Commit Graph
879 Commits
Author SHA1 Message Date
Noah Levitt 98a4be5eb1 fix user-agent test, add more tests 2012-06-29 07:05:44 -07:00
Noah Levitt 1a79348cf7 test a few more things 2012-06-29 02:55:26 -07:00
Noah Levitt 1a1d1b008c allow FetchHTTP2 infrastructure to fall back on default java hostname resolution 2012-06-29 02:40:34 -07:00
Noah Levitt 1f315a7cc7 unit tests for FetchHTTP and FetchHTTP2 2012-06-29 02:04:39 -07:00
Noah Levitt 2653583cdd unit tests for FetchHTTP and FetchHTTP2 2012-06-29 01:59:58 -07:00
Noah Levitt 58d827c85b Merge branch 'master' into fetchhttp-hc 2012-06-28 18:08:39 -07:00
Noah Levitt 6225f6fa0b Fix HER-2013 scheme-less relative URI not resolved correctly
* LaxURI.java
    parseUriReference() - fix logic testing if uri string is a path to respect scheme-less uris even if they contain colons
* UURIFactoryTest.java
    add test for scheme-less relative uri (with colons)
2012-06-28 16:49:50 -07:00
Noah Levitt 63c6ca59c2 Merge branch 'master' into fetchhttp-hc 2012-06-25 16:42:34 -07:00
Travis Wellman c40c24e97d Merge pull request #12 from travisfw/generics-fixes
generics warnings fixes
2012-06-25 11:37:18 -07:00
Travis Wellman 6626b44286 generics warnings fixes
This commit does not change any functionality, just
alters generics to reduce warnings in Eclipse.
2012-06-21 18:23:06 -07:00
Noah Levitt 2b0141fd0d bdb cookie store for FetchHTTP2 2012-06-20 18:49:57 -07:00
Noah Levitt 49c00c0387 do not rely on ToeThread for the ServerCache, pass it to where it's needed instead 2012-06-19 13:54:05 -07:00
Noah Levitt 56f0c92690 resolve hostnames from heritrix server cache (the bad old way) 2012-06-19 13:32:34 -07:00
Noah Levitt 1d0beedf2b use http 1.0 by default, other request configuration stuff 2012-06-19 12:09:46 -07:00
Noah Levitt 233b72cb34 get charset, encodings from headers 2012-06-19 11:28:01 -07:00
Noah Levitt de152eaa9e browser-compatibility mode for cookie matching; that's what old FetchHTTP does, and also avoids sending weird Cookie2:$Version1 2012-06-18 17:49:09 -07:00
Noah Levitt 3c7bd8afd6 no keep-alive 2012-06-18 17:06:46 -07:00
Noah Levitt a779e46ac7 set user-agent 2012-06-18 15:55:07 -07:00
Noah Levitt 7422a7b992 move httpclient helper classes to commons/ 2012-06-18 15:43:52 -07:00
Noah Levitt 7f0efbdac8 do recording through SessionOutputBuffer implementation like we have for SessionInputBuffer; refactor helper classes into their own file RecordingHttpClient.java 2012-06-18 15:42:50 -07:00
Noah Levitt 0d172be888 not using HcInputWrapper anymore 2012-06-18 12:31:52 -07:00
Noah Levitt 316391786e license headers 2012-06-17 23:23:39 -07:00
Noah Levitt 6c56ce41a0 different deeper way to insert our Recorder, letting us correctly note end of http headers. is there a problem with not buffering at this level? 2012-06-17 15:48:19 -07:00
Noah Levitt b27b3f67e5 continuing to mess with httpcomponents 2012-06-15 18:29:21 -07:00
Noah Levitt 9a5570c6f1 continuing to mess with httpcomponents 2012-06-15 16:34:55 -07:00
Noah Levitt 3a0f50d031 FetchHTTP2 - record, archive request as well 2012-06-14 19:01:15 -07:00
Noah Levitt 5043dcc35b FetchHTTP2, based on httpcomponents, is working in a rudimentary way. This commit also gets rid of most egregious commons-httpclient-specific code in CrawlURI 2012-06-14 18:34:43 -07:00
Noah Levitt 793d733f1e continuing to mess with httpcomponents 2012-06-14 14:00:16 -07:00
Noah Levitt e5668726a1 starting to mess with httpcomponents 2012-06-13 14:11:11 -07:00
Noah Levitt 6ab7415342 Revert "* ConfigFileEditor.java, ConfigPathEditor.java"
This reverts commit 6dae8f9a17.

Revert to fix this problem: ... Failed to convert property value of type 'java.lang.String' to required type 'org.archive.spring.ConfigPath' for property 'path'; nested exception is java.lang.IllegalStateException: Cannot convert value of type [java.lang.String] to required type [org.archive.spring.ConfigPath] for property 'path': no matching editors or conversion strategy found
2012-06-12 10:15:27 -07:00
Noah Levitt 6dae8f9a17 * ConfigFileEditor.java, ConfigPathEditor.java
remove unused classes
2012-06-11 17:21:23 -07:00
Noah Levitt e1f5bc5319 HER-1987 record charset used for link extraction in warc metadata record
* WARCWriterProcessor.java
    writeMetadata() - write metadata entry charsetForLinkExtraction
2012-06-06 12:02:41 -07:00
Noah Levitt 7eabc7ef73 HER-1987 record charset used for link extraction in warc metadata record
* WARCWriterProcessor.java
    writeMetadata() - if extractors have left annotations about the encoding used to interpret content, include them in the metadata (HER-1987)
2012-06-06 11:40:58 -07:00
Noah Levitt 99acaa9bfa Fix tests revealed to be failing
* UriUtils.java
    isLikelyFalsePositive() - do not flag urls starting with ../ as likely false positives
* UriUtilsTest.java
    remove test url that is now rejected
2012-06-06 09:00:13 -07:00
Noah Levitt 342ea33be5 Move test classes into test area of source tree
* LongToIntConsistentHashTest.java, UriUtilsTest.java
2012-06-05 18:03:33 -07:00
Noah Levitt 35378df2fa Merge branch 'master' into no-package-scope 2012-06-05 17:58:40 -07:00
Noah Levitt 3dce1e909d Change scope of most/all default-scope i.e. package-scoped class members to protected, to make heritrix more friendly to third-party extensions. (I don't think there was ever an intent to limit the scope of any of these to package-only.) 2012-06-05 17:55:41 -07:00
Noah Levitt e965421281 * WorkQueueFrontier.java
remove unused variable
2012-06-05 16:26:00 -07:00
Travis Wellman ee1e25ca69 Merge pull request #11 from travisfw/new-reports
deleting MultiReporter interface
2012-06-05 16:24:43 -07:00
Travis Wellman 07a2fc65ea use methods to specify which report rather than string constants
* MultiReporter
    deleted. direct method calls are a better way to specify the kind of report desired
    getReports was never used to choose from the array of strings. no use case for MultiReporter.
* ArchiveUtils
    writeReportToString was not used, and it used the now deleted MultiReporter#reportTo
* DevUtils, Frontier, ToePool, AbstractFrontier, WorkQueueFrontier
    MultiReporter -> Reporter and make methods that implemented named multireport public
* FrontierNonemptyReport, FrontierSummaryReport
    call the appropriate report method instead of passing a string constant
2012-06-01 17:57:25 -07:00
Travis Wellman 493e2423f6 begin removing MultiReporter
* MultiReporter.java
    deprecated
* ToePool, ToeThread, WorkQueue, WorkQueueFrontier, PrecedenceProvider, CrawlURI, FetchStats.java
    remove MultiReporter#getReports() and reportTo(String, PrintWriter)
    these are not used as MultiReporter specifies, and MultiReporter is going away anyhow
2012-06-01 17:14:41 -07:00
Noah Levitt 3bebf10674 * ScriptedDecideRule.java
getEngine() - tiny cosmetic fixups
2012-05-31 16:29:33 -07:00
Travis Wellman 392a4b3fd0 correct synchronization and threadlocal logic
Previously the whole method was synchronized even if threadlocals
were used (which are thread safe). And previously the sharedEngine
was set and returned only when it /was not/ needed ie when
getIsolateThreads() returned true.

Now getEngine does what it says it does.
2012-05-31 16:09:42 -07:00
Noah Levitt 1015c5fa00 Fix HER-1973 Regression in queue budget overrides
* WorkQueueFrontier.java
    findEligibleURI() - apply sheet overrides before refreshing queue budgets
2012-05-15 17:35:49 -07:00
Noah Levitt 809981ef02 * PersistProcessor.java
populatePersistEnvFromLog() - handle exception deserializing line
2012-05-10 19:00:27 -07:00
Noah Levitt 9fffa8fc53 **/pom.xml
bump version to 3.1.2-SNAPSHOT
2012-05-01 18:17:33 -07:00
Noah Levitt 43d57dcaf4 * README.txt
update for 3.1.1
3.1.1
2012-05-01 18:05:38 -07:00
Noah Levitt 79c40346fc **/pom.xml
* bump version number to 3.1.1 for release
2012-05-01 18:01:06 -07:00
Noah Levitt a7c63b8d00 * FetchHTTP.java
remove redundant nonsensical comment
2012-05-01 17:53:52 -07:00
Gordon Mohr e4f12bb20f Merge pull request #6 from ato/codemirror
HER-2001: Use the CodeMirror editor for crawl config and script console
    (Thanks Alex!)
2012-04-25 15:04:10 -07:00