testLaxUrlEncoding() - Tests a URL not correctly url-encoded, but that heritrix should let pass through to mimic browser behavior (test fails currently)
avoid redundancy by using extractor built in setUp()
makeExtractor() - call extractor.setExtractorJS(new ExtractorJS()) since we got rid of the static ExtractorJS stuff
testConditionalComment1() - override to skip the test since it fails with JerichoExtractorHTML
* UriUtils.java
new method isVeryLikelyUri() with tighter heuristic than isLikelyUri()
* ExtractorJS.java
use UriUtils.isVeryLikelyUri(), and change order of operations to do fixup before call to isVeryLikelyUri(), since it doesn't expect strings with javascript escaping and stuff
* StringExtractorTestBase.java
handle test data with expected value null, meaning no outlinks expected
* ExtractorHTMLTest.java
avoid redundancy by using the extractor created in ContentExtractorTestBase.setUp()
* ExtractorJSTest.java
some new tests
refactor so considerStrings() is not static, allowing it to be overridden in subclasses
* ExtractorHTML.java, ExtractorJS.java
add @Autowired parameter extractorJS used to process inline javascript, instead of call to static ExtractorJS.considerStrings()
move to modules/src/test
testFormAuth() - remove, there is FormAuthSelfTest.java, and this was the test that would fail in modules/
testHttpBindAddress() - incorporate ed65493db0 - alternate testing of multiple local binds more likely to work more places
createRecorder(String,String) - new method for creating test Recorder and specifying charset
createRecorder(String) - deprecate (use default charset as before)
shut down test servers in tearDown() (server startup/shutdown happens on every test, inefficient but it doesn't take all that long anyway)
* ContentDigestHistoryTest.java
shut down test servers in finally block
* ContentDigestHistoryTest.java
new test testWarcDedupe() fetches two different urls with same content, writes warc records, checks results
* FetchHTTPTest.java, WARCWriterProcessorTest.java
make test module code public and static for use in ContentDigestHistoryTest.java
* FetchHTTPTest.java
a bunch of new tests
* FetchHTTP.java
make sslTrustLevel work correctly (bug found by test; even with fix, only respects value at startup, no overrides)
* modules/pom.xml, .classpath
dependencies for testing
* engine/pom.xml
update jetty to 6.1.26 since we require that for the new tests
* CrawlURI.java
utility method hasContentDigestHistory()
* BdbContentDigestHistory.java
load() - make sure to call curi.getContentDigestHistory() in all cases so the value is initialized and WARCWriterProcessor knows it should put the info in there
* WARCWriterProcessor.java
updateMetadataAfterWrite() - update curi.getContentDigestHistory() only if curi.hasContentDigestHistory() for efficiency, like old uri-based fetch history; update the count after writing a revisit record
* RecordingSocketInputBuffer.java
readLine(CharArrayBuffer) - return -1 (not 0) on end of stream
* RecordingHttpClient.java
setHttpRequestRetryHandler() - never retry (may want to reevaluate later, old fetcher did retry sometimes)
* FetchHTTPTest.java
testNoResponse()
writeRevisitUriAgnosticDigest() - write several separate warc header fields for each value instead of one json blob
* WARCConstants.java
new header field names
Noah Levitt