* FetchHTTPTest.java
a bunch of new tests
* FetchHTTP.java
make sslTrustLevel work correctly (bug found by test; even with fix, only respects value at startup, no overrides)
* modules/pom.xml, .classpath
dependencies for testing
* engine/pom.xml
update jetty to 6.1.26 since we require that for the new tests
* CrawlURI.java
utility method hasContentDigestHistory()
* BdbContentDigestHistory.java
load() - make sure to call curi.getContentDigestHistory() in all cases so the value is initialized and WARCWriterProcessor knows it should put the info in there
* WARCWriterProcessor.java
updateMetadataAfterWrite() - update curi.getContentDigestHistory() only if curi.hasContentDigestHistory() for efficiency, like old uri-based fetch history; update the count after writing a revisit record
writeRevisitUriAgnosticDigest() - write several separate warc header fields for each value instead of one json blob
* WARCConstants.java
new header field names
remove rfc6648-deprecated "X-" prefix to "WARC-Reference-Location", and document the field
* WARCWriterProcessor.java
writeRevisitUriAgnosticDigest() - add some comments
* WARCConstants.java
new header keys "WARC-Refers-To" and "X-WARC-Reference-Location"
* CrawlURI.java
getContentDigestHistory() - return HashMap instead of Map so clone() can be called on the return value
* WARCWriterProcessor.java
writeHttpRecords(), writeRevisitUriAgnosticDigest() - write revisit record for URI-agnostic content digest duplicate if enabled
* WARCRecordInfo.java
new fields warcFilename, warcFileOffset
* WARCWriter.java
temporarily remember warc records written
* CrawlURI.java
convenience method getContentDigestHistory()
* ContentDigestHistory.java
start up properly; load into and store from curi.getContentDigestHistory()
* RecrawlAttributeConstants.java
bunch of hash keys for curi.getContentDigestHistory()
* WARCWriterProcessor.java
save info in curi.getContentDigestHistory()
* ANVLRecord.java
extend LinkedList instead of ArrayList (it was inane to guess the number of records everywhere, when it was never much more than ~10)
* ANVLRecords.java
remove (seemed to be an incomplete duplicate of ANVLRecord)
* WARCRecordInfo.java
addExtraHeader() - convenience method
* Arc2Warc.java, WARCWriter.java, WARCWriter.java, WARCWriterProcessor.java
use WARCRecordInfo.addExtraHeader() where appropriate, and use no-argument ANVLRecord constructor instead of deprecated ANVLRecord(int)
* WARCRecordInfo.java
get rid of constructor with arguments, add setUrl() and setType()
* WARCWriter.java
get rid of writeRecord() taking a million arguments
* Arc2Warc.java, WARCWriter.java, WARCWriterTest.java, WARCWriterProcessor.java, WARCWriterProcessorTest.java
construct WARCRecordInfo object and call WARCWriter.writeRecord(recordInfo) instead of million argument method
* WARCRecordInfo.java
new class to hold warc record info to write, so we don't have to pass around all these long lists of variables
* WARCConstants.java
new enum WARCRecordType replaces bunch of string constants
* WARCWriter.java, WARCWriterProcessor.java, WARCWriterTest.java
use WARCRecordInfo, WARCRecordType
* CrawlURI.java
getAnnotations() - use LinkedHashSet data structure for new annotations object to prevent duplicates
* Link.java
addRelativeToVia() - only log use of base for via the first time, before the annotation has been added
* CoreAttributeConstants.java
new constant A_HTTP_AUTH_CHALLENGES key for CrawlURI.getData() for http auth challenges
* CrawlURI.java
add A_HTTP_AUTH_CHALLENGES to keys to remember across URI processing; convenience getter and setter for auth challenges
* Credential.java, HtmlFormCredential.java, HttpAuthenticationCredential.java
add httpAuthChallenges argument to populate()
* HttpAuthenticationCredential.java
populate() - process httpAuthChallenges to set up auth state of http client to send the right auth header
* CrawlServer.java
cache httpAuthChallenges
* FetchHTTP.java
getAuthScheme() - cache auth challenges with CrawlURI
configureHttp() - do not set authentication preemptive - this is handled by the configuration done in HttpAuthenticationCredential.populate()
promoteCredentials() - cache auth challenges with the CrawlServer
* FetchHTTP.java
configureHttp() - enable preemptive authentication parameter
innerProcess() - do not call method.setDoAuthentication() (it's for automatic 401 handling which is not what we want)
* TooManyPathSegmentsDecideRule
Sometimes the path is null, throwing a NPE when the loop in evaluate(CrawlURI)
is entered. In such a case, revert to previous behavior.
* TooManyPathSegmentsDecideRule
evaluate(CrawlURI) previously counted slashes in the full URI.
Now it only counts slashes in the path of the URI, meaning a URI
with slashes in the query string will not be counted as having more
path segments than it has in its path.
* WARCWriterProcessor.java
writeMetadata() - if extractors have left annotations about the encoding used to interpret content, include them in the metadata (HER-1987)
* MultiReporter.java
deprecated
* ToePool, ToeThread, WorkQueue, WorkQueueFrontier, PrecedenceProvider, CrawlURI, FetchStats.java
remove MultiReporter#getReports() and reportTo(String, PrintWriter)
these are not used as MultiReporter specifies, and MultiReporter is going away anyhow
Previously the whole method was synchronized even if threadlocals
were used (which are thread safe). And previously the sharedEngine
was set and returned only when it /was not/ needed ie when
getIsolateThreads() returned true.
Now getEngine does what it says it does.
Noah Levitt