* CrawlURI.java
utility method hasContentDigestHistory()
* BdbContentDigestHistory.java
load() - make sure to call curi.getContentDigestHistory() in all cases so the value is initialized and WARCWriterProcessor knows it should put the info in there
* WARCWriterProcessor.java
updateMetadataAfterWrite() - update curi.getContentDigestHistory() only if curi.hasContentDigestHistory() for efficiency, like old uri-based fetch history; update the count after writing a revisit record
writeRevisitUriAgnosticDigest() - write several separate warc header fields for each value instead of one json blob
* WARCConstants.java
new header field names
remove rfc6648-deprecated "X-" prefix to "WARC-Reference-Location", and document the field
* WARCWriterProcessor.java
writeRevisitUriAgnosticDigest() - add some comments
* WARCConstants.java
new header keys "WARC-Refers-To" and "X-WARC-Reference-Location"
* CrawlURI.java
getContentDigestHistory() - return HashMap instead of Map so clone() can be called on the return value
* WARCWriterProcessor.java
writeHttpRecords(), writeRevisitUriAgnosticDigest() - write revisit record for URI-agnostic content digest duplicate if enabled
* WARCRecordInfo.java
new fields warcFilename, warcFileOffset
* WARCWriter.java
temporarily remember warc records written
* CrawlURI.java
convenience method getContentDigestHistory()
* ContentDigestHistory.java
start up properly; load into and store from curi.getContentDigestHistory()
* RecrawlAttributeConstants.java
bunch of hash keys for curi.getContentDigestHistory()
* WARCWriterProcessor.java
save info in curi.getContentDigestHistory()
* ANVLRecord.java
extend LinkedList instead of ArrayList (it was inane to guess the number of records everywhere, when it was never much more than ~10)
* ANVLRecords.java
remove (seemed to be an incomplete duplicate of ANVLRecord)
* WARCRecordInfo.java
addExtraHeader() - convenience method
* Arc2Warc.java, WARCWriter.java, WARCWriter.java, WARCWriterProcessor.java
use WARCRecordInfo.addExtraHeader() where appropriate, and use no-argument ANVLRecord constructor instead of deprecated ANVLRecord(int)
* WARCRecordInfo.java
get rid of constructor with arguments, add setUrl() and setType()
* WARCWriter.java
get rid of writeRecord() taking a million arguments
* Arc2Warc.java, WARCWriter.java, WARCWriterTest.java, WARCWriterProcessor.java, WARCWriterProcessorTest.java
construct WARCRecordInfo object and call WARCWriter.writeRecord(recordInfo) instead of million argument method
* WARCRecordInfo.java
new class to hold warc record info to write, so we don't have to pass around all these long lists of variables
* WARCConstants.java
new enum WARCRecordType replaces bunch of string constants
* WARCWriter.java, WARCWriterProcessor.java, WARCWriterTest.java
use WARCRecordInfo, WARCRecordType
* CrawlURI.java
getAnnotations() - use LinkedHashSet data structure for new annotations object to prevent duplicates
* Link.java
addRelativeToVia() - only log use of base for via the first time, before the annotation has been added
* JobRelatedResource.java
Added encoding to URL generation in makePresentableMapFor().
Added check to prevent URL generation from overwriting the job name portion of the path in addPresentableNestedNames()
* CoreAttributeConstants.java
new constant A_HTTP_AUTH_CHALLENGES key for CrawlURI.getData() for http auth challenges
* CrawlURI.java
add A_HTTP_AUTH_CHALLENGES to keys to remember across URI processing; convenience getter and setter for auth challenges
* Credential.java, HtmlFormCredential.java, HttpAuthenticationCredential.java
add httpAuthChallenges argument to populate()
* HttpAuthenticationCredential.java
populate() - process httpAuthChallenges to set up auth state of http client to send the right auth header
* CrawlServer.java
cache httpAuthChallenges
* FetchHTTP.java
getAuthScheme() - cache auth challenges with CrawlURI
configureHttp() - do not set authentication preemptive - this is handled by the configuration done in HttpAuthenticationCredential.populate()
promoteCredentials() - cache auth challenges with the CrawlServer
* FetchHTTP.java
configureHttp() - enable preemptive authentication parameter
innerProcess() - do not call method.setDoAuthentication() (it's for automatic 401 handling which is not what we want)
* TooManyPathSegmentsDecideRule
Sometimes the path is null, throwing a NPE when the loop in evaluate(CrawlURI)
is entered. In such a case, revert to previous behavior.
* TooManyPathSegmentsDecideRule
evaluate(CrawlURI) previously counted slashes in the full URI.
Now it only counts slashes in the path of the URI, meaning a URI
with slashes in the query string will not be counted as having more
path segments than it has in its path.
* LaxURI.java
parseUriReference() - fix logic testing if uri string is a path to respect scheme-less uris even if they contain colons
* UURIFactoryTest.java
add test for scheme-less relative uri (with colons)
Noah Levitt