createRecorder(String,String) - new method for creating test Recorder and specifying charset
createRecorder(String) - deprecate (use default charset as before)
shut down test servers in tearDown() (server startup/shutdown happens on every test, inefficient but it doesn't take all that long anyway)
* ContentDigestHistoryTest.java
shut down test servers in finally block
* ContentDigestHistoryTest.java
new test testWarcDedupe() fetches two different urls with same content, writes warc records, checks results
* FetchHTTPTest.java, WARCWriterProcessorTest.java
make test module code public and static for use in ContentDigestHistoryTest.java
* FetchHTTPTest.java
a bunch of new tests
* FetchHTTP.java
make sslTrustLevel work correctly (bug found by test; even with fix, only respects value at startup, no overrides)
* modules/pom.xml, .classpath
dependencies for testing
* engine/pom.xml
update jetty to 6.1.26 since we require that for the new tests
* CrawlURI.java
utility method hasContentDigestHistory()
* BdbContentDigestHistory.java
load() - make sure to call curi.getContentDigestHistory() in all cases so the value is initialized and WARCWriterProcessor knows it should put the info in there
* WARCWriterProcessor.java
updateMetadataAfterWrite() - update curi.getContentDigestHistory() only if curi.hasContentDigestHistory() for efficiency, like old uri-based fetch history; update the count after writing a revisit record
writeRevisitUriAgnosticDigest() - write several separate warc header fields for each value instead of one json blob
* WARCConstants.java
new header field names
remove rfc6648-deprecated "X-" prefix to "WARC-Reference-Location", and document the field
* WARCWriterProcessor.java
writeRevisitUriAgnosticDigest() - add some comments
* WARCConstants.java
new header keys "WARC-Refers-To" and "X-WARC-Reference-Location"
* CrawlURI.java
getContentDigestHistory() - return HashMap instead of Map so clone() can be called on the return value
* WARCWriterProcessor.java
writeHttpRecords(), writeRevisitUriAgnosticDigest() - write revisit record for URI-agnostic content digest duplicate if enabled
* WARCRecordInfo.java
new fields warcFilename, warcFileOffset
* WARCWriter.java
temporarily remember warc records written
* CrawlURI.java
convenience method getContentDigestHistory()
* ContentDigestHistory.java
start up properly; load into and store from curi.getContentDigestHistory()
* RecrawlAttributeConstants.java
bunch of hash keys for curi.getContentDigestHistory()
* WARCWriterProcessor.java
save info in curi.getContentDigestHistory()
* ANVLRecord.java
extend LinkedList instead of ArrayList (it was inane to guess the number of records everywhere, when it was never much more than ~10)
* ANVLRecords.java
remove (seemed to be an incomplete duplicate of ANVLRecord)
* WARCRecordInfo.java
addExtraHeader() - convenience method
* Arc2Warc.java, WARCWriter.java, WARCWriter.java, WARCWriterProcessor.java
use WARCRecordInfo.addExtraHeader() where appropriate, and use no-argument ANVLRecord constructor instead of deprecated ANVLRecord(int)
* WARCRecordInfo.java
get rid of constructor with arguments, add setUrl() and setType()
* WARCWriter.java
get rid of writeRecord() taking a million arguments
* Arc2Warc.java, WARCWriter.java, WARCWriterTest.java, WARCWriterProcessor.java, WARCWriterProcessorTest.java
construct WARCRecordInfo object and call WARCWriter.writeRecord(recordInfo) instead of million argument method
* WARCRecordInfo.java
new class to hold warc record info to write, so we don't have to pass around all these long lists of variables
* WARCConstants.java
new enum WARCRecordType replaces bunch of string constants
* WARCWriter.java, WARCWriterProcessor.java, WARCWriterTest.java
use WARCRecordInfo, WARCRecordType
* CrawlURI.java
getAnnotations() - use LinkedHashSet data structure for new annotations object to prevent duplicates
* Link.java
addRelativeToVia() - only log use of base for via the first time, before the annotation has been added
* CoreAttributeConstants.java
new constant A_HTTP_AUTH_CHALLENGES key for CrawlURI.getData() for http auth challenges
* CrawlURI.java
add A_HTTP_AUTH_CHALLENGES to keys to remember across URI processing; convenience getter and setter for auth challenges
* Credential.java, HtmlFormCredential.java, HttpAuthenticationCredential.java
add httpAuthChallenges argument to populate()
* HttpAuthenticationCredential.java
populate() - process httpAuthChallenges to set up auth state of http client to send the right auth header
* CrawlServer.java
cache httpAuthChallenges
* FetchHTTP.java
getAuthScheme() - cache auth challenges with CrawlURI
configureHttp() - do not set authentication preemptive - this is handled by the configuration done in HttpAuthenticationCredential.populate()
promoteCredentials() - cache auth challenges with the CrawlServer
* FetchHTTP.java
configureHttp() - enable preemptive authentication parameter
innerProcess() - do not call method.setDoAuthentication() (it's for automatic 401 handling which is not what we want)
Noah Levitt