lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-23 22:26:01 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,156 Commits 38 Branches 33 Tags
b84d443f00a75d68fa022c61ba5b8a106bd6344a
Commit Graph
839 Commits
Author SHA1 Message Date
Noah Levitt 4470ba512e Count url-agnostic content digest duplicates as duplicates in statistics 
* CrawledBytesHistotable.java
    accumulate() - count urls with warcRevisit:uriAgnosticDigest annotation as duplicates
* ContentDigestHistoryTest.java
    testWarcDedupe() - check for presence of warcRevisit:uriAgnosticDigest annotation
 2012-09-27 16:56:14 -07:00
Noah Levitt a1d135edbc * WARCWriterProcessor.java 
include warcFileOffset in CrawlURI extraInfo (optionally logged in crawl log if logExtraInfo is enabled)
 2012-09-27 16:51:50 -07:00
Noah Levitt e2a1e3bb53 * ContentDigestHistoryTest.java 
testWarcDedupe() - open the written with with WARCReader, check that each record is as expected
 2012-09-27 14:34:20 -07:00
Noah Levitt a1a5fc7379 * WARCWriterProcessor.java 
writeRevisitUriAgnosticDigest() - put angle brackets around WARC-Refers-To uri value, since warc record ids appear everywhere with those brackets
 2012-09-27 14:20:00 -07:00
Noah Levitt 7a92e1ff7c * FetchHTTPTest.java 
shut down test servers in tearDown() (server startup/shutdown happens on every test, inefficient but it doesn't take all that long anyway)
* ContentDigestHistoryTest.java
    shut down test servers in finally block
 2012-09-26 19:29:11 -07:00
Noah Levitt e6fbe45774 Another test for ContentDigestHistory 
* ContentDigestHistoryTest.java
    new test testWarcDedupe() fetches two different urls with same content, writes warc records, checks results
* FetchHTTPTest.java, WARCWriterProcessorTest.java
    make test module code public and static for use in ContentDigestHistoryTest.java
 2012-09-26 18:54:48 -07:00
Noah Levitt b9cbbbd248 Bunch of unit tests for FetchHTTP 
* FetchHTTPTest.java
    a bunch of new tests
* FetchHTTP.java
    make sslTrustLevel work correctly (bug found by test; even with fix, only respects value at startup, no overrides)
* modules/pom.xml, .classpath
    dependencies for testing
* engine/pom.xml
    update jetty to 6.1.26 since we require that for the new tests
 2012-09-26 14:41:49 -07:00
Noah Levitt f0b4cfeeac Merge branch 'master' into new-fetchhttp-only 
Conflicts:
	modules/src/main/java/org/archive/modules/CrawlURI.java
	modules/src/main/java/org/archive/modules/writer/WARCWriterProcessor.java
 2012-09-26 12:59:41 -07:00
Noah Levitt 3612396ff3 Unit test for content digest history 
* ContentDigestHistoryTest.java
    test basics
* BdbContentDigestHistory.java
    code formatting tweak
 2012-09-26 12:03:45 -07:00
Noah Levitt 50c8a09b63 * BdbContentDigestHistory.java 
class javadoc
 2012-09-25 16:04:44 -07:00
Noah Levitt 819371f6eb * BdbContentDigestHistory.java 
store() - avoid clobbering good content digest history with empty one
 2012-09-25 15:28:19 -07:00
Noah Levitt 34252c4949 * WARCWriterProcessor.java 
writeHttpRecords() - avoid inadvertently creating content digest history map
 2012-09-25 15:26:56 -07:00
Noah Levitt fe1df5883f Some tweaks on maintaining content digest history 
* CrawlURI.java
    utility method hasContentDigestHistory()
* BdbContentDigestHistory.java
    load() - make sure to call curi.getContentDigestHistory() in all cases so the value is initialized and WARCWriterProcessor knows it should put the info in there
* WARCWriterProcessor.java
    updateMetadataAfterWrite() - update curi.getContentDigestHistory() only if curi.hasContentDigestHistory() for efficiency, like old uri-based fetch history; update the count after writing a revisit record
 2012-09-25 14:35:26 -07:00
Noah Levitt 25a9155298 New parent class AbstractContentDigestHistory for implementations to extend 
* AbstractContentDigestHistory.java
    abstract methods load(), store(); non-abstract persistKeyFor(); javadocs
* BdbContentDigestHistory.java
    extend AbstractContentDigestHistory
* ContentDigestHistoryLoader.java, ContentDigestHistoryStorer.java
    use AbstractContentDigestHistory
 2012-09-25 13:38:25 -07:00
Noah Levitt b8e14ba1df Rename ContentDigestHistory -> BdbContentDigestHistory, since it's bdb-specific 2012-09-25 13:12:59 -07:00
Noah Levitt 7965a27b1d Fix infinite loop bug on server returning no response 
* RecordingSocketInputBuffer.java
    readLine(CharArrayBuffer) - return -1 (not 0) on end of stream
* RecordingHttpClient.java
    setHttpRequestRetryHandler() - never retry (may want to reevaluate later, old fetcher did retry sometimes)
* FetchHTTPTest.java
    testNoResponse()
 2012-09-13 19:10:11 -07:00
Noah Levitt 77559bf81c **/pom.xml 
bump version number because of api change
 2012-09-13 17:47:26 -07:00
Noah Levitt ef78e584c4 * WARCWriterProcessor.java 
writeRevisitUriAgnosticDigest() - write several separate warc header fields for each value instead of one json blob
* WARCConstants.java
    new header field names
 2012-09-13 17:21:29 -07:00
Noah Levitt a8aae91167 * WARCConstants.java 
remove rfc6648-deprecated "X-" prefix to "WARC-Reference-Location", and document the field
* WARCWriterProcessor.java
    writeRevisitUriAgnosticDigest() - add some comments
 2012-09-11 18:21:20 -07:00
Noah Levitt 1048536a25 * WARCWriterProcessor.java 
writeRevisitUriAgnosticDigest() - fix construction of X-WARC-Reference-Location json value
 2012-09-11 17:21:34 -07:00
Noah Levitt 38aefe4d16 Write revisit records for URI-agnostic content digest duplicates. 
* WARCConstants.java
    new header keys "WARC-Refers-To" and "X-WARC-Reference-Location"
* CrawlURI.java
    getContentDigestHistory() - return HashMap instead of Map so clone() can be called on the return value
* WARCWriterProcessor.java
    writeHttpRecords(), writeRevisitUriAgnosticDigest() - write revisit record for URI-agnostic content digest duplicate if enabled
 2012-09-11 17:12:24 -07:00
Noah Levitt a04e76f00f Load, store content digest history 
* WARCRecordInfo.java
    new fields warcFilename, warcFileOffset
* WARCWriter.java
    temporarily remember warc records written
* CrawlURI.java
    convenience method getContentDigestHistory()
* ContentDigestHistory.java
    start up properly; load into and store from curi.getContentDigestHistory()
* RecrawlAttributeConstants.java
    bunch of hash keys for curi.getContentDigestHistory()
* WARCWriterProcessor.java
    save info in curi.getContentDigestHistory()
 2012-09-11 10:10:49 -07:00
Noah Levitt 2407fc5dd0 Beginning work on deduplication by content digest (irrespective of url) 
* ContentDigestHistory.java, ContentDigestHistoryLoader.java, ContentDigestHistoryStorer.java
    initial checkin
 2012-09-10 18:18:12 -07:00
Noah Levitt a414f522f6 Continuing to work on cleaning up WARC writer code: simply creation of WARC record headers. 
* ANVLRecord.java
    extend LinkedList instead of ArrayList (it was inane to guess the number of records everywhere, when it was never much more than ~10)
* ANVLRecords.java
    remove (seemed to be an incomplete duplicate of ANVLRecord)
* WARCRecordInfo.java
    addExtraHeader() - convenience method
* Arc2Warc.java, WARCWriter.java, WARCWriter.java, WARCWriterProcessor.java
    use WARCRecordInfo.addExtraHeader() where appropriate, and use no-argument ANVLRecord constructor instead of deprecated ANVLRecord(int)
 2012-09-10 18:03:25 -07:00
Noah Levitt 360ac00000 Continuing to work on cleaning up WARC writer code. 
* WARCRecordInfo.java
    get rid of constructor with arguments, add setUrl() and setType()
* WARCWriter.java
    get rid of writeRecord() taking a million arguments
* Arc2Warc.java, WARCWriter.java, WARCWriterTest.java, WARCWriterProcessor.java, WARCWriterProcessorTest.java
    construct WARCRecordInfo object and call WARCWriter.writeRecord(recordInfo) instead of million argument method
 2012-09-10 17:41:43 -07:00
Noah Levitt dd5e51fd9d Working on cleaning up WARC writer code. 
* WARCRecordInfo.java
    new class to hold warc record info to write, so we don't have to pass around all these long lists of variables
* WARCConstants.java
    new enum WARCRecordType replaces bunch of string constants
* WARCWriter.java, WARCWriterProcessor.java, WARCWriterTest.java
    use WARCRecordInfo, WARCRecordType
 2012-09-10 15:43:37 -07:00
Noah Levitt 9699d6a6ed get rid of AbstractFetchHTTP 2012-09-08 01:17:32 -07:00
Noah Levitt 33df8faaf0 remove LegacyFetchHTTP and its classes 2012-09-08 00:49:30 -07:00
Noah Levitt 71761a72a7 rename FetchHTTP2 to FetchHTTP 2012-09-08 00:28:06 -07:00
Noah Levitt bc29599a3b oops forgot to add LegacyFetchHTTP.java ? 2012-09-08 00:27:14 -07:00
Noah Levitt 917986454b rename FetchHTTP to LegacyFetchHTTP 2012-09-08 00:25:14 -07:00
Noah Levitt d3acf337cc use ThreadLocal<HttpClient> since the connection manager doesn't like being shared among threads, and who knows what else; basically it seems to be designed for use by a single thread 2012-09-08 00:11:43 -07:00
Noah Levitt ec7c525c9d use Integer instead of int because null is allowed 2012-09-07 23:19:52 -07:00
Noah Levitt 24e813c858 test chunked transfer encoding 2012-09-07 13:28:41 -07:00
Noah Levitt 09f0e53119 testHttp11 2012-09-07 11:32:38 -07:00
Noah Levitt a346389a3a twiddles 2012-09-07 03:10:53 -07:00
Noah Levitt 9a0191a84c ssl trust level support for FetchHTTP2! another thing that doesn't even work properly in old FetchHTTP 2012-09-07 02:47:54 -07:00
Noah Levitt da76fe322a Merge branch 'master' into fetchhttp-hc 2012-09-06 15:51:36 -07:00
Noah Levitt 5b82d9abee Merge branch 'master' of github.com:internetarchive/heritrix3 2012-09-06 15:15:03 -07:00
Noah Levitt 5b35835d53 Unsupported auth scheme causing recoverableProblem with SEVERE level log line and stack trace 
* FetchHTTP.java
    getAuthScheme() - handle IllegalStateException thrown by AuthPolicy.getAuthScheme()
 2012-09-06 15:12:56 -07:00
Noah Levitt 948d641b14 socket/connection timeout 2012-09-06 12:39:44 -07:00
Noah Levitt d11338d620 testFetchTimeout 2012-09-05 19:24:50 -07:00
Noah Levitt fbd25b789a implement shouldFetchBodyRule 2012-09-05 14:19:17 -07:00
Noah Levitt d703fce31a add missing license header 2012-09-05 14:08:26 -07:00
Noah Levitt b19994e3bb Merge branch 'master' into fetchhttp-hc 2012-09-05 12:08:48 -07:00
Noah Levitt 1a07c6b741 Make unit tests pass 
* ContentExtractorTestBase.java
    assertNoSideEffects() - check curi.getAnnotations.isEmpty() instead of equality with Collections.EMPTY_LIST
 2012-09-05 11:19:18 -07:00
Noah Levitt cd0c169a4d Merge branch 'master' into fetchhttp-hc 2012-09-04 17:54:03 -07:00
Noah Levitt b704f4eeb7 Fix HER-2021 usedBaseForVia annotation repeated many times 
* CrawlURI.java
    getAnnotations() - use LinkedHashSet data structure for new annotations object to prevent duplicates
* Link.java
    addRelativeToVia() - only log use of base for via the first time, before the annotation has been added
 2012-09-04 17:49:01 -07:00
Noah Levitt baaf4f8092 testShouldFetchBodyRule 2012-09-04 13:25:51 -07:00
Noah Levitt ca606588ae test sendIfNoneMatch (etag) 2012-09-02 15:25:56 -07:00