lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-25 16:06:33 +00:00
Code Issues Packages Projects Releases Wiki Activity
913 Commits 38 Branches 32 Tags
4470ba512efe5cdf786aaf9e90c087e00ea892ce
Commit Graph
913 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Noah Levitt 4470ba512e Count url-agnostic content digest duplicates as duplicates in statistics 
* CrawledBytesHistotable.java
    accumulate() - count urls with warcRevisit:uriAgnosticDigest annotation as duplicates
* ContentDigestHistoryTest.java
    testWarcDedupe() - check for presence of warcRevisit:uriAgnosticDigest annotation
 2012-09-27 16:56:14 -07:00
Noah Levitt a1d135edbc * WARCWriterProcessor.java 
include warcFileOffset in CrawlURI extraInfo (optionally logged in crawl log if logExtraInfo is enabled)
 2012-09-27 16:51:50 -07:00
Noah Levitt e2a1e3bb53 * ContentDigestHistoryTest.java 
testWarcDedupe() - open the written with with WARCReader, check that each record is as expected
 2012-09-27 14:34:20 -07:00
Noah Levitt a1a5fc7379 * WARCWriterProcessor.java 
writeRevisitUriAgnosticDigest() - put angle brackets around WARC-Refers-To uri value, since warc record ids appear everywhere with those brackets
 2012-09-27 14:20:00 -07:00
Noah Levitt 7a92e1ff7c * FetchHTTPTest.java 
shut down test servers in tearDown() (server startup/shutdown happens on every test, inefficient but it doesn't take all that long anyway)
* ContentDigestHistoryTest.java
    shut down test servers in finally block
 2012-09-26 19:29:11 -07:00
Noah Levitt e6fbe45774 Another test for ContentDigestHistory 
* ContentDigestHistoryTest.java
    new test testWarcDedupe() fetches two different urls with same content, writes warc records, checks results
* FetchHTTPTest.java, WARCWriterProcessorTest.java
    make test module code public and static for use in ContentDigestHistoryTest.java
 2012-09-26 18:54:48 -07:00
Noah Levitt f9703fb5ef * engine/pom.xml 
remove incorrect, unnecessary dependency
 2012-09-26 14:55:03 -07:00
Noah Levitt b9cbbbd248 Bunch of unit tests for FetchHTTP 
* FetchHTTPTest.java
    a bunch of new tests
* FetchHTTP.java
    make sslTrustLevel work correctly (bug found by test; even with fix, only respects value at startup, no overrides)
* modules/pom.xml, .classpath
    dependencies for testing
* engine/pom.xml
    update jetty to 6.1.26 since we require that for the new tests
 2012-09-26 14:41:49 -07:00
Noah Levitt 3612396ff3 Unit test for content digest history 
* ContentDigestHistoryTest.java
    test basics
* BdbContentDigestHistory.java
    code formatting tweak
 2012-09-26 12:03:45 -07:00
Noah Levitt 190e2ff0f3 Fix unit test 
* WARCWriterTest.java
    writeRandomHTTPRecord() - fix method, had transposed some lines of code
 2012-09-26 10:58:01 -07:00
Noah Levitt 2e1d81fa80 Fix build 
* WARCConstants.java, WARCWriter.java
    remove imports of WARCWriterProcessor from the modules/ area (was only used in javadoc)
 2012-09-26 08:58:22 -07:00
Noah Levitt 50c8a09b63 * BdbContentDigestHistory.java 
class javadoc
 2012-09-25 16:04:44 -07:00
Noah Levitt 819371f6eb * BdbContentDigestHistory.java 
store() - avoid clobbering good content digest history with empty one
 2012-09-25 15:28:19 -07:00
Noah Levitt 34252c4949 * WARCWriterProcessor.java 
writeHttpRecords() - avoid inadvertently creating content digest history map
 2012-09-25 15:26:56 -07:00
Noah Levitt fe1df5883f Some tweaks on maintaining content digest history 
* CrawlURI.java
    utility method hasContentDigestHistory()
* BdbContentDigestHistory.java
    load() - make sure to call curi.getContentDigestHistory() in all cases so the value is initialized and WARCWriterProcessor knows it should put the info in there
* WARCWriterProcessor.java
    updateMetadataAfterWrite() - update curi.getContentDigestHistory() only if curi.hasContentDigestHistory() for efficiency, like old uri-based fetch history; update the count after writing a revisit record
 2012-09-25 14:35:26 -07:00
Noah Levitt 25a9155298 New parent class AbstractContentDigestHistory for implementations to extend 
* AbstractContentDigestHistory.java
    abstract methods load(), store(); non-abstract persistKeyFor(); javadocs
* BdbContentDigestHistory.java
    extend AbstractContentDigestHistory
* ContentDigestHistoryLoader.java, ContentDigestHistoryStorer.java
    use AbstractContentDigestHistory
 2012-09-25 13:38:25 -07:00
Noah Levitt b8e14ba1df Rename ContentDigestHistory -> BdbContentDigestHistory, since it's bdb-specific 2012-09-25 13:12:59 -07:00
Noah Levitt ef78e584c4 * WARCWriterProcessor.java 
writeRevisitUriAgnosticDigest() - write several separate warc header fields for each value instead of one json blob
* WARCConstants.java
    new header field names
 2012-09-13 17:21:29 -07:00
Noah Levitt a8aae91167 * WARCConstants.java 
remove rfc6648-deprecated "X-" prefix to "WARC-Reference-Location", and document the field
* WARCWriterProcessor.java
    writeRevisitUriAgnosticDigest() - add some comments
 2012-09-11 18:21:20 -07:00
Noah Levitt 1048536a25 * WARCWriterProcessor.java 
writeRevisitUriAgnosticDigest() - fix construction of X-WARC-Reference-Location json value
 2012-09-11 17:21:34 -07:00
Noah Levitt 38aefe4d16 Write revisit records for URI-agnostic content digest duplicates. 
* WARCConstants.java
    new header keys "WARC-Refers-To" and "X-WARC-Reference-Location"
* CrawlURI.java
    getContentDigestHistory() - return HashMap instead of Map so clone() can be called on the return value
* WARCWriterProcessor.java
    writeHttpRecords(), writeRevisitUriAgnosticDigest() - write revisit record for URI-agnostic content digest duplicate if enabled
 2012-09-11 17:12:24 -07:00
Noah Levitt a04e76f00f Load, store content digest history 
* WARCRecordInfo.java
    new fields warcFilename, warcFileOffset
* WARCWriter.java
    temporarily remember warc records written
* CrawlURI.java
    convenience method getContentDigestHistory()
* ContentDigestHistory.java
    start up properly; load into and store from curi.getContentDigestHistory()
* RecrawlAttributeConstants.java
    bunch of hash keys for curi.getContentDigestHistory()
* WARCWriterProcessor.java
    save info in curi.getContentDigestHistory()
 2012-09-11 10:10:49 -07:00
Noah Levitt 2407fc5dd0 Beginning work on deduplication by content digest (irrespective of url) 
* ContentDigestHistory.java, ContentDigestHistoryLoader.java, ContentDigestHistoryStorer.java
    initial checkin
 2012-09-10 18:18:12 -07:00
Noah Levitt c4faa444a2 * ANVLRecord.java 
remove unneeded(?) serialVersionUID
 2012-09-10 18:14:15 -07:00
Noah Levitt a414f522f6 Continuing to work on cleaning up WARC writer code: simply creation of WARC record headers. 
* ANVLRecord.java
    extend LinkedList instead of ArrayList (it was inane to guess the number of records everywhere, when it was never much more than ~10)
* ANVLRecords.java
    remove (seemed to be an incomplete duplicate of ANVLRecord)
* WARCRecordInfo.java
    addExtraHeader() - convenience method
* Arc2Warc.java, WARCWriter.java, WARCWriter.java, WARCWriterProcessor.java
    use WARCRecordInfo.addExtraHeader() where appropriate, and use no-argument ANVLRecord constructor instead of deprecated ANVLRecord(int)
 2012-09-10 18:03:25 -07:00
Noah Levitt 360ac00000 Continuing to work on cleaning up WARC writer code. 
* WARCRecordInfo.java
    get rid of constructor with arguments, add setUrl() and setType()
* WARCWriter.java
    get rid of writeRecord() taking a million arguments
* Arc2Warc.java, WARCWriter.java, WARCWriterTest.java, WARCWriterProcessor.java, WARCWriterProcessorTest.java
    construct WARCRecordInfo object and call WARCWriter.writeRecord(recordInfo) instead of million argument method
 2012-09-10 17:41:43 -07:00
Noah Levitt dd5e51fd9d Working on cleaning up WARC writer code. 
* WARCRecordInfo.java
    new class to hold warc record info to write, so we don't have to pass around all these long lists of variables
* WARCConstants.java
    new enum WARCRecordType replaces bunch of string constants
* WARCWriter.java, WARCWriterProcessor.java, WARCWriterTest.java
    use WARCRecordInfo, WARCRecordType
 2012-09-10 15:43:37 -07:00
Noah Levitt 5b82d9abee Merge branch 'master' of github.com:internetarchive/heritrix3 2012-09-06 15:15:03 -07:00
Noah Levitt 5b35835d53 Unsupported auth scheme causing recoverableProblem with SEVERE level log line and stack trace 
* FetchHTTP.java
    getAuthScheme() - handle IllegalStateException thrown by AuthPolicy.getAuthScheme()
 2012-09-06 15:12:56 -07:00
Noah Levitt 1a07c6b741 Make unit tests pass 
* ContentExtractorTestBase.java
    assertNoSideEffects() - check curi.getAnnotations.isEmpty() instead of equality with Collections.EMPTY_LIST
 2012-09-05 11:19:18 -07:00
Noah Levitt e711ff9506 * JobRelatedResource.java 
fix code formatting from 01be6d4b
 2012-09-04 18:03:38 -07:00
Noah Levitt c433c78e90 Merge pull request #17 from adam-miller/HER-2018 
Fix for HER-2018: XML representation for /engine/job/<jobName>/beans returns incorrect url for named beans
 2012-09-04 17:59:43 -07:00
Noah Levitt b704f4eeb7 Fix HER-2021 usedBaseForVia annotation repeated many times 
* CrawlURI.java
    getAnnotations() - use LinkedHashSet data structure for new annotations object to prevent duplicates
* Link.java
    addRelativeToVia() - only log use of base for via the first time, before the annotation has been added
 2012-09-04 17:49:01 -07:00
Noah Levitt d0882a18e6 * AbstractPersistProcessor.java 
shouldStore() - only save persist record if http, https or ftp
 2012-09-01 22:09:30 -07:00
Adam Miller 01be6d4b18 Fix for HER-2018: XML representation for /engine/job/<jobName>/beans returns incorrect url for named beans 
* JobRelatedResource.java
    Added encoding to URL generation in makePresentableMapFor().
    Added check to prevent URL generation from overwriting the job name portion of the path in addPresentableNestedNames()
 2012-08-31 15:07:31 -07:00
Noah Levitt 61cbd54059 Merge pull request #16 from adam-miller/HER-2017 
Fix for HER-2017 XML representation of beans uses root node of type "script"
 2012-08-30 16:04:04 -07:00
Adam Miller 52e70a10ef Fix for HER-2017 XML representation for /engine/job/<jobName>/beans uses root node of type "script" instead of "beans" 
* BeanBrowseResource.java
	replaced "script" string in root node with "beans"
 2012-08-30 15:35:27 -07:00
Noah Levitt 0d65d7e446 Fix for HER-2019 fetchHTTP httpBindAddress setting doesn't take effect 
* FetchHTTP.java
    use keyed properties for httpBindAddress setting, as expected in configureBindAddress(CrawlURI, HostConfiguration)
 2012-08-28 16:03:17 -07:00
Noah Levitt a7a2b3cf7e Make speculative link extractor ignore urls with two consecutive slashes (not preceded by a colon as in http://). We've seen the double slashes lead to crawler traps. 
* UriUtils.java
    isLikelyFalsePositive() - consider likely false positive if it contains '//' (but not '://')
* UriUtilsTest.java
    testDoubleSlashFalsePositives()
 2012-08-24 14:03:45 -07:00
Noah Levitt f072dfc83c * Credential.java, HtmlFormCredential.java, HttpAuthenticationCredential.java 
bump serialVersionUID because of changed method signature on populate()
 2012-08-16 12:28:19 -07:00
Noah Levitt 0144df024e Real fix for HER-2016 FetchHTTP tries every url twice without credentials before sending credentials: Remember WWW-Authenticate headers, aka auth challenges, with CrawlURI, and after successful auth, with the CrawlServer. It's necessary to do this to know what Authorization header to send without getting another 401 first. 
* CoreAttributeConstants.java
    new constant A_HTTP_AUTH_CHALLENGES key for CrawlURI.getData() for http auth challenges
* CrawlURI.java
    add A_HTTP_AUTH_CHALLENGES to keys to remember across URI processing; convenience getter and setter for auth challenges
* Credential.java, HtmlFormCredential.java, HttpAuthenticationCredential.java
    add httpAuthChallenges argument to populate()
* HttpAuthenticationCredential.java
    populate() - process httpAuthChallenges to set up auth state of http client to send the right auth header
* CrawlServer.java
    cache httpAuthChallenges
* FetchHTTP.java
    getAuthScheme() - cache auth challenges with CrawlURI
    configureHttp() - do not set authentication preemptive - this is handled by the configuration done in HttpAuthenticationCredential.populate()
    promoteCredentials() - cache auth challenges with the CrawlServer
 2012-08-16 12:17:01 -07:00
Noah Levitt c57d3ba97a Merge pull request #15 from kngenie/reuse-htmllinkcontext 
Reuse htmllinkcontext
 2012-08-07 15:41:16 -07:00
Kenji Nagahashi 541c2ecfe4 Merge branch 'master' into reuse-htmllinkcontext 2012-07-31 12:25:16 -07:00
Kenji Nagahashi 3078eac4ab test case for scheme-less relative URL 2012-07-19 14:15:22 -07:00
Kenji Nagahashi 8fc44b626c added a test case for catching HER-2013 2012-07-19 14:15:22 -07:00
Noah Levitt 6f4053cab7 HER-2016 FetchHTTP tries every url twice without credentials before sending credentials 
* FetchHTTP.java
    configureHttp() - enable preemptive authentication parameter
    innerProcess() - do not call method.setDoAuthentication() (it's for automatic 401 handling which is not what we want)
 2012-07-19 13:15:27 -07:00
Noah Levitt 9073d12f96 Revert "HER-2015 credentials cached to server never used" 
This reverts commit 37a8504fda.
 2012-07-19 12:36:05 -07:00
Noah Levitt 37a8504fda HER-2015 credentials cached to server never used 
* FetchHTTP.java
    populateCredentials() - return true when credentials populated from server cache
 2012-07-16 20:02:56 -07:00
Noah Levitt eee6c6cd30 * ArchiveRecord.java 
fix indentation only (no substantive changes)
 2012-07-16 09:57:24 -07:00
Noah Levitt c108c4c451 HER-2014 Make ArchiveRecord.getPosition() public instead of protected. 
* ArchiveRecord.java, HeaderedArchiveRecord.java
    getPosition() - change visibility to public
 2012-07-16 09:54:54 -07:00