lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-23 06:56:41 +00:00
Code Issues Packages Projects Releases Wiki Activity
926 Commits 40 Branches 31 Tags
3f8590fe44d0d033cb71bfa8a4d21bde08ac8b8c
Commit Graph
926 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
gojomo 3f8590fe44 when found in CrawlURI, add WARC-Simple-Form-Province-Status header 2013-03-11 20:57:41 -07:00
gojomo bf9cab8eb2 when POST/submit-data requested, alter request method accordingly 2013-03-11 20:57:09 -07:00
gojomo bbac7d7320 per (overlay) settings, spawn high-priority POST when appropriate; annotate/add-headers to describe 2013-03-11 20:56:31 -07:00
gojomo ed639c8342 extract HTMLForm INPUT details 2013-03-11 20:55:20 -07:00
gojomo 94a61b4fb7 retain offsets of discovered FORMs to aid later processing 2013-03-11 20:53:41 -07:00
gojomo 6b0abae8ee new constants, hop-type; refactoring to help support form-submissions 2013-03-11 20:52:59 -07:00
gojomo eb0760a37e refactor for runCandidatesChain; also run outCandidates 2013-03-11 20:50:18 -07:00
gojomo ed65493db0 alternate testing of multiple local binds more likely to work more places 2012-10-01 16:38:17 -07:00
gojomo b4baea0a13 make maven requested version match eclipse project classpath 2012-10-01 16:37:31 -07:00
gojomo 37e6ab802a Merge branch 'master' of github.com:internetarchive/heritrix3 2012-10-01 13:41:12 -07:00
Noah Levitt 4470ba512e Count url-agnostic content digest duplicates as duplicates in statistics 
* CrawledBytesHistotable.java
    accumulate() - count urls with warcRevisit:uriAgnosticDigest annotation as duplicates
* ContentDigestHistoryTest.java
    testWarcDedupe() - check for presence of warcRevisit:uriAgnosticDigest annotation
 2012-09-27 16:56:14 -07:00
Noah Levitt a1d135edbc * WARCWriterProcessor.java 
include warcFileOffset in CrawlURI extraInfo (optionally logged in crawl log if logExtraInfo is enabled)
 2012-09-27 16:51:50 -07:00
Noah Levitt e2a1e3bb53 * ContentDigestHistoryTest.java 
testWarcDedupe() - open the written with with WARCReader, check that each record is as expected
 2012-09-27 14:34:20 -07:00
Noah Levitt a1a5fc7379 * WARCWriterProcessor.java 
writeRevisitUriAgnosticDigest() - put angle brackets around WARC-Refers-To uri value, since warc record ids appear everywhere with those brackets
 2012-09-27 14:20:00 -07:00
Noah Levitt 7a92e1ff7c * FetchHTTPTest.java 
shut down test servers in tearDown() (server startup/shutdown happens on every test, inefficient but it doesn't take all that long anyway)
* ContentDigestHistoryTest.java
    shut down test servers in finally block
 2012-09-26 19:29:11 -07:00
Noah Levitt e6fbe45774 Another test for ContentDigestHistory 
* ContentDigestHistoryTest.java
    new test testWarcDedupe() fetches two different urls with same content, writes warc records, checks results
* FetchHTTPTest.java, WARCWriterProcessorTest.java
    make test module code public and static for use in ContentDigestHistoryTest.java
 2012-09-26 18:54:48 -07:00
Noah Levitt f9703fb5ef * engine/pom.xml 
remove incorrect, unnecessary dependency
 2012-09-26 14:55:03 -07:00
Noah Levitt b9cbbbd248 Bunch of unit tests for FetchHTTP 
* FetchHTTPTest.java
    a bunch of new tests
* FetchHTTP.java
    make sslTrustLevel work correctly (bug found by test; even with fix, only respects value at startup, no overrides)
* modules/pom.xml, .classpath
    dependencies for testing
* engine/pom.xml
    update jetty to 6.1.26 since we require that for the new tests
 2012-09-26 14:41:49 -07:00
Noah Levitt 3612396ff3 Unit test for content digest history 
* ContentDigestHistoryTest.java
    test basics
* BdbContentDigestHistory.java
    code formatting tweak
 2012-09-26 12:03:45 -07:00
Noah Levitt 190e2ff0f3 Fix unit test 
* WARCWriterTest.java
    writeRandomHTTPRecord() - fix method, had transposed some lines of code
 2012-09-26 10:58:01 -07:00
Noah Levitt 2e1d81fa80 Fix build 
* WARCConstants.java, WARCWriter.java
    remove imports of WARCWriterProcessor from the modules/ area (was only used in javadoc)
 2012-09-26 08:58:22 -07:00
Noah Levitt 50c8a09b63 * BdbContentDigestHistory.java 
class javadoc
 2012-09-25 16:04:44 -07:00
Noah Levitt 819371f6eb * BdbContentDigestHistory.java 
store() - avoid clobbering good content digest history with empty one
 2012-09-25 15:28:19 -07:00
Noah Levitt 34252c4949 * WARCWriterProcessor.java 
writeHttpRecords() - avoid inadvertently creating content digest history map
 2012-09-25 15:26:56 -07:00
Noah Levitt fe1df5883f Some tweaks on maintaining content digest history 
* CrawlURI.java
    utility method hasContentDigestHistory()
* BdbContentDigestHistory.java
    load() - make sure to call curi.getContentDigestHistory() in all cases so the value is initialized and WARCWriterProcessor knows it should put the info in there
* WARCWriterProcessor.java
    updateMetadataAfterWrite() - update curi.getContentDigestHistory() only if curi.hasContentDigestHistory() for efficiency, like old uri-based fetch history; update the count after writing a revisit record
 2012-09-25 14:35:26 -07:00
Noah Levitt 25a9155298 New parent class AbstractContentDigestHistory for implementations to extend 
* AbstractContentDigestHistory.java
    abstract methods load(), store(); non-abstract persistKeyFor(); javadocs
* BdbContentDigestHistory.java
    extend AbstractContentDigestHistory
* ContentDigestHistoryLoader.java, ContentDigestHistoryStorer.java
    use AbstractContentDigestHistory
 2012-09-25 13:38:25 -07:00
Noah Levitt b8e14ba1df Rename ContentDigestHistory -> BdbContentDigestHistory, since it's bdb-specific 2012-09-25 13:12:59 -07:00
Noah Levitt ef78e584c4 * WARCWriterProcessor.java 
writeRevisitUriAgnosticDigest() - write several separate warc header fields for each value instead of one json blob
* WARCConstants.java
    new header field names
 2012-09-13 17:21:29 -07:00
Noah Levitt a8aae91167 * WARCConstants.java 
remove rfc6648-deprecated "X-" prefix to "WARC-Reference-Location", and document the field
* WARCWriterProcessor.java
    writeRevisitUriAgnosticDigest() - add some comments
 2012-09-11 18:21:20 -07:00
Noah Levitt 1048536a25 * WARCWriterProcessor.java 
writeRevisitUriAgnosticDigest() - fix construction of X-WARC-Reference-Location json value
 2012-09-11 17:21:34 -07:00
Noah Levitt 38aefe4d16 Write revisit records for URI-agnostic content digest duplicates. 
* WARCConstants.java
    new header keys "WARC-Refers-To" and "X-WARC-Reference-Location"
* CrawlURI.java
    getContentDigestHistory() - return HashMap instead of Map so clone() can be called on the return value
* WARCWriterProcessor.java
    writeHttpRecords(), writeRevisitUriAgnosticDigest() - write revisit record for URI-agnostic content digest duplicate if enabled
 2012-09-11 17:12:24 -07:00
Noah Levitt a04e76f00f Load, store content digest history 
* WARCRecordInfo.java
    new fields warcFilename, warcFileOffset
* WARCWriter.java
    temporarily remember warc records written
* CrawlURI.java
    convenience method getContentDigestHistory()
* ContentDigestHistory.java
    start up properly; load into and store from curi.getContentDigestHistory()
* RecrawlAttributeConstants.java
    bunch of hash keys for curi.getContentDigestHistory()
* WARCWriterProcessor.java
    save info in curi.getContentDigestHistory()
 2012-09-11 10:10:49 -07:00
Noah Levitt 2407fc5dd0 Beginning work on deduplication by content digest (irrespective of url) 
* ContentDigestHistory.java, ContentDigestHistoryLoader.java, ContentDigestHistoryStorer.java
    initial checkin
 2012-09-10 18:18:12 -07:00
Noah Levitt c4faa444a2 * ANVLRecord.java 
remove unneeded(?) serialVersionUID
 2012-09-10 18:14:15 -07:00
Noah Levitt a414f522f6 Continuing to work on cleaning up WARC writer code: simply creation of WARC record headers. 
* ANVLRecord.java
    extend LinkedList instead of ArrayList (it was inane to guess the number of records everywhere, when it was never much more than ~10)
* ANVLRecords.java
    remove (seemed to be an incomplete duplicate of ANVLRecord)
* WARCRecordInfo.java
    addExtraHeader() - convenience method
* Arc2Warc.java, WARCWriter.java, WARCWriter.java, WARCWriterProcessor.java
    use WARCRecordInfo.addExtraHeader() where appropriate, and use no-argument ANVLRecord constructor instead of deprecated ANVLRecord(int)
 2012-09-10 18:03:25 -07:00
Noah Levitt 360ac00000 Continuing to work on cleaning up WARC writer code. 
* WARCRecordInfo.java
    get rid of constructor with arguments, add setUrl() and setType()
* WARCWriter.java
    get rid of writeRecord() taking a million arguments
* Arc2Warc.java, WARCWriter.java, WARCWriterTest.java, WARCWriterProcessor.java, WARCWriterProcessorTest.java
    construct WARCRecordInfo object and call WARCWriter.writeRecord(recordInfo) instead of million argument method
 2012-09-10 17:41:43 -07:00
Noah Levitt dd5e51fd9d Working on cleaning up WARC writer code. 
* WARCRecordInfo.java
    new class to hold warc record info to write, so we don't have to pass around all these long lists of variables
* WARCConstants.java
    new enum WARCRecordType replaces bunch of string constants
* WARCWriter.java, WARCWriterProcessor.java, WARCWriterTest.java
    use WARCRecordInfo, WARCRecordType
 2012-09-10 15:43:37 -07:00
Noah Levitt 5b82d9abee Merge branch 'master' of github.com:internetarchive/heritrix3 2012-09-06 15:15:03 -07:00
Noah Levitt 5b35835d53 Unsupported auth scheme causing recoverableProblem with SEVERE level log line and stack trace 
* FetchHTTP.java
    getAuthScheme() - handle IllegalStateException thrown by AuthPolicy.getAuthScheme()
 2012-09-06 15:12:56 -07:00
Noah Levitt 1a07c6b741 Make unit tests pass 
* ContentExtractorTestBase.java
    assertNoSideEffects() - check curi.getAnnotations.isEmpty() instead of equality with Collections.EMPTY_LIST
 2012-09-05 11:19:18 -07:00
Noah Levitt e711ff9506 * JobRelatedResource.java 
fix code formatting from 01be6d4b
 2012-09-04 18:03:38 -07:00
Noah Levitt c433c78e90 Merge pull request #17 from adam-miller/HER-2018 
Fix for HER-2018: XML representation for /engine/job/<jobName>/beans returns incorrect url for named beans
 2012-09-04 17:59:43 -07:00
Noah Levitt b704f4eeb7 Fix HER-2021 usedBaseForVia annotation repeated many times 
* CrawlURI.java
    getAnnotations() - use LinkedHashSet data structure for new annotations object to prevent duplicates
* Link.java
    addRelativeToVia() - only log use of base for via the first time, before the annotation has been added
 2012-09-04 17:49:01 -07:00
Noah Levitt d0882a18e6 * AbstractPersistProcessor.java 
shouldStore() - only save persist record if http, https or ftp
 2012-09-01 22:09:30 -07:00
Adam Miller 01be6d4b18 Fix for HER-2018: XML representation for /engine/job/<jobName>/beans returns incorrect url for named beans 
* JobRelatedResource.java
    Added encoding to URL generation in makePresentableMapFor().
    Added check to prevent URL generation from overwriting the job name portion of the path in addPresentableNestedNames()
 2012-08-31 15:07:31 -07:00
Noah Levitt 61cbd54059 Merge pull request #16 from adam-miller/HER-2017 
Fix for HER-2017 XML representation of beans uses root node of type "script"
 2012-08-30 16:04:04 -07:00
Adam Miller 52e70a10ef Fix for HER-2017 XML representation for /engine/job/<jobName>/beans uses root node of type "script" instead of "beans" 
* BeanBrowseResource.java
	replaced "script" string in root node with "beans"
 2012-08-30 15:35:27 -07:00
Noah Levitt 0d65d7e446 Fix for HER-2019 fetchHTTP httpBindAddress setting doesn't take effect 
* FetchHTTP.java
    use keyed properties for httpBindAddress setting, as expected in configureBindAddress(CrawlURI, HostConfiguration)
 2012-08-28 16:03:17 -07:00
Noah Levitt a7a2b3cf7e Make speculative link extractor ignore urls with two consecutive slashes (not preceded by a colon as in http://). We've seen the double slashes lead to crawler traps. 
* UriUtils.java
    isLikelyFalsePositive() - consider likely false positive if it contains '//' (but not '://')
* UriUtilsTest.java
    testDoubleSlashFalsePositives()
 2012-08-24 14:03:45 -07:00
Noah Levitt f072dfc83c * Credential.java, HtmlFormCredential.java, HttpAuthenticationCredential.java 
bump serialVersionUID because of changed method signature on populate()
 2012-08-16 12:28:19 -07:00