lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-26 08:26:53 +00:00
Code Issues Packages Projects Releases Wiki Activity
939 Commits 38 Branches 32 Tags
b981fd7423aca2c444bacbdfec37fc88fe9be20b
Commit Graph
939 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Noah Levitt b981fd7423 * ArchiveReader.java 
add "implements Closeable" since it already has the close() method
 2012-12-03 17:09:22 -08:00
Noah Levitt 92b302f164 * ExtractorMultipleRegexTest.java 
avoid "constant string too long" compile error
 2012-11-16 19:08:18 -08:00
Noah Levitt 94df568b4f * ExtractorMultipleRegex.java 
remove temporary performance testing code
 2012-11-16 18:34:28 -08:00
Noah Levitt aa98ec3327 * ExtractorMultipleRegex.java 
keep cache of groovy Template objects, since they are expensive to create
 2012-11-16 18:30:41 -08:00
Noah Levitt eeee01a52d * ExtractorMultipleRegex.java 
javadocs
 2012-11-16 16:42:43 -08:00
Noah Levitt 89dfef468c * ExtractorMultipleRegexTest.java 
test passes now; extractor gets what it can, which is most of the scroll down urls
 2012-11-15 19:18:59 -08:00
Noah Levitt dfd7a8a7d3 * ExtractorMultipleRegexTest.java 
turns out that __adt parameter can be found near the json blob - most, but not all, expected links are now found
 2012-11-15 18:58:11 -08:00
Noah Levitt 22ea489c57 * ExtractorMultipleRegex.java 
remove the "fooIndex" thing from available bindings, since it's kinda hacky and turned out not to be needed for our use case
 2012-11-15 18:56:54 -08:00
Noah Levitt 4af58f38e5 * CrawlURI.java 
outLinks, outCandidates - use LinkedHashSet to ensure predictable order (any reason not to do that?)
 2012-11-15 18:55:37 -08:00
Noah Levitt 75280a13a8 * ExtractorMultipleRegexTest.java 
working on making test work - first two scroll-down urls are extracted successfully, others fail
 2012-11-15 17:27:41 -08:00
Noah Levitt 13e1f9d64d * ExtractorMultipleRegex.java 
fix omission from last commit, uriRegex
 2012-11-15 14:01:01 -08:00
Noah Levitt 6ec4b5a973 * ExtractorMultipleRegex.java 
more refactoring to avoid redundant operations
 2012-11-15 13:55:11 -08:00
Noah Levitt 43dbcbfe81 * ExtractorMultipleRegex.java 
refactor for readability
 2012-11-15 13:35:16 -08:00
Noah Levitt 3f44a4bc66 Merge branch 'master' into MoreImpliedExtractor 2012-11-15 12:35:17 -08:00
Noah Levitt 8395e372e4 * ScriptedProcessor.java 
getEngine() - fix logic to respect isolateThreads setting
 2012-11-15 12:33:48 -08:00
Noah Levitt ba5bca807e * ExtractorMultipleRegex.java 
use groovy templating facility
 2012-11-15 12:32:42 -08:00
Noah Levitt 6e2d2129fe * ExtractorMultipleRegexTest.java 
use real use case actual content, actual urls from facebook for testing
* ExtractorMultipleRegex.java
    some comments, little twiddles
 2012-11-15 11:16:57 -08:00
Noah Levitt 1e2d86dc63 * ContentExtractorTestBase.java 
createRecorder(String,String) - new method for creating test Recorder and specifying charset
    createRecorder(String) - deprecate (use default charset as before)
 2012-11-15 11:14:25 -08:00
Noah Levitt 1d994493e2 more progress on ExtractorMultipleRegex 2012-11-14 19:58:43 -08:00
Travis Wellman 1998337645 tmp commit 2012-11-14 17:58:44 -08:00
gojomo ed65493db0 alternate testing of multiple local binds more likely to work more places 2012-10-01 16:38:17 -07:00
gojomo b4baea0a13 make maven requested version match eclipse project classpath 2012-10-01 16:37:31 -07:00
gojomo 37e6ab802a Merge branch 'master' of github.com:internetarchive/heritrix3 2012-10-01 13:41:12 -07:00
Noah Levitt 4470ba512e Count url-agnostic content digest duplicates as duplicates in statistics 
* CrawledBytesHistotable.java
    accumulate() - count urls with warcRevisit:uriAgnosticDigest annotation as duplicates
* ContentDigestHistoryTest.java
    testWarcDedupe() - check for presence of warcRevisit:uriAgnosticDigest annotation
 2012-09-27 16:56:14 -07:00
Noah Levitt a1d135edbc * WARCWriterProcessor.java 
include warcFileOffset in CrawlURI extraInfo (optionally logged in crawl log if logExtraInfo is enabled)
 2012-09-27 16:51:50 -07:00
Noah Levitt e2a1e3bb53 * ContentDigestHistoryTest.java 
testWarcDedupe() - open the written with with WARCReader, check that each record is as expected
 2012-09-27 14:34:20 -07:00
Noah Levitt a1a5fc7379 * WARCWriterProcessor.java 
writeRevisitUriAgnosticDigest() - put angle brackets around WARC-Refers-To uri value, since warc record ids appear everywhere with those brackets
 2012-09-27 14:20:00 -07:00
Noah Levitt 7a92e1ff7c * FetchHTTPTest.java 
shut down test servers in tearDown() (server startup/shutdown happens on every test, inefficient but it doesn't take all that long anyway)
* ContentDigestHistoryTest.java
    shut down test servers in finally block
 2012-09-26 19:29:11 -07:00
Noah Levitt e6fbe45774 Another test for ContentDigestHistory 
* ContentDigestHistoryTest.java
    new test testWarcDedupe() fetches two different urls with same content, writes warc records, checks results
* FetchHTTPTest.java, WARCWriterProcessorTest.java
    make test module code public and static for use in ContentDigestHistoryTest.java
 2012-09-26 18:54:48 -07:00
Noah Levitt f9703fb5ef * engine/pom.xml 
remove incorrect, unnecessary dependency
 2012-09-26 14:55:03 -07:00
Noah Levitt b9cbbbd248 Bunch of unit tests for FetchHTTP 
* FetchHTTPTest.java
    a bunch of new tests
* FetchHTTP.java
    make sslTrustLevel work correctly (bug found by test; even with fix, only respects value at startup, no overrides)
* modules/pom.xml, .classpath
    dependencies for testing
* engine/pom.xml
    update jetty to 6.1.26 since we require that for the new tests
 2012-09-26 14:41:49 -07:00
Noah Levitt 3612396ff3 Unit test for content digest history 
* ContentDigestHistoryTest.java
    test basics
* BdbContentDigestHistory.java
    code formatting tweak
 2012-09-26 12:03:45 -07:00
Noah Levitt 190e2ff0f3 Fix unit test 
* WARCWriterTest.java
    writeRandomHTTPRecord() - fix method, had transposed some lines of code
 2012-09-26 10:58:01 -07:00
Noah Levitt 2e1d81fa80 Fix build 
* WARCConstants.java, WARCWriter.java
    remove imports of WARCWriterProcessor from the modules/ area (was only used in javadoc)
 2012-09-26 08:58:22 -07:00
Noah Levitt 50c8a09b63 * BdbContentDigestHistory.java 
class javadoc
 2012-09-25 16:04:44 -07:00
Noah Levitt 819371f6eb * BdbContentDigestHistory.java 
store() - avoid clobbering good content digest history with empty one
 2012-09-25 15:28:19 -07:00
Noah Levitt 34252c4949 * WARCWriterProcessor.java 
writeHttpRecords() - avoid inadvertently creating content digest history map
 2012-09-25 15:26:56 -07:00
Noah Levitt fe1df5883f Some tweaks on maintaining content digest history 
* CrawlURI.java
    utility method hasContentDigestHistory()
* BdbContentDigestHistory.java
    load() - make sure to call curi.getContentDigestHistory() in all cases so the value is initialized and WARCWriterProcessor knows it should put the info in there
* WARCWriterProcessor.java
    updateMetadataAfterWrite() - update curi.getContentDigestHistory() only if curi.hasContentDigestHistory() for efficiency, like old uri-based fetch history; update the count after writing a revisit record
 2012-09-25 14:35:26 -07:00
Noah Levitt 25a9155298 New parent class AbstractContentDigestHistory for implementations to extend 
* AbstractContentDigestHistory.java
    abstract methods load(), store(); non-abstract persistKeyFor(); javadocs
* BdbContentDigestHistory.java
    extend AbstractContentDigestHistory
* ContentDigestHistoryLoader.java, ContentDigestHistoryStorer.java
    use AbstractContentDigestHistory
 2012-09-25 13:38:25 -07:00
Noah Levitt b8e14ba1df Rename ContentDigestHistory -> BdbContentDigestHistory, since it's bdb-specific 2012-09-25 13:12:59 -07:00
Noah Levitt ef78e584c4 * WARCWriterProcessor.java 
writeRevisitUriAgnosticDigest() - write several separate warc header fields for each value instead of one json blob
* WARCConstants.java
    new header field names
 2012-09-13 17:21:29 -07:00
Noah Levitt a8aae91167 * WARCConstants.java 
remove rfc6648-deprecated "X-" prefix to "WARC-Reference-Location", and document the field
* WARCWriterProcessor.java
    writeRevisitUriAgnosticDigest() - add some comments
 2012-09-11 18:21:20 -07:00
Noah Levitt 1048536a25 * WARCWriterProcessor.java 
writeRevisitUriAgnosticDigest() - fix construction of X-WARC-Reference-Location json value
 2012-09-11 17:21:34 -07:00
Noah Levitt 38aefe4d16 Write revisit records for URI-agnostic content digest duplicates. 
* WARCConstants.java
    new header keys "WARC-Refers-To" and "X-WARC-Reference-Location"
* CrawlURI.java
    getContentDigestHistory() - return HashMap instead of Map so clone() can be called on the return value
* WARCWriterProcessor.java
    writeHttpRecords(), writeRevisitUriAgnosticDigest() - write revisit record for URI-agnostic content digest duplicate if enabled
 2012-09-11 17:12:24 -07:00
Noah Levitt a04e76f00f Load, store content digest history 
* WARCRecordInfo.java
    new fields warcFilename, warcFileOffset
* WARCWriter.java
    temporarily remember warc records written
* CrawlURI.java
    convenience method getContentDigestHistory()
* ContentDigestHistory.java
    start up properly; load into and store from curi.getContentDigestHistory()
* RecrawlAttributeConstants.java
    bunch of hash keys for curi.getContentDigestHistory()
* WARCWriterProcessor.java
    save info in curi.getContentDigestHistory()
 2012-09-11 10:10:49 -07:00
Noah Levitt 2407fc5dd0 Beginning work on deduplication by content digest (irrespective of url) 
* ContentDigestHistory.java, ContentDigestHistoryLoader.java, ContentDigestHistoryStorer.java
    initial checkin
 2012-09-10 18:18:12 -07:00
Noah Levitt c4faa444a2 * ANVLRecord.java 
remove unneeded(?) serialVersionUID
 2012-09-10 18:14:15 -07:00
Noah Levitt a414f522f6 Continuing to work on cleaning up WARC writer code: simply creation of WARC record headers. 
* ANVLRecord.java
    extend LinkedList instead of ArrayList (it was inane to guess the number of records everywhere, when it was never much more than ~10)
* ANVLRecords.java
    remove (seemed to be an incomplete duplicate of ANVLRecord)
* WARCRecordInfo.java
    addExtraHeader() - convenience method
* Arc2Warc.java, WARCWriter.java, WARCWriter.java, WARCWriterProcessor.java
    use WARCRecordInfo.addExtraHeader() where appropriate, and use no-argument ANVLRecord constructor instead of deprecated ANVLRecord(int)
 2012-09-10 18:03:25 -07:00
Noah Levitt 360ac00000 Continuing to work on cleaning up WARC writer code. 
* WARCRecordInfo.java
    get rid of constructor with arguments, add setUrl() and setType()
* WARCWriter.java
    get rid of writeRecord() taking a million arguments
* Arc2Warc.java, WARCWriter.java, WARCWriterTest.java, WARCWriterProcessor.java, WARCWriterProcessorTest.java
    construct WARCRecordInfo object and call WARCWriter.writeRecord(recordInfo) instead of million argument method
 2012-09-10 17:41:43 -07:00
Noah Levitt dd5e51fd9d Working on cleaning up WARC writer code. 
* WARCRecordInfo.java
    new class to hold warc record info to write, so we don't have to pass around all these long lists of variables
* WARCConstants.java
    new enum WARCRecordType replaces bunch of string constants
* WARCWriter.java, WARCWriterProcessor.java, WARCWriterTest.java
    use WARCRecordInfo, WARCRecordType
 2012-09-10 15:43:37 -07:00