lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-13 10:11:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
980 Commits 40 Branches 31 Tags
60d00e3ef3b7c2f3eb6e1262d13fbcfacdf1ef90
Commit Graph
349 Commits
Author SHA1 Message Date
Noah Levitt 878e3b1afe Remove spurious "throws" declaration 2013-02-05 18:37:22 -08:00
Noah Levitt 543e8820ef Merge branch 'master' into archive-commons-refactor 2012-12-20 02:44:57 -08:00
Noah Levitt 3a65fa25d7 Working on refactoring some stuff into archive-commons 2012-12-18 14:46:52 -08:00
Noah Levitt 0eac126f7d Special WARC-Profile for uri-agnostic revisit records. See HER-2022, http://tech.groups.yahoo.com/group/archive-crawler/message/7806 2012-12-13 14:11:14 -08:00
Noah Levitt 5586d5452d * JerichoExtractorHTMLTest.java 
avoid redundancy by using extractor built in setUp()
    makeExtractor() - call extractor.setExtractorJS(new ExtractorJS()) since we got rid of the static ExtractorJS stuff
    testConditionalComment1() - override to skip the test since it fails with JerichoExtractorHTML
 2012-12-11 17:00:23 -08:00
Noah Levitt 85118c59c4 Tighten up javascript extraction based on real-world analysis (addresses HER-1523) 
* UriUtils.java
    new method isVeryLikelyUri() with tighter heuristic than isLikelyUri()
* ExtractorJS.java
    use UriUtils.isVeryLikelyUri(), and change order of operations to do fixup before call to isVeryLikelyUri(), since it doesn't expect strings with javascript escaping and stuff
* StringExtractorTestBase.java
    handle test data with expected value null, meaning no outlinks expected
* ExtractorHTMLTest.java
    avoid redundancy by using the extractor created in ContentExtractorTestBase.setUp()
* ExtractorJSTest.java
    some new tests
 2012-12-11 16:27:26 -08:00
Noah Levitt b1cd3d30a0 * ExtractorHTMLTest.java 
makeExtractor() - call setExtractorJS(new ExtractorJS()) so testSpeculativeLinkExtraction() passes
 2012-12-10 17:31:55 -08:00
Noah Levitt 256b16249a * ExtractorJS.java 
refactor so considerStrings() is not static, allowing it to be overridden in subclasses
* ExtractorHTML.java, ExtractorJS.java
    add @Autowired parameter extractorJS used to process inline javascript, instead of call to static ExtractorJS.considerStrings()
 2012-12-10 16:51:42 -08:00
Noah Levitt e44cf9ca9b A little cleanup of FetchHTTPTest refactoring 2012-12-04 18:43:15 -08:00
Noah Levitt 5354960f00 Refactor FetchHTTPTest to use a TestSuite so that starting and shutting down the test http servers happen only once at start and finish respectively 2012-12-04 17:25:09 -08:00
Noah Levitt e30cb83540 * FetchHTTPTest.java 
testLaxUrlEncoding() - Tests a URL not correctly url-encoded, but that heritrix lets pass through to mimic browser behavior.
 2012-12-04 16:25:09 -08:00
Noah Levitt 92b302f164 * ExtractorMultipleRegexTest.java 
avoid "constant string too long" compile error
 2012-11-16 19:08:18 -08:00
Noah Levitt 94df568b4f * ExtractorMultipleRegex.java 
remove temporary performance testing code
 2012-11-16 18:34:28 -08:00
Noah Levitt aa98ec3327 * ExtractorMultipleRegex.java 
keep cache of groovy Template objects, since they are expensive to create
 2012-11-16 18:30:41 -08:00
Noah Levitt eeee01a52d * ExtractorMultipleRegex.java 
javadocs
 2012-11-16 16:42:43 -08:00
Noah Levitt 89dfef468c * ExtractorMultipleRegexTest.java 
test passes now; extractor gets what it can, which is most of the scroll down urls
 2012-11-15 19:18:59 -08:00
Noah Levitt dfd7a8a7d3 * ExtractorMultipleRegexTest.java 
turns out that __adt parameter can be found near the json blob - most, but not all, expected links are now found
 2012-11-15 18:58:11 -08:00
Noah Levitt 22ea489c57 * ExtractorMultipleRegex.java 
remove the "fooIndex" thing from available bindings, since it's kinda hacky and turned out not to be needed for our use case
 2012-11-15 18:56:54 -08:00
Noah Levitt 4af58f38e5 * CrawlURI.java 
outLinks, outCandidates - use LinkedHashSet to ensure predictable order (any reason not to do that?)
 2012-11-15 18:55:37 -08:00
Noah Levitt 75280a13a8 * ExtractorMultipleRegexTest.java 
working on making test work - first two scroll-down urls are extracted successfully, others fail
 2012-11-15 17:27:41 -08:00
Noah Levitt 13e1f9d64d * ExtractorMultipleRegex.java 
fix omission from last commit, uriRegex
 2012-11-15 14:01:01 -08:00
Noah Levitt 6ec4b5a973 * ExtractorMultipleRegex.java 
more refactoring to avoid redundant operations
 2012-11-15 13:55:11 -08:00
Noah Levitt 43dbcbfe81 * ExtractorMultipleRegex.java 
refactor for readability
 2012-11-15 13:35:16 -08:00
Noah Levitt 3f44a4bc66 Merge branch 'master' into MoreImpliedExtractor 2012-11-15 12:35:17 -08:00
Noah Levitt 8395e372e4 * ScriptedProcessor.java 
getEngine() - fix logic to respect isolateThreads setting
 2012-11-15 12:33:48 -08:00
Noah Levitt ba5bca807e * ExtractorMultipleRegex.java 
use groovy templating facility
 2012-11-15 12:32:42 -08:00
Noah Levitt 6e2d2129fe * ExtractorMultipleRegexTest.java 
use real use case actual content, actual urls from facebook for testing
* ExtractorMultipleRegex.java
    some comments, little twiddles
 2012-11-15 11:16:57 -08:00
Noah Levitt 1e2d86dc63 * ContentExtractorTestBase.java 
createRecorder(String,String) - new method for creating test Recorder and specifying charset
    createRecorder(String) - deprecate (use default charset as before)
 2012-11-15 11:14:25 -08:00
Noah Levitt 1d994493e2 more progress on ExtractorMultipleRegex 2012-11-14 19:58:43 -08:00
Travis Wellman 1998337645 tmp commit 2012-11-14 17:58:44 -08:00
gojomo ed65493db0 alternate testing of multiple local binds more likely to work more places 2012-10-01 16:38:17 -07:00
Noah Levitt 4470ba512e Count url-agnostic content digest duplicates as duplicates in statistics 
* CrawledBytesHistotable.java
    accumulate() - count urls with warcRevisit:uriAgnosticDigest annotation as duplicates
* ContentDigestHistoryTest.java
    testWarcDedupe() - check for presence of warcRevisit:uriAgnosticDigest annotation
 2012-09-27 16:56:14 -07:00
Noah Levitt a1d135edbc * WARCWriterProcessor.java 
include warcFileOffset in CrawlURI extraInfo (optionally logged in crawl log if logExtraInfo is enabled)
 2012-09-27 16:51:50 -07:00
Noah Levitt e2a1e3bb53 * ContentDigestHistoryTest.java 
testWarcDedupe() - open the written with with WARCReader, check that each record is as expected
 2012-09-27 14:34:20 -07:00
Noah Levitt a1a5fc7379 * WARCWriterProcessor.java 
writeRevisitUriAgnosticDigest() - put angle brackets around WARC-Refers-To uri value, since warc record ids appear everywhere with those brackets
 2012-09-27 14:20:00 -07:00
Noah Levitt 7a92e1ff7c * FetchHTTPTest.java 
shut down test servers in tearDown() (server startup/shutdown happens on every test, inefficient but it doesn't take all that long anyway)
* ContentDigestHistoryTest.java
    shut down test servers in finally block
 2012-09-26 19:29:11 -07:00
Noah Levitt e6fbe45774 Another test for ContentDigestHistory 
* ContentDigestHistoryTest.java
    new test testWarcDedupe() fetches two different urls with same content, writes warc records, checks results
* FetchHTTPTest.java, WARCWriterProcessorTest.java
    make test module code public and static for use in ContentDigestHistoryTest.java
 2012-09-26 18:54:48 -07:00
Noah Levitt b9cbbbd248 Bunch of unit tests for FetchHTTP 
* FetchHTTPTest.java
    a bunch of new tests
* FetchHTTP.java
    make sslTrustLevel work correctly (bug found by test; even with fix, only respects value at startup, no overrides)
* modules/pom.xml, .classpath
    dependencies for testing
* engine/pom.xml
    update jetty to 6.1.26 since we require that for the new tests
 2012-09-26 14:41:49 -07:00
Noah Levitt 3612396ff3 Unit test for content digest history 
* ContentDigestHistoryTest.java
    test basics
* BdbContentDigestHistory.java
    code formatting tweak
 2012-09-26 12:03:45 -07:00
Noah Levitt 50c8a09b63 * BdbContentDigestHistory.java 
class javadoc
 2012-09-25 16:04:44 -07:00
Noah Levitt 819371f6eb * BdbContentDigestHistory.java 
store() - avoid clobbering good content digest history with empty one
 2012-09-25 15:28:19 -07:00
Noah Levitt 34252c4949 * WARCWriterProcessor.java 
writeHttpRecords() - avoid inadvertently creating content digest history map
 2012-09-25 15:26:56 -07:00
Noah Levitt fe1df5883f Some tweaks on maintaining content digest history 
* CrawlURI.java
    utility method hasContentDigestHistory()
* BdbContentDigestHistory.java
    load() - make sure to call curi.getContentDigestHistory() in all cases so the value is initialized and WARCWriterProcessor knows it should put the info in there
* WARCWriterProcessor.java
    updateMetadataAfterWrite() - update curi.getContentDigestHistory() only if curi.hasContentDigestHistory() for efficiency, like old uri-based fetch history; update the count after writing a revisit record
 2012-09-25 14:35:26 -07:00
Noah Levitt 25a9155298 New parent class AbstractContentDigestHistory for implementations to extend 
* AbstractContentDigestHistory.java
    abstract methods load(), store(); non-abstract persistKeyFor(); javadocs
* BdbContentDigestHistory.java
    extend AbstractContentDigestHistory
* ContentDigestHistoryLoader.java, ContentDigestHistoryStorer.java
    use AbstractContentDigestHistory
 2012-09-25 13:38:25 -07:00
Noah Levitt b8e14ba1df Rename ContentDigestHistory -> BdbContentDigestHistory, since it's bdb-specific 2012-09-25 13:12:59 -07:00
Noah Levitt ef78e584c4 * WARCWriterProcessor.java 
writeRevisitUriAgnosticDigest() - write several separate warc header fields for each value instead of one json blob
* WARCConstants.java
    new header field names
 2012-09-13 17:21:29 -07:00
Noah Levitt a8aae91167 * WARCConstants.java 
remove rfc6648-deprecated "X-" prefix to "WARC-Reference-Location", and document the field
* WARCWriterProcessor.java
    writeRevisitUriAgnosticDigest() - add some comments
 2012-09-11 18:21:20 -07:00
Noah Levitt 1048536a25 * WARCWriterProcessor.java 
writeRevisitUriAgnosticDigest() - fix construction of X-WARC-Reference-Location json value
 2012-09-11 17:21:34 -07:00
Noah Levitt 38aefe4d16 Write revisit records for URI-agnostic content digest duplicates. 
* WARCConstants.java
    new header keys "WARC-Refers-To" and "X-WARC-Reference-Location"
* CrawlURI.java
    getContentDigestHistory() - return HashMap instead of Map so clone() can be called on the return value
* WARCWriterProcessor.java
    writeHttpRecords(), writeRevisitUriAgnosticDigest() - write revisit record for URI-agnostic content digest duplicate if enabled
 2012-09-11 17:12:24 -07:00
Noah Levitt a04e76f00f Load, store content digest history 
* WARCRecordInfo.java
    new fields warcFilename, warcFileOffset
* WARCWriter.java
    temporarily remember warc records written
* CrawlURI.java
    convenience method getContentDigestHistory()
* ContentDigestHistory.java
    start up properly; load into and store from curi.getContentDigestHistory()
* RecrawlAttributeConstants.java
    bunch of hash keys for curi.getContentDigestHistory()
* WARCWriterProcessor.java
    save info in curi.getContentDigestHistory()
 2012-09-11 10:10:49 -07:00