lee/heritrix3
1
0 Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-10-07 04:41:23 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,068 Commits 41 Branches 34 Tags
d3db71e3bb05367c0d63f7c4e7c83e110f5035c1
Commit Graph
1068 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Noah Levitt d3db71e3bb Working on adapting to httpcomponents HEAD with custom patches (hoping to get those incorporated) 2012-12-21 16:42:30 -08:00
Noah Levitt e26ee96895 * FetchHTTPTests.java 
testLaxUrlEncoding() - Tests a URL not correctly url-encoded, but that heritrix should let pass through to mimic browser behavior (test fails currently)
 2012-12-20 11:14:16 -08:00
Noah Levitt c7813938e0 Merge branch 'master' into new-fetchhttp-only 
Conflicts:
	.classpath
	commons/src/main/java/org/archive/state/ModuleTestBase.java
	modules/src/test/java/org/archive/modules/fetcher/FetchHTTPTest.java
 2012-12-20 11:04:42 -08:00
Noah Levitt 0eac126f7d Special WARC-Profile for uri-agnostic revisit records. See HER-2022, http://tech.groups.yahoo.com/group/archive-crawler/message/7806 2012-12-13 14:11:14 -08:00
Noah Levitt 5586d5452d * JerichoExtractorHTMLTest.java 
avoid redundancy by using extractor built in setUp()
    makeExtractor() - call extractor.setExtractorJS(new ExtractorJS()) since we got rid of the static ExtractorJS stuff
    testConditionalComment1() - override to skip the test since it fails with JerichoExtractorHTML
 2012-12-11 17:00:23 -08:00
Noah Levitt 85118c59c4 Tighten up javascript extraction based on real-world analysis (addresses HER-1523) 
* UriUtils.java
    new method isVeryLikelyUri() with tighter heuristic than isLikelyUri()
* ExtractorJS.java
    use UriUtils.isVeryLikelyUri(), and change order of operations to do fixup before call to isVeryLikelyUri(), since it doesn't expect strings with javascript escaping and stuff
* StringExtractorTestBase.java
    handle test data with expected value null, meaning no outlinks expected
* ExtractorHTMLTest.java
    avoid redundancy by using the extractor created in ContentExtractorTestBase.setUp()
* ExtractorJSTest.java
    some new tests
 2012-12-11 16:27:26 -08:00
Noah Levitt b1cd3d30a0 * ExtractorHTMLTest.java 
makeExtractor() - call setExtractorJS(new ExtractorJS()) so testSpeculativeLinkExtraction() passes
 2012-12-10 17:31:55 -08:00
Noah Levitt 256b16249a * ExtractorJS.java 
refactor so considerStrings() is not static, allowing it to be overridden in subclasses
* ExtractorHTML.java, ExtractorJS.java
    add @Autowired parameter extractorJS used to process inline javascript, instead of call to static ExtractorJS.considerStrings()
 2012-12-10 16:51:42 -08:00
Noah Levitt b2a90ddc15 * UriUtils.java 
speculativeFixup() - improve detection of scheme-less intended-absolute-URIs
 2012-12-10 14:10:34 -08:00
Noah Levitt d3dc153c5f * ModuleTestBase.java 
runTest() - convenience logging of test failures
 2012-12-10 13:56:45 -08:00
Noah Levitt e44cf9ca9b A little cleanup of FetchHTTPTest refactoring 2012-12-04 18:43:15 -08:00
Noah Levitt 5354960f00 Refactor FetchHTTPTest to use a TestSuite so that starting and shutting down the test http servers happen only once at start and finish respectively 2012-12-04 17:25:09 -08:00
Noah Levitt 2c0dbed399 .classpath 
include junit as a regular dependency not managed by eclipse, so source jar can be attached
 2012-12-04 17:20:39 -08:00
Noah Levitt e30cb83540 * FetchHTTPTest.java 
testLaxUrlEncoding() - Tests a URL not correctly url-encoded, but that heritrix lets pass through to mimic browser behavior.
 2012-12-04 16:25:09 -08:00
Noah Levitt b981fd7423 * ArchiveReader.java 
add "implements Closeable" since it already has the close() method
 2012-12-03 17:09:22 -08:00
Noah Levitt 4496c5d755 * FetchHTTPTest.java 
move to modules/src/test
    testFormAuth() - remove, there is FormAuthSelfTest.java, and this was the test that would fail in modules/
    testHttpBindAddress() - incorporate ed65493db0 - alternate testing of multiple local binds more likely to work more places
 2012-11-30 17:57:36 -08:00
Noah Levitt dc7fea3252 * .classpath 
add missing libs
 2012-11-30 17:56:53 -08:00
Noah Levitt a20824aea7 Merge branch 'master' into new-fetchhttp-only 
Conflicts:
	.classpath
	engine/pom.xml
	modules/pom.xml
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java
	modules/src/test/java/org/archive/modules/fetcher/FetchHTTPTest.java
 2012-11-30 17:04:28 -08:00
Noah Levitt 92b302f164 * ExtractorMultipleRegexTest.java 
avoid "constant string too long" compile error
 2012-11-16 19:08:18 -08:00
Noah Levitt 94df568b4f * ExtractorMultipleRegex.java 
remove temporary performance testing code
 2012-11-16 18:34:28 -08:00
Noah Levitt aa98ec3327 * ExtractorMultipleRegex.java 
keep cache of groovy Template objects, since they are expensive to create
 2012-11-16 18:30:41 -08:00
Noah Levitt eeee01a52d * ExtractorMultipleRegex.java 
javadocs
 2012-11-16 16:42:43 -08:00
Noah Levitt 89dfef468c * ExtractorMultipleRegexTest.java 
test passes now; extractor gets what it can, which is most of the scroll down urls
 2012-11-15 19:18:59 -08:00
Noah Levitt dfd7a8a7d3 * ExtractorMultipleRegexTest.java 
turns out that __adt parameter can be found near the json blob - most, but not all, expected links are now found
 2012-11-15 18:58:11 -08:00
Noah Levitt 22ea489c57 * ExtractorMultipleRegex.java 
remove the "fooIndex" thing from available bindings, since it's kinda hacky and turned out not to be needed for our use case
 2012-11-15 18:56:54 -08:00
Noah Levitt 4af58f38e5 * CrawlURI.java 
outLinks, outCandidates - use LinkedHashSet to ensure predictable order (any reason not to do that?)
 2012-11-15 18:55:37 -08:00
Noah Levitt 75280a13a8 * ExtractorMultipleRegexTest.java 
working on making test work - first two scroll-down urls are extracted successfully, others fail
 2012-11-15 17:27:41 -08:00
Noah Levitt 13e1f9d64d * ExtractorMultipleRegex.java 
fix omission from last commit, uriRegex
 2012-11-15 14:01:01 -08:00
Noah Levitt 6ec4b5a973 * ExtractorMultipleRegex.java 
more refactoring to avoid redundant operations
 2012-11-15 13:55:11 -08:00
Noah Levitt 43dbcbfe81 * ExtractorMultipleRegex.java 
refactor for readability
 2012-11-15 13:35:16 -08:00
Noah Levitt 3f44a4bc66 Merge branch 'master' into MoreImpliedExtractor 2012-11-15 12:35:17 -08:00
Noah Levitt 8395e372e4 * ScriptedProcessor.java 
getEngine() - fix logic to respect isolateThreads setting
 2012-11-15 12:33:48 -08:00
Noah Levitt ba5bca807e * ExtractorMultipleRegex.java 
use groovy templating facility
 2012-11-15 12:32:42 -08:00
Noah Levitt 6e2d2129fe * ExtractorMultipleRegexTest.java 
use real use case actual content, actual urls from facebook for testing
* ExtractorMultipleRegex.java
    some comments, little twiddles
 2012-11-15 11:16:57 -08:00
Noah Levitt 1e2d86dc63 * ContentExtractorTestBase.java 
createRecorder(String,String) - new method for creating test Recorder and specifying charset
    createRecorder(String) - deprecate (use default charset as before)
 2012-11-15 11:14:25 -08:00
Noah Levitt 1d994493e2 more progress on ExtractorMultipleRegex 2012-11-14 19:58:43 -08:00
Travis Wellman 1998337645 tmp commit 2012-11-14 17:58:44 -08:00
gojomo ed65493db0 alternate testing of multiple local binds more likely to work more places 2012-10-01 16:38:17 -07:00
gojomo b4baea0a13 make maven requested version match eclipse project classpath 2012-10-01 16:37:31 -07:00
gojomo 37e6ab802a Merge branch 'master' of github.com:internetarchive/heritrix3 2012-10-01 13:41:12 -07:00
Noah Levitt 4470ba512e Count url-agnostic content digest duplicates as duplicates in statistics 
* CrawledBytesHistotable.java
    accumulate() - count urls with warcRevisit:uriAgnosticDigest annotation as duplicates
* ContentDigestHistoryTest.java
    testWarcDedupe() - check for presence of warcRevisit:uriAgnosticDigest annotation
 2012-09-27 16:56:14 -07:00
Noah Levitt a1d135edbc * WARCWriterProcessor.java 
include warcFileOffset in CrawlURI extraInfo (optionally logged in crawl log if logExtraInfo is enabled)
 2012-09-27 16:51:50 -07:00
Noah Levitt e2a1e3bb53 * ContentDigestHistoryTest.java 
testWarcDedupe() - open the written with with WARCReader, check that each record is as expected
 2012-09-27 14:34:20 -07:00
Noah Levitt a1a5fc7379 * WARCWriterProcessor.java 
writeRevisitUriAgnosticDigest() - put angle brackets around WARC-Refers-To uri value, since warc record ids appear everywhere with those brackets
 2012-09-27 14:20:00 -07:00
Noah Levitt 7a92e1ff7c * FetchHTTPTest.java 
shut down test servers in tearDown() (server startup/shutdown happens on every test, inefficient but it doesn't take all that long anyway)
* ContentDigestHistoryTest.java
    shut down test servers in finally block
 2012-09-26 19:29:11 -07:00
Noah Levitt e6fbe45774 Another test for ContentDigestHistory 
* ContentDigestHistoryTest.java
    new test testWarcDedupe() fetches two different urls with same content, writes warc records, checks results
* FetchHTTPTest.java, WARCWriterProcessorTest.java
    make test module code public and static for use in ContentDigestHistoryTest.java
 2012-09-26 18:54:48 -07:00
Noah Levitt f9703fb5ef * engine/pom.xml 
remove incorrect, unnecessary dependency
 2012-09-26 14:55:03 -07:00
Noah Levitt b9cbbbd248 Bunch of unit tests for FetchHTTP 
* FetchHTTPTest.java
    a bunch of new tests
* FetchHTTP.java
    make sslTrustLevel work correctly (bug found by test; even with fix, only respects value at startup, no overrides)
* modules/pom.xml, .classpath
    dependencies for testing
* engine/pom.xml
    update jetty to 6.1.26 since we require that for the new tests
 2012-09-26 14:41:49 -07:00
Noah Levitt f0b4cfeeac Merge branch 'master' into new-fetchhttp-only 
Conflicts:
	modules/src/main/java/org/archive/modules/CrawlURI.java
	modules/src/main/java/org/archive/modules/writer/WARCWriterProcessor.java
 2012-09-26 12:59:41 -07:00
Noah Levitt 3612396ff3 Unit test for content digest history 
* ContentDigestHistoryTest.java
    test basics
* BdbContentDigestHistory.java
    code formatting tweak
 2012-09-26 12:03:45 -07:00