lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-07-29 02:41:12 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,091 Commits 40 Branches 31 Tags
b6133aee5fca91dc6da50461fe7d2dbc664fc997
Commit Graph
466 Commits
Author SHA1 Message Date
Noah Levitt b6133aee5f Update httpcomponents.diff 2012-12-31 14:15:05 -08:00
Noah Levitt 163871d462 Tweaks and twiddles 2012-12-31 01:57:03 -08:00
Noah Levitt 25234898ef Somewhat redundant form auth integration test restored, passes 2012-12-30 22:35:28 -08:00
Noah Levitt fcece02646 Close recorders on failure to avoid exception from retry cycle within httpclient- now all the FetchHTTPTests pass 2012-12-30 20:45:05 -08:00
Noah Levitt cb725bb38e Fix socket timeout 2012-12-30 19:05:55 -08:00
Noah Levitt dac492f2c6 DnsResolver using ServerCache (should add a test for this) 2012-12-30 01:00:07 -08:00
Noah Levitt d1c41d624f SSL tests pass 2012-12-30 00:18:44 -08:00
Noah Levitt 7cb05c6961 Make sure wire logger is disabled for testMaxFetchKBSec(), otherwise it can slow things down enough that it fails 2012-12-29 21:37:46 -08:00
Noah Levitt 2a6a2aa6d4 testHttpProxyAuth() passes 2012-12-29 21:05:37 -08:00
Noah Levitt 6e8d4efe35 Refactor code building the request into new class FetchHTTPRequest; testHttpProxy() passes 2012-12-29 20:33:47 -08:00
Noah Levitt 01a9c89088 testHttpBindAddress() passes 2012-12-29 15:09:46 -08:00
Noah Levitt 959504b7f0 testAcceptCompression() passes 2012-12-29 14:59:59 -08:00
Noah Levitt c01e50a5a5 Digest auth test passes 2012-12-29 14:29:47 -08:00
Noah Levitt 70875f1605 Basic auth test passes 2012-12-29 14:00:36 -08:00
Noah Levitt 34e0ba96ce Get rid of "UmmSession*Buffer* classes and use Session*BufferImpl directly 2012-12-28 21:23:18 -08:00
Noah Levitt 6fe901d1f8 New approach to overriding SessionInputBuffer seems promising. 
(httpcomponents.diff is the patch to httpcomponents head that is needed for this h3 commit to work, will keep up to date with each commit)
 2012-12-28 21:06:15 -08:00
Noah Levitt 6300caf367 A little more work on the approach of wrapping the SessionInputBuffer, to remember that overriding BHttpConnectionBase.getSessionInputBuffer() makes it possible (but it doesn't work because we need to get in there and no-op the buffering since we do our own, and it eats up the stream too soon...) 2012-12-28 18:46:39 -08:00
Noah Levitt 33a500b213 Some work on an approach to overriding SessionInputBuffer that looks like is not the way 2012-12-28 15:37:58 -08:00
Noah Levitt c9223a9a02 More work on adapting to httpcomponents HEAD with custom patches 2012-12-22 17:10:38 -08:00
Noah Levitt d3db71e3bb Working on adapting to httpcomponents HEAD with custom patches (hoping to get those incorporated) 2012-12-21 16:42:30 -08:00
Noah Levitt e26ee96895 * FetchHTTPTests.java 
testLaxUrlEncoding() - Tests a URL not correctly url-encoded, but that heritrix should let pass through to mimic browser behavior (test fails currently)
 2012-12-20 11:14:16 -08:00
Noah Levitt c7813938e0 Merge branch 'master' into new-fetchhttp-only 
Conflicts:
	.classpath
	commons/src/main/java/org/archive/state/ModuleTestBase.java
	modules/src/test/java/org/archive/modules/fetcher/FetchHTTPTest.java
 2012-12-20 11:04:42 -08:00
Noah Levitt 0eac126f7d Special WARC-Profile for uri-agnostic revisit records. See HER-2022, http://tech.groups.yahoo.com/group/archive-crawler/message/7806 2012-12-13 14:11:14 -08:00
Noah Levitt 5586d5452d * JerichoExtractorHTMLTest.java 
avoid redundancy by using extractor built in setUp()
    makeExtractor() - call extractor.setExtractorJS(new ExtractorJS()) since we got rid of the static ExtractorJS stuff
    testConditionalComment1() - override to skip the test since it fails with JerichoExtractorHTML
 2012-12-11 17:00:23 -08:00
Noah Levitt 85118c59c4 Tighten up javascript extraction based on real-world analysis (addresses HER-1523) 
* UriUtils.java
    new method isVeryLikelyUri() with tighter heuristic than isLikelyUri()
* ExtractorJS.java
    use UriUtils.isVeryLikelyUri(), and change order of operations to do fixup before call to isVeryLikelyUri(), since it doesn't expect strings with javascript escaping and stuff
* StringExtractorTestBase.java
    handle test data with expected value null, meaning no outlinks expected
* ExtractorHTMLTest.java
    avoid redundancy by using the extractor created in ContentExtractorTestBase.setUp()
* ExtractorJSTest.java
    some new tests
 2012-12-11 16:27:26 -08:00
Noah Levitt b1cd3d30a0 * ExtractorHTMLTest.java 
makeExtractor() - call setExtractorJS(new ExtractorJS()) so testSpeculativeLinkExtraction() passes
 2012-12-10 17:31:55 -08:00
Noah Levitt 256b16249a * ExtractorJS.java 
refactor so considerStrings() is not static, allowing it to be overridden in subclasses
* ExtractorHTML.java, ExtractorJS.java
    add @Autowired parameter extractorJS used to process inline javascript, instead of call to static ExtractorJS.considerStrings()
 2012-12-10 16:51:42 -08:00
Noah Levitt e44cf9ca9b A little cleanup of FetchHTTPTest refactoring 2012-12-04 18:43:15 -08:00
Noah Levitt 5354960f00 Refactor FetchHTTPTest to use a TestSuite so that starting and shutting down the test http servers happen only once at start and finish respectively 2012-12-04 17:25:09 -08:00
Noah Levitt e30cb83540 * FetchHTTPTest.java 
testLaxUrlEncoding() - Tests a URL not correctly url-encoded, but that heritrix lets pass through to mimic browser behavior.
 2012-12-04 16:25:09 -08:00
Noah Levitt 4496c5d755 * FetchHTTPTest.java 
move to modules/src/test
    testFormAuth() - remove, there is FormAuthSelfTest.java, and this was the test that would fail in modules/
    testHttpBindAddress() - incorporate ed65493db0 - alternate testing of multiple local binds more likely to work more places
 2012-11-30 17:57:36 -08:00
Noah Levitt a20824aea7 Merge branch 'master' into new-fetchhttp-only 
Conflicts:
	.classpath
	engine/pom.xml
	modules/pom.xml
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java
	modules/src/test/java/org/archive/modules/fetcher/FetchHTTPTest.java
 2012-11-30 17:04:28 -08:00
Noah Levitt 92b302f164 * ExtractorMultipleRegexTest.java 
avoid "constant string too long" compile error
 2012-11-16 19:08:18 -08:00
Noah Levitt 94df568b4f * ExtractorMultipleRegex.java 
remove temporary performance testing code
 2012-11-16 18:34:28 -08:00
Noah Levitt aa98ec3327 * ExtractorMultipleRegex.java 
keep cache of groovy Template objects, since they are expensive to create
 2012-11-16 18:30:41 -08:00
Noah Levitt eeee01a52d * ExtractorMultipleRegex.java 
javadocs
 2012-11-16 16:42:43 -08:00
Noah Levitt 89dfef468c * ExtractorMultipleRegexTest.java 
test passes now; extractor gets what it can, which is most of the scroll down urls
 2012-11-15 19:18:59 -08:00
Noah Levitt dfd7a8a7d3 * ExtractorMultipleRegexTest.java 
turns out that __adt parameter can be found near the json blob - most, but not all, expected links are now found
 2012-11-15 18:58:11 -08:00
Noah Levitt 22ea489c57 * ExtractorMultipleRegex.java 
remove the "fooIndex" thing from available bindings, since it's kinda hacky and turned out not to be needed for our use case
 2012-11-15 18:56:54 -08:00
Noah Levitt 4af58f38e5 * CrawlURI.java 
outLinks, outCandidates - use LinkedHashSet to ensure predictable order (any reason not to do that?)
 2012-11-15 18:55:37 -08:00
Noah Levitt 75280a13a8 * ExtractorMultipleRegexTest.java 
working on making test work - first two scroll-down urls are extracted successfully, others fail
 2012-11-15 17:27:41 -08:00
Noah Levitt 13e1f9d64d * ExtractorMultipleRegex.java 
fix omission from last commit, uriRegex
 2012-11-15 14:01:01 -08:00
Noah Levitt 6ec4b5a973 * ExtractorMultipleRegex.java 
more refactoring to avoid redundant operations
 2012-11-15 13:55:11 -08:00
Noah Levitt 43dbcbfe81 * ExtractorMultipleRegex.java 
refactor for readability
 2012-11-15 13:35:16 -08:00
Noah Levitt 3f44a4bc66 Merge branch 'master' into MoreImpliedExtractor 2012-11-15 12:35:17 -08:00
Noah Levitt 8395e372e4 * ScriptedProcessor.java 
getEngine() - fix logic to respect isolateThreads setting
 2012-11-15 12:33:48 -08:00
Noah Levitt ba5bca807e * ExtractorMultipleRegex.java 
use groovy templating facility
 2012-11-15 12:32:42 -08:00
Noah Levitt 6e2d2129fe * ExtractorMultipleRegexTest.java 
use real use case actual content, actual urls from facebook for testing
* ExtractorMultipleRegex.java
    some comments, little twiddles
 2012-11-15 11:16:57 -08:00
Noah Levitt 1e2d86dc63 * ContentExtractorTestBase.java 
createRecorder(String,String) - new method for creating test Recorder and specifying charset
    createRecorder(String) - deprecate (use default charset as before)
 2012-11-15 11:14:25 -08:00
Noah Levitt 1d994493e2 more progress on ExtractorMultipleRegex 2012-11-14 19:58:43 -08:00