lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-22 21:55:54 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,156 Commits 38 Branches 33 Tags
b84d443f00a75d68fa022c61ba5b8a106bd6344a
Commit Graph
839 Commits
Author SHA1 Message Date
Noah Levitt a2977490c4 update for httpclient 4.3-beta3-SNAPSHOT 2013-08-09 18:33:15 -07:00
Noah Levitt 317ff7eb65 Merge branch 'master' into hc43 2013-08-09 18:17:24 -07:00
Noah Levitt 705a375daf uses of UriUtils.isLikelyUri() in Extractor{HTML,SWF,XML} with UriUtils.isVeryLikelyUri() to reap the benefits of HER-1523 improvements (should address archive-it issue ARI-3492) 2013-08-09 18:17:11 -07:00
Noah Levitt 750a688670 Merge branch 'master' into hc43 
Conflicts:
	commons/src/main/java/org/apache/commons/httpclient/HttpParser.java
	engine/pom.xml
	modules/src/main/java/org/archive/modules/CrawlURI.java
	modules/src/main/java/org/archive/modules/deciderules/MatchesStatusCodeDecideRule.java
	modules/src/main/java/org/archive/modules/deciderules/NotMatchesStatusCodeDecideRule.java
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java
 2013-08-08 21:14:11 -07:00
Noah Levitt 8c64cba9ae TransclusionDecideRule.java - consider form submission "S" hop as navlink like "L", not transcluded 2013-08-05 16:05:02 -07:00
Noah Levitt f4f3b9be38 add logging to BdbUriUniqFilter.forgetSchemeHost(), clean up some javadocs 2013-07-26 17:50:36 -07:00
Noah Levitt 72094213a4 MirrorWriterProcessor.java - change "path" setting to a ConfigPath and set default value "${launchId}/mirror" 2013-07-22 17:54:06 -07:00
Adam Miller b85828135b Convert MatchesStatusCodeDecideRule and NotMatchesStatusCodeDecideRule to use CrawlURI.getFetchStatus() to avoid null status codes and broaden use. 
Added test cases for MatchesStatusCodeDecideRule and NotMatchesStatusCodeDecideRule
Fix comments for youtube extractor.
 2013-06-21 12:03:41 -07:00
Noah Levitt f2984881f6 address HER-2042 - new annotation duplicate:uriAgnosticDigest for all url agnostic duplicates, whether written to warcs or not; stats consult this annotation so that duplicate/novel numbers are correct 2013-06-07 17:17:24 -07:00
Noah Levitt ffd248f780 HER-2040 ExtractorJS shouldExtract when content-type is application/json 2013-06-03 11:23:16 -07:00
Kristinn Sigurðsson 858523db94 HER-2039: Initial work. Eliminated Link class and dealt with resultant 
errors. Not tested any further than that.
 2013-05-15 13:36:01 +00:00
Noah Levitt d4c5bd6e98 avoid npe by setting default <input> type "text" when unspecified 2013-05-14 18:17:09 -07:00
Noah Levitt d972731958 ExtractorHTMLForms.java - handle a corner case extracting attribute values 2013-05-14 16:23:41 -07:00
Noah Levitt 7bd010e3c7 HTMLForm.java - default html <input> type is text, so consider <input> with no type as candidateUsernameInput 2013-05-13 22:53:37 -07:00
Noah Levitt bea5c6f267 followup on HER-2031 ExtractorHTMLForms.java - improve regexes looking for form and input attributes, mainly to avoid maxing cpu for long periods of time for certain input 2013-05-13 22:37:02 -07:00
Noah Levitt ec824e9724 avoid need for getRawData() method by introducing Link.hasData() 2013-05-13 17:43:45 -07:00
Kristinn Sigurðsson 7c5b2ceb00 Added a data map to Link. Becomes the CrawlURI's data map when Link is 
promoted to CrawlURI
 2013-05-13 15:18:02 +00:00
Noah Levitt d5693563f8 fix bug pointed out in https://webarchive.jira.com/browse/ARI-3176?focusedCommentId=35314 - url-agnostic dedupe not reported in host report (and other reports) 2013-04-29 18:11:24 -07:00
Noah Levitt 6a0ed067c0 consolidate disparate versions of WARCConstants into one version that lives in ia-web-commons 2013-03-29 17:12:32 -07:00
Kenji Nagahashi a4a3ce60cd fix failing ContentDigestHistoryTest 2013-03-26 15:06:31 -07:00
Noah Levitt 5d8fe0443d Merge branch 'master' into HER-2031 2013-03-23 19:16:05 -07:00
Noah Levitt 0631123783 close recorders on failure to avoid exception from retry cycle within httpclient (restore functionality from fcece02646 to work with current httpcomponents 2013-03-20 16:59:06 -07:00
Noah Levitt 75abd2509f bit of a hack to support shouldFetchBodyRule, given httpcomponents limitation discussed at https://issues.apache.org/jira/browse/HTTPCORE-325 2013-03-20 16:17:34 -07:00
Noah Levitt 1993c4388c Merge branch 'master' into hc43 
Conflicts:
	.classpath
 2013-03-20 14:28:13 -07:00
Noah Levitt 8fd361e518 update source/target version to 1.6 since we use 1.6 features, specifically @Override for methods only implementing an interface (not sure why maven wasn't enforcing this); also remove maven-antrun-plugin created timestamp.txt, doesn't seem to be used anywhere 2013-03-20 14:25:22 -07:00
Noah Levitt c509b6e552 remove more eclipse stuff, people like to manage their own 2013-03-20 13:39:28 -07:00
Noah Levitt b4bff3ffb9 new method RecordingInputStream.readToEndOfContent(), because now we can't rely on stopping right after the headers, which is an assumption of readFullyOrUntil() 2013-03-19 18:24:02 -07:00
Noah Levitt 0be1b8e318 partly done not working update to latest httpcomponents 2013-03-19 13:01:35 -07:00
Noah Levitt 12da824316 Merge branch 'master' into hc43 2013-03-18 23:32:51 -07:00
Noah Levitt b0c6619f80 Merge branch 'master' of github.com:internetarchive/heritrix3 2013-03-18 20:02:03 -07:00
Noah Levitt 0c9e2a1f4d ContentDigestHistory{Loader,Storer} - ignore uris with empty content body (deduping that is counterproductive) 2013-03-18 20:01:52 -07:00
Noah Levitt ff36546eda fix serverless whois url example in comment 2013-03-18 14:03:39 -07:00
gojomo c5c54ac138 comment-out debug output 2013-03-12 11:42:47 -07:00
gojomo 7c8295ee82 checkpoint support 2013-03-12 11:42:29 -07:00
gojomo a844878e7d Merge branch 'simple_logins' into HER-2031 2013-03-12 01:16:31 -07:00
gojomo 3f8590fe44 when found in CrawlURI, add WARC-Simple-Form-Province-Status header 2013-03-11 20:57:41 -07:00
gojomo bf9cab8eb2 when POST/submit-data requested, alter request method accordingly 2013-03-11 20:57:09 -07:00
gojomo bbac7d7320 per (overlay) settings, spawn high-priority POST when appropriate; annotate/add-headers to describe 2013-03-11 20:56:31 -07:00
gojomo ed639c8342 extract HTMLForm INPUT details 2013-03-11 20:55:20 -07:00
gojomo 94a61b4fb7 retain offsets of discovered FORMs to aid later processing 2013-03-11 20:53:41 -07:00
gojomo 6b0abae8ee new constants, hop-type; refactoring to help support form-submissions 2013-03-11 20:52:59 -07:00
Noah Levitt 8cbb84b8df Merge branch 'master' of github.com:internetarchive/heritrix3 2013-02-22 15:43:30 -08:00
Noah Levitt fb758e53e3 Avoid pulling in hadoop-core.jar and its dependencies; add plugin versions to get rid of warnings 2013-02-22 15:43:15 -08:00
Noah Levitt 878e3b1afe Remove spurious "throws" declaration 2013-02-05 18:37:22 -08:00
Noah Levitt c171a98c07 Bring up to date with latest httpcomponents after commit for https://issues.apache.org/jira/browse/HTTPCLIENT-1288 2013-01-03 19:13:53 -08:00
Noah Levitt 7d3fe554af Merge branch 'master' into hc43 
Conflicts:
	.classpath
	commons/src/main/java/org/apache/commons/httpclient/HttpMethodBase.java
 2013-01-03 18:18:22 -08:00
Noah Levitt b6133aee5f Update httpcomponents.diff 2012-12-31 14:15:05 -08:00
Noah Levitt 163871d462 Tweaks and twiddles 2012-12-31 01:57:03 -08:00
Noah Levitt 25234898ef Somewhat redundant form auth integration test restored, passes 2012-12-30 22:35:28 -08:00
Noah Levitt fcece02646 Close recorders on failure to avoid exception from retry cycle within httpclient- now all the FetchHTTPTests pass 2012-12-30 20:45:05 -08:00