lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-25 16:06:33 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,554 Commits 38 Branches 32 Tags
035a7b0a0899ae91cea6eaa58ba87be36df1c5cd
Commit Graph
1554 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
vonrosen 035a7b0a08 Merge pull request #80 from nlevitt/ari-3617-data-remote 
ExtractorHTML when a/@href links include the attribute data-remote="true...
 2014-09-04 16:08:33 -07:00
Noah Levitt 25c3b3e680 Merge pull request #77 from vonrosen/ari-3943 
Force urls received from AMQP to be recrawled so custom http headers can...
 2014-08-29 15:26:37 -07:00
Kenji Nagahashi 01af800b35 Merge pull request #91 from adam-miller/FetchHTTP_Empty_Content_Length 
FetchHTTP.java
 2014-08-27 17:43:34 -07:00
Adam Miller bf248f407b FetchHTTP.java 
Handle response header with a null or empty Content-Length
 2014-08-27 17:14:05 -07:00
Noah Levitt 9e1ade3de6 Merge pull request #90 from kngenie/wbm-recrawl 
Move Wayback-dedup module to heritrix-contrib
 2014-08-27 10:24:37 -07:00
Kenji Nagahashi f343ce2ae1 suppress JUnit 4 dependency through hbase-client. 
rewrite WbmPersistLoadProcessorTest for JUnit 3.
  (also disable a test that depends on production CDX server)
 2014-08-26 10:56:35 -07:00
Kenji Nagahashi fc38175182 FIX: WbmPersistLoadProcessor.requestHeaders are ignored. 
(code commented out during update for httpclient 4.3.3)
 2014-08-25 17:05:50 -07:00
Kenji Nagahashi bb511796ec Update POMs for newly imported Wayback-dedup module. 
lock httpclient version to 4.3.3 stated by
	heritrix-commons. have to exclude hadoop’s
	dependency on jets3t 0.9.0 as it pulls in		httpcore 4.1.2.
	add easy mock 3.1 to heritrix-contrib.
 2014-08-25 16:44:48 -07:00
Kenji Nagahashi 651de1584d import Wayback-dedup modules from crawltools project to Heritrix3 contrib 2014-08-25 15:38:04 -07:00
Kenji Nagahashi d88e19716b move WbmPersistLoadProcessor to final location in the tree 2014-08-25 15:37:22 -07:00
Kenji Nagahashi 93dd896217 Merge pull request #86 from nlevitt/remove-cruft 
remove unused Transform* classes
 2014-08-25 15:30:08 -07:00
Kenji Nagahashi b6599d3b22 Merge pull request #89 from kngenie/beanbrowser-fix 
Don’t let exception from property getter fail entire bean-browser.
 2014-08-25 14:56:18 -07:00
Kenji Nagahashi 2a38fa0b12 Don’t let exception from property getter fail entire bean-browser. 2014-08-25 13:36:25 -07:00
Kenji Nagahashi 3450cf37e4 Update HQ modules and WBM-dedup module to match Heritrix 3.3.0. 
HQ modules: for method removed in Heritrix, API deprecated in httpclient
  WBM-dedup: fix NPE in FetchHistoryProcessor due to changed expectation.
 2014-08-24 00:45:04 -07:00
Kenji Nagahashi 6868a9ee5c Merge pull request #88 from nlevitt/crawluri-compare-bug 
fix bug in CrawlURI.compare() discovered by Kenji, add unit test CrawlUR...
 2014-08-19 17:17:03 -07:00
Noah Levitt 04921073ca fix bug in CrawlURI.compare() discovered by Kenji, add unit test CrawlURITest.testOrdering() 2014-08-19 17:14:32 -07:00
Noah Levitt f1b0633a3d remove unused Transform* classes 2014-08-13 10:53:03 -07:00
vonrosen 4298c3de69 Merge pull request #83 from nlevitt/revisit-redux 
oops! count novel urls/bytes for hosts report, etc
 2014-07-25 16:08:33 -07:00
Noah Levitt e5ecde6c2f oops! count novel urls/bytes for hosts report, etc 2014-07-25 15:57:53 -07:00
Noah Levitt e7b034c60e Merge pull request #82 from kris-sigur/HER-2071 
Fix for HER-2071
 2014-07-25 12:13:02 -07:00
Kristinn Sigurðsson 8edebc0943 Fix for HER-2071 2014-07-25 13:32:08 +00:00
Vinay Goel 1c6746c6c8 Merge pull request #81 from nlevitt/hbase-cdh5 
Hbase cdh5
 2014-07-23 13:34:54 -07:00
Noah Levitt 40eae4cbce Merge branch 'master' into hbase-cdh5 
Conflicts:
	modules/src/main/java/org/archive/modules/writer/WARCWriterProcessor.java
 2014-07-23 13:33:58 -07:00
vonrosen ba48f96d93 Merge pull request #78 from nlevitt/extract-html-no-mimetype 
treat content as html and extract links if it looks like html, even if m...
 2014-07-22 15:41:44 -07:00
Kristinn Sigurðsson d0ebd40578 Merge pull request #79 from nlevitt/revisit-redux 
Revisit redux
 2014-07-21 08:40:30 +00:00
Noah Levitt 410e1709b7 ExtractorHTML when a/@href links include the attribute data-remote="true", include that info the viaContext; and in FetchHTTPRequest, when a CrawlURI has data-remote="true", add the http header "X-Requested-With: XMLHttpRequest"; data-remote is a jquery thing that means the url can be requested via ajax in a browser... some urls work wrong without the X-Requested-With header... this is not a great place for a non-generic thing like this, but where else could it go... 2014-07-18 19:48:15 -07:00
Noah Levitt 4b75dfa16f treat content as html and extract links if it looks like html, even if mime type belies that 2014-07-17 21:09:51 -07:00
Hunter Stern bdeaa8bf0e Remove fetch status setting in AMQP publisher. 2014-07-16 15:46:53 -07:00
Hunter Stern c3d5996314 Force urls received from AMQP to be recrawled so custom http headers can be used to make the request for that url. 2014-07-14 17:29:15 -07:00
Noah Levitt cb6ac8b8b5 restore writeRevisitFor* settings, but issue warning in setter and ignore the value 2014-07-14 10:53:37 -07:00
Noah Levitt da30439d61 oops, fix 304 tests 2014-07-11 12:47:15 -07:00
Noah Levitt 77492802b3 test 304 not modified revist profile in FetchHTTPTest 2014-07-10 21:30:54 -07:00
Noah Levitt 2292ea9287 remove deprecated unused stuff 2014-07-10 21:29:47 -07:00
Noah Levitt 810ad91b9d set revisit profile on 304 not modified 2014-07-10 21:28:52 -07:00
Noah Levitt 905a98c0c2 use RevisitProfile for tallying stats 2014-07-10 21:27:50 -07:00
Noah Levitt d9fd66878b Merge branch 'master' into revisit-redux 
Conflicts:
	modules/src/main/java/org/archive/modules/CrawlURI.java
	modules/src/main/java/org/archive/modules/writer/WARCWriterProcessor.java
 2014-07-10 15:01:43 -07:00
Kristinn Sigurðsson f101bba509 Merge pull request #76 from nlevitt/HER-2039 
HER-2039 remove class Link, use CrawlURI

It should be noted that this is an API change that will affect any custom link extractors that people may have built.
 2014-07-10 09:41:00 +00:00
Noah Levitt ee52f48be9 remove LinksScoper (deprecated since 2009) and remove recently added code only used in LinksScoper 2014-07-09 10:14:24 -07:00
Noah Levitt 7bd9f3acaf switch usage of Link to CrawlURI in heritrix-contrib 2014-07-08 11:56:56 -07:00
Noah Levitt 6e9a0a4d05 only write WARC-Refers-To header when value is not null 2014-07-07 19:21:07 -07:00
Noah Levitt 2854798964 new parameter keySuffix-- If not null, keySuffix is appended to the lookup key when loading and storing digest history. Thus the key looks like {digest}{keySuffix}, e.g. "sha1:22SFHXERHNFOEY6WK7YOUN4PFIPZSB4D-1193". The purpose is to support multiple namespaces in a single hbase table, to avoid proliferation of small tables. The reason we use a suffix instead of a prefix is to leave open the possibility of deduplication across these different namespaces at some point in the future. 2014-07-07 19:05:39 -07:00
Noah Levitt bed1e58adf copy over and adjust equals(), hashCode(), compareTo() from old Link class to CrawlURI, so that tests pass 2014-07-07 16:59:18 -07:00
Noah Levitt bfac3c8bbc ensure non-null pathFromSeed to fix some unit tests 2014-07-07 12:25:27 -07:00
Noah Levitt 3c2e1315bb other changes required for merge from master into HER-2039 2014-07-07 11:48:36 -07:00
Noah Levitt f23a59a918 Merge branch 'master' into HER-2039 
Conflicts:
	modules/src/main/java/org/archive/modules/CrawlURI.java
	modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
	modules/src/main/java/org/archive/modules/extractor/ExtractorSWF.java
	modules/src/main/java/org/archive/modules/extractor/Link.java
	modules/src/main/java/org/archive/modules/writer/WARCWriterProcessor.java
 2014-07-07 11:45:27 -07:00
Adam Miller c48de14663 Merge pull request #75 from nlevitt/uri-inherited-data-fix 
in CrawlURI.createCrawlURI(), avoid clobbering inherited data with data ...
 2014-07-02 15:45:56 -07:00
Noah Levitt f2055ee2ae in CrawlURI.createCrawlURI(), avoid clobbering inherited data with data from Link 2014-07-02 13:17:42 -07:00
Noah Levitt 47e4e98608 Merge pull request #74 from vonrosen/ari-3943 
Fix for https://webarchive.jira.com/browse/ARI-3943
 2014-06-27 17:00:40 -07:00
Hunter Stern 5eade58de4 Fix for https://webarchive.jira.com/browse/ARI-3943 2014-06-27 16:20:42 -07:00
Noah Levitt 3e35c967ab upgrade hbase to cdh5 2014-06-16 18:33:25 -07:00