lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-13 17:26:01 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,576 Commits 39 Branches 33 Tags
47b7caad290cc5dde37cbdca06cd4fa3ab6b0d2b
Commit Graph
1576 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Noah Levitt 47b7caad29 initial bdb implementation; facade by topmost assigned domain, since cookies can apply to subdomains 2014-09-26 20:56:45 -07:00
Noah Levitt 886a0f2136 (forgot to add SimpleHostCookieSetStore) initial proof of concept fix for HER-2070 "cookie monster" bug 2014-09-26 20:05:37 -07:00
Noah Levitt f769e0636d initial proof of concept fix for HER-2070 "cookie monster" bug 2014-09-26 20:01:52 -07:00
Kenji Nagahashi 29379c7e19 Merge pull request #84 from nlevitt/iipc-webarchive-commons 
switch to mainline iipc webarchive-commons latest release (1.1.4)
 2014-09-11 17:19:08 -07:00
Kenji Nagahashi 6ada6801de Merge pull request #94 from nlevitt/ari-3724 
do not require "+" (plus sign) before @OPERATOR_CONTACT_URL@ in user-age...
 2014-09-11 17:17:55 -07:00
Kenji Nagahashi 8136046442 Merge pull request #92 from nlevitt/remove-cruft 
remove more unused classes (S3URLConnection)
 2014-09-11 17:13:48 -07:00
Noah Levitt e2064237cf Merge pull request #87 from vonrosen/ari-3977 
Allow xml extractor to handle urls in CDATA.
 2014-09-10 12:42:34 -07:00
Noah Levitt 1d0f60a2be webarchive-commons 1.1.4 has been released, bump dependency 2014-09-10 12:31:01 -07:00
Noah Levitt 066cd17b5e Merge pull request #93 from vonrosen/ari-4020 
Allow urls in JavaScript between unicode quotes to be detected.
 2014-09-08 18:03:34 -07:00
Hunter Stern 2af19fa03b Revise regex instead of unescaping js. 2014-09-08 16:37:03 -07:00
Noah Levitt 98a6c08a86 remove extraneous backslashes (thanks Kris) 2014-09-05 14:38:14 -07:00
Noah Levitt df7d3b506c do not require "+" (plus sign) before @OPERATOR_CONTACT_URL@ in user-agent -- many bots including googlebot have the +, but some don't, so having it in the default template seems sufficient 2014-09-05 12:59:17 -07:00
vonrosen 035a7b0a08 Merge pull request #80 from nlevitt/ari-3617-data-remote 
ExtractorHTML when a/@href links include the attribute data-remote="true...
 2014-09-04 16:08:33 -07:00
Hunter Stern 2e4f513a4c Revise parsing of string with unicode quoted data 2014-09-03 15:27:58 -07:00
Hunter Stern ea235f4466 Allow ExtractorJS to detect urls within unicode code points for " and \' 2014-09-02 15:18:13 -07:00
Noah Levitt 935b8f9658 remove unused dependency jets3t (thanks kenji) 2014-09-02 11:00:34 -07:00
Hunter Stern 27d632159d Add test case for ExtractorXML 2014-09-01 10:13:25 -07:00
Hunter Stern 2705bd3179 Merge remote-tracking branch 'internetarchive/master' into ari-3977 2014-09-01 09:23:26 -07:00
Noah Levitt 25c3b3e680 Merge pull request #77 from vonrosen/ari-3943 
Force urls received from AMQP to be recrawled so custom http headers can...
 2014-08-29 15:26:37 -07:00
Noah Levitt ec0839a6fe remove more unused classes 2014-08-29 15:23:21 -07:00
Kenji Nagahashi 01af800b35 Merge pull request #91 from adam-miller/FetchHTTP_Empty_Content_Length 
FetchHTTP.java
 2014-08-27 17:43:34 -07:00
Adam Miller bf248f407b FetchHTTP.java 
Handle response header with a null or empty Content-Length
 2014-08-27 17:14:05 -07:00
Noah Levitt 9e1ade3de6 Merge pull request #90 from kngenie/wbm-recrawl 
Move Wayback-dedup module to heritrix-contrib
 2014-08-27 10:24:37 -07:00
Kenji Nagahashi f343ce2ae1 suppress JUnit 4 dependency through hbase-client. 
rewrite WbmPersistLoadProcessorTest for JUnit 3.
  (also disable a test that depends on production CDX server)
 2014-08-26 10:56:35 -07:00
Kenji Nagahashi fc38175182 FIX: WbmPersistLoadProcessor.requestHeaders are ignored. 
(code commented out during update for httpclient 4.3.3)
 2014-08-25 17:05:50 -07:00
Kenji Nagahashi bb511796ec Update POMs for newly imported Wayback-dedup module. 
lock httpclient version to 4.3.3 stated by
	heritrix-commons. have to exclude hadoop’s
	dependency on jets3t 0.9.0 as it pulls in		httpcore 4.1.2.
	add easy mock 3.1 to heritrix-contrib.
 2014-08-25 16:44:48 -07:00
Kenji Nagahashi 651de1584d import Wayback-dedup modules from crawltools project to Heritrix3 contrib 2014-08-25 15:38:04 -07:00
Kenji Nagahashi d88e19716b move WbmPersistLoadProcessor to final location in the tree 2014-08-25 15:37:22 -07:00
Kenji Nagahashi 93dd896217 Merge pull request #86 from nlevitt/remove-cruft 
remove unused Transform* classes
 2014-08-25 15:30:08 -07:00
Kenji Nagahashi b6599d3b22 Merge pull request #89 from kngenie/beanbrowser-fix 
Don’t let exception from property getter fail entire bean-browser.
 2014-08-25 14:56:18 -07:00
Kenji Nagahashi 2a38fa0b12 Don’t let exception from property getter fail entire bean-browser. 2014-08-25 13:36:25 -07:00
Kenji Nagahashi 3450cf37e4 Update HQ modules and WBM-dedup module to match Heritrix 3.3.0. 
HQ modules: for method removed in Heritrix, API deprecated in httpclient
  WBM-dedup: fix NPE in FetchHistoryProcessor due to changed expectation.
 2014-08-24 00:45:04 -07:00
Kenji Nagahashi 6868a9ee5c Merge pull request #88 from nlevitt/crawluri-compare-bug 
fix bug in CrawlURI.compare() discovered by Kenji, add unit test CrawlUR...
 2014-08-19 17:17:03 -07:00
Noah Levitt 04921073ca fix bug in CrawlURI.compare() discovered by Kenji, add unit test CrawlURITest.testOrdering() 2014-08-19 17:14:32 -07:00
Noah Levitt f1b0633a3d remove unused Transform* classes 2014-08-13 10:53:03 -07:00
Noah Levitt 65cf7942c2 replace use of another method removed in updated guava library 2014-08-11 11:23:59 -07:00
Noah Levitt 651d18beb1 replace use of method removed from guava library 2014-08-08 17:38:51 -07:00
Noah Levitt 21f17a0d9f switch to mainline iipc webarchive-commons latest release 2014-08-08 17:26:21 -07:00
Hunter Stern 7f1700400b Allow xml extractor to handle urls in CDATA. 2014-08-01 17:37:31 -07:00
vonrosen 4298c3de69 Merge pull request #83 from nlevitt/revisit-redux 
oops! count novel urls/bytes for hosts report, etc
 2014-07-25 16:08:33 -07:00
Noah Levitt e5ecde6c2f oops! count novel urls/bytes for hosts report, etc 2014-07-25 15:57:53 -07:00
Noah Levitt e7b034c60e Merge pull request #82 from kris-sigur/HER-2071 
Fix for HER-2071
 2014-07-25 12:13:02 -07:00
Kristinn Sigurðsson 8edebc0943 Fix for HER-2071 2014-07-25 13:32:08 +00:00
Vinay Goel 1c6746c6c8 Merge pull request #81 from nlevitt/hbase-cdh5 
Hbase cdh5
 2014-07-23 13:34:54 -07:00
Noah Levitt 40eae4cbce Merge branch 'master' into hbase-cdh5 
Conflicts:
	modules/src/main/java/org/archive/modules/writer/WARCWriterProcessor.java
 2014-07-23 13:33:58 -07:00
vonrosen ba48f96d93 Merge pull request #78 from nlevitt/extract-html-no-mimetype 
treat content as html and extract links if it looks like html, even if m...
 2014-07-22 15:41:44 -07:00
Kristinn Sigurðsson d0ebd40578 Merge pull request #79 from nlevitt/revisit-redux 
Revisit redux
 2014-07-21 08:40:30 +00:00
Noah Levitt 410e1709b7 ExtractorHTML when a/@href links include the attribute data-remote="true", include that info the viaContext; and in FetchHTTPRequest, when a CrawlURI has data-remote="true", add the http header "X-Requested-With: XMLHttpRequest"; data-remote is a jquery thing that means the url can be requested via ajax in a browser... some urls work wrong without the X-Requested-With header... this is not a great place for a non-generic thing like this, but where else could it go... 2014-07-18 19:48:15 -07:00
Noah Levitt 4b75dfa16f treat content as html and extract links if it looks like html, even if mime type belies that 2014-07-17 21:09:51 -07:00
Hunter Stern bdeaa8bf0e Remove fetch status setting in AMQP publisher. 2014-07-16 15:46:53 -07:00