lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-25 07:56:35 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,613 Commits 38 Branches 32 Tags
8c318e53f6ebd13c1937535795fd57e973ea692e
Commit Graph
1613 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Hunter Stern 8c318e53f6 Remove forcefetch for AMQP received urls so they don't get crawled twice. 2014-10-16 22:02:43 +00:00
vonrosen 5909042b84 Merge pull request #99 from nlevitt/aitfive-39 
AMQPCrawlLogFeed, DecideRuleSequenceWithAMQPFeed, DecideRuleSequence.logExtraInfo
 2014-10-16 17:24:37 +00:00
Noah Levitt a27dbd182d change DecideRuleSequence extraInfo field name 'source' to 'seed' (for reason external to heritrix) 2014-10-14 15:40:07 -07:00
Noah Levitt 79aadf919b rename AMQPCrawlLogFeed fields to match James's names in druid (looks like there's no way to translate) 2014-10-14 14:00:55 -07:00
Noah Levitt 5f7dc49599 Merge pull request #95 from kngenie/patch-beanbrowser-failure 
FIX corner-case of bean browser failing due to an exception from hashCode()
 2014-10-03 19:21:27 -07:00
Noah Levitt 0f1760fbfe Merge pull request #97 from kris-sigur/HER-2074 
Fix for HER-2074
 2014-10-03 18:48:42 -07:00
Kristinn Sigurðsson 0b16ff0fad Fix for HER-2074 
https://webarchive.jira.com/browse/HER-2074
 2014-10-03 11:30:38 +00:00
Kristinn Sigurðsson 86ebed04cc Merge pull request #96 from nlevitt/her-2070 
new cookie store system to address HER-2070 "cookie monster" bug
 2014-10-03 09:33:04 +00:00
Noah Levitt 5cb5f05437 add field "extra info" to DecideRuleSequence log, which is a json object with entries for host, via, source and hop path 2014-10-02 02:34:32 -07:00
Noah Levitt 743d76c966 set <encoding>UTF-8</encoding> in maven-compiler-plugin; project.build.sourceEncoding seems not to be enough, see http://stackoverflow.com/questions/8978013/error-unmappable-character-for-encoding-utf8-during-maven-compilation 2014-10-02 01:48:13 -07:00
Noah Levitt 78e6b8b6ab replace accidental U+2212 MINUS SIGN with normal ascii 0x2d '-' to avoid compilation error "unmappable character for encoding ASCII" 2014-10-01 21:31:14 -07:00
Noah Levitt e5e5cd87e4 configure maven-compiler-plugin in root pom to avoid redundancy, inconsistency 2014-10-01 13:38:39 -07:00
Noah Levitt 060054e6b3 avoid java 7 api 2014-10-01 13:38:14 -07:00
Noah Levitt 6faa1b8c1c fix misleading variable name 2014-09-30 11:43:09 -07:00
Noah Levitt 8d98882eb7 an optimization suggested by Gordon: organize cookies by host (domain), rather than top private domain, and supply a composite list of cookies from the host and each parent domain to the http client library 2014-09-29 18:32:31 -07:00
Noah Levitt 5fd120dffd some javadocs and other comments, some minor refactoring, change delimiter to ';' for the sortable key 2014-09-29 17:51:24 -07:00
Noah Levitt 87b408cc23 rename limited cookie store facade class 2014-09-28 14:35:22 -07:00
Noah Levitt 610a1e1e97 cookie store is called BdbCookieStore again 2014-09-28 14:32:34 -07:00
Noah Levitt 54eb1c950f rudimentary test of cookie at url with ip address as host 2014-09-28 14:30:01 -07:00
Noah Levitt 5c21ad1762 AbstractCookieStore.getCookieFor(CrawlURI) computes top private domain, so FetchHTTP doesn't have to know anything about that 2014-09-28 14:29:04 -07:00
Noah Levitt 4aa85af1b0 a few tests exercising some basic domain/subdomain cookie behaviors 2014-09-28 14:15:27 -07:00
Noah Levitt a37a8de4a2 improved implementation not using TreeSet, and some initial work on integration testing of cookie handling 2014-09-28 11:59:14 -07:00
Noah Levitt 6588641017 restore saving and loading, checkpoints, testing 2014-09-27 13:55:07 -07:00
Noah Levitt 111294e887 oops, commit classes for initial bdb implementation 2014-09-27 12:22:50 -07:00
Noah Levitt 47b7caad29 initial bdb implementation; facade by topmost assigned domain, since cookies can apply to subdomains 2014-09-26 20:56:45 -07:00
Noah Levitt 886a0f2136 (forgot to add SimpleHostCookieSetStore) initial proof of concept fix for HER-2070 "cookie monster" bug 2014-09-26 20:05:37 -07:00
Noah Levitt f769e0636d initial proof of concept fix for HER-2070 "cookie monster" bug 2014-09-26 20:01:52 -07:00
Kenji Nagahashi 29379c7e19 Merge pull request #84 from nlevitt/iipc-webarchive-commons 
switch to mainline iipc webarchive-commons latest release (1.1.4)
 2014-09-11 17:19:08 -07:00
Kenji Nagahashi 6ada6801de Merge pull request #94 from nlevitt/ari-3724 
do not require "+" (plus sign) before @OPERATOR_CONTACT_URL@ in user-age...
 2014-09-11 17:17:55 -07:00
Kenji Nagahashi 8136046442 Merge pull request #92 from nlevitt/remove-cruft 
remove more unused classes (S3URLConnection)
 2014-09-11 17:13:48 -07:00
Kenji Nagahashi 5821fe7b1e FIX corner-case of bean browser failing due to an exception from hashCode(). 2014-09-11 11:22:01 -07:00
Noah Levitt e2064237cf Merge pull request #87 from vonrosen/ari-3977 
Allow xml extractor to handle urls in CDATA.
 2014-09-10 12:42:34 -07:00
Noah Levitt 1d0f60a2be webarchive-commons 1.1.4 has been released, bump dependency 2014-09-10 12:31:01 -07:00
Noah Levitt 066cd17b5e Merge pull request #93 from vonrosen/ari-4020 
Allow urls in JavaScript between unicode quotes to be detected.
 2014-09-08 18:03:34 -07:00
Hunter Stern 2af19fa03b Revise regex instead of unescaping js. 2014-09-08 16:37:03 -07:00
Noah Levitt 98a6c08a86 remove extraneous backslashes (thanks Kris) 2014-09-05 14:38:14 -07:00
Noah Levitt df7d3b506c do not require "+" (plus sign) before @OPERATOR_CONTACT_URL@ in user-agent -- many bots including googlebot have the +, but some don't, so having it in the default template seems sufficient 2014-09-05 12:59:17 -07:00
vonrosen 035a7b0a08 Merge pull request #80 from nlevitt/ari-3617-data-remote 
ExtractorHTML when a/@href links include the attribute data-remote="true...
 2014-09-04 16:08:33 -07:00
Hunter Stern 2e4f513a4c Revise parsing of string with unicode quoted data 2014-09-03 15:27:58 -07:00
Hunter Stern ea235f4466 Allow ExtractorJS to detect urls within unicode code points for " and \' 2014-09-02 15:18:13 -07:00
Noah Levitt 935b8f9658 remove unused dependency jets3t (thanks kenji) 2014-09-02 11:00:34 -07:00
Hunter Stern 27d632159d Add test case for ExtractorXML 2014-09-01 10:13:25 -07:00
Hunter Stern 2705bd3179 Merge remote-tracking branch 'internetarchive/master' into ari-3977 2014-09-01 09:23:26 -07:00
Noah Levitt 51a1cad300 new class DecideRuleSequenceWithAMQPFeed publishes scoping decisions to amqp 2014-08-29 17:08:32 -07:00
Noah Levitt 37442104cc move AMQP producer code to a separate class so it can be used by classes that are not heritrix processors 2014-08-29 17:05:16 -07:00
Noah Levitt 25c3b3e680 Merge pull request #77 from vonrosen/ari-3943 
Force urls received from AMQP to be recrawled so custom http headers can...
 2014-08-29 15:26:37 -07:00
Noah Levitt ec0839a6fe remove more unused classes 2014-08-29 15:23:21 -07:00
Kenji Nagahashi 01af800b35 Merge pull request #91 from adam-miller/FetchHTTP_Empty_Content_Length 
FetchHTTP.java
 2014-08-27 17:43:34 -07:00
Adam Miller bf248f407b FetchHTTP.java 
Handle response header with a null or empty Content-Length
 2014-08-27 17:14:05 -07:00
Noah Levitt 152c4aeb9f use JSONObject.NULL for null values so that they appear explicitly in the json 2014-08-27 15:33:19 -07:00