lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-26 00:16:52 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,650 Commits 38 Branches 32 Tags
4ea7efdfc8a3560cb71f0f5f96ab70b9d2cf006b
Commit Graph
678 Commits
Author SHA1 Message Date
vonrosen 3609abe38c Merge pull request #108 from nlevitt/aitfive-203 
kafka crawl log feed
 2014-11-14 10:29:41 -08:00
Noah Levitt d96b1f8eca Merge pull request #100 from vonrosen/ari-4061 
Allow discovery of urls in content attribute of meta tags.
 2014-11-08 00:05:00 -08:00
Noah Levitt 7312ad597e so that it can appear in the crawl log, add contentSize to CrawlURI extraInfo, for http transactions (the only case in heritrix currently where contentLength and contentSize differ) 2014-11-07 13:21:42 -08:00
Hunter Stern 9a41b31264 Handle case where form does not have an action defined. 2014-11-04 11:22:38 -08:00
Hunter Stern 78e98365d6 Remove unused variable. 2014-10-16 17:51:08 +00:00
vonrosen 5909042b84 Merge pull request #99 from nlevitt/aitfive-39 
AMQPCrawlLogFeed, DecideRuleSequenceWithAMQPFeed, DecideRuleSequence.logExtraInfo
 2014-10-16 17:24:37 +00:00
Noah Levitt a27dbd182d change DecideRuleSequence extraInfo field name 'source' to 'seed' (for reason external to heritrix) 2014-10-14 15:40:07 -07:00
Hunter Stern 3a82b37e52 Ensure Jerico extractor html doesn't call new metadata test method. 2014-10-10 23:33:22 +00:00
Hunter Stern 3347de3d39 Make meta tag content url extraction rules more generate so as to apply to all meta tags 'content' attributes, not just specific og:video. 2014-10-09 23:42:24 +00:00
Hunter Stern ff5c59c555 Allow extraction of content url in opengraph video meta tag - og:video 2014-10-09 17:57:05 +00:00
Kristinn Sigurðsson 0b16ff0fad Fix for HER-2074 
https://webarchive.jira.com/browse/HER-2074
 2014-10-03 11:30:38 +00:00
Noah Levitt 5cb5f05437 add field "extra info" to DecideRuleSequence log, which is a json object with entries for host, via, source and hop path 2014-10-02 02:34:32 -07:00
Noah Levitt e5e5cd87e4 configure maven-compiler-plugin in root pom to avoid redundancy, inconsistency 2014-10-01 13:38:39 -07:00
Noah Levitt 060054e6b3 avoid java 7 api 2014-10-01 13:38:14 -07:00
Noah Levitt 6faa1b8c1c fix misleading variable name 2014-09-30 11:43:09 -07:00
Noah Levitt 8d98882eb7 an optimization suggested by Gordon: organize cookies by host (domain), rather than top private domain, and supply a composite list of cookies from the host and each parent domain to the http client library 2014-09-29 18:32:31 -07:00
Noah Levitt 5fd120dffd some javadocs and other comments, some minor refactoring, change delimiter to ';' for the sortable key 2014-09-29 17:51:24 -07:00
Noah Levitt 87b408cc23 rename limited cookie store facade class 2014-09-28 14:35:22 -07:00
Noah Levitt 54eb1c950f rudimentary test of cookie at url with ip address as host 2014-09-28 14:30:01 -07:00
Noah Levitt 5c21ad1762 AbstractCookieStore.getCookieFor(CrawlURI) computes top private domain, so FetchHTTP doesn't have to know anything about that 2014-09-28 14:29:04 -07:00
Noah Levitt 4aa85af1b0 a few tests exercising some basic domain/subdomain cookie behaviors 2014-09-28 14:15:27 -07:00
Noah Levitt a37a8de4a2 improved implementation not using TreeSet, and some initial work on integration testing of cookie handling 2014-09-28 11:59:14 -07:00
Noah Levitt 6588641017 restore saving and loading, checkpoints, testing 2014-09-27 13:55:07 -07:00
Noah Levitt 111294e887 oops, commit classes for initial bdb implementation 2014-09-27 12:22:50 -07:00
Noah Levitt 47b7caad29 initial bdb implementation; facade by topmost assigned domain, since cookies can apply to subdomains 2014-09-26 20:56:45 -07:00
Noah Levitt 886a0f2136 (forgot to add SimpleHostCookieSetStore) initial proof of concept fix for HER-2070 "cookie monster" bug 2014-09-26 20:05:37 -07:00
Noah Levitt f769e0636d initial proof of concept fix for HER-2070 "cookie monster" bug 2014-09-26 20:01:52 -07:00
Kenji Nagahashi 29379c7e19 Merge pull request #84 from nlevitt/iipc-webarchive-commons 
switch to mainline iipc webarchive-commons latest release (1.1.4)
 2014-09-11 17:19:08 -07:00
Kenji Nagahashi 6ada6801de Merge pull request #94 from nlevitt/ari-3724 
do not require "+" (plus sign) before @OPERATOR_CONTACT_URL@ in user-age...
 2014-09-11 17:17:55 -07:00
Noah Levitt e2064237cf Merge pull request #87 from vonrosen/ari-3977 
Allow xml extractor to handle urls in CDATA.
 2014-09-10 12:42:34 -07:00
Noah Levitt 066cd17b5e Merge pull request #93 from vonrosen/ari-4020 
Allow urls in JavaScript between unicode quotes to be detected.
 2014-09-08 18:03:34 -07:00
Hunter Stern 2af19fa03b Revise regex instead of unescaping js. 2014-09-08 16:37:03 -07:00
Noah Levitt 98a6c08a86 remove extraneous backslashes (thanks Kris) 2014-09-05 14:38:14 -07:00
Noah Levitt df7d3b506c do not require "+" (plus sign) before @OPERATOR_CONTACT_URL@ in user-agent -- many bots including googlebot have the +, but some don't, so having it in the default template seems sufficient 2014-09-05 12:59:17 -07:00
vonrosen 035a7b0a08 Merge pull request #80 from nlevitt/ari-3617-data-remote 
ExtractorHTML when a/@href links include the attribute data-remote="true...
 2014-09-04 16:08:33 -07:00
Hunter Stern 2e4f513a4c Revise parsing of string with unicode quoted data 2014-09-03 15:27:58 -07:00
Hunter Stern ea235f4466 Allow ExtractorJS to detect urls within unicode code points for " and \' 2014-09-02 15:18:13 -07:00
Hunter Stern 27d632159d Add test case for ExtractorXML 2014-09-01 10:13:25 -07:00
Hunter Stern 2705bd3179 Merge remote-tracking branch 'internetarchive/master' into ari-3977 2014-09-01 09:23:26 -07:00
Noah Levitt 51a1cad300 new class DecideRuleSequenceWithAMQPFeed publishes scoping decisions to amqp 2014-08-29 17:08:32 -07:00
Adam Miller bf248f407b FetchHTTP.java 
Handle response header with a null or empty Content-Length
 2014-08-27 17:14:05 -07:00
Noah Levitt 04921073ca fix bug in CrawlURI.compare() discovered by Kenji, add unit test CrawlURITest.testOrdering() 2014-08-19 17:14:32 -07:00
Noah Levitt 9a0c0656b2 move HostsReport.fixup(String) to CrawlHost.fixUpName() so it can be called elsewhere 2014-08-12 16:45:53 -07:00
Noah Levitt 65cf7942c2 replace use of another method removed in updated guava library 2014-08-11 11:23:59 -07:00
Hunter Stern 7f1700400b Allow xml extractor to handle urls in CDATA. 2014-08-01 17:37:31 -07:00
vonrosen 4298c3de69 Merge pull request #83 from nlevitt/revisit-redux 
oops! count novel urls/bytes for hosts report, etc
 2014-07-25 16:08:33 -07:00
Noah Levitt e5ecde6c2f oops! count novel urls/bytes for hosts report, etc 2014-07-25 15:57:53 -07:00
Noah Levitt e7b034c60e Merge pull request #82 from kris-sigur/HER-2071 
Fix for HER-2071
 2014-07-25 12:13:02 -07:00
Kristinn Sigurðsson 8edebc0943 Fix for HER-2071 2014-07-25 13:32:08 +00:00
vonrosen ba48f96d93 Merge pull request #78 from nlevitt/extract-html-no-mimetype 
treat content as html and extract links if it looks like html, even if m...
 2014-07-22 15:41:44 -07:00