lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-25 16:06:33 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,634 Commits 38 Branches 32 Tags
a0f70d0865198908e2fa700772ca0694ee079a96
Commit Graph
1634 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Noah Levitt a0f70d0865 Revert "switch to org.apache.kafka.clients.producer.KafkaProducer to see if we can avoid the InterruptedException at the end of the crawl (and log warnings on failure to send message)" 
This reverts commit d2f70e5c97.

Conflicts:
	contrib/src/main/java/org/archive/modules/postprocessor/KafkaCrawlLogFeed.java
 2014-11-10 13:55:13 -08:00
Noah Levitt fa0a28f78b Merge branch 'aitfive-203' of github.com:nlevitt/heritrix3 into aitfive-203 2014-11-07 15:59:24 -08:00
Noah Levitt 192b3f9b4b for kafka/amqp crawl log feeds, remove now redundant contentSize from extraInfo 2014-11-07 15:59:14 -08:00
Noah Levitt 4527d5af29 Merge branch 'extra-info-size' into aitfive-203 2014-11-07 15:58:08 -08:00
Noah Levitt 7312ad597e so that it can appear in the crawl log, add contentSize to CrawlURI extraInfo, for http transactions (the only case in heritrix currently where contentLength and contentSize differ) 2014-11-07 13:21:42 -08:00
Noah Levitt e2be3bbf9c new style kafka configuration properties 2014-11-06 19:53:41 -08:00
Noah Levitt d2f70e5c97 switch to org.apache.kafka.clients.producer.KafkaProducer to see if we can avoid the InterruptedException at the end of the crawl (and log warnings on failure to send message) 2014-11-06 19:26:32 -08:00
Noah Levitt 23fa29d673 fix log messages to say "kafka" 2014-11-05 14:03:54 -08:00
Noah Levitt 77e2a47c4c AITFIVE-203 kafka crawl log feed 2014-11-05 12:59:10 -08:00
Noah Levitt 4d0eff23d2 Merge pull request #107 from vonrosen/ari-4102 
Handle case where form does not have an action defined.
 2014-11-04 15:55:50 -08:00
Hunter Stern 9a41b31264 Handle case where form does not have an action defined. 2014-11-04 11:22:38 -08:00
vonrosen f5cf30ec3e Merge pull request #102 from nlevitt/amqp-receiver-deadlock 
avoid deadlock in AMQPUrlReceiver hopefully
 2014-11-03 08:11:00 -08:00
Noah Levitt 716e0cc7ef use "fair" ReentrantLock -- I stepped through heritrix that appeared to be deadlocked, found that the thread trying to shut down was being starved of the lock by another thread (the StarterRestarter) locking and unlocking it repeatedly, which can apparently happen because the lock was "unfair" 2014-10-27 17:34:41 -07:00
jkafader edcc7cef21 Merge pull request #106 from nlevitt/aitfive-39 
seriously, fix extraInfo handling in AMQPCrawlLogFeed
 2014-10-26 19:20:38 -07:00
Noah Levitt 6f21ede979 seriously, fix extraInfo handling in AMQPCrawlLogFeed 2014-10-26 19:17:23 -07:00
jkafader 23a07d5cef Merge pull request #105 from nlevitt/aitfive-39 
fix extraInfo handling in AMQPCrawlLogFeed
 2014-10-26 18:29:21 -07:00
Noah Levitt 7e4b1717d1 fix extraInfo handling in AMQPCrawlLogFeed 2014-10-26 18:24:47 -07:00
jkafader f25c171a79 Merge pull request #104 from nlevitt/aitfive-39 
change field names to match new druid config
 2014-10-26 17:11:57 -07:00
Noah Levitt 13a9d3a1be change field names to match new druid config 2014-10-22 15:28:40 -07:00
Noah Levitt 0712cb381c Merge pull request #101 from vonrosen/ari-4086 
Remove forcefetch for AMQP received urls so they don't get crawled twice...
 2014-10-19 17:38:35 -07:00
Noah Levitt d154cfbf36 avoid deadlock in AMQPUrlReceiver hopefully 2014-10-17 15:27:08 -07:00
Hunter Stern 8c318e53f6 Remove forcefetch for AMQP received urls so they don't get crawled twice. 2014-10-16 22:02:43 +00:00
vonrosen 5909042b84 Merge pull request #99 from nlevitt/aitfive-39 
AMQPCrawlLogFeed, DecideRuleSequenceWithAMQPFeed, DecideRuleSequence.logExtraInfo
 2014-10-16 17:24:37 +00:00
Noah Levitt a27dbd182d change DecideRuleSequence extraInfo field name 'source' to 'seed' (for reason external to heritrix) 2014-10-14 15:40:07 -07:00
Noah Levitt 79aadf919b rename AMQPCrawlLogFeed fields to match James's names in druid (looks like there's no way to translate) 2014-10-14 14:00:55 -07:00
Noah Levitt 5f7dc49599 Merge pull request #95 from kngenie/patch-beanbrowser-failure 
FIX corner-case of bean browser failing due to an exception from hashCode()
 2014-10-03 19:21:27 -07:00
Noah Levitt 0f1760fbfe Merge pull request #97 from kris-sigur/HER-2074 
Fix for HER-2074
 2014-10-03 18:48:42 -07:00
Kristinn Sigurðsson 0b16ff0fad Fix for HER-2074 
https://webarchive.jira.com/browse/HER-2074
 2014-10-03 11:30:38 +00:00
Kristinn Sigurðsson 86ebed04cc Merge pull request #96 from nlevitt/her-2070 
new cookie store system to address HER-2070 "cookie monster" bug
 2014-10-03 09:33:04 +00:00
Noah Levitt 5cb5f05437 add field "extra info" to DecideRuleSequence log, which is a json object with entries for host, via, source and hop path 2014-10-02 02:34:32 -07:00
Noah Levitt 743d76c966 set <encoding>UTF-8</encoding> in maven-compiler-plugin; project.build.sourceEncoding seems not to be enough, see http://stackoverflow.com/questions/8978013/error-unmappable-character-for-encoding-utf8-during-maven-compilation 2014-10-02 01:48:13 -07:00
Noah Levitt 78e6b8b6ab replace accidental U+2212 MINUS SIGN with normal ascii 0x2d '-' to avoid compilation error "unmappable character for encoding ASCII" 2014-10-01 21:31:14 -07:00
Noah Levitt e5e5cd87e4 configure maven-compiler-plugin in root pom to avoid redundancy, inconsistency 2014-10-01 13:38:39 -07:00
Noah Levitt 060054e6b3 avoid java 7 api 2014-10-01 13:38:14 -07:00
Noah Levitt 6faa1b8c1c fix misleading variable name 2014-09-30 11:43:09 -07:00
Noah Levitt 8d98882eb7 an optimization suggested by Gordon: organize cookies by host (domain), rather than top private domain, and supply a composite list of cookies from the host and each parent domain to the http client library 2014-09-29 18:32:31 -07:00
Noah Levitt 5fd120dffd some javadocs and other comments, some minor refactoring, change delimiter to ';' for the sortable key 2014-09-29 17:51:24 -07:00
Noah Levitt 87b408cc23 rename limited cookie store facade class 2014-09-28 14:35:22 -07:00
Noah Levitt 610a1e1e97 cookie store is called BdbCookieStore again 2014-09-28 14:32:34 -07:00
Noah Levitt 54eb1c950f rudimentary test of cookie at url with ip address as host 2014-09-28 14:30:01 -07:00
Noah Levitt 5c21ad1762 AbstractCookieStore.getCookieFor(CrawlURI) computes top private domain, so FetchHTTP doesn't have to know anything about that 2014-09-28 14:29:04 -07:00
Noah Levitt 4aa85af1b0 a few tests exercising some basic domain/subdomain cookie behaviors 2014-09-28 14:15:27 -07:00
Noah Levitt a37a8de4a2 improved implementation not using TreeSet, and some initial work on integration testing of cookie handling 2014-09-28 11:59:14 -07:00
Noah Levitt 6588641017 restore saving and loading, checkpoints, testing 2014-09-27 13:55:07 -07:00
Noah Levitt 111294e887 oops, commit classes for initial bdb implementation 2014-09-27 12:22:50 -07:00
Noah Levitt 47b7caad29 initial bdb implementation; facade by topmost assigned domain, since cookies can apply to subdomains 2014-09-26 20:56:45 -07:00
Noah Levitt 886a0f2136 (forgot to add SimpleHostCookieSetStore) initial proof of concept fix for HER-2070 "cookie monster" bug 2014-09-26 20:05:37 -07:00
Noah Levitt f769e0636d initial proof of concept fix for HER-2070 "cookie monster" bug 2014-09-26 20:01:52 -07:00
Kenji Nagahashi 29379c7e19 Merge pull request #84 from nlevitt/iipc-webarchive-commons 
switch to mainline iipc webarchive-commons latest release (1.1.4)
 2014-09-11 17:19:08 -07:00
Kenji Nagahashi 6ada6801de Merge pull request #94 from nlevitt/ari-3724 
do not require "+" (plus sign) before @OPERATOR_CONTACT_URL@ in user-age...
 2014-09-11 17:17:55 -07:00