lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-17 11:15:54 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,650 Commits 38 Branches 33 Tags
4ea7efdfc8a3560cb71f0f5f96ab70b9d2cf006b
Commit Graph
1650 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Kristinn Sigurðsson 4ea7efdfc8 Set character set to UTF-8 when passing through files. 2015-02-10 16:07:37 +00:00
vonrosen 2c9e82a261 Merge pull request #112 from nlevitt/amqp-receiver-deadlock 
Amqp receiver deadlock
 2014-11-21 10:15:53 -08:00
vonrosen 3609abe38c Merge pull request #108 from nlevitt/aitfive-203 
kafka crawl log feed
 2014-11-14 10:29:41 -08:00
Noah Levitt 54555e8566 Merge pull request #103 from adam-miller/SeedsReportRedirectFix 
CandidatesProcessor.java
 2014-11-11 18:04:51 -08:00
Noah Levitt 40bc02c03f Merge pull request #110 from kris-sigur/httpclient-4.3.6 
Upgrade to HttpClient 4.3.6
 2014-11-11 17:35:26 -08:00
Kristinn Sigurðsson 0212a554e6 Upgrade to httpclient 4.3.6 2014-11-11 14:40:33 +00:00
Noah Levitt 7be1fc606b make sure the kafka producer thread is not in the ToePool thread group, to stop it from being interrupted at the end of the crawl 2014-11-10 16:45:08 -08:00
Noah Levitt 743ab3c9c0 Revert "Revert "switch to org.apache.kafka.clients.producer.KafkaProducer to see if we can avoid the InterruptedException at the end of the crawl (and log warnings on failure to send message)"" 
This reverts commit a0f70d0865.
 2014-11-10 16:44:37 -08:00
Noah Levitt a0f70d0865 Revert "switch to org.apache.kafka.clients.producer.KafkaProducer to see if we can avoid the InterruptedException at the end of the crawl (and log warnings on failure to send message)" 
This reverts commit d2f70e5c97.

Conflicts:
	contrib/src/main/java/org/archive/modules/postprocessor/KafkaCrawlLogFeed.java
 2014-11-10 13:55:13 -08:00
Noah Levitt d96b1f8eca Merge pull request #100 from vonrosen/ari-4061 
Allow discovery of urls in content attribute of meta tags.
 2014-11-08 00:05:00 -08:00
Noah Levitt fa0a28f78b Merge branch 'aitfive-203' of github.com:nlevitt/heritrix3 into aitfive-203 2014-11-07 15:59:24 -08:00
Noah Levitt 192b3f9b4b for kafka/amqp crawl log feeds, remove now redundant contentSize from extraInfo 2014-11-07 15:59:14 -08:00
Noah Levitt 4527d5af29 Merge branch 'extra-info-size' into aitfive-203 2014-11-07 15:58:08 -08:00
Noah Levitt 7312ad597e so that it can appear in the crawl log, add contentSize to CrawlURI extraInfo, for http transactions (the only case in heritrix currently where contentLength and contentSize differ) 2014-11-07 13:21:42 -08:00
Noah Levitt e2be3bbf9c new style kafka configuration properties 2014-11-06 19:53:41 -08:00
Noah Levitt 82bef2a337 only release the lock if we actually own it (avoid java.lang.IllegalMonitorStateException at java.util.concurrent.locks.ReentrantLock.tryRelease(ReentrantLock.java:155)) 2014-11-06 19:33:35 -08:00
Noah Levitt a37b879a4a logging improvement 2014-11-06 19:28:19 -08:00
Noah Levitt d2f70e5c97 switch to org.apache.kafka.clients.producer.KafkaProducer to see if we can avoid the InterruptedException at the end of the crawl (and log warnings on failure to send message) 2014-11-06 19:26:32 -08:00
Noah Levitt 23fa29d673 fix log messages to say "kafka" 2014-11-05 14:03:54 -08:00
Noah Levitt 77e2a47c4c AITFIVE-203 kafka crawl log feed 2014-11-05 12:59:10 -08:00
Noah Levitt 4d0eff23d2 Merge pull request #107 from vonrosen/ari-4102 
Handle case where form does not have an action defined.
 2014-11-04 15:55:50 -08:00
Hunter Stern 9a41b31264 Handle case where form does not have an action defined. 2014-11-04 11:22:38 -08:00
vonrosen f5cf30ec3e Merge pull request #102 from nlevitt/amqp-receiver-deadlock 
avoid deadlock in AMQPUrlReceiver hopefully
 2014-11-03 08:11:00 -08:00
Noah Levitt 716e0cc7ef use "fair" ReentrantLock -- I stepped through heritrix that appeared to be deadlocked, found that the thread trying to shut down was being starved of the lock by another thread (the StarterRestarter) locking and unlocking it repeatedly, which can apparently happen because the lock was "unfair" 2014-10-27 17:34:41 -07:00
jkafader edcc7cef21 Merge pull request #106 from nlevitt/aitfive-39 
seriously, fix extraInfo handling in AMQPCrawlLogFeed
 2014-10-26 19:20:38 -07:00
Noah Levitt 6f21ede979 seriously, fix extraInfo handling in AMQPCrawlLogFeed 2014-10-26 19:17:23 -07:00
jkafader 23a07d5cef Merge pull request #105 from nlevitt/aitfive-39 
fix extraInfo handling in AMQPCrawlLogFeed
 2014-10-26 18:29:21 -07:00
Noah Levitt 7e4b1717d1 fix extraInfo handling in AMQPCrawlLogFeed 2014-10-26 18:24:47 -07:00
jkafader f25c171a79 Merge pull request #104 from nlevitt/aitfive-39 
change field names to match new druid config
 2014-10-26 17:11:57 -07:00
Noah Levitt 13a9d3a1be change field names to match new druid config 2014-10-22 15:28:40 -07:00
Adam Miller fb23b40e74 CandidatesProcessor.java 
Keep CrawlURI outlinks intact for use in SeedRecord reporting.
 2014-10-22 13:18:25 -07:00
Noah Levitt 0712cb381c Merge pull request #101 from vonrosen/ari-4086 
Remove forcefetch for AMQP received urls so they don't get crawled twice...
 2014-10-19 17:38:35 -07:00
Noah Levitt d154cfbf36 avoid deadlock in AMQPUrlReceiver hopefully 2014-10-17 15:27:08 -07:00
Hunter Stern 8c318e53f6 Remove forcefetch for AMQP received urls so they don't get crawled twice. 2014-10-16 22:02:43 +00:00
Hunter Stern 78e98365d6 Remove unused variable. 2014-10-16 17:51:08 +00:00
vonrosen 5909042b84 Merge pull request #99 from nlevitt/aitfive-39 
AMQPCrawlLogFeed, DecideRuleSequenceWithAMQPFeed, DecideRuleSequence.logExtraInfo
 2014-10-16 17:24:37 +00:00
Noah Levitt a27dbd182d change DecideRuleSequence extraInfo field name 'source' to 'seed' (for reason external to heritrix) 2014-10-14 15:40:07 -07:00
Noah Levitt 79aadf919b rename AMQPCrawlLogFeed fields to match James's names in druid (looks like there's no way to translate) 2014-10-14 14:00:55 -07:00
Hunter Stern 3a82b37e52 Ensure Jerico extractor html doesn't call new metadata test method. 2014-10-10 23:33:22 +00:00
Hunter Stern 3347de3d39 Make meta tag content url extraction rules more generate so as to apply to all meta tags 'content' attributes, not just specific og:video. 2014-10-09 23:42:24 +00:00
Hunter Stern ff5c59c555 Allow extraction of content url in opengraph video meta tag - og:video 2014-10-09 17:57:05 +00:00
Noah Levitt 5f7dc49599 Merge pull request #95 from kngenie/patch-beanbrowser-failure 
FIX corner-case of bean browser failing due to an exception from hashCode()
 2014-10-03 19:21:27 -07:00
Noah Levitt 0f1760fbfe Merge pull request #97 from kris-sigur/HER-2074 
Fix for HER-2074
 2014-10-03 18:48:42 -07:00
Kristinn Sigurðsson 0b16ff0fad Fix for HER-2074 
https://webarchive.jira.com/browse/HER-2074
 2014-10-03 11:30:38 +00:00
Kristinn Sigurðsson 86ebed04cc Merge pull request #96 from nlevitt/her-2070 
new cookie store system to address HER-2070 "cookie monster" bug
 2014-10-03 09:33:04 +00:00
Noah Levitt 5cb5f05437 add field "extra info" to DecideRuleSequence log, which is a json object with entries for host, via, source and hop path 2014-10-02 02:34:32 -07:00
Noah Levitt 743d76c966 set <encoding>UTF-8</encoding> in maven-compiler-plugin; project.build.sourceEncoding seems not to be enough, see http://stackoverflow.com/questions/8978013/error-unmappable-character-for-encoding-utf8-during-maven-compilation 2014-10-02 01:48:13 -07:00
Noah Levitt 78e6b8b6ab replace accidental U+2212 MINUS SIGN with normal ascii 0x2d '-' to avoid compilation error "unmappable character for encoding ASCII" 2014-10-01 21:31:14 -07:00
Noah Levitt e5e5cd87e4 configure maven-compiler-plugin in root pom to avoid redundancy, inconsistency 2014-10-01 13:38:39 -07:00
Noah Levitt 060054e6b3 avoid java 7 api 2014-10-01 13:38:14 -07:00