1660 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Kristinn Sigurðsson	3ffe5b8b4f	Stop automatically writing WARC-Payload-Digest for revisit records. ... Refer that functionality to the relevant revisit profile classes.	2015-03-11 13:52:31 +00:00
vonrosen	b5da93d04b	Merge pull request #117 from nlevitt/ari-4222 ... avoid java.lang.NullPointerException at org.archive.modules.writer.Write...	2015-02-26 10:04:34 -08:00
Noah Levitt	a0c3fcdae0	avoid java.lang.NullPointerException at org.archive.modules.writer.WriterPoolProcessor.copyForwardWriteTagIfDupe(WriterPoolProcessor.java:504); this exception has not always been possible, but is possible now because IdenticalDigestDecideRule.hasIdenticalDigest(curi) can return true even if there is no curi.getFetchHistory(), since uri-agnostic dedup doesn't use fetchHistory	2015-02-23 14:09:25 -08:00
Adam Miller	1b6d78bf7a	Merge pull request #116 from nlevitt/log4j-breaks-tests ... make sure log4j is configured when running unit tests, to avoid log4j er...	2015-02-23 11:59:50 -08:00
Noah Levitt	8009275075	make sure log4j is configured when running unit tests, to avoid log4j error messages polluting captured output and causing tests to fail, like so: https://builds.archive.org/job/Heritrix-3/org.archive.heritrix$heritrix-commons/587/testReport/org.archive.io.arc/ARCWriterTest/testLengthTooLongCompressed/	2015-02-23 11:36:29 -08:00
vonrosen	0a418a8425	Merge pull request #114 from nlevitt/webarchive-commons-issue-38 ... remove RecordingOutputStreamTest.java (moving to webarchive-commons)	2015-02-12 15:33:22 -08:00
Noah Levitt	1bd8b141aa	bump webarchive-commons dependency to 1.1.5 to pick up https://github.com/iipc/webarchive-commons/pull/39	2015-02-11 18:53:34 -08:00
Noah Levitt	789bd8ac7c	Merge pull request #115 from kris-sigur/report-encoding ... Set character set to UTF-8 when passing through files.	2015-02-10 23:54:21 -08:00
Kristinn Sigurðsson	4ea7efdfc8	Set character set to UTF-8 when passing through files.	2015-02-10 16:07:37 +00:00
Noah Levitt	2078a9f0e2	remove TmpDirTestCase.java (moving to webarchive-commons)	2015-01-06 16:08:18 -08:00
Noah Levitt	64e3b1b02a	remove RecordingOutputStreamTest.java (moving to webarchive-commons)	2015-01-06 16:01:16 -08:00
vonrosen	2c9e82a261	Merge pull request #112 from nlevitt/amqp-receiver-deadlock ... Amqp receiver deadlock	2014-11-21 10:15:53 -08:00
vonrosen	3609abe38c	Merge pull request #108 from nlevitt/aitfive-203 ... kafka crawl log feed	2014-11-14 10:29:41 -08:00
Noah Levitt	54555e8566	Merge pull request #103 from adam-miller/SeedsReportRedirectFix ... CandidatesProcessor.java	2014-11-11 18:04:51 -08:00
Noah Levitt	40bc02c03f	Merge pull request #110 from kris-sigur/httpclient-4.3.6 ... Upgrade to HttpClient 4.3.6	2014-11-11 17:35:26 -08:00
Kristinn Sigurðsson	0212a554e6	Upgrade to httpclient 4.3.6	2014-11-11 14:40:33 +00:00
Noah Levitt	7be1fc606b	make sure the kafka producer thread is not in the ToePool thread group, to stop it from being interrupted at the end of the crawl	2014-11-10 16:45:08 -08:00
Noah Levitt	743ab3c9c0	Revert "Revert "switch to org.apache.kafka.clients.producer.KafkaProducer to see if we can avoid the InterruptedException at the end of the crawl (and log warnings on failure to send message)"" ... This reverts commit a0f70d0865.	2014-11-10 16:44:37 -08:00
Noah Levitt	a0f70d0865	Revert "switch to org.apache.kafka.clients.producer.KafkaProducer to see if we can avoid the InterruptedException at the end of the crawl (and log warnings on failure to send message)" ... This reverts commit d2f70e5c97. Conflicts: contrib/src/main/java/org/archive/modules/postprocessor/KafkaCrawlLogFeed.java	2014-11-10 13:55:13 -08:00
Noah Levitt	d96b1f8eca	Merge pull request #100 from vonrosen/ari-4061 ... Allow discovery of urls in content attribute of meta tags.	2014-11-08 00:05:00 -08:00
Noah Levitt	fa0a28f78b	Merge branch 'aitfive-203' of github.com:nlevitt/heritrix3 into aitfive-203	2014-11-07 15:59:24 -08:00
Noah Levitt	192b3f9b4b	for kafka/amqp crawl log feeds, remove now redundant contentSize from extraInfo	2014-11-07 15:59:14 -08:00
Noah Levitt	4527d5af29	Merge branch 'extra-info-size' into aitfive-203	2014-11-07 15:58:08 -08:00
Noah Levitt	7312ad597e	so that it can appear in the crawl log, add contentSize to CrawlURI extraInfo, for http transactions (the only case in heritrix currently where contentLength and contentSize differ)	2014-11-07 13:21:42 -08:00
Noah Levitt	e2be3bbf9c	new style kafka configuration properties	2014-11-06 19:53:41 -08:00
Noah Levitt	82bef2a337	only release the lock if we actually own it (avoid java.lang.IllegalMonitorStateException at java.util.concurrent.locks.ReentrantLock.tryRelease(ReentrantLock.java:155))	2014-11-06 19:33:35 -08:00
Noah Levitt	a37b879a4a	logging improvement	2014-11-06 19:28:19 -08:00
Noah Levitt	d2f70e5c97	switch to org.apache.kafka.clients.producer.KafkaProducer to see if we can avoid the InterruptedException at the end of the crawl (and log warnings on failure to send message)	2014-11-06 19:26:32 -08:00
Noah Levitt	23fa29d673	fix log messages to say "kafka"	2014-11-05 14:03:54 -08:00
Noah Levitt	77e2a47c4c	AITFIVE-203 kafka crawl log feed	2014-11-05 12:59:10 -08:00
Noah Levitt	4d0eff23d2	Merge pull request #107 from vonrosen/ari-4102 ... Handle case where form does not have an action defined.	2014-11-04 15:55:50 -08:00
Hunter Stern	9a41b31264	Handle case where form does not have an action defined.	2014-11-04 11:22:38 -08:00
vonrosen	f5cf30ec3e	Merge pull request #102 from nlevitt/amqp-receiver-deadlock ... avoid deadlock in AMQPUrlReceiver hopefully	2014-11-03 08:11:00 -08:00
Noah Levitt	716e0cc7ef	use "fair" ReentrantLock -- I stepped through heritrix that appeared to be deadlocked, found that the thread trying to shut down was being starved of the lock by another thread (the StarterRestarter) locking and unlocking it repeatedly, which can apparently happen because the lock was "unfair"	2014-10-27 17:34:41 -07:00
jkafader	edcc7cef21	Merge pull request #106 from nlevitt/aitfive-39 ... seriously, fix extraInfo handling in AMQPCrawlLogFeed	2014-10-26 19:20:38 -07:00
Noah Levitt	6f21ede979	seriously, fix extraInfo handling in AMQPCrawlLogFeed	2014-10-26 19:17:23 -07:00
jkafader	23a07d5cef	Merge pull request #105 from nlevitt/aitfive-39 ... fix extraInfo handling in AMQPCrawlLogFeed	2014-10-26 18:29:21 -07:00
Noah Levitt	7e4b1717d1	fix extraInfo handling in AMQPCrawlLogFeed	2014-10-26 18:24:47 -07:00
jkafader	f25c171a79	Merge pull request #104 from nlevitt/aitfive-39 ... change field names to match new druid config	2014-10-26 17:11:57 -07:00
Noah Levitt	13a9d3a1be	change field names to match new druid config	2014-10-22 15:28:40 -07:00
Adam Miller	fb23b40e74	CandidatesProcessor.java ... Keep CrawlURI outlinks intact for use in SeedRecord reporting.	2014-10-22 13:18:25 -07:00
Noah Levitt	0712cb381c	Merge pull request #101 from vonrosen/ari-4086 ... Remove forcefetch for AMQP received urls so they don't get crawled twice...	2014-10-19 17:38:35 -07:00
Noah Levitt	d154cfbf36	avoid deadlock in AMQPUrlReceiver hopefully	2014-10-17 15:27:08 -07:00
Hunter Stern	8c318e53f6	Remove forcefetch for AMQP received urls so they don't get crawled twice.	2014-10-16 22:02:43 +00:00
Hunter Stern	78e98365d6	Remove unused variable.	2014-10-16 17:51:08 +00:00
vonrosen	5909042b84	Merge pull request #99 from nlevitt/aitfive-39 ... AMQPCrawlLogFeed, DecideRuleSequenceWithAMQPFeed, DecideRuleSequence.logExtraInfo	2014-10-16 17:24:37 +00:00
Noah Levitt	a27dbd182d	change DecideRuleSequence extraInfo field name 'source' to 'seed' (for reason external to heritrix)	2014-10-14 15:40:07 -07:00
Noah Levitt	79aadf919b	rename AMQPCrawlLogFeed fields to match James's names in druid (looks like there's no way to translate)	2014-10-14 14:00:55 -07:00
Hunter Stern	3a82b37e52	Ensure Jerico extractor html doesn't call new metadata test method.	2014-10-10 23:33:22 +00:00
Hunter Stern	3347de3d39	Make meta tag content url extraction rules more generate so as to apply to all meta tags 'content' attributes, not just specific og:video.	2014-10-09 23:42:24 +00:00