Kristinn Sigurðsson
dff219e884
Harmonizing whitespace use
2015-03-13 14:32:23 +00:00
Kristinn Sigurðsson
843f6468c4
Use digest from previous capture when encountering 304s.
...
The previous capture may also have been a 304 but there must be an
original 200 capture at the start. This assumption may fail when using
an index generated by a version of Heritrix without this fix. But even
then you should be no worse off than before.
2015-03-13 14:30:39 +00:00
Kristinn Sigurðsson
3ffe5b8b4f
Stop automatically writing WARC-Payload-Digest for revisit records.
...
Refer that functionality to the relevant revisit profile classes.
2015-03-11 13:52:31 +00:00
vonrosen
b5da93d04b
Merge pull request #117 from nlevitt/ari-4222
...
avoid java.lang.NullPointerException at org.archive.modules.writer.Write...
2015-02-26 10:04:34 -08:00
Noah Levitt
a0c3fcdae0
avoid java.lang.NullPointerException at org.archive.modules.writer.WriterPoolProcessor.copyForwardWriteTagIfDupe(WriterPoolProcessor.java:504); this exception has not always been possible, but is possible now because IdenticalDigestDecideRule.hasIdenticalDigest(curi) can return true even if there is no curi.getFetchHistory(), since uri-agnostic dedup doesn't use fetchHistory
2015-02-23 14:09:25 -08:00
Adam Miller
1b6d78bf7a
Merge pull request #116 from nlevitt/log4j-breaks-tests
...
make sure log4j is configured when running unit tests, to avoid log4j er...
2015-02-23 11:59:50 -08:00
Noah Levitt
8009275075
make sure log4j is configured when running unit tests, to avoid log4j error messages polluting captured output and causing tests to fail, like so: https://builds.archive.org/job/Heritrix-3/org.archive.heritrix$heritrix-commons/587/testReport/org.archive.io.arc/ARCWriterTest/testLengthTooLongCompressed/
2015-02-23 11:36:29 -08:00
vonrosen
0a418a8425
Merge pull request #114 from nlevitt/webarchive-commons-issue-38
...
remove RecordingOutputStreamTest.java (moving to webarchive-commons)
2015-02-12 15:33:22 -08:00
Noah Levitt
1bd8b141aa
bump webarchive-commons dependency to 1.1.5 to pick up https://github.com/iipc/webarchive-commons/pull/39
2015-02-11 18:53:34 -08:00
Noah Levitt
789bd8ac7c
Merge pull request #115 from kris-sigur/report-encoding
...
Set character set to UTF-8 when passing through files.
2015-02-10 23:54:21 -08:00
Kristinn Sigurðsson
4ea7efdfc8
Set character set to UTF-8 when passing through files.
2015-02-10 16:07:37 +00:00
Noah Levitt
2078a9f0e2
remove TmpDirTestCase.java (moving to webarchive-commons)
2015-01-06 16:08:18 -08:00
Noah Levitt
64e3b1b02a
remove RecordingOutputStreamTest.java (moving to webarchive-commons)
2015-01-06 16:01:16 -08:00
vonrosen
2c9e82a261
Merge pull request #112 from nlevitt/amqp-receiver-deadlock
...
Amqp receiver deadlock
2014-11-21 10:15:53 -08:00
vonrosen
3609abe38c
Merge pull request #108 from nlevitt/aitfive-203
...
kafka crawl log feed
2014-11-14 10:29:41 -08:00
Noah Levitt
54555e8566
Merge pull request #103 from adam-miller/SeedsReportRedirectFix
...
CandidatesProcessor.java
2014-11-11 18:04:51 -08:00
Noah Levitt
40bc02c03f
Merge pull request #110 from kris-sigur/httpclient-4.3.6
...
Upgrade to HttpClient 4.3.6
2014-11-11 17:35:26 -08:00
Kristinn Sigurðsson
0212a554e6
Upgrade to httpclient 4.3.6
2014-11-11 14:40:33 +00:00
Noah Levitt
7be1fc606b
make sure the kafka producer thread is not in the ToePool thread group, to stop it from being interrupted at the end of the crawl
2014-11-10 16:45:08 -08:00
Noah Levitt
743ab3c9c0
Revert "Revert "switch to org.apache.kafka.clients.producer.KafkaProducer to see if we can avoid the InterruptedException at the end of the crawl (and log warnings on failure to send message)""
...
This reverts commit a0f70d0865
2014-11-10 16:44:37 -08:00
Noah Levitt
a0f70d0865
Revert "switch to org.apache.kafka.clients.producer.KafkaProducer to see if we can avoid the InterruptedException at the end of the crawl (and log warnings on failure to send message)"
...
This reverts commit d2f70e5c97
2014-11-10 13:55:13 -08:00
Noah Levitt
d96b1f8eca
Merge pull request #100 from vonrosen/ari-4061
...
Allow discovery of urls in content attribute of meta tags.
2014-11-08 00:05:00 -08:00
Noah Levitt
fa0a28f78b
Merge branch 'aitfive-203' of github.com:nlevitt/heritrix3 into aitfive-203
2014-11-07 15:59:24 -08:00
Noah Levitt
192b3f9b4b
for kafka/amqp crawl log feeds, remove now redundant contentSize from extraInfo
2014-11-07 15:59:14 -08:00
Noah Levitt
4527d5af29
Merge branch 'extra-info-size' into aitfive-203
2014-11-07 15:58:08 -08:00
Noah Levitt
7312ad597e
so that it can appear in the crawl log, add contentSize to CrawlURI extraInfo, for http transactions (the only case in heritrix currently where contentLength and contentSize differ)
2014-11-07 13:21:42 -08:00
Noah Levitt
e2be3bbf9c
new style kafka configuration properties
2014-11-06 19:53:41 -08:00
Noah Levitt
82bef2a337
only release the lock if we actually own it (avoid java.lang.IllegalMonitorStateException at java.util.concurrent.locks.ReentrantLock.tryRelease(ReentrantLock.java:155))
2014-11-06 19:33:35 -08:00
Noah Levitt
a37b879a4a
logging improvement
2014-11-06 19:28:19 -08:00
Noah Levitt
d2f70e5c97
switch to org.apache.kafka.clients.producer.KafkaProducer to see if we can avoid the InterruptedException at the end of the crawl (and log warnings on failure to send message)
2014-11-06 19:26:32 -08:00
Noah Levitt
23fa29d673
fix log messages to say "kafka"
2014-11-05 14:03:54 -08:00
Noah Levitt
77e2a47c4c
AITFIVE-203 kafka crawl log feed
2014-11-05 12:59:10 -08:00
Noah Levitt
4d0eff23d2
Merge pull request #107 from vonrosen/ari-4102
...
Handle case where form does not have an action defined.
2014-11-04 15:55:50 -08:00
Hunter Stern
9a41b31264
Handle case where form does not have an action defined.
2014-11-04 11:22:38 -08:00
vonrosen
f5cf30ec3e
Merge pull request #102 from nlevitt/amqp-receiver-deadlock
...
avoid deadlock in AMQPUrlReceiver hopefully
2014-11-03 08:11:00 -08:00
Noah Levitt
716e0cc7ef
use "fair" ReentrantLock -- I stepped through heritrix that appeared to be deadlocked, found that the thread trying to shut down was being starved of the lock by another thread (the StarterRestarter) locking and unlocking it repeatedly, which can apparently happen because the lock was "unfair"
2014-10-27 17:34:41 -07:00
jkafader
edcc7cef21
Merge pull request #106 from nlevitt/aitfive-39
...
seriously, fix extraInfo handling in AMQPCrawlLogFeed
2014-10-26 19:20:38 -07:00
Noah Levitt
6f21ede979
seriously, fix extraInfo handling in AMQPCrawlLogFeed
2014-10-26 19:17:23 -07:00
jkafader
23a07d5cef
Merge pull request #105 from nlevitt/aitfive-39
...
fix extraInfo handling in AMQPCrawlLogFeed
2014-10-26 18:29:21 -07:00
Noah Levitt
7e4b1717d1
fix extraInfo handling in AMQPCrawlLogFeed
2014-10-26 18:24:47 -07:00
jkafader
f25c171a79
Merge pull request #104 from nlevitt/aitfive-39
...
change field names to match new druid config
2014-10-26 17:11:57 -07:00
Noah Levitt
13a9d3a1be
change field names to match new druid config
2014-10-22 15:28:40 -07:00
Adam Miller
fb23b40e74
CandidatesProcessor.java
...
Keep CrawlURI outlinks intact for use in SeedRecord reporting.
2014-10-22 13:18:25 -07:00
Noah Levitt
0712cb381c
Merge pull request #101 from vonrosen/ari-4086
...
Remove forcefetch for AMQP received urls so they don't get crawled twice...
2014-10-19 17:38:35 -07:00
Noah Levitt
d154cfbf36
avoid deadlock in AMQPUrlReceiver hopefully
2014-10-17 15:27:08 -07:00
Hunter Stern
8c318e53f6
Remove forcefetch for AMQP received urls so they don't get crawled twice.
2014-10-16 22:02:43 +00:00
Hunter Stern
78e98365d6
Remove unused variable.
2014-10-16 17:51:08 +00:00
vonrosen
5909042b84
Merge pull request #99 from nlevitt/aitfive-39
...
AMQPCrawlLogFeed, DecideRuleSequenceWithAMQPFeed, DecideRuleSequence.logExtraInfo
2014-10-16 17:24:37 +00:00
Noah Levitt
a27dbd182d
change DecideRuleSequence extraInfo field name 'source' to 'seed' (for reason external to heritrix)
2014-10-14 15:40:07 -07:00
Noah Levitt
79aadf919b
rename AMQPCrawlLogFeed fields to match James's names in druid (looks like there's no way to translate)
2014-10-14 14:00:55 -07:00
Kristinn Sigurðsson