Andrew Jackson
e883977de6
Downgrade Throwable to Exception.
2015-09-23 23:14:11 +01:00
Andrew Jackson
d1ea01c75d
Updated with additional optional fields and separated isConsuming from isRunning.
2015-09-19 21:39:19 +01:00
Andrew Jackson
cd3fb8d717
Configurable and 3.3 compatible AMQP receiver for #1
2015-09-18 15:48:09 +01:00
Noah Levitt
490a257cfe
Merge pull request #126 from caofangkun/her-001
...
'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing
2015-08-25 10:40:59 -07:00
caofangkun
6cf0d4a0c5
'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing
2015-08-24 17:56:56 +08:00
Noah Levitt
f5a4ecbb03
Merge pull request #125 from ldko/AMQP-declarations-fix
...
Amqp declarations fix
2015-08-03 14:26:38 -07:00
Lauren Ko
c94f515956
Sets the default clientId to the same as on Receiver side
2015-07-31 15:45:49 -05:00
Lauren Ko
592c70fa3d
declare AMQP exchange in AMQPUrlReceiver to prevent exception when exchange not yet created
2015-07-30 17:34:48 -05:00
Hunter
7afe03540a
Merge pull request #120 from nlevitt/extractorjs-twodots
...
for ARI-4267 accept possible uris with two dots in the filename part if ...
2015-05-04 15:45:16 -07:00
Hunter
eb78de6963
Merge pull request #121 from nlevitt/knowledgable-extractorjs
...
new contrib module KnowledgableExtractorJS, a subclass of ExtractorJS th...
2015-05-04 15:41:21 -07:00
Noah Levitt
fea49f0e30
expand stubbed javadoc comment
2015-05-04 15:16:11 -07:00
Noah Levitt
0f43a3dae6
add license header
2015-05-04 15:08:52 -07:00
Noah Levitt
e9fcb34508
new contrib module KnowledgableExtractorJS, a subclass of ExtractorJS that has some customized behavior for specific kinds of web pages; initially, the one special behavior it has is for drupal generated pages
2015-04-21 01:19:33 -07:00
Noah Levitt
1027f854ec
for ARI-4267 accept possible uris with two dots in the filename part if they end with a known good extension (e.g. "foo.min.js")
2015-04-20 17:08:47 -07:00
Noah Levitt
0923a7ca18
Merge pull request #119 from adam-miller/HER-2082
...
Fix for HER-2082
2015-03-26 16:30:49 -07:00
Adam Miller
458cbb51b0
Fix for HER-2082
...
https://webarchive.jira.com/browse/HER-2082
WARCWriterProcessor.java
Add last hop character and link context to outlinks in metadata records
2015-03-26 16:19:09 -07:00
Noah Levitt
bcae99daa8
Merge pull request #118 from kris-sigur/HER-2080
...
Fix for ServerNotModified WARC revisit records incorrectly record WARC-Payload-Digest
2015-03-23 11:16:36 -07:00
Kristinn Sigurðsson
f5107f60b8
Add annotation for server-not-modified duplicates.
2015-03-20 13:46:20 +00:00
Kristinn Sigurðsson
dff219e884
Harmonizing whitespace use
2015-03-13 14:32:23 +00:00
Kristinn Sigurðsson
843f6468c4
Use digest from previous capture when encountering 304s.
...
The previous capture may also have been a 304 but there must be an
original 200 capture at the start. This assumption may fail when using
an index generated by a version of Heritrix without this fix. But even
then you should be no worse off than before.
2015-03-13 14:30:39 +00:00
Kristinn Sigurðsson
3ffe5b8b4f
Stop automatically writing WARC-Payload-Digest for revisit records.
...
Refer that functionality to the relevant revisit profile classes.
2015-03-11 13:52:31 +00:00
vonrosen
b5da93d04b
Merge pull request #117 from nlevitt/ari-4222
...
avoid java.lang.NullPointerException at org.archive.modules.writer.Write...
2015-02-26 10:04:34 -08:00
Noah Levitt
a0c3fcdae0
avoid java.lang.NullPointerException at org.archive.modules.writer.WriterPoolProcessor.copyForwardWriteTagIfDupe(WriterPoolProcessor.java:504); this exception has not always been possible, but is possible now because IdenticalDigestDecideRule.hasIdenticalDigest(curi) can return true even if there is no curi.getFetchHistory(), since uri-agnostic dedup doesn't use fetchHistory
2015-02-23 14:09:25 -08:00
Adam Miller
1b6d78bf7a
Merge pull request #116 from nlevitt/log4j-breaks-tests
...
make sure log4j is configured when running unit tests, to avoid log4j er...
2015-02-23 11:59:50 -08:00
Noah Levitt
8009275075
make sure log4j is configured when running unit tests, to avoid log4j error messages polluting captured output and causing tests to fail, like so: https://builds.archive.org/job/Heritrix-3/org.archive.heritrix$heritrix-commons/587/testReport/org.archive.io.arc/ARCWriterTest/testLengthTooLongCompressed/
2015-02-23 11:36:29 -08:00
vonrosen
0a418a8425
Merge pull request #114 from nlevitt/webarchive-commons-issue-38
...
remove RecordingOutputStreamTest.java (moving to webarchive-commons)
2015-02-12 15:33:22 -08:00
Noah Levitt
1bd8b141aa
bump webarchive-commons dependency to 1.1.5 to pick up https://github.com/iipc/webarchive-commons/pull/39
2015-02-11 18:53:34 -08:00
Noah Levitt
789bd8ac7c
Merge pull request #115 from kris-sigur/report-encoding
...
Set character set to UTF-8 when passing through files.
2015-02-10 23:54:21 -08:00
Kristinn Sigurðsson
4ea7efdfc8
Set character set to UTF-8 when passing through files.
2015-02-10 16:07:37 +00:00
Noah Levitt
2078a9f0e2
remove TmpDirTestCase.java (moving to webarchive-commons)
2015-01-06 16:08:18 -08:00
Noah Levitt
64e3b1b02a
remove RecordingOutputStreamTest.java (moving to webarchive-commons)
2015-01-06 16:01:16 -08:00
vonrosen
2c9e82a261
Merge pull request #112 from nlevitt/amqp-receiver-deadlock
...
Amqp receiver deadlock
2014-11-21 10:15:53 -08:00
vonrosen
3609abe38c
Merge pull request #108 from nlevitt/aitfive-203
...
kafka crawl log feed
2014-11-14 10:29:41 -08:00
Noah Levitt
54555e8566
Merge pull request #103 from adam-miller/SeedsReportRedirectFix
...
CandidatesProcessor.java
2014-11-11 18:04:51 -08:00
Noah Levitt
40bc02c03f
Merge pull request #110 from kris-sigur/httpclient-4.3.6
...
Upgrade to HttpClient 4.3.6
2014-11-11 17:35:26 -08:00
Kristinn Sigurðsson
0212a554e6
Upgrade to httpclient 4.3.6
2014-11-11 14:40:33 +00:00
Noah Levitt
7be1fc606b
make sure the kafka producer thread is not in the ToePool thread group, to stop it from being interrupted at the end of the crawl
2014-11-10 16:45:08 -08:00
Noah Levitt
743ab3c9c0
Revert "Revert "switch to org.apache.kafka.clients.producer.KafkaProducer to see if we can avoid the InterruptedException at the end of the crawl (and log warnings on failure to send message)""
...
This reverts commit a0f70d0865
2014-11-10 16:44:37 -08:00
Noah Levitt
a0f70d0865
Revert "switch to org.apache.kafka.clients.producer.KafkaProducer to see if we can avoid the InterruptedException at the end of the crawl (and log warnings on failure to send message)"
...
This reverts commit d2f70e5c97
2014-11-10 13:55:13 -08:00
Noah Levitt
d96b1f8eca
Merge pull request #100 from vonrosen/ari-4061
...
Allow discovery of urls in content attribute of meta tags.
2014-11-08 00:05:00 -08:00
Noah Levitt
fa0a28f78b
Merge branch 'aitfive-203' of github.com:nlevitt/heritrix3 into aitfive-203
2014-11-07 15:59:24 -08:00
Noah Levitt
192b3f9b4b
for kafka/amqp crawl log feeds, remove now redundant contentSize from extraInfo
2014-11-07 15:59:14 -08:00
Noah Levitt
4527d5af29
Merge branch 'extra-info-size' into aitfive-203
2014-11-07 15:58:08 -08:00
Noah Levitt
7312ad597e
so that it can appear in the crawl log, add contentSize to CrawlURI extraInfo, for http transactions (the only case in heritrix currently where contentLength and contentSize differ)
2014-11-07 13:21:42 -08:00
Noah Levitt
e2be3bbf9c
new style kafka configuration properties
2014-11-06 19:53:41 -08:00
Noah Levitt
82bef2a337
only release the lock if we actually own it (avoid java.lang.IllegalMonitorStateException at java.util.concurrent.locks.ReentrantLock.tryRelease(ReentrantLock.java:155))
2014-11-06 19:33:35 -08:00
Noah Levitt
a37b879a4a
logging improvement
2014-11-06 19:28:19 -08:00
Noah Levitt
d2f70e5c97
switch to org.apache.kafka.clients.producer.KafkaProducer to see if we can avoid the InterruptedException at the end of the crawl (and log warnings on failure to send message)
2014-11-06 19:26:32 -08:00
Noah Levitt
23fa29d673
fix log messages to say "kafka"
2014-11-05 14:03:54 -08:00
Noah Levitt
77e2a47c4c
AITFIVE-203 kafka crawl log feed
2014-11-05 12:59:10 -08:00
Andrew Jackson