Hunter
79ebb0e03c
Merge pull request #129 from nlevitt/java8keytool
...
load keytool utility main class dynamically, trying both the old and …
2015-10-06 16:09:43 -07:00
Hunter
a9c4cb42c9
Merge pull request #130 from nlevitt/stay-alive-statistics-tracker
...
catch exceptions and log error in StatisticsTracker.run(), to make su…
2015-10-06 16:09:11 -07:00
Hunter
98d41051b8
Merge pull request #131 from nlevitt/amqp-url-receiver-changes
...
set isRunning=true so that stop() gets called to avoid leaking connec…
2015-10-06 15:52:05 -07:00
Noah Levitt
af5c6c3dde
set isRunning=true so that stop() gets called to avoid leaking connections when crawls finish
2015-10-06 15:50:47 -07:00
Noah Levitt
11d4bab1ac
catch exceptions and log error in StatisticsTracker.run(), to make sure statistics events don't stop happening
2015-10-05 11:46:35 -07:00
Noah Levitt
41dac25cb6
Merge pull request #128 from ukwa/amqp-url-receiver-changes
...
AMQPUrlReceiver changes to support RabbitMQ >= 3.3
2015-09-24 16:52:04 -07:00
Noah Levitt
b7afc627ed
fix typo
2015-09-24 15:31:12 -07:00
Noah Levitt
88c7f3c3d4
Merge pull request #124 from vonrosen/ari-4371
...
Allow realm to be set by server for basic auth.
2015-09-24 10:26:49 -07:00
Andrew Jackson
e883977de6
Downgrade Throwable to Exception.
2015-09-23 23:14:11 +01:00
Noah Levitt
9aabcb3272
add license header to new class
2015-09-23 12:07:38 -07:00
Noah Levitt
12dde2b693
load keytool utility main class dynamically, trying both the old and new (java 8) class names
2015-09-23 12:06:11 -07:00
Andrew Jackson
d1ea01c75d
Updated with additional optional fields and separated isConsuming from isRunning.
2015-09-19 21:39:19 +01:00
Andrew Jackson
cd3fb8d717
Configurable and 3.3 compatible AMQP receiver for #1
2015-09-18 15:48:09 +01:00
Noah Levitt
490a257cfe
Merge pull request #126 from caofangkun/her-001
...
'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing
2015-08-25 10:40:59 -07:00
caofangkun
6cf0d4a0c5
'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing
2015-08-24 17:56:56 +08:00
Hunter Stern
f097939a1c
Don't call setRealm so other realms that can be used.
2015-08-14 13:01:54 -07:00
Noah Levitt
f5a4ecbb03
Merge pull request #125 from ldko/AMQP-declarations-fix
...
Amqp declarations fix
2015-08-03 14:26:38 -07:00
Lauren Ko
c94f515956
Sets the default clientId to the same as on Receiver side
2015-07-31 15:45:49 -05:00
Lauren Ko
592c70fa3d
declare AMQP exchange in AMQPUrlReceiver to prevent exception when exchange not yet created
2015-07-30 17:34:48 -05:00
Hunter Stern
4e22f1f809
Allow realm to be set by server for basic auth.
2015-07-07 13:06:29 -07:00
Hunter
7afe03540a
Merge pull request #120 from nlevitt/extractorjs-twodots
...
for ARI-4267 accept possible uris with two dots in the filename part if ...
2015-05-04 15:45:16 -07:00
Hunter
eb78de6963
Merge pull request #121 from nlevitt/knowledgable-extractorjs
...
new contrib module KnowledgableExtractorJS, a subclass of ExtractorJS th...
2015-05-04 15:41:21 -07:00
Noah Levitt
fea49f0e30
expand stubbed javadoc comment
2015-05-04 15:16:11 -07:00
Noah Levitt
0f43a3dae6
add license header
2015-05-04 15:08:52 -07:00
Noah Levitt
e9fcb34508
new contrib module KnowledgableExtractorJS, a subclass of ExtractorJS that has some customized behavior for specific kinds of web pages; initially, the one special behavior it has is for drupal generated pages
2015-04-21 01:19:33 -07:00
Noah Levitt
1027f854ec
for ARI-4267 accept possible uris with two dots in the filename part if they end with a known good extension (e.g. "foo.min.js")
2015-04-20 17:08:47 -07:00
Noah Levitt
0923a7ca18
Merge pull request #119 from adam-miller/HER-2082
...
Fix for HER-2082
2015-03-26 16:30:49 -07:00
Adam Miller
458cbb51b0
Fix for HER-2082
...
https://webarchive.jira.com/browse/HER-2082
WARCWriterProcessor.java
Add last hop character and link context to outlinks in metadata records
2015-03-26 16:19:09 -07:00
Noah Levitt
bcae99daa8
Merge pull request #118 from kris-sigur/HER-2080
...
Fix for ServerNotModified WARC revisit records incorrectly record WARC-Payload-Digest
2015-03-23 11:16:36 -07:00
Kristinn Sigurðsson
f5107f60b8
Add annotation for server-not-modified duplicates.
2015-03-20 13:46:20 +00:00
Kristinn Sigurðsson
dff219e884
Harmonizing whitespace use
2015-03-13 14:32:23 +00:00
Kristinn Sigurðsson
843f6468c4
Use digest from previous capture when encountering 304s.
...
The previous capture may also have been a 304 but there must be an
original 200 capture at the start. This assumption may fail when using
an index generated by a version of Heritrix without this fix. But even
then you should be no worse off than before.
2015-03-13 14:30:39 +00:00
Kristinn Sigurðsson
3ffe5b8b4f
Stop automatically writing WARC-Payload-Digest for revisit records.
...
Refer that functionality to the relevant revisit profile classes.
2015-03-11 13:52:31 +00:00
vonrosen
b5da93d04b
Merge pull request #117 from nlevitt/ari-4222
...
avoid java.lang.NullPointerException at org.archive.modules.writer.Write...
2015-02-26 10:04:34 -08:00
Noah Levitt
a0c3fcdae0
avoid java.lang.NullPointerException at org.archive.modules.writer.WriterPoolProcessor.copyForwardWriteTagIfDupe(WriterPoolProcessor.java:504); this exception has not always been possible, but is possible now because IdenticalDigestDecideRule.hasIdenticalDigest(curi) can return true even if there is no curi.getFetchHistory(), since uri-agnostic dedup doesn't use fetchHistory
2015-02-23 14:09:25 -08:00
Adam Miller
1b6d78bf7a
Merge pull request #116 from nlevitt/log4j-breaks-tests
...
make sure log4j is configured when running unit tests, to avoid log4j er...
2015-02-23 11:59:50 -08:00
Noah Levitt
8009275075
make sure log4j is configured when running unit tests, to avoid log4j error messages polluting captured output and causing tests to fail, like so: https://builds.archive.org/job/Heritrix-3/org.archive.heritrix$heritrix-commons/587/testReport/org.archive.io.arc/ARCWriterTest/testLengthTooLongCompressed/
2015-02-23 11:36:29 -08:00
vonrosen
0a418a8425
Merge pull request #114 from nlevitt/webarchive-commons-issue-38
...
remove RecordingOutputStreamTest.java (moving to webarchive-commons)
2015-02-12 15:33:22 -08:00
Noah Levitt
1bd8b141aa
bump webarchive-commons dependency to 1.1.5 to pick up https://github.com/iipc/webarchive-commons/pull/39
2015-02-11 18:53:34 -08:00
Noah Levitt
789bd8ac7c
Merge pull request #115 from kris-sigur/report-encoding
...
Set character set to UTF-8 when passing through files.
2015-02-10 23:54:21 -08:00
Kristinn Sigurðsson
4ea7efdfc8
Set character set to UTF-8 when passing through files.
2015-02-10 16:07:37 +00:00
Noah Levitt
2078a9f0e2
remove TmpDirTestCase.java (moving to webarchive-commons)
2015-01-06 16:08:18 -08:00
Noah Levitt
64e3b1b02a
remove RecordingOutputStreamTest.java (moving to webarchive-commons)
2015-01-06 16:01:16 -08:00
vonrosen
2c9e82a261
Merge pull request #112 from nlevitt/amqp-receiver-deadlock
...
Amqp receiver deadlock
2014-11-21 10:15:53 -08:00
vonrosen
3609abe38c
Merge pull request #108 from nlevitt/aitfive-203
...
kafka crawl log feed
2014-11-14 10:29:41 -08:00
Noah Levitt
54555e8566
Merge pull request #103 from adam-miller/SeedsReportRedirectFix
...
CandidatesProcessor.java
2014-11-11 18:04:51 -08:00
Noah Levitt
40bc02c03f
Merge pull request #110 from kris-sigur/httpclient-4.3.6
...
Upgrade to HttpClient 4.3.6
2014-11-11 17:35:26 -08:00
Kristinn Sigurðsson
0212a554e6
Upgrade to httpclient 4.3.6
2014-11-11 14:40:33 +00:00
Noah Levitt
7be1fc606b
make sure the kafka producer thread is not in the ToePool thread group, to stop it from being interrupted at the end of the crawl
2014-11-10 16:45:08 -08:00
Noah Levitt
743ab3c9c0
Revert "Revert "switch to org.apache.kafka.clients.producer.KafkaProducer to see if we can avoid the InterruptedException at the end of the crawl (and log warnings on failure to send message)""
...
This reverts commit a0f70d0865
2014-11-10 16:44:37 -08:00
Hunter