lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-25 07:05:49 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,696 Commits 38 Branches 33 Tags
44ac1030f2b496f4e8380533bbc48b0527f254a2
Commit Graph
1696 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Hunter Stern 44ac1030f2 Make sure to synchronize in addCookie now that domain count limits are imposed and modify unit test to also take into account domain limits 2015-10-16 17:12:42 -07:00
Hunter Stern d9684e913b Restrict number of cookies per domain to 50 2015-10-16 14:05:56 -07:00
Hunter Stern 5ec6c738a6 Revert "Possible fix that probably isn't needed" 
This reverts commit c6f019a095.
 2015-10-13 17:21:28 -07:00
Hunter Stern c6f019a095 Possible fix that probably isn't needed 2015-10-13 14:18:06 -07:00
Hunter 79ebb0e03c Merge pull request #129 from nlevitt/java8keytool 
load keytool utility main class dynamically, trying both the old and …
 2015-10-06 16:09:43 -07:00
Hunter a9c4cb42c9 Merge pull request #130 from nlevitt/stay-alive-statistics-tracker 
catch exceptions and log error in StatisticsTracker.run(), to make su…
 2015-10-06 16:09:11 -07:00
Hunter 98d41051b8 Merge pull request #131 from nlevitt/amqp-url-receiver-changes 
set isRunning=true so that stop() gets called to avoid leaking connec…
 2015-10-06 15:52:05 -07:00
Noah Levitt af5c6c3dde set isRunning=true so that stop() gets called to avoid leaking connections when crawls finish 2015-10-06 15:50:47 -07:00
Noah Levitt 11d4bab1ac catch exceptions and log error in StatisticsTracker.run(), to make sure statistics events don't stop happening 2015-10-05 11:46:35 -07:00
Noah Levitt 41dac25cb6 Merge pull request #128 from ukwa/amqp-url-receiver-changes 
AMQPUrlReceiver changes to support RabbitMQ >= 3.3
 2015-09-24 16:52:04 -07:00
Noah Levitt b7afc627ed fix typo 2015-09-24 15:31:12 -07:00
Noah Levitt 88c7f3c3d4 Merge pull request #124 from vonrosen/ari-4371 
Allow realm to be set by server for basic auth.
 2015-09-24 10:26:49 -07:00
Andrew Jackson e883977de6 Downgrade Throwable to Exception. 2015-09-23 23:14:11 +01:00
Noah Levitt 9aabcb3272 add license header to new class 2015-09-23 12:07:38 -07:00
Noah Levitt 12dde2b693 load keytool utility main class dynamically, trying both the old and new (java 8) class names 2015-09-23 12:06:11 -07:00
Andrew Jackson d1ea01c75d Updated with additional optional fields and separated isConsuming from isRunning. 2015-09-19 21:39:19 +01:00
Andrew Jackson cd3fb8d717 Configurable and 3.3 compatible AMQP receiver for #1 2015-09-18 15:48:09 +01:00
Noah Levitt 490a257cfe Merge pull request #126 from caofangkun/her-001 
'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing
 2015-08-25 10:40:59 -07:00
caofangkun 6cf0d4a0c5 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing 2015-08-24 17:56:56 +08:00
Hunter Stern f097939a1c Don't call setRealm so other realms that can be used. 2015-08-14 13:01:54 -07:00
Noah Levitt f5a4ecbb03 Merge pull request #125 from ldko/AMQP-declarations-fix 
Amqp declarations fix
 2015-08-03 14:26:38 -07:00
Lauren Ko c94f515956 Sets the default clientId to the same as on Receiver side 2015-07-31 15:45:49 -05:00
Lauren Ko 592c70fa3d declare AMQP exchange in AMQPUrlReceiver to prevent exception when exchange not yet created 2015-07-30 17:34:48 -05:00
Hunter Stern 4e22f1f809 Allow realm to be set by server for basic auth. 2015-07-07 13:06:29 -07:00
Hunter 7afe03540a Merge pull request #120 from nlevitt/extractorjs-twodots 
for ARI-4267 accept possible uris with two dots in the filename part if ...
 2015-05-04 15:45:16 -07:00
Hunter eb78de6963 Merge pull request #121 from nlevitt/knowledgable-extractorjs 
new contrib module KnowledgableExtractorJS, a subclass of ExtractorJS th...
 2015-05-04 15:41:21 -07:00
Noah Levitt fea49f0e30 expand stubbed javadoc comment 2015-05-04 15:16:11 -07:00
Noah Levitt 0f43a3dae6 add license header 2015-05-04 15:08:52 -07:00
Noah Levitt e9fcb34508 new contrib module KnowledgableExtractorJS, a subclass of ExtractorJS that has some customized behavior for specific kinds of web pages; initially, the one special behavior it has is for drupal generated pages 2015-04-21 01:19:33 -07:00
Noah Levitt 1027f854ec for ARI-4267 accept possible uris with two dots in the filename part if they end with a known good extension (e.g. "foo.min.js") 2015-04-20 17:08:47 -07:00
Noah Levitt 0923a7ca18 Merge pull request #119 from adam-miller/HER-2082 
Fix for HER-2082
 2015-03-26 16:30:49 -07:00
Adam Miller 458cbb51b0 Fix for HER-2082 
https://webarchive.jira.com/browse/HER-2082
WARCWriterProcessor.java
	Add last hop character and link context to outlinks in metadata records
 2015-03-26 16:19:09 -07:00
Noah Levitt bcae99daa8 Merge pull request #118 from kris-sigur/HER-2080 
Fix for ServerNotModified WARC revisit records incorrectly record WARC-Payload-Digest
 2015-03-23 11:16:36 -07:00
Kristinn Sigurðsson f5107f60b8 Add annotation for server-not-modified duplicates. 2015-03-20 13:46:20 +00:00
Kristinn Sigurðsson dff219e884 Harmonizing whitespace use 2015-03-13 14:32:23 +00:00
Kristinn Sigurðsson 843f6468c4 Use digest from previous capture when encountering 304s. 
The previous capture may also have been a 304 but there must be an
original 200 capture at the start. This assumption may fail when using
an index generated by a version of Heritrix without this fix. But even
then you should be no worse off than before.
 2015-03-13 14:30:39 +00:00
Kristinn Sigurðsson 3ffe5b8b4f Stop automatically writing WARC-Payload-Digest for revisit records. 
Refer that functionality to the relevant revisit profile classes.
 2015-03-11 13:52:31 +00:00
vonrosen b5da93d04b Merge pull request #117 from nlevitt/ari-4222 
avoid java.lang.NullPointerException at org.archive.modules.writer.Write...
 2015-02-26 10:04:34 -08:00
Noah Levitt a0c3fcdae0 avoid java.lang.NullPointerException at org.archive.modules.writer.WriterPoolProcessor.copyForwardWriteTagIfDupe(WriterPoolProcessor.java:504); this exception has not always been possible, but is possible now because IdenticalDigestDecideRule.hasIdenticalDigest(curi) can return true even if there is no curi.getFetchHistory(), since uri-agnostic dedup doesn't use fetchHistory 2015-02-23 14:09:25 -08:00
Adam Miller 1b6d78bf7a Merge pull request #116 from nlevitt/log4j-breaks-tests 
make sure log4j is configured when running unit tests, to avoid log4j er...
 2015-02-23 11:59:50 -08:00
Noah Levitt 8009275075 make sure log4j is configured when running unit tests, to avoid log4j error messages polluting captured output and causing tests to fail, like so: https://builds.archive.org/job/Heritrix-3/org.archive.heritrix$heritrix-commons/587/testReport/org.archive.io.arc/ARCWriterTest/testLengthTooLongCompressed/ 2015-02-23 11:36:29 -08:00
vonrosen 0a418a8425 Merge pull request #114 from nlevitt/webarchive-commons-issue-38 
remove RecordingOutputStreamTest.java (moving to webarchive-commons)
 2015-02-12 15:33:22 -08:00
Noah Levitt 1bd8b141aa bump webarchive-commons dependency to 1.1.5 to pick up https://github.com/iipc/webarchive-commons/pull/39 2015-02-11 18:53:34 -08:00
Noah Levitt 789bd8ac7c Merge pull request #115 from kris-sigur/report-encoding 
Set character set to UTF-8 when passing through files.
 2015-02-10 23:54:21 -08:00
Kristinn Sigurðsson 4ea7efdfc8 Set character set to UTF-8 when passing through files. 2015-02-10 16:07:37 +00:00
Noah Levitt 2078a9f0e2 remove TmpDirTestCase.java (moving to webarchive-commons) 2015-01-06 16:08:18 -08:00
Noah Levitt 64e3b1b02a remove RecordingOutputStreamTest.java (moving to webarchive-commons) 2015-01-06 16:01:16 -08:00
vonrosen 2c9e82a261 Merge pull request #112 from nlevitt/amqp-receiver-deadlock 
Amqp receiver deadlock
 2014-11-21 10:15:53 -08:00
vonrosen 3609abe38c Merge pull request #108 from nlevitt/aitfive-203 
kafka crawl log feed
 2014-11-14 10:29:41 -08:00
Noah Levitt 54555e8566 Merge pull request #103 from adam-miller/SeedsReportRedirectFix 
CandidatesProcessor.java
 2014-11-11 18:04:51 -08:00