lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-24 07:26:36 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,670 Commits 40 Branches 31 Tags
eb78de69634bc0959e52b5bfbf235d5fd9734bf4
Commit Graph
1670 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Hunter eb78de6963 Merge pull request #121 from nlevitt/knowledgable-extractorjs 
new contrib module KnowledgableExtractorJS, a subclass of ExtractorJS th...
 2015-05-04 15:41:21 -07:00
Noah Levitt fea49f0e30 expand stubbed javadoc comment 2015-05-04 15:16:11 -07:00
Noah Levitt 0f43a3dae6 add license header 2015-05-04 15:08:52 -07:00
Noah Levitt e9fcb34508 new contrib module KnowledgableExtractorJS, a subclass of ExtractorJS that has some customized behavior for specific kinds of web pages; initially, the one special behavior it has is for drupal generated pages 2015-04-21 01:19:33 -07:00
Noah Levitt 0923a7ca18 Merge pull request #119 from adam-miller/HER-2082 
Fix for HER-2082
 2015-03-26 16:30:49 -07:00
Adam Miller 458cbb51b0 Fix for HER-2082 
https://webarchive.jira.com/browse/HER-2082
WARCWriterProcessor.java
	Add last hop character and link context to outlinks in metadata records
 2015-03-26 16:19:09 -07:00
Noah Levitt bcae99daa8 Merge pull request #118 from kris-sigur/HER-2080 
Fix for ServerNotModified WARC revisit records incorrectly record WARC-Payload-Digest
 2015-03-23 11:16:36 -07:00
Kristinn Sigurðsson f5107f60b8 Add annotation for server-not-modified duplicates. 2015-03-20 13:46:20 +00:00
Kristinn Sigurðsson dff219e884 Harmonizing whitespace use 2015-03-13 14:32:23 +00:00
Kristinn Sigurðsson 843f6468c4 Use digest from previous capture when encountering 304s. 
The previous capture may also have been a 304 but there must be an
original 200 capture at the start. This assumption may fail when using
an index generated by a version of Heritrix without this fix. But even
then you should be no worse off than before.
 2015-03-13 14:30:39 +00:00
Kristinn Sigurðsson 3ffe5b8b4f Stop automatically writing WARC-Payload-Digest for revisit records. 
Refer that functionality to the relevant revisit profile classes.
 2015-03-11 13:52:31 +00:00
vonrosen b5da93d04b Merge pull request #117 from nlevitt/ari-4222 
avoid java.lang.NullPointerException at org.archive.modules.writer.Write...
 2015-02-26 10:04:34 -08:00
Noah Levitt a0c3fcdae0 avoid java.lang.NullPointerException at org.archive.modules.writer.WriterPoolProcessor.copyForwardWriteTagIfDupe(WriterPoolProcessor.java:504); this exception has not always been possible, but is possible now because IdenticalDigestDecideRule.hasIdenticalDigest(curi) can return true even if there is no curi.getFetchHistory(), since uri-agnostic dedup doesn't use fetchHistory 2015-02-23 14:09:25 -08:00
Adam Miller 1b6d78bf7a Merge pull request #116 from nlevitt/log4j-breaks-tests 
make sure log4j is configured when running unit tests, to avoid log4j er...
 2015-02-23 11:59:50 -08:00
Noah Levitt 8009275075 make sure log4j is configured when running unit tests, to avoid log4j error messages polluting captured output and causing tests to fail, like so: https://builds.archive.org/job/Heritrix-3/org.archive.heritrix$heritrix-commons/587/testReport/org.archive.io.arc/ARCWriterTest/testLengthTooLongCompressed/ 2015-02-23 11:36:29 -08:00
vonrosen 0a418a8425 Merge pull request #114 from nlevitt/webarchive-commons-issue-38 
remove RecordingOutputStreamTest.java (moving to webarchive-commons)
 2015-02-12 15:33:22 -08:00
Noah Levitt 1bd8b141aa bump webarchive-commons dependency to 1.1.5 to pick up https://github.com/iipc/webarchive-commons/pull/39 2015-02-11 18:53:34 -08:00
Noah Levitt 789bd8ac7c Merge pull request #115 from kris-sigur/report-encoding 
Set character set to UTF-8 when passing through files.
 2015-02-10 23:54:21 -08:00
Kristinn Sigurðsson 4ea7efdfc8 Set character set to UTF-8 when passing through files. 2015-02-10 16:07:37 +00:00
Noah Levitt 2078a9f0e2 remove TmpDirTestCase.java (moving to webarchive-commons) 2015-01-06 16:08:18 -08:00
Noah Levitt 64e3b1b02a remove RecordingOutputStreamTest.java (moving to webarchive-commons) 2015-01-06 16:01:16 -08:00
vonrosen 2c9e82a261 Merge pull request #112 from nlevitt/amqp-receiver-deadlock 
Amqp receiver deadlock
 2014-11-21 10:15:53 -08:00
vonrosen 3609abe38c Merge pull request #108 from nlevitt/aitfive-203 
kafka crawl log feed
 2014-11-14 10:29:41 -08:00
Noah Levitt 54555e8566 Merge pull request #103 from adam-miller/SeedsReportRedirectFix 
CandidatesProcessor.java
 2014-11-11 18:04:51 -08:00
Noah Levitt 40bc02c03f Merge pull request #110 from kris-sigur/httpclient-4.3.6 
Upgrade to HttpClient 4.3.6
 2014-11-11 17:35:26 -08:00
Kristinn Sigurðsson 0212a554e6 Upgrade to httpclient 4.3.6 2014-11-11 14:40:33 +00:00
Noah Levitt 7be1fc606b make sure the kafka producer thread is not in the ToePool thread group, to stop it from being interrupted at the end of the crawl 2014-11-10 16:45:08 -08:00
Noah Levitt 743ab3c9c0 Revert "Revert "switch to org.apache.kafka.clients.producer.KafkaProducer to see if we can avoid the InterruptedException at the end of the crawl (and log warnings on failure to send message)"" 
This reverts commit a0f70d0865.
 2014-11-10 16:44:37 -08:00
Noah Levitt a0f70d0865 Revert "switch to org.apache.kafka.clients.producer.KafkaProducer to see if we can avoid the InterruptedException at the end of the crawl (and log warnings on failure to send message)" 
This reverts commit d2f70e5c97.

Conflicts:
	contrib/src/main/java/org/archive/modules/postprocessor/KafkaCrawlLogFeed.java
 2014-11-10 13:55:13 -08:00
Noah Levitt d96b1f8eca Merge pull request #100 from vonrosen/ari-4061 
Allow discovery of urls in content attribute of meta tags.
 2014-11-08 00:05:00 -08:00
Noah Levitt fa0a28f78b Merge branch 'aitfive-203' of github.com:nlevitt/heritrix3 into aitfive-203 2014-11-07 15:59:24 -08:00
Noah Levitt 192b3f9b4b for kafka/amqp crawl log feeds, remove now redundant contentSize from extraInfo 2014-11-07 15:59:14 -08:00
Noah Levitt 4527d5af29 Merge branch 'extra-info-size' into aitfive-203 2014-11-07 15:58:08 -08:00
Noah Levitt 7312ad597e so that it can appear in the crawl log, add contentSize to CrawlURI extraInfo, for http transactions (the only case in heritrix currently where contentLength and contentSize differ) 2014-11-07 13:21:42 -08:00
Noah Levitt e2be3bbf9c new style kafka configuration properties 2014-11-06 19:53:41 -08:00
Noah Levitt 82bef2a337 only release the lock if we actually own it (avoid java.lang.IllegalMonitorStateException at java.util.concurrent.locks.ReentrantLock.tryRelease(ReentrantLock.java:155)) 2014-11-06 19:33:35 -08:00
Noah Levitt a37b879a4a logging improvement 2014-11-06 19:28:19 -08:00
Noah Levitt d2f70e5c97 switch to org.apache.kafka.clients.producer.KafkaProducer to see if we can avoid the InterruptedException at the end of the crawl (and log warnings on failure to send message) 2014-11-06 19:26:32 -08:00
Noah Levitt 23fa29d673 fix log messages to say "kafka" 2014-11-05 14:03:54 -08:00
Noah Levitt 77e2a47c4c AITFIVE-203 kafka crawl log feed 2014-11-05 12:59:10 -08:00
Noah Levitt 4d0eff23d2 Merge pull request #107 from vonrosen/ari-4102 
Handle case where form does not have an action defined.
 2014-11-04 15:55:50 -08:00
Hunter Stern 9a41b31264 Handle case where form does not have an action defined. 2014-11-04 11:22:38 -08:00
vonrosen f5cf30ec3e Merge pull request #102 from nlevitt/amqp-receiver-deadlock 
avoid deadlock in AMQPUrlReceiver hopefully
 2014-11-03 08:11:00 -08:00
Noah Levitt 716e0cc7ef use "fair" ReentrantLock -- I stepped through heritrix that appeared to be deadlocked, found that the thread trying to shut down was being starved of the lock by another thread (the StarterRestarter) locking and unlocking it repeatedly, which can apparently happen because the lock was "unfair" 2014-10-27 17:34:41 -07:00
jkafader edcc7cef21 Merge pull request #106 from nlevitt/aitfive-39 
seriously, fix extraInfo handling in AMQPCrawlLogFeed
 2014-10-26 19:20:38 -07:00
Noah Levitt 6f21ede979 seriously, fix extraInfo handling in AMQPCrawlLogFeed 2014-10-26 19:17:23 -07:00
jkafader 23a07d5cef Merge pull request #105 from nlevitt/aitfive-39 
fix extraInfo handling in AMQPCrawlLogFeed
 2014-10-26 18:29:21 -07:00
Noah Levitt 7e4b1717d1 fix extraInfo handling in AMQPCrawlLogFeed 2014-10-26 18:24:47 -07:00
jkafader f25c171a79 Merge pull request #104 from nlevitt/aitfive-39 
change field names to match new druid config
 2014-10-26 17:11:57 -07:00
Noah Levitt 13a9d3a1be change field names to match new druid config 2014-10-22 15:28:40 -07:00