Hunter Stern
5f630e4268
Make regex more inclusive for parsing out youtube videoId and add youtube watch page as relative to base
2016-01-14 09:03:42 -08:00
Hunter Stern
7bcdf03a78
Merge branch 'master' of github.com:internetarchive/heritrix3 into ARI-4562
2016-01-14 08:48:39 -08:00
Noah Levitt
9e4c7f0024
Merge branch 'master' into novel-quotas
...
* master:
license header
check that sourceTag of CrawlURI actually matches configured sourceTag
remove already-outdated stuff from javadoc
handle multiple clauses for same user agent in robots.txt
Hook in submitted seeds properly.
avoid spurious logging
try very hard to start url consumer, and therefore bind the queue to the routing key, so that no messages are dropped, before crawling starts (should always work unless rabbitmq is down); some other tweaks for clarity and stability
2016-01-13 13:44:54 -08:00
Adam Miller
b82896927a
Merge pull request #132 from nlevitt/amqp-url-receiver-changes
...
try very hard to start url consumer, and therefore bind the queue to …
2016-01-13 13:35:43 -08:00
Noah Levitt
df5748d1f1
Merge pull request #136 from ukwa/amqp-declare-recieved-seeds
...
Register seeds send in via AMQP
2016-01-13 13:27:43 -08:00
Noah Levitt
8918ce99c6
check that sourceTag of CrawlURI actually matches configured sourceTag
2015-12-15 11:24:53 -08:00
Noah Levitt
7a8a614804
avoid exception in case applyToSubdomains is enabled and configured host is an ip address
2015-12-15 11:06:33 -08:00
Noah Levitt
a95519543d
avoid exception in case applyToSubdomains is enabled and CrawlURI host is a nonstandard host like "dns:" in HostQuotaEnforcer
2015-12-15 11:02:38 -08:00
Noah Levitt
e3e6bfeb11
new HostQuotaEnforcer option applyToSubdomains, to additionally apply the quotas (separately) to each subdomain of the configured host
2015-12-14 18:17:44 -08:00
Noah Levitt
2f82d43110
remove already-outdated stuff from javadoc
2015-12-14 15:24:08 -08:00
Noah Levitt
013029d9ad
fix annotation copy-paste-o
2015-12-14 15:22:51 -08:00
Noah Levitt
2a56886c01
add license header
2015-12-14 10:46:14 -08:00
Noah Levitt
8deead15ec
new class to enforce quotas on a host
2015-12-11 17:02:32 -08:00
Noah Levitt
c20a06ec27
Merge branch 'seed-limits' into novel-quotas -- so I can add tests to StatisticsSelfTest.java in this branch
...
* seed-limits:
change class originally known as SeedLimitsEnforcer to SourceQuotaEnforcer; make it a Processor instead of a DecideRule (because checking quota at link scoping time doesn't work, since many urls which would go over quota can be added to the frontier); support quotas on any of the fields tracked by CrawledBytesHistotable
fix checkpointing problems with new statsBySource
SeedLimitsEnforcer (contrib) DecideRule that rejects CrawlURI if source seed byte or document limit has been reached
SourceSeedDecideRule applies the configured decision for any URI with discovered from one of a set of seeds
add support to StatisticsTracker to keep a CrawledBytesHistotable per source tag when trackSources is enabled; integration test for this functionality
2015-12-11 11:35:11 -08:00
Noah Levitt
87f70c3e6a
WARCLimitEnforcer to enforce limits (by stopping the crawl job) on any of the stats kept by WARCWriterProcessor
2015-12-11 11:26:11 -08:00
Noah Levitt
566c1a600d
remove warning which can be misleading if crawl is configured to write only certain urls to warcs
2015-12-11 11:01:26 -08:00
Noah Levitt
ce31cada5f
change class originally known as SeedLimitsEnforcer to
...
SourceQuotaEnforcer; make it a Processor instead of a DecideRule
(because checking quota at link scoping time doesn't work, since many
urls which would go over quota can be added to the frontier); support
quotas on any of the fields tracked by CrawledBytesHistotable
2015-12-08 13:45:25 -08:00
Noah Levitt
de22096dde
SeedLimitsEnforcer (contrib) DecideRule that rejects CrawlURI if source seed byte or document limit has been reached
2015-12-03 17:50:46 -08:00
Andrew Jackson
0f6d385b24
Hook in submitted seeds properly.
2015-11-27 20:51:35 +00:00
Hunter Stern
d3070cef54
Allow KnowledgableExtractorJS to parse out youtube watch from youtube javascript embed
2015-11-06 11:39:41 -08:00
Noah Levitt
e37b2b42cb
avoid spurious logging
2015-10-08 18:11:09 -07:00
Noah Levitt
e4acb6c512
try very hard to start url consumer, and therefore bind the queue to the routing key, so that no messages are dropped, before crawling starts (should always work unless rabbitmq is down); some other tweaks for clarity and stability
2015-10-08 17:22:44 -07:00
Noah Levitt
af5c6c3dde
set isRunning=true so that stop() gets called to avoid leaking connections when crawls finish
2015-10-06 15:50:47 -07:00
Andrew Jackson
e883977de6
Downgrade Throwable to Exception.
2015-09-23 23:14:11 +01:00
Andrew Jackson
d1ea01c75d
Updated with additional optional fields and separated isConsuming from isRunning.
2015-09-19 21:39:19 +01:00
Andrew Jackson
cd3fb8d717
Configurable and 3.3 compatible AMQP receiver for #1
2015-09-18 15:48:09 +01:00
Lauren Ko
c94f515956
Sets the default clientId to the same as on Receiver side
2015-07-31 15:45:49 -05:00
Lauren Ko
592c70fa3d
declare AMQP exchange in AMQPUrlReceiver to prevent exception when exchange not yet created
2015-07-30 17:34:48 -05:00
Noah Levitt
fea49f0e30
expand stubbed javadoc comment
2015-05-04 15:16:11 -07:00
Noah Levitt
0f43a3dae6
add license header
2015-05-04 15:08:52 -07:00
Noah Levitt
e9fcb34508
new contrib module KnowledgableExtractorJS, a subclass of ExtractorJS that has some customized behavior for specific kinds of web pages; initially, the one special behavior it has is for drupal generated pages
2015-04-21 01:19:33 -07:00
Noah Levitt
8009275075
make sure log4j is configured when running unit tests, to avoid log4j error messages polluting captured output and causing tests to fail, like so: https://builds.archive.org/job/Heritrix-3/org.archive.heritrix$heritrix-commons/587/testReport/org.archive.io.arc/ARCWriterTest/testLengthTooLongCompressed/
2015-02-23 11:36:29 -08:00
vonrosen
2c9e82a261
Merge pull request #112 from nlevitt/amqp-receiver-deadlock
...
Amqp receiver deadlock
2014-11-21 10:15:53 -08:00
Noah Levitt
7be1fc606b
make sure the kafka producer thread is not in the ToePool thread group, to stop it from being interrupted at the end of the crawl
2014-11-10 16:45:08 -08:00
Noah Levitt
743ab3c9c0
Revert "Revert "switch to org.apache.kafka.clients.producer.KafkaProducer to see if we can avoid the InterruptedException at the end of the crawl (and log warnings on failure to send message)""
...
This reverts commit a0f70d0865
2014-11-10 16:44:37 -08:00
Noah Levitt
a0f70d0865
Revert "switch to org.apache.kafka.clients.producer.KafkaProducer to see if we can avoid the InterruptedException at the end of the crawl (and log warnings on failure to send message)"
...
This reverts commit d2f70e5c97
2014-11-10 13:55:13 -08:00
Noah Levitt
fa0a28f78b
Merge branch 'aitfive-203' of github.com:nlevitt/heritrix3 into aitfive-203
2014-11-07 15:59:24 -08:00
Noah Levitt
192b3f9b4b
for kafka/amqp crawl log feeds, remove now redundant contentSize from extraInfo
2014-11-07 15:59:14 -08:00
Noah Levitt
e2be3bbf9c
new style kafka configuration properties
2014-11-06 19:53:41 -08:00
Noah Levitt
82bef2a337
only release the lock if we actually own it (avoid java.lang.IllegalMonitorStateException at java.util.concurrent.locks.ReentrantLock.tryRelease(ReentrantLock.java:155))
2014-11-06 19:33:35 -08:00
Noah Levitt
a37b879a4a
logging improvement
2014-11-06 19:28:19 -08:00
Noah Levitt
d2f70e5c97
switch to org.apache.kafka.clients.producer.KafkaProducer to see if we can avoid the InterruptedException at the end of the crawl (and log warnings on failure to send message)
2014-11-06 19:26:32 -08:00
Noah Levitt
23fa29d673
fix log messages to say "kafka"
2014-11-05 14:03:54 -08:00
Noah Levitt
77e2a47c4c
AITFIVE-203 kafka crawl log feed
2014-11-05 12:59:10 -08:00
vonrosen
f5cf30ec3e
Merge pull request #102 from nlevitt/amqp-receiver-deadlock
...
avoid deadlock in AMQPUrlReceiver hopefully
2014-11-03 08:11:00 -08:00
Noah Levitt
716e0cc7ef
use "fair" ReentrantLock -- I stepped through heritrix that appeared to be deadlocked, found that the thread trying to shut down was being starved of the lock by another thread (the StarterRestarter) locking and unlocking it repeatedly, which can apparently happen because the lock was "unfair"
2014-10-27 17:34:41 -07:00
jkafader
edcc7cef21
Merge pull request #106 from nlevitt/aitfive-39
...
seriously, fix extraInfo handling in AMQPCrawlLogFeed
2014-10-26 19:20:38 -07:00
Noah Levitt
6f21ede979
seriously, fix extraInfo handling in AMQPCrawlLogFeed
2014-10-26 19:17:23 -07:00
jkafader
23a07d5cef
Merge pull request #105 from nlevitt/aitfive-39
...
fix extraInfo handling in AMQPCrawlLogFeed
2014-10-26 18:29:21 -07:00
Noah Levitt
7e4b1717d1
fix extraInfo handling in AMQPCrawlLogFeed
2014-10-26 18:24:47 -07:00
Hunter Stern