2016-02-18 01:44:46.434 SEVERE thread-51 org.archive.crawler.spring.SheetOverlaysManager.applyOverlaysTo() problem determining whether to apply overlays, so not applying [ignoreCookies] to https://static.xx.fbcdn.net/rsrc.php/v2/yY/r/z9xrgSKFbeZ.js
java.lang.NullPointerException
at org.archive.spring.KeyedProperties.get(KeyedProperties.java:65)
at org.archive.modules.deciderules.DecideRule.getEnabled(DecideRule.java:38)
at org.archive.modules.deciderules.DecideRule.decisionFor(DecideRule.java:57)
at org.archive.modules.deciderules.DecideRule.accepts(DecideRule.java:77)
at org.archive.crawler.spring.SheetOverlaysManager.applyOverlaysTo(SheetOverlaysManager.java:331)
at org.archive.crawler.postprocessor.CandidatesProcessor.runCandidateChain(CandidatesProcessor.java:170)
at org.archive.crawler.frontier.AMQPUrlReceiver$UrlConsumer.handleDelivery(AMQPUrlReceiver.java:321)
at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:140)
at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:76)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
* master:
license header
check that sourceTag of CrawlURI actually matches configured sourceTag
remove already-outdated stuff from javadoc
handle multiple clauses for same user agent in robots.txt
Hook in submitted seeds properly.
avoid spurious logging
try very hard to start url consumer, and therefore bind the queue to the routing key, so that no messages are dropped, before crawling starts (should always work unless rabbitmq is down); some other tweaks for clarity and stability
* seed-limits:
change class originally known as SeedLimitsEnforcer to SourceQuotaEnforcer; make it a Processor instead of a DecideRule (because checking quota at link scoping time doesn't work, since many urls which would go over quota can be added to the frontier); support quotas on any of the fields tracked by CrawledBytesHistotable
fix checkpointing problems with new statsBySource
SeedLimitsEnforcer (contrib) DecideRule that rejects CrawlURI if source seed byte or document limit has been reached
SourceSeedDecideRule applies the configured decision for any URI with discovered from one of a set of seeds
add support to StatisticsTracker to keep a CrawledBytesHistotable per source tag when trackSources is enabled; integration test for this functionality
Noah Levitt