There is already a clause for this in logging.properties, but it's using
log4j. It was dumping stack traces every time the client was dubious of
heritrix's self-signed certificate.
Why do we have so many identical log4j.xml's? 🤷♂️
These changes come from a private fork of H3, originally made by Kenji
Nagahashi, to create org.archive.crawler.frontier.PullingBdbFrontier,
which we intend to merge into 'contrib' of the official version in the
near future.
Sometimes our crawl jobs get stuck in the NEW state (due to hbase
problems). But the checkpoint service has started and sees fit to
checkpoint every five minutes. Evidently, at this stage the seeds have
not been queued yet, the frontier is empty. Thus, if we try to resume
from one of these checkpoints, the crawl has an empty frontier and ends
immediately.
The fix is to avoid checkpointing in this NEW state before crawling has
really started.
This is a bad bug. Stupid, too. When multiple crawls are running with
`DecideRuleSequence.logToFile` enabled, all the scope log lines from all
the crawls are going to all the scope logs for all the crawls. It was
because of this line:
Logger logger = Logger.getLogger(logName + ".log");
`logName` here is "scope" normally, for all crawl jobs.
`Logger` is shared across the whole java vm, so
`Logger.getLogger("scope.log")` returns the same logger for every
crawl job. Each time a crawl starts, it adds another output file to
the logger.
Fix is to give the logger a name specific to the crawl job.
😳🥵🥶💩
* origin/master:
Setup TravisCI
Set fetch status on curis when testing link extraction
No link extraction on URI not successfully downloaded
Fixes issue #158 : Noisy alerts about 401s without auth challenge (#159)
Make Content-Location header url INFERRED not REFFER hop type since Content-Location is not for redirection (#151)
fixes for kafka 0.9 (?)
upgrade to kafka 0.9
somewhat ugly fix to handle exceptions from the bean browser like java.lang.RuntimeException: not implemented at org.archive.modules.fetcher.BdbCookieStore$RestrictedCollectionWrappedList.get(BdbCookieStore.java:92)
* master:
license header
check that sourceTag of CrawlURI actually matches configured sourceTag
remove already-outdated stuff from javadoc
handle multiple clauses for same user agent in robots.txt
Hook in submitted seeds properly.
avoid spurious logging
try very hard to start url consumer, and therefore bind the queue to the routing key, so that no messages are dropped, before crawling starts (should always work unless rabbitmq is down); some other tweaks for clarity and stability
Noah Levitt