There is already a clause for this in logging.properties, but it's using
log4j. It was dumping stack traces every time the client was dubious of
heritrix's self-signed certificate.
Why do we have so many identical log4j.xml's? 🤷♂️
These changes come from a private fork of H3, originally made by Kenji
Nagahashi, to create org.archive.crawler.frontier.PullingBdbFrontier,
which we intend to merge into 'contrib' of the official version in the
near future.
Sometimes our crawl jobs get stuck in the NEW state (due to hbase
problems). But the checkpoint service has started and sees fit to
checkpoint every five minutes. Evidently, at this stage the seeds have
not been queued yet, the frontier is empty. Thus, if we try to resume
from one of these checkpoints, the crawl has an empty frontier and ends
immediately.
The fix is to avoid checkpointing in this NEW state before crawling has
really started.
This is a bad bug. Stupid, too. When multiple crawls are running with
`DecideRuleSequence.logToFile` enabled, all the scope log lines from all
the crawls are going to all the scope logs for all the crawls. It was
because of this line:
Logger logger = Logger.getLogger(logName + ".log");
`logName` here is "scope" normally, for all crawl jobs.
`Logger` is shared across the whole java vm, so
`Logger.getLogger("scope.log")` returns the same logger for every
crawl job. Each time a crawl starts, it adds another output file to
the logger.
Fix is to give the logger a name specific to the crawl job.
😳🥵🥶💩
* origin/master:
Setup TravisCI
Set fetch status on curis when testing link extraction
No link extraction on URI not successfully downloaded
Fixes issue #158 : Noisy alerts about 401s without auth challenge (#159)
Make Content-Location header url INFERRED not REFFER hop type since Content-Location is not for redirection (#151)
fixes for kafka 0.9 (?)
upgrade to kafka 0.9
somewhat ugly fix to handle exceptions from the bean browser like java.lang.RuntimeException: not implemented at org.archive.modules.fetcher.BdbCookieStore$RestrictedCollectionWrappedList.get(BdbCookieStore.java:92)
* master:
license header
check that sourceTag of CrawlURI actually matches configured sourceTag
remove already-outdated stuff from javadoc
handle multiple clauses for same user agent in robots.txt
Hook in submitted seeds properly.
avoid spurious logging
try very hard to start url consumer, and therefore bind the queue to the routing key, so that no messages are dropped, before crawling starts (should always work unless rabbitmq is down); some other tweaks for clarity and stability
* seed-limits:
change class originally known as SeedLimitsEnforcer to SourceQuotaEnforcer; make it a Processor instead of a DecideRule (because checking quota at link scoping time doesn't work, since many urls which would go over quota can be added to the frontier); support quotas on any of the fields tracked by CrawledBytesHistotable
fix checkpointing problems with new statsBySource
SeedLimitsEnforcer (contrib) DecideRule that rejects CrawlURI if source seed byte or document limit has been reached
SourceSeedDecideRule applies the configured decision for any URI with discovered from one of a set of seeds
add support to StatisticsTracker to keep a CrawledBytesHistotable per source tag when trackSources is enabled; integration test for this functionality
Until now hosts that have never been crawled are included in the hosts
report. This can be a very large number of hosts and is infrequently of
interest. This makes it possible to exclude the empty hosts from the
report. Default behavior remains to include them.
The option is highlighted by including it in the relevant, commented-out
section of the default profile.
Can now set a maximum size for sorted reports. Longer reports are
unsorted. Default behavior is still to sort all reports.
Default profile updated to reflect this new option in a commented-out
section.
Noah Levitt