... configure using rethinkdb url and segment id, instead of write url,
which means it can work if the segment gets reassigned and so forth
***backward incompatible change***
- make sure we "promote" segments (persist them to hdfs) after we're
entirely finished writing to theme; in other words, when we reach the
FINISH state, not while STOPPING
- shut down the promoter thread at crawl finish
plus a couple of tweaks and cleanups
Sometimes our crawl jobs get stuck in the NEW state (due to hbase
problems). But the checkpoint service has started and sees fit to
checkpoint every five minutes. Evidently, at this stage the seeds have
not been queued yet, the frontier is empty. Thus, if we try to resume
from one of these checkpoints, the crawl has an empty frontier and ends
immediately.
The fix is to avoid checkpointing in this NEW state before crawling has
really started.
This is a bad bug. Stupid, too. When multiple crawls are running with
`DecideRuleSequence.logToFile` enabled, all the scope log lines from all
the crawls are going to all the scope logs for all the crawls. It was
because of this line:
Logger logger = Logger.getLogger(logName + ".log");
`logName` here is "scope" normally, for all crawl jobs.
`Logger` is shared across the whole java vm, so
`Logger.getLogger("scope.log")` returns the same logger for every
crawl job. Each time a crawl starts, it adds another output file to
the logger.
Fix is to give the logger a name specific to the crawl job.
😳🥵🥶💩
realized that we're leaking zookeeper connections, and it might be
because the HConnection has been thread local, which probably means that
at shutdown only one of the possibly many connections is cleaned up
zookeeper by default allows 60 connections per client and it looks
likely that we start having trouble when we hit that limit
DecideRuleSequence already has the optimization that I was looking for,
namely, don't bother evaluating a DecideRule if we know it won't change
the current result. For some reason PredicatedDecideRule, which is a
parent class to most of the decide rules in heritrix, didn't implement
onlyDecision(). Certain crawl configurations could see significant
performance improvement with this change.
Catch exceptions and attempt to reset the hbase connection. The intent
is for this to fix the problem where the hbase connection (under the
hood the problem is the zookeeper connection) gets into a borked state,
and we never reset it.
Noah Levitt