dnsjava will create its global selector thread on first DNS lookup.
Since the first lookup occurs on a toe thread then this will cause the
selector thead to join the job's toe thread group. This is a problem
because when the job is terminated ToePool sends an interrupt to all
threads in the thread group which the dnsjava selector thread is
not expected and never clears. The interrupted state then causes
select() to return immediately instead of sleeping so it consumes
100% CPU.
So workaround this by doing a dummy lookup when FetchDNS is loaded,
this ensures the first lookup happens on the web server thread that
handles the job build action instead of a toe thread.
Fixes: 045b2516da ("Update to latest version of dnsjava")
Since the cache is global we only need to configure it once, not every
time we start a job. By moving it we keep the code that interfaces with
dnsjava centralized in FetchDNS.
Looks like dnsjava no longer uses a cleaner thread so remove the comment
referring to it.
Data URIs can be very large. ExtractorHTML mostly works with off-heap
CharSequences so by delaying the conversion of outlinks to strings
until after filtering out data URIs we can potentially avoid some
very large String allocations.
JavaDoc comments need to be directly above a class, method or field
declaration to be used by IDEs and generated documentation. We had some
doc comments that were attached to instance initialization blocks and
thus were being ignored. This invalid doc comment positioning was
presumably an accidental consequence of the conversion of fields to
KeyedProperties.
This change moves most of the dangling doc comments to setter methods
where it can be seen by tools. There were a couple of dangling doc
comments above package statements. These are moved above the class
declaration or removed entirely when empty.
The vast majority of protocols we use will have a server IP address,
so rather than having a separate one for each protocol let's just use
a single common one.
This eliminates one of the jars that Heritrix depends on that isn't in
Maven Central.
There were only two usages. The first usage was a method in a
CrawlerJournal which as far as I can tell is unused and can be removed.
The second usage was an unnecessary use of MutableString as a line
buffer when saving cookies to a file. Since there doesn't seem any
particular reason buffer per-line here let's just use a BufferedWriter
instead on the whole file which is also slightly simpler and more
efficient.
Alex Osborne