dnsjava will create its global selector thread on first DNS lookup.
Since the first lookup occurs on a toe thread then this will cause the
selector thead to join the job's toe thread group. This is a problem
because when the job is terminated ToePool sends an interrupt to all
threads in the thread group which the dnsjava selector thread is
not expected and never clears. The interrupted state then causes
select() to return immediately instead of sleeping so it consumes
100% CPU.
So workaround this by doing a dummy lookup when FetchDNS is loaded,
this ensures the first lookup happens on the web server thread that
handles the job build action instead of a toe thread.
Fixes: 045b2516da ("Update to latest version of dnsjava")
Since the cache is global we only need to configure it once, not every
time we start a job. By moving it we keep the code that interfaces with
dnsjava centralized in FetchDNS.
Looks like dnsjava no longer uses a cleaner thread so remove the comment
referring to it.
The parsing throws StringIndexOutOfBoundsException when the version
does not contain '.' such as version "17". This check could never
actually trigger as the class cannot even be loaded on 1.7 or earlier,
so there's no reason to keep it.
Fixes#439
Colin reported this exception. I'm uncertain how this can occur though
as FetchHTTP should populate the response headers. Perhaps a different
Fetch module was used?
We hit a timeout during CI. The timeout is just a safety measure in case
the browser hangs so it doesn't hurt to have it higher. Using a higher
value will hopefully help if the system temporarily stalls for some
reason (garbage collection, IO issues, VM migration etc).
Each direct subclass of BaseResource defines an identical getEngine()
method so let's pull it up to BaseResource. We can also don't need the
type cast anymore as BaseResource.getApplication() does it for us.
Data URIs can be very large. ExtractorHTML mostly works with off-heap
CharSequences so by delaying the conversion of outlinks to strings
until after filtering out data URIs we can potentially avoid some
very large String allocations.
Thread interruption and certain other exceptions can cause a toe thread
to exit without informing the frontier that the current CrawlURI is
finished. This causes the job to get permanently stuck in the STOPPING
state.
This change adds a section to the finally block that will finish any
unfinished CrawlURI.
We also move the continueCheck() call after setCurrentCuri() to ensure
there's no window where InterruptedException can be thrown after the
frontier returns the next CrawlURI but before it gets assigned to
currentCuri.
Fixes#420
Since we now run extractors on subresources there's no reason to
schedule and refetch them again.
Note that duplicate fetches can still occur if the URI was already
scheduled or if the browser itself refetches the resource.
This ensures we discover links in subresources even if the browser
doesn't happen to load them. For example a CSS file might link to images
that the browser won't load as they're gated by media queries.
By intercepting the browser's request and fulfilling it using the
response previously recorded by FetchHTTP we avoid sending duplicate
requests for the CrawlURI to the web server.
A size limit (maxReplayLength) is applied as a safety measure since the
browser's Fetch.fulfillRequest API requires us to load the entire
response body into memory.
Note: This only applies to the main CrawlURI. The browser can still
make duplicate requests when loading sub-resources. Solving this for
sub-resources will require implementing the ability to read back
previously written WARC records.
Alex Osborne