The parsing throws StringIndexOutOfBoundsException when the version
does not contain '.' such as version "17". This check could never
actually trigger as the class cannot even be loaded on 1.7 or earlier,
so there's no reason to keep it.
Fixes#439
Colin reported this exception. I'm uncertain how this can occur though
as FetchHTTP should populate the response headers. Perhaps a different
Fetch module was used?
We hit a timeout during CI. The timeout is just a safety measure in case
the browser hangs so it doesn't hurt to have it higher. Using a higher
value will hopefully help if the system temporarily stalls for some
reason (garbage collection, IO issues, VM migration etc).
Each direct subclass of BaseResource defines an identical getEngine()
method so let's pull it up to BaseResource. We can also don't need the
type cast anymore as BaseResource.getApplication() does it for us.
Data URIs can be very large. ExtractorHTML mostly works with off-heap
CharSequences so by delaying the conversion of outlinks to strings
until after filtering out data URIs we can potentially avoid some
very large String allocations.
Thread interruption and certain other exceptions can cause a toe thread
to exit without informing the frontier that the current CrawlURI is
finished. This causes the job to get permanently stuck in the STOPPING
state.
This change adds a section to the finally block that will finish any
unfinished CrawlURI.
We also move the continueCheck() call after setCurrentCuri() to ensure
there's no window where InterruptedException can be thrown after the
frontier returns the next CrawlURI but before it gets assigned to
currentCuri.
Fixes#420
Since we now run extractors on subresources there's no reason to
schedule and refetch them again.
Note that duplicate fetches can still occur if the URI was already
scheduled or if the browser itself refetches the resource.
This ensures we discover links in subresources even if the browser
doesn't happen to load them. For example a CSS file might link to images
that the browser won't load as they're gated by media queries.
By intercepting the browser's request and fulfilling it using the
response previously recorded by FetchHTTP we avoid sending duplicate
requests for the CrawlURI to the web server.
A size limit (maxReplayLength) is applied as a safety measure since the
browser's Fetch.fulfillRequest API requires us to load the entire
response body into memory.
Note: This only applies to the main CrawlURI. The browser can still
make duplicate requests when loading sub-resources. Solving this for
sub-resources will require implementing the ability to read back
previously written WARC records.
JavaDoc comments need to be directly above a class, method or field
declaration to be used by IDEs and generated documentation. We had some
doc comments that were attached to instance initialization blocks and
thus were being ignored. This invalid doc comment positioning was
presumably an accidental consequence of the conversion of fields to
KeyedProperties.
This change moves most of the dangling doc comments to setter methods
where it can be seen by tools. There were a couple of dangling doc
comments above package statements. These are moved above the class
declaration or removed entirely when empty.
This enables us to generate documentation for more bean properties,
although a number of beans have javadoc on initializer code blocks which
makes it hard to access. This affects javadoc and IDE contextual
documentation too so should probably be fixed in the source code itself.
Alex Osborne