We hit a timeout during CI. The timeout is just a safety measure in case
the browser hangs so it doesn't hurt to have it higher. Using a higher
value will hopefully help if the system temporarily stalls for some
reason (garbage collection, IO issues, VM migration etc).
Data URIs can be very large. ExtractorHTML mostly works with off-heap
CharSequences so by delaying the conversion of outlinks to strings
until after filtering out data URIs we can potentially avoid some
very large String allocations.
Thread interruption and certain other exceptions can cause a toe thread
to exit without informing the frontier that the current CrawlURI is
finished. This causes the job to get permanently stuck in the STOPPING
state.
This change adds a section to the finally block that will finish any
unfinished CrawlURI.
We also move the continueCheck() call after setCurrentCuri() to ensure
there's no window where InterruptedException can be thrown after the
frontier returns the next CrawlURI but before it gets assigned to
currentCuri.
Fixes#420
Since we now run extractors on subresources there's no reason to
schedule and refetch them again.
Note that duplicate fetches can still occur if the URI was already
scheduled or if the browser itself refetches the resource.
This ensures we discover links in subresources even if the browser
doesn't happen to load them. For example a CSS file might link to images
that the browser won't load as they're gated by media queries.
By intercepting the browser's request and fulfilling it using the
response previously recorded by FetchHTTP we avoid sending duplicate
requests for the CrawlURI to the web server.
A size limit (maxReplayLength) is applied as a safety measure since the
browser's Fetch.fulfillRequest API requires us to load the entire
response body into memory.
Note: This only applies to the main CrawlURI. The browser can still
make duplicate requests when loading sub-resources. Solving this for
sub-resources will require implementing the ability to read back
previously written WARC records.
JavaDoc comments need to be directly above a class, method or field
declaration to be used by IDEs and generated documentation. We had some
doc comments that were attached to instance initialization blocks and
thus were being ignored. This invalid doc comment positioning was
presumably an accidental consequence of the conversion of fields to
KeyedProperties.
This change moves most of the dangling doc comments to setter methods
where it can be seen by tools. There were a couple of dangling doc
comments above package statements. These are moved above the class
declaration or removed entirely when empty.
This enables us to generate documentation for more bean properties,
although a number of beans have javadoc on initializer code blocks which
makes it hard to access. This affects javadoc and IDE contextual
documentation too so should probably be fixed in the source code itself.
This is meant to complement the javadoc by providing a reference
more suitable for users trying to configure crawls rather than
developers writing new modules.
The doc generation could still do with some improving and some of the
source javadoc comments need fixing up but this is already useful so I'm
committing what I have so far.
Large portions of the docs were first published much earlier in other
places and some have been updated since then. Rather than trying to
keep a date range up to date let's just remove the date. We don't use
dates in the source code boilerplate and my understanding is copyright
notices are not mandatory in almost all countries due to the Berne
Convention anyway so it's really just informational.
This adds a `captureRequests` flag to ExtractorChrome which is enabled
by default and causes requests made by the browser to be captured via
the devtools Network domain. Captured browser requests are sent to the
disposition chain for WARC writing and also to statistics tracker and
crawl log.
Browser requests are given the annotation "browser" so they can be
easily distinguished in the log from normal requests.
There are a quite a few limitations that will be addressed in followup
work:
* The frontier is entirely unaware of browser requests. This means they
bypass quotas, ignore scope rules, politeness and don't count towards
the statistics tracked by the frontier itself.
* There's no replay of previously saved resources so duplicate requests
for the same URL end up being made.
* Heritrix's extractors do not currently process browser requests.
* Various error and failure cases likely need improving.
Alex Osborne