BdbMultipleWorkQueues.calculateInsertKey() silently clips precedence
values to 127 for queue ordering. This can cause unexpected behavior
for custom cost assignment policies that return values above 127.
Log a warning when clipping occurs and fix the misleading comment in
CrawlURI.holderCost that said "should not exceed 255" (the actual
limit is 127, not 255).
Fixes#502
WWWAuthenticationProtocolHandler and ProxyAuthenticationProtocolHandler unhelpfully throw exceptions if a 401 or 407 response has a larger than 16KB body or is missing the appropriate -Authenticate header.
FetchHTTP2 doesn't yet support HTTP auth, thus they aren't currently being used, so let's just remove them for now. This also adds regression tests so that if in the future we implement auth support, we don't accidentally reintroduce this.
Fixes#714Fixes#717
If there are multiple addresses then ignore 0.0.0.0. If the only address is 0.0.0.0 treat it as a DNS resolution failure. This should prevent us from throwing IllegalStateException later in DefaultServerCache.getHostFor().
Fixes#428
After I added them to the Bean reference they seemed out of place with the DecideRules and RecordBuilders.
We had two classes called 'Browser' which is potentially confusing. It's also probably good to more clearly differentiate the ExtractLinks browser behavior from the Extractor processors.
The Browser processor can load a fetched page in a local web browser, record any requests the browser makes and run behaviors that interact with the page such as scrolling down and extracting links.
This differs from my previous attempt (ExtractorChrome) in a few ways:
- Uses the new WebDriver BiDi standard instead of the Chrome Devtools Protocol. The new protocol is mostly browser-agnostic, more consistent and hopefully more stable.
- Uses a MITM proxy instead of CDP request interception for recording sub-resources. That's partly because BiDi is still missing some key interception APIs. Even so in practice I found the proxy method loads pages faster and more reliably, likely because responses can be streamed incrementally, which helps a lot for large resources or server-sent events.
- Even when HTTP/2 is unavailable, the new FetchHTTP2 module does connection pooling which makes loading browser requests a lot faster. The original FetchHTTP opened a new connection for every request.
- The Browser processor can be configured with a list of behavior beans making it more customizable and extensible.
Obvious areas for future development:
- More Behavior beans: take screenshots, saveg the rendered DOM, run Browsertrix-compatible behavior scripts
- Support for remote WebDrivers (e.g. Selenium Server or cloud services)
This uses Jetty HttpClient since it speaks both protocols, and we
already have it as a dependency via Restlet. This doesn't support all
the options of FetchHTTP, notably proxy and POST requests are missing.
Jetty currently has the HTTP/3 client marked as "experimental, not for
production use" so we disable it by default and don't ship the large
quiche native jar it requires. It does seem to work OK though, at least
in my limited testing so far. The HTTP/3 support currently only responds
to Alt-Svc headers not other ways of discovering HTTP/3 availability
(e.g. HTTPS DNS record).
Fetches that were made via HTTP/2 or HTTP/3 are annotated 'h2' and 'h3'
in the crawl.log. The messages are recorded in the WARC files using
HTTP/1.1 syntax with a WARC-Protocol header. FetchHTTP2 also currently
records HTTP/1.1 messages without transfer-encoding rather than the raw
wire messages.
The Jetty API has changed, which mostly affects test code.
Jetty now does a strict SNI host check which unfortunately causes it to
return "SNI error" for our existing ad-hoc certificates. For now, I've
disabled it to avoid breaking existing deployments but added a
--sni-host-check command-line option so you can re-enable it if
you've configured your own certificate appropriately.
When enabled this option causes regular links annotated with rel=nofollow to not be extracted. This is useful for sites that use rel=nofollow to hint crawler traps.
fastutil is our largest dependency, consuming about a third of the
total Heritrix distribution size but we only use a couple of trivial
classes from it.
FPMergeUriUniqFilter (which I'm not sure anyone uses anyway), uses
LongArrayList so this change replaces it with a basic version that does
just enough.
The unsynchronized FastBufferedOutputStream usages are likely
unnecessary these days thanks to the JVM's lock optimisations and for
the one in CrawlerJournal, the GZIPOutputStream is still going to
be synchronizing anyway.
Spring 6 removed @Required and they suggest using constructor injection instead. If we switched our beans to that we'd break existing Heritrix crawl configs. So this change implements our own basic version so we still get errors when a @Required property is null.
Adds a list of regular expressions that URLs being processed by the
ConfigurableExtractorJS are evaluted against. If they match the
extraction is performed in strict mode, even if strict mode is not set.
This requires a minor modification to ExtractorJS so that the CrawlURI
is passed to the shouldAddUri method that ConfigurableExtractorJS
overrides.
Links from manifests (e.g. sitemaps) should not receive the preferential
treatment sometimes accorded to "transitive" hops. Most commonly this is
about giving priority to discovered (probable) embeds.
Manifests should be regarded as more analogous with a directory page.
Alex Osborne