This fixes a bug where POST requests would get recorded with a doubled header and a bunch of nulls at the end.
It seems that the onRequestContent() listener doesn't pass us the right buffer when proxying. The buffer seems to contain client-to-server data and the buffer's limit isn't set. So instead this records the request content by overriding newProxyToServerRequestContent.
Foundation 4 has been obsolete since 2013, and these libraries trigger security alerts. We're also not using them to do much, just the modal dialogs, alert close buttons and collapsing the top-bar on mobile. This replaces the modals with the native HTML dialog element and implements the trivial click event handlers needed for hiding and showing things.
This means:
- Job profiles can be used without having to hunt for them among the regular jobs.
- The Groovy default profile can be selected as an alternative to Spring XML one.
- The default profiles can be customised by creating a job profile with the same name.
BdbMultipleWorkQueues.calculateInsertKey() silently clips precedence
values to 127 for queue ordering. This can cause unexpected behavior
for custom cost assignment policies that return values above 127.
Log a warning when clipping occurs and fix the misleading comment in
CrawlURI.holderCost that said "should not exceed 255" (the actual
limit is 127, not 255).
Fixes#502
Heritrix's Spring XML based job config format by design allows arbitrary code execution, so this doesn't realistically make things much safer, but I also don't see any reason we need external entity resolution enabled. I suppose there's a chance this could mitigate a generic automated attack that uses XXE but doesn't target Heritrix or Spring XML specifically.
Fixes#711
This adds context-sensitive autocompletion for bean classes, bean ids, property names, basic Spring XML tags and attributes. The bean completions use a new `/engine/beandoc` endpoint that serves the combination of all the `/META-INF/heritrix-beans.json` files generated at compile-time by the heritrix-docgen annotation processor.
After I added them to the Bean reference they seemed out of place with the DecideRules and RecordBuilders.
We had two classes called 'Browser' which is potentially confusing. It's also probably good to more clearly differentiate the ExtractLinks browser behavior from the Extractor processors.
The Browser processor can load a fetched page in a local web browser, record any requests the browser makes and run behaviors that interact with the page such as scrolling down and extracting links.
This differs from my previous attempt (ExtractorChrome) in a few ways:
- Uses the new WebDriver BiDi standard instead of the Chrome Devtools Protocol. The new protocol is mostly browser-agnostic, more consistent and hopefully more stable.
- Uses a MITM proxy instead of CDP request interception for recording sub-resources. That's partly because BiDi is still missing some key interception APIs. Even so in practice I found the proxy method loads pages faster and more reliably, likely because responses can be streamed incrementally, which helps a lot for large resources or server-sent events.
- Even when HTTP/2 is unavailable, the new FetchHTTP2 module does connection pooling which makes loading browser requests a lot faster. The original FetchHTTP opened a new connection for every request.
- The Browser processor can be configured with a list of behavior beans making it more customizable and extensible.
Obvious areas for future development:
- More Behavior beans: take screenshots, saveg the rendered DOM, run Browsertrix-compatible behavior scripts
- Support for remote WebDrivers (e.g. Selenium Server or cloud services)
This option enables HTTP Basic authentication for the web interface instead of the default Digest authentication. This is useful when running Heritrix behind a reverse proxy that adds external authentication as typically they don't support Digest auth for the upstream server.
#641
Graceful shutdown would be useful if you could deploy the UI in a high-availability configuration and direct new requests to a different instance while the current instance finished its outstanding ones. But as you can't, it's just making restarting Heritrix slow for little benefit.
This resolves some browser incompatibilities, allowing CodeMirror’s own find function to be re-enabled for reliable text search of content far outside the viewport.
Alex Osborne