This fixes a bug where POST requests would get recorded with a doubled header and a bunch of nulls at the end.
It seems that the onRequestContent() listener doesn't pass us the right buffer when proxying. The buffer seems to contain client-to-server data and the buffer's limit isn't set. So instead this records the request content by overriding newProxyToServerRequestContent.
Foundation 4 has been obsolete since 2013, and these libraries trigger security alerts. We're also not using them to do much, just the modal dialogs, alert close buttons and collapsing the top-bar on mobile. This replaces the modals with the native HTML dialog element and implements the trivial click event handlers needed for hiding and showing things.
This means:
- Job profiles can be used without having to hunt for them among the regular jobs.
- The Groovy default profile can be selected as an alternative to Spring XML one.
- The default profiles can be customised by creating a job profile with the same name.
BdbMultipleWorkQueues.calculateInsertKey() silently clips precedence
values to 127 for queue ordering. This can cause unexpected behavior
for custom cost assignment policies that return values above 127.
Log a warning when clipping occurs and fix the misleading comment in
CrawlURI.holderCost that said "should not exceed 255" (the actual
limit is 127, not 255).
Fixes#502
Heritrix's Spring XML based job config format by design allows arbitrary code execution, so this doesn't realistically make things much safer, but I also don't see any reason we need external entity resolution enabled. I suppose there's a chance this could mitigate a generic automated attack that uses XXE but doesn't target Heritrix or Spring XML specifically.
Fixes#711
This adds context-sensitive autocompletion for bean classes, bean ids, property names, basic Spring XML tags and attributes. The bean completions use a new `/engine/beandoc` endpoint that serves the combination of all the `/META-INF/heritrix-beans.json` files generated at compile-time by the heritrix-docgen annotation processor.
After I added them to the Bean reference they seemed out of place with the DecideRules and RecordBuilders.
We had two classes called 'Browser' which is potentially confusing. It's also probably good to more clearly differentiate the ExtractLinks browser behavior from the Extractor processors.
Leslie Bellony