The Browser processor can load a fetched page in a local web browser, record any requests the browser makes and run behaviors that interact with the page such as scrolling down and extracting links.
This differs from my previous attempt (ExtractorChrome) in a few ways:
- Uses the new WebDriver BiDi standard instead of the Chrome Devtools Protocol. The new protocol is mostly browser-agnostic, more consistent and hopefully more stable.
- Uses a MITM proxy instead of CDP request interception for recording sub-resources. That's partly because BiDi is still missing some key interception APIs. Even so in practice I found the proxy method loads pages faster and more reliably, likely because responses can be streamed incrementally, which helps a lot for large resources or server-sent events.
- Even when HTTP/2 is unavailable, the new FetchHTTP2 module does connection pooling which makes loading browser requests a lot faster. The original FetchHTTP opened a new connection for every request.
- The Browser processor can be configured with a list of behavior beans making it more customizable and extensible.
Obvious areas for future development:
- More Behavior beans: take screenshots, saveg the rendered DOM, run Browsertrix-compatible behavior scripts
- Support for remote WebDrivers (e.g. Selenium Server or cloud services)
This resolves some browser incompatibilities, allowing CodeMirror’s own find function to be re-enabled for reliable text search of content far outside the viewport.
The Jetty API has changed, which mostly affects test code.
Jetty now does a strict SNI host check which unfortunately causes it to
return "SNI error" for our existing ad-hoc certificates. For now, I've
disabled it to avoid breaking existing deployments but added a
--sni-host-check command-line option so you can re-enable it if
you've configured your own certificate appropriately.
littleproxy has not been updated since 2017 and has various out of date
dependencies. Since we're using jetty for HTTP server testing anyway, we
may as well use it as the proxy too.
Alex Osborne