After I added them to the Bean reference they seemed out of place with the DecideRules and RecordBuilders.
We had two classes called 'Browser' which is potentially confusing. It's also probably good to more clearly differentiate the ExtractLinks browser behavior from the Extractor processors.
This option enables HTTP Basic authentication for the web interface instead of the default Digest authentication. This is useful when running Heritrix behind a reverse proxy that adds external authentication as typically they don't support Digest auth for the upstream server.
#641
Looks like the javalang parser doesn't handle some newer syntax.
For now just catch the errors. Long term we probably need to switch
to a different parser.
It seems to be failing because the placeholder is missing:
```
File "/home/docs/checkouts/readthedocs.org/user_builds/heritrix/envs/latest/lib/python3.12/site-packages/sphinx/ext/extlinks.py", line 103, in role
title = caption % part
~~~~~~~~^~~~~~
TypeError: not all arguments converted during string formatting
```
- instead of "borrowing" the configured proxy from
the fetchHttp bean, use proxy values defined via
global options, to avoid interference with other
jobs running in parallel (or at least make them
explicit).
The "fetchHttp" bean also uses these settings,
if no bean specific settings are used.
- remove the "enableDnsOverHttpResolves", and rely
on a non-empty "dnsOverHttpServer" value to signal
that DoH should be used.
- use the "DohResolver" from the dnsjava library
to make DoH lookups
- to enable and configure it, add two new
properties
* "enableDnsOverHttpResolves" (boolean)
* "dnsOverHttpServer" URL to the DoH Server
- as one use case for DoH is being located
behind a firewall, also support using a proxy
to access the DoH server; the proxy from
the FetchHTTP bean is reused in that case
Fixes#211
This enables us to generate documentation for more bean properties,
although a number of beans have javadoc on initializer code blocks which
makes it hard to access. This affects javadoc and IDE contextual
documentation too so should probably be fixed in the source code itself.
This is meant to complement the javadoc by providing a reference
more suitable for users trying to configure crawls rather than
developers writing new modules.
The doc generation could still do with some improving and some of the
source javadoc comments need fixing up but this is already useful so I'm
committing what I have so far.
Alex Osborne