This means:
- Job profiles can be used without having to hunt for them among the regular jobs.
- The Groovy default profile can be selected as an alternative to Spring XML one.
- The default profiles can be customised by creating a job profile with the same name.
This means we can rely on the Java compiler to parse the source code instead of the old python library which no longer works as it only supported Java 8 source code. The new processor generates a META-INF/heritrix-beans.json file for each module at compile time which the sphinx beandoc plugin then reads.
My intent is to eventually use the heritrix-beans.json files to also implement autocomplete and context-sensitive help in the Heritrix config editor.
After I added them to the Bean reference they seemed out of place with the DecideRules and RecordBuilders.
We had two classes called 'Browser' which is potentially confusing. It's also probably good to more clearly differentiate the ExtractLinks browser behavior from the Extractor processors.
This option enables HTTP Basic authentication for the web interface instead of the default Digest authentication. This is useful when running Heritrix behind a reverse proxy that adds external authentication as typically they don't support Digest auth for the upstream server.
#641
Looks like the javalang parser doesn't handle some newer syntax.
For now just catch the errors. Long term we probably need to switch
to a different parser.
It seems to be failing because the placeholder is missing:
```
File "/home/docs/checkouts/readthedocs.org/user_builds/heritrix/envs/latest/lib/python3.12/site-packages/sphinx/ext/extlinks.py", line 103, in role
title = caption % part
~~~~~~~~^~~~~~
TypeError: not all arguments converted during string formatting
```
- instead of "borrowing" the configured proxy from
the fetchHttp bean, use proxy values defined via
global options, to avoid interference with other
jobs running in parallel (or at least make them
explicit).
The "fetchHttp" bean also uses these settings,
if no bean specific settings are used.
- remove the "enableDnsOverHttpResolves", and rely
on a non-empty "dnsOverHttpServer" value to signal
that DoH should be used.
- use the "DohResolver" from the dnsjava library
to make DoH lookups
- to enable and configure it, add two new
properties
* "enableDnsOverHttpResolves" (boolean)
* "dnsOverHttpServer" URL to the DoH Server
- as one use case for DoH is being located
behind a firewall, also support using a proxy
to access the DoH server; the proxy from
the FetchHTTP bean is reused in that case
Fixes#211
This enables us to generate documentation for more bean properties,
although a number of beans have javadoc on initializer code blocks which
makes it hard to access. This affects javadoc and IDE contextual
documentation too so should probably be fixed in the source code itself.
Alex Osborne