ExtractorSVG parses SVG with the JDK's StAX parser and extracts hrefs
from elements like <image>, <use> and <script> as embeds and from <a>
as links, along with <?xml-stylesheet?>, url() in presentation
attributes, <style>, <script>, animated hrefs and HTML in
<foreignObject>. Like browsers it requires the SVG namespace, decides
by the root element whether text/xml and application/xml responses
are SVG, and keeps the links found before any XML error. It's added
to the default profile before extractorHtml.
ExtractorHTML and JerichoExtractorHTML now treat hrefs on those inline
SVG elements as embeds rather than links, and extract url() from
presentation attributes like fill. ExtractorCSS now ignores
same-document references like url(#gradient).
Fixes#796
Add S_MIDFETCH_ABORTED (-5005) and set it in FetchHTTP when
shouldFetchBodyRule rejects a response, in place of S_OUT_OF_SCOPE.
The code means something distinct from a scope rejection, so crawl logs
and reports can now tell the two apart. AbstractFrontier.isDisregarded()
treats it as disregarded, CrawlURI gives it a readable string and the
glossary lists it.
getting-started:
- Use a Java 17 path in the JAVA_HOME example (17 is the minimum)
- Link to the local Configuring Jobs page instead of the wiki
- Fix typos
operating:
- Correct --jobs-dir option name and explain its default
- Document --proxy-host, --proxy-port and --sni-host-check
- Fix option list formatting that broke rendering (-c, --web-auth)
- Remove contradictory note about credentials in heritrix_out.log
- crawl.log fetch duration is separated by '+', not ';'
- Renumber progress-statistics.log fields (field 3 was duplicated)
- Fix sourceTagSeeds property name in the XML example
- ActionDirectory is already in the default profile
- Use the correct frontier.recover.gz file name throughout
- The action directory is inside the job directory, not beside bin/
configuring-jobs:
- Fix maxDocumentsDownload, obeyRelNofollow and logger class names
- Replace removed/renamed decide rules with their current names
- Fix javadoc links for decide rules in subpackages and in engine
- Fix malformed useHTTP3 XML example and FTP/SFTP seed examples
- Replace dead Confluence glossary link
- Drop DoH caveats that only applied to Java 11 / DisableExplicitGC
api:
- Fix Create Job field list not rendering (missing blank line)
- Script engines: list Groovy/BeanShell; js and AppleScript are not
available on supported Java versions
- Make JSON sample response consistent with the XML one
glossary:
- SHA-1 is used for content digests, not encryption
- Fix garbled Host entry (logical vs physical host)
- Fix SURT and SURT prefix examples
- Status -6 refers to DNS, not WHOIS; add -9, 1001, 2001, 2002
- Link status codes to RFC 9110 instead of RFC 2616
Detects responses produced by common bot-blocking services and adds a "botblock:service" annotation. This is intended for reporting and troubleshooting but may also be useful as a signal to stop crawling or to skip writing WARC records.
This means:
- Job profiles can be used without having to hunt for them among the regular jobs.
- The Groovy default profile can be selected as an alternative to Spring XML one.
- The default profiles can be customised by creating a job profile with the same name.
This means we can rely on the Java compiler to parse the source code instead of the old python library which no longer works as it only supported Java 8 source code. The new processor generates a META-INF/heritrix-beans.json file for each module at compile time which the sphinx beandoc plugin then reads.
My intent is to eventually use the heritrix-beans.json files to also implement autocomplete and context-sensitive help in the Heritrix config editor.
After I added them to the Bean reference they seemed out of place with the DecideRules and RecordBuilders.
We had two classes called 'Browser' which is potentially confusing. It's also probably good to more clearly differentiate the ExtractLinks browser behavior from the Extractor processors.
This option enables HTTP Basic authentication for the web interface instead of the default Digest authentication. This is useful when running Heritrix behind a reverse proxy that adds external authentication as typically they don't support Digest auth for the upstream server.
#641
Looks like the javalang parser doesn't handle some newer syntax.
For now just catch the errors. Long term we probably need to switch
to a different parser.
It seems to be failing because the placeholder is missing:
```
File "/home/docs/checkouts/readthedocs.org/user_builds/heritrix/envs/latest/lib/python3.12/site-packages/sphinx/ext/extlinks.py", line 103, in role
title = caption % part
~~~~~~~~^~~~~~
TypeError: not all arguments converted during string formatting
```
- instead of "borrowing" the configured proxy from
the fetchHttp bean, use proxy values defined via
global options, to avoid interference with other
jobs running in parallel (or at least make them
explicit).
The "fetchHttp" bean also uses these settings,
if no bean specific settings are used.
- remove the "enableDnsOverHttpResolves", and rely
on a non-empty "dnsOverHttpServer" value to signal
that DoH should be used.
- use the "DohResolver" from the dnsjava library
to make DoH lookups
- to enable and configure it, add two new
properties
* "enableDnsOverHttpResolves" (boolean)
* "dnsOverHttpServer" URL to the DoH Server
- as one use case for DoH is being located
behind a firewall, also support using a proxy
to access the DoH server; the proxy from
the FetchHTTP bean is reused in that case
Fixes#211
This enables us to generate documentation for more bean properties,
although a number of beans have javadoc on initializer code blocks which
makes it hard to access. This affects javadoc and IDE contextual
documentation too so should probably be fixed in the source code itself.