Commit Graph
81 Commits
Author SHA1 Message Date
Alex Osborne 0d1eae1082 Add missing processors to bean reference (#815) 2026-11-06 14:41:51 +09:00
Alex Osborne 9d79ef7372 Add ExtractorSVG and extract links from inline SVG
ExtractorSVG parses SVG with the JDK's StAX parser and extracts hrefs
from elements like <image>, <use> and <script> as embeds and from <a>
as links, along with <?xml-stylesheet?>, url() in presentation
attributes, <style>, <script>, animated hrefs and HTML in
<foreignObject>. Like browsers it requires the SVG namespace, decides
by the root element whether text/xml and application/xml responses
are SVG, and keeps the links found before any XML error. It's added
to the default profile before extractorHtml.

ExtractorHTML and JerichoExtractorHTML now treat hrefs on those inline
SVG elements as embeds rather than links, and extract url() from
presentation attributes like fill. ExtractorCSS now ignores
same-document references like url(#gradient).

Fixes #796
2026-11-05 16:17:28 +09:00
Matt Van Horn a769651bf5 fix: give midfetch aborts their own fetch status code
Add S_MIDFETCH_ABORTED (-5005) and set it in FetchHTTP when
shouldFetchBodyRule rejects a response, in place of S_OUT_OF_SCOPE.
The code means something distinct from a scope rejection, so crawl logs
and reports can now tell the two apart. AbstractFrontier.isDisregarded()
treats it as disregarded, CrawlURI gives it a readable string and the
glossary lists it.
2026-11-02 06:01:06 -08:00
Alex Osborne e3f9b11394 docs: fix factual errors in user documentation (#787)
getting-started:
- Use a Java 17 path in the JAVA_HOME example (17 is the minimum)
- Link to the local Configuring Jobs page instead of the wiki
- Fix typos

operating:
- Correct --jobs-dir option name and explain its default
- Document --proxy-host, --proxy-port and --sni-host-check
- Fix option list formatting that broke rendering (-c, --web-auth)
- Remove contradictory note about credentials in heritrix_out.log
- crawl.log fetch duration is separated by '+', not ';'
- Renumber progress-statistics.log fields (field 3 was duplicated)
- Fix sourceTagSeeds property name in the XML example
- ActionDirectory is already in the default profile
- Use the correct frontier.recover.gz file name throughout
- The action directory is inside the job directory, not beside bin/

configuring-jobs:
- Fix maxDocumentsDownload, obeyRelNofollow and logger class names
- Replace removed/renamed decide rules with their current names
- Fix javadoc links for decide rules in subpackages and in engine
- Fix malformed useHTTP3 XML example and FTP/SFTP seed examples
- Replace dead Confluence glossary link
- Drop DoH caveats that only applied to Java 11 / DisableExplicitGC

api:
- Fix Create Job field list not rendering (missing blank line)
- Script engines: list Groovy/BeanShell; js and AppleScript are not
  available on supported Java versions
- Make JSON sample response consistent with the XML one

glossary:
- SHA-1 is used for content digests, not encryption
- Fix garbled Host entry (logical vs physical host)
- Fix SURT and SURT prefix examples
- Status -6 refers to DNS, not WHOIS; add -9, 1001, 2001, 2002
- Link status codes to RFC 9110 instead of RFC 2616
2026-10-13 11:34:43 +09:00
Alex Osborne cec6cfbb64 Add BotBlockDetector
Detects responses produced by common bot-blocking services and adds a "botblock:service" annotation. This is intended for reporting and troubleshooting but may also be useful as a signal to stop crawling or to skip writing WARC records.
2026-08-18 16:11:35 +09:00
Alex Osborne 3718480cd5 docs: Add Groovy versions of config examples
Uses sphinx-inline-tabs to render tabs offering the choice between XML and Groovy syntax for each example.
2026-05-03 12:07:01 +09:00
Alex Osborne b1e13c550b Add profile selector to job creation form
This means:

- Job profiles can be used without having to hunt for them among the regular jobs.
- The Groovy default profile can be selected as an alternative to Spring XML one.
- The default profiles can be customised by creating a job profile with the same name.
2026-04-27 17:40:37 +09:00
Erik Körner c1bedebef1 Add job deletion to API and UI 2026-01-05 14:19:17 +01:00
Erik Körner 76d3847f35 Update documentation 2025-12-17 18:31:16 +01:00
Alex Osborne 34f390af42 Add ExtractorJson to bean reference #701 2025-12-11 13:40:08 +09:00
Alex Osborne 89e5ecc7d6 docs: replace javalang with an annotation processor
This means we can rely on the Java compiler to parse the source code instead of the old python library which no longer works as it only supported Java 8 source code. The new processor generates a META-INF/heritrix-beans.json file for each module at compile time which the sphinx beandoc plugin then reads.

My intent is to eventually use the heritrix-beans.json files to also implement autocomplete and context-sensitive help in the Heritrix config editor.
2025-10-28 16:26:19 +09:00
Alex Osborne d484007f57 FetchHTTP2: Note that WARC doesn't specify the recording of HTTP/2 and 3
iipc/warc-specifications#106
2025-09-06 10:29:24 +09:00
Alex Osborne 020ea34cb6 Bean reference: Add record builders under WARCWriterChainProcessor 2025-06-09 15:42:13 +09:00
Alex Osborne e68e4975c9 Rename to BrowserProcessor and -Behavior
After I added them to the Bean reference they seemed out of place with the DecideRules and RecordBuilders.

We had two classes called 'Browser' which is potentially confusing. It's also probably good to more clearly differentiate the ExtractLinks browser behavior from the Extractor processors.
2025-06-09 15:35:25 +09:00
Alex Osborne cf447f3bd9 Add Browser processor and behaviors to bean reference 2025-06-09 15:19:09 +09:00
Alex Osborne c7b7ee1842 Support * and $ wildcards in robots.txt 2025-06-07 10:37:11 +09:00
Alex Osborne eaf72b5ce2 Add --web-auth basic command-line option
This option enables HTTP Basic authentication for the web interface instead of the default Digest authentication. This is useful when running Heritrix behind a reverse proxy that adds external authentication as typically they don't support Digest auth for the upstream server.

#641
2025-05-25 00:29:14 +09:00
Alex Osborne 51347f7698 docs: Use javalang17 fork
Fixes some parse errors but not all of them
2025-05-13 14:23:08 +09:00
Alex Osborne 7db91b53c6 docs: HTTP/2: correct code quotes instead of italics 2025-05-13 14:19:35 +09:00
Alex Osborne e1ac68f533 docs: Correct bean-example path for FetchHTTP2 2025-05-13 14:16:36 +09:00
Alex Osborne fee534c3db docs: Get version from pom.xml 2025-05-13 14:15:20 +09:00
Alex Osborne 81ac46be45 docs: Catch parse errors and continue generating
Looks like the javalang parser doesn't handle some newer syntax.
For now just catch the errors. Long term we probably need to switch
to a different parser.
2025-05-13 14:09:09 +09:00
Alex Osborne e09106cc15 Document HTTP/2 and HTTP/3 support 2025-05-09 21:03:38 +09:00
Alex Osborne 3f323bb0da Remove pillow from docs requirements.txt
It's triggering dependabot alerts and it appears as if we don't
actually use it.
2025-03-21 13:23:03 +09:00
Alex Osborne 22f8f7f754 Restore accidentally removed word 2025-02-14 11:29:22 +09:00
Alex Osborne 88854d8dfd Add caption to deciderule extlinks
It seems to be failing because the placeholder is missing:

```
  File "/home/docs/checkouts/readthedocs.org/user_builds/heritrix/envs/latest/lib/python3.12/site-packages/sphinx/ext/extlinks.py", line 103, in role
    title = caption % part
            ~~~~~~~~^~~~~~
TypeError: not all arguments converted during string formatting
```
2025-02-14 11:23:02 +09:00
Alex Osborne 71c2fdb618 Add missing readthedocs dependencies
https://blog.readthedocs.com/defaulting-latest-build-tools/
2025-02-14 11:15:35 +09:00
Alex Osborne 4093871de1 Merge pull request #626 from internetarchive/checkpoint-on-shutdown
Add checkpoint on shutdown and --checkpoint CLI option
2024-11-28 16:43:45 +09:00
Alex Osborne ec689dbce3 Document checkpointOnShutdown and --checkpoint options in the operating guide 2024-11-24 14:35:52 +09:00
Alex Osborne 585fed8575 Add --checkpoint command-line option
This makes it possible to select a checkpoint from the command-line when using the --run-job option.
2024-11-24 13:40:33 +09:00
Alex Osborne 055f4f9f22 Require Java 17 minimum 2024-11-20 15:31:36 +09:00
Alex Osborne 3d1b7e37ca Remove log4j 2024-10-17 19:25:53 +09:00
Alex Osborne 55a179c610 Remove ExtractorChrome
This never worked that well, is causing random test failures and I don't have any plans to continue developing it.
2024-08-08 07:21:46 +09:00
Christian Clauss c60af1eba2 Fix typo 2022-10-24 15:06:21 +09:00
Alex Osborne b328ded92e docs: Add 'Checkpointing' section to operating guide
Fixes #500 #506
2022-09-30 15:31:21 +09:00
Alex Osborne f736bf2854 docs: Add DecideRuleSequence to bean-reference 2022-09-06 18:02:30 +09:00
Alex Osborne c1ba694add Document robotsTxtOnly robots policy in config guide 2022-07-07 16:18:39 +09:00
Alex Osborne 09f75e389a docs/glossary: Add manifest (M) to discovery path 2022-06-22 15:44:44 +09:00
Alex Osborne a35cd44a73 Docs: Add section to config guide about http and form credentials
Based on https://github.com/internetarchive/heritrix3/wiki/Credentials
2022-05-24 12:01:56 +09:00
Clemens Klein-Robbenhaar f5c83ce368 Add support for DNS-over-HTTPS lookups
- docs: lets see if we can set a link
  to another chapter
2022-04-09 23:32:55 +02:00
Clemens Klein-Robbenhaar fdde6d87a0 Add support for DNS-over-HTTPS lookups
- instead of "borrowing" the configured proxy from
  the fetchHttp bean, use proxy values defined via
  global options, to avoid interference with other
  jobs running in parallel (or at least make them
  explicit).
  The "fetchHttp" bean also uses these settings,
  if no bean specific settings are used.
- remove the "enableDnsOverHttpResolves", and rely
  on a non-empty "dnsOverHttpServer" value to signal
  that DoH should be used.
2022-04-09 23:04:37 +02:00
Clemens Klein-Robbenhaar 9648c98c77 [misc] rst formatting 2022-04-08 12:14:11 +02:00
Clemens Klein-Robbenhaar 2144abb2cb Add support for DNS-over-HTTPS lookups
- use the "DohResolver" from the dnsjava library
  to make DoH lookups
- to enable and configure it, add two new
  properties
  * "enableDnsOverHttpResolves" (boolean)
  * "dnsOverHttpServer" URL to the DoH Server
- as one use case for DoH is being located
  behind a firewall, also support using a proxy
  to access the DoH server; the proxy from
  the FetchHTTP bean is reused in that case

Fixes #211
2022-04-08 11:48:52 +02:00
Jamie Hoyle dd01689ace FetchHTTP: SOCKS5 proxy support
Closes #454
2022-03-28 10:24:07 +09:00
Oliver Schihin c4edadf1a8 Configuration guide: clarify usage of ContentTypeMatchesRegexDecideRule 2021-12-21 19:10:22 +09:00
Erik Körner 3a5b758d5e Docs (temp) 2021-12-10 16:39:17 +01:00
Erik Körner 17c9694039 Add Dockerfile 2021-12-10 15:35:03 +01:00
Alex Osborne d11d02eeac docs: Add 'Loading Cookies' section to config guide 2021-11-29 12:11:54 +09:00
Alex Osborne 780abb0b1d docs: Update AdoptOpenJDK link as it has been rebranded Adoptium
https://blog.adoptopenjdk.net/2021/08/goodbye-adoptopenjdk-hello-adoptium/
2021-09-11 07:53:18 +09:00
Alex Osborne ea42630608 docs: Add decide rules to bean reference 2021-07-13 21:49:27 +09:00