Commit Graph
16 Commits
Author SHA1 Message Date
Alex Osborne 486dffbedf BrowserProcessor: allow per-browser concurrency, executable and options 2026-11-13 10:12:38 +09:00
Alex Osborne 0d1eae1082 Add missing processors to bean reference (#815) 2026-11-06 14:41:51 +09:00
Alex Osborne 9d79ef7372 Add ExtractorSVG and extract links from inline SVG
ExtractorSVG parses SVG with the JDK's StAX parser and extracts hrefs
from elements like <image>, <use> and <script> as embeds and from <a>
as links, along with <?xml-stylesheet?>, url() in presentation
attributes, <style>, <script>, animated hrefs and HTML in
<foreignObject>. Like browsers it requires the SVG namespace, decides
by the root element whether text/xml and application/xml responses
are SVG, and keeps the links found before any XML error. It's added
to the default profile before extractorHtml.

ExtractorHTML and JerichoExtractorHTML now treat hrefs on those inline
SVG elements as embeds rather than links, and extract url() from
presentation attributes like fill. ExtractorCSS now ignores
same-document references like url(#gradient).

Fixes #796
2026-11-05 16:17:28 +09:00
Alex Osborne cec6cfbb64 Add BotBlockDetector
Detects responses produced by common bot-blocking services and adds a "botblock:service" annotation. This is intended for reporting and troubleshooting but may also be useful as a signal to stop crawling or to skip writing WARC records.
2026-08-18 16:11:35 +09:00
Alex Osborne 34f390af42 Add ExtractorJson to bean reference #701 2025-12-11 13:40:08 +09:00
Alex Osborne 89e5ecc7d6 docs: replace javalang with an annotation processor
This means we can rely on the Java compiler to parse the source code instead of the old python library which no longer works as it only supported Java 8 source code. The new processor generates a META-INF/heritrix-beans.json file for each module at compile time which the sphinx beandoc plugin then reads.

My intent is to eventually use the heritrix-beans.json files to also implement autocomplete and context-sensitive help in the Heritrix config editor.
2025-10-28 16:26:19 +09:00
Alex Osborne 020ea34cb6 Bean reference: Add record builders under WARCWriterChainProcessor 2025-06-09 15:42:13 +09:00
Alex Osborne e68e4975c9 Rename to BrowserProcessor and -Behavior
After I added them to the Bean reference they seemed out of place with the DecideRules and RecordBuilders.

We had two classes called 'Browser' which is potentially confusing. It's also probably good to more clearly differentiate the ExtractLinks browser behavior from the Extractor processors.
2025-06-09 15:35:25 +09:00
Alex Osborne cf447f3bd9 Add Browser processor and behaviors to bean reference 2025-06-09 15:19:09 +09:00
Alex Osborne e09106cc15 Document HTTP/2 and HTTP/3 support 2025-05-09 21:03:38 +09:00
Alex Osborne 55a179c610 Remove ExtractorChrome
This never worked that well, is causing random test failures and I don't have any plans to continue developing it.
2024-08-08 07:21:46 +09:00
Alex Osborne f736bf2854 docs: Add DecideRuleSequence to bean-reference 2022-09-06 18:02:30 +09:00
Alex Osborne ea42630608 docs: Add decide rules to bean reference 2021-07-13 21:49:27 +09:00
Alex Osborne c54f18e3ff docs: Add most of the default config beans to the bean reference
Notably Decide Rules are still to be done.
2021-07-12 11:39:06 +09:00
Alex Osborne 4a5fdcfaf0 docs: Add remaining link extractor to the bean reference 2021-07-12 09:23:08 +09:00
Alex Osborne b3be9d01fe docs: Start a 'Bean Reference' document generated from the source code
This is meant to complement the javadoc by providing a reference
more suitable for users trying to configure crawls rather than
developers writing new modules.

The doc generation could still do with some improving and some of the
source javadoc comments need fixing up but this is already useful so I'm
committing what I have so far.
2021-07-11 15:56:37 +09:00