Files
heritrix3/modules
Alex Osborne cec6cfbb64 Add BotBlockDetector
Detects responses produced by common bot-blocking services and adds a "botblock:service" annotation. This is intended for reporting and troubleshooting but may also be useful as a signal to stop crawling or to skip writing WARC records.
2026-08-18 16:11:35 +09:00
..
2026-08-18 16:11:35 +09:00