mirror of
https://github.com/internetarchive/heritrix3.git
synced 2026-08-23 15:06:46 +00:00
Alex Osborne
cec6cfbb64
Detects responses produced by common bot-blocking services and adds a "botblock:service" annotation. This is intended for reporting and troubleshooting but may also be useful as a signal to stop crawling or to skip writing WARC records.
Heritrix Documentation
This directory contains structured documentation we're in the process of migrating and reformatting. It can be viewed online.
The main bulk of Heritrix's documentation currently lives on the Github wiki.