Files
heritrix3/engine
Alex Osborne d614ea3b34 Soft404Detector: detect soft 404s and optionally skip extraction
Sites that return 200 for nonexistent URLs, combined with relative
links, can generate an endless series of fake URLs (#729).

The first time a server returns a successful response, a probe for a
random two-segment path on it is scheduled. If the probe gets a 200,
its content is fingerprinted, and later 200 responses from that server
with the same fingerprint are annotated "soft404". The fingerprint is a
hash of the content after removing echoes of the requested path and
replacing tokens that contain digits, so nonces and timestamps don't
prevent a match. Seeds and the root path are exempt, and a probe
redirect that keeps the probe path is treated as the probe.

With skipExtraction (the default) the rest of the fetch chain is
skipped for matching pages, so neither the extractors nor the browser
processor see them. The default profiles enable the detector with
skipExtraction=false for now, so it only annotates and counts soft
404s until it has seen more real-world use.

Per-server probe state is stored on CrawlServer.
2026-11-05 16:12:17 +09:00
..