mirror of
https://github.com/internetarchive/heritrix3.git
synced 2026-11-22 13:47:02 +00:00
Sites that return 200 for nonexistent URLs, combined with relative links, can generate an endless series of fake URLs (#729). The first time a server returns a successful response, a probe for a random two-segment path on it is scheduled. If the probe gets a 200, its content is fingerprinted, and later 200 responses from that server with the same fingerprint are annotated "soft404". The fingerprint is a hash of the content after removing echoes of the requested path and replacing tokens that contain digits, so nonces and timestamps don't prevent a match. Seeds and the root path are exempt, and a probe redirect that keeps the probe path is treated as the probe. With skipExtraction (the default) the rest of the fetch chain is skipped for matching pages, so neither the extractors nor the browser processor see them. The default profiles enable the detector with skipExtraction=false for now, so it only annotates and counts soft 404s until it has seen more real-world use. Per-server probe state is stored on CrawlServer.