mirror of
https://github.com/internetarchive/heritrix3.git
synced 2026-07-12 18:47:09 +00:00
Alex Osborne
1209b2d42f
This policy obeys robots.txt but ignores meta nofollow. We're increasingly frequently encountering sites which fail to be archived properly due to usage of this meta tag but where the robots.txt file contains reasonable rules. In many cases it seems to be unintentional, and we've even had questions from webmasters asking why archiving was failing when they were explicitly allowing us in robots.txt.