mirror of
https://github.com/internetarchive/heritrix3.git
synced 2026-10-01 10:05:34 +00:00
Alex Osborne
4e8bda1a07
When enabled this option causes regular links annotated with rel=nofollow to not be extracted. This is useful for sites that use rel=nofollow to hint crawler traps.