lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-07-13 11:07:12 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
4e8bda1a0757de7fd46485f0c94f8eadb241e1ec
heritrix3/modules
T
History
Alex Osborne 4e8bda1a07 ExtractorHTML: Add obeyRelNofollow option 
When enabled this option causes regular links annotated with rel=nofollow to not be extracted. This is useful for sites that use rel=nofollow to hint crawler traps.
2025-01-13 15:34:38 +09:00
..
src
ExtractorHTML: Add obeyRelNofollow option
2025-01-13 15:34:38 +09:00
pom.xml
Add Groovy crawl configs
2024-12-01 00:14:42 +09:00