lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-07-16 04:27:09 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
4e8bda1a0757de7fd46485f0c94f8eadb241e1ec
heritrix3/engine/src
T
History
Alex Osborne 4e8bda1a07 ExtractorHTML: Add obeyRelNofollow option 
When enabled this option causes regular links annotated with rel=nofollow to not be extracted. This is useful for sites that use rel=nofollow to hint crawler traps.
2025-01-13 15:34:38 +09:00
..
design
…
main
ExtractorHTML: Add obeyRelNofollow option
2025-01-13 15:34:38 +09:00
test/java/org/archive
Add Groovy crawl configs
2024-12-01 00:14:42 +09:00
webapps/selftest
…
xsd/arc/1.0
…