mirror of
https://github.com/internetarchive/heritrix3.git
synced 2026-09-10 07:45:42 +00:00
* RobotsDirectives.java
- use plain ConcurrentSkipListSet<String> instead of PrefixSet, to
maintain complete list of prefixes, so we can know the longest match
- allows(String) - return true if longest matching Allow prefix is longer
than or equal to longest matching Disallow prefix
* RobotstxtTest.java
flip expected result of test of generic Allow against specific Disallow