Files
heritrix3/modules
nlevitt 2df2584573 Fix HER-1880 robots.txt less specific Allow overrides more specific Disallow
* RobotsDirectives.java
    - use plain ConcurrentSkipListSet<String> instead of PrefixSet, to
      maintain complete list of prefixes, so we can know the longest match
    - allows(String) - return true if longest matching Allow prefix is longer
      than or equal to longest matching Disallow prefix
* RobotstxtTest.java
    flip expected result of test of generic Allow against specific Disallow
2011-04-14 02:25:43 +00:00
..