heritrix3

mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-24 23:46:42 +00:00

T

nlevitt 2df2584573 Fix HER-1880 robots.txt less specific Allow overrides more specific Disallow

* RobotsDirectives.java
    - use plain ConcurrentSkipListSet<String> instead of PrefixSet, to
      maintain complete list of prefixes, so we can know the longest match
    - allows(String) - return true if longest matching Allow prefix is longer
      than or equal to longest matching Disallow prefix
* RobotstxtTest.java
    flip expected result of test of generic Allow against specific Disallow

2011-04-14 02:25:43 +00:00

.settings

commit project-specific eclipse settings

2009-11-11 02:05:30 +00:00

commons

[HER-1821] H3: implement hard-link-based BDB/state-dir checkpointing

2011-04-12 00:09:51 +00:00

dist

[HER-1856] remove redundant profile-crawler-beans.cxml

2011-01-12 01:11:36 +00:00

engine

* XmlMarshaller.java