nlevitt 2df2584573 Fix HER-1880 robots.txt less specific Allow overrides more specific Disallow ...

* RobotsDirectives.java
    - use plain ConcurrentSkipListSet<String> instead of PrefixSet, to
      maintain complete list of prefixes, so we can know the longest match
    - allows(String) - return true if longest matching Allow prefix is longer
      than or equal to longest matching Disallow prefix
* RobotstxtTest.java
    flip expected result of test of generic Allow against specific Disallow

2011-04-14 02:25:43 +00:00

..

canonicalize

tests cleanup

2010-11-06 01:26:20 +00:00

credential

Discard 'CredentialAvatar' -- no longer needed with Spring settings -- use Credential instances directly

2010-11-06 03:45:00 +00:00

deciderules

HER-1645 whois fetcher

2011-02-17 22:33:28 +00:00

extractor

[HER-1053] compressed HTTP fetch: "Accept-encoding: gzip"

2011-04-06 20:54:16 +00:00

fetcher

* FetchWhois.java

2011-04-07 17:17:48 +00:00

net

Fix HER-1880 robots.txt less specific Allow overrides more specific Disallow

2011-04-14 02:25:43 +00:00

recrawl

Different fix for HER-1872: additional property preloadSourceUrl. Use either

2011-03-02 01:24:28 +00:00

seeds

Experiment to assist large seed-loads

2010-11-05 23:27:43 +00:00

writer

improved comments about new naming conventions/default-template

2011-04-11 22:49:01 +00:00

CandidateChain.java

[HER-1605] Refactor parameterizable policies out of frontier

2009-10-05 23:41:59 +00:00

CoreAttributeConstants.java

Consolidate ModuleAttributeConstants into CoreAttributeConstants

2010-11-09 23:47:37 +00:00

CrawlMetadata.java

[HER-1803] H3: make it easier to change robots settings

2010-09-10 00:27:20 +00:00

CrawlURI.java

HER-1645 whois fetcher

2011-02-17 22:33:28 +00:00

DispositionChain.java

[HER-1801] H3: Unreleased lock causes deadlock when checkpointing

2010-09-03 02:14:41 +00:00

FetchChain.java

[HER-1605] Refactor parameterizable policies out of frontier

2009-10-05 23:41:59 +00:00

package-info.java

…

Processor.java

Discard 'CredentialAvatar' -- no longer needed with Spring settings -- use Credential instances directly

2010-11-06 03:45:00 +00:00

ProcessorChain.java

fix for HER-1744 Incorrect comparison of Strings in ProcessorChain.process() with "!=" operator

2010-02-16 19:13:43 +00:00

ProcessorTestBase.java

[HER-1561] update license to Apache License 2.0

2009-11-10 21:03:27 +00:00

ProcessResult.java

[HER-1605] Refactor parameterizable policies out of frontier

2009-10-05 23:41:59 +00:00

SchedulingConstants.java

…

ScriptedProcessor.java

[HER-1703] convert all maps that operator might want to change on the fly to concurrent maps

2009-11-07 03:07:51 +00:00

SimpleFileLoggerProvider.java

Fixes and improvements to to DecideRuleSequence/logToFile.

2011-01-07 02:09:31 +00:00