gojomo 647a0b3edf HER-1754 treat robots.txt non-response as 404 (optionally?) ...

* CrawlServer.java
    enable a specific kind of connect-lost (no response given) to be counted as 'deemed 404' on robots retries over a minimum
* FetchStatusCodes.java
    add a synthetic code: -404, S_DEEMED_NOT_FOUND
* FetchHTTP.java
    better distinguish CONNECT_FAILED and CONNECT_LOST
* DispositionProcessor.java
    treat CONNECT_LOST same as _FAILED; simplify a test

2010-04-02 03:55:12 +00:00

..

.settings

copy 'springy' branch to heritrix3

2009-05-11 22:56:36 +00:00

src

HER-1754 treat robots.txt non-response as 404 (optionally?)

2010-04-02 03:55:12 +00:00

.classpath

copy 'springy' branch to heritrix3

2009-05-11 22:56:36 +00:00

.project

copy 'springy' branch to heritrix3

2009-05-11 22:56:36 +00:00

pom.xml

post-3.0.0 release build

2009-12-05 23:48:38 +00:00