gojomo 647a0b3edf HER-1754 treat robots.txt non-response as 404 (optionally?) ...

* CrawlServer.java
    enable a specific kind of connect-lost (no response given) to be counted as 'deemed 404' on robots retries over a minimum
* FetchStatusCodes.java
    add a synthetic code: -404, S_DEEMED_NOT_FOUND
* FetchHTTP.java
    better distinguish CONNECT_FAILED and CONNECT_LOST
* DispositionProcessor.java
    treat CONNECT_LOST same as _FAILED; simplify a test

2010-04-02 03:55:12 +00:00

..

crawler

HER-1754 treat robots.txt non-response as 404 (optionally?)

2010-04-02 03:55:12 +00:00

extractor

[HER-1561] update license to Apache License 2.0

2009-11-10 21:03:27 +00:00

overview.html

…