nlevitt 706869e586 Fix handling of server error on robots.txt - change to what appears to be the intended behavior ...

* DispositionProcessor.java
    innerProcess() - call server.updateRobots(curi) after special ignore-robots short-circuiting of retry cycle rather than before, so that updateRobots() deemed-not-found handling applies
* FetchHTTP.java
    call curi.setHttpMethod() right after creating the HttpMethod, instead of when processing the response; this is needed so that updateRobots() check of fetchType works properly (before this change, fetchType was "UNKNOWN" on server error)
* PreconditionEnforcer.java, CrawlServer.java
    fix typos

2011-08-02 23:28:20 +00:00

..

.settings

…

src

Fix handling of server error on robots.txt - change to what appears to be the intended behavior

2011-08-02 23:28:20 +00:00

.classpath

…

.project

…

pom.xml

Post 3.1.0-RC1

2011-07-26 03:42:02 +00:00