nlevitt b898d55cea Certain invalid "Crawl-delay" values, e.g. "/", in robots.txt cause split() to return empty string, resulting in ArrayIndexOutOfBoundsException - reported by John Lekashman ...

* Robotstxt.java
    initializeFromReader() - catch ArrayIndexOutOfBoundsException and ignore -- matches existing handling of other invalid crawl-delay values (catch NumberFormatException and ignore)

2011-10-06 17:20:13 +00:00

..

java/org/archive/modules

Certain invalid "Crawl-delay" values, e.g. "/", in robots.txt cause split() to return empty string, resulting in ArrayIndexOutOfBoundsException - reported by John Lekashman

2011-10-06 17:20:13 +00:00

resources/org/archive/modules

…