nlevitt b898d55cea Certain invalid "Crawl-delay" values, e.g. "/", in robots.txt cause split() to return empty string, resulting in ArrayIndexOutOfBoundsException - reported by John Lekashman ...

* Robotstxt.java
    initializeFromReader() - catch ArrayIndexOutOfBoundsException and ignore -- matches existing handling of other invalid crawl-delay values (catch NumberFormatException and ignore)

2011-10-06 17:20:13 +00:00

.settings

commit project-specific eclipse settings

2009-11-11 02:05:30 +00:00

commons

"Fix" for HER-1951 ExtractorJS produces many false positives

2011-09-30 00:48:00 +00:00

dist

Fix HER-1935 Many calls to File.mkdirs() and other file/dir methods don't check the return value. Also fix bug where pointless empty directories were created in scratch dir.

2011-09-12 20:12:10 +00:00

engine

Fix HER-1955 Some annoying interaction between new creation of latest link and disk full java bean

2011-09-30 20:07:15 +00:00

modules

Certain invalid "Crawl-delay" values, e.g. "/", in robots.txt cause split() to return empty string, resulting in ArrayIndexOutOfBoundsException - reported by John Lekashman

2011-10-06 17:20:13 +00:00

.classpath

Various little cleanups including several discovered by Aaron using an automated tool.

2011-09-12 02:12:44 +00:00

.project

Fix for [HER-1524] speculativeFixup in ExtractorJS should maintain URL scheme

2009-05-12 22:09:22 +00:00

java.hprof.txt

prep for 3.0.0 release

2009-12-05 09:28:19 +00:00

java.hprof.txt-cpusamples

prep for 3.0.0 release

2009-12-05 09:28:19 +00:00

java.hprof.txt-heapsites

prep for 3.0.0 release

2009-12-05 09:28:19 +00:00

pom.xml

* **/pom.xml

2011-08-05 17:46:57 +00:00

S

Description

No description provided

heritrixjavawarcwebcrawling

15 MiB

Languages

Java 93.3%

HTML 2.6%

Rich Text Format 1.1%

JavaScript 1%

FreeMarker 0.8%

Other 1%