nlevitt 4991b295bc HER-1645 whois fetcher ...

* URIAuthorityBasedQueueAssignmentPolicy.java
    whois urls all go in special "whois" queue
* PreconditionEnforcer.java, CrawlURI.java
    move markPrerequisite() method from PreconditionEnforcer to CrawlURI
* profile-crawler-beans.cxml
    commented-out whois fetcher clause
* FetchStatusCodes.java
    new status codes for whois
* ServerCache.java
    getHostFor() - return "whois:" for whois urls, following dns: convention
* UURIFactory.java
    ugly hack for whois urls
* SchemeNotInSetDecideRule.java
    add whois to list of supported schemes
* WriterPoolProcessor.java
    shouldWrite() - we should write successful whois fetches
* WARCWriterProcessor.java
    write whois records
* FetchWhois.java
    the fetcher

2011-02-17 22:33:28 +00:00

.settings

commit project-specific eclipse settings

2009-11-11 02:05:30 +00:00

commons

HER-1645 whois fetcher

2011-02-17 22:33:28 +00:00

dist

[HER-1856] remove redundant profile-crawler-beans.cxml

2011-01-12 01:11:36 +00:00

engine

HER-1645 whois fetcher

2011-02-17 22:33:28 +00:00

modules

HER-1645 whois fetcher

2011-02-17 22:33:28 +00:00

.classpath

Update Guava library to r08

2011-02-16 01:15:13 +00:00

.project

Fix for [HER-1524] speculativeFixup in ExtractorJS should maintain URL scheme

2009-05-12 22:09:22 +00:00

java.hprof.txt

prep for 3.0.0 release

2009-12-05 09:28:19 +00:00

java.hprof.txt-cpusamples

prep for 3.0.0 release

2009-12-05 09:28:19 +00:00

java.hprof.txt-heapsites

prep for 3.0.0 release

2009-12-05 09:28:19 +00:00

pom.xml

[HER-1831] change version numbering convention to be less confusing

2010-09-30 19:48:35 +00:00

S

Description

No description provided

heritrixjavawarcwebcrawling

14 MiB

Languages

Java 93.3%

HTML 2.6%

Rich Text Format 1.1%

JavaScript 1%

FreeMarker 0.8%

Other 1%