gojomo 18e165f21e [HER-1714] fixed-interval rescheduling ...

* AbstractFrontier.java, WorkQueueFrontier.java, BdbFrontier.java
    add 'futureUris' map of CrawlURIs to be reenqueued at specific future dates
    the usual findEligibleUri also checks for newly-eligible rescheduled URIs
    the frontier is no longer empty unless the future count is also 0
* CrawlURI.java
    add rescheduleTime property
* ReschedulingProcessor.java
    optional post-processor to set a CrawlURI's reschedulingTime
* CrawlStatSnapshot.java, CrawlJob.java
    add futureUriCount to stats/uriTotalsReport

2009-11-23 08:47:14 +00:00

.settings

commit project-specific eclipse settings

2009-11-11 02:05:30 +00:00

commons

BDBJE updates, refactoring

2009-11-23 08:18:41 +00:00

dist

[HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls

2009-11-20 18:35:13 +00:00

engine

[HER-1714] fixed-interval rescheduling

2009-11-23 08:47:14 +00:00

modules

[HER-1714] fixed-interval rescheduling

2009-11-23 08:47:14 +00:00

.classpath

BDBJE updates, refactoring

2009-11-23 04:41:32 +00:00

.project

Fix for [HER-1524] speculativeFixup in ExtractorJS should maintain URL scheme

2009-05-12 22:09:22 +00:00

pom.xml

Finishing up [HER-1561]

2009-11-11 03:23:09 +00:00

S

Description

No description provided

heritrixjavawarcwebcrawling

14 MiB

Languages

Java 93.3%

HTML 2.6%

Rich Text Format 1.1%

JavaScript 1%

FreeMarker 0.8%

Other 1%