mirror of
https://github.com/internetarchive/heritrix3.git
synced 2026-07-16 04:27:09 +00:00
18e165f21e
* AbstractFrontier.java, WorkQueueFrontier.java, BdbFrontier.java
add 'futureUris' map of CrawlURIs to be reenqueued at specific future dates
the usual findEligibleUri also checks for newly-eligible rescheduled URIs
the frontier is no longer empty unless the future count is also 0
* CrawlURI.java
add rescheduleTime property
* ReschedulingProcessor.java
optional post-processor to set a CrawlURI's reschedulingTime
* CrawlStatSnapshot.java, CrawlJob.java
add futureUriCount to stats/uriTotalsReport