* WorkQueueFrontier.java
add snoozedOverflow map of (wakeTime)->(DelayedWorkQueue) for snoozes over a capped size
maintain in parallel to snoozedClassQueues DelayQueue, maintain atomic size count
in standard report, presort snoozedClassQueues before displaying
* DelayedWorkQueue.java
move to top-level so serialization doesn't pull out whole frontier
* BdbFrontier.java
initialize snoozedOverflow
* BdbMultipleWorkQueues.java
improve error message seen on problematic checkpoint-recovery
* EngineResource.java
give add-job options their own header, reorder/reword for clarity
* JobResource.java
return to always showing 'checkpoint' button, disable if necessary, so main control buttons aren't appearing/disappearing or moving
==========
2009-12-03 01:36:16.726 SEVERE thread-10 com.noelios.restlet.http.HttpServerConverter.commit() An exception occured writing the response entity
java.io.IOException: Stream closed
at sun.nio.cs.StreamEncoder.ensureOpen(StreamEncoder.java:26)
[...]
2009-12-03 01:36:16.733::WARN: Committed before 500 An exception occured writing the response entity
2009-12-03 01:36:16.735 WARNING thread-10 com.noelios.restlet.http.HttpServerHelper.handle() Error while handling an HTTP server call:
==========
* JobResource.java
writeHtml(Writer) - do not close writer
* CrawlController.java
send event of crawl-end before appCtx.stop()
* StatisticsTracker.java
force final progressStatisticsEvent and dumpReports at crawlEnded() before stop()
add force-dump of reports (so dumpReports never reuses prior reports)
* CrawledBytesHistotable.java
add getTotalBytes(), a total of just the byte-counts (getTotal() includes URI counts)
* CrawlStatSnapshot.java
use getTotalBytes()
offer html
* XmlMarshaller.java
new class to write xml based on nested map structure
* BeanBrowseResource.java ScriptResource.java EngineResource.java
JobResource.java JobRelatedResource.java
support xml representation
* WorkQueueFrontier.java PrecedenceProvider.java WorkQueue.java ToeThread.java
ToePool.java CrawlJob.java Engine.java Frontier.java CrawlController.java
StatisticsTracker.java CrawledBytesHistotable.java FetchStats.java
ProcessorChain.java CrawlURI.java Reporter.java
offer same data that is returned in the single line reports in a map
suitable for use with XmlMarshaller
* FileUtils.java
new method tryToCanonicalize(File)
* AbstractFrontier.java, WorkQueueFrontier.java, BdbFrontier.java
add 'futureUris' map of CrawlURIs to be reenqueued at specific future dates
the usual findEligibleUri also checks for newly-eligible rescheduled URIs
the frontier is no longer empty unless the future count is also 0
* CrawlURI.java
add rescheduleTime property
* ReschedulingProcessor.java
optional post-processor to set a CrawlURI's reschedulingTime
* CrawlStatSnapshot.java, CrawlJob.java
add futureUriCount to stats/uriTotalsReport
* commons/pom.xml, .classpath
update to BDB-JE 4.0.71
* BdbModule.java
(getStoredQueue) added
(getStoredMap) made more general
(openDatabase) all databases now 'managed'
* TempStoredSortedMap.java -> DisposableStoredSortedMap.java
renamed
* StatisticsTracker.java, **Report.java
use new name
* StoredQueue.java, StoredQueueTest.java
move to org.archive.bdb package
* BdbMultipleWorkQueues.java, PrefixFinderTest.java
update for BDB-JE 4 changes
* BdbFrontier.java
use getStoredQueue
drop unneeded custom serialization support
* WorkQueueFrontier.java
drop unneeded custom serialization support
* JobResource.java
suppress checkpoint-selector after terminate
* Checkpoint.java
adapt to work without direct reference to CheckpointService
* CheckpointService.java
configure Checkpoint without making it dependent on CheckpointService
UI support
* JobResource.java
enable 'checkpoint' button as appropriate
display active recovery-checkpoint or select bos of recovery-options when available
* CrawlJob.java
(getCheckpointService) accessor
* BdbFrontier.java
implement Checkpointable; save state via JSON or BDB; restore queues-of-queues on recovery
* WorkQueueFrontier.java
(deactivateQueue) (retireQueue) protected
* BdbUriUniqFilter.java
implement Checkpointable; save state via JSON or BDB; restore
* WorkQueue.java
make peekItem transient -- should not be held through checkpoint
make retired protected
* Frontier.java, AbstractFrontier.java
remove/update obsolete code
* CrawlController.java
remove older checkpoint-functionality passthrough
implement Checkpointable so behavior can vary during recovery
only trigger annnounce of seeds on 'cold' (non-recovery) start
* PersistLogProcessor.java
rotate logs on checkpoint
* CrawlerJournal.java, FrontierJournal.java
rotate for checkpoint support without full Checkpointable
stats & logging checkpoint support
* StatisticsTracker.java
implement Checkpointable; save small state in JSON, recycle BDB data when recovering
* CrawlerLoggerModule.java
implement Checkpointable; rotate logs on checkpoint
* GenerationFileHandler.java
avoid clobbering leftover logs from earlier futures via moveAside
* CrawlStatSnapshot.java
(sameProgressAs) test for lack-of-progress
* ObjectIdentityMemCache.java, TopNSet.java
offer deeper access to assist checkpoint/resume
core checkpointing system classes:
* Checkpointable.java
basic interface for checkpoint aware/capable beans
* Checkpoint.java
one Checkpoint, either in progress or recovery -- a name + storage directory
* CheckpointService.java
helper bean to enable, trigger checkpoints and recovery
* Checkpointer.java(*2), CheckpointInputStream.java, CheckpointRecovery.java DefaultCheckpointRecovery.java, RecoverAction.java
(deleted) no longer used
* IoUtils.java
remove class that confusingly-overlaps with commons IOUtils; move remaining methods to ArchiveUtils or archive's FileUtils
* ArchiveUtils.java, FileUtils.java
receive relocated methods; eliminated deprecated or unused methods
* JSONUtils.java
new class to collect common JSON actions
* Iteratorable.java
wrap Iterator as Iterable for foreach usage
* (many)
update to use alternate utils methods
* ServerCache.java
make abstract, implement getHostFor(UURI) and getServerFor(UURI)
* DefaultServerCache.java
extend ServerCache, remove getHostFor(UURI) and getServerFor(UURI)
* ServerCacheUtil.java
removed
* other classes
change usage of ServerCacheUtil to ServerCache
* BloomUriUniqFilter.java
make bloom size configurable via bean properties
change implementing class to BloomFilter64bit (given growing prevalance of 64bit JVMs and larger heaps)
* BloomUriUniqFilterTest.java
update test for property-set sizing
* BloomFilter64bit.java
improve comment
* CrawlJob.java
renamed isContainerOk() to hasApplicationContext()
increased granularity of isLaunchable() to offer launch button
* JobResource.java
more coarse check for hasApplicationContext() to disable build button
* DefaultServerCache.java
getHostFor(UURI): special case dns uris
* ServerCache.java
add getHostFor(UURI) to public interface
* ServerCacheUtil.java
remove now redundant wrapper method
* DispositionProcessor.java AbstractFrontier.java IPQueueAssignmentPolicy.java
BucketQueueAssignmentPolicy.java PreconditionEnforcer.java QuotaEnforcer.java
StatisticsTracker.java ServerCacheTest.java FetchHTTP.java
Kw3WriterProcessor.java WriterPoolProcessor.java
changed to use serverCache.getHostFor(UURI) (a few cases remain that use
getHostFor(String))
* BdbModule.java
allow Database instance reuse if BdbConfig is identical instance
* PersistProcessor.java
convert shared BdbConfig for history db to static constant HISTORY_DB_CONFIG
* PersistOnlineProcessor.java
use HISTORY_DB_CONFIG, don't close possibly-shared DB on stop()
* PrecedenceLoader.java, PreloadedUriPrecedencePolicy.java
use HISTORY_DB_CONFIG
gojomo