* commons/pom.xml, .classpath
update to BDB-JE 4.0.71
* BdbModule.java
(getStoredQueue) added
(getStoredMap) made more general
(openDatabase) all databases now 'managed'
* TempStoredSortedMap.java -> DisposableStoredSortedMap.java
renamed
* StatisticsTracker.java, **Report.java
use new name
* StoredQueue.java, StoredQueueTest.java
move to org.archive.bdb package
* BdbMultipleWorkQueues.java, PrefixFinderTest.java
update for BDB-JE 4 changes
* BdbFrontier.java
use getStoredQueue
drop unneeded custom serialization support
* WorkQueueFrontier.java
drop unneeded custom serialization support
* JobResource.java
suppress checkpoint-selector after terminate
* Checkpoint.java
adapt to work without direct reference to CheckpointService
* CheckpointService.java
configure Checkpoint without making it dependent on CheckpointService
UI support
* JobResource.java
enable 'checkpoint' button as appropriate
display active recovery-checkpoint or select bos of recovery-options when available
* CrawlJob.java
(getCheckpointService) accessor
* BdbFrontier.java
implement Checkpointable; save state via JSON or BDB; restore queues-of-queues on recovery
* WorkQueueFrontier.java
(deactivateQueue) (retireQueue) protected
* BdbUriUniqFilter.java
implement Checkpointable; save state via JSON or BDB; restore
* WorkQueue.java
make peekItem transient -- should not be held through checkpoint
make retired protected
* Frontier.java, AbstractFrontier.java
remove/update obsolete code
* CrawlController.java
remove older checkpoint-functionality passthrough
implement Checkpointable so behavior can vary during recovery
only trigger annnounce of seeds on 'cold' (non-recovery) start
* PersistLogProcessor.java
rotate logs on checkpoint
* CrawlerJournal.java, FrontierJournal.java
rotate for checkpoint support without full Checkpointable
stats & logging checkpoint support
* StatisticsTracker.java
implement Checkpointable; save small state in JSON, recycle BDB data when recovering
* CrawlerLoggerModule.java
implement Checkpointable; rotate logs on checkpoint
* GenerationFileHandler.java
avoid clobbering leftover logs from earlier futures via moveAside
* CrawlStatSnapshot.java
(sameProgressAs) test for lack-of-progress
* ObjectIdentityMemCache.java, TopNSet.java
offer deeper access to assist checkpoint/resume
processor-related bean checkpoint support:
* Processor.java
implement Checkpointable and BeanNameAware; methods to support JSON state-saving
* DispositionChain.java
implement Checkpointable and during-checkpoint disposition-lockout for state stability
processor-related bean checkpoint support:
* Processor.java
implement Checkpointable and BeanNameAware; methods to support JSON state-saving
* DispositionChain.java
implement Checkpointable and during-checkpoint disposition-lockout for state stability
* **Extractor**.java
pull common reporting & checkpointing up to Extractor.java
* WriterPoolProcessor.java
implement Checkpointable; freeze and roll writer-pool during checkpoint
* PersistLogProcessor.java
roll log on checkpoint
* TextSeedModule.java, SeedModule.java
eliminate obsolete (and probably unnecessary) checkpoint support
* (others)
adapt to above changes
core checkpointing system classes:
* Checkpointable.java
basic interface for checkpoint aware/capable beans
* Checkpoint.java
one Checkpoint, either in progress or recovery -- a name + storage directory
* CheckpointService.java
helper bean to enable, trigger checkpoints and recovery
* Checkpointer.java(*2), CheckpointInputStream.java, CheckpointRecovery.java DefaultCheckpointRecovery.java, RecoverAction.java
(deleted) no longer used
* IoUtils.java
remove class that confusingly-overlaps with commons IOUtils; move remaining methods to ArchiveUtils or archive's FileUtils
* ArchiveUtils.java, FileUtils.java
receive relocated methods; eliminated deprecated or unused methods
* JSONUtils.java
new class to collect common JSON actions
* Iteratorable.java
wrap Iterator as Iterable for foreach usage
* (many)
update to use alternate utils methods
* ServerCache.java
make abstract, implement getHostFor(UURI) and getServerFor(UURI)
* DefaultServerCache.java
extend ServerCache, remove getHostFor(UURI) and getServerFor(UURI)
* ServerCacheUtil.java
removed
* other classes
change usage of ServerCacheUtil to ServerCache
* BloomUriUniqFilter.java
make bloom size configurable via bean properties
change implementing class to BloomFilter64bit (given growing prevalance of 64bit JVMs and larger heaps)
* BloomUriUniqFilterTest.java
update test for property-set sizing
* BloomFilter64bit.java
improve comment
* CrawlJob.java
renamed isContainerOk() to hasApplicationContext()
increased granularity of isLaunchable() to offer launch button
* JobResource.java
more coarse check for hasApplicationContext() to disable build button
* DefaultServerCache.java
getHostFor(UURI): special case dns uris
* ServerCache.java
add getHostFor(UURI) to public interface
* ServerCacheUtil.java
remove now redundant wrapper method
* DispositionProcessor.java AbstractFrontier.java IPQueueAssignmentPolicy.java
BucketQueueAssignmentPolicy.java PreconditionEnforcer.java QuotaEnforcer.java
StatisticsTracker.java ServerCacheTest.java FetchHTTP.java
Kw3WriterProcessor.java WriterPoolProcessor.java
changed to use serverCache.getHostFor(UURI) (a few cases remain that use
getHostFor(String))
* ScopePlusOneDecideRule.java
removing, in favor of SurtPrefixedDecideRule's 'alsoCheckVia' option
* RecyclingFastBufferedOutputStream.java
use library by deriving as subclass (rather than c&p), now that class is available in an included library (fastutil)
* BdbModule.java
allow Database instance reuse if BdbConfig is identical instance
* PersistProcessor.java
convert shared BdbConfig for history db to static constant HISTORY_DB_CONFIG
* PersistOnlineProcessor.java
use HISTORY_DB_CONFIG, don't close possibly-shared DB on stop()
* PrecedenceLoader.java, PreloadedUriPrecedencePolicy.java
use HISTORY_DB_CONFIG
gojomo