* UURIFactoryTest.java
specify stirng literal with escape-encoding, rather than as raw UTF-8
* **/pom.xml
include 'project.build.sourceEncoding' property to suppress platform-encoding warnings
(unfortunately, neither this nor similar 'maven.compile.encoding' property actually casues maven build to interpret Java source as UTF-8)
offer html
* XmlMarshaller.java
new class to write xml based on nested map structure
* BeanBrowseResource.java ScriptResource.java EngineResource.java
JobResource.java JobRelatedResource.java
support xml representation
* WorkQueueFrontier.java PrecedenceProvider.java WorkQueue.java ToeThread.java
ToePool.java CrawlJob.java Engine.java Frontier.java CrawlController.java
StatisticsTracker.java CrawledBytesHistotable.java FetchStats.java
ProcessorChain.java CrawlURI.java Reporter.java
offer same data that is returned in the single line reports in a map
suitable for use with XmlMarshaller
* FileUtils.java
new method tryToCanonicalize(File)
* AbstractFrontier.java, WorkQueueFrontier.java, BdbFrontier.java
add 'futureUris' map of CrawlURIs to be reenqueued at specific future dates
the usual findEligibleUri also checks for newly-eligible rescheduled URIs
the frontier is no longer empty unless the future count is also 0
* CrawlURI.java
add rescheduleTime property
* ReschedulingProcessor.java
optional post-processor to set a CrawlURI's reschedulingTime
* CrawlStatSnapshot.java, CrawlJob.java
add futureUriCount to stats/uriTotalsReport
* commons/pom.xml, .classpath
update to BDB-JE 4.0.71
* BdbModule.java
(getStoredQueue) added
(getStoredMap) made more general
(openDatabase) all databases now 'managed'
* TempStoredSortedMap.java -> DisposableStoredSortedMap.java
renamed
* StatisticsTracker.java, **Report.java
use new name
* StoredQueue.java, StoredQueueTest.java
move to org.archive.bdb package
* BdbMultipleWorkQueues.java, PrefixFinderTest.java
update for BDB-JE 4 changes
* BdbFrontier.java
use getStoredQueue
drop unneeded custom serialization support
* WorkQueueFrontier.java
drop unneeded custom serialization support
* JobResource.java
suppress checkpoint-selector after terminate
* Checkpoint.java
adapt to work without direct reference to CheckpointService
* CheckpointService.java
configure Checkpoint without making it dependent on CheckpointService
UI support
* JobResource.java
enable 'checkpoint' button as appropriate
display active recovery-checkpoint or select bos of recovery-options when available
* CrawlJob.java
(getCheckpointService) accessor
* BdbFrontier.java
implement Checkpointable; save state via JSON or BDB; restore queues-of-queues on recovery
* WorkQueueFrontier.java
(deactivateQueue) (retireQueue) protected
* BdbUriUniqFilter.java
implement Checkpointable; save state via JSON or BDB; restore
* WorkQueue.java
make peekItem transient -- should not be held through checkpoint
make retired protected
* Frontier.java, AbstractFrontier.java
remove/update obsolete code
* CrawlController.java
remove older checkpoint-functionality passthrough
implement Checkpointable so behavior can vary during recovery
only trigger annnounce of seeds on 'cold' (non-recovery) start
* PersistLogProcessor.java
rotate logs on checkpoint
* CrawlerJournal.java, FrontierJournal.java
rotate for checkpoint support without full Checkpointable
stats & logging checkpoint support
* StatisticsTracker.java
implement Checkpointable; save small state in JSON, recycle BDB data when recovering
* CrawlerLoggerModule.java
implement Checkpointable; rotate logs on checkpoint
* GenerationFileHandler.java
avoid clobbering leftover logs from earlier futures via moveAside
* CrawlStatSnapshot.java
(sameProgressAs) test for lack-of-progress
* ObjectIdentityMemCache.java, TopNSet.java
offer deeper access to assist checkpoint/resume
processor-related bean checkpoint support:
* Processor.java
implement Checkpointable and BeanNameAware; methods to support JSON state-saving
* DispositionChain.java
implement Checkpointable and during-checkpoint disposition-lockout for state stability
processor-related bean checkpoint support:
* Processor.java
implement Checkpointable and BeanNameAware; methods to support JSON state-saving
* DispositionChain.java
implement Checkpointable and during-checkpoint disposition-lockout for state stability
* **Extractor**.java
pull common reporting & checkpointing up to Extractor.java
* WriterPoolProcessor.java
implement Checkpointable; freeze and roll writer-pool during checkpoint
* PersistLogProcessor.java
roll log on checkpoint
* TextSeedModule.java, SeedModule.java
eliminate obsolete (and probably unnecessary) checkpoint support
* (others)
adapt to above changes
core checkpointing system classes:
* Checkpointable.java
basic interface for checkpoint aware/capable beans
* Checkpoint.java
one Checkpoint, either in progress or recovery -- a name + storage directory
* CheckpointService.java
helper bean to enable, trigger checkpoints and recovery
* Checkpointer.java(*2), CheckpointInputStream.java, CheckpointRecovery.java DefaultCheckpointRecovery.java, RecoverAction.java
(deleted) no longer used
* IoUtils.java
remove class that confusingly-overlaps with commons IOUtils; move remaining methods to ArchiveUtils or archive's FileUtils
* ArchiveUtils.java, FileUtils.java
receive relocated methods; eliminated deprecated or unused methods
* JSONUtils.java
new class to collect common JSON actions
* Iteratorable.java
wrap Iterator as Iterable for foreach usage
* (many)
update to use alternate utils methods
* ServerCache.java
make abstract, implement getHostFor(UURI) and getServerFor(UURI)
* DefaultServerCache.java
extend ServerCache, remove getHostFor(UURI) and getServerFor(UURI)
* ServerCacheUtil.java
removed
* other classes
change usage of ServerCacheUtil to ServerCache
* BloomUriUniqFilter.java
make bloom size configurable via bean properties
change implementing class to BloomFilter64bit (given growing prevalance of 64bit JVMs and larger heaps)
* BloomUriUniqFilterTest.java
update test for property-set sizing
* BloomFilter64bit.java
improve comment
* CrawlJob.java
renamed isContainerOk() to hasApplicationContext()
increased granularity of isLaunchable() to offer launch button
* JobResource.java
more coarse check for hasApplicationContext() to disable build button
gojomo