* PersistLogProcessor.java
rotate logs on checkpoint
* CrawlerJournal.java, FrontierJournal.java
rotate for checkpoint support without full Checkpointable
stats & logging checkpoint support
* StatisticsTracker.java
implement Checkpointable; save small state in JSON, recycle BDB data when recovering
* CrawlerLoggerModule.java
implement Checkpointable; rotate logs on checkpoint
* GenerationFileHandler.java
avoid clobbering leftover logs from earlier futures via moveAside
* CrawlStatSnapshot.java
(sameProgressAs) test for lack-of-progress
* ObjectIdentityMemCache.java, TopNSet.java
offer deeper access to assist checkpoint/resume
core checkpointing system classes:
* Checkpointable.java
basic interface for checkpoint aware/capable beans
* Checkpoint.java
one Checkpoint, either in progress or recovery -- a name + storage directory
* CheckpointService.java
helper bean to enable, trigger checkpoints and recovery
* Checkpointer.java(*2), CheckpointInputStream.java, CheckpointRecovery.java DefaultCheckpointRecovery.java, RecoverAction.java
(deleted) no longer used
* IoUtils.java
remove class that confusingly-overlaps with commons IOUtils; move remaining methods to ArchiveUtils or archive's FileUtils
* ArchiveUtils.java, FileUtils.java
receive relocated methods; eliminated deprecated or unused methods
* JSONUtils.java
new class to collect common JSON actions
* Iteratorable.java
wrap Iterator as Iterable for foreach usage
* (many)
update to use alternate utils methods
* ServerCache.java
make abstract, implement getHostFor(UURI) and getServerFor(UURI)
* DefaultServerCache.java
extend ServerCache, remove getHostFor(UURI) and getServerFor(UURI)
* ServerCacheUtil.java
removed
* other classes
change usage of ServerCacheUtil to ServerCache
* BloomUriUniqFilter.java
make bloom size configurable via bean properties
change implementing class to BloomFilter64bit (given growing prevalance of 64bit JVMs and larger heaps)
* BloomUriUniqFilterTest.java
update test for property-set sizing
* BloomFilter64bit.java
improve comment
* CrawlJob.java
renamed isContainerOk() to hasApplicationContext()
increased granularity of isLaunchable() to offer launch button
* JobResource.java
more coarse check for hasApplicationContext() to disable build button
* DefaultServerCache.java
getHostFor(UURI): special case dns uris
* ServerCache.java
add getHostFor(UURI) to public interface
* ServerCacheUtil.java
remove now redundant wrapper method
* DispositionProcessor.java AbstractFrontier.java IPQueueAssignmentPolicy.java
BucketQueueAssignmentPolicy.java PreconditionEnforcer.java QuotaEnforcer.java
StatisticsTracker.java ServerCacheTest.java FetchHTTP.java
Kw3WriterProcessor.java WriterPoolProcessor.java
changed to use serverCache.getHostFor(UURI) (a few cases remain that use
getHostFor(String))
* BdbModule.java
allow Database instance reuse if BdbConfig is identical instance
* PersistProcessor.java
convert shared BdbConfig for history db to static constant HISTORY_DB_CONFIG
* PersistOnlineProcessor.java
use HISTORY_DB_CONFIG, don't close possibly-shared DB on stop()
* PrecedenceLoader.java, PreloadedUriPrecedencePolicy.java
use HISTORY_DB_CONFIG
* ConfigPathConfigurer, PathFixupListener
notify beans when paths fixed
* SurtPrefixedDecideRule.java
only read source prefixes after path fixup; dump after read or any batch of seeds
* SeedModule.java, TextSeedModule.java, SeedListener.java
new concludedSeedBatch event
* AbstractFrontier.java, StatisticsTracker.java
ignore new concluded-batch event
* AlertThreadGroup.java, JobResource.java
new mechanism for threads not part of AlertThreadGroup to temporarily redirect their logging to provided (job) logger
* URIAuthorityBasedQueueAssignmentPolicy.java
choose numbered subqueue on first-path-segment only, so similar URIs land in same subqueue
* CrawlMapper.java
add queueKey to diversion lines
* HashCrawlMapper.java
update license
* Engine.java
protect against disappeared jobsDir in findJobConfigs()
check for .jobpath files in findJobConfigs()
added considerAsJobPath()
log added jobs in considerAsJobDirectory()
make considerAsJobDirectory() return true when jobConfig (already) exists
added leaveJobPathFile() to write .jobpath file for newly added jobs
* EngineResource.java
call leaveJobPathFile() when job directory added successfully
added messageDiv() to make messages more uniform and prominent
re-scan jobConfigs on each page load - may obviate need for "rescan" button
* Engine.java
createNewJobWithDefaults() to write profile-crawler-beans.cxml resoure into new job
* EngineResource.java
added FORM for new job dir and "create" action
* profile-crawler-beans.cxml
added resource from dist/src/main/conf/jobs/profile-defaults to bootstrap new jobs
* Engine.java
return 'false' rather than NPE when given directory non-existent/contains no .cxml
* EngineResource.java
better handle empty 'add' path; show NACK flash when 'add' has no effect
* H1toH3.map, migrate-template-crawler-beans.cxml, .classpath
move resources to 'resources' src subdirectory, so included in built JAR
move 'resources' directorys to library rather than source directories for dev-time reachability
* MigrateH1to3Tool.java
handle H1 values that just need capitalization
report all 'no rule' situations as 'needs attention'
touchup explanations and comments
enable uriCanonicalizationRule overrides
* BdbModule.java
be more robust about 0/-1 as 'don't care' values
* AbstractFrontier.java
reorder settable properties for ease of discovery
* FetchHTTP.java
correct type to allow setting property
gojomo