* BloomFilter64bit.java
include the split-to-subarrays (for larger bitfields) and round-up-to-power-of-2 (for performance) options previously in largely-redundant classes
fit a number of problems with int/long overflow and bitwise ops
add methods for reporting/testing
* BloomFilter.java
add methods for reporting/testing
* BloomFilterTest.java, BloomFilter64bitTest.java
more extensive tests, including two lengthy tests of default/oversized blooms usually disabled by renaming
* BloomFilter32bit.java, BloomFilter32bitSplit.java, BloomFilter32bp2.java, BloomFilter32bp2Split.java
deleted as buggy or redundant
* BenchmarkBlooms.java
move to test source dir
* BloomUriUniqFilter.java
change to accept filter instance (rather than parameters) for added configuration flexibility
fix comments
* BloomUriUniqFilterTest.java
supply filter not paramters
* BloomFilter64bit.java
fix long arithmetic calculating bit-length m in constructor
slight optimization of add()
* BloomFilterTest.java
abstract superclass for bloom impl class tests
* BloomFilter64bitTest.java
sanity check that set bits, after a handful of adds into default-sized filter, occupy top and bottom 20% of range (which would have caught this bug earlier)
* ReplayCharSequence.java, GenericReplayCharSequence.java, InMemoryReplayCharSequence.java
store actual first exception for reference
* ExtractorHTML.java
report exception in nonfatal-log, rather than WARNING (which was still polluting job log more than desired -- but should still be logged to allow future investigation of sites where encoding issues may exist)
* ReplayCharSequence.java
new interface method: getDecoderExceptionCount(), reporting >0 if any decoding problems were encountered and silently replaced
* GenericReplayCharSequence.java
count decoding problems, and only log as FINE rather than WARNING
* InMemoryReplayCharSequence.java
try decoding with reporting; if any problems note at least one then decode without reporting errors
* ExtractorHTML.java
after using ReplayCharSequence, log WARNING if nonzero decoding errors
* GenericReplayCharSequence.java
* Scoper.java
* FetchFTP.java
* FetchHTTP.java
demote a bunch of INFO logging to FINE so that be default, console isn't filled with output for every URI handled by a busy crawl
* CheckpointService.java
reorganize so that checkpoint timedTask is reset whenever interval is reset on a running crawl
* CheckpointSuccessEvent.java, CrawlJob.java
report successes as new kind of ApplicationEvent; propagagte to job log
* BdbModule.java
remove obsolete unconsulted checkpointCopyLogs setting
* UriUtils.java
move likely-URI heuristics here for better consistency, reuse, testing, improvement
* UriUtilsTest.java
testing for UriUtils
* ExtractorJS.java
move likely-URI support to UriUtils; use new test
* ExtractoSWF.java
use new UriUtils likely-URI testing
* ExtractorHTML.java
move likely-URI testing to UriUtils, use new test
* JerichoExtractorHTML.java
use new UriUtils likely-URI testing
* CachedBdbMap.java
@deprecated
* BdbModule.java
note (unused) 'CBM' option is deprecated
* ObjectIdentityBdbCache.java
tighten logging to FINE
* ObjectIdentityBdbCacheTest.java
fix test that was only trying to log, rather than fail, problems
* ArchiveRecord.java
do not decrement position after EOR, port from H1
* ARCWriterTest.java
added testArchiveRecordEORConsistent(), testArchiveRecordMarkSupport()
port from H1
* EnhancedEnvironment.java
in getClassCatalog(), when read-only env, insist on read-only catalog
* PersistLoadProcessor.java
remove existing curi members from prior to prevent clobbering source
* PersistProcessor.java
set historyDbConfig read-only, set je.log.faultReadSize for
compatability with BDBModule.setup()
* Arc2Warc.java
when HTTP response, write response record rather than resource
enable reconstruction of ARC from WARC via namedFields
* ARCRecord.java
added headerString and accessor to save verbatim ARC header string
offer html
* XmlMarshaller.java
new class to write xml based on nested map structure
* BeanBrowseResource.java ScriptResource.java EngineResource.java
JobResource.java JobRelatedResource.java
support xml representation
* WorkQueueFrontier.java PrecedenceProvider.java WorkQueue.java ToeThread.java
ToePool.java CrawlJob.java Engine.java Frontier.java CrawlController.java
StatisticsTracker.java CrawledBytesHistotable.java FetchStats.java
ProcessorChain.java CrawlURI.java Reporter.java
offer same data that is returned in the single line reports in a map
suitable for use with XmlMarshaller
* FileUtils.java
new method tryToCanonicalize(File)
* commons/pom.xml, .classpath
update to BDB-JE 4.0.71
* BdbModule.java
(getStoredQueue) added
(getStoredMap) made more general
(openDatabase) all databases now 'managed'
* TempStoredSortedMap.java -> DisposableStoredSortedMap.java
renamed
* StatisticsTracker.java, **Report.java
use new name
* StoredQueue.java, StoredQueueTest.java
move to org.archive.bdb package
* BdbMultipleWorkQueues.java, PrefixFinderTest.java
update for BDB-JE 4 changes
* BdbFrontier.java
use getStoredQueue
drop unneeded custom serialization support
* WorkQueueFrontier.java
drop unneeded custom serialization support
* JobResource.java
suppress checkpoint-selector after terminate
* Checkpoint.java
adapt to work without direct reference to CheckpointService
* CheckpointService.java
configure Checkpoint without making it dependent on CheckpointService
* PersistLogProcessor.java
rotate logs on checkpoint
* CrawlerJournal.java, FrontierJournal.java
rotate for checkpoint support without full Checkpointable
stats & logging checkpoint support
* StatisticsTracker.java
implement Checkpointable; save small state in JSON, recycle BDB data when recovering
* CrawlerLoggerModule.java
implement Checkpointable; rotate logs on checkpoint
* GenerationFileHandler.java
avoid clobbering leftover logs from earlier futures via moveAside
* CrawlStatSnapshot.java
(sameProgressAs) test for lack-of-progress
* ObjectIdentityMemCache.java, TopNSet.java
offer deeper access to assist checkpoint/resume
core checkpointing system classes:
* Checkpointable.java
basic interface for checkpoint aware/capable beans
* Checkpoint.java
one Checkpoint, either in progress or recovery -- a name + storage directory
* CheckpointService.java
helper bean to enable, trigger checkpoints and recovery
* Checkpointer.java(*2), CheckpointInputStream.java, CheckpointRecovery.java DefaultCheckpointRecovery.java, RecoverAction.java
(deleted) no longer used
* IoUtils.java
remove class that confusingly-overlaps with commons IOUtils; move remaining methods to ArchiveUtils or archive's FileUtils
* ArchiveUtils.java, FileUtils.java
receive relocated methods; eliminated deprecated or unused methods
* JSONUtils.java
new class to collect common JSON actions
* Iteratorable.java
wrap Iterator as Iterable for foreach usage
* (many)
update to use alternate utils methods
* ServerCache.java
make abstract, implement getHostFor(UURI) and getServerFor(UURI)
* DefaultServerCache.java
extend ServerCache, remove getHostFor(UURI) and getServerFor(UURI)
* ServerCacheUtil.java
removed
* other classes
change usage of ServerCacheUtil to ServerCache
* BloomUriUniqFilter.java
make bloom size configurable via bean properties
change implementing class to BloomFilter64bit (given growing prevalance of 64bit JVMs and larger heaps)
* BloomUriUniqFilterTest.java
update test for property-set sizing
* BloomFilter64bit.java
improve comment
gojomo