* ServerCache.java
make abstract, implement getHostFor(UURI) and getServerFor(UURI)
* DefaultServerCache.java
extend ServerCache, remove getHostFor(UURI) and getServerFor(UURI)
* ServerCacheUtil.java
removed
* other classes
change usage of ServerCacheUtil to ServerCache
* BloomUriUniqFilter.java
make bloom size configurable via bean properties
change implementing class to BloomFilter64bit (given growing prevalance of 64bit JVMs and larger heaps)
* BloomUriUniqFilterTest.java
update test for property-set sizing
* BloomFilter64bit.java
improve comment
* ScopePlusOneDecideRule.java
removing, in favor of SurtPrefixedDecideRule's 'alsoCheckVia' option
* RecyclingFastBufferedOutputStream.java
use library by deriving as subclass (rather than c&p), now that class is available in an included library (fastutil)
* BdbModule.java
allow Database instance reuse if BdbConfig is identical instance
* PersistProcessor.java
convert shared BdbConfig for history db to static constant HISTORY_DB_CONFIG
* PersistOnlineProcessor.java
use HISTORY_DB_CONFIG, don't close possibly-shared DB on stop()
* PrecedenceLoader.java, PreloadedUriPrecedencePolicy.java
use HISTORY_DB_CONFIG
* ConfigPathConfigurer, PathFixupListener
notify beans when paths fixed
* SurtPrefixedDecideRule.java
only read source prefixes after path fixup; dump after read or any batch of seeds
* SeedModule.java, TextSeedModule.java, SeedListener.java
new concludedSeedBatch event
* AbstractFrontier.java, StatisticsTracker.java
ignore new concluded-batch event
* AlertThreadGroup.java, JobResource.java
new mechanism for threads not part of AlertThreadGroup to temporarily redirect their logging to provided (job) logger
* ConfigPathConfigurer, PathFixupListener
notify beans when paths fixed
* SurtPrefixedDecideRule.java
only read source prefixes after path fixup; dump after read or any batch of seeds
* SeedModule.java, TextSeedModule.java, SeedListener.java
new concludedSeedBatch event
* AbstractFrontier.java, StatisticsTracker.java
ignore new concluded-batch event
* AlertThreadGroup.java, JobResource.java
new mechanism for threads not part of AlertThreadGroup to temporarily redirect their logging to provided (job) logger
* BdbModule.java
be more robust about 0/-1 as 'don't care' values
* AbstractFrontier.java
reorder settable properties for ease of discovery
* FetchHTTP.java
correct type to allow setting property
* profile-crawler-beans.cxml
add comments for all unstated default values an operator might want to change
* (many)
reorder, refactor, rename to better minimize/match simple configuration
* ArchiveUtils.java
(checkContinue) utility method relocated from ToeThread for converting interrupt status to exception
* ClassKeyMatchesRegExpDecideRule.java
properly set up crawlController property for auto-wiring
* ObjectIdentityBdbCache.java
explicitly page-out lingering entry;
temporary System.err logging to see if problem is momentary lag in RefQueue-appearance
* BdbModule.java
split openDatabase() to openManagedDatabase() (auto-closed) and plain openDatabase() (caller-closes)
set TempStoredSortedMap to use unmanaged openDatabase
* (others)
use openManagedDatabase()
* BdbModule.java
add temp-stored-map service method
remove unused secondary-db support
* TempStoredSortedMap.java
StoredSortedMap that can destroy its underlying database after temporary use
* StatisticsTracker.java
update various sorted-by-decreasing-frequency methods to use temp StoredMaps, with duplicate keys that are negative counts
* (Multiple)Report.java
update to use new duplicate-keyed frequency maps
problematic (illegal control character?) outlink
catch IllegalArgumentException in ANVLRecord.addLabelValue() to allow
WARCWriter to continue processing, and move
WARCWriter.createRecordHeader() into its own try-catch block to with
logging prevent fouling record when problems occur creating header.
* ANVLRecord.java
catch IllegalArgumentException in addLabelValue and log warning to
give WARCWriter to continue processing
* WARCWriter.java
move createRecordHeader() into its own try-catch block to log
severe problem writing record without fouling the WARC, and have
baseCharacterCheck() throw IllegalArgumentException in accord with
ANVLRecord.addLabelValue
* WARCWriterTest.java
throw IllegalArgumentException in accord with
WARCWriter.createRecordHeader()
* BdbModuleTest.java, CachedBdbMapTest.java
use putIfAbsent() in preference to put() (which logs warning)
* BdbModule.java
register big-cache DBs in implementation-specific methods, so always auto-closed on env close
alternate simplified implementation of our object-cache need
* ObjectIdentityCache.java
new interface, far less than (Concurrent)Map, for big seems-like-in-memory object-cache
* Supplier.java
trivial interface for deferred-provision of new instance
* CachedBdbMap.java
implement ObjectIdentityCache, mainly via getOrUse()
* ObjectIdentityMemCache.java
trivial ConcurrentHashMap-based all-in-memory ObjectIdentityCache implementation
* ObjectIdentityBdbCache.java
BDB-backed ObjectIdentityCache implementation, carved from CachedBdbMap
* BdbModule.java
refactor utility methods to offer CachedBdbMap or ObjectIdentityBdbCache instances, via source toggle
* EnhancedEnvironment.java
convenience test-environment method
* (others)
update to use ObjectIdentityCache/ObjectIdentityMemCache in place of ConcurrentMap/ConcurrentHashMap
alternate simplified implementation of our object-cache need
* ObjectIdentityCache.java
new interface, far less than (Concurrent)Map, for big seems-like-in-memory object-cache
* Supplier.java
trivial interface for deferred-provision of new instance
* CachedBdbMap.java
implement ObjectIdentityCache, mainly via getOrUse()
* ObjectIdentityMemCache.java
trivial ConcurrentHashMap-based all-in-memory ObjectIdentityCache implementation
* ObjectIdentityBdbCache.java
BDB-backed ObjectIdentityCache implementation, carved from CachedBdbMap
* BdbModule.java
refactor utility methods to offer CachedBdbMap or ObjectIdentityBdbCache instances, via source toggle
* EnhancedEnvironment.java
convenience test-environment method
* (others)
update to use ObjectIdentityCache/ObjectIdentityMemCache in place of ConcurrentMap/ConcurrentHashMap
WARConstants.java
update WARC_VERSION=1.0 according to ISO 28500:2009 as of May 2009
WARCWriterProcessor.java
update format and conformsTo fields according to ISO 28500:2009,
as of May 2009, as described at http://bibnum.bnf.fr/WARC/,
latest draft as of November 2008
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf
* CachedBdbMap.java
make all WARNING logs include stacks
add WARNINGs if memMap.remove() has no expected effect
verify reported expunction, WARN and try to fix if memMap expunction not evident
adjust indentation/bracketing
* BdbModule.java
fail-fast if not started properly; stop with fewer errors in problem situations
* CrawlController.java
continue past runtimeException on stop
* CrawlJob.java
less redundant error-reporting
* AbstractFrontier.java
consider PAUSE or FINISH valid targets after unexpected exception
* StatisticsTracker.java
avoid promoting runtimeException in report-generating up
* PersistOnlineProcessor.java
delegate DB-closing to BdbModule to avoid double-close
* SurtPrefixSet.java
refactor to allow add-directives under outside control
* SurtPrefixedDecideRule.java
extend nonseedLine() mechanism to trigger off '+' or '-' depending on decision being ACCEPT or REJECT
gojomo