* ArchiveUtils.java
(checkContinue) utility method relocated from ToeThread for converting interrupt status to exception
* ClassKeyMatchesRegExpDecideRule.java
properly set up crawlController property for auto-wiring
* (many)
update license notice to Apache where appropriate
eliminate ProcessorURI, DefaultProcessorURI in favor of CrawlURI (now in modules package)
other comment/warning/unused code cleanup
* ObjectIdentityBdbCache.java
explicitly page-out lingering entry;
temporary System.err logging to see if problem is momentary lag in RefQueue-appearance
* BdbModule.java
split openDatabase() to openManagedDatabase() (auto-closed) and plain openDatabase() (caller-closes)
set TempStoredSortedMap to use unmanaged openDatabase
* (others)
use openManagedDatabase()
* BdbModule.java
add temp-stored-map service method
remove unused secondary-db support
* TempStoredSortedMap.java
StoredSortedMap that can destroy its underlying database after temporary use
* StatisticsTracker.java
update various sorted-by-decreasing-frequency methods to use temp StoredMaps, with duplicate keys that are negative counts
* (Multiple)Report.java
update to use new duplicate-keyed frequency maps
problematic (illegal control character?) outlink
catch IllegalArgumentException in ANVLRecord.addLabelValue() to allow
WARCWriter to continue processing, and move
WARCWriter.createRecordHeader() into its own try-catch block to with
logging prevent fouling record when problems occur creating header.
* ANVLRecord.java
catch IllegalArgumentException in addLabelValue and log warning to
give WARCWriter to continue processing
* WARCWriter.java
move createRecordHeader() into its own try-catch block to log
severe problem writing record without fouling the WARC, and have
baseCharacterCheck() throw IllegalArgumentException in accord with
ANVLRecord.addLabelValue
* WARCWriterTest.java
throw IllegalArgumentException in accord with
WARCWriter.createRecordHeader()
implementation in 1.14.3/heritrix2 to latest ISO document,
heritrix3 - in warcinfo record, "robots" field should be
"robots: classic" rather than "robots: CLASSIC"
* WARCWriterProcessor.java
made robots entry in metadata lower-case
* BdbModuleTest.java, CachedBdbMapTest.java
use putIfAbsent() in preference to put() (which logs warning)
* BdbModule.java
register big-cache DBs in implementation-specific methods, so always auto-closed on env close
* PreconditionEnforcer.java, CrawlServer.java
in testing found some ordering issues affecting robots-scheduling if parallelQueues>1
reordered/refactored to put more responsibility in CrawlServer to resolve
fold in H3 updates to http transaction constant for revisit records.
* CoreAttributeConstants.java
removed A_HTTP_TRANSACTION, in favor checking CrawlURI method with isHttpTran$
* ModelAttributeConstants.java
removed A_HTTP_TRANSACTION, in favor checking CrawlURI method with isHttpTran$
* FetchHistoryProcessor.java
check for HTTP transaction using CrawlURI.isHttpTransaction()
use HttpMethod in saveHeader() instead of HttpMethodBase
* WARCWriterProcessor.java
check for HTTP transaction using CrawlURI.isHttpTransaction()
use HttpMethod in saveHeader() instead of HttpMethodBase
* CrawlURI.java
(getPolicyBasisUURI) return either own UURI, or -- for prereqs -- via UURI, as basis for overlays/policy calculations
* URIAuthorityBasedQueueAssignmentPolicy.java
use getPolicyBasisUURI for queue-name decisions
* SheetOverlaysManager.java
use getPolicyBasisUURI for surt-based overlay decisions
alternate simplified implementation of our object-cache need
* ObjectIdentityCache.java
new interface, far less than (Concurrent)Map, for big seems-like-in-memory object-cache
* Supplier.java
trivial interface for deferred-provision of new instance
* CachedBdbMap.java
implement ObjectIdentityCache, mainly via getOrUse()
* ObjectIdentityMemCache.java
trivial ConcurrentHashMap-based all-in-memory ObjectIdentityCache implementation
* ObjectIdentityBdbCache.java
BDB-backed ObjectIdentityCache implementation, carved from CachedBdbMap
* BdbModule.java
refactor utility methods to offer CachedBdbMap or ObjectIdentityBdbCache instances, via source toggle
* EnhancedEnvironment.java
convenience test-environment method
* (others)
update to use ObjectIdentityCache/ObjectIdentityMemCache in place of ConcurrentMap/ConcurrentHashMap
alternate simplified implementation of our object-cache need
* ObjectIdentityCache.java
new interface, far less than (Concurrent)Map, for big seems-like-in-memory object-cache
* Supplier.java
trivial interface for deferred-provision of new instance
* CachedBdbMap.java
implement ObjectIdentityCache, mainly via getOrUse()
* ObjectIdentityMemCache.java
trivial ConcurrentHashMap-based all-in-memory ObjectIdentityCache implementation
* ObjectIdentityBdbCache.java
BDB-backed ObjectIdentityCache implementation, carved from CachedBdbMap
* BdbModule.java
refactor utility methods to offer CachedBdbMap or ObjectIdentityBdbCache instances, via source toggle
* EnhancedEnvironment.java
convenience test-environment method
* (others)
update to use ObjectIdentityCache/ObjectIdentityMemCache in place of ConcurrentMap/ConcurrentHashMap
WARConstants.java
update WARC_VERSION=1.0 according to ISO 28500:2009 as of May 2009
WARCWriterProcessor.java
update format and conformsTo fields according to ISO 28500:2009,
as of May 2009, as described at http://bibnum.bnf.fr/WARC/,
latest draft as of November 2008
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf
* CachedBdbMap.java
make all WARNING logs include stacks
add WARNINGs if memMap.remove() has no expected effect
verify reported expunction, WARN and try to fix if memMap expunction not evident
adjust indentation/bracketing
* PersistProcessor.java
add readOnly option for environment only opened for copying-from
* PersistLoadProcessor.java
make start() safe to call redundantly (per Lifecycle contract)
* BdbModule.java
fail-fast if not started properly; stop with fewer errors in problem situations
* CrawlController.java
continue past runtimeException on stop
* CrawlJob.java
less redundant error-reporting
* AbstractFrontier.java
consider PAUSE or FINISH valid targets after unexpected exception
* StatisticsTracker.java
avoid promoting runtimeException in report-generating up
* PersistOnlineProcessor.java
delegate DB-closing to BdbModule to avoid double-close
* BeanBrowseResource.java
coerce edits to type of value they're replacing; makes for safer mid-crawl edits of values in maps
* JobRelatedResource.java
improve rendering/clickability of map, indexed objects
* SurtPrefixedDecideRule.java
allow seeds to be null (as when rule is intentionally completely
independent of seeds; may also need to disqualify rule from
Spring autowiring in such a case)
gojomo