* ArchiveUtils.java
(checkContinue) utility method relocated from ToeThread for converting interrupt status to exception
* ClassKeyMatchesRegExpDecideRule.java
properly set up crawlController property for auto-wiring
* (many)
update license notice to Apache where appropriate
eliminate ProcessorURI, DefaultProcessorURI in favor of CrawlURI (now in modules package)
other comment/warning/unused code cleanup
* BdbModule.java
split openDatabase() to openManagedDatabase() (auto-closed) and plain openDatabase() (caller-closes)
set TempStoredSortedMap to use unmanaged openDatabase
* (others)
use openManagedDatabase()
* BdbModule.java
add temp-stored-map service method
remove unused secondary-db support
* TempStoredSortedMap.java
StoredSortedMap that can destroy its underlying database after temporary use
* StatisticsTracker.java
update various sorted-by-decreasing-frequency methods to use temp StoredMaps, with duplicate keys that are negative counts
* (Multiple)Report.java
update to use new duplicate-keyed frequency maps
* PreconditionEnforcer.java, CrawlServer.java
in testing found some ordering issues affecting robots-scheduling if parallelQueues>1
reordered/refactored to put more responsibility in CrawlServer to resolve
* CrawlURI.java
(getPolicyBasisUURI) return either own UURI, or -- for prereqs -- via UURI, as basis for overlays/policy calculations
* URIAuthorityBasedQueueAssignmentPolicy.java
use getPolicyBasisUURI for queue-name decisions
* SheetOverlaysManager.java
use getPolicyBasisUURI for surt-based overlay decisions
alternate simplified implementation of our object-cache need
* ObjectIdentityCache.java
new interface, far less than (Concurrent)Map, for big seems-like-in-memory object-cache
* Supplier.java
trivial interface for deferred-provision of new instance
* CachedBdbMap.java
implement ObjectIdentityCache, mainly via getOrUse()
* ObjectIdentityMemCache.java
trivial ConcurrentHashMap-based all-in-memory ObjectIdentityCache implementation
* ObjectIdentityBdbCache.java
BDB-backed ObjectIdentityCache implementation, carved from CachedBdbMap
* BdbModule.java
refactor utility methods to offer CachedBdbMap or ObjectIdentityBdbCache instances, via source toggle
* EnhancedEnvironment.java
convenience test-environment method
* (others)
update to use ObjectIdentityCache/ObjectIdentityMemCache in place of ConcurrentMap/ConcurrentHashMap
* BdbModule.java
fail-fast if not started properly; stop with fewer errors in problem situations
* CrawlController.java
continue past runtimeException on stop
* CrawlJob.java
less redundant error-reporting
* AbstractFrontier.java
consider PAUSE or FINISH valid targets after unexpected exception
* StatisticsTracker.java
avoid promoting runtimeException in report-generating up
* PersistOnlineProcessor.java
delegate DB-closing to BdbModule to avoid double-close
* BeanBrowseResource.java
coerce edits to type of value they're replacing; makes for safer mid-crawl edits of values in maps
* JobRelatedResource.java
improve rendering/clickability of map, indexed objects
* ActionDirectory.java
support for .seeds, .recover, .include, .schedule, .force files
* CrawlController.java
change default for pause-at-start to true; ensure enters PAUSED rather than PAUSING
* Frontier.java
(importRecoverFormat) support method for .recover, .include, .schedule, .force additions
(considerIncluded) change to accept CrawlURI
* AbstractFrontier.java
(importRecoverFormat) support method for .recover, .include, .schedule, .force additions
* BdbFrontier.java
remove too-strict assertion
* FrontierJournal.java
rename recover-log 'frontier.recover.gz' (so that '.recover' is a dot-suffix)
* WorkQueueFrontier.java
(considerIncluded) change to accept CrawlURI
* CrawlerJournal.java, IoUtils.java
move getBufferedReader to IoUtils
* FrontierJournal.java
add new 'included' tag for log lines ("Fi")
* (others)
update for relocated method
[HER-1660] allow adding seeds, recovery-journal-like URI data at any point in crawl via 'action directory'
[HER-769] Support case where millions of seeds.
* AbstractFrontier.java
move sourceTagSeeds setting to SeedModule
trigger and schedule seeds via SeedModule's announcements and SeedListener protocols
* StatisticsTracker.java
implement SeedListener, use processedSeedRecords as list of all seeds
* ProcessorURI.java, DefaultProcessorURI.java
(getURI) convenience accessor for plain URI string
* SurtPrefixedDecideRule.java
use seeds as prefixes via SeedListener notifications, rather than separate scan of seeds source text
* SeedListener.java
simplify to notifications of new seed, and nonseed text mixed with seeds (as with SURT directives)
* SeedModule.java
methods for SeedListener interface, announceSeeds, actOn(File)
* TextSeedModule.java
eliminate interators; add methods for announcing seeds and scanning seed text sources (including arbitrary Files)
* PrefixSet.java, SurtPrefixSet.java
derive from ConcurrentSkipListSet; eliminate now-unneeded extra synchronization and clone()s
offer considerAsDirective() for direct application of directive Strings
* CachedBdbMap.java
do expunge on put(), replace()
add low-memory-sensitive 'canary' to force expunge even if otherwise untriggered
* CachedBdbMapTest.java
add test of idle expunge in low-memory conditions
* CrawlURI.java
(getURI) direct access to String URI (don't reuse toString() functionally)
* SeedRecord.java
support for updating record with later/repeat report
* StatisticsTracker.java
remove put()s for processedSeedRecords, hostsLastFinished
* BdbFrontier.java
(getQueueFor) remove put, do in concurrent-compliant manner
* LongToIntConsistentHash.java + Test
consistent-hashing utility class
* URIAuthorityBasedQueueAssignmentPolicy.java
shared superclass for the hostname and surtauthority QAPs
setting for 'deferToPrevious' -- avoid changing assignments
setting for 'parallelQueues' -- when > 1, consistent-hash URIs across that many separate queues (with numerical suffix)
* SurtAuthorityQueueAssignmentPolicy.java, HostnameQueueAssignmentPolicy.java
refactor to derive from URIAuthorityBasedQueueAssignmentPolicy
* QueueAssignmentPolicy.java
add Apache license
* Hop.java
add String representation for convenience
* ConfigPathConfigurer.java
work in two passes: one (BeanPostProcessor) collects all candidates for fixup, two (ApplicationListener) actually performs fixup
suppress fixup of own base path, preventing circulare ref/StackOverflow
* ConfigPath.java
change sense of 'merge' to merge into receiver
* SurtPrefixedDecideRule.java
use ConfigFile, use 'merge' in assignment
* CrawlerLoggerModule.java
use merge in assignment
* CrawlJob.java
(getBeanpathTarget) utility method
* JobResource.java
respect unset ConfigPaths
* ConfigPathEdito.java, ConfigFileEditor.java
better comments, ConfigFile variant
* CrawlController.java
(isFinished) added
* CrawlJob.java
rename reset to teardown
wait for CrawlController to report finished before discarding AppContext
if not finished in time, fail reporting 'false'
* JobResource.java
show Flash if teardown did not succeed; user must retry
* BdbUriUniqFilter.java
don't throw DatabaseException from simple accessor
rename 'discard' to 'teardown'
enable browse/edit of paths outside jobdir via new /anypath/ service
* EngineResource.java, Engine.java
enable addition of job dirs from outside main jobs directory
* EngineApplication.java
register new /engine/anypath service for outside-job-dir paths
* HopCrossesAssignmentLevelDomainDecideRule.java
port H2 IsCrossTopmostAssignedSurtHopDecideRule to H3 under this new name
* PublicSuffixes.java
update comment, public method for 'Assignment Level Domain'/'Assignment Level SURT' term
* AssignmentLevelSurtQueueAssignmentPolicy
renamed TopmostAssignedSurtQueueAssignmentPolicy
* (all, especially CachedBdbMap.java)
pbaclace-contributed patch, ported to H3, plus...
* StatisticsTracker.java
replace LongWrapper with AtomicLong
use non-recursive putIfAbsent-based idiom for get-or-create
* DefaultServerCache.java
clean up host/server get-or-create around non-recursive concurrent idiom
* BdbModule.java
add expectedConcurrency property & je.lock.nLockTables tuning
* TopNSet.java
add synchronization, as calling StatisticsTracker method has lost it
* Heritrix.java
fail-fast with helpful error message if launched with Java < 1.6
* EditRepresentation.java
"save changes" rather than "update content"
* EngineApplication.java
customize restlet error to include back-button and full stack (instead of link to HTTP spec!)
gojomo