* ScopePlusOneDecideRule.java
removing, in favor of SurtPrefixedDecideRule's 'alsoCheckVia' option
* RecyclingFastBufferedOutputStream.java
use library by deriving as subclass (rather than c&p), now that class is available in an included library (fastutil)
* BdbModule.java
allow Database instance reuse if BdbConfig is identical instance
* PersistProcessor.java
convert shared BdbConfig for history db to static constant HISTORY_DB_CONFIG
* PersistOnlineProcessor.java
use HISTORY_DB_CONFIG, don't close possibly-shared DB on stop()
* PrecedenceLoader.java, PreloadedUriPrecedencePolicy.java
use HISTORY_DB_CONFIG
* ConfigPathConfigurer, PathFixupListener
notify beans when paths fixed
* SurtPrefixedDecideRule.java
only read source prefixes after path fixup; dump after read or any batch of seeds
* SeedModule.java, TextSeedModule.java, SeedListener.java
new concludedSeedBatch event
* AbstractFrontier.java, StatisticsTracker.java
ignore new concluded-batch event
* AlertThreadGroup.java, JobResource.java
new mechanism for threads not part of AlertThreadGroup to temporarily redirect their logging to provided (job) logger
* BdbModule.java
be more robust about 0/-1 as 'don't care' values
* AbstractFrontier.java
reorder settable properties for ease of discovery
* FetchHTTP.java
correct type to allow setting property
* profile-crawler-beans.cxml
add comments for all unstated default values an operator might want to change
* (many)
reorder, refactor, rename to better minimize/match simple configuration
* defaults.xml
update bundled profile for new chains refactoring
* CrawlController.java
move to three processor-chains rather than one
* Frontier.java
(loadSeeds) removed
* ToeThread.java
delegate most processing-loop to FetchChain and DispositionChain
* AbstractFrontier.java, WorkQueueFrontier.java
move policies/calculation out to processors
* CandidatesProcessor.java
new processor for DispositionChain that runs every outlink through CandidateChain
* CrawlStateUpdater.java -> DispositionProcessor.java
rename, expand to prep CrawlURI for frontier
* FrontierScheduler.java
deleted; use CandidatesProcessor/CandidateChain
* LinksScoper.java
deprecated; use CandidatesProcessor/CandidateChain/CandidateScoper
(only temporarily retained for ease of comparison)
* CandidateScoper.java
simple single-URI scope-testing for CandidateChain
* FrontierPreparer.java
precalculate all frontier-policies in CandidateChain, before scheduling
* PreconditionEnforcer.java
ProcessorURI->CrawlURI; take on some prerequisite preparation previously deferred to elsewhere
* ProcessorsReport.java
update for 3-chains of Processors
* SheetOverlaysManager.java
(applyOverridesTo) moved here for broader use
* CandidateChain.java, FetchChain.java, DispositionChain.java
role-specific subclasses of ProcessorChain (suitable for type-based autowiring)
* CrawlURI.java
new fields/accessors of use to new chains/frontier
* PostProcessor.java
deleted; skip-to-'postprocessing' is now skip-to-end-of-chain
* ProcessorChain.java
take-on control loop formerly in ToeThread
* ProcessResult.java
absorb ProcessStatus
eliminate problematic STUCK result
* (many)
update license notice to Apache where appropriate
eliminate ProcessorURI, DefaultProcessorURI in favor of CrawlURI (now in modules package)
other comment/warning/unused code cleanup
* BdbModule.java
split openDatabase() to openManagedDatabase() (auto-closed) and plain openDatabase() (caller-closes)
set TempStoredSortedMap to use unmanaged openDatabase
* (others)
use openManagedDatabase()
implementation in 1.14.3/heritrix2 to latest ISO document,
heritrix3 - in warcinfo record, "robots" field should be
"robots: classic" rather than "robots: CLASSIC"
* WARCWriterProcessor.java
made robots entry in metadata lower-case
* PreconditionEnforcer.java, CrawlServer.java
in testing found some ordering issues affecting robots-scheduling if parallelQueues>1
reordered/refactored to put more responsibility in CrawlServer to resolve
fold in H3 updates to http transaction constant for revisit records.
* CoreAttributeConstants.java
removed A_HTTP_TRANSACTION, in favor checking CrawlURI method with isHttpTran$
* ModelAttributeConstants.java
removed A_HTTP_TRANSACTION, in favor checking CrawlURI method with isHttpTran$
* FetchHistoryProcessor.java
check for HTTP transaction using CrawlURI.isHttpTransaction()
use HttpMethod in saveHeader() instead of HttpMethodBase
* WARCWriterProcessor.java
check for HTTP transaction using CrawlURI.isHttpTransaction()
use HttpMethod in saveHeader() instead of HttpMethodBase
* CrawlURI.java
(getPolicyBasisUURI) return either own UURI, or -- for prereqs -- via UURI, as basis for overlays/policy calculations
* URIAuthorityBasedQueueAssignmentPolicy.java
use getPolicyBasisUURI for queue-name decisions
* SheetOverlaysManager.java
use getPolicyBasisUURI for surt-based overlay decisions
alternate simplified implementation of our object-cache need
* ObjectIdentityCache.java
new interface, far less than (Concurrent)Map, for big seems-like-in-memory object-cache
* Supplier.java
trivial interface for deferred-provision of new instance
* CachedBdbMap.java
implement ObjectIdentityCache, mainly via getOrUse()
* ObjectIdentityMemCache.java
trivial ConcurrentHashMap-based all-in-memory ObjectIdentityCache implementation
* ObjectIdentityBdbCache.java
BDB-backed ObjectIdentityCache implementation, carved from CachedBdbMap
* BdbModule.java
refactor utility methods to offer CachedBdbMap or ObjectIdentityBdbCache instances, via source toggle
* EnhancedEnvironment.java
convenience test-environment method
* (others)
update to use ObjectIdentityCache/ObjectIdentityMemCache in place of ConcurrentMap/ConcurrentHashMap
WARConstants.java
update WARC_VERSION=1.0 according to ISO 28500:2009 as of May 2009
WARCWriterProcessor.java
update format and conformsTo fields according to ISO 28500:2009,
as of May 2009, as described at http://bibnum.bnf.fr/WARC/,
latest draft as of November 2008
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf
* PersistProcessor.java
add readOnly option for environment only opened for copying-from
* PersistLoadProcessor.java
make start() safe to call redundantly (per Lifecycle contract)
* BdbModule.java
fail-fast if not started properly; stop with fewer errors in problem situations
* CrawlController.java
continue past runtimeException on stop
* CrawlJob.java
less redundant error-reporting
* AbstractFrontier.java
consider PAUSE or FINISH valid targets after unexpected exception
* StatisticsTracker.java
avoid promoting runtimeException in report-generating up
* PersistOnlineProcessor.java
delegate DB-closing to BdbModule to avoid double-close
* SurtPrefixedDecideRule.java
allow seeds to be null (as when rule is intentionally completely
independent of seeds; may also need to disqualify rule from
Spring autowiring in such a case)
* SurtPrefixSet.java
refactor to allow add-directives under outside control
* SurtPrefixedDecideRule.java
extend nonseedLine() mechanism to trigger off '+' or '-' depending on decision being ACCEPT or REJECT
* CrawlerJournal.java, IoUtils.java
move getBufferedReader to IoUtils
* FrontierJournal.java
add new 'included' tag for log lines ("Fi")
* (others)
update for relocated method
[HER-1660] allow adding seeds, recovery-journal-like URI data at any point in crawl via 'action directory'
[HER-769] Support case where millions of seeds.
* AbstractFrontier.java
move sourceTagSeeds setting to SeedModule
trigger and schedule seeds via SeedModule's announcements and SeedListener protocols
* StatisticsTracker.java
implement SeedListener, use processedSeedRecords as list of all seeds
* ProcessorURI.java, DefaultProcessorURI.java
(getURI) convenience accessor for plain URI string
* SurtPrefixedDecideRule.java
use seeds as prefixes via SeedListener notifications, rather than separate scan of seeds source text
* SeedListener.java
simplify to notifications of new seed, and nonseed text mixed with seeds (as with SURT directives)
* SeedModule.java
methods for SeedListener interface, announceSeeds, actOn(File)
* TextSeedModule.java
eliminate interators; add methods for announcing seeds and scanning seed text sources (including arbitrary Files)
* PrefixSet.java, SurtPrefixSet.java
derive from ConcurrentSkipListSet; eliminate now-unneeded extra synchronization and clone()s
offer considerAsDirective() for direct application of directive Strings
* LongToIntConsistentHash.java + Test
consistent-hashing utility class
* URIAuthorityBasedQueueAssignmentPolicy.java
shared superclass for the hostname and surtauthority QAPs
setting for 'deferToPrevious' -- avoid changing assignments
setting for 'parallelQueues' -- when > 1, consistent-hash URIs across that many separate queues (with numerical suffix)
* SurtAuthorityQueueAssignmentPolicy.java, HostnameQueueAssignmentPolicy.java
refactor to derive from URIAuthorityBasedQueueAssignmentPolicy
* QueueAssignmentPolicy.java
add Apache license
* Hop.java
add String representation for convenience
nlevitt