* profile-crawler-beans.cxml
add comments for all unstated default values an operator might want to change
* (many)
reorder, refactor, rename to better minimize/match simple configuration
* defaults.xml
update bundled profile for new chains refactoring
* CrawlController.java
move to three processor-chains rather than one
* Frontier.java
(loadSeeds) removed
* ToeThread.java
delegate most processing-loop to FetchChain and DispositionChain
* AbstractFrontier.java, WorkQueueFrontier.java
move policies/calculation out to processors
* CandidatesProcessor.java
new processor for DispositionChain that runs every outlink through CandidateChain
* CrawlStateUpdater.java -> DispositionProcessor.java
rename, expand to prep CrawlURI for frontier
* FrontierScheduler.java
deleted; use CandidatesProcessor/CandidateChain
* LinksScoper.java
deprecated; use CandidatesProcessor/CandidateChain/CandidateScoper
(only temporarily retained for ease of comparison)
* CandidateScoper.java
simple single-URI scope-testing for CandidateChain
* FrontierPreparer.java
precalculate all frontier-policies in CandidateChain, before scheduling
* PreconditionEnforcer.java
ProcessorURI->CrawlURI; take on some prerequisite preparation previously deferred to elsewhere
* ProcessorsReport.java
update for 3-chains of Processors
* SheetOverlaysManager.java
(applyOverridesTo) moved here for broader use
* CandidateChain.java, FetchChain.java, DispositionChain.java
role-specific subclasses of ProcessorChain (suitable for type-based autowiring)
* CrawlURI.java
new fields/accessors of use to new chains/frontier
* PostProcessor.java
deleted; skip-to-'postprocessing' is now skip-to-end-of-chain
* ProcessorChain.java
take-on control loop formerly in ToeThread
* ProcessResult.java
absorb ProcessStatus
eliminate problematic STUCK result
* ArchiveUtils.java
(checkContinue) utility method relocated from ToeThread for converting interrupt status to exception
* ClassKeyMatchesRegExpDecideRule.java
properly set up crawlController property for auto-wiring
* (many)
update license notice to Apache where appropriate
eliminate ProcessorURI, DefaultProcessorURI in favor of CrawlURI (now in modules package)
other comment/warning/unused code cleanup
* ObjectIdentityBdbCache.java
explicitly page-out lingering entry;
temporary System.err logging to see if problem is momentary lag in RefQueue-appearance
* BdbModule.java
split openDatabase() to openManagedDatabase() (auto-closed) and plain openDatabase() (caller-closes)
set TempStoredSortedMap to use unmanaged openDatabase
* (others)
use openManagedDatabase()
* BdbModule.java
add temp-stored-map service method
remove unused secondary-db support
* TempStoredSortedMap.java
StoredSortedMap that can destroy its underlying database after temporary use
* StatisticsTracker.java
update various sorted-by-decreasing-frequency methods to use temp StoredMaps, with duplicate keys that are negative counts
* (Multiple)Report.java
update to use new duplicate-keyed frequency maps
problematic (illegal control character?) outlink
catch IllegalArgumentException in ANVLRecord.addLabelValue() to allow
WARCWriter to continue processing, and move
WARCWriter.createRecordHeader() into its own try-catch block to with
logging prevent fouling record when problems occur creating header.
* ANVLRecord.java
catch IllegalArgumentException in addLabelValue and log warning to
give WARCWriter to continue processing
* WARCWriter.java
move createRecordHeader() into its own try-catch block to log
severe problem writing record without fouling the WARC, and have
baseCharacterCheck() throw IllegalArgumentException in accord with
ANVLRecord.addLabelValue
* WARCWriterTest.java
throw IllegalArgumentException in accord with
WARCWriter.createRecordHeader()
implementation in 1.14.3/heritrix2 to latest ISO document,
heritrix3 - in warcinfo record, "robots" field should be
"robots: classic" rather than "robots: CLASSIC"
* WARCWriterProcessor.java
made robots entry in metadata lower-case
* BdbModuleTest.java, CachedBdbMapTest.java
use putIfAbsent() in preference to put() (which logs warning)
* BdbModule.java
register big-cache DBs in implementation-specific methods, so always auto-closed on env close
* PreconditionEnforcer.java, CrawlServer.java
in testing found some ordering issues affecting robots-scheduling if parallelQueues>1
reordered/refactored to put more responsibility in CrawlServer to resolve
fold in H3 updates to http transaction constant for revisit records.
* CoreAttributeConstants.java
removed A_HTTP_TRANSACTION, in favor checking CrawlURI method with isHttpTran$
* ModelAttributeConstants.java
removed A_HTTP_TRANSACTION, in favor checking CrawlURI method with isHttpTran$
* FetchHistoryProcessor.java
check for HTTP transaction using CrawlURI.isHttpTransaction()
use HttpMethod in saveHeader() instead of HttpMethodBase
* WARCWriterProcessor.java
check for HTTP transaction using CrawlURI.isHttpTransaction()
use HttpMethod in saveHeader() instead of HttpMethodBase
szznax