* WriterPoolProcessor.java
change description of WriterPoolProcessor.directory ConfigPath to not imply as strongly that is where writing much be occurring
* UURIFactoryTest.java
specify stirng literal with escape-encoding, rather than as raw UTF-8
* **/pom.xml
include 'project.build.sourceEncoding' property to suppress platform-encoding warnings
(unfortunately, neither this nor similar 'maven.compile.encoding' property actually casues maven build to interpret Java source as UTF-8)
offer html
* XmlMarshaller.java
new class to write xml based on nested map structure
* BeanBrowseResource.java ScriptResource.java EngineResource.java
JobResource.java JobRelatedResource.java
support xml representation
* WorkQueueFrontier.java PrecedenceProvider.java WorkQueue.java ToeThread.java
ToePool.java CrawlJob.java Engine.java Frontier.java CrawlController.java
StatisticsTracker.java CrawledBytesHistotable.java FetchStats.java
ProcessorChain.java CrawlURI.java Reporter.java
offer same data that is returned in the single line reports in a map
suitable for use with XmlMarshaller
* FileUtils.java
new method tryToCanonicalize(File)
* AbstractFrontier.java, WorkQueueFrontier.java, BdbFrontier.java
add 'futureUris' map of CrawlURIs to be reenqueued at specific future dates
the usual findEligibleUri also checks for newly-eligible rescheduled URIs
the frontier is no longer empty unless the future count is also 0
* CrawlURI.java
add rescheduleTime property
* ReschedulingProcessor.java
optional post-processor to set a CrawlURI's reschedulingTime
* CrawlStatSnapshot.java, CrawlJob.java
add futureUriCount to stats/uriTotalsReport
* PersistLogProcessor.java
rotate logs on checkpoint
* CrawlerJournal.java, FrontierJournal.java
rotate for checkpoint support without full Checkpointable
processor-related bean checkpoint support:
* Processor.java
implement Checkpointable and BeanNameAware; methods to support JSON state-saving
* DispositionChain.java
implement Checkpointable and during-checkpoint disposition-lockout for state stability
processor-related bean checkpoint support:
* Processor.java
implement Checkpointable and BeanNameAware; methods to support JSON state-saving
* DispositionChain.java
implement Checkpointable and during-checkpoint disposition-lockout for state stability
* **Extractor**.java
pull common reporting & checkpointing up to Extractor.java
* WriterPoolProcessor.java
implement Checkpointable; freeze and roll writer-pool during checkpoint
* PersistLogProcessor.java
roll log on checkpoint
* TextSeedModule.java, SeedModule.java
eliminate obsolete (and probably unnecessary) checkpoint support
* (others)
adapt to above changes
* IoUtils.java
remove class that confusingly-overlaps with commons IOUtils; move remaining methods to ArchiveUtils or archive's FileUtils
* ArchiveUtils.java, FileUtils.java
receive relocated methods; eliminated deprecated or unused methods
* JSONUtils.java
new class to collect common JSON actions
* Iteratorable.java
wrap Iterator as Iterable for foreach usage
* (many)
update to use alternate utils methods
* ServerCache.java
make abstract, implement getHostFor(UURI) and getServerFor(UURI)
* DefaultServerCache.java
extend ServerCache, remove getHostFor(UURI) and getServerFor(UURI)
* ServerCacheUtil.java
removed
* other classes
change usage of ServerCacheUtil to ServerCache
* DefaultServerCache.java
getHostFor(UURI): special case dns uris
* ServerCache.java
add getHostFor(UURI) to public interface
* ServerCacheUtil.java
remove now redundant wrapper method
* DispositionProcessor.java AbstractFrontier.java IPQueueAssignmentPolicy.java
BucketQueueAssignmentPolicy.java PreconditionEnforcer.java QuotaEnforcer.java
StatisticsTracker.java ServerCacheTest.java FetchHTTP.java
Kw3WriterProcessor.java WriterPoolProcessor.java
changed to use serverCache.getHostFor(UURI) (a few cases remain that use
getHostFor(String))
* ScopePlusOneDecideRule.java
removing, in favor of SurtPrefixedDecideRule's 'alsoCheckVia' option
* RecyclingFastBufferedOutputStream.java
use library by deriving as subclass (rather than c&p), now that class is available in an included library (fastutil)
* BdbModule.java
allow Database instance reuse if BdbConfig is identical instance
* PersistProcessor.java
convert shared BdbConfig for history db to static constant HISTORY_DB_CONFIG
* PersistOnlineProcessor.java
use HISTORY_DB_CONFIG, don't close possibly-shared DB on stop()
* PrecedenceLoader.java, PreloadedUriPrecedencePolicy.java
use HISTORY_DB_CONFIG
* ConfigPathConfigurer, PathFixupListener
notify beans when paths fixed
* SurtPrefixedDecideRule.java
only read source prefixes after path fixup; dump after read or any batch of seeds
* SeedModule.java, TextSeedModule.java, SeedListener.java
new concludedSeedBatch event
* AbstractFrontier.java, StatisticsTracker.java
ignore new concluded-batch event
* AlertThreadGroup.java, JobResource.java
new mechanism for threads not part of AlertThreadGroup to temporarily redirect their logging to provided (job) logger
* BdbModule.java
be more robust about 0/-1 as 'don't care' values
* AbstractFrontier.java
reorder settable properties for ease of discovery
* FetchHTTP.java
correct type to allow setting property
* profile-crawler-beans.cxml
add comments for all unstated default values an operator might want to change
* (many)
reorder, refactor, rename to better minimize/match simple configuration
* defaults.xml
update bundled profile for new chains refactoring
* CrawlController.java
move to three processor-chains rather than one
* Frontier.java
(loadSeeds) removed
* ToeThread.java
delegate most processing-loop to FetchChain and DispositionChain
* AbstractFrontier.java, WorkQueueFrontier.java
move policies/calculation out to processors
* CandidatesProcessor.java
new processor for DispositionChain that runs every outlink through CandidateChain
* CrawlStateUpdater.java -> DispositionProcessor.java
rename, expand to prep CrawlURI for frontier
* FrontierScheduler.java
deleted; use CandidatesProcessor/CandidateChain
* LinksScoper.java
deprecated; use CandidatesProcessor/CandidateChain/CandidateScoper
(only temporarily retained for ease of comparison)
* CandidateScoper.java
simple single-URI scope-testing for CandidateChain
* FrontierPreparer.java
precalculate all frontier-policies in CandidateChain, before scheduling
* PreconditionEnforcer.java
ProcessorURI->CrawlURI; take on some prerequisite preparation previously deferred to elsewhere
* ProcessorsReport.java
update for 3-chains of Processors
* SheetOverlaysManager.java
(applyOverridesTo) moved here for broader use
* CandidateChain.java, FetchChain.java, DispositionChain.java
role-specific subclasses of ProcessorChain (suitable for type-based autowiring)
* CrawlURI.java
new fields/accessors of use to new chains/frontier
* PostProcessor.java
deleted; skip-to-'postprocessing' is now skip-to-end-of-chain
* ProcessorChain.java
take-on control loop formerly in ToeThread
* ProcessResult.java
absorb ProcessStatus
eliminate problematic STUCK result
* (many)
update license notice to Apache where appropriate
eliminate ProcessorURI, DefaultProcessorURI in favor of CrawlURI (now in modules package)
other comment/warning/unused code cleanup
* BdbModule.java
split openDatabase() to openManagedDatabase() (auto-closed) and plain openDatabase() (caller-closes)
set TempStoredSortedMap to use unmanaged openDatabase
* (others)
use openManagedDatabase()
implementation in 1.14.3/heritrix2 to latest ISO document,
heritrix3 - in warcinfo record, "robots" field should be
"robots: classic" rather than "robots: CLASSIC"
* WARCWriterProcessor.java
made robots entry in metadata lower-case
* PreconditionEnforcer.java, CrawlServer.java
in testing found some ordering issues affecting robots-scheduling if parallelQueues>1
reordered/refactored to put more responsibility in CrawlServer to resolve
fold in H3 updates to http transaction constant for revisit records.
* CoreAttributeConstants.java
removed A_HTTP_TRANSACTION, in favor checking CrawlURI method with isHttpTran$
* ModelAttributeConstants.java
removed A_HTTP_TRANSACTION, in favor checking CrawlURI method with isHttpTran$
* FetchHistoryProcessor.java
check for HTTP transaction using CrawlURI.isHttpTransaction()
use HttpMethod in saveHeader() instead of HttpMethodBase
* WARCWriterProcessor.java
check for HTTP transaction using CrawlURI.isHttpTransaction()
use HttpMethod in saveHeader() instead of HttpMethodBase
* CrawlURI.java
(getPolicyBasisUURI) return either own UURI, or -- for prereqs -- via UURI, as basis for overlays/policy calculations
* URIAuthorityBasedQueueAssignmentPolicy.java
use getPolicyBasisUURI for queue-name decisions
* SheetOverlaysManager.java
use getPolicyBasisUURI for surt-based overlay decisions
gojomo