* BdbModule.java
allow Database instance reuse if BdbConfig is identical instance
* PersistProcessor.java
convert shared BdbConfig for history db to static constant HISTORY_DB_CONFIG
* PersistOnlineProcessor.java
use HISTORY_DB_CONFIG, don't close possibly-shared DB on stop()
* PrecedenceLoader.java, PreloadedUriPrecedencePolicy.java
use HISTORY_DB_CONFIG
* ConfigPathConfigurer, PathFixupListener
notify beans when paths fixed
* SurtPrefixedDecideRule.java
only read source prefixes after path fixup; dump after read or any batch of seeds
* SeedModule.java, TextSeedModule.java, SeedListener.java
new concludedSeedBatch event
* AbstractFrontier.java, StatisticsTracker.java
ignore new concluded-batch event
* AlertThreadGroup.java, JobResource.java
new mechanism for threads not part of AlertThreadGroup to temporarily redirect their logging to provided (job) logger
* URIAuthorityBasedQueueAssignmentPolicy.java
choose numbered subqueue on first-path-segment only, so similar URIs land in same subqueue
* CrawlMapper.java
add queueKey to diversion lines
* HashCrawlMapper.java
update license
* Engine.java
protect against disappeared jobsDir in findJobConfigs()
check for .jobpath files in findJobConfigs()
added considerAsJobPath()
log added jobs in considerAsJobDirectory()
make considerAsJobDirectory() return true when jobConfig (already) exists
added leaveJobPathFile() to write .jobpath file for newly added jobs
* EngineResource.java
call leaveJobPathFile() when job directory added successfully
added messageDiv() to make messages more uniform and prominent
re-scan jobConfigs on each page load - may obviate need for "rescan" button
* Engine.java
createNewJobWithDefaults() to write profile-crawler-beans.cxml resoure into new job
* EngineResource.java
added FORM for new job dir and "create" action
* profile-crawler-beans.cxml
added resource from dist/src/main/conf/jobs/profile-defaults to bootstrap new jobs
* Engine.java
return 'false' rather than NPE when given directory non-existent/contains no .cxml
* EngineResource.java
better handle empty 'add' path; show NACK flash when 'add' has no effect
* H1toH3.map, migrate-template-crawler-beans.cxml, .classpath
move resources to 'resources' src subdirectory, so included in built JAR
move 'resources' directorys to library rather than source directories for dev-time reachability
* MigrateH1to3Tool.java
handle H1 values that just need capitalization
report all 'no rule' situations as 'needs attention'
touchup explanations and comments
enable uriCanonicalizationRule overrides
* BdbModule.java
be more robust about 0/-1 as 'don't care' values
* AbstractFrontier.java
reorder settable properties for ease of discovery
* FetchHTTP.java
correct type to allow setting property
* profile-crawler-beans.cxml
add comments for all unstated default values an operator might want to change
* (many)
reorder, refactor, rename to better minimize/match simple configuration
* defaults.xml
update bundled profile for new chains refactoring
* CrawlController.java
move to three processor-chains rather than one
* Frontier.java
(loadSeeds) removed
* ToeThread.java
delegate most processing-loop to FetchChain and DispositionChain
* AbstractFrontier.java, WorkQueueFrontier.java
move policies/calculation out to processors
* CandidatesProcessor.java
new processor for DispositionChain that runs every outlink through CandidateChain
* CrawlStateUpdater.java -> DispositionProcessor.java
rename, expand to prep CrawlURI for frontier
* FrontierScheduler.java
deleted; use CandidatesProcessor/CandidateChain
* LinksScoper.java
deprecated; use CandidatesProcessor/CandidateChain/CandidateScoper
(only temporarily retained for ease of comparison)
* CandidateScoper.java
simple single-URI scope-testing for CandidateChain
* FrontierPreparer.java
precalculate all frontier-policies in CandidateChain, before scheduling
* PreconditionEnforcer.java
ProcessorURI->CrawlURI; take on some prerequisite preparation previously deferred to elsewhere
* ProcessorsReport.java
update for 3-chains of Processors
* SheetOverlaysManager.java
(applyOverridesTo) moved here for broader use
* CandidateChain.java, FetchChain.java, DispositionChain.java
role-specific subclasses of ProcessorChain (suitable for type-based autowiring)
* CrawlURI.java
new fields/accessors of use to new chains/frontier
* PostProcessor.java
deleted; skip-to-'postprocessing' is now skip-to-end-of-chain
* ProcessorChain.java
take-on control loop formerly in ToeThread
* ProcessResult.java
absorb ProcessStatus
eliminate problematic STUCK result
gojomo