new property textSource, an org.archive.io.ReadSource, allowing
specification inline in cxml as well as in external file
* profile-crawler-beans.cxml
commented out clauses demonstrating use of textSource
setting "independentExtractors" - when enabled, extractors run regardless of
whether other extractors have run
* profile-crawler-beans.cxml, AbstractFrontier.java, ExtractorParameters.java,
Extractor.java
new setting independentExtractors
* ContentExtractor.java
shouldProcess() - respect independentExtractors
* RobotsPolicy.java
simplified unification of RobotsHonoring and RobotsExclusion policy classes; subclasses specialize
* ObeyRobotsPolicy.java
classic obey-robots-as-declared policy
* IgnoreRobotsPolicy.java
ignore declared robots policy
* CustomRobotsPolicy.java
follow custom-crafted policy in robots.txt format
* MostFavoredRobotsPolicy.java
follow the most-permissive policy available to a set of user-agents; optionally adopt ('masquerade') the user-agent whose rules are chosen
* RobotsExclusionPolicy.java, RobotsHonoringPolicy.java
deleted
* CrawlMetadata.java
use new string robotsPolicyName as main determinant of policy
maintain map of available policies ('obey','ignore', and any the operator added)
* CrawlServer.java
retain Robotstxt instance rather than customized RobotsExclusionPolicy
* PreconditionEnforcer.java
use full CrawlMetadata rather than just UserAgentProvider subset
consult RobotsPolicy looked up through CrawlMetadata for allow/disallow decision
* DispositionProcessor.java
update CrawlServer with Robotstxt only
consult Robotstxt for crawl-delay
* profile-crawler-beans.cxml (*2), migrate-template-crawler-beans.cxml, selftest-crawler-beans.cxml
remove old declarations/examples; add new
* ExtractorHTML.java, JerichoExtractorHTML.java
consult RobotsPolicy.obeyMetaRobotsNofollow for NOFOLLOW
* Robotstxt.java
new constructors for new use patterns
* **/*Test.java
update/remove to match new use patterns
* LaxURI.java
increase tolerance with regard to path-segments
* UURIFactory.java
allow digits in scheme regex
discard old supported-schemes code
* UURIFactoryTest.java
tests for roundtrip serialization of problematic hostlike-scheme and bars-in-path
* CrawlerLoggerModule.java
discard special-casing of URI scheme errors
* SchemeNotInSetDecideRule.java
rule to apply to URIs of unknown schemes; by default REJECTs those not usually handled by Heritrix
* **/profile-crawler-beans.cxml
add REJECT SchemeNotInSetDecideRule
* logging.properties
streamline to bare essentials; allow INFO for org.archive classes
* GenericReplayCharSequence.java, BdbUriUniqFilter.java
demote some INFO logging to FINE
* profile-crawler-beans.cxml
Added a commented-out configuration for new DiskSpaceMonitor
* DiskSpaceMonitor.java
Implements the required functionality
* package-info.java
Description of new package for monitoring utilities
* LowDiskPauseProcessor.java
Now marked as deprecated
* AbstractFrontier.java
change config and defaults for 'inbound' and 'outbound': allowing any BlockinqQueue bean, of any size, to be specified if desired
* **/profile-crawler-beans.cxml
changed commented-out descriptions of defaults to reflect what's used if nothing is specified for 'outbound' and 'inbound'
* dist/src/main/conf/jobs/profile-defaults/profile-crawler-beans.cxml
* engine/src/main/resources/org/archive/crawler/restlet/profile-crawler-beans.cxml
add intended REJECT decision for the late (rules[4]) MatchesListRegexDecideRule
* **/pom.xml
bump version identifiers to "3.1.1-SNAPSHOT" (odd minor number meaning dev version, odd patch number indicating expected next patch number of 3.0.x release if before a 3.2.0 release)
* UURIFactoryTest.java
specify stirng literal with escape-encoding, rather than as raw UTF-8
* **/pom.xml
include 'project.build.sourceEncoding' property to suppress platform-encoding warnings
(unfortunately, neither this nor similar 'maven.compile.encoding' property actually casues maven build to interpret Java source as UTF-8)
* profile-crawler-beans.cxml
add comments for all unstated default values an operator might want to change
* (many)
reorder, refactor, rename to better minimize/match simple configuration
* defaults.xml
update bundled profile for new chains refactoring
* CrawlController.java
move to three processor-chains rather than one
* Frontier.java
(loadSeeds) removed
* ToeThread.java
delegate most processing-loop to FetchChain and DispositionChain
* AbstractFrontier.java, WorkQueueFrontier.java
move policies/calculation out to processors
* CandidatesProcessor.java
new processor for DispositionChain that runs every outlink through CandidateChain
* CrawlStateUpdater.java -> DispositionProcessor.java
rename, expand to prep CrawlURI for frontier
* FrontierScheduler.java
deleted; use CandidatesProcessor/CandidateChain
* LinksScoper.java
deprecated; use CandidatesProcessor/CandidateChain/CandidateScoper
(only temporarily retained for ease of comparison)
* CandidateScoper.java
simple single-URI scope-testing for CandidateChain
* FrontierPreparer.java
precalculate all frontier-policies in CandidateChain, before scheduling
* PreconditionEnforcer.java
ProcessorURI->CrawlURI; take on some prerequisite preparation previously deferred to elsewhere
* ProcessorsReport.java
update for 3-chains of Processors
* SheetOverlaysManager.java
(applyOverridesTo) moved here for broader use
* CandidateChain.java, FetchChain.java, DispositionChain.java
role-specific subclasses of ProcessorChain (suitable for type-based autowiring)
* CrawlURI.java
new fields/accessors of use to new chains/frontier
* PostProcessor.java
deleted; skip-to-'postprocessing' is now skip-to-end-of-chain
* ProcessorChain.java
take-on control loop formerly in ToeThread
* ProcessResult.java
absorb ProcessStatus
eliminate problematic STUCK result
* HOWTO-Launch-Heritrix.txt
removed; info mostly about H2 variants
* README.txt
updated wiki link, launch info to match H3
* LICENSE.txt
updated to Apache2
nlevitt