gojomo d6a4d158b0 [HER-1803] H3: make it easier to change robots settings ...

* RobotsPolicy.java
    simplified unification of RobotsHonoring and RobotsExclusion policy classes; subclasses specialize
* ObeyRobotsPolicy.java
    classic obey-robots-as-declared policy
* IgnoreRobotsPolicy.java
    ignore declared robots policy
* CustomRobotsPolicy.java
    follow custom-crafted policy in robots.txt format
* MostFavoredRobotsPolicy.java
    follow the most-permissive policy available to a set of user-agents; optionally adopt ('masquerade') the user-agent whose rules are chosen
* RobotsExclusionPolicy.java, RobotsHonoringPolicy.java
    deleted
* CrawlMetadata.java
    use new string robotsPolicyName as main determinant of policy
    maintain map of available policies ('obey','ignore', and any the operator added)
* CrawlServer.java
    retain Robotstxt instance rather than customized RobotsExclusionPolicy
* PreconditionEnforcer.java
    use full CrawlMetadata rather than just UserAgentProvider subset
    consult RobotsPolicy looked up through CrawlMetadata for allow/disallow decision
* DispositionProcessor.java
    update CrawlServer with Robotstxt only
    consult Robotstxt for crawl-delay
* profile-crawler-beans.cxml (*2), migrate-template-crawler-beans.cxml, selftest-crawler-beans.cxml
    remove old declarations/examples; add new
* ExtractorHTML.java, JerichoExtractorHTML.java
    consult RobotsPolicy.obeyMetaRobotsNofollow for NOFOLLOW
* Robotstxt.java
    new constructors for new use patterns
* **/*Test.java
    update/remove to match new use patterns

2010-09-10 00:27:20 +00:00

..

selftest

[HER-1803] H3: make it easier to change robots settings

2010-09-10 00:27:20 +00:00