lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-07-10 17:47:23 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,985 Commits 40 Branches 31 Tags
master
Commit Graph

16 Commits

Author SHA1 Message Date
Edward Betts aa705bef2e Correct spelling mistakes. 2018-11-04 19:49:37 +09:00
Andrew Jackson 8384a59b47 Fix up tests to account for new Base URI behaviour. 2018-07-06 15:15:53 +01:00
Andrew Jackson d178c689de Add some comments to try to avoid misuse. 2017-10-12 22:13:00 +01:00
Andrew Jackson b0594ac0f4 Allow JavaDNS to be disabled and use that to avoid issues in StatisticsSelfTest. 2017-10-12 21:29:34 +01:00
Noah Levitt ce4a865764 add support to StatisticsTracker to keep a CrawledBytesHistotable per source tag when trackSources is enabled; integration test for this functionality 2015-12-03 16:13:34 -08:00
Noah Levitt 552200b2fa fix form login processor by saving login post data as url-encoded query string, which FetchHTTP now expects; tweaks to make FormLoginSelfTest pass 2014-03-31 20:38:17 -07:00
Noah Levitt 20a5f31e19 ExtractorHTMLForms+FormLoginProcessor full crawl job integration test (not passing yet) 2014-03-31 18:57:33 -07:00
Noah Levitt 16fa2b25f1 selftest-crawler-beans.cxml - replace obsolete BdbCookieStorage with BdbCookieStore 2014-01-12 13:11:25 -08:00
Noah Levitt 239f616c17 * **/.gitignore 
dummy files to make git keep these directories which are expected by unit tests
 2011-10-21 22:40:44 -07:00
gojomo 4221aee537 fix selftests 
* selftest-crawler-beans.cxml
    write logs, arcs to old constant locations, for ease of post-test verification
 2011-07-15 21:34:59 +00:00
gojomo a41809b8a7 [HER-1908] H3: upgrade to Spring 3 
* pom.xml, .classpath
    update references to necessary spring-3.0.5 packages
* SeedModule
    merge rather than replace event listeners, so that (now later) autowiring doesn't clobber the self-insertion done by anonymous (non-top-level) SurtPrefixedDecideRule bean
* HeritrixLifecycleProcessor, PathScharingContext
    use this new non-default LifecycleProcessor to avoid new Spring3 behavior of auto-start()ing context on refresh(build)
* CrawlController
    example of using @Value annotation to set default value: will offer benefits for auto-discovery of defaults for configuration interface, or enforcing maximally-explicit configurations (see [HER-1897])
* profile-crawler-beans.cxml, selftest-crawler-beans.cxml
    update templates with spring3 preamble/boilerplate
 2011-07-08 04:31:54 +00:00
gojomo d6a4d158b0 [HER-1803] H3: make it easier to change robots settings 
* RobotsPolicy.java
    simplified unification of RobotsHonoring and RobotsExclusion policy classes; subclasses specialize
* ObeyRobotsPolicy.java
    classic obey-robots-as-declared policy
* IgnoreRobotsPolicy.java
    ignore declared robots policy
* CustomRobotsPolicy.java
    follow custom-crafted policy in robots.txt format
* MostFavoredRobotsPolicy.java
    follow the most-permissive policy available to a set of user-agents; optionally adopt ('masquerade') the user-agent whose rules are chosen
* RobotsExclusionPolicy.java, RobotsHonoringPolicy.java
    deleted
* CrawlMetadata.java
    use new string robotsPolicyName as main determinant of policy
    maintain map of available policies ('obey','ignore', and any the operator added)
* CrawlServer.java
    retain Robotstxt instance rather than customized RobotsExclusionPolicy
* PreconditionEnforcer.java
    use full CrawlMetadata rather than just UserAgentProvider subset
    consult RobotsPolicy looked up through CrawlMetadata for allow/disallow decision
* DispositionProcessor.java
    update CrawlServer with Robotstxt only
    consult Robotstxt for crawl-delay
* profile-crawler-beans.cxml (*2), migrate-template-crawler-beans.cxml, selftest-crawler-beans.cxml
    remove old declarations/examples; add new
* ExtractorHTML.java, JerichoExtractorHTML.java
    consult RobotsPolicy.obeyMetaRobotsNofollow for NOFOLLOW
* Robotstxt.java
    new constructors for new use patterns
* **/*Test.java
    update/remove to match new use patterns
 2010-09-10 00:27:20 +00:00
gojomo f75f95648a [HER-1795] H3: Wrong schemaLocation in .cxml 
* **/*crawler-beans.cxml
    update all 2.0 spring schema refs to 2.5
 2010-07-02 00:58:23 +00:00
gojomo 047a157fe8 [HER-1605] Refactor parameterizable policies out of frontier 
* selftest-crawler.beans.cxml, FormAuthSelfTest.java, Precedence[134]SelfTest.java
    update self-tests for new chains refactoring
 2009-10-05 23:44:30 +00:00
gojomo b78fce045b Fix selftests hanging due to pauseAtStart default true 
* selftest-crawler-beans.cxml
    make selftests not pauseAtStart
 2009-08-13 10:03:56 +00:00
gojomo e047bf68e0 copy 'springy' branch to heritrix3 2009-05-11 22:56:36 +00:00