lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-07-31 20:01:18 +00:00
Code Issues Packages Projects Releases Wiki Activity
450 Commits 40 Branches 31 Tags
9bfa8fc1dbb9bcd576ca4e63539ec86e672a8564
Commit Graph
36 Commits
Author SHA1 Message Date
kristinn_sig c410c90b76 Implementation of [HER-1811] Replace the LowDiskPauseProcessor with something that works 
* profile-crawler-beans.cxml
    Added a commented-out configuration for new DiskSpaceMonitor
* DiskSpaceMonitor.java
    Implements the required functionality
* package-info.java
    Description of new package for monitoring utilities
* LowDiskPauseProcessor.java
    Now marked as deprecated
 2010-08-19 10:47:22 +00:00
gojomo f75f95648a [HER-1795] H3: Wrong schemaLocation in .cxml 
* **/*crawler-beans.cxml
    update all 2.0 spring schema refs to 2.5
 2010-07-02 00:58:23 +00:00
gojomo 943ecf3088 [HER-1762] H3 performance tuning 
* AbstractFrontier.java
    change config and defaults for 'inbound' and 'outbound': allowing any BlockinqQueue bean, of any size, to be specified if desired
* **/profile-crawler-beans.cxml
    changed commented-out descriptions of defaults to reflect what's used if nothing is specified for 'outbound' and 'inbound'
 2010-06-18 21:17:40 +00:00
gojomo e1f76dbc07 [HER-1784] default profile MatchesListRegexDecideRule intended to REJECT doesn't 
* dist/src/main/conf/jobs/profile-defaults/profile-crawler-beans.cxml
* engine/src/main/resources/org/archive/crawler/restlet/profile-crawler-beans.cxml
    add intended REJECT decision for the late (rules[4]) MatchesListRegexDecideRule
 2010-06-14 22:22:34 +00:00
szznax d1f27aaecd removed no-longer-referenced "recoveryDir" property from "frontier" bean 2010-05-27 17:46:44 +00:00
gojomo 67bf73026f remove obsolete/moved example properties 2010-04-23 04:05:31 +00:00
gojomo cb7df416d4 post-3.0.0 release build 
* **/pom.xml
    bump version identifiers to "3.1.1-SNAPSHOT" (odd minor number meaning dev version, odd patch number indicating expected next patch number of 3.0.x release if before a 3.2.0 release)
 2009-12-05 23:48:38 +00:00
gojomo 5661658b1b prep for 3.0.0 release 
* **/pom.xml
    bump version-id to "3.0.0"
* README.txt
    refer to exactly 3.0.0 release notes
 2009-12-05 09:28:19 +00:00
gojomo ac255d0ebe Prep for 3.0.0 
* README.txt
    updated, pointers to wiki references
 2009-12-05 00:12:14 +00:00
szznax d423af6838 changed cachePercentage to cachePercent 2009-12-04 22:09:49 +00:00
nlevitt a49159a2cc Use specific includes to define contents of source distribution, to avoid 
packaging whatever junk might be lying around your source checkout.
* dist/src/main/assembly/src.xml
 2009-12-03 22:42:12 +00:00
gojomo ed2061560e Improve Maven build robustness 
* UURIFactoryTest.java
    specify stirng literal with escape-encoding, rather than as raw UTF-8
* **/pom.xml
    include 'project.build.sourceEncoding' property to suppress platform-encoding warnings
    (unfortunately, neither this nor similar 'maven.compile.encoding' property actually casues maven build to interpret Java source as UTF-8)
 2009-12-02 01:34:40 +00:00
nlevitt aa3bfc7f4c Remove unused jasper dependency. 
* .classpath
* dist/pom.xml
 2009-11-26 01:43:15 +00:00
gojomo 79fd64a2b5 Post-RC1-build 
* **/pom.xml
    revert version name to SNAPSHOT
 2009-11-23 10:23:28 +00:00
gojomo 0c7a74bd35 RC1 
* **/pom.xml
    rename version to '3.0.0-RC1' for distinguished 'RC1' identified build (instead of SNAPSHOT/timestamp)
 2009-11-23 09:57:42 +00:00
gojomo 8251620481 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls 
* profile-crawler-beans.cxml*2, migrate-template-crawler-beans.cxml
    add CheckpointService as recommended bean
 2009-11-20 18:35:13 +00:00
nlevitt 94e38c5b99 * logging.properties: default PersistLoadProcessor log level INFO to see if 
dedup info is loaded
 2009-11-13 01:50:13 +00:00
nlevitt 2740cc7bb0 Change the artifactId from commons to heritrix-commons, modules to 
heritrix-modules, engine to heritrix-engine.
 2009-11-12 02:48:38 +00:00
nlevitt fe1a852974 Finishing up [HER-1561] 
* LICENSE.txt: list individual source files that are not (yet) apache licensed
* pom.xml: include apache license in <licenses> section
 2009-11-11 03:23:09 +00:00
gojomo 69736535a7 * profile-crawler-beans.cxml 
correct missing <property> tags in example commented-out canonicalization rules
 2009-10-23 22:34:41 +00:00
gojomo 4a0f809c0d Post-beta-build 
* **/pom.xml
    revert version name to SNAPSHOT
 2009-10-07 11:54:12 +00:00
gojomo af9345a3be * **/pom.xml 
rename version to 'beta' for distinguished 'beta' identified build (instead of SNAPSHOT/timestamp)
 2009-10-07 11:25:20 +00:00
gojomo 21353cc26d more profile cleanup 
* profile-crawler-beans.cxml
   add overlooked AbstractFrontier settings
 2009-10-07 01:39:30 +00:00
gojomo eb659a9fad more profile cleanup 
* profile-crawler-beans.cxml
   add alsoCheckVia
   improve description of commented-out defaults
   add CrawlLimitEnforcer
* SurtPrefixedDecideRule.java
    remove obsolete property
 2009-10-07 01:05:59 +00:00
gojomo 258da53d57 [HER-1676] improve bundled profiles to match best-practice, be self-documenting 
* profile-crawler-beans.cxml
    add comments for all unstated default values an operator might want to change
* (many) 
    reorder, refactor, rename to better minimize/match simple configuration
 2009-10-06 19:35:05 +00:00
gojomo 6f7301d994 [HER-1676] improve bundled profiles to match best-practice, be self-documenting 
* defaults.xml
   deleted
 2009-10-06 04:16:52 +00:00
gojomo 5c50f9c104 [HER-1676] improve bundled profiles to match best-practice, be self-documenting 
* profile-crawler-beans.cxml
    merge 'defaults' back in, beef up comments, strip unnecessary declarations
 2009-10-06 04:07:25 +00:00
gojomo 13c9cc1915 [HER-1605] Refactor parameterizable policies out of frontier 
* defaults.xml
    update bundled profile for new chains refactoring
* CrawlController.java
    move to three processor-chains rather than one
* Frontier.java
    (loadSeeds) removed
* ToeThread.java
    delegate most processing-loop to FetchChain and DispositionChain
* AbstractFrontier.java, WorkQueueFrontier.java
    move policies/calculation out to processors
* CandidatesProcessor.java
    new processor for DispositionChain that runs every outlink through CandidateChain
* CrawlStateUpdater.java -> DispositionProcessor.java
    rename, expand to prep CrawlURI for frontier
* FrontierScheduler.java
    deleted; use CandidatesProcessor/CandidateChain
* LinksScoper.java
    deprecated; use CandidatesProcessor/CandidateChain/CandidateScoper
    (only temporarily retained for ease of comparison)
* CandidateScoper.java
    simple single-URI scope-testing for CandidateChain
* FrontierPreparer.java
    precalculate all frontier-policies in CandidateChain, before scheduling
* PreconditionEnforcer.java
    ProcessorURI->CrawlURI; take on some prerequisite preparation previously deferred to elsewhere
* ProcessorsReport.java
    update for 3-chains of Processors
* SheetOverlaysManager.java
    (applyOverridesTo) moved here for broader use
* CandidateChain.java, FetchChain.java, DispositionChain.java
    role-specific subclasses of ProcessorChain (suitable for type-based autowiring)
* CrawlURI.java
    new fields/accessors of use to new chains/frontier
* PostProcessor.java
    deleted; skip-to-'postprocessing' is now skip-to-end-of-chain
* ProcessorChain.java
    take-on control loop formerly in ToeThread
* ProcessResult.java
    absorb ProcessStatus
    eliminate problematic STUCK result
 2009-10-05 23:41:59 +00:00
gojomo d955e31963 [HER-1676] improve bundled profiles to match best-practice, be self-documenting 
* defaults.xml
    add REJECTing SurtPrefixedDecideRule and MatchesListRegExpDecideRule
 2009-09-10 22:39:26 +00:00
gojomo c721d34fde [HER-1642] fix HOWTO-launch document 
* HOWTO-Launch-Heritrix.txt
    removed; info mostly about H2 variants
* README.txt
    updated wiki link, launch info to match H3
* LICENSE.txt
    updated to Apache2
 2009-07-13 23:06:23 +00:00
gojomo 721089044a * **/pom.xml 
revert version name to SNAPSHOT
 2009-06-11 09:09:43 +00:00
gojomo e8e5fdd9a4 * **/pom.xml 
rename version to 'alpha' for distinguished 'alpha' identified build (instead of SNAPSHOT/timestamp)
 2009-06-11 08:06:19 +00:00
gojomo 3862c361a4 Fixes from trial guide testing 
* defaults.xml
    remove error-causing redundant BdbCookieStorage declaration
* TextUtils.java
    exception- and deprecation-free UTF8 URL escaping
* CrawlJob.java, JobResource.java
    ensure job log shows INFO-level items
    better handle job directories with spaces in them
 2009-06-08 23:30:55 +00:00
gojomo 7ca3fbff72 * heritrix.cmd 
remove JMX-related paths; streamline comments; link to Heritrix3-on-Windows wiki page
 2009-05-28 00:05:44 +00:00
gojomo daf5ed7590 Update pom.xml * 4 
more naming improvements
    ignore test cruft
    update SCM url
 2009-05-12 00:17:01 +00:00
gojomo e047bf68e0 copy 'springy' branch to heritrix3 2009-05-11 22:56:36 +00:00