lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-22 14:36:37 +00:00
Code Issues Packages Projects Releases Wiki Activity
234 Commits 40 Branches 31 Tags
648ef7deef94d46132b375e937ef2ad331963b5f
Commit Graph
234 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
gojomo 648ef7deef Fix issues with surt-source, surt-dump file paths 
* ConfigPathConfigurer, PathFixupListener
    notify beans when paths fixed
* SurtPrefixedDecideRule.java
    only read source prefixes after path fixup; dump after read or any batch of seeds
* SeedModule.java, TextSeedModule.java, SeedListener.java
    new concludedSeedBatch event 
* AbstractFrontier.java, StatisticsTracker.java
    ignore new concluded-batch event
* AlertThreadGroup.java, JobResource.java
    new mechanism for threads not part of AlertThreadGroup to temporarily redirect their logging to provided (job) logger
 2009-10-26 11:29:58 +00:00
gojomo 59003e22ab Fix issues with surt-source, surt-dump file paths 
* ConfigPathConfigurer, PathFixupListener
    notify beans when paths fixed
* SurtPrefixedDecideRule.java
    only read source prefixes after path fixup; dump after read or any batch of seeds
* SeedModule.java, TextSeedModule.java, SeedListener.java
    new concludedSeedBatch event 
* AbstractFrontier.java, StatisticsTracker.java
    ignore new concluded-batch event
* AlertThreadGroup.java, JobResource.java
    new mechanism for threads not part of AlertThreadGroup to temporarily redirect their logging to provided (job) logger
 2009-10-26 11:27:09 +00:00
gojomo 4ef460ac00 Crawl-distribution-related improvements 
* URIAuthorityBasedQueueAssignmentPolicy.java
    choose numbered subqueue on first-path-segment only, so similar URIs land in same subqueue
* CrawlMapper.java
    add queueKey to diversion lines
* HashCrawlMapper.java
    update license
 2009-10-26 09:34:12 +00:00
gojomo 2033c99fea * AbstractFrontier.java 
promote priority of manageThread (where that is respected)
    remove stray debug output
 2009-10-23 22:38:21 +00:00
gojomo 69736535a7 * profile-crawler-beans.cxml 
correct missing <property> tags in example commented-out canonicalization rules
 2009-10-23 22:34:41 +00:00
gojomo 2e31d58687 * ObjectIdentityBdbCache.java 
replace costly bdb size() operation with maintained counter
 2009-10-23 22:33:36 +00:00
gojomo 06223b2794 BDB JE tuning 
* BdbModule.java
    increase je.log.faultReadSize from 2K to 6K -- stats showed many faults (ideal is zero)
 2009-10-23 21:55:06 +00:00
gojomo f0c9608e59 Followup for [HER-1605] Refactor parameterizable policies out of frontier 
* CandidatesProcessor.java
    remove misguided lock preventing intended parallelism of pre-enqueue URL prep
 2009-10-21 22:55:06 +00:00
gojomo ea114357fb * FrontierPreparer.java 
fix inconsistency in canonicalizationPolicy getter/setter names making explicit spring-config impossible
 2009-10-20 21:32:26 +00:00
szznax efee001bbc Fix for [HER-1688] remember external job directories 
* Engine.java
    protect against disappeared jobsDir in findJobConfigs()
    check for .jobpath files in findJobConfigs()
    added considerAsJobPath()
    log added jobs in considerAsJobDirectory()
    make considerAsJobDirectory() return true when jobConfig (already) exists
    added leaveJobPathFile() to write .jobpath file for newly added jobs
* EngineResource.java
    call leaveJobPathFile() when job directory added successfully
    added messageDiv() to make messages more uniform and prominent
    re-scan jobConfigs on each page load - may obviate need for "rescan" button
 2009-10-15 01:03:29 +00:00
szznax 8311db6a26 minor style change 2009-10-09 16:58:54 +00:00
szznax 72136dea31 handle null path on "create" action 2009-10-09 00:04:31 +00:00
szznax 1a136b21d5 Fix for [HER-1419] can't create job in web ui without profile/job to base it on in jobs dir? 
* Engine.java
  createNewJobWithDefaults() to write profile-crawler-beans.cxml resoure into new job
* EngineResource.java
  added FORM for new job dir and "create" action
* profile-crawler-beans.cxml
  added resource from dist/src/main/conf/jobs/profile-defaults to bootstrap new jobs
 2009-10-08 23:55:42 +00:00
gojomo 58513a8940 [HER-1687] H3 web UI: adding non-existent job directory gives unhelpful NPE 
* Engine.java
    return 'false' rather than NPE when given directory non-existent/contains no .cxml
* EngineResource.java
    better handle empty 'add' path; show NACK flash when 'add' has no effect
 2009-10-08 20:57:04 +00:00
gojomo 4a0f809c0d Post-beta-build 
* **/pom.xml
    revert version name to SNAPSHOT
 2009-10-07 11:54:12 +00:00
gojomo af9345a3be * **/pom.xml 
rename version to 'beta' for distinguished 'beta' identified build (instead of SNAPSHOT/timestamp)
 2009-10-07 11:25:20 +00:00
gojomo 64a79ea9e6 Fix issues revealed in extra tests 
* H1toH3.map, migrate-template-crawler-beans.cxml, .classpath
    move resources to 'resources' src subdirectory, so included in built JAR
    move 'resources' directorys to library rather than source directories for dev-time reachability
* MigrateH1to3Tool.java
    handle H1 values that just need capitalization
    report all 'no rule' situations as 'needs attention' 
    touchup explanations and comments
    enable uriCanonicalizationRule overrides
 2009-10-07 10:24:05 +00:00
gojomo f114448e6f Fix issues revealed in extra tests 
* H1toH3.map, migrate-template-crawler-beans.cxml, .classpath
    move resources to 'resources' src subdirectory, so included in built JAR
    move 'resources' directorys to library rather than source directories for dev-time reachability
* MigrateH1to3Tool.java
    handle H1 values that just need capitalization
    report all 'no rule' situations as 'needs attention' 
    touchup explanations and comments
    enable uriCanonicalizationRule overrides
 2009-10-07 10:23:36 +00:00
gojomo 45f09818ae Fix intermittent ClassCastExceptions on job console 
* Histotable.java, ToePool.java
    only tally Strings; convert all histotable keys to strings for sorting
 2009-10-07 10:18:21 +00:00
gojomo f960a018c1 [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 
* MigrateH1to3Tool.java
    add count of successfully migrated settings
 2009-10-07 05:04:39 +00:00
gojomo 9b3a65ab3a [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 
* MigrateH1to3Tool.java
    improve output: explain skipped settings, affected paths
* H1toH3.map
    truncate $na to $
 2009-10-07 05:01:16 +00:00
gojomo 47cbdded9a [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 
* MigrateH1to3Tool.java
    comment, clean unused code
* H1toH3.map
    text map of {H3-override-beanpath, H1-settings-XPath, H1-default-value}
* migrate-template-crawler-beans.cxml
    version of bundled profile slightly changed for better mapping to H1
    (target replace range, seeds.txt, arcWriter)
 2009-10-07 04:28:10 +00:00
gojomo b7631e2b02 Misc fixes from H1-to-H3 testing 
* BdbModule.java
    be more robust about 0/-1 as 'don't care' values
* AbstractFrontier.java
    reorder settable properties for ease of discovery
* FetchHTTP.java
    correct type to allow setting property
 2009-10-07 04:22:05 +00:00
gojomo 07adde3a73 Fix impoliteness regression 
* CrawlURI.java
    return true calculated politeness delay, not 0!
 2009-10-07 04:20:11 +00:00
gojomo a582596092 Fix bean browser 
* StatisticsTracker.java, SeedsReport.java
    avoid naming method as getter, causing bean browser problems
 2009-10-07 04:19:29 +00:00
szznax f097c07d51 Updates to be merged ... 2009-10-07 02:06:06 +00:00
gojomo 21353cc26d more profile cleanup 
* profile-crawler-beans.cxml
   add overlooked AbstractFrontier settings
 2009-10-07 01:39:30 +00:00
gojomo 791df1b78e [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 
* H1toH3.map
    order-XPath-to-override-key map for translating simple H1 settings to H3 overrides
 2009-10-07 01:23:37 +00:00
gojomo 78b3b0101f [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 
* H1toH3.map
    order-XPath-to-override-key map for translating simple H1 settings to H3 overrides
 2009-10-07 01:14:44 +00:00
gojomo 4cf3c0b151 [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 
* H1toH3.map
    order-XPath-to-override-key map for translating simple H1 settings to H3 overrides
 2009-10-07 01:11:12 +00:00
gojomo 329204b9ff [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 
* MigrateSettings.java, MigrateH1toH3Tool.java
    renamings, 'flatten' method for building map
* H1toH3.map
    order-XPath-to-override-key map for translating simple H1 settings to H3 overrides
 2009-10-07 01:07:33 +00:00
gojomo eb659a9fad more profile cleanup 
* profile-crawler-beans.cxml
   add alsoCheckVia
   improve description of commented-out defaults
   add CrawlLimitEnforcer
* SurtPrefixedDecideRule.java
    remove obsolete property
 2009-10-07 01:05:59 +00:00
szznax e2ee037786 Working commit... 2009-10-06 22:24:19 +00:00
gojomo 258da53d57 [HER-1676] improve bundled profiles to match best-practice, be self-documenting 
* profile-crawler-beans.cxml
    add comments for all unstated default values an operator might want to change
* (many) 
    reorder, refactor, rename to better minimize/match simple configuration
 2009-10-06 19:35:05 +00:00
szznax 6b1d1963d8 Incremental checkin. 2009-10-06 18:06:46 +00:00
gojomo 6f7301d994 [HER-1676] improve bundled profiles to match best-practice, be self-documenting 
* defaults.xml
   deleted
 2009-10-06 04:16:52 +00:00
gojomo 3f7d40412b Fix broken unit tests 
* CandidatesProcessor.java
    properly set full-via CrawlURI on prereqs
 2009-10-06 04:16:14 +00:00
gojomo 742e4e8cad [HER-1676] improve bundled profiles to match best-practice, be self-documenting 
* DispositionProcessor.java
    indirect robots-through-metadata, to obviate explicit declaration in beans.xml
 2009-10-06 04:14:29 +00:00
gojomo e2a61792a7 * (many) 
harmonize our usage on '[Rr]egex' as the abbrieviation for regular-expression
    (replacing RegExp/etc.; 'regex' is 30-40% more prevalent by google hits)
 2009-10-06 04:09:49 +00:00
gojomo 5c50f9c104 [HER-1676] improve bundled profiles to match best-practice, be self-documenting 
* profile-crawler-beans.cxml
    merge 'defaults' back in, beef up comments, strip unnecessary declarations
 2009-10-06 04:07:25 +00:00
gojomo da95932598 * CrawlMetadata.java 
set default RobotsHonoringPolicy, making explicit declaration unnecessary
 * RobotsHonoringPolicy.java
    apache license
 2009-10-06 04:00:58 +00:00
gojomo 2d3f569f15 * InterruptibleCharSequenceTest.java 
better output to diagnose intermittent failure
 2009-10-06 03:58:22 +00:00
gojomo 3654e330cc * (many) 
harmonize our usage on '[Rr]egex' as the abbrieviation for regular-expression
    (replacing RegExp/etc.; 'regex' is 30-40% more prevalent by google hits)
 2009-10-06 03:47:38 +00:00
gojomo d86f5f7403 * (many) 
harmonize our usage on '[Rr]egex' as the abbrieviation for regular-expression
    (replacing RegExp/etc.; 'regex' is 30-40% more prevalent by google hits)
 2009-10-06 03:45:44 +00:00
gojomo dbc1c1b411 * KeyWordProcessor.java 
update shouldProcess() to be based on via CrawlURI's type
 2009-10-06 02:28:47 +00:00
gojomo 4ce8aef47a Remove obsolete test class 2009-10-06 01:40:22 +00:00
gojomo 6769cdb2dc Fix inadvertent build-breaking upward dependency from modules to engine 
* modules//ProcessorChain.java
    send updates to ChainStatusReceiver interface
* engine//ToeThread.java
    implement ChainStatusReceiver interface
 2009-10-06 01:01:32 +00:00
szznax 92da7ab8cc Incremental check-in with DOM search and XPath interrogation of scope node from existing order XML. 2009-10-06 00:36:00 +00:00
gojomo 047a157fe8 [HER-1605] Refactor parameterizable policies out of frontier 
* selftest-crawler.beans.cxml, FormAuthSelfTest.java, Precedence[134]SelfTest.java
    update self-tests for new chains refactoring
 2009-10-05 23:44:30 +00:00
gojomo 13c9cc1915 [HER-1605] Refactor parameterizable policies out of frontier 
* defaults.xml
    update bundled profile for new chains refactoring
* CrawlController.java
    move to three processor-chains rather than one
* Frontier.java
    (loadSeeds) removed
* ToeThread.java
    delegate most processing-loop to FetchChain and DispositionChain
* AbstractFrontier.java, WorkQueueFrontier.java
    move policies/calculation out to processors
* CandidatesProcessor.java
    new processor for DispositionChain that runs every outlink through CandidateChain
* CrawlStateUpdater.java -> DispositionProcessor.java
    rename, expand to prep CrawlURI for frontier
* FrontierScheduler.java
    deleted; use CandidatesProcessor/CandidateChain
* LinksScoper.java
    deprecated; use CandidatesProcessor/CandidateChain/CandidateScoper
    (only temporarily retained for ease of comparison)
* CandidateScoper.java
    simple single-URI scope-testing for CandidateChain
* FrontierPreparer.java
    precalculate all frontier-policies in CandidateChain, before scheduling
* PreconditionEnforcer.java
    ProcessorURI->CrawlURI; take on some prerequisite preparation previously deferred to elsewhere
* ProcessorsReport.java
    update for 3-chains of Processors
* SheetOverlaysManager.java
    (applyOverridesTo) moved here for broader use
* CandidateChain.java, FetchChain.java, DispositionChain.java
    role-specific subclasses of ProcessorChain (suitable for type-based autowiring)
* CrawlURI.java
    new fields/accessors of use to new chains/frontier
* PostProcessor.java
    deleted; skip-to-'postprocessing' is now skip-to-end-of-chain
* ProcessorChain.java
    take-on control loop formerly in ToeThread
* ProcessResult.java
    absorb ProcessStatus
    eliminate problematic STUCK result
 2009-10-05 23:41:59 +00:00