lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-22 14:36:37 +00:00
Code Issues Packages Projects Releases Wiki Activity
223 Commits 40 Branches 31 Tags
72136dea311d5edd16f56d29cfab46f756367cee
Commit Graph
223 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
szznax 72136dea31 handle null path on "create" action 2009-10-09 00:04:31 +00:00
szznax 1a136b21d5 Fix for [HER-1419] can't create job in web ui without profile/job to base it on in jobs dir? 
* Engine.java
  createNewJobWithDefaults() to write profile-crawler-beans.cxml resoure into new job
* EngineResource.java
  added FORM for new job dir and "create" action
* profile-crawler-beans.cxml
  added resource from dist/src/main/conf/jobs/profile-defaults to bootstrap new jobs
 2009-10-08 23:55:42 +00:00
gojomo 58513a8940 [HER-1687] H3 web UI: adding non-existent job directory gives unhelpful NPE 
* Engine.java
    return 'false' rather than NPE when given directory non-existent/contains no .cxml
* EngineResource.java
    better handle empty 'add' path; show NACK flash when 'add' has no effect
 2009-10-08 20:57:04 +00:00
gojomo 4a0f809c0d Post-beta-build 
* **/pom.xml
    revert version name to SNAPSHOT
 2009-10-07 11:54:12 +00:00
gojomo af9345a3be * **/pom.xml 
rename version to 'beta' for distinguished 'beta' identified build (instead of SNAPSHOT/timestamp)
 2009-10-07 11:25:20 +00:00
gojomo 64a79ea9e6 Fix issues revealed in extra tests 
* H1toH3.map, migrate-template-crawler-beans.cxml, .classpath
    move resources to 'resources' src subdirectory, so included in built JAR
    move 'resources' directorys to library rather than source directories for dev-time reachability
* MigrateH1to3Tool.java
    handle H1 values that just need capitalization
    report all 'no rule' situations as 'needs attention' 
    touchup explanations and comments
    enable uriCanonicalizationRule overrides
 2009-10-07 10:24:05 +00:00
gojomo f114448e6f Fix issues revealed in extra tests 
* H1toH3.map, migrate-template-crawler-beans.cxml, .classpath
    move resources to 'resources' src subdirectory, so included in built JAR
    move 'resources' directorys to library rather than source directories for dev-time reachability
* MigrateH1to3Tool.java
    handle H1 values that just need capitalization
    report all 'no rule' situations as 'needs attention' 
    touchup explanations and comments
    enable uriCanonicalizationRule overrides
 2009-10-07 10:23:36 +00:00
gojomo 45f09818ae Fix intermittent ClassCastExceptions on job console 
* Histotable.java, ToePool.java
    only tally Strings; convert all histotable keys to strings for sorting
 2009-10-07 10:18:21 +00:00
gojomo f960a018c1 [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 
* MigrateH1to3Tool.java
    add count of successfully migrated settings
 2009-10-07 05:04:39 +00:00
gojomo 9b3a65ab3a [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 
* MigrateH1to3Tool.java
    improve output: explain skipped settings, affected paths
* H1toH3.map
    truncate $na to $
 2009-10-07 05:01:16 +00:00
gojomo 47cbdded9a [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 
* MigrateH1to3Tool.java
    comment, clean unused code
* H1toH3.map
    text map of {H3-override-beanpath, H1-settings-XPath, H1-default-value}
* migrate-template-crawler-beans.cxml
    version of bundled profile slightly changed for better mapping to H1
    (target replace range, seeds.txt, arcWriter)
 2009-10-07 04:28:10 +00:00
gojomo b7631e2b02 Misc fixes from H1-to-H3 testing 
* BdbModule.java
    be more robust about 0/-1 as 'don't care' values
* AbstractFrontier.java
    reorder settable properties for ease of discovery
* FetchHTTP.java
    correct type to allow setting property
 2009-10-07 04:22:05 +00:00
gojomo 07adde3a73 Fix impoliteness regression 
* CrawlURI.java
    return true calculated politeness delay, not 0!
 2009-10-07 04:20:11 +00:00
gojomo a582596092 Fix bean browser 
* StatisticsTracker.java, SeedsReport.java
    avoid naming method as getter, causing bean browser problems
 2009-10-07 04:19:29 +00:00
szznax f097c07d51 Updates to be merged ... 2009-10-07 02:06:06 +00:00
gojomo 21353cc26d more profile cleanup 
* profile-crawler-beans.cxml
   add overlooked AbstractFrontier settings
 2009-10-07 01:39:30 +00:00
gojomo 791df1b78e [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 
* H1toH3.map
    order-XPath-to-override-key map for translating simple H1 settings to H3 overrides
 2009-10-07 01:23:37 +00:00
gojomo 78b3b0101f [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 
* H1toH3.map
    order-XPath-to-override-key map for translating simple H1 settings to H3 overrides
 2009-10-07 01:14:44 +00:00
gojomo 4cf3c0b151 [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 
* H1toH3.map
    order-XPath-to-override-key map for translating simple H1 settings to H3 overrides
 2009-10-07 01:11:12 +00:00
gojomo 329204b9ff [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 
* MigrateSettings.java, MigrateH1toH3Tool.java
    renamings, 'flatten' method for building map
* H1toH3.map
    order-XPath-to-override-key map for translating simple H1 settings to H3 overrides
 2009-10-07 01:07:33 +00:00
gojomo eb659a9fad more profile cleanup 
* profile-crawler-beans.cxml
   add alsoCheckVia
   improve description of commented-out defaults
   add CrawlLimitEnforcer
* SurtPrefixedDecideRule.java
    remove obsolete property
 2009-10-07 01:05:59 +00:00
szznax e2ee037786 Working commit... 2009-10-06 22:24:19 +00:00
gojomo 258da53d57 [HER-1676] improve bundled profiles to match best-practice, be self-documenting 
* profile-crawler-beans.cxml
    add comments for all unstated default values an operator might want to change
* (many) 
    reorder, refactor, rename to better minimize/match simple configuration
 2009-10-06 19:35:05 +00:00
szznax 6b1d1963d8 Incremental checkin. 2009-10-06 18:06:46 +00:00
gojomo 6f7301d994 [HER-1676] improve bundled profiles to match best-practice, be self-documenting 
* defaults.xml
   deleted
 2009-10-06 04:16:52 +00:00
gojomo 3f7d40412b Fix broken unit tests 
* CandidatesProcessor.java
    properly set full-via CrawlURI on prereqs
 2009-10-06 04:16:14 +00:00
gojomo 742e4e8cad [HER-1676] improve bundled profiles to match best-practice, be self-documenting 
* DispositionProcessor.java
    indirect robots-through-metadata, to obviate explicit declaration in beans.xml
 2009-10-06 04:14:29 +00:00
gojomo e2a61792a7 * (many) 
harmonize our usage on '[Rr]egex' as the abbrieviation for regular-expression
    (replacing RegExp/etc.; 'regex' is 30-40% more prevalent by google hits)
 2009-10-06 04:09:49 +00:00
gojomo 5c50f9c104 [HER-1676] improve bundled profiles to match best-practice, be self-documenting 
* profile-crawler-beans.cxml
    merge 'defaults' back in, beef up comments, strip unnecessary declarations
 2009-10-06 04:07:25 +00:00
gojomo da95932598 * CrawlMetadata.java 
set default RobotsHonoringPolicy, making explicit declaration unnecessary
 * RobotsHonoringPolicy.java
    apache license
 2009-10-06 04:00:58 +00:00
gojomo 2d3f569f15 * InterruptibleCharSequenceTest.java 
better output to diagnose intermittent failure
 2009-10-06 03:58:22 +00:00
gojomo 3654e330cc * (many) 
harmonize our usage on '[Rr]egex' as the abbrieviation for regular-expression
    (replacing RegExp/etc.; 'regex' is 30-40% more prevalent by google hits)
 2009-10-06 03:47:38 +00:00
gojomo d86f5f7403 * (many) 
harmonize our usage on '[Rr]egex' as the abbrieviation for regular-expression
    (replacing RegExp/etc.; 'regex' is 30-40% more prevalent by google hits)
 2009-10-06 03:45:44 +00:00
gojomo dbc1c1b411 * KeyWordProcessor.java 
update shouldProcess() to be based on via CrawlURI's type
 2009-10-06 02:28:47 +00:00
gojomo 4ce8aef47a Remove obsolete test class 2009-10-06 01:40:22 +00:00
gojomo 6769cdb2dc Fix inadvertent build-breaking upward dependency from modules to engine 
* modules//ProcessorChain.java
    send updates to ChainStatusReceiver interface
* engine//ToeThread.java
    implement ChainStatusReceiver interface
 2009-10-06 01:01:32 +00:00
szznax 92da7ab8cc Incremental check-in with DOM search and XPath interrogation of scope node from existing order XML. 2009-10-06 00:36:00 +00:00
gojomo 047a157fe8 [HER-1605] Refactor parameterizable policies out of frontier 
* selftest-crawler.beans.cxml, FormAuthSelfTest.java, Precedence[134]SelfTest.java
    update self-tests for new chains refactoring
 2009-10-05 23:44:30 +00:00
gojomo 13c9cc1915 [HER-1605] Refactor parameterizable policies out of frontier 
* defaults.xml
    update bundled profile for new chains refactoring
* CrawlController.java
    move to three processor-chains rather than one
* Frontier.java
    (loadSeeds) removed
* ToeThread.java
    delegate most processing-loop to FetchChain and DispositionChain
* AbstractFrontier.java, WorkQueueFrontier.java
    move policies/calculation out to processors
* CandidatesProcessor.java
    new processor for DispositionChain that runs every outlink through CandidateChain
* CrawlStateUpdater.java -> DispositionProcessor.java
    rename, expand to prep CrawlURI for frontier
* FrontierScheduler.java
    deleted; use CandidatesProcessor/CandidateChain
* LinksScoper.java
    deprecated; use CandidatesProcessor/CandidateChain/CandidateScoper
    (only temporarily retained for ease of comparison)
* CandidateScoper.java
    simple single-URI scope-testing for CandidateChain
* FrontierPreparer.java
    precalculate all frontier-policies in CandidateChain, before scheduling
* PreconditionEnforcer.java
    ProcessorURI->CrawlURI; take on some prerequisite preparation previously deferred to elsewhere
* ProcessorsReport.java
    update for 3-chains of Processors
* SheetOverlaysManager.java
    (applyOverridesTo) moved here for broader use
* CandidateChain.java, FetchChain.java, DispositionChain.java
    role-specific subclasses of ProcessorChain (suitable for type-based autowiring)
* CrawlURI.java
    new fields/accessors of use to new chains/frontier
* PostProcessor.java
    deleted; skip-to-'postprocessing' is now skip-to-end-of-chain
* ProcessorChain.java
    take-on control loop formerly in ToeThread
* ProcessResult.java
    absorb ProcessStatus
    eliminate problematic STUCK result
 2009-10-05 23:41:59 +00:00
gojomo 30c729edb3 More ProcessorURI -> CrawlURI updates 2009-10-05 23:25:12 +00:00
gojomo b7d1a79b4d More ProcessorURI -> CrawlURI updates 2009-10-05 23:08:12 +00:00
gojomo 8f7ad4eae4 work for [HER-1605] Refactor parameterizable policies out of frontier 
* QueueAssignmentPolicy.java, URIAuthorityBasedQueueAssignmentPolicy.java
    move 'force-queue-assignment' setting from frontier to policy
 2009-10-05 23:04:37 +00:00
gojomo 10275d7fa1 More ProcessorURI -> CrawlURI updates 2009-10-05 23:01:40 +00:00
gojomo c7ed3e775d Misc 
* ArchiveUtils.java
   (checkContinue) utility method relocated from ToeThread for converting interrupt status to exception
* ClassKeyMatchesRegExpDecideRule.java
   properly set up crawlController property for auto-wiring
 2009-10-05 22:59:46 +00:00
gojomo 16538f175c Cleanup, refactoring 
* (many)
   update license notice to Apache where appropriate
   eliminate ProcessorURI, DefaultProcessorURI in favor of CrawlURI (now in modules package)
   other comment/warning/unused code cleanup
 2009-10-05 22:49:19 +00:00
gojomo c04c5dbdbc Improve test robustness 
* Engine.java
    wait another second before testing for job wrap-up
 2009-10-05 22:25:44 +00:00
szznax 62690268b7 Initial checkin for [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 2009-10-05 20:00:56 +00:00
szznax ba5bc1a20c Fix unit tests (carried over from H1) 
* WARCWriterTest.java
    (testSpaceInUrl),(testTabInURL): don't expect exception; expect nothing-written
 2009-10-05 18:35:06 +00:00
gojomo 86974e2e1d Cleanup post-fix 
* ObjectIdentityBdbCache.java
    with late ref-enqueuing confirmed as cause of problems, cleaning out unnecessary steps, improving comments
 2009-10-04 06:15:35 +00:00
gojomo 75676ac905 Attempt at fixing build-box-only warnings 
* ObjectIdentityBdbCache.java
    explicitly page-out lingering entry; 
    temporary System.err logging to see if problem is momentary lag in RefQueue-appearance
 2009-10-04 05:14:59 +00:00