lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-22 06:26:32 +00:00
Code Issues Packages Projects Releases Wiki Activity
241 Commits 40 Branches 31 Tags
9e0584b02138f57cba3edd4fcfdca94a478c09fb
Commit Graph
241 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
gojomo 9e0584b021 Followup for [HER-1703] convert all maps that operator might want to change on the fly to concurrent maps 
* HTTPContentDigest.java
    avoid using null for overlayable KeyedProperty -- fix broken NPE in test
 2009-11-07 20:10:19 +00:00
gojomo 11048de24c [HER-1703] convert all maps that operator might want to change on the fly to concurrent maps 
* KeyedProperties.java
    derive from ConcurrentHashMap
* Sheet.java
    ensure internal map is always a concurrent-map (even when setMap is passed something else)
* SheetOverlaysManager.java
    convert ruleAssociations, sheetNamesBySurt, sheetsByName to concurrent implementation classes
* ScriptedProcessor.java, ScriptedDecideRule.java
    convert sharedMap to concurrent implementation
* DecideRuledSheetAssociation.java
    ensure non-equal associations don't compareTo()==0 (implying equals, causing problems in sorted maps/sets)
 2009-11-07 03:07:51 +00:00
gojomo 06c63ffc5b * FrontierSummaryReport.java 
allow report even on empty frontier (as at pasue-at-end)
 2009-11-07 03:02:09 +00:00
gojomo bda89f5617 * JobResource.java 
add descriptive status to TITLE
 2009-11-07 03:00:49 +00:00
gojomo 9b211165df [HER-1698] report generation gives text of path, without showing report 
* JobResource.java
    make getHrefPath public static utility method
* ReportGenResource.java
    use getHrefPath so '/anypath/' URLs are used where needed
 2009-11-06 22:18:47 +00:00
gojomo a7a990b638 [HER-1699] pause-at-finish leaves crawl in 'pausing' not 'paused' state 
* CrawlController.java
    if frontier pauses and pauseAtFinish is true; enter PAUSED immediately
 2009-11-05 23:19:00 +00:00
gojomo 7cf25a844e * StatisticsTracker.java 
make expensive seeds-result tracking optional
 2009-10-26 12:08:49 +00:00
gojomo 648ef7deef Fix issues with surt-source, surt-dump file paths 
* ConfigPathConfigurer, PathFixupListener
    notify beans when paths fixed
* SurtPrefixedDecideRule.java
    only read source prefixes after path fixup; dump after read or any batch of seeds
* SeedModule.java, TextSeedModule.java, SeedListener.java
    new concludedSeedBatch event 
* AbstractFrontier.java, StatisticsTracker.java
    ignore new concluded-batch event
* AlertThreadGroup.java, JobResource.java
    new mechanism for threads not part of AlertThreadGroup to temporarily redirect their logging to provided (job) logger
 2009-10-26 11:29:58 +00:00
gojomo 59003e22ab Fix issues with surt-source, surt-dump file paths 
* ConfigPathConfigurer, PathFixupListener
    notify beans when paths fixed
* SurtPrefixedDecideRule.java
    only read source prefixes after path fixup; dump after read or any batch of seeds
* SeedModule.java, TextSeedModule.java, SeedListener.java
    new concludedSeedBatch event 
* AbstractFrontier.java, StatisticsTracker.java
    ignore new concluded-batch event
* AlertThreadGroup.java, JobResource.java
    new mechanism for threads not part of AlertThreadGroup to temporarily redirect their logging to provided (job) logger
 2009-10-26 11:27:09 +00:00
gojomo 4ef460ac00 Crawl-distribution-related improvements 
* URIAuthorityBasedQueueAssignmentPolicy.java
    choose numbered subqueue on first-path-segment only, so similar URIs land in same subqueue
* CrawlMapper.java
    add queueKey to diversion lines
* HashCrawlMapper.java
    update license
 2009-10-26 09:34:12 +00:00
gojomo 2033c99fea * AbstractFrontier.java 
promote priority of manageThread (where that is respected)
    remove stray debug output
 2009-10-23 22:38:21 +00:00
gojomo 69736535a7 * profile-crawler-beans.cxml 
correct missing <property> tags in example commented-out canonicalization rules
 2009-10-23 22:34:41 +00:00
gojomo 2e31d58687 * ObjectIdentityBdbCache.java 
replace costly bdb size() operation with maintained counter
 2009-10-23 22:33:36 +00:00
gojomo 06223b2794 BDB JE tuning 
* BdbModule.java
    increase je.log.faultReadSize from 2K to 6K -- stats showed many faults (ideal is zero)
 2009-10-23 21:55:06 +00:00
gojomo f0c9608e59 Followup for [HER-1605] Refactor parameterizable policies out of frontier 
* CandidatesProcessor.java
    remove misguided lock preventing intended parallelism of pre-enqueue URL prep
 2009-10-21 22:55:06 +00:00
gojomo ea114357fb * FrontierPreparer.java 
fix inconsistency in canonicalizationPolicy getter/setter names making explicit spring-config impossible
 2009-10-20 21:32:26 +00:00
szznax efee001bbc Fix for [HER-1688] remember external job directories 
* Engine.java
    protect against disappeared jobsDir in findJobConfigs()
    check for .jobpath files in findJobConfigs()
    added considerAsJobPath()
    log added jobs in considerAsJobDirectory()
    make considerAsJobDirectory() return true when jobConfig (already) exists
    added leaveJobPathFile() to write .jobpath file for newly added jobs
* EngineResource.java
    call leaveJobPathFile() when job directory added successfully
    added messageDiv() to make messages more uniform and prominent
    re-scan jobConfigs on each page load - may obviate need for "rescan" button
 2009-10-15 01:03:29 +00:00
szznax 8311db6a26 minor style change 2009-10-09 16:58:54 +00:00
szznax 72136dea31 handle null path on "create" action 2009-10-09 00:04:31 +00:00
szznax 1a136b21d5 Fix for [HER-1419] can't create job in web ui without profile/job to base it on in jobs dir? 
* Engine.java
  createNewJobWithDefaults() to write profile-crawler-beans.cxml resoure into new job
* EngineResource.java
  added FORM for new job dir and "create" action
* profile-crawler-beans.cxml
  added resource from dist/src/main/conf/jobs/profile-defaults to bootstrap new jobs
 2009-10-08 23:55:42 +00:00
gojomo 58513a8940 [HER-1687] H3 web UI: adding non-existent job directory gives unhelpful NPE 
* Engine.java
    return 'false' rather than NPE when given directory non-existent/contains no .cxml
* EngineResource.java
    better handle empty 'add' path; show NACK flash when 'add' has no effect
 2009-10-08 20:57:04 +00:00
gojomo 4a0f809c0d Post-beta-build 
* **/pom.xml
    revert version name to SNAPSHOT
 2009-10-07 11:54:12 +00:00
gojomo af9345a3be * **/pom.xml 
rename version to 'beta' for distinguished 'beta' identified build (instead of SNAPSHOT/timestamp)
 2009-10-07 11:25:20 +00:00
gojomo 64a79ea9e6 Fix issues revealed in extra tests 
* H1toH3.map, migrate-template-crawler-beans.cxml, .classpath
    move resources to 'resources' src subdirectory, so included in built JAR
    move 'resources' directorys to library rather than source directories for dev-time reachability
* MigrateH1to3Tool.java
    handle H1 values that just need capitalization
    report all 'no rule' situations as 'needs attention' 
    touchup explanations and comments
    enable uriCanonicalizationRule overrides
 2009-10-07 10:24:05 +00:00
gojomo f114448e6f Fix issues revealed in extra tests 
* H1toH3.map, migrate-template-crawler-beans.cxml, .classpath
    move resources to 'resources' src subdirectory, so included in built JAR
    move 'resources' directorys to library rather than source directories for dev-time reachability
* MigrateH1to3Tool.java
    handle H1 values that just need capitalization
    report all 'no rule' situations as 'needs attention' 
    touchup explanations and comments
    enable uriCanonicalizationRule overrides
 2009-10-07 10:23:36 +00:00
gojomo 45f09818ae Fix intermittent ClassCastExceptions on job console 
* Histotable.java, ToePool.java
    only tally Strings; convert all histotable keys to strings for sorting
 2009-10-07 10:18:21 +00:00
gojomo f960a018c1 [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 
* MigrateH1to3Tool.java
    add count of successfully migrated settings
 2009-10-07 05:04:39 +00:00
gojomo 9b3a65ab3a [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 
* MigrateH1to3Tool.java
    improve output: explain skipped settings, affected paths
* H1toH3.map
    truncate $na to $
 2009-10-07 05:01:16 +00:00
gojomo 47cbdded9a [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 
* MigrateH1to3Tool.java
    comment, clean unused code
* H1toH3.map
    text map of {H3-override-beanpath, H1-settings-XPath, H1-default-value}
* migrate-template-crawler-beans.cxml
    version of bundled profile slightly changed for better mapping to H1
    (target replace range, seeds.txt, arcWriter)
 2009-10-07 04:28:10 +00:00
gojomo b7631e2b02 Misc fixes from H1-to-H3 testing 
* BdbModule.java
    be more robust about 0/-1 as 'don't care' values
* AbstractFrontier.java
    reorder settable properties for ease of discovery
* FetchHTTP.java
    correct type to allow setting property
 2009-10-07 04:22:05 +00:00
gojomo 07adde3a73 Fix impoliteness regression 
* CrawlURI.java
    return true calculated politeness delay, not 0!
 2009-10-07 04:20:11 +00:00
gojomo a582596092 Fix bean browser 
* StatisticsTracker.java, SeedsReport.java
    avoid naming method as getter, causing bean browser problems
 2009-10-07 04:19:29 +00:00
szznax f097c07d51 Updates to be merged ... 2009-10-07 02:06:06 +00:00
gojomo 21353cc26d more profile cleanup 
* profile-crawler-beans.cxml
   add overlooked AbstractFrontier settings
 2009-10-07 01:39:30 +00:00
gojomo 791df1b78e [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 
* H1toH3.map
    order-XPath-to-override-key map for translating simple H1 settings to H3 overrides
 2009-10-07 01:23:37 +00:00
gojomo 78b3b0101f [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 
* H1toH3.map
    order-XPath-to-override-key map for translating simple H1 settings to H3 overrides
 2009-10-07 01:14:44 +00:00
gojomo 4cf3c0b151 [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 
* H1toH3.map
    order-XPath-to-override-key map for translating simple H1 settings to H3 overrides
 2009-10-07 01:11:12 +00:00
gojomo 329204b9ff [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 
* MigrateSettings.java, MigrateH1toH3Tool.java
    renamings, 'flatten' method for building map
* H1toH3.map
    order-XPath-to-override-key map for translating simple H1 settings to H3 overrides
 2009-10-07 01:07:33 +00:00
gojomo eb659a9fad more profile cleanup 
* profile-crawler-beans.cxml
   add alsoCheckVia
   improve description of commented-out defaults
   add CrawlLimitEnforcer
* SurtPrefixedDecideRule.java
    remove obsolete property
 2009-10-07 01:05:59 +00:00
szznax e2ee037786 Working commit... 2009-10-06 22:24:19 +00:00
gojomo 258da53d57 [HER-1676] improve bundled profiles to match best-practice, be self-documenting 
* profile-crawler-beans.cxml
    add comments for all unstated default values an operator might want to change
* (many) 
    reorder, refactor, rename to better minimize/match simple configuration
 2009-10-06 19:35:05 +00:00
szznax 6b1d1963d8 Incremental checkin. 2009-10-06 18:06:46 +00:00
gojomo 6f7301d994 [HER-1676] improve bundled profiles to match best-practice, be self-documenting 
* defaults.xml
   deleted
 2009-10-06 04:16:52 +00:00
gojomo 3f7d40412b Fix broken unit tests 
* CandidatesProcessor.java
    properly set full-via CrawlURI on prereqs
 2009-10-06 04:16:14 +00:00
gojomo 742e4e8cad [HER-1676] improve bundled profiles to match best-practice, be self-documenting 
* DispositionProcessor.java
    indirect robots-through-metadata, to obviate explicit declaration in beans.xml
 2009-10-06 04:14:29 +00:00
gojomo e2a61792a7 * (many) 
harmonize our usage on '[Rr]egex' as the abbrieviation for regular-expression
    (replacing RegExp/etc.; 'regex' is 30-40% more prevalent by google hits)
 2009-10-06 04:09:49 +00:00
gojomo 5c50f9c104 [HER-1676] improve bundled profiles to match best-practice, be self-documenting 
* profile-crawler-beans.cxml
    merge 'defaults' back in, beef up comments, strip unnecessary declarations
 2009-10-06 04:07:25 +00:00
gojomo da95932598 * CrawlMetadata.java 
set default RobotsHonoringPolicy, making explicit declaration unnecessary
 * RobotsHonoringPolicy.java
    apache license
 2009-10-06 04:00:58 +00:00
gojomo 2d3f569f15 * InterruptibleCharSequenceTest.java 
better output to diagnose intermittent failure
 2009-10-06 03:58:22 +00:00
gojomo 3654e330cc * (many) 
harmonize our usage on '[Rr]egex' as the abbrieviation for regular-expression
    (replacing RegExp/etc.; 'regex' is 30-40% more prevalent by google hits)
 2009-10-06 03:47:38 +00:00