lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-23 15:06:46 +00:00
Code Issues Packages Projects Releases Wiki Activity
217 Commits 40 Branches 31 Tags
f114448e6fb3ff2e6aebda5ba0d5ef1f2263a474
Commit Graph
217 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
gojomo f114448e6f Fix issues revealed in extra tests 
* H1toH3.map, migrate-template-crawler-beans.cxml, .classpath
    move resources to 'resources' src subdirectory, so included in built JAR
    move 'resources' directorys to library rather than source directories for dev-time reachability
* MigrateH1to3Tool.java
    handle H1 values that just need capitalization
    report all 'no rule' situations as 'needs attention' 
    touchup explanations and comments
    enable uriCanonicalizationRule overrides
 2009-10-07 10:23:36 +00:00
gojomo 45f09818ae Fix intermittent ClassCastExceptions on job console 
* Histotable.java, ToePool.java
    only tally Strings; convert all histotable keys to strings for sorting
 2009-10-07 10:18:21 +00:00
gojomo f960a018c1 [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 
* MigrateH1to3Tool.java
    add count of successfully migrated settings
 2009-10-07 05:04:39 +00:00
gojomo 9b3a65ab3a [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 
* MigrateH1to3Tool.java
    improve output: explain skipped settings, affected paths
* H1toH3.map
    truncate $na to $
 2009-10-07 05:01:16 +00:00
gojomo 47cbdded9a [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 
* MigrateH1to3Tool.java
    comment, clean unused code
* H1toH3.map
    text map of {H3-override-beanpath, H1-settings-XPath, H1-default-value}
* migrate-template-crawler-beans.cxml
    version of bundled profile slightly changed for better mapping to H1
    (target replace range, seeds.txt, arcWriter)
 2009-10-07 04:28:10 +00:00
gojomo b7631e2b02 Misc fixes from H1-to-H3 testing 
* BdbModule.java
    be more robust about 0/-1 as 'don't care' values
* AbstractFrontier.java
    reorder settable properties for ease of discovery
* FetchHTTP.java
    correct type to allow setting property
 2009-10-07 04:22:05 +00:00
gojomo 07adde3a73 Fix impoliteness regression 
* CrawlURI.java
    return true calculated politeness delay, not 0!
 2009-10-07 04:20:11 +00:00
gojomo a582596092 Fix bean browser 
* StatisticsTracker.java, SeedsReport.java
    avoid naming method as getter, causing bean browser problems
 2009-10-07 04:19:29 +00:00
szznax f097c07d51 Updates to be merged ... 2009-10-07 02:06:06 +00:00
gojomo 21353cc26d more profile cleanup 
* profile-crawler-beans.cxml
   add overlooked AbstractFrontier settings
 2009-10-07 01:39:30 +00:00
gojomo 791df1b78e [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 
* H1toH3.map
    order-XPath-to-override-key map for translating simple H1 settings to H3 overrides
 2009-10-07 01:23:37 +00:00
gojomo 78b3b0101f [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 
* H1toH3.map
    order-XPath-to-override-key map for translating simple H1 settings to H3 overrides
 2009-10-07 01:14:44 +00:00
gojomo 4cf3c0b151 [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 
* H1toH3.map
    order-XPath-to-override-key map for translating simple H1 settings to H3 overrides
 2009-10-07 01:11:12 +00:00
gojomo 329204b9ff [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 
* MigrateSettings.java, MigrateH1toH3Tool.java
    renamings, 'flatten' method for building map
* H1toH3.map
    order-XPath-to-override-key map for translating simple H1 settings to H3 overrides
 2009-10-07 01:07:33 +00:00
gojomo eb659a9fad more profile cleanup 
* profile-crawler-beans.cxml
   add alsoCheckVia
   improve description of commented-out defaults
   add CrawlLimitEnforcer
* SurtPrefixedDecideRule.java
    remove obsolete property
 2009-10-07 01:05:59 +00:00
szznax e2ee037786 Working commit... 2009-10-06 22:24:19 +00:00
gojomo 258da53d57 [HER-1676] improve bundled profiles to match best-practice, be self-documenting 
* profile-crawler-beans.cxml
    add comments for all unstated default values an operator might want to change
* (many) 
    reorder, refactor, rename to better minimize/match simple configuration
 2009-10-06 19:35:05 +00:00
szznax 6b1d1963d8 Incremental checkin. 2009-10-06 18:06:46 +00:00
gojomo 6f7301d994 [HER-1676] improve bundled profiles to match best-practice, be self-documenting 
* defaults.xml
   deleted
 2009-10-06 04:16:52 +00:00
gojomo 3f7d40412b Fix broken unit tests 
* CandidatesProcessor.java
    properly set full-via CrawlURI on prereqs
 2009-10-06 04:16:14 +00:00
gojomo 742e4e8cad [HER-1676] improve bundled profiles to match best-practice, be self-documenting 
* DispositionProcessor.java
    indirect robots-through-metadata, to obviate explicit declaration in beans.xml
 2009-10-06 04:14:29 +00:00
gojomo e2a61792a7 * (many) 
harmonize our usage on '[Rr]egex' as the abbrieviation for regular-expression
    (replacing RegExp/etc.; 'regex' is 30-40% more prevalent by google hits)
 2009-10-06 04:09:49 +00:00
gojomo 5c50f9c104 [HER-1676] improve bundled profiles to match best-practice, be self-documenting 
* profile-crawler-beans.cxml
    merge 'defaults' back in, beef up comments, strip unnecessary declarations
 2009-10-06 04:07:25 +00:00
gojomo da95932598 * CrawlMetadata.java 
set default RobotsHonoringPolicy, making explicit declaration unnecessary
 * RobotsHonoringPolicy.java
    apache license
 2009-10-06 04:00:58 +00:00
gojomo 2d3f569f15 * InterruptibleCharSequenceTest.java 
better output to diagnose intermittent failure
 2009-10-06 03:58:22 +00:00
gojomo 3654e330cc * (many) 
harmonize our usage on '[Rr]egex' as the abbrieviation for regular-expression
    (replacing RegExp/etc.; 'regex' is 30-40% more prevalent by google hits)
 2009-10-06 03:47:38 +00:00
gojomo d86f5f7403 * (many) 
harmonize our usage on '[Rr]egex' as the abbrieviation for regular-expression
    (replacing RegExp/etc.; 'regex' is 30-40% more prevalent by google hits)
 2009-10-06 03:45:44 +00:00
gojomo dbc1c1b411 * KeyWordProcessor.java 
update shouldProcess() to be based on via CrawlURI's type
 2009-10-06 02:28:47 +00:00
gojomo 4ce8aef47a Remove obsolete test class 2009-10-06 01:40:22 +00:00
gojomo 6769cdb2dc Fix inadvertent build-breaking upward dependency from modules to engine 
* modules//ProcessorChain.java
    send updates to ChainStatusReceiver interface
* engine//ToeThread.java
    implement ChainStatusReceiver interface
 2009-10-06 01:01:32 +00:00
szznax 92da7ab8cc Incremental check-in with DOM search and XPath interrogation of scope node from existing order XML. 2009-10-06 00:36:00 +00:00
gojomo 047a157fe8 [HER-1605] Refactor parameterizable policies out of frontier 
* selftest-crawler.beans.cxml, FormAuthSelfTest.java, Precedence[134]SelfTest.java
    update self-tests for new chains refactoring
 2009-10-05 23:44:30 +00:00
gojomo 13c9cc1915 [HER-1605] Refactor parameterizable policies out of frontier 
* defaults.xml
    update bundled profile for new chains refactoring
* CrawlController.java
    move to three processor-chains rather than one
* Frontier.java
    (loadSeeds) removed
* ToeThread.java
    delegate most processing-loop to FetchChain and DispositionChain
* AbstractFrontier.java, WorkQueueFrontier.java
    move policies/calculation out to processors
* CandidatesProcessor.java
    new processor for DispositionChain that runs every outlink through CandidateChain
* CrawlStateUpdater.java -> DispositionProcessor.java
    rename, expand to prep CrawlURI for frontier
* FrontierScheduler.java
    deleted; use CandidatesProcessor/CandidateChain
* LinksScoper.java
    deprecated; use CandidatesProcessor/CandidateChain/CandidateScoper
    (only temporarily retained for ease of comparison)
* CandidateScoper.java
    simple single-URI scope-testing for CandidateChain
* FrontierPreparer.java
    precalculate all frontier-policies in CandidateChain, before scheduling
* PreconditionEnforcer.java
    ProcessorURI->CrawlURI; take on some prerequisite preparation previously deferred to elsewhere
* ProcessorsReport.java
    update for 3-chains of Processors
* SheetOverlaysManager.java
    (applyOverridesTo) moved here for broader use
* CandidateChain.java, FetchChain.java, DispositionChain.java
    role-specific subclasses of ProcessorChain (suitable for type-based autowiring)
* CrawlURI.java
    new fields/accessors of use to new chains/frontier
* PostProcessor.java
    deleted; skip-to-'postprocessing' is now skip-to-end-of-chain
* ProcessorChain.java
    take-on control loop formerly in ToeThread
* ProcessResult.java
    absorb ProcessStatus
    eliminate problematic STUCK result
 2009-10-05 23:41:59 +00:00
gojomo 30c729edb3 More ProcessorURI -> CrawlURI updates 2009-10-05 23:25:12 +00:00
gojomo b7d1a79b4d More ProcessorURI -> CrawlURI updates 2009-10-05 23:08:12 +00:00
gojomo 8f7ad4eae4 work for [HER-1605] Refactor parameterizable policies out of frontier 
* QueueAssignmentPolicy.java, URIAuthorityBasedQueueAssignmentPolicy.java
    move 'force-queue-assignment' setting from frontier to policy
 2009-10-05 23:04:37 +00:00
gojomo 10275d7fa1 More ProcessorURI -> CrawlURI updates 2009-10-05 23:01:40 +00:00
gojomo c7ed3e775d Misc 
* ArchiveUtils.java
   (checkContinue) utility method relocated from ToeThread for converting interrupt status to exception
* ClassKeyMatchesRegExpDecideRule.java
   properly set up crawlController property for auto-wiring
 2009-10-05 22:59:46 +00:00
gojomo 16538f175c Cleanup, refactoring 
* (many)
   update license notice to Apache where appropriate
   eliminate ProcessorURI, DefaultProcessorURI in favor of CrawlURI (now in modules package)
   other comment/warning/unused code cleanup
 2009-10-05 22:49:19 +00:00
gojomo c04c5dbdbc Improve test robustness 
* Engine.java
    wait another second before testing for job wrap-up
 2009-10-05 22:25:44 +00:00
szznax 62690268b7 Initial checkin for [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml 2009-10-05 20:00:56 +00:00
szznax ba5bc1a20c Fix unit tests (carried over from H1) 
* WARCWriterTest.java
    (testSpaceInUrl),(testTabInURL): don't expect exception; expect nothing-written
 2009-10-05 18:35:06 +00:00
gojomo 86974e2e1d Cleanup post-fix 
* ObjectIdentityBdbCache.java
    with late ref-enqueuing confirmed as cause of problems, cleaning out unnecessary steps, improving comments
 2009-10-04 06:15:35 +00:00
gojomo 75676ac905 Attempt at fixing build-box-only warnings 
* ObjectIdentityBdbCache.java
    explicitly page-out lingering entry; 
    temporary System.err logging to see if problem is momentary lag in RefQueue-appearance
 2009-10-04 05:14:59 +00:00
gojomo c05044d51b Eliminate double-database close log messages 
* BdbModule.java
   split openDatabase() to openManagedDatabase() (auto-closed) and plain openDatabase() (caller-closes)
   set TempStoredSortedMap to use unmanaged openDatabase
* (others)
   use openManagedDatabase()
 2009-10-04 05:06:14 +00:00
gojomo c3914ea943 Attempt at fixing build-box-only warnings 
* ObjectIdentityBdbCache.java
    add earlier check/better logging for error state
 2009-10-04 03:31:45 +00:00
gojomo 839e895a8d Attempt at fixing build-box-only warnings 
* ObjectIdentityBdbCache.java
    synchronize critical sections on memMap, diskMap rather than enclosing object
 2009-10-03 23:55:59 +00:00
gojomo 3cb6b59b8c [HER-769] Support case where millions of seeds. 
* BdbModule.java
    add temp-stored-map service method
    remove unused secondary-db support
* TempStoredSortedMap.java
    StoredSortedMap that can destroy its underlying database after temporary use
* StatisticsTracker.java
    update various sorted-by-decreasing-frequency methods to use temp StoredMaps, with duplicate keys that are negative counts
* (Multiple)Report.java
    update to use new duplicate-keyed frequency maps
 2009-10-03 23:28:11 +00:00
szznax 97778493e2 As part of fix for [HER-1671] WARCWriterProcessor error when writing 
problematic (illegal control character?) outlink
catch IllegalArgumentException in ANVLRecord.addLabelValue() to allow
WARCWriter to continue processing, and move
WARCWriter.createRecordHeader() into its own try-catch block to with
logging prevent fouling record when problems occur creating header.
* ANVLRecord.java
    catch IllegalArgumentException in addLabelValue and log warning to
    give WARCWriter to continue processing
* WARCWriter.java
    move createRecordHeader() into its own try-catch block to log
    severe problem writing record without fouling the WARC, and have
    baseCharacterCheck() throw IllegalArgumentException in accord with
    ANVLRecord.addLabelValue
* WARCWriterTest.java
    throw IllegalArgumentException in accord with
    WARCWriter.createRecordHeader()
 2009-10-03 00:41:59 +00:00
szznax 03df2a9a99 As part of fix for [HER-1671] WARCWriterProcessor error when writing problematic (illegal control character?) outlink - percent encode control-characters in 'userinfo' part of URI as we do for uriPath and uriQuery, so we don't raise exception when adding outlinks to metadata record 
* UURIFactory.java
  ensureMinimalEscaping() on 'userinfo' in UURIFactory.fixupAuthority()
 2009-10-02 01:39:44 +00:00