lee/heritrix3
1
0 Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-10-08 13:21:24 +00:00
Code Issues Packages Projects Releases Wiki Activity
298 Commits 43 Branches 34 Tags
18e165f21e138b1a17853cc548e2fa8e596b2e53
Commit Graph
163 Commits
Author SHA1 Message Date
gojomo 18e165f21e [HER-1714] fixed-interval rescheduling 
* AbstractFrontier.java, WorkQueueFrontier.java, BdbFrontier.java
    add 'futureUris' map of CrawlURIs to be reenqueued at specific future dates
    the usual findEligibleUri also checks for newly-eligible rescheduled URIs
    the frontier is no longer empty unless the future count is also 0
* CrawlURI.java
    add rescheduleTime property
* ReschedulingProcessor.java
    optional post-processor to set a CrawlURI's reschedulingTime
* CrawlStatSnapshot.java, CrawlJob.java
    add futureUriCount to stats/uriTotalsReport
 2009-11-23 08:47:14 +00:00
gojomo 3c9df9a8c6 BDBJE updates, refactoring 
* BdbFrontier.java
    use BdbModule.openDatabase()
 2009-11-23 08:24:54 +00:00
gojomo 5b6b2fe954 BDBJE updates, refactoring 
* BdbModule.java
    (openDatabase) all databases now 'managed'
    (getStoredMap) all reuse of prior data as for checkpoints
* (others)
    match above changes
 2009-11-23 08:18:41 +00:00
gojomo 4127286a1c Checkpoint cleanup 
* Checkpoint.java
    calculate shortName correctly
* CrawlerLoggerModule.java
    use full checkpoint-name for log-rotation-suffix
 2009-11-23 08:10:07 +00:00
gojomo eb9bf5a8e2 BDBJE updates, refactoring 
* commons/pom.xml, .classpath
    update to BDB-JE 4.0.71
* BdbModule.java
    (getStoredQueue) added
    (getStoredMap) made more general
    (openDatabase) all databases now 'managed'
* TempStoredSortedMap.java -> DisposableStoredSortedMap.java
    renamed
* StatisticsTracker.java, **Report.java
    use new name
* StoredQueue.java, StoredQueueTest.java
    move to org.archive.bdb package
* BdbMultipleWorkQueues.java, PrefixFinderTest.java
    update for BDB-JE 4 changes
* BdbFrontier.java
    use getStoredQueue 
    drop unneeded custom serialization support
* WorkQueueFrontier.java
    drop unneeded custom serialization support
 2009-11-23 04:41:32 +00:00
gojomo 8251620481 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls 
* profile-crawler-beans.cxml*2, migrate-template-crawler-beans.cxml
    add CheckpointService as recommended bean
 2009-11-20 18:35:13 +00:00
gojomo f9d56e3f6b move json dependency to commons from engine 2009-11-20 02:03:27 +00:00
gojomo 25a568fff7 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls 
* JobResource.java
    suppress checkpoint-selector after terminate
* Checkpoint.java
    adapt to work without direct reference to CheckpointService
* CheckpointService.java
    configure Checkpoint without making it dependent on CheckpointService
 2009-11-20 01:24:57 +00:00
gojomo daab317c94 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls 
processor-related bean checkpoint support:
* Checkpoint.java
    move to commons for proper dependency graph
* (others) 
    recognize above
 2009-11-20 01:10:32 +00:00
gojomo dfacba51e5 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls 
UI support
* JobResource.java
    enable 'checkpoint' button as appropriate
    display active recovery-checkpoint or select bos of recovery-options when available
* CrawlJob.java
    (getCheckpointService) accessor
 2009-11-20 01:02:14 +00:00
gojomo 50ee78f6d2 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls 
* BdbFrontier.java
    implement Checkpointable; save state via JSON or BDB; restore queues-of-queues on recovery
* WorkQueueFrontier.java
    (deactivateQueue) (retireQueue) protected
* BdbUriUniqFilter.java
    implement Checkpointable; save state via JSON or BDB; restore
* WorkQueue.java
    make peekItem transient -- should not be held through checkpoint
    make retired protected
* Frontier.java, AbstractFrontier.java
    remove/update obsolete code
 2009-11-20 00:59:00 +00:00
gojomo be17d5ed33 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls 
* CrawlController.java
    remove older checkpoint-functionality passthrough
    implement Checkpointable so behavior can vary during recovery
    only trigger annnounce of seeds on 'cold' (non-recovery) start
 2009-11-20 00:51:59 +00:00
gojomo ac6b58cb0e [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls 
* Checkpoint.java
    (generateFrom) version more suitable for unit testing
 2009-11-20 00:49:31 +00:00
gojomo fbd3af35ac [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls 
* PersistLogProcessor.java
    rotate logs on checkpoint
* CrawlerJournal.java, FrontierJournal.java
    rotate for checkpoint support without full Checkpointable
 2009-11-20 00:47:54 +00:00
gojomo f01ad9ada8 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls 
stats & logging checkpoint support
* StatisticsTracker.java
    implement Checkpointable; save small state in JSON, recycle BDB data when recovering
* CrawlerLoggerModule.java
    implement Checkpointable; rotate logs on checkpoint
* GenerationFileHandler.java
    avoid clobbering leftover logs from earlier futures via moveAside
* CrawlStatSnapshot.java
    (sameProgressAs) test for lack-of-progress
* ObjectIdentityMemCache.java, TopNSet.java
    offer deeper access to assist checkpoint/resume
 2009-11-19 23:18:15 +00:00
gojomo b63c2bdc77 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls 
processor-related bean checkpoint support:
* ToeThread.java
    use getBeanName
 2009-11-19 23:12:47 +00:00
gojomo bdf4437ca2 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls 
core checkpointing system classes:
* Checkpointable.java
    basic interface for checkpoint aware/capable beans
* Checkpoint.java
    one Checkpoint, either in progress or recovery -- a name + storage directory
* CheckpointService.java
    helper bean to enable, trigger checkpoints and recovery 
* Checkpointer.java(*2), CheckpointInputStream.java, CheckpointRecovery.java DefaultCheckpointRecovery.java, RecoverAction.java
    (deleted) no longer used
 2009-11-19 22:45:07 +00:00
gojomo e42e2ed7f6 Cleanup/consolidation/expansion of *Utils classes 
* IoUtils.java
    remove class that confusingly-overlaps with commons IOUtils; move remaining methods to ArchiveUtils or archive's FileUtils
* ArchiveUtils.java, FileUtils.java
    receive relocated methods; eliminated deprecated or unused methods
* JSONUtils.java
    new class to collect common JSON actions
* Iteratorable.java
    wrap Iterator as Iterable for foreach usage
* (many)
    update to use alternate utils methods
 2009-11-19 22:39:53 +00:00
gojomo 300ce15a06 * EditRepresentation.java 
add filename to TITLE to help identify browser tabs
 2009-11-19 21:58:12 +00:00
gojomo b97319fb07 reformat 2009-11-19 21:57:05 +00:00
gojomo f9e1191144 Fix build - cleanup leak of incomplete code to prior commit 2009-11-18 20:53:41 +00:00
gojomo 56055d8236 [HER-1709] h3 - dumpPendingAtClose doesn't work 
* CrawlController.java
    avoid unnecessarily early/redundant close of logs files (and checkpointer cleanup)
* WorkQueueFrontier.java
    add stop() that does close()
* BdbMultipleWorkQueues.java
    add cursor.close()
* AbstractFrontier.java, BdbFrontier.java
    use closeQuietly
 2009-11-18 00:29:37 +00:00
nlevitt 148c7332c4 Merge ServerCache+ServerCacheUtil abstract superclass ServerCache 
* ServerCache.java
    make abstract, implement getHostFor(UURI) and getServerFor(UURI)
* DefaultServerCache.java
    extend ServerCache, remove getHostFor(UURI) and getServerFor(UURI)
* ServerCacheUtil.java
    removed
* other classes
    change usage of ServerCacheUtil to ServerCache
 2009-11-17 06:10:42 +00:00
nlevitt e57ad68230 * AbstractFrontier.java 
implement ApplicationListener like it seems to want to, being that it has a
    method onApplicationEvent()
 2009-11-17 04:53:11 +00:00
szznax 92f3bfe0ec check for exactly 2 args, tightened up SAXException handling by making sourceOrderXmlDom 
a protected field, cleaned up unused import and whitespace.
 2009-11-16 23:45:04 +00:00
gojomo 9059395617 Make Bloom size configurable without system properties 
* BloomUriUniqFilter.java
    make bloom size configurable via bean properties
    change implementing class to BloomFilter64bit (given growing prevalance of 64bit JVMs and larger heaps)
* BloomUriUniqFilterTest.java
    update test for property-set sizing
* BloomFilter64bit.java
    improve comment
 2009-11-16 22:42:18 +00:00
szznax 69bedb8240 a bit more error-handling for input args 2009-11-16 19:31:30 +00:00
szznax 24463da8bc Fix for [HER-1689] build, launch buttons remain enabled after problem building 
* CrawlJob.java
  renamed isContainerOk() to hasApplicationContext()
  increased granularity of isLaunchable() to offer launch button
* JobResource.java
  more coarse check for hasApplicationContext() to disable build button
 2009-11-13 23:23:52 +00:00
nlevitt 2740cc7bb0 Change the artifactId from commons to heritrix-commons, modules to 
heritrix-modules, engine to heritrix-engine.
 2009-11-12 02:48:38 +00:00
nlevitt f6b2066d6e [HER-1693] dedup numbers in crawl,host reports - patch contributed by Matt 
Warhaftig
 2009-11-12 01:38:28 +00:00
nlevitt 9743be5d8c Fix for [HER-1683] h3: inconsistent host classification for dns: urls 
* DefaultServerCache.java
    getHostFor(UURI): special case dns uris
* ServerCache.java
    add getHostFor(UURI) to public interface
* ServerCacheUtil.java
    remove now redundant wrapper method
* DispositionProcessor.java AbstractFrontier.java IPQueueAssignmentPolicy.java
  BucketQueueAssignmentPolicy.java PreconditionEnforcer.java QuotaEnforcer.java
  StatisticsTracker.java ServerCacheTest.java FetchHTTP.java
  Kw3WriterProcessor.java WriterPoolProcessor.java 
    changed to use serverCache.getHostFor(UURI) (a few cases remain that use
    getHostFor(String))
 2009-11-12 01:28:22 +00:00
nlevitt c77de5fc08 [HER-1561] put apache license boilerplate on a couple of stragglers 2009-11-11 03:19:06 +00:00
nlevitt 3e08d49640 Use json library jar instead of copy&pasted code. 2009-11-11 02:30:39 +00:00
gojomo ce79e8a76e [HER-1561] update license to Apache License 2.0 
* SeekReader.java
   update to Apache
* RecoveryLogMapper.java
   revert to LGPL pending verification of authors' approval for relicensing
 2009-11-11 00:28:03 +00:00
gojomo dd0ca367c3 [HER-1706] PersistLoadProcessor and PersistStoreProcessor can't share same BdbModule 
* BdbModule.java
    allow Database instance reuse if BdbConfig is identical instance
* PersistProcessor.java
    convert shared BdbConfig for history db to static constant HISTORY_DB_CONFIG
* PersistOnlineProcessor.java
    use HISTORY_DB_CONFIG, don't close possibly-shared DB on stop()
* PrecedenceLoader.java, PreloadedUriPrecedencePolicy.java
    use HISTORY_DB_CONFIG
 2009-11-10 23:48:10 +00:00
gojomo 0497efbb32 [HER-1561] update license to Apache License 2.0 
* (many)
    update license boilerplate
* (few)
    delete no-longer used classes
 2009-11-10 21:03:27 +00:00
gojomo f5e0d333a2 [HER-1702] console rates (eg URIs/sec) sometimes way off, infinite, NaN (�) 
* CrawlStatSnapshot.java
    force double math, preventing premature loss of precision, zeros in denominators
 2009-11-07 20:13:00 +00:00
gojomo 11048de24c [HER-1703] convert all maps that operator might want to change on the fly to concurrent maps 
* KeyedProperties.java
    derive from ConcurrentHashMap
* Sheet.java
    ensure internal map is always a concurrent-map (even when setMap is passed something else)
* SheetOverlaysManager.java
    convert ruleAssociations, sheetNamesBySurt, sheetsByName to concurrent implementation classes
* ScriptedProcessor.java, ScriptedDecideRule.java
    convert sharedMap to concurrent implementation
* DecideRuledSheetAssociation.java
    ensure non-equal associations don't compareTo()==0 (implying equals, causing problems in sorted maps/sets)
 2009-11-07 03:07:51 +00:00
gojomo 06c63ffc5b * FrontierSummaryReport.java 
allow report even on empty frontier (as at pasue-at-end)
 2009-11-07 03:02:09 +00:00
gojomo bda89f5617 * JobResource.java 
add descriptive status to TITLE
 2009-11-07 03:00:49 +00:00
gojomo 9b211165df [HER-1698] report generation gives text of path, without showing report 
* JobResource.java
    make getHrefPath public static utility method
* ReportGenResource.java
    use getHrefPath so '/anypath/' URLs are used where needed
 2009-11-06 22:18:47 +00:00
gojomo a7a990b638 [HER-1699] pause-at-finish leaves crawl in 'pausing' not 'paused' state 
* CrawlController.java
    if frontier pauses and pauseAtFinish is true; enter PAUSED immediately
 2009-11-05 23:19:00 +00:00
gojomo 7cf25a844e * StatisticsTracker.java 
make expensive seeds-result tracking optional
 2009-10-26 12:08:49 +00:00
gojomo 59003e22ab Fix issues with surt-source, surt-dump file paths 
* ConfigPathConfigurer, PathFixupListener
    notify beans when paths fixed
* SurtPrefixedDecideRule.java
    only read source prefixes after path fixup; dump after read or any batch of seeds
* SeedModule.java, TextSeedModule.java, SeedListener.java
    new concludedSeedBatch event 
* AbstractFrontier.java, StatisticsTracker.java
    ignore new concluded-batch event
* AlertThreadGroup.java, JobResource.java
    new mechanism for threads not part of AlertThreadGroup to temporarily redirect their logging to provided (job) logger
 2009-10-26 11:27:09 +00:00
gojomo 4ef460ac00 Crawl-distribution-related improvements 
* URIAuthorityBasedQueueAssignmentPolicy.java
    choose numbered subqueue on first-path-segment only, so similar URIs land in same subqueue
* CrawlMapper.java
    add queueKey to diversion lines
* HashCrawlMapper.java
    update license
 2009-10-26 09:34:12 +00:00
gojomo 2033c99fea * AbstractFrontier.java 
promote priority of manageThread (where that is respected)
    remove stray debug output
 2009-10-23 22:38:21 +00:00
gojomo f0c9608e59 Followup for [HER-1605] Refactor parameterizable policies out of frontier 
* CandidatesProcessor.java
    remove misguided lock preventing intended parallelism of pre-enqueue URL prep
 2009-10-21 22:55:06 +00:00
gojomo ea114357fb * FrontierPreparer.java 
fix inconsistency in canonicalizationPolicy getter/setter names making explicit spring-config impossible
 2009-10-20 21:32:26 +00:00
szznax efee001bbc Fix for [HER-1688] remember external job directories 
* Engine.java
    protect against disappeared jobsDir in findJobConfigs()
    check for .jobpath files in findJobConfigs()
    added considerAsJobPath()
    log added jobs in considerAsJobDirectory()
    make considerAsJobDirectory() return true when jobConfig (already) exists
    added leaveJobPathFile() to write .jobpath file for newly added jobs
* EngineResource.java
    call leaveJobPathFile() when job directory added successfully
    added messageDiv() to make messages more uniform and prominent
    re-scan jobConfigs on each page load - may obviate need for "rescan" button
 2009-10-15 01:03:29 +00:00
szznax 8311db6a26 minor style change 2009-10-09 16:58:54 +00:00