lee/heritrix3
1
0 Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-11-15 10:20:47 +00:00
Code Issues Packages Projects Releases Wiki Activity
384 Commits 47 Branches 34 Tags
9fd648f3efabd31439f38141b13682d445757c55
Commit Graph
18 Commits
Author SHA1 Message Date
gojomo 9fd648f3ef HER-1467 Allow periodic checkpointing and checkpoint-copy-logs to be set/unset during crawl 
* CheckpointService.java
    reorganize so that checkpoint timedTask is reset whenever interval is reset on a running crawl
* CheckpointSuccessEvent.java, CrawlJob.java
    report successes as new kind of ApplicationEvent; propagagte to job log
* BdbModule.java
    remove obsolete unconsulted checkpointCopyLogs setting
 2010-04-23 00:15:44 +00:00
gojomo 9e12e4abe5 test/deprecation cleanup 
* CachedBdbMap.java
   @deprecated
* BdbModule.java
   note (unused) 'CBM' option is deprecated
* ObjectIdentityBdbCache.java
   tighten logging to FINE
* ObjectIdentityBdbCacheTest.java
   fix test that was only trying to log, rather than fail, problems
 2010-04-13 02:15:53 +00:00
gojomo 5b6b2fe954 BDBJE updates, refactoring 
* BdbModule.java
    (openDatabase) all databases now 'managed'
    (getStoredMap) all reuse of prior data as for checkpoints
* (others)
    match above changes
 2009-11-23 08:18:41 +00:00
gojomo eb9bf5a8e2 BDBJE updates, refactoring 
* commons/pom.xml, .classpath
    update to BDB-JE 4.0.71
* BdbModule.java
    (getStoredQueue) added
    (getStoredMap) made more general
    (openDatabase) all databases now 'managed'
* TempStoredSortedMap.java -> DisposableStoredSortedMap.java
    renamed
* StatisticsTracker.java, **Report.java
    use new name
* StoredQueue.java, StoredQueueTest.java
    move to org.archive.bdb package
* BdbMultipleWorkQueues.java, PrefixFinderTest.java
    update for BDB-JE 4 changes
* BdbFrontier.java
    use getStoredQueue 
    drop unneeded custom serialization support
* WorkQueueFrontier.java
    drop unneeded custom serialization support
 2009-11-23 04:41:32 +00:00
gojomo daab317c94 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls 
processor-related bean checkpoint support:
* Checkpoint.java
    move to commons for proper dependency graph
* (others) 
    recognize above
 2009-11-20 01:10:32 +00:00
gojomo 8591002573 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls 
* BdbModule.java
    implement Checkpointable; save/restore consistent sets of jdb log files
* BdbModuleTest.java
    testDoCheckpoint
 2009-11-20 00:45:07 +00:00
gojomo dd0ca367c3 [HER-1706] PersistLoadProcessor and PersistStoreProcessor can't share same BdbModule 
* BdbModule.java
    allow Database instance reuse if BdbConfig is identical instance
* PersistProcessor.java
    convert shared BdbConfig for history db to static constant HISTORY_DB_CONFIG
* PersistOnlineProcessor.java
    use HISTORY_DB_CONFIG, don't close possibly-shared DB on stop()
* PrecedenceLoader.java, PreloadedUriPrecedencePolicy.java
    use HISTORY_DB_CONFIG
 2009-11-10 23:48:10 +00:00
gojomo 06223b2794 BDB JE tuning 
* BdbModule.java
    increase je.log.faultReadSize from 2K to 6K -- stats showed many faults (ideal is zero)
 2009-10-23 21:55:06 +00:00
gojomo b7631e2b02 Misc fixes from H1-to-H3 testing 
* BdbModule.java
    be more robust about 0/-1 as 'don't care' values
* AbstractFrontier.java
    reorder settable properties for ease of discovery
* FetchHTTP.java
    correct type to allow setting property
 2009-10-07 04:22:05 +00:00
gojomo c05044d51b Eliminate double-database close log messages 
* BdbModule.java
   split openDatabase() to openManagedDatabase() (auto-closed) and plain openDatabase() (caller-closes)
   set TempStoredSortedMap to use unmanaged openDatabase
* (others)
   use openManagedDatabase()
 2009-10-04 05:06:14 +00:00
gojomo 3cb6b59b8c [HER-769] Support case where millions of seeds. 
* BdbModule.java
    add temp-stored-map service method
    remove unused secondary-db support
* TempStoredSortedMap.java
    StoredSortedMap that can destroy its underlying database after temporary use
* StatisticsTracker.java
    update various sorted-by-decreasing-frequency methods to use temp StoredMaps, with duplicate keys that are negative counts
* (Multiple)Report.java
    update to use new duplicate-keyed frequency maps
 2009-10-03 23:28:11 +00:00
gojomo 5a0fe30366 Quiet JUnite test issues 
* BdbModuleTest.java, CachedBdbMapTest.java
    use putIfAbsent() in preference to put() (which logs warning)
* BdbModule.java
    register big-cache DBs in implementation-specific methods, so always auto-closed on env close
 2009-09-29 19:22:23 +00:00
gojomo e7ab73dd5d [HER-1677] threads stuck in CachedBdbMap.get/_getMem 
alternate simplified implementation of our object-cache need
* ObjectIdentityCache.java
    new interface, far less than (Concurrent)Map, for big seems-like-in-memory object-cache
* Supplier.java
    trivial interface for deferred-provision of new instance
* CachedBdbMap.java
    implement ObjectIdentityCache, mainly via getOrUse()
* ObjectIdentityMemCache.java
    trivial ConcurrentHashMap-based all-in-memory ObjectIdentityCache implementation
* ObjectIdentityBdbCache.java
    BDB-backed ObjectIdentityCache implementation, carved from CachedBdbMap
* BdbModule.java
    refactor utility methods to offer CachedBdbMap or ObjectIdentityBdbCache instances, via source toggle
* EnhancedEnvironment.java
    convenience test-environment method
* (others)
    update to use ObjectIdentityCache/ObjectIdentityMemCache in place of ConcurrentMap/ConcurrentHashMap
 2009-09-25 21:16:53 +00:00
gojomo 163dcb7826 [HER-1607] handle persist state when crawl starts up 
* BdbModule.java
    fail-fast if not started properly; stop with fewer errors in problem situations
* CrawlController.java
    continue past runtimeException on stop
* CrawlJob.java
    less redundant error-reporting
* AbstractFrontier.java
    consider PAUSE or FINISH valid targets after unexpected exception 
* StatisticsTracker.java
    avoid promoting runtimeException in report-generating up
* PersistOnlineProcessor.java
    delegate DB-closing to BdbModule to avoid double-close
 2009-09-11 02:26:04 +00:00
gojomo 608a3ddc8b [HER-1607] handle persist state when crawl starts up 
* PersistProcessor.java, PersistLoadProcessor.java
    port [HER-1586] preloadSource setting/refactoring to H3
* IoUtils.java
    (getBufferedReader) utility method
* BdbModule.java
    explicit passthrough of settable deferredWrite property
 2009-08-27 01:32:08 +00:00
gojomo 8d09ee38ae CachedBdbMap cleanup 2009-08-11 02:07:22 +00:00
gojomo 149fc414df [HER-1609] wide crawl performance is limited by lock contention in ServerCache.getServerFor(String) 
* (all, especially CachedBdbMap.java) 
    pbaclace-contributed patch, ported to H3, plus...
* StatisticsTracker.java
    replace LongWrapper with AtomicLong
    use non-recursive putIfAbsent-based idiom for get-or-create
* DefaultServerCache.java
    clean up host/server get-or-create around non-recursive concurrent idiom
* BdbModule.java
    add expectedConcurrency property & je.lock.nLockTables tuning
* TopNSet.java
    add synchronization, as calling StatisticsTracker method has lost it
 2009-07-06 23:36:17 +00:00
gojomo e047bf68e0 copy 'springy' branch to heritrix3 2009-05-11 22:56:36 +00:00