lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-29 09:05:53 +00:00
Code Issues Packages Projects Releases Wiki Activity
299 Commits 38 Branches 33 Tags
faa8e899db66f8c397b68bb06bec103d12530bf5
Commit Graph
14 Commits
Author SHA1 Message Date
gojomo faa8e899db * CachedBdbMap.java 
remove unnecessary/build-breaking CrawlController import
 2009-11-23 09:26:08 +00:00
gojomo d93129115e BDBJE updates, refactoring 
* CachedBdbMap.java
    update for BDB-JE 4.0.71 exception changes
 2009-11-23 08:03:30 +00:00
gojomo c2c0e5116e [HER-1705] 'harmless' low-memory-canary causing heap reference leak/OOME 
* CachedBdbMap.java, ObjectIdentityBdbCache.java
    only reinstall canary if map/cache still has 'db' field
 2009-11-09 03:55:12 +00:00
gojomo a285a65576 * CachedBdbMap.java 
remove superfluous (build-breaking) imports
 2009-09-25 22:50:43 +00:00
gojomo e7ab73dd5d [HER-1677] threads stuck in CachedBdbMap.get/_getMem 
alternate simplified implementation of our object-cache need
* ObjectIdentityCache.java
    new interface, far less than (Concurrent)Map, for big seems-like-in-memory object-cache
* Supplier.java
    trivial interface for deferred-provision of new instance
* CachedBdbMap.java
    implement ObjectIdentityCache, mainly via getOrUse()
* ObjectIdentityMemCache.java
    trivial ConcurrentHashMap-based all-in-memory ObjectIdentityCache implementation
* ObjectIdentityBdbCache.java
    BDB-backed ObjectIdentityCache implementation, carved from CachedBdbMap
* BdbModule.java
    refactor utility methods to offer CachedBdbMap or ObjectIdentityBdbCache instances, via source toggle
* EnhancedEnvironment.java
    convenience test-environment method
* (others)
    update to use ObjectIdentityCache/ObjectIdentityMemCache in place of ConcurrentMap/ConcurrentHashMap
 2009-09-25 21:16:53 +00:00
gojomo 2bd9334627 [HER-1677] threads stuck in CachedBdbMap.get/_getMem 
* CachedBdbMap.java
    make all WARNING logs include stacks
    add WARNINGs if memMap.remove() has no expected effect
    verify reported expunction, WARN and try to fix if memMap expunction not evident
    adjust indentation/bracketing
 2009-09-22 21:16:21 +00:00
gojomo 8b95e7fb36 CachedBdbMap cleanup - intended commit comment 
* CachedBdbMap.java
    discard code to open/manage own BDB Environments (and related constructor) 
    cleanup comments/FIXMEs
* BdbModule.java
    return CachedBdbMap, rather than more general ConcurrentMap, in case requester wants it
* CachedBdbMapTest.java
    get instance via BdbModule (to manage env/classCatalog)
 2009-08-11 02:10:28 +00:00
gojomo 8d09ee38ae CachedBdbMap cleanup 2009-08-11 02:07:22 +00:00
gojomo 272ae19961 Cleanup 
* CachedBdbMap.java
    remove debug output
* TestUtils.java
    add info logging
* CachedBdbMapTest.java
    make more robust against prior heap usage, different platforms
 2009-08-04 06:55:31 +00:00
gojomo c196a1de77 [HER-1658] CachedBdbMaps not expunging as expected (especially StatisticsTracker.processedSeedsRecords) 
* CachedBdbMap.java
    do expunge on put(), replace()
    add low-memory-sensitive 'canary' to force expunge even if otherwise untriggered
* CachedBdbMapTest.java
    add test of idle expunge in low-memory conditions
* CrawlURI.java
    (getURI) direct access to String URI (don't reuse toString() functionally)
* SeedRecord.java
    support for updating record with later/repeat report
* StatisticsTracker.java
    remove put()s for processedSeedRecords, hostsLastFinished
* BdbFrontier.java
    (getQueueFor) remove put, do in concurrent-compliant manner
 2009-08-04 00:06:22 +00:00
gojomo d2a82228a3 TODO/FIXME comments where things don't seem quite right but aren't definitively buggy 
* CachedBdbMap.java
    reevaluate if clear() should include close()
* CrawlURI.java
    investigate possible confusion in meaning of isPrerequisite()
 2009-07-28 22:56:54 +00:00
gojomo 1128ad638b [HER-1609] wide crawl performance is limited by lock contention in ServerCache.getServerFor(String) 
* CachedBdbMap.java
    remove unnecessary import of CrawlController (from engine subproject)
 2009-07-07 01:35:56 +00:00
gojomo 149fc414df [HER-1609] wide crawl performance is limited by lock contention in ServerCache.getServerFor(String) 
* (all, especially CachedBdbMap.java) 
    pbaclace-contributed patch, ported to H3, plus...
* StatisticsTracker.java
    replace LongWrapper with AtomicLong
    use non-recursive putIfAbsent-based idiom for get-or-create
* DefaultServerCache.java
    clean up host/server get-or-create around non-recursive concurrent idiom
* BdbModule.java
    add expectedConcurrency property & je.lock.nLockTables tuning
* TopNSet.java
    add synchronization, as calling StatisticsTracker method has lost it
 2009-07-06 23:36:17 +00:00
gojomo e047bf68e0 copy 'springy' branch to heritrix3 2009-05-11 22:56:36 +00:00