lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-28 09:26:54 +00:00
Code Issues Packages Projects Releases Wiki Activity
158 Commits 38 Branches 32 Tags
7bd6017448e7f0dc82e0c19e84b4cae1f79b14d8
Commit Graph
158 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
gojomo 7bd6017448 * ObjectIdentityBdbCache.java 
avoid error if size() called after close()
 2009-09-26 04:12:23 +00:00
gojomo 71aa2ad51d * StatisticsTracker.java, HostsReport.java 
rename getReverseSortedHostsDistribution() so it isn't misinterpreted as beans property
 2009-09-26 04:09:20 +00:00
gojomo a285a65576 * CachedBdbMap.java 
remove superfluous (build-breaking) imports
 2009-09-25 22:50:43 +00:00
gojomo 7521375128 [HER-1677] threads stuck in CachedBdbMap.get/_getMem 
alternate simplified implementation of our object-cache need
* ObjectIdentityCache.java
    new interface, far less than (Concurrent)Map, for big seems-like-in-memory object-cache
* Supplier.java
    trivial interface for deferred-provision of new instance
* CachedBdbMap.java
    implement ObjectIdentityCache, mainly via getOrUse()
* ObjectIdentityMemCache.java
    trivial ConcurrentHashMap-based all-in-memory ObjectIdentityCache implementation
* ObjectIdentityBdbCache.java
    BDB-backed ObjectIdentityCache implementation, carved from CachedBdbMap
* BdbModule.java
    refactor utility methods to offer CachedBdbMap or ObjectIdentityBdbCache instances, via source toggle
* EnhancedEnvironment.java
    convenience test-environment method
* (others)
    update to use ObjectIdentityCache/ObjectIdentityMemCache in place of ConcurrentMap/ConcurrentHashMap
 2009-09-25 21:18:14 +00:00
gojomo e7ab73dd5d [HER-1677] threads stuck in CachedBdbMap.get/_getMem 
alternate simplified implementation of our object-cache need
* ObjectIdentityCache.java
    new interface, far less than (Concurrent)Map, for big seems-like-in-memory object-cache
* Supplier.java
    trivial interface for deferred-provision of new instance
* CachedBdbMap.java
    implement ObjectIdentityCache, mainly via getOrUse()
* ObjectIdentityMemCache.java
    trivial ConcurrentHashMap-based all-in-memory ObjectIdentityCache implementation
* ObjectIdentityBdbCache.java
    BDB-backed ObjectIdentityCache implementation, carved from CachedBdbMap
* BdbModule.java
    refactor utility methods to offer CachedBdbMap or ObjectIdentityBdbCache instances, via source toggle
* EnhancedEnvironment.java
    convenience test-environment method
* (others)
    update to use ObjectIdentityCache/ObjectIdentityMemCache in place of ConcurrentMap/ConcurrentHashMap
 2009-09-25 21:16:53 +00:00
szznax 78dd7f2086 Fix for [HER-1648] harmonize declared WARC versions in protocol, warcinfo record metadata 
WARConstants.java
  update WARC_VERSION=1.0 according to ISO 28500:2009 as of May 2009
WARCWriterProcessor.java
  update format and conformsTo fields according to ISO 28500:2009,
  as of May 2009, as described at http://bibnum.bnf.fr/WARC/,
  latest draft as of November 2008
  format: WARC File Format 1.0
  conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf
 2009-09-23 18:42:25 +00:00
gojomo 2bd9334627 [HER-1677] threads stuck in CachedBdbMap.get/_getMem 
* CachedBdbMap.java
    make all WARNING logs include stacks
    add WARNINGs if memMap.remove() has no expected effect
    verify reported expunction, WARN and try to fix if memMap expunction not evident
    adjust indentation/bracketing
 2009-09-22 21:16:21 +00:00
gojomo 142c72fc04 [HER-1668] support '-' SURT directives in seed files analogous to '+' (if a REJECTing SurtPrefixedDecideRule is present) 
* SurtPrefixedDecideRule.java
    pass only portion after directive ('+' or '-') to SurtPrefixSet
* SurtPrefixSet.java
    better comment/param name
 2009-09-16 01:05:23 +00:00
gojomo ee1c9e2ce4 [HER-1607] handle persist state when crawl starts up 
* PersistProcessor.java
    add readOnly option for environment only opened for copying-from
* PersistLoadProcessor.java
    make start() safe to call redundantly (per Lifecycle contract)
 2009-09-12 00:22:06 +00:00
gojomo 163dcb7826 [HER-1607] handle persist state when crawl starts up 
* BdbModule.java
    fail-fast if not started properly; stop with fewer errors in problem situations
* CrawlController.java
    continue past runtimeException on stop
* CrawlJob.java
    less redundant error-reporting
* AbstractFrontier.java
    consider PAUSE or FINISH valid targets after unexpected exception 
* StatisticsTracker.java
    avoid promoting runtimeException in report-generating up
* PersistOnlineProcessor.java
    delegate DB-closing to BdbModule to avoid double-close
 2009-09-11 02:26:04 +00:00
gojomo d5e088ed6a * WorkQueueFrontier.java 
improve frontier report: move longest-queue to top, add labels/counters to long lists of queue groups
 2009-09-10 22:40:30 +00:00
gojomo d955e31963 [HER-1676] improve bundled profiles to match best-practice, be self-documenting 
* defaults.xml
    add REJECTing SurtPrefixedDecideRule and MatchesListRegExpDecideRule
 2009-09-10 22:39:26 +00:00
gojomo 1c451021d1 * PagedRepresentation.java 
make 'end' link go to true end, not end-as-of-moment-of-rendering
 2009-09-10 21:57:21 +00:00
gojomo 1e3ec9fe35 * SheetAssociation.java 
initialize list of names for easier in-crawl construction/insertion
 2009-09-10 21:56:44 +00:00
gojomo a564b7a867 [HER-1675] FetchStats/CrawlSubstats not tallying fetchResponses correctly; as a result QuotaEnforcer doesn't work 
* FetchStats.java
    increment fetchResponses appropriately
 2009-09-09 00:12:54 +00:00
gojomo 0b10862c8d Make during-crawl sheets/overlay changes easier 
* SheetOverlaysManager.java
    convenience methods for during-crawl changes:
    (getOrCreateSheet)
    (deleteSheet)
    (putSheetOverlay)
    (removeSheetOverlay)
* Sheet.java
    improved comments
 2009-09-03 06:33:25 +00:00
gojomo c88744ab27 Improve bean-browse UI 
* BeanBrowseResource.java
    coerce edits to type of value they're replacing; makes for safer mid-crawl edits of values in maps
* JobRelatedResource.java
    improve rendering/clickability of map, indexed objects
 2009-09-03 03:32:07 +00:00
gojomo f26803bd4f UI tweaks 
* JobResource.java
    make jaob main page begin with job title -- easier to distinguish tabs with different jobs in them
 2009-09-03 03:27:05 +00:00
gojomo 635d27b390 More action directory capabilities 
* ActionDirectory.java
    execute scripts found in action directory, determining language by their suffix
 2009-09-03 03:15:54 +00:00
gojomo e03329a035 [HER-1668] support '-' SURT directives in seed files analogous to '+' (if a REJECTing SurtPrefixedDecideRule is present) 
* SurtPrefixedDecideRule.java
    allow seeds to be null (as when rule is intentionally completely 
    independent of seeds; may also need to disqualify rule from
    Spring autowiring in such a case)
 2009-09-02 23:24:30 +00:00
gojomo 21d90f2a3a UI annoyances 
* ReportGenResource.java
    ensure redirect after regeneration of report doesn't show stale cached version
 2009-09-02 23:15:18 +00:00
gojomo d4eabb9631 Improve log display 
* JobResource.java
    form proper alerts-rev-tail link
    for any file ending ".log", use rev-tail-paged view
 2009-09-01 22:42:52 +00:00
gojomo 81a658daa3 Update BDB-JE to 3.3.87 
* commons/pom.xml, .classpath
    update version reference
 2009-09-01 22:40:21 +00:00
gojomo f0421e1f13 * ActionDirectory.java 
also include 'Fi ' lines -- for recoveries of recoveries
 2009-08-28 17:08:59 +00:00
gojomo ad937c58e5 [HER-1662] use fully qualified hostname 
* FetchHTTP.java
    replace one more InetAddress.getHostName() with InetAddress.getCanonicalHostName() (for consistency)
 2009-08-28 06:56:20 +00:00
gojomo d253ff1ace [HER-1662] use fully qualified hostname 
* WriterPoolProcessor.java, ARCWriterProcessor.java, WARCWriterProcessor.java, InetAddressUtil.java 
    replace all InetAddress.getHostName() with InetAddress.getCanonicalHostName() (for consistency)
 2009-08-27 23:35:07 +00:00
gojomo 211cd7c659 [HER-1669] slow crawl/contention n WARCWriterProcessor 
* CrawlController.java
    increase default size of in-memory recording buffers 
    outBuffer (requests): 4K -> 16K
    inBuffer (responses): 64K -> 512K
 2009-08-27 23:13:11 +00:00
gojomo 8ba6d47a93 Apache License 2009-08-27 23:10:38 +00:00
gojomo dc89ce0d95 [HER-1668] support '-' SURT directives in seed files analogous to '+' (if a REJECTing SurtPrefixedDecideRule is present) 
* SurtPrefixSet.java
    refactor to allow add-directives under outside control
* SurtPrefixedDecideRule.java
    extend nonseedLine() mechanism to trigger off '+' or '-' depending on decision being ACCEPT or REJECT
 2009-08-27 01:40:26 +00:00
gojomo 3292c8b7f6 Apache License, import cleanup 2009-08-27 01:34:11 +00:00
gojomo 608a3ddc8b [HER-1607] handle persist state when crawl starts up 
* PersistProcessor.java, PersistLoadProcessor.java
    port [HER-1586] preloadSource setting/refactoring to H3
* IoUtils.java
    (getBufferedReader) utility method
* BdbModule.java
    explicit passthrough of settable deferredWrite property
 2009-08-27 01:32:08 +00:00
gojomo beb858369b Apache License 2009-08-27 01:27:28 +00:00
gojomo f7af3e016b [HER-1655] Excessive charset/character-encoding exceptions; is fallback behavior correct? 
* ReplayCharSequence.java, 
    return to ISO8859_1 as fallback-encoding
 2009-08-27 00:20:37 +00:00
gojomo 783ecfd9d7 [HER-1655] Excessive charset/character-encoding exceptions; is fallback behavior correct? 
* ReplayCharSequence.java, GenericReplayCharSequence.java, InMemoryReplayCharSequence.java
    refactor to put fallback-encoding in single place
 2009-08-27 00:12:15 +00:00
gojomo 3f9ecc4dfa * PagedRepresentation.java 
fix negative estimated size error for log-tail viewing
 2009-08-27 00:10:48 +00:00
gojomo c07a70eb21 Fix SubElement IllegalArgumentException (via WarcWriterProcessor.writeMetadata) 
* SubElement.java
    accept zero-length strings as valid values
* WarcWriterProcessor.java
    don't write 'via' or 'hopsFromSeed' unless non-blank
 2009-08-19 21:15:40 +00:00
gojomo 5c93d7bd28 Rename DecideResult.PASS to DecideResult.NONE for clarity 
* DecideResult.java + references
    PASS was confusable with ACCEPT; NONE better conveys "no decision"
 2009-08-14 21:31:11 +00:00
gojomo b78fce045b Fix selftests hanging due to pauseAtStart default true 
* selftest-crawler-beans.cxml
    make selftests not pauseAtStart
 2009-08-13 10:03:56 +00:00
gojomo d654bb5651 [HER-1660] allow adding seeds, recovery-journal-like URI data at any point in crawl via 'action directory' 
* ActionDirectory.java
     support for .seeds, .recover, .include, .schedule, .force files
* CrawlController.java
    change default for pause-at-start to true; ensure enters PAUSED rather than PAUSING
* Frontier.java 
    (importRecoverFormat) support method for .recover, .include, .schedule, .force additions
    (considerIncluded) change to accept CrawlURI
* AbstractFrontier.java
    (importRecoverFormat) support method for .recover, .include, .schedule, .force additions
* BdbFrontier.java
    remove too-strict assertion
* FrontierJournal.java
    rename recover-log 'frontier.recover.gz' (so that '.recover' is a dot-suffix)
* WorkQueueFrontier.java
    (considerIncluded) change to accept CrawlURI
 2009-08-12 01:50:51 +00:00
gojomo bba028ce21 * StatisticsTracker.java, SeedsReport.java 
remove redundant method
 2009-08-12 01:35:19 +00:00
gojomo a37a1846b7 Move getBufferedReader() -> IoUtils 
* CrawlerJournal.java, IoUtils.java
    move getBufferedReader to IoUtils
* FrontierJournal.java
    add new 'included' tag for log lines ("Fi")
* (others)
    update for relocated method
 2009-08-11 02:17:20 +00:00
gojomo 8b95e7fb36 CachedBdbMap cleanup - intended commit comment 
* CachedBdbMap.java
    discard code to open/manage own BDB Environments (and related constructor) 
    cleanup comments/FIXMEs
* BdbModule.java
    return CachedBdbMap, rather than more general ConcurrentMap, in case requester wants it
* CachedBdbMapTest.java
    get instance via BdbModule (to manage env/classCatalog)
 2009-08-11 02:10:28 +00:00
gojomo 8d09ee38ae CachedBdbMap cleanup 2009-08-11 02:07:22 +00:00
gojomo 023efe4aba [HER-1660] allow adding seeds, recovery-journal-like URI data at any point in crawl via 'action directory' 
* ActionDirectory.java
    initial implementation; handle '.seeds' files
 2009-08-06 21:33:31 +00:00
gojomo 0400d81ee5 support for... 
[HER-1660] allow adding seeds, recovery-journal-like URI data at any point in crawl via 'action directory' 
[HER-769] Support case where millions of seeds. 
* AbstractFrontier.java
    move sourceTagSeeds setting to SeedModule
    trigger and schedule seeds via SeedModule's announcements and SeedListener protocols
* StatisticsTracker.java
    implement SeedListener, use processedSeedRecords as list of all seeds
* ProcessorURI.java, DefaultProcessorURI.java
    (getURI) convenience accessor for plain URI string
* SurtPrefixedDecideRule.java
    use seeds as prefixes via SeedListener notifications, rather than separate scan of seeds source text
* SeedListener.java
    simplify to notifications of new seed, and nonseed text mixed with seeds (as with SURT directives)
* SeedModule.java
    methods for SeedListener interface, announceSeeds, actOn(File)
* TextSeedModule.java
    eliminate interators; add methods for announcing seeds and scanning seed text sources (including arbitrary Files)
* PrefixSet.java, SurtPrefixSet.java
    derive from ConcurrentSkipListSet; eliminate now-unneeded extra synchronization and clone()s 
    offer considerAsDirective() for direct application of directive Strings
 2009-08-06 21:24:04 +00:00
gojomo 4d06b239ee Push CrawlURI and related constants down to modules subproject/package 
(many)
 2009-08-06 20:22:00 +00:00
gojomo 0ebac2f115 * CachedBdbMapTest.java 
more timing/heap voodoo for test robustness
 2009-08-06 20:11:04 +00:00
gojomo 64d3fd212f [HER-1659] make default WARC size comply with spec; adjust default pool size for fewer odd-sized (W)ARCs 
* ArchiveFileConstants.java
    remove superfluous constant
* WriterPool.java
    DEFAULT_MAX_ACTIVE -> 1
* WriterPoolMember.java
    update constant use; remove deprecated+unused methods
* WriterPoolProcessor.java
    pull storePaths up from subclasses, but set default via subclass method
    set defaultMaxSize via subclass method
* WARCWriterProcessor.java, ARCWriterProcessor.java
   (getDefaultMaxFileSize), (getDefaultStorePaths) as appropriate
 2009-08-06 00:55:27 +00:00
gojomo 272ae19961 Cleanup 
* CachedBdbMap.java
    remove debug output
* TestUtils.java
    add info logging
* CachedBdbMapTest.java
    make more robust against prior heap usage, different platforms
 2009-08-04 06:55:31 +00:00
gojomo 4f0961c371 Fix NPE 
* URIAuthorityBasedQueueAssignmentPolicy.java
    tolerate null getRawPathQuery response
 2009-08-04 01:31:38 +00:00