lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-25 16:06:33 +00:00
Code Issues Packages Projects Releases Wiki Activity
168 Commits 38 Branches 32 Tags
03df2a9a993b5b2d959f75ceb22d88a67eab8da1
Commit Graph
168 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
szznax 03df2a9a99 As part of fix for [HER-1671] WARCWriterProcessor error when writing problematic (illegal control character?) outlink - percent encode control-characters in 'userinfo' part of URI as we do for uriPath and uriQuery, so we don't raise exception when adding outlinks to metadata record 
* UURIFactory.java
  ensureMinimalEscaping() on 'userinfo' in UURIFactory.fixupAuthority()
 2009-10-02 01:39:44 +00:00
szznax 677a816db4 Fix for [HER-1681] FetchFTP module doesn't set content-begin mark correctly. 
Bug reported and fix suggested by Tomas Ukkonen.
* FetchFTP.java
    saveToRecorder(): call recorder.markContentBegin() after recorder.outputWrap()
 2009-10-01 02:21:17 +00:00
szznax fef7a93bad As part of fix for [HER-1634] verify/port/update latest WARC 
implementation in 1.14.3/heritrix2 to latest ISO document,
heritrix3 - in warcinfo record, "robots" field should be
"robots: classic" rather than "robots: CLASSIC"
* WARCWriterProcessor.java
  made robots entry in metadata lower-case
 2009-09-30 17:06:53 +00:00
szznax 3e65e09927 As part of fix for [HER-1634] verify/port/update latest WARC 
implementation in 1.14.3/heritrix2 to latest ISO document,
heritrix3 - update REVISIT profile URIs per WARC spec (May 2009)
* WARCConstants.java
  PROFILE_REVISIT_IDENTICAL_DIGEST =
    "http://netpreserve.org/warc/1.0/revisit/identical-payload-digest"
  PROFILE_REVISIT_NOT_MODIFIED =
    "http://netpreserve.org/warc/1.0/revisit/server-not-modified"
 2009-09-29 21:52:33 +00:00
gojomo 1951930842 Improve test robustness in face of unpredictable GC 
* ObjectIdentityBdbCache.java
   (pageOutStaleEntries) package-protection for unit-test visibility
* ObjectIdentityBdbCacheTest.java
   explicit pageOutStaleEntries, better assert-fail message
 2009-09-29 20:20:14 +00:00
gojomo 5a0fe30366 Quiet JUnite test issues 
* BdbModuleTest.java, CachedBdbMapTest.java
    use putIfAbsent() in preference to put() (which logs warning)
* BdbModule.java
    register big-cache DBs in implementation-specific methods, so always auto-closed on env close
 2009-09-29 19:22:23 +00:00
gojomo 99020c4981 [HER-1672] parallelQueues with overlays - prerequisites not landing in right queues (causes massive DNS-fetch repeats) 
* PreconditionEnforcer.java, CrawlServer.java
    in testing found some ordering issues affecting robots-scheduling if parallelQueues>1
    reordered/refactored to put more responsibility in CrawlServer to resolve
 2009-09-29 00:47:38 +00:00
szznax 1cfcf8e0a5 As part of fix for [HER-1634] verify/port/update latest WARC implementation in $ 
fold in H3 updates to http transaction constant for revisit records.
* CoreAttributeConstants.java
  removed A_HTTP_TRANSACTION, in favor checking CrawlURI method with isHttpTran$
* ModelAttributeConstants.java
  removed A_HTTP_TRANSACTION, in favor checking CrawlURI method with isHttpTran$
* FetchHistoryProcessor.java
  check for HTTP transaction using CrawlURI.isHttpTransaction()
  use HttpMethod in saveHeader() instead of HttpMethodBase
* WARCWriterProcessor.java
  check for HTTP transaction using CrawlURI.isHttpTransaction()
  use HttpMethod in saveHeader() instead of HttpMethodBase
 2009-09-29 00:41:57 +00:00
gojomo c2b71a6f95 * WriterPoolProcessor.java 
fix serialization test: make servercache field transient
 2009-09-26 04:41:09 +00:00
gojomo 1c1a6501ba [HER-1672] parallelQueues with overlays - prerequisites not landing in right queues (causes massive DNS-fetch repeats) 
* CrawlURI.java
    (getPolicyBasisUURI) return either own UURI, or -- for prereqs -- via UURI, as basis for overlays/policy calculations
* URIAuthorityBasedQueueAssignmentPolicy.java
    use getPolicyBasisUURI for queue-name decisions
* SheetOverlaysManager.java
    use getPolicyBasisUURI for surt-based overlay decisions
 2009-09-26 04:18:25 +00:00
gojomo 7bd6017448 * ObjectIdentityBdbCache.java 
avoid error if size() called after close()
 2009-09-26 04:12:23 +00:00
gojomo 71aa2ad51d * StatisticsTracker.java, HostsReport.java 
rename getReverseSortedHostsDistribution() so it isn't misinterpreted as beans property
 2009-09-26 04:09:20 +00:00
gojomo a285a65576 * CachedBdbMap.java 
remove superfluous (build-breaking) imports
 2009-09-25 22:50:43 +00:00
gojomo 7521375128 [HER-1677] threads stuck in CachedBdbMap.get/_getMem 
alternate simplified implementation of our object-cache need
* ObjectIdentityCache.java
    new interface, far less than (Concurrent)Map, for big seems-like-in-memory object-cache
* Supplier.java
    trivial interface for deferred-provision of new instance
* CachedBdbMap.java
    implement ObjectIdentityCache, mainly via getOrUse()
* ObjectIdentityMemCache.java
    trivial ConcurrentHashMap-based all-in-memory ObjectIdentityCache implementation
* ObjectIdentityBdbCache.java
    BDB-backed ObjectIdentityCache implementation, carved from CachedBdbMap
* BdbModule.java
    refactor utility methods to offer CachedBdbMap or ObjectIdentityBdbCache instances, via source toggle
* EnhancedEnvironment.java
    convenience test-environment method
* (others)
    update to use ObjectIdentityCache/ObjectIdentityMemCache in place of ConcurrentMap/ConcurrentHashMap
 2009-09-25 21:18:14 +00:00
gojomo e7ab73dd5d [HER-1677] threads stuck in CachedBdbMap.get/_getMem 
alternate simplified implementation of our object-cache need
* ObjectIdentityCache.java
    new interface, far less than (Concurrent)Map, for big seems-like-in-memory object-cache
* Supplier.java
    trivial interface for deferred-provision of new instance
* CachedBdbMap.java
    implement ObjectIdentityCache, mainly via getOrUse()
* ObjectIdentityMemCache.java
    trivial ConcurrentHashMap-based all-in-memory ObjectIdentityCache implementation
* ObjectIdentityBdbCache.java
    BDB-backed ObjectIdentityCache implementation, carved from CachedBdbMap
* BdbModule.java
    refactor utility methods to offer CachedBdbMap or ObjectIdentityBdbCache instances, via source toggle
* EnhancedEnvironment.java
    convenience test-environment method
* (others)
    update to use ObjectIdentityCache/ObjectIdentityMemCache in place of ConcurrentMap/ConcurrentHashMap
 2009-09-25 21:16:53 +00:00
szznax 78dd7f2086 Fix for [HER-1648] harmonize declared WARC versions in protocol, warcinfo record metadata 
WARConstants.java
  update WARC_VERSION=1.0 according to ISO 28500:2009 as of May 2009
WARCWriterProcessor.java
  update format and conformsTo fields according to ISO 28500:2009,
  as of May 2009, as described at http://bibnum.bnf.fr/WARC/,
  latest draft as of November 2008
  format: WARC File Format 1.0
  conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf
 2009-09-23 18:42:25 +00:00
gojomo 2bd9334627 [HER-1677] threads stuck in CachedBdbMap.get/_getMem 
* CachedBdbMap.java
    make all WARNING logs include stacks
    add WARNINGs if memMap.remove() has no expected effect
    verify reported expunction, WARN and try to fix if memMap expunction not evident
    adjust indentation/bracketing
 2009-09-22 21:16:21 +00:00
gojomo 142c72fc04 [HER-1668] support '-' SURT directives in seed files analogous to '+' (if a REJECTing SurtPrefixedDecideRule is present) 
* SurtPrefixedDecideRule.java
    pass only portion after directive ('+' or '-') to SurtPrefixSet
* SurtPrefixSet.java
    better comment/param name
 2009-09-16 01:05:23 +00:00
gojomo ee1c9e2ce4 [HER-1607] handle persist state when crawl starts up 
* PersistProcessor.java
    add readOnly option for environment only opened for copying-from
* PersistLoadProcessor.java
    make start() safe to call redundantly (per Lifecycle contract)
 2009-09-12 00:22:06 +00:00
gojomo 163dcb7826 [HER-1607] handle persist state when crawl starts up 
* BdbModule.java
    fail-fast if not started properly; stop with fewer errors in problem situations
* CrawlController.java
    continue past runtimeException on stop
* CrawlJob.java
    less redundant error-reporting
* AbstractFrontier.java
    consider PAUSE or FINISH valid targets after unexpected exception 
* StatisticsTracker.java
    avoid promoting runtimeException in report-generating up
* PersistOnlineProcessor.java
    delegate DB-closing to BdbModule to avoid double-close
 2009-09-11 02:26:04 +00:00
gojomo d5e088ed6a * WorkQueueFrontier.java 
improve frontier report: move longest-queue to top, add labels/counters to long lists of queue groups
 2009-09-10 22:40:30 +00:00
gojomo d955e31963 [HER-1676] improve bundled profiles to match best-practice, be self-documenting 
* defaults.xml
    add REJECTing SurtPrefixedDecideRule and MatchesListRegExpDecideRule
 2009-09-10 22:39:26 +00:00
gojomo 1c451021d1 * PagedRepresentation.java 
make 'end' link go to true end, not end-as-of-moment-of-rendering
 2009-09-10 21:57:21 +00:00
gojomo 1e3ec9fe35 * SheetAssociation.java 
initialize list of names for easier in-crawl construction/insertion
 2009-09-10 21:56:44 +00:00
gojomo a564b7a867 [HER-1675] FetchStats/CrawlSubstats not tallying fetchResponses correctly; as a result QuotaEnforcer doesn't work 
* FetchStats.java
    increment fetchResponses appropriately
 2009-09-09 00:12:54 +00:00
gojomo 0b10862c8d Make during-crawl sheets/overlay changes easier 
* SheetOverlaysManager.java
    convenience methods for during-crawl changes:
    (getOrCreateSheet)
    (deleteSheet)
    (putSheetOverlay)
    (removeSheetOverlay)
* Sheet.java
    improved comments
 2009-09-03 06:33:25 +00:00
gojomo c88744ab27 Improve bean-browse UI 
* BeanBrowseResource.java
    coerce edits to type of value they're replacing; makes for safer mid-crawl edits of values in maps
* JobRelatedResource.java
    improve rendering/clickability of map, indexed objects
 2009-09-03 03:32:07 +00:00
gojomo f26803bd4f UI tweaks 
* JobResource.java
    make jaob main page begin with job title -- easier to distinguish tabs with different jobs in them
 2009-09-03 03:27:05 +00:00
gojomo 635d27b390 More action directory capabilities 
* ActionDirectory.java
    execute scripts found in action directory, determining language by their suffix
 2009-09-03 03:15:54 +00:00
gojomo e03329a035 [HER-1668] support '-' SURT directives in seed files analogous to '+' (if a REJECTing SurtPrefixedDecideRule is present) 
* SurtPrefixedDecideRule.java
    allow seeds to be null (as when rule is intentionally completely 
    independent of seeds; may also need to disqualify rule from
    Spring autowiring in such a case)
 2009-09-02 23:24:30 +00:00
gojomo 21d90f2a3a UI annoyances 
* ReportGenResource.java
    ensure redirect after regeneration of report doesn't show stale cached version
 2009-09-02 23:15:18 +00:00
gojomo d4eabb9631 Improve log display 
* JobResource.java
    form proper alerts-rev-tail link
    for any file ending ".log", use rev-tail-paged view
 2009-09-01 22:42:52 +00:00
gojomo 81a658daa3 Update BDB-JE to 3.3.87 
* commons/pom.xml, .classpath
    update version reference
 2009-09-01 22:40:21 +00:00
gojomo f0421e1f13 * ActionDirectory.java 
also include 'Fi ' lines -- for recoveries of recoveries
 2009-08-28 17:08:59 +00:00
gojomo ad937c58e5 [HER-1662] use fully qualified hostname 
* FetchHTTP.java
    replace one more InetAddress.getHostName() with InetAddress.getCanonicalHostName() (for consistency)
 2009-08-28 06:56:20 +00:00
gojomo d253ff1ace [HER-1662] use fully qualified hostname 
* WriterPoolProcessor.java, ARCWriterProcessor.java, WARCWriterProcessor.java, InetAddressUtil.java 
    replace all InetAddress.getHostName() with InetAddress.getCanonicalHostName() (for consistency)
 2009-08-27 23:35:07 +00:00
gojomo 211cd7c659 [HER-1669] slow crawl/contention n WARCWriterProcessor 
* CrawlController.java
    increase default size of in-memory recording buffers 
    outBuffer (requests): 4K -> 16K
    inBuffer (responses): 64K -> 512K
 2009-08-27 23:13:11 +00:00
gojomo 8ba6d47a93 Apache License 2009-08-27 23:10:38 +00:00
gojomo dc89ce0d95 [HER-1668] support '-' SURT directives in seed files analogous to '+' (if a REJECTing SurtPrefixedDecideRule is present) 
* SurtPrefixSet.java
    refactor to allow add-directives under outside control
* SurtPrefixedDecideRule.java
    extend nonseedLine() mechanism to trigger off '+' or '-' depending on decision being ACCEPT or REJECT
 2009-08-27 01:40:26 +00:00
gojomo 3292c8b7f6 Apache License, import cleanup 2009-08-27 01:34:11 +00:00
gojomo 608a3ddc8b [HER-1607] handle persist state when crawl starts up 
* PersistProcessor.java, PersistLoadProcessor.java
    port [HER-1586] preloadSource setting/refactoring to H3
* IoUtils.java
    (getBufferedReader) utility method
* BdbModule.java
    explicit passthrough of settable deferredWrite property
 2009-08-27 01:32:08 +00:00
gojomo beb858369b Apache License 2009-08-27 01:27:28 +00:00
gojomo f7af3e016b [HER-1655] Excessive charset/character-encoding exceptions; is fallback behavior correct? 
* ReplayCharSequence.java, 
    return to ISO8859_1 as fallback-encoding
 2009-08-27 00:20:37 +00:00
gojomo 783ecfd9d7 [HER-1655] Excessive charset/character-encoding exceptions; is fallback behavior correct? 
* ReplayCharSequence.java, GenericReplayCharSequence.java, InMemoryReplayCharSequence.java
    refactor to put fallback-encoding in single place
 2009-08-27 00:12:15 +00:00
gojomo 3f9ecc4dfa * PagedRepresentation.java 
fix negative estimated size error for log-tail viewing
 2009-08-27 00:10:48 +00:00
gojomo c07a70eb21 Fix SubElement IllegalArgumentException (via WarcWriterProcessor.writeMetadata) 
* SubElement.java
    accept zero-length strings as valid values
* WarcWriterProcessor.java
    don't write 'via' or 'hopsFromSeed' unless non-blank
 2009-08-19 21:15:40 +00:00
gojomo 5c93d7bd28 Rename DecideResult.PASS to DecideResult.NONE for clarity 
* DecideResult.java + references
    PASS was confusable with ACCEPT; NONE better conveys "no decision"
 2009-08-14 21:31:11 +00:00
gojomo b78fce045b Fix selftests hanging due to pauseAtStart default true 
* selftest-crawler-beans.cxml
    make selftests not pauseAtStart
 2009-08-13 10:03:56 +00:00
gojomo d654bb5651 [HER-1660] allow adding seeds, recovery-journal-like URI data at any point in crawl via 'action directory' 
* ActionDirectory.java
     support for .seeds, .recover, .include, .schedule, .force files
* CrawlController.java
    change default for pause-at-start to true; ensure enters PAUSED rather than PAUSING
* Frontier.java 
    (importRecoverFormat) support method for .recover, .include, .schedule, .force additions
    (considerIncluded) change to accept CrawlURI
* AbstractFrontier.java
    (importRecoverFormat) support method for .recover, .include, .schedule, .force additions
* BdbFrontier.java
    remove too-strict assertion
* FrontierJournal.java
    rename recover-log 'frontier.recover.gz' (so that '.recover' is a dot-suffix)
* WorkQueueFrontier.java
    (considerIncluded) change to accept CrawlURI
 2009-08-12 01:50:51 +00:00
gojomo bba028ce21 * StatisticsTracker.java, SeedsReport.java 
remove redundant method
 2009-08-12 01:35:19 +00:00