lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-20 12:46:00 +00:00
Code Issues Packages Projects Releases Wiki Activity
398 Commits 38 Branches 33 Tags
e10a14df5415d4408a08035289db862def8ecfbe
Commit Graph
398 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
gojomo e10a14df54 HER-1753 offer means of setting login/password that doesn't leak to process info (as when literally specified to '-a' option) 
* Heritrix.java
    allow [username:]password to come from file, supplied to '-a' with '@' prefix
    also, clean up options names/help text to match current behavior
    also, offer warning about particularly weak (<8 characters, all letters, all numbers) passwords
 2010-04-27 01:36:38 +00:00
gojomo b639873dae Fix broken test 
* Engine.java
    tolerate (with return-false-forfailure rather than NPE) non-job directories
* EngineTest.java
    use empty temp job directory rather than (possibly full-of-junk) general junit-tests directory
 2010-04-27 00:44:36 +00:00
szznax 57c857d5c6 Refinement of fixes for HER-1688 remember external job directories 
* EngineResource.java
    avoid calling findJobConfigs() on each page load
 2010-04-26 23:33:40 +00:00
gojomo 5034c7d96e correct spaces-for-tabs 2010-04-26 22:28:43 +00:00
gojomo c4946d04d0 HER-1688 remember external job directories 
* EngineResource.java, Engine.java
    refactor for less branching/redundancy
 2010-04-26 22:22:09 +00:00
szznax 14b8649aa9 minor cleanup/clarification on review 2010-04-26 21:49:41 +00:00
gojomo 27cca4da60 Refinement of HER-1737 excessive job.log WARNINGs for misreported encoding (FLV reported as text/html;charset=UTF-8 
* ReplayCharSequence.java, GenericReplayCharSequence.java, InMemoryReplayCharSequence.java
    store actual first exception for reference
* ExtractorHTML.java
    report exception in nonfatal-log, rather than WARNING (which was still polluting job log more than desired -- but should still be logged to allow future investigation of sites where encoding issues may exist)
 2010-04-24 02:22:24 +00:00
gojomo 9fc5e42cfe Fix failing test 
* InMemoryReplayCharSequence.java
    ensure redecode after problem restarts at right position
 2010-04-23 22:57:50 +00:00
gojomo 792d8e528b HER-1766 The use of the MatchesFilePatternDecideRule with the CUSTOM preset causes a fatal error (infinite loop) 
* MatchesFilePatternDecideRule.java
    fix getRegex() to super.getRegex()
 2010-04-23 22:16:25 +00:00
gojomo 2855572278 ? 2010-04-23 04:20:12 +00:00
gojomo a7caceb8ad HER-1737 excessive job.log WARNINGs for misreported encoding (FLV reported as text/html;charset=UTF-8 
* ReplayCharSequence.java
    new interface method: getDecoderExceptionCount(), reporting >0 if any decoding problems were encountered and silently replaced
* GenericReplayCharSequence.java
    count decoding problems, and only log as FINE rather than WARNING
* InMemoryReplayCharSequence.java
    try decoding with reporting; if any problems note at least one then decode without reporting errors
* ExtractorHTML.java
    after using ReplayCharSequence, log WARNING if nonzero decoding errors
 2010-04-23 04:11:53 +00:00
gojomo 67bf73026f remove obsolete/moved example properties 2010-04-23 04:05:31 +00:00
gojomo cb5ec19b6c HER-1765 quiet excessive INFO level logging 
* GenericReplayCharSequence.java
* Scoper.java
* FetchFTP.java
* FetchHTTP.java
    demote a bunch of INFO logging to FINE so that be default, console isn't filled with output for every URI handled by a busy crawl
 2010-04-23 00:44:25 +00:00
gojomo 3f0059c8a8 * BeanBrowseResource.java 
remove extraneous close() that seemed to be causing (harmless but distracting) console error stacks after web hits
 2010-04-23 00:41:25 +00:00
gojomo 9fd648f3ef HER-1467 Allow periodic checkpointing and checkpoint-copy-logs to be set/unset during crawl 
* CheckpointService.java
    reorganize so that checkpoint timedTask is reset whenever interval is reset on a running crawl
* CheckpointSuccessEvent.java, CrawlJob.java
    report successes as new kind of ApplicationEvent; propagagte to job log
* BdbModule.java
    remove obsolete unconsulted checkpointCopyLogs setting
 2010-04-23 00:15:44 +00:00
gojomo ca32d0d448 HER-1692 Cannot create profile with same name as job 
* JobResource.java
    if destination of copy-job fails, show flash error
 2010-04-22 01:59:41 +00:00
gojomo c1207582c4 HER-1728 Heritrix could be enhanced to parse flashvars parameter to correctly extract the links 
* ExtractorHTMLTest.java
    tests for embed/@flshvars, param[@name=flashvars]/@value likely-URI discovery
* ExtractorHTML.java
    move likely-URI testing to UriUtils
    add FLASHVARS support
* JerichoExtractorHTML.java
    use UriUtils ikely-URI testing
    add FLASHVARS support
 2010-04-22 01:27:24 +00:00
gojomo f933dc922d HER-1763 unify/improve/document likely-URI string tests 
* UriUtils.java
    move likely-URI heuristics here for better consistency, reuse, testing, improvement
* UriUtilsTest.java
    testing for UriUtils
* ExtractorJS.java
    move likely-URI support to UriUtils; use new test
* ExtractoSWF.java
    use new UriUtils likely-URI testing
* ExtractorHTML.java
    move likely-URI testing to UriUtils, use new test
* JerichoExtractorHTML.java
    use new UriUtils likely-URI testing
 2010-04-22 01:08:14 +00:00
gojomo 72fb1c55bd Move TLD list to resource file for easier drop-in updating 
* ArchiveUtils.java
    read TLDs from tlds-alpha-by-domain.txt
* ArchiveUtilsTest.java
    verify TLDs read
* tlds-alpha-by-domain.txt
    from http://data.iana.org/TLD/tlds-alpha-by-domain.txt
 2010-04-22 01:05:23 +00:00
gojomo f018a2e6f1 * StatisticsTracker.java 
correct outdated comment
 2010-04-13 22:47:19 +00:00
gojomo affa3b672c HER-1620 Tolerate trailing '*' in robots disallow (& allow?) rules 
* Robotstxt.java
    trim single trailing '*' if present (assume unintentional or redundant)
 2010-04-13 22:46:23 +00:00
gojomo 9e12e4abe5 test/deprecation cleanup 
* CachedBdbMap.java
   @deprecated
* BdbModule.java
   note (unused) 'CBM' option is deprecated
* ObjectIdentityBdbCache.java
   tighten logging to FINE
* ObjectIdentityBdbCacheTest.java
   fix test that was only trying to log, rather than fail, problems
 2010-04-13 02:15:53 +00:00
gojomo a82838cd27 HER-1756 improved crawl status reporting: more definitive start-after-setup, conclusion-after-logging of crawling 
* AbstractFrontier.java
    call new finalTasks just before announcing FINISH state
* WorkQueueFrontier.java
    eliminate closeQueue
* BdbFrontier.java
    move dumpAllPendingToLog from closeQueue (triggered by CrawlController FINISHED) to finalTasks (before Frontier FINISH)
    replace closeQueue with close() override
 2010-04-13 02:10:56 +00:00
gojomo 8be7be2983 fix race causing intermittent test failures on MacOS 
* CrawlJob.java
    synchronize methods which mutate or change behavior based on field 'ac'; previously, some self-tests were exitting/failing immediately because they were not seeing a non-null 'ac' value.
 2010-04-13 00:55:09 +00:00
gojomo 1698412130 HER-1450 Reading at EOF using read(byte[]) causes position pointer to decrement. 
* ARCWriterTest.java
    one more test to ensure the problem with read(void) was exercised/fixed
 2010-04-09 21:25:26 +00:00
szznax cc6fb841d9 part of fix for [HER-1450] Reading at EOF using read(byte[]) causes position pointer to decrement. 
* ArchiveRecord.java
    do not decrement position after EOR, port from H1
* ARCWriterTest.java
    added testArchiveRecordEORConsistent(), testArchiveRecordMarkSupport()
        port from H1
 2010-04-09 18:25:16 +00:00
gojomo 647a0b3edf HER-1754 treat robots.txt non-response as 404 (optionally?) 
* CrawlServer.java
    enable a specific kind of connect-lost (no response given) to be counted as 'deemed 404' on robots retries over a minimum
* FetchStatusCodes.java
    add a synthetic code: -404, S_DEEMED_NOT_FOUND
* FetchHTTP.java
    better distinguish CONNECT_FAILED and CONNECT_LOST
* DispositionProcessor.java
    treat CONNECT_LOST same as _FAILED; simplify a test
 2010-04-02 03:55:12 +00:00
szznax 693f050fb7 successfully adding external job dir is ACK flash kind 2010-03-29 15:56:09 +00:00
gojomo fd5644f109 HER-1627 followup: show error rather than match faulReadSize 
* PersistProcessor.java
    rename setupEnvironment to setupCopyEnvironment to make limited purpose of env clear; catch and rethrow IllegalArgumentException with hint that second-open may be cause
* PersistLoadProcessor.java 
    catch and display IllegalArgumentException same as other failure-to-open SEVERE but proceedable exceptions
* PrecedenceLoader.java
    update method names
 2010-03-26 01:02:48 +00:00
gojomo 37c32b4e5c Maven cleanup to enable warningless build on fresh machine with Maven2.2.0: 
* engine/pom.xml
    restore a reference to a (new version of a) necessary servlet library
* pom.xml
    remove deprecated {pom.artifactId}; add needed javadoc module version decl
* .classpath
    update ref to currently-used servlet jar
 2010-03-23 22:57:27 +00:00
szznax 9f77412581 part of fix for HER-1627 incorrect source tag in crawl.log (of persist-loaded crawl) 
* EnhancedEnvironment.java
    in getClassCatalog(), when read-only env, insist on read-only catalog
* PersistLoadProcessor.java
    remove existing curi members from prior to prevent clobbering source
* PersistProcessor.java
    set historyDbConfig read-only, set je.log.faultReadSize for
        compatability with BDBModule.setup()
 2010-03-23 20:52:40 +00:00
nlevitt 20b8a7bc90 mention url in question in debug level logging 2010-03-19 18:39:14 +00:00
szznax af85fed7f4 part of fix for HER-1748 Arc2Warc should generally write WARC "response" records for ARC HTTP responses 
* Arc2Warc.java
    reverted writing ARC namedFields, deferred for further analysis as part of HER-1750
 2010-03-12 23:44:07 +00:00
szznax cc8ea6494a changed namedField "ARC-Compressed-Offset" to "ARC-Gzip-Offset" for even more clarity 2010-03-11 15:59:10 +00:00
szznax 4802e5093e part of fix for HER-1748 Arc2Warc should generally write WARC "response" records for ARC HTTP responses 
* Arc2Warc.java
    changed namedField "ARC-Offset" to "ARC-Compressed-Offset" for clarity
 2010-03-11 01:45:49 +00:00
szznax 8a567c9bc0 part of fix for HER-1748 Arc2Warc should generally write WARC "response" records for ARC HTTP responses 
* Arc2Warc.java
    removed unnecessary URISyntaxException import
    moved ARC named fields outside of check for IP in header field
 2010-03-10 00:33:38 +00:00
szznax 65f962dc25 whitespace corrections 2010-03-08 18:07:30 +00:00
szznax 65d552ce5c fix for HER-1748 Arc2Warc should generally write WARC "response" records for ARC HTTP responses 
* Arc2Warc.java
    when HTTP response, write response record rather than resource
    enable reconstruction of ARC from WARC via namedFields
* ARCRecord.java
    added headerString and accessor to save verbatim ARC header string
 2010-03-08 17:43:42 +00:00
szznax 3e5cf29315 nit correction in javadoc 2010-03-08 17:35:47 +00:00
nlevitt 86268def97 * LongToIntConsistentHash.java 
Skip unnecessary conversion from CharSequence to String
 2010-02-25 18:19:46 +00:00
szznax e9df65d446 fix for HER-1744 Incorrect comparison of Strings in ProcessorChain.process() with "!=" operator 
* ProcessorChain.java
    use equals() to compare string values rather than objects (with "!=")
 2010-02-16 19:13:43 +00:00
szznax ed3d64d91e part of fix for HER-1736 long pause at startup reading large per-job job.log(s) 
* CrawlJob.java
 2010-02-16 19:03:59 +00:00
gojomo acfe95fea3 naming/branching/visibility review for HER-1736 long pause at startup reading large per-job job.log(s) 
* CrawlJob.java
    added accessor, renamed a bit, unified 2 branches
* JobResource.java
    use accessor
 2010-02-13 00:47:41 +00:00
szznax e1308e80f5 reverting last checkin, also testing passwordless checkin with Karmic Eclipse 3.5.1 SVNKit 1.6.7 update 2010-02-13 00:21:22 +00:00
szznax 6c13f42242 testing Karmic Eclipse 3.5.1 (w/updated SVNKit 1.6.7) 2010-02-13 00:18:10 +00:00
szznax 2d441d611e part of fix for HER-1736 long pause at startup reading large per-job job.log(s) 
* CrawlJob.java
    added boolean partialJobLogScan, and print "at least X launches" when true
* JobResource.java
    print "at least X launches" when partialJobLogScan
 2010-02-12 23:21:59 +00:00
szznax 18d459f6b9 part of fix for HER-1736 long pause at startup reading large per-job job.log(s) 
* CrawlJob.java
    check for excessively long job log, scan near end of file for lastLaunch
    inform user OK to move big job log aside
 2010-02-12 18:54:57 +00:00
szznax 4097e29d50 part of fix for HER-1736 long pause at startup reading large per-job job.log(s) 
* CrawlJob.java
    return early from scanJobLog() when job log > 1GB
 2010-02-09 19:45:23 +00:00
szznax a53d9ea733 per Gordon's suggestions; renamed "Engine.css" to "engine.css" and changed flashACK class background-color to palegreen 2010-01-26 20:54:16 +00:00
szznax 0b5e5b30e3 removed references to unused repos, and updated servlet-api to use newer jetty referenced repo 2010-01-26 20:31:36 +00:00