Commit Graph
314 Commits
Author SHA1 Message Date
Noah Levitt d71fbb3a1a stick a basic auth 401 in the middle of testDigestAuth to check it doesn't mess with the digest auth we're working on 2012-07-26 18:45:29 -07:00
Noah Levitt 4e50c32578 Merge branch 'master' into fetchhttp-hc
Conflicts:
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java
2012-07-20 21:36:23 -07:00
Kenji Nagahashi 3078eac4ab test case for scheme-less relative URL 2012-07-19 14:15:22 -07:00
Kenji Nagahashi 8fc44b626c added a test case for catching HER-2013 2012-07-19 14:15:22 -07:00
Noah Levitt e9b6bb3fd6 Merge branch 'master' into fetchhttp-hc 2012-07-16 17:33:02 -07:00
Noah Levitt eee6c6cd30 * ArchiveRecord.java
fix indentation only (no substantive changes)
2012-07-16 09:57:24 -07:00
Noah Levitt c108c4c451 HER-2014 Make ArchiveRecord.getPosition() public instead of protected.
* ArchiveRecord.java, HeaderedArchiveRecord.java
    getPosition() - change visibility to public
2012-07-16 09:54:54 -07:00
Noah Levitt 82c983d2b3 Merge branch 'master' into fetchhttp-hc
Conflicts:
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java

(conflicts are resolved)
2012-07-15 11:58:41 -07:00
Travis Wellman c5d740a6d4 remove some dead or redundant code
* ArchiveUtils
    sb.length() was tested twice and int i was never used
* PrefixFinder
    opCount is never used
* BdbMultipleWorkQueues
    tries is never used
2012-07-11 16:57:13 -07:00
Travis Wellman c1a0525107 suppress unused warnings for serialVersionUid
Toward the goal of having Eclipse warnings be useful to look at, suppressing
unused warnings for serialVersionUid will significantly reduce clutter.
2012-07-10 14:12:33 -07:00
Travis Wellman 4ca6e9cc99 revert a change that may break bdb history 2012-07-03 17:38:43 -07:00
Noah Levitt 93958abfcd these help classes for fetchhttp2 need to be in modules/ since they use stuff from modules/ 2012-06-29 11:19:56 -07:00
Noah Levitt 1a1d1b008c allow FetchHTTP2 infrastructure to fall back on default java hostname resolution 2012-06-29 02:40:34 -07:00
Noah Levitt 1f315a7cc7 unit tests for FetchHTTP and FetchHTTP2 2012-06-29 02:04:39 -07:00
Noah Levitt 58d827c85b Merge branch 'master' into fetchhttp-hc 2012-06-28 18:08:39 -07:00
Noah Levitt 6225f6fa0b Fix HER-2013 scheme-less relative URI not resolved correctly
* LaxURI.java
    parseUriReference() - fix logic testing if uri string is a path to respect scheme-less uris even if they contain colons
* UURIFactoryTest.java
    add test for scheme-less relative uri (with colons)
2012-06-28 16:49:50 -07:00
Noah Levitt 63c6ca59c2 Merge branch 'master' into fetchhttp-hc 2012-06-25 16:42:34 -07:00
Travis Wellman 6626b44286 generics warnings fixes
This commit does not change any functionality, just
alters generics to reduce warnings in Eclipse.
2012-06-21 18:23:06 -07:00
Noah Levitt 2b0141fd0d bdb cookie store for FetchHTTP2 2012-06-20 18:49:57 -07:00
Noah Levitt 49c00c0387 do not rely on ToeThread for the ServerCache, pass it to where it's needed instead 2012-06-19 13:54:05 -07:00
Noah Levitt 56f0c92690 resolve hostnames from heritrix server cache (the bad old way) 2012-06-19 13:32:34 -07:00
Noah Levitt 1d0beedf2b use http 1.0 by default, other request configuration stuff 2012-06-19 12:09:46 -07:00
Noah Levitt de152eaa9e browser-compatibility mode for cookie matching; that's what old FetchHTTP does, and also avoids sending weird Cookie2:$Version1 2012-06-18 17:49:09 -07:00
Noah Levitt 3c7bd8afd6 no keep-alive 2012-06-18 17:06:46 -07:00
Noah Levitt 7422a7b992 move httpclient helper classes to commons/ 2012-06-18 15:43:52 -07:00
Noah Levitt 6ab7415342 Revert "* ConfigFileEditor.java, ConfigPathEditor.java"
This reverts commit 6dae8f9a17.

Revert to fix this problem: ... Failed to convert property value of type 'java.lang.String' to required type 'org.archive.spring.ConfigPath' for property 'path'; nested exception is java.lang.IllegalStateException: Cannot convert value of type [java.lang.String] to required type [org.archive.spring.ConfigPath] for property 'path': no matching editors or conversion strategy found
2012-06-12 10:15:27 -07:00
Noah Levitt 6dae8f9a17 * ConfigFileEditor.java, ConfigPathEditor.java
remove unused classes
2012-06-11 17:21:23 -07:00
Noah Levitt 99acaa9bfa Fix tests revealed to be failing
* UriUtils.java
    isLikelyFalsePositive() - do not flag urls starting with ../ as likely false positives
* UriUtilsTest.java
    remove test url that is now rejected
2012-06-06 09:00:13 -07:00
Noah Levitt 342ea33be5 Move test classes into test area of source tree
* LongToIntConsistentHashTest.java, UriUtilsTest.java
2012-06-05 18:03:33 -07:00
Noah Levitt 35378df2fa Merge branch 'master' into no-package-scope 2012-06-05 17:58:40 -07:00
Noah Levitt 3dce1e909d Change scope of most/all default-scope i.e. package-scoped class members to protected, to make heritrix more friendly to third-party extensions. (I don't think there was ever an intent to limit the scope of any of these to package-only.) 2012-06-05 17:55:41 -07:00
Travis Wellman 07a2fc65ea use methods to specify which report rather than string constants
* MultiReporter
    deleted. direct method calls are a better way to specify the kind of report desired
    getReports was never used to choose from the array of strings. no use case for MultiReporter.
* ArchiveUtils
    writeReportToString was not used, and it used the now deleted MultiReporter#reportTo
* DevUtils, Frontier, ToePool, AbstractFrontier, WorkQueueFrontier
    MultiReporter -> Reporter and make methods that implemented named multireport public
* FrontierNonemptyReport, FrontierSummaryReport
    call the appropriate report method instead of passing a string constant
2012-06-01 17:57:25 -07:00
Travis Wellman 493e2423f6 begin removing MultiReporter
* MultiReporter.java
    deprecated
* ToePool, ToeThread, WorkQueue, WorkQueueFrontier, PrecedenceProvider, CrawlURI, FetchStats.java
    remove MultiReporter#getReports() and reportTo(String, PrintWriter)
    these are not used as MultiReporter specifies, and MultiReporter is going away anyhow
2012-06-01 17:14:41 -07:00
Noah Levitt 52d5307a9b Merge branch 'her-1948' 2012-04-20 12:33:19 -07:00
Noah Levitt 8d0a1678de * effective_tld_names.dat
update to latest from publicsuffix.org
2012-04-20 12:25:57 -07:00
Noah Levitt da28258f90 Option to set bdb cache size as an absolute number instead of a percentage (part of HER-1948) 2012-04-19 18:16:53 -07:00
Kenji Nagahashi 27d621f21a renamed PublicSuffixes2 to PublicSuffixes.
deleted debug println.
2012-04-18 10:14:51 -07:00
Kenji Nagahashi a0d52e6c17 PublicSuffixes2Test still had tests for PublicSuffix. fixed. 2012-04-18 10:14:51 -07:00
Kenji Nagahashi b2d62a81ff trial rewrite of PublicSuffixes for small memory footprint (no significant performance gain) 2012-04-18 10:14:50 -07:00
Noah Levitt 31df3a8f03 Merge branch 'master' into speculative-links 2012-04-13 18:16:08 -07:00
Noah Levitt 52703c6eb4 * ArchiveUtils.java
doubleToString() - return "NaN" for NaN (inf symbol didn't quite make sense to me)
2012-03-15 18:46:01 -07:00
Kenji Nagahashi adc82c1c6f Second round of HTML cleanup:
- added missing <html> and <body>
- fixed non-working CSS link, added CSS link to all pages
- replaced layout by <br> and &nbsp; with structural tags with CSS
- removed superfluous output, notably in BeanBrowseResource
- <code>-ify variable names in ScriptResource
- added special handling of NaN in doubleToString()
2012-03-03 00:46:56 -08:00
Noah Levitt db801b3e0f Fix problems handling crawl job paths with characters that require url-encoding, since we sometimes refer to them with file: urls
* PathSharingContext.java, CrawlJob.java
    use java.net.URI to convert paths to and from file: urls
2012-02-10 15:48:09 -08:00
Noah Levitt cfac7ead91 Fix for HER-1985 H3: SurtPrefixDecideRule forgets learned/seed-derived/seed-directive SURT prefixes in checkpoint-resume
* SurtPrefixedDecideRule.java
    make class implement Checkpointable, save surt prefixes to json on checkpoint and load them on recover
* profile-crawler-beans.cxml
    make main SurtPrefixedDecideRule a top-level bean so that it can be checkpointed
* Checkpoint.java, CheckpointService.java
    add some FINE level logging during checkpointing and recovery
2012-02-09 10:05:07 -08:00
Noah Levitt 9dbe673bc3 * UriUtils.java
some improvements to likely-uri detection based on crawl log analysis
2012-01-20 11:23:45 -08:00
Noah Levitt 803f9ca979 HER-1984 save script state - implement by adding a map to the application context for arbitrary use, and make sure the app context is available in all scripting environments
* PathSharingContext.java
    new member variable ConcurrentHashMap data and accessor getData()
* ScriptedProcessor.java, ScriptedDecideRule.java
    make appCtx available to scripts; also remove unused member sharedMap
* ActionDirectory.java
    formatting fix
2012-01-19 13:31:57 -08:00
Noah Levitt 995133dc40 Fix for HER-1962 NPE from missing sheet
* SheetOverlaysManager.java
    getOverlayMap(String) - return null if sheet missing instead of triggering npe
* KeyedProperties.java
    get(String) - check for null return value from getOverlayMap() and log warning
2011-10-27 10:40:30 -07:00
nlevitt 32680b8f73 Fix for HER-1960 kryo buffer can hog memory, lead to oome (bug report, patch from Kenji)
* KryoBinding.java
    wrap ObjectBuffers with WeakReference so that can be garbage collected if necessary (when recreated they'll be back at the default size of 16k)
2011-10-20 02:21:54 +00:00
nlevitt 351a89c5a9 Fix java.lang.IllegalArgumentException from CharBuffer.allocate() in GenericReplayCharSequence.decode()
Reported by Kenji who says, "My guess is that root cause is Recorder.calcRecommendedCharBufferSize():
    return Math.min(inStream.getRecordedBufferLength()/2,(int)inStream.getSize());
    URL above returns HTML larger than 5GB (infinite smileys!! what the heck), and (int)intStream.getSize() became negative."
* Recorder.java
    -        return Math.min(inStream.getRecordedBufferLength()/2,(int)inStream.getSize());
    +        return (int) Math.min(inStream.getRecordedBufferLength()/2, inStream.getSize());
2011-10-13 15:47:02 +00:00
nlevitt a09c8e69b8 * Recorder.java
remove TODOs that have already been done
2011-10-12 17:52:45 +00:00