lee/heritrix3
1
0 Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-10-07 12:51:21 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,068 Commits 42 Branches 34 Tags
d3db71e3bb05367c0d63f7c4e7c83e110f5035c1
Commit Graph
478 Commits
Author SHA1 Message Date
Noah Levitt 4496c5d755 * FetchHTTPTest.java 
move to modules/src/test
    testFormAuth() - remove, there is FormAuthSelfTest.java, and this was the test that would fail in modules/
    testHttpBindAddress() - incorporate ed65493db0 - alternate testing of multiple local binds more likely to work more places
 2012-11-30 17:57:36 -08:00
Noah Levitt a20824aea7 Merge branch 'master' into new-fetchhttp-only 
Conflicts:
	.classpath
	engine/pom.xml
	modules/pom.xml
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java
	modules/src/test/java/org/archive/modules/fetcher/FetchHTTPTest.java
 2012-11-30 17:04:28 -08:00
Noah Levitt 4470ba512e Count url-agnostic content digest duplicates as duplicates in statistics 
* CrawledBytesHistotable.java
    accumulate() - count urls with warcRevisit:uriAgnosticDigest annotation as duplicates
* ContentDigestHistoryTest.java
    testWarcDedupe() - check for presence of warcRevisit:uriAgnosticDigest annotation
 2012-09-27 16:56:14 -07:00
Noah Levitt f9703fb5ef * engine/pom.xml 
remove incorrect, unnecessary dependency
 2012-09-26 14:55:03 -07:00
Noah Levitt b9cbbbd248 Bunch of unit tests for FetchHTTP 
* FetchHTTPTest.java
    a bunch of new tests
* FetchHTTP.java
    make sslTrustLevel work correctly (bug found by test; even with fix, only respects value at startup, no overrides)
* modules/pom.xml, .classpath
    dependencies for testing
* engine/pom.xml
    update jetty to 6.1.26 since we require that for the new tests
 2012-09-26 14:41:49 -07:00
Noah Levitt 7965a27b1d Fix infinite loop bug on server returning no response 
* RecordingSocketInputBuffer.java
    readLine(CharArrayBuffer) - return -1 (not 0) on end of stream
* RecordingHttpClient.java
    setHttpRequestRetryHandler() - never retry (may want to reevaluate later, old fetcher did retry sometimes)
* FetchHTTPTest.java
    testNoResponse()
 2012-09-13 19:10:11 -07:00
Noah Levitt dd7e0c5bd2 * profile-crawler-beans.cxml 
replace defunct commons-httpclient BdbCookieStorage with new httpcomponents BdbCookieStore
 2012-09-13 17:58:47 -07:00
Noah Levitt 77559bf81c **/pom.xml 
bump version number because of api change
 2012-09-13 17:47:26 -07:00
Noah Levitt 9699d6a6ed get rid of AbstractFetchHTTP 2012-09-08 01:17:32 -07:00
Noah Levitt 33df8faaf0 remove LegacyFetchHTTP and its classes 2012-09-08 00:49:30 -07:00
Noah Levitt 71761a72a7 rename FetchHTTP2 to FetchHTTP 2012-09-08 00:28:06 -07:00
Noah Levitt 917986454b rename FetchHTTP to LegacyFetchHTTP 2012-09-08 00:25:14 -07:00
Noah Levitt 24e813c858 test chunked transfer encoding 2012-09-07 13:28:41 -07:00
Noah Levitt 09f0e53119 testHttp11 2012-09-07 11:32:38 -07:00
Noah Levitt 9a0191a84c ssl trust level support for FetchHTTP2! another thing that doesn't even work properly in old FetchHTTP 2012-09-07 02:47:54 -07:00
Noah Levitt 0764096eb0 skip slow tests for old FetchHTTP for now 2012-09-07 02:42:13 -07:00
Noah Levitt f2f0ca59f6 embedded jetty https server 2012-09-06 19:28:07 -07:00
Noah Levitt 948d641b14 socket/connection timeout 2012-09-06 12:39:44 -07:00
Noah Levitt d11338d620 testFetchTimeout 2012-09-05 19:24:50 -07:00
Noah Levitt b19994e3bb Merge branch 'master' into fetchhttp-hc 2012-09-05 12:08:48 -07:00
Noah Levitt e711ff9506 * JobRelatedResource.java 
fix code formatting from 01be6d4b
 2012-09-04 18:03:38 -07:00
Noah Levitt baaf4f8092 testShouldFetchBodyRule 2012-09-04 13:25:51 -07:00
Noah Levitt ca606588ae test sendIfNoneMatch (etag) 2012-09-02 15:25:56 -07:00
Noah Levitt c03273230b sendIfModifiedSince 2012-09-02 13:00:01 -07:00
Noah Levitt 1fe4781cf5 axLengthBytes, sendRange 2012-09-02 01:37:00 -07:00
Noah Levitt 2d8687e55c maxFetchKBSec 2012-09-01 23:05:50 -07:00
Noah Levitt 5cb3b914e8 Merge branch 'master' into fetchhttp-hc 2012-09-01 22:11:59 -07:00
Adam Miller 01be6d4b18 Fix for HER-2018: XML representation for /engine/job/<jobName>/beans returns incorrect url for named beans 
* JobRelatedResource.java
    Added encoding to URL generation in makePresentableMapFor().
    Added check to prevent URL generation from overwriting the job name portion of the path in addPresentableNestedNames()
 2012-08-31 15:07:31 -07:00
Adam Miller 52e70a10ef Fix for HER-2017 XML representation for /engine/job/<jobName>/beans uses root node of type "script" instead of "beans" 
* BeanBrowseResource.java
	replaced "script" string in root node with "beans"
 2012-08-30 15:35:27 -07:00
Noah Levitt 3f2295353a test dependencies belong in engine/ now too 2012-08-30 12:16:25 -07:00
Noah Levitt cdd95cbb1f move FetchHTTP*Test* to engine/ subproject because testFormAuth() makes use of PreconditionEnforcer, part of engine/, and we want to keep that test 2012-08-30 09:55:20 -07:00
Noah Levitt eb3419aa59 deprecate method (GET/POST) in HtmlFormCredential, since PreconditionEnforcer always sets fetch type of login urls to POST (who sends password in a GET request anyway?) 2012-08-15 13:53:07 -07:00
Noah Levitt 864fea938b working on form auth test 2012-07-27 19:13:19 -07:00
Noah Levitt 82c983d2b3 Merge branch 'master' into fetchhttp-hc 
Conflicts:
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java

(conflicts are resolved)
 2012-07-15 11:58:41 -07:00
Travis Wellman c5d740a6d4 remove some dead or redundant code 
* ArchiveUtils
    sb.length() was tested twice and int i was never used
* PrefixFinder
    opCount is never used
* BdbMultipleWorkQueues
    tries is never used
 2012-07-11 16:57:13 -07:00
Travis Wellman c1a0525107 suppress unused warnings for serialVersionUid 
Toward the goal of having Eclipse warnings be useful to look at, suppressing
unused warnings for serialVersionUid will significantly reduce clutter.
 2012-07-10 14:12:33 -07:00
Noah Levitt 373abbbe81 latest jetty 6.x; starting on basic-auth test 2012-06-29 17:19:46 -07:00
Travis Wellman 6626b44286 generics warnings fixes 
This commit does not change any functionality, just
alters generics to reduce warnings in Eclipse.
 2012-06-21 18:23:06 -07:00
Noah Levitt 35378df2fa Merge branch 'master' into no-package-scope 2012-06-05 17:58:40 -07:00
Noah Levitt 3dce1e909d Change scope of most/all default-scope i.e. package-scoped class members to protected, to make heritrix more friendly to third-party extensions. (I don't think there was ever an intent to limit the scope of any of these to package-only.) 2012-06-05 17:55:41 -07:00
Noah Levitt e965421281 * WorkQueueFrontier.java 
remove unused variable
 2012-06-05 16:26:00 -07:00
Travis Wellman 07a2fc65ea use methods to specify which report rather than string constants 
* MultiReporter
    deleted. direct method calls are a better way to specify the kind of report desired
    getReports was never used to choose from the array of strings. no use case for MultiReporter.
* ArchiveUtils
    writeReportToString was not used, and it used the now deleted MultiReporter#reportTo
* DevUtils, Frontier, ToePool, AbstractFrontier, WorkQueueFrontier
    MultiReporter -> Reporter and make methods that implemented named multireport public
* FrontierNonemptyReport, FrontierSummaryReport
    call the appropriate report method instead of passing a string constant
 2012-06-01 17:57:25 -07:00
Travis Wellman 493e2423f6 begin removing MultiReporter 
* MultiReporter.java
    deprecated
* ToePool, ToeThread, WorkQueue, WorkQueueFrontier, PrecedenceProvider, CrawlURI, FetchStats.java
    remove MultiReporter#getReports() and reportTo(String, PrintWriter)
    these are not used as MultiReporter specifies, and MultiReporter is going away anyhow
 2012-06-01 17:14:41 -07:00
Noah Levitt 1015c5fa00 Fix HER-1973 Regression in queue budget overrides 
* WorkQueueFrontier.java
    findEligibleURI() - apply sheet overrides before refreshing queue budgets
 2012-05-15 17:35:49 -07:00
Noah Levitt 9fffa8fc53 **/pom.xml 
bump version to 3.1.2-SNAPSHOT
 2012-05-01 18:17:33 -07:00
Noah Levitt 79c40346fc **/pom.xml 
* bump version number to 3.1.1 for release
 2012-05-01 18:01:06 -07:00
Gordon Mohr e4f12bb20f Merge pull request #6 from ato/codemirror 
HER-2001: Use the CodeMirror editor for crawl config and script console
    (Thanks Alex!)
 2012-04-25 15:04:10 -07:00
Noah Levitt 42e759c01d List correct defaults for BdbModule cacheSize and cachePercentage with an explanatory comment. 2012-04-25 11:10:56 -07:00
Noah Levitt da28258f90 Option to set bdb cache size as an absolute number instead of a percentage (part of HER-1948) 2012-04-19 18:16:53 -07:00
Noah Levitt 4a64c6ba59 Revert "Altered FetchStats to also keep track of the number of URIs that have" 
This reverts commit 4c1740ce1e.
 2012-04-18 15:53:49 -07:00