Commit Graph
441 Commits
Author SHA1 Message Date
Noah Levitt 25234898ef Somewhat redundant form auth integration test restored, passes 2012-12-30 22:35:28 -08:00
Noah Levitt 4496c5d755 * FetchHTTPTest.java
move to modules/src/test
    testFormAuth() - remove, there is FormAuthSelfTest.java, and this was the test that would fail in modules/
    testHttpBindAddress() - incorporate ed65493db0 - alternate testing of multiple local binds more likely to work more places
2012-11-30 17:57:36 -08:00
Noah Levitt a20824aea7 Merge branch 'master' into new-fetchhttp-only
Conflicts:
	.classpath
	engine/pom.xml
	modules/pom.xml
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java
	modules/src/test/java/org/archive/modules/fetcher/FetchHTTPTest.java
2012-11-30 17:04:28 -08:00
Noah Levitt 4470ba512e Count url-agnostic content digest duplicates as duplicates in statistics
* CrawledBytesHistotable.java
    accumulate() - count urls with warcRevisit:uriAgnosticDigest annotation as duplicates
* ContentDigestHistoryTest.java
    testWarcDedupe() - check for presence of warcRevisit:uriAgnosticDigest annotation
2012-09-27 16:56:14 -07:00
Noah Levitt 7965a27b1d Fix infinite loop bug on server returning no response
* RecordingSocketInputBuffer.java
    readLine(CharArrayBuffer) - return -1 (not 0) on end of stream
* RecordingHttpClient.java
    setHttpRequestRetryHandler() - never retry (may want to reevaluate later, old fetcher did retry sometimes)
* FetchHTTPTest.java
    testNoResponse()
2012-09-13 19:10:11 -07:00
Noah Levitt dd7e0c5bd2 * profile-crawler-beans.cxml
replace defunct commons-httpclient BdbCookieStorage with new httpcomponents BdbCookieStore
2012-09-13 17:58:47 -07:00
Noah Levitt 9699d6a6ed get rid of AbstractFetchHTTP 2012-09-08 01:17:32 -07:00
Noah Levitt 33df8faaf0 remove LegacyFetchHTTP and its classes 2012-09-08 00:49:30 -07:00
Noah Levitt 71761a72a7 rename FetchHTTP2 to FetchHTTP 2012-09-08 00:28:06 -07:00
Noah Levitt 917986454b rename FetchHTTP to LegacyFetchHTTP 2012-09-08 00:25:14 -07:00
Noah Levitt 24e813c858 test chunked transfer encoding 2012-09-07 13:28:41 -07:00
Noah Levitt 09f0e53119 testHttp11 2012-09-07 11:32:38 -07:00
Noah Levitt 9a0191a84c ssl trust level support for FetchHTTP2! another thing that doesn't even work properly in old FetchHTTP 2012-09-07 02:47:54 -07:00
Noah Levitt 0764096eb0 skip slow tests for old FetchHTTP for now 2012-09-07 02:42:13 -07:00
Noah Levitt f2f0ca59f6 embedded jetty https server 2012-09-06 19:28:07 -07:00
Noah Levitt 948d641b14 socket/connection timeout 2012-09-06 12:39:44 -07:00
Noah Levitt d11338d620 testFetchTimeout 2012-09-05 19:24:50 -07:00
Noah Levitt b19994e3bb Merge branch 'master' into fetchhttp-hc 2012-09-05 12:08:48 -07:00
Noah Levitt e711ff9506 * JobRelatedResource.java
fix code formatting from 01be6d4b
2012-09-04 18:03:38 -07:00
Noah Levitt baaf4f8092 testShouldFetchBodyRule 2012-09-04 13:25:51 -07:00
Noah Levitt ca606588ae test sendIfNoneMatch (etag) 2012-09-02 15:25:56 -07:00
Noah Levitt c03273230b sendIfModifiedSince 2012-09-02 13:00:01 -07:00
Noah Levitt 1fe4781cf5 axLengthBytes, sendRange 2012-09-02 01:37:00 -07:00
Noah Levitt 2d8687e55c maxFetchKBSec 2012-09-01 23:05:50 -07:00
Noah Levitt 5cb3b914e8 Merge branch 'master' into fetchhttp-hc 2012-09-01 22:11:59 -07:00
Adam Miller 01be6d4b18 Fix for HER-2018: XML representation for /engine/job/<jobName>/beans returns incorrect url for named beans
* JobRelatedResource.java
    Added encoding to URL generation in makePresentableMapFor().
    Added check to prevent URL generation from overwriting the job name portion of the path in addPresentableNestedNames()
2012-08-31 15:07:31 -07:00
Adam Miller 52e70a10ef Fix for HER-2017 XML representation for /engine/job/<jobName>/beans uses root node of type "script" instead of "beans"
* BeanBrowseResource.java
	replaced "script" string in root node with "beans"
2012-08-30 15:35:27 -07:00
Noah Levitt cdd95cbb1f move FetchHTTP*Test* to engine/ subproject because testFormAuth() makes use of PreconditionEnforcer, part of engine/, and we want to keep that test 2012-08-30 09:55:20 -07:00
Noah Levitt eb3419aa59 deprecate method (GET/POST) in HtmlFormCredential, since PreconditionEnforcer always sets fetch type of login urls to POST (who sends password in a GET request anyway?) 2012-08-15 13:53:07 -07:00
Noah Levitt 864fea938b working on form auth test 2012-07-27 19:13:19 -07:00
Travis Wellman c5d740a6d4 remove some dead or redundant code
* ArchiveUtils
    sb.length() was tested twice and int i was never used
* PrefixFinder
    opCount is never used
* BdbMultipleWorkQueues
    tries is never used
2012-07-11 16:57:13 -07:00
Travis Wellman c1a0525107 suppress unused warnings for serialVersionUid
Toward the goal of having Eclipse warnings be useful to look at, suppressing
unused warnings for serialVersionUid will significantly reduce clutter.
2012-07-10 14:12:33 -07:00
Travis Wellman 6626b44286 generics warnings fixes
This commit does not change any functionality, just
alters generics to reduce warnings in Eclipse.
2012-06-21 18:23:06 -07:00
Noah Levitt 35378df2fa Merge branch 'master' into no-package-scope 2012-06-05 17:58:40 -07:00
Noah Levitt 3dce1e909d Change scope of most/all default-scope i.e. package-scoped class members to protected, to make heritrix more friendly to third-party extensions. (I don't think there was ever an intent to limit the scope of any of these to package-only.) 2012-06-05 17:55:41 -07:00
Noah Levitt e965421281 * WorkQueueFrontier.java
remove unused variable
2012-06-05 16:26:00 -07:00
Travis Wellman 07a2fc65ea use methods to specify which report rather than string constants
* MultiReporter
    deleted. direct method calls are a better way to specify the kind of report desired
    getReports was never used to choose from the array of strings. no use case for MultiReporter.
* ArchiveUtils
    writeReportToString was not used, and it used the now deleted MultiReporter#reportTo
* DevUtils, Frontier, ToePool, AbstractFrontier, WorkQueueFrontier
    MultiReporter -> Reporter and make methods that implemented named multireport public
* FrontierNonemptyReport, FrontierSummaryReport
    call the appropriate report method instead of passing a string constant
2012-06-01 17:57:25 -07:00
Travis Wellman 493e2423f6 begin removing MultiReporter
* MultiReporter.java
    deprecated
* ToePool, ToeThread, WorkQueue, WorkQueueFrontier, PrecedenceProvider, CrawlURI, FetchStats.java
    remove MultiReporter#getReports() and reportTo(String, PrintWriter)
    these are not used as MultiReporter specifies, and MultiReporter is going away anyhow
2012-06-01 17:14:41 -07:00
Noah Levitt 1015c5fa00 Fix HER-1973 Regression in queue budget overrides
* WorkQueueFrontier.java
    findEligibleURI() - apply sheet overrides before refreshing queue budgets
2012-05-15 17:35:49 -07:00
Gordon Mohr e4f12bb20f Merge pull request #6 from ato/codemirror
HER-2001: Use the CodeMirror editor for crawl config and script console
    (Thanks Alex!)
2012-04-25 15:04:10 -07:00
Noah Levitt 42e759c01d List correct defaults for BdbModule cacheSize and cachePercentage with an explanatory comment. 2012-04-25 11:10:56 -07:00
Noah Levitt da28258f90 Option to set bdb cache size as an absolute number instead of a percentage (part of HER-1948) 2012-04-19 18:16:53 -07:00
Noah Levitt 4a64c6ba59 Revert "Altered FetchStats to also keep track of the number of URIs that have"
This reverts commit 4c1740ce1e.
2012-04-18 15:53:49 -07:00
Noah Levitt 9551c774af Merge branch 'master' into http-status-deciderule 2012-04-18 15:47:24 -07:00
Alex Osborne f19a72efaf HER-2001: Use the CodeMirror editor for crawl config and script console
This enables syntax-highlighting, line-numbering and auto-indent for the crawl
configuration editor and the script console (for BeanShell, JavaScript and
Groovy).

* BaseResource.java
    add a more generic getStaticRef() in addition to getStylesheetRef()
* EditRepresentation.java
    wrap textarea with CodeMirror in purexml mode.
    disable the "save changes" button until config is edited.
    dynamically resize CodeMirror edit element to fit window height.
* ScriptResource.java
    wrap textarea with CodeMirror.
    switch CodeMirror language modes depending on selected script engine.
* resources/.../restlet/codemirror/
    include just the files from codemirror-2.23.zip necessary for the above
    plus the CodeMirror LICENSE file (MIT)
    plus a README linking to the codemirror website
2012-03-31 18:35:06 +11:00
Noah Levitt 9626b8454a * EditRepresentation.java
make edit page (for cxml mainly) prettier with big textarea (see Kenji's comment on HER-1993 dated 2012-03-27 18:03)
2012-03-30 11:44:28 -07:00
Noah Levitt ea42c0c653 * EngineResource.java
acceptRepresentation() - "I'm sure" parameter is now "im_sure"
2012-03-19 17:30:18 -07:00
Noah Levitt f7505e6673 More HTML cleanup. Html5 doctype declaration, get rid of <br> and
xml-style "/>" tag endings, other cleanup to make it validate.
2012-03-15 18:47:18 -07:00
Noah Levitt 7cde3b4f17 Merge branch 'master' of github.com:internetarchive/heritrix3 into html-cleanup 2012-03-15 15:59:10 -07:00
Kenji Nagahashi 64f24d4c66 HER-1993: In JobResource, write error message instead of throwing RuntimeException when it failed to read
job log or crawl log.
2012-03-09 16:48:11 -08:00