lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-23 06:05:46 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,156 Commits 38 Branches 33 Tags
b84d443f00a75d68fa022c61ba5b8a106bd6344a
Commit Graph
700 Commits
Author SHA1 Message Date
gojomo eb0760a37e refactor for runCandidatesChain; also run outCandidates 2013-03-11 20:50:18 -07:00
Adam Miller 0e3a2ec80a CrawlJobModel.java 
Adding check for application context
 2013-03-11 17:07:15 -07:00
Adam Miller b7b7953dc9 EngineResource.java 
Pulling individual values for necessary data from crawl jobs instead of building the full CrawlJobModel
CrawlJobModel.java
	Moving some data from methods into fields so they are exposed in the XML representation
EngineModel.java
	Removing full building of CrawlJobModel. Handling instead in EngineResource
Job.ftl
	Updating fields that relied on CrawlJobModel methods which were moved to fields.
 2013-03-11 17:07:15 -07:00
Adam Miller 03f4f381af JobResource.java 
Removing temporary template loading. Finishing converting XML and HTML generation to use the same data model
CrawlJobModel.java
	Converting to same Map based reports used by XML API
EngineModel.java
	Converting to same Map based reports used by XML API
Engine.ftl
	Creating template for Map based reports
Job.ftl
	Creating template for Map based reports
 2013-03-11 17:07:15 -07:00
Adam Miller ef7fd37a0a BeanBrowseResource.java 
Removing obsolete method: makePresentableMap()
 2013-03-11 17:07:15 -07:00
Adam Miller d26a712211 Fixed bad merge conflict 2013-03-11 17:07:15 -07:00
Adam Miller 60d00e3ef3 Converted data models to be usable for conversion to HTML and XML. Updated Models and freemarker templates to match XML naming 2013-03-11 17:05:18 -07:00
Adam Miller 50e49b6399 Removed commented out code. 2013-03-11 17:05:18 -07:00
Adam Miller 7426c78d73 Removed temporary template loading from file system. 
Removed unused method: getAvailableActions
 2013-03-11 17:05:18 -07:00
Adam Miller d545899fd2 Removed temporary template loading from file system 2013-03-11 17:05:18 -07:00
Adam Miller 929f3cb489 Scripts and Beans converted to freemarker 2013-03-11 17:05:17 -07:00
Adam Miller 783d48b49c Removing old commented out code 2013-03-11 16:58:28 -07:00
Adam Miller a32f00c2b3 Finishing /engine and /engine/job/{job} 2013-03-11 16:58:28 -07:00
Adam Miller e4ad60dda6 Beginning conversion to freemarker templates. /engine updated 2013-03-11 16:58:28 -07:00
Noah Levitt fb758e53e3 Avoid pulling in hadoop-core.jar and its dependencies; add plugin versions to get rid of warnings 2013-02-22 15:43:15 -08:00
Travis Wellman 56e6a0d504 ScriptResource actually uses the BeanLookupBindings 2013-01-24 10:56:16 -08:00
Travis Wellman 1883d81bc8 bindings on top of beans instead of vice versa 2013-01-23 18:27:40 -08:00
Travis Wellman 91aeb6d0bf disable bean lookup by default 
bean lookup may be enabled by setting a variable named beanBindings to
true in the Bindings or in the script.
 2013-01-11 12:09:54 -08:00
Travis Wellman 03914a246c BeanLookupBindings for simpler script access to beans 
The new class BeanLookupBindings allows scripts to skip getBean calls.
Lines like 'beanname = appCtx.getBean("beanname")' can be left out.
Past scripts remain compatible.
This change effects action directory scripts and rest console scripts.
 2013-01-10 18:54:33 -08:00
Noah Levitt 7d3fe554af Merge branch 'master' into hc43 
Conflicts:
	.classpath
	commons/src/main/java/org/apache/commons/httpclient/HttpMethodBase.java
 2013-01-03 18:18:22 -08:00
Noah Levitt 25234898ef Somewhat redundant form auth integration test restored, passes 2012-12-30 22:35:28 -08:00
Noah Levitt 3a65fa25d7 Working on refactoring some stuff into archive-commons 2012-12-18 14:46:52 -08:00
Noah Levitt 4496c5d755 * FetchHTTPTest.java 
move to modules/src/test
    testFormAuth() - remove, there is FormAuthSelfTest.java, and this was the test that would fail in modules/
    testHttpBindAddress() - incorporate ed65493db0 - alternate testing of multiple local binds more likely to work more places
 2012-11-30 17:57:36 -08:00
Noah Levitt a20824aea7 Merge branch 'master' into new-fetchhttp-only 
Conflicts:
	.classpath
	engine/pom.xml
	modules/pom.xml
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java
	modules/src/test/java/org/archive/modules/fetcher/FetchHTTPTest.java
 2012-11-30 17:04:28 -08:00
Noah Levitt 4470ba512e Count url-agnostic content digest duplicates as duplicates in statistics 
* CrawledBytesHistotable.java
    accumulate() - count urls with warcRevisit:uriAgnosticDigest annotation as duplicates
* ContentDigestHistoryTest.java
    testWarcDedupe() - check for presence of warcRevisit:uriAgnosticDigest annotation
 2012-09-27 16:56:14 -07:00
Noah Levitt f9703fb5ef * engine/pom.xml 
remove incorrect, unnecessary dependency
 2012-09-26 14:55:03 -07:00
Noah Levitt b9cbbbd248 Bunch of unit tests for FetchHTTP 
* FetchHTTPTest.java
    a bunch of new tests
* FetchHTTP.java
    make sslTrustLevel work correctly (bug found by test; even with fix, only respects value at startup, no overrides)
* modules/pom.xml, .classpath
    dependencies for testing
* engine/pom.xml
    update jetty to 6.1.26 since we require that for the new tests
 2012-09-26 14:41:49 -07:00
Noah Levitt 7965a27b1d Fix infinite loop bug on server returning no response 
* RecordingSocketInputBuffer.java
    readLine(CharArrayBuffer) - return -1 (not 0) on end of stream
* RecordingHttpClient.java
    setHttpRequestRetryHandler() - never retry (may want to reevaluate later, old fetcher did retry sometimes)
* FetchHTTPTest.java
    testNoResponse()
 2012-09-13 19:10:11 -07:00
Noah Levitt dd7e0c5bd2 * profile-crawler-beans.cxml 
replace defunct commons-httpclient BdbCookieStorage with new httpcomponents BdbCookieStore
 2012-09-13 17:58:47 -07:00
Noah Levitt 77559bf81c **/pom.xml 
bump version number because of api change
 2012-09-13 17:47:26 -07:00
Noah Levitt 9699d6a6ed get rid of AbstractFetchHTTP 2012-09-08 01:17:32 -07:00
Noah Levitt 33df8faaf0 remove LegacyFetchHTTP and its classes 2012-09-08 00:49:30 -07:00
Noah Levitt 71761a72a7 rename FetchHTTP2 to FetchHTTP 2012-09-08 00:28:06 -07:00
Noah Levitt 917986454b rename FetchHTTP to LegacyFetchHTTP 2012-09-08 00:25:14 -07:00
Noah Levitt 24e813c858 test chunked transfer encoding 2012-09-07 13:28:41 -07:00
Noah Levitt 09f0e53119 testHttp11 2012-09-07 11:32:38 -07:00
Noah Levitt 9a0191a84c ssl trust level support for FetchHTTP2! another thing that doesn't even work properly in old FetchHTTP 2012-09-07 02:47:54 -07:00
Noah Levitt 0764096eb0 skip slow tests for old FetchHTTP for now 2012-09-07 02:42:13 -07:00
Noah Levitt f2f0ca59f6 embedded jetty https server 2012-09-06 19:28:07 -07:00
Noah Levitt 948d641b14 socket/connection timeout 2012-09-06 12:39:44 -07:00
Noah Levitt d11338d620 testFetchTimeout 2012-09-05 19:24:50 -07:00
Noah Levitt b19994e3bb Merge branch 'master' into fetchhttp-hc 2012-09-05 12:08:48 -07:00
Noah Levitt e711ff9506 * JobRelatedResource.java 
fix code formatting from 01be6d4b
 2012-09-04 18:03:38 -07:00
Noah Levitt baaf4f8092 testShouldFetchBodyRule 2012-09-04 13:25:51 -07:00
Noah Levitt ca606588ae test sendIfNoneMatch (etag) 2012-09-02 15:25:56 -07:00
Noah Levitt c03273230b sendIfModifiedSince 2012-09-02 13:00:01 -07:00
Noah Levitt 1fe4781cf5 axLengthBytes, sendRange 2012-09-02 01:37:00 -07:00
Noah Levitt 2d8687e55c maxFetchKBSec 2012-09-01 23:05:50 -07:00
Noah Levitt 5cb3b914e8 Merge branch 'master' into fetchhttp-hc 2012-09-01 22:11:59 -07:00
Adam Miller 01be6d4b18 Fix for HER-2018: XML representation for /engine/job/<jobName>/beans returns incorrect url for named beans 
* JobRelatedResource.java
    Added encoding to URL generation in makePresentableMapFor().
    Added check to prevent URL generation from overwriting the job name portion of the path in addPresentableNestedNames()
 2012-08-31 15:07:31 -07:00