lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-25 16:06:33 +00:00
Code Issues Packages Projects Releases Wiki Activity
976 Commits 38 Branches 32 Tags
948d641b1460c0d5a1e66cd4c9ac00a12bb9e831
Commit Graph
976 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Noah Levitt 948d641b14 socket/connection timeout 2012-09-06 12:39:44 -07:00
Noah Levitt d11338d620 testFetchTimeout 2012-09-05 19:24:50 -07:00
Noah Levitt fbd25b789a implement shouldFetchBodyRule 2012-09-05 14:19:17 -07:00
Noah Levitt d703fce31a add missing license header 2012-09-05 14:08:26 -07:00
Noah Levitt b19994e3bb Merge branch 'master' into fetchhttp-hc 2012-09-05 12:08:48 -07:00
Noah Levitt 1a07c6b741 Make unit tests pass 
* ContentExtractorTestBase.java
    assertNoSideEffects() - check curi.getAnnotations.isEmpty() instead of equality with Collections.EMPTY_LIST
 2012-09-05 11:19:18 -07:00
Noah Levitt e711ff9506 * JobRelatedResource.java 
fix code formatting from 01be6d4b
 2012-09-04 18:03:38 -07:00
Noah Levitt c433c78e90 Merge pull request #17 from adam-miller/HER-2018 
Fix for HER-2018: XML representation for /engine/job/<jobName>/beans returns incorrect url for named beans
 2012-09-04 17:59:43 -07:00
Noah Levitt cd0c169a4d Merge branch 'master' into fetchhttp-hc 2012-09-04 17:54:03 -07:00
Noah Levitt b704f4eeb7 Fix HER-2021 usedBaseForVia annotation repeated many times 
* CrawlURI.java
    getAnnotations() - use LinkedHashSet data structure for new annotations object to prevent duplicates
* Link.java
    addRelativeToVia() - only log use of base for via the first time, before the annotation has been added
 2012-09-04 17:49:01 -07:00
Noah Levitt baaf4f8092 testShouldFetchBodyRule 2012-09-04 13:25:51 -07:00
Noah Levitt ca606588ae test sendIfNoneMatch (etag) 2012-09-02 15:25:56 -07:00
Noah Levitt c03273230b sendIfModifiedSince 2012-09-02 13:00:01 -07:00
Noah Levitt f3963d5f89 convenience method CrawlURI.getFetchHistory() 2012-09-02 12:25:19 -07:00
Noah Levitt 1fe4781cf5 axLengthBytes, sendRange 2012-09-02 01:37:00 -07:00
Noah Levitt 2d8687e55c maxFetchKBSec 2012-09-01 23:05:50 -07:00
Noah Levitt 5cb3b914e8 Merge branch 'master' into fetchhttp-hc 2012-09-01 22:11:59 -07:00
Noah Levitt d0882a18e6 * AbstractPersistProcessor.java 
shouldStore() - only save persist record if http, https or ftp
 2012-09-01 22:09:30 -07:00
Adam Miller 01be6d4b18 Fix for HER-2018: XML representation for /engine/job/<jobName>/beans returns incorrect url for named beans 
* JobRelatedResource.java
    Added encoding to URL generation in makePresentableMapFor().
    Added check to prevent URL generation from overwriting the job name portion of the path in addPresentableNestedNames()
 2012-08-31 15:07:31 -07:00
Noah Levitt 61cbd54059 Merge pull request #16 from adam-miller/HER-2017 
Fix for HER-2017 XML representation of beans uses root node of type "script"
 2012-08-30 16:04:04 -07:00
Adam Miller 52e70a10ef Fix for HER-2017 XML representation for /engine/job/<jobName>/beans uses root node of type "script" instead of "beans" 
* BeanBrowseResource.java
	replaced "script" string in root node with "beans"
 2012-08-30 15:35:27 -07:00
Noah Levitt 3f2295353a test dependencies belong in engine/ now too 2012-08-30 12:16:25 -07:00
Noah Levitt cdd95cbb1f move FetchHTTP*Test* to engine/ subproject because testFormAuth() makes use of PreconditionEnforcer, part of engine/, and we want to keep that test 2012-08-30 09:55:20 -07:00
Noah Levitt f502292eef commons-codec was already pulled in from elsewhere 2012-08-29 19:06:37 -07:00
Noah Levitt 731d382ae0 fetchhttp2 support for proxy auth, somewhat elaborately done, sharing principles and code with http "target" auth 2012-08-29 17:41:50 -07:00
Noah Levitt 0e02aa3523 check for Proxy-Connection header, but don't worry about the value for now (does it matter?) 2012-08-29 11:12:49 -07:00
Noah Levitt b112fdd9f7 http proxy support for FetchHTTP2; also, avoid credentials contamination across requests and across threads 2012-08-29 10:57:28 -07:00
Noah Levitt c17f74a7b0 add check that our little proxy server really handled a request 2012-08-29 10:06:20 -07:00
Noah Levitt a2d65044f0 progress on http proxy test 2012-08-28 18:25:53 -07:00
Noah Levitt 6c1b762fe4 remove unnecessary finally blocks unsetting test settings, since each test gets a fresh fetcher 2012-08-28 16:57:45 -07:00
Noah Levitt aa33887e64 Merge remote-tracking branch 'origin/master' into fetchhttp-hc 2012-08-28 16:07:35 -07:00
Noah Levitt 0d65d7e446 Fix for HER-2019 fetchHTTP httpBindAddress setting doesn't take effect 
* FetchHTTP.java
    use keyed properties for httpBindAddress setting, as expected in configureBindAddress(CrawlURI, HostConfiguration)
 2012-08-28 16:03:17 -07:00
Noah Levitt 2be26573ba httpBindAddress 2012-08-28 16:00:14 -07:00
Noah Levitt d820fe946e Merge remote-tracking branch 'travisfw/nlevitt-fetchhttp-hc' into fetchhttp-hc 2012-08-27 18:51:37 -07:00
Travis Wellman 60954dbbc0 set accept compression back to false (shared) 2012-08-27 18:44:55 -07:00
Travis Wellman bc71578c68 testing setAcceptCompression 2012-08-27 18:41:49 -07:00
Noah Levitt 9f433cbde4 use eclipse junit thing 2012-08-27 17:09:49 -07:00
Noah Levitt 6eb1e4cefd Merge branch 'master' into fetchhttp-hc 
Conflicts:
	modules/src/main/java/org/archive/modules/CoreAttributeConstants.java
	modules/src/main/java/org/archive/modules/CrawlURI.java
	modules/src/main/java/org/archive/modules/credential/Credential.java
	modules/src/main/java/org/archive/modules/credential/HtmlFormCredential.java
	modules/src/main/java/org/archive/modules/credential/HttpAuthenticationCredential.java
	modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java
	modules/src/main/java/org/archive/modules/net/CrawlServer.java
 2012-08-24 14:28:33 -07:00
Noah Levitt a7a2b3cf7e Make speculative link extractor ignore urls with two consecutive slashes (not preceded by a colon as in http://). We've seen the double slashes lead to crawler traps. 
* UriUtils.java
    isLikelyFalsePositive() - consider likely false positive if it contains '//' (but not '://')
* UriUtilsTest.java
    testDoubleSlashFalsePositives()
 2012-08-24 14:03:45 -07:00
Noah Levitt 9c00bd009e form auth 2012-08-21 16:40:47 -07:00
Noah Levitt bc30eabe72 disable automatic redirect handling 2012-08-21 15:48:06 -07:00
Noah Levitt 6941842fa8 make http auth work in FetchHTTP2 the new good way, like we did with FetchHTTP 2012-08-21 15:23:31 -07:00
Noah Levitt f072dfc83c * Credential.java, HtmlFormCredential.java, HttpAuthenticationCredential.java 
bump serialVersionUID because of changed method signature on populate()
 2012-08-16 12:28:19 -07:00
Noah Levitt 0144df024e Real fix for HER-2016 FetchHTTP tries every url twice without credentials before sending credentials: Remember WWW-Authenticate headers, aka auth challenges, with CrawlURI, and after successful auth, with the CrawlServer. It's necessary to do this to know what Authorization header to send without getting another 401 first. 
* CoreAttributeConstants.java
    new constant A_HTTP_AUTH_CHALLENGES key for CrawlURI.getData() for http auth challenges
* CrawlURI.java
    add A_HTTP_AUTH_CHALLENGES to keys to remember across URI processing; convenience getter and setter for auth challenges
* Credential.java, HtmlFormCredential.java, HttpAuthenticationCredential.java
    add httpAuthChallenges argument to populate()
* HttpAuthenticationCredential.java
    populate() - process httpAuthChallenges to set up auth state of http client to send the right auth header
* CrawlServer.java
    cache httpAuthChallenges
* FetchHTTP.java
    getAuthScheme() - cache auth challenges with CrawlURI
    configureHttp() - do not set authentication preemptive - this is handled by the configuration done in HttpAuthenticationCredential.populate()
    promoteCredentials() - cache auth challenges with the CrawlServer
 2012-08-16 12:17:01 -07:00
Noah Levitt 201e941b69 tell CrawlURI to remember auth challenges across URI processings; couple other tweaks 2012-08-16 12:00:17 -07:00
Noah Levitt 883ca25925 remember www-authenticate headers to make http auth work as intended, without lots of extra 401s 2012-08-15 22:11:34 -07:00
Noah Levitt 5272b0b8c8 make form auth test work and test stuff for real 2012-08-15 13:53:27 -07:00
Noah Levitt eb3419aa59 deprecate method (GET/POST) in HtmlFormCredential, since PreconditionEnforcer always sets fetch type of login urls to POST (who sends password in a GET request anyway?) 2012-08-15 13:53:07 -07:00
Noah Levitt fe7eb22b2a oops... was swallowing problems so all junit tests passed! throw that Throwable 2012-08-15 13:51:22 -07:00
Noah Levitt 5bc1fb0b6d import doesn't belong 2012-08-14 16:12:22 -07:00