lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-26 07:35:39 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,156 Commits 38 Branches 33 Tags
b84d443f00a75d68fa022c61ba5b8a106bd6344a
Commit Graph
839 Commits
Author SHA1 Message Date
Noah Levitt c03273230b sendIfModifiedSince 2012-09-02 13:00:01 -07:00
Noah Levitt f3963d5f89 convenience method CrawlURI.getFetchHistory() 2012-09-02 12:25:19 -07:00
Noah Levitt 1fe4781cf5 axLengthBytes, sendRange 2012-09-02 01:37:00 -07:00
Noah Levitt 2d8687e55c maxFetchKBSec 2012-09-01 23:05:50 -07:00
Noah Levitt 5cb3b914e8 Merge branch 'master' into fetchhttp-hc 2012-09-01 22:11:59 -07:00
Noah Levitt d0882a18e6 * AbstractPersistProcessor.java 
shouldStore() - only save persist record if http, https or ftp
 2012-09-01 22:09:30 -07:00
Noah Levitt 3f2295353a test dependencies belong in engine/ now too 2012-08-30 12:16:25 -07:00
Noah Levitt cdd95cbb1f move FetchHTTP*Test* to engine/ subproject because testFormAuth() makes use of PreconditionEnforcer, part of engine/, and we want to keep that test 2012-08-30 09:55:20 -07:00
Noah Levitt f502292eef commons-codec was already pulled in from elsewhere 2012-08-29 19:06:37 -07:00
Noah Levitt 731d382ae0 fetchhttp2 support for proxy auth, somewhat elaborately done, sharing principles and code with http "target" auth 2012-08-29 17:41:50 -07:00
Noah Levitt 0e02aa3523 check for Proxy-Connection header, but don't worry about the value for now (does it matter?) 2012-08-29 11:12:49 -07:00
Noah Levitt b112fdd9f7 http proxy support for FetchHTTP2; also, avoid credentials contamination across requests and across threads 2012-08-29 10:57:28 -07:00
Noah Levitt c17f74a7b0 add check that our little proxy server really handled a request 2012-08-29 10:06:20 -07:00
Noah Levitt a2d65044f0 progress on http proxy test 2012-08-28 18:25:53 -07:00
Noah Levitt 6c1b762fe4 remove unnecessary finally blocks unsetting test settings, since each test gets a fresh fetcher 2012-08-28 16:57:45 -07:00
Noah Levitt 0d65d7e446 Fix for HER-2019 fetchHTTP httpBindAddress setting doesn't take effect 
* FetchHTTP.java
    use keyed properties for httpBindAddress setting, as expected in configureBindAddress(CrawlURI, HostConfiguration)
 2012-08-28 16:03:17 -07:00
Noah Levitt 2be26573ba httpBindAddress 2012-08-28 16:00:14 -07:00
Travis Wellman 60954dbbc0 set accept compression back to false (shared) 2012-08-27 18:44:55 -07:00
Travis Wellman bc71578c68 testing setAcceptCompression 2012-08-27 18:41:49 -07:00
Noah Levitt 6eb1e4cefd Merge branch 'master' into fetchhttp-hc 
Conflicts:
	modules/src/main/java/org/archive/modules/CoreAttributeConstants.java
	modules/src/main/java/org/archive/modules/CrawlURI.java
	modules/src/main/java/org/archive/modules/credential/Credential.java
	modules/src/main/java/org/archive/modules/credential/HtmlFormCredential.java
	modules/src/main/java/org/archive/modules/credential/HttpAuthenticationCredential.java
	modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java
	modules/src/main/java/org/archive/modules/net/CrawlServer.java
 2012-08-24 14:28:33 -07:00
Noah Levitt 9c00bd009e form auth 2012-08-21 16:40:47 -07:00
Noah Levitt bc30eabe72 disable automatic redirect handling 2012-08-21 15:48:06 -07:00
Noah Levitt 6941842fa8 make http auth work in FetchHTTP2 the new good way, like we did with FetchHTTP 2012-08-21 15:23:31 -07:00
Noah Levitt f072dfc83c * Credential.java, HtmlFormCredential.java, HttpAuthenticationCredential.java 
bump serialVersionUID because of changed method signature on populate()
 2012-08-16 12:28:19 -07:00
Noah Levitt 0144df024e Real fix for HER-2016 FetchHTTP tries every url twice without credentials before sending credentials: Remember WWW-Authenticate headers, aka auth challenges, with CrawlURI, and after successful auth, with the CrawlServer. It's necessary to do this to know what Authorization header to send without getting another 401 first. 
* CoreAttributeConstants.java
    new constant A_HTTP_AUTH_CHALLENGES key for CrawlURI.getData() for http auth challenges
* CrawlURI.java
    add A_HTTP_AUTH_CHALLENGES to keys to remember across URI processing; convenience getter and setter for auth challenges
* Credential.java, HtmlFormCredential.java, HttpAuthenticationCredential.java
    add httpAuthChallenges argument to populate()
* HttpAuthenticationCredential.java
    populate() - process httpAuthChallenges to set up auth state of http client to send the right auth header
* CrawlServer.java
    cache httpAuthChallenges
* FetchHTTP.java
    getAuthScheme() - cache auth challenges with CrawlURI
    configureHttp() - do not set authentication preemptive - this is handled by the configuration done in HttpAuthenticationCredential.populate()
    promoteCredentials() - cache auth challenges with the CrawlServer
 2012-08-16 12:17:01 -07:00
Noah Levitt 201e941b69 tell CrawlURI to remember auth challenges across URI processings; couple other tweaks 2012-08-16 12:00:17 -07:00
Noah Levitt 883ca25925 remember www-authenticate headers to make http auth work as intended, without lots of extra 401s 2012-08-15 22:11:34 -07:00
Noah Levitt 5272b0b8c8 make form auth test work and test stuff for real 2012-08-15 13:53:27 -07:00
Noah Levitt eb3419aa59 deprecate method (GET/POST) in HtmlFormCredential, since PreconditionEnforcer always sets fetch type of login urls to POST (who sends password in a GET request anyway?) 2012-08-15 13:53:07 -07:00
Kenji Nagahashi 541c2ecfe4 Merge branch 'master' into reuse-htmllinkcontext 2012-07-31 12:25:16 -07:00
Noah Levitt 864fea938b working on form auth test 2012-07-27 19:13:19 -07:00
Noah Levitt d71fbb3a1a stick a basic auth 401 in the middle of testDigestAuth to check it doesn't mess with the digest auth we're working on 2012-07-26 18:45:29 -07:00
Noah Levitt 3efe8221cf commons-httpclient, don't try to handle 401s internally 2012-07-26 18:44:30 -07:00
Noah Levitt 4e50c32578 Merge branch 'master' into fetchhttp-hc 
Conflicts:
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java
 2012-07-20 21:36:23 -07:00
Noah Levitt 6f4053cab7 HER-2016 FetchHTTP tries every url twice without credentials before sending credentials 
* FetchHTTP.java
    configureHttp() - enable preemptive authentication parameter
    innerProcess() - do not call method.setDoAuthentication() (it's for automatic 401 handling which is not what we want)
 2012-07-19 13:15:27 -07:00
Noah Levitt 9073d12f96 Revert "HER-2015 credentials cached to server never used" 
This reverts commit 37a8504fda.
 2012-07-19 12:36:05 -07:00
Noah Levitt 93c3a706dd Merge branch 'master' into fetchhttp-hc 
Conflicts:
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java
 2012-07-16 20:05:05 -07:00
Noah Levitt 37a8504fda HER-2015 credentials cached to server never used 
* FetchHTTP.java
    populateCredentials() - return true when credentials populated from server cache
 2012-07-16 20:02:56 -07:00
Noah Levitt ea7e7ed304 check that successful basic/digest auth credentials are cached with server and volunteered on subsequent requests 2012-07-16 19:33:11 -07:00
Noah Levitt cb6be2e9ca license header 2012-07-16 18:07:11 -07:00
Noah Levitt fe739e8c2e remove unnecessary dependency on httpcomponents from CrawlURI 2012-07-16 17:45:44 -07:00
Noah Levitt 82c983d2b3 Merge branch 'master' into fetchhttp-hc 
Conflicts:
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java

(conflicts are resolved)
 2012-07-15 11:58:41 -07:00
Noah Levitt daac7fb96f more work on http authentication (is it done now?) digest auth test passes 2012-07-15 11:56:30 -07:00
Travis Wellman 1eb5ba704e Merge commit '787f7d' 2012-07-11 13:51:13 -07:00
Travis Wellman d0add8c186 Merge branch 'master' of github.com:internetarchive/heritrix3 2012-07-11 13:50:28 -07:00
Travis Wellman 787f7d9234 If the URI doesn't have a path, don't count segments. 
* TooManyPathSegmentsDecideRule#evaluate(CrawlURI)
    return false if a path can't be obtained from the uri
 2012-07-11 13:40:01 -07:00
Travis Wellman 9505edc9c3 fix for null paths 
* TooManyPathSegmentsDecideRule
    Sometimes the path is null, throwing a NPE when the loop in evaluate(CrawlURI)
    is entered. In such a case, revert to previous behavior.
 2012-07-10 18:01:53 -07:00
Travis Wellman c1a0525107 suppress unused warnings for serialVersionUid 
Toward the goal of having Eclipse warnings be useful to look at, suppressing
unused warnings for serialVersionUid will significantly reduce clutter.
 2012-07-10 14:12:33 -07:00
Travis Wellman 4b047dba64 count path segments by looking at the path 
* TooManyPathSegmentsDecideRule
    evaluate(CrawlURI) previously counted slashes in the full URI.
    Now it only counts slashes in the path of the URI, meaning a URI
    with slashes in the query string will not be counted as having more
    path segments than it has in its path.
 2012-07-03 13:13:50 -07:00
Noah Levitt 3f58893d1e disassociate Credential type from commons-httpclient library by moving populate() method into utility class 2012-07-02 17:34:23 -07:00