lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-26 08:26:53 +00:00
Code Issues Packages Projects Releases Wiki Activity
930 Commits 38 Branches 32 Tags
6941842fa812bb75fb65ffb4c5896f2816fb8024
Commit Graph
930 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Noah Levitt 6941842fa8 make http auth work in FetchHTTP2 the new good way, like we did with FetchHTTP 2012-08-21 15:23:31 -07:00
Noah Levitt 201e941b69 tell CrawlURI to remember auth challenges across URI processings; couple other tweaks 2012-08-16 12:00:17 -07:00
Noah Levitt 883ca25925 remember www-authenticate headers to make http auth work as intended, without lots of extra 401s 2012-08-15 22:11:34 -07:00
Noah Levitt 5272b0b8c8 make form auth test work and test stuff for real 2012-08-15 13:53:27 -07:00
Noah Levitt eb3419aa59 deprecate method (GET/POST) in HtmlFormCredential, since PreconditionEnforcer always sets fetch type of login urls to POST (who sends password in a GET request anyway?) 2012-08-15 13:53:07 -07:00
Noah Levitt fe7eb22b2a oops... was swallowing problems so all junit tests passed! throw that Throwable 2012-08-15 13:51:22 -07:00
Noah Levitt 5bc1fb0b6d import doesn't belong 2012-08-14 16:12:22 -07:00
Noah Levitt 864fea938b working on form auth test 2012-07-27 19:13:19 -07:00
Noah Levitt d71fbb3a1a stick a basic auth 401 in the middle of testDigestAuth to check it doesn't mess with the digest auth we're working on 2012-07-26 18:45:29 -07:00
Noah Levitt 3efe8221cf commons-httpclient, don't try to handle 401s internally 2012-07-26 18:44:30 -07:00
Noah Levitt 4e50c32578 Merge branch 'master' into fetchhttp-hc 
Conflicts:
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java
 2012-07-20 21:36:23 -07:00
Kenji Nagahashi 3078eac4ab test case for scheme-less relative URL 2012-07-19 14:15:22 -07:00
Kenji Nagahashi 8fc44b626c added a test case for catching HER-2013 2012-07-19 14:15:22 -07:00
Noah Levitt 6f4053cab7 HER-2016 FetchHTTP tries every url twice without credentials before sending credentials 
* FetchHTTP.java
    configureHttp() - enable preemptive authentication parameter
    innerProcess() - do not call method.setDoAuthentication() (it's for automatic 401 handling which is not what we want)
 2012-07-19 13:15:27 -07:00
Noah Levitt 9073d12f96 Revert "HER-2015 credentials cached to server never used" 
This reverts commit 37a8504fda.
 2012-07-19 12:36:05 -07:00
Noah Levitt 93c3a706dd Merge branch 'master' into fetchhttp-hc 
Conflicts:
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java
 2012-07-16 20:05:05 -07:00
Noah Levitt 37a8504fda HER-2015 credentials cached to server never used 
* FetchHTTP.java
    populateCredentials() - return true when credentials populated from server cache
 2012-07-16 20:02:56 -07:00
Noah Levitt ea7e7ed304 check that successful basic/digest auth credentials are cached with server and volunteered on subsequent requests 2012-07-16 19:33:11 -07:00
Noah Levitt cb6be2e9ca license header 2012-07-16 18:07:11 -07:00
Noah Levitt fe739e8c2e remove unnecessary dependency on httpcomponents from CrawlURI 2012-07-16 17:45:44 -07:00
Noah Levitt e9b6bb3fd6 Merge branch 'master' into fetchhttp-hc 2012-07-16 17:33:02 -07:00
Noah Levitt eee6c6cd30 * ArchiveRecord.java 
fix indentation only (no substantive changes)
 2012-07-16 09:57:24 -07:00
Noah Levitt c108c4c451 HER-2014 Make ArchiveRecord.getPosition() public instead of protected. 
* ArchiveRecord.java, HeaderedArchiveRecord.java
    getPosition() - change visibility to public
 2012-07-16 09:54:54 -07:00
Noah Levitt 82c983d2b3 Merge branch 'master' into fetchhttp-hc 
Conflicts:
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java

(conflicts are resolved)
 2012-07-15 11:58:41 -07:00
Noah Levitt daac7fb96f more work on http authentication (is it done now?) digest auth test passes 2012-07-15 11:56:30 -07:00
Travis Wellman c5d740a6d4 remove some dead or redundant code 
* ArchiveUtils
    sb.length() was tested twice and int i was never used
* PrefixFinder
    opCount is never used
* BdbMultipleWorkQueues
    tries is never used
 2012-07-11 16:57:13 -07:00
Travis Wellman 1eb5ba704e Merge commit '787f7d' 2012-07-11 13:51:13 -07:00
Travis Wellman d0add8c186 Merge branch 'master' of github.com:internetarchive/heritrix3 2012-07-11 13:50:28 -07:00
Travis Wellman 787f7d9234 If the URI doesn't have a path, don't count segments. 
* TooManyPathSegmentsDecideRule#evaluate(CrawlURI)
    return false if a path can't be obtained from the uri
 2012-07-11 13:40:01 -07:00
Travis Wellman 9505edc9c3 fix for null paths 
* TooManyPathSegmentsDecideRule
    Sometimes the path is null, throwing a NPE when the loop in evaluate(CrawlURI)
    is entered. In such a case, revert to previous behavior.
 2012-07-10 18:01:53 -07:00
Travis Wellman c1a0525107 suppress unused warnings for serialVersionUid 
Toward the goal of having Eclipse warnings be useful to look at, suppressing
unused warnings for serialVersionUid will significantly reduce clutter.
 2012-07-10 14:12:33 -07:00
Travis Wellman 4ca6e9cc99 revert a change that may break bdb history 2012-07-03 17:38:43 -07:00
Travis Wellman 73640e0cd8 Merge pull request #13 from travisfw/master 
have TooManyPathSegmentsDecideRule count path segments only
 2012-07-03 13:31:23 -07:00
Travis Wellman 4b047dba64 count path segments by looking at the path 
* TooManyPathSegmentsDecideRule
    evaluate(CrawlURI) previously counted slashes in the full URI.
    Now it only counts slashes in the path of the URI, meaning a URI
    with slashes in the query string will not be counted as having more
    path segments than it has in its path.
 2012-07-03 13:13:50 -07:00
Noah Levitt 3f58893d1e disassociate Credential type from commons-httpclient library by moving populate() method into utility class 2012-07-02 17:34:23 -07:00
Noah Levitt 0eaf1c1d3e missed commit 2012-07-02 17:15:18 -07:00
Noah Levitt 2e5af085bc Merge branch 'fetchhttp-hc' of ssh://desktop-nlevitt.sf.archive.org/~nlevitt/workspace/heritrix3 into fetchhttp-hc 
Conflicts:
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP2.java
	modules/src/test/java/org/archive/modules/fetcher/FetchHTTPTestBase.java
 2012-07-02 17:07:32 -07:00
Noah Levitt 22b327bdc1 progress on basic auth 2012-07-02 15:38:52 -07:00
Noah Levitt 1f700e5cac poking around at basic auth, nothing close to working yet 2012-07-02 10:31:32 -07:00
Noah Levitt 29abbecdf3 make data key "httpResponseHeaders" a constant 2012-07-02 10:30:27 -07:00
Noah Levitt 8abe871af6 configure logging 2012-07-02 10:29:19 -07:00
Noah Levitt 4790be501c fix up saving of http response headers in CrawlURI 2012-06-29 22:00:35 -07:00
Noah Levitt 5140a3f5bb basic-auth test works! 2012-06-29 18:34:17 -07:00
Noah Levitt 373abbbe81 latest jetty 6.x; starting on basic-auth test 2012-06-29 17:19:46 -07:00
Noah Levitt 09d1839e6c test sending cookies and ignoring cookies 2012-06-29 15:42:31 -07:00
Noah Levitt 69d7b0a0cd make jetty server not require anything external; also make it work with maven 2012-06-29 13:43:54 -07:00
Noah Levitt 72303cc103 make jetty server not require anything external; also make it work with maven 2012-06-29 12:33:20 -07:00
Noah Levitt 93958abfcd these help classes for fetchhttp2 need to be in modules/ since they use stuff from modules/ 2012-06-29 11:19:56 -07:00
Noah Levitt c7ba0675cf more license headers 2012-06-29 07:40:53 -07:00
Noah Levitt 9ef01de7b4 twiddles 2012-06-29 07:36:36 -07:00