Commit Graph
347 Commits
Author SHA1 Message Date
Noah Levitt 82c983d2b3 Merge branch 'master' into fetchhttp-hc
Conflicts:
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java

(conflicts are resolved)
2012-07-15 11:58:41 -07:00
Noah Levitt daac7fb96f more work on http authentication (is it done now?) digest auth test passes 2012-07-15 11:56:30 -07:00
Travis Wellman 1eb5ba704e Merge commit '787f7d' 2012-07-11 13:51:13 -07:00
Travis Wellman d0add8c186 Merge branch 'master' of github.com:internetarchive/heritrix3 2012-07-11 13:50:28 -07:00
Travis Wellman 787f7d9234 If the URI doesn't have a path, don't count segments.
* TooManyPathSegmentsDecideRule#evaluate(CrawlURI)
    return false if a path can't be obtained from the uri
2012-07-11 13:40:01 -07:00
Travis Wellman 9505edc9c3 fix for null paths
* TooManyPathSegmentsDecideRule
    Sometimes the path is null, throwing a NPE when the loop in evaluate(CrawlURI)
    is entered. In such a case, revert to previous behavior.
2012-07-10 18:01:53 -07:00
Travis Wellman c1a0525107 suppress unused warnings for serialVersionUid
Toward the goal of having Eclipse warnings be useful to look at, suppressing
unused warnings for serialVersionUid will significantly reduce clutter.
2012-07-10 14:12:33 -07:00
Travis Wellman 4b047dba64 count path segments by looking at the path
* TooManyPathSegmentsDecideRule
    evaluate(CrawlURI) previously counted slashes in the full URI.
    Now it only counts slashes in the path of the URI, meaning a URI
    with slashes in the query string will not be counted as having more
    path segments than it has in its path.
2012-07-03 13:13:50 -07:00
Noah Levitt 3f58893d1e disassociate Credential type from commons-httpclient library by moving populate() method into utility class 2012-07-02 17:34:23 -07:00
Noah Levitt 0eaf1c1d3e missed commit 2012-07-02 17:15:18 -07:00
Noah Levitt 2e5af085bc Merge branch 'fetchhttp-hc' of ssh://desktop-nlevitt.sf.archive.org/~nlevitt/workspace/heritrix3 into fetchhttp-hc
Conflicts:
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP2.java
	modules/src/test/java/org/archive/modules/fetcher/FetchHTTPTestBase.java
2012-07-02 17:07:32 -07:00
Noah Levitt 22b327bdc1 progress on basic auth 2012-07-02 15:38:52 -07:00
Noah Levitt 1f700e5cac poking around at basic auth, nothing close to working yet 2012-07-02 10:31:32 -07:00
Noah Levitt 29abbecdf3 make data key "httpResponseHeaders" a constant 2012-07-02 10:30:27 -07:00
Noah Levitt 8abe871af6 configure logging 2012-07-02 10:29:19 -07:00
Noah Levitt 4790be501c fix up saving of http response headers in CrawlURI 2012-06-29 22:00:35 -07:00
Noah Levitt 5140a3f5bb basic-auth test works! 2012-06-29 18:34:17 -07:00
Noah Levitt 373abbbe81 latest jetty 6.x; starting on basic-auth test 2012-06-29 17:19:46 -07:00
Noah Levitt 09d1839e6c test sending cookies and ignoring cookies 2012-06-29 15:42:31 -07:00
Noah Levitt 69d7b0a0cd make jetty server not require anything external; also make it work with maven 2012-06-29 13:43:54 -07:00
Noah Levitt 72303cc103 make jetty server not require anything external; also make it work with maven 2012-06-29 12:33:20 -07:00
Noah Levitt 93958abfcd these help classes for fetchhttp2 need to be in modules/ since they use stuff from modules/ 2012-06-29 11:19:56 -07:00
Noah Levitt c7ba0675cf more license headers 2012-06-29 07:40:53 -07:00
Noah Levitt 9ef01de7b4 twiddles 2012-06-29 07:36:36 -07:00
Noah Levitt b6cc860438 refactor to get rid of redundancy 2012-06-29 07:24:09 -07:00
Noah Levitt 98a4be5eb1 fix user-agent test, add more tests 2012-06-29 07:05:44 -07:00
Noah Levitt 1a79348cf7 test a few more things 2012-06-29 02:55:26 -07:00
Noah Levitt 1f315a7cc7 unit tests for FetchHTTP and FetchHTTP2 2012-06-29 02:04:39 -07:00
Noah Levitt 2653583cdd unit tests for FetchHTTP and FetchHTTP2 2012-06-29 01:59:58 -07:00
Noah Levitt 63c6ca59c2 Merge branch 'master' into fetchhttp-hc 2012-06-25 16:42:34 -07:00
Travis Wellman 6626b44286 generics warnings fixes
This commit does not change any functionality, just
alters generics to reduce warnings in Eclipse.
2012-06-21 18:23:06 -07:00
Noah Levitt 2b0141fd0d bdb cookie store for FetchHTTP2 2012-06-20 18:49:57 -07:00
Noah Levitt 49c00c0387 do not rely on ToeThread for the ServerCache, pass it to where it's needed instead 2012-06-19 13:54:05 -07:00
Noah Levitt 56f0c92690 resolve hostnames from heritrix server cache (the bad old way) 2012-06-19 13:32:34 -07:00
Noah Levitt 1d0beedf2b use http 1.0 by default, other request configuration stuff 2012-06-19 12:09:46 -07:00
Noah Levitt 233b72cb34 get charset, encodings from headers 2012-06-19 11:28:01 -07:00
Noah Levitt de152eaa9e browser-compatibility mode for cookie matching; that's what old FetchHTTP does, and also avoids sending weird Cookie2:$Version1 2012-06-18 17:49:09 -07:00
Noah Levitt 3c7bd8afd6 no keep-alive 2012-06-18 17:06:46 -07:00
Noah Levitt a779e46ac7 set user-agent 2012-06-18 15:55:07 -07:00
Noah Levitt 7422a7b992 move httpclient helper classes to commons/ 2012-06-18 15:43:52 -07:00
Noah Levitt 7f0efbdac8 do recording through SessionOutputBuffer implementation like we have for SessionInputBuffer; refactor helper classes into their own file RecordingHttpClient.java 2012-06-18 15:42:50 -07:00
Noah Levitt 0d172be888 not using HcInputWrapper anymore 2012-06-18 12:31:52 -07:00
Noah Levitt 316391786e license headers 2012-06-17 23:23:39 -07:00
Noah Levitt 6c56ce41a0 different deeper way to insert our Recorder, letting us correctly note end of http headers. is there a problem with not buffering at this level? 2012-06-17 15:48:19 -07:00
Noah Levitt b27b3f67e5 continuing to mess with httpcomponents 2012-06-15 18:29:21 -07:00
Noah Levitt 9a5570c6f1 continuing to mess with httpcomponents 2012-06-15 16:34:55 -07:00
Noah Levitt 3a0f50d031 FetchHTTP2 - record, archive request as well 2012-06-14 19:01:15 -07:00
Noah Levitt 5043dcc35b FetchHTTP2, based on httpcomponents, is working in a rudimentary way. This commit also gets rid of most egregious commons-httpclient-specific code in CrawlURI 2012-06-14 18:34:43 -07:00
Noah Levitt 793d733f1e continuing to mess with httpcomponents 2012-06-14 14:00:16 -07:00
Noah Levitt e5668726a1 starting to mess with httpcomponents 2012-06-13 14:11:11 -07:00