lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-24 15:36:35 +00:00
Code Issues Packages Projects Releases Wiki Activity
907 Commits 40 Branches 31 Tags
82c983d2b3d96a26becaebab6995b759d6c8a353
Commit Graph
907 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Noah Levitt 82c983d2b3 Merge branch 'master' into fetchhttp-hc 
Conflicts:
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java

(conflicts are resolved)
 2012-07-15 11:58:41 -07:00
Noah Levitt daac7fb96f more work on http authentication (is it done now?) digest auth test passes 2012-07-15 11:56:30 -07:00
Travis Wellman c5d740a6d4 remove some dead or redundant code 
* ArchiveUtils
    sb.length() was tested twice and int i was never used
* PrefixFinder
    opCount is never used
* BdbMultipleWorkQueues
    tries is never used
 2012-07-11 16:57:13 -07:00
Travis Wellman 1eb5ba704e Merge commit '787f7d' 2012-07-11 13:51:13 -07:00
Travis Wellman d0add8c186 Merge branch 'master' of github.com:internetarchive/heritrix3 2012-07-11 13:50:28 -07:00
Travis Wellman 787f7d9234 If the URI doesn't have a path, don't count segments. 
* TooManyPathSegmentsDecideRule#evaluate(CrawlURI)
    return false if a path can't be obtained from the uri
 2012-07-11 13:40:01 -07:00
Travis Wellman 9505edc9c3 fix for null paths 
* TooManyPathSegmentsDecideRule
    Sometimes the path is null, throwing a NPE when the loop in evaluate(CrawlURI)
    is entered. In such a case, revert to previous behavior.
 2012-07-10 18:01:53 -07:00
Travis Wellman c1a0525107 suppress unused warnings for serialVersionUid 
Toward the goal of having Eclipse warnings be useful to look at, suppressing
unused warnings for serialVersionUid will significantly reduce clutter.
 2012-07-10 14:12:33 -07:00
Travis Wellman 4ca6e9cc99 revert a change that may break bdb history 2012-07-03 17:38:43 -07:00
Travis Wellman 73640e0cd8 Merge pull request #13 from travisfw/master 
have TooManyPathSegmentsDecideRule count path segments only
 2012-07-03 13:31:23 -07:00
Travis Wellman 4b047dba64 count path segments by looking at the path 
* TooManyPathSegmentsDecideRule
    evaluate(CrawlURI) previously counted slashes in the full URI.
    Now it only counts slashes in the path of the URI, meaning a URI
    with slashes in the query string will not be counted as having more
    path segments than it has in its path.
 2012-07-03 13:13:50 -07:00
Noah Levitt 3f58893d1e disassociate Credential type from commons-httpclient library by moving populate() method into utility class 2012-07-02 17:34:23 -07:00
Noah Levitt 0eaf1c1d3e missed commit 2012-07-02 17:15:18 -07:00
Noah Levitt 2e5af085bc Merge branch 'fetchhttp-hc' of ssh://desktop-nlevitt.sf.archive.org/~nlevitt/workspace/heritrix3 into fetchhttp-hc 
Conflicts:
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP2.java
	modules/src/test/java/org/archive/modules/fetcher/FetchHTTPTestBase.java
 2012-07-02 17:07:32 -07:00
Noah Levitt 22b327bdc1 progress on basic auth 2012-07-02 15:38:52 -07:00
Noah Levitt 1f700e5cac poking around at basic auth, nothing close to working yet 2012-07-02 10:31:32 -07:00
Noah Levitt 29abbecdf3 make data key "httpResponseHeaders" a constant 2012-07-02 10:30:27 -07:00
Noah Levitt 8abe871af6 configure logging 2012-07-02 10:29:19 -07:00
Noah Levitt 4790be501c fix up saving of http response headers in CrawlURI 2012-06-29 22:00:35 -07:00
Noah Levitt 5140a3f5bb basic-auth test works! 2012-06-29 18:34:17 -07:00
Noah Levitt 373abbbe81 latest jetty 6.x; starting on basic-auth test 2012-06-29 17:19:46 -07:00
Noah Levitt 09d1839e6c test sending cookies and ignoring cookies 2012-06-29 15:42:31 -07:00
Noah Levitt 69d7b0a0cd make jetty server not require anything external; also make it work with maven 2012-06-29 13:43:54 -07:00
Noah Levitt 72303cc103 make jetty server not require anything external; also make it work with maven 2012-06-29 12:33:20 -07:00
Noah Levitt 93958abfcd these help classes for fetchhttp2 need to be in modules/ since they use stuff from modules/ 2012-06-29 11:19:56 -07:00
Noah Levitt c7ba0675cf more license headers 2012-06-29 07:40:53 -07:00
Noah Levitt 9ef01de7b4 twiddles 2012-06-29 07:36:36 -07:00
Noah Levitt b6cc860438 refactor to get rid of redundancy 2012-06-29 07:24:09 -07:00
Noah Levitt 98a4be5eb1 fix user-agent test, add more tests 2012-06-29 07:05:44 -07:00
Noah Levitt 1a79348cf7 test a few more things 2012-06-29 02:55:26 -07:00
Noah Levitt 1a1d1b008c allow FetchHTTP2 infrastructure to fall back on default java hostname resolution 2012-06-29 02:40:34 -07:00
Noah Levitt 1f315a7cc7 unit tests for FetchHTTP and FetchHTTP2 2012-06-29 02:04:39 -07:00
Noah Levitt 2653583cdd unit tests for FetchHTTP and FetchHTTP2 2012-06-29 01:59:58 -07:00
Noah Levitt 58d827c85b Merge branch 'master' into fetchhttp-hc 2012-06-28 18:08:39 -07:00
Noah Levitt 6225f6fa0b Fix HER-2013 scheme-less relative URI not resolved correctly 
* LaxURI.java
    parseUriReference() - fix logic testing if uri string is a path to respect scheme-less uris even if they contain colons
* UURIFactoryTest.java
    add test for scheme-less relative uri (with colons)
 2012-06-28 16:49:50 -07:00
Noah Levitt 63c6ca59c2 Merge branch 'master' into fetchhttp-hc 2012-06-25 16:42:34 -07:00
Travis Wellman c40c24e97d Merge pull request #12 from travisfw/generics-fixes 
generics warnings fixes
 2012-06-25 11:37:18 -07:00
Travis Wellman 6626b44286 generics warnings fixes 
This commit does not change any functionality, just
alters generics to reduce warnings in Eclipse.
 2012-06-21 18:23:06 -07:00
Noah Levitt 2b0141fd0d bdb cookie store for FetchHTTP2 2012-06-20 18:49:57 -07:00
Noah Levitt 49c00c0387 do not rely on ToeThread for the ServerCache, pass it to where it's needed instead 2012-06-19 13:54:05 -07:00
Noah Levitt 56f0c92690 resolve hostnames from heritrix server cache (the bad old way) 2012-06-19 13:32:34 -07:00
Noah Levitt 1d0beedf2b use http 1.0 by default, other request configuration stuff 2012-06-19 12:09:46 -07:00
Noah Levitt 233b72cb34 get charset, encodings from headers 2012-06-19 11:28:01 -07:00
Noah Levitt de152eaa9e browser-compatibility mode for cookie matching; that's what old FetchHTTP does, and also avoids sending weird Cookie2:$Version1 2012-06-18 17:49:09 -07:00
Noah Levitt 3c7bd8afd6 no keep-alive 2012-06-18 17:06:46 -07:00
Noah Levitt a779e46ac7 set user-agent 2012-06-18 15:55:07 -07:00
Noah Levitt 7422a7b992 move httpclient helper classes to commons/ 2012-06-18 15:43:52 -07:00
Noah Levitt 7f0efbdac8 do recording through SessionOutputBuffer implementation like we have for SessionInputBuffer; refactor helper classes into their own file RecordingHttpClient.java 2012-06-18 15:42:50 -07:00
Noah Levitt 0d172be888 not using HcInputWrapper anymore 2012-06-18 12:31:52 -07:00
Noah Levitt 316391786e license headers 2012-06-17 23:23:39 -07:00