lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-23 22:26:01 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,156 Commits 38 Branches 33 Tags
b84d443f00a75d68fa022c61ba5b8a106bd6344a
Commit Graph
839 Commits
Author SHA1 Message Date
Noah Levitt 0eaf1c1d3e missed commit 2012-07-02 17:15:18 -07:00
Noah Levitt 2e5af085bc Merge branch 'fetchhttp-hc' of ssh://desktop-nlevitt.sf.archive.org/~nlevitt/workspace/heritrix3 into fetchhttp-hc 
Conflicts:
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP2.java
	modules/src/test/java/org/archive/modules/fetcher/FetchHTTPTestBase.java
 2012-07-02 17:07:32 -07:00
Noah Levitt 22b327bdc1 progress on basic auth 2012-07-02 15:38:52 -07:00
Noah Levitt 1f700e5cac poking around at basic auth, nothing close to working yet 2012-07-02 10:31:32 -07:00
Noah Levitt 29abbecdf3 make data key "httpResponseHeaders" a constant 2012-07-02 10:30:27 -07:00
Noah Levitt 8abe871af6 configure logging 2012-07-02 10:29:19 -07:00
Noah Levitt 4790be501c fix up saving of http response headers in CrawlURI 2012-06-29 22:00:35 -07:00
Noah Levitt 5140a3f5bb basic-auth test works! 2012-06-29 18:34:17 -07:00
Noah Levitt 373abbbe81 latest jetty 6.x; starting on basic-auth test 2012-06-29 17:19:46 -07:00
Noah Levitt 09d1839e6c test sending cookies and ignoring cookies 2012-06-29 15:42:31 -07:00
Noah Levitt 69d7b0a0cd make jetty server not require anything external; also make it work with maven 2012-06-29 13:43:54 -07:00
Noah Levitt 72303cc103 make jetty server not require anything external; also make it work with maven 2012-06-29 12:33:20 -07:00
Noah Levitt 93958abfcd these help classes for fetchhttp2 need to be in modules/ since they use stuff from modules/ 2012-06-29 11:19:56 -07:00
Noah Levitt c7ba0675cf more license headers 2012-06-29 07:40:53 -07:00
Noah Levitt 9ef01de7b4 twiddles 2012-06-29 07:36:36 -07:00
Noah Levitt b6cc860438 refactor to get rid of redundancy 2012-06-29 07:24:09 -07:00
Noah Levitt 98a4be5eb1 fix user-agent test, add more tests 2012-06-29 07:05:44 -07:00
Noah Levitt 1a79348cf7 test a few more things 2012-06-29 02:55:26 -07:00
Noah Levitt 1f315a7cc7 unit tests for FetchHTTP and FetchHTTP2 2012-06-29 02:04:39 -07:00
Noah Levitt 2653583cdd unit tests for FetchHTTP and FetchHTTP2 2012-06-29 01:59:58 -07:00
Noah Levitt 63c6ca59c2 Merge branch 'master' into fetchhttp-hc 2012-06-25 16:42:34 -07:00
Travis Wellman 6626b44286 generics warnings fixes 
This commit does not change any functionality, just
alters generics to reduce warnings in Eclipse.
 2012-06-21 18:23:06 -07:00
Noah Levitt 2b0141fd0d bdb cookie store for FetchHTTP2 2012-06-20 18:49:57 -07:00
Noah Levitt 49c00c0387 do not rely on ToeThread for the ServerCache, pass it to where it's needed instead 2012-06-19 13:54:05 -07:00
Noah Levitt 56f0c92690 resolve hostnames from heritrix server cache (the bad old way) 2012-06-19 13:32:34 -07:00
Noah Levitt 1d0beedf2b use http 1.0 by default, other request configuration stuff 2012-06-19 12:09:46 -07:00
Noah Levitt 233b72cb34 get charset, encodings from headers 2012-06-19 11:28:01 -07:00
Noah Levitt de152eaa9e browser-compatibility mode for cookie matching; that's what old FetchHTTP does, and also avoids sending weird Cookie2:$Version1 2012-06-18 17:49:09 -07:00
Noah Levitt 3c7bd8afd6 no keep-alive 2012-06-18 17:06:46 -07:00
Noah Levitt a779e46ac7 set user-agent 2012-06-18 15:55:07 -07:00
Noah Levitt 7422a7b992 move httpclient helper classes to commons/ 2012-06-18 15:43:52 -07:00
Noah Levitt 7f0efbdac8 do recording through SessionOutputBuffer implementation like we have for SessionInputBuffer; refactor helper classes into their own file RecordingHttpClient.java 2012-06-18 15:42:50 -07:00
Noah Levitt 0d172be888 not using HcInputWrapper anymore 2012-06-18 12:31:52 -07:00
Noah Levitt 316391786e license headers 2012-06-17 23:23:39 -07:00
Noah Levitt 6c56ce41a0 different deeper way to insert our Recorder, letting us correctly note end of http headers. is there a problem with not buffering at this level? 2012-06-17 15:48:19 -07:00
Noah Levitt b27b3f67e5 continuing to mess with httpcomponents 2012-06-15 18:29:21 -07:00
Noah Levitt 9a5570c6f1 continuing to mess with httpcomponents 2012-06-15 16:34:55 -07:00
Noah Levitt 3a0f50d031 FetchHTTP2 - record, archive request as well 2012-06-14 19:01:15 -07:00
Noah Levitt 5043dcc35b FetchHTTP2, based on httpcomponents, is working in a rudimentary way. This commit also gets rid of most egregious commons-httpclient-specific code in CrawlURI 2012-06-14 18:34:43 -07:00
Noah Levitt 793d733f1e continuing to mess with httpcomponents 2012-06-14 14:00:16 -07:00
Noah Levitt e5668726a1 starting to mess with httpcomponents 2012-06-13 14:11:11 -07:00
Noah Levitt e1f5bc5319 HER-1987 record charset used for link extraction in warc metadata record 
* WARCWriterProcessor.java
    writeMetadata() - write metadata entry charsetForLinkExtraction
 2012-06-06 12:02:41 -07:00
Noah Levitt 7eabc7ef73 HER-1987 record charset used for link extraction in warc metadata record 
* WARCWriterProcessor.java
    writeMetadata() - if extractors have left annotations about the encoding used to interpret content, include them in the metadata (HER-1987)
 2012-06-06 11:40:58 -07:00
Noah Levitt 35378df2fa Merge branch 'master' into no-package-scope 2012-06-05 17:58:40 -07:00
Noah Levitt 3dce1e909d Change scope of most/all default-scope i.e. package-scoped class members to protected, to make heritrix more friendly to third-party extensions. (I don't think there was ever an intent to limit the scope of any of these to package-only.) 2012-06-05 17:55:41 -07:00
Travis Wellman 493e2423f6 begin removing MultiReporter 
* MultiReporter.java
    deprecated
* ToePool, ToeThread, WorkQueue, WorkQueueFrontier, PrecedenceProvider, CrawlURI, FetchStats.java
    remove MultiReporter#getReports() and reportTo(String, PrintWriter)
    these are not used as MultiReporter specifies, and MultiReporter is going away anyhow
 2012-06-01 17:14:41 -07:00
Noah Levitt 3bebf10674 * ScriptedDecideRule.java 
getEngine() - tiny cosmetic fixups
 2012-05-31 16:29:33 -07:00
Travis Wellman 392a4b3fd0 correct synchronization and threadlocal logic 
Previously the whole method was synchronized even if threadlocals
were used (which are thread safe). And previously the sharedEngine
was set and returned only when it /was not/ needed ie when
getIsolateThreads() returned true.

Now getEngine does what it says it does.
 2012-05-31 16:09:42 -07:00
Noah Levitt 809981ef02 * PersistProcessor.java 
populatePersistEnvFromLog() - handle exception deserializing line
 2012-05-10 19:00:27 -07:00
Noah Levitt 9fffa8fc53 **/pom.xml 
bump version to 3.1.2-SNAPSHOT
 2012-05-01 18:17:33 -07:00