Commit Graph
322 Commits
Author SHA1 Message Date
Noah Levitt 98a4be5eb1 fix user-agent test, add more tests 2012-06-29 07:05:44 -07:00
Noah Levitt 1a79348cf7 test a few more things 2012-06-29 02:55:26 -07:00
Noah Levitt 1f315a7cc7 unit tests for FetchHTTP and FetchHTTP2 2012-06-29 02:04:39 -07:00
Noah Levitt 2653583cdd unit tests for FetchHTTP and FetchHTTP2 2012-06-29 01:59:58 -07:00
Noah Levitt 63c6ca59c2 Merge branch 'master' into fetchhttp-hc 2012-06-25 16:42:34 -07:00
Travis Wellman 6626b44286 generics warnings fixes
This commit does not change any functionality, just
alters generics to reduce warnings in Eclipse.
2012-06-21 18:23:06 -07:00
Noah Levitt 2b0141fd0d bdb cookie store for FetchHTTP2 2012-06-20 18:49:57 -07:00
Noah Levitt 49c00c0387 do not rely on ToeThread for the ServerCache, pass it to where it's needed instead 2012-06-19 13:54:05 -07:00
Noah Levitt 56f0c92690 resolve hostnames from heritrix server cache (the bad old way) 2012-06-19 13:32:34 -07:00
Noah Levitt 1d0beedf2b use http 1.0 by default, other request configuration stuff 2012-06-19 12:09:46 -07:00
Noah Levitt 233b72cb34 get charset, encodings from headers 2012-06-19 11:28:01 -07:00
Noah Levitt de152eaa9e browser-compatibility mode for cookie matching; that's what old FetchHTTP does, and also avoids sending weird Cookie2:$Version1 2012-06-18 17:49:09 -07:00
Noah Levitt 3c7bd8afd6 no keep-alive 2012-06-18 17:06:46 -07:00
Noah Levitt a779e46ac7 set user-agent 2012-06-18 15:55:07 -07:00
Noah Levitt 7422a7b992 move httpclient helper classes to commons/ 2012-06-18 15:43:52 -07:00
Noah Levitt 7f0efbdac8 do recording through SessionOutputBuffer implementation like we have for SessionInputBuffer; refactor helper classes into their own file RecordingHttpClient.java 2012-06-18 15:42:50 -07:00
Noah Levitt 0d172be888 not using HcInputWrapper anymore 2012-06-18 12:31:52 -07:00
Noah Levitt 316391786e license headers 2012-06-17 23:23:39 -07:00
Noah Levitt 6c56ce41a0 different deeper way to insert our Recorder, letting us correctly note end of http headers. is there a problem with not buffering at this level? 2012-06-17 15:48:19 -07:00
Noah Levitt b27b3f67e5 continuing to mess with httpcomponents 2012-06-15 18:29:21 -07:00
Noah Levitt 9a5570c6f1 continuing to mess with httpcomponents 2012-06-15 16:34:55 -07:00
Noah Levitt 3a0f50d031 FetchHTTP2 - record, archive request as well 2012-06-14 19:01:15 -07:00
Noah Levitt 5043dcc35b FetchHTTP2, based on httpcomponents, is working in a rudimentary way. This commit also gets rid of most egregious commons-httpclient-specific code in CrawlURI 2012-06-14 18:34:43 -07:00
Noah Levitt 793d733f1e continuing to mess with httpcomponents 2012-06-14 14:00:16 -07:00
Noah Levitt e5668726a1 starting to mess with httpcomponents 2012-06-13 14:11:11 -07:00
Noah Levitt e1f5bc5319 HER-1987 record charset used for link extraction in warc metadata record
* WARCWriterProcessor.java
    writeMetadata() - write metadata entry charsetForLinkExtraction
2012-06-06 12:02:41 -07:00
Noah Levitt 7eabc7ef73 HER-1987 record charset used for link extraction in warc metadata record
* WARCWriterProcessor.java
    writeMetadata() - if extractors have left annotations about the encoding used to interpret content, include them in the metadata (HER-1987)
2012-06-06 11:40:58 -07:00
Noah Levitt 35378df2fa Merge branch 'master' into no-package-scope 2012-06-05 17:58:40 -07:00
Noah Levitt 3dce1e909d Change scope of most/all default-scope i.e. package-scoped class members to protected, to make heritrix more friendly to third-party extensions. (I don't think there was ever an intent to limit the scope of any of these to package-only.) 2012-06-05 17:55:41 -07:00
Travis Wellman 493e2423f6 begin removing MultiReporter
* MultiReporter.java
    deprecated
* ToePool, ToeThread, WorkQueue, WorkQueueFrontier, PrecedenceProvider, CrawlURI, FetchStats.java
    remove MultiReporter#getReports() and reportTo(String, PrintWriter)
    these are not used as MultiReporter specifies, and MultiReporter is going away anyhow
2012-06-01 17:14:41 -07:00
Noah Levitt 3bebf10674 * ScriptedDecideRule.java
getEngine() - tiny cosmetic fixups
2012-05-31 16:29:33 -07:00
Travis Wellman 392a4b3fd0 correct synchronization and threadlocal logic
Previously the whole method was synchronized even if threadlocals
were used (which are thread safe). And previously the sharedEngine
was set and returned only when it /was not/ needed ie when
getIsolateThreads() returned true.

Now getEngine does what it says it does.
2012-05-31 16:09:42 -07:00
Noah Levitt 809981ef02 * PersistProcessor.java
populatePersistEnvFromLog() - handle exception deserializing line
2012-05-10 19:00:27 -07:00
Noah Levitt 9fffa8fc53 **/pom.xml
bump version to 3.1.2-SNAPSHOT
2012-05-01 18:17:33 -07:00
Noah Levitt 79c40346fc **/pom.xml
* bump version number to 3.1.1 for release
2012-05-01 18:01:06 -07:00
Noah Levitt a7c63b8d00 * FetchHTTP.java
remove redundant nonsensical comment
2012-05-01 17:53:52 -07:00
Noah Levitt a2880c4306 HER-1990 large robots.txt problematic
* Robotstxt.java
   enforce max size of robots.txt
2012-04-20 13:57:53 -07:00
Noah Levitt 4a64c6ba59 Revert "Altered FetchStats to also keep track of the number of URIs that have"
This reverts commit 4c1740ce1e.
2012-04-18 15:53:49 -07:00
Noah Levitt 9551c774af Merge branch 'master' into http-status-deciderule 2012-04-18 15:47:24 -07:00
Noah Levitt 8e853c5f8d HER-2005 existing content length decide rule is not flexible enough
* ResponseContentLengthDecideRule.java
    new predicated decide rule that will ACCEPT or REJECT a uri if the content body is within a specified size range
2012-04-18 13:15:15 -07:00
Noah Levitt 0a4d56b638 Fix for HER-2004 SEVERE error in FetchWhois on some government domains
* FetchWhois.java
    addWhoisLinks() - handle IllegalStateException from InternetDomainName.topPrivateDomain()
2012-04-18 10:37:24 -07:00
Travis Wellman 52eef84658 note about preselector.recheckScope for use of IpAddressSetDecideRule 2012-04-12 17:34:21 -07:00
Travis Wellman 57d2f42954 formatting 2012-04-12 16:50:22 -07:00
Travis Wellman 64e9cc367b remove println, make class public 2012-04-12 16:46:16 -07:00
Travis Wellman 2a3945ff79 Merge branch 'master' of github.com:internetarchive/heritrix3 into IpAddressSetDecideRule 2012-04-02 18:17:44 -07:00
Travis Wellman f9f9e62f4c Reject or accept matching a set of IP addresses. 2012-04-02 18:16:23 -07:00
Noah Levitt 335b32aecc * ExtractorHTML.java
some fixup on RELEVANT_TAG_EXTRACTOR javadoc comment
2012-03-20 15:14:14 -07:00
Adam Miller a108795d50 HER-1998 - Adding comments, adjusting test case to prevent subclasses from failing new tests 2012-03-19 18:19:00 -07:00
Adam Miller 1a333fbd35 HER-1998 - ExtractorHTML modified to parse html inside conditional comments. Still ignores normal comments 2012-03-16 14:36:15 -07:00
Noah Levitt e7d75cbb5a * AbstractPersistProcessor.java, PersistProcessor.java, PersistStoreProcessor.java
refactor some common persist processor code into new class AbstractPersistProcessor
2012-03-06 16:20:58 -08:00