Noah Levitt
98a4be5eb1
fix user-agent test, add more tests
2012-06-29 07:05:44 -07:00
Noah Levitt
1a79348cf7
test a few more things
2012-06-29 02:55:26 -07:00
Noah Levitt
1f315a7cc7
unit tests for FetchHTTP and FetchHTTP2
2012-06-29 02:04:39 -07:00
Noah Levitt
2653583cdd
unit tests for FetchHTTP and FetchHTTP2
2012-06-29 01:59:58 -07:00
Noah Levitt
63c6ca59c2
Merge branch 'master' into fetchhttp-hc
2012-06-25 16:42:34 -07:00
Travis Wellman
6626b44286
generics warnings fixes
...
This commit does not change any functionality, just
alters generics to reduce warnings in Eclipse.
2012-06-21 18:23:06 -07:00
Noah Levitt
2b0141fd0d
bdb cookie store for FetchHTTP2
2012-06-20 18:49:57 -07:00
Noah Levitt
49c00c0387
do not rely on ToeThread for the ServerCache, pass it to where it's needed instead
2012-06-19 13:54:05 -07:00
Noah Levitt
56f0c92690
resolve hostnames from heritrix server cache (the bad old way)
2012-06-19 13:32:34 -07:00
Noah Levitt
1d0beedf2b
use http 1.0 by default, other request configuration stuff
2012-06-19 12:09:46 -07:00
Noah Levitt
233b72cb34
get charset, encodings from headers
2012-06-19 11:28:01 -07:00
Noah Levitt
de152eaa9e
browser-compatibility mode for cookie matching; that's what old FetchHTTP does, and also avoids sending weird Cookie2:$Version1
2012-06-18 17:49:09 -07:00
Noah Levitt
3c7bd8afd6
no keep-alive
2012-06-18 17:06:46 -07:00
Noah Levitt
a779e46ac7
set user-agent
2012-06-18 15:55:07 -07:00
Noah Levitt
7422a7b992
move httpclient helper classes to commons/
2012-06-18 15:43:52 -07:00
Noah Levitt
7f0efbdac8
do recording through SessionOutputBuffer implementation like we have for SessionInputBuffer; refactor helper classes into their own file RecordingHttpClient.java
2012-06-18 15:42:50 -07:00
Noah Levitt
0d172be888
not using HcInputWrapper anymore
2012-06-18 12:31:52 -07:00
Noah Levitt
316391786e
license headers
2012-06-17 23:23:39 -07:00
Noah Levitt
6c56ce41a0
different deeper way to insert our Recorder, letting us correctly note end of http headers. is there a problem with not buffering at this level?
2012-06-17 15:48:19 -07:00
Noah Levitt
b27b3f67e5
continuing to mess with httpcomponents
2012-06-15 18:29:21 -07:00
Noah Levitt
9a5570c6f1
continuing to mess with httpcomponents
2012-06-15 16:34:55 -07:00
Noah Levitt
3a0f50d031
FetchHTTP2 - record, archive request as well
2012-06-14 19:01:15 -07:00
Noah Levitt
5043dcc35b
FetchHTTP2, based on httpcomponents, is working in a rudimentary way. This commit also gets rid of most egregious commons-httpclient-specific code in CrawlURI
2012-06-14 18:34:43 -07:00
Noah Levitt
793d733f1e
continuing to mess with httpcomponents
2012-06-14 14:00:16 -07:00
Noah Levitt
e5668726a1
starting to mess with httpcomponents
2012-06-13 14:11:11 -07:00
Noah Levitt
e1f5bc5319
HER-1987 record charset used for link extraction in warc metadata record
...
* WARCWriterProcessor.java
writeMetadata() - write metadata entry charsetForLinkExtraction
2012-06-06 12:02:41 -07:00
Noah Levitt
7eabc7ef73
HER-1987 record charset used for link extraction in warc metadata record
...
* WARCWriterProcessor.java
writeMetadata() - if extractors have left annotations about the encoding used to interpret content, include them in the metadata (HER-1987)
2012-06-06 11:40:58 -07:00
Noah Levitt
35378df2fa
Merge branch 'master' into no-package-scope
2012-06-05 17:58:40 -07:00
Noah Levitt
3dce1e909d
Change scope of most/all default-scope i.e. package-scoped class members to protected, to make heritrix more friendly to third-party extensions. (I don't think there was ever an intent to limit the scope of any of these to package-only.)
2012-06-05 17:55:41 -07:00
Travis Wellman
493e2423f6
begin removing MultiReporter
...
* MultiReporter.java
deprecated
* ToePool, ToeThread, WorkQueue, WorkQueueFrontier, PrecedenceProvider, CrawlURI, FetchStats.java
remove MultiReporter#getReports() and reportTo(String, PrintWriter)
these are not used as MultiReporter specifies, and MultiReporter is going away anyhow
2012-06-01 17:14:41 -07:00
Noah Levitt
3bebf10674
* ScriptedDecideRule.java
...
getEngine() - tiny cosmetic fixups
2012-05-31 16:29:33 -07:00
Travis Wellman
392a4b3fd0
correct synchronization and threadlocal logic
...
Previously the whole method was synchronized even if threadlocals
were used (which are thread safe). And previously the sharedEngine
was set and returned only when it /was not/ needed ie when
getIsolateThreads() returned true.
Now getEngine does what it says it does.
2012-05-31 16:09:42 -07:00
Noah Levitt
809981ef02
* PersistProcessor.java
...
populatePersistEnvFromLog() - handle exception deserializing line
2012-05-10 19:00:27 -07:00
Noah Levitt
9fffa8fc53
**/pom.xml
...
bump version to 3.1.2-SNAPSHOT
2012-05-01 18:17:33 -07:00
Noah Levitt
79c40346fc
**/pom.xml
...
* bump version number to 3.1.1 for release
2012-05-01 18:01:06 -07:00
Noah Levitt
a7c63b8d00
* FetchHTTP.java
...
remove redundant nonsensical comment
2012-05-01 17:53:52 -07:00
Noah Levitt
a2880c4306
HER-1990 large robots.txt problematic
...
* Robotstxt.java
enforce max size of robots.txt
2012-04-20 13:57:53 -07:00
Noah Levitt
4a64c6ba59
Revert "Altered FetchStats to also keep track of the number of URIs that have"
...
This reverts commit 4c1740ce1e .
2012-04-18 15:53:49 -07:00
Noah Levitt
9551c774af
Merge branch 'master' into http-status-deciderule
2012-04-18 15:47:24 -07:00
Noah Levitt
8e853c5f8d
HER-2005 existing content length decide rule is not flexible enough
...
* ResponseContentLengthDecideRule.java
new predicated decide rule that will ACCEPT or REJECT a uri if the content body is within a specified size range
2012-04-18 13:15:15 -07:00
Noah Levitt
0a4d56b638
Fix for HER-2004 SEVERE error in FetchWhois on some government domains
...
* FetchWhois.java
addWhoisLinks() - handle IllegalStateException from InternetDomainName.topPrivateDomain()
2012-04-18 10:37:24 -07:00
Travis Wellman
52eef84658
note about preselector.recheckScope for use of IpAddressSetDecideRule
2012-04-12 17:34:21 -07:00
Travis Wellman
57d2f42954
formatting
2012-04-12 16:50:22 -07:00
Travis Wellman
64e9cc367b
remove println, make class public
2012-04-12 16:46:16 -07:00
Travis Wellman
2a3945ff79
Merge branch 'master' of github.com:internetarchive/heritrix3 into IpAddressSetDecideRule
2012-04-02 18:17:44 -07:00
Travis Wellman
f9f9e62f4c
Reject or accept matching a set of IP addresses.
2012-04-02 18:16:23 -07:00
Noah Levitt
335b32aecc
* ExtractorHTML.java
...
some fixup on RELEVANT_TAG_EXTRACTOR javadoc comment
2012-03-20 15:14:14 -07:00
Adam Miller
a108795d50
HER-1998 - Adding comments, adjusting test case to prevent subclasses from failing new tests
2012-03-19 18:19:00 -07:00
Adam Miller
1a333fbd35
HER-1998 - ExtractorHTML modified to parse html inside conditional comments. Still ignores normal comments
2012-03-16 14:36:15 -07:00
Noah Levitt
e7d75cbb5a
* AbstractPersistProcessor.java, PersistProcessor.java, PersistStoreProcessor.java
...
refactor some common persist processor code into new class AbstractPersistProcessor
2012-03-06 16:20:58 -08:00