Noah Levitt
2b0141fd0d
bdb cookie store for FetchHTTP2
2012-06-20 18:49:57 -07:00
Noah Levitt
49c00c0387
do not rely on ToeThread for the ServerCache, pass it to where it's needed instead
2012-06-19 13:54:05 -07:00
Noah Levitt
56f0c92690
resolve hostnames from heritrix server cache (the bad old way)
2012-06-19 13:32:34 -07:00
Noah Levitt
1d0beedf2b
use http 1.0 by default, other request configuration stuff
2012-06-19 12:09:46 -07:00
Noah Levitt
233b72cb34
get charset, encodings from headers
2012-06-19 11:28:01 -07:00
Noah Levitt
de152eaa9e
browser-compatibility mode for cookie matching; that's what old FetchHTTP does, and also avoids sending weird Cookie2:$Version1
2012-06-18 17:49:09 -07:00
Noah Levitt
3c7bd8afd6
no keep-alive
2012-06-18 17:06:46 -07:00
Noah Levitt
a779e46ac7
set user-agent
2012-06-18 15:55:07 -07:00
Noah Levitt
7422a7b992
move httpclient helper classes to commons/
2012-06-18 15:43:52 -07:00
Noah Levitt
7f0efbdac8
do recording through SessionOutputBuffer implementation like we have for SessionInputBuffer; refactor helper classes into their own file RecordingHttpClient.java
2012-06-18 15:42:50 -07:00
Noah Levitt
0d172be888
not using HcInputWrapper anymore
2012-06-18 12:31:52 -07:00
Noah Levitt
316391786e
license headers
2012-06-17 23:23:39 -07:00
Noah Levitt
6c56ce41a0
different deeper way to insert our Recorder, letting us correctly note end of http headers. is there a problem with not buffering at this level?
2012-06-17 15:48:19 -07:00
Noah Levitt
b27b3f67e5
continuing to mess with httpcomponents
2012-06-15 18:29:21 -07:00
Noah Levitt
9a5570c6f1
continuing to mess with httpcomponents
2012-06-15 16:34:55 -07:00
Noah Levitt
3a0f50d031
FetchHTTP2 - record, archive request as well
2012-06-14 19:01:15 -07:00
Noah Levitt
5043dcc35b
FetchHTTP2, based on httpcomponents, is working in a rudimentary way. This commit also gets rid of most egregious commons-httpclient-specific code in CrawlURI
2012-06-14 18:34:43 -07:00
Noah Levitt
793d733f1e
continuing to mess with httpcomponents
2012-06-14 14:00:16 -07:00
Noah Levitt
e5668726a1
starting to mess with httpcomponents
2012-06-13 14:11:11 -07:00
Noah Levitt
6ab7415342
Revert "* ConfigFileEditor.java, ConfigPathEditor.java"
...
This reverts commit 6dae8f9a17
2012-06-12 10:15:27 -07:00
Noah Levitt
6dae8f9a17
* ConfigFileEditor.java, ConfigPathEditor.java
...
remove unused classes
2012-06-11 17:21:23 -07:00
Noah Levitt
e1f5bc5319
HER-1987 record charset used for link extraction in warc metadata record
...
* WARCWriterProcessor.java
writeMetadata() - write metadata entry charsetForLinkExtraction
2012-06-06 12:02:41 -07:00
Noah Levitt
7eabc7ef73
HER-1987 record charset used for link extraction in warc metadata record
...
* WARCWriterProcessor.java
writeMetadata() - if extractors have left annotations about the encoding used to interpret content, include them in the metadata (HER-1987)
2012-06-06 11:40:58 -07:00
Noah Levitt
99acaa9bfa
Fix tests revealed to be failing
...
* UriUtils.java
isLikelyFalsePositive() - do not flag urls starting with ../ as likely false positives
* UriUtilsTest.java
remove test url that is now rejected
2012-06-06 09:00:13 -07:00
Noah Levitt
342ea33be5
Move test classes into test area of source tree
...
* LongToIntConsistentHashTest.java, UriUtilsTest.java
2012-06-05 18:03:33 -07:00
Noah Levitt
35378df2fa
Merge branch 'master' into no-package-scope
2012-06-05 17:58:40 -07:00
Noah Levitt
3dce1e909d
Change scope of most/all default-scope i.e. package-scoped class members to protected, to make heritrix more friendly to third-party extensions. (I don't think there was ever an intent to limit the scope of any of these to package-only.)
2012-06-05 17:55:41 -07:00
Noah Levitt
e965421281
* WorkQueueFrontier.java
...
remove unused variable
2012-06-05 16:26:00 -07:00
Travis Wellman
ee1e25ca69
Merge pull request #11 from travisfw/new-reports
...
deleting MultiReporter interface
2012-06-05 16:24:43 -07:00
Travis Wellman
07a2fc65ea
use methods to specify which report rather than string constants
...
* MultiReporter
deleted. direct method calls are a better way to specify the kind of report desired
getReports was never used to choose from the array of strings. no use case for MultiReporter.
* ArchiveUtils
writeReportToString was not used, and it used the now deleted MultiReporter#reportTo
* DevUtils, Frontier, ToePool, AbstractFrontier, WorkQueueFrontier
MultiReporter -> Reporter and make methods that implemented named multireport public
* FrontierNonemptyReport, FrontierSummaryReport
call the appropriate report method instead of passing a string constant
2012-06-01 17:57:25 -07:00
Travis Wellman
493e2423f6
begin removing MultiReporter
...
* MultiReporter.java
deprecated
* ToePool, ToeThread, WorkQueue, WorkQueueFrontier, PrecedenceProvider, CrawlURI, FetchStats.java
remove MultiReporter#getReports() and reportTo(String, PrintWriter)
these are not used as MultiReporter specifies, and MultiReporter is going away anyhow
2012-06-01 17:14:41 -07:00
Noah Levitt
3bebf10674
* ScriptedDecideRule.java
...
getEngine() - tiny cosmetic fixups
2012-05-31 16:29:33 -07:00
Travis Wellman
392a4b3fd0
correct synchronization and threadlocal logic
...
Previously the whole method was synchronized even if threadlocals
were used (which are thread safe). And previously the sharedEngine
was set and returned only when it /was not/ needed ie when
getIsolateThreads() returned true.
Now getEngine does what it says it does.
2012-05-31 16:09:42 -07:00
Noah Levitt
1015c5fa00
Fix HER-1973 Regression in queue budget overrides
...
* WorkQueueFrontier.java
findEligibleURI() - apply sheet overrides before refreshing queue budgets
2012-05-15 17:35:49 -07:00
Noah Levitt
809981ef02
* PersistProcessor.java
...
populatePersistEnvFromLog() - handle exception deserializing line
2012-05-10 19:00:27 -07:00
Noah Levitt
9fffa8fc53
**/pom.xml
...
bump version to 3.1.2-SNAPSHOT
2012-05-01 18:17:33 -07:00
Noah Levitt
43d57dcaf4
* README.txt
...
update for 3.1.1
2012-05-01 18:05:38 -07:00
Noah Levitt
79c40346fc
**/pom.xml
...
* bump version number to 3.1.1 for release
2012-05-01 18:01:06 -07:00
Noah Levitt
a7c63b8d00
* FetchHTTP.java
...
remove redundant nonsensical comment
2012-05-01 17:53:52 -07:00
Gordon Mohr
e4f12bb20f
Merge pull request #6 from ato/codemirror
...
HER-2001: Use the CodeMirror editor for crawl config and script console
(Thanks Alex!)
2012-04-25 15:04:10 -07:00
Noah Levitt
42e759c01d
List correct defaults for BdbModule cacheSize and cachePercentage with an explanatory comment.
2012-04-25 11:10:56 -07:00
Noah Levitt
a2880c4306
HER-1990 large robots.txt problematic
...
* Robotstxt.java
enforce max size of robots.txt
2012-04-20 13:57:53 -07:00
Noah Levitt
52d5307a9b
Merge branch 'her-1948'
2012-04-20 12:33:19 -07:00
Noah Levitt
8d0a1678de
* effective_tld_names.dat
...
update to latest from publicsuffix.org
2012-04-20 12:25:57 -07:00
Noah Levitt
3fed078817
Merge pull request #9 from kngenie/publicsuffixes2
...
Publicsuffixes2
2012-04-20 12:19:47 -07:00
Noah Levitt
da28258f90
Option to set bdb cache size as an absolute number instead of a percentage (part of HER-1948)
2012-04-19 18:16:53 -07:00
Noah Levitt
4a64c6ba59
Revert "Altered FetchStats to also keep track of the number of URIs that have"
...
This reverts commit 4c1740ce1e
2012-04-18 15:53:49 -07:00
Noah Levitt
9551c774af
Merge branch 'master' into http-status-deciderule
2012-04-18 15:47:24 -07:00
Noah Levitt
8e853c5f8d
HER-2005 existing content length decide rule is not flexible enough
...
* ResponseContentLengthDecideRule.java
new predicated decide rule that will ACCEPT or REJECT a uri if the content body is within a specified size range
2012-04-18 13:15:15 -07:00
Noah Levitt
0a4d56b638
Fix for HER-2004 SEVERE error in FetchWhois on some government domains
...
* FetchWhois.java
addWhoisLinks() - handle IllegalStateException from InternetDomainName.topPrivateDomain()
2012-04-18 10:37:24 -07:00
Noah Levitt