Commit Graph

  • 7422a7b992 move httpclient helper classes to commons/ Noah Levitt 2012-06-18 15:43:52 -07:00
  • 7f0efbdac8 do recording through SessionOutputBuffer implementation like we have for SessionInputBuffer; refactor helper classes into their own file RecordingHttpClient.java Noah Levitt 2012-06-18 15:42:50 -07:00
  • 0d172be888 not using HcInputWrapper anymore Noah Levitt 2012-06-18 12:31:52 -07:00
  • 316391786e license headers Noah Levitt 2012-06-17 23:23:39 -07:00
  • 6c56ce41a0 different deeper way to insert our Recorder, letting us correctly note end of http headers. is there a problem with not buffering at this level? Noah Levitt 2012-06-17 15:48:19 -07:00
  • b27b3f67e5 continuing to mess with httpcomponents Noah Levitt 2012-06-15 18:29:21 -07:00
  • 9a5570c6f1 continuing to mess with httpcomponents Noah Levitt 2012-06-15 16:34:55 -07:00
  • 3a0f50d031 FetchHTTP2 - record, archive request as well Noah Levitt 2012-06-14 19:01:15 -07:00
  • 5043dcc35b FetchHTTP2, based on httpcomponents, is working in a rudimentary way. This commit also gets rid of most egregious commons-httpclient-specific code in CrawlURI Noah Levitt 2012-06-14 18:34:43 -07:00
  • 793d733f1e continuing to mess with httpcomponents Noah Levitt 2012-06-14 14:00:16 -07:00
  • e5668726a1 starting to mess with httpcomponents Noah Levitt 2012-06-13 14:11:11 -07:00
  • 6ab7415342 Revert "* ConfigFileEditor.java, ConfigPathEditor.java" Noah Levitt 2012-06-12 10:15:27 -07:00
  • 6dae8f9a17 * ConfigFileEditor.java, ConfigPathEditor.java remove unused classes Noah Levitt 2012-06-11 17:21:23 -07:00
  • e1f5bc5319 HER-1987 record charset used for link extraction in warc metadata record * WARCWriterProcessor.java writeMetadata() - write metadata entry charsetForLinkExtraction Noah Levitt 2012-06-06 12:02:41 -07:00
  • 7eabc7ef73 HER-1987 record charset used for link extraction in warc metadata record * WARCWriterProcessor.java writeMetadata() - if extractors have left annotations about the encoding used to interpret content, include them in the metadata (HER-1987) Noah Levitt 2012-06-06 11:40:58 -07:00
  • 99acaa9bfa Fix tests revealed to be failing * UriUtils.java isLikelyFalsePositive() - do not flag urls starting with ../ as likely false positives * UriUtilsTest.java remove test url that is now rejected Noah Levitt 2012-06-06 09:00:13 -07:00
  • 342ea33be5 Move test classes into test area of source tree * LongToIntConsistentHashTest.java, UriUtilsTest.java Noah Levitt 2012-06-05 18:03:33 -07:00
  • 35378df2fa Merge branch 'master' into no-package-scope Noah Levitt 2012-06-05 17:58:40 -07:00
  • 3dce1e909d Change scope of most/all default-scope i.e. package-scoped class members to protected, to make heritrix more friendly to third-party extensions. (I don't think there was ever an intent to limit the scope of any of these to package-only.) Noah Levitt 2012-06-05 17:55:41 -07:00
  • e965421281 * WorkQueueFrontier.java remove unused variable Noah Levitt 2012-06-05 16:26:00 -07:00
  • ee1e25ca69 Merge pull request #11 from travisfw/new-reports Travis Wellman 2012-06-05 16:24:43 -07:00
  • 07a2fc65ea use methods to specify which report rather than string constants Travis Wellman 2012-06-01 17:57:25 -07:00
  • 493e2423f6 begin removing MultiReporter Travis Wellman 2012-06-01 17:14:41 -07:00
  • 3bebf10674 * ScriptedDecideRule.java getEngine() - tiny cosmetic fixups Noah Levitt 2012-05-31 16:29:33 -07:00
  • 392a4b3fd0 correct synchronization and threadlocal logic Travis Wellman 2012-05-31 16:09:42 -07:00
  • 1015c5fa00 Fix HER-1973 Regression in queue budget overrides * WorkQueueFrontier.java findEligibleURI() - apply sheet overrides before refreshing queue budgets Noah Levitt 2012-05-15 17:35:49 -07:00
  • 809981ef02 * PersistProcessor.java populatePersistEnvFromLog() - handle exception deserializing line Noah Levitt 2012-05-10 19:00:27 -07:00
  • 440129ad23 Merge branch 'master' of git@github.com:internetarchive/heritrix3.git gojomo 2012-05-02 01:48:21 -07:00
  • 9fffa8fc53 **/pom.xml bump version to 3.1.2-SNAPSHOT Noah Levitt 2012-05-01 18:17:33 -07:00
  • 43d57dcaf4 * README.txt update for 3.1.1 3.1.1 Noah Levitt 2012-05-01 18:05:38 -07:00
  • 79c40346fc **/pom.xml * bump version number to 3.1.1 for release Noah Levitt 2012-05-01 18:01:06 -07:00
  • a7c63b8d00 * FetchHTTP.java remove redundant nonsensical comment Noah Levitt 2012-05-01 17:53:52 -07:00
  • b05f64e2ab Merge branch 'master' of git@github.com:internetarchive/heritrix3.git gojomo 2012-04-25 15:09:41 -07:00
  • e4f12bb20f Merge pull request #6 from ato/codemirror Gordon Mohr 2012-04-25 15:04:10 -07:00
  • b80966ebd9 HER-2001: Use the CodeMirror editor for crawl config and script console Alex Osborne 2012-03-31 18:34:11 +11:00
  • 42e759c01d List correct defaults for BdbModule cacheSize and cachePercentage with an explanatory comment. Noah Levitt 2012-04-25 11:10:56 -07:00
  • a2880c4306 HER-1990 large robots.txt problematic * Robotstxt.java enforce max size of robots.txt Noah Levitt 2012-04-20 13:57:53 -07:00
  • 52d5307a9b Merge branch 'her-1948' Noah Levitt 2012-04-20 12:33:19 -07:00
  • 8d0a1678de * effective_tld_names.dat update to latest from publicsuffix.org Noah Levitt 2012-04-20 12:25:57 -07:00
  • 3fed078817 Merge pull request #9 from kngenie/publicsuffixes2 Noah Levitt 2012-04-20 12:19:47 -07:00
  • da28258f90 Option to set bdb cache size as an absolute number instead of a percentage (part of HER-1948) Noah Levitt 2012-04-19 18:16:53 -07:00
  • 4a64c6ba59 Revert "Altered FetchStats to also keep track of the number of URIs that have" Noah Levitt 2012-04-18 15:53:49 -07:00
  • 9551c774af Merge branch 'master' into http-status-deciderule Noah Levitt 2012-04-18 15:47:24 -07:00
  • 8e853c5f8d HER-2005 existing content length decide rule is not flexible enough * ResponseContentLengthDecideRule.java new predicated decide rule that will ACCEPT or REJECT a uri if the content body is within a specified size range Noah Levitt 2012-04-18 13:15:15 -07:00
  • 0a4d56b638 Fix for HER-2004 SEVERE error in FetchWhois on some government domains * FetchWhois.java addWhoisLinks() - handle IllegalStateException from InternetDomainName.topPrivateDomain() Noah Levitt 2012-04-18 10:37:24 -07:00
  • 27d621f21a renamed PublicSuffixes2 to PublicSuffixes. deleted debug println. Kenji Nagahashi 2011-11-05 23:26:55 -07:00
  • a0d52e6c17 PublicSuffixes2Test still had tests for PublicSuffix. fixed. Kenji Nagahashi 2011-10-20 19:39:36 -07:00
  • b2d62a81ff trial rewrite of PublicSuffixes for small memory footprint (no significant performance gain) Kenji Nagahashi 2011-10-11 17:11:06 -07:00
  • 31df3a8f03 Merge branch 'master' into speculative-links Noah Levitt 2012-04-13 18:16:08 -07:00
  • 52eef84658 note about preselector.recheckScope for use of IpAddressSetDecideRule Travis Wellman 2012-04-12 17:34:21 -07:00
  • 54b1e245eb gitignore for maven build artifacts Travis Wellman 2012-04-12 17:32:57 -07:00
  • 563da14330 Merge pull request #7 from travisfw/IpAddressSetDecideRule Travis Wellman 2012-04-12 16:52:00 -07:00
  • 57d2f42954 formatting Travis Wellman 2012-04-12 16:50:22 -07:00
  • 64e9cc367b remove println, make class public Travis Wellman 2012-04-12 16:46:16 -07:00
  • 2a3945ff79 Merge branch 'master' of github.com:internetarchive/heritrix3 into IpAddressSetDecideRule Travis Wellman 2012-04-02 18:17:44 -07:00
  • f9f9e62f4c Reject or accept matching a set of IP addresses. Travis Wellman 2012-04-02 18:16:23 -07:00
  • f19a72efaf HER-2001: Use the CodeMirror editor for crawl config and script console Alex Osborne 2012-03-31 18:34:11 +11:00
  • 9626b8454a * EditRepresentation.java make edit page (for cxml mainly) prettier with big textarea (see Kenji's comment on HER-1993 dated 2012-03-27 18:03) Noah Levitt 2012-03-30 11:44:28 -07:00
  • 335b32aecc * ExtractorHTML.java some fixup on RELEVANT_TAG_EXTRACTOR javadoc comment Noah Levitt 2012-03-20 15:14:14 -07:00
  • 4eab6461ed Merge remote-tracking branch 'adam-miller/master' Noah Levitt 2012-03-20 15:06:58 -07:00
  • a108795d50 HER-1998 - Adding comments, adjusting test case to prevent subclasses from failing new tests Adam Miller 2012-03-19 18:19:00 -07:00
  • ea42c0c653 * EngineResource.java acceptRepresentation() - "I'm sure" parameter is now "im_sure" Noah Levitt 2012-03-19 17:30:18 -07:00
  • 1a333fbd35 HER-1998 - ExtractorHTML modified to parse html inside conditional comments. Still ignores normal comments Adam Miller 2012-03-16 14:36:15 -07:00
  • f7505e6673 More HTML cleanup. Html5 doctype declaration, get rid of <br> and xml-style "/>" tag endings, other cleanup to make it validate. Noah Levitt 2012-03-15 18:47:18 -07:00
  • 52703c6eb4 * ArchiveUtils.java doubleToString() - return "NaN" for NaN (inf symbol didn't quite make sense to me) Noah Levitt 2012-03-15 18:46:01 -07:00
  • 7cde3b4f17 Merge branch 'master' of github.com:internetarchive/heritrix3 into html-cleanup Noah Levitt 2012-03-15 15:59:10 -07:00
  • 64f24d4c66 HER-1993: In JobResource, write error message instead of throwing RuntimeException when it failed to read job log or crawl log. Kenji Nagahashi 2012-03-09 16:48:11 -08:00
  • f05287c8ca restored deleted space after header "Crawl Log" Kenji Nagahashi 2012-03-09 16:29:52 -08:00
  • e7d75cbb5a * AbstractPersistProcessor.java, PersistProcessor.java, PersistStoreProcessor.java refactor some common persist processor code into new class AbstractPersistProcessor Noah Levitt 2012-03-06 16:20:58 -08:00
  • adc82c1c6f Second round of HTML cleanup: - added missing <html> and <body> - fixed non-working CSS link, added CSS link to all pages - replaced layout by <br> and &nbsp; with structural tags with CSS - removed superfluous output, notably in BeanBrowseResource - <code>-ify variable names in ScriptResource - added special handling of NaN in doubleToString() Kenji Nagahashi 2012-03-03 00:46:56 -08:00
  • 46c335c8a3 BeanBrowseResource: fixed malformed HTML. Kenji Nagahashi 2011-05-11 01:25:41 -07:00
  • 1d05df7fbe * StatisticsTracker.java writeReportFile() - reduce log level from warning to info when reusing existing report (normal after crawl is finished) Noah Levitt 2012-02-29 11:02:39 -08:00
  • 7af7cbf18c Merge branch 'master' of github.com:internetarchive/heritrix3 Noah Levitt 2012-02-10 15:51:15 -08:00
  • db801b3e0f Fix problems handling crawl job paths with characters that require url-encoding, since we sometimes refer to them with file: urls * PathSharingContext.java, CrawlJob.java use java.net.URI to convert paths to and from file: urls Noah Levitt 2012-02-10 15:48:09 -08:00
  • d83ba3070b Merge branch 'master' of https://github.com/travisfw/heritrix3 Noah Levitt 2012-02-09 16:43:19 -08:00
  • cfac7ead91 Fix for HER-1985 H3: SurtPrefixDecideRule forgets learned/seed-derived/seed-directive SURT prefixes in checkpoint-resume * SurtPrefixedDecideRule.java make class implement Checkpointable, save surt prefixes to json on checkpoint and load them on recover * profile-crawler-beans.cxml make main SurtPrefixedDecideRule a top-level bean so that it can be checkpointed * Checkpoint.java, CheckpointService.java add some FINE level logging during checkpointing and recovery Noah Levitt 2012-02-09 10:05:07 -08:00
  • e90fc2965b Sort script engines alphabetically so that they appear consistently. Note that the name of the engine may be different than the name of the language it parses. A good example is the Rhino script engine that parses javascript. Travis Wellman 2012-01-26 00:37:19 -08:00
  • a652a9afd0 * profile-crawler-beans.cxml commented-out default value for new CandidatesProcessor settings processErrorOutlinks Noah Levitt 2012-01-20 11:30:24 -08:00
  • 9dbe673bc3 * UriUtils.java some improvements to likely-uri detection based on crawl log analysis Noah Levitt 2012-01-20 11:23:45 -08:00
  • 4e05c89156 Merge branch 'master' of github.com:internetarchive/heritrix3 Noah Levitt 2012-01-19 13:41:14 -08:00
  • 803f9ca979 HER-1984 save script state - implement by adding a map to the application context for arbitrary use, and make sure the app context is available in all scripting environments * PathSharingContext.java new member variable ConcurrentHashMap data and accessor getData() * ScriptedProcessor.java, ScriptedDecideRule.java make appCtx available to scripts; also remove unused member sharedMap * ActionDirectory.java formatting fix Noah Levitt 2012-01-19 13:31:57 -08:00
  • 20502f54d5 add processErrorOutlinks setting to CandidatesProcessor default false is old behavior, skip candidates-handling of outlinks from response codes <200 and >=400 if true, these outlinks will be treated the same as others (get scope-tested and enqueued) gojomo 2012-01-18 01:23:26 -08:00
  • 4384312647 reuse pre-allocated shared HTMLLinkContext instance. Kenji Nagahashi 2011-01-25 14:06:43 -08:00
  • 55ec01c4e4 Fix for HER-1983 mangled dns records in w/arcs * FetchDNS.java recordDNS() - use local variable for dummy read buffer, instead of member variable that was shared among threads, since RecordingInputStream.read() used the passed buffer as a work area Noah Levitt 2012-01-11 14:38:57 -08:00
  • d3ea17baf5 [HER-1976] query-portion ignored when checking against robots.txt rules (using only path) (RobotsPolicyTest) added for case that failed pre-fix (RobotsTxtTest) added case similar to problem (but was already OK at RobotsTxt level) (RobotsPolicy, *RobotsPolicy) rename method to reflect new behavior gojomo 2012-01-05 23:37:00 -08:00
  • 4829160c84 [HER-1981] H3: overlay setting of alternate queue-precedence (via BaseQueuePrecedencePolicy.basePrecedence) doesn't stick (CrawlURI) don't null overlayNames in processingCleanup (WorkQueueFrontier) move queue precedence recalc to only in handleQueue, not all (post-wake) reenqueues gojomo 2012-01-05 23:01:54 -08:00
  • 4ffb6865e1 [HER-1976] H3: query-portion ignored when checking against robots.txt rules (using only path) * RobotsPolicy.java: use UURI.getPathQuery -- like H1 does Gordon Mohr 2011-12-22 19:04:19 -08:00
  • 4c1740ce1e Altered FetchStats to also keep track of the number of URIs that have an HTTP status between 200 and 299. Christopher Miles 2011-11-17 12:34:05 -05:00
  • 86cf3e1fcd Added two new decide rules that make decisions against the HTTP status code for the provided URI. This is likely most useful when you want to keep the size of the stored crawled data down be rejecting 404s, etc. Christopher Miles 2011-11-16 14:53:11 -05:00
  • a4380db177 .classpath remove unusued, unneeded dependency asm-3.2.jar, see http://tech.groups.yahoo.com/group/archive-crawler/message/7384 Noah Levitt 2011-11-03 16:15:20 -07:00
  • abda8a01fb HER-1963 Unvisited hosts being written twice into H3-generated hosts-report.txt * HostsReport.java write() - remove obsolete code described by this no longer correct comment - "StatisticsTracker doesn't know of zero-completion hosts; so supplement report with those entries from host cache" - StatisticsTracker does know of zero-completion hosts Noah Levitt 2011-11-02 13:01:40 -07:00
  • dff143b301 HER-1964 record whois server ip address in warc * FetchWhois.java fetch() - record server ip address in CrawlURI attributes list * CoreAttributeConstants.java new string constant A_WHOIS_SERVER_IP, key for whois server ip address CrawlURI attribute * WARCWriterProcessor.java writeWhoisRecords() - include warc header WARC-IP-Address Noah Levitt 2011-11-01 16:57:22 -07:00
  • 48434a6293 Merge branch 'master' of github.com:internetarchive/heritrix3 Noah Levitt 2011-11-01 16:56:43 -07:00
  • 450a4c44c4 * CrawlURI.java markPrerequisite() - call setPrerequisite(true) on the prerequisite CrawlURI Noah Levitt 2011-10-28 09:55:55 -07:00
  • 264d3c6f5f * CrawlURI.java markPrerequisite() - call setPrerequisite(true) on the prerequisite CrawlURI Noah Levitt 2011-10-28 09:55:55 -07:00
  • 995133dc40 Fix for HER-1962 NPE from missing sheet * SheetOverlaysManager.java getOverlayMap(String) - return null if sheet missing instead of triggering npe * KeyedProperties.java get(String) - check for null return value from getOverlayMap() and log warning Noah Levitt 2011-10-27 10:40:30 -07:00
  • 9b418bbb76 * LICENSE.txt revert previous commit - we still distribute jars and licenses in the binary distribution (duh) Noah Levitt 2011-10-21 23:10:25 -07:00
  • 239f616c17 * **/.gitignore dummy files to make git keep these directories which are expected by unit tests Noah Levitt 2011-10-21 22:40:44 -07:00
  • 23028ccab1 * dist/LICENSE.txt remove clause about lib/ directory, which doesn't apply to h3 (XXX replace with something about maven downloading dependencies?) Noah Levitt 2011-10-21 22:39:22 -07:00
  • e8cdfcd82f * README.txt symlink to dist/README.txt just to have one in the root of the project Noah Levitt 2011-10-21 15:56:33 -07:00