lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-23 15:06:46 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,335 Commits 40 Branches 31 Tags
b33d3faa756dfac94d0a5e1df4feeb358c94e72b
Commit Graph
1335 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Eldon b33d3faa75 Better Log messages 2014-01-24 18:49:04 -08:00
Eldon ab3aadf199 Pull out most umbra refs, only do http/https 2014-01-24 18:47:50 -08:00
Eldon d837536dc6 Use toString method instead 2014-01-24 18:28:38 -08:00
Eldon d3e96b2770 First pass at a processor to publish crawluris to AMQP channels 2014-01-24 18:11:47 -08:00
Kenji Nagahashi 7353f62d84 Merge pull request #32 from nlevitt/hc43 
Switch to BasicHttpClientConnectionManager instead of PoolingHttpClientConnectionManager
 2014-01-24 16:43:06 -08:00
Noah Levitt 6b72ac9705 Switch to BasicHttpClientConnectionManager instead of 
PoolingHttpClientConnectionManager, and make the connection manager a member
variable of HttpRequest. We had been seeing a strange bug where once in
every few thousand urls or so, the connection manager would get gc'd
while the http request was being sent, and RecordingOutputStream.flush()
would get called, as in the stack trace below.  The ToeThread was using
it at the same time, so if conditions were right, the recorded stream
would get mangled. This commit makes the connection manager a member of
FetchHTTPRequest so it doesn't get gc'd prematurely. Also switches to
BasicHttpClientConnectionManager, since we're not doing any pooling.

  it.unimi.dsi.fastutil.io.FastBufferedOutputStream.dumpBuffer(boolean) line: 133
  it.unimi.dsi.fastutil.io.FastBufferedOutputStream.flush() line: 166
  org.archive.io.RecordingOutputStream.flush() line: 469
  org.apache.http.impl.io.SessionOutputBufferImpl.flushStream() line: 128
  org.apache.http.impl.io.SessionOutputBufferImpl.flush() line: 143
  org.archive.modules.fetcher.FetchHTTPRequest$RecordingHttpClientConnection(org.apache.http.impl.BHttpConnectionBase).close() line: 314
  org.archive.modules.fetcher.FetchHTTPRequest$RecordingHttpClientConnection.close() line: 520
  org.apache.http.impl.conn.CPoolEntry.closeConnection() line: 69
  org.apache.http.impl.conn.CPoolEntry.close() line: 95
  org.apache.http.impl.conn.CPool(org.apache.http.pool.AbstractConnPool<T,C,E>).shutdown() line: 130
  org.apache.http.impl.conn.PoolingHttpClientConnectionManager.shutdown() line: 347
  org.apache.http.impl.conn.PoolingHttpClientConnectionManager.finalize() line: 168
  java.lang.ref.Finalizer.invokeFinalizeMethod(java.lang.Object) line: not available [native method]
 2014-01-24 16:07:04 -08:00
Kenji Nagahashi b1b2f2bc42 Merge pull request #31 from nlevitt/proxy-port-configurable 
make http proxy port configurable in cxml, avoiding this: org.springfram...
 2014-01-22 16:05:44 -08:00
Noah Levitt c5c8788227 make http proxy port configurable in cxml, avoiding this: org.springframework.beans.NotWritablePropertyException: Invalid property 'httpProxyPort' of bean class [org.archive.modules.fetcher.FetchHTTP]: Bean property 'httpProxyPort' is not writable or has an invalid setter method. Did you mean 'httpProxyHost'? 2014-01-22 16:01:21 -08:00
Kenji Nagahashi fafcc1e28b Merge pull request #30 from nlevitt/fix-bdb-cookie-store 
Fix bdb cookie store (cookie is piling up due to lack of proper equality method in Cookie)
 2014-01-21 12:01:10 -08:00
Noah Levitt 3ba476c39b restore useful comment 2014-01-20 20:05:12 -08:00
Noah Levitt cba663a107 fix BdbCookieStore so that cookies get replaced correctly! and test it 2014-01-20 20:02:56 -08:00
Noah Levitt 9a7e868022 BdbCookieStoreTest.java partly done 2014-01-20 09:13:55 -08:00
Roger G. Coram 4b8567dca2 Merge pull request #29 from kris-sigur/HER-2062 
HER-2062 Fix for WorkQueueFrontier.deleteURIs handling of retired queues
 2014-01-16 02:48:27 -08:00
Noah Levitt e36b9e74e4 remove debug logging 2014-01-15 13:41:36 -08:00
Kristinn Sigurðsson df72bc8afc Minor tidying up 2014-01-15 12:51:35 +00:00
Kristinn Sigurðsson bc77e82da5 Fix for WorkQueueFrontier.deleteURIs mishandles deletions from retired 
queues (HER-2062)
 2014-01-15 12:49:41 +00:00
Kenji Nagahashi 07293fbfb2 Merge pull request #28 from nlevitt/hc43 
switch to httpcomponents, get rid of archive-overlay-commons-httpclient
 2014-01-14 14:58:36 -08:00
Noah Levitt 420452195c pull in stock commons-httpclient dependency through ia-web-commons instead of specifying it explicitly 2014-01-14 14:52:24 -08:00
Noah Levitt 5930561042 fix contrib modules that had been using CrawlURI.getHttpMethod() 2014-01-14 14:27:34 -08:00
Noah Levitt c50393fe53 remove some cruft from testing 2014-01-13 21:29:23 -08:00
Noah Levitt 9f88da1735 add javadoc explaining newfound absence of getHttpMethod() 2014-01-13 20:51:08 -08:00
Noah Levitt 2f137dab50 remove obsolete httpcomponents.diff 2014-01-13 12:38:42 -08:00
Noah Levitt a86efb419f bump httpcomponents httpclient dependency to latest 4.3.1 2014-01-12 13:24:48 -08:00
Noah Levitt 16fa2b25f1 selftest-crawler-beans.cxml - replace obsolete BdbCookieStorage with BdbCookieStore 2014-01-12 13:11:25 -08:00
Noah Levitt 41ffae253e mark beginning of content at beginning of test payload, now that Recorder tries to auto-detect 2014-01-12 12:42:46 -08:00
Noah Levitt 3a429ffa44 Merge branch 'master' into hc43 
Conflicts:
	commons/pom.xml
	commons/src/main/java/org/archive/io/RecordingInputStream.java
	commons/src/main/java/org/archive/io/RecordingOutputStream.java
	commons/src/main/java/org/archive/io/ReplayInputStream.java
	dist/pom.xml
	engine/pom.xml
	modules/pom.xml
	modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
	modules/src/main/java/org/archive/modules/recrawl/FetchHistoryProcessor.java
	pom.xml
 2014-01-10 17:52:03 -08:00
Noah Levitt 6bdfec7e7f Merge branch 'master' of github.com:internetarchive/heritrix3 2014-01-10 16:22:51 -08:00
Noah Levitt 749bd6a214 bump version to 3.3.0-SNAPSHOT after 3.2.0 release 2014-01-10 16:22:43 -08:00
Noah Levitt 78a6bea665 Merge pull request #27 from nlevitt/release-3.2.0 
rename dist/README.md to dist/README.txt so that maven bundles it in the...
 2014-01-10 14:16:07 -08:00
Noah Levitt 563c5b034d rename dist/README.md to dist/README.txt so that maven bundles it in the distribution 2014-01-10 14:13:32 -08:00
Noah Levitt 484b3574d7 Merge pull request #26 from nlevitt/release-3.2.0 
update readme for 3.2.0 release
3.2.0 		2014-01-10 13:17:46 -08:00
Noah Levitt 0467304f76 update readme for 3.2.0 release 2014-01-10 13:15:48 -08:00
Eldon 752e288ef8 Merge pull request #25 from nlevitt/release-3.2.0 
bump version number to 3.2.0 for release
 2014-01-10 12:47:58 -08:00
Noah Levitt 32c083f720 bump version number to 3.2.0 for release 2014-01-10 12:36:45 -08:00
Adam Miller 9ac0ee057f Merge pull request #24 from nlevitt/url-agnostic-standard 
for url-agnostic dedup, follow "Proposal for Standardizing the Recording...
 2014-01-08 11:38:51 -08:00
Noah Levitt 84e78db40a for url-agnostic dedup, follow "Proposal for Standardizing the Recording of Arbitrary Duplicates in WARC Files" https://docs.google.com/document/d/1QyQBA7Ykgxie75V8Jziz_O7hbhwf7PF6_u9O6w6zgp0 2014-01-07 14:43:26 -08:00
ikreymer 506e68316a Merge pull request #23 from nlevitt/her-1979 
fix HER-1979 so heritrix can run on windows xp
 2013-12-18 01:12:16 -08:00
Noah Levitt 7dd67bd876 fix HER-1979 so heritrix can run on windows xp 2013-12-17 19:25:12 -08:00
Noah Levitt 2388a35a06 useProjectArtifact=false to avoid [WARNING] Cannot include project artifact: ... it doesn't have an associated file or directory 2013-12-15 11:54:36 -08:00
Noah Levitt 0ea10665f1 abbreviate maven repository clauses, reduce redundancy, and let default settings prevail, such as updatePolicy=daily, pointed out by Erik Hetzner 2013-12-15 11:53:18 -08:00
Noah Levitt 6f171b575a fix report links 2013-12-11 21:14:04 -08:00
Noah Levitt 3b06c88537 test fix (in ia-web-commons) for HER-2059 2013-12-06 19:22:16 -08:00
Noah Levitt 3c8a0a389f moving a bunch of stuff from heritrix-commons to ia-web-commons so that wayback doesn't have to depend on heritrix-commons 2013-12-05 17:48:57 -08:00
Noah Levitt 98d0901531 moving org.archive.net.PublicSuffixes to ia-web-commons 2013-12-05 16:13:30 -08:00
Noah Levitt 552794dc89 don't let the exception from commit df2a313797 derail the whole crawl job 2013-11-26 16:08:38 -08:00
Noah Levitt 1163847644 try to avoid nearly endless loop of "java.lang.IllegalStateException: Can't open a cursor Database was closed." when startup fails and dumpPendingToLog is attempted 2013-11-21 18:51:16 -08:00
Noah Levitt 8e1125d062 bump version to 3.2.0-SNAPSHOT since there are some incompatible api changes since 3.1.1 (requested by kris sigurðsson) 2013-11-20 09:18:31 -08:00
Noah Levitt d93b5449be new CrawlURI annotation "duplicate:digest" to simplify handling dups in stats, etc 2013-11-19 11:05:14 -08:00
Noah Levitt 7089caf3c4 remove unused engine.css 2013-11-15 19:11:19 -08:00
Noah Levitt 119e565acc Adam's responsive UI 2013-11-15 18:59:14 -08:00