lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-07-29 02:41:12 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,339 Commits 40 Branches 31 Tags
b803ea0ec03b4e9d2b6a7bdaead8ccfc4ae760f9
Commit Graph
537 Commits
Author SHA1 Message Date
Noah Levitt b803ea0ec0 whoops! send escaped path+query on http request line; had been sending raw spaces (includes unit test) 2014-01-27 16:57:37 -08:00
Noah Levitt 26a5dfebdc fix NullPointerException in case of 401 with no auth challenge (includes unit test) 2014-01-27 11:47:37 -08:00
Noah Levitt 6b72ac9705 Switch to BasicHttpClientConnectionManager instead of 
PoolingHttpClientConnectionManager, and make the connection manager a member
variable of HttpRequest. We had been seeing a strange bug where once in
every few thousand urls or so, the connection manager would get gc'd
while the http request was being sent, and RecordingOutputStream.flush()
would get called, as in the stack trace below.  The ToeThread was using
it at the same time, so if conditions were right, the recorded stream
would get mangled. This commit makes the connection manager a member of
FetchHTTPRequest so it doesn't get gc'd prematurely. Also switches to
BasicHttpClientConnectionManager, since we're not doing any pooling.

  it.unimi.dsi.fastutil.io.FastBufferedOutputStream.dumpBuffer(boolean) line: 133
  it.unimi.dsi.fastutil.io.FastBufferedOutputStream.flush() line: 166
  org.archive.io.RecordingOutputStream.flush() line: 469
  org.apache.http.impl.io.SessionOutputBufferImpl.flushStream() line: 128
  org.apache.http.impl.io.SessionOutputBufferImpl.flush() line: 143
  org.archive.modules.fetcher.FetchHTTPRequest$RecordingHttpClientConnection(org.apache.http.impl.BHttpConnectionBase).close() line: 314
  org.archive.modules.fetcher.FetchHTTPRequest$RecordingHttpClientConnection.close() line: 520
  org.apache.http.impl.conn.CPoolEntry.closeConnection() line: 69
  org.apache.http.impl.conn.CPoolEntry.close() line: 95
  org.apache.http.impl.conn.CPool(org.apache.http.pool.AbstractConnPool<T,C,E>).shutdown() line: 130
  org.apache.http.impl.conn.PoolingHttpClientConnectionManager.shutdown() line: 347
  org.apache.http.impl.conn.PoolingHttpClientConnectionManager.finalize() line: 168
  java.lang.ref.Finalizer.invokeFinalizeMethod(java.lang.Object) line: not available [native method]
 2014-01-24 16:07:04 -08:00
Noah Levitt c5c8788227 make http proxy port configurable in cxml, avoiding this: org.springframework.beans.NotWritablePropertyException: Invalid property 'httpProxyPort' of bean class [org.archive.modules.fetcher.FetchHTTP]: Bean property 'httpProxyPort' is not writable or has an invalid setter method. Did you mean 'httpProxyHost'? 2014-01-22 16:01:21 -08:00
Noah Levitt 3ba476c39b restore useful comment 2014-01-20 20:05:12 -08:00
Noah Levitt cba663a107 fix BdbCookieStore so that cookies get replaced correctly! and test it 2014-01-20 20:02:56 -08:00
Noah Levitt 9a7e868022 BdbCookieStoreTest.java partly done 2014-01-20 09:13:55 -08:00
Noah Levitt e36b9e74e4 remove debug logging 2014-01-15 13:41:36 -08:00
Noah Levitt 5930561042 fix contrib modules that had been using CrawlURI.getHttpMethod() 2014-01-14 14:27:34 -08:00
Noah Levitt c50393fe53 remove some cruft from testing 2014-01-13 21:29:23 -08:00
Noah Levitt 9f88da1735 add javadoc explaining newfound absence of getHttpMethod() 2014-01-13 20:51:08 -08:00
Noah Levitt 41ffae253e mark beginning of content at beginning of test payload, now that Recorder tries to auto-detect 2014-01-12 12:42:46 -08:00
Noah Levitt 3a429ffa44 Merge branch 'master' into hc43 
Conflicts:
	commons/pom.xml
	commons/src/main/java/org/archive/io/RecordingInputStream.java
	commons/src/main/java/org/archive/io/RecordingOutputStream.java
	commons/src/main/java/org/archive/io/ReplayInputStream.java
	dist/pom.xml
	engine/pom.xml
	modules/pom.xml
	modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
	modules/src/main/java/org/archive/modules/recrawl/FetchHistoryProcessor.java
	pom.xml
 2014-01-10 17:52:03 -08:00
Noah Levitt 84e78db40a for url-agnostic dedup, follow "Proposal for Standardizing the Recording of Arbitrary Duplicates in WARC Files" https://docs.google.com/document/d/1QyQBA7Ykgxie75V8Jziz_O7hbhwf7PF6_u9O6w6zgp0 2014-01-07 14:43:26 -08:00
Noah Levitt 3b06c88537 test fix (in ia-web-commons) for HER-2059 2013-12-06 19:22:16 -08:00
Noah Levitt d93b5449be new CrawlURI annotation "duplicate:digest" to simplify handling dups in stats, etc 2013-11-19 11:05:14 -08:00
Noah Levitt 2b87219496 oops, avoid npe 2013-11-15 17:59:34 -08:00
Noah Levitt df2a313797 throw exception when host cache returns 0.0.0.0 for any host (trying to track down a strange problem) 2013-11-15 17:20:55 -08:00
Noah Levitt b137022b79 Include in checkpoint data a list of ".open" WARCs and their current raw on-disk length, so the exact extent of pre-checkpoint content can be deduced/reconstructed. Also when flushing or closing writers, block to wait on any that remain active. (In context of heritrix crawl this is not normally necessary because of the dispositionInProgressLock managed by the frontier, but seems like a good idea in case WriterPool is used elsewhere.) 2013-10-28 17:35:55 -07:00
Noah Levitt 38a12b2008 avoid NPE resuming from checkpoint if "stats" is {} in warcWriter checkpoint json 2013-10-11 14:44:34 -07:00
Noah Levitt a1f84d3cc4 HER-2056 new option startNewFilesOnCheckpoint on writers, true by default for backward compatibility 2013-10-10 17:57:18 -07:00
Noah Levitt 071cbe99cd include urlsWritten in checkpoint, and be backward compatible with old checkpoints 2013-09-07 23:40:17 -07:00
Noah Levitt c49b465d82 custom kryo serialization for Inet4Address in CrawlHost since default doesn't work right, and junit test 2013-09-07 19:40:11 -07:00
Noah Levitt 52d72e661d part of HER-2048 - WARCWriterProcessor stats resume from checkpoint 2013-09-07 11:35:03 -07:00
Noah Levitt e0f78976d7 part of HER-2048 - restore BdbServerCache from checkpoint 2013-09-07 10:53:22 -07:00
Noah Levitt 25d4de186f don't log warning in normal case 2013-09-06 22:04:56 -07:00
Noah Levitt 418c7138a9 handle case where there is no whois server for a domain 2013-08-28 18:13:53 -07:00
Noah Levitt 4abfcb01b0 HER-1895 support HTTP Refresh header 2013-08-26 18:33:02 -07:00
Noah Levitt 6d451d8285 Merge branch 'master' into hc43 2013-08-09 19:15:14 -07:00
Noah Levitt 68adaa70fd need to close the recorder here so it can calculate the content length properly 2013-08-09 19:14:58 -07:00
Noah Levitt a2977490c4 update for httpclient 4.3-beta3-SNAPSHOT 2013-08-09 18:33:15 -07:00
Noah Levitt 317ff7eb65 Merge branch 'master' into hc43 2013-08-09 18:17:24 -07:00
Noah Levitt 705a375daf uses of UriUtils.isLikelyUri() in Extractor{HTML,SWF,XML} with UriUtils.isVeryLikelyUri() to reap the benefits of HER-1523 improvements (should address archive-it issue ARI-3492) 2013-08-09 18:17:11 -07:00
Noah Levitt 750a688670 Merge branch 'master' into hc43 
Conflicts:
	commons/src/main/java/org/apache/commons/httpclient/HttpParser.java
	engine/pom.xml
	modules/src/main/java/org/archive/modules/CrawlURI.java
	modules/src/main/java/org/archive/modules/deciderules/MatchesStatusCodeDecideRule.java
	modules/src/main/java/org/archive/modules/deciderules/NotMatchesStatusCodeDecideRule.java
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java
 2013-08-08 21:14:11 -07:00
Noah Levitt 8c64cba9ae TransclusionDecideRule.java - consider form submission "S" hop as navlink like "L", not transcluded 2013-08-05 16:05:02 -07:00
Noah Levitt f4f3b9be38 add logging to BdbUriUniqFilter.forgetSchemeHost(), clean up some javadocs 2013-07-26 17:50:36 -07:00
Noah Levitt 72094213a4 MirrorWriterProcessor.java - change "path" setting to a ConfigPath and set default value "${launchId}/mirror" 2013-07-22 17:54:06 -07:00
Adam Miller b85828135b Convert MatchesStatusCodeDecideRule and NotMatchesStatusCodeDecideRule to use CrawlURI.getFetchStatus() to avoid null status codes and broaden use. 
Added test cases for MatchesStatusCodeDecideRule and NotMatchesStatusCodeDecideRule
Fix comments for youtube extractor.
 2013-06-21 12:03:41 -07:00
Noah Levitt f2984881f6 address HER-2042 - new annotation duplicate:uriAgnosticDigest for all url agnostic duplicates, whether written to warcs or not; stats consult this annotation so that duplicate/novel numbers are correct 2013-06-07 17:17:24 -07:00
Noah Levitt ffd248f780 HER-2040 ExtractorJS shouldExtract when content-type is application/json 2013-06-03 11:23:16 -07:00
Noah Levitt d4c5bd6e98 avoid npe by setting default <input> type "text" when unspecified 2013-05-14 18:17:09 -07:00
Noah Levitt d972731958 ExtractorHTMLForms.java - handle a corner case extracting attribute values 2013-05-14 16:23:41 -07:00
Noah Levitt 7bd010e3c7 HTMLForm.java - default html <input> type is text, so consider <input> with no type as candidateUsernameInput 2013-05-13 22:53:37 -07:00
Noah Levitt bea5c6f267 followup on HER-2031 ExtractorHTMLForms.java - improve regexes looking for form and input attributes, mainly to avoid maxing cpu for long periods of time for certain input 2013-05-13 22:37:02 -07:00
Noah Levitt ec824e9724 avoid need for getRawData() method by introducing Link.hasData() 2013-05-13 17:43:45 -07:00
Kristinn Sigurðsson 7c5b2ceb00 Added a data map to Link. Becomes the CrawlURI's data map when Link is 
promoted to CrawlURI
 2013-05-13 15:18:02 +00:00
Noah Levitt d5693563f8 fix bug pointed out in https://webarchive.jira.com/browse/ARI-3176?focusedCommentId=35314 - url-agnostic dedupe not reported in host report (and other reports) 2013-04-29 18:11:24 -07:00
Noah Levitt 6a0ed067c0 consolidate disparate versions of WARCConstants into one version that lives in ia-web-commons 2013-03-29 17:12:32 -07:00
Kenji Nagahashi a4a3ce60cd fix failing ContentDigestHistoryTest 2013-03-26 15:06:31 -07:00
Noah Levitt 5d8fe0443d Merge branch 'master' into HER-2031 2013-03-23 19:16:05 -07:00