lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-07-26 09:21:43 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,515 Commits 40 Branches 31 Tags
8edebc09439254c8eaee0396d166ce6562a10f4e
Commit Graph
627 Commits
Author SHA1 Message Date
Kristinn Sigurðsson 8edebc0943 Fix for HER-2071 2014-07-25 13:32:08 +00:00
Noah Levitt cb6ac8b8b5 restore writeRevisitFor* settings, but issue warning in setter and ignore the value 2014-07-14 10:53:37 -07:00
Noah Levitt da30439d61 oops, fix 304 tests 2014-07-11 12:47:15 -07:00
Noah Levitt 77492802b3 test 304 not modified revist profile in FetchHTTPTest 2014-07-10 21:30:54 -07:00
Noah Levitt 2292ea9287 remove deprecated unused stuff 2014-07-10 21:29:47 -07:00
Noah Levitt 810ad91b9d set revisit profile on 304 not modified 2014-07-10 21:28:52 -07:00
Noah Levitt 905a98c0c2 use RevisitProfile for tallying stats 2014-07-10 21:27:50 -07:00
Noah Levitt d9fd66878b Merge branch 'master' into revisit-redux 
Conflicts:
	modules/src/main/java/org/archive/modules/CrawlURI.java
	modules/src/main/java/org/archive/modules/writer/WARCWriterProcessor.java
 2014-07-10 15:01:43 -07:00
Noah Levitt ee52f48be9 remove LinksScoper (deprecated since 2009) and remove recently added code only used in LinksScoper 2014-07-09 10:14:24 -07:00
Noah Levitt bed1e58adf copy over and adjust equals(), hashCode(), compareTo() from old Link class to CrawlURI, so that tests pass 2014-07-07 16:59:18 -07:00
Noah Levitt bfac3c8bbc ensure non-null pathFromSeed to fix some unit tests 2014-07-07 12:25:27 -07:00
Noah Levitt 3c2e1315bb other changes required for merge from master into HER-2039 2014-07-07 11:48:36 -07:00
Noah Levitt f23a59a918 Merge branch 'master' into HER-2039 
Conflicts:
	modules/src/main/java/org/archive/modules/CrawlURI.java
	modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
	modules/src/main/java/org/archive/modules/extractor/ExtractorSWF.java
	modules/src/main/java/org/archive/modules/extractor/Link.java
	modules/src/main/java/org/archive/modules/writer/WARCWriterProcessor.java
 2014-07-07 11:45:27 -07:00
Noah Levitt f2055ee2ae in CrawlURI.createCrawlURI(), avoid clobbering inherited data with data from Link 2014-07-02 13:17:42 -07:00
Hunter Stern 5eade58de4 Fix for https://webarchive.jira.com/browse/ARI-3943 2014-06-27 16:20:42 -07:00
Kristinn Sigurðsson ece0149339 Fix for HER-2068 2014-06-04 14:59:13 +00:00
Noah Levitt 67a0c59758 add A_ANNOTATIONS to persistentKeys so that CrawlURI doesn't lose its annotations when deferred 2014-05-22 16:48:01 -07:00
Noah Levitt aff5b9ac4e for ARI-3712, add extracted links relative to both via and base, and annotate with "extractorSWFRelToVia", "extractorSWFRelToBase", or "extractorSWFRelToBoth" if resulting link is the same whether relative to base or via 2014-05-15 13:31:29 -07:00
Noah Levitt 519717fd7b make addRelativeToBase() and addRelativeToVia() return the link added, or null; copy getAnnotations() from CrawlURI 2014-05-15 13:27:50 -07:00
Adam Miller ae878d29cf Code cleanup 2014-05-01 15:22:16 -07:00
Adam Miller f2bebd41b0 Cleanup / Optimization 2014-05-01 12:48:23 -07:00
Adam MillerandAdam Miller 753404f662 Adding ViaSurtPrefixedDecideRule. Returns true if a surtform 'via' matches any surt in a specified list 2014-05-01 12:43:26 -07:00
Noah Levitt e73810d0a6 fix unit test now that we accept speculative urls with query params with no value 2014-04-10 17:25:27 -07:00
Noah Levitt 36a207ce07 fix FormLoginProcessor checkpointing 2014-04-02 17:33:09 -07:00
Noah Levitt 878fd9f7a9 oops, update test to expect post data as url-encoded query string 2014-04-01 15:18:17 -07:00
Noah Levitt 552200b2fa fix form login processor by saving login post data as url-encoded query string, which FetchHTTP now expects; tweaks to make FormLoginSelfTest pass 2014-03-31 20:38:17 -07:00
Noah Levitt a82886b747 restore support for sending http post data, add unit test 2014-03-31 18:09:23 -07:00
Noah Levitt 23e8d41651 avoid NPE; root cause was assumption that LoadingCache.asMap().get() would call CacheLoader.load() but it doesnt, see https://code.google.com/p/guava-libraries/wiki/MapMakerMigration 2014-03-31 13:40:29 -07:00
Noah Levitt 26a577ff6e basic unit tests for FormLoginProcessor 2014-03-31 13:38:39 -07:00
Noah Levitt 0d7f633fdb pull methods getRecorder() and makeCrawlURI(String) into ModuleTestBase, and move ModuleTestBase into heritrix-modules so that CrawlURI is available 2014-03-31 13:36:44 -07:00
Shriphani Palakodety 84f29e4e10 build with the right key and value types 2014-03-18 06:42:03 -04:00
Shriphani Palakodety 2f60b5f935 migration to latest guava complete 2014-03-18 06:42:02 -04:00
Noah Levitt 6a92e88b1c Merge pull request #40 from internetarchive/her-2064 
For https://webarchive.jira.com/browse/HER-2064
 2014-02-28 16:48:42 -08:00
Noah Levitt cf775bec94 remove extraneous code related to logging removed in last commit 2014-02-21 15:46:52 -08:00
Noah Levitt fd806a514d remove extra logging from performance test 2014-02-21 13:28:13 -08:00
Noah Levitt 39f06a1f1c fix slowness caused by BdbCookieStore by maintaining cookie list in memory parallel to the one stored on disk; add tests for consistency and performance under load 2014-02-21 13:26:58 -08:00
Hunter Stern d6e61b83da For https://webarchive.jira.com/browse/HER-2064 2014-02-13 16:59:05 -08:00
Hunter Stern 06ba50c410 Reverting previous commit: https://github.com/internetarchive/heritrix3/commit/164d0fbdc5fa32cd90edd9f1b4091ed876552232 2014-02-13 16:50:21 -08:00
Hunter Stern 164d0fbdc5 For https://webarchive.jira.com/browse/HER-2064 2014-02-13 16:42:58 -08:00
Noah Levitt af44522fdf FetchHTTPRequest - include CrawlURI.getData().get("customHttpRequestHeaders"), if any, in http request 2014-02-03 18:41:23 -08:00
Noah Levitt edd62f3de0 fix HER-2063 - omit port in Host request header when it is default for the scheme 2014-02-03 17:18:40 -08:00
Kristinn Sigurðsson cc18521759 Merge remote-tracking branch 'internetarchive/master' into revisit-redux 2014-01-30 15:06:01 +00:00
Kristinn Sigurðsson 701457ed4b ContentDigestHistoryLoader now sets the revisit profile on the CrawlURI 
after determining that it is a revisit.
 2014-01-30 15:03:48 +00:00
Kristinn Sigurðsson 2905cc556c Revisit headers now come correctly formated from the revisit profile. 2014-01-30 15:02:40 +00:00
Kristinn Sigurðsson 685f5d5d6a Store refersToDate in String format (as intended to write in WARC). 
Retain long setter for convenience.
 2014-01-30 15:01:41 +00:00
Noah Levitt 31aea014d9 also handle case where Charset.forName() throws IllegalCharsetNameException, as opposed to UnsupportedCharsetException 2014-01-28 16:36:51 -08:00
Noah Levitt 161b359979 Avoid the exception below by handling bad charsets in FetchHTTP. Restore annotation "unsatisfiableCharsetInHeader:...". Includes unit tests for good and bad charsets. 
SEVERE thread-5973 org.archive.crawler.framework.ToeThread.recoverableProblem() Problem java.nio.charset.UnsupportedCharsetException: binary occured when trying to process 'http://example.com/' at step ABOUT_TO_BEGIN_PROCESSOR in
java.nio.charset.UnsupportedCharsetException: binary
        at java.nio.charset.Charset.forName(Charset.java:543)
        at org.apache.http.entity.ContentType.<init>(ContentType.java:111)
        at org.apache.http.entity.ContentType.create(ContentType.java:210)
        at org.apache.http.entity.ContentType.get(ContentType.java:258)
        at org.apache.http.entity.ContentType.getOrDefault(ContentType.java:277)
        at org.archive.modules.fetcher.FetchHTTP.setCharacterEncoding(FetchHTTP.java:580)
        at org.archive.modules.fetcher.FetchHTTP.innerProcess(FetchHTTP.java:693)
        at org.archive.modules.Processor.innerProcessResult(Processor.java:175)
        at org.archive.modules.Processor.process(Processor.java:142)
        at org.archive.modules.ProcessorChain.process(ProcessorChain.java:131)
        at org.archive.crawler.framework.ToeThread.run(ToeThread.java:148)
 2014-01-28 16:19:44 -08:00
Noah Levitt b803ea0ec0 whoops! send escaped path+query on http request line; had been sending raw spaces (includes unit test) 2014-01-27 16:57:37 -08:00
Noah Levitt 26a5dfebdc fix NullPointerException in case of 401 with no auth challenge (includes unit test) 2014-01-27 11:47:37 -08:00
Noah Levitt 6b72ac9705 Switch to BasicHttpClientConnectionManager instead of 
PoolingHttpClientConnectionManager, and make the connection manager a member
variable of HttpRequest. We had been seeing a strange bug where once in
every few thousand urls or so, the connection manager would get gc'd
while the http request was being sent, and RecordingOutputStream.flush()
would get called, as in the stack trace below.  The ToeThread was using
it at the same time, so if conditions were right, the recorded stream
would get mangled. This commit makes the connection manager a member of
FetchHTTPRequest so it doesn't get gc'd prematurely. Also switches to
BasicHttpClientConnectionManager, since we're not doing any pooling.

  it.unimi.dsi.fastutil.io.FastBufferedOutputStream.dumpBuffer(boolean) line: 133
  it.unimi.dsi.fastutil.io.FastBufferedOutputStream.flush() line: 166
  org.archive.io.RecordingOutputStream.flush() line: 469
  org.apache.http.impl.io.SessionOutputBufferImpl.flushStream() line: 128
  org.apache.http.impl.io.SessionOutputBufferImpl.flush() line: 143
  org.archive.modules.fetcher.FetchHTTPRequest$RecordingHttpClientConnection(org.apache.http.impl.BHttpConnectionBase).close() line: 314
  org.archive.modules.fetcher.FetchHTTPRequest$RecordingHttpClientConnection.close() line: 520
  org.apache.http.impl.conn.CPoolEntry.closeConnection() line: 69
  org.apache.http.impl.conn.CPoolEntry.close() line: 95
  org.apache.http.impl.conn.CPool(org.apache.http.pool.AbstractConnPool<T,C,E>).shutdown() line: 130
  org.apache.http.impl.conn.PoolingHttpClientConnectionManager.shutdown() line: 347
  org.apache.http.impl.conn.PoolingHttpClientConnectionManager.finalize() line: 168
  java.lang.ref.Finalizer.invokeFinalizeMethod(java.lang.Object) line: not available [native method]
 2014-01-24 16:07:04 -08:00