lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-22 22:46:43 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,837 Commits 40 Branches 31 Tags
a4a841c826be290ea0cb2740c03bbcdfeab71b3f
Commit Graph
1837 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Andrew Jackson a4a841c826 Removing comment from old version of this code. 2016-10-11 20:43:32 +01:00
Andrew Jackson b3dff8d9d6 Defaulting the message prefetch count to 1000 instead of infinity. 2016-10-11 20:42:52 +01:00
Andrew Jackson 4d7a64d127 Reverted Kryo version change. 2016-10-11 20:41:20 +01:00
Andrew Jackson b3e1a83034 Merge branch 'amqp-fine-control' of github.com:ukwa/heritrix3 into amqp-fine-control 2016-10-03 16:52:48 +01:00
Andrew Jackson 0e40869f83 Fix package name. 2016-10-03 16:51:28 +01:00
Andrew Jackson fdae4b6297 Update Kryo to latest version. 2016-10-03 16:51:28 +01:00
Andrew Jackson eaec682dd4 Bump incoming links to MEDIUM to compete with e.g. runs of redirects. 2016-10-03 16:51:28 +01:00
Andrew Jackson c6c396af8b Use sensible defaults for normal links. 2016-10-03 16:51:28 +01:00
Andrew Jackson 553eae2ddd Add support for setting the hop and the prefetch count. 2016-10-03 16:51:28 +01:00
Noah Levitt fbb0173459 Merge pull request #166 from internetarchive/amqpWaiterStarter 
Don't wait to receive Umbra urls if Heritrix sends no url to Umbra
 2016-07-21 18:05:10 -05:00
Barbara Miller cbc73a6e8d check controller state, not frontier.isEmpty) 2016-07-21 11:40:58 -07:00
Barbara Miller 40d8ee77bf don't wait if h3 sends no url 2016-07-13 14:32:07 -07:00
Noah Levitt af2530edc4 Merge pull request #165 from galgeek/amqpWaiter 
AMQP URL Waiter
 2016-07-07 19:35:04 -05:00
Barbara Miller 4485b8c8c7 AMQP URL Waiter 2016-07-01 15:09:50 -07:00
Noah Levitt c04162a0f7 Merge pull request #164 from nlevitt/fix-test-errors 
Fixes for apparent build errors (extends #154)
 2016-06-08 14:31:58 -05:00
Noah Levitt 36ea1a9fb2 try to fix remaining serialization tests in oraclejdk8 by using ConcurrentSkipListMap instead of ConcurrentHashMap 2016-06-07 17:55:45 -05:00
Noah Levitt 99d87e8f23 hopefully fix serialization tests in oraclejdk8 by using TreeSet instead of HashSet in KeyedProperties.java 2016-06-07 14:53:11 -05:00
Noah Levitt cf9498d2ed clear the history store at the beginning of testBasics(), because the other test might have run first 2016-06-07 14:12:20 -05:00
Noah Levitt b09b078075 yeesh... "cd .." to get back to the right place to see the failure reports 2016-06-07 13:36:45 -05:00
Noah Levitt bc8b0baa59 Merge remote-tracking branch 'origin/master' into fix-test-errors 
* origin/master:
  Setup TravisCI
  Set fetch status on curis when testing link extraction
  No link extraction on URI not successfully downloaded
  Fixes issue #158 : Noisy alerts about 401s without auth challenge (#159)
  Make Content-Location header url INFERRED not REFFER hop type since Content-Location is not for redirection (#151)
  fixes for kafka 0.9 (?)
  upgrade to kafka 0.9
  somewhat ugly fix to handle exceptions from the bean browser like java.lang.RuntimeException: not implemented at org.archive.modules.fetcher.BdbCookieStore$RestrictedCollectionWrappedList.get(BdbCookieStore.java:92)
 2016-06-07 13:09:51 -05:00
Noah Levitt ac41da69c2 let me see what failed, travis 2016-06-07 13:07:24 -05:00
Noah Levitt ca3b65ed03 exclude dns from stats tests because the size of the dns records can vary depending on the environment 2016-06-06 15:43:02 -05:00
Noah Levitt f7d1ad7f1f Merge pull request #111 from nlevitt/bean-browse-cookiestore-exception 
somewhat ugly fix to handle exceptions from the bean browser like java.l...
 2016-06-06 11:11:45 -07:00
Noah Levitt bb10b4a80c Merge pull request #161 from kris-sigur/NoNonSuccessLinkExtraction 
No link extraction on URI not successfully downloaded
 2016-06-06 11:11:06 -07:00
Andrew Jackson aba94c7ac1 Fix package name. 2016-05-23 12:11:15 +01:00
Andrew Jackson d349c19298 Update Kryo to latest version. 2016-05-23 11:00:59 +01:00
jkafader c80f0cee2b Merge pull request #163 from nlevitt/kafka-0.9 
Kafka 0.9
 2016-05-09 12:01:53 -07:00
nruest ecd27e9e65 Setup TravisCI 2016-05-03 10:56:24 -07:00
Kristinn Sigurðsson 70867efdac Set fetch status on curis when testing link extraction 2016-05-03 15:58:50 +00:00
Kristinn Sigurðsson 08e3d055cf No link extraction on URI not successfully downloaded 2016-05-03 14:40:24 +00:00
Kristinn Sigurðsson c5848ae803 Fixes issue #158 : Noisy alerts about 401s without auth challenge (#159) 
* Fixes issue #158 : Noisy alerts about 401s without auth challenge

* Update test to account for non-fatal-error log not being empty on
non-auth 401s.
 2016-05-02 10:15:32 -07:00
Andy Jackson 8437a2724c Merge pull request #3 from ruebot/fix-test-errors 
Setup TravisCI
 2016-04-14 17:54:37 +00:00
nruest 3ee34d886f Setup TravisCI 2016-04-14 07:18:26 -04:00
Hunter a02a32cdfe Make Content-Location header url INFERRED not REFFER hop type since Content-Location is not for redirection (#151) 2016-04-13 15:19:21 -07:00
Andrew Jackson 48f5e71e76 Add delay to avoid race condition during test. 2016-04-07 21:48:20 +01:00
Andrew Jackson 07d610a1a7 Merge branch 'fix-test-errors' into amqp-fine-control 2016-04-02 20:58:43 +01:00
Andrew Jackson 9145fedd40 Fix unexpected change in count of novel bytes in one selftest. 2016-04-02 20:57:05 +01:00
Andrew Jackson cdb79b8868 Fix serialisation tests by defaulting to TreeMaps. 2016-04-02 20:35:37 +01:00
Andrew Jackson 621468107b Bump incoming links to MEDIUM to compete with e.g. runs of redirects. 2016-03-13 20:22:27 +00:00
Andrew Jackson 576b1d234b Use sensible defaults for normal links. 2016-03-10 14:41:40 +00:00
Andrew Jackson 1341a61a3c Merge branch 'master' into amqp-fine-control 2016-03-09 19:48:57 +00:00
Noah Levitt 124907341f Merge pull request #152 from ukwa/switch-to-java-7 
Switch to Java 7
 2016-03-08 16:47:58 -08:00
Andrew Jackson 8ff09dc769 Switch to Java 1.7 2016-03-08 23:53:59 +00:00
Andrew Jackson e1476401ac Merged with upstream changes. 2016-03-08 23:50:07 +00:00
Adam Miller 7f752d8d45 Merge pull request #150 from nlevitt/ari-4638 
various changes to amqp publish and receive
 2016-03-08 09:26:58 -08:00
Hunter 0dc95ab7f7 Merge pull request #144 from nlevitt/ari-4528 
run received urls through the candidates processor, to check scope an…
 2016-02-29 14:24:43 -08:00
Kristinn Sigurðsson ae413b1d9e Merge pull request #149 from eleclerc/patch-1 
Update to ExtractorHTML.java for cond. comments
 2016-02-19 09:49:34 +00:00
Noah Levitt 9bdafc3c33 remove stray line 2016-02-18 18:19:44 -08:00
Noah Levitt bc9a6cc640 new parameter extraInfo, "Arbitrary additional information to include in the json payload", on AMQPPublishProcessor; used KeyedProperties so parameters can be overridden with the sheet/overlay system 2016-02-18 18:17:30 -08:00
Noah Levitt f4b8468564 set custom http headers last, so they override headers set elsewhere (e.g. cookies) 2016-02-18 18:16:03 -08:00