Commit Graph
1837 Commits
Author SHA1 Message Date
Neil Minton 4d6969554c Merge branch 'bitbaron/master' into ait-qa 2016-08-19 19:49:38 -07:00
Neil Minton 8c78297ded Add annotation for reject decisions to write crawl information. 2016-08-19 19:47:47 -07:00
Neil Minton 2ee5db7f3b Formating fixes in POM file. 2016-08-19 19:47:47 -07:00
Noah Levitt fbb0173459 Merge pull request #166 from internetarchive/amqpWaiterStarter
Don't wait to receive Umbra urls if Heritrix sends no url to Umbra
2016-07-21 18:05:10 -05:00
Barbara Miller b5fdd49ec8 check controller state, not frontier.isEmpty) 2016-07-21 12:51:16 -07:00
Barbara Miller cbc73a6e8d check controller state, not frontier.isEmpty) 2016-07-21 11:40:58 -07:00
Barbara Miller 63dda8bd8b don't wait if h3 sends no url 2016-07-13 14:45:15 -07:00
Barbara Miller 40d8ee77bf don't wait if h3 sends no url 2016-07-13 14:32:07 -07:00
Barbara Miller 0060adec29 minor merge tweaks 2016-07-12 22:45:03 -07:00
Noah Levitt af2530edc4 Merge pull request #165 from galgeek/amqpWaiter
AMQP URL Waiter
2016-07-07 19:35:04 -05:00
Barbara Miller 4485b8c8c7 AMQP URL Waiter 2016-07-01 15:09:50 -07:00
Barbara Miller 264b32ab56 autowired afterall 2016-06-28 19:21:43 -07:00
Barbara Miller 1542f04e13 3rd time? 2016-06-28 19:01:42 -07:00
Barbara Miller c85e2072ad more better frontier checking 2016-06-28 18:14:34 -07:00
Barbara Miller c90bea7b7c better frontier checking 2016-06-28 17:39:13 -07:00
Barbara Miller c8ece72d3c tweak 2016-06-26 18:59:59 -07:00
Barbara Miller 9071e669ad public AMQPUrlWaiter(); 2016-06-26 18:27:20 -07:00
Barbara Miller 1f7ff71319 last cannot find symbols 2016-06-24 17:38:00 -07:00
Barbara Miller 5400e31594 better Event params 2016-06-24 17:14:37 -07:00
Barbara Miller 0d5853f72c missing imports 2016-06-24 17:00:45 -07:00
Barbara Miller 6b6c689df1 move AMQPURLReceivedEvent.java under contrib 2016-06-24 16:00:03 -07:00
Barbara Miller 1d83d47b07 AMQP URL Waiter 2016-06-24 13:33:39 -07:00
Noah Levitt c04162a0f7 Merge pull request #164 from nlevitt/fix-test-errors
Fixes for apparent build errors (extends #154)
2016-06-08 14:31:58 -05:00
Noah Levitt 36ea1a9fb2 try to fix remaining serialization tests in oraclejdk8 by using ConcurrentSkipListMap instead of ConcurrentHashMap 2016-06-07 17:55:45 -05:00
Noah Levitt 99d87e8f23 hopefully fix serialization tests in oraclejdk8 by using TreeSet instead of HashSet in KeyedProperties.java 2016-06-07 14:53:11 -05:00
Noah Levitt cf9498d2ed clear the history store at the beginning of testBasics(), because the other test might have run first 2016-06-07 14:12:20 -05:00
Noah Levitt b09b078075 yeesh... "cd .." to get back to the right place to see the failure reports 2016-06-07 13:36:45 -05:00
Noah Levitt bc8b0baa59 Merge remote-tracking branch 'origin/master' into fix-test-errors
* origin/master:
  Setup TravisCI
  Set fetch status on curis when testing link extraction
  No link extraction on URI not successfully downloaded
  Fixes issue #158 : Noisy alerts about 401s without auth challenge (#159)
  Make Content-Location header url INFERRED not REFFER hop type since Content-Location is not for redirection (#151)
  fixes for kafka 0.9 (?)
  upgrade to kafka 0.9
  somewhat ugly fix to handle exceptions from the bean browser like java.lang.RuntimeException: not implemented at org.archive.modules.fetcher.BdbCookieStore$RestrictedCollectionWrappedList.get(BdbCookieStore.java:92)
2016-06-07 13:09:51 -05:00
Noah Levitt ac41da69c2 let me see what failed, travis 2016-06-07 13:07:24 -05:00
Noah Levitt ca3b65ed03 exclude dns from stats tests because the size of the dns records can vary depending on the environment 2016-06-06 15:43:02 -05:00
Noah Levitt f7d1ad7f1f Merge pull request #111 from nlevitt/bean-browse-cookiestore-exception
somewhat ugly fix to handle exceptions from the bean browser like java.l...
2016-06-06 11:11:45 -07:00
Noah Levitt bb10b4a80c Merge pull request #161 from kris-sigur/NoNonSuccessLinkExtraction
No link extraction on URI not successfully downloaded
2016-06-06 11:11:06 -07:00
jkafader c80f0cee2b Merge pull request #163 from nlevitt/kafka-0.9
Kafka 0.9
2016-05-09 12:01:53 -07:00
nruest ecd27e9e65 Setup TravisCI 2016-05-03 10:56:24 -07:00
Kristinn Sigurðsson 70867efdac Set fetch status on curis when testing link extraction 2016-05-03 15:58:50 +00:00
Kristinn Sigurðsson 08e3d055cf No link extraction on URI not successfully downloaded 2016-05-03 14:40:24 +00:00
Kristinn Sigurðsson c5848ae803 Fixes issue #158 : Noisy alerts about 401s without auth challenge (#159)
* Fixes issue #158 : Noisy alerts about 401s without auth challenge

* Update test to account for non-fatal-error log not being empty on
non-auth 401s.
2016-05-02 10:15:32 -07:00
Andy Jackson 8437a2724c Merge pull request #3 from ruebot/fix-test-errors
Setup TravisCI
2016-04-14 17:54:37 +00:00
nruest 3ee34d886f Setup TravisCI 2016-04-14 07:18:26 -04:00
Hunter a02a32cdfe Make Content-Location header url INFERRED not REFFER hop type since Content-Location is not for redirection (#151) 2016-04-13 15:19:21 -07:00
Andrew Jackson 48f5e71e76 Add delay to avoid race condition during test. 2016-04-07 21:48:20 +01:00
Andrew Jackson 9145fedd40 Fix unexpected change in count of novel bytes in one selftest. 2016-04-02 20:57:05 +01:00
Andrew Jackson cdb79b8868 Fix serialisation tests by defaulting to TreeMaps. 2016-04-02 20:35:37 +01:00
Noah Levitt 124907341f Merge pull request #152 from ukwa/switch-to-java-7
Switch to Java 7
2016-03-08 16:47:58 -08:00
Andrew Jackson 8ff09dc769 Switch to Java 1.7 2016-03-08 23:53:59 +00:00
Adam Miller 7f752d8d45 Merge pull request #150 from nlevitt/ari-4638
various changes to amqp publish and receive
2016-03-08 09:26:58 -08:00
Hunter 0dc95ab7f7 Merge pull request #144 from nlevitt/ari-4528
run received urls through the candidates processor, to check scope an…
2016-02-29 14:24:43 -08:00
Kristinn Sigurðsson ae413b1d9e Merge pull request #149 from eleclerc/patch-1
Update to ExtractorHTML.java for cond. comments
2016-02-19 09:49:34 +00:00
Noah Levitt 9bdafc3c33 remove stray line 2016-02-18 18:19:44 -08:00
Noah Levitt bc9a6cc640 new parameter extraInfo, "Arbitrary additional information to include in the json payload", on AMQPPublishProcessor; used KeyedProperties so parameters can be overridden with the sheet/overlay system 2016-02-18 18:17:30 -08:00