Neil Minton
4d6969554c
Merge branch 'bitbaron/master' into ait-qa
2016-08-19 19:49:38 -07:00
Neil Minton
8c78297ded
Add annotation for reject decisions to write crawl information.
2016-08-19 19:47:47 -07:00
Neil Minton
2ee5db7f3b
Formating fixes in POM file.
2016-08-19 19:47:47 -07:00
Noah Levitt
fbb0173459
Merge pull request #166 from internetarchive/amqpWaiterStarter
...
Don't wait to receive Umbra urls if Heritrix sends no url to Umbra
2016-07-21 18:05:10 -05:00
Barbara Miller
b5fdd49ec8
check controller state, not frontier.isEmpty)
2016-07-21 12:51:16 -07:00
Barbara Miller
cbc73a6e8d
check controller state, not frontier.isEmpty)
2016-07-21 11:40:58 -07:00
Barbara Miller
63dda8bd8b
don't wait if h3 sends no url
2016-07-13 14:45:15 -07:00
Barbara Miller
40d8ee77bf
don't wait if h3 sends no url
2016-07-13 14:32:07 -07:00
Barbara Miller
0060adec29
minor merge tweaks
2016-07-12 22:45:03 -07:00
Noah Levitt
af2530edc4
Merge pull request #165 from galgeek/amqpWaiter
...
AMQP URL Waiter
2016-07-07 19:35:04 -05:00
Barbara Miller
4485b8c8c7
AMQP URL Waiter
2016-07-01 15:09:50 -07:00
Barbara Miller
264b32ab56
autowired afterall
2016-06-28 19:21:43 -07:00
Barbara Miller
1542f04e13
3rd time?
2016-06-28 19:01:42 -07:00
Barbara Miller
c85e2072ad
more better frontier checking
2016-06-28 18:14:34 -07:00
Barbara Miller
c90bea7b7c
better frontier checking
2016-06-28 17:39:13 -07:00
Barbara Miller
c8ece72d3c
tweak
2016-06-26 18:59:59 -07:00
Barbara Miller
9071e669ad
public AMQPUrlWaiter();
2016-06-26 18:27:20 -07:00
Barbara Miller
1f7ff71319
last cannot find symbols
2016-06-24 17:38:00 -07:00
Barbara Miller
5400e31594
better Event params
2016-06-24 17:14:37 -07:00
Barbara Miller
0d5853f72c
missing imports
2016-06-24 17:00:45 -07:00
Barbara Miller
6b6c689df1
move AMQPURLReceivedEvent.java under contrib
2016-06-24 16:00:03 -07:00
Barbara Miller
1d83d47b07
AMQP URL Waiter
2016-06-24 13:33:39 -07:00
Noah Levitt
c04162a0f7
Merge pull request #164 from nlevitt/fix-test-errors
...
Fixes for apparent build errors (extends #154 )
2016-06-08 14:31:58 -05:00
Noah Levitt
36ea1a9fb2
try to fix remaining serialization tests in oraclejdk8 by using ConcurrentSkipListMap instead of ConcurrentHashMap
2016-06-07 17:55:45 -05:00
Noah Levitt
99d87e8f23
hopefully fix serialization tests in oraclejdk8 by using TreeSet instead of HashSet in KeyedProperties.java
2016-06-07 14:53:11 -05:00
Noah Levitt
cf9498d2ed
clear the history store at the beginning of testBasics(), because the other test might have run first
2016-06-07 14:12:20 -05:00
Noah Levitt
b09b078075
yeesh... "cd .." to get back to the right place to see the failure reports
2016-06-07 13:36:45 -05:00
Noah Levitt
bc8b0baa59
Merge remote-tracking branch 'origin/master' into fix-test-errors
...
* origin/master:
Setup TravisCI
Set fetch status on curis when testing link extraction
No link extraction on URI not successfully downloaded
Fixes issue #158 : Noisy alerts about 401s without auth challenge (#159 )
Make Content-Location header url INFERRED not REFFER hop type since Content-Location is not for redirection (#151 )
fixes for kafka 0.9 (?)
upgrade to kafka 0.9
somewhat ugly fix to handle exceptions from the bean browser like java.lang.RuntimeException: not implemented at org.archive.modules.fetcher.BdbCookieStore$RestrictedCollectionWrappedList.get(BdbCookieStore.java:92)
2016-06-07 13:09:51 -05:00
Noah Levitt
ac41da69c2
let me see what failed, travis
2016-06-07 13:07:24 -05:00
Noah Levitt
ca3b65ed03
exclude dns from stats tests because the size of the dns records can vary depending on the environment
2016-06-06 15:43:02 -05:00
Noah Levitt
f7d1ad7f1f
Merge pull request #111 from nlevitt/bean-browse-cookiestore-exception
...
somewhat ugly fix to handle exceptions from the bean browser like java.l...
2016-06-06 11:11:45 -07:00
Noah Levitt
bb10b4a80c
Merge pull request #161 from kris-sigur/NoNonSuccessLinkExtraction
...
No link extraction on URI not successfully downloaded
2016-06-06 11:11:06 -07:00
jkafader
c80f0cee2b
Merge pull request #163 from nlevitt/kafka-0.9
...
Kafka 0.9
2016-05-09 12:01:53 -07:00
nruest
ecd27e9e65
Setup TravisCI
2016-05-03 10:56:24 -07:00
Kristinn Sigurðsson
70867efdac
Set fetch status on curis when testing link extraction
2016-05-03 15:58:50 +00:00
Kristinn Sigurðsson
08e3d055cf
No link extraction on URI not successfully downloaded
2016-05-03 14:40:24 +00:00
Kristinn Sigurðsson
c5848ae803
Fixes issue #158 : Noisy alerts about 401s without auth challenge ( #159 )
...
* Fixes issue #158 : Noisy alerts about 401s without auth challenge
* Update test to account for non-fatal-error log not being empty on
non-auth 401s.
2016-05-02 10:15:32 -07:00
Andy Jackson
8437a2724c
Merge pull request #3 from ruebot/fix-test-errors
...
Setup TravisCI
2016-04-14 17:54:37 +00:00
nruest
3ee34d886f
Setup TravisCI
2016-04-14 07:18:26 -04:00
Hunter
a02a32cdfe
Make Content-Location header url INFERRED not REFFER hop type since Content-Location is not for redirection ( #151 )
2016-04-13 15:19:21 -07:00
Andrew Jackson
48f5e71e76
Add delay to avoid race condition during test.
2016-04-07 21:48:20 +01:00
Andrew Jackson
9145fedd40
Fix unexpected change in count of novel bytes in one selftest.
2016-04-02 20:57:05 +01:00
Andrew Jackson
cdb79b8868
Fix serialisation tests by defaulting to TreeMaps.
2016-04-02 20:35:37 +01:00
Noah Levitt
124907341f
Merge pull request #152 from ukwa/switch-to-java-7
...
Switch to Java 7
2016-03-08 16:47:58 -08:00
Andrew Jackson
8ff09dc769
Switch to Java 1.7
2016-03-08 23:53:59 +00:00
Adam Miller
7f752d8d45
Merge pull request #150 from nlevitt/ari-4638
...
various changes to amqp publish and receive
2016-03-08 09:26:58 -08:00
Hunter
0dc95ab7f7
Merge pull request #144 from nlevitt/ari-4528
...
run received urls through the candidates processor, to check scope an…
2016-02-29 14:24:43 -08:00
Kristinn Sigurðsson
ae413b1d9e
Merge pull request #149 from eleclerc/patch-1
...
Update to ExtractorHTML.java for cond. comments
2016-02-19 09:49:34 +00:00
Noah Levitt
9bdafc3c33
remove stray line
2016-02-18 18:19:44 -08:00
Noah Levitt
bc9a6cc640
new parameter extraInfo, "Arbitrary additional information to include in the json payload", on AMQPPublishProcessor; used KeyedProperties so parameters can be overridden with the sheet/overlay system
2016-02-18 18:17:30 -08:00