Noah Levitt
fb039291ef
fix indentation (spaces only)
2014-03-16 18:14:29 -07:00
Noah Levitt
fb2be86e43
ExpressionDecideRule - evaluate groovy template with variable "curi" bound
2014-03-16 18:13:12 -07:00
Noah Levitt
5138a01240
warn if youtube video url is missing signature parameter
2014-03-06 16:08:07 -08:00
Noah Levitt
6e3bd2a2e8
update to handle latest changes from youtube
2014-03-05 19:05:31 -08:00
Noah Levitt
d1eeb43e40
Merge branch 'master' into umbra
2014-03-01 16:10:56 -08:00
Noah Levitt
e46c9ed759
Continue processing url after sending to amqp (fetch, etc). Solves 2 problems, 1) crawl finishing after sending off umbra seeds and before umbra has a chance to send anything back, and 2) seeds sent to umbra not being fetched because they were already seen. We may want to change this in the future as umbra gets fancier, e.g. sending back html from DOM, but for now it should work.
2014-03-01 16:10:41 -08:00
Adam Miller
7fa3809828
fixing formatting
2014-02-28 14:33:55 -08:00
Adam Miller
09c113163e
Adjusting youtube extraction itag priority
2014-02-28 14:15:49 -08:00
Noah Levitt
61539ab930
update ExtractorYoutubeFormatStreamTest.testDefaultItag()
2014-02-27 17:41:38 -08:00
Noah Levitt
8b124ecaa9
update youtube default video format priority (see https://webarchive.jira.com/browse/ARI-3730 )
2014-02-27 17:37:45 -08:00
Eldon
06a75c6fcf
Adding the queue declaration and binding to the UrlReceiver
2014-02-25 17:11:56 -05:00
Noah Levitt
d130e3fc44
call the extra info we send to umbra "clientId", kinda clarifies the purpose of it
2014-02-13 19:25:09 -08:00
Noah Levitt
e389499aa7
don't try to stop consuming amqp when crawl is shutting down, can end up reopening the connection after it's been closed
2014-02-13 19:24:31 -08:00
Noah Levitt
72b2d58034
pause amqp consumption on crawl pause, and resume on unpause
2014-02-10 21:04:19 -08:00
Noah Levitt
c493248863
formatting change only - use 4 spaces to indent
2014-02-10 20:43:50 -08:00
Noah Levitt
eccddb3df3
use parent url metadata received via amqp in building CrawlURI
2014-02-10 20:38:37 -08:00
Noah Levitt
8d82c440d3
send url metadata with amqp message so that umbra can send it back with child urls
2014-02-10 20:36:51 -08:00
Noah Levitt
6dbcac40d9
syncronize start() and stop()
2014-02-03 20:27:51 -08:00
Noah Levitt
500dafd390
AMQPPublishProcessor - annotate urls sent to AMQP; set special fetch status; add default amqpUri; replace queueName setting with exchange and routingKey, which seem to be what are needed; avoid sending urls that were received from AMQP, and robots.txt urls; set content-type of amqp message to application/json, which umbra seems to need
2014-02-03 18:41:33 -08:00
Noah Levitt
ecb0eaf435
AMQPUrlReceiver - ack amqp messages, get rid of misguided thread starting infinite consumers
2014-02-03 18:36:52 -08:00
Noah Levitt
c980b452c9
annotate CrawlURI with "receivedViaAMQP"
2014-02-01 18:26:07 -08:00
Noah Levitt
c7cf62bf80
license header
2014-02-01 18:23:49 -08:00
Noah Levitt
bf0590a083
spaces not tabs for indentation
2014-02-01 18:23:05 -08:00
Noah Levitt
c163f31851
AMQPUrlReceiver - receive urls and request headers via amqp in the format produced by https://github.com/internetarchive/umbra and add to frontier
2014-01-31 20:42:27 -08:00
Eldon
b33d3faa75
Better Log messages
2014-01-24 18:49:04 -08:00
Eldon
ab3aadf199
Pull out most umbra refs, only do http/https
2014-01-24 18:47:50 -08:00
Eldon
d837536dc6
Use toString method instead
2014-01-24 18:28:38 -08:00
Eldon
d3e96b2770
First pass at a processor to publish crawluris to AMQP channels
2014-01-24 18:11:47 -08:00
Noah Levitt
5930561042
fix contrib modules that had been using CrawlURI.getHttpMethod()
2014-01-14 14:27:34 -08:00
Noah Levitt
749bd6a214
bump version to 3.3.0-SNAPSHOT after 3.2.0 release
2014-01-10 16:22:43 -08:00
Noah Levitt
32c083f720
bump version number to 3.2.0 for release
2014-01-10 12:36:45 -08:00
Noah Levitt
0ea10665f1
abbreviate maven repository clauses, reduce redundancy, and let default settings prevail, such as updatePolicy=daily, pointed out by Erik Hetzner
2013-12-15 11:53:18 -08:00
Noah Levitt
8e1125d062
bump version to 3.2.0-SNAPSHOT since there are some incompatible api changes since 3.1.1 (requested by kris sigurðsson)
2013-11-20 09:18:31 -08:00
Noah Levitt
57311db117
use libs version ${project.version}
2013-09-10 15:24:58 -07:00
Adam Miller
dd135c9383
Making youtube extractor log messages less noisy.
2013-06-25 11:33:24 -07:00
Adam Miller
b85828135b
Convert MatchesStatusCodeDecideRule and NotMatchesStatusCodeDecideRule to use CrawlURI.getFetchStatus() to avoid null status codes and broaden use.
...
Added test cases for MatchesStatusCodeDecideRule and NotMatchesStatusCodeDecideRule
Fix comments for youtube extractor.
2013-06-21 12:03:41 -07:00
Kenji Nagahashi
1e3e66ac4c
SingleHBaseTable: FIX error message typo, print friendlier message for
...
TableNotFoundException.
2013-06-16 08:23:12 -07:00
Kenji Nagahashi
030006ddd5
drop debug="true" from log4j.xml in contrib
2013-06-13 18:45:55 -07:00
Kenji Nagahashi
b2a0495a08
merged Kenji's change to hbase de-duplication module:
...
- SingleHBaseTable using single HTable instances (no HTablePool) with
backing-off upon communication errors.
- FetchHistoryHelper for allowing multiple crawl history sources.
- row key backward compatibility mode.
2013-06-13 17:21:28 -07:00
Adam Miller
646133ce4e
pom.xml
...
Adding heritrix-engine dependency
XmlCrawlSummaryReport.java
XML version of the CrawlSummaryReport
ExtractorYoutubeFormatStream.java
Adding sample bean to docs
2013-06-12 16:43:28 -07:00
Noah Levitt
1743817e1e
make code formatting consistent with main heritrix proj by changing spaces to tabs; also remove unused imports from HBase.java, no other substantive changes
2013-06-11 15:55:35 -07:00
Adam Miller
ed6cc473f8
ExtractorYoutubeFormatStream.java
...
Adjusting json extraction regex to account for semi-colons within the content
ExtractorYoutubeFormatStreamTest.java
New test case for semi-colon in json content
2013-06-11 15:40:07 -07:00
Noah Levitt
f478798935
implement itagPriority and extractLimit parameters
2013-06-06 16:15:13 -07:00
Noah Levitt
593cae448e
test crawl reveals generate_204 urls apparently not needed
2013-06-06 14:41:26 -07:00
Noah Levitt
9a9e49d92d
only extract urls that look like youtube watch pages
2013-06-06 14:28:53 -07:00
Noah Levitt
9828310f60
make ExtractorYoutubeFormatStream extend Extractor instead of ContentExtractor because we don't really want the extra stuff that ContentExtractor does
2013-06-06 14:13:26 -07:00
Adam Miller
a7dda8b785
ExtractorYoutubeFormatStream.java
...
Parse parameters instead of attempting to regex desired values
Add priority itag list and extract limit
ExtractorYoutubeFormatStreamTest.java
Test cases for extracting all videos and extracting single video
2013-06-05 16:39:10 -07:00
Noah Levitt
11f8fd51c6
extractIndependently=true so that earlier ExtractorHTML extraction doesn't stop us from extracting here; also use java.net.URLDecoder
2013-06-05 13:25:25 -07:00
Noah Levitt
feec423b52
import Adam's ExtractorYoutubeFormatStream from its former home https://github.com/adam-miller/ExtractorYoutubeFormatStream
2013-06-05 13:16:55 -07:00
Noah Levitt
0508abb9ad
new contrib subproject for extra generally useful modules not part of the main distribution
2013-06-05 13:12:36 -07:00
Noah Levitt