lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-13 01:05:47 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,400 Commits 39 Branches 33 Tags
1e8e2e2016b5ab3d5fd8a32669a4010ccc2cd006
Commit Graph
1400 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Shriphani Palakodety 1e8e2e2016 prevent intellij from complaining 2014-03-18 06:42:02 -04:00
Shriphani Palakodety b0ad9358e8 bump version in pom.xml 2014-03-18 06:42:02 -04:00
Shriphani Palakodety d0e0ee6bc6 forgot to add expireAfterWrite 2014-03-18 06:42:02 -04:00
Shriphani Palakodety bd1047d2d2 bumped guava version to 11 2014-03-18 06:42:02 -04:00
Adam Miller 2018da4cd6 Merge pull request #51 from nlevitt/expression-deciderule 
Expression deciderule
 2014-03-17 17:49:12 -07:00
Noah Levitt fb039291ef fix indentation (spaces only) 2014-03-16 18:14:29 -07:00
Noah Levitt fb2be86e43 ExpressionDecideRule - evaluate groovy template with variable "curi" bound 2014-03-16 18:13:12 -07:00
Adam Miller a36acbacac Merge pull request #49 from nlevitt/youtube-march-2014 
Youtube march 2014
 2014-03-06 16:14:12 -08:00
Noah Levitt 5138a01240 warn if youtube video url is missing signature parameter 2014-03-06 16:08:07 -08:00
Noah Levitt 6e3bd2a2e8 update to handle latest changes from youtube 2014-03-05 19:05:31 -08:00
vonrosen 8a2533c95c Merge pull request #48 from nlevitt/umbra 
Umbra
 2014-03-03 08:21:13 -08:00
Noah Levitt d1eeb43e40 Merge branch 'master' into umbra 2014-03-01 16:10:56 -08:00
Noah Levitt e46c9ed759 Continue processing url after sending to amqp (fetch, etc). Solves 2 problems, 1) crawl finishing after sending off umbra seeds and before umbra has a chance to send anything back, and 2) seeds sent to umbra not being fetched because they were already seen. We may want to change this in the future as umbra gets fancier, e.g. sending back html from DOM, but for now it should work. 2014-03-01 16:10:41 -08:00
Noah Levitt 6a92e88b1c Merge pull request #40 from internetarchive/her-2064 
For https://webarchive.jira.com/browse/HER-2064
 2014-02-28 16:48:42 -08:00
Noah Levitt 54f572c356 Merge pull request #47 from adam-miller/youtube-itag-priority 
Adjusting Youtube itag priority
 2014-02-28 14:43:50 -08:00
Adam Miller 7fa3809828 fixing formatting 2014-02-28 14:33:55 -08:00
Adam Miller 09c113163e Adjusting youtube extraction itag priority 2014-02-28 14:15:49 -08:00
ikreymer 5e55921abb Merge pull request #46 from nlevitt/switch-to-ia-webarchive-commons 
switch dependency from ia-web-commons 1.1.1-SNAPSHOT to webarchive-commo...
 2014-02-28 12:48:44 -08:00
Noah Levitt e3efa3a5f2 switch dependency from ia-web-commons 1.1.1-SNAPSHOT to webarchive-commons 1.1.1-IA-SNAPSHOT 2014-02-28 12:46:36 -08:00
Adam Miller 22cc1222b2 Merge pull request #45 from nlevitt/update-youtube-itags 
Update youtube itags
 2014-02-28 11:13:11 -08:00
Noah Levitt 61539ab930 update ExtractorYoutubeFormatStreamTest.testDefaultItag() 2014-02-27 17:41:38 -08:00
Noah Levitt 8b124ecaa9 update youtube default video format priority (see https://webarchive.jira.com/browse/ARI-3730) 2014-02-27 17:37:45 -08:00
Noah Levitt da8e284581 Merge pull request #43 from martinsbalodis/master 
fix job.log file handler was left open when jobdir is removed
 2014-02-27 09:34:07 -08:00
Mārtiņš Balodis eba9a880be Merge pull request #1 from nlevitt/pull43 
handle main job log handler like current launch job log handler, close on teardown
 2014-02-27 18:42:55 +02:00
Noah Levitt ea19dacbd2 handle main job log handler like current launch job log handler, close on teardown 2014-02-26 17:19:50 -08:00
Eldon a297abb02d Merge pull request #44 from nlevitt/update-hc4_3_3 
update httpcomponents, should address NPE we've seen https://issues.apac...
 2014-02-26 14:39:47 -05:00
Noah Levitt cf5e295d42 update httpcomponents, should address NPE we've seen https://issues.apache.org/jira/browse/HTTPCLIENT-1417 2014-02-26 11:35:26 -08:00
Martins Balodis 88dc01305f fix job.log file handler was left open when jobdir is removed 2014-02-26 21:15:38 +02:00
Noah Levitt 659c5e886f Merge pull request #42 from eldondev/master 
Adding the queue declaration and binding to the UrlReceiver
 2014-02-25 14:15:30 -08:00
Eldon 06a75c6fcf Adding the queue declaration and binding to the UrlReceiver 2014-02-25 17:11:56 -05:00
Eldon bbd818e6e4 Merge pull request #39 from nlevitt/umbra 
progress and formatting changes
 2014-02-25 16:24:38 -05:00
Adam Miller e08936d029 Merge pull request #41 from nlevitt/fix-slow-cookies 
Fix slow cookies
 2014-02-21 15:51:49 -08:00
Noah Levitt cf775bec94 remove extraneous code related to logging removed in last commit 2014-02-21 15:46:52 -08:00
Noah Levitt fd806a514d remove extra logging from performance test 2014-02-21 13:28:13 -08:00
Noah Levitt 39f06a1f1c fix slowness caused by BdbCookieStore by maintaining cookie list in memory parallel to the one stored on disk; add tests for consistency and performance under load 2014-02-21 13:26:58 -08:00
Noah Levitt d130e3fc44 call the extra info we send to umbra "clientId", kinda clarifies the purpose of it 2014-02-13 19:25:09 -08:00
Noah Levitt e389499aa7 don't try to stop consuming amqp when crawl is shutting down, can end up reopening the connection after it's been closed 2014-02-13 19:24:31 -08:00
Hunter Stern d6e61b83da For https://webarchive.jira.com/browse/HER-2064 2014-02-13 16:59:05 -08:00
Hunter Stern 06ba50c410 Reverting previous commit: https://github.com/internetarchive/heritrix3/commit/164d0fbdc5fa32cd90edd9f1b4091ed876552232 2014-02-13 16:50:21 -08:00
Hunter Stern 164d0fbdc5 For https://webarchive.jira.com/browse/HER-2064 2014-02-13 16:42:58 -08:00
Noah Levitt 72b2d58034 pause amqp consumption on crawl pause, and resume on unpause 2014-02-10 21:04:19 -08:00
Noah Levitt c493248863 formatting change only - use 4 spaces to indent 2014-02-10 20:43:50 -08:00
Noah Levitt eccddb3df3 use parent url metadata received via amqp in building CrawlURI 2014-02-10 20:38:37 -08:00
Noah Levitt 8d82c440d3 send url metadata with amqp message so that umbra can send it back with child urls 2014-02-10 20:36:51 -08:00
Eldon 704f4aa443 Merge pull request #38 from nlevitt/umbra 
Umbra - AMQPUrlReceiver.java receive urls via amqp and add to frontier, related changes
 2014-02-06 16:51:03 -05:00
Noah Levitt 6dbcac40d9 syncronize start() and stop() 2014-02-03 20:27:51 -08:00
Noah Levitt 500dafd390 AMQPPublishProcessor - annotate urls sent to AMQP; set special fetch status; add default amqpUri; replace queueName setting with exchange and routingKey, which seem to be what are needed; avoid sending urls that were received from AMQP, and robots.txt urls; set content-type of amqp message to application/json, which umbra seems to need 2014-02-03 18:41:33 -08:00
Noah Levitt af44522fdf FetchHTTPRequest - include CrawlURI.getData().get("customHttpRequestHeaders"), if any, in http request 2014-02-03 18:41:23 -08:00
Noah Levitt ecb0eaf435 AMQPUrlReceiver - ack amqp messages, get rid of misguided thread starting infinite consumers 2014-02-03 18:36:52 -08:00
Noah Levitt e90ba8278b Merge branch 'master' into umbra 2014-02-03 18:22:25 -08:00