lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-25 16:06:33 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,451 Commits 38 Branches 32 Tags
e6d646d4b4eeec0c148d6bd5e9cc0e67cbcf3479
Commit Graph
1451 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Adam Miller e6d646d4b4 Cleanup / optimization 2014-04-30 12:14:32 -07:00
Adam Miller 34a9172e69 Organize Imports 2014-04-29 17:25:43 -07:00
Adam Miller 5181c1ef64 Cleaning up test cases and removing unused code 2014-04-29 17:11:18 -07:00
Adam Miller 33277445f6 Merge branch 'master' of git://github.com/internetarchive/heritrix3 into ExtractorPDFContent 2014-04-29 17:06:55 -07:00
Adam Miller 49069f2dd1 Adding comment for library dependencies 2014-04-29 17:06:11 -07:00
Adam Miller 7e8a66e7d8 New Test cases, fixing bugs 2014-04-29 10:50:04 -07:00
vonrosen 747bbeba97 Merge pull request #62 from nlevitt/ari-3765 
Ari 3765 gracefully handle amqp server going up and down
 2014-04-28 09:39:37 -07:00
Kenji Nagahashi 50a38e1376 Merge pull request #61 from nlevitt/her-2065 
HER-2065 synchronize on inactiveQueuesByPrecedence inside of synchronize...
 2014-04-25 11:55:14 -07:00
Noah Levitt 381442dfb6 ARI-3765 - make AMQPUrlReceiver gracefully handle amqp server going up and down 2014-04-24 21:23:03 -07:00
Noah Levitt 10e0f262b3 ARI-3765 - make AMQPPublishProcessor gracefully handle amqp server going up and down 2014-04-24 20:31:40 -07:00
Noah Levitt d777c198bd HER-2065 - in activateInactiveQueue() close synchronized block earlier, and move updateHighestWaiting() up to keep it inside the block; now the call to deactivateQueue() is not in the synchronized block, so the deadlock should be avoided (fix suggested by kenji) 2014-04-24 17:23:01 -07:00
Adam Miller 3a456638ca Merge pull request #60 from nlevitt/cosmetics 
Cosmetics
 2014-04-21 15:52:43 -07:00
Noah Levitt 0cd1043cb1 change memory bar color, use consistent non-capitalization on buttons 2014-04-21 11:53:40 -07:00
Noah Levitt 1b72ce5dd8 remove extra </div> 2014-04-21 11:53:05 -07:00
Noah Levitt cfb9474f7a cosmetic changes 2014-04-21 09:06:15 -07:00
Adam Miller a48a29f9bd Adding ExtractorPDFContent to find nav links from the text content of PDFs 2014-04-14 16:02:35 -07:00
vonrosen 67a7a4fb6c Merge pull request #59 from nlevitt/ari-3723 
fix unit test now that we accept speculative urls with query params with...
 2014-04-10 20:22:15 -07:00
Noah Levitt e73810d0a6 fix unit test now that we accept speculative urls with query params with no value 2014-04-10 17:25:27 -07:00
Eldon 5e75ba6506 Merge pull request #57 from nlevitt/umbra 
AMQPUrlReceiver - improve handling of case where rabbitmq is unreachable...
 2014-04-10 15:35:48 -04:00
vonrosen 63c4264de6 Merge pull request #58 from nlevitt/ari-3723 
for ARI-3723, accept speculative urls with query params with no value
 2014-04-10 12:09:16 -07:00
Noah Levitt 339e6ec87a for ARI-3723, accept speculative urls with query params with no value 2014-04-04 10:15:07 -07:00
Noah Levitt ac9d8e6ba7 AMQPUrlReceiver - improve handling of case where rabbitmq is unreachable at crawl startup - spawn off a thread to start up the amqp consumer to avoid the crawl launch thread getting stuck 2014-04-03 11:39:36 -07:00
vonrosen 030063f7b3 Merge pull request #56 from nlevitt/fix-form-login 
fix FormLoginProcessor checkpointing
 2014-04-03 09:10:44 -07:00
Noah Levitt 36a207ce07 fix FormLoginProcessor checkpointing 2014-04-02 17:33:09 -07:00
vonrosen 20350091af Merge pull request #54 from nlevitt/fix-form-login 
oops, update test to expect post data as url-encoded query string
 2014-04-01 15:30:30 -07:00
Noah Levitt 878fd9f7a9 oops, update test to expect post data as url-encoded query string 2014-04-01 15:18:17 -07:00
vonrosen 4bd8045bfb Merge pull request #53 from nlevitt/fix-form-login 
Fix form login
 2014-04-01 12:29:50 -07:00
Noah Levitt 552200b2fa fix form login processor by saving login post data as url-encoded query string, which FetchHTTP now expects; tweaks to make FormLoginSelfTest pass 2014-03-31 20:38:17 -07:00
Noah Levitt 20a5f31e19 ExtractorHTMLForms+FormLoginProcessor full crawl job integration test (not passing yet) 2014-03-31 18:57:33 -07:00
Noah Levitt a82886b747 restore support for sending http post data, add unit test 2014-03-31 18:09:23 -07:00
Noah Levitt 23e8d41651 avoid NPE; root cause was assumption that LoadingCache.asMap().get() would call CacheLoader.load() but it doesnt, see https://code.google.com/p/guava-libraries/wiki/MapMakerMigration 2014-03-31 13:40:29 -07:00
Noah Levitt cc838747af pull methods getRecorder() and makeCrawlURI(String) into ModuleTestBase, and move ModuleTestBase into heritrix-modules so that CrawlURI is available 2014-03-31 13:38:54 -07:00
Noah Levitt 26a577ff6e basic unit tests for FormLoginProcessor 2014-03-31 13:38:39 -07:00
Noah Levitt 0d7f633fdb pull methods getRecorder() and makeCrawlURI(String) into ModuleTestBase, and move ModuleTestBase into heritrix-modules so that CrawlURI is available 2014-03-31 13:36:44 -07:00
Adam Miller 19e8cd5b66 Merge pull request #52 from nlevitt/expression-deciderule 
Implicitly add the ${} around groovyExpression. When cxml contains ${}, ...
 2014-03-18 14:10:36 -07:00
Noah Levitt ef2889cf17 Merge pull request #50 from shriphani/master 
Replace deprecated routines in guava
 2014-03-18 11:27:49 -07:00
Shriphani Palakodety 74f017c53e Merge branch 'master' of https://github.com/shriphani/heritrix3 2014-03-18 06:42:54 -04:00
Shriphani Palakodety 84f29e4e10 build with the right key and value types 2014-03-18 06:42:03 -04:00
Shriphani Palakodety c81eeced7a added removal listener 2014-03-18 06:42:02 -04:00
Shriphani Palakodety 2f60b5f935 migration to latest guava complete 2014-03-18 06:42:02 -04:00
Shriphani Palakodety 8c9892586a removed guava dependency from heritrix. use the webarchive-commons version 2014-03-18 06:42:02 -04:00
Shriphani Palakodety 1e8e2e2016 prevent intellij from complaining 2014-03-18 06:42:02 -04:00
Shriphani Palakodety b0ad9358e8 bump version in pom.xml 2014-03-18 06:42:02 -04:00
Shriphani Palakodety d0e0ee6bc6 forgot to add expireAfterWrite 2014-03-18 06:42:02 -04:00
Shriphani Palakodety bd1047d2d2 bumped guava version to 11 2014-03-18 06:42:02 -04:00
Shriphani Palakodety 60cc78dac8 Merge branch 'master' of https://github.com/internetarchive/heritrix3 2014-03-18 06:38:12 -04:00
Noah Levitt 863ff90157 Implicitly add the ${} around groovyExpression. When cxml contains ${}, it fights with spring's PropertyPlaceholderConfigurer (if enabled); and since what we're interested in is an expression, not a template for producing a string, it's more appropriate to leave it out anyway. 2014-03-17 18:11:47 -07:00
Adam Miller 2018da4cd6 Merge pull request #51 from nlevitt/expression-deciderule 
Expression deciderule
 2014-03-17 17:49:12 -07:00
Noah Levitt fb039291ef fix indentation (spaces only) 2014-03-16 18:14:29 -07:00
Noah Levitt fb2be86e43 ExpressionDecideRule - evaluate groovy template with variable "curi" bound 2014-03-16 18:13:12 -07:00