lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-23 06:05:46 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,469 Commits 38 Branches 33 Tags
963e04e50cd16ee1383c871fed5cec0ca92998dc
Commit Graph
1469 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Noah Levitt 963e04e50c handle exception determining whether to apply overlay 2014-05-13 15:47:21 -07:00
Adam Miller 3e213c82bc Merge pull request #67 from nlevitt/umbra 
don't log severe with stack trace on normal amqp shutdown
 2014-05-13 14:44:09 -07:00
Noah Levitt 08a4aafa23 don't log severe with stack trace on normal amqp shutdown 2014-05-13 14:39:35 -07:00
Noah Levitt 38cd14ecac Merge pull request #63 from adam-miller/ExtractorPDFContent 
Contrib - ExtractorPDFContent
 2014-05-13 14:09:20 -07:00
Adam Miller aaecd6d78c Merge branch 'master' of git://github.com/internetarchive/heritrix3 into ExtractorPDFContent 2014-05-13 14:04:29 -07:00
Adam Miller fb42c07c92 bugfix, new test case. 2014-05-13 14:03:03 -07:00
Adam Miller ab7a60f9e3 Merge pull request #66 from nlevitt/cosmetics 
oops, make "exit java process" button work again
 2014-05-13 13:31:11 -07:00
Adam Miller 85e7a5ee78 Bugfix & new test cases. 2014-05-12 16:52:32 -07:00
Noah Levitt ba68649ba0 Merge remote-tracking branch 'origin/master' into cosmetics 2014-05-07 10:02:06 -07:00
Noah Levitt b7f8a0669f Merge branch 'cosmetics' of github.com:nlevitt/heritrix3 into cosmetics 2014-05-07 10:02:00 -07:00
Noah Levitt 33e782e2fd oops, make "exit java process" button work again 2014-05-07 10:01:47 -07:00
Adam Miller eb32d7a8b8 Merge pull request #65 from nlevitt/umbra 
shut down the starter-restarter thread at crawl finish!!
 2014-05-02 15:47:44 -07:00
Noah Levitt 949c9f086c shut down the starter-restarter thread at crawl finish!! 2014-05-02 15:42:29 -07:00
Noah Levitt dd510cbf3b Merge pull request #64 from adam-miller/ViaSurtPrefixedDecideRule 
Via surt prefixed decide rule
 2014-05-01 18:14:25 -07:00
Adam Miller ae878d29cf Code cleanup 2014-05-01 15:22:16 -07:00
Adam Miller f2bebd41b0 Cleanup / Optimization 2014-05-01 12:48:23 -07:00
Adam Miller 753404f662 Adding ViaSurtPrefixedDecideRule. Returns true if a surtform 'via' matches any surt in a specified list 2014-05-01 12:43:26 -07:00
Adam Miller 2b1c330be6 Further cleanup 2014-04-30 12:18:14 -07:00
Adam Miller e6d646d4b4 Cleanup / optimization 2014-04-30 12:14:32 -07:00
Adam Miller 34a9172e69 Organize Imports 2014-04-29 17:25:43 -07:00
Adam Miller 5181c1ef64 Cleaning up test cases and removing unused code 2014-04-29 17:11:18 -07:00
Adam Miller 33277445f6 Merge branch 'master' of git://github.com/internetarchive/heritrix3 into ExtractorPDFContent 2014-04-29 17:06:55 -07:00
Adam Miller 49069f2dd1 Adding comment for library dependencies 2014-04-29 17:06:11 -07:00
Adam Miller 7e8a66e7d8 New Test cases, fixing bugs 2014-04-29 10:50:04 -07:00
vonrosen 747bbeba97 Merge pull request #62 from nlevitt/ari-3765 
Ari 3765 gracefully handle amqp server going up and down
 2014-04-28 09:39:37 -07:00
Kenji Nagahashi 50a38e1376 Merge pull request #61 from nlevitt/her-2065 
HER-2065 synchronize on inactiveQueuesByPrecedence inside of synchronize...
 2014-04-25 11:55:14 -07:00
Noah Levitt 381442dfb6 ARI-3765 - make AMQPUrlReceiver gracefully handle amqp server going up and down 2014-04-24 21:23:03 -07:00
Noah Levitt 10e0f262b3 ARI-3765 - make AMQPPublishProcessor gracefully handle amqp server going up and down 2014-04-24 20:31:40 -07:00
Noah Levitt d777c198bd HER-2065 - in activateInactiveQueue() close synchronized block earlier, and move updateHighestWaiting() up to keep it inside the block; now the call to deactivateQueue() is not in the synchronized block, so the deadlock should be avoided (fix suggested by kenji) 2014-04-24 17:23:01 -07:00
Adam Miller 3a456638ca Merge pull request #60 from nlevitt/cosmetics 
Cosmetics
 2014-04-21 15:52:43 -07:00
Noah Levitt 0cd1043cb1 change memory bar color, use consistent non-capitalization on buttons 2014-04-21 11:53:40 -07:00
Noah Levitt 1b72ce5dd8 remove extra </div> 2014-04-21 11:53:05 -07:00
Noah Levitt cfb9474f7a cosmetic changes 2014-04-21 09:06:15 -07:00
Adam Miller a48a29f9bd Adding ExtractorPDFContent to find nav links from the text content of PDFs 2014-04-14 16:02:35 -07:00
vonrosen 67a7a4fb6c Merge pull request #59 from nlevitt/ari-3723 
fix unit test now that we accept speculative urls with query params with...
 2014-04-10 20:22:15 -07:00
Noah Levitt e73810d0a6 fix unit test now that we accept speculative urls with query params with no value 2014-04-10 17:25:27 -07:00
Eldon 5e75ba6506 Merge pull request #57 from nlevitt/umbra 
AMQPUrlReceiver - improve handling of case where rabbitmq is unreachable...
 2014-04-10 15:35:48 -04:00
vonrosen 63c4264de6 Merge pull request #58 from nlevitt/ari-3723 
for ARI-3723, accept speculative urls with query params with no value
 2014-04-10 12:09:16 -07:00
Noah Levitt 339e6ec87a for ARI-3723, accept speculative urls with query params with no value 2014-04-04 10:15:07 -07:00
Noah Levitt ac9d8e6ba7 AMQPUrlReceiver - improve handling of case where rabbitmq is unreachable at crawl startup - spawn off a thread to start up the amqp consumer to avoid the crawl launch thread getting stuck 2014-04-03 11:39:36 -07:00
vonrosen 030063f7b3 Merge pull request #56 from nlevitt/fix-form-login 
fix FormLoginProcessor checkpointing
 2014-04-03 09:10:44 -07:00
Noah Levitt 36a207ce07 fix FormLoginProcessor checkpointing 2014-04-02 17:33:09 -07:00
vonrosen 20350091af Merge pull request #54 from nlevitt/fix-form-login 
oops, update test to expect post data as url-encoded query string
 2014-04-01 15:30:30 -07:00
Noah Levitt 878fd9f7a9 oops, update test to expect post data as url-encoded query string 2014-04-01 15:18:17 -07:00
vonrosen 4bd8045bfb Merge pull request #53 from nlevitt/fix-form-login 
Fix form login
 2014-04-01 12:29:50 -07:00
Noah Levitt 552200b2fa fix form login processor by saving login post data as url-encoded query string, which FetchHTTP now expects; tweaks to make FormLoginSelfTest pass 2014-03-31 20:38:17 -07:00
Noah Levitt 20a5f31e19 ExtractorHTMLForms+FormLoginProcessor full crawl job integration test (not passing yet) 2014-03-31 18:57:33 -07:00
Noah Levitt a82886b747 restore support for sending http post data, add unit test 2014-03-31 18:09:23 -07:00
Noah Levitt 23e8d41651 avoid NPE; root cause was assumption that LoadingCache.asMap().get() would call CacheLoader.load() but it doesnt, see https://code.google.com/p/guava-libraries/wiki/MapMakerMigration 2014-03-31 13:40:29 -07:00
Noah Levitt cc838747af pull methods getRecorder() and makeCrawlURI(String) into ModuleTestBase, and move ModuleTestBase into heritrix-modules so that CrawlURI is available 2014-03-31 13:38:54 -07:00