lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-07-26 09:21:43 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,431 Commits 40 Branches 31 Tags
63c4264de6f85bedb2ffe33bbba92e19aaa4e35d
Commit Graph
1431 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
vonrosen 63c4264de6 Merge pull request #58 from nlevitt/ari-3723 
for ARI-3723, accept speculative urls with query params with no value
 2014-04-10 12:09:16 -07:00
Noah Levitt 339e6ec87a for ARI-3723, accept speculative urls with query params with no value 2014-04-04 10:15:07 -07:00
vonrosen 030063f7b3 Merge pull request #56 from nlevitt/fix-form-login 
fix FormLoginProcessor checkpointing
 2014-04-03 09:10:44 -07:00
Noah Levitt 36a207ce07 fix FormLoginProcessor checkpointing 2014-04-02 17:33:09 -07:00
vonrosen 20350091af Merge pull request #54 from nlevitt/fix-form-login 
oops, update test to expect post data as url-encoded query string
 2014-04-01 15:30:30 -07:00
Noah Levitt 878fd9f7a9 oops, update test to expect post data as url-encoded query string 2014-04-01 15:18:17 -07:00
vonrosen 4bd8045bfb Merge pull request #53 from nlevitt/fix-form-login 
Fix form login
 2014-04-01 12:29:50 -07:00
Noah Levitt 552200b2fa fix form login processor by saving login post data as url-encoded query string, which FetchHTTP now expects; tweaks to make FormLoginSelfTest pass 2014-03-31 20:38:17 -07:00
Noah Levitt 20a5f31e19 ExtractorHTMLForms+FormLoginProcessor full crawl job integration test (not passing yet) 2014-03-31 18:57:33 -07:00
Noah Levitt a82886b747 restore support for sending http post data, add unit test 2014-03-31 18:09:23 -07:00
Noah Levitt 23e8d41651 avoid NPE; root cause was assumption that LoadingCache.asMap().get() would call CacheLoader.load() but it doesnt, see https://code.google.com/p/guava-libraries/wiki/MapMakerMigration 2014-03-31 13:40:29 -07:00
Noah Levitt cc838747af pull methods getRecorder() and makeCrawlURI(String) into ModuleTestBase, and move ModuleTestBase into heritrix-modules so that CrawlURI is available 2014-03-31 13:38:54 -07:00
Noah Levitt 26a577ff6e basic unit tests for FormLoginProcessor 2014-03-31 13:38:39 -07:00
Noah Levitt 0d7f633fdb pull methods getRecorder() and makeCrawlURI(String) into ModuleTestBase, and move ModuleTestBase into heritrix-modules so that CrawlURI is available 2014-03-31 13:36:44 -07:00
Adam Miller 19e8cd5b66 Merge pull request #52 from nlevitt/expression-deciderule 
Implicitly add the ${} around groovyExpression. When cxml contains ${}, ...
 2014-03-18 14:10:36 -07:00
Noah Levitt ef2889cf17 Merge pull request #50 from shriphani/master 
Replace deprecated routines in guava
 2014-03-18 11:27:49 -07:00
Shriphani Palakodety 74f017c53e Merge branch 'master' of https://github.com/shriphani/heritrix3 2014-03-18 06:42:54 -04:00
Shriphani Palakodety 84f29e4e10 build with the right key and value types 2014-03-18 06:42:03 -04:00
Shriphani Palakodety c81eeced7a added removal listener 2014-03-18 06:42:02 -04:00
Shriphani Palakodety 2f60b5f935 migration to latest guava complete 2014-03-18 06:42:02 -04:00
Shriphani Palakodety 8c9892586a removed guava dependency from heritrix. use the webarchive-commons version 2014-03-18 06:42:02 -04:00
Shriphani Palakodety 1e8e2e2016 prevent intellij from complaining 2014-03-18 06:42:02 -04:00
Shriphani Palakodety b0ad9358e8 bump version in pom.xml 2014-03-18 06:42:02 -04:00
Shriphani Palakodety d0e0ee6bc6 forgot to add expireAfterWrite 2014-03-18 06:42:02 -04:00
Shriphani Palakodety bd1047d2d2 bumped guava version to 11 2014-03-18 06:42:02 -04:00
Shriphani Palakodety 60cc78dac8 Merge branch 'master' of https://github.com/internetarchive/heritrix3 2014-03-18 06:38:12 -04:00
Noah Levitt 863ff90157 Implicitly add the ${} around groovyExpression. When cxml contains ${}, it fights with spring's PropertyPlaceholderConfigurer (if enabled); and since what we're interested in is an expression, not a template for producing a string, it's more appropriate to leave it out anyway. 2014-03-17 18:11:47 -07:00
Adam Miller 2018da4cd6 Merge pull request #51 from nlevitt/expression-deciderule 
Expression deciderule
 2014-03-17 17:49:12 -07:00
Noah Levitt fb039291ef fix indentation (spaces only) 2014-03-16 18:14:29 -07:00
Noah Levitt fb2be86e43 ExpressionDecideRule - evaluate groovy template with variable "curi" bound 2014-03-16 18:13:12 -07:00
Shriphani Palakodety 4c302f84be build with the right key and value types 2014-03-16 21:00:10 -04:00
Shriphani Palakodety d4d23639b6 added removal listener 2014-03-14 16:20:50 -04:00
Shriphani Palakodety 439f78b038 migration to latest guava complete 2014-03-14 15:36:22 -04:00
Shriphani Palakodety abd61abbf2 removed guava dependency from heritrix. use the webarchive-commons version 2014-03-14 15:13:34 -04:00
Shriphani Palakodety ad8cfd7075 prevent intellij from complaining 2014-03-13 21:50:06 -04:00
Shriphani Palakodety 25ce16e893 bump version in pom.xml 2014-03-13 20:02:58 -04:00
Shriphani Palakodety ee6a3bddb3 forgot to add expireAfterWrite 2014-03-13 15:29:37 -04:00
Shriphani Palakodety 2cac0ebf80 bumped guava version to 11 2014-03-13 15:26:57 -04:00
Adam Miller a36acbacac Merge pull request #49 from nlevitt/youtube-march-2014 
Youtube march 2014
 2014-03-06 16:14:12 -08:00
Noah Levitt 5138a01240 warn if youtube video url is missing signature parameter 2014-03-06 16:08:07 -08:00
Noah Levitt 6e3bd2a2e8 update to handle latest changes from youtube 2014-03-05 19:05:31 -08:00
vonrosen 8a2533c95c Merge pull request #48 from nlevitt/umbra 
Umbra
 2014-03-03 08:21:13 -08:00
Noah Levitt d1eeb43e40 Merge branch 'master' into umbra 2014-03-01 16:10:56 -08:00
Noah Levitt e46c9ed759 Continue processing url after sending to amqp (fetch, etc). Solves 2 problems, 1) crawl finishing after sending off umbra seeds and before umbra has a chance to send anything back, and 2) seeds sent to umbra not being fetched because they were already seen. We may want to change this in the future as umbra gets fancier, e.g. sending back html from DOM, but for now it should work. 2014-03-01 16:10:41 -08:00
Noah Levitt 6a92e88b1c Merge pull request #40 from internetarchive/her-2064 
For https://webarchive.jira.com/browse/HER-2064
 2014-02-28 16:48:42 -08:00
Noah Levitt 54f572c356 Merge pull request #47 from adam-miller/youtube-itag-priority 
Adjusting Youtube itag priority
 2014-02-28 14:43:50 -08:00
Adam Miller 7fa3809828 fixing formatting 2014-02-28 14:33:55 -08:00
Adam Miller 09c113163e Adjusting youtube extraction itag priority 2014-02-28 14:15:49 -08:00
ikreymer 5e55921abb Merge pull request #46 from nlevitt/switch-to-ia-webarchive-commons 
switch dependency from ia-web-commons 1.1.1-SNAPSHOT to webarchive-commo...
 2014-02-28 12:48:44 -08:00
Noah Levitt e3efa3a5f2 switch dependency from ia-web-commons 1.1.1-SNAPSHOT to webarchive-commons 1.1.1-IA-SNAPSHOT 2014-02-28 12:46:36 -08:00