lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-25 15:16:24 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,870 Commits 38 Branches 33 Tags
09ee95f37bcd48bde6d39cf81c3ab9e58fa8eeef
Commit Graph
1870 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Andrew Jackson 09ee95f37b Allow results with or without DNS lookup of localhost. 2017-10-11 23:30:17 +01:00
Andrew Jackson 46fa54da0c Set an appropriate Content Size when using non-DNS resolves. 2017-10-11 17:08:15 +01:00
Andrew Jackson a5381440fd Use build folder for temp files as tmp gets wiped on Travis CI sometimes. 2017-10-09 22:13:53 +01:00
Andrew Jackson 401e27fc64 Adapt test to cope with variation in how localhost gets resolved. 2017-10-09 21:50:05 +01:00
Andrew Jackson 1de6f7d2a0 Update Travis CI file to suitable Java 7 + 8 variants. 2017-10-09 09:54:16 +01:00
Barbara Miller 0ec055c211 Merge pull request #186 from nlevitt/trough-logging-tweak 
reduce batch size to 400 and avoid ridiculously long log lines
 2017-10-04 16:32:11 -07:00
Noah Levitt d01f2c3a30 fix copy&paste mistake 2017-10-04 16:29:22 -07:00
Noah Levitt 8fea6c4783 reduce batch size to 400 and avoid ridiculously long log lines 2017-10-04 15:35:16 -07:00
jkafader 65004ea748 Merge pull request #185 from internetarchive/trough-esc-sql 
escape strings in sql posted to trough
 2017-09-21 12:19:35 -07:00
Noah Levitt 7fc1c2f25f escape strings in sql posted to trough 2017-09-21 12:16:25 -07:00
jkafader 4c438d3852 Merge pull request #180 from internetarchive/aitfive-1176 
trough feed
 2017-09-05 17:37:36 -07:00
Noah Levitt 1c92be2d74 realized this stuff needs to be thread safe 2017-06-27 16:01:15 -07:00
Noah Levitt 66ead24762 missed another comma 2017-06-27 15:50:33 -07:00
Noah Levitt 698a02a5b7 missed a comma 2017-06-27 15:45:55 -07:00
Noah Levitt 2f29949b88 include http response payload when logging errors from trough 2017-06-26 17:37:39 -07:00
Noah Levitt 3892c17b56 ughhh appease java 8 javadoc rules 2017-06-15 17:29:22 -07:00
Noah Levitt a1a7232530 update for new data model queued_url->uncrawled_url and some new fields 2017-06-14 18:31:39 -07:00
Noah Levitt b670c2f59a missed the maven tarball assembly file 2017-06-08 11:39:08 -07:00
Noah Levitt a1f59da000 oops, String.join is a java 8 feature 2017-06-08 11:35:50 -07:00
Noah Levitt 87a15bddc8 Merge pull request #168 from internetarchive/gh-167 
fix for race-condition when first using the WARC writers https://gith…
 2017-06-07 18:15:19 -07:00
Noah Levitt 66fabf790e hacked up tarball with heritrix, contrib and dependencies 2017-06-07 18:05:52 -07:00
Noah Levitt eed0210a8f batch trough writes 2017-06-07 18:05:52 -07:00
Noah Levitt d1452c1d21 initial version of TroughCrawlLogFeed.java 2017-06-07 18:05:52 -07:00
Noah Levitt a4bb99e828 Merge pull request #179 from BitBaron/aitfive-1225 
Add parsing for srcset attributes
 2017-06-02 12:01:17 -07:00
Neil Minton 10541184b5 Clean up after code review. 
- Parse with string split instead of regex.
- Move srcset logic up to processEmbed(CrawlURI, CharSequence, CharSequence, Hop).
 2017-06-02 11:53:16 -07:00
Neil Minton 4c0e52af61 Add parsing for srcset attributes. 
- Includes Unit Tests
 2017-05-01 17:17:05 -07:00
Noah Levitt 752ea028f6 fix floating point format 2017-03-09 10:41:31 -08:00
Mohamed Elsayed d13bfb6a37 Remove unit tests which moved over from Heritrix3 to webarchive-commons 2017-03-09 10:26:32 -08:00
Barbara Miller 548ea5eb84 Merge pull request #178 from internetarchive/kafka-mem-hog 
KafkaCrawlLogFeed had been using lots of heap because each callback i…
 2017-03-07 11:11:23 -08:00
Noah Levitt bd6607739d KafkaCrawlLogFeed had been using lots of heap because each callback instance was keeping around a CrawlURI and in some cases there could be many of thousands of callbacks waiting around; instead, just keep some stats and log them periodically 2017-02-13 17:29:13 -08:00
Noah Levitt 0ca88c6071 Merge pull request #171 from ukwa/amqp-fine-control 
AMQP fine control
 2016-10-13 17:10:29 -07:00
Andrew Jackson 7e965f3512 No need to do anything for the usual link types as the FrontierPreparer handles them. 2016-10-11 20:56:46 +01:00
Andrew Jackson a4a841c826 Removing comment from old version of this code. 2016-10-11 20:43:32 +01:00
Andrew Jackson b3dff8d9d6 Defaulting the message prefetch count to 1000 instead of infinity. 2016-10-11 20:42:52 +01:00
Andrew Jackson 4d7a64d127 Reverted Kryo version change. 2016-10-11 20:41:20 +01:00
Andrew Jackson b3e1a83034 Merge branch 'amqp-fine-control' of github.com:ukwa/heritrix3 into amqp-fine-control 2016-10-03 16:52:48 +01:00
Andrew Jackson 0e40869f83 Fix package name. 2016-10-03 16:51:28 +01:00
Andrew Jackson fdae4b6297 Update Kryo to latest version. 2016-10-03 16:51:28 +01:00
Andrew Jackson eaec682dd4 Bump incoming links to MEDIUM to compete with e.g. runs of redirects. 2016-10-03 16:51:28 +01:00
Andrew Jackson c6c396af8b Use sensible defaults for normal links. 2016-10-03 16:51:28 +01:00
Andrew Jackson 553eae2ddd Add support for setting the hop and the prefetch count. 2016-10-03 16:51:28 +01:00
Noah Levitt 0b4671c291 fix for race-condition when first using the WARC writers https://github.com/internetarchive/heritrix3/issues/167 2016-08-15 13:36:33 -07:00
Noah Levitt fbb0173459 Merge pull request #166 from internetarchive/amqpWaiterStarter 
Don't wait to receive Umbra urls if Heritrix sends no url to Umbra
 2016-07-21 18:05:10 -05:00
Barbara Miller cbc73a6e8d check controller state, not frontier.isEmpty) 2016-07-21 11:40:58 -07:00
Barbara Miller 40d8ee77bf don't wait if h3 sends no url 2016-07-13 14:32:07 -07:00
Noah Levitt af2530edc4 Merge pull request #165 from galgeek/amqpWaiter 
AMQP URL Waiter
 2016-07-07 19:35:04 -05:00
Barbara Miller 4485b8c8c7 AMQP URL Waiter 2016-07-01 15:09:50 -07:00
Noah Levitt c04162a0f7 Merge pull request #164 from nlevitt/fix-test-errors 
Fixes for apparent build errors (extends #154)
 2016-06-08 14:31:58 -05:00
Noah Levitt 36ea1a9fb2 try to fix remaining serialization tests in oraclejdk8 by using ConcurrentSkipListMap instead of ConcurrentHashMap 2016-06-07 17:55:45 -05:00
Noah Levitt 99d87e8f23 hopefully fix serialization tests in oraclejdk8 by using TreeSet instead of HashSet in KeyedProperties.java 2016-06-07 14:53:11 -05:00