lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-23 06:56:41 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,853 Commits 40 Branches 31 Tags
2f29949b88be2f97069a1cbe0dcd61ca341b12e5
Commit Graph
1853 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Noah Levitt 2f29949b88 include http response payload when logging errors from trough 2017-06-26 17:37:39 -07:00
Noah Levitt a1a7232530 update for new data model queued_url->uncrawled_url and some new fields 2017-06-14 18:31:39 -07:00
Noah Levitt b670c2f59a missed the maven tarball assembly file 2017-06-08 11:39:08 -07:00
Noah Levitt a1f59da000 oops, String.join is a java 8 feature 2017-06-08 11:35:50 -07:00
Noah Levitt 66fabf790e hacked up tarball with heritrix, contrib and dependencies 2017-06-07 18:05:52 -07:00
Noah Levitt eed0210a8f batch trough writes 2017-06-07 18:05:52 -07:00
Noah Levitt d1452c1d21 initial version of TroughCrawlLogFeed.java 2017-06-07 18:05:52 -07:00
Noah Levitt a4bb99e828 Merge pull request #179 from BitBaron/aitfive-1225 
Add parsing for srcset attributes
 2017-06-02 12:01:17 -07:00
Neil Minton 10541184b5 Clean up after code review. 
- Parse with string split instead of regex.
- Move srcset logic up to processEmbed(CrawlURI, CharSequence, CharSequence, Hop).
 2017-06-02 11:53:16 -07:00
Neil Minton 4c0e52af61 Add parsing for srcset attributes. 
- Includes Unit Tests
 2017-05-01 17:17:05 -07:00
Noah Levitt 752ea028f6 fix floating point format 2017-03-09 10:41:31 -08:00
Mohamed Elsayed d13bfb6a37 Remove unit tests which moved over from Heritrix3 to webarchive-commons 2017-03-09 10:26:32 -08:00
Barbara Miller 548ea5eb84 Merge pull request #178 from internetarchive/kafka-mem-hog 
KafkaCrawlLogFeed had been using lots of heap because each callback i…
 2017-03-07 11:11:23 -08:00
Noah Levitt bd6607739d KafkaCrawlLogFeed had been using lots of heap because each callback instance was keeping around a CrawlURI and in some cases there could be many of thousands of callbacks waiting around; instead, just keep some stats and log them periodically 2017-02-13 17:29:13 -08:00
Noah Levitt 0ca88c6071 Merge pull request #171 from ukwa/amqp-fine-control 
AMQP fine control
 2016-10-13 17:10:29 -07:00
Andrew Jackson 7e965f3512 No need to do anything for the usual link types as the FrontierPreparer handles them. 2016-10-11 20:56:46 +01:00
Andrew Jackson a4a841c826 Removing comment from old version of this code. 2016-10-11 20:43:32 +01:00
Andrew Jackson b3dff8d9d6 Defaulting the message prefetch count to 1000 instead of infinity. 2016-10-11 20:42:52 +01:00
Andrew Jackson 4d7a64d127 Reverted Kryo version change. 2016-10-11 20:41:20 +01:00
Andrew Jackson b3e1a83034 Merge branch 'amqp-fine-control' of github.com:ukwa/heritrix3 into amqp-fine-control 2016-10-03 16:52:48 +01:00
Andrew Jackson 0e40869f83 Fix package name. 2016-10-03 16:51:28 +01:00
Andrew Jackson fdae4b6297 Update Kryo to latest version. 2016-10-03 16:51:28 +01:00
Andrew Jackson eaec682dd4 Bump incoming links to MEDIUM to compete with e.g. runs of redirects. 2016-10-03 16:51:28 +01:00
Andrew Jackson c6c396af8b Use sensible defaults for normal links. 2016-10-03 16:51:28 +01:00
Andrew Jackson 553eae2ddd Add support for setting the hop and the prefetch count. 2016-10-03 16:51:28 +01:00
Noah Levitt fbb0173459 Merge pull request #166 from internetarchive/amqpWaiterStarter 
Don't wait to receive Umbra urls if Heritrix sends no url to Umbra
 2016-07-21 18:05:10 -05:00
Barbara Miller cbc73a6e8d check controller state, not frontier.isEmpty) 2016-07-21 11:40:58 -07:00
Barbara Miller 40d8ee77bf don't wait if h3 sends no url 2016-07-13 14:32:07 -07:00
Noah Levitt af2530edc4 Merge pull request #165 from galgeek/amqpWaiter 
AMQP URL Waiter
 2016-07-07 19:35:04 -05:00
Barbara Miller 4485b8c8c7 AMQP URL Waiter 2016-07-01 15:09:50 -07:00
Noah Levitt c04162a0f7 Merge pull request #164 from nlevitt/fix-test-errors 
Fixes for apparent build errors (extends #154)
 2016-06-08 14:31:58 -05:00
Noah Levitt 36ea1a9fb2 try to fix remaining serialization tests in oraclejdk8 by using ConcurrentSkipListMap instead of ConcurrentHashMap 2016-06-07 17:55:45 -05:00
Noah Levitt 99d87e8f23 hopefully fix serialization tests in oraclejdk8 by using TreeSet instead of HashSet in KeyedProperties.java 2016-06-07 14:53:11 -05:00
Noah Levitt cf9498d2ed clear the history store at the beginning of testBasics(), because the other test might have run first 2016-06-07 14:12:20 -05:00
Noah Levitt b09b078075 yeesh... "cd .." to get back to the right place to see the failure reports 2016-06-07 13:36:45 -05:00
Noah Levitt bc8b0baa59 Merge remote-tracking branch 'origin/master' into fix-test-errors 
* origin/master:
  Setup TravisCI
  Set fetch status on curis when testing link extraction
  No link extraction on URI not successfully downloaded
  Fixes issue #158 : Noisy alerts about 401s without auth challenge (#159)
  Make Content-Location header url INFERRED not REFFER hop type since Content-Location is not for redirection (#151)
  fixes for kafka 0.9 (?)
  upgrade to kafka 0.9
  somewhat ugly fix to handle exceptions from the bean browser like java.lang.RuntimeException: not implemented at org.archive.modules.fetcher.BdbCookieStore$RestrictedCollectionWrappedList.get(BdbCookieStore.java:92)
 2016-06-07 13:09:51 -05:00
Noah Levitt ac41da69c2 let me see what failed, travis 2016-06-07 13:07:24 -05:00
Noah Levitt ca3b65ed03 exclude dns from stats tests because the size of the dns records can vary depending on the environment 2016-06-06 15:43:02 -05:00
Noah Levitt f7d1ad7f1f Merge pull request #111 from nlevitt/bean-browse-cookiestore-exception 
somewhat ugly fix to handle exceptions from the bean browser like java.l...
 2016-06-06 11:11:45 -07:00
Noah Levitt bb10b4a80c Merge pull request #161 from kris-sigur/NoNonSuccessLinkExtraction 
No link extraction on URI not successfully downloaded
 2016-06-06 11:11:06 -07:00
Andrew Jackson aba94c7ac1 Fix package name. 2016-05-23 12:11:15 +01:00
Andrew Jackson d349c19298 Update Kryo to latest version. 2016-05-23 11:00:59 +01:00
jkafader c80f0cee2b Merge pull request #163 from nlevitt/kafka-0.9 
Kafka 0.9
 2016-05-09 12:01:53 -07:00
nruest ecd27e9e65 Setup TravisCI 2016-05-03 10:56:24 -07:00
Kristinn Sigurðsson 70867efdac Set fetch status on curis when testing link extraction 2016-05-03 15:58:50 +00:00
Kristinn Sigurðsson 08e3d055cf No link extraction on URI not successfully downloaded 2016-05-03 14:40:24 +00:00
Kristinn Sigurðsson c5848ae803 Fixes issue #158 : Noisy alerts about 401s without auth challenge (#159) 
* Fixes issue #158 : Noisy alerts about 401s without auth challenge

* Update test to account for non-fatal-error log not being empty on
non-auth 401s.
 2016-05-02 10:15:32 -07:00
Andy Jackson 8437a2724c Merge pull request #3 from ruebot/fix-test-errors 
Setup TravisCI
 2016-04-14 17:54:37 +00:00
nruest 3ee34d886f Setup TravisCI 2016-04-14 07:18:26 -04:00
Hunter a02a32cdfe Make Content-Location header url INFERRED not REFFER hop type since Content-Location is not for redirection (#151) 2016-04-13 15:19:21 -07:00