lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-22 14:36:37 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,819 Commits 40 Branches 31 Tags
cbc73a6e8d6525430aded73048c04d034a098a59
Commit Graph
1819 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Barbara Miller cbc73a6e8d check controller state, not frontier.isEmpty) 2016-07-21 11:40:58 -07:00
Barbara Miller 40d8ee77bf don't wait if h3 sends no url 2016-07-13 14:32:07 -07:00
Noah Levitt af2530edc4 Merge pull request #165 from galgeek/amqpWaiter 
AMQP URL Waiter
 2016-07-07 19:35:04 -05:00
Barbara Miller 4485b8c8c7 AMQP URL Waiter 2016-07-01 15:09:50 -07:00
Noah Levitt c04162a0f7 Merge pull request #164 from nlevitt/fix-test-errors 
Fixes for apparent build errors (extends #154)
 2016-06-08 14:31:58 -05:00
Noah Levitt 36ea1a9fb2 try to fix remaining serialization tests in oraclejdk8 by using ConcurrentSkipListMap instead of ConcurrentHashMap 2016-06-07 17:55:45 -05:00
Noah Levitt 99d87e8f23 hopefully fix serialization tests in oraclejdk8 by using TreeSet instead of HashSet in KeyedProperties.java 2016-06-07 14:53:11 -05:00
Noah Levitt cf9498d2ed clear the history store at the beginning of testBasics(), because the other test might have run first 2016-06-07 14:12:20 -05:00
Noah Levitt b09b078075 yeesh... "cd .." to get back to the right place to see the failure reports 2016-06-07 13:36:45 -05:00
Noah Levitt bc8b0baa59 Merge remote-tracking branch 'origin/master' into fix-test-errors 
* origin/master:
  Setup TravisCI
  Set fetch status on curis when testing link extraction
  No link extraction on URI not successfully downloaded
  Fixes issue #158 : Noisy alerts about 401s without auth challenge (#159)
  Make Content-Location header url INFERRED not REFFER hop type since Content-Location is not for redirection (#151)
  fixes for kafka 0.9 (?)
  upgrade to kafka 0.9
  somewhat ugly fix to handle exceptions from the bean browser like java.lang.RuntimeException: not implemented at org.archive.modules.fetcher.BdbCookieStore$RestrictedCollectionWrappedList.get(BdbCookieStore.java:92)
 2016-06-07 13:09:51 -05:00
Noah Levitt ac41da69c2 let me see what failed, travis 2016-06-07 13:07:24 -05:00
Noah Levitt ca3b65ed03 exclude dns from stats tests because the size of the dns records can vary depending on the environment 2016-06-06 15:43:02 -05:00
Noah Levitt f7d1ad7f1f Merge pull request #111 from nlevitt/bean-browse-cookiestore-exception 
somewhat ugly fix to handle exceptions from the bean browser like java.l...
 2016-06-06 11:11:45 -07:00
Noah Levitt bb10b4a80c Merge pull request #161 from kris-sigur/NoNonSuccessLinkExtraction 
No link extraction on URI not successfully downloaded
 2016-06-06 11:11:06 -07:00
jkafader c80f0cee2b Merge pull request #163 from nlevitt/kafka-0.9 
Kafka 0.9
 2016-05-09 12:01:53 -07:00
nruest ecd27e9e65 Setup TravisCI 2016-05-03 10:56:24 -07:00
Kristinn Sigurðsson 70867efdac Set fetch status on curis when testing link extraction 2016-05-03 15:58:50 +00:00
Kristinn Sigurðsson 08e3d055cf No link extraction on URI not successfully downloaded 2016-05-03 14:40:24 +00:00
Kristinn Sigurðsson c5848ae803 Fixes issue #158 : Noisy alerts about 401s without auth challenge (#159) 
* Fixes issue #158 : Noisy alerts about 401s without auth challenge

* Update test to account for non-fatal-error log not being empty on
non-auth 401s.
 2016-05-02 10:15:32 -07:00
Andy Jackson 8437a2724c Merge pull request #3 from ruebot/fix-test-errors 
Setup TravisCI
 2016-04-14 17:54:37 +00:00
nruest 3ee34d886f Setup TravisCI 2016-04-14 07:18:26 -04:00
Hunter a02a32cdfe Make Content-Location header url INFERRED not REFFER hop type since Content-Location is not for redirection (#151) 2016-04-13 15:19:21 -07:00
Andrew Jackson 48f5e71e76 Add delay to avoid race condition during test. 2016-04-07 21:48:20 +01:00
Andrew Jackson 9145fedd40 Fix unexpected change in count of novel bytes in one selftest. 2016-04-02 20:57:05 +01:00
Andrew Jackson cdb79b8868 Fix serialisation tests by defaulting to TreeMaps. 2016-04-02 20:35:37 +01:00
Noah Levitt 124907341f Merge pull request #152 from ukwa/switch-to-java-7 
Switch to Java 7
 2016-03-08 16:47:58 -08:00
Andrew Jackson 8ff09dc769 Switch to Java 1.7 2016-03-08 23:53:59 +00:00
Adam Miller 7f752d8d45 Merge pull request #150 from nlevitt/ari-4638 
various changes to amqp publish and receive
 2016-03-08 09:26:58 -08:00
Hunter 0dc95ab7f7 Merge pull request #144 from nlevitt/ari-4528 
run received urls through the candidates processor, to check scope an…
 2016-02-29 14:24:43 -08:00
Kristinn Sigurðsson ae413b1d9e Merge pull request #149 from eleclerc/patch-1 
Update to ExtractorHTML.java for cond. comments
 2016-02-19 09:49:34 +00:00
Noah Levitt 9bdafc3c33 remove stray line 2016-02-18 18:19:44 -08:00
Noah Levitt bc9a6cc640 new parameter extraInfo, "Arbitrary additional information to include in the json payload", on AMQPPublishProcessor; used KeyedProperties so parameters can be overridden with the sheet/overlay system 2016-02-18 18:17:30 -08:00
Noah Levitt f4b8468564 set custom http headers last, so they override headers set elsewhere (e.g. cookies) 2016-02-18 18:16:03 -08:00
Noah Levitt a64551296a new boolean forceFetch setting; REQUEST_HEADER_BLACKLIST to exclude certain http headers, since with https://github.com/internetarchive/umbra/pull/55 umbra sends over more stuff 2016-02-18 18:15:22 -08:00
Noah Levitt 8a6a49f7fc ARI-4528 avoid this exception 
2016-02-18 01:44:46.434 SEVERE thread-51 org.archive.crawler.spring.SheetOverlaysManager.applyOverlaysTo() problem determining whether to apply overlays, so not applying [ignoreCookies] to https://static.xx.fbcdn.net/rsrc.php/v2/yY/r/z9xrgSKFbeZ.js
java.lang.NullPointerException
        at org.archive.spring.KeyedProperties.get(KeyedProperties.java:65)
        at org.archive.modules.deciderules.DecideRule.getEnabled(DecideRule.java:38)
        at org.archive.modules.deciderules.DecideRule.decisionFor(DecideRule.java:57)
        at org.archive.modules.deciderules.DecideRule.accepts(DecideRule.java:77)
        at org.archive.crawler.spring.SheetOverlaysManager.applyOverlaysTo(SheetOverlaysManager.java:331)
        at org.archive.crawler.postprocessor.CandidatesProcessor.runCandidateChain(CandidatesProcessor.java:170)
        at org.archive.crawler.frontier.AMQPUrlReceiver$UrlConsumer.handleDelivery(AMQPUrlReceiver.java:321)
        at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:140)
        at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:76)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
 2016-02-17 17:53:05 -08:00
Eric Leclerc 5f431de215 Update to ExtractorHTML.java for cond. comments 
Lot's of websites are using "Downlevel Revealed" conditional comments with a twist to have a page with valid html. this is an update to the RegEx to allow such a case. ex:

    <!--[if expression]><!-->
      HTML
    <!--<![endif]-->

- reference: https://css-tricks.com/downlevel-hidden-downlevel-revealed/
- reference: http://www.sitepoint.com/web-foundations/internet-explorer-conditional-comments/
- example site where this type of tag can be found: https://www.canada.ca/index.html
 2016-02-17 14:49:50 -05:00
Noah Levitt 847400776b Merge pull request #148 from vonrosen/ARI-4738 
Don't canonicalize source tag so that SourceSeedDecideRule will work.…
 2016-02-17 11:43:30 -08:00
Hunter Stern 3e219af148 Set source tag to uri as it exists before possible prepending of scheme. 2016-02-17 08:10:07 -08:00
Hunter Stern b9b2343bc0 Don't canonicalize source tag so that SourceSeedDecideRule will work. Trailing slash of canonicalized seed currently causes it not to work. 2016-02-16 08:52:16 -08:00
Hunter bfbd61edd7 Merge pull request #145 from vonrosen/ARI-4713 
Make some urls with whitespace acceptable to JavaScript extractor.
 2016-02-11 08:27:52 -08:00
Noah Levitt cbb5e3ab99 Simplify logic for urls with spaces, and make it better follow the pattern of UriUtils.isVeryLikelyUri(). Involves a subtle adjustment to the regex LIKELY_RELATIVE_URI_PATTERN to ensure the 2nd capturing group always gets the file extension. Also add a couple of tests of strings with spaces and file extensions that are not known good extensions. 2016-02-09 15:02:11 -08:00
Noah Levitt d80cf8244b Merge pull request #146 from vonrosen/ARI-4656-2 
More fixes for mutlipart form submission
 2016-02-09 10:48:07 -08:00
Hunter Stern 30ce2f09a0 Take care of possible NullPointerException 2016-02-09 08:22:39 -08:00
Noah Levitt 44139e7e3b fixes for kafka 0.9 (?) 2016-02-08 16:08:00 -08:00
Noah Levitt f665ce844f upgrade to kafka 0.9 2016-02-08 12:49:47 -08:00
Hunter Stern e7c46753ce Fix formatting 2016-01-29 18:26:18 -08:00
Hunter Stern e83ff04e86 More fixes for https://webarchive.jira.com/browse/ARI-4656 2016-01-29 17:25:06 -08:00
Hunter Stern 088189ee8b Fix up code based on pull requests comments. Add test for urls with spaces in them. 2016-01-28 15:11:49 -08:00
Hunter Stern 4518bc2da8 Make sure no urls with whitespace and not having good extension slip through 2016-01-20 14:29:42 -08:00
Hunter Stern 12a88d6d23 Allow spaces in urls extracted from JS. 2016-01-20 10:44:20 -08:00