lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-24 07:26:36 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,791 Commits 40 Branches 31 Tags
e1476401ac71290e94110eca235173b6662dad82
Commit Graph
1791 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Andrew Jackson e1476401ac Merged with upstream changes. 2016-03-08 23:50:07 +00:00
Adam Miller 7f752d8d45 Merge pull request #150 from nlevitt/ari-4638 
various changes to amqp publish and receive
 2016-03-08 09:26:58 -08:00
Hunter 0dc95ab7f7 Merge pull request #144 from nlevitt/ari-4528 
run received urls through the candidates processor, to check scope an…
 2016-02-29 14:24:43 -08:00
Kristinn Sigurðsson ae413b1d9e Merge pull request #149 from eleclerc/patch-1 
Update to ExtractorHTML.java for cond. comments
 2016-02-19 09:49:34 +00:00
Noah Levitt 9bdafc3c33 remove stray line 2016-02-18 18:19:44 -08:00
Noah Levitt bc9a6cc640 new parameter extraInfo, "Arbitrary additional information to include in the json payload", on AMQPPublishProcessor; used KeyedProperties so parameters can be overridden with the sheet/overlay system 2016-02-18 18:17:30 -08:00
Noah Levitt f4b8468564 set custom http headers last, so they override headers set elsewhere (e.g. cookies) 2016-02-18 18:16:03 -08:00
Noah Levitt a64551296a new boolean forceFetch setting; REQUEST_HEADER_BLACKLIST to exclude certain http headers, since with https://github.com/internetarchive/umbra/pull/55 umbra sends over more stuff 2016-02-18 18:15:22 -08:00
Noah Levitt 8a6a49f7fc ARI-4528 avoid this exception 
2016-02-18 01:44:46.434 SEVERE thread-51 org.archive.crawler.spring.SheetOverlaysManager.applyOverlaysTo() problem determining whether to apply overlays, so not applying [ignoreCookies] to https://static.xx.fbcdn.net/rsrc.php/v2/yY/r/z9xrgSKFbeZ.js
java.lang.NullPointerException
        at org.archive.spring.KeyedProperties.get(KeyedProperties.java:65)
        at org.archive.modules.deciderules.DecideRule.getEnabled(DecideRule.java:38)
        at org.archive.modules.deciderules.DecideRule.decisionFor(DecideRule.java:57)
        at org.archive.modules.deciderules.DecideRule.accepts(DecideRule.java:77)
        at org.archive.crawler.spring.SheetOverlaysManager.applyOverlaysTo(SheetOverlaysManager.java:331)
        at org.archive.crawler.postprocessor.CandidatesProcessor.runCandidateChain(CandidatesProcessor.java:170)
        at org.archive.crawler.frontier.AMQPUrlReceiver$UrlConsumer.handleDelivery(AMQPUrlReceiver.java:321)
        at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:140)
        at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:76)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
 2016-02-17 17:53:05 -08:00
Eric Leclerc 5f431de215 Update to ExtractorHTML.java for cond. comments 
Lot's of websites are using "Downlevel Revealed" conditional comments with a twist to have a page with valid html. this is an update to the RegEx to allow such a case. ex:

    <!--[if expression]><!-->
      HTML
    <!--<![endif]-->

- reference: https://css-tricks.com/downlevel-hidden-downlevel-revealed/
- reference: http://www.sitepoint.com/web-foundations/internet-explorer-conditional-comments/
- example site where this type of tag can be found: https://www.canada.ca/index.html
 2016-02-17 14:49:50 -05:00
Noah Levitt 847400776b Merge pull request #148 from vonrosen/ARI-4738 
Don't canonicalize source tag so that SourceSeedDecideRule will work.…
 2016-02-17 11:43:30 -08:00
Hunter Stern 3e219af148 Set source tag to uri as it exists before possible prepending of scheme. 2016-02-17 08:10:07 -08:00
Hunter Stern b9b2343bc0 Don't canonicalize source tag so that SourceSeedDecideRule will work. Trailing slash of canonicalized seed currently causes it not to work. 2016-02-16 08:52:16 -08:00
Andrew Jackson 7b3c9e51e3 Add support for setting the hop and the prefetch count. 2016-02-12 11:20:51 +00:00
Hunter bfbd61edd7 Merge pull request #145 from vonrosen/ARI-4713 
Make some urls with whitespace acceptable to JavaScript extractor.
 2016-02-11 08:27:52 -08:00
Noah Levitt cbb5e3ab99 Simplify logic for urls with spaces, and make it better follow the pattern of UriUtils.isVeryLikelyUri(). Involves a subtle adjustment to the regex LIKELY_RELATIVE_URI_PATTERN to ensure the 2nd capturing group always gets the file extension. Also add a couple of tests of strings with spaces and file extensions that are not known good extensions. 2016-02-09 15:02:11 -08:00
Noah Levitt d80cf8244b Merge pull request #146 from vonrosen/ARI-4656-2 
More fixes for mutlipart form submission
 2016-02-09 10:48:07 -08:00
Hunter Stern 30ce2f09a0 Take care of possible NullPointerException 2016-02-09 08:22:39 -08:00
Hunter Stern e7c46753ce Fix formatting 2016-01-29 18:26:18 -08:00
Hunter Stern e83ff04e86 More fixes for https://webarchive.jira.com/browse/ARI-4656 2016-01-29 17:25:06 -08:00
Hunter Stern 088189ee8b Fix up code based on pull requests comments. Add test for urls with spaces in them. 2016-01-28 15:11:49 -08:00
Hunter Stern 4518bc2da8 Make sure no urls with whitespace and not having good extension slip through 2016-01-20 14:29:42 -08:00
Hunter Stern 12a88d6d23 Allow spaces in urls extracted from JS. 2016-01-20 10:44:20 -08:00
Hunter Stern 23885557bb Allow spaces in JavaScript urls, but only if they have a known good file extension 2016-01-19 18:07:05 -08:00
Hunter 34c90966a8 Merge pull request #143 from nlevitt/form-login-extra-inputs 
handle login forms with <input type="text"> fields in addition to use…
 2016-01-19 08:37:05 -08:00
Noah Levitt 13322111b5 run received urls through the candidates processor, to check scope and schedule them with the frontier 2016-01-15 17:46:27 -08:00
Noah Levitt 25a7aa9c8d Merge pull request #141 from vonrosen/ARI-4659 
Disable SNI for a request if that request failed due to an SNI error …
 2016-01-15 11:13:42 -08:00
Noah Levitt cc97f01656 Merge pull request #123 from kris-sigur/HostsReport 
Hosts report
 2016-01-15 10:33:07 -08:00
Noah Levitt 70c688a4f5 Merge pull request #135 from vonrosen/ARI-4562 
Allow KnowledgableExtractorJS to parse out youtube watch from youtube…
 2016-01-15 10:15:23 -08:00
Noah Levitt e92e5f2bb3 Merge branch 'master' into form-login-extra-inputs 
* also fix new FormLoginProcessorTest.testFormLoginExtraInputs() in context of form-login-multipart changes

* master:
  fix NullPointerException when using old HtmlFormCredential login
  to make this branch only about multipart/form-data, undo the changes to handle more complicated forms with extra input fields; add unit test; store CrawlURI submit data in http-library agnostic way and remove httpcomponents-specific code from HTMLForm, refactor creation of post data accordingly
  Be sure to return at least one of the candidates
  Remove unneeded constant
  Make sure to check form encoding type when addign form to curi data
  More changes for https://webarchive.jira.com/browse/ARI-4656
  More changes
  Save enctype of form in submission curi
  Add ability to submit forms with enctype="multipart/form-data"
  If multiple username inputs in login form, favor the one with the word 'login' in it.
 2016-01-15 09:39:24 -08:00
Hunter 02e3a554f0 Merge pull request #142 from nlevitt/form-login-multipart 
Form login multipart
 2016-01-15 08:19:23 -08:00
Noah Levitt c1abbfa5bd handle login forms with <input type="text"> fields in addition to username (for this to work there must be exactly one input field whose name contains the string "username") 2016-01-14 18:02:54 -08:00
Noah Levitt 368d586eb1 fix NullPointerException when using old HtmlFormCredential login 2016-01-14 16:00:08 -08:00
Noah Levitt 3b12a751e1 to make this branch only about multipart/form-data, undo the changes to handle more complicated forms with extra input fields; add unit test; store CrawlURI submit data in http-library agnostic way and remove httpcomponents-specific code from HTMLForm, refactor creation of post data accordingly 2016-01-14 12:17:19 -08:00
Hunter Stern 309847cc40 Be sure to return at least one of the candidates 2016-01-14 12:05:38 -08:00
Hunter Stern 704ec62487 Remove unneeded constant 2016-01-14 12:05:38 -08:00
Hunter Stern a2a39fe052 Make sure to check form encoding type when addign form to curi data 2016-01-14 12:05:38 -08:00
Hunter Stern 93bca3d55c More changes for https://webarchive.jira.com/browse/ARI-4656 2016-01-14 12:05:38 -08:00
Hunter Stern 4997014b34 More changes 2016-01-14 12:05:38 -08:00
Hunter Stern ad329ceea8 Save enctype of form in submission curi 2016-01-14 12:05:38 -08:00
Hunter Stern 34cffd512a Add ability to submit forms with enctype="multipart/form-data" 2016-01-14 12:05:38 -08:00
Hunter Stern afa4f6535b If multiple username inputs in login form, favor the one with the word 'login' in it. 2016-01-14 12:05:38 -08:00
Hunter Stern 5f630e4268 Make regex more inclusive for parsing out youtube videoId and add youtube watch page as relative to base 2016-01-14 09:03:42 -08:00
Hunter Stern 7bcdf03a78 Merge branch 'master' of github.com:internetarchive/heritrix3 into ARI-4562 2016-01-14 08:48:39 -08:00
Hunter eeeea6f0c4 Merge pull request #122 from nlevitt/form-login-checked-boxes 
only submit checkbox and radio button form fields if they are on by d…
 2016-01-14 08:00:49 -08:00
Adam Miller bf2a887688 Merge pull request #138 from nlevitt/novel-quotas 
crawl level and host level limits on *novel* (not deduplicated) bytes and urls
 2016-01-13 16:25:24 -08:00
Noah Levitt 9e4c7f0024 Merge branch 'master' into novel-quotas 
* master:
  license header
  check that sourceTag of CrawlURI actually matches configured sourceTag
  remove already-outdated stuff from javadoc
  handle multiple clauses for same user agent in robots.txt
  Hook in submitted seeds properly.
  avoid spurious logging
  try very hard to start url consumer, and therefore bind the queue to the routing key, so that no messages are dropped, before crawling starts (should always work unless rabbitmq is down); some other tweaks for clarity and stability
 2016-01-13 13:44:54 -08:00
Adam Miller b82896927a Merge pull request #132 from nlevitt/amqp-url-receiver-changes 
try very hard to start url consumer, and therefore bind the queue to …
 2016-01-13 13:35:43 -08:00
Noah Levitt df5748d1f1 Merge pull request #136 from ukwa/amqp-declare-recieved-seeds 
Register seeds send in via AMQP
 2016-01-13 13:27:43 -08:00
Noah Levitt 22336fb71b fix stats in unusual case of "failed" fetch with response > 0 (only case where this can happen currently is if basic auth is configured for a url, but fails and url returns "401 Unauthorized") 2016-01-12 17:34:27 -08:00