lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-07-22 07:21:36 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,776 Commits 40 Branches 31 Tags
bfbd61edd7cd6c1069205f13fe2e16cfeffc648b
Commit Graph
1776 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Hunter bfbd61edd7 Merge pull request #145 from vonrosen/ARI-4713 
Make some urls with whitespace acceptable to JavaScript extractor.
 2016-02-11 08:27:52 -08:00
Noah Levitt cbb5e3ab99 Simplify logic for urls with spaces, and make it better follow the pattern of UriUtils.isVeryLikelyUri(). Involves a subtle adjustment to the regex LIKELY_RELATIVE_URI_PATTERN to ensure the 2nd capturing group always gets the file extension. Also add a couple of tests of strings with spaces and file extensions that are not known good extensions. 2016-02-09 15:02:11 -08:00
Noah Levitt d80cf8244b Merge pull request #146 from vonrosen/ARI-4656-2 
More fixes for mutlipart form submission
 2016-02-09 10:48:07 -08:00
Hunter Stern 30ce2f09a0 Take care of possible NullPointerException 2016-02-09 08:22:39 -08:00
Hunter Stern e7c46753ce Fix formatting 2016-01-29 18:26:18 -08:00
Hunter Stern e83ff04e86 More fixes for https://webarchive.jira.com/browse/ARI-4656 2016-01-29 17:25:06 -08:00
Hunter Stern 088189ee8b Fix up code based on pull requests comments. Add test for urls with spaces in them. 2016-01-28 15:11:49 -08:00
Hunter Stern 4518bc2da8 Make sure no urls with whitespace and not having good extension slip through 2016-01-20 14:29:42 -08:00
Hunter Stern 12a88d6d23 Allow spaces in urls extracted from JS. 2016-01-20 10:44:20 -08:00
Hunter Stern 23885557bb Allow spaces in JavaScript urls, but only if they have a known good file extension 2016-01-19 18:07:05 -08:00
Hunter 34c90966a8 Merge pull request #143 from nlevitt/form-login-extra-inputs 
handle login forms with <input type="text"> fields in addition to use…
 2016-01-19 08:37:05 -08:00
Noah Levitt 25a7aa9c8d Merge pull request #141 from vonrosen/ARI-4659 
Disable SNI for a request if that request failed due to an SNI error …
 2016-01-15 11:13:42 -08:00
Noah Levitt cc97f01656 Merge pull request #123 from kris-sigur/HostsReport 
Hosts report
 2016-01-15 10:33:07 -08:00
Noah Levitt 70c688a4f5 Merge pull request #135 from vonrosen/ARI-4562 
Allow KnowledgableExtractorJS to parse out youtube watch from youtube…
 2016-01-15 10:15:23 -08:00
Noah Levitt e92e5f2bb3 Merge branch 'master' into form-login-extra-inputs 
* also fix new FormLoginProcessorTest.testFormLoginExtraInputs() in context of form-login-multipart changes

* master:
  fix NullPointerException when using old HtmlFormCredential login
  to make this branch only about multipart/form-data, undo the changes to handle more complicated forms with extra input fields; add unit test; store CrawlURI submit data in http-library agnostic way and remove httpcomponents-specific code from HTMLForm, refactor creation of post data accordingly
  Be sure to return at least one of the candidates
  Remove unneeded constant
  Make sure to check form encoding type when addign form to curi data
  More changes for https://webarchive.jira.com/browse/ARI-4656
  More changes
  Save enctype of form in submission curi
  Add ability to submit forms with enctype="multipart/form-data"
  If multiple username inputs in login form, favor the one with the word 'login' in it.
 2016-01-15 09:39:24 -08:00
Hunter 02e3a554f0 Merge pull request #142 from nlevitt/form-login-multipart 
Form login multipart
 2016-01-15 08:19:23 -08:00
Noah Levitt c1abbfa5bd handle login forms with <input type="text"> fields in addition to username (for this to work there must be exactly one input field whose name contains the string "username") 2016-01-14 18:02:54 -08:00
Noah Levitt 368d586eb1 fix NullPointerException when using old HtmlFormCredential login 2016-01-14 16:00:08 -08:00
Noah Levitt 3b12a751e1 to make this branch only about multipart/form-data, undo the changes to handle more complicated forms with extra input fields; add unit test; store CrawlURI submit data in http-library agnostic way and remove httpcomponents-specific code from HTMLForm, refactor creation of post data accordingly 2016-01-14 12:17:19 -08:00
Hunter SternandNoah Levitt 309847cc40 Be sure to return at least one of the candidates 2016-01-14 12:05:38 -08:00
Hunter SternandNoah Levitt 704ec62487 Remove unneeded constant 2016-01-14 12:05:38 -08:00
Hunter SternandNoah Levitt a2a39fe052 Make sure to check form encoding type when addign form to curi data 2016-01-14 12:05:38 -08:00
Hunter SternandNoah Levitt 93bca3d55c More changes for https://webarchive.jira.com/browse/ARI-4656 2016-01-14 12:05:38 -08:00
Hunter SternandNoah Levitt 4997014b34 More changes 2016-01-14 12:05:38 -08:00
Hunter SternandNoah Levitt ad329ceea8 Save enctype of form in submission curi 2016-01-14 12:05:38 -08:00
Hunter SternandNoah Levitt 34cffd512a Add ability to submit forms with enctype="multipart/form-data" 2016-01-14 12:05:38 -08:00
Hunter SternandNoah Levitt afa4f6535b If multiple username inputs in login form, favor the one with the word 'login' in it. 2016-01-14 12:05:38 -08:00
Hunter Stern 5f630e4268 Make regex more inclusive for parsing out youtube videoId and add youtube watch page as relative to base 2016-01-14 09:03:42 -08:00
Hunter Stern 7bcdf03a78 Merge branch 'master' of github.com:internetarchive/heritrix3 into ARI-4562 2016-01-14 08:48:39 -08:00
Hunter eeeea6f0c4 Merge pull request #122 from nlevitt/form-login-checked-boxes 
only submit checkbox and radio button form fields if they are on by d…
 2016-01-14 08:00:49 -08:00
Adam Miller bf2a887688 Merge pull request #138 from nlevitt/novel-quotas 
crawl level and host level limits on *novel* (not deduplicated) bytes and urls
 2016-01-13 16:25:24 -08:00
Noah Levitt 9e4c7f0024 Merge branch 'master' into novel-quotas 
* master:
  license header
  check that sourceTag of CrawlURI actually matches configured sourceTag
  remove already-outdated stuff from javadoc
  handle multiple clauses for same user agent in robots.txt
  Hook in submitted seeds properly.
  avoid spurious logging
  try very hard to start url consumer, and therefore bind the queue to the routing key, so that no messages are dropped, before crawling starts (should always work unless rabbitmq is down); some other tweaks for clarity and stability
 2016-01-13 13:44:54 -08:00
Adam Miller b82896927a Merge pull request #132 from nlevitt/amqp-url-receiver-changes 
try very hard to start url consumer, and therefore bind the queue to …
 2016-01-13 13:35:43 -08:00
Noah Levitt df5748d1f1 Merge pull request #136 from ukwa/amqp-declare-recieved-seeds 
Register seeds send in via AMQP
 2016-01-13 13:27:43 -08:00
Noah Levitt 22336fb71b fix stats in unusual case of "failed" fetch with response > 0 (only case where this can happen currently is if basic auth is configured for a url, but fails and url returns "401 Unauthorized") 2016-01-12 17:34:27 -08:00
Hunter Stern 73be17af06 Disable SNI for a request if that request failed due to an SNI error 'javax.net.ssl.SSLProtocolException: handshake alert: unrecognized_name' 2016-01-12 16:58:48 -08:00
Adam Miller 3dbf754d56 Merge pull request #137 from nlevitt/seed-limits 
SourceSeedDecideRule, SeedLimitsEnforcer
 2016-01-12 16:39:17 -08:00
Noah Levitt 65a1077425 license header 2016-01-12 14:59:45 -08:00
Neil Minton 32dd07375d Merge pull request #139 from nlevitt/robots-sep-sections 
handle multiple clauses for same user agent in robots.txt
 2016-01-11 18:39:25 -06:00
Noah Levitt 8918ce99c6 check that sourceTag of CrawlURI actually matches configured sourceTag 2015-12-15 11:24:53 -08:00
Noah Levitt 7a8a614804 avoid exception in case applyToSubdomains is enabled and configured host is an ip address 2015-12-15 11:06:33 -08:00
Noah Levitt a95519543d avoid exception in case applyToSubdomains is enabled and CrawlURI host is a nonstandard host like "dns:" in HostQuotaEnforcer 2015-12-15 11:02:38 -08:00
Noah Levitt e3e6bfeb11 new HostQuotaEnforcer option applyToSubdomains, to additionally apply the quotas (separately) to each subdomain of the configured host 2015-12-14 18:17:44 -08:00
Noah Levitt 2f82d43110 remove already-outdated stuff from javadoc 2015-12-14 15:24:08 -08:00
Noah Levitt 6f9c235fd0 fix javadoc 2015-12-14 15:23:16 -08:00
Noah Levitt 013029d9ad fix annotation copy-paste-o 2015-12-14 15:22:51 -08:00
Noah Levitt 99c5af0cf2 comment out currently failing test (needs webarchive-commons pull request merged) 2015-12-14 10:46:50 -08:00
Noah Levitt 2a56886c01 add license header 2015-12-14 10:46:14 -08:00
Noah Levitt 8deead15ec new class to enforce quotas on a host 2015-12-11 17:02:32 -08:00
Noah Levitt f1173789f7 fix test now that ContentDigestHistoryTest.shouldProcess() is smarter 2015-12-11 16:13:17 -08:00