lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-23 06:56:41 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,866 Commits 40 Branches 31 Tags
ARI-5439
Commit Graph
754 Commits
Author SHA1 Message Date
Barbara Miller d4b8566fea updateRobots deem S_CONNECT_FAILED 404 2017-10-06 12:55:29 -07:00
Noah Levitt 3892c17b56 ughhh appease java 8 javadoc rules 2017-06-15 17:29:22 -07:00
Noah Levitt 87a15bddc8 Merge pull request #168 from internetarchive/gh-167 
fix for race-condition when first using the WARC writers https://gith…
 2017-06-07 18:15:19 -07:00
Neil Minton 10541184b5 Clean up after code review. 
- Parse with string split instead of regex.
- Move srcset logic up to processEmbed(CrawlURI, CharSequence, CharSequence, Hop).
 2017-06-02 11:53:16 -07:00
Neil Minton 4c0e52af61 Add parsing for srcset attributes. 
- Includes Unit Tests
 2017-05-01 17:17:05 -07:00
Noah Levitt 0b4671c291 fix for race-condition when first using the WARC writers https://github.com/internetarchive/heritrix3/issues/167 2016-08-15 13:36:33 -07:00
Noah Levitt cf9498d2ed clear the history store at the beginning of testBasics(), because the other test might have run first 2016-06-07 14:12:20 -05:00
Noah Levitt bc8b0baa59 Merge remote-tracking branch 'origin/master' into fix-test-errors 
* origin/master:
  Setup TravisCI
  Set fetch status on curis when testing link extraction
  No link extraction on URI not successfully downloaded
  Fixes issue #158 : Noisy alerts about 401s without auth challenge (#159)
  Make Content-Location header url INFERRED not REFFER hop type since Content-Location is not for redirection (#151)
  fixes for kafka 0.9 (?)
  upgrade to kafka 0.9
  somewhat ugly fix to handle exceptions from the bean browser like java.lang.RuntimeException: not implemented at org.archive.modules.fetcher.BdbCookieStore$RestrictedCollectionWrappedList.get(BdbCookieStore.java:92)
 2016-06-07 13:09:51 -05:00
Kristinn Sigurðsson 70867efdac Set fetch status on curis when testing link extraction 2016-05-03 15:58:50 +00:00
Kristinn Sigurðsson 08e3d055cf No link extraction on URI not successfully downloaded 2016-05-03 14:40:24 +00:00
Kristinn Sigurðsson c5848ae803 Fixes issue #158 : Noisy alerts about 401s without auth challenge (#159) 
* Fixes issue #158 : Noisy alerts about 401s without auth challenge

* Update test to account for non-fatal-error log not being empty on
non-auth 401s.
 2016-05-02 10:15:32 -07:00
Hunter a02a32cdfe Make Content-Location header url INFERRED not REFFER hop type since Content-Location is not for redirection (#151) 2016-04-13 15:19:21 -07:00
Andrew Jackson 48f5e71e76 Add delay to avoid race condition during test. 2016-04-07 21:48:20 +01:00
Andrew Jackson cdb79b8868 Fix serialisation tests by defaulting to TreeMaps. 2016-04-02 20:35:37 +01:00
Adam Miller 7f752d8d45 Merge pull request #150 from nlevitt/ari-4638 
various changes to amqp publish and receive
 2016-03-08 09:26:58 -08:00
Noah Levitt f4b8468564 set custom http headers last, so they override headers set elsewhere (e.g. cookies) 2016-02-18 18:16:03 -08:00
Eric Leclerc 5f431de215 Update to ExtractorHTML.java for cond. comments 
Lot's of websites are using "Downlevel Revealed" conditional comments with a twist to have a page with valid html. this is an update to the RegEx to allow such a case. ex:

    <!--[if expression]><!-->
      HTML
    <!--<![endif]-->

- reference: https://css-tricks.com/downlevel-hidden-downlevel-revealed/
- reference: http://www.sitepoint.com/web-foundations/internet-explorer-conditional-comments/
- example site where this type of tag can be found: https://www.canada.ca/index.html
 2016-02-17 14:49:50 -05:00
Hunter Stern 3e219af148 Set source tag to uri as it exists before possible prepending of scheme. 2016-02-17 08:10:07 -08:00
Hunter Stern b9b2343bc0 Don't canonicalize source tag so that SourceSeedDecideRule will work. Trailing slash of canonicalized seed currently causes it not to work. 2016-02-16 08:52:16 -08:00
Hunter bfbd61edd7 Merge pull request #145 from vonrosen/ARI-4713 
Make some urls with whitespace acceptable to JavaScript extractor.
 2016-02-11 08:27:52 -08:00
Noah Levitt cbb5e3ab99 Simplify logic for urls with spaces, and make it better follow the pattern of UriUtils.isVeryLikelyUri(). Involves a subtle adjustment to the regex LIKELY_RELATIVE_URI_PATTERN to ensure the 2nd capturing group always gets the file extension. Also add a couple of tests of strings with spaces and file extensions that are not known good extensions. 2016-02-09 15:02:11 -08:00
Hunter Stern 30ce2f09a0 Take care of possible NullPointerException 2016-02-09 08:22:39 -08:00
Hunter Stern e7c46753ce Fix formatting 2016-01-29 18:26:18 -08:00
Hunter Stern e83ff04e86 More fixes for https://webarchive.jira.com/browse/ARI-4656 2016-01-29 17:25:06 -08:00
Hunter Stern 088189ee8b Fix up code based on pull requests comments. Add test for urls with spaces in them. 2016-01-28 15:11:49 -08:00
Hunter Stern 12a88d6d23 Allow spaces in urls extracted from JS. 2016-01-20 10:44:20 -08:00
Hunter Stern 23885557bb Allow spaces in JavaScript urls, but only if they have a known good file extension 2016-01-19 18:07:05 -08:00
Hunter 34c90966a8 Merge pull request #143 from nlevitt/form-login-extra-inputs 
handle login forms with <input type="text"> fields in addition to use…
 2016-01-19 08:37:05 -08:00
Noah Levitt 25a7aa9c8d Merge pull request #141 from vonrosen/ARI-4659 
Disable SNI for a request if that request failed due to an SNI error …
 2016-01-15 11:13:42 -08:00
Noah Levitt e92e5f2bb3 Merge branch 'master' into form-login-extra-inputs 
* also fix new FormLoginProcessorTest.testFormLoginExtraInputs() in context of form-login-multipart changes

* master:
  fix NullPointerException when using old HtmlFormCredential login
  to make this branch only about multipart/form-data, undo the changes to handle more complicated forms with extra input fields; add unit test; store CrawlURI submit data in http-library agnostic way and remove httpcomponents-specific code from HTMLForm, refactor creation of post data accordingly
  Be sure to return at least one of the candidates
  Remove unneeded constant
  Make sure to check form encoding type when addign form to curi data
  More changes for https://webarchive.jira.com/browse/ARI-4656
  More changes
  Save enctype of form in submission curi
  Add ability to submit forms with enctype="multipart/form-data"
  If multiple username inputs in login form, favor the one with the word 'login' in it.
 2016-01-15 09:39:24 -08:00
Noah Levitt c1abbfa5bd handle login forms with <input type="text"> fields in addition to username (for this to work there must be exactly one input field whose name contains the string "username") 2016-01-14 18:02:54 -08:00
Noah Levitt 368d586eb1 fix NullPointerException when using old HtmlFormCredential login 2016-01-14 16:00:08 -08:00
Noah Levitt 3b12a751e1 to make this branch only about multipart/form-data, undo the changes to handle more complicated forms with extra input fields; add unit test; store CrawlURI submit data in http-library agnostic way and remove httpcomponents-specific code from HTMLForm, refactor creation of post data accordingly 2016-01-14 12:17:19 -08:00
Hunter Stern 309847cc40 Be sure to return at least one of the candidates 2016-01-14 12:05:38 -08:00
Hunter Stern 704ec62487 Remove unneeded constant 2016-01-14 12:05:38 -08:00
Hunter Stern a2a39fe052 Make sure to check form encoding type when addign form to curi data 2016-01-14 12:05:38 -08:00
Hunter Stern 93bca3d55c More changes for https://webarchive.jira.com/browse/ARI-4656 2016-01-14 12:05:38 -08:00
Hunter Stern 4997014b34 More changes 2016-01-14 12:05:38 -08:00
Hunter Stern ad329ceea8 Save enctype of form in submission curi 2016-01-14 12:05:38 -08:00
Hunter Stern 34cffd512a Add ability to submit forms with enctype="multipart/form-data" 2016-01-14 12:05:38 -08:00
Hunter Stern afa4f6535b If multiple username inputs in login form, favor the one with the word 'login' in it. 2016-01-14 12:05:38 -08:00
Hunter eeeea6f0c4 Merge pull request #122 from nlevitt/form-login-checked-boxes 
only submit checkbox and radio button form fields if they are on by d…
 2016-01-14 08:00:49 -08:00
Noah Levitt 9e4c7f0024 Merge branch 'master' into novel-quotas 
* master:
  license header
  check that sourceTag of CrawlURI actually matches configured sourceTag
  remove already-outdated stuff from javadoc
  handle multiple clauses for same user agent in robots.txt
  Hook in submitted seeds properly.
  avoid spurious logging
  try very hard to start url consumer, and therefore bind the queue to the routing key, so that no messages are dropped, before crawling starts (should always work unless rabbitmq is down); some other tweaks for clarity and stability
 2016-01-13 13:44:54 -08:00
Noah Levitt 22336fb71b fix stats in unusual case of "failed" fetch with response > 0 (only case where this can happen currently is if basic auth is configured for a url, but fails and url returns "401 Unauthorized") 2016-01-12 17:34:27 -08:00
Hunter Stern 73be17af06 Disable SNI for a request if that request failed due to an SNI error 'javax.net.ssl.SSLProtocolException: handshake alert: unrecognized_name' 2016-01-12 16:58:48 -08:00
Adam Miller 3dbf754d56 Merge pull request #137 from nlevitt/seed-limits 
SourceSeedDecideRule, SeedLimitsEnforcer
 2016-01-12 16:39:17 -08:00
Noah Levitt 99c5af0cf2 comment out currently failing test (needs webarchive-commons pull request merged) 2015-12-14 10:46:50 -08:00
Noah Levitt f1173789f7 fix test now that ContentDigestHistoryTest.shouldProcess() is smarter 2015-12-11 16:13:17 -08:00
Noah Levitt 38931f152a handle multiple clauses for same user agent in robots.txt 2015-12-11 15:42:16 -08:00
Noah Levitt dfc1dd940e tally warc novel content bytes and warc novel urls in CrawledBytesHistotable, with tests 2015-12-11 12:49:02 -08:00