lee/heritrix3
1
0 Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-11-04 13:05:24 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,907 Commits 42 Branches 34 Tags
48908ee5b4aca1cd41ae8b939a764d06ae00a8d3
Commit Graph
767 Commits
Author SHA1 Message Date
Kenji Nagahashi a7b7c6cf5a fix for test failures in a workspace on NFS-mounted filesystem 
ContentDigestHistoryTest does not close BdbModule. It results in failure to delete bdb directory in following tests.
Added tearDown() method that closes BdbModule.
 2017-12-08 17:15:03 -08:00
Barbara Miller 998edf3479 max size for extracted form elements 2017-11-16 08:48:06 -08:00
Alex Osborne d6d7f1b162 Fix ArrayIndexOutOfBoundsException when all lines are blank 2017-11-01 10:05:26 +09:00
Alex Osborne 68ceceedd2 Use simpler read 500k chars and split method suggested by @nlevitt 
While we temporarily use a little more memory this version is a lot
less codes. It also allows us to do away with the BufferedReader.
 2017-11-01 09:29:28 +09:00
Alex Osborne 1fae77c64d Enforce robots.txt character limit per char not per line 
Since we were only enforcing the size limit after reading each
line a single very long line could cause us to to run out of
memory.

This change enforces the character limit on each read character
not just on line boundaries.

We also correct the count of processed characters in the warning
message, which was not counting newline and linefeed characters.
 2017-10-31 18:41:00 +09:00
Barbara Miller f0d1c0422b Merge pull request #187 from nlevitt/robots-allow-failed-fetch 
treat a failed fetch (e.g. socket timeout) of robots.txt the same way…
 2017-10-13 11:23:54 -07:00
Andrew Jackson aee83dfe26 Allow DNS settings to be overridden via sheets. 2017-10-13 10:29:15 +01:00
Andrew Jackson e7ce1f56b4 Ensure disabling JavaDNS always fails over to the InetAddress approach. 2017-10-12 23:33:54 +01:00
Andrew Jackson d178c689de Add some comments to try to avoid misuse. 2017-10-12 22:13:00 +01:00
Andrew Jackson b0594ac0f4 Allow JavaDNS to be disabled and use that to avoid issues in StatisticsSelfTest. 2017-10-12 21:29:34 +01:00
Andrew Jackson f7a5376fbe Merge branch 'master' into fix-ci-and-add-java-8 2017-10-12 09:31:47 +01:00
Noah Levitt ae8a0ef95f update webarchive-commons dependency to incorporate fix to warc size calculation, and uncomment test that exercises the fix 2017-10-11 14:54:37 -07:00
Andrew Jackson 46fa54da0c Set an appropriate Content Size when using non-DNS resolves. 2017-10-11 17:08:15 +01:00
Noah Levitt 84debcfe4d treat a failed fetch (e.g. socket timeout) of robots.txt the same way as http errors: full allow 2017-10-06 12:00:31 -07:00
Noah Levitt 3892c17b56 ughhh appease java 8 javadoc rules 2017-06-15 17:29:22 -07:00
Noah Levitt 87a15bddc8 Merge pull request #168 from internetarchive/gh-167 
fix for race-condition when first using the WARC writers https://gith…
 2017-06-07 18:15:19 -07:00
Neil Minton 10541184b5 Clean up after code review. 
- Parse with string split instead of regex.
- Move srcset logic up to processEmbed(CrawlURI, CharSequence, CharSequence, Hop).
 2017-06-02 11:53:16 -07:00
Neil Minton 4c0e52af61 Add parsing for srcset attributes. 
- Includes Unit Tests
 2017-05-01 17:17:05 -07:00
Noah Levitt 0b4671c291 fix for race-condition when first using the WARC writers https://github.com/internetarchive/heritrix3/issues/167 2016-08-15 13:36:33 -07:00
Noah Levitt cf9498d2ed clear the history store at the beginning of testBasics(), because the other test might have run first 2016-06-07 14:12:20 -05:00
Noah Levitt bc8b0baa59 Merge remote-tracking branch 'origin/master' into fix-test-errors 
* origin/master:
  Setup TravisCI
  Set fetch status on curis when testing link extraction
  No link extraction on URI not successfully downloaded
  Fixes issue #158 : Noisy alerts about 401s without auth challenge (#159)
  Make Content-Location header url INFERRED not REFFER hop type since Content-Location is not for redirection (#151)
  fixes for kafka 0.9 (?)
  upgrade to kafka 0.9
  somewhat ugly fix to handle exceptions from the bean browser like java.lang.RuntimeException: not implemented at org.archive.modules.fetcher.BdbCookieStore$RestrictedCollectionWrappedList.get(BdbCookieStore.java:92)
 2016-06-07 13:09:51 -05:00
Kristinn Sigurðsson 70867efdac Set fetch status on curis when testing link extraction 2016-05-03 15:58:50 +00:00
Kristinn Sigurðsson 08e3d055cf No link extraction on URI not successfully downloaded 2016-05-03 14:40:24 +00:00
Kristinn Sigurðsson c5848ae803 Fixes issue #158 : Noisy alerts about 401s without auth challenge (#159) 
* Fixes issue #158 : Noisy alerts about 401s without auth challenge

* Update test to account for non-fatal-error log not being empty on
non-auth 401s.
 2016-05-02 10:15:32 -07:00
Hunter a02a32cdfe Make Content-Location header url INFERRED not REFFER hop type since Content-Location is not for redirection (#151) 2016-04-13 15:19:21 -07:00
Andrew Jackson 48f5e71e76 Add delay to avoid race condition during test. 2016-04-07 21:48:20 +01:00
Andrew Jackson cdb79b8868 Fix serialisation tests by defaulting to TreeMaps. 2016-04-02 20:35:37 +01:00
Adam Miller 7f752d8d45 Merge pull request #150 from nlevitt/ari-4638 
various changes to amqp publish and receive
 2016-03-08 09:26:58 -08:00
Noah Levitt f4b8468564 set custom http headers last, so they override headers set elsewhere (e.g. cookies) 2016-02-18 18:16:03 -08:00
Eric Leclerc 5f431de215 Update to ExtractorHTML.java for cond. comments 
Lot's of websites are using "Downlevel Revealed" conditional comments with a twist to have a page with valid html. this is an update to the RegEx to allow such a case. ex:

    <!--[if expression]><!-->
      HTML
    <!--<![endif]-->

- reference: https://css-tricks.com/downlevel-hidden-downlevel-revealed/
- reference: http://www.sitepoint.com/web-foundations/internet-explorer-conditional-comments/
- example site where this type of tag can be found: https://www.canada.ca/index.html
 2016-02-17 14:49:50 -05:00
Hunter Stern 3e219af148 Set source tag to uri as it exists before possible prepending of scheme. 2016-02-17 08:10:07 -08:00
Hunter Stern b9b2343bc0 Don't canonicalize source tag so that SourceSeedDecideRule will work. Trailing slash of canonicalized seed currently causes it not to work. 2016-02-16 08:52:16 -08:00
Hunter bfbd61edd7 Merge pull request #145 from vonrosen/ARI-4713 
Make some urls with whitespace acceptable to JavaScript extractor.
 2016-02-11 08:27:52 -08:00
Noah Levitt cbb5e3ab99 Simplify logic for urls with spaces, and make it better follow the pattern of UriUtils.isVeryLikelyUri(). Involves a subtle adjustment to the regex LIKELY_RELATIVE_URI_PATTERN to ensure the 2nd capturing group always gets the file extension. Also add a couple of tests of strings with spaces and file extensions that are not known good extensions. 2016-02-09 15:02:11 -08:00
Hunter Stern 30ce2f09a0 Take care of possible NullPointerException 2016-02-09 08:22:39 -08:00
Hunter Stern e7c46753ce Fix formatting 2016-01-29 18:26:18 -08:00
Hunter Stern e83ff04e86 More fixes for https://webarchive.jira.com/browse/ARI-4656 2016-01-29 17:25:06 -08:00
Hunter Stern 088189ee8b Fix up code based on pull requests comments. Add test for urls with spaces in them. 2016-01-28 15:11:49 -08:00
Hunter Stern 12a88d6d23 Allow spaces in urls extracted from JS. 2016-01-20 10:44:20 -08:00
Hunter Stern 23885557bb Allow spaces in JavaScript urls, but only if they have a known good file extension 2016-01-19 18:07:05 -08:00
Hunter 34c90966a8 Merge pull request #143 from nlevitt/form-login-extra-inputs 
handle login forms with <input type="text"> fields in addition to use…
 2016-01-19 08:37:05 -08:00
Noah Levitt 25a7aa9c8d Merge pull request #141 from vonrosen/ARI-4659 
Disable SNI for a request if that request failed due to an SNI error …
 2016-01-15 11:13:42 -08:00
Noah Levitt e92e5f2bb3 Merge branch 'master' into form-login-extra-inputs 
* also fix new FormLoginProcessorTest.testFormLoginExtraInputs() in context of form-login-multipart changes

* master:
  fix NullPointerException when using old HtmlFormCredential login
  to make this branch only about multipart/form-data, undo the changes to handle more complicated forms with extra input fields; add unit test; store CrawlURI submit data in http-library agnostic way and remove httpcomponents-specific code from HTMLForm, refactor creation of post data accordingly
  Be sure to return at least one of the candidates
  Remove unneeded constant
  Make sure to check form encoding type when addign form to curi data
  More changes for https://webarchive.jira.com/browse/ARI-4656
  More changes
  Save enctype of form in submission curi
  Add ability to submit forms with enctype="multipart/form-data"
  If multiple username inputs in login form, favor the one with the word 'login' in it.
 2016-01-15 09:39:24 -08:00
Noah Levitt c1abbfa5bd handle login forms with <input type="text"> fields in addition to username (for this to work there must be exactly one input field whose name contains the string "username") 2016-01-14 18:02:54 -08:00
Noah Levitt 368d586eb1 fix NullPointerException when using old HtmlFormCredential login 2016-01-14 16:00:08 -08:00
Noah Levitt 3b12a751e1 to make this branch only about multipart/form-data, undo the changes to handle more complicated forms with extra input fields; add unit test; store CrawlURI submit data in http-library agnostic way and remove httpcomponents-specific code from HTMLForm, refactor creation of post data accordingly 2016-01-14 12:17:19 -08:00
Hunter Stern 309847cc40 Be sure to return at least one of the candidates 2016-01-14 12:05:38 -08:00
Hunter Stern 704ec62487 Remove unneeded constant 2016-01-14 12:05:38 -08:00
Hunter Stern a2a39fe052 Make sure to check form encoding type when addign form to curi data 2016-01-14 12:05:38 -08:00
Hunter Stern 93bca3d55c More changes for https://webarchive.jira.com/browse/ARI-4656 2016-01-14 12:05:38 -08:00