lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-23 23:16:33 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,446 Commits 40 Branches 31 Tags
c5d9bbd0c09556d445d64ca252fe2220e59debc9
Commit Graph
2446 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Andy Jackson c5d9bbd0c0 Fix changelog following tag merge. 2022-07-28 10:58:48 +01:00
Andy Jackson e83f4f9e02 Reset to SNAPSHOT 2022-07-28 10:56:59 +01:00
Andy Jackson 9278651b32 Merge tag '3.4.0-20220727' 
[maven-release-plugin] copy for tag 3.4.0-20220727
 2022-07-28 10:53:55 +01:00
Andy Jackson a00a9618a5 Resync changelog. 2022-07-28 09:06:02 +01:00
Andy Jackson 7c4453cf66 [maven-release-plugin] prepare for next development iteration 2022-07-27 21:41:58 +01:00
Andy Jackson b0505662d7 [maven-release-plugin] prepare release 3.4.0-20220727 2022-07-27 21:41:58 +01:00
Andy Jackson 292d2998cb [maven-release-plugin] prepare release 3.4.0-20220727 3.4.0-20220727 2022-07-27 21:36:19 +01:00
Andy Jackson d17a6a8711 Merge pull request #493 from ukwa/master 
Update changelog as of 2022-07-27
 2022-07-27 21:26:06 +01:00
Andy Jackson b8b6c5e209 Update changelog. 2022-07-27 21:24:33 +01:00
dependabot[bot] 1a91a2c8c1 Bump jsch from 0.1.52 to 0.1.54 in /commons 
Bumps jsch from 0.1.52 to 0.1.54.

---
updated-dependencies:
- dependency-name: com.jcraft:jsch
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-07-07 23:43:35 +09:00
dependabot[bot] 46ea8e1047 Bump spring-core from 5.3.19 to 5.3.20 in /commons 
Bumps [spring-core](https://github.com/spring-projects/spring-framework) from 5.3.19 to 5.3.20.
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v5.3.19...v5.3.20)

---
updated-dependencies:
- dependency-name: org.springframework:spring-core
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-07-07 23:30:29 +09:00
dependabot[bot] 82dee7c89c Bump jsch from 0.1.52 to 0.1.54 in /modules 
Bumps jsch from 0.1.52 to 0.1.54.

---
updated-dependencies:
- dependency-name: com.jcraft:jsch
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-07-07 23:27:52 +09:00
Alex Osborne c1ba694add Document robotsTxtOnly robots policy in config guide 2022-07-07 16:18:39 +09:00
Alex Osborne 1209b2d42f Add robotsTxtOnly robots policy 
This policy obeys robots.txt but ignores meta nofollow. We're
increasingly frequently encountering sites which fail to be archived
properly due to usage of this meta tag but where the robots.txt file
contains reasonable rules. In many cases it seems to be unintentional,
and we've even had questions from webmasters asking why archiving was
failing when they were explicitly allowing us in robots.txt.
 2022-07-07 16:18:39 +09:00
Alex Osborne 09f75e389a docs/glossary: Add manifest (M) to discovery path 2022-06-22 15:44:44 +09:00
Alex Osborne 1d490c779b Merge pull request #488 from netarchivesuite/crawluri-npe 
Removed a potential NPE in hashCode method to CrawlURI which was fata…
 2022-06-08 13:35:23 +09:00
csrster 702bfa5170 Removed a potential NPE in hashCode method to CrawlURI which was fatal for heritrix Browse Beans functionality. 
(cherry picked from commit 96e47a2f695a3116d89f02e7cb529ca181d21444)
 2022-06-07 17:48:29 +02:00
dependabot[bot] e8be86f7cf Bump gson from 2.8.6 to 2.8.9 in /contrib 
Bumps [gson](https://github.com/google/gson) from 2.8.6 to 2.8.9.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.6...gson-parent-2.8.9)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-05-24 12:32:27 +09:00
Alex Osborne a35cd44a73 Docs: Add section to config guide about http and form credentials 
Based on https://github.com/internetarchive/heritrix3/wiki/Credentials
 2022-05-24 12:01:56 +09:00
Alex Osborne 9644780a54 ExtractorHTML: Add --robots option to main() 
Useful for testing link extraction on pages that use a robots meta tag.
 2022-05-11 16:31:48 +09:00
dependabot[bot] e1e7adf088 Bump spring-core from 5.3.18 to 5.3.19 in /commons 
Bumps [spring-core](https://github.com/spring-projects/spring-framework) from 5.3.18 to 5.3.19.
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v5.3.18...v5.3.19)

---
updated-dependencies:
- dependency-name: org.springframework:spring-core
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-04-27 15:09:33 +09:00
Alex Osborne 9fe64a6f0c ExtractorChrome: If disabled only connect the browser process on demand 
This avoids creating browser processes for jobs that have it disabled.
If a job has it disabled by default but enables it with a sheet we'll
start it when needed.

We still connect on start when enabled to provide early error feedback.
 2022-04-27 14:17:00 +09:00
Alex Osborne 207adec12c Merge pull request #478 from internetarchive/srcset-fix 2022-04-27 14:11:09 +09:00
Alex Osborne 914756f155 ExtractorHTML: Add a main() method to run the extractor standalone 
This makes troubleshooting link extraction problems much easier.
 2022-04-22 16:44:54 +09:00
Alex Osborne aac74411a1 ExtractorHTML: Fix srcset by normalizing elementContext() to lowercase 
This ensures that when we later compare the context in processEmbed()
we don't need to deal with variants like srcSet or SRCSET. Note that
we're already sometimes lowercasing it later in HTMLLinkContext.get().

Fixes #477.
 2022-04-22 16:44:53 +09:00
Alex Osborne 55078c078d Merge pull request #476 from ClemensRobbenhaar/issue211-dns-over-https 2022-04-10 08:59:59 +09:00
Clemens Klein-Robbenhaar f5c83ce368 Add support for DNS-over-HTTPS lookups 
- docs: lets see if we can set a link
  to another chapter
 2022-04-09 23:32:55 +02:00
Clemens Klein-Robbenhaar ed332bcb9e Add support for DNS-over-HTTPS lookups 
- scrap setter for "enableDnsOverHttpResolve", too
 2022-04-09 23:26:07 +02:00
Clemens Klein-Robbenhaar fdde6d87a0 Add support for DNS-over-HTTPS lookups 
- instead of "borrowing" the configured proxy from
  the fetchHttp bean, use proxy values defined via
  global options, to avoid interference with other
  jobs running in parallel (or at least make them
  explicit).
  The "fetchHttp" bean also uses these settings,
  if no bean specific settings are used.
- remove the "enableDnsOverHttpResolves", and rely
  on a non-empty "dnsOverHttpServer" value to signal
  that DoH should be used.
 2022-04-09 23:04:37 +02:00
Clemens Klein-Robbenhaar 9648c98c77 [misc] rst formatting 2022-04-08 12:14:11 +02:00
Clemens Klein-Robbenhaar 2144abb2cb Add support for DNS-over-HTTPS lookups 
- use the "DohResolver" from the dnsjava library
  to make DoH lookups
- to enable and configure it, add two new
  properties
  * "enableDnsOverHttpResolves" (boolean)
  * "dnsOverHttpServer" URL to the DoH Server
- as one use case for DoH is being located
  behind a firewall, also support using a proxy
  to access the DoH server; the proxy from
  the FetchHTTP bean is reused in that case

Fixes #211
 2022-04-08 11:48:52 +02:00
Alex Osborne 8557533d11 ChromeWindow: Ignore WebsocketNotConnectedException in window.close() 
This usually means the browser has exited and so there's no window left
to close.
 2022-04-08 12:25:30 +09:00
Alex Osborne adc68583cc ChromeWindow: Ignore events once we've requested the window be closed 2022-04-08 12:25:30 +09:00
Alex Osborne e5d8d53993 ExtractorChrome: Only run on pages with 2xx status codes 
Let's not waste time running the browser on error pages.
 2022-04-08 12:25:30 +09:00
Alex Osborne 42ac612e1a ExtractorChrome: Add a counter to give each recorder a unique basename 
We use a static (global) counter so that if a job uses multiple
instances of ExtractorChrome they don't clobber each other.
 2022-04-08 12:25:30 +09:00
Alex Osborne f2b3722b0b ChromeProcess: Disable features that prompt or cause background traffic 
Disables browser features like phishing protection, translation, crash
and metrics reporting, OS keychain integration and such that can cause
unnecessary network traffic or that can cause the browser to get stuck
in a prompt. Some of these options may be unnecessary in headless mode,
but we may as well keep them in case in future we want to support headed
mode for debugging.
 2022-04-08 12:25:30 +09:00
Alex Osborne e5bff6a06a ExtractorChrome: Add commandLineOptions property 
Allows for custom browser configuration such as using a proxy server or
increasing log verbosity.
 2022-04-08 12:25:30 +09:00
Alex Osborne 0378e0d123 ChromeProcess: Use the --disable-crash-reporter command-line option 
This avoids an ownership conflict over /tmp/Crashpad.
 2022-04-08 12:25:30 +09:00
Alex Osborne f2af65122c ChromeProcess: Add macOS default install path to DEFAULT_EXECUTABLES 2022-04-08 12:25:30 +09:00
dependabot[bot] ce0b5e3a24 Bump spring-beans from 5.3.14 to 5.3.18 in /commons 
Bumps [spring-beans](https://github.com/spring-projects/spring-framework) from 5.3.14 to 5.3.18.
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v5.3.14...v5.3.18)

---
updated-dependencies:
- dependency-name: org.springframework:spring-beans
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-04-06 10:21:56 +09:00
Alex Osborne 0698545eec JDK18 support: Run GitHub CI tests on JDK 18 2022-04-01 11:34:11 +09:00
Alex Osborne c3f140b9b5 JDK18 support: Update Groovy to 3.0.10 
Fixes #473 "Unsupported class file major version 62".
 2022-04-01 11:34:11 +09:00
Alex Osborne 971f04dd23 TransclusionDecideRule: Don't treat sitemap links ('M') as transclusions 
We should only crawl sitemaps and sitemap links if they are in the
primary SURT scope. Otherwise we'll start crawling the sitemaps of
every site an embedded resource is pulled in from even when they
should be out of scope.

Fixes #469
 2022-03-28 13:47:49 +09:00
Alex Osborne d796491995 FetchHTTPTest: Fix SOCKS server startup race 
testSocksProxy() is failing on the GitHub action CI runs but I can't
reproduce the issue locally. I'm just guessing this is what's causing it
but this should be fixed anyway to avoid future intermittent failures.
 2022-03-28 13:26:53 +09:00
Alex Osborne 0dc05408da FetchHTTPTest: Don't test IP binding on macOS utun network interfaces 
This avoids a test failure on macOS machines that are running Wireguard.
 2022-03-28 11:40:48 +09:00
Alex Osborne 0d6c04fda5 FetchHTTPTest: Update to junit 4 
By replacing the custom TestSuite with junit 4 @BeforeClass and
@AfterClass annotations running individual tests by themselves now works
correctly.

Since FetchHTTPTest now just contains the utility code for the test
servers we rename it to FetchHTTPTestServers. FetchHTTPTests (which
contains the actual tests) can now follow the standard naming
scheme as FetchHTTPTest.
 2022-03-28 11:35:45 +09:00
Jamie Hoyle dd01689ace FetchHTTP: SOCKS5 proxy support 
Closes #454
 2022-03-28 10:24:07 +09:00
Alex Osborne 69f231b8ab Merge pull request #468 from internetarchive/remove-jna 2022-03-03 17:16:34 +09:00
Alex Osborne c6d4251e4f Merge pull request #459 from internetarchive/parse-link-rel 2022-03-03 17:16:22 +09:00
Alex Osborne 98f42de863 Merge pull request #460 from internetarchive/dependabot/maven/commons/org.springframework-spring-core-5.3.14 2022-03-03 17:14:46 +09:00