lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-07-21 06:51:38 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,848 Commits 40 Branches 31 Tags
3.12.0
Commit Graph
1040 Commits
Author SHA1 Message Date
Alex Osborne b626f52d67 [maven-release-plugin] prepare release 3.12.0 2025-10-30 10:25:11 +09:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>Alex Osborne
 		d627118307 Bump the minor-and-patch group across 1 directory with 18 updates (#680) 
* Bump the minor-and-patch group across 1 directory with 18 updates

Bumps the minor-and-patch group with 18 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [org.eclipse.jetty:jetty-bom](https://github.com/jetty/jetty.project) | `12.0.27` | `12.0.29` |
| [org.eclipse.jetty.ee10:jetty-ee10-bom](https://github.com/jetty/jetty.project) | `12.0.27` | `12.0.29` |
| [org.apache.maven.plugins:maven-compiler-plugin](https://github.com/apache/maven-compiler-plugin) | `3.13.0` | `3.14.1` |
| [org.codehaus.mojo:versions-maven-plugin](https://github.com/mojohaus/versions) | `2.18.0` | `2.19.1` |
| [org.sonatype.central:central-publishing-maven-plugin](https://github.com/sonatype/central-publishing-maven-plugin) | `0.7.0` | `0.9.0` |
| [org.apache.maven.plugins:maven-gpg-plugin](https://github.com/apache/maven-gpg-plugin) | `3.2.7` | `3.2.8` |
| [org.springframework:spring-core](https://github.com/spring-projects/spring-framework) | `6.2.11` | `6.2.12` |
| [org.springframework:spring-beans](https://github.com/spring-projects/spring-framework) | `6.2.11` | `6.2.12` |
| [org.springframework:spring-context](https://github.com/spring-projects/spring-framework) | `6.2.11` | `6.2.12` |
| [org.springframework:spring-expression](https://github.com/spring-projects/spring-framework) | `6.2.11` | `6.2.12` |
| [org.netpreserve.commons:webarchive-commons](https://github.com/iipc/webarchive-commons) | `3.0.0` | `3.0.1` |
| [com.github.mwiede:jsch](https://github.com/mwiede/jsch) | `2.27.3` | `2.27.4` |
| org.apache.pdfbox:pdfbox | `3.0.5` | `3.0.6` |
| org.glassfish.jaxb:jaxb-runtime | `4.0.5` | `4.0.6` |
| [org.webjars:webjars-locator-lite](https://github.com/webjars/webjars-locator-lite) | `1.1.0` | `1.1.2` |
| [org.apache.maven.plugins:maven-assembly-plugin](https://github.com/apache/maven-assembly-plugin) | `3.3.0` | `3.7.1` |
| [com.rabbitmq:amqp-client](https://github.com/rabbitmq/rabbitmq-java-client) | `5.26.0` | `5.27.0` |
| [com.rethinkdb:rethinkdb-driver](https://github.com/rethinkdb/rethinkdb-java) | `2.3.3` | `2.4.4` |



Updates `org.eclipse.jetty:jetty-bom` from 12.0.27 to 12.0.29
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.27...jetty-12.0.29)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.27 to 12.0.29
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.27...jetty-12.0.29)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.27 to 12.0.29
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.27...jetty-12.0.29)

Updates `org.apache.maven.plugins:maven-compiler-plugin` from 3.13.0 to 3.14.1
- [Release notes](https://github.com/apache/maven-compiler-plugin/releases)
- [Commits](https://github.com/apache/maven-compiler-plugin/compare/maven-compiler-plugin-3.13.0...maven-compiler-plugin-3.14.1)

Updates `org.codehaus.mojo:versions-maven-plugin` from 2.18.0 to 2.19.1
- [Release notes](https://github.com/mojohaus/versions/releases)
- [Changelog](https://github.com/mojohaus/versions/blob/master/ReleaseNotes.md)
- [Commits](https://github.com/mojohaus/versions/compare/2.18.0...2.19.1)

Updates `org.sonatype.central:central-publishing-maven-plugin` from 0.7.0 to 0.9.0
- [Commits](https://github.com/sonatype/central-publishing-maven-plugin/commits)

Updates `org.apache.maven.plugins:maven-gpg-plugin` from 3.2.7 to 3.2.8
- [Release notes](https://github.com/apache/maven-gpg-plugin/releases)
- [Commits](https://github.com/apache/maven-gpg-plugin/compare/maven-gpg-plugin-3.2.7...maven-gpg-plugin-3.2.8)

Updates `org.springframework:spring-core` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12)

Updates `org.springframework:spring-beans` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12)

Updates `org.springframework:spring-context` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12)

Updates `org.springframework:spring-expression` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12)

Updates `org.springframework:spring-beans` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12)

Updates `org.springframework:spring-context` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12)

Updates `org.springframework:spring-expression` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12)

Updates `org.netpreserve.commons:webarchive-commons` from 3.0.0 to 3.0.1
- [Release notes](https://github.com/iipc/webarchive-commons/releases)
- [Changelog](https://github.com/iipc/webarchive-commons/blob/master/CHANGES.md)
- [Commits](https://github.com/iipc/webarchive-commons/compare/webarchive-commons-3.0.0...webarchive-commons-3.0.1)

Updates `com.github.mwiede:jsch` from 2.27.3 to 2.27.4
- [Release notes](https://github.com/mwiede/jsch/releases)
- [Changelog](https://github.com/mwiede/jsch/blob/master/ChangeLog)
- [Commits](https://github.com/mwiede/jsch/compare/jsch-2.27.3...jsch-2.27.4)

Updates `org.apache.pdfbox:pdfbox` from 3.0.5 to 3.0.6

Updates `org.glassfish.jaxb:jaxb-runtime` from 4.0.5 to 4.0.6

Updates `org.webjars:webjars-locator-lite` from 1.1.0 to 1.1.2
- [Release notes](https://github.com/webjars/webjars-locator-lite/releases)
- [Commits](https://github.com/webjars/webjars-locator-lite/compare/webjars-locator-lite-1.1.0...webjars-locator-lite-1.1.2)

Updates `org.apache.maven.plugins:maven-assembly-plugin` from 3.3.0 to 3.7.1
- [Release notes](https://github.com/apache/maven-assembly-plugin/releases)
- [Commits](https://github.com/apache/maven-assembly-plugin/compare/maven-assembly-plugin-3.3.0...maven-assembly-plugin-3.7.1)

Updates `com.rabbitmq:amqp-client` from 5.26.0 to 5.27.0
- [Release notes](https://github.com/rabbitmq/rabbitmq-java-client/releases)
- [Commits](https://github.com/rabbitmq/rabbitmq-java-client/compare/v5.26.0...v5.27.0)

Updates `com.rethinkdb:rethinkdb-driver` from 2.3.3 to 2.4.4
- [Release notes](https://github.com/rethinkdb/rethinkdb-java/releases)
- [Commits](https://github.com/rethinkdb/rethinkdb-java/commits/v2.4.4)

---
updated-dependencies:
- dependency-name: org.eclipse.jetty:jetty-bom
  dependency-version: 12.0.29
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.29
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.29
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-compiler-plugin
  dependency-version: 3.14.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.codehaus.mojo:versions-maven-plugin
  dependency-version: 2.19.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.sonatype.central:central-publishing-maven-plugin
  dependency-version: 0.9.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-gpg-plugin
  dependency-version: 3.2.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-core
  dependency-version: 6.2.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
  dependency-version: 6.2.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
  dependency-version: 6.2.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
  dependency-version: 6.2.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
  dependency-version: 6.2.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
  dependency-version: 6.2.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
  dependency-version: 6.2.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.netpreserve.commons:webarchive-commons
  dependency-version: 3.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.github.mwiede:jsch
  dependency-version: 2.27.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.apache.pdfbox:pdfbox
  dependency-version: 3.0.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.glassfish.jaxb:jaxb-runtime
  dependency-version: 4.0.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars:webjars-locator-lite
  dependency-version: 1.1.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-assembly-plugin
  dependency-version: 3.7.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: com.rabbitmq:amqp-client
  dependency-version: 5.27.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: com.rethinkdb:rethinkdb-driver
  dependency-version: 2.4.4
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add json-simple as contrib dependency

Needed for TroughClient and no longer included via rethinkdb.

* Exclude junit from json-simple

* Add Jackson BOM 2.20.0 to dependency management

This ensures rethinkdb doesn't pull in an old version.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Alex Osborne <aosborne@nla.gov.au>
 2025-10-27 15:42:40 +09:00
dependabot[bot]andGitHub 237d19a5d4 Bump org.apache.maven.plugins:maven-surefire-plugin from 2.22.2 to 3.5.4 
Bumps [org.apache.maven.plugins:maven-surefire-plugin](https://github.com/apache/maven-surefire) from 2.22.2 to 3.5.4.
- [Release notes](https://github.com/apache/maven-surefire/releases)
- [Commits](https://github.com/apache/maven-surefire/compare/surefire-2.22.2...surefire-3.5.4)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-surefire-plugin
  dependency-version: 3.5.4
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2025-10-27 01:14:45 +00:00
Adam Miller 8d09cc4e50 feat: Pass along script tag attributes to JS extractor for configurable rejection. 2025-10-07 11:04:12 -07:00
Adam Miller e92475ac43 fix: sheet setting of extractorJs.strict doesn't work without kp keyname matching 2025-09-30 14:14:06 -07:00
Alex Osborne b988940eff [maven-release-plugin] prepare for next development iteration 2025-09-22 13:57:47 +09:00
Alex Osborne e3bdd7220e [maven-release-plugin] prepare release 3.11.0 2025-09-22 13:57:41 +09:00
Alex Osborne d484007f57 FetchHTTP2: Note that WARC doesn't specify the recording of HTTP/2 and 3 
iipc/warc-specifications#106
 2025-09-06 10:29:24 +09:00
Alex Osborne 20dcc89244 [maven-release-plugin] prepare for next development iteration 2025-08-29 17:26:11 +09:00
Alex Osborne 7a2c676dc9 [maven-release-plugin] prepare release 3.10.2 2025-08-29 17:26:05 +09:00
Alex Osborne 31539e2d62 Use Groovy BOM to ensure consistent versions 2025-08-28 15:35:14 +09:00
Alex Osborne e42306ceb1 Use slf4j BOM to ensure consistent versions 2025-08-28 15:32:48 +09:00
Alex Osborne 3cde816e95 Use jetty BOM to ensure consistent versions 2025-08-28 15:27:10 +09:00
Alex Osborne 219139e4e9 FetchDNS: Treat 0.0.0.0 as resolution failure 
If there are multiple addresses then ignore 0.0.0.0. If the only address is 0.0.0.0 treat it as a DNS resolution failure. This should prevent us from throwing IllegalStateException later in DefaultServerCache.getHostFor().

Fixes #428
 2025-08-21 09:47:48 +09:00
Alex Osborne ea67b59e51 [maven-release-plugin] prepare for next development iteration 2025-07-21 17:15:17 +09:00
Alex Osborne 1cf6762c7f [maven-release-plugin] prepare release 3.10.1 2025-07-21 17:15:12 +09:00
Alex Osborne 2388db4229 Bump commons-lang from 2.6 to 3.18.0 2025-07-21 16:55:31 +09:00
Alex Osborne 53f154574d Bump crawler-commons from 1.4 to 1.5 2025-07-21 16:10:15 +09:00
Alex Osborne acf41d598d FetchHTTP2: Use HTTP/1.1 when the server doesn't support ALPN 
This fixes `IOException: frame_size_error/invalid_frame_length`.
 2025-06-20 18:52:32 +09:00
Alex Osborne 8bfde99af3 FetchHTTP2: Handle getRemoteSocketAddress() returning null 2025-06-20 17:27:32 +09:00
Alex Osborne 01378ddbbf [maven-release-plugin] prepare for next development iteration 2025-06-12 22:01:16 +09:00
Alex Osborne ecc46a38c3 [maven-release-plugin] prepare release 3.10.0 2025-06-12 22:01:10 +09:00
Alex Osborne 9b5900b528 FetchHTTP2: Avoid silently decoding gzip 
Fixes `Recorder getContentReplayInputStream
WARNING: gzip problem; using raw entity instead`.
 2025-06-12 21:36:14 +09:00
Adam Miller 1ab9b4a206 fix: avoid bdb environment corruption when terminate is called during shutdown 2025-06-11 17:44:52 -07:00
Alex Osborne e0bae9a1b5 FetchHTTP2: Fallback to async DNS resolver 2025-06-11 16:55:11 +09:00
Alex Osborne c3f9afe64b BrowserProcessor: use FetchHTTP2's configured proxy as an upstream proxy 2025-06-10 09:01:02 +09:00
Alex Osborne cc50af72be FetchHTTP2: Add HTTP proxy support 2025-06-09 22:28:49 +09:00
Alex Osborne e68e4975c9 Rename to BrowserProcessor and -Behavior 
After I added them to the Bean reference they seemed out of place with the DecideRules and RecordBuilders.

We had two classes called 'Browser' which is potentially confusing. It's also probably good to more clearly differentiate the ExtractLinks browser behavior from the Extractor processors.
 2025-06-09 15:35:25 +09:00
Alex Osborne cf447f3bd9 Add Browser processor and behaviors to bean reference 2025-06-09 15:19:09 +09:00
Alex OsborneandGitHub 52bbd8091b Merge pull request #653 from internetarchive/bidi 
Add Browser processor using WebDriver BiDi
 2025-06-09 09:36:06 +09:00
Alex Osborne c7b7ee1842 Support * and $ wildcards in robots.txt 2025-06-07 10:37:11 +09:00
Alex Osborne 1161d87217 Browser: Add processor report 2025-06-05 10:48:15 +09:00
Alex Osborne 722552feef Add Browser processor using WebDriver BiDi 
The Browser processor can load a fetched page in a local web browser, record any requests the browser makes and run behaviors that interact with the page such as scrolling down and extracting links.

This differs from my previous attempt (ExtractorChrome) in a few ways:

- Uses the new WebDriver BiDi standard instead of the Chrome Devtools Protocol. The new protocol is mostly browser-agnostic, more consistent and hopefully more stable.

- Uses a MITM proxy instead of CDP request interception for recording sub-resources. That's partly because BiDi is still missing some key interception APIs. Even so in practice I found the proxy method loads pages faster and more reliably, likely because responses can be streamed incrementally, which helps a lot for large resources or server-sent events.

- Even when HTTP/2 is unavailable, the new FetchHTTP2 module does connection pooling which makes loading browser requests a lot faster. The original FetchHTTP opened a new connection for every request.

- The Browser processor can be configured with a list of behavior beans making it more customizable and extensible.

Obvious areas for future development:

- More Behavior beans: take screenshots, saveg the rendered DOM, run Browsertrix-compatible behavior scripts

- Support for remote WebDrivers (e.g. Selenium Server or cloud services)
 2025-06-04 17:44:42 +09:00
Alex Osborne 567e27181d Upgrade webarchive-commons from 1.3.0 to 2.0.1 (removes httpclient 3) 2025-05-21 17:39:21 +09:00
Alex Osborne 1827d93fe6 [maven-release-plugin] prepare for next development iteration 2025-05-13 12:25:24 +09:00
Alex Osborne dd40970f15 [maven-release-plugin] prepare release 3.9.0 2025-05-13 12:25:16 +09:00
Alex Osborne 4a2cdfaa81 FetchHTTP2: Print the jetty-quiche-native warning earlier at build time 2025-05-09 20:40:33 +09:00
Alex Osborne 8692bdf655 Add WARC-Protocol to request records too when using HTTP/2 or HTTP/3 
While this could be inferred from the response, it's more accurate to
include them on the request as the request was not really HTTP/1.1.
 2025-05-09 19:54:22 +09:00
Alex Osborne eb1205f6d5 Upgrade all tests to junit 5 2025-05-09 00:07:18 +09:00
Alex Osborne d5e91eb1f3 Bump pdfbox from 3.0.4 to 3.0.5 2025-05-08 17:36:12 +09:00
Alex Osborne fcfef68aff FetchHTTP2: A new fetch module for HTTP/2 and HTTP/3 
This uses Jetty HttpClient since it speaks both protocols, and we
already have it as a dependency via Restlet. This doesn't support all
the options of FetchHTTP, notably proxy and POST requests are missing.

Jetty currently has the HTTP/3 client marked as "experimental, not for
production use" so we disable it by default and don't ship the large
quiche native jar it requires. It does seem to work OK though, at least
in my limited testing so far. The HTTP/3 support currently only responds
to Alt-Svc headers not other ways of discovering HTTP/3 availability
(e.g. HTTPS DNS record).

Fetches that were made via HTTP/2 or HTTP/3 are annotated 'h2' and 'h3'
in the crawl.log. The messages are recorded in the WARC files using
HTTP/1.1 syntax with a WARC-Protocol header. FetchHTTP2 also currently
records HTTP/1.1 messages without transfer-encoding rather than the raw
wire messages.
 2025-04-29 12:47:57 +09:00
Alex Osborne bc259728a0 Update to Jetty 12.0.16 and Restlet 2.6.0-M2 
The Jetty API has changed, which mostly affects test code.

Jetty now does a strict SNI host check which unfortunately causes it to
return "SNI error" for our existing ad-hoc certificates. For now, I've
disabled it to avoid breaking existing deployments but added a
--sni-host-check command-line option so you can re-enable it if
you've configured your own certificate appropriately.
 2025-04-24 17:40:20 +09:00
Alex Osborne e9b60404a6 [maven-release-plugin] prepare for next development iteration 2025-04-01 17:43:31 +09:00
Alex Osborne 2323e2251e [maven-release-plugin] prepare release 3.8.0 2025-04-01 17:43:25 +09:00
Alex Osborne 13052a1465 Fix more broken references in javadoc 2025-04-01 16:06:10 +09:00
Alex Osborne f956b2ec1f Fix broken references in javadoc 2025-04-01 15:59:38 +09:00
Alex Osborne 7f76f5abd1 FetchHTTPTest: Disable testHostHeaderDefaultPort due to random failures 
Probably some of the GitHub test runners are getting blocked.
 2025-04-01 15:37:21 +09:00
Alex Osborne b17e3fcab3 Bump beanshell from 2.0b5 to 2.0b6 2025-04-01 15:27:22 +09:00
Alex Osborne f17000e14f Fix warning: Building a command line with string concatenation 2025-03-31 15:17:28 +09:00
Alex Osborne 60f512b4b0 Fix compiler warning: deprecated item is not annotated with @Deprecated 2025-03-31 15:12:21 +09:00