lee/heritrix3
1
0 Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-11-06 14:02:51 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,847 Commits 46 Branches 34 Tags
89bd3474ba658103cc2c9e71b4eef26e1841f921
Commit Graph
135 Commits
Author SHA1 Message Date
dependabot[bot]andAlex Osborne d627118307 Bump the minor-and-patch group across 1 directory with 18 updates (#680) 
* Bump the minor-and-patch group across 1 directory with 18 updates

Bumps the minor-and-patch group with 18 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [org.eclipse.jetty:jetty-bom](https://github.com/jetty/jetty.project) | `12.0.27` | `12.0.29` |
| [org.eclipse.jetty.ee10:jetty-ee10-bom](https://github.com/jetty/jetty.project) | `12.0.27` | `12.0.29` |
| [org.apache.maven.plugins:maven-compiler-plugin](https://github.com/apache/maven-compiler-plugin) | `3.13.0` | `3.14.1` |
| [org.codehaus.mojo:versions-maven-plugin](https://github.com/mojohaus/versions) | `2.18.0` | `2.19.1` |
| [org.sonatype.central:central-publishing-maven-plugin](https://github.com/sonatype/central-publishing-maven-plugin) | `0.7.0` | `0.9.0` |
| [org.apache.maven.plugins:maven-gpg-plugin](https://github.com/apache/maven-gpg-plugin) | `3.2.7` | `3.2.8` |
| [org.springframework:spring-core](https://github.com/spring-projects/spring-framework) | `6.2.11` | `6.2.12` |
| [org.springframework:spring-beans](https://github.com/spring-projects/spring-framework) | `6.2.11` | `6.2.12` |
| [org.springframework:spring-context](https://github.com/spring-projects/spring-framework) | `6.2.11` | `6.2.12` |
| [org.springframework:spring-expression](https://github.com/spring-projects/spring-framework) | `6.2.11` | `6.2.12` |
| [org.netpreserve.commons:webarchive-commons](https://github.com/iipc/webarchive-commons) | `3.0.0` | `3.0.1` |
| [com.github.mwiede:jsch](https://github.com/mwiede/jsch) | `2.27.3` | `2.27.4` |
| org.apache.pdfbox:pdfbox | `3.0.5` | `3.0.6` |
| org.glassfish.jaxb:jaxb-runtime | `4.0.5` | `4.0.6` |
| [org.webjars:webjars-locator-lite](https://github.com/webjars/webjars-locator-lite) | `1.1.0` | `1.1.2` |
| [org.apache.maven.plugins:maven-assembly-plugin](https://github.com/apache/maven-assembly-plugin) | `3.3.0` | `3.7.1` |
| [com.rabbitmq:amqp-client](https://github.com/rabbitmq/rabbitmq-java-client) | `5.26.0` | `5.27.0` |
| [com.rethinkdb:rethinkdb-driver](https://github.com/rethinkdb/rethinkdb-java) | `2.3.3` | `2.4.4` |



Updates `org.eclipse.jetty:jetty-bom` from 12.0.27 to 12.0.29
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.27...jetty-12.0.29)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.27 to 12.0.29
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.27...jetty-12.0.29)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.27 to 12.0.29
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.27...jetty-12.0.29)

Updates `org.apache.maven.plugins:maven-compiler-plugin` from 3.13.0 to 3.14.1
- [Release notes](https://github.com/apache/maven-compiler-plugin/releases)
- [Commits](https://github.com/apache/maven-compiler-plugin/compare/maven-compiler-plugin-3.13.0...maven-compiler-plugin-3.14.1)

Updates `org.codehaus.mojo:versions-maven-plugin` from 2.18.0 to 2.19.1
- [Release notes](https://github.com/mojohaus/versions/releases)
- [Changelog](https://github.com/mojohaus/versions/blob/master/ReleaseNotes.md)
- [Commits](https://github.com/mojohaus/versions/compare/2.18.0...2.19.1)

Updates `org.sonatype.central:central-publishing-maven-plugin` from 0.7.0 to 0.9.0
- [Commits](https://github.com/sonatype/central-publishing-maven-plugin/commits)

Updates `org.apache.maven.plugins:maven-gpg-plugin` from 3.2.7 to 3.2.8
- [Release notes](https://github.com/apache/maven-gpg-plugin/releases)
- [Commits](https://github.com/apache/maven-gpg-plugin/compare/maven-gpg-plugin-3.2.7...maven-gpg-plugin-3.2.8)

Updates `org.springframework:spring-core` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12)

Updates `org.springframework:spring-beans` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12)

Updates `org.springframework:spring-context` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12)

Updates `org.springframework:spring-expression` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12)

Updates `org.springframework:spring-beans` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12)

Updates `org.springframework:spring-context` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12)

Updates `org.springframework:spring-expression` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12)

Updates `org.netpreserve.commons:webarchive-commons` from 3.0.0 to 3.0.1
- [Release notes](https://github.com/iipc/webarchive-commons/releases)
- [Changelog](https://github.com/iipc/webarchive-commons/blob/master/CHANGES.md)
- [Commits](https://github.com/iipc/webarchive-commons/compare/webarchive-commons-3.0.0...webarchive-commons-3.0.1)

Updates `com.github.mwiede:jsch` from 2.27.3 to 2.27.4
- [Release notes](https://github.com/mwiede/jsch/releases)
- [Changelog](https://github.com/mwiede/jsch/blob/master/ChangeLog)
- [Commits](https://github.com/mwiede/jsch/compare/jsch-2.27.3...jsch-2.27.4)

Updates `org.apache.pdfbox:pdfbox` from 3.0.5 to 3.0.6

Updates `org.glassfish.jaxb:jaxb-runtime` from 4.0.5 to 4.0.6

Updates `org.webjars:webjars-locator-lite` from 1.1.0 to 1.1.2
- [Release notes](https://github.com/webjars/webjars-locator-lite/releases)
- [Commits](https://github.com/webjars/webjars-locator-lite/compare/webjars-locator-lite-1.1.0...webjars-locator-lite-1.1.2)

Updates `org.apache.maven.plugins:maven-assembly-plugin` from 3.3.0 to 3.7.1
- [Release notes](https://github.com/apache/maven-assembly-plugin/releases)
- [Commits](https://github.com/apache/maven-assembly-plugin/compare/maven-assembly-plugin-3.3.0...maven-assembly-plugin-3.7.1)

Updates `com.rabbitmq:amqp-client` from 5.26.0 to 5.27.0
- [Release notes](https://github.com/rabbitmq/rabbitmq-java-client/releases)
- [Commits](https://github.com/rabbitmq/rabbitmq-java-client/compare/v5.26.0...v5.27.0)

Updates `com.rethinkdb:rethinkdb-driver` from 2.3.3 to 2.4.4
- [Release notes](https://github.com/rethinkdb/rethinkdb-java/releases)
- [Commits](https://github.com/rethinkdb/rethinkdb-java/commits/v2.4.4)

---
updated-dependencies:
- dependency-name: org.eclipse.jetty:jetty-bom
  dependency-version: 12.0.29
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.29
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.29
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-compiler-plugin
  dependency-version: 3.14.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.codehaus.mojo:versions-maven-plugin
  dependency-version: 2.19.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.sonatype.central:central-publishing-maven-plugin
  dependency-version: 0.9.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-gpg-plugin
  dependency-version: 3.2.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-core
  dependency-version: 6.2.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
  dependency-version: 6.2.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
  dependency-version: 6.2.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
  dependency-version: 6.2.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
  dependency-version: 6.2.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
  dependency-version: 6.2.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
  dependency-version: 6.2.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.netpreserve.commons:webarchive-commons
  dependency-version: 3.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.github.mwiede:jsch
  dependency-version: 2.27.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.apache.pdfbox:pdfbox
  dependency-version: 3.0.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.glassfish.jaxb:jaxb-runtime
  dependency-version: 4.0.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars:webjars-locator-lite
  dependency-version: 1.1.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-assembly-plugin
  dependency-version: 3.7.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: com.rabbitmq:amqp-client
  dependency-version: 5.27.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: com.rethinkdb:rethinkdb-driver
  dependency-version: 2.4.4
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add json-simple as contrib dependency

Needed for TroughClient and no longer included via rethinkdb.

* Exclude junit from json-simple

* Add Jackson BOM 2.20.0 to dependency management

This ensures rethinkdb doesn't pull in an old version.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Alex Osborne <aosborne@nla.gov.au>
 2025-10-27 15:42:40 +09:00
dependabot[bot] 237d19a5d4 Bump org.apache.maven.plugins:maven-surefire-plugin from 2.22.2 to 3.5.4 
Bumps [org.apache.maven.plugins:maven-surefire-plugin](https://github.com/apache/maven-surefire) from 2.22.2 to 3.5.4.
- [Release notes](https://github.com/apache/maven-surefire/releases)
- [Commits](https://github.com/apache/maven-surefire/compare/surefire-2.22.2...surefire-3.5.4)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-surefire-plugin
  dependency-version: 3.5.4
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2025-10-27 01:14:45 +00:00
Alex Osborne b988940eff [maven-release-plugin] prepare for next development iteration 2025-09-22 13:57:47 +09:00
Alex Osborne e3bdd7220e [maven-release-plugin] prepare release 3.11.0 2025-09-22 13:57:41 +09:00
Alex Osborne 20dcc89244 [maven-release-plugin] prepare for next development iteration 2025-08-29 17:26:11 +09:00
Alex Osborne 7a2c676dc9 [maven-release-plugin] prepare release 3.10.2 2025-08-29 17:26:05 +09:00
Alex Osborne 31539e2d62 Use Groovy BOM to ensure consistent versions 2025-08-28 15:35:14 +09:00
Alex Osborne e42306ceb1 Use slf4j BOM to ensure consistent versions 2025-08-28 15:32:48 +09:00
Alex Osborne 3cde816e95 Use jetty BOM to ensure consistent versions 2025-08-28 15:27:10 +09:00
Alex Osborne ea67b59e51 [maven-release-plugin] prepare for next development iteration 2025-07-21 17:15:17 +09:00
Alex Osborne 1cf6762c7f [maven-release-plugin] prepare release 3.10.1 2025-07-21 17:15:12 +09:00
Alex Osborne 53f154574d Bump crawler-commons from 1.4 to 1.5 2025-07-21 16:10:15 +09:00
Alex Osborne 01378ddbbf [maven-release-plugin] prepare for next development iteration 2025-06-12 22:01:16 +09:00
Alex Osborne ecc46a38c3 [maven-release-plugin] prepare release 3.10.0 2025-06-12 22:01:10 +09:00
Alex Osborne 722552feef Add Browser processor using WebDriver BiDi 
The Browser processor can load a fetched page in a local web browser, record any requests the browser makes and run behaviors that interact with the page such as scrolling down and extracting links.

This differs from my previous attempt (ExtractorChrome) in a few ways:

- Uses the new WebDriver BiDi standard instead of the Chrome Devtools Protocol. The new protocol is mostly browser-agnostic, more consistent and hopefully more stable.

- Uses a MITM proxy instead of CDP request interception for recording sub-resources. That's partly because BiDi is still missing some key interception APIs. Even so in practice I found the proxy method loads pages faster and more reliably, likely because responses can be streamed incrementally, which helps a lot for large resources or server-sent events.

- Even when HTTP/2 is unavailable, the new FetchHTTP2 module does connection pooling which makes loading browser requests a lot faster. The original FetchHTTP opened a new connection for every request.

- The Browser processor can be configured with a list of behavior beans making it more customizable and extensible.

Obvious areas for future development:

- More Behavior beans: take screenshots, saveg the rendered DOM, run Browsertrix-compatible behavior scripts

- Support for remote WebDrivers (e.g. Selenium Server or cloud services)
 2025-06-04 17:44:42 +09:00
Alex Osborne 1827d93fe6 [maven-release-plugin] prepare for next development iteration 2025-05-13 12:25:24 +09:00
Alex Osborne dd40970f15 [maven-release-plugin] prepare release 3.9.0 2025-05-13 12:25:16 +09:00
Alex Osborne eb1205f6d5 Upgrade all tests to junit 5 2025-05-09 00:07:18 +09:00
Alex Osborne d5e91eb1f3 Bump pdfbox from 3.0.4 to 3.0.5 2025-05-08 17:36:12 +09:00
Alex Osborne fcfef68aff FetchHTTP2: A new fetch module for HTTP/2 and HTTP/3 
This uses Jetty HttpClient since it speaks both protocols, and we
already have it as a dependency via Restlet. This doesn't support all
the options of FetchHTTP, notably proxy and POST requests are missing.

Jetty currently has the HTTP/3 client marked as "experimental, not for
production use" so we disable it by default and don't ship the large
quiche native jar it requires. It does seem to work OK though, at least
in my limited testing so far. The HTTP/3 support currently only responds
to Alt-Svc headers not other ways of discovering HTTP/3 availability
(e.g. HTTPS DNS record).

Fetches that were made via HTTP/2 or HTTP/3 are annotated 'h2' and 'h3'
in the crawl.log. The messages are recorded in the WARC files using
HTTP/1.1 syntax with a WARC-Protocol header. FetchHTTP2 also currently
records HTTP/1.1 messages without transfer-encoding rather than the raw
wire messages.
 2025-04-29 12:47:57 +09:00
Alex Osborne bc259728a0 Update to Jetty 12.0.16 and Restlet 2.6.0-M2 
The Jetty API has changed, which mostly affects test code.

Jetty now does a strict SNI host check which unfortunately causes it to
return "SNI error" for our existing ad-hoc certificates. For now, I've
disabled it to avoid breaking existing deployments but added a
--sni-host-check command-line option so you can re-enable it if
you've configured your own certificate appropriately.
 2025-04-24 17:40:20 +09:00
Alex Osborne e9b60404a6 [maven-release-plugin] prepare for next development iteration 2025-04-01 17:43:31 +09:00
Alex Osborne 2323e2251e [maven-release-plugin] prepare release 3.8.0 2025-04-01 17:43:25 +09:00
Alex Osborne b17e3fcab3 Bump beanshell from 2.0b5 to 2.0b6 2025-04-01 15:27:22 +09:00
Alex Osborne e9227e7f84 Bump pdfbox from 3.0.3 to 3.0.4 2025-03-31 14:51:40 +09:00
Alex Osborne d21a053c6d [maven-release-plugin] prepare for next development iteration 2025-02-03 14:11:22 +09:00
Alex Osborne 370f0480e5 [maven-release-plugin] prepare release 3.7.0 2025-02-03 14:11:18 +09:00
Alex Osborne 8cf3e476d1 Bump ftpserver-core from 1.2.0 to 1.2.1 2025-02-03 14:08:51 +09:00
Alex Osborne 19c24b9006 Add Groovy crawl configs 
This enables crawl configuration files to use Spring's [Groovy Bean Definition DSL] as an optional alternative to Spring XML. It uses the same bean configuration model but the syntax is more terse and human-readable. No more need for `&amp;` in seed URLs. :-)

```groovy
   checkpointService(CheckpointService) {
        checkpointIntervalMinutes = 15
        checkpointsDir = 'checkpoints'
        forgetAllButLatest = true
   }
```

It also enables some powerful scripting capabilities. For example, defining a custom DecideRule directly in the crawl scope:

```groovy
scope(DecideRuleSequence) {
    rules = [
        new RejectDecideRule(),
        // ACCEPT everything linked from a .pdf file
        new PredicatedDecideRule() {
             boolean evaluate(CrawlURI uri) {
                 return uri.via?.path?.endsWith(".pdf")
             }
        },
        // ...
    ]
}
```

The main downsides are defining nested inner beans can be a bit awkward, some of the errors can be cryptic, and you can't just manipulate the config files with an XML parser.

This commit includes a Groovy version of the default crawl profile for reference, but doesn't expose a way to use it in the UI yet. For now, you need to manually create a `crawler-beans.groovy` file in your job directory.

[Groovy Bean Definition DSL]: https://docs.spring.io/spring-framework/reference/core/beans/basics.html#beans-factory-groovy
 2024-12-01 00:14:42 +09:00
Alex Osborne 2b2179d9c7 Update pom.xml for 3.6.1-SNAPSHOT 2024-11-29 21:06:57 +09:00
Alex Osborne be81b3a407 Update maven release plugins for Java compat 2024-11-29 20:57:01 +09:00
Alex Osborne c67ff98b2d [maven-release-plugin] prepare for next development iteration 2024-11-29 20:46:02 +09:00
Alex Osborne 2d903ae772 [maven-release-plugin] prepare release 3.6.0 2024-11-29 20:42:40 +09:00
Alex Osborne d5b4391cce Bump java-socks-proxy-server from 1.1.0 to 4.1.2 2024-11-29 17:12:43 +09:00
Alex Osborne 5778747527 Bump crawler-commons from 1.0 to 1.4 2024-11-29 17:12:43 +09:00
Alex Osborne faf26adbb5 Bump jsch from 0.1.54 (jcraft) to 0.2.21 (mwiede fork) 
The original project has been unmaintained since 2018 and doesn't
support the algorithms to connect to current versions of openssh.
 2024-11-29 17:12:43 +09:00
Alex Osborne 631620c92b Bump groovy from 4.0.21 to 4.0.24 2024-11-29 17:12:43 +09:00
Alex Osborne a93761d0d5 Bump pdfbox from 3.0.1 to 3.0.3 2024-11-29 13:40:27 +09:00
Alex Osborne b736bd6a24 Bump slf4j from 1.6.6 to 2.0.16 2024-11-28 21:05:12 +09:00
Alex Osborne 25c73da7e3 [maven-release-plugin] prepare for next development iteration 2024-10-29 15:18:54 +09:00
Alex Osborne 0aec63b70f [maven-release-plugin] prepare release 3.5.0 2024-10-29 15:01:16 +09:00
Alex Osborne c25e57a4a1 Bump version to 3.4.1-SNAPSHOT 2024-10-18 20:54:12 +09:00
Alex Osborne 7a3d29ad4d Bump ftpserver-core from 1.1.1 to 1.2.0 2024-10-18 20:38:49 +09:00
Alex Osborne cbd7f37ed9 Replace littleproxy with jetty-proxy (to remove dependency on log4j) 
littleproxy has not been updated since 2017 and has various out of date
dependencies. Since we're using jetty for HTTP server testing anyway, we
 may as well use it as the proxy too.
 2024-10-18 17:28:03 +09:00
Alex Osborne c7c492b507 [maven-release-plugin] prepare for next development iteration 2024-09-09 15:28:44 +09:00
Alex Osborne 11fcbe71f3 [maven-release-plugin] prepare release 3.4.0-20240909 2024-09-09 15:28:44 +09:00
Alex Osborne 0c251c7c34 Update groovy to 4.0.21 for JDK 22 support 2024-05-15 23:13:18 +09:00
Alex Osborne 3700628cb2 Switch PDFParser and ExtractorPDF to pdfbox 2024-02-21 14:48:54 +09:00
Alex Osborne 1c07e43ade Update groovy to 4.0.15 for JDK 20 support 2023-09-19 11:21:05 +09:00
Alex Osborne b9cb43e910 Update groovy to 3.0.13 for JDK 19 support 2022-11-09 21:11:45 +09:00