ec62a9b0ad
[maven-release-plugin] prepare release 3.13.0
2025-12-11 13:54:07 +09:00
8a7db14a51
Add sizeOnDisk on job status ( #699 )
2025-12-09 15:59:03 +01:00
e4523d63ee
fix 690 issue Consolidate and log on deleting URIs from BdB database
2025-11-22 18:42:36 +05:30
7c1b1639cb
Merge pull request #684 from internetarchive/ato/autocomplete
...
Config editor: implement autocompletion
2025-10-30 10:42:19 +09:00
ab41c7a658
[maven-release-plugin] prepare for next development iteration
2025-10-30 10:25:16 +09:00
b626f52d67
[maven-release-plugin] prepare release 3.12.0
2025-10-30 10:25:11 +09:00
c0e5018eef
Config editor: implement autocompletion
...
This adds context-sensitive autocompletion for bean classes, bean ids, property names, basic Spring XML tags and attributes. The bean completions use a new `/engine/beandoc` endpoint that serves the combination of all the `/META-INF/heritrix-beans.json` files generated at compile-time by the heritrix-docgen annotation processor.
2025-10-30 09:53:16 +09:00
d627118307
Bump the minor-and-patch group across 1 directory with 18 updates ( #680 )
...
* Bump the minor-and-patch group across 1 directory with 18 updates
Bumps the minor-and-patch group with 18 updates in the / directory:
| Package | From | To |
| --- | --- | --- |
| [org.eclipse.jetty:jetty-bom](https://github.com/jetty/jetty.project ) | `12.0.27` | `12.0.29` |
| [org.eclipse.jetty.ee10:jetty-ee10-bom](https://github.com/jetty/jetty.project ) | `12.0.27` | `12.0.29` |
| [org.apache.maven.plugins:maven-compiler-plugin](https://github.com/apache/maven-compiler-plugin ) | `3.13.0` | `3.14.1` |
| [org.codehaus.mojo:versions-maven-plugin](https://github.com/mojohaus/versions ) | `2.18.0` | `2.19.1` |
| [org.sonatype.central:central-publishing-maven-plugin](https://github.com/sonatype/central-publishing-maven-plugin ) | `0.7.0` | `0.9.0` |
| [org.apache.maven.plugins:maven-gpg-plugin](https://github.com/apache/maven-gpg-plugin ) | `3.2.7` | `3.2.8` |
| [org.springframework:spring-core](https://github.com/spring-projects/spring-framework ) | `6.2.11` | `6.2.12` |
| [org.springframework:spring-beans](https://github.com/spring-projects/spring-framework ) | `6.2.11` | `6.2.12` |
| [org.springframework:spring-context](https://github.com/spring-projects/spring-framework ) | `6.2.11` | `6.2.12` |
| [org.springframework:spring-expression](https://github.com/spring-projects/spring-framework ) | `6.2.11` | `6.2.12` |
| [org.netpreserve.commons:webarchive-commons](https://github.com/iipc/webarchive-commons ) | `3.0.0` | `3.0.1` |
| [com.github.mwiede:jsch](https://github.com/mwiede/jsch ) | `2.27.3` | `2.27.4` |
| org.apache.pdfbox:pdfbox | `3.0.5` | `3.0.6` |
| org.glassfish.jaxb:jaxb-runtime | `4.0.5` | `4.0.6` |
| [org.webjars:webjars-locator-lite](https://github.com/webjars/webjars-locator-lite ) | `1.1.0` | `1.1.2` |
| [org.apache.maven.plugins:maven-assembly-plugin](https://github.com/apache/maven-assembly-plugin ) | `3.3.0` | `3.7.1` |
| [com.rabbitmq:amqp-client](https://github.com/rabbitmq/rabbitmq-java-client ) | `5.26.0` | `5.27.0` |
| [com.rethinkdb:rethinkdb-driver](https://github.com/rethinkdb/rethinkdb-java ) | `2.3.3` | `2.4.4` |
Updates `org.eclipse.jetty:jetty-bom` from 12.0.27 to 12.0.29
- [Release notes](https://github.com/jetty/jetty.project/releases )
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.27...jetty-12.0.29 )
Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.27 to 12.0.29
- [Release notes](https://github.com/jetty/jetty.project/releases )
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.27...jetty-12.0.29 )
Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.27 to 12.0.29
- [Release notes](https://github.com/jetty/jetty.project/releases )
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.27...jetty-12.0.29 )
Updates `org.apache.maven.plugins:maven-compiler-plugin` from 3.13.0 to 3.14.1
- [Release notes](https://github.com/apache/maven-compiler-plugin/releases )
- [Commits](https://github.com/apache/maven-compiler-plugin/compare/maven-compiler-plugin-3.13.0...maven-compiler-plugin-3.14.1 )
Updates `org.codehaus.mojo:versions-maven-plugin` from 2.18.0 to 2.19.1
- [Release notes](https://github.com/mojohaus/versions/releases )
- [Changelog](https://github.com/mojohaus/versions/blob/master/ReleaseNotes.md )
- [Commits](https://github.com/mojohaus/versions/compare/2.18.0...2.19.1 )
Updates `org.sonatype.central:central-publishing-maven-plugin` from 0.7.0 to 0.9.0
- [Commits](https://github.com/sonatype/central-publishing-maven-plugin/commits )
Updates `org.apache.maven.plugins:maven-gpg-plugin` from 3.2.7 to 3.2.8
- [Release notes](https://github.com/apache/maven-gpg-plugin/releases )
- [Commits](https://github.com/apache/maven-gpg-plugin/compare/maven-gpg-plugin-3.2.7...maven-gpg-plugin-3.2.8 )
Updates `org.springframework:spring-core` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases )
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12 )
Updates `org.springframework:spring-beans` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases )
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12 )
Updates `org.springframework:spring-context` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases )
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12 )
Updates `org.springframework:spring-expression` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases )
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12 )
Updates `org.springframework:spring-beans` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases )
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12 )
Updates `org.springframework:spring-context` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases )
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12 )
Updates `org.springframework:spring-expression` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases )
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12 )
Updates `org.netpreserve.commons:webarchive-commons` from 3.0.0 to 3.0.1
- [Release notes](https://github.com/iipc/webarchive-commons/releases )
- [Changelog](https://github.com/iipc/webarchive-commons/blob/master/CHANGES.md )
- [Commits](https://github.com/iipc/webarchive-commons/compare/webarchive-commons-3.0.0...webarchive-commons-3.0.1 )
Updates `com.github.mwiede:jsch` from 2.27.3 to 2.27.4
- [Release notes](https://github.com/mwiede/jsch/releases )
- [Changelog](https://github.com/mwiede/jsch/blob/master/ChangeLog )
- [Commits](https://github.com/mwiede/jsch/compare/jsch-2.27.3...jsch-2.27.4 )
Updates `org.apache.pdfbox:pdfbox` from 3.0.5 to 3.0.6
Updates `org.glassfish.jaxb:jaxb-runtime` from 4.0.5 to 4.0.6
Updates `org.webjars:webjars-locator-lite` from 1.1.0 to 1.1.2
- [Release notes](https://github.com/webjars/webjars-locator-lite/releases )
- [Commits](https://github.com/webjars/webjars-locator-lite/compare/webjars-locator-lite-1.1.0...webjars-locator-lite-1.1.2 )
Updates `org.apache.maven.plugins:maven-assembly-plugin` from 3.3.0 to 3.7.1
- [Release notes](https://github.com/apache/maven-assembly-plugin/releases )
- [Commits](https://github.com/apache/maven-assembly-plugin/compare/maven-assembly-plugin-3.3.0...maven-assembly-plugin-3.7.1 )
Updates `com.rabbitmq:amqp-client` from 5.26.0 to 5.27.0
- [Release notes](https://github.com/rabbitmq/rabbitmq-java-client/releases )
- [Commits](https://github.com/rabbitmq/rabbitmq-java-client/compare/v5.26.0...v5.27.0 )
Updates `com.rethinkdb:rethinkdb-driver` from 2.3.3 to 2.4.4
- [Release notes](https://github.com/rethinkdb/rethinkdb-java/releases )
- [Commits](https://github.com/rethinkdb/rethinkdb-java/commits/v2.4.4 )
---
updated-dependencies:
- dependency-name: org.eclipse.jetty:jetty-bom
dependency-version: 12.0.29
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
dependency-version: 12.0.29
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
dependency-version: 12.0.29
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-compiler-plugin
dependency-version: 3.14.1
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: org.codehaus.mojo:versions-maven-plugin
dependency-version: 2.19.1
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: org.sonatype.central:central-publishing-maven-plugin
dependency-version: 0.9.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-gpg-plugin
dependency-version: 3.2.8
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-core
dependency-version: 6.2.12
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
dependency-version: 6.2.12
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
dependency-version: 6.2.12
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
dependency-version: 6.2.12
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
dependency-version: 6.2.12
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
dependency-version: 6.2.12
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
dependency-version: 6.2.12
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.netpreserve.commons:webarchive-commons
dependency-version: 3.0.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: com.github.mwiede:jsch
dependency-version: 2.27.4
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.apache.pdfbox:pdfbox
dependency-version: 3.0.6
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.glassfish.jaxb:jaxb-runtime
dependency-version: 4.0.6
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.webjars:webjars-locator-lite
dependency-version: 1.1.2
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-assembly-plugin
dependency-version: 3.7.1
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: com.rabbitmq:amqp-client
dependency-version: 5.27.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: com.rethinkdb:rethinkdb-driver
dependency-version: 2.4.4
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
...
Signed-off-by: dependabot[bot] <support@github.com >
* Add json-simple as contrib dependency
Needed for TroughClient and no longer included via rethinkdb.
* Exclude junit from json-simple
* Add Jackson BOM 2.20.0 to dependency management
This ensures rethinkdb doesn't pull in an old version.
---------
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Alex Osborne <aosborne@nla.gov.au >
2025-10-27 15:42:40 +09:00
bfe45b25cd
Merge pull request #675 from internetarchive/dependabot/maven/org.apache.maven.plugins-maven-surefire-plugin-3.5.4
...
Bump org.apache.maven.plugins:maven-surefire-plugin from 2.22.2 to 3.5.4
2025-10-27 10:20:26 +09:00
237d19a5d4
Bump org.apache.maven.plugins:maven-surefire-plugin from 2.22.2 to 3.5.4
...
Bumps [org.apache.maven.plugins:maven-surefire-plugin](https://github.com/apache/maven-surefire ) from 2.22.2 to 3.5.4.
- [Release notes](https://github.com/apache/maven-surefire/releases )
- [Commits](https://github.com/apache/maven-surefire/compare/surefire-2.22.2...surefire-3.5.4 )
---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-surefire-plugin
dependency-version: 3.5.4
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com >
2025-10-27 01:14:45 +00:00
558727ac37
fix handle more bdb shutdown interrupts
2025-10-03 17:17:46 -07:00
38e7156ef6
fix: don't restore crawlEndTime when resuming from checkpoint.
2025-09-25 10:50:47 -07:00
b988940eff
[maven-release-plugin] prepare for next development iteration
2025-09-22 13:57:47 +09:00
e3bdd7220e
[maven-release-plugin] prepare release 3.11.0
2025-09-22 13:57:41 +09:00
25b621ce61
Bump jakarta.xml.bind-api from 4.0.2 to 4.0.4
2025-09-22 13:51:48 +09:00
cb986a0ed2
Bump codemirror__language from 6.11.2 to 6.11.3
2025-09-22 13:50:01 +09:00
fd2aed89fb
feat: strip invalid chars from xml rest api output
2025-09-11 17:42:07 -07:00
20dcc89244
[maven-release-plugin] prepare for next development iteration
2025-08-29 17:26:11 +09:00
7a2c676dc9
[maven-release-plugin] prepare release 3.10.2
2025-08-29 17:26:05 +09:00
e42306ceb1
Use slf4j BOM to ensure consistent versions
2025-08-28 15:32:48 +09:00
3cde816e95
Use jetty BOM to ensure consistent versions
2025-08-28 15:27:10 +09:00
b1be6fbe0d
Bump codemirror__view from 6.37.2 to 6.38.1
2025-08-20 17:05:03 +09:00
5983f61626
Bump codemirror@legacy-modes from 6.5.0 to 6.5.1
2025-08-20 16:53:23 +09:00
c8c2512e77
Bump codemirror@language from 6.11.1 to 6.11.2
2025-08-20 16:53:10 +09:00
ea67b59e51
[maven-release-plugin] prepare for next development iteration
2025-07-21 17:15:17 +09:00
1cf6762c7f
[maven-release-plugin] prepare release 3.10.1
2025-07-21 17:15:12 +09:00
2388db4229
Bump commons-lang from 2.6 to 3.18.0
2025-07-21 16:55:31 +09:00
8eaaa27ae7
Bump codemirror@search to 6.5.11 and codemirror@state to 6.37.2
2025-07-21 16:06:32 +09:00
c45544a47b
Bump restlet from 2.6.0-rc1 to 2.6.0
2025-07-21 16:03:24 +09:00
85ceab905c
BrowserProcessor: create scratchDir on start
...
This fixes an exception on first launch.
2025-07-21 11:55:37 +09:00
db63c7ec85
Seeds report: record redirect URI for statuses 303, 307 and 308
...
These can also redirect, not just 301 and 302.
Fixes #564
2025-06-30 22:24:29 +09:00
01378ddbbf
[maven-release-plugin] prepare for next development iteration
2025-06-12 22:01:16 +09:00
ecc46a38c3
[maven-release-plugin] prepare release 3.10.0
2025-06-12 22:01:10 +09:00
9b5900b528
FetchHTTP2: Avoid silently decoding gzip
...
Fixes `Recorder getContentReplayInputStream
WARNING: gzip problem; using raw entity instead`.
2025-06-12 21:36:14 +09:00
0494d63ea2
fix: Avoid a NullPointerException when terminate is called again during shutdown
2025-06-11 17:44:52 -07:00
61af1d83c4
MitmProxy: Handle URLs generated by browsers but disallowed by java.net.URI
2025-06-11 20:38:54 +09:00
8a15156fa1
BrowserProcessor: Catch WebDriver timeouts and exceptions
2025-06-11 17:17:02 +09:00
c3f9afe64b
BrowserProcessor: use FetchHTTP2's configured proxy as an upstream proxy
2025-06-10 09:01:02 +09:00
e68e4975c9
Rename to BrowserProcessor and -Behavior
...
After I added them to the Bean reference they seemed out of place with the DecideRules and RecordBuilders.
We had two classes called 'Browser' which is potentially confusing. It's also probably good to more clearly differentiate the ExtractLinks browser behavior from the Extractor processors.
2025-06-09 15:35:25 +09:00
cf447f3bd9
Add Browser processor and behaviors to bean reference
2025-06-09 15:19:09 +09:00
6782fc065d
Update dependencies
...
- **junit-jupiter**: 5.12.2 → 5.13.1
- **spring**: 6.2.6 → 6.2.7
- **codemirror__language**: 6.11.0 → 6.11.1
- **codemirror__view**: 6.36.5 → 6.37.1
2025-06-09 10:17:16 +09:00
1161d87217
Browser: Add processor report
2025-06-05 10:48:15 +09:00
2210cb7262
Browser: Handle navigation abort from downloads starting
2025-06-04 17:44:42 +09:00
5131ea4a80
Browser: Disable downloads in Firefox and Chrome
...
This hopefully will stop us filling up ~/Downloads with random junk.
2025-06-04 17:44:42 +09:00
722552feef
Add Browser processor using WebDriver BiDi
...
The Browser processor can load a fetched page in a local web browser, record any requests the browser makes and run behaviors that interact with the page such as scrolling down and extracting links.
This differs from my previous attempt (ExtractorChrome) in a few ways:
- Uses the new WebDriver BiDi standard instead of the Chrome Devtools Protocol. The new protocol is mostly browser-agnostic, more consistent and hopefully more stable.
- Uses a MITM proxy instead of CDP request interception for recording sub-resources. That's partly because BiDi is still missing some key interception APIs. Even so in practice I found the proxy method loads pages faster and more reliably, likely because responses can be streamed incrementally, which helps a lot for large resources or server-sent events.
- Even when HTTP/2 is unavailable, the new FetchHTTP2 module does connection pooling which makes loading browser requests a lot faster. The original FetchHTTP opened a new connection for every request.
- The Browser processor can be configured with a list of behavior beans making it more customizable and extensible.
Obvious areas for future development:
- More Behavior beans: take screenshots, saveg the rendered DOM, run Browsertrix-compatible behavior scripts
- Support for remote WebDrivers (e.g. Selenium Server or cloud services)
2025-06-04 17:44:42 +09:00
eaf72b5ce2
Add --web-auth basic command-line option
...
This option enables HTTP Basic authentication for the web interface instead of the default Digest authentication. This is useful when running Heritrix behind a reverse proxy that adds external authentication as typically they don't support Digest auth for the upstream server.
#641
2025-05-25 00:29:14 +09:00
0f0db370ee
UI: Disable Jetty graceful shutdown for faster restarts
...
Graceful shutdown would be useful if you could deploy the UI in a high-availability configuration and direct new requests to a different instance while the current instance finished its outstanding ones. But as you can't, it's just making restarting Heritrix slow for little benefit.
2025-05-24 22:46:42 +09:00
679fde42c6
SelfTestBase: Use a dynamic port (port 0) for Heritrix web port
...
This stops the tests from failing when you happen to be running Heritrix or something else on port 8443.
2025-05-22 15:38:45 +09:00
567e27181d
Upgrade webarchive-commons from 1.3.0 to 2.0.1 (removes httpclient 3)
2025-05-21 17:39:21 +09:00
33446c7a2b
Upgrade to CodeMirror 6
...
This resolves some browser incompatibilities, allowing CodeMirror’s own find function to be re-enabled for reliable text search of content far outside the viewport.
2025-05-16 18:34:30 +09:00
Alex Osborne