lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-19 20:25:45 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,958 Commits 38 Branches 33 Tags
2cbc5ec018a178bb44e7d4163aa7b0fa6d26bfa9
Commit Graph
2958 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
dependabot[bot] 2cbc5ec018 Bump actions/cache from 4 to 5 
Bumps [actions/cache](https://github.com/actions/cache) from 4 to 5.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-06-17 17:17:37 +09:00
dependabot[bot] e7fd9e0c3f Bump actions/checkout from 2 to 6 
Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 6.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v2...v6)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-06-17 17:17:19 +09:00
Alex Osborne a6adb1f4f9 dependabot: update github-actions monthly 2026-06-17 17:01:41 +09:00
Alex Osborne a904253f3c Merge pull request #736 from internetarchive/pagination-behavior 
PaginationBehavior: repeatedly clicks next-page and extracts links
 2026-06-12 17:20:19 +09:00
Alex Osborne 709ac00dd2 Add PaginationBehavior: repeatedly clicks next-page and extracts links 2026-06-11 13:34:55 +09:00
Alex Osborne 54b95f5e97 groovy profile: remove stray } 2026-06-10 18:23:12 +09:00
Alex Osborne fc76935fea webdriver: Support Cookie.sameSite value "default" 2026-06-10 18:19:04 +09:00
Alex Osborne bd34a7a0a9 Log the exception when creating new jobs fails 2026-06-10 17:43:57 +09:00
Alex Osborne 4b1c1d1e57 Merge pull request #735 from internetarchive/dependabot/maven/minor-and-patch-e83b0aec52 
Bump the minor-and-patch group with 11 updates
 2026-06-08 15:58:53 +09:00
dependabot[bot] 3406c14632 Bump the minor-and-patch group with 11 updates 
Bumps the minor-and-patch group with 11 updates:

| Package | From | To |
| --- | --- | --- |
| [org.junit.jupiter:junit-jupiter](https://github.com/junit-team/junit-framework) | `6.0.3` | `6.1.0` |
| [org.apache.groovy:groovy-bom](https://github.com/apache/groovy) | `5.0.5` | `5.0.6` |
| [com.fasterxml.jackson:jackson-bom](https://github.com/FasterXML/jackson-bom) | `2.21.3` | `2.22.0` |
| [org.eclipse.jetty:jetty-bom](https://github.com/jetty/jetty.project) | `12.0.34` | `12.0.36` |
| [org.eclipse.jetty.ee10:jetty-ee10-bom](https://github.com/jetty/jetty.project) | `12.0.34` | `12.0.36` |
| [org.slf4j:slf4j-bom](https://github.com/qos-ch/slf4j) | `2.0.17` | `2.0.18` |
| [org.apache.maven.plugins:maven-surefire-plugin](https://github.com/apache/maven-surefire) | `3.5.5` | `3.5.6` |
| [dnsjava:dnsjava](https://github.com/dnsjava/dnsjava) | `3.6.4` | `3.6.5` |
| org.glassfish.jaxb:jaxb-runtime | `4.0.7` | `4.0.9` |
| [com.rabbitmq:amqp-client](https://github.com/rabbitmq/rabbitmq-java-client) | `5.30.0` | `5.31.0` |
| org.apache.kafka:kafka-clients | `4.2.0` | `4.3.0` |


Updates `org.junit.jupiter:junit-jupiter` from 6.0.3 to 6.1.0
- [Release notes](https://github.com/junit-team/junit-framework/releases)
- [Commits](https://github.com/junit-team/junit-framework/compare/r6.0.3...r6.1.0)

Updates `org.apache.groovy:groovy-bom` from 5.0.5 to 5.0.6
- [Commits](https://github.com/apache/groovy/commits)

Updates `com.fasterxml.jackson:jackson-bom` from 2.21.3 to 2.22.0
- [Commits](https://github.com/FasterXML/jackson-bom/compare/jackson-bom-2.21.3...jackson-bom-2.22.0)

Updates `org.eclipse.jetty:jetty-bom` from 12.0.34 to 12.0.36
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.34...jetty-12.0.36)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.34 to 12.0.36
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.34...jetty-12.0.36)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.34 to 12.0.36
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.34...jetty-12.0.36)

Updates `org.slf4j:slf4j-bom` from 2.0.17 to 2.0.18
- [Release notes](https://github.com/qos-ch/slf4j/releases)
- [Commits](https://github.com/qos-ch/slf4j/compare/v_2.0.17...v_2.0.18)

Updates `org.apache.maven.plugins:maven-surefire-plugin` from 3.5.5 to 3.5.6
- [Release notes](https://github.com/apache/maven-surefire/releases)
- [Commits](https://github.com/apache/maven-surefire/compare/surefire-3.5.5...surefire-3.5.6)

Updates `dnsjava:dnsjava` from 3.6.4 to 3.6.5
- [Release notes](https://github.com/dnsjava/dnsjava/releases)
- [Changelog](https://github.com/dnsjava/dnsjava/blob/master/Changelog)
- [Commits](https://github.com/dnsjava/dnsjava/commits)

Updates `org.glassfish.jaxb:jaxb-runtime` from 4.0.7 to 4.0.9

Updates `com.rabbitmq:amqp-client` from 5.30.0 to 5.31.0
- [Release notes](https://github.com/rabbitmq/rabbitmq-java-client/releases)
- [Commits](https://github.com/rabbitmq/rabbitmq-java-client/compare/v5.30.0...v5.31.0)

Updates `org.apache.kafka:kafka-clients` from 4.2.0 to 4.3.0

---
updated-dependencies:
- dependency-name: org.junit.jupiter:junit-jupiter
  dependency-version: 6.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.apache.groovy:groovy-bom
  dependency-version: 5.0.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.fasterxml.jackson:jackson-bom
  dependency-version: 2.22.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty:jetty-bom
  dependency-version: 12.0.36
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.36
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.36
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.slf4j:slf4j-bom
  dependency-version: 2.0.18
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-surefire-plugin
  dependency-version: 3.5.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: dnsjava:dnsjava
  dependency-version: 3.6.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.glassfish.jaxb:jaxb-runtime
  dependency-version: 4.0.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.rabbitmq:amqp-client
  dependency-version: 5.31.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.apache.kafka:kafka-clients
  dependency-version: 4.3.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-06-03 16:56:20 +00:00
Alex Osborne e0b46789bc [maven-release-plugin] prepare for next development iteration 2026-05-26 11:51:31 +09:00
Alex Osborne bc28f4d1ae [maven-release-plugin] prepare release 3.15.0 3.15.0 2026-05-26 11:51:25 +09:00
Alex Osborne b4dd4949d7 Update CHANGELOG with links for 3.15.0 2026-05-26 11:50:38 +09:00
Alex Osborne 36a3088994 Update CHANGELOG for 3.15.0 2026-05-26 11:47:37 +09:00
Alex Osborne f919ba2629 Merge pull request #734 from matasaru/master 
fixed maven-central badge
 2026-05-20 14:56:02 +09:00
Alex Osborne b9b4823c26 Merge pull request #733 from adam-miller/adam/apply_attribute_name_limit_to_all_names 
fix: cap dynamic attribute names to avoid pathological loop.
 2026-05-20 14:55:05 +09:00
radu 540cea06c1 fixed maven-central badge 2026-05-18 13:22:33 -04:00
Adam Miller e09b33069a fix: cap dynamic attribute names to avoid pathological loop. 2026-05-15 11:29:31 -07:00
Alex Osborne 4058c10238 Merge pull request #731 from internetarchive/groovy-docs 
docs: Add Groovy versions of config examples
 2026-05-12 16:44:32 +09:00
Alex Osborne d3defaacd2 Merge pull request #732 from internetarchive/remove-foundation-js 
Remove obsolete JavaScript libraries
 2026-05-12 16:44:16 +09:00
Alex Osborne 56a1fc2198 Remove Internet Explorer compatibility shims 
There's no need for these anymore, IE has been sunset.
 2026-05-04 08:45:41 +09:00
Alex Osborne 05f5243207 Remove Foundation JS libraries 
Foundation 4 has been obsolete since 2013, and these libraries trigger security alerts. We're also not using them to do much, just the modal dialogs, alert close buttons and collapsing the top-bar on mobile. This replaces the modals with the native HTML dialog element and implements the trivial click event handlers needed for hiding and showing things.
 2026-05-04 08:45:40 +09:00
Alex Osborne 3718480cd5 docs: Add Groovy versions of config examples 
Uses sphinx-inline-tabs to render tabs offering the choice between XML and Groovy syntax for each example.
 2026-05-03 12:07:01 +09:00
Alex Osborne 71ae9a013b Merge pull request #728 from internetarchive/groovy-config-editor 
Add Groovy mode to config editor
 2026-05-03 12:06:26 +09:00
Alex Osborne 40930e260b Merge pull request #725 from internetarchive/profile-selector 
Add profile selector to job creation form
 2026-05-03 12:06:12 +09:00
Alex Osborne ca9a8ec465 Merge pull request #730 from internetarchive/dependabot/maven/minor-and-patch-8ce37d9795 
Bump the minor-and-patch group with 12 updates
 2026-05-02 14:31:12 +09:00
dependabot[bot] bb219db589 Bump the minor-and-patch group with 12 updates 
Bumps the minor-and-patch group with 12 updates:

| Package | From | To |
| --- | --- | --- |
| [com.fasterxml.jackson:jackson-bom](https://github.com/FasterXML/jackson-bom) | `2.21.2` | `2.21.3` |
| commons-io:commons-io | `2.21.0` | `2.22.0` |
| [commons-codec:commons-codec](https://github.com/apache/commons-codec) | `1.21.0` | `1.22.0` |
| [org.springframework:spring-core](https://github.com/spring-projects/spring-framework) | `7.0.6` | `7.0.7` |
| [org.springframework:spring-beans](https://github.com/spring-projects/spring-framework) | `7.0.6` | `7.0.7` |
| [org.springframework:spring-context](https://github.com/spring-projects/spring-framework) | `7.0.6` | `7.0.7` |
| [org.springframework:spring-expression](https://github.com/spring-projects/spring-framework) | `7.0.6` | `7.0.7` |
| [org.netpreserve.commons:webarchive-commons](https://github.com/iipc/webarchive-commons) | `3.0.3` | `3.0.4` |
| [com.github.mwiede:jsch](https://github.com/mwiede/jsch) | `2.28.0` | `2.28.2` |
| [com.rabbitmq:amqp-client](https://github.com/rabbitmq/rabbitmq-java-client) | `5.29.0` | `5.30.0` |
| [at.yawk.lz4:lz4-java](https://github.com/yawkat/lz4-java) | `1.10.4` | `1.11.0` |
| [com.google.code.gson:gson](https://github.com/google/gson) | `2.13.2` | `2.14.0` |


Updates `com.fasterxml.jackson:jackson-bom` from 2.21.2 to 2.21.3
- [Commits](https://github.com/FasterXML/jackson-bom/compare/jackson-bom-2.21.2...jackson-bom-2.21.3)

Updates `commons-io:commons-io` from 2.21.0 to 2.22.0

Updates `commons-codec:commons-codec` from 1.21.0 to 1.22.0
- [Changelog](https://github.com/apache/commons-codec/blob/master/RELEASE-NOTES.txt)
- [Commits](https://github.com/apache/commons-codec/compare/rel/commons-codec-1.21.0...rel/commons-codec-1.22.0)

Updates `org.springframework:spring-core` from 7.0.6 to 7.0.7
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.6...v7.0.7)

Updates `org.springframework:spring-beans` from 7.0.6 to 7.0.7
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.6...v7.0.7)

Updates `org.springframework:spring-context` from 7.0.6 to 7.0.7
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.6...v7.0.7)

Updates `org.springframework:spring-expression` from 7.0.6 to 7.0.7
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.6...v7.0.7)

Updates `org.springframework:spring-beans` from 7.0.6 to 7.0.7
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.6...v7.0.7)

Updates `org.springframework:spring-context` from 7.0.6 to 7.0.7
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.6...v7.0.7)

Updates `org.springframework:spring-expression` from 7.0.6 to 7.0.7
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.6...v7.0.7)

Updates `org.netpreserve.commons:webarchive-commons` from 3.0.3 to 3.0.4
- [Release notes](https://github.com/iipc/webarchive-commons/releases)
- [Changelog](https://github.com/iipc/webarchive-commons/blob/master/CHANGES.md)
- [Commits](https://github.com/iipc/webarchive-commons/compare/webarchive-commons-3.0.3...webarchive-commons-3.0.4)

Updates `com.github.mwiede:jsch` from 2.28.0 to 2.28.2
- [Release notes](https://github.com/mwiede/jsch/releases)
- [Changelog](https://github.com/mwiede/jsch/blob/master/ChangeLog)
- [Commits](https://github.com/mwiede/jsch/compare/jsch-2.28.0...jsch-2.28.2)

Updates `com.rabbitmq:amqp-client` from 5.29.0 to 5.30.0
- [Release notes](https://github.com/rabbitmq/rabbitmq-java-client/releases)
- [Commits](https://github.com/rabbitmq/rabbitmq-java-client/compare/v5.29.0...v5.30.0)

Updates `at.yawk.lz4:lz4-java` from 1.10.4 to 1.11.0
- [Release notes](https://github.com/yawkat/lz4-java/releases)
- [Changelog](https://github.com/yawkat/lz4-java/blob/main/CHANGES.md)
- [Commits](https://github.com/yawkat/lz4-java/compare/v1.10.4...v1.11.0)

Updates `com.google.code.gson:gson` from 2.13.2 to 2.14.0
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/main/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.13.2...gson-parent-2.14.0)

---
updated-dependencies:
- dependency-name: com.fasterxml.jackson:jackson-bom
  dependency-version: 2.21.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: commons-io:commons-io
  dependency-version: 2.22.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: commons-codec:commons-codec
  dependency-version: 1.22.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-core
  dependency-version: 7.0.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
  dependency-version: 7.0.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
  dependency-version: 7.0.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
  dependency-version: 7.0.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
  dependency-version: 7.0.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
  dependency-version: 7.0.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
  dependency-version: 7.0.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.netpreserve.commons:webarchive-commons
  dependency-version: 3.0.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.github.mwiede:jsch
  dependency-version: 2.28.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.rabbitmq:amqp-client
  dependency-version: 5.30.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: at.yawk.lz4:lz4-java
  dependency-version: 1.11.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: com.google.code.gson:gson
  dependency-version: 2.14.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-05-02 03:05:43 +00:00
Alex Osborne 82181f3d9e Add Groovy mode to config editor 
Includes bean and property name autocomplete, similar to the XML mode. Completion of short class names auto-inserts import statements as needed.
 2026-04-28 09:08:23 +09:00
Alex Osborne 9e43fde053 Recognise Groovy profiles during job discovery 2026-04-27 17:56:23 +09:00
Alex Osborne b1e13c550b Add profile selector to job creation form 
This means:

- Job profiles can be used without having to hunt for them among the regular jobs.
- The Groovy default profile can be selected as an alternative to Spring XML one.
- The default profiles can be customised by creating a job profile with the same name.
 2026-04-27 17:40:37 +09:00
Alex Osborne 55c1c20295 Remove httpProxyHost and httpProxyPort from fetcher properties after test execution 2026-04-25 21:28:44 +09:00
Alex Osborne bb0c7e918d Correct date typos in CHANGELOG.md 2026-04-18 19:24:43 +09:00
Alex Osborne c77a3a4310 Merge pull request #723 from larsnaesbye/master 
Update Dockerfile to latest Heritrix
 2026-04-18 19:22:40 +09:00
Lars Næsbye Christensen b016e44b19 we should use COPY instead of ADD when only copying local files 2026-04-16 13:01:40 +02:00
Lars Næsbye Christensen 00d1eaefa0 update Dockerfile to build version 3.14.1 of Heritrix 2026-04-16 12:52:14 +02:00
Alex Osborne 94aaebfca2 [maven-release-plugin] prepare for next development iteration 2026-04-06 17:11:14 +09:00
Alex Osborne 56f8aee836 [maven-release-plugin] prepare release 3.14.1 3.14.1 2026-04-06 17:11:04 +09:00
Alex Osborne 08dfe5aa97 Update CHANGELOG.md for 3.14.1 2026-04-06 17:06:33 +09:00
Alex Osborne 7490998843 Merge pull request #721 from simons-hub/fix/warn-on-precedence-clipping 
Log warning when URI precedence exceeds maximum 127
 2026-04-06 16:54:56 +09:00
Alex Osborne b229185932 Merge pull request #719 from internetarchive/ato/FetchHTTP2-remove-auth-protocol-handlers 
FetchHTTP2: Remove HttpClient default auth protocol handlers
 2026-04-06 16:41:02 +09:00
Alex Osborne 1620dc06ce Merge pull request #713 from internetarchive/xxe 
Enable XXE protection when parsing crawl job XML
 2026-04-06 16:40:29 +09:00
Alex Osborne b260ef8abd Merge pull request #722 from internetarchive/dependabot/maven/minor-and-patch-17bda1ffa9 
Bump the minor-and-patch group with 21 updates
 2026-04-02 15:27:21 +09:00
dependabot[bot] 46b198a180 Bump the minor-and-patch group with 21 updates 
Bumps the minor-and-patch group with 21 updates:

| Package | From | To |
| --- | --- | --- |
| [org.apache.groovy:groovy-bom](https://github.com/apache/groovy) | `5.0.4` | `5.0.5` |
| [com.fasterxml.jackson:jackson-bom](https://github.com/FasterXML/jackson-bom) | `2.21.1` | `2.21.2` |
| [org.eclipse.jetty:jetty-bom](https://github.com/jetty/jetty.project) | `12.0.32` | `12.0.34` |
| [org.eclipse.jetty.ee10:jetty-ee10-bom](https://github.com/jetty/jetty.project) | `12.0.32` | `12.0.34` |
| [commons-net:commons-net](https://github.com/apache/commons-net) | `3.12.0` | `3.13.0` |
| [org.springframework:spring-core](https://github.com/spring-projects/spring-framework) | `7.0.5` | `7.0.6` |
| [org.springframework:spring-beans](https://github.com/spring-projects/spring-framework) | `7.0.5` | `7.0.6` |
| [org.springframework:spring-context](https://github.com/spring-projects/spring-framework) | `7.0.5` | `7.0.6` |
| [org.springframework:spring-expression](https://github.com/spring-projects/spring-framework) | `7.0.5` | `7.0.6` |
| [com.github.mwiede:jsch](https://github.com/mwiede/jsch) | `2.27.8` | `2.28.0` |
| [org.apache.maven.plugins:maven-resources-plugin](https://github.com/apache/maven-resources-plugin) | `3.4.0` | `3.5.0` |
| org.apache.pdfbox:pdfbox | `3.0.6` | `3.0.7` |
| org.glassfish.jaxb:jaxb-runtime | `4.0.6` | `4.0.7` |
| [org.webjars.npm:codemirror__autocomplete](https://github.com/codemirror/autocomplete) | `6.20.0` | `6.20.1` |
| [org.webjars.npm:codemirror__commands](https://github.com/codemirror/commands) | `6.10.1` | `6.10.2` |
| [org.webjars.npm:codemirror__language](https://github.com/codemirror/language) | `6.12.1` | `6.12.2` |
| [org.webjars.npm:codemirror__view](https://github.com/codemirror/view) | `6.39.11` | `6.39.16` |
| [org.webjars.npm:codemirror__lint](https://github.com/codemirror/lint) | `6.9.2` | `6.9.5` |
| [org.webjars.npm:lezer__common](https://github.com/lezer-parser/common) | `1.5.0` | `1.5.1` |
| [org.webjars.npm:lezer__lr](https://github.com/lezer-parser/lr) | `1.4.7` | `1.4.8` |
| [at.yawk.lz4:lz4-java](https://github.com/yawkat/lz4-java) | `1.10.3` | `1.10.4` |


Updates `org.apache.groovy:groovy-bom` from 5.0.4 to 5.0.5
- [Commits](https://github.com/apache/groovy/commits)

Updates `com.fasterxml.jackson:jackson-bom` from 2.21.1 to 2.21.2
- [Commits](https://github.com/FasterXML/jackson-bom/compare/jackson-bom-2.21.1...jackson-bom-2.21.2)

Updates `org.eclipse.jetty:jetty-bom` from 12.0.32 to 12.0.34
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.32...jetty-12.0.34)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.32 to 12.0.34
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.32...jetty-12.0.34)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.32 to 12.0.34
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.32...jetty-12.0.34)

Updates `commons-net:commons-net` from 3.12.0 to 3.13.0
- [Changelog](https://github.com/apache/commons-net/blob/master/RELEASE-NOTES.txt)
- [Commits](https://github.com/apache/commons-net/compare/rel/commons-net-3.12.0...rel/commons-net-3.13.0)

Updates `org.springframework:spring-core` from 7.0.5 to 7.0.6
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.5...v7.0.6)

Updates `org.springframework:spring-beans` from 7.0.5 to 7.0.6
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.5...v7.0.6)

Updates `org.springframework:spring-context` from 7.0.5 to 7.0.6
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.5...v7.0.6)

Updates `org.springframework:spring-expression` from 7.0.5 to 7.0.6
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.5...v7.0.6)

Updates `org.springframework:spring-beans` from 7.0.5 to 7.0.6
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.5...v7.0.6)

Updates `org.springframework:spring-context` from 7.0.5 to 7.0.6
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.5...v7.0.6)

Updates `org.springframework:spring-expression` from 7.0.5 to 7.0.6
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.5...v7.0.6)

Updates `com.github.mwiede:jsch` from 2.27.8 to 2.28.0
- [Release notes](https://github.com/mwiede/jsch/releases)
- [Changelog](https://github.com/mwiede/jsch/blob/master/ChangeLog)
- [Commits](https://github.com/mwiede/jsch/compare/jsch-2.27.8...jsch-2.28.0)

Updates `org.apache.maven.plugins:maven-resources-plugin` from 3.4.0 to 3.5.0
- [Release notes](https://github.com/apache/maven-resources-plugin/releases)
- [Commits](https://github.com/apache/maven-resources-plugin/compare/v3.4.0...maven-resources-plugin-3.5.0)

Updates `org.apache.pdfbox:pdfbox` from 3.0.6 to 3.0.7

Updates `org.glassfish.jaxb:jaxb-runtime` from 4.0.6 to 4.0.7

Updates `org.webjars.npm:codemirror__autocomplete` from 6.20.0 to 6.20.1
- [Changelog](https://github.com/codemirror/autocomplete/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codemirror/autocomplete/compare/6.20.0...6.20.1)

Updates `org.webjars.npm:codemirror__commands` from 6.10.1 to 6.10.2
- [Changelog](https://github.com/codemirror/commands/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codemirror/commands/compare/6.10.1...6.10.2)

Updates `org.webjars.npm:codemirror__language` from 6.12.1 to 6.12.2
- [Changelog](https://github.com/codemirror/language/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codemirror/language/compare/6.12.1...6.12.2)

Updates `org.webjars.npm:codemirror__view` from 6.39.11 to 6.39.16
- [Changelog](https://github.com/codemirror/view/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codemirror/view/compare/6.39.11...6.39.16)

Updates `org.webjars.npm:codemirror__lint` from 6.9.2 to 6.9.5
- [Changelog](https://github.com/codemirror/lint/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codemirror/lint/compare/6.9.2...6.9.5)

Updates `org.webjars.npm:lezer__common` from 1.5.0 to 1.5.1
- [Changelog](https://github.com/lezer-parser/common/blob/main/CHANGELOG.md)
- [Commits](https://github.com/lezer-parser/common/compare/1.5.0...1.5.1)

Updates `org.webjars.npm:lezer__lr` from 1.4.7 to 1.4.8
- [Changelog](https://github.com/lezer-parser/lr/blob/main/CHANGELOG.md)
- [Commits](https://github.com/lezer-parser/lr/compare/1.4.7...1.4.8)

Updates `at.yawk.lz4:lz4-java` from 1.10.3 to 1.10.4
- [Release notes](https://github.com/yawkat/lz4-java/releases)
- [Changelog](https://github.com/yawkat/lz4-java/blob/main/CHANGES.md)
- [Commits](https://github.com/yawkat/lz4-java/compare/v1.10.3...v1.10.4)

---
updated-dependencies:
- dependency-name: org.apache.groovy:groovy-bom
  dependency-version: 5.0.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.fasterxml.jackson:jackson-bom
  dependency-version: 2.21.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty:jetty-bom
  dependency-version: 12.0.34
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.34
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.34
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: commons-net:commons-net
  dependency-version: 3.13.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-core
  dependency-version: 7.0.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
  dependency-version: 7.0.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
  dependency-version: 7.0.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
  dependency-version: 7.0.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
  dependency-version: 7.0.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
  dependency-version: 7.0.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
  dependency-version: 7.0.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.github.mwiede:jsch
  dependency-version: 2.28.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-resources-plugin
  dependency-version: 3.5.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.apache.pdfbox:pdfbox
  dependency-version: 3.0.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.glassfish.jaxb:jaxb-runtime
  dependency-version: 4.0.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__autocomplete
  dependency-version: 6.20.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__commands
  dependency-version: 6.10.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__language
  dependency-version: 6.12.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__view
  dependency-version: 6.39.16
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__lint
  dependency-version: 6.9.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:lezer__common
  dependency-version: 1.5.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:lezer__lr
  dependency-version: 1.4.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: at.yawk.lz4:lz4-java
  dependency-version: 1.10.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-04-02 04:04:31 +00:00
Simon 47632eb4c4 Log warning when URI precedence exceeds maximum 127 
BdbMultipleWorkQueues.calculateInsertKey() silently clips precedence
values to 127 for queue ordering. This can cause unexpected behavior
for custom cost assignment policies that return values above 127.

Log a warning when clipping occurs and fix the misleading comment in
CrawlURI.holderCost that said "should not exceed 255" (the actual
limit is 127, not 255).

Fixes #502
 2026-03-31 00:37:36 -07:00
Alex Osborne 3c3ae15a7c Fix test failures due to missing bdb close 2026-03-10 11:37:22 +09:00
Alex Osborne 5b3117644b FetchHTTP2: Fix IllegalArgumentException when host contains underscores 
Fixes #718
 2026-03-10 11:22:32 +09:00
Alex Osborne 9af99add3c FetchHTTP2: Remove HttpClient default auth protocol handlers 
WWWAuthenticationProtocolHandler and ProxyAuthenticationProtocolHandler unhelpfully throw exceptions if a 401 or 407 response has a larger than 16KB body or is missing the appropriate -Authenticate header.

FetchHTTP2 doesn't yet support HTTP auth, thus they aren't currently being used, so let's just remove them for now. This also adds regression tests so that if in the future we implement auth support, we don't accidentally reintroduce this.

Fixes #714
Fixes #717
 2026-03-09 22:44:57 +09:00
Alex Osborne 96737799ed Enable XXE protection when parsing crawl job XML 
Heritrix's Spring XML based job config format by design allows arbitrary code execution, so this doesn't realistically make things much safer, but I also don't see any reason we need external entity resolution enabled. I suppose there's a chance this could mitigate a generic automated attack that uses XXE but doesn't target Heritrix or Spring XML specifically.

Fixes #711
 2026-03-02 10:11:38 +09:00
Alex Osborne bc82cf5ab8 Merge pull request #712 from internetarchive/dependabot/maven/minor-and-patch-574bb45e44 
Bump the minor-and-patch group with 11 updates
 2026-03-02 09:09:05 +09:00
dependabot[bot] 5b2d677498 Bump the minor-and-patch group with 11 updates 
Bumps the minor-and-patch group with 11 updates:

| Package | From | To |
| --- | --- | --- |
| [org.junit.jupiter:junit-jupiter](https://github.com/junit-team/junit-framework) | `6.0.2` | `6.0.3` |
| [com.fasterxml.jackson:jackson-bom](https://github.com/FasterXML/jackson-bom) | `2.21.0` | `2.21.1` |
| [org.apache.maven.plugins:maven-surefire-plugin](https://github.com/apache/maven-surefire) | `3.5.4` | `3.5.5` |
| [org.springframework:spring-core](https://github.com/spring-projects/spring-framework) | `7.0.3` | `7.0.5` |
| [org.springframework:spring-beans](https://github.com/spring-projects/spring-framework) | `7.0.3` | `7.0.5` |
| [org.springframework:spring-context](https://github.com/spring-projects/spring-framework) | `7.0.3` | `7.0.5` |
| [org.springframework:spring-expression](https://github.com/spring-projects/spring-framework) | `7.0.3` | `7.0.5` |
| [com.github.mwiede:jsch](https://github.com/mwiede/jsch) | `2.27.7` | `2.27.8` |
| [jakarta.xml.bind:jakarta.xml.bind-api](https://github.com/jakartaee/jaxb-api) | `4.0.4` | `4.0.5` |
| [com.rabbitmq:amqp-client](https://github.com/rabbitmq/rabbitmq-java-client) | `5.28.0` | `5.29.0` |
| org.apache.kafka:kafka-clients | `4.1.1` | `4.2.0` |


Updates `org.junit.jupiter:junit-jupiter` from 6.0.2 to 6.0.3
- [Release notes](https://github.com/junit-team/junit-framework/releases)
- [Commits](https://github.com/junit-team/junit-framework/compare/r6.0.2...r6.0.3)

Updates `com.fasterxml.jackson:jackson-bom` from 2.21.0 to 2.21.1
- [Commits](https://github.com/FasterXML/jackson-bom/compare/jackson-bom-2.21.0...jackson-bom-2.21.1)

Updates `org.apache.maven.plugins:maven-surefire-plugin` from 3.5.4 to 3.5.5
- [Release notes](https://github.com/apache/maven-surefire/releases)
- [Commits](https://github.com/apache/maven-surefire/compare/surefire-3.5.4...surefire-3.5.5)

Updates `org.springframework:spring-core` from 7.0.3 to 7.0.5
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.3...v7.0.5)

Updates `org.springframework:spring-beans` from 7.0.3 to 7.0.5
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.3...v7.0.5)

Updates `org.springframework:spring-context` from 7.0.3 to 7.0.5
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.3...v7.0.5)

Updates `org.springframework:spring-expression` from 7.0.3 to 7.0.5
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.3...v7.0.5)

Updates `org.springframework:spring-beans` from 7.0.3 to 7.0.5
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.3...v7.0.5)

Updates `org.springframework:spring-context` from 7.0.3 to 7.0.5
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.3...v7.0.5)

Updates `org.springframework:spring-expression` from 7.0.3 to 7.0.5
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.3...v7.0.5)

Updates `com.github.mwiede:jsch` from 2.27.7 to 2.27.8
- [Release notes](https://github.com/mwiede/jsch/releases)
- [Changelog](https://github.com/mwiede/jsch/blob/master/ChangeLog)
- [Commits](https://github.com/mwiede/jsch/compare/jsch-2.27.7...jsch-2.27.8)

Updates `jakarta.xml.bind:jakarta.xml.bind-api` from 4.0.4 to 4.0.5
- [Release notes](https://github.com/jakartaee/jaxb-api/releases)
- [Commits](https://github.com/jakartaee/jaxb-api/compare/4.0.4...4.0.5)

Updates `com.rabbitmq:amqp-client` from 5.28.0 to 5.29.0
- [Release notes](https://github.com/rabbitmq/rabbitmq-java-client/releases)
- [Commits](https://github.com/rabbitmq/rabbitmq-java-client/compare/v5.28.0...v5.29.0)

Updates `org.apache.kafka:kafka-clients` from 4.1.1 to 4.2.0

---
updated-dependencies:
- dependency-name: org.junit.jupiter:junit-jupiter
  dependency-version: 6.0.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.fasterxml.jackson:jackson-bom
  dependency-version: 2.21.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-surefire-plugin
  dependency-version: 3.5.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-core
  dependency-version: 7.0.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
  dependency-version: 7.0.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
  dependency-version: 7.0.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
  dependency-version: 7.0.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
  dependency-version: 7.0.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
  dependency-version: 7.0.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
  dependency-version: 7.0.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.github.mwiede:jsch
  dependency-version: 2.27.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: jakarta.xml.bind:jakarta.xml.bind-api
  dependency-version: 4.0.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.rabbitmq:amqp-client
  dependency-version: 5.29.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.apache.kafka:kafka-clients
  dependency-version: 4.2.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-03-01 23:42:42 +00:00