lee/heritrix3
1
0 Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-10-09 13:51:31 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,045 Commits 41 Branches 34 Tags
a09f38570da5fbc69ca856175cf52002e85feda8
Commit Graph
3045 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Alex Osborne a09f38570d Only write heritrix_dmesg.log when requested (#779) 
Only write heritrix_dmesg.log when requested

Heritrix.java always created heritrix_dmesg.log in $HERITRIX_HOME, even
though only the bin/heritrix script reads it, and only when it starts
Heritrix in the background. This got in the way of making
$HERITRIX_HOME read-only for Docker and systemd setups.

The startup log is now written only when the heritrix.dmesg system
property gives its path. bin/heritrix sets this property in background
mode and keeps the file in the same place as before.

Also fix a typo in bin/heritrix ($startmessage) that stopped it
removing a heritrix_dmesg.log left over from a previous start.

Set -Dheritrix.dmesg in heritrix.cmd too.
 2026-10-06 11:29:51 +09:00
Alex Osborne 0ca9e5dc2a Merge pull request #778 from internetarchive/ato/botblock-detector2 
BotBlockDetector: add additional services and enable by default
 2026-10-06 11:20:02 +09:00
Alex Osborne 838523d33a BotBlockDetector: enable in default profiles 2026-10-02 09:24:44 +09:00
dependabot[bot] 4f26ed82f2 Bump actions/setup-java from 6.0.0 to 6.0.1 
Bumps [actions/setup-java](https://github.com/actions/setup-java) from 6.0.0 to 6.0.1.
- [Release notes](https://github.com/actions/setup-java/releases)
- [Commits](https://github.com/actions/setup-java/compare/v6.0.0...v6.0.1)

---
updated-dependencies:
- dependency-name: actions/setup-java
  dependency-version: 6.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-10-02 08:51:47 +09:00
dependabot[bot] c36460a0f3 Bump the minor-and-patch group with 13 updates 
Bumps the minor-and-patch group with 13 updates:

| Package | From | To |
| --- | --- | --- |
| [org.eclipse.jetty:jetty-bom](https://github.com/jetty/jetty.project) | `12.0.38` | `12.0.39` |
| [org.eclipse.jetty.ee10:jetty-ee10-bom](https://github.com/jetty/jetty.project) | `12.0.38` | `12.0.39` |
| [org.slf4j:slf4j-bom](https://github.com/qos-ch/slf4j) | `2.0.18` | `2.0.20` |
| [org.apache.maven.plugins:maven-compiler-plugin](https://github.com/apache/maven-compiler-plugin) | `3.15.0` | `3.16.0` |
| [org.apache.maven.plugins:maven-surefire-plugin](https://github.com/apache/maven-surefire) | `3.5.6` | `3.6.0` |
| [org.codehaus.mojo:versions-maven-plugin](https://github.com/mojohaus/versions) | `2.21.0` | `2.22.0` |
| [org.webjars:webjars-locator-lite](https://github.com/webjars/webjars-locator-lite) | `1.1.4` | `1.1.5` |
| org.webjars.npm:codemirror__commands | `6.10.4` | `6.11.0` |
| org.webjars.npm:codemirror__state | `6.7.1` | `6.7.4` |
| org.webjars.npm:codemirror__view | `6.43.7` | `6.43.11` |
| org.webjars.npm:marijn__find-cluster-break | `1.0.3` | `1.0.4` |
| [com.rabbitmq:amqp-client](https://github.com/rabbitmq/rabbitmq-java-client) | `5.35.0` | `5.36.0` |
| [at.yawk.lz4:lz4-java](https://github.com/yawkat/lz4-java) | `1.11.2` | `1.12.0` |


Updates `org.eclipse.jetty:jetty-bom` from 12.0.38 to 12.0.39
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.38...jetty-12.0.39)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.38 to 12.0.39
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.38...jetty-12.0.39)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.38 to 12.0.39
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.38...jetty-12.0.39)

Updates `org.slf4j:slf4j-bom` from 2.0.18 to 2.0.20
- [Release notes](https://github.com/qos-ch/slf4j/releases)
- [Commits](https://github.com/qos-ch/slf4j/compare/v_2.0.18...v_2.0.20)

Updates `org.apache.maven.plugins:maven-compiler-plugin` from 3.15.0 to 3.16.0
- [Release notes](https://github.com/apache/maven-compiler-plugin/releases)
- [Commits](https://github.com/apache/maven-compiler-plugin/compare/maven-compiler-plugin-3.15.0...maven-compiler-plugin-3.16.0)

Updates `org.apache.maven.plugins:maven-surefire-plugin` from 3.5.6 to 3.6.0
- [Release notes](https://github.com/apache/maven-surefire/releases)
- [Commits](https://github.com/apache/maven-surefire/compare/surefire-3.5.6...surefire-3.6.0)

Updates `org.codehaus.mojo:versions-maven-plugin` from 2.21.0 to 2.22.0
- [Release notes](https://github.com/mojohaus/versions/releases)
- [Changelog](https://github.com/mojohaus/versions/blob/master/ReleaseNotes.md)
- [Commits](https://github.com/mojohaus/versions/compare/2.21.0...2.22.0)

Updates `org.webjars:webjars-locator-lite` from 1.1.4 to 1.1.5
- [Release notes](https://github.com/webjars/webjars-locator-lite/releases)
- [Commits](https://github.com/webjars/webjars-locator-lite/compare/webjars-locator-lite-1.1.4...webjars-locator-lite-1.1.5)

Updates `org.webjars.npm:codemirror__commands` from 6.10.4 to 6.11.0

Updates `org.webjars.npm:codemirror__state` from 6.7.1 to 6.7.4

Updates `org.webjars.npm:codemirror__view` from 6.43.7 to 6.43.11

Updates `org.webjars.npm:marijn__find-cluster-break` from 1.0.3 to 1.0.4

Updates `com.rabbitmq:amqp-client` from 5.35.0 to 5.36.0
- [Release notes](https://github.com/rabbitmq/rabbitmq-java-client/releases)
- [Commits](https://github.com/rabbitmq/rabbitmq-java-client/compare/v5.35.0...v5.36.0)

Updates `at.yawk.lz4:lz4-java` from 1.11.2 to 1.12.0
- [Release notes](https://github.com/yawkat/lz4-java/releases)
- [Changelog](https://github.com/yawkat/lz4-java/blob/main/CHANGES.md)
- [Commits](https://github.com/yawkat/lz4-java/compare/v1.11.2...v1.12.0)

---
updated-dependencies:
- dependency-name: org.eclipse.jetty:jetty-bom
  dependency-version: 12.0.39
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.39
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.39
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.slf4j:slf4j-bom
  dependency-version: 2.0.20
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-compiler-plugin
  dependency-version: 3.16.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-surefire-plugin
  dependency-version: 3.6.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.codehaus.mojo:versions-maven-plugin
  dependency-version: 2.22.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.webjars:webjars-locator-lite
  dependency-version: 1.1.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__commands
  dependency-version: 6.11.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__state
  dependency-version: 6.7.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__view
  dependency-version: 6.43.11
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:marijn__find-cluster-break
  dependency-version: 1.0.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.rabbitmq:amqp-client
  dependency-version: 5.36.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: at.yawk.lz4:lz4-java
  dependency-version: 1.12.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-10-02 08:51:34 +09:00
dependabot[bot] a0771568c2 Bump org.apache.groovy:groovy-bom from 5.1.1 to 6.0.0 
Bumps [org.apache.groovy:groovy-bom](https://github.com/apache/groovy) from 5.1.1 to 6.0.0.
- [Commits](https://github.com/apache/groovy/commits)

---
updated-dependencies:
- dependency-name: org.apache.groovy:groovy-bom
  dependency-version: 6.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-10-02 08:51:13 +09:00
Alex Osborne a19bd61a33 BotBlockDetector: add aws-waf, kasada, lua-anti-ddos, perimeterx 2026-10-02 08:25:52 +09:00
Alex Osborne 022f1d8050 Bump jackson-bom from 2.22.2 to 2.22.3 2026-10-02 07:44:54 +09:00
Alex Osborne f6a14ac1f6 Implement invoker commands for older browsers like Firefox ESR 
Fixes #773
 2026-10-01 19:05:53 +09:00
Shubham Padkonde 16deaaf4eb Decode JavaScript Unicode code-point escapes in extracted URLs (#772) 
Signed-off-by: Shubham Padkonde <shubhampadkonde12@gmail.com>
 2026-09-22 17:21:42 +09:00
Francesco Marinoni Moretto ed2805f6b7 Extract URIs nested inside JSON arrays (#771) 
`parse()` called `propertyStream()` on array nodes, which Jackson returns empty
for anything that is not an object, so URIs inside an array held as an object
property value were never extracted. Those array elements now go through the same
value handling as object values, which is also what lets a bare URI string inside
an array be extracted and not only an object holding one.

A known limit this does not change: `parse()` still starts from
`rootNode.properties()`, so a document whose root is itself an array yields no
links.
 2026-09-22 17:16:43 +09:00
Alex Osborne 89294d11ba Merge pull request #770 from internetarchive/ato/fix-chunked-digest 
FetchHTTP: digest chunked responses after transfer decoding
 2026-09-17 17:20:59 +09:00
Alex Osborne 6c32aaa8c1 FetchHTTP: digest chunked responses after transfer decoding 
Recalculate content digests from the entity replay stream for chunked responses so that chunk framing is not included in WARC-Payload-Digest.

Fixes #769
 2026-09-15 10:13:56 +09:00
Alex Osborne abd17fa3c7 Merge pull request #767 from internetarchive/adam/improve_amqp_url_parsing 
chore: refactor AMQPUrlReceiver parsing to handle null values coming …
 2026-09-14 14:55:37 +09:00
Adam Miller d15311698e chore: refactor AMQPUrlReceiver parsing to handle null values coming from different json libraries. 2026-09-10 14:56:48 -07:00
Alex Osborne 0d3582a136 [maven-release-plugin] prepare for next development iteration 2026-09-10 17:29:57 +09:00
Alex Osborne 48f9c37f83 [maven-release-plugin] prepare release 3.17.1 3.17.1 2026-09-10 17:29:50 +09:00
Alex Osborne 3405087a6c Update CHANGELOG.md for 3.17.1 2026-09-10 17:25:21 +09:00
Alex Osborne 2fda0a0021 Merge pull request #766 from bnfleb/issue748 
Include Heritrix status code in extended SourceTags report
 2026-09-10 08:07:05 +09:00
Leslie Bellony b0beb39518 Call tallySourceStats only if the URI is effectively crawled 2026-09-09 17:12:29 +02:00
Alex Osborne ac615ae6f5 Merge pull request #765 from bnfleb/issue752 
New Heritrix status for SSLHandshakeException
 2026-09-09 14:22:13 +09:00
Leslie Bellony 7b43a91e38 Include Heritrix status code in extended SourceTags report 2026-09-08 11:13:33 +02:00
Leslie Bellony 33984a68d2 Issue #752 new Heritrix status for SSLHandshakeException 2026-09-08 10:56:57 +02:00
dependabot[bot] f261b5beec Bump the minor-and-patch group with 2 updates 
Bumps the minor-and-patch group with 2 updates: [org.apache.groovy:groovy-bom](https://github.com/apache/groovy) and org.freemarker:freemarker.


Updates `org.apache.groovy:groovy-bom` from 5.1.0 to 5.1.1
- [Commits](https://github.com/apache/groovy/commits)

Updates `org.freemarker:freemarker` from 2.3.34 to 2.3.35

---
updated-dependencies:
- dependency-name: org.apache.groovy:groovy-bom
  dependency-version: 5.1.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.freemarker:freemarker
  dependency-version: 2.3.35
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-09-02 10:17:33 +09:00
dependabot[bot] c1c9f92b8c Bump actions/setup-java from 5.6.0 to 6.0.0 
Bumps [actions/setup-java](https://github.com/actions/setup-java) from 5.6.0 to 6.0.0.
- [Release notes](https://github.com/actions/setup-java/releases)
- [Commits](https://github.com/actions/setup-java/compare/v5.6.0...v6.0.0)

---
updated-dependencies:
- dependency-name: actions/setup-java
  dependency-version: 6.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-09-02 10:17:16 +09:00
Alex Osborne b5f32616c8 Fix javadoc warnings 2026-08-25 11:02:15 +09:00
Alex Osborne 527b202752 [maven-release-plugin] prepare for next development iteration 2026-08-25 10:17:24 +09:00
Alex Osborne 6eeb779923 [maven-release-plugin] prepare release 3.17.0 3.17.0 2026-08-25 10:17:12 +09:00
Alex Osborne e9d245837a Update CHANGELOG.md for 3.17.0 2026-08-25 10:11:55 +09:00
Alex Osborne 8b0755eadd Bump webarchive-commons from 3.0.4 to 3.0.5 2026-08-25 09:53:22 +09:00
Alex Osborne c419ad3ce5 Merge pull request #762 from internetarchive/dependabot/maven/minor-and-patch-3ea853afca 
Bump the minor-and-patch group with 14 updates
 2026-08-25 09:48:29 +09:00
dependabot[bot] 1ece1be25c Bump the minor-and-patch group with 14 updates 
Bumps the minor-and-patch group with 14 updates:

| Package | From | To |
| --- | --- | --- |
| [org.junit.jupiter:junit-jupiter](https://github.com/junit-team/junit-framework) | `6.1.2` | `6.1.3` |
| [org.apache.groovy:groovy-bom](https://github.com/apache/groovy) | `5.0.8` | `5.1.0` |
| [com.fasterxml.jackson:jackson-bom](https://github.com/FasterXML/jackson-bom) | `2.22.1` | `2.22.2` |
| [org.eclipse.jetty:jetty-bom](https://github.com/jetty/jetty.project) | `12.0.37` | `12.0.38` |
| [org.eclipse.jetty.ee10:jetty-ee10-bom](https://github.com/jetty/jetty.project) | `12.0.37` | `12.0.38` |
| [org.springframework:spring-core](https://github.com/spring-projects/spring-framework) | `7.0.8` | `7.0.9` |
| [org.springframework:spring-beans](https://github.com/spring-projects/spring-framework) | `7.0.8` | `7.0.9` |
| [org.springframework:spring-context](https://github.com/spring-projects/spring-framework) | `7.0.8` | `7.0.9` |
| [org.springframework:spring-expression](https://github.com/spring-projects/spring-framework) | `7.0.8` | `7.0.9` |
| [com.github.mwiede:jsch](https://github.com/mwiede/jsch) | `2.28.6` | `2.28.7` |
| org.webjars.npm:codemirror__commands | `6.10.2` | `6.10.4` |
| org.webjars.npm:codemirror__view | `6.43.6` | `6.43.7` |
| [com.rabbitmq:amqp-client](https://github.com/rabbitmq/rabbitmq-java-client) | `5.34.0` | `5.35.0` |
| [at.yawk.lz4:lz4-java](https://github.com/yawkat/lz4-java) | `1.11.1` | `1.11.2` |


Updates `org.junit.jupiter:junit-jupiter` from 6.1.2 to 6.1.3
- [Release notes](https://github.com/junit-team/junit-framework/releases)
- [Commits](https://github.com/junit-team/junit-framework/compare/r6.1.2...r6.1.3)

Updates `org.apache.groovy:groovy-bom` from 5.0.8 to 5.1.0
- [Commits](https://github.com/apache/groovy/commits)

Updates `com.fasterxml.jackson:jackson-bom` from 2.22.1 to 2.22.2
- [Commits](https://github.com/FasterXML/jackson-bom/compare/jackson-bom-2.22.1...jackson-bom-2.22.2)

Updates `org.eclipse.jetty:jetty-bom` from 12.0.37 to 12.0.38
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.37...jetty-12.0.38)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.37 to 12.0.38
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.37...jetty-12.0.38)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.37 to 12.0.38
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.37...jetty-12.0.38)

Updates `org.springframework:spring-core` from 7.0.8 to 7.0.9
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.8...v7.0.9)

Updates `org.springframework:spring-beans` from 7.0.8 to 7.0.9
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.8...v7.0.9)

Updates `org.springframework:spring-context` from 7.0.8 to 7.0.9
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.8...v7.0.9)

Updates `org.springframework:spring-expression` from 7.0.8 to 7.0.9
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.8...v7.0.9)

Updates `org.springframework:spring-beans` from 7.0.8 to 7.0.9
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.8...v7.0.9)

Updates `org.springframework:spring-context` from 7.0.8 to 7.0.9
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.8...v7.0.9)

Updates `org.springframework:spring-expression` from 7.0.8 to 7.0.9
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.8...v7.0.9)

Updates `com.github.mwiede:jsch` from 2.28.6 to 2.28.7
- [Release notes](https://github.com/mwiede/jsch/releases)
- [Changelog](https://github.com/mwiede/jsch/blob/master/ChangeLog)
- [Commits](https://github.com/mwiede/jsch/compare/jsch-2.28.6...jsch-2.28.7)

Updates `org.webjars.npm:codemirror__commands` from 6.10.2 to 6.10.4

Updates `org.webjars.npm:codemirror__view` from 6.43.6 to 6.43.7

Updates `com.rabbitmq:amqp-client` from 5.34.0 to 5.35.0
- [Release notes](https://github.com/rabbitmq/rabbitmq-java-client/releases)
- [Commits](https://github.com/rabbitmq/rabbitmq-java-client/compare/v5.34.0...v5.35.0)

Updates `at.yawk.lz4:lz4-java` from 1.11.1 to 1.11.2
- [Release notes](https://github.com/yawkat/lz4-java/releases)
- [Changelog](https://github.com/yawkat/lz4-java/blob/main/CHANGES.md)
- [Commits](https://github.com/yawkat/lz4-java/compare/v1.11.1...v1.11.2)

---
updated-dependencies:
- dependency-name: org.junit.jupiter:junit-jupiter
  dependency-version: 6.1.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.apache.groovy:groovy-bom
  dependency-version: 5.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: com.fasterxml.jackson:jackson-bom
  dependency-version: 2.22.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty:jetty-bom
  dependency-version: 12.0.38
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.38
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.38
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-core
  dependency-version: 7.0.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
  dependency-version: 7.0.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
  dependency-version: 7.0.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
  dependency-version: 7.0.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
  dependency-version: 7.0.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
  dependency-version: 7.0.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
  dependency-version: 7.0.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.github.mwiede:jsch
  dependency-version: 2.28.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__commands
  dependency-version: 6.10.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__view
  dependency-version: 6.43.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.rabbitmq:amqp-client
  dependency-version: 5.35.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: at.yawk.lz4:lz4-java
  dependency-version: 1.11.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-08-25 00:40:49 +00:00
Alex Osborne 38f88e5b16 Merge pull request #760 from internetarchive/ato/botblock-detector 
Add BotBlockDetector
 2026-08-21 09:31:06 +09:00
Alex Osborne cd4f910c57 BotBlockDetector: Look for cf-mitigated without server header 
Might catch unusual cases where something else is in front of Cloudflare.
 2026-08-19 10:49:55 +09:00
Alex Osborne 9973a9f2f0 BotBlockDetector: Add DataDome 2026-08-19 10:49:55 +09:00
Alex Osborne cec6cfbb64 Add BotBlockDetector 
Detects responses produced by common bot-blocking services and adds a "botblock:service" annotation. This is intended for reporting and troubleshooting but may also be useful as a signal to stop crawling or to skip writing WARC records.
 2026-08-18 16:11:35 +09:00
Alex Osborne dea9227de4 Merge pull request #759 from bnfleb/issue753 
Issue #753 verify if bdb is started before generating reports
 2026-08-05 12:09:02 +09:00
Leslie Bellony 78cc54eb1f Issue #753 verify if bdb is started before generating reports 2026-08-04 18:36:16 +02:00
Alex Osborne 96b16d0e68 Add mina-core 2.2.9 as explicit dependency 
The transitive version pulled in via ftpserver-core has vulnerabilities.
 2026-08-04 16:53:36 +09:00
Alex Osborne 26e8891e97 Merge pull request #758 from bnfleb/issue748 
Extend SourceTags report with status code
 2026-08-04 16:45:58 +09:00
Leslie Bellony 71bffa42e6 Issue #748 extend SourceTags report with status code 2026-08-03 14:25:53 +02:00
Alex Osborne 4f3980aad2 Merge pull request #757 from bnfleb/issue747 
Add Heritrix response codes in ResponseCode report
 2026-08-03 19:21:24 +09:00
Leslie Bellony 58054b962d Issue #747 add heritrix response codes in ResponseCode report 2026-08-03 11:18:11 +02:00
Alex Osborne 1a7dc27387 Merge pull request #755 from internetarchive/dependabot/github_actions/actions/setup-java-5.6.0 
Bump actions/setup-java from 5 to 5.6.0
 2026-08-03 17:03:26 +09:00
Alex Osborne 61a099c415 Merge pull request #756 from internetarchive/dependabot/maven/minor-and-patch-1f48d049a8 
Bump the minor-and-patch group with 21 updates
 2026-08-03 17:03:10 +09:00
dependabot[bot] b1b07fc477 Bump the minor-and-patch group with 21 updates 
Bumps the minor-and-patch group with 21 updates:

| Package | From | To |
| --- | --- | --- |
| [org.junit.jupiter:junit-jupiter](https://github.com/junit-team/junit-framework) | `6.1.1` | `6.1.2` |
| [org.apache.groovy:groovy-bom](https://github.com/apache/groovy) | `5.0.6` | `5.0.8` |
| [com.fasterxml.jackson:jackson-bom](https://github.com/FasterXML/jackson-bom) | `2.22.0` | `2.22.1` |
| [org.eclipse.jetty:jetty-bom](https://github.com/jetty/jetty.project) | `12.0.36` | `12.0.37` |
| [org.eclipse.jetty.ee10:jetty-ee10-bom](https://github.com/jetty/jetty.project) | `12.0.36` | `12.0.37` |
| [commons-codec:commons-codec](https://github.com/apache/commons-codec) | `1.22.0` | `1.22.1` |
| [com.github.mwiede:jsch](https://github.com/mwiede/jsch) | `2.28.3` | `2.28.6` |
| [com.github.bbottema:java-socks-proxy-server](https://github.com/bbottema/java-socks-proxy-server) | `4.1.2` | `4.2.0` |
| org.apache.pdfbox:pdfbox | `3.0.7` | `3.0.8` |
| [org.webjars:webjars-locator-lite](https://github.com/webjars/webjars-locator-lite) | `1.1.3` | `1.1.4` |
| org.webjars.npm:codemirror__autocomplete | `6.20.1` | `6.20.3` |
| org.webjars.npm:codemirror__language | `6.12.2` | `6.12.4` |
| org.webjars.npm:codemirror__state | `6.5.4` | `6.7.1` |
| org.webjars.npm:codemirror__view | `6.39.16` | `6.43.6` |
| org.webjars.npm:codemirror__lint | `6.9.5` | `6.9.7` |
| org.webjars.npm:crelt | `1.0.6` | `1.0.7` |
| org.webjars.npm:marijn__find-cluster-break | `1.0.2` | `1.0.3` |
| [org.webjars.npm:lezer__common](https://github.com/lezer-parser/common) | `1.5.1` | `1.5.2` |
| org.webjars.npm:lezer__lr | `1.4.8` | `1.4.10` |
| [com.rabbitmq:amqp-client](https://github.com/rabbitmq/rabbitmq-java-client) | `5.33.0` | `5.34.0` |
| [at.yawk.lz4:lz4-java](https://github.com/yawkat/lz4-java) | `1.11.0` | `1.11.1` |


Updates `org.junit.jupiter:junit-jupiter` from 6.1.1 to 6.1.2
- [Release notes](https://github.com/junit-team/junit-framework/releases)
- [Commits](https://github.com/junit-team/junit-framework/compare/r6.1.1...r6.1.2)

Updates `org.apache.groovy:groovy-bom` from 5.0.6 to 5.0.8
- [Commits](https://github.com/apache/groovy/commits)

Updates `com.fasterxml.jackson:jackson-bom` from 2.22.0 to 2.22.1
- [Commits](https://github.com/FasterXML/jackson-bom/compare/jackson-bom-2.22.0...jackson-bom-2.22.1)

Updates `org.eclipse.jetty:jetty-bom` from 12.0.36 to 12.0.37
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.36...jetty-12.0.37)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.36 to 12.0.37
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.36...jetty-12.0.37)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.36 to 12.0.37
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.36...jetty-12.0.37)

Updates `commons-codec:commons-codec` from 1.22.0 to 1.22.1
- [Changelog](https://github.com/apache/commons-codec/blob/master/RELEASE-NOTES.txt)
- [Commits](https://github.com/apache/commons-codec/compare/rel/commons-codec-1.22.0...rel/commons-codec-1.22.1)

Updates `com.github.mwiede:jsch` from 2.28.3 to 2.28.6
- [Release notes](https://github.com/mwiede/jsch/releases)
- [Changelog](https://github.com/mwiede/jsch/blob/master/ChangeLog)
- [Commits](https://github.com/mwiede/jsch/compare/jsch-2.28.3...jsch-2.28.6)

Updates `com.github.bbottema:java-socks-proxy-server` from 4.1.2 to 4.2.0
- [Release notes](https://github.com/bbottema/java-socks-proxy-server/releases)
- [Changelog](https://github.com/bbottema/java-socks-proxy-server/blob/master/RELEASE.txt)
- [Commits](https://github.com/bbottema/java-socks-proxy-server/compare/4.1.2...4.2.0)

Updates `org.apache.pdfbox:pdfbox` from 3.0.7 to 3.0.8

Updates `org.webjars:webjars-locator-lite` from 1.1.3 to 1.1.4
- [Release notes](https://github.com/webjars/webjars-locator-lite/releases)
- [Commits](https://github.com/webjars/webjars-locator-lite/compare/webjars-locator-lite-1.1.3...webjars-locator-lite-1.1.4)

Updates `org.webjars.npm:codemirror__autocomplete` from 6.20.1 to 6.20.3

Updates `org.webjars.npm:codemirror__language` from 6.12.2 to 6.12.4

Updates `org.webjars.npm:codemirror__state` from 6.5.4 to 6.7.1

Updates `org.webjars.npm:codemirror__view` from 6.39.16 to 6.43.6

Updates `org.webjars.npm:codemirror__lint` from 6.9.5 to 6.9.7

Updates `org.webjars.npm:crelt` from 1.0.6 to 1.0.7

Updates `org.webjars.npm:marijn__find-cluster-break` from 1.0.2 to 1.0.3

Updates `org.webjars.npm:lezer__common` from 1.5.1 to 1.5.2
- [Changelog](https://github.com/lezer-parser/common/blob/main/CHANGELOG.md)
- [Commits](https://github.com/lezer-parser/common/compare/1.5.1...1.5.2)

Updates `org.webjars.npm:lezer__lr` from 1.4.8 to 1.4.10

Updates `com.rabbitmq:amqp-client` from 5.33.0 to 5.34.0
- [Release notes](https://github.com/rabbitmq/rabbitmq-java-client/releases)
- [Commits](https://github.com/rabbitmq/rabbitmq-java-client/compare/v5.33.0...v5.34.0)

Updates `at.yawk.lz4:lz4-java` from 1.11.0 to 1.11.1
- [Release notes](https://github.com/yawkat/lz4-java/releases)
- [Changelog](https://github.com/yawkat/lz4-java/blob/main/CHANGES.md)
- [Commits](https://github.com/yawkat/lz4-java/compare/v1.11.0...v1.11.1)

---
updated-dependencies:
- dependency-name: org.junit.jupiter:junit-jupiter
  dependency-version: 6.1.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.apache.groovy:groovy-bom
  dependency-version: 5.0.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.fasterxml.jackson:jackson-bom
  dependency-version: 2.22.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty:jetty-bom
  dependency-version: 12.0.37
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.37
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.37
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: commons-codec:commons-codec
  dependency-version: 1.22.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.github.mwiede:jsch
  dependency-version: 2.28.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.github.bbottema:java-socks-proxy-server
  dependency-version: 4.2.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.apache.pdfbox:pdfbox
  dependency-version: 3.0.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars:webjars-locator-lite
  dependency-version: 1.1.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__autocomplete
  dependency-version: 6.20.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__language
  dependency-version: 6.12.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__state
  dependency-version: 6.7.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__view
  dependency-version: 6.43.6
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__lint
  dependency-version: 6.9.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:crelt
  dependency-version: 1.0.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:marijn__find-cluster-break
  dependency-version: 1.0.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:lezer__common
  dependency-version: 1.5.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:lezer__lr
  dependency-version: 1.4.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.rabbitmq:amqp-client
  dependency-version: 5.34.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: at.yawk.lz4:lz4-java
  dependency-version: 1.11.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-08-01 23:41:17 +00:00
dependabot[bot] 12006c5618 Bump actions/setup-java from 5 to 5.6.0 
Bumps [actions/setup-java](https://github.com/actions/setup-java) from 5 to 5.6.0.
- [Release notes](https://github.com/actions/setup-java/releases)
- [Commits](https://github.com/actions/setup-java/compare/v5...v5.6.0)

---
updated-dependencies:
- dependency-name: actions/setup-java
  dependency-version: 5.6.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-08-01 23:36:10 +00:00
Alex Osborne 10ebb393df Merge pull request #750 from internetarchive/browser-processor-reliability 
BrowserProcessor: improve reliability
 2026-07-15 14:07:03 +09:00
Alex Osborne 1c9b064f26 Merge pull request #751 from adam-miller/adam/fix_warc_writer_chain_null_via_check 
fix: Add null check to curi.getVia() before isNotBlank() check
 2026-07-14 14:25:48 +09:00
Adam Miller fa4184928f fix: Add null check to curi.getVia() before isNotBlank() check 2026-07-13 16:57:06 -07:00