lee/heritrix3
1
0 Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-10-08 13:21:24 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,056 Commits 43 Branches 34 Tags
master
Commit Graph
936 Commits
Author SHA1 Message Date
Alex Osborne 94a6761a17 BrowserProcessorTest: make tempDir static (#785) 
We're seeing occasional random test failures. I think it may be because we're changing the CrawlController scratchDir before each test, which means the profile of a running browser can get wiped out.
 2026-10-08 18:36:09 +09:00
Alex Osborne 8bb0a65349 Merge pull request #783 from internetarchive/toethread-always-end-disposition 
Always release the disposition lock when a ToeThread exits
 2026-10-07 12:42:19 +09:00
Alex Osborne d6747686e3 Don't let browsers cache job directory files for 10 minutes (#782) 
Restlet's Directory gives every file an Expires header 10 minutes in the
future. Job files (logs, reports, crawler-beans.cxml) change constantly, so
browsers could show stale copies. Worst case, the config editor loads a
cached crawler-beans.cxml and saving it silently reverts newer changes.

Clear the expiration date and send Cache-Control: no-cache instead, so
browsers revalidate against Last-Modified on each use.
 2026-10-07 12:42:02 +09:00
Alex Osborne be6b49fa6c [maven-release-plugin] prepare for next development iteration 2026-10-06 12:39:19 +09:00
Alex Osborne eac2c96daf [maven-release-plugin] prepare release 3.18.0 2026-10-06 12:39:12 +09:00
Alex Osborne 7939d0d7e1 Reject cross-site requests to the web UI 
Add CrossSiteRequestFilter in front of the web UI's router to protect
against cross-site request forgery. POST, PUT and DELETE requests are
rejected with 403 when the browser's Sec-Fetch-Site header says they
came from another site.

Sec-Fetch-Site is set by the browser from its own view of the page and
target URL, so unlike comparing Origin with Host it isn't affected by
reverse proxies rewriting the host or scheme. Requests without the
header, such as from curl and other API clients, are allowed.

Origins listed in the heritrix.trustedOrigins system property (comma
separated) are allowed to send cross-site requests.
 2026-10-06 12:21:40 +09:00
Alex Osborne 55bef0aa32 Merge pull request #780 from internetarchive/paged-log-viewer-search 
Crawl log search
 2026-10-06 12:09:09 +09:00
Alex Osborne 6f9a82c8fb Fix HTML escaping in templates 
Enables autoescaping in Freemarker and adds some missing escaping
where we generate HTML in Java methods.
 2026-10-06 12:04:23 +09:00
Alex Osborne a09f38570d Only write heritrix_dmesg.log when requested (#779) 
Only write heritrix_dmesg.log when requested

Heritrix.java always created heritrix_dmesg.log in $HERITRIX_HOME, even
though only the bin/heritrix script reads it, and only when it starts
Heritrix in the background. This got in the way of making
$HERITRIX_HOME read-only for Docker and systemd setups.

The startup log is now written only when the heritrix.dmesg system
property gives its path. bin/heritrix sets this property in background
mode and keeps the file in the same place as before.

Also fix a typo in bin/heritrix ($startmessage) that stopped it
removing a heritrix_dmesg.log left over from a previous start.

Set -Dheritrix.dmesg in heritrix.cmd too.
 2026-10-06 11:29:51 +09:00
Alex Osborne 0ca9e5dc2a Merge pull request #778 from internetarchive/ato/botblock-detector2 
BotBlockDetector: add additional services and enable by default
 2026-10-06 11:20:02 +09:00
Alex OsborneandClaude Opus 5.5 f5070f6203 Paged log viewer: search and filter with a small query language 
The paged log viewer gains a search box taking a search-engine style
query, passed as the q parameter:

  status:4xx host:example.com -type:image size>1MB duration>10s

- Terms are ANDed; '-' negates; double quotes allow spaces.
- 'field:value' is a case-insensitive substring match with '*'
  wildcards, 'field=value' is exact and 'field~regex' is a Java regex.
  Numeric fields take the same values with ':' or '=', and no regex;
  a comparison needs no operator, as in size>1MB.
- Bare words and the 'line' field search the whole line in any log.
  crawl.log also supports status (404, 4xx, <0, 500..599, 404,410),
  size (>1MB, 100KB..10MB; units are powers of 1024), duration (>10s,
  500..2000 in ms), depth (hops from the seed: 0, >5), hops (the hop
  path, e.g. hops:*E), url, host (includes subdomains), via and viahost
  (the URI it was discovered from, and its host), type and annot.
- Bare words and line terms are highlighted in the results.

Filtering keeps the viewer's stateless, byte-position paging and works
across checkpoint logs: each request scans forward or backward from pos
until it has enough matches or reaches the start or end, streaming a
progress bar to the browser. A scan stops after 5 minutes, offering a
link to continue from where it stopped, or as soon as the client
disconnects. Regex matching is aborted at the same deadline to guard against
catastrophic backtracking. The query is carried through all navigation links.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-02 16:54:19 +09:00
Alex OsborneandClaude Opus 5.5 b1a36159ae Paged log viewer: view a log together with its checkpoint generations 
Checkpointing renames each log in place (crawl.log becomes
crawl.log.cp00001-<timestamp>) and starts a new one, so earlier entries
could only be viewed one file at a time. The paged viewer now offers an
"include N checkpoint logs" link (all=y) that shows the log and its
rotated generations in the current launch as one virtual file, oldest
first, with each run of lines labelled with its file.

Positions in the virtual file stay valid across checkpoints, since a
rotated file keeps its place and the new log continues after it, and
they are converted when toggling so the view stays where it was.
Gzipped generations can't be read at arbitrary positions and are
left out.

LogSeries opens the files and fixes their lengths up front, so a
rotation mid-request doesn't change what is read. Paging uses a new
FilteredLineScanner over a LogSeries instead of FileUtils.pagedLines,
which can only read a single file. Results are the same for every
position the navigation links generate; a hand-edited pos in the
middle of a line now starts at that line rather than the next one.
Lines are capped at 256 KiB so a file with few or no newlines can't
exhaust the heap.

Also fixes in the crawl.log rendering: a rotated crawl.log gets the
same status highlighting, status codes unknown to Jetty (e.g. 999) and
numbers too long to be a status no longer fail the page, and the file
name, status titles and navigation links are HTML escaped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-02 16:46:36 +09:00
Alex Osborne 838523d33a BotBlockDetector: enable in default profiles 2026-10-02 09:24:44 +09:00
dependabot[bot] c36460a0f3 Bump the minor-and-patch group with 13 updates 
Bumps the minor-and-patch group with 13 updates:

| Package | From | To |
| --- | --- | --- |
| [org.eclipse.jetty:jetty-bom](https://github.com/jetty/jetty.project) | `12.0.38` | `12.0.39` |
| [org.eclipse.jetty.ee10:jetty-ee10-bom](https://github.com/jetty/jetty.project) | `12.0.38` | `12.0.39` |
| [org.slf4j:slf4j-bom](https://github.com/qos-ch/slf4j) | `2.0.18` | `2.0.20` |
| [org.apache.maven.plugins:maven-compiler-plugin](https://github.com/apache/maven-compiler-plugin) | `3.15.0` | `3.16.0` |
| [org.apache.maven.plugins:maven-surefire-plugin](https://github.com/apache/maven-surefire) | `3.5.6` | `3.6.0` |
| [org.codehaus.mojo:versions-maven-plugin](https://github.com/mojohaus/versions) | `2.21.0` | `2.22.0` |
| [org.webjars:webjars-locator-lite](https://github.com/webjars/webjars-locator-lite) | `1.1.4` | `1.1.5` |
| org.webjars.npm:codemirror__commands | `6.10.4` | `6.11.0` |
| org.webjars.npm:codemirror__state | `6.7.1` | `6.7.4` |
| org.webjars.npm:codemirror__view | `6.43.7` | `6.43.11` |
| org.webjars.npm:marijn__find-cluster-break | `1.0.3` | `1.0.4` |
| [com.rabbitmq:amqp-client](https://github.com/rabbitmq/rabbitmq-java-client) | `5.35.0` | `5.36.0` |
| [at.yawk.lz4:lz4-java](https://github.com/yawkat/lz4-java) | `1.11.2` | `1.12.0` |


Updates `org.eclipse.jetty:jetty-bom` from 12.0.38 to 12.0.39
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.38...jetty-12.0.39)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.38 to 12.0.39
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.38...jetty-12.0.39)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.38 to 12.0.39
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.38...jetty-12.0.39)

Updates `org.slf4j:slf4j-bom` from 2.0.18 to 2.0.20
- [Release notes](https://github.com/qos-ch/slf4j/releases)
- [Commits](https://github.com/qos-ch/slf4j/compare/v_2.0.18...v_2.0.20)

Updates `org.apache.maven.plugins:maven-compiler-plugin` from 3.15.0 to 3.16.0
- [Release notes](https://github.com/apache/maven-compiler-plugin/releases)
- [Commits](https://github.com/apache/maven-compiler-plugin/compare/maven-compiler-plugin-3.15.0...maven-compiler-plugin-3.16.0)

Updates `org.apache.maven.plugins:maven-surefire-plugin` from 3.5.6 to 3.6.0
- [Release notes](https://github.com/apache/maven-surefire/releases)
- [Commits](https://github.com/apache/maven-surefire/compare/surefire-3.5.6...surefire-3.6.0)

Updates `org.codehaus.mojo:versions-maven-plugin` from 2.21.0 to 2.22.0
- [Release notes](https://github.com/mojohaus/versions/releases)
- [Changelog](https://github.com/mojohaus/versions/blob/master/ReleaseNotes.md)
- [Commits](https://github.com/mojohaus/versions/compare/2.21.0...2.22.0)

Updates `org.webjars:webjars-locator-lite` from 1.1.4 to 1.1.5
- [Release notes](https://github.com/webjars/webjars-locator-lite/releases)
- [Commits](https://github.com/webjars/webjars-locator-lite/compare/webjars-locator-lite-1.1.4...webjars-locator-lite-1.1.5)

Updates `org.webjars.npm:codemirror__commands` from 6.10.4 to 6.11.0

Updates `org.webjars.npm:codemirror__state` from 6.7.1 to 6.7.4

Updates `org.webjars.npm:codemirror__view` from 6.43.7 to 6.43.11

Updates `org.webjars.npm:marijn__find-cluster-break` from 1.0.3 to 1.0.4

Updates `com.rabbitmq:amqp-client` from 5.35.0 to 5.36.0
- [Release notes](https://github.com/rabbitmq/rabbitmq-java-client/releases)
- [Commits](https://github.com/rabbitmq/rabbitmq-java-client/compare/v5.35.0...v5.36.0)

Updates `at.yawk.lz4:lz4-java` from 1.11.2 to 1.12.0
- [Release notes](https://github.com/yawkat/lz4-java/releases)
- [Changelog](https://github.com/yawkat/lz4-java/blob/main/CHANGES.md)
- [Commits](https://github.com/yawkat/lz4-java/compare/v1.11.2...v1.12.0)

---
updated-dependencies:
- dependency-name: org.eclipse.jetty:jetty-bom
  dependency-version: 12.0.39
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.39
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.39
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.slf4j:slf4j-bom
  dependency-version: 2.0.20
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-compiler-plugin
  dependency-version: 3.16.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-surefire-plugin
  dependency-version: 3.6.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.codehaus.mojo:versions-maven-plugin
  dependency-version: 2.22.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.webjars:webjars-locator-lite
  dependency-version: 1.1.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__commands
  dependency-version: 6.11.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__state
  dependency-version: 6.7.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__view
  dependency-version: 6.43.11
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:marijn__find-cluster-break
  dependency-version: 1.0.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.rabbitmq:amqp-client
  dependency-version: 5.36.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: at.yawk.lz4:lz4-java
  dependency-version: 1.12.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-10-02 08:51:34 +09:00
Alex Osborne f6a14ac1f6 Implement invoker commands for older browsers like Firefox ESR 
Fixes #773
 2026-10-01 19:05:53 +09:00
Alex Osborne 0d3582a136 [maven-release-plugin] prepare for next development iteration 2026-09-10 17:29:57 +09:00
Alex Osborne 48f9c37f83 [maven-release-plugin] prepare release 3.17.1 2026-09-10 17:29:50 +09:00
Leslie Bellony b0beb39518 Call tallySourceStats only if the URI is effectively crawled 2026-09-09 17:12:29 +02:00
Leslie Bellony 7b43a91e38 Include Heritrix status code in extended SourceTags report 2026-09-08 11:13:33 +02:00
dependabot[bot] f261b5beec Bump the minor-and-patch group with 2 updates 
Bumps the minor-and-patch group with 2 updates: [org.apache.groovy:groovy-bom](https://github.com/apache/groovy) and org.freemarker:freemarker.


Updates `org.apache.groovy:groovy-bom` from 5.1.0 to 5.1.1
- [Commits](https://github.com/apache/groovy/commits)

Updates `org.freemarker:freemarker` from 2.3.34 to 2.3.35

---
updated-dependencies:
- dependency-name: org.apache.groovy:groovy-bom
  dependency-version: 5.1.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.freemarker:freemarker
  dependency-version: 2.3.35
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-09-02 10:17:33 +09:00
Alex Osborne b5f32616c8 Fix javadoc warnings 2026-08-25 11:02:15 +09:00
Alex Osborne 527b202752 [maven-release-plugin] prepare for next development iteration 2026-08-25 10:17:24 +09:00
Alex Osborne 6eeb779923 [maven-release-plugin] prepare release 3.17.0 2026-08-25 10:17:12 +09:00
dependabot[bot] 1ece1be25c Bump the minor-and-patch group with 14 updates 
Bumps the minor-and-patch group with 14 updates:

| Package | From | To |
| --- | --- | --- |
| [org.junit.jupiter:junit-jupiter](https://github.com/junit-team/junit-framework) | `6.1.2` | `6.1.3` |
| [org.apache.groovy:groovy-bom](https://github.com/apache/groovy) | `5.0.8` | `5.1.0` |
| [com.fasterxml.jackson:jackson-bom](https://github.com/FasterXML/jackson-bom) | `2.22.1` | `2.22.2` |
| [org.eclipse.jetty:jetty-bom](https://github.com/jetty/jetty.project) | `12.0.37` | `12.0.38` |
| [org.eclipse.jetty.ee10:jetty-ee10-bom](https://github.com/jetty/jetty.project) | `12.0.37` | `12.0.38` |
| [org.springframework:spring-core](https://github.com/spring-projects/spring-framework) | `7.0.8` | `7.0.9` |
| [org.springframework:spring-beans](https://github.com/spring-projects/spring-framework) | `7.0.8` | `7.0.9` |
| [org.springframework:spring-context](https://github.com/spring-projects/spring-framework) | `7.0.8` | `7.0.9` |
| [org.springframework:spring-expression](https://github.com/spring-projects/spring-framework) | `7.0.8` | `7.0.9` |
| [com.github.mwiede:jsch](https://github.com/mwiede/jsch) | `2.28.6` | `2.28.7` |
| org.webjars.npm:codemirror__commands | `6.10.2` | `6.10.4` |
| org.webjars.npm:codemirror__view | `6.43.6` | `6.43.7` |
| [com.rabbitmq:amqp-client](https://github.com/rabbitmq/rabbitmq-java-client) | `5.34.0` | `5.35.0` |
| [at.yawk.lz4:lz4-java](https://github.com/yawkat/lz4-java) | `1.11.1` | `1.11.2` |


Updates `org.junit.jupiter:junit-jupiter` from 6.1.2 to 6.1.3
- [Release notes](https://github.com/junit-team/junit-framework/releases)
- [Commits](https://github.com/junit-team/junit-framework/compare/r6.1.2...r6.1.3)

Updates `org.apache.groovy:groovy-bom` from 5.0.8 to 5.1.0
- [Commits](https://github.com/apache/groovy/commits)

Updates `com.fasterxml.jackson:jackson-bom` from 2.22.1 to 2.22.2
- [Commits](https://github.com/FasterXML/jackson-bom/compare/jackson-bom-2.22.1...jackson-bom-2.22.2)

Updates `org.eclipse.jetty:jetty-bom` from 12.0.37 to 12.0.38
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.37...jetty-12.0.38)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.37 to 12.0.38
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.37...jetty-12.0.38)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.37 to 12.0.38
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.37...jetty-12.0.38)

Updates `org.springframework:spring-core` from 7.0.8 to 7.0.9
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.8...v7.0.9)

Updates `org.springframework:spring-beans` from 7.0.8 to 7.0.9
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.8...v7.0.9)

Updates `org.springframework:spring-context` from 7.0.8 to 7.0.9
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.8...v7.0.9)

Updates `org.springframework:spring-expression` from 7.0.8 to 7.0.9
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.8...v7.0.9)

Updates `org.springframework:spring-beans` from 7.0.8 to 7.0.9
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.8...v7.0.9)

Updates `org.springframework:spring-context` from 7.0.8 to 7.0.9
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.8...v7.0.9)

Updates `org.springframework:spring-expression` from 7.0.8 to 7.0.9
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.8...v7.0.9)

Updates `com.github.mwiede:jsch` from 2.28.6 to 2.28.7
- [Release notes](https://github.com/mwiede/jsch/releases)
- [Changelog](https://github.com/mwiede/jsch/blob/master/ChangeLog)
- [Commits](https://github.com/mwiede/jsch/compare/jsch-2.28.6...jsch-2.28.7)

Updates `org.webjars.npm:codemirror__commands` from 6.10.2 to 6.10.4

Updates `org.webjars.npm:codemirror__view` from 6.43.6 to 6.43.7

Updates `com.rabbitmq:amqp-client` from 5.34.0 to 5.35.0
- [Release notes](https://github.com/rabbitmq/rabbitmq-java-client/releases)
- [Commits](https://github.com/rabbitmq/rabbitmq-java-client/compare/v5.34.0...v5.35.0)

Updates `at.yawk.lz4:lz4-java` from 1.11.1 to 1.11.2
- [Release notes](https://github.com/yawkat/lz4-java/releases)
- [Changelog](https://github.com/yawkat/lz4-java/blob/main/CHANGES.md)
- [Commits](https://github.com/yawkat/lz4-java/compare/v1.11.1...v1.11.2)

---
updated-dependencies:
- dependency-name: org.junit.jupiter:junit-jupiter
  dependency-version: 6.1.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.apache.groovy:groovy-bom
  dependency-version: 5.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: com.fasterxml.jackson:jackson-bom
  dependency-version: 2.22.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty:jetty-bom
  dependency-version: 12.0.38
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.38
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.38
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-core
  dependency-version: 7.0.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
  dependency-version: 7.0.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
  dependency-version: 7.0.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
  dependency-version: 7.0.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
  dependency-version: 7.0.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
  dependency-version: 7.0.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
  dependency-version: 7.0.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.github.mwiede:jsch
  dependency-version: 2.28.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__commands
  dependency-version: 6.10.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__view
  dependency-version: 6.43.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.rabbitmq:amqp-client
  dependency-version: 5.35.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: at.yawk.lz4:lz4-java
  dependency-version: 1.11.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-08-25 00:40:49 +00:00
Leslie Bellony 78cc54eb1f Issue #753 verify if bdb is started before generating reports 2026-08-04 18:36:16 +02:00
Leslie Bellony 71bffa42e6 Issue #748 extend SourceTags report with status code 2026-08-03 14:25:53 +02:00
Leslie Bellony 58054b962d Issue #747 add heritrix response codes in ResponseCode report 2026-08-03 11:18:11 +02:00
dependabot[bot] b1b07fc477 Bump the minor-and-patch group with 21 updates 
Bumps the minor-and-patch group with 21 updates:

| Package | From | To |
| --- | --- | --- |
| [org.junit.jupiter:junit-jupiter](https://github.com/junit-team/junit-framework) | `6.1.1` | `6.1.2` |
| [org.apache.groovy:groovy-bom](https://github.com/apache/groovy) | `5.0.6` | `5.0.8` |
| [com.fasterxml.jackson:jackson-bom](https://github.com/FasterXML/jackson-bom) | `2.22.0` | `2.22.1` |
| [org.eclipse.jetty:jetty-bom](https://github.com/jetty/jetty.project) | `12.0.36` | `12.0.37` |
| [org.eclipse.jetty.ee10:jetty-ee10-bom](https://github.com/jetty/jetty.project) | `12.0.36` | `12.0.37` |
| [commons-codec:commons-codec](https://github.com/apache/commons-codec) | `1.22.0` | `1.22.1` |
| [com.github.mwiede:jsch](https://github.com/mwiede/jsch) | `2.28.3` | `2.28.6` |
| [com.github.bbottema:java-socks-proxy-server](https://github.com/bbottema/java-socks-proxy-server) | `4.1.2` | `4.2.0` |
| org.apache.pdfbox:pdfbox | `3.0.7` | `3.0.8` |
| [org.webjars:webjars-locator-lite](https://github.com/webjars/webjars-locator-lite) | `1.1.3` | `1.1.4` |
| org.webjars.npm:codemirror__autocomplete | `6.20.1` | `6.20.3` |
| org.webjars.npm:codemirror__language | `6.12.2` | `6.12.4` |
| org.webjars.npm:codemirror__state | `6.5.4` | `6.7.1` |
| org.webjars.npm:codemirror__view | `6.39.16` | `6.43.6` |
| org.webjars.npm:codemirror__lint | `6.9.5` | `6.9.7` |
| org.webjars.npm:crelt | `1.0.6` | `1.0.7` |
| org.webjars.npm:marijn__find-cluster-break | `1.0.2` | `1.0.3` |
| [org.webjars.npm:lezer__common](https://github.com/lezer-parser/common) | `1.5.1` | `1.5.2` |
| org.webjars.npm:lezer__lr | `1.4.8` | `1.4.10` |
| [com.rabbitmq:amqp-client](https://github.com/rabbitmq/rabbitmq-java-client) | `5.33.0` | `5.34.0` |
| [at.yawk.lz4:lz4-java](https://github.com/yawkat/lz4-java) | `1.11.0` | `1.11.1` |


Updates `org.junit.jupiter:junit-jupiter` from 6.1.1 to 6.1.2
- [Release notes](https://github.com/junit-team/junit-framework/releases)
- [Commits](https://github.com/junit-team/junit-framework/compare/r6.1.1...r6.1.2)

Updates `org.apache.groovy:groovy-bom` from 5.0.6 to 5.0.8
- [Commits](https://github.com/apache/groovy/commits)

Updates `com.fasterxml.jackson:jackson-bom` from 2.22.0 to 2.22.1
- [Commits](https://github.com/FasterXML/jackson-bom/compare/jackson-bom-2.22.0...jackson-bom-2.22.1)

Updates `org.eclipse.jetty:jetty-bom` from 12.0.36 to 12.0.37
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.36...jetty-12.0.37)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.36 to 12.0.37
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.36...jetty-12.0.37)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.36 to 12.0.37
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.36...jetty-12.0.37)

Updates `commons-codec:commons-codec` from 1.22.0 to 1.22.1
- [Changelog](https://github.com/apache/commons-codec/blob/master/RELEASE-NOTES.txt)
- [Commits](https://github.com/apache/commons-codec/compare/rel/commons-codec-1.22.0...rel/commons-codec-1.22.1)

Updates `com.github.mwiede:jsch` from 2.28.3 to 2.28.6
- [Release notes](https://github.com/mwiede/jsch/releases)
- [Changelog](https://github.com/mwiede/jsch/blob/master/ChangeLog)
- [Commits](https://github.com/mwiede/jsch/compare/jsch-2.28.3...jsch-2.28.6)

Updates `com.github.bbottema:java-socks-proxy-server` from 4.1.2 to 4.2.0
- [Release notes](https://github.com/bbottema/java-socks-proxy-server/releases)
- [Changelog](https://github.com/bbottema/java-socks-proxy-server/blob/master/RELEASE.txt)
- [Commits](https://github.com/bbottema/java-socks-proxy-server/compare/4.1.2...4.2.0)

Updates `org.apache.pdfbox:pdfbox` from 3.0.7 to 3.0.8

Updates `org.webjars:webjars-locator-lite` from 1.1.3 to 1.1.4
- [Release notes](https://github.com/webjars/webjars-locator-lite/releases)
- [Commits](https://github.com/webjars/webjars-locator-lite/compare/webjars-locator-lite-1.1.3...webjars-locator-lite-1.1.4)

Updates `org.webjars.npm:codemirror__autocomplete` from 6.20.1 to 6.20.3

Updates `org.webjars.npm:codemirror__language` from 6.12.2 to 6.12.4

Updates `org.webjars.npm:codemirror__state` from 6.5.4 to 6.7.1

Updates `org.webjars.npm:codemirror__view` from 6.39.16 to 6.43.6

Updates `org.webjars.npm:codemirror__lint` from 6.9.5 to 6.9.7

Updates `org.webjars.npm:crelt` from 1.0.6 to 1.0.7

Updates `org.webjars.npm:marijn__find-cluster-break` from 1.0.2 to 1.0.3

Updates `org.webjars.npm:lezer__common` from 1.5.1 to 1.5.2
- [Changelog](https://github.com/lezer-parser/common/blob/main/CHANGELOG.md)
- [Commits](https://github.com/lezer-parser/common/compare/1.5.1...1.5.2)

Updates `org.webjars.npm:lezer__lr` from 1.4.8 to 1.4.10

Updates `com.rabbitmq:amqp-client` from 5.33.0 to 5.34.0
- [Release notes](https://github.com/rabbitmq/rabbitmq-java-client/releases)
- [Commits](https://github.com/rabbitmq/rabbitmq-java-client/compare/v5.33.0...v5.34.0)

Updates `at.yawk.lz4:lz4-java` from 1.11.0 to 1.11.1
- [Release notes](https://github.com/yawkat/lz4-java/releases)
- [Changelog](https://github.com/yawkat/lz4-java/blob/main/CHANGES.md)
- [Commits](https://github.com/yawkat/lz4-java/compare/v1.11.0...v1.11.1)

---
updated-dependencies:
- dependency-name: org.junit.jupiter:junit-jupiter
  dependency-version: 6.1.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.apache.groovy:groovy-bom
  dependency-version: 5.0.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.fasterxml.jackson:jackson-bom
  dependency-version: 2.22.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty:jetty-bom
  dependency-version: 12.0.37
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.37
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.37
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: commons-codec:commons-codec
  dependency-version: 1.22.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.github.mwiede:jsch
  dependency-version: 2.28.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.github.bbottema:java-socks-proxy-server
  dependency-version: 4.2.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.apache.pdfbox:pdfbox
  dependency-version: 3.0.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars:webjars-locator-lite
  dependency-version: 1.1.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__autocomplete
  dependency-version: 6.20.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__language
  dependency-version: 6.12.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__state
  dependency-version: 6.7.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__view
  dependency-version: 6.43.6
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__lint
  dependency-version: 6.9.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:crelt
  dependency-version: 1.0.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:marijn__find-cluster-break
  dependency-version: 1.0.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:lezer__common
  dependency-version: 1.5.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:lezer__lr
  dependency-version: 1.4.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.rabbitmq:amqp-client
  dependency-version: 5.34.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: at.yawk.lz4:lz4-java
  dependency-version: 1.11.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-08-01 23:41:17 +00:00
Alex Osborne 97d8307420 BrowserProcessor: restart the browser automatically if it crashes 2026-07-10 16:12:52 +09:00
Alex Osborne d7676b1b67 BrowserProcessor: log failures continuing intercepted requests 2026-07-10 16:12:47 +09:00
Alex Osborne 1b107f06c1 BrowserProcessor: guard stop() against a partially failed start() 
If start() throws after the proxy is started but before the browser
launches, stop() could NPE on the null fields.
 2026-07-10 15:59:47 +09:00
Alex Osborne d03a43266a BrowserProcessor: catch tab close failures 
If closing the tab in the finally block throws, just log it rather than propagating to the toe thread.
 2026-07-10 15:59:46 +09:00
Alex Osborne e276d83f4b BrowserProcessor: avoid NPE when a WebDriverException has no message 2026-07-10 15:48:42 +09:00
Alex Osborne 789603c1b7 [maven-release-plugin] prepare for next development iteration 2026-07-03 10:20:27 +09:00
Alex Osborne 7af0bf56ea [maven-release-plugin] prepare release 3.16.0 2026-07-03 10:20:16 +09:00
Alex Osborne 6c7a54b5c5 [maven-release-plugin] prepare for next development iteration 2026-07-03 08:49:28 +09:00
Alex Osborne 9fdd5543af [maven-release-plugin] prepare release 3.16.0 2026-07-03 08:49:14 +09:00
Alex Osborne d3da2f9afe BrowserProcessor: handle result.isFailed() in SubresourceRecorder.onComplete() 2026-06-18 23:30:48 +09:00
Alex Osborne 2c9f167ae3 BrowserProcessor: handle recording truncation with length and timeout limits 2026-06-18 23:25:20 +09:00
Alex Osborne 4cb48ec5dc BrowserProcessor: fix content digest and recording limits for subresources 2026-06-18 23:18:23 +09:00
Alex Osborne ff6f98d38d BrowserProcessor: set User-Agent header 
Ensures requests made by the browser use the user-agent string configured for the page in the job config.
 2026-06-18 23:00:21 +09:00
Alex Osborne e156997dbd MitmProxy: Fix request recording 
This fixes a bug where POST requests would get recorded with a doubled header and a bunch of nulls at the end.

It seems that the onRequestContent() listener doesn't pass us the right buffer when proxying. The buffer seems to contain client-to-server data and the buffer's limit isn't set. So instead this records the request content by overriding newProxyToServerRequestContent.
 2026-06-17 16:56:24 +09:00
Alex Osborne 709ac00dd2 Add PaginationBehavior: repeatedly clicks next-page and extracts links 2026-06-11 13:34:55 +09:00
Alex Osborne 54b95f5e97 groovy profile: remove stray } 2026-06-10 18:23:12 +09:00
Alex Osborne bd34a7a0a9 Log the exception when creating new jobs fails 2026-06-10 17:43:57 +09:00
dependabot[bot] 3406c14632 Bump the minor-and-patch group with 11 updates 
Bumps the minor-and-patch group with 11 updates:

| Package | From | To |
| --- | --- | --- |
| [org.junit.jupiter:junit-jupiter](https://github.com/junit-team/junit-framework) | `6.0.3` | `6.1.0` |
| [org.apache.groovy:groovy-bom](https://github.com/apache/groovy) | `5.0.5` | `5.0.6` |
| [com.fasterxml.jackson:jackson-bom](https://github.com/FasterXML/jackson-bom) | `2.21.3` | `2.22.0` |
| [org.eclipse.jetty:jetty-bom](https://github.com/jetty/jetty.project) | `12.0.34` | `12.0.36` |
| [org.eclipse.jetty.ee10:jetty-ee10-bom](https://github.com/jetty/jetty.project) | `12.0.34` | `12.0.36` |
| [org.slf4j:slf4j-bom](https://github.com/qos-ch/slf4j) | `2.0.17` | `2.0.18` |
| [org.apache.maven.plugins:maven-surefire-plugin](https://github.com/apache/maven-surefire) | `3.5.5` | `3.5.6` |
| [dnsjava:dnsjava](https://github.com/dnsjava/dnsjava) | `3.6.4` | `3.6.5` |
| org.glassfish.jaxb:jaxb-runtime | `4.0.7` | `4.0.9` |
| [com.rabbitmq:amqp-client](https://github.com/rabbitmq/rabbitmq-java-client) | `5.30.0` | `5.31.0` |
| org.apache.kafka:kafka-clients | `4.2.0` | `4.3.0` |


Updates `org.junit.jupiter:junit-jupiter` from 6.0.3 to 6.1.0
- [Release notes](https://github.com/junit-team/junit-framework/releases)
- [Commits](https://github.com/junit-team/junit-framework/compare/r6.0.3...r6.1.0)

Updates `org.apache.groovy:groovy-bom` from 5.0.5 to 5.0.6
- [Commits](https://github.com/apache/groovy/commits)

Updates `com.fasterxml.jackson:jackson-bom` from 2.21.3 to 2.22.0
- [Commits](https://github.com/FasterXML/jackson-bom/compare/jackson-bom-2.21.3...jackson-bom-2.22.0)

Updates `org.eclipse.jetty:jetty-bom` from 12.0.34 to 12.0.36
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.34...jetty-12.0.36)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.34 to 12.0.36
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.34...jetty-12.0.36)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.34 to 12.0.36
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.34...jetty-12.0.36)

Updates `org.slf4j:slf4j-bom` from 2.0.17 to 2.0.18
- [Release notes](https://github.com/qos-ch/slf4j/releases)
- [Commits](https://github.com/qos-ch/slf4j/compare/v_2.0.17...v_2.0.18)

Updates `org.apache.maven.plugins:maven-surefire-plugin` from 3.5.5 to 3.5.6
- [Release notes](https://github.com/apache/maven-surefire/releases)
- [Commits](https://github.com/apache/maven-surefire/compare/surefire-3.5.5...surefire-3.5.6)

Updates `dnsjava:dnsjava` from 3.6.4 to 3.6.5
- [Release notes](https://github.com/dnsjava/dnsjava/releases)
- [Changelog](https://github.com/dnsjava/dnsjava/blob/master/Changelog)
- [Commits](https://github.com/dnsjava/dnsjava/commits)

Updates `org.glassfish.jaxb:jaxb-runtime` from 4.0.7 to 4.0.9

Updates `com.rabbitmq:amqp-client` from 5.30.0 to 5.31.0
- [Release notes](https://github.com/rabbitmq/rabbitmq-java-client/releases)
- [Commits](https://github.com/rabbitmq/rabbitmq-java-client/compare/v5.30.0...v5.31.0)

Updates `org.apache.kafka:kafka-clients` from 4.2.0 to 4.3.0

---
updated-dependencies:
- dependency-name: org.junit.jupiter:junit-jupiter
  dependency-version: 6.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.apache.groovy:groovy-bom
  dependency-version: 5.0.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.fasterxml.jackson:jackson-bom
  dependency-version: 2.22.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty:jetty-bom
  dependency-version: 12.0.36
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.36
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.36
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.slf4j:slf4j-bom
  dependency-version: 2.0.18
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-surefire-plugin
  dependency-version: 3.5.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: dnsjava:dnsjava
  dependency-version: 3.6.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.glassfish.jaxb:jaxb-runtime
  dependency-version: 4.0.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.rabbitmq:amqp-client
  dependency-version: 5.31.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.apache.kafka:kafka-clients
  dependency-version: 4.3.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-06-03 16:56:20 +00:00
Alex Osborne e0b46789bc [maven-release-plugin] prepare for next development iteration 2026-05-26 11:51:31 +09:00
Alex Osborne bc28f4d1ae [maven-release-plugin] prepare release 3.15.0 2026-05-26 11:51:25 +09:00
Alex Osborne d3defaacd2 Merge pull request #732 from internetarchive/remove-foundation-js 
Remove obsolete JavaScript libraries
 2026-05-12 16:44:16 +09:00
Alex Osborne 56a1fc2198 Remove Internet Explorer compatibility shims 
There's no need for these anymore, IE has been sunset.
 2026-05-04 08:45:41 +09:00