lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-07-11 01:57:19 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,995 Commits 40 Branches 31 Tags
browser-processor-reliability
Commit Graph

2995 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Alex Osborne 58258eb640 BiDiJson: map results to null for commands that return Void 
Fixes java.lang.UnsupportedOperationException: Unsupported type: class java.lang.Void
 2026-07-10 16:12:54 +09:00
Alex Osborne 97d8307420 BrowserProcessor: restart the browser automatically if it crashes 2026-07-10 16:12:52 +09:00
Alex Osborne d7676b1b67 BrowserProcessor: log failures continuing intercepted requests 2026-07-10 16:12:47 +09:00
Alex Osborne 1b107f06c1 BrowserProcessor: guard stop() against a partially failed start() 
If start() throws after the proxy is started but before the browser
launches, stop() could NPE on the null fields.
 2026-07-10 15:59:47 +09:00
Alex Osborne d03a43266a BrowserProcessor: catch tab close failures 
If closing the tab in the finally block throws, just log it rather than propagating to the toe thread.
 2026-07-10 15:59:46 +09:00
Alex Osborne e276d83f4b BrowserProcessor: avoid NPE when a WebDriverException has no message 2026-07-10 15:48:42 +09:00
Alex Osborne 1392da4fa6 LocalWebDriverBiDi: make close() reliably terminate the browser 2026-07-10 15:43:50 +09:00
Alex Osborne 27d0f05010 LocalWebDriverBiDi: time out and clean up if the browser fails to start 2026-07-10 15:41:28 +09:00
Alex Osborne 99cd6d942d LocalWebDriverBiDi: fail pending commands when the connection is lost 2026-07-10 15:39:34 +09:00
Alex Osborne 4b396426e5 LocalWebDriverBiDi: serialize WebSocket sends 
The JDK WebSocket only permits one outstanding sendText() at a time.
With many toe threads issuing BiDi commands concurrently sends can fail with IllegalStateException, so chain them.
 2026-07-10 15:36:00 +09:00
Alex Osborne b68042158b RELEASING: Move docker image build after GitHub release 
The build fetches the release artifact from GitHub so needs to be done after.
 2026-07-03 11:04:51 +09:00
Alex Osborne d42e1405a0 RELEASING.md: add steps to upload distribution artifacts to Github release 2026-07-03 10:50:34 +09:00
Alex Osborne 79ec034562 Update distribution package links to Github for 3.16.0 2026-07-03 10:42:24 +09:00
Alex Osborne 789603c1b7 [maven-release-plugin] prepare for next development iteration 2026-07-03 10:20:27 +09:00
Alex Osborne 7af0bf56ea [maven-release-plugin] prepare release 3.16.0 3.16.0 2026-07-03 10:20:16 +09:00
Alex Osborne df1fdb3593 Exclude dist artifacts from Maven Central release bundle 
Maven Central now has an 80MB total size limit on releases so we can't include them.
 2026-07-03 10:10:13 +09:00
Alex Osborne dafcc2ef14 RELEASING.md: set version dynamically based on last tag 2026-07-03 10:06:39 +09:00
Alex Osborne 6c7a54b5c5 [maven-release-plugin] prepare for next development iteration 2026-07-03 08:49:28 +09:00
Alex Osborne 9fdd5543af [maven-release-plugin] prepare release 3.16.0 2026-07-03 08:49:14 +09:00
Alex Osborne 7846215a0e Update CHANGELOG for 3.16.0 release 2026-07-03 08:44:58 +09:00
Alex Osborne 0b451ed437 Update CHANGELOG for 3.16.0 release 2026-07-03 08:35:21 +09:00
Alex Osborne 5ada2a784a Merge pull request #744 from internetarchive/dependabot/github_actions/actions/cache-6 
Bump actions/cache from 5 to 6
 2026-07-02 08:53:40 +09:00
Alex Osborne 987ba1ac46 Merge pull request #745 from internetarchive/dependabot/github_actions/actions/checkout-7 
Bump actions/checkout from 6 to 7
 2026-07-02 08:53:22 +09:00
Alex Osborne 9ee5b4285b Merge pull request #746 from internetarchive/dependabot/maven/minor-and-patch-607f52c7b4 
Bump the minor-and-patch group with 3 updates
 2026-07-02 08:53:07 +09:00
dependabot[bot] 52de3d940c Bump the minor-and-patch group with 3 updates 
Bumps the minor-and-patch group with 3 updates: [org.junit.jupiter:junit-jupiter](https://github.com/junit-team/junit-framework), [com.rabbitmq:amqp-client](https://github.com/rabbitmq/rabbitmq-java-client) and org.apache.kafka:kafka-clients.


Updates `org.junit.jupiter:junit-jupiter` from 6.1.0 to 6.1.1
- [Release notes](https://github.com/junit-team/junit-framework/releases)
- [Commits](https://github.com/junit-team/junit-framework/compare/r6.1.0...r6.1.1)

Updates `com.rabbitmq:amqp-client` from 5.31.0 to 5.33.0
- [Release notes](https://github.com/rabbitmq/rabbitmq-java-client/releases)
- [Commits](https://github.com/rabbitmq/rabbitmq-java-client/compare/v5.31.0...v5.33.0)

Updates `org.apache.kafka:kafka-clients` from 4.3.0 to 4.3.1

---
updated-dependencies:
- dependency-name: org.junit.jupiter:junit-jupiter
  dependency-version: 6.1.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.rabbitmq:amqp-client
  dependency-version: 5.33.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.apache.kafka:kafka-clients
  dependency-version: 4.3.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-07-01 23:40:25 +00:00
dependabot[bot] a0eb9e5eac Bump actions/checkout from 6 to 7 
Bumps [actions/checkout](https://github.com/actions/checkout) from 6 to 7.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v6...v7)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-07-01 23:37:47 +00:00
dependabot[bot] d63e5c710d Bump actions/cache from 5 to 6 
Bumps [actions/cache](https://github.com/actions/cache) from 5 to 6.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v5...v6)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-07-01 23:37:44 +00:00
Alex Osborne 6344f8fdcf Merge pull request #742 from internetarchive/ato/BrowserProcessor-fixes 
BrowserProcessor fixes
 2026-06-19 14:29:34 +09:00
Alex Osborne 1f145f7926 CI: try running browser tests 2026-06-18 23:50:34 +09:00
Alex Osborne d3da2f9afe BrowserProcessor: handle result.isFailed() in SubresourceRecorder.onComplete() 2026-06-18 23:30:48 +09:00
Alex Osborne 2c9f167ae3 BrowserProcessor: handle recording truncation with length and timeout limits 2026-06-18 23:25:20 +09:00
Alex Osborne 4cb48ec5dc BrowserProcessor: fix content digest and recording limits for subresources 2026-06-18 23:18:23 +09:00
Alex Osborne ff6f98d38d BrowserProcessor: set User-Agent header 
Ensures requests made by the browser use the user-agent string configured for the page in the job config.
 2026-06-18 23:00:21 +09:00
Alex Osborne 3b468c05c9 Merge pull request #737 from internetarchive/ato/fix-mitmproxy-request-content 
BrowserProcessor/MitmProxy: Fix request recording
 2026-06-18 09:36:10 +09:00
dependabot[bot] b1c1f1f207 Bump the minor-and-patch group with 6 updates 
Bumps the minor-and-patch group with 6 updates:

| Package | From | To |
| --- | --- | --- |
| [org.sonatype.central:central-publishing-maven-plugin](https://github.com/sonatype/central-publishing-maven-plugin) | `0.10.0` | `0.11.0` |
| [org.springframework:spring-core](https://github.com/spring-projects/spring-framework) | `7.0.7` | `7.0.8` |
| [org.springframework:spring-beans](https://github.com/spring-projects/spring-framework) | `7.0.7` | `7.0.8` |
| [org.springframework:spring-context](https://github.com/spring-projects/spring-framework) | `7.0.7` | `7.0.8` |
| [org.springframework:spring-expression](https://github.com/spring-projects/spring-framework) | `7.0.7` | `7.0.8` |
| [com.github.mwiede:jsch](https://github.com/mwiede/jsch) | `2.28.2` | `2.28.3` |


Updates `org.sonatype.central:central-publishing-maven-plugin` from 0.10.0 to 0.11.0
- [Commits](https://github.com/sonatype/central-publishing-maven-plugin/commits)

Updates `org.springframework:spring-core` from 7.0.7 to 7.0.8
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.7...v7.0.8)

Updates `org.springframework:spring-beans` from 7.0.7 to 7.0.8
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.7...v7.0.8)

Updates `org.springframework:spring-context` from 7.0.7 to 7.0.8
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.7...v7.0.8)

Updates `org.springframework:spring-expression` from 7.0.7 to 7.0.8
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.7...v7.0.8)

Updates `org.springframework:spring-beans` from 7.0.7 to 7.0.8
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.7...v7.0.8)

Updates `org.springframework:spring-context` from 7.0.7 to 7.0.8
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.7...v7.0.8)

Updates `org.springframework:spring-expression` from 7.0.7 to 7.0.8
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.7...v7.0.8)

Updates `com.github.mwiede:jsch` from 2.28.2 to 2.28.3
- [Release notes](https://github.com/mwiede/jsch/releases)
- [Changelog](https://github.com/mwiede/jsch/blob/master/ChangeLog)
- [Commits](https://github.com/mwiede/jsch/compare/jsch-2.28.2...jsch-2.28.3)

---
updated-dependencies:
- dependency-name: org.sonatype.central:central-publishing-maven-plugin
  dependency-version: 0.11.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-core
  dependency-version: 7.0.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
  dependency-version: 7.0.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
  dependency-version: 7.0.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
  dependency-version: 7.0.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
  dependency-version: 7.0.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
  dependency-version: 7.0.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
  dependency-version: 7.0.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.github.mwiede:jsch
  dependency-version: 2.28.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-06-17 17:18:10 +09:00
dependabot[bot] 5744aa7cc3 Bump actions/setup-java from 2 to 5 
Bumps [actions/setup-java](https://github.com/actions/setup-java) from 2 to 5.
- [Release notes](https://github.com/actions/setup-java/releases)
- [Commits](https://github.com/actions/setup-java/compare/v2...v5)

---
updated-dependencies:
- dependency-name: actions/setup-java
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-06-17 17:17:53 +09:00
dependabot[bot] 2cbc5ec018 Bump actions/cache from 4 to 5 
Bumps [actions/cache](https://github.com/actions/cache) from 4 to 5.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-06-17 17:17:37 +09:00
dependabot[bot] e7fd9e0c3f Bump actions/checkout from 2 to 6 
Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 6.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v2...v6)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-06-17 17:17:19 +09:00
Alex Osborne a6adb1f4f9 dependabot: update github-actions monthly 2026-06-17 17:01:41 +09:00
Alex Osborne e156997dbd MitmProxy: Fix request recording 
This fixes a bug where POST requests would get recorded with a doubled header and a bunch of nulls at the end.

It seems that the onRequestContent() listener doesn't pass us the right buffer when proxying. The buffer seems to contain client-to-server data and the buffer's limit isn't set. So instead this records the request content by overriding newProxyToServerRequestContent.
 2026-06-17 16:56:24 +09:00
Alex Osborne a904253f3c Merge pull request #736 from internetarchive/pagination-behavior 
PaginationBehavior: repeatedly clicks next-page and extracts links
 2026-06-12 17:20:19 +09:00
Alex Osborne 709ac00dd2 Add PaginationBehavior: repeatedly clicks next-page and extracts links 2026-06-11 13:34:55 +09:00
Alex Osborne 54b95f5e97 groovy profile: remove stray } 2026-06-10 18:23:12 +09:00
Alex Osborne fc76935fea webdriver: Support Cookie.sameSite value "default" 2026-06-10 18:19:04 +09:00
Alex Osborne bd34a7a0a9 Log the exception when creating new jobs fails 2026-06-10 17:43:57 +09:00
Alex Osborne 4b1c1d1e57 Merge pull request #735 from internetarchive/dependabot/maven/minor-and-patch-e83b0aec52 
Bump the minor-and-patch group with 11 updates
 2026-06-08 15:58:53 +09:00
dependabot[bot] 3406c14632 Bump the minor-and-patch group with 11 updates 
Bumps the minor-and-patch group with 11 updates:

| Package | From | To |
| --- | --- | --- |
| [org.junit.jupiter:junit-jupiter](https://github.com/junit-team/junit-framework) | `6.0.3` | `6.1.0` |
| [org.apache.groovy:groovy-bom](https://github.com/apache/groovy) | `5.0.5` | `5.0.6` |
| [com.fasterxml.jackson:jackson-bom](https://github.com/FasterXML/jackson-bom) | `2.21.3` | `2.22.0` |
| [org.eclipse.jetty:jetty-bom](https://github.com/jetty/jetty.project) | `12.0.34` | `12.0.36` |
| [org.eclipse.jetty.ee10:jetty-ee10-bom](https://github.com/jetty/jetty.project) | `12.0.34` | `12.0.36` |
| [org.slf4j:slf4j-bom](https://github.com/qos-ch/slf4j) | `2.0.17` | `2.0.18` |
| [org.apache.maven.plugins:maven-surefire-plugin](https://github.com/apache/maven-surefire) | `3.5.5` | `3.5.6` |
| [dnsjava:dnsjava](https://github.com/dnsjava/dnsjava) | `3.6.4` | `3.6.5` |
| org.glassfish.jaxb:jaxb-runtime | `4.0.7` | `4.0.9` |
| [com.rabbitmq:amqp-client](https://github.com/rabbitmq/rabbitmq-java-client) | `5.30.0` | `5.31.0` |
| org.apache.kafka:kafka-clients | `4.2.0` | `4.3.0` |


Updates `org.junit.jupiter:junit-jupiter` from 6.0.3 to 6.1.0
- [Release notes](https://github.com/junit-team/junit-framework/releases)
- [Commits](https://github.com/junit-team/junit-framework/compare/r6.0.3...r6.1.0)

Updates `org.apache.groovy:groovy-bom` from 5.0.5 to 5.0.6
- [Commits](https://github.com/apache/groovy/commits)

Updates `com.fasterxml.jackson:jackson-bom` from 2.21.3 to 2.22.0
- [Commits](https://github.com/FasterXML/jackson-bom/compare/jackson-bom-2.21.3...jackson-bom-2.22.0)

Updates `org.eclipse.jetty:jetty-bom` from 12.0.34 to 12.0.36
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.34...jetty-12.0.36)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.34 to 12.0.36
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.34...jetty-12.0.36)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.34 to 12.0.36
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.34...jetty-12.0.36)

Updates `org.slf4j:slf4j-bom` from 2.0.17 to 2.0.18
- [Release notes](https://github.com/qos-ch/slf4j/releases)
- [Commits](https://github.com/qos-ch/slf4j/compare/v_2.0.17...v_2.0.18)

Updates `org.apache.maven.plugins:maven-surefire-plugin` from 3.5.5 to 3.5.6
- [Release notes](https://github.com/apache/maven-surefire/releases)
- [Commits](https://github.com/apache/maven-surefire/compare/surefire-3.5.5...surefire-3.5.6)

Updates `dnsjava:dnsjava` from 3.6.4 to 3.6.5
- [Release notes](https://github.com/dnsjava/dnsjava/releases)
- [Changelog](https://github.com/dnsjava/dnsjava/blob/master/Changelog)
- [Commits](https://github.com/dnsjava/dnsjava/commits)

Updates `org.glassfish.jaxb:jaxb-runtime` from 4.0.7 to 4.0.9

Updates `com.rabbitmq:amqp-client` from 5.30.0 to 5.31.0
- [Release notes](https://github.com/rabbitmq/rabbitmq-java-client/releases)
- [Commits](https://github.com/rabbitmq/rabbitmq-java-client/compare/v5.30.0...v5.31.0)

Updates `org.apache.kafka:kafka-clients` from 4.2.0 to 4.3.0

---
updated-dependencies:
- dependency-name: org.junit.jupiter:junit-jupiter
  dependency-version: 6.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.apache.groovy:groovy-bom
  dependency-version: 5.0.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.fasterxml.jackson:jackson-bom
  dependency-version: 2.22.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty:jetty-bom
  dependency-version: 12.0.36
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.36
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.36
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.slf4j:slf4j-bom
  dependency-version: 2.0.18
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-surefire-plugin
  dependency-version: 3.5.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: dnsjava:dnsjava
  dependency-version: 3.6.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.glassfish.jaxb:jaxb-runtime
  dependency-version: 4.0.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.rabbitmq:amqp-client
  dependency-version: 5.31.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.apache.kafka:kafka-clients
  dependency-version: 4.3.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-06-03 16:56:20 +00:00
Alex Osborne e0b46789bc [maven-release-plugin] prepare for next development iteration 2026-05-26 11:51:31 +09:00
Alex Osborne bc28f4d1ae [maven-release-plugin] prepare release 3.15.0 3.15.0 2026-05-26 11:51:25 +09:00
Alex Osborne b4dd4949d7 Update CHANGELOG with links for 3.15.0 2026-05-26 11:50:38 +09:00