lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-22 05:35:49 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,889 Commits 38 Branches 33 Tags
107a249a5d617caa025458371abb8a0318dd9df0
Commit Graph
867 Commits
Author SHA1 Message Date
Erik Körner 107a249a5d Limit height of Exit Java joblist, set auto-scroll behaviour on overflow 2025-12-18 12:26:08 +01:00
Erik Körner 78b76345d0 Update other endpoints to use new JsonMarshaller, fix some edge cases 2025-12-17 23:25:37 +01:00
Erik Körner cf87b045b9 Add /beans endpoint JSON handling 2025-12-17 22:51:37 +01:00
Erik Körner 18b569736c Add org.restlet.ext.json dependency, add application/json REST API responses 2025-12-17 18:02:36 +01:00
Alex Osborne 3a1974528a [maven-release-plugin] prepare for next development iteration 2025-12-11 13:54:13 +09:00
Alex Osborne ec62a9b0ad [maven-release-plugin] prepare release 3.13.0 2025-12-11 13:54:07 +09:00
Leslie Bellony 8a7db14a51 Add sizeOnDisk on job status (#699) 2025-12-09 15:59:03 +01:00
Manikya Rathore e4523d63ee fix 690 issue Consolidate and log on deleting URIs from BdB database 2025-11-22 18:42:36 +05:30
Alex Osborne 7c1b1639cb Merge pull request #684 from internetarchive/ato/autocomplete 
Config editor: implement autocompletion
 2025-10-30 10:42:19 +09:00
Alex Osborne ab41c7a658 [maven-release-plugin] prepare for next development iteration 2025-10-30 10:25:16 +09:00
Alex Osborne b626f52d67 [maven-release-plugin] prepare release 3.12.0 2025-10-30 10:25:11 +09:00
Alex Osborne c0e5018eef Config editor: implement autocompletion 
This adds context-sensitive autocompletion for bean classes, bean ids, property names, basic Spring XML tags and attributes. The bean completions use a new `/engine/beandoc` endpoint that serves the combination of all the `/META-INF/heritrix-beans.json` files generated at compile-time by the heritrix-docgen annotation processor.
 2025-10-30 09:53:16 +09:00
dependabot[bot]andAlex Osborne d627118307 Bump the minor-and-patch group across 1 directory with 18 updates (#680) 
* Bump the minor-and-patch group across 1 directory with 18 updates

Bumps the minor-and-patch group with 18 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [org.eclipse.jetty:jetty-bom](https://github.com/jetty/jetty.project) | `12.0.27` | `12.0.29` |
| [org.eclipse.jetty.ee10:jetty-ee10-bom](https://github.com/jetty/jetty.project) | `12.0.27` | `12.0.29` |
| [org.apache.maven.plugins:maven-compiler-plugin](https://github.com/apache/maven-compiler-plugin) | `3.13.0` | `3.14.1` |
| [org.codehaus.mojo:versions-maven-plugin](https://github.com/mojohaus/versions) | `2.18.0` | `2.19.1` |
| [org.sonatype.central:central-publishing-maven-plugin](https://github.com/sonatype/central-publishing-maven-plugin) | `0.7.0` | `0.9.0` |
| [org.apache.maven.plugins:maven-gpg-plugin](https://github.com/apache/maven-gpg-plugin) | `3.2.7` | `3.2.8` |
| [org.springframework:spring-core](https://github.com/spring-projects/spring-framework) | `6.2.11` | `6.2.12` |
| [org.springframework:spring-beans](https://github.com/spring-projects/spring-framework) | `6.2.11` | `6.2.12` |
| [org.springframework:spring-context](https://github.com/spring-projects/spring-framework) | `6.2.11` | `6.2.12` |
| [org.springframework:spring-expression](https://github.com/spring-projects/spring-framework) | `6.2.11` | `6.2.12` |
| [org.netpreserve.commons:webarchive-commons](https://github.com/iipc/webarchive-commons) | `3.0.0` | `3.0.1` |
| [com.github.mwiede:jsch](https://github.com/mwiede/jsch) | `2.27.3` | `2.27.4` |
| org.apache.pdfbox:pdfbox | `3.0.5` | `3.0.6` |
| org.glassfish.jaxb:jaxb-runtime | `4.0.5` | `4.0.6` |
| [org.webjars:webjars-locator-lite](https://github.com/webjars/webjars-locator-lite) | `1.1.0` | `1.1.2` |
| [org.apache.maven.plugins:maven-assembly-plugin](https://github.com/apache/maven-assembly-plugin) | `3.3.0` | `3.7.1` |
| [com.rabbitmq:amqp-client](https://github.com/rabbitmq/rabbitmq-java-client) | `5.26.0` | `5.27.0` |
| [com.rethinkdb:rethinkdb-driver](https://github.com/rethinkdb/rethinkdb-java) | `2.3.3` | `2.4.4` |



Updates `org.eclipse.jetty:jetty-bom` from 12.0.27 to 12.0.29
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.27...jetty-12.0.29)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.27 to 12.0.29
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.27...jetty-12.0.29)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.27 to 12.0.29
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.27...jetty-12.0.29)

Updates `org.apache.maven.plugins:maven-compiler-plugin` from 3.13.0 to 3.14.1
- [Release notes](https://github.com/apache/maven-compiler-plugin/releases)
- [Commits](https://github.com/apache/maven-compiler-plugin/compare/maven-compiler-plugin-3.13.0...maven-compiler-plugin-3.14.1)

Updates `org.codehaus.mojo:versions-maven-plugin` from 2.18.0 to 2.19.1
- [Release notes](https://github.com/mojohaus/versions/releases)
- [Changelog](https://github.com/mojohaus/versions/blob/master/ReleaseNotes.md)
- [Commits](https://github.com/mojohaus/versions/compare/2.18.0...2.19.1)

Updates `org.sonatype.central:central-publishing-maven-plugin` from 0.7.0 to 0.9.0
- [Commits](https://github.com/sonatype/central-publishing-maven-plugin/commits)

Updates `org.apache.maven.plugins:maven-gpg-plugin` from 3.2.7 to 3.2.8
- [Release notes](https://github.com/apache/maven-gpg-plugin/releases)
- [Commits](https://github.com/apache/maven-gpg-plugin/compare/maven-gpg-plugin-3.2.7...maven-gpg-plugin-3.2.8)

Updates `org.springframework:spring-core` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12)

Updates `org.springframework:spring-beans` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12)

Updates `org.springframework:spring-context` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12)

Updates `org.springframework:spring-expression` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12)

Updates `org.springframework:spring-beans` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12)

Updates `org.springframework:spring-context` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12)

Updates `org.springframework:spring-expression` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12)

Updates `org.netpreserve.commons:webarchive-commons` from 3.0.0 to 3.0.1
- [Release notes](https://github.com/iipc/webarchive-commons/releases)
- [Changelog](https://github.com/iipc/webarchive-commons/blob/master/CHANGES.md)
- [Commits](https://github.com/iipc/webarchive-commons/compare/webarchive-commons-3.0.0...webarchive-commons-3.0.1)

Updates `com.github.mwiede:jsch` from 2.27.3 to 2.27.4
- [Release notes](https://github.com/mwiede/jsch/releases)
- [Changelog](https://github.com/mwiede/jsch/blob/master/ChangeLog)
- [Commits](https://github.com/mwiede/jsch/compare/jsch-2.27.3...jsch-2.27.4)

Updates `org.apache.pdfbox:pdfbox` from 3.0.5 to 3.0.6

Updates `org.glassfish.jaxb:jaxb-runtime` from 4.0.5 to 4.0.6

Updates `org.webjars:webjars-locator-lite` from 1.1.0 to 1.1.2
- [Release notes](https://github.com/webjars/webjars-locator-lite/releases)
- [Commits](https://github.com/webjars/webjars-locator-lite/compare/webjars-locator-lite-1.1.0...webjars-locator-lite-1.1.2)

Updates `org.apache.maven.plugins:maven-assembly-plugin` from 3.3.0 to 3.7.1
- [Release notes](https://github.com/apache/maven-assembly-plugin/releases)
- [Commits](https://github.com/apache/maven-assembly-plugin/compare/maven-assembly-plugin-3.3.0...maven-assembly-plugin-3.7.1)

Updates `com.rabbitmq:amqp-client` from 5.26.0 to 5.27.0
- [Release notes](https://github.com/rabbitmq/rabbitmq-java-client/releases)
- [Commits](https://github.com/rabbitmq/rabbitmq-java-client/compare/v5.26.0...v5.27.0)

Updates `com.rethinkdb:rethinkdb-driver` from 2.3.3 to 2.4.4
- [Release notes](https://github.com/rethinkdb/rethinkdb-java/releases)
- [Commits](https://github.com/rethinkdb/rethinkdb-java/commits/v2.4.4)

---
updated-dependencies:
- dependency-name: org.eclipse.jetty:jetty-bom
  dependency-version: 12.0.29
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.29
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.29
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-compiler-plugin
  dependency-version: 3.14.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.codehaus.mojo:versions-maven-plugin
  dependency-version: 2.19.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.sonatype.central:central-publishing-maven-plugin
  dependency-version: 0.9.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-gpg-plugin
  dependency-version: 3.2.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-core
  dependency-version: 6.2.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
  dependency-version: 6.2.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
  dependency-version: 6.2.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
  dependency-version: 6.2.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
  dependency-version: 6.2.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
  dependency-version: 6.2.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
  dependency-version: 6.2.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.netpreserve.commons:webarchive-commons
  dependency-version: 3.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.github.mwiede:jsch
  dependency-version: 2.27.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.apache.pdfbox:pdfbox
  dependency-version: 3.0.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.glassfish.jaxb:jaxb-runtime
  dependency-version: 4.0.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars:webjars-locator-lite
  dependency-version: 1.1.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-assembly-plugin
  dependency-version: 3.7.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: com.rabbitmq:amqp-client
  dependency-version: 5.27.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: com.rethinkdb:rethinkdb-driver
  dependency-version: 2.4.4
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add json-simple as contrib dependency

Needed for TroughClient and no longer included via rethinkdb.

* Exclude junit from json-simple

* Add Jackson BOM 2.20.0 to dependency management

This ensures rethinkdb doesn't pull in an old version.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Alex Osborne <aosborne@nla.gov.au>
 2025-10-27 15:42:40 +09:00
Alex Osborne bfe45b25cd Merge pull request #675 from internetarchive/dependabot/maven/org.apache.maven.plugins-maven-surefire-plugin-3.5.4 
Bump org.apache.maven.plugins:maven-surefire-plugin from 2.22.2 to 3.5.4
 2025-10-27 10:20:26 +09:00
dependabot[bot] 237d19a5d4 Bump org.apache.maven.plugins:maven-surefire-plugin from 2.22.2 to 3.5.4 
Bumps [org.apache.maven.plugins:maven-surefire-plugin](https://github.com/apache/maven-surefire) from 2.22.2 to 3.5.4.
- [Release notes](https://github.com/apache/maven-surefire/releases)
- [Commits](https://github.com/apache/maven-surefire/compare/surefire-2.22.2...surefire-3.5.4)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-surefire-plugin
  dependency-version: 3.5.4
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2025-10-27 01:14:45 +00:00
Adam Miller 558727ac37 fix handle more bdb shutdown interrupts 2025-10-03 17:17:46 -07:00
Adam Miller 38e7156ef6 fix: don't restore crawlEndTime when resuming from checkpoint. 2025-09-25 10:50:47 -07:00
Alex Osborne b988940eff [maven-release-plugin] prepare for next development iteration 2025-09-22 13:57:47 +09:00
Alex Osborne e3bdd7220e [maven-release-plugin] prepare release 3.11.0 2025-09-22 13:57:41 +09:00
Alex Osborne 25b621ce61 Bump jakarta.xml.bind-api from 4.0.2 to 4.0.4 2025-09-22 13:51:48 +09:00
Alex Osborne cb986a0ed2 Bump codemirror__language from 6.11.2 to 6.11.3 2025-09-22 13:50:01 +09:00
Adam Miller fd2aed89fb feat: strip invalid chars from xml rest api output 2025-09-11 17:42:07 -07:00
Alex Osborne 20dcc89244 [maven-release-plugin] prepare for next development iteration 2025-08-29 17:26:11 +09:00
Alex Osborne 7a2c676dc9 [maven-release-plugin] prepare release 3.10.2 2025-08-29 17:26:05 +09:00
Alex Osborne e42306ceb1 Use slf4j BOM to ensure consistent versions 2025-08-28 15:32:48 +09:00
Alex Osborne 3cde816e95 Use jetty BOM to ensure consistent versions 2025-08-28 15:27:10 +09:00
Alex Osborne b1be6fbe0d Bump codemirror__view from 6.37.2 to 6.38.1 2025-08-20 17:05:03 +09:00
Alex Osborne 5983f61626 Bump codemirror@legacy-modes from 6.5.0 to 6.5.1 2025-08-20 16:53:23 +09:00
Alex Osborne c8c2512e77 Bump codemirror@language from 6.11.1 to 6.11.2 2025-08-20 16:53:10 +09:00
Alex Osborne ea67b59e51 [maven-release-plugin] prepare for next development iteration 2025-07-21 17:15:17 +09:00
Alex Osborne 1cf6762c7f [maven-release-plugin] prepare release 3.10.1 2025-07-21 17:15:12 +09:00
Alex Osborne 2388db4229 Bump commons-lang from 2.6 to 3.18.0 2025-07-21 16:55:31 +09:00
Alex Osborne 8eaaa27ae7 Bump codemirror@search to 6.5.11 and codemirror@state to 6.37.2 2025-07-21 16:06:32 +09:00
Alex Osborne c45544a47b Bump restlet from 2.6.0-rc1 to 2.6.0 2025-07-21 16:03:24 +09:00
Alex Osborne 85ceab905c BrowserProcessor: create scratchDir on start 
This fixes an exception on first launch.
 2025-07-21 11:55:37 +09:00
Alex Osborne db63c7ec85 Seeds report: record redirect URI for statuses 303, 307 and 308 
These can also redirect, not just 301 and 302.

Fixes #564
 2025-06-30 22:24:29 +09:00
Alex Osborne 01378ddbbf [maven-release-plugin] prepare for next development iteration 2025-06-12 22:01:16 +09:00
Alex Osborne ecc46a38c3 [maven-release-plugin] prepare release 3.10.0 2025-06-12 22:01:10 +09:00
Alex Osborne 9b5900b528 FetchHTTP2: Avoid silently decoding gzip 
Fixes `Recorder getContentReplayInputStream
WARNING: gzip problem; using raw entity instead`.
 2025-06-12 21:36:14 +09:00
Adam Miller 0494d63ea2 fix: Avoid a NullPointerException when terminate is called again during shutdown 2025-06-11 17:44:52 -07:00
Alex Osborne 61af1d83c4 MitmProxy: Handle URLs generated by browsers but disallowed by java.net.URI 2025-06-11 20:38:54 +09:00
Alex Osborne 8a15156fa1 BrowserProcessor: Catch WebDriver timeouts and exceptions 2025-06-11 17:17:02 +09:00
Alex Osborne c3f9afe64b BrowserProcessor: use FetchHTTP2's configured proxy as an upstream proxy 2025-06-10 09:01:02 +09:00
Alex Osborne e68e4975c9 Rename to BrowserProcessor and -Behavior 
After I added them to the Bean reference they seemed out of place with the DecideRules and RecordBuilders.

We had two classes called 'Browser' which is potentially confusing. It's also probably good to more clearly differentiate the ExtractLinks browser behavior from the Extractor processors.
 2025-06-09 15:35:25 +09:00
Alex Osborne cf447f3bd9 Add Browser processor and behaviors to bean reference 2025-06-09 15:19:09 +09:00
Alex Osborne 6782fc065d Update dependencies 
- **junit-jupiter**: 5.12.2 → 5.13.1
- **spring**: 6.2.6 → 6.2.7
- **codemirror__language**: 6.11.0 → 6.11.1
- **codemirror__view**: 6.36.5 → 6.37.1
 2025-06-09 10:17:16 +09:00
Alex Osborne 1161d87217 Browser: Add processor report 2025-06-05 10:48:15 +09:00
Alex Osborne 2210cb7262 Browser: Handle navigation abort from downloads starting 2025-06-04 17:44:42 +09:00
Alex Osborne 5131ea4a80 Browser: Disable downloads in Firefox and Chrome 
This hopefully will stop us filling up ~/Downloads with random junk.
 2025-06-04 17:44:42 +09:00
Alex Osborne 722552feef Add Browser processor using WebDriver BiDi 
The Browser processor can load a fetched page in a local web browser, record any requests the browser makes and run behaviors that interact with the page such as scrolling down and extracting links.

This differs from my previous attempt (ExtractorChrome) in a few ways:

- Uses the new WebDriver BiDi standard instead of the Chrome Devtools Protocol. The new protocol is mostly browser-agnostic, more consistent and hopefully more stable.

- Uses a MITM proxy instead of CDP request interception for recording sub-resources. That's partly because BiDi is still missing some key interception APIs. Even so in practice I found the proxy method loads pages faster and more reliably, likely because responses can be streamed incrementally, which helps a lot for large resources or server-sent events.

- Even when HTTP/2 is unavailable, the new FetchHTTP2 module does connection pooling which makes loading browser requests a lot faster. The original FetchHTTP opened a new connection for every request.

- The Browser processor can be configured with a list of behavior beans making it more customizable and extensible.

Obvious areas for future development:

- More Behavior beans: take screenshots, saveg the rendered DOM, run Browsertrix-compatible behavior scripts

- Support for remote WebDrivers (e.g. Selenium Server or cloud services)
 2025-06-04 17:44:42 +09:00