lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-25 16:06:33 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,847 Commits 38 Branches 32 Tags
89bd3474ba658103cc2c9e71b4eef26e1841f921
Commit Graph
2847 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Alex Osborne 89bd3474ba Add docgen/target to .gitignore 2025-10-30 10:21:16 +09:00
Alex Osborne cf40d8513e Update CHANGELOG.md for 3.12.0 release 2025-10-30 10:20:36 +09:00
Alex Osborne a76a91ac3b readthedocs: use mvn --batch-mode 
Disables color and download progress output for more readable build logs.
 2025-10-28 16:50:00 +09:00
Alex Osborne 03b9c27247 Merge pull request #683 from internetarchive/ato/beandoc 
docs: replace javalang with an annotation processor
 2025-10-28 16:32:52 +09:00
Alex Osborne 89e5ecc7d6 docs: replace javalang with an annotation processor 
This means we can rely on the Java compiler to parse the source code instead of the old python library which no longer works as it only supported Java 8 source code. The new processor generates a META-INF/heritrix-beans.json file for each module at compile time which the sphinx beandoc plugin then reads.

My intent is to eventually use the heritrix-beans.json files to also implement autocomplete and context-sensitive help in the Heritrix config editor.
 2025-10-28 16:26:19 +09:00
Alex Osborne ab589e6b18 CI: Replace JDK 24 with 25 in build matrix 2025-10-27 15:46:11 +09:00
dependabot[bot]dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>Alex Osborne
 		d627118307 Bump the minor-and-patch group across 1 directory with 18 updates (#680) 
* Bump the minor-and-patch group across 1 directory with 18 updates

Bumps the minor-and-patch group with 18 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [org.eclipse.jetty:jetty-bom](https://github.com/jetty/jetty.project) | `12.0.27` | `12.0.29` |
| [org.eclipse.jetty.ee10:jetty-ee10-bom](https://github.com/jetty/jetty.project) | `12.0.27` | `12.0.29` |
| [org.apache.maven.plugins:maven-compiler-plugin](https://github.com/apache/maven-compiler-plugin) | `3.13.0` | `3.14.1` |
| [org.codehaus.mojo:versions-maven-plugin](https://github.com/mojohaus/versions) | `2.18.0` | `2.19.1` |
| [org.sonatype.central:central-publishing-maven-plugin](https://github.com/sonatype/central-publishing-maven-plugin) | `0.7.0` | `0.9.0` |
| [org.apache.maven.plugins:maven-gpg-plugin](https://github.com/apache/maven-gpg-plugin) | `3.2.7` | `3.2.8` |
| [org.springframework:spring-core](https://github.com/spring-projects/spring-framework) | `6.2.11` | `6.2.12` |
| [org.springframework:spring-beans](https://github.com/spring-projects/spring-framework) | `6.2.11` | `6.2.12` |
| [org.springframework:spring-context](https://github.com/spring-projects/spring-framework) | `6.2.11` | `6.2.12` |
| [org.springframework:spring-expression](https://github.com/spring-projects/spring-framework) | `6.2.11` | `6.2.12` |
| [org.netpreserve.commons:webarchive-commons](https://github.com/iipc/webarchive-commons) | `3.0.0` | `3.0.1` |
| [com.github.mwiede:jsch](https://github.com/mwiede/jsch) | `2.27.3` | `2.27.4` |
| org.apache.pdfbox:pdfbox | `3.0.5` | `3.0.6` |
| org.glassfish.jaxb:jaxb-runtime | `4.0.5` | `4.0.6` |
| [org.webjars:webjars-locator-lite](https://github.com/webjars/webjars-locator-lite) | `1.1.0` | `1.1.2` |
| [org.apache.maven.plugins:maven-assembly-plugin](https://github.com/apache/maven-assembly-plugin) | `3.3.0` | `3.7.1` |
| [com.rabbitmq:amqp-client](https://github.com/rabbitmq/rabbitmq-java-client) | `5.26.0` | `5.27.0` |
| [com.rethinkdb:rethinkdb-driver](https://github.com/rethinkdb/rethinkdb-java) | `2.3.3` | `2.4.4` |



Updates `org.eclipse.jetty:jetty-bom` from 12.0.27 to 12.0.29
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.27...jetty-12.0.29)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.27 to 12.0.29
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.27...jetty-12.0.29)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.27 to 12.0.29
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.27...jetty-12.0.29)

Updates `org.apache.maven.plugins:maven-compiler-plugin` from 3.13.0 to 3.14.1
- [Release notes](https://github.com/apache/maven-compiler-plugin/releases)
- [Commits](https://github.com/apache/maven-compiler-plugin/compare/maven-compiler-plugin-3.13.0...maven-compiler-plugin-3.14.1)

Updates `org.codehaus.mojo:versions-maven-plugin` from 2.18.0 to 2.19.1
- [Release notes](https://github.com/mojohaus/versions/releases)
- [Changelog](https://github.com/mojohaus/versions/blob/master/ReleaseNotes.md)
- [Commits](https://github.com/mojohaus/versions/compare/2.18.0...2.19.1)

Updates `org.sonatype.central:central-publishing-maven-plugin` from 0.7.0 to 0.9.0
- [Commits](https://github.com/sonatype/central-publishing-maven-plugin/commits)

Updates `org.apache.maven.plugins:maven-gpg-plugin` from 3.2.7 to 3.2.8
- [Release notes](https://github.com/apache/maven-gpg-plugin/releases)
- [Commits](https://github.com/apache/maven-gpg-plugin/compare/maven-gpg-plugin-3.2.7...maven-gpg-plugin-3.2.8)

Updates `org.springframework:spring-core` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12)

Updates `org.springframework:spring-beans` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12)

Updates `org.springframework:spring-context` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12)

Updates `org.springframework:spring-expression` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12)

Updates `org.springframework:spring-beans` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12)

Updates `org.springframework:spring-context` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12)

Updates `org.springframework:spring-expression` from 6.2.11 to 6.2.12
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v6.2.11...v6.2.12)

Updates `org.netpreserve.commons:webarchive-commons` from 3.0.0 to 3.0.1
- [Release notes](https://github.com/iipc/webarchive-commons/releases)
- [Changelog](https://github.com/iipc/webarchive-commons/blob/master/CHANGES.md)
- [Commits](https://github.com/iipc/webarchive-commons/compare/webarchive-commons-3.0.0...webarchive-commons-3.0.1)

Updates `com.github.mwiede:jsch` from 2.27.3 to 2.27.4
- [Release notes](https://github.com/mwiede/jsch/releases)
- [Changelog](https://github.com/mwiede/jsch/blob/master/ChangeLog)
- [Commits](https://github.com/mwiede/jsch/compare/jsch-2.27.3...jsch-2.27.4)

Updates `org.apache.pdfbox:pdfbox` from 3.0.5 to 3.0.6

Updates `org.glassfish.jaxb:jaxb-runtime` from 4.0.5 to 4.0.6

Updates `org.webjars:webjars-locator-lite` from 1.1.0 to 1.1.2
- [Release notes](https://github.com/webjars/webjars-locator-lite/releases)
- [Commits](https://github.com/webjars/webjars-locator-lite/compare/webjars-locator-lite-1.1.0...webjars-locator-lite-1.1.2)

Updates `org.apache.maven.plugins:maven-assembly-plugin` from 3.3.0 to 3.7.1
- [Release notes](https://github.com/apache/maven-assembly-plugin/releases)
- [Commits](https://github.com/apache/maven-assembly-plugin/compare/maven-assembly-plugin-3.3.0...maven-assembly-plugin-3.7.1)

Updates `com.rabbitmq:amqp-client` from 5.26.0 to 5.27.0
- [Release notes](https://github.com/rabbitmq/rabbitmq-java-client/releases)
- [Commits](https://github.com/rabbitmq/rabbitmq-java-client/compare/v5.26.0...v5.27.0)

Updates `com.rethinkdb:rethinkdb-driver` from 2.3.3 to 2.4.4
- [Release notes](https://github.com/rethinkdb/rethinkdb-java/releases)
- [Commits](https://github.com/rethinkdb/rethinkdb-java/commits/v2.4.4)

---
updated-dependencies:
- dependency-name: org.eclipse.jetty:jetty-bom
  dependency-version: 12.0.29
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.29
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.29
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-compiler-plugin
  dependency-version: 3.14.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.codehaus.mojo:versions-maven-plugin
  dependency-version: 2.19.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.sonatype.central:central-publishing-maven-plugin
  dependency-version: 0.9.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-gpg-plugin
  dependency-version: 3.2.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-core
  dependency-version: 6.2.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
  dependency-version: 6.2.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
  dependency-version: 6.2.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
  dependency-version: 6.2.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
  dependency-version: 6.2.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
  dependency-version: 6.2.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
  dependency-version: 6.2.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.netpreserve.commons:webarchive-commons
  dependency-version: 3.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.github.mwiede:jsch
  dependency-version: 2.27.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.apache.pdfbox:pdfbox
  dependency-version: 3.0.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.glassfish.jaxb:jaxb-runtime
  dependency-version: 4.0.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars:webjars-locator-lite
  dependency-version: 1.1.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-assembly-plugin
  dependency-version: 3.7.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: com.rabbitmq:amqp-client
  dependency-version: 5.27.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: com.rethinkdb:rethinkdb-driver
  dependency-version: 2.4.4
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add json-simple as contrib dependency

Needed for TroughClient and no longer included via rethinkdb.

* Exclude junit from json-simple

* Add Jackson BOM 2.20.0 to dependency management

This ensures rethinkdb doesn't pull in an old version.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Alex Osborne <aosborne@nla.gov.au>
 2025-10-27 15:42:40 +09:00
dependabot[bot] 14cb366928 Bump org.junit.jupiter:junit-jupiter from 5.13.4 to 6.0.0 
Bumps [org.junit.jupiter:junit-jupiter](https://github.com/junit-team/junit-framework) from 5.13.4 to 6.0.0.
- [Release notes](https://github.com/junit-team/junit-framework/releases)
- [Commits](https://github.com/junit-team/junit-framework/compare/r5.13.4...r6.0.0)

---
updated-dependencies:
- dependency-name: org.junit.jupiter:junit-jupiter
  dependency-version: 6.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2025-10-27 15:21:39 +09:00
dependabot[bot] cbe879fb37 Bump org.apache.kafka:kafka-clients from 3.9.1 to 4.1.0 
Bumps org.apache.kafka:kafka-clients from 3.9.1 to 4.1.0.

---
updated-dependencies:
- dependency-name: org.apache.kafka:kafka-clients
  dependency-version: 4.1.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2025-10-27 15:19:34 +09:00
Alex Osborne d5b40f580f dependabot: add ignore for net.htmlparser.jericho:jericho-html 2025-10-27 15:15:25 +09:00
Alex Osborne 291164a09e dependabot: add ignore for com.sleepycat:je 
It has a larger version number in central that's actually older.
 2025-10-27 15:14:35 +09:00
Alex Osborne d69dc04528 dependabot: add ignores for codemirror and jetty 12.1 2025-10-27 14:44:46 +09:00
Alex Osborne a50c012d2d Merge pull request #677 from internetarchive/dependabot/maven/org.apache.groovy-groovy-bom-5.0.2 
Bump org.apache.groovy:groovy-bom from 4.0.28 to 5.0.2
 2025-10-27 10:21:22 +09:00
Alex Osborne bfe45b25cd Merge pull request #675 from internetarchive/dependabot/maven/org.apache.maven.plugins-maven-surefire-plugin-3.5.4 
Bump org.apache.maven.plugins:maven-surefire-plugin from 2.22.2 to 3.5.4
 2025-10-27 10:20:26 +09:00
dependabot[bot] c50f4dc461 Bump org.apache.maven.plugins:maven-resources-plugin from 2.5 to 3.3.1 (#674) 2025-10-27 01:17:23 +00:00
dependabot[bot] 8fe22b503c Bump org.apache.maven.plugins:maven-javadoc-plugin from 2.6.1 to 3.12.0 (#673) 2025-10-27 01:16:59 +00:00
dependabot[bot] 6c25801d13 Bump org.apache.groovy:groovy-bom from 4.0.28 to 5.0.2 
Bumps [org.apache.groovy:groovy-bom](https://github.com/apache/groovy) from 4.0.28 to 5.0.2.
- [Commits](https://github.com/apache/groovy/commits)

---
updated-dependencies:
- dependency-name: org.apache.groovy:groovy-bom
  dependency-version: 5.0.2
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2025-10-27 01:15:06 +00:00
dependabot[bot] 237d19a5d4 Bump org.apache.maven.plugins:maven-surefire-plugin from 2.22.2 to 3.5.4 
Bumps [org.apache.maven.plugins:maven-surefire-plugin](https://github.com/apache/maven-surefire) from 2.22.2 to 3.5.4.
- [Release notes](https://github.com/apache/maven-surefire/releases)
- [Commits](https://github.com/apache/maven-surefire/compare/surefire-2.22.2...surefire-3.5.4)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-surefire-plugin
  dependency-version: 3.5.4
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2025-10-27 01:14:45 +00:00
Alex Osborne 3a27aeb37d Merge pull request #671 from internetarchive/adam/handle_more_bdb_shutdown_interrupts 
fix: handle more bdb shutdown interrupts
 2025-10-27 10:06:20 +09:00
Alex Osborne dd11f87ce7 Enable dependabot updates monthly with minor updates grouped 2025-10-27 10:05:16 +09:00
Alex Osborne 26bb59621b Merge pull request #672 from internetarchive/adam/add_configurable_extractor_js_script_attribute_rules 
feat: Add configurable regex rules to block extraction of script tags based on the attributes of the tag.
 2025-10-21 22:11:28 +09:00
Adam Miller 8d09cc4e50 feat: Pass along script tag attributes to JS extractor for configurable rejection. 2025-10-07 11:04:12 -07:00
Adam Miller 558727ac37 fix handle more bdb shutdown interrupts 2025-10-03 17:17:46 -07:00
Kristinn Sigurðsson 29d2d7919b Merge pull request #670 from internetarchive/adam/fix_configurable_js_extractor_strict_overlay 
fix: sheet setting of extractorJs.strict doesn't work without kp keyname matching
 2025-10-01 08:35:50 +00:00
Adam Miller e92475ac43 fix: sheet setting of extractorJs.strict doesn't work without kp keyname matching 2025-09-30 14:14:06 -07:00
Alex Osborne 978345c51a Merge pull request #669 from internetarchive/adam/fix_crawl_end_time_checkpoint_resumption 
fix: don't restore crawlEndTime when resuming from checkpoint.
 2025-09-30 07:21:15 +10:00
Adam Miller 38e7156ef6 fix: don't restore crawlEndTime when resuming from checkpoint. 2025-09-25 10:50:47 -07:00
Alex Osborne b988940eff [maven-release-plugin] prepare for next development iteration 2025-09-22 13:57:47 +09:00
Alex Osborne e3bdd7220e [maven-release-plugin] prepare release 3.11.0 3.11.0 2025-09-22 13:57:41 +09:00
Alex Osborne 41aa946adb Update CHANGELOG for 3.11.0 release 2025-09-22 13:56:26 +09:00
Alex Osborne 25b621ce61 Bump jakarta.xml.bind-api from 4.0.2 to 4.0.4 2025-09-22 13:51:48 +09:00
Alex Osborne cb986a0ed2 Bump codemirror__language from 6.11.2 to 6.11.3 2025-09-22 13:50:01 +09:00
Alex Osborne e439f3568b Bump jetty from 12.0.25 to 12.0.27 2025-09-22 13:48:49 +09:00
Alex Osborne 1ed64142ea Bump jsch from 2.27.2 to 2.27.3 2025-09-22 13:46:42 +09:00
Alex Osborne 94c0daeec5 Bump spring from 6.2.10 to 6.2.11 2025-09-22 13:45:11 +09:00
Alex Osborne 95d74ab4e7 Bump gson from 2.13.1 to 2.13.2 2025-09-22 13:43:40 +09:00
Alex Osborne 5e2809f7c6 Merge pull request #668 from internetarchive/adam/knowledgable_extractorjs_extends_configurable_extractorjs 
feat: KnowledgableExtractorJS extends ConfigurableExtractorJS to inhe…
 2025-09-22 13:39:21 +09:00
Adam Miller ec3cbe679f feat: KnowledgableExtractorJS extends ConfigurableExtractorJS to inherit new config options 2025-09-18 15:13:01 -07:00
Alex Osborne 351cc2bea2 Update CHANGELOG.md for #667 2025-09-12 16:45:51 +09:00
Alex Osborne 96b6480f9c Merge pull request #667 from internetarchive/adam/strip_invalid_xml_chars_from_rest_api_output 
feat: strip invalid chars from xml rest api output
 2025-09-12 16:38:55 +09:00
Adam Miller fd2aed89fb feat: strip invalid chars from xml rest api output 2025-09-11 17:42:07 -07:00
Alex Osborne d484007f57 FetchHTTP2: Note that WARC doesn't specify the recording of HTTP/2 and 3 
iipc/warc-specifications#106
 2025-09-06 10:29:24 +09:00
Alex Osborne 20dcc89244 [maven-release-plugin] prepare for next development iteration 2025-08-29 17:26:11 +09:00
Alex Osborne 7a2c676dc9 [maven-release-plugin] prepare release 3.10.2 3.10.2 2025-08-29 17:26:05 +09:00
Alex Osborne de47ab97a6 Update CHANGELOG for 3.10.2 2025-08-29 17:21:13 +09:00
Alex Osborne 31539e2d62 Use Groovy BOM to ensure consistent versions 2025-08-28 15:35:14 +09:00
Alex Osborne e42306ceb1 Use slf4j BOM to ensure consistent versions 2025-08-28 15:32:48 +09:00
Alex Osborne bb04d1e0a9 Bump jetty from 12.0.22 to 12.0.25 2025-08-28 15:27:42 +09:00
Alex Osborne 3cde816e95 Use jetty BOM to ensure consistent versions 2025-08-28 15:27:10 +09:00
Alex Osborne c79de0bec5 Merge pull request #665 from internetarchive/fetchdns-zero-addr 
FetchDNS: Treat 0.0.0.0 as resolution failure
 2025-08-26 09:58:44 +09:00