lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-24 07:26:36 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,895 Commits 40 Branches 31 Tags
938c68dbd8a8a495316fd4bea30825375efc88ba
T
Clone
Open with VS Code Open with VSCodium Open with Intellij IDEA
Download ZIP Download TAR.GZ Download BUNDLE
dependabot[bot] 938c68dbd8 Bump the minor-and-patch group with 25 updates 
Bumps the minor-and-patch group with 25 updates:

| Package | From | To |
| --- | --- | --- |
| [org.junit.jupiter:junit-jupiter](https://github.com/junit-team/junit-framework) | `6.0.1` | `6.0.2` |
| [org.apache.groovy:groovy-bom](https://github.com/apache/groovy) | `5.0.3` | `5.0.4` |
| [com.fasterxml.jackson:jackson-bom](https://github.com/FasterXML/jackson-bom) | `2.20.1` | `2.21.0` |
| [org.eclipse.jetty:jetty-bom](https://github.com/jetty/jetty.project) | `12.0.31` | `12.0.32` |
| [org.eclipse.jetty.ee10:jetty-ee10-bom](https://github.com/jetty/jetty.project) | `12.0.31` | `12.0.32` |
| [org.apache.maven.plugins:maven-compiler-plugin](https://github.com/apache/maven-compiler-plugin) | `3.14.1` | `3.15.0` |
| [org.codehaus.mojo:versions-maven-plugin](https://github.com/mojohaus/versions) | `2.20.1` | `2.21.0` |
| [org.sonatype.central:central-publishing-maven-plugin](https://github.com/sonatype/central-publishing-maven-plugin) | `0.9.0` | `0.10.0` |
| [commons-codec:commons-codec](https://github.com/apache/commons-codec) | `1.20.0` | `1.21.0` |
| [dnsjava:dnsjava](https://github.com/dnsjava/dnsjava) | `3.6.3` | `3.6.4` |
| [org.springframework:spring-core](https://github.com/spring-projects/spring-framework) | `7.0.2` | `7.0.3` |
| [org.springframework:spring-beans](https://github.com/spring-projects/spring-framework) | `7.0.2` | `7.0.3` |
| [org.springframework:spring-context](https://github.com/spring-projects/spring-framework) | `7.0.2` | `7.0.3` |
| [org.springframework:spring-expression](https://github.com/spring-projects/spring-framework) | `7.0.2` | `7.0.3` |
| [org.webjars.npm:codemirror__autocomplete](https://github.com/codemirror/autocomplete) | `6.18.6` | `6.20.0` |
| [org.webjars.npm:codemirror__commands](https://github.com/codemirror/commands) | `6.8.1` | `6.10.1` |
| [org.webjars.npm:codemirror__language](https://github.com/codemirror/language) | `6.11.3` | `6.12.1` |
| [org.webjars.npm:codemirror__state](https://github.com/codemirror/state) | `6.5.2` | `6.5.4` |
| [org.webjars.npm:codemirror__view](https://github.com/codemirror/view) | `6.38.1` | `6.39.11` |
| [org.webjars.npm:codemirror__lint](https://github.com/codemirror/lint) | `6.8.5` | `6.9.2` |
| [org.webjars.npm:lezer__common](https://github.com/lezer-parser/common) | `1.2.3` | `1.5.0` |
| [org.webjars.npm:lezer__lr](https://github.com/lezer-parser/lr) | `1.4.2` | `1.4.7` |
| [org.webjars.npm:lezer__highlight](https://github.com/lezer-parser/highlight) | `1.2.1` | `1.2.3` |
| [org.webjars.npm:style-mod](https://github.com/marijnh/style-mod) | `4.1.2` | `4.1.3` |
| [at.yawk.lz4:lz4-java](https://github.com/yawkat/lz4-java) | `1.10.2` | `1.10.3` |


Updates `org.junit.jupiter:junit-jupiter` from 6.0.1 to 6.0.2
- [Release notes](https://github.com/junit-team/junit-framework/releases)
- [Commits](https://github.com/junit-team/junit-framework/compare/r6.0.1...r6.0.2)

Updates `org.apache.groovy:groovy-bom` from 5.0.3 to 5.0.4
- [Commits](https://github.com/apache/groovy/commits)

Updates `com.fasterxml.jackson:jackson-bom` from 2.20.1 to 2.21.0
- [Commits](https://github.com/FasterXML/jackson-bom/compare/jackson-bom-2.20.1...jackson-bom-2.21.0)

Updates `org.eclipse.jetty:jetty-bom` from 12.0.31 to 12.0.32
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.31...jetty-12.0.32)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.31 to 12.0.32
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.31...jetty-12.0.32)

Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.31 to 12.0.32
- [Release notes](https://github.com/jetty/jetty.project/releases)
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.31...jetty-12.0.32)

Updates `org.apache.maven.plugins:maven-compiler-plugin` from 3.14.1 to 3.15.0
- [Release notes](https://github.com/apache/maven-compiler-plugin/releases)
- [Commits](https://github.com/apache/maven-compiler-plugin/compare/maven-compiler-plugin-3.14.1...maven-compiler-plugin-3.15.0)

Updates `org.codehaus.mojo:versions-maven-plugin` from 2.20.1 to 2.21.0
- [Release notes](https://github.com/mojohaus/versions/releases)
- [Changelog](https://github.com/mojohaus/versions/blob/master/ReleaseNotes.md)
- [Commits](https://github.com/mojohaus/versions/compare/2.20.1...2.21.0)

Updates `org.sonatype.central:central-publishing-maven-plugin` from 0.9.0 to 0.10.0
- [Commits](https://github.com/sonatype/central-publishing-maven-plugin/commits)

Updates `commons-codec:commons-codec` from 1.20.0 to 1.21.0
- [Changelog](https://github.com/apache/commons-codec/blob/master/RELEASE-NOTES.txt)
- [Commits](https://github.com/apache/commons-codec/compare/rel/commons-codec-1.20.0...rel/commons-codec-1.21.0)

Updates `dnsjava:dnsjava` from 3.6.3 to 3.6.4
- [Release notes](https://github.com/dnsjava/dnsjava/releases)
- [Changelog](https://github.com/dnsjava/dnsjava/blob/master/Changelog)
- [Commits](https://github.com/dnsjava/dnsjava/compare/v3.6.3...v3.6.4)

Updates `org.springframework:spring-core` from 7.0.2 to 7.0.3
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.2...v7.0.3)

Updates `org.springframework:spring-beans` from 7.0.2 to 7.0.3
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.2...v7.0.3)

Updates `org.springframework:spring-context` from 7.0.2 to 7.0.3
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.2...v7.0.3)

Updates `org.springframework:spring-expression` from 7.0.2 to 7.0.3
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.2...v7.0.3)

Updates `org.springframework:spring-beans` from 7.0.2 to 7.0.3
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.2...v7.0.3)

Updates `org.springframework:spring-context` from 7.0.2 to 7.0.3
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.2...v7.0.3)

Updates `org.springframework:spring-expression` from 7.0.2 to 7.0.3
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.2...v7.0.3)

Updates `org.webjars.npm:codemirror__autocomplete` from 6.18.6 to 6.20.0
- [Changelog](https://github.com/codemirror/autocomplete/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codemirror/autocomplete/compare/6.18.6...6.20.0)

Updates `org.webjars.npm:codemirror__commands` from 6.8.1 to 6.10.1
- [Changelog](https://github.com/codemirror/commands/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codemirror/commands/compare/6.8.1...6.10.1)

Updates `org.webjars.npm:codemirror__language` from 6.11.3 to 6.12.1
- [Changelog](https://github.com/codemirror/language/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codemirror/language/compare/6.11.3...6.12.1)

Updates `org.webjars.npm:codemirror__state` from 6.5.2 to 6.5.4
- [Changelog](https://github.com/codemirror/state/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codemirror/state/compare/6.5.2...6.5.4)

Updates `org.webjars.npm:codemirror__view` from 6.38.1 to 6.39.11
- [Changelog](https://github.com/codemirror/view/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codemirror/view/compare/6.38.1...6.39.11)

Updates `org.webjars.npm:codemirror__lint` from 6.8.5 to 6.9.2
- [Changelog](https://github.com/codemirror/lint/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codemirror/lint/compare/6.8.5...6.9.2)

Updates `org.webjars.npm:lezer__common` from 1.2.3 to 1.5.0
- [Changelog](https://github.com/lezer-parser/common/blob/main/CHANGELOG.md)
- [Commits](https://github.com/lezer-parser/common/compare/1.2.3...1.5.0)

Updates `org.webjars.npm:lezer__lr` from 1.4.2 to 1.4.7
- [Changelog](https://github.com/lezer-parser/lr/blob/main/CHANGELOG.md)
- [Commits](https://github.com/lezer-parser/lr/compare/1.4.2...1.4.7)

Updates `org.webjars.npm:lezer__highlight` from 1.2.1 to 1.2.3
- [Changelog](https://github.com/lezer-parser/highlight/blob/main/CHANGELOG.md)
- [Commits](https://github.com/lezer-parser/highlight/compare/1.2.1...1.2.3)

Updates `org.webjars.npm:style-mod` from 4.1.2 to 4.1.3
- [Commits](https://github.com/marijnh/style-mod/compare/4.1.2...4.1.3)

Updates `at.yawk.lz4:lz4-java` from 1.10.2 to 1.10.3
- [Release notes](https://github.com/yawkat/lz4-java/releases)
- [Changelog](https://github.com/yawkat/lz4-java/blob/main/CHANGES.md)
- [Commits](https://github.com/yawkat/lz4-java/compare/v1.10.2...v1.10.3)

---
updated-dependencies:
- dependency-name: org.junit.jupiter:junit-jupiter
  dependency-version: 6.0.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.apache.groovy:groovy-bom
  dependency-version: 5.0.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: com.fasterxml.jackson:jackson-bom
  dependency-version: 2.21.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty:jetty-bom
  dependency-version: 12.0.32
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.32
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
  dependency-version: 12.0.32
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-compiler-plugin
  dependency-version: 3.15.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.codehaus.mojo:versions-maven-plugin
  dependency-version: 2.21.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.sonatype.central:central-publishing-maven-plugin
  dependency-version: 0.10.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: commons-codec:commons-codec
  dependency-version: 1.21.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: dnsjava:dnsjava
  dependency-version: 3.6.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-core
  dependency-version: 7.0.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
  dependency-version: 7.0.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
  dependency-version: 7.0.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
  dependency-version: 7.0.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
  dependency-version: 7.0.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
  dependency-version: 7.0.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
  dependency-version: 7.0.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__autocomplete
  dependency-version: 6.20.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__commands
  dependency-version: 6.10.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__language
  dependency-version: 6.12.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__state
  dependency-version: 6.5.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__view
  dependency-version: 6.39.11
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__lint
  dependency-version: 6.9.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:lezer__common
  dependency-version: 1.5.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:lezer__lr
  dependency-version: 1.4.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:lezer__highlight
  dependency-version: 1.2.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:style-mod
  dependency-version: 4.1.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: at.yawk.lz4:lz4-java
  dependency-version: 1.10.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-02-01 23:44:24 +00:00
.github
Config editor: implement autocompletion
2025-10-30 09:53:16 +09:00
commons
Bump the minor-and-patch group with 25 updates
2026-02-01 23:44:24 +00:00
contrib
Bump the minor-and-patch group with 25 updates
2026-02-01 23:44:24 +00:00
dist
[maven-release-plugin] prepare for next development iteration
2025-12-11 13:54:13 +09:00
docgen
[maven-release-plugin] prepare for next development iteration
2025-12-11 13:54:13 +09:00
docker
Update CHANGELOG.md for 3.10.1 release
2025-07-21 17:13:11 +09:00
docs
Add job deletion to API and UI
2026-01-05 14:19:17 +01:00
engine
Bump the minor-and-patch group with 25 updates
2026-02-01 23:44:24 +00:00
modules
Bump the minor-and-patch group with 12 updates
2026-01-01 23:24:56 +00:00
.gitignore
Add docgen/target to .gitignore
2025-10-30 10:21:16 +09:00
.readthedocs.yml
readthedocs: use mvn --batch-mode
2025-10-28 16:50:00 +09:00
CHANGELOG.md
Update CHANGELOG.md for #700 and #701
2025-12-11 13:38:50 +09:00
LICENSE
…
pom.xml
Bump the minor-and-patch group with 25 updates
2026-02-01 23:44:24 +00:00
README.md
…
RELEASING.md
Add RELEASING.md
2025-06-13 00:44:40 +09:00
SECURITY.md
…

README.md

Heritrix

Maven Central Docker Javadoc LICENSE

Introduction

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix (sometimes spelled heretrix, or misspelled or missaid as heratrix/heritix/heretix/heratix) is an archaic word for heiress (woman who inherits). Since our crawler seeks to collect and preserve the digital artifacts of our culture for the benefit of future researchers and generations, this name seemed apt.

Crawl Operators!

Heritrix is designed to respect the robots.txt exclusion directives and META nofollow tags. Please consider the load your crawl will place on seed sites and set politeness policies accordingly. Also, always identify your crawl with contact information in the User-Agent so sites that may be adversely affected by your crawl can contact you or adapt their server behavior accordingly.

Documentation

Developer Documentation

Latest Releases

Information about releases can be found here.

License

Heritrix is free software; you can redistribute it and/or modify it under the terms of the Apache License, Version 2.0

Some individual source code files are subject to or offered under other licenses. See the included LICENSE.txt file for more information.

Heritrix is distributed with the libraries it depends upon. The libraries can be found under the lib directory in the release distribution, and are used under the terms of their respective licenses, which are included alongside the libraries in the lib directory.

Reference in New Issue View Git Blame Copy Permalink
S
Description
No description provided
heritrixjavawarcwebcrawling
Readme
14 MiB
Languages
Java 93.3%
HTML 2.6%
Rich Text Format 1.1%
JavaScript 1%
FreeMarker 0.8%
Other 1%