Alex Osborne
208d62b79c
[maven-release-plugin] prepare for next development iteration
2026-11-19 14:15:42 +09:00
Alex Osborne
a0107e3895
[maven-release-plugin] prepare release 3.19.0
2026-11-19 14:15:34 +09:00
Alex Osborne
7eedafddc7
Merge pull request #825 from internetarchive/link-rel-preload
...
Fix navlink/embed classification of link rel types and duplicate outlinks
2026-11-18 17:46:35 +09:00
Alex Osborne
aa746df854
Schedule outlinks as embeds when a page also embeds them
...
When a page links to the same URL both as an embed and via another hop
type, only schedule the embed. Previously whichever came first in the page
was scheduled and the rest dropped as duplicates by the frontier, so e.g. in
<a href=image.jpg><img src=image.jpg></a>
the image was queued as a navlink. Likewise, until rel=preload was treated
as an embed, the stylesheet in
<link rel=preload href=style.css as=style>
<link rel=stylesheet href=style.css>
was queued as a navlink (#824 ).
2026-11-18 17:32:40 +09:00
Alex Osborne
30b88543e8
Add @codemirror/streamparser to import maps
...
@codemirror/language 6.13 and @codemirror/legacy-modes 6.5.5 now import
@codemirror/streamparser, so the editors failed to load with an
unmapped bare specifier error.
2026-11-18 17:02:28 +09:00
Alex Osborne
aa4dd213df
Merge pull request #814 from internetarchive/soft404-detector
...
Soft404Detector: detect soft 404s and optionally skip extraction
2026-11-18 16:35:14 +09:00
dependabot[bot]
e4660006ca
Bump the minor-and-patch group with 18 updates ( #822 )
...
Bumps the minor-and-patch group with 18 updates:
| Package | From | To |
| --- | --- | --- |
| [com.fasterxml.jackson:jackson-bom](https://github.com/FasterXML/jackson-bom ) | `2.22.4` | `2.23.0` |
| [org.eclipse.jetty:jetty-bom](https://github.com/jetty/jetty.project ) | `12.0.40` | `12.0.41` |
| [org.eclipse.jetty.ee10:jetty-ee10-bom](https://github.com/jetty/jetty.project ) | `12.0.40` | `12.0.41` |
| [org.codehaus.mojo:versions-maven-plugin](https://github.com/mojohaus/versions ) | `2.23.0` | `2.25.1` |
| [dnsjava:dnsjava](https://github.com/dnsjava/dnsjava ) | `3.7.0` | `3.7.1` |
| org.webjars.npm:codemirror__autocomplete | `6.20.3` | `6.20.4` |
| org.webjars.npm:codemirror__commands | `6.11.0` | `6.11.1` |
| org.webjars.npm:codemirror__language | `6.12.4` | `6.13.1` |
| org.webjars.npm:codemirror__legacy-modes | `6.5.1` | `6.5.5` |
| org.webjars.npm:codemirror__search | `6.5.11` | `6.7.2` |
| org.webjars.npm:codemirror__state | `6.7.4` | `6.7.6` |
| org.webjars.npm:codemirror__view | `6.43.11` | `6.43.15` |
| org.webjars.npm:lezer__common | `1.5.2` | `1.5.5` |
| org.webjars.npm:lezer__lr | `1.4.10` | `1.4.12` |
| org.webjars.npm:lezer__highlight | `1.2.3` | `1.2.5` |
| org.webjars.npm:style-mod | `4.1.3` | `4.1.4` |
| [com.rabbitmq:amqp-client](https://github.com/rabbitmq/rabbitmq-java-client ) | `5.37.0` | `5.37.1` |
| [at.yawk.lz4:lz4-java](https://github.com/yawkat/lz4-java ) | `1.12.1` | `1.14.0` |
Updates `com.fasterxml.jackson:jackson-bom` from 2.22.4 to 2.23.0
- [Commits](https://github.com/FasterXML/jackson-bom/compare/jackson-bom-2.22.4...jackson-bom-2.23.0 )
Updates `org.eclipse.jetty:jetty-bom` from 12.0.40 to 12.0.41
- [Release notes](https://github.com/jetty/jetty.project/releases )
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.40...jetty-12.0.41 )
Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.40 to 12.0.41
- [Release notes](https://github.com/jetty/jetty.project/releases )
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.40...jetty-12.0.41 )
Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.40 to 12.0.41
- [Release notes](https://github.com/jetty/jetty.project/releases )
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.40...jetty-12.0.41 )
Updates `org.codehaus.mojo:versions-maven-plugin` from 2.23.0 to 2.25.1
- [Release notes](https://github.com/mojohaus/versions/releases )
- [Changelog](https://github.com/mojohaus/versions/blob/master/ReleaseNotes.md )
- [Commits](https://github.com/mojohaus/versions/compare/2.23.0...2.25.1 )
Updates `dnsjava:dnsjava` from 3.7.0 to 3.7.1
- [Release notes](https://github.com/dnsjava/dnsjava/releases )
- [Changelog](https://github.com/dnsjava/dnsjava/blob/master/Changelog )
- [Commits](https://github.com/dnsjava/dnsjava/compare/v3.7.0...v3.7.1 )
Updates `org.webjars.npm:codemirror__autocomplete` from 6.20.3 to 6.20.4
Updates `org.webjars.npm:codemirror__commands` from 6.11.0 to 6.11.1
Updates `org.webjars.npm:codemirror__language` from 6.12.4 to 6.13.1
Updates `org.webjars.npm:codemirror__legacy-modes` from 6.5.1 to 6.5.5
Updates `org.webjars.npm:codemirror__search` from 6.5.11 to 6.7.2
Updates `org.webjars.npm:codemirror__state` from 6.7.4 to 6.7.6
Updates `org.webjars.npm:codemirror__view` from 6.43.11 to 6.43.15
Updates `org.webjars.npm:lezer__common` from 1.5.2 to 1.5.5
Updates `org.webjars.npm:lezer__lr` from 1.4.10 to 1.4.12
Updates `org.webjars.npm:lezer__highlight` from 1.2.3 to 1.2.5
Updates `org.webjars.npm:style-mod` from 4.1.3 to 4.1.4
Updates `com.rabbitmq:amqp-client` from 5.37.0 to 5.37.1
- [Release notes](https://github.com/rabbitmq/rabbitmq-java-client/releases )
- [Commits](https://github.com/rabbitmq/rabbitmq-java-client/compare/v5.37.0...v5.37.1 )
Updates `at.yawk.lz4:lz4-java` from 1.12.1 to 1.14.0
- [Release notes](https://github.com/yawkat/lz4-java/releases )
- [Changelog](https://github.com/yawkat/lz4-java/blob/main/CHANGES.md )
- [Commits](https://github.com/yawkat/lz4-java/compare/v1.12.1...v1.14.0 )
---
updated-dependencies:
- dependency-name: com.fasterxml.jackson:jackson-bom
dependency-version: 2.23.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty:jetty-bom
dependency-version: 12.0.41
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
dependency-version: 12.0.41
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
dependency-version: 12.0.41
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.codehaus.mojo:versions-maven-plugin
dependency-version: 2.25.1
dependency-type: direct:development
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: dnsjava:dnsjava
dependency-version: 3.7.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__autocomplete
dependency-version: 6.20.4
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__commands
dependency-version: 6.11.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__language
dependency-version: 6.13.1
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__legacy-modes
dependency-version: 6.5.5
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__search
dependency-version: 6.7.2
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__state
dependency-version: 6.7.6
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__view
dependency-version: 6.43.15
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:lezer__common
dependency-version: 1.5.5
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:lezer__lr
dependency-version: 1.4.12
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:lezer__highlight
dependency-version: 1.2.5
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:style-mod
dependency-version: 4.1.4
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: com.rabbitmq:amqp-client
dependency-version: 5.37.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: at.yawk.lz4:lz4-java
dependency-version: 1.14.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-17 15:22:08 +09:00
Alex Osborne
6f8201cc60
Comment out Soft404Detector in the default profiles
...
It's still new, so leave it opt-in until it has seen more real-world use.
2026-11-17 15:03:33 +09:00
Leslie Bellony
13c6572ad6
Merge pull request #821 from bnfleb/statisticstracker
...
ToeThread dies on `http://0.0.0.0/ ` (IllegalStateException "got suspicious value 0.0.0.0"), job stuck in EMPTY forever
2026-11-15 07:48:03 +09:00
Alex Osborne
db156f3427
Write browser behaviors inline in the Groovy profile and docs
2026-11-13 16:08:42 +09:00
Alex Osborne
fb3452a6af
Behaviors: log unparseable links through the Page instead of a constructor argument
...
ExtractLinksBehavior and PaginationBehavior took the UriErrorLoggerModule as a
constructor argument, so they had to be Spring beans to get it. That meant the
Groovy config couldn't create them inline in the behaviors list, as Spring's
Groovy reader doesn't turn inner beans in a list into beans, and had to declare
each one as a named bean and refer to it. BrowserProcessor already has the
logger module, so it now passes it to behaviors through the page and they can
be plain objects.
2026-11-13 16:08:36 +09:00
Alex Osborne
27ba6ba850
Add a commented-out BrowserProcessor bean to the Groovy default profile
2026-11-13 16:01:35 +09:00
Alex Osborne
336d0d6c52
BrowserProcessor: turn off Firefox site isolation
2026-11-13 10:12:38 +09:00
Alex Osborne
7b2a485091
BrowserProcessor: don't mistake a crashed Firefox tab for a download
2026-11-13 10:12:38 +09:00
Alex Osborne
486dffbedf
BrowserProcessor: allow per-browser concurrency, executable and options
2026-11-13 10:12:38 +09:00
Alex Osborne
d7c9a5581e
BrowserProcessor: launch browsers on other machines over ssh
2026-11-13 10:12:38 +09:00
Alex Osborne
dc070b4af1
BrowserProcessor: close idle browsers and relaunch them when a page needs one
...
Many small jobs in one JVM would otherwise each hold browsers open for as long as the job runs or sits paused.
2026-11-12 15:47:09 +09:00
Alex Osborne
51668638b0
BrowserProcessor: track each browser's state with an enum instead of a flag
2026-11-12 15:29:43 +09:00
Alex Osborne
0f8644fd65
LocalWebDriverBiDi: make the command timeout configurable
2026-11-12 13:50:20 +09:00
Alex Osborne
9cefc32df0
LocalWebDriverBiDi: kill the browser if it stops answering
2026-11-12 13:50:20 +09:00
Alex Osborne
d7f82bce4d
BrowserProcessor: launch browsers in parallel and retry failed launches with backoff
2026-11-12 13:44:03 +09:00
Alex Osborne
ad8e3fb51b
BrowserProcessor: support several browsers
2026-11-12 13:44:03 +09:00
Alex Osborne
7369ec2980
BrowserProcessor: move per-browser state into a nested Browser class
2026-11-12 13:43:55 +09:00
Alex Osborne
ec320998c2
CI: run the browser tests in their own job against Firefox and Chrome
2026-11-12 13:19:14 +09:00
Alex Osborne
74d236ee57
BrowserProcessor: reject proxy requests that don't belong to an open page
2026-11-11 22:21:44 +09:00
Alex Osborne
832279989d
Add ExtractorSVG to the Groovy default profile
2026-11-06 13:56:11 +09:00
Alex Osborne
9d79ef7372
Add ExtractorSVG and extract links from inline SVG
...
ExtractorSVG parses SVG with the JDK's StAX parser and extracts hrefs
from elements like <image>, <use> and <script> as embeds and from <a>
as links, along with <?xml-stylesheet?>, url() in presentation
attributes, <style>, <script>, animated hrefs and HTML in
<foreignObject>. Like browsers it requires the SVG namespace, decides
by the root element whether text/xml and application/xml responses
are SVG, and keeps the links found before any XML error. It's added
to the default profile before extractorHtml.
ExtractorHTML and JerichoExtractorHTML now treat hrefs on those inline
SVG elements as embeds rather than links, and extract url() from
presentation attributes like fill. ExtractorCSS now ignores
same-document references like url(#gradient).
Fixes #796
2026-11-05 16:17:28 +09:00
Alex Osborne
d614ea3b34
Soft404Detector: detect soft 404s and optionally skip extraction
...
Sites that return 200 for nonexistent URLs, combined with relative
links, can generate an endless series of fake URLs (#729 ).
The first time a server returns a successful response, a probe for a
random two-segment path on it is scheduled. If the probe gets a 200,
its content is fingerprinted, and later 200 responses from that server
with the same fingerprint are annotated "soft404". The fingerprint is a
hash of the content after removing echoes of the requested path and
replacing tokens that contain digits, so nonces and timestamps don't
prevent a match. Seeds and the root path are exempt, and a probe
redirect that keeps the probe path is treated as the probe.
With skipExtraction (the default) the rest of the fetch chain is
skipped for matching pages, so neither the extractors nor the browser
processor see them. The default profiles enable the detector with
skipExtraction=false for now, so it only annotates and counts soft
404s until it has seen more real-world use.
Per-server probe state is stored on CrawlServer.
2026-11-05 16:12:17 +09:00
Alex Osborne
7a9e2689b4
Use CrawlURI.hasContentType() for exact media type checks
...
Replaces startsWith() prefix checks, which also matched longer subtypes,
e.g. "application/vnd.wap.wml" matched the binary
"application/vnd.wap.wmlc". The truncated "application/xhtml" and
"application/vnd.wap.xhtml" checks now name the full "+xml" types. Also
fixes case-sensitive comparisons in ExtractorPDF and ExtractorPDFContent,
and drops BotBlockDetector's private equivalent.
Substring heuristics like ExtractorJS's indexOf("javascript") and
ExtractorXML's indexOf("xml") are intentionally left as they are.
2026-11-05 09:58:58 +09:00
Alex Osborne
5b1f492cc2
CrawlURI: add hasContentType() for media type comparison
...
Compares the content type's type/subtype with the given media type,
ignoring case and any parameters, so "Text/HTML; charset=utf-8" matches
"text/html" but "text/htmlfoo" doesn't. This replaces ad hoc checks like
toLowerCase().startsWith(...), which also match longer subtypes.
2026-11-05 09:53:25 +09:00
Matt Van Horn
a769651bf5
fix: give midfetch aborts their own fetch status code
...
Add S_MIDFETCH_ABORTED (-5005) and set it in FetchHTTP when
shouldFetchBodyRule rejects a response, in place of S_OUT_OF_SCOPE.
The code means something distinct from a scope rejection, so crawl logs
and reports can now tell the two apart. AbstractFrontier.isDisregarded()
treats it as disregarded, CrawlURI gives it a readable string and the
glossary lists it.
2026-11-02 06:01:06 -08:00
dependabot[bot]
ce13a99f71
Bump the minor-and-patch group with 23 updates ( #791 )
...
Bumps the minor-and-patch group with 23 updates:
| Package | From | To |
| --- | --- | --- |
| [com.fasterxml.jackson:jackson-bom](https://github.com/FasterXML/jackson-bom ) | `2.22.3` | `2.22.4` |
| [org.eclipse.jetty:jetty-bom](https://github.com/jetty/jetty.project ) | `12.0.39` | `12.0.40` |
| [org.eclipse.jetty.ee10:jetty-ee10-bom](https://github.com/jetty/jetty.project ) | `12.0.39` | `12.0.40` |
| [org.apache.maven.plugins:maven-surefire-plugin](https://github.com/apache/maven-surefire ) | `3.6.0` | `3.6.1` |
| [org.codehaus.mojo:versions-maven-plugin](https://github.com/mojohaus/versions ) | `2.22.0` | `2.23.0` |
| [org.sonatype.central:central-publishing-maven-plugin](https://github.com/sonatype-nexus-community/central-publishing-maven-plugin ) | `0.11.0` | `0.12.0` |
| [commons-codec:commons-codec](https://github.com/apache/commons-codec ) | `1.22.1` | `1.23.0` |
| [dnsjava:dnsjava](https://github.com/dnsjava/dnsjava ) | `3.6.5` | `3.7.0` |
| [org.springframework:spring-core](https://github.com/spring-projects/spring-framework ) | `7.0.9` | `7.0.10` |
| [org.springframework:spring-beans](https://github.com/spring-projects/spring-framework ) | `7.0.9` | `7.0.10` |
| [org.springframework:spring-context](https://github.com/spring-projects/spring-framework ) | `7.0.9` | `7.0.10` |
| [org.springframework:spring-expression](https://github.com/spring-projects/spring-framework ) | `7.0.9` | `7.0.10` |
| [com.esotericsoftware:kryo](https://github.com/EsotericSoftware/kryo ) | `5.6.2` | `5.7.1` |
| [com.github.mwiede:jsch](https://github.com/mwiede/jsch ) | `2.28.7` | `2.28.8` |
| [org.apache.mina:mina-core](https://github.com/apache/mina ) | `2.2.9` | `2.2.10` |
| [org.restlet:org.restlet](https://github.com/restlet/restlet-framework-java ) | `2.6.0` | `2.6.1` |
| org.restlet:org.restlet.ext.jetty | `2.6.0` | `2.6.1` |
| [org.restlet:org.restlet.ext.xml](https://github.com/restlet/restlet-framework-java ) | `2.6.0` | `2.6.1` |
| [org.restlet:org.restlet.ext.json](https://github.com/restlet/restlet-framework-java ) | `2.6.0` | `2.6.1` |
| [org.restlet:org.restlet.ext.crypto](https://github.com/restlet/restlet-framework-java ) | `2.6.0` | `2.6.1` |
| org.apache.kafka:kafka-clients | `4.3.1` | `4.4.0` |
| [at.yawk.lz4:lz4-java](https://github.com/yawkat/lz4-java ) | `1.12.0` | `1.12.1` |
| [com.google.code.gson:gson](https://github.com/google/gson ) | `2.14.0` | `2.15.0` |
Updates `com.fasterxml.jackson:jackson-bom` from 2.22.3 to 2.22.4
- [Commits](https://github.com/FasterXML/jackson-bom/compare/jackson-bom-2.22.3...jackson-bom-2.22.4 )
Updates `org.eclipse.jetty:jetty-bom` from 12.0.39 to 12.0.40
- [Release notes](https://github.com/jetty/jetty.project/releases )
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.39...jetty-12.0.40 )
Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.39 to 12.0.40
- [Release notes](https://github.com/jetty/jetty.project/releases )
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.39...jetty-12.0.40 )
Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.39 to 12.0.40
- [Release notes](https://github.com/jetty/jetty.project/releases )
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.39...jetty-12.0.40 )
Updates `org.apache.maven.plugins:maven-surefire-plugin` from 3.6.0 to 3.6.1
- [Release notes](https://github.com/apache/maven-surefire/releases )
- [Commits](https://github.com/apache/maven-surefire/compare/surefire-3.6.0...surefire-3.6.1 )
Updates `org.codehaus.mojo:versions-maven-plugin` from 2.22.0 to 2.23.0
- [Release notes](https://github.com/mojohaus/versions/releases )
- [Changelog](https://github.com/mojohaus/versions/blob/master/ReleaseNotes.md )
- [Commits](https://github.com/mojohaus/versions/compare/2.22.0...2.23.0 )
Updates `org.sonatype.central:central-publishing-maven-plugin` from 0.11.0 to 0.12.0
- [Release notes](https://github.com/sonatype-nexus-community/central-publishing-maven-plugin/releases )
- [Commits](https://github.com/sonatype-nexus-community/central-publishing-maven-plugin/commits/release-0.12.0 )
Updates `commons-codec:commons-codec` from 1.22.1 to 1.23.0
- [Changelog](https://github.com/apache/commons-codec/blob/master/RELEASE-NOTES.txt )
- [Commits](https://github.com/apache/commons-codec/compare/rel/commons-codec-1.22.1...rel/commons-codec-1.23.0 )
Updates `dnsjava:dnsjava` from 3.6.5 to 3.7.0
- [Release notes](https://github.com/dnsjava/dnsjava/releases )
- [Changelog](https://github.com/dnsjava/dnsjava/blob/master/Changelog )
- [Commits](https://github.com/dnsjava/dnsjava/commits/v3.7.0 )
Updates `org.springframework:spring-core` from 7.0.9 to 7.0.10
- [Release notes](https://github.com/spring-projects/spring-framework/releases )
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.9...v7.0.10 )
Updates `org.springframework:spring-beans` from 7.0.9 to 7.0.10
- [Release notes](https://github.com/spring-projects/spring-framework/releases )
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.9...v7.0.10 )
Updates `org.springframework:spring-context` from 7.0.9 to 7.0.10
- [Release notes](https://github.com/spring-projects/spring-framework/releases )
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.9...v7.0.10 )
Updates `org.springframework:spring-expression` from 7.0.9 to 7.0.10
- [Release notes](https://github.com/spring-projects/spring-framework/releases )
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.9...v7.0.10 )
Updates `org.springframework:spring-beans` from 7.0.9 to 7.0.10
- [Release notes](https://github.com/spring-projects/spring-framework/releases )
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.9...v7.0.10 )
Updates `org.springframework:spring-context` from 7.0.9 to 7.0.10
- [Release notes](https://github.com/spring-projects/spring-framework/releases )
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.9...v7.0.10 )
Updates `org.springframework:spring-expression` from 7.0.9 to 7.0.10
- [Release notes](https://github.com/spring-projects/spring-framework/releases )
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.9...v7.0.10 )
Updates `com.esotericsoftware:kryo` from 5.6.2 to 5.7.1
- [Release notes](https://github.com/EsotericSoftware/kryo/releases )
- [Commits](https://github.com/EsotericSoftware/kryo/compare/kryo-parent-5.6.2...kryo-parent-5.7.1 )
Updates `com.github.mwiede:jsch` from 2.28.7 to 2.28.8
- [Release notes](https://github.com/mwiede/jsch/releases )
- [Changelog](https://github.com/mwiede/jsch/blob/master/ChangeLog )
- [Commits](https://github.com/mwiede/jsch/compare/jsch-2.28.7...jsch-2.28.8 )
Updates `org.apache.mina:mina-core` from 2.2.9 to 2.2.10
- [Commits](https://github.com/apache/mina/compare/2.2.9...2.2.10 )
Updates `org.restlet:org.restlet` from 2.6.0 to 2.6.1
- [Changelog](https://github.com/restlet/restlet-framework-java/blob/2.6.1/changes.md )
- [Commits](https://github.com/restlet/restlet-framework-java/compare/2.6.0...2.6.1 )
Updates `org.restlet:org.restlet.ext.jetty` from 2.6.0 to 2.6.1
Updates `org.restlet:org.restlet.ext.xml` from 2.6.0 to 2.6.1
- [Changelog](https://github.com/restlet/restlet-framework-java/blob/2.6.1/changes.md )
- [Commits](https://github.com/restlet/restlet-framework-java/compare/2.6.0...2.6.1 )
Updates `org.restlet:org.restlet.ext.json` from 2.6.0 to 2.6.1
- [Changelog](https://github.com/restlet/restlet-framework-java/blob/2.6.1/changes.md )
- [Commits](https://github.com/restlet/restlet-framework-java/compare/2.6.0...2.6.1 )
Updates `org.restlet:org.restlet.ext.crypto` from 2.6.0 to 2.6.1
- [Changelog](https://github.com/restlet/restlet-framework-java/blob/2.6.1/changes.md )
- [Commits](https://github.com/restlet/restlet-framework-java/compare/2.6.0...2.6.1 )
Updates `org.restlet:org.restlet.ext.xml` from 2.6.0 to 2.6.1
- [Changelog](https://github.com/restlet/restlet-framework-java/blob/2.6.1/changes.md )
- [Commits](https://github.com/restlet/restlet-framework-java/compare/2.6.0...2.6.1 )
Updates `org.restlet:org.restlet.ext.json` from 2.6.0 to 2.6.1
- [Changelog](https://github.com/restlet/restlet-framework-java/blob/2.6.1/changes.md )
- [Commits](https://github.com/restlet/restlet-framework-java/compare/2.6.0...2.6.1 )
Updates `org.restlet:org.restlet.ext.crypto` from 2.6.0 to 2.6.1
- [Changelog](https://github.com/restlet/restlet-framework-java/blob/2.6.1/changes.md )
- [Commits](https://github.com/restlet/restlet-framework-java/compare/2.6.0...2.6.1 )
Updates `org.apache.kafka:kafka-clients` from 4.3.1 to 4.4.0
Updates `at.yawk.lz4:lz4-java` from 1.12.0 to 1.12.1
- [Release notes](https://github.com/yawkat/lz4-java/releases )
- [Changelog](https://github.com/yawkat/lz4-java/blob/main/CHANGES.md )
- [Commits](https://github.com/yawkat/lz4-java/compare/v1.12.0...v1.12.1 )
Updates `com.google.code.gson:gson` from 2.14.0 to 2.15.0
- [Release notes](https://github.com/google/gson/releases )
- [Changelog](https://github.com/google/gson/blob/main/CHANGELOG.md )
- [Commits](https://github.com/google/gson/compare/gson-parent-2.14.0...gson-parent-2.15.0 )
---
updated-dependencies:
- dependency-name: com.fasterxml.jackson:jackson-bom
dependency-version: 2.22.4
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty:jetty-bom
dependency-version: 12.0.40
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
dependency-version: 12.0.40
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
dependency-version: 12.0.40
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-surefire-plugin
dependency-version: 3.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.codehaus.mojo:versions-maven-plugin
dependency-version: 2.23.0
dependency-type: direct:development
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: org.sonatype.central:central-publishing-maven-plugin
dependency-version: 0.12.0
dependency-type: direct:development
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: commons-codec:commons-codec
dependency-version: 1.23.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: dnsjava:dnsjava
dependency-version: 3.7.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-core
dependency-version: 7.0.10
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
dependency-version: 7.0.10
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
dependency-version: 7.0.10
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
dependency-version: 7.0.10
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
dependency-version: 7.0.10
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
dependency-version: 7.0.10
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
dependency-version: 7.0.10
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: com.esotericsoftware:kryo
dependency-version: 5.7.1
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: com.github.mwiede:jsch
dependency-version: 2.28.8
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.apache.mina:mina-core
dependency-version: 2.2.10
dependency-type: direct:development
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.restlet:org.restlet
dependency-version: 2.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.restlet:org.restlet.ext.jetty
dependency-version: 2.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.restlet:org.restlet.ext.xml
dependency-version: 2.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.restlet:org.restlet.ext.json
dependency-version: 2.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.restlet:org.restlet.ext.crypto
dependency-version: 2.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.restlet:org.restlet.ext.xml
dependency-version: 2.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.restlet:org.restlet.ext.json
dependency-version: 2.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.restlet:org.restlet.ext.crypto
dependency-version: 2.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.apache.kafka:kafka-clients
dependency-version: 4.4.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: at.yawk.lz4:lz4-java
dependency-version: 1.12.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: com.google.code.gson:gson
dependency-version: 2.15.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 16:52:47 +09:00
Leslie Bellony
91a869bd4b
sourcetags: dispose of the sorted map after use ( #788 )
2026-10-12 18:34:12 +09:00
Alex Osborne
94a6761a17
BrowserProcessorTest: make tempDir static ( #785 )
...
We're seeing occasional random test failures. I think it may be because we're changing the CrawlController scratchDir before each test, which means the profile of a running browser can get wiped out.
2026-10-08 18:36:09 +09:00
Alex Osborne
8bb0a65349
Merge pull request #783 from internetarchive/toethread-always-end-disposition
...
Always release the disposition lock when a ToeThread exits
2026-10-07 12:42:19 +09:00
Alex Osborne
d6747686e3
Don't let browsers cache job directory files for 10 minutes ( #782 )
...
Restlet's Directory gives every file an Expires header 10 minutes in the
future. Job files (logs, reports, crawler-beans.cxml) change constantly, so
browsers could show stale copies. Worst case, the config editor loads a
cached crawler-beans.cxml and saving it silently reverts newer changes.
Clear the expiration date and send Cache-Control: no-cache instead, so
browsers revalidate against Last-Modified on each use.
2026-10-07 12:42:02 +09:00
Alex Osborne
be6b49fa6c
[maven-release-plugin] prepare for next development iteration
2026-10-06 12:39:19 +09:00
Alex Osborne
eac2c96daf
[maven-release-plugin] prepare release 3.18.0
2026-10-06 12:39:12 +09:00
Alex Osborne
7939d0d7e1
Reject cross-site requests to the web UI
...
Add CrossSiteRequestFilter in front of the web UI's router to protect
against cross-site request forgery. POST, PUT and DELETE requests are
rejected with 403 when the browser's Sec-Fetch-Site header says they
came from another site.
Sec-Fetch-Site is set by the browser from its own view of the page and
target URL, so unlike comparing Origin with Host it isn't affected by
reverse proxies rewriting the host or scheme. Requests without the
header, such as from curl and other API clients, are allowed.
Origins listed in the heritrix.trustedOrigins system property (comma
separated) are allowed to send cross-site requests.
2026-10-06 12:21:40 +09:00
Alex Osborne
55bef0aa32
Merge pull request #780 from internetarchive/paged-log-viewer-search
...
Crawl log search
2026-10-06 12:09:09 +09:00
Alex Osborne
6f9a82c8fb
Fix HTML escaping in templates
...
Enables autoescaping in Freemarker and adds some missing escaping
where we generate HTML in Java methods.
2026-10-06 12:04:23 +09:00
Alex Osborne
a09f38570d
Only write heritrix_dmesg.log when requested ( #779 )
...
Only write heritrix_dmesg.log when requested
Heritrix.java always created heritrix_dmesg.log in $HERITRIX_HOME, even
though only the bin/heritrix script reads it, and only when it starts
Heritrix in the background. This got in the way of making
$HERITRIX_HOME read-only for Docker and systemd setups.
The startup log is now written only when the heritrix.dmesg system
property gives its path. bin/heritrix sets this property in background
mode and keeps the file in the same place as before.
Also fix a typo in bin/heritrix ($startmessage) that stopped it
removing a heritrix_dmesg.log left over from a previous start.
Set -Dheritrix.dmesg in heritrix.cmd too.
2026-10-06 11:29:51 +09:00
Alex Osborne
0ca9e5dc2a
Merge pull request #778 from internetarchive/ato/botblock-detector2
...
BotBlockDetector: add additional services and enable by default
2026-10-06 11:20:02 +09:00
Alex Osborne and Claude Opus 5.5
f5070f6203
Paged log viewer: search and filter with a small query language
...
The paged log viewer gains a search box taking a search-engine style
query, passed as the q parameter:
status:4xx host:example.com -type:image size>1MB duration>10s
- Terms are ANDed; '-' negates; double quotes allow spaces.
- 'field:value' is a case-insensitive substring match with '*'
wildcards, 'field=value' is exact and 'field~regex' is a Java regex.
Numeric fields take the same values with ':' or '=', and no regex;
a comparison needs no operator, as in size>1MB.
- Bare words and the 'line' field search the whole line in any log.
crawl.log also supports status (404, 4xx, <0, 500..599, 404,410),
size (>1MB, 100KB..10MB; units are powers of 1024), duration (>10s,
500..2000 in ms), depth (hops from the seed: 0, >5), hops (the hop
path, e.g. hops:*E), url, host (includes subdomains), via and viahost
(the URI it was discovered from, and its host), type and annot.
- Bare words and line terms are highlighted in the results.
Filtering keeps the viewer's stateless, byte-position paging and works
across checkpoint logs: each request scans forward or backward from pos
until it has enough matches or reaches the start or end, streaming a
progress bar to the browser. A scan stops after 5 minutes, offering a
link to continue from where it stopped, or as soon as the client
disconnects. Regex matching is aborted at the same deadline to guard against
catastrophic backtracking. The query is carried through all navigation links.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-02 16:54:19 +09:00
Alex Osborne and Claude Opus 5.5
b1a36159ae
Paged log viewer: view a log together with its checkpoint generations
...
Checkpointing renames each log in place (crawl.log becomes
crawl.log.cp00001-<timestamp>) and starts a new one, so earlier entries
could only be viewed one file at a time. The paged viewer now offers an
"include N checkpoint logs" link (all=y) that shows the log and its
rotated generations in the current launch as one virtual file, oldest
first, with each run of lines labelled with its file.
Positions in the virtual file stay valid across checkpoints, since a
rotated file keeps its place and the new log continues after it, and
they are converted when toggling so the view stays where it was.
Gzipped generations can't be read at arbitrary positions and are
left out.
LogSeries opens the files and fixes their lengths up front, so a
rotation mid-request doesn't change what is read. Paging uses a new
FilteredLineScanner over a LogSeries instead of FileUtils.pagedLines,
which can only read a single file. Results are the same for every
position the navigation links generate; a hand-edited pos in the
middle of a line now starts at that line rather than the next one.
Lines are capped at 256 KiB so a file with few or no newlines can't
exhaust the heap.
Also fixes in the crawl.log rendering: a rotated crawl.log gets the
same status highlighting, status codes unknown to Jetty (e.g. 999) and
numbers too long to be a status no longer fail the page, and the file
name, status titles and navigation links are HTML escaped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-02 16:46:36 +09:00
Alex Osborne
838523d33a
BotBlockDetector: enable in default profiles
2026-10-02 09:24:44 +09:00
dependabot[bot]
c36460a0f3
Bump the minor-and-patch group with 13 updates
...
Bumps the minor-and-patch group with 13 updates:
| Package | From | To |
| --- | --- | --- |
| [org.eclipse.jetty:jetty-bom](https://github.com/jetty/jetty.project ) | `12.0.38` | `12.0.39` |
| [org.eclipse.jetty.ee10:jetty-ee10-bom](https://github.com/jetty/jetty.project ) | `12.0.38` | `12.0.39` |
| [org.slf4j:slf4j-bom](https://github.com/qos-ch/slf4j ) | `2.0.18` | `2.0.20` |
| [org.apache.maven.plugins:maven-compiler-plugin](https://github.com/apache/maven-compiler-plugin ) | `3.15.0` | `3.16.0` |
| [org.apache.maven.plugins:maven-surefire-plugin](https://github.com/apache/maven-surefire ) | `3.5.6` | `3.6.0` |
| [org.codehaus.mojo:versions-maven-plugin](https://github.com/mojohaus/versions ) | `2.21.0` | `2.22.0` |
| [org.webjars:webjars-locator-lite](https://github.com/webjars/webjars-locator-lite ) | `1.1.4` | `1.1.5` |
| org.webjars.npm:codemirror__commands | `6.10.4` | `6.11.0` |
| org.webjars.npm:codemirror__state | `6.7.1` | `6.7.4` |
| org.webjars.npm:codemirror__view | `6.43.7` | `6.43.11` |
| org.webjars.npm:marijn__find-cluster-break | `1.0.3` | `1.0.4` |
| [com.rabbitmq:amqp-client](https://github.com/rabbitmq/rabbitmq-java-client ) | `5.35.0` | `5.36.0` |
| [at.yawk.lz4:lz4-java](https://github.com/yawkat/lz4-java ) | `1.11.2` | `1.12.0` |
Updates `org.eclipse.jetty:jetty-bom` from 12.0.38 to 12.0.39
- [Release notes](https://github.com/jetty/jetty.project/releases )
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.38...jetty-12.0.39 )
Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.38 to 12.0.39
- [Release notes](https://github.com/jetty/jetty.project/releases )
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.38...jetty-12.0.39 )
Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.38 to 12.0.39
- [Release notes](https://github.com/jetty/jetty.project/releases )
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.38...jetty-12.0.39 )
Updates `org.slf4j:slf4j-bom` from 2.0.18 to 2.0.20
- [Release notes](https://github.com/qos-ch/slf4j/releases )
- [Commits](https://github.com/qos-ch/slf4j/compare/v_2.0.18...v_2.0.20 )
Updates `org.apache.maven.plugins:maven-compiler-plugin` from 3.15.0 to 3.16.0
- [Release notes](https://github.com/apache/maven-compiler-plugin/releases )
- [Commits](https://github.com/apache/maven-compiler-plugin/compare/maven-compiler-plugin-3.15.0...maven-compiler-plugin-3.16.0 )
Updates `org.apache.maven.plugins:maven-surefire-plugin` from 3.5.6 to 3.6.0
- [Release notes](https://github.com/apache/maven-surefire/releases )
- [Commits](https://github.com/apache/maven-surefire/compare/surefire-3.5.6...surefire-3.6.0 )
Updates `org.codehaus.mojo:versions-maven-plugin` from 2.21.0 to 2.22.0
- [Release notes](https://github.com/mojohaus/versions/releases )
- [Changelog](https://github.com/mojohaus/versions/blob/master/ReleaseNotes.md )
- [Commits](https://github.com/mojohaus/versions/compare/2.21.0...2.22.0 )
Updates `org.webjars:webjars-locator-lite` from 1.1.4 to 1.1.5
- [Release notes](https://github.com/webjars/webjars-locator-lite/releases )
- [Commits](https://github.com/webjars/webjars-locator-lite/compare/webjars-locator-lite-1.1.4...webjars-locator-lite-1.1.5 )
Updates `org.webjars.npm:codemirror__commands` from 6.10.4 to 6.11.0
Updates `org.webjars.npm:codemirror__state` from 6.7.1 to 6.7.4
Updates `org.webjars.npm:codemirror__view` from 6.43.7 to 6.43.11
Updates `org.webjars.npm:marijn__find-cluster-break` from 1.0.3 to 1.0.4
Updates `com.rabbitmq:amqp-client` from 5.35.0 to 5.36.0
- [Release notes](https://github.com/rabbitmq/rabbitmq-java-client/releases )
- [Commits](https://github.com/rabbitmq/rabbitmq-java-client/compare/v5.35.0...v5.36.0 )
Updates `at.yawk.lz4:lz4-java` from 1.11.2 to 1.12.0
- [Release notes](https://github.com/yawkat/lz4-java/releases )
- [Changelog](https://github.com/yawkat/lz4-java/blob/main/CHANGES.md )
- [Commits](https://github.com/yawkat/lz4-java/compare/v1.11.2...v1.12.0 )
---
updated-dependencies:
- dependency-name: org.eclipse.jetty:jetty-bom
dependency-version: 12.0.39
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
dependency-version: 12.0.39
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
dependency-version: 12.0.39
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.slf4j:slf4j-bom
dependency-version: 2.0.20
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-compiler-plugin
dependency-version: 3.16.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-surefire-plugin
dependency-version: 3.6.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: org.codehaus.mojo:versions-maven-plugin
dependency-version: 2.22.0
dependency-type: direct:development
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: org.webjars:webjars-locator-lite
dependency-version: 1.1.5
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__commands
dependency-version: 6.11.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__state
dependency-version: 6.7.4
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__view
dependency-version: 6.43.11
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:marijn__find-cluster-break
dependency-version: 1.0.4
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: com.rabbitmq:amqp-client
dependency-version: 5.36.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: at.yawk.lz4:lz4-java
dependency-version: 1.12.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
...
Signed-off-by: dependabot[bot] <support@github.com >
2026-10-02 08:51:34 +09:00
Alex Osborne
f6a14ac1f6
Implement invoker commands for older browsers like Firefox ESR
...
Fixes #773
2026-10-01 19:05:53 +09:00
Alex Osborne
0d3582a136
[maven-release-plugin] prepare for next development iteration
2026-09-10 17:29:57 +09:00
Alex Osborne
48f9c37f83
[maven-release-plugin] prepare release 3.17.1
2026-09-10 17:29:50 +09:00