Alex Osborne
208d62b79c
[maven-release-plugin] prepare for next development iteration
2026-11-19 14:15:42 +09:00
Alex Osborne
a0107e3895
[maven-release-plugin] prepare release 3.19.0
2026-11-19 14:15:34 +09:00
Alex Osborne
c05bf8194f
Treat link rel=prefetch and rel=prerender as navlinks
...
These are generally used for likely navigation targets rather than resources the current page needs.
2026-11-18 17:21:25 +09:00
Alex Osborne
48a3a6cb20
Treat link rel=preload as an embed
...
Fixes #824
2026-11-18 17:19:56 +09:00
Alex Osborne
aa4dd213df
Merge pull request #814 from internetarchive/soft404-detector
...
Soft404Detector: detect soft 404s and optionally skip extraction
2026-11-18 16:35:14 +09:00
Alex Osborne
b7ce8f9782
Fix ExtractorJSTest for commons-lang3 3.22.0
...
The webarchive-commons 3.0.6 bump pulls in commons-lang3 3.22.0, whose
UnicodeUnescaper no longer throws on malformed \u escapes. Only test
the invalid code points ExtractorJS rejects itself.
2026-11-17 15:32:33 +09:00
Alex Osborne
a8cd8190ea
Soft404Detector: count a redirected probe as its target's probe
...
Otherwise the target server scheduled the same probe again.
2026-11-17 14:36:59 +09:00
Alex Osborne
ee7f2c694c
Soft404Detector: also remove echoes without a trailing slash
...
A canonical link to /a/b on the page /a/b/ prevented a match.
2026-11-17 14:36:52 +09:00
Alex Osborne
c3d062bee2
Soft404Detector: only probe servers reached by seeds or links
...
Servers reached by a redirect or speculative link, such as other sites a
crawled site redirects to, got a probe too.
2026-11-17 14:36:46 +09:00
Alex Osborne
82e1c5ad59
Soft404Detector: drop the content length check
...
It compared raw lengths, but echoes of the probe path and the page's own
path make those differ, so pages that repeat their path often were
missed even with matching fingerprints.
2026-11-17 14:36:38 +09:00
Alex Osborne
e1c61d3fe8
Soft404Detector: only collapse URL-like tokens containing digits
...
Tokens were runs of anything but whitespace, quotes and angle brackets,
so a meta refresh's "0;url=..." or a paragraph of CJK text containing
a digit collapsed to "0", making unrelated pages match.
2026-11-17 14:36:32 +09:00
Alex Osborne
fb3452a6af
Behaviors: log unparseable links through the Page instead of a constructor argument
...
ExtractLinksBehavior and PaginationBehavior took the UriErrorLoggerModule as a
constructor argument, so they had to be Spring beans to get it. That meant the
Groovy config couldn't create them inline in the behaviors list, as Spring's
Groovy reader doesn't turn inner beans in a list into beans, and had to declare
each one as a named bean and refer to it. BrowserProcessor already has the
logger module, so it now passes it to behaviors through the page and they can
be plain objects.
2026-11-13 16:08:36 +09:00
Alex Osborne
dec91bca04
Merge pull request #803 from RaphaelFakhri/fix-alt-svc-default-max-age
...
FetchHTTP2: Default Alt-Svc max age to 24 hours
2026-11-07 09:04:07 +09:00
Raphael Fakhri
1a120ea3c2
FetchHTTP2Test: Check the Alt-Svc expiry instead of sleeping
2026-11-06 20:36:21 +00:00
Alex Osborne
5838dbf1f9
ExtractorSVG: obey ExtractorHTML obeyRelNofollow setting
2026-11-05 16:17:28 +09:00
Alex Osborne
9d79ef7372
Add ExtractorSVG and extract links from inline SVG
...
ExtractorSVG parses SVG with the JDK's StAX parser and extracts hrefs
from elements like <image>, <use> and <script> as embeds and from <a>
as links, along with <?xml-stylesheet?>, url() in presentation
attributes, <style>, <script>, animated hrefs and HTML in
<foreignObject>. Like browsers it requires the SVG namespace, decides
by the root element whether text/xml and application/xml responses
are SVG, and keeps the links found before any XML error. It's added
to the default profile before extractorHtml.
ExtractorHTML and JerichoExtractorHTML now treat hrefs on those inline
SVG elements as embeds rather than links, and extract url() from
presentation attributes like fill. ExtractorCSS now ignores
same-document references like url(#gradient).
Fixes #796
2026-11-05 16:17:28 +09:00
Alex Osborne
a1f7103d12
Soft404Detector: tighten probe handling and only check HTML
...
- Generate one probe path per crawl (saved in checkpoints) and only
treat URLs ending in it as the probe, so relative links from the
probe page aren't mistaken for probes.
- Never extract links from the probe itself unless it's a redirect.
- Only check HTML responses, using CrawlURI.hasContentType(). Minified
JSON could otherwise normalize to the same fingerprint.
- Schedule the probe with an inferred hop rather than a prerequisite
hop, so the crawl scope's reject rules apply to it, and don't probe
servers reached via an embed.
- Store the fingerprint as an immutable record and keep the first one
recorded, so a probe redirected to another server doesn't overwrite
it or count that server twice.
2026-11-05 16:12:17 +09:00
Alex Osborne
d614ea3b34
Soft404Detector: detect soft 404s and optionally skip extraction
...
Sites that return 200 for nonexistent URLs, combined with relative
links, can generate an endless series of fake URLs (#729 ).
The first time a server returns a successful response, a probe for a
random two-segment path on it is scheduled. If the probe gets a 200,
its content is fingerprinted, and later 200 responses from that server
with the same fingerprint are annotated "soft404". The fingerprint is a
hash of the content after removing echoes of the requested path and
replacing tokens that contain digits, so nonces and timestamps don't
prevent a match. Seeds and the root path are exempt, and a probe
redirect that keeps the probe path is treated as the probe.
With skipExtraction (the default) the rest of the fetch chain is
skipped for matching pages, so neither the extractors nor the browser
processor see them. The default profiles enable the detector with
skipExtraction=false for now, so it only annotates and counts soft
404s until it has seen more real-world use.
Per-server probe state is stored on CrawlServer.
2026-11-05 16:12:17 +09:00
Alex Osborne
7a9e2689b4
Use CrawlURI.hasContentType() for exact media type checks
...
Replaces startsWith() prefix checks, which also matched longer subtypes,
e.g. "application/vnd.wap.wml" matched the binary
"application/vnd.wap.wmlc". The truncated "application/xhtml" and
"application/vnd.wap.xhtml" checks now name the full "+xml" types. Also
fixes case-sensitive comparisons in ExtractorPDF and ExtractorPDFContent,
and drops BotBlockDetector's private equivalent.
Substring heuristics like ExtractorJS's indexOf("javascript") and
ExtractorXML's indexOf("xml") are intentionally left as they are.
2026-11-05 09:58:58 +09:00
Alex Osborne
5b1f492cc2
CrawlURI: add hasContentType() for media type comparison
...
Compares the content type's type/subtype with the given media type,
ignoring case and any parameters, so "Text/HTML; charset=utf-8" matches
"text/html" but "text/htmlfoo" doesn't. This replaces ad hoc checks like
toLowerCase().startsWith(...), which also match longer subtypes.
2026-11-05 09:53:25 +09:00
Alex Osborne
a6f69a4473
Merge pull request #808 from internetarchive/extractorjs-es-module-imports
...
ExtractorJS: resolve ES module imports relative to script's URL
2026-11-04 16:41:13 +09:00
Alex Osborne
b8c22606b6
ExtractorHTML: extract links from iframe srcdoc
2026-11-04 15:16:19 +09:00
Alex Osborne
889a6b0082
JerichoExtractorHTML: parse sequentially to ignore tags inside attribute values
2026-11-04 15:10:48 +09:00
Alex Osborne
65621b48d4
ExtractorHTML: don't end tags at '>' inside quoted attribute values
...
Previously a '>' inside a quoted attribute value, e.g.
onclick="if (a > b) ..." or an iframe srcdoc, ended the tag early and
any attributes after it were lost. Quotes are now only treated as value
delimiters directly after '=', as in browsers.
2026-11-04 15:08:46 +09:00
Alex Osborne
91d41c2084
ExtractorHTML: extract formaction attribute
...
Only extracted when formmethod=get or extractOnlyFormGets is disabled,
since without formmethod the form's own method applies and would
require tracking state between tags.
2026-11-04 14:54:34 +09:00
Alex Osborne
a356bc477d
ExtractorHTML: extract link imagesrcset attribute as embed
2026-11-04 14:43:12 +09:00
Alex Osborne
a76e5221cc
ExtractorHTML: extract poster attribute as embed
2026-11-04 14:40:36 +09:00
Raphael Fakhri
4cfc68cda3
FetchHTTP2: Default Alt-Svc max age to 24 hours
...
An Alt-Svc header without an ma parameter set the HTTP/3 expiry to
the current time, so the alternative was discarded straight away and
HTTP/3 was never used for that server. RFC 7838 defines the default
freshness lifetime as 24 hours, which the code comment already
states.
2026-11-03 13:54:10 +00:00
Matt Van Horn
a769651bf5
fix: give midfetch aborts their own fetch status code
...
Add S_MIDFETCH_ABORTED (-5005) and set it in FetchHTTP when
shouldFetchBodyRule rejects a response, in place of S_OUT_OF_SCOPE.
The code means something distinct from a scope rejection, so crawl logs
and reports can now tell the two apart. AbstractFrontier.isDisregarded()
treats it as disregarded, CrawlURI gives it a readable string and the
glossary lists it.
2026-11-02 06:01:06 -08:00
Matt Van Horn
1c27e2e7b0
fix: disregard HTTP responses rejected before body fetching
...
A rejection by shouldFetchBodyRule now sets S_OUT_OF_SCOPE, causing the
frontier to disregard the URI without charging the queue budget or
retrying it, and skipping extraction and WARC writing. Tests cover
rejected responses and preserve successful handling for accepted or
disabled rules, prerequisites, size limits, and timeouts.
Fixes #455
2026-11-02 00:32:19 -08:00
dependabot[bot]
ce13a99f71
Bump the minor-and-patch group with 23 updates ( #791 )
...
Bumps the minor-and-patch group with 23 updates:
| Package | From | To |
| --- | --- | --- |
| [com.fasterxml.jackson:jackson-bom](https://github.com/FasterXML/jackson-bom ) | `2.22.3` | `2.22.4` |
| [org.eclipse.jetty:jetty-bom](https://github.com/jetty/jetty.project ) | `12.0.39` | `12.0.40` |
| [org.eclipse.jetty.ee10:jetty-ee10-bom](https://github.com/jetty/jetty.project ) | `12.0.39` | `12.0.40` |
| [org.apache.maven.plugins:maven-surefire-plugin](https://github.com/apache/maven-surefire ) | `3.6.0` | `3.6.1` |
| [org.codehaus.mojo:versions-maven-plugin](https://github.com/mojohaus/versions ) | `2.22.0` | `2.23.0` |
| [org.sonatype.central:central-publishing-maven-plugin](https://github.com/sonatype-nexus-community/central-publishing-maven-plugin ) | `0.11.0` | `0.12.0` |
| [commons-codec:commons-codec](https://github.com/apache/commons-codec ) | `1.22.1` | `1.23.0` |
| [dnsjava:dnsjava](https://github.com/dnsjava/dnsjava ) | `3.6.5` | `3.7.0` |
| [org.springframework:spring-core](https://github.com/spring-projects/spring-framework ) | `7.0.9` | `7.0.10` |
| [org.springframework:spring-beans](https://github.com/spring-projects/spring-framework ) | `7.0.9` | `7.0.10` |
| [org.springframework:spring-context](https://github.com/spring-projects/spring-framework ) | `7.0.9` | `7.0.10` |
| [org.springframework:spring-expression](https://github.com/spring-projects/spring-framework ) | `7.0.9` | `7.0.10` |
| [com.esotericsoftware:kryo](https://github.com/EsotericSoftware/kryo ) | `5.6.2` | `5.7.1` |
| [com.github.mwiede:jsch](https://github.com/mwiede/jsch ) | `2.28.7` | `2.28.8` |
| [org.apache.mina:mina-core](https://github.com/apache/mina ) | `2.2.9` | `2.2.10` |
| [org.restlet:org.restlet](https://github.com/restlet/restlet-framework-java ) | `2.6.0` | `2.6.1` |
| org.restlet:org.restlet.ext.jetty | `2.6.0` | `2.6.1` |
| [org.restlet:org.restlet.ext.xml](https://github.com/restlet/restlet-framework-java ) | `2.6.0` | `2.6.1` |
| [org.restlet:org.restlet.ext.json](https://github.com/restlet/restlet-framework-java ) | `2.6.0` | `2.6.1` |
| [org.restlet:org.restlet.ext.crypto](https://github.com/restlet/restlet-framework-java ) | `2.6.0` | `2.6.1` |
| org.apache.kafka:kafka-clients | `4.3.1` | `4.4.0` |
| [at.yawk.lz4:lz4-java](https://github.com/yawkat/lz4-java ) | `1.12.0` | `1.12.1` |
| [com.google.code.gson:gson](https://github.com/google/gson ) | `2.14.0` | `2.15.0` |
Updates `com.fasterxml.jackson:jackson-bom` from 2.22.3 to 2.22.4
- [Commits](https://github.com/FasterXML/jackson-bom/compare/jackson-bom-2.22.3...jackson-bom-2.22.4 )
Updates `org.eclipse.jetty:jetty-bom` from 12.0.39 to 12.0.40
- [Release notes](https://github.com/jetty/jetty.project/releases )
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.39...jetty-12.0.40 )
Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.39 to 12.0.40
- [Release notes](https://github.com/jetty/jetty.project/releases )
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.39...jetty-12.0.40 )
Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.39 to 12.0.40
- [Release notes](https://github.com/jetty/jetty.project/releases )
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.39...jetty-12.0.40 )
Updates `org.apache.maven.plugins:maven-surefire-plugin` from 3.6.0 to 3.6.1
- [Release notes](https://github.com/apache/maven-surefire/releases )
- [Commits](https://github.com/apache/maven-surefire/compare/surefire-3.6.0...surefire-3.6.1 )
Updates `org.codehaus.mojo:versions-maven-plugin` from 2.22.0 to 2.23.0
- [Release notes](https://github.com/mojohaus/versions/releases )
- [Changelog](https://github.com/mojohaus/versions/blob/master/ReleaseNotes.md )
- [Commits](https://github.com/mojohaus/versions/compare/2.22.0...2.23.0 )
Updates `org.sonatype.central:central-publishing-maven-plugin` from 0.11.0 to 0.12.0
- [Release notes](https://github.com/sonatype-nexus-community/central-publishing-maven-plugin/releases )
- [Commits](https://github.com/sonatype-nexus-community/central-publishing-maven-plugin/commits/release-0.12.0 )
Updates `commons-codec:commons-codec` from 1.22.1 to 1.23.0
- [Changelog](https://github.com/apache/commons-codec/blob/master/RELEASE-NOTES.txt )
- [Commits](https://github.com/apache/commons-codec/compare/rel/commons-codec-1.22.1...rel/commons-codec-1.23.0 )
Updates `dnsjava:dnsjava` from 3.6.5 to 3.7.0
- [Release notes](https://github.com/dnsjava/dnsjava/releases )
- [Changelog](https://github.com/dnsjava/dnsjava/blob/master/Changelog )
- [Commits](https://github.com/dnsjava/dnsjava/commits/v3.7.0 )
Updates `org.springframework:spring-core` from 7.0.9 to 7.0.10
- [Release notes](https://github.com/spring-projects/spring-framework/releases )
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.9...v7.0.10 )
Updates `org.springframework:spring-beans` from 7.0.9 to 7.0.10
- [Release notes](https://github.com/spring-projects/spring-framework/releases )
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.9...v7.0.10 )
Updates `org.springframework:spring-context` from 7.0.9 to 7.0.10
- [Release notes](https://github.com/spring-projects/spring-framework/releases )
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.9...v7.0.10 )
Updates `org.springframework:spring-expression` from 7.0.9 to 7.0.10
- [Release notes](https://github.com/spring-projects/spring-framework/releases )
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.9...v7.0.10 )
Updates `org.springframework:spring-beans` from 7.0.9 to 7.0.10
- [Release notes](https://github.com/spring-projects/spring-framework/releases )
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.9...v7.0.10 )
Updates `org.springframework:spring-context` from 7.0.9 to 7.0.10
- [Release notes](https://github.com/spring-projects/spring-framework/releases )
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.9...v7.0.10 )
Updates `org.springframework:spring-expression` from 7.0.9 to 7.0.10
- [Release notes](https://github.com/spring-projects/spring-framework/releases )
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.9...v7.0.10 )
Updates `com.esotericsoftware:kryo` from 5.6.2 to 5.7.1
- [Release notes](https://github.com/EsotericSoftware/kryo/releases )
- [Commits](https://github.com/EsotericSoftware/kryo/compare/kryo-parent-5.6.2...kryo-parent-5.7.1 )
Updates `com.github.mwiede:jsch` from 2.28.7 to 2.28.8
- [Release notes](https://github.com/mwiede/jsch/releases )
- [Changelog](https://github.com/mwiede/jsch/blob/master/ChangeLog )
- [Commits](https://github.com/mwiede/jsch/compare/jsch-2.28.7...jsch-2.28.8 )
Updates `org.apache.mina:mina-core` from 2.2.9 to 2.2.10
- [Commits](https://github.com/apache/mina/compare/2.2.9...2.2.10 )
Updates `org.restlet:org.restlet` from 2.6.0 to 2.6.1
- [Changelog](https://github.com/restlet/restlet-framework-java/blob/2.6.1/changes.md )
- [Commits](https://github.com/restlet/restlet-framework-java/compare/2.6.0...2.6.1 )
Updates `org.restlet:org.restlet.ext.jetty` from 2.6.0 to 2.6.1
Updates `org.restlet:org.restlet.ext.xml` from 2.6.0 to 2.6.1
- [Changelog](https://github.com/restlet/restlet-framework-java/blob/2.6.1/changes.md )
- [Commits](https://github.com/restlet/restlet-framework-java/compare/2.6.0...2.6.1 )
Updates `org.restlet:org.restlet.ext.json` from 2.6.0 to 2.6.1
- [Changelog](https://github.com/restlet/restlet-framework-java/blob/2.6.1/changes.md )
- [Commits](https://github.com/restlet/restlet-framework-java/compare/2.6.0...2.6.1 )
Updates `org.restlet:org.restlet.ext.crypto` from 2.6.0 to 2.6.1
- [Changelog](https://github.com/restlet/restlet-framework-java/blob/2.6.1/changes.md )
- [Commits](https://github.com/restlet/restlet-framework-java/compare/2.6.0...2.6.1 )
Updates `org.restlet:org.restlet.ext.xml` from 2.6.0 to 2.6.1
- [Changelog](https://github.com/restlet/restlet-framework-java/blob/2.6.1/changes.md )
- [Commits](https://github.com/restlet/restlet-framework-java/compare/2.6.0...2.6.1 )
Updates `org.restlet:org.restlet.ext.json` from 2.6.0 to 2.6.1
- [Changelog](https://github.com/restlet/restlet-framework-java/blob/2.6.1/changes.md )
- [Commits](https://github.com/restlet/restlet-framework-java/compare/2.6.0...2.6.1 )
Updates `org.restlet:org.restlet.ext.crypto` from 2.6.0 to 2.6.1
- [Changelog](https://github.com/restlet/restlet-framework-java/blob/2.6.1/changes.md )
- [Commits](https://github.com/restlet/restlet-framework-java/compare/2.6.0...2.6.1 )
Updates `org.apache.kafka:kafka-clients` from 4.3.1 to 4.4.0
Updates `at.yawk.lz4:lz4-java` from 1.12.0 to 1.12.1
- [Release notes](https://github.com/yawkat/lz4-java/releases )
- [Changelog](https://github.com/yawkat/lz4-java/blob/main/CHANGES.md )
- [Commits](https://github.com/yawkat/lz4-java/compare/v1.12.0...v1.12.1 )
Updates `com.google.code.gson:gson` from 2.14.0 to 2.15.0
- [Release notes](https://github.com/google/gson/releases )
- [Changelog](https://github.com/google/gson/blob/main/CHANGELOG.md )
- [Commits](https://github.com/google/gson/compare/gson-parent-2.14.0...gson-parent-2.15.0 )
---
updated-dependencies:
- dependency-name: com.fasterxml.jackson:jackson-bom
dependency-version: 2.22.4
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty:jetty-bom
dependency-version: 12.0.40
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
dependency-version: 12.0.40
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
dependency-version: 12.0.40
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-surefire-plugin
dependency-version: 3.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.codehaus.mojo:versions-maven-plugin
dependency-version: 2.23.0
dependency-type: direct:development
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: org.sonatype.central:central-publishing-maven-plugin
dependency-version: 0.12.0
dependency-type: direct:development
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: commons-codec:commons-codec
dependency-version: 1.23.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: dnsjava:dnsjava
dependency-version: 3.7.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-core
dependency-version: 7.0.10
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
dependency-version: 7.0.10
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
dependency-version: 7.0.10
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
dependency-version: 7.0.10
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
dependency-version: 7.0.10
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
dependency-version: 7.0.10
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
dependency-version: 7.0.10
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: com.esotericsoftware:kryo
dependency-version: 5.7.1
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: com.github.mwiede:jsch
dependency-version: 2.28.8
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.apache.mina:mina-core
dependency-version: 2.2.10
dependency-type: direct:development
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.restlet:org.restlet
dependency-version: 2.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.restlet:org.restlet.ext.jetty
dependency-version: 2.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.restlet:org.restlet.ext.xml
dependency-version: 2.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.restlet:org.restlet.ext.json
dependency-version: 2.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.restlet:org.restlet.ext.crypto
dependency-version: 2.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.restlet:org.restlet.ext.xml
dependency-version: 2.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.restlet:org.restlet.ext.json
dependency-version: 2.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.restlet:org.restlet.ext.crypto
dependency-version: 2.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.apache.kafka:kafka-clients
dependency-version: 4.4.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: at.yawk.lz4:lz4-java
dependency-version: 1.12.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: com.google.code.gson:gson
dependency-version: 2.15.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 16:52:47 +09:00
Alex Osborne
f4f88a4335
BotBlockDetector: add azure-waf and vercel ( #784 )
2026-10-09 08:32:00 +09:00
Alex Osborne
be6b49fa6c
[maven-release-plugin] prepare for next development iteration
2026-10-06 12:39:19 +09:00
Alex Osborne
eac2c96daf
[maven-release-plugin] prepare release 3.18.0
2026-10-06 12:39:12 +09:00
Alex Osborne
0ca9e5dc2a
Merge pull request #778 from internetarchive/ato/botblock-detector2
...
BotBlockDetector: add additional services and enable by default
2026-10-06 11:20:02 +09:00
dependabot[bot]
c36460a0f3
Bump the minor-and-patch group with 13 updates
...
Bumps the minor-and-patch group with 13 updates:
| Package | From | To |
| --- | --- | --- |
| [org.eclipse.jetty:jetty-bom](https://github.com/jetty/jetty.project ) | `12.0.38` | `12.0.39` |
| [org.eclipse.jetty.ee10:jetty-ee10-bom](https://github.com/jetty/jetty.project ) | `12.0.38` | `12.0.39` |
| [org.slf4j:slf4j-bom](https://github.com/qos-ch/slf4j ) | `2.0.18` | `2.0.20` |
| [org.apache.maven.plugins:maven-compiler-plugin](https://github.com/apache/maven-compiler-plugin ) | `3.15.0` | `3.16.0` |
| [org.apache.maven.plugins:maven-surefire-plugin](https://github.com/apache/maven-surefire ) | `3.5.6` | `3.6.0` |
| [org.codehaus.mojo:versions-maven-plugin](https://github.com/mojohaus/versions ) | `2.21.0` | `2.22.0` |
| [org.webjars:webjars-locator-lite](https://github.com/webjars/webjars-locator-lite ) | `1.1.4` | `1.1.5` |
| org.webjars.npm:codemirror__commands | `6.10.4` | `6.11.0` |
| org.webjars.npm:codemirror__state | `6.7.1` | `6.7.4` |
| org.webjars.npm:codemirror__view | `6.43.7` | `6.43.11` |
| org.webjars.npm:marijn__find-cluster-break | `1.0.3` | `1.0.4` |
| [com.rabbitmq:amqp-client](https://github.com/rabbitmq/rabbitmq-java-client ) | `5.35.0` | `5.36.0` |
| [at.yawk.lz4:lz4-java](https://github.com/yawkat/lz4-java ) | `1.11.2` | `1.12.0` |
Updates `org.eclipse.jetty:jetty-bom` from 12.0.38 to 12.0.39
- [Release notes](https://github.com/jetty/jetty.project/releases )
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.38...jetty-12.0.39 )
Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.38 to 12.0.39
- [Release notes](https://github.com/jetty/jetty.project/releases )
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.38...jetty-12.0.39 )
Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.38 to 12.0.39
- [Release notes](https://github.com/jetty/jetty.project/releases )
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.38...jetty-12.0.39 )
Updates `org.slf4j:slf4j-bom` from 2.0.18 to 2.0.20
- [Release notes](https://github.com/qos-ch/slf4j/releases )
- [Commits](https://github.com/qos-ch/slf4j/compare/v_2.0.18...v_2.0.20 )
Updates `org.apache.maven.plugins:maven-compiler-plugin` from 3.15.0 to 3.16.0
- [Release notes](https://github.com/apache/maven-compiler-plugin/releases )
- [Commits](https://github.com/apache/maven-compiler-plugin/compare/maven-compiler-plugin-3.15.0...maven-compiler-plugin-3.16.0 )
Updates `org.apache.maven.plugins:maven-surefire-plugin` from 3.5.6 to 3.6.0
- [Release notes](https://github.com/apache/maven-surefire/releases )
- [Commits](https://github.com/apache/maven-surefire/compare/surefire-3.5.6...surefire-3.6.0 )
Updates `org.codehaus.mojo:versions-maven-plugin` from 2.21.0 to 2.22.0
- [Release notes](https://github.com/mojohaus/versions/releases )
- [Changelog](https://github.com/mojohaus/versions/blob/master/ReleaseNotes.md )
- [Commits](https://github.com/mojohaus/versions/compare/2.21.0...2.22.0 )
Updates `org.webjars:webjars-locator-lite` from 1.1.4 to 1.1.5
- [Release notes](https://github.com/webjars/webjars-locator-lite/releases )
- [Commits](https://github.com/webjars/webjars-locator-lite/compare/webjars-locator-lite-1.1.4...webjars-locator-lite-1.1.5 )
Updates `org.webjars.npm:codemirror__commands` from 6.10.4 to 6.11.0
Updates `org.webjars.npm:codemirror__state` from 6.7.1 to 6.7.4
Updates `org.webjars.npm:codemirror__view` from 6.43.7 to 6.43.11
Updates `org.webjars.npm:marijn__find-cluster-break` from 1.0.3 to 1.0.4
Updates `com.rabbitmq:amqp-client` from 5.35.0 to 5.36.0
- [Release notes](https://github.com/rabbitmq/rabbitmq-java-client/releases )
- [Commits](https://github.com/rabbitmq/rabbitmq-java-client/compare/v5.35.0...v5.36.0 )
Updates `at.yawk.lz4:lz4-java` from 1.11.2 to 1.12.0
- [Release notes](https://github.com/yawkat/lz4-java/releases )
- [Changelog](https://github.com/yawkat/lz4-java/blob/main/CHANGES.md )
- [Commits](https://github.com/yawkat/lz4-java/compare/v1.11.2...v1.12.0 )
---
updated-dependencies:
- dependency-name: org.eclipse.jetty:jetty-bom
dependency-version: 12.0.39
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
dependency-version: 12.0.39
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
dependency-version: 12.0.39
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.slf4j:slf4j-bom
dependency-version: 2.0.20
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-compiler-plugin
dependency-version: 3.16.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-surefire-plugin
dependency-version: 3.6.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: org.codehaus.mojo:versions-maven-plugin
dependency-version: 2.22.0
dependency-type: direct:development
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: org.webjars:webjars-locator-lite
dependency-version: 1.1.5
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__commands
dependency-version: 6.11.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__state
dependency-version: 6.7.4
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__view
dependency-version: 6.43.11
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:marijn__find-cluster-break
dependency-version: 1.0.4
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: com.rabbitmq:amqp-client
dependency-version: 5.36.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: at.yawk.lz4:lz4-java
dependency-version: 1.12.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
...
Signed-off-by: dependabot[bot] <support@github.com >
2026-10-02 08:51:34 +09:00
Alex Osborne
a19bd61a33
BotBlockDetector: add aws-waf, kasada, lua-anti-ddos, perimeterx
2026-10-02 08:25:52 +09:00
Shubham Padkonde
16deaaf4eb
Decode JavaScript Unicode code-point escapes in extracted URLs ( #772 )
...
Signed-off-by: Shubham Padkonde <shubhampadkonde12@gmail.com >
2026-09-22 17:21:42 +09:00
Francesco Marinoni Moretto
ed2805f6b7
Extract URIs nested inside JSON arrays ( #771 )
...
`parse()` called `propertyStream()` on array nodes, which Jackson returns empty
for anything that is not an object, so URIs inside an array held as an object
property value were never extracted. Those array elements now go through the same
value handling as object values, which is also what lets a bare URI string inside
an array be extracted and not only an object holding one.
A known limit this does not change: `parse()` still starts from
`rootNode.properties()`, so a document whose root is itself an array yields no
links.
2026-09-22 17:16:43 +09:00
Alex Osborne
6c32aaa8c1
FetchHTTP: digest chunked responses after transfer decoding
...
Recalculate content digests from the entity replay stream for chunked responses so that chunk framing is not included in WARC-Payload-Digest.
Fixes #769
2026-09-15 10:13:56 +09:00
Alex Osborne
0d3582a136
[maven-release-plugin] prepare for next development iteration
2026-09-10 17:29:57 +09:00
Alex Osborne
48f9c37f83
[maven-release-plugin] prepare release 3.17.1
2026-09-10 17:29:50 +09:00
Leslie Bellony
33984a68d2
Issue #752 new Heritrix status for SSLHandshakeException
2026-09-08 10:56:57 +02:00
Alex Osborne
b5f32616c8
Fix javadoc warnings
2026-08-25 11:02:15 +09:00
Alex Osborne
527b202752
[maven-release-plugin] prepare for next development iteration
2026-08-25 10:17:24 +09:00
Alex Osborne
6eeb779923
[maven-release-plugin] prepare release 3.17.0
2026-08-25 10:17:12 +09:00
Alex Osborne
cd4f910c57
BotBlockDetector: Look for cf-mitigated without server header
...
Might catch unusual cases where something else is in front of Cloudflare.
2026-08-19 10:49:55 +09:00
Alex Osborne
9973a9f2f0
BotBlockDetector: Add DataDome
2026-08-19 10:49:55 +09:00
Alex Osborne
cec6cfbb64
Add BotBlockDetector
...
Detects responses produced by common bot-blocking services and adds a "botblock:service" annotation. This is intended for reporting and troubleshooting but may also be useful as a signal to stop crawling or to skip writing WARC records.
2026-08-18 16:11:35 +09:00
Alex Osborne
96b16d0e68
Add mina-core 2.2.9 as explicit dependency
...
The transitive version pulled in via ftpserver-core has vulnerabilities.
2026-08-04 16:53:36 +09:00