Alex Osborne
a35dcb86a6
Merge pull request #811 from internetarchive/crawluri-has-content-type
...
Add CrawlURI.hasContentType() for media type checks
2026-11-05 16:17:25 +09:00
Alex Osborne
db8b626247
Add .editorconfig
2026-11-05 11:21:32 +09:00
Alex Osborne
7a9e2689b4
Use CrawlURI.hasContentType() for exact media type checks
...
Replaces startsWith() prefix checks, which also matched longer subtypes,
e.g. "application/vnd.wap.wml" matched the binary
"application/vnd.wap.wmlc". The truncated "application/xhtml" and
"application/vnd.wap.xhtml" checks now name the full "+xml" types. Also
fixes case-sensitive comparisons in ExtractorPDF and ExtractorPDFContent,
and drops BotBlockDetector's private equivalent.
Substring heuristics like ExtractorJS's indexOf("javascript") and
ExtractorXML's indexOf("xml") are intentionally left as they are.
2026-11-05 09:58:58 +09:00
Alex Osborne
5b1f492cc2
CrawlURI: add hasContentType() for media type comparison
...
Compares the content type's type/subtype with the given media type,
ignoring case and any parameters, so "Text/HTML; charset=utf-8" matches
"text/html" but "text/htmlfoo" doesn't. This replaces ad hoc checks like
toLowerCase().startsWith(...), which also match longer subtypes.
2026-11-05 09:53:25 +09:00
Alex Osborne
a6f69a4473
Merge pull request #808 from internetarchive/extractorjs-es-module-imports
...
ExtractorJS: resolve ES module imports relative to script's URL
2026-11-04 16:41:13 +09:00
Alex Osborne
8cc8a7493b
Merge pull request #804 from internetarchive/html-extraction-gaps
...
ExtractorHTML: extract poster, imagesrcset, formaction and iframe srcdoc
2026-11-04 16:38:38 +09:00
Alex Osborne
b8c22606b6
ExtractorHTML: extract links from iframe srcdoc
2026-11-04 15:16:19 +09:00
Alex Osborne
889a6b0082
JerichoExtractorHTML: parse sequentially to ignore tags inside attribute values
2026-11-04 15:10:48 +09:00
Alex Osborne
65621b48d4
ExtractorHTML: don't end tags at '>' inside quoted attribute values
...
Previously a '>' inside a quoted attribute value, e.g.
onclick="if (a > b) ..." or an iframe srcdoc, ended the tag early and
any attributes after it were lost. Quotes are now only treated as value
delimiters directly after '=', as in browsers.
2026-11-04 15:08:46 +09:00
Alex Osborne
91d41c2084
ExtractorHTML: extract formaction attribute
...
Only extracted when formmethod=get or extractOnlyFormGets is disabled,
since without formmethod the form's own method applies and would
require tracking state between tags.
2026-11-04 14:54:34 +09:00
Alex Osborne
a356bc477d
ExtractorHTML: extract link imagesrcset attribute as embed
2026-11-04 14:43:12 +09:00
Alex Osborne
a76e5221cc
ExtractorHTML: extract poster attribute as embed
2026-11-04 14:40:36 +09:00
Alex Osborne
0f46340d26
Merge pull request #792 from mvanhorn/fix/455-midfetch-abort-statistics
...
fix: disregard HTTP responses rejected before body fetching
2026-11-02 23:38:13 +09:00
Matt Van Horn
a769651bf5
fix: give midfetch aborts their own fetch status code
...
Add S_MIDFETCH_ABORTED (-5005) and set it in FetchHTTP when
shouldFetchBodyRule rejects a response, in place of S_OUT_OF_SCOPE.
The code means something distinct from a scope rejection, so crawl logs
and reports can now tell the two apart. AbstractFrontier.isDisregarded()
treats it as disregarded, CrawlURI gives it a readable string and the
glossary lists it.
2026-11-02 06:01:06 -08:00
Matt Van Horn
1c27e2e7b0
fix: disregard HTTP responses rejected before body fetching
...
A rejection by shouldFetchBodyRule now sets S_OUT_OF_SCOPE, causing the
frontier to disregard the URI without charging the queue budget or
retrying it, and skipping extraction and WARC writing. Tests cover
rejected responses and preserve successful handling for accepted or
disabled rules, prerequisites, size limits, and timeouts.
Fixes #455
2026-11-02 00:32:19 -08:00
dependabot[bot]
2dfc290224
Bump actions/setup-java from 6.0.1 to 6.1.0 ( #790 )
...
Bumps [actions/setup-java](https://github.com/actions/setup-java ) from 6.0.1 to 6.1.0.
- [Release notes](https://github.com/actions/setup-java/releases )
- [Commits](https://github.com/actions/setup-java/compare/v6.0.1...v6.1.0 )
---
updated-dependencies:
- dependency-name: actions/setup-java
dependency-version: 6.1.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 16:53:05 +09:00
dependabot[bot]
ce13a99f71
Bump the minor-and-patch group with 23 updates ( #791 )
...
Bumps the minor-and-patch group with 23 updates:
| Package | From | To |
| --- | --- | --- |
| [com.fasterxml.jackson:jackson-bom](https://github.com/FasterXML/jackson-bom ) | `2.22.3` | `2.22.4` |
| [org.eclipse.jetty:jetty-bom](https://github.com/jetty/jetty.project ) | `12.0.39` | `12.0.40` |
| [org.eclipse.jetty.ee10:jetty-ee10-bom](https://github.com/jetty/jetty.project ) | `12.0.39` | `12.0.40` |
| [org.apache.maven.plugins:maven-surefire-plugin](https://github.com/apache/maven-surefire ) | `3.6.0` | `3.6.1` |
| [org.codehaus.mojo:versions-maven-plugin](https://github.com/mojohaus/versions ) | `2.22.0` | `2.23.0` |
| [org.sonatype.central:central-publishing-maven-plugin](https://github.com/sonatype-nexus-community/central-publishing-maven-plugin ) | `0.11.0` | `0.12.0` |
| [commons-codec:commons-codec](https://github.com/apache/commons-codec ) | `1.22.1` | `1.23.0` |
| [dnsjava:dnsjava](https://github.com/dnsjava/dnsjava ) | `3.6.5` | `3.7.0` |
| [org.springframework:spring-core](https://github.com/spring-projects/spring-framework ) | `7.0.9` | `7.0.10` |
| [org.springframework:spring-beans](https://github.com/spring-projects/spring-framework ) | `7.0.9` | `7.0.10` |
| [org.springframework:spring-context](https://github.com/spring-projects/spring-framework ) | `7.0.9` | `7.0.10` |
| [org.springframework:spring-expression](https://github.com/spring-projects/spring-framework ) | `7.0.9` | `7.0.10` |
| [com.esotericsoftware:kryo](https://github.com/EsotericSoftware/kryo ) | `5.6.2` | `5.7.1` |
| [com.github.mwiede:jsch](https://github.com/mwiede/jsch ) | `2.28.7` | `2.28.8` |
| [org.apache.mina:mina-core](https://github.com/apache/mina ) | `2.2.9` | `2.2.10` |
| [org.restlet:org.restlet](https://github.com/restlet/restlet-framework-java ) | `2.6.0` | `2.6.1` |
| org.restlet:org.restlet.ext.jetty | `2.6.0` | `2.6.1` |
| [org.restlet:org.restlet.ext.xml](https://github.com/restlet/restlet-framework-java ) | `2.6.0` | `2.6.1` |
| [org.restlet:org.restlet.ext.json](https://github.com/restlet/restlet-framework-java ) | `2.6.0` | `2.6.1` |
| [org.restlet:org.restlet.ext.crypto](https://github.com/restlet/restlet-framework-java ) | `2.6.0` | `2.6.1` |
| org.apache.kafka:kafka-clients | `4.3.1` | `4.4.0` |
| [at.yawk.lz4:lz4-java](https://github.com/yawkat/lz4-java ) | `1.12.0` | `1.12.1` |
| [com.google.code.gson:gson](https://github.com/google/gson ) | `2.14.0` | `2.15.0` |
Updates `com.fasterxml.jackson:jackson-bom` from 2.22.3 to 2.22.4
- [Commits](https://github.com/FasterXML/jackson-bom/compare/jackson-bom-2.22.3...jackson-bom-2.22.4 )
Updates `org.eclipse.jetty:jetty-bom` from 12.0.39 to 12.0.40
- [Release notes](https://github.com/jetty/jetty.project/releases )
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.39...jetty-12.0.40 )
Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.39 to 12.0.40
- [Release notes](https://github.com/jetty/jetty.project/releases )
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.39...jetty-12.0.40 )
Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.39 to 12.0.40
- [Release notes](https://github.com/jetty/jetty.project/releases )
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.39...jetty-12.0.40 )
Updates `org.apache.maven.plugins:maven-surefire-plugin` from 3.6.0 to 3.6.1
- [Release notes](https://github.com/apache/maven-surefire/releases )
- [Commits](https://github.com/apache/maven-surefire/compare/surefire-3.6.0...surefire-3.6.1 )
Updates `org.codehaus.mojo:versions-maven-plugin` from 2.22.0 to 2.23.0
- [Release notes](https://github.com/mojohaus/versions/releases )
- [Changelog](https://github.com/mojohaus/versions/blob/master/ReleaseNotes.md )
- [Commits](https://github.com/mojohaus/versions/compare/2.22.0...2.23.0 )
Updates `org.sonatype.central:central-publishing-maven-plugin` from 0.11.0 to 0.12.0
- [Release notes](https://github.com/sonatype-nexus-community/central-publishing-maven-plugin/releases )
- [Commits](https://github.com/sonatype-nexus-community/central-publishing-maven-plugin/commits/release-0.12.0 )
Updates `commons-codec:commons-codec` from 1.22.1 to 1.23.0
- [Changelog](https://github.com/apache/commons-codec/blob/master/RELEASE-NOTES.txt )
- [Commits](https://github.com/apache/commons-codec/compare/rel/commons-codec-1.22.1...rel/commons-codec-1.23.0 )
Updates `dnsjava:dnsjava` from 3.6.5 to 3.7.0
- [Release notes](https://github.com/dnsjava/dnsjava/releases )
- [Changelog](https://github.com/dnsjava/dnsjava/blob/master/Changelog )
- [Commits](https://github.com/dnsjava/dnsjava/commits/v3.7.0 )
Updates `org.springframework:spring-core` from 7.0.9 to 7.0.10
- [Release notes](https://github.com/spring-projects/spring-framework/releases )
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.9...v7.0.10 )
Updates `org.springframework:spring-beans` from 7.0.9 to 7.0.10
- [Release notes](https://github.com/spring-projects/spring-framework/releases )
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.9...v7.0.10 )
Updates `org.springframework:spring-context` from 7.0.9 to 7.0.10
- [Release notes](https://github.com/spring-projects/spring-framework/releases )
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.9...v7.0.10 )
Updates `org.springframework:spring-expression` from 7.0.9 to 7.0.10
- [Release notes](https://github.com/spring-projects/spring-framework/releases )
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.9...v7.0.10 )
Updates `org.springframework:spring-beans` from 7.0.9 to 7.0.10
- [Release notes](https://github.com/spring-projects/spring-framework/releases )
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.9...v7.0.10 )
Updates `org.springframework:spring-context` from 7.0.9 to 7.0.10
- [Release notes](https://github.com/spring-projects/spring-framework/releases )
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.9...v7.0.10 )
Updates `org.springframework:spring-expression` from 7.0.9 to 7.0.10
- [Release notes](https://github.com/spring-projects/spring-framework/releases )
- [Commits](https://github.com/spring-projects/spring-framework/compare/v7.0.9...v7.0.10 )
Updates `com.esotericsoftware:kryo` from 5.6.2 to 5.7.1
- [Release notes](https://github.com/EsotericSoftware/kryo/releases )
- [Commits](https://github.com/EsotericSoftware/kryo/compare/kryo-parent-5.6.2...kryo-parent-5.7.1 )
Updates `com.github.mwiede:jsch` from 2.28.7 to 2.28.8
- [Release notes](https://github.com/mwiede/jsch/releases )
- [Changelog](https://github.com/mwiede/jsch/blob/master/ChangeLog )
- [Commits](https://github.com/mwiede/jsch/compare/jsch-2.28.7...jsch-2.28.8 )
Updates `org.apache.mina:mina-core` from 2.2.9 to 2.2.10
- [Commits](https://github.com/apache/mina/compare/2.2.9...2.2.10 )
Updates `org.restlet:org.restlet` from 2.6.0 to 2.6.1
- [Changelog](https://github.com/restlet/restlet-framework-java/blob/2.6.1/changes.md )
- [Commits](https://github.com/restlet/restlet-framework-java/compare/2.6.0...2.6.1 )
Updates `org.restlet:org.restlet.ext.jetty` from 2.6.0 to 2.6.1
Updates `org.restlet:org.restlet.ext.xml` from 2.6.0 to 2.6.1
- [Changelog](https://github.com/restlet/restlet-framework-java/blob/2.6.1/changes.md )
- [Commits](https://github.com/restlet/restlet-framework-java/compare/2.6.0...2.6.1 )
Updates `org.restlet:org.restlet.ext.json` from 2.6.0 to 2.6.1
- [Changelog](https://github.com/restlet/restlet-framework-java/blob/2.6.1/changes.md )
- [Commits](https://github.com/restlet/restlet-framework-java/compare/2.6.0...2.6.1 )
Updates `org.restlet:org.restlet.ext.crypto` from 2.6.0 to 2.6.1
- [Changelog](https://github.com/restlet/restlet-framework-java/blob/2.6.1/changes.md )
- [Commits](https://github.com/restlet/restlet-framework-java/compare/2.6.0...2.6.1 )
Updates `org.restlet:org.restlet.ext.xml` from 2.6.0 to 2.6.1
- [Changelog](https://github.com/restlet/restlet-framework-java/blob/2.6.1/changes.md )
- [Commits](https://github.com/restlet/restlet-framework-java/compare/2.6.0...2.6.1 )
Updates `org.restlet:org.restlet.ext.json` from 2.6.0 to 2.6.1
- [Changelog](https://github.com/restlet/restlet-framework-java/blob/2.6.1/changes.md )
- [Commits](https://github.com/restlet/restlet-framework-java/compare/2.6.0...2.6.1 )
Updates `org.restlet:org.restlet.ext.crypto` from 2.6.0 to 2.6.1
- [Changelog](https://github.com/restlet/restlet-framework-java/blob/2.6.1/changes.md )
- [Commits](https://github.com/restlet/restlet-framework-java/compare/2.6.0...2.6.1 )
Updates `org.apache.kafka:kafka-clients` from 4.3.1 to 4.4.0
Updates `at.yawk.lz4:lz4-java` from 1.12.0 to 1.12.1
- [Release notes](https://github.com/yawkat/lz4-java/releases )
- [Changelog](https://github.com/yawkat/lz4-java/blob/main/CHANGES.md )
- [Commits](https://github.com/yawkat/lz4-java/compare/v1.12.0...v1.12.1 )
Updates `com.google.code.gson:gson` from 2.14.0 to 2.15.0
- [Release notes](https://github.com/google/gson/releases )
- [Changelog](https://github.com/google/gson/blob/main/CHANGELOG.md )
- [Commits](https://github.com/google/gson/compare/gson-parent-2.14.0...gson-parent-2.15.0 )
---
updated-dependencies:
- dependency-name: com.fasterxml.jackson:jackson-bom
dependency-version: 2.22.4
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty:jetty-bom
dependency-version: 12.0.40
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
dependency-version: 12.0.40
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
dependency-version: 12.0.40
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-surefire-plugin
dependency-version: 3.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.codehaus.mojo:versions-maven-plugin
dependency-version: 2.23.0
dependency-type: direct:development
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: org.sonatype.central:central-publishing-maven-plugin
dependency-version: 0.12.0
dependency-type: direct:development
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: commons-codec:commons-codec
dependency-version: 1.23.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: dnsjava:dnsjava
dependency-version: 3.7.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-core
dependency-version: 7.0.10
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
dependency-version: 7.0.10
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
dependency-version: 7.0.10
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
dependency-version: 7.0.10
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-beans
dependency-version: 7.0.10
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-context
dependency-version: 7.0.10
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.springframework:spring-expression
dependency-version: 7.0.10
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: com.esotericsoftware:kryo
dependency-version: 5.7.1
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: com.github.mwiede:jsch
dependency-version: 2.28.8
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.apache.mina:mina-core
dependency-version: 2.2.10
dependency-type: direct:development
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.restlet:org.restlet
dependency-version: 2.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.restlet:org.restlet.ext.jetty
dependency-version: 2.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.restlet:org.restlet.ext.xml
dependency-version: 2.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.restlet:org.restlet.ext.json
dependency-version: 2.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.restlet:org.restlet.ext.crypto
dependency-version: 2.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.restlet:org.restlet.ext.xml
dependency-version: 2.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.restlet:org.restlet.ext.json
dependency-version: 2.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.restlet:org.restlet.ext.crypto
dependency-version: 2.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.apache.kafka:kafka-clients
dependency-version: 4.4.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: at.yawk.lz4:lz4-java
dependency-version: 1.12.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: com.google.code.gson:gson
dependency-version: 2.15.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 16:52:47 +09:00
Alex Osborne
65529301b5
Merge pull request #781 from internetarchive/adam/fix_amqp_consumer_leak
...
feat: Refactor AMQPUrlReceiver and AMQPProducer to improve recovery from network issues.
2026-10-20 00:36:15 +09:00
Adam Miller
6b3ab9c937
feat: Adds configuration to enable persistence of messages in the penumbra queue
2026-10-16 10:26:48 -07:00
Adam Miller
3a2865efb6
feat: add liveliness check to AMQP StarterRestarter. Reduce queue prefetch to 100
2026-10-15 15:44:20 -07:00
Adam Miller
428b35ed8e
fix: stop AMQPProducer abandoning connections and channels
...
Abort connections and channels before replacing them, and in stop(), so
the client stops recovering what this producer no longer references.
Automatic recovery stays enabled here, unlike AMQPUrlReceiver: connect()
runs on the publishing thread under a lock, so making it the only recovery
path would block every ToeThread while the broker is down.
2026-10-14 16:16:41 -07:00
Adam Miller
458484c50c
fix: restart the AMQPUrlReceiver consumer after a broker-side cancel
...
Handle basic.cancel, and clear consumerTag whether or not basicCancel
succeeded, so a lost subscription is rebuilt instead of stranding the
receiver.
2026-10-14 16:15:45 -07:00
Alex Osborne
e3f9b11394
docs: fix factual errors in user documentation ( #787 )
...
getting-started:
- Use a Java 17 path in the JAVA_HOME example (17 is the minimum)
- Link to the local Configuring Jobs page instead of the wiki
- Fix typos
operating:
- Correct --jobs-dir option name and explain its default
- Document --proxy-host, --proxy-port and --sni-host-check
- Fix option list formatting that broke rendering (-c, --web-auth)
- Remove contradictory note about credentials in heritrix_out.log
- crawl.log fetch duration is separated by '+', not ';'
- Renumber progress-statistics.log fields (field 3 was duplicated)
- Fix sourceTagSeeds property name in the XML example
- ActionDirectory is already in the default profile
- Use the correct frontier.recover.gz file name throughout
- The action directory is inside the job directory, not beside bin/
configuring-jobs:
- Fix maxDocumentsDownload, obeyRelNofollow and logger class names
- Replace removed/renamed decide rules with their current names
- Fix javadoc links for decide rules in subpackages and in engine
- Fix malformed useHTTP3 XML example and FTP/SFTP seed examples
- Replace dead Confluence glossary link
- Drop DoH caveats that only applied to Java 11 / DisableExplicitGC
api:
- Fix Create Job field list not rendering (missing blank line)
- Script engines: list Groovy/BeanShell; js and AppleScript are not
available on supported Java versions
- Make JSON sample response consistent with the XML one
glossary:
- SHA-1 is used for content digests, not encryption
- Fix garbled Host entry (logical vs physical host)
- Fix SURT and SURT prefix examples
- Status -6 refers to DNS, not WHOIS; add -9, 1001, 2001, 2002
- Link status codes to RFC 9110 instead of RFC 2616
2026-10-13 11:34:43 +09:00
Leslie Bellony
91a869bd4b
sourcetags: dispose of the sorted map after use ( #788 )
2026-10-12 18:34:12 +09:00
Alex Osborne
f4f88a4335
BotBlockDetector: add azure-waf and vercel ( #784 )
2026-10-09 08:32:00 +09:00
dependabot[bot]
7cdb1d5ed9
Bump com.rabbitmq:amqp-client ( #786 )
...
Bumps the maven group with 1 update in the /contrib directory: [com.rabbitmq:amqp-client](https://github.com/rabbitmq/rabbitmq-java-client ).
Updates `com.rabbitmq:amqp-client` from 5.36.0 to 5.37.0
- [Release notes](https://github.com/rabbitmq/rabbitmq-java-client/releases )
- [Commits](https://github.com/rabbitmq/rabbitmq-java-client/compare/v5.36.0...v5.37.0 )
---
updated-dependencies:
- dependency-name: com.rabbitmq:amqp-client
dependency-version: 5.37.0
dependency-type: direct:production
dependency-group: maven
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-09 08:31:29 +09:00
Alex Osborne
94a6761a17
BrowserProcessorTest: make tempDir static ( #785 )
...
We're seeing occasional random test failures. I think it may be because we're changing the CrawlController scratchDir before each test, which means the profile of a running browser can get wiped out.
2026-10-08 18:36:09 +09:00
Alex Osborne
8bb0a65349
Merge pull request #783 from internetarchive/toethread-always-end-disposition
...
Always release the disposition lock when a ToeThread exits
2026-10-07 12:42:19 +09:00
Alex Osborne
d6747686e3
Don't let browsers cache job directory files for 10 minutes ( #782 )
...
Restlet's Directory gives every file an Expires header 10 minutes in the
future. Job files (logs, reports, crawler-beans.cxml) change constantly, so
browsers could show stale copies. Worst case, the config editor loads a
cached crawler-beans.cxml and saving it silently reverts newer changes.
Clear the expiration date and send Cache-Control: no-cache instead, so
browsers revalidate against Last-Modified on each use.
2026-10-07 12:42:02 +09:00
Adam Miller
d5ec767bff
fix: make AMQPUrlReceiver resilient to dropped connections
...
Disable the client's automatic recovery and leave StarterRestarter as the
sole owner of the consumer lifecycle, so only one mechanism re-registers a
consumer after a failure.
Abort dead connections and channels before replacing them, so they cannot
go on consuming once the bean has stopped referencing them.
Catch everything in handleDelivery, including the json parsing and the ack,
so one bad message or one dead channel cannot close the channel out from
under every other consumer on it.
Hold consumerTag in an AtomicReference and clear it with compareAndSet, so a
late shutdown signal cannot clear a newer consumer's tag. A lock would risk
deadlocking against stop().
2026-10-06 14:43:12 -07:00
Alex Osborne
be6b49fa6c
[maven-release-plugin] prepare for next development iteration
2026-10-06 12:39:19 +09:00
Alex Osborne
eac2c96daf
[maven-release-plugin] prepare release 3.18.0
3.18.0
2026-10-06 12:39:12 +09:00
Alex Osborne
dbbd820305
Add release notes for 3.18.0
2026-10-06 12:24:16 +09:00
Alex Osborne
7939d0d7e1
Reject cross-site requests to the web UI
...
Add CrossSiteRequestFilter in front of the web UI's router to protect
against cross-site request forgery. POST, PUT and DELETE requests are
rejected with 403 when the browser's Sec-Fetch-Site header says they
came from another site.
Sec-Fetch-Site is set by the browser from its own view of the page and
target URL, so unlike comparing Origin with Host it isn't affected by
reverse proxies rewriting the host or scheme. Requests without the
header, such as from curl and other API clients, are allowed.
Origins listed in the heritrix.trustedOrigins system property (comma
separated) are allowed to send cross-site requests.
2026-10-06 12:21:40 +09:00
Alex Osborne
55bef0aa32
Merge pull request #780 from internetarchive/paged-log-viewer-search
...
Crawl log search
2026-10-06 12:09:09 +09:00
Alex Osborne
6f9a82c8fb
Fix HTML escaping in templates
...
Enables autoescaping in Freemarker and adds some missing escaping
where we generate HTML in Java methods.
2026-10-06 12:04:23 +09:00
Alex Osborne
a09f38570d
Only write heritrix_dmesg.log when requested ( #779 )
...
Only write heritrix_dmesg.log when requested
Heritrix.java always created heritrix_dmesg.log in $HERITRIX_HOME, even
though only the bin/heritrix script reads it, and only when it starts
Heritrix in the background. This got in the way of making
$HERITRIX_HOME read-only for Docker and systemd setups.
The startup log is now written only when the heritrix.dmesg system
property gives its path. bin/heritrix sets this property in background
mode and keeps the file in the same place as before.
Also fix a typo in bin/heritrix ($startmessage) that stopped it
removing a heritrix_dmesg.log left over from a previous start.
Set -Dheritrix.dmesg in heritrix.cmd too.
2026-10-06 11:29:51 +09:00
Alex Osborne
0ca9e5dc2a
Merge pull request #778 from internetarchive/ato/botblock-detector2
...
BotBlockDetector: add additional services and enable by default
2026-10-06 11:20:02 +09:00
Alex Osborne and Claude Opus 5.5
f5070f6203
Paged log viewer: search and filter with a small query language
...
The paged log viewer gains a search box taking a search-engine style
query, passed as the q parameter:
status:4xx host:example.com -type:image size>1MB duration>10s
- Terms are ANDed; '-' negates; double quotes allow spaces.
- 'field:value' is a case-insensitive substring match with '*'
wildcards, 'field=value' is exact and 'field~regex' is a Java regex.
Numeric fields take the same values with ':' or '=', and no regex;
a comparison needs no operator, as in size>1MB.
- Bare words and the 'line' field search the whole line in any log.
crawl.log also supports status (404, 4xx, <0, 500..599, 404,410),
size (>1MB, 100KB..10MB; units are powers of 1024), duration (>10s,
500..2000 in ms), depth (hops from the seed: 0, >5), hops (the hop
path, e.g. hops:*E), url, host (includes subdomains), via and viahost
(the URI it was discovered from, and its host), type and annot.
- Bare words and line terms are highlighted in the results.
Filtering keeps the viewer's stateless, byte-position paging and works
across checkpoint logs: each request scans forward or backward from pos
until it has enough matches or reaches the start or end, streaming a
progress bar to the browser. A scan stops after 5 minutes, offering a
link to continue from where it stopped, or as soon as the client
disconnects. Regex matching is aborted at the same deadline to guard against
catastrophic backtracking. The query is carried through all navigation links.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-02 16:54:19 +09:00
Alex Osborne and Claude Opus 5.5
b1a36159ae
Paged log viewer: view a log together with its checkpoint generations
...
Checkpointing renames each log in place (crawl.log becomes
crawl.log.cp00001-<timestamp>) and starts a new one, so earlier entries
could only be viewed one file at a time. The paged viewer now offers an
"include N checkpoint logs" link (all=y) that shows the log and its
rotated generations in the current launch as one virtual file, oldest
first, with each run of lines labelled with its file.
Positions in the virtual file stay valid across checkpoints, since a
rotated file keeps its place and the new log continues after it, and
they are converted when toggling so the view stays where it was.
Gzipped generations can't be read at arbitrary positions and are
left out.
LogSeries opens the files and fixes their lengths up front, so a
rotation mid-request doesn't change what is read. Paging uses a new
FilteredLineScanner over a LogSeries instead of FileUtils.pagedLines,
which can only read a single file. Results are the same for every
position the navigation links generate; a hand-edited pos in the
middle of a line now starts at that line rather than the next one.
Lines are capped at 256 KiB so a file with few or no newlines can't
exhaust the heap.
Also fixes in the crawl.log rendering: a rotated crawl.log gets the
same status highlighting, status codes unknown to Jetty (e.g. 999) and
numbers too long to be a status no longer fail the page, and the file
name, status titles and navigation links are HTML escaped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-02 16:46:36 +09:00
Alex Osborne
838523d33a
BotBlockDetector: enable in default profiles
2026-10-02 09:24:44 +09:00
dependabot[bot]
4f26ed82f2
Bump actions/setup-java from 6.0.0 to 6.0.1
...
Bumps [actions/setup-java](https://github.com/actions/setup-java ) from 6.0.0 to 6.0.1.
- [Release notes](https://github.com/actions/setup-java/releases )
- [Commits](https://github.com/actions/setup-java/compare/v6.0.0...v6.0.1 )
---
updated-dependencies:
- dependency-name: actions/setup-java
dependency-version: 6.0.1
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com >
2026-10-02 08:51:47 +09:00
dependabot[bot]
c36460a0f3
Bump the minor-and-patch group with 13 updates
...
Bumps the minor-and-patch group with 13 updates:
| Package | From | To |
| --- | --- | --- |
| [org.eclipse.jetty:jetty-bom](https://github.com/jetty/jetty.project ) | `12.0.38` | `12.0.39` |
| [org.eclipse.jetty.ee10:jetty-ee10-bom](https://github.com/jetty/jetty.project ) | `12.0.38` | `12.0.39` |
| [org.slf4j:slf4j-bom](https://github.com/qos-ch/slf4j ) | `2.0.18` | `2.0.20` |
| [org.apache.maven.plugins:maven-compiler-plugin](https://github.com/apache/maven-compiler-plugin ) | `3.15.0` | `3.16.0` |
| [org.apache.maven.plugins:maven-surefire-plugin](https://github.com/apache/maven-surefire ) | `3.5.6` | `3.6.0` |
| [org.codehaus.mojo:versions-maven-plugin](https://github.com/mojohaus/versions ) | `2.21.0` | `2.22.0` |
| [org.webjars:webjars-locator-lite](https://github.com/webjars/webjars-locator-lite ) | `1.1.4` | `1.1.5` |
| org.webjars.npm:codemirror__commands | `6.10.4` | `6.11.0` |
| org.webjars.npm:codemirror__state | `6.7.1` | `6.7.4` |
| org.webjars.npm:codemirror__view | `6.43.7` | `6.43.11` |
| org.webjars.npm:marijn__find-cluster-break | `1.0.3` | `1.0.4` |
| [com.rabbitmq:amqp-client](https://github.com/rabbitmq/rabbitmq-java-client ) | `5.35.0` | `5.36.0` |
| [at.yawk.lz4:lz4-java](https://github.com/yawkat/lz4-java ) | `1.11.2` | `1.12.0` |
Updates `org.eclipse.jetty:jetty-bom` from 12.0.38 to 12.0.39
- [Release notes](https://github.com/jetty/jetty.project/releases )
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.38...jetty-12.0.39 )
Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.38 to 12.0.39
- [Release notes](https://github.com/jetty/jetty.project/releases )
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.38...jetty-12.0.39 )
Updates `org.eclipse.jetty.ee10:jetty-ee10-bom` from 12.0.38 to 12.0.39
- [Release notes](https://github.com/jetty/jetty.project/releases )
- [Commits](https://github.com/jetty/jetty.project/compare/jetty-12.0.38...jetty-12.0.39 )
Updates `org.slf4j:slf4j-bom` from 2.0.18 to 2.0.20
- [Release notes](https://github.com/qos-ch/slf4j/releases )
- [Commits](https://github.com/qos-ch/slf4j/compare/v_2.0.18...v_2.0.20 )
Updates `org.apache.maven.plugins:maven-compiler-plugin` from 3.15.0 to 3.16.0
- [Release notes](https://github.com/apache/maven-compiler-plugin/releases )
- [Commits](https://github.com/apache/maven-compiler-plugin/compare/maven-compiler-plugin-3.15.0...maven-compiler-plugin-3.16.0 )
Updates `org.apache.maven.plugins:maven-surefire-plugin` from 3.5.6 to 3.6.0
- [Release notes](https://github.com/apache/maven-surefire/releases )
- [Commits](https://github.com/apache/maven-surefire/compare/surefire-3.5.6...surefire-3.6.0 )
Updates `org.codehaus.mojo:versions-maven-plugin` from 2.21.0 to 2.22.0
- [Release notes](https://github.com/mojohaus/versions/releases )
- [Changelog](https://github.com/mojohaus/versions/blob/master/ReleaseNotes.md )
- [Commits](https://github.com/mojohaus/versions/compare/2.21.0...2.22.0 )
Updates `org.webjars:webjars-locator-lite` from 1.1.4 to 1.1.5
- [Release notes](https://github.com/webjars/webjars-locator-lite/releases )
- [Commits](https://github.com/webjars/webjars-locator-lite/compare/webjars-locator-lite-1.1.4...webjars-locator-lite-1.1.5 )
Updates `org.webjars.npm:codemirror__commands` from 6.10.4 to 6.11.0
Updates `org.webjars.npm:codemirror__state` from 6.7.1 to 6.7.4
Updates `org.webjars.npm:codemirror__view` from 6.43.7 to 6.43.11
Updates `org.webjars.npm:marijn__find-cluster-break` from 1.0.3 to 1.0.4
Updates `com.rabbitmq:amqp-client` from 5.35.0 to 5.36.0
- [Release notes](https://github.com/rabbitmq/rabbitmq-java-client/releases )
- [Commits](https://github.com/rabbitmq/rabbitmq-java-client/compare/v5.35.0...v5.36.0 )
Updates `at.yawk.lz4:lz4-java` from 1.11.2 to 1.12.0
- [Release notes](https://github.com/yawkat/lz4-java/releases )
- [Changelog](https://github.com/yawkat/lz4-java/blob/main/CHANGES.md )
- [Commits](https://github.com/yawkat/lz4-java/compare/v1.11.2...v1.12.0 )
---
updated-dependencies:
- dependency-name: org.eclipse.jetty:jetty-bom
dependency-version: 12.0.39
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
dependency-version: 12.0.39
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-bom
dependency-version: 12.0.39
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.slf4j:slf4j-bom
dependency-version: 2.0.20
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-compiler-plugin
dependency-version: 3.16.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: org.apache.maven.plugins:maven-surefire-plugin
dependency-version: 3.6.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: org.codehaus.mojo:versions-maven-plugin
dependency-version: 2.22.0
dependency-type: direct:development
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: org.webjars:webjars-locator-lite
dependency-version: 1.1.5
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__commands
dependency-version: 6.11.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__state
dependency-version: 6.7.4
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:codemirror__view
dependency-version: 6.43.11
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: org.webjars.npm:marijn__find-cluster-break
dependency-version: 1.0.4
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: minor-and-patch
- dependency-name: com.rabbitmq:amqp-client
dependency-version: 5.36.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
- dependency-name: at.yawk.lz4:lz4-java
dependency-version: 1.12.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: minor-and-patch
...
Signed-off-by: dependabot[bot] <support@github.com >
2026-10-02 08:51:34 +09:00
dependabot[bot]
a0771568c2
Bump org.apache.groovy:groovy-bom from 5.1.1 to 6.0.0
...
Bumps [org.apache.groovy:groovy-bom](https://github.com/apache/groovy ) from 5.1.1 to 6.0.0.
- [Commits](https://github.com/apache/groovy/commits )
---
updated-dependencies:
- dependency-name: org.apache.groovy:groovy-bom
dependency-version: 6.0.0
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com >
2026-10-02 08:51:13 +09:00
Alex Osborne
a19bd61a33
BotBlockDetector: add aws-waf, kasada, lua-anti-ddos, perimeterx
2026-10-02 08:25:52 +09:00
Alex Osborne
022f1d8050
Bump jackson-bom from 2.22.2 to 2.22.3
2026-10-02 07:44:54 +09:00
Alex Osborne
f6a14ac1f6
Implement invoker commands for older browsers like Firefox ESR
...
Fixes #773
2026-10-01 19:05:53 +09:00
Shubham Padkonde
16deaaf4eb
Decode JavaScript Unicode code-point escapes in extracted URLs ( #772 )
...
Signed-off-by: Shubham Padkonde <shubhampadkonde12@gmail.com >
2026-09-22 17:21:42 +09:00
Francesco Marinoni Moretto
ed2805f6b7
Extract URIs nested inside JSON arrays ( #771 )
...
`parse()` called `propertyStream()` on array nodes, which Jackson returns empty
for anything that is not an object, so URIs inside an array held as an object
property value were never extracted. Those array elements now go through the same
value handling as object values, which is also what lets a bare URI string inside
an array be extracted and not only an object holding one.
A known limit this does not change: `parse()` still starts from
`rootNode.properties()`, so a document whose root is itself an array yields no
links.
2026-09-22 17:16:43 +09:00
Alex Osborne
89294d11ba
Merge pull request #770 from internetarchive/ato/fix-chunked-digest
...
FetchHTTP: digest chunked responses after transfer decoding
2026-09-17 17:20:59 +09:00