Alex Osborne
e1ac68f533
docs: Correct bean-example path for FetchHTTP2
2025-05-13 14:16:36 +09:00
Alex Osborne
fee534c3db
docs: Get version from pom.xml
2025-05-13 14:15:20 +09:00
Alex Osborne
81ac46be45
docs: Catch parse errors and continue generating
...
Looks like the javalang parser doesn't handle some newer syntax.
For now just catch the errors. Long term we probably need to switch
to a different parser.
2025-05-13 14:09:09 +09:00
Alex Osborne
1827d93fe6
[maven-release-plugin] prepare for next development iteration
2025-05-13 12:25:24 +09:00
Alex Osborne
dd40970f15
[maven-release-plugin] prepare release 3.9.0
2025-05-13 12:25:16 +09:00
Alex Osborne
d60161b6be
Merge pull request #650 from internetarchive/deps-3.9
...
Upgrade dependencies (notably junit 5) for 3.9.0 release
2025-05-09 22:05:42 +09:00
Alex Osborne
e09106cc15
Document HTTP/2 and HTTP/3 support
2025-05-09 21:03:38 +09:00
Alex Osborne
4a2cdfaa81
FetchHTTP2: Print the jetty-quiche-native warning earlier at build time
2025-05-09 20:40:33 +09:00
Alex Osborne
8692bdf655
Add WARC-Protocol to request records too when using HTTP/2 or HTTP/3
...
While this could be inferred from the response, it's more accurate to
include them on the request as the request was not really HTTP/1.1.
2025-05-09 19:54:22 +09:00
Alex Osborne
1e502fba10
Limit permissions on github actions
2025-05-09 18:52:32 +09:00
Alex Osborne
8e177f778b
Update CHANGELOG for 3.9.0
2025-05-09 18:52:21 +09:00
Alex Osborne
f5732e1a98
Bump jsch from 0.2.26 to 2.27.0
2025-05-09 18:19:21 +09:00
Alex Osborne
eb1205f6d5
Upgrade all tests to junit 5
2025-05-09 00:07:18 +09:00
Alex Osborne
bd8d6fb2c5
Bump commons-io from 2.18.0 to 2.19.0
2025-05-08 17:41:39 +09:00
Alex Osborne
d25a0b96ca
Bump spring from 6.2.5 to 6.2.6
2025-05-08 17:41:14 +09:00
Alex Osborne
083a60b04c
Bump gson from 2.12.1 to 2.13.1
2025-05-08 17:40:10 +09:00
Alex Osborne
d5e91eb1f3
Bump pdfbox from 3.0.4 to 3.0.5
2025-05-08 17:36:12 +09:00
Alex Osborne
227d51f467
Bump jetty from 12.0.16 to 12.0.17
2025-05-08 17:31:49 +09:00
Alex Osborne
4465c1b9f6
Bump restlet from 2.6.0-m2 to 2.6.0-rc1
2025-05-08 17:31:30 +09:00
Alex Osborne
1e3b0f6c39
Add /target, __pycache__, adhoc.keystore, heritrix_dmesg.log to .gitignore
2025-05-08 17:27:01 +09:00
Alex Osborne
9807b93db3
Merge pull request #649 from internetarchive/http2
...
FetchHTTP2: A new fetch module for HTTP/2 and HTTP/3
2025-05-08 17:25:08 +09:00
Alex Osborne
fcfef68aff
FetchHTTP2: A new fetch module for HTTP/2 and HTTP/3
...
This uses Jetty HttpClient since it speaks both protocols, and we
already have it as a dependency via Restlet. This doesn't support all
the options of FetchHTTP, notably proxy and POST requests are missing.
Jetty currently has the HTTP/3 client marked as "experimental, not for
production use" so we disable it by default and don't ship the large
quiche native jar it requires. It does seem to work OK though, at least
in my limited testing so far. The HTTP/3 support currently only responds
to Alt-Svc headers not other ways of discovering HTTP/3 availability
(e.g. HTTPS DNS record).
Fetches that were made via HTTP/2 or HTTP/3 are annotated 'h2' and 'h3'
in the crawl.log. The messages are recorded in the WARC files using
HTTP/1.1 syntax with a WARC-Protocol header. FetchHTTP2 also currently
records HTTP/1.1 messages without transfer-encoding rather than the raw
wire messages.
2025-04-29 12:47:57 +09:00
Alex Osborne
bc9b21e2ee
Merge pull request #648 from internetarchive/jetty-12
...
Update to Jetty 12.0.16 and Restlet 2.6.0-M2
2025-04-29 12:47:25 +09:00
Alex Osborne
db16b60960
Fix 'anypath' URL generation and avoid unnecessary usages of it
...
Restlet 2.6 refuses paths with double slash in the static path and we
were inadvertently generating them.
2025-04-28 17:09:33 +09:00
Alex Osborne
0f5358ffef
Exclude jetty-quiche-native from restlet
...
It's very large and we don't use Restlet's HTTP/3 support currently.
2025-04-28 16:29:22 +09:00
Alex Osborne
bc259728a0
Update to Jetty 12.0.16 and Restlet 2.6.0-M2
...
The Jetty API has changed, which mostly affects test code.
Jetty now does a strict SNI host check which unfortunately causes it to
return "SNI error" for our existing ad-hoc certificates. For now, I've
disabled it to avoid breaking existing deployments but added a
--sni-host-check command-line option so you can re-enable it if
you've configured your own certificate appropriately.
2025-04-24 17:40:20 +09:00
Alex Osborne
d73e2f6b87
Update CHANGELOG.md for #647
2025-04-24 11:02:17 +09:00
Alex Osborne
91cd4b815c
Merge pull request #647 from adam-miller/add-histotable-serializer-for-HighestUriPrecedenceProvider
...
Add histotable serializer for highest uri precedence provider
2025-04-24 10:58:03 +09:00
Adam Miller
02c4c8ee09
feat: add unit test for HighestUriPrecedenceProvider Kryo serialization
2025-04-23 16:25:58 -07:00
Adam Miller
fdada4c5ec
chore: formatting
2025-04-23 15:20:40 -07:00
Adam Miller
cb98bfc522
chore: fix line endings
2025-04-22 17:33:32 -07:00
Adam Miller
e7fe566919
fix: autoregister Histotable and HighestUriPrecedenceProvider
2025-04-22 17:29:52 -07:00
Adam Miller
75377c5a3b
fix: FieldSerializer wasn't passing unit tests. Switch to JavaSerializer
2025-04-21 18:16:47 -07:00
Adam Miller
c7784f18e9
feat: Add serializer for Histotable used by HighestUriPrecedenceProvider
2025-04-21 17:20:38 -07:00
Alex Osborne
e9b60404a6
[maven-release-plugin] prepare for next development iteration
2025-04-01 17:43:31 +09:00
Alex Osborne
2323e2251e
[maven-release-plugin] prepare release 3.8.0
2025-04-01 17:43:25 +09:00
Alex Osborne
edb3f3aa00
Merge pull request #645 from internetarchive/deps-3.8
...
Update dependencies and fix some warnings for 3.8.0 release
2025-04-01 17:41:02 +09:00
Alex Osborne
fb9b286d95
Update CHANGELOG for 3.8.0
2025-04-01 17:38:06 +09:00
Alex Osborne
575a2653b9
Fix warning: assert statement with side effects
2025-04-01 16:27:13 +09:00
Alex Osborne
b8dbe8bb2d
Remove unused utility class LogUtils
2025-04-01 16:25:44 +09:00
Alex Osborne
ed72112d9c
Fix missing try-with-resources warning
2025-04-01 16:19:58 +09:00
Alex Osborne
1a605cf9a6
Remove unused utility class JavaLiterals
...
Apache Commons StringEscapeUtils which we use for other things provides
this anyway.
2025-04-01 16:11:10 +09:00
Alex Osborne
13052a1465
Fix more broken references in javadoc
2025-04-01 16:06:10 +09:00
Alex Osborne
f956b2ec1f
Fix broken references in javadoc
2025-04-01 15:59:38 +09:00
Alex Osborne
7f76f5abd1
FetchHTTPTest: Disable testHostHeaderDefaultPort due to random failures
...
Probably some of the GitHub test runners are getting blocked.
2025-04-01 15:37:21 +09:00
Alex Osborne
b17e3fcab3
Bump beanshell from 2.0b5 to 2.0b6
2025-04-01 15:27:22 +09:00
Alex Osborne
7ff1078d21
Fix warning about implicit narrowing conversion for contentSinceCheck
2025-03-31 15:31:04 +09:00
Alex Osborne
f17000e14f
Fix warning: Building a command line with string concatenation
2025-03-31 15:17:28 +09:00
Alex Osborne
60f512b4b0
Fix compiler warning: deprecated item is not annotated with @Deprecated
2025-03-31 15:12:21 +09:00
Alex Osborne
f6b083ecfb
Remove usages of deprecated Integer and Long constructors
2025-03-31 15:04:00 +09:00
Alex Osborne