WWWAuthenticationProtocolHandler and ProxyAuthenticationProtocolHandler unhelpfully throw exceptions if a 401 or 407 response has a larger than 16KB body or is missing the appropriate -Authenticate header.
FetchHTTP2 doesn't yet support HTTP auth, thus they aren't currently being used, so let's just remove them for now. This also adds regression tests so that if in the future we implement auth support, we don't accidentally reintroduce this.
Fixes#714Fixes#717
If there are multiple addresses then ignore 0.0.0.0. If the only address is 0.0.0.0 treat it as a DNS resolution failure. This should prevent us from throwing IllegalStateException later in DefaultServerCache.getHostFor().
Fixes#428
This uses Jetty HttpClient since it speaks both protocols, and we
already have it as a dependency via Restlet. This doesn't support all
the options of FetchHTTP, notably proxy and POST requests are missing.
Jetty currently has the HTTP/3 client marked as "experimental, not for
production use" so we disable it by default and don't ship the large
quiche native jar it requires. It does seem to work OK though, at least
in my limited testing so far. The HTTP/3 support currently only responds
to Alt-Svc headers not other ways of discovering HTTP/3 availability
(e.g. HTTPS DNS record).
Fetches that were made via HTTP/2 or HTTP/3 are annotated 'h2' and 'h3'
in the crawl.log. The messages are recorded in the WARC files using
HTTP/1.1 syntax with a WARC-Protocol header. FetchHTTP2 also currently
records HTTP/1.1 messages without transfer-encoding rather than the raw
wire messages.
The Jetty API has changed, which mostly affects test code.
Jetty now does a strict SNI host check which unfortunately causes it to
return "SNI error" for our existing ad-hoc certificates. For now, I've
disabled it to avoid breaking existing deployments but added a
--sni-host-check command-line option so you can re-enable it if
you've configured your own certificate appropriately.
When enabled this option causes regular links annotated with rel=nofollow to not be extracted. This is useful for sites that use rel=nofollow to hint crawler traps.
littleproxy has not been updated since 2017 and has various out of date
dependencies. Since we're using jetty for HTTP server testing anyway, we
may as well use it as the proxy too.
This eliminates a few more very old dependencies that aren't in Maven Central.
Our direct usage of the unsupported sun.reflect.ReflectionFactory JDK API (which newer compilers complain about) is no longer needed as Kryo now has a SerializingInstantiatorStrategy that does roughly the same thing.
This ensures that when we later compare the context in processEmbed()
we don't need to deal with variants like srcSet or SRCSET. Note that
we're already sometimes lowercasing it later in HTMLLinkContext.get().
Fixes#477.
testSocksProxy() is failing on the GitHub action CI runs but I can't
reproduce the issue locally. I'm just guessing this is what's causing it
but this should be fixed anyway to avoid future intermittent failures.
By replacing the custom TestSuite with junit 4 @BeforeClass and
@AfterClass annotations running individual tests by themselves now works
correctly.
Since FetchHTTPTest now just contains the utility code for the test
servers we rename it to FetchHTTPTestServers. FetchHTTPTests (which
contains the actual tests) can now follow the standard naming
scheme as FetchHTTPTest.
Only treat icons, stylesheets and (tentatively) resource preloads as
embedded resources. Treat most other link types as navigation links.
Previously we were treating all link tags as embedded resources
including links like 'next', 'prev', 'canonical', 'author' which could
pull in resources that should normally be considered out of scope.
For example crawling a page containing an author link lke:
<link href="http://twitter.com/example" rel="author">
Would end up transitively pulling 200+ MB of Twitter profile pages in
multiple languages into a crawl for which twitter.com is not even
supposed to be in scope at all.
Fixes#263
- do not send the scheme/host/port part of the request
even when using a proxy, if the scheme is https
- in case of a https request send over a proxy, do not
wrap the first input/output streams (which only contain
the data for the `CONNECT` request), but the ones after
them, which contain the actual data
Fixes#191
JavaDoc comments need to be directly above a class, method or field
declaration to be used by IDEs and generated documentation. We had some
doc comments that were attached to instance initialization blocks and
thus were being ignored. This invalid doc comment positioning was
presumably an accidental consequence of the conversion of fields to
KeyedProperties.
This change moves most of the dangling doc comments to setter methods
where it can be seen by tools. There were a couple of dangling doc
comments above package statements. These are moved above the class
declaration or removed entirely when empty.
Noah wrote in #280:
> Maybe we should just drop the test. The assumption when we wrote the
> test was that a race condition would not be so frequent in practice.
> We've seen that under the contrived conditions created by the test
> case, it is frequent. But that's ok
The arbitary value `25` was used but in prace it's quite possible
for more than 25 writing threads to have checked the cookie count
limit before adding their cookie. In practice we see Travis failing
on this test quite often, every few builds in fact.
I think using `threads.length` (i.e. 200) should cover the worst
case possibility where every thread reads a stale count and tries
to add their cookie.
Fixes#274
Alex Osborne