When there's less than 5 GiB free BDB throws DiskLimitException which
Heritrix will likely be unable to handle gracefully and crawl job will
break in various ways. #499
This policy obeys robots.txt but ignores meta nofollow. We're
increasingly frequently encountering sites which fail to be archived
properly due to usage of this meta tag but where the robots.txt file
contains reasonable rules. In many cases it seems to be unintentional,
and we've even had questions from webmasters asking why archiving was
failing when they were explicitly allowing us in robots.txt.
This avoids creating browser processes for jobs that have it disabled.
If a job has it disabled by default but enables it with a sheet we'll
start it when needed.
We still connect on start when enabled to provide early error feedback.
This ensures that when we later compare the context in processEmbed()
we don't need to deal with variants like srcSet or SRCSET. Note that
we're already sometimes lowercasing it later in HTMLLinkContext.get().
Fixes#477.
- instead of "borrowing" the configured proxy from
the fetchHttp bean, use proxy values defined via
global options, to avoid interference with other
jobs running in parallel (or at least make them
explicit).
The "fetchHttp" bean also uses these settings,
if no bean specific settings are used.
- remove the "enableDnsOverHttpResolves", and rely
on a non-empty "dnsOverHttpServer" value to signal
that DoH should be used.
- use the "DohResolver" from the dnsjava library
to make DoH lookups
- to enable and configure it, add two new
properties
* "enableDnsOverHttpResolves" (boolean)
* "dnsOverHttpServer" URL to the DoH Server
- as one use case for DoH is being located
behind a firewall, also support using a proxy
to access the DoH server; the proxy from
the FetchHTTP bean is reused in that case
Fixes#211
Disables browser features like phishing protection, translation, crash
and metrics reporting, OS keychain integration and such that can cause
unnecessary network traffic or that can cause the browser to get stuck
in a prompt. Some of these options may be unnecessary in headless mode,
but we may as well keep them in case in future we want to support headed
mode for debugging.
We should only crawl sitemaps and sitemap links if they are in the
primary SURT scope. Otherwise we'll start crawling the sitemaps of
every site an embedded resource is pulled in from even when they
should be out of scope.
Fixes#469
testSocksProxy() is failing on the GitHub action CI runs but I can't
reproduce the issue locally. I'm just guessing this is what's causing it
but this should be fixed anyway to avoid future intermittent failures.
By replacing the custom TestSuite with junit 4 @BeforeClass and
@AfterClass annotations running individual tests by themselves now works
correctly.
Since FetchHTTPTest now just contains the utility code for the test
servers we rename it to FetchHTTPTestServers. FetchHTTPTests (which
contains the actual tests) can now follow the standard naming
scheme as FetchHTTPTest.
Alex Osborne