littleproxy has not been updated since 2017 and has various out of date
dependencies. Since we're using jetty for HTTP server testing anyway, we
may as well use it as the proxy too.
This eliminates a few more very old dependencies that aren't in Maven Central.
Our direct usage of the unsupported sun.reflect.ReflectionFactory JDK API (which newer compilers complain about) is no longer needed as Kryo now has a SerializingInstantiatorStrategy that does roughly the same thing.
The digest authentication method requires an initial request to get a challenge to use for the subsequent authenmtication digest. This initial request is not an authentication attempt and should not be handled as a failed such.
The initial sleepMs value will be the negative of the current time minus 6 seconds. Using that value to calculate lastFailureTime will increase that large negative number with 6000 milliseconds for every failure which means that rate limiting won't occur until after millions of failed attempts.
When there's less than 5 GiB free BDB throws DiskLimitException which
Heritrix will likely be unable to handle gracefully and crawl job will
break in various ways. #499
- instead of "borrowing" the configured proxy from
the fetchHttp bean, use proxy values defined via
global options, to avoid interference with other
jobs running in parallel (or at least make them
explicit).
The "fetchHttp" bean also uses these settings,
if no bean specific settings are used.
- remove the "enableDnsOverHttpResolves", and rely
on a non-empty "dnsOverHttpServer" value to signal
that DoH should be used.
- use the "DohResolver" from the dnsjava library
to make DoH lookups
- to enable and configure it, add two new
properties
* "enableDnsOverHttpResolves" (boolean)
* "dnsOverHttpServer" URL to the DoH Server
- as one use case for DoH is being located
behind a firewall, also support using a proxy
to access the DoH server; the proxy from
the FetchHTTP bean is reused in that case
Fixes#211
We don't need to support Java < 7 anymore. This simplifies the
code and should improve portability to platforms our old version of
JNA doesn't support like Apple Silicon.
Fixes#467
Since the cache is global we only need to configure it once, not every
time we start a job. By moving it we keep the code that interfaces with
dnsjava centralized in FetchDNS.
Looks like dnsjava no longer uses a cleaner thread so remove the comment
referring to it.
The parsing throws StringIndexOutOfBoundsException when the version
does not contain '.' such as version "17". This check could never
actually trigger as the class cannot even be loaded on 1.7 or earlier,
so there's no reason to keep it.
Fixes#439
Each direct subclass of BaseResource defines an identical getEngine()
method so let's pull it up to BaseResource. We can also don't need the
type cast anymore as BaseResource.getApplication() does it for us.
Thread interruption and certain other exceptions can cause a toe thread
to exit without informing the frontier that the current CrawlURI is
finished. This causes the job to get permanently stuck in the STOPPING
state.
This change adds a section to the finally block that will finish any
unfinished CrawlURI.
We also move the continueCheck() call after setCurrentCuri() to ensure
there's no window where InterruptedException can be thrown after the
frontier returns the next CrawlURI but before it gets assigned to
currentCuri.
Fixes#420
JavaDoc comments need to be directly above a class, method or field
declaration to be used by IDEs and generated documentation. We had some
doc comments that were attached to instance initialization blocks and
thus were being ignored. This invalid doc comment positioning was
presumably an accidental consequence of the conversion of fields to
KeyedProperties.
This change moves most of the dangling doc comments to setter methods
where it can be seen by tools. There were a couple of dangling doc
comments above package statements. These are moved above the class
declaration or removed entirely when empty.
I noticed there was a non-functional and undocumented -r command-line
option which automatically runs a given job when Heritrix starts. I
suspect this was never finished due to the bug that was fixed in commit
643f16d where launch() returns before the job has been launched.
This option seems like it would be very useful so you could run a job
from cron or another scheduling program without having to use the
REST API to start it. Therefore I've enabled it and extended it so it
also unpauses, waits for the crawl to finish and then exits.
I think the sleep is supposed to make launch() not return until the job
has actually been launched but it doesn't work as launch()
and getCrawlController() are both synchronized therefore the
launcher thread can't actually call startContext() until launch()
returns after sleeping.
So let's replace the sleep call with join and unsynchronize launch()
so it doesn't deadlock. All the relevant methods it calls seem to be
synchronized so I think it's no worse to not synchronize it itself.
Lots of 1 second sleeps add up fast. On my PC this reduces the runtime
of `mvn clean test` from about 3m 40s to about 1m 50s.
Ideally we'd probably re-architect some this to use a notification
mechanism instead of polling but that's easier said than done and this
is a pretty big improvement by itself.
I think the sleep is supposed to make launch() not return until the job
has actually been launched but it doesn't work as launch()
and getCrawlController() are both synchronized therefore the
launcher thread can't actually call startContext() until launch()
returns after sleeping.
So let's replace the sleep call with join and unsynchronize launch()
so it doesn't deadlock. All the relevant methods it calls seem to be
synchronized so I think it's no worse to not synchronize it itself.
Alex Osborne