Guava 12 from hbase-client is closer to the root of the dependency tree than guava 17 from webarchive-commons so Maven prefers it. But recent changes to heritrix-commons rely on classes in the newer version of Guava. So let's ensure webarchive-commons wins.
Hopefully this doesn't break the hbase module, I have no way of testing it.
Fixes#311
CrawlURI already had the accessor method, and the use of the constant
was a bit inconsistent. This change adds the corresponding mutator
method to make working with the CrawlURI history a bit simpler.
We were seeing OOME due to large youtoube-dl json (for playlists and
such). So instead of storing the json in ram, stream through it, and
stash the contents in an thread-local anonymous tempfile so it can be
written to to warc.
and equivalent for other sites. Usually we find these links through
normal link extraction, but we have the info here, so we may as well use
it to make sure.
Jetty 9.4.12+ is required for TLS to work correctly under JDK11
(due to SSL handshake failures). In order to upgrade
Jetty we also need to upgrade Restlet.
There is one intentional change in behaviour to simplify
upgrading. We remove a workaround for an old [webkit bug]
where the browser claimed to prefer application/xml. The
bug was fixed in 2011.
[webkit bug]: https://bugs.webkit.org/show_bug.cgi?id=27267
Summary of Jetty API changes:
- package names changed such as org.mortbay -> org.eclipse
- SocketConnector and SslSocketConnector merged to ServerConnector
- HashUserRealm split into UserStore and HashLoginService
- SecurityHandler -> ConstraintSecurityHandler
Summary of Restlet API changes:
- some classes have moved package (Request, Response, Router etc)
- ServerResource replaces Resource
- represent(), acceptRepresentation() renamed to get(), post()
- constructors were replaced by an init() method
- setModifiable() was removed
- getRequest().getEntityAsForm() -> new Form(entity)
- Guard -> ChallengeAuthenticator
Fixes#275
tools.jar is no longer included in the JDK as of JDK 11 (per JEP 220) so
exclude it as a dependency. According to [HBASE-13963] it is only
required when compiling hbase itself and was unintentionally leaked
as a transitive dependency.
Fixes#265
[HBASE-13963]: https://issues.apache.org/jira/browse/HBASE-13963
There is already a clause for this in logging.properties, but it's using
log4j. It was dumping stack traces every time the client was dubious of
heritrix's self-signed certificate.
Why do we have so many identical log4j.xml's? 🤷♂️
... configure using rethinkdb url and segment id, instead of write url,
which means it can work if the segment gets reassigned and so forth
***backward incompatible change***
- make sure we "promote" segments (persist them to hdfs) after we're
entirely finished writing to theme; in other words, when we reach the
FINISH state, not while STOPPING
- shut down the promoter thread at crawl finish
plus a couple of tweaks and cleanups
