Jetty 9.4.12+ is required for TLS to work correctly under JDK11
(due to SSL handshake failures). In order to upgrade
Jetty we also need to upgrade Restlet.
There is one intentional change in behaviour to simplify
upgrading. We remove a workaround for an old [webkit bug]
where the browser claimed to prefer application/xml. The
bug was fixed in 2011.
[webkit bug]: https://bugs.webkit.org/show_bug.cgi?id=27267
Summary of Jetty API changes:
- package names changed such as org.mortbay -> org.eclipse
- SocketConnector and SslSocketConnector merged to ServerConnector
- HashUserRealm split into UserStore and HashLoginService
- SecurityHandler -> ConstraintSecurityHandler
Summary of Restlet API changes:
- some classes have moved package (Request, Response, Router etc)
- ServerResource replaces Resource
- represent(), acceptRepresentation() renamed to get(), post()
- constructors were replaced by an init() method
- setModifiable() was removed
- getRequest().getEntityAsForm() -> new Form(entity)
- Guard -> ChallengeAuthenticator
Fixes#275
There is already a clause for this in logging.properties, but it's using
log4j. It was dumping stack traces every time the client was dubious of
heritrix's self-signed certificate.
Why do we have so many identical log4j.xml's? 🤷♂️
These changes come from a private fork of H3, originally made by Kenji
Nagahashi, to create org.archive.crawler.frontier.PullingBdbFrontier,
which we intend to merge into 'contrib' of the official version in the
near future.
Sometimes our crawl jobs get stuck in the NEW state (due to hbase
problems). But the checkpoint service has started and sees fit to
checkpoint every five minutes. Evidently, at this stage the seeds have
not been queued yet, the frontier is empty. Thus, if we try to resume
from one of these checkpoints, the crawl has an empty frontier and ends
immediately.
The fix is to avoid checkpointing in this NEW state before crawling has
really started.
This is a bad bug. Stupid, too. When multiple crawls are running with
`DecideRuleSequence.logToFile` enabled, all the scope log lines from all
the crawls are going to all the scope logs for all the crawls. It was
because of this line:
Logger logger = Logger.getLogger(logName + ".log");
`logName` here is "scope" normally, for all crawl jobs.
`Logger` is shared across the whole java vm, so
`Logger.getLogger("scope.log")` returns the same logger for every
crawl job. Each time a crawl starts, it adds another output file to
the logger.
Fix is to give the logger a name specific to the crawl job.
😳🥵🥶💩
* origin/master:
Setup TravisCI
Set fetch status on curis when testing link extraction
No link extraction on URI not successfully downloaded
Fixes issue #158 : Noisy alerts about 401s without auth challenge (#159)
Make Content-Location header url INFERRED not REFFER hop type since Content-Location is not for redirection (#151)
fixes for kafka 0.9 (?)
upgrade to kafka 0.9
somewhat ugly fix to handle exceptions from the bean browser like java.lang.RuntimeException: not implemented at org.archive.modules.fetcher.BdbCookieStore$RestrictedCollectionWrappedList.get(BdbCookieStore.java:92)
Alex Osborne