Commit Graph

700 Commits

Author SHA1 Message Date
Clara Wiatrowski eea3914059 Add missing import 2020-04-21 16:00:21 +02:00
Clara Wiatrowski b61f7e574e Add real crawlStatus in the crawlReport 2020-04-17 14:53:00 +02:00
Andrew Jackson ff05de44b6 [maven-release-plugin] prepare for next development iteration 2020-03-04 23:05:41 +00:00
Andrew Jackson a1bdcb1be7 [maven-release-plugin] prepare release 3.4.0-20200304 2020-03-04 23:05:33 +00:00
Andy Jackson e0c82f70e9 Merge pull request #289 from netarchivesuite/bdb-frontier-access
Bdb frontier access
2020-03-04 13:41:27 +00:00
Jonathan Leitschuh bc6a15b41d Use HTTPS to resolve dependencies in Maven Build where possible 2020-02-11 17:48:20 +09:00
Lauren Ko 8c1c8009c6 Fix stream closed exception for Paged view 2020-01-30 16:48:38 -06:00
Alex Osborne b45f774066 Merge pull request #300 from hennekey/fix-299
Use Guice instead of custom implementation
2020-01-30 19:12:05 +09:00
Alex Osborne a4ce59dbd6 Fix stream closed exception by not closing output stream
ServerCall.writeResponseBody() flushes it after we return so it must
remain open.

Fixes #305
2020-01-25 08:25:59 +09:00
Tim Hennekey 33458f1518 Fix assertions
By using assertEquals and seting the expected and actual values, the
failure messages become a bit more useful.
2020-01-22 17:13:58 -05:00
Noah Levitt 784b3a5dec fix logging config
by setting system property java.util.logging.config.file, because new
version restlet reconfigures logging after heritrix has already
configured it
2020-01-22 13:39:24 -08:00
Alex Osborne 34be934da5 Fix 'Method Not Allowed' on POST of config editor form
Under Restlet 2 `getVariants()` always returns null for a POST so our
post() method was never called and 405 Method Not Allowed was returned.
We don't need content negotiation for the POST response anyway so let's
instead override the no-variants post() like we do for put().

Fixes #293
2020-01-15 16:51:52 +09:00
jkafader 3a8b58998c Merge pull request #285 from nlevitt/warc-writer-chain
Warc writer chain
2020-01-13 16:39:42 -08:00
Nicholas Clarke c611c46c8a Merged frontier-management with upgrade to bdb 7 2019-11-15 10:33:13 +01:00
Noah Levitt 74c0739bb0 fix line ending and indentation issues 2019-10-11 13:49:49 -07:00
Adam Miller 0cee24922f AssignmentLevelSurtQueueAssignmentPolicy.java - Add support for forced queue assignment and parallel queues
URIAuthorityBasedQueueAssignmentPolicy.java - Add interoperability between forced queue assignment and parallel queues
QuotaEnforcer.java - Fix javadoc to match default behavior
2019-10-11 19:44:35 +00:00
Alex Osborne eeddfd763f Override PUT so it doesn't change the file extension
Fixes #282 and HER-1907
2019-08-28 12:46:29 +09:00
Alex Osborne 5612fa249b Use super.getVariants() rather than super.getVariants(GET)
This was a regression introduced in the upgrade to Restlet 2. I
encountered a NullPointerException here when upgrading and misunderstood
the cause of it. Since PUT and DELETE return no content they are
actually supposed to return null.
2019-08-27 23:53:34 +09:00
Alex Osborne 38c44e8e8e Add missing UUID import (interactive commit fail) 2019-08-12 17:50:35 +09:00
Alex Osborne 36003b8626 Fix digest authentication
In Restlet 2 it appears we need to use DigestAuthenticator.
(Previously both digest and basic auth were handled by the same
Guard class.)
2019-08-12 17:36:28 +09:00
Noah Levitt cefad96036 Merge pull request #276 from internetarchive/jdk11-upgrade-jetty
JDK11 support: upgrade to Jetty 9.4.19, Restlet 2.4.0 and drop JDK 7 support
2019-08-07 10:48:51 -07:00
Noah Levitt 6811f49bd9 Merge pull request #272 from internetarchive/jdk11-upgrade-surefire
JDK11 support: upgrade maven-surefire-plugin to 2.22.2
2019-08-07 10:48:29 -07:00
Alex Osborne 97b9c3c91c Fix restlet child component context warning 2019-08-06 17:13:29 +09:00
Alex Osborne 6c2afc36b0 Update statistics assertions as Jetty 9 produces different headers 2019-08-06 17:13:02 +09:00
Alex Osborne 4012a5f4dd Fix 'No session data store configured' test error 2019-08-06 16:56:10 +09:00
Alex Osborne 0782fb0481 JDK11: upgrade engine to jetty 9.4.19 and restlet 2.4.0
Jetty 9.4.12+ is required for TLS to work correctly under JDK11
(due to SSL handshake failures). In order to upgrade
Jetty we also need to upgrade Restlet.

There is one intentional change in behaviour to simplify
upgrading. We remove a workaround for an old [webkit bug]
where the browser claimed to prefer application/xml. The
bug was fixed in 2011.

[webkit bug]: https://bugs.webkit.org/show_bug.cgi?id=27267

Summary of Jetty API changes:
- package names changed such as org.mortbay -> org.eclipse
- SocketConnector and SslSocketConnector merged to ServerConnector
- HashUserRealm split into UserStore and HashLoginService
- SecurityHandler -> ConstraintSecurityHandler

Summary of Restlet API changes:
- some classes have moved package (Request, Response, Router etc)
- ServerResource replaces Resource
- represent(), acceptRepresentation() renamed to get(), post()
- constructors were replaced by an init() method
- setModifiable() was removed
- getRequest().getEntityAsForm() -> new Form(entity)
- Guard -> ChallengeAuthenticator

Fixes #275
2019-08-06 15:16:37 +09:00
Alex Osborne a2920c622d JDK11 support: explicitly depend on JAXB
JAXB is no longer included in the JDK as of JDK 11 (JEP 320) so add it
as an external dependency.

Fixes #264
2019-08-02 16:47:20 +09:00
Alex Osborne 412885fd7d JDK11 support: upgrade maven-surefire-plugin to 2.22.2
Under JDK11 our old version of it throws ClassNotFoundExceptions when
tests load some builtin classes like javax.transaction.xa.Xid.

Fixes #266
2019-08-02 16:43:20 +09:00
Noah Levitt 7c31b0752a use WARCWriterChainProcessor for these tests 2019-06-12 12:56:49 -07:00
Noah Levitt 8c4c443a88 default chain in code 2019-06-12 12:54:00 -07:00
Noah Levitt 9435f761a6 configurable warc writer chain
exercised only lightly at this point
2019-06-11 13:45:51 -07:00
Noah Levitt 1bd8b713c6 quiet org.mortbay.log (jetty?) logging
There is already a clause for this in logging.properties, but it's using
log4j. It was dumping stack traces every time the client was dubious of
heritrix's self-signed certificate.

Why do we have so many identical log4j.xml's? 🤷‍♂️
2019-04-30 16:17:35 -07:00
Noah Levitt 79a0d34adf ExtractorYoutubeDL 2019-04-29 15:29:16 -07:00
Andrew Jackson b3961a2f96 [maven-release-plugin] prepare for next development iteration 2019-04-18 15:36:28 +01:00
Andrew Jackson c7c6141ee1 [maven-release-plugin] prepare release 3.4.0-20190418 2019-04-18 15:36:20 +01:00
Noah Levitt a60b2ef387 Merge pull request #249 from ruebot/remove-suffix-crawler-bean
Remove suffix from warcWriter since it is no longer used.
2019-04-10 11:15:37 -07:00
David Van Duzer 6c6e460c8d set of frontier management changes to support CrawlHQ module
These changes come from a private fork of H3, originally made by Kenji
Nagahashi, to create org.archive.crawler.frontier.PullingBdbFrontier,
which we intend to merge into 'contrib' of the official version in the
near future.
2019-04-08 17:15:24 -06:00
Noah Levitt 2c6ab5a44a replace System.err.println with logger.info 2019-04-04 17:06:01 -07:00
nruest 5ee7528369 Remove suffix from warcWriter since it is no longer used. 2019-03-29 12:40:51 -04:00
Andrew Jackson afa88c025e Ensure we start parsing full lines, for #239. 2019-03-13 21:00:19 +09:00
Andrew Jackson 91b18e9bbd Allow failed lookups to expire, for #234. 2019-02-21 23:04:40 +00:00
Andrew Jackson ad9e74d8ad [maven-release-plugin] prepare for next development iteration 2019-02-07 13:52:56 +00:00
Andrew Jackson 83c7044b22 [maven-release-plugin] prepare release 3.4.0-20190207 2019-02-07 13:52:49 +00:00
Andy Jackson 18d3bb23aa Merge pull request #231 from ukwa/synchronized-enqueuecounts
Add synchronized statements for internetarchive/heritrix3#221.
2019-02-07 13:49:50 +00:00
Andrew Jackson 317b19e7e2 [maven-release-plugin] prepare for next development iteration 2019-02-05 12:33:08 +00:00
Andrew Jackson 9c7d0c299d [maven-release-plugin] prepare release 3.4.0-20190205 2019-02-05 12:32:57 +00:00
Andrew Jackson ab34a54566 [maven-release-plugin] prepare for next development iteration 2019-02-05 12:29:33 +00:00
Andrew Jackson d1fc5cedb4 [maven-release-plugin] prepare release 3.4.0-20190205-2 2019-02-05 12:29:26 +00:00
Andrew Jackson 096ba4e9c3 [maven-release-plugin] prepare for next development iteration 2019-02-05 12:15:26 +00:00
Andrew Jackson 3b435cb0a9 Add synchronized statements for internetarchive/heritrix3#221. 2019-02-04 10:41:40 +00:00