Commit Graph
721 Commits
Author SHA1 Message Date
Andrew Jackson 846f2bc2cc [maven-release-plugin] prepare release 3.4.0-20210621 2021-06-21 15:42:46 +01:00
Kristinn Sigurdsson 37913cbb2e Graceful UI shutdown 2021-06-18 10:56:20 +00:00
Andrew Jackson a5beb7d6e9 [maven-release-plugin] prepare for next development iteration 2021-06-18 09:41:30 +01:00
Andrew Jackson 30598f8402 [maven-release-plugin] prepare release 3.4.0-20210618 2021-06-18 09:41:24 +01:00
Andrew Jackson 3423cab6a6 [maven-release-plugin] prepare for next development iteration 2021-06-17 14:39:00 +01:00
Andrew Jackson 7368ded9c2 [maven-release-plugin] prepare release 3.4.0-20210617 2021-06-17 14:38:54 +01:00
Andy Jackson 5caa83d327 Only update last checkpoint stats if the checkpoint completed, for #392. 2021-05-28 12:42:17 +01:00
Andrew Jackson d3dfb34f83 [maven-release-plugin] prepare for next development iteration 2021-05-27 10:03:10 +01:00
Andrew Jackson 9dcbb240a6 [maven-release-plugin] prepare release 3.4.0-20210527 2021-05-27 10:03:04 +01:00
Andy Jackson a2c65f4618 Update dependencies for security issues, including changes to tests. 2021-05-26 13:39:30 +01:00
Andy Jackson c0dbfb2a0a Merge pull request #383 from ukwa/upgrade-dnsjava
Update to latest version of dnsjava, for #344
2021-05-20 21:05:59 +01:00
Andy Jackson 396467c535 Merge branch 'master' into sitemaps 2021-05-20 21:03:54 +01:00
Andy Jackson c7b1dfa198 Merge pull request #368 from webdev4422/patch-1
Fix misspell in comments
2021-05-20 20:58:41 +01:00
Andy Jackson 045b2516da Update to latest version of dnsjava, for #344
This commit updates to version 3.3.1 of the dnsjava library.

Incidentally, the restlet.org certificate is currently invalid, so this
patch switched to maven.restlet.com.
2021-05-20 20:44:19 +01:00
Alex Osborne a09479364a Merge pull request #379 from ldko/bean-url-missing-fix
Avoid error when bean properties have no url available
2021-05-12 09:31:27 +09:00
Lauren Ko ac33e2ef55 Avoid error when bean properties have no url available 2021-05-11 16:42:33 -05:00
Alex Osborne 71b3065172 Handle empty Optionals when browsing beans
addPresentableNestedNames() recursively walks the properties of beans
calling getters. Likely unintentionally this includes calling getClass()
and recursing into the reflection API. When running on JDK 11 the
reflection API has some methods (e.g. in java.lang.module) that return
instances of Optional. With the newer version of Spring's
BeanWrapperImpl encounters an Optional it attempts to unwrap it and
throws IllegalArgumentException if it is empty.

Let's fix this in two ways. Firstly let's avoid walking the reflection
API entirely as its irrelevant for the purposes of bean browsing by
not inspecting the properties of instances of java.lang.Class.

Secondly in case any Heritrix beans start using Optional in future lets
also handle the empty case by also not inspecting, the same as we do for
null.

Fixes #376
Reported-By: Lauren Ko <lauren.ko@unt.edu>
2021-05-11 21:55:03 +09:00
webdev4422 5ec7fb87d9 Fix misspell in comments 2021-02-26 08:50:13 +02:00
Andrew Jackson e215b89963 [maven-release-plugin] prepare for next development iteration 2020-05-18 14:08:12 +01:00
Andrew Jackson ef38dc08c3 [maven-release-plugin] prepare release 3.4.0-20200518 2020-05-18 14:08:04 +01:00
Clara Wiatrowski eea3914059 Add missing import 2020-04-21 16:00:21 +02:00
Clara Wiatrowski b61f7e574e Add real crawlStatus in the crawlReport 2020-04-17 14:53:00 +02:00
Andrew Jackson ff05de44b6 [maven-release-plugin] prepare for next development iteration 2020-03-04 23:05:41 +00:00
Andrew Jackson a1bdcb1be7 [maven-release-plugin] prepare release 3.4.0-20200304 2020-03-04 23:05:33 +00:00
Andy Jackson e0c82f70e9 Merge pull request #289 from netarchivesuite/bdb-frontier-access
Bdb frontier access
2020-03-04 13:41:27 +00:00
Jonathan Leitschuh bc6a15b41d Use HTTPS to resolve dependencies in Maven Build where possible 2020-02-11 17:48:20 +09:00
Lauren Ko 8c1c8009c6 Fix stream closed exception for Paged view 2020-01-30 16:48:38 -06:00
Alex Osborne b45f774066 Merge pull request #300 from hennekey/fix-299
Use Guice instead of custom implementation
2020-01-30 19:12:05 +09:00
Alex Osborne a4ce59dbd6 Fix stream closed exception by not closing output stream
ServerCall.writeResponseBody() flushes it after we return so it must
remain open.

Fixes #305
2020-01-25 08:25:59 +09:00
Tim Hennekey 33458f1518 Fix assertions
By using assertEquals and seting the expected and actual values, the
failure messages become a bit more useful.
2020-01-22 17:13:58 -05:00
Noah Levitt 784b3a5dec fix logging config
by setting system property java.util.logging.config.file, because new
version restlet reconfigures logging after heritrix has already
configured it
2020-01-22 13:39:24 -08:00
Alex Osborne 34be934da5 Fix 'Method Not Allowed' on POST of config editor form
Under Restlet 2 `getVariants()` always returns null for a POST so our
post() method was never called and 405 Method Not Allowed was returned.
We don't need content negotiation for the POST response anyway so let's
instead override the no-variants post() like we do for put().

Fixes #293
2020-01-15 16:51:52 +09:00
jkafader 3a8b58998c Merge pull request #285 from nlevitt/warc-writer-chain
Warc writer chain
2020-01-13 16:39:42 -08:00
Nicholas Clarke c611c46c8a Merged frontier-management with upgrade to bdb 7 2019-11-15 10:33:13 +01:00
Noah Levitt 74c0739bb0 fix line ending and indentation issues 2019-10-11 13:49:49 -07:00
Adam Miller 0cee24922f AssignmentLevelSurtQueueAssignmentPolicy.java - Add support for forced queue assignment and parallel queues
URIAuthorityBasedQueueAssignmentPolicy.java - Add interoperability between forced queue assignment and parallel queues
QuotaEnforcer.java - Fix javadoc to match default behavior
2019-10-11 19:44:35 +00:00
Alex Osborne eeddfd763f Override PUT so it doesn't change the file extension
Fixes #282 and HER-1907
2019-08-28 12:46:29 +09:00
Alex Osborne 5612fa249b Use super.getVariants() rather than super.getVariants(GET)
This was a regression introduced in the upgrade to Restlet 2. I
encountered a NullPointerException here when upgrading and misunderstood
the cause of it. Since PUT and DELETE return no content they are
actually supposed to return null.
2019-08-27 23:53:34 +09:00
Alex Osborne 38c44e8e8e Add missing UUID import (interactive commit fail) 2019-08-12 17:50:35 +09:00
Alex Osborne 36003b8626 Fix digest authentication
In Restlet 2 it appears we need to use DigestAuthenticator.
(Previously both digest and basic auth were handled by the same
Guard class.)
2019-08-12 17:36:28 +09:00
Noah Levitt cefad96036 Merge pull request #276 from internetarchive/jdk11-upgrade-jetty
JDK11 support: upgrade to Jetty 9.4.19, Restlet 2.4.0 and drop JDK 7 support
2019-08-07 10:48:51 -07:00
Noah Levitt 6811f49bd9 Merge pull request #272 from internetarchive/jdk11-upgrade-surefire
JDK11 support: upgrade maven-surefire-plugin to 2.22.2
2019-08-07 10:48:29 -07:00
Alex Osborne 97b9c3c91c Fix restlet child component context warning 2019-08-06 17:13:29 +09:00
Alex Osborne 6c2afc36b0 Update statistics assertions as Jetty 9 produces different headers 2019-08-06 17:13:02 +09:00
Alex Osborne 4012a5f4dd Fix 'No session data store configured' test error 2019-08-06 16:56:10 +09:00
Alex Osborne 0782fb0481 JDK11: upgrade engine to jetty 9.4.19 and restlet 2.4.0
Jetty 9.4.12+ is required for TLS to work correctly under JDK11
(due to SSL handshake failures). In order to upgrade
Jetty we also need to upgrade Restlet.

There is one intentional change in behaviour to simplify
upgrading. We remove a workaround for an old [webkit bug]
where the browser claimed to prefer application/xml. The
bug was fixed in 2011.

[webkit bug]: https://bugs.webkit.org/show_bug.cgi?id=27267

Summary of Jetty API changes:
- package names changed such as org.mortbay -> org.eclipse
- SocketConnector and SslSocketConnector merged to ServerConnector
- HashUserRealm split into UserStore and HashLoginService
- SecurityHandler -> ConstraintSecurityHandler

Summary of Restlet API changes:
- some classes have moved package (Request, Response, Router etc)
- ServerResource replaces Resource
- represent(), acceptRepresentation() renamed to get(), post()
- constructors were replaced by an init() method
- setModifiable() was removed
- getRequest().getEntityAsForm() -> new Form(entity)
- Guard -> ChallengeAuthenticator

Fixes #275
2019-08-06 15:16:37 +09:00
Alex Osborne a2920c622d JDK11 support: explicitly depend on JAXB
JAXB is no longer included in the JDK as of JDK 11 (JEP 320) so add it
as an external dependency.

Fixes #264
2019-08-02 16:47:20 +09:00
Alex Osborne 412885fd7d JDK11 support: upgrade maven-surefire-plugin to 2.22.2
Under JDK11 our old version of it throws ClassNotFoundExceptions when
tests load some builtin classes like javax.transaction.xa.Xid.

Fixes #266
2019-08-02 16:43:20 +09:00
kristinn 308cee85a3 Add sitemap extraction to default profile 2019-07-05 11:28:53 +00:00
Noah Levitt 7c31b0752a use WARCWriterChainProcessor for these tests 2019-06-12 12:56:49 -07:00