Commit Graph
548 Commits
Author SHA1 Message Date
Alex Osborne 636948cb09 KeyTool wrapper: fallback to running keytool as a subprocess on JDK 16+
JDK 16 defaults to --illegal-access=deny which means trying to call
KeyTool via reflection now throws IllegalAccessException.

Fixes #417
2021-07-25 11:15:45 +09:00
Adam Miller d2d9d4f157 Adding configurable MaxLogFileSize option to BdbModule 2021-07-19 15:49:56 +09:00
Alex Osborne 6835d0940e Merge pull request #407 from internetarchive/fix-maven-warnings
Fix a couple of boring maven warnings
2021-06-25 11:52:53 +09:00
Alex Osborne 039634cee0 Update relocated dependencies
Fixes the Maven warnings:

    [WARNING] The artifact poi:poi:jar:2.5.1 has been relocated to
        poi:poi:jar:2.5.1-final-20040804
    [WARNING] The artifact itext:itext:jar:1.3 has been relocated to
        com.lowagie:itext:jar:1.3
2021-06-22 20:52:55 +09:00
Alex Osborne d643770a7c Put slowest unit tests behind a -DrunSlowTests=true option
Shaves about 2 minutes off the Heritrix build time.
2021-06-22 17:10:50 +09:00
Andrew Jackson 0e93e22ba9 [maven-release-plugin] prepare for next development iteration 2021-06-21 15:42:52 +01:00
Andrew Jackson 846f2bc2cc [maven-release-plugin] prepare release 3.4.0-20210621 2021-06-21 15:42:46 +01:00
Alex Osborne 5d0a5b2ec4 Remove dependency on mg4j
This eliminates one of the jars that Heritrix depends on that isn't in
Maven Central.

There were only two usages. The first usage was a method in a
CrawlerJournal which as far as I can tell is unused and can be removed.

The second usage was an unnecessary use of MutableString as a line
buffer when saving cookies to a file. Since there doesn't seem any
particular reason buffer per-line here let's just use a BufferedWriter
instead on the whole file which is also slightly simpler and more
efficient.
2021-06-20 23:30:41 +09:00
Andrew Jackson a5beb7d6e9 [maven-release-plugin] prepare for next development iteration 2021-06-18 09:41:30 +01:00
Andrew Jackson 30598f8402 [maven-release-plugin] prepare release 3.4.0-20210618 2021-06-18 09:41:24 +01:00
Andrew Jackson 3423cab6a6 [maven-release-plugin] prepare for next development iteration 2021-06-17 14:39:00 +01:00
Andrew Jackson 7368ded9c2 [maven-release-plugin] prepare release 3.4.0-20210617 2021-06-17 14:38:54 +01:00
Andrew Jackson d3dfb34f83 [maven-release-plugin] prepare for next development iteration 2021-05-27 10:03:10 +01:00
Andrew Jackson 9dcbb240a6 [maven-release-plugin] prepare release 3.4.0-20210527 2021-05-27 10:03:04 +01:00
Andy Jackson 669cf1ed2b Merge branch 'master' into update-dependencies 2021-05-26 13:40:15 +01:00
Andy Jackson a2c65f4618 Update dependencies for security issues, including changes to tests. 2021-05-26 13:39:30 +01:00
Andy Jackson 276ced0de3 Allow tuning of BDB-JE evictor and cleaner threads. 2021-05-20 22:22:00 +01:00
Andy Jackson c0dbfb2a0a Merge pull request #383 from ukwa/upgrade-dnsjava
Update to latest version of dnsjava, for #344
2021-05-20 21:05:59 +01:00
Andy Jackson 396467c535 Merge branch 'master' into sitemaps 2021-05-20 21:03:54 +01:00
Andy Jackson 045b2516da Update to latest version of dnsjava, for #344
This commit updates to version 3.3.1 of the dnsjava library.

Incidentally, the restlet.org certificate is currently invalid, so this
patch switched to maven.restlet.com.
2021-05-20 20:44:19 +01:00
Alex Osborne 5f52a14afc Upgrade to Spring 5.3.3
Spring 3.0.5 is 10 years old now and is incompatible with classes using
Java 9+ bytecode and even on Java 8 classes using lambdas (see #337).

We remove spring-asm as it is now part of spring-core module.
2021-02-15 16:20:00 +09:00
Andrew Jackson e215b89963 [maven-release-plugin] prepare for next development iteration 2020-05-18 14:08:12 +01:00
Andrew Jackson ef38dc08c3 [maven-release-plugin] prepare release 3.4.0-20200518 2020-05-18 14:08:04 +01:00
Leslie Bellony 63ca9d4c6a Remove static fields and byte loop + rename variable 2020-04-23 17:10:42 +02:00
Leslie Bellony 40725372d6 Add support for the SFTP protocol 2020-04-01 14:47:34 +02:00
Andrew Jackson ff05de44b6 [maven-release-plugin] prepare for next development iteration 2020-03-04 23:05:41 +00:00
Andrew Jackson a1bdcb1be7 [maven-release-plugin] prepare release 3.4.0-20200304 2020-03-04 23:05:33 +00:00
Alex Osborne fd514b7bf6 Merge pull request #315 from ukwa/fix-oracle-doc-link
Use the Wayback Machine to repair a link to Oracle docs.
2020-03-05 01:40:39 +09:00
Andrew Jackson b4494a7ffb Use the Wayback Machine to repair a link to Oracle docs. 2020-03-04 15:42:29 +00:00
Tim Hennekey 348b5330bc Utilize the d parameter
There are uses of this class that necessitate a lower false positive
rate than the value that was hard-coded. By making use of the parameter,
as indicated in the JavaDoc, the resulting delegate should meet the
demands of the caller.
2020-03-04 09:37:31 -05:00
Andy Jackson 7f80b4fd98 Merge pull request #281 from ukwa/upgrade-bdb-je
WIP: Upgrade BDB JE to version 7.5.11
2020-03-04 13:36:03 +00:00
Jonathan Leitschuh bc6a15b41d Use HTTPS to resolve dependencies in Maven Build where possible 2020-02-11 17:48:20 +09:00
Alex Osborne b45f774066 Merge pull request #300 from hennekey/fix-299
Use Guice instead of custom implementation
2020-01-30 19:12:05 +09:00
Alex Osborne fd9136dbfa Merge pull request #304 from hennekey/remove-custom-base32
Replace custom Base32 encoding
2020-01-30 19:07:09 +09:00
Tim Hennekey 54e05a7864 Correct encoding
The previous implementation appears to always have returned upper case,
was able to encode either case, and did not reutrn padding.
2020-01-28 15:12:54 -05:00
Tim Hennekey b5f95c5e06 Replace custom Base32 encoding
Guava is available so a custom implementation is unnecessary.
2020-01-24 12:46:37 -05:00
Alex Osborne 9927e1325f Merge pull request #296 from hennekey/update-junit
Set JUnit version to latest
2020-01-23 16:44:59 +09:00
Tim Hennekey 9cb9563da3 Increment the count only when the filter notes it
Otherwise this is a count of how many times this add method is called,
not how many times an element was noted as being actually added.
2020-01-22 17:15:24 -05:00
Tim Hennekey 66723aea68 Remove version
Allow the parent POM to specify the version
2020-01-21 10:52:15 -05:00
Tim Hennekey 080380b18a Use Guice instead of custom implementation
This uses the avaialable code in Guice rather than a custom
implementation. It also provides a performance increase (as demonstrated
by the unit tests)
2020-01-15 18:13:20 -05:00
Tim Hennekey 0fa87a36e8 Add default constructor to IdentityCacheWrapper
Kryo is spending a lot of time during serialization handling a
NoSuchMethodException. By adding this default constructor we can skip
doing all that work.
2020-01-15 13:39:27 -05:00
Tim Hennekey 55308ba90a Move IdentityCacheableWrapper
This is utilized only in tests so it belongs there
2020-01-15 13:39:27 -05:00
Tim Hennekey 0fe8463e81 Remove AtomicInteger loop counter
This test isn't multithreaded so I see no need for this
2020-01-15 13:39:27 -05:00
Tim Hennekey 611fa325df Remove unused code
It reduces the clarity of the surrounding code
2020-01-15 13:39:27 -05:00
Tim Hennekey bb61aa0bcb Fix unchecked warning 2020-01-15 13:39:27 -05:00
Andrew Jackson 45cd46e030 Try upping to BDB JE 7.5.11 2019-08-11 12:42:03 +01:00
Noah Levitt cefad96036 Merge pull request #276 from internetarchive/jdk11-upgrade-jetty
JDK11 support: upgrade to Jetty 9.4.19, Restlet 2.4.0 and drop JDK 7 support
2019-08-07 10:48:51 -07:00
Noah Levitt a1f2c9513a Merge pull request #273 from internetarchive/jdk11-remove-old-bdbcache
JDK11 support: remove unused class ObjectIdentityBdbCache and tests
2019-08-07 10:48:40 -07:00
Alex Osborne 0782fb0481 JDK11: upgrade engine to jetty 9.4.19 and restlet 2.4.0
Jetty 9.4.12+ is required for TLS to work correctly under JDK11
(due to SSL handshake failures). In order to upgrade
Jetty we also need to upgrade Restlet.

There is one intentional change in behaviour to simplify
upgrading. We remove a workaround for an old [webkit bug]
where the browser claimed to prefer application/xml. The
bug was fixed in 2011.

[webkit bug]: https://bugs.webkit.org/show_bug.cgi?id=27267

Summary of Jetty API changes:
- package names changed such as org.mortbay -> org.eclipse
- SocketConnector and SslSocketConnector merged to ServerConnector
- HashUserRealm split into UserStore and HashLoginService
- SecurityHandler -> ConstraintSecurityHandler

Summary of Restlet API changes:
- some classes have moved package (Request, Response, Router etc)
- ServerResource replaces Resource
- represent(), acceptRepresentation() renamed to get(), post()
- constructors were replaced by an init() method
- setModifiable() was removed
- getRequest().getEntityAsForm() -> new Form(entity)
- Guard -> ChallengeAuthenticator

Fixes #275
2019-08-06 15:16:37 +09:00
Alex Osborne a5c2b46208 JDK11 support: remove unused class ObjectIdentityBdbCache and tests
In 2011 ObjectIdentityBdbCache was replaced by
ObjectIdentityBdbManualCache in order to "remove dependence on
GC/finalization/PhantomReference magic". It hasn't been used since.
As of JDK11 the ObjectIdentityBdbCache unit tests are failing so let's
remove it.
2019-08-02 17:10:13 +09:00