Commit Graph
2088 Commits
Author SHA1 Message Date
Barbara Miller dbade18b58 Merge pull request #301 from nlevitt/fix-logging-config
fix logging config
2020-01-22 14:42:22 -08:00
Noah Levitt 784b3a5dec fix logging config
by setting system property java.util.logging.config.file, because new
version restlet reconfigures logging after heritrix has already
configured it
2020-01-22 13:39:24 -08:00
Alex Osborne a8c41aa014 Merge pull request #298 from hennekey/fix-297
Speed up ObjectIdentityBdbManualCacheTest
2020-01-16 15:42:06 +09:00
jkafader 739cf7fdb3 Merge pull request #287 from nlevitt/trough-dedup-fix
change trough dedup `date` type to varchar.
2020-01-15 11:21:06 -08:00
Noah Levitt 00a1f2c9b0 Merge pull request #294 from internetarchive/fix-config-post
Fix 'Method Not Allowed' on POST of config editor form
2020-01-15 11:01:00 -08:00
Noah Levitt ccbf678f4a Merge pull request #295 from internetarchive/disable-wbm-test
Disable test that connects to wwwb-dedup.us.archive.org
2020-01-15 10:56:31 -08:00
Tim Hennekey 0fa87a36e8 Add default constructor to IdentityCacheWrapper
Kryo is spending a lot of time during serialization handling a
NoSuchMethodException. By adding this default constructor we can skip
doing all that work.
2020-01-15 13:39:27 -05:00
Tim Hennekey 55308ba90a Move IdentityCacheableWrapper
This is utilized only in tests so it belongs there
2020-01-15 13:39:27 -05:00
Tim Hennekey 0fe8463e81 Remove AtomicInteger loop counter
This test isn't multithreaded so I see no need for this
2020-01-15 13:39:27 -05:00
Tim Hennekey 611fa325df Remove unused code
It reduces the clarity of the surrounding code
2020-01-15 13:39:27 -05:00
Tim Hennekey bb61aa0bcb Fix unchecked warning 2020-01-15 13:39:27 -05:00
Alex Osborne 615578b22d Add IntelliJ IDEA and Heritrix runtime files to .gitignore
I've seen a few people accidentally commit them.
2020-01-15 18:19:25 +09:00
Alex Osborne dcdaba3b10 Disable test that connects to wwwb-dedup.us.archive.org
http://wwwb-dedup.us.archive.org:8083/web/timemap/cdx is currently
returning 503 intermittently and so this is breaking Travis builds for
unrelated changes.
2020-01-15 17:48:31 +09:00
Alex Osborne 34be934da5 Fix 'Method Not Allowed' on POST of config editor form
Under Restlet 2 `getVariants()` always returns null for a POST so our
post() method was never called and 405 Method Not Allowed was returned.
We don't need content negotiation for the POST response anyway so let's
instead override the no-variants post() like we do for put().

Fixes #293
2020-01-15 16:51:52 +09:00
Alex Osborne 8e875bb17f Merge pull request #288 from netarchivesuite/inline-image-filter
Attempt to filter out embedded images.
2020-01-15 12:48:54 +09:00
jkafader 3a8b58998c Merge pull request #285 from nlevitt/warc-writer-chain
Warc writer chain
2020-01-13 16:39:42 -08:00
Noah Levitt 31b9960503 basic level of documentation 2019-12-30 16:02:10 -08:00
Noah Levitt 125547b93d change trough dedup date type to varchar.
By parsing/unparsing to/from java.util.Date, we ended up with a
different date format in trough (sqlite) than warcprox, which is no
good; see https://github.com/internetarchive/warcprox/pull/144
2019-11-19 14:34:49 -08:00
Colin Rosenthal 204491c357 Attempt to filter out embedded images.
(cherry picked from commit aa5ff1dbdaefd04652a9c66506d20f1a6ae01dc3)
2019-11-19 15:04:50 +01:00
Noah Levitt ff8ebc4c2e use JSONObject.isNull()
because opt() returns org.json.JSONObject.Null
2019-11-18 14:42:38 -08:00
Noah Levitt 6e7bc54c9d use org.json like everybody else 2019-11-15 16:14:17 -08:00
Noah Levitt c4b08166f2 extract watch page links from youtube playlists
and equivalent for other sites. Usually we find these links through
normal link extraction, but we have the info here, so we may as well use
it to make sure.
2019-11-15 15:52:36 -08:00
Noah Levitt 4983e65929 fix non-playlist case (oops!) 2019-11-15 15:42:59 -08:00
Noah Levitt 1b95453748 be consistent and null-safe with concurrentTo 2019-10-15 10:50:26 -07:00
Noah Levitt 7a83aecb28 Merge pull request #286 from internetarchive/adds-subqueue-support-forced-queue-assignment
Add support for forced queue assignment and parallel queues
2019-10-11 13:52:47 -07:00
Noah Levitt 74c0739bb0 fix line ending and indentation issues 2019-10-11 13:49:49 -07:00
Adam Miller 0cee24922f AssignmentLevelSurtQueueAssignmentPolicy.java - Add support for forced queue assignment and parallel queues
URIAuthorityBasedQueueAssignmentPolicy.java - Add interoperability between forced queue assignment and parallel queues
QuotaEnforcer.java - Fix javadoc to match default behavior
2019-10-11 19:44:35 +00:00
Noah Levitt feea47b2c7 Merge pull request #283 from internetarchive/jobdir-put-fix
Fix jobdir PUT
2019-08-28 12:41:00 -07:00
Alex Osborne eeddfd763f Override PUT so it doesn't change the file extension
Fixes #282 and HER-1907
2019-08-28 12:46:29 +09:00
Alex Osborne 5612fa249b Use super.getVariants() rather than super.getVariants(GET)
This was a regression introduced in the upgrade to Restlet 2. I
encountered a NullPointerException here when upgrading and misunderstood
the cause of it. Since PUT and DELETE return no content they are
actually supposed to return null.
2019-08-27 23:53:34 +09:00
Noah Levitt 2eafb5a0c4 Merge pull request #280 from internetarchive/fix-cookie-test-failures
Mitigate random CookieStore.testConcurrentLoad test failures
2019-08-16 10:08:31 -07:00
Alex Osborne 2a5bd87e51 Remove testConcurrentLoad
Noah wrote in #280:

> Maybe we should just drop the test. The assumption when we wrote the
> test was that a race condition would not be so frequent in practice.
> We've seen that under the contrived conditions created by the test
> case, it is frequent. But that's ok
2019-08-16 22:11:26 +09:00
Alex Osborne a5c03d9b73 Mitigate random CookieStore.testConcurrentLoad test failures
The arbitary value `25` was used but in prace it's quite possible
for more than 25 writing threads to have checked the cookie count
limit before adding their cookie. In practice we see Travis failing
on this test quite often, every few builds in fact.

I think using `threads.length` (i.e. 200) should cover the worst
case possibility where every thread reads a stale count and tries
to add their cookie.

Fixes #274
2019-08-12 18:08:43 +09:00
Alex Osborne 38c44e8e8e Add missing UUID import (interactive commit fail) 2019-08-12 17:50:35 +09:00
Alex Osborne 36003b8626 Fix digest authentication
In Restlet 2 it appears we need to use DigestAuthenticator.
(Previously both digest and basic auth were handled by the same
Guard class.)
2019-08-12 17:36:28 +09:00
Alex Osborne 92ffdede0d Link to javadoc.io for more recent api docs 2019-08-12 15:03:52 +09:00
Alex Osborne 8b72be7d1d Now jdk11 support is merged we can disallow test failures on it 2019-08-08 10:37:38 +09:00
Noah Levitt cefad96036 Merge pull request #276 from internetarchive/jdk11-upgrade-jetty
JDK11 support: upgrade to Jetty 9.4.19, Restlet 2.4.0 and drop JDK 7 support
2019-08-07 10:48:51 -07:00
Noah Levitt a1f2c9513a Merge pull request #273 from internetarchive/jdk11-remove-old-bdbcache
JDK11 support: remove unused class ObjectIdentityBdbCache and tests
2019-08-07 10:48:40 -07:00
Noah Levitt 6811f49bd9 Merge pull request #272 from internetarchive/jdk11-upgrade-surefire
JDK11 support: upgrade maven-surefire-plugin to 2.22.2
2019-08-07 10:48:29 -07:00
Noah Levitt ebb0d3fd48 Merge pull request #271 from internetarchive/jdk11-toolsjar
JDK11 support: exclude tools.jar from hbase-client dependency
2019-08-07 10:48:19 -07:00
Noah Levitt d5956d1443 Merge pull request #269 from internetarchive/jdk11-jaxb
JDK11 support: explicitly depend on JAXB
2019-08-07 10:48:02 -07:00
Alex Osborne 97b9c3c91c Fix restlet child component context warning 2019-08-06 17:13:29 +09:00
Alex Osborne 6c2afc36b0 Update statistics assertions as Jetty 9 produces different headers 2019-08-06 17:13:02 +09:00
Alex Osborne 4012a5f4dd Fix 'No session data store configured' test error 2019-08-06 16:56:10 +09:00
Alex Osborne 996d1366b0 Drop support for JDK 7
Jetty 9.4 requires JDK 8 or later. JDK 11 requires Jetty 9.4.12+.
Therefore unfortunately we cannot easily support both JDK 11 and JDK 7
at the same time.
2019-08-06 16:41:09 +09:00
Alex Osborne 0782fb0481 JDK11: upgrade engine to jetty 9.4.19 and restlet 2.4.0
Jetty 9.4.12+ is required for TLS to work correctly under JDK11
(due to SSL handshake failures). In order to upgrade
Jetty we also need to upgrade Restlet.

There is one intentional change in behaviour to simplify
upgrading. We remove a workaround for an old [webkit bug]
where the browser claimed to prefer application/xml. The
bug was fixed in 2011.

[webkit bug]: https://bugs.webkit.org/show_bug.cgi?id=27267

Summary of Jetty API changes:
- package names changed such as org.mortbay -> org.eclipse
- SocketConnector and SslSocketConnector merged to ServerConnector
- HashUserRealm split into UserStore and HashLoginService
- SecurityHandler -> ConstraintSecurityHandler

Summary of Restlet API changes:
- some classes have moved package (Request, Response, Router etc)
- ServerResource replaces Resource
- represent(), acceptRepresentation() renamed to get(), post()
- constructors were replaced by an init() method
- setModifiable() was removed
- getRequest().getEntityAsForm() -> new Form(entity)
- Guard -> ChallengeAuthenticator

Fixes #275
2019-08-06 15:16:37 +09:00
Alex Osborne 9d1341c731 JDK11 support: upgrade jetty to 9.4.19 in modules tests
Fixes #268 ssl handshake_failure. Support for jdk11 was added in jetty
9.4.14 but we may as well bump to the latest stable version.

In Jetty 9 it appears the request is logged after the response is sent.
Thus it was racing with the assertions that check the client IP. So to
fix this rather stashing the 'lastRequest' we just make the server
echo the client's IP in a response header.

Some other minor tweaks were needed due to changes in Jetty behaviour:

- We stop checking the length of the raw response. It doesn't tell us
  anything and easily varies.

- Jetty now generates Set-Cookie with a space after the ;

- Jetty now lowercases the word "basic" in WWW-Authenticate header

- testLaxUrlEncoding(): Jetty now rejects bad paths with a 400 error so
  we disable the response checks. The actual request line is
  still checked which is the important thing.

Note: This patch does not affect the version of jetty used by the
Heritrix admin console. That will be tackled separately.
2019-08-05 14:35:09 +09:00
Alex Osborne fea6a9c60c Merge pull request #256 from ruebot/readme-updates
Update README and add LICENSE.txt
2019-08-05 13:15:19 +09:00
Alex Osborne d3d19e2248 Prevent newer surefire from invoking FetchHTTPTests directly
It seems older versions only matched *Test.java but the new version
is now incorrectly including it.
2019-08-02 17:57:09 +09:00