Commit Graph
2156 Commits
Author SHA1 Message Date
Noah Levitt 6f69dd73d6 Merge branch 'ydl-streaming' into ait-qa
* ydl-streaming:
  java 8 compatibility
2020-01-22 14:13:22 -08:00
Noah Levitt 6667186826 java 8 compatibility 2020-01-22 14:12:36 -08:00
Noah Levitt b4cf06d087 Merge branch 'ydl-streaming' into ait-qa
* ydl-streaming:
  limit ExtractorYoutubeDL heap usage
2020-01-22 14:03:20 -08:00
Noah Levitt 515bcca6d6 Merge branch 'fix-logging-config' into ait-qa
* fix-logging-config:
  fix logging config
  Add default constructor to IdentityCacheWrapper
  Move IdentityCacheableWrapper
  Remove AtomicInteger loop counter
  Remove unused code
  Fix unchecked warning
  Add IntelliJ IDEA and Heritrix runtime files to .gitignore
  Disable test that connects to wwwb-dedup.us.archive.org
  Fix 'Method Not Allowed' on POST of config editor form
  basic level of documentation
  Attempt to filter out embedded images.
2020-01-22 14:03:09 -08:00
Noah Levitt 9229a51dcb limit ExtractorYoutubeDL heap usage
We were seeing OOME due to large youtoube-dl json (for playlists and
such). So instead of storing the json in ram, stream through it, and
stash the contents in an thread-local anonymous tempfile so it can be
written to to warc.
2020-01-22 13:49:24 -08:00
Noah Levitt 784b3a5dec fix logging config
by setting system property java.util.logging.config.file, because new
version restlet reconfigures logging after heritrix has already
configured it
2020-01-22 13:39:24 -08:00
Alex Osborne a8c41aa014 Merge pull request #298 from hennekey/fix-297
Speed up ObjectIdentityBdbManualCacheTest
2020-01-16 15:42:06 +09:00
jkafader 739cf7fdb3 Merge pull request #287 from nlevitt/trough-dedup-fix
change trough dedup `date` type to varchar.
2020-01-15 11:21:06 -08:00
Noah Levitt 00a1f2c9b0 Merge pull request #294 from internetarchive/fix-config-post
Fix 'Method Not Allowed' on POST of config editor form
2020-01-15 11:01:00 -08:00
Noah Levitt ccbf678f4a Merge pull request #295 from internetarchive/disable-wbm-test
Disable test that connects to wwwb-dedup.us.archive.org
2020-01-15 10:56:31 -08:00
Tim Hennekey 0fa87a36e8 Add default constructor to IdentityCacheWrapper
Kryo is spending a lot of time during serialization handling a
NoSuchMethodException. By adding this default constructor we can skip
doing all that work.
2020-01-15 13:39:27 -05:00
Tim Hennekey 55308ba90a Move IdentityCacheableWrapper
This is utilized only in tests so it belongs there
2020-01-15 13:39:27 -05:00
Tim Hennekey 0fe8463e81 Remove AtomicInteger loop counter
This test isn't multithreaded so I see no need for this
2020-01-15 13:39:27 -05:00
Tim Hennekey 611fa325df Remove unused code
It reduces the clarity of the surrounding code
2020-01-15 13:39:27 -05:00
Tim Hennekey bb61aa0bcb Fix unchecked warning 2020-01-15 13:39:27 -05:00
Alex Osborne 615578b22d Add IntelliJ IDEA and Heritrix runtime files to .gitignore
I've seen a few people accidentally commit them.
2020-01-15 18:19:25 +09:00
Alex Osborne dcdaba3b10 Disable test that connects to wwwb-dedup.us.archive.org
http://wwwb-dedup.us.archive.org:8083/web/timemap/cdx is currently
returning 503 intermittently and so this is breaking Travis builds for
unrelated changes.
2020-01-15 17:48:31 +09:00
Alex Osborne 34be934da5 Fix 'Method Not Allowed' on POST of config editor form
Under Restlet 2 `getVariants()` always returns null for a POST so our
post() method was never called and 405 Method Not Allowed was returned.
We don't need content negotiation for the POST response anyway so let's
instead override the no-variants post() like we do for put().

Fixes #293
2020-01-15 16:51:52 +09:00
Alex Osborne 8e875bb17f Merge pull request #288 from netarchivesuite/inline-image-filter
Attempt to filter out embedded images.
2020-01-15 12:48:54 +09:00
jkafader 3a8b58998c Merge pull request #285 from nlevitt/warc-writer-chain
Warc writer chain
2020-01-13 16:39:42 -08:00
Noah Levitt 31b9960503 basic level of documentation 2019-12-30 16:02:10 -08:00
Noah Levitt 808ff9a5b7 Merge branch 'trough-dedup-fix' into ait-qa
* trough-dedup-fix:
  change trough dedup `date` type to varchar.
2019-11-19 14:35:30 -08:00
Noah Levitt 125547b93d change trough dedup date type to varchar.
By parsing/unparsing to/from java.util.Date, we ended up with a
different date format in trough (sqlite) than warcprox, which is no
good; see https://github.com/internetarchive/warcprox/pull/144
2019-11-19 14:34:49 -08:00
Colin Rosenthal 204491c357 Attempt to filter out embedded images.
(cherry picked from commit aa5ff1dbdaefd04652a9c66506d20f1a6ae01dc3)
2019-11-19 15:04:50 +01:00
Noah Levitt 14e1eee662 Merge branch 'warc-writer-chain' into ait-qa
* warc-writer-chain:
  use JSONObject.isNull()
2019-11-18 14:43:58 -08:00
Noah Levitt ff8ebc4c2e use JSONObject.isNull()
because opt() returns org.json.JSONObject.Null
2019-11-18 14:42:38 -08:00
Noah Levitt 35c65ccb5a Merge remote-tracking branch 'origin/master' into ait-qa
* origin/master:
  fix line ending and indentation issues
  AssignmentLevelSurtQueueAssignmentPolicy.java - Add support for forced queue assignment and parallel queues URIAuthorityBasedQueueAssignmentPolicy.java - Add interoperability between forced queue assignment and parallel queues QuotaEnforcer.java - Fix javadoc to match default behavior
  Override PUT so it doesn't change the file extension
  Use super.getVariants() rather than super.getVariants(GET)
  Remove testConcurrentLoad
  Mitigate random CookieStore.testConcurrentLoad test failures
  Add missing UUID import (interactive commit fail)
  Fix digest authentication
  Link to javadoc.io for more recent api docs
  Now jdk11 support is merged we can disallow test failures on it
  Fix restlet child component context warning
  Update statistics assertions as Jetty 9 produces different headers
  Fix 'No session data store configured' test error
  Drop support for JDK 7
  JDK11: upgrade engine to jetty 9.4.19 and restlet 2.4.0
  JDK11 support: upgrade jetty to 9.4.19 in modules tests
  Prevent newer surefire from invoking FetchHTTPTests directly
  JDK11 support: remove unused class ObjectIdentityBdbCache and tests
  JDK11 support: exclude tools.jar from hbase-client dependency
  JDK11 support: explicitly depend on JAXB
  JDK11 support: upgrade maven-surefire-plugin to 2.22.2
  Travis fixes
  Update README and add LICENSE.txt
2019-11-15 16:21:39 -08:00
Noah Levitt 7c76dfba7c Merge branch 'warc-writer-chain' into ait-qa
* warc-writer-chain:
  use org.json like everybody else
2019-11-15 16:14:51 -08:00
Noah Levitt 6e7bc54c9d use org.json like everybody else 2019-11-15 16:14:17 -08:00
Noah Levitt 4524f24dc7 Merge branch 'warc-writer-chain' into ait-qa
* warc-writer-chain:
  extract watch page links from youtube playlists
  fix non-playlist case (oops!)
  be consistent and null-safe with concurrentTo
2019-11-15 15:54:31 -08:00
Noah Levitt c4b08166f2 extract watch page links from youtube playlists
and equivalent for other sites. Usually we find these links through
normal link extraction, but we have the info here, so we may as well use
it to make sure.
2019-11-15 15:52:36 -08:00
Noah Levitt 4983e65929 fix non-playlist case (oops!) 2019-11-15 15:42:59 -08:00
Noah Levitt 1b95453748 be consistent and null-safe with concurrentTo 2019-10-15 10:50:26 -07:00
Noah Levitt 7a83aecb28 Merge pull request #286 from internetarchive/adds-subqueue-support-forced-queue-assignment
Add support for forced queue assignment and parallel queues
2019-10-11 13:52:47 -07:00
Noah Levitt 74c0739bb0 fix line ending and indentation issues 2019-10-11 13:49:49 -07:00
Adam Miller 0cee24922f AssignmentLevelSurtQueueAssignmentPolicy.java - Add support for forced queue assignment and parallel queues
URIAuthorityBasedQueueAssignmentPolicy.java - Add interoperability between forced queue assignment and parallel queues
QuotaEnforcer.java - Fix javadoc to match default behavior
2019-10-11 19:44:35 +00:00
Noah Levitt feea47b2c7 Merge pull request #283 from internetarchive/jobdir-put-fix
Fix jobdir PUT
2019-08-28 12:41:00 -07:00
Alex Osborne eeddfd763f Override PUT so it doesn't change the file extension
Fixes #282 and HER-1907
2019-08-28 12:46:29 +09:00
Alex Osborne 5612fa249b Use super.getVariants() rather than super.getVariants(GET)
This was a regression introduced in the upgrade to Restlet 2. I
encountered a NullPointerException here when upgrading and misunderstood
the cause of it. Since PUT and DELETE return no content they are
actually supposed to return null.
2019-08-27 23:53:34 +09:00
Noah Levitt 2eafb5a0c4 Merge pull request #280 from internetarchive/fix-cookie-test-failures
Mitigate random CookieStore.testConcurrentLoad test failures
2019-08-16 10:08:31 -07:00
Alex Osborne 2a5bd87e51 Remove testConcurrentLoad
Noah wrote in #280:

> Maybe we should just drop the test. The assumption when we wrote the
> test was that a race condition would not be so frequent in practice.
> We've seen that under the contrived conditions created by the test
> case, it is frequent. But that's ok
2019-08-16 22:11:26 +09:00
Alex Osborne a5c03d9b73 Mitigate random CookieStore.testConcurrentLoad test failures
The arbitary value `25` was used but in prace it's quite possible
for more than 25 writing threads to have checked the cookie count
limit before adding their cookie. In practice we see Travis failing
on this test quite often, every few builds in fact.

I think using `threads.length` (i.e. 200) should cover the worst
case possibility where every thread reads a stale count and tries
to add their cookie.

Fixes #274
2019-08-12 18:08:43 +09:00
Alex Osborne 38c44e8e8e Add missing UUID import (interactive commit fail) 2019-08-12 17:50:35 +09:00
Alex Osborne 36003b8626 Fix digest authentication
In Restlet 2 it appears we need to use DigestAuthenticator.
(Previously both digest and basic auth were handled by the same
Guard class.)
2019-08-12 17:36:28 +09:00
Alex Osborne 92ffdede0d Link to javadoc.io for more recent api docs 2019-08-12 15:03:52 +09:00
Alex Osborne 8b72be7d1d Now jdk11 support is merged we can disallow test failures on it 2019-08-08 10:37:38 +09:00
Noah Levitt cefad96036 Merge pull request #276 from internetarchive/jdk11-upgrade-jetty
JDK11 support: upgrade to Jetty 9.4.19, Restlet 2.4.0 and drop JDK 7 support
2019-08-07 10:48:51 -07:00
Noah Levitt a1f2c9513a Merge pull request #273 from internetarchive/jdk11-remove-old-bdbcache
JDK11 support: remove unused class ObjectIdentityBdbCache and tests
2019-08-07 10:48:40 -07:00
Noah Levitt 6811f49bd9 Merge pull request #272 from internetarchive/jdk11-upgrade-surefire
JDK11 support: upgrade maven-surefire-plugin to 2.22.2
2019-08-07 10:48:29 -07:00
Noah Levitt ebb0d3fd48 Merge pull request #271 from internetarchive/jdk11-toolsjar
JDK11 support: exclude tools.jar from hbase-client dependency
2019-08-07 10:48:19 -07:00