lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-24 22:55:45 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,156 Commits 38 Branches 33 Tags
b84d443f00a75d68fa022c61ba5b8a106bd6344a
Commit Graph
2156 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Barbara Miller 8ffcfa9020 Merge pull request #302 from nlevitt/ydl-streaming 
limit ExtractorYoutubeDL heap usage
 2020-01-22 15:08:26 -08:00
Barbara Miller dbade18b58 Merge pull request #301 from nlevitt/fix-logging-config 
fix logging config
 2020-01-22 14:42:22 -08:00
Tim Hennekey 9cb9563da3 Increment the count only when the filter notes it 
Otherwise this is a count of how many times this add method is called,
not how many times an element was noted as being actually added.
 2020-01-22 17:15:24 -05:00
Tim Hennekey 33458f1518 Fix assertions 
By using assertEquals and seting the expected and actual values, the
failure messages become a bit more useful.
 2020-01-22 17:13:58 -05:00
Noah Levitt 6667186826 java 8 compatibility 2020-01-22 14:12:36 -08:00
Noah Levitt 9229a51dcb limit ExtractorYoutubeDL heap usage 
We were seeing OOME due to large youtoube-dl json (for playlists and
such). So instead of storing the json in ram, stream through it, and
stash the contents in an thread-local anonymous tempfile so it can be
written to to warc.
 2020-01-22 13:49:24 -08:00
Noah Levitt 784b3a5dec fix logging config 
by setting system property java.util.logging.config.file, because new
version restlet reconfigures logging after heritrix has already
configured it
 2020-01-22 13:39:24 -08:00
Tim Hennekey 66723aea68 Remove version 
Allow the parent POM to specify the version
 2020-01-21 10:52:15 -05:00
Alex Osborne a8c41aa014 Merge pull request #298 from hennekey/fix-297 
Speed up ObjectIdentityBdbManualCacheTest
 2020-01-16 15:42:06 +09:00
Tim Hennekey 080380b18a Use Guice instead of custom implementation 
This uses the avaialable code in Guice rather than a custom
implementation. It also provides a performance increase (as demonstrated
by the unit tests)
 2020-01-15 18:13:20 -05:00
jkafader 739cf7fdb3 Merge pull request #287 from nlevitt/trough-dedup-fix 
change trough dedup `date` type to varchar.
 2020-01-15 11:21:06 -08:00
Noah Levitt 00a1f2c9b0 Merge pull request #294 from internetarchive/fix-config-post 
Fix 'Method Not Allowed' on POST of config editor form
 2020-01-15 11:01:00 -08:00
Noah Levitt ccbf678f4a Merge pull request #295 from internetarchive/disable-wbm-test 
Disable test that connects to wwwb-dedup.us.archive.org
 2020-01-15 10:56:31 -08:00
Tim Hennekey 0fa87a36e8 Add default constructor to IdentityCacheWrapper 
Kryo is spending a lot of time during serialization handling a
NoSuchMethodException. By adding this default constructor we can skip
doing all that work.
 2020-01-15 13:39:27 -05:00
Tim Hennekey 55308ba90a Move IdentityCacheableWrapper 
This is utilized only in tests so it belongs there
 2020-01-15 13:39:27 -05:00
Tim Hennekey 0fe8463e81 Remove AtomicInteger loop counter 
This test isn't multithreaded so I see no need for this
 2020-01-15 13:39:27 -05:00
Tim Hennekey 611fa325df Remove unused code 
It reduces the clarity of the surrounding code
 2020-01-15 13:39:27 -05:00
Tim Hennekey bb61aa0bcb Fix unchecked warning 2020-01-15 13:39:27 -05:00
Tim Hennekey 4be3e620c3 Set JUnit version to latest 
Updating JUnit to begin updating the unit tests
 2020-01-15 11:08:31 -05:00
Alex Osborne 615578b22d Add IntelliJ IDEA and Heritrix runtime files to .gitignore 
I've seen a few people accidentally commit them.
 2020-01-15 18:19:25 +09:00
Alex Osborne dcdaba3b10 Disable test that connects to wwwb-dedup.us.archive.org 
http://wwwb-dedup.us.archive.org:8083/web/timemap/cdx is currently
returning 503 intermittently and so this is breaking Travis builds for
unrelated changes.
 2020-01-15 17:48:31 +09:00
Alex Osborne 34be934da5 Fix 'Method Not Allowed' on POST of config editor form 
Under Restlet 2 `getVariants()` always returns null for a POST so our
post() method was never called and 405 Method Not Allowed was returned.
We don't need content negotiation for the POST response anyway so let's
instead override the no-variants post() like we do for put().

Fixes #293
 2020-01-15 16:51:52 +09:00
Alex Osborne 8e875bb17f Merge pull request #288 from netarchivesuite/inline-image-filter 
Attempt to filter out embedded images.
 2020-01-15 12:48:54 +09:00
jkafader 3a8b58998c Merge pull request #285 from nlevitt/warc-writer-chain 
Warc writer chain
 2020-01-13 16:39:42 -08:00
Noah Levitt 31b9960503 basic level of documentation 2019-12-30 16:02:10 -08:00
Noah Levitt 125547b93d change trough dedup date type to varchar. 
By parsing/unparsing to/from java.util.Date, we ended up with a
different date format in trough (sqlite) than warcprox, which is no
good; see https://github.com/internetarchive/warcprox/pull/144
 2019-11-19 14:34:49 -08:00
Colin Rosenthal 204491c357 Attempt to filter out embedded images. 
(cherry picked from commit aa5ff1dbdaefd04652a9c66506d20f1a6ae01dc3)
 2019-11-19 15:04:50 +01:00
Noah Levitt ff8ebc4c2e use JSONObject.isNull() 
because opt() returns org.json.JSONObject.Null
 2019-11-18 14:42:38 -08:00
Noah Levitt 6e7bc54c9d use org.json like everybody else 2019-11-15 16:14:17 -08:00
Noah Levitt c4b08166f2 extract watch page links from youtube playlists 
and equivalent for other sites. Usually we find these links through
normal link extraction, but we have the info here, so we may as well use
it to make sure.
 2019-11-15 15:52:36 -08:00
Noah Levitt 4983e65929 fix non-playlist case (oops!) 2019-11-15 15:42:59 -08:00
Colin Rosenthal 5e536862db Removed mistaken additions 2019-11-15 14:38:45 +01:00
Colin Rosenthal bf5842c02e removed .iml file 2019-11-15 14:37:11 +01:00
Colin Rosenthal f76ec2f362 Merge remote-tracking branch 'origin/crawltrap-regex-timeout' into crawltrap-regex-timeout 2019-11-15 14:34:11 +01:00
Colin Rosenthal 718cc5ae32 Updated manually to new SNAPSHOT version 2019-11-15 14:34:03 +01:00
Colin Rosenthal f22a97875d Added a timeout to crawlertrap regex matching 2019-11-15 14:31:40 +01:00
Nicholas Clarke c611c46c8a Merged frontier-management with upgrade to bdb 7 2019-11-15 10:33:13 +01:00
Andrew Jackson 73db3ac628 Merge branch 'master' into upgrade-bdb-je 2019-10-17 14:10:22 +01:00
Noah Levitt 1b95453748 be consistent and null-safe with concurrentTo 2019-10-15 10:50:26 -07:00
Noah Levitt 7a83aecb28 Merge pull request #286 from internetarchive/adds-subqueue-support-forced-queue-assignment 
Add support for forced queue assignment and parallel queues
 2019-10-11 13:52:47 -07:00
Noah Levitt 74c0739bb0 fix line ending and indentation issues 2019-10-11 13:49:49 -07:00
Adam Miller 0cee24922f AssignmentLevelSurtQueueAssignmentPolicy.java - Add support for forced queue assignment and parallel queues 
URIAuthorityBasedQueueAssignmentPolicy.java - Add interoperability between forced queue assignment and parallel queues
QuotaEnforcer.java - Fix javadoc to match default behavior
 2019-10-11 19:44:35 +00:00
Noah Levitt feea47b2c7 Merge pull request #283 from internetarchive/jobdir-put-fix 
Fix jobdir PUT
 2019-08-28 12:41:00 -07:00
Alex Osborne eeddfd763f Override PUT so it doesn't change the file extension 
Fixes #282 and HER-1907
 2019-08-28 12:46:29 +09:00
Alex Osborne 5612fa249b Use super.getVariants() rather than super.getVariants(GET) 
This was a regression introduced in the upgrade to Restlet 2. I
encountered a NullPointerException here when upgrading and misunderstood
the cause of it. Since PUT and DELETE return no content they are
actually supposed to return null.
 2019-08-27 23:53:34 +09:00
Noah Levitt 2eafb5a0c4 Merge pull request #280 from internetarchive/fix-cookie-test-failures 
Mitigate random CookieStore.testConcurrentLoad test failures
 2019-08-16 10:08:31 -07:00
Alex Osborne 2a5bd87e51 Remove testConcurrentLoad 
Noah wrote in #280:

> Maybe we should just drop the test. The assumption when we wrote the
> test was that a race condition would not be so frequent in practice.
> We've seen that under the contrived conditions created by the test
> case, it is frequent. But that's ok
 2019-08-16 22:11:26 +09:00
Andrew Jackson f63570f977 Merge branch 'master' into upgrade-bdb-je 2019-08-12 16:21:29 +01:00
Alex Osborne a5c03d9b73 Mitigate random CookieStore.testConcurrentLoad test failures 
The arbitary value `25` was used but in prace it's quite possible
for more than 25 writing threads to have checked the cookie count
limit before adding their cookie. In practice we see Travis failing
on this test quite often, every few builds in fact.

I think using `threads.length` (i.e. 200) should cover the worst
case possibility where every thread reads a stale count and tries
to add their cookie.

Fixes #274
 2019-08-12 18:08:43 +09:00
Alex Osborne 38c44e8e8e Add missing UUID import (interactive commit fail) 2019-08-12 17:50:35 +09:00