lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-22 13:46:01 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,100 Commits 38 Branches 33 Tags
ea374b3a99103d11eef2a66bceed7b16918b093d
Commit Graph
2100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
jkafader ea374b3a99 Merge pull request #317 from internetarchive/fixes-extractor-multiple-regex-matcher-recycle 
Recycle the regex Matcher after use.
 2020-04-06 16:04:54 -07:00
Adam Miller a384dea2f9 Recycle the Matcher after use. 2020-03-25 22:33:06 +00:00
Alex Osborne 3528c990df Merge pull request #303 from hennekey/refactor-history 
Replace constant with accessor methods
 2020-01-28 14:49:00 +09:00
Alex Osborne 00fd8c2960 Merge pull request #306 from internetarchive/fix-stream-closed 
Fix stream closed exception by not closing output stream
 2020-01-28 02:17:07 +09:00
Alex Osborne a4ce59dbd6 Fix stream closed exception by not closing output stream 
ServerCall.writeResponseBody() flushes it after we return so it must
remain open.

Fixes #305
 2020-01-25 08:25:59 +09:00
Tim Hennekey 3003a8e751 Replace constant with accessor methods 
CrawlURI already had the accessor method, and the use of the constant
was a bit inconsistent. This change adds the corresponding mutator
method to make working with the CrawlURI history a bit simpler.
 2020-01-24 12:28:11 -05:00
Alex Osborne 9927e1325f Merge pull request #296 from hennekey/update-junit 
Set JUnit version to latest
 2020-01-23 16:44:59 +09:00
Barbara Miller 8ffcfa9020 Merge pull request #302 from nlevitt/ydl-streaming 
limit ExtractorYoutubeDL heap usage
 2020-01-22 15:08:26 -08:00
Barbara Miller dbade18b58 Merge pull request #301 from nlevitt/fix-logging-config 
fix logging config
 2020-01-22 14:42:22 -08:00
Noah Levitt 6667186826 java 8 compatibility 2020-01-22 14:12:36 -08:00
Noah Levitt 9229a51dcb limit ExtractorYoutubeDL heap usage 
We were seeing OOME due to large youtoube-dl json (for playlists and
such). So instead of storing the json in ram, stream through it, and
stash the contents in an thread-local anonymous tempfile so it can be
written to to warc.
 2020-01-22 13:49:24 -08:00
Noah Levitt 784b3a5dec fix logging config 
by setting system property java.util.logging.config.file, because new
version restlet reconfigures logging after heritrix has already
configured it
 2020-01-22 13:39:24 -08:00
Tim Hennekey 66723aea68 Remove version 
Allow the parent POM to specify the version
 2020-01-21 10:52:15 -05:00
Alex Osborne a8c41aa014 Merge pull request #298 from hennekey/fix-297 
Speed up ObjectIdentityBdbManualCacheTest
 2020-01-16 15:42:06 +09:00
jkafader 739cf7fdb3 Merge pull request #287 from nlevitt/trough-dedup-fix 
change trough dedup `date` type to varchar.
 2020-01-15 11:21:06 -08:00
Noah Levitt 00a1f2c9b0 Merge pull request #294 from internetarchive/fix-config-post 
Fix 'Method Not Allowed' on POST of config editor form
 2020-01-15 11:01:00 -08:00
Noah Levitt ccbf678f4a Merge pull request #295 from internetarchive/disable-wbm-test 
Disable test that connects to wwwb-dedup.us.archive.org
 2020-01-15 10:56:31 -08:00
Tim Hennekey 0fa87a36e8 Add default constructor to IdentityCacheWrapper 
Kryo is spending a lot of time during serialization handling a
NoSuchMethodException. By adding this default constructor we can skip
doing all that work.
 2020-01-15 13:39:27 -05:00
Tim Hennekey 55308ba90a Move IdentityCacheableWrapper 
This is utilized only in tests so it belongs there
 2020-01-15 13:39:27 -05:00
Tim Hennekey 0fe8463e81 Remove AtomicInteger loop counter 
This test isn't multithreaded so I see no need for this
 2020-01-15 13:39:27 -05:00
Tim Hennekey 611fa325df Remove unused code 
It reduces the clarity of the surrounding code
 2020-01-15 13:39:27 -05:00
Tim Hennekey bb61aa0bcb Fix unchecked warning 2020-01-15 13:39:27 -05:00
Tim Hennekey 4be3e620c3 Set JUnit version to latest 
Updating JUnit to begin updating the unit tests
 2020-01-15 11:08:31 -05:00
Alex Osborne 615578b22d Add IntelliJ IDEA and Heritrix runtime files to .gitignore 
I've seen a few people accidentally commit them.
 2020-01-15 18:19:25 +09:00
Alex Osborne dcdaba3b10 Disable test that connects to wwwb-dedup.us.archive.org 
http://wwwb-dedup.us.archive.org:8083/web/timemap/cdx is currently
returning 503 intermittently and so this is breaking Travis builds for
unrelated changes.
 2020-01-15 17:48:31 +09:00
Alex Osborne 34be934da5 Fix 'Method Not Allowed' on POST of config editor form 
Under Restlet 2 `getVariants()` always returns null for a POST so our
post() method was never called and 405 Method Not Allowed was returned.
We don't need content negotiation for the POST response anyway so let's
instead override the no-variants post() like we do for put().

Fixes #293
 2020-01-15 16:51:52 +09:00
Alex Osborne 8e875bb17f Merge pull request #288 from netarchivesuite/inline-image-filter 
Attempt to filter out embedded images.
 2020-01-15 12:48:54 +09:00
jkafader 3a8b58998c Merge pull request #285 from nlevitt/warc-writer-chain 
Warc writer chain
 2020-01-13 16:39:42 -08:00
Noah Levitt 31b9960503 basic level of documentation 2019-12-30 16:02:10 -08:00
Noah Levitt 125547b93d change trough dedup date type to varchar. 
By parsing/unparsing to/from java.util.Date, we ended up with a
different date format in trough (sqlite) than warcprox, which is no
good; see https://github.com/internetarchive/warcprox/pull/144
 2019-11-19 14:34:49 -08:00
Colin Rosenthal 204491c357 Attempt to filter out embedded images. 
(cherry picked from commit aa5ff1dbdaefd04652a9c66506d20f1a6ae01dc3)
 2019-11-19 15:04:50 +01:00
Noah Levitt ff8ebc4c2e use JSONObject.isNull() 
because opt() returns org.json.JSONObject.Null
 2019-11-18 14:42:38 -08:00
Noah Levitt 6e7bc54c9d use org.json like everybody else 2019-11-15 16:14:17 -08:00
Noah Levitt c4b08166f2 extract watch page links from youtube playlists 
and equivalent for other sites. Usually we find these links through
normal link extraction, but we have the info here, so we may as well use
it to make sure.
 2019-11-15 15:52:36 -08:00
Noah Levitt 4983e65929 fix non-playlist case (oops!) 2019-11-15 15:42:59 -08:00
Noah Levitt 1b95453748 be consistent and null-safe with concurrentTo 2019-10-15 10:50:26 -07:00
Noah Levitt 7a83aecb28 Merge pull request #286 from internetarchive/adds-subqueue-support-forced-queue-assignment 
Add support for forced queue assignment and parallel queues
 2019-10-11 13:52:47 -07:00
Noah Levitt 74c0739bb0 fix line ending and indentation issues 2019-10-11 13:49:49 -07:00
Adam Miller 0cee24922f AssignmentLevelSurtQueueAssignmentPolicy.java - Add support for forced queue assignment and parallel queues 
URIAuthorityBasedQueueAssignmentPolicy.java - Add interoperability between forced queue assignment and parallel queues
QuotaEnforcer.java - Fix javadoc to match default behavior
 2019-10-11 19:44:35 +00:00
Noah Levitt feea47b2c7 Merge pull request #283 from internetarchive/jobdir-put-fix 
Fix jobdir PUT
 2019-08-28 12:41:00 -07:00
Alex Osborne eeddfd763f Override PUT so it doesn't change the file extension 
Fixes #282 and HER-1907
 2019-08-28 12:46:29 +09:00
Alex Osborne 5612fa249b Use super.getVariants() rather than super.getVariants(GET) 
This was a regression introduced in the upgrade to Restlet 2. I
encountered a NullPointerException here when upgrading and misunderstood
the cause of it. Since PUT and DELETE return no content they are
actually supposed to return null.
 2019-08-27 23:53:34 +09:00
Noah Levitt 2eafb5a0c4 Merge pull request #280 from internetarchive/fix-cookie-test-failures 
Mitigate random CookieStore.testConcurrentLoad test failures
 2019-08-16 10:08:31 -07:00
Alex Osborne 2a5bd87e51 Remove testConcurrentLoad 
Noah wrote in #280:

> Maybe we should just drop the test. The assumption when we wrote the
> test was that a race condition would not be so frequent in practice.
> We've seen that under the contrived conditions created by the test
> case, it is frequent. But that's ok
 2019-08-16 22:11:26 +09:00
Alex Osborne a5c03d9b73 Mitigate random CookieStore.testConcurrentLoad test failures 
The arbitary value `25` was used but in prace it's quite possible
for more than 25 writing threads to have checked the cookie count
limit before adding their cookie. In practice we see Travis failing
on this test quite often, every few builds in fact.

I think using `threads.length` (i.e. 200) should cover the worst
case possibility where every thread reads a stale count and tries
to add their cookie.

Fixes #274
 2019-08-12 18:08:43 +09:00
Alex Osborne 38c44e8e8e Add missing UUID import (interactive commit fail) 2019-08-12 17:50:35 +09:00
Alex Osborne 36003b8626 Fix digest authentication 
In Restlet 2 it appears we need to use DigestAuthenticator.
(Previously both digest and basic auth were handled by the same
Guard class.)
 2019-08-12 17:36:28 +09:00
Alex Osborne 92ffdede0d Link to javadoc.io for more recent api docs 2019-08-12 15:03:52 +09:00
Alex Osborne 8b72be7d1d Now jdk11 support is merged we can disallow test failures on it 2019-08-08 10:37:38 +09:00
Noah Levitt cefad96036 Merge pull request #276 from internetarchive/jdk11-upgrade-jetty 
JDK11 support: upgrade to Jetty 9.4.19, Restlet 2.4.0 and drop JDK 7 support
 2019-08-07 10:48:51 -07:00