fd514b7bf6
Merge pull request #315 from ukwa/fix-oracle-doc-link
...
Use the Wayback Machine to repair a link to Oracle docs.
2020-03-05 01:40:39 +09:00
9f7887a5fa
Merge pull request #314 from hennekey/parameterize-bloom-filter
...
Utilize the `d` parameter
2020-03-04 16:36:17 +00:00
b4494a7ffb
Use the Wayback Machine to repair a link to Oracle docs.
2020-03-04 15:42:29 +00:00
348b5330bc
Utilize the d parameter
...
There are uses of this class that necessitate a lower false positive
rate than the value that was hard-coded. By making use of the parameter,
as indicated in the JavaDoc, the resulting delegate should meet the
demands of the caller.
2020-03-04 09:37:31 -05:00
e0c82f70e9
Merge pull request #289 from netarchivesuite/bdb-frontier-access
...
Bdb frontier access
2020-03-04 13:41:27 +00:00
7f80b4fd98
Merge pull request #281 from ukwa/upgrade-bdb-je
...
WIP: Upgrade BDB JE to version 7.5.11
2020-03-04 13:36:03 +00:00
977f608c74
Merge pull request #290 from netarchivesuite/crawltrap-regex-timeout
...
Crawltrap regex timeout
2020-03-03 19:27:42 +09:00
e54258756a
Fixed regex timeout handling following suggestion https://github.com/internetarchive/heritrix3/pull/290#discussion_r366711640
2020-03-03 09:02:07 +01:00
60e4053180
Commented this test back in to make Travis happy
2020-03-03 08:52:09 +01:00
66535370ff
Merge pull request #312 from internetarchive/guava-dep
...
Exclude hbase-client's guava 12 transitive dependency
2020-02-28 10:16:55 +09:00
6fcc5e808a
Exclude hbase-client's guava 12 transitive dependency
...
Guava 12 from hbase-client is closer to the root of the dependency tree than guava 17 from webarchive-commons so Maven prefers it. But recent changes to heritrix-commons rely on classes in the newer version of Guava. So let's ensure webarchive-commons wins.
Hopefully this doesn't break the hbase module, I have no way of testing it.
Fixes #311
2020-02-13 16:59:13 +09:00
bc6a15b41d
Use HTTPS to resolve dependencies in Maven Build where possible
2020-02-11 17:48:20 +09:00
015c5e7f80
Merge pull request #308 from ldko/fix-restlet-errors
...
Fix stream closed exception for Paged view
2020-01-31 09:33:26 +09:00
8c1c8009c6
Fix stream closed exception for Paged view
2020-01-30 16:48:38 -06:00
b45f774066
Merge pull request #300 from hennekey/fix-299
...
Use Guice instead of custom implementation
2020-01-30 19:12:05 +09:00
fd9136dbfa
Merge pull request #304 from hennekey/remove-custom-base32
...
Replace custom Base32 encoding
2020-01-30 19:07:09 +09:00
54e05a7864
Correct encoding
...
The previous implementation appears to always have returned upper case,
was able to encode either case, and did not reutrn padding.
2020-01-28 15:12:54 -05:00
3528c990df
Merge pull request #303 from hennekey/refactor-history
...
Replace constant with accessor methods
2020-01-28 14:49:00 +09:00
00fd8c2960
Merge pull request #306 from internetarchive/fix-stream-closed
...
Fix stream closed exception by not closing output stream
2020-01-28 02:17:07 +09:00
a4ce59dbd6
Fix stream closed exception by not closing output stream
...
ServerCall.writeResponseBody() flushes it after we return so it must
remain open.
Fixes #305
2020-01-25 08:25:59 +09:00
b5f95c5e06
Replace custom Base32 encoding
...
Guava is available so a custom implementation is unnecessary.
2020-01-24 12:46:37 -05:00
3003a8e751
Replace constant with accessor methods
...
CrawlURI already had the accessor method, and the use of the constant
was a bit inconsistent. This change adds the corresponding mutator
method to make working with the CrawlURI history a bit simpler.
2020-01-24 12:28:11 -05:00
9927e1325f
Merge pull request #296 from hennekey/update-junit
...
Set JUnit version to latest
2020-01-23 16:44:59 +09:00
8ffcfa9020
Merge pull request #302 from nlevitt/ydl-streaming
...
limit ExtractorYoutubeDL heap usage
2020-01-22 15:08:26 -08:00
dbade18b58
Merge pull request #301 from nlevitt/fix-logging-config
...
fix logging config
2020-01-22 14:42:22 -08:00
9cb9563da3
Increment the count only when the filter notes it
...
Otherwise this is a count of how many times this add method is called,
not how many times an element was noted as being actually added.
2020-01-22 17:15:24 -05:00
33458f1518
Fix assertions
...
By using assertEquals and seting the expected and actual values, the
failure messages become a bit more useful.
2020-01-22 17:13:58 -05:00
6667186826
java 8 compatibility
2020-01-22 14:12:36 -08:00
9229a51dcb
limit ExtractorYoutubeDL heap usage
...
We were seeing OOME due to large youtoube-dl json (for playlists and
such). So instead of storing the json in ram, stream through it, and
stash the contents in an thread-local anonymous tempfile so it can be
written to to warc.
2020-01-22 13:49:24 -08:00
784b3a5dec
fix logging config
...
by setting system property java.util.logging.config.file, because new
version restlet reconfigures logging after heritrix has already
configured it
2020-01-22 13:39:24 -08:00
66723aea68
Remove version
...
Allow the parent POM to specify the version
2020-01-21 10:52:15 -05:00
a8c41aa014
Merge pull request #298 from hennekey/fix-297
...
Speed up ObjectIdentityBdbManualCacheTest
2020-01-16 15:42:06 +09:00
080380b18a
Use Guice instead of custom implementation
...
This uses the avaialable code in Guice rather than a custom
implementation. It also provides a performance increase (as demonstrated
by the unit tests)
2020-01-15 18:13:20 -05:00
739cf7fdb3
Merge pull request #287 from nlevitt/trough-dedup-fix
...
change trough dedup `date` type to varchar.
2020-01-15 11:21:06 -08:00
00a1f2c9b0
Merge pull request #294 from internetarchive/fix-config-post
...
Fix 'Method Not Allowed' on POST of config editor form
2020-01-15 11:01:00 -08:00
ccbf678f4a
Merge pull request #295 from internetarchive/disable-wbm-test
...
Disable test that connects to wwwb-dedup.us.archive.org
2020-01-15 10:56:31 -08:00
0fa87a36e8
Add default constructor to IdentityCacheWrapper
...
Kryo is spending a lot of time during serialization handling a
NoSuchMethodException. By adding this default constructor we can skip
doing all that work.
2020-01-15 13:39:27 -05:00
55308ba90a
Move IdentityCacheableWrapper
...
This is utilized only in tests so it belongs there
2020-01-15 13:39:27 -05:00
0fe8463e81
Remove AtomicInteger loop counter
...
This test isn't multithreaded so I see no need for this
2020-01-15 13:39:27 -05:00
611fa325df
Remove unused code
...
It reduces the clarity of the surrounding code
2020-01-15 13:39:27 -05:00
bb61aa0bcb
Fix unchecked warning
2020-01-15 13:39:27 -05:00
4be3e620c3
Set JUnit version to latest
...
Updating JUnit to begin updating the unit tests
2020-01-15 11:08:31 -05:00
615578b22d
Add IntelliJ IDEA and Heritrix runtime files to .gitignore
...
I've seen a few people accidentally commit them.
2020-01-15 18:19:25 +09:00
dcdaba3b10
Disable test that connects to wwwb-dedup.us.archive.org
...
http://wwwb-dedup.us.archive.org:8083/web/timemap/cdx is currently
returning 503 intermittently and so this is breaking Travis builds for
unrelated changes.
2020-01-15 17:48:31 +09:00
34be934da5
Fix 'Method Not Allowed' on POST of config editor form
...
Under Restlet 2 `getVariants()` always returns null for a POST so our
post() method was never called and 405 Method Not Allowed was returned.
We don't need content negotiation for the POST response anyway so let's
instead override the no-variants post() like we do for put().
Fixes #293
2020-01-15 16:51:52 +09:00
8e875bb17f
Merge pull request #288 from netarchivesuite/inline-image-filter
...
Attempt to filter out embedded images.
2020-01-15 12:48:54 +09:00
3a8b58998c
Merge pull request #285 from nlevitt/warc-writer-chain
...
Warc writer chain
2020-01-13 16:39:42 -08:00
31b9960503
basic level of documentation
2019-12-30 16:02:10 -08:00
125547b93d
change trough dedup date type to varchar.
...
By parsing/unparsing to/from java.util.Date, we ended up with a
different date format in trough (sqlite) than warcprox, which is no
good; see https://github.com/internetarchive/warcprox/pull/144
2019-11-19 14:34:49 -08:00
204491c357
Attempt to filter out embedded images.
...
(cherry picked from commit aa5ff1dbdaefd04652a9c66506d20f1a6ae01dc3)
2019-11-19 15:04:50 +01:00
Alex Osborne