Andrew Jackson
4cd416a5e4
Update complier plugin and use compatible flip().
2021-06-17 14:01:04 +01:00
Andrew Jackson
d892026731
Expire cookies when requested.
2021-06-15 00:31:04 +01:00
Andrew Jackson
81fa60d870
Upgrade HTTPClient to 4.5.x for #245
2021-06-14 23:35:46 +01:00
Andrew Jackson
d3dfb34f83
[maven-release-plugin] prepare for next development iteration
2021-05-27 10:03:10 +01:00
Andrew Jackson
9dcbb240a6
[maven-release-plugin] prepare release 3.4.0-20210527
2021-05-27 10:03:04 +01:00
Andy Jackson
a2c65f4618
Update dependencies for security issues, including changes to tests.
2021-05-26 13:39:30 +01:00
Andy Jackson
c0dbfb2a0a
Merge pull request #383 from ukwa/upgrade-dnsjava
...
Update to latest version of dnsjava, for #344
2021-05-20 21:05:59 +01:00
Andy Jackson
396467c535
Merge branch 'master' into sitemaps
2021-05-20 21:03:54 +01:00
Andy Jackson
c1bcdd9f9e
Merge pull request #341 from internetarchive/noplaylist-ydl
...
youtube-dl --no-playlist
2021-05-20 21:01:39 +01:00
Andy Jackson
b2a07b290f
Merge pull request #348 from internetarchive/fixes-leaky-file-handles
...
Fixes leaky file handles
2021-05-20 21:00:31 +01:00
Andy Jackson
045b2516da
Update to latest version of dnsjava, for #344
...
This commit updates to version 3.3.1 of the dnsjava library.
Incidentally, the restlet.org certificate is currently invalid, so this
patch switched to maven.restlet.com.
2021-05-20 20:44:19 +01:00
Kristinn Sigurdsson
52093eb25f
Only copy source tag if not null.
2021-04-20 13:18:29 +00:00
Neil Minton
2505effadc
Strip quotes from URL value.
2020-09-01 10:57:58 -04:00
Adam Miller
52a8f345ed
Fixing up logging and comments
2020-07-30 23:30:08 +00:00
Adam Miller
00d1c46d60
Ensure Replay Input Stream and File Channels are closed after writing
2020-07-30 23:19:43 +00:00
Alex Osborne
947507b8ec
Revert "Warc convention for storing ftp responses has been to use a WARC reso…"
2020-06-01 13:56:12 +09:00
Alex Osborne
4763a8a04b
Merge pull request #334 from internetarchive/fixes-ftp-response-record-builder
...
Warc convention for storing ftp responses has been to use a WARC reso…
2020-06-01 13:54:59 +09:00
Adam Miller
60daef90b2
Merge branch 'master' into fixes-extractor-multiple-regex-matcher-recycle
2020-05-20 16:46:14 +00:00
Adam Miller
c0fe2064a1
Merge branch 'fixes-ftp-response-record-builder'
2020-05-20 16:32:36 +00:00
Andrew Jackson
e215b89963
[maven-release-plugin] prepare for next development iteration
2020-05-18 14:08:12 +01:00
Andrew Jackson
ef38dc08c3
[maven-release-plugin] prepare release 3.4.0-20200518
2020-05-18 14:08:04 +01:00
Adam Miller
0db661fb36
Warc convention for storing ftp responses has been to use a WARC resource record and not a response record. Changing to be consisten with the default WARCWriterProcessor
2020-05-05 23:09:53 +00:00
morokosi
13ba0b2b67
optimize imports
2020-04-30 22:40:00 +09:00
morokosi
18bee39a4c
fix discarded future value
2020-04-30 22:39:51 +09:00
Alex Osborne
7764f9265f
Merge pull request #320 from bnfleb/sftp
...
Add support for the SFTP protocol
2020-04-24 08:13:39 +09:00
Leslie Bellony
aad54c3cb3
rename variable + javadoc
2020-04-23 15:43:30 +02:00
Clara Wiatrowski
5e948883fa
Add overrides in JerichoExtractorHTMLTest
2020-04-22 15:33:23 +02:00
Clara Wiatrowski
0d1f4ea84f
Add parsing case data-*
2020-04-22 11:48:05 +02:00
Clara Wiatrowski
cc29bb97d1
Revert ExtractorHTML
2020-04-22 11:33:24 +02:00
Clara Wiatrowski
6710c8dba3
Add parsing for HTML tags (data-src, data-srcset, data-original and data-original-set)
2020-04-10 15:59:50 +02:00
Leslie Bellony
40725372d6
Add support for the SFTP protocol
2020-04-01 14:47:34 +02:00
Adam Miller
a384dea2f9
Recycle the Matcher after use.
2020-03-25 22:33:06 +00:00
Andrew Jackson
ff05de44b6
[maven-release-plugin] prepare for next development iteration
2020-03-04 23:05:41 +00:00
Andrew Jackson
a1bdcb1be7
[maven-release-plugin] prepare release 3.4.0-20200304
2020-03-04 23:05:33 +00:00
Andy Jackson
7f80b4fd98
Merge pull request #281 from ukwa/upgrade-bdb-je
...
WIP: Upgrade BDB JE to version 7.5.11
2020-03-04 13:36:03 +00:00
Alex Osborne
977f608c74
Merge pull request #290 from netarchivesuite/crawltrap-regex-timeout
...
Crawltrap regex timeout
2020-03-03 19:27:42 +09:00
Colin Rosenthal
e54258756a
Fixed regex timeout handling following suggestion https://github.com/internetarchive/heritrix3/pull/290#discussion_r366711640
2020-03-03 09:02:07 +01:00
Colin Rosenthal
60e4053180
Commented this test back in to make Travis happy
2020-03-03 08:52:09 +01:00
Alex Osborne
3528c990df
Merge pull request #303 from hennekey/refactor-history
...
Replace constant with accessor methods
2020-01-28 14:49:00 +09:00
Tim Hennekey
3003a8e751
Replace constant with accessor methods
...
CrawlURI already had the accessor method, and the use of the constant
was a bit inconsistent. This change adds the corresponding mutator
method to make working with the CrawlURI history a bit simpler.
2020-01-24 12:28:11 -05:00
Alex Osborne
8e875bb17f
Merge pull request #288 from netarchivesuite/inline-image-filter
...
Attempt to filter out embedded images.
2020-01-15 12:48:54 +09:00
jkafader
3a8b58998c
Merge pull request #285 from nlevitt/warc-writer-chain
...
Warc writer chain
2020-01-13 16:39:42 -08:00
Noah Levitt
31b9960503
basic level of documentation
2019-12-30 16:02:10 -08:00
Colin Rosenthal
204491c357
Attempt to filter out embedded images.
...
(cherry picked from commit aa5ff1dbdaefd04652a9c66506d20f1a6ae01dc3)
2019-11-19 15:04:50 +01:00
Colin Rosenthal
718cc5ae32
Updated manually to new SNAPSHOT version
2019-11-15 14:34:03 +01:00
Andrew Jackson
73db3ac628
Merge branch 'master' into upgrade-bdb-je
2019-10-17 14:10:22 +01:00
Noah Levitt
1b95453748
be consistent and null-safe with concurrentTo
2019-10-15 10:50:26 -07:00
Alex Osborne
2a5bd87e51
Remove testConcurrentLoad
...
Noah wrote in #280 :
> Maybe we should just drop the test. The assumption when we wrote the
> test was that a race condition would not be so frequent in practice.
> We've seen that under the contrived conditions created by the test
> case, it is frequent. But that's ok
2019-08-16 22:11:26 +09:00
Alex Osborne
a5c03d9b73
Mitigate random CookieStore.testConcurrentLoad test failures
...
The arbitary value `25` was used but in prace it's quite possible
for more than 25 writing threads to have checked the cookie count
limit before adding their cookie. In practice we see Travis failing
on this test quite often, every few builds in fact.
I think using `threads.length` (i.e. 200) should cover the worst
case possibility where every thread reads a stale count and tries
to add their cookie.
Fixes #274
2019-08-12 18:08:43 +09:00
Andrew Jackson
7dab6ec275
Avoid using Thread.interrupt as this freaks BDB-JE.
2019-08-11 22:03:55 +01:00