Andrew Jackson
2d64e005cb
Switch to standard cookie handling.
2021-06-17 14:23:20 +01:00
Andrew Jackson
1d1d2f5abe
Avoid two slf4j implementations.
2021-06-17 14:19:12 +01:00
Andrew Jackson
4cd416a5e4
Update complier plugin and use compatible flip().
2021-06-17 14:01:04 +01:00
Andrew Jackson
d892026731
Expire cookies when requested.
2021-06-15 00:31:04 +01:00
Andrew Jackson
81fa60d870
Upgrade HTTPClient to 4.5.x for #245
2021-06-14 23:35:46 +01:00
Andrew Jackson
d3dfb34f83
[maven-release-plugin] prepare for next development iteration
2021-05-27 10:03:10 +01:00
Andrew Jackson
9dcbb240a6
[maven-release-plugin] prepare release 3.4.0-20210527
2021-05-27 10:03:04 +01:00
Andy Jackson
a2c65f4618
Update dependencies for security issues, including changes to tests.
2021-05-26 13:39:30 +01:00
Andy Jackson
c0dbfb2a0a
Merge pull request #383 from ukwa/upgrade-dnsjava
...
Update to latest version of dnsjava, for #344
2021-05-20 21:05:59 +01:00
Andy Jackson
396467c535
Merge branch 'master' into sitemaps
2021-05-20 21:03:54 +01:00
Andy Jackson
c1bcdd9f9e
Merge pull request #341 from internetarchive/noplaylist-ydl
...
youtube-dl --no-playlist
2021-05-20 21:01:39 +01:00
Andy Jackson
b2a07b290f
Merge pull request #348 from internetarchive/fixes-leaky-file-handles
...
Fixes leaky file handles
2021-05-20 21:00:31 +01:00
Andy Jackson
045b2516da
Update to latest version of dnsjava, for #344
...
This commit updates to version 3.3.1 of the dnsjava library.
Incidentally, the restlet.org certificate is currently invalid, so this
patch switched to maven.restlet.com.
2021-05-20 20:44:19 +01:00
Kristinn Sigurdsson
52093eb25f
Only copy source tag if not null.
2021-04-20 13:18:29 +00:00
Neil Minton
2505effadc
Strip quotes from URL value.
2020-09-01 10:57:58 -04:00
Adam Miller
52a8f345ed
Fixing up logging and comments
2020-07-30 23:30:08 +00:00
Adam Miller
00d1c46d60
Ensure Replay Input Stream and File Channels are closed after writing
2020-07-30 23:19:43 +00:00
Alex Osborne
947507b8ec
Revert "Warc convention for storing ftp responses has been to use a WARC reso…"
2020-06-01 13:56:12 +09:00
Alex Osborne
4763a8a04b
Merge pull request #334 from internetarchive/fixes-ftp-response-record-builder
...
Warc convention for storing ftp responses has been to use a WARC reso…
2020-06-01 13:54:59 +09:00
Adam Miller
60daef90b2
Merge branch 'master' into fixes-extractor-multiple-regex-matcher-recycle
2020-05-20 16:46:14 +00:00
Adam Miller
c0fe2064a1
Merge branch 'fixes-ftp-response-record-builder'
2020-05-20 16:32:36 +00:00
Andrew Jackson
e215b89963
[maven-release-plugin] prepare for next development iteration
2020-05-18 14:08:12 +01:00
Andrew Jackson
ef38dc08c3
[maven-release-plugin] prepare release 3.4.0-20200518
2020-05-18 14:08:04 +01:00
Adam Miller
0db661fb36
Warc convention for storing ftp responses has been to use a WARC resource record and not a response record. Changing to be consisten with the default WARCWriterProcessor
2020-05-05 23:09:53 +00:00
morokosi
13ba0b2b67
optimize imports
2020-04-30 22:40:00 +09:00
morokosi
18bee39a4c
fix discarded future value
2020-04-30 22:39:51 +09:00
Alex Osborne
7764f9265f
Merge pull request #320 from bnfleb/sftp
...
Add support for the SFTP protocol
2020-04-24 08:13:39 +09:00
Leslie Bellony
aad54c3cb3
rename variable + javadoc
2020-04-23 15:43:30 +02:00
Clara Wiatrowski
5e948883fa
Add overrides in JerichoExtractorHTMLTest
2020-04-22 15:33:23 +02:00
Clara Wiatrowski
0d1f4ea84f
Add parsing case data-*
2020-04-22 11:48:05 +02:00
Clara Wiatrowski
cc29bb97d1
Revert ExtractorHTML
2020-04-22 11:33:24 +02:00
Clara Wiatrowski
6710c8dba3
Add parsing for HTML tags (data-src, data-srcset, data-original and data-original-set)
2020-04-10 15:59:50 +02:00
Leslie Bellony
40725372d6
Add support for the SFTP protocol
2020-04-01 14:47:34 +02:00
Adam Miller
a384dea2f9
Recycle the Matcher after use.
2020-03-25 22:33:06 +00:00
Andrew Jackson
ff05de44b6
[maven-release-plugin] prepare for next development iteration
2020-03-04 23:05:41 +00:00
Andrew Jackson
a1bdcb1be7
[maven-release-plugin] prepare release 3.4.0-20200304
2020-03-04 23:05:33 +00:00
Andy Jackson
7f80b4fd98
Merge pull request #281 from ukwa/upgrade-bdb-je
...
WIP: Upgrade BDB JE to version 7.5.11
2020-03-04 13:36:03 +00:00
Alex Osborne
977f608c74
Merge pull request #290 from netarchivesuite/crawltrap-regex-timeout
...
Crawltrap regex timeout
2020-03-03 19:27:42 +09:00
Colin Rosenthal
e54258756a
Fixed regex timeout handling following suggestion https://github.com/internetarchive/heritrix3/pull/290#discussion_r366711640
2020-03-03 09:02:07 +01:00
Colin Rosenthal
60e4053180
Commented this test back in to make Travis happy
2020-03-03 08:52:09 +01:00
Alex Osborne
3528c990df
Merge pull request #303 from hennekey/refactor-history
...
Replace constant with accessor methods
2020-01-28 14:49:00 +09:00
Tim Hennekey
3003a8e751
Replace constant with accessor methods
...
CrawlURI already had the accessor method, and the use of the constant
was a bit inconsistent. This change adds the corresponding mutator
method to make working with the CrawlURI history a bit simpler.
2020-01-24 12:28:11 -05:00
Alex Osborne
8e875bb17f
Merge pull request #288 from netarchivesuite/inline-image-filter
...
Attempt to filter out embedded images.
2020-01-15 12:48:54 +09:00
jkafader
3a8b58998c
Merge pull request #285 from nlevitt/warc-writer-chain
...
Warc writer chain
2020-01-13 16:39:42 -08:00
Noah Levitt
31b9960503
basic level of documentation
2019-12-30 16:02:10 -08:00
Colin Rosenthal
204491c357
Attempt to filter out embedded images.
...
(cherry picked from commit aa5ff1dbdaefd04652a9c66506d20f1a6ae01dc3)
2019-11-19 15:04:50 +01:00
Colin Rosenthal
718cc5ae32
Updated manually to new SNAPSHOT version
2019-11-15 14:34:03 +01:00
Andrew Jackson
73db3ac628
Merge branch 'master' into upgrade-bdb-je
2019-10-17 14:10:22 +01:00
Noah Levitt
1b95453748
be consistent and null-safe with concurrentTo
2019-10-15 10:50:26 -07:00
Alex Osborne
2a5bd87e51
Remove testConcurrentLoad
...
Noah wrote in #280 :
> Maybe we should just drop the test. The assumption when we wrote the
> test was that a race condition would not be so frequent in practice.
> We've seen that under the contrived conditions created by the test
> case, it is frequent. But that's ok
2019-08-16 22:11:26 +09:00