Commit Graph
868 Commits
Author SHA1 Message Date
Andrew Jackson 2d64e005cb Switch to standard cookie handling. 2021-06-17 14:23:20 +01:00
Andrew Jackson 1d1d2f5abe Avoid two slf4j implementations. 2021-06-17 14:19:12 +01:00
Andrew Jackson 4cd416a5e4 Update complier plugin and use compatible flip(). 2021-06-17 14:01:04 +01:00
Andrew Jackson d892026731 Expire cookies when requested. 2021-06-15 00:31:04 +01:00
Andrew Jackson 81fa60d870 Upgrade HTTPClient to 4.5.x for #245 2021-06-14 23:35:46 +01:00
Andrew Jackson d3dfb34f83 [maven-release-plugin] prepare for next development iteration 2021-05-27 10:03:10 +01:00
Andrew Jackson 9dcbb240a6 [maven-release-plugin] prepare release 3.4.0-20210527 2021-05-27 10:03:04 +01:00
Andy Jackson a2c65f4618 Update dependencies for security issues, including changes to tests. 2021-05-26 13:39:30 +01:00
Andy Jackson c0dbfb2a0a Merge pull request #383 from ukwa/upgrade-dnsjava
Update to latest version of dnsjava, for #344
2021-05-20 21:05:59 +01:00
Andy Jackson 396467c535 Merge branch 'master' into sitemaps 2021-05-20 21:03:54 +01:00
Andy Jackson c1bcdd9f9e Merge pull request #341 from internetarchive/noplaylist-ydl
youtube-dl --no-playlist
2021-05-20 21:01:39 +01:00
Andy Jackson b2a07b290f Merge pull request #348 from internetarchive/fixes-leaky-file-handles
Fixes leaky file handles
2021-05-20 21:00:31 +01:00
Andy Jackson 045b2516da Update to latest version of dnsjava, for #344
This commit updates to version 3.3.1 of the dnsjava library.

Incidentally, the restlet.org certificate is currently invalid, so this
patch switched to maven.restlet.com.
2021-05-20 20:44:19 +01:00
Kristinn Sigurdsson 52093eb25f Only copy source tag if not null. 2021-04-20 13:18:29 +00:00
Neil Minton 2505effadc Strip quotes from URL value. 2020-09-01 10:57:58 -04:00
Adam Miller 52a8f345ed Fixing up logging and comments 2020-07-30 23:30:08 +00:00
Adam Miller 00d1c46d60 Ensure Replay Input Stream and File Channels are closed after writing 2020-07-30 23:19:43 +00:00
Alex Osborne 947507b8ec Revert "Warc convention for storing ftp responses has been to use a WARC reso…" 2020-06-01 13:56:12 +09:00
Alex Osborne 4763a8a04b Merge pull request #334 from internetarchive/fixes-ftp-response-record-builder
Warc convention for storing ftp responses has been to use a WARC reso…
2020-06-01 13:54:59 +09:00
Adam Miller 60daef90b2 Merge branch 'master' into fixes-extractor-multiple-regex-matcher-recycle 2020-05-20 16:46:14 +00:00
Adam Miller c0fe2064a1 Merge branch 'fixes-ftp-response-record-builder' 2020-05-20 16:32:36 +00:00
Andrew Jackson e215b89963 [maven-release-plugin] prepare for next development iteration 2020-05-18 14:08:12 +01:00
Andrew Jackson ef38dc08c3 [maven-release-plugin] prepare release 3.4.0-20200518 2020-05-18 14:08:04 +01:00
Adam Miller 0db661fb36 Warc convention for storing ftp responses has been to use a WARC resource record and not a response record. Changing to be consisten with the default WARCWriterProcessor 2020-05-05 23:09:53 +00:00
morokosi 13ba0b2b67 optimize imports 2020-04-30 22:40:00 +09:00
morokosi 18bee39a4c fix discarded future value 2020-04-30 22:39:51 +09:00
Alex Osborne 7764f9265f Merge pull request #320 from bnfleb/sftp
Add support for the SFTP protocol
2020-04-24 08:13:39 +09:00
Leslie Bellony aad54c3cb3 rename variable + javadoc 2020-04-23 15:43:30 +02:00
Clara Wiatrowski 5e948883fa Add overrides in JerichoExtractorHTMLTest 2020-04-22 15:33:23 +02:00
Clara Wiatrowski 0d1f4ea84f Add parsing case data-* 2020-04-22 11:48:05 +02:00
Clara Wiatrowski cc29bb97d1 Revert ExtractorHTML 2020-04-22 11:33:24 +02:00
Clara Wiatrowski 6710c8dba3 Add parsing for HTML tags (data-src, data-srcset, data-original and data-original-set) 2020-04-10 15:59:50 +02:00
Leslie Bellony 40725372d6 Add support for the SFTP protocol 2020-04-01 14:47:34 +02:00
Adam Miller a384dea2f9 Recycle the Matcher after use. 2020-03-25 22:33:06 +00:00
Andrew Jackson ff05de44b6 [maven-release-plugin] prepare for next development iteration 2020-03-04 23:05:41 +00:00
Andrew Jackson a1bdcb1be7 [maven-release-plugin] prepare release 3.4.0-20200304 2020-03-04 23:05:33 +00:00
Andy Jackson 7f80b4fd98 Merge pull request #281 from ukwa/upgrade-bdb-je
WIP: Upgrade BDB JE to version 7.5.11
2020-03-04 13:36:03 +00:00
Alex Osborne 977f608c74 Merge pull request #290 from netarchivesuite/crawltrap-regex-timeout
Crawltrap regex timeout
2020-03-03 19:27:42 +09:00
Colin Rosenthal e54258756a Fixed regex timeout handling following suggestion https://github.com/internetarchive/heritrix3/pull/290#discussion_r366711640 2020-03-03 09:02:07 +01:00
Colin Rosenthal 60e4053180 Commented this test back in to make Travis happy 2020-03-03 08:52:09 +01:00
Alex Osborne 3528c990df Merge pull request #303 from hennekey/refactor-history
Replace constant with accessor methods
2020-01-28 14:49:00 +09:00
Tim Hennekey 3003a8e751 Replace constant with accessor methods
CrawlURI already had the accessor method, and the use of the constant
was a bit inconsistent. This change adds the corresponding mutator
method to make working with the CrawlURI history a bit simpler.
2020-01-24 12:28:11 -05:00
Alex Osborne 8e875bb17f Merge pull request #288 from netarchivesuite/inline-image-filter
Attempt to filter out embedded images.
2020-01-15 12:48:54 +09:00
jkafader 3a8b58998c Merge pull request #285 from nlevitt/warc-writer-chain
Warc writer chain
2020-01-13 16:39:42 -08:00
Noah Levitt 31b9960503 basic level of documentation 2019-12-30 16:02:10 -08:00
Colin Rosenthal 204491c357 Attempt to filter out embedded images.
(cherry picked from commit aa5ff1dbdaefd04652a9c66506d20f1a6ae01dc3)
2019-11-19 15:04:50 +01:00
Colin Rosenthal 718cc5ae32 Updated manually to new SNAPSHOT version 2019-11-15 14:34:03 +01:00
Andrew Jackson 73db3ac628 Merge branch 'master' into upgrade-bdb-je 2019-10-17 14:10:22 +01:00
Noah Levitt 1b95453748 be consistent and null-safe with concurrentTo 2019-10-15 10:50:26 -07:00
Alex Osborne 2a5bd87e51 Remove testConcurrentLoad
Noah wrote in #280:

> Maybe we should just drop the test. The assumption when we wrote the
> test was that a race condition would not be so frequent in practice.
> We've seen that under the contrived conditions created by the test
> case, it is frequent. But that's ok
2019-08-16 22:11:26 +09:00