Commit Graph
866 Commits
Author SHA1 Message Date
Andrew Jackson 4cd416a5e4 Update complier plugin and use compatible flip(). 2021-06-17 14:01:04 +01:00
Andrew Jackson d892026731 Expire cookies when requested. 2021-06-15 00:31:04 +01:00
Andrew Jackson 81fa60d870 Upgrade HTTPClient to 4.5.x for #245 2021-06-14 23:35:46 +01:00
Andrew Jackson d3dfb34f83 [maven-release-plugin] prepare for next development iteration 2021-05-27 10:03:10 +01:00
Andrew Jackson 9dcbb240a6 [maven-release-plugin] prepare release 3.4.0-20210527 2021-05-27 10:03:04 +01:00
Andy Jackson a2c65f4618 Update dependencies for security issues, including changes to tests. 2021-05-26 13:39:30 +01:00
Andy Jackson c0dbfb2a0a Merge pull request #383 from ukwa/upgrade-dnsjava
Update to latest version of dnsjava, for #344
2021-05-20 21:05:59 +01:00
Andy Jackson 396467c535 Merge branch 'master' into sitemaps 2021-05-20 21:03:54 +01:00
Andy Jackson c1bcdd9f9e Merge pull request #341 from internetarchive/noplaylist-ydl
youtube-dl --no-playlist
2021-05-20 21:01:39 +01:00
Andy Jackson b2a07b290f Merge pull request #348 from internetarchive/fixes-leaky-file-handles
Fixes leaky file handles
2021-05-20 21:00:31 +01:00
Andy Jackson 045b2516da Update to latest version of dnsjava, for #344
This commit updates to version 3.3.1 of the dnsjava library.

Incidentally, the restlet.org certificate is currently invalid, so this
patch switched to maven.restlet.com.
2021-05-20 20:44:19 +01:00
Kristinn Sigurdsson 52093eb25f Only copy source tag if not null. 2021-04-20 13:18:29 +00:00
Neil Minton 2505effadc Strip quotes from URL value. 2020-09-01 10:57:58 -04:00
Adam Miller 52a8f345ed Fixing up logging and comments 2020-07-30 23:30:08 +00:00
Adam Miller 00d1c46d60 Ensure Replay Input Stream and File Channels are closed after writing 2020-07-30 23:19:43 +00:00
Alex Osborne 947507b8ec Revert "Warc convention for storing ftp responses has been to use a WARC reso…" 2020-06-01 13:56:12 +09:00
Alex Osborne 4763a8a04b Merge pull request #334 from internetarchive/fixes-ftp-response-record-builder
Warc convention for storing ftp responses has been to use a WARC reso…
2020-06-01 13:54:59 +09:00
Adam Miller 60daef90b2 Merge branch 'master' into fixes-extractor-multiple-regex-matcher-recycle 2020-05-20 16:46:14 +00:00
Adam Miller c0fe2064a1 Merge branch 'fixes-ftp-response-record-builder' 2020-05-20 16:32:36 +00:00
Andrew Jackson e215b89963 [maven-release-plugin] prepare for next development iteration 2020-05-18 14:08:12 +01:00
Andrew Jackson ef38dc08c3 [maven-release-plugin] prepare release 3.4.0-20200518 2020-05-18 14:08:04 +01:00
Adam Miller 0db661fb36 Warc convention for storing ftp responses has been to use a WARC resource record and not a response record. Changing to be consisten with the default WARCWriterProcessor 2020-05-05 23:09:53 +00:00
morokosi 13ba0b2b67 optimize imports 2020-04-30 22:40:00 +09:00
morokosi 18bee39a4c fix discarded future value 2020-04-30 22:39:51 +09:00
Alex Osborne 7764f9265f Merge pull request #320 from bnfleb/sftp
Add support for the SFTP protocol
2020-04-24 08:13:39 +09:00
Leslie Bellony aad54c3cb3 rename variable + javadoc 2020-04-23 15:43:30 +02:00
Clara Wiatrowski 5e948883fa Add overrides in JerichoExtractorHTMLTest 2020-04-22 15:33:23 +02:00
Clara Wiatrowski 0d1f4ea84f Add parsing case data-* 2020-04-22 11:48:05 +02:00
Clara Wiatrowski cc29bb97d1 Revert ExtractorHTML 2020-04-22 11:33:24 +02:00
Clara Wiatrowski 6710c8dba3 Add parsing for HTML tags (data-src, data-srcset, data-original and data-original-set) 2020-04-10 15:59:50 +02:00
Leslie Bellony 40725372d6 Add support for the SFTP protocol 2020-04-01 14:47:34 +02:00
Adam Miller a384dea2f9 Recycle the Matcher after use. 2020-03-25 22:33:06 +00:00
Andrew Jackson ff05de44b6 [maven-release-plugin] prepare for next development iteration 2020-03-04 23:05:41 +00:00
Andrew Jackson a1bdcb1be7 [maven-release-plugin] prepare release 3.4.0-20200304 2020-03-04 23:05:33 +00:00
Andy Jackson 7f80b4fd98 Merge pull request #281 from ukwa/upgrade-bdb-je
WIP: Upgrade BDB JE to version 7.5.11
2020-03-04 13:36:03 +00:00
Alex Osborne 977f608c74 Merge pull request #290 from netarchivesuite/crawltrap-regex-timeout
Crawltrap regex timeout
2020-03-03 19:27:42 +09:00
Colin Rosenthal e54258756a Fixed regex timeout handling following suggestion https://github.com/internetarchive/heritrix3/pull/290#discussion_r366711640 2020-03-03 09:02:07 +01:00
Colin Rosenthal 60e4053180 Commented this test back in to make Travis happy 2020-03-03 08:52:09 +01:00
Alex Osborne 3528c990df Merge pull request #303 from hennekey/refactor-history
Replace constant with accessor methods
2020-01-28 14:49:00 +09:00
Tim Hennekey 3003a8e751 Replace constant with accessor methods
CrawlURI already had the accessor method, and the use of the constant
was a bit inconsistent. This change adds the corresponding mutator
method to make working with the CrawlURI history a bit simpler.
2020-01-24 12:28:11 -05:00
Alex Osborne 8e875bb17f Merge pull request #288 from netarchivesuite/inline-image-filter
Attempt to filter out embedded images.
2020-01-15 12:48:54 +09:00
jkafader 3a8b58998c Merge pull request #285 from nlevitt/warc-writer-chain
Warc writer chain
2020-01-13 16:39:42 -08:00
Noah Levitt 31b9960503 basic level of documentation 2019-12-30 16:02:10 -08:00
Colin Rosenthal 204491c357 Attempt to filter out embedded images.
(cherry picked from commit aa5ff1dbdaefd04652a9c66506d20f1a6ae01dc3)
2019-11-19 15:04:50 +01:00
Colin Rosenthal 718cc5ae32 Updated manually to new SNAPSHOT version 2019-11-15 14:34:03 +01:00
Andrew Jackson 73db3ac628 Merge branch 'master' into upgrade-bdb-je 2019-10-17 14:10:22 +01:00
Noah Levitt 1b95453748 be consistent and null-safe with concurrentTo 2019-10-15 10:50:26 -07:00
Alex Osborne 2a5bd87e51 Remove testConcurrentLoad
Noah wrote in #280:

> Maybe we should just drop the test. The assumption when we wrote the
> test was that a race condition would not be so frequent in practice.
> We've seen that under the contrived conditions created by the test
> case, it is frequent. But that's ok
2019-08-16 22:11:26 +09:00
Alex Osborne a5c03d9b73 Mitigate random CookieStore.testConcurrentLoad test failures
The arbitary value `25` was used but in prace it's quite possible
for more than 25 writing threads to have checked the cookie count
limit before adding their cookie. In practice we see Travis failing
on this test quite often, every few builds in fact.

I think using `threads.length` (i.e. 200) should cover the worst
case possibility where every thread reads a stale count and tries
to add their cookie.

Fixes #274
2019-08-12 18:08:43 +09:00
Andrew Jackson 7dab6ec275 Avoid using Thread.interrupt as this freaks BDB-JE. 2019-08-11 22:03:55 +01:00