Adam Miller and Alex Osborne
ca3397e346
Fixing race condition on totalBytesWritten
2021-07-03 12:10:21 +09:00
Adam Miller and Alex Osborne
111b77b4aa
Add checkpoint store/load functionality for warc writer stats
2021-07-03 12:10:21 +09:00
Alex Osborne
200f1cf38f
DNS: Use the new protocol-agnostic server IP attribute
...
Also fixes an issue where in some cases we would incorrectly output
WARC-IP-Address for DNS records as "/127.0.0.53:53" instead of
"127.0.0.53".
2021-06-25 16:39:59 +09:00
Alex Osborne
c378c46e49
WHOIS: Replace A_WHOIS_SERVER_IP with the generic A_SERVER_IP one
2021-06-25 14:39:26 +09:00
Alex Osborne
6b42c608a5
BaseWARCRecordBuilder: remove now unused getHostAddress(curi) helper
...
We now save the server IP at fetch time and retrieve it with
CrawlURI.getServerIP()
2021-06-25 14:39:26 +09:00
Alex Osborne
ef3b428061
FTP: Save server IP at fetch time and use in WARC-IP-Address
2021-06-25 14:39:26 +09:00
Alex Osborne
13c0f65294
FTP: Implement a basic fetch test
2021-06-25 14:09:38 +09:00
Alex Osborne
28ea9c4ba5
HTTP: Populate WARC-IP-Address with server's IP address
...
Fixes #396
2021-06-25 13:34:40 +09:00
Alex Osborne
5a6531b296
CrawlURI: Introduce a protocol-agnostic server-ip attribute
...
The vast majority of protocols we use will have a server IP address,
so rather than having a separate one for each protocol let's just use
a single common one.
2021-06-25 13:24:20 +09:00
Andrew Jackson
0e93e22ba9
[maven-release-plugin] prepare for next development iteration
2021-06-21 15:42:52 +01:00
Andrew Jackson
846f2bc2cc
[maven-release-plugin] prepare release 3.4.0-20210621
2021-06-21 15:42:46 +01:00
Alex Osborne
5d0a5b2ec4
Remove dependency on mg4j
...
This eliminates one of the jars that Heritrix depends on that isn't in
Maven Central.
There were only two usages. The first usage was a method in a
CrawlerJournal which as far as I can tell is unused and can be removed.
The second usage was an unnecessary use of MutableString as a line
buffer when saving cookies to a file. Since there doesn't seem any
particular reason buffer per-line here let's just use a BufferedWriter
instead on the whole file which is also slightly simpler and more
efficient.
2021-06-20 23:30:41 +09:00
Kristinn Sigurdsson
627befe07e
Remove unnecessary fiddling with VIA path
2021-06-18 08:42:05 +00:00
Andrew Jackson
3423cab6a6
[maven-release-plugin] prepare for next development iteration
2021-06-17 14:39:00 +01:00
Andrew Jackson
7368ded9c2
[maven-release-plugin] prepare release 3.4.0-20210617
2021-06-17 14:38:54 +01:00
Andy Jackson and GitHub
4b82d6060c
Merge pull request #398 from kris-sigur/sitemap-tweak
...
Annotate nested sitemap links as sitemaps
2021-06-17 10:10:58 +01:00
Kristinn Sigurdsson
81bef22270
Mark nested sitemap links as sitemaps
2021-06-16 13:07:18 +00:00
Andrew Jackson
d3dfb34f83
[maven-release-plugin] prepare for next development iteration
2021-05-27 10:03:10 +01:00
Andrew Jackson
9dcbb240a6
[maven-release-plugin] prepare release 3.4.0-20210527
2021-05-27 10:03:04 +01:00
Andy Jackson
a2c65f4618
Update dependencies for security issues, including changes to tests.
2021-05-26 13:39:30 +01:00
Andy Jackson and GitHub
c0dbfb2a0a
Merge pull request #383 from ukwa/upgrade-dnsjava
...
Update to latest version of dnsjava, for #344
2021-05-20 21:05:59 +01:00
Andy Jackson and GitHub
396467c535
Merge branch 'master' into sitemaps
2021-05-20 21:03:54 +01:00
Andy Jackson and GitHub
c1bcdd9f9e
Merge pull request #341 from internetarchive/noplaylist-ydl
...
youtube-dl --no-playlist
2021-05-20 21:01:39 +01:00
Andy Jackson and GitHub
b2a07b290f
Merge pull request #348 from internetarchive/fixes-leaky-file-handles
...
Fixes leaky file handles
2021-05-20 21:00:31 +01:00
Andy Jackson
045b2516da
Update to latest version of dnsjava, for #344
...
This commit updates to version 3.3.1 of the dnsjava library.
Incidentally, the restlet.org certificate is currently invalid, so this
patch switched to maven.restlet.com.
2021-05-20 20:44:19 +01:00
Kristinn Sigurdsson
52093eb25f
Only copy source tag if not null.
2021-04-20 13:18:29 +00:00
Neil Minton
2505effadc
Strip quotes from URL value.
2020-09-01 10:57:58 -04:00
Adam Miller
52a8f345ed
Fixing up logging and comments
2020-07-30 23:30:08 +00:00
Adam Miller
00d1c46d60
Ensure Replay Input Stream and File Channels are closed after writing
2020-07-30 23:19:43 +00:00
Alex Osborne and GitHub
947507b8ec
Revert "Warc convention for storing ftp responses has been to use a WARC reso…"
2020-06-01 13:56:12 +09:00
Alex Osborne and GitHub
4763a8a04b
Merge pull request #334 from internetarchive/fixes-ftp-response-record-builder
...
Warc convention for storing ftp responses has been to use a WARC reso…
2020-06-01 13:54:59 +09:00
Adam Miller
60daef90b2
Merge branch 'master' into fixes-extractor-multiple-regex-matcher-recycle
2020-05-20 16:46:14 +00:00
Adam Miller
c0fe2064a1
Merge branch 'fixes-ftp-response-record-builder'
2020-05-20 16:32:36 +00:00
Andrew Jackson
e215b89963
[maven-release-plugin] prepare for next development iteration
2020-05-18 14:08:12 +01:00
Andrew Jackson
ef38dc08c3
[maven-release-plugin] prepare release 3.4.0-20200518
2020-05-18 14:08:04 +01:00
Adam Miller
0db661fb36
Warc convention for storing ftp responses has been to use a WARC resource record and not a response record. Changing to be consisten with the default WARCWriterProcessor
2020-05-05 23:09:53 +00:00
morokosi
13ba0b2b67
optimize imports
2020-04-30 22:40:00 +09:00
morokosi
18bee39a4c
fix discarded future value
2020-04-30 22:39:51 +09:00
Alex Osborne and GitHub
7764f9265f
Merge pull request #320 from bnfleb/sftp
...
Add support for the SFTP protocol
2020-04-24 08:13:39 +09:00
Leslie Bellony
aad54c3cb3
rename variable + javadoc
2020-04-23 15:43:30 +02:00
Clara Wiatrowski
5e948883fa
Add overrides in JerichoExtractorHTMLTest
2020-04-22 15:33:23 +02:00
Clara Wiatrowski
0d1f4ea84f
Add parsing case data-*
2020-04-22 11:48:05 +02:00
Clara Wiatrowski
cc29bb97d1
Revert ExtractorHTML
2020-04-22 11:33:24 +02:00
Clara Wiatrowski
6710c8dba3
Add parsing for HTML tags (data-src, data-srcset, data-original and data-original-set)
2020-04-10 15:59:50 +02:00
Leslie Bellony
40725372d6
Add support for the SFTP protocol
2020-04-01 14:47:34 +02:00
Adam Miller
a384dea2f9
Recycle the Matcher after use.
2020-03-25 22:33:06 +00:00
Andrew Jackson
ff05de44b6
[maven-release-plugin] prepare for next development iteration
2020-03-04 23:05:41 +00:00
Andrew Jackson
a1bdcb1be7
[maven-release-plugin] prepare release 3.4.0-20200304
2020-03-04 23:05:33 +00:00
Andy Jackson and GitHub
7f80b4fd98
Merge pull request #281 from ukwa/upgrade-bdb-je
...
WIP: Upgrade BDB JE to version 7.5.11
2020-03-04 13:36:03 +00:00
Alex Osborne and GitHub
977f608c74
Merge pull request #290 from netarchivesuite/crawltrap-regex-timeout
...
Crawltrap regex timeout
2020-03-03 19:27:42 +09:00
