Andrew Jackson
ddb695abad
[maven-release-plugin] prepare for next development iteration
2021-08-03 09:56:26 +01:00
Andrew Jackson
ba44cce7f2
[maven-release-plugin] prepare release 3.4.0-20210803
2021-08-03 09:56:20 +01:00
Alex Osborne
eb67cfb215
Merge pull request #423 from internetarchive/dont-extract-data-uris
...
Don't extract data URIs
2021-08-03 07:43:17 +09:00
Andy Jackson
4d5b2dc3fe
Merge pull request #397 from ukwa/upgrade-httpclient-4-5
...
Upgrade httpclient to 4.5
2021-08-02 09:19:30 +01:00
Alex Osborne
33a5d70209
ExtractorSitemap: Use logUriError() helper like other extractors
2021-07-30 15:57:09 +09:00
Alex Osborne
1032c0ec40
ExtractorHTML: Avoid allocating strings for data: URIs when possible
...
Data URIs can be very large. ExtractorHTML mostly works with off-heap
CharSequences so by delaying the conversion of outlinks to strings
until after filtering out data URIs we can potentially avoid some
very large String allocations.
2021-07-30 15:57:09 +09:00
Alex Osborne
e33e1ff398
Extractor: ignore data URIs when adding outlinks
2021-07-30 15:57:09 +09:00
Alex Osborne
91ef23eddf
ExtractorPDFContext, ExtractorYoutubeDL: use addOutlink() helper method
2021-07-30 14:25:34 +09:00
Alex Osborne
f4bd5d1015
Upgrade Groovy to latest stable version (3.0.8) for JDK 16 compatibility
...
Fixes #419
2021-07-25 14:11:13 +09:00
Alex Osborne
09e3ff3568
Everywhere: Fix dangling JavaDoc comments
...
JavaDoc comments need to be directly above a class, method or field
declaration to be used by IDEs and generated documentation. We had some
doc comments that were attached to instance initialization blocks and
thus were being ignored. This invalid doc comment positioning was
presumably an accidental consequence of the conversion of fields to
KeyedProperties.
This change moves most of the dangling doc comments to setter methods
where it can be seen by tools. There were a couple of dangling doc
comments above package statements. These are moved above the class
declaration or removed entirely when empty.
2021-07-12 21:38:48 +09:00
Andrew Jackson
0640cae4a9
Enabled AuthCache but not for DigestScheme.
2021-07-05 21:29:16 +01:00
Adam Miller
ca3397e346
Fixing race condition on totalBytesWritten
2021-07-03 12:10:21 +09:00
Adam Miller
111b77b4aa
Add checkpoint store/load functionality for warc writer stats
2021-07-03 12:10:21 +09:00
Alex Osborne
200f1cf38f
DNS: Use the new protocol-agnostic server IP attribute
...
Also fixes an issue where in some cases we would incorrectly output
WARC-IP-Address for DNS records as "/127.0.0.53:53" instead of
"127.0.0.53".
2021-06-25 16:39:59 +09:00
Alex Osborne
c378c46e49
WHOIS: Replace A_WHOIS_SERVER_IP with the generic A_SERVER_IP one
2021-06-25 14:39:26 +09:00
Alex Osborne
6b42c608a5
BaseWARCRecordBuilder: remove now unused getHostAddress(curi) helper
...
We now save the server IP at fetch time and retrieve it with
CrawlURI.getServerIP()
2021-06-25 14:39:26 +09:00
Alex Osborne
ef3b428061
FTP: Save server IP at fetch time and use in WARC-IP-Address
2021-06-25 14:39:26 +09:00
Alex Osborne
13c0f65294
FTP: Implement a basic fetch test
2021-06-25 14:09:38 +09:00
Alex Osborne
28ea9c4ba5
HTTP: Populate WARC-IP-Address with server's IP address
...
Fixes #396
2021-06-25 13:34:40 +09:00
Alex Osborne
5a6531b296
CrawlURI: Introduce a protocol-agnostic server-ip attribute
...
The vast majority of protocols we use will have a server IP address,
so rather than having a separate one for each protocol let's just use
a single common one.
2021-06-25 13:24:20 +09:00
Andrew Jackson
0e93e22ba9
[maven-release-plugin] prepare for next development iteration
2021-06-21 15:42:52 +01:00
Andrew Jackson
846f2bc2cc
[maven-release-plugin] prepare release 3.4.0-20210621
2021-06-21 15:42:46 +01:00
Andrew Jackson
d372aacc33
Fix merge error.
2021-06-21 13:50:58 +01:00
Andrew Jackson
1e144e23dc
Merge branch 'upgrade-httpclient-4-5' of github.com:ukwa/heritrix3 into upgrade-httpclient-4-5
2021-06-21 13:49:43 +01:00
Andrew Jackson
595c8d1293
Resolve issues raised during review by @ato.
2021-06-21 13:49:21 +01:00
Andy Jackson and Alex Osborne
bd6b4f1e0f
Switch to StandardCharsets to avoid try/catch.
...
Co-authored-by: Alex Osborne <ato@meshy.org >
2021-06-21 13:42:13 +01:00
Andrew Jackson
df4a328048
Fix imports.
2021-06-20 23:16:42 +01:00
Andy Jackson
a9002e1e3b
Merge branch 'master' into upgrade-httpclient-4-5
2021-06-20 23:09:59 +01:00
Alex Osborne
5d0a5b2ec4
Remove dependency on mg4j
...
This eliminates one of the jars that Heritrix depends on that isn't in
Maven Central.
There were only two usages. The first usage was a method in a
CrawlerJournal which as far as I can tell is unused and can be removed.
The second usage was an unnecessary use of MutableString as a line
buffer when saving cookies to a file. Since there doesn't seem any
particular reason buffer per-line here let's just use a BufferedWriter
instead on the whole file which is also slightly simpler and more
efficient.
2021-06-20 23:30:41 +09:00
Andrew Jackson
e3b2eea12d
Merge branch 'master' into upgrade-httpclient-4-5
2021-06-18 10:22:31 +01:00
Kristinn Sigurdsson
627befe07e
Remove unnecessary fiddling with VIA path
2021-06-18 08:42:05 +00:00
Andrew Jackson
3423cab6a6
[maven-release-plugin] prepare for next development iteration
2021-06-17 14:39:00 +01:00
Andrew Jackson
7368ded9c2
[maven-release-plugin] prepare release 3.4.0-20210617
2021-06-17 14:38:54 +01:00
Andrew Jackson
998eaa3ed8
Merge branch 'master' into upgrade-httpclient-4-5
2021-06-17 14:32:43 +01:00
Andrew Jackson
e07b1bb938
Support cookie expiration and extend testing to cover that case.
2021-06-17 14:23:42 +01:00
Andrew Jackson
2d64e005cb
Switch to standard cookie handling.
2021-06-17 14:23:20 +01:00
Andrew Jackson
1d1d2f5abe
Avoid two slf4j implementations.
2021-06-17 14:19:12 +01:00
Andrew Jackson
4cd416a5e4
Update complier plugin and use compatible flip().
2021-06-17 14:01:04 +01:00
Andy Jackson
4b82d6060c
Merge pull request #398 from kris-sigur/sitemap-tweak
...
Annotate nested sitemap links as sitemaps
2021-06-17 10:10:58 +01:00
Kristinn Sigurdsson
81bef22270
Mark nested sitemap links as sitemaps
2021-06-16 13:07:18 +00:00
Andrew Jackson
d892026731
Expire cookies when requested.
2021-06-15 00:31:04 +01:00
Andrew Jackson
81fa60d870
Upgrade HTTPClient to 4.5.x for #245
2021-06-14 23:35:46 +01:00
Andrew Jackson
d3dfb34f83
[maven-release-plugin] prepare for next development iteration
2021-05-27 10:03:10 +01:00
Andrew Jackson
9dcbb240a6
[maven-release-plugin] prepare release 3.4.0-20210527
2021-05-27 10:03:04 +01:00
Andy Jackson
a2c65f4618
Update dependencies for security issues, including changes to tests.
2021-05-26 13:39:30 +01:00
Andy Jackson
c0dbfb2a0a
Merge pull request #383 from ukwa/upgrade-dnsjava
...
Update to latest version of dnsjava, for #344
2021-05-20 21:05:59 +01:00
Andy Jackson
396467c535
Merge branch 'master' into sitemaps
2021-05-20 21:03:54 +01:00
Andy Jackson
c1bcdd9f9e
Merge pull request #341 from internetarchive/noplaylist-ydl
...
youtube-dl --no-playlist
2021-05-20 21:01:39 +01:00
Andy Jackson
b2a07b290f
Merge pull request #348 from internetarchive/fixes-leaky-file-handles
...
Fixes leaky file handles
2021-05-20 21:00:31 +01:00
Andy Jackson
045b2516da
Update to latest version of dnsjava, for #344
...
This commit updates to version 3.3.1 of the dnsjava library.
Incidentally, the restlet.org certificate is currently invalid, so this
patch switched to maven.restlet.com.
2021-05-20 20:44:19 +01:00
Andrew Jackson