Andy Jackson
3f4535a933
Add safer cookie iteration, as per #427 .
2021-09-06 11:05:12 +01:00
Alex Osborne
eb67cfb215
Merge pull request #423 from internetarchive/dont-extract-data-uris
...
Don't extract data URIs
2021-08-03 07:43:17 +09:00
Andy Jackson
4d5b2dc3fe
Merge pull request #397 from ukwa/upgrade-httpclient-4-5
...
Upgrade httpclient to 4.5
2021-08-02 09:19:30 +01:00
Alex Osborne
33a5d70209
ExtractorSitemap: Use logUriError() helper like other extractors
2021-07-30 15:57:09 +09:00
Alex Osborne
1032c0ec40
ExtractorHTML: Avoid allocating strings for data: URIs when possible
...
Data URIs can be very large. ExtractorHTML mostly works with off-heap
CharSequences so by delaying the conversion of outlinks to strings
until after filtering out data URIs we can potentially avoid some
very large String allocations.
2021-07-30 15:57:09 +09:00
Alex Osborne
e33e1ff398
Extractor: ignore data URIs when adding outlinks
2021-07-30 15:57:09 +09:00
Alex Osborne
91ef23eddf
ExtractorPDFContext, ExtractorYoutubeDL: use addOutlink() helper method
2021-07-30 14:25:34 +09:00
Alex Osborne
09e3ff3568
Everywhere: Fix dangling JavaDoc comments
...
JavaDoc comments need to be directly above a class, method or field
declaration to be used by IDEs and generated documentation. We had some
doc comments that were attached to instance initialization blocks and
thus were being ignored. This invalid doc comment positioning was
presumably an accidental consequence of the conversion of fields to
KeyedProperties.
This change moves most of the dangling doc comments to setter methods
where it can be seen by tools. There were a couple of dangling doc
comments above package statements. These are moved above the class
declaration or removed entirely when empty.
2021-07-12 21:38:48 +09:00
Andrew Jackson
0640cae4a9
Enabled AuthCache but not for DigestScheme.
2021-07-05 21:29:16 +01:00
Adam Miller
ca3397e346
Fixing race condition on totalBytesWritten
2021-07-03 12:10:21 +09:00
Adam Miller
111b77b4aa
Add checkpoint store/load functionality for warc writer stats
2021-07-03 12:10:21 +09:00
Alex Osborne
200f1cf38f
DNS: Use the new protocol-agnostic server IP attribute
...
Also fixes an issue where in some cases we would incorrectly output
WARC-IP-Address for DNS records as "/127.0.0.53:53" instead of
"127.0.0.53".
2021-06-25 16:39:59 +09:00
Alex Osborne
c378c46e49
WHOIS: Replace A_WHOIS_SERVER_IP with the generic A_SERVER_IP one
2021-06-25 14:39:26 +09:00
Alex Osborne
6b42c608a5
BaseWARCRecordBuilder: remove now unused getHostAddress(curi) helper
...
We now save the server IP at fetch time and retrieve it with
CrawlURI.getServerIP()
2021-06-25 14:39:26 +09:00
Alex Osborne
ef3b428061
FTP: Save server IP at fetch time and use in WARC-IP-Address
2021-06-25 14:39:26 +09:00
Alex Osborne
13c0f65294
FTP: Implement a basic fetch test
2021-06-25 14:09:38 +09:00
Alex Osborne
28ea9c4ba5
HTTP: Populate WARC-IP-Address with server's IP address
...
Fixes #396
2021-06-25 13:34:40 +09:00
Alex Osborne
5a6531b296
CrawlURI: Introduce a protocol-agnostic server-ip attribute
...
The vast majority of protocols we use will have a server IP address,
so rather than having a separate one for each protocol let's just use
a single common one.
2021-06-25 13:24:20 +09:00
Andrew Jackson
d372aacc33
Fix merge error.
2021-06-21 13:50:58 +01:00
Andrew Jackson
1e144e23dc
Merge branch 'upgrade-httpclient-4-5' of github.com:ukwa/heritrix3 into upgrade-httpclient-4-5
2021-06-21 13:49:43 +01:00
Andrew Jackson
595c8d1293
Resolve issues raised during review by @ato.
2021-06-21 13:49:21 +01:00
Andy Jackson and Alex Osborne
bd6b4f1e0f
Switch to StandardCharsets to avoid try/catch.
...
Co-authored-by: Alex Osborne <ato@meshy.org >
2021-06-21 13:42:13 +01:00
Andrew Jackson
df4a328048
Fix imports.
2021-06-20 23:16:42 +01:00
Andy Jackson
a9002e1e3b
Merge branch 'master' into upgrade-httpclient-4-5
2021-06-20 23:09:59 +01:00
Alex Osborne
5d0a5b2ec4
Remove dependency on mg4j
...
This eliminates one of the jars that Heritrix depends on that isn't in
Maven Central.
There were only two usages. The first usage was a method in a
CrawlerJournal which as far as I can tell is unused and can be removed.
The second usage was an unnecessary use of MutableString as a line
buffer when saving cookies to a file. Since there doesn't seem any
particular reason buffer per-line here let's just use a BufferedWriter
instead on the whole file which is also slightly simpler and more
efficient.
2021-06-20 23:30:41 +09:00
Andrew Jackson
e3b2eea12d
Merge branch 'master' into upgrade-httpclient-4-5
2021-06-18 10:22:31 +01:00
Kristinn Sigurdsson
627befe07e
Remove unnecessary fiddling with VIA path
2021-06-18 08:42:05 +00:00
Andrew Jackson
998eaa3ed8
Merge branch 'master' into upgrade-httpclient-4-5
2021-06-17 14:32:43 +01:00
Andrew Jackson
e07b1bb938
Support cookie expiration and extend testing to cover that case.
2021-06-17 14:23:42 +01:00
Andrew Jackson
2d64e005cb
Switch to standard cookie handling.
2021-06-17 14:23:20 +01:00
Andrew Jackson
4cd416a5e4
Update complier plugin and use compatible flip().
2021-06-17 14:01:04 +01:00
Kristinn Sigurdsson
81bef22270
Mark nested sitemap links as sitemaps
2021-06-16 13:07:18 +00:00
Andrew Jackson
d892026731
Expire cookies when requested.
2021-06-15 00:31:04 +01:00
Andrew Jackson
81fa60d870
Upgrade HTTPClient to 4.5.x for #245
2021-06-14 23:35:46 +01:00
Andy Jackson
c0dbfb2a0a
Merge pull request #383 from ukwa/upgrade-dnsjava
...
Update to latest version of dnsjava, for #344
2021-05-20 21:05:59 +01:00
Andy Jackson
396467c535
Merge branch 'master' into sitemaps
2021-05-20 21:03:54 +01:00
Andy Jackson
c1bcdd9f9e
Merge pull request #341 from internetarchive/noplaylist-ydl
...
youtube-dl --no-playlist
2021-05-20 21:01:39 +01:00
Andy Jackson
b2a07b290f
Merge pull request #348 from internetarchive/fixes-leaky-file-handles
...
Fixes leaky file handles
2021-05-20 21:00:31 +01:00
Andy Jackson
045b2516da
Update to latest version of dnsjava, for #344
...
This commit updates to version 3.3.1 of the dnsjava library.
Incidentally, the restlet.org certificate is currently invalid, so this
patch switched to maven.restlet.com.
2021-05-20 20:44:19 +01:00
Kristinn Sigurdsson
52093eb25f
Only copy source tag if not null.
2021-04-20 13:18:29 +00:00
Neil Minton
2505effadc
Strip quotes from URL value.
2020-09-01 10:57:58 -04:00
Adam Miller
52a8f345ed
Fixing up logging and comments
2020-07-30 23:30:08 +00:00
Adam Miller
00d1c46d60
Ensure Replay Input Stream and File Channels are closed after writing
2020-07-30 23:19:43 +00:00
Alex Osborne
947507b8ec
Revert "Warc convention for storing ftp responses has been to use a WARC reso…"
2020-06-01 13:56:12 +09:00
Alex Osborne
4763a8a04b
Merge pull request #334 from internetarchive/fixes-ftp-response-record-builder
...
Warc convention for storing ftp responses has been to use a WARC reso…
2020-06-01 13:54:59 +09:00
Adam Miller
60daef90b2
Merge branch 'master' into fixes-extractor-multiple-regex-matcher-recycle
2020-05-20 16:46:14 +00:00
Adam Miller
c0fe2064a1
Merge branch 'fixes-ftp-response-record-builder'
2020-05-20 16:32:36 +00:00
Adam Miller
0db661fb36
Warc convention for storing ftp responses has been to use a WARC resource record and not a response record. Changing to be consisten with the default WARCWriterProcessor
2020-05-05 23:09:53 +00:00
morokosi
13ba0b2b67
optimize imports
2020-04-30 22:40:00 +09:00
morokosi
18bee39a4c
fix discarded future value
2020-04-30 22:39:51 +09:00
Andy Jackson