Adam Miller
d6eb68a265
Revert "Fix error log reporting of batch size for trough crawl logs"
2021-04-05 16:01:27 -07:00
jkafader
12eef6c70d
Merge pull request #367 from internetarchive/fixes-trough-crawl-log-feed-synchronization
...
Fix error log reporting of batch size for trough crawl logs
2021-02-26 15:41:03 -08:00
Adam Miller
00947b0167
Fixing sync issues on uncrawled urls batch as well
2021-02-25 22:31:25 +00:00
Adam Miller
79e56337b2
Fix error log reporting of batch size for trough crawl logs
2021-02-18 23:06:51 +00:00
Barbara Miller
6a8b754ba7
Merge branch 'log_sleep' into ait-qa
2021-02-01 21:24:54 -08:00
Barbara Miller
7e82722da3
log sleep
2021-02-01 21:24:24 -08:00
jkafader
5e7a20d19a
Merge pull request #364 from internetarchive/fixes-trough-crawl-log-feed-synchronization
...
Fixing synchronization issues in TroughCrawlLogFeed.java to reduce bl…
2021-01-28 15:31:53 -08:00
Adam Miller
14be8752e9
Fixing synchronization issues in TroughCrawlLogFeed.java to reduce blocking
2021-01-28 22:54:06 +00:00
Barbara Miller
147467c326
Merge branch 'fix_restlet_npe' into ait-qa
2021-01-12 18:50:16 -08:00
Barbara Miller
b5b204b40a
avoid NPE https://webarchive.jira.com/browse/WT-28?focusedCommentId=139095
2021-01-12 18:49:43 -08:00
Adam Miller
6ff10be094
Merge pull request #357 from internetarchive/fixes-prerequisite-status
...
Reset CrawlURI status for hasPrerequisite() so that it isn't preserve…
2020-11-12 15:46:16 -08:00
Adam Miller
6ebb28db18
Merge pull request #356 from internetarchive/fixes-WARCWriterChainProcessor-checkpoint-stats
...
Fixing race condition on totalBytesWritten
2020-11-12 15:20:14 -08:00
Adam Miller
1f357c8daf
Fixing race condition on totalBytesWritten
2020-11-11 00:02:09 +00:00
Adam Miller
e694a4b5c8
Reset CrawlURI status for hasPrerequisite() so that it isn't preserved between attempts
2020-11-07 01:12:56 +00:00
jkafader
288b167399
Merge pull request #354 from internetarchive/fixes-WARCWriterChainProcessor-checkpoint-stats
...
Add checkpoint store/load functionality for warc writer stats
2020-10-15 13:51:09 -07:00
Adam Miller
85052d6cb2
Merge branch 'fixes-WARCWriterChainProcessor-checkpoint-stats' into ait-qa
2020-10-09 22:14:23 +00:00
Adam Miller
5504d492a1
Add checkpoint store/load functionality for warc writer stats
2020-10-09 22:14:08 +00:00
jkafader
87ac0dacb2
Merge pull request #349 from BitBaron/absolute-meta-urls
...
Better parsing of absolute meta element URLs.
2020-08-17 12:38:02 -07:00
Neil Minton
13a1028e2f
Merge branch 'absolute-meta-urls' into ait-qa
2020-08-14 10:59:17 -04:00
Neil Minton
941c6c50ad
Strip quotes from URL value.
2020-08-14 10:54:18 -04:00
Neil Minton
6b1e88934a
Merge branch 'absolute-meta-urls' into ait-qa
2020-08-12 13:16:30 -04:00
Neil Minton
5cefd18357
Strip quotes from URL value.
2020-08-12 13:16:14 -04:00
Neil Minton
09884d1f57
Merge branch 'absolute-meta-urls' into ait-qa
2020-08-11 14:02:59 -04:00
Neil Minton
36f7aea625
Enable parsing of absolute URLs in meta HTML tags.
...
- Refactors several utility methods in UriUtils for URI parsing.
- Tests URLs in meta tags for absolute URIs.
The current W3C spec indicates those links should be relative. Several
examples have been observed in the wild with absolute URLs.
Browsers do not throw a parsing error and do redirect approprately. This
change conforms to that behavior.
Reference:
https://www.w3.org/TR/html53/document-metadata.html#statedef-http-equiv-refresh
2020-08-11 14:00:21 -04:00
Neil Minton
5b014e7971
Merge branch 'absolute-meta-urls' into ait-qa
2020-08-06 19:38:11 -04:00
Neil Minton
0dd65dfd0b
Enable parsing of absolute URLs in meta HTML tags.
...
- Refactors several utility methods in UriUtils for URI parsing.
- Tests URLs in meta tags for absolute URIs.
The current W3C spec indicates those links should be relative. Several
examples have been observed in the wild with absolute URLs.
Browsers do not throw a parsing error and do redirect approprately. This
change conforms to that behavior.
Reference:
https://www.w3.org/TR/html53/document-metadata.html#statedef-http-equiv-refresh
2020-08-06 19:23:50 -04:00
Adam Miller
f879226c8e
Merge pull request #342 from internetarchive/noplaylist-ydl
...
youtube-dl --no-playlist
2020-08-03 16:40:54 -07:00
jkafader
34bb996e5c
Merge pull request #346 from internetarchive/fixes-leaky-file-handles
...
Fixes leaky file handles
2020-08-03 16:22:23 -07:00
Adam Miller
46100d65b2
Merge pull request #347 from internetarchive/fixes-leaky-file-handles
...
Manage youtube dl temp files which can be closed in the warc writer.
2020-07-31 15:27:09 -07:00
Adam Miller
a9c0c6588b
Manage youtube dl temp files which can be closed in the warc writer.
2020-07-31 22:22:25 +00:00
jkafader
ab27ff93e9
Merge pull request #345 from internetarchive/fixes-leaky-file-handles
...
Ensure Replay Input Stream and File Channels are closed after writing
2020-07-30 16:37:03 -07:00
Adam Miller
52a8f345ed
Fixing up logging and comments
2020-07-30 23:30:08 +00:00
Adam Miller
00d1c46d60
Ensure Replay Input Stream and File Channels are closed after writing
2020-07-30 23:19:43 +00:00
Barbara Miller
1a50ff538b
Merge branch 'noplaylist-ydl' into ait-qa
2020-07-29 11:15:02 -07:00
Barbara Miller
1b8e7f73ca
youtube-dl --no-playlist
2020-07-29 11:12:33 -07:00
Alex Osborne
812e96d2c0
Merge pull request #336 from internetarchive/revert-334-fixes-ftp-response-record-builder
...
Revert "Warc convention for storing ftp responses has been to use a WARC reso…"
2020-06-01 13:56:37 +09:00
Alex Osborne
947507b8ec
Revert "Warc convention for storing ftp responses has been to use a WARC reso…"
2020-06-01 13:56:12 +09:00
Alex Osborne
4763a8a04b
Merge pull request #334 from internetarchive/fixes-ftp-response-record-builder
...
Warc convention for storing ftp responses has been to use a WARC reso…
2020-06-01 13:54:59 +09:00
Barbara Miller
f34684d7c6
Merge branch 'ait-qa' of github.com:internetarchive/heritrix3 into ait-qa
2020-05-29 18:00:14 -07:00
Barbara Miller
aea704c9b7
Merge branch 'ffmpeg' into ait-qa
2020-05-29 17:59:47 -07:00
Barbara Miller
c0f57a5f54
use ffmpeg for segmented downloads
2020-05-29 17:56:56 -07:00
Neil Minton
f9c4057074
Merge pull request #331 from internetarchive/fixes-ftp-response-record-builder
...
Warc convention for storing ftp responses has been to use a WARC reso…
2020-05-06 12:03:34 -04:00
Adam Miller
0db661fb36
Warc convention for storing ftp responses has been to use a WARC resource record and not a response record. Changing to be consisten with the default WARCWriterProcessor
2020-05-05 23:09:53 +00:00
jkafader
ce409c03c1
Merge pull request #329 from internetarchive/ydl-no-cache
...
youtube-dl no cache dir
2020-04-30 15:03:23 -07:00
jkafader
0a5d46d927
Merge pull request #330 from internetarchive/less_youtubedl
...
don't youtubedl receivedFromAMQP
2020-04-30 14:31:18 -07:00
Barbara Miller
3a8236a836
Merge branch 'less_youtubedl' into ait-qa
2020-04-30 11:53:01 -07:00
Barbara Miller
aabd16b41e
don't youtube-dl receivedFromAMQP
2020-04-30 11:06:13 -07:00
Barbara Miller
ba9bdcea7d
Merge branch 'less_youtubedl' into ait-qa
2020-04-29 13:34:37 -07:00
Barbara Miller
5cbeb1d5dd
try skipping all receivedFromAMQP
2020-04-29 13:33:38 -07:00
Barbara Miller
147b56dd8f
Merge branch 'ydl-no-cache' into ait-qa
2020-04-27 18:16:28 -07:00
Adam Miller