Adam Miller
85052d6cb2
Merge branch 'fixes-WARCWriterChainProcessor-checkpoint-stats' into ait-qa
2020-10-09 22:14:23 +00:00
Adam Miller
5504d492a1
Add checkpoint store/load functionality for warc writer stats
2020-10-09 22:14:08 +00:00
Neil Minton
13a1028e2f
Merge branch 'absolute-meta-urls' into ait-qa
2020-08-14 10:59:17 -04:00
Neil Minton
941c6c50ad
Strip quotes from URL value.
2020-08-14 10:54:18 -04:00
Neil Minton
6b1e88934a
Merge branch 'absolute-meta-urls' into ait-qa
2020-08-12 13:16:30 -04:00
Neil Minton
5cefd18357
Strip quotes from URL value.
2020-08-12 13:16:14 -04:00
Neil Minton
09884d1f57
Merge branch 'absolute-meta-urls' into ait-qa
2020-08-11 14:02:59 -04:00
Neil Minton
36f7aea625
Enable parsing of absolute URLs in meta HTML tags.
...
- Refactors several utility methods in UriUtils for URI parsing.
- Tests URLs in meta tags for absolute URIs.
The current W3C spec indicates those links should be relative. Several
examples have been observed in the wild with absolute URLs.
Browsers do not throw a parsing error and do redirect approprately. This
change conforms to that behavior.
Reference:
https://www.w3.org/TR/html53/document-metadata.html#statedef-http-equiv-refresh
2020-08-11 14:00:21 -04:00
Neil Minton
5b014e7971
Merge branch 'absolute-meta-urls' into ait-qa
2020-08-06 19:38:11 -04:00
Neil Minton
0dd65dfd0b
Enable parsing of absolute URLs in meta HTML tags.
...
- Refactors several utility methods in UriUtils for URI parsing.
- Tests URLs in meta tags for absolute URIs.
The current W3C spec indicates those links should be relative. Several
examples have been observed in the wild with absolute URLs.
Browsers do not throw a parsing error and do redirect approprately. This
change conforms to that behavior.
Reference:
https://www.w3.org/TR/html53/document-metadata.html#statedef-http-equiv-refresh
2020-08-06 19:23:50 -04:00
Adam Miller
f879226c8e
Merge pull request #342 from internetarchive/noplaylist-ydl
...
youtube-dl --no-playlist
2020-08-03 16:40:54 -07:00
jkafader
34bb996e5c
Merge pull request #346 from internetarchive/fixes-leaky-file-handles
...
Fixes leaky file handles
2020-08-03 16:22:23 -07:00
Adam Miller
46100d65b2
Merge pull request #347 from internetarchive/fixes-leaky-file-handles
...
Manage youtube dl temp files which can be closed in the warc writer.
2020-07-31 15:27:09 -07:00
Adam Miller
a9c0c6588b
Manage youtube dl temp files which can be closed in the warc writer.
2020-07-31 22:22:25 +00:00
jkafader
ab27ff93e9
Merge pull request #345 from internetarchive/fixes-leaky-file-handles
...
Ensure Replay Input Stream and File Channels are closed after writing
2020-07-30 16:37:03 -07:00
Adam Miller
52a8f345ed
Fixing up logging and comments
2020-07-30 23:30:08 +00:00
Adam Miller
00d1c46d60
Ensure Replay Input Stream and File Channels are closed after writing
2020-07-30 23:19:43 +00:00
Barbara Miller
1a50ff538b
Merge branch 'noplaylist-ydl' into ait-qa
2020-07-29 11:15:02 -07:00
Barbara Miller
1b8e7f73ca
youtube-dl --no-playlist
2020-07-29 11:12:33 -07:00
Alex Osborne
812e96d2c0
Merge pull request #336 from internetarchive/revert-334-fixes-ftp-response-record-builder
...
Revert "Warc convention for storing ftp responses has been to use a WARC reso…"
2020-06-01 13:56:37 +09:00
Alex Osborne
947507b8ec
Revert "Warc convention for storing ftp responses has been to use a WARC reso…"
2020-06-01 13:56:12 +09:00
Alex Osborne
4763a8a04b
Merge pull request #334 from internetarchive/fixes-ftp-response-record-builder
...
Warc convention for storing ftp responses has been to use a WARC reso…
2020-06-01 13:54:59 +09:00
Barbara Miller
f34684d7c6
Merge branch 'ait-qa' of github.com:internetarchive/heritrix3 into ait-qa
2020-05-29 18:00:14 -07:00
Barbara Miller
aea704c9b7
Merge branch 'ffmpeg' into ait-qa
2020-05-29 17:59:47 -07:00
Barbara Miller
c0f57a5f54
use ffmpeg for segmented downloads
2020-05-29 17:56:56 -07:00
Neil Minton
f9c4057074
Merge pull request #331 from internetarchive/fixes-ftp-response-record-builder
...
Warc convention for storing ftp responses has been to use a WARC reso…
2020-05-06 12:03:34 -04:00
Adam Miller
0db661fb36
Warc convention for storing ftp responses has been to use a WARC resource record and not a response record. Changing to be consisten with the default WARCWriterProcessor
2020-05-05 23:09:53 +00:00
jkafader
ce409c03c1
Merge pull request #329 from internetarchive/ydl-no-cache
...
youtube-dl no cache dir
2020-04-30 15:03:23 -07:00
jkafader
0a5d46d927
Merge pull request #330 from internetarchive/less_youtubedl
...
don't youtubedl receivedFromAMQP
2020-04-30 14:31:18 -07:00
Barbara Miller
3a8236a836
Merge branch 'less_youtubedl' into ait-qa
2020-04-30 11:53:01 -07:00
Barbara Miller
aabd16b41e
don't youtube-dl receivedFromAMQP
2020-04-30 11:06:13 -07:00
Barbara Miller
ba9bdcea7d
Merge branch 'less_youtubedl' into ait-qa
2020-04-29 13:34:37 -07:00
Barbara Miller
5cbeb1d5dd
try skipping all receivedFromAMQP
2020-04-29 13:33:38 -07:00
Barbara Miller
147b56dd8f
Merge branch 'ydl-no-cache' into ait-qa
2020-04-27 18:16:28 -07:00
Barbara Miller
633a7cffea
no youtube-dl cache dir
2020-04-27 18:15:54 -07:00
Barbara Miller
197c7f900b
Merge branch 'less_youtubedl' into ait-qa
2020-04-24 17:45:52 -07:00
Barbara Miller
1ffe0bc8c6
NPEs begone!
2020-04-24 17:45:34 -07:00
Barbara Miller
b39b68ae57
Merge branch 'less_youtubedl' into ait-qa
2020-04-24 16:48:54 -07:00
Barbara Miller
9c9da328d5
logging
2020-04-24 16:47:13 -07:00
Barbara Miller
5aa395333d
Merge branch 'less_youtubedl' into ait-qa
2020-04-23 17:53:20 -07:00
Barbara Miller
cf57532dbd
try again
2020-04-23 17:53:05 -07:00
Barbara Miller
479d3b8e5e
Merge branch 'less_youtubedl' into ait-qa
2020-04-23 16:46:12 -07:00
Barbara Miller
5849b2ecf5
fix typos
2020-04-23 16:45:55 -07:00
Barbara Miller
c946cdba80
Merge branch 'less_youtubedl' into ait-qa
2020-04-23 16:28:55 -07:00
Barbara Miller
0bec2ca59a
don't youtube-dl seeds twice
2020-04-23 16:28:21 -07:00
Neil Minton
0f8fbaf8f2
Merge pull request #327 from internetarchive/yt-dl-format-medium-ish
...
best medium-ish size
2020-04-20 18:20:00 -04:00
Barbara Miller
4999843ade
best medium-ish size
2020-04-20 14:48:51 -07:00
jkafader
ea374b3a99
Merge pull request #317 from internetarchive/fixes-extractor-multiple-regex-matcher-recycle
...
Recycle the regex Matcher after use.
2020-04-06 16:04:54 -07:00
Barbara Miller
1ecf69dd64
best medium-ish size
2020-04-03 16:25:10 -07:00
Adam Miller
a384dea2f9
Recycle the Matcher after use.
2020-03-25 22:33:06 +00:00
Adam Miller