lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-27 17:06:33 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,221 Commits 38 Branches 32 Tags
09884d1f57f0eececaeb340f49b9296499f7fc73
Commit Graph
2221 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Neil Minton 09884d1f57 Merge branch 'absolute-meta-urls' into ait-qa 2020-08-11 14:02:59 -04:00
Neil Minton 36f7aea625 Enable parsing of absolute URLs in meta HTML tags. 
- Refactors several utility methods in UriUtils for URI parsing.
- Tests URLs in meta tags for absolute URIs.

The current W3C spec indicates those links should be relative. Several
examples have been observed in the wild with absolute URLs.

Browsers do not throw a parsing error and do redirect approprately. This
change conforms to that behavior.

Reference:
https://www.w3.org/TR/html53/document-metadata.html#statedef-http-equiv-refresh
 2020-08-11 14:00:21 -04:00
Neil Minton 5b014e7971 Merge branch 'absolute-meta-urls' into ait-qa 2020-08-06 19:38:11 -04:00
Neil Minton 0dd65dfd0b Enable parsing of absolute URLs in meta HTML tags. 
- Refactors several utility methods in UriUtils for URI parsing.
- Tests URLs in meta tags for absolute URIs.

The current W3C spec indicates those links should be relative. Several
examples have been observed in the wild with absolute URLs.

Browsers do not throw a parsing error and do redirect approprately. This
change conforms to that behavior.

Reference:
https://www.w3.org/TR/html53/document-metadata.html#statedef-http-equiv-refresh
 2020-08-06 19:23:50 -04:00
Adam Miller f879226c8e Merge pull request #342 from internetarchive/noplaylist-ydl 
youtube-dl --no-playlist
 2020-08-03 16:40:54 -07:00
jkafader 34bb996e5c Merge pull request #346 from internetarchive/fixes-leaky-file-handles 
Fixes leaky file handles
 2020-08-03 16:22:23 -07:00
Adam Miller 46100d65b2 Merge pull request #347 from internetarchive/fixes-leaky-file-handles 
Manage youtube dl temp files which can be closed in the warc writer.
 2020-07-31 15:27:09 -07:00
Adam Miller a9c0c6588b Manage youtube dl temp files which can be closed in the warc writer. 2020-07-31 22:22:25 +00:00
jkafader ab27ff93e9 Merge pull request #345 from internetarchive/fixes-leaky-file-handles 
Ensure Replay Input Stream and File Channels are closed after writing
 2020-07-30 16:37:03 -07:00
Adam Miller 52a8f345ed Fixing up logging and comments 2020-07-30 23:30:08 +00:00
Adam Miller 00d1c46d60 Ensure Replay Input Stream and File Channels are closed after writing 2020-07-30 23:19:43 +00:00
Barbara Miller 1a50ff538b Merge branch 'noplaylist-ydl' into ait-qa 2020-07-29 11:15:02 -07:00
Barbara Miller 1b8e7f73ca youtube-dl --no-playlist 2020-07-29 11:12:33 -07:00
Alex Osborne 812e96d2c0 Merge pull request #336 from internetarchive/revert-334-fixes-ftp-response-record-builder 
Revert "Warc convention for storing ftp responses has been to use a WARC reso…"
 2020-06-01 13:56:37 +09:00
Alex Osborne 947507b8ec Revert "Warc convention for storing ftp responses has been to use a WARC reso…" 2020-06-01 13:56:12 +09:00
Alex Osborne 4763a8a04b Merge pull request #334 from internetarchive/fixes-ftp-response-record-builder 
Warc convention for storing ftp responses has been to use a WARC reso…
 2020-06-01 13:54:59 +09:00
Barbara Miller f34684d7c6 Merge branch 'ait-qa' of github.com:internetarchive/heritrix3 into ait-qa 2020-05-29 18:00:14 -07:00
Barbara Miller aea704c9b7 Merge branch 'ffmpeg' into ait-qa 2020-05-29 17:59:47 -07:00
Barbara Miller c0f57a5f54 use ffmpeg for segmented downloads 2020-05-29 17:56:56 -07:00
Neil Minton f9c4057074 Merge pull request #331 from internetarchive/fixes-ftp-response-record-builder 
Warc convention for storing ftp responses has been to use a WARC reso…
 2020-05-06 12:03:34 -04:00
Adam Miller 0db661fb36 Warc convention for storing ftp responses has been to use a WARC resource record and not a response record. Changing to be consisten with the default WARCWriterProcessor 2020-05-05 23:09:53 +00:00
jkafader ce409c03c1 Merge pull request #329 from internetarchive/ydl-no-cache 
youtube-dl no cache dir
 2020-04-30 15:03:23 -07:00
jkafader 0a5d46d927 Merge pull request #330 from internetarchive/less_youtubedl 
don't youtubedl receivedFromAMQP
 2020-04-30 14:31:18 -07:00
Barbara Miller 3a8236a836 Merge branch 'less_youtubedl' into ait-qa 2020-04-30 11:53:01 -07:00
Barbara Miller aabd16b41e don't youtube-dl receivedFromAMQP 2020-04-30 11:06:13 -07:00
Barbara Miller ba9bdcea7d Merge branch 'less_youtubedl' into ait-qa 2020-04-29 13:34:37 -07:00
Barbara Miller 5cbeb1d5dd try skipping all receivedFromAMQP 2020-04-29 13:33:38 -07:00
Barbara Miller 147b56dd8f Merge branch 'ydl-no-cache' into ait-qa 2020-04-27 18:16:28 -07:00
Barbara Miller 633a7cffea no youtube-dl cache dir 2020-04-27 18:15:54 -07:00
Barbara Miller 197c7f900b Merge branch 'less_youtubedl' into ait-qa 2020-04-24 17:45:52 -07:00
Barbara Miller 1ffe0bc8c6 NPEs begone! 2020-04-24 17:45:34 -07:00
Barbara Miller b39b68ae57 Merge branch 'less_youtubedl' into ait-qa 2020-04-24 16:48:54 -07:00
Barbara Miller 9c9da328d5 logging 2020-04-24 16:47:13 -07:00
Barbara Miller 5aa395333d Merge branch 'less_youtubedl' into ait-qa 2020-04-23 17:53:20 -07:00
Barbara Miller cf57532dbd try again 2020-04-23 17:53:05 -07:00
Barbara Miller 479d3b8e5e Merge branch 'less_youtubedl' into ait-qa 2020-04-23 16:46:12 -07:00
Barbara Miller 5849b2ecf5 fix typos 2020-04-23 16:45:55 -07:00
Barbara Miller c946cdba80 Merge branch 'less_youtubedl' into ait-qa 2020-04-23 16:28:55 -07:00
Barbara Miller 0bec2ca59a don't youtube-dl seeds twice 2020-04-23 16:28:21 -07:00
Neil Minton 0f8fbaf8f2 Merge pull request #327 from internetarchive/yt-dl-format-medium-ish 
best medium-ish size
 2020-04-20 18:20:00 -04:00
Barbara Miller 4999843ade best medium-ish size 2020-04-20 14:48:51 -07:00
jkafader ea374b3a99 Merge pull request #317 from internetarchive/fixes-extractor-multiple-regex-matcher-recycle 
Recycle the regex Matcher after use.
 2020-04-06 16:04:54 -07:00
Barbara Miller 1ecf69dd64 best medium-ish size 2020-04-03 16:25:10 -07:00
Adam Miller a384dea2f9 Recycle the Matcher after use. 2020-03-25 22:33:06 +00:00
Neil Minton 009458f2db Merge branch 'master-ait-contrib' into ait-qa 2020-02-13 15:40:58 -05:00
Neil Minton 97545664b2 Remove Kafka support. 2020-02-13 15:38:38 -05:00
Neil Minton f5a49e70b2 Remove Hbase support. 2020-02-13 15:38:21 -05:00
Alex Osborne 015c5e7f80 Merge pull request #308 from ldko/fix-restlet-errors 
Fix stream closed exception for Paged view
 2020-01-31 09:33:26 +09:00
Lauren Ko 8c1c8009c6 Fix stream closed exception for Paged view 2020-01-30 16:48:38 -06:00
Alex Osborne b45f774066 Merge pull request #300 from hennekey/fix-299 
Use Guice instead of custom implementation
 2020-01-30 19:12:05 +09:00