Adam Miller
fc402acd06
Merge branch 'ait-qa' into adds-trough-dedup-performance-rework
2021-05-13 10:39:02 -07:00
Adam Miller
18edb3e413
Refactor crawl log batch posting. Add configuration options for trough dedup batches. Fix posting of trough dedup at end of crawl.
2021-05-13 17:32:26 +00:00
Adam Miller
4625a78a2c
Merge pull request #374 from internetarchive/adds-trough-dedup-performance-rework
...
Adds trough dedup performance rework
2021-04-05 16:49:50 -07:00
Adam Miller
e353ccb097
Merge branch 'ait-qa' into adds-trough-dedup-performance-rework
2021-04-05 16:17:47 -07:00
Adam Miller
21e84ad015
Merge pull request #373 from internetarchive/revert-367-fixes-trough-crawl-log-feed-synchronization
...
Revert "Fix error log reporting of batch size for trough crawl logs"
2021-04-05 16:01:51 -07:00
Adam Miller
d6eb68a265
Revert "Fix error log reporting of batch size for trough crawl logs"
2021-04-05 16:01:27 -07:00
Adam Miller
0b09340e84
Fix inconsistency with cached batch sizes and refine logic for forcing final batch posting at end of crawl
2021-04-05 22:56:23 +00:00
Adam Miller
d72175635e
Ensure that the final batch is posted during shutdown. Make posting of uncrawled urls optional.
2021-04-05 21:06:56 +00:00
Adam Miller
466c6795b3
Merge branch 'fixes-trough-crawl-log-feed-synchronization' into adds-trough-dedup-performance-rework
2021-03-10 21:36:33 +00:00
Adam Miller
54abea3157
Merge branch 'adds-trough-local-dedup-write-cache' into adds-trough-dedup-performance-rework
2021-03-10 21:36:24 +00:00
Adam Miller
8724b14a9d
Replacing some synchronized lists with threadsafe objects and fixing discrepency between crawled and uncrawled lists
2021-03-10 21:21:59 +00:00
jkafader
12eef6c70d
Merge pull request #367 from internetarchive/fixes-trough-crawl-log-feed-synchronization
...
Fix error log reporting of batch size for trough crawl logs
2021-02-26 15:41:03 -08:00
Adam Miller
e700f384d5
Switching to in memory sqlite for dedup cache
2021-02-26 23:32:20 +00:00
Adam Miller
88b7eeaf02
On disk - per-seed sqlite dedup
2021-02-25 22:35:08 +00:00
Adam Miller
00947b0167
Fixing sync issues on uncrawled urls batch as well
2021-02-25 22:31:25 +00:00
Adam Miller
e30005128d
Merge branch 'adds-trough-local-dedup-write-cache' into adds-trough-dedup-performance-rework
2021-02-23 00:44:34 +00:00
Adam Miller
09485a5dcd
Merge branch 'fixes-trough-crawl-log-feed-synchronization' into adds-trough-dedup-performance-rework
2021-02-23 00:44:23 +00:00
Adam Miller
86e238e7f7
Create and insert into local sqlite dedup db shards
2021-02-23 00:43:26 +00:00
Adam Miller
79e56337b2
Fix error log reporting of batch size for trough crawl logs
2021-02-18 23:06:51 +00:00
Adam Miller
d22ef124d3
Trying to avoid hard locks on updating url caches
2021-02-02 23:58:48 +00:00
Barbara Miller
6a8b754ba7
Merge branch 'log_sleep' into ait-qa
2021-02-01 21:24:54 -08:00
Barbara Miller
7e82722da3
log sleep
2021-02-01 21:24:24 -08:00
jkafader
5e7a20d19a
Merge pull request #364 from internetarchive/fixes-trough-crawl-log-feed-synchronization
...
Fixing synchronization issues in TroughCrawlLogFeed.java to reduce bl…
2021-01-28 15:31:53 -08:00
Adam Miller
14be8752e9
Fixing synchronization issues in TroughCrawlLogFeed.java to reduce blocking
2021-01-28 22:54:06 +00:00
Barbara Miller
147467c326
Merge branch 'fix_restlet_npe' into ait-qa
2021-01-12 18:50:16 -08:00
Barbara Miller
b5b204b40a
avoid NPE https://webarchive.jira.com/browse/WT-28?focusedCommentId=139095
2021-01-12 18:49:43 -08:00
Adam Miller
6ff10be094
Merge pull request #357 from internetarchive/fixes-prerequisite-status
...
Reset CrawlURI status for hasPrerequisite() so that it isn't preserve…
2020-11-12 15:46:16 -08:00
Adam Miller
6ebb28db18
Merge pull request #356 from internetarchive/fixes-WARCWriterChainProcessor-checkpoint-stats
...
Fixing race condition on totalBytesWritten
2020-11-12 15:20:14 -08:00
Adam Miller
1f357c8daf
Fixing race condition on totalBytesWritten
2020-11-11 00:02:09 +00:00
Adam Miller
e694a4b5c8
Reset CrawlURI status for hasPrerequisite() so that it isn't preserved between attempts
2020-11-07 01:12:56 +00:00
jkafader
288b167399
Merge pull request #354 from internetarchive/fixes-WARCWriterChainProcessor-checkpoint-stats
...
Add checkpoint store/load functionality for warc writer stats
2020-10-15 13:51:09 -07:00
Adam Miller
85052d6cb2
Merge branch 'fixes-WARCWriterChainProcessor-checkpoint-stats' into ait-qa
2020-10-09 22:14:23 +00:00
Adam Miller
5504d492a1
Add checkpoint store/load functionality for warc writer stats
2020-10-09 22:14:08 +00:00
jkafader
87ac0dacb2
Merge pull request #349 from BitBaron/absolute-meta-urls
...
Better parsing of absolute meta element URLs.
2020-08-17 12:38:02 -07:00
Neil Minton
13a1028e2f
Merge branch 'absolute-meta-urls' into ait-qa
2020-08-14 10:59:17 -04:00
Neil Minton
941c6c50ad
Strip quotes from URL value.
2020-08-14 10:54:18 -04:00
Neil Minton
6b1e88934a
Merge branch 'absolute-meta-urls' into ait-qa
2020-08-12 13:16:30 -04:00
Neil Minton
5cefd18357
Strip quotes from URL value.
2020-08-12 13:16:14 -04:00
Neil Minton
09884d1f57
Merge branch 'absolute-meta-urls' into ait-qa
2020-08-11 14:02:59 -04:00
Neil Minton
36f7aea625
Enable parsing of absolute URLs in meta HTML tags.
...
- Refactors several utility methods in UriUtils for URI parsing.
- Tests URLs in meta tags for absolute URIs.
The current W3C spec indicates those links should be relative. Several
examples have been observed in the wild with absolute URLs.
Browsers do not throw a parsing error and do redirect approprately. This
change conforms to that behavior.
Reference:
https://www.w3.org/TR/html53/document-metadata.html#statedef-http-equiv-refresh
2020-08-11 14:00:21 -04:00
Neil Minton
5b014e7971
Merge branch 'absolute-meta-urls' into ait-qa
2020-08-06 19:38:11 -04:00
Neil Minton
0dd65dfd0b
Enable parsing of absolute URLs in meta HTML tags.
...
- Refactors several utility methods in UriUtils for URI parsing.
- Tests URLs in meta tags for absolute URIs.
The current W3C spec indicates those links should be relative. Several
examples have been observed in the wild with absolute URLs.
Browsers do not throw a parsing error and do redirect approprately. This
change conforms to that behavior.
Reference:
https://www.w3.org/TR/html53/document-metadata.html#statedef-http-equiv-refresh
2020-08-06 19:23:50 -04:00
Adam Miller
f879226c8e
Merge pull request #342 from internetarchive/noplaylist-ydl
...
youtube-dl --no-playlist
2020-08-03 16:40:54 -07:00
jkafader
34bb996e5c
Merge pull request #346 from internetarchive/fixes-leaky-file-handles
...
Fixes leaky file handles
2020-08-03 16:22:23 -07:00
Adam Miller
46100d65b2
Merge pull request #347 from internetarchive/fixes-leaky-file-handles
...
Manage youtube dl temp files which can be closed in the warc writer.
2020-07-31 15:27:09 -07:00
Adam Miller
a9c0c6588b
Manage youtube dl temp files which can be closed in the warc writer.
2020-07-31 22:22:25 +00:00
jkafader
ab27ff93e9
Merge pull request #345 from internetarchive/fixes-leaky-file-handles
...
Ensure Replay Input Stream and File Channels are closed after writing
2020-07-30 16:37:03 -07:00
Adam Miller
52a8f345ed
Fixing up logging and comments
2020-07-30 23:30:08 +00:00
Adam Miller
00d1c46d60
Ensure Replay Input Stream and File Channels are closed after writing
2020-07-30 23:19:43 +00:00
Barbara Miller
1a50ff538b
Merge branch 'noplaylist-ydl' into ait-qa
2020-07-29 11:15:02 -07:00
Adam Miller