lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-26 16:36:37 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,272 Commits 38 Branches 32 Tags
122417c0fbc2ee3cfd99f08a06dc334279ce5dd4
Commit Graph
2272 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Barbara Miller 122417c0fb Update ExtractorYoutubeDL.java 2023-09-06 11:47:59 -07:00
Barbara Miller 677744e756 use yt-dlp instead of youtube-dl 2022-05-10 15:27:58 -07:00
Adam Miller d2d8c490bc Merge pull request #412 from internetarchive/adds-bdbmodule-max-file-size-option 
Adds bdbmodule max file size option
 2021-07-16 10:21:56 -07:00
Adam Miller 953e7c9399 Adding configurable MaxLogFileSize option to BdbModule 2021-07-13 21:29:33 +00:00
Adam Miller c403d7b913 Merge pull request #395 from internetarchive/adds-trough-dedup-performance-rework 
Refactor Trough client URL cache put. Limit TroughContentDigestHistor…
 2021-05-28 16:15:22 -07:00
Adam Miller 58bf0cf45c Refactor Trough client URL cache put. Limit TroughContentDigestHistory query. 2021-05-28 23:10:11 +00:00
Adam Miller 8c59cdfbad Merge pull request #385 from internetarchive/adds-trough-dedup-performance-rework 
Add dedup load from in-memory dedup cache db
 2021-05-20 14:45:40 -07:00
Adam Miller 7fc648cad4 Add dedup load from in-memory dedup cache db 2021-05-20 21:37:11 +00:00
Adam Miller 1b9565ef45 Merge pull request #382 from internetarchive/adds-trough-dedup-performance-rework 
Ensure we set dedup schema when writing dedup shards
 2021-05-17 12:45:56 -07:00
Adam Miller 6fad293759 Ensure we set dedup schema when writing dedup shards 2021-05-17 19:41:27 +00:00
Adam Miller d1026bdc55 Merge pull request #381 from internetarchive/adds-trough-dedup-performance-rework 
Refactor crawl log batch posting. Add configuration options for troug…
 2021-05-13 10:39:20 -07:00
Adam Miller fc402acd06 Merge branch 'ait-qa' into adds-trough-dedup-performance-rework 2021-05-13 10:39:02 -07:00
Adam Miller 18edb3e413 Refactor crawl log batch posting. Add configuration options for trough dedup batches. Fix posting of trough dedup at end of crawl. 2021-05-13 17:32:26 +00:00
Adam Miller 4625a78a2c Merge pull request #374 from internetarchive/adds-trough-dedup-performance-rework 
Adds trough dedup performance rework
 2021-04-05 16:49:50 -07:00
Adam Miller e353ccb097 Merge branch 'ait-qa' into adds-trough-dedup-performance-rework 2021-04-05 16:17:47 -07:00
Adam Miller 21e84ad015 Merge pull request #373 from internetarchive/revert-367-fixes-trough-crawl-log-feed-synchronization 
Revert "Fix error log reporting of batch size for trough crawl logs"
 2021-04-05 16:01:51 -07:00
Adam Miller d6eb68a265 Revert "Fix error log reporting of batch size for trough crawl logs" 2021-04-05 16:01:27 -07:00
Adam Miller 0b09340e84 Fix inconsistency with cached batch sizes and refine logic for forcing final batch posting at end of crawl 2021-04-05 22:56:23 +00:00
Adam Miller d72175635e Ensure that the final batch is posted during shutdown. Make posting of uncrawled urls optional. 2021-04-05 21:06:56 +00:00
Adam Miller 466c6795b3 Merge branch 'fixes-trough-crawl-log-feed-synchronization' into adds-trough-dedup-performance-rework 2021-03-10 21:36:33 +00:00
Adam Miller 54abea3157 Merge branch 'adds-trough-local-dedup-write-cache' into adds-trough-dedup-performance-rework 2021-03-10 21:36:24 +00:00
Adam Miller 8724b14a9d Replacing some synchronized lists with threadsafe objects and fixing discrepency between crawled and uncrawled lists 2021-03-10 21:21:59 +00:00
jkafader 12eef6c70d Merge pull request #367 from internetarchive/fixes-trough-crawl-log-feed-synchronization 
Fix error log reporting of batch size for trough crawl logs
 2021-02-26 15:41:03 -08:00
Adam Miller e700f384d5 Switching to in memory sqlite for dedup cache 2021-02-26 23:32:20 +00:00
Adam Miller 88b7eeaf02 On disk - per-seed sqlite dedup 2021-02-25 22:35:08 +00:00
Adam Miller 00947b0167 Fixing sync issues on uncrawled urls batch as well 2021-02-25 22:31:25 +00:00
Adam Miller e30005128d Merge branch 'adds-trough-local-dedup-write-cache' into adds-trough-dedup-performance-rework 2021-02-23 00:44:34 +00:00
Adam Miller 09485a5dcd Merge branch 'fixes-trough-crawl-log-feed-synchronization' into adds-trough-dedup-performance-rework 2021-02-23 00:44:23 +00:00
Adam Miller 86e238e7f7 Create and insert into local sqlite dedup db shards 2021-02-23 00:43:26 +00:00
Adam Miller 79e56337b2 Fix error log reporting of batch size for trough crawl logs 2021-02-18 23:06:51 +00:00
Adam Miller d22ef124d3 Trying to avoid hard locks on updating url caches 2021-02-02 23:58:48 +00:00
Barbara Miller 6a8b754ba7 Merge branch 'log_sleep' into ait-qa 2021-02-01 21:24:54 -08:00
Barbara Miller 7e82722da3 log sleep 2021-02-01 21:24:24 -08:00
jkafader 5e7a20d19a Merge pull request #364 from internetarchive/fixes-trough-crawl-log-feed-synchronization 
Fixing synchronization issues in TroughCrawlLogFeed.java to reduce bl…
 2021-01-28 15:31:53 -08:00
Adam Miller 14be8752e9 Fixing synchronization issues in TroughCrawlLogFeed.java to reduce blocking 2021-01-28 22:54:06 +00:00
jkafader 77e2e9171f Merge pull request #362 from internetarchive/fixes-WARCWriterChainProcessor-checkpoint-stats 
Fixing race condition on totalBytesWritten
 2021-01-14 12:08:10 -08:00
jkafader 2f640fc974 Merge pull request #355 from internetarchive/fixes-prerequisite-status 
Reset CrawlURI status for hasPrerequisite() so that it isn't preserve…
 2021-01-14 12:08:01 -08:00
Barbara Miller 147467c326 Merge branch 'fix_restlet_npe' into ait-qa 2021-01-12 18:50:16 -08:00
Barbara Miller b5b204b40a avoid NPE https://webarchive.jira.com/browse/WT-28?focusedCommentId=139095 2021-01-12 18:49:43 -08:00
Adam Miller 6ff10be094 Merge pull request #357 from internetarchive/fixes-prerequisite-status 
Reset CrawlURI status for hasPrerequisite() so that it isn't preserve…
 2020-11-12 15:46:16 -08:00
Adam Miller 6ebb28db18 Merge pull request #356 from internetarchive/fixes-WARCWriterChainProcessor-checkpoint-stats 
Fixing race condition on totalBytesWritten
 2020-11-12 15:20:14 -08:00
Adam Miller 1f357c8daf Fixing race condition on totalBytesWritten 2020-11-11 00:02:09 +00:00
Adam Miller e694a4b5c8 Reset CrawlURI status for hasPrerequisite() so that it isn't preserved between attempts 2020-11-07 01:12:56 +00:00
jkafader 288b167399 Merge pull request #354 from internetarchive/fixes-WARCWriterChainProcessor-checkpoint-stats 
Add checkpoint store/load functionality for warc writer stats
 2020-10-15 13:51:09 -07:00
Adam Miller 85052d6cb2 Merge branch 'fixes-WARCWriterChainProcessor-checkpoint-stats' into ait-qa 2020-10-09 22:14:23 +00:00
Adam Miller 5504d492a1 Add checkpoint store/load functionality for warc writer stats 2020-10-09 22:14:08 +00:00
jkafader 87ac0dacb2 Merge pull request #349 from BitBaron/absolute-meta-urls 
Better parsing of absolute meta element URLs.
 2020-08-17 12:38:02 -07:00
Neil Minton 13a1028e2f Merge branch 'absolute-meta-urls' into ait-qa 2020-08-14 10:59:17 -04:00
Neil Minton 941c6c50ad Strip quotes from URL value. 2020-08-14 10:54:18 -04:00
Neil Minton 6b1e88934a Merge branch 'absolute-meta-urls' into ait-qa 2020-08-12 13:16:30 -04:00