lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-26 08:26:53 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,646 Commits 38 Branches 32 Tags
ait-qa
Commit Graph
2646 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Adam Miller 54c6b02040 chore: fix contrib dependencies 2024-06-21 22:48:40 +00:00
Adam Miller 13f63cd952 refactor: simplify sha1 calculation for yt-dlp json content. 2024-06-20 20:16:12 +00:00
Adam Miller 71911100cf fix: re-add imports where are used in this branch, but not others 2024-06-14 00:21:34 +00:00
Adam Miller 871db559e0 feat: Add unit tests for ExtractorYoutubeDL WARC record building 2024-06-13 23:28:14 +00:00
Adam Miller 5483407829 Make logging of metadata record optional, but default. 2024-06-13 23:28:05 +00:00
Adam Miller 2f3605540d feat: Add logging to crawl.log for metadata records created by ExtractorYoutubeDL 2024-06-13 23:27:39 +00:00
Barbara Miller 12691f5d9d Merge branch 'fix-yt-dlp-typo' into ait-qa 2024-03-19 13:06:58 -07:00
Barbara Miller 547aa7d954 720p 2024-03-19 13:01:41 -07:00
Barbara Miller 956d3a29c1 vcodec,res 2024-03-18 15:43:26 -07:00
Barbara Miller 737ea9f673 vcodec,res 2024-03-18 15:38:25 -07:00
Neil Minton a230f9e402 Merge pull request #577 from BitBaron/play-nice 
Nice spawned video extraction subprocesses.
 2024-03-14 07:04:41 -04:00
Neil Minton 6de3580794 Modify process array. 
Dealing with:
> java.io.IOException: Cannot run program "nice -n10": error=2, No such file or directory
 2024-03-13 21:39:16 -04:00
Neil Minton fcbdad584c Modify process array. 
Dealing with:
> java.io.IOException: Cannot run program "nice -n10": error=2, No such file or directory
 2024-03-13 21:32:48 -04:00
Neil Minton 32d356e967 Update logging and comments. 2024-03-13 12:58:02 -04:00
Neil Minton 32078dd418 Make video extraction nicer. 2024-03-13 12:58:02 -04:00
Neil Minton 7eed2bc03c Update logging and comments. 2024-03-13 12:53:53 -04:00
Neil Minton d608bb0b43 Make video extraction nicer. 2024-03-13 12:53:53 -04:00
Neil Minton ca079845c8 Merge pull request #574 from internetarchive/yt-dlp 
Update ExtractorYoutubeDL.java for yt-dlp v.2023.07.06 and better
 2024-02-09 06:31:30 -05:00
Barbara Miller 4b390404f0 more updates for newer yt-dlp 2024-02-09 06:27:29 -05:00
Barbara Miller 75b18e6117 yt-dlp (not youtube-dl) 2024-02-09 06:27:29 -05:00
Barbara Miller 3bf1c70eef Update ExtractorYoutubeDL.java 2024-02-09 06:27:29 -05:00
Barbara Miller a1500586a8 more updates for newer yt-dlp 2023-11-07 14:10:26 -08:00
Barbara Miller e19bdc0af1 yt-dlp (not youtube-dl) 2023-09-29 16:12:25 -07:00
Barbara Miller e86a5fedfe Revert "Merge branch 'yt-dlp-20230706' into ait-qa" 
This reverts commit 1074d95e2d, reversing
changes made to 122417c0fb.
 2023-09-29 16:11:20 -07:00
Barbara Miller 1074d95e2d Merge branch 'yt-dlp-20230706' into ait-qa 2023-09-29 15:48:35 -07:00
Barbara Miller bcfed0c1f7 yt-dlp (not youtube-dl) 2023-09-29 15:41:13 -07:00
Barbara Miller 122417c0fb Update ExtractorYoutubeDL.java 2023-09-06 11:47:59 -07:00
Barbara Miller d50158c012 Update ExtractorYoutubeDL.java 2023-09-06 11:25:12 -07:00
Andy Jackson 8563f491a5 Merge pull request #536 from claire9910/oscs_fix_cd5c2voau51t5k2gfi30 
fix(sec): upgrade com.itextpdf:itextpdf to 5.5.12
 2023-03-10 13:12:30 +00:00
Andy Jackson a3952be181 Merge pull request #552 from internetarchive/revert-375-fix-last-modified-millis 
Revert "Use seconds from the FetchCompletedTime for LastModified time"
 2023-03-10 13:05:55 +00:00
Andy Jackson d073b43b48 Revert "Use seconds from the FetchCompletedTime for LastModified time" 2023-03-10 13:05:39 +00:00
Andy Jackson d948364105 Merge pull request #549 from KBNLresearch/master 
Job Status description  now shows that a job is not running anymore
 2023-03-10 12:57:53 +00:00
Andy Jackson b25d71339e Merge pull request #551 from internetarchive/revert-445-adds-bdbmodule-max-file-size-option 
Revert "Adding configurable MaxLogFileSize option to BdbModule"
 2023-03-10 12:53:48 +00:00
Andy Jackson b6af2389fb Revert "Adding configurable MaxLogFileSize option to BdbModule" 2023-03-10 12:53:35 +00:00
Andy Jackson 87566eac61 Merge pull request #445 from internetarchive/adds-bdbmodule-max-file-size-option 
Adding configurable MaxLogFileSize option to BdbModule
 2023-03-10 11:23:06 +00:00
Andy Jackson 2d0afcdbd7 Merge pull request #375 from phanidv/fix-last-modified-millis 
Use seconds from the FetchCompletedTime for LastModified time
 2023-03-10 11:21:32 +00:00
fedor cdfcce8deb In case of a crashed job, the crawlsummaryreport says: "Finished - Abnormal exit from crawl while the ApplicationContext still reports running==true. So the console shows "Job is ACTIVE - Running", while the process crashed This change reports the exit status in this case. 2023-03-06 15:54:27 +01:00
dependabot[bot] c45d524e74 Bump commons-net from 2.0 to 3.9.0 in /commons 
Bumps commons-net from 2.0 to 3.9.0.

---
updated-dependencies:
- dependency-name: commons-net:commons-net
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-01-23 15:17:30 +09:00
Alex Osborne f4280012ae BdbModule: Fix resource leak on job teardown if sync() throws 
This change ensures that close() is always called by moving sync() to a
separate try-block.

Often when a BDB error occurs it causes all subsequent operations
including sync() also throw exceptions. Since we were calling sync()
and close() in the same try-block, sync() throwing would prevent close()
from ever being called and thus leak file handles and BDB worker
threads.
 2023-01-23 14:47:07 +09:00
Alex Osborne a34d9f3f5e Merge pull request #539 from krakan/dont-warn-for-non-authenticating-calls 
Don't warn for non authenticating calls
 2023-01-23 14:44:16 +09:00
Jonas Linde de5c419d8f Don't log authentication failures when header is missing 
The digest authentication method requires an initial request to get a challenge to use for the subsequent authenmtication digest. This initial request is not an authentication attempt and should not be handled as a failed such.
 2022-11-16 11:24:21 +01:00
Jonas Linde 4bad4713ba Set correct timestamp for lastFailureTime 
The initial sleepMs value will be the negative of the current time minus 6 seconds. Using that value to calculate lastFailureTime will increase that large negative number with 6000 milliseconds for every failure which means that rate limiting won't occur until after millions of failed attempts.
 2022-11-16 11:24:21 +01:00
Alex Osborne b9cb43e910 Update groovy to 3.0.13 for JDK 19 support 2022-11-09 21:11:45 +09:00
Alex Osborne b0fb41e7b5 Update CI tests from JDK 18 to 19 2022-11-09 21:03:30 +09:00
Christian Clauss c60af1eba2 Fix typo 2022-10-24 15:06:21 +09:00
claire9910 7cd4a914c3 update com.itextpdf:itextpdf 5.5.0 to 5.5.12 2022-10-15 22:19:49 +08:00
Alex Osborne b328ded92e docs: Add 'Checkpointing' section to operating guide 
Fixes #500 #506
 2022-09-30 15:31:21 +09:00
Alex Osborne 3c79a0b2b1 Increase DiskSpaceMonitor default pause threshold to 8 GiB 
When there's less than 5 GiB free BDB throws DiskLimitException which
Heritrix will likely be unable to handle gracefully and crawl job will
break in various ways. #499
 2022-09-10 23:17:53 +09:00
Alex Osborne f736bf2854 docs: Add DecideRuleSequence to bean-reference 2022-09-06 18:02:30 +09:00
Andy Jackson c5d9bbd0c0 Fix changelog following tag merge. 2022-07-28 10:58:48 +01:00