lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-21 22:16:50 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,497 Commits 40 Branches 31 Tags
778a57b3b47b20fd25df17dee3ff48fb526e68bc
Commit Graph
2497 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Kristinn Sigurðsson 778a57b3b4 Add a more general support for inferred path discovery 2024-07-08 13:05:56 +00:00
Alex Osborne 4a103f3e21 Merge pull request #592 from internetarchive/adam/fix-maven-junit3-test-discovery 
fix: maven was not running older junit3 style tests without some extr…
 2024-06-18 09:57:21 +09:00
Adam Miller cf5ccf7895 fix: maven was not running older junit3 style tests without some extra configuration. 2024-06-07 17:07:38 -07:00
Alex Osborne e38cd3528e Merge pull request #589 from internetarchive/jdk22 
JDK 22 support
 2024-05-15 23:17:26 +09:00
Alex Osborne 5424423e7d Add JDK 22 to test matrix 2024-05-15 23:14:51 +09:00
Alex Osborne 0c251c7c34 Update groovy to 4.0.21 for JDK 22 support 2024-05-15 23:13:18 +09:00
Alex Osborne a151937d4c Remove workaround for non-https builds.archive.org repository 
It's now accessible over HTTPS, so we don't need this anymore.
 2024-05-15 21:03:50 +09:00
Alex Osborne 64c7f0fb1c Merge pull request #572 from BulkSecurityGeneratorProjectV2/fix/JLL/use_https_to_resolve_dependencies_maven 
[SECURITY] Use HTTPS to resolve dependencies in Maven Build
 2024-05-15 15:50:07 +09:00
Alex Osborne f81a987110 Merge pull request #583 from internetarchive/dependabot/maven/commons/org.springframework-spring-expression-5.3.27 
Bump org.springframework:spring-expression from 5.3.20 to 5.3.27 in /commons
 2024-04-12 16:00:21 +09:00
dependabot[bot] 7975bf3d7c Bump org.springframework:spring-expression in /commons 
Bumps [org.springframework:spring-expression](https://github.com/spring-projects/spring-framework) from 5.3.20 to 5.3.27.
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v5.3.20...v5.3.27)

---
updated-dependencies:
- dependency-name: org.springframework:spring-expression
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-04-12 06:58:23 +00:00
Alex Osborne b3d276b6d8 Merge pull request #575 from internetarchive/pdfbox 
Switch ExtractorPDF and ExtractorPDFContent to pdfbox
 2024-04-12 15:49:17 +09:00
Barbara Miller 4552509941 Merge pull request #582 from galgeek/fix-yt-dlp-typo-too 2024-03-20 10:22:05 -07:00
Barbara Miller 3996195ab3 update comment 2024-03-20 10:17:34 -07:00
Barbara Miller d748f11ac7 Merge branch 'internetarchive:fix-yt-dlp-typo-too' into fix-yt-dlp-typo-too 2024-03-19 18:39:29 -07:00
Barbara Miller e84e125d94 720p 2024-03-19 12:50:59 -07:00
Barbara Miller a3d1873f42 Merge branch 'master' into fix-yt-dlp-typo-too 2024-03-18 17:26:46 -07:00
Barbara Miller 0bac560a9c rm ait updates 2024-03-18 17:24:26 -07:00
Barbara Miller 3396970d27 vcodec,res 2024-03-18 17:20:47 -07:00
Alex Osborne 19dd5c00de Switch ExtractorPDFContent to pdfbox 2024-02-21 14:48:58 +09:00
Alex Osborne 3700628cb2 Switch PDFParser and ExtractorPDF to pdfbox 2024-02-21 14:48:54 +09:00
Alex Osborne c467065b79 Add PDFParserTest 2024-02-21 14:48:50 +09:00
Barbara Miller 8549031e59 Merge pull request #565 from internetarchive/yt-dlp-20230706 
update ExtractorYoutubeDL.java for yt-dlp v.2023.07.06 and better
 2024-02-06 10:21:17 -08:00
Jonathan LeitschuhandModerne ea89c56c1d vuln-fix: Use HTTPS instead of HTTP to resolve deps CVE-2021-26291 
This fixes a security vulnerability in this project where the `pom.xml`
files were configuring Maven to resolve dependencies over HTTP instead of
HTTPS.

Weakness: CWE-829: Inclusion of Functionality from Untrusted Control Sphere
Severity: High
CVSS: 8.1
Detection: CodeQL & OpenRewrite (https://app.moderne.io/recipes/org.openrewrite.maven.security.UseHttpsForRepositories)

Reported-by: Jonathan Leitschuh <Jonathan.Leitschuh@gmail.com>
Signed-off-by: Jonathan Leitschuh <Jonathan.Leitschuh@gmail.com>

Bug-tracker: https://github.com/JLLeitschuh/security-research/issues/8
Detection: CodeQL (https://codeql.github.com/codeql-query-help/java/java-maven-non-https-url/) & OpenRewrite (https://app.moderne.io/recipes/org.openrewrite.maven.security.UseHttpsForRepositories)

Reported-by: Jonathan Leitschuh <Jonathan.Leitschuh@gmail.com>
Signed-off-by: Jonathan Leitschuh <Jonathan.Leitschuh@gmail.com>

Bug-tracker: https://github.com/JLLeitschuh/security-research/issues/8


Use this link to re-run the recipe: https://app.moderne.io/recipes/builder/IfHkrYfxx?organizationId=QWxsIEdpdEh1Yg%3D%3D

Co-authored-by: Moderne <team@moderne.io>
 2023-12-16 06:58:50 +00:00
Barbara Miller e4c9eb210a more updates for newer yt-dlp 2023-11-07 14:09:13 -08:00
Alex Osborne d3ca691f8d Update CI tests from JDK 20 to 21 2023-10-11 11:22:17 +09:00
Alex Osborne bd6c6bbff5 Merge pull request #566 from internetarchive/jdk20 
Update groovy to 4.0.15 for JDK 20 support
 2023-10-11 11:09:53 +09:00
Barbara Miller bcfed0c1f7 yt-dlp (not youtube-dl) 2023-09-29 15:41:13 -07:00
Alex Osborne 1a27551e60 Update CI tests from JDK 19 to 20 2023-09-19 11:21:05 +09:00
Alex Osborne 1c07e43ade Update groovy to 4.0.15 for JDK 20 support 2023-09-19 11:21:05 +09:00
Barbara Miller d50158c012 Update ExtractorYoutubeDL.java 2023-09-06 11:25:12 -07:00
Andy Jackson 8563f491a5 Merge pull request #536 from claire9910/oscs_fix_cd5c2voau51t5k2gfi30 
fix(sec): upgrade com.itextpdf:itextpdf to 5.5.12
 2023-03-10 13:12:30 +00:00
Andy Jackson a3952be181 Merge pull request #552 from internetarchive/revert-375-fix-last-modified-millis 
Revert "Use seconds from the FetchCompletedTime for LastModified time"
 2023-03-10 13:05:55 +00:00
Andy Jackson d073b43b48 Revert "Use seconds from the FetchCompletedTime for LastModified time" 2023-03-10 13:05:39 +00:00
Andy Jackson d948364105 Merge pull request #549 from KBNLresearch/master 
Job Status description  now shows that a job is not running anymore
 2023-03-10 12:57:53 +00:00
Andy Jackson 2d0afcdbd7 Merge pull request #375 from phanidv/fix-last-modified-millis 
Use seconds from the FetchCompletedTime for LastModified time
 2023-03-10 11:21:32 +00:00
fedor cdfcce8deb In case of a crashed job, the crawlsummaryreport says: "Finished - Abnormal exit from crawl while the ApplicationContext still reports running==true. So the console shows "Job is ACTIVE - Running", while the process crashed This change reports the exit status in this case. 2023-03-06 15:54:27 +01:00
dependabot[bot] c45d524e74 Bump commons-net from 2.0 to 3.9.0 in /commons 
Bumps commons-net from 2.0 to 3.9.0.

---
updated-dependencies:
- dependency-name: commons-net:commons-net
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-01-23 15:17:30 +09:00
Alex Osborne f4280012ae BdbModule: Fix resource leak on job teardown if sync() throws 
This change ensures that close() is always called by moving sync() to a
separate try-block.

Often when a BDB error occurs it causes all subsequent operations
including sync() also throw exceptions. Since we were calling sync()
and close() in the same try-block, sync() throwing would prevent close()
from ever being called and thus leak file handles and BDB worker
threads.
 2023-01-23 14:47:07 +09:00
Alex Osborne a34d9f3f5e Merge pull request #539 from krakan/dont-warn-for-non-authenticating-calls 
Don't warn for non authenticating calls
 2023-01-23 14:44:16 +09:00
Jonas Linde de5c419d8f Don't log authentication failures when header is missing 
The digest authentication method requires an initial request to get a challenge to use for the subsequent authenmtication digest. This initial request is not an authentication attempt and should not be handled as a failed such.
 2022-11-16 11:24:21 +01:00
Jonas Linde 4bad4713ba Set correct timestamp for lastFailureTime 
The initial sleepMs value will be the negative of the current time minus 6 seconds. Using that value to calculate lastFailureTime will increase that large negative number with 6000 milliseconds for every failure which means that rate limiting won't occur until after millions of failed attempts.
 2022-11-16 11:24:21 +01:00
Alex Osborne b9cb43e910 Update groovy to 3.0.13 for JDK 19 support 2022-11-09 21:11:45 +09:00
Alex Osborne b0fb41e7b5 Update CI tests from JDK 18 to 19 2022-11-09 21:03:30 +09:00
Christian Clauss c60af1eba2 Fix typo 2022-10-24 15:06:21 +09:00
claire9910 7cd4a914c3 update com.itextpdf:itextpdf 5.5.0 to 5.5.12 2022-10-15 22:19:49 +08:00
Alex Osborne b328ded92e docs: Add 'Checkpointing' section to operating guide 
Fixes #500 #506
 2022-09-30 15:31:21 +09:00
Alex Osborne 3c79a0b2b1 Increase DiskSpaceMonitor default pause threshold to 8 GiB 
When there's less than 5 GiB free BDB throws DiskLimitException which
Heritrix will likely be unable to handle gracefully and crawl job will
break in various ways. #499
 2022-09-10 23:17:53 +09:00
Alex Osborne f736bf2854 docs: Add DecideRuleSequence to bean-reference 2022-09-06 18:02:30 +09:00
Andy Jackson c5d9bbd0c0 Fix changelog following tag merge. 2022-07-28 10:58:48 +01:00
Andy Jackson e83f4f9e02 Reset to SNAPSHOT 2022-07-28 10:56:59 +01:00