lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-20 20:55:49 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,510 Commits 38 Branches 33 Tags
2d862c3e2811c3fef0f21ba8511f6bbc4e248f6e
Commit Graph
2510 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Adam Miller 2d862c3e28 Merge pull request #600 from internetarchive/adam/restore-has-prerequisite-behavior 
Reset CrawlURI status for hasPrerequisite() so that it isn't preserved between attempts
 2024-08-08 12:05:22 -07:00
Kristinn Sigurðsson cd3a424176 Merge pull request #597 from kris-sigur/extractorhttp-implicit 
Add a more general support for inferred path discovery
 2024-08-08 07:37:45 +00:00
Kristinn Sigurðsson 0faf338f91 Merge pull request #595 from internetarchive/meta-content-name 
Apply speculativeFixup before evaluating meta content
 2024-08-08 07:36:36 +00:00
Adam Miller ec164ec77f Reset CrawlURI status for hasPrerequisite() so that it isn't preserved between attempts 2024-08-07 10:30:35 -07:00
Adam Miller b22d6ce2e1 Merge pull request #593 from internetarchive/adam/add-crawl-log-logging-to-extractoryoutubedl 
feat: Add logging to crawl.log for metadata records created by ExtractorYoutubeDL
 2024-08-07 10:17:25 -07:00
Alex Osborne 3ae1300a21 Merge pull request #599 from miku/miku/scriptedprocessor-javadoc-fix 
fix: ScriptedProcessor function name in docs
 2024-08-07 17:04:02 +09:00
Martin Czygan 3128ae5b84 fix: ScriptedProcessor function name in docs 2024-07-22 14:15:40 +02:00
Kristinn Sigurðsson 778a57b3b4 Add a more general support for inferred path discovery 2024-07-08 13:05:56 +00:00
Kristinn Sigurðsson 0dd3a2506d Apply speculativeFixup before evaluating meta content 
This avoids treating meta conent values like "Example.com" as relative
urls as they are converted to absolute URLs. This is already done for
speculative JS extraction.

The example sited above is common in meta "sitename" elements where the
sitename is something dot com or similar.
 2024-07-03 11:09:57 +00:00
Adam Miller 59635f00f4 chore: fix contrib dependencies 2024-06-21 23:34:23 +00:00
Adam Miller e64629cddd refactor: simplify sha1 calculation for yt-dlp json content. 2024-06-20 10:59:21 -07:00
Alex Osborne 4a103f3e21 Merge pull request #592 from internetarchive/adam/fix-maven-junit3-test-discovery 
fix: maven was not running older junit3 style tests without some extr…
 2024-06-18 09:57:21 +09:00
Adam Miller cf5ccf7895 fix: maven was not running older junit3 style tests without some extra configuration. 2024-06-07 17:07:38 -07:00
Adam Miller 55dc23daaf feat: Add unit tests for ExtractorYoutubeDL WARC record building 2024-06-07 17:03:37 -07:00
Adam Miller d5e2ce6466 Make logging of metadata record optional, but default. 2024-06-06 14:32:45 -07:00
Adam Miller 3f764f7904 feat: Add logging to crawl.log for metadata records created by ExtractorYoutubeDL 2024-06-06 14:21:43 -07:00
Alex Osborne e38cd3528e Merge pull request #589 from internetarchive/jdk22 
JDK 22 support
 2024-05-15 23:17:26 +09:00
Alex Osborne 5424423e7d Add JDK 22 to test matrix 2024-05-15 23:14:51 +09:00
Alex Osborne 0c251c7c34 Update groovy to 4.0.21 for JDK 22 support 2024-05-15 23:13:18 +09:00
Alex Osborne a151937d4c Remove workaround for non-https builds.archive.org repository 
It's now accessible over HTTPS, so we don't need this anymore.
 2024-05-15 21:03:50 +09:00
Alex Osborne 64c7f0fb1c Merge pull request #572 from BulkSecurityGeneratorProjectV2/fix/JLL/use_https_to_resolve_dependencies_maven 
[SECURITY] Use HTTPS to resolve dependencies in Maven Build
 2024-05-15 15:50:07 +09:00
Alex Osborne f81a987110 Merge pull request #583 from internetarchive/dependabot/maven/commons/org.springframework-spring-expression-5.3.27 
Bump org.springframework:spring-expression from 5.3.20 to 5.3.27 in /commons
 2024-04-12 16:00:21 +09:00
dependabot[bot] 7975bf3d7c Bump org.springframework:spring-expression in /commons 
Bumps [org.springframework:spring-expression](https://github.com/spring-projects/spring-framework) from 5.3.20 to 5.3.27.
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v5.3.20...v5.3.27)

---
updated-dependencies:
- dependency-name: org.springframework:spring-expression
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-04-12 06:58:23 +00:00
Alex Osborne b3d276b6d8 Merge pull request #575 from internetarchive/pdfbox 
Switch ExtractorPDF and ExtractorPDFContent to pdfbox
 2024-04-12 15:49:17 +09:00
Barbara Miller 4552509941 Merge pull request #582 from galgeek/fix-yt-dlp-typo-too 2024-03-20 10:22:05 -07:00
Barbara Miller 3996195ab3 update comment 2024-03-20 10:17:34 -07:00
Barbara Miller d748f11ac7 Merge branch 'internetarchive:fix-yt-dlp-typo-too' into fix-yt-dlp-typo-too 2024-03-19 18:39:29 -07:00
Barbara Miller e84e125d94 720p 2024-03-19 12:50:59 -07:00
Barbara Miller a3d1873f42 Merge branch 'master' into fix-yt-dlp-typo-too 2024-03-18 17:26:46 -07:00
Barbara Miller 0bac560a9c rm ait updates 2024-03-18 17:24:26 -07:00
Barbara Miller 3396970d27 vcodec,res 2024-03-18 17:20:47 -07:00
Alex Osborne 19dd5c00de Switch ExtractorPDFContent to pdfbox 2024-02-21 14:48:58 +09:00
Alex Osborne 3700628cb2 Switch PDFParser and ExtractorPDF to pdfbox 2024-02-21 14:48:54 +09:00
Alex Osborne c467065b79 Add PDFParserTest 2024-02-21 14:48:50 +09:00
Barbara Miller 8549031e59 Merge pull request #565 from internetarchive/yt-dlp-20230706 
update ExtractorYoutubeDL.java for yt-dlp v.2023.07.06 and better
 2024-02-06 10:21:17 -08:00
Jonathan LeitschuhandModerne ea89c56c1d vuln-fix: Use HTTPS instead of HTTP to resolve deps CVE-2021-26291 
This fixes a security vulnerability in this project where the `pom.xml`
files were configuring Maven to resolve dependencies over HTTP instead of
HTTPS.

Weakness: CWE-829: Inclusion of Functionality from Untrusted Control Sphere
Severity: High
CVSS: 8.1
Detection: CodeQL & OpenRewrite (https://app.moderne.io/recipes/org.openrewrite.maven.security.UseHttpsForRepositories)

Reported-by: Jonathan Leitschuh <Jonathan.Leitschuh@gmail.com>
Signed-off-by: Jonathan Leitschuh <Jonathan.Leitschuh@gmail.com>

Bug-tracker: https://github.com/JLLeitschuh/security-research/issues/8
Detection: CodeQL (https://codeql.github.com/codeql-query-help/java/java-maven-non-https-url/) & OpenRewrite (https://app.moderne.io/recipes/org.openrewrite.maven.security.UseHttpsForRepositories)

Reported-by: Jonathan Leitschuh <Jonathan.Leitschuh@gmail.com>
Signed-off-by: Jonathan Leitschuh <Jonathan.Leitschuh@gmail.com>

Bug-tracker: https://github.com/JLLeitschuh/security-research/issues/8


Use this link to re-run the recipe: https://app.moderne.io/recipes/builder/IfHkrYfxx?organizationId=QWxsIEdpdEh1Yg%3D%3D

Co-authored-by: Moderne <team@moderne.io>
 2023-12-16 06:58:50 +00:00
Barbara Miller e4c9eb210a more updates for newer yt-dlp 2023-11-07 14:09:13 -08:00
Alex Osborne d3ca691f8d Update CI tests from JDK 20 to 21 2023-10-11 11:22:17 +09:00
Alex Osborne bd6c6bbff5 Merge pull request #566 from internetarchive/jdk20 
Update groovy to 4.0.15 for JDK 20 support
 2023-10-11 11:09:53 +09:00
Barbara Miller bcfed0c1f7 yt-dlp (not youtube-dl) 2023-09-29 15:41:13 -07:00
Alex Osborne 1a27551e60 Update CI tests from JDK 19 to 20 2023-09-19 11:21:05 +09:00
Alex Osborne 1c07e43ade Update groovy to 4.0.15 for JDK 20 support 2023-09-19 11:21:05 +09:00
Barbara Miller d50158c012 Update ExtractorYoutubeDL.java 2023-09-06 11:25:12 -07:00
Andy Jackson 8563f491a5 Merge pull request #536 from claire9910/oscs_fix_cd5c2voau51t5k2gfi30 
fix(sec): upgrade com.itextpdf:itextpdf to 5.5.12
 2023-03-10 13:12:30 +00:00
Andy Jackson a3952be181 Merge pull request #552 from internetarchive/revert-375-fix-last-modified-millis 
Revert "Use seconds from the FetchCompletedTime for LastModified time"
 2023-03-10 13:05:55 +00:00
Andy Jackson d073b43b48 Revert "Use seconds from the FetchCompletedTime for LastModified time" 2023-03-10 13:05:39 +00:00
Andy Jackson d948364105 Merge pull request #549 from KBNLresearch/master 
Job Status description  now shows that a job is not running anymore
 2023-03-10 12:57:53 +00:00
Andy Jackson 2d0afcdbd7 Merge pull request #375 from phanidv/fix-last-modified-millis 
Use seconds from the FetchCompletedTime for LastModified time
 2023-03-10 11:21:32 +00:00
fedor cdfcce8deb In case of a crashed job, the crawlsummaryreport says: "Finished - Abnormal exit from crawl while the ApplicationContext still reports running==true. So the console shows "Job is ACTIVE - Running", while the process crashed This change reports the exit status in this case. 2023-03-06 15:54:27 +01:00
dependabot[bot] c45d524e74 Bump commons-net from 2.0 to 3.9.0 in /commons 
Bumps commons-net from 2.0 to 3.9.0.

---
updated-dependencies:
- dependency-name: commons-net:commons-net
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-01-23 15:17:30 +09:00