lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-21 05:05:44 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,522 Commits 38 Branches 33 Tags
09ad15f7114fa247142ddbffa80d3f4fb8331f09
Commit Graph
2522 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
dependabot[bot] 09ad15f711 Bump org.springframework:spring-expression in /commons 
Bumps [org.springframework:spring-expression](https://github.com/spring-projects/spring-framework) from 5.3.27 to 5.3.39.
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v5.3.27...v5.3.39)

---
updated-dependencies:
- dependency-name: org.springframework:spring-expression
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-09-06 02:12:04 +00:00
Alex Osborne 2c9a5c97ab Merge pull request #586 from nla/kryo-5.6.0 
Update to kryo 5.6.0
 2024-09-06 11:10:17 +09:00
Alex Osborne ec57e2a8b1 Merge pull request #605 from bnfleb/extractorHTML 
Improve the HTML Parser for alternative resolution images
 2024-09-06 11:09:45 +09:00
Leslie Bellony ca3508f55f Add new HTML tags and attributes for ExtractorHTML (#604) 2024-09-04 17:49:40 +02:00
Kristinn Sigurðsson 9405cb9af1 Merge pull request #598 from kris-sigur/hostname-queues-with-limits 
Hostname based queue assignment variants that optionally limit queue name length
 2024-09-03 12:25:20 +00:00
Kristinn Sigurðsson 9ee8520604 ConfigurableExtractorJS (#602) 
Adds ConfigurableExtractorJS

Minor modification to ExtractorJS to make subclassing easier
 2024-09-03 12:24:08 +00:00
Alex Osborne 020c6d77b0 Merge pull request #603 from internetarchive/dependabot/maven/commons/dnsjava-dnsjava-3.6.0 
Bump dnsjava:dnsjava from 3.3.1 to 3.6.0 in /commons
 2024-09-03 11:46:15 +09:00
dependabot[bot] 3ec27f2097 Bump dnsjava:dnsjava from 3.3.1 to 3.6.0 in /commons 
Bumps [dnsjava:dnsjava](https://github.com/dnsjava/dnsjava) from 3.3.1 to 3.6.0.
- [Release notes](https://github.com/dnsjava/dnsjava/releases)
- [Changelog](https://github.com/dnsjava/dnsjava/blob/master/Changelog)
- [Commits](https://github.com/dnsjava/dnsjava/compare/v3.3.1...v3.6.0)

---
updated-dependencies:
- dependency-name: dnsjava:dnsjava
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-09-03 01:37:46 +00:00
Alex Osborne c35d72211c Merge pull request #601 from internetarchive/remove-extractor-chrome 
Remove ExtractorChrome
 2024-08-20 11:59:55 +09:00
Adam Miller 2d862c3e28 Merge pull request #600 from internetarchive/adam/restore-has-prerequisite-behavior 
Reset CrawlURI status for hasPrerequisite() so that it isn't preserved between attempts
 2024-08-08 12:05:22 -07:00
Kristinn Sigurðsson cd3a424176 Merge pull request #597 from kris-sigur/extractorhttp-implicit 
Add a more general support for inferred path discovery
 2024-08-08 07:37:45 +00:00
Kristinn Sigurðsson 0faf338f91 Merge pull request #595 from internetarchive/meta-content-name 
Apply speculativeFixup before evaluating meta content
 2024-08-08 07:36:36 +00:00
Alex Osborne 55a179c610 Remove ExtractorChrome 
This never worked that well, is causing random test failures and I don't have any plans to continue developing it.
 2024-08-08 07:21:46 +09:00
Adam Miller ec164ec77f Reset CrawlURI status for hasPrerequisite() so that it isn't preserved between attempts 2024-08-07 10:30:35 -07:00
Adam Miller b22d6ce2e1 Merge pull request #593 from internetarchive/adam/add-crawl-log-logging-to-extractoryoutubedl 
feat: Add logging to crawl.log for metadata records created by ExtractorYoutubeDL
 2024-08-07 10:17:25 -07:00
Alex Osborne 3ae1300a21 Merge pull request #599 from miku/miku/scriptedprocessor-javadoc-fix 
fix: ScriptedProcessor function name in docs
 2024-08-07 17:04:02 +09:00
Martin Czygan 3128ae5b84 fix: ScriptedProcessor function name in docs 2024-07-22 14:15:40 +02:00
Kristinn Sigurðsson 568db3cabc Added queue assignment variants that limit queue name length 2024-07-09 09:36:39 +00:00
Kristinn Sigurðsson 778a57b3b4 Add a more general support for inferred path discovery 2024-07-08 13:05:56 +00:00
Kristinn Sigurðsson 0dd3a2506d Apply speculativeFixup before evaluating meta content 
This avoids treating meta conent values like "Example.com" as relative
urls as they are converted to absolute URLs. This is already done for
speculative JS extraction.

The example sited above is common in meta "sitename" elements where the
sitename is something dot com or similar.
 2024-07-03 11:09:57 +00:00
Adam Miller 59635f00f4 chore: fix contrib dependencies 2024-06-21 23:34:23 +00:00
Adam Miller e64629cddd refactor: simplify sha1 calculation for yt-dlp json content. 2024-06-20 10:59:21 -07:00
Alex Osborne 74d6b37779 Update to kryo 5.6.0 
This eliminates a few more very old dependencies that aren't in Maven Central.

Our direct usage of the unsupported sun.reflect.ReflectionFactory JDK API (which newer compilers complain about) is no longer needed as Kryo now has a SerializingInstantiatorStrategy that does roughly the same thing.
 2024-06-18 10:05:56 +09:00
Alex Osborne 4a103f3e21 Merge pull request #592 from internetarchive/adam/fix-maven-junit3-test-discovery 
fix: maven was not running older junit3 style tests without some extr…
 2024-06-18 09:57:21 +09:00
Adam Miller cf5ccf7895 fix: maven was not running older junit3 style tests without some extra configuration. 2024-06-07 17:07:38 -07:00
Adam Miller 55dc23daaf feat: Add unit tests for ExtractorYoutubeDL WARC record building 2024-06-07 17:03:37 -07:00
Adam Miller d5e2ce6466 Make logging of metadata record optional, but default. 2024-06-06 14:32:45 -07:00
Adam Miller 3f764f7904 feat: Add logging to crawl.log for metadata records created by ExtractorYoutubeDL 2024-06-06 14:21:43 -07:00
Alex Osborne e38cd3528e Merge pull request #589 from internetarchive/jdk22 
JDK 22 support
 2024-05-15 23:17:26 +09:00
Alex Osborne 5424423e7d Add JDK 22 to test matrix 2024-05-15 23:14:51 +09:00
Alex Osborne 0c251c7c34 Update groovy to 4.0.21 for JDK 22 support 2024-05-15 23:13:18 +09:00
Alex Osborne a151937d4c Remove workaround for non-https builds.archive.org repository 
It's now accessible over HTTPS, so we don't need this anymore.
 2024-05-15 21:03:50 +09:00
Alex Osborne 64c7f0fb1c Merge pull request #572 from BulkSecurityGeneratorProjectV2/fix/JLL/use_https_to_resolve_dependencies_maven 
[SECURITY] Use HTTPS to resolve dependencies in Maven Build
 2024-05-15 15:50:07 +09:00
Alex Osborne f81a987110 Merge pull request #583 from internetarchive/dependabot/maven/commons/org.springframework-spring-expression-5.3.27 
Bump org.springframework:spring-expression from 5.3.20 to 5.3.27 in /commons
 2024-04-12 16:00:21 +09:00
dependabot[bot] 7975bf3d7c Bump org.springframework:spring-expression in /commons 
Bumps [org.springframework:spring-expression](https://github.com/spring-projects/spring-framework) from 5.3.20 to 5.3.27.
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v5.3.20...v5.3.27)

---
updated-dependencies:
- dependency-name: org.springframework:spring-expression
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-04-12 06:58:23 +00:00
Alex Osborne b3d276b6d8 Merge pull request #575 from internetarchive/pdfbox 
Switch ExtractorPDF and ExtractorPDFContent to pdfbox
 2024-04-12 15:49:17 +09:00
Barbara Miller 4552509941 Merge pull request #582 from galgeek/fix-yt-dlp-typo-too 2024-03-20 10:22:05 -07:00
Barbara Miller 3996195ab3 update comment 2024-03-20 10:17:34 -07:00
Barbara Miller d748f11ac7 Merge branch 'internetarchive:fix-yt-dlp-typo-too' into fix-yt-dlp-typo-too 2024-03-19 18:39:29 -07:00
Barbara Miller e84e125d94 720p 2024-03-19 12:50:59 -07:00
Barbara Miller a3d1873f42 Merge branch 'master' into fix-yt-dlp-typo-too 2024-03-18 17:26:46 -07:00
Barbara Miller 0bac560a9c rm ait updates 2024-03-18 17:24:26 -07:00
Barbara Miller 3396970d27 vcodec,res 2024-03-18 17:20:47 -07:00
Alex Osborne 19dd5c00de Switch ExtractorPDFContent to pdfbox 2024-02-21 14:48:58 +09:00
Alex Osborne 3700628cb2 Switch PDFParser and ExtractorPDF to pdfbox 2024-02-21 14:48:54 +09:00
Alex Osborne c467065b79 Add PDFParserTest 2024-02-21 14:48:50 +09:00
Barbara Miller 8549031e59 Merge pull request #565 from internetarchive/yt-dlp-20230706 
update ExtractorYoutubeDL.java for yt-dlp v.2023.07.06 and better
 2024-02-06 10:21:17 -08:00
Jonathan LeitschuhandModerne ea89c56c1d vuln-fix: Use HTTPS instead of HTTP to resolve deps CVE-2021-26291 
This fixes a security vulnerability in this project where the `pom.xml`
files were configuring Maven to resolve dependencies over HTTP instead of
HTTPS.

Weakness: CWE-829: Inclusion of Functionality from Untrusted Control Sphere
Severity: High
CVSS: 8.1
Detection: CodeQL & OpenRewrite (https://app.moderne.io/recipes/org.openrewrite.maven.security.UseHttpsForRepositories)

Reported-by: Jonathan Leitschuh <Jonathan.Leitschuh@gmail.com>
Signed-off-by: Jonathan Leitschuh <Jonathan.Leitschuh@gmail.com>

Bug-tracker: https://github.com/JLLeitschuh/security-research/issues/8
Detection: CodeQL (https://codeql.github.com/codeql-query-help/java/java-maven-non-https-url/) & OpenRewrite (https://app.moderne.io/recipes/org.openrewrite.maven.security.UseHttpsForRepositories)

Reported-by: Jonathan Leitschuh <Jonathan.Leitschuh@gmail.com>
Signed-off-by: Jonathan Leitschuh <Jonathan.Leitschuh@gmail.com>

Bug-tracker: https://github.com/JLLeitschuh/security-research/issues/8


Use this link to re-run the recipe: https://app.moderne.io/recipes/builder/IfHkrYfxx?organizationId=QWxsIEdpdEh1Yg%3D%3D

Co-authored-by: Moderne <team@moderne.io>
 2023-12-16 06:58:50 +00:00
Barbara Miller e4c9eb210a more updates for newer yt-dlp 2023-11-07 14:09:13 -08:00
Alex Osborne d3ca691f8d Update CI tests from JDK 20 to 21 2023-10-11 11:22:17 +09:00