lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-19 04:05:45 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,531 Commits 38 Branches 33 Tags
2e891bc4bbfc6cf0dfe1f9278f876f4c7eb71f5b
Commit Graph
2531 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Alex Osborne 2e891bc4bb Merge pull request #608 from internetarchive/cite-navlinks 
ExtractorHTML: Treat 'cite' attribute as navlink instead of embed
 2024-09-13 13:19:03 +09:00
manderson 3a447b5e61 ExtractorHTML: Treat 'cite' attribute as navlink instead of embed 
The cite attribute is used to identify the source document of a blockquote. But ExtractorHTML was treating it as an embed which can cause out of scope pages to be included in a crawl incorrectly. Browsers don't use the cite attribute currently so there might be an argument for ignoring it entirely but let's at least not treat it as an embed.
 2024-09-10 12:42:15 +09:00
Alex Osborne 791f1c81be Add changelog links for 3.4.0-20240909 2024-09-10 12:27:56 +09:00
Alex Osborne 8ac80be9d3 Add 3.4.0-20240909 to CHANGELOG 2024-09-10 12:24:06 +09:00
Alex Osborne c7c492b507 [maven-release-plugin] prepare for next development iteration 2024-09-09 15:28:44 +09:00
Alex Osborne 11fcbe71f3 [maven-release-plugin] prepare release 3.4.0-20240909 3.4.0-20240909 2024-09-09 15:28:44 +09:00
Alex Osborne ae7a025c35 Merge pull request #607 from internetarchive/dependabot/maven/contrib/com.rabbitmq-amqp-client-5.18.0 
Bump com.rabbitmq:amqp-client from 4.8.0 to 5.18.0 in /contrib
 2024-09-06 11:18:33 +09:00
Alex Osborne ee048f4129 Merge pull request #606 from internetarchive/dependabot/maven/commons/org.springframework-spring-expression-5.3.39 
Bump org.springframework:spring-expression from 5.3.27 to 5.3.39 in /commons
 2024-09-06 11:17:14 +09:00
dependabot[bot] 2cffefc798 Bump com.rabbitmq:amqp-client from 4.8.0 to 5.18.0 in /contrib 
Bumps [com.rabbitmq:amqp-client](https://github.com/rabbitmq/rabbitmq-java-client) from 4.8.0 to 5.18.0.
- [Release notes](https://github.com/rabbitmq/rabbitmq-java-client/releases)
- [Commits](https://github.com/rabbitmq/rabbitmq-java-client/compare/v4.8.0...v5.18.0)

---
updated-dependencies:
- dependency-name: com.rabbitmq:amqp-client
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-09-06 02:13:37 +00:00
dependabot[bot] 09ad15f711 Bump org.springframework:spring-expression in /commons 
Bumps [org.springframework:spring-expression](https://github.com/spring-projects/spring-framework) from 5.3.27 to 5.3.39.
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v5.3.27...v5.3.39)

---
updated-dependencies:
- dependency-name: org.springframework:spring-expression
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-09-06 02:12:04 +00:00
Alex Osborne 2c9a5c97ab Merge pull request #586 from nla/kryo-5.6.0 
Update to kryo 5.6.0
 2024-09-06 11:10:17 +09:00
Alex Osborne ec57e2a8b1 Merge pull request #605 from bnfleb/extractorHTML 
Improve the HTML Parser for alternative resolution images
 2024-09-06 11:09:45 +09:00
Leslie Bellony ca3508f55f Add new HTML tags and attributes for ExtractorHTML (#604) 2024-09-04 17:49:40 +02:00
Kristinn Sigurðsson 9405cb9af1 Merge pull request #598 from kris-sigur/hostname-queues-with-limits 
Hostname based queue assignment variants that optionally limit queue name length
 2024-09-03 12:25:20 +00:00
Kristinn Sigurðsson 9ee8520604 ConfigurableExtractorJS (#602) 
Adds ConfigurableExtractorJS

Minor modification to ExtractorJS to make subclassing easier
 2024-09-03 12:24:08 +00:00
Alex Osborne 020c6d77b0 Merge pull request #603 from internetarchive/dependabot/maven/commons/dnsjava-dnsjava-3.6.0 
Bump dnsjava:dnsjava from 3.3.1 to 3.6.0 in /commons
 2024-09-03 11:46:15 +09:00
dependabot[bot] 3ec27f2097 Bump dnsjava:dnsjava from 3.3.1 to 3.6.0 in /commons 
Bumps [dnsjava:dnsjava](https://github.com/dnsjava/dnsjava) from 3.3.1 to 3.6.0.
- [Release notes](https://github.com/dnsjava/dnsjava/releases)
- [Changelog](https://github.com/dnsjava/dnsjava/blob/master/Changelog)
- [Commits](https://github.com/dnsjava/dnsjava/compare/v3.3.1...v3.6.0)

---
updated-dependencies:
- dependency-name: dnsjava:dnsjava
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-09-03 01:37:46 +00:00
Alex Osborne c35d72211c Merge pull request #601 from internetarchive/remove-extractor-chrome 
Remove ExtractorChrome
 2024-08-20 11:59:55 +09:00
Adam Miller 2d862c3e28 Merge pull request #600 from internetarchive/adam/restore-has-prerequisite-behavior 
Reset CrawlURI status for hasPrerequisite() so that it isn't preserved between attempts
 2024-08-08 12:05:22 -07:00
Kristinn Sigurðsson cd3a424176 Merge pull request #597 from kris-sigur/extractorhttp-implicit 
Add a more general support for inferred path discovery
 2024-08-08 07:37:45 +00:00
Kristinn Sigurðsson 0faf338f91 Merge pull request #595 from internetarchive/meta-content-name 
Apply speculativeFixup before evaluating meta content
 2024-08-08 07:36:36 +00:00
Alex Osborne 55a179c610 Remove ExtractorChrome 
This never worked that well, is causing random test failures and I don't have any plans to continue developing it.
 2024-08-08 07:21:46 +09:00
Adam Miller ec164ec77f Reset CrawlURI status for hasPrerequisite() so that it isn't preserved between attempts 2024-08-07 10:30:35 -07:00
Adam Miller b22d6ce2e1 Merge pull request #593 from internetarchive/adam/add-crawl-log-logging-to-extractoryoutubedl 
feat: Add logging to crawl.log for metadata records created by ExtractorYoutubeDL
 2024-08-07 10:17:25 -07:00
Alex Osborne 3ae1300a21 Merge pull request #599 from miku/miku/scriptedprocessor-javadoc-fix 
fix: ScriptedProcessor function name in docs
 2024-08-07 17:04:02 +09:00
Martin Czygan 3128ae5b84 fix: ScriptedProcessor function name in docs 2024-07-22 14:15:40 +02:00
Kristinn Sigurðsson 568db3cabc Added queue assignment variants that limit queue name length 2024-07-09 09:36:39 +00:00
Kristinn Sigurðsson 778a57b3b4 Add a more general support for inferred path discovery 2024-07-08 13:05:56 +00:00
Kristinn Sigurðsson 0dd3a2506d Apply speculativeFixup before evaluating meta content 
This avoids treating meta conent values like "Example.com" as relative
urls as they are converted to absolute URLs. This is already done for
speculative JS extraction.

The example sited above is common in meta "sitename" elements where the
sitename is something dot com or similar.
 2024-07-03 11:09:57 +00:00
Adam Miller 59635f00f4 chore: fix contrib dependencies 2024-06-21 23:34:23 +00:00
Adam Miller e64629cddd refactor: simplify sha1 calculation for yt-dlp json content. 2024-06-20 10:59:21 -07:00
Alex Osborne 74d6b37779 Update to kryo 5.6.0 
This eliminates a few more very old dependencies that aren't in Maven Central.

Our direct usage of the unsupported sun.reflect.ReflectionFactory JDK API (which newer compilers complain about) is no longer needed as Kryo now has a SerializingInstantiatorStrategy that does roughly the same thing.
 2024-06-18 10:05:56 +09:00
Alex Osborne 4a103f3e21 Merge pull request #592 from internetarchive/adam/fix-maven-junit3-test-discovery 
fix: maven was not running older junit3 style tests without some extr…
 2024-06-18 09:57:21 +09:00
Adam Miller cf5ccf7895 fix: maven was not running older junit3 style tests without some extra configuration. 2024-06-07 17:07:38 -07:00
Adam Miller 55dc23daaf feat: Add unit tests for ExtractorYoutubeDL WARC record building 2024-06-07 17:03:37 -07:00
Adam Miller d5e2ce6466 Make logging of metadata record optional, but default. 2024-06-06 14:32:45 -07:00
Adam Miller 3f764f7904 feat: Add logging to crawl.log for metadata records created by ExtractorYoutubeDL 2024-06-06 14:21:43 -07:00
Alex Osborne e38cd3528e Merge pull request #589 from internetarchive/jdk22 
JDK 22 support
 2024-05-15 23:17:26 +09:00
Alex Osborne 5424423e7d Add JDK 22 to test matrix 2024-05-15 23:14:51 +09:00
Alex Osborne 0c251c7c34 Update groovy to 4.0.21 for JDK 22 support 2024-05-15 23:13:18 +09:00
Alex Osborne a151937d4c Remove workaround for non-https builds.archive.org repository 
It's now accessible over HTTPS, so we don't need this anymore.
 2024-05-15 21:03:50 +09:00
Alex Osborne 64c7f0fb1c Merge pull request #572 from BulkSecurityGeneratorProjectV2/fix/JLL/use_https_to_resolve_dependencies_maven 
[SECURITY] Use HTTPS to resolve dependencies in Maven Build
 2024-05-15 15:50:07 +09:00
Alex Osborne f81a987110 Merge pull request #583 from internetarchive/dependabot/maven/commons/org.springframework-spring-expression-5.3.27 
Bump org.springframework:spring-expression from 5.3.20 to 5.3.27 in /commons
 2024-04-12 16:00:21 +09:00
dependabot[bot] 7975bf3d7c Bump org.springframework:spring-expression in /commons 
Bumps [org.springframework:spring-expression](https://github.com/spring-projects/spring-framework) from 5.3.20 to 5.3.27.
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v5.3.20...v5.3.27)

---
updated-dependencies:
- dependency-name: org.springframework:spring-expression
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-04-12 06:58:23 +00:00
Alex Osborne b3d276b6d8 Merge pull request #575 from internetarchive/pdfbox 
Switch ExtractorPDF and ExtractorPDFContent to pdfbox
 2024-04-12 15:49:17 +09:00
Barbara Miller 4552509941 Merge pull request #582 from galgeek/fix-yt-dlp-typo-too 2024-03-20 10:22:05 -07:00
Barbara Miller 3996195ab3 update comment 2024-03-20 10:17:34 -07:00
Barbara Miller d748f11ac7 Merge branch 'internetarchive:fix-yt-dlp-typo-too' into fix-yt-dlp-typo-too 2024-03-19 18:39:29 -07:00
Barbara Miller e84e125d94 720p 2024-03-19 12:50:59 -07:00
Barbara Miller a3d1873f42 Merge branch 'master' into fix-yt-dlp-typo-too 2024-03-18 17:26:46 -07:00