lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-20 12:46:00 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,546 Commits 38 Branches 33 Tags
d8aa0902e44346d45a672432df2f4bd21fff5bb5
Commit Graph
2546 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Alex Osborne d8aa0902e4 Merge pull request #617 from internetarchive/kafka-3.8.0 
Bump kafka from 0.9.0.0 to kafka-clients 3.8.0
 2024-10-18 20:30:38 +09:00
Alex Osborne 80b174f1fd Bump kafka from 0.9.0.0 to kafka-clients 3.8.0 2024-10-18 20:06:23 +09:00
Alex Osborne f49dbda2b8 Merge pull request #615 from internetarchive/dependabot/maven/org.eclipse.jetty-jetty-server-9.4.55.v20240627 
Bump org.eclipse.jetty:jetty-server from 9.4.37.v20210219 to 9.4.55.v20240627
 2024-10-17 12:02:09 +09:00
Alex Osborne 1d37cbc237 StatisticsSelfTest: Update for new jetty version 
Jetty error messages are included in the selftest crawl and now have
slightly different lengths.
 2024-10-17 11:58:13 +09:00
dependabot[bot] ececa3b6a3 Bump org.eclipse.jetty:jetty-server 
Bumps org.eclipse.jetty:jetty-server from 9.4.37.v20210219 to 9.4.55.v20240627.

---
updated-dependencies:
- dependency-name: org.eclipse.jetty:jetty-server
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-10-17 02:46:12 +00:00
Alex Osborne 74cabbe798 Merge pull request #614 from internetarchive/remove-builds.archive.org 
Remove dependency on builds.archive.org Maven repository
 2024-10-17 11:40:20 +09:00
Alex Osborne 76cbe9343d Remove dependency on builds.archive.org Maven repository 
Use the IIPC fork of javaswf from Maven Central instead.
https://github.com/iipc/javaswf

#613 #481
 2024-10-17 11:28:53 +09:00
Alex Osborne 891dbe7615 Re-add restlet repository with new URL 
Turns out we do need this. Restlet isn't in central, the CI builds
were caching the artifacts.
 2024-10-15 19:19:28 +09:00
Alex Osborne 8e568187b6 Merge pull request #611 from internetarchive/dependabot/maven/commons/commons-io-commons-io-2.14.0 
Bump commons-io:commons-io from 2.7 to 2.14.0 in /commons
 2024-10-15 18:18:22 +09:00
dependabot[bot] 27259d8b82 Bump commons-io:commons-io from 2.7 to 2.14.0 in /commons 
Bumps commons-io:commons-io from 2.7 to 2.14.0.

---
updated-dependencies:
- dependency-name: commons-io:commons-io
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-10-15 07:35:22 +00:00
Alex Osborne 3b06226509 Merge pull request #610 from internetarchive/remove-restlet-repo 
Remove maven.restlet.com repository
 2024-10-09 12:39:34 +09:00
Alex Osborne cb1143e438 Remove maven.restlet.com repository 
It seems to have moved to maven.restlet.talend.com but the restlet artifacts are in Central anyway so we don't need it.
 2024-10-09 12:29:06 +09:00
Alex Osborne 043dc4e48d Merge pull request #609 from internetarchive/remove-joda-time 
Remove joda-time
 2024-10-09 11:29:15 +09:00
Alex Osborne 3c80dacabe CrawlJob: Replace joda-time with java.time (JSR-310) 2024-09-13 13:47:00 +09:00
Alex Osborne e02187ea3b Arc2Warc: Replace joda-time with java.time (JSR-310) 2024-09-13 13:47:00 +09:00
Alex Osborne 2e891bc4bb Merge pull request #608 from internetarchive/cite-navlinks 
ExtractorHTML: Treat 'cite' attribute as navlink instead of embed
 2024-09-13 13:19:03 +09:00
manderson 3a447b5e61 ExtractorHTML: Treat 'cite' attribute as navlink instead of embed 
The cite attribute is used to identify the source document of a blockquote. But ExtractorHTML was treating it as an embed which can cause out of scope pages to be included in a crawl incorrectly. Browsers don't use the cite attribute currently so there might be an argument for ignoring it entirely but let's at least not treat it as an embed.
 2024-09-10 12:42:15 +09:00
Alex Osborne 791f1c81be Add changelog links for 3.4.0-20240909 2024-09-10 12:27:56 +09:00
Alex Osborne 8ac80be9d3 Add 3.4.0-20240909 to CHANGELOG 2024-09-10 12:24:06 +09:00
Alex Osborne c7c492b507 [maven-release-plugin] prepare for next development iteration 2024-09-09 15:28:44 +09:00
Alex Osborne 11fcbe71f3 [maven-release-plugin] prepare release 3.4.0-20240909 3.4.0-20240909 2024-09-09 15:28:44 +09:00
Alex Osborne ae7a025c35 Merge pull request #607 from internetarchive/dependabot/maven/contrib/com.rabbitmq-amqp-client-5.18.0 
Bump com.rabbitmq:amqp-client from 4.8.0 to 5.18.0 in /contrib
 2024-09-06 11:18:33 +09:00
Alex Osborne ee048f4129 Merge pull request #606 from internetarchive/dependabot/maven/commons/org.springframework-spring-expression-5.3.39 
Bump org.springframework:spring-expression from 5.3.27 to 5.3.39 in /commons
 2024-09-06 11:17:14 +09:00
dependabot[bot] 2cffefc798 Bump com.rabbitmq:amqp-client from 4.8.0 to 5.18.0 in /contrib 
Bumps [com.rabbitmq:amqp-client](https://github.com/rabbitmq/rabbitmq-java-client) from 4.8.0 to 5.18.0.
- [Release notes](https://github.com/rabbitmq/rabbitmq-java-client/releases)
- [Commits](https://github.com/rabbitmq/rabbitmq-java-client/compare/v4.8.0...v5.18.0)

---
updated-dependencies:
- dependency-name: com.rabbitmq:amqp-client
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-09-06 02:13:37 +00:00
dependabot[bot] 09ad15f711 Bump org.springframework:spring-expression in /commons 
Bumps [org.springframework:spring-expression](https://github.com/spring-projects/spring-framework) from 5.3.27 to 5.3.39.
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v5.3.27...v5.3.39)

---
updated-dependencies:
- dependency-name: org.springframework:spring-expression
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-09-06 02:12:04 +00:00
Alex Osborne 2c9a5c97ab Merge pull request #586 from nla/kryo-5.6.0 
Update to kryo 5.6.0
 2024-09-06 11:10:17 +09:00
Alex Osborne ec57e2a8b1 Merge pull request #605 from bnfleb/extractorHTML 
Improve the HTML Parser for alternative resolution images
 2024-09-06 11:09:45 +09:00
Leslie Bellony ca3508f55f Add new HTML tags and attributes for ExtractorHTML (#604) 2024-09-04 17:49:40 +02:00
Kristinn Sigurðsson 9405cb9af1 Merge pull request #598 from kris-sigur/hostname-queues-with-limits 
Hostname based queue assignment variants that optionally limit queue name length
 2024-09-03 12:25:20 +00:00
Kristinn Sigurðsson 9ee8520604 ConfigurableExtractorJS (#602) 
Adds ConfigurableExtractorJS

Minor modification to ExtractorJS to make subclassing easier
 2024-09-03 12:24:08 +00:00
Alex Osborne 020c6d77b0 Merge pull request #603 from internetarchive/dependabot/maven/commons/dnsjava-dnsjava-3.6.0 
Bump dnsjava:dnsjava from 3.3.1 to 3.6.0 in /commons
 2024-09-03 11:46:15 +09:00
dependabot[bot] 3ec27f2097 Bump dnsjava:dnsjava from 3.3.1 to 3.6.0 in /commons 
Bumps [dnsjava:dnsjava](https://github.com/dnsjava/dnsjava) from 3.3.1 to 3.6.0.
- [Release notes](https://github.com/dnsjava/dnsjava/releases)
- [Changelog](https://github.com/dnsjava/dnsjava/blob/master/Changelog)
- [Commits](https://github.com/dnsjava/dnsjava/compare/v3.3.1...v3.6.0)

---
updated-dependencies:
- dependency-name: dnsjava:dnsjava
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-09-03 01:37:46 +00:00
Alex Osborne c35d72211c Merge pull request #601 from internetarchive/remove-extractor-chrome 
Remove ExtractorChrome
 2024-08-20 11:59:55 +09:00
Adam Miller 2d862c3e28 Merge pull request #600 from internetarchive/adam/restore-has-prerequisite-behavior 
Reset CrawlURI status for hasPrerequisite() so that it isn't preserved between attempts
 2024-08-08 12:05:22 -07:00
Kristinn Sigurðsson cd3a424176 Merge pull request #597 from kris-sigur/extractorhttp-implicit 
Add a more general support for inferred path discovery
 2024-08-08 07:37:45 +00:00
Kristinn Sigurðsson 0faf338f91 Merge pull request #595 from internetarchive/meta-content-name 
Apply speculativeFixup before evaluating meta content
 2024-08-08 07:36:36 +00:00
Alex Osborne 55a179c610 Remove ExtractorChrome 
This never worked that well, is causing random test failures and I don't have any plans to continue developing it.
 2024-08-08 07:21:46 +09:00
Adam Miller ec164ec77f Reset CrawlURI status for hasPrerequisite() so that it isn't preserved between attempts 2024-08-07 10:30:35 -07:00
Adam Miller b22d6ce2e1 Merge pull request #593 from internetarchive/adam/add-crawl-log-logging-to-extractoryoutubedl 
feat: Add logging to crawl.log for metadata records created by ExtractorYoutubeDL
 2024-08-07 10:17:25 -07:00
Alex Osborne 3ae1300a21 Merge pull request #599 from miku/miku/scriptedprocessor-javadoc-fix 
fix: ScriptedProcessor function name in docs
 2024-08-07 17:04:02 +09:00
Martin Czygan 3128ae5b84 fix: ScriptedProcessor function name in docs 2024-07-22 14:15:40 +02:00
Kristinn Sigurðsson 568db3cabc Added queue assignment variants that limit queue name length 2024-07-09 09:36:39 +00:00
Kristinn Sigurðsson 778a57b3b4 Add a more general support for inferred path discovery 2024-07-08 13:05:56 +00:00
Kristinn Sigurðsson 0dd3a2506d Apply speculativeFixup before evaluating meta content 
This avoids treating meta conent values like "Example.com" as relative
urls as they are converted to absolute URLs. This is already done for
speculative JS extraction.

The example sited above is common in meta "sitename" elements where the
sitename is something dot com or similar.
 2024-07-03 11:09:57 +00:00
Adam Miller 59635f00f4 chore: fix contrib dependencies 2024-06-21 23:34:23 +00:00
Adam Miller e64629cddd refactor: simplify sha1 calculation for yt-dlp json content. 2024-06-20 10:59:21 -07:00
Alex Osborne 74d6b37779 Update to kryo 5.6.0 
This eliminates a few more very old dependencies that aren't in Maven Central.

Our direct usage of the unsupported sun.reflect.ReflectionFactory JDK API (which newer compilers complain about) is no longer needed as Kryo now has a SerializingInstantiatorStrategy that does roughly the same thing.
 2024-06-18 10:05:56 +09:00
Alex Osborne 4a103f3e21 Merge pull request #592 from internetarchive/adam/fix-maven-junit3-test-discovery 
fix: maven was not running older junit3 style tests without some extr…
 2024-06-18 09:57:21 +09:00
Adam Miller cf5ccf7895 fix: maven was not running older junit3 style tests without some extra configuration. 2024-06-07 17:07:38 -07:00
Adam Miller 55dc23daaf feat: Add unit tests for ExtractorYoutubeDL WARC record building 2024-06-07 17:03:37 -07:00