lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-16 18:55:35 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,560 Commits 39 Branches 33 Tags
345e927d3208301b030ecaae0ff4c4e7c2e64e39
Commit Graph
2560 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Alex Osborne 345e927d32 Remove HBase from contrib 
The transitive dependencies pulled in via hbase are causing many security alerts for unpatched vulnerabilities in Dependabot, Maven Central and other tools. The newest version of hbase-client these modules seem to compile against still has many vulnerable transitive dependencies and the latest version of HBase appears to have many breaking API changes.

These can be re-added if someone volunteers to take over maintaining this code and updates it so that it doesn't pull in vulnerable dependencies.
 2024-10-25 14:33:26 +09:00
Alex Osborne ff84c8d32f Bump jetty from 9.4.55.v20240627 to 9.4.56.v20240826 2024-10-18 22:17:25 +09:00
Alex Osborne 3c649614ba Avoid old versions of commons-io and commons-collections from hbase 2024-10-18 22:14:51 +09:00
Alex Osborne c25e57a4a1 Bump version to 3.4.1-SNAPSHOT 2024-10-18 20:54:12 +09:00
Alex Osborne b0fad42e25 Merge pull request #619 from internetarchive/jdk23 
JDK 23 support
 2024-10-18 20:51:08 +09:00
Alex Osborne 070063ce0a Merge pull request #618 from internetarchive/ftpserver-core-1.2.0 
Bump ftpserver-core from 1.1.1 to 1.2.0
 2024-10-18 20:45:33 +09:00
Alex Osborne 5cfa5f02df Update CI tests from JDK 22 to 23 2024-10-18 20:45:11 +09:00
Alex Osborne 7a3d29ad4d Bump ftpserver-core from 1.1.1 to 1.2.0 2024-10-18 20:38:49 +09:00
Alex Osborne 6157645545 Merge pull request #612 from internetarchive/config-extractor-js-default-strict 
Set default value for ConfigurableExtractorJS strict property
 2024-10-18 20:32:58 +09:00
Alex Osborne 33f009516f Merge pull request #616 from internetarchive/webarchive-commons-1.1.10 
Remove log4j from main Heritrix distribution
 2024-10-18 20:32:42 +09:00
Alex Osborne d8aa0902e4 Merge pull request #617 from internetarchive/kafka-3.8.0 
Bump kafka from 0.9.0.0 to kafka-clients 3.8.0
 2024-10-18 20:30:38 +09:00
Alex Osborne 80b174f1fd Bump kafka from 0.9.0.0 to kafka-clients 3.8.0 2024-10-18 20:06:23 +09:00
Alex Osborne cbd7f37ed9 Replace littleproxy with jetty-proxy (to remove dependency on log4j) 
littleproxy has not been updated since 2017 and has various out of date
dependencies. Since we're using jetty for HTTP server testing anyway, we
 may as well use it as the proxy too.
 2024-10-18 17:28:03 +09:00
Alex Osborne 3d1b7e37ca Remove log4j 2024-10-17 19:25:53 +09:00
Alex Osborne c797985590 Bump webarchive-commons from 1.1.8 to 1.1.10 
We need to add log4j as a dependency temporarily as webarchive-commons
no longer pulls it in.
 2024-10-17 19:25:53 +09:00
Alex Osborne f49dbda2b8 Merge pull request #615 from internetarchive/dependabot/maven/org.eclipse.jetty-jetty-server-9.4.55.v20240627 
Bump org.eclipse.jetty:jetty-server from 9.4.37.v20210219 to 9.4.55.v20240627
 2024-10-17 12:02:09 +09:00
Alex Osborne 1d37cbc237 StatisticsSelfTest: Update for new jetty version 
Jetty error messages are included in the selftest crawl and now have
slightly different lengths.
 2024-10-17 11:58:13 +09:00
dependabot[bot] ececa3b6a3 Bump org.eclipse.jetty:jetty-server 
Bumps org.eclipse.jetty:jetty-server from 9.4.37.v20210219 to 9.4.55.v20240627.

---
updated-dependencies:
- dependency-name: org.eclipse.jetty:jetty-server
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-10-17 02:46:12 +00:00
Alex Osborne 74cabbe798 Merge pull request #614 from internetarchive/remove-builds.archive.org 
Remove dependency on builds.archive.org Maven repository
 2024-10-17 11:40:20 +09:00
Alex Osborne 76cbe9343d Remove dependency on builds.archive.org Maven repository 
Use the IIPC fork of javaswf from Maven Central instead.
https://github.com/iipc/javaswf

#613 #481
 2024-10-17 11:28:53 +09:00
Kristinn Sigurðsson af280584f0 Set default value for strict. 2024-10-16 09:25:02 +00:00
Alex Osborne 891dbe7615 Re-add restlet repository with new URL 
Turns out we do need this. Restlet isn't in central, the CI builds
were caching the artifacts.
 2024-10-15 19:19:28 +09:00
Alex Osborne 8e568187b6 Merge pull request #611 from internetarchive/dependabot/maven/commons/commons-io-commons-io-2.14.0 
Bump commons-io:commons-io from 2.7 to 2.14.0 in /commons
 2024-10-15 18:18:22 +09:00
dependabot[bot] 27259d8b82 Bump commons-io:commons-io from 2.7 to 2.14.0 in /commons 
Bumps commons-io:commons-io from 2.7 to 2.14.0.

---
updated-dependencies:
- dependency-name: commons-io:commons-io
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-10-15 07:35:22 +00:00
Alex Osborne 3b06226509 Merge pull request #610 from internetarchive/remove-restlet-repo 
Remove maven.restlet.com repository
 2024-10-09 12:39:34 +09:00
Alex Osborne cb1143e438 Remove maven.restlet.com repository 
It seems to have moved to maven.restlet.talend.com but the restlet artifacts are in Central anyway so we don't need it.
 2024-10-09 12:29:06 +09:00
Alex Osborne 043dc4e48d Merge pull request #609 from internetarchive/remove-joda-time 
Remove joda-time
 2024-10-09 11:29:15 +09:00
Alex Osborne 3c80dacabe CrawlJob: Replace joda-time with java.time (JSR-310) 2024-09-13 13:47:00 +09:00
Alex Osborne e02187ea3b Arc2Warc: Replace joda-time with java.time (JSR-310) 2024-09-13 13:47:00 +09:00
Alex Osborne 2e891bc4bb Merge pull request #608 from internetarchive/cite-navlinks 
ExtractorHTML: Treat 'cite' attribute as navlink instead of embed
 2024-09-13 13:19:03 +09:00
manderson 3a447b5e61 ExtractorHTML: Treat 'cite' attribute as navlink instead of embed 
The cite attribute is used to identify the source document of a blockquote. But ExtractorHTML was treating it as an embed which can cause out of scope pages to be included in a crawl incorrectly. Browsers don't use the cite attribute currently so there might be an argument for ignoring it entirely but let's at least not treat it as an embed.
 2024-09-10 12:42:15 +09:00
Alex Osborne 791f1c81be Add changelog links for 3.4.0-20240909 2024-09-10 12:27:56 +09:00
Alex Osborne 8ac80be9d3 Add 3.4.0-20240909 to CHANGELOG 2024-09-10 12:24:06 +09:00
Alex Osborne c7c492b507 [maven-release-plugin] prepare for next development iteration 2024-09-09 15:28:44 +09:00
Alex Osborne 11fcbe71f3 [maven-release-plugin] prepare release 3.4.0-20240909 3.4.0-20240909 2024-09-09 15:28:44 +09:00
Alex Osborne ae7a025c35 Merge pull request #607 from internetarchive/dependabot/maven/contrib/com.rabbitmq-amqp-client-5.18.0 
Bump com.rabbitmq:amqp-client from 4.8.0 to 5.18.0 in /contrib
 2024-09-06 11:18:33 +09:00
Alex Osborne ee048f4129 Merge pull request #606 from internetarchive/dependabot/maven/commons/org.springframework-spring-expression-5.3.39 
Bump org.springframework:spring-expression from 5.3.27 to 5.3.39 in /commons
 2024-09-06 11:17:14 +09:00
dependabot[bot] 2cffefc798 Bump com.rabbitmq:amqp-client from 4.8.0 to 5.18.0 in /contrib 
Bumps [com.rabbitmq:amqp-client](https://github.com/rabbitmq/rabbitmq-java-client) from 4.8.0 to 5.18.0.
- [Release notes](https://github.com/rabbitmq/rabbitmq-java-client/releases)
- [Commits](https://github.com/rabbitmq/rabbitmq-java-client/compare/v4.8.0...v5.18.0)

---
updated-dependencies:
- dependency-name: com.rabbitmq:amqp-client
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-09-06 02:13:37 +00:00
dependabot[bot] 09ad15f711 Bump org.springframework:spring-expression in /commons 
Bumps [org.springframework:spring-expression](https://github.com/spring-projects/spring-framework) from 5.3.27 to 5.3.39.
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v5.3.27...v5.3.39)

---
updated-dependencies:
- dependency-name: org.springframework:spring-expression
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-09-06 02:12:04 +00:00
Alex Osborne 2c9a5c97ab Merge pull request #586 from nla/kryo-5.6.0 
Update to kryo 5.6.0
 2024-09-06 11:10:17 +09:00
Alex Osborne ec57e2a8b1 Merge pull request #605 from bnfleb/extractorHTML 
Improve the HTML Parser for alternative resolution images
 2024-09-06 11:09:45 +09:00
Leslie Bellony ca3508f55f Add new HTML tags and attributes for ExtractorHTML (#604) 2024-09-04 17:49:40 +02:00
Kristinn Sigurðsson 9405cb9af1 Merge pull request #598 from kris-sigur/hostname-queues-with-limits 
Hostname based queue assignment variants that optionally limit queue name length
 2024-09-03 12:25:20 +00:00
Kristinn Sigurðsson 9ee8520604 ConfigurableExtractorJS (#602) 
Adds ConfigurableExtractorJS

Minor modification to ExtractorJS to make subclassing easier
 2024-09-03 12:24:08 +00:00
Alex Osborne 020c6d77b0 Merge pull request #603 from internetarchive/dependabot/maven/commons/dnsjava-dnsjava-3.6.0 
Bump dnsjava:dnsjava from 3.3.1 to 3.6.0 in /commons
 2024-09-03 11:46:15 +09:00
dependabot[bot] 3ec27f2097 Bump dnsjava:dnsjava from 3.3.1 to 3.6.0 in /commons 
Bumps [dnsjava:dnsjava](https://github.com/dnsjava/dnsjava) from 3.3.1 to 3.6.0.
- [Release notes](https://github.com/dnsjava/dnsjava/releases)
- [Changelog](https://github.com/dnsjava/dnsjava/blob/master/Changelog)
- [Commits](https://github.com/dnsjava/dnsjava/compare/v3.3.1...v3.6.0)

---
updated-dependencies:
- dependency-name: dnsjava:dnsjava
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-09-03 01:37:46 +00:00
Alex Osborne c35d72211c Merge pull request #601 from internetarchive/remove-extractor-chrome 
Remove ExtractorChrome
 2024-08-20 11:59:55 +09:00
Adam Miller 2d862c3e28 Merge pull request #600 from internetarchive/adam/restore-has-prerequisite-behavior 
Reset CrawlURI status for hasPrerequisite() so that it isn't preserved between attempts
 2024-08-08 12:05:22 -07:00
Kristinn Sigurðsson cd3a424176 Merge pull request #597 from kris-sigur/extractorhttp-implicit 
Add a more general support for inferred path discovery
 2024-08-08 07:37:45 +00:00
Kristinn Sigurðsson 0faf338f91 Merge pull request #595 from internetarchive/meta-content-name 
Apply speculativeFixup before evaluating meta content
 2024-08-08 07:36:36 +00:00