SerializingInstantiatorStrategy when compiled for Java 17 fails when running without `--add-opens java.base/java.io=ALL-UNNAMED`. StdInstantiatorStrategy seems to work OK though.
This is needed for Spring 6.1 to be able to reflect on parameter names. We have some autowiring that relies on parameter names such as `AbstractFrontier.setScope(DecideRule scope)`. Without this Spring doesn't know whether to inject the `scope` or `acceptSurts` beans as they both subclass DecideRule. Apparently older versions of Spring used to discover method names by parsing bytecode but this was mechanism removed in 6.1.
This enables a checkpoint to be automatically created during a graceful termination. This makes it easier to stop and restart Heritrix without having to manually checkpoint each running job.
Spring 6 removed @Required and they suggest using constructor injection instead. If we switched our beans to that we'd break existing Heritrix crawl configs. So this change implements our own basic version so we still get errors when a @Required property is null.
Adds a list of regular expressions that URLs being processed by the
ConfigurableExtractorJS are evaluted against. If they match the
extraction is performed in strict mode, even if strict mode is not set.
This requires a minor modification to ExtractorJS so that the CrawlURI
is passed to the shouldAddUri method that ConfigurableExtractorJS
overrides.
Links from manifests (e.g. sitemaps) should not receive the preferential
treatment sometimes accorded to "transitive" hops. Most commonly this is
about giving priority to discovered (probable) embeds.
Manifests should be regarded as more analogous with a directory page.
The transitive dependencies pulled in via hbase are causing many security alerts for unpatched vulnerabilities in Dependabot, Maven Central and other tools. The newest version of hbase-client these modules seem to compile against still has many vulnerable transitive dependencies and the latest version of HBase appears to have many breaking API changes.
These can be re-added if someone volunteers to take over maintaining this code and updates it so that it doesn't pull in vulnerable dependencies.
littleproxy has not been updated since 2017 and has various out of date
dependencies. Since we're using jetty for HTTP server testing anyway, we
may as well use it as the proxy too.
Alex Osborne