Spring 6 removed @Required and they suggest using constructor injection instead. If we switched our beans to that we'd break existing Heritrix crawl configs. So this change implements our own basic version so we still get errors when a @Required property is null.
Adds a list of regular expressions that URLs being processed by the
ConfigurableExtractorJS are evaluted against. If they match the
extraction is performed in strict mode, even if strict mode is not set.
This requires a minor modification to ExtractorJS so that the CrawlURI
is passed to the shouldAddUri method that ConfigurableExtractorJS
overrides.
Links from manifests (e.g. sitemaps) should not receive the preferential
treatment sometimes accorded to "transitive" hops. Most commonly this is
about giving priority to discovered (probable) embeds.
Manifests should be regarded as more analogous with a directory page.
littleproxy has not been updated since 2017 and has various out of date
dependencies. Since we're using jetty for HTTP server testing anyway, we
may as well use it as the proxy too.
The cite attribute is used to identify the source document of a blockquote. But ExtractorHTML was treating it as an embed which can cause out of scope pages to be included in a crawl incorrectly. Browsers don't use the cite attribute currently so there might be an argument for ignoring it entirely but let's at least not treat it as an embed.
This avoids treating meta conent values like "Example.com" as relative
urls as they are converted to absolute URLs. This is already done for
speculative JS extraction.
The example sited above is common in meta "sitename" elements where the
sitename is something dot com or similar.
This eliminates a few more very old dependencies that aren't in Maven Central.
Our direct usage of the unsupported sun.reflect.ReflectionFactory JDK API (which newer compilers complain about) is no longer needed as Kryo now has a SerializingInstantiatorStrategy that does roughly the same thing.
This policy obeys robots.txt but ignores meta nofollow. We're
increasingly frequently encountering sites which fail to be archived
properly due to usage of this meta tag but where the robots.txt file
contains reasonable rules. In many cases it seems to be unintentional,
and we've even had questions from webmasters asking why archiving was
failing when they were explicitly allowing us in robots.txt.
Alex Osborne