When enabled this option causes regular links annotated with rel=nofollow to not be extracted. This is useful for sites that use rel=nofollow to hint crawler traps.
fastutil is our largest dependency, consuming about a third of the
total Heritrix distribution size but we only use a couple of trivial
classes from it.
FPMergeUriUniqFilter (which I'm not sure anyone uses anyway), uses
LongArrayList so this change replaces it with a basic version that does
just enough.
The unsynchronized FastBufferedOutputStream usages are likely
unnecessary these days thanks to the JVM's lock optimisations and for
the one in CrawlerJournal, the GZIPOutputStream is still going to
be synchronizing anyway.
This enables crawl configuration files to use Spring's [Groovy Bean Definition DSL] as an optional alternative to Spring XML. It uses the same bean configuration model but the syntax is more terse and human-readable. No more need for `&` in seed URLs. :-)
```groovy
checkpointService(CheckpointService) {
checkpointIntervalMinutes = 15
checkpointsDir = 'checkpoints'
forgetAllButLatest = true
}
```
It also enables some powerful scripting capabilities. For example, defining a custom DecideRule directly in the crawl scope:
```groovy
scope(DecideRuleSequence) {
rules = [
new RejectDecideRule(),
// ACCEPT everything linked from a .pdf file
new PredicatedDecideRule() {
boolean evaluate(CrawlURI uri) {
return uri.via?.path?.endsWith(".pdf")
}
},
// ...
]
}
```
The main downsides are defining nested inner beans can be a bit awkward, some of the errors can be cryptic, and you can't just manipulate the config files with an XML parser.
This commit includes a Groovy version of the default crawl profile for reference, but doesn't expose a way to use it in the UI yet. For now, you need to manually create a `crawler-beans.groovy` file in your job directory.
[Groovy Bean Definition DSL]: https://docs.spring.io/spring-framework/reference/core/beans/basics.html#beans-factory-groovy
This enables a checkpoint to be automatically created during a graceful termination. This makes it easier to stop and restart Heritrix without having to manually checkpoint each running job.
Spring 6 removed @Required and they suggest using constructor injection instead. If we switched our beans to that we'd break existing Heritrix crawl configs. So this change implements our own basic version so we still get errors when a @Required property is null.
littleproxy has not been updated since 2017 and has various out of date
dependencies. Since we're using jetty for HTTP server testing anyway, we
may as well use it as the proxy too.
This eliminates a few more very old dependencies that aren't in Maven Central.
Our direct usage of the unsupported sun.reflect.ReflectionFactory JDK API (which newer compilers complain about) is no longer needed as Kryo now has a SerializingInstantiatorStrategy that does roughly the same thing.
The digest authentication method requires an initial request to get a challenge to use for the subsequent authenmtication digest. This initial request is not an authentication attempt and should not be handled as a failed such.
The initial sleepMs value will be the negative of the current time minus 6 seconds. Using that value to calculate lastFailureTime will increase that large negative number with 6000 milliseconds for every failure which means that rate limiting won't occur until after millions of failed attempts.
Alex Osborne