When enabled this option causes regular links annotated with rel=nofollow to not be extracted. This is useful for sites that use rel=nofollow to hint crawler traps.
fastutil is our largest dependency, consuming about a third of the
total Heritrix distribution size but we only use a couple of trivial
classes from it.
FPMergeUriUniqFilter (which I'm not sure anyone uses anyway), uses
LongArrayList so this change replaces it with a basic version that does
just enough.
The unsynchronized FastBufferedOutputStream usages are likely
unnecessary these days thanks to the JVM's lock optimisations and for
the one in CrawlerJournal, the GZIPOutputStream is still going to
be synchronizing anyway.
Spring 6 removed @Required and they suggest using constructor injection instead. If we switched our beans to that we'd break existing Heritrix crawl configs. So this change implements our own basic version so we still get errors when a @Required property is null.
Adds a list of regular expressions that URLs being processed by the
ConfigurableExtractorJS are evaluted against. If they match the
extraction is performed in strict mode, even if strict mode is not set.
This requires a minor modification to ExtractorJS so that the CrawlURI
is passed to the shouldAddUri method that ConfigurableExtractorJS
overrides.
Links from manifests (e.g. sitemaps) should not receive the preferential
treatment sometimes accorded to "transitive" hops. Most commonly this is
about giving priority to discovered (probable) embeds.
Manifests should be regarded as more analogous with a directory page.
littleproxy has not been updated since 2017 and has various out of date
dependencies. Since we're using jetty for HTTP server testing anyway, we
may as well use it as the proxy too.
The cite attribute is used to identify the source document of a blockquote. But ExtractorHTML was treating it as an embed which can cause out of scope pages to be included in a crawl incorrectly. Browsers don't use the cite attribute currently so there might be an argument for ignoring it entirely but let's at least not treat it as an embed.
This avoids treating meta conent values like "Example.com" as relative
urls as they are converted to absolute URLs. This is already done for
speculative JS extraction.
The example sited above is common in meta "sitename" elements where the
sitename is something dot com or similar.
This eliminates a few more very old dependencies that aren't in Maven Central.
Our direct usage of the unsupported sun.reflect.ReflectionFactory JDK API (which newer compilers complain about) is no longer needed as Kryo now has a SerializingInstantiatorStrategy that does roughly the same thing.
This policy obeys robots.txt but ignores meta nofollow. We're
increasingly frequently encountering sites which fail to be archived
properly due to usage of this meta tag but where the robots.txt file
contains reasonable rules. In many cases it seems to be unintentional,
and we've even had questions from webmasters asking why archiving was
failing when they were explicitly allowing us in robots.txt.
This ensures that when we later compare the context in processEmbed()
we don't need to deal with variants like srcSet or SRCSET. Note that
we're already sometimes lowercasing it later in HTMLLinkContext.get().
Fixes#477.
- instead of "borrowing" the configured proxy from
the fetchHttp bean, use proxy values defined via
global options, to avoid interference with other
jobs running in parallel (or at least make them
explicit).
The "fetchHttp" bean also uses these settings,
if no bean specific settings are used.
- remove the "enableDnsOverHttpResolves", and rely
on a non-empty "dnsOverHttpServer" value to signal
that DoH should be used.
- use the "DohResolver" from the dnsjava library
to make DoH lookups
- to enable and configure it, add two new
properties
* "enableDnsOverHttpResolves" (boolean)
* "dnsOverHttpServer" URL to the DoH Server
- as one use case for DoH is being located
behind a firewall, also support using a proxy
to access the DoH server; the proxy from
the FetchHTTP bean is reused in that case
Fixes#211
We should only crawl sitemaps and sitemap links if they are in the
primary SURT scope. Otherwise we'll start crawling the sitemaps of
every site an embedded resource is pulled in from even when they
should be out of scope.
Fixes#469
testSocksProxy() is failing on the GitHub action CI runs but I can't
reproduce the issue locally. I'm just guessing this is what's causing it
but this should be fixed anyway to avoid future intermittent failures.
By replacing the custom TestSuite with junit 4 @BeforeClass and
@AfterClass annotations running individual tests by themselves now works
correctly.
Since FetchHTTPTest now just contains the utility code for the test
servers we rename it to FetchHTTPTestServers. FetchHTTPTests (which
contains the actual tests) can now follow the standard naming
scheme as FetchHTTPTest.
Only treat icons, stylesheets and (tentatively) resource preloads as
embedded resources. Treat most other link types as navigation links.
Previously we were treating all link tags as embedded resources
including links like 'next', 'prev', 'canonical', 'author' which could
pull in resources that should normally be considered out of scope.
For example crawling a page containing an author link lke:
<link href="http://twitter.com/example" rel="author">
Would end up transitively pulling 200+ MB of Twitter profile pages in
multiple languages into a crawl for which twitter.com is not even
supposed to be in scope at all.
Fixes#263
- do not send the scheme/host/port part of the request
even when using a proxy, if the scheme is https
- in case of a https request send over a proxy, do not
wrap the first input/output streams (which only contain
the data for the `CONNECT` request), but the ones after
them, which contain the actual data
Fixes#191
If a null character appears in the Content-Length header Firefox and
Chrome seem to ignore the rest of the value so let's do the same.
Let's also handle the exception when parsing the content-length fails.
Fixes#449
Alex Osborne