JavaDoc comments need to be directly above a class, method or field
declaration to be used by IDEs and generated documentation. We had some
doc comments that were attached to instance initialization blocks and
thus were being ignored. This invalid doc comment positioning was
presumably an accidental consequence of the conversion of fields to
KeyedProperties.
This change moves most of the dangling doc comments to setter methods
where it can be seen by tools. There were a couple of dangling doc
comments above package statements. These are moved above the class
declaration or removed entirely when empty.
This enables us to generate documentation for more bean properties,
although a number of beans have javadoc on initializer code blocks which
makes it hard to access. This affects javadoc and IDE contextual
documentation too so should probably be fixed in the source code itself.
This is meant to complement the javadoc by providing a reference
more suitable for users trying to configure crawls rather than
developers writing new modules.
The doc generation could still do with some improving and some of the
source javadoc comments need fixing up but this is already useful so I'm
committing what I have so far.
Large portions of the docs were first published much earlier in other
places and some have been updated since then. Rather than trying to
keep a date range up to date let's just remove the date. We don't use
dates in the source code boilerplate and my understanding is copyright
notices are not mandatory in almost all countries due to the Berne
Convention anyway so it's really just informational.
This adds a `captureRequests` flag to ExtractorChrome which is enabled
by default and causes requests made by the browser to be captured via
the devtools Network domain. Captured browser requests are sent to the
disposition chain for WARC writing and also to statistics tracker and
crawl log.
Browser requests are given the annotation "browser" so they can be
easily distinguished in the log from normal requests.
There are a quite a few limitations that will be addressed in followup
work:
* The frontier is entirely unaware of browser requests. This means they
bypass quotas, ignore scope rules, politeness and don't count towards
the statistics tracked by the frontier itself.
* There's no replay of previously saved resources so duplicate requests
for the same URL end up being made.
* Heritrix's extractors do not currently process browser requests.
* Various error and failure cases likely need improving.
The vast majority of protocols we use will have a server IP address,
so rather than having a separate one for each protocol let's just use
a single common one.
Fixes the Maven warnings:
[WARNING] The artifact poi:poi:jar:2.5.1 has been relocated to
poi:poi:jar:2.5.1-final-20040804
[WARNING] The artifact itext:itext:jar:1.3 has been relocated to
com.lowagie:itext:jar:1.3
Maven 2.0.9 was 13 years ago. I think we can safely assume everyone has
a newer maven now. ;-)
This stops Maven printing this warning:
[WARNING] The project org.archive:heritrix:pom:3.4.0-SNAPSHOT uses
prerequisites which is only intended for maven-plugin projects but
not for non maven-plugin projects. For such purposes you should use
the maven-enforcer-plugin.
I noticed there was a non-functional and undocumented -r command-line
option which automatically runs a given job when Heritrix starts. I
suspect this was never finished due to the bug that was fixed in commit
643f16d where launch() returns before the job has been launched.
This option seems like it would be very useful so you could run a job
from cron or another scheduling program without having to use the
REST API to start it. Therefore I've enabled it and extended it so it
also unpauses, waits for the crawl to finish and then exits.
I think the sleep is supposed to make launch() not return until the job
has actually been launched but it doesn't work as launch()
and getCrawlController() are both synchronized therefore the
launcher thread can't actually call startContext() until launch()
returns after sleeping.
So let's replace the sleep call with join and unsynchronize launch()
so it doesn't deadlock. All the relevant methods it calls seem to be
synchronized so I think it's no worse to not synchronize it itself.
Lots of 1 second sleeps add up fast. On my PC this reduces the runtime
of `mvn clean test` from about 3m 40s to about 1m 50s.
Ideally we'd probably re-architect some this to use a notification
mechanism instead of polling but that's easier said than done and this
is a pretty big improvement by itself.
I think the sleep is supposed to make launch() not return until the job
has actually been launched but it doesn't work as launch()
and getCrawlController() are both synchronized therefore the
launcher thread can't actually call startContext() until launch()
returns after sleeping.
So let's replace the sleep call with join and unsynchronize launch()
so it doesn't deadlock. All the relevant methods it calls seem to be
synchronized so I think it's no worse to not synchronize it itself.
The CI builds are no longer working as travis-ci.org has been
discontinued so switch to using Github Actions instead.
We need to use a custom settings.xml for now as Maven now blocks
non-https repositories by default and builds.archive.org isn't
available via https.
Alex Osborne