This adds a `captureRequests` flag to ExtractorChrome which is enabled
by default and causes requests made by the browser to be captured via
the devtools Network domain. Captured browser requests are sent to the
disposition chain for WARC writing and also to statistics tracker and
crawl log.
Browser requests are given the annotation "browser" so they can be
easily distinguished in the log from normal requests.
There are a quite a few limitations that will be addressed in followup
work:
* The frontier is entirely unaware of browser requests. This means they
bypass quotas, ignore scope rules, politeness and don't count towards
the statistics tracked by the frontier itself.
* There's no replay of previously saved resources so duplicate requests
for the same URL end up being made.
* Heritrix's extractors do not currently process browser requests.
* Various error and failure cases likely need improving.
The vast majority of protocols we use will have a server IP address,
so rather than having a separate one for each protocol let's just use
a single common one.
Fixes the Maven warnings:
[WARNING] The artifact poi:poi:jar:2.5.1 has been relocated to
poi:poi:jar:2.5.1-final-20040804
[WARNING] The artifact itext:itext:jar:1.3 has been relocated to
com.lowagie:itext:jar:1.3
Maven 2.0.9 was 13 years ago. I think we can safely assume everyone has
a newer maven now. ;-)
This stops Maven printing this warning:
[WARNING] The project org.archive:heritrix:pom:3.4.0-SNAPSHOT uses
prerequisites which is only intended for maven-plugin projects but
not for non maven-plugin projects. For such purposes you should use
the maven-enforcer-plugin.
I noticed there was a non-functional and undocumented -r command-line
option which automatically runs a given job when Heritrix starts. I
suspect this was never finished due to the bug that was fixed in commit
643f16d where launch() returns before the job has been launched.
This option seems like it would be very useful so you could run a job
from cron or another scheduling program without having to use the
REST API to start it. Therefore I've enabled it and extended it so it
also unpauses, waits for the crawl to finish and then exits.
I think the sleep is supposed to make launch() not return until the job
has actually been launched but it doesn't work as launch()
and getCrawlController() are both synchronized therefore the
launcher thread can't actually call startContext() until launch()
returns after sleeping.
So let's replace the sleep call with join and unsynchronize launch()
so it doesn't deadlock. All the relevant methods it calls seem to be
synchronized so I think it's no worse to not synchronize it itself.
Lots of 1 second sleeps add up fast. On my PC this reduces the runtime
of `mvn clean test` from about 3m 40s to about 1m 50s.
Ideally we'd probably re-architect some this to use a notification
mechanism instead of polling but that's easier said than done and this
is a pretty big improvement by itself.
I think the sleep is supposed to make launch() not return until the job
has actually been launched but it doesn't work as launch()
and getCrawlController() are both synchronized therefore the
launcher thread can't actually call startContext() until launch()
returns after sleeping.
So let's replace the sleep call with join and unsynchronize launch()
so it doesn't deadlock. All the relevant methods it calls seem to be
synchronized so I think it's no worse to not synchronize it itself.
The CI builds are no longer working as travis-ci.org has been
discontinued so switch to using Github Actions instead.
We need to use a custom settings.xml for now as Maven now blocks
non-https repositories by default and builds.archive.org isn't
available via https.
This adds a link extractor that uses a headless browser via the Chrome
Devtools Protocol. It works with Chromium and Google Chrome and
while untested likely also other Blink-based browsers such as Brave and
Microsoft Edge. It also works with Firefox which implements a subset of
the protocol.
In addition to ExtractorChrome itself this adds a couple of classes
under org.archive.net.chrome for communicating with the browser:
* ChromeProcess - manages the lifecycle of the browser child process
* ChromeClient - client for the Chrome Devtools Protocol
* ChromeWindow - client for controlling a particular browser window
Why this extractor when we already have Umbra? The primary goal is to
have something that works out of the box without any additional
infrastructure beyond having a browser installed. A longer term goal is
to be able to take advantage of tighter integration between Heritrix
and the browser to do things like recording browser subrequests to WARC
files or running ExtractorHTML against the loaded DOM.
While this extractor is already usable it is currently quite bare bones
and only extracts links from <a> and <area> tags. The intent is to
expand it with more functionality over time with features like
intercepting subrequests, reusing previously saved resources, taking
screenshots or running JavaScript behaviours.
This eliminates one of the jars that Heritrix depends on that isn't in
Maven Central.
There were only two usages. The first usage was a method in a
CrawlerJournal which as far as I can tell is unused and can be removed.
The second usage was an unnecessary use of MutableString as a line
buffer when saving cookies to a file. Since there doesn't seem any
particular reason buffer per-line here let's just use a BufferedWriter
instead on the whole file which is also slightly simpler and more
efficient.
Alex Osborne