I noticed there was a non-functional and undocumented -r command-line
option which automatically runs a given job when Heritrix starts. I
suspect this was never finished due to the bug that was fixed in commit
643f16d where launch() returns before the job has been launched.
This option seems like it would be very useful so you could run a job
from cron or another scheduling program without having to use the
REST API to start it. Therefore I've enabled it and extended it so it
also unpauses, waits for the crawl to finish and then exits.
I think the sleep is supposed to make launch() not return until the job
has actually been launched but it doesn't work as launch()
and getCrawlController() are both synchronized therefore the
launcher thread can't actually call startContext() until launch()
returns after sleeping.
So let's replace the sleep call with join and unsynchronize launch()
so it doesn't deadlock. All the relevant methods it calls seem to be
synchronized so I think it's no worse to not synchronize it itself.
Lots of 1 second sleeps add up fast. On my PC this reduces the runtime
of `mvn clean test` from about 3m 40s to about 1m 50s.
Ideally we'd probably re-architect some this to use a notification
mechanism instead of polling but that's easier said than done and this
is a pretty big improvement by itself.
I think the sleep is supposed to make launch() not return until the job
has actually been launched but it doesn't work as launch()
and getCrawlController() are both synchronized therefore the
launcher thread can't actually call startContext() until launch()
returns after sleeping.
So let's replace the sleep call with join and unsynchronize launch()
so it doesn't deadlock. All the relevant methods it calls seem to be
synchronized so I think it's no worse to not synchronize it itself.
This commit updates to version 3.3.1 of the dnsjava library.
Incidentally, the restlet.org certificate is currently invalid, so this
patch switched to maven.restlet.com.
addPresentableNestedNames() recursively walks the properties of beans
calling getters. Likely unintentionally this includes calling getClass()
and recursing into the reflection API. When running on JDK 11 the
reflection API has some methods (e.g. in java.lang.module) that return
instances of Optional. With the newer version of Spring's
BeanWrapperImpl encounters an Optional it attempts to unwrap it and
throws IllegalArgumentException if it is empty.
Let's fix this in two ways. Firstly let's avoid walking the reflection
API entirely as its irrelevant for the purposes of bean browsing by
not inspecting the properties of instances of java.lang.Class.
Secondly in case any Heritrix beans start using Optional in future lets
also handle the empty case by also not inspecting, the same as we do for
null.
Fixes#376
Reported-By: Lauren Ko <lauren.ko@unt.edu>
Under Restlet 2 `getVariants()` always returns null for a POST so our
post() method was never called and 405 Method Not Allowed was returned.
We don't need content negotiation for the POST response anyway so let's
instead override the no-variants post() like we do for put().
Fixes#293
URIAuthorityBasedQueueAssignmentPolicy.java - Add interoperability between forced queue assignment and parallel queues
QuotaEnforcer.java - Fix javadoc to match default behavior
This was a regression introduced in the upgrade to Restlet 2. I
encountered a NullPointerException here when upgrading and misunderstood
the cause of it. Since PUT and DELETE return no content they are
actually supposed to return null.
