This is meant to complement the javadoc by providing a reference
more suitable for users trying to configure crawls rather than
developers writing new modules.
The doc generation could still do with some improving and some of the
source javadoc comments need fixing up but this is already useful so I'm
committing what I have so far.