From b328ded92ebb3c55ef56b4b08e09d97d9e18fe83 Mon Sep 17 00:00:00 2001 From: Alex Osborne Date: Fri, 30 Sep 2022 15:31:21 +0900 Subject: [PATCH] docs: Add 'Checkpointing' section to operating guide Fixes #500 #506 --- docs/operating.rst | 56 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) diff --git a/docs/operating.rst b/docs/operating.rst index f7f16fef..041c0357 100644 --- a/docs/operating.rst +++ b/docs/operating.rst @@ -653,6 +653,62 @@ Note that the recovery journal format's 'F+' lines may include a via the above mechanisms, but that this may not be a complete representation of all URI state from its discovery in a normal crawl. +Checkpointing +------------- + +Checkpointing a crawl job writes a representation of the current state of the job under the ``checkpoints`` directory +which can be used to restart the job from the same point. + +Checkpointed state includes serialization of the main crawl job objects, copies of the current set of bdbje log files, +and other files that represent the state of the crawl. The checkpoint directory contains all that is required to +recover a crawl. Checkpointing also rotates the crawl logs, including the recover.gz log, if enabled. Log files are +NOT copied to the checkpoint directory. They are left under the logs directory and are distinguished by a suffix. The +suffix is the checkpoint name. For example, for checkpoint cp00001-20220930061713 the crawl log would be named +crawl.log.cp00001-20220930061713. + +To make checkpointing faster and reduce disk space usage, hardlinks on systems that support them to collect the +BerkeleyDB-JE files required to reproduce the crawler's state. + +To run a checkpoint, click the checkpoint button on the job page of the WUI or invoke the checkpoint functionality +through the REST API. While checkpointing, the crawl status will show as CHECKPOINTING. When the checkpoint has +completed, the crawler will resume crawling, unless it was in the paused state when the checkpoint was invoked. +In this case, the crawler will re-enter the paused state. + +Recovery from a checkpoint has much in common with the recovery of a crawl using the frontier.recovery.log. + +Automated Checkpointing +~~~~~~~~~~~~~~~~~~~~~~~ + +To configure Heritrix to automatically run checkpoints periodically, set the +``checkpointService.checkpointIntervalMinutes`` property: + +.. code-block:: xml + + + + + + + +By default only the latest checkpoint will be kept. + +Restarting from a Checkpoint +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The web UI provides an option to restart a crawl from a checkpoint: + +1. Checkpoint the running crawl by clicking the "checkpoint" button. +2. When the checkpoint ends (a message will be displayed informing the operator of this event) terminate the crawl by + clicking the "terminate" button. +3. Teardown the job by clicking the "teardown" button. +4. Re-build the job by clicking the "build" button. At this point a dropdown box should appear under the command + buttons. The dropdown box has the names of the previously invoked checkpoints. +5. Select a checkpoint from the dropdown. The selected checkpoint will be used to start the newly built job. +6. Click launch +7. Click unpause + +The job will now begin running from the chosen checkpoint. + Crawl Recovery --------------