Files
heritrix3/docs/getting-started.rst
T
Alex Osborne e3f9b11394 docs: fix factual errors in user documentation (#787)
getting-started:
- Use a Java 17 path in the JAVA_HOME example (17 is the minimum)
- Link to the local Configuring Jobs page instead of the wiki
- Fix typos

operating:
- Correct --jobs-dir option name and explain its default
- Document --proxy-host, --proxy-port and --sni-host-check
- Fix option list formatting that broke rendering (-c, --web-auth)
- Remove contradictory note about credentials in heritrix_out.log
- crawl.log fetch duration is separated by '+', not ';'
- Renumber progress-statistics.log fields (field 3 was duplicated)
- Fix sourceTagSeeds property name in the XML example
- ActionDirectory is already in the default profile
- Use the correct frontier.recover.gz file name throughout
- The action directory is inside the job directory, not beside bin/

configuring-jobs:
- Fix maxDocumentsDownload, obeyRelNofollow and logger class names
- Replace removed/renamed decide rules with their current names
- Fix javadoc links for decide rules in subpackages and in engine
- Fix malformed useHTTP3 XML example and FTP/SFTP seed examples
- Replace dead Confluence glossary link
- Drop DoH caveats that only applied to Java 11 / DisableExplicitGC

api:
- Fix Create Job field list not rendering (missing blank line)
- Script engines: list Groovy/BeanShell; js and AppleScript are not
  available on supported Java versions
- Make JSON sample response consistent with the XML one

glossary:
- SHA-1 is used for content digests, not encryption
- Fix garbled Host entry (logical vs physical host)
- Fix SURT and SURT prefix examples
- Status -6 refers to DNS, not WHOIS; add -9, 1001, 2001, 2002
- Link status codes to RFC 9110 instead of RFC 2616
2026-10-13 11:34:43 +09:00

139 lines
5.6 KiB
ReStructuredText

Getting Started with Heritrix
=============================
System Requirements
^^^^^^^^^^^^^^^^^^^
Heritrix is primarily used on Linux. It may run on other platforms but is not regularly tested or supported on them.
Heritrix requires Java 17 or later. We recommend using your Linux distribution's OpenJDK packages. Alternatively up to
date builds of OpenJDK for several platforms are available from `Adoptium <https://adoptium.net/>`__.
The default Java heap for Heritrix is 256MB RAM, which is usually suitable for crawls that range over hundreds of
hosts. Assign more of your available RAM to the heap if you are crawling thousands of hosts or experience Java
out-of-memory problems. You can use the JAVA_OPTS variable to configure memory.
Installation
^^^^^^^^^^^^
Download the latest Heritrix distribution package linked from the `Heritrix releases page
<https://github.com/internetarchive/heritrix3/releases>`__ and unzip it somewhere.
The installation will contain the following subdirectories:
bin
contains shell scripts/batch files for launching Heritrix.
lib
contains the third-party .jar files the Heritrix application requires to run.
conf
contains various configuration files (such as the configuration for Java logging, and pristine versions of the bundled profiles)
jobs
the default location where operator-created jobs are stored
Environment Variables
^^^^^^^^^^^^^^^^^^^^^
#. Set the ``JAVA_HOME`` environment variable. The value should point to your Java installation.
.. code-block:: bash
export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
#. Set the ``HERITRIX_HOME`` environment variable. The value should be set to the path where Heritrix is installed.
.. code-block:: bash
export HERITRIX_HOME=/home/user/heritrix-3.x.y
#. Set execute permission on the Heritrix startup file.
.. code-block:: bash
chmod u+x $HERITRIX_HOME/bin/heritrix
#. To change the amount of memory allocated to Heritrix (the Java heap size), set the ``JAVA_OPTS`` environment
variable. The following example allocates 1GB of memory to Heritrix.
.. code-block:: bash
export JAVA_OPTS=-Xmx1024M
Running Heritrix
^^^^^^^^^^^^^^^^
To launch Heritrix with the Web UI enabled, enter the following command. The username and password for the Web UI are
set to "admin" and "admin", respectively.
.. code-block:: bash
$HERITRIX_HOME/bin/heritrix -a admin:admin
By default, the Web UI listening address is only bound to the 'localhost' address. Therefore, the Web UI can only be
accessed on the same machine from which it was launched. The '-b' option may be used to listen on
different/additional addresses. See :ref:`Security Considerations <security-considerations>` before changing this
default.
If the parameter supplied to the ``-a`` option is a file path beginning with "@", the admin username and password
will be read from a file. This adds an additional layer of protection to the admin username and password by ensuring
they don't appear directly in the command-line and can't be seen by other users running the ``ps`` command.
Accessing the User Interface
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
After Heritrix has been launched, the Web-based user interface (WUI) becomes accessible.
The URI to access the Web UI is typically
https://localhost:8443/
The initial login page prompts for the username and password. After login, your session will time-out after a period
of non-use.
Access to the WUI is through HTTPS. Heritrix is installed with a keystore containing a self-signed certificate. This
will cause your browser to display a prompt, warning that a self-signed certificate is being used. Follow the steps
below for your browser to login to Heritrix for the first time.
**Chrome:** The message "Your connection is not private" is displayed. Click the "Advanced" button and then click
"Proceed to localhost (unsafe)."
**Firefox:** The message "Warning: Potential Security Risk Ahead" is displayed. Click the "Advanced..." button and then
click "Accept the Risk and Continue."
Your First Crawl
^^^^^^^^^^^^^^^^
#. Enter the name of the new job in the text box with the "create new job" label. Then click "create".
#. Click on the name of the new job and you will be taken to the job page.
#. Click on the "Configuration" link at the top and the contents of the job configuration file will be displayed.
#. At this point you must enter several properties to make the job runnable.
#. First, add the URL of page explaining how webmasters can contact you to the metadata.operatorContactUrl property.
#. Next, populate the ``<prop>`` element of the ``longerOverrides`` bean with the seed values for the crawl. A
test seed is configured for reference.
#. When done click "save changes" at the top of the page.
For more detailed information on configuring jobs see :doc:`configuring-jobs`.
#. From the job screen, click "build." This command will validate the job configuration and load it into memory. In
the Job Log the following message will display: "INFO JOB instantiated."
#. Next, click the "launch" button. This command launches the job in "paused" mode. At this point the job is ready
to run.
#. To run the job, click the "unpause" button. The job will now begin sending requests to the seeds of your crawl.
The status of the job will be set to "Running." Refresh the page to see updated statistics.
#. When you want to stop your crawl, click the "terminate" button to finish and then "teardown" to unload the job
configuration from memory.
Exiting Heritrix
^^^^^^^^^^^^^^^^
To exit Heritrix get back to the main page by clicking "Engine" in the top bar. Then check the "I'm sure" box under
"Exit Java" and click the "exit java process" button.