Files
heritrix3/docs/api.rst
T
Alex Osborne e3f9b11394 docs: fix factual errors in user documentation (#787)
getting-started:
- Use a Java 17 path in the JAVA_HOME example (17 is the minimum)
- Link to the local Configuring Jobs page instead of the wiki
- Fix typos

operating:
- Correct --jobs-dir option name and explain its default
- Document --proxy-host, --proxy-port and --sni-host-check
- Fix option list formatting that broke rendering (-c, --web-auth)
- Remove contradictory note about credentials in heritrix_out.log
- crawl.log fetch duration is separated by '+', not ';'
- Renumber progress-statistics.log fields (field 3 was duplicated)
- Fix sourceTagSeeds property name in the XML example
- ActionDirectory is already in the default profile
- Use the correct frontier.recover.gz file name throughout
- The action directory is inside the job directory, not beside bin/

configuring-jobs:
- Fix maxDocumentsDownload, obeyRelNofollow and logger class names
- Replace removed/renamed decide rules with their current names
- Fix javadoc links for decide rules in subpackages and in engine
- Fix malformed useHTTP3 XML example and FTP/SFTP seed examples
- Replace dead Confluence glossary link
- Drop DoH caveats that only applied to Java 11 / DisableExplicitGC

api:
- Fix Create Job field list not rendering (missing blank line)
- Script engines: list Groovy/BeanShell; js and AppleScript are not
  available on supported Java versions
- Make JSON sample response consistent with the XML one

glossary:
- SHA-1 is used for content digests, not encryption
- Fix garbled Host entry (logical vs physical host)
- Fix SURT and SURT prefix examples
- Status -6 refers to DNS, not WHOIS; add -9, 1001, 2001, 2002
- Link status codes to RFC 9110 instead of RFC 2616
2026-10-13 11:34:43 +09:00

807 lines
28 KiB
ReStructuredText
Raw Blame History

This file contains invisible Unicode characters
This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
REST API
========
This manual describes the REST application programming interface (API)
of the Heritrix Web crawler.
This document is intended for application developers and administrators
interested in controlling the Heritrix Web crawler through its REST API.
Any client that supports HTTPS can be used to invoke the Heritrix API.
The examples in this document use the command line tool curl which
is typically found in most unix environments. Curl is
\ `available <https://curl.haxx.se/download.html>`__ for many systems
including Windows.
Get Engine Status
~~~~~~~~~~~~~~~~~
.. http:get:: https://(heritrixhost):8443/engine
Returns information about this instance of Heritrix such as version number, memory usage and the list of crawl jobs.
**XML Example:**
.. code:: bash
curl -v -k -u admin:admin --anyauth --location -H "Accept: application/xml" https://localhost:8443/engine
Response:
.. code:: xml
<engine>
<heritrixVersion>3.3.0-SNAPSHOT-2017-07-12T04:17:56Z</heritrixVersion>
<heapReport>
<usedBytes>69529904</usedBytes>
<totalBytes>589824000</totalBytes>
<maxBytes>2885681152</maxBytes>
</heapReport>
<jobsDir>/heritrix/jobs</jobsDir>
<jobsDirUrl>https://localhost:8443/engine/jobsdir/</jobsDirUrl>
<availableActions>
<value>rescan</value>
<value>add</value>
<value>create</value>
</availableActions>
<jobs>
<value>
<shortName>myjob</shortName>
<url>https://localhost:8443/engine/job/myjob</url>
<isProfile>false</isProfile>
<launchCount>0</launchCount>
<lastLaunch/>
<hasApplicationContext>false</hasApplicationContext>
<statusDescription>Unbuilt</statusDescription>
<isLaunchInfoPartial>false</isLaunchInfoPartial>
<primaryConfig>/heritrix/jobs/myjob/crawler-beans.cxml</primaryConfig>
<primaryConfigUrl>https://localhost:8443/engine/jobdir/crawler-beans.cxml</primaryConfigUrl>
<key>myjob</key>
</value>
</jobs>
<profiles>
<value>Defaults (XML)</value>
<value>Defaults (Groovy)</value>
<value>myprofile</value>
</profiles>
</engine>
**JSON Example:**
.. code:: bash
curl -v -k -u admin:admin --anyauth --location -H "Accept: application/json" https://localhost:8443/engine
Response:
.. code:: json
{
"availableActions": [
"rescan",
"add",
"create"
],
"heapReport": {
"usedBytes": 69529904,
"totalBytes": 589824000,
"maxBytes": 2885681152
},
"jobsDirUrl": "https://localhost:8443/engine/jobsdir/",
"heritrixVersion": "3.3.0-SNAPSHOT-2017-07-12T04:17:56Z",
"jobsDir": "/heritrix/jobs",
"jobs": [{
"isProfile": false,
"launchCount": 0,
"statusDescription": "Unbuilt",
"hasApplicationContext": false,
"shortName": "myjob",
"isLaunchInfoPartial": false,
"url": "https://localhost:8443/engine/job/myjob",
"key": "myjob",
"primaryConfig": "/heritrix/jobs/myjob/crawler-beans.cxml",
"primaryConfigUrl": "https://localhost:8443/engine/jobdir/crawler-beans.cxml"
}]
}
Create New Job
~~~~~~~~~~~~~~
.. http:post:: https://(heritrixhost):8443/engine [action=create]
Creates a new crawl job from the selected profile. If no profile is
supplied, ``Defaults (XML)`` is used. Built-in profiles include
``Defaults (XML)`` and ``Defaults (Groovy)``. Existing job profiles may
also be selected by passing the profile job's short name.
:form action: must be ``create``
:form createpath: the name of the new job
:form profile: optional profile name. Existing profiles use their job
short names.
**HTML Example:**
.. code:: bash
curl -v -d "createpath=myjob&action=create" -k -u admin:admin --anyauth --location \
https://localhost:8443/engine
To create a job from the Groovy defaults:
.. code:: bash
curl -v --data-urlencode "createpath=mygroovyjob" --data-urlencode "action=create" \
--data-urlencode "profile=Defaults (Groovy)" -k -u admin:admin --anyauth --location \
https://localhost:8443/engine
**XML Example:**
.. code:: bash
curl -v -d "createpath=myjob&action=create" -k -u admin:admin --anyauth --location \
-H "Accept: application/xml" https://localhost:8443/engine
**JSON Example:**
.. code:: bash
curl -v -d "createpath=myjob&action=create" -k -u admin:admin --anyauth --location \
-H "Accept: application/json" https://localhost:8443/engine
Add Job Directory
~~~~~~~~~~~~~~~~~
.. http:post:: https://(heritrixhost):8443/engine [action=add]
Adds a new job directory to the Heritrix configuration. The directory must
contain a cxml configuration file.
:form action: must be ``add``
:form addpath: the job directory to add
**HTML Example:**
.. code:: bash
curl -v -d "action=add&addpath=/Users/hstern/job" -k -u admin:admin --anyauth --location https://localhost:8443/engine
**XML Example:**
.. code:: bash
curl -v -d "action=add&addpath=/Users/hstern/job" -k -u admin:admin --anyauth --location -H "Accept: application/xml" https://localhost:8443/engine
**JSON Example:**
.. code:: bash
curl -v -d "action=add&addpath=/Users/hstern/job" -k -u admin:admin --anyauth --location -H "Accept: application/json" https://localhost:8443/engine
Get Job Status
~~~~~~~~~~~~~~
.. http:get:: https://(heritrixhost):8443/engine/job/(jobname)
Returns status information and statistics about the chosen job.
**XML Example:**
.. code:: bash
curl -v -k -u admin:admin --anyauth --location -H "Accept: application/xml" https://localhost:8443/engine/job/myjob
Response:
.. code:: xml
<job>
<shortName>myjob</shortName>
<crawlControllerState>FINISHED</crawlControllerState>
<crawlExitStatus>FINISHED</crawlExitStatus>
<statusDescription>Finished: FINISHED</statusDescription>
<availableActions>
<value>teardown</value>
</availableActions>
<launchCount>1</launchCount>
<lastLaunch>2020-04-01T02:07:42.531Z</lastLaunch>
<isProfile>false</isProfile>
<primaryConfig>/heritrix/jobs/myjob/crawler-beans.cxml</primaryConfig>
<primaryConfigUrl>https://localhost:8443/engine/job/myjob/jobdir/crawler-beans.cxml</primaryConfigUrl>
<url>https://localhost:8443/engine/job/myjob/job/myjob</url>
<jobLogTail>
<value>2020-04-01T03:50:44.708Z INFO FINISHED 20200401020744</value>
<value>2020-04-01T03:50:42.670Z INFO EMPTY 20200401020744</value>
<value>2020-04-01T03:50:42.669Z INFO STOPPING 20200401020744</value>
</jobLogTail>
<uriTotalsReport>
<downloadedUriCount>3920</downloadedUriCount>
<queuedUriCount>0</queuedUriCount>
<totalUriCount>3920</totalUriCount>
<futureUriCount>0</futureUriCount>
</uriTotalsReport>
<sizeTotalsReport>
<dupByHash>0</dupByHash>
<dupByHashCount>0</dupByHashCount>
<notModified>0</notModified>
<notModifiedCount>0</notModifiedCount>
<novel>2177235508</novel>
<novelCount>3920</novelCount>
<total>2177235508</total>
<totalCount>3920</totalCount>
<warcNovelContentBytes>2177235508</warcNovelContentBytes>
<warcNovelUrls>3920</warcNovelUrls>
</sizeTotalsReport>
<rateReport>
<currentDocsPerSecond>0.0</currentDocsPerSecond>
<averageDocsPerSecond>0.6354171124312226</averageDocsPerSecond>
<currentKiBPerSec>0</currentKiBPerSec>
<averageKiBPerSec>344</averageKiBPerSec>
</rateReport>
<loadReport>
<busyThreads>0</busyThreads>
<totalThreads>0</totalThreads>
<congestionRatio>NaN</congestionRatio>
<averageQueueDepth>0</averageQueueDepth>
<deepestQueueDepth>0</deepestQueueDepth>
</loadReport>
<elapsedReport>
<elapsedMilliseconds>6169176</elapsedMilliseconds>
<elapsedPretty>1h42m49s176ms</elapsedPretty>
</elapsedReport>
<threadReport/>
<frontierReport>
<totalQueues>1</totalQueues>
<inProcessQueues>0</inProcessQueues>
<readyQueues>0</readyQueues>
<snoozedQueues>0</snoozedQueues>
<activeQueues>0</activeQueues>
<inactiveQueues>0</inactiveQueues>
<ineligibleQueues>0</ineligibleQueues>
<retiredQueues>0</retiredQueues>
<exhaustedQueues>1</exhaustedQueues>
<lastReachedState>FINISH</lastReachedState>
</frontierReport>
<crawlLogTail>
...
</crawlLogTail>
<configFiles>
...
</configFiles>
<isLaunchInfoPartial>false</isLaunchInfoPartial>
<isRunning>false</isRunning>
<isLaunchable>false</isLaunchable>
<hasApplicationContext>true</hasApplicationContext>
<alertCount>549</alertCount>
<checkpointFiles></checkpointFiles>
<alertLogFilePath>/heritrix/jobs/myjob/20200401020744/logs/alerts.log</alertLogFilePath>
<crawlLogFilePath>/heritrix/jobs/myjob/20200401020744/logs/crawl.log</crawlLogFilePath>
<reports>
<value>
<className>CrawlSummaryReport</className>
<shortName>CrawlSummary</shortName>
</value>
...
</reports>
<heapReport>
<usedBytes>66893400</usedBytes>
<totalBytes>589824000</totalBytes>
<maxBytes>2885681152</maxBytes>
</heapReport>
</job>
**JSON Example:**
.. code:: bash
curl -v -k -u admin:admin --anyauth --location -H "Accept: application/json" https://localhost:8443/engine/job/myjob
Response:
.. code:: json
{
"uriTotalsReport": {
"downloadedUriCount": 3920,
"queuedUriCount": 0,
"futureUriCount": 0,
"totalUriCount": 3920
},
"reports": [
{
"shortName": "CrawlSummary",
"className": "CrawlSummaryReport"
},
// ...
],
"rateReport": {
"currentDocsPerSecond": 0,
"averageKiBPerSec": 344,
"currentKiBPerSec": 0,
"averageDocsPerSecond": 0.6354171124312226
},
"lastLaunch": "2020-04-01T02:07:42.531Z",
"frontierReport": {
"lastReachedState": "FINISH",
"totalQueues": 1,
"queueReadiedCount": 0,
"inProcessQueues": 0,
"readyQueues": 0,
"activeQueues": 0,
"inactiveQueues": 0,
"ineligibleQueues": 0,
"retiredQueues": 0,
"exhaustedQueues": 1,
"snoozedQueues": 0
},
"sizeTotalsReport": {
"notModifiedCount": 0,
"total": 2177235508,
"notModified": 0,
"dupByHashCount": 0,
"novelCount": 3920,
"sizeOnDisk": 0,
"totalCount": 3920,
"dupByHash": 0,
"novel": 2177235508,
"warcNovelContentBytes": 2177235508,
"warcNovelUrls": 3920
},
"checkpointFiles": [],
"crawlLogTail": [
// ...
],
"primaryConfig": "/heritrix/jobs/myjob/crawler-beans.cxml",
"primaryConfigUrl": "https://localhost:8443/engine/job/myjob/jobdir/crawler-beans.cxml",
"crawlLogFileUrl": "jobdir/20200401020744/logs/crawl.log",
"heapReport": {
"usedBytes": 66893400,
"totalBytes": 589824000,
"maxBytes": 2885681152
},
"isRunning": false,
"isLaunchable": false,
"isLaunchInfoPartial": false,
"alertLogFilePath": "/heritrix/jobs/myjob/20200401020744/logs/alerts.log",
"alertLogFileUrl": "jobdir/20200401020744/logs/alerts.log",
"availableActions": ["teardown"],
"launchCount": 1,
"isProfile": false,
"loadReport": {
"totalThreads": 0,
"busyThreads": 0,
"averageQueueDepth": 0,
"deepestQueueDepth": 0,
"congestionRatio": null
},
"crawlExitStatus": "FINISHED",
"jobLogTail": [
"2020-04-01T03:50:44.708Z INFO FINISHED 20200401020744",
"2020-04-01T03:50:42.670Z INFO EMPTY 20200401020744",
"2020-04-01T03:50:42.669Z INFO STOPPING 20200401020744"
],
"url": "https://localhost:8443/engine/job/myjob/job/myjob",
"elapsedReport": {
"elapsedPretty": "1h42m49s176ms",
"elapsedMilliseconds": 6169176
},
"statusDescription": "Finished: FINISHED",
"configFiles": [
// ...
],
"hasApplicationContext": true,
"alertCount": 549,
"crawlLogFilePath": "/heritrix/jobs/myjob/20200401020744/logs/crawl.log",
"shortName": "myjob",
"crawlControllerState": "FINISHED"
}
Build Job Configuration
~~~~~~~~~~~~~~~~~~~~~~~
.. http:post:: https://(heritrixhost):8443/engine/job/(jobname) [action=build]
Builds the job configuration for the chosen job. It reads an XML descriptor
file and uses Spring to build the Java objects that are necessary for
running the crawl. Before a crawl can be run it must be built.
:form action: must be ``build``
**HTML Example:**
.. code:: bash
curl -v -d "action=build" -k -u admin:admin --anyauth --location https://localhost:8443/engine/job/myjob
**XML Example:**
.. code:: bash
curl -v -d "action=build" -k -u admin:admin --anyauth --location -H "Accept: application/xml" https://localhost:8443/engine/job/myjob
**JSON Example:**
.. code:: bash
curl -v -d "action=build" -k -u admin:admin --anyauth --location -H "Accept: application/json" https://localhost:8443/engine/job/myjob
Launch Job
~~~~~~~~~~
.. http:post:: https://(heritrixhost):8443/engine/job/(jobname) [action=launch]
Launches a crawl job. The job can be launched in the "paused" state or the
"unpaused" state. If launched in the "unpaused" state the job will
immediately begin crawling.
:form action: must be ``launch``
:form checkpoint: optional field: If supplied, Heritrix will attempt to launch from a checkpoint. Should be the name of a checkpoint (e.g. ``cp00001-20180102121229``) or (since version 3.3) the special value ``latest``, which will automatically select the most recent checkpoint. If no ``checkpoint`` is specified (or if the ``latest`` checkpoint is requested and there are no valid checkpoints) a new crawl will be launched.
**HTML Example:**
.. code:: bash
curl -v -d "action=launch" -k -u admin:admin --anyauth --location https://localhost:8443/engine/job/myjob
**XML Example:**
.. code:: bash
curl -v -d "action=launch" -k -u admin:admin --anyauth --location -H "Accept: application/xml" https://localhost:8443/engine/job/myjob
**JSON Example:**
.. code:: bash
curl -v -d "action=launch" -k -u admin:admin --anyauth --location -H "Accept: application/json" https://localhost:8443/engine/job/myjob
Rescan Job Directory
~~~~~~~~~~~~~~~~~~~~
.. http:post:: https://(heritrixhost):8443/engine [action=rescan]
Rescans the main job directory and returns an HTML page containing all the
job names. It also returns information about the jobs, such as the location
of the job configuration file and the number of job launches.
:form action: must be ``rescan``
**HTML Example:**
.. code:: bash
curl -v -d "action=rescan" -k -u admin:admin --anyauth --location https://localhost:8443/engine
**XML Example:**
.. code:: bash
curl -v -d "action=rescan" -k -u admin:admin --anyauth --location -H "Accept: application/xml" https://localhost:8443/engine
**JSON Example:**
.. code:: bash
curl -v -d "action=rescan" -k -u admin:admin --anyauth --location -H "Accept: application/json" https://localhost:8443/engine
Pause Job
~~~~~~~~~
.. http:post:: https://(heritrixhost):8443/engine/job/(jobname) [action=pause]
Pauses an unpaused job. No crawling will occur while a job is paused.
:form action: must be ``pause``
**HTML Example:**
.. code:: bash
curl -v -d "action=pause" -k -u admin:admin --anyauth --location https://localhost:8443/engine/job/myjob
**XML Example:**
.. code:: bash
curl -v -d "action=pause" -k -u admin:admin --anyauth --location -H "Accept: application/xml" https://localhost:8443/engine/job/myjob
**JSON Example:**
.. code:: bash
curl -v -d "action=pause" -k -u admin:admin --anyauth --location -H "Accept: application/json" https://localhost:8443/engine/job/myjob
Unpause Job
~~~~~~~~~~~
.. http:post:: https://(heritrixhost):8443/engine/job/(jobname) [action=unpause]
This API unpauses a paused job. Crawling will resume (or begin, in the case
of a job launched in the paused state) if possible.
:form action: must be ``unpause``
**HTML Example:**
.. code:: bash
curl -v -d "action=unpause" -k -u admin:admin --anyauth --location https://localhost:8443/engine/job/myjob
**XML Example:**
.. code:: bash
curl -v -d "action=unpause" -k -u admin:admin --anyauth --location -H "Accept: application/xml" https://localhost:8443/engine/job/myjob
**JSON Example:**
.. code:: bash
curl -v -d "action=unpause" -k -u admin:admin --anyauth --location -H "Accept: application/json" https://localhost:8443/engine/job/myjob
Terminate Job
~~~~~~~~~~~~~
.. http:post:: https://(heritrixhost):8443/engine/job/(jobname) [action=terminate]
Terminates a running job.
:form action: must be ``terminate``
**HTML Example:**
.. code:: bash
curl -v -d "action=terminate" -k -u admin:admin --anyauth --location https://localhost:8443/engine/job/myjob
**XML Example:**
.. code:: bash
curl -v -d "action=terminate" -k -u admin:admin --anyauth --location -H "Accept: application/xml" https://localhost:8443/engine/job/myjob
**JSON Example:**
.. code:: bash
curl -v -d "action=terminate" -k -u admin:admin --anyauth --location -H "Accept: application/json" https://localhost:8443/engine/job/myjob
Teardown Job
~~~~~~~~~~~~
.. http:post:: https://(heritrixhost):8443/engine/job/(jobname) [action=teardown]
Removes the Spring code that is used to run the job. Once a job is torn down
it must be rebuilt in order to run.
:form action: must be ``teardown``
**HTML Example:**
.. code:: bash
curl -v -d "action=teardown" -k -u admin:admin --anyauth --location https://localhost:8443/engine/job/myjob
**XML Example:**
.. code:: bash
curl -v -d "action=teardown" -k -u admin:admin --anyauth --location -H "Accept: application/xml" https://localhost:8443/engine/job/myjob
**JSON Example:**
.. code:: bash
curl -v -d "action=teardown" -k -u admin:admin --anyauth --location -H "Accept: application/json" https://localhost:8443/engine/job/myjob
Copy Job
~~~~~~~~
.. http:post:: https://(heritrixhost):8443/engine/job/(jobname) [copyTo]
Copies an existing job configuration to a new job configuration. If the
``asProfile`` option is submitted with value ``on``, then the copy is a
non-runnable profile. Profiles are listed as options when creating new
jobs, and profiles with built-in names override the built-ins.
:form copyTo: the name of the new job or profile configuration
:form asProfile: whether to copy the job as a runnable configuration or as a
non-runnable profile. The value ``on`` means the job will be copied as a
profile. If omitted the job will be copied as a runnable configuration.
**HTML Example:**
.. code:: bash
curl -v -d "copyTo=mycopy&asProfile=on" -k -u admin:admin --anyauth --location https://localhost:8443/engine/job/myjob
**XML Example:**
.. code:: bash
curl -v -d "copyTo=mycopy&asProfile=on" -k -u admin:admin --anyauth --location -H "Accept: application/xml" https://localhost:8443/engine/job/myjob
**JSON Example:**
.. code:: bash
curl -v -d "copyTo=mycopy&asProfile=on" -k -u admin:admin --anyauth --location -H "Accept: application/json" https://localhost:8443/engine/job/myjob
Delete Job
~~~~~~~~~~
.. http:post:: https://(heritrixhost):8443/engine/job/(jobname) [delete]
Deletes an existing job. It will remove everything related to the job
(configuration, statistics, logs and results) including the whole job
folder. Everything to keep has to be copied outside of the job folder.
Deleting a job is only possible if no active application context for the job
exists which is the case for new or unbuilt jobs. If a job has been built,
it must first be torn down to allow deletion.
:form action: must be ``delete``
**HTML Example:**
.. code:: bash
curl -v -d "action=delete" -k -u admin:admin --anyauth --location https://localhost:8443/engine/job/myjob
**XML Example:**
.. code:: bash
curl -v -d "action=delete" -k -u admin:admin --anyauth --location -H "Accept: application/xml" https://localhost:8443/engine/job/myjob
**JSON Example:**
.. code:: bash
curl -v -d "action=delete" -k -u admin:admin --anyauth --location -H "Accept: application/json" https://localhost:8443/engine/job/myjob
Checkpoint Job
~~~~~~~~~~~~~~
.. http:post:: https://(heritrixhost):8443/engine/job/(jobname) [action=checkpoint]
This API checkpoints the chosen job. Checkpointing writes the current state
of a crawl to the file system so that the crawl can be recovered if it
fails.
:form action: must be ``checkpoint``
**HTML Example:**
.. code:: bash
curl -v -d "action=checkpoint" -k -u admin:admin --anyauth --location https://localhost:8443/engine/job/myjob
**XML Example:**
.. code:: bash
curl -v -d "action=checkpoint" -k -u admin:admin --anyauth --location -H "Accept: application/xml" https://localhost:8443/engine/job/myjob
**JSON Example:**
.. code:: bash
curl -v -d "action=checkpoint" -k -u admin:admin --anyauth --location -H "Accept: application/json" https://localhost:8443/engine/job/myjob
Execute Script in Job
~~~~~~~~~~~~~~~~~~~~~
.. http:post:: https://(heritrixhost):8443/engine/job/(jobname)/script
Executes a script. Any JSR-223 script engine available on the classpath
may be used. Heritrix includes Groovy and BeanShell.
:form engine: the name of the script engine to use, such as ``groovy`` or
``beanshell``.
:form script: the script code to execute
**HTML Example:**
.. code:: bash
curl -v -d "engine=beanshell&script=System.out.println%28%22test%22%29%3B" -k -u admin:admin --anyauth --location https://localhost:8443/engine/job/myjob/script
**XML Example:**
.. code:: bash
curl -v -d "engine=beanshell&script=System.out.println%28%22test%22%29%3B" -k -u admin:admin --anyauth --location -H "Accept: application/xml" https://localhost:8443/engine/job/myjob/script
**JSON Example:**
.. code:: bash
curl -v -d "engine=beanshell&script=System.out.println%28%22test%22%29%3B" -k -u admin:admin --anyauth --location -H "Accept: application/json" https://localhost:8443/engine/job/myjob/script
Submitting a CXML Job Configuration File
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. http:put:: https://(heritrixhost):8443/engine/job/(jobname)/jobdir/crawler-beans.cxml
Submits the contents of a CXML file for a chosen job. CXML files are the
configuration files used to control a crawl job. Each job has a single CXML
file.
**Example:**
.. code:: bash
curl -v -T my-crawler-beans.cxml -k -u admin:admin --anyauth --location https://localhost:8443/engine/job/myjob/jobdir/crawler-beans.cxml
:statuscode 200: On success, the Heritrix REST API will return a HTTP 200 with no body.
Conventions and Assumptions
~~~~~~~~~~~~~~~~~~~~~~~~~~~
The following curl parameters are used when invoking the API.
+-----------------------------------+-----------------------------------+
| | curl Parameter | | Description |
+===================================+===================================+
| -v | Verbose. Output a detailed |
| | account of the curl command to |
| | standard out. |
+-----------------------------------+-----------------------------------+
| -d | Data. These are the name/value |
| | pairs that are sent in the body |
| | of a POST. |
+-----------------------------------+-----------------------------------+
| -k | Insecure. Allows connections to |
| | SSL sites without certificates. |
+-----------------------------------+-----------------------------------+
| | -u | User. Allows the submission of a |
| | username and password to |
| | authenticate the HTTP request. |
+-----------------------------------+-----------------------------------+
| --anyauth | Any authentication type. Allows |
| | authentication of the request |
| | based on any type of |
| | authentication method. |
+-----------------------------------+-----------------------------------+
| --location | Follows HTTP redirects. This |
| | option is used so that API calls |
| | that return data (such as HTML) |
| | will not halt upon receipt of a |
| | redirect code (such as an HTTP |
| | 303). |
+-----------------------------------+-----------------------------------+
| | -H | Set the value of an HTTP header. |
| | For example, "Accept: |
| | application/xml". |
+-----------------------------------+-----------------------------------+
It is assumed that the reader has a working knowledge of the HTTP
protocol and Heritrix functionality.  Also, the examples assume that
Heritrix is run with an administrative username and password of "admin."
About the REST implementation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Representational State Transfer (REST) is a software architecture for
distributed hypermedia systems such as the World Wide Web (WWW). REST is
built on the concept of representations of resources. Resources can be
any coherent and meaningful concept that may be addressed. A URI is an
example of a resource. The representation of the resource is typically a
document that captures the current or intended state of the resource. An
example of a representation of a resource is an HTML page.
Heritrix uses REST to expose its functionality. The REST implementation
used by Heritrix is Restlet. Restlet implements the concepts defined by
REST, including resources and representations. It also provides a REST
container that processes RESTful requests. The container is the Noelios
Restlet Engine. For detailed information on Restlet,
visit \ http://www.restlet.org/.
Heritrix exposes its REST functionality through HTTPS. The HTTPS
protocol is used to send requests to retrieve or modify configuration
settings and manage crawl jobs.