From d11d02eeac649e830d2b198381423cea4ece932a Mon Sep 17 00:00:00 2001 From: Alex Osborne Date: Mon, 29 Nov 2021 12:11:54 +0900 Subject: [PATCH] docs: Add 'Loading Cookies' section to config guide --- docs/configuring-jobs.rst | 46 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/docs/configuring-jobs.rst b/docs/configuring-jobs.rst index 3f5767e5..828871b1 100644 --- a/docs/configuring-jobs.rst +++ b/docs/configuring-jobs.rst @@ -383,6 +383,52 @@ requests. +Loading Cookies +--------------- + +Heritrix can be configured to load a set of cookies from a file. This can be used for example to crawl a website behind +a login form by manually logging in through the browser and then copying the session cookie. + +To enable loading of cookies set the cookiesLoadFile property of the BdbCookieStore bean to a ConfigFile: + +.. code-block:: xml + + + + + + + + + +The cookies.txt should be in the 7-field tab-separated Netscape cookie file format. An entry might look like:: + + www.example.org FALSE / FALSE 1311699995 sessionid 60ddb868550a + +.. list-table:: Cookie file tab-separated fields + + * - 1 + - DOMAIN + - The domain that created and has access to the cookie. + * - 2 + - FLAG + - A TRUE or FALSE value indicating if subdomains within the given domain can access the cookie. + * - 3 + - PATH + - The path within the domain that the cookie is valid for. + * - 4 + - SECURE + - A TRUE or FALSE value indicating if the cookie should be sent over HTTPS only. + * - 5 + - EXPIRATION + - Expiration time in seconds since 1970-01-01T00:00:00Z, or -1 for no expiration + * - 6 + - NAME + - The name of the cookie. + * - 7 + - VALUE + - The value of the cookie. + Other Protocols ---------------