heritrix3/docs/bean-reference.rst

Bean Reference
==============

.. note::
    This reference is a work in progress and does not yet cover all available beans. For a more complete list of
    Heritrix beans please refer to the `javadoc <https://www.javadoc.io/doc/org.archive.heritrix/heritrix-modules>`_.

Core Beans
----------

ActionDirectory
~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.crawler.framework.ActionDirectory

BdbCookieStore
~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.fetcher.BdbCookieStore

BdbFrontier
~~~~~~~~~~~

.. bean-doc:: org.archive.crawler.frontier.BdbFrontier

BdbModule
~~~~~~~~~

.. bean-doc:: org.archive.bdb.BdbModule

BdbServerCache
~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.net.BdbServerCache

BdbUriUniqFilter
~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.crawler.util.BdbUriUniqFilter

CheckpointService
~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.crawler.framework.CheckpointService

CrawlController
~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.crawler.framework.CrawlController

CrawlerLoggerModule
~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.crawler.reporting.CrawlerLoggerModule

CrawlLimitEnforcer
~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.crawler.framework.CrawlLimitEnforcer

CrawlMetadata
~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.CrawlMetadata

CredentialStore
~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.credential.CredentialStore

DecideRuleSequence
~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.DecideRuleSequence

DiskSpaceMonitor
~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.crawler.monitor.DiskSpaceMonitor

RulesCanonicalizationPolicy
~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.canonicalize.RulesCanonicalizationPolicy

SheetOverlaysManager
~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.crawler.spring.SheetOverlaysManager

StatisticsTracker
~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.crawler.reporting.StatisticsTracker

TextSeedModule
~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.seeds.TextSeedModule

Decide Rules
------------

AcceptDecideRule
~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.AcceptDecideRule

ClassKeyMatchesRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.crawler.deciderules.ClassKeyMatchesRegexDecideRule

ContentLengthDecideRule
~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.ContentLengthDecideRule

ContentTypeMatchesRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.ContentTypeMatchesRegexDecideRule

ContentTypeNotMatchesRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.ContentTypeNotMatchesRegexDecideRule

ExpressionDecideRule (contrib)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.ExpressionDecideRule

ExternalGeoLocationDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.ExternalGeoLocationDecideRule

FetchStatusDecideRule
~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.FetchStatusDecideRule

FetchStatusMatchesRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.FetchStatusMatchesRegexDecideRule

FetchStatusNotMatchesRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.FetchStatusNotMatchesRegexDecideRule

HasViaDecideRule
~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.HasViaDecideRule

HopCrossesAssignmentLevelDomainDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.HopCrossesAssignmentLevelDomainDecideRule

HopsPathMatchesRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.HopsPathMatchesRegexDecideRule

IdenticalDigestDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.recrawl.IdenticalDigestDecideRule

IpAddressSetDecideRule
~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.IpAddressSetDecideRule

MatchesFilePatternDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.MatchesFilePatternDecideRule

MatchesListRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.MatchesListRegexDecideRule

MatchesRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.MatchesRegexDecideRule

MatchesStatusCodeDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.MatchesStatusCodeDecideRule

NotMatchesFilePatternDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.NotMatchesFilePatternDecideRule

NotMatchesListRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.NotMatchesListRegexDecideRule

NotMatchesRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.NotMatchesRegexDecideRule

NotMatchesStatusCodeDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.NotMatchesStatusCodeDecideRule

NotOnDomainsDecideRule
~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.surt.NotOnDomainsDecideRule

NotOnHostsDecideRule
~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.surt.NotOnHostsDecideRule

NotSurtPrefixedDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.surt.NotSurtPrefixedDecideRule

OnDomainsDecideRule
~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.surt.OnDomainsDecideRule

OnHostsDecideRule
~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.surt.OnHostsDecideRule

PathologicalPathDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.PathologicalPathDecideRule

PredicatedDecideRule
~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.PredicatedDecideRule

PrerequisiteAcceptDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.PrerequisiteAcceptDecideRule

RejectDecideRule
~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.RejectDecideRule

ResourceLongerThanDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.ResourceLongerThanDecideRule

ResourceNoLongerThanDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.ResourceNoLongerThanDecideRule

ResponseContentLengthDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.ResponseContentLengthDecideRule

SchemeNotInSetDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.SchemeNotInSetDecideRule

ScriptedDecideRule
~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.ScriptedDecideRule

SeedAcceptDecideRule
~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.SeedAcceptDecideRule

SourceSeedDecideRule
~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.SourceSeedDecideRule

SurtPrefixedDecideRule
~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.surt.SurtPrefixedDecideRule

TooManyHopsDecideRule
~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.TooManyHopsDecideRule

TooManyPathSegmentsDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.TooManyPathSegmentsDecideRule

TransclusionDecideRule
~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.TransclusionDecideRule

ViaSurtPrefixedDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.deciderules.ViaSurtPrefixedDecideRule

Candidate Processors
--------------------

CandidateScoper
~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.crawler.prefetch.CandidateScoper

FrontierPreparer
~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.crawler.prefetch.FrontierPreparer

Pre-Fetch Processors
--------------------

PreconditionEnforcer
~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.crawler.prefetch.PreconditionEnforcer

Preselector
~~~~~~~~~~~

.. bean-doc:: org.archive.crawler.prefetch.Preselector

Fetch Processors
----------------

FetchDNS
~~~~~~~~

.. bean-doc:: org.archive.modules.fetcher.FetchDNS

FetchFTP
~~~~~~~~

.. bean-doc:: org.archive.modules.fetcher.FetchFTP

FetchHTTP
~~~~~~~~~

.. bean-doc:: org.archive.modules.fetcher.FetchHTTP

FetchHTTP2
~~~~~~~~~~

.. bean-doc:: org.archive.modules.fetcher.FetchHTTP2

FetchSFTP
~~~~~~~~~

.. bean-doc:: org.archive.modules.fetcher.FetchSFTP

FetchWhois
~~~~~~~~~~

.. bean-doc:: org.archive.modules.fetcher.FetchWhois

Link Extractors
---------------

ExtractorCSS
~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.extractor.ExtractorCSS

ExtractorDOC
~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.extractor.ExtractorDOC

ExtractorHTML
~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.extractor.ExtractorHTML

AggressiveExtractorHTML
~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.extractor.AggressiveExtractorHTML

JerichoExtractorHTML
~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.extractor.JerichoExtractorHTML

ExtractorHTMLForms
~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.forms.ExtractorHTMLForms

ExtractorHTTP
~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.extractor.ExtractorHTTP

ExtractorImpliedURI
~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.extractor.ExtractorImpliedURI

ExtractorJS
~~~~~~~~~~~

.. bean-doc:: org.archive.modules.extractor.ExtractorJS

ExtractorJson
~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.extractor.ExtractorJson

KnowledgableExtractorJS (contrib)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.extractor.KnowledgableExtractorJS

ExtractorMultipleRegex
~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.extractor.ExtractorMultipleRegex

ExtractorPDF
~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.extractor.ExtractorPDF

ExtractorPDFContent (contrib)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.extractor.ExtractorPDFContent

ExtractorRobotsTxt
~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.extractor.ExtractorRobotsTxt

ExtractorSitemap
~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.extractor.ExtractorSitemap

ExtractorSWF
~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.extractor.ExtractorSWF

ExtractorUniversal
~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.extractor.ExtractorUniversal

ExtractorURI
~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.extractor.ExtractorURI

ExtractorXML
~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.extractor.ExtractorXML

ExtractorYoutubeDL (contrib)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.extractor.ExtractorYoutubeDL

ExtractorYoutubeFormatStream (contrib)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.extractor.ExtractorYoutubeFormatStream

ExtractorYoutubeChannelFormatStream (contrib)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.extractor.ExtractorYoutubeChannelFormatStream

TrapSuppressExtractor
~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.extractor.TrapSuppressExtractor

Browser Processor
-----------------

BrowserProcessor
~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.crawler.processor.BrowserProcessor

ExtractLinksBehavior
~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.behaviors.ExtractLinksBehavior

ScrollDownBehavior
~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.behaviors.ScrollDownBehavior

Post-Processors
---------------

CandidatesProcessor
~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.crawler.postprocessor.CandidatesProcessor

DispositionProcessor
~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.crawler.postprocessor.DispositionProcessor

ReschedulingProcessor
~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.crawler.postprocessor.ReschedulingProcessor

WARCWriterChainProcessor
~~~~~~~~~~~~~~~~~~~~~~~~

.. bean-doc:: org.archive.modules.writer.WARCWriterChainProcessor

DnsResponseRecordBuilder
^^^^^^^^^^^^^^^^^^^^^^^^

.. bean-doc:: org.archive.modules.warc.DnsResponseRecordBuilder

FtpControlConversationRecordBuilder
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. bean-doc:: org.archive.modules.warc.FtpControlConversationRecordBuilder

FtpResponseRecordBuilder
^^^^^^^^^^^^^^^^^^^^^^^^

.. bean-doc:: org.archive.modules.warc.FtpResponseRecordBuilder

HttpRequestRecordBuilder
^^^^^^^^^^^^^^^^^^^^^^^^

.. bean-doc:: org.archive.modules.warc.HttpRequestRecordBuilder

HttpResponseRecordBuilder
^^^^^^^^^^^^^^^^^^^^^^^^^

.. bean-doc:: org.archive.modules.warc.HttpResponseRecordBuilder

MetadataRecordBuilder
^^^^^^^^^^^^^^^^^^^^^

.. bean-doc:: org.archive.modules.warc.MetadataRecordBuilder

RevisitRecordBuilder
^^^^^^^^^^^^^^^^^^^^

.. bean-doc:: org.archive.modules.warc.RevisitRecordBuilder

WhoisResponseRecordBuilder
^^^^^^^^^^^^^^^^^^^^^^^^^^

.. bean-doc:: org.archive.modules.warc.WhoisResponseRecordBuilder