Files
heritrix3/docs/bean-reference.rst
T
Alex Osborne 9d79ef7372 Add ExtractorSVG and extract links from inline SVG
ExtractorSVG parses SVG with the JDK's StAX parser and extracts hrefs
from elements like <image>, <use> and <script> as embeds and from <a>
as links, along with <?xml-stylesheet?>, url() in presentation
attributes, <style>, <script>, animated hrefs and HTML in
<foreignObject>. Like browsers it requires the SVG namespace, decides
by the root element whether text/xml and application/xml responses
are SVG, and keeps the links found before any XML error. It's added
to the default profile before extractorHtml.

ExtractorHTML and JerichoExtractorHTML now treat hrefs on those inline
SVG elements as embeds rather than links, and extract url() from
presentation attributes like fill. ExtractorCSS now ignores
same-document references like url(#gradient).

Fixes #796
2026-11-05 16:17:28 +09:00

608 lines
13 KiB
ReStructuredText

Bean Reference
==============
.. note::
This reference is a work in progress and does not yet cover all available beans. For a more complete list of
Heritrix beans please refer to the `javadoc <https://www.javadoc.io/doc/org.archive.heritrix/heritrix-modules>`_.
Core Beans
----------
ActionDirectory
~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.framework.ActionDirectory
BdbCookieStore
~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.fetcher.BdbCookieStore
BdbFrontier
~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.frontier.BdbFrontier
BdbModule
~~~~~~~~~
.. bean-doc:: org.archive.bdb.BdbModule
BdbServerCache
~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.net.BdbServerCache
BdbUriUniqFilter
~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.util.BdbUriUniqFilter
CheckpointService
~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.framework.CheckpointService
CrawlController
~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.framework.CrawlController
CrawlerLoggerModule
~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.reporting.CrawlerLoggerModule
CrawlLimitEnforcer
~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.framework.CrawlLimitEnforcer
CrawlMetadata
~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.CrawlMetadata
CredentialStore
~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.credential.CredentialStore
DecideRuleSequence
~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.DecideRuleSequence
DiskSpaceMonitor
~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.monitor.DiskSpaceMonitor
RulesCanonicalizationPolicy
~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.canonicalize.RulesCanonicalizationPolicy
SheetOverlaysManager
~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.spring.SheetOverlaysManager
StatisticsTracker
~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.reporting.StatisticsTracker
TextSeedModule
~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.seeds.TextSeedModule
Decide Rules
------------
AcceptDecideRule
~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.AcceptDecideRule
ClassKeyMatchesRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.deciderules.ClassKeyMatchesRegexDecideRule
ContentLengthDecideRule
~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.ContentLengthDecideRule
ContentTypeMatchesRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.ContentTypeMatchesRegexDecideRule
ContentTypeNotMatchesRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.ContentTypeNotMatchesRegexDecideRule
ExpressionDecideRule (contrib)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.ExpressionDecideRule
ExternalGeoLocationDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.ExternalGeoLocationDecideRule
FetchStatusDecideRule
~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.FetchStatusDecideRule
FetchStatusMatchesRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.FetchStatusMatchesRegexDecideRule
FetchStatusNotMatchesRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.FetchStatusNotMatchesRegexDecideRule
HasViaDecideRule
~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.HasViaDecideRule
HopCrossesAssignmentLevelDomainDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.HopCrossesAssignmentLevelDomainDecideRule
HopsPathMatchesRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.HopsPathMatchesRegexDecideRule
IdenticalDigestDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.recrawl.IdenticalDigestDecideRule
IpAddressSetDecideRule
~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.IpAddressSetDecideRule
MatchesFilePatternDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.MatchesFilePatternDecideRule
MatchesListRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.MatchesListRegexDecideRule
MatchesRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.MatchesRegexDecideRule
MatchesStatusCodeDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.MatchesStatusCodeDecideRule
NotMatchesFilePatternDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.NotMatchesFilePatternDecideRule
NotMatchesListRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.NotMatchesListRegexDecideRule
NotMatchesRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.NotMatchesRegexDecideRule
NotMatchesStatusCodeDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.NotMatchesStatusCodeDecideRule
NotOnDomainsDecideRule
~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.surt.NotOnDomainsDecideRule
NotOnHostsDecideRule
~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.surt.NotOnHostsDecideRule
NotSurtPrefixedDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.surt.NotSurtPrefixedDecideRule
OnDomainsDecideRule
~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.surt.OnDomainsDecideRule
OnHostsDecideRule
~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.surt.OnHostsDecideRule
PathologicalPathDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.PathologicalPathDecideRule
PredicatedDecideRule
~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.PredicatedDecideRule
PrerequisiteAcceptDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.PrerequisiteAcceptDecideRule
RejectDecideRule
~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.RejectDecideRule
ResourceLongerThanDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.ResourceLongerThanDecideRule
ResourceNoLongerThanDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.ResourceNoLongerThanDecideRule
ResponseContentLengthDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.ResponseContentLengthDecideRule
SchemeNotInSetDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.SchemeNotInSetDecideRule
ScriptedDecideRule
~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.ScriptedDecideRule
SeedAcceptDecideRule
~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.SeedAcceptDecideRule
SourceSeedDecideRule
~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.SourceSeedDecideRule
SurtPrefixedDecideRule
~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.surt.SurtPrefixedDecideRule
TooManyHopsDecideRule
~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.TooManyHopsDecideRule
TooManyPathSegmentsDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.TooManyPathSegmentsDecideRule
TransclusionDecideRule
~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.TransclusionDecideRule
ViaSurtPrefixedDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.ViaSurtPrefixedDecideRule
Candidate Processors
--------------------
CandidateScoper
~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.prefetch.CandidateScoper
FrontierPreparer
~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.prefetch.FrontierPreparer
Pre-Fetch Processors
--------------------
PreconditionEnforcer
~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.prefetch.PreconditionEnforcer
Preselector
~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.prefetch.Preselector
Fetch Processors
----------------
FetchDNS
~~~~~~~~
.. bean-doc:: org.archive.modules.fetcher.FetchDNS
FetchFTP
~~~~~~~~
.. bean-doc:: org.archive.modules.fetcher.FetchFTP
FetchHTTP
~~~~~~~~~
.. bean-doc:: org.archive.modules.fetcher.FetchHTTP
FetchHTTP2
~~~~~~~~~~
.. bean-doc:: org.archive.modules.fetcher.FetchHTTP2
FetchSFTP
~~~~~~~~~
.. bean-doc:: org.archive.modules.fetcher.FetchSFTP
FetchWhois
~~~~~~~~~~
.. bean-doc:: org.archive.modules.fetcher.FetchWhois
Link Extractors
---------------
ExtractorCSS
~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorCSS
ExtractorDOC
~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorDOC
ExtractorHTML
~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorHTML
AggressiveExtractorHTML
~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.AggressiveExtractorHTML
JerichoExtractorHTML
~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.JerichoExtractorHTML
ExtractorHTMLForms
~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.forms.ExtractorHTMLForms
ExtractorHTTP
~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorHTTP
ExtractorImpliedURI
~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorImpliedURI
ExtractorJS
~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorJS
ExtractorJson
~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorJson
KnowledgableExtractorJS (contrib)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.KnowledgableExtractorJS
ExtractorMultipleRegex
~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorMultipleRegex
ExtractorPDF
~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorPDF
ExtractorPDFContent (contrib)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorPDFContent
ExtractorRobotsTxt
~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorRobotsTxt
ExtractorSitemap
~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorSitemap
ExtractorSVG
~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorSVG
ExtractorSWF
~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorSWF
ExtractorUniversal
~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorUniversal
ExtractorURI
~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorURI
ExtractorXML
~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorXML
ExtractorYoutubeDL (contrib)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorYoutubeDL
ExtractorYoutubeFormatStream (contrib)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorYoutubeFormatStream
ExtractorYoutubeChannelFormatStream (contrib)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorYoutubeChannelFormatStream
TrapSuppressExtractor
~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.TrapSuppressExtractor
Browser Processor
-----------------
BrowserProcessor
~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.processor.BrowserProcessor
ExtractLinksBehavior
~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.behaviors.ExtractLinksBehavior
ScrollDownBehavior
~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.behaviors.ScrollDownBehavior
Miscellaneous Processors
------------------------
BotBlockDetector
~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.processor.BotBlockDetector
HashCrawlMapper
~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.processor.HashCrawlMapper
LexicalCrawlMapper
~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.processor.LexicalCrawlMapper
Post-Processors
---------------
CandidatesProcessor
~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.postprocessor.CandidatesProcessor
DispositionProcessor
~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.postprocessor.DispositionProcessor
ReschedulingProcessor
~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.postprocessor.ReschedulingProcessor
WARCWriterChainProcessor
~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.writer.WARCWriterChainProcessor
DnsResponseRecordBuilder
^^^^^^^^^^^^^^^^^^^^^^^^
.. bean-doc:: org.archive.modules.warc.DnsResponseRecordBuilder
FtpControlConversationRecordBuilder
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. bean-doc:: org.archive.modules.warc.FtpControlConversationRecordBuilder
FtpResponseRecordBuilder
^^^^^^^^^^^^^^^^^^^^^^^^
.. bean-doc:: org.archive.modules.warc.FtpResponseRecordBuilder
HttpRequestRecordBuilder
^^^^^^^^^^^^^^^^^^^^^^^^
.. bean-doc:: org.archive.modules.warc.HttpRequestRecordBuilder
HttpResponseRecordBuilder
^^^^^^^^^^^^^^^^^^^^^^^^^
.. bean-doc:: org.archive.modules.warc.HttpResponseRecordBuilder
MetadataRecordBuilder
^^^^^^^^^^^^^^^^^^^^^
.. bean-doc:: org.archive.modules.warc.MetadataRecordBuilder
RevisitRecordBuilder
^^^^^^^^^^^^^^^^^^^^
.. bean-doc:: org.archive.modules.warc.RevisitRecordBuilder
WhoisResponseRecordBuilder
^^^^^^^^^^^^^^^^^^^^^^^^^^
.. bean-doc:: org.archive.modules.warc.WhoisResponseRecordBuilder