lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-07-12 02:27:14 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
3.12.0
heritrix3/docs/bean-reference.rst
T
Alex Osborne 89e5ecc7d6 docs: replace javalang with an annotation processor 
This means we can rely on the Java compiler to parse the source code instead of the old python library which no longer works as it only supported Java 8 source code. The new processor generates a META-INF/heritrix-beans.json file for each module at compile time which the sphinx beandoc plugin then reads.

My intent is to eventually use the heritrix-beans.json files to also implement autocomplete and context-sensitive help in the Heritrix config editor.
2025-10-28 16:26:19 +09:00

580 lines
13 KiB
ReStructuredText
Raw Permalink Blame History

Bean Reference
==============
.. note::
This reference is a work in progress and does not yet cover all available beans. For a more complete list of
Heritrix beans please refer to the `javadoc <https://www.javadoc.io/doc/org.archive.heritrix/heritrix-modules>`_.
Core Beans
----------
ActionDirectory
~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.framework.ActionDirectory
BdbCookieStore
~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.fetcher.BdbCookieStore
BdbFrontier
~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.frontier.BdbFrontier
BdbModule
~~~~~~~~~
.. bean-doc:: org.archive.bdb.BdbModule
BdbServerCache
~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.net.BdbServerCache
BdbUriUniqFilter
~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.util.BdbUriUniqFilter
CheckpointService
~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.framework.CheckpointService
CrawlController
~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.framework.CrawlController
CrawlerLoggerModule
~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.reporting.CrawlerLoggerModule
CrawlLimitEnforcer
~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.framework.CrawlLimitEnforcer
CrawlMetadata
~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.CrawlMetadata
CredentialStore
~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.credential.CredentialStore
DecideRuleSequence
~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.DecideRuleSequence
DiskSpaceMonitor
~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.monitor.DiskSpaceMonitor
RulesCanonicalizationPolicy
~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.canonicalize.RulesCanonicalizationPolicy
SheetOverlaysManager
~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.spring.SheetOverlaysManager
StatisticsTracker
~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.reporting.StatisticsTracker
TextSeedModule
~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.seeds.TextSeedModule
Decide Rules
------------
AcceptDecideRule
~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.AcceptDecideRule
ClassKeyMatchesRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.deciderules.ClassKeyMatchesRegexDecideRule
ContentLengthDecideRule
~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.ContentLengthDecideRule
ContentTypeMatchesRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.ContentTypeMatchesRegexDecideRule
ContentTypeNotMatchesRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.ContentTypeNotMatchesRegexDecideRule
ExpressionDecideRule (contrib)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.ExpressionDecideRule
ExternalGeoLocationDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.ExternalGeoLocationDecideRule
FetchStatusDecideRule
~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.FetchStatusDecideRule
FetchStatusMatchesRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.FetchStatusMatchesRegexDecideRule
FetchStatusNotMatchesRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.FetchStatusNotMatchesRegexDecideRule
HasViaDecideRule
~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.HasViaDecideRule
HopCrossesAssignmentLevelDomainDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.HopCrossesAssignmentLevelDomainDecideRule
HopsPathMatchesRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.HopsPathMatchesRegexDecideRule
IdenticalDigestDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.recrawl.IdenticalDigestDecideRule
IpAddressSetDecideRule
~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.IpAddressSetDecideRule
MatchesFilePatternDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.MatchesFilePatternDecideRule
MatchesListRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.MatchesListRegexDecideRule
MatchesRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.MatchesRegexDecideRule
MatchesStatusCodeDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.MatchesStatusCodeDecideRule
NotMatchesFilePatternDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.NotMatchesFilePatternDecideRule
NotMatchesListRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.NotMatchesListRegexDecideRule
NotMatchesRegexDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.NotMatchesRegexDecideRule
NotMatchesStatusCodeDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.NotMatchesStatusCodeDecideRule
NotOnDomainsDecideRule
~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.surt.NotOnDomainsDecideRule
NotOnHostsDecideRule
~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.surt.NotOnHostsDecideRule
NotSurtPrefixedDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.surt.NotSurtPrefixedDecideRule
OnDomainsDecideRule
~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.surt.OnDomainsDecideRule
OnHostsDecideRule
~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.surt.OnHostsDecideRule
PathologicalPathDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.PathologicalPathDecideRule
PredicatedDecideRule
~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.PredicatedDecideRule
PrerequisiteAcceptDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.PrerequisiteAcceptDecideRule
RejectDecideRule
~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.RejectDecideRule
ResourceLongerThanDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.ResourceLongerThanDecideRule
ResourceNoLongerThanDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.ResourceNoLongerThanDecideRule
ResponseContentLengthDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.ResponseContentLengthDecideRule
SchemeNotInSetDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.SchemeNotInSetDecideRule
ScriptedDecideRule
~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.ScriptedDecideRule
SeedAcceptDecideRule
~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.SeedAcceptDecideRule
SourceSeedDecideRule
~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.SourceSeedDecideRule
SurtPrefixedDecideRule
~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.surt.SurtPrefixedDecideRule
TooManyHopsDecideRule
~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.TooManyHopsDecideRule
TooManyPathSegmentsDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.TooManyPathSegmentsDecideRule
TransclusionDecideRule
~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.TransclusionDecideRule
ViaSurtPrefixedDecideRule
~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.deciderules.ViaSurtPrefixedDecideRule
Candidate Processors
--------------------
CandidateScoper
~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.prefetch.CandidateScoper
FrontierPreparer
~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.prefetch.FrontierPreparer
Pre-Fetch Processors
--------------------
PreconditionEnforcer
~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.prefetch.PreconditionEnforcer
Preselector
~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.prefetch.Preselector
Fetch Processors
----------------
FetchDNS
~~~~~~~~
.. bean-doc:: org.archive.modules.fetcher.FetchDNS
FetchFTP
~~~~~~~~
.. bean-doc:: org.archive.modules.fetcher.FetchFTP
FetchHTTP
~~~~~~~~~
.. bean-doc:: org.archive.modules.fetcher.FetchHTTP
FetchHTTP2
~~~~~~~~~~
.. bean-doc:: org.archive.modules.fetcher.FetchHTTP2
FetchSFTP
~~~~~~~~~
.. bean-doc:: org.archive.modules.fetcher.FetchSFTP
FetchWhois
~~~~~~~~~~
.. bean-doc:: org.archive.modules.fetcher.FetchWhois
Link Extractors
---------------
ExtractorCSS
~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorCSS
ExtractorDOC
~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorDOC
ExtractorHTML
~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorHTML
AggressiveExtractorHTML
~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.AggressiveExtractorHTML
JerichoExtractorHTML
~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.JerichoExtractorHTML
ExtractorHTMLForms
~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.forms.ExtractorHTMLForms
ExtractorHTTP
~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorHTTP
ExtractorImpliedURI
~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorImpliedURI
ExtractorJS
~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorJS
KnowledgableExtractorJS (contrib)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.KnowledgableExtractorJS
ExtractorMultipleRegex
~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorMultipleRegex
ExtractorPDF
~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorPDF
ExtractorPDFContent (contrib)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorPDFContent
ExtractorRobotsTxt
~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorRobotsTxt
ExtractorSitemap
~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorSitemap
ExtractorSWF
~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorSWF
ExtractorUniversal
~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorUniversal
ExtractorURI
~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorURI
ExtractorXML
~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorXML
ExtractorYoutubeDL (contrib)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorYoutubeDL
ExtractorYoutubeFormatStream (contrib)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorYoutubeFormatStream
ExtractorYoutubeChannelFormatStream (contrib)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.ExtractorYoutubeChannelFormatStream
TrapSuppressExtractor
~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.extractor.TrapSuppressExtractor
Browser Processor
-----------------
BrowserProcessor
~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.processor.BrowserProcessor
ExtractLinksBehavior
~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.behaviors.ExtractLinksBehavior
ScrollDownBehavior
~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.behaviors.ScrollDownBehavior
Post-Processors
---------------
CandidatesProcessor
~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.postprocessor.CandidatesProcessor
DispositionProcessor
~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.postprocessor.DispositionProcessor
ReschedulingProcessor
~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.crawler.postprocessor.ReschedulingProcessor
WARCWriterChainProcessor
~~~~~~~~~~~~~~~~~~~~~~~~
.. bean-doc:: org.archive.modules.writer.WARCWriterChainProcessor
DnsResponseRecordBuilder
^^^^^^^^^^^^^^^^^^^^^^^^
.. bean-doc:: org.archive.modules.warc.DnsResponseRecordBuilder
FtpControlConversationRecordBuilder
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. bean-doc:: org.archive.modules.warc.FtpControlConversationRecordBuilder
FtpResponseRecordBuilder
^^^^^^^^^^^^^^^^^^^^^^^^
.. bean-doc:: org.archive.modules.warc.FtpResponseRecordBuilder
HttpRequestRecordBuilder
^^^^^^^^^^^^^^^^^^^^^^^^
.. bean-doc:: org.archive.modules.warc.HttpRequestRecordBuilder
HttpResponseRecordBuilder
^^^^^^^^^^^^^^^^^^^^^^^^^
.. bean-doc:: org.archive.modules.warc.HttpResponseRecordBuilder
MetadataRecordBuilder
^^^^^^^^^^^^^^^^^^^^^
.. bean-doc:: org.archive.modules.warc.MetadataRecordBuilder
RevisitRecordBuilder
^^^^^^^^^^^^^^^^^^^^
.. bean-doc:: org.archive.modules.warc.RevisitRecordBuilder
WhoisResponseRecordBuilder
^^^^^^^^^^^^^^^^^^^^^^^^^^
.. bean-doc:: org.archive.modules.warc.WhoisResponseRecordBuilder
Reference in New Issue View Git Blame Copy Permalink