Bean Reference ============== .. note:: This reference is a work in progress and does not yet cover all available beans. For a more complete list of Heritrix beans please refer to the `javadoc `_. Core Beans ---------- ActionDirectory ~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.framework.ActionDirectory BdbCookieStore ~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.fetcher.BdbCookieStore BdbFrontier ~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.frontier.BdbFrontier BdbModule ~~~~~~~~~ .. bean-doc:: org.archive.bdb.BdbModule BdbServerCache ~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.net.BdbServerCache BdbUriUniqFilter ~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.util.BdbUriUniqFilter CheckpointService ~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.framework.CheckpointService CrawlController ~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.framework.CrawlController CrawlerLoggerModule ~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.reporting.CrawlerLoggerModule CrawlLimitEnforcer ~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.framework.CrawlLimitEnforcer CrawlMetadata ~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.CrawlMetadata CredentialStore ~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.credential.CredentialStore DecideRuleSequence ~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.DecideRuleSequence DiskSpaceMonitor ~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.monitor.DiskSpaceMonitor RulesCanonicalizationPolicy ~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.canonicalize.RulesCanonicalizationPolicy SheetOverlaysManager ~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.spring.SheetOverlaysManager StatisticsTracker ~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.reporting.StatisticsTracker TextSeedModule ~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.seeds.TextSeedModule Decide Rules ------------ AcceptDecideRule ~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.AcceptDecideRule ClassKeyMatchesRegexDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.deciderules.ClassKeyMatchesRegexDecideRule ContentLengthDecideRule ~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.ContentLengthDecideRule ContentTypeMatchesRegexDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.ContentTypeMatchesRegexDecideRule ContentTypeNotMatchesRegexDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.ContentTypeNotMatchesRegexDecideRule ExpressionDecideRule (contrib) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.ExpressionDecideRule ExternalGeoLocationDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.ExternalGeoLocationDecideRule FetchStatusDecideRule ~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.FetchStatusDecideRule FetchStatusMatchesRegexDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.FetchStatusMatchesRegexDecideRule FetchStatusNotMatchesRegexDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.FetchStatusNotMatchesRegexDecideRule HasViaDecideRule ~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.HasViaDecideRule HopCrossesAssignmentLevelDomainDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.HopCrossesAssignmentLevelDomainDecideRule HopsPathMatchesRegexDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.HopsPathMatchesRegexDecideRule IdenticalDigestDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.recrawl.IdenticalDigestDecideRule IpAddressSetDecideRule ~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.IpAddressSetDecideRule MatchesFilePatternDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.MatchesFilePatternDecideRule MatchesListRegexDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.MatchesListRegexDecideRule MatchesRegexDecideRule ~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.MatchesRegexDecideRule MatchesStatusCodeDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.MatchesStatusCodeDecideRule NotMatchesFilePatternDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.NotMatchesFilePatternDecideRule NotMatchesListRegexDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.NotMatchesListRegexDecideRule NotMatchesRegexDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.NotMatchesRegexDecideRule NotMatchesStatusCodeDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.NotMatchesStatusCodeDecideRule NotOnDomainsDecideRule ~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.surt.NotOnDomainsDecideRule NotOnHostsDecideRule ~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.surt.NotOnHostsDecideRule NotSurtPrefixedDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.surt.NotSurtPrefixedDecideRule OnDomainsDecideRule ~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.surt.OnDomainsDecideRule OnHostsDecideRule ~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.surt.OnHostsDecideRule PathologicalPathDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.PathologicalPathDecideRule PredicatedDecideRule ~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.PredicatedDecideRule PrerequisiteAcceptDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.PrerequisiteAcceptDecideRule RejectDecideRule ~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.RejectDecideRule ResourceLongerThanDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.ResourceLongerThanDecideRule ResourceNoLongerThanDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.ResourceNoLongerThanDecideRule ResponseContentLengthDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.ResponseContentLengthDecideRule SchemeNotInSetDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.SchemeNotInSetDecideRule ScriptedDecideRule ~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.ScriptedDecideRule SeedAcceptDecideRule ~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.SeedAcceptDecideRule SourceSeedDecideRule ~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.SourceSeedDecideRule SurtPrefixedDecideRule ~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.surt.SurtPrefixedDecideRule TooManyHopsDecideRule ~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.TooManyHopsDecideRule TooManyPathSegmentsDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.TooManyPathSegmentsDecideRule TransclusionDecideRule ~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.TransclusionDecideRule ViaSurtPrefixedDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.ViaSurtPrefixedDecideRule Candidate Processors -------------------- CandidateScoper ~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.prefetch.CandidateScoper FrontierPreparer ~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.prefetch.FrontierPreparer Pre-Fetch Processors -------------------- HostQuotaEnforcer (contrib) ~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.prefetch.HostQuotaEnforcer PreconditionEnforcer ~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.prefetch.PreconditionEnforcer Preselector ~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.prefetch.Preselector QuotaEnforcer ~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.prefetch.QuotaEnforcer RuntimeLimitEnforcer ~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.prefetch.RuntimeLimitEnforcer SourceQuotaEnforcer (contrib) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.prefetch.SourceQuotaEnforcer Fetch Processors ---------------- FetchDNS ~~~~~~~~ .. bean-doc:: org.archive.modules.fetcher.FetchDNS FetchFTP ~~~~~~~~ .. bean-doc:: org.archive.modules.fetcher.FetchFTP FetchHTTP ~~~~~~~~~ .. bean-doc:: org.archive.modules.fetcher.FetchHTTP FetchHTTP2 ~~~~~~~~~~ .. bean-doc:: org.archive.modules.fetcher.FetchHTTP2 FetchSFTP ~~~~~~~~~ .. bean-doc:: org.archive.modules.fetcher.FetchSFTP FetchWhois ~~~~~~~~~~ .. bean-doc:: org.archive.modules.fetcher.FetchWhois Link Extractors --------------- ExtractorCSS ~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorCSS ExtractorDOC ~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorDOC ExtractorHTML ~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorHTML AggressiveExtractorHTML ~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.AggressiveExtractorHTML JerichoExtractorHTML ~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.JerichoExtractorHTML ExtractorHTMLForms ~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.forms.ExtractorHTMLForms FormLoginProcessor ~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.forms.FormLoginProcessor ExtractorHTTP ~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorHTTP ExtractorImpliedURI ~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorImpliedURI ExtractorJS ~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorJS ConfigurableExtractorJS ~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ConfigurableExtractorJS ExtractorJson ~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorJson KnowledgableExtractorJS (contrib) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.KnowledgableExtractorJS ExtractorMultipleRegex ~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorMultipleRegex ExtractorPDF ~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorPDF ExtractorPDFContent (contrib) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorPDFContent ExtractorRobotsTxt ~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorRobotsTxt ExtractorSitemap ~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorSitemap ExtractorSVG ~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorSVG ExtractorSWF ~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorSWF ExtractorUniversal ~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorUniversal ExtractorURI ~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorURI ExtractorXML ~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorXML ExtractorYoutubeDL (contrib) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorYoutubeDL ExtractorYoutubeFormatStream (contrib) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorYoutubeFormatStream ExtractorYoutubeChannelFormatStream (contrib) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorYoutubeChannelFormatStream TrapSuppressExtractor ~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.TrapSuppressExtractor Browser Processor ----------------- BrowserProcessor ~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.processor.BrowserProcessor LocalBrowser ~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.processor.LocalBrowser SshBrowser ~~~~~~~~~~ .. bean-doc:: org.archive.crawler.processor.SshBrowser ExtractLinksBehavior ~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.behaviors.ExtractLinksBehavior ScrollDownBehavior ~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.behaviors.ScrollDownBehavior Miscellaneous Processors ------------------------ BotBlockDetector ~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.processor.BotBlockDetector HashCrawlMapper ~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.processor.HashCrawlMapper HTTPContentDigest ~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.HTTPContentDigest LexicalCrawlMapper ~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.processor.LexicalCrawlMapper ScriptedProcessor ~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.ScriptedProcessor Recrawl Processors ------------------ ContentDigestHistoryLoader ~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.recrawl.ContentDigestHistoryLoader ContentDigestHistoryStorer ~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.recrawl.ContentDigestHistoryStorer FetchHistoryProcessor ~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.recrawl.FetchHistoryProcessor PersistLoadProcessor ~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.recrawl.PersistLoadProcessor PersistLogProcessor ~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.recrawl.PersistLogProcessor PersistStoreProcessor ~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.recrawl.PersistStoreProcessor WbmPersistLoadProcessor (contrib) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.recrawl.wbm.WbmPersistLoadProcessor Post-Processors --------------- AMQPCrawlLogFeed (contrib) ~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.postprocessor.AMQPCrawlLogFeed AMQPPublishProcessor (contrib) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.AMQPPublishProcessor ARCWriterProcessor ~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.writer.ARCWriterProcessor CandidatesProcessor ~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.postprocessor.CandidatesProcessor DispositionProcessor ~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.postprocessor.DispositionProcessor KafkaCrawlLogFeed (contrib) ~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.postprocessor.KafkaCrawlLogFeed Kw3WriterProcessor ~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.writer.Kw3WriterProcessor MirrorWriterProcessor ~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.writer.MirrorWriterProcessor ReschedulingProcessor ~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.postprocessor.ReschedulingProcessor SupplementaryLinksScoper ~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.postprocessor.SupplementaryLinksScoper TroughCrawlLogFeed (contrib) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.postprocessor.TroughCrawlLogFeed WARCLimitEnforcer (contrib) ~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.postprocessor.WARCLimitEnforcer WARCWriterChainProcessor ~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.writer.WARCWriterChainProcessor DnsResponseRecordBuilder ^^^^^^^^^^^^^^^^^^^^^^^^ .. bean-doc:: org.archive.modules.warc.DnsResponseRecordBuilder FtpControlConversationRecordBuilder ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. bean-doc:: org.archive.modules.warc.FtpControlConversationRecordBuilder FtpResponseRecordBuilder ^^^^^^^^^^^^^^^^^^^^^^^^ .. bean-doc:: org.archive.modules.warc.FtpResponseRecordBuilder HttpRequestRecordBuilder ^^^^^^^^^^^^^^^^^^^^^^^^ .. bean-doc:: org.archive.modules.warc.HttpRequestRecordBuilder HttpResponseRecordBuilder ^^^^^^^^^^^^^^^^^^^^^^^^^ .. bean-doc:: org.archive.modules.warc.HttpResponseRecordBuilder MetadataRecordBuilder ^^^^^^^^^^^^^^^^^^^^^ .. bean-doc:: org.archive.modules.warc.MetadataRecordBuilder RevisitRecordBuilder ^^^^^^^^^^^^^^^^^^^^ .. bean-doc:: org.archive.modules.warc.RevisitRecordBuilder WhoisResponseRecordBuilder ^^^^^^^^^^^^^^^^^^^^^^^^^^ .. bean-doc:: org.archive.modules.warc.WhoisResponseRecordBuilder