diff --git a/engine/src/main/java/org/archive/crawler/package.html b/engine/src/main/java/org/archive/crawler/package.html index 055311fd..03de90c9 100644 --- a/engine/src/main/java/org/archive/crawler/package.html +++ b/engine/src/main/java/org/archive/crawler/package.html @@ -46,15 +46,15 @@ such essentials as the CandidateURI and CrawlURI classes that wrap the discovered URIs for processing. -

admin

+

restlet

- {@link org.archive.crawler.admin org.archive.crawler.admin} + {@link org.archive.crawler.restlet org.archive.crawler.restlet}

- The admin package contains classes that are used by the Web UI. - This includes some core classes and a specific implementation of the - Statistics Tracking interface found in the framework - package that is designed to provide the UI with information about - ongoing crawls. + The restlet package implements the Web UI and REST API. Its + resources provide access to the crawl engine, jobs, configuration, + scripts, and reports. The + {@link org.archive.crawler.reporting.StatisticsTracker StatisticsTracker} + supplies progress information and reports for ongoing crawls.

Pluggable modules

@@ -96,7 +96,7 @@

  • Post-processing chain Each of these chains contain any number of processors. The processors - all inherit from a generic {@link org.archive.crawler.framework.Processor + all inherit from a generic {@link org.archive.modules.Processor Processor}. While the processors are divided into the five categories above that is strictly a high level configuration and any processor can be in any chain (although doing link extraction before fetching a document is clearly of no @@ -105,12 +105,12 @@ Numerous processors are provided with Heritrix in the following packages:
    {@link org.archive.crawler.prefetch org.archive.crawler.prefetch} package contains processors run before the URI is fetched from the Internet.
    - {@link org.archive.crawler.fetcher org.archive.crawler.fetcher} package + {@link org.archive.modules.fetcher org.archive.modules.fetcher} package contains processors that fetch URI from the Internet. Typically each processor handles a different protocol.
    - {@link org.archive.crawler.extractor org.archive.crawler.extractor} package + {@link org.archive.modules.extractor org.archive.modules.extractor} package contains processors that perform link extractions on various document types.
    - {@link org.archive.crawler.writer org.archive.crawler.writer} package contains + {@link org.archive.modules.writer org.archive.modules.writer} package contains a processor that writes an ARC file with the fetched document.
    {@link org.archive.crawler.postprocessor org.archive.crawler.postprocessor} package contain processors that do wrapup on the processing, reporting links @@ -131,10 +131,10 @@ of a crawl can be more easily achived using one of the existing scopes and modifying it with appropriate filters.

    - {@link org.archive.crawler.framework.CrawlScope CrawlScope} - Base class for - scopes.
    - {@link org.archive.crawler.scope org.archive.crawler.scope} package. Contains - provided scopes. + {@link org.archive.modules.deciderules.DecideRule DecideRule} is the base + class for rules used to define scope. The + {@link org.archive.modules.deciderules org.archive.modules.deciderules} + package contains the provided rules.

    Statistics Tracking

    @@ -143,16 +143,12 @@

    These modules can both interrogate the Frontier for what sparse date it exposes but they can also subscribe to - {@link org.archive.crawler.event.CrawlURIDispositionListener Crawled URI + {@link org.archive.crawler.event.CrawlURIDispositionEvent Crawled URI Disposition} events to monitor the completion of each URI that is processed.

    - An interface for {@link org.archive.crawler.framework.StatisticsTracking - statistics tracking} is provided as well as a partial implementation - ({@link org.archive.crawler.framework.AbstractTracker AbstractTracker}) - that does much of the work common to most statistics tracking modules. -

    - Furthermore the admin package implements a statistics tracking - module ({@link org.archive.crawler.admin.StatisticsTracker StatisticsTracker}) + The {@link org.archive.crawler.reporting org.archive.crawler.reporting} + package provides a statistics tracking module + ({@link org.archive.crawler.reporting.StatisticsTracker StatisticsTracker}) that generates a log of the crawlers progress as well as providing information that the UI uses. It also compiles end-of-crawl reports that contain all of the information it has gathered in the course of the crawl.
    diff --git a/engine/src/main/java/org/archive/crawler/postprocessor/CandidatesProcessor.java b/engine/src/main/java/org/archive/crawler/postprocessor/CandidatesProcessor.java index 6e2e7367..d6bb62a3 100644 --- a/engine/src/main/java/org/archive/crawler/postprocessor/CandidatesProcessor.java +++ b/engine/src/main/java/org/archive/crawler/postprocessor/CandidatesProcessor.java @@ -123,7 +123,7 @@ public class CandidatesProcessor extends Processor { return (Boolean) kp.get("processErrorOutlinks"); } /** - * If true, outlinks from status codes <200 and >=400 + * If true, outlinks from status codes <200 and >=400 * will be sent through candidates processing. Default is * false. */ diff --git a/modules/src/main/java/org/archive/modules/extractor/ExtractorSitemap.java b/modules/src/main/java/org/archive/modules/extractor/ExtractorSitemap.java index 819d9665..9ad6e6be 100644 --- a/modules/src/main/java/org/archive/modules/extractor/ExtractorSitemap.java +++ b/modules/src/main/java/org/archive/modules/extractor/ExtractorSitemap.java @@ -23,7 +23,7 @@ import crawlercommons.sitemaps.UnknownFormatException; /** * - * @author Andrew Jackson + * @author Andrew Jackson <Andrew.Jackson@bl.uk> * */ public class ExtractorSitemap extends ContentExtractor { diff --git a/modules/src/main/java/org/archive/modules/warc/WARCRecordBuilder.java b/modules/src/main/java/org/archive/modules/warc/WARCRecordBuilder.java index c063386e..8bf03f8a 100644 --- a/modules/src/main/java/org/archive/modules/warc/WARCRecordBuilder.java +++ b/modules/src/main/java/org/archive/modules/warc/WARCRecordBuilder.java @@ -35,7 +35,7 @@ public interface WARCRecordBuilder { * @param concurrentTo implementations should do this: *

        if (concurrentTo != null) {
          *        recordInfo.addExtraHeader(HEADER_KEY_CONCURRENT_TO,
    -     *                "<" + concurrentTo + ">");
    +     *                "<" + concurrentTo + ">");
          *    }
    * @return the freshly built warc record * @throws IOException