add 'trackSources' to make hosts-by-source-tag report optional in large crawls, where it is very expensive (and an OOME risk until other changes are also made)
* Report.java
make StatisticsTracker stats an argument to write() instead of a field; new properties shouldReportAtEndOfCrawl and shouldReportDuringCrawl, both default true, but configurable if desired, replacing the LIVE_REPORTS/END_REPORTS stuff in StatisticsTracker
* StatisticsTracker.java
new property "reports", a List<Report>, which defaults (with lazy initialization) to the same reports we've always had, replacing hard-coded lists of reports classes and related code
* profile-crawler-beans.cxml
default reports list, commented out
* SeedsReport.java FrontierSummaryReport.java MimetypesReport.java SourceTagsReport.java FrontierNonemptyReport.java CrawlSummaryReport.java ResponseCodeReport.java HostsReport.java ProcessorsReport.java ToeThreadsReport.java
update write(PrintWriter, StatisticsTracker) arguments
* JobResource.java
replace use of StatisticsTracker.LIVE_REPORTS
* IdenticalDigestDecideRule.java
fix javadoc typo
* CoreAttributeConstants.java
"history-good-to-store" is history
* RecrawlAttributeConstants.java
new "write-tag" constant
* PersistProcessor.java
remove defunct "history-good-to-store" logic; nadd new option onlyStoreIfWriteTagPresent, and update shouldStore() to respect that option
* WriterPoolProcessor.java
remove defunct "history-good-to-store" logic; new method copyForwardWriteTagIfDupe(); override innerRejectProcess() to call copyForwardWriteTagIfDupe()
* ARCWriterProcessor.java, WARCWriterProcessor.java
remove defunct "history-good-to-store" logic; add writeTag to history on write; call copyForwardWriteTagIfDupe() on skipIdenticalDigest
contributed by Adam Wilmer
* FetchHTTP.java
properties, setup for http proxy user/password
* profile-crawler-beans.cxml
commented-out example settings for proxy auth properties
* ActionDirectory.java, CrawlerLoggerModule.java, StatisticsTracker.java, SurtPrefixedDecideRule.java, WriterPoolProcessor.java, profile-crawler-beans.cxml
use camelcase ${launchId}
* CheckpointService.java, JobResource.java
rename getAvailableCheckpointDirectories() to findAvailableCheckpointDirectories() so that it's not treated as a bean property
* WriterPoolProcessor.java, WriterPoolSettingsData.java, WriterPoolMember.java
rename getOutputDirs() to calcOutputDirs() so that it's not treated as a bean property
* ConfigPath.java
let ConfigPathConfigurer to do interpolation of ${launchId}
* ConfigFile.java
let ConfigPathConfigurer do the snapshotting of config file
* CrawlJob.java, PathSharingContext.java
remove launchDir initialization out of CrawlJob into PathSharingContext
* ConfigPathConfigurer.java
- remove special handling of WriterPoolProcessor store paths; instead, look in beans for ConfigPaths within Iterables
- have each ConfigPath hold reference to this ConfigPathConfigurer to use for interpolating ${launchId} and snapshotting config files
* Recorder
object to unsupported content-encodings when first set
* FetchHTTP
note as annotation unsupported content-encodings
* Link
not as annotation when base URI is used for absent via
* ExtractorHTML
downgrade char-sequence reading problem (often a chunking problem) to WARNING from SEVERE
* CrawlJob.java
decided to go with plain timestamp17 as launch id, no "launch-" prefix
* ConfigPathConfigurer.java
refactored special remembering of WriterPoolProcessor storePaths out of fixupPaths()
* UriProcessingFormatter, Preformatter, GenerationFileHandler
improve preformat-outside-synchronized optimization so that the LogRecord/CrawlURI doesn't linger until next displaces it
* CrawlURI
(processingCleanup) null more of last-processing-run collected values
* WriterPoolProcessor.java, ARCWriterProcessor.java, WARCWriterProcessor.java
change type of storePaths to List<ConfigPath> and handle appropriately
* ConfigPathConfigurer.java
fixupPaths() - old code did not touch WriterPoolProcessor storePaths, since they're deeply nested inside the bean, but they need to be remembered for later interpolation of ${launch-id}, so add special handling
* HardLinker.java
renamed FilesystemLinkMaker.java
* FilesystemLinkMaker.java
add support for symbolic links
* CLibrary.java
new method symlink()
* BdbModule.java
use new class name FilesystemLinkMaker
* CrawlJob.java
at crawl launch, create launch directory launch-{timestamp17}, copy cxml there, symlink "current" to launch dir, inform ConfigPaths
* ConfigPath.java
interpolate ${launch-id} in configured paths
* ConfigFile.java
obtainReader() - snapshot config files to launch dir when they are read
* ActionDirectory.java
default doneDir now ${launch-id}/actions-done
actOn() - symlink from old style done dir action/done to done files
* SurtPrefixedDecideRule.java
default surtsDumpFile now ${launch-id}/surts.dump
pathsFixedUp() - this gets called at build time, but we don't want anything written to disk until launch time, so remove call to dumpSurtPrefixSet() here
* CrawlerLoggerModule.java
default logs dir now ${launch-id}/logs
* StatisticsTracker.java
default reports dir now ${launch-id}/reports
* WriterPoolProcessor.java
default writer base path now ${launch-id}
* profile-crawler-beans.cxml
update to reflect new default paths under launch dirs
* PropertyUtils.java
fix javadoc typo
* WriterPool
make new LARGEST_MAX_ACTIVE (255) the capacity of the reuse-pool, so that the maxActive property may vary up to that and still work; also handle more gracefully the failure of trying to use a larger number than the capacity (writer gets closed early rather than left hanging open)
* pom.xml, .classpath
update references to necessary spring-3.0.5 packages
* SeedModule
merge rather than replace event listeners, so that (now later) autowiring doesn't clobber the self-insertion done by anonymous (non-top-level) SurtPrefixedDecideRule bean
* HeritrixLifecycleProcessor, PathScharingContext
use this new non-default LifecycleProcessor to avoid new Spring3 behavior of auto-start()ing context on refresh(build)
* CrawlController
example of using @Value annotation to set default value: will offer benefits for auto-discovery of defaults for configuration interface, or enforcing maximally-explicit configurations (see [HER-1897])
* profile-crawler-beans.cxml, selftest-crawler-beans.cxml
update templates with spring3 preamble/boilerplate
[HER-1910] H3: job-related pages very slow to render (pause mid-way through) in large crawls
* BdbModule
option to reuse bdb data on StoredQueue-creation
* Checkpoint
new saveWriter, loadReader utilities for creating extra checkpoint files (for the long list of ready queues)
* BdbFrontier
on checkpoint, remember ordered list of inProcess/ready/snoozed queues (so same queues are active upon resume)
on resume, reuse old retired/inactive StoredQueue data, and reload active queues from newly-saved list
* StoredQueue
restore tailIndex properly when resuing old data
more-efficient size() calculation (for HER-1910)
* JobRelatedResource
catch InvalidPropertyException, render as red error message rather than fouling entire trace of beans to web UI
* StatisticsTracker
rename getSnapshots to listSnapshots to prevent interpretation as (invalid) bean-property
* HardLinker.java
makeHardLink() - wrap unix link() and windows CreateHardLink() and choose depending on platform
* BdbModule.java
doCheckpoint() - use HardLinker.makeHardLink()
* TextSeedModule
refactor announceSeeds to occur in background thread if non-default blockAwaitingSeedLines value is set
signal CountDownLatch on each line, allowing calling thread to proceed at right count
* profile-crawler-beans.cxml
commented-out blockAwaitingSeedLines default settings (-1, meaning wait for all seed lines)
* CrawlController
change pauseAtFinish to runWhileEmpty
add EMPTY state
replace isStateRunning() with isActive() (RUNNING or EMPTY)
trigger finish-test off frontier EMPTY rather than PAUSE
in logging say 'running' rather than 'resumed'
* Frontier
add EMPTY state
* AbstractFrontier
handle EMPTY like RUNNING, transition EMPTY<->RUNNING when appropriate
* StatisticsTracker
handle EMPTY analogous to RUNNING
in logging say 'running' rather than 'resumed'
* CrawlJob, DiskSpaceMonitor
replace isStateRunning() with isActive()
* GenericReplayCharSequence.java
decode() - read another character to check for more content, rather than consult BufferedReader.ready(), since the latter sometimes returns false even when there is more to read
* ReplayCharSequence, GenericReplayCharSequence
use Charset instance rather than String-name
* Recorder
use Charset instance ratehr than String-name
(getContentReplayCharSequence) avoid reusing cached ReplayCharSequence when encoding-in-use has changed since it was created
(getContentReplayPrefixString) allow requested specific charset-interpretation
* ExtractorHTML, ExtractorXML
work with Charset instances
double-check that in-content-declarations are self-consistent before using
recycle matcher instances
change minor charset problems/decisions to a crawl.log annotation
gojomo