* JobRelatedResource.java
add more property names to blacklist
route all relevant requests through new utilitiy getPropertyDescriptors method that patches instances as necessary to suppress problems
* RobotsPolicy.java
simplified unification of RobotsHonoring and RobotsExclusion policy classes; subclasses specialize
* ObeyRobotsPolicy.java
classic obey-robots-as-declared policy
* IgnoreRobotsPolicy.java
ignore declared robots policy
* CustomRobotsPolicy.java
follow custom-crafted policy in robots.txt format
* MostFavoredRobotsPolicy.java
follow the most-permissive policy available to a set of user-agents; optionally adopt ('masquerade') the user-agent whose rules are chosen
* RobotsExclusionPolicy.java, RobotsHonoringPolicy.java
deleted
* CrawlMetadata.java
use new string robotsPolicyName as main determinant of policy
maintain map of available policies ('obey','ignore', and any the operator added)
* CrawlServer.java
retain Robotstxt instance rather than customized RobotsExclusionPolicy
* PreconditionEnforcer.java
use full CrawlMetadata rather than just UserAgentProvider subset
consult RobotsPolicy looked up through CrawlMetadata for allow/disallow decision
* DispositionProcessor.java
update CrawlServer with Robotstxt only
consult Robotstxt for crawl-delay
* profile-crawler-beans.cxml (*2), migrate-template-crawler-beans.cxml, selftest-crawler-beans.cxml
remove old declarations/examples; add new
* ExtractorHTML.java, JerichoExtractorHTML.java
consult RobotsPolicy.obeyMetaRobotsNofollow for NOFOLLOW
* Robotstxt.java
new constructors for new use patterns
* **/*Test.java
update/remove to match new use patterns
* Frontier.java
add beginDisposition(), endDisposition() methods
* AbstractFrontier.java
implement beginDisposition(), endDisposition() methods to maintain dispositions-in-progress lock
* BdbFrontier.java
acquire exclusive dispositionInProgressLock before allowing checkpoint to begin; release at completion of checkpoint
* ToeThread.java
call beginDisposition before dispositionChain/finished(); call endDisposition after finished() and on exceptions
* DispositionChain.java
shed all prior relation to locking/checkpointing
* Checkpoint.java
remove previous attempt to use Lifecycle to catch error
* Checkpointable.java
@Autowired(required=false) annotation for setRecoveryCheckpoint
* CheckpointService.java
HasValidator implementation for post-build validity check; also check at launch (probably redundant)
* CheckpointValidator.java
ensure validity stamp exists on any present Checkpoint
* LaxURI.java
increase tolerance with regard to path-segments
* UURIFactory.java
allow digits in scheme regex
discard old supported-schemes code
* UURIFactoryTest.java
tests for roundtrip serialization of problematic hostlike-scheme and bars-in-path
* CrawlerLoggerModule.java
discard special-casing of URI scheme errors
* SchemeNotInSetDecideRule.java
rule to apply to URIs of unknown schemes; by default REJECTs those not usually handled by Heritrix
* **/profile-crawler-beans.cxml
add REJECT SchemeNotInSetDecideRule
* logging.properties
streamline to bare essentials; allow INFO for org.archive classes
* GenericReplayCharSequence.java, BdbUriUniqFilter.java
demote some INFO logging to FINE
* .classpath, pom.xml, org/apache/commons/pool/impl/*
remove references to and patched classes based on commons-pool
* WriterPool.java
move instance creation/round-robin-pool/dispose into this class
close (rather then recycle) extra writers when crawl slows (via tracking time of last need/rollover)
* WriterPoolMember.java
add isOversize() for external check of ready-to-rollover file
avoid rollover before every record (so that set of related WARC records aren't split between files)
* ARCWriterPool.java, WARCWriterPool.java
remove BasePoolableObjectFactory reference, implement class-appropriate makeWriter()
* ARCWriterPoolTest.java
adjust for new never-timeout behavior
* ARCWriterPoolProcessor.java WARCWriterPoolProcessor.java
adjust constructor calls
* WriterPoolProcessor.java
replace poolMaxWaitMs (which caused eventual complete timeout from waiting for writer) with maxWaitForIdleMs, a much smaller value which only controls how long a thread waits for an reusable writer before considering creating a new one. (If creation isn't allowed, threads will now wait indefinitely for a writer to become available.)
* Heritrix.java
make heritrix.hostname, heritrix.pid, heritrix.port available as global properties
* PropertyUtils.java, PropertyUtilsTest.java
utility methods to interpolate string values from one or several supplied Properties instances
* ArchiveUtils.java
utility methods to give timestamps guaranteed larger/different than any previously-issued timestamp
* WriterPoolSettings.java, WriterPool.java, ARCWriterPool.java, WARCWriterPool.java, ARCWriterPoolTest.java
improve field names, change 'suffix' to more general 'template'
* WriterPoolMember.java
replace 'suffix' with 'template' which is interpolated when specific name is needed
set default template to pattern extremely unlikely to generate duplicate filenames
change default prefix to 'WEB'
centralize creation of new basenames into generateNewBasename() method, which internalizes timetamp/serialNo minting and interpolation
* WriterPoolProcessor.java, ARCWriterProcessor.java, WARCWriterProcessor.java
serve as own WriterPoolSettings instance
replace 'suffix' with 'template'
* ARCWriter.java
accept (but truncate) 17-digit timestamps at creation
* DefaultWriterPoolSettings.java, TimestampSerialNo.java
delete as superfluous
* profile-crawler-beans.cxml
Added a commented-out configuration for new DiskSpaceMonitor
* DiskSpaceMonitor.java
Implements the required functionality
* package-info.java
Description of new package for monitoring utilities
* LowDiskPauseProcessor.java
Now marked as deprecated
* CheckpointService.java
use static method on Checkpoint for validity check
* Checkpoint.java
throw early-startup exception if attempting resumption of invalid checkpoint
* CheckpointService.java
(getAvailableCheckpointDirectories) return List, filtered to leave out directories without a 'valid' stamp; log WARNING to _out so that the attentive operator can manually clean-up/investigate
* ExtractorHTTP.java
add property inferRootPage; if true (not the default), always consider '/' inferred from all HTTP(S) URIs
* Hop.java, LinkContext.java
add new hop-type 'I' for inferred URIs; add corresponding LinkContext
* ExtractorImpliedURI.java
use new 'I'nferred types
* CrawlURI.java
improve comments for fetchAttempts methods
(resetForRescheduling) added to clear per-scheduling state
* WorkQueueFrontier.java
call resetForRescheduling just before scheduling-at-specific-time
* FrontierJournal.java, AbstractFrontier.java
renamings: prefer 'reenqueue' instead of 'reschedule' for describing simple (non-timed) retries
* AbstractFrontier.java
change config and defaults for 'inbound' and 'outbound': allowing any BlockinqQueue bean, of any size, to be specified if desired
* **/profile-crawler-beans.cxml
changed commented-out descriptions of defaults to reflect what's used if nothing is specified for 'outbound' and 'inbound'
gojomo