* JobRelatedResource.java
add more property names to blacklist
route all relevant requests through new utilitiy getPropertyDescriptors method that patches instances as necessary to suppress problems
* RobotsPolicy.java
simplified unification of RobotsHonoring and RobotsExclusion policy classes; subclasses specialize
* ObeyRobotsPolicy.java
classic obey-robots-as-declared policy
* IgnoreRobotsPolicy.java
ignore declared robots policy
* CustomRobotsPolicy.java
follow custom-crafted policy in robots.txt format
* MostFavoredRobotsPolicy.java
follow the most-permissive policy available to a set of user-agents; optionally adopt ('masquerade') the user-agent whose rules are chosen
* RobotsExclusionPolicy.java, RobotsHonoringPolicy.java
deleted
* CrawlMetadata.java
use new string robotsPolicyName as main determinant of policy
maintain map of available policies ('obey','ignore', and any the operator added)
* CrawlServer.java
retain Robotstxt instance rather than customized RobotsExclusionPolicy
* PreconditionEnforcer.java
use full CrawlMetadata rather than just UserAgentProvider subset
consult RobotsPolicy looked up through CrawlMetadata for allow/disallow decision
* DispositionProcessor.java
update CrawlServer with Robotstxt only
consult Robotstxt for crawl-delay
* profile-crawler-beans.cxml (*2), migrate-template-crawler-beans.cxml, selftest-crawler-beans.cxml
remove old declarations/examples; add new
* ExtractorHTML.java, JerichoExtractorHTML.java
consult RobotsPolicy.obeyMetaRobotsNofollow for NOFOLLOW
* Robotstxt.java
new constructors for new use patterns
* **/*Test.java
update/remove to match new use patterns
* Frontier.java
add beginDisposition(), endDisposition() methods
* AbstractFrontier.java
implement beginDisposition(), endDisposition() methods to maintain dispositions-in-progress lock
* BdbFrontier.java
acquire exclusive dispositionInProgressLock before allowing checkpoint to begin; release at completion of checkpoint
* ToeThread.java
call beginDisposition before dispositionChain/finished(); call endDisposition after finished() and on exceptions
* DispositionChain.java
shed all prior relation to locking/checkpointing
* Checkpoint.java
remove previous attempt to use Lifecycle to catch error
* Checkpointable.java
@Autowired(required=false) annotation for setRecoveryCheckpoint
* CheckpointService.java
HasValidator implementation for post-build validity check; also check at launch (probably redundant)
* CheckpointValidator.java
ensure validity stamp exists on any present Checkpoint
* LaxURI.java
increase tolerance with regard to path-segments
* UURIFactory.java
allow digits in scheme regex
discard old supported-schemes code
* UURIFactoryTest.java
tests for roundtrip serialization of problematic hostlike-scheme and bars-in-path
* CrawlerLoggerModule.java
discard special-casing of URI scheme errors
* SchemeNotInSetDecideRule.java
rule to apply to URIs of unknown schemes; by default REJECTs those not usually handled by Heritrix
* **/profile-crawler-beans.cxml
add REJECT SchemeNotInSetDecideRule
* logging.properties
streamline to bare essentials; allow INFO for org.archive classes
* GenericReplayCharSequence.java, BdbUriUniqFilter.java
demote some INFO logging to FINE
* Heritrix.java
make heritrix.hostname, heritrix.pid, heritrix.port available as global properties
* PropertyUtils.java, PropertyUtilsTest.java
utility methods to interpolate string values from one or several supplied Properties instances
* ArchiveUtils.java
utility methods to give timestamps guaranteed larger/different than any previously-issued timestamp
* WriterPoolSettings.java, WriterPool.java, ARCWriterPool.java, WARCWriterPool.java, ARCWriterPoolTest.java
improve field names, change 'suffix' to more general 'template'
* WriterPoolMember.java
replace 'suffix' with 'template' which is interpolated when specific name is needed
set default template to pattern extremely unlikely to generate duplicate filenames
change default prefix to 'WEB'
centralize creation of new basenames into generateNewBasename() method, which internalizes timetamp/serialNo minting and interpolation
* WriterPoolProcessor.java, ARCWriterProcessor.java, WARCWriterProcessor.java
serve as own WriterPoolSettings instance
replace 'suffix' with 'template'
* ARCWriter.java
accept (but truncate) 17-digit timestamps at creation
* DefaultWriterPoolSettings.java, TimestampSerialNo.java
delete as superfluous
* profile-crawler-beans.cxml
Added a commented-out configuration for new DiskSpaceMonitor
* DiskSpaceMonitor.java
Implements the required functionality
* package-info.java
Description of new package for monitoring utilities
* LowDiskPauseProcessor.java
Now marked as deprecated
* CheckpointService.java
use static method on Checkpoint for validity check
* Checkpoint.java
throw early-startup exception if attempting resumption of invalid checkpoint
* CheckpointService.java
(getAvailableCheckpointDirectories) return List, filtered to leave out directories without a 'valid' stamp; log WARNING to _out so that the attentive operator can manually clean-up/investigate
* CrawlURI.java
improve comments for fetchAttempts methods
(resetForRescheduling) added to clear per-scheduling state
* WorkQueueFrontier.java
call resetForRescheduling just before scheduling-at-specific-time
* FrontierJournal.java, AbstractFrontier.java
renamings: prefer 'reenqueue' instead of 'reschedule' for describing simple (non-timed) retries
* AbstractFrontier.java
change config and defaults for 'inbound' and 'outbound': allowing any BlockinqQueue bean, of any size, to be specified if desired
* **/profile-crawler-beans.cxml
changed commented-out descriptions of defaults to reflect what's used if nothing is specified for 'outbound' and 'inbound'
* EngineResource.java
add 'exit' control area at bottom of page
allow JVM-exit if confirm-box checked and either no jobs built/running or extra confirm boxes for each built job checked
* Flash.java
fix to allow multiple flashes per page load
* JobResource.java, CrawlJob.java
move useful status string method to CrawlJob
* BloomFilter64bit.java
include the split-to-subarrays (for larger bitfields) and round-up-to-power-of-2 (for performance) options previously in largely-redundant classes
fit a number of problems with int/long overflow and bitwise ops
add methods for reporting/testing
* BloomFilter.java
add methods for reporting/testing
* BloomFilterTest.java, BloomFilter64bitTest.java
more extensive tests, including two lengthy tests of default/oversized blooms usually disabled by renaming
* BloomFilter32bit.java, BloomFilter32bitSplit.java, BloomFilter32bp2.java, BloomFilter32bp2Split.java
deleted as buggy or redundant
* BenchmarkBlooms.java
move to test source dir
* BloomUriUniqFilter.java
change to accept filter instance (rather than parameters) for added configuration flexibility
fix comments
* BloomUriUniqFilterTest.java
supply filter not paramters
* dist/src/main/conf/jobs/profile-defaults/profile-crawler-beans.cxml
* engine/src/main/resources/org/archive/crawler/restlet/profile-crawler-beans.cxml
add intended REJECT decision for the late (rules[4]) MatchesListRegexDecideRule
* Engine.java
suppress WARNING for common case of harmless failure during rescans: job already exists
* EngineResource.java
suppress rescan on each reload (ODD: steve had already committed this change -- I got the commit email -- but now the repo version is missing it. Sourceforge problems?)
* WorkQueueFrontier.java
override schedule() to keep unique-testing in calling thread; may remove bottleneck shown by time-profiling and thread-observation (most ToeThreads stuck in candidatesProcessor, schedule()ing to frontier)
* AbstractFrontier.java
add comment about implementation vs. subclass override
* Heritrix.java
allow [username:]password to come from file, supplied to '-a' with '@' prefix
also, clean up options names/help text to match current behavior
also, offer warning about particularly weak (<8 characters, all letters, all numbers) passwords
* Engine.java
tolerate (with return-false-forfailure rather than NPE) non-job directories
* EngineTest.java
use empty temp job directory rather than (possibly full-of-junk) general junit-tests directory
* GenericReplayCharSequence.java
* Scoper.java
* FetchFTP.java
* FetchHTTP.java
demote a bunch of INFO logging to FINE so that be default, console isn't filled with output for every URI handled by a busy crawl
* CheckpointService.java
reorganize so that checkpoint timedTask is reset whenever interval is reset on a running crawl
* CheckpointSuccessEvent.java, CrawlJob.java
report successes as new kind of ApplicationEvent; propagagte to job log
* BdbModule.java
remove obsolete unconsulted checkpointCopyLogs setting
gojomo