* profile-crawler-beans.cxml
Added a commented-out configuration for new DiskSpaceMonitor
* DiskSpaceMonitor.java
Implements the required functionality
* package-info.java
Description of new package for monitoring utilities
* LowDiskPauseProcessor.java
Now marked as deprecated
* CheckpointService.java
use static method on Checkpoint for validity check
* Checkpoint.java
throw early-startup exception if attempting resumption of invalid checkpoint
* CheckpointService.java
(getAvailableCheckpointDirectories) return List, filtered to leave out directories without a 'valid' stamp; log WARNING to _out so that the attentive operator can manually clean-up/investigate
* CrawlURI.java
improve comments for fetchAttempts methods
(resetForRescheduling) added to clear per-scheduling state
* WorkQueueFrontier.java
call resetForRescheduling just before scheduling-at-specific-time
* FrontierJournal.java, AbstractFrontier.java
renamings: prefer 'reenqueue' instead of 'reschedule' for describing simple (non-timed) retries
* AbstractFrontier.java
change config and defaults for 'inbound' and 'outbound': allowing any BlockinqQueue bean, of any size, to be specified if desired
* **/profile-crawler-beans.cxml
changed commented-out descriptions of defaults to reflect what's used if nothing is specified for 'outbound' and 'inbound'
* EngineResource.java
add 'exit' control area at bottom of page
allow JVM-exit if confirm-box checked and either no jobs built/running or extra confirm boxes for each built job checked
* Flash.java
fix to allow multiple flashes per page load
* JobResource.java, CrawlJob.java
move useful status string method to CrawlJob
* BloomFilter64bit.java
include the split-to-subarrays (for larger bitfields) and round-up-to-power-of-2 (for performance) options previously in largely-redundant classes
fit a number of problems with int/long overflow and bitwise ops
add methods for reporting/testing
* BloomFilter.java
add methods for reporting/testing
* BloomFilterTest.java, BloomFilter64bitTest.java
more extensive tests, including two lengthy tests of default/oversized blooms usually disabled by renaming
* BloomFilter32bit.java, BloomFilter32bitSplit.java, BloomFilter32bp2.java, BloomFilter32bp2Split.java
deleted as buggy or redundant
* BenchmarkBlooms.java
move to test source dir
* BloomUriUniqFilter.java
change to accept filter instance (rather than parameters) for added configuration flexibility
fix comments
* BloomUriUniqFilterTest.java
supply filter not paramters
* dist/src/main/conf/jobs/profile-defaults/profile-crawler-beans.cxml
* engine/src/main/resources/org/archive/crawler/restlet/profile-crawler-beans.cxml
add intended REJECT decision for the late (rules[4]) MatchesListRegexDecideRule
* Engine.java
suppress WARNING for common case of harmless failure during rescans: job already exists
* EngineResource.java
suppress rescan on each reload (ODD: steve had already committed this change -- I got the commit email -- but now the repo version is missing it. Sourceforge problems?)
* WorkQueueFrontier.java
override schedule() to keep unique-testing in calling thread; may remove bottleneck shown by time-profiling and thread-observation (most ToeThreads stuck in candidatesProcessor, schedule()ing to frontier)
* AbstractFrontier.java
add comment about implementation vs. subclass override
* Heritrix.java
allow [username:]password to come from file, supplied to '-a' with '@' prefix
also, clean up options names/help text to match current behavior
also, offer warning about particularly weak (<8 characters, all letters, all numbers) passwords
* Engine.java
tolerate (with return-false-forfailure rather than NPE) non-job directories
* EngineTest.java
use empty temp job directory rather than (possibly full-of-junk) general junit-tests directory
* GenericReplayCharSequence.java
* Scoper.java
* FetchFTP.java
* FetchHTTP.java
demote a bunch of INFO logging to FINE so that be default, console isn't filled with output for every URI handled by a busy crawl
* CheckpointService.java
reorganize so that checkpoint timedTask is reset whenever interval is reset on a running crawl
* CheckpointSuccessEvent.java, CrawlJob.java
report successes as new kind of ApplicationEvent; propagagte to job log
* BdbModule.java
remove obsolete unconsulted checkpointCopyLogs setting
* AbstractFrontier.java
call new finalTasks just before announcing FINISH state
* WorkQueueFrontier.java
eliminate closeQueue
* BdbFrontier.java
move dumpAllPendingToLog from closeQueue (triggered by CrawlController FINISHED) to finalTasks (before Frontier FINISH)
replace closeQueue with close() override
* CrawlJob.java
synchronize methods which mutate or change behavior based on field 'ac'; previously, some self-tests were exitting/failing immediately because they were not seeing a non-null 'ac' value.
* CrawlServer.java
enable a specific kind of connect-lost (no response given) to be counted as 'deemed 404' on robots retries over a minimum
* FetchStatusCodes.java
add a synthetic code: -404, S_DEEMED_NOT_FOUND
* FetchHTTP.java
better distinguish CONNECT_FAILED and CONNECT_LOST
* DispositionProcessor.java
treat CONNECT_LOST same as _FAILED; simplify a test
* PersistProcessor.java
rename setupEnvironment to setupCopyEnvironment to make limited purpose of env clear; catch and rethrow IllegalArgumentException with hint that second-open may be cause
* PersistLoadProcessor.java
catch and display IllegalArgumentException same as other failure-to-open SEVERE but proceedable exceptions
* PrecedenceLoader.java
update method names
* engine/pom.xml
restore a reference to a (new version of a) necessary servlet library
* pom.xml
remove deprecated {pom.artifactId}; add needed javadoc module version decl
* .classpath
update ref to currently-used servlet jar
* CrawlJob.java
added boolean partialJobLogScan, and print "at least X launches" when true
* JobResource.java
print "at least X launches" when partialJobLogScan
kristinn_sig