* CheckpointService.java
use static method on Checkpoint for validity check
* Checkpoint.java
throw early-startup exception if attempting resumption of invalid checkpoint
* CheckpointService.java
(getAvailableCheckpointDirectories) return List, filtered to leave out directories without a 'valid' stamp; log WARNING to _out so that the attentive operator can manually clean-up/investigate
* ExtractorHTTP.java
add property inferRootPage; if true (not the default), always consider '/' inferred from all HTTP(S) URIs
* Hop.java, LinkContext.java
add new hop-type 'I' for inferred URIs; add corresponding LinkContext
* ExtractorImpliedURI.java
use new 'I'nferred types
* CrawlURI.java
improve comments for fetchAttempts methods
(resetForRescheduling) added to clear per-scheduling state
* WorkQueueFrontier.java
call resetForRescheduling just before scheduling-at-specific-time
* FrontierJournal.java, AbstractFrontier.java
renamings: prefer 'reenqueue' instead of 'reschedule' for describing simple (non-timed) retries
* AbstractFrontier.java
change config and defaults for 'inbound' and 'outbound': allowing any BlockinqQueue bean, of any size, to be specified if desired
* **/profile-crawler-beans.cxml
changed commented-out descriptions of defaults to reflect what's used if nothing is specified for 'outbound' and 'inbound'
* EngineResource.java
add 'exit' control area at bottom of page
allow JVM-exit if confirm-box checked and either no jobs built/running or extra confirm boxes for each built job checked
* Flash.java
fix to allow multiple flashes per page load
* JobResource.java, CrawlJob.java
move useful status string method to CrawlJob
* BloomFilter64bit.java
include the split-to-subarrays (for larger bitfields) and round-up-to-power-of-2 (for performance) options previously in largely-redundant classes
fit a number of problems with int/long overflow and bitwise ops
add methods for reporting/testing
* BloomFilter.java
add methods for reporting/testing
* BloomFilterTest.java, BloomFilter64bitTest.java
more extensive tests, including two lengthy tests of default/oversized blooms usually disabled by renaming
* BloomFilter32bit.java, BloomFilter32bitSplit.java, BloomFilter32bp2.java, BloomFilter32bp2Split.java
deleted as buggy or redundant
* BenchmarkBlooms.java
move to test source dir
* BloomUriUniqFilter.java
change to accept filter instance (rather than parameters) for added configuration flexibility
fix comments
* BloomUriUniqFilterTest.java
supply filter not paramters
* dist/src/main/conf/jobs/profile-defaults/profile-crawler-beans.cxml
* engine/src/main/resources/org/archive/crawler/restlet/profile-crawler-beans.cxml
add intended REJECT decision for the late (rules[4]) MatchesListRegexDecideRule
* BloomFilter64bit.java
fix long arithmetic calculating bit-length m in constructor
slight optimization of add()
* BloomFilterTest.java
abstract superclass for bloom impl class tests
* BloomFilter64bitTest.java
sanity check that set bits, after a handful of adds into default-sized filter, occupy top and bottom 20% of range (which would have caught this bug earlier)
* ExtractorHTML.java
smaller caps on element-name, attribute-name, attribute-value lengths -- costly match-failures on bad/non-HTML may be eating a lot of cycles
* Engine.java
suppress WARNING for common case of harmless failure during rescans: job already exists
* EngineResource.java
suppress rescan on each reload (ODD: steve had already committed this change -- I got the commit email -- but now the repo version is missing it. Sourceforge problems?)
* WorkQueueFrontier.java
override schedule() to keep unique-testing in calling thread; may remove bottleneck shown by time-profiling and thread-observation (most ToeThreads stuck in candidatesProcessor, schedule()ing to frontier)
* AbstractFrontier.java
add comment about implementation vs. subclass override
* Heritrix.java
allow [username:]password to come from file, supplied to '-a' with '@' prefix
also, clean up options names/help text to match current behavior
also, offer warning about particularly weak (<8 characters, all letters, all numbers) passwords
* Engine.java
tolerate (with return-false-forfailure rather than NPE) non-job directories
* EngineTest.java
use empty temp job directory rather than (possibly full-of-junk) general junit-tests directory
* ReplayCharSequence.java, GenericReplayCharSequence.java, InMemoryReplayCharSequence.java
store actual first exception for reference
* ExtractorHTML.java
report exception in nonfatal-log, rather than WARNING (which was still polluting job log more than desired -- but should still be logged to allow future investigation of sites where encoding issues may exist)
szznax