* EngineResource.java
add 'exit' control area at bottom of page
allow JVM-exit if confirm-box checked and either no jobs built/running or extra confirm boxes for each built job checked
* Flash.java
fix to allow multiple flashes per page load
* JobResource.java, CrawlJob.java
move useful status string method to CrawlJob
* BloomFilter64bit.java
include the split-to-subarrays (for larger bitfields) and round-up-to-power-of-2 (for performance) options previously in largely-redundant classes
fit a number of problems with int/long overflow and bitwise ops
add methods for reporting/testing
* BloomFilter.java
add methods for reporting/testing
* BloomFilterTest.java, BloomFilter64bitTest.java
more extensive tests, including two lengthy tests of default/oversized blooms usually disabled by renaming
* BloomFilter32bit.java, BloomFilter32bitSplit.java, BloomFilter32bp2.java, BloomFilter32bp2Split.java
deleted as buggy or redundant
* BenchmarkBlooms.java
move to test source dir
* BloomUriUniqFilter.java
change to accept filter instance (rather than parameters) for added configuration flexibility
fix comments
* BloomUriUniqFilterTest.java
supply filter not paramters
* dist/src/main/conf/jobs/profile-defaults/profile-crawler-beans.cxml
* engine/src/main/resources/org/archive/crawler/restlet/profile-crawler-beans.cxml
add intended REJECT decision for the late (rules[4]) MatchesListRegexDecideRule
* Engine.java
suppress WARNING for common case of harmless failure during rescans: job already exists
* EngineResource.java
suppress rescan on each reload (ODD: steve had already committed this change -- I got the commit email -- but now the repo version is missing it. Sourceforge problems?)
* WorkQueueFrontier.java
override schedule() to keep unique-testing in calling thread; may remove bottleneck shown by time-profiling and thread-observation (most ToeThreads stuck in candidatesProcessor, schedule()ing to frontier)
* AbstractFrontier.java
add comment about implementation vs. subclass override
* Heritrix.java
allow [username:]password to come from file, supplied to '-a' with '@' prefix
also, clean up options names/help text to match current behavior
also, offer warning about particularly weak (<8 characters, all letters, all numbers) passwords
* Engine.java
tolerate (with return-false-forfailure rather than NPE) non-job directories
* EngineTest.java
use empty temp job directory rather than (possibly full-of-junk) general junit-tests directory
* GenericReplayCharSequence.java
* Scoper.java
* FetchFTP.java
* FetchHTTP.java
demote a bunch of INFO logging to FINE so that be default, console isn't filled with output for every URI handled by a busy crawl
* CheckpointService.java
reorganize so that checkpoint timedTask is reset whenever interval is reset on a running crawl
* CheckpointSuccessEvent.java, CrawlJob.java
report successes as new kind of ApplicationEvent; propagagte to job log
* BdbModule.java
remove obsolete unconsulted checkpointCopyLogs setting
* AbstractFrontier.java
call new finalTasks just before announcing FINISH state
* WorkQueueFrontier.java
eliminate closeQueue
* BdbFrontier.java
move dumpAllPendingToLog from closeQueue (triggered by CrawlController FINISHED) to finalTasks (before Frontier FINISH)
replace closeQueue with close() override
* CrawlJob.java
synchronize methods which mutate or change behavior based on field 'ac'; previously, some self-tests were exitting/failing immediately because they were not seeing a non-null 'ac' value.
* CrawlServer.java
enable a specific kind of connect-lost (no response given) to be counted as 'deemed 404' on robots retries over a minimum
* FetchStatusCodes.java
add a synthetic code: -404, S_DEEMED_NOT_FOUND
* FetchHTTP.java
better distinguish CONNECT_FAILED and CONNECT_LOST
* DispositionProcessor.java
treat CONNECT_LOST same as _FAILED; simplify a test
* PersistProcessor.java
rename setupEnvironment to setupCopyEnvironment to make limited purpose of env clear; catch and rethrow IllegalArgumentException with hint that second-open may be cause
* PersistLoadProcessor.java
catch and display IllegalArgumentException same as other failure-to-open SEVERE but proceedable exceptions
* PrecedenceLoader.java
update method names
* engine/pom.xml
restore a reference to a (new version of a) necessary servlet library
* pom.xml
remove deprecated {pom.artifactId}; add needed javadoc module version decl
* .classpath
update ref to currently-used servlet jar
* CrawlJob.java
added boolean partialJobLogScan, and print "at least X launches" when true
* JobResource.java
print "at least X launches" when partialJobLogScan
* Heritrix.java
added CLAP protocol to access representations via classloader
* EngineApplication.java
added staticDir resource to serve static files under "/engine/static"
also added org.archive.crawler.restlet package to hold static files
* EngineResource.java
added <link> to static CSS resource
* Engine.css
style for flashACK/NACK/ADVISORY divs
* Engine.java
removed ERROR and WARNING strings, rely on Flash class
log jobsDir check failure as SEVERE and return early
collapse error-handling in considerAsJobPath() and log SEVERE
renamed considerAsJobDirectory() to addJobDirectory()
remove catch-and-rethrow from leaveJobPathFile() when writing,
instead log SEVERE
renamed leaveJobPathFile() to writeJobPathFile()
added boolean "userRequest" arg to addJobDirectory(), and moved
call to writeJobPathFile() inside of addJobDirectory()
changed void considerAsJobPath(File) to String getJobPathFromFile
and removed call to addJobDirectory
make seperate calls to addJobDirectory() in findJobConfigs()
* EngineResource.java
restructured "create" logic for clarity
eliminate style code, rely on Flash class
remove limit on "addpath" input, change to "createpath"
removed all-caps and bangs ("!") from error messages
removed platform-specific path separators ("/")
call Engine.addJobDirectory() with userRequest=true in when
form action=add
* **/pom.xml
bump version identifiers to "3.1.1-SNAPSHOT" (odd minor number meaning dev version, odd patch number indicating expected next patch number of 3.0.x release if before a 3.2.0 release)
gojomo