* dist/src/main/conf/jobs/profile-defaults/profile-crawler-beans.cxml
* engine/src/main/resources/org/archive/crawler/restlet/profile-crawler-beans.cxml
add intended REJECT decision for the late (rules[4]) MatchesListRegexDecideRule
* Engine.java
suppress WARNING for common case of harmless failure during rescans: job already exists
* EngineResource.java
suppress rescan on each reload (ODD: steve had already committed this change -- I got the commit email -- but now the repo version is missing it. Sourceforge problems?)
* WorkQueueFrontier.java
override schedule() to keep unique-testing in calling thread; may remove bottleneck shown by time-profiling and thread-observation (most ToeThreads stuck in candidatesProcessor, schedule()ing to frontier)
* AbstractFrontier.java
add comment about implementation vs. subclass override
* Heritrix.java
allow [username:]password to come from file, supplied to '-a' with '@' prefix
also, clean up options names/help text to match current behavior
also, offer warning about particularly weak (<8 characters, all letters, all numbers) passwords
* Engine.java
tolerate (with return-false-forfailure rather than NPE) non-job directories
* EngineTest.java
use empty temp job directory rather than (possibly full-of-junk) general junit-tests directory
* GenericReplayCharSequence.java
* Scoper.java
* FetchFTP.java
* FetchHTTP.java
demote a bunch of INFO logging to FINE so that be default, console isn't filled with output for every URI handled by a busy crawl
* CheckpointService.java
reorganize so that checkpoint timedTask is reset whenever interval is reset on a running crawl
* CheckpointSuccessEvent.java, CrawlJob.java
report successes as new kind of ApplicationEvent; propagagte to job log
* BdbModule.java
remove obsolete unconsulted checkpointCopyLogs setting
* AbstractFrontier.java
call new finalTasks just before announcing FINISH state
* WorkQueueFrontier.java
eliminate closeQueue
* BdbFrontier.java
move dumpAllPendingToLog from closeQueue (triggered by CrawlController FINISHED) to finalTasks (before Frontier FINISH)
replace closeQueue with close() override
* CrawlJob.java
synchronize methods which mutate or change behavior based on field 'ac'; previously, some self-tests were exitting/failing immediately because they were not seeing a non-null 'ac' value.
* CrawlServer.java
enable a specific kind of connect-lost (no response given) to be counted as 'deemed 404' on robots retries over a minimum
* FetchStatusCodes.java
add a synthetic code: -404, S_DEEMED_NOT_FOUND
* FetchHTTP.java
better distinguish CONNECT_FAILED and CONNECT_LOST
* DispositionProcessor.java
treat CONNECT_LOST same as _FAILED; simplify a test
* PersistProcessor.java
rename setupEnvironment to setupCopyEnvironment to make limited purpose of env clear; catch and rethrow IllegalArgumentException with hint that second-open may be cause
* PersistLoadProcessor.java
catch and display IllegalArgumentException same as other failure-to-open SEVERE but proceedable exceptions
* PrecedenceLoader.java
update method names
* engine/pom.xml
restore a reference to a (new version of a) necessary servlet library
* pom.xml
remove deprecated {pom.artifactId}; add needed javadoc module version decl
* .classpath
update ref to currently-used servlet jar
* CrawlJob.java
added boolean partialJobLogScan, and print "at least X launches" when true
* JobResource.java
print "at least X launches" when partialJobLogScan
* Heritrix.java
added CLAP protocol to access representations via classloader
* EngineApplication.java
added staticDir resource to serve static files under "/engine/static"
also added org.archive.crawler.restlet package to hold static files
* EngineResource.java
added <link> to static CSS resource
* Engine.css
style for flashACK/NACK/ADVISORY divs
* Engine.java
removed ERROR and WARNING strings, rely on Flash class
log jobsDir check failure as SEVERE and return early
collapse error-handling in considerAsJobPath() and log SEVERE
renamed considerAsJobDirectory() to addJobDirectory()
remove catch-and-rethrow from leaveJobPathFile() when writing,
instead log SEVERE
renamed leaveJobPathFile() to writeJobPathFile()
added boolean "userRequest" arg to addJobDirectory(), and moved
call to writeJobPathFile() inside of addJobDirectory()
changed void considerAsJobPath(File) to String getJobPathFromFile
and removed call to addJobDirectory
make seperate calls to addJobDirectory() in findJobConfigs()
* EngineResource.java
restructured "create" logic for clarity
eliminate style code, rely on Flash class
remove limit on "addpath" input, change to "createpath"
removed all-caps and bangs ("!") from error messages
removed platform-specific path separators ("/")
call Engine.addJobDirectory() with userRequest=true in when
form action=add
* **/pom.xml
bump version identifiers to "3.1.1-SNAPSHOT" (odd minor number meaning dev version, odd patch number indicating expected next patch number of 3.0.x release if before a 3.2.0 release)
* AbstractFrontier.java
close recovery log in stop() (comes later, usu. from managerThread) rather than terminate() (earlier, may occur in other threads like progress-stats-snapshot thread)
gojomo