* Heritrix.java
allow [username:]password to come from file, supplied to '-a' with '@' prefix
also, clean up options names/help text to match current behavior
also, offer warning about particularly weak (<8 characters, all letters, all numbers) passwords
* Engine.java
tolerate (with return-false-forfailure rather than NPE) non-job directories
* EngineTest.java
use empty temp job directory rather than (possibly full-of-junk) general junit-tests directory
* GenericReplayCharSequence.java
* Scoper.java
* FetchFTP.java
* FetchHTTP.java
demote a bunch of INFO logging to FINE so that be default, console isn't filled with output for every URI handled by a busy crawl
* CheckpointService.java
reorganize so that checkpoint timedTask is reset whenever interval is reset on a running crawl
* CheckpointSuccessEvent.java, CrawlJob.java
report successes as new kind of ApplicationEvent; propagagte to job log
* BdbModule.java
remove obsolete unconsulted checkpointCopyLogs setting
* AbstractFrontier.java
call new finalTasks just before announcing FINISH state
* WorkQueueFrontier.java
eliminate closeQueue
* BdbFrontier.java
move dumpAllPendingToLog from closeQueue (triggered by CrawlController FINISHED) to finalTasks (before Frontier FINISH)
replace closeQueue with close() override
* CrawlJob.java
synchronize methods which mutate or change behavior based on field 'ac'; previously, some self-tests were exitting/failing immediately because they were not seeing a non-null 'ac' value.
* CrawlServer.java
enable a specific kind of connect-lost (no response given) to be counted as 'deemed 404' on robots retries over a minimum
* FetchStatusCodes.java
add a synthetic code: -404, S_DEEMED_NOT_FOUND
* FetchHTTP.java
better distinguish CONNECT_FAILED and CONNECT_LOST
* DispositionProcessor.java
treat CONNECT_LOST same as _FAILED; simplify a test
* PersistProcessor.java
rename setupEnvironment to setupCopyEnvironment to make limited purpose of env clear; catch and rethrow IllegalArgumentException with hint that second-open may be cause
* PersistLoadProcessor.java
catch and display IllegalArgumentException same as other failure-to-open SEVERE but proceedable exceptions
* PrecedenceLoader.java
update method names
* CrawlJob.java
added boolean partialJobLogScan, and print "at least X launches" when true
* JobResource.java
print "at least X launches" when partialJobLogScan
* Heritrix.java
added CLAP protocol to access representations via classloader
* EngineApplication.java
added staticDir resource to serve static files under "/engine/static"
also added org.archive.crawler.restlet package to hold static files
* EngineResource.java
added <link> to static CSS resource
* Engine.css
style for flashACK/NACK/ADVISORY divs
* Engine.java
removed ERROR and WARNING strings, rely on Flash class
log jobsDir check failure as SEVERE and return early
collapse error-handling in considerAsJobPath() and log SEVERE
renamed considerAsJobDirectory() to addJobDirectory()
remove catch-and-rethrow from leaveJobPathFile() when writing,
instead log SEVERE
renamed leaveJobPathFile() to writeJobPathFile()
added boolean "userRequest" arg to addJobDirectory(), and moved
call to writeJobPathFile() inside of addJobDirectory()
changed void considerAsJobPath(File) to String getJobPathFromFile
and removed call to addJobDirectory
make seperate calls to addJobDirectory() in findJobConfigs()
* EngineResource.java
restructured "create" logic for clarity
eliminate style code, rely on Flash class
remove limit on "addpath" input, change to "createpath"
removed all-caps and bangs ("!") from error messages
removed platform-specific path separators ("/")
call Engine.addJobDirectory() with userRequest=true in when
form action=add
* AbstractFrontier.java
close recovery log in stop() (comes later, usu. from managerThread) rather than terminate() (earlier, may occur in other threads like progress-stats-snapshot thread)
iPhone Safari, Chrome) use a screwy 'Accept' header that if taken literally
means they prefer 'application/xml' content. Thus trying to view web UI in
these browsers results in seeing the XML version, without element markup
* BaseResource.java
override getPreferredVariant() to bump up preference level of text/html for
any client can accept it
* EngineResource.java, JobResource.java
inherit from BaseResource
* WorkQueueFrontier.java
add snoozedOverflow map of (wakeTime)->(DelayedWorkQueue) for snoozes over a capped size
maintain in parallel to snoozedClassQueues DelayQueue, maintain atomic size count
in standard report, presort snoozedClassQueues before displaying
* DelayedWorkQueue.java
move to top-level so serialization doesn't pull out whole frontier
* BdbFrontier.java
initialize snoozedOverflow
* BdbMultipleWorkQueues.java
improve error message seen on problematic checkpoint-recovery
* EngineResource.java
give add-job options their own header, reorder/reword for clarity
* JobResource.java
return to always showing 'checkpoint' button, disable if necessary, so main control buttons aren't appearing/disappearing or moving
gojomo