* Heritrix.java
added CLAP protocol to access representations via classloader
* EngineApplication.java
added staticDir resource to serve static files under "/engine/static"
also added org.archive.crawler.restlet package to hold static files
* EngineResource.java
added <link> to static CSS resource
* Engine.css
style for flashACK/NACK/ADVISORY divs
* Engine.java
removed ERROR and WARNING strings, rely on Flash class
log jobsDir check failure as SEVERE and return early
collapse error-handling in considerAsJobPath() and log SEVERE
renamed considerAsJobDirectory() to addJobDirectory()
remove catch-and-rethrow from leaveJobPathFile() when writing,
instead log SEVERE
renamed leaveJobPathFile() to writeJobPathFile()
added boolean "userRequest" arg to addJobDirectory(), and moved
call to writeJobPathFile() inside of addJobDirectory()
changed void considerAsJobPath(File) to String getJobPathFromFile
and removed call to addJobDirectory
make seperate calls to addJobDirectory() in findJobConfigs()
* EngineResource.java
restructured "create" logic for clarity
eliminate style code, rely on Flash class
remove limit on "addpath" input, change to "createpath"
removed all-caps and bangs ("!") from error messages
removed platform-specific path separators ("/")
call Engine.addJobDirectory() with userRequest=true in when
form action=add
* **/pom.xml
bump version identifiers to "3.1.1-SNAPSHOT" (odd minor number meaning dev version, odd patch number indicating expected next patch number of 3.0.x release if before a 3.2.0 release)
* AbstractFrontier.java
close recovery log in stop() (comes later, usu. from managerThread) rather than terminate() (earlier, may occur in other threads like progress-stats-snapshot thread)
iPhone Safari, Chrome) use a screwy 'Accept' header that if taken literally
means they prefer 'application/xml' content. Thus trying to view web UI in
these browsers results in seeing the XML version, without element markup
* BaseResource.java
override getPreferredVariant() to bump up preference level of text/html for
any client can accept it
* EngineResource.java, JobResource.java
inherit from BaseResource
* WorkQueueFrontier.java
add snoozedOverflow map of (wakeTime)->(DelayedWorkQueue) for snoozes over a capped size
maintain in parallel to snoozedClassQueues DelayQueue, maintain atomic size count
in standard report, presort snoozedClassQueues before displaying
* DelayedWorkQueue.java
move to top-level so serialization doesn't pull out whole frontier
* BdbFrontier.java
initialize snoozedOverflow
* BdbMultipleWorkQueues.java
improve error message seen on problematic checkpoint-recovery
* EngineResource.java
give add-job options their own header, reorder/reword for clarity
* JobResource.java
return to always showing 'checkpoint' button, disable if necessary, so main control buttons aren't appearing/disappearing or moving
* WriterPoolProcessor.java
change description of WriterPoolProcessor.directory ConfigPath to not imply as strongly that is where writing much be occurring
==========
2009-12-03 01:36:16.726 SEVERE thread-10 com.noelios.restlet.http.HttpServerConverter.commit() An exception occured writing the response entity
java.io.IOException: Stream closed
at sun.nio.cs.StreamEncoder.ensureOpen(StreamEncoder.java:26)
[...]
2009-12-03 01:36:16.733::WARN: Committed before 500 An exception occured writing the response entity
2009-12-03 01:36:16.735 WARNING thread-10 com.noelios.restlet.http.HttpServerHelper.handle() Error while handling an HTTP server call:
==========
* JobResource.java
writeHtml(Writer) - do not close writer
* CrawlController.java
send event of crawl-end before appCtx.stop()
* StatisticsTracker.java
force final progressStatisticsEvent and dumpReports at crawlEnded() before stop()
add force-dump of reports (so dumpReports never reuses prior reports)
* CrawledBytesHistotable.java
add getTotalBytes(), a total of just the byte-counts (getTotal() includes URI counts)
* CrawlStatSnapshot.java
use getTotalBytes()
* UURIFactoryTest.java
specify stirng literal with escape-encoding, rather than as raw UTF-8
* **/pom.xml
include 'project.build.sourceEncoding' property to suppress platform-encoding warnings
(unfortunately, neither this nor similar 'maven.compile.encoding' property actually casues maven build to interpret Java source as UTF-8)
offer html
* XmlMarshaller.java
new class to write xml based on nested map structure
* BeanBrowseResource.java ScriptResource.java EngineResource.java
JobResource.java JobRelatedResource.java
support xml representation
* WorkQueueFrontier.java PrecedenceProvider.java WorkQueue.java ToeThread.java
ToePool.java CrawlJob.java Engine.java Frontier.java CrawlController.java
StatisticsTracker.java CrawledBytesHistotable.java FetchStats.java
ProcessorChain.java CrawlURI.java Reporter.java
offer same data that is returned in the single line reports in a map
suitable for use with XmlMarshaller
* FileUtils.java
new method tryToCanonicalize(File)
szznax