* Heritrix.java
added CLAP protocol to access representations via classloader
* EngineApplication.java
added staticDir resource to serve static files under "/engine/static"
also added org.archive.crawler.restlet package to hold static files
* EngineResource.java
added <link> to static CSS resource
* Engine.css
style for flashACK/NACK/ADVISORY divs
* Engine.java
removed ERROR and WARNING strings, rely on Flash class
log jobsDir check failure as SEVERE and return early
collapse error-handling in considerAsJobPath() and log SEVERE
renamed considerAsJobDirectory() to addJobDirectory()
remove catch-and-rethrow from leaveJobPathFile() when writing,
instead log SEVERE
renamed leaveJobPathFile() to writeJobPathFile()
added boolean "userRequest" arg to addJobDirectory(), and moved
call to writeJobPathFile() inside of addJobDirectory()
changed void considerAsJobPath(File) to String getJobPathFromFile
and removed call to addJobDirectory
make seperate calls to addJobDirectory() in findJobConfigs()
* EngineResource.java
restructured "create" logic for clarity
eliminate style code, rely on Flash class
remove limit on "addpath" input, change to "createpath"
removed all-caps and bangs ("!") from error messages
removed platform-specific path separators ("/")
call Engine.addJobDirectory() with userRequest=true in when
form action=add
* **/pom.xml
bump version identifiers to "3.1.1-SNAPSHOT" (odd minor number meaning dev version, odd patch number indicating expected next patch number of 3.0.x release if before a 3.2.0 release)
* AbstractFrontier.java
close recovery log in stop() (comes later, usu. from managerThread) rather than terminate() (earlier, may occur in other threads like progress-stats-snapshot thread)
iPhone Safari, Chrome) use a screwy 'Accept' header that if taken literally
means they prefer 'application/xml' content. Thus trying to view web UI in
these browsers results in seeing the XML version, without element markup
* BaseResource.java
override getPreferredVariant() to bump up preference level of text/html for
any client can accept it
* EngineResource.java, JobResource.java
inherit from BaseResource
* WorkQueueFrontier.java
add snoozedOverflow map of (wakeTime)->(DelayedWorkQueue) for snoozes over a capped size
maintain in parallel to snoozedClassQueues DelayQueue, maintain atomic size count
in standard report, presort snoozedClassQueues before displaying
* DelayedWorkQueue.java
move to top-level so serialization doesn't pull out whole frontier
* BdbFrontier.java
initialize snoozedOverflow
* BdbMultipleWorkQueues.java
improve error message seen on problematic checkpoint-recovery
* EngineResource.java
give add-job options their own header, reorder/reword for clarity
* JobResource.java
return to always showing 'checkpoint' button, disable if necessary, so main control buttons aren't appearing/disappearing or moving
==========
2009-12-03 01:36:16.726 SEVERE thread-10 com.noelios.restlet.http.HttpServerConverter.commit() An exception occured writing the response entity
java.io.IOException: Stream closed
at sun.nio.cs.StreamEncoder.ensureOpen(StreamEncoder.java:26)
[...]
2009-12-03 01:36:16.733::WARN: Committed before 500 An exception occured writing the response entity
2009-12-03 01:36:16.735 WARNING thread-10 com.noelios.restlet.http.HttpServerHelper.handle() Error while handling an HTTP server call:
==========
* JobResource.java
writeHtml(Writer) - do not close writer
* CrawlController.java
send event of crawl-end before appCtx.stop()
* StatisticsTracker.java
force final progressStatisticsEvent and dumpReports at crawlEnded() before stop()
add force-dump of reports (so dumpReports never reuses prior reports)
* CrawledBytesHistotable.java
add getTotalBytes(), a total of just the byte-counts (getTotal() includes URI counts)
* CrawlStatSnapshot.java
use getTotalBytes()
* UURIFactoryTest.java
specify stirng literal with escape-encoding, rather than as raw UTF-8
* **/pom.xml
include 'project.build.sourceEncoding' property to suppress platform-encoding warnings
(unfortunately, neither this nor similar 'maven.compile.encoding' property actually casues maven build to interpret Java source as UTF-8)
offer html
* XmlMarshaller.java
new class to write xml based on nested map structure
* BeanBrowseResource.java ScriptResource.java EngineResource.java
JobResource.java JobRelatedResource.java
support xml representation
* WorkQueueFrontier.java PrecedenceProvider.java WorkQueue.java ToeThread.java
ToePool.java CrawlJob.java Engine.java Frontier.java CrawlController.java
StatisticsTracker.java CrawledBytesHistotable.java FetchStats.java
ProcessorChain.java CrawlURI.java Reporter.java
offer same data that is returned in the single line reports in a map
suitable for use with XmlMarshaller
* FileUtils.java
new method tryToCanonicalize(File)
* AbstractFrontier.java, WorkQueueFrontier.java, BdbFrontier.java
add 'futureUris' map of CrawlURIs to be reenqueued at specific future dates
the usual findEligibleUri also checks for newly-eligible rescheduled URIs
the frontier is no longer empty unless the future count is also 0
* CrawlURI.java
add rescheduleTime property
* ReschedulingProcessor.java
optional post-processor to set a CrawlURI's reschedulingTime
* CrawlStatSnapshot.java, CrawlJob.java
add futureUriCount to stats/uriTotalsReport
* commons/pom.xml, .classpath
update to BDB-JE 4.0.71
* BdbModule.java
(getStoredQueue) added
(getStoredMap) made more general
(openDatabase) all databases now 'managed'
* TempStoredSortedMap.java -> DisposableStoredSortedMap.java
renamed
* StatisticsTracker.java, **Report.java
use new name
* StoredQueue.java, StoredQueueTest.java
move to org.archive.bdb package
* BdbMultipleWorkQueues.java, PrefixFinderTest.java
update for BDB-JE 4 changes
* BdbFrontier.java
use getStoredQueue
drop unneeded custom serialization support
* WorkQueueFrontier.java
drop unneeded custom serialization support
* JobResource.java
suppress checkpoint-selector after terminate
* Checkpoint.java
adapt to work without direct reference to CheckpointService
* CheckpointService.java
configure Checkpoint without making it dependent on CheckpointService
UI support
* JobResource.java
enable 'checkpoint' button as appropriate
display active recovery-checkpoint or select bos of recovery-options when available
* CrawlJob.java
(getCheckpointService) accessor
* BdbFrontier.java
implement Checkpointable; save state via JSON or BDB; restore queues-of-queues on recovery
* WorkQueueFrontier.java
(deactivateQueue) (retireQueue) protected
* BdbUriUniqFilter.java
implement Checkpointable; save state via JSON or BDB; restore
* WorkQueue.java
make peekItem transient -- should not be held through checkpoint
make retired protected
* Frontier.java, AbstractFrontier.java
remove/update obsolete code
* CrawlController.java
remove older checkpoint-functionality passthrough
implement Checkpointable so behavior can vary during recovery
only trigger annnounce of seeds on 'cold' (non-recovery) start
gojomo