* CrawlJob.java
decided to go with plain timestamp17 as launch id, no "launch-" prefix
* ConfigPathConfigurer.java
refactored special remembering of WriterPoolProcessor storePaths out of fixupPaths()
* UriProcessingFormatter, Preformatter, GenerationFileHandler
improve preformat-outside-synchronized optimization so that the LogRecord/CrawlURI doesn't linger until next displaces it
* CrawlURI
(processingCleanup) null more of last-processing-run collected values
* HardLinker.java
renamed FilesystemLinkMaker.java
* FilesystemLinkMaker.java
add support for symbolic links
* CLibrary.java
new method symlink()
* BdbModule.java
use new class name FilesystemLinkMaker
* CrawlJob.java
at crawl launch, create launch directory launch-{timestamp17}, copy cxml there, symlink "current" to launch dir, inform ConfigPaths
* ConfigPath.java
interpolate ${launch-id} in configured paths
* ConfigFile.java
obtainReader() - snapshot config files to launch dir when they are read
* ActionDirectory.java
default doneDir now ${launch-id}/actions-done
actOn() - symlink from old style done dir action/done to done files
* SurtPrefixedDecideRule.java
default surtsDumpFile now ${launch-id}/surts.dump
pathsFixedUp() - this gets called at build time, but we don't want anything written to disk until launch time, so remove call to dumpSurtPrefixSet() here
* CrawlerLoggerModule.java
default logs dir now ${launch-id}/logs
* StatisticsTracker.java
default reports dir now ${launch-id}/reports
* WriterPoolProcessor.java
default writer base path now ${launch-id}
* profile-crawler-beans.cxml
update to reflect new default paths under launch dirs
* PropertyUtils.java
fix javadoc typo
* pom.xml, .classpath
update references to necessary spring-3.0.5 packages
* SeedModule
merge rather than replace event listeners, so that (now later) autowiring doesn't clobber the self-insertion done by anonymous (non-top-level) SurtPrefixedDecideRule bean
* HeritrixLifecycleProcessor, PathScharingContext
use this new non-default LifecycleProcessor to avoid new Spring3 behavior of auto-start()ing context on refresh(build)
* CrawlController
example of using @Value annotation to set default value: will offer benefits for auto-discovery of defaults for configuration interface, or enforcing maximally-explicit configurations (see [HER-1897])
* profile-crawler-beans.cxml, selftest-crawler-beans.cxml
update templates with spring3 preamble/boilerplate
[HER-1910] H3: job-related pages very slow to render (pause mid-way through) in large crawls
* BdbModule
option to reuse bdb data on StoredQueue-creation
* Checkpoint
new saveWriter, loadReader utilities for creating extra checkpoint files (for the long list of ready queues)
* BdbFrontier
on checkpoint, remember ordered list of inProcess/ready/snoozed queues (so same queues are active upon resume)
on resume, reuse old retired/inactive StoredQueue data, and reload active queues from newly-saved list
* StoredQueue
restore tailIndex properly when resuing old data
more-efficient size() calculation (for HER-1910)
* JobRelatedResource
catch InvalidPropertyException, render as red error message rather than fouling entire trace of beans to web UI
* StatisticsTracker
rename getSnapshots to listSnapshots to prevent interpretation as (invalid) bean-property
* TextSeedModule
refactor announceSeeds to occur in background thread if non-default blockAwaitingSeedLines value is set
signal CountDownLatch on each line, allowing calling thread to proceed at right count
* profile-crawler-beans.cxml
commented-out blockAwaitingSeedLines default settings (-1, meaning wait for all seed lines)
* CrawlController
change pauseAtFinish to runWhileEmpty
add EMPTY state
replace isStateRunning() with isActive() (RUNNING or EMPTY)
trigger finish-test off frontier EMPTY rather than PAUSE
in logging say 'running' rather than 'resumed'
* Frontier
add EMPTY state
* AbstractFrontier
handle EMPTY like RUNNING, transition EMPTY<->RUNNING when appropriate
* StatisticsTracker
handle EMPTY analogous to RUNNING
in logging say 'running' rather than 'resumed'
* CrawlJob, DiskSpaceMonitor
replace isStateRunning() with isActive()
redirect url sometimes '0 NOTCRAWLED' in seeds-report.txt even when crawled"
* CandidatesProcessor.java
innerProcess() - set force-fetch on outlinks promoted to seeds
* CrawlURI.java
new field JSONObject extraInfo, methods getExtraInfo() and addExtraInfo()
* WARCWriterProcessor.java
write() - on success, add warcFilename to CrawlURI extraInfo
* CrawlerLoggerModule.java
new config option logExtraInfo, default false
* UriProcessingFormatter.java
new field logExtraInfo
format() - include CrawlURI extraInfo if logExtraInfo is enabled
also include "-" if CrawlURI has no annotations, since this is no longer the last field on the line
* profile-crawler-beans.cxml
<!-- <property name="logExtraInfo" value="false" /> -->
* NonFatalErrorFormatter.java, RuntimeErrorFormatter.java
new constructor to handle logExtraInfo since these subclass UriProcessingFormatter
even when crawled, when original seed also has a regular link to the redirect
url
* CandidatesProcessor.java
innerProcess() - present seed outlinks to the frontier ahead of non-seed outlinks, so that seed version of any duplicated outlink is always the one that's crawled
* FetchHTTP.java
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
by default - this is what my firefox 4.0 for mac sends
* profile-crawler-beans.cxml
commented-out default value
marshalDocument() - SAXException can wrap other exceptions, e.g.
EOFException when tcp connection is broken; thus, in case of IOException,
rather than throwing a RuntimeException, re-throw the wrapped IOException
(unfortunately in that case we still get a massive stack trace in the log,
but at least it comes from the appropriate spot and is not too misleading)
* URIAuthorityBasedQueueAssignmentPolicy.java
whois urls all go in special "whois" queue
* PreconditionEnforcer.java, CrawlURI.java
move markPrerequisite() method from PreconditionEnforcer to CrawlURI
* profile-crawler-beans.cxml
commented-out whois fetcher clause
* FetchStatusCodes.java
new status codes for whois
* ServerCache.java
getHostFor() - return "whois:" for whois urls, following dns: convention
* UURIFactory.java
ugly hack for whois urls
* SchemeNotInSetDecideRule.java
add whois to list of supported schemes
* WriterPoolProcessor.java
shouldWrite() - we should write successful whois fetches
* WARCWriterProcessor.java
write whois records
* FetchWhois.java
the fetcher
use heritrix.home, if available, to find default logging.properties (to let those who launch Heritrix from elsewhere save themselves from HttpClient's copious logging)
* StatisticsTracker
move small stat maps to in-memory ConcurrentMaps
update processedSeedRecords for new ObjectIdentityCache shape
move hostsDistribution/hostsBytes/hostsLastFinished tracking to CrawlHosts/hostsCache
* CrawlSummaryReport
gets hosts count from hostsCache
* SeedRecord
implement IdentityCacheable
* FetchStats
remember last-success times
* IdentityCacheable
new interface required of objects stored in ObjectIdentityCaches
* IdentityCacheableWrapper
wrapper for storing arbitrary objects in ObjectIdentityCaches
* ObjectIdentityCache
keys now always Strings
values now always IdentityCacheables
new dirtyKey() method to ensure a key is persisted
* ObjectIdentityBdbCache, ObjectIdentityMemCache
update for new ObjectIdentityCache shape; still using legacy GC magic
* ObjectIdentityBdbManualCache
alternate implementation relying on dirtying to ensure persistence
use Guava library MapMaker for soft memMap; capped dirtyMap
* BdbModule
use ObjectIdentityBdbManualCache by default
up expectedConcurrency default to 64
* BdbModuleTest, ObjectIdentityBdbCacheTest, ObjectIdentityBdbManualCacheTest
update, add tests
* Frontier
FrontierGroup as IdentityCacheable
* AbstractFrontier, WorkQueueFrontier, BdbFrontier
touchup queue/group accessors
make queue instances dirty whenever mutated
* WorkQueue, CrawlHost, CrawlServer
IdentityCacheable support; appropriate makeDirty()s
* ServerCache, DefaultServerCache
update for new ObjectIdentityCache shape
* CrawlURI
MAX_HOPS_DISPLAYED constant, 50
build new CrawlURI pathFromSeed value via extendHopsPath, which shows only MAX_HOPS_DISPLAYED hops, and precedes string with int value of hops unshown
* CrawlURITest
test for extendHopsPath
gojomo