Commit Graph
8 Commits
Author SHA1 Message Date
Noah Levitt db801b3e0f Fix problems handling crawl job paths with characters that require url-encoding, since we sometimes refer to them with file: urls
* PathSharingContext.java, CrawlJob.java
    use java.net.URI to convert paths to and from file: urls
2012-02-10 15:48:09 -08:00
Noah Levitt 803f9ca979 HER-1984 save script state - implement by adding a map to the application context for arbitrary use, and make sure the app context is available in all scripting environments
* PathSharingContext.java
    new member variable ConcurrentHashMap data and accessor getData()
* ScriptedProcessor.java, ScriptedDecideRule.java
    make appCtx available to scripts; also remove unused member sharedMap
* ActionDirectory.java
    formatting fix
2012-01-19 13:31:57 -08:00
nlevitt a577f1eb0b * PathSharingContext.java
start() - restore this override to call initLaunchDir() (how did I miss that), then super.start()
2011-09-29 23:12:26 +00:00
nlevitt 747fbc0a0f Fix for HER-1954 bdb closed at crawl finish instead of teardown -- refactor crawl finish and teardown
* PathSharingContext.java
    start(), doStart(), stop(), doStop() - remove these overloaded methods, because the spring bug they were working around seems to be fixed, not seeing any problems with cyclical dependencies - might be this one https://jira.springsource.org/browse/SPR-7266
    doClose() - remove because superclass version seems to work fine (this method generally wasn't being called anyway, though it would be now with other changes in this checkin)
* CrawlJob.java
    refactor teardown to call close() on the ApplicationContext, which calls destroy() on any beans that implement DisposableBean - this is now the way to have beans do stuff at teardown
* CrawlController.java
    send FINISHED crawl state event after calling appCtx.stop() so that isFinished() can indicate ready-ness for teardown
* BdbModule.java
    move close() to teardown, i.e. implementation of DisposableBean.destroy(); remove shutdown hook and rely on teardown; related tweaks
* WorkQueueFrontier.java, CrawlerLoggerModule.java, BdbUriUniqFilter.java
    move close() to teardown
* CrawlMapper.java, AbstractFrontier.java, PreloadedUriPrecedencePolicy.java, FetchWhois.java, FetchHTTP.java, PersistLogProcessor.java, WriterPoolProcessor.java
    add comments about cleanup that maybe should wait until teardown
* UriUniqFilter.java
    remove incorrect(?) comment
2011-09-29 00:39:55 +00:00
nlevitt b90aa89e4b More on HER-1901 - fix build by refactoring launch dir initialization into PathSharingContext; refactor config path interpolation mostly into ConfigPathConfigurer; various tweaks
* ActionDirectory.java, CrawlerLoggerModule.java, StatisticsTracker.java, SurtPrefixedDecideRule.java, WriterPoolProcessor.java, profile-crawler-beans.cxml
    use camelcase ${launchId} 
* CheckpointService.java, JobResource.java
    rename getAvailableCheckpointDirectories() to findAvailableCheckpointDirectories() so that it's not treated as a bean property
* WriterPoolProcessor.java, WriterPoolSettingsData.java, WriterPoolMember.java
    rename getOutputDirs() to calcOutputDirs() so that it's not treated as a bean property
* ConfigPath.java
    let ConfigPathConfigurer to do interpolation of ${launchId}
* ConfigFile.java
    let ConfigPathConfigurer do the snapshotting of config file
* CrawlJob.java, PathSharingContext.java
    remove launchDir initialization out of CrawlJob into PathSharingContext
* ConfigPathConfigurer.java
    - remove special handling of WriterPoolProcessor store paths; instead, look in beans for ConfigPaths within Iterables
    - have each ConfigPath hold reference to this ConfigPathConfigurer to use for interpolating ${launchId} and snapshotting config files
2011-07-15 19:39:32 +00:00
gojomo a41809b8a7 [HER-1908] H3: upgrade to Spring 3
* pom.xml, .classpath
    update references to necessary spring-3.0.5 packages
* SeedModule
    merge rather than replace event listeners, so that (now later) autowiring doesn't clobber the self-insertion done by anonymous (non-top-level) SurtPrefixedDecideRule bean
* HeritrixLifecycleProcessor, PathScharingContext
    use this new non-default LifecycleProcessor to avoid new Spring3 behavior of auto-start()ing context on refresh(build)
* CrawlController
    example of using @Value annotation to set default value: will offer benefits for auto-discovery of defaults for configuration interface, or enforcing maximally-explicit configurations (see [HER-1897])
* profile-crawler-beans.cxml, selftest-crawler-beans.cxml
    update templates with spring3 preamble/boilerplate
2011-07-08 04:31:54 +00:00
gojomo d336ea760f quiet excessive logging 2009-06-05 02:00:07 +00:00
gojomo e047bf68e0 copy 'springy' branch to heritrix3 2009-05-11 22:56:36 +00:00