Commit Graph
8 Commits
Author SHA1 Message Date
Noah Levitt 3dce1e909d Change scope of most/all default-scope i.e. package-scoped class members to protected, to make heritrix more friendly to third-party extensions. (I don't think there was ever an intent to limit the scope of any of these to package-only.) 2012-06-05 17:55:41 -07:00
nlevitt f20b3eee30 Fix HER-1935 Many calls to File.mkdirs() and other file/dir methods don't check the return value. Also fix bug where pointless empty directories were created in scratch dir.
* BasicProfileTest.java, SelfTestBase.java, CrawlControllerTest.java, PrecedenceLoader.java, MigrateH1to3Tool.java, CrawlerLoggerModule.java, StatisticsTracker.java, CheckpointUtils.java, BdbUriUniqFilter.java, ARCWriterProcessorTest.java, WARCWriterProcessorTest.java, PersistProcessor.java, WriterPoolProcessor.java, PrefixFinderTest.java, StoredQueueTest.java, FileUtilsTest.java, ObjectIdentityBdbManualCacheTest.java, ObjectIdentityBdbCacheTest.java, ObjectPlusFilesOutputStream.java, TestUtils.java, TmpDirTestCase.java, Engine.java
    Replace calls to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir). In these cases the calls were either already in a spot where the possible IOException would be handled appropriately, or the line was trivially moved into such a block.
* ActionDirectory.java, Engine.java
    replace calls to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir), and throw IllegalStateException on failure
* BdbModule.java
    setup() - replace call to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir) and add "throws IOException" - conveniently the place where this method is called was already in a try block that catches IOException
* Checkpoint.java
    generateFrom() - replace call to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir) and add "throws IOException"
* CheckpointService.java
    move call to Checkpoint.generateFrom() inside existing try block since it now can throw IOException
* Recorder.java
    ensure(File) - replace call to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir), and throw IllegalStateException on failure
    new Recorder(File,String,int,int) - call ensure() on the correct object, the containing directory; and remove redundant call to ensure()
2011-09-12 20:12:10 +00:00
gojomo 3f98fc248b * ArchiveUtils
(readFully) return actual number of bytes read into array
* FileUtils
    (readFullyToFile) simplify; also return total bytes read
2011-04-06 20:22:30 +00:00
nlevitt 8ba641ed4b Part of [HER-1603] Restletify: xml representations of all resources that also
offer html
* XmlMarshaller.java
    new class to write xml based on nested map structure
* BeanBrowseResource.java ScriptResource.java EngineResource.java
  JobResource.java JobRelatedResource.java
    support xml representation
* WorkQueueFrontier.java PrecedenceProvider.java WorkQueue.java ToeThread.java
  ToePool.java CrawlJob.java Engine.java Frontier.java CrawlController.java
  StatisticsTracker.java CrawledBytesHistotable.java FetchStats.java
  ProcessorChain.java CrawlURI.java Reporter.java
    offer same data that is returned in the single line reports in a map
    suitable for use with XmlMarshaller
* FileUtils.java
    new method tryToCanonicalize(File)
2009-11-25 01:43:57 +00:00
gojomo e42e2ed7f6 Cleanup/consolidation/expansion of *Utils classes
* IoUtils.java
    remove class that confusingly-overlaps with commons IOUtils; move remaining methods to ArchiveUtils or archive's FileUtils
* ArchiveUtils.java, FileUtils.java
    receive relocated methods; eliminated deprecated or unused methods
* JSONUtils.java
    new class to collect common JSON actions
* Iteratorable.java
    wrap Iterator as Iterable for foreach usage
* (many)
    update to use alternate utils methods
2009-11-19 22:39:53 +00:00
gojomo d86f5f7403 * (many)
harmonize our usage on '[Rr]egex' as the abbrieviation for regular-expression
    (replacing RegExp/etc.; 'regex' is 30-40% more prevalent by google hits)
2009-10-06 03:45:44 +00:00
gojomo a1e9d064da copy to h3
Fix for [HER-1520] Windows mapped-files cause intermittent ReplayCharSequenceTest.testReplayCharSequenceByteToStringMulti failures, risk other problems
* GenericReplayCharSequence.java
    use more diverse filenames, so subsequent decodes don't run into lingering open mapped sections of previous decodes
    do delete via new FileUtils.deleteSoonerOrLater utility
* FileUtils.java
   (deleteSoonerOrLater) try immediate delete, but if fails, add to a list of files to be retried later
2009-05-14 21:31:45 +00:00
gojomo e047bf68e0 copy 'springy' branch to heritrix3 2009-05-11 22:56:36 +00:00