Commit Graph
8 Commits
Author SHA1 Message Date
gojomo 04012279fc Misc logging-cleanup/refactoring
* CrawlerJournal.java
    improve exception logging
    combine writeLine() methods to use varargs
* FrontierJournal.java
    improve exception logging
* ArchiveUtils.java
    replace ignored  count parameter with meaingful boolean one
* CrawlURI.java
    use updated ArchiveUtils method
2009-12-04 22:23:48 +00:00
nlevitt 8ba641ed4b Part of [HER-1603] Restletify: xml representations of all resources that also
offer html
* XmlMarshaller.java
    new class to write xml based on nested map structure
* BeanBrowseResource.java ScriptResource.java EngineResource.java
  JobResource.java JobRelatedResource.java
    support xml representation
* WorkQueueFrontier.java PrecedenceProvider.java WorkQueue.java ToeThread.java
  ToePool.java CrawlJob.java Engine.java Frontier.java CrawlController.java
  StatisticsTracker.java CrawledBytesHistotable.java FetchStats.java
  ProcessorChain.java CrawlURI.java Reporter.java
    offer same data that is returned in the single line reports in a map
    suitable for use with XmlMarshaller
* FileUtils.java
    new method tryToCanonicalize(File)
2009-11-25 01:43:57 +00:00
gojomo e42e2ed7f6 Cleanup/consolidation/expansion of *Utils classes
* IoUtils.java
    remove class that confusingly-overlaps with commons IOUtils; move remaining methods to ArchiveUtils or archive's FileUtils
* ArchiveUtils.java, FileUtils.java
    receive relocated methods; eliminated deprecated or unused methods
* JSONUtils.java
    new class to collect common JSON actions
* Iteratorable.java
    wrap Iterator as Iterable for foreach usage
* (many)
    update to use alternate utils methods
2009-11-19 22:39:53 +00:00
nlevitt 3b5e9f7c45 Improve pretty printing of arrays, in particular the fetch-history array. 2009-11-13 02:21:38 +00:00
gojomo c7ed3e775d Misc
* ArchiveUtils.java
   (checkContinue) utility method relocated from ToeThread for converting interrupt status to exception
* ClassKeyMatchesRegExpDecideRule.java
   properly set up crawlController property for auto-wiring
2009-10-05 22:59:46 +00:00
gojomo f82432ca9d Improve conciseness/clarity of ReplayInputStream/ReplayCharSequence usage/cleanup
* ReplayCharSequence.java
    implement Closeable
* ArchiveUtils.java
    (closeQuietly) for ReplayCharSequence 
* ExtractorCSS.java, ExtractorDOC.java, ExtractorHTML.java, ExtractorJS.java, ExtractorXML.java, HTTPContentDigest.java, CrawlServer.java, MirrorWriterProcessor.java, WARCWriterProcessor.java
    cover more with finally blocks; use closeQuietly() variants
2009-07-15 23:30:22 +00:00
gojomo 58486c5f18 (copying from h2 to h3)
Fix for [HER-1277] speculative javascript link caused 404s for "http%3A" and "www.example.com" like strings
* ExtractorJS.java
    perform additional decoding or prepending of "http:" for speculative candidate URIs that appear to need it
* ArchiveUtils.java
    add utility set/method with list of all TLDs
    
Fix for [HER-1524] speculativeFixup in ExtractorJS should maintain URL scheme
* ExtractorJS.java
    (speculativeFixup) maintain origin URI scheme if apparent target host is same
2009-05-13 02:14:15 +00:00
gojomo e047bf68e0 copy 'springy' branch to heritrix3 2009-05-11 22:56:36 +00:00