Commit Graph
16 Commits
Author SHA1 Message Date
Noah Levitt 3dce1e909d Change scope of most/all default-scope i.e. package-scoped class members to protected, to make heritrix more friendly to third-party extensions. (I don't think there was ever an intent to limit the scope of any of these to package-only.) 2012-06-05 17:55:41 -07:00
Noah Levitt 52703c6eb4 * ArchiveUtils.java
doubleToString() - return "NaN" for NaN (inf symbol didn't quite make sense to me)
2012-03-15 18:46:01 -07:00
Kenji Nagahashi adc82c1c6f Second round of HTML cleanup:
- added missing <html> and <body>
- fixed non-working CSS link, added CSS link to all pages
- replaced layout by <br> and &nbsp; with structural tags with CSS
- removed superfluous output, notably in BeanBrowseResource
- <code>-ify variable names in ScriptResource
- added special handling of NaN in doubleToString()
2012-03-03 00:46:56 -08:00
gojomo 3f98fc248b * ArchiveUtils
(readFully) return actual number of bytes read into array
* FileUtils
    (readFullyToFile) simplify; also return total bytes read
2011-04-06 20:22:30 +00:00
gojomo 9b78770027 [HER-1865] JDK6u23 breaks GzippedInputStream & W/ARCReaders with different GZIP handling
* GZIPMembersInputStream
    new workaround class that offers choice of pre/post JDK6u23 behavior and new accessors for finding member boundary offsets
* GZIPMembersInputStreamTest
    test for above
* ArchiveUtils, ARCWriter
    move static compress utilities to ArchiveUtils
* ArchiveReader, (W)ARCReaderFactory, (W)ARCWriterTest
    adapt to use GZIPMembersInputStream
* GzippedInputStream(Test)
    deleted
2011-02-16 01:12:25 +00:00
gojomo b388479d32 [HER-1727] (W)ARC filename uniqueness guarantees: adminport option, process_id option(?), JVM-global-repeat-suppression
* Heritrix.java
    make heritrix.hostname, heritrix.pid, heritrix.port available as global properties
* PropertyUtils.java, PropertyUtilsTest.java
    utility methods to interpolate string values from one or several supplied Properties instances
* ArchiveUtils.java
    utility methods to give timestamps guaranteed larger/different than any previously-issued timestamp

* WriterPoolSettings.java, WriterPool.java, ARCWriterPool.java, WARCWriterPool.java, ARCWriterPoolTest.java
    improve field names, change 'suffix' to more general 'template'
* WriterPoolMember.java
    replace 'suffix' with 'template' which is interpolated when specific name is needed
    set default template to pattern extremely unlikely to generate duplicate filenames
    change default prefix to 'WEB'
    centralize creation of new basenames into generateNewBasename() method, which internalizes timetamp/serialNo minting and interpolation
* WriterPoolProcessor.java, ARCWriterProcessor.java, WARCWriterProcessor.java
    serve as own WriterPoolSettings instance
    replace 'suffix' with 'template' 
    
* ARCWriter.java
    accept (but truncate) 17-digit timestamps at creation
* DefaultWriterPoolSettings.java, TimestampSerialNo.java
    delete as superfluous
2010-08-20 00:21:43 +00:00
gojomo 72fb1c55bd Move TLD list to resource file for easier drop-in updating
* ArchiveUtils.java
    read TLDs from tlds-alpha-by-domain.txt
* ArchiveUtilsTest.java
    verify TLDs read
* tlds-alpha-by-domain.txt
    from http://data.iana.org/TLD/tlds-alpha-by-domain.txt
2010-04-22 01:05:23 +00:00
nlevitt df6e31697d Rename short report methods:
- singleLineReport() -> shortReportLine()
 - singleLineReportTo() -> shortReportLineTo()
 - singleLineReportData() -> shortReportMap()
2009-12-09 02:53:02 +00:00
gojomo 04012279fc Misc logging-cleanup/refactoring
* CrawlerJournal.java
    improve exception logging
    combine writeLine() methods to use varargs
* FrontierJournal.java
    improve exception logging
* ArchiveUtils.java
    replace ignored  count parameter with meaingful boolean one
* CrawlURI.java
    use updated ArchiveUtils method
2009-12-04 22:23:48 +00:00
nlevitt 8ba641ed4b Part of [HER-1603] Restletify: xml representations of all resources that also
offer html
* XmlMarshaller.java
    new class to write xml based on nested map structure
* BeanBrowseResource.java ScriptResource.java EngineResource.java
  JobResource.java JobRelatedResource.java
    support xml representation
* WorkQueueFrontier.java PrecedenceProvider.java WorkQueue.java ToeThread.java
  ToePool.java CrawlJob.java Engine.java Frontier.java CrawlController.java
  StatisticsTracker.java CrawledBytesHistotable.java FetchStats.java
  ProcessorChain.java CrawlURI.java Reporter.java
    offer same data that is returned in the single line reports in a map
    suitable for use with XmlMarshaller
* FileUtils.java
    new method tryToCanonicalize(File)
2009-11-25 01:43:57 +00:00
gojomo e42e2ed7f6 Cleanup/consolidation/expansion of *Utils classes
* IoUtils.java
    remove class that confusingly-overlaps with commons IOUtils; move remaining methods to ArchiveUtils or archive's FileUtils
* ArchiveUtils.java, FileUtils.java
    receive relocated methods; eliminated deprecated or unused methods
* JSONUtils.java
    new class to collect common JSON actions
* Iteratorable.java
    wrap Iterator as Iterable for foreach usage
* (many)
    update to use alternate utils methods
2009-11-19 22:39:53 +00:00
nlevitt 3b5e9f7c45 Improve pretty printing of arrays, in particular the fetch-history array. 2009-11-13 02:21:38 +00:00
gojomo c7ed3e775d Misc
* ArchiveUtils.java
   (checkContinue) utility method relocated from ToeThread for converting interrupt status to exception
* ClassKeyMatchesRegExpDecideRule.java
   properly set up crawlController property for auto-wiring
2009-10-05 22:59:46 +00:00
gojomo f82432ca9d Improve conciseness/clarity of ReplayInputStream/ReplayCharSequence usage/cleanup
* ReplayCharSequence.java
    implement Closeable
* ArchiveUtils.java
    (closeQuietly) for ReplayCharSequence 
* ExtractorCSS.java, ExtractorDOC.java, ExtractorHTML.java, ExtractorJS.java, ExtractorXML.java, HTTPContentDigest.java, CrawlServer.java, MirrorWriterProcessor.java, WARCWriterProcessor.java
    cover more with finally blocks; use closeQuietly() variants
2009-07-15 23:30:22 +00:00
gojomo 58486c5f18 (copying from h2 to h3)
Fix for [HER-1277] speculative javascript link caused 404s for "http%3A" and "www.example.com" like strings
* ExtractorJS.java
    perform additional decoding or prepending of "http:" for speculative candidate URIs that appear to need it
* ArchiveUtils.java
    add utility set/method with list of all TLDs
    
Fix for [HER-1524] speculativeFixup in ExtractorJS should maintain URL scheme
* ExtractorJS.java
    (speculativeFixup) maintain origin URI scheme if apparent target host is same
2009-05-13 02:14:15 +00:00
gojomo e047bf68e0 copy 'springy' branch to heritrix3 2009-05-11 22:56:36 +00:00