lee/heritrix3
1
0 Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-11-15 18:34:03 +00:00
Code Issues Packages Projects Releases Wiki Activity
442 Commits 47 Branches 34 Tags
b388479d32e07aba25754cec1ca6556407cd6155
Commit Graph
5 Commits
Author SHA1 Message Date
gojomo b388479d32 [HER-1727] (W)ARC filename uniqueness guarantees: adminport option, process_id option(?), JVM-global-repeat-suppression 
* Heritrix.java
    make heritrix.hostname, heritrix.pid, heritrix.port available as global properties
* PropertyUtils.java, PropertyUtilsTest.java
    utility methods to interpolate string values from one or several supplied Properties instances
* ArchiveUtils.java
    utility methods to give timestamps guaranteed larger/different than any previously-issued timestamp

* WriterPoolSettings.java, WriterPool.java, ARCWriterPool.java, WARCWriterPool.java, ARCWriterPoolTest.java
    improve field names, change 'suffix' to more general 'template'
* WriterPoolMember.java
    replace 'suffix' with 'template' which is interpolated when specific name is needed
    set default template to pattern extremely unlikely to generate duplicate filenames
    change default prefix to 'WEB'
    centralize creation of new basenames into generateNewBasename() method, which internalizes timetamp/serialNo minting and interpolation
* WriterPoolProcessor.java, ARCWriterProcessor.java, WARCWriterProcessor.java
    serve as own WriterPoolSettings instance
    replace 'suffix' with 'template' 
    
* ARCWriter.java
    accept (but truncate) 17-digit timestamps at creation
* DefaultWriterPoolSettings.java, TimestampSerialNo.java
    delete as superfluous
 2010-08-20 00:21:43 +00:00
gojomo e42e2ed7f6 Cleanup/consolidation/expansion of *Utils classes 
* IoUtils.java
    remove class that confusingly-overlaps with commons IOUtils; move remaining methods to ArchiveUtils or archive's FileUtils
* ArchiveUtils.java, FileUtils.java
    receive relocated methods; eliminated deprecated or unused methods
* JSONUtils.java
    new class to collect common JSON actions
* Iteratorable.java
    wrap Iterator as Iterable for foreach usage
* (many)
    update to use alternate utils methods
 2009-11-19 22:39:53 +00:00
gojomo 64d3fd212f [HER-1659] make default WARC size comply with spec; adjust default pool size for fewer odd-sized (W)ARCs 
* ArchiveFileConstants.java
    remove superfluous constant
* WriterPool.java
    DEFAULT_MAX_ACTIVE -> 1
* WriterPoolMember.java
    update constant use; remove deprecated+unused methods
* WriterPoolProcessor.java
    pull storePaths up from subclasses, but set default via subclass method
    set defaultMaxSize via subclass method
* WARCWriterProcessor.java, ARCWriterProcessor.java
   (getDefaultMaxFileSize), (getDefaultStorePaths) as appropriate
 2009-08-06 00:55:27 +00:00
gojomo d336ea760f quiet excessive logging 2009-06-05 02:00:07 +00:00
gojomo e047bf68e0 copy 'springy' branch to heritrix3 2009-05-11 22:56:36 +00:00