* .classpath, pom.xml, org/apache/commons/pool/impl/*
remove references to and patched classes based on commons-pool
* WriterPool.java
move instance creation/round-robin-pool/dispose into this class
close (rather then recycle) extra writers when crawl slows (via tracking time of last need/rollover)
* WriterPoolMember.java
add isOversize() for external check of ready-to-rollover file
avoid rollover before every record (so that set of related WARC records aren't split between files)
* ARCWriterPool.java, WARCWriterPool.java
remove BasePoolableObjectFactory reference, implement class-appropriate makeWriter()
* ARCWriterPoolTest.java
adjust for new never-timeout behavior
* ARCWriterPoolProcessor.java WARCWriterPoolProcessor.java
adjust constructor calls
* WriterPoolProcessor.java
replace poolMaxWaitMs (which caused eventual complete timeout from waiting for writer) with maxWaitForIdleMs, a much smaller value which only controls how long a thread waits for an reusable writer before considering creating a new one. (If creation isn't allowed, threads will now wait indefinitely for a writer to become available.)
* Heritrix.java
make heritrix.hostname, heritrix.pid, heritrix.port available as global properties
* PropertyUtils.java, PropertyUtilsTest.java
utility methods to interpolate string values from one or several supplied Properties instances
* ArchiveUtils.java
utility methods to give timestamps guaranteed larger/different than any previously-issued timestamp
* WriterPoolSettings.java, WriterPool.java, ARCWriterPool.java, WARCWriterPool.java, ARCWriterPoolTest.java
improve field names, change 'suffix' to more general 'template'
* WriterPoolMember.java
replace 'suffix' with 'template' which is interpolated when specific name is needed
set default template to pattern extremely unlikely to generate duplicate filenames
change default prefix to 'WEB'
centralize creation of new basenames into generateNewBasename() method, which internalizes timetamp/serialNo minting and interpolation
* WriterPoolProcessor.java, ARCWriterProcessor.java, WARCWriterProcessor.java
serve as own WriterPoolSettings instance
replace 'suffix' with 'template'
* ARCWriter.java
accept (but truncate) 17-digit timestamps at creation
* DefaultWriterPoolSettings.java, TimestampSerialNo.java
delete as superfluous
* CheckpointService.java
use static method on Checkpoint for validity check
* Checkpoint.java
throw early-startup exception if attempting resumption of invalid checkpoint
* BloomFilter64bit.java
include the split-to-subarrays (for larger bitfields) and round-up-to-power-of-2 (for performance) options previously in largely-redundant classes
fit a number of problems with int/long overflow and bitwise ops
add methods for reporting/testing
* BloomFilter.java
add methods for reporting/testing
* BloomFilterTest.java, BloomFilter64bitTest.java
more extensive tests, including two lengthy tests of default/oversized blooms usually disabled by renaming
* BloomFilter32bit.java, BloomFilter32bitSplit.java, BloomFilter32bp2.java, BloomFilter32bp2Split.java
deleted as buggy or redundant
* BenchmarkBlooms.java
move to test source dir
* BloomUriUniqFilter.java
change to accept filter instance (rather than parameters) for added configuration flexibility
fix comments
* BloomUriUniqFilterTest.java
supply filter not paramters
* BloomFilter64bit.java
fix long arithmetic calculating bit-length m in constructor
slight optimization of add()
* BloomFilterTest.java
abstract superclass for bloom impl class tests
* BloomFilter64bitTest.java
sanity check that set bits, after a handful of adds into default-sized filter, occupy top and bottom 20% of range (which would have caught this bug earlier)
* ReplayCharSequence.java, GenericReplayCharSequence.java, InMemoryReplayCharSequence.java
store actual first exception for reference
* ExtractorHTML.java
report exception in nonfatal-log, rather than WARNING (which was still polluting job log more than desired -- but should still be logged to allow future investigation of sites where encoding issues may exist)
* ReplayCharSequence.java
new interface method: getDecoderExceptionCount(), reporting >0 if any decoding problems were encountered and silently replaced
* GenericReplayCharSequence.java
count decoding problems, and only log as FINE rather than WARNING
* InMemoryReplayCharSequence.java
try decoding with reporting; if any problems note at least one then decode without reporting errors
* ExtractorHTML.java
after using ReplayCharSequence, log WARNING if nonzero decoding errors
* GenericReplayCharSequence.java
* Scoper.java
* FetchFTP.java
* FetchHTTP.java
demote a bunch of INFO logging to FINE so that be default, console isn't filled with output for every URI handled by a busy crawl
* CheckpointService.java
reorganize so that checkpoint timedTask is reset whenever interval is reset on a running crawl
* CheckpointSuccessEvent.java, CrawlJob.java
report successes as new kind of ApplicationEvent; propagagte to job log
* BdbModule.java
remove obsolete unconsulted checkpointCopyLogs setting
* UriUtils.java
move likely-URI heuristics here for better consistency, reuse, testing, improvement
* UriUtilsTest.java
testing for UriUtils
* ExtractorJS.java
move likely-URI support to UriUtils; use new test
* ExtractoSWF.java
use new UriUtils likely-URI testing
* ExtractorHTML.java
move likely-URI testing to UriUtils, use new test
* JerichoExtractorHTML.java
use new UriUtils likely-URI testing
* CachedBdbMap.java
@deprecated
* BdbModule.java
note (unused) 'CBM' option is deprecated
* ObjectIdentityBdbCache.java
tighten logging to FINE
* ObjectIdentityBdbCacheTest.java
fix test that was only trying to log, rather than fail, problems
* ArchiveRecord.java
do not decrement position after EOR, port from H1
* ARCWriterTest.java
added testArchiveRecordEORConsistent(), testArchiveRecordMarkSupport()
port from H1
* EnhancedEnvironment.java
in getClassCatalog(), when read-only env, insist on read-only catalog
* PersistLoadProcessor.java
remove existing curi members from prior to prevent clobbering source
* PersistProcessor.java
set historyDbConfig read-only, set je.log.faultReadSize for
compatability with BDBModule.setup()
* Arc2Warc.java
when HTTP response, write response record rather than resource
enable reconstruction of ARC from WARC via namedFields
* ARCRecord.java
added headerString and accessor to save verbatim ARC header string
* **/pom.xml
bump version identifiers to "3.1.1-SNAPSHOT" (odd minor number meaning dev version, odd patch number indicating expected next patch number of 3.0.x release if before a 3.2.0 release)
* UURIFactoryTest.java
specify stirng literal with escape-encoding, rather than as raw UTF-8
* **/pom.xml
include 'project.build.sourceEncoding' property to suppress platform-encoding warnings
(unfortunately, neither this nor similar 'maven.compile.encoding' property actually casues maven build to interpret Java source as UTF-8)
offer html
* XmlMarshaller.java
new class to write xml based on nested map structure
* BeanBrowseResource.java ScriptResource.java EngineResource.java
JobResource.java JobRelatedResource.java
support xml representation
* WorkQueueFrontier.java PrecedenceProvider.java WorkQueue.java ToeThread.java
ToePool.java CrawlJob.java Engine.java Frontier.java CrawlController.java
StatisticsTracker.java CrawledBytesHistotable.java FetchStats.java
ProcessorChain.java CrawlURI.java Reporter.java
offer same data that is returned in the single line reports in a map
suitable for use with XmlMarshaller
* FileUtils.java
new method tryToCanonicalize(File)
gojomo