* .classpath, pom.xml, org/apache/commons/pool/impl/*
remove references to and patched classes based on commons-pool
* WriterPool.java
move instance creation/round-robin-pool/dispose into this class
close (rather then recycle) extra writers when crawl slows (via tracking time of last need/rollover)
* WriterPoolMember.java
add isOversize() for external check of ready-to-rollover file
avoid rollover before every record (so that set of related WARC records aren't split between files)
* ARCWriterPool.java, WARCWriterPool.java
remove BasePoolableObjectFactory reference, implement class-appropriate makeWriter()
* ARCWriterPoolTest.java
adjust for new never-timeout behavior
* ARCWriterPoolProcessor.java WARCWriterPoolProcessor.java
adjust constructor calls
* WriterPoolProcessor.java
replace poolMaxWaitMs (which caused eventual complete timeout from waiting for writer) with maxWaitForIdleMs, a much smaller value which only controls how long a thread waits for an reusable writer before considering creating a new one. (If creation isn't allowed, threads will now wait indefinitely for a writer to become available.)
* Heritrix.java
make heritrix.hostname, heritrix.pid, heritrix.port available as global properties
* PropertyUtils.java, PropertyUtilsTest.java
utility methods to interpolate string values from one or several supplied Properties instances
* ArchiveUtils.java
utility methods to give timestamps guaranteed larger/different than any previously-issued timestamp
* WriterPoolSettings.java, WriterPool.java, ARCWriterPool.java, WARCWriterPool.java, ARCWriterPoolTest.java
improve field names, change 'suffix' to more general 'template'
* WriterPoolMember.java
replace 'suffix' with 'template' which is interpolated when specific name is needed
set default template to pattern extremely unlikely to generate duplicate filenames
change default prefix to 'WEB'
centralize creation of new basenames into generateNewBasename() method, which internalizes timetamp/serialNo minting and interpolation
* WriterPoolProcessor.java, ARCWriterProcessor.java, WARCWriterProcessor.java
serve as own WriterPoolSettings instance
replace 'suffix' with 'template'
* ARCWriter.java
accept (but truncate) 17-digit timestamps at creation
* DefaultWriterPoolSettings.java, TimestampSerialNo.java
delete as superfluous
* profile-crawler-beans.cxml
Added a commented-out configuration for new DiskSpaceMonitor
* DiskSpaceMonitor.java
Implements the required functionality
* package-info.java
Description of new package for monitoring utilities
* LowDiskPauseProcessor.java
Now marked as deprecated
* CheckpointService.java
use static method on Checkpoint for validity check
* Checkpoint.java
throw early-startup exception if attempting resumption of invalid checkpoint
* CheckpointService.java
(getAvailableCheckpointDirectories) return List, filtered to leave out directories without a 'valid' stamp; log WARNING to _out so that the attentive operator can manually clean-up/investigate
* ExtractorHTTP.java
add property inferRootPage; if true (not the default), always consider '/' inferred from all HTTP(S) URIs
* Hop.java, LinkContext.java
add new hop-type 'I' for inferred URIs; add corresponding LinkContext
* ExtractorImpliedURI.java
use new 'I'nferred types
* CrawlURI.java
improve comments for fetchAttempts methods
(resetForRescheduling) added to clear per-scheduling state
* WorkQueueFrontier.java
call resetForRescheduling just before scheduling-at-specific-time
* FrontierJournal.java, AbstractFrontier.java
renamings: prefer 'reenqueue' instead of 'reschedule' for describing simple (non-timed) retries
* AbstractFrontier.java
change config and defaults for 'inbound' and 'outbound': allowing any BlockinqQueue bean, of any size, to be specified if desired
* **/profile-crawler-beans.cxml
changed commented-out descriptions of defaults to reflect what's used if nothing is specified for 'outbound' and 'inbound'
* EngineResource.java
add 'exit' control area at bottom of page
allow JVM-exit if confirm-box checked and either no jobs built/running or extra confirm boxes for each built job checked
* Flash.java
fix to allow multiple flashes per page load
* JobResource.java, CrawlJob.java
move useful status string method to CrawlJob
* BloomFilter64bit.java
include the split-to-subarrays (for larger bitfields) and round-up-to-power-of-2 (for performance) options previously in largely-redundant classes
fit a number of problems with int/long overflow and bitwise ops
add methods for reporting/testing
* BloomFilter.java
add methods for reporting/testing
* BloomFilterTest.java, BloomFilter64bitTest.java
more extensive tests, including two lengthy tests of default/oversized blooms usually disabled by renaming
* BloomFilter32bit.java, BloomFilter32bitSplit.java, BloomFilter32bp2.java, BloomFilter32bp2Split.java
deleted as buggy or redundant
* BenchmarkBlooms.java
move to test source dir
* BloomUriUniqFilter.java
change to accept filter instance (rather than parameters) for added configuration flexibility
fix comments
* BloomUriUniqFilterTest.java
supply filter not paramters
* dist/src/main/conf/jobs/profile-defaults/profile-crawler-beans.cxml
* engine/src/main/resources/org/archive/crawler/restlet/profile-crawler-beans.cxml
add intended REJECT decision for the late (rules[4]) MatchesListRegexDecideRule
* BloomFilter64bit.java
fix long arithmetic calculating bit-length m in constructor
slight optimization of add()
* BloomFilterTest.java
abstract superclass for bloom impl class tests
* BloomFilter64bitTest.java
sanity check that set bits, after a handful of adds into default-sized filter, occupy top and bottom 20% of range (which would have caught this bug earlier)
* ExtractorHTML.java
smaller caps on element-name, attribute-name, attribute-value lengths -- costly match-failures on bad/non-HTML may be eating a lot of cycles
gojomo