* ANVLRecord.java
extend LinkedList instead of ArrayList (it was inane to guess the number of records everywhere, when it was never much more than ~10)
* ANVLRecords.java
remove (seemed to be an incomplete duplicate of ANVLRecord)
* WARCRecordInfo.java
addExtraHeader() - convenience method
* Arc2Warc.java, WARCWriter.java, WARCWriter.java, WARCWriterProcessor.java
use WARCRecordInfo.addExtraHeader() where appropriate, and use no-argument ANVLRecord constructor instead of deprecated ANVLRecord(int)
* WARCRecordInfo.java
get rid of constructor with arguments, add setUrl() and setType()
* WARCWriter.java
get rid of writeRecord() taking a million arguments
* Arc2Warc.java, WARCWriter.java, WARCWriterTest.java, WARCWriterProcessor.java, WARCWriterProcessorTest.java
construct WARCRecordInfo object and call WARCWriter.writeRecord(recordInfo) instead of million argument method
* WARCRecordInfo.java
new class to hold warc record info to write, so we don't have to pass around all these long lists of variables
* WARCConstants.java
new enum WARCRecordType replaces bunch of string constants
* WARCWriter.java, WARCWriterProcessor.java, WARCWriterTest.java
use WARCRecordInfo, WARCRecordType
* LaxURI.java
parseUriReference() - fix logic testing if uri string is a path to respect scheme-less uris even if they contain colons
* UURIFactoryTest.java
add test for scheme-less relative uri (with colons)
* UriUtils.java
isLikelyFalsePositive() - do not flag urls starting with ../ as likely false positives
* UriUtilsTest.java
remove test url that is now rejected
* BasicProfileTest.java, SelfTestBase.java, CrawlControllerTest.java, PrecedenceLoader.java, MigrateH1to3Tool.java, CrawlerLoggerModule.java, StatisticsTracker.java, CheckpointUtils.java, BdbUriUniqFilter.java, ARCWriterProcessorTest.java, WARCWriterProcessorTest.java, PersistProcessor.java, WriterPoolProcessor.java, PrefixFinderTest.java, StoredQueueTest.java, FileUtilsTest.java, ObjectIdentityBdbManualCacheTest.java, ObjectIdentityBdbCacheTest.java, ObjectPlusFilesOutputStream.java, TestUtils.java, TmpDirTestCase.java, Engine.java
Replace calls to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir). In these cases the calls were either already in a spot where the possible IOException would be handled appropriately, or the line was trivially moved into such a block.
* ActionDirectory.java, Engine.java
replace calls to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir), and throw IllegalStateException on failure
* BdbModule.java
setup() - replace call to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir) and add "throws IOException" - conveniently the place where this method is called was already in a try block that catches IOException
* Checkpoint.java
generateFrom() - replace call to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir) and add "throws IOException"
* CheckpointService.java
move call to Checkpoint.generateFrom() inside existing try block since it now can throw IOException
* Recorder.java
ensure(File) - replace call to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir), and throw IllegalStateException on failure
new Recorder(File,String,int,int) - call ensure() on the correct object, the containing directory; and remove redundant call to ensure()
(specifically, when they're in the query string)
* UURIFactory.java
fixup() - replace \ with / up to the first ? in the url, instead of
anywhere in the url
and update comments
* UURIFactoryTest.java
backslash test
* OpenJDK7GZIPInputStream.java, OpenJDK7InflatedInputStream.java
backport the working GZIP implementation from OpenJDK7, with minimal
renames/imports/private-to-protected changes to enable member-at-a-time reading
* GZIPMembersInputStream
base on OpenJDK7 implementation to avoid pre- & post-6u23 codepaths, and 6u23/24 readHeader bug
override readTrailer to set member-end; update comment to reflect end-uncertainty without EOF-per-member-mode
* GZIPMembersInputStreamTest, ARCWriterTest
test tweaks
* ArchiveReader
improve warning message
* WriterPoolMember
attempt pre-delete of target filename, warn on failure, for better odds on some filesystems
* ARCWriterTest, WARCWriterTest
ensure timely close of readers/writers in case filenames are reused
restore originally-intended unique target names
* GZIPMembersInputStream
new workaround class that offers choice of pre/post JDK6u23 behavior and new accessors for finding member boundary offsets
* GZIPMembersInputStreamTest
test for above
* ArchiveUtils, ARCWriter
move static compress utilities to ArchiveUtils
* ArchiveReader, (W)ARCReaderFactory, (W)ARCWriterTest
adapt to use GZIPMembersInputStream
* GzippedInputStream(Test)
deleted
* IdentityCacheable
new interface required of objects stored in ObjectIdentityCaches
* IdentityCacheableWrapper
wrapper for storing arbitrary objects in ObjectIdentityCaches
* ObjectIdentityCache
keys now always Strings
values now always IdentityCacheables
new dirtyKey() method to ensure a key is persisted
* ObjectIdentityBdbCache, ObjectIdentityMemCache
update for new ObjectIdentityCache shape; still using legacy GC magic
* ObjectIdentityBdbManualCache
alternate implementation relying on dirtying to ensure persistence
use Guava library MapMaker for soft memMap; capped dirtyMap
* BdbModule
use ObjectIdentityBdbManualCache by default
up expectedConcurrency default to 64
* BdbModuleTest, ObjectIdentityBdbCacheTest, ObjectIdentityBdbManualCacheTest
update, add tests
* Frontier
FrontierGroup as IdentityCacheable
* AbstractFrontier, WorkQueueFrontier, BdbFrontier
touchup queue/group accessors
make queue instances dirty whenever mutated
* WorkQueue, CrawlHost, CrawlServer
IdentityCacheable support; appropriate makeDirty()s
* ServerCache, DefaultServerCache
update for new ObjectIdentityCache shape
* Arc2Warc, Warc2Arc
adapt to new constructors, settings-object
* MiserOutputStream
stream to monitor position, optionally suppress flushes
* WriterPoolMember
use MiserOutputStream rather than special file-access to find compressed offsets
use shared settings object rather than copying multiple values
* WriterPoolSettings
add settings for frequentFlushes and writeBufferSize for IO optimization
* ARCReader, ARCWriter, ARCWriterPool
adapt to new constructors, settings-object
* WriterPoolSettingsData
impl of WriterPoolSettings for testing and adhoc use
* WARCWriter
adapt to new constructors, settings-object
remove checkSize rollover to new file (done elsewhere)
remove extraneous id-generator code/static-method
* WARCWriterPool
adapt to new constructors, settings-object
* WARCWriterPoolSettings, WARCWriterPoolSettingsData
add recordIDGenerator setting, impl class for testing/adhoc use
* WARCWriterTest
adapt to new constructors, settings-object
* Generator, GeneratorFactory
removed in favor of RecordIDGenerator interface
* RecordIDGenerator
common interface for classes that can provide WARC record IDs
* UUIDGenerator, UUIDGeneratorTest
adapt to derive from RecordIDGenerator
avoid throwing exceptions
* ARCWriterPoolTest, ARCWriterTest
adapt to new constructors, settings-object
* WARCWriterProcessor
do checkSize file-rollover here, so related records don't span WARCs
calculate processor's totalBytesWritten via position offset changes (as before stats-additions)
delegate record-ID generation
* WriterPoolProcessor
add frequentFlushes and writeBufferSize from new WARCWriterPoolSettings interface
* AutoKryo
extension of Kryo to allow classes to control their own registration, trigger registration of associated classes, and deserialize classes without no-arg constructors
* KryoBinding
binding for use with BDB that uses AutoKryo serialization for a 2X-4X reduction in byte[] size
* UURI
improved serialization via Externalizable and Kryo's CustomSerialization methods
* BdbModule
discard deprecated CachedBDBMap option
(getObjectCache) extend with both declaredClass and valueClass (for when map values are specializations of the declared type, as with frontier.allQueues)
adjust type declarations
* ObjectIdentityBdbCache
use KryoBinding rather than SerialBinding
* CachedBdbMapTest
discarded
* BdbFrontier, BdbServerCache, StatisticsTracker
adjust type declarations, objectCache creation
* BdbMultipleWorkQueues
use KryoBinding rather than (Recycling)SerialBinding
* BdbWorkQueue, CrawlServer, CrawlHost, CrawlURI
add autoregister support
* LinkContext
public for kryo registration
* LaxURI.java
increase tolerance with regard to path-segments
* UURIFactory.java
allow digits in scheme regex
discard old supported-schemes code
* UURIFactoryTest.java
tests for roundtrip serialization of problematic hostlike-scheme and bars-in-path
* CrawlerLoggerModule.java
discard special-casing of URI scheme errors
* SchemeNotInSetDecideRule.java
rule to apply to URIs of unknown schemes; by default REJECTs those not usually handled by Heritrix
* **/profile-crawler-beans.cxml
add REJECT SchemeNotInSetDecideRule
* .classpath, pom.xml, org/apache/commons/pool/impl/*
remove references to and patched classes based on commons-pool
* WriterPool.java
move instance creation/round-robin-pool/dispose into this class
close (rather then recycle) extra writers when crawl slows (via tracking time of last need/rollover)
* WriterPoolMember.java
add isOversize() for external check of ready-to-rollover file
avoid rollover before every record (so that set of related WARC records aren't split between files)
* ARCWriterPool.java, WARCWriterPool.java
remove BasePoolableObjectFactory reference, implement class-appropriate makeWriter()
* ARCWriterPoolTest.java
adjust for new never-timeout behavior
* ARCWriterPoolProcessor.java WARCWriterPoolProcessor.java
adjust constructor calls
* WriterPoolProcessor.java
replace poolMaxWaitMs (which caused eventual complete timeout from waiting for writer) with maxWaitForIdleMs, a much smaller value which only controls how long a thread waits for an reusable writer before considering creating a new one. (If creation isn't allowed, threads will now wait indefinitely for a writer to become available.)
* Heritrix.java
make heritrix.hostname, heritrix.pid, heritrix.port available as global properties
* PropertyUtils.java, PropertyUtilsTest.java
utility methods to interpolate string values from one or several supplied Properties instances
* ArchiveUtils.java
utility methods to give timestamps guaranteed larger/different than any previously-issued timestamp
* WriterPoolSettings.java, WriterPool.java, ARCWriterPool.java, WARCWriterPool.java, ARCWriterPoolTest.java
improve field names, change 'suffix' to more general 'template'
* WriterPoolMember.java
replace 'suffix' with 'template' which is interpolated when specific name is needed
set default template to pattern extremely unlikely to generate duplicate filenames
change default prefix to 'WEB'
centralize creation of new basenames into generateNewBasename() method, which internalizes timetamp/serialNo minting and interpolation
* WriterPoolProcessor.java, ARCWriterProcessor.java, WARCWriterProcessor.java
serve as own WriterPoolSettings instance
replace 'suffix' with 'template'
* ARCWriter.java
accept (but truncate) 17-digit timestamps at creation
* DefaultWriterPoolSettings.java, TimestampSerialNo.java
delete as superfluous
* BloomFilter64bit.java
include the split-to-subarrays (for larger bitfields) and round-up-to-power-of-2 (for performance) options previously in largely-redundant classes
fit a number of problems with int/long overflow and bitwise ops
add methods for reporting/testing
* BloomFilter.java
add methods for reporting/testing
* BloomFilterTest.java, BloomFilter64bitTest.java
more extensive tests, including two lengthy tests of default/oversized blooms usually disabled by renaming
* BloomFilter32bit.java, BloomFilter32bitSplit.java, BloomFilter32bp2.java, BloomFilter32bp2Split.java
deleted as buggy or redundant
* BenchmarkBlooms.java
move to test source dir
* BloomUriUniqFilter.java
change to accept filter instance (rather than parameters) for added configuration flexibility
fix comments
* BloomUriUniqFilterTest.java
supply filter not paramters
* BloomFilter64bit.java
fix long arithmetic calculating bit-length m in constructor
slight optimization of add()
* BloomFilterTest.java
abstract superclass for bloom impl class tests
* BloomFilter64bitTest.java
sanity check that set bits, after a handful of adds into default-sized filter, occupy top and bottom 20% of range (which would have caught this bug earlier)
Noah Levitt