* OpenJDK7GZIPInputStream.java, OpenJDK7InflatedInputStream.java
backport the working GZIP implementation from OpenJDK7, with minimal
renames/imports/private-to-protected changes to enable member-at-a-time reading
* GZIPMembersInputStream
base on OpenJDK7 implementation to avoid pre- & post-6u23 codepaths, and 6u23/24 readHeader bug
override readTrailer to set member-end; update comment to reflect end-uncertainty without EOF-per-member-mode
* GZIPMembersInputStreamTest, ARCWriterTest
test tweaks
ensure mark()ing more frequently, when reset to before current position won't be needed
* ArchiveReaderFactory
avoid misinterpreting a "c:" local path as a URI scheme
better logging of caught/recovery-attempted IOException
* GZIPMemberInputStream
handling of previously-unrecognized case in pre-JDK6u23: attempt at read() exhausts inflater with no new bytes available (previous nonzero-length read had not indicated inflater finished)
* URIAuthorityBasedQueueAssignmentPolicy.java
whois urls all go in special "whois" queue
* PreconditionEnforcer.java, CrawlURI.java
move markPrerequisite() method from PreconditionEnforcer to CrawlURI
* profile-crawler-beans.cxml
commented-out whois fetcher clause
* FetchStatusCodes.java
new status codes for whois
* ServerCache.java
getHostFor() - return "whois:" for whois urls, following dns: convention
* UURIFactory.java
ugly hack for whois urls
* SchemeNotInSetDecideRule.java
add whois to list of supported schemes
* WriterPoolProcessor.java
shouldWrite() - we should write successful whois fetches
* WARCWriterProcessor.java
write whois records
* FetchWhois.java
the fetcher
* ArchiveReader
improve warning message
* WriterPoolMember
attempt pre-delete of target filename, warn on failure, for better odds on some filesystems
* ARCWriterTest, WARCWriterTest
ensure timely close of readers/writers in case filenames are reused
restore originally-intended unique target names
* CookieSpecBase
use Guava InternetDomainName public-suffixes to both (1) prevent storing cookie on public-suffix; (2) prevent looking-up cookies for public-suffixes
* BdbCookieStorage
correct checkpoint/relaunch behavior: don't reuse persisted cookies on normal relaunches; do reuse on resume-from-checkpoint
* GZIPMembersInputStream
new workaround class that offers choice of pre/post JDK6u23 behavior and new accessors for finding member boundary offsets
* GZIPMembersInputStreamTest
test for above
* ArchiveUtils, ARCWriter
move static compress utilities to ArchiveUtils
* ArchiveReader, (W)ARCReaderFactory, (W)ARCWriterTest
adapt to use GZIPMembersInputStream
* GzippedInputStream(Test)
deleted
* WARCWriterProcessor
move checkSize into block protected by catch/finally cleanup of errors
* WARCWriterProcessorTest
test to simulate record-write-failure and verify does not cause pool lockup
* WARCWriterTest, WriterPoolMember
comment and warnings cleanup
* IdentityCacheable
new interface required of objects stored in ObjectIdentityCaches
* IdentityCacheableWrapper
wrapper for storing arbitrary objects in ObjectIdentityCaches
* ObjectIdentityCache
keys now always Strings
values now always IdentityCacheables
new dirtyKey() method to ensure a key is persisted
* ObjectIdentityBdbCache, ObjectIdentityMemCache
update for new ObjectIdentityCache shape; still using legacy GC magic
* ObjectIdentityBdbManualCache
alternate implementation relying on dirtying to ensure persistence
use Guava library MapMaker for soft memMap; capped dirtyMap
* BdbModule
use ObjectIdentityBdbManualCache by default
up expectedConcurrency default to 64
* BdbModuleTest, ObjectIdentityBdbCacheTest, ObjectIdentityBdbManualCacheTest
update, add tests
* Frontier
FrontierGroup as IdentityCacheable
* AbstractFrontier, WorkQueueFrontier, BdbFrontier
touchup queue/group accessors
make queue instances dirty whenever mutated
* WorkQueue, CrawlHost, CrawlServer
IdentityCacheable support; appropriate makeDirty()s
* ServerCache, DefaultServerCache
update for new ObjectIdentityCache shape
addStats() - use putIfAbsent() to avoid race conditions; move
creation of stats map to constructor to avoid another race condition
* WARCWriter.java
switch back to thread-unsafe data structures for stats since object is used
in one thread at a time according to javadoc
processing is finished.
* Recorder.java
getReplayCharSequence() - remember ReplayCharSequence, obtain new one if
none remembered or remembered has been closed
endReplays() - close replayCharSequence
* ToeThread.java
run() - call Recorder.endReplays() when finished with uri
* ReplayCharSequence.java
isOpen() - return false if close() has been called
* GenericReplayCharSequence.java, InMemoryReplayCharSequence.java
implement isOpen()
* KeyWordProcessor.java, ExtractorHTML.java, HTTPContentDigest.java,
ExtractorCSS.java, ExtractorXML.java
do not close ReplayCharSequence
* Arc2Warc, Warc2Arc
adapt to new constructors, settings-object
* MiserOutputStream
stream to monitor position, optionally suppress flushes
* WriterPoolMember
use MiserOutputStream rather than special file-access to find compressed offsets
use shared settings object rather than copying multiple values
* WriterPoolSettings
add settings for frequentFlushes and writeBufferSize for IO optimization
* ARCReader, ARCWriter, ARCWriterPool
adapt to new constructors, settings-object
* WriterPoolSettingsData
impl of WriterPoolSettings for testing and adhoc use
* WARCWriter
adapt to new constructors, settings-object
remove checkSize rollover to new file (done elsewhere)
remove extraneous id-generator code/static-method
* WARCWriterPool
adapt to new constructors, settings-object
* WARCWriterPoolSettings, WARCWriterPoolSettingsData
add recordIDGenerator setting, impl class for testing/adhoc use
* WARCWriterTest
adapt to new constructors, settings-object
* Generator, GeneratorFactory
removed in favor of RecordIDGenerator interface
* RecordIDGenerator
common interface for classes that can provide WARC record IDs
* UUIDGenerator, UUIDGeneratorTest
adapt to derive from RecordIDGenerator
avoid throwing exceptions
* ARCWriterPoolTest, ARCWriterTest
adapt to new constructors, settings-object
* WARCWriterProcessor
do checkSize file-rollover here, so related records don't span WARCs
calculate processor's totalBytesWritten via position offset changes (as before stats-additions)
delegate record-ID generation
* WriterPoolProcessor
add frequentFlushes and writeBufferSize from new WARCWriterPoolSettings interface
* WriterPoolMember.java
write(*), copyFrom() - return number of (uncompressed) bytes written, for
convenience
* WARCWriter.java
tally by record type: number of records written, content bytes, total
bytes, and (possibly compressed) size on disk
* WARCWriterProcessor.java
keep totals and write report to processors-report.txt
* KryoBinding.java
always default to registrationOptional, for now
* StatisticsTracker.java
explicitly set valueClass of sourceDistribution to ConcurrentHashMap
* StoredQueue.java
answer 0 for size() when backing DB closed
* BdbModule
store lengths in JDB manifest; limit extension of last JDB on recover
* CrawlerJournal
rotate frontier-recover on checkpoints like other logs
* RecordingInputStream.java
check for interrupt on each socket-timeout
* CrawlController.java
on second requestCrawlStop, interrupt threads via ToePool.cleanup
* ToePool.java
adjust for new earlier/repeated cleanup
* ToeThread.java
better warnings/recovery on forced-interrupts
* UriProcessingFormatter.java
allow pre-caching of a specific LogRecord's formatted version, outside the synchronized publish()
* GenerationFileHandler.java
force a format before publish(), to benefit above (a small hit in other cases where it's redundant)
* UURI
return to custom-serialization rather than externalization
* CachedBdbMap
remove obsolete class
* StripSessionCFIDs, StripSessionIDs
update patterns to restore intended case-insensitivity
* StripSessionCFIDsTest
put expected, actual in expected order
* AutoKryo
extension of Kryo to allow classes to control their own registration, trigger registration of associated classes, and deserialize classes without no-arg constructors
* KryoBinding
binding for use with BDB that uses AutoKryo serialization for a 2X-4X reduction in byte[] size
* UURI
improved serialization via Externalizable and Kryo's CustomSerialization methods
* BdbModule
discard deprecated CachedBDBMap option
(getObjectCache) extend with both declaredClass and valueClass (for when map values are specializations of the declared type, as with frontier.allQueues)
adjust type declarations
* ObjectIdentityBdbCache
use KryoBinding rather than SerialBinding
* CachedBdbMapTest
discarded
* BdbFrontier, BdbServerCache, StatisticsTracker
adjust type declarations, objectCache creation
* BdbMultipleWorkQueues
use KryoBinding rather than (Recycling)SerialBinding
* BdbWorkQueue, CrawlServer, CrawlHost, CrawlURI
add autoregister support
* LinkContext
public for kryo registration
* UURIFactory, canonicalize/**, PathologicalPathDecideRule, ExtractorHTML
use recycled matchers via TextUtils
* CrawlURI, KeyedProperties, OverlayContext, SheetOverlaysManager
use ArrayList and indexed access rather than LinkedList and iterator instances
* Histotable.java
expose static method useful for frequency-reporting
* TopNSet.java, TopNSet.java
improve handling of case where updates may decrease frequency counts
add convenience methods for common needs
* WorkQueueFrontier.java
replace single longestActiveQueue with largestQueues TopNSet, of configurable size
make max-queues-to-report-per-category configurable
gojomo