* WARCWriterProcessor
move checkSize into block protected by catch/finally cleanup of errors
* WARCWriterProcessorTest
test to simulate record-write-failure and verify does not cause pool lockup
* WARCWriterTest, WriterPoolMember
comment and warnings cleanup
* Robotstxt, RobotstxtTest
treat wildcardRules specially, so that it is possible to ask for only explicitly-named matches
* MostFavoredRobotsPolicy
improve comment, change masquerade default to true
* FirstNamedRobotsPolicy, FirstNamedRobotsPolicyTest
policy to consider alternate user-agents in order, using the first that has any explicit (non-wildcard) rules declared (or failing that, the original default user-agent)
* TooManyHopsDecideRule
remove inadvertent reversal of sense of comparison, making default scope reject almost everything (and breaking most self-tests)
* StatisticsTracker
move small stat maps to in-memory ConcurrentMaps
update processedSeedRecords for new ObjectIdentityCache shape
move hostsDistribution/hostsBytes/hostsLastFinished tracking to CrawlHosts/hostsCache
* CrawlSummaryReport
gets hosts count from hostsCache
* SeedRecord
implement IdentityCacheable
* FetchStats
remember last-success times
* IdentityCacheable
new interface required of objects stored in ObjectIdentityCaches
* IdentityCacheableWrapper
wrapper for storing arbitrary objects in ObjectIdentityCaches
* ObjectIdentityCache
keys now always Strings
values now always IdentityCacheables
new dirtyKey() method to ensure a key is persisted
* ObjectIdentityBdbCache, ObjectIdentityMemCache
update for new ObjectIdentityCache shape; still using legacy GC magic
* ObjectIdentityBdbManualCache
alternate implementation relying on dirtying to ensure persistence
use Guava library MapMaker for soft memMap; capped dirtyMap
* BdbModule
use ObjectIdentityBdbManualCache by default
up expectedConcurrency default to 64
* BdbModuleTest, ObjectIdentityBdbCacheTest, ObjectIdentityBdbManualCacheTest
update, add tests
* Frontier
FrontierGroup as IdentityCacheable
* AbstractFrontier, WorkQueueFrontier, BdbFrontier
touchup queue/group accessors
make queue instances dirty whenever mutated
* WorkQueue, CrawlHost, CrawlServer
IdentityCacheable support; appropriate makeDirty()s
* ServerCache, DefaultServerCache
update for new ObjectIdentityCache shape
* CrawlURI
MAX_HOPS_DISPLAYED constant, 50
build new CrawlURI pathFromSeed value via extendHopsPath, which shows only MAX_HOPS_DISPLAYED hops, and precedes string with int value of hops unshown
* CrawlURITest
test for extendHopsPath
* ExtractorJS
apply JS literal string deescaping before interpretation as link
* ExtractorJSTest
3 basic tests, including one for this issue
* Link
provide logging and fallback for case where via may be null; risk identified in creation of test cases could present again if a '.js' file is ever provided as a (via-less) seed
addStats() - use putIfAbsent() to avoid race conditions; move
creation of stats map to constructor to avoid another race condition
* WARCWriter.java
switch back to thread-unsafe data structures for stats since object is used
in one thread at a time according to javadoc
processing is finished.
* Recorder.java
getReplayCharSequence() - remember ReplayCharSequence, obtain new one if
none remembered or remembered has been closed
endReplays() - close replayCharSequence
* ToeThread.java
run() - call Recorder.endReplays() when finished with uri
* ReplayCharSequence.java
isOpen() - return false if close() has been called
* GenericReplayCharSequence.java, InMemoryReplayCharSequence.java
implement isOpen()
* KeyWordProcessor.java, ExtractorHTML.java, HTTPContentDigest.java,
ExtractorCSS.java, ExtractorXML.java
do not close ReplayCharSequence
* Arc2Warc, Warc2Arc
adapt to new constructors, settings-object
* MiserOutputStream
stream to monitor position, optionally suppress flushes
* WriterPoolMember
use MiserOutputStream rather than special file-access to find compressed offsets
use shared settings object rather than copying multiple values
* WriterPoolSettings
add settings for frequentFlushes and writeBufferSize for IO optimization
* ARCReader, ARCWriter, ARCWriterPool
adapt to new constructors, settings-object
* WriterPoolSettingsData
impl of WriterPoolSettings for testing and adhoc use
* WARCWriter
adapt to new constructors, settings-object
remove checkSize rollover to new file (done elsewhere)
remove extraneous id-generator code/static-method
* WARCWriterPool
adapt to new constructors, settings-object
* WARCWriterPoolSettings, WARCWriterPoolSettingsData
add recordIDGenerator setting, impl class for testing/adhoc use
* WARCWriterTest
adapt to new constructors, settings-object
* Generator, GeneratorFactory
removed in favor of RecordIDGenerator interface
* RecordIDGenerator
common interface for classes that can provide WARC record IDs
* UUIDGenerator, UUIDGeneratorTest
adapt to derive from RecordIDGenerator
avoid throwing exceptions
* ARCWriterPoolTest, ARCWriterTest
adapt to new constructors, settings-object
* WARCWriterProcessor
do checkSize file-rollover here, so related records don't span WARCs
calculate processor's totalBytesWritten via position offset changes (as before stats-additions)
delegate record-ID generation
* WriterPoolProcessor
add frequentFlushes and writeBufferSize from new WARCWriterPoolSettings interface
* WriterPoolMember.java
write(*), copyFrom() - return number of (uncompressed) bytes written, for
convenience
* WARCWriter.java
tally by record type: number of records written, content bytes, total
bytes, and (possibly compressed) size on disk
* WARCWriterProcessor.java
keep totals and write report to processors-report.txt
new property textSource, an org.archive.io.ReadSource, allowing
specification inline in cxml as well as in external file
* profile-crawler-beans.cxml
commented out clauses demonstrating use of textSource
* DecideRuleSequence.java
- use Spring Lifecycle start() to initialize logToFile
- inject logger module as SimpleFileLoggerProvider, since we're in
heritrix-modules and don't have access to CrawlerLoggerModule from
heritrix-engine
* CrawlerLoggerModule.java
- implement SimpleFileLoggerProvider
- setupSimpleLog() - use 'T' instead of '+' between date and time in
timestamp
* SimpleFileLoggerProvider.java
new interface with one method, setupSimpleLog()
* Scoper.java
do not call scope.start()/scope.stop(), these are handled using Spring
Lifecycle now
* DecideRule.java
remove unused, unneeded start()/stop()
* (many)
distinction between these two constant-collecting classes was fuzzy (to the point that they already referred to each other), and they were both in same subproject/package already as well; so, merged constants into the larger, older class
* UURI
return to custom-serialization rather than externalization
* CachedBdbMap
remove obsolete class
* StripSessionCFIDs, StripSessionIDs
update patterns to restore intended case-insensitivity
* StripSessionCFIDsTest
put expected, actual in expected order
* AutoKryo
extension of Kryo to allow classes to control their own registration, trigger registration of associated classes, and deserialize classes without no-arg constructors
* KryoBinding
binding for use with BDB that uses AutoKryo serialization for a 2X-4X reduction in byte[] size
* UURI
improved serialization via Externalizable and Kryo's CustomSerialization methods
* BdbModule
discard deprecated CachedBDBMap option
(getObjectCache) extend with both declaredClass and valueClass (for when map values are specializations of the declared type, as with frontier.allQueues)
adjust type declarations
* ObjectIdentityBdbCache
use KryoBinding rather than SerialBinding
* CachedBdbMapTest
discarded
* BdbFrontier, BdbServerCache, StatisticsTracker
adjust type declarations, objectCache creation
* BdbMultipleWorkQueues
use KryoBinding rather than (Recycling)SerialBinding
* BdbWorkQueue, CrawlServer, CrawlHost, CrawlURI
add autoregister support
* LinkContext
public for kryo registration
* UURIFactory, canonicalize/**, PathologicalPathDecideRule, ExtractorHTML
use recycled matchers via TextUtils
* CrawlURI, KeyedProperties, OverlayContext, SheetOverlaysManager
use ArrayList and indexed access rather than LinkedList and iterator instances
setting "independentExtractors" - when enabled, extractors run regardless of
whether other extractors have run
* profile-crawler-beans.cxml, AbstractFrontier.java, ExtractorParameters.java,
Extractor.java
new setting independentExtractors
* ContentExtractor.java
shouldProcess() - respect independentExtractors
* RobotsPolicy.java
simplified unification of RobotsHonoring and RobotsExclusion policy classes; subclasses specialize
* ObeyRobotsPolicy.java
classic obey-robots-as-declared policy
* IgnoreRobotsPolicy.java
ignore declared robots policy
* CustomRobotsPolicy.java
follow custom-crafted policy in robots.txt format
* MostFavoredRobotsPolicy.java
follow the most-permissive policy available to a set of user-agents; optionally adopt ('masquerade') the user-agent whose rules are chosen
* RobotsExclusionPolicy.java, RobotsHonoringPolicy.java
deleted
* CrawlMetadata.java
use new string robotsPolicyName as main determinant of policy
maintain map of available policies ('obey','ignore', and any the operator added)
* CrawlServer.java
retain Robotstxt instance rather than customized RobotsExclusionPolicy
* PreconditionEnforcer.java
use full CrawlMetadata rather than just UserAgentProvider subset
consult RobotsPolicy looked up through CrawlMetadata for allow/disallow decision
* DispositionProcessor.java
update CrawlServer with Robotstxt only
consult Robotstxt for crawl-delay
* profile-crawler-beans.cxml (*2), migrate-template-crawler-beans.cxml, selftest-crawler-beans.cxml
remove old declarations/examples; add new
* ExtractorHTML.java, JerichoExtractorHTML.java
consult RobotsPolicy.obeyMetaRobotsNofollow for NOFOLLOW
* Robotstxt.java
new constructors for new use patterns
* **/*Test.java
update/remove to match new use patterns
* Frontier.java
add beginDisposition(), endDisposition() methods
* AbstractFrontier.java
implement beginDisposition(), endDisposition() methods to maintain dispositions-in-progress lock
* BdbFrontier.java
acquire exclusive dispositionInProgressLock before allowing checkpoint to begin; release at completion of checkpoint
* ToeThread.java
call beginDisposition before dispositionChain/finished(); call endDisposition after finished() and on exceptions
* DispositionChain.java
shed all prior relation to locking/checkpointing
gojomo