* DecideRuleSequence.java
- use Spring Lifecycle start() to initialize logToFile
- inject logger module as SimpleFileLoggerProvider, since we're in
heritrix-modules and don't have access to CrawlerLoggerModule from
heritrix-engine
* CrawlerLoggerModule.java
- implement SimpleFileLoggerProvider
- setupSimpleLog() - use 'T' instead of '+' between date and time in
timestamp
* SimpleFileLoggerProvider.java
new interface with one method, setupSimpleLog()
* Scoper.java
do not call scope.start()/scope.stop(), these are handled using Spring
Lifecycle now
* DecideRule.java
remove unused, unneeded start()/stop()
* KryoBinding.java
always default to registrationOptional, for now
* StatisticsTracker.java
explicitly set valueClass of sourceDistribution to ConcurrentHashMap
* StoredQueue.java
answer 0 for size() when backing DB closed
* CheckpointService
Made checkpointIntervalMinutes setter accept a long instead of an int (was missed when the class variable was made a long). While it was non-harmful the way it was, this is more in line with Java conventions.
* AbstractFrontier
remove inbound/outbound queues; simplify managerThread
perform findEligible/schedule/receive/finish immediately
* WorkQueueFrontier
eliminate holdQueues setting
synchronize sendToQueue on target queue
make findEligibleUri reentrant; rely on [Blocking|Stored]Queue thread-safety
make arriving at front of ready trigger for 'active'/budget-session
simplify inactiveQueues management
simplify waking from overflow
* WorkQueue
redefine 'active' as in-budget-session
rename 'held' as 'managed'
synchronize major methods, relying on allQueues cache that operations on same intended queue use identical instance
split over-budget test to isOverSessionBudget and isOverTotalBudget
have toString show classKey
* BdbFrontier
checkpoint fixes: remember nextOrdinal, rotate frontier-recover log, reset queues on recovery
consistencyCheck method for probing state during stress tests
* RecordingInputStream.java
check for interrupt on each socket-timeout
* CrawlController.java
on second requestCrawlStop, interrupt threads via ToePool.cleanup
* ToePool.java
adjust for new earlier/repeated cleanup
* ToeThread.java
better warnings/recovery on forced-interrupts
* UriProcessingFormatter.java
allow pre-caching of a specific LogRecord's formatted version, outside the synchronized publish()
* GenerationFileHandler.java
force a format before publish(), to benefit above (a small hit in other cases where it's redundant)
* (many)
distinction between these two constant-collecting classes was fuzzy (to the point that they already referred to each other), and they were both in same subproject/package already as well; so, merged constants into the larger, older class
* WorkQueueFrontier
(findEligibleURI) avoid outbound.capacity-sensitive activation, which under a race created by recent changes led to infinite recursion here
* AbstractFrontier
(next) when nothing is immediately ready, try adding one-at-a-time to outbound, rather than fillOutbound()
* AbstractFrontier
(drainInbound) don't assume current thread will be able to take() full batch; poll() and exit early if other threads have drained queue first
* AbstractFrontier, WorkQueueFrontier
synchronize around all InEvent/findEligibleURI actions, so that drainInbound and fillOutbound may be called outside managerThread
whenever a ToeThread would block on enqueue() or next(), try the appropriate catch-up method before blocking
remove assertions no longer true with activity happening outside managerThread
* AutoKryo
extension of Kryo to allow classes to control their own registration, trigger registration of associated classes, and deserialize classes without no-arg constructors
* KryoBinding
binding for use with BDB that uses AutoKryo serialization for a 2X-4X reduction in byte[] size
* UURI
improved serialization via Externalizable and Kryo's CustomSerialization methods
* BdbModule
discard deprecated CachedBDBMap option
(getObjectCache) extend with both declaredClass and valueClass (for when map values are specializations of the declared type, as with frontier.allQueues)
adjust type declarations
* ObjectIdentityBdbCache
use KryoBinding rather than SerialBinding
* CachedBdbMapTest
discarded
* BdbFrontier, BdbServerCache, StatisticsTracker
adjust type declarations, objectCache creation
* BdbMultipleWorkQueues
use KryoBinding rather than (Recycling)SerialBinding
* BdbWorkQueue, CrawlServer, CrawlHost, CrawlURI
add autoregister support
* LinkContext
public for kryo registration
* UURIFactory, canonicalize/**, PathologicalPathDecideRule, ExtractorHTML
use recycled matchers via TextUtils
* CrawlURI, KeyedProperties, OverlayContext, SheetOverlaysManager
use ArrayList and indexed access rather than LinkedList and iterator instances
setting "independentExtractors" - when enabled, extractors run regardless of
whether other extractors have run
* profile-crawler-beans.cxml, AbstractFrontier.java, ExtractorParameters.java,
Extractor.java
new setting independentExtractors
* ContentExtractor.java
shouldProcess() - respect independentExtractors
* Histotable.java
expose static method useful for frequency-reporting
* TopNSet.java, TopNSet.java
improve handling of case where updates may decrease frequency counts
add convenience methods for common needs
* WorkQueueFrontier.java
replace single longestActiveQueue with largestQueues TopNSet, of configurable size
make max-queues-to-report-per-category configurable
* JobRelatedResource.java
add more property names to blacklist
route all relevant requests through new utilitiy getPropertyDescriptors method that patches instances as necessary to suppress problems
* RobotsPolicy.java
simplified unification of RobotsHonoring and RobotsExclusion policy classes; subclasses specialize
* ObeyRobotsPolicy.java
classic obey-robots-as-declared policy
* IgnoreRobotsPolicy.java
ignore declared robots policy
* CustomRobotsPolicy.java
follow custom-crafted policy in robots.txt format
* MostFavoredRobotsPolicy.java
follow the most-permissive policy available to a set of user-agents; optionally adopt ('masquerade') the user-agent whose rules are chosen
* RobotsExclusionPolicy.java, RobotsHonoringPolicy.java
deleted
* CrawlMetadata.java
use new string robotsPolicyName as main determinant of policy
maintain map of available policies ('obey','ignore', and any the operator added)
* CrawlServer.java
retain Robotstxt instance rather than customized RobotsExclusionPolicy
* PreconditionEnforcer.java
use full CrawlMetadata rather than just UserAgentProvider subset
consult RobotsPolicy looked up through CrawlMetadata for allow/disallow decision
* DispositionProcessor.java
update CrawlServer with Robotstxt only
consult Robotstxt for crawl-delay
* profile-crawler-beans.cxml (*2), migrate-template-crawler-beans.cxml, selftest-crawler-beans.cxml
remove old declarations/examples; add new
* ExtractorHTML.java, JerichoExtractorHTML.java
consult RobotsPolicy.obeyMetaRobotsNofollow for NOFOLLOW
* Robotstxt.java
new constructors for new use patterns
* **/*Test.java
update/remove to match new use patterns
* Frontier.java
add beginDisposition(), endDisposition() methods
* AbstractFrontier.java
implement beginDisposition(), endDisposition() methods to maintain dispositions-in-progress lock
* BdbFrontier.java
acquire exclusive dispositionInProgressLock before allowing checkpoint to begin; release at completion of checkpoint
* ToeThread.java
call beginDisposition before dispositionChain/finished(); call endDisposition after finished() and on exceptions
* DispositionChain.java
shed all prior relation to locking/checkpointing
* Checkpoint.java
remove previous attempt to use Lifecycle to catch error
* Checkpointable.java
@Autowired(required=false) annotation for setRecoveryCheckpoint
* CheckpointService.java
HasValidator implementation for post-build validity check; also check at launch (probably redundant)
* CheckpointValidator.java
ensure validity stamp exists on any present Checkpoint
* LaxURI.java
increase tolerance with regard to path-segments
* UURIFactory.java
allow digits in scheme regex
discard old supported-schemes code
* UURIFactoryTest.java
tests for roundtrip serialization of problematic hostlike-scheme and bars-in-path
* CrawlerLoggerModule.java
discard special-casing of URI scheme errors
* SchemeNotInSetDecideRule.java
rule to apply to URIs of unknown schemes; by default REJECTs those not usually handled by Heritrix
* **/profile-crawler-beans.cxml
add REJECT SchemeNotInSetDecideRule
* logging.properties
streamline to bare essentials; allow INFO for org.archive classes
* GenericReplayCharSequence.java, BdbUriUniqFilter.java
demote some INFO logging to FINE
nlevitt