* ExtractorHTML.java
smaller caps on element-name, attribute-name, attribute-value lengths -- costly match-failures on bad/non-HTML may be eating a lot of cycles
* ReplayCharSequence.java, GenericReplayCharSequence.java, InMemoryReplayCharSequence.java
store actual first exception for reference
* ExtractorHTML.java
report exception in nonfatal-log, rather than WARNING (which was still polluting job log more than desired -- but should still be logged to allow future investigation of sites where encoding issues may exist)
* ReplayCharSequence.java
new interface method: getDecoderExceptionCount(), reporting >0 if any decoding problems were encountered and silently replaced
* GenericReplayCharSequence.java
count decoding problems, and only log as FINE rather than WARNING
* InMemoryReplayCharSequence.java
try decoding with reporting; if any problems note at least one then decode without reporting errors
* ExtractorHTML.java
after using ReplayCharSequence, log WARNING if nonzero decoding errors
* GenericReplayCharSequence.java
* Scoper.java
* FetchFTP.java
* FetchHTTP.java
demote a bunch of INFO logging to FINE so that be default, console isn't filled with output for every URI handled by a busy crawl
* UriUtils.java
move likely-URI heuristics here for better consistency, reuse, testing, improvement
* UriUtilsTest.java
testing for UriUtils
* ExtractorJS.java
move likely-URI support to UriUtils; use new test
* ExtractoSWF.java
use new UriUtils likely-URI testing
* ExtractorHTML.java
move likely-URI testing to UriUtils, use new test
* JerichoExtractorHTML.java
use new UriUtils likely-URI testing
* CrawlServer.java
enable a specific kind of connect-lost (no response given) to be counted as 'deemed 404' on robots retries over a minimum
* FetchStatusCodes.java
add a synthetic code: -404, S_DEEMED_NOT_FOUND
* FetchHTTP.java
better distinguish CONNECT_FAILED and CONNECT_LOST
* DispositionProcessor.java
treat CONNECT_LOST same as _FAILED; simplify a test
* PersistProcessor.java
rename setupEnvironment to setupCopyEnvironment to make limited purpose of env clear; catch and rethrow IllegalArgumentException with hint that second-open may be cause
* PersistLoadProcessor.java
catch and display IllegalArgumentException same as other failure-to-open SEVERE but proceedable exceptions
* PrecedenceLoader.java
update method names
* EnhancedEnvironment.java
in getClassCatalog(), when read-only env, insist on read-only catalog
* PersistLoadProcessor.java
remove existing curi members from prior to prevent clobbering source
* PersistProcessor.java
set historyDbConfig read-only, set je.log.faultReadSize for
compatability with BDBModule.setup()
* **/pom.xml
bump version identifiers to "3.1.1-SNAPSHOT" (odd minor number meaning dev version, odd patch number indicating expected next patch number of 3.0.x release if before a 3.2.0 release)
* WriterPoolProcessor.java
change description of WriterPoolProcessor.directory ConfigPath to not imply as strongly that is where writing much be occurring
* UURIFactoryTest.java
specify stirng literal with escape-encoding, rather than as raw UTF-8
* **/pom.xml
include 'project.build.sourceEncoding' property to suppress platform-encoding warnings
(unfortunately, neither this nor similar 'maven.compile.encoding' property actually casues maven build to interpret Java source as UTF-8)
offer html
* XmlMarshaller.java
new class to write xml based on nested map structure
* BeanBrowseResource.java ScriptResource.java EngineResource.java
JobResource.java JobRelatedResource.java
support xml representation
* WorkQueueFrontier.java PrecedenceProvider.java WorkQueue.java ToeThread.java
ToePool.java CrawlJob.java Engine.java Frontier.java CrawlController.java
StatisticsTracker.java CrawledBytesHistotable.java FetchStats.java
ProcessorChain.java CrawlURI.java Reporter.java
offer same data that is returned in the single line reports in a map
suitable for use with XmlMarshaller
* FileUtils.java
new method tryToCanonicalize(File)
* AbstractFrontier.java, WorkQueueFrontier.java, BdbFrontier.java
add 'futureUris' map of CrawlURIs to be reenqueued at specific future dates
the usual findEligibleUri also checks for newly-eligible rescheduled URIs
the frontier is no longer empty unless the future count is also 0
* CrawlURI.java
add rescheduleTime property
* ReschedulingProcessor.java
optional post-processor to set a CrawlURI's reschedulingTime
* CrawlStatSnapshot.java, CrawlJob.java
add futureUriCount to stats/uriTotalsReport
* PersistLogProcessor.java
rotate logs on checkpoint
* CrawlerJournal.java, FrontierJournal.java
rotate for checkpoint support without full Checkpointable
processor-related bean checkpoint support:
* Processor.java
implement Checkpointable and BeanNameAware; methods to support JSON state-saving
* DispositionChain.java
implement Checkpointable and during-checkpoint disposition-lockout for state stability
processor-related bean checkpoint support:
* Processor.java
implement Checkpointable and BeanNameAware; methods to support JSON state-saving
* DispositionChain.java
implement Checkpointable and during-checkpoint disposition-lockout for state stability
* **Extractor**.java
pull common reporting & checkpointing up to Extractor.java
* WriterPoolProcessor.java
implement Checkpointable; freeze and roll writer-pool during checkpoint
* PersistLogProcessor.java
roll log on checkpoint
* TextSeedModule.java, SeedModule.java
eliminate obsolete (and probably unnecessary) checkpoint support
* (others)
adapt to above changes
* IoUtils.java
remove class that confusingly-overlaps with commons IOUtils; move remaining methods to ArchiveUtils or archive's FileUtils
* ArchiveUtils.java, FileUtils.java
receive relocated methods; eliminated deprecated or unused methods
* JSONUtils.java
new class to collect common JSON actions
* Iteratorable.java
wrap Iterator as Iterable for foreach usage
* (many)
update to use alternate utils methods
* ServerCache.java
make abstract, implement getHostFor(UURI) and getServerFor(UURI)
* DefaultServerCache.java
extend ServerCache, remove getHostFor(UURI) and getServerFor(UURI)
* ServerCacheUtil.java
removed
* other classes
change usage of ServerCacheUtil to ServerCache
* DefaultServerCache.java
getHostFor(UURI): special case dns uris
* ServerCache.java
add getHostFor(UURI) to public interface
* ServerCacheUtil.java
remove now redundant wrapper method
* DispositionProcessor.java AbstractFrontier.java IPQueueAssignmentPolicy.java
BucketQueueAssignmentPolicy.java PreconditionEnforcer.java QuotaEnforcer.java
StatisticsTracker.java ServerCacheTest.java FetchHTTP.java
Kw3WriterProcessor.java WriterPoolProcessor.java
changed to use serverCache.getHostFor(UURI) (a few cases remain that use
getHostFor(String))
* ScopePlusOneDecideRule.java
removing, in favor of SurtPrefixedDecideRule's 'alsoCheckVia' option
* RecyclingFastBufferedOutputStream.java
use library by deriving as subclass (rather than c&p), now that class is available in an included library (fastutil)
* BdbModule.java
allow Database instance reuse if BdbConfig is identical instance
* PersistProcessor.java
convert shared BdbConfig for history db to static constant HISTORY_DB_CONFIG
* PersistOnlineProcessor.java
use HISTORY_DB_CONFIG, don't close possibly-shared DB on stop()
* PrecedenceLoader.java, PreloadedUriPrecedencePolicy.java
use HISTORY_DB_CONFIG
gojomo