Invalid property 'timeoutSeconds' of bean class
[org.archive.modules.fetcher.FetchFTP]: Bean property 'timeoutSeconds' is not
writable or has an invalid setter method. Does the parameter type of the setter
match the return type of the getter?"
* FetchFTP.java
change setTimeoutSeconds(Integer) to setTimeoutSeconds(int)
* FetchFTP.java
setSoTimeoutMs(), getSoTimeoutMs()
(init) - set default value of 20000 ms, same as FetchHTTP
fetch() - set various timeouts on FTPClient - effectively sets connect
timeout and socket timeout on control connection, and socket timeout on
data connection (notably, does not set connect timeout on data connection
w/ current commons-net)
* OpenJDK7GZIPInputStream.java, OpenJDK7InflatedInputStream.java
backport the working GZIP implementation from OpenJDK7, with minimal
renames/imports/private-to-protected changes to enable member-at-a-time reading
* GZIPMembersInputStream
base on OpenJDK7 implementation to avoid pre- & post-6u23 codepaths, and 6u23/24 readHeader bug
override readTrailer to set member-end; update comment to reflect end-uncertainty without EOF-per-member-mode
* GZIPMembersInputStreamTest, ARCWriterTest
test tweaks
* OpenJDK7GZIPInputStream.java, OpenJDK7InflatedInputStream.java
backport the working GZIP implementation from OpenJDK7, with minimal
renames/imports/private-to-protected changes to enable member-at-a-time reading
* GZIPMembersInputStream
base on OpenJDK7 implementation to avoid pre- & post-6u23 codepaths, and 6u23/24 readHeader bug
override readTrailer to set member-end; update comment to reflect end-uncertainty without EOF-per-member-mode
* GZIPMembersInputStreamTest, ARCWriterTest
test tweaks
marshalDocument() - SAXException can wrap other exceptions, e.g.
EOFException when tcp connection is broken; thus, in case of IOException,
rather than throwing a RuntimeException, re-throw the wrapped IOException
(unfortunately in that case we still get a massive stack trace in the log,
but at least it comes from the appropriate spot and is not too misleading)
that or preloadSource (a filesystem path), rather than try to handle both
possibilities with one string.
* PersistLoadProcessor.java
* PersistProcessor.java
(regression from h1)
* PersistLoadProcessor.java
change preloadSource from ConfigFile to String, allowing the logic in
PersistProcessor.copyPersistSourceToHistoryMap() to decide how to load it
ensure mark()ing more frequently, when reset to before current position won't be needed
* ArchiveReaderFactory
avoid misinterpreting a "c:" local path as a URI scheme
better logging of caught/recovery-attempted IOException
* GZIPMemberInputStream
handling of previously-unrecognized case in pre-JDK6u23: attempt at read() exhausts inflater with no new bytes available (previous nonzero-length read had not indicated inflater finished)
* URIAuthorityBasedQueueAssignmentPolicy.java
whois urls all go in special "whois" queue
* PreconditionEnforcer.java, CrawlURI.java
move markPrerequisite() method from PreconditionEnforcer to CrawlURI
* profile-crawler-beans.cxml
commented-out whois fetcher clause
* FetchStatusCodes.java
new status codes for whois
* ServerCache.java
getHostFor() - return "whois:" for whois urls, following dns: convention
* UURIFactory.java
ugly hack for whois urls
* SchemeNotInSetDecideRule.java
add whois to list of supported schemes
* WriterPoolProcessor.java
shouldWrite() - we should write successful whois fetches
* WARCWriterProcessor.java
write whois records
* FetchWhois.java
the fetcher
* ArchiveReader
improve warning message
* WriterPoolMember
attempt pre-delete of target filename, warn on failure, for better odds on some filesystems
* ARCWriterTest, WARCWriterTest
ensure timely close of readers/writers in case filenames are reused
restore originally-intended unique target names
* CookieSpecBase
use Guava InternetDomainName public-suffixes to both (1) prevent storing cookie on public-suffix; (2) prevent looking-up cookies for public-suffixes
* BdbCookieStorage
correct checkpoint/relaunch behavior: don't reuse persisted cookies on normal relaunches; do reuse on resume-from-checkpoint
* GZIPMembersInputStream
new workaround class that offers choice of pre/post JDK6u23 behavior and new accessors for finding member boundary offsets
* GZIPMembersInputStreamTest
test for above
* ArchiveUtils, ARCWriter
move static compress utilities to ArchiveUtils
* ArchiveReader, (W)ARCReaderFactory, (W)ARCWriterTest
adapt to use GZIPMembersInputStream
* GzippedInputStream(Test)
deleted
use heritrix.home, if available, to find default logging.properties (to let those who launch Heritrix from elsewhere save themselves from HttpClient's copious logging)
* WARCWriterProcessor
move checkSize into block protected by catch/finally cleanup of errors
* WARCWriterProcessorTest
test to simulate record-write-failure and verify does not cause pool lockup
* WARCWriterTest, WriterPoolMember
comment and warnings cleanup
* Robotstxt, RobotstxtTest
treat wildcardRules specially, so that it is possible to ask for only explicitly-named matches
* MostFavoredRobotsPolicy
improve comment, change masquerade default to true
* FirstNamedRobotsPolicy, FirstNamedRobotsPolicyTest
policy to consider alternate user-agents in order, using the first that has any explicit (non-wildcard) rules declared (or failing that, the original default user-agent)
* TooManyHopsDecideRule
remove inadvertent reversal of sense of comparison, making default scope reject almost everything (and breaking most self-tests)
* StatisticsTracker
move small stat maps to in-memory ConcurrentMaps
update processedSeedRecords for new ObjectIdentityCache shape
move hostsDistribution/hostsBytes/hostsLastFinished tracking to CrawlHosts/hostsCache
* CrawlSummaryReport
gets hosts count from hostsCache
* SeedRecord
implement IdentityCacheable
* FetchStats
remember last-success times
* IdentityCacheable
new interface required of objects stored in ObjectIdentityCaches
* IdentityCacheableWrapper
wrapper for storing arbitrary objects in ObjectIdentityCaches
* ObjectIdentityCache
keys now always Strings
values now always IdentityCacheables
new dirtyKey() method to ensure a key is persisted
* ObjectIdentityBdbCache, ObjectIdentityMemCache
update for new ObjectIdentityCache shape; still using legacy GC magic
* ObjectIdentityBdbManualCache
alternate implementation relying on dirtying to ensure persistence
use Guava library MapMaker for soft memMap; capped dirtyMap
* BdbModule
use ObjectIdentityBdbManualCache by default
up expectedConcurrency default to 64
* BdbModuleTest, ObjectIdentityBdbCacheTest, ObjectIdentityBdbManualCacheTest
update, add tests
* Frontier
FrontierGroup as IdentityCacheable
* AbstractFrontier, WorkQueueFrontier, BdbFrontier
touchup queue/group accessors
make queue instances dirty whenever mutated
* WorkQueue, CrawlHost, CrawlServer
IdentityCacheable support; appropriate makeDirty()s
* ServerCache, DefaultServerCache
update for new ObjectIdentityCache shape
* CrawlURI
MAX_HOPS_DISPLAYED constant, 50
build new CrawlURI pathFromSeed value via extendHopsPath, which shows only MAX_HOPS_DISPLAYED hops, and precedes string with int value of hops unshown
* CrawlURITest
test for extendHopsPath
* ExtractorJS
apply JS literal string deescaping before interpretation as link
* ExtractorJSTest
3 basic tests, including one for this issue
* Link
provide logging and fallback for case where via may be null; risk identified in creation of test cases could present again if a '.js' file is ever provided as a (via-less) seed
addStats() - use putIfAbsent() to avoid race conditions; move
creation of stats map to constructor to avoid another race condition
* WARCWriter.java
switch back to thread-unsafe data structures for stats since object is used
in one thread at a time according to javadoc
nlevitt