(specifically, when they're in the query string)
* UURIFactory.java
fixup() - replace \ with / up to the first ? in the url, instead of
anywhere in the url
and update comments
* UURIFactoryTest.java
backslash test
* RobotsDirectives.java
- use plain ConcurrentSkipListSet<String> instead of PrefixSet, to
maintain complete list of prefixes, so we can know the longest match
- allows(String) - return true if longest matching Allow prefix is longer
than or equal to longest matching Disallow prefix
* RobotstxtTest.java
flip expected result of test of generic Allow against specific Disallow
* BdbModule.java
new setting, useHardLinkCheckpoints, default true
doCheckpoint now creates hard-links to active log files from the in-environment checkpoint-dir
doRecover now relinks from checkpoint-dir files, when available, in preference to whatever is already in the env-dir
* pom.xml
bring in JNA for native calls
* CLibrary
wrapper for the link() function
* GZIPMembersInputStream
override OpenJDK7GZIPInputStream.readTrailer to not rely on available() as indicator of end-of-stream
fix membersIterator hasNext() to also not rely on available() as indicator of end-of-stream
* ArchiveReader
change innerHasNext to peek one byte (using mark/reset) rather than rely on available() as indicator of end-of-stream
remove redundant getInputStream accessor in favor of field-name-matching getIn/setIn
* ArchiveReaderFactory, (W)ARCReaderFactory
remove no-longer-necessary RepositionableStream references
use getIn() rather than getInputStream()
[HER-728] Offer replay stream that has been un-chunked (whether because response was HTTP/1.1 or used chunked in HTTP/1.0 against spec)
[HER-1876] Offer HTTP/1.1 option - for chunked transfer-encoding (but not persistent connections)
* (all)
update to use new (decoded-as-necessary) content streams/CharSequences
[HER-728] Offer replay stream that has been un-chunked (whether because response was HTTP/1.1 or used chunked in HTTP/1.0 against spec)
[HER-1876] Offer HTTP/1.1 option - for chunked transfer-encoding (but not persistent connections)
* FetchHTTP
add 'acceptCompression' and 'useHTTP11' properties, both default false
* Recorder
track whether recorded-input is transfer-encoded (chunked) or content-encoded (gzip etc)
offer alternate replay streams for
(1) raw 'messageBody';
(2) entity (un-chunked if necessary)
(3) content (decompressed if necessary)
always content & GenericReplayCharSequence for CharSequence replays
* GenericReplayCharSequence
always use a stream (rather than random-access buffer)
always decode to prefix buffer first, so short content never touches disk no matter the encoding
* InMemoryReplayCharSequence
deleted; 'Generic' now works similarly for small content and anyway random-access for single-byte-encodings is now rarely possible (given deconding streams)
* RecordingInputStream, RecordingOutputStream
adjust for changed stream names, functionality moved to Recorder
* ReplayCharSequence
use Charset instances rather than names
* ReplayInputStream
add convenience constructor (and tmp-file-destroy) for copying any other inputStream into a seekable ReplayInputStream
Invalid property 'timeoutSeconds' of bean class
[org.archive.modules.fetcher.FetchFTP]: Bean property 'timeoutSeconds' is not
writable or has an invalid setter method. Does the parameter type of the setter
match the return type of the getter?"
* FetchFTP.java
change setTimeoutSeconds(Integer) to setTimeoutSeconds(int)
* FetchFTP.java
setSoTimeoutMs(), getSoTimeoutMs()
(init) - set default value of 20000 ms, same as FetchHTTP
fetch() - set various timeouts on FTPClient - effectively sets connect
timeout and socket timeout on control connection, and socket timeout on
data connection (notably, does not set connect timeout on data connection
w/ current commons-net)
* OpenJDK7GZIPInputStream.java, OpenJDK7InflatedInputStream.java
backport the working GZIP implementation from OpenJDK7, with minimal
renames/imports/private-to-protected changes to enable member-at-a-time reading
* GZIPMembersInputStream
base on OpenJDK7 implementation to avoid pre- & post-6u23 codepaths, and 6u23/24 readHeader bug
override readTrailer to set member-end; update comment to reflect end-uncertainty without EOF-per-member-mode
* GZIPMembersInputStreamTest, ARCWriterTest
test tweaks
* OpenJDK7GZIPInputStream.java, OpenJDK7InflatedInputStream.java
backport the working GZIP implementation from OpenJDK7, with minimal
renames/imports/private-to-protected changes to enable member-at-a-time reading
* GZIPMembersInputStream
base on OpenJDK7 implementation to avoid pre- & post-6u23 codepaths, and 6u23/24 readHeader bug
override readTrailer to set member-end; update comment to reflect end-uncertainty without EOF-per-member-mode
* GZIPMembersInputStreamTest, ARCWriterTest
test tweaks
marshalDocument() - SAXException can wrap other exceptions, e.g.
EOFException when tcp connection is broken; thus, in case of IOException,
rather than throwing a RuntimeException, re-throw the wrapped IOException
(unfortunately in that case we still get a massive stack trace in the log,
but at least it comes from the appropriate spot and is not too misleading)
that or preloadSource (a filesystem path), rather than try to handle both
possibilities with one string.
* PersistLoadProcessor.java
* PersistProcessor.java
(regression from h1)
* PersistLoadProcessor.java
change preloadSource from ConfigFile to String, allowing the logic in
PersistProcessor.copyPersistSourceToHistoryMap() to decide how to load it
ensure mark()ing more frequently, when reset to before current position won't be needed
* ArchiveReaderFactory
avoid misinterpreting a "c:" local path as a URI scheme
better logging of caught/recovery-attempted IOException
* GZIPMemberInputStream
handling of previously-unrecognized case in pre-JDK6u23: attempt at read() exhausts inflater with no new bytes available (previous nonzero-length read had not indicated inflater finished)
* URIAuthorityBasedQueueAssignmentPolicy.java
whois urls all go in special "whois" queue
* PreconditionEnforcer.java, CrawlURI.java
move markPrerequisite() method from PreconditionEnforcer to CrawlURI
* profile-crawler-beans.cxml
commented-out whois fetcher clause
* FetchStatusCodes.java
new status codes for whois
* ServerCache.java
getHostFor() - return "whois:" for whois urls, following dns: convention
* UURIFactory.java
ugly hack for whois urls
* SchemeNotInSetDecideRule.java
add whois to list of supported schemes
* WriterPoolProcessor.java
shouldWrite() - we should write successful whois fetches
* WARCWriterProcessor.java
write whois records
* FetchWhois.java
the fetcher
* ArchiveReader
improve warning message
* WriterPoolMember
attempt pre-delete of target filename, warn on failure, for better odds on some filesystems
* ARCWriterTest, WARCWriterTest
ensure timely close of readers/writers in case filenames are reused
restore originally-intended unique target names
* CookieSpecBase
use Guava InternetDomainName public-suffixes to both (1) prevent storing cookie on public-suffix; (2) prevent looking-up cookies for public-suffixes
* BdbCookieStorage
correct checkpoint/relaunch behavior: don't reuse persisted cookies on normal relaunches; do reuse on resume-from-checkpoint
* GZIPMembersInputStream
new workaround class that offers choice of pre/post JDK6u23 behavior and new accessors for finding member boundary offsets
* GZIPMembersInputStreamTest
test for above
* ArchiveUtils, ARCWriter
move static compress utilities to ArchiveUtils
* ArchiveReader, (W)ARCReaderFactory, (W)ARCWriterTest
adapt to use GZIPMembersInputStream
* GzippedInputStream(Test)
deleted
use heritrix.home, if available, to find default logging.properties (to let those who launch Heritrix from elsewhere save themselves from HttpClient's copious logging)
nlevitt