Fix for [HER-1612] lock contention in BeanShellDecideRule/BeanShellProcessor occurs even if isolate-threads=true
* BeanShellDecideRule.java
(getInterpreter) add synchronization
(decisionFor) remove method synchronization; sync on interpreter/getInterpreter makes it a superfluous bottleneck
* BeanShellProcessor.java
(getInterpreter) add synchronization
(innerProcess) remove method synchronization; sync on interpreter/getInterpreter makes it a superfluous bottleneck
Fix for [HER-1550] set-source-tag option nonfunctional
* CrawlURI.java
ensure 'heritable' key is itself heritable; also change heritable collection to a set
Fix for [HER-1520] Windows mapped-files cause intermittent ReplayCharSequenceTest.testReplayCharSequenceByteToStringMulti failures, risk other problems
* GenericReplayCharSequence.java
use more diverse filenames, so subsequent decodes don't run into lingering open mapped sections of previous decodes
do delete via new FileUtils.deleteSoonerOrLater utility
* FileUtils.java
(deleteSoonerOrLater) try immediate delete, but if fails, add to a list of files to be retried later
Fix for [HER-1506] ByteReplayCharSequence ignores specified/non-default character encodings, reported by Jean-No?\195?\171l Rivasseau
* RecordingOutputStream.java
use ByteReplayCharSequence only for iso8859-1
* ReplayCharSequenceTest.java
new tests for iso8859-1, windows-1252 and ascii
Fix for [HER-1533] bottleneck in StatisticsTracker.saveSourceStats
* StatisticsTrackerImpl.java
saveSourceStats() -- call put() on the CachedBdbMap only when entry is
added, not on every change
Fix for [HER-1225] HtmlFormCredential.java may create non-wellformed URLs
* HtmlFormCredential.java
only insert '&' if previous query-string was non-empty
Fix for [HER-1521] Make the UncompressedARCReader and CompressedARCReader in the ARCReaderFactory as public or protected
* ARCReaderFactory.java, WARCReaderFactory.java
make compressed/uncompressed implementation inner classes public to enable direct reuse
Fix for [HER-1558] ${HOSTNAME} in arc suffix is only replaced completely
* WriterPoolProcessor.java
replace ${HOSTNAME} in suffix, rather than test for exactly ${HOSTNAME}
Fix for [#HER-1554] META http-equiv refresh content containing only a number misinterpreted as a URI - IA Webteam JIRA:
* ExtractorHTML.java
don't use full 'content' as META refresh URL in absence of '='
Fix for [HER-1499] crawl.log has no annotation indicating that WARCWriterProcessor wrote a revisit record
* WARCWriterProcessor.java
add annotations "warcRevisit:digest" or "warcRevisit:notModified" when revisit record written
Fix for [HER-1532] robots.txt with html <BR> markup causing NumberFormatException
* RobotsTxtTest.java
test based on ecnountered robots.txt with HTML markup but clear intent
* RobotsTxt.java
strip HTML markup from each line before parsing
strip expected numbers in Crawl-Delay to first run of digits-and-decimal-points
catch any NumberFormatException in Crawl-Delay number and ignore, continuing other parsing
Fix for [HER-1277] speculative javascript link caused 404s for "http%3A" and "www.example.com" like strings
* ExtractorJS.java
perform additional decoding or prepending of "http:" for speculative candidate URIs that appear to need it
* ArchiveUtils.java
add utility set/method with list of all TLDs
Fix for [HER-1524] speculativeFixup in ExtractorJS should maintain URL scheme
* ExtractorJS.java
(speculativeFixup) maintain origin URI scheme if apparent target host is same
* FrontierJournal.java
refactor to always operate on CrawlURI instances (with hops-path and via info), and to write longer info on success/failure lines
* AbstractFrontier.java
change boolean 'respect-crawl-delay' to integer 'respect-crawl-delay-up-to-secs'
Fix for [HER-1516] in datamodel.Robotstxt, Integer.parseInt generates NumberFormatException on floating point Crawl-delay strings
* RobotsDirectives.java, RobotsTxt.java, RobotsExclusionPolicy.java
parse, store, return crawlDelay as float
* RobotsTxtTest.java
compare value as float
[HER-1510] meta refresh url from a seed considered as a seed
* LinksScoper.java
(considerAsSeed) return true any seed with hop type Hop.REFER, which
includes meta refreshes; also simplify the code a bit
* ExtractorHTMLTest.java, UURIFactoryTest.java
test that relative-URIs with late-position colons aren't interpreted as absolute URIs with long, illegal schemes
* UURIFactory.java
update RFC2396REGEX to require legal scheme
* LaxURI.java
allow illegal candidate scheme to just be interpreted as something else
* ExtractorHTMLTest.java
test for maintaining 'https' when speculative, leading-domain URI is on same host
* ExtractorJS.java
(speculativeFixup) maintain origin URI scheme if apparent target host is same
Fix for [HER-1526] SCRIPT writing script TYPE common trigger of bogus links (eg. 'text/javascript')
Fix for [HER-1611] more javascript false-positives from Google Analytics insert
* ExtractorJS.java
port of Steve's work in h2: blacklist for URIs, extracted strings to handle common mis-extractions
* ExtractorHTMLTest.java
port of test for 'text/javascript' false extract
* UURIFactoryTest.java
test that port-looking-part that's out of integer bounds throws URIException not NumberFormatException
* UURIFactory.java
catch NumberFormatException; allow to surface as URIException
* SelfTestBase.java
don't allow negative random passwords, in case leading '-' is causing problems
* Heritrix.java
echo arguments when showing usage info -- to understand what went wrong
gojomo