Fix for [HER-1612] lock contention in BeanShellDecideRule/BeanShellProcessor occurs even if isolate-threads=true
* BeanShellDecideRule.java
(getInterpreter) add synchronization
(decisionFor) remove method synchronization; sync on interpreter/getInterpreter makes it a superfluous bottleneck
* BeanShellProcessor.java
(getInterpreter) add synchronization
(innerProcess) remove method synchronization; sync on interpreter/getInterpreter makes it a superfluous bottleneck
Fix for [HER-1225] HtmlFormCredential.java may create non-wellformed URLs
* HtmlFormCredential.java
only insert '&' if previous query-string was non-empty
Fix for [HER-1558] ${HOSTNAME} in arc suffix is only replaced completely
* WriterPoolProcessor.java
replace ${HOSTNAME} in suffix, rather than test for exactly ${HOSTNAME}
Fix for [#HER-1554] META http-equiv refresh content containing only a number misinterpreted as a URI - IA Webteam JIRA:
* ExtractorHTML.java
don't use full 'content' as META refresh URL in absence of '='
Fix for [HER-1499] crawl.log has no annotation indicating that WARCWriterProcessor wrote a revisit record
* WARCWriterProcessor.java
add annotations "warcRevisit:digest" or "warcRevisit:notModified" when revisit record written
Fix for [HER-1532] robots.txt with html <BR> markup causing NumberFormatException
* RobotsTxtTest.java
test based on ecnountered robots.txt with HTML markup but clear intent
* RobotsTxt.java
strip HTML markup from each line before parsing
strip expected numbers in Crawl-Delay to first run of digits-and-decimal-points
catch any NumberFormatException in Crawl-Delay number and ignore, continuing other parsing
Fix for [HER-1277] speculative javascript link caused 404s for "http%3A" and "www.example.com" like strings
* ExtractorJS.java
perform additional decoding or prepending of "http:" for speculative candidate URIs that appear to need it
* ArchiveUtils.java
add utility set/method with list of all TLDs
Fix for [HER-1524] speculativeFixup in ExtractorJS should maintain URL scheme
* ExtractorJS.java
(speculativeFixup) maintain origin URI scheme if apparent target host is same
* AbstractFrontier.java
change boolean 'respect-crawl-delay' to integer 'respect-crawl-delay-up-to-secs'
Fix for [HER-1516] in datamodel.Robotstxt, Integer.parseInt generates NumberFormatException on floating point Crawl-delay strings
* RobotsDirectives.java, RobotsTxt.java, RobotsExclusionPolicy.java
parse, store, return crawlDelay as float
* RobotsTxtTest.java
compare value as float
* ExtractorHTMLTest.java, UURIFactoryTest.java
test that relative-URIs with late-position colons aren't interpreted as absolute URIs with long, illegal schemes
* UURIFactory.java
update RFC2396REGEX to require legal scheme
* LaxURI.java
allow illegal candidate scheme to just be interpreted as something else
* ExtractorHTMLTest.java
test for maintaining 'https' when speculative, leading-domain URI is on same host
* ExtractorJS.java
(speculativeFixup) maintain origin URI scheme if apparent target host is same
Fix for [HER-1526] SCRIPT writing script TYPE common trigger of bogus links (eg. 'text/javascript')
Fix for [HER-1611] more javascript false-positives from Google Analytics insert
* ExtractorJS.java
port of Steve's work in h2: blacklist for URIs, extracted strings to handle common mis-extractions
* ExtractorHTMLTest.java
port of test for 'text/javascript' false extract
gojomo