* DefaultProcessorURI.java
add constructor that supports setting via
* CustomSWFTags.java, CrawlUriSWFAction.java
look for links in a couple more places
* ExtractorSWF.java
ExtractorTagParser.parsePlaceObject2() - read 32 bit clip event
flags when flash version >= 6
* ExtractorSWFTest.java
xestHer1509() - test this fix (rename to enable the test)
* Extractor.java
added instance initializer for DEFAULT_PARAMETERS, and set
@Autowired(required=false) eliminates the need for test
classes to set them explicitly.
* ExtractorCSSTest.java
remove explicit setting of default parameters
* ExtractorHTMLTest.java
remove explicit setting of default parameters
* JerichoExtractorHTMLTest.java
remove explicit setting of default parameters
Fix for [HER-1532] robots.txt with html <BR> markup causing NumberFormatException
* RobotsTxtTest.java
test based on ecnountered robots.txt with HTML markup but clear intent
* RobotsTxt.java
strip HTML markup from each line before parsing
strip expected numbers in Crawl-Delay to first run of digits-and-decimal-points
catch any NumberFormatException in Crawl-Delay number and ignore, continuing other parsing
* AbstractFrontier.java
change boolean 'respect-crawl-delay' to integer 'respect-crawl-delay-up-to-secs'
Fix for [HER-1516] in datamodel.Robotstxt, Integer.parseInt generates NumberFormatException on floating point Crawl-delay strings
* RobotsDirectives.java, RobotsTxt.java, RobotsExclusionPolicy.java
parse, store, return crawlDelay as float
* RobotsTxtTest.java
compare value as float
* ExtractorHTMLTest.java, UURIFactoryTest.java
test that relative-URIs with late-position colons aren't interpreted as absolute URIs with long, illegal schemes
* UURIFactory.java
update RFC2396REGEX to require legal scheme
* LaxURI.java
allow illegal candidate scheme to just be interpreted as something else
* ExtractorHTMLTest.java
test for maintaining 'https' when speculative, leading-domain URI is on same host
* ExtractorJS.java
(speculativeFixup) maintain origin URI scheme if apparent target host is same
Fix for [HER-1526] SCRIPT writing script TYPE common trigger of bogus links (eg. 'text/javascript')
Fix for [HER-1611] more javascript false-positives from Google Analytics insert
* ExtractorJS.java
port of Steve's work in h2: blacklist for URIs, extracted strings to handle common mis-extractions
* ExtractorHTMLTest.java
port of test for 'text/javascript' false extract
szznax