* ServerCache.java
make abstract, implement getHostFor(UURI) and getServerFor(UURI)
* DefaultServerCache.java
extend ServerCache, remove getHostFor(UURI) and getServerFor(UURI)
* ServerCacheUtil.java
removed
* other classes
change usage of ServerCacheUtil to ServerCache
* DefaultServerCache.java
getHostFor(UURI): special case dns uris
* ServerCache.java
add getHostFor(UURI) to public interface
* ServerCacheUtil.java
remove now redundant wrapper method
* DispositionProcessor.java AbstractFrontier.java IPQueueAssignmentPolicy.java
BucketQueueAssignmentPolicy.java PreconditionEnforcer.java QuotaEnforcer.java
StatisticsTracker.java ServerCacheTest.java FetchHTTP.java
Kw3WriterProcessor.java WriterPoolProcessor.java
changed to use serverCache.getHostFor(UURI) (a few cases remain that use
getHostFor(String))
* (many)
update license notice to Apache where appropriate
eliminate ProcessorURI, DefaultProcessorURI in favor of CrawlURI (now in modules package)
other comment/warning/unused code cleanup
* BeanShellProcessor.java -> ScriptedProcessor.java; BeanShellDecideRule.java -> ScriptedDecideRule.java
update to use any present JSR-223 scripting engine (requested by free-text name)
update to allow script to come from inline string or file (just as with seeds)
* modules/pom.xml
include groovy and jsr-223-capable JARs
* .classpath
add new JARs to classpath
* BeanShellProcessorTest.java -> ScriptedProcessorTest.java
rename (empty) test class
* DefaultProcessorURI.java
add constructor that supports setting via
* CustomSWFTags.java, CrawlUriSWFAction.java
look for links in a couple more places
* ExtractorSWF.java
ExtractorTagParser.parsePlaceObject2() - read 32 bit clip event
flags when flash version >= 6
* ExtractorSWFTest.java
xestHer1509() - test this fix (rename to enable the test)
* Extractor.java
added instance initializer for DEFAULT_PARAMETERS, and set
@Autowired(required=false) eliminates the need for test
classes to set them explicitly.
* ExtractorCSSTest.java
remove explicit setting of default parameters
* ExtractorHTMLTest.java
remove explicit setting of default parameters
* JerichoExtractorHTMLTest.java
remove explicit setting of default parameters
Fix for [HER-1532] robots.txt with html <BR> markup causing NumberFormatException
* RobotsTxtTest.java
test based on ecnountered robots.txt with HTML markup but clear intent
* RobotsTxt.java
strip HTML markup from each line before parsing
strip expected numbers in Crawl-Delay to first run of digits-and-decimal-points
catch any NumberFormatException in Crawl-Delay number and ignore, continuing other parsing
* AbstractFrontier.java
change boolean 'respect-crawl-delay' to integer 'respect-crawl-delay-up-to-secs'
Fix for [HER-1516] in datamodel.Robotstxt, Integer.parseInt generates NumberFormatException on floating point Crawl-delay strings
* RobotsDirectives.java, RobotsTxt.java, RobotsExclusionPolicy.java
parse, store, return crawlDelay as float
* RobotsTxtTest.java
compare value as float
* ExtractorHTMLTest.java, UURIFactoryTest.java
test that relative-URIs with late-position colons aren't interpreted as absolute URIs with long, illegal schemes
* UURIFactory.java
update RFC2396REGEX to require legal scheme
* LaxURI.java
allow illegal candidate scheme to just be interpreted as something else
* ExtractorHTMLTest.java
test for maintaining 'https' when speculative, leading-domain URI is on same host
* ExtractorJS.java
(speculativeFixup) maintain origin URI scheme if apparent target host is same
Fix for [HER-1526] SCRIPT writing script TYPE common trigger of bogus links (eg. 'text/javascript')
Fix for [HER-1611] more javascript false-positives from Google Analytics insert
* ExtractorJS.java
port of Steve's work in h2: blacklist for URIs, extracted strings to handle common mis-extractions
* ExtractorHTMLTest.java
port of test for 'text/javascript' false extract
nlevitt