* LongToIntConsistentHash.java + Test
consistent-hashing utility class
* URIAuthorityBasedQueueAssignmentPolicy.java
shared superclass for the hostname and surtauthority QAPs
setting for 'deferToPrevious' -- avoid changing assignments
setting for 'parallelQueues' -- when > 1, consistent-hash URIs across that many separate queues (with numerical suffix)
* SurtAuthorityQueueAssignmentPolicy.java, HostnameQueueAssignmentPolicy.java
refactor to derive from URIAuthorityBasedQueueAssignmentPolicy
* QueueAssignmentPolicy.java
add Apache license
* Hop.java
add String representation for convenience
* ConfigPathConfigurer.java
work in two passes: one (BeanPostProcessor) collects all candidates for fixup, two (ApplicationListener) actually performs fixup
suppress fixup of own base path, preventing circulare ref/StackOverflow
* ConfigPath.java
change sense of 'merge' to merge into receiver
* SurtPrefixedDecideRule.java
use ConfigFile, use 'merge' in assignment
* CrawlerLoggerModule.java
use merge in assignment
* CrawlJob.java
(getBeanpathTarget) utility method
* JobResource.java
respect unset ConfigPaths
* ConfigPathEdito.java, ConfigFileEditor.java
better comments, ConfigFile variant
* HopCrossesAssignmentLevelDomainDecideRule.java
port H2 IsCrossTopmostAssignedSurtHopDecideRule to H3 under this new name
* PublicSuffixes.java
update comment, public method for 'Assignment Level Domain'/'Assignment Level SURT' term
* AssignmentLevelSurtQueueAssignmentPolicy
renamed TopmostAssignedSurtQueueAssignmentPolicy
* (all, especially CachedBdbMap.java)
pbaclace-contributed patch, ported to H3, plus...
* StatisticsTracker.java
replace LongWrapper with AtomicLong
use non-recursive putIfAbsent-based idiom for get-or-create
* DefaultServerCache.java
clean up host/server get-or-create around non-recursive concurrent idiom
* BdbModule.java
add expectedConcurrency property & je.lock.nLockTables tuning
* TopNSet.java
add synchronization, as calling StatisticsTracker method has lost it
* BeanBrowseResource.java
offers browsing (and limited editting of simple values) of beans in built crawl
* JobRelatedResource.java, ReportGenResource.java
refactor for reuse by other job sub-resources
* JobResource.java
link to job/beans and job/script
* DescriptorUpdater.java
helper for marking-up PropertyDescriptors to aid UI bean access
* CrawlJob.java
don't auto-instantiate context on request
* EngineApplication.java
add uri routes
* FetchHTTP.java, BdbCookieStorage.java, LongWrapper.java WorkQueueFrontier.java, BdbUriUniqFilter.java
improve display when browsed (esp. before launch)
* BeanShellProcessor.java -> ScriptedProcessor.java; BeanShellDecideRule.java -> ScriptedDecideRule.java
update to use any present JSR-223 scripting engine (requested by free-text name)
update to allow script to come from inline string or file (just as with seeds)
* modules/pom.xml
include groovy and jsr-223-capable JARs
* .classpath
add new JARs to classpath
* BeanShellProcessorTest.java -> ScriptedProcessorTest.java
rename (empty) test class
* DefaultProcessorURI.java
add constructor that supports setting via
* CustomSWFTags.java, CrawlUriSWFAction.java
look for links in a couple more places
* ExtractorSWF.java
ExtractorTagParser.parsePlaceObject2() - read 32 bit clip event
flags when flash version >= 6
* ExtractorSWFTest.java
xestHer1509() - test this fix (rename to enable the test)
* Extractor.java
added instance initializer for DEFAULT_PARAMETERS, and set
@Autowired(required=false) eliminates the need for test
classes to set them explicitly.
* ExtractorCSSTest.java
remove explicit setting of default parameters
* ExtractorHTMLTest.java
remove explicit setting of default parameters
* JerichoExtractorHTMLTest.java
remove explicit setting of default parameters
Fix for [HER-1533] robots.txt fetch referer header is via url which is confusing
* FetchHTTP.java
configureMethod() don't set referer header if this is a prerequisite
Fix for [HER-1612] lock contention in BeanShellDecideRule/BeanShellProcessor occurs even if isolate-threads=true
* BeanShellDecideRule.java
(getInterpreter) add synchronization
(decisionFor) remove method synchronization; sync on interpreter/getInterpreter makes it a superfluous bottleneck
* BeanShellProcessor.java
(getInterpreter) add synchronization
(innerProcess) remove method synchronization; sync on interpreter/getInterpreter makes it a superfluous bottleneck
Fix for [HER-1225] HtmlFormCredential.java may create non-wellformed URLs
* HtmlFormCredential.java
only insert '&' if previous query-string was non-empty
Fix for [HER-1558] ${HOSTNAME} in arc suffix is only replaced completely
* WriterPoolProcessor.java
replace ${HOSTNAME} in suffix, rather than test for exactly ${HOSTNAME}
Fix for [#HER-1554] META http-equiv refresh content containing only a number misinterpreted as a URI - IA Webteam JIRA:
* ExtractorHTML.java
don't use full 'content' as META refresh URL in absence of '='
Fix for [HER-1499] crawl.log has no annotation indicating that WARCWriterProcessor wrote a revisit record
* WARCWriterProcessor.java
add annotations "warcRevisit:digest" or "warcRevisit:notModified" when revisit record written
Fix for [HER-1532] robots.txt with html <BR> markup causing NumberFormatException
* RobotsTxtTest.java
test based on ecnountered robots.txt with HTML markup but clear intent
* RobotsTxt.java
strip HTML markup from each line before parsing
strip expected numbers in Crawl-Delay to first run of digits-and-decimal-points
catch any NumberFormatException in Crawl-Delay number and ignore, continuing other parsing
Fix for [HER-1277] speculative javascript link caused 404s for "http%3A" and "www.example.com" like strings
* ExtractorJS.java
perform additional decoding or prepending of "http:" for speculative candidate URIs that appear to need it
* ArchiveUtils.java
add utility set/method with list of all TLDs
Fix for [HER-1524] speculativeFixup in ExtractorJS should maintain URL scheme
* ExtractorJS.java
(speculativeFixup) maintain origin URI scheme if apparent target host is same
* AbstractFrontier.java
change boolean 'respect-crawl-delay' to integer 'respect-crawl-delay-up-to-secs'
Fix for [HER-1516] in datamodel.Robotstxt, Integer.parseInt generates NumberFormatException on floating point Crawl-delay strings
* RobotsDirectives.java, RobotsTxt.java, RobotsExclusionPolicy.java
parse, store, return crawlDelay as float
* RobotsTxtTest.java
compare value as float
* ExtractorHTMLTest.java, UURIFactoryTest.java
test that relative-URIs with late-position colons aren't interpreted as absolute URIs with long, illegal schemes
* UURIFactory.java
update RFC2396REGEX to require legal scheme
* LaxURI.java
allow illegal candidate scheme to just be interpreted as something else
* ExtractorHTMLTest.java
test for maintaining 'https' when speculative, leading-domain URI is on same host
* ExtractorJS.java
(speculativeFixup) maintain origin URI scheme if apparent target host is same
Fix for [HER-1526] SCRIPT writing script TYPE common trigger of bogus links (eg. 'text/javascript')
Fix for [HER-1611] more javascript false-positives from Google Analytics insert
* ExtractorJS.java
port of Steve's work in h2: blacklist for URIs, extracted strings to handle common mis-extractions
* ExtractorHTMLTest.java
port of test for 'text/javascript' false extract
gojomo