11 Commits

Author	SHA1	Message	Date
gojomo	db794496fb	copy to h3 ... Fix for [HER-1550] set-source-tag option nonfunctional * CrawlURI.java ensure 'heritable' key is itself heritable; also change heritable collection to a set	2009-05-14 22:20:07 +00:00
gojomo	dae03cb1de	port to h3 noah's fix ... Fix for [HER-1533] bottleneck in StatisticsTracker.saveSourceStats * StatisticsTrackerImpl.java saveSourceStats() -- call put() on the CachedBdbMap only when entry is added, not on every change	2009-05-14 20:51:46 +00:00
gojomo	7480d2933b	Cleanup for h3 of [HER-1045] [contrib] make it easier to set FetchHTTP local bind address, use rotation of alternate local addresses ... * FetchHTTP.java advertise key for CrawlURI-based bindAddress override as static constant HTTP_BIND_ADDRESS * CoreAttributeConstants.java remove unused A_HTTP_BIND_ADDRESS	2009-05-13 23:49:36 +00:00
gojomo	ab6942a60d	Fix for [HER-1291] accurate scoping not possible at log-recovery time; URIs may be recrawled against expectations ... * FrontierJournal.java refactor to always operate on CrawlURI instances (with hops-path and via info), and to write longer info on success/failure lines	2009-05-13 01:43:41 +00:00
gojomo	a530af6e5c	Improvement for [HER-1] Support 'Crawl-Delay' and 'Allow' robots.txt directives ... * AbstractFrontier.java change boolean 'respect-crawl-delay' to integer 'respect-crawl-delay-up-to-secs' Fix for [HER-1516] in datamodel.Robotstxt, Integer.parseInt generates NumberFormatException on floating point Crawl-delay strings * RobotsDirectives.java, RobotsTxt.java, RobotsExclusionPolicy.java parse, store, return crawlDelay as float * RobotsTxtTest.java compare value as float	2009-05-13 01:14:58 +00:00
gojomo	2c7c07ba64	apply to h3 noah's fix ... [HER-1510] meta refresh url from a seed considered as a seed * LinksScoper.java (considerAsSeed) return true any seed with hop type Hop.REFER, which includes meta refreshes; also simplify the code a bit	2009-05-12 22:29:54 +00:00
gojomo	2c11eb1dc2	Remove cruft comments	2009-05-12 19:50:05 +00:00
gojomo	8de6b2a9c5	Fix for [HER-1502] selftests break if 'localhost' does not resolve to 127.0.0.1 ... * (all) change 'localhost' to '127.0.0.1'	2009-05-12 18:40:18 +00:00
gojomo	c8a5446695	Test robustness ... * SelfTestBase.java don't allow negative random passwords, in case leading '-' is causing problems * Heritrix.java echo arguments when showing usage info -- to understand what went wrong	2009-05-12 02:10:37 +00:00
gojomo	daf5ed7590	Update pom.xml * 4 ... more naming improvements ignore test cruft update SCM url	2009-05-12 00:17:01 +00:00
gojomo	e047bf68e0	copy 'springy' branch to heritrix3	2009-05-11 22:56:36 +00:00