Note that the name of the engine may be different than the name of the language
it parses. A good example is the Rhino script engine that parses javascript.
* PathSharingContext.java
new member variable ConcurrentHashMap data and accessor getData()
* ScriptedProcessor.java, ScriptedDecideRule.java
make appCtx available to scripts; also remove unused member sharedMap
* ActionDirectory.java
formatting fix
default false is old behavior, skip candidates-handling of outlinks from response codes <200 and >=400
if true, these outlinks will be treated the same as others (get scope-tested and enqueued)
* FetchDNS.java
recordDNS() - use local variable for dummy read buffer, instead of member variable that was shared among threads, since RecordingInputStream.read() used the passed buffer as a work area
(using only path)
(RobotsPolicyTest) added for case that failed pre-fix
(RobotsTxtTest) added case similar to problem (but was already OK at
RobotsTxt level)
(RobotsPolicy, *RobotsPolicy) rename method to reflect new behavior
BaseQueuePrecedencePolicy.basePrecedence) doesn't stick
(CrawlURI) don't null overlayNames in processingCleanup
(WorkQueueFrontier) move queue precedence recalc to only in handleQueue,
not all (post-wake) reenqueues
* HostsReport.java
write() - remove obsolete code described by this no longer correct comment - "StatisticsTracker doesn't know of zero-completion hosts; so supplement report with those entries from host cache" - StatisticsTracker does know of zero-completion hosts
* FetchWhois.java
fetch() - record server ip address in CrawlURI attributes list
* CoreAttributeConstants.java
new string constant A_WHOIS_SERVER_IP, key for whois server ip address CrawlURI attribute
* WARCWriterProcessor.java
writeWhoisRecords() - include warc header WARC-IP-Address
* SheetOverlaysManager.java
getOverlayMap(String) - return null if sheet missing instead of triggering npe
* KeyedProperties.java
get(String) - check for null return value from getOverlayMap() and log warning
when starting in background, make better effort to write pid to $HERITRIX_HOME/heritrix.pid on successful start, plus this change:
- echo -n "`date` Starting heritrix"
+ echo -n "`date` Heritrix starting (pid $HERITRIX_PID)"
* KryoBinding.java
wrap ObjectBuffers with WeakReference so that can be garbage collected if necessary (when recreated they'll be back at the default size of 16k)
Reported by Kenji who says, "My guess is that root cause is Recorder.calcRecommendedCharBufferSize():
return Math.min(inStream.getRecordedBufferLength()/2,(int)inStream.getSize());
URL above returns HTML larger than 5GB (infinite smileys!! what the heck), and (int)intStream.getSize() became negative."
* Recorder.java
- return Math.min(inStream.getRecordedBufferLength()/2,(int)inStream.getSize());
+ return (int) Math.min(inStream.getRecordedBufferLength()/2, inStream.getSize());
* WorkQueueFrontier.java
activateInactiveQueue(), deactivateQueue() - synchronize around updates of inactiveQueuesByPrecedence and highestPrecedenceWaiting
* Heritrix.java
instanceMain() - move logging of java vendor/version ahead of version check, since the information could be particularly helpful when version check doesn't pass
* CrawlController.java
requestCrawlResume() - do not create ToePool (creation here appeared to be inherited cruft), paused crawl must already have toe pool
* UriUtils.java
isLikelyFalsePositive() - check for unusual characters, likely mimetypes, and a couple of other common traps
* UriUtilsTest.java
add some tests for this code
Elaboration: CrawlJob needs all beans to receive FINISHED event before teardown. Can't easily guarantee CrawlJob receives the event last. Actually, spring provides a way, org.springframework.core.Ordered, but to make a bean receive events last, *every other bean* must implement Ordered. So this way is simpler.
* CrawlController.java
new member variable isStopComplete and event StopCompleteEvent
* CrawlJob.java
rely on cc.isStopComplete() or StopCompleteEvent to indicate ready for teardown
instantiateContainer() - do not call teardown() on exception from ac.refresh() - can throw IllegalStateException
teardown() - put all uses of variable cc inside not-null check
* PathSharingContext.java
start(), doStart(), stop(), doStop() - remove these overloaded methods, because the spring bug they were working around seems to be fixed, not seeing any problems with cyclical dependencies - might be this one https://jira.springsource.org/browse/SPR-7266
doClose() - remove because superclass version seems to work fine (this method generally wasn't being called anyway, though it would be now with other changes in this checkin)
* CrawlJob.java
refactor teardown to call close() on the ApplicationContext, which calls destroy() on any beans that implement DisposableBean - this is now the way to have beans do stuff at teardown
* CrawlController.java
send FINISHED crawl state event after calling appCtx.stop() so that isFinished() can indicate ready-ness for teardown
* BdbModule.java
move close() to teardown, i.e. implementation of DisposableBean.destroy(); remove shutdown hook and rely on teardown; related tweaks
* WorkQueueFrontier.java, CrawlerLoggerModule.java, BdbUriUniqFilter.java
move close() to teardown
* CrawlMapper.java, AbstractFrontier.java, PreloadedUriPrecedencePolicy.java, FetchWhois.java, FetchHTTP.java, PersistLogProcessor.java, WriterPoolProcessor.java
add comments about cleanup that maybe should wait until teardown
* UriUniqFilter.java
remove incorrect(?) comment
* ToeThread.java
run() - set local variable curi=null when finished with it, because the jvm seems to be holding on to it even after the containing block is finished
* URIAuthorityBasedQueueAssignmentPolicy.java
remove field kp, accessor getKeyedProperties(); instead inherit these from superclass QueueAssignmentPolicy
Travis Wellman