* HostsReport.java
write() - remove obsolete code described by this no longer correct comment - "StatisticsTracker doesn't know of zero-completion hosts; so supplement report with those entries from host cache" - StatisticsTracker does know of zero-completion hosts
* FetchWhois.java
fetch() - record server ip address in CrawlURI attributes list
* CoreAttributeConstants.java
new string constant A_WHOIS_SERVER_IP, key for whois server ip address CrawlURI attribute
* WARCWriterProcessor.java
writeWhoisRecords() - include warc header WARC-IP-Address
* SheetOverlaysManager.java
getOverlayMap(String) - return null if sheet missing instead of triggering npe
* KeyedProperties.java
get(String) - check for null return value from getOverlayMap() and log warning
when starting in background, make better effort to write pid to $HERITRIX_HOME/heritrix.pid on successful start, plus this change:
- echo -n "`date` Starting heritrix"
+ echo -n "`date` Heritrix starting (pid $HERITRIX_PID)"
* KryoBinding.java
wrap ObjectBuffers with WeakReference so that can be garbage collected if necessary (when recreated they'll be back at the default size of 16k)
Reported by Kenji who says, "My guess is that root cause is Recorder.calcRecommendedCharBufferSize():
return Math.min(inStream.getRecordedBufferLength()/2,(int)inStream.getSize());
URL above returns HTML larger than 5GB (infinite smileys!! what the heck), and (int)intStream.getSize() became negative."
* Recorder.java
- return Math.min(inStream.getRecordedBufferLength()/2,(int)inStream.getSize());
+ return (int) Math.min(inStream.getRecordedBufferLength()/2, inStream.getSize());
* WorkQueueFrontier.java
activateInactiveQueue(), deactivateQueue() - synchronize around updates of inactiveQueuesByPrecedence and highestPrecedenceWaiting
* Heritrix.java
instanceMain() - move logging of java vendor/version ahead of version check, since the information could be particularly helpful when version check doesn't pass
* CrawlController.java
requestCrawlResume() - do not create ToePool (creation here appeared to be inherited cruft), paused crawl must already have toe pool
* UriUtils.java
isLikelyFalsePositive() - check for unusual characters, likely mimetypes, and a couple of other common traps
* UriUtilsTest.java
add some tests for this code
Elaboration: CrawlJob needs all beans to receive FINISHED event before teardown. Can't easily guarantee CrawlJob receives the event last. Actually, spring provides a way, org.springframework.core.Ordered, but to make a bean receive events last, *every other bean* must implement Ordered. So this way is simpler.
* CrawlController.java
new member variable isStopComplete and event StopCompleteEvent
* CrawlJob.java
rely on cc.isStopComplete() or StopCompleteEvent to indicate ready for teardown
instantiateContainer() - do not call teardown() on exception from ac.refresh() - can throw IllegalStateException
teardown() - put all uses of variable cc inside not-null check
* PathSharingContext.java
start(), doStart(), stop(), doStop() - remove these overloaded methods, because the spring bug they were working around seems to be fixed, not seeing any problems with cyclical dependencies - might be this one https://jira.springsource.org/browse/SPR-7266
doClose() - remove because superclass version seems to work fine (this method generally wasn't being called anyway, though it would be now with other changes in this checkin)
* CrawlJob.java
refactor teardown to call close() on the ApplicationContext, which calls destroy() on any beans that implement DisposableBean - this is now the way to have beans do stuff at teardown
* CrawlController.java
send FINISHED crawl state event after calling appCtx.stop() so that isFinished() can indicate ready-ness for teardown
* BdbModule.java
move close() to teardown, i.e. implementation of DisposableBean.destroy(); remove shutdown hook and rely on teardown; related tweaks
* WorkQueueFrontier.java, CrawlerLoggerModule.java, BdbUriUniqFilter.java
move close() to teardown
* CrawlMapper.java, AbstractFrontier.java, PreloadedUriPrecedencePolicy.java, FetchWhois.java, FetchHTTP.java, PersistLogProcessor.java, WriterPoolProcessor.java
add comments about cleanup that maybe should wait until teardown
* UriUniqFilter.java
remove incorrect(?) comment
* ToeThread.java
run() - set local variable curi=null when finished with it, because the jvm seems to be holding on to it even after the containing block is finished
* URIAuthorityBasedQueueAssignmentPolicy.java
remove field kp, accessor getKeyedProperties(); instead inherit these from superclass QueueAssignmentPolicy
* BasicProfileTest.java, SelfTestBase.java, CrawlControllerTest.java, PrecedenceLoader.java, MigrateH1to3Tool.java, CrawlerLoggerModule.java, StatisticsTracker.java, CheckpointUtils.java, BdbUriUniqFilter.java, ARCWriterProcessorTest.java, WARCWriterProcessorTest.java, PersistProcessor.java, WriterPoolProcessor.java, PrefixFinderTest.java, StoredQueueTest.java, FileUtilsTest.java, ObjectIdentityBdbManualCacheTest.java, ObjectIdentityBdbCacheTest.java, ObjectPlusFilesOutputStream.java, TestUtils.java, TmpDirTestCase.java, Engine.java
Replace calls to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir). In these cases the calls were either already in a spot where the possible IOException would be handled appropriately, or the line was trivially moved into such a block.
* ActionDirectory.java, Engine.java
replace calls to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir), and throw IllegalStateException on failure
* BdbModule.java
setup() - replace call to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir) and add "throws IOException" - conveniently the place where this method is called was already in a try block that catches IOException
* Checkpoint.java
generateFrom() - replace call to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir) and add "throws IOException"
* CheckpointService.java
move call to Checkpoint.generateFrom() inside existing try block since it now can throw IOException
* Recorder.java
ensure(File) - replace call to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir), and throw IllegalStateException on failure
new Recorder(File,String,int,int) - call ensure() on the correct object, the containing directory; and remove redundant call to ensure()
Noah Levitt