* hbase-refactor:
use non-deprecated hbase api
Correct spelling mistakes.
Note feature only applies to forthcoming 3.3 release
Update API with note about checkpoint launching.
Default to starting anew if no checkpoints are found.
Provide a simple hook to restart from the latest checkpoint.
Move sync to after Null check.
Attempt to cache dependencies.
Simplify contrib build.
Attempt to prevent build time-outs.
Add synchronisation around host and server stats.
Avoid overdoing the RAM allocation.
Ensure all WorkQueue modification as serialised across threads.
Tests now need more RAM.
fix exception starting DecideRuleSequence logging
Fix up tests to account for new Base URI behaviour.
Only set the BaseURI if not set already.
Same fix for JerichoExtractorHTML.
Test case and fix for internetarchive/heritrix3#208.
Fix link to User Guide
Fix typos
Replace API response section with statuscode tag
Reformat the rest of the API calls
Add actions to API URLs to distinguish them
Add explanatory note about documentation
Try nicer formatting for the first API example
Format HTTP request lines in API guide
Move conventions and REST section to the end of the document
Convert html tables to rst tables
Add skeleton sphinx docs for readthedocs
Point at GitHub wiki for latest release info
Add parameter to allow even distribution for parallel queues.
Catch exceptions and attempt to reset the hbase connection. The intent
is for this to fix the problem where the hbase connection (under the
hood the problem is the zookeeper connection) gets into a borked state,
and we never reset it.
I don't know why we've never seen this before, and now we suddenly have
a case of it, but this is the exception:
2018-07-23 17:47:01.123 SEVERE thread-2875088 org.archive.crawler.framework.CrawlJob.beansException() Failed to start bean 'scope'; nested exception is java.lang.IllegalStateException: java.nio.file.NoSuchFileException: /1/ait-h3-jobs/8144-20180723162745141/20180723174701/logs/scope.log.lck
org.springframework.context.ApplicationContextException: Failed to start bean 'scope'; nested exception is java.lang.IllegalStateException: java.nio.file.NoSuchFileException: /1/ait-h3-jobs/8144-20180723162745141/20180723174701/logs/scope.log.lck
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:169)
at org.springframework.context.support.DefaultLifecycleProcessor.access$1(DefaultLifecycleProcessor.java:154)
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:335)
at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:143)
at org.springframework.context.support.DefaultLifecycleProcessor.start(DefaultLifecycleProcessor.java:89)
at org.springframework.context.support.AbstractApplicationContext.start(AbstractApplicationContext.java:1236)
at org.archive.spring.PathSharingContext.start(PathSharingContext.java:115)
at org.archive.crawler.framework.CrawlJob.startContext(CrawlJob.java:455)
at org.archive.crawler.framework.CrawlJob$1.run(CrawlJob.java:428)
Caused by: java.lang.IllegalStateException: java.nio.file.NoSuchFileException: /1/ait-h3-jobs/8144-20180723162745141/20180723174701/logs/scope.log.lck
at org.archive.crawler.reporting.CrawlerLoggerModule.setupSimpleLog(CrawlerLoggerModule.java:298)
at org.archive.modules.deciderules.DecideRuleSequence.start(DecideRuleSequence.java:174)
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:166)
... 8 more
Caused by: java.nio.file.NoSuchFileException: /1/ait-h3-jobs/8144-20180723162745141/20180723174701/logs/scope.log.lck
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:335)
at java.util.logging.FileHandler.openFiles(FileHandler.java:478)
at java.util.logging.FileHandler.<init>(FileHandler.java:344)
at org.archive.io.GenerationFileHandler.<init>(GenerationFileHandler.java:63)
at org.archive.io.GenerationFileHandler.makeNew(GenerationFileHandler.java:158)
at org.archive.crawler.reporting.CrawlerLoggerModule.setupLogFile(CrawlerLoggerModule.java:275)
at org.archive.crawler.reporting.CrawlerLoggerModule.setupSimpleLog(CrawlerLoggerModule.java:296)
... 10 more
* ari-5630:
catch exceptions scoping outlinks to stop them from derailing processing of the parent url
fix for test failures in a workspace on NFS-mounted filesystem
max size for extracted form elements
ContentDigestHistoryTest does not close BdbModule. It results in failure to delete bdb directory in following tests.
Added tearDown() method that closes BdbModule.
Noah Levitt