DecideRuleSequence already has the optimization that I was looking for,
namely, don't bother evaluating a DecideRule if we know it won't change
the current result. For some reason PredicatedDecideRule, which is a
parent class to most of the decide rules in heritrix, didn't implement
onlyDecision(). Certain crawl configurations could see significant
performance improvement with this change.
I don't know why we've never seen this before, and now we suddenly have
a case of it, but this is the exception:
2018-07-23 17:47:01.123 SEVERE thread-2875088 org.archive.crawler.framework.CrawlJob.beansException() Failed to start bean 'scope'; nested exception is java.lang.IllegalStateException: java.nio.file.NoSuchFileException: /1/ait-h3-jobs/8144-20180723162745141/20180723174701/logs/scope.log.lck
org.springframework.context.ApplicationContextException: Failed to start bean 'scope'; nested exception is java.lang.IllegalStateException: java.nio.file.NoSuchFileException: /1/ait-h3-jobs/8144-20180723162745141/20180723174701/logs/scope.log.lck
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:169)
at org.springframework.context.support.DefaultLifecycleProcessor.access$1(DefaultLifecycleProcessor.java:154)
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:335)
at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:143)
at org.springframework.context.support.DefaultLifecycleProcessor.start(DefaultLifecycleProcessor.java:89)
at org.springframework.context.support.AbstractApplicationContext.start(AbstractApplicationContext.java:1236)
at org.archive.spring.PathSharingContext.start(PathSharingContext.java:115)
at org.archive.crawler.framework.CrawlJob.startContext(CrawlJob.java:455)
at org.archive.crawler.framework.CrawlJob$1.run(CrawlJob.java:428)
Caused by: java.lang.IllegalStateException: java.nio.file.NoSuchFileException: /1/ait-h3-jobs/8144-20180723162745141/20180723174701/logs/scope.log.lck
at org.archive.crawler.reporting.CrawlerLoggerModule.setupSimpleLog(CrawlerLoggerModule.java:298)
at org.archive.modules.deciderules.DecideRuleSequence.start(DecideRuleSequence.java:174)
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:166)
... 8 more
Caused by: java.nio.file.NoSuchFileException: /1/ait-h3-jobs/8144-20180723162745141/20180723174701/logs/scope.log.lck
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:335)
at java.util.logging.FileHandler.openFiles(FileHandler.java:478)
at java.util.logging.FileHandler.<init>(FileHandler.java:344)
at org.archive.io.GenerationFileHandler.<init>(GenerationFileHandler.java:63)
at org.archive.io.GenerationFileHandler.makeNew(GenerationFileHandler.java:158)
at org.archive.crawler.reporting.CrawlerLoggerModule.setupLogFile(CrawlerLoggerModule.java:275)
at org.archive.crawler.reporting.CrawlerLoggerModule.setupSimpleLog(CrawlerLoggerModule.java:296)
... 10 more
ContentDigestHistoryTest does not close BdbModule. It results in failure to delete bdb directory in following tests.
Added tearDown() method that closes BdbModule.
Since we were only enforcing the size limit after reading each
line a single very long line could cause us to to run out of
memory.
This change enforces the character limit on each read character
not just on line boundaries.
We also correct the count of processed characters in the warning
message, which was not counting newline and linefeed characters.
* origin/master:
Setup TravisCI
Set fetch status on curis when testing link extraction
No link extraction on URI not successfully downloaded
Fixes issue #158 : Noisy alerts about 401s without auth challenge (#159)
Make Content-Location header url INFERRED not REFFER hop type since Content-Location is not for redirection (#151)
fixes for kafka 0.9 (?)
upgrade to kafka 0.9
somewhat ugly fix to handle exceptions from the bean browser like java.lang.RuntimeException: not implemented at org.archive.modules.fetcher.BdbCookieStore$RestrictedCollectionWrappedList.get(BdbCookieStore.java:92)
Andrew Jackson