* master:
use constant from rethinkdb lib for default port
Update README.md
Handle missing closing paren in srcset descriptor
Teach jericho extractor srcset
Don't run srcset test against jericho, it doesn't handle it
Handle commas more compliantly when parsing srcset
Ensure we start parsing full lines, for #239.
Commas are allowed if they're in the middle of the URL. Consequently:
srcset="a,b,,c," => ["a,b,,c"]
srcset="a, b,, c," => ["a", "b", "c"]
They occur particularly commonly in data: URLs before the base64 value.
Commas are also allowed in descriptors if they are enclosed by parens:
srcset="a (b,c),d" => ["a", "d"]
Spec: https://html.spec.whatwg.org/multipage/images.html#parsing-a-srcset-attribute
* trough-dedup:
promote dirty segments at crawl finish
trough dedup!
Allow failed lookups to expire, for #234.
[maven-release-plugin] prepare for next development iteration
[maven-release-plugin] prepare release 3.4.0-20190207
As @nlevitt suggestion, a further check.
[maven-release-plugin] prepare for next development iteration
[maven-release-plugin] prepare release 3.4.0-20190205
[maven-release-plugin] prepare for next development iteration
[maven-release-plugin] prepare release 3.4.0-20190205-2
[maven-release-plugin] prepare for next development iteration
Extend to set additional property.
Use argument syntax.
Skip tests during release process (covered by CI).
Set consistent tag, and include contrib.
Wrong repo spec.
Add build profile for deployment to Maven Central.
Avoid headings being treated as lists
Clarification about APIs.
Swapped original and link to make maintenance simpler.
Tidy up markup and links.
Add synchronized statements for internetarchive/heritrix3#221.
Add checks to guard against server sending 304 in error, for #229.
do not checkpoint if crawl job has not started
DecideRuleSequence already has the optimization that I was looking for,
namely, don't bother evaluating a DecideRule if we know it won't change
the current result. For some reason PredicatedDecideRule, which is a
parent class to most of the decide rules in heritrix, didn't implement
onlyDecision(). Certain crawl configurations could see significant
performance improvement with this change.
* hbase-refactor:
use non-deprecated hbase api
Correct spelling mistakes.
Note feature only applies to forthcoming 3.3 release
Update API with note about checkpoint launching.
Default to starting anew if no checkpoints are found.
Provide a simple hook to restart from the latest checkpoint.
Move sync to after Null check.
Attempt to cache dependencies.
Simplify contrib build.
Attempt to prevent build time-outs.
Add synchronisation around host and server stats.
Avoid overdoing the RAM allocation.
Ensure all WorkQueue modification as serialised across threads.
Tests now need more RAM.
fix exception starting DecideRuleSequence logging
Fix up tests to account for new Base URI behaviour.
Only set the BaseURI if not set already.
Same fix for JerichoExtractorHTML.
Test case and fix for internetarchive/heritrix3#208.
Fix link to User Guide
Fix typos
Replace API response section with statuscode tag
Reformat the rest of the API calls
Add actions to API URLs to distinguish them
Add explanatory note about documentation
Try nicer formatting for the first API example
Format HTTP request lines in API guide
Move conventions and REST section to the end of the document
Convert html tables to rst tables
Add skeleton sphinx docs for readthedocs
Point at GitHub wiki for latest release info
Add parameter to allow even distribution for parallel queues.
I don't know why we've never seen this before, and now we suddenly have
a case of it, but this is the exception:
2018-07-23 17:47:01.123 SEVERE thread-2875088 org.archive.crawler.framework.CrawlJob.beansException() Failed to start bean 'scope'; nested exception is java.lang.IllegalStateException: java.nio.file.NoSuchFileException: /1/ait-h3-jobs/8144-20180723162745141/20180723174701/logs/scope.log.lck
org.springframework.context.ApplicationContextException: Failed to start bean 'scope'; nested exception is java.lang.IllegalStateException: java.nio.file.NoSuchFileException: /1/ait-h3-jobs/8144-20180723162745141/20180723174701/logs/scope.log.lck
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:169)
at org.springframework.context.support.DefaultLifecycleProcessor.access$1(DefaultLifecycleProcessor.java:154)
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:335)
at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:143)
at org.springframework.context.support.DefaultLifecycleProcessor.start(DefaultLifecycleProcessor.java:89)
at org.springframework.context.support.AbstractApplicationContext.start(AbstractApplicationContext.java:1236)
at org.archive.spring.PathSharingContext.start(PathSharingContext.java:115)
at org.archive.crawler.framework.CrawlJob.startContext(CrawlJob.java:455)
at org.archive.crawler.framework.CrawlJob$1.run(CrawlJob.java:428)
Caused by: java.lang.IllegalStateException: java.nio.file.NoSuchFileException: /1/ait-h3-jobs/8144-20180723162745141/20180723174701/logs/scope.log.lck
at org.archive.crawler.reporting.CrawlerLoggerModule.setupSimpleLog(CrawlerLoggerModule.java:298)
at org.archive.modules.deciderules.DecideRuleSequence.start(DecideRuleSequence.java:174)
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:166)
... 8 more
Caused by: java.nio.file.NoSuchFileException: /1/ait-h3-jobs/8144-20180723162745141/20180723174701/logs/scope.log.lck
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:335)
at java.util.logging.FileHandler.openFiles(FileHandler.java:478)
at java.util.logging.FileHandler.<init>(FileHandler.java:344)
at org.archive.io.GenerationFileHandler.<init>(GenerationFileHandler.java:63)
at org.archive.io.GenerationFileHandler.makeNew(GenerationFileHandler.java:158)
at org.archive.crawler.reporting.CrawlerLoggerModule.setupLogFile(CrawlerLoggerModule.java:275)
at org.archive.crawler.reporting.CrawlerLoggerModule.setupSimpleLog(CrawlerLoggerModule.java:296)
... 10 more
* ari-5630:
catch exceptions scoping outlinks to stop them from derailing processing of the parent url
fix for test failures in a workspace on NFS-mounted filesystem
max size for extracted form elements
ContentDigestHistoryTest does not close BdbModule. It results in failure to delete bdb directory in following tests.
Added tearDown() method that closes BdbModule.
Since we were only enforcing the size limit after reading each
line a single very long line could cause us to to run out of
memory.
This change enforces the character limit on each read character
not just on line boundaries.
We also correct the count of processed characters in the warning
message, which was not counting newline and linefeed characters.
Noah Levitt