* origin/master:
fix line ending and indentation issues
AssignmentLevelSurtQueueAssignmentPolicy.java - Add support for forced queue assignment and parallel queues URIAuthorityBasedQueueAssignmentPolicy.java - Add interoperability between forced queue assignment and parallel queues QuotaEnforcer.java - Fix javadoc to match default behavior
Override PUT so it doesn't change the file extension
Use super.getVariants() rather than super.getVariants(GET)
Remove testConcurrentLoad
Mitigate random CookieStore.testConcurrentLoad test failures
Add missing UUID import (interactive commit fail)
Fix digest authentication
Link to javadoc.io for more recent api docs
Now jdk11 support is merged we can disallow test failures on it
Fix restlet child component context warning
Update statistics assertions as Jetty 9 produces different headers
Fix 'No session data store configured' test error
Drop support for JDK 7
JDK11: upgrade engine to jetty 9.4.19 and restlet 2.4.0
JDK11 support: upgrade jetty to 9.4.19 in modules tests
Prevent newer surefire from invoking FetchHTTPTests directly
JDK11 support: remove unused class ObjectIdentityBdbCache and tests
JDK11 support: exclude tools.jar from hbase-client dependency
JDK11 support: explicitly depend on JAXB
JDK11 support: upgrade maven-surefire-plugin to 2.22.2
Travis fixes
Update README and add LICENSE.txt
Noah wrote in #280:
> Maybe we should just drop the test. The assumption when we wrote the
> test was that a race condition would not be so frequent in practice.
> We've seen that under the contrived conditions created by the test
> case, it is frequent. But that's ok
The arbitary value `25` was used but in prace it's quite possible
for more than 25 writing threads to have checked the cookie count
limit before adding their cookie. In practice we see Travis failing
on this test quite often, every few builds in fact.
I think using `threads.length` (i.e. 200) should cover the worst
case possibility where every thread reads a stale count and tries
to add their cookie.
Fixes#274
Jetty 9.4.12+ is required for TLS to work correctly under JDK11
(due to SSL handshake failures). In order to upgrade
Jetty we also need to upgrade Restlet.
There is one intentional change in behaviour to simplify
upgrading. We remove a workaround for an old [webkit bug]
where the browser claimed to prefer application/xml. The
bug was fixed in 2011.
[webkit bug]: https://bugs.webkit.org/show_bug.cgi?id=27267
Summary of Jetty API changes:
- package names changed such as org.mortbay -> org.eclipse
- SocketConnector and SslSocketConnector merged to ServerConnector
- HashUserRealm split into UserStore and HashLoginService
- SecurityHandler -> ConstraintSecurityHandler
Summary of Restlet API changes:
- some classes have moved package (Request, Response, Router etc)
- ServerResource replaces Resource
- represent(), acceptRepresentation() renamed to get(), post()
- constructors were replaced by an init() method
- setModifiable() was removed
- getRequest().getEntityAsForm() -> new Form(entity)
- Guard -> ChallengeAuthenticator
Fixes#275
Fixes#268 ssl handshake_failure. Support for jdk11 was added in jetty
9.4.14 but we may as well bump to the latest stable version.
In Jetty 9 it appears the request is logged after the response is sent.
Thus it was racing with the assertions that check the client IP. So to
fix this rather stashing the 'lastRequest' we just make the server
echo the client's IP in a response header.
Some other minor tweaks were needed due to changes in Jetty behaviour:
- We stop checking the length of the raw response. It doesn't tell us
anything and easily varies.
- Jetty now generates Set-Cookie with a space after the ;
- Jetty now lowercases the word "basic" in WWW-Authenticate header
- testLaxUrlEncoding(): Jetty now rejects bad paths with a 400 error so
we disable the response checks. The actual request line is
still checked which is the important thing.
Note: This patch does not affect the version of jetty used by the
Heritrix admin console. That will be tackled separately.
* warc-writer-chain:
write youtube-dl json to the warc
rename method so as not to conflict with Processor
make WARCRecordBuilder an interface
revisits only for http and ftp
accept any BaseWARCWriterProcessor
use WARCWriterChainProcessor for these tests
default chain in code
oops, handle https too
same test as for WARCWriterProcessor
configurable warc writer chain
* ydl:
whoops, better spawn the thread first
read stderr and stdout in separate threads...
do not drop any `CrawlURI.data` between processing
nothing private ever
quiet org.mortbay.log (jetty?) logging
making everything work
ExtractorYoutubeDL
Update changelog for 3.4.0-20190418
[maven-release-plugin] prepare for next development iteration
[maven-release-plugin] prepare release 3.4.0-20190418
set of frontier management changes to support CrawlHQ module
Remove suffix from warcWriter since it is no longer used.
Add CHANGELOG; address #233.
Without this change (or other measures), we sometimes get nulls in the
ExtractorYoutubeDL log for containing page information. We'll run this
on QA for a while and see if it causes any problems.
nlevitt [1:59 PM]
https://github.com/internetarchive/heritrix3/blob/master/modules/src/main/java/org/archive/modules/CrawlURI.java#L878
drops some stuff from `CrawlURI.data` after processing a uri, even if it needs to be processed again
there is a list of keys that shouldn’t be dropped (`persistentKeys`), but it is final and private
so if you’re writing your own heritrix module and you want to keep some information in CrawlURI.data, it usually works, except when the url is processed more than once (like when it needs a prereq like robots.txt the first time)
in practice it seems that most data is persisted, that is, most commonly used keys are in `persistentKeys`
in a crawl with pretty standard configuration i’m mostly seeing `prerequisite-uri` dropped and occasionally `fetch-completed-time` and `fetch-began-time` being dropped
i’m highly skeptical of the value of dropping keys at all and i’m tempted to get rid of this entirely, make all the keys persistent in other words
soliciting feedback (edited)
anjackson [2:39 PM]
My immediate reaction is HARD AGREE. It looks like Really Old Code though (https://github.com/internetarchive/heritrix3/blame/7d3eff5269142c77fa4b988396153f4c29d16caa/modules/src/main/java/org/archive/modules/CrawlURI.java#L878)
so the reasons for doing so may have been lost in time.
Hm, looking at usage: https://github.com/internetarchive/heritrix3/blob/a60b2ef3875ad47f57b0c6b3c0b19f86c40a12f7/engine/src/main/java/org/archive/crawler/frontier/WorkQueueFrontier.java#L954-L955
engine/src/main/java/org/archive/crawler/frontier/WorkQueueFrontier.java:954-955
curi.processingCleanup(); // lose state that shouldn't burden
// retry
I guess there's a concern that there may be state in there that is set during a fetch and may cause problems if the same CrawlURI is deferred?
But I'm not aware of anything in the fetch chain that behaves like that.
nlevitt [3:02 PM]
oh, i missed `CrawlURI.addDataPersistentMember(String)` et al. still...
There is already a clause for this in logging.properties, but it's using
log4j. It was dumping stack traces every time the client was dubious of
heritrix's self-signed certificate.
Why do we have so many identical log4j.xml's? 🤷♂️
* master:
use constant from rethinkdb lib for default port
Update README.md
Handle missing closing paren in srcset descriptor
Teach jericho extractor srcset
Don't run srcset test against jericho, it doesn't handle it
Handle commas more compliantly when parsing srcset
Ensure we start parsing full lines, for #239.
Commas are allowed if they're in the middle of the URL. Consequently:
srcset="a,b,,c," => ["a,b,,c"]
srcset="a, b,, c," => ["a", "b", "c"]
They occur particularly commonly in data: URLs before the base64 value.
Commas are also allowed in descriptors if they are enclosed by parens:
srcset="a (b,c),d" => ["a", "d"]
Spec: https://html.spec.whatwg.org/multipage/images.html#parsing-a-srcset-attribute
* trough-dedup:
promote dirty segments at crawl finish
trough dedup!
Allow failed lookups to expire, for #234.
[maven-release-plugin] prepare for next development iteration
[maven-release-plugin] prepare release 3.4.0-20190207
As @nlevitt suggestion, a further check.
[maven-release-plugin] prepare for next development iteration
[maven-release-plugin] prepare release 3.4.0-20190205
[maven-release-plugin] prepare for next development iteration
[maven-release-plugin] prepare release 3.4.0-20190205-2
[maven-release-plugin] prepare for next development iteration
Extend to set additional property.
Use argument syntax.
Skip tests during release process (covered by CI).
Set consistent tag, and include contrib.
Wrong repo spec.
Add build profile for deployment to Maven Central.
Avoid headings being treated as lists
Clarification about APIs.
Swapped original and link to make maintenance simpler.
Tidy up markup and links.
Add synchronized statements for internetarchive/heritrix3#221.
Add checks to guard against server sending 304 in error, for #229.
do not checkpoint if crawl job has not started
DecideRuleSequence already has the optimization that I was looking for,
namely, don't bother evaluating a DecideRule if we know it won't change
the current result. For some reason PredicatedDecideRule, which is a
parent class to most of the decide rules in heritrix, didn't implement
onlyDecision(). Certain crawl configurations could see significant
performance improvement with this change.
* hbase-refactor:
use non-deprecated hbase api
Correct spelling mistakes.
Note feature only applies to forthcoming 3.3 release
Update API with note about checkpoint launching.
Default to starting anew if no checkpoints are found.
Provide a simple hook to restart from the latest checkpoint.
Move sync to after Null check.
Attempt to cache dependencies.
Simplify contrib build.
Attempt to prevent build time-outs.
Add synchronisation around host and server stats.
Avoid overdoing the RAM allocation.
Ensure all WorkQueue modification as serialised across threads.
Tests now need more RAM.
fix exception starting DecideRuleSequence logging
Fix up tests to account for new Base URI behaviour.
Only set the BaseURI if not set already.
Same fix for JerichoExtractorHTML.
Test case and fix for internetarchive/heritrix3#208.
Fix link to User Guide
Fix typos
Replace API response section with statuscode tag
Reformat the rest of the API calls
Add actions to API URLs to distinguish them
Add explanatory note about documentation
Try nicer formatting for the first API example
Format HTTP request lines in API guide
Move conventions and REST section to the end of the document
Convert html tables to rst tables
Add skeleton sphinx docs for readthedocs
Point at GitHub wiki for latest release info
Add parameter to allow even distribution for parallel queues.
Noah Levitt