Commit Graph
92 Commits
Author SHA1 Message Date
Alex Osborne 1d37cbc237 StatisticsSelfTest: Update for new jetty version
Jetty error messages are included in the selftest crawl and now have
slightly different lengths.
2024-10-17 11:58:13 +09:00
Alex Osborne 09e3ff3568 Everywhere: Fix dangling JavaDoc comments
JavaDoc comments need to be directly above a class, method or field
declaration to be used by IDEs and generated documentation. We had some
doc comments that were attached to instance initialization blocks and
thus were being ignored. This invalid doc comment positioning was
presumably an accidental consequence of the conversion of fields to
KeyedProperties.

This change moves most of the dangling doc comments to setter methods
where it can be seen by tools. There were a couple of dangling doc
comments above package statements. These are moved above the class
declaration or removed entirely when empty.
2021-07-12 21:38:48 +09:00
Andy Jackson a2c65f4618 Update dependencies for security issues, including changes to tests. 2021-05-26 13:39:30 +01:00
Andy Jackson e0c82f70e9 Merge pull request #289 from netarchivesuite/bdb-frontier-access
Bdb frontier access
2020-03-04 13:41:27 +00:00
Tim Hennekey 33458f1518 Fix assertions
By using assertEquals and seting the expected and actual values, the
failure messages become a bit more useful.
2020-01-22 17:13:58 -05:00
jkafader 3a8b58998c Merge pull request #285 from nlevitt/warc-writer-chain
Warc writer chain
2020-01-13 16:39:42 -08:00
Nicholas Clarke c611c46c8a Merged frontier-management with upgrade to bdb 7 2019-11-15 10:33:13 +01:00
Alex Osborne 6c2afc36b0 Update statistics assertions as Jetty 9 produces different headers 2019-08-06 17:13:02 +09:00
Alex Osborne 4012a5f4dd Fix 'No session data store configured' test error 2019-08-06 16:56:10 +09:00
Alex Osborne 0782fb0481 JDK11: upgrade engine to jetty 9.4.19 and restlet 2.4.0
Jetty 9.4.12+ is required for TLS to work correctly under JDK11
(due to SSL handshake failures). In order to upgrade
Jetty we also need to upgrade Restlet.

There is one intentional change in behaviour to simplify
upgrading. We remove a workaround for an old [webkit bug]
where the browser claimed to prefer application/xml. The
bug was fixed in 2011.

[webkit bug]: https://bugs.webkit.org/show_bug.cgi?id=27267

Summary of Jetty API changes:
- package names changed such as org.mortbay -> org.eclipse
- SocketConnector and SslSocketConnector merged to ServerConnector
- HashUserRealm split into UserStore and HashLoginService
- SecurityHandler -> ConstraintSecurityHandler

Summary of Restlet API changes:
- some classes have moved package (Request, Response, Router etc)
- ServerResource replaces Resource
- represent(), acceptRepresentation() renamed to get(), post()
- constructors were replaced by an init() method
- setModifiable() was removed
- getRequest().getEntityAsForm() -> new Form(entity)
- Guard -> ChallengeAuthenticator

Fixes #275
2019-08-06 15:16:37 +09:00
Noah Levitt 7c31b0752a use WARCWriterChainProcessor for these tests 2019-06-12 12:56:49 -07:00
Edward Betts aa705bef2e Correct spelling mistakes. 2018-11-04 19:49:37 +09:00
Andrew Jackson 8384a59b47 Fix up tests to account for new Base URI behaviour. 2018-07-06 15:15:53 +01:00
Andrew Jackson b0594ac0f4 Allow JavaDNS to be disabled and use that to avoid issues in StatisticsSelfTest. 2017-10-12 21:29:34 +01:00
Andrew Jackson 09ee95f37b Allow results with or without DNS lookup of localhost. 2017-10-11 23:30:17 +01:00
Andrew Jackson 401e27fc64 Adapt test to cope with variation in how localhost gets resolved. 2017-10-09 21:50:05 +01:00
Noah Levitt 3892c17b56 ughhh appease java 8 javadoc rules 2017-06-15 17:29:22 -07:00
Noah Levitt ca3b65ed03 exclude dns from stats tests because the size of the dns records can vary depending on the environment 2016-06-06 15:43:02 -05:00
Andrew Jackson 9145fedd40 Fix unexpected change in count of novel bytes in one selftest. 2016-04-02 20:57:05 +01:00
Noah Levitt 9e4c7f0024 Merge branch 'master' into novel-quotas
* master:
  license header
  check that sourceTag of CrawlURI actually matches configured sourceTag
  remove already-outdated stuff from javadoc
  handle multiple clauses for same user agent in robots.txt
  Hook in submitted seeds properly.
  avoid spurious logging
  try very hard to start url consumer, and therefore bind the queue to the routing key, so that no messages are dropped, before crawling starts (should always work unless rabbitmq is down); some other tweaks for clarity and stability
2016-01-13 13:44:54 -08:00
Noah Levitt 65a1077425 license header 2016-01-12 14:59:45 -08:00
Noah Levitt c05678e296 CrawlLimitEnforcer support for maxWarcNovelBytes and maxWarcNovelUrls, with tests 2015-12-11 16:08:47 -08:00
Noah Levitt dfc1dd940e tally warc novel content bytes and warc novel urls in CrawledBytesHistotable, with tests 2015-12-11 12:49:02 -08:00
Noah Levitt c20a06ec27 Merge branch 'seed-limits' into novel-quotas -- so I can add tests to StatisticsSelfTest.java in this branch
* seed-limits:
  change class originally known as SeedLimitsEnforcer to SourceQuotaEnforcer; make it a Processor instead of a DecideRule (because checking quota at link scoping time doesn't work, since many urls which would go over quota can be added to the frontier); support quotas on any of the fields tracked by CrawledBytesHistotable
  fix checkpointing problems with new statsBySource
  SeedLimitsEnforcer (contrib) DecideRule that rejects CrawlURI if source seed byte or document limit has been reached
  SourceSeedDecideRule applies the configured decision for any URI with discovered from one of a set of seeds
  add support to StatisticsTracker to keep a CrawledBytesHistotable per source tag when trackSources is enabled; integration test for this functionality
2015-12-11 11:35:11 -08:00
Noah Levitt 35a7954882 refactor FetchStats to be a subclass of CrawledBytesHistotable, to remove redundant code 2015-12-11 11:05:44 -08:00
Noah Levitt cce07084e5 CrawlLimitEnforcer support for maxNovelBytes and maxNovelUrls, with tests 2015-12-09 17:15:34 -08:00
Noah Levitt 501efe4a7b QuotaEnforcer support for quotas on novelBytes and novelUrls, with tests 2015-12-09 15:54:36 -08:00
Noah Levitt ce4a865764 add support to StatisticsTracker to keep a CrawledBytesHistotable per source tag when trackSources is enabled; integration test for this functionality 2015-12-03 16:13:34 -08:00
Noah Levitt 04921073ca fix bug in CrawlURI.compare() discovered by Kenji, add unit test CrawlURITest.testOrdering() 2014-08-19 17:14:32 -07:00
Noah Levitt ee52f48be9 remove LinksScoper (deprecated since 2009) and remove recently added code only used in LinksScoper 2014-07-09 10:14:24 -07:00
Noah Levitt 20a5f31e19 ExtractorHTMLForms+FormLoginProcessor full crawl job integration test (not passing yet) 2014-03-31 18:57:33 -07:00
Noah Levitt 3a429ffa44 Merge branch 'master' into hc43
Conflicts:
	commons/pom.xml
	commons/src/main/java/org/archive/io/RecordingInputStream.java
	commons/src/main/java/org/archive/io/RecordingOutputStream.java
	commons/src/main/java/org/archive/io/ReplayInputStream.java
	dist/pom.xml
	engine/pom.xml
	modules/pom.xml
	modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
	modules/src/main/java/org/archive/modules/recrawl/FetchHistoryProcessor.java
	pom.xml
2014-01-10 17:52:03 -08:00
Noah Levitt a06c141740 rename BdbUriUniqFilter.forgetSchemeHost() to forgetAllSchemeAuthorityMatching(), improve performance 2013-08-21 11:31:05 -07:00
Noah Levitt 750a688670 Merge branch 'master' into hc43
Conflicts:
	commons/src/main/java/org/apache/commons/httpclient/HttpParser.java
	engine/pom.xml
	modules/src/main/java/org/archive/modules/CrawlURI.java
	modules/src/main/java/org/archive/modules/deciderules/MatchesStatusCodeDecideRule.java
	modules/src/main/java/org/archive/modules/deciderules/NotMatchesStatusCodeDecideRule.java
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java
2013-08-08 21:14:11 -07:00
Noah Levitt f4f3b9be38 add logging to BdbUriUniqFilter.forgetSchemeHost(), clean up some javadocs 2013-07-26 17:50:36 -07:00
Noah Levitt b9726c8d4f new method BdbUriUniqFilter.forgetSchemeHost(String schemeHost), and unit test 2013-07-25 18:51:26 -07:00
Kenji Nagahashi 358afd4486 revised scripting console xml response fix based on comments.
removed scripting console actions from ScriptModel into ScriptingConsole
class (resurrection of ScriptExec).
added basic ScriptingConsoleTest.
2013-06-10 14:17:27 -07:00
Kenji Nagahashi 3a54600a79 ADD unit test for XmlMarshaller 2013-05-29 12:03:49 -07:00
Noah Levitt 25234898ef Somewhat redundant form auth integration test restored, passes 2012-12-30 22:35:28 -08:00
Noah Levitt 4496c5d755 * FetchHTTPTest.java
move to modules/src/test
    testFormAuth() - remove, there is FormAuthSelfTest.java, and this was the test that would fail in modules/
    testHttpBindAddress() - incorporate ed65493db0 - alternate testing of multiple local binds more likely to work more places
2012-11-30 17:57:36 -08:00
Noah Levitt 7965a27b1d Fix infinite loop bug on server returning no response
* RecordingSocketInputBuffer.java
    readLine(CharArrayBuffer) - return -1 (not 0) on end of stream
* RecordingHttpClient.java
    setHttpRequestRetryHandler() - never retry (may want to reevaluate later, old fetcher did retry sometimes)
* FetchHTTPTest.java
    testNoResponse()
2012-09-13 19:10:11 -07:00
Noah Levitt 9699d6a6ed get rid of AbstractFetchHTTP 2012-09-08 01:17:32 -07:00
Noah Levitt 33df8faaf0 remove LegacyFetchHTTP and its classes 2012-09-08 00:49:30 -07:00
Noah Levitt 71761a72a7 rename FetchHTTP2 to FetchHTTP 2012-09-08 00:28:06 -07:00
Noah Levitt 917986454b rename FetchHTTP to LegacyFetchHTTP 2012-09-08 00:25:14 -07:00
Noah Levitt 24e813c858 test chunked transfer encoding 2012-09-07 13:28:41 -07:00
Noah Levitt 09f0e53119 testHttp11 2012-09-07 11:32:38 -07:00
Noah Levitt 9a0191a84c ssl trust level support for FetchHTTP2! another thing that doesn't even work properly in old FetchHTTP 2012-09-07 02:47:54 -07:00
Noah Levitt 0764096eb0 skip slow tests for old FetchHTTP for now 2012-09-07 02:42:13 -07:00
Noah Levitt f2f0ca59f6 embedded jetty https server 2012-09-06 19:28:07 -07:00