Noah Levitt
22336fb71b
fix stats in unusual case of "failed" fetch with response > 0 (only case where this can happen currently is if basic auth is configured for a url, but fails and url returns "401 Unauthorized")
2016-01-12 17:34:27 -08:00
Noah Levitt
7a8a614804
avoid exception in case applyToSubdomains is enabled and configured host is an ip address
2015-12-15 11:06:33 -08:00
Noah Levitt
a95519543d
avoid exception in case applyToSubdomains is enabled and CrawlURI host is a nonstandard host like "dns:" in HostQuotaEnforcer
2015-12-15 11:02:38 -08:00
Noah Levitt
e3e6bfeb11
new HostQuotaEnforcer option applyToSubdomains, to additionally apply the quotas (separately) to each subdomain of the configured host
2015-12-14 18:17:44 -08:00
Noah Levitt
6f9c235fd0
fix javadoc
2015-12-14 15:23:16 -08:00
Noah Levitt
013029d9ad
fix annotation copy-paste-o
2015-12-14 15:22:51 -08:00
Noah Levitt
99c5af0cf2
comment out currently failing test (needs webarchive-commons pull request merged)
2015-12-14 10:46:50 -08:00
Noah Levitt
2a56886c01
add license header
2015-12-14 10:46:14 -08:00
Noah Levitt
8deead15ec
new class to enforce quotas on a host
2015-12-11 17:02:32 -08:00
Noah Levitt
f1173789f7
fix test now that ContentDigestHistoryTest.shouldProcess() is smarter
2015-12-11 16:13:17 -08:00
Noah Levitt
c05678e296
CrawlLimitEnforcer support for maxWarcNovelBytes and maxWarcNovelUrls, with tests
2015-12-11 16:08:47 -08:00
Noah Levitt
dfc1dd940e
tally warc novel content bytes and warc novel urls in CrawledBytesHistotable, with tests
2015-12-11 12:49:02 -08:00
Noah Levitt
c20a06ec27
Merge branch 'seed-limits' into novel-quotas -- so I can add tests to StatisticsSelfTest.java in this branch
...
* seed-limits:
change class originally known as SeedLimitsEnforcer to SourceQuotaEnforcer; make it a Processor instead of a DecideRule (because checking quota at link scoping time doesn't work, since many urls which would go over quota can be added to the frontier); support quotas on any of the fields tracked by CrawledBytesHistotable
fix checkpointing problems with new statsBySource
SeedLimitsEnforcer (contrib) DecideRule that rejects CrawlURI if source seed byte or document limit has been reached
SourceSeedDecideRule applies the configured decision for any URI with discovered from one of a set of seeds
add support to StatisticsTracker to keep a CrawledBytesHistotable per source tag when trackSources is enabled; integration test for this functionality
2015-12-11 11:35:11 -08:00
Noah Levitt
87f70c3e6a
WARCLimitEnforcer to enforce limits (by stopping the crawl job) on any of the stats kept by WARCWriterProcessor
2015-12-11 11:26:11 -08:00
Noah Levitt
294cfa3524
limited test of warc writer stats
2015-12-11 11:22:03 -08:00
Noah Levitt
9ce12d2dbe
add warc stats to CrawlURI.getData() dynamic attributes list after writing records; new method WARCWriterProcessor.getStats() to expose cumulative stats
2015-12-11 11:21:05 -08:00
Noah Levitt
35a7954882
refactor FetchStats to be a subclass of CrawledBytesHistotable, to remove redundant code
2015-12-11 11:05:44 -08:00
Noah Levitt
566c1a600d
remove warning which can be misleading if crawl is configured to write only certain urls to warcs
2015-12-11 11:01:26 -08:00
Noah Levitt
cce07084e5
CrawlLimitEnforcer support for maxNovelBytes and maxNovelUrls, with tests
2015-12-09 17:15:34 -08:00
Noah Levitt
501efe4a7b
QuotaEnforcer support for quotas on novelBytes and novelUrls, with tests
2015-12-09 15:54:36 -08:00
Noah Levitt
ce31cada5f
change class originally known as SeedLimitsEnforcer to
...
SourceQuotaEnforcer; make it a Processor instead of a DecideRule
(because checking quota at link scoping time doesn't work, since many
urls which would go over quota can be added to the frontier); support
quotas on any of the fields tracked by CrawledBytesHistotable
2015-12-08 13:45:25 -08:00
Noah Levitt
7929e16d6a
fix checkpointing problems with new statsBySource
2015-12-04 21:33:40 -08:00
Noah Levitt
de22096dde
SeedLimitsEnforcer (contrib) DecideRule that rejects CrawlURI if source seed byte or document limit has been reached
2015-12-03 17:50:46 -08:00
Noah Levitt
5559a5353d
SourceSeedDecideRule applies the configured decision for any URI with discovered from one of a set of seeds
2015-12-03 17:43:26 -08:00
Noah Levitt
ce4a865764
add support to StatisticsTracker to keep a CrawledBytesHistotable per source tag when trackSources is enabled; integration test for this functionality
2015-12-03 16:13:34 -08:00
Noah Levitt
89cd085e76
Merge pull request #133 from vonrosen/ARI-4565
...
Add maximum to number of cookies to store for domain to BdbCookieStore
2015-11-10 17:00:23 -08:00
Hunter Stern
17f58f4bf8
Remove uneeded logger
2015-11-06 10:43:58 -08:00
Hunter Stern
580391e0a7
Move most of addCookie functionality into AbstractCookieStore
2015-11-06 10:40:43 -08:00
Hunter Stern
3142bbd485
Revise based on feedback in pull request from nlevitt
2015-11-02 13:07:11 -08:00
Hunter Stern
44ac1030f2
Make sure to synchronize in addCookie now that domain count limits are imposed and modify unit test to also take into account domain limits
2015-10-16 17:12:42 -07:00
Hunter Stern
d9684e913b
Restrict number of cookies per domain to 50
2015-10-16 14:05:56 -07:00
Hunter Stern
5ec6c738a6
Revert "Possible fix that probably isn't needed"
...
This reverts commit c6f019a095
2015-10-13 17:21:28 -07:00
Hunter Stern
c6f019a095
Possible fix that probably isn't needed
2015-10-13 14:18:06 -07:00
Hunter
79ebb0e03c
Merge pull request #129 from nlevitt/java8keytool
...
load keytool utility main class dynamically, trying both the old and …
2015-10-06 16:09:43 -07:00
Hunter
a9c4cb42c9
Merge pull request #130 from nlevitt/stay-alive-statistics-tracker
...
catch exceptions and log error in StatisticsTracker.run(), to make su…
2015-10-06 16:09:11 -07:00
Hunter
98d41051b8
Merge pull request #131 from nlevitt/amqp-url-receiver-changes
...
set isRunning=true so that stop() gets called to avoid leaking connec…
2015-10-06 15:52:05 -07:00
Noah Levitt
af5c6c3dde
set isRunning=true so that stop() gets called to avoid leaking connections when crawls finish
2015-10-06 15:50:47 -07:00
Noah Levitt
11d4bab1ac
catch exceptions and log error in StatisticsTracker.run(), to make sure statistics events don't stop happening
2015-10-05 11:46:35 -07:00
Noah Levitt
41dac25cb6
Merge pull request #128 from ukwa/amqp-url-receiver-changes
...
AMQPUrlReceiver changes to support RabbitMQ >= 3.3
2015-09-24 16:52:04 -07:00
Noah Levitt
b7afc627ed
fix typo
2015-09-24 15:31:12 -07:00
Noah Levitt
88c7f3c3d4
Merge pull request #124 from vonrosen/ari-4371
...
Allow realm to be set by server for basic auth.
2015-09-24 10:26:49 -07:00
Andrew Jackson
e883977de6
Downgrade Throwable to Exception.
2015-09-23 23:14:11 +01:00
Noah Levitt
9aabcb3272
add license header to new class
2015-09-23 12:07:38 -07:00
Noah Levitt
12dde2b693
load keytool utility main class dynamically, trying both the old and new (java 8) class names
2015-09-23 12:06:11 -07:00
Andrew Jackson
d1ea01c75d
Updated with additional optional fields and separated isConsuming from isRunning.
2015-09-19 21:39:19 +01:00
Andrew Jackson
cd3fb8d717
Configurable and 3.3 compatible AMQP receiver for #1
2015-09-18 15:48:09 +01:00
Noah Levitt
490a257cfe
Merge pull request #126 from caofangkun/her-001
...
'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing
2015-08-25 10:40:59 -07:00
caofangkun
6cf0d4a0c5
'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing
2015-08-24 17:56:56 +08:00
Hunter Stern
f097939a1c
Don't call setRealm so other realms that can be used.
2015-08-14 13:01:54 -07:00
Noah Levitt
f5a4ecbb03
Merge pull request #125 from ldko/AMQP-declarations-fix
...
Amqp declarations fix
2015-08-03 14:26:38 -07:00
Noah Levitt