Noah Levitt
8deead15ec
new class to enforce quotas on a host
2015-12-11 17:02:32 -08:00
Noah Levitt
f1173789f7
fix test now that ContentDigestHistoryTest.shouldProcess() is smarter
2015-12-11 16:13:17 -08:00
Noah Levitt
c05678e296
CrawlLimitEnforcer support for maxWarcNovelBytes and maxWarcNovelUrls, with tests
2015-12-11 16:08:47 -08:00
Noah Levitt
dfc1dd940e
tally warc novel content bytes and warc novel urls in CrawledBytesHistotable, with tests
2015-12-11 12:49:02 -08:00
Noah Levitt
c20a06ec27
Merge branch 'seed-limits' into novel-quotas -- so I can add tests to StatisticsSelfTest.java in this branch
...
* seed-limits:
change class originally known as SeedLimitsEnforcer to SourceQuotaEnforcer; make it a Processor instead of a DecideRule (because checking quota at link scoping time doesn't work, since many urls which would go over quota can be added to the frontier); support quotas on any of the fields tracked by CrawledBytesHistotable
fix checkpointing problems with new statsBySource
SeedLimitsEnforcer (contrib) DecideRule that rejects CrawlURI if source seed byte or document limit has been reached
SourceSeedDecideRule applies the configured decision for any URI with discovered from one of a set of seeds
add support to StatisticsTracker to keep a CrawledBytesHistotable per source tag when trackSources is enabled; integration test for this functionality
2015-12-11 11:35:11 -08:00
Noah Levitt
87f70c3e6a
WARCLimitEnforcer to enforce limits (by stopping the crawl job) on any of the stats kept by WARCWriterProcessor
2015-12-11 11:26:11 -08:00
Noah Levitt
294cfa3524
limited test of warc writer stats
2015-12-11 11:22:03 -08:00
Noah Levitt
9ce12d2dbe
add warc stats to CrawlURI.getData() dynamic attributes list after writing records; new method WARCWriterProcessor.getStats() to expose cumulative stats
2015-12-11 11:21:05 -08:00
Noah Levitt
35a7954882
refactor FetchStats to be a subclass of CrawledBytesHistotable, to remove redundant code
2015-12-11 11:05:44 -08:00
Noah Levitt
566c1a600d
remove warning which can be misleading if crawl is configured to write only certain urls to warcs
2015-12-11 11:01:26 -08:00
Noah Levitt
cce07084e5
CrawlLimitEnforcer support for maxNovelBytes and maxNovelUrls, with tests
2015-12-09 17:15:34 -08:00
Noah Levitt
501efe4a7b
QuotaEnforcer support for quotas on novelBytes and novelUrls, with tests
2015-12-09 15:54:36 -08:00
Noah Levitt
ce31cada5f
change class originally known as SeedLimitsEnforcer to
...
SourceQuotaEnforcer; make it a Processor instead of a DecideRule
(because checking quota at link scoping time doesn't work, since many
urls which would go over quota can be added to the frontier); support
quotas on any of the fields tracked by CrawledBytesHistotable
2015-12-08 13:45:25 -08:00
Noah Levitt
7929e16d6a
fix checkpointing problems with new statsBySource
2015-12-04 21:33:40 -08:00
Noah Levitt
de22096dde
SeedLimitsEnforcer (contrib) DecideRule that rejects CrawlURI if source seed byte or document limit has been reached
2015-12-03 17:50:46 -08:00
Noah Levitt
5559a5353d
SourceSeedDecideRule applies the configured decision for any URI with discovered from one of a set of seeds
2015-12-03 17:43:26 -08:00
Noah Levitt
ce4a865764
add support to StatisticsTracker to keep a CrawledBytesHistotable per source tag when trackSources is enabled; integration test for this functionality
2015-12-03 16:13:34 -08:00
Noah Levitt
89cd085e76
Merge pull request #133 from vonrosen/ARI-4565
...
Add maximum to number of cookies to store for domain to BdbCookieStore
2015-11-10 17:00:23 -08:00
Hunter Stern
17f58f4bf8
Remove uneeded logger
2015-11-06 10:43:58 -08:00
Hunter Stern
580391e0a7
Move most of addCookie functionality into AbstractCookieStore
2015-11-06 10:40:43 -08:00
Hunter Stern
3142bbd485
Revise based on feedback in pull request from nlevitt
2015-11-02 13:07:11 -08:00
Hunter Stern
44ac1030f2
Make sure to synchronize in addCookie now that domain count limits are imposed and modify unit test to also take into account domain limits
2015-10-16 17:12:42 -07:00
Hunter Stern
d9684e913b
Restrict number of cookies per domain to 50
2015-10-16 14:05:56 -07:00
Hunter Stern
5ec6c738a6
Revert "Possible fix that probably isn't needed"
...
This reverts commit c6f019a095
2015-10-13 17:21:28 -07:00
Hunter Stern
c6f019a095
Possible fix that probably isn't needed
2015-10-13 14:18:06 -07:00
Hunter
79ebb0e03c
Merge pull request #129 from nlevitt/java8keytool
...
load keytool utility main class dynamically, trying both the old and …
2015-10-06 16:09:43 -07:00
Hunter
a9c4cb42c9
Merge pull request #130 from nlevitt/stay-alive-statistics-tracker
...
catch exceptions and log error in StatisticsTracker.run(), to make su…
2015-10-06 16:09:11 -07:00
Hunter
98d41051b8
Merge pull request #131 from nlevitt/amqp-url-receiver-changes
...
set isRunning=true so that stop() gets called to avoid leaking connec…
2015-10-06 15:52:05 -07:00
Noah Levitt
af5c6c3dde
set isRunning=true so that stop() gets called to avoid leaking connections when crawls finish
2015-10-06 15:50:47 -07:00
Noah Levitt
11d4bab1ac
catch exceptions and log error in StatisticsTracker.run(), to make sure statistics events don't stop happening
2015-10-05 11:46:35 -07:00
Noah Levitt
41dac25cb6
Merge pull request #128 from ukwa/amqp-url-receiver-changes
...
AMQPUrlReceiver changes to support RabbitMQ >= 3.3
2015-09-24 16:52:04 -07:00
Noah Levitt
b7afc627ed
fix typo
2015-09-24 15:31:12 -07:00
Noah Levitt
88c7f3c3d4
Merge pull request #124 from vonrosen/ari-4371
...
Allow realm to be set by server for basic auth.
2015-09-24 10:26:49 -07:00
Andrew Jackson
e883977de6
Downgrade Throwable to Exception.
2015-09-23 23:14:11 +01:00
Noah Levitt
9aabcb3272
add license header to new class
2015-09-23 12:07:38 -07:00
Noah Levitt
12dde2b693
load keytool utility main class dynamically, trying both the old and new (java 8) class names
2015-09-23 12:06:11 -07:00
Andrew Jackson
d1ea01c75d
Updated with additional optional fields and separated isConsuming from isRunning.
2015-09-19 21:39:19 +01:00
Andrew Jackson
cd3fb8d717
Configurable and 3.3 compatible AMQP receiver for #1
2015-09-18 15:48:09 +01:00
Noah Levitt
490a257cfe
Merge pull request #126 from caofangkun/her-001
...
'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing
2015-08-25 10:40:59 -07:00
caofangkun
6cf0d4a0c5
'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing
2015-08-24 17:56:56 +08:00
Hunter Stern
f097939a1c
Don't call setRealm so other realms that can be used.
2015-08-14 13:01:54 -07:00
Noah Levitt
f5a4ecbb03
Merge pull request #125 from ldko/AMQP-declarations-fix
...
Amqp declarations fix
2015-08-03 14:26:38 -07:00
Lauren Ko
c94f515956
Sets the default clientId to the same as on Receiver side
2015-07-31 15:45:49 -05:00
Lauren Ko
592c70fa3d
declare AMQP exchange in AMQPUrlReceiver to prevent exception when exchange not yet created
2015-07-30 17:34:48 -05:00
Hunter Stern
4e22f1f809
Allow realm to be set by server for basic auth.
2015-07-07 13:06:29 -07:00
Hunter
7afe03540a
Merge pull request #120 from nlevitt/extractorjs-twodots
...
for ARI-4267 accept possible uris with two dots in the filename part if ...
2015-05-04 15:45:16 -07:00
Hunter
eb78de6963
Merge pull request #121 from nlevitt/knowledgable-extractorjs
...
new contrib module KnowledgableExtractorJS, a subclass of ExtractorJS th...
2015-05-04 15:41:21 -07:00
Noah Levitt
fea49f0e30
expand stubbed javadoc comment
2015-05-04 15:16:11 -07:00
Noah Levitt
0f43a3dae6
add license header
2015-05-04 15:08:52 -07:00
Noah Levitt
e9fcb34508
new contrib module KnowledgableExtractorJS, a subclass of ExtractorJS that has some customized behavior for specific kinds of web pages; initially, the one special behavior it has is for drupal generated pages
2015-04-21 01:19:33 -07:00
Noah Levitt