Hunter Stern
afa4f6535b
If multiple username inputs in login form, favor the one with the word 'login' in it.
2016-01-14 12:05:38 -08:00
Hunter
eeeea6f0c4
Merge pull request #122 from nlevitt/form-login-checked-boxes
...
only submit checkbox and radio button form fields if they are on by d…
2016-01-14 08:00:49 -08:00
Adam Miller
bf2a887688
Merge pull request #138 from nlevitt/novel-quotas
...
crawl level and host level limits on *novel* (not deduplicated) bytes and urls
2016-01-13 16:25:24 -08:00
Noah Levitt
9e4c7f0024
Merge branch 'master' into novel-quotas
...
* master:
license header
check that sourceTag of CrawlURI actually matches configured sourceTag
remove already-outdated stuff from javadoc
handle multiple clauses for same user agent in robots.txt
Hook in submitted seeds properly.
avoid spurious logging
try very hard to start url consumer, and therefore bind the queue to the routing key, so that no messages are dropped, before crawling starts (should always work unless rabbitmq is down); some other tweaks for clarity and stability
2016-01-13 13:44:54 -08:00
Adam Miller
b82896927a
Merge pull request #132 from nlevitt/amqp-url-receiver-changes
...
try very hard to start url consumer, and therefore bind the queue to …
2016-01-13 13:35:43 -08:00
Noah Levitt
df5748d1f1
Merge pull request #136 from ukwa/amqp-declare-recieved-seeds
...
Register seeds send in via AMQP
2016-01-13 13:27:43 -08:00
Noah Levitt
22336fb71b
fix stats in unusual case of "failed" fetch with response > 0 (only case where this can happen currently is if basic auth is configured for a url, but fails and url returns "401 Unauthorized")
2016-01-12 17:34:27 -08:00
Adam Miller
3dbf754d56
Merge pull request #137 from nlevitt/seed-limits
...
SourceSeedDecideRule, SeedLimitsEnforcer
2016-01-12 16:39:17 -08:00
Noah Levitt
65a1077425
license header
2016-01-12 14:59:45 -08:00
Neil Minton
32dd07375d
Merge pull request #139 from nlevitt/robots-sep-sections
...
handle multiple clauses for same user agent in robots.txt
2016-01-11 18:39:25 -06:00
Noah Levitt
8918ce99c6
check that sourceTag of CrawlURI actually matches configured sourceTag
2015-12-15 11:24:53 -08:00
Noah Levitt
7a8a614804
avoid exception in case applyToSubdomains is enabled and configured host is an ip address
2015-12-15 11:06:33 -08:00
Noah Levitt
a95519543d
avoid exception in case applyToSubdomains is enabled and CrawlURI host is a nonstandard host like "dns:" in HostQuotaEnforcer
2015-12-15 11:02:38 -08:00
Noah Levitt
e3e6bfeb11
new HostQuotaEnforcer option applyToSubdomains, to additionally apply the quotas (separately) to each subdomain of the configured host
2015-12-14 18:17:44 -08:00
Noah Levitt
2f82d43110
remove already-outdated stuff from javadoc
2015-12-14 15:24:08 -08:00
Noah Levitt
6f9c235fd0
fix javadoc
2015-12-14 15:23:16 -08:00
Noah Levitt
013029d9ad
fix annotation copy-paste-o
2015-12-14 15:22:51 -08:00
Noah Levitt
99c5af0cf2
comment out currently failing test (needs webarchive-commons pull request merged)
2015-12-14 10:46:50 -08:00
Noah Levitt
2a56886c01
add license header
2015-12-14 10:46:14 -08:00
Noah Levitt
8deead15ec
new class to enforce quotas on a host
2015-12-11 17:02:32 -08:00
Noah Levitt
f1173789f7
fix test now that ContentDigestHistoryTest.shouldProcess() is smarter
2015-12-11 16:13:17 -08:00
Noah Levitt
c05678e296
CrawlLimitEnforcer support for maxWarcNovelBytes and maxWarcNovelUrls, with tests
2015-12-11 16:08:47 -08:00
Noah Levitt
38931f152a
handle multiple clauses for same user agent in robots.txt
2015-12-11 15:42:16 -08:00
Noah Levitt
dfc1dd940e
tally warc novel content bytes and warc novel urls in CrawledBytesHistotable, with tests
2015-12-11 12:49:02 -08:00
Noah Levitt
c20a06ec27
Merge branch 'seed-limits' into novel-quotas -- so I can add tests to StatisticsSelfTest.java in this branch
...
* seed-limits:
change class originally known as SeedLimitsEnforcer to SourceQuotaEnforcer; make it a Processor instead of a DecideRule (because checking quota at link scoping time doesn't work, since many urls which would go over quota can be added to the frontier); support quotas on any of the fields tracked by CrawledBytesHistotable
fix checkpointing problems with new statsBySource
SeedLimitsEnforcer (contrib) DecideRule that rejects CrawlURI if source seed byte or document limit has been reached
SourceSeedDecideRule applies the configured decision for any URI with discovered from one of a set of seeds
add support to StatisticsTracker to keep a CrawledBytesHistotable per source tag when trackSources is enabled; integration test for this functionality
2015-12-11 11:35:11 -08:00
Noah Levitt
87f70c3e6a
WARCLimitEnforcer to enforce limits (by stopping the crawl job) on any of the stats kept by WARCWriterProcessor
2015-12-11 11:26:11 -08:00
Noah Levitt
294cfa3524
limited test of warc writer stats
2015-12-11 11:22:03 -08:00
Noah Levitt
9ce12d2dbe
add warc stats to CrawlURI.getData() dynamic attributes list after writing records; new method WARCWriterProcessor.getStats() to expose cumulative stats
2015-12-11 11:21:05 -08:00
Noah Levitt
35a7954882
refactor FetchStats to be a subclass of CrawledBytesHistotable, to remove redundant code
2015-12-11 11:05:44 -08:00
Noah Levitt
566c1a600d
remove warning which can be misleading if crawl is configured to write only certain urls to warcs
2015-12-11 11:01:26 -08:00
Noah Levitt
cce07084e5
CrawlLimitEnforcer support for maxNovelBytes and maxNovelUrls, with tests
2015-12-09 17:15:34 -08:00
Noah Levitt
501efe4a7b
QuotaEnforcer support for quotas on novelBytes and novelUrls, with tests
2015-12-09 15:54:36 -08:00
Noah Levitt
ce31cada5f
change class originally known as SeedLimitsEnforcer to
...
SourceQuotaEnforcer; make it a Processor instead of a DecideRule
(because checking quota at link scoping time doesn't work, since many
urls which would go over quota can be added to the frontier); support
quotas on any of the fields tracked by CrawledBytesHistotable
2015-12-08 13:45:25 -08:00
Noah Levitt
7929e16d6a
fix checkpointing problems with new statsBySource
2015-12-04 21:33:40 -08:00
Noah Levitt
de22096dde
SeedLimitsEnforcer (contrib) DecideRule that rejects CrawlURI if source seed byte or document limit has been reached
2015-12-03 17:50:46 -08:00
Noah Levitt
5559a5353d
SourceSeedDecideRule applies the configured decision for any URI with discovered from one of a set of seeds
2015-12-03 17:43:26 -08:00
Noah Levitt
ce4a865764
add support to StatisticsTracker to keep a CrawledBytesHistotable per source tag when trackSources is enabled; integration test for this functionality
2015-12-03 16:13:34 -08:00
Andrew Jackson
0f6d385b24
Hook in submitted seeds properly.
2015-11-27 20:51:35 +00:00
Noah Levitt
89cd085e76
Merge pull request #133 from vonrosen/ARI-4565
...
Add maximum to number of cookies to store for domain to BdbCookieStore
2015-11-10 17:00:23 -08:00
Hunter Stern
17f58f4bf8
Remove uneeded logger
2015-11-06 10:43:58 -08:00
Hunter Stern
580391e0a7
Move most of addCookie functionality into AbstractCookieStore
2015-11-06 10:40:43 -08:00
Hunter Stern
3142bbd485
Revise based on feedback in pull request from nlevitt
2015-11-02 13:07:11 -08:00
Hunter Stern
44ac1030f2
Make sure to synchronize in addCookie now that domain count limits are imposed and modify unit test to also take into account domain limits
2015-10-16 17:12:42 -07:00
Hunter Stern
d9684e913b
Restrict number of cookies per domain to 50
2015-10-16 14:05:56 -07:00
Hunter Stern
5ec6c738a6
Revert "Possible fix that probably isn't needed"
...
This reverts commit c6f019a095
2015-10-13 17:21:28 -07:00
Hunter Stern
c6f019a095
Possible fix that probably isn't needed
2015-10-13 14:18:06 -07:00
Noah Levitt
e37b2b42cb
avoid spurious logging
2015-10-08 18:11:09 -07:00
Noah Levitt
e4acb6c512
try very hard to start url consumer, and therefore bind the queue to the routing key, so that no messages are dropped, before crawling starts (should always work unless rabbitmq is down); some other tweaks for clarity and stability
2015-10-08 17:22:44 -07:00
Hunter
79ebb0e03c
Merge pull request #129 from nlevitt/java8keytool
...
load keytool utility main class dynamically, trying both the old and …
2015-10-06 16:09:43 -07:00
Hunter
a9c4cb42c9
Merge pull request #130 from nlevitt/stay-alive-statistics-tracker
...
catch exceptions and log error in StatisticsTracker.run(), to make su…
2015-10-06 16:09:11 -07:00
Hunter Stern