Noah Levitt
2f82d43110
remove already-outdated stuff from javadoc
2015-12-14 15:24:08 -08:00
Noah Levitt
ce31cada5f
change class originally known as SeedLimitsEnforcer to
...
SourceQuotaEnforcer; make it a Processor instead of a DecideRule
(because checking quota at link scoping time doesn't work, since many
urls which would go over quota can be added to the frontier); support
quotas on any of the fields tracked by CrawledBytesHistotable
2015-12-08 13:45:25 -08:00
Noah Levitt
7929e16d6a
fix checkpointing problems with new statsBySource
2015-12-04 21:33:40 -08:00
Noah Levitt
de22096dde
SeedLimitsEnforcer (contrib) DecideRule that rejects CrawlURI if source seed byte or document limit has been reached
2015-12-03 17:50:46 -08:00
Noah Levitt
5559a5353d
SourceSeedDecideRule applies the configured decision for any URI with discovered from one of a set of seeds
2015-12-03 17:43:26 -08:00
Noah Levitt
ce4a865764
add support to StatisticsTracker to keep a CrawledBytesHistotable per source tag when trackSources is enabled; integration test for this functionality
2015-12-03 16:13:34 -08:00
Noah Levitt
89cd085e76
Merge pull request #133 from vonrosen/ARI-4565
...
Add maximum to number of cookies to store for domain to BdbCookieStore
2015-11-10 17:00:23 -08:00
Hunter Stern
17f58f4bf8
Remove uneeded logger
2015-11-06 10:43:58 -08:00
Hunter Stern
580391e0a7
Move most of addCookie functionality into AbstractCookieStore
2015-11-06 10:40:43 -08:00
Hunter Stern
3142bbd485
Revise based on feedback in pull request from nlevitt
2015-11-02 13:07:11 -08:00
Hunter Stern
44ac1030f2
Make sure to synchronize in addCookie now that domain count limits are imposed and modify unit test to also take into account domain limits
2015-10-16 17:12:42 -07:00
Hunter Stern
d9684e913b
Restrict number of cookies per domain to 50
2015-10-16 14:05:56 -07:00
Hunter Stern
5ec6c738a6
Revert "Possible fix that probably isn't needed"
...
This reverts commit c6f019a095
2015-10-13 17:21:28 -07:00
Hunter Stern
c6f019a095
Possible fix that probably isn't needed
2015-10-13 14:18:06 -07:00
Hunter
79ebb0e03c
Merge pull request #129 from nlevitt/java8keytool
...
load keytool utility main class dynamically, trying both the old and …
2015-10-06 16:09:43 -07:00
Hunter
a9c4cb42c9
Merge pull request #130 from nlevitt/stay-alive-statistics-tracker
...
catch exceptions and log error in StatisticsTracker.run(), to make su…
2015-10-06 16:09:11 -07:00
Hunter
98d41051b8
Merge pull request #131 from nlevitt/amqp-url-receiver-changes
...
set isRunning=true so that stop() gets called to avoid leaking connec…
2015-10-06 15:52:05 -07:00
Noah Levitt
af5c6c3dde
set isRunning=true so that stop() gets called to avoid leaking connections when crawls finish
2015-10-06 15:50:47 -07:00
Noah Levitt
11d4bab1ac
catch exceptions and log error in StatisticsTracker.run(), to make sure statistics events don't stop happening
2015-10-05 11:46:35 -07:00
Noah Levitt
41dac25cb6
Merge pull request #128 from ukwa/amqp-url-receiver-changes
...
AMQPUrlReceiver changes to support RabbitMQ >= 3.3
2015-09-24 16:52:04 -07:00
Noah Levitt
b7afc627ed
fix typo
2015-09-24 15:31:12 -07:00
Noah Levitt
88c7f3c3d4
Merge pull request #124 from vonrosen/ari-4371
...
Allow realm to be set by server for basic auth.
2015-09-24 10:26:49 -07:00
Andrew Jackson
e883977de6
Downgrade Throwable to Exception.
2015-09-23 23:14:11 +01:00
Noah Levitt
9aabcb3272
add license header to new class
2015-09-23 12:07:38 -07:00
Noah Levitt
12dde2b693
load keytool utility main class dynamically, trying both the old and new (java 8) class names
2015-09-23 12:06:11 -07:00
Andrew Jackson
d1ea01c75d
Updated with additional optional fields and separated isConsuming from isRunning.
2015-09-19 21:39:19 +01:00
Andrew Jackson
cd3fb8d717
Configurable and 3.3 compatible AMQP receiver for #1
2015-09-18 15:48:09 +01:00
Noah Levitt
490a257cfe
Merge pull request #126 from caofangkun/her-001
...
'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing
2015-08-25 10:40:59 -07:00
caofangkun
6cf0d4a0c5
'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing
2015-08-24 17:56:56 +08:00
Hunter Stern
f097939a1c
Don't call setRealm so other realms that can be used.
2015-08-14 13:01:54 -07:00
Noah Levitt
f5a4ecbb03
Merge pull request #125 from ldko/AMQP-declarations-fix
...
Amqp declarations fix
2015-08-03 14:26:38 -07:00
Lauren Ko
c94f515956
Sets the default clientId to the same as on Receiver side
2015-07-31 15:45:49 -05:00
Lauren Ko
592c70fa3d
declare AMQP exchange in AMQPUrlReceiver to prevent exception when exchange not yet created
2015-07-30 17:34:48 -05:00
Hunter Stern
4e22f1f809
Allow realm to be set by server for basic auth.
2015-07-07 13:06:29 -07:00
Hunter
7afe03540a
Merge pull request #120 from nlevitt/extractorjs-twodots
...
for ARI-4267 accept possible uris with two dots in the filename part if ...
2015-05-04 15:45:16 -07:00
Hunter
eb78de6963
Merge pull request #121 from nlevitt/knowledgable-extractorjs
...
new contrib module KnowledgableExtractorJS, a subclass of ExtractorJS th...
2015-05-04 15:41:21 -07:00
Noah Levitt
fea49f0e30
expand stubbed javadoc comment
2015-05-04 15:16:11 -07:00
Noah Levitt
0f43a3dae6
add license header
2015-05-04 15:08:52 -07:00
Noah Levitt
e9fcb34508
new contrib module KnowledgableExtractorJS, a subclass of ExtractorJS that has some customized behavior for specific kinds of web pages; initially, the one special behavior it has is for drupal generated pages
2015-04-21 01:19:33 -07:00
Noah Levitt
1027f854ec
for ARI-4267 accept possible uris with two dots in the filename part if they end with a known good extension (e.g. "foo.min.js")
2015-04-20 17:08:47 -07:00
Noah Levitt
0923a7ca18
Merge pull request #119 from adam-miller/HER-2082
...
Fix for HER-2082
2015-03-26 16:30:49 -07:00
Adam Miller
458cbb51b0
Fix for HER-2082
...
https://webarchive.jira.com/browse/HER-2082
WARCWriterProcessor.java
Add last hop character and link context to outlinks in metadata records
2015-03-26 16:19:09 -07:00
Noah Levitt
bcae99daa8
Merge pull request #118 from kris-sigur/HER-2080
...
Fix for ServerNotModified WARC revisit records incorrectly record WARC-Payload-Digest
2015-03-23 11:16:36 -07:00
Kristinn Sigurðsson
f5107f60b8
Add annotation for server-not-modified duplicates.
2015-03-20 13:46:20 +00:00
Kristinn Sigurðsson
dff219e884
Harmonizing whitespace use
2015-03-13 14:32:23 +00:00
Kristinn Sigurðsson
843f6468c4
Use digest from previous capture when encountering 304s.
...
The previous capture may also have been a 304 but there must be an
original 200 capture at the start. This assumption may fail when using
an index generated by a version of Heritrix without this fix. But even
then you should be no worse off than before.
2015-03-13 14:30:39 +00:00
Kristinn Sigurðsson
3ffe5b8b4f
Stop automatically writing WARC-Payload-Digest for revisit records.
...
Refer that functionality to the relevant revisit profile classes.
2015-03-11 13:52:31 +00:00
vonrosen
b5da93d04b
Merge pull request #117 from nlevitt/ari-4222
...
avoid java.lang.NullPointerException at org.archive.modules.writer.Write...
2015-02-26 10:04:34 -08:00
Noah Levitt
a0c3fcdae0
avoid java.lang.NullPointerException at org.archive.modules.writer.WriterPoolProcessor.copyForwardWriteTagIfDupe(WriterPoolProcessor.java:504); this exception has not always been possible, but is possible now because IdenticalDigestDecideRule.hasIdenticalDigest(curi) can return true even if there is no curi.getFetchHistory(), since uri-agnostic dedup doesn't use fetchHistory
2015-02-23 14:09:25 -08:00
Adam Miller
1b6d78bf7a
Merge pull request #116 from nlevitt/log4j-breaks-tests
...
make sure log4j is configured when running unit tests, to avoid log4j er...
2015-02-23 11:59:50 -08:00
Noah Levitt