cc97f01656
Merge pull request #123 from kris-sigur/HostsReport
...
Hosts report
2016-01-15 10:33:07 -08:00
9e4c7f0024
Merge branch 'master' into novel-quotas
...
* master:
license header
check that sourceTag of CrawlURI actually matches configured sourceTag
remove already-outdated stuff from javadoc
handle multiple clauses for same user agent in robots.txt
Hook in submitted seeds properly.
avoid spurious logging
try very hard to start url consumer, and therefore bind the queue to the routing key, so that no messages are dropped, before crawling starts (should always work unless rabbitmq is down); some other tweaks for clarity and stability
2016-01-13 13:44:54 -08:00
65a1077425
license header
2016-01-12 14:59:45 -08:00
6f9c235fd0
fix javadoc
2015-12-14 15:23:16 -08:00
c05678e296
CrawlLimitEnforcer support for maxWarcNovelBytes and maxWarcNovelUrls, with tests
2015-12-11 16:08:47 -08:00
dfc1dd940e
tally warc novel content bytes and warc novel urls in CrawledBytesHistotable, with tests
2015-12-11 12:49:02 -08:00
c20a06ec27
Merge branch 'seed-limits' into novel-quotas -- so I can add tests to StatisticsSelfTest.java in this branch
...
* seed-limits:
change class originally known as SeedLimitsEnforcer to SourceQuotaEnforcer; make it a Processor instead of a DecideRule (because checking quota at link scoping time doesn't work, since many urls which would go over quota can be added to the frontier); support quotas on any of the fields tracked by CrawledBytesHistotable
fix checkpointing problems with new statsBySource
SeedLimitsEnforcer (contrib) DecideRule that rejects CrawlURI if source seed byte or document limit has been reached
SourceSeedDecideRule applies the configured decision for any URI with discovered from one of a set of seeds
add support to StatisticsTracker to keep a CrawledBytesHistotable per source tag when trackSources is enabled; integration test for this functionality
2015-12-11 11:35:11 -08:00
35a7954882
refactor FetchStats to be a subclass of CrawledBytesHistotable, to remove redundant code
2015-12-11 11:05:44 -08:00
cce07084e5
CrawlLimitEnforcer support for maxNovelBytes and maxNovelUrls, with tests
2015-12-09 17:15:34 -08:00
501efe4a7b
QuotaEnforcer support for quotas on novelBytes and novelUrls, with tests
2015-12-09 15:54:36 -08:00
7929e16d6a
fix checkpointing problems with new statsBySource
2015-12-04 21:33:40 -08:00
ce4a865764
add support to StatisticsTracker to keep a CrawledBytesHistotable per source tag when trackSources is enabled; integration test for this functionality
2015-12-03 16:13:34 -08:00
79ebb0e03c
Merge pull request #129 from nlevitt/java8keytool
...
load keytool utility main class dynamically, trying both the old and …
2015-10-06 16:09:43 -07:00
11d4bab1ac
catch exceptions and log error in StatisticsTracker.run(), to make sure statistics events don't stop happening
2015-10-05 11:46:35 -07:00
12dde2b693
load keytool utility main class dynamically, trying both the old and new (java 8) class names
2015-09-23 12:06:11 -07:00
3495851029
JavaDoc typo
2015-07-13 15:10:44 +00:00
b15fbbd13e
Enable suppression of empty hosts in report.
...
Until now hosts that have never been crawled are included in the hosts
report. This can be a very large number of hosts and is infrequently of
interest. This makes it possible to exclude the empty hosts from the
report. Default behavior remains to include them.
The option is highlighted by including it in the relevant, commented-out
section of the default profile.
2015-07-13 15:08:03 +00:00
0cf009eb4c
Use default value in sample profile.
2015-07-13 14:58:23 +00:00
96b927d9fa
Make sorting of host report configurable.
...
Can now set a maximum size for sorted reports. Longer reports are
unsorted. Default behavior is still to sort all reports.
Default profile updated to reflect this new option in a commented-out
section.
2015-07-13 14:55:26 +00:00
bf638939b0
Remove unused "top" hosts tracking.
...
This used to be used for an overview report that was available in the H1
UI. Had become vestigial code that served no purpose.
2015-07-13 14:32:34 +00:00
8009275075
make sure log4j is configured when running unit tests, to avoid log4j error messages polluting captured output and causing tests to fail, like so: https://builds.archive.org/job/Heritrix-3/org.archive.heritrix$heritrix-commons/587/testReport/org.archive.io.arc/ARCWriterTest/testLengthTooLongCompressed/
2015-02-23 11:36:29 -08:00
4ea7efdfc8
Set character set to UTF-8 when passing through files.
2015-02-10 16:07:37 +00:00
3609abe38c
Merge pull request #108 from nlevitt/aitfive-203
...
kafka crawl log feed
2014-11-14 10:29:41 -08:00
7312ad597e
so that it can appear in the crawl log, add contentSize to CrawlURI extraInfo, for http transactions (the only case in heritrix currently where contentLength and contentSize differ)
2014-11-07 13:21:42 -08:00
fb23b40e74
CandidatesProcessor.java
...
Keep CrawlURI outlinks intact for use in SeedRecord reporting.
2014-10-22 13:18:25 -07:00
5909042b84
Merge pull request #99 from nlevitt/aitfive-39
...
AMQPCrawlLogFeed, DecideRuleSequenceWithAMQPFeed, DecideRuleSequence.logExtraInfo
2014-10-16 17:24:37 +00:00
5f7dc49599
Merge pull request #95 from kngenie/patch-beanbrowser-failure
...
FIX corner-case of bean browser failing due to an exception from hashCode()
2014-10-03 19:21:27 -07:00
e5e5cd87e4
configure maven-compiler-plugin in root pom to avoid redundancy, inconsistency
2014-10-01 13:38:39 -07:00
610a1e1e97
cookie store is called BdbCookieStore again
2014-09-28 14:32:34 -07:00
a37a8de4a2
improved implementation not using TreeSet, and some initial work on integration testing of cookie handling
2014-09-28 11:59:14 -07:00
47b7caad29
initial bdb implementation; facade by topmost assigned domain, since cookies can apply to subdomains
2014-09-26 20:56:45 -07:00
f769e0636d
initial proof of concept fix for HER-2070 "cookie monster" bug
2014-09-26 20:01:52 -07:00
5821fe7b1e
FIX corner-case of bean browser failing due to an exception from hashCode().
2014-09-11 11:22:01 -07:00
2a38fa0b12
Don’t let exception from property getter fail entire bean-browser.
2014-08-25 13:36:25 -07:00
04921073ca
fix bug in CrawlURI.compare() discovered by Kenji, add unit test CrawlURITest.testOrdering()
2014-08-19 17:14:32 -07:00
591f0f9f0f
expose forAllPendingDo() as public so it can be used elsewhere
2014-08-13 13:02:28 -07:00
9a0c0656b2
move HostsReport.fixup(String) to CrawlHost.fixUpName() so it can be called elsewhere
2014-08-12 16:45:53 -07:00
51081a652c
make AbstractFrontier.needReenqueuing() public so it can be called elsewhere
2014-08-12 16:20:25 -07:00
47a62f7c6f
remove a couple of crufty lines of code
2014-08-12 16:13:59 -07:00
08e24bbdff
update guess at log line length based on real world data
2014-08-12 16:13:11 -07:00
905a98c0c2
use RevisitProfile for tallying stats
2014-07-10 21:27:50 -07:00
ee52f48be9
remove LinksScoper (deprecated since 2009) and remove recently added code only used in LinksScoper
2014-07-09 10:14:24 -07:00
f23a59a918
Merge branch 'master' into HER-2039
...
Conflicts:
modules/src/main/java/org/archive/modules/CrawlURI.java
modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
modules/src/main/java/org/archive/modules/extractor/ExtractorSWF.java
modules/src/main/java/org/archive/modules/extractor/Link.java
modules/src/main/java/org/archive/modules/writer/WARCWriterProcessor.java
2014-07-07 11:45:27 -07:00
fc43a29459
Merge pull request #68 from nlevitt/handle-apply-overlay-exc
...
handle exception determining whether to apply overlay
2014-05-28 09:59:54 +01:00
39814d1efd
avoid calling CheckpointService.hasAvailableCheckpoints() when crawl not launchable and therefore the info is not needed, because it calls isRunning() which is synchronized and blocks when checkpoint is in progress
2014-05-22 16:40:44 -07:00
963e04e50c
handle exception determining whether to apply overlay
2014-05-13 15:47:21 -07:00
ba68649ba0
Merge remote-tracking branch 'origin/master' into cosmetics
2014-05-07 10:02:06 -07:00
b7f8a0669f
Merge branch 'cosmetics' of github.com:nlevitt/heritrix3 into cosmetics
2014-05-07 10:02:00 -07:00
33e782e2fd
oops, make "exit java process" button work again
2014-05-07 10:01:47 -07:00
d777c198bd
HER-2065 - in activateInactiveQueue() close synchronized block earlier, and move updateHighestWaiting() up to keep it inside the block; now the call to deactivateQueue() is not in the synchronized block, so the deadlock should be avoided (fix suggested by kenji)
2014-04-24 17:23:01 -07:00
Noah Levitt