Noah Levitt
3892c17b56
ughhh appease java 8 javadoc rules
2017-06-15 17:29:22 -07:00
Noah Levitt
bc8b0baa59
Merge remote-tracking branch 'origin/master' into fix-test-errors
...
* origin/master:
Setup TravisCI
Set fetch status on curis when testing link extraction
No link extraction on URI not successfully downloaded
Fixes issue #158 : Noisy alerts about 401s without auth challenge (#159 )
Make Content-Location header url INFERRED not REFFER hop type since Content-Location is not for redirection (#151 )
fixes for kafka 0.9 (?)
upgrade to kafka 0.9
somewhat ugly fix to handle exceptions from the bean browser like java.lang.RuntimeException: not implemented at org.archive.modules.fetcher.BdbCookieStore$RestrictedCollectionWrappedList.get(BdbCookieStore.java:92)
2016-06-07 13:09:51 -05:00
Noah Levitt
ca3b65ed03
exclude dns from stats tests because the size of the dns records can vary depending on the environment
2016-06-06 15:43:02 -05:00
Noah Levitt
f7d1ad7f1f
Merge pull request #111 from nlevitt/bean-browse-cookiestore-exception
...
somewhat ugly fix to handle exceptions from the bean browser like java.l...
2016-06-06 11:11:45 -07:00
Andrew Jackson
9145fedd40
Fix unexpected change in count of novel bytes in one selftest.
2016-04-02 20:57:05 +01:00
Noah Levitt
13322111b5
run received urls through the candidates processor, to check scope and schedule them with the frontier
2016-01-15 17:46:27 -08:00
Noah Levitt
cc97f01656
Merge pull request #123 from kris-sigur/HostsReport
...
Hosts report
2016-01-15 10:33:07 -08:00
Noah Levitt
9e4c7f0024
Merge branch 'master' into novel-quotas
...
* master:
license header
check that sourceTag of CrawlURI actually matches configured sourceTag
remove already-outdated stuff from javadoc
handle multiple clauses for same user agent in robots.txt
Hook in submitted seeds properly.
avoid spurious logging
try very hard to start url consumer, and therefore bind the queue to the routing key, so that no messages are dropped, before crawling starts (should always work unless rabbitmq is down); some other tweaks for clarity and stability
2016-01-13 13:44:54 -08:00
Noah Levitt
65a1077425
license header
2016-01-12 14:59:45 -08:00
Noah Levitt
6f9c235fd0
fix javadoc
2015-12-14 15:23:16 -08:00
Noah Levitt
c05678e296
CrawlLimitEnforcer support for maxWarcNovelBytes and maxWarcNovelUrls, with tests
2015-12-11 16:08:47 -08:00
Noah Levitt
dfc1dd940e
tally warc novel content bytes and warc novel urls in CrawledBytesHistotable, with tests
2015-12-11 12:49:02 -08:00
Noah Levitt
c20a06ec27
Merge branch 'seed-limits' into novel-quotas -- so I can add tests to StatisticsSelfTest.java in this branch
...
* seed-limits:
change class originally known as SeedLimitsEnforcer to SourceQuotaEnforcer; make it a Processor instead of a DecideRule (because checking quota at link scoping time doesn't work, since many urls which would go over quota can be added to the frontier); support quotas on any of the fields tracked by CrawledBytesHistotable
fix checkpointing problems with new statsBySource
SeedLimitsEnforcer (contrib) DecideRule that rejects CrawlURI if source seed byte or document limit has been reached
SourceSeedDecideRule applies the configured decision for any URI with discovered from one of a set of seeds
add support to StatisticsTracker to keep a CrawledBytesHistotable per source tag when trackSources is enabled; integration test for this functionality
2015-12-11 11:35:11 -08:00
Noah Levitt
35a7954882
refactor FetchStats to be a subclass of CrawledBytesHistotable, to remove redundant code
2015-12-11 11:05:44 -08:00
Noah Levitt
cce07084e5
CrawlLimitEnforcer support for maxNovelBytes and maxNovelUrls, with tests
2015-12-09 17:15:34 -08:00
Noah Levitt
501efe4a7b
QuotaEnforcer support for quotas on novelBytes and novelUrls, with tests
2015-12-09 15:54:36 -08:00
Noah Levitt
7929e16d6a
fix checkpointing problems with new statsBySource
2015-12-04 21:33:40 -08:00
Noah Levitt
ce4a865764
add support to StatisticsTracker to keep a CrawledBytesHistotable per source tag when trackSources is enabled; integration test for this functionality
2015-12-03 16:13:34 -08:00
Hunter
79ebb0e03c
Merge pull request #129 from nlevitt/java8keytool
...
load keytool utility main class dynamically, trying both the old and …
2015-10-06 16:09:43 -07:00
Noah Levitt
11d4bab1ac
catch exceptions and log error in StatisticsTracker.run(), to make sure statistics events don't stop happening
2015-10-05 11:46:35 -07:00
Noah Levitt
12dde2b693
load keytool utility main class dynamically, trying both the old and new (java 8) class names
2015-09-23 12:06:11 -07:00
Kristinn Sigurðsson
3495851029
JavaDoc typo
2015-07-13 15:10:44 +00:00
Kristinn Sigurðsson
b15fbbd13e
Enable suppression of empty hosts in report.
...
Until now hosts that have never been crawled are included in the hosts
report. This can be a very large number of hosts and is infrequently of
interest. This makes it possible to exclude the empty hosts from the
report. Default behavior remains to include them.
The option is highlighted by including it in the relevant, commented-out
section of the default profile.
2015-07-13 15:08:03 +00:00
Kristinn Sigurðsson
0cf009eb4c
Use default value in sample profile.
2015-07-13 14:58:23 +00:00
Kristinn Sigurðsson
96b927d9fa
Make sorting of host report configurable.
...
Can now set a maximum size for sorted reports. Longer reports are
unsorted. Default behavior is still to sort all reports.
Default profile updated to reflect this new option in a commented-out
section.
2015-07-13 14:55:26 +00:00
Kristinn Sigurðsson
bf638939b0
Remove unused "top" hosts tracking.
...
This used to be used for an overview report that was available in the H1
UI. Had become vestigial code that served no purpose.
2015-07-13 14:32:34 +00:00
Noah Levitt
8009275075
make sure log4j is configured when running unit tests, to avoid log4j error messages polluting captured output and causing tests to fail, like so: https://builds.archive.org/job/Heritrix-3/org.archive.heritrix$heritrix-commons/587/testReport/org.archive.io.arc/ARCWriterTest/testLengthTooLongCompressed/
2015-02-23 11:36:29 -08:00
Kristinn Sigurðsson
4ea7efdfc8
Set character set to UTF-8 when passing through files.
2015-02-10 16:07:37 +00:00
vonrosen
3609abe38c
Merge pull request #108 from nlevitt/aitfive-203
...
kafka crawl log feed
2014-11-14 10:29:41 -08:00
Noah Levitt
5f17707a2e
somewhat ugly fix to handle exceptions from the bean browser like java.lang.RuntimeException: not implemented at org.archive.modules.fetcher.BdbCookieStore$RestrictedCollectionWrappedList.get(BdbCookieStore.java:92)
2014-11-11 12:45:24 -08:00
Noah Levitt
7312ad597e
so that it can appear in the crawl log, add contentSize to CrawlURI extraInfo, for http transactions (the only case in heritrix currently where contentLength and contentSize differ)
2014-11-07 13:21:42 -08:00
Adam Miller
fb23b40e74
CandidatesProcessor.java
...
Keep CrawlURI outlinks intact for use in SeedRecord reporting.
2014-10-22 13:18:25 -07:00
vonrosen
5909042b84
Merge pull request #99 from nlevitt/aitfive-39
...
AMQPCrawlLogFeed, DecideRuleSequenceWithAMQPFeed, DecideRuleSequence.logExtraInfo
2014-10-16 17:24:37 +00:00
Noah Levitt
5f7dc49599
Merge pull request #95 from kngenie/patch-beanbrowser-failure
...
FIX corner-case of bean browser failing due to an exception from hashCode()
2014-10-03 19:21:27 -07:00
Noah Levitt
610a1e1e97
cookie store is called BdbCookieStore again
2014-09-28 14:32:34 -07:00
Noah Levitt
a37a8de4a2
improved implementation not using TreeSet, and some initial work on integration testing of cookie handling
2014-09-28 11:59:14 -07:00
Noah Levitt
47b7caad29
initial bdb implementation; facade by topmost assigned domain, since cookies can apply to subdomains
2014-09-26 20:56:45 -07:00
Noah Levitt
f769e0636d
initial proof of concept fix for HER-2070 "cookie monster" bug
2014-09-26 20:01:52 -07:00
Kenji Nagahashi
5821fe7b1e
FIX corner-case of bean browser failing due to an exception from hashCode().
2014-09-11 11:22:01 -07:00
Kenji Nagahashi
2a38fa0b12
Don’t let exception from property getter fail entire bean-browser.
2014-08-25 13:36:25 -07:00
Noah Levitt
04921073ca
fix bug in CrawlURI.compare() discovered by Kenji, add unit test CrawlURITest.testOrdering()
2014-08-19 17:14:32 -07:00
Noah Levitt
591f0f9f0f
expose forAllPendingDo() as public so it can be used elsewhere
2014-08-13 13:02:28 -07:00
Noah Levitt
9a0c0656b2
move HostsReport.fixup(String) to CrawlHost.fixUpName() so it can be called elsewhere
2014-08-12 16:45:53 -07:00
Noah Levitt
51081a652c
make AbstractFrontier.needReenqueuing() public so it can be called elsewhere
2014-08-12 16:20:25 -07:00
Noah Levitt
47a62f7c6f
remove a couple of crufty lines of code
2014-08-12 16:13:59 -07:00
Noah Levitt
08e24bbdff
update guess at log line length based on real world data
2014-08-12 16:13:11 -07:00
Noah Levitt
905a98c0c2
use RevisitProfile for tallying stats
2014-07-10 21:27:50 -07:00
Noah Levitt
ee52f48be9
remove LinksScoper (deprecated since 2009) and remove recently added code only used in LinksScoper
2014-07-09 10:14:24 -07:00
Noah Levitt
f23a59a918
Merge branch 'master' into HER-2039
...
Conflicts:
modules/src/main/java/org/archive/modules/CrawlURI.java
modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
modules/src/main/java/org/archive/modules/extractor/ExtractorSWF.java
modules/src/main/java/org/archive/modules/extractor/Link.java
modules/src/main/java/org/archive/modules/writer/WARCWriterProcessor.java
2014-07-07 11:45:27 -07:00
Roger G. Coram
fc43a29459
Merge pull request #68 from nlevitt/handle-apply-overlay-exc
...
handle exception determining whether to apply overlay
2014-05-28 09:59:54 +01:00
Noah Levitt