Noah Levitt
9e4c7f0024
Merge branch 'master' into novel-quotas
...
* master:
license header
check that sourceTag of CrawlURI actually matches configured sourceTag
remove already-outdated stuff from javadoc
handle multiple clauses for same user agent in robots.txt
Hook in submitted seeds properly.
avoid spurious logging
try very hard to start url consumer, and therefore bind the queue to the routing key, so that no messages are dropped, before crawling starts (should always work unless rabbitmq is down); some other tweaks for clarity and stability
2016-01-13 13:44:54 -08:00
Noah Levitt
65a1077425
license header
2016-01-12 14:59:45 -08:00
Noah Levitt
6f9c235fd0
fix javadoc
2015-12-14 15:23:16 -08:00
Noah Levitt
c05678e296
CrawlLimitEnforcer support for maxWarcNovelBytes and maxWarcNovelUrls, with tests
2015-12-11 16:08:47 -08:00
Noah Levitt
dfc1dd940e
tally warc novel content bytes and warc novel urls in CrawledBytesHistotable, with tests
2015-12-11 12:49:02 -08:00
Noah Levitt
c20a06ec27
Merge branch 'seed-limits' into novel-quotas -- so I can add tests to StatisticsSelfTest.java in this branch
...
* seed-limits:
change class originally known as SeedLimitsEnforcer to SourceQuotaEnforcer; make it a Processor instead of a DecideRule (because checking quota at link scoping time doesn't work, since many urls which would go over quota can be added to the frontier); support quotas on any of the fields tracked by CrawledBytesHistotable
fix checkpointing problems with new statsBySource
SeedLimitsEnforcer (contrib) DecideRule that rejects CrawlURI if source seed byte or document limit has been reached
SourceSeedDecideRule applies the configured decision for any URI with discovered from one of a set of seeds
add support to StatisticsTracker to keep a CrawledBytesHistotable per source tag when trackSources is enabled; integration test for this functionality
2015-12-11 11:35:11 -08:00
Noah Levitt
35a7954882
refactor FetchStats to be a subclass of CrawledBytesHistotable, to remove redundant code
2015-12-11 11:05:44 -08:00
Noah Levitt
cce07084e5
CrawlLimitEnforcer support for maxNovelBytes and maxNovelUrls, with tests
2015-12-09 17:15:34 -08:00
Noah Levitt
501efe4a7b
QuotaEnforcer support for quotas on novelBytes and novelUrls, with tests
2015-12-09 15:54:36 -08:00
Noah Levitt
7929e16d6a
fix checkpointing problems with new statsBySource
2015-12-04 21:33:40 -08:00
Noah Levitt
ce4a865764
add support to StatisticsTracker to keep a CrawledBytesHistotable per source tag when trackSources is enabled; integration test for this functionality
2015-12-03 16:13:34 -08:00
Hunter
79ebb0e03c
Merge pull request #129 from nlevitt/java8keytool
...
load keytool utility main class dynamically, trying both the old and …
2015-10-06 16:09:43 -07:00
Noah Levitt
11d4bab1ac
catch exceptions and log error in StatisticsTracker.run(), to make sure statistics events don't stop happening
2015-10-05 11:46:35 -07:00
Noah Levitt
12dde2b693
load keytool utility main class dynamically, trying both the old and new (java 8) class names
2015-09-23 12:06:11 -07:00
Noah Levitt
8009275075
make sure log4j is configured when running unit tests, to avoid log4j error messages polluting captured output and causing tests to fail, like so: https://builds.archive.org/job/Heritrix-3/org.archive.heritrix$heritrix-commons/587/testReport/org.archive.io.arc/ARCWriterTest/testLengthTooLongCompressed/
2015-02-23 11:36:29 -08:00
Kristinn Sigurðsson
4ea7efdfc8
Set character set to UTF-8 when passing through files.
2015-02-10 16:07:37 +00:00
vonrosen
3609abe38c
Merge pull request #108 from nlevitt/aitfive-203
...
kafka crawl log feed
2014-11-14 10:29:41 -08:00
Noah Levitt
7312ad597e
so that it can appear in the crawl log, add contentSize to CrawlURI extraInfo, for http transactions (the only case in heritrix currently where contentLength and contentSize differ)
2014-11-07 13:21:42 -08:00
Adam Miller
fb23b40e74
CandidatesProcessor.java
...
Keep CrawlURI outlinks intact for use in SeedRecord reporting.
2014-10-22 13:18:25 -07:00
vonrosen
5909042b84
Merge pull request #99 from nlevitt/aitfive-39
...
AMQPCrawlLogFeed, DecideRuleSequenceWithAMQPFeed, DecideRuleSequence.logExtraInfo
2014-10-16 17:24:37 +00:00
Noah Levitt
5f7dc49599
Merge pull request #95 from kngenie/patch-beanbrowser-failure
...
FIX corner-case of bean browser failing due to an exception from hashCode()
2014-10-03 19:21:27 -07:00
Noah Levitt
610a1e1e97
cookie store is called BdbCookieStore again
2014-09-28 14:32:34 -07:00
Noah Levitt
a37a8de4a2
improved implementation not using TreeSet, and some initial work on integration testing of cookie handling
2014-09-28 11:59:14 -07:00
Noah Levitt
47b7caad29
initial bdb implementation; facade by topmost assigned domain, since cookies can apply to subdomains
2014-09-26 20:56:45 -07:00
Noah Levitt
f769e0636d
initial proof of concept fix for HER-2070 "cookie monster" bug
2014-09-26 20:01:52 -07:00
Kenji Nagahashi
5821fe7b1e
FIX corner-case of bean browser failing due to an exception from hashCode().
2014-09-11 11:22:01 -07:00
Kenji Nagahashi
2a38fa0b12
Don’t let exception from property getter fail entire bean-browser.
2014-08-25 13:36:25 -07:00
Noah Levitt
04921073ca
fix bug in CrawlURI.compare() discovered by Kenji, add unit test CrawlURITest.testOrdering()
2014-08-19 17:14:32 -07:00
Noah Levitt
591f0f9f0f
expose forAllPendingDo() as public so it can be used elsewhere
2014-08-13 13:02:28 -07:00
Noah Levitt
9a0c0656b2
move HostsReport.fixup(String) to CrawlHost.fixUpName() so it can be called elsewhere
2014-08-12 16:45:53 -07:00
Noah Levitt
51081a652c
make AbstractFrontier.needReenqueuing() public so it can be called elsewhere
2014-08-12 16:20:25 -07:00
Noah Levitt
47a62f7c6f
remove a couple of crufty lines of code
2014-08-12 16:13:59 -07:00
Noah Levitt
08e24bbdff
update guess at log line length based on real world data
2014-08-12 16:13:11 -07:00
Noah Levitt
905a98c0c2
use RevisitProfile for tallying stats
2014-07-10 21:27:50 -07:00
Noah Levitt
ee52f48be9
remove LinksScoper (deprecated since 2009) and remove recently added code only used in LinksScoper
2014-07-09 10:14:24 -07:00
Noah Levitt
f23a59a918
Merge branch 'master' into HER-2039
...
Conflicts:
modules/src/main/java/org/archive/modules/CrawlURI.java
modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
modules/src/main/java/org/archive/modules/extractor/ExtractorSWF.java
modules/src/main/java/org/archive/modules/extractor/Link.java
modules/src/main/java/org/archive/modules/writer/WARCWriterProcessor.java
2014-07-07 11:45:27 -07:00
Roger G. Coram
fc43a29459
Merge pull request #68 from nlevitt/handle-apply-overlay-exc
...
handle exception determining whether to apply overlay
2014-05-28 09:59:54 +01:00
Noah Levitt
39814d1efd
avoid calling CheckpointService.hasAvailableCheckpoints() when crawl not launchable and therefore the info is not needed, because it calls isRunning() which is synchronized and blocks when checkpoint is in progress
2014-05-22 16:40:44 -07:00
Noah Levitt
963e04e50c
handle exception determining whether to apply overlay
2014-05-13 15:47:21 -07:00
Noah Levitt
ba68649ba0
Merge remote-tracking branch 'origin/master' into cosmetics
2014-05-07 10:02:06 -07:00
Noah Levitt
b7f8a0669f
Merge branch 'cosmetics' of github.com:nlevitt/heritrix3 into cosmetics
2014-05-07 10:02:00 -07:00
Noah Levitt
33e782e2fd
oops, make "exit java process" button work again
2014-05-07 10:01:47 -07:00
Noah Levitt
d777c198bd
HER-2065 - in activateInactiveQueue() close synchronized block earlier, and move updateHighestWaiting() up to keep it inside the block; now the call to deactivateQueue() is not in the synchronized block, so the deadlock should be avoided (fix suggested by kenji)
2014-04-24 17:23:01 -07:00
Noah Levitt
0cd1043cb1
change memory bar color, use consistent non-capitalization on buttons
2014-04-21 11:53:40 -07:00
Noah Levitt
1b72ce5dd8
remove extra </div>
2014-04-21 11:53:05 -07:00
Noah Levitt
cfb9474f7a
cosmetic changes
2014-04-21 09:06:15 -07:00
Noah Levitt
20a5f31e19
ExtractorHTMLForms+FormLoginProcessor full crawl job integration test (not passing yet)
2014-03-31 18:57:33 -07:00
Shriphani Palakodety
2f60b5f935
migration to latest guava complete
2014-03-18 06:42:02 -04:00
Noah Levitt
ea19dacbd2
handle main job log handler like current launch job log handler, close on teardown
2014-02-26 17:19:50 -08:00
Martins Balodis
88dc01305f
fix job.log file handler was left open when jobdir is removed
2014-02-26 21:15:38 +02:00