From b10b33821f4a71eecd6b8656d2c52325684fa075 Mon Sep 17 00:00:00 2001 From: Alex Osborne Date: Mon, 15 Feb 2021 17:12:04 +0900 Subject: [PATCH] Fix broken robots.txt link --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index f5853c84..9c88e6d7 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ Heritrix is the Internet Archive's open-source, extensible, web-scale, archival- ## Crawl Operators! -Heritrix is designed to respect the [`robots.txt`](http://www.robotstxt.org/wc/robots.html) exclusion directives and [META robots tags](http://www.robotstxt.org/meta.html). Please consider the +Heritrix is designed to respect the [`robots.txt`](http://www.robotstxt.org/robotstxt.html) exclusion directives and [META robots tags](http://www.robotstxt.org/meta.html). Please consider the load your crawl will place on seed sites and set politeness policies accordingly. Also, always identify your crawl with contact information in the `User-Agent` so sites that may be adversely affected by your crawl can contact you or adapt their server behavior accordingly. ## Getting Started