lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-25 16:06:33 +00:00
Code Issues Packages Projects Releases Wiki Activity

Clarify robots protocol support

Browse Source 
Closes #351, #353
This commit is contained in:
Alex Osborne
2021-02-15 17:29:49 +09:00
committed by GitHub
parent b10b33821f
commit 35bb3286f4
1 changed files with 3 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+3 -1
View File
@@ -10,9 +10,11 @@ Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-
## Crawl Operators!
Heritrix is designed to respect the [`robots.txt`](http://www.robotstxt.org/robotstxt.html) exclusion directives and [META robots tags](http://www.robotstxt.org/meta.html). Please consider the
Heritrix is designed to respect the [`robots.txt`](http://www.robotstxt.org/robotstxt.html) exclusion directives<sup>†</sup> and [META nofollow tags](http://www.robotstxt.org/meta.html). Please consider the
load your crawl will place on seed sites and set politeness policies accordingly. Also, always identify your crawl with contact information in the `User-Agent` so sites that may be adversely affected by your crawl can contact you or adapt their server behavior accordingly.
<sup>†</sup> The newer wildcard extension to robots.txt is [not yet](https://github.com/internetarchive/heritrix3/issues/250) supported.
## Getting Started
- [User Manual](https://github.com/internetarchive/heritrix3/wiki)