mirror of
https://github.com/internetarchive/heritrix3.git
synced 2026-09-07 06:15:43 +00:00
Tidy up markup and links.
This commit is contained in:
Vendored
+14
-8
@@ -9,8 +9,9 @@ Readme for Heritrix
|
||||
6. License
|
||||
|
||||
|
||||
1. Introduction
|
||||
1) Introduction
|
||||
----------------
|
||||
|
||||
Heritrix is the Internet Archive's open-source, extensible, web-scale,
|
||||
archival-quality web crawler project. Heritrix (sometimes spelled heretrix, or
|
||||
misspelled or missaid as heratrix/heritix/heretix/heratix) is an archaic word
|
||||
@@ -19,8 +20,9 @@ preserve the digital artifacts of our culture for the benefit of future
|
||||
researchers and generations, this name seemed apt.
|
||||
|
||||
|
||||
2. Crawl Operators!
|
||||
2) Crawl Operators!
|
||||
--------------------
|
||||
|
||||
Heritrix is designed to respect the robots.txt
|
||||
<http://www.robotstxt.org/wc/robots.html> exclusion directives and META robots
|
||||
tags <http://www.robotstxt.org/wc/exclusion.html#meta>. Please consider the
|
||||
@@ -30,25 +32,29 @@ User-Agent so sites that may be adversely affected by your crawl can contact
|
||||
you or adapt their server behavior accordingly.
|
||||
|
||||
|
||||
3. Getting Started
|
||||
3) Getting Started
|
||||
-------------------
|
||||
See the User Manual at <https://github.com/internetarchive/heritrix3/wiki/Heritrix%203.0%20and%203.1%20User%20Guide>
|
||||
|
||||
See the User Manual, available from <https://github.com/internetarchive/heritrix3/wiki
|
||||
|
||||
|
||||
4. Developer Documentation
|
||||
4) Developer Documentation
|
||||
---------------------------
|
||||
|
||||
See <http://crawler.archive.org/articles/developer_manual/index.html>.
|
||||
For API documentation, see <https://webarchive.jira.com/wiki/display/Heritrix/Heritrix+3.x+API+Guide>
|
||||
For API documentation, see <https://heritrix.readthedocs.io/en/latest/api.html>
|
||||
and <http://builds.archive.org/javadoc/heritrix-3.2.0/>
|
||||
|
||||
|
||||
5. Latest Releases
|
||||
5) Latest Releases
|
||||
-------------------
|
||||
|
||||
Information about releases can be found at <https://github.com/internetarchive/heritrix3/wiki#latest-releases>
|
||||
|
||||
|
||||
6. License
|
||||
6) License
|
||||
-----------
|
||||
|
||||
Heritrix is free software; you can redistribute it and/or modify it
|
||||
under the terms of the Apache License, Version 2.0:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user