mirror of
https://github.com/jaypyles/Scraperr.git
synced 2025-10-30 14:07:04 +00:00
4 lines
67 B
Plaintext
4 lines
67 B
Plaintext
# https://www.robotstxt.org/robotstxt.html
|
|
User-agent: *
|
|
Disallow:
|