Alex Osborne 1fae77c64d Enforce robots.txt character limit per char not per line ...

Since we were only enforcing the size limit after reading each
line a single very long line could cause us to to run out of
memory.

This change enforces the character limit on each read character
not just on line boundaries.

We also correct the count of processed characters in the warning
message, which was not counting newline and linefeed characters.

2017-10-31 18:41:00 +09:00

..

main

Enforce robots.txt character limit per char not per line

2017-10-31 18:41:00 +09:00

test

Enforce robots.txt character limit per char not per line

2017-10-31 18:41:00 +09:00