mirror of
https://github.com/internetarchive/heritrix3.git
synced 2026-08-23 15:06:46 +00:00
269a7638601740b0764c8c0ff3644c07e7e7895b
* CrawlServer, Robotstxt
move autoregister details to Robotstxt, RobotsDirectives
* RobotsDirectives
use ReferenceFieldSerializer so that repeated instances of RobotsDirectives in same context are replaced with backrefs
* RobotstxtTest
(from Kenji) unit test for instance multiplication
Languages
Java
93.3%
HTML
2.6%
Rich Text Format
1.1%
JavaScript
1%
FreeMarker
0.8%
Other
1%