mirror of
https://github.com/internetarchive/heritrix3.git
synced 2026-10-12 07:12:02 +00:00
Alex OsborneandClaude Opus 5.5
b1a36159ae
Checkpointing renames each log in place (crawl.log becomes crawl.log.cp00001-<timestamp>) and starts a new one, so earlier entries could only be viewed one file at a time. The paged viewer now offers an "include N checkpoint logs" link (all=y) that shows the log and its rotated generations in the current launch as one virtual file, oldest first, with each run of lines labelled with its file. Positions in the virtual file stay valid across checkpoints, since a rotated file keeps its place and the new log continues after it, and they are converted when toggling so the view stays where it was. Gzipped generations can't be read at arbitrary positions and are left out. LogSeries opens the files and fixes their lengths up front, so a rotation mid-request doesn't change what is read. Paging uses a new FilteredLineScanner over a LogSeries instead of FileUtils.pagedLines, which can only read a single file. Results are the same for every position the navigation links generate; a hand-edited pos in the middle of a line now starts at that line rather than the next one. Lines are capped at 256 KiB so a file with few or no newlines can't exhaust the heap. Also fixes in the crawl.log rendering: a rotated crawl.log gets the same status highlighting, status codes unknown to Jetty (e.g. 999) and numbers too long to be a status no longer fail the page, and the file name, status titles and navigation links are HTML escaped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>