The dispatch maps (inlineCommands, blockCommands, environments) have
types that mention the parser monad, so they are compiled as functions
of the PandocMonad dictionary and were being rebuilt from scratch --
hundreds of Map entries -- every time a command or environment was
dispatched. SPECIALIZE pragmas can't fix this on the CLI path, since
the reader table abstracts over the monad dictionary.
Instead, add a ReaderT layer to the LP monad carrying a LaTeXEnv
record with the three dispatch maps, built once per parse in readLaTeX
(and once per chunk in rawLaTeXParser's callers), and consult it via
askEnv at the dispatch sites. Internal `lift $ runParserT` sub-parses
share the environment automatically.
Benchmark (1.8MB LaTeX file): allocations drop from 17.85GB to 9.97GB
and wall time from ~4.7s to ~3.4s. On a stress test of 100k \emph
commands, allocations drop from 36GB to 10.5GB.
Co-Authored-By: Claude <noreply@anthropic.com>
doMacros runs before every token consumption; when the head of the
stream is not a control sequence no expansion can occur, so avoid
scanning and rewriting the input stream (which allocated a new
TokStream per consumed token) in that case.
Co-Authored-By: Claude <noreply@anthropic.com>
peekTok went through lookAhead (satisfyTok (const True)), paying
for tokenPrim position bookkeeping, the raw-token state update, and
lookAhead state save/restore on every call; peekTok is used by the
main inline and block dispatchers for every element. Just examine
the head of the input stream after expanding macros.
Co-Authored-By: Claude <noreply@anthropic.com>
macroDef is tried for every control sequence in inline and block
context, and previously attempted ~30 controlSeq alternatives in
turn before failing. Peek at the next control sequence and check
membership in a Set of macro-defining command names first, so the
common case fails after a single lookup.
Co-Authored-By: Claude <noreply@anthropic.com>
untokenize used foldr with Text (<>), which repeatedly copies the
accumulated suffix and is quadratic in the total text length. Use
T.concat on a list of chunks instead, preserving the space
insertion after control words (#5836). Simplify untoken to return
the token text directly (equivalent to the old behavior with an
empty accumulator).
Co-Authored-By: Claude <noreply@anthropic.com>
Previously satisfyTok updated an IntMap of reversed token lists,
one entry per active withRaw scope, so every consumed token cost
O(number of nested withRaw scopes) time and allocation, and the
state was updated even when no withRaw was pending. Replace the
IntMap with a single accumulated list plus a count and a counter of
active scopes: satisfyTok now conses at most once per token (and
does nothing when no scope is active); withRaw takes its slice of
the accumulated list by count difference.
Co-Authored-By: Claude <noreply@anthropic.com>
`tokenize pos` already produces tokens positioned relative to pos,
but the result was then shifted again by pos's line and column,
roughly doubling the reported positions of tokens retokenized from
comments in URL arguments. Tokenize from the position just after
the '%' instead, and drop the extra shift.
Co-Authored-By: Claude <noreply@anthropic.com>
When tokenizing `##` not followed by digits, the continuation
restarted at column pos+1 instead of pos+2, shifting the source
positions of all subsequent tokens on the line one column left.
Co-Authored-By: Claude <noreply@anthropic.com>
Previously \newif\foobar would blindly do `T.drop 2` on the name,
defining a bogus \foobar -> \iffalse plus junk "obartrue"/"obarfalse"
macros; the \iffalse expansion could then swallow following text up
to \fi. Now newif checks the "if" prefix (as TeX requires) and
otherwise falls through to normal unknown-command handling.
Co-Authored-By: Claude <noreply@anthropic.com>
unescapeURL used T.splitOn "\\", which conflates consecutive
backslashes, so "\\" in a URL argument was left as two backslashes
even though '\' is in the escapable set. Rewrite as a direct scan:
a backslash followed by an escapable character is unescaped, and
"\\\\" now yields a single backslash.
Co-Authored-By: Claude <noreply@anthropic.com>
The string literal "\a0\x25FB" parses as BEL + '0' + WHITE MEDIUM
SQUARE; the intent (cf. qedSign in T.P.Readers.LaTeX.Math) was
"\xa0\x25FB", a non-breaking space before the square.
Co-Authored-By: Claude <noreply@anthropic.com>
Big performance improvements.
In addition, treatment of tables now aligns with asciidoctor;
the first line is only implicitly treated as a header if it
is on a line by itself, followed by a blank line.
The ODT reader was written in an idiosyncratic arrows-based style,
using a custom ArrowState arrow and dozens of special combinators.
Replace this with an ordinary monad:
type XMLConverter nsID extraState
= ExceptT () (State (XMLConverterState nsID extraState))
This preserves the original semantics (state changes persist through
failure; recovery via Alternative's <|>) while allowing all readers
to be written in plain do/applicative notation.
- Rewrite Generic/XMLConverter.hs around the new monad; replace the
parent-element stack with a single currentElement field, and make
matchContent matchers simple (namespace, name, reader) triples
whose results are mconcat'd in document order.
- Port StyleReader.hs and ContentReader.hs to monadic style.
- constructList now runs its child reader once instead of twice.
- Delete Arrows/State.hs, Arrows/Utils.hs, and Base.hs; trim
Generic/Fallible.hs and Generic/Utils.hs to what is still used.
Net -994 lines. No changes to test output.
Co-Authored-By: Claude <noreply@anthropic.com>
role="img" and an aria-label copied from alt were being added to
every element with a src-like attribute, not just inlined SVGs.
That is redundant on img elements, whose alt already provides the
accessible name, and actively wrong on elements like video, audio,
iframe, and embed, where role="img" misrepresents interactive
content as a static image to assistive technology.
The attributes are only needed when an img element is replaced by
an inline svg element, since the alt text loses its meaning there;
restrict them to that path. Inline SVG output is unchanged.
Co-Authored-By: Claude <noreply@anthropic.com>
Previously every occurrence of a resource was fetched, decompressed,
and CSS-rewritten independently, so a document referencing the same
image, script, or stylesheet N times triggered N network requests.
Add a cache to ConvertState, keyed on the mime type hint and url, and
generalize getData and the CSS rewriting functions to run in
StateT ConvertState. Failed fetches are cached too, so each missing
resource is now reported only once.
Co-Authored-By: Claude <noreply@anthropic.com>
Gzip.decompress throws an imprecise exception from pure code on
malformed input, which catchError cannot intercept, so a corrupt
.gz or .svgz resource crashed pandoc. Use the streaming zlib API
to catch decompression errors purely, report the failure as
CouldNotFetchResource, and leave the element unchanged.
Co-Authored-By: Claude <noreply@anthropic.com>
fixUrl used T.breakOn and only rewrote the first occurrence, so in
attributes like style="fill:url(#a);stroke:url(#b)" the second
reference kept its unprefixed id and could resolve to the wrong
element when several inline SVGs are combined in one document.
Co-Authored-By: Claude <noreply@anthropic.com>
HTML end tags match case-insensitively, so a fetched script
containing e.g. `</SCRIPT` in a string literal would pass the
guard, get inlined verbatim, and prematurely terminate the inline
script element, corrupting the document.
Co-Authored-By: Claude <noreply@anthropic.com>
When an @import could not be inlined (unfetchable resource, data:
URI, or fragment), the parser consumed the terminating `;` but did
not emit it, producing invalid CSS like `@import url(foo.css)body{...}`
that swallowed the following rule. Also stop consuming whitespace
after the `;` so it is preserved in the output.
Co-Authored-By: Claude <noreply@anthropic.com>
The String -> Text migration (90e436d496) translated
`';' `notElem` mime` as `T.any (== ';') mime`, dropping the
negation. As a result, textual mime types that already carried a
charset parameter got a duplicate `;charset=utf-8` appended, while
bare types like `text/css` got none at all, causing percent-encoded
UTF-8 payloads in data: URIs to be interpreted as US-ASCII
(RFC 2397 default).
Co-Authored-By: Claude <noreply@anthropic.com>
Rewrite the guard-isJust/fromJust combinations in pWebpSize as
case expressions, removing the partial fromJust calls, and drop
the now-unused Data.Maybe imports.
Also make checkDpi default to 72 for negative dpi values, not
just 0; a negative dpi would produce negative dimensions in
sizeInPoints.
Co-Authored-By: Claude <noreply@anthropic.com>
The ISO BMFF box parser treated the 32-bit size field as the
literal box size, but size == 1 means a 64-bit "largesize" field
follows the box type, and size == 0 means the box extends to the
end of the file. A new getBoxHeader helper handles all three
forms and is used everywhere box headers are read, fixing both
the header parsing and the size accounting in
searchAvifBoxesInRange, which subtracted the raw size field.
Also remove a double skip in searchAvifBoxes: tryParseDimensions
already skips the unconsumed contents of a box, so skipping
contentSize again caused parse failures whenever an unknown
top-level box preceded the meta box.
Co-Authored-By: Claude <noreply@anthropic.com>
AVIF files carry no resolution information in the boxes we parse, so
use the dpi from the writer options, as we already do for WebP and
SVG. With the default options this changes the assumed resolution
from 72 to 96 dpi.
Co-Authored-By: Claude <noreply@anthropic.com>
As with the earlier JPEG change: JuicyPixels' decodeImageWithMetadata
fully decodes paletted PNGs before returning any metadata (other PNG
color types are decoded lazily, but still allocate). Instead, read
the dimensions from the IHDR chunk and the resolution from the pHYs
chunk, using the same unit conversion as JuicyPixels, so results are
unchanged. If the header scan fails, we still fall back to the full
decoder.
Co-Authored-By: Claude <noreply@anthropic.com>
Instead of two full isInfixOf scans (one for "<svg", one for
"<SVG"), do a single pass that uses elemIndex (memchr) to jump
between '<' characters and checks both spellings at each one.
On a 26 MB non-SVG XML file this is 2.4x faster; on files with few
'<' characters, 60x.
Co-Authored-By: Claude <noreply@anthropic.com>
pPdfSize read the body of a compressed object stream with
`manyTill (satisfy (const True)) ...`, which builds a list with one
cons cell per byte and re-attempts the terminator parser at every
position. For a 4 MB stream this allocated about 3 GB, and larger
streams could exhaust memory. Replace it with a chunked scan for
the "endstream" keyword.
The trailing end-of-line before "endstream" is now passed along with
the stream data, which is harmless: zlib decompression stops at the
end of the compressed stream and ignores leftover input.
Co-Authored-By: Claude <noreply@anthropic.com>
Previously JPEG sizes were determined with JuicyPixels'
decodeImageWithMetadata, which fully decodes the image data even
though only the metadata is needed (for JPEG, both the metadata and
the Right/Left verdict depend on running the whole decoder). For a
4000x3000 photo this allocated over 1 GB.
Instead, scan the marker segments preceding the entropy-coded data:
the start-of-frame segment gives the dimensions, and the JFIF APP0
and Exif APP1 segments give the resolution. The dpi computation
follows JuicyPixels' metadata extraction (including Exif taking
precedence over JFIF), so results are unchanged; all JPEG files in
the test suite yield identical sizes. If the header scan fails, we
still fall back to the full decoder.
Co-Authored-By: Claude <noreply@anthropic.com>
The new Tests.ImageSize module constructs minimal binary files
in-memory to exercise imageType and imageSize for EPS, PDF (including
compressed object streams), SVG, EMF, WebP (lossless, lossy, and
extended), and AVIF (ispe and tkhd paths), covering regressions for
the bugs fixed in the preceding commits, plus fixture-based tests for
JPEG, PNG, and GIF.
[API change] ImageType now derives Eq.
Co-Authored-By: Claude <noreply@anthropic.com>
Per the SVG spec, the viewBox numbers may be separated by
whitespace and/or a comma, and may be fractional. Previously the
viewBox fallback (used when the svg element lacks usable width and
height attributes) failed for such values.
Co-Authored-By: Claude <noreply@anthropic.com>
The %%BoundingBox comment gives llx lly urx ury; the size was
computed from the upper corner alone, giving wrong dimensions for
EPS files whose bounding box has a non-zero origin. Also decode the
coordinates leniently, so that invalid UTF-8 on the line cannot
raise an exception from pure code.
Co-Authored-By: Claude <noreply@anthropic.com>
- imageType: only classify a file with an ISO BMFF ftyp box as Avif
if the major brand is avif or avis. Previously any ISO media file
(mp4, mov, heic, ...) was identified as AVIF (and then failed
size detection, since verifyFtyp checks the brand).
- Fix off-by-8 skips in the tkhd box parser: the fields preceding
width/height total 72 bytes (version 0) or 84 bytes (version 1)
after version/flags; the previous 64/76 landed inside the
transformation matrix, yielding bogus dimensions for animated
AVIF files lacking an ispe box.
Co-Authored-By: Claude <noreply@anthropic.com>
Codec.Compression.Zlib.decompress throws an exception from pure code
when the corrupt part of its lazy result is forced, so a PDF with a
malformed compressed object stream could crash pandoc. Decompress
total-ly via foldDecompressStreamWithInput instead, treating
malformed streams as a parse failure (so we keep scanning the rest
of the document for a /MediaBox).
Co-Authored-By: Claude <noreply@anthropic.com>
