Given RST like
.. _target:
See |sub|.
.. |sub| replace:: `text <target_>`_
'pandoc -f rst -t html' produces
<div id="target">
<p>See <a href="##REF##target">text</a>.</p>
</div>
instead of the expected
<div id="target">
<p>See <a href="#target">text</a>.</p>
</div>
It formerly worked and regressed with c8fda8f4d ("RST reader: Use a new
one-pass parsing strategy."), release 3.6.
What happens is that during parsing pass 1 the `replace::` value `text
<target_>`_ is parsed to
Link nullAttr [Str "text"] ("##REF##target", "")
and is stored in ParserState's substitution table. Separately, '|sub|'
usage is parsed to
Link nullAttr [Str "|sub|"] ("##SUBST##|sub|", "")
and is stored in the document tree. resolveReferences then replaces the
placeholder in the document node with substitution table node during
walkM. However, the freshly substituted ##REF## placeholder was not
revisited further, and appeared unresolved in the output.
To fix it, we resolve the node recursively until the result contains no
more placeholder. We must protect from self-references to avoid
endless recursion.
Word represents "restart numbering" on a style-based list by pointing
only the first item of the restarted list at a new `numId` that shares the
original list's abstract numbering definition but carries a
`w:startOverride`; the remaining items keep using the original `numId`.
Pandoc keyed list continuation and grouping on the `numId`, so the
restarted items continued the stale count from the earlier list (and
were split into a separate ordered list with the wrong start).
Key continuation and grouping off the `abstractNumId` instead (the real
running counter in Word), and treat `startOverride` as a restart that
resets the count.
Closes#8367.
Co-Authored-By: Claude <noreply@anthropic.com>
Word-95-style numbered and bulleted lists are encoded with a
`{\pntext ...}` auto-number destination at the start of each list
paragraph plus a `{\*\pn ...}` destination describing the numbering,
rather than the modern `\listtext`/`\listtable` mechanism. Two problems:
1. The `\pntext` marker text ("1.", "·", etc.) was captured as the
paragraph's first text run, which sits before the paragraph's
`\ls`/`\ilvl`, so emitBlocks (which reads list properties from the first
run) misclassified the paragraph as an ordinary paragraph. The first
item of each list therefore came out as a stray paragraph.
2. The numbering style was ignored, so numbered lists defaulted to
bullets.
Treat `\pntext` like `\listtext`: drop its visible marker text and flag the
start of a new list item. Parse the `{\*\pn ...}` destination
(`\pnlvlbody`/`\pnlvlblt`, `\pndec`, `\pnucltr`, `\pnlcltr`,
`\pnucrm`, `\pnlcrm`, `\pnstart`, and the `\ls`/`\ilvl` keys it
carries) into the list override table so numbered lists are
emitted as ordered lists with the right number style.
`\pn` is a paragraph property that remains in effect until reset
by `\pard`, auto-numbering every paragraph in scope. Track
this (`sPnActive`) so each paragraph becomes its own list item,
rather than merging markerless continuation paragraphs as is done
for modern `\listtext` lists.
Co-Authored-By: Claude <noreply@anthropic.com>
The reader treated every list paragraph as a new list item, so a list
item containing several paragraphs was split into one item per
paragraph. In the RTF list model a new item is marked by a
`{\listtext ...}` destination group; a list paragraph lacking one is a
continuation of the current item.
Track whether a `{\listtext}` group was seen (`sListText`) and, in
`emitBlocks`, append a continuation paragraph (no `\listtext`) to the
current item rather than starting a new one.
Adds a `list_multiparagraph` reader test.
Co-Authored-By: Claude <noreply@anthropic.com>
When parsing an inline note (`^[...]`) inside a quoted span,
`stateQuoteContext` was still set to `InSingleQuote`/`InDoubleQuote`,
so quotes within notes failed to parse as `Quoted` nodes.
Fix this by wrapping the note body parser in
`withQuoteContext NoQuote`.
Closes#11613.
Previously, when a w:p paragraph contained runs with textboxes,
the entire paragraph was replaced by just the textbox content,
discarding all other runs (including image-bearing runs).
Now we walk the paragraph's children in order, grouping
non-textbox content into copies of the original w:p and splicing
unwrapped textbox content in place, preserving the original order.
We also treat text inside a textbox containing an image as a
figure caption. (One often finds captioned images of this
kind in docx files.)
Closes#11510.
Closes#6893.
Closes#11412.
Closes#5394.
Closes#9633.
Co-Authored-By: Claude <noreply@anthropic.com>
pandoc generates a separate style class like P123 for each paragraph,
inheriting the desired style using `parent-style-name`.
Previously, we did not support that but it is pretty easy to add.
Though for simplicity this fix does not extend to finding the parent
style recursively when deeper hierarchies are involved.
LibreOffice uses `Preformatted_20_Text` as the default style for
code listings etc. Let’s detect if a paragraph has the `style-name`
and convert it to `CodeBlock` if it is the case.
Unfortunately, pandoc itself generates a separate style class like P123
for each paragraph so we will not yet support code listings from documents
generated by pandoc itself.
Closes#4841.
Table rows in Docx files can use `w:gridBefore` property to specify
a number of extra empty table cells to be inserted at the beginning
of a row. Without this, table columns in tables using the element
will become misaligned.
https://learn.microsoft.com/en-us/dotnet/api/documentformat.openxml.wordprocessing.gridbefore?view=openxml-3.0.1
In the worst case, this can lead to cells at the end of some rows
to be dropped by the parser.
The test file was produced as follows based on
the “3GPP TS 24.282 V19.5.0 (2025-12)” document obtained from
<https://www.3gpp.org/ftp/Specs/archive/24_series/24.282/24282-j50.zip>
where I encountered the issue:
1. Extracted the `document.xml`.
2. Formatted it with `xmllint --format`.
3. Found the raw XML for “Table 15.2.13-2: Payload content type”.
4. Removed `w:rsid*` and `w14:*` attributes, and `w:pStyle` elements.
5. Replaced the context of `test/docx/raw-blocks.native`.
6. Used pandoc to generate the docx.
Each worksheet turns into a section containing a table.
The common file `nativeDiff` has been extract from
the Docx and Pptx text files and put in Tests.Helpers.
New module `Text.Pandoc.Readers.Pptx`,
exporting `readPptx`. [API change]
Factored out some common OOXML functions from
Text.Pandoc.Readers.Docx.Util into a non-exported module
Text.Pandoc.Readers.OOXML.Shared.
If a `latex-pos` attribute is present on a figure, it will be used as the
optional positioning hint in LaTeX (e.g. `ht`).
With implicit figures, `latex-pos` will be added to the figure (and
removed from the image) if it is present on the image.
Closes#10369.
Sub- and superscript must be preceded by a string in Org mode. Some text
preceded by space or at the start of a paragraph was previously parsed
incorrectly as sub- or superscript.
Org mode makes a distinction between smart parsing of quotes, and smart
parsing of special strings like `...`. The finer grained control over
these features is necessary to truthfully reproduce Emacs Org mode
behavior. Special strings are enabled by default, while smart quotes are
disabled.
The behavior of `special_string` is brought closer to the reference
implementation in that `\-` is now treated as a soft hyphen.
Translating entities by name ultimately relies on
Commonmark.Entity.lookupEntity, which de facto requires the entity name
to be followed by a semicolon. Paste a semicolon onto the end of the
entity name read from POD to look it up.
Fixes#11015
If the `four_space_rule` extension is not enabled,
figure out the indentation needed for child blocks dynamically,
by looking at the first nonspace content after the `:` marker.
Previously the four-space rule was always obeyed.
Remove the old `compact_definition_lists` extension. This was
neded to preserve backwards compatibility after pandoc 1.12
was released, but at this point we can get rid of it.
T.P.Extensions: remove `Ext_compact_definition_lists` constructor
for `Extension` [API change].
Fix tight/loose detection for definition lists, to conform to
the documentation.
Closes#10889.
Previously inline TeX was handled in a way that was different
from org's own export, and that could lead to information loss.
This was particularly noticeable for inline math environments
such as `equation`. Previously, an `equation` environment
starting at the beginning of a line would create a raw block,
splitting up the paragraph containing it (see #10836).
On the other hand, an `equation` environment not at the beginning
of a line would be turned into regular inline elements
representing the math. (This would cause the equation number to
go missing and in some cases degrade the math formatting.)
Now, we parse all of these as raw "latex" inlines, which will be
omitted when converting to formats other than LaTeX (and other
formats like pandoc's Markdown that allow raw LaTex).
Closes#10836.
Previously we tried to handle things like commented out list
items:
- one
<!--
- two
-->
- three
and also things like:
- one `and
- two` and
But the code we added to handle these cases caused problems with
other, more straightforward things, like:
- one
- ```
code
```
- three
So we are rolling back all the fanciness, so that the markdown
parser now behaves more like the commonmark parser, in which
indicators of block-level structure always take priority over
indicators of inline structure.
Closes#9865. Closes#7778. See also #5628.
Once upon a time the only metadata element for links in Pandoc's AST was
a title, and it was hijacked to track certain links as having originated
in the wikilink syntax. Now we have Attrs and we can use a class to
handle wikilinks instead.
Requires coordinated changes to commonmark-hs.
Pod ("Plain old documentation") is a markup languaged used principally
to document Perl modules and programs. Since it was originally meant to
be translated pretty directly to man, the semantics are fairly simple.
This Pod reader was developed with reference to the canonical user and
implementer documentation of Pod: https://perldoc.perl.org/perlpod and
https://perldoc.perl.org/perlpodspec.
There are 1490 .pod, .pl, and .pm in the Perl 5.34 distribution found in
/System/Library/Perl on my mac. Of those, this reader dies with a parse
error on 7 of them. All of them seem to be cases where pod commands are
found within a non-colon-prefixed =begin/=end. perlpodspec says I may
treat this as an error.
[API change] adds readPod
This change introduces a reader for mdoc, a roff-derived semantic markup
language for manual pages. The two relevant contemporary implementations
of mdoc for manual pages are mandoc (https://mandoc.bsd.lv/), which
implements the language from scratch in C, and groff
(https://www.gnu.org/software/groff/), which implements it as roff macros.
mdoc has a lot of semantics specific to technical manuals that aren't
representable in Pandoc's AST. I've taken a cue from the mandoc HTML
output and many mdoc elements are encoded as Codes or Spans with classes
named for the mdoc macro that produced them.
Much like web browsers with HTML, mandoc attempts to produce best-effort
output given all kinds of weird and crappy mdoc input. Part of the
reason it's able to do this is it uses a very accommodating parse tree
and stateful output routines specialized to the output mode, and when it
encounters some macro it wasn't expecting, it can easily give up on
whatever it was outputting and output something else. I've encoded as
much flexibility as I reasonably could into the mdoc reader here, but I
don't know how to be as flexible as mandoc.
This branch has been developed almost exclusively against mandoc's
documentation and implementation of mdoc as a reference, and the
real-world manual pages tested against are those from the OpenBSD base
system. Of ~3500 manuals in mdoc format shipped with a fresh OpenBSD
install, 17 cause the mdoc reader to exit with a parse error. Any
further chasing of edge cases is deferred to future work.
Many of the tests in test/Tests/Readers/Mdoc.hs are derived directly
from mandoc's extensive regression tests.
[API change] Adds readMdoc to the public API
Create MetaInlines when possible, just as with markdown input.
MetaBlocks is still used when there are multiple paragraphs or
non-paragraph content.
This change also affects field lists.
Closes#7766.
Closes#6461.
Blockquotes are not really block containers in DokuWiki;
the lines are interpreted literally (so, e.g., you can't
start a list), and line breaks are added at the ends.
When the last parPart before a header was a bookmark, no span with an
anchor was added for a bookmark in the header. But the function that
adds header anchors to the anchor map, needs a span with an anchor.
So this commit adds that span.
This applies to both fenced and HTML-ish varieties.
Otherwise we face an exponential performance problem with
backtracking.
This also accords with the behavior of the `fenced_divs`
extension in commonmark.
A warning is issued when a div is implicitly closed.
Closes#9635.
John MacFarlane