2 Commits
Author SHA1 Message Date
safishamsi ada1c095ab fix(llm): bisect PDF slices on extracted text; reconcile splittable-PDF tests (#2906)
Three fixes landing the PDF-slicing set coherently:
- bisect_slice read raw container bytes, so the adaptive-retry path (a lone oversized
  slice that still overflows, #2880) searched for the newline cut in binary coordinates
  and could cut mid-line for a compressed PDF. Index unit_source_text instead, matching
  read_slice_text and expand_oversized_files; any converter failure falls back to None.
- #2900's test asserted a PDF is not splittable, which #2906 overturns; rewrite it to
  guard the real invariant (an image has no addressable text and is never sliced). The
  unreadable-PDF passthrough case is covered by test_pdf_slicing.
- add a bisect_slice-on-PDF regression test: the cut lands on an extracted-text line
  boundary and the halves tile the slice exactly.
2026-08-21 16:57:29 +01:00
abhay-codes07 b6b2fb17c8 fix(llm): slice every text document type, not just five of them (#2900)
The oversized-file slicer only split five suffixes while the doc allowlist reaching the
LLM was larger, so a large .qmd/.skill/.html/.yaml/.yml was truncated at the char cap
and silently dropped past it. Extend the splittable-text set to cover every text
document type, with a contract test so future allowlist additions fail loudly.
2026-08-21 16:51:00 +01:00