The semantic cache keyed entries on sha256(file content + path) alone, with
no component for the extraction prompt that produced them. After an upgrade
that changed the prompt, every unchanged file was a cache hit and replayed
the older prompt's extraction: the run exited 0, cost.json looked cheap, and
the graph silently carried two prompt generations side by side. The reporter
saw 506 of 512 docs replay an older vintage on a rebuild they expected to be
cold; deleting the whole cache was the only workaround.
output, and invalidating them on every release would re-bill extraction for
unchanged files. Fingerprinting the prompt itself keeps both properties:
entries survive releases that don't touch the prompt, and invalidate only
when it actually changed.
Semantic entries now live under cache/semantic/p{fingerprint}/, mirroring the
AST cache's v{version}/ layout. Both extraction paths pass their prompt: the
Python/CLI path from llm.py's _EXTRACTION_SYSTEM (shared by every backend),
and the skill path via a new prompt_file argument in Step B0/B3 naming the
references/extraction-spec.md the subagents were handed. The fingerprint
normalizes line endings so a CRLF checkout isn't mistaken for a new prompt.
Pre-existing entries predate fingerprinting and have unknowable vintage, so
they are still served rather than re-billing a whole corpus on upgrade — but
check_semantic_cache now warns with the count, turning "no signal at all"
into a visible one. merge_existing refuses to fuse such an entry into a
current-vintage write, which would mix two prompts inside one entry and then
attest the result to a prompt that produced half of it.
Old-fingerprint entries are pruned by liveness only, never swept wholesale
the way stale AST versions are: two hosts with different prompts (verbose vs
compact extraction-spec) can share one graphify-out/, and a wholesale sweep
would have each run delete the other's entries and re-bill on every
alternation. prune/clear/cached_files glob recursively so fingerprinted
entries can't become unprunable orphans (the #1527 failure mode).
The two monolith skills (aider, devin) inline their prompt instead of
shipping a spec sidecar and stay on the unfingerprinted path for now.
`graphify extract --mode deep` over a warm tree was a silent no-op, for
three stacked reasons:
1. The semantic cache ignored mode: deep runs were served standard-mode
entries (and vice versa). check_semantic_cache/save_semantic_cache now
take `mode` (default None, byte-identical when omitted so older
installed callers keep working) and map it to a namespaced kind —
cache/semantic/ for None, cache/semantic-{mode}/ otherwise. The
per-chunk checkpoint in llm.extract_corpus_parallel and the extract /
cache-check consumers thread the run's mode through (cache-check grows
--mode/--deep). cached_files, clear_cache, and prune_semantic_cache
sweep BOTH namespaces; prune uses the same live-hash set for both
(liveness is content-based, mode-independent) so semantic-deep/ can't
regrow the #1527 unbounded-orphan problem and inherits the
files_by_type-derived exclusion gating for free.
2. extract had no --force — the flag was silently swallowed by the
parser's unknown-arg fallthrough. It is now real (plus GRAPHIFY_FORCE
env parity with `update`): force disables the incremental gate so
detection is a full scan and skips the semantic cache READ so every
semantic file re-dispatches, while the post-run save and manifest
stamping still happen.
3. The incremental gate dispatched zero files on a warm unchanged tree
before the cache was ever consulted, so namespacing alone couldn't fix
the repro. In deep+incremental runs the semantic pass now widens to the
full live doc/paper/image set from detect_incremental's files_by_type
(already exclusion-filtered, #1908/#1909) and lets the mode-namespaced
cache decide hits/misses, with a loud count line so the first deep
run's full re-dispatch is visible.
Skill-side threading of mode is deliberately deferred to PR-2; mode
defaults keep generated skills byte-compatible.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
save_semantic_cache groups nodes/edges/hyperedges by their own source_file
and the write loop skips a group whose path is a ghost (not .is_file(),
silently) or out-of-scope per the #1757 guard — but an edge or hyperedge in
an ALLOWED group that references a node id from a skipped group was still
written verbatim, so on replay (check_semantic_cache) it dangled forever.
The #1895 filter does not cover this: it cleans the in-memory merged result,
while the checkpoint writes the cache BEFORE it runs and replay bypasses it
entirely.
Fix at the cache layer (the authoritative write path): before the write
loop, compute the node ids belonging to groups that will be skipped —
mirroring BOTH skip branches — minus ids also defined in a group that will
be written (duplicate-attribution nodes must not be over-pruned). Each
written group then drops edges whose source/target is a skipped id and
hyperedges whose member list intersects them (whole-hyperedge drop,
mirroring #1895). Pruning runs on the incoming result only, so with
merge_existing=True (the llm.py checkpoint path) the prior cached entry's
valid edges survive the union untouched. Everything is gated on
allowed_source_files being provided, so unscoped callers stay
byte-identical.
Complementary hardening in build_from_json: hyperedges were copied into
G.graph["hyperedges"] verbatim without member validation, so a dangling
hyperedge reached graph.json even from a live (non-cache) extraction.
Members are now remapped via the same normalization the pairwise-edge loop
uses and pruned when they still don't resolve; a hyperedge with no
surviving member is dropped whole with a stderr warning. (Single-member
hyperedges are legal in this codebase — per-file flows in the #1574 tests —
so the drop threshold is zero survivors, not two.)
Tests: scoped save with an edge to an out-of-scope real file, the same with
a ghost source_file, whole-hyperedge drop, unscoped save preserved
byte-identically (raw cache entry compared), merge_existing keeping the
prior slice's valid edges, and build_from_json pruning a dangling hyperedge
member / dropping an all-dangling hyperedge.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-chunk checkpoint feature added merge_existing without test coverage.
Add tests asserting merge_existing=True unions a file's slices across chunks and
the default still overwrites (the authoritative final write in extract).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The AST cache is version-swept but the semantic/LLM cache had no pruning,
so it grew unbounded: it is content-hash-keyed, so every content change
or file deletion leaves a permanent orphan entry (reporter saw 152
entries for 124 live docs). This matters for the committed-cache workflow
where the semantic cache is published for warm CI rebuilds.
Adds prune_semantic_cache(root, live_hashes) and wires it into the end of
the extract path, sweeping cache/semantic/*.json entries whose hash is not
in the live set. The live set is computed from the FULL detected document
set (not the incremental changed-subset, which would delete valid
entries), using the same file_hash recipe save_semantic_cache uses.
Best-effort (unlink guarded), only touches cache/semantic/ (.tmp and
cache/ast/** untouched), and keeps the semantic cache unversioned so
releases never re-bill LLM extraction.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_body_content used substring checks (startswith("---") / find("\n---")),
so `----` thematic breaks and `--- text` prose lines were mistaken for
frontmatter delimiters and everything above them was silently excluded
from the hash -- edits there never invalidated the cache.
Both delimiters must now be a whole line of exactly three dashes
(optional trailing whitespace). For well-formed frontmatter the stripped
body stays byte-identical to the previous implementation, so existing
semantic-cache hashes do not churn.
Co-authored-by: Cursor <cursoragent@cursor.com>
Fixes#777. Relativize manifest keys, .graphify_root, and cache source_file fields on persist; re-anchor on load. In-memory callers still see absolute paths. Symlink round-trip fixed in follow-up commit 8f09326.
style: replace all em dashes with hyphens
fix: explain hidden .graphify/ folder in skill output and README
fix: rename .graphify/ to graphify-out/ so output is visible by default
