`graphify extract --mode deep` over a warm tree was a silent no-op, for
three stacked reasons:
1. The semantic cache ignored mode: deep runs were served standard-mode
entries (and vice versa). check_semantic_cache/save_semantic_cache now
take `mode` (default None, byte-identical when omitted so older
installed callers keep working) and map it to a namespaced kind —
cache/semantic/ for None, cache/semantic-{mode}/ otherwise. The
per-chunk checkpoint in llm.extract_corpus_parallel and the extract /
cache-check consumers thread the run's mode through (cache-check grows
--mode/--deep). cached_files, clear_cache, and prune_semantic_cache
sweep BOTH namespaces; prune uses the same live-hash set for both
(liveness is content-based, mode-independent) so semantic-deep/ can't
regrow the #1527 unbounded-orphan problem and inherits the
files_by_type-derived exclusion gating for free.
2. extract had no --force — the flag was silently swallowed by the
parser's unknown-arg fallthrough. It is now real (plus GRAPHIFY_FORCE
env parity with `update`): force disables the incremental gate so
detection is a full scan and skips the semantic cache READ so every
semantic file re-dispatches, while the post-run save and manifest
stamping still happen.
3. The incremental gate dispatched zero files on a warm unchanged tree
before the cache was ever consulted, so namespacing alone couldn't fix
the repro. In deep+incremental runs the semantic pass now widens to the
full live doc/paper/image set from detect_incremental's files_by_type
(already exclusion-filtered, #1908/#1909) and lets the mode-namespaced
cache decide hits/misses, with a loud count line so the first deep
run's full re-dispatch is visible.
Skill-side threading of mode is deliberately deferred to PR-2; mode
defaults keep generated skills byte-compatible.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
save_semantic_cache groups nodes/edges/hyperedges by their own source_file
and the write loop skips a group whose path is a ghost (not .is_file(),
silently) or out-of-scope per the #1757 guard — but an edge or hyperedge in
an ALLOWED group that references a node id from a skipped group was still
written verbatim, so on replay (check_semantic_cache) it dangled forever.
The #1895 filter does not cover this: it cleans the in-memory merged result,
while the checkpoint writes the cache BEFORE it runs and replay bypasses it
entirely.
Fix at the cache layer (the authoritative write path): before the write
loop, compute the node ids belonging to groups that will be skipped —
mirroring BOTH skip branches — minus ids also defined in a group that will
be written (duplicate-attribution nodes must not be over-pruned). Each
written group then drops edges whose source/target is a skipped id and
hyperedges whose member list intersects them (whole-hyperedge drop,
mirroring #1895). Pruning runs on the incoming result only, so with
merge_existing=True (the llm.py checkpoint path) the prior cached entry's
valid edges survive the union untouched. Everything is gated on
allowed_source_files being provided, so unscoped callers stay
byte-identical.
Complementary hardening in build_from_json: hyperedges were copied into
G.graph["hyperedges"] verbatim without member validation, so a dangling
hyperedge reached graph.json even from a live (non-cache) extraction.
Members are now remapped via the same normalization the pairwise-edge loop
uses and pruned when they still don't resolve; a hyperedge with no
surviving member is dropped whole with a stderr warning. (Single-member
hyperedges are legal in this codebase — per-file flows in the #1574 tests —
so the drop threshold is zero survivors, not two.)
Tests: scoped save with an edge to an out-of-scope real file, the same with
a ghost source_file, whole-hyperedge drop, unscoped save preserved
byte-identically (raw cache entry compared), merge_existing keeping the
prior slice's valid edges, and build_from_json pruning a dangling hyperedge
member / dropping an all-dangling hyperedge.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-chunk checkpoint feature added merge_existing without test coverage.
Add tests asserting merge_existing=True unions a file's slices across chunks and
the default still overwrites (the authoritative final write in extract).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The AST cache is version-swept but the semantic/LLM cache had no pruning,
so it grew unbounded: it is content-hash-keyed, so every content change
or file deletion leaves a permanent orphan entry (reporter saw 152
entries for 124 live docs). This matters for the committed-cache workflow
where the semantic cache is published for warm CI rebuilds.
Adds prune_semantic_cache(root, live_hashes) and wires it into the end of
the extract path, sweeping cache/semantic/*.json entries whose hash is not
in the live set. The live set is computed from the FULL detected document
set (not the incremental changed-subset, which would delete valid
entries), using the same file_hash recipe save_semantic_cache uses.
Best-effort (unlink guarded), only touches cache/semantic/ (.tmp and
cache/ast/** untouched), and keeps the semantic cache unversioned so
releases never re-bill LLM extraction.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_body_content used substring checks (startswith("---") / find("\n---")),
so `----` thematic breaks and `--- text` prose lines were mistaken for
frontmatter delimiters and everything above them was silently excluded
from the hash -- edits there never invalidated the cache.
Both delimiters must now be a whole line of exactly three dashes
(optional trailing whitespace). For well-formed frontmatter the stripped
body stays byte-identical to the previous implementation, so existing
semantic-cache hashes do not churn.
Co-authored-by: Cursor <cursoragent@cursor.com>
Fixes#777. Relativize manifest keys, .graphify_root, and cache source_file fields on persist; re-anchor on load. In-memory callers still see absolute paths. Symlink round-trip fixed in follow-up commit 8f09326.
style: replace all em dashes with hyphens
fix: explain hidden .graphify/ folder in skill output and README
fix: rename .graphify/ to graphify-out/ so output is visible by default
safishamsi