Follow-up to #1956: a doc stamped complete on a prior run that truncates
(partial) this run must have its stale semantic_hash cleared so it is re-queued.
Partial files are dropped by _stamped_manifest_files and therefore land in
clear_semantic — pin that composed behavior.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The semantic cache keyed entries on sha256(file content + path) alone, with
no component for the extraction prompt that produced them. After an upgrade
that changed the prompt, every unchanged file was a cache hit and replayed
the older prompt's extraction: the run exited 0, cost.json looked cheap, and
the graph silently carried two prompt generations side by side. The reporter
saw 506 of 512 docs replay an older vintage on a rebuild they expected to be
cold; deleting the whole cache was the only workaround.
output, and invalidating them on every release would re-bill extraction for
unchanged files. Fingerprinting the prompt itself keeps both properties:
entries survive releases that don't touch the prompt, and invalidate only
when it actually changed.
Semantic entries now live under cache/semantic/p{fingerprint}/, mirroring the
AST cache's v{version}/ layout. Both extraction paths pass their prompt: the
Python/CLI path from llm.py's _EXTRACTION_SYSTEM (shared by every backend),
and the skill path via a new prompt_file argument in Step B0/B3 naming the
references/extraction-spec.md the subagents were handed. The fingerprint
normalizes line endings so a CRLF checkout isn't mistaken for a new prompt.
Pre-existing entries predate fingerprinting and have unknowable vintage, so
they are still served rather than re-billing a whole corpus on upgrade — but
check_semantic_cache now warns with the count, turning "no signal at all"
into a visible one. merge_existing refuses to fuse such an entry into a
current-vintage write, which would mix two prompts inside one entry and then
attest the result to a prompt that produced half of it.
Old-fingerprint entries are pruned by liveness only, never swept wholesale
the way stale AST versions are: two hosts with different prompts (verbose vs
compact extraction-spec) can share one graphify-out/, and a wholesale sweep
would have each run delete the other's entries and re-bill on every
alternation. prune/clear/cached_files glob recursively so fingerprinted
entries can't become unprunable orphans (the #1527 failure mode).
The two monolith skills (aider, devin) inline their prompt instead of
shipping a spec sidecar and stay on the unfingerprinted path for now.
- #1925: a missing manifest.json no longer degrades `extract --code-only`
into a full scan that discards the committed semantic layer. An existing
graph.json is a sufficient incremental baseline (detect_incremental treats
an absent manifest as "all new / none deleted"), so out-of-scope doc/paper/
image nodes are preserved while genuinely deleted sources still evict.
- #1920: _stamped_manifest_files now counts hyperedge output, so a doc whose
only chunk output is a hyperedge is stamped instead of re-extracted forever.
- #1923: new namespace/use-aware PHP resolver (mirrors the Java resolver, runs
before the unique-name rewire) so App\Models\Page and an imported
Filament\Pages\Page stay distinct — no more false inherits/imports edge.
- #1922: detect() records ignored files/dirs in a new `ignored` diagnostic
field (the nested-ignore scoping bug itself shipped in 0.9.16 / #1873).
Regression tests added for each; full suite 3325 passed, 3 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`graphify extract --mode deep` over a warm tree was a silent no-op, for
three stacked reasons:
1. The semantic cache ignored mode: deep runs were served standard-mode
entries (and vice versa). check_semantic_cache/save_semantic_cache now
take `mode` (default None, byte-identical when omitted so older
installed callers keep working) and map it to a namespaced kind —
cache/semantic/ for None, cache/semantic-{mode}/ otherwise. The
per-chunk checkpoint in llm.extract_corpus_parallel and the extract /
cache-check consumers thread the run's mode through (cache-check grows
--mode/--deep). cached_files, clear_cache, and prune_semantic_cache
sweep BOTH namespaces; prune uses the same live-hash set for both
(liveness is content-based, mode-independent) so semantic-deep/ can't
regrow the #1527 unbounded-orphan problem and inherits the
files_by_type-derived exclusion gating for free.
2. extract had no --force — the flag was silently swallowed by the
parser's unknown-arg fallthrough. It is now real (plus GRAPHIFY_FORCE
env parity with `update`): force disables the incremental gate so
detection is a full scan and skips the semantic cache READ so every
semantic file re-dispatches, while the post-run save and manifest
stamping still happen.
3. The incremental gate dispatched zero files on a warm unchanged tree
before the cache was ever consulted, so namespacing alone couldn't fix
the repro. In deep+incremental runs the semantic pass now widens to the
full live doc/paper/image set from detect_incremental's files_by_type
(already exclusion-filtered, #1908/#1909) and lets the mode-namespaced
cache decide hits/misses, with a loud count line so the first deep
run's full re-dispatch is visible.
Skill-side threading of mode is deliberately deferred to PR-2; mode
defaults keep generated skills byte-compatible.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two coupled excluded-vs-deleted fixes:
#1909 — incremental extract's prune set was derived from the manifest
alone (manifest - corpus), so a file that became excluded without ever
being manifest-listed (every pre-#1897 graph) kept its stale nodes in
graph.json forever. The prune set is now also derived from the existing
graph's own node source_files reconciled against the post-exclude detect
corpus (_stale_graph_sources), restricted to in-root paths; out-of-root
--include/symlinked entries and remote (://) sources are never pruned.
Relative source_files are anchored against both the scan root and the
--out root (the #555/#1899 relativized form). The --no-cluster
incremental early exit never runs build_merge, so an exclusion-only
change now prunes the raw graph.json in place instead.
#1908 — save_manifest retained any prior row whose file still existed
on disk, so an excluded-but-alive file survived as a permanent phantom
that detect_incremental reported as deleted on every run. Full-scan
callers (extract's saves, watch._rebuild_code's saves) now pass the RAW
detect corpus via a new scan_corpus parameter and in-root rows outside
it are dropped; the corpus is deliberately not the #933 stamp-filtered
files dict, so failed-chunk/omitted-doc rows and --code-only doc rows
survive. Subset saves (changed_paths hooks, #917) keep the seeding
default. detect_incremental now splits manifest rows that left the scan
into deleted_files (gone from disk) and excluded_files (alive but out of
scan), mirroring the watch-side #1795 distinction, and the extract
summaries report the two separately.
Ordering matters: extract's cleanup of newly-excluded nodes previously
worked only through the #1908 conflation, so the graph-source prune
lands together with the manifest split to avoid regressing #1909.
The #933 manifest filter built its extracted-set from node/edge
source_file values, which are root-relative on a fresh extraction, and
compared them against files_by_type entries, which are absolute (from
detect()). The raw string membership test therefore never matched, so
every freshly-extracted semantic doc was dropped from the manifest and
re-queued as changed on the next run; only code files and cache-replayed
docs got stamped. The filter now lives in _stamped_manifest_files(),
which resolves BOTH sides against the scan root before the membership
test — the same Path/is_absolute/resolve normalization the #1890
reconciliation uses in graphify.llm. Genuinely omitted zero-node docs
still have no source_file entry and stay unstamped, preserving the
intentional #933 re-queue behavior.
Regression tests: a CLI extract with a mocked corpus extractor returning
root-relative source_files lands the doc in manifest.json with a
non-empty semantic_hash while a zero-node doc stays out; a unit test
covers relative (fresh) and absolute (cache-hit) source_file shapes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds an opt-in --timing flag that prints per-stage wall-clock timings to stderr so
slow stages are visible on large corpora. extract reports detect / AST extract /
semantic extract / build / cluster / analyze / export (and write on --no-cluster);
cluster-only reports load / cluster / analyze / label / report / export; both end
with a total. A small _StageTimer helper uses monotonic perf_counter.
Off by default and stderr-only, so default output and machine-read stdout/graph.json
are byte-identical (the extract arg loop already swallows unknown flags). Added a
regression test asserting the lines appear with --timing and are absent without it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
graphify extract <path> --out R is documented to send all output to
R/graphify-out/, but the AST and semantic extraction caches were still
anchored at the scanned project (cache_root=target / root=target). That
re-created a graphify-out/ directory inside the project the user
explicitly asked to keep clean.
Anchor both caches at out_root instead. With --out unset, out_root
equals target, so existing in-project behavior is unchanged.
Adds test_extract_out_keeps_project_root_clean: runs extract from a
project root with --out pointing elsewhere and asserts the artifacts
land under --out while the project directory stays byte-identical.
Backend resolution now defers until after file detection. A code-only
corpus (pure AST, zero LLM calls) runs without any API key.
Key validation only fires when needs_llm=True (semantic_files non-empty
or --dedup-llm passed). Error message now names why a key is needed and
notes that code-only corpora need none.
Applied from PR #1123 with one fix: _clear_backend_keys in tests now
also clears AWS_PROFILE/REGION, OLLAMA_BASE_URL to prevent CI flakes
on machines with ambient credentials.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
If `graphify extract --backend claude` runs without the `anthropic`
package installed (pip install graphifyy doesn't pull it in), every
semantic chunk fails inside extract_corpus_parallel. The per-chunk
errors print to stderr but the function returns the empty merged
accumulator anyway, so extract proceeds to write an AST-only graph.json
and exit 0. CI that checks exit status sees success even though the
requested semantic pass produced no nodes.
Track per-chunk success via the existing on_chunk_done callback, which
only fires after a chunk succeeds. If fresh extraction was requested
(uncached_paths non-empty) and zero chunks completed, abort before the
merge/cluster/write phase with exit 1 and a message naming the backend.
The same shape covers other backends with optional SDK deps (openai,
google-generativeai). Cached-only runs are unaffected: uncached_paths
is empty and the guard does not fire.
Tests in tests/test_extract_cli.py simulate the all-failed and
one-succeeded paths by patching extract_corpus_parallel directly.
