build_merge now prunes a deleted file's nodes, edges, and hyperedges regardless
of whether their stored source_file is absolute or relative. When the caller
passed no root (the --update runbook), a node that kept an absolute path slipped
past the relative prune set and the deleted file's graph survived silently.
Matching is now form-insensitive (raw, normalized-relative, then an absolute-
identity fallback); a re-extracted file is still never pruned (#1796 preserved).
`graphify extract` also writes the .graphify_root marker after every graph write
so a later build_merge relativizes deleted-file paths correctly even under a
custom --out (its grandparent-of-graph.json fallback pointed at the wrong dir).
Regression tests in tests/test_build_merge_hyperedges_and_prune.py.
Three silent-data-loss fixes in the update/reconcile path:
- #2051: a full `graphify update` now evicts semantic nodes whose non-AST
source (a .txt/.pdf/.png with no code extractor) was deleted from disk.
The corpus sweep only checked re-extractable files, so deleted docs'/images'
LLM nodes survived as authoritative forever. Disk absence is now the deletion
signal; remote/virtual sources (`://`) are left untouched.
- #2056: an incremental rebuild whose change set names a present-but-
unextractable file no longer treats it as a deletion (which evicted its
semantic nodes and disabled the shrink guard). The guard now falls through to
per-source accounting instead of a wholesale bypass on any deletion.
- #2014: code-typed nodes the semantic pass surfaces from within a document now
count as that doc's semantic layer, so a rebuild doesn't re-scan and drop them.
Regression tests for each in tests/test_watch.py.
The barrel-chain resolver keyed (source, symbol) -> target as last-write-wins, so
a barrel re-exporting the same local name from two modules collapsed an importer's
edge onto whichever was learned last — a fabricated wrong edge. Learn into a set
per key and refuse to resolve when a name maps to more than one target; the edge
falls to the dangling-canonical fallback (dropped at build) instead. Adds a
regression test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ollama's own server has no OLLAMA_BASE_URL concept — it reads
OLLAMA_HOST (https://docs.ollama.com/faq#how-do-i-configure-ollama-server).
graphify only ever read OLLAMA_BASE_URL, so anyone who configured Ollama
the way its own docs describe (OLLAMA_HOST) had graphify silently ignore
it and fall back to the localhost:11434 default.
Add _resolve_ollama_base_url(default): OLLAMA_BASE_URL still wins when
set (unchanged behavior, graphify's existing convention for OpenAI-
compatible base_url overrides across backends). Otherwise falls back to
OLLAMA_HOST, normalized into an OpenAI-compatible URL (adds a scheme if
missing, ensures a trailing /v1). Wired into the BACKENDS dict's ollama
entry and the two ollama_url call sites that read the env var directly.
Fixes#1940.
With `graphify extract --out <dir>`, the semantic cache write and read
sides disagreed on both location and key anchoring, breaking the cache
round-trip in two ways:
- Checkpoints (#1990): `_checkpoint_chunk` called `save_semantic_cache`
with only `root=target`, so per-chunk recovery checkpoints were written
under `<corpus>/graphify-out/` while the reader consulted
`<out>/graphify-out/` — creating an unwanted graphify-out/ inside the
analyzed source tree and making every interrupted run re-extract (and
re-bill) completed chunks.
- Final save (#1991): cli.py passed `root=out_root`, so corpus-relative
`source_file` paths resolved against the --out directory, failed
`p.is_file()`, and every result group was silently skipped — the cache
the reader would consult was never populated at all, with no warning.
Fix, following the split the AST cache already uses (#1774):
- `save_semantic_cache` and `check_semantic_cache` gain a `cache_root`
parameter mirroring `load_cached`/`save_cached`: `root` stays the
source-key anchor (content-hash keys, source_file resolution and
relativization), `cache_root` selects where cache files live. Omitting
it keeps `root` for both, so existing callers are unchanged.
- `extract_corpus_parallel` plumbs `cache_root` into `_checkpoint_chunk`.
- cli.py extract passes `root=target, cache_root=out_root` at the cache
read, the checkpoint path, and the final save, and re-anchors the
prune sweep's live hashes to `target` (keys anchored to out_root would
mismatch every entry and sweep the fresh cache as orphaned).
- `save_semantic_cache` now warns loudly when every result group is
dropped because its source_file does not resolve to a real file — the
silent-0-writes failure mode #1991 asked to surface.
Regression tests cover: checkpoint written under cache_root (not the
corpus, no corpus graphify-out/ created), recovery read finds the
checkpoint via the same root/cache_root split, the final-save call shape
writes entries where the reader looks, the all-groups-dropped warning,
and backward compatibility when cache_root is omitted.
Fixes#1990Fixes#1991
ee1df22 narrowed the Claude Code search-guard matcher from "Glob|Grep" to
"Bash" on the premise that dedicated search tools were removed and searches
go through Bash. Current Claude Code routes content search through its
first-class Grep tool (its Bash tool description actively steers away from
shell grep), so the graphify-first nudge never fired on the agent's primary
exploration path and the graph was silently bypassed.
Three-part fix, per the issue's analysis:
- Matcher: "Bash" -> "Bash|Grep" in _claude_pretooluse_hooks. Glob already
fires the read nudge via "Read|Glob", so Grep was the only orphaned tool.
- Guard body: the hook-guard search branch only inspected tool_input.command,
which a Grep call doesn't carry (it has pattern/path/glob). A Grep-shaped
input (pattern present, no command) is now treated as a search — it IS one
by definition — and nudges whenever a fresh graph exists. The Bash
token-matching path is unchanged, and a command-carrying input never
triggers the Grep shape, so non-search Bash calls stay silent.
- Idempotency: "Bash|Grep" added to the four install/uninstall dedup filters
(claude + codebuddy), so upgrading replaces the stale "Bash" hook in place
instead of appending a duplicate — verified against a pre-fix settings.json.
Tests: new regression tests feed Grep-shaped tool_input through
hook-guard search and assert the nudge (with graph), silence (without),
valid PreToolUse JSON, and no blocking; plus a guard that a non-search Bash
command with a stray pattern key does not nudge. Existing matcher assertions
updated across test_search_hook/test_install/test_claude_md/test_codebuddy/
test_hook_strict. Hook+install suites: 397 passed. Full suite: 3224 passed;
the 13 failures are pre-existing on clean v8 in this environment.
Fixes#1986
Claude Code runs command-type PreToolUse hooks through Git Bash by default on
Windows. The resolved exe was emitted as a raw backslash path and quoted only
when it contained a space, so a space-free path like C:\Users\me\graphify.EXE
reached settings.json unquoted. Git Bash treats the unquoted backslashes as
escapes and strips them, producing 'C:Usersmegraphify.EXE: command not found',
so every graph guard silently fails. Normalize the path to forward slashes in
_resolve_graphify_exe (a no-op on POSIX), fixing the Claude, Gemini, and Codex
hook emitters at once.
The PR always wrote gitignore=not no_gitignore into .graphify_build.json, so a
flag-less `graphify extract` after `--no-gitignore` reset it to True and the
git-ignored code silently disappeared again — the exact #1971 complaint. Write
False only when the flag is set (None = leave as-is, mirroring #1886 excludes),
and honor the persisted value for the run when the flag is absent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`graphify prs` reads gh, git, and claude output via subprocess.run(text=True)
with no explicit encoding=, so on Windows stdout is decoded with the cp1252
locale codec. gh emits UTF-8 JSON whose PR titles and logins routinely carry
non-Latin1 bytes (emoji, or the Persian title in PR #1281), and cp1252 has no
mapping for bytes such as 0x81. The capture reader thread raises
UnicodeDecodeError, subprocess.run then returns stdout=None, and the caller dies
one line later on json.loads(None) (TypeError) or None.splitlines()
(AttributeError). Neither is caught by _gh's except clause, so the whole `prs`
command aborts, with a spurious reader-thread traceback on stderr, against any
repo that has non-ASCII PR metadata.
Pass encoding="utf-8", errors="replace" to all five subprocess reads in prs.py
so decoding matches Linux and macOS. This is the decode-side sibling of #1505,
which applied the same change to the llm.py claude-cli subprocess. CI runs Linux
only (a UTF-8 locale), so this path is never exercised there.
Reproduced on Windows 11 / CPython 3.13 with a cp1252 locale:
prs._gh("pr", "view", "1281", ...) crashed with TypeError before and returns the
decoded title after. The added regression tests assert encoding="utf-8" on each
call, mirroring tests/test_charmap_encoding.py, and are red without the fix.
The digest salts content with the path relative to root (portability, #1774),
but the stat-index memo was keyed by absolute path only — so the same file
hashed under two roots (which happens within one `--out` run) served whichever
digest was computed first, making file_hash order-dependent and poisoning the
persisted stat-index across runs. Store one digest per salt under a "hashes"
map; legacy un-salted "hash" entries are never trusted (recompute once). Digest
computation is byte-identical, so existing cache entries still hit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Agents routinely ignore the advisory "run graphify query first" nudge and read
raw files anyway. `graphify install --project --strict` (or `graphify claude
install --strict`) now installs a hook that BLOCKS the first raw source read of a
session via permissionDecision:"deny" with a redirect to graphify query, then
downgrades to the soft nudge — it fires at most once per session (atomic
per-session marker) so it can never strand the agent, and a recent
query/explain/path refreshes a stamp that suppresses it. Claude Code only;
Bash-grep and Glob stay nudge-only; Gemini/Codex/OpenCode are unchanged.
GRAPHIFY_HOOK_STRICT=1/0 toggles at runtime without a reinstall; default installs
are byte-identical (soft nudge).
Also fixes#1840 for the default soft nudge: the guard no longer fires for reads
of out-of-project files, and softens to a non-mandatory nudge when the graph is
stale for the target file. Gating is ~3 stat calls (no corpus walk) and fails
open. Begins the 0.9.19 cycle.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`graphify extract <root> --out <dir>` reduced every node's `source_file` to a
bare filename, so a graph.json could no longer be resolved back to files on
disk by joining source_file onto the scan root.
--out passes the output dir as cache_root to relocate the cache, but that value
also anchored relativization. Every scanned file then failed relative_to(root),
fell through to the #1899 out-of-root fallback, tripped its `updepth > 3`
walk-up guard -- written for a stray ProjectReference, not a whole corpus -- and
collapsed to a basename. On Windows an --out on another drive hit the
cross-drive branch and basenamed unconditionally, which is what the reporter
saw: 0 of ~120k source_files kept a separator. The directory survived only in
the node id slug, lossily (`.`, `/`, `\`, `-`, spaces all map to `_`), and no
other field carried it -- origin_file is stripped (#1516) and the export has no
file table -- so 0% of nodes resolved.
extract() now takes an explicit `root` anchor for source_file/ids/symbol
resolution, which the CLI pins to the scan root independent of where the cache
lives. This completes the cache/anchor decoupling #1774 started and matches
build(root=target), which already anchored on the scan root -- extract was the
lone component keying off --out.
cache_root keeps its fallback-anchor role, so callers that pass the scan root as
cache_root (watch, the no---out CLI path, tests) are unchanged, as are cache
location and out-of-root portability (#1899).
Follow-up to #1956: a doc stamped complete on a prior run that truncates
(partial) this run must have its stale semantic_hash cleared so it is re-queued.
Partial files are dropped by _stamped_manifest_files and therefore land in
clear_semantic — pin that composed behavior.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The #1943 carve-out rescues genuine source under secrets/ / credentials/, but
.tfvars sits in CODE_EXTENSIONS while being Terraform's canonical values store
(routinely real secrets), so it would now be indexed. Add .tfvars to
_SECRET_PRONE_DATA_EXTS so the shared graphable-source predicate drops it in
both Stage 1 and Stage 3; .tf/.hcl stay graphable as genuine infra source.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An explicit "weight": null in the extraction JSON survived .get("weight", 1.0)
(the key is present, so the default never applied) and reached Louvain/Leiden as
None, crashing modularity with a TypeError (graspologic's Leiden even panics on
NaN). build_from_json now coerces weight and confidence_score to float at the
ingest choke point, falling back to 1.0 for null / non-numeric / NaN / inf /
negative values while preserving valid ones. Repairs (not drops) the key so
graph.json round-trips clean and a cluster-only/--update reload never re-ingests
the null.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The PR flagged unverifiable code nodes as confidence="UNVERIFIED", but node
confidence is read by nothing and UNVERIFIED isn't in the validated (edge-only)
confidence vocabulary {EXTRACTED,INFERRED,AMBIGUOUS} — a dead, colliding field.
- Move the flag to a dedicated verification="unverified" node field, off the
confidence key. A node the model already hedged (INFERRED/AMBIGUOUS) is left
untouched, as before.
- Also check the node id's identifiers (not just the label): ids carry the
verbatim symbol, cutting false flags on prettified labels.
- Skip the (potentially PDF-re-extracting) source read entirely when a result
has no code-typed node with a source_file.
- Wire a consumer: diagnose_extraction now counts and reports unverified nodes,
so the persisted flag is actually surfaced.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The semantic (LLM) extraction runs on documents/papers/images; code files are
handled by the deterministic AST engine and never reach the model. A node the
model tags file_type="code" is therefore a symbol it surfaced from within a
document (a name in a fenced code block, an API referenced in a paper), and it
enters graph.json today with no check that the symbol actually appears in the
source the model read. `_out_of_scope` (#1895) only rejects nodes attributed to a
file that was NOT dispatched, so a fabricated symbol on a dispatched file passes.
`extract_files_direct` now verifies every file_type=="code" node whose
source_file was dispatched in the call: if no identifier from its label occurs
(case-insensitive substring) in that file's source bytes, the node's confidence
is downgraded to "UNVERIFIED" (never dropped) and the count is reported to
stderr. Concept/document nodes, nodes without a source_file, nodes attributed to
undispatched files, and labels with no checkable identifier are left untouched.
Best-effort; never aborts extraction; one unreadable file is skipped, not fatal.
The _partial item-marker approach couldn't fire on the most common truncation
shape: a mid-JSON cut parses to zero items, so a sliced document whose second
slice truncated empty was still stamped complete (the PR's own headline case).
- The adaptive-retry give-up sites now record the chunk's own source files in a
result-level _partial_files list, independent of parsed items; it propagates
through _merge_two / the recursion merges / _merge_into so it reaches both the
per-chunk checkpoint and the run-level manifest stamp.
- _partial_source_files unions _partial_files with the item markers.
- save_semantic_cache seeds an empty group for a named partial file with no
items so its entry is stamped partial, and carries a partial prev entry's flag
forward so a later clean slice merging over it can't re-promote it to complete.
- the CLI final save now passes partial_source_files (computed before the save)
so an empty-parse file isn't written back as a complete entry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A chunk whose LLM response is truncated (`finish_reason="length"`) and can't be
recovered by splitting, or that hits the adaptive-retry depth cap, returns a
partial node set. Today that set is checkpointed and written to the content-hash
semantic cache + manifest-stamped as complete, so the incomplete nodes are
served forever until the file content changes or `--force`.
Truncated give-up results are now tagged with an internal `_partial` marker.
`save_semantic_cache` stamps the affected file's entry `partial: True` (detected
from the marker or an explicit `partial_source_files` arg), and `load_cached`
treats a partial entry as a cache MISS, so the file re-dispatches and retries.
The file is also left unstamped in the manifest (like a failed chunk, #933) so
detect_incremental re-queues it on the next incremental run — not only on a full
/ `--force` / content-change run. A file sliced across chunks accumulates via a
partial-aware `merge_existing` peek (`load_cached(allow_partial=True)`) so a
truncated slice is never dropped or silently promoted to complete. Self-heals: a
later complete extraction overwrites the same key with a non-partial entry. The
marker is stripped after the final save so it never leaks into graph.json.
The atomic-write PR left several writers on the old truncate-then-write path.
Route them through write_json_atomic so a crash mid-write can't corrupt them:
- the --no-cluster raw graph.json dump (a core graph.json writer)
- merge-graphs / merge-chunks / merge-semantic output
- .graphify_analysis.json and .graphify_labels.json sidecars
- global_graph.py's global-graph.json and global-manifest.json
write_json_atomic gains an ensure_ascii flag so the raw-UTF-8 writers
(labels, merge outputs) keep byte-for-byte output. Adds tests for the
Windows PermissionError copy fallback and ensure_ascii=False.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
graph.json (the clustered `to_json` write and the `--no-cluster`/merge raw
dumps) and manifest.json were written with a direct `open()`/`write_text`, so a
crash, kill, or disk-full mid-write left a truncated, unparseable file that the
next load or `detect_incremental` then failed on.
Add `write_text_atomic`/`write_json_atomic` in graphify.paths (temp file in the
same directory + `os.replace`; JSON is streamed into the temp, not materialized
as one string) and route the graph.json writers (export.to_json,
cli._prune_graph_json_sources, the merge driver) plus detect.save_manifest
through them. The helper preserves the destination's mode (an atomic replace
never tightens 0644 to mkstemp's 0600), writes through a symlinked destination
(shared-output setups), and falls back to copy-then-delete on a Windows
os.replace lock — matching graphify.cache's existing atomic writer. On failure
the previous file is left intact and the temp removed. Not a power-loss
durability guarantee (no fsync, consistent with the rest of the codebase).
Two gaps the review found in the incomplete-build shrink guard:
- existing_graph_node_count() returned None ("proceed") on a present-but-
unparseable graph.json, so the --no-cluster path could clobber a complete
graph whose file was corrupt/mid-write. It now returns a MALFORMED_GRAPH
sentinel and the caller fails closed, matching to_json's #479 handling.
- A walk that couldn't fully enumerate the corpus (permission-denied subtree,
I/O error) is now treated as an incomplete extraction: detect()/
detect_incremental() already record walk_errors; the extract path consumes
them so a walk-truncated graph can't force-overwrite a complete one.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The clustered write is guarded by to_json's #479 shrink check, but the
`--no-cluster` raw-dump path writes graph.json directly and had no guard, so an
incomplete `--no-cluster` build could still overwrite a larger complete graph
with a partial one — the residual gap noted in the original change.
Add `existing_graph_node_count` in graphify.export (mirrors to_json's guard and
respects the graph-size cap) and, on the raw path, refuse the write with exit 1
before the manifest when the build was incomplete and the new graph has fewer
nodes than the existing one — unless --allow-partial is passed. Both write paths
now enforce the same guarantee.
A full `graphify extract` writes the final graph with `to_json(..., force=True)`,
which bypasses the #479 shrink guard. That is correct for a clean build that
legitimately shrinks (dedup collapse, deleted code), but when this run's
extraction was incomplete — an AST pass crashed, or some semantic chunks failed —
forcing the write lets a partial graph silently overwrite a good complete one.
The build now tracks incompleteness (AST-pass failure, semantic-pass crash, or
succeeded < total chunks) and falls back to the shrink guard (force=False) on an
incomplete run, so a smaller partial graph is refused rather than written. It
exits non-zero before the manifest is written, so the manifest is never stamped
for a graph we declined to write and the next run re-attempts. `--allow-partial`
restores force=True to override intentionally.
Note: the `--no-cluster` raw-dump path writes graph.json directly and has no
shrink guard; this change covers the normal clustered build path only.
`graphify merge-chunks` concatenates agent-written `.graphify_chunk_*.json`
files with only a JSON-decode guard, so an oversized payload or a crafted
node/edge id (e.g. `../../etc/passwd`) flowed straight into the merged graph.
Route each chunk through `load_validated_semantic_fragment`, which stats the
file size BEFORE reading it (a multi-GB chunk can't blow up memory), parses the
JSON, and validates the byte/count caps + the node/edge id charset that blocks
path traversal (#825). An invalid chunk is skipped with a warning (filter
semantics; never abort). Left OUT of build_from_json/load_graph_json on purpose:
those must keep loading valid pre-existing graphs.
Also relax two over-strict checks in the shared validator that would otherwise
silently drop whole legitimate chunks (a relaxation — never a new rejection):
- file_type is no longer gated: build coerces any value via _FILE_TYPE_SYNONYMS
(unknown -> "concept", #840), so synonyms like "markdown"/"tool" the loader
maps must not fail validation.
- the id charset now allows Unicode word chars (build's normalize_id preserves
CJK/Cyrillic/accented-Latin ids); the explicit path-separator/".." check still
blocks directory escape.
Corrects the stale module docstring (the validator serves the devin skill path
and merge-chunks, not skill-opencode/codex).
The semantic cache keyed entries on sha256(file content + path) alone, with
no component for the extraction prompt that produced them. After an upgrade
that changed the prompt, every unchanged file was a cache hit and replayed
the older prompt's extraction: the run exited 0, cost.json looked cheap, and
the graph silently carried two prompt generations side by side. The reporter
saw 506 of 512 docs replay an older vintage on a rebuild they expected to be
cold; deleting the whole cache was the only workaround.
output, and invalidating them on every release would re-bill extraction for
unchanged files. Fingerprinting the prompt itself keeps both properties:
entries survive releases that don't touch the prompt, and invalidate only
when it actually changed.
Semantic entries now live under cache/semantic/p{fingerprint}/, mirroring the
AST cache's v{version}/ layout. Both extraction paths pass their prompt: the
Python/CLI path from llm.py's _EXTRACTION_SYSTEM (shared by every backend),
and the skill path via a new prompt_file argument in Step B0/B3 naming the
references/extraction-spec.md the subagents were handed. The fingerprint
normalizes line endings so a CRLF checkout isn't mistaken for a new prompt.
Pre-existing entries predate fingerprinting and have unknowable vintage, so
they are still served rather than re-billing a whole corpus on upgrade — but
check_semantic_cache now warns with the count, turning "no signal at all"
into a visible one. merge_existing refuses to fuse such an entry into a
current-vintage write, which would mix two prompts inside one entry and then
attest the result to a prompt that produced half of it.
Old-fingerprint entries are pruned by liveness only, never swept wholesale
the way stale AST versions are: two hosts with different prompts (verbose vs
compact extraction-spec) can share one graphify-out/, and a wholesale sweep
would have each run delete the other's entries and re-bill on every
alternation. prune/clear/cached_files glob recursively so fingerprinted
entries can't become unprunable orphans (the #1527 failure mode).
The two monolith skills (aider, devin) inline their prompt instead of
shipping a spec sidecar and stay on the unfingerprinted path for now.
A C-language routine's body from information_schema.routines is just the C
symbol name, so its reconstructed stub (CREATE FUNCTION ... AS $gfx$ name
$gfx$ LANGUAGE c) is unparseable by tree-sitter-sql, and the parser's error
recovery consumes the statements that follow. With FK ALTER TABLEs emitted
last, every references edge was silently lost on any DB with a common
extension installed (uuid-ossp, pgcrypto, pg_trgm, fuzzystrmatch, ...).
Emit the FK block before the routine DDL. FK statements only reference
tables, which are emitted first, so the order is always safe. On a real DB
with 35 tables / 41 FKs / 41 extension routines: 41/41 references edges vs
1/41 before.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- #1925: a missing manifest.json no longer degrades `extract --code-only`
into a full scan that discards the committed semantic layer. An existing
graph.json is a sufficient incremental baseline (detect_incremental treats
an absent manifest as "all new / none deleted"), so out-of-scope doc/paper/
image nodes are preserved while genuinely deleted sources still evict.
- #1920: _stamped_manifest_files now counts hyperedge output, so a doc whose
only chunk output is a hyperedge is stamped instead of re-extracted forever.
- #1923: new namespace/use-aware PHP resolver (mirrors the Java resolver, runs
before the unique-name rewire) so App\Models\Page and an imported
Filament\Pages\Page stay distinct — no more false inherits/imports edge.
- #1922: detect() records ignored files/dirs in a new `ignored` diagnostic
field (the nested-ignore scoping bug itself shipped in 0.9.16 / #1873).
Regression tests added for each; full suite 3325 passed, 3 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An id whose canonical stem contains its legacy stem as a prefix (parent
dir name == file stem, e.g. .claude/CLAUDE.md) re-matched the legacy
branch every build and grew another stem segment, defeating the
same_topology/no_change short-circuits and churning graph.json +
clustering on every zero-delta update. Skip an id that already carries
its canonical stem (mirrors graph_has_legacy_ids); genuine legacy
migration still applies. Also records the #1889/#1918 query-scoring perf
work in the changelog.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PR #1918 replaced _pick_seeds(terms=) with best_seed_by_term= from
_score_query. Update the #1900 German-stopword seed test to the new
single-traversal API and add the missing trailing newline in the
(manual, non-collected) query-scoring benchmark.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add _score_query() producing combined ranking + per-term singleton winners
in a single graph traversal. _pick_seeds consumes precomputed winners
instead of rescoring each term via _score_nodes.
Behavior byte-identical to the legacy T+1 path. Supersedes the rescoring
loop from #1596 while preserving its per-term coverage guarantee; folds
in coverage scaling from #1724 and multiplicity penalty from #1832.
Orthogonal to the trigram prefilter from #1431.
Benchmark (full sweep in #1889):
1k-100k nodes, 1-10 terms: 1.43x-2.43x median speedup
Single-term: no regression (1.70x-1.89x improvement)
Traversals: T+1 -> 1 regardless of term count
Tests: 3221 passed, 3 skipped. Ruff clean. Graphify graph updated.
Refs #1889
`graphify extract --mode deep` over a warm tree was a silent no-op, for
three stacked reasons:
1. The semantic cache ignored mode: deep runs were served standard-mode
entries (and vice versa). check_semantic_cache/save_semantic_cache now
take `mode` (default None, byte-identical when omitted so older
installed callers keep working) and map it to a namespaced kind —
cache/semantic/ for None, cache/semantic-{mode}/ otherwise. The
per-chunk checkpoint in llm.extract_corpus_parallel and the extract /
cache-check consumers thread the run's mode through (cache-check grows
--mode/--deep). cached_files, clear_cache, and prune_semantic_cache
sweep BOTH namespaces; prune uses the same live-hash set for both
(liveness is content-based, mode-independent) so semantic-deep/ can't
regrow the #1527 unbounded-orphan problem and inherits the
files_by_type-derived exclusion gating for free.
2. extract had no --force — the flag was silently swallowed by the
parser's unknown-arg fallthrough. It is now real (plus GRAPHIFY_FORCE
env parity with `update`): force disables the incremental gate so
detection is a full scan and skips the semantic cache READ so every
semantic file re-dispatches, while the post-run save and manifest
stamping still happen.
3. The incremental gate dispatched zero files on a warm unchanged tree
before the cache was ever consulted, so namespacing alone couldn't fix
the repro. In deep+incremental runs the semantic pass now widens to the
full live doc/paper/image set from detect_incremental's files_by_type
(already exclusion-filtered, #1908/#1909) and lets the mode-namespaced
cache decide hits/misses, with a loud count line so the first deep
run's full re-dispatch is visible.
Skill-side threading of mode is deliberately deferred to PR-2; mode
defaults keep generated skills byte-compatible.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
save_semantic_cache groups nodes/edges/hyperedges by their own source_file
and the write loop skips a group whose path is a ghost (not .is_file(),
silently) or out-of-scope per the #1757 guard — but an edge or hyperedge in
an ALLOWED group that references a node id from a skipped group was still
written verbatim, so on replay (check_semantic_cache) it dangled forever.
The #1895 filter does not cover this: it cleans the in-memory merged result,
while the checkpoint writes the cache BEFORE it runs and replay bypasses it
entirely.
Fix at the cache layer (the authoritative write path): before the write
loop, compute the node ids belonging to groups that will be skipped —
mirroring BOTH skip branches — minus ids also defined in a group that will
be written (duplicate-attribution nodes must not be over-pruned). Each
written group then drops edges whose source/target is a skipped id and
hyperedges whose member list intersects them (whole-hyperedge drop,
mirroring #1895). Pruning runs on the incoming result only, so with
merge_existing=True (the llm.py checkpoint path) the prior cached entry's
valid edges survive the union untouched. Everything is gated on
allowed_source_files being provided, so unscoped callers stay
byte-identical.
Complementary hardening in build_from_json: hyperedges were copied into
G.graph["hyperedges"] verbatim without member validation, so a dangling
hyperedge reached graph.json even from a live (non-cache) extraction.
Members are now remapped via the same normalization the pairwise-edge loop
uses and pruned when they still don't resolve; a hyperedge with no
surviving member is dropped whole with a stderr warning. (Single-member
hyperedges are legal in this codebase — per-file flows in the #1574 tests —
so the drop threshold is zero survivors, not two.)
Tests: scoped save with an edge to an out-of-scope real file, the same with
a ghost source_file, whole-hyperedge drop, unscoped save preserved
byte-identically (raw cache entry compared), merge_existing keeping the
prior slice's valid edges, and build_from_json pruning a dangling hyperedge
member / dropping an all-dangling hyperedge.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_rebuild_code fed Markdown/doc files (.md/.mdx/.qmd/.skill) into the AST
quick-scan while _reconcile_existing_graph preserved the same docs'
semantic (LLM) nodes, so every semantic-backed doc was represented twice
(heading nodes + concept nodes) and the graph bloated ~4x vs the CLI
`graphify . --update` path.
Semantic now supersedes AST per doc source: before choosing
extract_targets, read the existing graph's source identities that carry
semantic doc nodes (non-_origin=="ast", gated on file_type=="document"
so pre-#1865 marker-less graphs aren't misread) and exclude those docs
from the quick-scan in both the full-rebuild and changed_paths branches.
They stay in code_files, so the #1795 fail-closed corpus check and the
shrink accounting still cover them — a previously-bloated graph
self-heals on the next full rebuild (the AST ownership rule drops the
stale heading nodes) without the shrink guard refusing the smaller
write. On incremental rebuilds the excluded docs never enter
rebuilt/node-evicted identities, mirroring #1865's tier-scoped edge rule
at the node level, so a doc's semantic nodes and edges survive. Docs
with no semantic layer (and brand-new docs) keep the no-LLM quick-scan
structure from #09b33b7.
.cjs was half-registered: the language maps in build.py and
extract.py already routed it to the JS grammar, but it was missing
from CODE_EXTENSIONS, the extractor _DISPATCH, _LANG_FAMILY, and the
JS resolution/cache suffix sets — so .cjs sources (Electron
main/preload scripts, CommonJS escape hatches in "type": "module"
packages) were silently skipped during a build: classified as
non-code and never handed to the JS extractor.
Add .cjs alongside .mjs in the six lists that gate/route JS files
(detect.py, extract.py _DISPATCH, analyze.py _LANG_FAMILY,
extractors/models.py cache-bypass, extractors/resolution.py resolve
exts, cli.py _HOOK_SOURCE_EXTS), with regression locks mirroring the
.mts/.cts fix (#1607).
Real-world impact: an Electron app whose ~2000-line main.cjs backend
was invisible went from 201 to 294 nodes and 256 to 441 edges on
rebuild.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
tree-sitter-sql parses PL/pgSQL CREATE FUNCTION statements (OUT/INOUT
params, tagged dollar quotes, PERFORM/:= body statements) as ERROR
nodes, and the dispatch loop had no branch for them, so the functions
were silently dropped from the graph.
Handle ERROR nodes inside walk() (they can nest inside a merged
create_function during multi-statement error recovery) and dispatch
top-level ERROR statements to it. The branch regex-scans the raw node
text for every CREATE [OR REPLACE] FUNCTION/PROCEDURE, mirroring the
existing fb_proc_or_trigger and has_error fallbacks, with a name class
that keeps schema-qualified names (exposed.important_function) whole.
The PL/pgSQL body is not scanned for FROM/JOIN references to avoid
junk reads_from targets, and node ids match the clean create_function
branch so seen_ids dedups consistently.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
safishamsi