Under #3503 (#3477) existing nodes from untouched files are protected from
dedup collapse, so the carried-hyperedge remap test must deliver the duplicate
via an incoming re-extracted chunk to fold into the protected survivor.
The symbol-resolution and cross-file import passes call Path.resolve() once
per import/export/use fact and per node — ~44k calls over a few hundred
distinct corpus paths, each walking nt._getfinalpathname on Windows. A
per-(path, cwd) memo collapses that to one syscall per distinct path. On a
364-file self-corpus a sequential extract drops from ~27s to ~14s, graph
byte-identical. The cwd is part of the key so a relative path resolved after
a chdir misses rather than replaying a stale answer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JJbLfztSxm2tH5cJwBbe9q
(cherry picked from commit 5ca30dfd8a)
The symbol-resolution facts pass and the cross-file import pass each parsed
the whole .py corpus, back to back, in the main process — every file read
and tree-sitter-parsed twice for nothing. _parse_python_tree now memoizes on
(path, mtime, size), so the second pass reuses the first pass's tree while a
file changed between watch runs still re-parses. On a 364-file self-corpus a
sequential extract drops from ~27s to ~24s, graph byte-identical.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JJbLfztSxm2tH5cJwBbe9q
(cherry picked from commit dcbab53879)
_walk_python_tree drove the symbol-resolution passes through a recursive
generator: one suspended frame per tree level, every node re-propagated up
the whole chain — ~25M frame resumptions on a 364-file corpus for ~2.8M
nodes. An explicit stack yields the identical preorder (node identity and
order verified) in ~30% less time, and no longer risks RecursionError on a
deeply nested tree. Same rewrite _walk_js_tree already carries.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JJbLfztSxm2tH5cJwBbe9q
(cherry picked from commit e1544e0ad4)
_extract_parallel falls back to sequential on BrokenProcessPool, but an
OSError raised while constructing the ProcessPoolExecutor escaped and
killed the whole extraction (and `graphify update`).
Seen on macOS: ProcessPoolExecutor -> multiprocessing SemLock -> sem_open
fails with OSError(ENOSPC), "No space left on device", once leaked POSIX
named semaphores exhaust kern.posix.sem.max — with the disk nowhere near
full. Sequential extraction needs no semaphore, so hand back False the
same way the BrokenProcessPool branch does, with a warning that names the
OS error rather than the Windows __main__ hint.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit c154c01d50)
Autouse conftest fixture clears every backend env var detect_backend() reads
(API keys, Azure/AWS/Ollama), with a self-guard test that the cleared list
covers every variable the detector reads. (#3484)
#3475 added a PATH probe for a claude-cli labelling fallback; the no-backend
test must neutralize it so it stays deterministic on machines with claude
installed (follow-up to #3475).
Two independent failures, both silent, found while repairing 16 repositories whose
graphs had quietly stopped tracking their code.
1. `graphify label` downgraded graphs while reporting success
`detect_backend()` is key-based, and `claude-cli` is the one backend with no API key
to find — `_get_backend_api_key` can never return one for it, and the fallback loop
excludes it by name. Nothing else looked for it. So on a machine with the Claude Code
CLI installed and no API key, labelling announced "no LLM backend configured",
replaced every real community name with a `Community N` placeholder, and exited 0.
Run over an existing graph, that overwrites good names with worse ones and looks like
success; taking the exit code at face value commits the damage.
The fallback is placed in the labelling path, NOT in `detect_backend()`. Widening
detection itself was the first attempt and it broke
`test_mixed_repo_without_key_errors_and_points_at_code_only`: extraction deliberately
refuses to run without a configured backend and points at `--code-only`, and on any
machine with the CLI present that contract would have silently changed to "shell out
to the CLI instead". That contract is unchanged here, and a test now pins it.
2. The post-commit hook pinned a path with an expiry date
`_pinned_python()` wrote `sys.executable` verbatim. graphify installed from inside a
snap-confined editor lives under `~/snap/<app>/<revision>/`, and snap swaps that
revision on update and prunes the old tree. Every hook pinned to it then dies —
observed across 15 repositories at once when an editor snap moved past revision 259.
Each commit printed "could not locate a Python with graphify installed" and the graphs
stopped following the code, with nothing failing loudly enough to notice.
Two changes: `_pinned_python()` declines to pin a path under a rotating revision
directory, which its docstring already describes as safe degradation; and the
uv-tools probe — which already scans tool environments — additionally scans
snap-confined HOMEs by glob, since an install made inside a snap lands in that snap's
private HOME and the plain `$HOME` roots never see it once the hook runs from an
ordinary shell.
Verified: same 25 pre-existing failures before and after (unrelated: terraform
fixtures, a TS scaling test); passing goes 5110 -> 5119, the nine tests added here.
Each change was mutation-tested — removing it fails a test that the others still pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit c6065e24ea)
`install()` copies the skill files first and writes the always-on
registration afterwards, so an unguarded write to `~/.claude/CLAUDE.md`
left a half-completed install plus a traceback whenever that file could
not be written. That is the normal shape for a declaratively managed
dotfile: nix/home-manager symlinks it into a read-only /nix/store, and
chezmoi or stow with read-only sources do the same.
Route the two registrations inside `install()` (CLAUDE.md and
CODEBUDDY.md, which had the identical unguarded pattern) through one
helper that reports the skip on stderr and lets the install finish, so
the skill files that were already laid down stay usable.
Refs #3474
Co-Authored-By: Claude Code <noreply@anthropic.com>
(cherry picked from commit 47041eef5d)
The Rust extractor gated every non-function item on struct_item,
enum_item and trait_item, so tree-sitter-rust's static_item and
const_item never produced a node: a constant existed in the graph only
through the files that referenced it, never from the Rust that defines
it (0 of 783 names in a jdx/mise snapshot).
Add a branch for both node types: a module-level declaration gets a
node with a file-level contains edge, an associated const inside an
impl is attributed to the impl like a method, and the declared type is
referenced the way a struct field type is.
Fixes#3471
(cherry picked from commit 70ff464064)
Adds an AST branch for create_index in the SQL extractor, emitting an index
node with an 'indexes' edge to its table (cross-file stub when the table is
defined elsewhere), mirroring the existing trigger handling. (#3470)
A workspace-barrel import (`import { evaluate } from '@scope/domain'`)
resolves through `export *` by probing each star target for a symbol
node with the imported name. The symbol map normalizes labels by
stripping the leading dot, so an interface/class METHOD `.evaluate()`
in the first star target satisfied the probe before the exported
function in the second one was reached. The import edge and the
cross-file call then bound to the method node, and the real function
lost its inbound `calls` edge (read as dead code).
Track member nodes separately and skip them in the star-export walk: a
module can only re-export top-level bindings. A member never shadows a
same-named top-level symbol in the map anymore either.
Fixes#3436
(cherry picked from commit 52e4a2b131)
`tag or source.parent.parent.name` is empty whenever the graph is not two levels
below a named directory: `graphify global add /tmp/graph.json`, or any relative
path. Every node is then prefixed `::`, the prune key is `""`, and the manifest
entry cannot be removed because the `remove` subcommand reads an empty tag as a
missing argument.
Infer through `distinct_repo_tags`, the helper `merge-graphs` already uses, so the
two entry points cannot diverge again and the degrade is `"repo"`; absolutize
first so a relative path names the repo dir rather than the cwd placeholder. An
omitted `remove` tag stays a usage error, while an explicitly empty one is now
addressable so stores written by earlier revisions can be cleaned up.
(cherry picked from commit 6a317e6ef2)
build.py annotates chunks and ast_sources with Iterable but never imports it, so
ruff reports four F821 on v8 and typing.get_type_hints() raises on
_tier_replacement_sources, merge_raw_extraction and build_merge. Nothing breaks at
import because `from __future__ import annotations` keeps the hints as strings.
The added test walks build.py's own functions rather than the package: nx under
if TYPE_CHECKING in cross_repo_calls.py, cross_repo_types.py and prs.py is
unresolvable on purpose, while build.py imports every annotated name at module scope.
(cherry picked from commit c315a15c24)
`script_invocation` edges (#1756) only ever resolved a path that is a
bare literal -- `./helpers.sh`, or `bash ./helpers.sh`. Both paths into
that branch go through `literal()`, which rejects by design any token
containing `$`, so the single most common way a shell script calls a
sibling is invisible:
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
"$script_dir/helpers.sh" --flag
`cmd` is None for that command node, the `elif cmd and ...` guard short-
circuits, and no edge is emitted. On a 206-file shell tree that uses the
`script_dir` idiom throughout, `script_invocation` produced *zero* edges
-- the feature was effectively dead for that whole style of codebase.
The `source` branch already solved the same problem for
`source "$DIR/lib/x.sh"` (#2079): strip the leading `${VAR}/` segments
with `_bash_source_suffix` and resolve the literal remainder against the
sourcing file's own directory, because the canonical `script_dir` idiom
makes that variable equal to exactly that. This applies the identical
treatment to the exec position -- read the command-name node's raw text
rather than its `literal()`, strip the same way, and fall through to the
existing `resolved.is_file()` gate.
Nothing new is fabricated. `_bash_source_suffix` still returns None when
the remainder holds another expansion (`"$dir/$name.sh"`) or a `..`
segment, and the on-disk check still means a wrong script-dir guess
emits nothing at all. The three existing negative tests -- missing,
shadowed, and dynamic invocations -- pass unchanged.
Measured on the same 206-file tree: 0 -> 16 edges across 10 distinct
file pairs, every one hand-checked to a real invocation of a real file,
no false positives. Full suite: no change in outcomes (the 25 failures
here are pre-existing and unrelated -- terraform, ollama, skillgen).
Fixes#3416.
(cherry picked from commit 2e112c39fc)
Review follow-up: a wide TS outDir keeping compiled files only under a
late-sorted module dir slipped past the 20-subdirectory cap. The level-two
probe now covers every subdirectory, bounded by total entries scanned.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JJbLfztSxm2tH5cJwBbe9q
(cherry picked from commit ff0a8f6d64)
An aliased use clause (use GuzzleHttp\Client as HttpClient) always
targeted the bare imported name, ignoring the alias entirely. Two
files importing the same external class under different local
aliases, a real pattern for disambiguating two same named classes
from different namespaces, produced two different stub targets for
one class, splitting its identity. The cross file resolver repoints
a stub through the file's own alias to FQN map, keyed by the alias
when the import has one, so a bare name derived stub could never be
found under that key and stayed stuck, unresolved. The alias, when
present, is also what the rest of the file actually references, so
preferring it here keeps this edge's target consistent with those
reference edges too.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017qfdzgbA5KedGEjD1AayNh
(cherry picked from commit 73f9f1667b)
`from . import brain, ledger` in a directory with no __init__.py — a
namespace package, which `python -m pkg.mod` runs without complaint —
emitted nothing: _resolve_python_module_path returns None when there is
no module file to probe, and the #1146 submodule branch sits behind that
None. So every `brain.think()` / `ledger.write()` call in such a repo was
invisible to the #1883 module arm, and the most-called functions carried
in-degree 0 in the graph.
_resolve_python_namespace_dir mirrors the module-path walk (relative
base, scan root, then sys.path-root ancestors per #2072) and returns only
a directory that exists inside the root and has no __init__.py. The fact
collector uses it when the module path resolves to no file; the existing
submodule probe then emits the same imports_from edges a regular package
gets. A name that is not a module file on disk still emits nothing, and
a namespace package binds no symbols of its own.
Measured on a 275-node Python repo laid out this way, same graphify
version with and without the change: calls 201 -> 307, nodes unchanged;
think() 0 -> 15 callers, remember() 0 -> 10, ledger write() 0 -> 20.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KTNEBgonu6ASph6w8R2Wcp
(cherry picked from commit e2ec796bfd)
perf_counter counts wall-clock, which includes time the process is preempted
off-CPU during a busy full-suite run, inflating the larger measurement and
flaking the ratio. process_time counts only CPU work, isolating the algorithmic
scaling. Follow-up to #3402.
extractors/dart.py hardcoded source_location: None at every node and
edge construction site, so a Dart graph could cite zero lines no
matter how many declarations it had (0% across three real repos,
~25k unciteable nodes in one of them).
Threads a line number through add_node/add_edge from each match's own
offset, anchored on the capture group rather than the whole match: a
pattern like the class declaration's leading ^\s* otherwise reports
the line too early when blank lines precede the declaration.
The comment stripping pass also needed a fix first: it deleted a
comment token outright, so a multi-line comment shifted every later
offset (and therefore every later line number) up by the comment's
own height. Blanking it to the same number of newlines instead keeps
src_clean's line count in step with the original file.
Sourceless reference stub nodes (an inherited class, a mixin, an
import target, and similar names the file only references rather
than defines) keep source_location: None, same as before, since there
is no single line in THIS file for a symbol that lives elsewhere.
Every node this file actually defines, and every edge attributed to a
real declaration site, now carries its real line.
Fixes#3365.
(cherry picked from commit 0dbcefc5ba)
`_normalize_ts_import_types` filtered each `import(...)` regex match against
every tree-sitter `type_arguments` byte range with a nested linear scan, in
both the pre-mask and post-mask passes:
matches = [m for m in matches
if not any(start <= m.start() < end
for start, end in original_type_ranges)]
That is O(matches x ranges), and `m.start()` is re-evaluated inside `any()`
for every range — which is why profiles blamed `re.Match.start`/`_sre` and
`sample` appeared to show a pathological regex. The regexes are all linear;
the quadratic caller was the real cost.
Both counts grow with file size in TypeScript that mixes generic calls
(`fn<A, B>(x)` — one `type_arguments` range each) with `import(...)` types,
which is the shape of generated and barrel modules in a large monorepo. On
the filter alone: a 0.8MB/10k-import file took ~21s, 2.1MB/25k ~2.6min, and
4.2MB/50k ~11.8min. With several such files `extract` never finished, and a
worker sat at 100% CPU holding no open files because the work is entirely
in-memory after the read.
Replace both scans with a binary search over start-sorted ranges plus a
prefix-maximum of `end` values, so a lookup touches nesting depth rather
than every range. Type-argument ranges nest but never partially overlap, so
the prefix maximum is a correct stop condition; a *closed* inner sibling
must not terminate the leftward walk, which a naive running-max break gets
wrong. Verified against the old predicate with 500k differential cases,
including non-nested random ranges: zero mismatches.
Normalization output is unchanged — byte-length and line-count preserving,
same masking decisions, and the runtime-vs-type disambiguation from 6baa5e0
still holds.
Also restore the emptiness guard in the second pass: the range list is now
an index tuple, which is always truthy, so it checks the spans it holds.
The new test asserts on scaling (2x input must not ~4x the time) rather than
wall-clock, so it is not machine-dependent. It fails at ~3.4x before this
change and passes after.
(cherry picked from commit b662361d8b)
The assertion recorded the order batches reached their LLM call inside worker
threads, which is nondeterministic; the product merges by cid on the main
thread and concurrency equivalence is covered by a sibling test. Takes only the
flaky-test fix from #3374 (its SECURITY.md/uv.lock changes are handled
separately).
`import X from '#services/x'` — Node's subpath-imports field, and the way
every AdonisJS 6 app reaches its own modules — was never resolved: the JS
resolver only knew tsconfig `paths` and workspace packages. In a real
Adonis app 438/680 alias imports produced no edge and `affected
"ZitadelService"` listed 2 nodes from a stray root script instead of the
7 controllers and the auth middleware that depend on it.
Resolve `#…` specifiers through the nearest package.json `imports` map,
after tsconfig `paths` (explicit aliases keep precedence, #1269) and
before workspace packages. Node semantics: exact key or a single `*`
wildcard, longest literal prefix wins, condition objects go through
_resolve_export_target, and only `./` targets are followed so an external
mapping keeps the ref-namespaced external id (#1638). The map applies to
every file in the package, so a nested tsconfig without the aliases
(e.g. `inertia/tsconfig.json`) no longer shadows it. Cached per start
dir and cleared per extract() run like the tsconfig caches (#2917).
Measured on a 966-file AdonisJS + Inertia repo: alias imports resolved
go from 242/680 to 680/680 in app/ and 17/17 in inertia/.
Claude-Session: https://claude.ai/code/session_011z6vqqu1uujGKjjUE9Dgbg
(cherry picked from commit 691cccc00f)
The no-SIGALRM fallback killing workers before os._exit does not
cover the primary path: on POSIX, where SIGALRM is available, a
TimeoutError raised while the main thread is waiting inside the
ProcessPoolExecutor with-block propagates straight through that
block's own __exit__, which calls shutdown(wait=True) and blocks
until every worker exits. Forever, for a worker stuck in a C-level
call the alarm firing does nothing to stop, the exact shape #3341
was. The except TimeoutError handler that would call sys.exit is
never reached, so the configured timeout bounds nothing in that case.
Reproduced directly: a worker in an unconditional loop, a SIGALRM
after 2s, and the process was still running 15s later.
Kills any live multiprocessing.active_children() inside the signal
handler itself, before it raises, so by the time the exception
reaches shutdown(wait=True) there is nothing left to wait for. Moved
the handler from an inline lambda (which cannot contain a statement,
only expressions) to a named function, matching the no-SIGALRM
fallback's own style.
Fixes#3396.
(cherry picked from commit ed948388db)
os._exit(1) terminates only the rebuild process itself, not any
ProcessPoolExecutor worker it spawned for a large corpus. It skips
every cleanup path, including the pool's own context-manager
shutdown, so a worker still running at the moment the watchdog fires
is orphaned, reparented to PID 1 on POSIX, and keeps going
unsupervised for as long as whatever it was doing takes. A worker
stuck in catastrophic regex backtracking, the exact shape #3341
fixed, was observed surviving 2.5 days that way on macOS.
extract.py already documents this risk for its own 1 worker special
case, but that only covers max_workers == 1, not a real multi worker
pool.
The watchdog fires on a separate timer thread with no reference to
the pool object, but multiprocessing.active_children() enumerates
every live worker process regardless of which thread asks, so the
fallback kills them before exiting itself. Kill, not terminate: a
worker stuck in a C level call like regex backtracking never gets a
chance to act on a signal a Python level handler would need to catch.
Fixes#3396.
(cherry picked from commit 319869e6c1)
_LANGUAGE_BUILTIN_GLOBALS is one union across every language, right
for a BARE call (String(x) really would become a god node) but wrong
for a MEMBER call. open is a Python builtin and Set a JavaScript one,
so session.open() in Swift or _server.Set() in C# named after another
language's builtin was silently discarded outright: no same file
edge, but also no raw_calls entry, so cross file resolution never got
a chance to try it.
A member call carries a receiver, so it is not the ambiguous case the
union guards against. Lets a builtin named member call reach the
existing tgt_nid = None deferral every receiver typed resolver
already relies on for other cases (a capitalized Python receiver, any
Java member call, and so on), so it can only ever reach an edge
through a guarded, receiver typed resolver, never the unguarded
bare name path a real god node would need.
Verified against real cross file setups for Python (class qualified
call) and Swift (field typed receiver, reproducing the reporter's own
AppleReadyStreamFinder.find -> RemuxPlayback.open example), plus that
a same file member call still cannot bind to an unrelated bare
function sharing the builtin name.
Scoped to the shared engine only. go.py and rust.py carry the same
outer filter, and the reported issue asks for the same change there
too, but tracing where their raw_calls entries actually go shows the
shared cross file resolver skips every is_member_call entry
unconditionally (extract.py, the "obj.log() has no import evidence"
guard) regardless of builtin status, and neither language has a
dedicated receiver typed resolver the way Python, Java, C#, Swift,
TypeScript, C++, and Objective-C do. Making the same change there
would be safe but currently inert, so left out rather than shipping a
change with no verifiable effect; worth revisiting once either
language gets its own receiver typed cross file resolver.
Fixes#3381.
(cherry picked from commit c0f9ff57e0)
`_load_graph` forces `directed: True` so renderers can recover stored arc
order (#2309), and `_bfs`/`_dfs` expand through `G.neighbors()`, which on a
DiGraph yields successors only. Seeded on a leaf that is only ever called,
imported and contained, the MCP `query_graph` therefore answered with the
seed alone, while the CLI `query` — which loads the same graph.json
undirected on purpose (#2080) — returned the callers, the test and the
neighbouring modules. Same shape as #1174: one read surface missed a fix
the others got.
Add `_traversal_view`: an undirected copy for the traversal with `_src`/
`_tgt` stashed per edge, so `_subgraph_to_text` keeps rendering
caller->callee regardless of the side the traversal reached the edge from.
An undirected input is returned as-is, so the CLI path is unchanged;
`_load_graph` is untouched because `get_neighbors` and `shortest_path`
rely on direction explicitly.
Regression tests seed the same `calls` edge from both endpoints through
`_load_graph`, in both traversal modes, with and without a context filter;
the callee-seeded ones fail on the prior code with `1 nodes found`.
(cherry picked from commit 341aec5286)
Follow-up to #3367: guards that signature-only trait methods are captured and
that a trait-declared method stays a distinct node from its impl definition.
The generated post-commit / post-checkout block is appended to whatever
hook a repo already has, and other tools chain their logic after it. Every
skip condition in the block was a bare `exit 0`, which in a flat sh script
ends the ENTIRE hook process — so anything after graphify's end marker
was silently dropped on every root commit (HEAD~1 does not exist), every
rebase/merge/cherry-pick, every linked worktree, every GRAPHIFY_SKIP_HOOK=1
and whenever no graphify-capable python could be found.
Both bodies now run inside a subshell `( ... )`, so an `exit` ends only
graphify's section: no rewriting of the seventeen skip sites, no
`return`-outside-a-function hazard for the _PYTHON_DETECT fragment that
tests run standalone, the detached rebuild launch unaffected, and the
hook's own exit status 0 as before. The markers stay outside the subshell
so reinstall/uninstall/status keep finding the block.
(cherry picked from commit 6f09919f10)
`claude -p --output-format json` shares stdout with the CLI's own
subsystems, so the envelope is not always the first thing on it. An
attached MCP server that advertises no tools makes every invocation emit
Client.listTools() called but server does not advertise tools
capability - returning empty list
ahead of the envelope, and `json.loads(stdout)` fails on the whole buffer.
The failure is raised after the model has already answered, so the chunk
is discarded with its tokens spent. On a real corpus this burned 149,582
input and 52,162 output tokens on chunk 1 of 3 and returned nothing, and
the error names the JSON rather than the preamble that caused it, so the
log points at the wrong thing. Anyone with an MCP server configured hits
it on every chunk.
`_envelope_after_preamble` scans for the first `[`/`{` that begins a
valid JSON document. `raw_decode` ignores trailing bytes, so a diagnostic
on either side is tolerated. The fast path is unchanged: clean stdout
still parses on the first `json.loads`, and stdout with no JSON at all
still returns None so the caller raises -- test_raises_on_garbage_envelope
holds.
(cherry picked from commit ede3a9e0a8)
Absence of a `node_kind` stamp is not proof that a node is an entity: an
extractor can mint a node standing for part of its file without stamping one
(detect.py's per-sheet document nodes do exactly that), and the predicate read
those as entities while its name and docstring claimed proof.
Renames it `_reads_as_file_entity` and splits the docstring into what is PROVEN
(not the file's own node — an `_id_prefixes` reconstruction that holds for every
stored-path spelling) and what is ASSUMED (not a section, resting on the
producer stamping `node_kind`). Adds the producer contract next to
`_FILE_STRUCTURE_NODE_KINDS` and a test pinning the stamped/unstamped split, so
the limit is enforced rather than described.
No behaviour change: the same nodes merge as before.
Co-Authored-By: Claude <noreply@anthropic.com>
(cherry picked from commit d123254ebf)
himanshupatro-334