`_extract_parallel` spawned a ProcessPoolExecutor whenever there were at least
_PARALLEL_THRESHOLD (20) uncached files, even when the resolved worker count was
1. A one-worker pool buys no parallelism: it still pays a process spawn plus an
IPC round trip per file, and it is the one residual case where the parent's
rebuild watchdog (os._exit) can orphan a worker that is mid-task.
The Windows post-commit hook exports GRAPHIFY_MAX_WORKERS=1, so this was the
default there for any rebuild touching 20+ uncached files.
Gate the pool on the resolved worker count -- after the GRAPHIFY_MAX_WORKERS
override and the win32/floor clamps -- and return False when it is 1. That reuses
the existing contract: the caller already falls back to `_extract_sequential`
in-process when `_extract_parallel` returns False.
Tests: no pool is constructed with GRAPHIFY_MAX_WORKERS=1 and 25 uncached files,
and a multi-worker run still takes the pool path.
Only item 2 of #2173 is addressed here. Item 1 (the `graphify watch` rebuild
timeout) needs a maintainer decision first: `watch()` currently arms no timeout
at all on any platform -- there is no signal.SIGALRM branch in graphify/watch.py
to add an `else` to -- so applying the hook's shape means adding a watchdog that
os._exit(1)s a long-running foreground watcher on a slow-but-healthy rebuild.
That is a behaviour change rather than a Windows-compat fix, so it is left out of
this PR.
#2169 canonicalizes cross-file edge targets to the root-relative file-node
id (the same id the target file gets as a node) instead of the old
absolute-path form. The #2153 baseUrl tests asserted the absolute form;
update them to the canonical id via a _cid() helper. Resolution behavior
is unchanged — only the expected id form.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The decorator reference edges added in #2154 fabricated sourceless stub
nodes for @property/@staticmethod/@dataclass/@functools.wraps and, via
the unique-function rewire, could stamp a false edge onto a corpus's own
def wraps(). Add _PYTHON_DECORATOR_NOISE (mirroring _PYTHON_ANNOTATION_NOISE)
and skip those names, same accepted tradeoff as patch/Mock in annotations.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An incremental `extract --no-cluster` wrote only the changed files over
graph.json with no merge, dropping every node/edge owned by an unchanged
file; and the id-canonicalization pass only learned batch files, so the
changed file's cross-file edges kept absolute-path target ids and
dangled. The raw path now merges the existing graph forward with the same
replace/prune semantics as the clustered path (new merge_raw_extraction
helper in build.py, shared loader), refuses to overwrite a corrupt
existing graph, and the remap pass now also learns in-root edge
target_file paths (existence-gated) so cross-file targets canonicalize.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The hook installers fell back to settings={} on any JSON parse error and
then overwrote the whole file, destroying the user's config (the likely
trigger is a UTF-8 BOM, same class as #2163). All four installers now
read utf-8-sig, refuse to modify a file that isn't a JSON object (naming
the path) instead of clobbering it, back up to <name>.graphify-bak before
any modifying write, skip the write when content is unchanged, and guard
the PreToolUse filter against non-dict entries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Several test files call install/uninstall functions that operate on the
real user home (~/.claude, ~/.gemini, ~/.codebuddy, ~/.copilot), so
running the suite deleted/overwrote the developer's actual config. An
autouse conftest fixture now points HOME/USERPROFILE/LOCALAPPDATA at a
throwaway dir and clears CLAUDE_CONFIG_DIR/XDG_CONFIG_HOME for every
test. Supersedes the per-file sandbox proposed in #2057 (thanks
@erlandl4g for surfacing it).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_LANGUAGE_BUILTIN_GLOBALS and _BUILTIN_NOISE_LABELS covered only JS/TS and
Python, so on Swift codebases framework symbols (Foundation, NSLock, View,
Data, Sendable, ...) ranked as god nodes, and the Swift member-call resolver
could bind a builtin-typed receiver (let d: Data) to a same-named user symbol
in another file — the same phantom-edge shape #1726 fixed for TypeScript.
- extractors/base.py: add Swift stdlib value types, conformance protocols,
Foundation types, and SwiftUI View/Color/Font to _LANGUAGE_BUILTIN_GLOBALS
- analyze.py: add the same set plus framework module names (Foundation,
SwiftUI, UIKit, AppKit, Combine) to _BUILTIN_NOISE_LABELS
- extract.py: _resolve_swift_member_calls now skips builtin receiver types,
matching the guard the TS/Python member-call resolvers already have (#1726)
- tests: god_nodes exclusion (parametrized) + Swift builtin-receiver
no-bind regression + user-type-still-resolves guard
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Eleven cases: the webpacker repro (jsconfig + baseUrl, no paths) for
static, dynamic and extensionless specifiers; the same for tsconfig;
tsconfig winning over jsconfig in one directory; and four preservation
guards that pass before and after — declared paths and directory-prefix
aliases are not shadowed, relative imports are untouched, an absent
baseUrl changes nothing, and an external package is not fabricated.
Non-relative imports produced no edge in a Rails/webpacker project, so
every module under `baseUrl` was orphaned and `affected` answered nothing.
Two defects. First, only `tsconfig.json` was probed, never
`jsconfig.json` — the plain-JS spelling of the same file, which
json_config.py already indexes, so the config's nodes appeared in the
graph while resolution ignored it entirely. Second, `baseUrl` was
consumed only as the base that `paths` targets resolve against, so a
config declaring `baseUrl` and NO `paths` produced an empty alias map and
every bare specifier died.
`_find_js_config` now probes both names, tsconfig winning within a
directory as tsc and editors do. `baseUrl` is exposed separately and used
as a resolution root of LAST RESORT, tried only when no declared alias
matches, so `paths` precedence (#1269, #927, #1531) is untouched. It is
deliberately not modelled as a synthesized `*` alias: that would score
(1, 0) in _match_tsconfig_alias and beat a declared non-wildcard
directory-prefix alias at (2, -len), silently shadowing it. The fallback
also returns a candidate only when it is a real file, so an external
package import is not fabricated into a <baseUrl>/<pkg> edge.
Threaded through the three regex-rescue dynamic-import paths (Svelte,
Astro, TS/TSX) as well as static resolution, since the issue reports both.
Nine cases: the issue's imported-decorator repro, same-file resolution to
the local definition, called and attribute decorators, stacked
decorators, class-qualified method owners, a decorated class, the #1050
@property class-qualification regression guard, and an absence control.
Applying a Python decorator emitted no edge to the decorator symbol, so
`affected <decorator>` answered "No affected nodes found" for every
function it wraps — a silent false negative on reverse-impact queries.
TS/JS already emit these edges via `_ts_emit_decorator_edges`. The Python
`decorated_definition` branch walked its children only to propagate the
parent class id (#1050) and never looked at the `decorator` children.
Python now emits the same shape: a `references` edge with
context="decorator" from the decorated function/class to each decorator
symbol. Owner ids reuse the definition branches' own formulas, so the
edge lands on the node the walk creates. Targets go through
`ensure_named_node`, so an imported decorator becomes a sourceless stub
the corpus rewire collapses onto its real definition. Stacked, called
(`@retry(3)`) and attribute (`@app.route`) decorators are covered; the
attribute form targets the symbol, not the module alias, matching
`_ts_decorator_name`.
The #2139 ${VAR} source handler now also records bash_sources so
resolve_bash_source_edges binds calls into the sourced lib's functions,
not just the source edge. Add the end-to-end oracle plus the previously
untested _bash_source_suffix guards (mid-path $, whole-var, .. traversal
fabricate nothing).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
.gitignore/.graphifyignore/info-exclude read with encoding=utf-8 kept a
leading BOM (U+FEFF) on the first line, so the first pattern (e.g. *.log
or .fable-wt/) silently matched nothing and a BOM'd full-line comment
became a bogus pattern. git strips a single leading BOM; switching the
two ignore read sites to utf-8-sig matches git exactly (strips at most
one, file-start only).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`source "${BENCH_DIR}/lib/x.sh"` (the `dirname "${BASH_SOURCE[0]}"` idiom)
took the bare-name branch, which baked the unexpanded `${BENCH_DIR}` text
into the target id via `_make_id`. That id matches no node, so the edge was
flagged dangling and dropped at export — shared shell libraries looked
orphaned and were split into separate communities.
Detect a `$`-expansion in the source argument, strip the leading
expansion segment(s), and resolve the literal suffix against the script's
own directory (which is what the canonical idiom makes the variable). Emit
`imports_from` as INFERRED only when it resolves to a real file on disk;
skip otherwise instead of emitting a dead id. Bare-name sources keep their
existing behavior.
extract_bash only linked calls whose callee was defined in the same file, so a
call to a function from a `source`d library was silently dropped and shell
scripts looked disconnected from the libraries they use. The resolver for exactly
this, resolve_bash_source_edges, already existed with tests but had no call site
and no raw_calls/bash_sources to work from.
extract_bash now emits `bash_sources` (the files it `source`s) and `raw_calls`
(calls whose callee isn't defined locally), and the pipeline runs them through
resolve_bash_source_edges after the id-remap passes, so caller and function node
ids are final and the already-emitted source edge is de-duped. Resolution is
scoped to the source relationship: a call to an external command never binds to a
same-named function in an unsourced file, and bash raw_calls are excluded from the
generic global-name resolver for the same reason.
The post-checkout body logs with [graphify] throughout, but the new
non-SIGALRM fallback used [graphify hook], so the same timeout read
differently depending on the platform. The post-commit body does use
[graphify hook] everywhere, so only the checkout copy was wrong.
Assert each body uses a single log prefix so this cannot drift again.
The GRAPHIFY_REBUILD_TIMEOUT watchdog in both embedded rebuild bodies was
guarded by hasattr(signal, 'SIGALRM') with no else branch, so on Windows it
never armed and a hung rebuild survived indefinitely -- the exact tail case
the #791 timeout was added to catch.
Fall back to a daemon threading.Timer that prints the same message and calls
os._exit(1). The hard exit is deliberate: the process is already stuck, so a
clean shutdown may itself be blocked, and _rebuild_lock degrades to a no-op
yield on platforms without fcntl, so there is no lock to leave stale.
Assert returncode == 0 so a malformed case snippet fails fast with
stderr instead of silently returning an empty string. Addresses
Copilot review feedback on #2133.
The post-commit/post-checkout hook's interpreter-detection allowlist
silently rejected valid Windows paths (C:\...\python.exe) on Git-Bash.
bash treats a lone backslash inside [...] as an escape that consumes
itself, so the emitted glob never matched a real backslash at runtime,
even though the pattern looked correct in Python's install-time re dialect.
Fix both allowlists (.graphify_python file path and shebang-parsed
launcher path) to emit [!a-zA-Z0-9/_.@:\\-], a doubled-backslash form
verified against bash and dash: it accepts Windows paths and still
rejects ; ` $ injection. The shebang allowlist additionally lacked :
and backslash entirely. Install-time _pinned_python() re is left as-is.
Add shell-runtime tests that execute the emitted case/esac glob directly
against Windows paths and shell metacharacters.
Copilot flagged that the #2137 regression test only exercised the
intra-file suppression path; a regression in the cross-file resolver
guard in extract.py would still pass. Add a cross-file test: class and
function imported from another module, asserting the imported class is
never an indirect_call target while a genuine imported callback still
emits its edge.
Classes are callable via their constructor but are frequently referenced as
descriptive values, not invoked: ORM args (select(Model), db.get(Model, id)),
exception tuples (except (ErrorA, ErrorB)), and string-literal getattr resolving
to a same-named class. The indirect_call guard treated any callable-def target
identically, so these produced false edges (~41% of indirect_call edges in the
reported sample targeted classes), inflating centrality and traversals.
Track class defs in a callable_class_nids set parallel to callable_def_nids,
mark class nodes with a _callable_class attribute, and exclude class targets
from indirect_call emission in both the intra-file (_emit_indirect_by_name) and
cross-file resolver paths. Marker is stripped before output like _callable.
Covers all languages: both class-node creation sites (the generic
config.class_types branch and the Ruby Struct.new/Class.new/Data.define
synthesis) register into the new set.
Tradeoff: suppression is context-blind, so a genuine higher-order class
callback (e.g. map(Point, coords)) also loses its indirect_call edge. This is
far rarer than the false-positive noise removed and matches the issue framing.
Verified before/after on the same input: 4 class-targeted indirect_call edges
-> 0, function callbacks preserved.
Apache 2.0 adds an explicit patent grant, a patent-retaliation clause,
and explicit inbound-contribution terms. MIT's sublicense right permits
relicensing the combined work, so this needs no per-contributor consent;
prior contributions were made under MIT and remain available under those
terms. The original MIT text is retained in LICENSE-MIT and referenced
from NOTICE.
- LICENSE: verbatim Apache License 2.0
- LICENSE-MIT: preserved MIT text for prior contributions
- NOTICE: attribution + pointer to LICENSE-MIT
- pyproject: license = "Apache-2.0" (PEP 639 SPDX), license-files, setuptools>=77
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The inline api.star-history.com SVG rate-limits on a repo this large and
serves a 503 instead of a chart, so the image was permanently broken.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The .graphifyinclude loader and its two matcher helpers had no consumers:
commit df40e4d (#873, index dot dirs) removed the blanket dot-prefix
exclusion and with it the only call sites, leaving detect() parsing the
file on every run and then discarding the result. A .graphifyinclude was
silently a no-op.
Delete _load_graphifyinclude, _is_included, _could_contain_included_path
and the orphaned assignment; add .graphifyinclude to _SKIP_FILES so a
leftover file no longer lands in unclassified; and print a one-time
stderr note when one is present at the scan root, pointing to ! negation
patterns in .graphifyignore. Bump to 0.9.25.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_xaml_csharp_class_nodes did `sorted(root.rglob("*.cs"))` where `root` comes from
_xaml_project_root walking up for a .csproj/.sln marker. On a standalone
extract_xaml call (no corpus boundary), a .xaml under a large/shared parent (a
temp dir, or a big monorepo) could resolve `root` to a broad ancestor, and the
rglob then recursively scanned the entire tree — effectively hanging (it stalled
the test suite intermittently once the shared temp dir grew). Replace the rglob
with an os.walk that prunes noise/hidden dirs (node_modules/.venv/.git/...) during
traversal and caps directories visited, so a real project scans fully while a
runaway root degrades to a fast, partial scan instead of a hang. Regression test
asserts a decoy .cs in node_modules is pruned and the real ViewModel still links.
The heuristic over-matched and silently dropped legitimate files:
- prose `.md`/`.rst` whose topic slug ends in a keyword (privacy-tokens.md,
token-economics.md) — only code was exempt, not prose;
- the unbounded Stage-2 `service.account` substring (regex `.` wildcard) matched
real source (google/oauth2/service_account.py) and prose slugs.
It also MISSED real secrets (.npmrc, .pypirc, secring, .git-credentials, and
case variants on case-insensitive filesystems), which were being indexed.
Fix: move service_account/aws_credentials to the boundary-checked keyword path
(so real source is spared, downloaded key files still drop), add a prose-note
carve-out (multi-word slugs indexed, bare `secrets.md`/`token.md` still dropped),
tighten id_rsa with a left boundary, add the missed secret dotfiles + secring,
lowercase the dir/segment comparisons, and count multi-dot slugs as multi-word.
Net effect is stricter on real secrets and stops the false-positive data loss.
Traceability: `graphify extract` now names the files skipped as sensitive (not
just a count), so a wrongly-flagged file is visible.
The claude-cli backend delivers the extraction schema in the user turn and
trusts the model to emit raw JSON. Newer Claude Code releases treat that
prompt as an agentic task and report the result in prose instead ("Knowledge
graph extracted — 21 nodes, 20 edges…"), so the graph parses empty, reads as
truncation, and adaptive-retry bisects without ever converging.
Pass --json-schema (structured output) when the CLI advertises it — probed
once via `claude --help` and cached — so the object shape is constrained
regardless of prompt framing. Older CLIs that predate the flag keep the
user-turn prompt as a fallback. The `result` envelope still carries the JSON
string, so the parse path is unchanged.
`from pkg import mod as alias` correctly emits the file-level
`imports_from` edge, but every downstream `alias.func()` call was
dropped: the module arm of the cross-file member-call resolver
(#1883, _resolve_python_member_calls in extract.py) matches a call
receiver against the imported module's own file stem, with no
awareness that the local binding in the importing file can be a
different name. `from pkg import mod` / `mod.func()` resolves because
the receiver ("mod") equals the stem; aliasing breaks the match
because the receiver ("alias") never does, and the calls edge
silently disappears while imports_from stays present -- the graph
looks connected, only the symbol-level reverse query comes back
empty. `import pkg.mod as alias` regresses the same way through the
same resolver.
Root cause is a missing propagation, not a missing feature: the local
alias is already parsed correctly in two places (_python_imported_names
in extractors/resolution.py, and the aliased_import branch of
_import_python in extract.py) but discarded before it reaches the
edges the module arm reads.
Fix threads the alias through as a `local_alias` field on the
`imports`/`imports_from` edge (mirroring the existing `target_file`
transient-hint pattern, #1814, including its pop-once-consumed
hygiene so the hint never reaches graph.json):
- _import_python now splits the alias off `import pkg.mod as alias`
and stamps it on the edge instead of only using it to compute the
bare module_name.
- _SymbolResolutionFacts.module_imports gains a 4th `local_name` slot
so the `from pkg import submod [as alias]` submodule path (#1146)
carries the binding through to _apply_symbol_resolution_facts,
which now stamps `local_alias` on the edge whenever it differs from
the submodule's own stem.
- The module arm's receiver match now accepts the tracked alias in
addition to the module's real stem, keyed per (importing file,
target module) so two files aliasing the same module differently
each match their own binding.
- extract() pops `local_alias` off every edge right after
run_language_resolvers runs, and build_from_json drops it from edge
attrs too -- the same two spots target_file is dropped at (#1814),
except the extract()-side pop has to happen AFTER the resolver
reads the field, not at the earlier point _disambiguate_colliding_
node_ids already pops target_file: that function runs before
run_language_resolvers, so popping local_alias there would strip it
before the resolver ever sees it and silently undo the fix above.
Known limitation, left out of scope: two different aliases bound to
the same module in the same file only resolve the last one
registered, since the match is one alias slot keyed per (importing
file, target module) -- not a regression, since the parent resolved
neither.
Adds five regression tests in tests/test_extract.py: the issue's own
shape (`from pkg import gate as m_gate`), its try/except-guarded
variant (the issue's literal repro, confirming the drop is
independent of try: nesting), the adjacent `import mathlib as m` and
dotted `import pkg.gate as g_alias` forms, and the relative `from .
import gate as r_gate` form -- plus an assertion that `local_alias`
never survives into the returned edges. All fail on the parent commit
with the exact missing-edge symptom and pass after the fix. Full
suite: 3554 passed vs. 3549 on the unfixed parent, a delta of exactly
these five new tests; ruff clean. #2080's calls-edge-direction
regression tests (test_serve.py) still pass unchanged.
`graphify explain "<node>"` sorts a node's connections by neighbor
degree and shows only the top 20, appending a bare "... and N more"
for the rest. On a high-degree node (a logging/error helper called
from dozens of places is typical) that leaves the exact question
explain is meant to answer - "who calls this, what's the impact?" -
unanswered for 97% of the callers, with nothing pointing at where
they live (#2009).
The top-20 list and its ordering are unchanged (no behavior change
for nodes at or under the cutoff). Past it, the cut connections are
now grouped by (direction, source_file) with counts and printed
under a "Grouped by file:" section, sorted by count descending, so
the caller/callee distribution is visible without falling back to a
repo-wide grep. The aggregation itself is capped at 20 files with
its own "... and N more files" line for the pathological case where
the remainder is spread across more files than that.
Full per-caller detail behind a flag (--callers --all / --group-by=dir
as proposed in the issue) is left for a follow-up: it's a larger,
more opinionated surface (flag naming, pagination semantics) than
the literal complaint needs, and the default output no longer hides
the answer either way.
Four regression tests in tests/test_explain_cli.py: the pre-existing
truncation notice on a 30-connection node is unchanged, the grouped-
by-file output carries real counts (3 files, one at 4 and two at 3)
that sum back to exactly the cut total (no silent loss), a
byte-for-byte no-op check for nodes at/under the cutoff, and a
boundary check pinning the cutoff itself at exactly 20 connections
(no section) vs exactly 21 (one grouped entry) — the earlier tests
sit well clear of the edge and wouldn't catch a future off-by-one.
`graphify query` builds an undirected nx.Graph (so BFS/DFS can explore
both callers and callees of the seed node), but its text renderer
assumed the BFS/DFS visit order (u, v) was always the edge's
(source, target). On an undirected graph that assumption only holds
when the seed happens to be the caller: seeding on the callee makes
BFS/DFS visit the callee first, so a `caller --calls--> callee` edge
was rendered backwards as `callee --calls--> caller`. graph.json's
own source/target fields stay correct on disk; only the query
rendering was wrong.
`graphify path` and `graphify explain` don't have this problem
because they force directed=True on load (#849, #853), and the MCP
query_graph tool's _load_graph() does the same. Doing that for CLI
`query` too was tried and reverted: forcing a DiGraph makes
G.neighbors() return successors only, so a query seeded on a
leaf/sink node (no outgoing edges) found zero neighbors instead of
its callers — a recall regression, not just a display fix, and it
would make the CLI and MCP query tools diverge in what they discover
even though they'd render direction identically.
Fix instead mirrors the _src/_tgt pattern graphify/build.py already
uses for the same underlying problem (undirected storage loses
direction): the CLI now stashes each link's true source/target on
its edge data as _src/_tgt before constructing the (still undirected)
graph, and _subgraph_to_text renders EDGE lines from _src/_tgt when
present, falling back to (u, v) otherwise. Traversal itself is
unchanged, so recall is unaffected — verified against the unpatched
CLI, the node counts returned for the same seeds are identical before
and after this fix, only the printed edge direction changes.
Adds two regression tests in tests/test_query_cli.py seeding the same
`calls` edge from both endpoints; the callee-seeded case fails on the
prior code with the exact backwards-edge symptom above.
get_neighbors and get_community render every edge/member line unbounded — on a
god node or a large community that floods an MCP client's context window with
100KB+ of text in one tool result. query_graph already solves this with the
~3-chars/token cut in _subgraph_to_text; this applies the same budget rule to
the two line-list tools via a shared helper (_cut_lines_to_budget): cut at a
line boundary, report how many lines were dropped, and point at the narrowing
path (relation_filter / get_node). Default 2000 like query_graph; output under
budget is byte-identical to today.
Souptik Chakraborty