These land together because they are interleaved in extract.py/engine.py.
#2584 (PR #2588, thanks @phudayyy): the 0.9.38 dynamic-import dedupe keyed
only on target, so an in-function import() suppressed the file-level edge
affected follows. Dedupe now keys on the importing file, emitting one
file-level dynamic_import edge per file/target while keeping the call-site
edge.
#2586 / #2417 (PR #2586, thanks @EZZEASY): a Python member call on an
untyped receiver (x.get(...)) no longer binds by name to a same-named
module function. walk_calls now defers non-self/cls/super Python member
calls to the evidence-gated resolver; super().method() still resolves.
Known trade: same-file x = Thing(); x.method() loses its evidence-free
edge (precision over recall, per #2553).
#2589/#2591 resolver arm (in _resolve_objc_member_calls): the @protocol
exclusion and the self.field/_ivar receiver resolution (paired with the
objc.py extractor changes committed separately).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#2543 (adopts PR #2546, thanks @michaelxer): a failed extraction is no
longer stamped in the incremental manifest as up-to-date, so graphify
update retries it instead of skipping it forever; a manifest already
poisoned by the old behavior is healed on the next run; genuinely
unchanged files are not re-processed. Extended to the watch save_manifest
paths too.
#2554 (adopts PR #2555, thanks @annieyii): the claude-cli backend now
inspects the stdout envelope for an is_error result (e.g. a rate limit
returned with exit code 0) and raises it on both the zero and non-zero
exit paths, instead of parsing it as an empty success and bisecting
against a live rate limit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#2411 (thanks @JensD-git): key the C# partial-class merge on assembly
(nearest ancestor .csproj/.fsproj/.vbproj) in addition to namespace and
name, so same-name partial classes in different assemblies stay distinct
while genuine partial halves within one project still merge. Fixes a
0.9.32 regression from #2332.
#2437/#2438 (thanks @aryanbonigala, builds on PR #2439): incremental
rebuilds no longer drop member-call and indirect_call edges from a
changed file into an unchanged target. Re-resolution now sees the
unchanged corpus (nodes, contains/method edges, and _callable markers,
which now persist to graph.json like _origin); edges to a genuinely
removed target are still evicted.
#2444/#2445 (thanks @Baziar, builds on PRs #2461/#2458): a
BrokenProcessPool triggers the sequential fallback instead of being
swallowed per future, a failed worker file is retried sequentially
rather than merged as empty, and a whole-pass AST failure on a fresh
build exits non-zero instead of writing a zero-node graph
(--allow-partial opts into a best-effort partial graph).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Absolute/machine-slug ids still leaked into edge endpoints from producers
the target_file-stamp loop didn't reach. Three fixes: apply id_remap to
raw_calls caller_nid so module-top-level indirect_call sources canonicalize
(#2231); a general backstop in the final relativization pass that learns
_make_id(abs source_file) -> canonical id for every node and rewrites all
node ids and edge endpoints (in-root -> _file_node_id, out-of-root -> ext_),
suffix-aware for __entry; and target_file stamps on bash source/entry edges
so they ride the same canonicalization. No node id or edge endpoint now
carries the scan-root slug for any file in the batch. Builds on #2250.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to #1899. That fix taught the relativization pass to catch a NODE
whose id was minted from an absolute out-of-root path and give it a portable
"ext_"-namespaced id, by matching the node's own id against
_make_id(str(its source_file)). But several cross-file resolvers (Python
relative imports, C/C++/ObjC quoted #include) only ever emit an EDGE for an
import target, no node -- so when that target lives outside the scan root,
the belt-and-braces pass has nothing to learn the old->new id from, and the
edge keeps the raw _make_id(str(absolute_path)) slug forever. The scan path,
including the OS username, ends up in links[].source/target, and differs
between machines/checkouts even though the node id sets are identical.
_import_c also never stamped the transient `target_file` hint (#1814/#2169)
its Python/JS siblings already use for exactly this kind of cross-file
target canonicalization, so it could not benefit from that machinery either.
Fix, in the two places this root cause actually lives:
- _import_c now stamps target_file on a resolved #include, mirroring
_import_python/_import_js.
- The id_remap pass that already walks target_file-stamped edges to
canonicalize in-root-but-unscanned targets now also handles the
out-of-root branch it previously skipped ("leave its ids alone"): an
existing out-of-root target gets the same portable ext_-namespaced id an
out-of-root NODE already gets, so an edge with no node of its own is
covered too. A target that does not exist on disk still stays dangling,
unchanged from before.
_portable_out_of_root_sf moved next to id_remap so both the new edge-target
branch and the existing node-level pass share one implementation.
Four tests in tests/test_extract.py: the out-of-root #include gets a
portable id instead of the raw slug, and its transient target_file hint
never leaks into the returned edge; the same corpus built from two
differently-nested checkout paths produces a byte-identical target (the
reported non-determinism, made explicit); an in-root, same-batch include
still resolves to the real node's id (negative/regression guard); and the
equivalent out-of-root Python relative import is fixed too, since the gap
was in the shared remap path, not language-specific.
Known limitation: this covers every current target_file-stamping resolver
(Python relative imports, C/C++/ObjC #include, JS/TS/Svelte/Astro/Vue
rescued imports). A resolver that mints a path-derived edge target WITHOUT
stamping target_file at all -- none do today -- would still leak; the fix
closes the gap in the shared mechanism, not a per-language allowlist.
#2079 resolved `source "${VAR}/lib/x.sh"` by stripping the leading expansion and
resolving the literal suffix against the sourcing script's own directory. That is
correct for the canonical
`DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"` idiom, but wrong whenever
the variable points somewhere else. With
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
source "${ROOT}/lib/utils.sh"
the real target is <root>/lib/utils.sh, yet a same-named decoy under the script
dir (<root>/scripts/lib/utils.sh) won: a wrong imports_from edge to a real node.
Track top-level variable assignments and use the assigned base for the leading
variable:
- the script-dir idiom, including any number of trailing `/..` hops (and the
`$0` spelling), resolves to the script dir walked up that many times
- a literal value resolves as-is when absolute, or against the script dir when
relative
- anything else -- a value built from other variables, or command substitution we
do not model -- stays untracked, so the previous script-dir guess is kept
Only the base changes. The suffix guards are untouched: an expansion left in the
suffix, an empty suffix, and a `..` component in the suffix are all still
rejected, the edge is still gated on is_file() and still emitted as INFERRED.
Two coverage gaps left by #2141 / #2157, both misses rather than fabrications.
1. Extensionless shebang scripts. _SHEBANG_DISPATCH already routes a
`#!/usr/bin/env bash` file with no extension to extract_bash, so its functions
are indexed, but the cross-file source-resolution pass picked participants by
filename suffix (`p.suffix in (".sh", ".bash")`). A sourced extensionless lib
was therefore excluded and calls into it never bound. Select by shape as well:
the bash extractor tags every node it emits with metadata.language == "bash".
The suffix check stays so an empty .sh file, which has no nodes to inspect,
still participates.
2. Bare `source lib.sh` (no ./ prefix). Only the `raw.startswith((".", "/"))`
branch recorded a bash_sources entry; a bare name fell through to the opaque
`imports` fallback, so neither the source edge nor calls into the lib resolved
even though the file usually sits beside the script. The else branch now binds
a sibling of that name when one exists.
The existence gate from the ./-prefixed branch carries over: a bare name that
resolves to no sibling keeps the old `imports` edge and records no bash_sources
entry, so nothing is invented. is_file() is wrapped against OSError so a name
that is invalid for the platform degrades instead of raising.
Transitive sources (a->b->c) remain unresolved, as the issue notes.
`_extract_parallel` spawned a ProcessPoolExecutor whenever there were at least
_PARALLEL_THRESHOLD (20) uncached files, even when the resolved worker count was
1. A one-worker pool buys no parallelism: it still pays a process spawn plus an
IPC round trip per file, and it is the one residual case where the parent's
rebuild watchdog (os._exit) can orphan a worker that is mid-task.
The Windows post-commit hook exports GRAPHIFY_MAX_WORKERS=1, so this was the
default there for any rebuild touching 20+ uncached files.
Gate the pool on the resolved worker count -- after the GRAPHIFY_MAX_WORKERS
override and the win32/floor clamps -- and return False when it is 1. That reuses
the existing contract: the caller already falls back to `_extract_sequential`
in-process when `_extract_parallel` returns False.
Tests: no pool is constructed with GRAPHIFY_MAX_WORKERS=1 and 25 uncached files,
and a multi-worker run still takes the pool path.
Only item 2 of #2173 is addressed here. Item 1 (the `graphify watch` rebuild
timeout) needs a maintainer decision first: `watch()` currently arms no timeout
at all on any platform -- there is no signal.SIGALRM branch in graphify/watch.py
to add an `else` to -- so applying the hook's shape means adding a watchdog that
os._exit(1)s a long-running foreground watcher on a slow-but-healthy rebuild.
That is a behaviour change rather than a Windows-compat fix, so it is left out of
this PR.
The #2139 ${VAR} source handler now also records bash_sources so
resolve_bash_source_edges binds calls into the sourced lib's functions,
not just the source edge. Add the end-to-end oracle plus the previously
untested _bash_source_suffix guards (mid-path $, whole-var, .. traversal
fabricate nothing).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`source "${BENCH_DIR}/lib/x.sh"` (the `dirname "${BASH_SOURCE[0]}"` idiom)
took the bare-name branch, which baked the unexpanded `${BENCH_DIR}` text
into the target id via `_make_id`. That id matches no node, so the edge was
flagged dangling and dropped at export — shared shell libraries looked
orphaned and were split into separate communities.
Detect a `$`-expansion in the source argument, strip the leading
expansion segment(s), and resolve the literal suffix against the script's
own directory (which is what the canonical idiom makes the variable). Emit
`imports_from` as INFERRED only when it resolves to a real file on disk;
skip otherwise instead of emitting a dead id. Bare-name sources keep their
existing behavior.
extract_bash only linked calls whose callee was defined in the same file, so a
call to a function from a `source`d library was silently dropped and shell
scripts looked disconnected from the libraries they use. The resolver for exactly
this, resolve_bash_source_edges, already existed with tests but had no call site
and no raw_calls/bash_sources to work from.
extract_bash now emits `bash_sources` (the files it `source`s) and `raw_calls`
(calls whose callee isn't defined locally), and the pipeline runs them through
resolve_bash_source_edges after the id-remap passes, so caller and function node
ids are final and the already-emitted source edge is de-duped. Resolution is
scoped to the source relationship: a call to an external command never binds to a
same-named function in an unsourced file, and bash raw_calls are excluded from the
generic global-name resolver for the same reason.
`from pkg import mod as alias` correctly emits the file-level
`imports_from` edge, but every downstream `alias.func()` call was
dropped: the module arm of the cross-file member-call resolver
(#1883, _resolve_python_member_calls in extract.py) matches a call
receiver against the imported module's own file stem, with no
awareness that the local binding in the importing file can be a
different name. `from pkg import mod` / `mod.func()` resolves because
the receiver ("mod") equals the stem; aliasing breaks the match
because the receiver ("alias") never does, and the calls edge
silently disappears while imports_from stays present -- the graph
looks connected, only the symbol-level reverse query comes back
empty. `import pkg.mod as alias` regresses the same way through the
same resolver.
Root cause is a missing propagation, not a missing feature: the local
alias is already parsed correctly in two places (_python_imported_names
in extractors/resolution.py, and the aliased_import branch of
_import_python in extract.py) but discarded before it reaches the
edges the module arm reads.
Fix threads the alias through as a `local_alias` field on the
`imports`/`imports_from` edge (mirroring the existing `target_file`
transient-hint pattern, #1814, including its pop-once-consumed
hygiene so the hint never reaches graph.json):
- _import_python now splits the alias off `import pkg.mod as alias`
and stamps it on the edge instead of only using it to compute the
bare module_name.
- _SymbolResolutionFacts.module_imports gains a 4th `local_name` slot
so the `from pkg import submod [as alias]` submodule path (#1146)
carries the binding through to _apply_symbol_resolution_facts,
which now stamps `local_alias` on the edge whenever it differs from
the submodule's own stem.
- The module arm's receiver match now accepts the tracked alias in
addition to the module's real stem, keyed per (importing file,
target module) so two files aliasing the same module differently
each match their own binding.
- extract() pops `local_alias` off every edge right after
run_language_resolvers runs, and build_from_json drops it from edge
attrs too -- the same two spots target_file is dropped at (#1814),
except the extract()-side pop has to happen AFTER the resolver
reads the field, not at the earlier point _disambiguate_colliding_
node_ids already pops target_file: that function runs before
run_language_resolvers, so popping local_alias there would strip it
before the resolver ever sees it and silently undo the fix above.
Known limitation, left out of scope: two different aliases bound to
the same module in the same file only resolve the last one
registered, since the match is one alias slot keyed per (importing
file, target module) -- not a regression, since the parent resolved
neither.
Adds five regression tests in tests/test_extract.py: the issue's own
shape (`from pkg import gate as m_gate`), its try/except-guarded
variant (the issue's literal repro, confirming the drop is
independent of try: nesting), the adjacent `import mathlib as m` and
dotted `import pkg.gate as g_alias` forms, and the relative `from .
import gate as r_gate` form -- plus an assertion that `local_alias`
never survives into the returned edges. All fail on the parent commit
with the exact missing-edge symptom and pass after the fix. Full
suite: 3554 passed vs. 3549 on the unfixed parent, a delta of exactly
these five new tests; ruff clean. #2080's calls-edge-direction
regression tests (test_serve.py) still pass unchanged.
`graphify extract <root> --out <dir>` reduced every node's `source_file` to a
bare filename, so a graph.json could no longer be resolved back to files on
disk by joining source_file onto the scan root.
--out passes the output dir as cache_root to relocate the cache, but that value
also anchored relativization. Every scanned file then failed relative_to(root),
fell through to the #1899 out-of-root fallback, tripped its `updepth > 3`
walk-up guard -- written for a stray ProjectReference, not a whole corpus -- and
collapsed to a basename. On Windows an --out on another drive hit the
cross-drive branch and basenamed unconditionally, which is what the reporter
saw: 0 of ~120k source_files kept a separator. The directory survived only in
the node id slug, lossily (`.`, `/`, `\`, `-`, spaces all map to `_`), and no
other field carried it -- origin_file is stripped (#1516) and the export has no
file table -- so 0% of nodes resolved.
extract() now takes an explicit `root` anchor for source_file/ids/symbol
resolution, which the CLI pins to the scan root independent of where the cache
lives. This completes the cache/anchor decoupling #1774 started and matches
build(root=target), which already anchored on the scan root -- extract was the
lone component keying off --out.
cache_root keeps its fallback-anchor role, so callers that pass the scan root as
cache_root (watch, the no---out CLI path, tests) are unchanged, as are cache
location and out-of-root portability (#1899).
Completes #1789. Two residual leaks into committed graph.json:
(a) Out-of-root reference targets (a .csproj ProjectReference, .sln
project, or bash `source` pointing outside the scan root) kept an
absolute source_file and an absolute-derived id, because the
relativization post-passes only handled paths under root and silently
left out-of-root ones absolute. They now get a portable walk-up
relative source_file and an `ext_`-namespaced id (basename fallback
for far-outside or cross-drive targets), with edge endpoints remapped.
(b) A symbol whose name normalizes to nothing (minified `$`, a JSONC
`"//"` key) collapsed _make_id(stem, name) to the bare absolute file
stem, leaking the path and colliding with the file node. Guard at the
three mint sites (json_config keys, engine function names) to skip
these no-signal symbols.
Adds regression tests: an out-of-root ProjectReference stays portable
(no scan-path in any id/source_file/edge), and a minified `$` is dropped
while the real function survives.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`module.function()` where `module` is imported produced no calls edge,
while bare-name calls did. The member call was captured, but the shared
cross-file pass skips all member calls and _resolve_python_member_calls
only handled capitalized class receivers, so a lowercase module receiver
fell through. Add a module arm: a lowercase receiver naming a module
imported into the caller's file resolves to the single callable that
module contains. Keyed on stable node ids (imports edge source is the
caller file node; contains maps file -> children), not source_file
strings, so it holds under the cache_root id-remap relativization. Also
drop the no-classes early return that skipped the whole resolver when a
corpus had functions but no class methods.
Guards: only modules imported into the caller's own file match (so
self/obj/local instances never match), and a single-definition god-node
guard on the callable. Adds a positive test and a false-edge negative.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
With no explicit cache_root, extract() wrote graphify-out/cache/ under the
inferred common parent of the inputs — the analyzed source tree — so
scanning a read-only/foreign corpus silently polluted it.
The naive fix (point the root at CWD) breaks two other things that shared
the same parameter: file_hash keys become absolute/non-portable for an
out-of-CWD corpus, and the XAML/C# project-scan boundary would scan CWD
instead of the corpus. So split cache LOCATION from key/id ANCHOR:
load_cached/save_cached gain a cache_root arg for where the dir lives,
while `root` (inferred common parent) still anchors file_hash keys,
source_file relativization, node ids, and the XAML boundary. extract()
now locates the cache at CWD (or cache_root) but anchors on `root`; the
parallel worker tuple carries both. Existing callers passing cache_root
(CLI, watcher) are unchanged.
Adopts @SimiSips's #1802 (the CWD default + the two location tests) and
adds the decoupling plus a regression test that keys stay relative for a
corpus outside CWD — the property the one-line version would have lost.
Co-Authored-By: SimiSips <SimiSips@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
extract_bash only created a cross-file edge for `source x.sh` / `. x.sh`. The
two most common ways one script runs another — `bash x.sh` and `./x.sh` —
produced no edge, so in any repo where scripts invoke each other by execution
the call topology was missing (each script left an isolated file+entry pair).
Emit a `calls` edge (context `script_invocation`) from the caller's entry (or
enclosing function) to the invoked script's entry node, for script-runner
commands (bash/sh/zsh/ksh/dash <path>) and bare `./x.sh`, but only when the
target resolves to a real .sh file on disk — so no phantom edges to missing or
function-shadowed names. Verified end-to-end: the edges land on real target
nodes (no dangling drop at build).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_rewire_unique_stub_nodes gated merge targets through _is_type_like_definition,
which rejects any label ending in `)`. So a function referenced from another
module (passed by name, e.g. FastAPI's Depends(get_db)) left its reference edge
dangling on a sourceless name-only stub while the real def had zero incoming
edges — "who references this function" returned nothing. Class/type symbols were
fine; only functions/methods suffered.
Top-level function defs (label `name()`, not `.name()` methods or `Class.m()`
qualifiers) are now eligible rewire targets, but only when:
- the label key matches exactly one such function corpus-wide (existing
unique-candidate guard — two same-named functions stay unresolved), AND
- the candidate shares a language family with the stub's referrers, so a
Python `get_db` reference can't bind to a unique Go `get_db()` (#1718/#1749
interop guard), AND
- the stub is not used as a supertype (inherits/implements/extends) — you
don't inherit from a function.
Types are unchanged. Regression tests: cross-module function ref binds to def;
cross-language, ambiguous, and supertype cases all correctly left unresolved.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
extract_json emitted `imports` edges for package.json dependencies and
`extends`/`$ref` edges for tsconfig.json to target ids (`_make_id("ref", ...)`
/ `_make_id(key)`) that it never created as nodes. build_from_json drops edges
to unknown node ids silently (that case is filtered out of real_errors), so
dependency and extends structure vanished from the graph on two of the most
common files in any JS/TS repo, surfaced only by diagnose_extraction after the
fact.
The extractor now adds the referenced target as a `concept` node (external ref,
not a corpus file) before emitting each edge, so the edges survive build.
Regression test asserts no dangling endpoints, the concept nodes exist, and the
import/extends edges land on real targets with no self-loops.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When the [sql] extra is absent, .sql files are counted as code and scanned but
extract_sql returns an error result and zero nodes — and the graph builds
"successfully" with the entire SQL corpus missing. Neither existing warning
catches it: #1666's zero-node warning skips results carrying an "error", and
#1689 only covers files with NO extractor at all (.sql HAS a dispatch entry).
extract() now scans per-file results for a "not installed" error, groups the
affected files by extension, and prints a warning naming the extra that
restores the language (pip install "graphifyy[sql]"), via a small
_EXTRA_FOR_EXTENSION map (sql, terraform, dm). The map is only consulted after
an extractor actually reports the dependency missing, so it can't mislabel a
language that has a working fallback.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ensure_named_node() tags the sourceless stub it creates for an
unresolved reference with origin_file, so _disambiguate_colliding_node_ids
can tell one file's unresolved reference apart from another's instead of
merging every file's same-named reference onto one shared bare id
(which can then collide with an unrelated same-named real definition
anywhere else in the corpus, since ids are case-normalized and global).
Five call sites duplicated that stub-creation logic inline instead of
calling ensure_named_node() -- Ruby's `Class.new(Super)` and
`class Foo < Base` inheritance, Python inheritance, Kotlin delegation
-specifier inheritance/conformance, and C++ base_class_clause
inheritance -- and none of them were updated when origin_file was added,
so all five still produce the un-disambiguated bare-id stub the fix was
meant to eliminate.
Four of the five sit directly inside _extract_generic, where
ensure_named_node() is already in scope as a closure, so they're
switched to call it directly. The fifth (Ruby's `Class.new(Super)`) is
handled by a separate helper, _ruby_extra_walk(), which doesn't have
that closure in scope; that one gets the same origin_file tag added
directly to its own inline stub dict, matching what ensure_named_node()
already does, without changing the helper's signature.
(A sixth occurrence of the same inline pattern exists in extract_apex(),
a fully separate regex-based extractor with no ensure_named_node()
equivalent of its own -- left out of this fix, which is scoped to the
shared _extract_generic path and its one directly-affiliated helper.)
Added a regression test: two different C++ files each inheriting from
the same undefined base class must produce two distinct stub nodes, not
one shared one. Fails on main (one shared 'base' id for both files),
passes with this fix.
`.m` is shared by Objective-C implementation files and MATLAB, but the extractor
dispatch routed every `.m` to extract_objc unconditionally. Feeding real MATLAB
to the Objective-C tree-sitter grammar yields root.has_error and garbage
nodes/edges — worse than skipping, because it pollutes the graph with wrong data.
_get_extractor now content-sniffs `.m` the same way `.h` already is: a genuine
Objective-C `.m` carries an ObjC directive (@implementation/@interface/@import/
#import) and still routes to extract_objc; a `.m` without one (MATLAB, Octave)
gets no extractor, so it is surfaced by the no-AST-extractor warning (#1689)
instead of mis-parsed. `.mm` is unambiguously Objective-C++ and is left untouched.
This stops the wrong-by-omission garbage; wiring a real tree-sitter-matlab
extractor (the issue's primary ask) remains a follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Intermediate progress lines count against len(uncached_work) ("X/Y uncached
files"), but the final line switched to total_files ("Y/Y files"), which includes
cached hits and files with no extractor that never entered uncached_work. On a
large corpus with unsupported-language files, the total jumped upward right after
99% with no explanation. Both the parallel and sequential final lines now report
the same uncached_work denominator, so the count no longer appears to change
mid-run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extensions like .r/.R (also .ejs, .ets) are in CODE_EXTENSIONS, so those files
are classified as code and counted in the scan, but there is no entry for them
in the extractor dispatch — so they produce zero nodes and are silently absent
from the graph while the CLI still reports success. The #1666 zero-node warning
deliberately skips them (it only fires when an extractor exists), so nothing
surfaced the gap.
extract() now emits a warning listing the offending extensions and counts
("N file(s) are classified as code but graphify has no AST extractor ...: .r (2)")
so a primarily-R (or .ejs/.ets) corpus no longer looks fully mapped when it is
not. Grouped by extension, fires only for files actually present with no
extractor (today: .ejs, .ets, .r). Adding real grammars for these remains the
follow-up; this removes the silent-data-loss now.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
detect.classify_file already labels extensionless files with a bash/python/
node/... shebang as CODE via _shebang_interpreter, but _get_extractor
dispatched purely on path.suffix — so a CLI entry point like `devctl` or
`manage` was detected as code and then silently contributed zero nodes to
the graph (its doc-referenced symbols stayed dangling stubs).
Resolve extensionless files through the same _shebang_interpreter and a
new _SHEBANG_DISPATCH map. Only interpreters with a real extractor are
mapped (python/bash-family/node/ruby/lua/php/julia); detect's wider set
(perl, fish, tcsh, Rscript) stays unmapped and skipped rather than being
mis-parsed by a wrong grammar.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When a callee had exactly one same-named definition repo-wide, the cross-file
resolver emitted a `calls` edge at INFERRED/0.8 even with no import path between
caller and callee. On a monorepo this fabricated dependencies: a 14-package repo
showed `platform`/`sidecar` depending on `registry-protocol` purely because it
exported generically-named symbols that unresolved calls collapsed onto.
JS/TS modules have no implicit cross-module scope, so a cross-file call is real
only if the caller imported it. Direct JS/TS cross-file `calls` attribution is
now gated on import evidence and left unresolved otherwise. Scoped to direct
calls: other languages keep the #1553 single-candidate resolution (C/C++
headers, Ruby autoload, same-package implicit scope), and the indirect_call path
(already INFERRED + callable-gated) is untouched.
Also hardens caller/candidate -> file mapping to resolve via the node's
`source_file` string (identifying the file node by its basename label) instead
of `relative_to(root.resolve())`, which threw on a path-resolution/symlink
mismatch and fell back to a non-matching absolute id — spuriously failing import
evidence. This both makes the new gate safe and fixes legitimate cross-file
calls being mislabeled INFERRED instead of EXTRACTED.
Full suite: 2898 passed, 3 skipped. Verified via CLI on the reporter's repro
(phantom dropped) and a control (imported call resolves EXTRACTED).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The cross-file call resolver bailed (#543/#1219 god-node guard) whenever a
bare callee name had 2+ definitions without unique import evidence — so a
single same-named test mock (or any same-named symbol) dropped the real
`calls` edge, erasing the call graph wherever a mock existed (the reporter
saw a 76-stub Pester suite wipe everything).
Replace the blunt bail with a smarter guard: when a name is ambiguous and
import evidence doesn't resolve it, apply tie-breakers — non-test
preference (a shared, segment-aware _is_test_path classifier) then path
proximity — and emit an INFERRED edge ONLY if exactly one candidate
survives, else keep bailing. A real def + a test mock resolves to the real
def; two genuine non-test defs still bail (god-node guard intact, no
fan-out). Wired into both the extract.py pass and the symbol_resolution.py
copy via the shared classifier.
Reported by @Schweinehund.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
0.9.0 made the node-ID stem the full repo-relative path, but normalize_id collapses
every non-word run to "_", so the path separator is indistinguishable from inner
punctuation: foo/bar_baz.py and foo_bar/baz.py both normalized to foo_bar_baz and
still silently merged (the residual of #1504). The existing _disambiguate_colliding
_node_ids salt didn't help — it salted with _make_id(source_key, old_id), which
re-normalizes the path with the same lossy recipe, so the two colliders produced an
identical salted id.
When two distinct source paths' naive salts still collide, append a short stable
sha1(source_key)[:6] — injective over distinct paths — so they separate. Computed in
code from source_file (never trusted from the LLM), so AST<->semantic parity holds.
Blast radius: minimal/non-breaking — only the actual residual colliders get a hash
suffix. Non-colliding ids (the 99%, incl. the common #1504 case like two README.md
in different dirs) are byte-identical to 0.9.0 (verified: src/auth/session.py ->
src_auth_session, docs/v1/api/README.md -> docs_v1_api_readme unchanged). This is a
0.9.1 patch, not another migration.
Reported by @sub4biz (#1522). Regression tests cover both the collider-separation
and the non-collider-unchanged cases.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The #1462 same-label cross-file stub disambiguation (the origin_file key) only
existed in the generic extractor, so the six dedicated extractors — Julia,
Fortran, Go, Rust, PowerShell, ObjC — still collapsed same-named imported-type
stubs from different files into one conflated bare-id node (a false cross-package
link). Each now sets origin_file on its sourceless stub, identical to the generic
extractor; the generic _node_disambiguation_source_key consumes it, so two files
importing the same type stay distinct while source_file stays empty (the #1402
rewire onto a real definition is unchanged).
Ported from PR #1515 by @TPAteeq. Must ship with #1516: this widens origin_file to
six more languages, and #1516 is what strips it from graph.json. Verified: a 2-file
Go corpus now yields 2 distinct Widget stubs AND graph.json carries no origin_file.
Resolved a test-insertion conflict with #1516 by keeping both tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The origin_file disambiguation field (#1462) is internal — it's consumed by the
colliding-id pass and must not be persisted. In 0.9.0 it shipped into graph.json
as an absolute, machine-specific path (the same portability bug class as #555,
#932), breaking on clone/move. It's now popped at the single extract() output
chokepoint, AFTER _disambiguate_colliding_node_ids has consumed it, so all persist
paths (clustered, --no-cluster, cluster-only) emit clean output. _origin is kept
(the incremental watcher relies on it, #1116).
Ported from PR #1516 by @TPAteeq. Verified: fresh extract (clustered and
--no-cluster) produces graph.json with zero origin_file fields; the #1462 same-
label disambiguation still works (it runs before the strip).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
BREAKING node-ID format change. The stem that prefixes every node id was the
immediate parent dir + filename, so same-named files in different directories
collided into one last-writer-wins node and silently dropped graph content
(docs/v1/api/README.md and docs/v2/api/README.md both -> api_readme). The stem is
now the full repo-relative path (docs_v1_api_readme vs docs_v2_api_readme);
top-level files are unchanged (setup.py -> setup).
- extractors/base.py::_file_stem -> full path (as_posix; make_id collapses
separators). The two hand-copied stems (symbol_resolution, mcp_ingest) now
import the canonical one, so they can't drift again.
- llm.py system prompt + extraction-spec fragments aligned to the same rule,
fixing the #1509 AST<->LLM divergence (prompt used zero parent dirs -> ghosts).
Regenerated + blessed the per-host specs.
- build_from_json deterministically re-keys every non-AST node id from its
source_file via the new stem (and registers old-stem aliases), so a cached or
pre-migration semantic fragment carrying an old short id reconciles with the
AST node instead of spawning a ghost. The semantic cache is unversioned, so
this code-side re-key (not LLM prose) is what makes it survive the format change
with no re-bill. AST nodes are already canonical and skipped.
- Migration: existing graphs migrate automatically on the next build/update (the
re-key runs in build_from_json, including the whole graph fed back through
build_merge); `graphify extract --force` for a clean rebuild. Neo4j persisted
stores need a re-import; GraphML layouts go stale.
Full suite green (2505 passed); #1504 collision fixed and old-id fragments
re-keyed verified by smoke. Lands on v9 for review before a 0.9.0 release.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two files that both import and use the same type as a bare name (e.g.
`from pathlib import Path` used as a type hint in a.py and b.py) collapsed into one
node, even with no project definition to anchor them. The referencing file is now
recorded in an internal `origin_file` field and used as the disambiguation key when
_disambiguate_colliding_node_ids splits same-id nodes — while source_file stays
empty, so the corpus-level rewire still collapses these stubs onto a real project
definition when one exists (the #1402 path is untouched). origin_file is read only
inside disambiguation; the rewire and the Java/C# type resolvers key off
source_file as before.
Ported from PR #1479 by @jiangyq9 (squash-merged onto current v8; the branch's
stale base made the raw diff appear to revert later features — the 3-way merge
applies only the surgical origin_file change). Resolved a test-adjacency conflict
with #1500 by keeping both cross-file tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Go's ensure_named_node emitted a SOURCED stub for a cross-file type reference,
unlike the generic/ObjC/Java extractors which emit sourceless ones. A sourced stub
bakes the referencing file's path into the node id at disambiguation time, which
blocks the corpus-level rewire from collapsing it onto the real definition — so a
type defined in one Go file and referenced from two others produced three nodes
(the real def plus two phantom per-file duplicates). Go now emits a sourceless
stub like the other languages, so the rewire collapses them to a single node.
Ported from PR #1501 by @TPAteeq. Closes the #1402 gap for Go.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cross-class qualified static calls like `CustomerTaskActions.approve(...)` did
not produce an EXTRACTED `calls` edge. Two compounding causes:
1. The shared cross-file pass skips all member calls (the #543/#1219 god-node
guard against bare `obj.method()` name collisions), and there was no Python
receiver-based resolver to recover the qualified ones.
2. When the called method shared its name with an in-file node — e.g. a viewset
action `approve()` delegating to a service `Service.approve()` — the in-file
bare-name lookup matched the caller's own node (tgt == caller), so the call
was silently dropped before any raw_call was recorded.
Fix: capture a simple-identifier receiver in the call walk (new
`call_accessor_object_field`, set to `object` for Python), defer capitalized-
receiver member calls to a new `_resolve_python_member_calls` pass (mirroring the
Swift resolver), and emit an EXTRACTED edge only when the receiver resolves to
exactly one class that owns the method (single-definition god-node guard).
Instance/module calls (`self.x()`, `obj.x()`, lowercase receivers) are unaffected.
Tests: cross-class resolution, the same-method-name collision shape from the
issue, instance-call non-over-connection, and the ambiguous-class guard.
Full suite 2337 passed; skillgen --check clean; ruff clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A class defined once but referenced via type annotations in N other files appeared
as 1+N nodes — the extras carrying the referencing file's path (with extension)
baked into the id (e.g. pkg_a_py_thing). ensure_named_node's cross-file fallback
called add_node, which stamps the referencing file as source_file; that sourced
stub then collided in _disambiguate_colliding_node_ids (baking the .py path into
the id) and _rewire_unique_stub_nodes skipped it (a node with a source_file is
treated as a real definition, not a stub).
The fallback now emits a SOURCELESS stub (mirroring the inheritance-base path), so
disambiguation ignores it and the rewire collapses it onto the canonical
definition. The helper is duplicated across all six language extractors, so the
fix is applied to all six. Genuinely-defined duplicates (same name, different
files) still stay separate — only cross-file references collapse.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extract JS/TS this.X=, exports.X=, prototype, class arrow fields, and function expressions (closes#1322). Validated locally against v8: full suite 2069 passed.
collect_files ran one full recursive rglob pass per supported extension
(~85 walks), descending into node_modules/.git/venv and filtering those
paths only after enumerating them, and re-evaluated gitignore patterns
per file without the shared cache.
Replace the loop with a single os.walk that prunes noise dirs (and, when
no negation patterns exist, ignored dirs) in place -- the same pattern
the follow_symlinks=True branch and detect.py's scan walk already use --
and pass the shared _is_ignored cache. Extension matching switches to
p.suffix in _EXTENSIONS, matching the follow_symlinks branch and
preserving the .f/.F Fortran case distinction.
Synthetic benchmark (200 source files + 5,000-file node_modules):
0.295s -> 0.007s, identical result set.
Tests: parity oracle against the old implementation on the fixtures and
on a synthetic tree (noise dirs, hidden dirs, gitignore negation), plus
a scandir-counting test asserting each directory is read at most once
and noise dirs are never entered.
Co-authored-by: Cursor <cursoragent@cursor.com>
- export.py: guard to_obsidian/to_canvas against dangling community member IDs
(KeyError crash when a node in communities dict is absent from graph, #1236)
- detect.py: NFC-normalize path before hashing Office sidecar filename to fix
macOS NFC/NFD mismatch causing --update to re-extract all Office files (#1226)
- extract.py: add _is_config_json() to skip data JSON files (only extract
package.json, tsconfig.json, eslint, deno, JSON Schema etc.) eliminating
561 orphan key-nodes on large repos (#1224)
- llm.py: add GRAPHIFY_LLM_TEMPERATURE env var + _resolve_temperature() helper;
auto-omit temperature for o1/o3/o4/gpt-5 reasoning models that reject temp=0;
mirrors GRAPHIFY_MAX_OUTPUT_TOKENS precedence pattern (#1191)
- tests: 20 new regression tests across obsidian, detect, extract, llm_backends
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add 'export_statement' to import_types for JS/TS/TSX configs
- Extend _import_js to detect 'export { X } from ./mod' re-exports
- Emit 're_exports' edges linking barrel files to source symbols
- Preserve walk-through for 'export function/const' declarations
- Add 're_exports' to clean_edges allowlist for cross-file edges
Tested on a 976-file Next.js codebase: detects 162 re_exports edges
and 5760 symbol-level imports (previously 0 for both).
* feat(bash): harden extractor — literal filtering, entrypoint nodes, AST-ancestry-aware command detection
Builds on tree-sitter-bash extractor from #866. Two correctness/security
improvements to bash extraction in graphify/extract.py:
1. Reject command/process substitutions at extraction time. Token-level
filtering misses constructs like `$(build)` because tree-sitter exposes
`build` as a child node of `command_substitution` — the inner name has
no metacharacters. Added `is_inside_expansion(node)` that walks
`node.parent` until it finds `command_substitution` or
`process_substitution`. Used as a gate in both `walk` and `walk_calls`.
Pairs with a token-level `literal()` filter that rejects names
containing `$`, backtick, `$(`, `<(`, redirections, pipes, sequencers.
2. Entrypoint node. Every .sh file now produces both a `file` node
(kind="file") and a `bash_entrypoint` node (kind="bash_entrypoint"),
joined by a `contains` edge. A separate top-level `walk_calls(root,
entry_nid, ...)` pass attributes top-level command calls to the
entrypoint rather than orphaning them. Matches the entrypoint pattern
other-language extractors use. Node metadata gains language+kind.
Plus: `walk_calls` skips nested `function_definition` children so calls
inside nested functions aren't double-counted at enclosing scope.
Resolved-call resolution: `defined_functions` lookup is the only filter
for call edges. User-defined functions named like external commands
(install, find, git, ...) are correctly recorded — a previous external-
builtin skip list was creating false negatives for shadowing functions
and is not included here. Skip list belongs with raw/unresolved call
recording (not in this PR).
Devtools (bundled): pyproject.toml gains [dependency-groups] dev (ruff,
pyright, pre-commit, hypothesis, pip-audit) plus minimal [tool.ruff],
[tool.ruff.lint], [tool.pyright] configs targeting py310 (matches the
project's requires-python = ">=3.10").
Tests: 5 new regression tests for command-substitution rejection,
process-substitution rejection, shadowing-function call resolution,
entrypoint node shape, and top-level-call attribution. 826/826 pass
(was 821); 15/15 bash-relevant tests pass (was 10).
* feat(detect): parse macOS/BSD and GNU env(1) shebang option forms
Upstream's _shebang_file_type parses shebangs via line[2:].split() and only
handles `#!/usr/bin/env <interp>`. Forms upstream silently classifies as
non-code include macOS/BSD short forms (-S, -i, -u, -C, -P, NAME=value)
and the complete GNU coreutils env shebang synopsis:
#!/usr/bin/env -[v]S[option]... [name=value]... command [args]...
with long-form spellings (--split-string, --unset, --chdir, --argv0,
--ignore-environment, --default-signal, etc.), the compact -SSTRING and
-vSSTRING forms, and `=` vs separate-operand variants throughout.
Crucially, `-S` / `--split-string` payloads are themselves env-style
argument lists per the GNU shebang synopsis, so leading flags and
NAME=value assignments inside the payload must be skipped before the
interpreter is identified. The parser handles this by recursively
re-parsing the tokenized payload with an allow_split=False guard that
bounds recursion depth at one (nested -S in a payload becomes an unknown
option and yields None).
Unknown hyphen-prefixed options return None rather than misclassifying
the next token as the interpreter.
_shebang_file_type becomes a 4-line wrapper. Read buffer raised 128 -> 256
to accommodate longer env -S strings.
Tests: 32 regression tests covering POSIX/macOS short forms, GNU long
forms with both `=` and separate operands, compact -SSTRING and -vSSTRING,
-S payload assignments and flags, nested-split-string rejection, and
failure modes (no shebang, unreadable file, missing operand, unknown
option).
* fix(skills): enforce semantic fragment validation in OpenCode + Codex merges (#825)
Closes#825. Adds graphify.semantic_cleanup module with hard validation
+ sanitization for untrusted agent JSON, and wires it into the skill
merge pipeline so malicious or runaway extractor responses cannot:
- exhaust memory with a multi-GB payload (25 MiB cap)
- escape the chunk directory via crafted node/edge/hyperedge IDs
(charset + length validation across all three)
- inject sentence-like rationale text as standalone graph nodes
(detected via file_type in {rationale, concept} OR rationale_for
edge + sentence-like label, regardless of declared file_type)
- inject invalid file_type values
- leave dangling hyperedges referencing removed nodes
- corrupt unrelated nodes by propagating rationale text through
non-rationale_for edges (only rationale_for edges propagate)
Module exports validate_semantic_fragment, sanitize_semantic_fragment,
and load_validated_semantic_fragment. Wired into skill-opencode.md and
skill-codex.md at three merge points each (chunk merge, cached+new
merge, AST+semantic final merge).
Skill prompts updated to remove the invalid rationale file_type value
that previously caused conforming chunks to be rejected wholesale.
Valid set is now {code, document, paper, image}.
Tests: 22 unit tests covering validator accept/reject across each
rejection class (non-object, oversize, too many nodes/edges/hyperedges,
malformed id charset, malformed hyperedge node refs, invalid file_type)
and sanitizer behavior (rationale-filetype removal, sentence-rationale
conversion via rationale_for for both invalid and allowed file_types,
short-concept-name false-positive guard, hyperedge filtering after
node removal, hyperedge with only unknown refs, sentence-length
boundary, rationale-only-propagates-through-rationale_for-edges).
880/880 tests pass.
* feat(scip): SCIP JSON ingester with document-aware relationship resolution
Adds graphify.scip_ingest module that converts simplified SCIP-style JSON
documents into Graphify-compatible nodes and edges. Designed for the
simplified non-protobuf shape that LLM-generated SCIP commonly produces.
Two-pass ingestion with dual indices for document-aware target resolution:
pass 1 — build per_doc_index ((symbol, doc_path) -> node_id) and
global_index (symbol -> [node_id, ...]) across every valid
symbol in every valid document. Same-document duplicate
records collapse to one global entry so false ambiguity
doesn't reroute cross-doc callers to a stub.
pass 2 — emit nodes for indexed symbols, then walk relationships.
Resolution order:
1. same-doc match (per_doc_index)
2. unique cross-doc match (global_index[symbol] len == 1)
3. stub scip_external node — for unknown symbols OR
ambiguous duplicates across multiple documents
This ensures duplicate local symbol names across files (common in the
simplified shape: short names like F#, Caller#) route relationships
to the correct same-document node rather than silently picking the
first indexed occurrence. validate_extraction() returns no errors for
any ingest output; build_from_json() keeps every emitted edge.
Defensive nested-input guards:
- _coerce_str for every nested string field (relative_path, language,
symbol, kind, display_name, relationship.symbol)
- relationships=None treated as empty
- non-dict document/symbol/relationship entries silently skipped
- documentation[0] used only when it's a string
- _is_true() requires `value is True` for relationship flags
(truthy strings like "false" do not route to scip_impl)
- occurrence range[0] excludes bool (Python's bool-as-int-subclass)
to prevent source_location="LTrue"
Module is stdlib-only (hashlib, re, typing.Any). Not wired to the CLI
in this phase — importable as `from graphify.scip_ingest import
ingest_scip_json`.
Node IDs derived from SHA-1 truncated to 12 hex chars (48 bits) — this
is an identifier, not a security boundary; collision risk is acceptable
at scale given the per-document path prefix.
Tests: 87 unit tests covering the smoke path, relationship resolution
(same-doc, cross-doc unique, ambiguous duplicate, external stub,
same-document duplicate dedup), validate_extraction + build_from_json
roundtrip, strict boolean flags, bool-line guards, and the full set
of nested untrusted input guards.
1044/1044 tests pass.
* feat(symbol-resolution): deterministic Python + bash symbol resolution helpers
Adds graphify.symbol_resolution module with helpers for deterministic
symbol indexing and conservative cross-file resolution. Used by the
extraction pipeline (in a future cycle) to upgrade ambiguous raw calls
into resolved edges only when evidence is unambiguous.
Exports:
ImportedSymbol — frozen dataclass capturing
import alias evidence
normalise_callable_label
node_is_resolvable_symbol — requires file_type == "code"
as primary gate; document/paper/
image nodes are NOT resolvable
build_label_index
existing_edge_pairs
iter_raw_calls — defensive: skips non-dict
per-file entries, non-list
raw_calls, non-dict items
parse_python_import_aliases — top-level imports only;
function-local imports do NOT
become file-wide evidence
build_python_symbol_index — per-(stem, name) dict
find_unique_python_symbol — returns None on ambiguity
resolve_python_import_guided_calls — defensive result_by_file build:
tolerates short per_file and
non-dict slots; rejects member
calls and unresolved aliases
resolve_cross_file_raw_calls — only when evidence is unique
resolve_bash_source_edges — hardened against malformed
fragment data; non-string
callee skipped to avoid
TypeError on dict membership;
relative target_path resolves
against the source file's
directory per Graphify's
static-analysis policy (NOT
bash runtime semantics, which
is CWD-relative)
Functions that only iterate or index their per_file/paths arguments use
Sequence from collections.abc for proper covariance. Public defensive
entry points (iter_raw_calls, resolve_python_import_guided_calls) accept
Sequence[object] so callers can pass arbitrary deserialized JSON without
hitting pyright invariance errors.
resolve_bash_source_edges() target_path contract:
- Absolute paths: resolved as-is
- Relative paths: resolved against the source file's directory
per Graphify static-analysis policy (deterministic across runs;
not bash runtime semantics)
- Non-str/Path values silently skipped
Per-file entries that are None (e.g. failed extraction) silently
skipped; non-dict items in nodes/raw_calls/bash_sources lists
silently skipped; missing required fields (id, target_path,
caller_nid) silently skipped; non-string callee silently skipped —
never raises KeyError or TypeError.
Module is stdlib-only (ast, re, dataclasses, pathlib, typing,
collections.abc). Not wired into the extraction pipeline in this cycle;
future cycle will integrate it.
Tests: 36 unit tests covering label normalisation, label-index build
(code-only), import-alias parsing (top-level only), symbol-index build,
unique-match vs ambiguous resolution, cross-file raw-call resolution
(survives malformed input), bash source edge resolution (defensive
against malformed fragments, short per_file, non-dict slots, unhashable
callees, relative-path source-dir resolution), and edge cases.
* feat(security): cap graph.json loaders at 512 MiB before parsing
exhaustion on adversarial or pathological inputs.
- graphify.security: add _MAX_GRAPH_FILE_BYTES + check_graph_file_size_cap
- graphify.serve._load_graph: call cap after existence check
- graphify.__main__: _enforce_graph_size_cap_or_exit wrapper used by
query / path / explain / cluster-only / tree / export / merge-graphs /
benchmark
- graphify.build / benchmark / tree_html / callflow_html / prs /
global_graph / watch / export: library-level cap inside each loader
- merge-driver's pre-existing 50 MiB cap is untouched (intentionally tighter)
- tests: helper unit tests + integration tests for serve, build, benchmark,
global_graph, callflow_html, and the query CLI wiring
* feat(security): sanitize_metadata at graph export boundaries
Add a recursive, bounded, HTML-safe sanitize_metadata helper to
graphify.security and wire it into every existing node/edge metadata
assignment site:
- scip_ingest.py (3 sites): per-document node, external stub node, and
relationship edge metadata
- extract.py (1 site): bash extractor's add_node metadata
- symbol_resolution.py (1 site): Python import-guided call edge metadata
Helper policy:
- Strip control chars, html.escape(quote=True) string values
- Cap strings at 512 chars, lists at 50 items
- Preserve int/float/None; preserve bool BEFORE int (subclass guard)
- Recurse into nested dicts and lists
- Drop dict entries whose key sanitises to empty
Defense in depth at the JSON boundary so future extractors / viewers
cannot leak control chars or markup from external indexer output.
* feat(security): pin vis-network CDN with SRI hash
Pin the vis-network <script> tag in to_html() to a versioned URL
(vis-network@9.1.6) with a sha384 Subresource Integrity hash and
crossorigin="anonymous". Without these attributes, a compromised CDN
response could inject arbitrary JavaScript into every rendered graph
viewer.
Hash verified live against
https://unpkg.com/vis-network@9.1.6/standalone/umd/vis-network.min.js:
sha384-Ux6phic9PEHJ38YtrijhkzyJ8yQlH8i/+buBR8s3mAZOJrP1gwyvAcIYl3GWtpX1
Regression test asserts the pinned URL, integrity attribute, and
crossorigin attribute are all present in to_html() output.
Follow-up: tree_html.py (D3) and callflow_html.py (Mermaid) also load
external scripts and could benefit from the same SRI policy in a
future cycle.
* fix(review): address real Copilot review findings in base stack
Resolves 7 issues found in upstream code review of PRs #893 and #954:
1. extract.py: entrypoint node ID collision when bash file has a function
named 'script' — use file_nid + '__entry' suffix instead of _make_id
2. extract.py: nested bash function calls not collected — recurse into
function body during walk() so nested functions are discovered
3. extract.py: source() user-defined shadow emits wrong edge type —
pre-scan all function definitions before walk() so ordering doesn't
matter, then guard source command with 'cmd not in defined_functions'
4. extract.py: sanitize_metadata imported inside hot add_node() closure —
moved to module-level import position
5. symbol_resolution.py: _bash_make_id() diverged from extract._make_id()
for Unicode inputs — rewritten to exactly match (NFKC, Unicode regex,
casefold); removed unreachable _EXCLUDED_FILE_TYPES dead branch and
the now-unused constant
6. semantic_cleanup.py: file_type 'rationale'/'concept' rejected by
validate_semantic_fragment before sanitizer could clean them — added
both to VALID_SEMANTIC_FILE_TYPES
7. scip_ingest.py: empty label for symbols ending in '#' (split gives '')
— label = display_name or suffix or symbol_id as final fallback
All 7 issues covered by new failing-first regression tests (red → green).
Full pytest suite: 1239 passed, 4 pre-existing env-specific failures.
* fix(review): address PR #956 Copilot findings in watch.py and symbol_resolution.py
- watch.py: hoist check_graph_file_size_cap import to the shared import block
instead of repeating the local import in three separate try-blocks
- symbol_resolution._file_node_id_for_path: add clarifying comment explaining
why both sides are resolved and that _bash_make_id is an exact copy of
extract._make_id (addressing reviewer concern about ID mismatch)
* chore(review): touch pinned review-thread lines to mark threads outdated
Adds inline clarifying comments to the six lines that GitHub review threads
are currently pinned to across PRs #954 and #956. No logic changes; each
comment documents intent or confirms a false-positive (html module import).
* feat(diagnostics): report multigraph edge-collapse risk
Add graphify.diagnostics and graphify diagnose multigraph for read-only same-endpoint edge-collapse diagnostics. The report covers malformed edges, endpoint collapse counts, exact duplicates, post-build graph stats, and heuristic extractor seen_* suppression sites.
Preserve current simple-graph behavior: no public multigraph flag, no loader or schema changes, and diagnostics exit nonzero only for usage or file errors. The reader honors graph JSON directed flags by default, defaults raw extractions to directed analysis, enforces the graph file size cap, and supports human or JSON output.
* feat(multigraph): add runtime compatibility probe
New module graphify.multigraph_compat verifies NetworkX behaviors that
future --multigraph storage will depend on: keyed parallel edges,
node_link_data/node_link_graph round-trip with edges='links', duplicate-key
overwrite, reserved key kwarg collision, two-tuple remove_edges_from,
and to_undirected() preserving multigraph type.
Behavior probe, not version check. Both NX 3.4.2 (Py 3.10 lane) and
NX 3.6.1+ (Py 3.11+ lane) pass. Result cached for the process lifetime.
No call sites added — this PR adds the API surface only. Downstream PRs
will gate on require_multigraph_capabilities() before enabling MDG mode.
Refs: Wave 1 MultiDiGraph implementation order.
* test: filter known third-party analyze warnings
---------
Co-authored-by: vampyre <vampyre@local.net>
Three independent Windows compatibility fixes shipped together because they
all surface during the same first /graphify run on Windows.
graphify/benchmark.py
print_benchmark() unconditionally printed U+2500 (box-drawing) and U+2192
(rightwards arrow), which UnicodeEncodeError'd on stdouts that can't encode
them — most notably the legacy Windows console at cp1252. New _safe()
helper falls back to ASCII when the active stdout encoding can't carry the
glyph; _hr() uses it. Two regression tests cover both paths and prove
print_benchmark survives a cp1252-strict stream.
graphify/extract.py
ProcessPoolExecutor on Windows uses spawn, so worker subprocesses
re-import the calling __main__. When the caller is `python -c "..."` or a
script without an `if __name__ == "__main__":` guard, the workers
recursively spawn themselves and the pool dies. The user-visible failure
was a 290-line traceback ending in BrokenProcessPool, hiding the actual
cause. _extract_parallel now catches BrokenProcessPool, prints a one-line
warning that names the __main__-guard idiom, and returns False so the
public extract() routes to the existing _extract_sequential fallback. Two
tests cover the parallel-returns-False contract and the sequential
fallback wiring.
graphify/skill-windows.md
Every `python -c "..."` block (30 in total) is replaced with a
Write+run+delete pattern using PowerShell's literal here-string @'...'@.
The old form was a quote-escaping minefield: any double-quote inside the
Python source had to be backslash-escaped for the shell, and PowerShell's
parser ate them inconsistently — failing on f-strings like
`f'AST: {len(result["nodes"])} nodes'`. The new form passes Python source
to disk literally, so what the model writes is what Python sees. The AST
step's script template now includes an explicit `if __name__ == "__main__":`
guard so multi-core extraction works even before the runtime fallback above
kicks in. All 31 resulting heredoc blocks parse cleanly under
`ast.parse`.
Co-authored-by: Nauman Hameed <Nauman.Hameed@enghouse.com>
tree-sitter-typescript ships two grammars:
- language_typescript: pure TypeScript, no JSX support
- language_tsx: JSX-aware variant for .tsx files
Currently both .ts and .tsx are parsed with language_typescript, which
treats JSX syntax as parse errors. Every function declaration, arrow
function, and call_expression nested inside a JSX tree is silently
dropped from the extracted graph.
Repro on a representative React+TypeScript codebase (a 13-file Tauri app):
parsing each .tsx with language_typescript produces ~276 ERROR nodes per
file. Only declarations that happen to live before the first JSX block
survive.
Fix: add _TSX_CONFIG that mirrors _TS_CONFIG but selects language_tsx,
and route .tsx files to it in extract_js().
Effect on the same repo (graphify update --force):
Nodes: 303 → 618 (+104%)
Edges: 482 → 779 (+62%)
Communities: 28 → 45 (+61%)
Parse errors 276 → 0 per .tsx file
Tests added:
- tsx fixture with helpers + JSX-returning component
- helpers and component are captured
- JSX expression calls ({fmtDate(now)}) resolve to call edges
- wiring check: .tsx uses language_tsx, .ts uses language_typescript
Note: this fixes the parsing layer. Calls inside deeply nested arrow
function callbacks (e.g. items.map(x => <T>{f(x)}</T>)) are still
missed by the call extraction logic — separate enhancement.
Co-authored-by: Serkan Gezici <serkan@quadroaipilot.com>
