perf_counter counts wall-clock, which includes time the process is preempted
off-CPU during a busy full-suite run, inflating the larger measurement and
flaking the ratio. process_time counts only CPU work, isolating the algorithmic
scaling. Follow-up to #3402.
extractors/dart.py hardcoded source_location: None at every node and
edge construction site, so a Dart graph could cite zero lines no
matter how many declarations it had (0% across three real repos,
~25k unciteable nodes in one of them).
Threads a line number through add_node/add_edge from each match's own
offset, anchored on the capture group rather than the whole match: a
pattern like the class declaration's leading ^\s* otherwise reports
the line too early when blank lines precede the declaration.
The comment stripping pass also needed a fix first: it deleted a
comment token outright, so a multi-line comment shifted every later
offset (and therefore every later line number) up by the comment's
own height. Blanking it to the same number of newlines instead keeps
src_clean's line count in step with the original file.
Sourceless reference stub nodes (an inherited class, a mixin, an
import target, and similar names the file only references rather
than defines) keep source_location: None, same as before, since there
is no single line in THIS file for a symbol that lives elsewhere.
Every node this file actually defines, and every edge attributed to a
real declaration site, now carries its real line.
Fixes#3365.
(cherry picked from commit 0dbcefc5ba)
The alias was written as a quoted string, which makes it a `str` *value*
rather than a type. Type checkers reject it: mypy reports "Variable
_TsRangeIndex is not valid as a type" at both annotation sites, plus two
cascading errors where the tuple loses its type and the unpacking/indexing
in _ts_ranges_containing and _normalize_ts_import_types become untyped.
`typing.get_type_hints` happened to resolve it at runtime because it
evaluates the annotation string and then the alias string again, so nothing
failed in practice -- but that is an accident of double evaluation, not a
working alias.
Unquote the right-hand side. The module already has
`from __future__ import annotations`, so the annotations themselves need no
quoting either. No behavior change.
(cherry picked from commit d902820ccf)
`_normalize_ts_import_types` filtered each `import(...)` regex match against
every tree-sitter `type_arguments` byte range with a nested linear scan, in
both the pre-mask and post-mask passes:
matches = [m for m in matches
if not any(start <= m.start() < end
for start, end in original_type_ranges)]
That is O(matches x ranges), and `m.start()` is re-evaluated inside `any()`
for every range — which is why profiles blamed `re.Match.start`/`_sre` and
`sample` appeared to show a pathological regex. The regexes are all linear;
the quadratic caller was the real cost.
Both counts grow with file size in TypeScript that mixes generic calls
(`fn<A, B>(x)` — one `type_arguments` range each) with `import(...)` types,
which is the shape of generated and barrel modules in a large monorepo. On
the filter alone: a 0.8MB/10k-import file took ~21s, 2.1MB/25k ~2.6min, and
4.2MB/50k ~11.8min. With several such files `extract` never finished, and a
worker sat at 100% CPU holding no open files because the work is entirely
in-memory after the read.
Replace both scans with a binary search over start-sorted ranges plus a
prefix-maximum of `end` values, so a lookup touches nesting depth rather
than every range. Type-argument ranges nest but never partially overlap, so
the prefix maximum is a correct stop condition; a *closed* inner sibling
must not terminate the leftward walk, which a naive running-max break gets
wrong. Verified against the old predicate with 500k differential cases,
including non-nested random ranges: zero mismatches.
Normalization output is unchanged — byte-length and line-count preserving,
same masking decisions, and the runtime-vs-type disambiguation from 6baa5e0
still holds.
Also restore the emptiness guard in the second pass: the range list is now
an index tuple, which is always truthy, so it checks the spans it holds.
The new test asserts on scaling (2x input must not ~4x the time) rather than
wall-clock, so it is not machine-dependent. It fails at ~3.4x before this
change and passes after.
(cherry picked from commit b662361d8b)
The assertion recorded the order batches reached their LLM call inside worker
threads, which is nondeterministic; the product merges by cid on the main
thread and concurrency equivalence is covered by a sibling test. Takes only the
flaky-test fix from #3374 (its SECURITY.md/uv.lock changes are handled
separately).
`import X from '#services/x'` — Node's subpath-imports field, and the way
every AdonisJS 6 app reaches its own modules — was never resolved: the JS
resolver only knew tsconfig `paths` and workspace packages. In a real
Adonis app 438/680 alias imports produced no edge and `affected
"ZitadelService"` listed 2 nodes from a stray root script instead of the
7 controllers and the auth middleware that depend on it.
Resolve `#…` specifiers through the nearest package.json `imports` map,
after tsconfig `paths` (explicit aliases keep precedence, #1269) and
before workspace packages. Node semantics: exact key or a single `*`
wildcard, longest literal prefix wins, condition objects go through
_resolve_export_target, and only `./` targets are followed so an external
mapping keeps the ref-namespaced external id (#1638). The map applies to
every file in the package, so a nested tsconfig without the aliases
(e.g. `inertia/tsconfig.json`) no longer shadows it. Cached per start
dir and cleared per extract() run like the tsconfig caches (#2917).
Measured on a 966-file AdonisJS + Inertia repo: alias imports resolved
go from 242/680 to 680/680 in app/ and 17/17 in inertia/.
Claude-Session: https://claude.ai/code/session_011z6vqqu1uujGKjjUE9Dgbg
(cherry picked from commit 691cccc00f)
The no-SIGALRM fallback killing workers before os._exit does not
cover the primary path: on POSIX, where SIGALRM is available, a
TimeoutError raised while the main thread is waiting inside the
ProcessPoolExecutor with-block propagates straight through that
block's own __exit__, which calls shutdown(wait=True) and blocks
until every worker exits. Forever, for a worker stuck in a C-level
call the alarm firing does nothing to stop, the exact shape #3341
was. The except TimeoutError handler that would call sys.exit is
never reached, so the configured timeout bounds nothing in that case.
Reproduced directly: a worker in an unconditional loop, a SIGALRM
after 2s, and the process was still running 15s later.
Kills any live multiprocessing.active_children() inside the signal
handler itself, before it raises, so by the time the exception
reaches shutdown(wait=True) there is nothing left to wait for. Moved
the handler from an inline lambda (which cannot contain a statement,
only expressions) to a named function, matching the no-SIGALRM
fallback's own style.
Fixes#3396.
(cherry picked from commit ed948388db)
os._exit(1) terminates only the rebuild process itself, not any
ProcessPoolExecutor worker it spawned for a large corpus. It skips
every cleanup path, including the pool's own context-manager
shutdown, so a worker still running at the moment the watchdog fires
is orphaned, reparented to PID 1 on POSIX, and keeps going
unsupervised for as long as whatever it was doing takes. A worker
stuck in catastrophic regex backtracking, the exact shape #3341
fixed, was observed surviving 2.5 days that way on macOS.
extract.py already documents this risk for its own 1 worker special
case, but that only covers max_workers == 1, not a real multi worker
pool.
The watchdog fires on a separate timer thread with no reference to
the pool object, but multiprocessing.active_children() enumerates
every live worker process regardless of which thread asks, so the
fallback kills them before exiting itself. Kill, not terminate: a
worker stuck in a C level call like regex backtracking never gets a
chance to act on a signal a Python level handler would need to catch.
Fixes#3396.
(cherry picked from commit 319869e6c1)
_LANGUAGE_BUILTIN_GLOBALS is one union across every language, right
for a BARE call (String(x) really would become a god node) but wrong
for a MEMBER call. open is a Python builtin and Set a JavaScript one,
so session.open() in Swift or _server.Set() in C# named after another
language's builtin was silently discarded outright: no same file
edge, but also no raw_calls entry, so cross file resolution never got
a chance to try it.
A member call carries a receiver, so it is not the ambiguous case the
union guards against. Lets a builtin named member call reach the
existing tgt_nid = None deferral every receiver typed resolver
already relies on for other cases (a capitalized Python receiver, any
Java member call, and so on), so it can only ever reach an edge
through a guarded, receiver typed resolver, never the unguarded
bare name path a real god node would need.
Verified against real cross file setups for Python (class qualified
call) and Swift (field typed receiver, reproducing the reporter's own
AppleReadyStreamFinder.find -> RemuxPlayback.open example), plus that
a same file member call still cannot bind to an unrelated bare
function sharing the builtin name.
Scoped to the shared engine only. go.py and rust.py carry the same
outer filter, and the reported issue asks for the same change there
too, but tracing where their raw_calls entries actually go shows the
shared cross file resolver skips every is_member_call entry
unconditionally (extract.py, the "obj.log() has no import evidence"
guard) regardless of builtin status, and neither language has a
dedicated receiver typed resolver the way Python, Java, C#, Swift,
TypeScript, C++, and Objective-C do. Making the same change there
would be safe but currently inert, so left out rather than shipping a
change with no verifiable effect; worth revisiting once either
language gets its own receiver typed cross file resolver.
Fixes#3381.
(cherry picked from commit c0f9ff57e0)
`_load_graph` forces `directed: True` so renderers can recover stored arc
order (#2309), and `_bfs`/`_dfs` expand through `G.neighbors()`, which on a
DiGraph yields successors only. Seeded on a leaf that is only ever called,
imported and contained, the MCP `query_graph` therefore answered with the
seed alone, while the CLI `query` — which loads the same graph.json
undirected on purpose (#2080) — returned the callers, the test and the
neighbouring modules. Same shape as #1174: one read surface missed a fix
the others got.
Add `_traversal_view`: an undirected copy for the traversal with `_src`/
`_tgt` stashed per edge, so `_subgraph_to_text` keeps rendering
caller->callee regardless of the side the traversal reached the edge from.
An undirected input is returned as-is, so the CLI path is unchanged;
`_load_graph` is untouched because `get_neighbors` and `shortest_path`
rely on direction explicitly.
Regression tests seed the same `calls` edge from both endpoints through
`_load_graph`, in both traversal modes, with and without a context filter;
the callee-seeded ones fail on the prior code with `1 nodes found`.
(cherry picked from commit 341aec5286)
Follow-up to #3367: guards that signature-only trait methods are captured and
that a trait-declared method stays a distinct node from its impl definition.
The generated post-commit / post-checkout block is appended to whatever
hook a repo already has, and other tools chain their logic after it. Every
skip condition in the block was a bare `exit 0`, which in a flat sh script
ends the ENTIRE hook process — so anything after graphify's end marker
was silently dropped on every root commit (HEAD~1 does not exist), every
rebase/merge/cherry-pick, every linked worktree, every GRAPHIFY_SKIP_HOOK=1
and whenever no graphify-capable python could be found.
Both bodies now run inside a subshell `( ... )`, so an `exit` ends only
graphify's section: no rewriting of the seventeen skip sites, no
`return`-outside-a-function hazard for the _PYTHON_DETECT fragment that
tests run standalone, the detached rebuild launch unaffected, and the
hook's own exit status 0 as before. The markers stay outside the subshell
so reinstall/uninstall/status keep finding the block.
(cherry picked from commit 6f09919f10)
A trait got a node; the methods it declares did not. Across three real Rust
repositories 504 of 654 trait methods (77%) had no node at all, so a trait — the
API contract — appeared in the graph as a bare name with no operations attached,
and "what must an implementor provide?" was unanswerable.
Two gaps, both here:
- `function_signature_item`, the node type for a bodyless `fn greet(&self);`,
had no branch anywhere in the package, so a signature-only method had no
extraction path.
- the struct/enum/trait branch returns before descending, so nothing walked a
trait's body — a default-bodied method inside a trait was never reached
either.
The methods that did appear came from impl blocks that happened to define the
same name, and were anchored at the impl rather than the declaration.
This adds a `function_signature_item` branch mirroring `function_item` (minus the
body walk, since there is none) and descends into the trait body the way
`impl_item` already descends into its own, attributing each method to the trait
node with a `method` edge.
Before / after on the reproducer in #3366:
src/lonely.rs (a trait with no implementor in the corpus)
before: Lonely
after: Lonely, .only_signature() L3, .with_default() L4
+ method edges from Lonely
On a 937-file Rust service: 112 trait methods declared, 70 missing -> 0 missing;
9426 -> 9574 nodes, 28638 -> 29177 edges.
Full suite is unchanged: 17 failed / 5296 passed / 93 skipped both with and
without this commit (the 17 are pre-existing on v8, all in test_skillgen.py).
Note for anyone verifying: the AST cache is keyed on file content, not extractor
version, so `graphify-out/cache` must be removed to observe the change — `--force`
alone re-runs extraction but still serves cached per-file results.
(cherry picked from commit e9fbe22503)
`claude -p --output-format json` shares stdout with the CLI's own
subsystems, so the envelope is not always the first thing on it. An
attached MCP server that advertises no tools makes every invocation emit
Client.listTools() called but server does not advertise tools
capability - returning empty list
ahead of the envelope, and `json.loads(stdout)` fails on the whole buffer.
The failure is raised after the model has already answered, so the chunk
is discarded with its tokens spent. On a real corpus this burned 149,582
input and 52,162 output tokens on chunk 1 of 3 and returned nothing, and
the error names the JSON rather than the preamble that caused it, so the
log points at the wrong thing. Anyone with an MCP server configured hits
it on every chunk.
`_envelope_after_preamble` scans for the first `[`/`{` that begins a
valid JSON document. `raw_decode` ignores trailing bytes, so a diagnostic
on either side is tolerated. The fast path is unchanged: clean stdout
still parses on the first `json.loads`, and stdout with no JSON at all
still returns None so the caller raises -- test_raises_on_garbage_envelope
holds.
(cherry picked from commit ede3a9e0a8)
Absence of a `node_kind` stamp is not proof that a node is an entity: an
extractor can mint a node standing for part of its file without stamping one
(detect.py's per-sheet document nodes do exactly that), and the predicate read
those as entities while its name and docstring claimed proof.
Renames it `_reads_as_file_entity` and splits the docstring into what is PROVEN
(not the file's own node — an `_id_prefixes` reconstruction that holds for every
stored-path spelling) and what is ASSUMED (not a section, resting on the
producer stamping `node_kind`). Adds the producer contract next to
`_FILE_STRUCTURE_NODE_KINDS` and a test pinning the stamped/unstamped split, so
the limit is enforced rather than described.
No behaviour change: the same nodes merge as before.
Co-Authored-By: Claude <noreply@anthropic.com>
(cherry picked from commit d123254ebf)
Per-file extraction over a note vault mints one node per mention of the same
person or project, and they never merge. Normalization is not the problem:
`_norm` already casefolds and collapses `[\W_]+`, so `@cyrilXBT`, `@cyrilxbt`
and `cyrilXBT` land in the same Pass 1 bucket. The bucket is then dropped,
because the cross-file union added in #2182 is gated to
`file_type == "concept"` — and an entity the extractor pulls out of a `.md`
note inherits `document` from the file's extension, not from anything about
the entity.
#1284's reasoning for that gate is right about files: two `README.md` in
different folders are two documents. It does not follow for a node that merely
lives in a document. This widens the Pass 1 cross-file residue to
`document`/`rationale` nodes that are provably not part of their file's own
structure, via a new `_provably_not_file_structure`:
* not the file's own node — reusing `_id_prefixes`, which already enumerates
the ID a node standing for `source_file` would carry in every spelling a
stored path may take;
* not one of the file's sections — `node_kind: "heading"`, and `"page"` for
the file node the markdown extractor labels outright.
A node that cannot be checked (no ID, no provenance) answers False and stays
blocked, so the predicate can only narrow what the gate treats as
file-anchored, never widen it on a guess.
Deliberately unchanged: the entropy and provenance guards, `code` (#1205),
`image`/`paper` shared basenames, and Pass 2's fuzzy
`_crossfile_fileanchored_blocked` — so #1284's near-identical boilerplate and
#3094's repeated `## Decisions` sections stay per-file, both now covered by
tests.
Co-Authored-By: Claude <noreply@anthropic.com>
(cherry picked from commit bc45e1cf8b)
ids.py:make_id strips leading/trailing underscores from every name
part before normalizing, so _get_connection/get_connection (and more
broadly any name pair differing only by underscore wrapping --
x/_x/__x/__x__ dunder variants all collapse the same way) mint the
SAME node id. add_node's first-wins dedup then silently drops
whichever declaration is walked second: a public method could be
entirely absent from the graph while its private-by-convention
sibling occupied the public name, with no warning and exit 0.
Adds a Python-scoped pre-scan (_python_pre_scan_underscore_collisions)
that groups module-level functions and direct class methods by their
would-be node id, plus a resolver (_python_underscore_salted_nid) that
salts every group member except a unique public (no leading
underscore) one -- mirroring the established convention the Go
extractor already uses for its own case-only collision (#2779/#2780):
when no unique public member exists, every member is salted, so the
outcome never depends on declaration order. The public member's id
stays exactly what it would be with no colliding sibling at all, so
adding/removing a private-by-convention sibling in an incremental
rebuild re-points nothing.
Both hooks are gated on config.ts_module == "tree_sitter_python" and
live inside the shared _extract_generic engine (Python has no
dedicated extractor file, unlike Go/Rust/SQL), following the existing
precedent for per-language pre-scans already there (csharp_interface_names,
swift_protocol_names). Call-site resolution needed no changes: label_to_nid
is rebuilt from the final nodes list after the walk completes, so it
naturally picks up whichever id (salted or not) a method actually got.
9 new tests in tests/test_python_underscore_resolution.py: the exact
repro, id stability across incremental add/remove of a private
sibling, the call edge correctly binding to the salted target, the
module-level (non-method) case, the no-unique-public-member case
(both members salted), a dunder+plain collision, an unaffected
no-collision file, and scope isolation (a collision in one class must
not touch an unrelated same-named-but-non-colliding member in another
class).
Note: tests/ on the current v8 HEAD has 7 pre-existing failures
unrelated to this change (test_cli_export.py, test_cross_repo_shared_types.py,
test_extract_code_only_cli.py, test_merge_graphs_cli.py) -- confirmed via
`git stash` that they fail identically on pristine v8 without this diff.
Full suite otherwise: 5116 passed. ruff and skillgen --check clean.
Fixes#3302.
(cherry picked from commit cb8ccebab9)
The #3345 ghost-merge fix cited #2963 (an unrelated Rust Self-keyword bug)
in its comments and test docstrings; the issue it actually resolves is #3344.
The #1145-extended ghost-merge (build_from_json) keys exclusively on
(source_file, label). When a semantic pass reads a document that only
*mentions* a real file by name (a saved graphify-out/memory/*.md query
answer, a runbook like "see App.tsx"/"backup.sh does X"), the new node
it mints gets source_file = the document being read, not the file
actually named in the prose. That wrong source_file means the key can
never match the real AST node's, so the ghost survives every rebuild.
Add a conservative label-only fallback pass reusing the existing
_is_file_node_label() predicate (already used for file-node label
disambiguation) against every AST file-self node collected in Pass 1.
Only remaps when the match is unique across the graph — 0 or 2+
candidates (e.g. two different index.ts files) are left alone, same
conservatism the existing (source_file, label) key already applies via
_loc_collisions.
Verified against a real ~4600-node project graph: 10 long-standing
ghost nodes removed across two unrelated documents (a memory query
file and an ops runbook), confirming this is a general failure mode of
the "document mentions a file by name" pattern, not a narrow edge
case. Both hyperedges that had cited the phantoms now correctly cite
only real, AST-backed nodes.
Fixes#3344
(cherry picked from commit a67e7060b0)
_get_docstring's loop checked only the FIRST child of a module/class/
function body for a docstring expression_statement, then unconditionally
broke -- so a leading comment (the #! shebang essentially every
executable Python script starts with, a coding declaration, a license
header) sat in that first-child slot instead of the docstring, and the
real docstring one position later was never seen. No warning, no
partial-extraction report, exit 0 -- module rationale for an entire
class of file (any shebang-led or comment-led script) was silently
absent from the graph.
Comments are not statements: a docstring must be the first STATEMENT
in the body, so the fix skips leading `comment` children before
checking whether the first real statement is a docstring. The same
shared helper also serves class/function bodies, so a comment as the
first line of a function/class body is fixed too, not just the
module-level case.
Fixes#3312.
(cherry picked from commit 87d4cb49b7)
The ImportError from _load_toml_module is caught by extract_package_manifest's
except-block and surfaced as a per-manifest error dict, so the behavior is a
visible per-file error, not an aborting failure. Correct the docstring to match
(follow-up to #3306).
Declare tomli with an environment marker and raise ImportError instead of returning None, so missing parsers no longer look like empty manifests.
Fixes#3283
(cherry picked from commit c06ba2731c)
Route to_canvas and _owned_write through paths.write_*_atomic so concurrent readers (e.g. git hashing) never see a truncated file.
Fixes#3282
(cherry picked from commit 39564e4eef)
Match the SKILL.md / references/ crash-safety pattern so an interrupted install cannot leave a truncated stamp, and replace managed symlinks instead of writing through them.
Fixes#3286
(cherry picked from commit 030c6372d1)
The extraction spec stores the WHY of a concept as a `rationale` attribute
on the node — deliberately not as a node of its own. `_score_query` matched
a question against `norm_label`, `label_tokens`, `source_file` and the node
id only, and `_node_search_text` indexed the same fields, so for a "why does
X …" question the node holding the answer was neither a trigram candidate
nor a seed unless the asker already knew its label (#2293: 0/5 answers
reached on a 5.5k-node corpus although every answer node existed and stated
the answer in its rationale).
Add a rationale tier, following the fix the issue proposes:
- `_RATIONALE_MATCH_BONUS = 0.75`: below the label substring tier (1.0),
above the source-path tier (0.5). A term found in the rationale adds to
the score and to the per-term singleton used for seed seating, but — like
the source tier — never to term coverage, so a long rationale adds recall
without winning back an exact-label tier it did not earn.
- `_node_rationale_text`: one normalizer (diacritics folded, lower-cased),
tolerant of the list an extractor may emit; "" when absent.
- `_node_search_text` appends the same text as a trailing field, only when
present, so the trigram prefilter stays a complete candidate generator and
every existing field position (#2467) is unchanged.
Nodes without a rationale score exactly as before.
Design credit: @nuboxworld-byte (issue #2293 and #2294). #2294 implements
the same idea but its diff carries the whole repository (+252k lines, 766
files) from a base mismatch and cannot be reviewed or merged as-is; this is
a fresh minimal implementation against v8.
Tests: rationale-only match ranks; tier order label-substring > rationale >
source; no coverage credit; list-valued attribute; search text carries the
field (and is unchanged without it); end to end a "why" question phrased
from the rationale seats the node as a seed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 95ff41101a)
save_query_result named files query_<second>_<slug50>.md and wrote them with
write_text, so two saves in the same second whose questions share the first
50 characters resolved to one path and the later one silently replaced the
earlier. Both calls returned normally. This is the common case when several
agents sweep one subsystem in parallel.
Add a short uuid to the name. The query_ prefix and .md suffix are unchanged
so reflect.load_memory_docs and existing tests keep working.
Fixes#3301
(cherry picked from commit 0bdf89f64b)
When a member call's receiver type resolves to zero in-corpus declarations, park
{callee, receiver_type, lang, line} (names only, #3150-safe) on the caller; a merge-time pass
emits one calls/cross_repo/INFERRED edge only when the receiver type resolves to exactly one
declaration in another repo owning exactly one matching member (language-suffix matched,
non-sourceless). Strong-signal, single-definition guarded, edges-only — composes with the
#3007 cross-repo type link and the #3068 community offset. Wired for Java/C++/C#/Swift.
god_nodes had no exclusion parameter, so --exclude-hubs only reached cluster() and god-node
ranking ignored it. Apply the same percentile threshold cluster() uses (byte-identical
formula), thread it through the build/analyze/MCP paths; the default (no exclusion) output is
unchanged and the node dict shape is preserved for wiki/report consumers.
A Python type reference to a name imported from another module dangled or mis-resolved.
Run Python cross-file import resolution before the generic stub rewire and repoint sourceless
AST reference/inherits/implements edges onto the exact imported definition using the
file-scoped (module, symbol) -> node_id binding (not a bare-name match), so two same-named
types in different modules bind to their own; orphaned stubs are pruned only when no edge
still touches them.
Companion to #3243: extend definition_file portability to the paths build_from_json does not
cover — the watch/incremental relativize + cache-root rebase and the direct extract() return
normalization — via a shared _PORTABLE_PATH_KEYS list so source_file and definition_file stay
in lockstep. Together with #3243 this closes all four persistence paths (graph.json, AST
cache, watch, direct extract).
#2990 stamped definition_file with the still-absolute implementation path, so graph.json
shipped the build host's absolute path beside a repo-relative source_file (and the AST cache
replayed it on every warm rebuild). Normalize definition_file through the same
_norm_source_file path as source_file in build_from_json and the cache round-trip, so it is
relative/portable and byte-stable across machines.
