`claude -p --output-format json` shares stdout with the CLI's own
subsystems, so the envelope is not always the first thing on it. An
attached MCP server that advertises no tools makes every invocation emit
Client.listTools() called but server does not advertise tools
capability - returning empty list
ahead of the envelope, and `json.loads(stdout)` fails on the whole buffer.
The failure is raised after the model has already answered, so the chunk
is discarded with its tokens spent. On a real corpus this burned 149,582
input and 52,162 output tokens on chunk 1 of 3 and returned nothing, and
the error names the JSON rather than the preamble that caused it, so the
log points at the wrong thing. Anyone with an MCP server configured hits
it on every chunk.
`_envelope_after_preamble` scans for the first `[`/`{` that begins a
valid JSON document. `raw_decode` ignores trailing bytes, so a diagnostic
on either side is tolerated. The fast path is unchanged: clean stdout
still parses on the first `json.loads`, and stdout with no JSON at all
still returns None so the caller raises -- test_raises_on_garbage_envelope
holds.
(cherry picked from commit ede3a9e0a8)
Absence of a `node_kind` stamp is not proof that a node is an entity: an
extractor can mint a node standing for part of its file without stamping one
(detect.py's per-sheet document nodes do exactly that), and the predicate read
those as entities while its name and docstring claimed proof.
Renames it `_reads_as_file_entity` and splits the docstring into what is PROVEN
(not the file's own node — an `_id_prefixes` reconstruction that holds for every
stored-path spelling) and what is ASSUMED (not a section, resting on the
producer stamping `node_kind`). Adds the producer contract next to
`_FILE_STRUCTURE_NODE_KINDS` and a test pinning the stamped/unstamped split, so
the limit is enforced rather than described.
No behaviour change: the same nodes merge as before.
Co-Authored-By: Claude <noreply@anthropic.com>
(cherry picked from commit d123254ebf)
Per-file extraction over a note vault mints one node per mention of the same
person or project, and they never merge. Normalization is not the problem:
`_norm` already casefolds and collapses `[\W_]+`, so `@cyrilXBT`, `@cyrilxbt`
and `cyrilXBT` land in the same Pass 1 bucket. The bucket is then dropped,
because the cross-file union added in #2182 is gated to
`file_type == "concept"` — and an entity the extractor pulls out of a `.md`
note inherits `document` from the file's extension, not from anything about
the entity.
#1284's reasoning for that gate is right about files: two `README.md` in
different folders are two documents. It does not follow for a node that merely
lives in a document. This widens the Pass 1 cross-file residue to
`document`/`rationale` nodes that are provably not part of their file's own
structure, via a new `_provably_not_file_structure`:
* not the file's own node — reusing `_id_prefixes`, which already enumerates
the ID a node standing for `source_file` would carry in every spelling a
stored path may take;
* not one of the file's sections — `node_kind: "heading"`, and `"page"` for
the file node the markdown extractor labels outright.
A node that cannot be checked (no ID, no provenance) answers False and stays
blocked, so the predicate can only narrow what the gate treats as
file-anchored, never widen it on a guess.
Deliberately unchanged: the entropy and provenance guards, `code` (#1205),
`image`/`paper` shared basenames, and Pass 2's fuzzy
`_crossfile_fileanchored_blocked` — so #1284's near-identical boilerplate and
#3094's repeated `## Decisions` sections stay per-file, both now covered by
tests.
Co-Authored-By: Claude <noreply@anthropic.com>
(cherry picked from commit bc45e1cf8b)
ids.py:make_id strips leading/trailing underscores from every name
part before normalizing, so _get_connection/get_connection (and more
broadly any name pair differing only by underscore wrapping --
x/_x/__x/__x__ dunder variants all collapse the same way) mint the
SAME node id. add_node's first-wins dedup then silently drops
whichever declaration is walked second: a public method could be
entirely absent from the graph while its private-by-convention
sibling occupied the public name, with no warning and exit 0.
Adds a Python-scoped pre-scan (_python_pre_scan_underscore_collisions)
that groups module-level functions and direct class methods by their
would-be node id, plus a resolver (_python_underscore_salted_nid) that
salts every group member except a unique public (no leading
underscore) one -- mirroring the established convention the Go
extractor already uses for its own case-only collision (#2779/#2780):
when no unique public member exists, every member is salted, so the
outcome never depends on declaration order. The public member's id
stays exactly what it would be with no colliding sibling at all, so
adding/removing a private-by-convention sibling in an incremental
rebuild re-points nothing.
Both hooks are gated on config.ts_module == "tree_sitter_python" and
live inside the shared _extract_generic engine (Python has no
dedicated extractor file, unlike Go/Rust/SQL), following the existing
precedent for per-language pre-scans already there (csharp_interface_names,
swift_protocol_names). Call-site resolution needed no changes: label_to_nid
is rebuilt from the final nodes list after the walk completes, so it
naturally picks up whichever id (salted or not) a method actually got.
9 new tests in tests/test_python_underscore_resolution.py: the exact
repro, id stability across incremental add/remove of a private
sibling, the call edge correctly binding to the salted target, the
module-level (non-method) case, the no-unique-public-member case
(both members salted), a dunder+plain collision, an unaffected
no-collision file, and scope isolation (a collision in one class must
not touch an unrelated same-named-but-non-colliding member in another
class).
Note: tests/ on the current v8 HEAD has 7 pre-existing failures
unrelated to this change (test_cli_export.py, test_cross_repo_shared_types.py,
test_extract_code_only_cli.py, test_merge_graphs_cli.py) -- confirmed via
`git stash` that they fail identically on pristine v8 without this diff.
Full suite otherwise: 5116 passed. ruff and skillgen --check clean.
Fixes#3302.
(cherry picked from commit cb8ccebab9)
The #3345 ghost-merge fix cited #2963 (an unrelated Rust Self-keyword bug)
in its comments and test docstrings; the issue it actually resolves is #3344.
The #1145-extended ghost-merge (build_from_json) keys exclusively on
(source_file, label). When a semantic pass reads a document that only
*mentions* a real file by name (a saved graphify-out/memory/*.md query
answer, a runbook like "see App.tsx"/"backup.sh does X"), the new node
it mints gets source_file = the document being read, not the file
actually named in the prose. That wrong source_file means the key can
never match the real AST node's, so the ghost survives every rebuild.
Add a conservative label-only fallback pass reusing the existing
_is_file_node_label() predicate (already used for file-node label
disambiguation) against every AST file-self node collected in Pass 1.
Only remaps when the match is unique across the graph — 0 or 2+
candidates (e.g. two different index.ts files) are left alone, same
conservatism the existing (source_file, label) key already applies via
_loc_collisions.
Verified against a real ~4600-node project graph: 10 long-standing
ghost nodes removed across two unrelated documents (a memory query
file and an ops runbook), confirming this is a general failure mode of
the "document mentions a file by name" pattern, not a narrow edge
case. Both hyperedges that had cited the phantoms now correctly cite
only real, AST-backed nodes.
Fixes#3344
(cherry picked from commit a67e7060b0)
_get_docstring's loop checked only the FIRST child of a module/class/
function body for a docstring expression_statement, then unconditionally
broke -- so a leading comment (the #! shebang essentially every
executable Python script starts with, a coding declaration, a license
header) sat in that first-child slot instead of the docstring, and the
real docstring one position later was never seen. No warning, no
partial-extraction report, exit 0 -- module rationale for an entire
class of file (any shebang-led or comment-led script) was silently
absent from the graph.
Comments are not statements: a docstring must be the first STATEMENT
in the body, so the fix skips leading `comment` children before
checking whether the first real statement is a docstring. The same
shared helper also serves class/function bodies, so a comment as the
first line of a function/class body is fixed too, not just the
module-level case.
Fixes#3312.
(cherry picked from commit 87d4cb49b7)
Declare tomli with an environment marker and raise ImportError instead of returning None, so missing parsers no longer look like empty manifests.
Fixes#3283
(cherry picked from commit c06ba2731c)
Route to_canvas and _owned_write through paths.write_*_atomic so concurrent readers (e.g. git hashing) never see a truncated file.
Fixes#3282
(cherry picked from commit 39564e4eef)
Match the SKILL.md / references/ crash-safety pattern so an interrupted install cannot leave a truncated stamp, and replace managed symlinks instead of writing through them.
Fixes#3286
(cherry picked from commit 030c6372d1)
The extraction spec stores the WHY of a concept as a `rationale` attribute
on the node — deliberately not as a node of its own. `_score_query` matched
a question against `norm_label`, `label_tokens`, `source_file` and the node
id only, and `_node_search_text` indexed the same fields, so for a "why does
X …" question the node holding the answer was neither a trigram candidate
nor a seed unless the asker already knew its label (#2293: 0/5 answers
reached on a 5.5k-node corpus although every answer node existed and stated
the answer in its rationale).
Add a rationale tier, following the fix the issue proposes:
- `_RATIONALE_MATCH_BONUS = 0.75`: below the label substring tier (1.0),
above the source-path tier (0.5). A term found in the rationale adds to
the score and to the per-term singleton used for seed seating, but — like
the source tier — never to term coverage, so a long rationale adds recall
without winning back an exact-label tier it did not earn.
- `_node_rationale_text`: one normalizer (diacritics folded, lower-cased),
tolerant of the list an extractor may emit; "" when absent.
- `_node_search_text` appends the same text as a trailing field, only when
present, so the trigram prefilter stays a complete candidate generator and
every existing field position (#2467) is unchanged.
Nodes without a rationale score exactly as before.
Design credit: @nuboxworld-byte (issue #2293 and #2294). #2294 implements
the same idea but its diff carries the whole repository (+252k lines, 766
files) from a base mismatch and cannot be reviewed or merged as-is; this is
a fresh minimal implementation against v8.
Tests: rationale-only match ranks; tier order label-substring > rationale >
source; no coverage credit; list-valued attribute; search text carries the
field (and is unchanged without it); end to end a "why" question phrased
from the rationale seats the node as a seed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 95ff41101a)
save_query_result named files query_<second>_<slug50>.md and wrote them with
write_text, so two saves in the same second whose questions share the first
50 characters resolved to one path and the later one silently replaced the
earlier. Both calls returned normally. This is the common case when several
agents sweep one subsystem in parallel.
Add a short uuid to the name. The query_ prefix and .md suffix are unchanged
so reflect.load_memory_docs and existing tests keep working.
Fixes#3301
(cherry picked from commit 0bdf89f64b)
When a member call's receiver type resolves to zero in-corpus declarations, park
{callee, receiver_type, lang, line} (names only, #3150-safe) on the caller; a merge-time pass
emits one calls/cross_repo/INFERRED edge only when the receiver type resolves to exactly one
declaration in another repo owning exactly one matching member (language-suffix matched,
non-sourceless). Strong-signal, single-definition guarded, edges-only — composes with the
#3007 cross-repo type link and the #3068 community offset. Wired for Java/C++/C#/Swift.
god_nodes had no exclusion parameter, so --exclude-hubs only reached cluster() and god-node
ranking ignored it. Apply the same percentile threshold cluster() uses (byte-identical
formula), thread it through the build/analyze/MCP paths; the default (no exclusion) output is
unchanged and the node dict shape is preserved for wiki/report consumers.
A Python type reference to a name imported from another module dangled or mis-resolved.
Run Python cross-file import resolution before the generic stub rewire and repoint sourceless
AST reference/inherits/implements edges onto the exact imported definition using the
file-scoped (module, symbol) -> node_id binding (not a bare-name match), so two same-named
types in different modules bind to their own; orphaned stubs are pruned only when no edge
still touches them.
Companion to #3243: extend definition_file portability to the paths build_from_json does not
cover — the watch/incremental relativize + cache-root rebase and the direct extract() return
normalization — via a shared _PORTABLE_PATH_KEYS list so source_file and definition_file stay
in lockstep. Together with #3243 this closes all four persistence paths (graph.json, AST
cache, watch, direct extract).
#2990 stamped definition_file with the still-absolute implementation path, so graph.json
shipped the build host's absolute path beside a repo-relative source_file (and the AST cache
replayed it on every warm rebuild). Normalize definition_file through the same
_norm_source_file path as source_file in build_from_json and the cache round-trip, so it is
relative/portable and byte-stable across machines.
A qualified Kotlin supertype (`class Foo : com.example.Base()`) resolved to the first
identifier (`com`), collapsing distinct bases onto a bogus node. Take the last segment of the
user type (handling generics like `Container<String>` -> `Container`), so the edge targets
the real base; simple and delegated supertypes are unaffected.
A qualified Scala base (`extends pkg.Base with other.Trait`) parses as a
stable_type_identifier (or generic_type wrapping one), which the heritage handler did not
match, so the edge was dropped or targeted the full qualified string. Add a helper that
takes the tail type name across all three shapes, emitting inherits for the first parent and
mixes_in for the rest, with no package/generic-arg false edges.
Only class_declaration was treated as a class type, so PHP interfaces, enums, and traits
produced no node and no heritage edges. Add them as class-like containers (and walk the
enum body) so their extends/implements/use heritage is captured as inherits/implements/
mixes_in edges to the right targets.
A plain JavaScript `class Dog extends Animal` produced no inherits edge — only the
TypeScript extends_clause/implements_clause shapes were handled, not the bare identifier
directly under class_heritage that JS emits. Fall back to reading heritage identifiers when
no clause is present, so JS class inheritance is captured; a dynamic base (mixin call) yields
nothing and TS is not double-emitted.
Add a Robot Framework extractor using the official robot.api parser (pure-Python, optional
[robot] extra), emitting suite/test/keyword nodes plus keyword-call, resource-import, and
library edges. Keyword resolution is case/space/underscore-insensitive (RF's real semantics)
and handles BDD prefixes and ${CURDIR}/${EXECDIR} path resolution; degrades to an actionable
error when robotframework is absent.
On the default incremental path a semantic chunk that returned successfully but under-produced
(prior semantic count > 1, fresh < prior) bypassed every guard, silently shrinking the graph.
Flag that per-source shrink (computed only from fresh nodes, so a genuinely deleted file is
never flagged) and route it through the existing shrink-guard refusal + manifest un-stamp so
the source re-extracts next run; --allow-partial overrides. Does not touch the prune path.
Follow-up to #3191: an authored markdown link whose target basename is present in the graph
but under a different spelling (e.g. Windows-case [[target]] vs TARGET.md) was dropped on a
code-only rebuild. Preserve such an edge when its target node is still live in the merged
graph, without loosening the deleted-target existence gate (a genuinely deleted target's node
is evicted first, so it is never preserved).
When the installed SKILL.md has diverged (user edit or old version), back it up beside the
install target before the atomic overwrite and print an actionable warning naming the stale
path and the exact reinstall command; a clean up-to-date reinstall backs up nothing (no
churn), and only the installed copy is touched, never the generated sources.
The hook-guard matched a search command appearing anywhere in the intercepted Bash string
(a comment, a quoted -m message, an echo), firing false nudges. Match only an executed
command position (start-of-segment after a shell operator, wrapper-aware), so prose lookalikes
stay quiet while genuine pipes/&&/;/$() invocations still fire.
Only a handful of control tokens were defanged, so an attacker-chosen token outside the set
(e.g. Llama-3 <|eot_id|>) could still reach the prompt. Generalize the neutraliser to match
the control-token form (<|...|> and [/?INST|SYSTEM]) at the single _wrap_untrusted chokepoint;
the bounded charset leaves legitimate content (|| , markdown tables, generics) untouched.
The GRAPH_REPORT headline figures disagreed: shown-community count included empty communities
the render loop skips, and the knowledge-gaps thin count used a hardcoded 3 beside a label
printing min_community_size. Drive every figure from one _real_count predicate so the counts
reconcile with what is actually rendered.
A god-node relation group capped at 20 targets showed no indication it was truncated; append
a '…and N more' line (exact count, degree-sorted so the lowest-degree are dropped) only when
the group actually exceeds the cap.
The reconcile relies on the extractor only stamping target_file when the link target exists
on disk. Add a negative test: link text intact but the target file deleted must leave no
reference edge and no edge pointing at the deleted document's node.
A code-only watch/update rebuild never re-creates a semantic-backed document node, so an
authored [[wikilink]] to such a doc resolved to a missing canonical id and reconcile
silently dropped the edge. Re-parse each markdown source and repoint authored references
edges onto a unique representative node for the target file (canonical id, else unique page
node, else unique document node; ambiguous -> keep the existing edge), preserving links to
the semantic tier across a code-only rebuild without resurrecting a deleted target.
graphify --help omitted the real prs and provider commands and the export format list. Add
them (verbatim from the dispatcher's own usage text), so every listed command/format exists
and the export formats match the supported allow-list.
T-SQL routine idioms (CREATE OR ALTER, the PROC shorthand, bracket-delimited names like
[dbo].[Get Widgets]) parse to tree-sitter ERROR nodes, so the routine names were lost. Hoist
a single routine-recovery pattern covering all three forms and run it over a
comment/string-masked copy of the file so commented-out or dynamic-SQL DDL cannot fabricate
nodes; labels keep the raw bracketed text and plain CREATE PROCEDURE/FUNCTION is unchanged.
This reverts #3155 (b568488): the source_file backfill also fired on a shared external stub
(e.g. a cross-file `Path` reference with no source_file), giving the stub a file identity so
the orphan prune swept it when that one file was deleted even though another file still
referenced it — regressing test_prune_sweeps_orphans. The fix needs to distinguish a
legacy node that genuinely lost its provenance from an external stub that must stay
sourceless; sending #3155 back as needs-work.
Wire Razor .razor/.cshtml @inject/@using through the existing scope-aware C# type resolver
instead of a bespoke path: @inject emits a references edge to a sourceless stub, @using
feeds scoping, and the type-resolution + stub-rewire passes now accept Razor sources. A
cross-file injected type resolves to its real definition; an external/undeclared type
fabricates nothing and a qualified inject does not mis-bind a same-named type.
Partially addresses #3187: a bare @inject of a type in a file-scoped namespace still dangles
(same as a bare C# cross-namespace reference without a using), and implicit _Imports.razor
global usings are not yet modeled — left as follow-up, so this does not auto-close the issue.
get_node used a raw substring scan and returned an iteration-order match, so it could miss
or mis-pick a node that get_neighbors resolves. Extract the shared tiered resolver
(_resolve_single_node) and route get_node through it — deterministic tiered winner or an
explicit ambiguity report, matching get_neighbors byte-for-byte; get_neighbors behavior and
return shapes are unchanged.
The prs MCP tools (list_prs, triage_prs, get_pr_impact) returned genuine failures (gh
missing/unauth, unresolved PR) as success text. Raise a ToolError propagated as isError on
both the 1.x and 2.x SDK paths, so a client sees the failure; an empty-but-successful result
(0 PRs, PR with no changed files) is unaffected. Mirrors the #2725 MCP-error pattern.
zfaustk