A full graphify update evicted every edge whose source_file was a
re-extracted source, including LLM semantic edges (e.g.
semantically_similar_to) from Markdown docs that also have an AST
extractor. The AST pass cannot regenerate those edges, so they were
silently lost while their concept nodes survived — node eviction is
provenance-aware via _origin (#1116), edge eviction was not.
Tag AST-extracted edges with the same _origin=ast marker nodes already
carry, and scope rebuild-driven edge eviction to that tier: re-extraction
replaces a source's AST edges, while its semantic edges survive until a
semantic re-extraction supersedes them. Deletion-driven eviction stays
provenance-blind, so edges of deleted or excluded sources are still
purged regardless of tier.
Edges from graphs built before this change lack the marker; a stale
AST edge from a file changed exactly once between the old and new
version can linger until its source is deleted — the same migration
trade-off the #1116 node marker made.
#1880 is the update-layer symptom of the #1873/#1887 nested-ignore
subtree-scoping regression (fixed in fb4d452, @Alwyn93): a nested broad
`.gitignore` zeroed update's re-scan, so it built 0 nodes and the
shrink-guard refused to overwrite. Adds a _rebuild_code regression test
asserting update sees the real files (and still scopes the nested ignore
correctly) so this can't recur at the update layer. Records both
regressions in the unreleased 0.9.16 changelog.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Non-anchored patterns from a nested .gitignore/.graphifyignore were
matched against root-relative paths first, so a nested ignore file's
patterns leaked outside its directory. In the wild, .hypothesis/.gitignore
(a bare "*" auto-written by the hypothesis library) ignored the entire
repository and detect() returned 0 files.
Per gitignore semantics, patterns from A/.gitignore apply only to paths
under A. Match every pattern against the path relative to its own anchor
(the anchor dir itself exempt — an ignore file governs its directory's
contents, not the directory), and skip patterns whose anchor does not
contain the target.
Regression introduced with nested-ignore support in 8a5287a (#1206).
tests/test_detect.py: 143 passed, including the existing nested-ignore
and nested-negation tests plus two new regressions.
Fixes#1873
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The #1757 batch-scoping followup built the per-chunk allowlist by reading
FileSlice.rel, which does not exist (a FileSlice carries its parent file
in .path). So every chunk containing a sliced oversized document leaked
the FileSlice object into the allowlist, save_semantic_cache raised
TypeError on Path(FileSlice), and the best-effort except swallowed it:
extraction finished but those chunks were never checkpointed, so a
re-run or a crash/rate-limit resume re-billed them.
Resolve each unit through the canonical unit_path() helper so a slice
maps to its parent file. Adds a regression test that slices a real
oversized .md and asserts the checkpoint writes without swallowing a
TypeError.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the regression test the #1855 fix was missing: _rebuild_code must
produce graph.json whose clustered nodes carry community_name, guarding
against the label-stripping regression recurring. Also records #1847 and
#1855 in the 0.9.15 changelog.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`graphify update` (and the git hooks that share its code path) wrote
graph.json with numeric-only community IDs instead of readable hub
names, because `_rebuild_code` called `to_json()` without the
`community_labels` parameter that `cluster-only` already passes.
Running `cluster-only` right after `update` "fixed" the names again,
which was the visible symptom.
- watch.py: forward the already-computed `labels` dict into `to_json`,
matching the `cluster-only` code path (cli.py:1203).
- watch.py: `_canonical_topology_for_compare` now also strips the new
`community_name` field before diffing topology. Without this, the
topology-unchanged cache (used to skip a full re-cluster) would
never match again once graph.json started carrying names, forcing
a full re-cluster on every subsequent `update` even with no code
changes.
Fixes#1808
Tighten the brittle negation test to use .py files so classification
lands in the deterministic `code` bucket (was checking a fuzzy
document+unclassified union), and add a composition test asserting a
nested `.gitignore` `!` re-include outranks both a root `.gitignore` and
`.git/info/exclude` (#1810) — locking the precedence across all three
ignore sources.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
detect() only read .gitignore/.graphifyignore in the scan root and its
ancestor directories (up to the nearest VCS root), loaded once before the
walk began. A .gitignore sitting in a descendant directory — e.g.
vendor/sub/.gitignore — was never read, so files/dirs it excluded leaked
into the graph. Real git (and every other gitignore-aware tool) honors
.gitignore at every directory level, not just the ancestor chain.
Extracts the per-directory read+parse logic into a shared
_load_dir_own_ignore() helper (used by both the existing ancestor-chain
loader and the new call site) and invokes it live inside detect()'s
os.walk loop for every directory visited, before that directory's
children are pruned — so a nested ignore file governs its own subtree
with the same closer-file-wins precedence git uses.
Adds three regression tests: nested file exclude, nested directory prune
(the walk never descends into it), and nested negation overriding a
broader root-level rule.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The HTML report's neighbor "focus" links dropped an unescaped
JSON.stringify(nid) into a double-quoted inline onclick. The stringified
value carries its own quotes, so the attribute was truncated on every
node (links never worked), and a node id/label containing a double-quote
broke out of the attribute and injected live event handlers. AST ids are
[a-z0-9_]-safe, but ids/labels from documents or titles scraped via
`graphify add <url>` are not, so a hostile source could plant an
executable handler into a locally-opened report.
Carry the id in an HTML-escaped data-nid attribute and dispatch via one
delegated listener bound to document (survives the innerHTML rebuild that
recreates #neighbors-list). Closes the injection and repairs the links.
Reported by @edgestack-ai.
Co-Authored-By: edgestack-ai <edgestack-ai@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes #1774. The prior fix redirected the AST cache dir to CWD but
file_hash still called _ensure_stat_index(root) without the cache
location, so the hash fastpath's stat-index.json kept anchoring on the
key-root (the analyzed corpus) — leaving a stray graphify-out/cache/
stat-index.json inside a writable foreign corpus even though the AST
cache itself had moved to CWD.
Thread cache_root through file_hash -> _ensure_stat_index (which already
accepts it, #1747). Surfaced by an out-of-CWD parallel-extract edge case:
the leak was masked in the in-process test suite because _stat_index_root
is a set-once module global that an earlier test had already pinned. The
regression test resets that global to simulate a fresh process and
asserts the corpus stays clean while the stat index lands under CWD.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
With no explicit cache_root, extract() wrote graphify-out/cache/ under the
inferred common parent of the inputs — the analyzed source tree — so
scanning a read-only/foreign corpus silently polluted it.
The naive fix (point the root at CWD) breaks two other things that shared
the same parameter: file_hash keys become absolute/non-portable for an
out-of-CWD corpus, and the XAML/C# project-scan boundary would scan CWD
instead of the corpus. So split cache LOCATION from key/id ANCHOR:
load_cached/save_cached gain a cache_root arg for where the dir lives,
while `root` (inferred common parent) still anchors file_hash keys,
source_file relativization, node ids, and the XAML boundary. extract()
now locates the cache at CWD (or cache_root) but anchors on `root`; the
parallel worker tuple carries both. Existing callers passing cache_root
(CLI, watcher) are unchanged.
Adopts @SimiSips's #1802 (the CWD default + the two location tests) and
adds the decoupling plus a regression test that keys stay relative for a
corpus outside CWD — the property the one-line version would have lost.
Co-Authored-By: SimiSips <SimiSips@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Many nodes sharing one generic label (framework route handlers all
labelled GET/POST, a repeated handler) consumed every BFS seed slot, so
query traversal explored near-identical neighborhoods and buried the
actual target. Seed selection now dedups by normalized label (GET/Get/get
collapse together), keeping one representative per label, and the per-term
guarantee loop honors the same cap so it can't reintroduce a dupe.
Adopts @devcool20's seed-dedup from #1832 but drops that PR's second
mechanism — a per-label multiplicity penalty applied inside the shared
_score_nodes. That scorer also resolves shortest_path/explain endpoints,
so dividing scores there silently reweighted path/explain (out of scope
for #1766 and able to flip endpoint selection); the dedup alone bounds
the flood. Also normalizes the dedup key (the PR keyed on the raw label).
Adds the tests the PR was missing: dedup of homonymous labels,
case/diacritic normalization, per-term-guarantee cap, and a guard that
identical-label nodes still score equally in _score_nodes.
Co-Authored-By: devcool20 <devcool20@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The #1835 fix scoped save_semantic_cache's final CLI write to an
allowed_source_files allowlist, but the per-chunk incremental checkpoint
in llm.py `_checkpoint_chunk` — the write that actually runs on every
`graphify extract`/`update` via extract_corpus_parallel — still called
save_semantic_cache with no allowlist. A chunk whose model result
mis-attributes a node's source_file to another corpus file would merge
that stray fragment into the victim's cache entry (merge_existing=True).
Scope the checkpoint write to the chunk's own dispatched files (FileSlice
-> .rel, bare Path -> the relative source_file). Also hoist the
`import warnings` in cache.py to module level.
Adds an extract_corpus_parallel integration test: a chunk dispatching
only A.py that returns a node attributed to already-cached B.py must
leave B.py's cache entry untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#1831 — `graphify export graphml` crashed on any dict/list-valued
attribute (per-node metadata dict, graph-level hyperedges list) because
nx.write_graphml only accepts scalars; a real ~2,300-node graph failed
every export and left a 0-byte .graphml behind. to_graphml now coerces
None->"" and JSON-serializes non-scalars across graph/node/edge scopes
(int/float/bool/str pass through), and writes atomically via a temp file
so a failed export can't leave a partial file. Closes#1830.
#1807 followup — adopt @varuntej07's explicit in-guard sys.stdout.flush()
from #1811: piped stdout is block-buffered, so a small fully-buffered
output would only flush at interpreter shutdown (outside the guard),
where a closed-pipe reader escapes as a noisy shutdown error and nonzero
exit. Flushing inside the try closes that gap. Closes#1811.
Reported by @hofmockel (#1831) and @varuntej07 (#1807/#1811).
Co-Authored-By: hofmockel <hofmockel@users.noreply.github.com>
Co-Authored-By: varuntej07 <varuntej07@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#1810 — detection read only .gitignore/.graphifyignore, never
.git/info/exclude, which is where git records local-only excludes and
where `git worktree add` writes nested worktree paths. graphify walked
into those worktree copies and the graph exploded (one 5-worktree repo:
9.4k nodes/10MB -> 210k nodes/311MB, ~77% duplicate). detect now loads
info/exclude at lowest precedence (below every per-dir .gitignore, per
git, so a nearer `!` still wins) and resolves the linked-worktree /
submodule case where `.git` is a file to the shared common git dir.
#1809 — two git-hook gaps: (a) post-checkout never honored
GRAPHIFY_SKIP_HOOK, so the var stopped commit rebuilds but not
branch-switch ones; now checked in both. (b) with core.hooksPath shared
across worktrees, a commit in any linked worktree fired post-commit,
which wrote a rogue delta-only graph.json into it and raced deploy/CI
`git clean` against the detached rebuild. Both hooks now short-circuit
in a linked worktree (git-dir != git-common-dir), comparing ABSOLUTE
paths so the primary checkout (where --git-common-dir is the relative
".git") is never false-positived and skipped.
Adds regression tests: info/exclude honored + negation precedence;
both hooks honor the skip env and carry the worktree guard; and an
end-to-end guard check against a real `git worktree`.
Reported by @cdahl86-cyber (#1810, #1809); the worktree guard was
co-developed with @Claude-Madera's PR #1806.
Co-Authored-By: cdahl86-cyber <cdahl86-cyber@users.noreply.github.com>
Co-Authored-By: Claude-Madera <Claude-Madera@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#1807 — piping graphify into a reader that stops early (head,
Select-Object -First N, sed q) disconnected stdout mid-write, raising an
unhandled BrokenPipeError (OSError(EINVAL) on Windows) and exiting 255,
so CI wrappers and agent harnesses read a successful query as a failure.
The console entry point now wraps the CLI body: a closed-pipe reader is
treated as success — stdout is redirected to devnull so shutdown flush
can't raise again, and the process exits 0. Adds a subprocess regression
test.
#1804 — .nox/ (nox virtualenvs, tox's successor, same .nox/ tree shape)
was missing from _SKIP_DIRS while .tox was present, so nox site-packages
got fully indexed (one repo came out 91% venv noise). Added next to .tox
with a regression test.
Reported by @varuntej07 (#1807) and @igorregoir-lgtm (#1804).
Co-Authored-By: varuntej07 <varuntej07@users.noreply.github.com>
Co-Authored-By: igorregoir-lgtm <igorregoir-lgtm@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A solution folder is a virtual grouping, not a file: VS writes its name
as both the display name and the "path" (name == path, no real file).
extract_sln resolved it to an absolute filesystem path anyway and keyed
the node id off that. The CLI id-relativization pass only remaps ids of
real files in the scan set, so a virtual folder never matched and its
absolute id (with the local username) survived into a committed
graph.json.
Detect solution folders (name == path) and key their id/source_file off
the folder name only; real project files still resolve as before. Adds a
regression test asserting the folder node id is relative.
The earlier fix (0.9.13) covered .csproj/.sln file nodes but missed the
virtual folders, so #1789 was closed prematurely; this completes it.
Reported and diagnosed by @fremat79.
Co-Authored-By: fremat79 <fremat79@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
querylog wrote every query/path/explain question + corpus path (and full
responses under GRAPHIFY_QUERY_LOG_RESPONSES) to a default-on, unbounded,
fail-silent plaintext file at ~/.cache/graphify-queries.log — outside any repo's
.gitignore/retention, and undocumented. A default-on plaintext record of
proprietary queries contradicts graphify's on-device / no-telemetry posture.
Flip to opt-in: _log_path() returns None unless GRAPHIFY_QUERY_LOG_ENABLE=1
(default path) or GRAPHIFY_QUERY_LOG=<path> is set; GRAPHIFY_QUERY_LOG_DISABLE=1
still forces it off (back-compat, wins). Document all four env vars in the
README (the old entries implied default-on). Regression tests cover
default-off, both enable paths, disable-wins, and that log_query writes nothing
without opt-in.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The semantic pass mints a document node <slug>_doc; the markdown quick-scan
(extract_markdown) mints the bare <slug>. After a semantic build, a `graphify
update` (AST path) re-runs the quick-scan and the graph ends up with BOTH — one
document as two disconnected nodes, the file's edges split between them (semantic
`references`/hyperedges on the _doc twin, quick-scan cross-links on the bare
one). path/query traversals dead-end on the wrong twin; degree and communities
split.
build_from_json now reconciles the pair: when <slug> and <slug>_doc both exist
with the same source_file and both are file_type=document, remap the bare node
into the semantic _doc node (canonical, richer edges) and repoint its edges and
hyperedges. Remap-induced self-loops are dropped; pre-existing ones are left
alone. Gated to document twins for the same file, so a code symbol `foo` and an
unrelated `foo_doc` never merge. Regression tests for both.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_reconcile_existing_graph treated "source identity absent from the collected
corpus" as deletion and evicted its nodes/edges/hyperedges. But corpus absence
is ambiguous: it's also what you see when a file still exists and merely stopped
being collected (ignore rules or filters changed). Upgrading into the merged-
.gitignore scan semantics (#1363) mass-evicted 655 nodes from a deliberately-
built, .gitignore'd docs dir whose files were present the whole time — reported
as a successful rebuild.
Fail-closed: before evicting a corpus-absent identity, require Path(identity)
.exists() is False (identity is an absolute path). Alive-but-excluded sources
are preserved (nodes, edges, hyperedges) and a loud line reports how many were
kept and why. True deletions and renames still evict (old path gone from disk);
a full extract --force still purges deliberate exclusions via the AST ownership
rule. Existence is memoized (one stat per file that left the corpus).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Switch every website URL from graphifylabs.ai to graphify.com — the hero logo,
the Penpax section, and the waitlist link — across the main README and the
translated READMEs. The contact email stays on graphifylabs.ai (mail is hosted
there); no mailto links were changed. graphify.com is the official graphify
product site; graphifylabs.ai remains the company site (org profile + email).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
build_merge already drops a re-extracted file's stale base nodes before merging
(replace-per-source), so on current code an EDITED file passed only in
new_chunks is handled correctly. But the prune step still removed every node
whose source_file was in prune_sources, with no guard for re-extracted files —
so a caller following the old edit-workflow (pass the changed file in BOTH
new_chunks and prune_sources) had its freshly-built nodes deleted after the
merge, silently losing a concept whose label survived the edit.
Exclude new_sources (files present in new_chunks) from prune_set: a re-extracted
file is being replaced, never deleted, so "replace" wins over a contradictory
"delete" of the same source. Genuine deletions (in prune_sources but not
new_chunks) still prune. Regression tests for both.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The absolute-path-in-node-ids leak reported on 0.8.19 is already fixed on v8:
detect() returns paths relative to the scan root, so the CLI-produced graph.json
uses relative structural node ids (portable, no username/home leak). Lock it with
a regression test that extracts the same corpus from two different absolute
checkout dirs and asserts identical, leak-free node ids.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`graphify path` committed each endpoint to _score_nodes()[0]. The full-query
bonus tier only fires when the query equals/prefixes a label, so a query that is
a token subset of the intended label ("Reject-everything judge" vs "Degenerate
Reject-Everything Judge") got no bonus and a node prefix-matching one rare token
("Rejection Summary") could out-score it on IDF alone — anchoring the path on an
unrelated, often disconnected node and returning a false "No path found".
_pick_scored_endpoint() scans the score-ordered list and takes the first
candidate whose label contains EVERY query token, falling back to scored[0] when
none does — so when the head already full-matches (the common case) resolution
is unchanged. Wired into both the `path` CLI and the MCP _tool_shortest_path.
The close-runner-up ambiguity warning now fires only when the picked endpoint is
the raw score head (a full-token override was chosen on coverage, not score, so
the head's margin is irrelevant).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
suggest_questions()'s "isolated/weakly-connected nodes" filter was missing the
`file_type != "rationale"` exclusion that report.py's Knowledge Gaps section
already applies, so the same GRAPH_REPORT.md reported two different counts for
the same concept (757 vs 245 on a real graph) — an internal inconsistency that
made a healthy graph look like a documentation problem. Add the same filter so
both computations agree.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
extract_bash only created a cross-file edge for `source x.sh` / `. x.sh`. The
two most common ways one script runs another — `bash x.sh` and `./x.sh` —
produced no edge, so in any repo where scripts invoke each other by execution
the call topology was missing (each script left an isolated file+entry pair).
Emit a `calls` edge (context `script_invocation`) from the caller's entry (or
enclosing function) to the invoked script's entry node, for script-runner
commands (bash/sh/zsh/ksh/dash <path>) and bare `./x.sh`, but only when the
target resolves to a real .sh file on disk — so no phantom edges to missing or
function-shadowed names. Verified end-to-end: the edges land on real target
nodes (no dangling drop at build).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
.rake files are plain Ruby (Rake's task DSL is ordinary method calls), but the
extension was gated out everywhere, so rake tasks were classified as
unsupported, skipped, and their calls invisible. Add `.rake` to all seven `.rb`
gates the reporter mapped:
- detect.CODE_EXTENSIONS (classification)
- extract._DISPATCH (extractor dispatch)
- extract._LANG_FAMILY_BY_EXT-adjacent language-name map (.rake -> ruby)
- the ruby_member_calls LanguageResolver suffix set
- both `.rb`-suffix filters in ruby_resolution.py (raw-call gather + class-def index)
- analyze language-stats map
- build repo-tag map
The extractor already parsed the content; this is purely extension routing.
Regression test: a `.rake` task's `Widget.tally` resolves cross-file to the
`.rb` definition.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The reported bug — a method chained directly onto a `new X(...)` expression
(no intermediate variable) producing no calls edge — is already fixed on v8:
`new Merger(ctx).Combine(...)` emits calls -> Merger.Combine. Add a regression
test so the fluent new-expression receiver stays covered.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_rewire_unique_stub_nodes gated merge targets through _is_type_like_definition,
which rejects any label ending in `)`. So a function referenced from another
module (passed by name, e.g. FastAPI's Depends(get_db)) left its reference edge
dangling on a sourceless name-only stub while the real def had zero incoming
edges — "who references this function" returned nothing. Class/type symbols were
fine; only functions/methods suffered.
Top-level function defs (label `name()`, not `.name()` methods or `Class.m()`
qualifiers) are now eligible rewire targets, but only when:
- the label key matches exactly one such function corpus-wide (existing
unique-candidate guard — two same-named functions stay unresolved), AND
- the candidate shares a language family with the stub's referrers, so a
Python `get_db` reference can't bind to a unique Go `get_db()` (#1718/#1749
interop guard), AND
- the stub is not used as a supertype (inherits/implements/extends) — you
don't inherit from a function.
Types are unchanged. Regression tests: cross-module function ref binds to def;
cross-language, ambiguous, and supertype cases all correctly left unresolved.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The TypeError reported on 0.1.14 is already fixed on v8: sanitize_label coerces
None ('if text is None: return ""') and the source_file call site guards with
str(data.get("source_file") or ""). Add regression tests (unit + to_html
integration with null label/source_file) so it can't silently regress.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The badge's href pointed at www.ycombinator.com/companies/graphify, which 404s
— the public S26 company page isn't published yet. Show the badge without a
link rather than ship a dead click; re-add the href once YC publishes the page.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The live-introspection FK query joined information_schema.referential_
constraints, which Postgres only exposes for constraints where the current user
has WRITE access to the referencing table. A read-only introspection role
therefore got zero FK rows — while tables/views/routines still appeared (SELECT
is enough for those views) — so the graph silently lost every `references`
edge, contradicting the documented FK-mapping behavior.
Switch to pg_catalog.pg_constraint (world-readable, not privilege-filtered),
keyed by constraint oid rather than name — which also fixes a latent bug where
same-named constraints on sibling tables could cross-match in the old
name-based key_column_usage joins. Composite-FK column order is preserved with
UNNEST(conkey/confkey) WITH ORDINALITY. Mock test asserts the query targets
pg_constraint and not the privilege-filtered view, plus composite-FK ordering.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
extract_json emitted `imports` edges for package.json dependencies and
`extends`/`$ref` edges for tsconfig.json to target ids (`_make_id("ref", ...)`
/ `_make_id(key)`) that it never created as nodes. build_from_json drops edges
to unknown node ids silently (that case is filtered out of real_errors), so
dependency and extends structure vanished from the graph on two of the most
common files in any JS/TS repo, surfaced only by diagnose_extraction after the
fact.
The extractor now adds the referenced target as a `concept` node (external ref,
not a corpus file) before emitting each edge, so the edges survive build.
Regression test asserts no dangling endpoints, the concept nodes exist, and the
import/extends edges land on real targets with no self-loops.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`graphify update` (the watch._rebuild_code / _reconcile_existing_graph path)
evicted every hyperedge whose source_file is in the corpus, because on a full
update every corpus file counts as "rebuilt" and hyperedge eviction reused the
node/edge eviction set. But the AST pass never emits hyperedges, so nothing
replaced them — doc-sourced hyperedges (what semantic extraction produces) were
permanently lost on the first update after a full build, even on a no-op run.
Split out a hyperedge_evicted_source_identities set scoped to genuinely deleted
(and symlink-target-outside) sources only, not merely-rebuilt ones. Replacement-
by-id (new_hyperedge_ids) and dangling-member cleanup are unchanged, so a real
semantic re-extraction still replaces its own hyperedges and orphaned ones are
still dropped. Parametrized regression test (full + incremental doc update).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Java member calls (`gw.charge()`) resolved by bare method name, so a call bound
to any same-named method in the corpus — e.g. `PaymentGateway.charge` and
`AuditLog.charge` were indistinguishable, producing phantom cross-class edges
and a false god node.
The extractor now preserves the receiver and its static type, and
_resolve_java_member_calls binds the call against the receiver's declared type:
explicit-type receivers and `this` are exact; current-class fields, method
parameters, and explicitly-typed locals resolve via a method-scoped type table;
a missing/ambiguous/inherited/chained receiver is skipped rather than falling
back to a bare name match (same single-owner god-node guard as the C#/Swift/Ruby
resolvers). Fully-qualified and nested-type receivers are deferred since they
need package- and nesting-aware type identity.
Verified: `gw.ping()/gw.charge()` (gw: PaymentGateway) bind to PaymentGateway,
the three charge() calls dedup to one edge, and no edge targets the same-named
AuditLog methods. Applies cleanly to the post-#1737 layout (extract.py +
extractors/engine.py). 13 new tests; full suite 3135 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Case 1 — `extract <corpus> --out <dir>`: the graph went to <dir> (cache_root is
already passed to the AST extractor), but detect()'s word-count/stat-index cache
uses the scan root, so a stray graphify-out/cache/ was created inside the corpus
(and left behind even when the run aborted at the no-LLM-key gate). Thread an
optional cache_root through detect() -> cached_word_count() -> _ensure_stat_index()
and pass out_root from the extract CLI, so the stat index lives under --out. Entry
keys are absolute paths, so relocating the index file is safe.
Case 2 — `cluster-only --graph <elsewhere>/graphify-out/graph.json`: outputs
(GRAPH_REPORT.md, re-clustered graph.json, labels, analysis, html) were written to
the CWD's graphify-out/, ignoring where --graph lives. They now write beside the
input graph when it sits in a graphify-out/ dir (another project/tenant's output),
while still falling back to the CWD for an arbitrary archived backup/graph.json —
the restore-into-place workflow #934 pins.
Regression tests for both cases; #934 still passes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The extraction spec forbids cross-language `calls` edges, and build already
dropped cross-language INFERRED `calls`. But `imports`/`references` had no such
guard: an unresolved Python `import time` resolved by bare stem (the #1504
old-stem alias) onto a `src/time.ts` file node, welding a polyglot repo's two
language halves together. In the reporter's repo three such edges were the only
bridge between 2409 Python and 1403 TS nodes, so every backend<->frontend
shortest path routed through time.ts, inflating its betweenness ~90x and making
it the #1 reported god node.
Hoist the interop-family map to a module constant and extend the edge-loop
guard to `imports`/`imports_from`/`references`. For these relations the edge is
dropped only when BOTH endpoints are known code languages of different families,
so a config/manifest -> code reference (unknown ext) is never mistaken for a
phantom. `calls` behavior is unchanged (still INFERRED-only, still drops when
either family differs). Regression tests: py->ts import dropped, ts->ts import
kept, config->code reference kept.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When the [sql] extra is absent, .sql files are counted as code and scanned but
extract_sql returns an error result and zero nodes — and the graph builds
"successfully" with the entire SQL corpus missing. Neither existing warning
catches it: #1666's zero-node warning skips results carrying an "error", and
#1689 only covers files with NO extractor at all (.sql HAS a dispatch entry).
extract() now scans per-file results for a "not installed" error, groups the
affected files by extension, and prints a warning naming the extra that
restores the language (pip install "graphifyy[sql]"), via a small
_EXTRA_FOR_EXTENSION map (sql, terraform, dm). The map is only consulted after
an extractor actually reports the dependency missing, so it can't mislabel a
language that has a working fallback.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
build_from_json's ghost-node merge iterated set(G.nodes()), so when two nodes
shared a (basename, label) key the "canonical" survivor was chosen by CPython's
per-process string-hash order — rebuilding the same extraction JSON in a fresh
process could pick a different survivor, silently changing which node id
represents a concept. That breaks any workflow persisting ids across a rebuild;
concretely it broke the cluster->relabel step (community membership referenced
an id that the second build merged away -> KeyError in report generation).
Two changes:
- Pass 1 and Pass 2 now iterate sorted(node_set), not set(node_set), the same
deterministic-order fix the edge loop below already uses on purpose.
- The #1257 ambiguity guard is extended to the case it did not cover: two
NON-AST nodes sharing a key but from DIFFERENT source files are distinct
concepts, not an AST ghost/canonical twin, so the key is marked ambiguous and
both survive rather than one arbitrarily merging away (data loss). A genuine
same-file duplicate (identical source_file) is not flagged and still
collapses to one node.
Reported with a precise root-cause, minimal repro, and real-world impact by
@erasmust-dotcom.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The extract_terraform move #1721 proposed already landed on v8 via the #1737
decomposition (extractors/terraform.py exists, extract.py re-exports it, and
extractors/LANGUAGE_EXTRACTORS registers it), so the code move is a no-op now.
But the regression test @Cekaru added with it had no equivalent on v8. Salvage
and generalize it: sweep every LANGUAGE_EXTRACTORS entry and assert graphify.
extract re-exports the SAME object (facade identity) and the registry maps to
it (registry identity), plus the concrete terraform anchor from the PR. This
institutionalizes the re-export-identity guarantee the split relies on, so a
future move that forgets a facade re-export fails loudly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two classes with the same simple name in different Maven modules
(FinancialEntryValidator in payment/ and core/) already survive as distinct
path-scoped nodes on v8 -- the "node silently disappears" report from 0.9.9 is
fixed. But a cross-module field/type `references` edge was still left dangling
on a sourceless phantom stub: _resolve_java_type_references (#1318) re-pointed
implements/inherits/extends/imports edges to the real definition using the
importing file's `import` statement, but its REPOINT_RELATIONS omitted
`references`, so bare-name resolution's shadow stub survived for field types.
A query about the referenced class could then miss it.
Add `references` to the Java resolver's REPOINT_RELATIONS. The C# sibling
already covers references; this brings Java to parity. The reference now
resolves to the imported package's class (falling back to same-package), and
the orphaned phantom is dropped. Regression test covers the ambiguous
two-module case: both reals present, no phantom, reference lands on the
imported class.
Reported with a precise root-cause and repro by @aviciot.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two correctness fixes found while analysing the reported 'graphify update
occasionally writes a partial graph.json' bug.
Enumeration (P0): detect()'s os.walk had no onerror handler, so any os.scandir
failure -- a transient PermissionError, or a directory created/deleted mid-walk
by concurrent writes (e.g. benchmarking racing the scan) -- was silently
swallowed and that entire subtree dropped out of the file list with no log, no
error. Downstream that becomes a silently partial graph.json. The walk now
records each skipped directory (surfaced as walk_errors in detect()'s result)
and warns to stderr, while still enumerating the rest of the tree. This stays
visible even when a --force/GRAPHIFY_FORCE rebuild bypasses the shrink guards.
Relatedly, to_json's #479 anti-shrink guard was fail-OPEN: a non-empty but
unreadable existing graph.json (corrupt or mid-write) proceeded with the
overwrite. It now fails SAFE -- refuse and point at force=True -- while an
empty/whitespace existing file (no nodes to lose) still proceeds. The size-cap
check keeps running before any read, so an oversized existing file is not
loaded into memory.
Pascal edges (P1): a class method declared in the interface section and defined
in the implementation section each emitted a "method" edge to the same node id,
and the edge helpers (unlike the node helpers) did not dedup, so ~half of a
Pascal/Delphi graph's method edges were doubled -- inflating degree/centrality
and tripping the #1739 cross-file resolver's single-owner god-node guard. Both
extractors now dedup edges on (source, target, relation).
Adds regression tests for all three behaviours.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both Pascal extractors resolved every call via a single file-wide
{method_name: node_id} dict, so two unrelated classes declaring a same-named
method (property accessors, generated COM/TLB wrapper classes) collapsed onto
whichever declaration was inserted last, producing wrong cross-class `calls`
edges. Resolution is now scoped: own class -> ancestor chain (inherits) ->
file-level free function -> unambiguous file-wide match; ambiguous at every
level emits no edge rather than guessing (same god-node guard as the Ruby
resolver).
Adds graphify/pascal_resolution.py, a corpus-wide post-extraction resolver
(registered via resolver_registry) that walks the inherits chain across file
boundaries, so a call from a manual descendant to a method it inherits from a
base class in a separate unit (the generated-base/manual-descendant split)
resolves. Also stops both extractors from emitting a duplicate base-class stub
carrying the referencing file's source_file, which collided with the real node
under cross-file id disambiguation. cache.py gives the new raw_calls bucket the
same portable-path treatment as nodes/edges so it round-trips.
Re-applied to the post-#1737 module layout (extractor hunks land in
graphify/extractors/pascal.py; registration stays in extract.py). Added one
adaptation the original PR predated: the cross-file resolver's god-node guard
now counts DISTINCT method nids, because the tree-sitter extractor emits a
method edge for both the interface declaration and the implementation, so the
same method_nid arrives twice -- without deduping, every inherited call looked
ambiguous and resolved to nothing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
