Node/edge ownership was keyed on source_file alone, but each file has an AST
tier and a semantic tier. A semantic-only re-extract deleted a doc's AST
headings, and a full _rebuild_code deleted document AST nodes. Add an
_is_ast_tier() predicate (with a source_location legacy fallback for the
unreliable _origin marker, #2334), backfill _origin on load so graphs
self-heal, make build_merge / merge_raw_extraction replace by (source_file,
tier) instead of source_file (coexist policy: an AST re-extract replaces only
AST nodes and keeps the semantic layer, and vice versa), and scope the
full-rebuild AST drop to sources actually regenerated (#2336) so a
semantic-backed doc keeps its AST layer. Deletion prune stays tier-blind.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The build_merge fix only covers the incremental path reached by
`graphify <path>`. `graphify update` rebuilds through _rebuild_code in
watch.py and never calls build_merge at all, so it still downgraded a
directed graph — which is the symptom named in the issue.
Two sites: the clustered path called build_from_json(result) with no
directed=, taking its directed=False default, and the --no-cluster path
built candidate_graph_data straight from the raw merged extraction, which
never carries a directed key, so the written file lost it entirely. Both
now inherit from existing_graph_data, defaulting to False when there is no
existing graph, matching build_merge's semantics.
_canonical_graph_for_compare normalizes a missing directed key to False so
an older graph.json without the key does not register as changed against a
candidate that now writes it explicitly.
_rebuild_code computed the correct output dir as `out = watch_path /
_GRAPHIFY_OUT` and routed every artifact through it except the three
save_manifest calls, which omitted manifest_path= and fell back to
detect._MANIFEST_PATH — a module-import-time constant built from the
relative out_path("graphify-out/manifest.json"), so it resolved against
the process CWD. `graphify update <target>` therefore wrote the target's
manifest into the invoking project and left the target with none.
save_manifest also read-modify-writes that path and full-scan callers
pass scan_corpus (#1908 pruning), so the invoking project's own rows were
pruned as out-of-corpus and overwritten. Both projects lost their
incremental baseline, not just the target.
Anchor the manifest keys to watch_root at the same three calls. root=
previously followed project_root, which is CWD-derived for a relative
invocation, so `graphify update ../other` stored out-of-root files under
absolute keys and broke #777/#1964 portability. project_root itself is
left alone: the persisted graph deliberately rehomes source_file against
it across invocation styles.
Also pass cwd= to _git_head. Without it `git rev-parse HEAD` inherited
the caller's directory, stamping the invoking repo's commit into the
target's graph.json built_at_commit (cf. #2081).
_reconcile_existing_graph loaded graph.json inside a swallowing try, so a
graph that was merely unreadable (over the size cap or unparseable) was
silently replaced by the code-only extraction, in both the clustered and
--no-cluster hook paths (force made it worse). It now loads through the
fail-closed build._load_existing_graph and _rebuild_code refuses the write
(prints and returns False) on a load failure, matching the CLI path; the
--no-cluster write is now atomic with a protected-graph backup.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Crossing MAX_NODES_FOR_VIZ left the project with no graph.html at all:
_rebuild_code unlinked the existing file and wrote nothing in its place. The
delete on its own is defensible — a kept graph.html would describe an older,
smaller graph — but the file is gone before the user reads the message, and
the next incremental rebuild silently removes it again, so a repo that grows
past the threshold just loses its visualization with no way to keep one.
The export path already solved this (#1019): over the cap it re-renders the
community-aggregation view rather than going without. Do the same here, so
the artifact is current AND present instead of current OR present.
GRAPHIFY_VIZ_NODE_LIMIT=0 still means "no HTML viz" (CI runners) rather than
"aggregate", and if the aggregated render also fails the old skip-and-remove
behaviour stands.
Community labels are saved keyed by community id, but re-clustering
reassigns those ids: after a rebuild that adds nodes, cid 30 can cover a
completely different community and its saved name is then simply wrong.
cluster-only already guards this — it validates each reused label against
the `.graphify_labels.json.sig` membership fingerprints and re-hubs any
community that changed (the case community_member_sigs() was written for).
_rebuild_code() skipped that check entirely: it reused every label whose
cid was present and hub-filled only the *missing* ones, so stale names
survived — and then wrote them back to labels.json, laundering them as
current. It also never refreshed the .sig sidecar, so the signatures kept
describing an older clustering and drifted out of step with the labels
they sit beside, leaving the cluster-only guard nothing accurate to check.
Adding ~3.5k nodes to a real graph re-clustered 463 -> 515 communities and
mislabeled 162 of them this way: a `domain.audit` namespace reading
"ACH / bank payments", a `domain.auth` namespace reading
"document-sensitivity.up.sql". Node and edge data stayed correct, so
nothing failed loudly — only the names lied.
Apply the same signature check in the incremental path, write the sidecar
in step with the labels, and print the same "run `graphify label`" notice
cluster-only emits so a drifted community set is visible rather than
silent.
A `cluster-only --no-label` run wrote "Community N" placeholders into
.graphify_labels.json plus a matching .sig, and the reuse path treated them as
fresh, so real labels were never regenerated on later runs. Two fixes: (a) don't
persist the labels sidecar (or its .sig) on a placeholder-only run, so a later
run generates real labels; (b) treat a stored "Community {cid}" as absent in the
reuse path so an already-polluted sidecar self-heals via the hub labeler while
genuine labels are still reused with no LLM call. The watch/update rebuild had
the same placeholder-perpetuation twin — fixed alongside.
The #2051 disk-absence sweep guarded remote/virtual sources with a literal
`"://"` check. But the write-side path normalization (Path.as_posix) collapses
the double slash, so a stored `gdoc://x` reads back as `gdoc:/x` on the next
update; the literal guard then missed it and the node fell into the disk-absence
branch (`Path('gdoc:/x').exists()` is False), evicting it on the second
`graphify update`. Match the scheme with a regex tolerant of the collapse, with
a 2+ char scheme so a Windows drive letter (C:/) is not misread as remote.
Regression test runs three consecutive updates and asserts the remote node
survives every one.
Three silent-data-loss fixes in the update/reconcile path:
- #2051: a full `graphify update` now evicts semantic nodes whose non-AST
source (a .txt/.pdf/.png with no code extractor) was deleted from disk.
The corpus sweep only checked re-extractable files, so deleted docs'/images'
LLM nodes survived as authoritative forever. Disk absence is now the deletion
signal; remote/virtual sources (`://`) are left untouched.
- #2056: an incremental rebuild whose change set names a present-but-
unextractable file no longer treats it as a deletion (which evicted its
semantic nodes and disabled the shrink guard). The guard now falls through to
per-source accounting instead of a wholesale bypass on any deletion.
- #2014: code-typed nodes the semantic pass surfaces from within a document now
count as that doc's semantic layer, so a rebuild doesn't re-scan and drop them.
Regression tests for each in tests/test_watch.py.
_rebuild_code fed Markdown/doc files (.md/.mdx/.qmd/.skill) into the AST
quick-scan while _reconcile_existing_graph preserved the same docs'
semantic (LLM) nodes, so every semantic-backed doc was represented twice
(heading nodes + concept nodes) and the graph bloated ~4x vs the CLI
`graphify . --update` path.
Semantic now supersedes AST per doc source: before choosing
extract_targets, read the existing graph's source identities that carry
semantic doc nodes (non-_origin=="ast", gated on file_type=="document"
so pre-#1865 marker-less graphs aren't misread) and exclude those docs
from the quick-scan in both the full-rebuild and changed_paths branches.
They stay in code_files, so the #1795 fail-closed corpus check and the
shrink accounting still cover them — a previously-bloated graph
self-heals on the next full rebuild (the AST ownership rule drops the
stale heading nodes) without the shrink guard refusing the smaller
write. On incremental rebuilds the excluded docs never enter
rebuilt/node-evicted identities, mirroring #1865's tier-scoped edge rule
at the node level, so a doc's semantic nodes and edges survive. Docs
with no semantic layer (and brand-new docs) keep the no-LLM quick-scan
structure from #09b33b7.
Two coupled excluded-vs-deleted fixes:
#1909 — incremental extract's prune set was derived from the manifest
alone (manifest - corpus), so a file that became excluded without ever
being manifest-listed (every pre-#1897 graph) kept its stale nodes in
graph.json forever. The prune set is now also derived from the existing
graph's own node source_files reconciled against the post-exclude detect
corpus (_stale_graph_sources), restricted to in-root paths; out-of-root
--include/symlinked entries and remote (://) sources are never pruned.
Relative source_files are anchored against both the scan root and the
--out root (the #555/#1899 relativized form). The --no-cluster
incremental early exit never runs build_merge, so an exclusion-only
change now prunes the raw graph.json in place instead.
#1908 — save_manifest retained any prior row whose file still existed
on disk, so an excluded-but-alive file survived as a permanent phantom
that detect_incremental reported as deleted on every run. Full-scan
callers (extract's saves, watch._rebuild_code's saves) now pass the RAW
detect corpus via a new scan_corpus parameter and in-root rows outside
it are dropped; the corpus is deliberately not the #933 stamp-filtered
files dict, so failed-chunk/omitted-doc rows and --code-only doc rows
survive. Subset saves (changed_paths hooks, #917) keep the seeding
default. detect_incremental now splits manifest rows that left the scan
into deleted_files (gone from disk) and excluded_files (alive but out of
scan), mirroring the watch-side #1795 distinction, and the extract
summaries report the two separately.
Ordering matters: extract's cleanup of newly-excluded nodes previously
worked only through the #1908 conflation, so the graph-source prune
lands together with the manifest split to avoid regressing #1909.
--exclude was honored only by the initial extract scan; update/watch/hook
rebuilds re-ran detect() without it and silently re-indexed excluded
paths. extract now writes the patterns to a graphify-out/.graphify_build
.json sidecar, and _rebuild_code reads and re-applies them on every
rebuild (layered after the ignore files, so they still win). Pre-existing
graphs without the sidecar behave as before. Adds a regression test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A full graphify update evicted every edge whose source_file was a
re-extracted source, including LLM semantic edges (e.g.
semantically_similar_to) from Markdown docs that also have an AST
extractor. The AST pass cannot regenerate those edges, so they were
silently lost while their concept nodes survived — node eviction is
provenance-aware via _origin (#1116), edge eviction was not.
Tag AST-extracted edges with the same _origin=ast marker nodes already
carry, and scope rebuild-driven edge eviction to that tier: re-extraction
replaces a source's AST edges, while its semantic edges survive until a
semantic re-extraction supersedes them. Deletion-driven eviction stays
provenance-blind, so edges of deleted or excluded sources are still
purged regardless of tier.
Edges from graphs built before this change lack the marker; a stale
AST edge from a file changed exactly once between the old and new
version can linger until its source is deleted — the same migration
trade-off the #1116 node marker made.
`graphify update` (and the git hooks that share its code path) wrote
graph.json with numeric-only community IDs instead of readable hub
names, because `_rebuild_code` called `to_json()` without the
`community_labels` parameter that `cluster-only` already passes.
Running `cluster-only` right after `update` "fixed" the names again,
which was the visible symptom.
- watch.py: forward the already-computed `labels` dict into `to_json`,
matching the `cluster-only` code path (cli.py:1203).
- watch.py: `_canonical_topology_for_compare` now also strips the new
`community_name` field before diffing topology. Without this, the
topology-unchanged cache (used to skip a full re-cluster) would
never match again once graph.json started carrying names, forcing
a full re-cluster on every subsequent `update` even with no code
changes.
Fixes#1808
_reconcile_existing_graph treated "source identity absent from the collected
corpus" as deletion and evicted its nodes/edges/hyperedges. But corpus absence
is ambiguous: it's also what you see when a file still exists and merely stopped
being collected (ignore rules or filters changed). Upgrading into the merged-
.gitignore scan semantics (#1363) mass-evicted 655 nodes from a deliberately-
built, .gitignore'd docs dir whose files were present the whole time — reported
as a successful rebuild.
Fail-closed: before evicting a corpus-absent identity, require Path(identity)
.exists() is False (identity is an absolute path). Alive-but-excluded sources
are preserved (nodes, edges, hyperedges) and a loud line reports how many were
kept and why. True deletions and renames still evict (old path gone from disk);
a full extract --force still purges deliberate exclusions via the AST ownership
rule. Existence is memoized (one stat per file that left the corpus).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`graphify update` (the watch._rebuild_code / _reconcile_existing_graph path)
evicted every hyperedge whose source_file is in the corpus, because on a full
update every corpus file counts as "rebuilt" and hyperedge eviction reused the
node/edge eviction set. But the AST pass never emits hyperedges, so nothing
replaced them — doc-sourced hyperedges (what semantic extraction produces) were
permanently lost on the first update after a full build, even on a no-op run.
Split out a hyperedge_evicted_source_identities set scoped to genuinely deleted
(and symlink-target-outside) sources only, not merely-rebuilt ones. Replacement-
by-id (new_hyperedge_ids) and dangling-member cleanup are unchanged, so a real
semantic re-extraction still replaces its own hyperedges and orphaned ones are
still dropped. Parametrized regression test (full + incremental doc update).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
What changed
- Stabilize relative rebuild execution before graphify-out queue/lock setup.
- Recover from deleted transient hook CWDs when GRAPHIFY_REPO_ROOT points at the repository root.
- Fail cleanly when the current working directory is gone and no repo root fallback is available.
- Add regression coverage for both fallback and clean-failure paths.
Why
- Detached post-commit/post-checkout rebuilds can inherit a transient CWD that is deleted before the background process starts.
- _rebuild_code previously used relative graphify-out paths before Path.cwd()/watch_path resolution could be handled, causing FileNotFoundError: [Errno 2] No such file or directory.
Validation
- .venv/bin/pytest tests/test_watch.py -q: 54 passed, 2 skipped.
- .venv/bin/ruff check graphify/watch.py tests/test_watch.py: passed.
- .venv/bin/python -m py_compile graphify/watch.py tests/test_watch.py: passed.
- env -u PYTHONPATH -u PYTHONHOME PYTHONHASHSEED=0 .venv/bin/pytest -q: 2904 passed, 30 skipped.
Notes
- Full suite without PYTHONHASHSEED=0 exposed an unrelated ordering-sensitive labeling test; with the deterministic hash seed used by graphify hooks, the suite passes.
Community labels defaulted to "Community N" whenever no LLM backend was configured,
making the report + suggested questions unreadable ("why does log_action connect
Community 70 to Community 129?"). Add `label_communities_by_hub`: name each community
after its highest-degree member — the structural hub — so the report reads "log_action"
/ "auth" with zero LLM cost. Ties break by node id for run-to-run stability; a community
with no members in the graph keeps the "Community N" placeholder.
Wired as the default base at both label-building sites — the label/standalone path in
__main__ and the watch/detect rebuild in watch.py. A configured LLM naming pass still
runs and overrides these with richer names; its no-backend placeholder fallback is
guarded so it can't clobber the hub labels.
Projects the verdicts `graphify reflect` already distills (preferred /
tentative / contested, exponential time-decayed) into a derived
experiential layer the read surfaces consume, so accumulated agent
experience actually shows up where you look — without polluting the
structural graph.
Design (grounded in agent-memory + provenance literature; a redesign of
the #1542 approach):
- SIDECAR, not graph.json stamping. `reflect` writes `.graphify_learning.json`
next to graph.json (an additional output, so the git hooks produce it
automatically). graph.json stays purely structural; nothing leaks into
GraphML; no graph.json churn. Mirrors the named-graph / event-sourcing
separation of durable truth from a derived layer.
- Reuses the existing reflect aggregate (its `_decay` is the
recency-weighted exponential model; `_finalize_sources` the
classification) — no new scoring.
- PROVENANCE: each verdict carries the source questions/dates that produced
it (cap 5, most-recent first).
- STALENESS: each verdict stores the node's file fingerprint; on read, a
changed source file flags the verdict stale ("code changed since —
re-verify") rather than presenting a confident lesson on rewritten code.
- CONTESTED surfaced distinctly (useful N / dead-end M), not averaged away.
- DEAD-ENDS stay QUERY-SCOPED — never a node-level status; they appear only
in the report as question -> nodes.
- Read surfaces (explain / query+MCP / GRAPH_REPORT / graph.html) merge the
overlay at read time, sanitized; un-annotated graphs are byte-identical.
Deferred (logged): letting verdicts influence query/seed traversal — the
recommender feedback-loop / Matthew-effect risk means that needs
propensity correction + exploration, not naive biasing.
Builds on the idea in #1441/#1542 (thanks @TPAteeq).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`graphify update` preserved old edges keyed only on endpoint-node membership, so a
removed import's edge survived forever whenever both endpoint nodes still existed
(e.g. a.py no longer imports b, but both a and b are still present). The stale edge
drove phantom circular-dependency findings; `--force` didn't help (it only gates
the node-count shrink guard), only a clean rebuild did.
_rebuild_code (watch.py) now also drops preserved edges whose source_file was
re-extracted this run — an edge is owned by the file it was extracted from, and the
fresh extraction re-emits whichever ones still exist. Scoped to source_file
ownership (and the full-rebuild case, where evict_sources lists only deleted
files), so edges with no source_file or owned by a non-re-extracted file are kept —
cross-file edges that merely point at a re-extracted file (#1402 sourceless
stubs/rewire) are not over-pruned.
Reported by @UltronOfSpace (#1521). Regression test: a removed import's edge is
gone after update; verified no over-prune (still-present imports survive) and no
regression to deleted-file pruning.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`graphify update` after deleting a function left the stale node in graph.json.
The build correctly dropped it (#1116), but _check_shrink then refused to write
the smaller graph ("Refusing to overwrite — you may be missing chunk files"),
so the deletion never persisted without --force. That also starved the
work-memory node-existence gate, which relies on graph.json reflecting deletions.
The shrink-guard now takes the set of source files re-extracted this run
(rebuilt_sources). A net shrink is allowed when every lost node belongs to a
rebuilt source (a symbol genuinely removed) or a deleted file; it is still
refused when a node vanishes from a file we did NOT touch — the silent
failed/partial-extraction case the guard exists to catch.
The #1116 e2e test now asserts the prune happens with force=False (was force=True);
added two _check_shrink unit tests (allowed within rebuilt sources, refused
outside). Full suite 2339 passed; skillgen --check clean; ruff clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#1418: build_from_json relativized source_file on nodes and edges but stored
graph.hyperedges[] verbatim, so a semantic subagent's absolute path leaked into
graph.json. Relativize hyperedges in build_from_json (to_json has no root to
relativize against), mirroring the existing node/edge handling.
#1423: consolidate the GRAPHIFY_OUT output-dir name into a single graphify.paths
module (was duplicated in __main__, cache, watch) and route the path guards
through it — security.validate_graph_path's base=None discovery + fallback,
callflow_html's project-root resolution, and the post-commit/post-checkout hook
bodies (which now read the env var at hook-run time). A renamed output dir is no
longer validated against the wrong base or missed by the hook.
Tests: hyperedge relativization (test_hypergraph), GRAPHIFY_OUT discovery
(test_security), updated the hook-body contract assertion (test_hooks).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adopts the approach from #1330 (thanks @duncan-daydream) on top of the v0.8.40
Swift import fix: _import_swift returns (id,label) module pairs, the extractor
materializes a type=module anchor node per import, and _disambiguate_colliding_node_ids
exempts type=module nodes so the same module imported from N files collapses to
one shared node (enables reverse traversal "what imports CoreKit"). The --no-cluster
writer now dedupes nodes by id and edges to match the clustered build_from_json path.
Replaces the interim _import_label/synthesize_import_module_nodes mechanism.
Adds tests/test_swift_import_resolution.py (cross-file collapse, build survival)
and dedupe_nodes coverage. Refs #1327, #1330.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- #1315: add .psm1 to CODE_EXTENSIONS + _DISPATCH so PowerShell modules are indexed
- #1327: synthesize a module node for Swift import targets (new LanguageConfig
flag synthesize_import_module_nodes) so imports edges survive build.py pruning;
strengthen the Swift dangling-edge test to also assert edge targets
- #1317: dedupe parallel edges by (source,target,relation) in the --no-cluster
and incremental update write paths so edge counts are deterministic and
`update` is idempotent
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes#1264
Also fixes a narrow watch-mode path-filtering bug found while running the required local
full-suite gate for this PR. Hidden directories inside the watched corpus are still ignored;
hidden ancestors outside the watched root no longer suppress valid file events.
#1118 — prune stale AST nodes on full re-extraction (#1116)
Stamps every AST-extracted node with _origin="ast" in extract(). On a
full rebuild _rebuild_code drops any AST-marked node absent from the
fresh output even when its source file survives, fixing stale symbols.
Backward-compat: marker-less nodes from pre-1118 graphs survive one
cycle then self-heal.
#1110 — stop reading images and PDFs as garbage in headless extract
Images route through per-backend vision payloads (base64/data-URI/bytes
for claude/openai/bedrock); non-vision backends get _strip_pixels for
graceful degradation. PDFs reuse pypdf. 5MB cap, 20-image chunk limit.
#1159 — Salesforce Apex extractor (.cls, .trigger)
Regex-based extractor: classes, interfaces, enums, methods, triggers,
SOQL/DML edges. No new dependency. Dispatched as .cls and .trigger.
#1107 — Azure OpenAI Service backend (--backend azure)
Uses AzureOpenAI SDK client (from existing openai package). Auto-detects
when AZURE_OPENAI_API_KEY + AZURE_OPENAI_ENDPOINT both set. Uses
max_completion_tokens (not deprecated max_tokens).
#1103 — live PostgreSQL introspection (--postgres DSN)
graphify extract --postgres "postgresql://..." introspects tables, views,
routines, and FK relations via information_schema (SERIALIZABLE READ ONLY).
Credentials sanitized on error. New graphify[postgres] extra (psycopg3).
Union-resolved llm.py conflict: Azure functions + bedrock images= param.
Fixed test_image_vision.py mock to accept timeout= kwarg (our #1112).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fixes#777. Relativize manifest keys, .graphify_root, and cache source_file fields on persist; re-anchor on load. In-memory callers still see absolute paths. Symlink round-trip fixed in follow-up commit 8f09326.
Three issues in _rebuild_code (watch.py):
1. _relativize_source_files was called on result after eviction list was built,
so existing nodes with absolute source_file were never normalized before comparison
2. deleted_paths and evict_sources used str() (backslashes on Windows) while
graph.json stores forward-slash paths via _norm_source_file
3. _relativize_source_files itself used str() instead of as_posix()
Also fix extract.py source_file relativization to use as_posix(). Closes#1007.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The post-commit hook passes `git diff --name-only HEAD~1 HEAD` as
changed_paths to `_rebuild_code`. That list includes deletions, and
`_rebuild_code` correctly identifies them (lines 352-367 in watch.py)
and evicts the stale nodes from the preserved set. The rebuilt graph
is intentionally smaller.
`_check_shrink` then refuses to overwrite the existing graph.json
because it only sees the node-count delta, not the cause. The guard
fires with "Refusing to overwrite — you may be missing chunk files
from a previous session. Pass --force to override."
Result: every commit that deletes a tracked file silently leaves stale
nodes in graph.json. The user must either pass --force (which also
disables the guard for legitimate failure modes) or manually re-run
`graphify update . --force` after delete-heavy commits.
Fix: thread a `had_explicit_deletions` flag from `_rebuild_code` into
`_check_shrink`. When the caller has declared the deletions, the
smaller graph is the expected outcome and the guard is skipped. The
guard remains intact for SILENT shrinkage — its actual purpose —
from failed semantic chunks or corrupted runs.
The fix is opt-in by design: callers that don't pass `changed_paths`
(e.g. the post-checkout full rebuild path) keep the old conservative
behavior. Only paths that explicitly track deletions get the bypass.
Tests added (tests/test_watch.py):
- `test_check_shrink_blocks_silent_shrink` — pre-existing behavior intact
- `test_check_shrink_allows_force_override` — pre-existing behavior intact
- `test_check_shrink_allows_explicit_deletions` — new: deletion bypass
- `test_check_shrink_allows_no_existing_data` — first-run case
- `test_check_shrink_allows_growth` — sanity
- `test_check_shrink_unlinks_tmp_on_refuse` — cleanup on refusal
- `test_check_shrink_keeps_tmp_when_deletions_declared` — no spurious unlink
- `test_rebuild_code_prunes_deleted_file_nodes` — end-to-end probe of the
exact scenario the post-commit hook triggers (git init, build, delete
one file, re-run with the deleted path in changed_paths, verify the
graph shrinks and the surviving file's nodes are preserved)
All 8 new tests pass; full test_watch.py + test_build.py + test_export.py
(74 tests) pass with no regressions.
* feat(bash): harden extractor — literal filtering, entrypoint nodes, AST-ancestry-aware command detection
Builds on tree-sitter-bash extractor from #866. Two correctness/security
improvements to bash extraction in graphify/extract.py:
1. Reject command/process substitutions at extraction time. Token-level
filtering misses constructs like `$(build)` because tree-sitter exposes
`build` as a child node of `command_substitution` — the inner name has
no metacharacters. Added `is_inside_expansion(node)` that walks
`node.parent` until it finds `command_substitution` or
`process_substitution`. Used as a gate in both `walk` and `walk_calls`.
Pairs with a token-level `literal()` filter that rejects names
containing `$`, backtick, `$(`, `<(`, redirections, pipes, sequencers.
2. Entrypoint node. Every .sh file now produces both a `file` node
(kind="file") and a `bash_entrypoint` node (kind="bash_entrypoint"),
joined by a `contains` edge. A separate top-level `walk_calls(root,
entry_nid, ...)` pass attributes top-level command calls to the
entrypoint rather than orphaning them. Matches the entrypoint pattern
other-language extractors use. Node metadata gains language+kind.
Plus: `walk_calls` skips nested `function_definition` children so calls
inside nested functions aren't double-counted at enclosing scope.
Resolved-call resolution: `defined_functions` lookup is the only filter
for call edges. User-defined functions named like external commands
(install, find, git, ...) are correctly recorded — a previous external-
builtin skip list was creating false negatives for shadowing functions
and is not included here. Skip list belongs with raw/unresolved call
recording (not in this PR).
Devtools (bundled): pyproject.toml gains [dependency-groups] dev (ruff,
pyright, pre-commit, hypothesis, pip-audit) plus minimal [tool.ruff],
[tool.ruff.lint], [tool.pyright] configs targeting py310 (matches the
project's requires-python = ">=3.10").
Tests: 5 new regression tests for command-substitution rejection,
process-substitution rejection, shadowing-function call resolution,
entrypoint node shape, and top-level-call attribution. 826/826 pass
(was 821); 15/15 bash-relevant tests pass (was 10).
* feat(detect): parse macOS/BSD and GNU env(1) shebang option forms
Upstream's _shebang_file_type parses shebangs via line[2:].split() and only
handles `#!/usr/bin/env <interp>`. Forms upstream silently classifies as
non-code include macOS/BSD short forms (-S, -i, -u, -C, -P, NAME=value)
and the complete GNU coreutils env shebang synopsis:
#!/usr/bin/env -[v]S[option]... [name=value]... command [args]...
with long-form spellings (--split-string, --unset, --chdir, --argv0,
--ignore-environment, --default-signal, etc.), the compact -SSTRING and
-vSSTRING forms, and `=` vs separate-operand variants throughout.
Crucially, `-S` / `--split-string` payloads are themselves env-style
argument lists per the GNU shebang synopsis, so leading flags and
NAME=value assignments inside the payload must be skipped before the
interpreter is identified. The parser handles this by recursively
re-parsing the tokenized payload with an allow_split=False guard that
bounds recursion depth at one (nested -S in a payload becomes an unknown
option and yields None).
Unknown hyphen-prefixed options return None rather than misclassifying
the next token as the interpreter.
_shebang_file_type becomes a 4-line wrapper. Read buffer raised 128 -> 256
to accommodate longer env -S strings.
Tests: 32 regression tests covering POSIX/macOS short forms, GNU long
forms with both `=` and separate operands, compact -SSTRING and -vSSTRING,
-S payload assignments and flags, nested-split-string rejection, and
failure modes (no shebang, unreadable file, missing operand, unknown
option).
* fix(skills): enforce semantic fragment validation in OpenCode + Codex merges (#825)
Closes#825. Adds graphify.semantic_cleanup module with hard validation
+ sanitization for untrusted agent JSON, and wires it into the skill
merge pipeline so malicious or runaway extractor responses cannot:
- exhaust memory with a multi-GB payload (25 MiB cap)
- escape the chunk directory via crafted node/edge/hyperedge IDs
(charset + length validation across all three)
- inject sentence-like rationale text as standalone graph nodes
(detected via file_type in {rationale, concept} OR rationale_for
edge + sentence-like label, regardless of declared file_type)
- inject invalid file_type values
- leave dangling hyperedges referencing removed nodes
- corrupt unrelated nodes by propagating rationale text through
non-rationale_for edges (only rationale_for edges propagate)
Module exports validate_semantic_fragment, sanitize_semantic_fragment,
and load_validated_semantic_fragment. Wired into skill-opencode.md and
skill-codex.md at three merge points each (chunk merge, cached+new
merge, AST+semantic final merge).
Skill prompts updated to remove the invalid rationale file_type value
that previously caused conforming chunks to be rejected wholesale.
Valid set is now {code, document, paper, image}.
Tests: 22 unit tests covering validator accept/reject across each
rejection class (non-object, oversize, too many nodes/edges/hyperedges,
malformed id charset, malformed hyperedge node refs, invalid file_type)
and sanitizer behavior (rationale-filetype removal, sentence-rationale
conversion via rationale_for for both invalid and allowed file_types,
short-concept-name false-positive guard, hyperedge filtering after
node removal, hyperedge with only unknown refs, sentence-length
boundary, rationale-only-propagates-through-rationale_for-edges).
880/880 tests pass.
* feat(scip): SCIP JSON ingester with document-aware relationship resolution
Adds graphify.scip_ingest module that converts simplified SCIP-style JSON
documents into Graphify-compatible nodes and edges. Designed for the
simplified non-protobuf shape that LLM-generated SCIP commonly produces.
Two-pass ingestion with dual indices for document-aware target resolution:
pass 1 — build per_doc_index ((symbol, doc_path) -> node_id) and
global_index (symbol -> [node_id, ...]) across every valid
symbol in every valid document. Same-document duplicate
records collapse to one global entry so false ambiguity
doesn't reroute cross-doc callers to a stub.
pass 2 — emit nodes for indexed symbols, then walk relationships.
Resolution order:
1. same-doc match (per_doc_index)
2. unique cross-doc match (global_index[symbol] len == 1)
3. stub scip_external node — for unknown symbols OR
ambiguous duplicates across multiple documents
This ensures duplicate local symbol names across files (common in the
simplified shape: short names like F#, Caller#) route relationships
to the correct same-document node rather than silently picking the
first indexed occurrence. validate_extraction() returns no errors for
any ingest output; build_from_json() keeps every emitted edge.
Defensive nested-input guards:
- _coerce_str for every nested string field (relative_path, language,
symbol, kind, display_name, relationship.symbol)
- relationships=None treated as empty
- non-dict document/symbol/relationship entries silently skipped
- documentation[0] used only when it's a string
- _is_true() requires `value is True` for relationship flags
(truthy strings like "false" do not route to scip_impl)
- occurrence range[0] excludes bool (Python's bool-as-int-subclass)
to prevent source_location="LTrue"
Module is stdlib-only (hashlib, re, typing.Any). Not wired to the CLI
in this phase — importable as `from graphify.scip_ingest import
ingest_scip_json`.
Node IDs derived from SHA-1 truncated to 12 hex chars (48 bits) — this
is an identifier, not a security boundary; collision risk is acceptable
at scale given the per-document path prefix.
Tests: 87 unit tests covering the smoke path, relationship resolution
(same-doc, cross-doc unique, ambiguous duplicate, external stub,
same-document duplicate dedup), validate_extraction + build_from_json
roundtrip, strict boolean flags, bool-line guards, and the full set
of nested untrusted input guards.
1044/1044 tests pass.
* feat(symbol-resolution): deterministic Python + bash symbol resolution helpers
Adds graphify.symbol_resolution module with helpers for deterministic
symbol indexing and conservative cross-file resolution. Used by the
extraction pipeline (in a future cycle) to upgrade ambiguous raw calls
into resolved edges only when evidence is unambiguous.
Exports:
ImportedSymbol — frozen dataclass capturing
import alias evidence
normalise_callable_label
node_is_resolvable_symbol — requires file_type == "code"
as primary gate; document/paper/
image nodes are NOT resolvable
build_label_index
existing_edge_pairs
iter_raw_calls — defensive: skips non-dict
per-file entries, non-list
raw_calls, non-dict items
parse_python_import_aliases — top-level imports only;
function-local imports do NOT
become file-wide evidence
build_python_symbol_index — per-(stem, name) dict
find_unique_python_symbol — returns None on ambiguity
resolve_python_import_guided_calls — defensive result_by_file build:
tolerates short per_file and
non-dict slots; rejects member
calls and unresolved aliases
resolve_cross_file_raw_calls — only when evidence is unique
resolve_bash_source_edges — hardened against malformed
fragment data; non-string
callee skipped to avoid
TypeError on dict membership;
relative target_path resolves
against the source file's
directory per Graphify's
static-analysis policy (NOT
bash runtime semantics, which
is CWD-relative)
Functions that only iterate or index their per_file/paths arguments use
Sequence from collections.abc for proper covariance. Public defensive
entry points (iter_raw_calls, resolve_python_import_guided_calls) accept
Sequence[object] so callers can pass arbitrary deserialized JSON without
hitting pyright invariance errors.
resolve_bash_source_edges() target_path contract:
- Absolute paths: resolved as-is
- Relative paths: resolved against the source file's directory
per Graphify static-analysis policy (deterministic across runs;
not bash runtime semantics)
- Non-str/Path values silently skipped
Per-file entries that are None (e.g. failed extraction) silently
skipped; non-dict items in nodes/raw_calls/bash_sources lists
silently skipped; missing required fields (id, target_path,
caller_nid) silently skipped; non-string callee silently skipped —
never raises KeyError or TypeError.
Module is stdlib-only (ast, re, dataclasses, pathlib, typing,
collections.abc). Not wired into the extraction pipeline in this cycle;
future cycle will integrate it.
Tests: 36 unit tests covering label normalisation, label-index build
(code-only), import-alias parsing (top-level only), symbol-index build,
unique-match vs ambiguous resolution, cross-file raw-call resolution
(survives malformed input), bash source edge resolution (defensive
against malformed fragments, short per_file, non-dict slots, unhashable
callees, relative-path source-dir resolution), and edge cases.
* feat(security): cap graph.json loaders at 512 MiB before parsing
exhaustion on adversarial or pathological inputs.
- graphify.security: add _MAX_GRAPH_FILE_BYTES + check_graph_file_size_cap
- graphify.serve._load_graph: call cap after existence check
- graphify.__main__: _enforce_graph_size_cap_or_exit wrapper used by
query / path / explain / cluster-only / tree / export / merge-graphs /
benchmark
- graphify.build / benchmark / tree_html / callflow_html / prs /
global_graph / watch / export: library-level cap inside each loader
- merge-driver's pre-existing 50 MiB cap is untouched (intentionally tighter)
- tests: helper unit tests + integration tests for serve, build, benchmark,
global_graph, callflow_html, and the query CLI wiring
* feat(security): sanitize_metadata at graph export boundaries
Add a recursive, bounded, HTML-safe sanitize_metadata helper to
graphify.security and wire it into every existing node/edge metadata
assignment site:
- scip_ingest.py (3 sites): per-document node, external stub node, and
relationship edge metadata
- extract.py (1 site): bash extractor's add_node metadata
- symbol_resolution.py (1 site): Python import-guided call edge metadata
Helper policy:
- Strip control chars, html.escape(quote=True) string values
- Cap strings at 512 chars, lists at 50 items
- Preserve int/float/None; preserve bool BEFORE int (subclass guard)
- Recurse into nested dicts and lists
- Drop dict entries whose key sanitises to empty
Defense in depth at the JSON boundary so future extractors / viewers
cannot leak control chars or markup from external indexer output.
* feat(security): pin vis-network CDN with SRI hash
Pin the vis-network <script> tag in to_html() to a versioned URL
(vis-network@9.1.6) with a sha384 Subresource Integrity hash and
crossorigin="anonymous". Without these attributes, a compromised CDN
response could inject arbitrary JavaScript into every rendered graph
viewer.
Hash verified live against
https://unpkg.com/vis-network@9.1.6/standalone/umd/vis-network.min.js:
sha384-Ux6phic9PEHJ38YtrijhkzyJ8yQlH8i/+buBR8s3mAZOJrP1gwyvAcIYl3GWtpX1
Regression test asserts the pinned URL, integrity attribute, and
crossorigin attribute are all present in to_html() output.
Follow-up: tree_html.py (D3) and callflow_html.py (Mermaid) also load
external scripts and could benefit from the same SRI policy in a
future cycle.
* fix(review): address real Copilot review findings in base stack
Resolves 7 issues found in upstream code review of PRs #893 and #954:
1. extract.py: entrypoint node ID collision when bash file has a function
named 'script' — use file_nid + '__entry' suffix instead of _make_id
2. extract.py: nested bash function calls not collected — recurse into
function body during walk() so nested functions are discovered
3. extract.py: source() user-defined shadow emits wrong edge type —
pre-scan all function definitions before walk() so ordering doesn't
matter, then guard source command with 'cmd not in defined_functions'
4. extract.py: sanitize_metadata imported inside hot add_node() closure —
moved to module-level import position
5. symbol_resolution.py: _bash_make_id() diverged from extract._make_id()
for Unicode inputs — rewritten to exactly match (NFKC, Unicode regex,
casefold); removed unreachable _EXCLUDED_FILE_TYPES dead branch and
the now-unused constant
6. semantic_cleanup.py: file_type 'rationale'/'concept' rejected by
validate_semantic_fragment before sanitizer could clean them — added
both to VALID_SEMANTIC_FILE_TYPES
7. scip_ingest.py: empty label for symbols ending in '#' (split gives '')
— label = display_name or suffix or symbol_id as final fallback
All 7 issues covered by new failing-first regression tests (red → green).
Full pytest suite: 1239 passed, 4 pre-existing env-specific failures.
* fix(review): address PR #956 Copilot findings in watch.py and symbol_resolution.py
- watch.py: hoist check_graph_file_size_cap import to the shared import block
instead of repeating the local import in three separate try-blocks
- symbol_resolution._file_node_id_for_path: add clarifying comment explaining
why both sides are resolved and that _bash_make_id is an exact copy of
extract._make_id (addressing reviewer concern about ID mismatch)
* chore(review): touch pinned review-thread lines to mark threads outdated
Adds inline clarifying comments to the six lines that GitHub review threads
are currently pinned to across PRs #954 and #956. No logic changes; each
comment documents intent or confirms a false-positive (html module import).
* feat(diagnostics): report multigraph edge-collapse risk
Add graphify.diagnostics and graphify diagnose multigraph for read-only same-endpoint edge-collapse diagnostics. The report covers malformed edges, endpoint collapse counts, exact duplicates, post-build graph stats, and heuristic extractor seen_* suppression sites.
Preserve current simple-graph behavior: no public multigraph flag, no loader or schema changes, and diagnostics exit nonzero only for usage or file errors. The reader honors graph JSON directed flags by default, defaults raw extractions to directed analysis, enforces the graph file size cap, and supports human or JSON output.
* feat(multigraph): add runtime compatibility probe
New module graphify.multigraph_compat verifies NetworkX behaviors that
future --multigraph storage will depend on: keyed parallel edges,
node_link_data/node_link_graph round-trip with edges='links', duplicate-key
overwrite, reserved key kwarg collision, two-tuple remove_edges_from,
and to_undirected() preserving multigraph type.
Behavior probe, not version check. Both NX 3.4.2 (Py 3.10 lane) and
NX 3.6.1+ (Py 3.11+ lane) pass. Result cached for the process lifetime.
No call sites added — this PR adds the API surface only. Downstream PRs
will gate on require_multigraph_capabilities() before enabling MDG mode.
Refs: Wave 1 MultiDiGraph implementation order.
* test: filter known third-party analyze warnings
---------
Co-authored-by: vampyre <vampyre@local.net>
Security: _is_sensitive now flags underscore-prefixed names (api_token.txt, oauth_token.json) by replacing \b with lookarounds; adds _SENSITIVE_DIRS check on parent path components (parts[:-1]) so .ssh/, secrets/, .aws/ directories are always skipped; aligns both patterns to (?![a-zA-Z]) for consistent underscore-after-keyword behavior (#920)
Fix: --wiki Relationships section always empty because _cross_community_links read community from node attrs (always None) instead of the communities dict; _god_node_article had the same bug and never linked to the owning community; fixed by building a node->community map in to_wiki() and threading it through (#925)
Fix: --watch now respects .graphifyignore; patterns loaded once at startup, handler checks _is_ignored before extension filter so node_modules/, .venv/, build/ churn no longer triggers rebuilds (#928)
Fix: C++ struct inheritance edges via base_class_clause; initialize base="" per iteration to prevent stale carryover (#915)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- skill-windows.md: add graphify-out/ prefix to 5 paths in Step B3 that
were being read/written without prefix (lines 244,245,373,387,388,407,413)
causing semantic cache to always appear empty on Windows (#862)
- detect.py: gate converted office/Google Workspace sidecar paths through
_is_ignored() before appending to files list so .graphifyignore rules
apply to graphify-out/converted/ sidecars (#861)
- detect.py: split manifest entries from {mtime,hash} to
{mtime,ast_hash,semantic_hash}; save_manifest() accepts kind="ast"|
"semantic"|"both"; detect_incremental() accepts kind= and checks the
right hash field so graphify update (kind=ast) no longer prevents
graphify extract from re-running semantic extraction (#857)
- watch.py: all three save_manifest() calls use kind="ast" (AST-only rebuild)
- __main__.py: both extract save_manifest() calls use kind="both"
- backwards compatible: legacy {mtime,hash} entries treated as ast_hash only
775 tests passing
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The rebuild lock file accumulated concatenated PIDs across post-commit
rebuilds without a separator, and was never removed when the rebuild
finished. Two practical consequences for users:
1. Downstream tooling that polls for `.rebuild.lock` to disappear before
doing post-rebuild work (publish scripts copying graph.html to a web
root, etc.) blocked forever / until its own timeout.
2. The accumulated digit string could not be parsed by humans or tooling
to find the owning PID.
The `_rebuild_lock` context manager now:
- Opens the lock file with `a+` so a non-acquiring caller does not
truncate the existing holder's PID.
- After flock acquisition, truncates and writes a single `<pid>\n` line
so external readers can `kill -0 $(cat .rebuild.lock)` to check
liveness.
- Unlinks the lock file in the finally block (only when *we* held the
lock), restoring the "signal-by-absence" convention users rely on.
Four regression tests added under `tests/test_watch.py` covering the
PID-with-newline payload, post-release unlink, no-accumulation across
sequential acquisitions, and the non-blocking-caller-does-not-clobber
invariant.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a pre-cluster topology comparison fast path in update rebuilds so unchanged graphs short-circuit before clustering and report generation, preventing residual run-to-run community-count drift.
Co-authored-by: Cursor <cursoragent@cursor.com>
Use safe JSON serialization fallbacks for deterministic sort keys in clustering and graph canonicalization, and skip invalid community IDs with a stderr warning instead of raising during update rebuilds.
Co-authored-by: Cursor <cursoragent@cursor.com>
Make `graphify update` idempotent by skipping output rewrites when graph/report content is unchanged, add `update --no-cluster`, and preserve community IDs across runs via overlap-based remapping with deterministic partition inputs.
Co-authored-by: Cursor <cursoragent@cursor.com>
#791: per-repo fcntl flock in _rebuild_code prevents concurrent hook rebuilds
from exhausting memory; changed_paths wired through so only modified files are
re-extracted; stale nodes evicted on deletion; SIGALRM watchdog with
GRAPHIFY_REBUILD_TIMEOUT; Darwin-aware RLIMIT_DATA memory cap
#792: remove hard 8-worker cap (GRAPHIFY_MAX_WORKERS env var); add
--max-workers, --token-budget, --max-concurrency, --api-timeout CLI flags
to graphify extract; fix ollama API key gate for loopback URLs; explicit
timeout on OpenAI client (GRAPHIFY_API_TIMEOUT, default 600s); per-chunk
progress prints during extraction
#453 + #785: rename .agent -> .agents throughout antigravity install/uninstall;
add trigger:always_on YAML frontmatter to _ANTIGRAVITY_RULES so Antigravity
recognises the rules file
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds support for Quarto markdown (.qmd) files by:
- Adding '.qmd' to document file extensions in detection
- Updating export logic to handle .qmd in filename sanitization
- Adding .qmd extractor dispatch using the existing markdown extractor
- Updating watch comments to include .qmd files
