One test sets a shared, absolute GRAPHIFY_OUT and confirms the marker
written by an update from a relative path is resolved to an absolute
form. A companion test confirms the ordinary, relative GRAPHIFY_OUT
case still preserves the caller supplied relative path exactly as the
existing 777 test already expects, so this fix only changes the
shared output case.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017qfdzgbA5KedGEjD1AayNh
(cherry picked from commit 181a46dca9)
The two test_issue_3695_symlink_worker_* tests monkeypatch _reconcile_existing_graph
to inject a deleted_source_identities collision, then assert the 'fail-closed: kept'
message prints — but on v8 that setup never reaches the corpus-sweep excluded-alive
branch, so the message never fires and the tests fail regardless of the fix. The
production fix stays genuinely guarded by the honest tests (fail_closed_preserved /
full_rebuild_preserves_alive / reextracted / shrink_involving), verified to fail
without the watch.py change and pass with it, plus the real-OS-symlink
test_requires_symlinks_rebuild_symlink_worker.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Simulates the VMware HGFS PermissionError this issue reports by
monkeypatching os.replace to always raise it, then confirms an
incremental no_cluster rebuild still succeeds and lands the new
content instead of aborting the whole run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017qfdzgbA5KedGEjD1AayNh
(cherry picked from commit 9e75593b02)
graphify extract has printed this since #1692, but the update and
watch rebuild path never did, so a corpus with no extractor for its
language, Lean 4 in the reported case, rebuilt successfully with
those files completely absent and nothing said about it. Prints the
same wording from the extract command, and threads the count into
the local detection summary the rebuild path already builds for
GRAPH_REPORT.md, which previously dropped it. Fixes#3511.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017qfdzgbA5KedGEjD1AayNh
(cherry picked from commit 120405673f)
Companion to #3243: extend definition_file portability to the paths build_from_json does not
cover — the watch/incremental relativize + cache-root rebase and the direct extract() return
normalization — via a shared _PORTABLE_PATH_KEYS list so source_file and definition_file stay
in lockstep. Together with #3243 this closes all four persistence paths (graph.json, AST
cache, watch, direct extract).
Follow-up to #3191: an authored markdown link whose target basename is present in the graph
but under a different spelling (e.g. Windows-case [[target]] vs TARGET.md) was dropped on a
code-only rebuild. Preserve such an edge when its target node is still live in the merged
graph, without loosening the deleted-target existence gate (a genuinely deleted target's node
is evicted first, so it is never preserved).
The reconcile relies on the extractor only stamping target_file when the link target exists
on disk. Add a negative test: link text intact but the target file deleted must leave no
reference edge and no edge pointing at the deleted document's node.
A code-only watch/update rebuild never re-creates a semantic-backed document node, so an
authored [[wikilink]] to such a doc resolved to a missing canonical id and reconcile
silently dropped the edge. Re-parse each markdown source and repoint authored references
edges onto a unique representative node for the target file (canonical id, else unique page
node, else unique document node; ambiguous -> keep the existing edge), preserving links to
the semantic tier across a code-only rebuild without resurrecting a deleted target.
On Linux, inotify emits opened / closed_no_write events for every file open and close,
including the watcher's own AST rebuild reading the tree, so the watcher fed itself a
rebuild loop. Drop those two read-only event types; close-after-write and
create/modify/move/delete still trigger, and the filter is a no-op on the macOS/Windows
backends that never emit them.
The label and cluster-only commands did not pass the viz node limit to to_html, so a
graph over the node limit raised and the except branch silently unlinked the graph.html
that update had produced. Always pass the limit (the aggregated community meta-graph
renders instead of raising), preserve the existing file on a failed render via an atomic
publish plus a stale marker, and regenerate a missing graph.html on the no-topology-change
fast path without reclustering.
Two small hardening tweaks on top of #2778: comment that in the hook path
Path.cwd() is the load-bearing anchor rescue (project_root == watch_root ==
the bad marker), and add hyperedges to the _anchors_stored_sources bucket
tuple to match the sibling loop. Adds a deletion-still-evicts test (the anchor
validation must not over-preserve) and an incremental==cold id-parity test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A graph built on a subfolder stores source_file relative to the repo root but
the skill writes .graphify_root as the absolute subfolder, so an incremental
rebuild anchored stored paths under the wrong root, judged every unchanged
source deleted, and collapsed the graph. The anchor is now adopted only if the
stored sources actually resolve under it, falling back to the build root / cwd
otherwise, so a stale subfolder marker can no longer evict the whole corpus.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
graphify watch only rebuilt on a code-file event, so deleting only doc
files while watching flagged needs_update but evicted nothing until the
next code change. A batch containing any vanished path now triggers a full
reconcile rebuild (which evicts the deleted source); a surviving modified
doc still writes the needs-LLM flag. The general deleted-file leak was
already fixed in 0.9.10; this closes the live-watcher residual.
Thanks @angmeng.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#2495 (thanks @alisson-acioli): graphify update now evicts a file newly
added to .graphifyignore/--exclude even though it still exists on disk
(positive ignore-rule evidence), with .gitignore-driven eviction gated on
an explicit full update; unchanged in-corpus files and fail-closed #1795
cases are preserved.
#2504 (thanks @te7ina-honey): the Java type resolver runs before the
unique-label stub rewire and parks imported-but-external types on their
FQN, and cross-file import resolution checks the package, so a local
class and a same-named external annotation (e.g. Spring @Component) no
longer collapse into one node. In-corpus annotation resolution unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#2411 (thanks @JensD-git): key the C# partial-class merge on assembly
(nearest ancestor .csproj/.fsproj/.vbproj) in addition to namespace and
name, so same-name partial classes in different assemblies stay distinct
while genuine partial halves within one project still merge. Fixes a
0.9.32 regression from #2332.
#2437/#2438 (thanks @aryanbonigala, builds on PR #2439): incremental
rebuilds no longer drop member-call and indirect_call edges from a
changed file into an unchanged target. Re-resolution now sees the
unchanged corpus (nodes, contains/method edges, and _callable markers,
which now persist to graph.json like _origin); edges to a genuinely
removed target are still evicted.
#2444/#2445 (thanks @Baziar, builds on PRs #2461/#2458): a
BrokenProcessPool triggers the sequential fallback instead of being
swallowed per future, a failed worker file is retried sequentially
rather than merged as empty, and a whole-pass AST failure on a fresh
build exits non-zero instead of writing a zero-node graph
(--allow-partial opts into a best-effort partial graph).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Node/edge ownership was keyed on source_file alone, but each file has an AST
tier and a semantic tier. A semantic-only re-extract deleted a doc's AST
headings, and a full _rebuild_code deleted document AST nodes. Add an
_is_ast_tier() predicate (with a source_location legacy fallback for the
unreliable _origin marker, #2334), backfill _origin on load so graphs
self-heal, make build_merge / merge_raw_extraction replace by (source_file,
tier) instead of source_file (coexist policy: an AST re-extract replaces only
AST nodes and keeps the semantic layer, and vice versa), and scope the
full-rebuild AST drop to sources actually regenerated (#2336) so a
semantic-backed doc keeps its AST layer. Deletion prune stays tier-blind.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The build_merge fix only covers the incremental path reached by
`graphify <path>`. `graphify update` rebuilds through _rebuild_code in
watch.py and never calls build_merge at all, so it still downgraded a
directed graph — which is the symptom named in the issue.
Two sites: the clustered path called build_from_json(result) with no
directed=, taking its directed=False default, and the --no-cluster path
built candidate_graph_data straight from the raw merged extraction, which
never carries a directed key, so the written file lost it entirely. Both
now inherit from existing_graph_data, defaulting to False when there is no
existing graph, matching build_merge's semantics.
_canonical_graph_for_compare normalizes a missing directed key to False so
an older graph.json without the key does not register as changed against a
candidate that now writes it explicitly.
_reconcile_existing_graph loaded graph.json inside a swallowing try, so a
graph that was merely unreadable (over the size cap or unparseable) was
silently replaced by the code-only extraction, in both the clustered and
--no-cluster hook paths (force made it worse). It now loads through the
fail-closed build._load_existing_graph and _rebuild_code refuses the write
(prints and returns False) on a load failure, matching the CLI path; the
--no-cluster write is now atomic with a protected-graph backup.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Crossing MAX_NODES_FOR_VIZ left the project with no graph.html at all:
_rebuild_code unlinked the existing file and wrote nothing in its place. The
delete on its own is defensible — a kept graph.html would describe an older,
smaller graph — but the file is gone before the user reads the message, and
the next incremental rebuild silently removes it again, so a repo that grows
past the threshold just loses its visualization with no way to keep one.
The export path already solved this (#1019): over the cap it re-renders the
community-aggregation view rather than going without. Do the same here, so
the artifact is current AND present instead of current OR present.
GRAPHIFY_VIZ_NODE_LIMIT=0 still means "no HTML viz" (CI runners) rather than
"aggregate", and if the aggregated render also fails the old skip-and-remove
behaviour stands.
Community labels are saved keyed by community id, but re-clustering
reassigns those ids: after a rebuild that adds nodes, cid 30 can cover a
completely different community and its saved name is then simply wrong.
cluster-only already guards this — it validates each reused label against
the `.graphify_labels.json.sig` membership fingerprints and re-hubs any
community that changed (the case community_member_sigs() was written for).
_rebuild_code() skipped that check entirely: it reused every label whose
cid was present and hub-filled only the *missing* ones, so stale names
survived — and then wrote them back to labels.json, laundering them as
current. It also never refreshed the .sig sidecar, so the signatures kept
describing an older clustering and drifted out of step with the labels
they sit beside, leaving the cluster-only guard nothing accurate to check.
Adding ~3.5k nodes to a real graph re-clustered 463 -> 515 communities and
mislabeled 162 of them this way: a `domain.audit` namespace reading
"ACH / bank payments", a `domain.auth` namespace reading
"document-sensitivity.up.sql". Node and edge data stayed correct, so
nothing failed loudly — only the names lied.
Apply the same signature check in the incremental path, write the sidecar
in step with the labels, and print the same "run `graphify label`" notice
cluster-only emits so a drifted community set is visible rather than
silent.
The #2051 disk-absence sweep guarded remote/virtual sources with a literal
`"://"` check. But the write-side path normalization (Path.as_posix) collapses
the double slash, so a stored `gdoc://x` reads back as `gdoc:/x` on the next
update; the literal guard then missed it and the node fell into the disk-absence
branch (`Path('gdoc:/x').exists()` is False), evicting it on the second
`graphify update`. Match the scheme with a regex tolerant of the collapse, with
a 2+ char scheme so a Windows drive letter (C:/) is not misread as remote.
Regression test runs three consecutive updates and asserts the remote node
survives every one.
Three silent-data-loss fixes in the update/reconcile path:
- #2051: a full `graphify update` now evicts semantic nodes whose non-AST
source (a .txt/.pdf/.png with no code extractor) was deleted from disk.
The corpus sweep only checked re-extractable files, so deleted docs'/images'
LLM nodes survived as authoritative forever. Disk absence is now the deletion
signal; remote/virtual sources (`://`) are left untouched.
- #2056: an incremental rebuild whose change set names a present-but-
unextractable file no longer treats it as a deletion (which evicted its
semantic nodes and disabled the shrink guard). The guard now falls through to
per-source accounting instead of a wholesale bypass on any deletion.
- #2014: code-typed nodes the semantic pass surfaces from within a document now
count as that doc's semantic layer, so a rebuild doesn't re-scan and drop them.
Regression tests for each in tests/test_watch.py.
_rebuild_code fed Markdown/doc files (.md/.mdx/.qmd/.skill) into the AST
quick-scan while _reconcile_existing_graph preserved the same docs'
semantic (LLM) nodes, so every semantic-backed doc was represented twice
(heading nodes + concept nodes) and the graph bloated ~4x vs the CLI
`graphify . --update` path.
Semantic now supersedes AST per doc source: before choosing
extract_targets, read the existing graph's source identities that carry
semantic doc nodes (non-_origin=="ast", gated on file_type=="document"
so pre-#1865 marker-less graphs aren't misread) and exclude those docs
from the quick-scan in both the full-rebuild and changed_paths branches.
They stay in code_files, so the #1795 fail-closed corpus check and the
shrink accounting still cover them — a previously-bloated graph
self-heals on the next full rebuild (the AST ownership rule drops the
stale heading nodes) without the shrink guard refusing the smaller
write. On incremental rebuilds the excluded docs never enter
rebuilt/node-evicted identities, mirroring #1865's tier-scoped edge rule
at the node level, so a doc's semantic nodes and edges survive. Docs
with no semantic layer (and brand-new docs) keep the no-LLM quick-scan
structure from #09b33b7.
#1837 (update silently fails to discover new files) was the same
nested-gitignore scoping bug (#1873/#1880), already fixed. The existing
test only does a single build; add the build -> add new file+dir ->
update -> discovered sequence the report describes, with a nested `*`
scratch dir as a scoping guard. No production change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
--exclude was honored only by the initial extract scan; update/watch/hook
rebuilds re-ran detect() without it and silently re-indexed excluded
paths. extract now writes the patterns to a graphify-out/.graphify_build
.json sidecar, and _rebuild_code reads and re-applies them on every
rebuild (layered after the ignore files, so they still win). Pre-existing
graphs without the sidecar behave as before. Adds a regression test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A full graphify update evicted every edge whose source_file was a
re-extracted source, including LLM semantic edges (e.g.
semantically_similar_to) from Markdown docs that also have an AST
extractor. The AST pass cannot regenerate those edges, so they were
silently lost while their concept nodes survived — node eviction is
provenance-aware via _origin (#1116), edge eviction was not.
Tag AST-extracted edges with the same _origin=ast marker nodes already
carry, and scope rebuild-driven edge eviction to that tier: re-extraction
replaces a source's AST edges, while its semantic edges survive until a
semantic re-extraction supersedes them. Deletion-driven eviction stays
provenance-blind, so edges of deleted or excluded sources are still
purged regardless of tier.
Edges from graphs built before this change lack the marker; a stale
AST edge from a file changed exactly once between the old and new
version can linger until its source is deleted — the same migration
trade-off the #1116 node marker made.
#1880 is the update-layer symptom of the #1873/#1887 nested-ignore
subtree-scoping regression (fixed in fb4d452, @Alwyn93): a nested broad
`.gitignore` zeroed update's re-scan, so it built 0 nodes and the
shrink-guard refused to overwrite. Adds a _rebuild_code regression test
asserting update sees the real files (and still scopes the nested ignore
correctly) so this can't recur at the update layer. Records both
regressions in the unreleased 0.9.16 changelog.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the regression test the #1855 fix was missing: _rebuild_code must
produce graph.json whose clustered nodes carry community_name, guarding
against the label-stripping regression recurring. Also records #1847 and
#1855 in the 0.9.15 changelog.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_reconcile_existing_graph treated "source identity absent from the collected
corpus" as deletion and evicted its nodes/edges/hyperedges. But corpus absence
is ambiguous: it's also what you see when a file still exists and merely stopped
being collected (ignore rules or filters changed). Upgrading into the merged-
.gitignore scan semantics (#1363) mass-evicted 655 nodes from a deliberately-
built, .gitignore'd docs dir whose files were present the whole time — reported
as a successful rebuild.
Fail-closed: before evicting a corpus-absent identity, require Path(identity)
.exists() is False (identity is an absolute path). Alive-but-excluded sources
are preserved (nodes, edges, hyperedges) and a loud line reports how many were
kept and why. True deletions and renames still evict (old path gone from disk);
a full extract --force still purges deliberate exclusions via the AST ownership
rule. Existence is memoized (one stat per file that left the corpus).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`graphify update` (the watch._rebuild_code / _reconcile_existing_graph path)
evicted every hyperedge whose source_file is in the corpus, because on a full
update every corpus file counts as "rebuilt" and hyperedge eviction reused the
node/edge eviction set. But the AST pass never emits hyperedges, so nothing
replaced them — doc-sourced hyperedges (what semantic extraction produces) were
permanently lost on the first update after a full build, even on a no-op run.
Split out a hyperedge_evicted_source_identities set scoped to genuinely deleted
(and symlink-target-outside) sources only, not merely-rebuilt ones. Replacement-
by-id (new_hyperedge_ids) and dangling-member cleanup are unchanged, so a real
semantic re-extraction still replaces its own hyperedges and orphaned ones are
still dropped. Parametrized regression test (full + incremental doc update).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
What changed
- Stabilize relative rebuild execution before graphify-out queue/lock setup.
- Recover from deleted transient hook CWDs when GRAPHIFY_REPO_ROOT points at the repository root.
- Fail cleanly when the current working directory is gone and no repo root fallback is available.
- Add regression coverage for both fallback and clean-failure paths.
Why
- Detached post-commit/post-checkout rebuilds can inherit a transient CWD that is deleted before the background process starts.
- _rebuild_code previously used relative graphify-out paths before Path.cwd()/watch_path resolution could be handled, causing FileNotFoundError: [Errno 2] No such file or directory.
Validation
- .venv/bin/pytest tests/test_watch.py -q: 54 passed, 2 skipped.
- .venv/bin/ruff check graphify/watch.py tests/test_watch.py: passed.
- .venv/bin/python -m py_compile graphify/watch.py tests/test_watch.py: passed.
- env -u PYTHONPATH -u PYTHONHOME PYTHONHASHSEED=0 .venv/bin/pytest -q: 2904 passed, 30 skipped.
Notes
- Full suite without PYTHONHASHSEED=0 exposed an unrelated ordering-sensitive labeling test; with the deterministic hash seed used by graphify hooks, the suite passes.
`graphify update` after deleting a function left the stale node in graph.json.
The build correctly dropped it (#1116), but _check_shrink then refused to write
the smaller graph ("Refusing to overwrite — you may be missing chunk files"),
so the deletion never persisted without --force. That also starved the
work-memory node-existence gate, which relies on graph.json reflecting deletions.
The shrink-guard now takes the set of source files re-extracted this run
(rebuilt_sources). A net shrink is allowed when every lost node belongs to a
rebuilt source (a symbol genuinely removed) or a deleted file; it is still
refused when a node vanishes from a file we did NOT touch — the silent
failed/partial-extraction case the guard exists to catch.
The #1116 e2e test now asserts the prune happens with force=False (was force=True);
added two _check_shrink unit tests (allowed within rebuilt sources, refused
outside). Full suite 2339 passed; skillgen --check clean; ruff clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes#1264
Also fixes a narrow watch-mode path-filtering bug found while running the required local
full-suite gate for this PR. Hidden directories inside the watched corpus are still ignored;
hidden ancestors outside the watched root no longer suppress valid file events.
#1118 — prune stale AST nodes on full re-extraction (#1116)
Stamps every AST-extracted node with _origin="ast" in extract(). On a
full rebuild _rebuild_code drops any AST-marked node absent from the
fresh output even when its source file survives, fixing stale symbols.
Backward-compat: marker-less nodes from pre-1118 graphs survive one
cycle then self-heal.
#1110 — stop reading images and PDFs as garbage in headless extract
Images route through per-backend vision payloads (base64/data-URI/bytes
for claude/openai/bedrock); non-vision backends get _strip_pixels for
graceful degradation. PDFs reuse pypdf. 5MB cap, 20-image chunk limit.
#1159 — Salesforce Apex extractor (.cls, .trigger)
Regex-based extractor: classes, interfaces, enums, methods, triggers,
SOQL/DML edges. No new dependency. Dispatched as .cls and .trigger.
#1107 — Azure OpenAI Service backend (--backend azure)
Uses AzureOpenAI SDK client (from existing openai package). Auto-detects
when AZURE_OPENAI_API_KEY + AZURE_OPENAI_ENDPOINT both set. Uses
max_completion_tokens (not deprecated max_tokens).
#1103 — live PostgreSQL introspection (--postgres DSN)
graphify extract --postgres "postgresql://..." introspects tables, views,
routines, and FK relations via information_schema (SERIALIZABLE READ ONLY).
Credentials sanitized on error. New graphify[postgres] extra (psycopg3).
Union-resolved llm.py conflict: Azure functions + bedrock images= param.
Fixed test_image_vision.py mock to accept timeout= kwarg (our #1112).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ayushcodes10