#1837 (update silently fails to discover new files) was the same
nested-gitignore scoping bug (#1873/#1880), already fixed. The existing
test only does a single build; add the build -> add new file+dir ->
update -> discovered sequence the report describes, with a nested `*`
scratch dir as a scoping guard. No production change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
--exclude was honored only by the initial extract scan; update/watch/hook
rebuilds re-ran detect() without it and silently re-indexed excluded
paths. extract now writes the patterns to a graphify-out/.graphify_build
.json sidecar, and _rebuild_code reads and re-applies them on every
rebuild (layered after the ignore files, so they still win). Pre-existing
graphs without the sidecar behave as before. Adds a regression test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A full graphify update evicted every edge whose source_file was a
re-extracted source, including LLM semantic edges (e.g.
semantically_similar_to) from Markdown docs that also have an AST
extractor. The AST pass cannot regenerate those edges, so they were
silently lost while their concept nodes survived — node eviction is
provenance-aware via _origin (#1116), edge eviction was not.
Tag AST-extracted edges with the same _origin=ast marker nodes already
carry, and scope rebuild-driven edge eviction to that tier: re-extraction
replaces a source's AST edges, while its semantic edges survive until a
semantic re-extraction supersedes them. Deletion-driven eviction stays
provenance-blind, so edges of deleted or excluded sources are still
purged regardless of tier.
Edges from graphs built before this change lack the marker; a stale
AST edge from a file changed exactly once between the old and new
version can linger until its source is deleted — the same migration
trade-off the #1116 node marker made.
#1880 is the update-layer symptom of the #1873/#1887 nested-ignore
subtree-scoping regression (fixed in fb4d452, @Alwyn93): a nested broad
`.gitignore` zeroed update's re-scan, so it built 0 nodes and the
shrink-guard refused to overwrite. Adds a _rebuild_code regression test
asserting update sees the real files (and still scopes the nested ignore
correctly) so this can't recur at the update layer. Records both
regressions in the unreleased 0.9.16 changelog.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the regression test the #1855 fix was missing: _rebuild_code must
produce graph.json whose clustered nodes carry community_name, guarding
against the label-stripping regression recurring. Also records #1847 and
#1855 in the 0.9.15 changelog.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_reconcile_existing_graph treated "source identity absent from the collected
corpus" as deletion and evicted its nodes/edges/hyperedges. But corpus absence
is ambiguous: it's also what you see when a file still exists and merely stopped
being collected (ignore rules or filters changed). Upgrading into the merged-
.gitignore scan semantics (#1363) mass-evicted 655 nodes from a deliberately-
built, .gitignore'd docs dir whose files were present the whole time — reported
as a successful rebuild.
Fail-closed: before evicting a corpus-absent identity, require Path(identity)
.exists() is False (identity is an absolute path). Alive-but-excluded sources
are preserved (nodes, edges, hyperedges) and a loud line reports how many were
kept and why. True deletions and renames still evict (old path gone from disk);
a full extract --force still purges deliberate exclusions via the AST ownership
rule. Existence is memoized (one stat per file that left the corpus).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`graphify update` (the watch._rebuild_code / _reconcile_existing_graph path)
evicted every hyperedge whose source_file is in the corpus, because on a full
update every corpus file counts as "rebuilt" and hyperedge eviction reused the
node/edge eviction set. But the AST pass never emits hyperedges, so nothing
replaced them — doc-sourced hyperedges (what semantic extraction produces) were
permanently lost on the first update after a full build, even on a no-op run.
Split out a hyperedge_evicted_source_identities set scoped to genuinely deleted
(and symlink-target-outside) sources only, not merely-rebuilt ones. Replacement-
by-id (new_hyperedge_ids) and dangling-member cleanup are unchanged, so a real
semantic re-extraction still replaces its own hyperedges and orphaned ones are
still dropped. Parametrized regression test (full + incremental doc update).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
What changed
- Stabilize relative rebuild execution before graphify-out queue/lock setup.
- Recover from deleted transient hook CWDs when GRAPHIFY_REPO_ROOT points at the repository root.
- Fail cleanly when the current working directory is gone and no repo root fallback is available.
- Add regression coverage for both fallback and clean-failure paths.
Why
- Detached post-commit/post-checkout rebuilds can inherit a transient CWD that is deleted before the background process starts.
- _rebuild_code previously used relative graphify-out paths before Path.cwd()/watch_path resolution could be handled, causing FileNotFoundError: [Errno 2] No such file or directory.
Validation
- .venv/bin/pytest tests/test_watch.py -q: 54 passed, 2 skipped.
- .venv/bin/ruff check graphify/watch.py tests/test_watch.py: passed.
- .venv/bin/python -m py_compile graphify/watch.py tests/test_watch.py: passed.
- env -u PYTHONPATH -u PYTHONHOME PYTHONHASHSEED=0 .venv/bin/pytest -q: 2904 passed, 30 skipped.
Notes
- Full suite without PYTHONHASHSEED=0 exposed an unrelated ordering-sensitive labeling test; with the deterministic hash seed used by graphify hooks, the suite passes.
`graphify update` after deleting a function left the stale node in graph.json.
The build correctly dropped it (#1116), but _check_shrink then refused to write
the smaller graph ("Refusing to overwrite — you may be missing chunk files"),
so the deletion never persisted without --force. That also starved the
work-memory node-existence gate, which relies on graph.json reflecting deletions.
The shrink-guard now takes the set of source files re-extracted this run
(rebuilt_sources). A net shrink is allowed when every lost node belongs to a
rebuilt source (a symbol genuinely removed) or a deleted file; it is still
refused when a node vanishes from a file we did NOT touch — the silent
failed/partial-extraction case the guard exists to catch.
The #1116 e2e test now asserts the prune happens with force=False (was force=True);
added two _check_shrink unit tests (allowed within rebuilt sources, refused
outside). Full suite 2339 passed; skillgen --check clean; ruff clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes#1264
Also fixes a narrow watch-mode path-filtering bug found while running the required local
full-suite gate for this PR. Hidden directories inside the watched corpus are still ignored;
hidden ancestors outside the watched root no longer suppress valid file events.
#1118 — prune stale AST nodes on full re-extraction (#1116)
Stamps every AST-extracted node with _origin="ast" in extract(). On a
full rebuild _rebuild_code drops any AST-marked node absent from the
fresh output even when its source file survives, fixing stale symbols.
Backward-compat: marker-less nodes from pre-1118 graphs survive one
cycle then self-heal.
#1110 — stop reading images and PDFs as garbage in headless extract
Images route through per-backend vision payloads (base64/data-URI/bytes
for claude/openai/bedrock); non-vision backends get _strip_pixels for
graceful degradation. PDFs reuse pypdf. 5MB cap, 20-image chunk limit.
#1159 — Salesforce Apex extractor (.cls, .trigger)
Regex-based extractor: classes, interfaces, enums, methods, triggers,
SOQL/DML edges. No new dependency. Dispatched as .cls and .trigger.
#1107 — Azure OpenAI Service backend (--backend azure)
Uses AzureOpenAI SDK client (from existing openai package). Auto-detects
when AZURE_OPENAI_API_KEY + AZURE_OPENAI_ENDPOINT both set. Uses
max_completion_tokens (not deprecated max_tokens).
#1103 — live PostgreSQL introspection (--postgres DSN)
graphify extract --postgres "postgresql://..." introspects tables, views,
routines, and FK relations via information_schema (SERIALIZABLE READ ONLY).
Credentials sanitized on error. New graphify[postgres] extra (psycopg3).
Union-resolved llm.py conflict: Azure functions + bedrock images= param.
Fixed test_image_vision.py mock to accept timeout= kwarg (our #1112).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fixes#777. Relativize manifest keys, .graphify_root, and cache source_file fields on persist; re-anchor on load. In-memory callers still see absolute paths. Symlink round-trip fixed in follow-up commit 8f09326.
The post-commit hook passes `git diff --name-only HEAD~1 HEAD` as
changed_paths to `_rebuild_code`. That list includes deletions, and
`_rebuild_code` correctly identifies them (lines 352-367 in watch.py)
and evicts the stale nodes from the preserved set. The rebuilt graph
is intentionally smaller.
`_check_shrink` then refuses to overwrite the existing graph.json
because it only sees the node-count delta, not the cause. The guard
fires with "Refusing to overwrite — you may be missing chunk files
from a previous session. Pass --force to override."
Result: every commit that deletes a tracked file silently leaves stale
nodes in graph.json. The user must either pass --force (which also
disables the guard for legitimate failure modes) or manually re-run
`graphify update . --force` after delete-heavy commits.
Fix: thread a `had_explicit_deletions` flag from `_rebuild_code` into
`_check_shrink`. When the caller has declared the deletions, the
smaller graph is the expected outcome and the guard is skipped. The
guard remains intact for SILENT shrinkage — its actual purpose —
from failed semantic chunks or corrupted runs.
The fix is opt-in by design: callers that don't pass `changed_paths`
(e.g. the post-checkout full rebuild path) keep the old conservative
behavior. Only paths that explicitly track deletions get the bypass.
Tests added (tests/test_watch.py):
- `test_check_shrink_blocks_silent_shrink` — pre-existing behavior intact
- `test_check_shrink_allows_force_override` — pre-existing behavior intact
- `test_check_shrink_allows_explicit_deletions` — new: deletion bypass
- `test_check_shrink_allows_no_existing_data` — first-run case
- `test_check_shrink_allows_growth` — sanity
- `test_check_shrink_unlinks_tmp_on_refuse` — cleanup on refusal
- `test_check_shrink_keeps_tmp_when_deletions_declared` — no spurious unlink
- `test_rebuild_code_prunes_deleted_file_nodes` — end-to-end probe of the
exact scenario the post-commit hook triggers (git init, build, delete
one file, re-run with the deleted path in changed_paths, verify the
graph shrinks and the surviving file's nodes are preserved)
All 8 new tests pass; full test_watch.py + test_build.py + test_export.py
(74 tests) pass with no regressions.
Security: _is_sensitive now flags underscore-prefixed names (api_token.txt, oauth_token.json) by replacing \b with lookarounds; adds _SENSITIVE_DIRS check on parent path components (parts[:-1]) so .ssh/, secrets/, .aws/ directories are always skipped; aligns both patterns to (?![a-zA-Z]) for consistent underscore-after-keyword behavior (#920)
Fix: --wiki Relationships section always empty because _cross_community_links read community from node attrs (always None) instead of the communities dict; _god_node_article had the same bug and never linked to the owning community; fixed by building a node->community map in to_wiki() and threading it through (#925)
Fix: --watch now respects .graphifyignore; patterns loaded once at startup, handler checks _is_ignored before extension filter so node_modules/, .venv/, build/ churn no longer triggers rebuilds (#928)
Fix: C++ struct inheritance edges via base_class_clause; initialize base="" per iteration to prevent stale carryover (#915)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add .sh, .bash, .json to CODE_EXTENSIONS in detect.py so files reach extractor
- Fix test_detect_incremental manifest path collision with new .json extension
- Update test_watch to reflect .json/.sh are now watched extensions
- B-1: only emit source imports for paths that exist on disk
- J-1: replace stat()+read() with bounded read to eliminate TOCTOU
- J-3: move pair_count cap inside loop so it is honoured exactly
- J-4: namespace $ref/extends refs with "ref_" prefix to prevent ID collision
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The rebuild lock file accumulated concatenated PIDs across post-commit
rebuilds without a separator, and was never removed when the rebuild
finished. Two practical consequences for users:
1. Downstream tooling that polls for `.rebuild.lock` to disappear before
doing post-rebuild work (publish scripts copying graph.html to a web
root, etc.) blocked forever / until its own timeout.
2. The accumulated digit string could not be parsed by humans or tooling
to find the owning PID.
The `_rebuild_lock` context manager now:
- Opens the lock file with `a+` so a non-acquiring caller does not
truncate the existing holder's PID.
- After flock acquisition, truncates and writes a single `<pid>\n` line
so external readers can `kill -0 $(cat .rebuild.lock)` to check
liveness.
- Unlinks the lock file in the finally block (only when *we* held the
lock), restoring the "signal-by-absence" convention users rely on.
Four regression tests added under `tests/test_watch.py` covering the
PID-with-newline payload, post-release unlink, no-accumulation across
sequential acquisitions, and the non-blocking-caller-does-not-clobber
invariant.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a pre-cluster topology comparison fast path in update rebuilds so unchanged graphs short-circuit before clustering and report generation, preventing residual run-to-run community-count drift.
Co-authored-by: Cursor <cursoragent@cursor.com>
Make `graphify update` idempotent by skipping output rewrites when graph/report content is unchanged, add `update --no-cluster`, and preserve community IDs across runs via overlap-based remapping with deterministic partition inputs.
Co-authored-by: Cursor <cursoragent@cursor.com>
style: replace all em dashes with hyphens
fix: explain hidden .graphify/ folder in skill output and README
fix: rename .graphify/ to graphify-out/ so output is visible by default
safishamsi