The rebuild lock file accumulated concatenated PIDs across post-commit
rebuilds without a separator, and was never removed when the rebuild
finished. Two practical consequences for users:
1. Downstream tooling that polls for `.rebuild.lock` to disappear before
doing post-rebuild work (publish scripts copying graph.html to a web
root, etc.) blocked forever / until its own timeout.
2. The accumulated digit string could not be parsed by humans or tooling
to find the owning PID.
The `_rebuild_lock` context manager now:
- Opens the lock file with `a+` so a non-acquiring caller does not
truncate the existing holder's PID.
- After flock acquisition, truncates and writes a single `<pid>\n` line
so external readers can `kill -0 $(cat .rebuild.lock)` to check
liveness.
- Unlinks the lock file in the finally block (only when *we* held the
lock), restoring the "signal-by-absence" convention users rely on.
Four regression tests added under `tests/test_watch.py` covering the
PID-with-newline payload, post-release unlink, no-accumulation across
sequential acquisitions, and the non-blocking-caller-does-not-clobber
invariant.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Astro files have a `---...---` TypeScript frontmatter block at the top
containing nearly all imports, followed by an HTML-with-expressions
template body, and optionally `<script>` blocks for client-side JS.
Feeding the whole file to the JS tree-sitter parser produces a top-level
ERROR node because the template isn't valid JS, so `import_statement`
nodes are never reached.
Mirrors `extract_svelte` (#713) — same regex-rescue strategy:
- Dynamic `import('...')` pass over the full source.
- Static `import X from '...'` pass over the frontmatter block plus any
`<script>` blocks.
Both static and dynamic imports honor tsconfig.json path aliases and
resolve relative paths through `_resolve_js_module_path` so node ids
match the targets that `_extract_generic` produces for sibling files.
Wired in:
- `detect.py:27` — `.astro` added to `CODE_EXTENSIONS`.
- `extract.py:5492` — `".astro": extract_astro` in the dispatch table.
Tests in `tests/test_astro_extraction.py` cover: extension registration,
frontmatter static imports, dynamic import in frontmatter, client-side
`<script>` block imports, frontmatter-less pages (must not crash), and
tsconfig.json path-alias resolution.
Add a pre-cluster topology comparison fast path in update rebuilds so unchanged graphs short-circuit before clustering and report generation, preventing residual run-to-run community-count drift.
Co-authored-by: Cursor <cursoragent@cursor.com>
Make `graphify update` idempotent by skipping output rewrites when graph/report content is unchanged, add `update --no-cluster`, and preserve community IDs across runs via overlap-based remapping with deterministic partition inputs.
Co-authored-by: Cursor <cursoragent@cursor.com>
#796: add edge_data()/edge_datas() helpers in build.py that tolerate
MultiGraph/MultiDiGraph; replace all G.edges[u,v] 2-tuple call sites in
__main__.py, serve.py, wiki.py, export.py, analyze.py, benchmark.py;
fix same pattern in 10 skill file inline heredocs
#795: all 12 skill files now short-circuit on /graphify --help or -h
and print the Usage block without running any pipeline steps
#792 (hollow response): add _response_is_hollow() predicate in llm.py;
when Ollama (or any backend) returns empty/null/whitespace content or a
parsed result with no nodes/edges, rewrite finish_reason="length" so
_extract_with_adaptive_retry bisects the chunk instead of silently
dropping it; applied to _call_openai_compat, _call_claude, _call_bedrock
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
tree-sitter-pascal is not on PyPI so extract_pascal() fell back to an
empty error result for all users. _extract_pascal_regex() now handles
unit/program/library headers, uses clauses, class/interface declarations
with inheritance, forward method decls, qualified impl headers, balanced
begin/end body extraction, and intra-file calls edges. All 15 previously
skipped pascal_required tests now run unconditionally and pass.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(extract): add Pascal/Delphi language support
Adds full AST extraction for Pascal and Delphi source files using
tree-sitter-pascal (https://github.com/Isopod/tree-sitter-pascal).
Supported file extensions: .pas, .pp, .dpr, .dpk, .inc
Extracted nodes:
- File node (the .pas file itself)
- unit / program / library declarations
- class, interface, and helper type declarations
- procedure and function implementations
Extracted edges:
- file --contains--> module
- module --imports--> dependency (via uses clause, resolved to path-based IDs)
- class --inherits--> base class / interface
- class/module --contains/method--> procedure or function
- procedure --calls--> procedure (in-file call resolution)
Key design: uses clause targets are resolved to path-based node IDs by
scanning all Pascal files under the project root (_pascal_project_root +
_pascal_resolve_unit helpers). This avoids dangling import edges that
result from resolving bare unit names like "SysUtils" to IDs that never
match any file node.
Bare procedure calls (e.g. `Reset;` without parentheses) are detected
by inspecting statement nodes whose sole named child is an identifier,
in addition to the standard exprCall nodes used for calls with arguments.
Requires: pip install tree-sitter-pascal
(https://github.com/Isopod/tree-sitter-pascal)
If not installed, extract_pascal returns {"nodes":[], "edges":[], "error": ...}
so the rest of the pipeline is unaffected.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(extract): add Lazarus .lfm and .lpk file support
Adds two new extractors for Lazarus IDE-specific file formats:
extract_lazarus_form() — .lfm (Lazarus Form files)
.lfm files are text-based UI component trees. The extractor parses
`object Name: TClassName ... end` blocks to build a containment graph
of form components, and captures `OnXxx = HandlerName` event bindings
as `references` edges (context: "event") linking each component to
its handler procedure.
extract_lazarus_package() — .lpk (Lazarus Package files)
.lpk files are XML package definitions. The extractor reads the
package name, required package dependencies (→ imports edges), and
listed unit files (→ contains edges). Unit names are resolved to
path-based node IDs via _pascal_resolve_unit so they connect to the
same nodes produced by extract_pascal on .pas files.
Both extensions added to CODE_EXTENSIONS in detect.py and to _DISPATCH.
13 new tests in test_pascal.py cover both extractors.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(pascal): fix dangling inherits-edge targets and capture all base classes
The declType/typeref handler built inherits edge targets with
_make_id(_read(child)) — just the bare class name. But class nodes
use _make_id(stem, type_name), so targets never matched, making the
entire class hierarchy invisible in the graph.
Add _pascal_class_stem_cache and _pascal_resolve_class(): strips the
conventional T/I prefix, locates the defining file by stem lookup
(same cache mechanism as _pascal_resolve_unit), and returns the
correct _make_id(file_stem, class_name) ID. RTL/unresolvable bases
(e.g. TObject) fall back to _make_id(bare_name) with an explicit
stub node, following the same pattern as the Python extractor.
Also remove the `break` that stopped after the first typeref, so
all parents are captured (e.g. class(TBase, IInterface)).
Extend test_pascal_no_dangling_edges to also assert that within-file
edge targets (contains, method, inherits, calls) resolve to real nodes.
* feat(extract): add Delphi .dfm form file support
Adds extract_delphi_form() for Delphi Form files (.dfm), which use the
same `object Name: TClassName ... end` text syntax as Lazarus .lfm files.
Binary .dfm files (FF 0A magic header) are skipped gracefully with an
informative error message so the pipeline is unaffected. Text .dfm files
are parsed identically to .lfm: component containment (`contains` edges)
and event handler references (`references`, context "event").
Adds .dfm to _DISPATCH and CODE_EXTENSIONS.
10 new tests in test_pascal.py, including a regression test for the
binary-format detection.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Add .lpr extension and fix removesuffix in Pascal extractor
Address review feedback from @safishamsi:
- Add .lpr (Lazarus program file, identical syntax to .dpr) to _DISPATCH
in extract.py and CODE_EXTENSIONS in detect.py so Lazarus project entry
points are indexed. Completes the promised Lazarus IDE support.
- Replace rstrip("()") with removesuffix("()") in the call-resolution
dict comprehension for precise suffix removal (rstrip strips individual
characters, not the literal string "()").
- Add .lpr assertions to test_pascal_dispatch_registered and
test_pascal_detect_extensions_registered.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Simeon Bodurov <simeon.bodurov@speedy.bg>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Three independent Windows compatibility fixes shipped together because they
all surface during the same first /graphify run on Windows.
graphify/benchmark.py
print_benchmark() unconditionally printed U+2500 (box-drawing) and U+2192
(rightwards arrow), which UnicodeEncodeError'd on stdouts that can't encode
them — most notably the legacy Windows console at cp1252. New _safe()
helper falls back to ASCII when the active stdout encoding can't carry the
glyph; _hr() uses it. Two regression tests cover both paths and prove
print_benchmark survives a cp1252-strict stream.
graphify/extract.py
ProcessPoolExecutor on Windows uses spawn, so worker subprocesses
re-import the calling __main__. When the caller is `python -c "..."` or a
script without an `if __name__ == "__main__":` guard, the workers
recursively spawn themselves and the pool dies. The user-visible failure
was a 290-line traceback ending in BrokenProcessPool, hiding the actual
cause. _extract_parallel now catches BrokenProcessPool, prints a one-line
warning that names the __main__-guard idiom, and returns False so the
public extract() routes to the existing _extract_sequential fallback. Two
tests cover the parallel-returns-False contract and the sequential
fallback wiring.
graphify/skill-windows.md
Every `python -c "..."` block (30 in total) is replaced with a
Write+run+delete pattern using PowerShell's literal here-string @'...'@.
The old form was a quote-escaping minefield: any double-quote inside the
Python source had to be backslash-escaped for the shell, and PowerShell's
parser ate them inconsistently — failing on f-strings like
`f'AST: {len(result["nodes"])} nodes'`. The new form passes Python source
to disk literally, so what the model writes is what Python sees. The AST
step's script template now includes an explicit `if __name__ == "__main__":`
guard so multi-core extraction works even before the runtime fallback above
kicks in. All 31 resulting heredoc blocks parse cleanly under
`ast.parse`.
Co-authored-by: Nauman Hameed <Nauman.Hameed@enghouse.com>
Adaptive retry only recovered from `finish_reason="length"` (output
truncation). It did not handle the other shape of overflow: the API
rejecting the prompt outright with a 400 because the input plus
`max_completion_tokens` doesn't fit in the model's context window.
This shows up immediately on local OpenAI-compatible servers (LM
Studio, llama.cpp, vLLM) where the default context is small (4K-32K)
and a 60K-token chunk packed for cloud Kimi/Claude blows past it.
Without retry the whole chunk fails with no output, even though the
two halves would each fit cleanly.
Catch a heuristic set of context-overflow exception messages,
classify them as the same kind of recoverable failure as
`finish_reason="length"`, and split-recurse on the same path. Single-
file overflow returns an empty fragment so the rest of the corpus
keeps running. Unrelated errors (rate limit, auth, etc.) still
propagate.
Tested with qwen3.5-9b on LM Studio (32K ctx) against a 215-file
corpus where chunks 4-12 of 12 previously failed; with this change
the overflowing chunks self-heal by splitting in half.
The SQL parser (`extract_sql`) previously only extracted foreign key
relationships defined inline within CREATE TABLE column definitions.
FK constraints added via ALTER TABLE ... ADD CONSTRAINT ... FOREIGN KEY
... REFERENCES were silently ignored.
Additionally, `_obj_name()` only read the first identifier child of
object_reference nodes, so schema-qualified names like `Sales.Customer`
were truncated to just `Sales`.
Changes:
- Add `alter_table` handler to `walk()` that extracts FK edges from
ALTER TABLE ... ADD CONSTRAINT ... FOREIGN KEY ... REFERENCES
- Fix `_obj_name()` to read the full object_reference text, preserving
schema-qualified names (e.g. `Sales.Customer`)
- Fix inline FK resolution in create_table and _walk_from_refs to use
full object_reference text instead of first identifier only
tree-sitter-typescript ships two grammars:
- language_typescript: pure TypeScript, no JSX support
- language_tsx: JSX-aware variant for .tsx files
Currently both .ts and .tsx are parsed with language_typescript, which
treats JSX syntax as parse errors. Every function declaration, arrow
function, and call_expression nested inside a JSX tree is silently
dropped from the extracted graph.
Repro on a representative React+TypeScript codebase (a 13-file Tauri app):
parsing each .tsx with language_typescript produces ~276 ERROR nodes per
file. Only declarations that happen to live before the first JSX block
survive.
Fix: add _TSX_CONFIG that mirrors _TS_CONFIG but selects language_tsx,
and route .tsx files to it in extract_js().
Effect on the same repo (graphify update --force):
Nodes: 303 → 618 (+104%)
Edges: 482 → 779 (+62%)
Communities: 28 → 45 (+61%)
Parse errors 276 → 0 per .tsx file
Tests added:
- tsx fixture with helpers + JSX-returning component
- helpers and component are captured
- JSX expression calls ({fmtDate(now)}) resolve to call edges
- wiring check: .tsx uses language_tsx, .ts uses language_typescript
Note: this fixes the parsing layer. Calls inside deeply nested arrow
function callbacks (e.g. items.map(x => <T>{f(x)}</T>)) are still
missed by the call extraction logic — separate enhancement.
Co-authored-by: Serkan Gezici <serkan@quadroaipilot.com>
The cross-file call resolver in `extract()` unconditionally marked every
resolved call edge as INFERRED with confidence_score 0.8 — even when the
caller's file had an explicit `imports` (symbol) or `imports_from`
(module) edge to the callee. The new CJS require handler made this gap
visible: imports were correctly EXTRACTED but the call edges that those
imports backed remained INFERRED, so downstream consumers couldn't tell
high-evidence calls apart from name-match guesses.
This pass runs after the file-id remap (line 4736), so we relativize
node `source_file` paths before computing file_nids — otherwise the
caller's computed file_nid (absolute-path-derived) wouldn't match the
imports_from edge source (already remapped to relative form).
Promotion rule:
- Symbol-level `imports` edge from caller's file -> callee node id
=> EXTRACTED, confidence_score 1.0
- Module-level `imports_from` edge from caller's file -> callee's file
=> EXTRACTED, confidence_score 1.0
- Otherwise => INFERRED, confidence_score 0.8 (existing behavior)
Validated on a 92-file CJS orchestrator: 5 previously-INFERRED edges
from runExecute() now resolve to EXTRACTED, and 88% of cross-file calls
in the corpus (104 of 118) promote, leaving INFERRED only for genuine
heuristic guesses with no import backing.
Adds two tests:
- test_cross_file_call_promoted_to_extracted_with_import_evidence
- test_cross_file_call_remains_inferred_without_import_evidence
The JS/TS extractor only handled ES `import` statements; CommonJS
`require()` calls produced no import edges. Downstream, the call-graph
pass could not resolve which symbols belonged to which file, so every
cross-file call in CJS Node.js codebases was downgraded to INFERRED
even when the binding was a top-of-file destructured require.
Adds three patterns to `_js_extra_walk` via a new `_require_imports_js`
helper:
const { foo, bar: alias } = require('./mod') -> imports_from + per-symbol imports
const mod = require('./mod') -> imports_from
const x = require('./mod').y -> imports_from + symbol edge for y
Refactors path-resolution out of `_import_js` into
`_resolve_js_import_target` so ES imports and CJS requires share the
relative / tsconfig-alias / bare-module logic.
Tested in a 92-file CJS Node.js orchestrator codebase: confirmed all
five previously INFERRED `runExecute -> {loadFoundation,
validateDispatchConfig, fetchSymphonyIssues, listSymphonyWorktrees,
workspacePathForIssue}` edges resolve to real top-of-file destructured
requires, so downstream calls would now be EXTRACTED instead of
INFERRED.
Add `.luau` to CODE_EXTENSIONS and route it through extract_lua
(tree-sitter-lua). Roblox first-party code uses .luau; without this,
graphify silently skipped 379/479 files on a real Roblox codebase
and the resulting graph was dominated by vendored .lua dependencies.
tree-sitter-lua doesn't parse Luau type annotations, but it
successfully extracts function declarations and call edges from
Luau source — verified on a 379-file Roblox codebase (1265 nodes,
1471 edges, 236 communities).
A dedicated tree-sitter-luau grammar would be a richer long-term
fix; this is the minimal change to make Luau projects work today.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The initial Gemini backend defaulted to 2.5 Flash, but large semantic extraction chunks can benefit from newer models and more output headroom. Move the default to Gemini 3 Flash Preview, add CLI and environment model overrides, and increase the Gemini completion budget while keeping low reasoning effort for cost control.
Constraint: Google exposes Gemini through an OpenAI-compatible chat-completions endpoint
Rejected: Hardcode Gemini 3.1 Pro as the default | higher cost for routine repository indexing
Confidence: medium
Scope-risk: narrow
Directive: Keep --model and GRAPHIFY_GEMINI_MODEL working before changing Gemini defaults again
Tested: uv run --directory vendor/graphify pytest tests/test_llm_backends.py tests/test_chunking.py -q
Not-tested: Live Gemini 3 extraction on the full cloud-edge repo before this commit
`detect_incremental(root)` always called `detect(root)` without forwarding
the `follow_symlinks` kwarg. As a result, corpora that include symlinked
sub-trees pointing to directories outside the scan root (e.g. a
`state_of_truth/` symlink pointing at `~/.hermes/state_of_truth/`) were
visible to a full `detect()` run with `follow_symlinks=True` but invisible
to any subsequent `--update` run. The incremental scan would then either
report no changes (silently dropping legitimate new files) or repeatedly
re-extract a phantom subset, depending on what was reachable without
crossing symlinks.
Add a keyword-only `follow_symlinks` parameter to `detect_incremental()`
and forward it. Default stays `False` for backwards compatibility — only
callers that already opt in to symlink following on `detect()` pick up
the new behaviour for incremental runs too.
Test: a corpus with a symlinked directory is invisible with
`follow_symlinks=False`, fully indexed with `follow_symlinks=True`, and
correctly reports zero new files on a second incremental scan after the
manifest is saved.
Gemini is often the cheaper available quota for low-stakes semantic graph extraction, while OpenAI is a useful fallback. Extend the direct extraction backend registry, CLI validation, docs, and tests so headless extraction can use GEMINI_API_KEY, GOOGLE_API_KEY, or OPENAI_API_KEY without changing the existing Claude and Kimi paths.
Constraint: Gemini supports OpenAI-compatible chat completions at the Google generative-language endpoint
Rejected: Native google-genai integration | higher dependency and response-shape churn for the same chat-completions path
Confidence: medium
Scope-risk: moderate
Directive: Keep backend detection explicit and test every accepted API-key environment variable before adding new providers
Tested: uv run --directory vendor/graphify pytest tests/test_llm_backends.py tests/test_chunking.py -q
Not-tested: Live Gemini/OpenAI API calls; no GEMINI_API_KEY or OPENAI_API_KEY present in this environment
Safi