Add `.luau` to CODE_EXTENSIONS and route it through extract_lua
(tree-sitter-lua). Roblox first-party code uses .luau; without this,
graphify silently skipped 379/479 files on a real Roblox codebase
and the resulting graph was dominated by vendored .lua dependencies.
tree-sitter-lua doesn't parse Luau type annotations, but it
successfully extracts function declarations and call edges from
Luau source — verified on a 379-file Roblox codebase (1265 nodes,
1471 edges, 236 communities).
A dedicated tree-sitter-luau grammar would be a richer long-term
fix; this is the minimal change to make Luau projects work today.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The initial Gemini backend defaulted to 2.5 Flash, but large semantic extraction chunks can benefit from newer models and more output headroom. Move the default to Gemini 3 Flash Preview, add CLI and environment model overrides, and increase the Gemini completion budget while keeping low reasoning effort for cost control.
Constraint: Google exposes Gemini through an OpenAI-compatible chat-completions endpoint
Rejected: Hardcode Gemini 3.1 Pro as the default | higher cost for routine repository indexing
Confidence: medium
Scope-risk: narrow
Directive: Keep --model and GRAPHIFY_GEMINI_MODEL working before changing Gemini defaults again
Tested: uv run --directory vendor/graphify pytest tests/test_llm_backends.py tests/test_chunking.py -q
Not-tested: Live Gemini 3 extraction on the full cloud-edge repo before this commit
`detect_incremental(root)` always called `detect(root)` without forwarding
the `follow_symlinks` kwarg. As a result, corpora that include symlinked
sub-trees pointing to directories outside the scan root (e.g. a
`state_of_truth/` symlink pointing at `~/.hermes/state_of_truth/`) were
visible to a full `detect()` run with `follow_symlinks=True` but invisible
to any subsequent `--update` run. The incremental scan would then either
report no changes (silently dropping legitimate new files) or repeatedly
re-extract a phantom subset, depending on what was reachable without
crossing symlinks.
Add a keyword-only `follow_symlinks` parameter to `detect_incremental()`
and forward it. Default stays `False` for backwards compatibility — only
callers that already opt in to symlink following on `detect()` pick up
the new behaviour for incremental runs too.
Test: a corpus with a symlinked directory is invisible with
`follow_symlinks=False`, fully indexed with `follow_symlinks=True`, and
correctly reports zero new files on a second incremental scan after the
manifest is saved.
Gemini is often the cheaper available quota for low-stakes semantic graph extraction, while OpenAI is a useful fallback. Extend the direct extraction backend registry, CLI validation, docs, and tests so headless extraction can use GEMINI_API_KEY, GOOGLE_API_KEY, or OPENAI_API_KEY without changing the existing Claude and Kimi paths.
Constraint: Gemini supports OpenAI-compatible chat completions at the Google generative-language endpoint
Rejected: Native google-genai integration | higher dependency and response-shape churn for the same chat-completions path
Confidence: medium
Scope-risk: moderate
Directive: Keep backend detection explicit and test every accepted API-key environment variable before adding new providers
Tested: uv run --directory vendor/graphify pytest tests/test_llm_backends.py tests/test_chunking.py -q
Not-tested: Live Gemini/OpenAI API calls; no GEMINI_API_KEY or OPENAI_API_KEY present in this environment
- Register .groovy and .gradle in CODE_EXTENSIONS, _DISPATCH, and collect_files
- Add _GROOVY_CONFIG (reuses Java import handler)
- Add regex-based _extract_spock_fallback for Spock spec files where
tree-sitter-groovy wraps the body in ERROR nodes due to def-string methods
- _is_spock_file detects via regex scan (def "...") instead of node-label
heuristic, avoiding false negatives on classes whose name differs from stem
- Fallback retains only file node + import edges from tree-sitter pass to
prevent orphaned constructor/method nodes
- Add tree-sitter-groovy>=0.1.2 dependency
- Add 11 tests covering plain Groovy and Spock paths, including apostrophe
in feature method names
Third call site that re-implemented the same .js→.ts rewrite in
isolation. Previously only handled the explicit .js→.ts case; bare
paths, multi-dot helper files, and alias-resolved dynamic imports
all dropped silently.
Now uses _resolve_js_module_path on both branches (relative and
alias) — same shape as the static-import and Svelte regex paths.
Real-world impact: TS files using `await import('./foo')` patterns
for code splitting (e.g. lazy-loading a profanity check) now produce
edges to the resolved target.
When both a file (foo.ts) and a directory (foo/) exist at the same path,
both TypeScript and Vite prefer the file. The previous ordering checked
directory first and fell through unchanged when the directory had no
index, silently dropping every import like 'from ./auth' when an
auth/ subdirectory existed alongside auth.ts.
The generalized resolver already handles .svelte.js because the append
loop iterates _JS_RESOLVE_EXTS = (.ts, .tsx, .svelte, .js, .jsx, .mjs).
Adds three explicit tests to pin the behaviour and document the priority
choice:
- test_resolve_svelte_to_svelte_js_for_javascript_rune_files
JS-only Svelte 5 project: .svelte → .svelte.js works the same
way as .svelte.ts in TS projects. No special-casing needed —
the generalized append loop covers both.
- test_resolve_svelte_prefers_svelte_ts_over_svelte_js
Hybrid case (both files exist, e.g. .svelte.ts source plus
.svelte.js build artifact): .ts wins. Documents the deliberate
source-first priority — graphify is a source-code tool, not a
runtime resolver, so we differ from Vite's default JS-first order.
- test_resolve_real_svelte_file_wins_over_svelte_ts_sibling
Existence check short-circuits before any extension append, so a
real .svelte file always wins over a .svelte.ts sibling.
Two changes that landed together because they share the same code path:
1. Generalize the bare-path append to handle multi-dot filenames
The previous resolver only appended extensions when path.suffix == ""
(truly bare paths). Real codebases use a lot of multi-dot patterns:
foo.shared.ts ← imported as './foo.shared'
foo.config.ts ← imported as './foo.config'
foo.compile.ts ← imported as './foo.compile'
foo.integration.ts ← imported as './foo.integration' (test helper)
foo.triggers.ts ← imported as './foo.triggers' (test helper)
foo.svelte.ts ← imported as './foo.svelte' (Svelte 5 rune)
foo.d.ts ← imported as './foo.d' (ambient types)
For all of these, .suffix is the meaningful middle segment (.shared,
.config, .integration, etc.) — not in the .js/.jsx/.svelte handled
list, so the resolver fell through and the import dropped to a phantom.
The fix unifies the bare-path and .svelte→.svelte.ts cases into a
single rule: append each candidate extension to the FULL filename, not
to the stripped stem. This subsumes:
bare path: foo → foo.ts
Svelte rune file: foo.svelte → foo.svelte.ts
multi-dot helper: foo.shared → foo.shared.ts
ambient declaration: foo.d → foo.d.ts
No behaviour change for paths that DO exist (.is_file() short-circuit)
or for the .js→.ts / .jsx→.tsx convention (handled before the append
loop so we don't accidentally match foo.js → foo.js.ts when foo.ts
is the real file).
2. Rename _resolve_with_extensions → _resolve_js_module_path
The function is JS/TS/Svelte-specific (Vite resolver order, mirrors the
convention used by _import_js, _JS_CONFIG, _TS_CONFIG). The original
name suggested it was a generic path utility. Renamed to make scope
explicit and align with the existing _import_js / _JS_CONFIG naming
pattern. Constants renamed to match: _JS_RESOLVE_EXTS, _JS_INDEX_FILES.
Tests
-----
4 new tests in tests/test_import_extension_resolution.py:
- test_resolve_multi_dot_helper_file: foo.shared → foo.shared.ts
- test_resolve_multi_dot_with_explicit_extension_still_works:
foo.shared.ts (explicit) still wins
- test_resolve_ambient_d_ts_via_bare_path: foo.d → foo.d.ts
- test_end_to_end_multi_dot_import_resolves: tree-sitter pipeline
sanity check via extract_js
Existing 28 tests updated for the rename. 32/32 pass; 7 pre-existing
unrelated failures elsewhere in the suite.
Validation
----------
On a 1,873-file SvelteKit codebase, applying both rules over the v0.7.5
baseline:
baseline: 12,096 edges
with the resolver fix: 20,151 edges (+8,055 = +67%)
The +2,652 over the previous version of this branch is attributable
entirely to multi-dot filename recovery, primarily test helper imports
('*.integration.ts', '*.triggers.ts'), domain-shared modules
('*.shared.ts'), and config files.
Adds 8 tests covering import shapes that came up during real-codebase
validation against a 1,873-file SvelteKit project:
- test_type_only_import_with_bare_path_resolves
`import type { X } from './foo'` — type-only imports must go
through the same resolver. Common pattern in TS codebases.
- test_named_imports_emit_symbol_edges_after_resolution
`import { foo, bar } from './module'` — verifies the per-symbol
`imports` edges (file → module.foo, file → module.bar) target the
correct stem after resolution. The symbol target_stem comes from
_file_stem(resolved), so resolution must happen first.
- test_alias_directory_import_resolves_to_index_ts
`from '$lib/queue'` — alias + directory composes correctly.
- test_resolve_does_not_match_partial_directory_name
Regression guard: `from './foo'` where only `foo-extra.ts` exists
must NOT accidentally resolve to it.
- test_resolve_directory_without_index_returns_unchanged
A directory with no index.* must fall through, not pick a random
.ts inside.
- test_resolve_handles_subpath_into_directory_with_index
`./foo/sub` where `./foo/sub/index.ts` exists.
- test_resolve_does_not_treat_dotfile_as_extension
Path('.env-types.ts').suffix is '.ts' (correct), but worth pinning.
- test_resolve_chain_alias_and_extension_compose
Two-layer resolution: alias → bare path → .svelte.ts. Verifies
the full chain works end-to-end for the Svelte 5 rune-file case.
Also expanded test_named_imports_emit_symbol_edges_after_resolution to
catch a subtle regression class: per-symbol import edges (line 319-340
in _import_js) build their target id from _file_stem(resolved). If
resolution fails or returns the wrong path, the symbol edges silently
target a different stem and downstream "where is X used?" queries miss
real callers.
_import_js previously only rewrote .js→.ts and .jsx→.tsx, leaving every
other common TypeScript / SvelteKit / Vite import shape unresolved. The
resulting node id wouldn't match the target file's own _make_id, so
build_from_json dropped the edge as external.
Three missed shapes:
1. Bare paths (no extension) — TS convention:
`import { foo } from './foo'` → real file is foo.ts
2. .svelte → .svelte.ts (Svelte 5 rune-only files):
`import { x } from './x.svelte'` → real file is x.svelte.ts
3. Directory imports / barrel index files:
`import { x } from './queue'` → real file is queue/index.ts
Fix
---
New helper _resolve_with_extensions(p: Path) -> Path mirrors Vite/TS
resolver order:
1. exact path (file)
2. .js→.ts, .jsx→.tsx (existing TS-ESM convention)
3. bare path → .ts/.tsx/.svelte/.js/.jsx/.mjs
4. bare path → directory's index.{ts,tsx,js,jsx}
5. .svelte → .svelte.ts (Svelte 5 rune file)
Falls back to the original path on no match — preserves pre-fix behaviour
for genuinely external modules (build_from_json drops them as phantoms).
Wired into _import_js (relative + alias branches) and extract_svelte's
regex pass for dynamic_import so static and dynamic imports both benefit.
Subtle: uses .is_file() / .is_dir() rather than .exists(). When the
import is a directory, .exists() returns True and would short-circuit
before the index.ts lookup ever ran.
Tests
-----
20 new tests in tests/test_import_extension_resolution.py:
Resolver unit tests (12):
- existing path returned unchanged
- bare path → .ts / .tsx / .svelte
- .ts wins over .svelte for ambiguous bare paths (Vite order)
- directory → index.ts
- directory prefers index.ts over index.js
- .svelte → .svelte.ts (Svelte 5 rune file)
- .js → .ts (TS ESM convention)
- .jsx → .tsx
- real .js stays .js when .ts doesn't exist
- unresolvable returns input unchanged
End-to-end (8):
- bare-path import resolves in TS file
- directory import resolves to index.ts
- .svelte import resolves to .svelte.ts rune file
- explicit .ts/.svelte imports still work (regression guard)
- external module specifiers unchanged
- alias + bare path resolves
- dynamic_import bare path resolves
1. NEW: extract_markdown() — structurally indexes .md/.mdx files into the
knowledge graph. Headings become nodes, code blocks become nodes with
language tags, and nesting produces 'contains' edges. Zero new deps
(pure regex/line-by-line parsing, no tree-sitter needed).
2. FIX: collect_files() _EXTENSIONS was hardcoded and missing 18 extensions
that _DISPATCH already supported (.jsx, .mjs, .ex, .exs, .jl, .vue,
.svelte, .dart, .v, .sv, .sql, .f, .F, .f90, etc). Now uses
set(_DISPATCH.keys()) to stay automatically in sync.
3. Added deploy_guide.md test fixture and 6 new test cases.
4. Updated test_collect_files_from_dir to use dynamic extension set.
- Add Fortran support (26th language): .f/.F/.f90/.F90/.f95/.F95/.f03/.F03/.f08/.F08
via tree-sitter-fortran; capital-F files preprocessed with cpp -w -P
- Add graphify export {html,obsidian,wiki,svg,graphml,neo4j} CLI subcommands
- Add graphify query/path/explain CLI subcommands
- Reduce skill.md from 63KB to 47KB by replacing Python heredocs with CLI calls
- Extend to_html() with node_limit param for auto-aggregation on large graphs
- Add integration tests for all export/query/path/explain subcommands
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Codex Desktop rejects hookSpecificOutput.additionalContext on PreToolUse.
hook-check is now a no-op — graph guidance reaches the agent via AGENTS.md/skill.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
`extract_kotlin` previously emitted zero `calls` edges (and zero
`raw_calls` entries) on the current PyPI grammar. The Kotlin branch
of `walk_calls` only matched node type `simple_identifier`, but
PyPI's `tree_sitter_kotlin` produces `identifier` for the equivalent
plain-identifier node. The `simple_identifier` ↔ `identifier` rename
is a generation gap between tree-sitter-kotlin grammar versions —
older forks (and the JVM `io.github.bonede:tree-sitter-kotlin`
binding) still use `simple_identifier`.
Accept both names so the extractor works across grammar generations.
Also widens `_KOTLIN_CONFIG.name_fallback_child_types` for the same
reason (defensive — currently the `name` field path covers
class/function name resolution, but if that field is dropped in a
future grammar update the fallback would face the same rename).
Tested against `tests/fixtures/sample.kt`: edges go from 6
(file-contains + class-method only) to 10 (adds 4 in-file `calls`
edges resolved by the walker:
- .get() → .buildRequest() @ L8
- .post() → .buildRequest() @ L12
- createClient() → Config @ L21
- createClient() → HttpClient @ L22).
A new regression test `test_kotlin_emits_in_file_calls` asserts the
four edges so this exact bug can't recur.
Found via graphify-kmp (Kotlin Multiplatform port of graphify) —
its `PythonParityTest` flagged 4 KMP-only edges that Python missed.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit adds full VB.NET language support to graphify, raising the
supported language count from 25 to 26. The implementation follows the
established LanguageConfig pattern used by all other tree-sitter-backed
extractors.
New dependency:
- Adds optional extra [vbnet] backed by tree-sitter-vbnet (published
to PyPI at https://pypi.org/project/tree-sitter-vbnet/0.1.0/).
Install with: pip install graphifyy[vbnet]
graphify/detect.py:
- Added .vb to CODE_EXTENSIONS so VB.NET files are discovered during
corpus ingestion and file-system watching.
graphify/extract.py:
- _import_vbnet(): import handler for imports_statement nodes; emits
imports edges using the namespace_name child text.
- _vbnet_extra_walk(): extra-walk hook that intercepts namespace_block
nodes, emits a namespace node, and recurses.
- _VBNET_CONFIG: full LanguageConfig covering class_block / module_block /
structure_block / interface_block as class types; method_declaration /
constructor_declaration / property_declaration as function types;
invocation call nodes with target/member_access fields.
- VB.NET-specific branches in _extract_generic:
* Class body: VB.NET has no wrapper body node; inherits and implements
are named fields directly on the class_block. Emits separate inherits
and implements edges for each base type, stripping generic arguments.
* Constructor name: constructor_declaration carries no name field in
the grammar; always resolves to New.
* Function body: uses the declaration node itself as body sentinel so
the call-graph pass can find invocations inside methods.
- extract_vbnet(path): public wrapper that delegates to _extract_generic.
- _DISPATCH['.vb']: routes .vb files to extract_vbnet.
pyproject.toml:
- Added vbnet = ['tree-sitter-vbnet'] optional dependency group.
- Added 'tree-sitter-vbnet' to the all extra.
tests/fixtures/sample.vb:
- New fixture file exercising: Imports statements, Namespace block,
Interface, Class with Inherits + Implements, Module, Structure,
Sub/Function/Property methods, and method calls.
tests/test_languages.py:
- Added 13 tests covering: no-error, class/interface/module/structure
detection, method detection, imports relation, inherits edge,
implements edge, and no-dangling-edges invariant.
README.md:
- Updated language count 25 to 26.
- Added VB.NET to language list and file-extension table.
- extract_sql(): deterministic tree-sitter extraction of tables, views,
functions, foreign key references, and FROM/JOIN reads_from edges
- .sql added to CODE_EXTENSIONS and dispatch table
- tree-sitter-sql added as optional dep under [sql] extra
- xlsx_extract_structure(): extracts sheet/table/column nodes from .xlsx
(utility — pipeline wiring in follow-up)
- 6 new SQL tests, 447 total passing
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Token-budget chunking cuts the truncation rate but doesn't eliminate
it. Output token cost scales with extractable concept density rather
than input tokens — a chunk that lands on a directory of dense design
docs can pack under the input budget while needing more than
`max_completion_tokens=8192` to express every named concept, so the
response is truncated mid-string and `_parse_llm_json` returns an
empty fragment.
Pre-tuning chunk size to be conservative enough that this never
happens leaves throughput on the table for the common case. Adding a
hard `max_files_per_chunk` cap on top of `token_budget` reintroduces
the "tune a static constant" problem the previous commit set out to
fix.
The fix uses the API's own truncation signal:
1. `_call_openai_compat` and `_call_claude` now expose `finish_reason`
on the result dict (Anthropic's `stop_reason == "max_tokens"` is
normalised to `"length"`).
2. `_extract_with_adaptive_retry` checks it: when truncated, splits
the chunk in half and recurses on each half. Recursion is bounded
by `max_retry_depth` (default 3 → at most 8x fanout per top-level
chunk).
3. Single-file chunks that truncate can't recover and surface a
warning rather than infinite-loop.
4. `extract_corpus_parallel` routes every chunk through the retry
wrapper. The `on_chunk_done` callback fires once per top-level
chunk with the merged result — recursive splits are invisible to
callers.
Three independent improvements to extract_corpus_parallel:
1. Token-aware chunking. Replaces `chunk_size=20` static packing with
a greedy packer keyed on `token_budget` (default 60_000), grouped
by parent directory so related artefacts share a chunk. Pass
`token_budget=None` to fall back to fixed-count packing.
2. Optional tiktoken (added to the [kimi] extra). When available,
`_estimate_file_tokens` uses cl100k_base for accurate counts;
without it, the existing chars/4 heuristic kicks in. Kimi-K2 ships
a tiktoken-based tokenizer so estimates against Moonshot are very
close to truth.
3. True parallelism. The function name said "parallel" but the body
was a sequential for-loop. Now uses ThreadPoolExecutor capped at
`max_concurrency` (default 4 — conservative against provider rate
limits). `on_chunk_done(idx, total, result)` still fires once per
chunk with the original submission idx so progress UIs work
unchanged. `max_concurrency=1` skips the pool to preserve
sequential semantics.
Plus failure tolerance: a chunk raising is now caught, logged to
stderr, and the run continues. Other chunks' results merge as normal.
On a 162-file repo (~125k words), the same work that took ~36 min
sequential under the old code finishes in ~7 min.
Grep and Glob tools removed in CC v2.1.117; searches now go through Bash.
Hook now reads stdin tool_input and pattern-matches on search commands.
Uninstall/reinstall handles both old and new matcher for clean upgrades.
Closes#578
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
import(`./handlers/${name}`) previously produced a garbage edge to a
path containing the unresolved ${name} expression. Now detects
template_substitution child nodes and breaks without emitting an edge.
Static template literals (no interpolation) still resolve correctly.
Adds 2 new tests: one asserting dynamic templates produce no edge,
one asserting static templates resolve like plain strings.
Adds _dynamic_import_js() helper (65 lines) that detects import() call
expressions in JS/TS, resolves the module path (same logic as static
imports including .js→.ts mapping and tsconfig aliases), and emits
imports_from edges from the enclosing function. Hooked into walk_calls
for JS/TS configs.
Also adds tests/fixtures/dynamic_import.ts fixture and 5 new tests
in tests/test_languages.py (all passing alongside 110 existing tests).
Safi