Add `.luau` to CODE_EXTENSIONS and route it through extract_lua
(tree-sitter-lua). Roblox first-party code uses .luau; without this,
graphify silently skipped 379/479 files on a real Roblox codebase
and the resulting graph was dominated by vendored .lua dependencies.
tree-sitter-lua doesn't parse Luau type annotations, but it
successfully extracts function declarations and call edges from
Luau source — verified on a 379-file Roblox codebase (1265 nodes,
1471 edges, 236 communities).
A dedicated tree-sitter-luau grammar would be a richer long-term
fix; this is the minimal change to make Luau projects work today.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The initial Gemini backend defaulted to 2.5 Flash, but large semantic extraction chunks can benefit from newer models and more output headroom. Move the default to Gemini 3 Flash Preview, add CLI and environment model overrides, and increase the Gemini completion budget while keeping low reasoning effort for cost control.
Constraint: Google exposes Gemini through an OpenAI-compatible chat-completions endpoint
Rejected: Hardcode Gemini 3.1 Pro as the default | higher cost for routine repository indexing
Confidence: medium
Scope-risk: narrow
Directive: Keep --model and GRAPHIFY_GEMINI_MODEL working before changing Gemini defaults again
Tested: uv run --directory vendor/graphify pytest tests/test_llm_backends.py tests/test_chunking.py -q
Not-tested: Live Gemini 3 extraction on the full cloud-edge repo before this commit
`detect_incremental(root)` always called `detect(root)` without forwarding
the `follow_symlinks` kwarg. As a result, corpora that include symlinked
sub-trees pointing to directories outside the scan root (e.g. a
`state_of_truth/` symlink pointing at `~/.hermes/state_of_truth/`) were
visible to a full `detect()` run with `follow_symlinks=True` but invisible
to any subsequent `--update` run. The incremental scan would then either
report no changes (silently dropping legitimate new files) or repeatedly
re-extract a phantom subset, depending on what was reachable without
crossing symlinks.
Add a keyword-only `follow_symlinks` parameter to `detect_incremental()`
and forward it. Default stays `False` for backwards compatibility — only
callers that already opt in to symlink following on `detect()` pick up
the new behaviour for incremental runs too.
Test: a corpus with a symlinked directory is invisible with
`follow_symlinks=False`, fully indexed with `follow_symlinks=True`, and
correctly reports zero new files on a second incremental scan after the
manifest is saved.
Gemini is often the cheaper available quota for low-stakes semantic graph extraction, while OpenAI is a useful fallback. Extend the direct extraction backend registry, CLI validation, docs, and tests so headless extraction can use GEMINI_API_KEY, GOOGLE_API_KEY, or OPENAI_API_KEY without changing the existing Claude and Kimi paths.
Constraint: Gemini supports OpenAI-compatible chat completions at the Google generative-language endpoint
Rejected: Native google-genai integration | higher dependency and response-shape churn for the same chat-completions path
Confidence: medium
Scope-risk: moderate
Directive: Keep backend detection explicit and test every accepted API-key environment variable before adding new providers
Tested: uv run --directory vendor/graphify pytest tests/test_llm_backends.py tests/test_chunking.py -q
Not-tested: Live Gemini/OpenAI API calls; no GEMINI_API_KEY or OPENAI_API_KEY present in this environment
- Register .groovy and .gradle in CODE_EXTENSIONS, _DISPATCH, and collect_files
- Add _GROOVY_CONFIG (reuses Java import handler)
- Add regex-based _extract_spock_fallback for Spock spec files where
tree-sitter-groovy wraps the body in ERROR nodes due to def-string methods
- _is_spock_file detects via regex scan (def "...") instead of node-label
heuristic, avoiding false negatives on classes whose name differs from stem
- Fallback retains only file node + import edges from tree-sitter pass to
prevent orphaned constructor/method nodes
- Add tree-sitter-groovy>=0.1.2 dependency
- Add 11 tests covering plain Groovy and Spock paths, including apostrophe
in feature method names
Third call site that re-implemented the same .js→.ts rewrite in
isolation. Previously only handled the explicit .js→.ts case; bare
paths, multi-dot helper files, and alias-resolved dynamic imports
all dropped silently.
Now uses _resolve_js_module_path on both branches (relative and
alias) — same shape as the static-import and Svelte regex paths.
Real-world impact: TS files using `await import('./foo')` patterns
for code splitting (e.g. lazy-loading a profanity check) now produce
edges to the resolved target.
When both a file (foo.ts) and a directory (foo/) exist at the same path,
both TypeScript and Vite prefer the file. The previous ordering checked
directory first and fell through unchanged when the directory had no
index, silently dropping every import like 'from ./auth' when an
auth/ subdirectory existed alongside auth.ts.
The generalized resolver already handles .svelte.js because the append
loop iterates _JS_RESOLVE_EXTS = (.ts, .tsx, .svelte, .js, .jsx, .mjs).
Adds three explicit tests to pin the behaviour and document the priority
choice:
- test_resolve_svelte_to_svelte_js_for_javascript_rune_files
JS-only Svelte 5 project: .svelte → .svelte.js works the same
way as .svelte.ts in TS projects. No special-casing needed —
the generalized append loop covers both.
- test_resolve_svelte_prefers_svelte_ts_over_svelte_js
Hybrid case (both files exist, e.g. .svelte.ts source plus
.svelte.js build artifact): .ts wins. Documents the deliberate
source-first priority — graphify is a source-code tool, not a
runtime resolver, so we differ from Vite's default JS-first order.
- test_resolve_real_svelte_file_wins_over_svelte_ts_sibling
Existence check short-circuits before any extension append, so a
real .svelte file always wins over a .svelte.ts sibling.
Two changes that landed together because they share the same code path:
1. Generalize the bare-path append to handle multi-dot filenames
The previous resolver only appended extensions when path.suffix == ""
(truly bare paths). Real codebases use a lot of multi-dot patterns:
foo.shared.ts ← imported as './foo.shared'
foo.config.ts ← imported as './foo.config'
foo.compile.ts ← imported as './foo.compile'
foo.integration.ts ← imported as './foo.integration' (test helper)
foo.triggers.ts ← imported as './foo.triggers' (test helper)
foo.svelte.ts ← imported as './foo.svelte' (Svelte 5 rune)
foo.d.ts ← imported as './foo.d' (ambient types)
For all of these, .suffix is the meaningful middle segment (.shared,
.config, .integration, etc.) — not in the .js/.jsx/.svelte handled
list, so the resolver fell through and the import dropped to a phantom.
The fix unifies the bare-path and .svelte→.svelte.ts cases into a
single rule: append each candidate extension to the FULL filename, not
to the stripped stem. This subsumes:
bare path: foo → foo.ts
Svelte rune file: foo.svelte → foo.svelte.ts
multi-dot helper: foo.shared → foo.shared.ts
ambient declaration: foo.d → foo.d.ts
No behaviour change for paths that DO exist (.is_file() short-circuit)
or for the .js→.ts / .jsx→.tsx convention (handled before the append
loop so we don't accidentally match foo.js → foo.js.ts when foo.ts
is the real file).
2. Rename _resolve_with_extensions → _resolve_js_module_path
The function is JS/TS/Svelte-specific (Vite resolver order, mirrors the
convention used by _import_js, _JS_CONFIG, _TS_CONFIG). The original
name suggested it was a generic path utility. Renamed to make scope
explicit and align with the existing _import_js / _JS_CONFIG naming
pattern. Constants renamed to match: _JS_RESOLVE_EXTS, _JS_INDEX_FILES.
Tests
-----
4 new tests in tests/test_import_extension_resolution.py:
- test_resolve_multi_dot_helper_file: foo.shared → foo.shared.ts
- test_resolve_multi_dot_with_explicit_extension_still_works:
foo.shared.ts (explicit) still wins
- test_resolve_ambient_d_ts_via_bare_path: foo.d → foo.d.ts
- test_end_to_end_multi_dot_import_resolves: tree-sitter pipeline
sanity check via extract_js
Existing 28 tests updated for the rename. 32/32 pass; 7 pre-existing
unrelated failures elsewhere in the suite.
Validation
----------
On a 1,873-file SvelteKit codebase, applying both rules over the v0.7.5
baseline:
baseline: 12,096 edges
with the resolver fix: 20,151 edges (+8,055 = +67%)
The +2,652 over the previous version of this branch is attributable
entirely to multi-dot filename recovery, primarily test helper imports
('*.integration.ts', '*.triggers.ts'), domain-shared modules
('*.shared.ts'), and config files.
Adds 8 tests covering import shapes that came up during real-codebase
validation against a 1,873-file SvelteKit project:
- test_type_only_import_with_bare_path_resolves
`import type { X } from './foo'` — type-only imports must go
through the same resolver. Common pattern in TS codebases.
- test_named_imports_emit_symbol_edges_after_resolution
`import { foo, bar } from './module'` — verifies the per-symbol
`imports` edges (file → module.foo, file → module.bar) target the
correct stem after resolution. The symbol target_stem comes from
_file_stem(resolved), so resolution must happen first.
- test_alias_directory_import_resolves_to_index_ts
`from '$lib/queue'` — alias + directory composes correctly.
- test_resolve_does_not_match_partial_directory_name
Regression guard: `from './foo'` where only `foo-extra.ts` exists
must NOT accidentally resolve to it.
- test_resolve_directory_without_index_returns_unchanged
A directory with no index.* must fall through, not pick a random
.ts inside.
- test_resolve_handles_subpath_into_directory_with_index
`./foo/sub` where `./foo/sub/index.ts` exists.
- test_resolve_does_not_treat_dotfile_as_extension
Path('.env-types.ts').suffix is '.ts' (correct), but worth pinning.
- test_resolve_chain_alias_and_extension_compose
Two-layer resolution: alias → bare path → .svelte.ts. Verifies
the full chain works end-to-end for the Svelte 5 rune-file case.
Also expanded test_named_imports_emit_symbol_edges_after_resolution to
catch a subtle regression class: per-symbol import edges (line 319-340
in _import_js) build their target id from _file_stem(resolved). If
resolution fails or returns the wrong path, the symbol edges silently
target a different stem and downstream "where is X used?" queries miss
real callers.
_import_js previously only rewrote .js→.ts and .jsx→.tsx, leaving every
other common TypeScript / SvelteKit / Vite import shape unresolved. The
resulting node id wouldn't match the target file's own _make_id, so
build_from_json dropped the edge as external.
Three missed shapes:
1. Bare paths (no extension) — TS convention:
`import { foo } from './foo'` → real file is foo.ts
2. .svelte → .svelte.ts (Svelte 5 rune-only files):
`import { x } from './x.svelte'` → real file is x.svelte.ts
3. Directory imports / barrel index files:
`import { x } from './queue'` → real file is queue/index.ts
Fix
---
New helper _resolve_with_extensions(p: Path) -> Path mirrors Vite/TS
resolver order:
1. exact path (file)
2. .js→.ts, .jsx→.tsx (existing TS-ESM convention)
3. bare path → .ts/.tsx/.svelte/.js/.jsx/.mjs
4. bare path → directory's index.{ts,tsx,js,jsx}
5. .svelte → .svelte.ts (Svelte 5 rune file)
Falls back to the original path on no match — preserves pre-fix behaviour
for genuinely external modules (build_from_json drops them as phantoms).
Wired into _import_js (relative + alias branches) and extract_svelte's
regex pass for dynamic_import so static and dynamic imports both benefit.
Subtle: uses .is_file() / .is_dir() rather than .exists(). When the
import is a directory, .exists() returns True and would short-circuit
before the index.ts lookup ever ran.
Tests
-----
20 new tests in tests/test_import_extension_resolution.py:
Resolver unit tests (12):
- existing path returned unchanged
- bare path → .ts / .tsx / .svelte
- .ts wins over .svelte for ambiguous bare paths (Vite order)
- directory → index.ts
- directory prefers index.ts over index.js
- .svelte → .svelte.ts (Svelte 5 rune file)
- .js → .ts (TS ESM convention)
- .jsx → .tsx
- real .js stays .js when .ts doesn't exist
- unresolvable returns input unchanged
End-to-end (8):
- bare-path import resolves in TS file
- directory import resolves to index.ts
- .svelte import resolves to .svelte.ts rune file
- explicit .ts/.svelte imports still work (regression guard)
- external module specifiers unchanged
- alias + bare path resolves
- dynamic_import bare path resolves
1. NEW: extract_markdown() — structurally indexes .md/.mdx files into the
knowledge graph. Headings become nodes, code blocks become nodes with
language tags, and nesting produces 'contains' edges. Zero new deps
(pure regex/line-by-line parsing, no tree-sitter needed).
2. FIX: collect_files() _EXTENSIONS was hardcoded and missing 18 extensions
that _DISPATCH already supported (.jsx, .mjs, .ex, .exs, .jl, .vue,
.svelte, .dart, .v, .sv, .sql, .f, .F, .f90, etc). Now uses
set(_DISPATCH.keys()) to stay automatically in sync.
3. Added deploy_guide.md test fixture and 6 new test cases.
4. Updated test_collect_files_from_dir to use dynamic extension set.
#700: _read_tsconfig_aliases() now handles // comments, /* */ block comments,
and trailing commas via a regex-based _strip_jsonc() helper. Tries plain
json.loads first, falls back to stripped parse, warns to stderr on failure
instead of silently returning {}.
#701 Bug A: removed startswith('.') filter from Svelte regex fallback so
aliased imports ($lib/, $partials/, @/) are no longer skipped.
#701 Bug B: fixed synthetic node IDs — source uses _make_id(str(path)),
target uses _make_id(str(normpath(parent/raw))) matching _extract_generic
and _import_js conventions so build_from_json no longer drops every edge.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Chris Stephens