73 Commits

Author SHA1 Message Date
Safi 6df69dce6c Fix #683: normalize source_file path separators + two-phase cohesion re-clustering
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-03 17:26:13 +01:00
Safi 893acb19df Fix #664: filter thin communities from GRAPH_REPORT.md by default
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-02 18:09:32 +01:00
Safi cf7ce3a450 fix #651: hook-check exits silently instead of emitting unsupported additionalContext
Codex Desktop rejects hookSpecificOutput.additionalContext on PreToolUse.
hook-check is now a no-op — graph guidance reaches the agent via AGENTS.md/skill.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-02 17:29:40 +01:00
Safi ba41258635 merge PR #663: parallel AST extraction with ProcessPoolExecutor (1.66x speedup on 84 files)
Co-Authored-By: hanzala-sohrab <hanzala-sohrab@users.noreply.github.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-02 17:24:57 +01:00
Safi 1e848458f9 remove VB.NET support: tree-sitter-vbnet only has Windows wheels, no Linux/macOS
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-02 17:02:24 +01:00
Safi 2ca3a5994b merge PR #579: dynamic import() extraction for JS/TS
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-02 16:52:19 +01:00
Safi 28047b502f merge PR #573: cross-language edge context filters in MCP query tool
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-02 16:49:22 +01:00
Safi 24808ef8e6 merge PR #625: token-aware chunking with split-and-retry on truncation
Co-Authored-By: Jason Matthew <jasonm4130@gmail.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-02 16:46:37 +01:00
Safi 7237cd3290 feat: add VB.NET (.vb) language support via tree-sitter (#648) 2026-05-02 16:41:55 +01:00
Hanzala Sohrab d04e487877 Merge branch 'safishamsi:v6' into feat/parallel-ast-extraction 2026-05-02 21:11:02 +05:30
Safi 996e82b3d9 test: add regression test for ambiguous cross-file call resolution (#632) 2026-05-02 16:39:53 +01:00
Michal Harakal 71423a1efb Kotlin call-walker: accept both simple_identifier and identifier (#659)
`extract_kotlin` previously emitted zero `calls` edges (and zero
`raw_calls` entries) on the current PyPI grammar. The Kotlin branch
of `walk_calls` only matched node type `simple_identifier`, but
PyPI's `tree_sitter_kotlin` produces `identifier` for the equivalent
plain-identifier node. The `simple_identifier` ↔ `identifier` rename
is a generation gap between tree-sitter-kotlin grammar versions —
older forks (and the JVM `io.github.bonede:tree-sitter-kotlin`
binding) still use `simple_identifier`.

Accept both names so the extractor works across grammar generations.
Also widens `_KOTLIN_CONFIG.name_fallback_child_types` for the same
reason (defensive — currently the `name` field path covers
class/function name resolution, but if that field is dropped in a
future grammar update the fallback would face the same rename).

Tested against `tests/fixtures/sample.kt`: edges go from 6
(file-contains + class-method only) to 10 (adds 4 in-file `calls`
edges resolved by the walker:
  - .get() → .buildRequest() @ L8
  - .post() → .buildRequest() @ L12
  - createClient() → Config @ L21
  - createClient() → HttpClient @ L22).

A new regression test `test_kotlin_emits_in_file_calls` asserts the
four edges so this exact bug can't recur.

Found via graphify-kmp (Kotlin Multiplatform port of graphify) —
its `PythonParityTest` flagged 4 KMP-only edges that Python missed.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 13:49:42 +01:00
Hanzala Sohrab 0fc2dc6ad3 feat: implement parallel AST extraction using ProcessPoolExecutor with benchmarking support 2026-05-02 12:54:08 +05:30
Rangarajan Ramaswamy 5dbbcf7dad feat: add VB.NET (.vb) language support via tree-sitter
This commit adds full VB.NET language support to graphify, raising the
supported language count from 25 to 26. The implementation follows the
established LanguageConfig pattern used by all other tree-sitter-backed
extractors.

New dependency:
- Adds optional extra [vbnet] backed by tree-sitter-vbnet (published
  to PyPI at https://pypi.org/project/tree-sitter-vbnet/0.1.0/).
  Install with: pip install graphifyy[vbnet]

graphify/detect.py:
- Added .vb to CODE_EXTENSIONS so VB.NET files are discovered during
  corpus ingestion and file-system watching.

graphify/extract.py:
- _import_vbnet(): import handler for imports_statement nodes; emits
  imports edges using the namespace_name child text.
- _vbnet_extra_walk(): extra-walk hook that intercepts namespace_block
  nodes, emits a namespace node, and recurses.
- _VBNET_CONFIG: full LanguageConfig covering class_block / module_block /
  structure_block / interface_block as class types; method_declaration /
  constructor_declaration / property_declaration as function types;
  invocation call nodes with target/member_access fields.
- VB.NET-specific branches in _extract_generic:
  * Class body: VB.NET has no wrapper body node; inherits and implements
    are named fields directly on the class_block. Emits separate inherits
    and implements edges for each base type, stripping generic arguments.
  * Constructor name: constructor_declaration carries no name field in
    the grammar; always resolves to New.
  * Function body: uses the declaration node itself as body sentinel so
    the call-graph pass can find invocations inside methods.
- extract_vbnet(path): public wrapper that delegates to _extract_generic.
- _DISPATCH['.vb']: routes .vb files to extract_vbnet.

pyproject.toml:
- Added vbnet = ['tree-sitter-vbnet'] optional dependency group.
- Added 'tree-sitter-vbnet' to the all extra.

tests/fixtures/sample.vb:
- New fixture file exercising: Imports statements, Namespace block,
  Interface, Class with Inherits + Implements, Module, Structure,
  Sub/Function/Property methods, and method calls.

tests/test_languages.py:
- Added 13 tests covering: no-error, class/interface/module/structure
  detection, method detection, imports relation, inherits edge,
  implements edge, and no-dangling-edges invariant.

README.md:
- Updated language count 25 to 26.
- Added VB.NET to language list and file-extension table.
2026-05-01 21:20:36 +03:00
Safi 7f336acfd9 fix .graphifyignore: correct gitignore semantics + hermetic non-VCS scan + skill auto-invoke
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-01 18:45:12 +01:00
Safi 7d604e8141 v6: SQL AST extractor + xlsx structural extraction utility (fixes #349)
- extract_sql(): deterministic tree-sitter extraction of tables, views,
  functions, foreign key references, and FROM/JOIN reads_from edges
- .sql added to CODE_EXTENSIONS and dispatch table
- tree-sitter-sql added as optional dep under [sql] extra
- xlsx_extract_structure(): extracts sheet/table/column nodes from .xlsx
  (utility — pipeline wiring in follow-up)
- 6 new SQL tests, 447 total passing

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-01 10:15:06 +01:00
Jason Matthew 2d13a17c3b feat(llm): split and retry chunks that hit max_completion_tokens truncation
Token-budget chunking cuts the truncation rate but doesn't eliminate
it. Output token cost scales with extractable concept density rather
than input tokens — a chunk that lands on a directory of dense design
docs can pack under the input budget while needing more than
`max_completion_tokens=8192` to express every named concept, so the
response is truncated mid-string and `_parse_llm_json` returns an
empty fragment.

Pre-tuning chunk size to be conservative enough that this never
happens leaves throughput on the table for the common case. Adding a
hard `max_files_per_chunk` cap on top of `token_budget` reintroduces
the "tune a static constant" problem the previous commit set out to
fix.

The fix uses the API's own truncation signal:

1. `_call_openai_compat` and `_call_claude` now expose `finish_reason`
   on the result dict (Anthropic's `stop_reason == "max_tokens"` is
   normalised to `"length"`).
2. `_extract_with_adaptive_retry` checks it: when truncated, splits
   the chunk in half and recurses on each half. Recursion is bounded
   by `max_retry_depth` (default 3 → at most 8x fanout per top-level
   chunk).
3. Single-file chunks that truncate can't recover and surface a
   warning rather than infinite-loop.
4. `extract_corpus_parallel` routes every chunk through the retry
   wrapper. The `on_chunk_done` callback fires once per top-level
   chunk with the merged result — recursive splits are invisible to
   callers.
2026-04-30 22:08:28 +10:00
Jason Matthew cc5c54574d feat(llm): pack chunks by token budget, parallelise, accept tiktoken
Three independent improvements to extract_corpus_parallel:

1. Token-aware chunking. Replaces `chunk_size=20` static packing with
   a greedy packer keyed on `token_budget` (default 60_000), grouped
   by parent directory so related artefacts share a chunk. Pass
   `token_budget=None` to fall back to fixed-count packing.

2. Optional tiktoken (added to the [kimi] extra). When available,
   `_estimate_file_tokens` uses cl100k_base for accurate counts;
   without it, the existing chars/4 heuristic kicks in. Kimi-K2 ships
   a tiktoken-based tokenizer so estimates against Moonshot are very
   close to truth.

3. True parallelism. The function name said "parallel" but the body
   was a sequential for-loop. Now uses ThreadPoolExecutor capped at
   `max_concurrency` (default 4 — conservative against provider rate
   limits). `on_chunk_done(idx, total, result)` still fires once per
   chunk with the original submission idx so progress UIs work
   unchanged. `max_concurrency=1` skips the pool to preserve
   sequential semantics.

Plus failure tolerance: a chunk raising is now caught, logged to
stderr, and the run continues. Other chunks' results merge as normal.

On a 162-file repo (~125k words), the same work that took ~36 min
sequential under the old code finishes in ~7 min.
2026-04-30 22:08:28 +10:00
Danil Tarasov a9946fb7a0 Merge remote-tracking branch 'upstream/v5' into v5
# Conflicts:
#	graphify/extract.py
2026-04-29 17:55:21 +03:00
Safi dd86271312 Fix SSRF DNS rebinding TOCTOU and yt-dlp URL bypass (#591, #592)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-28 15:04:52 +01:00
Safi ee1df22b25 Fix PreToolUse hook for Claude Code v2.1.117+ (Glob|Grep → Bash matcher)
Grep and Glob tools removed in CC v2.1.117; searches now go through Bash.
Hook now reads stdin tool_input and pattern-matches on search commands.
Uninstall/reinstall handles both old and new matcher for clean upgrades.

Closes #578

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-28 00:56:54 +01:00
Yalkowni 563ee80494 fix(extract): skip dynamic template literals in import() args
import(`./handlers/${name}`) previously produced a garbage edge to a
path containing the unresolved ${name} expression. Now detects
template_substitution child nodes and breaks without emitting an edge.
Static template literals (no interpolation) still resolve correctly.

Adds 2 new tests: one asserting dynamic templates produce no edge,
one asserting static templates resolve like plain strings.
2026-04-27 16:41:19 -07:00
Yalkowni a1dc610079 feat(extract): add dynamic import() extraction for JS/TS
Adds _dynamic_import_js() helper (65 lines) that detects import() call
expressions in JS/TS, resolves the module path (same logic as static
imports including .js→.ts mapping and tsconfig aliases), and emits
imports_from edges from the enclosing function. Hooked into walk_calls
for JS/TS configs.

Also adds tests/fixtures/dynamic_import.ts fixture and 5 new tests
in tests/test_languages.py (all passing alongside 110 existing tests).
2026-04-27 16:23:38 -07:00
Danil Tarasov 3ff7188fbf feat: add cross-language edge contexts and context-aware queries 2026-04-24 02:59:04 +03:00
Safi 64f38acf3e implement #488 #482 #472 #490: legacy schema canonicalization, Java inheritance, aggregated HTML viz, check-update subcommand 2026-04-22 23:28:09 +01:00
Safi e4bdcc2487 add tests for canvas vault-relative paths and hook .exe skip 2026-04-22 23:04:40 +01:00
Safi 86d6d9317f fix #498: opencode config to .opencode/opencode.json, fix skill temp file paths, add --wiki step 2026-04-22 09:19:53 +01:00
Safi 429e46a665 v0.4.15: VS Code Copilot Chat, OpenCode/Gemini Windows fixes, .mjs/.ejs, macOS watch, god_nodes degree rename
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-15 23:08:43 +01:00
Safi 01fb51ba78 Fix 9 issues: kiro package data, betweenness perf, wiki step, opencode plugin, cache root, PHP missing edges, Windows stability, cross-file calls
- #352: add skill-kiro.md to pyproject.toml package-data
- #341: guard edge_betweenness at >5000 nodes; use approximate k=100 for suggest_questions on large graphs
- #354/#229: add Step 6b in skill.md to call to_wiki() when --wiki given (before Step 9 cleanup)
- #356: call _install_opencode_plugin() from install --platform opencode path
- #350: add cache_root param to extract() so subdirectory runs keep cache at ./graphify-out/cache/
- #230: PHP class_constant_access_expression emits references_constant edges
- #232: PHP scoped_call_expression (static method calls) emits calls edges
- #287: os.replace fallback for Windows WinError 5; graphify update exits 1 on failure; templates use graphify update . instead of python3 -c
- #348: cross-file call resolution for all languages via raw_calls + global label map pass in extract()

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-15 00:26:11 +01:00
Safi 2993cce8b2 v0.4.12: add Kiro IDE/CLI support, fix cache portability across machines
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 22:46:13 +01:00
Safi 1fbcaf840f release: v0.4.9 — PHP extractor improvements, Dart, diacritics, Hermes, fixes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 08:39:28 +01:00
Safi 4210de270f Fix watch edge key, claw path, Blade support, WSL MCP docs (0.4.7)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-12 20:49:24 +01:00
Safi 2a7d2f4fdc Remove Anthropic API call from transcribe.py - agent generates Whisper prompt itself
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-10 15:55:22 +01:00
Safi db07e9f93a Fix CI: mock lazy anthropic import via sys.modules instead of module attribute
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-10 15:48:54 +01:00
Safi cf5d721bee Add video/audio corpus support with yt-dlp download and Whisper transcription
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-10 15:40:18 +01:00
Safi 6d39432058 fix MCP server path validation security issue (0.3.26) 2026-04-10 01:08:15 +01:00
Safi 9c829d7a2e release 0.3.25: Aider + Copilot CLI, directed graphs, frontmatter cache, graphifyignore parent discovery, MCP fixes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-10 00:56:22 +01:00
Safi df26f85648 Add Gemini CLI support and sponsor nudge at pipeline completion (#105)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-09 17:40:45 +01:00
Safi df77d5f8ce Add Cursor support, fix _rebuild_code KeyError and node_link_data crash (#137, #148, #149)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-09 17:19:40 +01:00
Safi ca842472e5 Fix AST call edges confidence: INFERRED/0.8 -> EXTRACTED/1.0 (#127)
Tree-sitter resolves call targets directly from source — marking them
INFERRED was incorrect. Cross-file class-level uses edges remain INFERRED.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-09 09:22:12 +01:00
Safi e5760e6b03 Add OpenCode tool.execute.before plugin via graphify opencode install (#71)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-09 08:55:07 +01:00
Safi c3817d6144 Apply PRs #82 #93 #102 #109: extension drift, click detection, skill coverage, .graphify_python persistence
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-09 08:22:01 +01:00
Safi 7ff7bd2e8f v0.3.17: Julia support, smarter chunking, tree-sitter pin, progress output
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-08 22:54:18 +01:00
azizur100389 22f79db72f fix: suppress graspologic ANSI output that corrupts PowerShell scroll buffer
* fix: suppress graspologic ANSI output that breaks PowerShell scrolling

graspologic's leiden() emits ANSI escape sequences (progress bars,
colored warnings) that corrupt PowerShell 5.1's scroll buffer on
Windows, disabling vertical scrolling. Redirect stdout/stderr to
StringIO during leiden() calls to prevent any escape codes from
reaching the terminal.

Add 2 tests verifying cluster() produces no stdout/stderr output.

Fixes #19

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: add PowerShell troubleshooting section to Windows skill

Document the PowerShell 5.1 scrolling issue and provide 4
workarounds: upgrade graphify, use Windows Terminal, reset
terminal, or uninstall graspologic to use Louvain fallback.

Fixes #19

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-08 19:40:10 +01:00
Safi bd24ddb1d6 fix: hook JSON format, Go pkg scoping, xcassets PDF, cross-file guard, skill file paths (#83, #85, #52, #81) 2026-04-08 19:20:48 +01:00
Safi 92b70ce5f4 fix: sanitize_label double-encoding and --wiki missing from skill (#66, #55) 2026-04-08 09:40:21 +01:00
Safi a78d25d4fc Add ObjC support, C# inheritance, CLI query, symlink support, bug fixes (v0.3.7-0.3.9)
- Add Objective-C extractor (.m/.mm) with @interface, @implementation, @protocol, imports, calls
- Add C# inheritance extraction via base_list nodes (inherits edges)
- Add --obsidian-dir flag to skill.md and skill-windows.md
- Add graphify query CLI command with --dfs, --budget, --graph flags
- Add follow_symlinks parameter to detect() and collect_files() with cycle detection
- Fix semantic cache relative path resolution (was saving only 4/17 files)
- Fix validate.py missing rationale file_type causing 75 warnings per run
- Add tree-sitter-objc dependency
- Update README with .m/.mm extensions, --obsidian-dir usage, Codex $ skill trigger
- 367 tests passing

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-07 20:20:49 +01:00
Safi 3d5da6039a Add Elixir language support (.ex/.exs)
Extracts defmodule, def/defp, alias/import/require/use, and call graph.
Follows same custom-walk pattern as Zig and PowerShell extractors.
2026-04-07 12:54:12 +01:00
Safi 9d998cc810 Add Zig and PowerShell language support 2026-04-07 09:56:53 +01:00
Safi f2423bc114 Add .graphifyignore support for excluding files and directories 2026-04-07 09:41:01 +01:00