mirror of
https://github.com/safishamsi/graphify.git
synced 2026-09-22 21:45:58 +00:00
Merge branch 'safishamsi:v6' into feat/parallel-ast-extraction
This commit is contained in:
Hanzala Sohrab
@@ -2,6 +2,37 @@
|
||||
|
||||
Full release notes with details on each version: [GitHub Releases](https://github.com/safishamsi/graphify/releases)
|
||||
|
||||
## 0.6.6 (2026-05-02)
|
||||
|
||||
- Fix: `skill-windows.md` rewritten from PowerShell to bash — Claude Code on Windows uses git-bash so PowerShell syntax (`$null`, `$LASTEXITCODE`, `Select-Object`, `& (Get-Content ...)`, `Remove-Item`) caused exit code 49 failures; now mirrors `skill.md` structure with `python` added as fallback after `python3` for Windows Conda (#39)
|
||||
- Fix: wiki `to_wiki()` now clears stale articles before regenerating, preventing orphan .md accumulation (#558)
|
||||
- Fix: `_safe_filename()` in wiki.py now strips Windows-reserved characters (`< > : " / \ | ? *`) and caps length at 200 chars (#594)
|
||||
- Fix: rationale-node leakage in cross-file INFERRED call resolution — rationale nodes now excluded from name lookup; edge direction (`calls`, `rationale_for`) preserved correctly at JSON export (#576)
|
||||
- Feat: `.graphifyinclude` hidden path allowlist — opt specific hidden dirs into traversal (e.g. `.hermes/plans/**/*.md`) (#583)
|
||||
- Feat: `--no-viz` flag wired in `cluster-only`; `GRAPHIFY_VIZ_NODE_LIMIT` env var overrides 5000-node HTML threshold (#565)
|
||||
- Fix: stray colon SyntaxError in `skill-trae.md` `--cluster-only` block (#603)
|
||||
- Docs: skill INFERRED confidence score guidance changed to discrete rubric (0.55/0.65/0.75/0.85/0.95) backed by calibration data (#546)
|
||||
- Docs: skill `--update` prune output clarified — splits no-drift vs drift cases (#544)
|
||||
- Docs: skill `--update` merge step now calls `save_manifest` to prevent deleted files reappearing (#545)
|
||||
|
||||
## 0.6.5 (2026-05-02)
|
||||
|
||||
- Fix: Kotlin call-walker now accepts both `simple_identifier` and `identifier` node types — PyPI's `tree_sitter_kotlin` grammar uses `identifier` while older forks use `simple_identifier`, causing zero `calls` edges to be emitted (#659)
|
||||
- Feat: community sidebar now uses checkbox-based multi-select instead of show/hide buttons — supports indeterminate "select all" state (#647)
|
||||
- Feat: `graphify update --force` and `GRAPHIFY_FORCE=1` env var — bypass the node-count safety check after refactors that legitimately shrink the graph (#639)
|
||||
- Fix: Codex PreToolUse hook on Windows — replaced `python3 -c "..."` inline command (fails on Conda where only `python` exists, and breaks PowerShell JSON parsing) with `graphify hook-check`, a new shell-agnostic subcommand. Re-run `graphify codex install` to regenerate the hook (#651, #522)
|
||||
|
||||
## 0.6.4 (2026-05-02)
|
||||
|
||||
- Fix: Codex PreToolUse hook failed on Windows — `[ -f ]` is bash-only and crashes on `cmd.exe`; replaced with a cross-platform Python one-liner (`pathlib.Path.exists()`) (#651)
|
||||
|
||||
## 0.6.3 (2026-05-02)
|
||||
|
||||
- Fix: incremental rebuild (`graphify update`, post-commit hook) dropped INFERRED/AMBIGUOUS semantic nodes extracted from code files — node preservation now filters by ID membership in the new AST output instead of `file_type`, so LLM-extracted call/data-flow edges survive code-only rebuilds (#653)
|
||||
- Fix: post-commit and post-checkout hooks blocked `git commit` for the full rebuild duration (hours on large repos) — rebuilds now detach via `nohup & disown`, git returns in ~100ms, log written to `~/.cache/graphify-rebuild.log` (#650)
|
||||
- Fix: cross-file INFERRED `calls` resolution used a last-write-wins name map, causing common short names (`log`, `execute`, `find`) to accumulate hundreds of spurious edges and dominate god_nodes ranking — resolution now skips any callee name that matches 2+ candidates (ambiguous, no import evidence to pick the right target) (#543)
|
||||
- Fix: `cluster-only` command crashed on graphs with >5000 nodes due to unguarded `to_html` call — now wrapped in try/except ValueError matching the watch/hook path (#541)
|
||||
|
||||
## 0.6.2 (2026-05-01)
|
||||
|
||||
- Fix: Kimi K2.6 reasoning mode consumed entire token budget leaving `content` empty — thinking now disabled on Moonshot calls so graphs actually populate (#623)
|
||||
|
||||
@@ -21,7 +21,7 @@
|
||||
</a>
|
||||
</p>
|
||||
|
||||
**An AI coding assistant skill.** Type `/graphify` in Claude Code, Codex, OpenCode, Cursor, Gemini CLI, GitHub Copilot CLI, VS Code Copilot Chat, Aider, OpenClaw, Factory Droid, Trae, Hermes, Kiro, or Google Antigravity - it reads your files, builds a knowledge graph, and gives you back structure you didn't know was there. Understand a codebase faster. Find the "why" behind architectural decisions.
|
||||
**An AI coding assistant skill.** Type `/graphify` in Claude Code, Codex, OpenCode, Cursor, Gemini CLI, GitHub Copilot CLI, VS Code Copilot Chat, Aider, OpenClaw, Factory Droid, Trae, Hermes, Kiro, Pi, or Google Antigravity - it reads your files, builds a knowledge graph, and gives you back structure you didn't know was there. Understand a codebase faster. Find the "why" behind architectural decisions.
|
||||
|
||||
Fully multimodal. Drop in code, PDFs, markdown, screenshots, diagrams, whiteboard photos, images in other languages, or video and audio files - graphify extracts concepts and relationships from all of it and connects them into one graph. Videos are transcribed with Whisper using a domain-aware prompt derived from your corpus. YAML/YML files (Kubernetes, Kustomize, Helm, config) are indexed for semantic extraction. SQL files are AST-extracted deterministically — tables, views, functions, foreign keys, and FROM/JOIN relationships map directly into the graph with no LLM needed. 25 languages supported via tree-sitter AST (Python, JS, TS, Go, Rust, Java, C, C++, Ruby, C#, Kotlin, Scala, PHP, Swift, Lua, Zig, PowerShell, Elixir, Objective-C, Julia, Verilog, SystemVerilog, Vue, Svelte, Dart).
|
||||
|
||||
@@ -61,7 +61,7 @@ Every relationship is tagged `EXTRACTED` (found directly in source), `INFERRED`
|
||||
|
||||
## Install
|
||||
|
||||
**Requires:** Python 3.10+ and one of: [Claude Code](https://claude.ai/code), [Codex](https://openai.com/codex), [OpenCode](https://opencode.ai), [Cursor](https://cursor.com), [Gemini CLI](https://github.com/google-gemini/gemini-cli), [GitHub Copilot CLI](https://docs.github.com/en/copilot/how-tos/copilot-cli), [VS Code Copilot Chat](https://code.visualstudio.com/docs/copilot/overview), [Aider](https://aider.chat), [OpenClaw](https://openclaw.ai), [Factory Droid](https://factory.ai), [Trae](https://trae.ai), [Kiro](https://kiro.dev), Hermes, or [Google Antigravity](https://antigravity.google)
|
||||
**Requires:** Python 3.10+ and one of: [Claude Code](https://claude.ai/code), [Codex](https://openai.com/codex), [OpenCode](https://opencode.ai), [Cursor](https://cursor.com), [Gemini CLI](https://github.com/google-gemini/gemini-cli), [GitHub Copilot CLI](https://docs.github.com/en/copilot/how-tos/copilot-cli), [VS Code Copilot Chat](https://code.visualstudio.com/docs/copilot/overview), [Aider](https://aider.chat), [OpenClaw](https://openclaw.ai), [Factory Droid](https://factory.ai), [Trae](https://trae.ai), [Kiro](https://kiro.dev), [Pi](https://pi.dev), Hermes, or [Google Antigravity](https://antigravity.google)
|
||||
|
||||
```bash
|
||||
# Recommended — works on Mac and Linux with no PATH setup needed
|
||||
@@ -94,6 +94,7 @@ pip install graphifyy && graphify install
|
||||
| Gemini CLI | `graphify install --platform gemini` |
|
||||
| Hermes | `graphify install --platform hermes` |
|
||||
| Kiro IDE/CLI | `graphify kiro install` |
|
||||
| Pi coding agent | `graphify install --platform pi` |
|
||||
| Cursor | `graphify cursor install` |
|
||||
| Google Antigravity | `graphify antigravity install` |
|
||||
|
||||
@@ -127,6 +128,7 @@ After building a graph, run this once in your project:
|
||||
| Gemini CLI | `graphify gemini install` |
|
||||
| Hermes | `graphify hermes install` |
|
||||
| Kiro IDE/CLI | `graphify kiro install` |
|
||||
| Pi coding agent | `graphify pi install` |
|
||||
| Google Antigravity | `graphify antigravity install` |
|
||||
|
||||
**Claude Code** does two things: writes a `CLAUDE.md` section telling Claude to read `graphify-out/GRAPH_REPORT.md` before answering architecture questions, and installs a **PreToolUse hook** (`settings.json`) that fires before every Glob and Grep call. If a knowledge graph exists, Claude sees: _"graphify: Knowledge graph exists. Read GRAPH_REPORT.md for god nodes and community structure before searching raw files."_ — so Claude navigates via the graph instead of grepping through every file.
|
||||
@@ -141,6 +143,8 @@ After building a graph, run this once in your project:
|
||||
|
||||
**Aider, OpenClaw, Factory Droid, Trae, and Hermes** write the same rules to `AGENTS.md` in your project root and copy the skill to the platform's global skill directory. These platforms don't support tool hooks, so AGENTS.md is the always-on mechanism.
|
||||
|
||||
**Pi coding agent** copies the skill to `~/.pi/agent/skills/graphify/SKILL.md`. Run `graphify pi install` to set it up.
|
||||
|
||||
**Kiro IDE/CLI** writes the skill to `.kiro/skills/graphify/SKILL.md` (invoked via `/graphify`) and a steering file to `.kiro/steering/graphify.md` with `inclusion: always` — Kiro injects this into every conversation automatically, no hook needed.
|
||||
|
||||
**Google Antigravity** writes `.agents/rules/graphify.md` (always-on rules) and `.agents/workflows/graphify.md` (registers `/graphify` as a slash command). No hook equivalent exists in Antigravity — rules are the always-on mechanism.
|
||||
@@ -408,7 +412,7 @@ Audio never leaves your machine. All transcription runs locally.
|
||||
|
||||
**Auto-sync** (`--watch`) - run in a background terminal and the graph updates itself as your codebase changes. Code file saves trigger an instant rebuild (AST only, no LLM). Doc/image changes notify you to run `--update` for the LLM re-pass.
|
||||
|
||||
**Git hooks** (`graphify hook install`) - installs post-commit and post-checkout hooks. Graph rebuilds automatically after every commit and every branch switch. If a rebuild fails, the hook exits with a non-zero code so git surfaces the error instead of silently continuing. No background process needed.
|
||||
**Git hooks** (`graphify hook install`) - installs post-commit and post-checkout hooks. Graph rebuilds automatically after every commit and every branch switch. Rebuilds run detached in the background so `git commit` returns instantly — log written to `~/.cache/graphify-rebuild.log` (`tail -f` for status).
|
||||
|
||||
**Wiki** (`--wiki`) - Wikipedia-style markdown articles per community and god node, with an `index.md` entry point. Point any agent at `index.md` and it can navigate the knowledge base by reading files instead of parsing JSON.
|
||||
|
||||
|
||||
+83
-11
@@ -124,6 +124,11 @@ _PLATFORM_CONFIG: dict[str, dict] = {
|
||||
"skill_dst": Path(".kiro") / "skills" / "graphify" / "SKILL.md",
|
||||
"claude_md": False,
|
||||
},
|
||||
"pi": {
|
||||
"skill_file": "skill-pi.md",
|
||||
"skill_dst": Path(".pi") / "agent" / "skills" / "graphify" / "SKILL.md",
|
||||
"claude_md": False,
|
||||
},
|
||||
"antigravity": {
|
||||
"skill_file": "skill.md",
|
||||
"skill_dst": Path(".agents") / "skills" / "graphify" / "SKILL.md",
|
||||
@@ -718,11 +723,11 @@ _CODEX_HOOK = {
|
||||
"hooks": [
|
||||
{
|
||||
"type": "command",
|
||||
"command": (
|
||||
"[ -f graphify-out/graph.json ] && "
|
||||
r"""echo '{"hookSpecificOutput":{"hookEventName":"PreToolUse","additionalContext":"graphify: Knowledge graph exists. Read graphify-out/GRAPH_REPORT.md for god nodes and community structure before searching raw files."}}' """
|
||||
"|| true"
|
||||
),
|
||||
# Use the graphify CLI itself so the hook is shell-agnostic:
|
||||
# no [ -f ] bash syntax, no python3 vs python Conda issue,
|
||||
# no JSON escaping inside PowerShell strings. Works on
|
||||
# Windows (PowerShell/cmd.exe), macOS, and Linux.
|
||||
"command": "graphify hook-check",
|
||||
}
|
||||
],
|
||||
}
|
||||
@@ -989,7 +994,7 @@ def main() -> None:
|
||||
print("Usage: graphify <command>")
|
||||
print()
|
||||
print("Commands:")
|
||||
print(" install [--platform P] copy skill to platform config dir (claude|windows|codex|opencode|aider|claw|droid|trae|trae-cn|gemini|cursor|antigravity|hermes|kiro)")
|
||||
print(" install [--platform P] copy skill to platform config dir (claude|windows|codex|opencode|aider|claw|droid|trae|trae-cn|gemini|cursor|antigravity|hermes|kiro|pi)")
|
||||
print(" path \"A\" \"B\" shortest path between two nodes in graph.json")
|
||||
print(" --graph <path> path to graph.json (default graphify-out/graph.json)")
|
||||
print(" explain \"X\" plain-language explanation of a node and its neighbors")
|
||||
@@ -1005,7 +1010,10 @@ def main() -> None:
|
||||
print(" --dir <path> target directory (default: ./raw)")
|
||||
print(" watch <path> watch a folder and rebuild the graph on code changes")
|
||||
print(" update <path> re-extract code files and update the graph (no LLM needed)")
|
||||
print(" --force overwrite graph.json even if the rebuild has fewer nodes")
|
||||
print(" (also: GRAPHIFY_FORCE=1 env var; use after refactors that delete code)")
|
||||
print(" cluster-only <path> rerun clustering on an existing graph.json and regenerate report")
|
||||
print(" --no-viz skip graph.html generation (useful for >5000 node graphs / CI)")
|
||||
print(" query \"<question>\" BFS traversal of graph.json for a question")
|
||||
print(" --dfs use depth-first instead of breadth-first")
|
||||
print(" --budget N cap output at N tokens (default 2000)")
|
||||
@@ -1051,6 +1059,8 @@ def main() -> None:
|
||||
print(" hermes uninstall remove skill from ~/.hermes/skills/graphify/")
|
||||
print(" kiro install write skill to .kiro/skills/graphify/ + steering file (Kiro IDE/CLI)")
|
||||
print(" kiro uninstall remove skill + steering file")
|
||||
print(" pi install write skill to ~/.pi/agent/skills/graphify/ (Pi coding agent)")
|
||||
print(" pi uninstall remove skill from ~/.pi/agent/skills/graphify/")
|
||||
print()
|
||||
return
|
||||
|
||||
@@ -1138,6 +1148,26 @@ def main() -> None:
|
||||
else:
|
||||
print("Usage: graphify kiro [install|uninstall]", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
elif cmd == "pi":
|
||||
subcmd = sys.argv[2] if len(sys.argv) > 2 else ""
|
||||
if subcmd == "install":
|
||||
install("pi")
|
||||
elif subcmd == "uninstall":
|
||||
skill_dst = Path.home() / ".pi" / "agent" / "skills" / "graphify" / "SKILL.md"
|
||||
if skill_dst.exists():
|
||||
skill_dst.unlink()
|
||||
print(f" skill removed -> {skill_dst}")
|
||||
version_file = skill_dst.parent / ".graphify_version"
|
||||
if version_file.exists():
|
||||
version_file.unlink()
|
||||
for d in (skill_dst.parent, skill_dst.parent.parent, skill_dst.parent.parent.parent):
|
||||
try:
|
||||
d.rmdir()
|
||||
except OSError:
|
||||
break
|
||||
else:
|
||||
print("Usage: graphify pi [install|uninstall]", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
elif cmd in ("aider", "codex", "opencode", "claw", "droid", "trae", "trae-cn", "hermes"):
|
||||
subcmd = sys.argv[2] if len(sys.argv) > 2 else ""
|
||||
if subcmd == "install":
|
||||
@@ -1383,6 +1413,7 @@ def main() -> None:
|
||||
|
||||
elif cmd == "cluster-only":
|
||||
watch_path = Path(sys.argv[2]) if len(sys.argv) > 2 else Path(".")
|
||||
no_viz = "--no-viz" in sys.argv
|
||||
graph_json = watch_path / "graphify-out" / "graph.json"
|
||||
if not graph_json.exists():
|
||||
print(f"error: no graph found at {graph_json} — run /graphify first", file=sys.stderr)
|
||||
@@ -1412,12 +1443,34 @@ def main() -> None:
|
||||
out = watch_path / "graphify-out"
|
||||
(out / "GRAPH_REPORT.md").write_text(report, encoding="utf-8")
|
||||
to_json(G, communities, str(out / "graph.json"))
|
||||
to_html(G, communities, str(out / "graph.html"), community_labels=labels or None)
|
||||
print(f"Done — {len(communities)} communities. GRAPH_REPORT.md, graph.json and graph.html updated.")
|
||||
|
||||
# Mirror watch.py pattern: gate to_html so core outputs (graph.json +
|
||||
# GRAPH_REPORT.md) always land. Honor --no-viz explicitly; otherwise
|
||||
# fall back to ValueError handling so an oversized graph doesn't crash
|
||||
# the CLI mid-write and leave a stale graph.html on disk.
|
||||
html_target = out / "graph.html"
|
||||
if no_viz:
|
||||
if html_target.exists():
|
||||
html_target.unlink()
|
||||
print(f"Done — {len(communities)} communities. GRAPH_REPORT.md and graph.json updated (--no-viz; graph.html removed).")
|
||||
else:
|
||||
try:
|
||||
to_html(G, communities, str(html_target), community_labels=labels or None)
|
||||
print(f"Done — {len(communities)} communities. GRAPH_REPORT.md, graph.json and graph.html updated.")
|
||||
except ValueError as viz_err:
|
||||
if html_target.exists():
|
||||
html_target.unlink()
|
||||
print(f"Skipped graph.html: {viz_err}")
|
||||
print(f"Done — {len(communities)} communities. GRAPH_REPORT.md and graph.json updated.")
|
||||
|
||||
elif cmd == "update":
|
||||
if len(sys.argv) > 2:
|
||||
watch_path = Path(sys.argv[2])
|
||||
force = os.environ.get("GRAPHIFY_FORCE", "").lower() in ("1", "true", "yes")
|
||||
argv = list(sys.argv)
|
||||
if "--force" in argv[2:]:
|
||||
force = True
|
||||
argv = [a for a in argv if a != "--force"]
|
||||
if len(argv) > 2:
|
||||
watch_path = Path(argv[2])
|
||||
else:
|
||||
# Try to recover the scan root saved by the last full build
|
||||
saved = Path("graphify-out/.graphify_root")
|
||||
@@ -1430,7 +1483,7 @@ def main() -> None:
|
||||
sys.exit(1)
|
||||
from graphify.watch import _rebuild_code
|
||||
print(f"Re-extracting code files in {watch_path} (no LLM needed)...")
|
||||
ok = _rebuild_code(watch_path)
|
||||
ok = _rebuild_code(watch_path, force=force)
|
||||
if ok:
|
||||
print("Code graph updated. For doc/paper/image changes run /graphify --update in your AI assistant.")
|
||||
if not os.environ.get("MOONSHOT_API_KEY") and not os.environ.get("GRAPHIFY_NO_TIPS"):
|
||||
@@ -1439,6 +1492,25 @@ def main() -> None:
|
||||
print("Nothing to update or rebuild failed — check output above.", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
elif cmd == "hook-check":
|
||||
# Shell-agnostic PreToolUse hook entry point for Codex (and any platform
|
||||
# where embedding Python/bash inline in a JSON hook command is fragile).
|
||||
# Prints the hookSpecificOutput JSON if graph.json exists, exits 0 silently
|
||||
# if not. Works on Windows PowerShell, cmd.exe, macOS, and Linux.
|
||||
graph = Path("graphify-out") / "graph.json"
|
||||
if graph.exists():
|
||||
import json as _json
|
||||
print(_json.dumps({
|
||||
"hookSpecificOutput": {
|
||||
"hookEventName": "PreToolUse",
|
||||
"additionalContext": (
|
||||
"graphify: Knowledge graph exists. "
|
||||
"Read graphify-out/GRAPH_REPORT.md for god nodes and "
|
||||
"community structure before searching raw files."
|
||||
),
|
||||
}
|
||||
}))
|
||||
sys.exit(0)
|
||||
elif cmd == "check-update":
|
||||
if len(sys.argv) < 3:
|
||||
print("Usage: graphify check-update <path>", file=sys.stderr)
|
||||
|
||||
+118
-5
@@ -509,6 +509,115 @@ def _is_ignored(path: Path, root: Path, patterns: list[tuple[Path, str]]) -> boo
|
||||
return result
|
||||
|
||||
|
||||
def _load_graphifyinclude(root: Path) -> list[tuple[Path, str]]:
|
||||
"""Read .graphifyinclude allowlist patterns from root and ancestors.
|
||||
|
||||
Include patterns opt matching hidden files/dirs into traversal. Sensitive
|
||||
files and hard-skipped noise directories are still excluded later.
|
||||
Uses the same VCS-root ceiling logic as _load_graphifyignore.
|
||||
"""
|
||||
root = root.resolve()
|
||||
ceiling = _find_vcs_root(root) or root
|
||||
|
||||
dirs: list[Path] = []
|
||||
current = root
|
||||
while True:
|
||||
dirs.append(current)
|
||||
if current == ceiling:
|
||||
break
|
||||
current = current.parent
|
||||
dirs.reverse()
|
||||
|
||||
patterns: list[tuple[Path, str]] = []
|
||||
for d in dirs:
|
||||
include_file = d / ".graphifyinclude"
|
||||
if include_file.exists():
|
||||
for raw in include_file.read_text(encoding="utf-8", errors="ignore").splitlines():
|
||||
line = _parse_gitignore_line(raw)
|
||||
if line:
|
||||
patterns.append((d, line))
|
||||
return patterns
|
||||
|
||||
|
||||
def _is_included(path: Path, root: Path, patterns: list[tuple[Path, str]]) -> bool:
|
||||
"""Return True if path matches any .graphifyinclude allowlist pattern."""
|
||||
if not patterns:
|
||||
return False
|
||||
|
||||
def _matches(rel: str, p: str) -> bool:
|
||||
parts = rel.split("/")
|
||||
if fnmatch.fnmatch(rel, p):
|
||||
return True
|
||||
if fnmatch.fnmatch(path.name, p):
|
||||
return True
|
||||
for i, part in enumerate(parts):
|
||||
if fnmatch.fnmatch(part, p):
|
||||
return True
|
||||
if fnmatch.fnmatch("/".join(parts[:i + 1]), p):
|
||||
return True
|
||||
return False
|
||||
|
||||
for anchor, pattern in patterns:
|
||||
anchored = pattern.startswith("/")
|
||||
p = pattern.strip("/")
|
||||
if not p:
|
||||
continue
|
||||
if anchored:
|
||||
try:
|
||||
rel_anchor = str(path.relative_to(anchor)).replace(os.sep, "/")
|
||||
if _matches(rel_anchor, p):
|
||||
return True
|
||||
except ValueError:
|
||||
pass
|
||||
else:
|
||||
try:
|
||||
rel = str(path.relative_to(root)).replace(os.sep, "/")
|
||||
if _matches(rel, p):
|
||||
return True
|
||||
except ValueError:
|
||||
pass
|
||||
if anchor != root:
|
||||
try:
|
||||
rel_anchor = str(path.relative_to(anchor)).replace(os.sep, "/")
|
||||
if _matches(rel_anchor, p):
|
||||
return True
|
||||
except ValueError:
|
||||
pass
|
||||
return False
|
||||
|
||||
|
||||
def _could_contain_included_path(path: Path, root: Path, patterns: list[tuple[Path, str]]) -> bool:
|
||||
"""Return True if a directory may contain files matched by .graphifyinclude."""
|
||||
if not patterns:
|
||||
return False
|
||||
|
||||
rels: list[str] = []
|
||||
try:
|
||||
rels.append(str(path.relative_to(root)).replace(os.sep, "/"))
|
||||
except ValueError:
|
||||
pass
|
||||
for anchor, _ in patterns:
|
||||
if anchor != root:
|
||||
try:
|
||||
rels.append(str(path.relative_to(anchor)).replace(os.sep, "/"))
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
for rel in rels:
|
||||
rel = rel.strip("/")
|
||||
if not rel:
|
||||
return True
|
||||
for _, pattern in patterns:
|
||||
p = pattern.strip("/")
|
||||
if not p:
|
||||
continue
|
||||
if p == rel or p.startswith(rel + "/"):
|
||||
return True
|
||||
if fnmatch.fnmatch(rel, p):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def detect(root: Path, *, follow_symlinks: bool = False) -> dict:
|
||||
root = root.resolve()
|
||||
files: dict[FileType, list[str]] = {
|
||||
@@ -522,6 +631,7 @@ def detect(root: Path, *, follow_symlinks: bool = False) -> dict:
|
||||
|
||||
skipped_sensitive: list[str] = []
|
||||
ignore_patterns = _load_graphifyignore(root)
|
||||
include_patterns = _load_graphifyinclude(root)
|
||||
|
||||
# Always include graphify-out/memory/ - query results filed back into the graph
|
||||
memory_dir = root / "graphify-out" / "memory"
|
||||
@@ -543,10 +653,12 @@ def detect(root: Path, *, follow_symlinks: bool = False) -> dict:
|
||||
dirnames.clear()
|
||||
continue
|
||||
if not in_memory_tree:
|
||||
# Prune noise dirs in-place so os.walk never descends into them
|
||||
# Prune noise dirs in-place so os.walk never descends into them.
|
||||
# Hidden dirs are allowed through if they could contain an
|
||||
# explicitly included path (.graphifyinclude allowlist).
|
||||
dirnames[:] = [
|
||||
d for d in dirnames
|
||||
if not d.startswith(".")
|
||||
if (not d.startswith(".") or _could_contain_included_path(dp / d, root, include_patterns))
|
||||
and not _is_noise_dir(d)
|
||||
and not _is_ignored(dp / d, root, ignore_patterns)
|
||||
]
|
||||
@@ -565,8 +677,9 @@ def detect(root: Path, *, follow_symlinks: bool = False) -> dict:
|
||||
in_memory = memory_dir.exists() and str(p).startswith(str(memory_dir))
|
||||
if not in_memory:
|
||||
# Hidden files are already excluded via dir pruning above,
|
||||
# but catch hidden files at the root level
|
||||
if p.name.startswith("."):
|
||||
# but catch hidden files at the root level. A .graphifyinclude
|
||||
# entry can opt a specific hidden file back in.
|
||||
if p.name.startswith(".") and not _is_included(p, root, include_patterns):
|
||||
continue
|
||||
# Skip files inside our own converted/ dir (avoid re-processing sidecars)
|
||||
if str(p).startswith(str(converted_dir)):
|
||||
@@ -623,7 +736,7 @@ def detect(root: Path, *, follow_symlinks: bool = False) -> dict:
|
||||
def _md5_file(path: Path) -> str:
|
||||
"""MD5 of file contents streamed in 64KB chunks — for change detection only."""
|
||||
import hashlib as _hl
|
||||
h = _hl.md5()
|
||||
h = _hl.md5(usedforsecurity=False)
|
||||
try:
|
||||
with path.open("rb") as f:
|
||||
for chunk in iter(lambda: f.read(65536), b""):
|
||||
|
||||
+78
-17
@@ -25,6 +25,22 @@ COMMUNITY_COLORS = [
|
||||
MAX_NODES_FOR_VIZ = 5_000
|
||||
|
||||
|
||||
def _viz_node_limit() -> int:
|
||||
"""Return the effective viz node limit, honoring GRAPHIFY_VIZ_NODE_LIMIT env var.
|
||||
|
||||
Falls back to MAX_NODES_FOR_VIZ when the env var is unset, empty, or non-integer.
|
||||
Set to 0 to disable HTML viz unconditionally (useful for CI runners).
|
||||
"""
|
||||
import os
|
||||
raw = os.environ.get("GRAPHIFY_VIZ_NODE_LIMIT")
|
||||
if raw is None or not raw.strip():
|
||||
return MAX_NODES_FOR_VIZ
|
||||
try:
|
||||
return int(raw)
|
||||
except ValueError:
|
||||
return MAX_NODES_FOR_VIZ
|
||||
|
||||
|
||||
def _html_styles() -> str:
|
||||
return """<style>
|
||||
* { box-sizing: border-box; margin: 0; padding: 0; }
|
||||
@@ -55,9 +71,14 @@ def _html_styles() -> str:
|
||||
.legend-label { flex: 1; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
|
||||
.legend-count { color: #666; font-size: 11px; }
|
||||
#stats { padding: 10px 14px; border-top: 1px solid #2a2a4e; font-size: 11px; color: #555; }
|
||||
#legend-controls { display: flex; gap: 6px; margin-bottom: 8px; }
|
||||
#legend-controls button { flex: 1; background: #0f0f1a; border: 1px solid #3a3a5e; color: #aaa; padding: 4px 0; border-radius: 4px; font-size: 11px; cursor: pointer; }
|
||||
#legend-controls button:hover { border-color: #4E79A7; color: #e0e0e0; }
|
||||
#legend-controls { display: flex; align-items: center; gap: 8px; margin-bottom: 8px; padding: 4px 0; }
|
||||
#legend-controls label { display: flex; align-items: center; gap: 6px; cursor: pointer; font-size: 12px; color: #aaa; user-select: none; }
|
||||
#legend-controls label:hover { color: #e0e0e0; }
|
||||
.legend-cb, #select-all-cb { appearance: none; -webkit-appearance: none; width: 14px; height: 14px; border: 1.5px solid #3a3a5e; border-radius: 3px; background: #0f0f1a; cursor: pointer; position: relative; flex-shrink: 0; }
|
||||
.legend-cb:checked, #select-all-cb:checked { background: #4E79A7; border-color: #4E79A7; }
|
||||
.legend-cb:checked::after, #select-all-cb:checked::after { content: ''; position: absolute; left: 3.5px; top: 1px; width: 4px; height: 7px; border: solid #fff; border-width: 0 2px 2px 0; transform: rotate(45deg); }
|
||||
#select-all-cb:indeterminate { background: #4E79A7; border-color: #4E79A7; }
|
||||
#select-all-cb:indeterminate::after { content: ''; position: absolute; left: 2px; top: 5px; width: 8px; height: 2px; background: #fff; border: none; transform: none; }
|
||||
</style>"""
|
||||
|
||||
|
||||
@@ -244,26 +265,41 @@ document.addEventListener('click', e => {{
|
||||
|
||||
const hiddenCommunities = new Set();
|
||||
|
||||
const selectAllCb = document.getElementById('select-all-cb');
|
||||
|
||||
function updateSelectAllState() {{
|
||||
const total = LEGEND.length;
|
||||
const hidden = hiddenCommunities.size;
|
||||
selectAllCb.checked = hidden === 0;
|
||||
selectAllCb.indeterminate = hidden > 0 && hidden < total;
|
||||
}}
|
||||
|
||||
function toggleAllCommunities(hide) {{
|
||||
document.querySelectorAll('.legend-item').forEach(item => {{
|
||||
hide ? item.classList.add('dimmed') : item.classList.remove('dimmed');
|
||||
}});
|
||||
document.querySelectorAll('.legend-cb').forEach(cb => {{
|
||||
cb.checked = !hide;
|
||||
}});
|
||||
LEGEND.forEach(c => {{
|
||||
if (hide) hiddenCommunities.add(c.cid); else hiddenCommunities.delete(c.cid);
|
||||
}});
|
||||
const updates = RAW_NODES.map(n => ({{ id: n.id, hidden: hide }}));
|
||||
nodesDS.update(updates);
|
||||
updateSelectAllState();
|
||||
}}
|
||||
|
||||
const legendEl = document.getElementById('legend');
|
||||
LEGEND.forEach(c => {{
|
||||
const item = document.createElement('div');
|
||||
item.className = 'legend-item';
|
||||
item.innerHTML = `<div class="legend-dot" style="background:${{c.color}}"></div>
|
||||
<span class="legend-label">${{c.label}}</span>
|
||||
<span class="legend-count">${{c.count}}</span>`;
|
||||
item.onclick = () => {{
|
||||
if (hiddenCommunities.has(c.cid)) {{
|
||||
const cb = document.createElement('input');
|
||||
cb.type = 'checkbox';
|
||||
cb.className = 'legend-cb';
|
||||
cb.checked = true;
|
||||
cb.addEventListener('change', (e) => {{
|
||||
e.stopPropagation();
|
||||
if (cb.checked) {{
|
||||
hiddenCommunities.delete(c.cid);
|
||||
item.classList.remove('dimmed');
|
||||
}} else {{
|
||||
@@ -272,8 +308,18 @@ LEGEND.forEach(c => {{
|
||||
}}
|
||||
const updates = RAW_NODES
|
||||
.filter(n => n.community === c.cid)
|
||||
.map(n => ({{ id: n.id, hidden: hiddenCommunities.has(c.cid) }}));
|
||||
.map(n => ({{ id: n.id, hidden: !cb.checked }}));
|
||||
nodesDS.update(updates);
|
||||
updateSelectAllState();
|
||||
}});
|
||||
item.innerHTML = `<div class="legend-dot" style="background:${{c.color}}"></div>
|
||||
<span class="legend-label">${{c.label}}</span>
|
||||
<span class="legend-count">${{c.count}}</span>`;
|
||||
item.prepend(cb);
|
||||
item.onclick = (e) => {{
|
||||
if (e.target === cb) return;
|
||||
cb.checked = !cb.checked;
|
||||
cb.dispatchEvent(new Event('change'));
|
||||
}};
|
||||
legendEl.appendChild(item);
|
||||
}});
|
||||
@@ -327,6 +373,15 @@ def to_json(G: nx.Graph, communities: dict[int, list[str]], output_path: str, *,
|
||||
if "confidence_score" not in link:
|
||||
conf = link.get("confidence", "EXTRACTED")
|
||||
link["confidence_score"] = _CONFIDENCE_SCORE_DEFAULTS.get(conf, 1.0)
|
||||
# Restore original edge direction. Undirected NetworkX storage may
|
||||
# canonicalize endpoint order, flipping `calls` and other directional
|
||||
# edges in graph.json. The build path stashes the true endpoints in
|
||||
# _src/_tgt for exactly this purpose (#563).
|
||||
true_src = link.pop("_src", None)
|
||||
true_tgt = link.pop("_tgt", None)
|
||||
if true_src is not None and true_tgt is not None:
|
||||
link["source"] = true_src
|
||||
link["target"] = true_tgt
|
||||
data["hyperedges"] = getattr(G, "graph", {}).get("hyperedges", [])
|
||||
with open(output_path, "w", encoding="utf-8") as f: # nosec
|
||||
json.dump(data, f, indent=2)
|
||||
@@ -391,10 +446,12 @@ def to_html(
|
||||
If member_counts is provided (aggregated community view), node sizes are
|
||||
based on community member counts rather than graph degree.
|
||||
"""
|
||||
if G.number_of_nodes() > MAX_NODES_FOR_VIZ:
|
||||
limit = _viz_node_limit()
|
||||
if G.number_of_nodes() > limit:
|
||||
raise ValueError(
|
||||
f"Graph has {G.number_of_nodes()} nodes - too large for HTML viz. "
|
||||
f"Use --no-viz or reduce input size."
|
||||
f"Graph has {G.number_of_nodes()} nodes - too large for HTML viz "
|
||||
f"(limit: {limit}). Use --no-viz, raise GRAPHIFY_VIZ_NODE_LIMIT, "
|
||||
f"or reduce input size."
|
||||
)
|
||||
|
||||
node_community = _node_community_map(communities)
|
||||
@@ -431,14 +488,19 @@ def to_html(
|
||||
"degree": deg,
|
||||
})
|
||||
|
||||
# Build edges list
|
||||
# Build edges list. Restore original edge direction from _src/_tgt
|
||||
# (stashed by build.py for exactly this reason): undirected NetworkX
|
||||
# canonicalizes endpoint order, which would otherwise flip the arrow
|
||||
# for `calls` and `rationale_for` in the rendered graph (#563).
|
||||
vis_edges = []
|
||||
for u, v, data in G.edges(data=True):
|
||||
confidence = data.get("confidence", "EXTRACTED")
|
||||
relation = data.get("relation", "")
|
||||
true_src = data.get("_src", u)
|
||||
true_tgt = data.get("_tgt", v)
|
||||
vis_edges.append({
|
||||
"from": u,
|
||||
"to": v,
|
||||
"from": true_src,
|
||||
"to": true_tgt,
|
||||
"label": relation,
|
||||
"title": _html.escape(f"{relation} [{confidence}]"),
|
||||
"dashes": confidence != "EXTRACTED",
|
||||
@@ -488,8 +550,7 @@ def to_html(
|
||||
<div id="legend-wrap">
|
||||
<h3>Communities</h3>
|
||||
<div id="legend-controls">
|
||||
<button onclick="toggleAllCommunities(false)">Show All</button>
|
||||
<button onclick="toggleAllCommunities(true)">Hide All</button>
|
||||
<label><input type="checkbox" id="select-all-cb" checked onchange="toggleAllCommunities(!this.checked)">Select All</label>
|
||||
</div>
|
||||
<div id="legend"></div>
|
||||
</div>
|
||||
|
||||
+75
-11
@@ -182,6 +182,7 @@ def _import_python(node, source: bytes, file_nid: str, stem: str, edges: list, s
|
||||
|
||||
|
||||
def _import_js(node, source: bytes, file_nid: str, stem: str, edges: list, str_path: str) -> None:
|
||||
resolved_path: "Path | None" = None
|
||||
for child in node.children:
|
||||
if child.type == "string":
|
||||
raw = _read_text(child, source).strip("'\"` ")
|
||||
@@ -197,6 +198,7 @@ def _import_js(node, source: bytes, file_nid: str, stem: str, edges: list, str_p
|
||||
elif resolved.suffix == ".jsx":
|
||||
resolved = resolved.with_suffix(".tsx")
|
||||
tgt_nid = _make_id(str(resolved))
|
||||
resolved_path = resolved
|
||||
else:
|
||||
# Check tsconfig.json path aliases (e.g. "@/" → "src/") before treating as external (#575)
|
||||
aliases = _load_tsconfig_aliases(Path(str_path).parent)
|
||||
@@ -208,6 +210,7 @@ def _import_js(node, source: bytes, file_nid: str, stem: str, edges: list, str_p
|
||||
break
|
||||
if resolved_alias is not None:
|
||||
tgt_nid = _make_id(str(resolved_alias))
|
||||
resolved_path = resolved_alias
|
||||
else:
|
||||
# Bare/scoped import (node_modules) - use last segment; dropped as external
|
||||
module_name = raw.split("/")[-1]
|
||||
@@ -225,6 +228,32 @@ def _import_js(node, source: bytes, file_nid: str, stem: str, edges: list, str_p
|
||||
})
|
||||
break
|
||||
|
||||
# Emit symbol-level edges for named imports from local/aliased files.
|
||||
# e.g. `import { Foo, type Bar } from './bar'` → file → Foo, file → Bar (EXTRACTED)
|
||||
# Uses the same _make_id(target_stem, name) key that _extract_generic emits when
|
||||
# defining the symbol, so these edges wire importers directly to existing symbol nodes.
|
||||
if resolved_path is not None:
|
||||
target_stem = _file_stem(resolved_path)
|
||||
line = node.start_point[0] + 1
|
||||
for child in node.children:
|
||||
if child.type == "import_clause":
|
||||
for sub in child.children:
|
||||
if sub.type == "named_imports":
|
||||
for spec in sub.children:
|
||||
if spec.type == "import_specifier":
|
||||
name_node = spec.child_by_field_name("name")
|
||||
if name_node:
|
||||
sym = _read_text(name_node, source)
|
||||
edges.append({
|
||||
"source": file_nid,
|
||||
"target": _make_id(target_stem, sym),
|
||||
"relation": "imports",
|
||||
"confidence": "EXTRACTED",
|
||||
"source_file": str_path,
|
||||
"source_location": f"L{line}",
|
||||
"weight": 1.0,
|
||||
})
|
||||
|
||||
|
||||
def _import_java(node, source: bytes, file_nid: str, stem: str, edges: list, str_path: str) -> None:
|
||||
def _walk_scoped(n) -> str:
|
||||
@@ -590,7 +619,11 @@ _KOTLIN_CONFIG = LanguageConfig(
|
||||
call_function_field="",
|
||||
call_accessor_node_types=frozenset({"navigation_expression"}),
|
||||
call_accessor_field="",
|
||||
name_fallback_child_types=("simple_identifier",),
|
||||
# Different tree-sitter-kotlin grammar versions name plain identifier
|
||||
# nodes differently: PyPI's `tree_sitter_kotlin` uses `identifier`,
|
||||
# older forks use `simple_identifier`. Accept both so the extractor
|
||||
# works across grammar generations.
|
||||
name_fallback_child_types=("simple_identifier", "identifier"),
|
||||
body_fallback_child_types=("function_body", "class_body"),
|
||||
function_boundary_types=frozenset({"function_declaration"}),
|
||||
import_handler=_import_kotlin,
|
||||
@@ -1069,15 +1102,19 @@ def _extract_generic(path: Path, config: LanguageConfig) -> dict:
|
||||
if sc.type == "simple_identifier":
|
||||
callee_name = _read_text(sc, source)
|
||||
elif config.ts_module == "tree_sitter_kotlin":
|
||||
# Kotlin: first child may be simple_identifier or navigation_expression
|
||||
# Kotlin: first child may be simple_identifier/identifier or
|
||||
# navigation_expression. PyPI's `tree_sitter_kotlin` produces
|
||||
# `identifier` for plain identifier nodes; older grammar
|
||||
# versions (including the JVM `io.github.bonede:tree-sitter-kotlin`
|
||||
# binding) produce `simple_identifier`. Accept both.
|
||||
first = node.children[0] if node.children else None
|
||||
if first:
|
||||
if first.type == "simple_identifier":
|
||||
if first.type in ("simple_identifier", "identifier"):
|
||||
callee_name = _read_text(first, source)
|
||||
elif first.type == "navigation_expression":
|
||||
is_member_call = True
|
||||
for child in reversed(first.children):
|
||||
if child.type == "simple_identifier":
|
||||
if child.type in ("simple_identifier", "identifier"):
|
||||
callee_name = _read_text(child, source)
|
||||
break
|
||||
elif config.ts_module == "tree_sitter_scala":
|
||||
@@ -2830,8 +2867,17 @@ def _resolve_cross_file_imports(
|
||||
stem = Path(src).stem
|
||||
label = node.get("label", "")
|
||||
nid = node.get("id", "")
|
||||
# Only index real classes/functions (not file nodes, not method stubs)
|
||||
if label and not label.endswith((")", ".py")) and "_" not in label[:1]:
|
||||
# Index class-level entities only. Function/method labels end in "()"
|
||||
# so are excluded by the `endswith(")")` filter; file nodes end in ".py";
|
||||
# private/internal labels start with "_"; rationale nodes carry
|
||||
# file_type=="rationale" and must never participate in cross-file
|
||||
# import resolution (#563).
|
||||
if (
|
||||
label
|
||||
and not label.endswith((")", ".py"))
|
||||
and "_" not in label[:1]
|
||||
and node.get("file_type") != "rationale"
|
||||
):
|
||||
stem_to_entities.setdefault(stem, {})[label] = nid
|
||||
|
||||
# Pass 2: for each file, find `from .X import A, B, C` and resolve
|
||||
@@ -2842,12 +2888,15 @@ def _resolve_cross_file_imports(
|
||||
stem = _file_stem(path)
|
||||
str_path = str(path)
|
||||
|
||||
# Find all classes defined in this file (the importers)
|
||||
# Find all classes defined in this file (the importers).
|
||||
# Excludes rationale nodes whose labels happen not to end in ")" or ".py"
|
||||
# but which must never be treated as importing entities (#563).
|
||||
local_classes = [
|
||||
n["id"] for n in file_result.get("nodes", [])
|
||||
if n.get("source_file") == str_path
|
||||
and not n["label"].endswith((")", ".py"))
|
||||
and n["id"] != _make_id(stem) # exclude file-level node
|
||||
and n.get("file_type") != "rationale"
|
||||
]
|
||||
if not local_classes:
|
||||
continue
|
||||
@@ -3697,12 +3746,21 @@ def extract(
|
||||
# Cross-file call resolution for all languages
|
||||
# Each extractor saved unresolved calls in raw_calls. Now that we have all
|
||||
# nodes from all files, resolve any callee that exists in another file.
|
||||
global_label_to_nid: dict[str, str] = {}
|
||||
# Build name → ALL matching node IDs so we can skip ambiguous common names
|
||||
# (e.g. "log", "execute", "find") that appear in multiple files — resolving
|
||||
# those inflates god_nodes ranking with spurious cross-file edges.
|
||||
# Build label -> node_id index for cross-file call resolution.
|
||||
# Skip rationale nodes (their labels are docstring text, not callable
|
||||
# identifiers, and they were polluting matches for short names — #563).
|
||||
global_label_to_nids: dict[str, list[str]] = {}
|
||||
for n in all_nodes:
|
||||
if n.get("file_type") == "rationale":
|
||||
continue
|
||||
raw = n.get("label", "")
|
||||
normalised = raw.strip("()").lstrip(".")
|
||||
if normalised:
|
||||
global_label_to_nid[normalised.lower()] = n["id"]
|
||||
key = normalised.lower()
|
||||
global_label_to_nids.setdefault(key, []).append(n["id"])
|
||||
|
||||
existing_pairs = {(e["source"], e["target"]) for e in all_edges}
|
||||
for result in per_file:
|
||||
@@ -3714,9 +3772,15 @@ def extract(
|
||||
# and collides with any top-level function named "log" in the corpus.
|
||||
if rc.get("is_member_call"):
|
||||
continue
|
||||
tgt = global_label_to_nid.get(callee.lower())
|
||||
candidates = global_label_to_nids.get(callee.lower(), [])
|
||||
# Skip ambiguous names that resolve to multiple nodes — these are
|
||||
# common short names (log, execute, find) with no import evidence
|
||||
# to pick the right target; emitting all edges inflates god_nodes.
|
||||
if len(candidates) != 1:
|
||||
continue
|
||||
tgt = candidates[0]
|
||||
caller = rc["caller_nid"]
|
||||
if tgt and tgt != caller and (caller, tgt) not in existing_pairs:
|
||||
if tgt != caller and (caller, tgt) not in existing_pairs:
|
||||
existing_pairs.add((caller, tgt))
|
||||
all_edges.append({
|
||||
"source": caller,
|
||||
|
||||
+21
-8
@@ -61,7 +61,13 @@ fi
|
||||
|
||||
""" + _PYTHON_DETECT + """
|
||||
export GRAPHIFY_CHANGED="$CHANGED"
|
||||
$GRAPHIFY_PYTHON -c "
|
||||
|
||||
# Run rebuild detached so git commit returns immediately.
|
||||
# Full repo rebuilds can take hours; blocking the post-commit hook stalls the shell.
|
||||
_GRAPHIFY_LOG="${HOME}/.cache/graphify-rebuild.log"
|
||||
mkdir -p "$(dirname "$_GRAPHIFY_LOG")"
|
||||
echo "[graphify hook] launching background rebuild (log: $_GRAPHIFY_LOG)"
|
||||
nohup $GRAPHIFY_PYTHON -c "
|
||||
import os, sys
|
||||
from pathlib import Path
|
||||
|
||||
@@ -74,12 +80,15 @@ if not changed:
|
||||
print(f'[graphify hook] {len(changed)} file(s) changed - rebuilding graph...')
|
||||
|
||||
try:
|
||||
import os as _os
|
||||
from graphify.watch import _rebuild_code
|
||||
_rebuild_code(Path('.'))
|
||||
_force = _os.environ.get('GRAPHIFY_FORCE', '').lower() in ('1', 'true', 'yes')
|
||||
_rebuild_code(Path('.'), force=_force)
|
||||
except Exception as exc:
|
||||
print(f'[graphify hook] Rebuild failed: {exc}')
|
||||
sys.exit(1)
|
||||
"
|
||||
" > "$_GRAPHIFY_LOG" 2>&1 < /dev/null &
|
||||
disown 2>/dev/null || true
|
||||
# graphify-hook-end
|
||||
"""
|
||||
|
||||
@@ -111,17 +120,21 @@ GIT_DIR=$(git rev-parse --git-dir 2>/dev/null)
|
||||
[ -f "$GIT_DIR/CHERRY_PICK_HEAD" ] && exit 0
|
||||
|
||||
""" + _PYTHON_DETECT + """
|
||||
echo "[graphify] Branch switched - rebuilding knowledge graph (code files)..."
|
||||
$GRAPHIFY_PYTHON -c "
|
||||
_GRAPHIFY_LOG="${HOME}/.cache/graphify-rebuild.log"
|
||||
mkdir -p "$(dirname "$_GRAPHIFY_LOG")"
|
||||
echo "[graphify] Branch switched - launching background rebuild (log: $_GRAPHIFY_LOG)"
|
||||
nohup $GRAPHIFY_PYTHON -c "
|
||||
from graphify.watch import _rebuild_code
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import os, sys
|
||||
try:
|
||||
_rebuild_code(Path('.'))
|
||||
_force = os.environ.get('GRAPHIFY_FORCE', '').lower() in ('1', 'true', 'yes')
|
||||
_rebuild_code(Path('.'), force=_force)
|
||||
except Exception as exc:
|
||||
print(f'[graphify] Rebuild failed: {exc}')
|
||||
sys.exit(1)
|
||||
"
|
||||
" > "$_GRAPHIFY_LOG" 2>&1 < /dev/null &
|
||||
disown 2>/dev/null || true
|
||||
# graphify-checkout-hook-end
|
||||
"""
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -826,7 +826,7 @@ G = json_graph.node_link_graph(data, edges='links')
|
||||
|
||||
detection = {'total_files': 0, 'total_words': 99999, 'needs_graph': True, 'warning': None,
|
||||
'files': {'code': [], 'document': [], 'paper': []}}
|
||||
tokens = {'input': 0, 'output':': 0}
|
||||
tokens = {'input': 0, 'output': 0}
|
||||
|
||||
communities = cluster(G)
|
||||
cohesion = score_all(G, communities)
|
||||
|
||||
@@ -279,10 +279,16 @@ If a file has YAML frontmatter (--- ... ---), copy source_url, captured_at, auth
|
||||
|
||||
confidence_score is REQUIRED on every edge - never omit it, never use 0.5 as a default:
|
||||
- EXTRACTED edges: confidence_score = 1.0 always
|
||||
- INFERRED edges: reason about each edge individually.
|
||||
Direct structural evidence (shared data structure, clear dependency): 0.8-0.9.
|
||||
Reasonable inference with some uncertainty: 0.6-0.7.
|
||||
Weak or speculative: 0.4-0.5. Most edges should be 0.6-0.9, not 0.5.
|
||||
- INFERRED edges: pick exactly ONE value from this set — never 0.5:
|
||||
0.95 direct structural evidence (shared data structure, named cross-file reference).
|
||||
0.85 strong inference (clear functional alignment, no direct symbol link).
|
||||
0.75 reasonable inference (shared problem domain + similar shape, requires interpretation).
|
||||
0.65 weak inference (thematically related, no shape evidence).
|
||||
0.55 speculative but plausible (surface-level co-occurrence only).
|
||||
Models follow discrete rubrics better than continuous ranges; the bimodal
|
||||
distribution observed in production (>50% at 0.5, >40% at 0.85+) shows the
|
||||
range guidance is being collapsed to a binary. If no value above fits, mark
|
||||
the edge AMBIGUOUS rather than picking 0.4 or below.
|
||||
- AMBIGUOUS edges: 0.1-0.3
|
||||
|
||||
Output exactly this JSON (no other text):
|
||||
@@ -785,9 +791,28 @@ G_existing = json_graph.node_link_graph(existing_data, edges='links')
|
||||
new_extraction = json.loads(Path('.graphify_extract.json').read_text())
|
||||
G_new = build_from_json(new_extraction)
|
||||
|
||||
# Prune nodes from deleted files
|
||||
incremental = json.loads(Path('.graphify_incremental.json').read_text())
|
||||
deleted = set(incremental.get('deleted_files', []))
|
||||
if deleted:
|
||||
to_remove = [n for n, d in G_existing.nodes(data=True) if d.get('source_file') in deleted]
|
||||
G_existing.remove_nodes_from(to_remove)
|
||||
if to_remove:
|
||||
print(f'Pruned {len(to_remove)} ghost node(s) from {len(deleted)} deleted file(s) — drift detected and corrected.')
|
||||
else:
|
||||
print(f'{len(deleted)} file(s) deleted since last run, but no ghost nodes were present in the graph — no drift.')
|
||||
|
||||
# Merge: new nodes/edges into existing graph
|
||||
G_existing.update(G_new)
|
||||
print(f'Merged: {G_existing.number_of_nodes()} nodes, {G_existing.number_of_edges()} edges')
|
||||
|
||||
# Save manifest with the CURRENT full file list so the next --update
|
||||
# diffs against today's filesystem state, not the prior --update's
|
||||
# baseline. Without this, deleted files get reported as ghosts again
|
||||
# on every subsequent --update until a full rebuild runs.
|
||||
from graphify.detect import save_manifest
|
||||
save_manifest(incremental['files'])
|
||||
print('[graphify update] Manifest saved.')
|
||||
"
|
||||
```
|
||||
|
||||
|
||||
+22
-5
@@ -337,10 +337,16 @@ If a file has YAML frontmatter (--- ... ---), copy source_url, captured_at, auth
|
||||
|
||||
confidence_score is REQUIRED on every edge - never omit it, never use 0.5 as a default:
|
||||
- EXTRACTED edges: confidence_score = 1.0 always
|
||||
- INFERRED edges: reason about each edge individually.
|
||||
Direct structural evidence (shared data structure, clear dependency): 0.8-0.9.
|
||||
Reasonable inference with some uncertainty: 0.6-0.7.
|
||||
Weak or speculative: 0.4-0.5. Most edges should be 0.6-0.9, not 0.5.
|
||||
- INFERRED edges: pick exactly ONE value from this set — never 0.5:
|
||||
0.95 direct structural evidence (shared data structure, named cross-file reference).
|
||||
0.85 strong inference (clear functional alignment, no direct symbol link).
|
||||
0.75 reasonable inference (shared problem domain + similar shape, requires interpretation).
|
||||
0.65 weak inference (thematically related, no shape evidence).
|
||||
0.55 speculative but plausible (surface-level co-occurrence only).
|
||||
Models follow discrete rubrics better than continuous ranges; the bimodal
|
||||
distribution observed in production (>50% at 0.5, >40% at 0.85+) shows the
|
||||
range guidance is being collapsed to a binary. If no value above fits, mark
|
||||
the edge AMBIGUOUS rather than picking 0.4 or below.
|
||||
- AMBIGUOUS edges: 0.1-0.3
|
||||
|
||||
Node ID format: lowercase, only `[a-z0-9_]`, no dots or slashes. Format: `{stem}_{entity}` where stem is the filename without extension and entity is the symbol name, both normalized (lowercase, non-alphanumeric chars replaced with `_`). Example: `src/auth/session.py` + `ValidateToken` → `session_validatetoken`. This must match the ID the AST extractor generates so cross-references between code and semantic nodes connect correctly. CRITICAL: never append chunk numbers, sequence numbers, or any suffix to an ID (no `_c1`, `_c2`, `_chunk2`, etc.). IDs must be deterministic from the label alone — the same entity must always produce the same ID regardless of which chunk processes it.
|
||||
@@ -923,7 +929,10 @@ deleted = set(incremental.get('deleted_files', []))
|
||||
if deleted:
|
||||
to_remove = [n for n, d in G_existing.nodes(data=True) if d.get('source_file') in deleted]
|
||||
G_existing.remove_nodes_from(to_remove)
|
||||
print(f'Pruned {len(to_remove)} ghost nodes from {len(deleted)} deleted file(s)')
|
||||
if to_remove:
|
||||
print(f'Pruned {len(to_remove)} ghost node(s) from {len(deleted)} deleted file(s) — drift detected and corrected.')
|
||||
else:
|
||||
print(f'{len(deleted)} file(s) deleted since last run, but no ghost nodes were present in the graph — no drift.')
|
||||
|
||||
# Merge: new nodes/edges into existing graph
|
||||
G_existing.update(G_new)
|
||||
@@ -939,6 +948,14 @@ merged_out = {
|
||||
}
|
||||
Path('graphify-out/.graphify_extract.json').write_text(json.dumps(merged_out))
|
||||
print(f'[graphify update] Merged extraction written ({len(merged_out[\"nodes\"])} nodes, {len(merged_out[\"edges\"])} edges)')
|
||||
|
||||
# Save manifest with the CURRENT full file list so the next --update
|
||||
# diffs against today's filesystem state, not the prior --update's
|
||||
# baseline. Without this, deleted files get reported as ghosts again
|
||||
# on every subsequent --update until a full rebuild runs.
|
||||
from graphify.detect import save_manifest
|
||||
save_manifest(incremental['files'])
|
||||
print('[graphify update] Manifest saved.')
|
||||
"
|
||||
```
|
||||
|
||||
|
||||
@@ -58,7 +58,7 @@ def download_audio(url: str, output_dir: Path) -> Path:
|
||||
|
||||
# yt-dlp uses %(title)s which can be long/weird — use a stable name based on URL hash
|
||||
import hashlib
|
||||
url_hash = hashlib.sha1(url.encode()).hexdigest()[:12]
|
||||
url_hash = hashlib.sha1(url.encode(), usedforsecurity=False).hexdigest()[:12]
|
||||
out_template = str(output_dir / f"yt_{url_hash}.%(ext)s")
|
||||
|
||||
# Check for already-downloaded file
|
||||
|
||||
+19
-10
@@ -33,9 +33,13 @@ def _relativize_source_files(payload: dict, root: Path) -> None:
|
||||
continue
|
||||
|
||||
|
||||
def _rebuild_code(watch_path: Path, *, follow_symlinks: bool = False) -> bool:
|
||||
def _rebuild_code(watch_path: Path, *, follow_symlinks: bool = False, force: bool = False) -> bool:
|
||||
"""Re-run AST extraction + build + cluster + report for code files. No LLM needed.
|
||||
|
||||
When ``force`` is True the node-count safety check in ``to_json`` is bypassed
|
||||
so the rebuilt graph overwrites graph.json even if it has fewer nodes.
|
||||
Use this after refactors that legitimately delete code.
|
||||
|
||||
Returns True on success, False on error.
|
||||
"""
|
||||
watch_root = watch_path.resolve()
|
||||
@@ -60,20 +64,25 @@ def _rebuild_code(watch_path: Path, *, follow_symlinks: bool = False) -> bool:
|
||||
result = extract(code_files, cache_root=watch_root)
|
||||
|
||||
# Preserve semantic nodes/edges from a previous full run.
|
||||
# AST-only rebuild replaces code nodes; doc/paper/image nodes are kept.
|
||||
# AST-only rebuild replaces nodes for changed files; everything else is kept.
|
||||
# Filter by node ID membership in the new AST output, not by file_type —
|
||||
# INFERRED/AMBIGUOUS nodes extracted from code files also carry file_type="code"
|
||||
# and would be wrongly dropped by a file_type-based filter.
|
||||
out = watch_path / "graphify-out"
|
||||
existing_graph = out / "graph.json"
|
||||
if existing_graph.exists():
|
||||
try:
|
||||
existing = json.loads(existing_graph.read_text(encoding="utf-8"))
|
||||
code_ids = {n["id"] for n in existing.get("nodes", []) if n.get("file_type") == "code"}
|
||||
sem_nodes = [n for n in existing.get("nodes", []) if n.get("file_type") != "code"]
|
||||
sem_edges = [e for e in existing.get("links", existing.get("edges", []))
|
||||
if e.get("confidence") in ("INFERRED", "AMBIGUOUS")
|
||||
or (e.get("source") not in code_ids and e.get("target") not in code_ids)]
|
||||
new_ast_ids = {n["id"] for n in result["nodes"]}
|
||||
preserved_nodes = [n for n in existing.get("nodes", []) if n["id"] not in new_ast_ids]
|
||||
all_ids = new_ast_ids | {n["id"] for n in preserved_nodes}
|
||||
preserved_edges = [
|
||||
e for e in existing.get("links", existing.get("edges", []))
|
||||
if e.get("source") in all_ids and e.get("target") in all_ids
|
||||
]
|
||||
result = {
|
||||
"nodes": result["nodes"] + sem_nodes,
|
||||
"edges": result["edges"] + sem_edges,
|
||||
"nodes": result["nodes"] + preserved_nodes,
|
||||
"edges": result["edges"] + preserved_edges,
|
||||
"hyperedges": existing.get("hyperedges", []),
|
||||
"input_tokens": 0,
|
||||
"output_tokens": 0,
|
||||
@@ -100,7 +109,7 @@ def _rebuild_code(watch_path: Path, *, follow_symlinks: bool = False) -> bool:
|
||||
out.mkdir(exist_ok=True)
|
||||
(out / ".graphify_root").write_text(str(watch_root), encoding="utf-8")
|
||||
|
||||
json_written = to_json(G, communities, str(out / "graph.json"))
|
||||
json_written = to_json(G, communities, str(out / "graph.json"), force=force)
|
||||
if not json_written:
|
||||
return False
|
||||
|
||||
|
||||
+21
-1
@@ -7,7 +7,18 @@ import networkx as nx
|
||||
|
||||
|
||||
def _safe_filename(name: str) -> str:
|
||||
return name.replace("/", "-").replace(" ", "_").replace(":", "-")
|
||||
"""Make a label safe for use as a filename across platforms.
|
||||
|
||||
Substitutes characters that Windows reserves in filenames
|
||||
(< > : " / \\ | ? *) and strips trailing dots/spaces, also reserved.
|
||||
Falls back to 'unnamed' for empty results and caps length at 200
|
||||
chars to stay well under common filesystem limits.
|
||||
"""
|
||||
import re
|
||||
s = name.replace("/", "-").replace(" ", "_").replace(":", "-")
|
||||
s = re.sub(r'[<>:"/\\|?*]', '_', s)
|
||||
s = s.strip('. ')
|
||||
return s[:200] if s else 'unnamed'
|
||||
|
||||
|
||||
def _cross_community_links(G: nx.Graph, nodes: list[str], own_cid: int, labels: dict[int, str]) -> list[tuple[str, int]]:
|
||||
@@ -185,6 +196,15 @@ def to_wiki(
|
||||
out = Path(output_dir)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Clear stale .md files from previous runs to prevent orphan accumulation.
|
||||
# Community labels are LLM-generated (per skill.md Step 5) and non-deterministic
|
||||
# across runs — the same conceptual community may be named differently each time
|
||||
# (e.g. "AutoAgent Skills" → "AutoAgent Methodology"), leaving the previous file
|
||||
# as an orphan. Since to_wiki() owns wiki/ entirely (always writes the full set),
|
||||
# it can safely clear .md files at the start of each call.
|
||||
for old_article in out.glob("*.md"):
|
||||
old_article.unlink()
|
||||
|
||||
labels = community_labels or {cid: f"Community {cid}" for cid in communities}
|
||||
cohesion = cohesion or {}
|
||||
god_nodes_data = god_nodes_data or []
|
||||
|
||||
+7
-7
@@ -4,11 +4,11 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "graphifyy"
|
||||
version = "0.6.2"
|
||||
description = "AI coding assistant skill (Claude Code, Codex, OpenCode, Cursor, Gemini CLI, Aider, OpenClaw, Factory Droid, Trae, Hermes, Kiro, Google Antigravity) - turn any folder of code, docs, papers, images, or videos into a queryable knowledge graph"
|
||||
version = "0.6.6"
|
||||
description = "AI coding assistant skill (Claude Code, Codex, OpenCode, Cursor, Gemini CLI, Aider, OpenClaw, Factory Droid, Trae, Hermes, Kiro, Pi, Google Antigravity) - turn any folder of code, docs, papers, images, or videos into a queryable knowledge graph"
|
||||
readme = "README.md"
|
||||
license = { file = "LICENSE" }
|
||||
keywords = ["claude", "claude-code", "codex", "opencode", "cursor", "gemini", "aider", "kiro", "knowledge-graph", "rag", "graphrag", "obsidian", "community-detection", "tree-sitter", "leiden", "llm"]
|
||||
keywords = ["claude", "claude-code", "codex", "opencode", "cursor", "gemini", "aider", "kiro", "pi", "knowledge-graph", "rag", "graphrag", "obsidian", "community-detection", "tree-sitter", "leiden", "llm"]
|
||||
requires-python = ">=3.10"
|
||||
dependencies = [
|
||||
"networkx",
|
||||
@@ -62,9 +62,9 @@ graphify = "graphify.__main__:main"
|
||||
# Run without installing: uvx graphifyy install
|
||||
package = true
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["."]
|
||||
include = ["graphify*"]
|
||||
[tool.setuptools]
|
||||
packages = ["graphify"]
|
||||
include-package-data = false
|
||||
|
||||
[tool.setuptools.package-data]
|
||||
graphify = ["skill.md", "skill-codex.md", "skill-opencode.md", "skill-aider.md", "skill-copilot.md", "skill-claw.md", "skill-windows.md", "skill-droid.md", "skill-trae.md", "skill-kiro.md", "skill-vscode.md"]
|
||||
graphify = ["skill.md", "skill-codex.md", "skill-opencode.md", "skill-aider.md", "skill-copilot.md", "skill-claw.md", "skill-windows.md", "skill-droid.md", "skill-trae.md", "skill-kiro.md", "skill-vscode.md", "skill-pi.md"]
|
||||
|
||||
@@ -168,3 +168,25 @@ def test_calls_deduplication():
|
||||
result = extract_python(FIXTURES / "sample_calls.py")
|
||||
call_pairs = [(e["source"], e["target"]) for e in result["edges"] if e["relation"] == "calls"]
|
||||
assert len(call_pairs) == len(set(call_pairs)), "Duplicate calls edges found"
|
||||
|
||||
|
||||
def test_cross_file_calls_skip_ambiguous_duplicate_labels(tmp_path):
|
||||
"""Unqualified cross-file calls must not guess between duplicate helper names."""
|
||||
caller = tmp_path / "caller.py"
|
||||
helper_a = tmp_path / "a.py"
|
||||
helper_b = tmp_path / "b.py"
|
||||
caller.write_text("def run():\n log()\n")
|
||||
helper_a.write_text("def log():\n return 'a'\n")
|
||||
helper_b.write_text("def log():\n return 'b'\n")
|
||||
|
||||
result = extract([caller, helper_a, helper_b], cache_root=tmp_path)
|
||||
nodes = {n["id"]: n for n in result["nodes"]}
|
||||
calls = [
|
||||
e for e in result["edges"]
|
||||
if e["relation"] == "calls" and e["confidence"] == "INFERRED"
|
||||
]
|
||||
|
||||
assert not any(
|
||||
nodes[e["source"]]["label"] == "run()" and nodes[e["target"]]["label"] == "log()"
|
||||
for e in calls
|
||||
)
|
||||
|
||||
@@ -188,6 +188,18 @@ def test_kotlin_finds_function():
|
||||
r = extract_kotlin(FIXTURES / "sample.kt")
|
||||
assert any("createClient" in l for l in _labels(r))
|
||||
|
||||
def test_kotlin_emits_in_file_calls():
|
||||
"""Regression test for the call-walker `simple_identifier` /
|
||||
`identifier` rename — see graphify-kmp's PythonParityTest."""
|
||||
r = extract_kotlin(FIXTURES / "sample.kt")
|
||||
calls = _calls(r)
|
||||
# In sample.kt: get() and post() both call buildRequest(), and
|
||||
# createClient() invokes Config and HttpClient (constructor calls).
|
||||
assert (".get()", ".buildRequest()") in calls
|
||||
assert (".post()", ".buildRequest()") in calls
|
||||
assert ("createClient()", "Config") in calls
|
||||
assert ("createClient()", "HttpClient") in calls
|
||||
|
||||
|
||||
# ── Scala ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
Reference in New Issue
Block a user