Graph queries filtered every token with len > 2, which dropped common two-character Chinese search terms while trying to suppress short English noise. Centralize query token selection and apply the length gate only to pure-English tokens so mixed or non-English terms remain searchable.
Constraint: Issue #962 reports space-separated Chinese query terms such as 前端, 依赖, and 安装 are lost by graphify query.
Rejected: Add Chinese segmentation now | the reported failure is fixed by preserving existing space-separated non-English tokens without expanding query behavior.
Confidence: high
Scope-risk: narrow
Directive: Keep CLI, MCP query, and benchmark query tokenization on one helper when changing query-term rules.
Tested: uv run --with pytest pytest tests/test_serve.py tests/test_query_cli.py tests/test_benchmark.py
Tested: graphify update .
Not-tested: Full test suite.
Co-authored-by: OmX <omx@oh-my-codex.dev>
tree-sitter-swift parses both `class Foo` and `extension Foo` as
`class_declaration`, and node ids carry the file stem, so `extension Foo`
in a sibling file produced a second `Foo` node instead of attaching to
the original. Same-file extensions already dedupe via seen_ids; only the
cross-file case leaked.
Per-file extraction now tags `extension` class_declarations, and the
corpus-level `extract()` runs a merge pass: when exactly one
non-extension declaration shares the label, the extension nodes redirect
onto it and their edges are rewritten (self-loops dropped, duplicates
collapsed). Extensions of types outside the corpus and ambiguous label
matches stay untouched.
On a 25-file Swift project this collapses Parser from 6 split nodes
(top of the god-node list, four entries) to one canonical node, and
lets the generic cross-file call resolver attach previously ambiguous
call edges to the right target.
Backends routed through _call_openai_compat (gemini, openai, kimi,
deepseek, ollama) silently ignored the documented env override when
their backend config dict carried a hardcoded max_completion_tokens.
The dispatcher used:
cfg.get("max_completion_tokens", max_out)
which always returned the config-dict value when the key was present,
shadowing the env-var-resolved max_out.
For gemini specifically, the hardcoded cap of 16384 truncated
extracted-graph JSON mid-response on multi-document chunks (~17 specs
of 100-1500 lines each pushing the output past 16k tokens). Symptom:
cascading 'LLM returned invalid JSON, skipping chunk: Unterminated
string at column 4XXXX' followed by bisect-retry storms that bill
input tokens without producing graph nodes.
Fix: route the same _resolve_max_tokens(...) call that the Claude and
Bedrock paths already use, so the override applies uniformly across
backends.
Verified with gemini-2.5-pro over a 20-doc / 76k-input-token chunk:
output of 36008 tokens emitted without truncation, producing 193
nodes / 223 edges / 23 communities in a single chunk.
- fix SQL extractor using bare path.stem as node ID prefix — collides across same-named files in different dirs; use _file_stem() (directory-qualified) instead
- fix Python import resolver keying stem_to_entities by bare stem; add bare_to_qualified secondary index for absolute imports so cross-file edges survive duplicate filenames
- add stat-based mtime fastpath to file_hash: skip full SHA256 when size+mtime_ns unchanged, flush index atomically via atexit (same trade-off as make)
- add cache-check, merge-chunks, merge-semantic CLI subcommands so the skill pipeline can use library functions instead of inline Python
- fix absolute source_file paths from semantic subagents not being relativized before graph storage (#932): add root param to build_from_json/build/build_merge, pass scan target at both call sites
- fix failed semantic chunks permanently freezing their files in the manifest (#933): filter _manifest_files to only include doc/paper/image files that appear in sem_result nodes/edges before calling save_manifest
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
detect: add scan_root to return dict so skill can strip absolute prefix
when computing relative subdirectory breakdown; remove stale --no-semantic
flag reference from large-corpus warning (flag does not exist)
skill: clarify fast path checks CWD graphify-out/graph.json (project root);
remove hardcoded --backend gemini from multi-subfolder example — users
should pass whichever backend key they have; expand large-corpus gate
instruction to use scan_root for relative paths, filter graphify-out/
converted sidecars, and handle flat repos with no subdirectories
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fast path: if graphify-out/graph.json exists and user is asking a question
(not an explicit rebuild), skip detect entirely and run graphify query —
prevents the skill from refusing large already-built corpora (#930)
- Raise FILE_COUNT_UPPER 200 → 500 so typical 200-500 file codebases no longer
hit the large-corpus size gate on fresh extraction (#930)
- Subdirectory breakdown now strips the scan-root prefix so agent shows
relative names (core/, service/) not absolute paths rooted at / (#930)
- Document multi-subfolder CLI pattern: graphify extract ./sub/ places
graphify-out/ inside each subfolder; skill clobbers single root graphify-out
when run on subfolders separately (#930)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Security: _is_sensitive now flags underscore-prefixed names (api_token.txt, oauth_token.json) by replacing \b with lookarounds; adds _SENSITIVE_DIRS check on parent path components (parts[:-1]) so .ssh/, secrets/, .aws/ directories are always skipped; aligns both patterns to (?![a-zA-Z]) for consistent underscore-after-keyword behavior (#920)
Fix: --wiki Relationships section always empty because _cross_community_links read community from node attrs (always None) instead of the communities dict; _god_node_article had the same bug and never linked to the owning community; fixed by building a node->community map in to_wiki() and threading it through (#925)
Fix: --watch now respects .graphifyignore; patterns loaded once at startup, handler checks _is_ignored before extension filter so node_modules/, .venv/, build/ churn no longer triggers rebuilds (#928)
Fix: C++ struct inheritance edges via base_class_clause; initialize base="" per iteration to prevent stale carryover (#915)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The OpenAI-compatible API can return HTTP 200 with an empty `choices`
list or with `choices[0].message = None` (e.g. content-filtered
responses on Gemini, overwhelmed Ollama instances). Without a guard,
both sites raise an unhandled IndexError or AttributeError.
`_call_openai_compat` already documents this hazard ("Ollama can return
HTTP 200 with empty/null content") and has `_response_is_hollow` logic
downstream, but `_response_is_hollow` is unreachable when the choices
list itself is empty. The new guard closes that gap.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
- hooks.py: drop --path-format=absolute (added git 2.31), validate no newlines in path, anchor relative paths on repo root (#907)
- detect.py: seed save_manifest from existing manifest before loop so incremental callers don't erase untouched file entries (#917)
- cluster.py: drop round(..., 2) from cohesion_score so split threshold 0.05 fires correctly; add resolution param to _partition and cluster; add exclude_hubs_percentile to cluster with majority-vote reattachment (#919)
- report.py: format cohesion with :.2f for display
- __main__.py: wire --resolution and --exclude-hubs into extract and cluster-only commands (#919)
- C++ inheritance already written to disk by analysis agent (#915)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(analyze): exclude npm dep-block keys from god-node selection
Extends _JSON_NOISE_LABELS in graphify/analyze.py with the six npm
package.json dependency-block keys (dependencies, devDependencies,
peerDependencies, optionalDependencies, bundledDependencies,
bundleDependencies — lowercased to match the existing .strip().lower()
comparison in _is_json_key_node).
On JS/TS corpora with non-trivial dependency counts, the dep-block key
node accumulates contains+imports edges to every package entry and was
surfacing as the top god-node in the report. The fix is a one-line
extension of the existing frozenset; no new helpers or code paths.
Includes a parametrized regression test in tests/test_analyze.py
covering all five npm keys: dependencies, devDependencies,
peerDependencies, optionalDependencies, bundledDependencies.
Validated live against rsl-siege-manager @ 6085fd66 — zero npm dep-block
keys in the top-10 god-nodes after the fix. Pre-existing failures
(Windows symlink + SQL tree-sitter): 17, unchanged.
Closes#2
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: drop fork-local issue refs from analyze comments
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(extract): force UTF-8 encoding on subprocess + loud failure on chunk errors
On Windows cp1252, subprocess.run(..., text=True) without encoding= raises
UnicodeEncodeError for chars like → ✅ ≥ in chunk content. Both
_call_claude_cli (llm.py:426) and the _call_llm claude-cli branch (llm.py:959)
lacked encoding=.
- Add encoding="utf-8" to both subprocess.run sites.
- Track failed_chunks in extract_corpus_parallel merged result dict.
- Print [graphify] WARNING: N/M semantic chunk(s) failed summary to stderr at
end of run when any chunk failed, so silent partial failures are visible.
- Add tests/test_charmap_encoding.py: 10 regression tests covering subprocess
encoding kwarg, loud-failure summary, and substitution validation.
Closes#3
* style: drop fork-local issue refs from llm/test comments
- Revert .h -> extract_c (C++ grammar rejects C++ keywords used as identifiers
in Linux-kernel-style headers; .hpp/.hxx/.hh already route to extract_cpp)
- Fix field_declaration block: use children_by_field_name("declarator") instead
of iterating all children with wrong type guard; replace ensure_node (undefined)
with add_node
- Fix _import_c include resolution: use _make_id(str(resolved)) to match the
file_nid scheme _extract_generic uses, not _make_id(_file_stem(resolved))
- Fix exact_merges counter in dedup Pass 1 to count only within-file merges
actually performed, not the raw unpartitioned group sizes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Route .h files through extract_cpp (was extract_c), fixing missing method nodes in C++ headers
- Extend _get_cpp_func_name to handle field_identifier, destructor_name, operator_name
- Add CPP-specific field_declaration branch in _extract_generic to emit class method/field nodes
- Partition dedup Pass 1 by source_file: only union same-label nodes within the same file;
cross-file matches fall through to Pass 2 fuzzy, preventing generic-label god nodes
- Add _resolve_c_include_path: resolve quoted #include paths to real files on disk so
target node IDs match what _extract_generic creates, fixing dangling include edges
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Common terms like 'error'/'exception' were stealing BFS seed slots from
rare identifiers like 'FooBarService', burning the token budget on noise.
- _compute_idf: weights query terms by inverse document frequency, cached
on G.graph so cost is paid once per graph load not per query
- _score_nodes: multiplies each tier bonus by IDF weight
- _pick_seeds: replaces fixed top-3 with gap-ratio selection — stops adding
seeds when score drops below 20% of the top match
- _subgraph_to_text: truncation hint now tells Claude to narrow with
context_filter or use get_node instead of just saying 'truncated'
Fixes#897
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Clean merge — fixes real token-bloat regression where GRAPH_REPORT.md was loaded on every question. All 10 install surfaces now route to graphify query first. Also fixes hidden bug where re-running graphify install after upgrade didn't refresh stale hook text.
Clean merge — fixes real silent data-loss bug in --update on symlinked repos, backward-compatible, inherits existing cycle protection. Follow-up: consider tightening _auto_follow_symlinks to is_symlink() and is_dir() to avoid tripping on incidental symlinked files at root.
The current install writes "ALWAYS read graphify-out/GRAPH_REPORT.md
before reading any source files, running grep/glob searches, or
answering codebase questions" into CLAUDE.md and equivalents, plus a
PreToolUse hook with the same instruction. On real corpora that report
is 47-91K characters, so Claude Code sessions pay roughly 12-25K tokens
of context up front for every search-able question. Three users on #580
reported this making token usage worse than no install at all.
Reproduced on a 1500-file Go monorepo: a "where is X defined" question
takes 5 tool calls and 34k agent tokens with stock graphify, 4 calls
and 30k tokens with no install, and 1 call and 30k tokens after this
patch. Stock graphify's Read of GRAPH_REPORT.md hit Claude Code's 25k
token cap and failed entirely, then recovered via a partial read plus
graphify explain.
Demote GRAPH_REPORT.md to a fallback for broad architecture review and
route first action to the existing scoped commands: graphify query,
path, explain. The 2k-budget BFS subgraph already exists in serve.py;
the install just wasn't pointing at it.
Updated across all ten install surfaces: _SETTINGS_HOOK,
_CLAUDE_MD_SECTION, _AGENTS_MD_SECTION, _GEMINI_MD_SECTION,
_GEMINI_HOOK, _VSCODE_INSTRUCTIONS_SECTION, _ANTIGRAVITY_RULES,
_KIRO_STEERING, _CURSOR_RULE, _OPENCODE_PLUGIN_JS. Plus the matching
sentence in README.md, which also fixes an inaccuracy about Codex
hooks (Codex's installed hook is intentionally a no-op because Codex
rejects additionalContext, so the guidance there comes from AGENTS.md,
not the hook).
Five installers (claude, agents, vscode, gemini, kiro, cursor) were
also writing their section only when no marker was present, so users
who installed pre-fix kept the old "ALWAYS read" text after upgrading.
Added _replace_or_append_section helper that updates in place when the
graphify marker is found. claude_install also no longer returns before
re-running _install_claude_hook, so stale settings.json hook payloads
get refreshed on upgrade.
Tests:
- tests/test_install_strings.py (3): every install constant still
mentions `graphify query` and matches no banned report-first regex.
- tests/test_install_upgrade.py (7): seeds each platform's instruction
file with pre-fix text, runs install, asserts the on-disk file
reflects the new policy.
- test_claude_md.py idempotency tests still pass.
Fixes#580.
Run with:
uv run python -m nuitka --project --mode=onefile \
--output-dir=build --output-filename=graphify \
--assume-yes-for-downloads
Produces build/graphify, a self-contained ~60 MB binary that bundles
the package's skill *.md files (picked up automatically from
[tool.setuptools.package-data] via Nuitka's --project flow). The
existing wheel-build path (uv build via setuptools.build_meta) is
unchanged.
[tool.nuitka] config in pyproject.toml is not used: in --project mode
Nuitka builds the option list from setuptools distribution metadata
(_dumpBuildConfiguration in nuitka/distutils/DistutilsCommands.py)
and never reads [tool.nuitka], so output-dir / output-filename / etc.
must be passed on the CLI.
patchelf is required at build time on Linux and macOS for Nuitka's
standalone/onefile linking; gated by sys_platform != 'win32' so
Windows dev installs skip it.
When `root` has at least one direct symlinked child, default to following
symlinks instead of silently dropping their contents. This makes "fake
working dir" patterns (a folder full of symlinks pointing at scattered
source dirs) work transparently — the caller no longer has to know to
pass `follow_symlinks=True`.
Concrete motivating shape:
~/projects/research-corpus/
├── papers -> /external/drive/papers
├── notes -> /external/drive/notes
└── code -> /external/drive/code
Before: `--update` invoked `detect_incremental(root)` without passing
`follow_symlinks=True`, so the scan saw the small set of files actually
inside the root and missed everything reachable only through a symlink.
Result: every legitimate new file was missed, and the manifest paths
were marked as deleted.
After: when `follow_symlinks` is left at its default `None`, `detect()`
runs `_auto_follow_symlinks(root)` (one cheap `iterdir()` + `is_symlink()`
loop) and follows symlinks if any direct child is one. Behaviour is
unchanged for ordinary scans (no direct symlinks → False, as before).
Override is always possible by passing an explicit `follow_symlinks=True`
or `follow_symlinks=False`; existing tests confirming the explicit
behaviour continue to pass unchanged.
Backwards compatibility:
- Type annotation: `bool` -> `bool | None`. Callers passing `True`/`False`
continue to work identically. Callers passing nothing get the new
auto-detect.
- No new dependencies.
- Cheap: one `iterdir()` call once per detect() invocation.
Tests:
- 3 new tests in tests/test_detect.py:
- test_detect_auto_detects_direct_symlink_child
- test_detect_default_does_not_follow_when_no_symlinks
- test_detect_explicit_false_overrides_auto_detect
- Full tests/test_detect.py + tests/test_incremental.py: 49/49 pass.
