From 238702b69754c5ecd011df1597edbe65dbdec766 Mon Sep 17 00:00:00 2001 From: eugene-krivosheyev Date: Sun, 24 May 2026 21:38:13 +0200 Subject: [PATCH] Constrained query expansion (#998) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add constrained query expansion step to /graphify query skill ## Problem `graphify query` matches via case-folded substring + IDF — no stemming, no synonyms, no cross-language match. When the user's question uses different vocabulary than the graph labels (Slavic → English, "handlers" → "handler", "обработчик" → "handler"), the literal matcher returns 0 hits and the LLM consumer either gets empty subgraph or improvises an ungrounded keyword list from training memory (e.g. expanding "auth" to `{passport, sso, saml, oauth, jwt, scim, …}` regardless of whether those tokens exist in the corpus). ## Fix Adds a `Step 0 — Constrained query expansion` block to the skill's `/graphify query` section. The LLM consumer extracts vocabulary from graph labels (CamelCase/snake_case split, length-filtered) and is instructed to pick **only** tokens present in that vocabulary, explicitly forbidden from inventing terms. Effects: - Bounded improvisation — fantom tokens (terms not in corpus) cannot be expanded, even when LLM "knows" they're related to the intent. - Honest negative signal — if vocab is poor on a query's topic, expansion returns [] and the LLM tells the user, instead of fabricating a search. - Auditability — selected tokens are printed to the user, and saved into `save-result` for the next --update to graph as Q&A nodes. ## Scope Patches the canonical `graphify/skill.md`. The 11 host-variant skills (skill-codex.md, skill-aider.md, …) follow the same query-section contract but inline Python rather than calling `graphify query` CLI; those need a parallel patch with the inline form. Happy to follow up in a separate PR after review on the canonical patch. ## Test On a graph built from the graphify repo itself (1284 nodes, 1454 vocab tokens), an unconstrained expansion of "укрупненная архитектура аутентификации" yields {auth, oauth, jwt, saml, sso, ldap, scim, mfa, 2fa, pin, passport, session, login, token} — of which 11/15 are absent from the corpus. Constrained expansion against the actual vocab yields {credential, security, token, signature, user, architecture, component, module, overview} — 9 tokens, 0 fantom. Same retrieval, dramatically higher precision. --- graphify/skill.md | 49 +++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 45 insertions(+), 4 deletions(-) diff --git a/graphify/skill.md b/graphify/skill.md index 2f5b3181..6fa688cb 100644 --- a/graphify/skill.md +++ b/graphify/skill.md @@ -965,20 +965,61 @@ Two traversal modes - choose based on the question: | BFS (default) | _(none)_ | "What is X connected to?" - broad context, nearest neighbors first | | DFS | `--dfs` | "How does X reach Y?" - trace a specific chain or dependency path | +### Step 0 — Constrained query expansion (REQUIRED before traversal) + +graphify's `query` CLI matches nodes via case-folded substring + IDF — there is **no stemming, no synonyms, no cross-language match** inside the binary. If the user's question uses different language or different domain vocabulary than the graph's labels (user says "обработчик" / graph says "handler"; user says "authentication" / graph says "Guardian"), the literal matcher returns 0 hits and the answer collapses to noise. + +Fix this **without inventing tokens** by expanding the query against the actual graph vocabulary first: + +1. Extract the token vocabulary from node labels: +```bash +$(cat graphify-out/.graphify_python) -c " +import json, re +from pathlib import Path +data = json.loads(Path('graphify-out/graph.json').read_text()) +vocab = set() +for n in data['nodes']: + for c in re.findall(r'[A-Za-z]+', n.get('label','') or ''): + for p in re.findall(r'[A-Z]+(?=[A-Z][a-z])|[A-Z]?[a-z]+|[A-Z]+', c): + t = p.lower() + if 3 <= len(t) <= 30: + vocab.add(t) +Path('graphify-out/.vocab.txt').write_text('\n'.join(sorted(vocab))) +print(f'vocab: {len(vocab)} tokens') +" +``` + +2. Read `graphify-out/.vocab.txt`. Then for the user's question, select **up to 12 tokens from this exact list** that semantically match the query intent. Hard constraints: + - You MUST pick only tokens present in the vocabulary file. Do NOT invent tokens. + - If a query concept has no plausible token in the vocab, skip it — do not substitute a near-synonym from training memory. + - If **no** vocab tokens match the query at all, output an empty list and tell the user the corpus has no relevant vocabulary for this question. Do not fabricate a search. + - Translate cross-language: Russian "аутентификация" → look for `auth`, `credential`, `token`, `security` IFF present in vocab. + - Morphology: "handlers" maps to `handler` IFF present; "todos" maps to `todo` IFF present. + +3. Print the selection explicitly to the user before running the query, so the expansion is auditable: +``` +Query expanded to (from graph vocab, N tokens): [token1, token2, ...] +``` +If the list is empty, say so plainly and stop — do not proceed to traversal. + +### Step 1 — Traversal + +Build the **expanded query string** by joining the selected tokens with spaces. Use this string as `QUESTION` below — NOT the original user question. (The original question is preserved only for `save-result` at the end.) + ```bash graphify query "QUESTION" # or: graphify query "QUESTION" --dfs --budget 3000 ``` -Replace `QUESTION` with the user's actual question. Answer using **only** what the graph output contains. Quote `source_location` when citing a specific fact. If the graph lacks enough information, say so - do not hallucinate edges. +Answer using **only** what the graph output contains. Quote `source_location` when citing a specific fact. If the graph lacks enough information, say so - do not hallucinate edges. -After writing the answer, save it back into the graph so it improves future queries: +After writing the answer, save it back into the graph so it improves future queries. Include the expanded tokens inside the `--answer` text (e.g. `"Expanded from original query via vocab: [tokens]. Then traversed..."`) so the next `--update` extracts the expansion history as a graph node: ```bash -$(cat graphify-out/.graphify_python) -m graphify save-result --question "QUESTION" --answer "ANSWER" --type query --nodes NODE1 NODE2 +$(cat graphify-out/.graphify_python) -m graphify save-result --question "ORIGINAL_QUESTION" --answer "ANSWER" --type query --nodes NODE1 NODE2 ``` -Replace `QUESTION` with the question, `ANSWER` with your full answer text, `SOURCE_NODES` with the list of node labels you cited. This closes the feedback loop: the next `--update` will extract this Q&A as a node in the graph. +Replace `ORIGINAL_QUESTION` with the user's verbatim question, `ANSWER` with your full answer text (containing the expanded-token trace), `NODE1 NODE2` with the list of node labels you cited. This closes the feedback loop: the next `--update` will extract this Q&A as a node in the graph. ---