From 7490d9e0ecb1494191652f21750c0799cab8d679 Mon Sep 17 00:00:00 2001 From: Safi Date: Fri, 17 Apr 2026 12:56:57 +0100 Subject: [PATCH] remove superpowers specs from repo (should be local only) --- .../2026-04-16-v5-rustworkx-github-design.md | 290 ------------------ .../specs/2026-04-16-v5.0-design.md | 238 -------------- .../specs/2026-04-16-v5.1-design.md | 284 ----------------- 3 files changed, 812 deletions(-) delete mode 100644 docs/superpowers/specs/2026-04-16-v5-rustworkx-github-design.md delete mode 100644 docs/superpowers/specs/2026-04-16-v5.0-design.md delete mode 100644 docs/superpowers/specs/2026-04-16-v5.1-design.md diff --git a/docs/superpowers/specs/2026-04-16-v5-rustworkx-github-design.md b/docs/superpowers/specs/2026-04-16-v5-rustworkx-github-design.md deleted file mode 100644 index 360eb67ef..000000000 --- a/docs/superpowers/specs/2026-04-16-v5-rustworkx-github-design.md +++ /dev/null @@ -1,290 +0,0 @@ -# graphify v5: rustworkx backend + GitHub repo ingestion - -**Date:** 2026-04-16 -**Branch:** v5 -**Status:** Approved (revised after senior engineering review) - ---- - -## Summary - -v5 introduces two major changes on a new branch: - -1. **GitHub repo ingestion** -- users can pass a GitHub URL directly instead of a local path. graphify clones the repo and runs the full pipeline on it. -2. **rustworkx graph backend** -- rustworkx replaces NetworkX as the in-memory graph type throughout, with a NetworkX fallback if rustworkx is not installed. Adds `--dag` flag for acyclic directed graphs and parallel betweenness/shortest-path. - -Both changes are independent. The user-facing API and `graph.json` format are unchanged. - ---- - -## Feature 1: GitHub repo ingestion - -### New file: `graphify/github.py` - -**`resolve_target(input: str) -> Path`** -Called by `__main__.py` before extraction. If input looks like a GitHub URL, delegates to `clone_or_update()` and returns the local clone path. Otherwise returns `Path(input)` unchanged. - -Recognised URL formats: -- `https://github.com/org/repo` -- `http://github.com/org/repo` -- `github.com/org/repo` -- `org/repo` (shorthand, only if it contains exactly one `/` and no dots) - -**`clone_or_update(org: str, repo: str, base_dir: Path) -> Path`** -- Clone destination: `~/.graphify/repos/org/repo/` -- First run: `git clone --depth 1 https://github.com/org/repo ` -- Subsequent runs (dest already exists): - ``` - git -C fetch --depth 1 origin - git -C reset --hard origin/HEAD - ``` - This unconditionally updates to the remote tip without requiring fast-forward eligibility and keeps history shallow. `git pull --ff-only` is explicitly avoided -- it fails on shallow clones when the upstream has rebased or advanced more than one commit. -- Returns the local path on success - -### Integration point - -`__main__.py`: single call to `resolve_target()` before the path is passed to `detect()` and `extract()`. No other changes to `__main__.py`. - -### Error handling - -| Condition | Behaviour | -|-----------|-----------| -| Repo not found / private | Clear error message, exit 1 | -| git not installed | `"git is required for GitHub repo ingestion. Install git and retry."`, exit 1 | -| Network timeout | Retry once, then fail with message | -| Partial clone (disk full, `.git` exists but incomplete) | Delete dest dir, report error, exit 1 | -| Already cloned, fetch/reset fails | Warn, continue with existing local copy | - ---- - -## Feature 2: rustworkx graph backend - -### Dependency - -- `rustworkx` added as optional dependency: `pip install graphifyy[fast]` -- If not installed: fall back to NetworkX with a one-time warning printed to stderr: - `"[graphify] rustworkx not installed -- using NetworkX. Install graphifyy[fast] for 2-10x speedup."` -- `pyproject.toml`: `fast = ["rustworkx"]`, added to `all` -- Note: NetworkX remains a hard dependency (required for Louvain community detection fallback -- rustworkx has no built-in community detection) - -### Graph type mapping - -| v4 (NetworkX) | v5 rustworkx backend | v5 NetworkX fallback | -|---------------|----------------------|----------------------| -| `nx.Graph` | `rustworkx.PyGraph` | `nx.Graph` | -| `nx.DiGraph` | `rustworkx.PyDiGraph` | `nx.DiGraph` | -| `nx.DiGraph` + `--dag` | `rustworkx.PyDAG(check_cycle=True)` | `nx.DiGraph` (no cycle enforcement) | - -### GraphBundle -- the central abstraction - -`PyGraph`/`PyDiGraph`/`PyDAG` are Rust extension types (pyo3 `#[pyclass]`) with no `__dict__` slot. Attribute assignment (`G._id_to_idx = ...`) raises `AttributeError`. The correct design is a thin dataclass returned by `build_from_json()` and passed through the entire pipeline: - -```python -# graphify/utils.py (new file) -from __future__ import annotations -from dataclasses import dataclass, field -from typing import Union -import networkx as nx - -try: - import rustworkx as rx - _RX_GRAPH_TYPES = (rx.PyGraph, rx.PyDiGraph, rx.PyDAG) - HAS_RUSTWORKX = True -except ImportError: - _RX_GRAPH_TYPES = () - HAS_RUSTWORKX = False - -AnyGraph = Union["rx.PyGraph", "rx.PyDiGraph", "rx.PyDAG", nx.Graph, nx.DiGraph] - -@dataclass -class GraphBundle: - graph: AnyGraph - id_to_idx: dict[str, int] = field(default_factory=dict) # empty for NetworkX backend - idx_to_id: dict[int, str] = field(default_factory=dict) # empty for NetworkX backend - -def is_rustworkx(bundle: GraphBundle) -> bool: - return isinstance(bundle.graph, _RX_GRAPH_TYPES) -``` - -Every function that currently accepts `nx.Graph` is updated to accept `GraphBundle`. The internal graph and lookup dicts are accessed via `bundle.graph`, `bundle.id_to_idx`, `bundle.idx_to_id`. - -`is_rustworkx()` lives in `graphify/utils.py`. It is imported by every module that needs to branch on backend. No copies. - -### ID mapping - -rustworkx uses integer node indices internally. `GraphBundle` carries two dicts: -- `id_to_idx: dict[str, int]` -- string node ID → rustworkx index -- `idx_to_id: dict[int, str]` -- rustworkx index → string node ID - -These are populated in `build_from_json()` as nodes are added and carried through the pipeline in the `GraphBundle`. The NetworkX fallback leaves both dicts empty (not needed). - -### API translation reference - -The following access patterns appear ~35 times across `analyze.py`, `cluster.py`, `export.py`, `serve.py`, `wiki.py`. Each must be dual-pathed via `is_rustworkx()`: - -| NetworkX | rustworkx equivalent | -|----------|---------------------| -| `G.nodes[nid]` | `G[id_to_idx[nid]]` | -| `G.nodes(data=True)` | `zip(G.node_indices(), G.nodes())` → use `idx_to_id[idx]` for ID | -| `G.edges(nid, data=True)` | `[(idx_to_id[u], idx_to_id[v], G.get_edge_data(u,v)) for u,v in G.incident_edges(id_to_idx[nid])]` | -| `G.degree(nid)` | `G.degree(id_to_idx[nid])` | -| `G.neighbors(nid)` → string IDs | `[idx_to_id[i] for i in G.neighbors(id_to_idx[nid])]` | -| `G.edges[u, v]` | `G.get_edge_data(id_to_idx[u], id_to_idx[v])` | -| `G.number_of_nodes()` | `G.num_nodes()` | -| `G.number_of_edges()` | `G.num_edges()` | - -### Module changes - -**`graphify/utils.py`** (new) -- `GraphBundle` dataclass -- `is_rustworkx(bundle)` helper -- `AnyGraph` type alias - -**`graphify/build.py`** -- `build_from_json()` returns `GraphBundle` (not a bare graph) -- Nodes added via `G.add_node(payload_dict)` → captures returned index → populates `id_to_idx`/`idx_to_id` -- Edges: `src_idx = id_to_idx.get(src)`, `tgt_idx = id_to_idx.get(tgt)` -- missing indices skip the edge (same semantics as v4 node_set check) -- ID normalization from v0.4.18 preserved (normalize before lookup) -- `--dag` edge-add: wrap in `try/except rustworkx.DAGWouldBeCyclic` -- drop edge, print warning to stderr. Do NOT use `rustworkx.is_directed_acyclic_graph()` for pre-checking (it cannot pre-check a prospective edge) -- NetworkX fallback: `GraphBundle(graph=nx.Graph(), id_to_idx={}, idx_to_id={})` - -**`graphify/cluster.py`** -- `_partition(bundle)` replaces `_partition(G)` -- Leiden (graspologic): graspologic's `leiden()` accepts a NetworkX graph. When rustworkx backend is active, convert to NetworkX for leiden only: - ```python - if is_rustworkx(bundle): - G_nx = nx.Graph() - for u, v in bundle.graph.edge_list(): - G_nx.add_edge(bundle.idx_to_id[u], bundle.idx_to_id[v]) - communities = leiden(G_nx) - else: - communities = leiden(bundle.graph) - ``` -- Louvain fallback: stays `nx.community.louvain_communities()` -- rustworkx has no built-in community detection. When rustworkx backend is active, same edge-list conversion as above. -- Node list extraction from leiden/louvain results uses `idx_to_id` where needed - -**`graphify/analyze.py`** -- All public functions updated to accept `GraphBundle` -- `betweenness_centrality`: `rustworkx.betweenness_centrality(bundle.graph)` returns `dict[int, float]` -- remap to string IDs via `idx_to_id` -- `edge_betweenness_centrality`: `rustworkx.edge_betweenness_centrality(bundle.graph)` returns `dict[(int,int), float]` -- remap edge tuples to string ID pairs -- `shortest_path`: `rustworkx.dijkstra_shortest_paths(bundle.graph, src_idx)` returns `dict[int, list[int]]` -- decode path using `idx_to_id` at every position -- `suggest_questions()`: calls `nx.betweenness_centrality(G, k=k)` with approximation parameter `k`. rustworkx's `betweenness_centrality()` has no `k` parameter (always exact, parallel). When rustworkx backend active, drop `k` and call `rustworkx.betweenness_centrality(bundle.graph)`. This is always exact but faster due to parallelism; behavior change is documented. -- `_is_rustworkx()` removed -- use `is_rustworkx()` from `utils.py` - -**`graphify/export.py`** -- Replace `json_graph.node_link_data()` with `_bundle_to_json(bundle)` -- custom serializer that produces the same schema as `node_link_data()` (see JSON schema below) -- SVG: `rustworkx.spring_layout(bundle.graph)` returns `dict[int, list[float]]` (integer-keyed). Map to string IDs via `idx_to_id` before passing to matplotlib. Node drawing iterates `zip(bundle.graph.node_indices(), bundle.graph.nodes())`. - -**`graphify/serve.py`** -- `_load_graph()` uses same custom deserializer as export.py (loads `graph.json` → `GraphBundle`) -- MCP tool handlers updated: node lookups via `bundle.id_to_idx[node_id]`, neighbour traversal via API translation table above - -**`graphify/wiki.py`** -- Accepts `GraphBundle`, uses `is_rustworkx()` + API translation table for all graph traversal - -### JSON serializer schema - -The custom serializer `_bundle_to_json(bundle)` must produce output byte-compatible with `networkx.readwrite.json_graph.node_link_data()` so v4 `graph.json` files load without modification in v5. The schema: - -```json -{ - "directed": true, - "multigraph": false, - "graph": {}, - "nodes": [ - {"id": "session_validatetoken", "label": "ValidateToken", "file_type": "code", ...} - ], - "links": [ - {"source": "session_validatetoken", "target": "other_node", - "relation": "calls", "confidence": "EXTRACTED", "weight": 1.0, ...} - ] -} -``` - -Key points: -- Top-level key is `"links"` not `"edges"` (this is what `node_link_data()` produces; `build.py` already handles both via the `"links"` → `"edges"` remap on load) -- Node dicts include all attributes from `bundle.graph.nodes()` plus `"id"` key -- Edge dicts include all attributes from `bundle.graph.get_edge_data()` plus `"source"` and `"target"` string IDs - -### `--dag` flag - -- New CLI flag: `graphify /path --dag` -- `build_from_json()` receives `dag=True`, uses `rustworkx.PyDAG(check_cycle=True)` -- Cycle violations: `except rustworkx.DAGWouldBeCyclic` → drop edge, print `"[graphify] warning: skipping edge {src} → {tgt} (would create cycle)"` to stderr -- Report includes topological sort order of god nodes via `rustworkx.topological_sort(bundle.graph)` decoded with `idx_to_id` -- NetworkX fallback when rustworkx absent: `--dag` flag accepted but cycle enforcement is silently skipped (no PyDAG available); warning printed once -- `"dag": true` written to `graph.json` metadata so serve.py can surface it in `get_graph_info` MCP tool. DAG enforcement is build-time only -- reloaded graphs are not re-enforced. -- `skill.md` updated to document `--dag` - -### `graphify path` shortest-path speedup - -- `analyze.py`: `shortest_path()` uses `rustworkx.dijkstra_shortest_paths(bundle.graph, src_idx)` -- no `parallel_threshold` parameter (rustworkx Dijkstra is always Rust-backed; per-query overhead reduction vs NetworkX is already ~10x) -- Path result decoded via `idx_to_id` at every element -- No CLI change -- transparent speedup - ---- - -## Compatibility - -### graph.json - -Format unchanged -- the custom serializer produces identical output to `node_link_data()`. v5 reads v4 `graph.json` files without modification. The integer index mapping is rebuilt from the JSON node list on load. - -### pip install - -| Install | Graph backend | GitHub ingest | -|---------|--------------|---------------| -| `pip install graphifyy` | NetworkX (fallback) | yes | -| `pip install graphifyy[fast]` | rustworkx | yes | -| `pip install graphifyy[all]` | rustworkx | yes | - -NetworkX remains a hard dependency in all cases (required for community detection). - -### Python version - -Unchanged: Python 3.10+ - ---- - -## Testing - -- All 433 existing tests must pass on the NetworkX fallback path (rustworkx not installed) -- Dual-backend coverage: `conftest.py` adds a `graph_backend` pytest fixture parametrized over `["networkx", "rustworkx"]`. Tests that create graphs import the fixture and get a `GraphBundle` built with the appropriate backend. This gives dual-backend coverage without duplicating test files. -- New tests: - - `tests/test_github.py`: URL parsing (all four formats), clone logic (mocked `subprocess.run`), update logic (mocked fetch+reset), each error case - - `tests/test_build_rustworkx.py`: `GraphBundle` round-trip, `id_to_idx`/`idx_to_id` correctness, DAG cycle rejection (`DAGWouldBeCyclic` caught), JSON serializer output matches `node_link_data()` byte-for-byte on a fixture graph - - `tests/test_analyze_rustworkx.py`: betweenness output matches NetworkX within 1e-6 tolerance; `suggest_questions()` betweenness behavior change documented in test comment - - `tests/test_cluster_rustworkx.py`: leiden edge-list conversion produces same community structure as direct NetworkX call on same graph - ---- - -## Files changed - -| File | Change | -|------|--------| -| `graphify/github.py` | New -- GitHub URL resolution + clone/update | -| `graphify/utils.py` | New -- `GraphBundle`, `is_rustworkx()`, `AnyGraph` | -| `graphify/build.py` | Returns `GraphBundle`; rustworkx + NetworkX dual backend | -| `graphify/cluster.py` | `GraphBundle` input; leiden edge-list conversion | -| `graphify/analyze.py` | `GraphBundle` input; rustworkx parallel betweenness + path | -| `graphify/export.py` | `GraphBundle` input; custom JSON serializer; matplotlib layout fix | -| `graphify/serve.py` | `GraphBundle` input; custom deserializer; MCP handler updates | -| `graphify/wiki.py` | `GraphBundle` input; dual-path graph traversal | -| `graphify/__main__.py` | `resolve_target()` call; `--dag` flag | -| `graphify/skill.md` | Document `--dag`; GitHub URL input | -| `pyproject.toml` | `fast = ["rustworkx"]`; add to `all` | -| `tests/conftest.py` | `graph_backend` fixture parametrized over both backends | -| `tests/test_github.py` | New | -| `tests/test_build_rustworkx.py` | New | -| `tests/test_analyze_rustworkx.py` | New | -| `tests/test_cluster_rustworkx.py` | New | - ---- - -## Out of scope for v5 - -- Private repo support (requires GitHub token -- future work) -- Incremental re-extraction after `git pull` (`--update` already handles this once cloned) -- GraphQL / GitHub API (issues, PRs, file-level fetch) -- future work -- rustworkx GPU acceleration -- future work -- DAG cycle enforcement on graph reload (enforcement is build-time only) diff --git a/docs/superpowers/specs/2026-04-16-v5.0-design.md b/docs/superpowers/specs/2026-04-16-v5.0-design.md deleted file mode 100644 index b30649e15..000000000 --- a/docs/superpowers/specs/2026-04-16-v5.0-design.md +++ /dev/null @@ -1,238 +0,0 @@ -# graphify v5.0 design spec - -**Date:** 2026-04-16 -**Branch:** v5 -**Status:** Draft -**Milestone:** v5.0 -- foundation layer - ---- - -## Summary - -v5.0 is the foundation of the graphify enterprise layer. Four independent but coordinated changes: - -1. **rustworkx graph backend** -- replaces NetworkX in-memory with a `GraphBundle` abstraction, NetworkX fallback retained -2. **GitHub repo ingestion** -- `graphify add github.com/org/repo` clones and extracts -3. **Within-document chunking + section nodes** -- PDFs and markdown split into sections before LLM extraction; sections become first-class nodes anchoring concepts -4. **Content-based exact deduplication** -- cache keyed on body hash only (not path), same content never extracted twice regardless of filename - -These four changes compose: a GitHub repo clone goes through the same chunking + dedup pipeline as a local corpus. - ---- - -## Change 1: rustworkx graph backend - -*(Full detail already in `2026-04-16-v5-rustworkx-github-design.md` -- this section summarises only the additions made after senior engineering review)* - -### GraphBundle - -```python -# graphify/utils.py (new) -@dataclass -class GraphBundle: - graph: AnyGraph # PyGraph | PyDiGraph | PyDAG | nx.Graph | nx.DiGraph - id_to_idx: dict[str, int] # empty for NetworkX backend - idx_to_id: dict[int, str] # empty for NetworkX backend - -def is_rustworkx(bundle: GraphBundle) -> bool: ... -``` - -`build_from_json()` returns `GraphBundle`. All downstream modules (`cluster`, `analyze`, `export`, `serve`, `wiki`) accept `GraphBundle`. - -### Key corrections from engineering review - -- No `rustworkx.community` module exists -- Louvain stays NetworkX-backed -- graspologic `leiden()` needs a NetworkX graph -- convert via edge list when rustworkx backend active -- `PyGraph`/`PyDiGraph` are pyo3 types, no `__dict__` -- monkey-patching forbidden, hence `GraphBundle` -- DAG cycle handling: `try/except rustworkx.DAGWouldBeCyclic`, not `is_directed_acyclic_graph()` -- `dijkstra_shortest_paths()` has no `parallel_threshold` -- drop it -- `git pull --ff-only` broken on shallow clones -- use `git fetch --depth 1 && git reset --hard origin/HEAD` - -### Dual-backend testing - -`tests/conftest.py`: `graph_backend` fixture parametrized over `["networkx", "rustworkx"]`. Existing 433 tests run on NetworkX fallback; new tests parametrized over both. - ---- - -## Change 2: GitHub repo ingestion - -### New file: `graphify/github.py` - -**`resolve_target(input: str) -> Path`** -Called by `__main__.py` before extraction. Recognises: -- `https://github.com/org/repo` -- `github.com/org/repo` -- `org/repo` (exactly one `/`, no dots) - -Returns local clone path or `Path(input)` unchanged. - -**`clone_or_update(org, repo, base_dir) -> Path`** -- Clone: `~/.graphify/repos/org/repo/` -- First run: `git clone --depth 1 https://github.com/org/repo ` -- Update: `git -C fetch --depth 1 origin && git -C reset --hard origin/HEAD` - -### Error handling - -| Condition | Behaviour | -|-----------|-----------| -| Repo not found / private | Clear message, exit 1 | -| git not installed | Message pointing to git install, exit 1 | -| Network timeout | Retry once, fail with message | -| Partial clone | Delete dest, report, exit 1 | -| Fetch/reset fails | Warn, use existing local copy | - ---- - -## Change 3: within-document chunking + section nodes - -### The problem - -Currently the LLM subagent receives entire file contents. A 300-page PDF = ~150k tokens in one context, risking truncation and shallow extraction. There is no within-document structure in the graph -- a book produces a flat bag of concept nodes with no hierarchy. - -### Solution: two-level split - -**Level 1 -- processing chunks (invisible in graph)** -Documents are split into processing units before being sent to LLM subagents. These are purely a compute concern -- they do not become nodes. - -| File type | Split strategy | -|-----------|---------------| -| PDF | Per page (pypdf `page.extract_text()`) -- pages grouped into batches of 10 | -| Markdown / RST | Per heading (`## `, `### `) -- sections split at H2/H3 boundaries | -| Plain text | Per 2000 words | -| DOCX | Per heading style (Heading 1 / Heading 2) | -| Images | One per subagent (unchanged) | -| Code | AST extraction unchanged, no LLM chunking | - -**Level 2 -- section nodes (visible in graph)** -Each processing unit produces one **section node** in addition to its concept nodes. Section nodes: -- `file_type: "section"` -- `id`: `{doc_stem}_{section_index}` e.g. `attention_paper_p012` (page 12), `readme_s03` (section 3) -- `label`: heading text (markdown) or `"Page 12"` (PDF) or `"Part 3"` (plain text) -- `source_file`: parent document path -- `source_location`: page number or heading anchor - -Every concept node extracted from a section gets an `EXTRACTED` edge to its section node (`contained_in`). The section node gets a `contained_in` edge to the file node. This gives a navigable three-level hierarchy: - -``` -file node - └─ contained_in ← section node (page / heading) - └─ contained_in ← concept node (LLM-extracted) -``` - -Concepts are still LLM-extracted and non-deterministic -- but they are now **bounded per section**. The same section on re-run produces the same section node ID, so the structure is reproducible even when concept labels vary. - -### Subagent prompt changes - -The subagent prompt gains: - -``` -Section context: {section_label} ({doc_path}, {location}) -Section ID: {section_node_id} - -For every concept node you extract, add a "contained_in" edge from the concept to -the section node ID above (confidence: EXTRACTED, weight: 1.0). -Also emit the section node itself as a node with file_type="section". -``` - -### Cache key for sections - -Sections are cached individually. Cache key: `SHA256(section_text)` -- content only, no path. If the same section appears in two files (e.g. a copied intro paragraph), only one LLM extraction runs. The second file gets the cached nodes with its own section node added. - -### New module: `graphify/splitter.py` - -```python -def split_document(path: Path) -> list[DocumentSection]: - """Split a document into sections for chunked LLM extraction.""" - -@dataclass -class DocumentSection: - doc_path: Path - section_index: int - label: str # heading text or "Page N" - location: str # "p12", "§3.2", etc. - text: str # content to send to LLM - node_id: str # deterministic section node ID - node: dict # pre-built section node dict -``` - -`splitter.py` is called in the skill before subagent dispatch. Its output replaces the flat file list with a section list. Each section becomes an item in the chunk assignment. - -### Chunk assignment changes - -Currently: chunks of 20-25 **files**. -v5.0: chunks of 20-25 **sections** (images still get their own chunk). - -A 300-page PDF produces 30 sections (10 pages each) → 2 chunks of 15 sections each, running in parallel. Token load per subagent drops from ~150k to ~15k. - ---- - -## Change 4: content-based exact deduplication - -### The problem - -Current cache key: `SHA256(content + path)`. Same file, different name = two extractions, two sets of duplicate nodes, double LLM cost. - -### Fix: content-only hash - -Change `file_hash()` in `cache.py`: - -```python -# v4 (path-dependent) -h.update(content) -h.update(b"\x00") -h.update(str(rel).encode()) # ← causes duplicate cache misses for same content - -# v5.0 (content-only) -h.update(content) -# path removed -``` - -For sections: `SHA256(section_text)` -- section text only, no path or index. - -### Dedup at graph build time - -When `build_from_json()` encounters two nodes with the same `id` (possible if duplicate files were extracted before this fix landed), last-write wins (existing NetworkX behavior, preserved in GraphBundle). No change needed. - -When the same cache entry is loaded for two different paths, the nodes carry `source_file` of the first file that produced them. v5.0 adds a `also_found_in: list[str]` attribute to nodes that are deduplication hits -- surfaced in GRAPH_REPORT as "N duplicate sources collapsed." - -### Backward compatibility - -Existing cache entries (path-dependent keys) become orphaned -- they will never match the new content-only keys. On first run after upgrade, all files re-extract. This is acceptable: one-time cost, correct behavior from that point forward. A migration note is printed: `"[graphify] Cache format updated in v5.0 -- re-extracting all files (one-time cost)."` - ---- - -## Files changed - -| File | Change | -|------|--------| -| `graphify/utils.py` | New -- `GraphBundle`, `is_rustworkx()`, `AnyGraph` | -| `graphify/github.py` | New -- GitHub URL resolution + clone/update | -| `graphify/splitter.py` | New -- `split_document()`, `DocumentSection` | -| `graphify/build.py` | `GraphBundle` return; rustworkx + NetworkX dual backend; `also_found_in` dedup attr | -| `graphify/cache.py` | Content-only hash; section cache; migration notice | -| `graphify/cluster.py` | `GraphBundle` input; leiden edge-list conversion | -| `graphify/analyze.py` | `GraphBundle` input; rustworkx parallel betweenness + path | -| `graphify/export.py` | `GraphBundle` input; custom JSON serializer; matplotlib layout | -| `graphify/serve.py` | `GraphBundle` input; custom deserializer; MCP handler updates | -| `graphify/wiki.py` | `GraphBundle` input; dual-path graph traversal | -| `graphify/__main__.py` | `resolve_target()` call; `--dag` flag | -| `graphify/skill.md` | Section node prompt; `--dag`; GitHub URL input; chunking by section | -| `pyproject.toml` | `fast = ["rustworkx"]`; add to `all` | -| `tests/conftest.py` | `graph_backend` fixture | -| `tests/test_github.py` | New | -| `tests/test_splitter.py` | New -- section splitting for PDF, markdown, plain text | -| `tests/test_build_rustworkx.py` | New | -| `tests/test_analyze_rustworkx.py` | New | -| `tests/test_cluster_rustworkx.py` | New | -| `tests/test_dedup.py` | New -- same content different path → single cache entry | - ---- - -## Out of scope (v5.1) - -- Multi-tenant silos and federated graph queries -- Near-deduplication (SimHash/MinHash for ~similar content) -- Entity type registry (Concept, Claim, Person, Method, Dataset, Decision) -- KG storage backend evaluation (Neo4j, Kuzu, LanceDB, TigerGraph) -- Document metadata store (separate from node attributes) -- Private GitHub repo support (token auth) diff --git a/docs/superpowers/specs/2026-04-16-v5.1-design.md b/docs/superpowers/specs/2026-04-16-v5.1-design.md deleted file mode 100644 index 5fed33edd..000000000 --- a/docs/superpowers/specs/2026-04-16-v5.1-design.md +++ /dev/null @@ -1,284 +0,0 @@ -# graphify v5.1 design spec - -**Date:** 2026-04-16 -**Branch:** v5 -**Status:** Draft -- depends on v5.0 -**Milestone:** v5.1 -- enterprise + scaling research - ---- - -## Summary - -v5.1 builds the enterprise layer on top of v5.0's foundation. Four areas: - -1. **Silos** -- multi-tenant graph namespacing with federated cross-silo queries -2. **Near-deduplication** -- SimHash/MinHash fingerprinting to collapse near-duplicate documents before LLM extraction -3. **Entity type registry** -- strict typed entity model replacing the LLM's ad-hoc node decisions -4. **KG scaling research** -- systematic evaluation of storage backends for graphs that exceed RAM - -These are independent and can ship incrementally within the v5.1 milestone. - ---- - -## Change 1: Silos - -### What a silo is - -A silo is a named, isolated graph namespace. Each silo has its own: -- `graph.json` (its node/edge set) -- `cache/` (its extraction cache) -- `manifest.json` (its file manifest) -- Access label (who owns it) - -Silos live under a shared base directory, defaulting to `~/.graphify/silos/`: - -``` -~/.graphify/silos/ - myapp/ - graph.json - cache/ - manifest.json - meta.json ← silo metadata (owner, created_at, description, tags) - research-2026/ - graph.json - cache/ - ... -``` - -### CLI - -```bash -graphify silo create myapp --description "main product repo" -graphify silo list -graphify silo delete myapp -graphify silo info myapp - -# Build graph into a specific silo -graphify . --silo myapp -graphify add github.com/org/repo --silo myapp - -# Query a silo -graphify query "auth flow" --silo myapp -graphify path "SessionManager" "Database" --silo myapp - -# Federated query across silos -graphify query "auth flow" --silos myapp,research-2026 -graphify query "auth flow" --silos all -``` - -### Silo metadata (`meta.json`) - -```json -{ - "name": "myapp", - "description": "main product repo", - "owner": "safishamsi98@gmail.com", - "created_at": "2026-04-16T00:00:00Z", - "updated_at": "2026-04-16T00:00:00Z", - "tags": ["backend", "python"], - "sources": [ - {"type": "github", "url": "github.com/org/repo", "cloned_at": "2026-04-16T00:00:00Z"}, - {"type": "local", "path": "/home/user/docs", "added_at": "2026-04-16T00:00:00Z"} - ], - "node_count": 1243, - "edge_count": 4821 -} -``` - -### Federated queries - -A federated query loads multiple `GraphBundle`s and merges them for query purposes only -- the individual silo graphs are not mutated. The merge is shallow: nodes from different silos with the same ID are kept separate (prefixed with silo name internally). Cross-silo edges can only be INFERRED -- there are no EXTRACTED cross-silo edges unless explicitly added. - -The result of a federated query surfaces which silo each node came from: - -``` -NODE: SessionManager [silo: myapp] - → calls → validate_token [silo: myapp] - → semantically_similar_to → AuthHandler [silo: research-2026, confidence: 0.82] -``` - -### New module: `graphify/silo.py` - -```python -def create_silo(name: str, base_dir: Path, description: str = "") -> Path -def delete_silo(name: str, base_dir: Path) -> None -def list_silos(base_dir: Path) -> list[SiloMeta] -def load_silo(name: str, base_dir: Path) -> GraphBundle -def merge_silos(names: list[str], base_dir: Path) -> GraphBundle # federated, read-only -def update_silo_meta(name: str, base_dir: Path, **fields) -> None -``` - -### Access control (v5.1 scope) - -Owner field in `meta.json` is informational only in v5.1. No authentication or enforcement. True multi-tenant auth (API keys, org membership) is v6 territory. - ---- - -## Change 2: Near-deduplication - -### The problem - -v5.0 exact dedup (SHA256 body-only) handles identical files. Near-dedup handles: -- v1 and v2 of the same paper (85% similar) -- A README copied with minor edits into a wiki -- The same email thread quoted at different levels of truncation - -Without near-dedup, near-duplicate documents produce overlapping concept nodes that pollute community detection and inflate god node scores. - -### Approach: MinHash + LSH - -**Fingerprinting:** Each document (or section in v5.0's model) is shingled (k=5 word shingles) and hashed to a MinHash signature (128 hash functions). Signatures are stored in `~/.graphify/fingerprints/{silo}.bin`. - -**Similarity threshold:** Documents with Jaccard similarity ≥ 0.85 are considered near-duplicates. Threshold is configurable: `--dedup-threshold 0.85`. - -**On detection:** -1. The lower-priority document (later ingested) skips LLM extraction -2. Its nodes are merged into the canonical document's nodes: `also_found_in` list extended -3. A `EXTRACTED` edge `superseded_by` connects the duplicate file node to the canonical file node -4. GRAPH_REPORT surfaces: "3 near-duplicate documents collapsed into 1 canonical source" - -**Library:** `datasketch` (pure Python, no native dependencies). Added as optional dependency: `pip install graphifyy[dedup]`, added to `all`. - -### New module: `graphify/dedup.py` - -```python -def fingerprint(text: str) -> MinHashSignature -def find_near_duplicates( - paths: list[Path], - threshold: float = 0.85, - fingerprint_store: Path | None = None, -) -> list[tuple[Path, Path, float]] # (canonical, duplicate, similarity) -def load_fingerprints(store: Path) -> FingerprintStore -def save_fingerprints(store: Path, fps: FingerprintStore) -> None -``` - ---- - -## Change 3: Entity type registry - -### The problem - -v5.0 section nodes add structure but the concepts within each section are still fully LLM-determined. The same paper produces `"attention mechanism"` in one run and `"self-attention"` in another. Federated queries across silos fail when the same concept has different labels. - -### Solution: typed entity model - -Replace the untyped `file_type: "document"|"paper"|"image"` with a mandatory `entity_type` field on every semantic node: - -| entity_type | Description | Examples | -|-------------|-------------|---------| -| `Concept` | Named idea, algorithm, pattern | "Attention Mechanism", "Leiden Community Detection" | -| `Claim` | Assertion made in source | "BERT outperforms GPT on GLUE" | -| `Person` | Author, researcher, contributor | "Vaswani et al.", "Andrej Karpathy" | -| `Method` | Technique, algorithm, procedure | "Scaled Dot-Product Attention", "Adam optimizer" | -| `Dataset` | Named dataset or benchmark | "ImageNet", "GLUE", "HumanEval" | -| `Decision` | Design decision, rationale node | "Use LayerNorm before attention (Pre-LN)" | -| `Section` | Document section (from splitter.py) | "Page 12", "§3.2 Encoder" | -| `File` | File-level node (code or document) | "session.py", "paper.pdf" | - -### Skill prompt change - -The subagent schema gains `entity_type` as a required field. The node schema: - -```json -{ - "id": "attention_paper_s03_attention_mechanism", - "label": "Attention Mechanism", - "entity_type": "Concept", - "file_type": "paper", - "source_file": "attention_paper.pdf", - "source_location": "§3", - "contained_in": "attention_paper_s03" -} -``` - -### Normalisation - -Entity labels are normalised at build time: lowercased, stripped, deduplicated by (label, entity_type, source_file). Two subagents extracting "Attention Mechanism" and "attention mechanism" from the same section produce one node. - -### Validation - -`validate.py` updated to enforce `entity_type` is one of the registered values. Nodes missing `entity_type` are assigned `"Concept"` with a warning (backward compatibility with v5.0 graphs). - ---- - -## Change 4: KG scaling research - -### The problem - -graphify builds the full graph in RAM. This works for corpora up to ~50k nodes (~500MB RAM). Beyond that: -- `betweenness_centrality` becomes prohibitively slow even with rustworkx parallelism -- `graph.json` serialization produces files >1GB -- Leiden community detection on the full graph fails - -### Research scope - -v5.1 does not pick a storage backend. It **evaluates** four candidates against graphify's specific query patterns: - -| Backend | Type | Key property | -|---------|------|-------------| -| Neo4j | Property graph DB | Mature, Cypher query language, graphify already has `--neo4j` export | -| Kuzu | Embedded property graph | DuckDB-style, no server, fast analytical queries, columnar storage | -| LanceDB | Vector + graph hybrid | Native embedding storage, good for semantic similarity queries | -| TigerGraph | Distributed graph DB | Horizontal scaling, GSQL, designed for 100B+ edge graphs | - -### Evaluation criteria - -For each backend, measure against a 500k-node, 2M-edge synthetic graphify corpus: - -1. **Ingest time** -- time to load `graph.json` into the backend -2. **Betweenness centrality** -- wall time for full graph betweenness -3. **BFS/DFS traversal** -- `graphify query` workload (3-hop neighbourhood) -4. **Shortest path** -- `graphify path` workload -5. **Subgraph extraction** -- pull a community as a subgraph -6. **Memory footprint** -- RSS at peak -7. **Operational complexity** -- setup, persistence, backup - -### Deliverable - -A research report: `docs/scaling-research/2026-KG-backend-evaluation.md` with benchmark numbers, trade-off analysis, and a recommendation for v6 integration. The report is committed to the repo. - -No backend is integrated into graphify in v5.1. The recommendation informs v6. - -### Synthetic corpus generator - -`scripts/gen_corpus.py` -- generates a synthetic `graph.json` at configurable scale (nodes, edges, communities) for reproducible benchmarking. Not shipped in the wheel. - ---- - -## Files changed - -| File | Change | -|------|--------| -| `graphify/silo.py` | New -- silo CRUD, federated merge | -| `graphify/dedup.py` | New -- MinHash fingerprinting, near-dedup detection | -| `graphify/__main__.py` | Silo CLI commands; `--dedup-threshold`; federated query flag | -| `graphify/validate.py` | `entity_type` enforcement | -| `graphify/skill.md` | `entity_type` in node schema; silo-aware subagent prompt | -| `graphify/build.py` | Label normalisation; `entity_type` default assignment | -| `graphify/report.py` | Near-dedup summary; silo source attribution | -| `pyproject.toml` | `dedup = ["datasketch"]`; add to `all` | -| `tests/test_silo.py` | New | -| `tests/test_dedup.py` | New -- MinHash, threshold behaviour, fingerprint persistence | -| `tests/test_entity_types.py` | New -- registry validation, label normalisation | -| `scripts/gen_corpus.py` | New -- synthetic corpus generator (not in wheel) | -| `docs/scaling-research/` | New -- benchmark results directory | - ---- - -## Dependencies on v5.0 - -- `GraphBundle` (utils.py) -- silos load graphs as bundles; federated merge operates on bundles -- Section nodes (splitter.py) -- entity type registry includes `Section`; near-dedup fingerprints sections not whole files -- Content-only cache hash -- near-dedup and exact dedup share the same hash function - -v5.1 cannot ship without v5.0 complete. - ---- - -## Out of scope (v6) - -- True multi-tenant authentication (API keys, org membership, RBAC) -- Streaming graph updates (append-only graph mutation without full rebuild) -- Real-time federated queries (live cross-silo joins) -- Integration of winning storage backend from v5.1 scaling research -- GraphQL API over the knowledge graph