diff --git a/docs/superpowers/specs/2026-04-16-v5.0-design.md b/docs/superpowers/specs/2026-04-16-v5.0-design.md new file mode 100644 index 00000000..b30649e1 --- /dev/null +++ b/docs/superpowers/specs/2026-04-16-v5.0-design.md @@ -0,0 +1,238 @@ +# graphify v5.0 design spec + +**Date:** 2026-04-16 +**Branch:** v5 +**Status:** Draft +**Milestone:** v5.0 -- foundation layer + +--- + +## Summary + +v5.0 is the foundation of the graphify enterprise layer. Four independent but coordinated changes: + +1. **rustworkx graph backend** -- replaces NetworkX in-memory with a `GraphBundle` abstraction, NetworkX fallback retained +2. **GitHub repo ingestion** -- `graphify add github.com/org/repo` clones and extracts +3. **Within-document chunking + section nodes** -- PDFs and markdown split into sections before LLM extraction; sections become first-class nodes anchoring concepts +4. **Content-based exact deduplication** -- cache keyed on body hash only (not path), same content never extracted twice regardless of filename + +These four changes compose: a GitHub repo clone goes through the same chunking + dedup pipeline as a local corpus. + +--- + +## Change 1: rustworkx graph backend + +*(Full detail already in `2026-04-16-v5-rustworkx-github-design.md` -- this section summarises only the additions made after senior engineering review)* + +### GraphBundle + +```python +# graphify/utils.py (new) +@dataclass +class GraphBundle: + graph: AnyGraph # PyGraph | PyDiGraph | PyDAG | nx.Graph | nx.DiGraph + id_to_idx: dict[str, int] # empty for NetworkX backend + idx_to_id: dict[int, str] # empty for NetworkX backend + +def is_rustworkx(bundle: GraphBundle) -> bool: ... +``` + +`build_from_json()` returns `GraphBundle`. All downstream modules (`cluster`, `analyze`, `export`, `serve`, `wiki`) accept `GraphBundle`. + +### Key corrections from engineering review + +- No `rustworkx.community` module exists -- Louvain stays NetworkX-backed +- graspologic `leiden()` needs a NetworkX graph -- convert via edge list when rustworkx backend active +- `PyGraph`/`PyDiGraph` are pyo3 types, no `__dict__` -- monkey-patching forbidden, hence `GraphBundle` +- DAG cycle handling: `try/except rustworkx.DAGWouldBeCyclic`, not `is_directed_acyclic_graph()` +- `dijkstra_shortest_paths()` has no `parallel_threshold` -- drop it +- `git pull --ff-only` broken on shallow clones -- use `git fetch --depth 1 && git reset --hard origin/HEAD` + +### Dual-backend testing + +`tests/conftest.py`: `graph_backend` fixture parametrized over `["networkx", "rustworkx"]`. Existing 433 tests run on NetworkX fallback; new tests parametrized over both. + +--- + +## Change 2: GitHub repo ingestion + +### New file: `graphify/github.py` + +**`resolve_target(input: str) -> Path`** +Called by `__main__.py` before extraction. Recognises: +- `https://github.com/org/repo` +- `github.com/org/repo` +- `org/repo` (exactly one `/`, no dots) + +Returns local clone path or `Path(input)` unchanged. + +**`clone_or_update(org, repo, base_dir) -> Path`** +- Clone: `~/.graphify/repos/org/repo/` +- First run: `git clone --depth 1 https://github.com/org/repo ` +- Update: `git -C fetch --depth 1 origin && git -C reset --hard origin/HEAD` + +### Error handling + +| Condition | Behaviour | +|-----------|-----------| +| Repo not found / private | Clear message, exit 1 | +| git not installed | Message pointing to git install, exit 1 | +| Network timeout | Retry once, fail with message | +| Partial clone | Delete dest, report, exit 1 | +| Fetch/reset fails | Warn, use existing local copy | + +--- + +## Change 3: within-document chunking + section nodes + +### The problem + +Currently the LLM subagent receives entire file contents. A 300-page PDF = ~150k tokens in one context, risking truncation and shallow extraction. There is no within-document structure in the graph -- a book produces a flat bag of concept nodes with no hierarchy. + +### Solution: two-level split + +**Level 1 -- processing chunks (invisible in graph)** +Documents are split into processing units before being sent to LLM subagents. These are purely a compute concern -- they do not become nodes. + +| File type | Split strategy | +|-----------|---------------| +| PDF | Per page (pypdf `page.extract_text()`) -- pages grouped into batches of 10 | +| Markdown / RST | Per heading (`## `, `### `) -- sections split at H2/H3 boundaries | +| Plain text | Per 2000 words | +| DOCX | Per heading style (Heading 1 / Heading 2) | +| Images | One per subagent (unchanged) | +| Code | AST extraction unchanged, no LLM chunking | + +**Level 2 -- section nodes (visible in graph)** +Each processing unit produces one **section node** in addition to its concept nodes. Section nodes: +- `file_type: "section"` +- `id`: `{doc_stem}_{section_index}` e.g. `attention_paper_p012` (page 12), `readme_s03` (section 3) +- `label`: heading text (markdown) or `"Page 12"` (PDF) or `"Part 3"` (plain text) +- `source_file`: parent document path +- `source_location`: page number or heading anchor + +Every concept node extracted from a section gets an `EXTRACTED` edge to its section node (`contained_in`). The section node gets a `contained_in` edge to the file node. This gives a navigable three-level hierarchy: + +``` +file node + └─ contained_in ← section node (page / heading) + └─ contained_in ← concept node (LLM-extracted) +``` + +Concepts are still LLM-extracted and non-deterministic -- but they are now **bounded per section**. The same section on re-run produces the same section node ID, so the structure is reproducible even when concept labels vary. + +### Subagent prompt changes + +The subagent prompt gains: + +``` +Section context: {section_label} ({doc_path}, {location}) +Section ID: {section_node_id} + +For every concept node you extract, add a "contained_in" edge from the concept to +the section node ID above (confidence: EXTRACTED, weight: 1.0). +Also emit the section node itself as a node with file_type="section". +``` + +### Cache key for sections + +Sections are cached individually. Cache key: `SHA256(section_text)` -- content only, no path. If the same section appears in two files (e.g. a copied intro paragraph), only one LLM extraction runs. The second file gets the cached nodes with its own section node added. + +### New module: `graphify/splitter.py` + +```python +def split_document(path: Path) -> list[DocumentSection]: + """Split a document into sections for chunked LLM extraction.""" + +@dataclass +class DocumentSection: + doc_path: Path + section_index: int + label: str # heading text or "Page N" + location: str # "p12", "§3.2", etc. + text: str # content to send to LLM + node_id: str # deterministic section node ID + node: dict # pre-built section node dict +``` + +`splitter.py` is called in the skill before subagent dispatch. Its output replaces the flat file list with a section list. Each section becomes an item in the chunk assignment. + +### Chunk assignment changes + +Currently: chunks of 20-25 **files**. +v5.0: chunks of 20-25 **sections** (images still get their own chunk). + +A 300-page PDF produces 30 sections (10 pages each) → 2 chunks of 15 sections each, running in parallel. Token load per subagent drops from ~150k to ~15k. + +--- + +## Change 4: content-based exact deduplication + +### The problem + +Current cache key: `SHA256(content + path)`. Same file, different name = two extractions, two sets of duplicate nodes, double LLM cost. + +### Fix: content-only hash + +Change `file_hash()` in `cache.py`: + +```python +# v4 (path-dependent) +h.update(content) +h.update(b"\x00") +h.update(str(rel).encode()) # ← causes duplicate cache misses for same content + +# v5.0 (content-only) +h.update(content) +# path removed +``` + +For sections: `SHA256(section_text)` -- section text only, no path or index. + +### Dedup at graph build time + +When `build_from_json()` encounters two nodes with the same `id` (possible if duplicate files were extracted before this fix landed), last-write wins (existing NetworkX behavior, preserved in GraphBundle). No change needed. + +When the same cache entry is loaded for two different paths, the nodes carry `source_file` of the first file that produced them. v5.0 adds a `also_found_in: list[str]` attribute to nodes that are deduplication hits -- surfaced in GRAPH_REPORT as "N duplicate sources collapsed." + +### Backward compatibility + +Existing cache entries (path-dependent keys) become orphaned -- they will never match the new content-only keys. On first run after upgrade, all files re-extract. This is acceptable: one-time cost, correct behavior from that point forward. A migration note is printed: `"[graphify] Cache format updated in v5.0 -- re-extracting all files (one-time cost)."` + +--- + +## Files changed + +| File | Change | +|------|--------| +| `graphify/utils.py` | New -- `GraphBundle`, `is_rustworkx()`, `AnyGraph` | +| `graphify/github.py` | New -- GitHub URL resolution + clone/update | +| `graphify/splitter.py` | New -- `split_document()`, `DocumentSection` | +| `graphify/build.py` | `GraphBundle` return; rustworkx + NetworkX dual backend; `also_found_in` dedup attr | +| `graphify/cache.py` | Content-only hash; section cache; migration notice | +| `graphify/cluster.py` | `GraphBundle` input; leiden edge-list conversion | +| `graphify/analyze.py` | `GraphBundle` input; rustworkx parallel betweenness + path | +| `graphify/export.py` | `GraphBundle` input; custom JSON serializer; matplotlib layout | +| `graphify/serve.py` | `GraphBundle` input; custom deserializer; MCP handler updates | +| `graphify/wiki.py` | `GraphBundle` input; dual-path graph traversal | +| `graphify/__main__.py` | `resolve_target()` call; `--dag` flag | +| `graphify/skill.md` | Section node prompt; `--dag`; GitHub URL input; chunking by section | +| `pyproject.toml` | `fast = ["rustworkx"]`; add to `all` | +| `tests/conftest.py` | `graph_backend` fixture | +| `tests/test_github.py` | New | +| `tests/test_splitter.py` | New -- section splitting for PDF, markdown, plain text | +| `tests/test_build_rustworkx.py` | New | +| `tests/test_analyze_rustworkx.py` | New | +| `tests/test_cluster_rustworkx.py` | New | +| `tests/test_dedup.py` | New -- same content different path → single cache entry | + +--- + +## Out of scope (v5.1) + +- Multi-tenant silos and federated graph queries +- Near-deduplication (SimHash/MinHash for ~similar content) +- Entity type registry (Concept, Claim, Person, Method, Dataset, Decision) +- KG storage backend evaluation (Neo4j, Kuzu, LanceDB, TigerGraph) +- Document metadata store (separate from node attributes) +- Private GitHub repo support (token auth) diff --git a/docs/superpowers/specs/2026-04-16-v5.1-design.md b/docs/superpowers/specs/2026-04-16-v5.1-design.md new file mode 100644 index 00000000..5fed33ed --- /dev/null +++ b/docs/superpowers/specs/2026-04-16-v5.1-design.md @@ -0,0 +1,284 @@ +# graphify v5.1 design spec + +**Date:** 2026-04-16 +**Branch:** v5 +**Status:** Draft -- depends on v5.0 +**Milestone:** v5.1 -- enterprise + scaling research + +--- + +## Summary + +v5.1 builds the enterprise layer on top of v5.0's foundation. Four areas: + +1. **Silos** -- multi-tenant graph namespacing with federated cross-silo queries +2. **Near-deduplication** -- SimHash/MinHash fingerprinting to collapse near-duplicate documents before LLM extraction +3. **Entity type registry** -- strict typed entity model replacing the LLM's ad-hoc node decisions +4. **KG scaling research** -- systematic evaluation of storage backends for graphs that exceed RAM + +These are independent and can ship incrementally within the v5.1 milestone. + +--- + +## Change 1: Silos + +### What a silo is + +A silo is a named, isolated graph namespace. Each silo has its own: +- `graph.json` (its node/edge set) +- `cache/` (its extraction cache) +- `manifest.json` (its file manifest) +- Access label (who owns it) + +Silos live under a shared base directory, defaulting to `~/.graphify/silos/`: + +``` +~/.graphify/silos/ + myapp/ + graph.json + cache/ + manifest.json + meta.json ← silo metadata (owner, created_at, description, tags) + research-2026/ + graph.json + cache/ + ... +``` + +### CLI + +```bash +graphify silo create myapp --description "main product repo" +graphify silo list +graphify silo delete myapp +graphify silo info myapp + +# Build graph into a specific silo +graphify . --silo myapp +graphify add github.com/org/repo --silo myapp + +# Query a silo +graphify query "auth flow" --silo myapp +graphify path "SessionManager" "Database" --silo myapp + +# Federated query across silos +graphify query "auth flow" --silos myapp,research-2026 +graphify query "auth flow" --silos all +``` + +### Silo metadata (`meta.json`) + +```json +{ + "name": "myapp", + "description": "main product repo", + "owner": "safishamsi98@gmail.com", + "created_at": "2026-04-16T00:00:00Z", + "updated_at": "2026-04-16T00:00:00Z", + "tags": ["backend", "python"], + "sources": [ + {"type": "github", "url": "github.com/org/repo", "cloned_at": "2026-04-16T00:00:00Z"}, + {"type": "local", "path": "/home/user/docs", "added_at": "2026-04-16T00:00:00Z"} + ], + "node_count": 1243, + "edge_count": 4821 +} +``` + +### Federated queries + +A federated query loads multiple `GraphBundle`s and merges them for query purposes only -- the individual silo graphs are not mutated. The merge is shallow: nodes from different silos with the same ID are kept separate (prefixed with silo name internally). Cross-silo edges can only be INFERRED -- there are no EXTRACTED cross-silo edges unless explicitly added. + +The result of a federated query surfaces which silo each node came from: + +``` +NODE: SessionManager [silo: myapp] + → calls → validate_token [silo: myapp] + → semantically_similar_to → AuthHandler [silo: research-2026, confidence: 0.82] +``` + +### New module: `graphify/silo.py` + +```python +def create_silo(name: str, base_dir: Path, description: str = "") -> Path +def delete_silo(name: str, base_dir: Path) -> None +def list_silos(base_dir: Path) -> list[SiloMeta] +def load_silo(name: str, base_dir: Path) -> GraphBundle +def merge_silos(names: list[str], base_dir: Path) -> GraphBundle # federated, read-only +def update_silo_meta(name: str, base_dir: Path, **fields) -> None +``` + +### Access control (v5.1 scope) + +Owner field in `meta.json` is informational only in v5.1. No authentication or enforcement. True multi-tenant auth (API keys, org membership) is v6 territory. + +--- + +## Change 2: Near-deduplication + +### The problem + +v5.0 exact dedup (SHA256 body-only) handles identical files. Near-dedup handles: +- v1 and v2 of the same paper (85% similar) +- A README copied with minor edits into a wiki +- The same email thread quoted at different levels of truncation + +Without near-dedup, near-duplicate documents produce overlapping concept nodes that pollute community detection and inflate god node scores. + +### Approach: MinHash + LSH + +**Fingerprinting:** Each document (or section in v5.0's model) is shingled (k=5 word shingles) and hashed to a MinHash signature (128 hash functions). Signatures are stored in `~/.graphify/fingerprints/{silo}.bin`. + +**Similarity threshold:** Documents with Jaccard similarity ≥ 0.85 are considered near-duplicates. Threshold is configurable: `--dedup-threshold 0.85`. + +**On detection:** +1. The lower-priority document (later ingested) skips LLM extraction +2. Its nodes are merged into the canonical document's nodes: `also_found_in` list extended +3. A `EXTRACTED` edge `superseded_by` connects the duplicate file node to the canonical file node +4. GRAPH_REPORT surfaces: "3 near-duplicate documents collapsed into 1 canonical source" + +**Library:** `datasketch` (pure Python, no native dependencies). Added as optional dependency: `pip install graphifyy[dedup]`, added to `all`. + +### New module: `graphify/dedup.py` + +```python +def fingerprint(text: str) -> MinHashSignature +def find_near_duplicates( + paths: list[Path], + threshold: float = 0.85, + fingerprint_store: Path | None = None, +) -> list[tuple[Path, Path, float]] # (canonical, duplicate, similarity) +def load_fingerprints(store: Path) -> FingerprintStore +def save_fingerprints(store: Path, fps: FingerprintStore) -> None +``` + +--- + +## Change 3: Entity type registry + +### The problem + +v5.0 section nodes add structure but the concepts within each section are still fully LLM-determined. The same paper produces `"attention mechanism"` in one run and `"self-attention"` in another. Federated queries across silos fail when the same concept has different labels. + +### Solution: typed entity model + +Replace the untyped `file_type: "document"|"paper"|"image"` with a mandatory `entity_type` field on every semantic node: + +| entity_type | Description | Examples | +|-------------|-------------|---------| +| `Concept` | Named idea, algorithm, pattern | "Attention Mechanism", "Leiden Community Detection" | +| `Claim` | Assertion made in source | "BERT outperforms GPT on GLUE" | +| `Person` | Author, researcher, contributor | "Vaswani et al.", "Andrej Karpathy" | +| `Method` | Technique, algorithm, procedure | "Scaled Dot-Product Attention", "Adam optimizer" | +| `Dataset` | Named dataset or benchmark | "ImageNet", "GLUE", "HumanEval" | +| `Decision` | Design decision, rationale node | "Use LayerNorm before attention (Pre-LN)" | +| `Section` | Document section (from splitter.py) | "Page 12", "§3.2 Encoder" | +| `File` | File-level node (code or document) | "session.py", "paper.pdf" | + +### Skill prompt change + +The subagent schema gains `entity_type` as a required field. The node schema: + +```json +{ + "id": "attention_paper_s03_attention_mechanism", + "label": "Attention Mechanism", + "entity_type": "Concept", + "file_type": "paper", + "source_file": "attention_paper.pdf", + "source_location": "§3", + "contained_in": "attention_paper_s03" +} +``` + +### Normalisation + +Entity labels are normalised at build time: lowercased, stripped, deduplicated by (label, entity_type, source_file). Two subagents extracting "Attention Mechanism" and "attention mechanism" from the same section produce one node. + +### Validation + +`validate.py` updated to enforce `entity_type` is one of the registered values. Nodes missing `entity_type` are assigned `"Concept"` with a warning (backward compatibility with v5.0 graphs). + +--- + +## Change 4: KG scaling research + +### The problem + +graphify builds the full graph in RAM. This works for corpora up to ~50k nodes (~500MB RAM). Beyond that: +- `betweenness_centrality` becomes prohibitively slow even with rustworkx parallelism +- `graph.json` serialization produces files >1GB +- Leiden community detection on the full graph fails + +### Research scope + +v5.1 does not pick a storage backend. It **evaluates** four candidates against graphify's specific query patterns: + +| Backend | Type | Key property | +|---------|------|-------------| +| Neo4j | Property graph DB | Mature, Cypher query language, graphify already has `--neo4j` export | +| Kuzu | Embedded property graph | DuckDB-style, no server, fast analytical queries, columnar storage | +| LanceDB | Vector + graph hybrid | Native embedding storage, good for semantic similarity queries | +| TigerGraph | Distributed graph DB | Horizontal scaling, GSQL, designed for 100B+ edge graphs | + +### Evaluation criteria + +For each backend, measure against a 500k-node, 2M-edge synthetic graphify corpus: + +1. **Ingest time** -- time to load `graph.json` into the backend +2. **Betweenness centrality** -- wall time for full graph betweenness +3. **BFS/DFS traversal** -- `graphify query` workload (3-hop neighbourhood) +4. **Shortest path** -- `graphify path` workload +5. **Subgraph extraction** -- pull a community as a subgraph +6. **Memory footprint** -- RSS at peak +7. **Operational complexity** -- setup, persistence, backup + +### Deliverable + +A research report: `docs/scaling-research/2026-KG-backend-evaluation.md` with benchmark numbers, trade-off analysis, and a recommendation for v6 integration. The report is committed to the repo. + +No backend is integrated into graphify in v5.1. The recommendation informs v6. + +### Synthetic corpus generator + +`scripts/gen_corpus.py` -- generates a synthetic `graph.json` at configurable scale (nodes, edges, communities) for reproducible benchmarking. Not shipped in the wheel. + +--- + +## Files changed + +| File | Change | +|------|--------| +| `graphify/silo.py` | New -- silo CRUD, federated merge | +| `graphify/dedup.py` | New -- MinHash fingerprinting, near-dedup detection | +| `graphify/__main__.py` | Silo CLI commands; `--dedup-threshold`; federated query flag | +| `graphify/validate.py` | `entity_type` enforcement | +| `graphify/skill.md` | `entity_type` in node schema; silo-aware subagent prompt | +| `graphify/build.py` | Label normalisation; `entity_type` default assignment | +| `graphify/report.py` | Near-dedup summary; silo source attribution | +| `pyproject.toml` | `dedup = ["datasketch"]`; add to `all` | +| `tests/test_silo.py` | New | +| `tests/test_dedup.py` | New -- MinHash, threshold behaviour, fingerprint persistence | +| `tests/test_entity_types.py` | New -- registry validation, label normalisation | +| `scripts/gen_corpus.py` | New -- synthetic corpus generator (not in wheel) | +| `docs/scaling-research/` | New -- benchmark results directory | + +--- + +## Dependencies on v5.0 + +- `GraphBundle` (utils.py) -- silos load graphs as bundles; federated merge operates on bundles +- Section nodes (splitter.py) -- entity type registry includes `Section`; near-dedup fingerprints sections not whole files +- Content-only cache hash -- near-dedup and exact dedup share the same hash function + +v5.1 cannot ship without v5.0 complete. + +--- + +## Out of scope (v6) + +- True multi-tenant authentication (API keys, org membership, RBAC) +- Streaming graph updates (append-only graph mutation without full rebuild) +- Real-time federated queries (live cross-silo joins) +- Integration of winning storage backend from v5.1 scaling research +- GraphQL API over the knowledge graph