docs: v5.0 and v5.1 design specs -- enterprise foundation

2026-07-12 10:27:11 +00:00 · 2026-04-16 13:25:02 +01:00
parent 2a4608cbb0
commit 022feee5fb
2 changed files with 522 additions and 0 deletions
@@ -0,0 +1,238 @@
+# graphify v5.0 design spec
+
+**Date:** 2026-04-16  
+**Branch:** v5  
+**Status:** Draft  
+**Milestone:** v5.0 -- foundation layer
+
+---
+
+## Summary
+
+v5.0 is the foundation of the graphify enterprise layer. Four independent but coordinated changes:
+
+1. **rustworkx graph backend** -- replaces NetworkX in-memory with a `GraphBundle` abstraction, NetworkX fallback retained
+2. **GitHub repo ingestion** -- `graphify add github.com/org/repo` clones and extracts
+3. **Within-document chunking + section nodes** -- PDFs and markdown split into sections before LLM extraction; sections become first-class nodes anchoring concepts
+4. **Content-based exact deduplication** -- cache keyed on body hash only (not path), same content never extracted twice regardless of filename
+
+These four changes compose: a GitHub repo clone goes through the same chunking + dedup pipeline as a local corpus.
+
+---
+
+## Change 1: rustworkx graph backend
+
+*(Full detail already in `2026-04-16-v5-rustworkx-github-design.md` -- this section summarises only the additions made after senior engineering review)*
+
+### GraphBundle
+
+```python
+# graphify/utils.py  (new)
+@dataclass
+class GraphBundle:
+    graph: AnyGraph                    # PyGraph | PyDiGraph | PyDAG | nx.Graph | nx.DiGraph
+    id_to_idx: dict[str, int]          # empty for NetworkX backend
+    idx_to_id: dict[int, str]          # empty for NetworkX backend
+
+def is_rustworkx(bundle: GraphBundle) -> bool: ...
+```
+
+`build_from_json()` returns `GraphBundle`. All downstream modules (`cluster`, `analyze`, `export`, `serve`, `wiki`) accept `GraphBundle`.
+
+### Key corrections from engineering review
+
+- No `rustworkx.community` module exists -- Louvain stays NetworkX-backed
+- graspologic `leiden()` needs a NetworkX graph -- convert via edge list when rustworkx backend active
+- `PyGraph`/`PyDiGraph` are pyo3 types, no `__dict__` -- monkey-patching forbidden, hence `GraphBundle`
+- DAG cycle handling: `try/except rustworkx.DAGWouldBeCyclic`, not `is_directed_acyclic_graph()`
+- `dijkstra_shortest_paths()` has no `parallel_threshold` -- drop it
+- `git pull --ff-only` broken on shallow clones -- use `git fetch --depth 1 && git reset --hard origin/HEAD`
+
+### Dual-backend testing
+
+`tests/conftest.py`: `graph_backend` fixture parametrized over `["networkx", "rustworkx"]`. Existing 433 tests run on NetworkX fallback; new tests parametrized over both.
+
+---
+
+## Change 2: GitHub repo ingestion
+
+### New file: `graphify/github.py`
+
+**`resolve_target(input: str) -> Path`**  
+Called by `__main__.py` before extraction. Recognises:
+- `https://github.com/org/repo`
+- `github.com/org/repo`
+- `org/repo` (exactly one `/`, no dots)
+
+Returns local clone path or `Path(input)` unchanged.
+
+**`clone_or_update(org, repo, base_dir) -> Path`**  
+- Clone: `~/.graphify/repos/org/repo/`
+- First run: `git clone --depth 1 https://github.com/org/repo <dest>`
+- Update: `git -C <dest> fetch --depth 1 origin && git -C <dest> reset --hard origin/HEAD`
+
+### Error handling
+
+| Condition | Behaviour |
+|-----------|-----------|
+| Repo not found / private | Clear message, exit 1 |
+| git not installed | Message pointing to git install, exit 1 |
+| Network timeout | Retry once, fail with message |
+| Partial clone | Delete dest, report, exit 1 |
+| Fetch/reset fails | Warn, use existing local copy |
+
+---
+
+## Change 3: within-document chunking + section nodes
+
+### The problem
+
+Currently the LLM subagent receives entire file contents. A 300-page PDF = ~150k tokens in one context, risking truncation and shallow extraction. There is no within-document structure in the graph -- a book produces a flat bag of concept nodes with no hierarchy.
+
+### Solution: two-level split
+
+**Level 1 -- processing chunks (invisible in graph)**  
+Documents are split into processing units before being sent to LLM subagents. These are purely a compute concern -- they do not become nodes.
+
+| File type | Split strategy |
+|-----------|---------------|
+| PDF | Per page (pypdf `page.extract_text()`) -- pages grouped into batches of 10 |
+| Markdown / RST | Per heading (`## `, `### `) -- sections split at H2/H3 boundaries |
+| Plain text | Per 2000 words |
+| DOCX | Per heading style (Heading 1 / Heading 2) |
+| Images | One per subagent (unchanged) |
+| Code | AST extraction unchanged, no LLM chunking |
+
+**Level 2 -- section nodes (visible in graph)**  
+Each processing unit produces one **section node** in addition to its concept nodes. Section nodes:
+- `file_type: "section"`
+- `id`: `{doc_stem}_{section_index}` e.g. `attention_paper_p012` (page 12), `readme_s03` (section 3)
+- `label`: heading text (markdown) or `"Page 12"` (PDF) or `"Part 3"` (plain text)
+- `source_file`: parent document path
+- `source_location`: page number or heading anchor
+
+Every concept node extracted from a section gets an `EXTRACTED` edge to its section node (`contained_in`). The section node gets a `contained_in` edge to the file node. This gives a navigable three-level hierarchy:
+
+```
+file node
+  └─ contained_in ← section node (page / heading)
+                        └─ contained_in ← concept node (LLM-extracted)
+```
+
+Concepts are still LLM-extracted and non-deterministic -- but they are now **bounded per section**. The same section on re-run produces the same section node ID, so the structure is reproducible even when concept labels vary.
+
+### Subagent prompt changes
+
+The subagent prompt gains:
+
+```
+Section context: {section_label} ({doc_path}, {location})
+Section ID: {section_node_id}
+
+For every concept node you extract, add a "contained_in" edge from the concept to
+the section node ID above (confidence: EXTRACTED, weight: 1.0).
+Also emit the section node itself as a node with file_type="section".
+```
+
+### Cache key for sections
+
+Sections are cached individually. Cache key: `SHA256(section_text)` -- content only, no path. If the same section appears in two files (e.g. a copied intro paragraph), only one LLM extraction runs. The second file gets the cached nodes with its own section node added.
+
+### New module: `graphify/splitter.py`
+
+```python
+def split_document(path: Path) -> list[DocumentSection]:
+    """Split a document into sections for chunked LLM extraction."""
+
+@dataclass
+class DocumentSection:
+    doc_path: Path
+    section_index: int
+    label: str               # heading text or "Page N"
+    location: str            # "p12", "§3.2", etc.
+    text: str                # content to send to LLM
+    node_id: str             # deterministic section node ID
+    node: dict               # pre-built section node dict
+```
+
+`splitter.py` is called in the skill before subagent dispatch. Its output replaces the flat file list with a section list. Each section becomes an item in the chunk assignment.
+
+### Chunk assignment changes
+
+Currently: chunks of 20-25 **files**.  
+v5.0: chunks of 20-25 **sections** (images still get their own chunk).
+
+A 300-page PDF produces 30 sections (10 pages each) → 2 chunks of 15 sections each, running in parallel. Token load per subagent drops from ~150k to ~15k.
+
+---
+
+## Change 4: content-based exact deduplication
+
+### The problem
+
+Current cache key: `SHA256(content + path)`. Same file, different name = two extractions, two sets of duplicate nodes, double LLM cost.
+
+### Fix: content-only hash
+
+Change `file_hash()` in `cache.py`:
+
+```python
+# v4 (path-dependent)
+h.update(content)
+h.update(b"\x00")
+h.update(str(rel).encode())   # ← causes duplicate cache misses for same content
+
+# v5.0 (content-only)
+h.update(content)
+# path removed
+```
+
+For sections: `SHA256(section_text)` -- section text only, no path or index.
+
+### Dedup at graph build time
+
+When `build_from_json()` encounters two nodes with the same `id` (possible if duplicate files were extracted before this fix landed), last-write wins (existing NetworkX behavior, preserved in GraphBundle). No change needed.
+
+When the same cache entry is loaded for two different paths, the nodes carry `source_file` of the first file that produced them. v5.0 adds a `also_found_in: list[str]` attribute to nodes that are deduplication hits -- surfaced in GRAPH_REPORT as "N duplicate sources collapsed."
+
+### Backward compatibility
+
+Existing cache entries (path-dependent keys) become orphaned -- they will never match the new content-only keys. On first run after upgrade, all files re-extract. This is acceptable: one-time cost, correct behavior from that point forward. A migration note is printed: `"[graphify] Cache format updated in v5.0 -- re-extracting all files (one-time cost)."`
+
+---
+
+## Files changed
+
+| File | Change |
+|------|--------|
+| `graphify/utils.py` | New -- `GraphBundle`, `is_rustworkx()`, `AnyGraph` |
+| `graphify/github.py` | New -- GitHub URL resolution + clone/update |
+| `graphify/splitter.py` | New -- `split_document()`, `DocumentSection` |
+| `graphify/build.py` | `GraphBundle` return; rustworkx + NetworkX dual backend; `also_found_in` dedup attr |
+| `graphify/cache.py` | Content-only hash; section cache; migration notice |
+| `graphify/cluster.py` | `GraphBundle` input; leiden edge-list conversion |
+| `graphify/analyze.py` | `GraphBundle` input; rustworkx parallel betweenness + path |
+| `graphify/export.py` | `GraphBundle` input; custom JSON serializer; matplotlib layout |
+| `graphify/serve.py` | `GraphBundle` input; custom deserializer; MCP handler updates |
+| `graphify/wiki.py` | `GraphBundle` input; dual-path graph traversal |
+| `graphify/__main__.py` | `resolve_target()` call; `--dag` flag |
+| `graphify/skill.md` | Section node prompt; `--dag`; GitHub URL input; chunking by section |
+| `pyproject.toml` | `fast = ["rustworkx"]`; add to `all` |
+| `tests/conftest.py` | `graph_backend` fixture |
+| `tests/test_github.py` | New |
+| `tests/test_splitter.py` | New -- section splitting for PDF, markdown, plain text |
+| `tests/test_build_rustworkx.py` | New |
+| `tests/test_analyze_rustworkx.py` | New |
+| `tests/test_cluster_rustworkx.py` | New |
+| `tests/test_dedup.py` | New -- same content different path → single cache entry |
+
+---
+
+## Out of scope (v5.1)
+
+- Multi-tenant silos and federated graph queries
+- Near-deduplication (SimHash/MinHash for ~similar content)
+- Entity type registry (Concept, Claim, Person, Method, Dataset, Decision)
+- KG storage backend evaluation (Neo4j, Kuzu, LanceDB, TigerGraph)
+- Document metadata store (separate from node attributes)
+- Private GitHub repo support (token auth)
@@ -0,0 +1,284 @@
+# graphify v5.1 design spec
+
+**Date:** 2026-04-16  
+**Branch:** v5  
+**Status:** Draft -- depends on v5.0  
+**Milestone:** v5.1 -- enterprise + scaling research
+
+---
+
+## Summary
+
+v5.1 builds the enterprise layer on top of v5.0's foundation. Four areas:
+
+1. **Silos** -- multi-tenant graph namespacing with federated cross-silo queries
+2. **Near-deduplication** -- SimHash/MinHash fingerprinting to collapse near-duplicate documents before LLM extraction
+3. **Entity type registry** -- strict typed entity model replacing the LLM's ad-hoc node decisions
+4. **KG scaling research** -- systematic evaluation of storage backends for graphs that exceed RAM
+
+These are independent and can ship incrementally within the v5.1 milestone.
+
+---
+
+## Change 1: Silos
+
+### What a silo is
+
+A silo is a named, isolated graph namespace. Each silo has its own:
+- `graph.json` (its node/edge set)
+- `cache/` (its extraction cache)
+- `manifest.json` (its file manifest)
+- Access label (who owns it)
+
+Silos live under a shared base directory, defaulting to `~/.graphify/silos/`:
+
+```
+~/.graphify/silos/
+  myapp/
+    graph.json
+    cache/
+    manifest.json
+    meta.json          ← silo metadata (owner, created_at, description, tags)
+  research-2026/
+    graph.json
+    cache/
+    ...
+```
+
+### CLI
+
+```bash
+graphify silo create myapp --description "main product repo"
+graphify silo list
+graphify silo delete myapp
+graphify silo info myapp
+
+# Build graph into a specific silo
+graphify . --silo myapp
+graphify add github.com/org/repo --silo myapp
+
+# Query a silo
+graphify query "auth flow" --silo myapp
+graphify path "SessionManager" "Database" --silo myapp
+
+# Federated query across silos
+graphify query "auth flow" --silos myapp,research-2026
+graphify query "auth flow" --silos all
+```
+
+### Silo metadata (`meta.json`)
+
+```json
+{
+  "name": "myapp",
+  "description": "main product repo",
+  "owner": "safishamsi98@gmail.com",
+  "created_at": "2026-04-16T00:00:00Z",
+  "updated_at": "2026-04-16T00:00:00Z",
+  "tags": ["backend", "python"],
+  "sources": [
+    {"type": "github", "url": "github.com/org/repo", "cloned_at": "2026-04-16T00:00:00Z"},
+    {"type": "local", "path": "/home/user/docs", "added_at": "2026-04-16T00:00:00Z"}
+  ],
+  "node_count": 1243,
+  "edge_count": 4821
+}
+```
+
+### Federated queries
+
+A federated query loads multiple `GraphBundle`s and merges them for query purposes only -- the individual silo graphs are not mutated. The merge is shallow: nodes from different silos with the same ID are kept separate (prefixed with silo name internally). Cross-silo edges can only be INFERRED -- there are no EXTRACTED cross-silo edges unless explicitly added.
+
+The result of a federated query surfaces which silo each node came from:
+
+```
+NODE: SessionManager  [silo: myapp]
+  → calls → validate_token  [silo: myapp]
+  → semantically_similar_to → AuthHandler  [silo: research-2026, confidence: 0.82]
+```
+
+### New module: `graphify/silo.py`
+
+```python
+def create_silo(name: str, base_dir: Path, description: str = "") -> Path
+def delete_silo(name: str, base_dir: Path) -> None
+def list_silos(base_dir: Path) -> list[SiloMeta]
+def load_silo(name: str, base_dir: Path) -> GraphBundle
+def merge_silos(names: list[str], base_dir: Path) -> GraphBundle   # federated, read-only
+def update_silo_meta(name: str, base_dir: Path, **fields) -> None
+```
+
+### Access control (v5.1 scope)
+
+Owner field in `meta.json` is informational only in v5.1. No authentication or enforcement. True multi-tenant auth (API keys, org membership) is v6 territory.
+
+---
+
+## Change 2: Near-deduplication
+
+### The problem
+
+v5.0 exact dedup (SHA256 body-only) handles identical files. Near-dedup handles:
+- v1 and v2 of the same paper (85% similar)
+- A README copied with minor edits into a wiki
+- The same email thread quoted at different levels of truncation
+
+Without near-dedup, near-duplicate documents produce overlapping concept nodes that pollute community detection and inflate god node scores.
+
+### Approach: MinHash + LSH
+
+**Fingerprinting:** Each document (or section in v5.0's model) is shingled (k=5 word shingles) and hashed to a MinHash signature (128 hash functions). Signatures are stored in `~/.graphify/fingerprints/{silo}.bin`.
+
+**Similarity threshold:** Documents with Jaccard similarity ≥ 0.85 are considered near-duplicates. Threshold is configurable: `--dedup-threshold 0.85`.
+
+**On detection:**
+1. The lower-priority document (later ingested) skips LLM extraction
+2. Its nodes are merged into the canonical document's nodes: `also_found_in` list extended
+3. A `EXTRACTED` edge `superseded_by` connects the duplicate file node to the canonical file node
+4. GRAPH_REPORT surfaces: "3 near-duplicate documents collapsed into 1 canonical source"
+
+**Library:** `datasketch` (pure Python, no native dependencies). Added as optional dependency: `pip install graphifyy[dedup]`, added to `all`.
+
+### New module: `graphify/dedup.py`
+
+```python
+def fingerprint(text: str) -> MinHashSignature
+def find_near_duplicates(
+    paths: list[Path],
+    threshold: float = 0.85,
+    fingerprint_store: Path | None = None,
+) -> list[tuple[Path, Path, float]]   # (canonical, duplicate, similarity)
+def load_fingerprints(store: Path) -> FingerprintStore
+def save_fingerprints(store: Path, fps: FingerprintStore) -> None
+```
+
+---
+
+## Change 3: Entity type registry
+
+### The problem
+
+v5.0 section nodes add structure but the concepts within each section are still fully LLM-determined. The same paper produces `"attention mechanism"` in one run and `"self-attention"` in another. Federated queries across silos fail when the same concept has different labels.
+
+### Solution: typed entity model
+
+Replace the untyped `file_type: "document"|"paper"|"image"` with a mandatory `entity_type` field on every semantic node:
+
+| entity_type | Description | Examples |
+|-------------|-------------|---------|
+| `Concept` | Named idea, algorithm, pattern | "Attention Mechanism", "Leiden Community Detection" |
+| `Claim` | Assertion made in source | "BERT outperforms GPT on GLUE" |
+| `Person` | Author, researcher, contributor | "Vaswani et al.", "Andrej Karpathy" |
+| `Method` | Technique, algorithm, procedure | "Scaled Dot-Product Attention", "Adam optimizer" |
+| `Dataset` | Named dataset or benchmark | "ImageNet", "GLUE", "HumanEval" |
+| `Decision` | Design decision, rationale node | "Use LayerNorm before attention (Pre-LN)" |
+| `Section` | Document section (from splitter.py) | "Page 12", "§3.2 Encoder" |
+| `File` | File-level node (code or document) | "session.py", "paper.pdf" |
+
+### Skill prompt change
+
+The subagent schema gains `entity_type` as a required field. The node schema:
+
+```json
+{
+  "id": "attention_paper_s03_attention_mechanism",
+  "label": "Attention Mechanism",
+  "entity_type": "Concept",
+  "file_type": "paper",
+  "source_file": "attention_paper.pdf",
+  "source_location": "§3",
+  "contained_in": "attention_paper_s03"
+}
+```
+
+### Normalisation
+
+Entity labels are normalised at build time: lowercased, stripped, deduplicated by (label, entity_type, source_file). Two subagents extracting "Attention Mechanism" and "attention mechanism" from the same section produce one node.
+
+### Validation
+
+`validate.py` updated to enforce `entity_type` is one of the registered values. Nodes missing `entity_type` are assigned `"Concept"` with a warning (backward compatibility with v5.0 graphs).
+
+---
+
+## Change 4: KG scaling research
+
+### The problem
+
+graphify builds the full graph in RAM. This works for corpora up to ~50k nodes (~500MB RAM). Beyond that:
+- `betweenness_centrality` becomes prohibitively slow even with rustworkx parallelism
+- `graph.json` serialization produces files >1GB
+- Leiden community detection on the full graph fails
+
+### Research scope
+
+v5.1 does not pick a storage backend. It **evaluates** four candidates against graphify's specific query patterns:
+
+| Backend | Type | Key property |
+|---------|------|-------------|
+| Neo4j | Property graph DB | Mature, Cypher query language, graphify already has `--neo4j` export |
+| Kuzu | Embedded property graph | DuckDB-style, no server, fast analytical queries, columnar storage |
+| LanceDB | Vector + graph hybrid | Native embedding storage, good for semantic similarity queries |
+| TigerGraph | Distributed graph DB | Horizontal scaling, GSQL, designed for 100B+ edge graphs |
+
+### Evaluation criteria
+
+For each backend, measure against a 500k-node, 2M-edge synthetic graphify corpus:
+
+1. **Ingest time** -- time to load `graph.json` into the backend
+2. **Betweenness centrality** -- wall time for full graph betweenness
+3. **BFS/DFS traversal** -- `graphify query` workload (3-hop neighbourhood)
+4. **Shortest path** -- `graphify path` workload
+5. **Subgraph extraction** -- pull a community as a subgraph
+6. **Memory footprint** -- RSS at peak
+7. **Operational complexity** -- setup, persistence, backup
+
+### Deliverable
+
+A research report: `docs/scaling-research/2026-KG-backend-evaluation.md` with benchmark numbers, trade-off analysis, and a recommendation for v6 integration. The report is committed to the repo.
+
+No backend is integrated into graphify in v5.1. The recommendation informs v6.
+
+### Synthetic corpus generator
+
+`scripts/gen_corpus.py` -- generates a synthetic `graph.json` at configurable scale (nodes, edges, communities) for reproducible benchmarking. Not shipped in the wheel.
+
+---
+
+## Files changed
+
+| File | Change |
+|------|--------|
+| `graphify/silo.py` | New -- silo CRUD, federated merge |
+| `graphify/dedup.py` | New -- MinHash fingerprinting, near-dedup detection |
+| `graphify/__main__.py` | Silo CLI commands; `--dedup-threshold`; federated query flag |
+| `graphify/validate.py` | `entity_type` enforcement |
+| `graphify/skill.md` | `entity_type` in node schema; silo-aware subagent prompt |
+| `graphify/build.py` | Label normalisation; `entity_type` default assignment |
+| `graphify/report.py` | Near-dedup summary; silo source attribution |
+| `pyproject.toml` | `dedup = ["datasketch"]`; add to `all` |
+| `tests/test_silo.py` | New |
+| `tests/test_dedup.py` | New -- MinHash, threshold behaviour, fingerprint persistence |
+| `tests/test_entity_types.py` | New -- registry validation, label normalisation |
+| `scripts/gen_corpus.py` | New -- synthetic corpus generator (not in wheel) |
+| `docs/scaling-research/` | New -- benchmark results directory |
+
+---
+
+## Dependencies on v5.0
+
+- `GraphBundle` (utils.py) -- silos load graphs as bundles; federated merge operates on bundles
+- Section nodes (splitter.py) -- entity type registry includes `Section`; near-dedup fingerprints sections not whole files
+- Content-only cache hash -- near-dedup and exact dedup share the same hash function
+
+v5.1 cannot ship without v5.0 complete.
+
+---
+
+## Out of scope (v6)
+
+- True multi-tenant authentication (API keys, org membership, RBAC)
+- Streaming graph updates (append-only graph mutation without full rebuild)
+- Real-time federated queries (live cross-silo joins)
+- Integration of winning storage backend from v5.1 scaling research
+- GraphQL API over the knowledge graph