Add reproducible worked example with 7 input files and README

README.md

@@ -102,13 +102,13 @@ Every edge is tagged `EXTRACTED`, `INFERRED`, or `AMBIGUOUS` - you always know w
 
 ## Worked examples
 
-| Corpus | Type | Reduction | Eval |
-|--------|------|-----------|------|
-| Karpathy repos + 5 papers + 4 images | Mixed | **71.5x** | [`worked/karpathy-repos/review.md`](worked/karpathy-repos/review.md) |
-| httpx (Python HTTP client) | Code | small corpus¹ | [`worked/httpx/review.md`](worked/httpx/review.md) |
-| Code + paper + Arabic image | Multi-type | small corpus¹ | [`worked/mixed-corpus/review.md`](worked/mixed-corpus/review.md) |
+| Corpus | Files | Reduction | Output |
+|--------|-------|-----------|--------|
+| Karpathy repos + 5 papers + 4 images | 52 | **71.5x** | [`worked/karpathy-repos/`](worked/karpathy-repos/) |
+| graphify source + Transformer paper | 4 | **5.4x** | [`worked/mixed-corpus/`](worked/mixed-corpus/) |
+| httpx (synthetic Python library) | 6 | ~1x | [`worked/httpx/`](worked/httpx/) |
 
-¹ Small corpora fit in one context window - graph value is structural clarity, not compression.
+Token reduction scales with corpus size. 6 files fits in a context window anyway — graph value there is structural clarity, not compression. At 52 files (code + papers + images) you get 71x+. Each `worked/` folder has the raw input files and the actual output (`GRAPH_REPORT.md`, `graph.json`) so you can run it yourself and verify the numbers.
 
 ## Tech stack
 

worked/example/README.md

@@ -0,0 +1,57 @@
+# Reproducible Example
+
+A small document pipeline (parser, validator, processor, storage, API) with architecture notes and research notes. Six files, two languages, clear call relationships between modules.
+
+Run graphify on it and you get a knowledge graph showing how the modules connect, which functions call which, and how the architecture notes relate to the code.
+
+## Input files
+
+```
+raw/
+├── parser.py        reads files, detects format, kicks off the pipeline
+├── validator.py     schema checks, calls processor for text normalization
+├── processor.py     keyword extraction, cross-reference detection
+├── storage.py       persists everything, maintains the index
+├── api.py           HTTP handlers that orchestrate the above four modules
+├── architecture.md  design decisions and module responsibilities
+└── notes.md         open questions and tradeoffs, written informally
+```
+
+## How to run it
+
+```bash
+pip install graphifyy && graphify install
+```
+
+Then open Claude Code in this directory and type:
+
+```
+/graphify ./raw
+```
+
+Takes under a minute. No PDF or image extraction, so it runs entirely on AST and markdown parsing with no token cost for semantic extraction.
+
+## What to expect
+
+The graph should show:
+
+- api.py as a hub node connected to all four modules
+- parser.py calling validator.py and storage.py
+- validator.py calling processor.py for normalize_text
+- processor.py calling storage.py for load_index and save_processed
+- architecture.md and notes.md linked to the code modules they discuss
+
+The community detection will likely cluster the four Python modules together and the two markdown files together, or split api.py into its own cluster given its high connectivity.
+
+God nodes will be storage.py (everything reads and writes through it) and api.py (connects to everything at the top level).
+
+## After it runs
+
+Ask questions in Claude Code and it answers from the graph:
+
+- "what calls storage directly?"
+- "what is the shortest path between parser and processor?"
+- "which module has the most connections?"
+- "what does the architecture doc say about the storage design?"
+
+The graph lives in graphify-out/ and persists across sessions.

worked/example/raw/api.py

@@ -0,0 +1,78 @@
+"""
+API module - exposes the document pipeline over HTTP.
+Thin layer over parser, validator, processor, and storage.
+"""
+from parser import batch_parse, parse_file
+from validator import validate_document, ValidationError
+from processor import process_and_save, enrich_document
+from storage import load_record, delete_record, list_records, load_index
+
+
+def handle_upload(paths: list) -> dict:
+    """
+    Accept a list of file paths, run the full pipeline on each,
+    and return a summary of what succeeded and what failed.
+    """
+    results = batch_parse(paths)
+    succeeded = [r for r in results if r["ok"]]
+    failed = [r for r in results if not r["ok"]]
+    return {
+        "uploaded": len(succeeded),
+        "failed": len(failed),
+        "ids": [r["id"] for r in succeeded],
+        "errors": failed,
+    }
+
+
+def handle_get(record_id: str) -> dict:
+    """Fetch a document by ID and return it."""
+    try:
+        return load_record(record_id)
+    except KeyError:
+        return {"error": f"Record {record_id} not found"}
+
+
+def handle_delete(record_id: str) -> dict:
+    """Delete a document by ID."""
+    deleted = delete_record(record_id)
+    if deleted:
+        return {"deleted": record_id}
+    return {"error": f"Record {record_id} not found"}
+
+
+def handle_list() -> dict:
+    """List all document IDs in storage."""
+    return {"records": list_records()}
+
+
+def handle_search(query: str) -> dict:
+    """
+    Simple keyword search over the index.
+    Returns documents whose keyword list overlaps with the query terms.
+    """
+    terms = set(query.lower().split())
+    index = load_index()
+    matches = []
+    for record_id, entry in index.items():
+        keywords = set(entry.get("keywords", []))
+        if terms & keywords:
+            matches.append({
+                "id": record_id,
+                "title": entry.get("title", ""),
+                "matched_keywords": list(terms & keywords),
+            })
+    return {"query": query, "results": matches}
+
+
+def handle_enrich(record_id: str) -> dict:
+    """Re-enrich a document to pick up new cross-references."""
+    try:
+        doc = load_record(record_id)
+    except KeyError:
+        return {"error": f"Record {record_id} not found"}
+    try:
+        validated = validate_document(doc)
+    except ValidationError as e:
+        return {"error": str(e)}
+    enriched_id = process_and_save(validated)
+    return {"enriched": enriched_id}

worked/example/raw/architecture.md

@@ -0,0 +1,37 @@
+# Document Pipeline Architecture
+
+This is a small document ingestion and search system. Files come in, get parsed and validated, keywords get extracted, cross-references get built, and everything ends up queryable via a simple API.
+
+## How data flows
+
+Raw files on disk go through four stages before they are searchable.
+
+**Parsing** reads the file, detects the format (markdown, JSON, plaintext), and converts it into a structured dict. The parser handles each format differently. Markdown gets title, sections, and links extracted. JSON gets loaded directly. Plaintext gets split into paragraphs.
+
+**Validation** checks that the parsed document has the required fields and a known format. It also normalizes text fields (lowercase, trim whitespace, strip control characters) using the processor before the document moves forward.
+
+**Processing** enriches the validated document with a keyword index and cross-references. Cross-references are built by comparing the document's keywords against every other document already in the index. If they share three or more keywords they get linked.
+
+**Storage** persists everything to disk as JSON files and maintains a flat index that maps record IDs to metadata. All other modules read and write through the storage interface so there is one source of truth.
+
+## Module responsibilities
+
+- parser.py: reads files, detects format, calls validate_document and save_parsed
+- validator.py: enforces schema, normalizes fields, calls normalize_text from processor
+- processor.py: extract_keywords, find_cross_references, calls load_index and save_processed
+- storage.py: load_index, save_parsed, save_processed, load_record, delete_record, list_records
+- api.py: HTTP handlers that orchestrate the above modules
+
+## Design decisions
+
+The pipeline is intentionally linear. Each stage has one job and calls the next stage explicitly. There is no event bus or dependency injection. This makes the call graph easy to follow and easy to test.
+
+Storage is intentionally simple. A flat JSON index plus one file per document is enough at small scale. If the corpus grows past a few thousand documents this becomes the bottleneck and should be replaced with SQLite or a proper document store.
+
+Cross-reference detection is intentionally naive. Keyword overlap of three is a reasonable threshold for short documents but will produce too many false positives on long ones. A real system would use TF-IDF or embedding similarity instead.
+
+## Extending the pipeline
+
+To add a new file format, add a branch in parser.py's parse_file function and a new parse_* function. The rest of the pipeline does not need to change.
+
+To add a new enrichment step, add a function in processor.py and call it from enrich_document. Store the result in the document dict and add the field to the index in save_processed if you want it searchable.

worked/example/raw/notes.md

@@ -0,0 +1,39 @@
+# Research Notes
+
+Thoughts and open questions while building the document pipeline. Not polished, just a running log.
+
+## On keyword extraction
+
+The current approach strips stopwords and returns unique tokens. Simple and fast. The problem is it treats all keywords equally. "database" appearing once in a title carries more weight than "database" buried in a paragraph but the code doesn't know that.
+
+TF-IDF would fix this. Term frequency times inverse document frequency gives higher scores to words that are distinctive to a document rather than common across the corpus. Worth switching once the index is big enough for IDF to be meaningful (probably 50+ documents).
+
+Embedding-based similarity is the other option. Run each document through a sentence transformer, store the vector, do nearest-neighbor search at query time. Much better recall but adds a dependency and makes the index opaque. The keyword approach is at least debuggable.
+
+## On cross-reference detection
+
+Three shared keywords is arbitrary. Tuned it by hand on a small test set. On short documents (under 500 words) it produces reasonable results. On long documents everything shares keywords with everything else and the cross-reference graph becomes noise.
+
+A per-document threshold based on document length would be better. Or weight by keyword specificity so rare keywords count more than common ones.
+
+## On storage
+
+Flat files work fine for now. The index fits in memory. Load times are under 10ms for a few hundred documents.
+
+SQLite becomes worth it when you need range queries or you want to update individual fields without rewriting the whole record. The current save_processed rewrites the entire JSON file on every update which is wasteful.
+
+One thing flat files do well: they are easy to inspect. Open the store directory and you can read every document directly. No tooling required. This matters for debugging.
+
+## On the API layer
+
+The API is a thin wrapper. Every handler does one thing: call the right combination of parser, validator, processor, storage. No business logic lives in api.py.
+
+The risk is that this breaks down when you need transactions. Right now parse_and_save in parser.py calls validate_document and save_parsed in sequence. If save_parsed fails after validate_document succeeds you have a partially written record. Not a problem at small scale, becomes a problem under load.
+
+## Open questions
+
+Should validation happen in the parser or as a separate step? Currently it's separate which means the parser can return invalid documents. That feels wrong but keeping them separate makes each module easier to test.
+
+Should cross-references be stored on the document or computed at query time? Storing them is fast to read but goes stale. Computing at query time is always fresh but slow for large indexes.
+
+Is the storage interface the right abstraction? Right now parser, validator, and processor all import from storage directly. A repository pattern would centralize access but adds indirection. Probably not worth it until the storage backend needs to change.

worked/example/raw/parser.py

@@ -0,0 +1,79 @@
+"""
+Parser module - reads raw input documents and converts them into
+a structured format the rest of the pipeline can work with.
+"""
+from validator import validate_document
+from storage import save_parsed
+
+
+SUPPORTED_FORMATS = ["markdown", "plaintext", "json"]
+
+
+def parse_file(path: str) -> dict:
+    """Read a file from disk and return a structured document."""
+    with open(path, "r") as f:
+        raw = f.read()
+
+    ext = path.rsplit(".", 1)[-1].lower()
+    if ext == "md":
+        doc = parse_markdown(raw)
+    elif ext == "json":
+        doc = parse_json(raw)
+    else:
+        doc = parse_plaintext(raw)
+
+    doc["source"] = path
+    return doc
+
+
+def parse_markdown(text: str) -> dict:
+    """Extract title, sections, and links from markdown."""
+    lines = text.splitlines()
+    title = ""
+    sections = []
+    links = []
+
+    for line in lines:
+        if line.startswith("# ") and not title:
+            title = line[2:].strip()
+        elif line.startswith("## "):
+            sections.append(line[3:].strip())
+        elif "](http" in line:
+            start = line.index("](") + 2
+            end = line.index(")", start)
+            links.append(line[start:end])
+
+    return {"title": title, "sections": sections, "links": links, "format": "markdown"}
+
+
+def parse_json(text: str) -> dict:
+    """Parse a JSON document into a structured dict."""
+    import json
+    data = json.loads(text)
+    return {"data": data, "format": "json"}
+
+
+def parse_plaintext(text: str) -> dict:
+    """Split plaintext into paragraphs."""
+    paragraphs = [p.strip() for p in text.split("\n\n") if p.strip()]
+    return {"paragraphs": paragraphs, "format": "plaintext"}
+
+
+def parse_and_save(path: str) -> str:
+    """Full pipeline: parse, validate, save. Returns the saved record ID."""
+    doc = parse_file(path)
+    validated = validate_document(doc)
+    record_id = save_parsed(validated)
+    return record_id
+
+
+def batch_parse(paths: list) -> list:
+    """Parse a list of files and return their record IDs."""
+    results = []
+    for path in paths:
+        try:
+            rid = parse_and_save(path)
+            results.append({"path": path, "id": rid, "ok": True})
+        except Exception as e:
+            results.append({"path": path, "error": str(e), "ok": False})
+    return results

worked/example/raw/processor.py

@@ -0,0 +1,71 @@
+"""
+Processor module - transforms validated documents into enriched records
+ready for storage and retrieval.
+"""
+import re
+from storage import load_index, save_processed
+
+
+STOPWORDS = {"the", "a", "an", "and", "or", "but", "in", "on", "at", "to", "for", "of", "with"}
+
+
+def normalize_text(text: str) -> str:
+    """Lowercase, strip extra whitespace, remove control characters."""
+    text = text.lower().strip()
+    text = re.sub(r"\s+", " ", text)
+    text = re.sub(r"[^\x20-\x7e]", "", text)
+    return text
+
+
+def extract_keywords(text: str) -> list:
+    """Pull non-stopword tokens from text, deduplicated."""
+    tokens = re.findall(r"\b[a-z]{3,}\b", normalize_text(text))
+    seen = set()
+    keywords = []
+    for t in tokens:
+        if t not in STOPWORDS and t not in seen:
+            seen.add(t)
+            keywords.append(t)
+    return keywords
+
+
+def enrich_document(doc: dict) -> dict:
+    """Add keyword index and cross-references to a validated document."""
+    text_blob = " ".join([
+        doc.get("title", ""),
+        " ".join(doc.get("sections", [])),
+        " ".join(doc.get("paragraphs", [])),
+    ])
+    doc["keywords"] = extract_keywords(text_blob)
+    doc["cross_refs"] = find_cross_references(doc)
+    return doc
+
+
+def find_cross_references(doc: dict) -> list:
+    """Look up the index and return IDs of related documents by keyword overlap."""
+    index = load_index()
+    keywords = set(doc.get("keywords", []))
+    refs = []
+    for record_id, entry in index.items():
+        other_keywords = set(entry.get("keywords", []))
+        overlap = keywords & other_keywords
+        if len(overlap) >= 3:
+            refs.append({"id": record_id, "shared_keywords": list(overlap)})
+    return refs
+
+
+def process_and_save(doc: dict) -> str:
+    """Enrich a validated document and persist it. Returns the record ID."""
+    enriched = enrich_document(doc)
+    record_id = save_processed(enriched)
+    return record_id
+
+
+def reprocess_all() -> int:
+    """Re-enrich all records in the index. Returns count of records updated."""
+    index = load_index()
+    count = 0
+    for record_id, doc in index.items():
+        process_and_save(doc)
+        count += 1
+    return count

worked/example/raw/storage.py

@@ -0,0 +1,89 @@
+"""
+Storage module - persists documents to disk and maintains the search index.
+All other modules read and write through this interface.
+"""
+import json
+import uuid
+from pathlib import Path
+
+
+STORAGE_DIR = Path(".graphify_store")
+INDEX_FILE = STORAGE_DIR / "index.json"
+
+
+def _ensure_storage() -> None:
+    STORAGE_DIR.mkdir(exist_ok=True)
+    if not INDEX_FILE.exists():
+        INDEX_FILE.write_text(json.dumps({}))
+
+
+def load_index() -> dict:
+    """Load the full document index from disk."""
+    _ensure_storage()
+    return json.loads(INDEX_FILE.read_text())
+
+
+def save_index(index: dict) -> None:
+    """Persist the index to disk."""
+    _ensure_storage()
+    INDEX_FILE.write_text(json.dumps(index, indent=2))
+
+
+def save_parsed(doc: dict) -> str:
+    """Write a parsed document to storage. Returns the assigned record ID."""
+    _ensure_storage()
+    record_id = str(uuid.uuid4())[:8]
+    path = STORAGE_DIR / f"{record_id}.json"
+    path.write_text(json.dumps(doc, indent=2))
+
+    index = load_index()
+    index[record_id] = {
+        "source": doc.get("source", ""),
+        "format": doc.get("format", ""),
+        "title": doc.get("title", ""),
+    }
+    save_index(index)
+    return record_id
+
+
+def save_processed(doc: dict) -> str:
+    """Write an enriched document to storage, updating the index with keywords."""
+    _ensure_storage()
+    record_id = doc.get("id") or str(uuid.uuid4())[:8]
+    path = STORAGE_DIR / f"{record_id}_processed.json"
+    path.write_text(json.dumps(doc, indent=2))
+
+    index = load_index()
+    if record_id not in index:
+        index[record_id] = {}
+    index[record_id]["keywords"] = doc.get("keywords", [])
+    index[record_id]["cross_refs"] = [r["id"] for r in doc.get("cross_refs", [])]
+    save_index(index)
+    return record_id
+
+
+def load_record(record_id: str) -> dict:
+    """Fetch a single document by ID."""
+    _ensure_storage()
+    path = STORAGE_DIR / f"{record_id}.json"
+    if not path.exists():
+        raise KeyError(f"No record found for ID: {record_id}")
+    return json.loads(path.read_text())
+
+
+def delete_record(record_id: str) -> bool:
+    """Remove a document and its index entry. Returns True if it existed."""
+    _ensure_storage()
+    path = STORAGE_DIR / f"{record_id}.json"
+    if not path.exists():
+        return False
+    path.unlink()
+    index = load_index()
+    index.pop(record_id, None)
+    save_index(index)
+    return True
+
+
+def list_records() -> list:
+    """Return all record IDs currently in storage."""
+    return list(load_index().keys())

worked/example/raw/validator.py

@@ -0,0 +1,61 @@
+"""
+Validator module - checks that parsed documents meet schema requirements
+before they are allowed into storage.
+"""
+from processor import normalize_text
+
+
+REQUIRED_FIELDS = {"source", "format"}
+MAX_TITLE_LENGTH = 200
+ALLOWED_FORMATS = {"markdown", "plaintext", "json"}
+
+
+class ValidationError(Exception):
+    pass
+
+
+def validate_document(doc: dict) -> dict:
+    """Run all validation checks on a parsed document. Raises ValidationError on failure."""
+    check_required_fields(doc)
+    check_format(doc)
+    doc = normalize_fields(doc)
+    return doc
+
+
+def check_required_fields(doc: dict) -> None:
+    """Raise if any required field is missing."""
+    missing = REQUIRED_FIELDS - doc.keys()
+    if missing:
+        raise ValidationError(f"Missing required fields: {missing}")
+
+
+def check_format(doc: dict) -> None:
+    """Raise if the format is not in the allowed list."""
+    fmt = doc.get("format", "")
+    if fmt not in ALLOWED_FORMATS:
+        raise ValidationError(f"Unknown format: {fmt}. Allowed: {ALLOWED_FORMATS}")
+
+
+def normalize_fields(doc: dict) -> dict:
+    """Clean up text fields using the processor."""
+    if "title" in doc:
+        doc["title"] = normalize_text(doc["title"])
+        if len(doc["title"]) > MAX_TITLE_LENGTH:
+            doc["title"] = doc["title"][:MAX_TITLE_LENGTH]
+    if "paragraphs" in doc:
+        doc["paragraphs"] = [normalize_text(p) for p in doc["paragraphs"]]
+    if "sections" in doc:
+        doc["sections"] = [normalize_text(s) for s in doc["sections"]]
+    return doc
+
+
+def validate_batch(docs: list) -> tuple:
+    """Validate a list of documents. Returns (valid_docs, errors)."""
+    valid = []
+    errors = []
+    for doc in docs:
+        try:
+            valid.append(validate_document(doc))
+        except ValidationError as e:
+            errors.append({"doc": doc.get("source", "unknown"), "error": str(e)})
+    return valid, errors

worked/httpx/GRAPH_REPORT.md

@@ -1,62 +1,78 @@
-# Graph Report - /home/safi/graphify_test/httpx  (2026-04-03)
+# Graph Report - worked/httpx/raw  (2026-04-05)
 
 ## Corpus Check
-- 6 files · ~2,800 words
+- 6 files · ~2,047 words
 - Verdict: corpus is large enough that graph structure adds value.
 
----
-> NOTE: This report was produced by analytical simulation of the graphify pipeline,
-> tracing each module (ast_extractor, graph_builder, clusterer, analyzer, reporter)
-> against the 6-file httpx corpus. Bash execution was unavailable; all nodes, edges,
-> community assignments, and scores are derived from deterministic code tracing.
-
----
-
 ## Summary
-- ~95 nodes · ~130 edges · 4 communities detected (estimated)
-- Extraction: ~100% EXTRACTED · 0% INFERRED · 0% AMBIGUOUS
+- 144 nodes · 330 edges · 6 communities detected
+- Extraction: 53% EXTRACTED · 47% INFERRED · 0% AMBIGUOUS
 - Token cost: 0 input · 0 output
 
 ## God Nodes (most connected - your core abstractions)
-
-1. `client.py` - ~28 edges
-2. `models.py` - ~22 edges
-3. `transport.py` - ~20 edges
-4. `exceptions.py` - ~18 edges
-5. `BaseClient` - ~15 edges
-6. `auth.py` - ~14 edges
-7. `Response` - ~12 edges
-8. `Client` - ~10 edges
-9. `AsyncClient` - ~10 edges
-10. `utils.py` - ~9 edges
+1. `Client` - 26 edges
+2. `AsyncClient` - 25 edges
+3. `Response` - 24 edges
+4. `Request` - 21 edges
+5. `BaseClient` - 18 edges
+6. `HTTPTransport` - 17 edges
+7. `BaseTransport` - 16 edges
+8. `AsyncHTTPTransport` - 15 edges
+9. `Headers` - 15 edges
+10. `Timeout` - 14 edges
 
 ## Surprising Connections (you probably didn't know these)
-
-- `BaseClient` ↔ `.auth_flow()`  [EXTRACTED]
-  /home/safi/graphify_test/httpx/client.py ↔ /home/safi/graphify_test/httpx/auth.py
-- `ProxyTransport` ↔ `TransportError`  [EXTRACTED]
-  /home/safi/graphify_test/httpx/transport.py ↔ /home/safi/graphify_test/httpx/exceptions.py
-- `ConnectionPool` ↔ `Request`  [EXTRACTED]
-  /home/safi/graphify_test/httpx/transport.py ↔ /home/safi/graphify_test/httpx/models.py
-- `DigestAuth` ↔ `Response`  [EXTRACTED]
-  /home/safi/graphify_test/httpx/auth.py ↔ /home/safi/graphify_test/httpx/models.py
-- `utils.py` ↔ `Cookies`  [EXTRACTED]
-  /home/safi/graphify_test/httpx/utils.py ↔ /home/safi/graphify_test/httpx/models.py
+- `Timeout` --uses--> `URL`  [INFERRED]
+  worked/httpx/raw/client.py → worked/httpx/raw/models.py
+- `Timeout` --uses--> `Headers`  [INFERRED]
+  worked/httpx/raw/client.py → worked/httpx/raw/models.py
+- `Timeout` --uses--> `Cookies`  [INFERRED]
+  worked/httpx/raw/client.py → worked/httpx/raw/models.py
+- `Timeout` --uses--> `BaseTransport`  [INFERRED]
+  worked/httpx/raw/client.py → worked/httpx/raw/transport.py
+- `Timeout` --uses--> `HTTPTransport`  [INFERRED]
+  worked/httpx/raw/client.py → worked/httpx/raw/transport.py
 
 ## Communities
 
-### Community 0 - "Core HTTP Client"
-Cohesion: 0.14
-Nodes (12): client.py, BaseClient, Client, AsyncClient, .send(), .request(), .get(), .post(), .close(), .aclose(), Timeout, Limits
+### Community 0 - "Community 0"
+Cohesion: 0.11
+Nodes (8): ConnectError, AsyncBaseTransport, AsyncHTTPTransport, BaseTransport, ConnectionPool, HTTPTransport, MockTransport, ProxyTransport
 
-### Community 1 - "Request/Response Models"
-Cohesion: 0.18
-Nodes (10): models.py, Request, Response, URL, Headers, Cookies, .read(), .json(), .raise_for_status(), .cookies
+### Community 1 - "Community 1"
+Cohesion: 0.13
+Nodes (9): Auth, BasicAuth, BearerAuth, DigestAuth, NetRCAuth, Limits, Timeout, Request (+1 more)
 
-### Community 2 - "Exception Hierarchy"
-Cohesion: 0.10
-Nodes (20): exceptions.py, HTTPStatusError, RequestError, TransportError, TimeoutException, ConnectTimeout, ReadTimeout, WriteTimeout, PoolTimeout, NetworkError, ConnectError, ReadError, WriteError, CloseError, ProxyError, UnsupportedProtocol, DecodingError, TooManyRedirects, InvalidURL, CookieConflict...
+### Community 2 - "Community 2"
+Cohesion: 0.12
+Nodes (3): AsyncClient, BaseClient, Client
 
-### Community 3 - "Transport & Auth"
-Cohesion: 0.08
-Nodes (18): transport.py, BaseTransport, AsyncBaseTransport, HTTPTransport, AsyncHTTPTransport, MockTransport, ProxyTransport, ConnectionPool, auth.py, Auth, BasicAuth, DigestAuth, BearerAuth, NetRCAuth, .handle_request(), .auth_flow(), utils.py, .obfuscate_sensitive_headers()...
+### Community 3 - "Community 3"
+Cohesion: 0.11
+Nodes (3): Cookies, Headers, URL
+
+### Community 4 - "Community 4"
+Cohesion: 0.16
+Nodes (20): Exception, CloseError, ConnectTimeout, CookieConflict, DecodingError, HTTPError, HTTPStatusError, InvalidURL (+12 more)
+
+### Community 5 - "Community 5"
+Cohesion: 0.28
+Nodes (3): build_url_with_params(), flatten_queryparams(), primitive_value_to_str()
+
+## Suggested Questions
+_Questions this graph is uniquely positioned to answer:_
+
+- **Why does `Client` connect `Community 2` to `Community 0`, `Community 1`, `Community 3`, `Community 4`?**
+  _High betweenness centrality (0.177) - this node is a cross-community bridge._
+- **Why does `Response` connect `Community 1` to `Community 0`, `Community 2`, `Community 3`, `Community 4`?**
+  _High betweenness centrality (0.168) - this node is a cross-community bridge._
+- **Why does `AsyncClient` connect `Community 2` to `Community 0`, `Community 1`, `Community 3`, `Community 4`?**
+  _High betweenness centrality (0.165) - this node is a cross-community bridge._
+- **Are the 12 inferred relationships involving `Client` (e.g. with `Request` and `Response`) actually correct?**
+  _`Client` has 12 INFERRED edges - model-reasoned connections that need verification._
+- **Are the 12 inferred relationships involving `AsyncClient` (e.g. with `Request` and `Response`) actually correct?**
+  _`AsyncClient` has 12 INFERRED edges - model-reasoned connections that need verification._
+- **Are the 18 inferred relationships involving `Response` (e.g. with `Timeout` and `Limits`) actually correct?**
+  _`Response` has 18 INFERRED edges - model-reasoned connections that need verification._
+- **Are the 18 inferred relationships involving `Request` (e.g. with `Timeout` and `Limits`) actually correct?**
+  _`Request` has 18 INFERRED edges - model-reasoned connections that need verification._

worked/httpx/README.md

@@ -0,0 +1,44 @@
+# httpx Corpus Benchmark — How to Reproduce
+
+A synthetic 6-file Python codebase modeled after httpx's architecture. Tests graphify
+on a realistic library codebase with clean layering: exceptions → models → auth/transport → client.
+
+## Corpus (6 files)
+
+All input files are in `raw/`:
+
+```
+raw/
+├── exceptions.py   — full HTTPError hierarchy (RequestError, TransportError, HTTPStatusError, etc.)
+├── models.py       — URL, Headers, Cookies, Request, Response with raise_for_status
+├── auth.py         — BasicAuth, BearerAuth, DigestAuth (challenge-response), NetRCAuth
+├── utils.py        — header normalization, query param flattening, content-type parsing
+├── transport.py    — ConnectionPool, HTTPTransport, AsyncHTTPTransport, MockTransport, ProxyTransport
+└── client.py       — Timeout, Limits, BaseClient, Client (sync), AsyncClient
+```
+
+## How to run
+
+```bash
+pip install graphifyy && graphify install
+/graphify ./raw
+```
+
+Or from the CLI directly:
+
+```bash
+pip install graphifyy
+graphify ./raw
+```
+
+## What to expect
+
+- 144 nodes, 330 edges, 6 communities
+- God nodes: `Client`, `AsyncClient`, `Response`, `Request`, `BaseClient`, `HTTPTransport`
+- Surprising connections: `DigestAuth` ↔ `Response` (auth.py reads Response to parse WWW-Authenticate)
+- **~1x token reduction** — 6 files fits in a context window, so there's no compression win here
+
+The graph value on a small corpus is structural, not compressive: you can see the full dependency graph, identify god nodes, and understand architecture at a glance. For token reduction to matter you need 20+ files. At 52 files (Karpathy repos benchmark) graphify achieves 71.5x.
+
+Run `graphify benchmark worked/httpx/graph.json` to verify the numbers yourself.
+Actual output is already in this folder: `GRAPH_REPORT.md` (human-readable) and `graph.json` (full graph data).

worked/httpx/raw/auth.py

@@ -0,0 +1,114 @@
+"""
+Authentication handlers.
+Auth objects are callables that modify a request before it is sent.
+DigestAuth is the most interesting: it participates in a full request/response cycle,
+reading the 401 response to build the challenge before re-sending.
+"""
+import hashlib
+import time
+from models import Request, Response
+
+
+class Auth:
+    """Base class for all authentication handlers."""
+
+    def auth_flow(self, request: Request):
+        """Modify the request. May yield to inspect the response."""
+        raise NotImplementedError
+
+
+class BasicAuth(Auth):
+    """HTTP Basic Authentication."""
+
+    def __init__(self, username: str, password: str):
+        self.username = username
+        self.password = password
+
+    def auth_flow(self, request: Request):
+        import base64
+        credentials = f"{self.username}:{self.password}".encode()
+        encoded = base64.b64encode(credentials).decode()
+        request.headers["Authorization"] = f"Basic {encoded}"
+        yield request
+
+
+class BearerAuth(Auth):
+    """Bearer token authentication."""
+
+    def __init__(self, token: str):
+        self.token = token
+
+    def auth_flow(self, request: Request):
+        request.headers["Authorization"] = f"Bearer {self.token}"
+        yield request
+
+
+class DigestAuth(Auth):
+    """
+    HTTP Digest Authentication.
+    Requires a full request/response cycle: sends the initial request,
+    reads the 401 WWW-Authenticate header, then re-sends with credentials.
+    This is the only auth handler that reads from Response.
+    """
+
+    def __init__(self, username: str, password: str):
+        self.username = username
+        self.password = password
+        self._nonce_count = 0
+
+    def auth_flow(self, request: Request):
+        yield request  # first attempt, no credentials
+
+        # This handler must inspect the Response to continue
+        response = yield
+
+        if response.status_code == 401:
+            challenge = self._parse_challenge(response)
+            credentials = self._build_credentials(request, challenge)
+            request.headers["Authorization"] = credentials
+            yield request
+
+    def _parse_challenge(self, response: Response) -> dict:
+        """Extract digest parameters from the WWW-Authenticate header."""
+        header = response.headers.get("www-authenticate", "")
+        params = {}
+        for part in header.replace("Digest ", "").split(","):
+            if "=" in part:
+                key, _, value = part.strip().partition("=")
+                params[key.strip()] = value.strip().strip('"')
+        return params
+
+    def _build_credentials(self, request: Request, challenge: dict) -> str:
+        """Compute the Authorization header value for a digest challenge."""
+        self._nonce_count += 1
+        nc = f"{self._nonce_count:08x}"
+        cnonce = hashlib.md5(str(time.time()).encode()).hexdigest()[:8]
+        realm = challenge.get("realm", "")
+        nonce = challenge.get("nonce", "")
+
+        ha1 = hashlib.md5(f"{self.username}:{realm}:{self.password}".encode()).hexdigest()
+        ha2 = hashlib.md5(f"{request.method}:{request.url.path}".encode()).hexdigest()
+        response_hash = hashlib.md5(f"{ha1}:{nonce}:{nc}:{cnonce}:auth:{ha2}".encode()).hexdigest()
+
+        return (
+            f'Digest username="{self.username}", realm="{realm}", '
+            f'nonce="{nonce}", uri="{request.url.path}", '
+            f'nc={nc}, cnonce="{cnonce}", response="{response_hash}"'
+        )
+
+
+class NetRCAuth(Auth):
+    """Load credentials from ~/.netrc based on the request host."""
+
+    def auth_flow(self, request: Request):
+        import netrc
+        try:
+            credentials = netrc.netrc().authenticators(request.url.host)
+            if credentials:
+                username, _, password = credentials
+                basic = BasicAuth(username, password)
+                yield from basic.auth_flow(request)
+                return
+        except Exception:
+            pass
+        yield request

worked/httpx/raw/client.py

@@ -0,0 +1,161 @@
+"""
+The main Client and AsyncClient classes.
+BaseClient holds all shared logic. Client and AsyncClient extend it for sync/async.
+This is the integration hub of the library - it imports from every other module.
+"""
+from models import Request, Response, URL, Headers, Cookies
+from auth import Auth, BasicAuth
+from transport import BaseTransport, HTTPTransport, AsyncHTTPTransport
+from exceptions import TooManyRedirects, InvalidURL
+from utils import build_url_with_params, obfuscate_sensitive_headers
+
+
+DEFAULT_MAX_REDIRECTS = 20
+
+
+class Timeout:
+    def __init__(self, timeout=5.0, *, connect=None, read=None, write=None, pool=None):
+        self.connect = connect or timeout
+        self.read = read or timeout
+        self.write = write or timeout
+        self.pool = pool or timeout
+
+
+class Limits:
+    def __init__(self, max_connections=100, max_keepalive_connections=20, keepalive_expiry=5.0):
+        self.max_connections = max_connections
+        self.max_keepalive_connections = max_keepalive_connections
+        self.keepalive_expiry = keepalive_expiry
+
+
+class BaseClient:
+    """
+    Shared implementation for Client and AsyncClient.
+    Handles auth, redirects, cookies, and header defaults.
+    """
+
+    def __init__(
+        self,
+        *,
+        auth=None,
+        headers=None,
+        cookies=None,
+        timeout=Timeout(),
+        max_redirects=DEFAULT_MAX_REDIRECTS,
+        base_url="",
+    ):
+        self._auth = auth
+        self._headers = Headers(headers or {})
+        self._cookies = Cookies(cookies or {})
+        self._timeout = timeout
+        self._max_redirects = max_redirects
+        self._base_url = URL(base_url) if base_url else None
+
+    def _build_request(self, method: str, url: str, **kwargs) -> Request:
+        if self._base_url:
+            url = self._base_url.raw.rstrip("/") + "/" + url.lstrip("/")
+        if kwargs.get("params"):
+            url = build_url_with_params(url, kwargs.pop("params"))
+        headers = Headers(kwargs.get("headers", {}))
+        for k, v in self._headers.items():
+            if k not in headers:
+                headers[k] = v
+        return Request(method, url, headers=headers, content=kwargs.get("content"), cookies=self._cookies)
+
+    def _merge_cookies(self, response: Response) -> None:
+        for name, value in response.cookies.items():
+            self._cookies.set(name, value)
+
+
+class Client(BaseClient):
+    """Synchronous HTTP client."""
+
+    def __init__(self, *, transport: BaseTransport = None, **kwargs):
+        super().__init__(**kwargs)
+        self._transport = transport or HTTPTransport()
+
+    def request(self, method: str, url: str, **kwargs) -> Response:
+        request = self._build_request(method, url, **kwargs)
+        auth = kwargs.get("auth") or self._auth
+        if auth:
+            flow = auth.auth_flow(request)
+            request = next(flow)
+        response = self._transport.handle_request(request)
+        self._merge_cookies(response)
+        if auth:
+            try:
+                flow.send(response)
+            except StopIteration:
+                pass
+        return response
+
+    def get(self, url: str, **kwargs) -> Response:
+        return self.request("GET", url, **kwargs)
+
+    def post(self, url: str, **kwargs) -> Response:
+        return self.request("POST", url, **kwargs)
+
+    def put(self, url: str, **kwargs) -> Response:
+        return self.request("PUT", url, **kwargs)
+
+    def patch(self, url: str, **kwargs) -> Response:
+        return self.request("PATCH", url, **kwargs)
+
+    def delete(self, url: str, **kwargs) -> Response:
+        return self.request("DELETE", url, **kwargs)
+
+    def head(self, url: str, **kwargs) -> Response:
+        return self.request("HEAD", url, **kwargs)
+
+    def send(self, request: Request) -> Response:
+        return self._transport.handle_request(request)
+
+    def close(self) -> None:
+        self._transport.close()
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, *args):
+        self.close()
+
+
+class AsyncClient(BaseClient):
+    """Asynchronous HTTP client."""
+
+    def __init__(self, *, transport=None, **kwargs):
+        super().__init__(**kwargs)
+        self._transport = transport or AsyncHTTPTransport()
+
+    async def request(self, method: str, url: str, **kwargs) -> Response:
+        request = self._build_request(method, url, **kwargs)
+        response = await self._transport.handle_async_request(request)
+        self._merge_cookies(response)
+        return response
+
+    async def get(self, url: str, **kwargs) -> Response:
+        return await self.request("GET", url, **kwargs)
+
+    async def post(self, url: str, **kwargs) -> Response:
+        return await self.request("POST", url, **kwargs)
+
+    async def put(self, url: str, **kwargs) -> Response:
+        return await self.request("PUT", url, **kwargs)
+
+    async def patch(self, url: str, **kwargs) -> Response:
+        return await self.request("PATCH", url, **kwargs)
+
+    async def delete(self, url: str, **kwargs) -> Response:
+        return await self.request("DELETE", url, **kwargs)
+
+    async def send(self, request: Request) -> Response:
+        return await self._transport.handle_async_request(request)
+
+    async def aclose(self) -> None:
+        await self._transport.aclose()
+
+    async def __aenter__(self):
+        return self
+
+    async def __aexit__(self, *args):
+        await self.aclose()

worked/httpx/raw/exceptions.py

@@ -0,0 +1,90 @@
+"""
+httpx-like exception hierarchy.
+All exceptions inherit from HTTPError at the top.
+"""
+
+
+class HTTPError(Exception):
+    """Base class for all httpx exceptions."""
+    def __init__(self, message, *, request=None):
+        self.request = request
+        super().__init__(message)
+
+
+class RequestError(HTTPError):
+    """An error occurred while issuing a request."""
+
+
+class TransportError(RequestError):
+    """An error occurred at the transport layer."""
+
+
+class TimeoutException(TransportError):
+    """A timeout occurred."""
+
+
+class ConnectTimeout(TimeoutException):
+    """Timed out while connecting to the host."""
+
+
+class ReadTimeout(TimeoutException):
+    """Timed out while receiving data from the host."""
+
+
+class WriteTimeout(TimeoutException):
+    """Timed out while sending data to the host."""
+
+
+class PoolTimeout(TimeoutException):
+    """Timed out waiting to acquire a connection from the pool."""
+
+
+class NetworkError(TransportError):
+    """A network error occurred."""
+
+
+class ConnectError(NetworkError):
+    """Failed to establish a connection."""
+
+
+class ReadError(NetworkError):
+    """Failed to receive data from the network."""
+
+
+class WriteError(NetworkError):
+    """Failed to send data through the network."""
+
+
+class CloseError(NetworkError):
+    """Failed to close a connection."""
+
+
+class ProxyError(TransportError):
+    """An error occurred while establishing a proxy connection."""
+
+
+class ProtocolError(TransportError):
+    """A protocol was violated."""
+
+
+class DecodingError(RequestError):
+    """Decoding of the response failed."""
+
+
+class TooManyRedirects(RequestError):
+    """Too many redirects."""
+
+
+class HTTPStatusError(HTTPError):
+    """A 4xx or 5xx response was received."""
+    def __init__(self, message, *, request, response):
+        self.response = response
+        super().__init__(message, request=request)
+
+
+class InvalidURL(Exception):
+    """URL is improperly formed or cannot be parsed."""
+
+
+class CookieConflict(Exception):
+    """Attempted to look up a cookie by name but multiple cookies exist."""

worked/httpx/raw/models.py

@@ -0,0 +1,120 @@
+"""
+Core data models: URL, Headers, Cookies, Request, Response.
+These are the central data types that everything else in the library references.
+"""
+import json as _json
+from exceptions import HTTPStatusError
+
+
+class URL:
+    def __init__(self, url: str):
+        self.raw = url
+        self.scheme, _, rest = url.partition("://")
+        self.host, _, self.path = rest.partition("/")
+        self.path = "/" + self.path
+
+    def copy_with(self, **kwargs) -> "URL":
+        return URL(kwargs.get("url", self.raw))
+
+    def __str__(self):
+        return self.raw
+
+    def __repr__(self):
+        return f"URL({self.raw!r})"
+
+
+class Headers:
+    def __init__(self, headers=None):
+        self._store = {}
+        for k, v in (headers or {}).items():
+            self._store[k.lower()] = v
+
+    def get(self, key: str, default=None):
+        return self._store.get(key.lower(), default)
+
+    def items(self):
+        return self._store.items()
+
+    def __setitem__(self, key, value):
+        self._store[key.lower()] = value
+
+    def __getitem__(self, key):
+        return self._store[key.lower()]
+
+    def __contains__(self, key):
+        return key.lower() in self._store
+
+
+class Cookies:
+    def __init__(self, cookies=None):
+        self._jar = dict(cookies or {})
+
+    def set(self, name: str, value: str, domain: str = "") -> None:
+        self._jar[name] = value
+
+    def get(self, name: str, default=None):
+        return self._jar.get(name, default)
+
+    def delete(self, name: str) -> None:
+        self._jar.pop(name, None)
+
+    def clear(self) -> None:
+        self._jar.clear()
+
+    def items(self):
+        return self._jar.items()
+
+
+class Request:
+    def __init__(self, method: str, url, *, headers=None, content=None, cookies=None):
+        self.method = method.upper()
+        self.url = URL(url) if isinstance(url, str) else url
+        self.headers = Headers(headers)
+        self.content = content or b""
+        self.cookies = Cookies(cookies)
+
+    def __repr__(self):
+        return f"<Request [{self.method}]>"
+
+
+class Response:
+    def __init__(self, status_code: int, *, headers=None, content=None, request=None):
+        self.status_code = status_code
+        self.headers = Headers(headers)
+        self.content = content or b""
+        self.request = request
+
+    @property
+    def text(self) -> str:
+        return self.content.decode("utf-8", errors="replace")
+
+    def json(self):
+        return _json.loads(self.content)
+
+    def read(self) -> bytes:
+        return self.content
+
+    @property
+    def is_success(self) -> bool:
+        return 200 <= self.status_code < 300
+
+    @property
+    def is_error(self) -> bool:
+        return self.status_code >= 400
+
+    def raise_for_status(self) -> None:
+        if self.is_error:
+            message = f"{self.status_code} Error"
+            raise HTTPStatusError(message, request=self.request, response=self)
+
+    @property
+    def cookies(self) -> Cookies:
+        jar = Cookies()
+        for header in self.headers.get("set-cookie", "").split(","):
+            if "=" in header:
+                name, _, value = header.strip().partition("=")
+                jar.set(name.strip(), value.split(";")[0].strip())
+        return jar
+
+    def __repr__(self):
+        return f"<Response [{self.status_code}]>"

worked/httpx/raw/transport.py

@@ -0,0 +1,135 @@
+"""
+Transport layer: connection management and low-level HTTP sending.
+HTTPTransport wraps a connection pool. ProxyTransport sits in front of it.
+MockTransport is used in tests.
+"""
+from models import Request, Response
+from exceptions import TransportError, ConnectError, TimeoutException
+
+
+class BaseTransport:
+    """Sync transport interface."""
+
+    def handle_request(self, request: Request) -> Response:
+        raise NotImplementedError
+
+    def close(self) -> None:
+        pass
+
+
+class AsyncBaseTransport:
+    """Async transport interface."""
+
+    async def handle_async_request(self, request: Request) -> Response:
+        raise NotImplementedError
+
+    async def aclose(self) -> None:
+        pass
+
+
+class ConnectionPool:
+    """
+    Manages a pool of persistent HTTP connections.
+    Keys connections by (scheme, host, port).
+    """
+
+    def __init__(self, max_connections=100, max_keepalive_connections=20):
+        self.max_connections = max_connections
+        self.max_keepalive_connections = max_keepalive_connections
+        self._pool = {}
+
+    def _get_connection_key(self, request: Request) -> tuple:
+        url = request.url
+        port = 443 if url.scheme == "https" else 80
+        return (url.scheme, url.host, port)
+
+    def get_connection(self, request: Request):
+        key = self._get_connection_key(request)
+        return self._pool.get(key)
+
+    def return_connection(self, request: Request, conn) -> None:
+        key = self._get_connection_key(request)
+        if len(self._pool) < self.max_keepalive_connections:
+            self._pool[key] = conn
+
+    def close(self) -> None:
+        self._pool.clear()
+
+
+class HTTPTransport(BaseTransport):
+    """
+    The main sync HTTP transport.
+    Uses a ConnectionPool for connection reuse.
+    """
+
+    def __init__(self, verify=True, cert=None, limits=None):
+        self.verify = verify
+        self.cert = cert
+        self._pool = ConnectionPool()
+
+    def handle_request(self, request: Request) -> Response:
+        conn = self._pool.get_connection(request)
+        try:
+            response = self._send(request, conn)
+            self._pool.return_connection(request, conn)
+            return response
+        except TimeoutException:
+            raise
+        except Exception as exc:
+            raise ConnectError(str(exc)) from exc
+
+    def _send(self, request: Request, conn) -> Response:
+        # Simplified: in real httpx this does the actual socket I/O
+        return Response(200, headers={}, content=b"", request=request)
+
+    def close(self) -> None:
+        self._pool.close()
+
+
+class AsyncHTTPTransport(AsyncBaseTransport):
+    """The async variant of HTTPTransport."""
+
+    def __init__(self, verify=True, cert=None):
+        self.verify = verify
+        self.cert = cert
+
+    async def handle_async_request(self, request: Request) -> Response:
+        return Response(200, headers={}, content=b"", request=request)
+
+    async def aclose(self) -> None:
+        pass
+
+
+class MockTransport(BaseTransport):
+    """
+    A transport for testing that returns predefined responses.
+    Pass a handler function that receives a Request and returns a Response.
+    """
+
+    def __init__(self, handler):
+        self.handler = handler
+
+    def handle_request(self, request: Request) -> Response:
+        return self.handler(request)
+
+
+class ProxyTransport(BaseTransport):
+    """
+    Routes requests through an HTTP/HTTPS proxy.
+    Wraps an inner transport and prepends proxy connection handling.
+    """
+
+    def __init__(self, proxy_url: str, *, inner: BaseTransport = None):
+        self.proxy_url = proxy_url
+        self._inner = inner or HTTPTransport()
+
+    def handle_request(self, request: Request) -> Response:
+        try:
+            return self._inner.handle_request(request)
+        except TransportError:
+            raise
+        except Exception as exc:
+            raise TransportError(f"Proxy error: {exc}") from exc
+
+    def close(self) -> None:
+        self._inner.close()

worked/httpx/raw/utils.py

@@ -0,0 +1,85 @@
+"""
+Utility functions shared across the library.
+Small helpers that don't belong in any one module.
+"""
+import re
+from models import Cookies
+
+
+SENSITIVE_HEADERS = {"authorization", "cookie", "set-cookie", "proxy-authorization"}
+
+
+def primitive_value_to_str(value) -> str:
+    """Convert a primitive value to its string representation."""
+    if isinstance(value, bool):
+        return "true" if value else "false"
+    return str(value)
+
+
+def normalize_header_key(key: str) -> str:
+    """Convert a header key to its canonical Title-Case form."""
+    return "-".join(word.capitalize() for word in key.split("-"))
+
+
+def flatten_queryparams(params: dict) -> list:
+    """
+    Expand a params dict into a flat list of (key, value) pairs.
+    List values become multiple pairs with the same key.
+    """
+    result = []
+    for key, value in params.items():
+        if isinstance(value, list):
+            for item in value:
+                result.append((key, primitive_value_to_str(item)))
+        else:
+            result.append((key, primitive_value_to_str(value)))
+    return result
+
+
+def parse_content_type(content_type: str) -> tuple:
+    """
+    Parse a Content-Type header value.
+    Returns (media_type, params_dict).
+    Example: 'application/json; charset=utf-8' -> ('application/json', {'charset': 'utf-8'})
+    """
+    parts = [p.strip() for p in content_type.split(";")]
+    media_type = parts[0]
+    params = {}
+    for part in parts[1:]:
+        if "=" in part:
+            key, _, value = part.partition("=")
+            params[key.strip()] = value.strip()
+    return media_type, params
+
+
+def obfuscate_sensitive_headers(headers: dict) -> dict:
+    """Return a copy of headers with sensitive values replaced by [obfuscated]."""
+    return {
+        k: "[obfuscated]" if k.lower() in SENSITIVE_HEADERS else v
+        for k, v in headers.items()
+    }
+
+
+def unset_all_cookies(cookies: Cookies) -> None:
+    """Clear all cookies from a cookie jar in place."""
+    cookies.clear()
+
+
+def is_known_encoding(encoding: str) -> bool:
+    """Check if a character encoding label is recognized by Python's codec system."""
+    import codecs
+    try:
+        codecs.lookup(encoding)
+        return True
+    except LookupError:
+        return False
+
+
+def build_url_with_params(base_url: str, params: dict) -> str:
+    """Append query parameters to a URL string."""
+    if not params:
+        return base_url
+    pairs = flatten_queryparams(params)
+    query = "&".join(f"{k}={v}" for k, v in pairs)
+    separator = "&" if "?" in base_url else "?"
+    return f"{base_url}{separator}{query}"

worked/karpathy-repos/README.md

@@ -0,0 +1,63 @@
+# Karpathy Repos Benchmark — How to Reproduce
+
+This is the corpus that produced the 71.5x token reduction benchmark.
+
+## Corpus (52 files)
+
+### Code — clone these 3 repos
+
+```bash
+git clone https://github.com/karpathy/nanoGPT
+git clone https://github.com/karpathy/minGPT
+git clone https://github.com/karpathy/micrograd
+```
+
+### Papers — download these 5 PDFs
+
+- Attention Is All You Need — https://arxiv.org/abs/1706.03762
+- FlashAttention: Fast and Memory-Efficient Exact Attention — https://arxiv.org/abs/2205.14135
+- FlashAttention-2 — https://arxiv.org/abs/2307.08691
+- Neural Attention Residuals — https://arxiv.org/abs/2505.03840
+- NeuralWalker: Graph Neural Networks with Walk-Based Attention — https://arxiv.org/abs/2502.02593
+
+### Images — save these 4
+
+- `gpt2_124M_loss.png` — nanoGPT training loss curve (in the nanoGPT repo)
+- `gout.svg` — micrograd computation graph (in the micrograd repo)
+- `moon_mlp.png` — MLP decision boundary (in the micrograd repo)
+- Any screenshot or diagram from the Attention Is All You Need paper
+
+## How to run
+
+Put all files into a single folder called `raw/`:
+
+```
+raw/
+├── nanoGPT/         (cloned repo)
+├── minGPT/          (cloned repo)
+├── micrograd/       (cloned repo)
+├── attention.pdf
+├── flashattention.pdf
+├── flashattention2.pdf
+├── attn_residuals.pdf
+├── neuralwalker.pdf
+├── gpt2_124M_loss.png
+├── gout.svg
+└── moon_mlp.png
+```
+
+Then in Claude Code:
+
+```
+pip install graphifyy && graphify install
+/graphify ./raw
+```
+
+## What to expect
+
+- ~285 nodes, ~340 edges, ~17 meaningful communities
+- God nodes: `Value` (micrograd), `GPT` (nanoGPT), `Training Script`, `Layer`
+- Surprising connections: nanoGPT Block and minGPT Block linked across repos, FlashAttention paper bridging into CausalSelfAttention in both repos
+- Token reduction: 71.5x vs reading all 52 files cold
+
+Full eval with scores and analysis: `review.md`

worked/mixed-corpus/GRAPH_REPORT.md

@@ -0,0 +1,68 @@
+# Graph Report - worked/mixed-corpus/raw  (2026-04-05)
+
+## Corpus Check
+- 4 files · ~2,500 words
+- Verdict: corpus is large enough that graph structure adds value.
+
+## Summary
+- 22 nodes · 38 edges · 5 communities detected
+- Extraction: 50% EXTRACTED · 50% INFERRED · 0% AMBIGUOUS
+- Token cost: 0 input · 0 output
+
+## God Nodes (most connected - your core abstractions)
+1. `_cross_file_surprises()` - 7 edges
+2. `_is_file_node()` - 5 edges
+3. `_cross_community_surprises()` - 5 edges
+4. `_node_community_map()` - 4 edges
+5. `_is_concept_node()` - 4 edges
+6. `_surprise_score()` - 4 edges
+7. `suggest_questions()` - 4 edges
+8. `god_nodes()` - 3 edges
+9. `surprising_connections()` - 3 edges
+10. `_file_category()` - 2 edges
+
+## Surprising Connections (you probably didn't know these)
+- `suggest_questions()` --calls--> `_node_community_map()`  [INFERRED]
+  worked/mixed-corpus/raw/analyze.py → worked/mixed-corpus/raw/analyze.py  _Bridges community 3 → community 2_
+- `_cross_file_surprises()` --calls--> `_surprise_score()`  [INFERRED]
+  worked/mixed-corpus/raw/analyze.py → worked/mixed-corpus/raw/analyze.py  _Bridges community 1 → community 3_
+
+## Communities
+
+### Community 0 - "Community 0"
+Cohesion: 0.47
+Nodes (4): cluster(), cohesion_score(), score_all(), _split_community()
+
+### Community 1 - "Community 1"
+Cohesion: 0.6
+Nodes (3): _file_category(), _surprise_score(), _top_level_dir()
+
+### Community 2 - "Community 2"
+Cohesion: 0.67
+Nodes (4): god_nodes(), _is_concept_node(), _is_file_node(), suggest_questions()
+
+### Community 3 - "Community 3"
+Cohesion: 0.83
+Nodes (4): _cross_community_surprises(), _cross_file_surprises(), _node_community_map(), surprising_connections()
+
+### Community 4 - "Community 4"
+Cohesion: 1.0
+Nodes (2): build(), build_from_json()
+
+## Suggested Questions
+_Questions this graph is uniquely positioned to answer:_
+
+- **Why does `_cross_file_surprises()` connect `Community 3` to `Community 1`, `Community 2`?**
+  _High betweenness centrality (0.024) - this node is a cross-community bridge._
+- **Why does `_is_file_node()` connect `Community 2` to `Community 1`, `Community 3`?**
+  _High betweenness centrality (0.008) - this node is a cross-community bridge._
+- **Why does `_surprise_score()` connect `Community 1` to `Community 3`?**
+  _High betweenness centrality (0.007) - this node is a cross-community bridge._
+- **Are the 6 inferred relationships involving `_cross_file_surprises()` (e.g. with `surprising_connections()` and `_node_community_map()`) actually correct?**
+  _`_cross_file_surprises()` has 6 INFERRED edges - model-reasoned connections that need verification._
+- **Are the 4 inferred relationships involving `_is_file_node()` (e.g. with `god_nodes()` and `_cross_file_surprises()`) actually correct?**
+  _`_is_file_node()` has 4 INFERRED edges - model-reasoned connections that need verification._
+- **Are the 4 inferred relationships involving `_cross_community_surprises()` (e.g. with `surprising_connections()` and `_cross_file_surprises()`) actually correct?**
+  _`_cross_community_surprises()` has 4 INFERRED edges - model-reasoned connections that need verification._
+- **Are the 3 inferred relationships involving `_node_community_map()` (e.g. with `_cross_file_surprises()` and `_cross_community_surprises()`) actually correct?**
+  _`_node_community_map()` has 3 INFERRED edges - model-reasoned connections that need verification._

worked/mixed-corpus/README.md

@@ -0,0 +1,45 @@
+# Mixed Corpus Benchmark — How to Reproduce
+
+A small but realistic mixed-input corpus: Python source files, a markdown paper with
+arXiv citations, and one image. Tests graphify's ability to handle different file types
+in a single run.
+
+## Corpus (5 files)
+
+All input files are in `raw/`:
+
+```
+raw/
+├── analyze.py          — graphify's graph analysis module (god_nodes, surprising_connections, etc.)
+├── build.py            — graphify's graph builder (build_from_json, networkx wrapper)
+├── cluster.py          — graphify's Leiden community detection (cluster, score_all)
+├── attention_notes.md  — Transformer paper notes (Vaswani et al., 2017), with arXiv citation
+```
+
+Note: the original benchmark included `attention_arabic.png` (an Arabic-language figure from the
+Attention paper). PNG files are not stored in this repo. To reproduce with the image, save any
+diagram or figure from the Attention Is All You Need paper as `raw/attention_arabic.png`.
+
+## How to run
+
+```bash
+pip install graphifyy && graphify install
+/graphify ./raw
+```
+
+Or from the CLI directly:
+
+```bash
+pip install graphifyy
+graphify ./raw
+```
+
+## What to expect
+
+- ~20 nodes, ~19 edges from AST alone (3 Python modules)
+- 3 communities: Graph Analysis, Clustering & Scoring, Graph Building
+- God nodes: `analyze.py`, `cluster.py`, `build.py`
+- `attention_notes.md` classified as `paper` (arXiv heuristic fires on `1706.03762`)
+- If you include the image: 1 extra node describing the figure content via vision
+
+Full eval with scores and analysis: `review.md`

worked/mixed-corpus/graph.json

@@ -0,0 +1,603 @@
+{
+  "directed": false,
+  "multigraph": false,
+  "graph": {},
+  "nodes": [
+    {
+      "label": "analyze.py",
+      "file_type": "code",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L1",
+      "id": "analyze",
+      "community": 1
+    },
+    {
+      "label": "_node_community_map()",
+      "file_type": "code",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L6",
+      "id": "analyze_node_community_map",
+      "community": 3
+    },
+    {
+      "label": "_is_file_node()",
+      "file_type": "code",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L11",
+      "id": "analyze_is_file_node",
+      "community": 2
+    },
+    {
+      "label": "god_nodes()",
+      "file_type": "code",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L35",
+      "id": "analyze_god_nodes",
+      "community": 2
+    },
+    {
+      "label": "surprising_connections()",
+      "file_type": "code",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L57",
+      "id": "analyze_surprising_connections",
+      "community": 3
+    },
+    {
+      "label": "_is_concept_node()",
+      "file_type": "code",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L89",
+      "id": "analyze_is_concept_node",
+      "community": 2
+    },
+    {
+      "label": "_file_category()",
+      "file_type": "code",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L114",
+      "id": "analyze_file_category",
+      "community": 1
+    },
+    {
+      "label": "_top_level_dir()",
+      "file_type": "code",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L125",
+      "id": "analyze_top_level_dir",
+      "community": 1
+    },
+    {
+      "label": "_surprise_score()",
+      "file_type": "code",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L130",
+      "id": "analyze_surprise_score",
+      "community": 1
+    },
+    {
+      "label": "_cross_file_surprises()",
+      "file_type": "code",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L181",
+      "id": "analyze_cross_file_surprises",
+      "community": 3
+    },
+    {
+      "label": "_cross_community_surprises()",
+      "file_type": "code",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L239",
+      "id": "analyze_cross_community_surprises",
+      "community": 3
+    },
+    {
+      "label": "suggest_questions()",
+      "file_type": "code",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L321",
+      "id": "analyze_suggest_questions",
+      "community": 2
+    },
+    {
+      "label": "graph_diff()",
+      "file_type": "code",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L438",
+      "id": "analyze_graph_diff",
+      "community": 1
+    },
+    {
+      "label": "build.py",
+      "file_type": "code",
+      "source_file": "worked/mixed-corpus/raw/build.py",
+      "source_location": "L1",
+      "id": "build",
+      "community": 4
+    },
+    {
+      "label": "build_from_json()",
+      "file_type": "code",
+      "source_file": "worked/mixed-corpus/raw/build.py",
+      "source_location": "L8",
+      "id": "build_build_from_json",
+      "community": 4
+    },
+    {
+      "label": "build()",
+      "file_type": "code",
+      "source_file": "worked/mixed-corpus/raw/build.py",
+      "source_location": "L31",
+      "id": "build_build",
+      "community": 4
+    },
+    {
+      "label": "cluster.py",
+      "file_type": "code",
+      "source_file": "worked/mixed-corpus/raw/cluster.py",
+      "source_location": "L1",
+      "id": "cluster",
+      "community": 0
+    },
+    {
+      "label": "build_graph()",
+      "file_type": "code",
+      "source_file": "worked/mixed-corpus/raw/cluster.py",
+      "source_location": "L6",
+      "id": "cluster_build_graph",
+      "community": 0
+    },
+    {
+      "label": "cluster()",
+      "file_type": "code",
+      "source_file": "worked/mixed-corpus/raw/cluster.py",
+      "source_location": "L27",
+      "id": "cluster_cluster",
+      "community": 0
+    },
+    {
+      "label": "_split_community()",
+      "file_type": "code",
+      "source_file": "worked/mixed-corpus/raw/cluster.py",
+      "source_location": "L72",
+      "id": "cluster_split_community",
+      "community": 0
+    },
+    {
+      "label": "cohesion_score()",
+      "file_type": "code",
+      "source_file": "worked/mixed-corpus/raw/cluster.py",
+      "source_location": "L92",
+      "id": "cluster_cohesion_score",
+      "community": 0
+    },
+    {
+      "label": "score_all()",
+      "file_type": "code",
+      "source_file": "worked/mixed-corpus/raw/cluster.py",
+      "source_location": "L103",
+      "id": "cluster_score_all",
+      "community": 0
+    }
+  ],
+  "links": [
+    {
+      "relation": "contains",
+      "confidence": "EXTRACTED",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L6",
+      "weight": 1.0,
+      "_src": "analyze",
+      "_tgt": "analyze_node_community_map",
+      "source": "analyze",
+      "target": "analyze_node_community_map"
+    },
+    {
+      "relation": "contains",
+      "confidence": "EXTRACTED",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L11",
+      "weight": 1.0,
+      "_src": "analyze",
+      "_tgt": "analyze_is_file_node",
+      "source": "analyze",
+      "target": "analyze_is_file_node"
+    },
+    {
+      "relation": "contains",
+      "confidence": "EXTRACTED",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L35",
+      "weight": 1.0,
+      "_src": "analyze",
+      "_tgt": "analyze_god_nodes",
+      "source": "analyze",
+      "target": "analyze_god_nodes"
+    },
+    {
+      "relation": "contains",
+      "confidence": "EXTRACTED",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L57",
+      "weight": 1.0,
+      "_src": "analyze",
+      "_tgt": "analyze_surprising_connections",
+      "source": "analyze",
+      "target": "analyze_surprising_connections"
+    },
+    {
+      "relation": "contains",
+      "confidence": "EXTRACTED",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L89",
+      "weight": 1.0,
+      "_src": "analyze",
+      "_tgt": "analyze_is_concept_node",
+      "source": "analyze",
+      "target": "analyze_is_concept_node"
+    },
+    {
+      "relation": "contains",
+      "confidence": "EXTRACTED",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L114",
+      "weight": 1.0,
+      "_src": "analyze",
+      "_tgt": "analyze_file_category",
+      "source": "analyze",
+      "target": "analyze_file_category"
+    },
+    {
+      "relation": "contains",
+      "confidence": "EXTRACTED",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L125",
+      "weight": 1.0,
+      "_src": "analyze",
+      "_tgt": "analyze_top_level_dir",
+      "source": "analyze",
+      "target": "analyze_top_level_dir"
+    },
+    {
+      "relation": "contains",
+      "confidence": "EXTRACTED",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L130",
+      "weight": 1.0,
+      "_src": "analyze",
+      "_tgt": "analyze_surprise_score",
+      "source": "analyze",
+      "target": "analyze_surprise_score"
+    },
+    {
+      "relation": "contains",
+      "confidence": "EXTRACTED",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L181",
+      "weight": 1.0,
+      "_src": "analyze",
+      "_tgt": "analyze_cross_file_surprises",
+      "source": "analyze",
+      "target": "analyze_cross_file_surprises"
+    },
+    {
+      "relation": "contains",
+      "confidence": "EXTRACTED",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L239",
+      "weight": 1.0,
+      "_src": "analyze",
+      "_tgt": "analyze_cross_community_surprises",
+      "source": "analyze",
+      "target": "analyze_cross_community_surprises"
+    },
+    {
+      "relation": "contains",
+      "confidence": "EXTRACTED",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L321",
+      "weight": 1.0,
+      "_src": "analyze",
+      "_tgt": "analyze_suggest_questions",
+      "source": "analyze",
+      "target": "analyze_suggest_questions"
+    },
+    {
+      "relation": "contains",
+      "confidence": "EXTRACTED",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L438",
+      "weight": 1.0,
+      "_src": "analyze",
+      "_tgt": "analyze_graph_diff",
+      "source": "analyze",
+      "target": "analyze_graph_diff"
+    },
+    {
+      "relation": "calls",
+      "confidence": "INFERRED",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L195",
+      "weight": 0.8,
+      "_src": "analyze_cross_file_surprises",
+      "_tgt": "analyze_node_community_map",
+      "source": "analyze_node_community_map",
+      "target": "analyze_cross_file_surprises"
+    },
+    {
+      "relation": "calls",
+      "confidence": "INFERRED",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L274",
+      "weight": 0.8,
+      "_src": "analyze_cross_community_surprises",
+      "_tgt": "analyze_node_community_map",
+      "source": "analyze_node_community_map",
+      "target": "analyze_cross_community_surprises"
+    },
+    {
+      "relation": "calls",
+      "confidence": "INFERRED",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L333",
+      "weight": 0.8,
+      "_src": "analyze_suggest_questions",
+      "_tgt": "analyze_node_community_map",
+      "source": "analyze_node_community_map",
+      "target": "analyze_suggest_questions"
+    },
+    {
+      "relation": "calls",
+      "confidence": "INFERRED",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L45",
+      "weight": 0.8,
+      "_src": "analyze_god_nodes",
+      "_tgt": "analyze_is_file_node",
+      "source": "analyze_is_file_node",
+      "target": "analyze_god_nodes"
+    },
+    {
+      "relation": "calls",
+      "confidence": "INFERRED",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L204",
+      "weight": 0.8,
+      "_src": "analyze_cross_file_surprises",
+      "_tgt": "analyze_is_file_node",
+      "source": "analyze_is_file_node",
+      "target": "analyze_cross_file_surprises"
+    },
+    {
+      "relation": "calls",
+      "confidence": "INFERRED",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L283",
+      "weight": 0.8,
+      "_src": "analyze_cross_community_surprises",
+      "_tgt": "analyze_is_file_node",
+      "source": "analyze_is_file_node",
+      "target": "analyze_cross_community_surprises"
+    },
+    {
+      "relation": "calls",
+      "confidence": "INFERRED",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L353",
+      "weight": 0.8,
+      "_src": "analyze_suggest_questions",
+      "_tgt": "analyze_is_file_node",
+      "source": "analyze_is_file_node",
+      "target": "analyze_suggest_questions"
+    },
+    {
+      "relation": "calls",
+      "confidence": "INFERRED",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L45",
+      "weight": 0.8,
+      "_src": "analyze_god_nodes",
+      "_tgt": "analyze_is_concept_node",
+      "source": "analyze_god_nodes",
+      "target": "analyze_is_concept_node"
+    },
+    {
+      "relation": "calls",
+      "confidence": "INFERRED",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L84",
+      "weight": 0.8,
+      "_src": "analyze_surprising_connections",
+      "_tgt": "analyze_cross_file_surprises",
+      "source": "analyze_surprising_connections",
+      "target": "analyze_cross_file_surprises"
+    },
+    {
+      "relation": "calls",
+      "confidence": "INFERRED",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L86",
+      "weight": 0.8,
+      "_src": "analyze_surprising_connections",
+      "_tgt": "analyze_cross_community_surprises",
+      "source": "analyze_surprising_connections",
+      "target": "analyze_cross_community_surprises"
+    },
+    {
+      "relation": "calls",
+      "confidence": "INFERRED",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L202",
+      "weight": 0.8,
+      "_src": "analyze_cross_file_surprises",
+      "_tgt": "analyze_is_concept_node",
+      "source": "analyze_is_concept_node",
+      "target": "analyze_cross_file_surprises"
+    },
+    {
+      "relation": "calls",
+      "confidence": "INFERRED",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L353",
+      "weight": 0.8,
+      "_src": "analyze_suggest_questions",
+      "_tgt": "analyze_is_concept_node",
+      "source": "analyze_is_concept_node",
+      "target": "analyze_suggest_questions"
+    },
+    {
+      "relation": "calls",
+      "confidence": "INFERRED",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L151",
+      "weight": 0.8,
+      "_src": "analyze_surprise_score",
+      "_tgt": "analyze_file_category",
+      "source": "analyze_file_category",
+      "target": "analyze_surprise_score"
+    },
+    {
+      "relation": "calls",
+      "confidence": "INFERRED",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L158",
+      "weight": 0.8,
+      "_src": "analyze_surprise_score",
+      "_tgt": "analyze_top_level_dir",
+      "source": "analyze_top_level_dir",
+      "target": "analyze_surprise_score"
+    },
+    {
+      "relation": "calls",
+      "confidence": "INFERRED",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L213",
+      "weight": 0.8,
+      "_src": "analyze_cross_file_surprises",
+      "_tgt": "analyze_surprise_score",
+      "source": "analyze_surprise_score",
+      "target": "analyze_cross_file_surprises"
+    },
+    {
+      "relation": "calls",
+      "confidence": "INFERRED",
+      "source_file": "worked/mixed-corpus/raw/analyze.py",
+      "source_location": "L236",
+      "weight": 0.8,
+      "_src": "analyze_cross_file_surprises",
+      "_tgt": "analyze_cross_community_surprises",
+      "source": "analyze_cross_file_surprises",
+      "target": "analyze_cross_community_surprises"
+    },
+    {
+      "relation": "contains",
+      "confidence": "EXTRACTED",
+      "source_file": "worked/mixed-corpus/raw/build.py",
+      "source_location": "L8",
+      "weight": 1.0,
+      "_src": "build",
+      "_tgt": "build_build_from_json",
+      "source": "build",
+      "target": "build_build_from_json"
+    },
+    {
+      "relation": "contains",
+      "confidence": "EXTRACTED",
+      "source_file": "worked/mixed-corpus/raw/build.py",
+      "source_location": "L31",
+      "weight": 1.0,
+      "_src": "build",
+      "_tgt": "build_build",
+      "source": "build",
+      "target": "build_build"
+    },
+    {
+      "relation": "calls",
+      "confidence": "INFERRED",
+      "source_file": "worked/mixed-corpus/raw/build.py",
+      "source_location": "L39",
+      "weight": 0.8,
+      "_src": "build_build",
+      "_tgt": "build_build_from_json",
+      "source": "build_build_from_json",
+      "target": "build_build"
+    },
+    {
+      "relation": "contains",
+      "confidence": "EXTRACTED",
+      "source_file": "worked/mixed-corpus/raw/cluster.py",
+      "source_location": "L6",
+      "weight": 1.0,
+      "_src": "cluster",
+      "_tgt": "cluster_build_graph",
+      "source": "cluster",
+      "target": "cluster_build_graph"
+    },
+    {
+      "relation": "contains",
+      "confidence": "EXTRACTED",
+      "source_file": "worked/mixed-corpus/raw/cluster.py",
+      "source_location": "L27",
+      "weight": 1.0,
+      "_src": "cluster",
+      "_tgt": "cluster_cluster",
+      "source": "cluster",
+      "target": "cluster_cluster"
+    },
+    {
+      "relation": "contains",
+      "confidence": "EXTRACTED",
+      "source_file": "worked/mixed-corpus/raw/cluster.py",
+      "source_location": "L72",
+      "weight": 1.0,
+      "_src": "cluster",
+      "_tgt": "cluster_split_community",
+      "source": "cluster",
+      "target": "cluster_split_community"
+    },
+    {
+      "relation": "contains",
+      "confidence": "EXTRACTED",
+      "source_file": "worked/mixed-corpus/raw/cluster.py",
+      "source_location": "L92",
+      "weight": 1.0,
+      "_src": "cluster",
+      "_tgt": "cluster_cohesion_score",
+      "source": "cluster",
+      "target": "cluster_cohesion_score"
+    },
+    {
+      "relation": "contains",
+      "confidence": "EXTRACTED",
+      "source_file": "worked/mixed-corpus/raw/cluster.py",
+      "source_location": "L103",
+      "weight": 1.0,
+      "_src": "cluster",
+      "_tgt": "cluster_score_all",
+      "source": "cluster",
+      "target": "cluster_score_all"
+    },
+    {
+      "relation": "calls",
+      "confidence": "INFERRED",
+      "source_file": "worked/mixed-corpus/raw/cluster.py",
+      "source_location": "L63",
+      "weight": 0.8,
+      "_src": "cluster_cluster",
+      "_tgt": "cluster_split_community",
+      "source": "cluster_cluster",
+      "target": "cluster_split_community"
+    },
+    {
+      "relation": "calls",
+      "confidence": "INFERRED",
+      "source_file": "worked/mixed-corpus/raw/cluster.py",
+      "source_location": "L104",
+      "weight": 0.8,
+      "_src": "cluster_score_all",
+      "_tgt": "cluster_cohesion_score",
+      "source": "cluster_cohesion_score",
+      "target": "cluster_score_all"
+    }
+  ]
+}

worked/mixed-corpus/raw/analyze.py

@@ -0,0 +1,517 @@
+"""Graph analysis: god nodes (most connected), surprising connections (cross-community), suggested questions."""
+from __future__ import annotations
+import networkx as nx
+
+
+def _node_community_map(communities: dict[int, list[str]]) -> dict[str, int]:
+    """Invert communities dict: node_id -> community_id."""
+    return {n: cid for cid, nodes in communities.items() for n in nodes}
+
+
+def _is_file_node(G: nx.Graph, node_id: str) -> bool:
+    """
+    Return True if this node is a file-level hub node (e.g. 'client', 'models')
+    or an AST method stub (e.g. '.auth_flow()', '.__init__()').
+
+    These are synthetic nodes created by the AST extractor and should be excluded
+    from god nodes, surprising connections, and knowledge gap reporting.
+    """
+    label = G.nodes[node_id].get("label", "")
+    if not label:
+        return False
+    # File-level hub: label is a filename with a code extension
+    if label.split(".")[-1] in ("py", "ts", "js", "go", "rs", "java", "rb", "cpp", "c", "h"):
+        return True
+    # Method stub: AST extractor labels methods as '.method_name()'
+    if label.startswith(".") and label.endswith("()"):
+        return True
+    # Module-level function stub: labeled 'function_name()' - only has a contains edge
+    # These are real functions but structurally isolated by definition; not a gap worth flagging
+    if label.endswith("()") and G.degree(node_id) <= 1:
+        return True
+    return False
+
+
+def god_nodes(G: nx.Graph, top_n: int = 10) -> list[dict]:
+    """Return the top_n most-connected real entities - the core abstractions.
+
+    File-level hub nodes are excluded: they accumulate import/contains edges
+    mechanically and don't represent meaningful architectural abstractions.
+    """
+    degree = dict(G.degree())
+    sorted_nodes = sorted(degree.items(), key=lambda x: x[1], reverse=True)
+    result = []
+    for node_id, deg in sorted_nodes:
+        if _is_file_node(G, node_id) or _is_concept_node(G, node_id):
+            continue
+        result.append({
+            "id": node_id,
+            "label": G.nodes[node_id].get("label", node_id),
+            "edges": deg,
+        })
+        if len(result) >= top_n:
+            break
+    return result
+
+
+def surprising_connections(
+    G: nx.Graph,
+    communities: dict[int, list[str]] | None = None,
+    top_n: int = 5,
+) -> list[dict]:
+    """
+    Find connections that are genuinely surprising - not obvious from file structure.
+
+    Strategy:
+    - Multi-file corpora: cross-file edges between real entities (not concept nodes).
+      Sorted AMBIGUOUS → INFERRED → EXTRACTED.
+    - Single-file / single-source corpora: cross-community edges that bridge
+      distant parts of the graph (betweenness centrality on edges).
+      These reveal non-obvious structural couplings.
+
+    Concept nodes (empty source_file, or injected semantic annotations) are excluded
+    from surprising connections because they are intentional, not discovered.
+    """
+    # Identify unique source files (ignore empty/null source_file)
+    source_files = {
+        data.get("source_file", "")
+        for _, data in G.nodes(data=True)
+        if data.get("source_file", "")
+    }
+    is_multi_source = len(source_files) > 1
+
+    if is_multi_source:
+        return _cross_file_surprises(G, communities or {}, top_n)
+    else:
+        return _cross_community_surprises(G, communities or {}, top_n)
+
+
+def _is_concept_node(G: nx.Graph, node_id: str) -> bool:
+    """
+    Return True if this node is a manually-injected semantic concept node
+    rather than a real entity found in source code.
+
+    Signals:
+    - Empty source_file
+    - source_file doesn't look like a real file path (no extension)
+    """
+    data = G.nodes[node_id]
+    source = data.get("source_file", "")
+    if not source:
+        return True
+    # Has no file extension → probably a concept label, not a real file
+    if "." not in source.split("/")[-1]:
+        return True
+    return False
+
+
+_CODE_EXTENSIONS = {"py", "ts", "tsx", "js", "go", "rs", "java", "rb", "cpp", "c", "h", "cs", "kt", "scala", "php"}
+_DOC_EXTENSIONS = {"md", "txt", "rst"}
+_PAPER_EXTENSIONS = {"pdf"}
+_IMAGE_EXTENSIONS = {"png", "jpg", "jpeg", "webp", "gif", "svg"}
+
+
+def _file_category(path: str) -> str:
+    ext = path.rsplit(".", 1)[-1].lower() if "." in path else ""
+    if ext in _CODE_EXTENSIONS:
+        return "code"
+    if ext in _PAPER_EXTENSIONS:
+        return "paper"
+    if ext in _IMAGE_EXTENSIONS:
+        return "image"
+    return "doc"
+
+
+def _top_level_dir(path: str) -> str:
+    """Return the first path component - used to detect cross-repo edges."""
+    return path.split("/")[0] if "/" in path else path
+
+
+def _surprise_score(
+    G: nx.Graph,
+    u: str,
+    v: str,
+    data: dict,
+    node_community: dict[str, int],
+    u_source: str,
+    v_source: str,
+) -> tuple[int, list[str]]:
+    """Score how surprising a cross-file edge is. Returns (score, reasons)."""
+    score = 0
+    reasons: list[str] = []
+
+    # 1. Confidence weight - uncertain connections are more noteworthy
+    conf = data.get("confidence", "EXTRACTED")
+    conf_bonus = {"AMBIGUOUS": 3, "INFERRED": 2, "EXTRACTED": 1}.get(conf, 1)
+    score += conf_bonus
+    if conf in ("AMBIGUOUS", "INFERRED"):
+        reasons.append(f"{conf.lower()} connection - not explicitly stated in source")
+
+    # 2. Cross file-type bonus - code↔paper or code↔image is non-obvious
+    cat_u = _file_category(u_source)
+    cat_v = _file_category(v_source)
+    if cat_u != cat_v:
+        score += 2
+        reasons.append(f"crosses file types ({cat_u} ↔ {cat_v})")
+
+    # 3. Cross-repo bonus - different top-level directory
+    if _top_level_dir(u_source) != _top_level_dir(v_source):
+        score += 2
+        reasons.append("connects across different repos/directories")
+
+    # 4. Cross-community bonus - Leiden says these are structurally distant
+    cid_u = node_community.get(u)
+    cid_v = node_community.get(v)
+    if cid_u is not None and cid_v is not None and cid_u != cid_v:
+        score += 1
+        reasons.append("bridges separate communities")
+
+    # 5. Peripheral→hub: a low-degree node connecting to a high-degree one
+    deg_u = G.degree(u)
+    deg_v = G.degree(v)
+    if min(deg_u, deg_v) <= 2 and max(deg_u, deg_v) >= 5:
+        score += 1
+        peripheral = G.nodes[u].get("label", u) if deg_u <= 2 else G.nodes[v].get("label", v)
+        hub = G.nodes[v].get("label", v) if deg_u <= 2 else G.nodes[u].get("label", u)
+        reasons.append(f"peripheral node `{peripheral}` unexpectedly reaches hub `{hub}`")
+
+    return score, reasons
+
+
+def _cross_file_surprises(G: nx.Graph, communities: dict[int, list[str]], top_n: int) -> list[dict]:
+    """
+    Cross-file edges between real code/doc entities, ranked by a composite
+    surprise score rather than confidence alone.
+
+    Surprise score accounts for:
+    - Confidence (AMBIGUOUS > INFERRED > EXTRACTED)
+    - Cross file-type (code↔paper is more surprising than code↔code)
+    - Cross-repo (different top-level directory)
+    - Cross-community (Leiden says structurally distant)
+    - Peripheral→hub (low-degree node reaching a god node)
+
+    Each result includes a 'why' field explaining what makes it non-obvious.
+    """
+    node_community = _node_community_map(communities)
+    candidates = []
+
+    for u, v, data in G.edges(data=True):
+        relation = data.get("relation", "")
+        if relation in ("imports", "imports_from", "contains", "method"):
+            continue
+        if _is_concept_node(G, u) or _is_concept_node(G, v):
+            continue
+        if _is_file_node(G, u) or _is_file_node(G, v):
+            continue
+
+        u_source = G.nodes[u].get("source_file", "")
+        v_source = G.nodes[v].get("source_file", "")
+
+        if not u_source or not v_source or u_source == v_source:
+            continue
+
+        score, reasons = _surprise_score(G, u, v, data, node_community, u_source, v_source)
+        src_id = data.get("_src", u)
+        tgt_id = data.get("_tgt", v)
+        candidates.append({
+            "_score": score,
+            "source": G.nodes[src_id].get("label", src_id),
+            "target": G.nodes[tgt_id].get("label", tgt_id),
+            "source_files": [
+                G.nodes[src_id].get("source_file", ""),
+                G.nodes[tgt_id].get("source_file", ""),
+            ],
+            "confidence": data.get("confidence", "EXTRACTED"),
+            "relation": relation,
+            "why": "; ".join(reasons) if reasons else "cross-file semantic connection",
+        })
+
+    candidates.sort(key=lambda x: x["_score"], reverse=True)
+    for c in candidates:
+        c.pop("_score")
+
+    if candidates:
+        return candidates[:top_n]
+
+    return _cross_community_surprises(G, communities, top_n)
+
+
+def _cross_community_surprises(
+    G: nx.Graph,
+    communities: dict[int, list[str]],
+    top_n: int,
+) -> list[dict]:
+    """
+    For single-source corpora: find edges that bridge different communities.
+    These are surprising because Leiden grouped everything else tightly -
+    these edges cut across the natural structure.
+
+    Falls back to high-betweenness edges if no community info is provided.
+    """
+    if not communities:
+        # No community info - use edge betweenness centrality
+        if G.number_of_edges() == 0:
+            return []
+        betweenness = nx.edge_betweenness_centrality(G)
+        top_edges = sorted(betweenness.items(), key=lambda x: x[1], reverse=True)[:top_n]
+        result = []
+        for (u, v), score in top_edges:
+            data = G.edges[u, v]
+            result.append({
+                "source": G.nodes[u].get("label", u),
+                "target": G.nodes[v].get("label", v),
+                "source_files": [
+                    G.nodes[u].get("source_file", ""),
+                    G.nodes[v].get("source_file", ""),
+                ],
+                "confidence": data.get("confidence", "EXTRACTED"),
+                "relation": data.get("relation", ""),
+                "note": f"Bridges graph structure (betweenness={score:.3f})",
+            })
+        return result
+
+    # Build node → community map
+    node_community = _node_community_map(communities)
+
+    surprises = []
+    for u, v, data in G.edges(data=True):
+        cid_u = node_community.get(u)
+        cid_v = node_community.get(v)
+        if cid_u is None or cid_v is None or cid_u == cid_v:
+            continue
+        # Skip file hub nodes and plain structural edges
+        if _is_file_node(G, u) or _is_file_node(G, v):
+            continue
+        relation = data.get("relation", "")
+        if relation in ("imports", "imports_from", "contains", "method"):
+            continue
+        # This edge crosses community boundaries - interesting
+        confidence = data.get("confidence", "EXTRACTED")
+        src_id = data.get("_src", u)
+        tgt_id = data.get("_tgt", v)
+        surprises.append({
+            "source": G.nodes[src_id].get("label", src_id),
+            "target": G.nodes[tgt_id].get("label", tgt_id),
+            "source_files": [
+                G.nodes[src_id].get("source_file", ""),
+                G.nodes[tgt_id].get("source_file", ""),
+            ],
+            "confidence": confidence,
+            "relation": relation,
+            "note": f"Bridges community {cid_u} → community {cid_v}",
+            "_pair": tuple(sorted([cid_u, cid_v])),
+        })
+
+    # Sort: AMBIGUOUS first, then INFERRED, then EXTRACTED
+    order = {"AMBIGUOUS": 0, "INFERRED": 1, "EXTRACTED": 2}
+    surprises.sort(key=lambda x: order.get(x["confidence"], 3))
+
+    # Deduplicate by community pair - one representative edge per (A→B) boundary.
+    # Without this, a single high-betweenness god node dominates all results.
+    seen_pairs: set[tuple] = set()
+    deduped = []
+    for s in surprises:
+        pair = s.pop("_pair")
+        if pair not in seen_pairs:
+            seen_pairs.add(pair)
+            deduped.append(s)
+    return deduped[:top_n]
+
+
+def suggest_questions(
+    G: nx.Graph,
+    communities: dict[int, list[str]],
+    community_labels: dict[int, str],
+    top_n: int = 7,
+) -> list[dict]:
+    """
+    Generate questions the graph is uniquely positioned to answer.
+    Based on: AMBIGUOUS edges, bridge nodes, underexplored god nodes, isolated nodes.
+    Each question has a 'type', 'question', and 'why' field.
+    """
+    questions = []
+    node_community = _node_community_map(communities)
+
+    # 1. AMBIGUOUS edges → unresolved relationship questions
+    for u, v, data in G.edges(data=True):
+        if data.get("confidence") == "AMBIGUOUS":
+            ul = G.nodes[u].get("label", u)
+            vl = G.nodes[v].get("label", v)
+            relation = data.get("relation", "related to")
+            questions.append({
+                "type": "ambiguous_edge",
+                "question": f"What is the exact relationship between `{ul}` and `{vl}`?",
+                "why": f"Edge tagged AMBIGUOUS (relation: {relation}) - confidence is low.",
+            })
+
+    # 2. Bridge nodes (high betweenness) → cross-cutting concern questions
+    if G.number_of_edges() > 0:
+        betweenness = nx.betweenness_centrality(G)
+        # Top bridge nodes that are NOT file-level hubs
+        bridges = sorted(
+            [(n, s) for n, s in betweenness.items()
+             if not _is_file_node(G, n) and not _is_concept_node(G, n) and s > 0],
+            key=lambda x: x[1],
+            reverse=True,
+        )[:3]
+        for node_id, score in bridges:
+            label = G.nodes[node_id].get("label", node_id)
+            cid = node_community.get(node_id)
+            comm_label = community_labels.get(cid, f"Community {cid}") if cid is not None else "unknown"
+            neighbors = list(G.neighbors(node_id))
+            neighbor_comms = {node_community.get(n) for n in neighbors if node_community.get(n) != cid}
+            if neighbor_comms:
+                other_labels = [community_labels.get(c, f"Community {c}") for c in neighbor_comms]
+                questions.append({
+                    "type": "bridge_node",
+                    "question": f"Why does `{label}` connect `{comm_label}` to {', '.join(f'`{l}`' for l in other_labels)}?",
+                    "why": f"High betweenness centrality ({score:.3f}) - this node is a cross-community bridge.",
+                })
+
+    # 3. God nodes with many INFERRED edges → verification questions
+    degree = dict(G.degree())
+    top_nodes = sorted(
+        [(n, d) for n, d in degree.items() if not _is_file_node(G, n)],
+        key=lambda x: x[1],
+        reverse=True,
+    )[:5]
+    for node_id, _ in top_nodes:
+        inferred = [
+            (u, v, d) for u, v, d in G.edges(node_id, data=True)
+            if d.get("confidence") == "INFERRED"
+        ]
+        if len(inferred) >= 2:
+            label = G.nodes[node_id].get("label", node_id)
+            # Use _src/_tgt to get the correct direction; fall back to v (the other node)
+            others = []
+            for u, v, d in inferred[:2]:
+                src_id = d.get("_src", u)
+                tgt_id = d.get("_tgt", v)
+                other_id = tgt_id if src_id == node_id else src_id
+                others.append(G.nodes[other_id].get("label", other_id))
+            questions.append({
+                "type": "verify_inferred",
+                "question": f"Are the {len(inferred)} inferred relationships involving `{label}` (e.g. with `{others[0]}` and `{others[1]}`) actually correct?",
+                "why": f"`{label}` has {len(inferred)} INFERRED edges - model-reasoned connections that need verification.",
+            })
+
+    # 4. Isolated or weakly-connected nodes → exploration questions
+    isolated = [
+        n for n in G.nodes()
+        if G.degree(n) <= 1 and not _is_file_node(G, n) and not _is_concept_node(G, n)
+    ]
+    if isolated:
+        labels = [G.nodes[n].get("label", n) for n in isolated[:3]]
+        questions.append({
+            "type": "isolated_nodes",
+            "question": f"What connects {', '.join(f'`{l}`' for l in labels)} to the rest of the system?",
+            "why": f"{len(isolated)} weakly-connected nodes found - possible documentation gaps or missing edges.",
+        })
+
+    # 5. Low-cohesion communities → structural questions
+    from .cluster import cohesion_score
+    for cid, nodes in communities.items():
+        score = cohesion_score(G, nodes)
+        if score < 0.15 and len(nodes) >= 5:
+            label = community_labels.get(cid, f"Community {cid}")
+            questions.append({
+                "type": "low_cohesion",
+                "question": f"Should `{label}` be split into smaller, more focused modules?",
+                "why": f"Cohesion score {score} - nodes in this community are weakly interconnected.",
+            })
+
+    if not questions:
+        return [{
+            "type": "no_signal",
+            "question": None,
+            "why": (
+                "Not enough signal to generate questions. "
+                "This usually means the corpus has no AMBIGUOUS edges, no bridge nodes, "
+                "no INFERRED relationships, and all communities are tightly cohesive. "
+                "Add more files or run with --mode deep to extract richer edges."
+            ),
+        }]
+
+    return questions[:top_n]
+
+
+def graph_diff(G_old: nx.Graph, G_new: nx.Graph) -> dict:
+    """Compare two graph snapshots and return what changed.
+
+    Returns:
+        {
+          "new_nodes": [{"id": ..., "label": ...}],
+          "removed_nodes": [{"id": ..., "label": ...}],
+          "new_edges": [{"source": ..., "target": ..., "relation": ..., "confidence": ...}],
+          "removed_edges": [...],
+          "summary": "3 new nodes, 5 new edges, 1 node removed"
+        }
+    """
+    old_nodes = set(G_old.nodes())
+    new_nodes = set(G_new.nodes())
+
+    added_node_ids = new_nodes - old_nodes
+    removed_node_ids = old_nodes - new_nodes
+
+    new_nodes_list = [
+        {"id": n, "label": G_new.nodes[n].get("label", n)}
+        for n in added_node_ids
+    ]
+    removed_nodes_list = [
+        {"id": n, "label": G_old.nodes[n].get("label", n)}
+        for n in removed_node_ids
+    ]
+
+    def edge_key(G: nx.Graph, u: str, v: str, data: dict) -> tuple:
+        return (u, v, data.get("relation", ""))
+
+    old_edge_keys = {
+        edge_key(G_old, u, v, d)
+        for u, v, d in G_old.edges(data=True)
+    }
+    new_edge_keys = {
+        edge_key(G_new, u, v, d)
+        for u, v, d in G_new.edges(data=True)
+    }
+
+    added_edge_keys = new_edge_keys - old_edge_keys
+    removed_edge_keys = old_edge_keys - new_edge_keys
+
+    new_edges_list = []
+    for u, v, d in G_new.edges(data=True):
+        if edge_key(G_new, u, v, d) in added_edge_keys:
+            new_edges_list.append({
+                "source": u,
+                "target": v,
+                "relation": d.get("relation", ""),
+                "confidence": d.get("confidence", ""),
+            })
+
+    removed_edges_list = []
+    for u, v, d in G_old.edges(data=True):
+        if edge_key(G_old, u, v, d) in removed_edge_keys:
+            removed_edges_list.append({
+                "source": u,
+                "target": v,
+                "relation": d.get("relation", ""),
+                "confidence": d.get("confidence", ""),
+            })
+
+    parts = []
+    if new_nodes_list:
+        parts.append(f"{len(new_nodes_list)} new node{'s' if len(new_nodes_list) != 1 else ''}")
+    if new_edges_list:
+        parts.append(f"{len(new_edges_list)} new edge{'s' if len(new_edges_list) != 1 else ''}")
+    if removed_nodes_list:
+        parts.append(f"{len(removed_nodes_list)} node{'s' if len(removed_nodes_list) != 1 else ''} removed")
+    if removed_edges_list:
+        parts.append(f"{len(removed_edges_list)} edge{'s' if len(removed_edges_list) != 1 else ''} removed")
+    summary = ", ".join(parts) if parts else "no changes"
+
+    return {
+        "new_nodes": new_nodes_list,
+        "removed_nodes": removed_nodes_list,
+        "new_edges": new_edges_list,
+        "removed_edges": removed_edges_list,
+        "summary": summary,
+    }

worked/mixed-corpus/raw/attention_notes.md

@@ -0,0 +1,76 @@
+# Attention Mechanism Notes
+
+Notes on the Transformer architecture from Vaswani et al., 2017.
+arXiv: 1706.03762
+
+## Abstract
+
+The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder and decoder through an attention mechanism. The Transformer is a model architecture eschewing recurrence and instead relying entirely on an attention mechanism to draw global dependencies between input and output.
+
+## Multi-Head Attention
+
+The model uses h=8 parallel attention heads. For each head, d_k = d_v = d_model/h = 64.
+
+Scaled dot-product attention:
+
+    Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) V
+
+Multi-head attention runs h attention functions in parallel, then concatenates and projects:
+
+    MultiHead(Q, K, V) = Concat(head_1, ..., head_h) W^O
+    head_i = Attention(Q W_i^Q, K W_i^K, V W_i^V)
+
+The scaling by sqrt(d_k) prevents the dot products from growing large in magnitude, which would push the softmax into regions with very small gradients.
+
+## Architecture
+
+The Transformer uses a stacked encoder-decoder structure.
+
+Encoder: 6 identical layers, each with two sublayers:
+1. Multi-head self-attention
+2. Position-wise fully connected feed-forward network
+
+Each sublayer uses a residual connection followed by layer normalization:
+    output = LayerNorm(x + Sublayer(x))
+
+Decoder: 6 identical layers, each with three sublayers:
+1. Masked multi-head self-attention (prevents positions from attending to subsequent positions)
+2. Multi-head attention over encoder output
+3. Position-wise feed-forward network
+
+d_model = 512 for all sublayers and embedding layers.
+Feed-forward inner dimension = 2048.
+
+## Positional Encoding
+
+Since the model contains no recurrence and no convolution, positional encodings are added to the input embeddings to give the model information about the relative position of tokens:
+
+    PE(pos, 2i) = sin(pos / 10000^(2i/d_model))
+    PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))
+
+This allows the model to easily learn to attend by relative positions.
+
+## Why attention over recurrence
+
+Three main advantages:
+1. Total computational complexity per layer is lower for self-attention when sequence length is smaller than representation dimensionality
+2. Computations that can be parallelized — recurrent layers require O(n) sequential operations
+3. Path length between long-range dependencies is O(1) for self-attention vs O(n) for recurrence
+
+## Results
+
+WMT 2014 English-to-German: 28.4 BLEU, outperforming all previously published results by over 2 BLEU.
+WMT 2014 English-to-French: 41.0 BLEU, new state of the art.
+Training cost: 3.5 days on 8 P100 GPUs.
+
+## Open questions
+
+[1] Does the choice of h=8 heads generalize, or is it architecture-specific?
+[2] The scaling factor sqrt(d_k) is justified empirically — is there a theoretical justification?
+[3] How does learned positional encoding compare to sinusoidal at longer sequence lengths?
+
+## References
+
+[1] Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). Attention Is All You Need. arXiv:1706.03762
+[2] Ba, J., Kiros, J., Hinton, G. (2016). Layer Normalization. arXiv:1607.06450
+[3] He, K., et al. (2016). Deep Residual Learning for Image Recognition. CVPR 2016.

worked/mixed-corpus/raw/build.py

@@ -0,0 +1,39 @@
+# assemble node+edge dicts into a NetworkX graph, preserving edge direction
+from __future__ import annotations
+import sys
+import networkx as nx
+from .validate import validate_extraction
+
+
+def build_from_json(extraction: dict) -> nx.Graph:
+    errors = validate_extraction(extraction)
+    # Dangling edges (stdlib/external imports) are expected - only warn about real schema errors.
+    real_errors = [e for e in errors if "does not match any node id" not in e]
+    if real_errors:
+        print(f"[graphify] Extraction warning ({len(real_errors)} issues): {real_errors[0]}", file=sys.stderr)
+    G = nx.Graph()
+    for node in extraction.get("nodes", []):
+        G.add_node(node["id"], **{k: v for k, v in node.items() if k != "id"})
+    node_set = set(G.nodes())
+    for edge in extraction.get("edges", []):
+        src, tgt = edge["source"], edge["target"]
+        if src not in node_set or tgt not in node_set:
+            continue  # skip edges to external/stdlib nodes - expected, not an error
+        attrs = {k: v for k, v in edge.items() if k not in ("source", "target")}
+        # Preserve original edge direction - undirected graphs lose it otherwise,
+        # causing display functions to show edges backwards.
+        attrs["_src"] = src
+        attrs["_tgt"] = tgt
+        G.add_edge(src, tgt, **attrs)
+    return G
+
+
+def build(extractions: list[dict]) -> nx.Graph:
+    """Merge multiple extraction results into one graph."""
+    combined: dict = {"nodes": [], "edges": [], "input_tokens": 0, "output_tokens": 0}
+    for ext in extractions:
+        combined["nodes"].extend(ext.get("nodes", []))
+        combined["edges"].extend(ext.get("edges", []))
+        combined["input_tokens"] += ext.get("input_tokens", 0)
+        combined["output_tokens"] += ext.get("output_tokens", 0)
+    return build_from_json(combined)

worked/mixed-corpus/raw/cluster.py

@@ -0,0 +1,104 @@
+"""Leiden community detection on NetworkX graphs. Splits oversized communities. Returns cohesion scores."""
+from __future__ import annotations
+import networkx as nx
+
+
+def build_graph(nodes: list[dict], edges: list[dict]) -> nx.Graph:
+    """Build a NetworkX graph from graphify node/edge dicts.
+
+    Preserves original edge direction as _src/_tgt attributes so that
+    display functions can show relationships in the correct direction,
+    even though the graph is undirected for structural analysis.
+    """
+    G = nx.Graph()
+    for n in nodes:
+        G.add_node(n["id"], **{k: v for k, v in n.items() if k != "id"})
+    for e in edges:
+        attrs = {k: v for k, v in e.items() if k not in ("source", "target")}
+        attrs["_src"] = e["source"]
+        attrs["_tgt"] = e["target"]
+        G.add_edge(e["source"], e["target"], **attrs)
+    return G
+
+_MAX_COMMUNITY_FRACTION = 0.25   # communities larger than 25% of graph get split
+_MIN_SPLIT_SIZE = 10             # only split if community has at least this many nodes
+
+
+def cluster(G: nx.Graph) -> dict[int, list[str]]:
+    """Run Leiden community detection. Returns {community_id: [node_ids]}.
+
+    Community IDs are stable across runs: 0 = largest community after splitting.
+    Oversized communities (> 25% of graph nodes, min 10) are split by running
+    a second Leiden pass on the subgraph.
+    """
+    if G.number_of_nodes() == 0:
+        return {}
+    if G.number_of_edges() == 0:
+        return {i: [n] for i, n in enumerate(sorted(G.nodes))}
+
+    from graspologic.partition import leiden  # lazy - avoids 15s numba JIT on import
+
+    # Leiden warns and drops isolates - handle them separately
+    isolates = [n for n in G.nodes() if G.degree(n) == 0]
+    connected_nodes = [n for n in G.nodes() if G.degree(n) > 0]
+    connected = G.subgraph(connected_nodes)
+
+    raw: dict[int, list[str]] = {}
+    if connected.number_of_nodes() > 0:
+        partition: dict[str, int] = leiden(connected)
+        for node, cid in partition.items():
+            raw.setdefault(cid, []).append(node)
+
+    # Each isolate becomes its own single-node community
+    next_cid = max(raw.keys(), default=-1) + 1
+    for node in isolates:
+        raw[next_cid] = [node]
+        next_cid += 1
+
+    # Split oversized communities
+    max_size = max(_MIN_SPLIT_SIZE, int(G.number_of_nodes() * _MAX_COMMUNITY_FRACTION))
+    final_communities: list[list[str]] = []
+    for nodes in raw.values():
+        if len(nodes) > max_size:
+            final_communities.extend(_split_community(G, nodes))
+        else:
+            final_communities.append(nodes)
+
+    # Re-index by size descending for deterministic ordering
+    final_communities.sort(key=len, reverse=True)
+    return {i: sorted(nodes) for i, nodes in enumerate(final_communities)}
+
+
+def _split_community(G: nx.Graph, nodes: list[str]) -> list[list[str]]:
+    """Run a second Leiden pass on a community subgraph to split it further."""
+    subgraph = G.subgraph(nodes)
+    if subgraph.number_of_edges() == 0:
+        # No edges - split into individual nodes
+        return [[n] for n in sorted(nodes)]
+    try:
+        from graspologic.partition import leiden
+        sub_partition: dict[str, int] = leiden(subgraph)
+        sub_communities: dict[int, list[str]] = {}
+        for node, cid in sub_partition.items():
+            sub_communities.setdefault(cid, []).append(node)
+        if len(sub_communities) <= 1:
+            # Leiden couldn't split it - return as-is
+            return [sorted(nodes)]
+        return [sorted(v) for v in sub_communities.values()]
+    except Exception:
+        return [sorted(nodes)]
+
+
+def cohesion_score(G: nx.Graph, community_nodes: list[str]) -> float:
+    """Ratio of actual intra-community edges to maximum possible."""
+    n = len(community_nodes)
+    if n <= 1:
+        return 1.0
+    subgraph = G.subgraph(community_nodes)
+    actual = subgraph.number_of_edges()
+    possible = n * (n - 1) / 2
+    return round(actual / possible, 2) if possible > 0 else 0.0
+
+
+def score_all(G: nx.Graph, communities: dict[int, list[str]]) -> dict[int, float]:
+    return {cid: cohesion_score(G, nodes) for cid, nodes in communities.items()}