+
+
+
+
Entry Points
+
+
+
build()
+
graphify/build.py:119
+
Merges multiple extractions, then calls deduplicate_entities(nodes, edges, communities={}) before build_from_json()
+
Flag: dedup=True (default)
+
+
+
build_merge()
+
graphify/build.py:197
+
Incremental mode: loads existing graph.json, merges new chunks, calls build() with dedup=True
+
Shrink-guard skipped when dedup is active
+
+
+
__main__.py extract
+
graphify/__main__.py
+
Passes --dedup-llm flag through to enable LLM tiebreaker in Pass 3
+
Also triggers via /graphify skill
+
+
+
+
+
↓nodes: list[dict], edges: list[dict], communities: dict
+
+
+
+
Pre-pass — ID Deduplication
+
+ Collapse nodes with identical id fields (last-wins). Prevents AST extractors generating "UserService" and "userservice" as separate nodes (both normalize to the same id) from confusing the union-find. O(n) dict pass.
+
+
+
+
↓
+
+
+
+
Pass 1 — Exact Normalization
+
+ For every node, compute _norm(label): lowercase, strip all non-alphanumeric characters, collapse whitespace.
+ Group nodes sharing the same norm key into union-find clusters. O(n).
+
+ Examples: "HTTP Client" = "http client" = "HTTPClient" = "http_client"
+
+
+
+
↓unmerged pairs only
+
+
+
+
Pass 2 — Fuzzy Matching (per candidate pair via MinHash/LSH blocking)
+
+ Candidate pairs are generated by MinHash LSH (not all-pairs), then scored by Jaro-Winkler. Community membership boosts the score.
+
+
+
+
+ ① Entropy Gate
+ _entropy(label)
+ Shannon bits/char < 2.5 → skip fuzzy
+ Short/low-info labels like "A", "get", "fn" would generate false positives at scale
+
+
→
+
+ ② MinHash / LSH Blocking
+ 3-gram shingles (spaces stripped), 128 permutations, Jaccard threshold 0.7
+ datasketch.MinHashLSH
+ Space-stripping: "graph extractor" ≡ "graphextractor" at shingling level
+
+
→
+
+ ③ Jaro-Winkler Score
+ JaroWinkler.normalized_similarity(a,b) × 100
+ rapidfuzz.distance.JaroWinkler
+ Merge if score ≥ 92.0 after boost
+
+
→
+
+ ④ Community Boost
+ Same community (from clustering) → +5.0 pts
+ Entities in the same module/cluster are more likely to be the same concept
+
+
→
+
+ ⑤ Union-Find Merge
+ _UF class, path compression
+ All connected pairs → single cluster
+ Transitivity: if A~B and B~C then A,B,C all merge
+
+
+
+
+
↓ambiguous zone 75–92 pts (only with --dedup-llm)
+
+
+
+
Pass 3 — LLM Tiebreaker (optional, --dedup-llm)
+
+ Pairs scoring 75.0–92.0 after community boost are batched in groups of 30 and sent to Claude for a semantic judgement call. One API call per batch. LLM-approved pairs are fed back into the union-find for merging.
+
+ Disabled by default — catches cases like "Synchronous HTTP client." vs "Asynchronous HTTP client." (JW=98.6) where string similarity is high but meaning differs. Without this flag, such pairs merge at Pass 2; with it, the LLM rejects the merge.
+
+
+
+
↓
+
+
+
+
Remap — Winner Selection & Edge Rewiring
+
+ For each cluster of merged nodes, _pick_winner(cluster) selects the canonical id:
+
1. Prefer ids without chunk suffix (_c\d+)
+
2. Prefer shorter id on tie
+
+ All edges are rewritten: source/target remapped to winner ids. Self-loops created by the merge are dropped. Surviving nodes list uses only winners.
+
+
+
+
↓
+
+
+
+
Output → build_from_json()
+
+ Returns (deduped_nodes, deduped_edges) — passed directly into build_from_json() to construct the NetworkX graph. On a 17,497-node corpus: 4,938 nodes merged (3,831 exact + 1,107 fuzzy) → 12,559 nodes in final graph.
+
+
+
+
+
_ENTROPY_THRESHOLD = 2.5 bits/char — below this, skip fuzzy (short/generic labels)
+
_LSH_THRESHOLD = 0.7 Jaccard — MinHash blocking gate
+
_MERGE_THRESHOLD = 92.0 JW — auto-merge above this score
+
_COMMUNITY_BOOST = +5.0 pts — same community bonus
+
LLM tiebreak zone = 75.0–92.0 JW (only with --dedup-llm)
+
MinHash permutations = 128, shingle size = 3-gram (spaces stripped)
+