mirror of
https://github.com/safishamsi/graphify.git
synced 2026-09-27 07:55:43 +00:00
perf(dedup): remove O(nodes x components) scan from remap construction
deduplicate_entities() rebuilt its remap table by re-scanning the whole
unique_nodes list once per merged union-find component, costing
O(nodes x components) -- roughly 137M membership checks on a 50k-node
corpus with 2.5k merged components, about a third of dedup wall-clock.
Build an id -> (position, node) index once and slice each component out of
it instead: O(members log members) per component.
The index carries the enumeration position, not just the node, because
_pick_winner() selects via min(), which returns the first minimum -- so
ties (equal chunk-suffix status and equal id length) are resolved by list
order. Sorting the group by id instead changed 179 of 2358 survivors on a
tie-heavy corpus. Sorting by position reproduces unique_nodes order
exactly, leaving survivors and edges byte-identical.
Benchmark, median of 5 runs:
nodes before after speedup
5000 2.37s 2.29s 1.03x
20000 10.58s 9.70s 1.09x
50000 31.72s 21.92s 1.45x
Isolated remap loop at 50k nodes / 2.5k components: 6.838s -> 0.019s (370x).
Merge counts and output node counts are identical at every scale.
This does not address the pre-existing arrival-order dependence of the
component structure itself, which originates in the fuzzy-merge loop's
uf.find() short-circuit and is out of scope here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
committed by
safishamsi
co-authored by
Claude Opus 5
parent
1b93c08697
commit
0c01da1932
+17
-1
@@ -574,10 +574,26 @@ def deduplicate_entities(
|
||||
components = uf.components()
|
||||
remap: dict[str, str] = {}
|
||||
|
||||
# id -> (position, node), built once. Previously each component re-scanned
|
||||
# the whole unique_nodes list, making remap construction O(nodes x
|
||||
# components) — 31% of dedup wall-clock on a 50k-node corpus.
|
||||
# The position is carried so group_nodes keeps unique_nodes order: _pick_winner
|
||||
# resolves ties (equal chunk-suffix status and equal id length) via min(),
|
||||
# which returns the first minimum, so reordering here would silently change
|
||||
# which node survives.
|
||||
nodes_by_id: dict[str, tuple[int, dict]] = {
|
||||
n["id"]: (i, n) for i, n in enumerate(unique_nodes)
|
||||
}
|
||||
|
||||
for root, members in components.items():
|
||||
if len(members) == 1:
|
||||
continue
|
||||
group_nodes = [n for n in unique_nodes if n["id"] in members]
|
||||
group_nodes = [
|
||||
n for _, n in sorted(
|
||||
(nodes_by_id[m] for m in members if m in nodes_by_id),
|
||||
key=lambda pair: pair[0],
|
||||
)
|
||||
]
|
||||
winner = _pick_winner(group_nodes) if group_nodes else {"id": root}
|
||||
winner_id = winner["id"]
|
||||
for member in members:
|
||||
|
||||
Reference in New Issue
Block a user