Import the canonical Unicode aware id normalizer into dedup.py

Toward issue 3352: _id_prefixes reimplements id slugification with
its own ASCII only regex instead of the canonical normalize_id every
real extractor uses to mint an id, so it silently drops non Latin
characters. This adds the import with no behavior change yet; the
swap lands in the next commit.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017qfdzgbA5KedGEjD1AayNh
(cherry picked from commit 237f02d135)
This commit is contained in:
ayushcodes10 authored and safishamsi committed 2026-09-14 18:49:17 +01:00
1 parent e4399cb9ca
commit b5247b5a88
1 file changed
+1
+1
View File
@@ -12,6 +12,7 @@ from collections import defaultdict
from pathlib import Path
from graphify._minhash import MinHash, MinHashLSH
from graphify.ids import normalize_id
from rapidfuzz.distance import DamerauLevenshtein, Jaro, JaroWinkler