`_find_node_tiers` builds two normalizations of the query: `term`, which
tokenizes on \w+ so punctuation becomes a space, and `norm_query`, which
keeps it. The exact tier compared the node id against `term` only, so
`term == nid_lower` was false for every id carrying punctuation, and
`norm_query` — which already held the right form, and is even one of the
two trigram needles — was never compared against the id at all. Comparing
`norm_query` to the folded id closes that half.
It does not reach ids carrying non-ASCII text. `_node_search_text`
indexed the id raw while every query path folds through
`_strip_diacritics`, which NFKD-decomposes. Hangul syllables decompose
into conjoining jamo, and jamo have combining class 0, so they survive
the combining-character filter: the needle's trigrams and the posting's
trigrams were disjoint, `_trigram_candidates` returned a candidate list
without the node, and it was dropped before any predicate ran. The
folded id is now part of the indexed text.
Both halves are additive. An id that resolved before resolves to the
same node; only ids that previously resolved to nothing can now resolve.
The folded field is appended, and only when the fold actually differs,
so field positions do not move and an all-ASCII graph indexes byte for
byte what it indexed before. Index build, median of 7 runs:
graph trigrams postings build
5k all-ASCII before 1723 238876 96ms
5k all-ASCII after 1723 238876 97ms
17k real before 33442 2311784 998ms
17k real after 33490 2312462 990ms
5k half-Hangul before 1735 256381 116ms
5k half-Hangul after 1739 266381 124ms
The real graph is the 17269-node one measured below; 354 of its ids are
non-ASCII, so the index grows 0.03% and the build stays inside run-to-run
noise. The half-Hangul row is a deliberate worst case — every other node
id Korean — and even there the cost is paid once per graph load, on a
graph where id lookup previously returned nothing at all.
On a real 17269-node graph with Korean source filenames, every node id
fed back to itself, full population:
id class total before after
contains punctuation 2326 0 2326
contains Hangul 354 0 354
ASCII, no punctuation 14589 14589 14589
And every query that graph can produce — all 17269 ids plus all 16537
distinct labels — through `_find_node_tiers` on both variants in one
process: 31126 identical, 2680 that returned nothing before and resolve
now, 0 with a changed first result, 0 lost, 0 with a widened exact tier.
Every difference is a query that previously returned nothing.
Left alone deliberately: `_score_query` compares the id raw in the same
way, so `path` and query seeding still cannot take a punctuated id, and
the prefix tier also matches ids against `term` only. Both are behaviour
changes beyond this defect rather than part of it.
One note for the regression tests: the non-ASCII case needs a graph of
at least ~10 nodes. `_trigram_candidates` bails out to a full scan when
`min(present) > int(n * 0.10)`, so on a small synthetic graph the index
path is never taken and the test passes with the defect still present.
sean-soomgo
1fdd11fa76