Confirms the reconstructed prefix set for an ASCII path is byte for
byte identical to what the old ASCII only regex produced, so the fix
changes behavior only for paths the old code was already getting
wrong.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017qfdzgbA5KedGEjD1AayNh
(cherry picked from commit 97bd1f37f0)
Runs deduplicate_entities on the defining and referencing Korean
node pair in both arrival orders and asserts the defining file own
node survives either way, matching the concrete case the issue
reports: a page whose filename equals the concept name losing its own
node to a page that merely mentions it in passing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017qfdzgbA5KedGEjD1AayNh
(cherry picked from commit 852c61098f)
Confirms the layer above _id_prefixes actually benefits from the
fix: a node whose id and source_file both encode a Korean path is
recognized as defining that id, and a node sharing that id from a
different file is correctly recognized as merely referencing it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017qfdzgbA5KedGEjD1AayNh
(cherry picked from commit 7d15c1bb26)
Confirms the exact repro shape from the issue: the reconstructed
prefix set for a Korean file path preserves every segment instead of
collapsing them to nothing, matching what the file's own id is
actually minted as.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017qfdzgbA5KedGEjD1AayNh
(cherry picked from commit ab4bde5413)
Absence of a `node_kind` stamp is not proof that a node is an entity: an
extractor can mint a node standing for part of its file without stamping one
(detect.py's per-sheet document nodes do exactly that), and the predicate read
those as entities while its name and docstring claimed proof.
Renames it `_reads_as_file_entity` and splits the docstring into what is PROVEN
(not the file's own node — an `_id_prefixes` reconstruction that holds for every
stored-path spelling) and what is ASSUMED (not a section, resting on the
producer stamping `node_kind`). Adds the producer contract next to
`_FILE_STRUCTURE_NODE_KINDS` and a test pinning the stamped/unstamped split, so
the limit is enforced rather than described.
No behaviour change: the same nodes merge as before.
Co-Authored-By: Claude <noreply@anthropic.com>
(cherry picked from commit d123254ebf)
Per-file extraction over a note vault mints one node per mention of the same
person or project, and they never merge. Normalization is not the problem:
`_norm` already casefolds and collapses `[\W_]+`, so `@cyrilXBT`, `@cyrilxbt`
and `cyrilXBT` land in the same Pass 1 bucket. The bucket is then dropped,
because the cross-file union added in #2182 is gated to
`file_type == "concept"` — and an entity the extractor pulls out of a `.md`
note inherits `document` from the file's extension, not from anything about
the entity.
#1284's reasoning for that gate is right about files: two `README.md` in
different folders are two documents. It does not follow for a node that merely
lives in a document. This widens the Pass 1 cross-file residue to
`document`/`rationale` nodes that are provably not part of their file's own
structure, via a new `_provably_not_file_structure`:
* not the file's own node — reusing `_id_prefixes`, which already enumerates
the ID a node standing for `source_file` would carry in every spelling a
stored path may take;
* not one of the file's sections — `node_kind: "heading"`, and `"page"` for
the file node the markdown extractor labels outright.
A node that cannot be checked (no ID, no provenance) answers False and stays
blocked, so the predicate can only narrow what the gate treats as
file-anchored, never widen it on a guess.
Deliberately unchanged: the entropy and provenance guards, `code` (#1205),
`image`/`paper` shared basenames, and Pass 2's fuzzy
`_crossfile_fileanchored_blocked` — so #1284's near-identical boilerplate and
#3094's repeated `## Decisions` sections stay per-file, both now covered by
tests.
Co-Authored-By: Claude <noreply@anthropic.com>
(cherry picked from commit bc45e1cf8b)
Fuzzy dedup compared same-file labels with prefix-weighted Jaro-Winkler, so
two distinct entities differing by one content word (asset contribution
flow vs asset consumption flow) cleared the threshold and one was lost. A
one-token difference is now judged on the differing tokens (any distinct
content word blocks; stopword/typo variants still merge), with a
same-length Damerau-Levenshtein typo escape. Genuine typo and
whitespace/case/punct variants still collapse. The #2532 collision path is
untouched.
Adapts PR #2587 (thanks @wilyan09007) with two hardening deltas.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_collision_rank no longer falls through to raw lexical source_file order.
It now ranks by a lifecycle penalty (active over archived) computed on the
root-relative path, then a reversed-segment tie-break, so the winner of a
collision (e.g. plans/_done/x.md vs plans/in-progress/x.md) no longer
depends on ASCII filename order, absolute-vs-relative path form, or the
checkout directory name.
Active/archived idea from PR #2540 (thanks @michaelxer); the root-relative
and reversed-segment determinism is implemented here.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pass 1 deferred cross-file exact matches to Pass 2, but Pass 2's candidate
filter keeps only the first node per normalized label, so identical-label
cross-file concept pairs could never merge (while fuzzy pairs did). Pass 1
now unions the cross-file residue of each label group, gated to concept
nodes with provenance and above the entropy floor, so code/rationale/
document/image/empty-source and cross-repo guards are all preserved.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two tweaks on top of @bchan84x's #1852:
1. The definer heuristic decided the survivor pairwise, so when several
same-file nodes co-defined one ID the surviving label still depended on
arrival order (the exact 3-node case #1851 reports). Choose the survivor
by a total order (definer first, then shorter/canonical label, then
lexically) via a min over _collision_rank, so it is order-independent.
Direction preserves #1504 (lexically-first source path wins).
2. _defines_id now also recognizes a bare file-level node whose id is
exactly the slugified path (nid == prefix), not only <path>_<entity>.
Adds an order-independence test over all 6 permutations of definer +
same-file relabel + cross-file reference, and a bare-file-node test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Node IDs are <source-path>_<entity>, so a doc that merely references an entity
mints the entity's own ID and collides with the entity's node by construction.
The pre-dedup pass kept whichever arrived first, so chunk order decided whether
an entity kept its own attributes or a passing cross-reference's, and the
warning that fired (#1504) told the user a same-name-different-directory clash
had lost their data — while the one drop that really is lossy, two labels for
one ID from the same file, was silent because the warning was gated on
source_file differing.
The survivor is now the node whose source_file is the file its ID encodes (any
trailing slice of the path, so absolute, repo-relative, and pre-#1504
bare-stem IDs all resolve), falling back to first-seen. Reporting follows what
the drop actually costs: a cross-reference folding into the node it references
loses nothing (edges are keyed by ID and rewire to the survivor) and is silent;
a same-file relabel notes the discarded label; two files that both encode the
ID are distinct entities and still WARNING, now stating the ID-scheme cause
rather than a filename clash.
Refs #1851
When two LLM extraction chunks each process a file with the same name in
different directories, they independently generate the same node IDs and
deduplicate_entities() silently drops one node (first-writer-wins). The
data loss had no indication in any log, counter, or output.
Adds a stderr WARNING when a duplicate ID comes from a different
source_file, telling the user which files collided and recommending the
per-subfolder extract + merge-graphs workflow to avoid it.
Three pass-2 guards (mirrored in the --dedup-llm pair collection): block merges
when labels' embedded numbers differ as zero-padding-insensitive multisets;
block cross-file merges of file-anchored rationale/document nodes (same-file
still merges); and score cross-file long labels on plain Jaro instead of
Jaro-Winkler so the prefix bonus can't fabricate merges of shared-prefix but
token-divergent entities (jest-native vs react-native), while genuine cross-file
duplicates still clear Jaro and same-file near-duplicates keep Jaro-Winkler.
Co-Authored-By: van4oza <van4oza@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pass 2 selected the merge winner from the union of both normalized-label
groups, so never-compared same-label/cross-file nodes could be pulled
into the union, bypassing the #1046/#1178 guards. Pick the winner from
[node, neighbor] only; group members that belong together still merge
via pass 1 (same file) or their own verified comparison.
Co-authored-by: Cursor <cursoragent@cursor.com>
- analyze.py: pass length_bound=max_cycle_length to nx.simple_cycles() so
networkx prunes during enumeration instead of post-filtering; drops report
generation from never-returns to ~0.1s on dense graphs (#1196)
- llm.py: replace hardcoded min(40+16*n,4096) label_communities token budget
with _resolve_max_tokens(min(64+24*n,8192)) — 24 tok/community covers 5-word
JSON entries; 8192 cap fits 16k-context models; env var now honoured (#1200)
- dedup.py: add prefix-extension guard in Pass 2 and _llm_tiebreak — skip merge
when one normalised label is a strict prefix of the other (getActiveSession /
getActiveSessions, parseConfig / parseConfigFile). Option (a) rejected: dropping
the >=12 early-out from _short_label_blocked breaks test_typo_merged (#1201)
- tests/test_dedup.py: two new regression tests verifying prefix guard fires for
extension pairs and does not fire for same-length typo pairs
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
abhirupone