Files
rea/experiments/atomic-codemode/native-engine.md
T
499fe56a62 docs(experiment): map verified primitive coverage
Separate task sufficiency, patched-engine requirements and remaining unknowns. Explain identity and relation grain without claiming universal RE completeness or provider parity.

Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com>
Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
2026-10-16 16:22:38 +00:00

688 lines
45 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Native recovered IR experiment
Stock Kuna's diagnostic text omits CFG successors and does not encode operand or
phi edge identities. `kuna-native-ir.patch` is a read-only exploratory exporter
against [Kuna v1.833](https://github.com/Noelo-Lab/kuna/tree/fe01d6294eb0d1053ce343c8943646df88e143fc).
It is not an upstream release or an integrated REA capability. No upstream commit,
PR or merge is being made.
The sections below record successive pinned prototypes and their historical
limits. For current comparison scope use the [capability map](capability-map.md),
[full-graph transport study](native-graph-transport-study.md) and
[independent switch study](switch-study.md), together with the live contracts.
`decompile --json --atomic-facts` captures the live function that produced C.
The default path does not collect the facet. Graph records include ordered CFG
edges, native value identities, indexed operands, outputs and phi input bindings
to particular incoming edges. Duplicate instruction origins or predecessor blocks
do not collapse identities. Values are local to one snapshot. Constant offsets
remain hexadecimal u64 address-space offsets, with width and annotation status
separate; they are not arbitrary-width computed bitvectors.
`native-ir.mjs` owns validation, immutable indexes and selected reads.
`control-dependence.mjs` derives control dependence using immediate
postdominators of the reversed graph, with linear retained graph state and a
bounded work count. It treats zero-successor recovered blocks as exits and
considers finite paths only. Exit-unreachable blocks, indirect-branch operations,
missing definitions, runtime successors and memory alias limitations stay
explicit. This is not a proof of feasible runtime paths.
The shared API adds `values`, `blocks` and `relations(kind:"controllers")`.
Diagnostic SSA sessions report these missing capabilities explicitly. Native
definition/use edges include operand and phi bindings; diagnostic printed
occurrences retain their original, weaker semantics. CLI, SDK and both MCP
adapters use the same owner. Ordinary source summaries are compact; `evidence`
retains the exact command argv and separate display argv, complete output and
digest. Earlier trials substituted display paths in `arguments`; they must not
be retroactively claimed to have captured exact argv.
To reproduce, check out the pinned upstream revision in a disposable directory
and apply this patch with `git apply`. Reuse already compiled `.sla` files from
the corresponding stock release under the same relative `specs/` paths; do not
recompile specs or replace existing unrelated files. Build from the REA root:
```sh
env CARGO_INCREMENTAL=0 CARGO_PROFILE_RELEASE_DEBUG=0 \
CARGO_PROFILE_RELEASE_OPT_LEVEL=1 CARGO_PROFILE_RELEASE_LTO=false \
CARGO_PROFILE_RELEASE_CODEGEN_UNITS=16 \
python3 experiments/analysis-efficiency/resource-run.py --observe-payload-exit -- \
cargo build --manifest-path /absolute/kuna/decompiler/Cargo.toml \
--locked --release -j1 -p kuna-cli --features embed-specs
```
Select that executable through `REA_KUNA_COMMAND` and explicitly pin its actual
SHA-256 through `REA_ATOMIC_NATIVE_IR_ENGINE_SHA256`. Native IR requires embedded
specs and rejects a caller-supplied `--sleighpath`; the provider creates a fresh
owned engine cache per lease. The executable digest therefore covers its embedded
assets, without reusing an unvalidated external spec cache. The current prototype
uses optimization level 1, not the stock release's assumed optimization profile;
comparisons must retain this difference or use the same engine in all lanes.
Native extraction bounds blocks/operations at 12,000, values at 36,000, input and
CFG edges at 50,000 and display text at 2 MiB, before the 8 MiB transport bound.
Capacity overflow makes this facet unavailable, preserving C and a reason. The
provider retains at most two 8 MiB native indexes (two 2 MiB diagnostic indexes)
and 64 MiB of complete source captures per lease. The outer observer still
enforces one CPU, 4 GiB hard memory and early termination at 3 GiB.
Real Linux zlib verification produced 3,860 operations, 5,898 values, 472 blocks,
779 CFG edges and 790 phi operations. Its 5,533,649-byte native JSON has no omitted
definitions or reported producer issues. C is byte-identical to the previously
captured stock 1.833 JSON (52,431 characters), and line mappings are identical.
The CLI composition independently discovers the string, owning function, store
and its two local controllers. Its observed cold session takes 9.191 seconds,
8.887 process-family CPU seconds and 198.9 MiB kernel memory peak, with verified
cleanup and zero OOM kills. This scripted lane is not a real-model cost result.
Targeted JS checks cover malformed/foreign identities, dangling relations, phi
order, exact hex offsets, immutable facts, loop/exit coverage, resource failure,
and all 64 four-node DAGs against an independent path-enumeration oracle.
Upstream `check-spec` passes. The full upstream corpus, stages and workspace gates
have not run; this patch is not declared upstream-ready. The model comparisons
and subsequent boundary findings below do not establish wider provider parity.
The value-origin model comparison exposed a representation boundary: optimized
recovered blocks can absorb a machine predecessor, so an IR edge is not an actual
branch instruction target. The experimental API now names operation locations
`origin_address`, their selector `origin_addresses`, and block locations
`origin_start`; printed CFG targets use `target_origin`. These replace the old
ambiguous fields without aliases. Historical captures retain their pinned old
contracts. Raw upstream JSON is preserved unchanged in Evidence.
The added `instructions` method batches selected machine-address windows, using
Kuna's existing [disassemble CLI](https://github.com/Noelo-Lab/kuna/blob/fe01d6294eb0d1053ce343c8943646df88e143fc/decompiler/crates/kuna-cli/src/disassemble.rs),
with exact bytes, decoded text, extent, coverage and per-window errors. Kuna
already has focused agent-oriented read surfaces; its port ancestry does not
establish which primitives are missing. No assembly decoding or branch-target
parser is duplicated in the MCP adapter.
Windows are capped at 4,096 decoded rows and 16 windows per call, sharing the
8 MiB capture and 64 MiB retained-source budgets. A real maximum-row CLI test
returns 4,096 rows in an 842,106-byte source, taking 1.227 family wall seconds,
1.042 CPU seconds and 112.0 MiB kernel memory peak. Both real stock and patched
MCP tests preserve good windows alongside an unmapped-address failure and report
the actual `JA 0xe26d` / `JMP 0xd9c9` bytes. The representation revision had 33
targeted tests; its subsequent real-model trial is recorded below.
That fresh trial now verifies the missing machine path correctly; see
[actual task costs](trials.md). The SQLite task then exposes a separate missing
primitive: name/alias lookup across distinct callable entries. `functions` now
reuses the production inventory parser, preserving every matching entry, import
stub ambiguity, original aliases, clipped extents and unknown runtime binding.
The inventory is read/validated once per lease, and query/field/page selection
returns only the requested view. Batch xref and instruction reads share one
partial-outcome owner, retaining good observations and source IDs on failures.
A real `sqlite3VdbeExec` capture at `0xea6d0` also exceeds the graph export's
operation capacity while producing 126,519 characters of valid C. The old adapter
incorrectly rejects the entire capture. The revised owner validates the function
envelope and input identity independently, then models graph availability as a
facet result. Code and its language remain available under the same handle;
graph counts stay unknown and graph reads return their reason and Evidence ID.
The real MCP regression reads 3,654 code lines after the graph failure and verifies
that `sqlite3Fts3ErrMsg` has distinct entries `0x268f0` and `0x5efc0`. Native graph
capacity remains a real limitation, not a resolved scalability claim.
`verify-cache.mjs` measures a real cold/warm pair within one open lease. Cold zlib
snapshot reading takes 6,934.4 ms / 6.928 family CPU seconds; warm reading takes
67.7 ms / 0.068 CPU seconds and preserves the exact snapshot/Evidence ID. The
cumulative kernel memory peak is 275.1 MiB for both phases; the warm number is an
upper bound on its isolated phase peak. This is an API-only measurement, with no
model token result or cross-session cache claim. Targeted tests now total 38,
including late cancellation retaining partial facts and their provenance.
A subsequent real boundary sweep found a different loss of usable facts:
`sqlite3Pragma` (`0xb90e0`) and `sqlite3Parser` (`0xb3d00`) stay below the row
limits but overflow the provider's 8 MiB stdout budget. Both public snapshots
fail, losing C as well as IR. A separate diagnostic capture with a stated 32 MiB
observer limit establishes the exact producer sizes; it is not relabeled as a
successful public-provider workflow. The old engine is preserved locally by its
`7d357e75…` digest for historical reproduction.
| Function | Previous pretty JSON bytes | Revised source bytes | C characters | Revised graph |
| --------------- | -------------------------: | -------------------: | -----------: | ----------------------------------------------- |
| `sqlite3Parser` | 8,492,810 | 4,526,811 | 50,508 | All 5,164 operations retained |
| `sqlite3Pragma` | 16,728,407 | 136,501 | 67,300 | Explicitly unavailable at encoded-byte capacity |
The structural fix is at the producing serialization boundary. Atomic output
uses the existing record producer and shared compact ASCII-safe writer.
`--atomic-output-bytes` takes an explicit encoded-document budget (default 8 MiB);
the provider passes the same bound it supervises. The writer checks each encoded
write before growing its bounded buffer, accounting for escaping and newline.
If necessary, the document drops only the graph and records an exact facet
failure. C, line mappings and metadata survive under the same Evidence. If even
the base document cannot fit, the command fails rather than clipping it or
repeating the analysis. Ordinary non-atomic JSON keeps its previous formatting.
Row limits alone never promise that the whole encoded graph fits.
The rebuilt local engine is SHA-256
`2290e8b13caabe30aab6832ab163d92925b2e652eebf7fc0cdbe65b889cb251b`
(same pinned source/release profile plus the revised patch). Real Code Mode MCP
verification succeeds for both functions. The preserved C and line mappings are
identical to the earlier full diagnostic captures; the Parser graph is also
deep-equal, and retained source hashes verify. Pragma's unknown graph counts and
failed graph reads retain the correct source ID. No limit was increased.
The first successful revised MCP check took 13.721 family wall seconds, 13.461
CPU seconds and 326.9 MiB kernel family peak, with verified cleanup and no OOM.
This is a two-function API regression, not a model-cost comparison or a claimed
analysis speedup. Native writer/escaping checks (14) and graph-only fallback (1)
pass in the release test binary; the shared JS tests (38), SDK ownership probe,
lint and patch apply/reverse checks pass. Full upstream corpus/stages/workspace
gates still have not run. The first MCP launch used a stale pre-format-build hash
and was correctly refused; the retained successful retry uses the digest above.
The `bytes` primitive reuses Kuna's existing `read --as data --json` protocol.
Unlike instruction decoding, it asks for an exact byte range. It disables the
global `listing` / `fast_funcdisc` passes for this data-only query; instructions
retain their existing profile. A native probe returns exactly equal JSON with
and without those flags. Its 484 / 290 ms timings share a workspace and run in
that order, so they do not establish an isolated speedup. Native/source ownership,
freshness, decoded-window admission and partial batches remain shared.
Real public MCP checks pass in native Code Mode, native direct and stock 1.833
Code Mode: a known 53-byte string, a 4-of-16-byte read at a mapped-image boundary,
an unmapped address, same-source cache reuse and the 64 KiB maximum. Both adapter
catalogs validate after SDK conversion. The byte parser rejects wrong subjects,
contradictory duplicated hex, invalid extents and rows, and preserves exact u64
addresses. Completeness follows examined byte coverage rather than the native
truncation flag. It never claims these loaded-image bytes are live memory or
proves an original-file mapping. Targeted JS tests now total 41.
## Structured call relations (wire version 2)
The next observed gap was input zero of CALL: a native `fspec` annotation is an
internal reference, not a code address. The read-only `atomic_calls.rs` exporter
now uses the existing `FuncCallSpecs` owner for recovered entry/name, model flags,
argument passing storage and stack offset. A single operation-to-spec index
avoids a linear lookup for every call. Input slots and SSA IDs remain separate
from passing locations. Finalized storage is admitted only at matching arity;
locked prototype storage is the alternative. Shifted inputs, placeholders,
missing specs, unavailable storage and unresolved indirect entries remain
explicit. There is no display-text parser or original-source ordinal claim.
`calls` provides ID/origin filtering, field projection and paging through the
shared SDK/CLI/MCP owner. Callee `values(is_input:true)` exposes retained input
values for exact storage candidate matching. Source schemas, unique call
ownership, call coverage, operand indexes/values and output identities are
validated before exposing an index; a bad or old-version graph preserves valid
C. Machine CALL instructions remain the independent check of a recovered edge.
Engine `64c6b3fe5737056fc29c10b0aad6c7cbd489ad7d17bd1726141b77bdcf5e15b7`
uses the same upstream revision, embedded specs and one-job build profile.
The rebuild took 317.090 family wall seconds, 315.808 CPU seconds and a 2,145.695
MiB kernel process-family memory peak (including charged file cache), OOM 0.
The previous compact exporter engine is preserved by its SHA-256 locally.
Real SQLite MCP verification passes both adapters, including a composed Code
Mode join: caller `0x65211 -> 0x64060`, argument passing storage
`register:0x38/8`, one matching callee input, and a bounded definition slice from
the context updates to that input. Formatter calls `0x5f031`/`0x5f062` point to
`0x2adc0`/`0x27e50`; their fspec offsets remain `0x10000`/`0x10001`.
All three functions' C, ordinary metadata/line mappings, blocks, operations,
values and existing coverage equal retained earlier captures. Those earlier
sources were pretty-printed: encoded-size changes are not a call-exporter
performance result. The structured call counts are 8, 33 and 2 respectively.
The Code Mode verifier took 7.872 family wall / 7.573 CPU seconds, 297.1 MiB;
the direct verifier took 7.771 / 7.499 seconds, 278.7 MiB. These differ in assertion
calls and warm file-cache state, so they are verification receipts, not a
controlled adapter speed comparison. Both verified cleanup and OOM 0. The
existing real zlib controller/phi/instruction verifier also passes with this
engine (10.992 wall / 10.795 CPU seconds, 285.7 MiB, cleanup verified, OOM 0).
`check-spec` passes in its default lenient mode; full upstream gates still have
not run. The independent cross-function question and acceptance criteria are in
[call-oracle.md](call-oracle.md), pinned before model evaluation.
The strengthened verifier additionally reads all five retained indirect calls in
this parser and checks their unknown targets. Machine bytes `ff5328` at
`0x644eb` independently describe a register-indirect CALL. The Code Mode rerun
passes (16.628 wall / 8.239 CPU seconds, 300.5 MiB); native test compilation was
also active on the same CPU, so its wall time is not an isolated performance
sample. The source checks pass 45 JS tests across both prototype directories and
two Python ledger tests; changed source lint and formatting pass.
A current coverage limit remains: call facts are part of the recovered IR
facet. A function that exceeds graph capacity also loses these call records,
even if a selected call-target view could fit. This is a reason to explore
independent fact facets; it must not be mistaken for proof of no calls or full
coverage on large functions. No full provider parity claim is made.
The current producer also passes fourteen native JSON-format tests. The initial
command compiled unrelated integration targets and a non-embedded CLI build,
which replaced Cargo's mutable `target/release/kuna`. Its 181.103 wall / 172.430
CPU seconds, 651.6 MiB peak and OOM 0 describe that broader compilation, not just
JSON test execution. Future focused commands should select `--bin kuna` and
`--features embed-specs`. The original embedded build restores from Cargo cache
and has the same `64c6b3f...` digest.
The first model launch after those tests correctly fails at `open_binary` with
`engine_profile_mismatch`; it produces no target Evidence and is not a completed
answer. Its rejected-trial costs are retained in `trials.md`. The producing
boundary is now fixed by preserving an independent executable copy in a directory
named by the full SHA-256, outside Cargo outputs. The model harness checks the
explicit experimental engine pin before copying credentials or launching a model,
and verifies the same bytes after the task. Changed/missing engines reject the
comparison while retaining costs. Other provider trials keep this field unknown.
Three regression tests cover missing/malformed pins, replacement and removal;
a real driver invocation with a wrong pin rejects before output/profile creation.
## Shallow outgoing references without decompilation
Kuna already supplies native `xrefs --from`, including exact-instruction and
owning-function queries. The shared `KunaReferenceFiles.ts` boundary now owns
both directions, validates direction-dependent peer fields and query ownership,
and preserves exact u64 endpoints and source-row occurrences. The old
incoming-facts-only parser is removed; both experimental consumers use the same
named query contract. Production incoming address-only queries keep their
existing projection and bounded cache semantics.
The experimental `references` method adds direction, explicit address/function
scope, native kind filtering and fields. Incoming function scope is rejected;
unknown outgoing function applicability, unattributed/excluded rows and unresolved
indirect/discovery/runtime coverage remain explicit. Native source cache keys
cover result-affecting query choices while scope, fields and pages reuse the same
source. `reference-reads.mjs` owns protocol selection/projection; the resource
owner and CLI/MCP adapters do not reinterpret native endpoint fields. Empty
field selection now requests the existing always-retained identities, rather
than being rejected without a format/resource reason.
Real outgoing CLI probes return 396 VM call-reference rows in 205,432 bytes,
versus one row/737 bytes for selected call site `0xf238b`. Four sequential probes
(VM function/instruction and wrapper function/instruction) take 6.772 family wall
/ 6.613 CPU seconds and 120.0 MiB peak. Each command takes about 1.6–1.7 seconds;
these are native metadata-query measurements with warm uncontrolled file caches,
not real-model speed/token results.
Independent GNU objdump decoding over the exported VM symbol (`0xea6d0`, size
`0x7e7e`) confirms all 396 `(origin,target)` pairs, 171 distinct exact targets,
and 16 calls to `0x296c0`. It also finds 21 indirect calls, outside these
known-target counts. In the wrapper, nine lifted machine call references compare
with eight retained IR calls: machine `0x6534b -> 0x29370` is an omitted
`__stack_chk_fail@plt` call. Its absence from optimized IR/C is not absence from
the executable. No native analysis or C behavior is changed to repair that
representation difference.
The real Code Mode MCP verifier passes paging, identity-only fields, scope-cache
reuse, exact sites, unknown applicability and local aggregation. It also captures
the VM anew: C is available, IR fails with `atomic operation capacity exceeded`,
and shallow references still work. This verifier takes 48.142 family wall /
47.803 CPU seconds, 610.1 MiB peak; VM decompilation/source capture itself takes
35.945 seconds. These are different requested capabilities, not an equivalent
workflow speedup claim. The direct verifier passes the same reference contract
without that VM decompilation (12.001 / 11.759 seconds, 256.8 MiB). Both verify
cleanup and OOM 0. An earlier direct-verifier attempt incorrectly expected target
metadata on an SDK input-validation error; the corrected verifier respects the
SDK rejection boundary. That failed assertion receipt is retained.
The exact focused boundary suites pass 38 tests, with a fresh strict production
build; changed-test strict typechecking and 45 experimental JS tests also pass.
Full lint/module boundaries and changed-file formatting pass. Repository lint
warnings remain outside this change. The focused build/test scope peaks at
3,030.6 MiB, below the 3 GiB early-stop line, with OOM 0. Two full `check:fast`
attempts (including serial execution and Go runtime memory controls) reach that
early-stop budget and are terminated before OOM; they are not reported as passed.
The changed-test typecheck uses 70.1 MiB and standalone full lint 919.0 MiB.
Full-repository typechecking and full upstream Kuna gates remain unverified.
The [next question/oracle](reference-oracle.md) are committed before model trials.
## Independent call and boundary-value facets
Graph schema 3 removes embedded call records. An independent `atomic_calls`
version-1 facet now carries every retained block-member CALL/CALLIND, target-input
SSA identity, origin/block/opcode/dead status, call specs and boundary values.
`atomic_values.rs` owns common native value interpretation. The call scan walks
native block membership without allocating an operation vector for every block;
its own call/input/value/text budgets do not depend on the 12,000-op graph cap.
The default native path still does not collect either facet and C analysis phases
are unchanged. All captures occur after the same C rendering.
The bounded serializer retains fitting facets independently and preserves ordinary
C/metadata. It tries the full document, graph without calls, calls without graph,
then the base record with both reasons. This keeps existing graph availability
when both facets individually fit but their combination does not. Facet contents
are moved for these bounded alternatives rather than cloned; no reanalysis or
clipped collection labeled complete is introduced.
The JS owner validates each facet independently with shared ID/value contracts
and row-wise admission. When both parse, call identities, origins, operands,
outputs and boundary values must agree with the graph owner. Bad call facts do
not erase a valid graph; a bad/oversized graph does not erase valid calls. Code
survives either failure. `values` uses the complete retained graph when present,
otherwise the explicitly scoped call-boundary set; absent IDs in that set do not
prove absence from the native function. Defining operation identities may remain
unreadable, and source signatures, source parameter ordinals, aliases and runtime
binding stay unknown. `is_dead` is always retained on call rows, like an
instruction decode status. CLI, SDK and both MCP adapters share these views.
The new preserved engine is
`e7d3f0cc8d8aae814bd6a2625a567d6aaad96c3fe2405a4f08dc8e065b851588`.
Its full rebuild took 315.815 family wall / 315.305 CPU seconds, 2,119.4 MiB
kernel process-family peak. Four focused native atomic-JSON tests pass, including
both overflow directions, both-overflow C preservation and graph preference when
the combined document is too large (95.981 / 95.631 seconds including test-target
compilation, 626.8 MiB). The final CLI rebuild after tests is 26.203 / 25.952
seconds, 492.7 MiB. It is copied outside Cargo outputs and pinned by full digest.
All these runs verify cleanup and OOM 0; full upstream gates remain unexecuted.
Real public MCP verification keeps VM C available, reports graph operation
capacity failure, and now exposes 416 calls plus 1,294 boundary values. The call
scan examines 21,349 operations over 1,229 blocks. It preserves 395 direct known
targets and 21 unresolved indirect calls; all direct pairs match independent
machine pairs except the omitted `0xf2493 -> 0x29370` stack-check call. All 21
indirect origin addresses also match independent disassembly. Selected
`0xf115c`/`0xf238b` calls target `0x296c0`; their fspec inputs remain separate
annotation values. Repeated snapshots reuse the same handle and source, and an
explicit unavailable graph read fails without hiding these retained facts.
The VM source grows from 273,703 to 793,791 bytes to retain this new data; the
wrapper grows from 202,206 to 208,083 bytes. Their ordinary records, including C,
variables and line mappings, are identical to the prior capture. The wrapper's
blocks/operations/values/coverage also remain identical. These are fidelity and
source-volume observations, not model-token or speed results.
The real reference/VM verifier passes in 49.471 wall / 49.130 CPU seconds,
615.8 MiB. The existing cross-function call/input verifier passes in 8.377 /
8.094 seconds, 292.9 MiB; encoded Pragma/parser C/graph checks pass in 12.403 /
12.195 seconds, 335.0 MiB. All verify cleanup and OOM 0. The updated contract
suite passes 47 JS tests, including independent facet failure, cross-facet
conflicts, input-zero identity and boundary-only value reads. `check-spec` passes
in default lenient mode, lint and changed-file formatting pass, and the patch
applies forward to the pinned upstream revision and reverses from the clone.
Full repository typechecking retains its previously recorded resource limitation.
The [boundary question/oracle](boundary-oracle.md) are fixed before model runs.
The subsequent real-model pair passes that oracle in both calling modes. Each
uses one VM analysis and the same retained function record; neither invents a
complete graph from the available call-boundary values. Source digests, target
and preserved-engine pins, selected machine checks, cleanup and OOM-zero receipts
verify. See [whole-task costs](trials.md): the capability survives real model use,
but a single mixed-context-cost pair does not establish general efficiency.
## Compact counts from the validated call owner
The following API-only change keeps the preserved engine and native source
format unchanged. Snapshot call metadata includes counts by recovered opcode
and known/unknown native target, plus call coverage. The owner computes these
once after validation, before the shared CLI/SDK/MCP views. Counts include dead
rows and do not infer target knowledge from CALL versus CALLIND. Available empty
sets have zeros; unavailable sets have no counts. Selected call reads do not
change this whole-facet summary or require a new analysis.
The focused suite passes 49 tests, including known indirect/unknown direct
targets, dead rows, immutable summaries and empty/unavailable distinction.
Both real MCP adapters verify the wrapper's eight direct calls and parser's
28 direct/five unresolved indirect calls while preserving the existing
cross-function checks and validating advertised schemas after SDK conversion.
Code Mode verification takes 29.263 family wall / 21.548 CPU seconds, 283.9 MiB
kernel family peak; direct takes 32.893 / 22.781 seconds, 280.6 MiB. Both have
verified cleanup and OOM 0. These are validation runs, not a controlled speed
comparison with previous verifier timings. An owner replay of the prior
digest-verified VM source yields the expected 395/21 distribution; it is not
relabeled as a fresh provider or model run. The identical boundary question and
oracle are retained for subsequent real-model comparison.
That comparison now passes in both modes. Both consume the compact counts and
read just the two selected calls, reusing one VM analysis. All native VM function
records match the preceding pair and all retained sources/digests verify. Direct
has lower observed task costs than Code Mode in this sample; the before/after
results remain mixed. [trials.md](trials.md) records cumulative/cached/uncached
tokens, model responses, elapsed time, CPU and process-family memory, including
the slower Code Mode result. No native engine or analysis-speed improvement is
claimed for this shared owner metadata change.
## Tagged cleanup ownership regression
Source inspection finds that the experimental owner awaits but ignores
`KunaTargetCommands.close()`'s tagged `Result`. On error it would clear its state
and let `AtomicSession` drop the owner while claiming a successful close. The
SDK verifier reproduces this before the fix: an injected `ProviderCleanupError`
does not reject close. Its fallback cleanup still removes the real owned state;
the original failing result is retained as `cleanup-before.json`.
The owning boundary now uses the existing `requireResult` projection before
closing its cache root or clearing state. The existing session queue therefore
retains the owner on failure. Both close and target replacement report
`cleanup_incomplete`, preserve the staged file identity and original lease, and
succeed on retry; the file is then absent. MCP's open/close error adapter now
preserves context and resource details rather than dropping that evidence. The
CLI already propagates the thrown error as a failed command. No alternate
cleanup owner or downstream retry workaround is added.
`verify-cleanup.mjs` runs against real provider reads and resources, injecting
one tagged failure per close/replacement case. `verify-cleanup-mcp.mjs` injects
one explicit test-only failure in an isolated server, then verifies public
Code Mode and direct error metadata and successful retry. These are controlled
boundary failures, not a claim that the real engine failed to stop. The SDK
check takes 1.730 family wall / 1.567 CPU seconds, 130.4 MiB kernel peak; MCP Code
Mode takes 1.931 / 1.639 seconds, 200.0 MiB; direct takes 1.730 / 1.489 seconds,
192.8 MiB. All verify process cleanup and OOM 0. The original failing verifier
takes 1.226 / 0.985 seconds, 122.7 MiB and also verifies fallback cleanup/OOM 0.
The 49 focused JS tests, lint and formatting pass. This lifecycle-only change
follows the recorded model pairs; no new model token or speed claim is made.
## Callable bounded definition projection
The next native exporter retains a version-1 `atomic_projection` independently
of graph version 3 and call version 1. One checked native membership scan owns
call roots and an optional index of at most 65,536 operation keys; exceeding the
index budget disables the projection without discarding calls. Full-graph row
capacity is checked before rendering its rows. Full and selected facts share
one operation/value normalizer, including exact identities, indexed operands,
outputs, phi predecessor edges, flags and display. Default non-atomic analysis
and C generation are unchanged.
All recovered CALL/CALLIND operations are retained as projection roots, followed
by backward operand definitions in breadth-first order. The projection bounds
4,096 operations, 12,288 values, 16,384 inputs and 2 MiB text. Too many roots or
another admission failure makes the facet explicitly unavailable. Omitted
definitions retain exact IDs and operation-budget/outside-membership reasons.
A fitting full graph omits its redundant projection; an encoded-overflow graph
restores the fitting projection. Facets are moved through bounded alternatives,
and a valid C/call record survives projection failure. There is no repeated
decompilation or clipped collection labeled complete.
The JS boundary shares operand/value validation and query indexes between full
and sparse views. It checks root reachability, exact frontier membership,
call/value agreement and phi metadata without inventing a CFG. In the absence
of a full graph, conflicting projection facts do not erase independently valid
calls. `snapshot.facets.operations` and `facets.values` expose their scopes;
selected definitions/uses and slices work over the retained set. Uses remain
selected-only, CFG/controller queries retain their graph error and Evidence ID,
and slice missing definitions keep their native reasons. A missing projection
preserves the original graph failure instead of hiding it behind a generic
not-captured error. All adapters use this owner.
Coverage returns counts instead of global omitted-ID inventories, at most eight
160-character issue summaries, full issue counts and omission metadata. Complete
native coverage/frontier details remain in source Evidence. Native operation
rows retain `is_dead` with their identities; constant summaries add `value_id`
and `input_index`, preserving exact offset/width semantics. No original-signature,
arbitrary-width computation, memory-alias or runtime proof is introduced.
Preserved engine SHA-256:
`a0685486518e09d3c65f756325a7e9fed8df123a2a988aca2972a72335ce8edc`.
The same one-job/embedded-spec build takes 309.805 family wall / 309.523 CPU
seconds, 2,099.5 MiB kernel family peak. Six focused native JSON tests pass,
including projection restoration and independent overflow (99.752 / 98.662
seconds with test compilation, 626.9 MiB). The final CLI build reuses artifacts
(0.320 / 0.146 seconds, 28.8 MiB), leaving the preserved digest unchanged.
Real SQLite VM public reads expose 4,096 operations, 7,469 values and 326
unexpanded definitions, while the complete graph remains unavailable. Source is
4,008,032 bytes, inside the unchanged 8 MiB public budget. Every selected operation
and value equals the earlier 17.8 MB diagnostic capture; ordinary C/metadata and
the complete call facet also equal it. An independent BFS over that retained
full graph produces exactly the same operation order and frontier. The extended
`measure-native-projection.py --verify-capture` reproduces these comparisons.
This is producer-fidelity evidence, not independent machine/runtime equivalence.
Both public MCP adapters read the selected PHI definitions and slices from one
VM source. Code Mode verifies unavailable controllers preserve the source ID.
The corrected Code Mode verifier takes 52.706 wall / 47.550 CPU seconds,
597.4 MiB peak; direct takes 39.696 / 39.418 seconds, 592.5 MiB. The first
Code Mode verifier incorrectly expected INDIRECT definitions; independent source
inspection shows MULTIEQUAL. Its 37.070 / 36.781 seconds, 593.3 MiB failed
assertion is retained, then the verifier is corrected without changing native
facts. These scripted runs are validation costs, not an adapter speed comparison.
Small-function cross-call checks pass (8.476 / 8.204 seconds, 293.6 MiB); all three
complete function records equal the prior exporter. Encoded Pragma/Parser checks
pass (13.712 / 13.507 seconds, 361.5 MiB): Pragma retains 3,627 projected operations
after graph encoding overflow, while Parser keeps its full 5,164-operation graph
without a duplicate projection. Zlib control/phi regression passes (10.893 /
10.700 seconds, 286.3 MiB). Real CLI reads verify both selected PHIs and repeated
snapshot/source identity (37.367 / 37.174 seconds, 527.1 MiB).
The real VM MCP cache benchmark observes cold snapshot wall 36,448.8 ms / family
CPU 36.425 seconds, then warm 40.2 ms / 0.041 seconds with the identical handle
and source. Cumulative family peak is 590.1 MiB, an upper bound on isolated warm
phase memory. Its complete run is 37.667 / 37.492 seconds. This is same-lease
capture reuse, not cross-session persistence or a model-token result; OS caches
remain uncontrolled. All resource runs above verify cleanup and OOM 0.
The focused JS suite passes 53 tests; lint/format, default lenient `check-spec`,
Python compilation and patch forward/reverse checks pass. Full upstream corpus,
stages and workspace gates remain unexecuted; the patch is not upstream-ready.
No whole-repository typecheck pass is claimed beyond the previously recorded
resource limitation. The [PHI-format task/oracle](projection-oracle.md) are fixed
before real-model launches. No model benefit is assumed from these smaller facts.
## Named reference discovery and input schemas
The first PHI model pair completes correctly but exposes a documentation boundary:
direct help displays `facts_*` names while lookup only recognizes raw method
substrings. A multi-name query returns no matches and causes a full-reference
fallback. Registration and help now share one explicit name map, with canonical
identity separate from presentation. Exact lists preserve caller order,
deduplicate names and retain unmatched terms. Topic/all-contract lookup remain;
ambiguous labels fail instead of choosing by catalog position.
The real client verifier also catches that plain `z.toJSONSchema` defaults to
output semantics, making defaulted `offset`/`limit` falsely required in advertised
input schemas. Reference generation now explicitly selects `io:input`. Converted
schemas are frozen and weakly cached by schema identity; a replacement schema
invalidates the conversion, while description/presentation is read separately.
The documentation module removes the adapter rename pass. No target analysis,
live-result caching or native-engine change is involved.
The shared suite passes 55 tests. `verify-discovery.mjs` checks both real SDK
catalogs and reference responses without opening a target, compiles every catalog
and all fourteen reference schemas with Ajv2020, and validates the same
omitted-default requests against named reference/direct contracts. It proves the
six-name query matches all requested contracts without a miss. The all-schema
verifier takes 1.936 family wall / 1.760 CPU seconds, 138.8 MiB kernel peak,
cleanup verified and OOM 0. The earlier six-schema probe takes 1.628 / 1.449
seconds, 128.9 MiB. Its original 1.730 / 1.487 seconds, 128.1 MiB failed
omitted-default check remains recorded. Six contracts occupy 10,835 Code Mode /
10,360 direct bytes, versus 21,746 / 20,699 for all fourteen. Bytes do not prove
token or time savings. The same PHI question/oracle are retained for the next
actual model pair.
## Wide native operand relations
A retained-source replay exposes a whole-row cost: VM PHI
`0xea998:14649` has 156 inputs and 156 definition edges. Its normalized old
operation is 44,447 JSON bytes, and a default one-node slice returns 44,980 data
bytes. Real QuickJS returns `output_budget` while preserving the source ID.
This replays real producer facts through the shared parser/executor, without
a fresh native analysis or model call (`wide-phi-before.json`). Explicitly
requesting only `inputs` already fits; the gap is default size and lack of
independently selected/paged operands, not inability to obtain the data at all.
`relations kind:operands` now owns slot-level paging and optional input-index
selection. Snapshot + operation + input_index identifies the relationship;
value identity, native definition and exact phi edge are separate fields. It
retains constants, function inputs and nulls that definition edges cannot cover.
Requested out-of-range slots of retained seeds are absent, while unretained
seeds stay explicit/unknown. Seeds and selectors deduplicate in caller order;
total counts filtered rows before paging. Full graph and projection share this
owner; diagnostic SSA rejects the unsupported relation with source context.
Paging counts existing relation groups before materializing selected rows,
avoiding a complete flattened relation copy. Native operation defaults are
compact summaries with exact input/definition/constant counts; detailed fields
remain selectable. Repeated value/block identity lookups deduplicate at their
shared read boundary. A discriminated contract replaces the prior runtime-only
relation refinement, so advertised schemas also enforce seed types and prohibit
input_indices on other relationship kinds. No native rebuild, higher native
capacity, new method or additional authority is involved.
The shared suite passes 57 tests; changed JS lint has zero warnings and formatting
passes. Both real SDK catalogs and all reference schemas compile; selected
operand inputs and invalid selector applicability agree with runtime validation.
That discovery run takes 7.443 family wall / 4.928 CPU seconds, 141.8 MiB peak
including charged file cache. The real Code Mode VM verifier passes ten calls
in 54.327 / 47.069 seconds, 597.6 MiB peak. Its default slice fits, selected
slots preserve exact native facts, and explicit all-input reads still work.
Both runs verify cleanup and OOM 0. This first public run precedes the final
identity-deduplication adjustment; subsequent direct/model reads cover it.
Scripted bytes and costs are not model-token savings or an adapter speed claim.
The subsequent direct verifier passes all ten calls, including duplicate-value
deduplication, in 39.385 wall / 39.163 CPU seconds, 591.3 MiB peak, with cleanup
verified and OOM 0. The different scripted read paths/host state do not isolate
adapter costs. The [wide-operand oracle](operands-oracle.md) fixes the next seeded
real-model task before launch; it is not a before/after of the PHI-format task.
## Static method index
The wide-operand pair exposes a discovery cost: Code Mode asks four times for
help before using the same seven fact requests as direct. `reference view:index`
now returns the same fourteen canonical/display names and descriptions without
constructing schemas, return contracts or examples. `view:contracts` preserves
the existing default and exact-name/topic behavior. Both presentations share one
named documentation contract/owner and the API's registry; no analysis change or
engine rebuild is involved. An index is static declared support, never live
provider capability. Runtime facets remain authoritative.
The shared suite passes 58 tests, including a getter that proves an index does
not access an unrequested input contract. Changed JS lint/format and both real
SDK catalogs/reference schemas pass. The first discovery probe takes 5.284 family
wall / 4.028 CPU seconds, 137.6 MiB peak, cleanup verified and OOM 0. Code Mode's
index is 7,212 JSON bytes versus 24,010 for all contracts; direct is 7,240 versus
22,939. These bytes are static discovery output, not demonstrated token/time
savings. A subsequent focused check and real probe cover moving the input schema
into the documentation owner and unifying presentation construction. The same
wide-operand question/oracle are retained for a new real-model pair.
The final real SDK probe passes in 4.380 wall / 3.807 CPU seconds, 141.4 MiB peak,
cleanup verified and OOM 0. Both subsequent real-model answers satisfy the
unchanged oracle. Code Mode uses index plus one exact contract query, reducing
its discovery lookups from four to two. Whole-task costs remain mixed; all
attempts and limits are recorded in [trials.md](trials.md).
## Fact granularity versus computation granularity
The eight successful real-model VM captures using the preserved `a0685486...`
engine each retain 4,008,032 bytes, including 126,519 bytes of reconstructed C
that the selected-operand models never read. Fresh-lease native request wall
ranges from 36.702 to 53.124 seconds. A separate failed first attempt hits the
120-second timeout before one successful capture; its cause remains unknown.
These are fresh private native caches, not controlled OS-cold samples. Same-lease
warm source reuse remains separately measured above.
Source inspection explains the current computation boundary: the project core
drives whole-function decompilation, renders C and extracts metadata, then
collects independent call/graph/projection facets from the live function. The
new slot atom reduces selected read/serialization scope, but does not implement
selected-slot native analysis. The complete VM CFG is still unavailable and
arbitrary definitions outside the retained call-rooted projection cannot be
expanded by a fact read. Provider parity and general complex-software superiority
remain unestablished.
At that checkpoint, pipeline/render/metadata/export costs were not separately
profiled. The [subsequent native computation study](native-compute-study.md) now
measures them: VM drive about 34 seconds, C rendering 0.22 seconds and projection
0.03 seconds. Skipping C is a small opportunity for this workload. An offline
lossless encoding probe also puts the earlier full 17.8 MB diagnostic record
inside the unchanged 8 MiB transport. The [subsequent codec study](native-graph-transport-study.md)
now implements and verifies this through both public experimental MCP adapters,
including complete decoded Evidence and shared text-budget ownership. Actual
guard/control model behavior is now checked in two real-model pairs. Model
comparisons share the same computation/fact owners, avoiding a capability
difference between adapters. The [switch extension](switch-study.md) adds an
independent native table facet with exact labels and row/edge mapping, selected
queries and a verified 2 MiB fallback that retains cases without a full graph.
Its final engine is `a9ca8a8ac224936afb08a9d4be9c1225feb69473ad9f35764e8d1c4f04875b5b`;
the fixed case/default model question is separate from the earlier guard pairs.