rea/experiments/atomic-codemode/trials.md

Real-model trials (ongoing)

The first trial used the committed 5aa1c5c64 runtime, requested model alias gpt-6.1-sol with high reasoning, Codex CLI 0.161.0 and the zlib question in this directory. The resolved server-side model build is not inferred. Its actual registered MCP server name was rea-codemode, with only the four Code Mode tools. Shell/file tools, web search, delegation and client-side Code Mode were disabled; the actual trace establishes server-side execute_javascript use.

The model independently found the string and xref, requested the owning function's snapshot, used operations/relations/slices and inspected reconstructed C. Its answer identifies both error-block branches, the message/STORE origins, wrapper bit condition and modulo-31 check, and correctly distinguishes local guards from runtime reachability. It names memory, phi, traversal-budget and CFG limitations. It records the exact target digest and real source Evidence IDs and closes the target. These observations agree with the previous independent machine-byte and arithmetic oracles; this is one stochastic local-question trial, not evidence of general RE superiority.

Measurement	First Code Mode trial
Completed outer MCP calls	14
Actual API calls	30
Reconstructed C page reads	16
Native source captures	3
Cumulative model input tokens	581,617
Cached input tokens (included above)	530,176
Output tokens	3,822
Reasoning output tokens (included in output)	869
Model runner wall seconds	130.348
Bounded family elapsed seconds	130.824
Bounded family CPU seconds	11.842
Kernel process-family memory peak MiB	540.7
OOM kills	0

Artifact bytes were unchanged. The copied-authentication profile was removed; both owner/observer cleanup checks passed. Memory includes charged file cache. Actual token usage is cumulative across turns, including discovery and retries, not a tokenizer estimate of the final answer. Cached and uncached tokens are reported separately without assigning an unverified price.

The first empty workspace was nested beneath the trial output directory in the repository. No source or oracle was copied into it and shell/file tools were disabled. Subsequent trials move the profile under /tmp to remove ancestor repository-instruction discovery as a possible confound. The first trial must not be silently relabeled as having that stronger isolation.

The trace exposes two design shortcomings. Reference lookup supplied parameters but omitted result shapes, making initial calls sequential shape discovery. The C reader's arbitrary 128-line maximum made code compose many page reads even though the complete function easily fits guest memory. Return contracts and byte-budgeted internal C reads are being tested next. The missing native CFG and phi predecessor relationships remain capability gaps, not problems to conceal with reconstructed-C guesses.

Private captures remain under .cache/atomic-codemode/zlib-codemode-v1/, with events.jsonl, codex-tool-context.jsonl, answer.txt, summary.json, explicit trial settings and cleanup proof. The adjacent trace retains actual model code and inner API calls; the *-evidence/ directory retains the three complete native sources and their identity/digest metadata. Credentials and raw captures are not committed.

The second round used the pinned c5cefd622 runtime, the same target, question, model alias/reasoning and client version, with the empty model profiles outside the repository under /tmp. Both atomic modes use the same eight fact methods; only the calling adapter changes. The traditional comparison forces the real Kuna provider and selects nine relevant production tools in the client: open_binary, close_binary, binary_session, search_strings, xrefs, procedure_address, procedure_pseudo_code, procedure_assembly, analyze_function. The production catalog remains complete. This is a favorable focused baseline, not the default full-catalog experience.

Measurement	Atomic direct v2	Atomic + Code Mode v2	Traditional v2
Outer MCP attempts, including failures	27	12	9
Completed / failed outer attempts	27 / 0	12 / 0	7 / 2
Internal Code Mode API reads	—	21	—
Cumulative model input tokens	590,872	453,734	344,105
Cached input tokens (included)	496,512	401,664	297,344
Uncached input tokens	94,360	52,070	46,761
Output tokens	5,874	4,738	3,076
Reasoning tokens (included in output)	2,228	1,716	832
Model runner wall seconds	170.813	148.397	106.352
Bounded family elapsed seconds	171.801	149.585	106.747
Bounded family CPU seconds	11.898	22.129	20.666
Kernel process-family memory peak MiB	578.2	544.3	843.4
OOM kills	0	0	0

The traditional trace includes a client approval-policy rejection for binary_session and a failed name lookup for inflate; the model recovered through string/xref/address discovery. Both attempts and their costs remain in the comparison. A completed transport call does not itself establish a valid analysis result; answers and the retained calls were reviewed separately.

All three answers identify the same two local error branches and modulo-31 condition, with supporting addresses and limitations consistent with the independent instruction/arithmetic checks. This establishes a completed local task comparison. It does not prove global runtime reachability or correctness on arbitrary binaries. All target digests remained unchanged, temporary model profiles were removed, and observed payload cleanup succeeded.

Code Mode reduces direct atomic round trips on this trial, but traditional tools win both cumulative input tokens and wall time. The prototype still makes the model assemble incomplete CFG/phi/value relationships and repeatedly inspect operations or C. Fixing the arbitrary C page cap reduced internal C reads from 16 to 2; the overall first-to-second trial changes are not a controlled causal measurement because return contracts and profile placement changed too.

These are single stochastic samples, with fresh provider/session caches, warm and uncontrolled OS file caches, and actual model prompt-cache usage reported. No warm-session paired reruns, confidence interval, server-side model compute or pricing was measured. Reproduce the ledger with summarize-trials.py over the private trial directories; it reads actual model usage and the payload-exit cgroup receipt and deliberately does not grade answers.