From 7259ea1deef0fc9ebe7d02e9966c00285bccff0d Mon Sep 17 00:00:00 2001 From: N0zoM1z0 <161784452+N0zoM1z0@users.noreply.github.com> Date: Fri, 16 Oct 2026 19:32:32 +0000 Subject: [PATCH] docs(experiment): retain mixed discovery repeat results Both real-model answers satisfy the unchanged oracle. Record the smaller actual help path, all task costs, and the unexplained 189-second gap between completed client executions without removing it from elapsed time. Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com> Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com> --- experiments/atomic-codemode/README.md | 5 ++ .../discovery-summary-study.md | 78 +++++++++++++++++++ 2 files changed, 83 insertions(+) diff --git a/experiments/atomic-codemode/README.md b/experiments/atomic-codemode/README.md index 1c16e991a..615629738 100644 --- a/experiments/atomic-codemode/README.md +++ b/experiments/atomic-codemode/README.md @@ -34,6 +34,11 @@ summaries; all full contracts and advertised tools remain available. Contracts and session admission have separate owners, with native loading deferred to open. Its repeat uses the same frozen control question and retains the earlier results. +That completed repeat again answers correctly in both lanes. The smaller index +does not establish an overall task improvement: actual help use adds a Code Mode +call, and cumulative/uncached tokens, local resources and end-to-end time have +mixed changes. A large observed inter-call gap remains charged and unexplained. + The [exact-value query study](value-query-study.md) adds location-to-value and value-to-operand-use queries, including inputs/constants without defining operations. Real stripped OpenSSH checks preserve all 250 SSA versions / 369 occurrences at diff --git a/experiments/atomic-codemode/discovery-summary-study.md b/experiments/atomic-codemode/discovery-summary-study.md index 800d2309e..0a1146713 100644 --- a/experiments/atomic-codemode/discovery-summary-study.md +++ b/experiments/atomic-codemode/discovery-summary-study.md @@ -84,3 +84,81 @@ isolated leases/private caches; OS/model caches and host contention remain uncontrolled. Review complete answers and paths before comparing whole-task usage, CPU and family memory. Module loading and model choices also change, so the result cannot isolate a pure byte-count effect. + +## Completed real-model repeat + +Runtime was frozen at `65b701df5`; the question/oracle remain byte-unchanged from +`8d13c67e7`. Both complete answers again pass the frozen source/oracle review: +ownership, PHI/call binding, both controller levels, four guards and widths, +exact shared/distinct SSA identities, valid avoidance witnesses, selected machine +checks and explicit unknowns. Both use one retained VM capture and close the +lease. Direct factors a longer common path suffix; every expanded edge is valid. +Code Mode gives the shorter witnesses. Each cites ten observed source IDs with +matching target/engine identity; both complete native function records equal the +prior golden. The cost summarizer does not perform natural-language grading. + +| Whole-task measure | Direct atomic v2 | Code Mode v2 | +| ------------------------------ | ---------------: | -----------: | +| Cumulative input tokens | 834,800 | 546,797 | +| Included cached input | 768,000 | 482,560 | +| Uncached input | 66,800 | 64,237 | +| Output tokens | 7,782 | 7,917 | +| Included reasoning output | 1,002 | 1,322 | +| Observed model responses | 25 | 19 | +| Client executions / waits | 23 / 1 | 17 / 1 | +| MCP attempts | 38 | 18 | +| Rejected MCP / inner API reads | 0 / 0 | 0 / 0 | +| Successful atomic fact reads | 35 | 32 | +| Model wall seconds | 251.325 | 422.804 | +| Family wall seconds | 251.925 | 423.457 | +| Family CPU seconds | 51.979 | 54.169 | +| Kernel process-family peak MiB | 922.7 | 937.3 | +| Native source-capture wall s | 37.061 | 36.472 | + +Both request `gpt-6.1-sol/high` through Codex 0.161.0 with the unchanged complete +18/4-tool manifests and common guidance. Both actually use client JavaScript +`exec`/`wait`; feature flags do not establish absence of that surface. Per-response +usage sums agree. Cached input is included in input, reasoning in output. +Runtime, engine and artifact fingerprints remain unchanged. Both profiles and +payloads have verified cleanup, OOM/kill 0 and no resource abort under the same +bounds. Resolved service build, remote compute and OS/model cache effects remain +unknown. These are fresh leases/private native caches, not warm task results. + +Actual help use matters. Direct v2 skips the index and reads selected complete +contracts once (24,238 bytes), versus two calls/24,733 bytes in v1. Code Mode v2 +uses the 2,849-byte index, then two contract calls: three help calls/30,520 bytes +total, versus two/36,504 in v1. Thus the complete Code Mode help path shrinks by +about 16%, with one extra call; the index's 71% byte reduction is not a 71% +whole-task token reduction. + +Compared with its own v1 run, Code Mode v2 has slightly lower uncached input +(66,377 to 64,237), but higher cumulative input (503,113 to 546,797), output and +elapsed time. Direct v2 has lower cumulative input but higher uncached input and +elapsed time than direct v1. Neither before/after pair proves a consistent overall +benefit. Different model choices, module loading, reversed order and uncontrolled +service/host/cache behavior prevent attributing changes solely to summaries. + +Captured client call/output timestamps locate a 189.366-second Code Mode gap +between two executions that both report `Script completed`; the executions +themselves last approximately 0.171/0.154 seconds. Matched client-tool interval +unions are 45.909 seconds direct and 50.383 Code Mode. Such intervals are not pure +native work, network or model latency, and gaps after yielded calls can overlap +background work. Here the large gap follows a completed call/PHI read. Its cause +is not established; stderr has no retry/reconnect/timeout diagnostic. The gap +remains charged in the reported wall time. Similar native-capture times and local +CPU do not establish a cause for the end-to-end latency difference. + +Both models avoid the C tool. They still read numerous CFG neighborhoods to +construct a few path witnesses; Code Mode again searches an internally collected +successor inventory. This recurring composition cost motivates testing a shared, +bounded recovered-path query next. It is not implemented by this metadata change. +Availability behavior under a stock/unconfigured provider is not newly established +by this native-IR model pair. No broad complex-software superiority or stable +Code Mode advantage is claimed. + +The earlier [v1 results](control-query-study.md) are retained rather than replaced. +Local v2 records are `sqlite-controller-{direct,codemode}-v2*`, +`controller-summary-repeat-costs.json`, `discovery-summary-answer-source-checks.json` +and `discovery-summary-client-timing.json`. Kernel peaks include charged file +cache, not RSS; startup/final generation outside the first/last client call is +not covered by the interval union. All whole-task costs remain in the main table.