docs(experiment): retain mixed discovery repeat results

Both real-model answers satisfy the unchanged oracle. Record the smaller actual help path, all task costs, and the unexplained 189-second gap between completed client executions without removing it from elapsed time.

Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com>
Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
This commit is contained in:
committed 2026-10-16 19:32:32 +00:00
1 parent 65b701df54
commit 7259ea1dee
2 files changed
+83

No files matched your search

+5
View File
@@ -34,6 +34,11 @@ summaries; all full contracts and advertised tools remain available. Contracts
and session admission have separate owners, with native loading deferred to open.
Its repeat uses the same frozen control question and retains the earlier results.
That completed repeat again answers correctly in both lanes. The smaller index
does not establish an overall task improvement: actual help use adds a Code Mode
call, and cumulative/uncached tokens, local resources and end-to-end time have
mixed changes. A large observed inter-call gap remains charged and unexplained.
The [exact-value query study](value-query-study.md) adds location-to-value and
value-to-operand-use queries, including inputs/constants without defining operations.
Real stripped OpenSSH checks preserve all 250 SSA versions / 369 occurrences at
@@ -84,3 +84,81 @@ isolated leases/private caches; OS/model caches and host contention remain
uncontrolled. Review complete answers and paths before comparing whole-task
usage, CPU and family memory. Module loading and model choices also change, so
the result cannot isolate a pure byte-count effect.
## Completed real-model repeat
Runtime was frozen at `65b701df5`; the question/oracle remain byte-unchanged from
`8d13c67e7`. Both complete answers again pass the frozen source/oracle review:
ownership, PHI/call binding, both controller levels, four guards and widths,
exact shared/distinct SSA identities, valid avoidance witnesses, selected machine
checks and explicit unknowns. Both use one retained VM capture and close the
lease. Direct factors a longer common path suffix; every expanded edge is valid.
Code Mode gives the shorter witnesses. Each cites ten observed source IDs with
matching target/engine identity; both complete native function records equal the
prior golden. The cost summarizer does not perform natural-language grading.
| Whole-task measure | Direct atomic v2 | Code Mode v2 |
| ------------------------------ | ---------------: | -----------: |
| Cumulative input tokens | 834,800 | 546,797 |
| Included cached input | 768,000 | 482,560 |
| Uncached input | 66,800 | 64,237 |
| Output tokens | 7,782 | 7,917 |
| Included reasoning output | 1,002 | 1,322 |
| Observed model responses | 25 | 19 |
| Client executions / waits | 23 / 1 | 17 / 1 |
| MCP attempts | 38 | 18 |
| Rejected MCP / inner API reads | 0 / 0 | 0 / 0 |
| Successful atomic fact reads | 35 | 32 |
| Model wall seconds | 251.325 | 422.804 |
| Family wall seconds | 251.925 | 423.457 |
| Family CPU seconds | 51.979 | 54.169 |
| Kernel process-family peak MiB | 922.7 | 937.3 |
| Native source-capture wall s | 37.061 | 36.472 |
Both request `gpt-6.1-sol/high` through Codex 0.161.0 with the unchanged complete
18/4-tool manifests and common guidance. Both actually use client JavaScript
`exec`/`wait`; feature flags do not establish absence of that surface. Per-response
usage sums agree. Cached input is included in input, reasoning in output.
Runtime, engine and artifact fingerprints remain unchanged. Both profiles and
payloads have verified cleanup, OOM/kill 0 and no resource abort under the same
bounds. Resolved service build, remote compute and OS/model cache effects remain
unknown. These are fresh leases/private native caches, not warm task results.
Actual help use matters. Direct v2 skips the index and reads selected complete
contracts once (24,238 bytes), versus two calls/24,733 bytes in v1. Code Mode v2
uses the 2,849-byte index, then two contract calls: three help calls/30,520 bytes
total, versus two/36,504 in v1. Thus the complete Code Mode help path shrinks by
about 16%, with one extra call; the index's 71% byte reduction is not a 71%
whole-task token reduction.
Compared with its own v1 run, Code Mode v2 has slightly lower uncached input
(66,377 to 64,237), but higher cumulative input (503,113 to 546,797), output and
elapsed time. Direct v2 has lower cumulative input but higher uncached input and
elapsed time than direct v1. Neither before/after pair proves a consistent overall
benefit. Different model choices, module loading, reversed order and uncontrolled
service/host/cache behavior prevent attributing changes solely to summaries.
Captured client call/output timestamps locate a 189.366-second Code Mode gap
between two executions that both report `Script completed`; the executions
themselves last approximately 0.171/0.154 seconds. Matched client-tool interval
unions are 45.909 seconds direct and 50.383 Code Mode. Such intervals are not pure
native work, network or model latency, and gaps after yielded calls can overlap
background work. Here the large gap follows a completed call/PHI read. Its cause
is not established; stderr has no retry/reconnect/timeout diagnostic. The gap
remains charged in the reported wall time. Similar native-capture times and local
CPU do not establish a cause for the end-to-end latency difference.
Both models avoid the C tool. They still read numerous CFG neighborhoods to
construct a few path witnesses; Code Mode again searches an internally collected
successor inventory. This recurring composition cost motivates testing a shared,
bounded recovered-path query next. It is not implemented by this metadata change.
Availability behavior under a stock/unconfigured provider is not newly established
by this native-IR model pair. No broad complex-software superiority or stable
Code Mode advantage is claimed.
The earlier [v1 results](control-query-study.md) are retained rather than replaced.
Local v2 records are `sqlite-controller-{direct,codemode}-v2*`,
`controller-summary-repeat-costs.json`, `discovery-summary-answer-source-checks.json`
and `discovery-summary-client-timing.json`. Kernel peaks include charged file
cache, not RSS; startup/final generation outside the first/last client call is
not covered by the interval union. All whole-task costs remain in the main table.