rea/experiments/analysis-efficiency/saz-selection-workflows.md

Fiddler SAZ selection checkpoint

This experiment addresses #3132. Its runtime baseline is ca1b2f8608c52ed721d3576ba5eeb1d66717b08c. Production APIs and defaults are unchanged. The measured cohort establishes no stable whole-task token/time benefit; no new production preset, cache or API is justified by this checkpoint.

Genuine public corpus

Both archives come from Microsoft's Xbox Live Trace Analyzer at immutable revision ad880ff00b26c4cae5e3c8864d3df438a03890ba. The repository's MIT license and original source files remain upstream. No sample binaries, raw HTTP material, authentication values, configs or detailed results are committed.

Artifact	Archive bytes	Expanded member bytes	Sessions	Members	Producer archive comment
xbox-clear_nomp.saz	43,515	62,609	13	42	Fiddler v4.5.1.2
xbox-clear_mp_cs10.saz	5,223,519	9,223,820	1,311	3,939	Fiddler v4.6.2.0

corpus.json pins download URLs, file sizes, SHA-256, Git blob IDs, directory coverage and question ordinals. Downloaded bytes match their Git blob and SHA-256. A third public 103-session archive is pinned as an admission-negative probe, outside completed-question comparisons. Archive comments report their producer versions; no local Fiddler installation or independent producer-version proof is claimed.

Fixed analyst question

Before examining transaction facts, select ordinals [12, 0, 6] in the control and [1310, 0, 655] in the larger archive. Return that caller-selected order, original padded session IDs and archive-member coordinates, request method/URL, response status/version, all recorded Content-Type pairs in their original spelling/order, and stored response-body byte length/SHA-256.

Bodies retain their archived encoding. CONNECT trailing bytes are stored data, not an inferred HTTP application body or certificate interpretation. Empty bodies have zero bytes and the empty digest; an absent response remains absent. Header duplication/folding, Latin-1 spelling and binary body bytes are independently read without REA imports. Synthetic reader tests cover those semantics and malformed input; they establish no provider coverage.

The oracle admits only immutable corpus size/digest matches before constructing ZIP objects, preserves central-directory session/member order, reads selected members with size sentinels and CRC validation, and checks the source digest again. Necessary facts are independently checked for all six selected records.

python3 experiments/analysis-efficiency/test_saz_oracle.py
python3 experiments/analysis-efficiency/saz-oracle.py xbox-clear_nomp.saz /absolute/clear_nomp.saz
python3 experiments/analysis-efficiency/saz-oracle.py xbox-clear_mp_cs10.saz /absolute/clear_mp_cs10.saz

The baseline build, five oracle regressions and the SDK checker semantics passed on Linux. Real MCP preflight checks verify control retained pages/repeated reads, the large ordinal-selected result, and large complete-workflow recovery. The default large MCP return completes analysis but exceeds the existing 10 MiB stdio receive-buffer budget; its exact returned retained Evidence reference supports summary and complete pages. Recovery preserves the failed RPC and every follow-up call, and matches the independently checked necessary facts from direct selection. This transport limit is not a decoding timeout. Formal repetitions and actual model tasks are reported below; preflight observations are not eligible cost pairs.

The 103-session xbox-x49_mp_cs21.saz candidate contains an empty stored response member raw/099_s.txt, lacking an HTTP header/body separator. On this baseline, whole-capture admission fails with invalid_request / invalid_format. Its ordinals [102, 0, 51] were fixed before inspecting its transaction facts. The failed attempt/source member is preserved as a negative probe; no parser repair or successful-task comparison is claimed for that archive.

Other platform behavior remains unverified. Detailed outputs stay private; missing resource counters remain unknown.

The large selected CLI workflow also matches independent bytes and MCP facts, parameters and Evidence ID. A separate default complete CLI attempt hit the provider's 30-second timeout under the one-CPU guard; that failed attempt is preserved separately from the successful-analysis MCP receive-buffer failure. It does not prove a general CLI failure or identify an engine-stage cause.

Ten grading preflight cases cover valid actual Evidence, wrong final digests, unknown citations, missing limitations, answer order, failed calls before/after valid data, semantic-ID corruption and actual large retained recovery. Their answers/events are injected checker tests, not actual model trials. An invalid later duplicate is rejected by the domain's semantic-ID owner and does not erase earlier authenticated observations.

SDK and model reproduction

node experiments/analysis-efficiency/measure.mjs \
  --runtime-root /absolute/frozen-runtime --revision ca1b2f8608c52ed721d3576ba5eeb1d66717b08c \
  --artifact xbox-clear_nomp.saz --path /absolute/clear_nomp.saz \
  --output /absolute/new-result-dir --schemas selected --saz-selection focused
node experiments/analysis-efficiency/saz-model-prompt.mjs \
  --artifact xbox-clear_nomp.saz --selection focused > /absolute/prompt.txt

Use the same runtime, inputs, explicit provider settings and resource profile within a cost cohort; retain runtime/workflow/oracle/settings fingerprints. Repeat both core archives with baseline/focused/focused/baseline order and retain all failures/recovery RPCs. The retained diagnostic workflow also measures preview, two complete pages and a repeated first page. Preview omits headers, so preview alone is not equivalent to the complete header/body question.

Run real Codex tasks through agent-trial.py with an isolated temporary profile, the two exact inspect_web_network_capture / inspect_analysis_view tools, the same requested model/reasoning and at least two baseline/focused pairs for the control. Grade cited observations with review-saz.mjs, an independently produced oracle and the tested runtime. Count whole-task input/output tokens, cached input, elapsed time and all calls; parsed JSON bytes are not model tokens. The executed scope and all failed/deviating trials follow below.

Executed SDK cohort

Runtime is frozen at ca1b2f8608c52ed721d3576ba5eeb1d66717b08c; corrected experiment source is 286c3e48ad50d9e84baa3af00ca522a8cef36e95. All corrected SDK runtime/workflow/oracle/provider-setting fingerprints match. Node is 24.18.0 and MCP SDK is 2.3.1. SDK processes have a 1 GiB Node heap ceiling; model MCP profiles separately record 512 MiB. Compare within each cohort.

The initial batch preserves 12 workflows / 36 RPCs. Two control baseline collectors incorrectly expected omitted request ordinals to remain omitted in Evidence; the owning application records the resolved full scope. Original RPC arguments preserve omission. Both saved replies independently validate correct typed Evidence/facts after the checker correction. Original failures, hashes and costs remain unchanged and are not pooled into the corrected batch.

The corrected batch executes 12 workflows / 32 RPCs: 11 workflows pass all necessary byte/header/member checks with equal facts across successful arms. One large default complete query hits provider_timeout at the 30-second provider budget. Failed complete-return RPCs in successful retained recovery also stay in cost.

Archive / workflow	Repetition outcomes	All parsed reply bytes	RPC ms	Whole runner ms
control / baseline	passed / passed	249272 / 249272	817.904 / 817.032	5393.982 / 4922.524
control / focused	passed / passed	65390 / 65390	805.492 / 835.598	4897.670 / 4924.772
control / retained	passed / passed	146981 / 146981	1846.344 / 8258.135	5861.567 / 18108.568
large / baseline	failed / passed	520 / 114294	32303.203 / 13623.033	52432.342 / 18064.360
large / focused	passed / passed	535767 / 535767	32917.417 / 23536.051	52933.865 / 37045.394
large / retained	passed / passed	174477 / 174477	10970.973 / 11550.158	15140.224 / 15669.775

Control direct selection reduces parsed replies 73.8%. Large complete-result recovery delivers 114,294 bytes, while direct ordinal selection retains the original container inventory and delivers 535,767 bytes. The five-call retained diagnostic delivers 174,477 bytes, including preview and repeated-page verification. A smaller record set alone does not guarantee less delivery; bytes are not wire bytes or model tokens.

Initial RPCs combine decoding, validation, canonicalization and delivery; no decode-only/format-only attribution is claimed. First-use and repeated retained reads remain separate RPC observations. Fresh client/server processes do not establish cold OS/provider caches. Substantial time variation precludes a stable general speedup claim.

Actual Codex outcomes

Four actual Codex CLI 0.161.0 trials request gpt-5.6-terra/high with the same two enabled/preapproved passive tools in baseline/focused/focused/baseline order. The requested alias is pinned; resolved backend identity is unknown. Temporary profiles preserve original bytes. The requested mode is direct, but owned tool-context logs show actual exec discovery/dispatch; neither guaranteed direct presentation nor exact submitted model requests are established.

Trial	Facts / fixed plan	MCP calls	Total input	Cached input	Uncached input, derived	Output	Seconds
baseline-1	passed / failed	4	97,988	53,504	44,484	1,556	51.266
focused-1	passed / failed	2	96,433	51,456	44,977	1,400	52.288
focused-2	failed / failed	0	21,379	9,984	11,395	317	16.301
baseline-2	passed / failed	4	172,043	125,696	46,347	1,943	62.570

Three of four tasks answer correctly. All three add compatible retained-view calls and differ from the initially prescribed single MCP-call plan. These successful completed-task costs stay visible separately from plan conformance. Baselines read three exact record views; the successful focus reads one complete selected-parent page.

The second focus makes zero MCP calls and reports unavailable tools. Its owned exact-name discovery returns an empty list, while the same names work in other trials; stderr is empty. The cause is unknown, not proof of a REA outage. Keep its 21,379 input tokens / 16.301 seconds as a failed task, without a replacement retry or an apparent speed win.

Only the first pair completes both questions: total input decreases 1.6%, time increases 2.0%, and derived uncached input increases 1.1%. The second pair has no completed focused question; no paired speed/token claim is valid. Limitation prose was inspected: successful answers qualify stored bytes, ZIP-directory order and absent transfer/content decoding. This small mixed cohort justifies no production optimization.

Resource and compatibility scope

Heavy work is serial with one CPU, 3 GiB early stop / 4 GiB hard family memory, no swap, task ceilings and host-availability/ownership checks. Corrected SDK payload-exit peak is 906,825,728 bytes (865 MiB); model peak is 873,148,416 bytes (833 MiB). All 16 corrected SDK/model payload phases verify cleanup, no observed OOM/abort and captured CPU counters. Terminal whole-scope counters remain unknown. Failed tool/model tasks do not become correct because their process exits cleanly.

Family CPU seconds at payload exit span 5.898–44.331 for corrected SDK workflows (median 15.808), including failed/recovered attempts. Model baseline/focus/focus/ baseline observations are 7.573 / 7.303 / 5.810 / 7.618 CPU seconds; the third model task failed and cannot establish an efficiency gain. These counters include the owned runner/client/server/observer family, not decoder-only CPU.

Verification covers Linux, two immutable positive archives plus one genuine partial-message negative source, one installed parser stack and one requested model alias. Other platforms, Fiddler acquisition/import, live deployment or certificate semantics, arbitrary layouts, direct presentation guarantees and unmeasured engine stages remain unknown. Complete defaults, caller choices, encoded bytes, redactions, semantic Evidence identity and session-owned views remain the compatibility baseline. Reuse existing selectors/views; expand SDK/cache/default behavior only after completed tasks show a benefit.