357996cc89
* test(perf): pin genuine Fiddler captures and raw HTTP oracles Pin two public Microsoft SAZ captures with 13 and 1311 sessions, immutable source/digest/member coverage and predeclared caller-ordered transaction questions. Independently preserve stored HTTP headers, binary/empty bodies and archive coordinates; admit only known corpus bytes before ZIP allocation. Five reader regressions and both genuine selected-record checks pass. No production or performance-gain claim. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com> * test(perf): measure SAZ selection and retained recovery workflows Add authenticated complete/ordinal/retained-page SDK experiments, fixed real-Codex prompts and independent cited-byte grading. Preserve actual failed RPCs and exact retained-reference recovery when complete MCP delivery exceeds its receive buffer. Verify selected CLI/MCP facts, parameters and semantic Evidence identity. Keep a genuine partial-message admission-negative probe. Focused 12 tests and 10 actual-Evidence grading cases pass; no formal model or gain claim yet. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com> Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com> * test(perf): verify resolved SAZ Evidence selection parameters Distinguish omitted request ordinals from the complete resolved ordinal scope recorded by WebNetworkCaptureService. Share the expected sanitized Evidence representation between SDK checks and model grading; preserve original caller arguments. A focused regression and both saved genuine baseline RPCs confirm the correction. Keep original collector failures and costs unchanged; freeze a new driver before repeated measurements. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com> Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com> * docs(perf): retain mixed SAZ whole-task costs and failure boundaries Record both SDK batches, twelve corrected workflows, all four actual Codex attempts, independent byte equivalence and verified payload-phase CPU/memory. Distinguish successful-analysis MCP transport limits from real provider timeouts and negative partial-message admission. Three model questions complete; the only completed pair saves 1.6 percent input but takes 2 percent longer, with no consistent benefit. Keep production APIs/defaults/cache unchanged and preserve all deviations, failed attempts and unknowns. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com> Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com> --------- Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com> Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com>
215 lines
13 KiB
Markdown
215 lines
13 KiB
Markdown
# Fiddler SAZ selection checkpoint
|
||
|
||
This experiment addresses [#3132](https://github.com/morluto/rea/issues/3132).
|
||
Its runtime baseline is `ca1b2f8608c52ed721d3576ba5eeb1d66717b08c`.
|
||
Production APIs and defaults are unchanged. The measured cohort establishes no
|
||
stable whole-task token/time benefit; no new production preset, cache or API is
|
||
justified by this checkpoint.
|
||
|
||
## Genuine public corpus
|
||
|
||
Both archives come from Microsoft's
|
||
[Xbox Live Trace Analyzer](https://github.com/microsoft/xbox-live-trace-analyzer/tree/ad880ff00b26c4cae5e3c8864d3df438a03890ba/Source/TestData/CertWarnings)
|
||
at immutable revision `ad880ff00b26c4cae5e3c8864d3df438a03890ba`.
|
||
The repository's [MIT license](https://github.com/microsoft/xbox-live-trace-analyzer/blob/ad880ff00b26c4cae5e3c8864d3df438a03890ba/LICENSE.md)
|
||
and original source files remain upstream. No sample binaries, raw HTTP material,
|
||
authentication values, configs or detailed results are committed.
|
||
|
||
| Artifact | Archive bytes | Expanded member bytes | Sessions | Members | Producer archive comment |
|
||
| ------------------------ | ------------: | --------------------: | -------: | ------: | ------------------------ |
|
||
| `xbox-clear_nomp.saz` | 43,515 | 62,609 | 13 | 42 | Fiddler v4.5.1.2 |
|
||
| `xbox-clear_mp_cs10.saz` | 5,223,519 | 9,223,820 | 1,311 | 3,939 | Fiddler v4.6.2.0 |
|
||
|
||
`corpus.json` pins download URLs, file sizes, SHA-256, Git blob IDs, directory
|
||
coverage and question ordinals. Downloaded bytes match their Git blob and
|
||
SHA-256. A third public 103-session archive is pinned as an admission-negative
|
||
probe, outside completed-question comparisons. Archive comments report their producer
|
||
versions; no local Fiddler installation or independent producer-version proof
|
||
is claimed.
|
||
|
||
## Fixed analyst question
|
||
|
||
Before examining transaction facts, select ordinals `[12, 0, 6]` in the control
|
||
and `[1310, 0, 655]` in the larger archive. Return that caller-selected order,
|
||
original padded session IDs and archive-member coordinates, request method/URL,
|
||
response status/version, all recorded Content-Type pairs in their original
|
||
spelling/order, and stored response-body byte length/SHA-256.
|
||
|
||
Bodies retain their archived encoding. CONNECT trailing bytes are stored data,
|
||
not an inferred HTTP application body or certificate interpretation. Empty bodies
|
||
have zero bytes and the empty digest; an absent response remains absent. Header
|
||
duplication/folding, Latin-1 spelling and binary body bytes are independently
|
||
read without REA imports. Synthetic reader tests cover those semantics and
|
||
malformed input; they establish no provider coverage.
|
||
|
||
The oracle admits only immutable corpus size/digest matches before constructing
|
||
ZIP objects, preserves central-directory session/member order, reads selected
|
||
members with size sentinels and CRC validation, and checks the source digest
|
||
again. Necessary facts are independently checked for all six selected records.
|
||
|
||
```sh
|
||
python3 experiments/analysis-efficiency/test_saz_oracle.py
|
||
python3 experiments/analysis-efficiency/saz-oracle.py xbox-clear_nomp.saz /absolute/clear_nomp.saz
|
||
python3 experiments/analysis-efficiency/saz-oracle.py xbox-clear_mp_cs10.saz /absolute/clear_mp_cs10.saz
|
||
```
|
||
|
||
The baseline build, five oracle regressions and the SDK checker semantics passed
|
||
on Linux. Real MCP preflight checks verify control retained pages/repeated reads,
|
||
the large ordinal-selected result, and large complete-workflow recovery. The
|
||
default large MCP return completes analysis but exceeds the existing 10 MiB stdio
|
||
receive-buffer budget; its exact returned retained Evidence reference supports
|
||
summary and complete pages. Recovery preserves the failed RPC and every follow-up
|
||
call, and matches the independently checked necessary facts from direct selection.
|
||
This transport limit is not a decoding timeout. Formal repetitions and actual
|
||
model tasks are reported below; preflight observations are not eligible cost pairs.
|
||
|
||
The 103-session `xbox-x49_mp_cs21.saz` candidate contains an empty stored response
|
||
member `raw/099_s.txt`, lacking an HTTP header/body separator. On this baseline,
|
||
whole-capture admission fails with `invalid_request` / `invalid_format`. Its
|
||
ordinals `[102, 0, 51]` were fixed before inspecting its transaction facts. The
|
||
failed attempt/source member is preserved as a negative probe; no parser repair
|
||
or successful-task comparison is claimed for that archive.
|
||
|
||
Other platform behavior remains unverified.
|
||
Detailed outputs stay private; missing resource counters remain unknown.
|
||
|
||
The large selected CLI workflow also matches independent bytes and MCP facts,
|
||
parameters and Evidence ID. A separate default complete CLI attempt hit the
|
||
provider's 30-second timeout under the one-CPU guard; that failed attempt is
|
||
preserved separately from the successful-analysis MCP receive-buffer failure.
|
||
It does not prove a general CLI failure or identify an engine-stage cause.
|
||
|
||
Ten grading preflight cases cover valid actual Evidence, wrong final digests,
|
||
unknown citations, missing limitations, answer order, failed calls before/after
|
||
valid data, semantic-ID corruption and actual large retained recovery. Their
|
||
answers/events are injected checker tests, not actual model trials. An invalid
|
||
later duplicate is rejected by the domain's semantic-ID owner and does not erase
|
||
earlier authenticated observations.
|
||
|
||
## SDK and model reproduction
|
||
|
||
```sh
|
||
node experiments/analysis-efficiency/measure.mjs \
|
||
--runtime-root /absolute/frozen-runtime --revision ca1b2f8608c52ed721d3576ba5eeb1d66717b08c \
|
||
--artifact xbox-clear_nomp.saz --path /absolute/clear_nomp.saz \
|
||
--output /absolute/new-result-dir --schemas selected --saz-selection focused
|
||
node experiments/analysis-efficiency/saz-model-prompt.mjs \
|
||
--artifact xbox-clear_nomp.saz --selection focused > /absolute/prompt.txt
|
||
```
|
||
|
||
Use the same runtime, inputs, explicit provider settings and resource profile
|
||
within a cost cohort; retain runtime/workflow/oracle/settings fingerprints. Repeat
|
||
both core archives with baseline/focused/focused/baseline order and retain all
|
||
failures/recovery RPCs. The `retained` diagnostic workflow also measures preview,
|
||
two complete pages and a repeated first page. Preview omits headers, so preview
|
||
alone is not equivalent to the complete header/body question.
|
||
|
||
Run real Codex tasks through `agent-trial.py` with an isolated temporary profile,
|
||
the two exact `inspect_web_network_capture` / `inspect_analysis_view` tools,
|
||
the same requested model/reasoning and at least two baseline/focused pairs for
|
||
the control. Grade cited observations with `review-saz.mjs`, an independently
|
||
produced oracle and the tested runtime. Count whole-task input/output tokens,
|
||
cached input, elapsed time and all calls; parsed JSON bytes are not model tokens.
|
||
The executed scope and all failed/deviating trials follow below.
|
||
|
||
## Executed SDK cohort
|
||
|
||
Runtime is frozen at `ca1b2f8608c52ed721d3576ba5eeb1d66717b08c`; corrected
|
||
experiment source is `286c3e48ad50d9e84baa3af00ca522a8cef36e95`. All corrected
|
||
SDK runtime/workflow/oracle/provider-setting fingerprints match. Node is 24.18.0
|
||
and MCP SDK is 2.3.1. SDK processes have a 1 GiB Node heap ceiling; model MCP
|
||
profiles separately record 512 MiB. Compare within each cohort.
|
||
|
||
The initial batch preserves 12 workflows / 36 RPCs. Two control baseline collectors
|
||
incorrectly expected omitted request ordinals to remain omitted in Evidence;
|
||
the owning application records the resolved full scope. Original RPC arguments
|
||
preserve omission. Both saved replies independently validate correct typed
|
||
Evidence/facts after the checker correction. Original failures, hashes and costs
|
||
remain unchanged and are not pooled into the corrected batch.
|
||
|
||
The corrected batch executes 12 workflows / 32 RPCs: **11 workflows pass** all necessary
|
||
byte/header/member checks with equal facts across successful arms. One large
|
||
default complete query hits `provider_timeout` at the 30-second provider budget.
|
||
Failed complete-return RPCs in successful retained recovery also stay in cost.
|
||
|
||
| Archive / workflow | Repetition outcomes | All parsed reply bytes | RPC ms | Whole runner ms |
|
||
| ------------------ | ------------------- | ---------------------: | --------------------- | --------------------- |
|
||
| control / baseline | passed / passed | 249272 / 249272 | 817.904 / 817.032 | 5393.982 / 4922.524 |
|
||
| control / focused | passed / passed | 65390 / 65390 | 805.492 / 835.598 | 4897.670 / 4924.772 |
|
||
| control / retained | passed / passed | 146981 / 146981 | 1846.344 / 8258.135 | 5861.567 / 18108.568 |
|
||
| large / baseline | failed / passed | 520 / 114294 | 32303.203 / 13623.033 | 52432.342 / 18064.360 |
|
||
| large / focused | passed / passed | 535767 / 535767 | 32917.417 / 23536.051 | 52933.865 / 37045.394 |
|
||
| large / retained | passed / passed | 174477 / 174477 | 10970.973 / 11550.158 | 15140.224 / 15669.775 |
|
||
|
||
Control direct selection reduces parsed replies **73.8%**. Large complete-result
|
||
recovery delivers 114,294 bytes, while direct ordinal selection retains the original
|
||
container inventory and delivers 535,767 bytes. The five-call retained diagnostic
|
||
delivers 174,477 bytes, including preview and repeated-page verification. A smaller
|
||
record set alone does not guarantee less delivery; bytes are not wire bytes or
|
||
model tokens.
|
||
|
||
Initial RPCs combine decoding, validation, canonicalization and delivery; no
|
||
decode-only/format-only attribution is claimed. First-use and repeated retained
|
||
reads remain separate RPC observations. Fresh client/server processes do not
|
||
establish cold OS/provider caches. Substantial time variation precludes a stable
|
||
general speedup claim.
|
||
|
||
## Actual Codex outcomes
|
||
|
||
Four actual Codex CLI 0.161.0 trials request `gpt-5.6-terra`/`high` with the same
|
||
two enabled/preapproved passive tools in baseline/focused/focused/baseline order.
|
||
The requested alias is pinned; resolved backend identity is unknown. Temporary
|
||
profiles preserve original bytes. The requested mode is `direct`, but owned
|
||
tool-context logs show actual `exec` discovery/dispatch; neither guaranteed direct
|
||
presentation nor exact submitted model requests are established.
|
||
|
||
| Trial | Facts / fixed plan | MCP calls | Total input | Cached input | Uncached input, derived | Output | Seconds |
|
||
| ---------- | ------------------ | --------: | ----------: | -----------: | ----------------------: | -----: | ------: |
|
||
| baseline-1 | passed / failed | 4 | 97,988 | 53,504 | 44,484 | 1,556 | 51.266 |
|
||
| focused-1 | passed / failed | 2 | 96,433 | 51,456 | 44,977 | 1,400 | 52.288 |
|
||
| focused-2 | failed / failed | 0 | 21,379 | 9,984 | 11,395 | 317 | 16.301 |
|
||
| baseline-2 | passed / failed | 4 | 172,043 | 125,696 | 46,347 | 1,943 | 62.570 |
|
||
|
||
**Three of four tasks answer correctly.** All three add compatible retained-view
|
||
calls and differ from the initially prescribed single MCP-call plan. These
|
||
successful completed-task costs stay visible separately from plan conformance.
|
||
Baselines read three exact record views; the successful focus reads one complete
|
||
selected-parent page.
|
||
|
||
The second focus makes zero MCP calls and reports unavailable tools. Its owned
|
||
exact-name discovery returns an empty list, while the same names work in other
|
||
trials; stderr is empty. The cause is unknown, not proof of a REA outage. Keep
|
||
its 21,379 input tokens / 16.301 seconds as a failed task, without a replacement retry
|
||
or an apparent speed win.
|
||
|
||
Only the first pair completes both questions: total input decreases **1.6%**,
|
||
time increases **2.0%**, and derived uncached input increases **1.1%**. The second
|
||
pair has no completed focused question; no paired speed/token claim is valid.
|
||
Limitation prose was inspected: successful answers qualify stored bytes,
|
||
ZIP-directory order and absent transfer/content decoding. This small mixed cohort
|
||
justifies **no production optimization**.
|
||
|
||
## Resource and compatibility scope
|
||
|
||
Heavy work is serial with one CPU, 3 GiB early stop / 4 GiB hard family memory, no swap,
|
||
task ceilings and host-availability/ownership checks. Corrected SDK payload-exit
|
||
peak is 906,825,728 bytes (**865 MiB**); model peak is 873,148,416 bytes (**833 MiB**).
|
||
All 16 corrected SDK/model payload phases verify cleanup, no observed OOM/abort
|
||
and captured CPU counters. Terminal whole-scope counters remain unknown. Failed
|
||
tool/model tasks do not become correct because their process exits cleanly.
|
||
|
||
Family CPU seconds at payload exit span 5.898–44.331 for corrected SDK workflows
|
||
(median 15.808), including failed/recovered attempts. Model baseline/focus/focus/
|
||
baseline observations are 7.573 / 7.303 / 5.810 / 7.618 CPU seconds; the third
|
||
model task failed and cannot establish an efficiency gain. These counters include
|
||
the owned runner/client/server/observer family, not decoder-only CPU.
|
||
|
||
Verification covers Linux, two immutable positive archives plus one genuine
|
||
partial-message negative source, one installed parser stack and one requested
|
||
model alias. Other platforms, Fiddler acquisition/import, live deployment or
|
||
certificate semantics, arbitrary layouts, direct presentation guarantees and
|
||
unmeasured engine stages remain unknown. Complete defaults, caller choices,
|
||
encoded bytes, redactions, semantic Evidence identity and session-owned views
|
||
remain the compatibility baseline. Reuse existing selectors/views; expand
|
||
SDK/cache/default behavior only after completed tasks show a benefit.
|