rea/experiments/analysis-efficiency/model-context.md

# Observe the client boundary before claiming token savings

Continues [#2016](https://github.com/morluto/rea/issues/2016) after the managed
summary [PR #2413](https://github.com/morluto/rea/pull/2413) merged as
`82cfcee19f18b0eeae9a330ba04e0411cdf85225`. That opt-in view has verified smaller
delivery and complete-default migration, while its original real-model cohort
does not establish token savings. This follow-up measures why raw reply bytes
and aggregate model usage cannot be treated as interchangeable observations.

## Concrete diagnostic observation

An individual real Codex CLI 0.161.0 control uses frozen runtime
`ae8051302b5834322fa00cfa31c968ae74ab6ed9`, the unchanged ShareX identity question,
requested `gpt-5.6-terra` high and the same two enabled REA tools. A private
diagnostic copy removes the original runner's explicit `--ephemeral` flag.
Authentication/profile cleanup still runs; only selected tool/usage items are
retained after filtering the owned rollout. Its source rollout SHA-256 is
`d4df41d6187880d177f00b7533d85ad032c9c4ec502a39e7d5a566180e10461b` and the retained
filtered-item SHA-256 is
`0d5b4fb67c5674d1ba14ac32ae37e9a72b0368f4d9f5f9f81eb4d13193cda260`.
The original whole-copy is removed; account metadata, world state, messages and
internal passthrough are not retained in the filtered receipt.

Despite the requested `direct` configuration label, the observed client uses
three JavaScript `exec` calls. First it filters `ALL_TOOLS` with a broad
`/REA|reverse|artifact|managed|assembly|metadata/i` expression and prints matching
metadata. The client reports truncation of a 164,005-token original output.
This is the client's output measurement, not billed input or a complete inventory.
Next it invokes complete managed inspection but selects requested normalized
identity/classification fields before printing. Finally it invokes native-boundary
inspection and prints selected fields. The actual original complete MCP replies
remain available in the separate public-event transcript.

The persisted custom-tool output values are 41,704 / 2,154 / 13,489 UTF-8 bytes.
These are the serialized recorded output **values**, not raw wire bytes or exact
submitted model requests. The trace reports four per-response usage observations:

| Response |  Input | Cached input | Output |
| -------- | -----: | -----------: | -----: |
| 1        | 10,400 |            0 |    132 |
| 2        | 19,992 |        9,984 |    456 |
| 3        | 21,242 |       19,200 |    251 |
| 4        | 24,799 |       20,224 |  1,535 |

Their totals match public turn usage: 76,433 input, 49,408 cached, 27,025 uncached
and 2,374 output, with 61.436 seconds observed wall time. Two MCP calls alone do
not describe the client's discovery/execution rounds. This proves the observed
control can project a complete result before printing it; it does not establish
that every earlier control/candidate did the same, or attribute every token
difference to discovery. The earlier `direct` label specifies requested client
configuration, not a verified absence of an executor.

Initial transcript-capture attempts retained empty source lists because the
original runner intentionally passed `--ephemeral`. A first private flag-removal
edit missed its newline and failed; that candidate remains an unsuccessful
capture attempt. The later corrected control is an individual diagnostic,
not a comparable pair with that candidate. Other exploratory aggregates remain
private and are not silently folded into the original four-trial cohort.

## Reproducible optional capture

`agent-trial.py --capture-tool-context` omits `--ephemeral` only for that explicitly
selected temporary trial. Its default is unchanged. The profile still disappears
in `finally`, including when capture fails. The capture reads only its owned
`sessions/*.jsonl` records, rejects symlink sources, and retains:

- Custom/function tool calls and replies, preserving call IDs and exact values.
- Per-response token counts and cumulative counts, with original response IDs.
- Client token-count events, retaining explicit unknown `info: null`.

It excludes session/account metadata, world state, user/developer messages,
encrypted reasoning, internal passthrough and unknown nested usage fields. It
does not copy authentication, configuration or the ambient environment. Receipts
record source/retained digests, event counts, malformed-line failures and missing
rollouts as unavailable. Capture is an observation of persisted client items,
not proof of the exact request accepted by a remote model service.

Run the same frozen runtime, question, tool selection and capture mode on both
sides. Preserve source/settings hashes and separate requested versus observed
behavior. Use the existing resource wrapper with unchanged one CPU, 4 GiB hard
memory, 3 GiB early-stop, no swap and task/host safeguards. Final physical counters
remain unknown when transient cgroups disappear; this instrumentation does not
repair that measurement limitation.

The focused Python tests run through the existing CI `test:python` gate.
They cover account/ambient exclusion while preserving caller data, malformed and
missing records, foreign symlinks, and capture failure without copied-credential
leakage. The cleanup test simulates a CLI and establishes only cleanup semantics;
it is not real-model/provider evidence.

## An explicit discovery-guidance experiment

`--guide-enabled-tools` is another opt-in experiment flag. It requires a nonempty
`--enabled-tools` list and adds the exact corresponding `mcp__rea__...` function
names to the temporary developer instructions, requesting only exact-name
metadata discovery. It preserves the analyst question, operation order, tool
selection and provider inputs. The default instruction is unchanged. Missing or
invalid selected tools fail before creating an output/profile. The test gate
also verifies the new prerequisite without acquiring a profile.

Compare guided and unguided runs on the **same** frozen REA runtime with matching
capture mode and requested facts. A smaller discovery response or additional
hint is not itself a task/token success: grade answers, count all response usage,
preserve failed/truncated runs and repeat both orders before recommending this
guidance. This is a caller-side experiment, not a new REA tool or provider preset.

## First committed capture pair

Checkpoint `9ebdcc458c399b9bdecaaca2ef47422888231181` runs the unchanged question
against frozen control `ae8051302b5834322fa00cfa31c968ae74ab6ed9` and candidate
`8bc1601ca51e417ca472d90ea0367e9418a1a0f2`, both with capture enabled and no tool
guidance. Both captures succeed with 10 retained events, unchanged source bytes,
removed profiles, two successful MCP calls and passing selected factual checks.
Full grader gates remain false because final physical resource counters are
unavailable; factual subsets do not override that limitation.

| Side              |  Input | Cached | Uncached | Output | Wall seconds |
| ----------------- | -----: | -----: | -------: | -----: | -----------: |
| Complete control  | 52,656 | 27,136 |   25,520 |  2,015 |       57.736 |
| Summary candidate | 56,655 | 29,184 |   27,471 |  1,534 |       55.055 |

Both clients make two executor calls and three response-usage observations.
First they print broad discovered metadata, with truncation. Then each executor
sequentially performs both MCP operations, preserving selected-target dependency
order. The control prints a 2,086-byte output value; the candidate prints 22,770
bytes. Their discovery output values are 41,723 and 41,675 bytes. These are
recorded serialized values and remain distinct from raw MCP envelopes, billed
tokens and exact submitted requests. Both requested modes are `direct`; the
actual executor use is separately observed.

The candidate's smaller raw metadata reply does not result in a smaller printed
client value in this pair. Candidate input/uncached tokens remain higher. The
observed projection and round counts are useful constraints for the next
experiment; this one pair does not attribute all usage differences to either
formatting or discovery, or establish a reliable speedup.

## Same-runtime guided discovery, repeated in both orders

Checkpoint `e20bd0e2a905f06bc8c56fe04b0e61f3da88636a` runs four actual Codex trials
in unguided/guided/guided/unguided order. All use the **same** frozen candidate
runtime, artifact, original question, requested model/reasoning, two selected
tools, provider settings and capture mode. Only the explicit discovery-guidance
treatment changes. Every trial selects artifact summary and makes the same two
successful MCP calls in dependency order; source bytes stay unchanged, captures
succeed and profiles are removed.

| Trial      |   Input | Cached | Uncached | Output | Seconds | Observed responses |
| ---------- | ------: | -----: | -------: | -----: | ------: | -----------------: |
| Unguided 1 |  77,251 | 50,432 |   26,819 |  1,931 |  62.066 |                  4 |
| Guided 1   |  39,175 | 23,040 |   16,135 |  1,763 |  55.658 |                  3 |
| Guided 2   |  58,336 | 38,144 |   20,192 |  1,626 |  50.685 |                  4 |
| Unguided 2 | 103,301 | 72,704 |   30,597 |  1,983 |  60.336 |                  5 |

Both guided discovery output values repeat at 12,336 bytes without truncation,
versus 41,709 bytes with truncation on both unguided trials. The recorded
executor/output usage explains observable discovery and round differences; it
does not certify exact remote request contents or attribute every token to the
hint. CPU samples are 6.303–6.554 seconds and peak family memory 684,519,424–
712,028,160 bytes, with no guard abort. Final resource counters remain unknown.
These samples do not establish a memory or CPU improvement.

Unguided 1 omits `classification.container` and fails the selected factual gate;
it is not a fully acceptable comparison answer. Guided 1 and Unguided 2 pass
that gate. Guided 2 initially fails exact coverage-object equality because it
adds `module_ref_count: 0` and `pinvoke_import_count: 0` to the explicitly named
native scope. Both numbers exactly match the producer's returned arrays. The
grader now explicitly verifies only these two known additional counts, including
integer type, while still requiring all original coverage facts. Unknown extra
claims, conflicting aliases, wrong counts, boolean/string counts and omitted
issues continue to fail. Source coverage remains unchanged; returned-row counts
do not establish global absence. Original failure receipts remain preserved;
the regrade separately records its new source hash. Guided 2 then passes selected
facts, but every **full** gate still fails unknown final physical resource checks.

The second comparison has two factually acceptable answers: observed guided input
falls about 43.5%, uncached input about 34.0%, and wall time about 16.0%. The first
comparison has an incomplete unguided answer and is reported without counting it
as equivalent completed work. Two trials per treatment on one question/application
remain a preliminary signal, not a general default recommendation. The hint stays
opt-in; wider questions/domains, count/coverage representation and client round
behavior need further verification before promoting it into general guidance.

## Real Android follow-up: the primary outcomes are mixed

Checkpoint `22834c7d73056e2480b426848f58c31af0c307bc` checks the pinned real
NewPipe 0.29.1 APK using actual configured JADX 0.7.1/Java, the original Android
question and the same frozen candidate runtime. First a fresh SDK workflow
validates three live advertised operations and produces the comparison facts:
package/version identity, every case-sensitive MainActivity match and all
reported onCreate entries, preserving signatures, overload indices and null DEX
descriptors. It takes 26.961 seconds; the owned scope samples 1,234,735,104 bytes.

Both subsequent actual Codex trials use the same three selected tool names,
capture mode, model/reasoning, question and provider settings. Only discovery
guidance changes. Each makes three successful MCP calls and five observed model
responses, passes selected facts against the fresh actual producer, preserves
source bytes and removes its temporary profile. No APK execution is requested.

| Trial    |  Input | Cached | Uncached | Output | Wall seconds | Sampled peak bytes |
| -------- | -----: | -----: | -------: | -----: | -----------: | -----------------: |
| Unguided | 99,913 | 80,640 |   19,273 |  1,527 |       66.797 |      1,405,829,120 |
| Guided   | 89,220 | 62,464 |   26,756 |  2,142 |       76.936 |      1,345,576,960 |

Total input falls about 10.7%, but uncached input increases and wall time is
longer. One pair does not demonstrate improved Android performance or memory.
No guard abort occurs; final counters remain unknown and full resource gates
remain false. This second domain is a concrete counterexample to generalizing
the favorable managed comparison. Keep both flags optional and continue
task-specific experiments before changing installed/default agent guidance.