diff --git a/docs/mcp-catalog-evaluation.md b/docs/mcp-catalog-evaluation.md new file mode 100644 index 0000000000..eff14206d8 --- /dev/null +++ b/docs/mcp-catalog-evaluation.md @@ -0,0 +1,134 @@ +# Measure MCP catalog costs in Codex + +Use actual harness usage to evaluate catalog changes. A `tools/list` JSON byte +count measures transport size. It does not establish how many tokens a client +submits to a model, whether it defers schemas, or the total cost of an analysis. +The pinned [Codex observations](verification/mcp-catalog-codex-2026-11-05.md) +include actual loading controls, an unavailable-host negative control and a +completed real-package pilot. + +## Run a controlled loading comparison + +Build the selected source checkout with the pinned Node and npm versions first: + +```sh +npm ci +npm run build:cached +npm run eval:mcp-catalog -- --model gpt-6.1-sol \ + --output "$HOME/.cache/rea-catalog-run" --repetitions 3 --plan +``` + +Remove `--plan` to execute the matrix. This makes real model requests using the +local Codex authentication. The output directory must not already exist. No +account configuration is changed. The runner copies only authentication into +an external disposable profile and removes that profile after each trial. +Execution stays outside both source checkouts even when results are saved inside +one, so ancestor repository instructions and skills do not pollute the trial. +The npm entrypoint holds the driver's artifact lock throughout the matrix. A +selected external checkout has its own lock, so the matrix also pins one runtime +fingerprint and binds its catalog receipt to that manifest. Freshness checks +stop further model calls if the runtime changes between trials. Each trial's +before/after fingerprints must match the matrix pin; different builds never form +a valid pair. Matrix admission/freshness audits sit outside individual guarded +trial resource measurements. Do not rebuild the selected runtime concurrently. + +This lane requires Linux, Python 3, a working user systemd manager with cgroup v2, +and an installed Codex CLI supporting the recorded flags. It reuses the existing +resource observer: one CPU, 4 GiB hard memory limit, 3 GiB early stop, no swap, +and a 192-task ceiling. Runs are sequential. Unsupported clients, resource stops, +failed startup, missing usage, and unverified cleanup remain failed measurements. +They do not become zero-cost successes. + +Each repetition compares no server, all REA tools, and a client-selected subset +(`binary_session` by default), in the supported client-code host mode. Alternate +repetitions reverse ordering. All variants receive the same developer instructions +and exact-response prompt. A valid loading trial completes one turn, returns +`CATALOG_OK`, and performs no tool calls. REA is marked `required` in every +server-enabled configuration. The runner explicitly disables/enables the stable +`code_mode_host` feature for direct/client-code modes. The old `code_mode` and +`code_mode_only` switches do not override Codex 0.165's enabled host default. +The requested flags and CLI version are retained; verify the actual call shape +in completed-task traces too. `--modes direct client-code` adds a diagnostic +host-disabled control. This installed Codex reports that disabling the host +makes tool dispatch unavailable; it does not provide a qualified eager/direct +comparison. An empty response alone cannot qualify tool availability. Errors +invalidate that profile even when the model still emits the expected marker. + +Useful controls: + +```sh +# Compare full input guidance with the default compact input presentation. +npm run eval:mcp-catalog -- --model gpt-6.1-sol \ + --output "$HOME/.cache/rea-catalog-full" --input-profile full --repetitions 3 + +# Choose exact tool names in the working host mode. +npm run eval:mcp-catalog -- --model gpt-6.1-sol \ + --output "$HOME/.cache/rea-catalog-selected" --modes client-code \ + --selected-tools analyze_javascript_application inspect_analysis_view +``` + +`enabled_tools` filters the catalog after client discovery. It does not reduce +the server's original `tools/list` transport. Server tool-family scope is a +separate existing control; do not attribute client filtering results to it. +The output-schema profile remains REA's default in these experiments. + +## Read the evidence + +`report.json` retains every attempted trial and compares each enabled-server +trial with its same-mode, same-repetition no-server control. It reports valid +pair counts and the median/range of observed input-token differences. Failed +pairs are explicit and excluded from the median. A small or negative delta can +reflect client framing, randomized workspace paths, and measurement variation; +it does not establish a negative schema cost. + +Usage, cleanup and resource receipts are collected independently of the verdict. +A failed host, malformed event or wrong marker can retain observed usage and +resource costs. Stage-specific issues explain invalidity; missing observations +remain unknown. A changed matrix records its unattempted cells explicitly. + +Each trial retains its exact prompt, runtime file digests, requested model alias, +Codex version, settings/source hashes, JSONL events, actual usage, answer, cleanup +receipt, and process-family resource receipt. The selected rollout capture keeps +per-response usage and tool exchanges; authentication, ambient environment, +account metadata, and encrypted reasoning are excluded. Tool output and local +paths remain evidence and should be reviewed before sharing the raw files. + +The metrics have distinct meanings: + +| Metric | Meaning and limit | +| ----------------------------------- | --------------------------------------------------------------------------------------------------------------- | +| Catalog bytes and SHA-256 | Built server advertisement, including output schemas; not model input. | +| `turn.completed.usage.input_tokens` | Observed whole-turn usage reported by Codex; includes cached input. | +| Cached/uncached input | Report both. Cache hits do not remove tokens from the context. | +| Per-response usage | Retained usage records help locate discovery and continuation costs. Do not add cumulative thread counts again. | +| Paired input delta | Incremental loading observation including instructions/client framing; not an exact schema-only token count. | +| Elapsed and family CPU | Whole guarded trial, including local setup and client/network waits; remote inference CPU is unmeasured. | +| Family peak memory | Kernel cgroup peak at payload exit, including observer and file cache; not final whole-scope counters. | + +These are fresh client/server sessions. OS caches and API prefix caches can +already be warm. No cold-cache claim follows from a new session. The requested +model alias is recorded; a resolved server-side model version is not inferred. +The harness usage is not an independent billing reconciliation. +The usage event shape follows the documented +[Codex non-interactive JSON output](https://developers.openai.com/codex/noninteractive#make-output-machine-readable). + +## Evaluate completed investigations too + +A no-tool response can avoid deferred discovery entirely. Before changing the +catalog based on this lane, compare completed tasks through the real harness. +Use `npm run verify:agent` and its known-answer rubric, or run a selected real +artifact through `tests/support/analysis-efficiency/agent-trial.py` under +`resource-run.py --observe-payload-exit`. Use the same task, artifact digest, +model/reasoning, authority, result coverage, and correctness oracle in every arm. +The existing runner accepts exact `--enabled-tools` and defaults to client-code +host mode. Its direct option is a host-disabled diagnostic, not a claim of +working native dispatch in the installed client. +Use `--capture-tool-context` to locate per-response costs. Its task summary +explicitly leaves factual correctness unassessed until an independent oracle or +transcript review establishes the requested facts. + +Count discovery, arguments, replies, retries, and continuations. Check the +requested facts against source bytes or another independent oracle, real produced +Evidence IDs and source links, partial/unknown coverage, and cleanup. Preserve +failed attempts. A successful small-package task is a pilot, not proof of broad +application performance or provider compatibility. diff --git a/docs/testing-lanes.md b/docs/testing-lanes.md index ff3cd9d5ea..e19786ff9f 100644 --- a/docs/testing-lanes.md +++ b/docs/testing-lanes.md @@ -1230,6 +1230,10 @@ package entrypoint as the script's first argument for package coverage. ## Agent evaluation and conformance records +For catalog-loading measurements, use the real Codex +[MCP catalog evaluation](mcp-catalog-evaluation.md). It separates observed usage +from advertisement bytes and from completed-task correctness. + `npm run verify:agent` evaluates investigation scenarios through a real local Codex CLI. The lane runs sequentially; select scenarios with `REA_AGENT_EVAL_SCENARIOS` and choose the installed client/model with diff --git a/docs/verification/mcp-catalog-codex-2026-11-05.json b/docs/verification/mcp-catalog-codex-2026-11-05.json new file mode 100644 index 0000000000..604180dd39 --- /dev/null +++ b/docs/verification/mcp-catalog-codex-2026-11-05.json @@ -0,0 +1,1027 @@ +{ + "version": 1, + "date_utc": "2026-11-05", + "source_baseline": "ec3410998b70290847f930538e57d0af0ac1bffb", + "environment": { + "platform": "Linux-6.1.0-54-cloud-amd64-x86_64-with-glibc2.36", + "node": "v24.22.1", + "npm": "11.16.0", + "codex": "codex-cli 0.165.0", + "model_requested": "gpt-6.1-sol", + "reasoning": "low", + "resolved_model_identity": "unknown", + "cache_scope": "Fresh processes/profiles; warm OS/API prefix caches possible. Observed cached input is included." + }, + "raw_artifact_root": "/home/yann/yann/rea-mcp-catalog-eval/.cache/catalog-eval", + "catalog": { + "path": "/home/yann/yann/rea-mcp-catalog-eval/dist/mcp-catalog.json", + "sha256": "f92bf1c16abb224b4702208ee54e12707d9312d106f7424ac2e44e3940ee6c41", + "bytes": 4029995, + "tool_count": 234, + "selected_tools": ["binary_session"], + "tools_json_bytes": 4018383, + "description_json_bytes": 148355, + "input_schema_json_bytes": 138747, + "output_schema_json_bytes": 3681386, + "scope": "Built server catalog, not submitted model context. Client filtering still discovers the server catalog." + }, + "measurement_sources_sha256": { + "catalog-eval.py": "232b13eb625ba2b45c6f30ddc0840fd3ff5347812c8ba04ad4650245337cd3e1", + "agent-trial.py": "31e4180207c2f3fb9ecb2a777ecfdccb69dec7028f2e6022575591085e45d4db", + "codex_context.py": "79c5ec253c836f67d05e4da19f80283c1d40c292dcbb78f3f984095031bcc52c", + "resource-run.py": "7a9642ab86ba34290564f636eb080cacf38b4185a254028680b238bff172993b", + "resource_payload.py": "6f089e645459eba80c5d73e39c21606c78173a44daf49fabc3b88c4631ce51c6", + "provider-environment.json": "59103e26d51bca9f82f18b43c5544de4522866ff79d9cfe718cc91cbab13d7be" + }, + "load_trials": [ + { + "repetition": 1, + "mode": "direct", + "arm": "none", + "status": "invalid", + "usage": { + "input_tokens": 11118, + "cached_input_tokens": 4992, + "cache_write_input_tokens": 0, + "output_tokens": 8, + "reasoning_output_tokens": 0, + "uncached_input_tokens": 6126 + }, + "runtime_revision": "5038280f96af74e4e116ce78c4b3b77017cc69ac", + "runtime_fingerprint_sha256": "5dccf8e4e9a300a402153887ddd8f018e392be9b29630df9f3a30ea70265570c", + "runtime_bytes_unchanged": true, + "events_sha256": "7bb0c7cdd2de9e96d914b894501025c9a38007545a7c0ba69ec89e9b67fd7150", + "retained_context_sha256": "3b144d6967690df64cac69803c17238aa8fee7a1b7700dbe3089b65a2d4bceb6", + "resources": { + "elapsed_seconds": 14.084, + "cpu_seconds": 1.726859, + "peak_process_family_bytes": 197263360, + "payload_exit_code": 0, + "payload_cleanup_verified": true + }, + "client_errors": [ + "Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`." + ] + }, + { + "repetition": 1, + "mode": "direct", + "arm": "all", + "status": "invalid", + "usage": { + "input_tokens": 11121, + "cached_input_tokens": 9216, + "cache_write_input_tokens": 0, + "output_tokens": 8, + "reasoning_output_tokens": 0, + "uncached_input_tokens": 1905 + }, + "runtime_revision": "5038280f96af74e4e116ce78c4b3b77017cc69ac", + "runtime_fingerprint_sha256": "5dccf8e4e9a300a402153887ddd8f018e392be9b29630df9f3a30ea70265570c", + "runtime_bytes_unchanged": true, + "events_sha256": "d85e4997858c8418b4d1a208541c4f08ebc38f80c9b24e22a8fa9471b2128f5b", + "retained_context_sha256": "043369767e917cccb92c5eb2f941f86fc123021068a3e861ebc528ef10b6d0eb", + "resources": { + "elapsed_seconds": 13.606, + "cpu_seconds": 2.629986, + "peak_process_family_bytes": 350830592, + "payload_exit_code": 0, + "payload_cleanup_verified": true + }, + "client_errors": [ + "Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`." + ] + }, + { + "repetition": 1, + "mode": "direct", + "arm": "selected", + "status": "invalid", + "usage": { + "input_tokens": 11118, + "cached_input_tokens": 9216, + "cache_write_input_tokens": 0, + "output_tokens": 8, + "reasoning_output_tokens": 0, + "uncached_input_tokens": 1902 + }, + "runtime_revision": "5038280f96af74e4e116ce78c4b3b77017cc69ac", + "runtime_fingerprint_sha256": "5dccf8e4e9a300a402153887ddd8f018e392be9b29630df9f3a30ea70265570c", + "runtime_bytes_unchanged": true, + "events_sha256": "7c7b8096234d6fab79ed8c09f03291095a0a1936f4a821ca317a0219ef3af61e", + "retained_context_sha256": "eb3cb2abad8fad861e13f4b1a0ab9d58fc2f4a7da1d48a1b0c492c66e086653c", + "resources": { + "elapsed_seconds": 12.881, + "cpu_seconds": 2.542219, + "peak_process_family_bytes": 351158272, + "payload_exit_code": 0, + "payload_cleanup_verified": true + }, + "client_errors": [ + "Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`." + ] + }, + { + "repetition": 1, + "mode": "client-code", + "arm": "none", + "status": "valid", + "usage": { + "input_tokens": 11112, + "cached_input_tokens": 9216, + "cache_write_input_tokens": 0, + "output_tokens": 8, + "reasoning_output_tokens": 0, + "uncached_input_tokens": 1896 + }, + "runtime_revision": "5038280f96af74e4e116ce78c4b3b77017cc69ac", + "runtime_fingerprint_sha256": "5dccf8e4e9a300a402153887ddd8f018e392be9b29630df9f3a30ea70265570c", + "runtime_bytes_unchanged": true, + "events_sha256": "bff8c370cf6ee92f4768f4ab02271e6064a2dc3c17fd8330775850f18f4b7d1a", + "retained_context_sha256": "0716689f3e06ef22768b6b2a1a7e21445bd00f11fd894885e8c2f19307329f07", + "resources": { + "elapsed_seconds": 15.492, + "cpu_seconds": 1.862212, + "peak_process_family_bytes": 196980736, + "payload_exit_code": 0, + "payload_cleanup_verified": true + } + }, + { + "repetition": 1, + "mode": "client-code", + "arm": "all", + "status": "valid", + "usage": { + "input_tokens": 11115, + "cached_input_tokens": 9216, + "cache_write_input_tokens": 0, + "output_tokens": 8, + "reasoning_output_tokens": 0, + "uncached_input_tokens": 1899 + }, + "runtime_revision": "5038280f96af74e4e116ce78c4b3b77017cc69ac", + "runtime_fingerprint_sha256": "5dccf8e4e9a300a402153887ddd8f018e392be9b29630df9f3a30ea70265570c", + "runtime_bytes_unchanged": true, + "events_sha256": "da50c705016a585201be8509f280aee158b8e84d43c56065c539d07c2d3cd64d", + "retained_context_sha256": "838a1a7fb916d7ff01ce0b34a2981d041477991ff113f2284fb2dcf4cfd2deb6", + "resources": { + "elapsed_seconds": 14.788, + "cpu_seconds": 2.684778, + "peak_process_family_bytes": 364081152, + "payload_exit_code": 0, + "payload_cleanup_verified": true + } + }, + { + "repetition": 1, + "mode": "client-code", + "arm": "selected", + "status": "valid", + "usage": { + "input_tokens": 11115, + "cached_input_tokens": 9216, + "cache_write_input_tokens": 0, + "output_tokens": 8, + "reasoning_output_tokens": 0, + "uncached_input_tokens": 1899 + }, + "runtime_revision": "5038280f96af74e4e116ce78c4b3b77017cc69ac", + "runtime_fingerprint_sha256": "5dccf8e4e9a300a402153887ddd8f018e392be9b29630df9f3a30ea70265570c", + "runtime_bytes_unchanged": true, + "events_sha256": "52fc7e3bed7fda1d6f1ee03886e2aa8e6072d2da85953318468822150f3aec34", + "retained_context_sha256": "0ffd05dac2b61105dc3e668586921fe01b9742808a016858a78c175bc7848f93", + "resources": { + "elapsed_seconds": 13.792, + "cpu_seconds": 2.587098, + "peak_process_family_bytes": 352436224, + "payload_exit_code": 0, + "payload_cleanup_verified": true + } + }, + { + "repetition": 2, + "mode": "client-code", + "arm": "selected", + "status": "valid", + "usage": { + "input_tokens": 11121, + "cached_input_tokens": 9216, + "cache_write_input_tokens": 0, + "output_tokens": 8, + "reasoning_output_tokens": 0, + "uncached_input_tokens": 1905 + }, + "runtime_revision": "5038280f96af74e4e116ce78c4b3b77017cc69ac", + "runtime_fingerprint_sha256": "5dccf8e4e9a300a402153887ddd8f018e392be9b29630df9f3a30ea70265570c", + "runtime_bytes_unchanged": true, + "events_sha256": "fd2147324479e63137ae2f8acc9746d70e56225c128f2ec25429796f2836faf8", + "retained_context_sha256": "fa6ad44c48da1ff316a4618da8178bac85402dcf26c13c72a52e1b2d28643569", + "resources": { + "elapsed_seconds": 14.791, + "cpu_seconds": 2.572715, + "peak_process_family_bytes": 348119040, + "payload_exit_code": 0, + "payload_cleanup_verified": true + } + }, + { + "repetition": 2, + "mode": "client-code", + "arm": "all", + "status": "valid", + "usage": { + "input_tokens": 11121, + "cached_input_tokens": 9216, + "cache_write_input_tokens": 0, + "output_tokens": 8, + "reasoning_output_tokens": 0, + "uncached_input_tokens": 1905 + }, + "runtime_revision": "5038280f96af74e4e116ce78c4b3b77017cc69ac", + "runtime_fingerprint_sha256": "5dccf8e4e9a300a402153887ddd8f018e392be9b29630df9f3a30ea70265570c", + "runtime_bytes_unchanged": true, + "events_sha256": "e4ce2f31c53ae9fdcb7e917f7700add84f39dcee6f2d2c16e46f843bd95e2ed4", + "retained_context_sha256": "7f0a85262e1419356f46f7e1176dcbe20d8d1c3b108e3c89db4c46b0e8fbc976", + "resources": { + "elapsed_seconds": 13.49, + "cpu_seconds": 2.697786, + "peak_process_family_bytes": 349622272, + "payload_exit_code": 0, + "payload_cleanup_verified": true + } + }, + { + "repetition": 2, + "mode": "client-code", + "arm": "none", + "status": "valid", + "usage": { + "input_tokens": 11118, + "cached_input_tokens": 9216, + "cache_write_input_tokens": 0, + "output_tokens": 8, + "reasoning_output_tokens": 0, + "uncached_input_tokens": 1902 + }, + "runtime_revision": "5038280f96af74e4e116ce78c4b3b77017cc69ac", + "runtime_fingerprint_sha256": "5dccf8e4e9a300a402153887ddd8f018e392be9b29630df9f3a30ea70265570c", + "runtime_bytes_unchanged": true, + "events_sha256": "4c19474977e782bc5b92ab1bed354ff9d8b050cf3ba32b83fb5d206350289156", + "retained_context_sha256": "8b6fe2584c93435ef1a04bd632ff431253c660fae082625de7704afb90726eed", + "resources": { + "elapsed_seconds": 14.693, + "cpu_seconds": 2.038174, + "peak_process_family_bytes": 191598592, + "payload_exit_code": 0, + "payload_cleanup_verified": true + } + }, + { + "repetition": 2, + "mode": "direct", + "arm": "selected", + "status": "invalid", + "usage": { + "input_tokens": 11115, + "cached_input_tokens": 9216, + "cache_write_input_tokens": 0, + "output_tokens": 8, + "reasoning_output_tokens": 0, + "uncached_input_tokens": 1899 + }, + "runtime_revision": "5038280f96af74e4e116ce78c4b3b77017cc69ac", + "runtime_fingerprint_sha256": "5dccf8e4e9a300a402153887ddd8f018e392be9b29630df9f3a30ea70265570c", + "runtime_bytes_unchanged": true, + "events_sha256": "9ed44dde66c7c61ec3e527a284e78455ee2bbec21af02ca8c09c3119ef0d6b7a", + "retained_context_sha256": "2cb4970caf1be698a1291aa4044284911c363181d78b85fe076098e5eadca732", + "resources": { + "elapsed_seconds": 12.78, + "cpu_seconds": 2.63344, + "peak_process_family_bytes": 376070144, + "payload_exit_code": 0, + "payload_cleanup_verified": true + }, + "client_errors": [ + "Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`." + ] + }, + { + "repetition": 2, + "mode": "direct", + "arm": "all", + "status": "invalid", + "usage": { + "input_tokens": 11115, + "cached_input_tokens": 9216, + "cache_write_input_tokens": 0, + "output_tokens": 8, + "reasoning_output_tokens": 0, + "uncached_input_tokens": 1899 + }, + "runtime_revision": "5038280f96af74e4e116ce78c4b3b77017cc69ac", + "runtime_fingerprint_sha256": "5dccf8e4e9a300a402153887ddd8f018e392be9b29630df9f3a30ea70265570c", + "runtime_bytes_unchanged": true, + "events_sha256": "112d7f79df74670ca5c03e48ed494bd24e0c19c74c774de745e1b5792dbd1bb5", + "retained_context_sha256": "b3a208e08fa99d383fc8101218d1046da89f12b24fcb3f12669bda0ec16e603c", + "resources": { + "elapsed_seconds": 13.99, + "cpu_seconds": 2.834572, + "peak_process_family_bytes": 390422528, + "payload_exit_code": 0, + "payload_cleanup_verified": true + }, + "client_errors": [ + "Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`." + ] + }, + { + "repetition": 2, + "mode": "direct", + "arm": "none", + "status": "invalid", + "usage": { + "input_tokens": 11118, + "cached_input_tokens": 9216, + "cache_write_input_tokens": 0, + "output_tokens": 8, + "reasoning_output_tokens": 0, + "uncached_input_tokens": 1902 + }, + "runtime_revision": "5038280f96af74e4e116ce78c4b3b77017cc69ac", + "runtime_fingerprint_sha256": "5dccf8e4e9a300a402153887ddd8f018e392be9b29630df9f3a30ea70265570c", + "runtime_bytes_unchanged": true, + "events_sha256": "46261f1778ab3d1963d19b6bf499953ba22bfa17aa9c9e51bd43483948445ed7", + "retained_context_sha256": "fea4491dcf29afa2502cd0583c0b05361b50c69693a3412a9f5a6ae3a3f422b9", + "resources": { + "elapsed_seconds": 14.092, + "cpu_seconds": 1.839927, + "peak_process_family_bytes": 200450048, + "payload_exit_code": 0, + "payload_cleanup_verified": true + }, + "client_errors": [ + "Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`." + ] + }, + { + "repetition": 3, + "mode": "direct", + "arm": "none", + "status": "invalid", + "usage": { + "input_tokens": 11121, + "cached_input_tokens": 9216, + "cache_write_input_tokens": 0, + "output_tokens": 8, + "reasoning_output_tokens": 0, + "uncached_input_tokens": 1905 + }, + "runtime_revision": "5038280f96af74e4e116ce78c4b3b77017cc69ac", + "runtime_fingerprint_sha256": "5dccf8e4e9a300a402153887ddd8f018e392be9b29630df9f3a30ea70265570c", + "runtime_bytes_unchanged": true, + "events_sha256": "21c54ea3b9dd49ba9534c837b2c016c3a7e397e29c3f990287b5960a9217901d", + "retained_context_sha256": "f2bd1356c7bf0b6ac3547e73fbf340686c89149755fc1c216d64a7f712c3d875", + "resources": { + "elapsed_seconds": 13.999, + "cpu_seconds": 1.902008, + "peak_process_family_bytes": 205639680, + "payload_exit_code": 0, + "payload_cleanup_verified": true + }, + "client_errors": [ + "Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`." + ] + }, + { + "repetition": 3, + "mode": "direct", + "arm": "all", + "status": "invalid", + "usage": { + "input_tokens": 11121, + "cached_input_tokens": 9216, + "cache_write_input_tokens": 0, + "output_tokens": 8, + "reasoning_output_tokens": 0, + "uncached_input_tokens": 1905 + }, + "runtime_revision": "5038280f96af74e4e116ce78c4b3b77017cc69ac", + "runtime_fingerprint_sha256": "5dccf8e4e9a300a402153887ddd8f018e392be9b29630df9f3a30ea70265570c", + "runtime_bytes_unchanged": true, + "events_sha256": "fcd5f7fe6cf094933ee7c957d4e18d84a596174fdcf7655d6e0ea0f524399a57", + "retained_context_sha256": "019d568d86a324f99ce0c1512240e56933055fc6f898e993de033195e8205105", + "resources": { + "elapsed_seconds": 15.395, + "cpu_seconds": 2.840376, + "peak_process_family_bytes": 362033152, + "payload_exit_code": 0, + "payload_cleanup_verified": true + }, + "client_errors": [ + "Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`." + ] + }, + { + "repetition": 3, + "mode": "direct", + "arm": "selected", + "status": "invalid", + "usage": { + "input_tokens": 11118, + "cached_input_tokens": 9216, + "cache_write_input_tokens": 0, + "output_tokens": 8, + "reasoning_output_tokens": 0, + "uncached_input_tokens": 1902 + }, + "runtime_revision": "5038280f96af74e4e116ce78c4b3b77017cc69ac", + "runtime_fingerprint_sha256": "5dccf8e4e9a300a402153887ddd8f018e392be9b29630df9f3a30ea70265570c", + "runtime_bytes_unchanged": true, + "events_sha256": "8e56b7f5f7f3f6c788092bedc9fd922bc5bc2b05f899d7b17a331e340aaeb1e1", + "retained_context_sha256": "8f6a6ac7f2e45b447fb026760679366a9103ccfc5b61fd634739380052468909", + "resources": { + "elapsed_seconds": 13.979, + "cpu_seconds": 2.374572, + "peak_process_family_bytes": 373174272, + "payload_exit_code": 0, + "payload_cleanup_verified": true + }, + "client_errors": [ + "Code Mode is unavailable because code-mode host is disabled. Code mode will fail closed; enable `features.code_mode_host` and install `codex-code-mode-host`." + ] + }, + { + "repetition": 3, + "mode": "client-code", + "arm": "none", + "status": "valid", + "usage": { + "input_tokens": 11118, + "cached_input_tokens": 9216, + "cache_write_input_tokens": 0, + "output_tokens": 8, + "reasoning_output_tokens": 0, + "uncached_input_tokens": 1902 + }, + "runtime_revision": "5038280f96af74e4e116ce78c4b3b77017cc69ac", + "runtime_fingerprint_sha256": "5dccf8e4e9a300a402153887ddd8f018e392be9b29630df9f3a30ea70265570c", + "runtime_bytes_unchanged": true, + "events_sha256": "2f73790f6c166d7ef0f431b154b5db6a6847df7008e606537af83e487cfdf9ed", + "retained_context_sha256": "e2db00371547ec85cee1c6ac7c1de31151c509bea65a23631bf7ffc00f9304d5", + "resources": { + "elapsed_seconds": 13.384, + "cpu_seconds": 1.96487, + "peak_process_family_bytes": 209338368, + "payload_exit_code": 0, + "payload_cleanup_verified": true + } + }, + { + "repetition": 3, + "mode": "client-code", + "arm": "all", + "status": "valid", + "usage": { + "input_tokens": 11118, + "cached_input_tokens": 9216, + "cache_write_input_tokens": 0, + "output_tokens": 8, + "reasoning_output_tokens": 0, + "uncached_input_tokens": 1902 + }, + "runtime_revision": "5038280f96af74e4e116ce78c4b3b77017cc69ac", + "runtime_fingerprint_sha256": "5dccf8e4e9a300a402153887ddd8f018e392be9b29630df9f3a30ea70265570c", + "runtime_bytes_unchanged": true, + "events_sha256": "de27cfa4fed633ee4834f7408a9d72530997630e0651655853c578364049def0", + "retained_context_sha256": "609867297100f977eea52ecee270beabbf628794a5eaa6c532593e871e1221f2", + "resources": { + "elapsed_seconds": 13.198, + "cpu_seconds": 2.631436, + "peak_process_family_bytes": 401825792, + "payload_exit_code": 0, + "payload_cleanup_verified": true + } + }, + { + "repetition": 3, + "mode": "client-code", + "arm": "selected", + "status": "valid", + "usage": { + "input_tokens": 11115, + "cached_input_tokens": 9216, + "cache_write_input_tokens": 0, + "output_tokens": 8, + "reasoning_output_tokens": 0, + "uncached_input_tokens": 1899 + }, + "runtime_revision": "5038280f96af74e4e116ce78c4b3b77017cc69ac", + "runtime_fingerprint_sha256": "5dccf8e4e9a300a402153887ddd8f018e392be9b29630df9f3a30ea70265570c", + "runtime_bytes_unchanged": true, + "events_sha256": "89a066866cf1275c2685d41866e322788bed7d3a0aa29f0ac4428c262a52bb83", + "retained_context_sha256": "24d8f0f6735ba70221b75c01c78f91f85ba2e6546d6639f5c56b3ec546920542", + "resources": { + "elapsed_seconds": 13.396, + "cpu_seconds": 2.490836, + "peak_process_family_bytes": 381198336, + "payload_exit_code": 0, + "payload_cleanup_verified": true + } + } + ], + "load_comparison": { + "pairs": [ + { + "mode": "direct", + "arm": "all", + "repetition": 1, + "status": "invalid", + "input_token_delta": null + }, + { + "mode": "direct", + "arm": "selected", + "repetition": 1, + "status": "invalid", + "input_token_delta": null + }, + { + "mode": "client-code", + "arm": "all", + "repetition": 1, + "status": "valid", + "input_token_delta": 3 + }, + { + "mode": "client-code", + "arm": "selected", + "repetition": 1, + "status": "valid", + "input_token_delta": 3 + }, + { + "mode": "client-code", + "arm": "selected", + "repetition": 2, + "status": "valid", + "input_token_delta": 3 + }, + { + "mode": "client-code", + "arm": "all", + "repetition": 2, + "status": "valid", + "input_token_delta": 3 + }, + { + "mode": "direct", + "arm": "selected", + "repetition": 2, + "status": "invalid", + "input_token_delta": null + }, + { + "mode": "direct", + "arm": "all", + "repetition": 2, + "status": "invalid", + "input_token_delta": null + }, + { + "mode": "direct", + "arm": "all", + "repetition": 3, + "status": "invalid", + "input_token_delta": null + }, + { + "mode": "direct", + "arm": "selected", + "repetition": 3, + "status": "invalid", + "input_token_delta": null + }, + { + "mode": "client-code", + "arm": "all", + "repetition": 3, + "status": "valid", + "input_token_delta": 0 + }, + { + "mode": "client-code", + "arm": "selected", + "repetition": 3, + "status": "valid", + "input_token_delta": -3 + } + ], + "groups": [ + { + "mode": "client-code", + "arm": "all", + "valid_pairs": 3, + "attempted_pairs": 3, + "median_input_token_delta": 3, + "min_input_token_delta": 0, + "max_input_token_delta": 3 + }, + { + "mode": "client-code", + "arm": "selected", + "valid_pairs": 3, + "attempted_pairs": 3, + "median_input_token_delta": 3, + "min_input_token_delta": -3, + "max_input_token_delta": 3 + }, + { + "mode": "direct", + "arm": "all", + "valid_pairs": 0, + "attempted_pairs": 3, + "median_input_token_delta": null, + "min_input_token_delta": null, + "max_input_token_delta": null + }, + { + "mode": "direct", + "arm": "selected", + "valid_pairs": 0, + "attempted_pairs": 3, + "median_input_token_delta": null, + "min_input_token_delta": null, + "max_input_token_delta": null + } + ] + }, + "default_smoke": { + "runtime_revision": "61a364210d217bc4cac82ac6e7540f44e73f6100", + "statuses": ["valid", "valid", "valid"], + "report_sha256": "e7cbd97a735724d4ff24106ef469deb0022b92ff6dfa6bab54d06fb2777467c4" + }, + "task_oracle": { + "path": "/home/yann/yann/rea-mcp-catalog-eval/node_modules/semver/package.json", + "sha256": "7c94cb7f2a53c27b20d76386ec144c062894dbcc909cfabd0f728c37874b1776", + "expected_fields": { + "package_name": "semver", + "package_version": "7.8.5", + "entry_file": "index.js" + }, + "npm_resolved": "https://registry.npmjs.org/semver/-/semver-7.8.5.tgz", + "npm_integrity": "sha512-Y7/KDsb8LjooZpwaqGyulO6DQlksgCncchHGk+sZIY4SBvUocMBEFH5Ur1fI4dV+Jvl0w6cjvucaIi40puRioA==" + }, + "task_prompt": "Use REA to statically determine the package name, version, and declared entry file of the local npm package at /home/yann/yann/rea-mcp-catalog-eval/node_modules/semver from package.json only. Select only package.json for source analysis; leave dependency modules unexamined. Return a concise JSON object with package_name, package_version, entry_file, evidence_ids, and limitations. Cite actual produced Evidence IDs. State that runtime behavior was not observed. Stop when these three metadata facts are supported.\n", + "task_prompt_sha256": "d95b8400eee2dc8442dfbc5336704a12bd5bb0aca372e73ecdd9b6687e05edb8", + "task_trials": [ + { + "arm": "all", + "mode": "client-code", + "runtime_revision": "61a364210d217bc4cac82ac6e7540f44e73f6100", + "runtime_fingerprint_sha256": "5dccf8e4e9a300a402153887ddd8f018e392be9b29630df9f3a30ea70265570c", + "settings_sources_sha256": { + "agent-trial.py": "3776556d2a9b209c211022349c363120c91731a38cc68e02d84f779997958387", + "codex_context.py": "79c5ec253c836f67d05e4da19f80283c1d40c292dcbb78f3f984095031bcc52c", + "provider-environment.json": "59103e26d51bca9f82f18b43c5544de4522866ff79d9cfe718cc91cbab13d7be" + }, + "usage": { + "input_tokens": 271084, + "cached_input_tokens": 228224, + "cache_write_input_tokens": 0, + "output_tokens": 699, + "reasoning_output_tokens": 15 + }, + "per_response_usage": [ + { + "input_tokens": 11349, + "cached_input_tokens": 9216, + "cache_write_input_tokens": 0, + "output_tokens": 65, + "reasoning_output_tokens": 0, + "total_tokens": 11414 + }, + { + "input_tokens": 20481, + "cached_input_tokens": 11264, + "cache_write_input_tokens": 0, + "output_tokens": 75, + "reasoning_output_tokens": 0, + "total_tokens": 20556 + }, + { + "input_tokens": 30845, + "cached_input_tokens": 20352, + "cache_write_input_tokens": 0, + "output_tokens": 73, + "reasoning_output_tokens": 0, + "total_tokens": 30918 + }, + { + "input_tokens": 31409, + "cached_input_tokens": 30720, + "cache_write_input_tokens": 0, + "output_tokens": 69, + "reasoning_output_tokens": 0, + "total_tokens": 31478 + }, + { + "input_tokens": 36632, + "cached_input_tokens": 31360, + "cache_write_input_tokens": 0, + "output_tokens": 102, + "reasoning_output_tokens": 0, + "total_tokens": 36734 + }, + { + "input_tokens": 37614, + "cached_input_tokens": 36608, + "cache_write_input_tokens": 0, + "output_tokens": 68, + "reasoning_output_tokens": 0, + "total_tokens": 37682 + }, + { + "input_tokens": 51227, + "cached_input_tokens": 37504, + "cache_write_input_tokens": 0, + "output_tokens": 109, + "reasoning_output_tokens": 15, + "total_tokens": 51336 + }, + { + "input_tokens": 51527, + "cached_input_tokens": 51200, + "cache_write_input_tokens": 0, + "output_tokens": 138, + "reasoning_output_tokens": 0, + "total_tokens": 51665 + } + ], + "mcp_calls": [ + { + "server": "rea", + "tool": "analyze_javascript_application", + "arguments": { + "input_path": "/home/yann/yann/rea-mcp-catalog-eval/node_modules/semver", + "source_paths": ["package.json"], + "detail": "summary" + }, + "status": "completed", + "error": null + }, + { + "server": "rea", + "tool": "get_evidence_bundle", + "arguments": { + "detail": "records", + "filters": { + "evidence_id": "ev_77708c6f32646b879ed36c30f67ac2c592ce0ad4b78b284ad213d4e9a0eb98ef" + } + }, + "status": "completed", + "error": null + }, + { + "server": "rea", + "tool": "close_binary", + "arguments": {}, + "status": "completed", + "error": null + } + ], + "answer": { + "package_name": "semver", + "package_version": "7.8.5", + "entry_file": "index.js", + "evidence_ids": [ + "ev_77708c6f32646b879ed36c30f67ac2c592ce0ad4b78b284ad213d4e9a0eb98ef" + ], + "limitations": [ + "Metadata derived only from package.json; entry_file is the declared main value.", + "Only package.json was selected for source analysis; dependency modules remained unexamined.", + "Runtime behavior was not observed." + ] + }, + "factual_correctness": "passed for the three declared package.json fields only", + "produced_evidence_reference_presence": true, + "runtime_bytes_unchanged": true, + "events_sha256": "229de2788bd0503689d7cb8f6276ed84efe0a8cc168233e17a0ad3a0efccbed9", + "retained_context_sha256": "b1da36b3269bacad55e3c534b1ff8f9d3e8f05dafda7b28d1e854a6944fe7727", + "resources": { + "elapsed_seconds": 48.952, + "cpu_seconds": 6.531794, + "peak_process_family_bytes": 704335872, + "payload_exit_code": 0, + "payload_cleanup_verified": true + } + }, + { + "arm": "selected", + "mode": "client-code", + "runtime_revision": "61a364210d217bc4cac82ac6e7540f44e73f6100", + "runtime_fingerprint_sha256": "5dccf8e4e9a300a402153887ddd8f018e392be9b29630df9f3a30ea70265570c", + "settings_sources_sha256": { + "agent-trial.py": "3776556d2a9b209c211022349c363120c91731a38cc68e02d84f779997958387", + "codex_context.py": "79c5ec253c836f67d05e4da19f80283c1d40c292dcbb78f3f984095031bcc52c", + "provider-environment.json": "59103e26d51bca9f82f18b43c5544de4522866ff79d9cfe718cc91cbab13d7be" + }, + "usage": { + "input_tokens": 160438, + "cached_input_tokens": 138496, + "cache_write_input_tokens": 0, + "output_tokens": 637, + "reasoning_output_tokens": 0 + }, + "per_response_usage": [ + { + "input_tokens": 11349, + "cached_input_tokens": 9216, + "cache_write_input_tokens": 0, + "output_tokens": 65, + "reasoning_output_tokens": 0, + "total_tokens": 11414 + }, + { + "input_tokens": 20139, + "cached_input_tokens": 11264, + "cache_write_input_tokens": 0, + "output_tokens": 47, + "reasoning_output_tokens": 0, + "total_tokens": 20186 + }, + { + "input_tokens": 20304, + "cached_input_tokens": 19968, + "cache_write_input_tokens": 0, + "output_tokens": 34, + "reasoning_output_tokens": 0, + "total_tokens": 20338 + }, + { + "input_tokens": 21617, + "cached_input_tokens": 20096, + "cache_write_input_tokens": 0, + "output_tokens": 69, + "reasoning_output_tokens": 0, + "total_tokens": 21686 + }, + { + "input_tokens": 26840, + "cached_input_tokens": 21504, + "cache_write_input_tokens": 0, + "output_tokens": 127, + "reasoning_output_tokens": 0, + "total_tokens": 26967 + }, + { + "input_tokens": 29702, + "cached_input_tokens": 26752, + "cache_write_input_tokens": 0, + "output_tokens": 156, + "reasoning_output_tokens": 0, + "total_tokens": 29858 + }, + { + "input_tokens": 30487, + "cached_input_tokens": 29696, + "cache_write_input_tokens": 0, + "output_tokens": 139, + "reasoning_output_tokens": 0, + "total_tokens": 30626 + } + ], + "mcp_calls": [ + { + "server": "rea", + "tool": "analyze_javascript_application", + "arguments": { + "input_path": "/home/yann/yann/rea-mcp-catalog-eval/node_modules/semver", + "source_paths": ["package.json"], + "detail": "summary" + }, + "status": "completed", + "error": null + }, + { + "server": "rea", + "tool": "inspect_analysis_view", + "arguments": { + "source": { + "kind": "retained-evidence", + "evidence_id": "ev_77708c6f32646b879ed36c30f67ac2c592ce0ad4b78b284ad213d4e9a0eb98ef" + }, + "view": { + "kind": "javascript-facts", + "where": { + "paths": ["package.json"] + }, + "include_available_property_keys": true, + "property_keys": [], + "offset": 0, + "limit": 20 + } + }, + "status": "completed", + "error": null + }, + { + "server": "rea", + "tool": "inspect_analysis_view", + "arguments": { + "source": { + "kind": "retained-evidence", + "evidence_id": "ev_77708c6f32646b879ed36c30f67ac2c592ce0ad4b78b284ad213d4e9a0eb98ef" + }, + "view": { + "kind": "javascript-facts", + "where": { + "paths": ["package.json"], + "kinds": ["package"] + }, + "property_keys": [ + "name", + "version", + "main", + "main_status", + "parse_status" + ], + "offset": 0, + "limit": 5 + } + }, + "status": "completed", + "error": null + } + ], + "answer": { + "package_name": "semver", + "package_version": "7.8.5", + "entry_file": "index.js", + "evidence_ids": [ + "ev_3c19f3e54f80c0d9a9ae1479f2ea30f9da57f31c390e94a7019db5d22900d60a" + ], + "limitations": [ + "Only package.json was selected for source analysis; dependency modules and 49 other inventoried files remained unexamined.", + "entry_file reflects the declared main field.", + "Runtime behavior was not observed." + ] + }, + "factual_correctness": "passed for the three declared package.json fields only", + "produced_evidence_reference_presence": true, + "runtime_bytes_unchanged": true, + "events_sha256": "1b2cb64ba0f903d127ac0bf5d2b0fce923b015c036c957a065db0347f308aed9", + "retained_context_sha256": "cbbf91f27127190f146357e988c86940a2837d3d789879bec133204a8b0adf31", + "resources": { + "elapsed_seconds": 42.928, + "cpu_seconds": 5.727509, + "peak_process_family_bytes": 610402304, + "payload_exit_code": 0, + "payload_cleanup_verified": true + } + } + ], + "excluded_cohorts": [ + { + "path": "matrix-1", + "reason": "Repository-nested workspace inherited project instructions/skills; aborted and invalidated." + }, + { + "path": "isolated-1; final; final-task-*", + "reason": "Legacy flags did not establish distinct execution modes; exploratory traces excluded from mode claims." + }, + { + "path": "package-*", + "reason": "Concurrent artifact rebuild interfered with runtime availability; whole cohort excluded, including two failed MCP handshakes." + } + ], + "limitations": [ + "Only one Codex CLI/model/backend; no qualified eager/native or other-client comparison.", + "Nine host-disabled diagnostic trials are invalid despite successful marker replies.", + "One real small-package task per arm; no general quality, latency, or token advantage established.", + "Whole-turn usage includes discovery/continuations; per-response usage is not added a second time.", + "Raw submitted request bodies and independent billing reconciliation were not captured.", + "Remote inference resources, dependency acquisition/build cost and full dependency-tree byte identity are unmeasured.", + "Kernel peaks include observer/file cache at payload exit, not final scope counters." + ], + "resource_bounds_each_trial": { + "hard_memory_bytes": 4294967296, + "early_stop_bytes": 3221225472, + "cpu_quota_percent": 100, + "cpu_affinity": [63], + "task_ceiling": 192 + }, + "resource_events_each_trial": { + "memory.events": { + "low": 0, + "high": 0, + "max": 0, + "oom": 0, + "oom_kill": 0, + "oom_group_kill": 0 + }, + "pids.events": { + "max": 0 + } + } +} diff --git a/docs/verification/mcp-catalog-codex-2026-11-05.md b/docs/verification/mcp-catalog-codex-2026-11-05.md new file mode 100644 index 0000000000..f39f683b9a --- /dev/null +++ b/docs/verification/mcp-catalog-codex-2026-11-05.md @@ -0,0 +1,111 @@ +# Codex MCP catalog observations, 2026-11-05 + +The supported Codex host path showed no material input-token increase for an +empty response with REA enabled. Completing a real package question still used +hundreds of thousands of cumulative input tokens across discovery and +continuations. Neither raw advertisement bytes nor an empty response measures +that whole-task cost. + +REA source baseline: `ec3410998b70290847f930538e57d0af0ac1bffb`. Loading driver: +`5038280f9`; completed-task/default-smoke driver: `61a364210`. The +[machine-readable receipts](mcp-catalog-codex-2026-11-05.json) retain exact usage, +source/runtime hashes, response counts, resource counters, answers, and invalid +profiles. Full local transcripts remain under the recorded `raw_artifact_root`; +they are not committed. See the [evaluation procedure](../mcp-catalog-evaluation.md) +to reproduce with a built source checkout. + +Linux x64; Node 24.22.1/npm 11.16.0; Codex CLI 0.165.0; requested model +`gpt-6.1-sol`, reasoning `low`. The resolved backend model version is unknown. +Each guarded trial used a fresh client/server/profile, one CPU, 4 GiB hard memory, +3 GiB early stop, no swap, and at most 192 tasks. OS/API prefix caches could be +warm. Reported input includes cached input. All valid payloads exited normally +and verified cleanup; no resource stops or OOM events were observed. + +## Advertisement and empty-response costs + +The built default catalog contains 234 tools and 4,029,995 JSON bytes, including +3,681,386 output-schema JSON bytes, 138,747 input-schema bytes, and 148,355 +description bytes. This is server advertisement size, not submitted model input. +Production `mcp doctor` independently observed all 234 tools and six prompts, +matched catalog identity, and completed `binary_session` successfully. + +Three repetitions per profile, with alternating order; default compact input +schemas and default full output schemas. The client-selected profile exposes +only `binary_session` after discovering the same server catalog. + +| Host-enabled profile | Valid samples | Paired input delta, median (range) | Median guarded wall s | Median family CPU s | Median family peak MiB | +| -------------------- | ------------: | ---------------------------------: | --------------------: | ------------------: | ---------------------: | +| No MCP | 3 | control | 14.693 | 1.965 | 187.9 | +| All tools | 3 | +3 (0 to +3) | 13.490 | 2.685 | 347.2 | +| Selected tool | 3 | +3 (-3 to +3) | 13.792 | 2.573 | 336.1 | + +Every valid response returned exactly `CATALOG_OK` without tool calls. Input was +11,112–11,121 tokens, including 9,216 cached input tokens, with eight output +tokens. Random workspace paths and client framing remain in the comparison. +These small differences do not establish a schema-only token count. Network +waits dominate elapsed time; no stable latency improvement is established. +The local CPU/memory increase when enabling MCP remains measurable in this setup. +Family peaks are kernel cgroup counters including observer/file cache at payload +exit, rather than final whole-scope counters. + +Nine additional host-disabled diagnostic samples emitted: + +> Code Mode is unavailable because code-mode host is disabled. + +The client also reported that tool dispatch would fail closed. It still returned +the requested marker, illustrating why that marker alone is insufficient. +All nine profiles are **invalid**; their paired deltas remain unknown. This +diagnostic matrix exits nonzero intentionally. It does not establish an eager +or native-dispatch comparison. The final command defaults to the working host +profile; a separate three-cell default smoke run passed. + +## Completed real-package pilot + +Both arms answered the same question from the installed public `semver@7.8.5` +package, analyzing only `package.json`: declared name `semver`, version `7.8.5`, +and entry `index.js`. Those three fields were checked against the source JSON +independently of REA. Answers cited Evidence references emitted by the actual +tools and stated that dependency modules were unexamined and runtime behavior +was not observed. Both runs retained the same analysis parent Evidence identity. + +The selected arm allowed `analyze_javascript_application` and +`inspect_analysis_view`; the other allowed the whole catalog. Both used the +working host, the identical question/instructions, and unchanged target bytes. + +| Tool selection | Whole-turn input (cached subset) | Output | Model responses / MCP calls | Guarded wall / family CPU s | Family peak MiB | +| -------------- | -------------------------------: | -----: | --------------------------: | --------------------------: | --------------: | +| All | 271,084 (228,224) | 699 | 8 / 3 | 48.952 / 6.532 | 671.7 | +| Selected | 160,438 (138,496) | 637 | 7 / 3 | 42.928 / 5.728 | 582.1 | + +The first model input was 11,349 tokens in both cases. The subsequent +`exec`/`ALL_TOOLS` discovery, schema reads, replies and context replay account for +the later cumulative input. Per-response input counts sum exactly to the reported +whole-turn input; they must not be added to it again. Counting only the three MCP +calls would miss several model responses. + +This is one task per arm. The model chose different follow-up tools and +projections. The numbers demonstrate a completed-task measurement method; they +do not isolate a causal catalog reduction or establish a general advantage for +filtered tools, complex applications, or other clients/providers. + +## Verification and exclusions + +The new runner separates execution workspaces from artifact output, holds the +driver's artifact lock, pins one matrix runtime and checks freshness across +trials and at completion. Its +named usage parser preserves missing counts as unknown and rejects failed client +profiles, wrong answers, unexpected actions, runtime drift and unverified cleanup. +All 18 retained control transcripts were reassessed after the reporting fixes: +verdicts and token comparisons stayed identical, and failed profiles retained +their available usage, cleanup and resource observations. + +Exploratory cohorts remain recorded separately: repository-nested workspaces +inherited project instructions/skills; a concurrent artifact rebuild interrupted +two MCP handshakes; and old `code_mode`/`code_mode_only` switches did not establish +distinct dispatch modes. Those cohorts do not support the tables above. The +sibling JavaScript-facts harness already explicitly enables the required host. + +Dependency acquisition/build costs, complete dependency-tree byte identity, +remote inference CPU/memory, exact submitted request bodies and independent +billing reconciliation are unmeasured. No REA production contracts, Evidence +semantics, or catalog defaults changed in this work. diff --git a/package.json b/package.json index 06654f2d42..6635d5ef71 100644 --- a/package.json +++ b/package.json @@ -154,6 +154,7 @@ "test:with-build": "npm run test:prepare && vitest run", "test:run": "vitest run", "test:python": "python3 scripts/run-python-tests.py", + "eval:mcp-catalog": "node scripts/run-exclusive.mjs artifacts python3 tests/support/analysis-efficiency/catalog-eval.py", "test:local": "node scripts/run-development-tests.mjs local", "test:fast": "node scripts/ci/test-group.mjs source", "test:built": "node scripts/ci/test-group.mjs built", diff --git a/tests/support/analysis-efficiency/README.md b/tests/support/analysis-efficiency/README.md index fd504226d4..1970ab7074 100644 --- a/tests/support/analysis-efficiency/README.md +++ b/tests/support/analysis-efficiency/README.md @@ -1,5 +1,10 @@ # Analysis verification test support +The maintained `catalog-eval.py` entrypoint (`npm run eval:mcp-catalog`) uses the +real local Codex CLI and the retained resource/agent runners. See +[MCP catalog evaluation](../../../docs/mcp-catalog-evaluation.md) for its matrix, +observed-token accounting, isolation and completed-task limits. + These files support the existing comparison, independent-oracle, receipt and cleanup regressions. The TypeScript consumers live in `tests/boundary/`, `tests/evaluation/` and `tests/conformance/`; `npm run test:python` discovers @@ -17,7 +22,7 @@ belong to documentation. Their numerical sections are produced and checked by: node tests/support/analysis-efficiency/mcp-integer-report.mjs --check ``` -The full experiment suite was retired. For historical reproduction, use the +The former broad experiment suite was retired. For historical reproduction, use the recorded runtime and producer revisions; the [archived source tree](https://github.com/morluto/rea/tree/a2e2c29b59e8775b0e6aedb09a9eb16adf32646b/experiments/analysis-efficiency) preserves the former layout. Current deterministic checks do not requalify those historical measurements. diff --git a/tests/support/analysis-efficiency/agent-trial.py b/tests/support/analysis-efficiency/agent-trial.py index d040d636af..f5db80f9b9 100644 --- a/tests/support/analysis-efficiency/agent-trial.py +++ b/tests/support/analysis-efficiency/agent-trial.py @@ -12,32 +12,13 @@ import tempfile import time from codex_context import capture_tool_context - - -def file_digest(path): - digest = hashlib.sha256() - with path.open('rb') as data: - for chunk in iter(lambda: data.read(1024 * 1024), b''): - digest.update(chunk) - return digest.hexdigest() - - -def runtime_fingerprint(root): - files = [] - for name in ['dist', 'bridge']: - for path in sorted((root / name).rglob('*')): - if path.is_symlink(): - raise ValueError('Runtime fingerprint does not follow symlinks') - if path.is_file(): - files.append({'path': path.relative_to(root).as_posix(), 'sha256': file_digest(path)}) - files.sort(key=lambda row: row['path']) - text = json.dumps(files, separators=(',', ':'), ensure_ascii=False) - return files, hashlib.sha256(text.encode()).hexdigest() +from runtime_identity import file_digest, runtime_fingerprint def main(): parser = argparse.ArgumentParser() - parser.add_argument('--mode', choices=['direct', 'client-code'], default='direct') + parser.add_argument('--mode', choices=['direct', 'client-code'], default='client-code', + help='client-code enables code_mode_host; direct is a host-disabled diagnostic') parser.add_argument('--runtime-root', type=Path, required=True) parser.add_argument('--revision', required=True) parser.add_argument('--prompt', type=Path, required=True) @@ -49,12 +30,22 @@ def main(): parser.add_argument('--codex', default='/usr/bin/codex') parser.add_argument('--providers', type=Path) parser.add_argument('--enabled-tools', type=Path) + parser.add_argument('--catalog-load', action='store_true', + help='Measure a no-tool response with identical instructions across catalogs') + parser.add_argument('--no-mcp', action='store_true', + help='No-server control; requires --catalog-load') + parser.add_argument('--profile-parent', type=Path, + help='External disposable-profile parent (default: cache beside the source Codex home)') parser.add_argument('--authority', choices=['static', 'browser'], default='static') parser.add_argument('--capture-tool-context', action='store_true', help='Retain owned tool/usage items; excludes account/ambient state') parser.add_argument('--guide-enabled-tools', action='store_true', help='Opt-in exact-name discovery guidance for selected REA tools') args = parser.parse_args() + if args.no_mcp and not args.catalog_load: + parser.error('--no-mcp requires --catalog-load') + if args.catalog_load and (args.guide_enabled_tools or args.artifact_path): + parser.error('--catalog-load excludes tool guidance and artifact tasks') if args.guide_enabled_tools and args.enabled_tools is None: parser.error('--guide-enabled-tools requires --enabled-tools') enabled_tools = None @@ -101,7 +92,13 @@ def main(): args.output.mkdir(parents=True, exist_ok=False, mode=0o700) runtime_files, runtime_digest = runtime_fingerprint(repository) (args.output / 'runtime-files.json').write_text(json.dumps(runtime_files)) - profile = Path(tempfile.mkdtemp(prefix='profile-', dir=args.output)) + # Output artifacts may live inside a checkout. Execution must not: Codex + # discovers ancestor AGENTS.md and skills independently of its user config. + profile_parent = (args.profile_parent or source_auth_home.parent / '.cache' / 'rea-agent-trials').resolve() + if any(profile_parent.is_relative_to(root.resolve()) for root in [repository, driver_repository]): + raise ValueError('--profile-parent must be outside the runtime and driver checkouts') + profile_parent.mkdir(parents=True, exist_ok=True, mode=0o700) + profile = Path(tempfile.mkdtemp(prefix='profile-', dir=profile_parent)) trial_codex_home = profile / 'codex' account = profile / 'account' workspace = profile / 'workspace' @@ -156,6 +153,11 @@ def main(): 'return only their metadata. Preserve the original analyst question, ' 'dependency order and source Evidence.' ) + if args.catalog_load: + instruction = ( + 'This is a catalog loading measurement. Do not call any tools, search, ' + 'or delegate. Reply with exactly the text requested by the user.' + ) quote = json.dumps config = '\n'.join([ 'approval_policy = "never"', 'sandbox_mode = "read-only"', @@ -164,20 +166,21 @@ def main(): 'developer_instructions = ' + quote(instruction), '[features]', 'multi_agent = false', 'shell_snapshot = false', 'shell_tool = false', 'unified_exec = false', - 'code_mode = ' + ('true' if args.mode == 'client-code' else 'false'), - 'code_mode_only = ' + ('true' if args.mode == 'client-code' else 'false'), + 'code_mode_host = ' + ('true' if args.mode == 'client-code' else 'false'), '[history]', 'persistence = "none"', - '[mcp_servers.rea]', 'command = ' + quote(node), - 'args = ' + quote(command_args), 'startup_timeout_sec = 60', - 'tool_timeout_sec = 600', - *([] if enabled_tools is None else ['enabled_tools = ' + quote(enabled_tools)]), - '[mcp_servers.rea.env]', - *[key + ' = ' + quote(value) for key, value in provider_env.items()], ]) + '\n' + if not args.no_mcp: + config += '\n'.join([ + '[mcp_servers.rea]', 'required = true', 'command = ' + quote(node), + 'args = ' + quote(command_args), 'startup_timeout_sec = 60', + 'tool_timeout_sec = 600', + *([] if enabled_tools is None else ['enabled_tools = ' + quote(enabled_tools)]), + '[mcp_servers.rea.env]', + *[key + ' = ' + quote(value) for key, value in provider_env.items()], + ]) + '\n' # MCP read-only hints cover normal static tools. Explicitly authorize only # the session-opening primitive needed by the requested native tasks. - config += '[mcp_servers.rea.tools.open_binary]\napproval_mode = "approve"\n' - for tool in ['analyze_javascript_application', 'inspect_analysis_view', + approved_tools = ['open_binary', 'analyze_javascript_application', 'inspect_analysis_view', 'inspect_android_package', 'search_android_classes', 'inspect_android_class', 'inspect_android_method', 'inspect_firmware_regions', 'inspect_managed_artifact', @@ -199,8 +202,10 @@ def main(): 'import_evidence_bundle', 'list_unknowns', 'verify_unknown_resolution', 'record_unknown', 'update_unknown', 'reset_investigation', 'binary_session', 'capture_browser_scenario', - 'resolve_containing_procedure', 'analyze_function', 'close_binary']: - config += '[mcp_servers.rea.tools.' + tool + ']\napproval_mode = "approve"\n' + 'resolve_containing_procedure', 'analyze_function', 'close_binary'] + if not args.no_mcp: + for tool in approved_tools: + config += '[mcp_servers.rea.tools.' + tool + ']\napproval_mode = "approve"\n' try: if source_auth.exists(): shutil.copyfile(source_auth, auth) @@ -220,10 +225,13 @@ def main(): 'mode': args.mode, 'model': args.model, 'reasoning': args.reasoning, 'capture_tool_context': args.capture_tool_context, 'guide_enabled_tools': args.guide_enabled_tools, + 'catalog_load': args.catalog_load, 'mcp_enabled': not args.no_mcp, 'driver_commit': subprocess.check_output( ['git', 'rev-parse', 'HEAD'], cwd=driver_repository, text=True).strip(), 'runtime_revision': args.revision, 'runtime_root': str(repository), + 'workspace': str(workspace), + 'workspace_isolated_from_runtime': not workspace.is_relative_to(repository.resolve()), 'selected_artifact_path': args.artifact_path, 'selected_artifact_sha256_before': target_digest, 'enabled_tools': enabled_tools, @@ -233,7 +241,7 @@ def main(): 'package_lock_sha256': file_digest(repository / 'package-lock.json'), 'experiment_sources_sha256': { name: hashlib.sha256((experiment / name).read_bytes()).hexdigest() - for name in ['agent-trial.py', 'codex_context.py', 'provider-environment.json'] + for name in ['agent-trial.py', 'codex_context.py', 'runtime_identity.py', 'provider-environment.json'] }, 'developer_instructions': instruction, 'developer_instructions_sha256': hashlib.sha256(instruction.encode()).hexdigest(), @@ -246,8 +254,7 @@ def main(): 'provider_environment_policy_sha256': hashlib.sha256( provider_policy_text.encode()).hexdigest(), 'features': {'shell_tool': False, 'unified_exec': False, - 'multi_agent': False, 'code_mode': args.mode == 'client-code', - 'code_mode_only': args.mode == 'client-code'}, + 'multi_agent': False, 'code_mode_host': args.mode == 'client-code'}, }, indent=2) + '\n') invocation = [args.codex, '--no-daemon', 'exec', '--strict-config', *([] if args.capture_tool_context else ['--ephemeral']), @@ -290,9 +297,17 @@ def main(): ).encode()), }) final_target_digest = file_digest(Path(args.artifact_path)) if args.artifact_path else None + try: + _, final_runtime_digest = runtime_fingerprint(repository) + except OSError: + final_runtime_digest = None summary = { 'mode': args.mode, 'requested_model': args.model, 'reasoning': args.reasoning, 'runtime_revision': args.revision, + 'runtime_files_sha256_before': runtime_digest, + 'runtime_files_sha256_after': final_runtime_digest, + 'runtime_bytes_unchanged': runtime_digest == final_runtime_digest, + 'catalog_load': args.catalog_load, 'mcp_enabled': not args.no_mcp, 'capture_tool_context_requested': args.capture_tool_context, 'selected_artifact_sha256_before': target_digest, 'selected_artifact_sha256_after': final_target_digest, @@ -315,7 +330,7 @@ def main(): # Capture failure must never prevent copied-credential cleanup. shutil.rmtree(profile) (args.output / 'cleanup.json').write_text(json.dumps({'profile_removed': not profile.exists()}) + '\n') - raise SystemExit(124 if timeout else exit_code) + raise SystemExit(124 if timeout else 125 if final_runtime_digest != runtime_digest else exit_code) if __name__ == '__main__': diff --git a/tests/support/analysis-efficiency/catalog-eval.py b/tests/support/analysis-efficiency/catalog-eval.py new file mode 100644 index 0000000000..6081992a76 --- /dev/null +++ b/tests/support/analysis-efficiency/catalog-eval.py @@ -0,0 +1,284 @@ +"""Measure catalog loading in real Codex sessions, never by a bytes/token ratio.""" + +import argparse +import hashlib +import json +from pathlib import Path +import statistics +import subprocess +import sys + +from codex_context import token_counts +from runtime_identity import runtime_fingerprint + +HERE = Path(__file__).resolve().parent +ROOT = HERE.parents[2] +MARKER = 'CATALOG_OK' +PROMPT = 'Reply with exactly ' + MARKER + '.\n' +TOKEN_KEYS = ('input_tokens', 'cached_input_tokens', 'output_tokens') + + +def write_json(path, value): + path.write_text(json.dumps(value, indent=2, ensure_ascii=False) + '\n') + + +def usage_counts(value): + counts = token_counts(value) + if not all(key in counts for key in TOKEN_KEYS) \ + or counts['input_tokens'] == 0 or counts['cached_input_tokens'] > counts['input_tokens']: + raise ValueError('Missing or invalid observed token usage') + return counts + + +def assess_trial(directory, returncode, expected_runtime_digest): + """Collect independent partial observations before grading the trial.""" + result = {'status': 'invalid', 'directory': str(directory), 'exit_code': returncode} + issues = [] + + def observe(stage, work): + try: + return work() + except (OSError, ValueError, KeyError, TypeError) as error: + issues.append({'stage': stage, 'reason': str(error)}) + return None + + def require(condition, stage, reason): + if not condition: + issues.append({'stage': stage, 'reason': reason}) + + def read_object(name): + value = json.loads((directory / name).read_text()) + if not isinstance(value, dict): + raise ValueError('Expected an object in ' + name) + return value + + summary = observe('summary', lambda: read_object('summary.json')) + settings = observe('settings', lambda: read_object('trial-settings.json')) + cleanup = observe('cleanup', lambda: read_object('cleanup.json')) + if summary is not None: + result['summary'] = summary + result['runtime_fingerprint_sha256'] = summary.get('runtime_files_sha256_before') + require(returncode == 0 and summary.get('exit_code') == 0 and summary.get('timeout') is False, + 'completion', 'Trial failed or timed out') + require(expected_runtime_digest is not None and summary.get('runtime_bytes_unchanged') is True + and summary.get('runtime_files_sha256_before') == expected_runtime_digest + and summary.get('runtime_files_sha256_after') == expected_runtime_digest, + 'identity', 'Runtime does not match the pinned matrix fingerprint') + if settings is not None: + require(settings.get('catalog_load') is True and settings.get('workspace_isolated_from_runtime') is True, + 'settings', 'Trial was not an isolated catalog-load measurement') + if cleanup is not None: + result['cleanup'] = cleanup + require(cleanup.get('profile_removed') is True, 'cleanup', 'Trial profile cleanup was not verified') + + events = [] + + def read_events(): + with (directory / 'events.jsonl').open() as lines: + for ordinal, line in enumerate(lines, 1): + if not line.strip(): + continue + def parse(): + value = json.loads(line) + if not isinstance(value, dict): + raise ValueError('Expected a JSONL event object') + return value + event = observe('events line ' + str(ordinal), parse) + if event is not None: + events.append(event) + + observe('events', read_events) + turns = [event for event in events if event.get('type') == 'turn.completed'] + require(len(turns) == 1, 'usage', 'Expected exactly one completed turn') + usage = observe('usage', lambda: usage_counts(turns[0].get('usage'))) if len(turns) == 1 else None + if usage is not None: + result['usage'] = usage + result['usage']['uncached_input_tokens'] = ( + usage['input_tokens'] - usage['cached_input_tokens']) + for event in events: + require(event.get('type') not in ['error', 'turn.failed'], 'client', + 'Client reported an error or failed turn: ' + str(event.get('message', event.get('error')))) + items = [event.get('item') for event in events if event.get('type') in ['item.started', 'item.completed']] + for item in items: + if isinstance(item, dict) and item.get('type') == 'error': + require(False, 'client', 'Client item error: ' + str(item.get('message'))) + else: + require(isinstance(item, dict) and item.get('type') in ['agent_message', 'reasoning'], + 'actions', 'A load-only trial performed a tool/action or an unknown item') + completed = [event.get('item') for event in events if event.get('type') == 'item.completed'] + answers = [item.get('text') for item in completed if isinstance(item, dict) and item.get('type') == 'agent_message'] + require(answers == [MARKER], 'answer', 'Load-only answer did not match the oracle') + + def read_resources(): + receipts = [] + for line in directory.with_suffix('.log').read_text().splitlines(): + try: + row = json.loads(line) + except ValueError: + continue + if isinstance(row, dict) and 'resource_scope' in row: + receipts.append(row) + if len(receipts) != 1: + raise ValueError('Missing or ambiguous resource receipt') + return receipts[0] + + receipt = observe('resources', read_resources) + if receipt is not None: + result['resources'] = receipt + observation = receipt.get('payload_resource_observation') + require(isinstance(observation, dict) and receipt.get('abort') is None + and receipt.get('payload_resource_observation_verified') is True + and observation.get('payload_exit_code') == 0 and observation.get('payload_cleanup_verified') is True, + 'resources', 'Resource stop, payload failure, or unverified process cleanup') + def metrics(): + cpu = observation['cpu_stat']['usage_usec'] + peak = observation['memory_peak_bytes'] + if any(not isinstance(value, int) or isinstance(value, bool) or value < 0 for value in [cpu, peak]): + raise ValueError('Invalid resource counters') + return {'elapsed_seconds': receipt.get('elapsed_seconds'), 'cpu_seconds': cpu / 1_000_000, + 'peak_process_family_bytes': peak, + 'scope': 'Kernel cgroup counters at payload exit, including observer and file cache; not final scope counters.'} + measured = observe('resource metrics', metrics) + if measured is not None: + result['resource_metrics'] = measured + result['status'] = 'invalid' if issues else 'valid' + if issues: + result['issues'] = issues + result['reason'] = '; '.join(issue['reason'] for issue in issues) + return result + + +def comparison(rows, expected_runtime_digest): + """Pair each catalog with its same-mode, same-repetition no-server control.""" + controls = {(row['mode'], row['repetition']): row for row in rows if row['arm'] == 'none'} + pairs = [] + for row in rows: + if row['arm'] == 'none': + continue + baseline = controls.get((row['mode'], row['repetition'])) + valid = expected_runtime_digest is not None and row['status'] == 'valid' and baseline is not None and baseline['status'] == 'valid' \ + and row.get('runtime_fingerprint_sha256') == expected_runtime_digest \ + and baseline.get('runtime_fingerprint_sha256') == expected_runtime_digest + pairs.append({ + 'mode': row['mode'], 'arm': row['arm'], 'repetition': row['repetition'], + 'status': 'valid' if valid else 'invalid', + 'input_token_delta': row['usage']['input_tokens'] - baseline['usage']['input_tokens'] if valid else None, + }) + groups = [] + for mode, arm in sorted({(pair['mode'], pair['arm']) for pair in pairs}): + selected = [pair for pair in pairs if pair['mode'] == mode and pair['arm'] == arm] + deltas = [pair['input_token_delta'] for pair in selected if pair['status'] == 'valid'] + groups.append({'mode': mode, 'arm': arm, 'valid_pairs': len(deltas), 'attempted_pairs': len(selected), + 'median_input_token_delta': statistics.median(deltas) if deltas else None, + 'min_input_token_delta': min(deltas) if deltas else None, + 'max_input_token_delta': max(deltas) if deltas else None}) + return {'pairs': pairs, 'groups': groups} + + +def catalog_receipt(root, profile, selected): + source = root / 'dist' / ('mcp-catalog-full.json' if profile == 'full' else 'mcp-catalog.json') + data = source.read_bytes() + catalog = json.loads(data) + names = {tool['name'] for tool in catalog['tools']} + if len(selected) != len(set(selected)) or not selected or not set(selected) <= names: + raise ValueError('Selected tools must be unique exact names in the built catalog') + size = lambda value: len(json.dumps(value, ensure_ascii=False, separators=(',', ':')).encode()) + return {'path': str(source), 'sha256': hashlib.sha256(data).hexdigest(), 'bytes': len(data), + 'tool_count': len(names), 'selected_tools': selected, + 'tools_json_bytes': size(catalog['tools']), + 'description_json_bytes': sum(size(tool.get('description')) for tool in catalog['tools']), + 'input_schema_json_bytes': sum(size(tool.get('inputSchema')) for tool in catalog['tools']), + 'output_schema_json_bytes': sum(size(tool.get('outputSchema')) for tool in catalog['tools']), + 'scope': 'Built server catalog, not submitted model context. Client filtering still discovers the server catalog.'} + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--runtime-root', type=Path, default=ROOT) + parser.add_argument('--output', type=Path, required=True) + parser.add_argument('--model', required=True, help='Explicit Codex model alias; never inferred from user config') + parser.add_argument('--reasoning', default='low') + parser.add_argument('--modes', nargs='+', choices=['direct', 'client-code'], default=['client-code'], + help='Default: supported host mode. direct disables the host as a diagnostic control.') + parser.add_argument('--input-profile', choices=['compact', 'full'], default='compact') + parser.add_argument('--selected-tools', nargs='+', default=['binary_session']) + parser.add_argument('--repetitions', type=int, default=1) + parser.add_argument('--timeout', type=int, default=180) + parser.add_argument('--codex', default='codex') + parser.add_argument('--plan', action='store_true', help='Show the matrix without invoking a model') + args = parser.parse_args() + if args.repetitions < 1 or args.timeout < 1 or len(set(args.modes)) != len(args.modes): + parser.error('Repetitions/timeout must be positive; modes must be unique') + root = args.runtime_root.resolve() + output = args.output.resolve() + runtime_files, pinned_digest = (None, None) if args.plan else runtime_fingerprint(root) + receipt = catalog_receipt(root, args.input_profile, args.selected_tools) + if not args.plan: + catalog_path = Path(receipt['path']).relative_to(root).as_posix() + pinned_catalog = next((row for row in runtime_files if row['path'] == catalog_path), None) + if pinned_catalog is None or pinned_catalog['sha256'] != receipt['sha256']: + raise ValueError('Catalog changed while pinning the matrix runtime; rebuild and retry') + revision = subprocess.check_output(['git', 'rev-parse', 'HEAD'], cwd=root, text=True).strip() + matrix = [] + for repetition in range(args.repetitions): + # Reverse both mode and arm ordering on alternating rounds to expose + # order/cache effects. Every invocation still has a fresh client/server. + for mode in args.modes[::1 if repetition % 2 == 0 else -1]: + for arm in ['none', 'all', 'selected'][::1 if repetition % 2 == 0 else -1]: + matrix.append({'repetition': repetition + 1, 'mode': mode, 'arm': arm}) + experiment = {'version': 1, 'runtime_revision': revision, 'model': args.model, 'reasoning': args.reasoning, + 'input_profile': args.input_profile, 'catalog': receipt, 'matrix': matrix, + 'sources_sha256': {name: hashlib.sha256((HERE / name).read_bytes()).hexdigest() + for name in ['catalog-eval.py', 'agent-trial.py', 'codex_context.py', 'runtime_identity.py', + 'resource-run.py', 'resource_payload.py', 'provider-environment.json']}, + 'scope': 'Fresh-session load-only usage. Warm OS/API caches are possible; no cache-miss claim or task-quality claim.', + 'usage_scope': 'Codex turn.completed observed usage; deltas include server instructions and client framing, not an exact schema-only token count.'} + if args.plan: + print(json.dumps(experiment, indent=2)) + return 0 + experiment['runtime_files_sha256'] = pinned_digest + output.mkdir(parents=True, exist_ok=False, mode=0o700) + write_json(output / 'experiment.json', experiment) + write_json(output / 'runtime-files.json', runtime_files) + (output / 'prompt.txt').write_text(PROMPT) + write_json(output / 'selected-tools.json', args.selected_tools) + write_json(output / 'providers.json', {'REA_MCP_INPUT_SCHEMA_PROFILE': args.input_profile}) + rows = [] + for index, cell in enumerate(matrix): + reason = 'Runtime changed between trials; no further model calls were made' + try: + _, current_digest = runtime_fingerprint(root) + except (OSError, ValueError) as error: + current_digest = None + reason = 'Runtime identity unavailable; no further model calls were made: ' + str(error) + if current_digest != pinned_digest: + report = {'experiment': experiment, 'trials': rows, 'comparison': comparison(rows, pinned_digest), + 'unattempted': matrix[index:], 'observed_runtime_digest': current_digest, 'reason': reason} + write_json(output / 'report.json', report) + print(report['reason'], flush=True) + return 1 + directory = output / ('{repetition:02d}-{mode}-{arm}'.format(**cell)) + command = [sys.executable, str(HERE / 'resource-run.py'), '--observe-payload-exit', '--', + sys.executable, str(HERE / 'agent-trial.py'), '--runtime-root', str(root), '--revision', revision, + '--prompt', str(output / 'prompt.txt'), '--output', str(directory), '--model', args.model, + '--reasoning', args.reasoning, '--mode', cell['mode'], '--timeout', str(args.timeout), + '--codex', args.codex, '--providers', str(output / 'providers.json'), + '--catalog-load', '--capture-tool-context'] + if cell['arm'] == 'none': + command.append('--no-mcp') + elif cell['arm'] == 'selected': + command.extend(['--enabled-tools', str(output / 'selected-tools.json')]) + print(json.dumps({'starting': cell}), flush=True) + with directory.with_suffix('.log').open('w') as log: + outcome = subprocess.run(command, stdout=log, stderr=subprocess.STDOUT, check=False) + row = {**cell, **assess_trial(directory, outcome.returncode, pinned_digest)} + rows.append(row) + report = {'experiment': experiment, 'trials': rows, 'comparison': comparison(rows, pinned_digest)} + write_json(output / 'report.json', report) + print(json.dumps({'completed': cell, 'status': row['status'], 'usage': row.get('usage'), 'reason': row.get('reason')}), flush=True) + print(json.dumps(report['comparison']['groups'], indent=2)) + return 0 if all(row['status'] == 'valid' for row in rows) else 1 + + +if __name__ == '__main__': + sys.exit(main()) diff --git a/tests/support/analysis-efficiency/runtime_identity.py b/tests/support/analysis-efficiency/runtime_identity.py new file mode 100644 index 0000000000..b432866095 --- /dev/null +++ b/tests/support/analysis-efficiency/runtime_identity.py @@ -0,0 +1,25 @@ +"""One byte-identity owner shared by matrix admission and guarded trials.""" + +import hashlib +import json + + +def file_digest(path): + digest = hashlib.sha256() + with path.open('rb') as data: + for chunk in iter(lambda: data.read(1024 * 1024), b''): + digest.update(chunk) + return digest.hexdigest() + + +def runtime_fingerprint(root): + files = [] + for name in ['dist', 'bridge']: + for path in sorted((root / name).rglob('*')): + if path.is_symlink(): + raise ValueError('Runtime fingerprint does not follow symlinks') + if path.is_file(): + files.append({'path': path.relative_to(root).as_posix(), 'sha256': file_digest(path)}) + files.sort(key=lambda row: row['path']) + text = json.dumps(files, separators=(',', ':'), ensure_ascii=False) + return files, hashlib.sha256(text.encode()).hexdigest() diff --git a/tests/support/analysis-efficiency/test_catalog_eval.py b/tests/support/analysis-efficiency/test_catalog_eval.py new file mode 100644 index 0000000000..4907f87420 --- /dev/null +++ b/tests/support/analysis-efficiency/test_catalog_eval.py @@ -0,0 +1,236 @@ +"""Catalog measurement oracles and missing/failed measurement semantics.""" + +import copy +import importlib.util +import json +from pathlib import Path +import tempfile +import unittest +from unittest.mock import patch +import io +import sys +import subprocess + +spec = importlib.util.spec_from_file_location('catalog_eval', Path(__file__).with_name('catalog-eval.py')) +catalog = importlib.util.module_from_spec(spec) +spec.loader.exec_module(catalog) + + +class CatalogEvaluationTests(unittest.TestCase): + def test_plan_does_not_spawn_a_model_or_acquire_trial_directories(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + (root / 'dist').mkdir() + catalog.write_json(root / 'dist/mcp-catalog.json', { + 'tools': [{'name': 'binary_session', 'inputSchema': {}}]}) + output = root / 'unacquired-output' + arguments = ['catalog-eval.py', '--runtime-root', str(root), '--output', str(output), + '--model', 'explicit-test-alias', '--plan'] + stream = io.StringIO() + with patch.object(sys, 'argv', arguments), patch('sys.stdout', stream), \ + patch.object(catalog.subprocess, 'check_output', return_value='pinned\n'), \ + patch.object(catalog, 'runtime_fingerprint', side_effect=AssertionError('Plan must not hash the runtime')), \ + patch.object(catalog.subprocess, 'run') as launch: + self.assertEqual(catalog.main(), 0) + launch.assert_not_called() + self.assertFalse(output.exists()) + plan = json.loads(stream.getvalue()) + self.assertEqual(plan['model'], 'explicit-test-alias') + self.assertEqual({cell['mode'] for cell in plan['matrix']}, {'client-code'}) + self.assertEqual({cell['arm'] for cell in plan['matrix']}, {'none', 'all', 'selected'}) + + def test_unknown_or_invalid_usage_never_becomes_zero(self): + for value in [None, {}, {'input_tokens': 10}, + {'input_tokens': True, 'cached_input_tokens': 0, 'output_tokens': 1}, + {'input_tokens': 10, 'cached_input_tokens': 11, 'output_tokens': 1}, + {'input_tokens': 0, 'cached_input_tokens': 0, 'output_tokens': 1}, + {'input_tokens': 10, 'cached_input_tokens': 0, 'output_tokens': -1}]: + with self.subTest(value=value), self.assertRaises(ValueError): + catalog.usage_counts(value) + + def test_pairs_with_the_same_mode_and_round_without_hiding_failures(self): + rows = [ + {'mode': 'direct', 'arm': 'none', 'repetition': 1, 'status': 'valid', 'usage': {'input_tokens': 100}}, + {'mode': 'direct', 'arm': 'all', 'repetition': 1, 'status': 'valid', 'usage': {'input_tokens': 130}}, + {'mode': 'client-code', 'arm': 'none', 'repetition': 1, 'status': 'valid', 'usage': {'input_tokens': 1000}}, + {'mode': 'client-code', 'arm': 'all', 'repetition': 1, 'status': 'valid', 'usage': {'input_tokens': 1200}}, + {'mode': 'direct', 'arm': 'all', 'repetition': 2, 'status': 'valid', 'usage': {'input_tokens': 140}}, + {'mode': 'direct', 'arm': 'none', 'repetition': 3, 'status': 'invalid'}, + {'mode': 'direct', 'arm': 'all', 'repetition': 3, 'status': 'valid', 'usage': {'input_tokens': 150}}, + ] + for row in rows: + row['runtime_fingerprint_sha256'] = 'a' * 64 + result = catalog.comparison(rows, 'a' * 64) + self.assertEqual([pair['input_token_delta'] for pair in result['pairs']], [30, 200, None, None]) + direct = next(group for group in result['groups'] if group['mode'] == 'direct') + self.assertEqual(direct['attempted_pairs'], 3) + self.assertEqual(direct['valid_pairs'], 1) + self.assertEqual(direct['median_input_token_delta'], 30) + + def test_rejects_different_builds_even_if_each_trial_is_individually_valid(self): + def row(arm, digest): + return {'mode': 'client-code', 'arm': arm, 'repetition': 1, 'status': 'valid', + 'usage': {'input_tokens': 100}, 'runtime_fingerprint_sha256': digest} + for left, right, pin in [('a' * 64, 'b' * 64, 'a' * 64), + ('b' * 64, 'b' * 64, 'a' * 64), (None, None, None)]: + result = catalog.comparison([row('none', left), row('all', right)], pin) + self.assertEqual(result['pairs'][0]['status'], 'invalid') + self.assertIsNone(result['groups'][0]['median_input_token_delta']) + + def test_matrix_stops_before_more_model_calls_when_another_checkout_rebuilds(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + runtime = root / 'other-checkout' + (runtime / 'dist').mkdir(parents=True) + module = runtime / 'dist/module.js' + module.write_text('build A') + catalog.write_json(runtime / 'dist/mcp-catalog.json', {'tools': [{'name': 'binary_session'}]}) + _, pin = catalog.runtime_fingerprint(runtime) + output = root / 'output' + arguments = ['catalog-eval.py', '--runtime-root', str(runtime), '--output', str(output), '--model', 'test-alias'] + + def launch(command, **options): + directory = Path(command[command.index('--output') + 1]) + directory.mkdir() + catalog.write_json(directory / 'summary.json', { + 'exit_code': 0, 'timeout': False, 'runtime_bytes_unchanged': True, + 'runtime_files_sha256_before': pin, 'runtime_files_sha256_after': pin}) + catalog.write_json(directory / 'trial-settings.json', {'catalog_load': True, 'workspace_isolated_from_runtime': True}) + catalog.write_json(directory / 'cleanup.json', {'profile_removed': True}) + (directory / 'events.jsonl').write_text('\n'.join(json.dumps(event) for event in [ + {'type': 'item.completed', 'item': {'type': 'agent_message', 'text': catalog.MARKER}}, + {'type': 'turn.completed', 'usage': {'input_tokens': 100, 'cached_input_tokens': 0, 'output_tokens': 8}}, + ]) + '\n') + options['stdout'].write(json.dumps({ + 'resource_scope': 'fixture-scope', 'abort': None, 'payload_resource_observation_verified': True, + 'payload_resource_observation': {'payload_exit_code': 0, 'payload_cleanup_verified': True, + 'cpu_stat': {'usage_usec': 1}, 'memory_peak_bytes': 1}, + }) + '\n') + # The completed first trial is valid; the external build happens + # between trials, outside the driver's artifact lock. + module.write_text('build B') + return subprocess.CompletedProcess(command, 0) + + with patch.object(sys, 'argv', arguments), patch('builtins.print'), \ + patch.object(catalog.subprocess, 'check_output', return_value='fixed\n'), \ + patch.object(catalog.subprocess, 'run', side_effect=launch) as spawn: + self.assertEqual(catalog.main(), 1) + self.assertEqual(spawn.call_count, 1) + report = json.loads((output / 'report.json').read_text()) + self.assertEqual(report['experiment']['runtime_files_sha256'], pin) + self.assertEqual(report['trials'][0]['status'], 'valid') + self.assertEqual(len(report['unattempted']), 2) + + def test_catalog_receipt_must_belong_to_the_pinned_manifest(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + (root / 'dist').mkdir() + source = root / 'dist/mcp-catalog.json' + catalog.write_json(source, {'tools': [{'name': 'binary_session'}]}) + original = catalog.runtime_fingerprint + + def rebuild_after_pin(runtime): + snapshot = original(runtime) + catalog.write_json(source, {'tools': [{'name': 'binary_session', 'description': 'new build'}]}) + return snapshot + + output = root / 'unacquired' + arguments = ['catalog-eval.py', '--runtime-root', str(root), '--output', str(output), '--model', 'test-alias'] + with patch.object(sys, 'argv', arguments), \ + patch.object(catalog.subprocess, 'check_output', return_value='fixed\n'), \ + patch.object(catalog, 'runtime_fingerprint', side_effect=rebuild_after_pin), \ + patch.object(catalog.subprocess, 'run') as spawn: + with self.assertRaisesRegex(ValueError, 'Catalog changed'): + catalog.main() + spawn.assert_not_called() + self.assertFalse(output.exists()) + + def test_load_oracle_rejects_actions_failures_and_unverified_cleanup(self): + summary = {'exit_code': 0, 'timeout': False, 'runtime_bytes_unchanged': True, + 'runtime_files_sha256_before': 'a' * 64, 'runtime_files_sha256_after': 'a' * 64} + events = [ + {'type': 'item.completed', 'item': {'type': 'agent_message', 'text': catalog.MARKER}}, + {'type': 'turn.completed', 'usage': {'input_tokens': 100, 'cached_input_tokens': 80, 'output_tokens': 8}}, + ] + resources = {'resource_scope': 'fixture-scope', 'abort': None, + 'payload_resource_observation_verified': True, + 'payload_resource_observation': {'payload_exit_code': 0, 'payload_cleanup_verified': True, + 'cpu_stat': {'usage_usec': 1500}, 'memory_peak_bytes': 1000}} + with tempfile.TemporaryDirectory() as temporary: + directory = Path(temporary) / 'trial' + directory.mkdir() + + def assess(changed_events=None, changed_resources=None, changed_summary=None, cleaned=True, code=0): + catalog.write_json(directory / 'summary.json', summary if changed_summary is None else changed_summary) + catalog.write_json(directory / 'trial-settings.json', {'catalog_load': True, 'workspace_isolated_from_runtime': True}) + catalog.write_json(directory / 'cleanup.json', {'profile_removed': cleaned}) + (directory / 'events.jsonl').write_text(''.join( + json.dumps(event) + '\n' for event in (events if changed_events is None else changed_events))) + directory.with_suffix('.log').write_text(json.dumps( + resources if changed_resources is None else changed_resources) + '\n') + return catalog.assess_trial(directory, code, 'a' * 64) + + valid = assess() + self.assertEqual(valid['status'], 'valid') + self.assertEqual(valid['usage']['uncached_input_tokens'], 20) + for item in ['mcp_tool_call', 'command_execution', 'tool_search', 'error', 'new_unknown_action']: + action = {'type': 'item.completed', 'item': {'type': item}} + self.assertEqual(assess(changed_events=[action, *events])['status'], 'invalid') + self.assertEqual(assess(changed_events=events + [events[-1]])['status'], 'invalid') + self.assertEqual(assess(changed_events=events[:1])['status'], 'invalid') + notice = {'type': 'item.completed', 'item': {'type': 'error', 'message': 'Host disabled; tool dispatch is unavailable'}} + failure = assess(changed_events=[notice, *events]) + self.assertIn(notice['item']['message'], failure['reason']) + self.assertEqual(failure['usage']['input_tokens'], 100) + self.assertEqual(failure['resources'], resources) + self.assertTrue(failure['cleanup']['profile_removed']) + failure = assess(changed_events=[{'type': 'error', 'message': 'outer failure'}, *events]) + self.assertEqual(failure['usage']['input_tokens'], 100) + self.assertEqual(failure['resource_metrics']['cpu_seconds'], 0.0015) + failure = assess(changed_events=[{'type': 'turn.failed'}]) + self.assertNotIn('usage', failure) + self.assertEqual(failure['resources'], resources) + assess() + transcript = directory / 'events.jsonl' + transcript.write_text('not-json\n' + transcript.read_text()) + failure = catalog.assess_trial(directory, 0, 'a' * 64) + self.assertEqual(failure['status'], 'invalid') + self.assertEqual(failure['usage']['input_tokens'], 100) + self.assertEqual(failure['resources'], resources) + for event in [{'type': 'turn.failed'}, {'type': 'error'}, None, + {'type': 'item.started', 'item': {'type': 'mcp_tool_call'}}, + {'type': 'item.completed', 'item': []}]: + self.assertEqual(assess(changed_events=[event, *events])['status'], 'invalid') + wrong_answer = copy.deepcopy(events) + wrong_answer[0]['item']['text'] = 'unverified facts' + self.assertEqual(assess(changed_events=wrong_answer)['status'], 'invalid') + self.assertEqual(assess(changed_summary={'exit_code': 0, 'timeout': True})['status'], 'invalid') + self.assertEqual(assess(changed_summary={**summary, 'runtime_bytes_unchanged': False})['status'], 'invalid') + self.assertEqual(assess(changed_summary={**summary, 'runtime_files_sha256_before': 'b' * 64, + 'runtime_files_sha256_after': 'b' * 64})['status'], 'invalid') + self.assertEqual(assess(cleaned=False)['status'], 'invalid') + self.assertEqual(assess(code=1)['status'], 'invalid') + for key, value in [('abort', 'memory_stop'), ('payload_resource_observation_verified', False), + ('payload_resource_observation', {'payload_exit_code': 0, 'payload_cleanup_verified': False})]: + failed = {**resources, key: value} + self.assertEqual(assess(changed_resources=failed)['status'], 'invalid') + + def test_built_catalog_keeps_transport_bytes_separate_from_client_selection(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + (root / 'dist').mkdir() + catalog.write_json(root / 'dist/mcp-catalog.json', { + 'tools': [{'name': 'one', 'inputSchema': {}, 'outputSchema': {}, 'description': 'one'}, + {'name': 'two', 'inputSchema': {}, 'outputSchema': {}, 'description': 'two'}]}) + one = catalog.catalog_receipt(root, 'compact', ['one']) + both = catalog.catalog_receipt(root, 'compact', ['one', 'two']) + self.assertEqual(one['bytes'], both['bytes']) + self.assertEqual(one['tool_count'], 2) + self.assertEqual(one['sha256'], both['sha256']) + for selected in [[], ['missing'], ['one', 'one']]: + with self.assertRaises(ValueError): + catalog.catalog_receipt(root, 'compact', selected) + + +if __name__ == '__main__': + unittest.main() diff --git a/tests/support/analysis-efficiency/test_codex_context.py b/tests/support/analysis-efficiency/test_codex_context.py index 8bd6d8c8a5..6b5c1350e7 100644 --- a/tests/support/analysis-efficiency/test_codex_context.py +++ b/tests/support/analysis-efficiency/test_codex_context.py @@ -673,9 +673,62 @@ class ContextCaptureTests(unittest.TestCase): with self.assertRaisesRegex(OSError, 'fixture capture failure'): agent.main() self.assertFalse(list(output.glob('profile-*'))) + self.assertFalse(list((root / '.cache/rea-agent-trials').glob('profile-*'))) self.assertTrue(json.loads((output / 'cleanup.json').read_text())['profile_removed']) self.assertEqual((auth_home / 'auth.json').read_text(), 'synthetic-credential') + def test_artifact_directory_does_not_choose_the_codex_workspace_or_enable_mcp_control(self): + spec = importlib.util.spec_from_file_location('agent_trial', Path(__file__).with_name('agent-trial.py')) + agent = importlib.util.module_from_spec(spec) + spec.loader.exec_module(agent) + for control, drift, mode in [(False, False, 'direct'), (True, False, 'direct'), + (False, True, 'direct'), (False, False, 'client-code')]: + with self.subTest(control=control, drift=drift, mode=mode), tempfile.TemporaryDirectory() as directory: + root = Path(directory) + runtime = root / 'runtime' + (runtime / 'scripts').mkdir(parents=True) + (runtime / 'scripts/rea.mjs').write_text('') + (runtime / 'package-lock.json').write_text('{}') + (runtime / '.nvmrc').write_text('24.22.1\n') + (runtime / 'AGENTS.md').write_text('Foreign project instructions must not reach the trial') + question = root / 'question' + question.write_text('Reply with exactly CATALOG_OK.') + output = runtime / '.cache/trial' + arguments = ['agent-trial.py', '--runtime-root', str(runtime), '--revision', 'fixture', + '--prompt', str(question), '--output', str(output), '--catalog-load', '--mode', mode, + *(['--no-mcp'] if control else [])] + observed = {} + + def launch(_invocation, **options): + workspace = options['cwd'] + self.assertFalse(workspace.is_relative_to(runtime)) + config = Path(options['env']['CODEX_HOME']) / 'config.toml' + observed['config'] = config.read_text() + if drift: + (runtime / 'dist').mkdir() + (runtime / 'dist/changed.js').write_text('Runtime changed during the trial') + child = Mock(stdin=io.StringIO()) + child.wait.return_value = 0 + return child + + with patch.dict(os.environ, {'CODEX_HOME': str(root / 'source-auth')}, clear=True), \ + patch.object(sys, 'argv', arguments), \ + patch.object(agent.shutil, 'which', return_value='/fixture/node'), \ + patch.object(agent.subprocess, 'check_output', return_value='v24.22.1\n'), \ + patch.object(agent.subprocess, 'Popen', side_effect=launch), patch('builtins.print'): + with self.assertRaises(SystemExit) as completed: + agent.main() + self.assertEqual(completed.exception.code, 125 if drift else 0) + self.assertEqual('[mcp_servers.rea]' in observed['config'], not control) + self.assertEqual('required = true' in observed['config'], not control) + self.assertNotIn('Use the registered REA MCP server', observed['config']) + self.assertIn('code_mode_host = ' + ('true' if mode == 'client-code' else 'false'), observed['config']) + self.assertNotIn('code_mode =', observed['config']) + self.assertNotIn('code_mode_only =', observed['config']) + self.assertTrue(json.loads((output / 'trial-settings.json').read_text())['workspace_isolated_from_runtime']) + self.assertEqual(json.loads((output / 'summary.json').read_text())['runtime_bytes_unchanged'], not drift) + self.assertFalse(list((root / '.cache/rea-agent-trials').glob('profile-*'))) + def test_refuses_foreign_session_directory_and_symlink_records(self): with tempfile.TemporaryDirectory() as directory: root = Path(directory)