merge: inherit resolved environment changes
This commit is contained in:
commit
bbce4fc2ea
622 files changed
+24509
-196582
No files matched your search
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"name": "rea",
|
||||
"interface": {
|
||||
"displayName": "REA"
|
||||
},
|
||||
"plugins": [
|
||||
{
|
||||
"name": "rea",
|
||||
"source": {
|
||||
"source": "npm",
|
||||
"package": "rea-agents",
|
||||
"version": "6.8.0"
|
||||
},
|
||||
"policy": {
|
||||
"installation": "AVAILABLE",
|
||||
"authentication": "ON_INSTALL"
|
||||
},
|
||||
"category": "Developer Tools"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -17,6 +17,9 @@ readiness or provider commands.
|
||||
|
||||
Installing this skill supplies instructions; it does not register the REA MCP
|
||||
server, install analysis engines, or add tools to an already-running agent.
|
||||
The exception is a Claude Code or Codex plugin install: the plugin bundles
|
||||
this skill and registers its own `rea` server, outside the state that
|
||||
`rea setup` and `rea doctor` manage.
|
||||
|
||||
If REA tools are available and their registration is not known to be stale,
|
||||
proceed directly to the target. Do not run diagnostics or setup before every
|
||||
@@ -37,37 +40,18 @@ repeat or known not to have started. If it may already have changed the
|
||||
target, or it fails again, tell the user and ask them to restart the MCP
|
||||
client. If REA could not start its connection, retry once, then tell the user
|
||||
and ask them to restart the MCP client. Do not retry a startup failure that
|
||||
returns the same unchanged result. Run `rea doctor` when REA could not start
|
||||
returns the same unchanged result. For plugin installs, follow
|
||||
[plugin recovery](references/connection-and-recovery.md#rea-installed-as-a-client-plugin).
|
||||
For setup-owned registrations, run `rea doctor` when REA could not start
|
||||
or tools stay missing after that restart. For `cleanup_incomplete`, a retryable
|
||||
capacity limit, or `configuration_invalid`, follow the recovery guide's ladder
|
||||
before asking for a restart. Read that guide before a second repair attempt.
|
||||
|
||||
1. Diagnose without changing REA or client configuration. `npx` may download
|
||||
the package and write to npm's cache. For Codex, run
|
||||
`npx -y rea-agents@latest doctor --client codex --json`. Substitute the current
|
||||
supported client: `claude_code`, `claude_desktop`, `codex`, `cursor`,
|
||||
`gemini_cli`, `windsurf`, `devin`, `opencode`, `antigravity`, `copilot_cli`,
|
||||
`commandcode`, `vscode`, `grok_build`, `omp`, or `grok_bot`. `grok_bot` has no local
|
||||
file registration: doctor reports the Grok Bot chat step, and setup does not
|
||||
write `mcp.json`. If the client is unknown, use `doctor --json` and
|
||||
inspect its registration results before choosing a setup scope.
|
||||
2. Distinguish the reason. Missing, malformed, or stale registration needs a
|
||||
scoped configuration repair. An aligned registration with no tools in the
|
||||
active session needs a restart/reconnection; doctor cannot prove that the
|
||||
current agent has connected. A missing provider affects only tasks requiring
|
||||
that provider: static JavaScript inspection needs neither Hopper nor Ghidra,
|
||||
and Android inspection has separate bring-your-own JADX/Java prerequisites.
|
||||
3. For a configuration repair, prepare the read-only plan:
|
||||
`npx -y rea-agents@latest setup --client codex --dry-run --json`.
|
||||
Use the current client's ID, show its exact proposed paths, backups, and
|
||||
changes, and obtain approval before setup writes configuration or installs
|
||||
Hopper. Setup normally installs the matching bundled skill too; include that
|
||||
replacement in the reviewed plan. After approval, apply the same scope with
|
||||
`npx -y rea-agents@latest setup --client codex --yes`. Add `--install-hopper`
|
||||
only if that separate installation was needed and explicitly approved.
|
||||
4. Restart/reconnect the affected agent. Verify that REA tools actually appear
|
||||
in the session, then resume the original investigation. If they remain
|
||||
absent, inspect the client's MCP launch error rather than repeating setup.
|
||||
Choose the installation owner before repairing registration. Plugin installs use
|
||||
[plugin recovery](references/connection-and-recovery.md#rea-installed-as-a-client-plugin);
|
||||
only setup-owned registrations use the guide's
|
||||
[registration repair plan](references/connection-and-recovery.md#repair-registration-when-needed).
|
||||
A plugin-only doctor result of `missing` does not call for `rea setup`.
|
||||
|
||||
REA setup never installs or upgrades Node.js, npm, Homebrew, Java, Ghidra, IDA,
|
||||
JADX, Kuna, Binwalk, or Unblob. Use existing prerequisites; do not install unrelated
|
||||
@@ -86,7 +70,7 @@ returns the Evidence record directly. This fallback does not establish that
|
||||
MCP is configured. Native CLI tasks still require their selected engine.
|
||||
Prefer an existing `rea` CLI. When none is installed and REA package acquisition
|
||||
is within the authorized scope, use `npx -y rea-agents@latest` in its place.
|
||||
Configuration repair starts with a read-only setup plan; show paths, backups and
|
||||
For setup-owned registration, configuration repair starts with a read-only setup plan; show paths, backups and
|
||||
skill replacement before applying authorized changes. Preserve existing approval
|
||||
and target-execution boundaries.
|
||||
|
||||
|
||||
@@ -13,6 +13,27 @@ own APK path and explicit selectors; do not open a native session first.
|
||||
Choose an `overload_index` from the class inventory when methods are ambiguous
|
||||
instead of guessing.
|
||||
|
||||
For JNI implementations, discover `inspect_android_jni_bindings`. Select raw
|
||||
DEX class/method identities and optional descriptors, ABI or logical library
|
||||
paths. Prefer `detail: "summary"`; complete Evidence remains retained and
|
||||
`android-jni` views page declarations, libraries, candidates or binding rows
|
||||
with exact fields. Global native-pair ambiguity survives class selection.
|
||||
Preserve slot/relocation provenance and source coverage: matched means observed
|
||||
candidate presence; partial scans do not establish uniqueness or absence.
|
||||
This reads DEX/ELF bytes without loading the APK into the decompiler or executing
|
||||
target code. Runtime registration and loading remain unknown. See the
|
||||
[JNI guide](https://github.com/morluto/rea/blob/main/docs/android-analysis.md#link-native-declarations-to-jni-candidates).
|
||||
|
||||
For an R8/ProGuard release build with its `mapping.txt`, pass the absolute
|
||||
`mapping_path` to `inspect_android_class` or `inspect_android_method`. Prefer
|
||||
`detail: "summary"` to receive normalized symbols or the selected body without
|
||||
duplicate raw class metadata. Complete Evidence stays retained under the linked
|
||||
parent ID and can be exported; CLI summary delivery requires `--evidence_output`.
|
||||
Omitted or complete detail preserves the full inline response. Keep
|
||||
selecting by observed names; the separate `mapping` block reports each symbol
|
||||
as `mapped`, `ambiguous` or `unmapped` after checking DEX types and prototypes.
|
||||
REA does not verify that the mapping belongs to the APK build.
|
||||
|
||||
The provider requires a separately supplied headless JADX MCP JAR 0.7.1 and
|
||||
Java 17 or newer. `REA_JADX_MCP_JAR` selects the absolute JAR path; `JAVA_HOME`
|
||||
can select Java. Use `rea doctor --provider jadx --json` to check the JADX
|
||||
|
||||
@@ -17,7 +17,7 @@ does not establish which server an approval wildcard covers. See
|
||||
|
||||
| Observation | Meaning and next step |
|
||||
| --------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| No REA tools in the active session | Inspect registration with `rea doctor --client codex --json`. A skill installation alone does not register MCP. |
|
||||
| No REA tools in the active session | If REA was installed as a client plugin, follow the plugin section below. Otherwise inspect registration with `rea doctor --client codex --json`. A skill installation alone does not register MCP. |
|
||||
| Missing, malformed or stale registration | Prepare a scoped setup plan; apply only authorized changes with backups. |
|
||||
| Aligned registration, but no connected tools | Restart/reconnect the client. Doctor checks registration; it cannot establish the current agent's connection. Inspect the client's launch error if reconnection fails. |
|
||||
| MCP connection dropped or REA could not start | If the connection drops, retry that call once only when it is safe to repeat or known not to have started. If it may already have changed the target, or the same failure returns, tell the user REA's connection failed and ask them to restart the MCP client. If REA could not start its connection, retry once, then tell the user and ask them to restart the MCP client. Do not retry a startup failure that returns the same unchanged result. Run `rea doctor` when REA could not start or tools stay missing after restart. Doctor cannot restore a live connection by itself. |
|
||||
@@ -116,6 +116,32 @@ is not a recommendation to change it. Do not invent a setting. Host memory and
|
||||
- `invalid_request`: correct the reported fields from the current schema. Do
|
||||
not repeat the unchanged call.
|
||||
|
||||
## REA installed as a client plugin
|
||||
|
||||
A Claude Code or Codex plugin owns its `rea` server and this skill. Doctor's
|
||||
`--client` and `--skill` checks and `rea setup` read and write only
|
||||
setup-owned state, so for a plugin-only install they report `missing`, and
|
||||
applying setup would add a second `rea` server. If this skill was loaded from
|
||||
a plugin, or the client lists a `rea` plugin, do not prepare a setup plan for
|
||||
missing tools. Instead:
|
||||
|
||||
- Check that the plugin is installed and enabled with the client's own
|
||||
listing: `claude plugin list` and `/mcp` in Claude Code, `codex plugin list`
|
||||
in Codex.
|
||||
- Restart or reconnect the client, then read its launch error for the `rea`
|
||||
server if tools are still absent. The plugin starts its bundled runtime; a missing build or version mismatch
|
||||
requires a reinstall, or a fresh build when developing from source.
|
||||
- Check analysis engines through the connected server's
|
||||
`binary_session.result.tool_availability`. `rea doctor --provider <id> --json`
|
||||
from a shell reports that shell's environment, which can differ from the
|
||||
environment the client gave the plugin server. If you still run it through
|
||||
the package runner, use the version pinned in the plugin manifest's server
|
||||
entry (`rea-agents@<version>`), not `@latest`, so it matches the server the
|
||||
plugin starts.
|
||||
|
||||
Use the setup repair below only when the user wants a setup-owned registration
|
||||
instead of the plugin, and remove or disable the plugin first.
|
||||
|
||||
## Repair registration when needed
|
||||
|
||||
For Codex:
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"$schema": "https://anthropic.com/claude-code/marketplace.schema.json",
|
||||
"name": "rea",
|
||||
"description": "Reverse engineer anything with agents, from app behavior down to native binaries.",
|
||||
"owner": {
|
||||
"name": "morluto",
|
||||
"url": "https://github.com/morluto"
|
||||
},
|
||||
"plugins": [
|
||||
{
|
||||
"name": "rea",
|
||||
"description": "Reverse engineer anything from your terminal or agent with one CLI and MCP server.",
|
||||
"source": {
|
||||
"source": "npm",
|
||||
"package": "rea-agents",
|
||||
"version": "6.8.0"
|
||||
},
|
||||
"category": "development"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"name": "rea",
|
||||
"version": "6.8.0",
|
||||
"description": "Reverse engineer anything from your terminal or agent with one CLI and MCP server.",
|
||||
"author": {
|
||||
"name": "morluto",
|
||||
"url": "https://github.com/morluto"
|
||||
},
|
||||
"homepage": "https://rea.tools/",
|
||||
"repository": "https://github.com/morluto/rea",
|
||||
"license": "MIT",
|
||||
"keywords": [
|
||||
"reverse-engineering",
|
||||
"binary-analysis",
|
||||
"decompiler",
|
||||
"mcp-server"
|
||||
],
|
||||
"skills": "./skills",
|
||||
"mcpServers": {
|
||||
"rea": {
|
||||
"command": "node",
|
||||
"args": [
|
||||
"${CLAUDE_PLUGIN_ROOT}/scripts/plugin-mcp-launcher.mjs",
|
||||
"6.8.0"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"mcpServers": {
|
||||
"rea": {
|
||||
"command": "node",
|
||||
"args": [
|
||||
"./scripts/plugin-mcp-launcher.mjs",
|
||||
"6.8.0"
|
||||
],
|
||||
"cwd": ".",
|
||||
"startup_timeout_sec": 30
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"name": "rea",
|
||||
"version": "6.8.0",
|
||||
"description": "Reverse engineer anything from your terminal or agent with one CLI and MCP server.",
|
||||
"author": {
|
||||
"name": "morluto",
|
||||
"url": "https://github.com/morluto"
|
||||
},
|
||||
"homepage": "https://rea.tools/",
|
||||
"repository": "https://github.com/morluto/rea",
|
||||
"license": "MIT",
|
||||
"keywords": [
|
||||
"reverse-engineering",
|
||||
"binary-analysis",
|
||||
"decompiler",
|
||||
"mcp-server"
|
||||
],
|
||||
"skills": "./skills",
|
||||
"mcpServers": "./.codex-plugin/mcp.json",
|
||||
"interface": {
|
||||
"displayName": "REA",
|
||||
"shortDescription": "Reverse engineer anything with agents",
|
||||
"longDescription": "Use REA to explain shipped behavior from native binaries, managed assemblies, Electron and JavaScript bundles, packages, firmware, and browser targets, with evidence that separates observations from inferences.",
|
||||
"developerName": "morluto",
|
||||
"category": "Developer Tools",
|
||||
"capabilities": [
|
||||
"Interactive",
|
||||
"Read",
|
||||
"Write"
|
||||
],
|
||||
"websiteURL": "https://rea.tools/",
|
||||
"defaultPrompt": [
|
||||
"Use REA to explain how this binary implements a feature.",
|
||||
"Use REA to compare two versions of this application.",
|
||||
"Use REA to recover the sources from this JavaScript bundle."
|
||||
]
|
||||
}
|
||||
}
|
||||
+18
-16
@@ -855,6 +855,24 @@ jobs:
|
||||
# The matching native artifact establishes real Job Object and private
|
||||
# runtime boundaries for these fixtures. Ghidra and Java are not needed;
|
||||
# real engine claims remain in the separate controlled-provider workflow.
|
||||
# Run native proofs before the broad curated suite so an unrelated
|
||||
# fixture failure cannot hide the real control and recovery checks.
|
||||
- uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
|
||||
with:
|
||||
name: windows-process-boundary-fixture
|
||||
path: .cache/windows-process-boundary/
|
||||
# Passing the fixture adds the breakaway, caller/child token, environment
|
||||
# semantics and ancestor reparse observations, which the lane skips
|
||||
# entirely when it is absent.
|
||||
- run: >-
|
||||
npm run verify:windows-native -- .
|
||||
"${{ github.workspace }}\.cache\windows-process-boundary\process-boundary.exe"
|
||||
- uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
|
||||
with:
|
||||
name: windows-native-inspection-fixture
|
||||
path: .cache/windows-native-inspection-fixture/
|
||||
- name: Verify native inspection failure ownership and recovery
|
||||
run: node scripts/verify-windows-native-inspection.mjs
|
||||
- name: Run curated Windows capability tests
|
||||
run: >-
|
||||
npx vitest run --coverage.enabled=false
|
||||
@@ -907,22 +925,6 @@ jobs:
|
||||
tests/process-global/runnerCancellation.test.ts
|
||||
tests/process-global/windowsOwnedCommand.test.ts
|
||||
tests/conformance/providers/ghidra/realGhidraWindowsWorkflow.test.ts
|
||||
- uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
|
||||
with:
|
||||
name: windows-process-boundary-fixture
|
||||
path: .cache/windows-process-boundary/
|
||||
# Passing the fixture adds the breakaway, caller/child token, environment
|
||||
# semantics and ancestor reparse observations, which the lane skips
|
||||
# entirely when it is absent.
|
||||
- run: >-
|
||||
npm run verify:windows-native -- .
|
||||
"${{ github.workspace }}\.cache\windows-process-boundary\process-boundary.exe"
|
||||
- uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
|
||||
with:
|
||||
name: windows-native-inspection-fixture
|
||||
path: .cache/windows-native-inspection-fixture/
|
||||
- name: Verify native inspection failure ownership and recovery
|
||||
run: node scripts/verify-windows-native-inspection.mjs
|
||||
- name: Verify real Node Inspector discovery and observation
|
||||
timeout-minutes: 10
|
||||
env:
|
||||
|
||||
+1
-1
@@ -50,7 +50,7 @@ Thumbs.db
|
||||
|
||||
# Local reference sources and analysis artifacts
|
||||
/_reference/
|
||||
/experiments/analysis-efficiency/results/
|
||||
/experiments/analysis-efficiency/
|
||||
|
||||
# Generated website download
|
||||
/website/public/examples/notes-example.zip
|
||||
|
||||
+4
-1
@@ -10,7 +10,10 @@
|
||||
"node_modules/**",
|
||||
"CHANGELOG.md",
|
||||
"package-lock.json",
|
||||
".claude-plugin/plugin.json",
|
||||
".codex-plugin/plugin.json",
|
||||
"_reference/**",
|
||||
"third_party/**"
|
||||
"third_party/**",
|
||||
".codex-plugin/mcp.json"
|
||||
]
|
||||
}
|
||||
@@ -88,7 +88,7 @@ configuration. Restart your agent afterward.
|
||||
Setup supports Claude Code, Codex, Cursor, Gemini CLI, Grok Build and
|
||||
[other agents](docs/installation.md#supported-agents). See
|
||||
[installation and setup](docs/installation.md) for provider configuration and
|
||||
manual MCP registration.
|
||||
manual MCP registration and Claude Code/Codex plugin installation.
|
||||
|
||||
MCP uses compact input schemas by default to reduce context overhead. Set
|
||||
`REA_MCP_INPUT_SCHEMA_PROFILE=full` in your MCP registration when you need the
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
import com.atxx.jhmcp.*;
|
||||
import com.google.gson.*;
|
||||
import com.android.tools.smali.dexlib2.Opcodes;
|
||||
import com.android.tools.smali.dexlib2.dexbacked.DexBackedDexFile;
|
||||
import com.android.tools.smali.dexlib2.dexbacked.DexBackedClassDef;
|
||||
import com.android.tools.smali.dexlib2.dexbacked.DexBackedMethod;
|
||||
import io.modelcontextprotocol.kotlin.sdk.server.*;
|
||||
import io.modelcontextprotocol.kotlin.sdk.types.*;
|
||||
import jadx.api.*;
|
||||
@@ -9,14 +13,21 @@ import jadx.api.plugins.input.data.ICodeReader;
|
||||
import jadx.api.plugins.input.data.IMethodRef;
|
||||
import jadx.api.plugins.input.insns.InsnIndexType;
|
||||
import jadx.api.plugins.input.insns.Opcode;
|
||||
import jadx.core.codegen.TypeGen;
|
||||
import jadx.core.dex.info.MethodInfo;
|
||||
import jadx.core.dex.nodes.*;
|
||||
import java.io.PrintStream;
|
||||
import java.io.StringReader;
|
||||
import java.io.InputStream;
|
||||
import java.io.IOException;
|
||||
import java.nio.file.Path;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.*;
|
||||
import java.util.concurrent.CountDownLatch;
|
||||
import java.util.stream.Collectors;
|
||||
import java.util.zip.CRC32;
|
||||
import java.util.zip.ZipEntry;
|
||||
import java.util.zip.ZipFile;
|
||||
import kotlin.Unit;
|
||||
import kotlin.ResultKt;
|
||||
import kotlin.coroutines.Continuation;
|
||||
@@ -178,6 +189,205 @@ class ReaJadxBridge {
|
||||
manifestSnapshot = null;
|
||||
}
|
||||
|
||||
// One bounded DEX at a time, beside the already admitted analysis JVM. The
|
||||
// response allowance accounts for worst-case JSON escaping before retention.
|
||||
private static final int NATIVE_DEX_BYTES = 64 * 1024 * 1024;
|
||||
private static final long NATIVE_TOTAL_DEX_BYTES = 256L * 1024 * 1024;
|
||||
// This JSON is escaped again inside an 8 MiB MCP frame.
|
||||
private static final long NATIVE_JSON_BYTES = 4L * 1024 * 1024;
|
||||
|
||||
/** The bundled decoder allocates its declared UTF-16 length before validating bytes. */
|
||||
private static final class NativeDex extends DexBackedDexFile {
|
||||
private OptionalIndexedSection<String> boundedStrings;
|
||||
NativeDex(byte[] bytes) { super(Opcodes.getDefault(), bytes); }
|
||||
|
||||
@Override public OptionalIndexedSection<String> getStringSection() {
|
||||
if (boundedStrings == null) {
|
||||
var source = super.getStringSection();
|
||||
boundedStrings = new OptionalIndexedSection<>() {
|
||||
@Override public int size() { return source.size(); }
|
||||
@Override public int getOffset(int index) { return source.getOffset(index); }
|
||||
@Override public String getOptional(int index) { return index == -1 ? null : get(index); }
|
||||
@Override public String get(int index) {
|
||||
int offset = getBuffer().readSmallUint(source.getOffset(index));
|
||||
int units = getDataBuffer().readerAt(offset).readSmallUleb128();
|
||||
if (units > NATIVE_JSON_BYTES / 6
|
||||
|| units > getDataBuffer().getBuf().length - offset)
|
||||
throw new IllegalArgumentException("DEX string exceeds the native decoding allowance");
|
||||
return source.get(index);
|
||||
}
|
||||
};
|
||||
}
|
||||
return boundedStrings;
|
||||
}
|
||||
}
|
||||
|
||||
private static void checkNativeInterrupted() {
|
||||
if (Thread.currentThread().isInterrupted())
|
||||
throw new java.util.concurrent.CancellationException("Native declaration inventory interrupted");
|
||||
}
|
||||
|
||||
private static final class NativeInventory {
|
||||
final List<Map<String, Object>> methods = new ArrayList<>();
|
||||
final List<String> unknowns = new ArrayList<>();
|
||||
long retainedBytes = 1024;
|
||||
int dexEntries;
|
||||
int examinedDexEntries;
|
||||
int examinedClasses;
|
||||
int examinedMethods;
|
||||
boolean partial;
|
||||
boolean diagnosticLimit;
|
||||
|
||||
void unknown(String reason) {
|
||||
partial = true;
|
||||
long bytes = 64L + reason.length() * 6L;
|
||||
if (unknowns.size() < 16 && retainedBytes + bytes <= NATIVE_JSON_BYTES - 4096) {
|
||||
unknowns.add(reason);
|
||||
retainedBytes += bytes;
|
||||
} else if (!diagnosticLimit) {
|
||||
diagnosticLimit = true;
|
||||
unknowns.add("Additional native-inventory limitations exceed the retained diagnostic allowance.");
|
||||
}
|
||||
}
|
||||
|
||||
boolean retain(Map<String, Object> method, long bytes) {
|
||||
if (retainedBytes + bytes > NATIVE_JSON_BYTES - 4096) {
|
||||
unknown("Native declarations exceed the 4 MiB retained JSON allowance; remaining declarations are unknown.");
|
||||
return false;
|
||||
}
|
||||
retainedBytes += bytes;
|
||||
methods.add(method);
|
||||
return true;
|
||||
}
|
||||
|
||||
Map<String, Object> result() {
|
||||
return Map.of("scope", "apk-root-dex-declarations", "methods", List.copyOf(methods),
|
||||
"coverage", Map.of("status", partial ? "partial" : "complete",
|
||||
"dex_entries", dexEntries, "examined_dex_entries", examinedDexEntries,
|
||||
"examined_classes", examinedClasses, "examined_methods", examinedMethods),
|
||||
"unknowns", List.copyOf(unknowns));
|
||||
}
|
||||
}
|
||||
|
||||
/** Match Android's standard root multidex carriers, without a basename guess. */
|
||||
private static boolean nativeDexEntry(String name) {
|
||||
return name.equals("classes.dex") || name.matches("classes[0-9]+\\.dex");
|
||||
}
|
||||
|
||||
private static byte[] readNativeDex(ZipFile archive, ZipEntry entry) throws IOException {
|
||||
long size = entry.getSize();
|
||||
if (size < 0 || size > NATIVE_DEX_BYTES)
|
||||
throw new IllegalArgumentException("DEX exceeds the 64 MiB single-member read allowance");
|
||||
byte[] bytes = new byte[(int) size];
|
||||
CRC32 crc = new CRC32();
|
||||
try (InputStream input = archive.getInputStream(entry)) {
|
||||
int offset = 0;
|
||||
while (offset < bytes.length) {
|
||||
checkNativeInterrupted();
|
||||
int read = input.read(bytes, offset, Math.min(65536, bytes.length - offset));
|
||||
if (read < 0) throw new IOException("DEX is shorter than its ZIP declaration");
|
||||
if (read == 0) continue;
|
||||
crc.update(bytes, offset, read);
|
||||
offset += read;
|
||||
}
|
||||
if (input.read() != -1) throw new IOException("DEX exceeds its ZIP declaration");
|
||||
}
|
||||
if (entry.getCrc() < 0 || entry.getCrc() != crc.getValue())
|
||||
throw new IOException("DEX does not match its ZIP CRC32 declaration");
|
||||
return bytes;
|
||||
}
|
||||
|
||||
/** Raw DEX ownership precedes JADX's duplicate-class selection and aliases. */
|
||||
static Map<String, Object> nativeDeclarations(Path apk) {
|
||||
NativeInventory inventory = new NativeInventory();
|
||||
try (ZipFile archive = new ZipFile(apk.toFile())) {
|
||||
Map<String, Integer> names = new HashMap<>();
|
||||
var directory = archive.entries();
|
||||
while (directory.hasMoreElements()) {
|
||||
checkNativeInterrupted();
|
||||
ZipEntry entry = directory.nextElement();
|
||||
if (!entry.isDirectory() && nativeDexEntry(entry.getName())) {
|
||||
inventory.dexEntries++;
|
||||
names.merge(entry.getName(), 1, Integer::sum);
|
||||
}
|
||||
}
|
||||
long totalBytes = 0;
|
||||
int entryIndex = -1;
|
||||
directory = archive.entries();
|
||||
while (directory.hasMoreElements()) {
|
||||
checkNativeInterrupted();
|
||||
ZipEntry entry = directory.nextElement();
|
||||
entryIndex++;
|
||||
String path = entry.getName();
|
||||
if (entry.isDirectory() || !nativeDexEntry(path)) continue;
|
||||
if (names.get(path) != 1) {
|
||||
inventory.unknown("Duplicate root DEX entry name cannot be selected uniquely by the ZIP reader: " + path);
|
||||
continue;
|
||||
}
|
||||
if (entry.getSize() < 0 || entry.getSize() > NATIVE_DEX_BYTES
|
||||
|| entry.getSize() > NATIVE_TOTAL_DEX_BYTES - totalBytes) {
|
||||
inventory.unknown("Root DEX member exceeds the 64 MiB member or 256 MiB total read allowance: " + path);
|
||||
continue;
|
||||
}
|
||||
totalBytes += entry.getSize();
|
||||
try {
|
||||
DexBackedDexFile dex = new NativeDex(readNativeDex(archive, entry));
|
||||
var classes = dex.getClassSection();
|
||||
for (int classIndex = 0; classIndex < classes.size(); classIndex++) {
|
||||
checkNativeInterrupted();
|
||||
DexBackedClassDef cls = classes.get(classIndex);
|
||||
String type = cls.getType();
|
||||
if (!type.startsWith("L") || !type.endsWith(";"))
|
||||
throw new IllegalArgumentException("Class definition has no reference type");
|
||||
String className = type.substring(1, type.length() - 1).replace('/', '.');
|
||||
// Passing false retains duplicate encoded method references too.
|
||||
for (Iterable<? extends DexBackedMethod> group : List.of(
|
||||
cls.getDirectMethods(false), cls.getVirtualMethods(false))) {
|
||||
for (DexBackedMethod method : group) {
|
||||
checkNativeInterrupted();
|
||||
inventory.examinedMethods++;
|
||||
if ((method.getAccessFlags() & 0x100) == 0) continue;
|
||||
String methodName = method.getName();
|
||||
String returnType = method.getReturnType();
|
||||
long characters = 2L + returnType.length();
|
||||
for (CharSequence parameter : method.getParameterTypes()) {
|
||||
characters += parameter.length();
|
||||
if (characters * 6L > NATIVE_JSON_BYTES) break;
|
||||
}
|
||||
long bytes = 512L + 6L * (className.length() + methodName.length()
|
||||
+ path.length() + characters);
|
||||
if (inventory.retainedBytes + bytes > NATIVE_JSON_BYTES - 4096) {
|
||||
inventory.unknown("Native declarations exceed the 4 MiB retained JSON allowance; remaining declarations are unknown.");
|
||||
return inventory.result();
|
||||
}
|
||||
StringBuilder descriptor = new StringBuilder((int) characters).append('(');
|
||||
for (CharSequence parameter : method.getParameterTypes()) descriptor.append(parameter);
|
||||
descriptor.append(')').append(returnType);
|
||||
Map<String, Object> row = Map.of("source_index", inventory.methods.size(),
|
||||
"class_name", className, "method_name", methodName, "descriptor", descriptor.toString(),
|
||||
"dex_entry", path, "dex_entry_index", entryIndex, "class_index", classIndex,
|
||||
"method_index", method.getMethodIndex());
|
||||
if (!inventory.retain(row, bytes)) return inventory.result();
|
||||
}
|
||||
}
|
||||
inventory.examinedClasses++;
|
||||
}
|
||||
inventory.examinedDexEntries++;
|
||||
} catch (IOException | RuntimeException failure) {
|
||||
if (failure instanceof java.util.concurrent.CancellationException) throw (RuntimeException) failure;
|
||||
inventory.unknown("Root DEX native declarations could not be fully read from " + path + ": "
|
||||
+ failure.getClass().getSimpleName()
|
||||
+ (failure.getMessage() == null ? "" : " (" + failure.getMessage() + ")"));
|
||||
}
|
||||
}
|
||||
} catch (IOException failure) {
|
||||
// A failed archive close must fail the operation so the provider retires
|
||||
// its owned JVM; do not cache a result with lost file-resource ownership.
|
||||
throw new IllegalArgumentException("APK ZIP native declaration inventory could not finish its reader", failure);
|
||||
}
|
||||
return inventory.result();
|
||||
}
|
||||
|
||||
/** Original archive identity, never a deobfuscated basename or name suffix. */
|
||||
private ManifestSnapshot manifest(JadxSession current) {
|
||||
if (manifestSession != current || manifestSnapshot == null) {
|
||||
@@ -270,7 +480,7 @@ class ReaJadxBridge {
|
||||
.map(Object::toString).collect(Collectors.joining(", "))
|
||||
+ "): " + method.getReturnType().toString();
|
||||
return new MethodSnapshot(name, rawName, shortId, List.copyOf(bodyAbsenceMarkers), Map.of(
|
||||
"name", name, "signature", signature,
|
||||
"name", name, "raw_name", rawName, "signature", signature,
|
||||
"dex_descriptor", descriptor,
|
||||
"is_constructor", method.isConstructor(), "def_pos", 0));
|
||||
}).toList();
|
||||
@@ -280,8 +490,11 @@ class ReaJadxBridge {
|
||||
List<Map<String, Object>> valuedFields = new ArrayList<>();
|
||||
boolean anyValue = false;
|
||||
for (FieldNode field : orderedFields) {
|
||||
// Raw DEX identity for mapping lookups; the display type can carry generics.
|
||||
Map<String, Object> entry = Map.of("name", field.getAlias(),
|
||||
"type", field.getType().toString(), "def_pos", 0);
|
||||
"raw_name", field.getFieldInfo().getName(),
|
||||
"type", field.getType().toString(),
|
||||
"dex_type", TypeGen.signature(field.getFieldInfo().getType()), "def_pos", 0);
|
||||
fields.add(entry);
|
||||
Map<String, Object> value = dexStaticValue(field);
|
||||
if (value == null) valuedFields.add(entry);
|
||||
@@ -293,10 +506,11 @@ class ReaJadxBridge {
|
||||
}
|
||||
}
|
||||
List<String> inner = node.getInnerClasses().stream().map(ClassNode::getFullName).toList();
|
||||
List<String> innerRaw = node.getInnerClasses().stream().map(ClassNode::getRawName).toList();
|
||||
Map<String, Object> summary = Map.of(
|
||||
"full_name", cls.getFullName(), "raw_name", cls.getRawName(), "name", cls.getName(),
|
||||
"method_count", methods.size(), "field_count", fields.size(),
|
||||
"inner_class_count", inner.size(), "inner_classes", inner,
|
||||
"inner_class_count", inner.size(), "inner_classes", inner, "inner_class_raw_names", innerRaw,
|
||||
"methods", methods.stream().map(MethodSnapshot::summary).toList(), "fields", List.copyOf(fields));
|
||||
// DEX static values are opt-in, so method and reference lookups keep the member-only reply.
|
||||
Map<String, Object> summaryWithValues = summary;
|
||||
@@ -470,6 +684,8 @@ class ReaJadxBridge {
|
||||
"available_processors", Runtime.getRuntime().availableProcessors(),
|
||||
"java_version", System.getProperty("java.version"),
|
||||
"metadata_scope", "parsed_members"));
|
||||
if (name.equals("rea_native_declarations"))
|
||||
return json(nativeDeclarations(Path.of(string(args, "path"))));
|
||||
JadxSession current = session();
|
||||
if (name.equals("get_app_info")) return json(manifest(current).summary());
|
||||
if (name.equals("get_android_manifest"))
|
||||
@@ -581,7 +797,7 @@ class ReaJadxBridge {
|
||||
SessionHolder holder = new SessionHolder(defaults);
|
||||
ServerCapabilities capabilities = new ServerCapabilities(
|
||||
new ServerCapabilities.Tools(null), null, null, null, null, null, null, Map.of());
|
||||
Server server = new Server(new Implementation("rea-jadx-bridge", "7", null, null, null),
|
||||
Server server = new Server(new Implementation("rea-jadx-bridge", "9", null, null, null),
|
||||
new ServerOptions(capabilities), "", ignored -> Unit.INSTANCE);
|
||||
SessionToolsKt.registerSessionTools(server, holder);
|
||||
ClassToolsKt.registerClassTools(server, holder);
|
||||
@@ -601,6 +817,11 @@ class ReaJadxBridge {
|
||||
new ToolSchema(null, new kotlinx.serialization.json.JsonObject(Map.of()), List.of(), null),
|
||||
"Observed worker JVM and metadata scope", null, null, null, null, null, null);
|
||||
server.addTool(diagnostics, (connection, request, continuation) -> bridge.handle("rea_jvm_status", request.getArguments()));
|
||||
registerBridgeTool(server, bridge, "rea_native_declarations",
|
||||
new ToolSchema(null, new kotlinx.serialization.json.JsonObject(Map.of("path",
|
||||
new kotlinx.serialization.json.JsonObject(Map.of("type",
|
||||
kotlinx.serialization.json.JsonElementKt.JsonPrimitive("string"))))), List.of("path"), null),
|
||||
"Original ACC_NATIVE declarations from the admitted APK's standard root DEX members, without loading or decompiling the APK.");
|
||||
for (String name : List.of("get_app_info", "get_android_manifest")) {
|
||||
String description = name.equals("get_app_info")
|
||||
? "Namespace-aware summary of the exact root AndroidManifest.xml resource."
|
||||
|
||||
@@ -286,6 +286,162 @@ instructions into field declarations, leaving an empty or incomplete method
|
||||
body. The selected method's disassembly preserves those instructions and retains
|
||||
the same source byte limit and explicit truncation reporting.
|
||||
|
||||
## Apply an R8/ProGuard mapping
|
||||
|
||||
Release builds shrunk by R8 or ProGuard rename classes and members. Select the
|
||||
build's `mapping.txt` with `mapping_path` on `inspect_android_class` or
|
||||
`inspect_android_method`, or `--mapping-path` on the matching CLI command:
|
||||
|
||||
```sh
|
||||
rea inspect-android-class /path/Example.apk a.b --mapping-path /path/mapping.txt \
|
||||
--detail summary --evidence_output /tmp/class-evidence.json
|
||||
```
|
||||
|
||||
For agent workflows, select `detail: "summary"` on class and method inspection.
|
||||
It preserves the complete normalized member inventory or selected method body,
|
||||
including mapping annotations, while retaining duplicate raw provider calls in
|
||||
the complete parent Evidence. The reply links that parent's identity and digest;
|
||||
export it with `export_evidence_bundle` or read its summary again through
|
||||
`inspect_analysis_view`. CLI summaries require `--evidence_output` and return
|
||||
the portable bundle path. Omitted or `complete` detail preserves complete inline
|
||||
Evidence. Search and reference operations retain their existing contracts.
|
||||
Symbol summary views admit at most **1 MiB**, including view context. An admitted
|
||||
method's source can already consume that amount; a larger combined summary
|
||||
fails explicitly instead of silently truncating facts. MCP refusal includes
|
||||
the retained parent ID. Select complete detail, or inspect one method rather
|
||||
than a whole class, when the requested summary does not fit.
|
||||
|
||||
The mapping path must be absolute. Selectors still take the observed names, such as
|
||||
`a.b`, and observed rows are unchanged. The result adds one `mapping` block with
|
||||
the file's path, SHA-256 and size, the header values the file reports
|
||||
(`compiler`, `compiler_version`, `min_api`, `map_version`, `pg_map_id`,
|
||||
`pg_map_hash`), valid record counts and up to 16 malformed line samples. Its
|
||||
`symbols` list has one entry per observed class, inner class, method and field;
|
||||
`inspect_android_method` lists the class and the selected method. `coverage`
|
||||
counts each resolution status. With summary delivery, these unchanged facts are
|
||||
under the view's `summary` field; the view separately reports source coverage.
|
||||
|
||||
Lookups use the bridge's raw DEX names, not JADX display aliases such as a
|
||||
renamed reserved word. Each symbol resolves as:
|
||||
|
||||
- `mapped`: exactly one record matches. A class needs one record for its raw
|
||||
name (`verified_by: "class"`). A method must also match its DEX prototype
|
||||
(`"descriptor"`) and a field its DEX type (`"type"`). R8 residual signature
|
||||
metadata is used when present; otherwise original types are translated
|
||||
through the file's class records.
|
||||
- `ambiguous`: `multiple-records`, `descriptor-mismatch`, `type-mismatch`,
|
||||
`ambiguous-class` or `residual-signature-conflict`, with `candidate_count` and
|
||||
up to 8 candidates. Conflicting residual metadata never selects a winner. A name match
|
||||
whose prototype or type does not verify is never reported as mapped.
|
||||
- `unmapped`: no valid record matched the raw identity.
|
||||
- `unknown` with reason `projection-budget`: this claim did not fit the combined
|
||||
reply budget. The observed raw symbol remains present; `coverage.unknown`
|
||||
counts these omissions. Inspect that specific class or method separately,
|
||||
or consult the selected local mapping file for complete source records.
|
||||
|
||||
A line continues an inline frame stack when it repeats the previous line's
|
||||
minified range and obfuscated name and gives a single call-site line, such as
|
||||
`:7`. Only the outermost original of a stack becomes a candidate. Original
|
||||
line numbers and source files are not projected. Malformed records are counted
|
||||
and skipped, so an `unmapped` symbol might be described by one of them. A file
|
||||
that is missing, not a regular file, not UTF-8, larger than the byte limit or
|
||||
without any class record, such as `usage.txt` or `seeds.txt`, is refused before
|
||||
JADX is used. Requests read their mappings one at a time, in the same queue as
|
||||
the JADX work and under the same operation deadline. A mapping failure therefore never starts or retires the JADX
|
||||
session, and a different mapping reuses the loaded APK.
|
||||
|
||||
## Link native declarations to JNI candidates
|
||||
|
||||
When advertised, `inspect_android_jni_bindings` links original `ACC_NATIVE`
|
||||
declarations from standard root `classes*.dex` members to bundled ELF libraries.
|
||||
The supplied JAR's DEX reader preserves duplicate declarations before decompiler
|
||||
deduplication. This operation does not load the APK into the decompiler.
|
||||
MCP defaults to retained summary delivery when `detail` is omitted; select
|
||||
`complete` explicitly to receive the full report inline.
|
||||
|
||||
```json
|
||||
{
|
||||
"path": "/absolute/Example.apk",
|
||||
"class_name": "org.conscrypt.NativeCrypto",
|
||||
"method_names": ["EVP_PKEY_new_RSA"],
|
||||
"abis": ["arm64-v8a"],
|
||||
"detail": "summary"
|
||||
}
|
||||
```
|
||||
|
||||
Class/method names and optional `descriptors` select raw DEX identities.
|
||||
`library_paths` selects exact logical APK paths such as
|
||||
`lib/arm64-v8a/libconscrypt_jni.so`; each library's `source_path` preserves its
|
||||
original ZIP spelling. Omitted filters include all scoped facts; `[]` matches
|
||||
none. Class selection never narrows the global inventory used for table-class
|
||||
attribution.
|
||||
|
||||
Independent `declarations`, `libraries`, `candidates` and `bindings` collections
|
||||
have stable IDs and source ordinals. Candidates retain APK/library digests,
|
||||
ABI, function addresses and file offsets. Table candidates also retain section,
|
||||
table and entry identity plus all three name/signature/function pointer slots
|
||||
and their relocation sources. JNI exports follow the
|
||||
[UTF-16 escaping and short-before-long rules](https://docs.oracle.com/en/java/javase/21/docs/specs/jni/design.html#resolving-native-method-names);
|
||||
failed escaping does not acquire an invented export match. Tables require
|
||||
NUL-terminated [Modified UTF-8 strings](https://docs.oracle.com/en/java/javase/21/docs/specs/jni/types.html#modified-utf-8-strings),
|
||||
valid descriptors and executable load mappings. REL, RELA and RELR retain
|
||||
separate provenance; unsupported packed relocations and unresolved pointer
|
||||
spans stay explicit unknowns.
|
||||
|
||||
A static `JNINativeMethod` entry names no class. Attribute it uniquely only
|
||||
when exactly one declaration in the complete recorded native inventory has
|
||||
that `(name, descriptor)` pair. Multiple classes remain ambiguous after a
|
||||
class filter; a partial inventory cannot establish unique attribution.
|
||||
|
||||
Each selected declaration has a row per selected/observed ABI. `matched` means
|
||||
candidate presence was observed; `ambiguous` means multiple observed targets
|
||||
or class associations remain. Binding `coverage` is separate: a matched
|
||||
partial row does not establish complete uniqueness, and an empty partial scan
|
||||
is `unknown`, never `unmatched`. ELF/ABI label mismatches preserve facts without
|
||||
asserting a binding. No result establishes library loading, runtime registration
|
||||
or method invocation.
|
||||
|
||||
Summary returns the first 64 selected binding rows, their declaration/candidate
|
||||
closure and every selected library's identity, coverage and failure reason;
|
||||
counts and page coverage describe the full selected binding set. Selected
|
||||
declaration IDs have separate preview coverage, so native declarations remain
|
||||
visible even when the APK has no bundled ABI libraries. Complete
|
||||
Evidence stays retained. Read exact fields or further pages without reopening
|
||||
files or rerunning analysis:
|
||||
|
||||
```json
|
||||
{
|
||||
"source": { "kind": "retained-evidence", "evidence_id": "ev_<digest>" },
|
||||
"view": {
|
||||
"kind": "android-jni",
|
||||
"collection": "bindings",
|
||||
"offset": 64,
|
||||
"limit": 64,
|
||||
"fields": ["declaration_id", "abi", "status", "coverage", "candidate_ids"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Collections also include `declarations`, `candidates` and `libraries`, with
|
||||
exact filters and batch lookup by declaration/candidate/library IDs. `row_index`
|
||||
is the original parent collection index; DEX and
|
||||
ZIP source ordinals stay separate. Complete detail returns the full report.
|
||||
CLI uses the same selection and delivery contracts. Its common one-shot
|
||||
default remains complete; summary delivery requires an explicit Evidence path:
|
||||
|
||||
```sh
|
||||
rea inspect-android-jni-bindings /absolute/Example.apk \
|
||||
--query '{"class_name":"org.conscrypt.NativeCrypto","method_names":["EVP_PKEY_new_RSA"],"abis":["arm64-v8a"]}' \
|
||||
--detail summary --evidence_output /absolute/fresh-jni-evidence.json
|
||||
```
|
||||
|
||||
Optional `--view` accepts the same JSON page and requires summary delivery.
|
||||
Evidence bundles never overwrite files. Complete declarations and one library
|
||||
selection are cached within the immutable APK session. Changing ABI/library
|
||||
selection replaces the scan; class/method selection reuses it. Fresh analysis
|
||||
still validates current caller APK bytes. Idle/close releases the caches;
|
||||
retained Evidence remains historical and independently readable.
|
||||
|
||||
## Interpret the result
|
||||
|
||||
- Package inspection retains decoded manifest XML, summary declarations,
|
||||
@@ -322,6 +478,11 @@ the same source byte limit and explicit truncation reporting.
|
||||
`dex_descriptor`, for example `(I)V`. Display names can be JADX aliases;
|
||||
the descriptor alone is not a complete raw DEX method ID. Overload indices
|
||||
are tied to this artifact and engine, not stable identities across builds.
|
||||
- A `mapping` block appears only when the caller selected `mapping_path`. Its
|
||||
names are claims from that file, not observations of the APK. REA does not
|
||||
check that the mapping belongs to this build; `pg_map_id` is reported, not
|
||||
compared. Evidence keeps the selected path in its parameters and the file's
|
||||
SHA-256 in the result.
|
||||
- Method inspection reports Java or smali, fallback markers and complete/partial
|
||||
returned text. Decompiled source is a derived representation. A complete text
|
||||
result does not establish complete semantics or a byte-identical reconstruction.
|
||||
@@ -411,6 +572,19 @@ target/configuration changes, failure, timeout, cancellation, MCP disconnect or
|
||||
server shutdown. One-shot CLI commands always join cleanup before returning.
|
||||
REA never writes to the original APK or launches target code.
|
||||
|
||||
JNI scans reuse the analyzer process without loading the APK into the
|
||||
decompiler. They admit at most 64 MiB per DEX member and 256 MiB of DEX bytes,
|
||||
with a 4 MiB native-declaration JSON allowance before MCP string escaping.
|
||||
Native libraries are processed sequentially: one 64 MiB buffer, 256 MiB total
|
||||
selected read work, and a 16 MiB parsed-library-facts allowance. ZIP inventory
|
||||
and library counts also have explicit resource bounds. Unexamined occurrences
|
||||
remain unknown. A complete relationship report has a separate 16 MiB serialized
|
||||
allowance; summaries/pages allow 1 MiB including their provenance envelope.
|
||||
These are resource admissions, not claims about maximum valid Android formats.
|
||||
Select fewer ABI/library occurrences or smaller retained pages when needed.
|
||||
Completed parsed library facts survive failed reader finalization in typed
|
||||
partial observations; the exact reader owner remains available for cleanup retry.
|
||||
|
||||
On Windows, stdin writes preserve stream backpressure and run outside the Node
|
||||
event loop. Cancellation terminates the retained job, including descendants,
|
||||
before workspace cleanup. On Windows, a detached REA helper owns the private
|
||||
@@ -496,6 +670,27 @@ complete, case-sensitive search contract. A protocol frame above **8 MiB**, or c
|
||||
**32 MiB per operation**, fails with a diagnostic instead of returning an incomplete inventory
|
||||
as complete. These are memory-safety boundaries, not undocumented result caps.
|
||||
|
||||
A selected R8/ProGuard mapping is re-read and SHA-256 checked on every call, up
|
||||
to **64 MiB**. The serialized provider keeps one successful parsed index, keyed
|
||||
by byte length and digest with fixed parser options. Identical bytes reuse that
|
||||
index; changed bytes are reparsed, missing files still fail, and each response
|
||||
preserves the currently selected path. The index is released on provider idle
|
||||
retirement or close. Omitting `mapping_path` never applies the cached mapping.
|
||||
During replacement, the old index can coexist with the new parse until it
|
||||
succeeds; these transient allocations count toward process memory.
|
||||
|
||||
Before retaining records, the parser admits an estimated **128 MiB** index
|
||||
representation and **8 MiB** of expanded mapped claims per residual class,
|
||||
including shared header facts. The index allowance accounts for strings and
|
||||
containers; it is not a measured V8 heap cap. Parameter and metadata expansion
|
||||
is admitted before splitting or JSON decoding. These fixed resource bounds
|
||||
have no runtime override; refusal reports `resource_constraint`, the source
|
||||
line and mapping SHA-256. Ambiguous lookups count all candidates while keeping
|
||||
only the first eight in stable file order. A separate **8 MiB** combined claim
|
||||
budget applies to the actual query set, including outer and inner classes,
|
||||
before cloning the report. Claims that do not fit become explicit `unknown`
|
||||
entries; other admitted claims and all observed identities remain available.
|
||||
|
||||
## Real verification
|
||||
|
||||
Download the explicit public fixtures once; this installs no tools:
|
||||
|
||||
@@ -75,7 +75,7 @@ graph TD
|
||||
end
|
||||
|
||||
subgraph "Domain Layer"
|
||||
DOMAIN["src/domain/<br/>Pure side-effect-free modules<br/>format declarations, Evidence, comparisons, application graphs"]
|
||||
DOMAIN["src/domain/<br/>Pure side-effect-free modules<br/>format declarations, provider selection, Evidence, comparisons, application graphs"]
|
||||
end
|
||||
|
||||
subgraph "Configuration"
|
||||
|
||||
@@ -497,17 +497,51 @@ comments and unrelated settings. It resolves that file from `HERMES_HOME`, else
|
||||
from the platform default Hermes itself uses — `%LOCALAPPDATA%\hermes` on
|
||||
Windows, `~/.hermes` elsewhere — including any `HERMES_DATA_DIR_SUFFIX`. Hermes
|
||||
connects an entry unless `enabled` is false, so setup states `enabled: true`.
|
||||
Aliases whose anchor is defined in the `rea` entry retain their original resolved
|
||||
values when setup refreshes or uninstall removes that entry. Setup does not
|
||||
redirect another server's alias to REA's new launcher or policy.
|
||||
This also covers an anchor on the removed registration key. References to
|
||||
definitions outside the entry keep sharing the original value, including when
|
||||
another definition later reuses the same anchor name.
|
||||
|
||||
Hermes personal skills install under the active Hermes home's `skills` directory.
|
||||
Setup, doctor and uninstall follow the root's `active_profile` selection, as
|
||||
plain Hermes CLI commands do. An explicit `HERMES_HOME` pointing to
|
||||
`profiles/<name>` selects that named profile directly. An invalid, missing or
|
||||
deleted sticky profile is reported instead of writing into the default home;
|
||||
it also remains an unresolved skill selection, so setup cannot report a
|
||||
successful empty installation. Overrides nested beneath the native Hermes home
|
||||
use that native home's sticky selection and profile root together, matching
|
||||
[Hermes's root rules](https://github.com/NousResearch/hermes-agent/blob/748c79a1c6873124d84071cdabb4062a75913e1f/hermes_constants.py#L83).
|
||||
Repair the profile or run `hermes profile use default` before retrying.
|
||||
Setup, doctor and uninstall share REA's sticky-profile resolver.
|
||||
A custom `HERMES_HOME` outside the platform default home
|
||||
reads its own `active_profile`. An override physically inside the default home
|
||||
uses that default home's sticky selection and profile root together, including
|
||||
when the final override directory does not yet exist.
|
||||
|
||||
For example, with `HERMES_HOME=~/.hermes/sub` and `~/.hermes/active_profile` set
|
||||
to `coder`, setup selects `~/.hermes/profiles/coder/config.yaml` and that
|
||||
profile's skills directory. It does not read `~/.hermes/sub/active_profile`.
|
||||
Without an outer named selection, this override uses `~/.hermes/sub/config.yaml`.
|
||||
Use a home outside the platform default tree for an independent sticky selection.
|
||||
|
||||
This describes REA's current routing, not full Hermes CLI equivalence.
|
||||
At Hermes commit `748c79a1`, the
|
||||
[sticky lookup](https://github.com/NousResearch/hermes-agent/blob/748c79a1c6873124d84071cdabb4062a75913e1f/hermes_cli/main.py#L612-L632)
|
||||
also reads the outer selection, but
|
||||
[`resolve_profile_env`](https://github.com/NousResearch/hermes-agent/blob/748c79a1c6873124d84071cdabb4062a75913e1f/hermes_cli/profiles.py#L2431-L2461)
|
||||
resolves it beneath the configured `HERMES_HOME`. For nested overrides, confirm
|
||||
the profile Hermes actually uses and point `HERMES_HOME` directly at that
|
||||
`profiles/<name>` directory before setup. Compatibility is tracked in
|
||||
[#3939](https://github.com/morluto/rea/issues/3939).
|
||||
|
||||
REA resolves existing ancestor directories before checking physical containment,
|
||||
including symlinks and junctions, even when the override directory does not exist.
|
||||
An override beneath a symlinked default home therefore uses the default home's
|
||||
sticky selection. If `~/.hermes/link` points outside the default home,
|
||||
`HERMES_HOME=~/.hermes/link/new` instead uses that override root, whether or not
|
||||
`new` exists. Previewed destinations retain the configured path spelling.
|
||||
An unreadable path or dangling alias leaves the home unresolved; repair that path
|
||||
before configuring Hermes. Inspect setup's previewed destinations before applying
|
||||
changes.
|
||||
|
||||
An explicit `HERMES_HOME` pointing to `profiles/<name>` selects that named
|
||||
profile directly, bypassing sticky selection. An invalid, missing or deleted
|
||||
sticky profile is reported instead of writing into the default home; repair the
|
||||
profile or run `hermes profile use default` before retrying. It also remains an
|
||||
unresolved skill selection, so setup cannot report a successful empty installation.
|
||||
REA follows `HERMES_HOME` whitespace trimming, environment-variable expansion,
|
||||
and home expansion before resolving that profile. Setup previews the resolved
|
||||
configuration and skill destinations; doctor and uninstall use those same paths.
|
||||
@@ -524,10 +558,12 @@ installs its skill copy there.
|
||||
## Factory Droid
|
||||
|
||||
For [Factory Droid](https://docs.factory.com/harness/mcp), setup writes
|
||||
`mcpServers.rea` into the user-level `~/.factory/mcp.json`. Entries carry
|
||||
`command`, `args`, and `env`. Droid discovers personal skills from
|
||||
`~/.factory/skills` ([skills](https://docs.factory.com/harness/skills)), so
|
||||
setup installs its skill copy there.
|
||||
`mcpServers.rea` into the user-level `~/.factory/mcp.json` (the
|
||||
`FACTORY_HOME_OVERRIDE` override replaces the home that contains `.factory`).
|
||||
Entries carry `command`, `args`, and `env`. Droid discovers personal skills
|
||||
from `~/.factory/skills` ([skills](https://docs.factory.com/harness/skills),
|
||||
following `FACTORY_HOME_OVERRIDE` when set), so setup installs its skill copy
|
||||
there.
|
||||
|
||||
## Trae
|
||||
|
||||
@@ -703,10 +739,6 @@ can override an inherited disable. Lower files and other server entries remain
|
||||
unchanged; a legacy server named `servers` that conflicts with the native table
|
||||
requires manual repair before setup can write.
|
||||
|
||||
When Hermes aliases refer to an anchored REA entry, refreshing or removing REA
|
||||
retains the aliases' original resolved values. Setup does not redirect another
|
||||
server's alias to REA's new launcher or policy.
|
||||
|
||||
With no client selected, the interactive skill choice targets the shared
|
||||
`~/.agents/skills` copy. Selecting a client targets its personal destination
|
||||
when supported. Use `rea setup --skill-client shared --skill` for the shared
|
||||
|
||||
@@ -36,7 +36,7 @@ The MCP tool `project_harmony_application_graph` takes the same
|
||||
|
||||
- exact component paths, artifact IDs, and hashes (manifests, Ark bytecode,
|
||||
resources, native libraries, JavaScript, signing)
|
||||
- a packaging-model hint: Stage is claimed only from a `module.json` entry
|
||||
- a packaging-model hint: Stage is claimed only from an exactly cased `module.json` entry
|
||||
outside a classified resource path, FA only from `config.json` outside a
|
||||
classified resource path, otherwise `unknown`
|
||||
- FA resources under `assets/<module>/resources/` as well as Stage `resources/`
|
||||
@@ -46,6 +46,9 @@ The MCP tool `project_harmony_application_graph` takes the same
|
||||
- path-based N-API bridge hypotheses between bytecode and `libs/<abi>` native
|
||||
libraries
|
||||
|
||||
An unavailable manifest can still supply a path-based model hint; its content
|
||||
identity remains unknown. Manifest names follow case-sensitive ZIP member names.
|
||||
|
||||
File and slice occurrences supply component and child-package paths; directory
|
||||
trees and symlinks retain their identities in the source inventory without
|
||||
being classified as code files. Ark and JavaScript runtime hints summarize the
|
||||
|
||||
@@ -240,7 +240,7 @@ The verifier requires the resolved procedure in both inventory and procedure sea
|
||||
|
||||
The initial real workflows use stdio registrations for the legacy Windows GUI profile and a Windows x64 headless supervisor with IDA 9.3. The headless workflow has also passed on macOS arm64 with IDA Professional 9.1 (build 9.1.250226) against the same upstream commit `c133c3853faa111a9b00ee615c013b720d0c4acd`, using a source-built arm64 Mach-O fixture; Mach-O C symbols carry a leading underscore, so the procedure name is `_rea_fixture_add`. Streamable HTTP is tested against a loopback MCP fixture with the pinned client SDK; a real IDA HTTP-supervisor workflow remains unverified. Linux headless engine operation, other IDA releases, architectures, and modern attached GUI tools require their corresponding real workflows before support is claimed.
|
||||
|
||||
Native Windows verification uses IDA Pro 9.3 with the public two-function fixture and Windows `whoami.exe`, using upstream `c9d15d98e8a143fc74f6e14037c48f63889b6d6c` with a loopback-listener change. It covers production CLI/MCP parity, distinct-function switching, identity and connection cleanup. See the [correctness and performance measurements](https://github.com/morluto/rea/blob/main/experiments/analysis-efficiency/ida-native-windows.md). This follow-up did not run Windows IDA Free or native macOS.
|
||||
Native Windows verification uses IDA Pro 9.3 with the public two-function fixture and Windows `whoami.exe`, using upstream `c9d15d98e8a143fc74f6e14037c48f63889b6d6c` with a loopback-listener change. It covers production CLI/MCP parity, distinct-function switching, identity and connection cleanup. See the [correctness and performance measurements](https://github.com/morluto/rea/blob/a2e2c29b59e8775b0e6aedb09a9eb16adf32646b/experiments/analysis-efficiency/ida-native-windows.md). This follow-up did not run Windows IDA Free or native macOS.
|
||||
|
||||
Headless readiness distinguishes an explicit `auto_analysis_ready: false` from
|
||||
unknown readiness (`null`). The former returns a retryable capability refusal
|
||||
|
||||
+107
-7
@@ -3,13 +3,14 @@
|
||||
Run REA, connect your selected agents, and check the tools needed for your task.
|
||||
Setup previews configuration changes and asks for approval before applying them.
|
||||
|
||||
| Task | Section |
|
||||
| --------------------------------- | --------------------------------------------------------------------------------------------------- |
|
||||
| Connect an agent | [Start setup](#start-setup), then [review changes](#review-setup-changes) |
|
||||
| Diagnose one task | [Scoped readiness checks](#check-readiness-for-your-task) |
|
||||
| Configure an analysis engine | [Hopper](#hopper), [Ghidra](#ghidra), the [IDA guide](ida-provider.md), or [Kuna](kuna-provider.md) |
|
||||
| Update or remove REA | [Maintenance](#diagnose-update-and-remove) |
|
||||
| Resolve a client-specific problem | [Client configuration](client-configuration.md) |
|
||||
| Task | Section |
|
||||
| ---------------------------------- | --------------------------------------------------------------------------------------------------- |
|
||||
| Connect an agent | [Start setup](#start-setup), then [review changes](#review-setup-changes) |
|
||||
| Install a Claude Code/Codex plugin | [Agent plugins and marketplaces](#agent-plugins-and-marketplaces) |
|
||||
| Diagnose one task | [Scoped readiness checks](#check-readiness-for-your-task) |
|
||||
| Configure an analysis engine | [Hopper](#hopper), [Ghidra](#ghidra), the [IDA guide](ida-provider.md), or [Kuna](kuna-provider.md) |
|
||||
| Update or remove REA | [Maintenance](#diagnose-update-and-remove) |
|
||||
| Resolve a client-specific problem | [Client configuration](client-configuration.md) |
|
||||
|
||||
## Start setup
|
||||
|
||||
@@ -444,6 +445,9 @@ and other MCP servers. Purging removes REA's legacy cache and state under
|
||||
archives under `${XDG_CACHE_HOME:-~/.cache}/rea/hopper-releases`. Other cache
|
||||
namespaces remain intact. A caller-selected `REA_KUNA_CACHE_DIR` is retained
|
||||
and reported because REA cannot establish ownership of its entire contents.
|
||||
Lexically equivalent spellings of that default directory, including a trailing
|
||||
separator, a `.` segment, or a repeated separator, share its purge result.
|
||||
The comparison preserves POSIX `..` traversal and literal backslashes.
|
||||
A client configuration that is malformed, unreadable, or at an unsafe
|
||||
path stops the operation before anything is removed, and a client that fails
|
||||
while being updated stops the remaining removals. Uninstall reports each
|
||||
@@ -1433,6 +1437,21 @@ current setting. For Windows PE resource analysis, the exact name is
|
||||
`WindowsResourceReference`. Disabling an analyzer skips the facts it would have
|
||||
produced.
|
||||
|
||||
Ghidra's Objective-C Message Analyzer can log
|
||||
`Required Objective-C data type not found in data type manager` /
|
||||
`Try adding libobjc.dylib` when a Mach-O links Objective-C runtime symbols but
|
||||
defines no class metadata (thin Cocoa glue in a C++ binary is a common case).
|
||||
That text is non-fatal Ghidra analyzer output: auto-analysis continues, and the
|
||||
analyzer simply skips `objc_msgSend` call-target recovery. REA's headless import
|
||||
does not accept a "add this library" option, and on modern macOS
|
||||
`/usr/lib/libobjc.A.dylib` exists only in the dyld shared cache, so there is no
|
||||
on-disk library to add. Treat the line as a coverage limitation, not an REA
|
||||
failure; check the session's `failure_kind` (and related provider errors) for
|
||||
any real abort. To silence that analyzer, set
|
||||
`REA_GHIDRA_DISABLED_ANALYZERS="Objective-C Message Analyzer"`. For large
|
||||
executables that miss the startup deadline instead, raise
|
||||
`REA_GHIDRA_STARTUP_TIMEOUT_MS` and the [Ghidra heap](#ghidra-heap-and-cpu-controls).
|
||||
|
||||
### Ghidra heap and CPU controls
|
||||
|
||||
Set resource controls in the environment that launches the CLI or MCP server,
|
||||
@@ -1534,6 +1553,87 @@ read-only operations, digest identity, and runtime cleanup. The separate
|
||||
ordinary user. It requires the built Windows native bundle and an existing
|
||||
Ghidra/JDK installation; neither is inferred from startup alone.
|
||||
|
||||
## Agent plugins and marketplaces
|
||||
|
||||
REA's npm package is the shared Claude Code and Codex plugin artifact. It
|
||||
contains `.claude-plugin/plugin.json`, `.codex-plugin/plugin.json`, and the
|
||||
canonical generated `skills/` tree. Both manifests select `./skills` and run a plain Node bootstrap pinned to
|
||||
that exact package version. The client must have a supported standalone `node`
|
||||
on its PATH. Claude expands its documented `CLAUDE_PLUGIN_ROOT` in the script
|
||||
argument; Codex resolves `cwd: "."` from the installed plugin root before
|
||||
launching the relative script. The bootstrap verifies the package version and runs its bundled runtime in
|
||||
the same process, preserving stdio and REA's provider/worker cleanup. It does
|
||||
not start another package runner or download a different runtime. The existing release skill
|
||||
generator also pins CLI recovery commands to the same version.
|
||||
|
||||
The repository's single-plugin marketplaces are declared in
|
||||
`.claude-plugin/marketplace.json` and `.agents/plugins/marketplace.json`.
|
||||
Both select an exact `rea-agents` npm version instead of mutable source-checkout
|
||||
contents. Release Please updates their selectors alongside the plugin manifests.
|
||||
No plugin-only publication or generated skill files committed to Git are needed.
|
||||
The qualified release archive contains a runtime-only manifest and
|
||||
`npm-shrinkwrap.json` plus bundled runtime dependencies, including PTY packages
|
||||
for each supported OS and CPU; development dependencies and development-only overrides
|
||||
stay in the source manifest and lockfile. Both clients receive this same archive.
|
||||
|
||||
This distribution becomes installable after the first qualified npm release
|
||||
containing these plugin manifests. Earlier published packages are not changed
|
||||
by this source proposal. For local source development, run `npm ci && npm run build:cached` to generate
|
||||
the runtime and packaged skill before loading the repository as a plugin.
|
||||
A source plugin uses the locally built runtime and generated skill; rebuild after
|
||||
source changes. Missing builds and mismatched version pins fail with an actionable
|
||||
error. The authored `.agents/skills/` tree is not the plugin's shipped skill layout.
|
||||
Windows source workflows that need native controls still require the
|
||||
[Windows source-checkout build](https://github.com/morluto/rea/blob/main/native/windows/README.md#build).
|
||||
|
||||
In Claude Code:
|
||||
|
||||
```text
|
||||
/plugin marketplace add morluto/rea
|
||||
/plugin install rea@rea
|
||||
```
|
||||
|
||||
In Codex CLI, add the marketplace and install REA:
|
||||
|
||||
```bash
|
||||
codex plugin marketplace add morluto/rea
|
||||
codex plugin add rea@rea
|
||||
```
|
||||
|
||||
Each release updates both manifest versions and their pinned server version
|
||||
together, which is when installed copies refresh the bundled skill and move to
|
||||
the new server. A plugin does not run `rea setup`, so provider
|
||||
configuration still follows the sections above. Use the plugin or a
|
||||
`rea setup` registration for a client, not both, to avoid two `rea` servers.
|
||||
|
||||
`rea doctor` does not see plugin-owned state. Its client and skill checks read
|
||||
only the registrations and skill locations that `rea setup` manages, so for a
|
||||
plugin-only install `rea doctor --client <id>` and `rea doctor --skill` report
|
||||
`missing` and recommend `rea setup`. Do not follow that remediation. Check the
|
||||
plugin with the client instead (`claude plugin list` and `/mcp` in Claude
|
||||
Code, `codex plugin list` in Codex).
|
||||
|
||||
For analysis engines, ask the connected server: have the agent call
|
||||
`binary_session` and read `result.tool_availability`, which reports what the
|
||||
server the plugin started can use, with reasons and remediation. See
|
||||
[MCP contracts](mcp-contracts.md).
|
||||
|
||||
A plugin does not put `rea` on your `PATH`. To check an engine from a
|
||||
terminal, use the package runner at the version the plugin pins:
|
||||
|
||||
<!-- x-release-please-start-version -->
|
||||
|
||||
```bash
|
||||
npx -y [email protected] doctor --provider ghidra --json
|
||||
```
|
||||
|
||||
<!-- x-release-please-end -->
|
||||
|
||||
If your installed plugin is older, use the version in its manifest instead.
|
||||
That report describes the terminal's environment, not the plugin server's.
|
||||
The server inherits the client's environment, so a setting such as
|
||||
`GHIDRA_INSTALL_DIR` that exists only in your terminal does not reach it.
|
||||
|
||||
## MCP Registry
|
||||
|
||||
REA is published in the official MCP Registry as `io.github.morluto/rea`. Registry
|
||||
|
||||
+13
-2
@@ -1248,11 +1248,22 @@ failures for such entries name this cause and the operation to rerun with
|
||||
Advertised input and output schemas contain no reachable recursive references.
|
||||
Input schemas inline repeated definitions before applying the selected
|
||||
presentation profile. Both `full` and `compact` start from this self-contained
|
||||
projection; output schemas can share schema-local definitions. Full inputs
|
||||
retain nested fields, descriptions and valid literal examples. The compatibility
|
||||
projection; output schemas can share schema-local definitions. Analyst-authored
|
||||
full inputs retain nested fields, descriptions and valid literal examples. The compatibility
|
||||
projections below can relax advertised constraints; canonical runtime validation
|
||||
still enforces the complete input contract.
|
||||
|
||||
`compare_web_captures` declares its complete producer observations as object
|
||||
operands instead of repeating their full output schemas in the input catalog.
|
||||
Pass the original `inspect_web_page` normalized result in `before.inspection`
|
||||
and `after.inspection`, optional complete `discover_webmcp_tools` results in
|
||||
`webmcp`, or complete `capture_browser_scenario` results in the scenario fields.
|
||||
Do not construct partial observations from the abbreviated declaration: the
|
||||
server still validates every canonical field before comparison. Input groups
|
||||
and analyst-authored normalization rules remain explicit. This presentation
|
||||
changes no capture bytes, identity, completeness, or comparison semantics and
|
||||
does not reduce `inspect_analysis_view` selectors or other tools' declarations.
|
||||
|
||||
When a host forwards input schemas to the model, their examples consume context.
|
||||
REA's advertised examples therefore never copy complete producer Evidence.
|
||||
An embedded Evidence record is shown as its exact `{"kind":"retained-evidence","evidence_id":"ev_..."}`
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
@@ -1,112 +0,0 @@
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"repository": "morluto/rea",
|
||||
"version": "6.7.0",
|
||||
"base": "2c4338db1fdb7773e05376301c558cff2a0b4b80",
|
||||
"target": "b83b9caf4a5f0241a5fd370792f1aed1bffbb2cf",
|
||||
"commits": [
|
||||
{
|
||||
"sha": "8cbaf6c7021b97dd4de6b5771c35fbed06b240e9",
|
||||
"subject": "fix(tests): isolate command finalization module mocks",
|
||||
"body": "fix(tests): isolate command finalization module mocks\n\nMove the five registered WASM finalization cases into the existing per-file isolated project. Preserve all result, exit-code, observation and cleanup assertions. A preloaded shared CLI graph reproduces three failures before the move; the isolated covered packet passes.\n\nFixes #4959\n",
|
||||
"parents": ["2c4338db1fdb7773e05376301c558cff2a0b4b80"],
|
||||
"authors": [
|
||||
{
|
||||
"name": "morluto",
|
||||
"email": "[email protected]",
|
||||
"user": {
|
||||
"login": "morluto",
|
||||
"__typename": "User"
|
||||
}
|
||||
}
|
||||
],
|
||||
"pullRequests": []
|
||||
},
|
||||
{
|
||||
"sha": "6b180783ed04ebc87548c8385c80e05725804c47",
|
||||
"subject": "fix(tests): budget CLI coverage from parsed options",
|
||||
"body": "fix(tests): budget CLI coverage from parsed options\n\nUse the pinned Vitest parser for coverage and shard flags. Keep ordinary CLI workers at 384 MiB and covered workers at the tested 512 MiB with unchanged concurrency. Retain complete coverage and add actual-config regressions for false, provider-only and spaced-shard options.\n\nValidation: 27 focused cases; 16 covered cases with CLI preloaded; 5 docs checks; typecheck/lint; exact discovery of 1578 files. Measured covered control: 22.18s elapsed, 46.52s CPU, about 2.33 GiB sampled summed process-family RSS on macOS arm64. Full Linux CI pending. No runtime/device verification claimed.\n\nFixes #4960\n",
|
||||
"parents": ["8cbaf6c7021b97dd4de6b5771c35fbed06b240e9"],
|
||||
"authors": [
|
||||
{
|
||||
"name": "morluto",
|
||||
"email": "[email protected]",
|
||||
"user": {
|
||||
"login": "morluto",
|
||||
"__typename": "User"
|
||||
}
|
||||
}
|
||||
],
|
||||
"pullRequests": []
|
||||
},
|
||||
{
|
||||
"sha": "bb212ffecb47e129cc569b9867b2b6e7c9b65685",
|
||||
"subject": "fix(tests): inherit server identity deadline policy",
|
||||
"body": "fix(tests): inherit server identity deadline policy\n\nRemove per-case ten-second overrides from MCP catalog and identity checks. Keep the suite-owned bounded ordinary and coverage deadlines, all assertions, and session cleanup.\n\nFixes #4967\n",
|
||||
"parents": ["6b180783ed04ebc87548c8385c80e05725804c47"],
|
||||
"authors": [
|
||||
{
|
||||
"name": "morluto",
|
||||
"email": "[email protected]",
|
||||
"user": {
|
||||
"login": "morluto",
|
||||
"__typename": "User"
|
||||
}
|
||||
}
|
||||
],
|
||||
"pullRequests": []
|
||||
},
|
||||
{
|
||||
"sha": "b83b9caf4a5f0241a5fd370792f1aed1bffbb2cf",
|
||||
"subject": "fix(setup): edit the duplicate JSON member that clients read (#4957)",
|
||||
"body": "fix(setup): edit the duplicate JSON member that clients read (#4957)\n\nJSON.parse, JSONC parse, and native Claude Code keep the last duplicate\nobject member, and parseClientConfiguration therefore validates the last\none. The shared setup/uninstall serializer instead edited text through\njsonc-parser's modify and findNodeAtLocation, which select the first\nmatching member, so setup and uninstall on configurations with repeated\nserver tables or rea members wrote into a shadowed member and failed\nreadback.\n\nLocate every JSON text edit by the effective (last) member in one place,\nreplacing the first-member modify path. Removing an effective member\nthat shadows an earlier duplicate would activate that duplicate, whatever\nits ownership, so the serializer now returns a tagged shadowed_member\nfailure and setup and uninstall refuse before writing a backup or the\nconfiguration.\n\nFixes #4952\n\nCo-authored-by: Claude Opus 5.5 <[email protected]>",
|
||||
"parents": ["bb212ffecb47e129cc569b9867b2b6e7c9b65685"],
|
||||
"authors": [
|
||||
{
|
||||
"name": "Kaoru0822",
|
||||
"email": "[email protected]",
|
||||
"user": {
|
||||
"login": "kaoru0822-kitauji",
|
||||
"__typename": "User"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Claude Opus 5.5",
|
||||
"email": "[email protected]",
|
||||
"user": {
|
||||
"login": "claude",
|
||||
"__typename": "User"
|
||||
}
|
||||
}
|
||||
],
|
||||
"pullRequests": [4957]
|
||||
}
|
||||
],
|
||||
"pullRequests": [
|
||||
{
|
||||
"number": 4957,
|
||||
"title": "fix(setup): edit the duplicate JSON member that clients read",
|
||||
"url": "https://github.com/morluto/rea/pull/4957",
|
||||
"author": {
|
||||
"login": "kaoru0822-kitauji",
|
||||
"__typename": "User"
|
||||
},
|
||||
"mergeCommit": {
|
||||
"oid": "b83b9caf4a5f0241a5fd370792f1aed1bffbb2cf"
|
||||
},
|
||||
"issues": [
|
||||
{
|
||||
"number": 4952,
|
||||
"url": "https://github.com/morluto/rea/issues/4952",
|
||||
"author": {
|
||||
"login": "xu91102",
|
||||
"__typename": "User"
|
||||
},
|
||||
"repository": {
|
||||
"nameWithOwner": "morluto/rea"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
+19
-1
@@ -349,7 +349,10 @@ tag remains in main's ancestry. Keep the released tag unchanged.
|
||||
|
||||
The synchronization must include `.release-please-manifest.json`,
|
||||
`package.json`, both root versions in `package-lock.json`, `CHANGELOG.md`,
|
||||
`server.json`, and the versioned documentation examples. Run
|
||||
`server.json`, both plugin manifests (`.claude-plugin/plugin.json` and
|
||||
`.codex-plugin/plugin.json`), the Codex server declaration (`.codex-plugin/mcp.json`), both
|
||||
marketplaces (`.claude-plugin/marketplace.json` and
|
||||
`.agents/plugins/marketplace.json`), and the versioned documentation examples. Run
|
||||
`npm run docs:generate`, `npm run docs:check`, and the release configuration
|
||||
tests against the combined tree. After the merge, verify that the released
|
||||
tag is an ancestor of main:
|
||||
@@ -383,3 +386,18 @@ tag. Do not dispatch a fresh publish phase to repair an already-created
|
||||
release: Release Please will not create the same release again. Do not move
|
||||
the tag, delete the release, or unpublish npm as a retry. A defective public
|
||||
package requires a reviewed correction and a new version.
|
||||
|
||||
### Plugin artifact metadata
|
||||
|
||||
Package verification selects files with npm, then packs an owned staging directory.
|
||||
The release manifest omits development dependencies and development-only overrides;
|
||||
its generated `npm-shrinkwrap.json` preserves the runtime dependency resolutions
|
||||
and integrity values from `package-lock.json`. Runtime or unresolved overrides
|
||||
fail packaging rather than being dropped. Source metadata is never rewritten.
|
||||
The archive bundles production dependencies because Codex fetches npm plugins
|
||||
without installing dependencies. The producer includes all locked PTY platform
|
||||
archives after checking their integrity, without re-resolving the runtime tree.
|
||||
Optional Frida stays outside the bundle so ordinary npm installation can retain
|
||||
its host-specific installer. This also lets Claude install locked dependencies under its plugin policy while
|
||||
retaining development security overrides. Both plugin marketplaces select the
|
||||
same qualified npm artifact, which the release workflow publishes unchanged.
|
||||
+20
-5
@@ -5,6 +5,12 @@ lane. Start with the commands below; use [real-toolchain lanes](#real-toolchain-
|
||||
for engine prerequisites and [client verification](client-verification.md) for
|
||||
installed-client and agent runs.
|
||||
|
||||
Historical verification records may refer to `experiments/analysis-efficiency`.
|
||||
Those experiment scripts are available in the [archived source tree](https://github.com/morluto/rea/tree/a2e2c29b59e8775b0e6aedb09a9eb16adf32646b/experiments/analysis-efficiency);
|
||||
they are not included in current checkouts. The retained [test support](https://github.com/morluto/rea/blob/main/tests/support/analysis-efficiency/README.md)
|
||||
contains deterministic checkers and fixtures. The [allocation report](verification/analysis-efficiency/mcp-integer-allocation.md)
|
||||
and its original receipt JSON remain documentation, checked by the report producer.
|
||||
|
||||
## Developer commands
|
||||
|
||||
Use source feedback while editing, explicit boundary checks for the changed
|
||||
@@ -204,7 +210,11 @@ changing a callback solely to satisfy its declared `void` return type.
|
||||
Ordinary Knip remains. `check`, `check:pr`, and `check:ci:deep` run reachability
|
||||
then typed lint sequentially after cached work. Typed lint builds the project
|
||||
type graph and stays separate from fast feedback. Its default Go settings are
|
||||
`GOMAXPROCS=2` and `GOMEMLIMIT=1024MiB`; caller-selected settings take precedence.
|
||||
`GOMAXPROCS=1` and `GOMEMLIMIT=1024MiB`; caller-selected settings take precedence.
|
||||
One Go worker reuses checker caches across the file queue without changing the
|
||||
file inventory or TypeScript project scope. The Go memory target is soft and
|
||||
does not cap the complete process family's memory; measure the full invocation
|
||||
when enforcing a host budget.
|
||||
|
||||
Routine iterations and rebases need focused regressions and relevant checks.
|
||||
Before handing off a PR, run `npm run check` and generated-document checks when
|
||||
@@ -1164,8 +1174,8 @@ its metadata regressions against the same checksum-pinned engine and public APK.
|
||||
It runs on branch pushes affecting its bridge, fixture or engine inputs, and can
|
||||
also be dispatched manually or called by another workflow. It uses the runner's
|
||||
JDK with two active processors and a 512 MiB heap; it needs no npm dependency
|
||||
installation. This lane checks exact raw/display class selectors and the existing
|
||||
cold-metadata regressions. It does not establish CLI/MCP parity or application
|
||||
installation. This lane checks exact raw/display class selectors, raw member names and field
|
||||
DEX types used by mapping lookups, and the existing cold-metadata regressions. It does not establish CLI/MCP parity or application
|
||||
execution behavior. The broader `verify:android` lane checks CLI/MCP parity;
|
||||
neither lane runs the APK.
|
||||
|
||||
@@ -1211,7 +1221,7 @@ HarmonyOS `.har` libraries are deliberately not suffix-classified because
|
||||
Synthetic producer regressions run independently:
|
||||
|
||||
```sh
|
||||
npm run test:focused -- tests/boundary/process/jadxIntegration.test.ts tests/boundary/process/androidAnalysisMcp.test.ts
|
||||
npm run test:focused -- tests/boundary/process/jadxIntegration.test.ts tests/boundary/process/androidAnalysisMcp.test.ts tests/boundary/process/androidMappingMcp.test.ts
|
||||
```
|
||||
|
||||
## Apktool resource decoding
|
||||
@@ -1374,7 +1384,7 @@ other engine versions and architectures remain unverified; see the
|
||||
|
||||
Native IDA MCP verification uses `"protocol": "native"` with an already-running native plugin and requires `--second-procedure other_function --second-marker unique_second_body_text` to exercise GUI function switching and a return to the first function. These options remain optional for other profiles. The literal marker must appear in the second function's pseudocode and be absent from the first. Native cloud/widget failures are reported without hidden retries.
|
||||
|
||||
The initial Linux x64 IDA Free 9.4 run used native MCP 1.1.2 with a local navigation correction. Windows x64 verification uses **IDA Pro 9.3**, the two-function public fixture, and the shipped Windows `whoami.exe` application. The native plugin was built from upstream `c9d15d98e8a143fc74f6e14037c48f63889b6d6c`, including the merged navigation fix, with a local loopback-listener change. Production CLI/MCP parity, switching, identity and cleanup passed. Correctness and performance measurements are recorded in the [Windows native verification notes](https://github.com/morluto/rea/blob/main/experiments/analysis-efficiency/ida-native-windows.md). Stock 1.1.2 switching, native macOS, and Windows IDA Free remain unverified. See the [native bridge prerequisites](ida-provider.md#optional-native-ida-mcp-bridge).
|
||||
The initial Linux x64 IDA Free 9.4 run used native MCP 1.1.2 with a local navigation correction. Windows x64 verification uses **IDA Pro 9.3**, the two-function public fixture, and the shipped Windows `whoami.exe` application. The native plugin was built from upstream `c9d15d98e8a143fc74f6e14037c48f63889b6d6c`, including the merged navigation fix, with a local loopback-listener change. Production CLI/MCP parity, switching, identity and cleanup passed. Correctness and performance measurements are recorded in the [Windows native verification notes](https://github.com/morluto/rea/blob/a2e2c29b59e8775b0e6aedb09a9eb16adf32646b/experiments/analysis-efficiency/ida-native-windows.md). Stock 1.1.2 switching, native macOS, and Windows IDA Free remain unverified. See the [native bridge prerequisites](ida-provider.md#optional-native-ida-mcp-bridge).
|
||||
|
||||
## Kuna lane
|
||||
|
||||
@@ -1624,6 +1634,11 @@ results after a failure. Available failed-shard reports remain diagnostic only:
|
||||
failed, cancelled, skipped, or missing selected shards cannot pass the gate or
|
||||
publish successful timing history. This deliberately avoids relying on an
|
||||
import graph for generated files, filesystem inputs, subprocesses, or bridges.
|
||||
The portable report rejects an entirely skipped module inventory and lists
|
||||
fully skipped modules in its summary. A mixed passed/skipped inventory is
|
||||
allowed: Vitest's skip state does not establish whether a skipped module was
|
||||
required on that runner. Review the listed skips against platform and provider
|
||||
prerequisites; the inventory alone does not prove every required case ran.
|
||||
|
||||
Scheduled qualification is admitted hourly at minute 23. A forced full run at
|
||||
03:23 UTC refreshes runner and dependency drift every day. Other hourly runs
|
||||
|
||||
File renamed without changes.
+11
-4
@@ -22,7 +22,7 @@ Seven dependency-boundary variants match parsed values and complete error issues
|
||||
|
||||
## Tasks, artifacts and bounds
|
||||
|
||||
Reuse the shared SDK task driver and independent ELF/section/zlib/JSON oracles from [the catalog study](mcp-input-catalog.md): official Alpine v3.21 x86_64 ripgrep 14.1.1-r0 and Zellij 0.40.0-r0. All 42/422 packages and 88/1,063 edges match the independent oracles. Select serde name/version/kind, repeat exact retained-parent reads twice, verify a canonical negative-offset refusal, export/check complete Evidence and confirm the unchanged directory. `kind: null` stays unknown. Targets are not executed; dependency presence does not establish runtime use. Acquisition URLs, archive/executable/oracle hashes and complete answers are in [the receipts](mcp-integer-allocation.json). Previously acquired public bytes are rehashed, not redownloaded.
|
||||
Reuse the shared SDK task driver and independent ELF/section/zlib/JSON oracles from [the archived catalog study](https://github.com/morluto/rea/blob/a2e2c29b59e8775b0e6aedb09a9eb16adf32646b/experiments/analysis-efficiency/mcp-input-catalog.md): official Alpine v3.21 x86_64 ripgrep 14.1.1-r0 and Zellij 0.40.0-r0. All 42/422 packages and 88/1,063 edges match the independent oracles. Select serde name/version/kind, repeat exact retained-parent reads twice, verify a canonical negative-offset refusal, export/check complete Evidence and confirm the unchanged directory. `kind: null` stays unknown. Targets are not executed; dependency presence does not establish runtime use. Acquisition URLs, archive/executable/oracle hashes and complete answers are in [the receipts](https://github.com/morluto/rea/blob/main/docs/verification/analysis-efficiency/mcp-integer-allocation.json). Previously acquired public bytes are rehashed, not redownloaded.
|
||||
|
||||
Normal cohort: three alternating baseline/candidate pairs for each task × cold/warm server compile cache × default compact/full inputs: 72 trials, plus four excluded full-task warm priming runs. Separate diagnostic cohort: three alternating pairs per profile, cold Zellij tasks, 12 trials. All pass, with identical complete parent IDs/hashes and answers.
|
||||
|
||||
@@ -115,6 +115,12 @@ Question, explicit recipe, server instructions, one complete name/description/in
|
||||
- Both runtime builds and all six static checks plus architecture guards pass. Before the final main synchronization, 5,972 source/contract/public CLI/MCP tests pass across 323 files; four skip for unavailable/guarded coverage. After synchronization, both builds/static checks repeat successfully, 77 affected producer and CLI/MCP tests pass, and all 974 registered numeric-parameter parser cases match again. Includes real SQLite CLI/MCP record/column selection and JavaScript heap/deadline preservation. A first direct schema-validity invocation lacked generated catalog metadata; the supported focused entry generates it and all 15 cases pass. A later affected-test invocation selected two nonexistent paths; the corrected three-file producer/consumer run is the 77-case result above. Neither setup failure was counted as a passing test, and no production or timeout change was needed.
|
||||
- All 84 measured tasks and four warm priming tasks pass independent oracles, canonical refusal, parent/directory identity and owned cleanup checks. This does not verify external Hopper/Ghidra/IDA, Android devices, all providers or foreign OS behavior. The earlier macOS numbers in #4360 describe a different revision/platform and are not combined with these results.
|
||||
|
||||
The historical cohort commands below use the retired experiment layout. Run them
|
||||
from the matching recorded producer checkout; the [archived source tree](https://github.com/morluto/rea/tree/a2e2c29b59e8775b0e6aedb09a9eb16adf32646b/experiments/analysis-efficiency)
|
||||
provides the former files for reference. Current checkouts retain the report and
|
||||
its conformance producer, rather than the full measurement suite. Moving these
|
||||
files does not requalify the historical measurements against current REA.
|
||||
|
||||
Build the pinned baseline/candidate in separate clean worktrees with the locked toolchain. Use the candidate’s producer scripts for both roles. `measurement_inputs` in the JSON records the complete matrix, questions/recipes, runtime bindings and producer hashes; extract it unchanged as the bindings file. Historical receipts require their matching producer revision. One normal task:
|
||||
|
||||
```bash
|
||||
@@ -133,11 +139,12 @@ python experiments/analysis-efficiency/mcp-startup-costs.py "$RECEIPT_ROOT" \
|
||||
|
||||
The aggregator requires the recorded tiktoken version, exact catalog/parent/runtime/producer/guard identities, clean source, successful payload cleanup and matching matrix coverage. Default driver behavior continues without an inspector; the heap flag is confined to the experiment.
|
||||
|
||||
Regenerate or verify every numerical section directly from the committed JSON:
|
||||
In a current checkout, regenerate or verify every numerical section directly
|
||||
from the committed JSON:
|
||||
|
||||
```bash
|
||||
node experiments/analysis-efficiency/mcp-integer-report.mjs
|
||||
node experiments/analysis-efficiency/mcp-integer-report.mjs --check
|
||||
node tests/support/analysis-efficiency/mcp-integer-report.mjs
|
||||
node tests/support/analysis-efficiency/mcp-integer-report.mjs --check
|
||||
```
|
||||
|
||||
The documentation conformance gate checks current receipts, rejects stale normal/heap tables, and refuses missing producer markers without writing.
|
||||
@@ -0,0 +1,320 @@
|
||||
# Static Android JNI verification, 2026-10-27
|
||||
|
||||
Issue #6403 asks which bundled native functions could implement original DEX
|
||||
native declarations. The implementation reads immutable APK bytes, preserving
|
||||
declarations, library occurrences, exported/table candidates and selected
|
||||
per-ABI bindings as independently queryable facts. No target, library,
|
||||
`JNI_OnLoad` or `RegisterNatives` code was executed.
|
||||
|
||||
## Revisions and independent oracles
|
||||
|
||||
The full qualification and alternating delivery measurements below use
|
||||
implementation `4a1839de6f091ade19e82557874c358ed34b8e36`, based on main
|
||||
`03f17cf41d3c844f9383d7792d9b1a14330d0bb2`. Subsequent merge
|
||||
`e8a8a50725cd384ae38ce66a94bafdaba1138108` incorporates main
|
||||
`d6f88becd531cabafca1d968b89cf7fc77a80041`; its browser input-presentation and
|
||||
SQLite test changes are separate from JNI analysis. Post-merge confirmation is
|
||||
recorded below. The native parser's built SHA-256 is
|
||||
`3bf0a2c342f1c404f5861b7c913f7f1b931f740a8c808b518474fc8de20821d5`.
|
||||
|
||||
Linux x86-64, Node 24.22.1/npm 11.16.0 and JDK 21.0.12.1 qualified the workflows.
|
||||
JADX MCP 0.7.1 remains the supplied unmodified JAR, SHA-256
|
||||
`6e5eacf500b64292bfb73c49797c1958f6ee44646e43e868039ae7feb573ff75`.
|
||||
The REA metadata bridge is version 9. JNI declarations use the JAR's existing
|
||||
dexlib2 reader before decompiler deduplication, with no `load_apk` call.
|
||||
|
||||
The genuine Android NDK r27d package is revision 27.3.13750724, clang/LLD 18.0.4.
|
||||
Its official Linux ZIP is 663,956,036 bytes, publisher SHA-1
|
||||
`22105e410cf29afcf163760cc95522b9fb981121`; observed SHA-256 is
|
||||
`601246087a682d1944e1e16dd85bc6e49560fe8b6d61255be2829178c8ed15d9`.
|
||||
Pins come from the [official repository manifest](https://dl.google.com/android/repository/repository2-3.xml).
|
||||
The package was streamed, hashed and selectively extracted into a private task
|
||||
cache, without installing or changing global tools. The selected compiler,
|
||||
sysroot and four additional official libunwind archives total 319,933,537 bytes.
|
||||
An initial link failure exposed those omitted archives; the corrected fixture
|
||||
build passed in 4.606 seconds, CPU 6.026 seconds, peak family RSS 110.4 MiB.
|
||||
|
||||
[Committed source fixtures and reproduction](https://github.com/morluto/rea/blob/main/tests/fixtures/android/jni-bindings/README.md)
|
||||
build native declarations, javac headers and two libraries for arm64-v8a,
|
||||
armeabi-v7a, x86 and x86_64. Ordinary REL/RELA, RELR and Android packed APS2
|
||||
variants include overloads, BMP/supplementary names, explicit Modified UTF-8,
|
||||
`.data` and `.data.rel.ro` tables, repeated exports in two libraries, an
|
||||
unmatched declaration and two classes sharing one registered pair. Source,
|
||||
`javac -h`, independent DEX flag/index reads, readelf and a separate byte/relocation
|
||||
oracle establish expected facts. Oracle SHA-256 is
|
||||
`fa634493bc8559a80eda3b70f01d44550c2f06e176cfc2a7cc7e10794b1eb793`.
|
||||
|
||||
| Static APK | SHA-256 |
|
||||
| -------------------- | ------------------------------------------------------------------ |
|
||||
| Ordinary | `c4eb50ef8bdecc6f56c454294c0c8f198fa51a4174d101dc720a849976546649` |
|
||||
| RELR | `5a2ce56191f633575e87bd239702f988d0d1ad82795a203e13a4f907f8624658` |
|
||||
| APS2 | `d2f0d613bc8f22ed1a6435f13b5ae1f088e904216aa8a04dd70c27c5cb606778` |
|
||||
| Conscrypt derivative | `5985c40d5a6b14b64c202c80335d851adb7e8a987e6154a1488a194253856385` |
|
||||
|
||||
The Conscrypt derivative packages the public Android 2.8.0 AAR's original
|
||||
library bytes with its class files transformed by D8 8.0.40. It is a static
|
||||
test container, not a vendor APK or a claimed launchable application. The AAR
|
||||
SHA-256 is `752044c289bc98e2dcabc00a1f88ca158b3aa3f505b38f124552bf4b2a0902b1`.
|
||||
The original-name DEX is used, not an R8-renamed DEX paired with unchanged JNI
|
||||
strings. D8's missing platform-adapter dependency warning remains a qualification
|
||||
limit; no full application linkage or signature verification is claimed.
|
||||
|
||||
## Executed correctness and Evidence checks
|
||||
|
||||
The built pure parser passed independent checks over 25 real libraries:
|
||||
24 genuine NDK outputs plus Conscrypt arm64. All 33 golden table entries match
|
||||
their strings, slot addresses/file offsets, relocated values, encodings,
|
||||
function addresses and code offsets. Conscrypt yields 367 structural entries;
|
||||
the requested `EVP_PKEY_new_RSA ([B[B[B[B[B[B[B[B)J` candidate is `0xfa11c`.
|
||||
Its name/signature/function slots are `0x2ada68`, `0x2ada70`, `0x2ada78`,
|
||||
with relocated values `0x749a8`, `0x7affa`, `0xfa11c`. This built-artifact
|
||||
qualification took 0.684 seconds, CPU 0.736 seconds, peak family RSS 99.5 MiB.
|
||||
|
||||
Real SDK/stdio MCP and CLI checks independently validate:
|
||||
|
||||
- All 12 source native declarations across four ABIs, exact library digests,
|
||||
overloads, UTF-16 names and per-library alternatives. Ordinary/RELR return
|
||||
32 matched, 12 ambiguous and four unknown rows. The source-known unmatched
|
||||
method remains unknown because symbolic writes/NOBITS leave scanning partial.
|
||||
- APS2 preserves 20 observed export matches, four ambiguous export rows and
|
||||
24 unknown rows. It explicitly reports unsupported table relocation encoding,
|
||||
retaining each selected library's reason even without a positive candidate.
|
||||
- Class selection never hides the other class sharing a registered pair.
|
||||
All selected native declarations survive independently of available ABI rows;
|
||||
the no-library selection case is covered by a domain regression, not claimed
|
||||
as a separate real SDK artifact qualification.
|
||||
- All four retained collections support exact fields, empty filters and batch
|
||||
ID retrieval. Parent-array `row_index` remains separate from raw DEX/ZIP
|
||||
source ordinals. Advertised converted schemas validate with the real SDK/Ajv.
|
||||
- Complete portable parents preserve the same normalized facts through summary
|
||||
and CLI delivery. Ordinary and each Conscrypt cohort copy the source to an
|
||||
exclusively created owned file, inspect it, remove only that copy, then read
|
||||
retained summary, exact table slots, binding fields and a full export.
|
||||
Original fixture bytes remain unchanged. Post-removal success is observed;
|
||||
hidden internal engine activity is not directly instrumented by this harness.
|
||||
|
||||
The first three four-ABI cohorts execute 107 SDK calls and three cold CLI parity
|
||||
checks. Each Conscrypt cohort adds 28 investigation calls, five separately
|
||||
accounted retained-removal calls and one CLI parity check. Four alternating
|
||||
Conscrypt cohorts preserve the same normalized fact digest:
|
||||
`7e4738d2dd9e518277d5268ce5c334d9d25728198f66b2102cdc3411865f2d46`.
|
||||
|
||||
Raw parent transcripts show one native inventory RPC on the first investigation,
|
||||
then only JVM-status RPCs on later main queries; normalized native facts remain
|
||||
present. No JNI parent contains `load_apk`. Library reuse is owned by the same
|
||||
immutable APK session; ABI/library selection replaces one scan, while class
|
||||
and method selection reuses it. Fresh analysis admission still hashes current
|
||||
caller APK bytes. Historical views never reopen them.
|
||||
|
||||
Type/lint/build and 396 tests across ten exact files pass together: native/domain
|
||||
projection, producer parsing, real ZIP occurrence/cleanup boundaries, unchanged
|
||||
JADX session/idle/cancellation behavior, public contracts and existing IL2CPP
|
||||
decoding/ownership. The gate took 215.198 seconds, CPU 257.470 seconds, peak
|
||||
family RSS 4396.1 MiB. A full real JDK/JADX metadata run additionally passed
|
||||
native reads before any APK load, malformed/duplicate DEX and CRC negatives,
|
||||
raw aliases, constants, absence facts, manifest identity and downstream CLI/MCP
|
||||
bodies/direct-invoke parity from both DEX entries: 154.890 seconds, CPU 233.275
|
||||
seconds, peak family RSS 984.8 MiB.
|
||||
|
||||
Finalization regressions retain already parsed library facts/digests in typed
|
||||
partial observations before ZIP close. Caller/deadline-first cancellation,
|
||||
failed close, blocked admission and retry preserve their first cause and exact
|
||||
resource lease. Interrupted later libraries preserve earlier completed facts.
|
||||
Stable table candidate identity uses physical pointer-slot address/file offset,
|
||||
not inferred run grouping. Known ZIP size claims stay separate from verified
|
||||
read size/hash.
|
||||
|
||||
## Completed-task costs
|
||||
|
||||
Earlier main cannot decode/register JNI tables with equivalent coverage, so
|
||||
manual export lookup is not an equivalent before workflow. The controlled
|
||||
comparison is complete versus summary delivery of the same new investigation,
|
||||
same bytes, query, selected pages and proof export. Conscrypt trials alternate
|
||||
summary/complete/summary/complete, each with one cold and one warm task.
|
||||
|
||||
Each fixture task has four investigations, 12 selected collection pages and a
|
||||
proof export (17 calls). Each Conscrypt task has one investigation, the same
|
||||
12 page checks and proof export (14 calls). Task time includes public calls and
|
||||
local oracle checks; one-time startup/catalog, separate retained-removal
|
||||
qualification, final cleanup and a cold CLI parity run are included in the
|
||||
whole-process columns, not hidden in task timings. Ordinary includes removal
|
||||
qualification; RELR/APS2 do not, so their whole-process costs are not a paired
|
||||
performance comparison. All four Conscrypt cohorts use the same phase flags.
|
||||
|
||||
| Cohort | Cold task s | Warm task s | Whole s | CPU s | Peak family MiB | Warm text tokens | Warm SDK tokens |
|
||||
| -------------------- | ----------: | ----------: | ------: | -----: | --------------: | ---------------: | --------------: |
|
||||
| Ordinary | 11.058 | 0.606 | 32.658 | 45.124 | 1215.7 | 58,319 | 401,681 |
|
||||
| RELR | 12.140 | 0.797 | 29.122 | 38.375 | 1266.2 | 58,231 | 401,502 |
|
||||
| APS2 | 12.578 | 0.917 | 30.892 | 41.071 | 1268.0 | 50,900 | 386,634 |
|
||||
| Conscrypt summary | 11.237 | 0.444 | 33.999 | 46.953 | 1282.0 | 32,265 | 349,128 |
|
||||
| Conscrypt complete | 10.846 | 0.456 | 32.907 | 45.948 | 1398.3 | 224,954 | 737,723 |
|
||||
| Conscrypt summary 2 | 19.205 | 1.200 | 57.545 | 77.644 | 1456.2 | 32,223 | 349,055 |
|
||||
| Conscrypt complete 2 | 20.978 | 0.925 | 64.513 | 81.218 | 1502.0 | 224,526 | 736,950 |
|
||||
|
||||
Warm text estimates are about 85.7% lower for summary delivery. Cold text
|
||||
estimates are 32,087/32,202 versus 247,593/247,912. SDK-inclusive warm estimates
|
||||
are about 52.6% lower. Latency/CPU/peak-memory samples vary substantially and do
|
||||
not establish a general improvement. Cold costs include the first generic
|
||||
view-schema compilation in the client; subsequent tasks reuse it. Startup/
|
||||
catalog alone ranges 2.692–4.955 seconds across these Conscrypt cohorts.
|
||||
|
||||
Token estimates use tiktoken 0.14.0/o200k_base. Every task counts its analyst
|
||||
question, discovered input catalog once, arguments and all text replies,
|
||||
including pages and export receipts. SDK totals additionally count output
|
||||
schemas and structured-content duplication. Qualifications and CLI response
|
||||
costs are reported separately by the cost script. Exported file contents are
|
||||
verified locally and are not counted as model context unless actually returned
|
||||
inline. Actual billing, hidden prompts, client rendering/pruning and OS cache
|
||||
state are unmeasured.
|
||||
|
||||
### Latest-main confirmation
|
||||
|
||||
After merging main `d6f88becd` into `e8a8a5072`, type/lint/build and the eight
|
||||
input-presentation/public-contract cases pass. The first direct schema-suite
|
||||
invocation could not load its generated test catalog; `test:focused` supplied
|
||||
the required artifacts and all 16 schema tests then passed (63.310 seconds,
|
||||
CPU 72.830 seconds, peak family RSS 1943.5 MiB). No schema assertion was relaxed.
|
||||
|
||||
Fresh full Conscrypt SDK/CLI cohorts use the same proof, 12 pages and owned-copy
|
||||
removal flags. Both retain the identical fact digest above:
|
||||
|
||||
| Delivery | Cold task s | Warm task s | Whole s | CPU s | Peak family MiB | Warm text tokens | Warm SDK tokens |
|
||||
| -------- | ----------: | ----------: | ------: | -----: | --------------: | ---------------: | --------------: |
|
||||
| Summary | 11.739 | 0.680 | 36.712 | 51.209 | 1395.1 | 31,932 | 348,522 |
|
||||
| Complete | 11.161 | 0.459 | 34.692 | 48.319 | 1385.2 | 224,954 | 737,723 |
|
||||
|
||||
Summary text estimates remain about 85.8% lower. This pair does not show a
|
||||
latency or peak-memory improvement. Each adds 33 SDK calls and one cold CLI
|
||||
parity check, bringing the executed public qualification to 305 SDK calls and
|
||||
nine CLI checks across the pinned cohorts.
|
||||
|
||||
`docs:check`, `format:check` and `docs:build` pass together: 44.365 seconds,
|
||||
CPU 56.935 seconds, peak sampled family RSS 1300.6 MiB. Generated catalogs,
|
||||
schemas, packaged skills and provider binaries remain uncommitted outputs.
|
||||
|
||||
All expensive checks use CPU affinity 0,1, bounded heaps and one active process
|
||||
family, with an 8 GiB RSS guard and 4 GiB host-availability guard. Peak memory is
|
||||
the sum of family RSS sampled every 100 ms: shared pages can be double-counted
|
||||
and short peaks can be missed. No guard stop or OOM occurred.
|
||||
|
||||
### Default-summary delivery confirmation
|
||||
|
||||
The new MCP operation now defaults to retained `summary` when `detail` is omitted.
|
||||
Explicit `complete` remains supported; the CLI retains its common explicit-detail
|
||||
policy. The actual SDK regression verifies both modes, caller selectors and
|
||||
reading an unselected declaration from the complete parent after source removal.
|
||||
Type/lint/build and all five cases in that MCP boundary file pass: 80.012 seconds,
|
||||
CPU 110.240 seconds, peak sampled family RSS 4747.6 MiB. An earlier attempt stopped
|
||||
at the test-function length lint rule; the fixture helper was extracted before
|
||||
this successful gate.
|
||||
|
||||
A fresh genuine Conscrypt workflow omits `detail` on JNI investigation calls and
|
||||
passes 33 SDK calls, all 12 retained pages, proof export, one cold CLI parity
|
||||
check and the separate owned-copy removal qualification. The same investigation
|
||||
fact digest above is retained. Cold/warm task times are 15.608/0.481 seconds;
|
||||
whole process-family time is 45.239 seconds, CPU 61.038 seconds, peak sampled RSS
|
||||
1431.2 MiB. Warm text/SDK token estimates are 31,928/348,516 using the same cost
|
||||
method. This adds a tenth CLI check and brings the successful public qualification
|
||||
total to 338 SDK calls. Explicit-complete removal qualification remains separate
|
||||
from the investigation task; default summaries do not discard the full parent.
|
||||
|
||||
Final `docs:check`, `format:check` and `docs:build` also pass: 77.269 seconds,
|
||||
CPU 70.506 seconds, peak sampled family RSS 1191.1 MiB.
|
||||
|
||||
The updated harness adds `--omit-detail` (with `--detail summary`), SHA-256
|
||||
`c2a5d090f32429c852e3c974083bb9c21a0dc6a608407687031f155a722b53d7`.
|
||||
The earlier cohort tables use the original frozen harness below. Default-run
|
||||
summary/catalog/call receipt SHA-256 values are respectively
|
||||
`c10e4d702b9694b0ccc69926a4e1f688fa8dea33a3de2e2837ea15190a706b37`,
|
||||
`0c36244f5ff788fabd5491144b0711e99ad211ce3c09e1812d1d4da5753624ac`, and
|
||||
`e1f021f0a7a92f101a037a1467a3428b4c620482eb556d81501a5ea64226ab82`.
|
||||
|
||||
### Review corrections and final qualification
|
||||
|
||||
Four review findings were reproduced or confirmed at their producing boundaries:
|
||||
|
||||
- The shared relocation reader had charged every file-backed write against a
|
||||
sparse retention limit. Its single immutable decoder now streams writes;
|
||||
IL2CPP applies them to its existing private copy and retains only bounded
|
||||
unresolved/zero-fill state and a file-offset bitmap. JNI keeps its bounded
|
||||
sparse provenance index. The two actual IL2CPP registration-read regressions
|
||||
each include 65,537 extra file-backed relative writes (AArch64 and x86-64)
|
||||
and verify the original bytes remain unchanged.
|
||||
- GNU version headers could replay shared auxiliary chains and names. The shared
|
||||
export owner now bounds total record visits from section bytes and the existing
|
||||
export-row allowance, caches decoded/failed string offsets within 16 MiB, and
|
||||
yields checkpoints through the same generator used by synchronous callers.
|
||||
Refusal retains exports with partial coverage and unknown version names.
|
||||
Both GNU table variants exercise 10,000 headers replaying 65,535 auxiliary
|
||||
records and a 65,535-byte name, verifying one decode and cooperative yields.
|
||||
- A named mapped-string read-limit failure now distinguishes budget exhaustion
|
||||
from invalid text. JNI preserves later candidates, marks tables partial and
|
||||
projects absent candidates as unknown. Both long names and descriptors are
|
||||
covered. An additional actual JNI-consumer regression queues cancellation
|
||||
after GNU name decoding begins and stops before export-symbol decoding.
|
||||
- The static verifier now requires the Conscrypt oracle only when expectations
|
||||
advertise it. Executed controls pass with 25 libraries/33 entries when present,
|
||||
and 24 libraries/32 entries for NDK-only expectations; an advertised missing
|
||||
oracle rejects with ENOENT and writes no success receipt.
|
||||
|
||||
Type/lint/build and 330 cases in 13 related domain, parser, archive, IL2CPP and
|
||||
actual CLI/stdio MCP files pass together: 192.437 seconds, CPU 224.376 seconds,
|
||||
peak sampled family RSS 4190.2 MiB. After adding the during-version cancellation
|
||||
case, type/lint and all 38 JNI parser cases pass: 39.842 seconds, CPU 67.581
|
||||
seconds, peak RSS 3928.0 MiB. These overlap earlier gates; counts are not summed
|
||||
as unique tests.
|
||||
|
||||
The final built parser again passes the independent 25-library/33-entry oracle:
|
||||
0.975 seconds, CPU 1.009 seconds, peak RSS 103.9 MiB. Built parser SHA-256 is
|
||||
`7c0638751427b2b282b8e55bb21c95f6c3f05c246b0ea66a31d0280694f9ffeb`;
|
||||
its receipt also fingerprints every shared parsing dependency. The fixed oracle
|
||||
SHA-256 above is unchanged. Large-capacity IL2CPP verification uses valid producer
|
||||
fixtures, not a newly qualified real IL2CPP application.
|
||||
|
||||
A fresh Conscrypt investigation qualification again passes 33 SDK calls,
|
||||
12 pages, proof, a cold CLI check and owned-copy removal. Investigation facts
|
||||
retain the same digest above. Cold/warm task times are 23.529/1.015 seconds;
|
||||
whole time 66.284 seconds, CPU 77.940 seconds, peak RSS 1402.6 MiB. Warm text/SDK
|
||||
estimates are 31,955/348,569. Host/time variation does not establish a general
|
||||
latency improvement. Cumulative successful public qualification is now 371 SDK
|
||||
calls and eleven CLI parity checks, across the explicitly separated revisions.
|
||||
The final cohort summary/calls SHA-256 values are
|
||||
`84df4f4ed97f6d1a931010db8e16420e5f06df371940ed9225eb99a7a7c01094` and
|
||||
`2c22b89e5ec4b26f15f0e9c5f8fdfd4af8583ad0926ad32edb359ecd991cd7fd`;
|
||||
its catalog hash matches the default-summary cohort. No resource guard or OOM
|
||||
occurred. Final `docs:check`, `format:check` and `docs:build` pass in
|
||||
38.602 seconds, CPU 49.448 seconds, peak sampled family RSS 1286.2 MiB.
|
||||
|
||||
## Reproduction and limitations
|
||||
|
||||
Build the runtime, generate the source fixtures using selected existing JDK,
|
||||
NDK and D8 paths, then run the maintained
|
||||
[pure parser verifier](https://github.com/morluto/rea/blob/main/tests/support/analysis-efficiency/android-jni-library.mjs)
|
||||
against the built parser. The
|
||||
[public workflow harness](https://github.com/morluto/rea/blob/main/tests/support/analysis-efficiency/android-jni.mjs)
|
||||
accepts `--variant ordinary|relr|aps2|conscrypt`, `--detail complete|summary`,
|
||||
`--proof --pages --cli` and optional `--retained-removal`, explicit selected
|
||||
repository/artifact/JDK/JAR/oracle paths and a fresh output directory. Frozen
|
||||
harness SHA-256:
|
||||
`14836895659de0f161c479b2b013ec39d754d172fd9935898872ab9cb5937035`.
|
||||
The [cost script](https://github.com/morluto/rea/blob/main/tests/support/analysis-efficiency/android-jni-costs.py)
|
||||
consumes every case's catalog, calls and summary receipts, fingerprints them
|
||||
and keeps qualification costs separate.
|
||||
|
||||
Initial source integration type/lint errors were corrected before qualification.
|
||||
A first CLI verifier attempt incorrectly treated its summary wrapper as flat
|
||||
Evidence; the CLI had written its valid bundle. The corrected harness saves raw
|
||||
stdout/stderr before assertions and validates the unwrapped named contract.
|
||||
One driver used the wrong Conscrypt derivative filename and failed before
|
||||
analysis; corrected cohorts use the pinned file above. A direct schema-suite
|
||||
invocation lacked generated test catalog metadata; artifact-aware verification
|
||||
is recorded separately. These attempts are not counted as successful runs.
|
||||
|
||||
Only Linux/JDK21/JADX0.7.1 and the fixed genuine NDK variants/public Conscrypt
|
||||
arm64 sample are qualified. Cross-ABI ELF fixtures do not establish real
|
||||
Windows/macOS provider lifecycle, device/emulator operation or runtime JNI
|
||||
registration. Existing IL2CPP regressions cover the shared relocation refactor;
|
||||
no new real IL2CPP application qualification is claimed. Runtime-built tables,
|
||||
obfuscated strings, unsupported encodings and missing split/runtime-loaded code
|
||||
remain unknown. Byte digests identify artifacts, not authenticity or build
|
||||
ownership.
|
||||
@@ -0,0 +1,366 @@
|
||||
# Caller-selected Android mapping investigation — 2026-10-27
|
||||
|
||||
## Question, revisions and oracle
|
||||
|
||||
Apply a caller-selected local R8 mapping while preserving observed DEX identities,
|
||||
original mapping-derived names/types, overloads, uncertainty, both artifact
|
||||
identities and complete source Evidence. Inspect the requested inventory and
|
||||
method bodies, then export complete Evidence. APKs and native code are never
|
||||
executed, installed or modified.
|
||||
|
||||
Both controls use main `4e67bc399951768737eeaf539f7bc9fa8373b244`:
|
||||
|
||||
- Before: `3a588662d916f801a8c444dd86f1c65c81c76bf9`, the integrated
|
||||
#7137 mapping implementation, without parsed-index reuse or retained symbol
|
||||
summaries. This is a feature-equivalent optimization control, not a claim
|
||||
that unmodified main already supported mappings.
|
||||
- After: `6b65d3a4a1a8307417365b9a1e9060e607538aa3`, bounded/cached mapping
|
||||
and explicit retained summary delivery. Later documentation changes do not
|
||||
alter the measured implementation. Following main’s experiment archive, the
|
||||
maintained harness/fixtures now live under tests and this report under
|
||||
docs/verification; the original measured scripts remain available at that pinned revision.
|
||||
- The original #7137 author is 555aaditya; integration retains their co-author
|
||||
credit and preserves #7379's shared manifest contracts/default-package tests.
|
||||
|
||||
The [source fixture](https://github.com/morluto/rea/blob/main/tests/fixtures/android/r8-mapping/README.md) has four classes,
|
||||
eight fields and fifteen methods, including constructors, overloads, Unicode
|
||||
names, nested/default-package classes, arrays and renamed types in prototypes.
|
||||
The same actual R8 invocation emits DEX and mapping. Handwritten source
|
||||
expectations are checked against pre-minification `javap -s -p`; an independently
|
||||
compiled dexlib2 reader inventories raw DEX identities without REA's parser,
|
||||
bridge or JADX aliases. All four classes and fifteen selected bodies are checked
|
||||
on each investigation. Seventeen valid method mapping records include three
|
||||
line ranges for one constructor; they are not seventeen distinct DEX methods.
|
||||
|
||||
The public-library task transforms the shipped `classes.jar` from Conscrypt
|
||||
Android 2.8.0 with actual R8. Its 949,641-byte mapping has 530 class, 1,263 field
|
||||
and 10,342 method records. A separate D8 conversion of the original class files
|
||||
provides the unminified descriptor oracle. Every selected `NativeCrypto` field
|
||||
(28) and method (390), plus inner-class identity, is verified; bodies are selected
|
||||
for `EVP_PKEY_new_RSA`, `EVP_DigestSignInit`, `SSL_new`, `SSL_free` and
|
||||
`SSL_get_error`. These are static DEX-only ZIP containers, not runnable APKs.
|
||||
D8 warned that a platform adapter's `com.android.org.conscrypt.OpenSSLSocketImpl`
|
||||
was missing; complete dependency/runtime linkage is unverified. R8 additionally
|
||||
used `-dontwarn **`, with shrinking/optimization disabled and name minification
|
||||
allowed. Stable R8 did not consume an existing APK's DEX.
|
||||
|
||||
## Pinned artifacts
|
||||
|
||||
| Artifact | Observed SHA-256 |
|
||||
| ----------------------------- | ------------------------------------------------------------------ |
|
||||
| R8 8.0.40 Maven JAR | `ab1379835c7d3e5f21f80347c3c81e2f762e0b9b02748ae5232c3afa14adf702` |
|
||||
| JADX MCP 0.7.1 JAR | `6e5eacf500b64292bfb73c49797c1958f6ee44646e43e868039ae7feb573ff75` |
|
||||
| Android 36 platform JAR | `d9eb9da824d9e247a352f570f01e1169e725b2954bca9e283a71786c59b59f9a` |
|
||||
| Source fixture input JAR | `88f3d46018ac2effb166ceed79fdd6e8b58b6bb08d2649cd0aa4f90dc260260c` |
|
||||
| Source fixture DEX | `0b4c49cf3d706f5c2c4c15a10ff27dfbabb9b07f463d508f6f348eafb3245909` |
|
||||
| Source fixture static APK | `f8c0849978ed7eceeeb293223cf2139c1749f672148dce1e16d018fa727445c9` |
|
||||
| Source fixture mapping | `5ca162587794af1bd1da00f79ee64fd4ecde6f7c91048ea5596260104df6e483` |
|
||||
| Conscrypt 2.8.0 AAR | `752044c289bc98e2dcabc00a1f88ca158b3aa3f505b38f124552bf4b2a0902b1` |
|
||||
| Conscrypt shipped classes JAR | `2cd9b6bc444cfca69048e8327383b7f5a13fc7978465e6e0fd5111a2000822f6` |
|
||||
| Conscrypt R8 DEX | `cd4d7eb9b05e1a84d064f7f9e4b33d68a8f92a9f58a6a62e4041972964ef6d27` |
|
||||
| Conscrypt static APK | `9a0224400b91b030213e0f6ba63d899a4ad6333f930414c768ce88bbef423e91` |
|
||||
| Conscrypt mapping | `67cd69e706afee50a1cc4e13bd9fd26056932c569d0ea7706dfe6fe8215a2e3c` |
|
||||
|
||||
R8 publisher and independent official Bazel checksum pin are linked from the
|
||||
fixture README. Conscrypt was downloaded from
|
||||
[Maven Central](https://repo.maven.apache.org/maven2/org/conscrypt/conscrypt-android/2.8.0/conscrypt-android-2.8.0.aar)
|
||||
and verified against its publisher SHA-256 sidecar. No generated binary or
|
||||
mapping file is committed. JAR/ZIP timestamps can change a source rebuild's
|
||||
artifact hash; the table identifies these measured bytes.
|
||||
|
||||
## Owning changes and safe bounds
|
||||
|
||||
The provider keeps one successful parsed index. Each call still performs a
|
||||
stable regular-file read and SHA-256 check. Only identical digest/length with
|
||||
fixed parser options reuses parsed facts; current path provenance comes from
|
||||
the new read. Changed content, including same-size/same-mtime rewrites, is
|
||||
reparsed; missing or cancelled input never falls back to stale facts. Idle
|
||||
retirement and close release the index. During replacement, old and new parsed
|
||||
indices can briefly coexist. APK/query results are not cached by this mechanism.
|
||||
|
||||
The domain parser scans lines without a whole-file split, accounts retained
|
||||
strings/containers before admitting records, and admits parameter/metadata
|
||||
expansion before split/JSON parsing. The fixed 64 MiB input, estimated 128 MiB
|
||||
index and 8 MiB per-residual-class claim allowances are resource constraints,
|
||||
not format claims or measured V8 heap caps. Replaced metadata, deduplicated
|
||||
range splits and overwritten inline frames are not cumulatively billed.
|
||||
Ambiguity scans count all alternatives but retain only eight stable candidates;
|
||||
conflicting residual metadata stays ambiguous. A separate combined 8 MiB claim
|
||||
budget covers actual queries including all inner classes; omitted claims become
|
||||
explicit `unknown`, preserving every raw identity and source SHA. It is not a
|
||||
whole-report/wire-byte cap.
|
||||
|
||||
Existing shared retained-Evidence projection now supports class/method summaries
|
||||
through both CLI and MCP. They preserve complete normalized facts and selected
|
||||
bodies, omitting duplicate raw RPC metadata from the inline response. Complete
|
||||
parent Evidence remains exportable. Default/complete delivery is compatible;
|
||||
summary is explicit, and CLI requires a non-overwriting portable output bundle.
|
||||
Whole summary views admit 1 MiB including context: an admitted one-MiB body may
|
||||
therefore require complete detail. Refusal preserves the MCP parent ID and is
|
||||
nonretryable for the same selection; no facts are silently truncated.
|
||||
|
||||
## Completed-task measurements
|
||||
|
||||
Linux x86-64, Node 24.22.1/npm 11.16.0, Temurin JDK 21.0.12.1, actual JADX
|
||||
0.7.1, one analysis worker, 1 GiB JVM heap and two active processors. Harness,
|
||||
server and children were restricted to CPUs 0/1; Node heap was 3 GiB. Expensive
|
||||
checks ran serially under an 8 GiB sampled family-RSS stop and a 4 GiB host-free
|
||||
memory guard. No guard stopped a completed task and no OOM occurred.
|
||||
|
||||
The [public stdio MCP harness](https://github.com/morluto/rea/blob/main/tests/support/analysis-efficiency/android-mapping.mjs) uses named live tool
|
||||
contracts and Ajv 2020 validation of SDK-converted advertised outputs. All
|
||||
correctness comparisons, hash reads, discovery and cleanup are included in the
|
||||
whole-harness cost. First-provider calls use fresh processes/JVMs; OS file caches
|
||||
were warm, and the host was shared. Conscrypt order was before/after/after/before.
|
||||
Each small task makes 19 inspections and one proof export; each library task
|
||||
makes six inspections and one proof export. Every exported parent is checked
|
||||
against the exact inline normalized facts, source parameters and APK SHA, and
|
||||
must retain raw calls. The complete/summary lanes use the same requested facts,
|
||||
provider, options, source bytes, discovery families (`android,session`) and proof.
|
||||
|
||||
Peak memory is summed process-family RSS sampled every 100 ms, including client,
|
||||
server, JVM and children. Shared pages can be counted twice and short peaks can
|
||||
be missed; this is not PSS, kernel peak accounting or an enforced production
|
||||
heap measurement. CPU is completed-family user+system time.
|
||||
|
||||
| Cohort | First task s | Warm task s | Whole harness s | CPU s | Peak family MiB |
|
||||
| ----------------- | -----------: | -----------: | --------------: | -----: | --------------: |
|
||||
| small-before | 7.342 | 1.611 | 12.060 | 16.680 | 781.7 |
|
||||
| small-after | 7.960 | 1.842 | 12.945 | 17.413 | 791.5 |
|
||||
| conscrypt-before | 8.057 | 1.015, 0.893 | 13.264 | 17.774 | 920.0 |
|
||||
| conscrypt-after | 6.975 | 0.833, 0.817 | 11.743 | 17.242 | 855.0 |
|
||||
| conscrypt-after2 | 6.987 | 0.725, 0.748 | 11.561 | 16.553 | 872.6 |
|
||||
| conscrypt-before2 | 7.046 | 0.917, 0.886 | 11.855 | 17.093 | 949.7 |
|
||||
|
||||
Warm Conscrypt task medians were 0.905 s before and 0.783 s after (about 14%
|
||||
lower across four warm samples per implementation). Cold/whole-task and memory
|
||||
samples vary; no universal startup or memory improvement is established. The
|
||||
small fixture pays extra summary/retention overhead: 12.060 to 12.945 seconds
|
||||
for the whole harness, with CPU 16.680 to 17.413 seconds. Explicit complete
|
||||
delivery remains available.
|
||||
|
||||
An earlier separate **cache-only** cohort used complete responses without proof
|
||||
export, on the same base. Conscrypt warm six-call medians fell from about
|
||||
0.981 to 0.765 s; before ranges were 0.950–0.996 s and after 0.743–0.850 s.
|
||||
Whole harness samples were 12.861–13.876 to 11.641–11.647 s, CPU
|
||||
16.916–18.297 to 16.454–16.893 s, and peak family 910.1–938.1 to
|
||||
853.1–876.8 MiB. Small before/after whole costs were 18.493/16.110 s,
|
||||
17.495/16.589 CPU s and 763.4/776.3 MiB. These samples motivated reuse but
|
||||
are not substituted for the completed proof-export task above.
|
||||
|
||||
## Whole-task token estimates
|
||||
|
||||
[android-mapping-costs.py](https://github.com/morluto/rea/blob/main/tests/support/analysis-efficiency/android-mapping-costs.py) executed with tiktoken
|
||||
0.14.0/o200k_base. Text projection counts a fixed task question, all discovered
|
||||
name/description/input schemas once per investigation, every RPC argument and
|
||||
all text replies, including proof export. SDK transport additionally counts
|
||||
output schemas and structured-content duplicates. Exported file contents are
|
||||
checked offline rather than injected into model context; export IO/CPU/memory
|
||||
is included above. Prompt replay, host pruning/rendering, hidden prompts,
|
||||
protocol handshake/framing and actual model billing are unmeasured. Estimates
|
||||
are not billed usage. Each table row is one warm completed investigation.
|
||||
|
||||
| Cohort | Estimated text tokens | Estimated SDK transport tokens |
|
||||
| ----------------- | --------------------: | -----------------------------: |
|
||||
| small-before | 43,526 | 167,094 |
|
||||
| small-after | 41,755 | 159,507 |
|
||||
| conscrypt-before | 180,213 | 510,351 |
|
||||
| conscrypt-after | 69,836 | 217,646 |
|
||||
| conscrypt-after2 | 69,908 | 217,789 |
|
||||
| conscrypt-before2 | 177,902 | 505,723 |
|
||||
|
||||
Conscrypt text estimates fall about 61%, from approximately 178–180k to 70k,
|
||||
while preserving all requested facts and complete proof. Small tasks improve
|
||||
only about 4%; summary context has its own cost. The cache-only cohort kept
|
||||
similar token counts (~172–174k library text tokens without proof export),
|
||||
revealing that parse caching alone did not address repeated raw class metadata.
|
||||
Full inventories still cost more context than a direct known-method summary.
|
||||
|
||||
Normalized investigation digests match exactly across all before/after lanes:
|
||||
|
||||
- Source fixture: `ec76d1e086c8561370fc1ed5dfef67bb88b185e8fb57c7e30afe9f1f153634eb`.
|
||||
- Conscrypt: `7ebf70795dfc23bf2b0d0e813837325fa221f15560ca2384fdb06f26d69d78a5`.
|
||||
|
||||
## Post-review ownership and final workflow confirmation
|
||||
|
||||
Greptile found three concrete boundaries subsequently fixed in
|
||||
`160573cb36f478a088b8a5284fa571cd97dc9a86`: translated prototypes could expand
|
||||
before comparison; a warm request without a mapping could bypass a pending
|
||||
mapping descriptor's cleanup; and timeout classification inspected the outer
|
||||
cleanup wrapper. Type matching now compares incrementally within the observed
|
||||
descriptor, retained file cleanup gates shared request admission and idle
|
||||
retirement independently of engine cleanup, and the first interruption is
|
||||
classified before cleanup composition. All 126 related cases pass serially:
|
||||
55 parser, five file-reader, four first-cause/close outcomes, three cleanup,
|
||||
52 session/idle/cleanup and seven MCP mapping cases.
|
||||
|
||||
A separate bounded diagnostic found and corrected source-string retention.
|
||||
On Node 24.22.1/V8 13.6.233.17-node.58, four small selected class/method results
|
||||
from four distinct mappings containing 10 MiB of ignored comments each retained
|
||||
41,944,028 extra external string bytes through the actual immutable Evidence
|
||||
ledger. Clearing it released that backing; detached-string controls retained
|
||||
zero extra external bytes. The parser now owns admitted UTF-16 strings once,
|
||||
sharing section names and copying only retained outermost/deduplicated facts.
|
||||
It preserves lone surrogate code units and never clones the entire index or
|
||||
copies names per lookup. The identical post-fix ledger experiment retained
|
||||
zero extra external bytes; its RSS delta was 151,552 bytes, versus 62,898,176
|
||||
before. These are engine-specific observations, not portable heap guarantees.
|
||||
The unchanged Evidence/ledger modules and changed parser are fingerprinted by
|
||||
[the diagnostic](https://github.com/morluto/rea/blob/main/tests/support/analysis-efficiency/android-mapping-retention.mjs).
|
||||
Before/after whole diagnostic costs were 2.995/2.582 seconds, CPU 4.976/4.382
|
||||
seconds and peak sampled family RSS 145,903,616/113,287,168 bytes, with explicit
|
||||
256 MiB old-space selection and a 1 GiB family guard.
|
||||
|
||||
Final equivalent Conscrypt investigations compare the integrated unoptimized
|
||||
control `5ecc6ccc662678c00a45bd171f40798634774155` with fixed `160573cb3`, both
|
||||
on main `e99b6c2c3bcc5f4f050b51b432145144d0d47dc0`, using the same artifacts,
|
||||
facts, seven calls including full proof export, and before/after/after/before
|
||||
order. Every normal-fact digest remains
|
||||
`7ebf70795dfc23bf2b0d0e813837325fa221f15560ca2384fdb06f26d69d78a5`.
|
||||
The final source-fixture summary/proof/negative qualification also passes all
|
||||
43 calls from the maintained harness path.
|
||||
|
||||
| Final cohort | First s | Warm s | Whole s | CPU s | Peak family MiB | Warm text estimate |
|
||||
| ----------------- | ------: | -----------: | ------: | -----: | --------------: | -----------------: |
|
||||
| conscrypt-before | 7.187 | 1.027, 1.006 | 12.370 | 17.307 | 917.7 | 177,855 |
|
||||
| conscrypt-after | 7.376 | 0.813, 0.866 | 12.402 | 17.522 | 868.7 | 69,819 |
|
||||
| conscrypt-after2 | 6.878 | 0.707, 0.726 | 11.439 | 16.383 | 883.4 | 69,902 |
|
||||
| conscrypt-before2 | 7.111 | 0.989, 1.186 | 12.566 | 17.479 | 930.1 | 177,719 |
|
||||
|
||||
The final warm ranges were 0.989–1.186 seconds before and 0.707–0.866 after;
|
||||
text estimates remain about 61% lower, ~177.7–177.9k to ~69.8–69.9k. Whole-task
|
||||
and cold samples overlap and are retained alongside the original cohort above;
|
||||
these measurements do not establish a general startup or memory advantage.
|
||||
|
||||
### Follow-up complexity review
|
||||
|
||||
Two additional review findings exposed repeated linear work despite bounded
|
||||
retention. The parsed owner now records one agreed residual alias (or explicit
|
||||
conflict) per original type, while preserving every class record and its count.
|
||||
Method identities have a direct residual-signature index; missing signatures
|
||||
still merge into the first record, including the initial unsigned-to-signed
|
||||
transition. Superseded alias arrays and candidate searches were removed.
|
||||
The representation allowance includes the new index containers/entries.
|
||||
|
||||
The reproducible
|
||||
[complexity diagnostic](https://github.com/morluto/rea/blob/main/tests/support/analysis-efficiency/android-mapping-complexity.mjs)
|
||||
compares the built parser from `160573cb3` with the follow-up parser SHA-256
|
||||
`9c4c24f7a10a17b8ec08e74b2588b9ba87bc30f3e3d16690c3e07c03fbce233b`.
|
||||
The copied before module only rewrites its three unchanged dependency imports
|
||||
to absolute URLs. Each scenario verifies late matches, counts and answer digest;
|
||||
answers match across revisions. Node 24.22.1, two CPUs, 512 MiB old space,
|
||||
8 GiB sampled family guard, one before/after trial per size:
|
||||
|
||||
| Scenario | Revision | Parse ms | Lookup ms | Whole s | CPU s | Peak family MiB |
|
||||
| ----------------------------------------------------- | -------- | -------: | --------: | ------: | ----: | --------------: |
|
||||
| 20k identical aliases, 1k candidates × 255 parameters | before | 212.60 | 5169.61 | 5.635 | 5.690 | 149.3 |
|
||||
| same | after | 207.31 | 22.84 | 0.449 | 0.559 | 117.1 |
|
||||
| 40k distinct signatures of one original method | before | 8032.93 | 5.70 | 8.300 | 8.371 | 117.5 |
|
||||
| same | after | 321.58 | 5.43 | 0.562 | 0.532 | 117.8 |
|
||||
|
||||
The smaller alias cohort (1k aliases, otherwise identical) looked up in
|
||||
289.18/24.64 ms before/after. The smaller signature cohort (10k) parsed in
|
||||
785.65/89.68 ms. These synthetic cases establish removal of the reported
|
||||
growth, not universal latency or memory gains; they involve no agent tokens.
|
||||
All 60 parser, two report and seven public MCP mapping cases pass serially,
|
||||
along with type/lint/build (79.084 s, CPU 110.273 s, peak family 4393.9 MiB).
|
||||
|
||||
The final indexed implementation also passes the real source fixture's
|
||||
43 summary/proof/negative calls (13.838 s, CPU 18.212 s, peak 797.5 MiB),
|
||||
and the Conscrypt 21-call investigation including proof export. The latter's
|
||||
normal-fact digest remains the value above: first task 7.670 s, warm tasks
|
||||
0.860/0.981 s, whole 12.706 s, CPU 17.870 s, peak 868.8 MiB. Warm whole-task
|
||||
text estimates are 69,940, SDK-inclusive 217,975; tokenizer/scope remain the
|
||||
same. Complete class/method facts and provenance remain accessible.
|
||||
|
||||
For CI head `160573cb3`, all failed/cancelled shard logs and source-test logs
|
||||
were audited. Its executed merge `ee2398458bd8c56666ecaf120c1c3c21f350eea4`
|
||||
uses main parent `0676ad254c206871e0f4b120ba300cbbf9383b62`. The 90 observed
|
||||
failed test identities were selected on a pristine worktree at that exact
|
||||
parent: 88 reproduced in the primary run (128.320 s, CPU 147.775 s, peak
|
||||
2593.3 MiB), then the two newline-bearing ZCode titles reproduced in an exact
|
||||
follow-up (2.388 s, CPU 2.788 s, peak 416.9 MiB). No feature code was present
|
||||
in either control. These are baseline failures; cancelled cases without a
|
||||
completed result remain unverified by CI, with local verification above.
|
||||
|
||||
## Executed verification and limits
|
||||
|
||||
- Parser/reader/protocol/provider mapping regressions cover raw names, exact
|
||||
descriptors/types, overloads, inline outermost frames, malformed/partial
|
||||
records, stable candidate samples, preallocation refusal, repeated metadata,
|
||||
cache freshness/path aliases/deletion, cancellation and descriptor ownership.
|
||||
- Real R8 source and public-library stdio MCP tasks verify all selected symbols
|
||||
and advertised schemas. Actual CLI complete and summary inspections verify
|
||||
raw/mapped identity, exact normalized parity and complete portable parents.
|
||||
- Actual Java metadata regression and downstream CLI/MCP body/direct-invoke
|
||||
parity pass, including reserved raw names, frozen aliases, default packages,
|
||||
encoded constants, native absence and classes.dex/classes2.dex identity.
|
||||
- Summary regressions preserve partial/unknown facts and optional historical
|
||||
omissions, validate SDK output contracts, read retained data after APK/mapping
|
||||
deletion, reject missing portable CLI output and exercise the one-MiB boundary.
|
||||
- Type/lint/build, named schema/profile checks and relevant source/boundary checks
|
||||
are executed separately. Final PR validation records the current command set.
|
||||
|
||||
The final combined 12-file Android run passed 219 of 220 cases, including all
|
||||
52 session/idle/cleanup integration cases. One existing package-summary SDK
|
||||
case hit its 30-second limit under combined load. The unchanged four-case file
|
||||
then passed with one worker: its SDK case took 11.479 seconds; whole isolated
|
||||
harness 31.867 seconds, CPU 35.064 seconds and peak family 1,551,081,472 bytes.
|
||||
No assertion or timeout was relaxed. The combined timeout remains a recorded
|
||||
resource/concurrency limitation rather than being reported as an all-green run.
|
||||
|
||||
Earlier failed attempts are retained rather than counted as passes: a resolver
|
||||
refactor used `only.mapped` after `only` already held the mapped fact; its
|
||||
regressions failed and were fixed. Tagged-error migration exposed one stale
|
||||
string-only test. New SDK schema test typing and a redundant JSON preadmission
|
||||
were corrected at their owning boundaries. A first summary MCP case timed out
|
||||
while enumerating unrelated full catalog contracts; selecting the actual
|
||||
Android/application/session families preserves every assertion and passes
|
||||
without raising timeouts. An initial library harness requested absent
|
||||
`SSL_shutdown` and skipped it; the harness now rejects absent requests, and
|
||||
reported final cohorts explicitly verify all five requested methods.
|
||||
|
||||
The mapping cancellation regression distinguishes caller-first from deadline-
|
||||
first finalization; the original implementation relabelled caller cancellation
|
||||
as timeout (one failing caller-first case and one passing deadline-first case
|
||||
on the unchanged optimization control). An initial test's public deadline code was corrected to the actual
|
||||
`provider_timeout` contract. Fixed code preserves the first cause and cleanup.
|
||||
|
||||
Only Linux x86-64/JDK21/JADX0.7.1 and R8 8.0.40 mapping-format 2.1 were qualified.
|
||||
Residual-signature conflicts use derived negative mappings, not a real mapping-
|
||||
format 2.2 producer. No Windows/macOS, emulator/device, app execution, full APK
|
||||
rebuild/linkage, source/line recovery, APK signature verification or model-usage
|
||||
trial is claimed. Mapping/build association remains explicitly caller-selected.
|
||||
|
||||
## Reproduction
|
||||
|
||||
Generate source/DEX/oracle using the fixture README with selected existing tools.
|
||||
Keep separate before/after worktrees, pin their revisions above, build both and
|
||||
use identical target/mapping/oracle bytes. Then run the same harness for each:
|
||||
|
||||
```sh
|
||||
node tests/support/analysis-efficiency/android-mapping.mjs \
|
||||
--repository "$SELECTED_REPOSITORY" --target "$STATIC_APK" \
|
||||
--mapping "$MAPPING_FILE" --output "$FRESH_RESULTS" \
|
||||
--java-home "$SELECTED_JDK" --jar "$JADX_MCP_JAR" \
|
||||
--oracle "$RAW_DEX_ORACLE" --expectations "$ORIGINAL_EXPECTATIONS" \
|
||||
--proof
|
||||
```
|
||||
|
||||
Add `--detail summary` only on the after implementation. The small fixture
|
||||
selects all classes/bodies. For Conscrypt add `--class-name U0 --repeats 3` and
|
||||
`--methods EVP_PKEY_new_RSA,EVP_DigestSignInit,SSL_new,SSL_free,SSL_get_error`;
|
||||
`U0` is the measured artifact's raw class identity, not a universal residual name.
|
||||
Use the original NativeCrypto D8 descriptor inventory for that expectation file.
|
||||
Run `--variants` separately on the source fixture to exercise derived malformed,
|
||||
partial, duplicate and conflicting mappings; those extra qualification calls
|
||||
are not part of the timing/token table's repeated task.
|
||||
|
||||
```sh
|
||||
python tests/support/analysis-efficiency/android-mapping-costs.py \
|
||||
--output "$TOKEN_RESULTS" "$BEFORE_RESULTS" "$AFTER_RESULTS"
|
||||
```
|
||||
|
||||
Capture elapsed/CPU/family RSS with the bounded host runner or an equivalent
|
||||
process-family sampler; alternate trials and keep each label's raw receipts and
|
||||
failed attempts. Complete Evidence stays in the exported per-pass bundles.
|
||||
File diff suppressed because it is too large.
Load diff
@@ -4,10 +4,14 @@ Issue [#4772](https://github.com/morluto/rea/issues/4772). Qualified runtime
|
||||
`b49ee40bf330b7bf1c5180b14dcc104c1c613a8e`, tree `0e5f8fe2feae1796c340c07c26e0e28e5d4328d8`,
|
||||
integrates main `c8ed32853881b08aa83459d4da55a28a4e09b0c8`. Recorded at
|
||||
`2026-10-22T08:30:58.269Z`. Subsequent verification-only
|
||||
commits change these reports. The [JSON record](javascript-route-inventory-2026-10-21.json)
|
||||
commits updated these reports. The [archived JSON record](https://github.com/morluto/rea/blob/7d0efef62d56d0a10134bea62d709302daa1565f/docs/verification/javascript-route-inventory-2026-10-21.json)
|
||||
pins original inputs, exact commands, profiles, failed controls and historical revisions.
|
||||
Current-head GitHub CI and automated review must be checked separately.
|
||||
|
||||
The full JSON archive is 2,681,223 bytes, with SHA-256
|
||||
`9d7ea2582a6f33ff0741578aec9f01b9327b3256ce50c582437164868e544d51`. It remains available in Git
|
||||
at the linked immutable revision; this page retains the historical summary.
|
||||
|
||||
The latest main contracts, selected environment and ASAR admission remain shared
|
||||
by CLI and MCP. The staged-code hook runs lint fixes before formatting, resolving the
|
||||
producing order behind the earlier CI format failure.
|
||||
|
||||
@@ -1,680 +0,0 @@
|
||||
# Analysis efficiency measurements
|
||||
|
||||
The [Android component checkpoint](android-components.md) records exact selected
|
||||
declaration views, real JADX verification and completed-task token estimates,
|
||||
including increased schema cost and no demonstrated speed or memory gain.
|
||||
|
||||
The [2026-10-21 coverage reconciliation](coverage-review-20261021.md) maps all
|
||||
46 source directories to concrete workflow evidence, including original feature
|
||||
PRs, historical no-benefit results and remaining verification boundaries.
|
||||
|
||||
The [GGUF validation checkpoint](model-utf8-validation.md) removes decoding of
|
||||
omitted strings while retaining full UTF-8 validation. It records real-model
|
||||
phase gains, unchanged task token costs and the higher main-task memory peak.
|
||||
|
||||
The [WAT framing checkpoint](wasm-string-framing.md) measures structural scanning
|
||||
on real SQLite and VS Code components, with unchanged Evidence and token costs,
|
||||
small whole-task timing gains and no memory improvement.
|
||||
|
||||
The [native inventory view checkpoint](native-inventory-views.md) compares
|
||||
complete delivery with summaries and selected rows on real `true` and jq
|
||||
inventories, including oversized-response recovery, portable old Evidence,
|
||||
CLI/MCP parity, tokenizer estimates and independent-engine variation.
|
||||
|
||||
The [retained-capture validation checkpoint](retained-capture-validation.md)
|
||||
reconciles the earlier audit queue and measures immutable historical-capture
|
||||
validation reuse on public compact and large qlogs, including first-view costs,
|
||||
small-input limits and unchanged Evidence delivery.
|
||||
|
||||
The [Ghidra Evidence projection checkpoint](ghidra-evidence-projection.md)
|
||||
measures removal of redundant raw copies on real `true` and jq workflows,
|
||||
with complete provenance, unchanged response budgets, native CLI/MCP checks
|
||||
and explicit cold-analysis variance.
|
||||
|
||||
For Hopper literal string queries, see the [decoded-record checkpoint](hopper-string-records.md).
|
||||
It measures record reuse through real Linux CLI/MCP workflows, preserving
|
||||
native recognition gaps, guard-stopped attempts and negative whole-task costs.
|
||||
|
||||
The [Linux embedding-runtime follow-up](hopper-linux-python-runtime.md) checks
|
||||
actual clean setup, dependency repair and normal-user analysis, retaining
|
||||
failed attempts and distinguishing prerequisite correctness from speedups.
|
||||
|
||||
For JEB on Linux, see the [CE native audit](jeb-linux-native.md). It qualifies
|
||||
the direct headless launch route and measures selected function reads through
|
||||
real CLI and production stdio MCP, preserving timeout attempts and the limits
|
||||
of wrapper-cache and token claims.
|
||||
|
||||
For Apktool locale queries, see the
|
||||
[shared-decode and table-only checkpoint](apktool-string-tables.md). It separates
|
||||
the existing decode reuse from duplicate string delivery, with three real APKs,
|
||||
CLI/MCP parity, whole-task estimates and negative timing results.
|
||||
|
||||
For immutable imported native/JavaScript parents, see the
|
||||
[validation-reuse checkpoint](immutable-parent-validation.md). It measures the
|
||||
owning format change with the existing retained-query driver, including first
|
||||
view costs, small cases, unchanged token estimates, and noisy whole-task results.
|
||||
|
||||
For authentic old Evidence and mixed investigations, see the
|
||||
[retained-query checkpoint](retained-investigation-workflows.md).
|
||||
`retained-investigation.mjs` compares existing complete delivery with selected
|
||||
pages, including capacity-refusal recovery and original-record round trips.
|
||||
It records negative model results and separates imported-format validation work
|
||||
from the query contract.
|
||||
|
||||
For genuine Fiddler captures, see the [SAZ selection checkpoint](saz-selection-workflows.md).
|
||||
`measure.mjs --saz-selection baseline|focused|retained` compares default complete
|
||||
delivery (including retained-reference recovery), caller-selected ordinals, and
|
||||
summary/preview/complete/repeated pages. Each failed RPC remains in task costs.
|
||||
The body question uses stored bytes, not decoded HTTP application payloads.
|
||||
|
||||
Tracks the experiments for [#2016](https://github.com/morluto/rea/issues/2016).
|
||||
The production CLI/MCP contracts and defaults remain the compatibility baseline.
|
||||
Downloaded artifacts, runtime copies, exported Evidence, transcripts and detailed
|
||||
JSON measurements belong outside the checkout. This document keeps the measured
|
||||
findings, verification scope and revision references; the scripts and `corpus.json`
|
||||
keep the reproduction inputs. Generated `results/` is ignored.
|
||||
|
||||
Follow-up: [latest-main capability audit](latest-main-audit.md) records the
|
||||
Flutter, SQLite, TShark and sigrok priorities and the first shipped Dart AOT
|
||||
baseline, preserving existing controls before introducing another selector.
|
||||
|
||||
Follow-up: [Dart AOT summary and retained string pools](flutter-aot-summary.md)
|
||||
records real LocalSend delivery pairs and old Evidence continuation, preserving
|
||||
ABI selection, snapshot identity, original lower bounds and partial coverage.
|
||||
|
||||
Follow-up: [SQLite existing-control measurements](sqlite-existing-controls.md)
|
||||
tests QGIS's shipped GeoPackage through the existing column selection, preserving
|
||||
full selected-table schema and typed cells before considering another API.
|
||||
|
||||
Follow-up: [small real PCAP recordings](pcap-small-captures.md) compares existing
|
||||
complete delivery with summary/pages. Preserve the negative SDK/model input
|
||||
results: compact captures do not justify another production mechanism.
|
||||
|
||||
Follow-up: [native PE/DOS entry views and real model verification](native-entry-views.md)
|
||||
records the post-merge native continuation, including a small-function case where
|
||||
four selected replies cost more bytes than the complete dossier.
|
||||
|
||||
Follow-up: [EVM descriptor and real-template controls](evm-descriptor.md) extends
|
||||
the offline corpus with digest-pinned Uniswap runtime templates and an independent
|
||||
published ABI selector oracle, preserving unknown deployment/immutable metadata.
|
||||
|
||||
Follow-up: [EVM interface summary delivery](evm-interface-summary.md) records
|
||||
the opt-in retained view, old-client/Evidence migration checks and repeated SDK
|
||||
and real-model pairs, including mixed token/time results and strict-answer limits.
|
||||
|
||||
Follow-up: [task-scoped preset lifecycle verification](preset-lifecycle.md) checks
|
||||
real Ghidra profile/snapshot boundaries, startup cancellation and retained cleanup
|
||||
retry while preserving defaults and distinguishing verification from performance.
|
||||
|
||||
Follow-up: [EVM descriptor lifecycle acceptance](evm-lifecycle.md) verifies actual
|
||||
worker replies, late SDK cancellation, permission-induced cleanup failures,
|
||||
healthy WASM peers and old CLI parity with explicit runtime and platform scope.
|
||||
|
||||
Follow-up: [managed artifact delivery and retained views](managed-artifact-summary.md)
|
||||
records two shipped assembly checks, previous-runtime portable Evidence, and four
|
||||
actual Codex trials. Selected replies shrink substantially; observed model input
|
||||
tokens increase and latency is mixed, so model-token savings are not claimed.
|
||||
|
||||
Follow-up: [model tool context and per-response usage](model-context.md) separates
|
||||
raw MCP replies from observed client execution/discovery and usage. Optional
|
||||
trial capture retains tool/usage items while excluding account and ambient state;
|
||||
existing ephemeral trials keep their default behavior.
|
||||
|
||||
Follow-up: [Android package delivery and retained manifest](android-package-summary.md)
|
||||
records real NewPipe SDK pairs, unchanged old requests/Evidence, exact retained
|
||||
text, and actual Codex outcomes including client errors and a failed discovery.
|
||||
Reply bytes fall; model-token/time gains are not established.
|
||||
|
||||
Follow-up: [resource counters at payload completion](resource-counters.md) adds
|
||||
optional owned counter observations while preserving unknown whole-scope final
|
||||
counters, default receipt shape and the unchanged CPU/memory safeguards.
|
||||
|
||||
The 12 detailed checkpoint receipts were archived byte-for-byte before removal
|
||||
from the final PR tree. The archive `review-receipts-00d51e0fc-20261012` is in the
|
||||
private experiment lab outside the checkout; its `archive-index.json` records
|
||||
source revision, filenames, sizes and SHA-256.
|
||||
Raw producer replies, model transcripts and exported Evidence remain in the same
|
||||
private lab. These local files are not required to build or use REA. Historical
|
||||
checkpoint commits retain their original receipts; no history was rewritten.
|
||||
|
||||
## Optimization priorities and gradual migration
|
||||
|
||||
The [larger/non-HTTP packet checkpoint](packet-field-workflows.md) records
|
||||
same-question SDK and actual Codex comparisons using existing input limits,
|
||||
field/filter selection, discovery and output windows. Lower delivered JSON did
|
||||
not establish stable total-token or speed gains; failed plans and negative
|
||||
controls remain visible.
|
||||
|
||||
The [sigrok selection checkpoint](signal-selection-workflows.md) records two
|
||||
real hardware recordings, independently verified bytes, repeated decoder/pages,
|
||||
and an eight-task Codex cohort with mixed token/time results. Existing annotation
|
||||
controls reduce replies; these measurements introduce no production cache,
|
||||
default or API changes.
|
||||
|
||||
Agent token consumption and end-to-end task time are the primary objectives.
|
||||
Memory and CPU consumption are also measured and controlled. Correctness,
|
||||
Evidence provenance, caller choices and existing-user compatibility are
|
||||
acceptance conditions for every optimization.
|
||||
|
||||
- Tokens: minimize unnecessary model-visible data while returning the requested
|
||||
facts and useful provenance inline. Count discovery, repeated calls, errors,
|
||||
retries and follow-up turns; report total/cached/uncached input and output
|
||||
separately. Compact JSON bytes are a delivery measurement, not a token count.
|
||||
- Time: measure completion of the same analyst question, including startup,
|
||||
provider work, transport, client processing and model corrections. Separate
|
||||
measured stages and cold/warm state; use repeated comparable trials before
|
||||
generalizing an observed latency improvement.
|
||||
- Memory: record aggregate process-tree peaks and limit/cleanup events, account
|
||||
for retained Evidence and representation expansion, and keep host safeguards.
|
||||
- CPU: record aggregate CPU time for the completed task. A one-CPU quota is an
|
||||
experiment control; it does not itself establish reduced CPU consumption.
|
||||
|
||||
Migrate domain by domain with focused real-artifact checks and relevant shared
|
||||
contract checks. The native follow-up verifies selected PE and DOS MZ entry facts;
|
||||
DOS COM and complete native application behavior remain unverified. The domain
|
||||
matrix is a growing roadmap. A passed lane does not imply all
|
||||
advertised platforms/providers have passed. Newly discovered REA bugs remain
|
||||
in the separate local handoff document for later team repair.
|
||||
|
||||
Old clients using existing CLI/MCP requests against the candidate retain the
|
||||
complete default path. New summary/view options require explicit selection.
|
||||
Actual old Evidence import/view/export checks preserve original authenticated
|
||||
records and identities. These checks establish the recorded baseline/workflow
|
||||
scope, not every historical release or third-party client. A new client targeting
|
||||
an older server must discover the actual schema and use a supported equivalent
|
||||
where available; this study does not implement a universal automatic fallback.
|
||||
The catalog intentionally gains more precise/additive schemas, so its bytes and
|
||||
digest are not frozen: measured runtime `259e99dd9` has about 2.6% larger complete
|
||||
catalog JSON than the frozen baseline. The separate post-integration receipt
|
||||
records the newer runtime, including upstream changes. Optional views can reduce
|
||||
model-visible result data; old complete requests do not silently switch delivery
|
||||
or acquire that reduction automatically.
|
||||
|
||||
## Reproduce a measurement
|
||||
|
||||
Use the repository's pinned Node/npm and locked dependencies. Build each selected
|
||||
revision, then copy its `dist/`, `bridge/`, `scripts/`, `skills/`, `package.json`
|
||||
and `package-lock.json` to a separate runtime directory before editing production
|
||||
sources. That copy needs the matching locked dependencies. The initial frozen
|
||||
baseline is `42b0ed126df7d5b908ac369715378d5e519051b7`; this study does not
|
||||
silently substitute a newer `main` or the previous Code Mode experiment.
|
||||
|
||||
The public artifacts and acquisition digests are in `corpus.json`. Supply their
|
||||
actual local paths and an optional JSON object of explicit provider environment
|
||||
settings. Do not add credentials to that file. Provider settings are inherited
|
||||
by the child, but the report persists only the selected settings' digest. Runs
|
||||
reject undeclared inherited REA/Hopper and Ghidra/Java configuration keys before
|
||||
starting a measured server or creating a model profile; include those settings
|
||||
explicitly in `--providers`. Both drivers use `provider-environment.json` and
|
||||
record its digest. The system/runtime environment still needs comparable
|
||||
controlled conditions; its contents are not serialized. Both drivers also
|
||||
fingerprint the effective provider `PATH` without recording its value, and
|
||||
the SDK comparator rejects different or unrecorded search paths. This detects different
|
||||
search configuration, not replacement of executable bytes at the same path.
|
||||
Historical receipts
|
||||
predate this guard, so their selected-settings digest alone does not certify
|
||||
that no undeclared provider setting was inherited.
|
||||
|
||||
```sh
|
||||
export NODE_OPTIONS=--max-old-space-size=512
|
||||
export UV_THREADPOOL_SIZE=1
|
||||
export GOMAXPROCS=1
|
||||
export GOMEMLIMIT=2GiB
|
||||
python3 experiments/analysis-efficiency/resource-run.py -- \
|
||||
node experiments/analysis-efficiency/measure.mjs \
|
||||
--runtime-root /absolute/frozen-runtime \
|
||||
--revision 42b0ed126df7d5b908ac369715378d5e519051b7 \
|
||||
--artifact obsidian.asar --path /absolute/obsidian-1.14.4.asar \
|
||||
--providers /absolute/selected-providers.json \
|
||||
--output /absolute/new-measurement-directory
|
||||
```
|
||||
|
||||
The wrapper requires Linux cgroup v2 and a working user systemd manager. It owns
|
||||
a unique transient scope, restricts the whole process tree to one CPU, caps
|
||||
memory at 4 GiB with no swap and 192 tasks, and samples memory/tasks/CPU every
|
||||
100 ms. It stops its own scope above 3 GiB or below 4 GiB host available memory;
|
||||
startup requires 8 GiB available. Cleanup failures retain the owned scope's
|
||||
identity. Run heavy builds, providers and model trials serially. Sampled peaks
|
||||
can miss short spikes; cgroup limit events are separately reported.
|
||||
|
||||
An actual Codex + Chromium trial exceeded the initial 192-task allowance and
|
||||
was stopped automatically below 1 GiB sampled memory, with no OOM. The isolated
|
||||
browser model retry uses `resource-run.py --task-ceiling 256 -- COMMAND...` to
|
||||
account for both process trees; memory, swap and CPU controls are unchanged.
|
||||
Keep the original aborted run and cleanup receipt. This is experiment capacity,
|
||||
not a REA defect or a successful browser analysis.
|
||||
|
||||
## What the numbers mean
|
||||
|
||||
`summary.json` records the selected artifact's before/after SHA-256, runtime
|
||||
file-table digest, dispatcher and lockfile digests, runner source digest,
|
||||
Node/SDK versions, catalog sizes, caller arguments and producer Evidence IDs.
|
||||
`runtime-files.json` identifies the exact compiled files measured. Actual
|
||||
provider settings and runtime files must stay fixed during a comparison.
|
||||
|
||||
Every tool call measures three intervals at the public SDK Transport boundary:
|
||||
|
||||
- Client work before `send`, including first-use schema preparation.
|
||||
- Transport/server roundtrip, including framing, I/O, server work and client
|
||||
JSON parsing. This interval is **not** a Ghidra-only or provider-only timer.
|
||||
- Client work after the parsed response reaches `onmessage`.
|
||||
|
||||
The SDK call's total excludes the runner's subsequent encoding and disk writes.
|
||||
Result byte counts measure compact JSON of the parsed client result, not raw
|
||||
wire bytes or tokens. Progress notifications retain their actual arrival times;
|
||||
they do not imply more precise internal engine timings than the producer emits.
|
||||
Scope CPU time and aggregate process-tree memory come from the wrapper's final
|
||||
JSON line. Fresh processes have cold application caches; host page caches are
|
||||
not globally flushed.
|
||||
|
||||
`--delivery summary` is an explicit candidate workflow, initially for artifact
|
||||
inventory. It must be supported by the selected server's advertised schema.
|
||||
The default `complete` runs the existing calls unchanged. Compare the same
|
||||
analyst facts and complete source Evidence before crediting a smaller response.
|
||||
Projection can reduce delivery bytes without reducing upstream analysis time.
|
||||
For WASM, the runner validates a summary request against the actual advertised
|
||||
2020-12 input schema, including SDK-generated references. Supporting servers
|
||||
return the first-call summary; older servers retain their complete path and use
|
||||
summary/export views after an observed transport constraint. Original errors
|
||||
remain in the receipts, and the complete source Evidence ID is a compared fact.
|
||||
`--javascript-question selected` analyzes only the exact Obsidian sources
|
||||
`package.json` and `main.js`, keeps inventory identity, and reads each observed
|
||||
`main.js` module by its exact node ID. Package metadata is not a module selector.
|
||||
|
||||
Compare paired runs with the same runner source and provider settings:
|
||||
|
||||
```sh
|
||||
node experiments/analysis-efficiency/compare.mjs \
|
||||
--left /absolute/complete-run --right /absolute/summary-run \
|
||||
--output /absolute/new-comparison.json
|
||||
```
|
||||
|
||||
This checks all selected analyst facts, including inventory source IDs, exact
|
||||
path records, manifest, counts, completeness, contradictions, coverage,
|
||||
unexplored branches and limitations. It excludes only delivery observation IDs
|
||||
and the named recovery-path description. Different settings, changed facts or
|
||||
failed workflows make the comparator fail. Binwalk's per-invocation region UUIDs
|
||||
are separately preserved with both original values and JSON paths; only those
|
||||
opaque lookup values are excluded from physical-region fact equality. Original
|
||||
producer replies are never rewritten. Selected JavaScript comparisons likewise
|
||||
preserve both values of the two named parent-heap-dependent byte budgets before
|
||||
excluding their values from fact equality; fixed limits and coverage rows remain
|
||||
checked. Counts include extra view calls and
|
||||
observed tool errors. Use repeated runs with alternating order before claiming
|
||||
a distribution or a speedup; one pair is only one observation.
|
||||
|
||||
For a completed WASM result exceeding the MCP response budget, the runner keeps
|
||||
the original error, exports retained Evidence through the public tool, and
|
||||
authenticates the exact reported record using the selected runtime's parser.
|
||||
The report names this recovery; it does not relabel the oversized inline reply
|
||||
as successful delivery.
|
||||
|
||||
## Initial observations and exclusions
|
||||
|
||||
The frozen baseline has 2,326,609 bytes of complete catalog JSON, of which
|
||||
494,207 bytes describe names, descriptions and input schemas. One Obsidian
|
||||
trial spent about 21.9 s before the first request was sent, then about 0.77 s
|
||||
in that request's transport/server roundtrip. Other cold runs differed
|
||||
substantially. These single observations identify a measurement boundary;
|
||||
they do not establish a speedup or a distribution.
|
||||
|
||||
The initial matrix exercised shipped Obsidian, NewPipe, ShareX, OpenWrt, SQLite
|
||||
WASM and jq artifacts. SQLite produced a complete result larger than the
|
||||
transport frame and requires an explicit recovery run. A first jq attempt hit
|
||||
the SDK's default 60 s timeout because the experimental runner used a superseded
|
||||
three-argument call signature. SDK 2.3.1 accepts request options as argument two;
|
||||
the corrected jq run completed real Ghidra analysis in about 265 s. That earlier
|
||||
driver timeout is excluded from REA regression claims. Provisional runs with
|
||||
different runner source digests are not one paired statistical cohort.
|
||||
|
||||
No cross-session engine database cache, default analyzer changes, external
|
||||
plugin loading, or broad plugin SDK is enabled by this experiment.
|
||||
|
||||
## Obsidian delivery checkpoint
|
||||
|
||||
The two recorded pairs compare frozen baseline
|
||||
`42b0ed12` and candidate `f1106181` in complete → summary and summary → complete
|
||||
order using the same runner source, bytes and provider settings. Both preserve
|
||||
every selected fact and the atomic inventory Evidence ID.
|
||||
|
||||
| Measurement | Complete | Summary + exact path view |
|
||||
| --------------------------------------------------- | --------------: | ------------------------: |
|
||||
| Tool-result compact JSON, including lifecycle calls | 937,404 bytes | 16,322 bytes |
|
||||
| Tool calls per question | 3 | 4 |
|
||||
| Complete catalog compact JSON | 2,326,609 bytes | 2,370,505 bytes |
|
||||
| Workflow elapsed, first pair | 11.04 s | 10.23 s |
|
||||
| Workflow elapsed, second pair | 10.57 s | 10.18 s |
|
||||
|
||||
Reply JSON falls by about 98.3%; catalog JSON grows by about 1.9%. The two elapsed
|
||||
observations do not establish a statistical speedup. Inventory coverage and
|
||||
upstream byte acquisition are unchanged. The candidate remains opt-in; existing
|
||||
requests continue to deliver the complete inspection. Each run verifies the
|
||||
selected artifact's before/after digest and reports no observed OOM or task
|
||||
limit events. Source typing and projection improve selected delivery without
|
||||
introducing an engine database cache or an external plugin protocol.
|
||||
|
||||
## Task-scoped analyzer experiment
|
||||
|
||||
The recorded jq pair explicitly disables
|
||||
`Decompiler Switch Analysis` through the existing `REA_GHIDRA_DISABLED_ANALYZERS`
|
||||
setting and compares it with the unchanged Ghidra default. Both use the same
|
||||
frozen runtime and real jq bytes, one CPU, a fresh ephemeral database and the
|
||||
same entry/main question. Completed main dossiers exceed the MCP frame in both
|
||||
runs; existing retained native views recover the selected facts, preserving the
|
||||
observed transport errors in the reports.
|
||||
|
||||
One disabled/default pair takes 174.77/276.23 seconds. Entry and main pseudocode,
|
||||
main strings and its block count agree. However, callee `0x40daa0` has 604
|
||||
recovered body bytes in eight ranges by default, and 164 bytes in four ranges
|
||||
with the same 646-byte span after disabling switch analysis. Equal top-level
|
||||
counts do not establish call-graph equivalence. **Keep the production default.**
|
||||
Use the existing control only for a caller-selected task whose required facts
|
||||
have been checked; no general fast preset is introduced. A separate real
|
||||
coreutils/Ghidra probe rejects a nonexistent analyzer before bridge readiness,
|
||||
preserves the named failure and closes the session. Source bytes remain unchanged.
|
||||
|
||||
`--schemas selected` is a separate SDK 2.3.1 client experiment. It passes the exact
|
||||
live named `tools/list` definition to the SDK's public `callTool` option, retaining
|
||||
output validation. It does not change REA or the default runner mode. Selected
|
||||
definitions compile per call rather than preparing the entire cached catalog;
|
||||
report this client effect separately from engine analysis or model behavior.
|
||||
|
||||
## Real Codex CLI trials
|
||||
|
||||
`agent-trial.py` accepts an explicit frozen runtime and revision, the same
|
||||
question templates in `questions/`, and optional explicit provider settings.
|
||||
It fingerprints the actual runtime, runs pinned local Codex CLI with a private
|
||||
temporary profile, copies only the authentication file with mode 0600, disables
|
||||
shell/web/multi-agent tools, and removes copied profile state afterward. Raw
|
||||
events and answers remain outside Git. A zero exit code is not a factual grade.
|
||||
Review the original tool calls, failed requests, result limitations and answer.
|
||||
|
||||
The optional `--enabled-tools FILE` supplies a list of exact names to the
|
||||
temporary Codex MCP configuration's `enabled_tools`, as described in the
|
||||
[official MCP configuration](https://learn.chatgpt.com/docs/extend/mcp?surface=cli).
|
||||
It leaves the REA catalog and other clients intact. Initial Obsidian trials
|
||||
showed parameter-correction attempts and different delivery choices even with
|
||||
the same question; filtering alone has not established a cost or quality win.
|
||||
Requested model aliases are recorded without inventing a resolved model identity.
|
||||
|
||||
The browser question uses `--authority browser` to permit only its explicitly
|
||||
selected public scenario in a fresh, owned headless profile. Browser observations
|
||||
and cleanup are graded separately from static artifact analysis.
|
||||
|
||||
## Real artifact compatibility checkpoint
|
||||
|
||||
Six frozen baseline/pilot pairs exercise
|
||||
Obsidian, NewPipe, ShareX, OpenWrt, SQLite WASM and published historical WETH code
|
||||
through public MCP tools. All selected facts agree under the stated comparison
|
||||
rules. Obsidian complete delivery and atomic inventory Evidence identity remain
|
||||
unchanged. ShareX native-boundary claims remain scoped to the selected DLL;
|
||||
NewPipe retains a null DEX descriptor rather than inventing one.
|
||||
|
||||
SQLite's complete 15,382,128-code-unit WAT exceeds the MCP response budget in both
|
||||
revisions. The control recovers complete Evidence through file export; the pilot
|
||||
reads summary and four exact export forms inline, without running WABT again.
|
||||
Inline byte totals exclude file exports, so they do not establish a size win.
|
||||
OpenWrt comparison preserves both sets of upstream-generated region UUIDs before
|
||||
comparing offsets, extents, signatures, coverage and limitations. WETH is a
|
||||
published historical example; a live RPC request returned HTTP 403, so this is
|
||||
not live-chain verification. Every pair records unchanged target bytes and zero
|
||||
observed OOM/task-limit events. The Linux x64 results do not establish behavior
|
||||
for IDA, Hopper, macOS, Windows or Harmony.
|
||||
|
||||
## Same-question model and selected-source follow-up
|
||||
|
||||
Two actual SQLite model trials preserve all
|
||||
12 requested facts. The question, developer instruction, provider settings and
|
||||
tool allowlist hashes match. A description-only follow-up supplies the exact
|
||||
summary kind and its fields. Calls change from 42 to 3, workflow time from
|
||||
292.64 to 63.79 seconds, and reported input tokens from 304,698 to 74,387. The
|
||||
follow-up reads no unnecessary WAT pages. Both keep the original completed
|
||||
analysis's transport error and authenticate the retained source/view identities.
|
||||
This is one stochastic model pair, not a statistical or general causal claim.
|
||||
|
||||
The selected JavaScript pair preserves
|
||||
all 24 observed `main.js` module identities and the exact two-source selection.
|
||||
The strict comparison initially differs on parent budget values derived from
|
||||
observed heap availability; both values and paths remain recorded separately.
|
||||
Other coverage facts, per-family rows, fixed worker limits, omitted/unknown
|
||||
counts, statistics and Electron observations agree. Coverage remains partial,
|
||||
with 36 unselected relevant files and 384 semantic unknowns. These static facts
|
||||
do not establish runtime behavior. The comparator does not rewrite Evidence.
|
||||
|
||||
Ten real CLI trial reviews cover native,
|
||||
managed, Android, firmware, Electron/JavaScript, WASM, EVM and browser tasks.
|
||||
The selected JavaScript model preserves core facts and partial coverage but
|
||||
omits full budget/family provenance. The browser producer captures complete
|
||||
DOM/accessibility/URL evidence and terminates its owned process; direct model
|
||||
answers lose two accessibility metadata fields to client context clipping and
|
||||
report them unknown. A requested under-development Codex code-mode configuration
|
||||
still produces a direct MCP call in the actual trace, so no executor/composition
|
||||
win is credited. Profile cleanup, original errors, actual calls, usage, facts and
|
||||
scope limits remain independently reviewed. Enabling a configuration flag does
|
||||
not prove that a capability was used.
|
||||
|
||||
Discovered REA defects are recorded separately for team repair at the user's
|
||||
request. This follow-up performs no additional REA bug fixes.
|
||||
|
||||
## Nonempty managed boundaries and final verification
|
||||
|
||||
A second shipped managed library checks
|
||||
11 ModuleRefs, 147 PInvoke declarations and 225 native implementations from
|
||||
ShareX HelpersLib 19.0.2. All complete normalized artifact and native-boundary
|
||||
fields match between baseline and pilot, including source Evidence identities.
|
||||
This covers nonempty declarations beyond the main ShareX DLL's empty import
|
||||
tables. It does not verify native exports, execute the assembly or resolve CLR
|
||||
dependencies. The curated receipt keeps identities, coverage, all ModuleRefs,
|
||||
the first three PInvoke records and digests of the complete boundary payloads.
|
||||
Full producer tables and comparison inputs stay in the private lab outputs.
|
||||
|
||||
The earlier verification includes 42 real
|
||||
WABT CLI/MCP cases and the complete real SQLite CLI JSON: the unchanged default
|
||||
returns the same original source Evidence ID, normalized result and raw result
|
||||
as the baseline exported MCP record. Source bytes are unchanged. The seven-file
|
||||
contract/composition run passes 51/52 assertions and hits the existing 20-second
|
||||
presentation-test deadline while sharing one CPU. That same unmodified assertion
|
||||
passes alone in 10.835 seconds. The original combined failure and isolated pass
|
||||
remain recorded; no test deadline or REA bug was patched. Build, type/lint/module
|
||||
checks and generated-document checks pass. No full `check:pr` or unverified
|
||||
provider/platform coverage is implied.
|
||||
|
||||
## Actual old Evidence migration
|
||||
|
||||
The migration experiment imports the original
|
||||
baseline SQLite Evidence bundle into the pilot runtime, reads a typed summary
|
||||
and four export forms, and exports the retained bundle again. The complete old
|
||||
WASM record is deeply equal after import/export, with its original authenticated
|
||||
Evidence ID. The original baseline Obsidian complete inspection also works as
|
||||
inline input to the new inventory summary, retaining the old parent and atomic
|
||||
inventory identities and manifest. Original input files remain byte-identical.
|
||||
|
||||
The runtime explicitly selects an absent WABT directory and requests no producer
|
||||
operation. These views consume existing Evidence without another provider run;
|
||||
this is portable-result reuse through existing import/export, not an engine
|
||||
database cache. The five-call workflow completes in 5.458 seconds with a sampled
|
||||
aggregate peak of 700,542,976 bytes and no observed limit or OOM events.
|
||||
|
||||
To repeat it with genuine old public outputs:
|
||||
|
||||
```sh
|
||||
python3 experiments/analysis-efficiency/resource-run.py -- \
|
||||
node experiments/analysis-efficiency/verify-migration.mjs \
|
||||
--runtime-root /absolute/frozen-pilot-runtime \
|
||||
--revision 259e99dd906290189217cf0fb1a19769bcf87101 \
|
||||
--bundle /absolute/baseline-exported-evidence.json \
|
||||
--inspection-reply /absolute/baseline-complete-inspection-reply.json \
|
||||
--output /absolute/new-migration-directory
|
||||
```
|
||||
|
||||
## PR integration verification
|
||||
|
||||
The separate post-integration verification pins runtime
|
||||
`0eb42a1eb8ab17aed04240174cb77612e733a1d8`, integrated with main
|
||||
`c22111eca2c4bc84a084f12711f405b3e7422a40`. Conflict resolution preserves main's
|
||||
HAR API-surface views, portable capture admission, exact layout/JavaScript
|
||||
summary carriers and existing CLI tests alongside inventory/WASM views.
|
||||
Earlier real-model observations keep their original runtime fingerprints.
|
||||
|
||||
Build, the complete `check` static gate and generated documentation checks pass.
|
||||
Twelve focused files pass 123 assertions, including the presentation test run
|
||||
separately with its original deadline. Actual package creation, private-prefix
|
||||
installation and 42 real WABT CLI/MCP cases pass. No package is published.
|
||||
The complete generic package verifier and full local `check:pr` were not run.
|
||||
|
||||
Genuine old SQLite/Obsidian Evidence migration passes again with WABT unavailable.
|
||||
A fresh public SDK cohort preserves the original Obsidian default result size
|
||||
(937,404 compact JSON bytes), atomic inventory identity and every selected fact;
|
||||
explicit summary plus exact paths still returns 16,322 bytes. Complete catalog
|
||||
JSON is 2,359,082 bytes versus the original baseline's 2,326,609 (about 1.4%
|
||||
larger). That difference includes intervening main changes and does not isolate
|
||||
this PR's catalog contribution. One cohort is not a latency distribution.
|
||||
|
||||
Recorded integration scopes have no OOM/OOM-kill or hard-limit abort. The full
|
||||
source-check scope reports three `memory.high` events near its 3 GiB soft
|
||||
threshold. The subsequent cold pre-push type/lint scope reports 37 such events,
|
||||
with 3,221,155,840 sampled peak bytes and no hard-limit/OOM event. Heavy lanes stay
|
||||
serial with the existing CPU, memory and no-swap controls. No further REA bug repairs
|
||||
are included; separate local findings remain for team follow-up.
|
||||
|
||||
## CI attribution
|
||||
|
||||
Review of the actual main and PR job logs separates two PR-related
|
||||
description/test mismatches from pre-existing main failures. The complete-graph
|
||||
wording check now accepts capitalization changes; the existing effects audit
|
||||
records why returning WASM execution/network metadata is not foreign execution.
|
||||
Those two files pass 11 focused assertions without production changes. Main's
|
||||
own CI already fails the same live prompt-completion assertion and Darwin
|
||||
packaged-completion checks; the affected implementation and fixture files are
|
||||
byte-identical here. Those failures remain unchanged for team review at the
|
||||
maintainer's request. CI failures are not all infrastructure faults, and the
|
||||
failed CI gate is not presented as passed.
|
||||
|
||||
| Failure | Main `c22111eca` | Reviewed PR `8d6f7eb1c` |
|
||||
| ---------------------------------------- | ---------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
|
||||
| Empty prompt completion after open | [main shard](https://github.com/morluto/rea/actions/runs/38224038323/job/114727940506) | [PR shard](https://github.com/morluto/rea/actions/runs/38228724531/job/114742955092) |
|
||||
| Packaged prompt completion, Darwin arm64 | [main package](https://github.com/morluto/rea/actions/runs/38224038323/job/114728030045) | [PR package](https://github.com/morluto/rea/actions/runs/38228724531/job/114743092314) |
|
||||
| Packaged prompt completion, Darwin x64 | [main package](https://github.com/morluto/rea/actions/runs/38224038323/job/114728029923) | [PR package](https://github.com/morluto/rea/actions/runs/38228724531/job/114743092079) |
|
||||
|
||||
`src/server/promptCompletion.ts`, `src/application/binary/BinarySession.ts`,
|
||||
`tests/boundary/mcp/promptCompletion.test.ts` and
|
||||
`scripts/verify-package-prompts.mjs` are byte-identical to that main revision.
|
||||
Main commit `07b36c148946f46b99a4ce04e55726e97d0ee03a` introduces ready-provider-only
|
||||
completion; the failing fixtures expect populated completion immediately after
|
||||
opening. Whether those expectations or the readiness behavior should change is
|
||||
left for team review. Commit `00d51e0fcd1d1bba60bf56281725e82d525a27ae`
|
||||
synchronizes only the two PR-related description checks.
|
||||
|
||||
## Frozen runtime references
|
||||
|
||||
### Review follow-up
|
||||
|
||||
Two substantive bot findings were reproduced and addressed before merge. Atomic
|
||||
inventory projections now reuse the owning subject/digest/format validation.
|
||||
Actual public SDK calls with original Obsidian data demonstrated the old
|
||||
acceptance of null subjects and inconsistent formats; the corrected runtime
|
||||
rejects both with `invalid_request`. Six boundary assertions also cover missing
|
||||
subject, wrong format and wrong digest across summary and node views.
|
||||
|
||||
The related three-file run passes 37 assertions, and full static/docs gates
|
||||
pass. Genuine baseline SQLite/Obsidian import/view/export still preserves whole
|
||||
original records and source/view identities with WABT unavailable. A new real
|
||||
Obsidian summary workflow preserves every selected fact, atomic source identity
|
||||
and 16,322 delivered JSON bytes; catalog bytes remain 2,359,082.
|
||||
|
||||
Both experimental drivers reject undeclared inherited provider controls before
|
||||
server/profile creation and log no values. Explicit settings still override
|
||||
the corresponding inherited values. This prevents claiming equal declared
|
||||
settings while silently inheriting a different analysis option; it does not
|
||||
certify every aspect of the inherited system/runtime environment. These are
|
||||
research-driver controls, with no production configuration changes.
|
||||
|
||||
The source/static scope records 98 `memory.high` events, no hard limit/OOM event,
|
||||
and a sampled peak of 3,221,209,088 bytes. The real migration/Obsidian scope peaks
|
||||
at 677,367,808 bytes with no limit/OOM events. All raw receipts remain outside Git.
|
||||
|
||||
A later WASM finding was also reproduced: portable sources with absent subject
|
||||
or a different subject path could report another artifact path. The producer and
|
||||
view now share an exact path/digest identity rule. Actual WABT inspection of the
|
||||
published VS Code Tree-sitter INI grammar (0.3.1, 4,716 bytes, SHA-256
|
||||
`1e8cdf3027209100fa54a43d0faed69721ae1116428164c8ad6dcc496a9aa6af`)
|
||||
supplied the unchanged report for public SDK before/after checks. The old runtime
|
||||
accepts both inconsistencies; the corrected runtime rejects both with
|
||||
`invalid_request` and WABT unavailable. Its 26 focused assertions and type/lint
|
||||
checks pass, and genuine old records still round-trip unchanged. The verification
|
||||
scope peaks at 3,204,509,696 bytes with no observed limit/OOM event. The initial
|
||||
CSS producer exceeded the MCP reply budget as expected; that failed probe is
|
||||
preserved separately and is not credited as a completed identity reproduction.
|
||||
|
||||
These revisions identify different experiment stages; model measurements were
|
||||
not rerun or relabeled as post-integration measurements.
|
||||
|
||||
| Stage | Source revision |
|
||||
| ---------------------------------------------- | ------------------------------------------ |
|
||||
| Original baseline | `42b0ed126df7d5b908ac369715378d5e519051b7` |
|
||||
| Inventory delivery pairs | `f110618184e99f89fa776bc5e59703a61baa7c57` |
|
||||
| Initial WASM pilot | `0002caa07ffd7319672a21ce901dbfbc55e21aac` |
|
||||
| Exact WASM summary guidance | `259e99dd906290189217cf0fb1a19769bcf87101` |
|
||||
| Main-integrated package/migration verification | `0eb42a1eb8ab17aed04240174cb77612e733a1d8` |
|
||||
|
||||
The baseline runtime file-table SHA-256 is
|
||||
`3a1765a6fec0031db0b288112d4db9bc0dd503c9c3aa0946f07ce57f7673ab4b`;
|
||||
the integrated runtime file-table SHA-256 is
|
||||
`e99d3bdc851822b77e7d5bfaa9b60c57e7b99bd66e27a739bdea2e9300560d5f`.
|
||||
The integrated lockfile SHA-256 is
|
||||
`0a55042141899a442a39dce3d49755c65a861f86d1930bd4f9c6d89cd53f5eba`.
|
||||
Actual old migration inputs have SHA-256
|
||||
`e1ff3d28801436bcc11693f79cc847a4f31234f49d88449f9e0d77215c3765fc`
|
||||
(SQLite bundle) and
|
||||
`7cecd07012bb79f3afad2ea9e70357e056e9eab48c1aa13bbbd310cdcb43eb9f`
|
||||
(Obsidian complete inspection reply). The source WASM identity
|
||||
`ev_16b9f6e0fcefcab9603f14fd5e63ea00c4749accd94fef4a4e9fbb260d2e0140`
|
||||
and atomic inventory identity
|
||||
`ev_22c17e52489f3ec0e7298a62da37bd985dbbec202d7208a0bb1c20d863790aed`
|
||||
remain unchanged after the recorded import/view/export workflows.
|
||||
|
||||
### Complete carrier identity and provider search configuration
|
||||
|
||||
The outer `inspect_artifact` carrier now uses the same owning subject/digest/format
|
||||
validator as its atomic inventory. Invalid imported/inline carriers are rejected
|
||||
before selected-view construction; the generic historical Evidence parser and
|
||||
valid complete records are unchanged. Six additional boundary assertions cover
|
||||
missing subject, wrong format and wrong digest across summary/node projections.
|
||||
The negative checks use the small keyed-archive workflow so the original large
|
||||
ZIP retention/capacity workflow keeps its existing deadline and distinct coverage.
|
||||
An initial large-workflow placement timed out and is retained as a failed attempt.
|
||||
The three public MCP workflows, build, fast static gate and generated docs pass.
|
||||
The final static/test scope observed one soft `memory.high` event, with no hard
|
||||
memory, task-limit, OOM or OOM-kill event; it is not a performance trial.
|
||||
|
||||
Both research drivers fingerprint the effective provider `PATH`, including an
|
||||
explicit caller override, without serializing its value or the ambient environment.
|
||||
The SDK comparator rejects absent or unequal search-path fingerprints. This
|
||||
records search configuration, not executable replacement at an unchanged path;
|
||||
provider/runtime identities and controlled host conditions still matter.
|
||||
|
||||
Actual public SDK calls using the original shipped Obsidian inspection reproduce
|
||||
accepted null/wrong-format outer subjects on frozen `2ef0467bb` and `invalid_request`
|
||||
on the corrected compiled runtime. Two fresh Obsidian replays with different
|
||||
inherited search paths preserve selected facts and source identities, but the
|
||||
comparator rejects matching settings with only `provider_search_path_sha256`
|
||||
different. The original records and failed/accepted replies stay in private lab
|
||||
receipts; no environment value is copied into them.
|
||||
|
||||
### Resource-counter closeout qualification
|
||||
|
||||
The experimental wrapper now attempts a final cgroup event read before waiting
|
||||
or cleanup. Disappeared/incomplete counters or still-owned descendants make
|
||||
`resource_counters_verified` false; last observations remain available separately.
|
||||
The model grader treats missing final counters as unverified, never as zero,
|
||||
and any recorded soft/hard/task-limit event prevents an unrestricted resource
|
||||
pass. Lightweight parser probes cover missing/incomplete and nonzero counters;
|
||||
an actual short command confirms disappearance is reported as unknown. Regrading
|
||||
one already-correct SQLite model answer preserves its facts but rejects the old
|
||||
receipt's absent final resource counters. Earlier receipts record last-sampled
|
||||
observations, not verified final counters. Their bytes, facts and token reports
|
||||
remain original observations; resource-qualified performance conclusions need
|
||||
this additional closeout evidence. No analyzer/build/schema/package behavior is
|
||||
changed, so unrelated verification is not repeated for this research-driver fix.
|
||||
|
||||
The next bounded WASM delivery cohort is documented in [first-call summary measurements](wasm-first-call.md).
|
||||
@@ -1,65 +0,0 @@
|
||||
"""Count completed Android MCP tasks; estimates do not establish model billing."""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import tiktoken
|
||||
|
||||
|
||||
def compact(value):
|
||||
return json.dumps(value, ensure_ascii=False, separators=(",", ":"), allow_nan=False)
|
||||
|
||||
|
||||
def inspect(path, encoding):
|
||||
report = json.loads((path / "report.json").read_text())
|
||||
catalog = json.loads((path / "catalog.json").read_text())
|
||||
calls = json.loads((path / "calls.json").read_text())
|
||||
assert len(report["runs"]) == 2 and len(calls) == 4
|
||||
assert all(run["answer_verified"] and run["calls"] == 2 for run in report["runs"])
|
||||
count = lambda text: len(encoding.encode(text, disallowed_special=()))
|
||||
inputs = [{key: tool[key] for key in ["name", "description", "inputSchema"]}
|
||||
for tool in catalog["tools"]]
|
||||
question = (f"For {report['target']}, find the receiver declaring "
|
||||
"android.provider.Telephony.SMS_RECEIVED and report its identity, permission, "
|
||||
"enabled/process declarations, per-filter priority and source position. "
|
||||
"Preserve source coverage and export complete parent Evidence. "
|
||||
"Static declarations do not establish runtime behavior.")
|
||||
rows = []
|
||||
for index, run in enumerate(report["runs"]):
|
||||
selected = calls[index * 2:index * 2 + 2]
|
||||
assert [call["name"] for call in selected] == ["inspect_android_package", "export_evidence_bundle"]
|
||||
assert not any(call["result"].get("isError") for call in selected)
|
||||
arguments = sum(count(compact({"name": call["name"], "arguments": call["arguments"]}))
|
||||
for call in selected)
|
||||
text = sum(count(block["text"]) for call in selected
|
||||
for block in call["result"]["content"] if block["type"] == "text")
|
||||
wire_calls = [{key: call[key] for key in ["name", "arguments", "result"]} for call in selected]
|
||||
rows.append({**run, "estimated_tokens": {
|
||||
"question": count(question), "input_catalog_once": count(compact(inputs)),
|
||||
"arguments": arguments, "text_replies": text,
|
||||
"whole_task_text_projection": count(question) + count(compact(inputs)) + arguments + text,
|
||||
"sdk_catalog_and_calls": count(compact({"catalog": catalog, "calls": wire_calls})),
|
||||
}, "inspection_sdk_reply_bytes": len(compact(selected[0]["result"]).encode()),
|
||||
"inspection_sdk_reply_tokens": count(compact(selected[0]["result"]))})
|
||||
return {"case": str(path), "mode": report["mode"], "sha256": report["sha256"],
|
||||
"runs": rows, "actual_model_usage": None,
|
||||
"receipts_sha256": {name: hashlib.sha256((path / name).read_bytes()).hexdigest()
|
||||
for name in ["report.json", "catalog.json", "calls.json"]}}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument("cases", type=Path, nargs="+")
|
||||
args = parser.parse_args()
|
||||
encoding = tiktoken.get_encoding("o200k_base")
|
||||
records = [inspect(path, encoding) for path in args.cases]
|
||||
assert len({record["sha256"] for record in records}) == 1
|
||||
args.output.write_text(json.dumps({"tokenizer": f"tiktoken {tiktoken.__version__}",
|
||||
"encoding": encoding.name, "scope": "Text projection counts a fixed question, discovered name/description/input schemas once, all RPC arguments and text replies including proof export. Full SDK transport also counts output schemas and structuredContent. Host context injection and model billing are unmeasured.",
|
||||
"records": records}, indent=2) + "\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,222 +0,0 @@
|
||||
# Android component declarations and selected retained facts
|
||||
|
||||
Addresses [#4448](https://github.com/morluto/rea/issues/4448), extending the inventory
|
||||
already on main. Builds on Harsh's partial [#7264](https://github.com/morluto/rea/pull/7264)
|
||||
revision `987b5aceb66c7526e47cf6d064c22029c2f48c04`, including its process budget.
|
||||
The historical-field correction preserves omission and marks unknown coverage;
|
||||
new fields are never invented as absent in old Evidence.
|
||||
|
||||
## Owner and compatibility
|
||||
|
||||
The Android manifest parser owns declaration interpretation. A shared domain
|
||||
contract replaces duplicated provider/domain declarations; a single namespace
|
||||
reader and one nested-declaration pass produce structured filters and existing
|
||||
flattened action/category fields. Source positions address the exact retained
|
||||
decoded text. Unsupported data constraints and filter constructs mark partial
|
||||
coverage, including scheme-specific-part attributes outside the projected fields.
|
||||
The producer emits all current fields; historical readers accept omitted fields.
|
||||
|
||||
Existing complete requests and the existing default summary remain available.
|
||||
An explicit components view selects exact names/kinds/actions/exported declarations,
|
||||
fields and pages. Component summaries count explicit/default exports separately.
|
||||
CLI and MCP share the retained projection and retain complete parent Evidence.
|
||||
Repeated views reuse validated sealed records under the existing session lifetime;
|
||||
no new persistent provider cache or live-state freshness claim is introduced.
|
||||
Rows always retain the parent array ordinal and declaration coverage. The APK
|
||||
subject, engine identity, source status, limitations and parent linkage remain
|
||||
available. Nothing here establishes runtime enabling, resource resolution, Android
|
||||
URI matching, signature authenticity or execution of an app.
|
||||
|
||||
## Reproduction inputs
|
||||
|
||||
The old runtime is main `15a8f6847ccac581c7ff717b42062a54b1ba8084`.
|
||||
The timing/token qualification below was captured on that pinned base before the
|
||||
feature was rebased onto main `6207843814e5267e3a1d9796d5800346f69dc7da`.
|
||||
The Android producer, bridge and view sources had no upstream changes in that
|
||||
range; latest-main integration is checked separately rather than relabeling the
|
||||
older runtime receipts.
|
||||
Actual verification uses Linux x86-64, Node 24.22.1, Temurin 21.0.12.1,
|
||||
JADX headless MCP 0.7.1 and the existing REA Java bridge protocol 7.
|
||||
JAR SHA-256: `6e5eacf500b64292bfb73c49797c1958f6ee44646e43e868039ae7feb573ff75`.
|
||||
The public [Appium ApiDemos 6.0.18 APK](https://github.com/appium/android-apidemos/releases/download/v6.0.18/ApiDemos-debug.apk)
|
||||
is 6,521,217 bytes, SHA-256
|
||||
`a9eecf37b26cd084855c530db81c2bb1b91f4c1b095a04f47aa7c20e2791f686`.
|
||||
It has 355 declared components and 152,541 decoded manifest UTF-16 units.
|
||||
The recorded sorted relative-path/SHA-256 tables cover compiled `dist/*.js` and
|
||||
JSON recursively, the dispatcher/environment helper and Android bridge. Table
|
||||
SHA-256: old runtime `9581afe8ba48c15d803dd7ed3ffc4d10e4e4b99857a0b90206c8012e266daffe`
|
||||
(2,435 files), final candidate
|
||||
`a78698c12ae1dc514ffd3f09ca1b881747c9dde7d1a8677ceed7de3233334c40`
|
||||
(2,439 files). Raw tables remain with the private receipts.
|
||||
|
||||
`android-components.mjs` exercises actual public stdio MCP tools and validates
|
||||
results against the selected named SDK tool definition. Supply `--repository`,
|
||||
`--target`, a fresh `--output` directory, `--mode before|after`, `--java-home` and
|
||||
`--jar`. It captures the selected catalog, every RPC, complete exported parents,
|
||||
first-capture and warm-session results, and checks APK identity and an independent
|
||||
stdlib ElementTree oracle. No production manifest parser is reused by that oracle.
|
||||
Keep raw results, APKs and resource receipts outside Git. The selected tool families
|
||||
are android/application/session, with compact input schemas, two allowed CPUs,
|
||||
one analysis worker, 1 GiB JVM heap and a 3 GiB Node heap. A separate supervisor
|
||||
samples process-family memory every 100 ms, with an 8 GiB early stop and a 4 GiB
|
||||
host-available-memory floor. No guard stop or OOM occurred in these measurements.
|
||||
|
||||
## Equivalent completed question
|
||||
|
||||
Find the receiver declaring `android.provider.Telephony.SMS_RECEIVED` and report
|
||||
its identity, permission, enabled/process declarations, per-filter priority and
|
||||
source position. The old runtime needs complete manifest delivery for the new
|
||||
facets. The selected candidate returns the matching row directly. Both treatments
|
||||
make one inspection plus one complete Evidence export per capture; the oracle
|
||||
verifies one positive match and the total count of 355 source declarations. An additional capture
|
||||
uses the same server/engine session. First capture is a fresh process/session,
|
||||
not a flushed OS page cache. Export costs and the extra call are included.
|
||||
|
||||
Initial candidate qualification passed the independent oracle. It measured
|
||||
first/warm inspection RPCs of 11.395/0.329 seconds versus the old runtime's
|
||||
10.534/0.314 seconds. The complete two-capture process family took 19.065 seconds,
|
||||
27.687 CPU seconds and 1,193,615,360 peak sampled RSS bytes versus
|
||||
16.534 seconds, 24.675 CPU seconds and 1,107,955,712 bytes. This run predates the
|
||||
additional unsupported-data-attribute coverage correction. It establishes no
|
||||
latency, CPU or memory improvement; retained data has more declaration facts.
|
||||
|
||||
The final coverage-corrected candidate and a repeated immutable-main control
|
||||
also passed the identical oracle. Preserve all four receipts rather than choosing
|
||||
a favorable timing:
|
||||
|
||||
| Runtime, execution order | First/warm inspection RPC seconds | Whole two-capture seconds | CPU seconds | Peak sampled family RSS bytes |
|
||||
| ------------------------ | --------------------------------: | ------------------------: | ----------: | ----------------------------: |
|
||||
| Main control 1 | 10.534 / 0.314 | 16.534 | 24.675 | 1,107,955,712 |
|
||||
| Initial candidate | 11.395 / 0.329 | 19.065 | 27.687 | 1,193,615,360 |
|
||||
| Final candidate | 14.558 / 0.571 | 23.369 | 36.031 | 1,145,073,664 |
|
||||
| Main control 2 | 26.348 / 0.511 | 36.537 | 36.381 | 1,168,494,592 |
|
||||
|
||||
The repeated main control shows substantial host/run variation. These samples
|
||||
establish neither a deterministic slowdown nor a speedup; warm candidate RPCs
|
||||
are slightly slower in both observations. RSS is the sum of family RSS sampled
|
||||
every 100 ms, so shared pages may count more than once and short peaks may be
|
||||
missed; it is not a kernel/cgroup peak. Tokenizer processing is outside the timed
|
||||
SDK runs. Local oracle work, hashing, discovery, serialization, proof export and
|
||||
cleanup are inside. Cold/warm CPU and memory are not separately measured.
|
||||
|
||||
For the first main control and final candidate, `o200k_base` estimates give:
|
||||
|
||||
| Cost per first capture | Old complete | Selected components |
|
||||
| ------------------------------------------------------------ | -----------: | ------------------: |
|
||||
| Inspection reply bytes | 978,381 | 11,307 |
|
||||
| Inspection reply estimated tokens | 225,926 | 2,715 |
|
||||
| Selected family name/description/input-schema catalog tokens | 14,855 | 15,346 |
|
||||
| Question + input catalog + both call arguments/text replies | 118,430 | 16,893 |
|
||||
| Full SDK catalog, including output schemas, + both calls | 516,533 | 299,566 |
|
||||
|
||||
The primary text projection counts a fixed question, the input catalog once, all
|
||||
call arguments and every text reply, including export receipts. The SDK transport
|
||||
row additionally counts output schemas and structuredContent; it is not equated
|
||||
with model context. Reproduce with `android-component-costs.py --output costs.json
|
||||
BEFORE_DIRECTORY AFTER_DIRECTORY` (requires tiktoken). The manifest file exported locally is retained
|
||||
proof, not inline model context. These are tokenizer estimates of scripted SDK
|
||||
workflows, not billed model usage or a measured autonomous-agent task. They do
|
||||
not establish fewer model rounds or a generic analysis speedup. Actual provider
|
||||
work still reads the complete manifest, and schema discovery grows.
|
||||
|
||||
## Global catalog compatibility
|
||||
|
||||
The actual CI merge `10746a4e8f8c622357cf8dd529f98adc8bf06d2b` combines the feature
|
||||
with main parent `86d4a21215cb0fbafaf555b5c7d9fcda3f255d82`. Real SDK discovery,
|
||||
using the same complete contract registrations as the existing profile test, gives:
|
||||
|
||||
| Explicit wire profiles | Pure main bytes | CI merge bytes |
|
||||
| ------------------------------ | --------------: | -------------: |
|
||||
| Full input / full output | 4,398,561 | 4,425,163 |
|
||||
| Compact input / compact output | 556,450 | 558,653 |
|
||||
| Compact input / output off | 302,913 | 305,116 |
|
||||
| Full input / output off | 1,047,039 | 1,052,239 |
|
||||
|
||||
Input presentation defaults to compact; output presentation defaults to full.
|
||||
For frame-capped hosts, compact/off outputs are explicit compatibility choices.
|
||||
The bounded compatibility combinations retain the existing one-megabyte test
|
||||
limit and per-schema budgets. New selectors add 5,200 bytes with full inputs:
|
||||
2,699 input bytes and 379 description bytes on `inspect_android_package`, plus
|
||||
2,122 input bytes on the retained-view operation. Compact-input discovery grows
|
||||
2,203 bytes. These global discovery costs are distinct from the selected-family
|
||||
task token estimates above.
|
||||
|
||||
This change crosses the old test assertion for full input with output off.
|
||||
That assertion was broader than its default-compact test name and the existing
|
||||
profile documentation: full inputs deliberately preserve complete guidance and
|
||||
can dominate the frame even without output schemas. The corrected test retains
|
||||
both compact-input frame limits, compares every full input schema with its named
|
||||
canonical advertisement under full/off outputs, preserves the entire tool set,
|
||||
and verifies that off removes only output advertisements. No production limit is
|
||||
raised and no full input validation/guidance is discarded. The owning guide now
|
||||
states this full-profile boundary explicitly.
|
||||
|
||||
## Verification boundaries and failed attempts
|
||||
|
||||
The real Java regression exercises the existing raw/default-package class selector,
|
||||
native absence, references and frozen metadata around neighbouring source generation.
|
||||
An initial assertion compared the entire usage object: JADX legitimately populated
|
||||
`def_pos` from 0 to 103 after generating source. The revised assertion excludes only
|
||||
that presentation position and keeps class/method identity, raw DEX invoke IDs,
|
||||
descriptor/overload facts and invoke offsets. Production bridge behavior and version
|
||||
are unchanged.
|
||||
|
||||
One combined focused run passed 121 checks but hit the existing 30-second MCP test
|
||||
timeout while advertising every unrelated tool family to the SDK. Its failed receipt
|
||||
is retained. The Android-specific test now selects the same relevant families as
|
||||
the real SDK workflow; its named schemas, retained manifest/source-removal behavior,
|
||||
full/summary compatibility and complete Evidence export assertions remain enabled.
|
||||
A subsequent run exposed a test assertion that incorrectly expected repeated
|
||||
acquisitions to reuse the first Evidence ID. Each selected capture now proves its
|
||||
own retained parent exists and has the same artifact and normalized facts; engine
|
||||
reuse does not collapse distinct acquisition identities.
|
||||
|
||||
ApiDemos contains no URI data declarations. The checked-in manifest/resources and
|
||||
Smali container permit a second source-built APK with real AAPT2 binary XML and
|
||||
JADX decoding, including two filters, URI data, metadata, a resource boolean,
|
||||
explicit process and provider URI grants. Compiled APK/DEX outputs are private.
|
||||
Static declaration verification does not execute these component classes.
|
||||
The source-built APK SHA-256 was
|
||||
`aea6ac64630ad70e93f4d85de0237cc51a7012b077ca0d0e7b24cf22c5737532`.
|
||||
Its real CLI selected view and retained bundle passed `verify-fixture.py`, including
|
||||
AAPT2 dump comparison, all source positions, grouping, raw resource references,
|
||||
metadata names and provider URI grants. The real SDK VIEW query passed the
|
||||
independent ElementTree oracle on first and warm capture. An initial fixture build
|
||||
used Smali's CLI and failed because the engine JAR does not bundle optional
|
||||
JCommander; `BuildDex.java` now calls the already bundled Smali API directly.
|
||||
|
||||
The real metadata helper passed all Java markers and downstream CLI/MCP method
|
||||
body and direct-invoke parity checks for both classes.dex and classes2.dex:
|
||||
149.436 seconds, 204.026 CPU seconds and 1,070,514,176 sampled peak family RSS bytes.
|
||||
No production bridge change was needed for default-package selectors.
|
||||
|
||||
A genuine complete parent from the old runtime also passed a new CLI retained
|
||||
component query with an explicitly nonexistent engine JAR. Its newly introduced
|
||||
fields remained omitted and row coverage remained unknown; no APK/provider
|
||||
reopening was required. The current source checks cover 111 parser cases,
|
||||
six projection cases, four CLI early-validation cases and four MCP cases,
|
||||
including named SDK-converted schemas and retained source-removal behavior.
|
||||
|
||||
After the rebase onto `6207843814e5267e3a1d9796d5800346f69dc7da`, type/lint,
|
||||
all 125 focused cases and the nine documentation conformance cases passed again.
|
||||
A fresh real ApiDemos SDK qualification also passed the independent oracle:
|
||||
first/warm RPCs 11.291/0.290 seconds, whole two-capture elapsed 17.008 seconds,
|
||||
CPU 26.425 seconds and peak sampled family RSS 1,166,245,888 bytes. This is latest-main
|
||||
integration evidence; it is not paired with a new baseline performance cohort.
|
||||
|
||||
Review follow-up: Greptile correctly identified that the original oracle filtered
|
||||
before comparison, so the timing cohorts above verified selected-receiver facts
|
||||
and the inventory count, not every unselected component's facts. The oracle now
|
||||
returns an unfiltered inventory and derives the selected list from it. The driver
|
||||
checks every retained parent's identity, enabled/raw-enabled/process/permission,
|
||||
filter grouping/data/priority, metadata names, declaration coverage and component
|
||||
source location before checking the selected page. The receipt records both the
|
||||
verified count and scope; historical mode checks the available identity/permission
|
||||
facets without inventing the new observations. A fresh real ApiDemos first/warm
|
||||
run passed all 355 rows in both captures (16.538 seconds, 24.977 CPU seconds,
|
||||
1,176,817,664 sampled peak family RSS bytes). The older receipts remain unchanged. The unfiltered historical-mode run also
|
||||
passed all 355 available identity/permission rows in both captures; the compiled
|
||||
manifest fixture passed all three current rows plus its selected VIEW page.
|
||||
|
||||
Platform coverage is Linux/JDK21/JADX0.7.1 only. Android devices, emulators, actual
|
||||
runtime enabling/registration, other JDKs and Windows/macOS are unverified here.
|
||||
@@ -1,294 +0,0 @@
|
||||
/** Real APK task through public MCP, with complete retained Evidence and an independent XML oracle. */
|
||||
import assert from "node:assert/strict";
|
||||
import { createHash } from "node:crypto";
|
||||
import { createReadStream } from "node:fs";
|
||||
import { mkdir, readFile, writeFile } from "node:fs/promises";
|
||||
import { createRequire } from "node:module";
|
||||
import { dirname, join, resolve } from "node:path";
|
||||
import { fileURLToPath, pathToFileURL } from "node:url";
|
||||
import { parseArgs } from "node:util";
|
||||
import { performance } from "node:perf_hooks";
|
||||
import { createVerifierCommands } from "../../scripts/lib/verify-commands.mjs";
|
||||
|
||||
const { values } = parseArgs({
|
||||
options: {
|
||||
repository: { type: "string" },
|
||||
target: { type: "string" },
|
||||
output: { type: "string" },
|
||||
mode: { type: "string" },
|
||||
"java-home": { type: "string" },
|
||||
jar: { type: "string" },
|
||||
action: {
|
||||
type: "string",
|
||||
default: "android.provider.Telephony.SMS_RECEIVED",
|
||||
},
|
||||
},
|
||||
});
|
||||
for (const name of [
|
||||
"repository",
|
||||
"target",
|
||||
"output",
|
||||
"mode",
|
||||
"java-home",
|
||||
"jar",
|
||||
])
|
||||
assert.ok(values[name], `Missing --${name}`);
|
||||
assert.ok(["before", "after"].includes(values.mode));
|
||||
const repo = resolve(values.repository),
|
||||
target = resolve(values.target),
|
||||
output = resolve(values.output);
|
||||
await mkdir(output, { mode: 0o700 });
|
||||
const digest = createHash("sha256");
|
||||
for await (const chunk of createReadStream(target)) digest.update(chunk);
|
||||
const sha256 = digest.digest("hex");
|
||||
const require = createRequire(join(repo, "package.json"));
|
||||
const { Client } = await import(
|
||||
pathToFileURL(require.resolve("@modelcontextprotocol/client"))
|
||||
);
|
||||
const { StdioClientTransport } = await import(
|
||||
pathToFileURL(require.resolve("@modelcontextprotocol/client/stdio"))
|
||||
);
|
||||
const commands = await createVerifierCommands();
|
||||
const env = {
|
||||
PATH: process.env.PATH,
|
||||
JAVA_HOME: resolve(values["java-home"]),
|
||||
REA_JADX_MCP_JAR: resolve(values.jar),
|
||||
REA_JADX_HEAP_MIB: "1024",
|
||||
REA_JADX_ACTIVE_PROCESSOR_COUNT: "2",
|
||||
NODE_OPTIONS: "--max-old-space-size=3072",
|
||||
UV_THREADPOOL_SIZE: "2",
|
||||
REA_LOG_LEVEL: "silent",
|
||||
REA_MCP_TOOL_FAMILIES: "android,application,session",
|
||||
REA_MCP_INPUT_SCHEMA_PROFILE: "compact",
|
||||
};
|
||||
const transport = new StdioClientTransport({
|
||||
command: process.execPath,
|
||||
args: [join(repo, "scripts/rea.mjs"), "mcp"],
|
||||
cwd: repo,
|
||||
env,
|
||||
stderr: "pipe",
|
||||
maxBufferSize: 16 * 1024 * 1024,
|
||||
});
|
||||
const diagnostics = [];
|
||||
let diagnosticBytes = 0;
|
||||
let omittedDiagnosticBytes = 0;
|
||||
transport.stderr?.on("data", (chunk) => {
|
||||
const bytes = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk);
|
||||
const retained = bytes.subarray(0, Math.max(0, 65536 - diagnosticBytes));
|
||||
if (retained.length > 0) diagnostics.push(retained);
|
||||
diagnosticBytes += retained.length;
|
||||
omittedDiagnosticBytes += bytes.length - retained.length;
|
||||
});
|
||||
const client = new Client({
|
||||
name: "rea-real-android-components",
|
||||
version: "1",
|
||||
});
|
||||
let catalog;
|
||||
const calls = [],
|
||||
runs = [];
|
||||
const invoke = async (name, arguments_) => {
|
||||
const definitions = catalog.tools.filter((tool) => tool.name === name);
|
||||
assert.equal(definitions.length, 1);
|
||||
const start = performance.now();
|
||||
const result = await client.callTool(
|
||||
{ name, arguments: arguments_ },
|
||||
{ toolDefinition: definitions[0], timeout: 120000 },
|
||||
);
|
||||
calls.push({
|
||||
name,
|
||||
arguments: arguments_,
|
||||
result,
|
||||
elapsed_ms: performance.now() - start,
|
||||
});
|
||||
assert.notEqual(
|
||||
result.isError,
|
||||
true,
|
||||
`${name}: ${JSON.stringify(result.content)}`,
|
||||
);
|
||||
return result.structuredContent;
|
||||
};
|
||||
const selectedFields = [
|
||||
"kind",
|
||||
"declared_name",
|
||||
"name",
|
||||
"exported",
|
||||
"exported_default",
|
||||
"permission",
|
||||
"permission_source",
|
||||
"enabled",
|
||||
"enabled_declaration",
|
||||
"process",
|
||||
"intent_filters",
|
||||
"meta_data",
|
||||
"location",
|
||||
];
|
||||
const stripLocations = (row) => {
|
||||
const { location: _location, ...fields } = row;
|
||||
return {
|
||||
...fields,
|
||||
intent_filters: row.intent_filters?.map(
|
||||
({
|
||||
location: _filterLocation,
|
||||
source_index: _filterIndex,
|
||||
coverage: _coverage,
|
||||
...filter
|
||||
}) => ({
|
||||
...filter,
|
||||
data: filter.data.map(({ location: _dataLocation, ...data }) => data),
|
||||
}),
|
||||
),
|
||||
meta_data: row.meta_data?.map(
|
||||
({ location: _metadataLocation, ...metadata }) => metadata,
|
||||
),
|
||||
};
|
||||
};
|
||||
const assertOracleFacts = (row, expected, source) => {
|
||||
assert.equal(row.declaration_coverage, "complete");
|
||||
const cleaned = stripLocations(row);
|
||||
for (const [key, value] of Object.entries(expected))
|
||||
assert.deepEqual(
|
||||
cleaned[key],
|
||||
value,
|
||||
`Independent oracle: ${expected.source_index}.${key}`,
|
||||
);
|
||||
assert.equal(
|
||||
source.slice(
|
||||
row.location.offset,
|
||||
row.location.offset + row.kind.length + 1,
|
||||
),
|
||||
"<" + row.kind,
|
||||
);
|
||||
};
|
||||
try {
|
||||
await client.connect(transport);
|
||||
catalog = await client.listTools();
|
||||
await writeFile(join(output, "catalog.json"), JSON.stringify(catalog));
|
||||
for (const temperature of ["first-capture", "warm-session"]) {
|
||||
const firstCall = calls.length;
|
||||
const start = performance.now();
|
||||
const result = await invoke(
|
||||
"inspect_android_package",
|
||||
values.mode === "before"
|
||||
? { path: target }
|
||||
: {
|
||||
path: target,
|
||||
detail: "summary",
|
||||
view: {
|
||||
kind: "android",
|
||||
facet: "components",
|
||||
actions: [values.action],
|
||||
fields: selectedFields,
|
||||
limit: 64,
|
||||
},
|
||||
},
|
||||
);
|
||||
const rpcMs = performance.now() - start;
|
||||
const normalized = result.normalized_result;
|
||||
const parentId =
|
||||
values.mode === "before"
|
||||
? result.evidence_id
|
||||
: normalized.parent_evidence_id;
|
||||
const path = join(output, temperature + "-evidence.json");
|
||||
await invoke("export_evidence_bundle", { path });
|
||||
const bundle = JSON.parse(await readFile(path, "utf8"));
|
||||
const parent = bundle.records.find(
|
||||
(record) => record.evidence_id === parentId,
|
||||
);
|
||||
assert.ok(parent);
|
||||
assert.equal(parent.subject.digest.sha256, sha256);
|
||||
const oraclePath = join(
|
||||
dirname(fileURLToPath(import.meta.url)),
|
||||
"fixtures/android-components/manifest-oracle.py",
|
||||
);
|
||||
const oracle = JSON.parse(
|
||||
(
|
||||
await commands.exec(
|
||||
"python3",
|
||||
[oraclePath, path, parentId, values.action],
|
||||
{ timeout: 15000, maxBuffer: 4 * 1024 * 1024 },
|
||||
)
|
||||
).stdout,
|
||||
);
|
||||
assert.ok(
|
||||
oracle.components.length > 0,
|
||||
"The completed task must exercise a real positive action match",
|
||||
);
|
||||
const complete = parent.normalized_result;
|
||||
assert.equal(complete.components.items.length, oracle.inventory.length);
|
||||
assert.equal(oracle.inventory.length, oracle.source_components);
|
||||
for (const [index, expected] of oracle.inventory.entries()) {
|
||||
const row = complete.components.items[index];
|
||||
if (values.mode === "after")
|
||||
assertOracleFacts(
|
||||
{ ...row, source_index: index },
|
||||
expected,
|
||||
complete.manifest.text,
|
||||
);
|
||||
else {
|
||||
for (const key of ["kind", "declared_name", "name", "permission"])
|
||||
assert.deepEqual(
|
||||
row[key],
|
||||
expected[key],
|
||||
`Legacy inventory: ${index}.${key}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
if (values.mode === "after") {
|
||||
assert.equal(normalized.facet, "components");
|
||||
assert.equal(normalized.source_status, "complete");
|
||||
assert.equal(normalized.source_components, oracle.source_components);
|
||||
assert.equal(normalized.coverage.exhausted, true);
|
||||
assert.equal(normalized.coverage.total, oracle.components.length);
|
||||
assert.equal(normalized.items.length, oracle.components.length);
|
||||
for (let i = 0; i < oracle.components.length; i++) {
|
||||
const row = normalized.items[i],
|
||||
expected = oracle.components[i];
|
||||
assertOracleFacts(row, expected, complete.manifest.text);
|
||||
}
|
||||
} else {
|
||||
for (const expected of oracle.components) {
|
||||
const row = normalized.components.items[expected.source_index];
|
||||
assert.equal(row.name, expected.name);
|
||||
assert.ok(row.actions.includes(values.action));
|
||||
}
|
||||
}
|
||||
runs.push({
|
||||
temperature,
|
||||
rpc_ms: rpcMs,
|
||||
calls: calls.length - firstCall,
|
||||
matched_components: oracle.components.length,
|
||||
source_components: oracle.source_components,
|
||||
parent_evidence_id: parentId,
|
||||
answer_verified: true,
|
||||
inventory_verified: oracle.inventory.length,
|
||||
inventory_scope:
|
||||
values.mode === "after"
|
||||
? "declared-structured-facets"
|
||||
: "legacy-identity-permission",
|
||||
});
|
||||
}
|
||||
} finally {
|
||||
// exec drains already adopted owners; adopt the live MCP client only at its final cleanup boundary.
|
||||
commands.adopt(client, "real Android component MCP client");
|
||||
await commands.close();
|
||||
await writeFile(join(output, "stderr.log"), Buffer.concat(diagnostics));
|
||||
await writeFile(join(output, "calls.json"), JSON.stringify(calls), {
|
||||
mode: 0o600,
|
||||
});
|
||||
await writeFile(
|
||||
join(output, "report.json"),
|
||||
JSON.stringify(
|
||||
{
|
||||
mode: values.mode,
|
||||
target,
|
||||
sha256,
|
||||
runs,
|
||||
tool_count: catalog?.tools.length,
|
||||
omitted_diagnostic_bytes: omittedDiagnosticBytes,
|
||||
},
|
||||
null,
|
||||
2,
|
||||
),
|
||||
);
|
||||
}
|
||||
console.log(JSON.stringify({ mode: values.mode, runs, verified: true }));
|
||||
@@ -1,166 +0,0 @@
|
||||
# Android package summary and exact retained manifest
|
||||
|
||||
Continues [#2016](https://github.com/morluto/rea/issues/2016) and
|
||||
[#2438](https://github.com/morluto/rea/pull/2438). The recurring NewPipe question
|
||||
asks for APK/package/version identity, all case-sensitive MainActivity matches
|
||||
and all reported onCreate entries, including explicit null DEX descriptors.
|
||||
Complete package replies delivered decoded manifest text and duplicate upstream
|
||||
calls unrelated to those selected facts.
|
||||
|
||||
## Compatibility and ownership
|
||||
|
||||
MCP `inspect_android_package` accepts explicit `detail: "summary"`; omitted and
|
||||
`"complete"` requests preserve the complete output and provider request. The
|
||||
one-shot CLI stays complete. Summary selection happens in the application
|
||||
workflow, independently of the engine protocol, using the existing retained
|
||||
analysis-view machinery. No third-party plugin API or persistent analysis cache
|
||||
is introduced. Servers lacking retention reject summary before provider work.
|
||||
|
||||
The complete original Evidence stays under its original ID and session lifetime.
|
||||
A distinct linked derived view returns all package/SDK/permission declarations,
|
||||
engine identity, class/resource counts and signature-verification status.
|
||||
Manifest metadata retains the producer's partial/complete status, nullable
|
||||
reported byte total, and observed UTF-16 length. Text and duplicate raw calls
|
||||
remain in the original record, rather than being rewritten under its ID.
|
||||
|
||||
Use `inspect_analysis_view` with the parent's ID and
|
||||
`view: {kind: "android", facet: "manifest", offset: 0, limit: 4096}` to read exact
|
||||
retained decoded text. Follow `coverage.next_offset`; surrogate boundaries use
|
||||
the existing text-page implementation. Exhaustion describes the retained text,
|
||||
not global completeness when the producer says partial. APK signatures and
|
||||
runtime behavior remain unverified. Engine artifact SHA-256 identifies the JAR;
|
||||
views require the authenticated APK subject and never substitute the engine
|
||||
hash for a missing target identity. Older servers require schema discovery and
|
||||
the original complete request; automatic fallback is not introduced.
|
||||
|
||||
## Frozen runtime and real SDK pairs
|
||||
|
||||
Source runtime: `28c84e78cf2fbfde4aceed345b773c59eb889f68`; later test-only schema
|
||||
typing corrections do not relabel these receipts. Compiled dist/bridge file-table
|
||||
SHA-256: `5332dda13981f043e146b3601ebec1f380d9f698280ca53d75786b638a912e11`.
|
||||
The frozen old runtime is `8bc1601ca51e417ca472d90ea0367e9418a1a0f2`, digest
|
||||
`9e7380f537301bf0cfe2926f5619011af848af7c220750e75189498e65d825e1`.
|
||||
|
||||
Real NewPipe 0.29.1 bytes, acquisition and digest are pinned in `corpus.json`.
|
||||
Actual configured JADX headless MCP 0.7.1/Java runs use the same selected
|
||||
provider settings and search-path digests, one worker and 768 MiB Java heap.
|
||||
Node is 24.18.0; actual SDK client/server are 2.3.1. Processes and engines are
|
||||
fresh per trial; host page caches are not flushed. SDK schema mode is selected
|
||||
in both treatments. Heavy runs are serial, with one CPU, unchanged 3 GiB early
|
||||
stop/4 GiB hard memory, no swap, and 192 tasks.
|
||||
|
||||
All four candidate workflows use three successful public MCP calls. Only the
|
||||
first package delivery changes. Both pairs pass settings and selected-fact
|
||||
comparison, including all declarations, manifest status/length, engine identity,
|
||||
original limitations, matching class rows and onCreate entries. Original replies
|
||||
and IDs remain separate. A fresh old-runtime complete workflow also passes
|
||||
comparison against the candidate complete request; both complete package replies
|
||||
are exactly 119,527 compact client-result JSON bytes. This is scoped compatibility
|
||||
evidence, not a guarantee for every historical release/client/host.
|
||||
|
||||
| Trial | Package reply bytes | SDK workflow seconds | Payload CPU seconds | Kernel peak bytes |
|
||||
| -------------- | ------------------: | -------------------: | ------------------: | ----------------: |
|
||||
| sdk-complete-1 | 119,527 | 24.059 | 25.259 | 1,225,076,736 |
|
||||
| sdk-summary-1 | 8,659 | 23.694 | 24.894 | 1,226,870,784 |
|
||||
| sdk-summary-2 | 8,659 | 24.407 | 25.611 | 1,228,296,192 |
|
||||
| sdk-complete-2 | 119,527 | 24.509 | 25.740 | 1,229,844,480 |
|
||||
|
||||
Summary replies are 8,659 bytes, about 92.8% smaller; all three replies together
|
||||
fall from 149,577 to 38,709 bytes. This is a delivery result, not a model-token
|
||||
or engine-work saving. The SDK timing differences are small. CPU and memory do
|
||||
not establish a reduction. No provider setting or analysis scope changes.
|
||||
Transport roundtrip includes server/client work and is not an engine-only timer.
|
||||
|
||||
The complete catalog is 2,506,213 compact JSON bytes versus 2,497,635 on the old
|
||||
runtime: about 0.34% growth, including intervening main changes. Discovery cost
|
||||
remains visible alongside selected-delivery savings.
|
||||
|
||||
## Authentic old Evidence continuation
|
||||
|
||||
A real package record from old runtime 8bc160 imports into the candidate and
|
||||
supports retained summary, exact manifest pages, inline portable summary and
|
||||
export through actual public SDK tools. The new session explicitly selects an
|
||||
absent JADX JAR; this workflow performs no package analysis. Eleven successful
|
||||
calls preserve the original record, original ID, all declarations and all 25,222
|
||||
manifest UTF-16 units. Manifest UTF-8 SHA-256:
|
||||
`fee192fe2a6d7892c22cba115184c74c7e2c42ce464e8bb37be6289f2c16881d`.
|
||||
The exported original record is deeply equal to the imported record.
|
||||
|
||||
## Actual Codex task outcomes
|
||||
|
||||
Four serial trials run in complete/summary/summary/complete order on the same
|
||||
frozen candidate. Requested configuration: Codex CLI 0.161.0, `gpt-5.6-terra` high,
|
||||
three exact selected tool names and the same optional context/discovery capture
|
||||
settings. The requested alias does not certify the server-resolved model. The
|
||||
original analyst question is unchanged; equal treatment instructions explicitly
|
||||
choose package detail and explain the summary carrier. Temporary configuration
|
||||
points to the frozen modified REA MCP; no global configuration is edited.
|
||||
|
||||
| Trial | Input | Cached | Uncached | Output | Seconds | MCP calls | Usage records | Facts |
|
||||
| ---------------- | ------: | -----: | -------: | -----: | ------: | --------: | ------------: | ----- |
|
||||
| model-complete-1 | 39,267 | 23,040 | 16,227 | 1,690 | 94.053 | 3 | 3 | Pass |
|
||||
| model-summary-1 | 128,417 | 93,440 | 34,977 | 3,633 | 150.401 | 8 | 9 | Pass |
|
||||
| model-summary-2 | 54,437 | 37,120 | 17,317 | 2,185 | 123.677 | 3 | 4 | Pass |
|
||||
| model-complete-2 | 21,332 | 9,984 | 11,348 | 667 | 20.774 | 0 | 2 | Fail |
|
||||
|
||||
Three answers pass the original selected-fact gate against the actual SDK
|
||||
producer. Each trial preserves APK bytes and removes its isolated profile.
|
||||
The first summary trial makes five successful calls and three rejected requests:
|
||||
an invented `case_sensitive` argument, empty class-search input and empty
|
||||
class-inspection input. It then repeats successful package/search work. These
|
||||
errors and correction turns remain part of measured task cost; strict existing
|
||||
schemas correctly reject them. The second summary trial uses the intended three
|
||||
successful calls, but still does not establish improved total/uncached input or
|
||||
latency against the factually acceptable first complete control.
|
||||
|
||||
The final complete control's first client executor filters `ALL_TOOLS` for the
|
||||
same three exact names and receives an empty array. It makes zero MCP calls,
|
||||
reports all requested facts unknown and fails the factual gate. Identical name
|
||||
filters worked in the other trials. Stderr contains no explanatory error; the
|
||||
client visibility failure remains unexplained and is not attributed to REA or
|
||||
summary delivery. Its short duration is not comparable completed work. The
|
||||
original failed receipt is retained rather than replaced by a successful trial.
|
||||
|
||||
These measurements demonstrate no real-model token or speed gain. Smaller raw
|
||||
MCP envelopes do not establish smaller model context; observed usage also
|
||||
includes discovery, execution context, corrections and extra responses. Usage
|
||||
records count captured usage items rather than asserting exact remote request
|
||||
contents or equating MCP calls with model rounds. The complete default and
|
||||
opt-in choice remain unchanged; this is useful delivery/retention capability,
|
||||
not a general performance recommendation.
|
||||
|
||||
All model runs have verified payload-completion witnesses with only the observer
|
||||
remaining and zero memory/task limit events at that phase. Kernel peaks are
|
||||
602,472,448–1,444,945,920 bytes; the failed no-analysis control is included in that
|
||||
range and is not evidence of lower analysis memory. Whole-scope final counters
|
||||
remain unknown and the existing full resource grader stays false. No broader
|
||||
CPU, memory, OS/provider or model claim is made.
|
||||
|
||||
## Resource interpretation
|
||||
|
||||
Every SDK run and portable continuation has a verified owned
|
||||
payload-exited-observer-alive witness, with only the observer remaining and no
|
||||
memory/task limit events at that phase. Kernel peak and CPU include that opt-in
|
||||
observer. Whole-scope terminal counters remain unknown; the existing complete
|
||||
physical resource grader is not changed or called successful. No OOM stress or
|
||||
resource-limit increase is used.
|
||||
|
||||
## Verification and scope
|
||||
|
||||
Twenty-seven relevant converted-schema, SDK, domain and existing Android
|
||||
failure/cancellation/cleanup checks pass. A three-case focused rerun passes after
|
||||
a test-only schema typing correction. The 1 MiB manifest protocol fixture verifies
|
||||
original default/full output, reconstruction after source removal, unchanged
|
||||
exported Evidence and one engine launch. Other cases preserve nulls and partial
|
||||
status, Unicode boundaries and tagged malformed/missing-identity failures, and
|
||||
reject unavailable retention or cancelled requests before provider work. These
|
||||
fixtures verify production boundaries; actual JADX claims use the real NewPipe
|
||||
runs above. Full supported single-thread compiler and unchanged lint pass under
|
||||
the same memory safeguards. No other Android OS/device/provider lane is inferred.
|
||||
|
||||
Raw SDK/model replies, private profiles and resource receipts stay outside Git.
|
||||
Reproduce SDK pairs using the existing `measure.mjs` with `--artifact newpipe.apk`,
|
||||
`--schemas selected`, identical explicit provider configuration and frozen runtime,
|
||||
and alternating `--delivery complete` / `--delivery summary`. Preserve failed
|
||||
attempts and run `compare.mjs` on both pair orders. Model findings require their
|
||||
own identical analyst facts and explicit delivery controls.
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,171 +0,0 @@
|
||||
# Apktool shared decoding and table-only projection
|
||||
|
||||
The measured question is the same in every comparison: inspect an APK's
|
||||
metadata/manifest and its default, German and Brazilian Portuguese string tables.
|
||||
One existing batch removes repeated decoding. An optional table-only projection
|
||||
then removes repeated primary strings without changing the selected facts.
|
||||
|
||||
This is a static resource-query measurement, not an agent evaluation or a claim
|
||||
about reconstructing an application's behavior. The machine-readable measured
|
||||
summary is [apktool-string-tables.json](apktool-string-tables.json).
|
||||
|
||||
## Revisions, inputs and execution
|
||||
|
||||
- Baseline: `8f7d12e30d63f3d3b4cfe2a90336362841438f56`, latest main at issue start.
|
||||
- Projection source: `406af90023a545fb280601bf51f445e122c36378`.
|
||||
- Existing shared decode: [#4777](https://github.com/morluto/rea/issues/4777),
|
||||
integrated by [#4804](https://github.com/morluto/rea/pull/4804).
|
||||
- Projection follow-up: [#4982](https://github.com/morluto/rea/issues/4982).
|
||||
- Linux x86-64, kernel 6.1.0-53-cloud-amd64, Node 24.18.0, npm 11.16.0,
|
||||
Temurin 21.0.12.1, Apktool 3.0.3. The selected JAR's closeout digest is
|
||||
`dbf930b076c6b9be08d57c449cacefc3bdd6b71ebd59b3066fc0e1f5b14f9423`;
|
||||
decoder contents were not independently hashed for every request.
|
||||
|
||||
- [newpipe](https://github.com/TeamNewPipe/NewPipe/releases/tag/v0.29.1): 11,507,122 bytes; SHA-256 `18447bfb1e06d113edc88df93f471827280de06f6f3d4dc42f56f28b9c1bab79`. Valid projected counts: 1126/833/779; unsupported entries: 1/1/1.
|
||||
- [apidemos](https://github.com/appium/android-apidemos/releases/tag/v6.0.18): 6,521,217 bytes; SHA-256 `a9eecf37b26cd084855c530db81c2bb1b91f4c1b095a04f47aa7c20e2791f686`. Valid projected counts: 967/29/29; unsupported entries: 8/0/0.
|
||||
- [settings](https://github.com/appium/io.appium.settings/releases/tag/v8.2.1): 3,099,589 bytes; SHA-256 `e4024b3ee348929252bcaf5eb422bcab9e4ace96703499a84f3332281c03375f`. Valid projected counts: 25/19/19; unsupported entries: 0/0/0.
|
||||
|
||||
The counts are default/de/pt-BR respectively. Source paths, XML digests, exact
|
||||
caller paths, runtime hashes and per-run identities/costs are in the JSON summary.
|
||||
The APKs were never installed or executed. Input digests were checked before and
|
||||
after each completed task; provider-owned workspaces were confirmed absent.
|
||||
|
||||
Each task uses a fresh production stdio MCP runtime, complete paginated tool
|
||||
discovery, client inspection, resource call(s), `close_binary`, transport close
|
||||
and owned cleanup. OS file caches were not flushed. Every decode has a fresh JVM,
|
||||
APK snapshot, framework directory and workspace: these are not warm retained
|
||||
engine/database queries, nor cold-disk measurements.
|
||||
|
||||
Tasks run serially on one logical CPU, with an 8 GiB family hard limit, 7 GiB
|
||||
early stop and no swap. Admission requires 16 GiB host availability; running
|
||||
tasks stop below 8 GiB host availability. CPU and kernel `memory.peak` cover
|
||||
the complete owned cgroup, including subprocesses, observer and file cache.
|
||||
Counters are captured at payload exit while the observer is alive; disappeared
|
||||
final whole-scope counters remain unknown. All completed payload receipts show
|
||||
successful owned cleanup and zero observed OOM/limit events at that checkpoint.
|
||||
|
||||
## Correctness and owning change
|
||||
|
||||
Separate native decodes were retained for independent Python Expat validation.
|
||||
It checks direct string entries, source entity spelling, unsupported markup
|
||||
counts, ordering and emitted prefixes against the XML bytes. Manifest text and
|
||||
package identity are checked separately. All six serial/batch pairs agree on
|
||||
selected strings, manifest, metadata and target identity. All nine projection
|
||||
pairs additionally preserve table counts and coverage; batch/table-only counts
|
||||
and coverage match the XML oracle. This qualifies REA's projection of these
|
||||
native outputs, not independent binary ARSC decoder completeness.
|
||||
|
||||
The existing default batch repeats its primary table in top-level `strings` and
|
||||
`string_tables[0]`. On NewPipe this made the full resource replies larger than
|
||||
three serial replies despite reducing decode calls: 616,059 → 680,188 bytes.
|
||||
ApiDemos' larger common manifest produces the opposite tradeoff. The existing
|
||||
batch remains the compatibility default.
|
||||
|
||||
`string_tables_only: true` (CLI `--string-tables-only`) keeps the complete
|
||||
requested table facts inline in `string_tables` and leaves `strings` empty by
|
||||
request. An unrequested field does not establish that a source table is empty.
|
||||
This also supports one table without `locales`. `include_strings: false` still
|
||||
suppresses string projection. Omitting the new option preserves old requests:
|
||||
three real legacy workflows produce identical complete MCP replies, including
|
||||
Evidence identity, at the new source revision.
|
||||
|
||||
The resource projection owner computes the output choice and charges aggregate
|
||||
entries for the copies actually emitted. The same 10,000-entry policy can thus
|
||||
return five distinct 2,000-entry tables rather than consuming capacity on a
|
||||
duplicate primary copy. Per-table limits, XML budgets, physical alias reuse,
|
||||
coverage, partial observations and cancellation remain at the existing owner.
|
||||
CLI and MCP share this implementation; there is no new tool or cross-call cache.
|
||||
|
||||
## Existing shared decode: three serial calls versus one batch
|
||||
|
||||
Three alternating fresh-session pairs per APK, all at the same main revision.
|
||||
The table reports marginal medians; paired speedups are reported separately.
|
||||
Elapsed/CPU include discovery, client/Java startup, calls, transcript/hash checks
|
||||
and cleanup. Native oracle creation, tokenization and verification gates are
|
||||
separate costs in the JSON, not hidden inside these task timings.
|
||||
|
||||
| Target | Resource calls, s | Whole elapsed, s | Whole CPU, s | Family peak, MiB |
|
||||
| -------- | ----------------: | ---------------: | ------------: | ---------------: |
|
||||
| newpipe | 23.19 → 7.40 | 35.32 → 19.86 | 34.95 → 19.65 | 1341.7 → 1309.5 |
|
||||
| apidemos | 17.43 → 6.22 | 30.32 → 18.85 | 30.05 → 18.63 | 1187.1 → 1202.7 |
|
||||
|
||||
Paired resource-call speedup medians: NewPipe **3.23×** (2.90–3.23), ApiDemos
|
||||
**2.80×** (2.66–3.09). Instrumentation confirms three versus one actual decoder
|
||||
invocations. These call timings include protocol/projection costs; they do not
|
||||
isolate the Java engine phase. NewPipe's whole raw token estimate increases
|
||||
951,332 → 962,330 with the default batch; ApiDemos decreases
|
||||
1,054,893 → 948,591. Fewer calls alone do not establish less context.
|
||||
|
||||
## New selector: default batch versus table-only batch
|
||||
|
||||
Three alternating pairs per APK; NewPipe repetitions 1/4/3, others 1/2/3.
|
||||
Repetition 4 replaces a refused start, described below. Both variants still
|
||||
perform one decode and return the same required facts.
|
||||
|
||||
| Target | Resource call, s | Whole elapsed, s | Whole CPU, s | Family peak, MiB |
|
||||
| -------- | ---------------: | ---------------: | ------------: | ---------------: |
|
||||
| newpipe | 9.18 → 9.35 | 23.95 → 24.85 | 23.65 → 23.82 | 1327.5 → 1323.6 |
|
||||
| apidemos | 6.46 → 5.81 | 23.28 → 19.79 | 22.90 → 19.47 | 1247.8 → 1197.0 |
|
||||
| settings | 4.21 → 4.06 | 17.66 → 19.39 | 17.42 → 19.04 | 1191.6 → 1122.4 |
|
||||
|
||||
There is no consistent latency or whole-family memory improvement established
|
||||
for the selector. NewPipe and Settings have higher whole-task elapsed and CPU
|
||||
medians. Elapsed ranges overlap: NewPipe 23.53–34.46 → 21.59–42.80 s;
|
||||
ApiDemos 18.55–27.07 → 19.25–25.96 s; Settings 16.55–23.39 → 16.14–21.72 s.
|
||||
The cause of this variation was not established; a passing or faster run does
|
||||
not explain a slower one. The selector's supported benefit is reduced duplicate
|
||||
delivery, with its full task costs retained here.
|
||||
|
||||
Full resource reply bytes fall 680,188 → 506,481 for NewPipe,
|
||||
616,841 → 463,588 for ApiDemos and 38,725 → 34,199 for Settings.
|
||||
|
||||
Offline `tiktoken` 0.12.0 / `cl100k_base` estimates count one complete catalog
|
||||
page-array plus every call name/argument/reply, including inspection and close.
|
||||
Full replies include both textual and structured copies. Text-only replies are
|
||||
also counted separately. These are serialization estimates, not actual model
|
||||
usage; initialization/framing, prompts, client filtering/rendering/replay and
|
||||
continuations are unmeasured.
|
||||
|
||||
| Target | Text reply tokens | Whole raw transport tokens | Whole estimated reduction |
|
||||
| -------- | ----------------: | -------------------------: | ------------------------: |
|
||||
| newpipe | 73,661 → 55,797 | 962,330 → 924,352 | 3.95% |
|
||||
| apidemos | 65,693 → 49,162 | 948,591 → 913,693 | 3.68% |
|
||||
| settings | 4,387 → 3,942 | 814,859 → 814,029 | 0.10% |
|
||||
|
||||
Discovery still has 207 tools; only `decode_android_resources` changes.
|
||||
The added schema costs 557 catalog bytes / 113 estimated catalog tokens
|
||||
(805,217 → 805,330). Raw discovery dominates this transport estimate; it does
|
||||
not establish what a client supplies to its model. Smaller replies and these
|
||||
estimates do not prove an actual agent context or investigation advantage.
|
||||
|
||||
## Verification, attempts and limits
|
||||
|
||||
- 94 source tests: provider/resource projection/service, including alias reads,
|
||||
aggregate output exhaustion, malformed/physical failures, partial facts,
|
||||
cancellation and cleanup. The table-only capacity, single-table provider
|
||||
and CLI cases fail on the old implementation (three selected failures,
|
||||
82 unrelated tests skipped).
|
||||
- Nine compiled CLI/MCP tests: schema conversion/advertisement, option forwarding,
|
||||
Evidence parameters, legacy locale grammar and existing boolean choices.
|
||||
- 36 completed real MCP tasks and three real CLI tasks. Nine exact projection
|
||||
oracle pairs; three legacy full-reply comparisons; alias/missing, single-table
|
||||
and omitted-string cases pass. CLI normalized data, subjects and Evidence IDs
|
||||
match MCP exactly on all three APKs.
|
||||
- `check:fast`, `build:cached` and `verify:test-discovery` passed; the last
|
||||
discovered all 1,579 test files exactly once. `npm run check` and `npm run docs:check` also passed.
|
||||
- An authored test group initially exceeded the 150-line lint limit (153).
|
||||
That gate failed after 189.31 s / 186.19 CPU s, peak 4,487,798,784 bytes;
|
||||
build/compiled checks did not run in that attempt. The capacity test was moved
|
||||
out of that group, and the complete corrected gate passed after 343.97 s /
|
||||
338.67 CPU s, peak 4,311,293,952 bytes. No rule was disabled.
|
||||
- `projection-newpipe-after-2` was refused before scope/payload creation because
|
||||
host availability was below 16 GiB; a following poll showed about 10.5 GiB.
|
||||
It has no payload timing/counters and is not a completed measurement. The
|
||||
refusal stays recorded; a fresh whole repetition-4 pair ran after recovery.
|
||||
No existing receipts were overwritten and resource thresholds were unchanged.
|
||||
|
||||
Raw transcripts, native outputs, intermediate failures and per-request workspace
|
||||
arguments remain in ignored local audit storage; the committed JSON is a curated
|
||||
summary. Small Settings resources and negative time results are retained.
|
||||
Actual models, APK execution, other providers/platforms and general complex
|
||||
software effectiveness were not verified by this checkpoint.
|
||||
@@ -1,713 +0,0 @@
|
||||
/** Compare complete real PocketBase create/update/read tasks through public MCP. */
|
||||
import assert from "node:assert/strict";
|
||||
import { readFile, writeFile, mkdir, mkdtemp, rm } from "node:fs/promises";
|
||||
import { join, resolve } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import { createRequire } from "node:module";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { parseArgs } from "node:util";
|
||||
import { performance } from "node:perf_hooks";
|
||||
import { createVerifierCommands } from "../../scripts/lib/verify-commands.mjs";
|
||||
import { withVerifierCleanup } from "../../scripts/lib/verifier-cleanup.mjs";
|
||||
const { values } = parseArgs({
|
||||
options: {
|
||||
repository: { type: "string" },
|
||||
mode: { type: "string" },
|
||||
output: { type: "string" },
|
||||
origin: { type: "string" },
|
||||
credentials: { type: "string" },
|
||||
browser: { type: "string" },
|
||||
"source-only": { type: "boolean", default: false },
|
||||
},
|
||||
});
|
||||
assert.ok(
|
||||
values.repository &&
|
||||
values.output &&
|
||||
values.origin &&
|
||||
values.credentials &&
|
||||
values.browser,
|
||||
);
|
||||
assert.ok(["before", "after"].includes(values.mode));
|
||||
const repo = resolve(values.repository),
|
||||
output = resolve(values.output),
|
||||
origin = new URL(values.origin).origin;
|
||||
assert.equal(
|
||||
new URL(origin).hostname,
|
||||
"127.0.0.1",
|
||||
"Use an owned loopback fixture",
|
||||
);
|
||||
await mkdir(output, { mode: 0o700 });
|
||||
const credentials = JSON.parse(await readFile(values.credentials, "utf8"));
|
||||
const require = createRequire(join(repo, "package.json"));
|
||||
const { chromium } = require("playwright-core");
|
||||
const { Client } = await import(
|
||||
pathToFileURL(require.resolve("@modelcontextprotocol/client"))
|
||||
);
|
||||
const { StdioClientTransport } = await import(
|
||||
pathToFileURL(require.resolve("@modelcontextprotocol/client/stdio"))
|
||||
);
|
||||
const commands = await createVerifierCommands();
|
||||
const profile = await mkdtemp(join(tmpdir(), "rea-saved-values-"));
|
||||
const env = {
|
||||
PATH: process.env.PATH,
|
||||
NODE_OPTIONS: "--max-old-space-size=3072",
|
||||
UV_THREADPOOL_SIZE: "2",
|
||||
REA_LOG_LEVEL: "silent",
|
||||
REA_MCP_TOOL_FAMILIES: "browser,application,session",
|
||||
REA_MCP_INPUT_SCHEMA_PROFILE: "compact",
|
||||
REA_TEST_EMAIL: credentials.email,
|
||||
REA_TEST_PASSWORD: credentials.password,
|
||||
REA_TEST_WHOLE_JSON: JSON.stringify({ token: "private-token-729d" }),
|
||||
};
|
||||
const calls = [];
|
||||
const runs = [];
|
||||
let catalog;
|
||||
const transport = new StdioClientTransport({
|
||||
command: process.execPath,
|
||||
args: [join(repo, "scripts/rea.mjs"), "mcp"],
|
||||
cwd: repo,
|
||||
env,
|
||||
stderr: "pipe",
|
||||
});
|
||||
transport.stderr?.resume();
|
||||
const client = new Client({ name: "rea-saved-values-real-app", version: "1" });
|
||||
commands.adopt(client, "saved-values MCP client");
|
||||
const tokenHeader = (saved) => ({
|
||||
name: "Authorization",
|
||||
value: saved
|
||||
? { source: "saved", name: "token" }
|
||||
: { source: "local_storage", key: "rea_auth", json_pointer: "/token" },
|
||||
});
|
||||
const decodeBodies = (value) => {
|
||||
const bodies = [];
|
||||
const visit = (x) => {
|
||||
if (x && typeof x === "object") {
|
||||
if (x.encoding === "base64" && typeof x.content === "string") {
|
||||
try {
|
||||
bodies.push(
|
||||
JSON.parse(Buffer.from(x.content, "base64").toString("utf8")),
|
||||
);
|
||||
} catch {}
|
||||
}
|
||||
for (const v of Object.values(x)) visit(v);
|
||||
}
|
||||
};
|
||||
visit(value);
|
||||
return bodies;
|
||||
};
|
||||
async function invoke(name, args) {
|
||||
const defs = catalog.tools.filter((t) => t.name === name);
|
||||
assert.equal(defs.length, 1);
|
||||
const start = performance.now();
|
||||
const result = await client.callTool(
|
||||
{ name, arguments: args },
|
||||
{ toolDefinition: defs[0], timeout: 90000 },
|
||||
);
|
||||
calls.push({
|
||||
name,
|
||||
arguments: args,
|
||||
result,
|
||||
elapsed_ms: performance.now() - start,
|
||||
});
|
||||
assert.notEqual(
|
||||
result.isError,
|
||||
true,
|
||||
`${name} failed; retain the local receipt`,
|
||||
);
|
||||
return result.structuredContent;
|
||||
}
|
||||
const loginActions = (saved) => [
|
||||
{
|
||||
step_id: "email",
|
||||
action: "fill",
|
||||
locator: { kind: "css", selector: "#email" },
|
||||
value: { source: "secret", secret_id: "email" },
|
||||
},
|
||||
{
|
||||
step_id: "password",
|
||||
action: "fill",
|
||||
locator: { kind: "css", selector: "#password" },
|
||||
value: { source: "secret", secret_id: "password" },
|
||||
},
|
||||
{
|
||||
step_id: "login",
|
||||
action: "click",
|
||||
locator: { kind: "css", selector: "#login" },
|
||||
...(saved
|
||||
? {
|
||||
save_response: {
|
||||
method: "POST",
|
||||
path: "/api/collections/_superusers/auth-with-password",
|
||||
save: [{ name: "token", json_pointer: "/token", secret: true }],
|
||||
},
|
||||
}
|
||||
: {}),
|
||||
timeout_ms: 10000,
|
||||
},
|
||||
{
|
||||
step_id: "ready",
|
||||
action: "wait_for",
|
||||
locator: { kind: "css", selector: '#result[data-ready="yes"]' },
|
||||
state: "visible",
|
||||
timeout_ms: 10000,
|
||||
},
|
||||
];
|
||||
const edits = (id, saved) => [
|
||||
{
|
||||
step_id: "update",
|
||||
action: "request",
|
||||
method: "PATCH",
|
||||
path: `/api/collections/rea_items/records/${id}`,
|
||||
headers: [tokenHeader(saved)],
|
||||
json: { title: `record-${id}`, quantity: saved ? "{saved:quantity}" : 11 },
|
||||
expected_status: [200],
|
||||
},
|
||||
{
|
||||
step_id: "read",
|
||||
action: "request",
|
||||
method: "GET",
|
||||
path: `/api/collections/rea_items/records/${id}?at=${id}`,
|
||||
headers: [tokenHeader(saved)],
|
||||
expected_status: [200],
|
||||
},
|
||||
{
|
||||
step_id: "open",
|
||||
action: "goto",
|
||||
destination: { url: `${origin}/api/collections/rea_items/records/${id}` },
|
||||
wait_until: "load",
|
||||
},
|
||||
];
|
||||
const captureOptions = {
|
||||
after_each_step: ["url"],
|
||||
at_end: ["url", "accessibility"],
|
||||
events: ["network"],
|
||||
network: { request_body: true, response_body: true, header_values: true },
|
||||
};
|
||||
try {
|
||||
const context = await chromium.launchPersistentContext(profile, {
|
||||
executablePath: resolve(values.browser),
|
||||
headless: true,
|
||||
args: ["--remote-debugging-port=0", "--no-sandbox"],
|
||||
});
|
||||
commands.adopt(context, "owned saved-values Chromium context");
|
||||
const page = context.pages()[0];
|
||||
await page.goto(origin, { waitUntil: "networkidle" });
|
||||
const port = (
|
||||
await readFile(join(profile, "DevToolsActivePort"), "utf8")
|
||||
).split("\n")[0];
|
||||
const endpoint = `http://127.0.0.1:${port}`;
|
||||
const targets = await (await fetch(endpoint + "/json/list")).json();
|
||||
const target = targets.find(
|
||||
(t) => t.type === "page" && t.url === origin + "/",
|
||||
);
|
||||
assert.ok(target);
|
||||
const browser = {
|
||||
mode: "connect",
|
||||
cdp_endpoint: endpoint,
|
||||
target_id: target.id,
|
||||
};
|
||||
const declared = [
|
||||
{ secret_id: "email", environment_variable: "REA_TEST_EMAIL" },
|
||||
{ secret_id: "password", environment_variable: "REA_TEST_PASSWORD" },
|
||||
];
|
||||
const scenario = (actions, secrets = declared) => ({
|
||||
browser,
|
||||
start_url: { url: origin + "/" },
|
||||
secrets,
|
||||
actions,
|
||||
capture: captureOptions,
|
||||
detail: "summary",
|
||||
});
|
||||
await client.connect(transport);
|
||||
catalog = await client.listTools();
|
||||
await writeFile(join(output, "catalog.json"), JSON.stringify(catalog));
|
||||
for (const temperature of values["source-only"] ? [] : ["cold", "warm"]) {
|
||||
const saved = values.mode === "after";
|
||||
const start = performance.now(),
|
||||
firstCall = calls.length;
|
||||
const create = {
|
||||
step_id: "create",
|
||||
action: "request",
|
||||
method: "POST",
|
||||
path: "/api/collections/rea_items/records",
|
||||
headers: [tokenHeader(saved)],
|
||||
json: { title: `created-${temperature}`, quantity: 11 },
|
||||
expected_status: [200, 201],
|
||||
...(saved
|
||||
? {
|
||||
save: [
|
||||
{ name: "record_id", json_pointer: "/id" },
|
||||
{ name: "quantity", json_pointer: "/quantity" },
|
||||
],
|
||||
}
|
||||
: {}),
|
||||
};
|
||||
const firstActions = [...loginActions(saved), create];
|
||||
if (saved) firstActions.push(...edits("{saved:record_id}", true));
|
||||
const first = await invoke(
|
||||
"capture_browser_scenario",
|
||||
scenario(firstActions),
|
||||
);
|
||||
const firstSummary = first.normalized_result;
|
||||
assert.ok(
|
||||
firstSummary.summary.steps.every((step) => step.status === "completed"),
|
||||
"Initial capture must complete",
|
||||
);
|
||||
let final = first;
|
||||
let id;
|
||||
let legacyPreReadObserved;
|
||||
if (!saved) {
|
||||
const selected = await invoke("inspect_analysis_view", {
|
||||
source: {
|
||||
kind: "retained-evidence",
|
||||
evidence_id: firstSummary.parent_evidence_id,
|
||||
},
|
||||
view: {
|
||||
kind: "browser-scenario",
|
||||
collection: "events",
|
||||
step_indices: [firstActions.length],
|
||||
event_kinds: ["network-content"],
|
||||
limit: 16,
|
||||
},
|
||||
});
|
||||
assert.equal(selected.normalized_result.coverage.exhausted, true);
|
||||
const records = decodeBodies(selected.normalized_result).filter(
|
||||
(b) => b?.collectionName === "rea_items" && typeof b.id === "string",
|
||||
);
|
||||
assert.equal(records.length, 1);
|
||||
id = records[0].id;
|
||||
final = await invoke(
|
||||
"capture_browser_scenario",
|
||||
scenario(edits(id, false), []),
|
||||
);
|
||||
}
|
||||
assert.ok(
|
||||
final.normalized_result.summary.steps.every(
|
||||
(step) => step.status === "completed",
|
||||
),
|
||||
"Completed task must not hide failed actions",
|
||||
);
|
||||
const completedMs = performance.now() - start,
|
||||
taskCalls = calls.length - firstCall;
|
||||
const path = join(output, `${temperature}-evidence.json`);
|
||||
await invoke("export_evidence_bundle", { path });
|
||||
const bundle = JSON.parse(await readFile(path, "utf8"));
|
||||
const parent = bundle.records.find(
|
||||
(r) => r.evidence_id === final.normalized_result.parent_evidence_id,
|
||||
);
|
||||
assert.ok(parent);
|
||||
if (!saved) {
|
||||
const legacy = bundle.records.find(
|
||||
(record) => record.evidence_id === firstSummary.parent_evidence_id,
|
||||
);
|
||||
assert.ok(legacy);
|
||||
const login = decodeBodies(legacy.normalized_result).filter(
|
||||
(body) => typeof body?.token === "string",
|
||||
);
|
||||
assert.equal(login.length, 1);
|
||||
const preReadLogin = decodeBodies({
|
||||
events: legacy.normalized_result.events.items.filter(
|
||||
(event) => event.step_index < firstActions.length,
|
||||
),
|
||||
}).filter((body) => typeof body?.token === "string");
|
||||
legacyPreReadObserved = preReadLogin.length === 1;
|
||||
if (legacyPreReadObserved) {
|
||||
assert.ok(
|
||||
!preReadLogin[0].token.startsWith("[REDACTED:"),
|
||||
"An observed pre-read login response must preserve its unmarked token",
|
||||
);
|
||||
const later = legacy.normalized_result.events.items.filter(
|
||||
(event) => event.step_index >= firstActions.length,
|
||||
);
|
||||
assert.ok(
|
||||
!JSON.stringify({
|
||||
events: later,
|
||||
bodies: decodeBodies({ events: later }),
|
||||
}).includes(preReadLogin[0].token),
|
||||
"Post-read facts must use the current page-value policy",
|
||||
);
|
||||
}
|
||||
}
|
||||
const recordBodies = decodeBodies(parent.normalized_result).filter(
|
||||
(b) => b?.collectionName === "rea_items" && typeof b.id === "string",
|
||||
);
|
||||
assert.ok(recordBodies.length >= 2);
|
||||
if (saved) id = recordBodies[0].id;
|
||||
assert.ok(recordBodies.every((b) => b.id === id));
|
||||
assert.ok(
|
||||
recordBodies.some((b) => b.title === `record-${id}` && b.quantity === 11),
|
||||
);
|
||||
const read = await fetch(
|
||||
`${origin}/api/collections/rea_items/records/${id}`,
|
||||
);
|
||||
assert.equal(read.status, 200);
|
||||
const actual = await read.json();
|
||||
assert.equal(actual.title, `record-${id}`);
|
||||
assert.equal(actual.quantity, 11);
|
||||
const requestBodies = decodeBodies(parent.normalized_result).filter(
|
||||
(b) => b?.title === `record-${id}` && !Object.hasOwn(b, "collectionName"),
|
||||
);
|
||||
assert.ok(requestBodies.length > 0);
|
||||
assert.ok(requestBodies.every((b) => typeof b.quantity === "number"));
|
||||
if (saved) {
|
||||
const loginBodies = decodeBodies(parent.normalized_result).filter(
|
||||
(b) => typeof b?.token === "string",
|
||||
);
|
||||
assert.equal(loginBodies.length, 1);
|
||||
assert.ok(loginBodies[0].token.startsWith("[REDACTED:"));
|
||||
}
|
||||
runs.push({
|
||||
temperature,
|
||||
legacy_pre_read_observed: legacyPreReadObserved,
|
||||
elapsed_ms: completedMs,
|
||||
task_calls: taskCalls,
|
||||
evidence_id: parent.evidence_id,
|
||||
record_id: id,
|
||||
quantity_type: typeof actual.quantity,
|
||||
});
|
||||
}
|
||||
if (values["source-only"]) {
|
||||
assert.equal(values.mode, "after");
|
||||
const sourceScenario = scenario(
|
||||
[
|
||||
{
|
||||
step_id: "source",
|
||||
action: "request",
|
||||
method: "GET",
|
||||
path: "/escaped-values.json?credential=source-token",
|
||||
save: [
|
||||
{ name: "token", json_pointer: "/token", secret: true },
|
||||
{ name: "code", json_pointer: "/code", secret: true },
|
||||
],
|
||||
},
|
||||
{
|
||||
step_id: "later",
|
||||
action: "request",
|
||||
method: "GET",
|
||||
path: "/escaped-values.json",
|
||||
},
|
||||
],
|
||||
[],
|
||||
);
|
||||
const captured = await invoke("capture_browser_scenario", sourceScenario);
|
||||
assert.ok(
|
||||
captured.normalized_result.summary.steps.every(
|
||||
(step) => step.status === "completed",
|
||||
),
|
||||
);
|
||||
const path = join(output, "source-evidence.json");
|
||||
await invoke("export_evidence_bundle", { path });
|
||||
const bundle = JSON.parse(await readFile(path, "utf8"));
|
||||
const parent = bundle.records.find(
|
||||
(record) =>
|
||||
record.evidence_id === captured.normalized_result.parent_evidence_id,
|
||||
);
|
||||
assert.ok(parent);
|
||||
const sourceRequests = parent.normalized_result.events.items.filter(
|
||||
(event) =>
|
||||
event.kind === "request" &&
|
||||
event.step_index === 1 &&
|
||||
event.url.url.includes("/escaped-values.json"),
|
||||
);
|
||||
assert.equal(sourceRequests.length, 1);
|
||||
const transaction = sourceRequests[0].transaction_id;
|
||||
const sourceEvents = parent.normalized_result.events.items.filter(
|
||||
(event) => event.transaction_id === transaction,
|
||||
);
|
||||
assert.ok(sourceEvents.some((event) => event.kind === "response"));
|
||||
assert.ok(
|
||||
!JSON.stringify(sourceEvents).includes("source-token"),
|
||||
"Source transaction metadata must not retain the value learned from its response",
|
||||
);
|
||||
const bodies = decodeBodies(parent.normalized_result).filter(
|
||||
(body) =>
|
||||
Object.hasOwn(body ?? {}, "token") && Object.hasOwn(body, "code"),
|
||||
);
|
||||
assert.equal(
|
||||
bodies.length,
|
||||
2,
|
||||
"Both actual source and later response must be observed",
|
||||
);
|
||||
for (const body of bodies) {
|
||||
assert.equal(body.token, "[REDACTED:page:source.token]");
|
||||
assert.equal(body.code, "[REDACTED:page:source.code]");
|
||||
}
|
||||
const expected = Buffer.from(
|
||||
'{"token":"[REDACTED:page:source.token]","code":"[REDACTED:page:source.code]","unknown":1e-400}',
|
||||
).toString("base64");
|
||||
assert.equal(
|
||||
JSON.stringify(parent.normalized_result).split(JSON.stringify(expected))
|
||||
.length - 1,
|
||||
2,
|
||||
"Both source bodies must preserve the unmarked numeric spelling exactly",
|
||||
);
|
||||
const metadata = await invoke("capture_browser_scenario", {
|
||||
...sourceScenario,
|
||||
capture: {
|
||||
...sourceScenario.capture,
|
||||
network: {
|
||||
request_body: false,
|
||||
response_body: false,
|
||||
header_values: false,
|
||||
},
|
||||
},
|
||||
});
|
||||
assert.ok(
|
||||
metadata.normalized_result.summary.steps.every(
|
||||
(step) => step.status === "completed",
|
||||
),
|
||||
);
|
||||
const metadataPath = join(output, "metadata-evidence.json");
|
||||
await invoke("export_evidence_bundle", { path: metadataPath });
|
||||
const metadataBundle = JSON.parse(await readFile(metadataPath, "utf8"));
|
||||
const metadataParent = metadataBundle.records.find(
|
||||
(record) =>
|
||||
record.evidence_id === metadata.normalized_result.parent_evidence_id,
|
||||
);
|
||||
assert.ok(metadataParent);
|
||||
const metadataRequest = metadataParent.normalized_result.events.items.find(
|
||||
(event) =>
|
||||
event.kind === "request" &&
|
||||
event.step_index === 1 &&
|
||||
event.url.url.includes("/escaped-values.json"),
|
||||
);
|
||||
assert.ok(metadataRequest);
|
||||
assert.equal(metadataRequest.url.origin, origin);
|
||||
assert.ok(metadataRequest.url.url.includes("/escaped-values.json"));
|
||||
assert.ok(!metadataRequest.url.url.includes("source-token"));
|
||||
assert.ok(
|
||||
!metadataParent.normalized_result.events.items.some(
|
||||
(event) => event.kind === "network-content",
|
||||
),
|
||||
);
|
||||
const countUrl = `${origin}/api/collections/rea_items/records?perPage=1`;
|
||||
const beforeCount = (await (await fetch(countUrl)).json()).totalItems;
|
||||
assert.ok(Number.isSafeInteger(beforeCount));
|
||||
const whole = await invoke("capture_browser_scenario", {
|
||||
...sourceScenario,
|
||||
start_url: { url: origin + "/whole-json.html" },
|
||||
secrets: [
|
||||
{
|
||||
secret_id: "whole_json",
|
||||
environment_variable: "REA_TEST_WHOLE_JSON",
|
||||
},
|
||||
],
|
||||
storage: {
|
||||
local_storage: [
|
||||
{
|
||||
origin,
|
||||
entries: [
|
||||
{
|
||||
name: "whole_json",
|
||||
value: { source: "secret", secret_id: "whole_json" },
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
actions: [
|
||||
{
|
||||
step_id: "settled",
|
||||
action: "wait_for",
|
||||
locator: { kind: "css", selector: '#result[data-ready="yes"]' },
|
||||
state: "visible",
|
||||
timeout_ms: 10000,
|
||||
},
|
||||
],
|
||||
});
|
||||
assert.ok(
|
||||
whole.normalized_result.summary.steps.every(
|
||||
(step) => step.status === "completed",
|
||||
),
|
||||
);
|
||||
const wholePath = join(output, "whole-json-evidence.json");
|
||||
await invoke("export_evidence_bundle", { path: wholePath });
|
||||
const wholeBundle = JSON.parse(await readFile(wholePath, "utf8"));
|
||||
const wholeParent = wholeBundle.records.find(
|
||||
(record) =>
|
||||
record.evidence_id === whole.normalized_result.parent_evidence_id,
|
||||
);
|
||||
assert.ok(wholeParent);
|
||||
const post = wholeParent.normalized_result.events.items.find(
|
||||
(event) => event.kind === "request" && event.method === "POST",
|
||||
);
|
||||
assert.ok(post);
|
||||
const postEvents = wholeParent.normalized_result.events.items.filter(
|
||||
(event) => event.transaction_id === post.transaction_id,
|
||||
);
|
||||
assert.ok(
|
||||
postEvents.some(
|
||||
(event) =>
|
||||
event.kind === "network-content" &&
|
||||
event.phase === "request" &&
|
||||
event.body.state === "unavailable",
|
||||
),
|
||||
);
|
||||
const postResponse = postEvents.find((event) => event.kind === "response");
|
||||
assert.ok(postResponse && [400, 401, 403].includes(postResponse.status));
|
||||
assert.ok(!JSON.stringify(wholeParent).includes("private-token-729d"));
|
||||
assert.equal(
|
||||
(await (await fetch(countUrl)).json()).totalItems,
|
||||
beforeCount,
|
||||
);
|
||||
const followup = await invoke("capture_browser_scenario", {
|
||||
...sourceScenario,
|
||||
start_url: { url: origin + "/immediate-followup.html" },
|
||||
capture: {
|
||||
...sourceScenario.capture,
|
||||
events: [...sourceScenario.capture.events, "console"],
|
||||
after_each_step: [
|
||||
...sourceScenario.capture.after_each_step,
|
||||
"screenshot",
|
||||
],
|
||||
at_end: [...sourceScenario.capture.at_end, "screenshot"],
|
||||
},
|
||||
actions: [
|
||||
{
|
||||
step_id: "exchange",
|
||||
action: "click",
|
||||
locator: { kind: "css", selector: "#run" },
|
||||
save_response: {
|
||||
method: "GET",
|
||||
path: "/escaped-values.json",
|
||||
save: [{ name: "token", json_pointer: "/token", secret: true }],
|
||||
},
|
||||
timeout_ms: 10000,
|
||||
},
|
||||
{
|
||||
step_id: "settled",
|
||||
action: "wait_for",
|
||||
locator: { kind: "css", selector: '#result[data-ready="yes"]' },
|
||||
state: "visible",
|
||||
timeout_ms: 10000,
|
||||
},
|
||||
],
|
||||
});
|
||||
const followupPath = join(output, "followup-evidence.json");
|
||||
await invoke("export_evidence_bundle", { path: followupPath });
|
||||
const followupBundle = JSON.parse(await readFile(followupPath, "utf8"));
|
||||
const followupParent = followupBundle.records.find(
|
||||
(record) =>
|
||||
record.evidence_id === followup.normalized_result.parent_evidence_id,
|
||||
);
|
||||
assert.ok(followupParent);
|
||||
assert.equal(await page.locator("#sensitive").innerText(), "source-token");
|
||||
assert.equal(followupParent.normalized_result.steps.length, 3);
|
||||
assert.ok(
|
||||
followupParent.normalized_result.steps.every(
|
||||
(step) =>
|
||||
step.artifacts.screenshot.state === "missing" &&
|
||||
step.completeness.missing_sections.includes("screenshot"),
|
||||
),
|
||||
);
|
||||
assert.ok(
|
||||
followupParent.normalized_result.completeness.missing_sections.includes(
|
||||
"screenshot",
|
||||
),
|
||||
);
|
||||
const consoleEvents = followupParent.normalized_result.events.items.filter(
|
||||
(event) => event.kind === "console" && event.text.includes("REDACTED"),
|
||||
);
|
||||
assert.equal(consoleEvents.length, 1);
|
||||
assert.ok(
|
||||
!JSON.stringify(followupParent.normalized_result.events.items).includes(
|
||||
"source-token",
|
||||
),
|
||||
);
|
||||
assert.ok(
|
||||
followup.normalized_result.summary.steps.every(
|
||||
(step) => step.status === "completed",
|
||||
),
|
||||
);
|
||||
|
||||
const followupPost = followupParent.normalized_result.events.items.find(
|
||||
(event) => event.kind === "request" && event.method === "POST",
|
||||
);
|
||||
assert.ok(followupPost);
|
||||
const followupEvents = followupParent.normalized_result.events.items.filter(
|
||||
(event) => event.transaction_id === followupPost.transaction_id,
|
||||
);
|
||||
assert.ok(!JSON.stringify(followupEvents).includes("source-token"));
|
||||
const followupBodies = decodeBodies({ events: followupEvents });
|
||||
assert.ok(
|
||||
followupBodies.some(
|
||||
(body) => body?.token === "[REDACTED:page:exchange.token]",
|
||||
),
|
||||
);
|
||||
assert.ok(
|
||||
followupEvents.some(
|
||||
(event) => event.kind === "response" && event.status === 403,
|
||||
),
|
||||
);
|
||||
assert.equal(
|
||||
(await (await fetch(countUrl)).json()).totalItems,
|
||||
beforeCount,
|
||||
);
|
||||
const publicScreenshot = await invoke("capture_browser_scenario", {
|
||||
...sourceScenario,
|
||||
start_url: { url: origin + "/immediate-followup.html" },
|
||||
actions: [
|
||||
{ step_id: "settle", action: "wait_for_timeout", duration_ms: 1 },
|
||||
],
|
||||
capture: {
|
||||
...sourceScenario.capture,
|
||||
events: [],
|
||||
network: {
|
||||
request_body: false,
|
||||
response_body: false,
|
||||
header_values: false,
|
||||
},
|
||||
after_each_step: [],
|
||||
at_end: ["screenshot"],
|
||||
},
|
||||
});
|
||||
const publicPath = join(output, "public-screenshot-evidence.json");
|
||||
await invoke("export_evidence_bundle", { path: publicPath });
|
||||
const publicBundle = JSON.parse(await readFile(publicPath, "utf8"));
|
||||
const publicParent = publicBundle.records.find(
|
||||
(record) =>
|
||||
record.evidence_id ===
|
||||
publicScreenshot.normalized_result.parent_evidence_id,
|
||||
);
|
||||
assert.ok(publicParent);
|
||||
assert.equal(await page.locator("#sensitive").innerText(), "");
|
||||
const png =
|
||||
publicParent.normalized_result.steps.at(-1).artifacts.screenshot;
|
||||
assert.equal(png.state, "captured");
|
||||
assert.equal(
|
||||
Buffer.from(png.value.data_base64, "base64")
|
||||
.subarray(0, 8)
|
||||
.toString("hex"),
|
||||
"89504e470d0a1a0a",
|
||||
);
|
||||
runs.push({
|
||||
verification: "source-and-later-JSON-spellings",
|
||||
verified_bodies: bodies.length,
|
||||
metadata_only_source_url: true,
|
||||
whole_json_secret_withheld: true,
|
||||
immediate_followup_protected: true,
|
||||
immediate_console_protected: true,
|
||||
rendered_secret_screenshots_withheld: true,
|
||||
nonsecret_screenshot_captured: true,
|
||||
whole_json_status: postResponse.status,
|
||||
});
|
||||
}
|
||||
await writeFile(join(output, "calls.json"), JSON.stringify(calls), {
|
||||
mode: 0o600,
|
||||
});
|
||||
await writeFile(
|
||||
join(output, "metrics.json"),
|
||||
JSON.stringify({ mode: values.mode, runs }, null, 2),
|
||||
);
|
||||
console.log(JSON.stringify({ mode: values.mode, runs }));
|
||||
} finally {
|
||||
await withVerifierCleanup(
|
||||
() => commands.close(),
|
||||
[
|
||||
async () => {
|
||||
if (commands.cleanupResources.length === 0)
|
||||
await rm(profile, { recursive: true, force: true });
|
||||
},
|
||||
],
|
||||
);
|
||||
}
|
||||
@@ -1,63 +0,0 @@
|
||||
# Browser scenario summary verification
|
||||
|
||||
The verifier runs the actual compiled CLI/MCP and caller-selected Chromium against
|
||||
TodoMVC's JavaScript ES5 distribution served on loopback. Ten actions add five
|
||||
todos. Every step captures URL, DOM, accessibility and a screenshot; the oracle
|
||||
checks all completed steps, the five authored labels, retained events and
|
||||
transaction identities. Summary runs also read step, text and transaction pages
|
||||
and authenticate an exported complete parent.
|
||||
|
||||
Download the public distribution at the pinned revision into a disposable folder:
|
||||
|
||||
```bash
|
||||
export REA_TODOMVC_APP="$(mktemp -d)"
|
||||
python3 - "$REA_TODOMVC_APP" <<'PY'
|
||||
import hashlib, json, sys, urllib.request
|
||||
from pathlib import Path
|
||||
root = Path(sys.argv[1])
|
||||
revision = "ff43b02e59dfa604386bb382034b2cd07c2bcd8a"
|
||||
files = ["index.html", "app.js", "base.css", "base.js", "controller.js",
|
||||
"helpers.js", "index.css", "model.js", "store.js", "template.js", "view.js"]
|
||||
manifest = []
|
||||
for name in files:
|
||||
url = f"https://raw.githubusercontent.com/tastejs/todomvc/{revision}/examples/javascript-es5/dist/{name}"
|
||||
with urllib.request.urlopen(url, timeout=30) as reply:
|
||||
data = reply.read()
|
||||
(root / name).write_bytes(data)
|
||||
manifest.append({"path": name, "url": url, "bytes": len(data),
|
||||
"sha256": hashlib.sha256(data).hexdigest()})
|
||||
(root / "provenance.json").write_text(json.dumps({"revision": revision, "files": manifest}, indent=2))
|
||||
PY
|
||||
```
|
||||
|
||||
Build each selected checkout with the repository's pinned Node/npm toolchain.
|
||||
Use new output directories; exports never overwrite an earlier run:
|
||||
|
||||
```bash
|
||||
node experiments/analysis-efficiency/browser-scenario-summary.mjs \
|
||||
--repository /absolute/path/to/baseline --mode complete \
|
||||
--application "$REA_TODOMVC_APP" --browser /absolute/path/to/chromium \
|
||||
--output /absolute/path/to/new-baseline-report
|
||||
node experiments/analysis-efficiency/browser-scenario-summary.mjs \
|
||||
--mode summary --application "$REA_TODOMVC_APP" \
|
||||
--browser /absolute/path/to/chromium \
|
||||
--output /absolute/path/to/new-summary-report
|
||||
```
|
||||
|
||||
`cold` is the first capture in a fresh MCP process; `warm` repeats the capture in
|
||||
that process with a new owned browser/profile. Neither label establishes cold OS
|
||||
file caches. Summary mode intentionally performs additional retained queries.
|
||||
Compare equivalent answers and include those calls when reporting task cost;
|
||||
smaller replies alone do not establish a faster task.
|
||||
|
||||
Add `--verify-boundaries` to a separate summary run for a real failed-step focus,
|
||||
CLI summary publication, and MCP text readback from the CLI's complete bundle.
|
||||
Those extra checks make its whole-run resource cost unsuitable for comparison
|
||||
against a complete-only run.
|
||||
|
||||
Reports retain the actual catalog, arguments, MCP text and complete Evidence.
|
||||
When estimating tokens, name the tokenizer and count discovery, arguments and
|
||||
all replies needed for the chosen task. Raw serialized catalog estimates are
|
||||
not actual model usage or a measurement of a client's rendered tool definitions.
|
||||
Measure CPU and peak process-family memory externally, state the accounting
|
||||
method, and include cleanup; the verifier's elapsed phase excludes final teardown.
|
||||
@@ -1,412 +0,0 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { createServer } from "node:http";
|
||||
import { readFile, writeFile, mkdir, readdir } from "node:fs/promises";
|
||||
import { resolve, join, extname } from "node:path";
|
||||
import { pathToFileURL, fileURLToPath } from "node:url";
|
||||
import { createRequire } from "node:module";
|
||||
import { performance } from "node:perf_hooks";
|
||||
import { execFile } from "node:child_process";
|
||||
import { promisify, parseArgs } from "node:util";
|
||||
// A developer verifier: the caller selects the pinned TodoMVC distribution,
|
||||
// browser executable, output directory and baseline/final repository checkout.
|
||||
const { values } = parseArgs({
|
||||
options: {
|
||||
repository: {
|
||||
type: "string",
|
||||
default: fileURLToPath(new URL("../../", import.meta.url)),
|
||||
},
|
||||
mode: { type: "string", default: "summary" },
|
||||
application: { type: "string" },
|
||||
browser: { type: "string" },
|
||||
output: { type: "string" },
|
||||
port: { type: "string", default: "0" },
|
||||
"verify-boundaries": { type: "boolean", default: false },
|
||||
},
|
||||
});
|
||||
assert.ok(
|
||||
values.application && values.browser && values.output,
|
||||
"Pass --application <TodoMVC javascript-es5 dist> --browser <executable> --output <new directory>",
|
||||
);
|
||||
const repo = resolve(values.repository);
|
||||
const mode = values.mode;
|
||||
const output = resolve(values.output);
|
||||
const appRoot = resolve(values.application);
|
||||
const executablePath = resolve(values.browser);
|
||||
assert.ok(["complete", "summary"].includes(mode));
|
||||
assert.ok(
|
||||
!values["verify-boundaries"] || mode === "summary",
|
||||
"--verify-boundaries requires summary mode",
|
||||
);
|
||||
process.chdir(repo);
|
||||
const require = createRequire(join(repo, "package.json"));
|
||||
const { Client } = await import(
|
||||
pathToFileURL(require.resolve("@modelcontextprotocol/client")).href
|
||||
);
|
||||
const { StdioClientTransport } = await import(
|
||||
pathToFileURL(require.resolve("@modelcontextprotocol/client/stdio")).href
|
||||
);
|
||||
await mkdir(output);
|
||||
const files = await readdir(appRoot);
|
||||
const server = createServer(async (request, response) => {
|
||||
try {
|
||||
const name =
|
||||
decodeURIComponent(
|
||||
new URL(request.url, "http://127.0.0.1").pathname,
|
||||
).slice(1) || "index.html";
|
||||
assert.ok(files.includes(name));
|
||||
const data = await readFile(join(appRoot, name));
|
||||
response.setHeader(
|
||||
"Content-Type",
|
||||
{ ".html": "text/html", ".js": "text/javascript", ".css": "text/css" }[
|
||||
extname(name)
|
||||
] ?? "application/octet-stream",
|
||||
);
|
||||
response.end(data);
|
||||
} catch {
|
||||
response.writeHead(404);
|
||||
response.end();
|
||||
}
|
||||
});
|
||||
await new Promise((done, reject) => {
|
||||
server.once("error", reject);
|
||||
server.listen(Number(values.port), "127.0.0.1", done);
|
||||
});
|
||||
const address = server.address();
|
||||
assert.ok(address && typeof address === "object");
|
||||
const origin = `http://127.0.0.1:${address.port}`;
|
||||
const environment = {
|
||||
PATH: process.env.PATH,
|
||||
NODE_OPTIONS: "--max-old-space-size=3072",
|
||||
UV_THREADPOOL_SIZE: "2",
|
||||
REA_LOG_LEVEL: "silent",
|
||||
REA_MCP_INPUT_SCHEMA_PROFILE: "compact",
|
||||
};
|
||||
const transport = new StdioClientTransport({
|
||||
command: process.execPath,
|
||||
args: [join(repo, "scripts/rea.mjs"), "mcp"],
|
||||
cwd: repo,
|
||||
env: environment,
|
||||
stderr: "pipe",
|
||||
});
|
||||
let stderr = "";
|
||||
transport.stderr?.on("data", (chunk) => {
|
||||
if (stderr.length < 65536) stderr += chunk;
|
||||
});
|
||||
const client = new Client({
|
||||
name: "rea-browser-scenario-summary-verifier",
|
||||
version: "1",
|
||||
});
|
||||
const calls = [];
|
||||
const invoke = async (name, args, tools) => {
|
||||
const toolDefinition = tools.find((tool) => tool.name === name);
|
||||
assert.ok(
|
||||
toolDefinition?.outputSchema,
|
||||
`Missing named output schema: ${name}`,
|
||||
);
|
||||
const started = performance.now();
|
||||
const result = await client.callTool(
|
||||
{ name, arguments: args },
|
||||
{ toolDefinition, timeout: 90000 },
|
||||
);
|
||||
calls.push({
|
||||
name,
|
||||
arguments: args,
|
||||
content: result.content,
|
||||
elapsed_ms: performance.now() - started,
|
||||
});
|
||||
assert.notEqual(result.isError, true, JSON.stringify(result));
|
||||
return result.structuredContent;
|
||||
};
|
||||
const allStarted = performance.now();
|
||||
try {
|
||||
await client.connect(transport);
|
||||
const discoveryStarted = performance.now();
|
||||
const catalog = await client.listTools();
|
||||
const discoveryMs = performance.now() - discoveryStarted;
|
||||
await writeFile(join(output, "catalog.json"), JSON.stringify(catalog));
|
||||
const actions = Array.from({ length: 5 }, (_, index) => [
|
||||
{
|
||||
step_id: `fill_${index}`,
|
||||
action: "fill",
|
||||
locator: { kind: "css", selector: ".new-todo" },
|
||||
value: { source: "literal", value: `Review evidence ${index + 1}` },
|
||||
timeout_ms: 10000,
|
||||
},
|
||||
{
|
||||
step_id: `add_${index}`,
|
||||
action: "press",
|
||||
locator: { kind: "css", selector: ".new-todo" },
|
||||
key: "Enter",
|
||||
timeout_ms: 10000,
|
||||
},
|
||||
]).flat();
|
||||
const scenario = {
|
||||
browser: {
|
||||
mode: "launch",
|
||||
executable_path: executablePath,
|
||||
headless: true,
|
||||
},
|
||||
start_url: { url: origin, query: [] },
|
||||
actions,
|
||||
capture: {
|
||||
after_each_step: ["url", "dom", "accessibility", "screenshot"],
|
||||
at_end: ["url", "dom", "accessibility"],
|
||||
events: ["network"],
|
||||
},
|
||||
};
|
||||
const runs = [];
|
||||
for (const temperature of ["cold", "warm"]) {
|
||||
const callStart = calls.length;
|
||||
const result = await invoke(
|
||||
"capture_browser_scenario",
|
||||
{ ...scenario, ...(mode === "summary" ? { detail: "summary" } : {}) },
|
||||
catalog.tools,
|
||||
);
|
||||
const parentId =
|
||||
mode === "summary"
|
||||
? result.normalized_result.parent_evidence_id
|
||||
: result.evidence_id;
|
||||
const bundlePath = join(output, `${temperature}-evidence.json`);
|
||||
await invoke("export_evidence_bundle", { path: bundlePath }, catalog.tools);
|
||||
const bundle = JSON.parse(await readFile(bundlePath, "utf8"));
|
||||
const parent = bundle.records.find((row) => row.evidence_id === parentId);
|
||||
assert.ok(parent);
|
||||
const capture = parent.normalized_result;
|
||||
assert.equal(capture.steps.length, 11);
|
||||
assert.ok(capture.steps.every((step) => step.status === "completed"));
|
||||
assert.equal(capture.browser.cleanup, "terminated-owned-process");
|
||||
const final = capture.steps.at(-1).artifacts.accessibility;
|
||||
assert.equal(final.state, "captured");
|
||||
for (let index = 1; index <= 5; index++)
|
||||
assert.ok(final.value.text.includes(`Review evidence ${index}`));
|
||||
assert.ok(final.value.text.includes('strong: "5"'));
|
||||
assert.ok(final.value.text.includes("text: items left"));
|
||||
const transactionIds = new Set(
|
||||
capture.events.items
|
||||
.filter((event) => "transaction_id" in event)
|
||||
.map((event) => event.transaction_id),
|
||||
);
|
||||
const details = {
|
||||
temperature,
|
||||
parentId,
|
||||
steps: capture.steps.length,
|
||||
retained_events: capture.events.retained,
|
||||
transactions: transactionIds.size,
|
||||
};
|
||||
if (mode === "summary") {
|
||||
const summary = result.normalized_result.summary;
|
||||
assert.equal(summary.steps.length, 11);
|
||||
assert.equal(
|
||||
summary.final_accessibility.accessibility.value.text,
|
||||
final.value.text,
|
||||
);
|
||||
assert.equal(summary.events.retained, capture.events.retained);
|
||||
assert.equal(summary.limitation_count, parent.limitations.length);
|
||||
assert.equal(summary.network.transactions, transactionIds.size);
|
||||
const source = { kind: "retained-evidence", evidence_id: parentId };
|
||||
const pagingStarted = performance.now();
|
||||
for (const stepIndex of [1, 5, 10]) {
|
||||
const page = await invoke(
|
||||
"inspect_analysis_view",
|
||||
{
|
||||
source,
|
||||
view: {
|
||||
kind: "browser-scenario",
|
||||
collection: "steps",
|
||||
step_indices: [stepIndex],
|
||||
artifacts: [],
|
||||
limit: 1,
|
||||
},
|
||||
},
|
||||
catalog.tools,
|
||||
);
|
||||
assert.equal(
|
||||
page.normalized_result.items.rows[0].step_id,
|
||||
capture.steps[stepIndex].step_id,
|
||||
);
|
||||
}
|
||||
details.warm_page_total_ms = performance.now() - pagingStarted;
|
||||
const text = await invoke(
|
||||
"inspect_analysis_view",
|
||||
{
|
||||
source,
|
||||
view: {
|
||||
kind: "browser-scenario",
|
||||
collection: "text",
|
||||
step_index: 10,
|
||||
artifact: "accessibility",
|
||||
offset: 0,
|
||||
limit: 32,
|
||||
},
|
||||
},
|
||||
catalog.tools,
|
||||
);
|
||||
assert.equal(
|
||||
text.normalized_result.items.text,
|
||||
final.value.text.slice(0, 32),
|
||||
);
|
||||
const tx = await invoke(
|
||||
"inspect_analysis_view",
|
||||
{
|
||||
source,
|
||||
view: {
|
||||
kind: "browser-scenario",
|
||||
collection: "transactions",
|
||||
limit: 2,
|
||||
},
|
||||
},
|
||||
catalog.tools,
|
||||
);
|
||||
assert.deepEqual(
|
||||
tx.normalized_result.items.rows.map((row) => row.transaction_id),
|
||||
[...transactionIds].slice(0, 2),
|
||||
);
|
||||
details.returned_summary = summary;
|
||||
}
|
||||
details.calls = calls.slice(callStart);
|
||||
runs.push(details);
|
||||
}
|
||||
const boundaryChecks = [];
|
||||
if (values["verify-boundaries"]) {
|
||||
const failed = await invoke(
|
||||
"capture_browser_scenario",
|
||||
{
|
||||
...scenario,
|
||||
actions: [
|
||||
{
|
||||
step_id: "missing",
|
||||
action: "click",
|
||||
locator: {
|
||||
kind: "css",
|
||||
selector: "#rea-control-that-does-not-exist",
|
||||
},
|
||||
timeout_ms: 100,
|
||||
},
|
||||
],
|
||||
capture: {
|
||||
after_each_step: ["url", "accessibility"],
|
||||
at_end: ["url", "accessibility"],
|
||||
events: ["network"],
|
||||
},
|
||||
detail: "summary",
|
||||
},
|
||||
catalog.tools,
|
||||
);
|
||||
assert.equal(failed.normalized_result.summary.steps[1].status, "failed");
|
||||
assert.equal(
|
||||
failed.normalized_result.summary.final_accessibility.focus_step_index,
|
||||
1,
|
||||
);
|
||||
assert.equal(
|
||||
failed.normalized_result.summary.completeness.equality_eligible,
|
||||
false,
|
||||
);
|
||||
boundaryChecks.push("real failed-step focus and incomplete coverage");
|
||||
const execute = promisify(execFile);
|
||||
const bundlePath = join(output, "cli-evidence.json");
|
||||
const { stdout } = await execute(
|
||||
process.execPath,
|
||||
[
|
||||
join(repo, "scripts/rea.mjs"),
|
||||
"capture-browser-scenario",
|
||||
JSON.stringify(scenario),
|
||||
"--detail",
|
||||
"summary",
|
||||
"--evidence_output",
|
||||
bundlePath,
|
||||
"--json",
|
||||
],
|
||||
{
|
||||
cwd: repo,
|
||||
env: environment,
|
||||
maxBuffer: 64 * 1024 * 1024,
|
||||
timeout: 90000,
|
||||
},
|
||||
);
|
||||
const cli = JSON.parse(stdout);
|
||||
const parentId = cli.summary.normalized_result.parent_evidence_id;
|
||||
const bundle = JSON.parse(await readFile(bundlePath, "utf8"));
|
||||
const parent = bundle.records.find((row) => row.evidence_id === parentId);
|
||||
assert.ok(parent);
|
||||
assert.equal(parent.normalized_result.steps.length, 11);
|
||||
assert.ok(
|
||||
parent.normalized_result.steps.every(
|
||||
(step) => step.status === "completed",
|
||||
),
|
||||
);
|
||||
assert.ok(!("detail" in parent.parameters));
|
||||
assert.equal(
|
||||
cli.summary.normalized_result.summary.final_accessibility.accessibility
|
||||
.value.text,
|
||||
parent.normalized_result.steps.at(-1).artifacts.accessibility.value.text,
|
||||
);
|
||||
const text = await invoke(
|
||||
"inspect_analysis_view",
|
||||
{
|
||||
source: {
|
||||
kind: "evidence-file",
|
||||
path: bundlePath,
|
||||
evidence_id: parentId,
|
||||
},
|
||||
view: {
|
||||
kind: "browser-scenario",
|
||||
collection: "text",
|
||||
step_index: 10,
|
||||
artifact: "accessibility",
|
||||
offset: 0,
|
||||
limit: 32,
|
||||
},
|
||||
},
|
||||
catalog.tools,
|
||||
);
|
||||
assert.equal(
|
||||
text.normalized_result.items.text,
|
||||
parent.normalized_result.steps
|
||||
.at(-1)
|
||||
.artifacts.accessibility.value.text.slice(0, 32),
|
||||
);
|
||||
boundaryChecks.push(
|
||||
"real CLI summary, complete bundle and MCP evidence-file readback",
|
||||
);
|
||||
}
|
||||
await writeFile(
|
||||
join(output, "result.json"),
|
||||
JSON.stringify(
|
||||
{
|
||||
repo,
|
||||
mode,
|
||||
application: appRoot,
|
||||
browser: executablePath,
|
||||
input_paths: {
|
||||
repository: values.repository,
|
||||
application: values.application,
|
||||
browser: values.browser,
|
||||
output: values.output,
|
||||
},
|
||||
origin,
|
||||
discovery_ms: discoveryMs,
|
||||
elapsed_ms: performance.now() - allStarted,
|
||||
runs,
|
||||
boundaryChecks,
|
||||
stderr,
|
||||
},
|
||||
null,
|
||||
2,
|
||||
),
|
||||
);
|
||||
console.log(
|
||||
JSON.stringify({
|
||||
mode,
|
||||
discovery_ms: discoveryMs,
|
||||
runs: runs.map(
|
||||
({ calls: _calls, returned_summary: _summary, ...rest }) => rest,
|
||||
),
|
||||
boundaryChecks,
|
||||
elapsed_ms: performance.now() - allStarted,
|
||||
}),
|
||||
);
|
||||
} finally {
|
||||
await client.close();
|
||||
server.closeAllConnections();
|
||||
await new Promise((done) => server.close(done));
|
||||
}
|
||||
@@ -1,94 +0,0 @@
|
||||
"""Verify 18 completed entropy tasks and report whole-task resource/token costs."""
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
import hashlib, json, statistics
|
||||
import tiktoken
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--receipts', type=Path, required=True)
|
||||
parser.add_argument('--output', type=Path, required=True)
|
||||
options = parser.parse_args()
|
||||
b = options.receipts
|
||||
cohort = b / 'cohort'
|
||||
index = json.loads((cohort / 'index.json').read_text())
|
||||
assert len(index) == 18 and all((r['exit'] == 0 for r in index))
|
||||
assert len({item['name'] for item in index}) == 18
|
||||
enc = tiktoken.get_encoding('cl100k_base')
|
||||
compact = lambda d: json.dumps(d, ensure_ascii=False, separators=(',', ':'), allow_nan=False)
|
||||
count = lambda s: len(enc.encode(s, disallowed_special=()))
|
||||
digest = lambda p: hashlib.sha256(p.read_bytes()).hexdigest()
|
||||
rows = []
|
||||
for item in index:
|
||||
p = cohort / item['name']
|
||||
r = json.loads((p / 'report.json').read_text())
|
||||
catalog = json.loads((p / 'catalog.json').read_text())
|
||||
tx = json.loads((p / 'transcript.json').read_text())
|
||||
scope = next((json.loads(l) for l in reversed((cohort / (item['name'] + '.log')).read_text().splitlines()) if l.startswith('{"resource_scope":')))
|
||||
obs = scope['payload_resource_observation']
|
||||
events = obs['resource_events']
|
||||
assert r['oracle_sha256'] == digest(b / f"{item['case']}-oracle.json")
|
||||
assert item['exit'] == 0 and r['status'] == 'passed' and r['complete_source_export_verified'] and (not r['cleanup_failures']) and (not r['receipt_failures'])
|
||||
assert scope['abort'] is None and scope['payload_resource_observation_verified'] and (obs['payload_exit_code'] == 0) and obs['payload_cleanup_verified']
|
||||
assert all((events['memory.events'][k] == 0 for k in ['high', 'max', 'oom', 'oom_kill'])) and events['pids.events']['max'] == 0
|
||||
assert [e['name'] for e in tx] == ['inspect_byte_entropy', 'export_evidence_bundle']
|
||||
assert all((not e['reply'].get('isError', False) for e in tx))
|
||||
inp = count(compact([{k: t[k] for k in ['name', 'description', 'inputSchema']} for t in catalog['tools']]))
|
||||
args = sum((count(compact({'name': e['name'], 'arguments': e['input']})) for e in tx))
|
||||
replies = sum((count(c['text']) for e in tx for c in e['reply']['content'] if c['type'] == 'text'))
|
||||
prompt = count(r['question'])
|
||||
instructions = count(r['server_instructions'])
|
||||
rows.append({
|
||||
**item,
|
||||
'catalog_tools': len(catalog['tools']),
|
||||
'catalog_sha256': digest(p / 'catalog.json'),
|
||||
'parent_evidence_id': r['parent_evidence_id'],
|
||||
'source_report_sha256': r['source_report_sha256'],
|
||||
'runtime_identity_sha256': r['runtime_identity_sha256'],
|
||||
'wall_seconds': scope['elapsed_seconds'],
|
||||
'cpu_seconds': obs['cpu_stat']['usage_usec'] / 1000000.0,
|
||||
'kernel_family_peak_bytes': obs['memory_peak_bytes'],
|
||||
'inspection_ms': r['calls'][0]['elapsed_ms'],
|
||||
'discovery_ms': r['phases']['discovery_ms'],
|
||||
'token_estimate': {
|
||||
'question': prompt,
|
||||
'instructions': instructions,
|
||||
'directory_once': inp,
|
||||
'arguments': args,
|
||||
'text_replies': replies,
|
||||
'whole_task': prompt + instructions + inp + args + replies,
|
||||
'raw_catalog': count(compact(catalog)),
|
||||
},
|
||||
'receipt_sha256': {n: digest(p / n) for n in ['report.json', 'transcript.json', 'catalog.json', 'source.evidence.json']},
|
||||
'payload_exit_events': events,
|
||||
'payload_cleanup_verified': True,
|
||||
'whole_scope_events_final': scope['resource_events_final'],
|
||||
})
|
||||
summary = {}
|
||||
for case in ['llvm-small-block', 'llvm-large-block', 'ls']:
|
||||
rs = [r for r in rows if r['case'] == case]
|
||||
for k in ['catalog_sha256', 'source_report_sha256', 'parent_evidence_id']:
|
||||
assert len({r[k] for r in rs}) == 1, (case, k)
|
||||
summary[case] = {}
|
||||
for lane in ['before', 'after']:
|
||||
ls = [r for r in rs if r['lane'] == lane]
|
||||
assert len(ls) == 3
|
||||
assert {r['repeat'] for r in ls} == {1, 2, 3}
|
||||
metrics = {k: [r[k] for r in ls] for k in ['wall_seconds', 'cpu_seconds', 'kernel_family_peak_bytes', 'inspection_ms', 'discovery_ms']}
|
||||
metrics['whole_task_tokens'] = [r['token_estimate']['whole_task'] for r in ls]
|
||||
summary[case][lane] = {k: {'median': statistics.median(v), 'min': min(v), 'max': max(v)} for k, v in metrics.items()}
|
||||
identities = {lane: json.loads((b / f'{lane}-runtime-identity.json').read_text()) for lane in ['before', 'after']}
|
||||
assert all(identity['source_clean'] for identity in identities.values())
|
||||
for row in rows:
|
||||
assert row['runtime_identity_sha256'] == digest(b / f"{row['lane']}-runtime-identity.json")
|
||||
result = {
|
||||
'tiktoken': tiktoken.__version__,
|
||||
'encoding': enc.name,
|
||||
'actual_model_usage': None,
|
||||
'revisions': {l: i['source_revision'] for l, i in identities.items()},
|
||||
'identity_receipts': {l: digest(b / f'{l}-runtime-identity.json') for l in identities},
|
||||
'oracles': {c: {'sha256': digest(b / f'{c}-oracle.json'), 'artifact': json.loads((b / f'{c}-oracle.json').read_text())['expected']['artifact']} for c in summary},
|
||||
'summary': summary,
|
||||
'recordings': rows,
|
||||
'measurement': 'Three alternating fresh production stdio SDK sessions per lane/case; exact discovered toolDefinition/default validator, fresh HOME, Node compile cache disabled, OS caches unflushed. Each task discovers complete catalog, inspects bytes and exports/compares full source Evidence; all oracle facts checked. Whole process-family CPU and kernel peak include client/server, runtime fingerprint verification, serialization, export and cleanup. Oracle generation and tokenization outside measured tasks. One-CPU quota, 3GiB early stop/4GiB hard/no swap; payload exit/cleanup/events verified, final whole-scope counters unavailable. Token serialization estimate includes question/instructions/all input name+description+schema once/all arguments/all text replies, raw catalog separately. Host framing, replay, model inference quality and billing unmeasured.',
|
||||
}
|
||||
options.output.write_text(json.dumps(result, indent=2) + '\n')
|
||||
print(json.dumps(summary, indent=2))
|
||||
@@ -1,56 +0,0 @@
|
||||
"""Independent Python histogram oracle; reads artifact bytes without execution."""
|
||||
import argparse
|
||||
from collections import Counter
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
from pathlib import Path
|
||||
|
||||
def entropy(counts, size):
|
||||
if not size:
|
||||
return 0
|
||||
value = -sum((counts[k] / size) * math.log2(counts[k] / size) for k in sorted(counts) if counts[k])
|
||||
return math.floor(value * 10000 + 0.5) / 10000
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('--path', required=True)
|
||||
parser.add_argument('--output', required=True)
|
||||
parser.add_argument('--block-size', type=int, required=True)
|
||||
parser.add_argument('--offset', type=int, default=0)
|
||||
parser.add_argument('--length', type=int)
|
||||
args = parser.parse_args()
|
||||
path = Path(args.path)
|
||||
before = path.stat()
|
||||
counts = Counter()
|
||||
digest = hashlib.sha256()
|
||||
with path.open('rb') as source:
|
||||
while chunk := source.read(1024 * 1024):
|
||||
digest.update(chunk)
|
||||
counts.update(chunk)
|
||||
length = args.length if args.length is not None else before.st_size - args.offset
|
||||
assert 0 <= args.offset <= before.st_size and 0 <= length <= before.st_size - args.offset
|
||||
blocks, runs = [], []
|
||||
source.seek(args.offset)
|
||||
for offset in range(args.offset, args.offset + length, args.block_size):
|
||||
chunk = source.read(min(args.block_size, args.offset + length - offset))
|
||||
value = entropy(Counter(chunk), len(chunk))
|
||||
block = {'offset': offset, 'length': len(chunk), 'entropy': value}
|
||||
blocks.append(block)
|
||||
classification = 'high' if value >= 7.2 else 'low' if value <= 1 else None
|
||||
if classification is not None:
|
||||
if runs and runs[-1]['classification'] == classification and runs[-1]['offset'] + runs[-1]['length'] == offset:
|
||||
runs[-1]['length'] += len(chunk)
|
||||
else:
|
||||
runs.append({'offset': offset, 'length': len(chunk), 'classification': classification})
|
||||
after = path.stat()
|
||||
assert (before.st_dev, before.st_ino, before.st_size, before.st_mtime_ns, before.st_ctime_ns) == (after.st_dev, after.st_ino, after.st_size, after.st_mtime_ns, after.st_ctime_ns)
|
||||
expected = {'artifact': {'path': args.path, 'bytes': before.st_size, 'sha256': digest.hexdigest()}, 'block_size':args.block_size,'window':{'offset':args.offset,'length':length},'file_entropy':entropy(counts,before.st_size),'thresholds':{'high':7.2,'low':1},'blocks':blocks,'runs':runs}
|
||||
request = {'path':args.path,'block_size':args.block_size,'offset':args.offset}
|
||||
if args.length is not None:
|
||||
request['length'] = args.length
|
||||
Path(args.output).write_text(json.dumps({'input':request,'expected':expected},indent=2)+'\n')
|
||||
print(json.dumps({'path':args.path,'sha256':digest.hexdigest(),'blocks':len(blocks),'runs':len(runs)}))
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,158 +0,0 @@
|
||||
# Byte entropy: separate physical reads from report resolution
|
||||
|
||||
Issue: [#6947](https://github.com/morluto/rea/issues/6947).
|
||||
|
||||
The question is to return the full-file SHA-256 and entropy, every block in a
|
||||
selected byte window, and adjacent high/low runs, preserving complete source
|
||||
Evidence. `ByteEntropyService` previously used `block_size` for both reporting
|
||||
and physical reads, including outside the selected window. Small report blocks
|
||||
therefore caused hundreds of thousands of asynchronous reads of a large file.
|
||||
|
||||
The shared service now fills a bounded 64 KiB buffer before calculating the
|
||||
caller-selected blocks. Supported power-of-two block sizes divide that buffer;
|
||||
the window starts its own block sequence, and only its final block can be short.
|
||||
Short physical reads are joined before interpretation. The same stable-file,
|
||||
whole-file hash, cancellation, capacity and descriptor-cleanup owners remain.
|
||||
No new cache, parser, contract or output selection is introduced.
|
||||
|
||||
## Fixed comparison
|
||||
|
||||
Baseline: `1d896e03b776453282beca7fd89338be778bfca2`.
|
||||
Measured implementation: `3ed3276d0ee79c291e1006901ace5b06626c1d4e`.
|
||||
Both were clean and built with Node 24.22.1, npm 11.16.0 and SDK 2.3.1.
|
||||
The runtime receipt fingerprints 2,582 compiled/catalog/entrypoint/dependency
|
||||
and driver files in each lane, checked before and after every task.
|
||||
|
||||
Real artifacts are installed Linux x64 binaries, inspected without execution:
|
||||
|
||||
| Artifact | Bytes | SHA-256 |
|
||||
| ----------------------------------------------------- | ----------: | ------------------------------------------------------------------ |
|
||||
| LLVM 16 (`/usr/lib/x86_64-linux-gnu/libLLVM-16.so.1`) | 123,379,936 | `f62d254b7f2bf42df8c8b07d46ee3bb4c2cafeca436b2e6bc6ccbe4581f58f40` |
|
||||
| GNU ls (`/usr/bin/ls`) | 151,344 | `cb30d69b24245bf2ecdc9e7f53bbad19159999970b6d82c0c00c7d32d9e37aa4` |
|
||||
|
||||
`byte-entropy-oracle.py` independently computes Python byte histograms,
|
||||
Shannon entropy, selected blocks, rounded-value run classifications and digests.
|
||||
It checks file state around its reads. Every returned fact agrees with this
|
||||
oracle, and both lanes have identical complete report hashes and Evidence IDs.
|
||||
|
||||
There are three alternating fresh production stdio MCP sessions per lane/case,
|
||||
18 tasks total. Each discovers all 222 tools, uses exact discovered
|
||||
`toolDefinition` validation, inspects the bytes, exports the complete Evidence
|
||||
bundle and verifies its parent record. Fresh HOME/client/server, Node compile
|
||||
cache disabled, OS caches unflushed. These are fresh sessions; no warm-runtime
|
||||
speedup is claimed. Cached-catalog SDK bulk validation was not measured here.
|
||||
|
||||
| Task (3 samples per lane) | Whole elapsed median, s | Family CPU median, s | Kernel family peak median, MiB | Whole-task estimated tokens |
|
||||
| ----------------------------------------------------------- | ----------------------: | -------------------: | -----------------------------: | --------------------------: |
|
||||
| LLVM: 256-byte blocks, offset 3, length 65,539 (257 blocks) | 29.842 → 8.584 | 25.286 → 6.358 | 629.5 → 672.4 | 62,000 → 62,000 |
|
||||
| LLVM: whole file, 65,536-byte blocks (1,883 blocks) | 12.314 → 12.115 | 7.581 → 7.325 | 667.9 → 674.3 | 91,900 → 91,900 |
|
||||
| ls: whole file, 256-byte blocks (592 blocks) | 7.075 → 8.286 | 5.804 → 5.528 | 659.1 → 659.3 | 67,215 → 67,215 |
|
||||
|
||||
The large fine-resolution task improves elapsed median by 71.2% and CPU by
|
||||
74.9%. Elapsed ranges are 28.541–33.290 → 7.981–10.396 seconds; CPU ranges
|
||||
23.600–25.429 → 6.049–6.633 seconds. Its kernel peak increases by about
|
||||
43 MiB (ranges 620.4–637.5 → 665.7–673.2 MiB). This is a speed/CPU improvement
|
||||
with higher measured peak memory, not a memory optimization.
|
||||
|
||||
The other controls do not establish a stable whole-task speedup. The small
|
||||
file's elapsed median is worse; ranges overlap at 6.469–9.892 → 6.169–9.188
|
||||
seconds. No general latency or memory benefit is claimed. Full-file reading
|
||||
and hashing remain necessary even when the selected window is short or empty.
|
||||
|
||||
A separate single real-file service probe delegated to actual `open/stat/read`
|
||||
while counting reads: 481,955 → 1,885 physical reads of all 123,379,936 bytes,
|
||||
maximum request 256 → 65,536 bytes, service elapsed 17.09 → 1.34 seconds.
|
||||
The complete Evidence objects were equal and independently checked. This probe
|
||||
locates the work removed; it is not the repeated public-task performance table.
|
||||
|
||||
The [machine-readable report](./byte-entropy-reads.json) preserves ranges,
|
||||
per-sample phase timings, calls, receipt hashes, artifact identities and commands.
|
||||
`cl100k_base` via tiktoken 0.14.0 counts the fixed question, server instructions,
|
||||
all tool names/descriptions/input schemas once, every argument and every text
|
||||
reply including export. Input-directory cost is 56,685 tokens; raw catalog with
|
||||
output schemas is separately 838,375. Actual model billing, host framing/replay,
|
||||
model reasoning quality and inference latency were not measured. No token saving
|
||||
is claimed: replies and catalog remain unchanged.
|
||||
|
||||
## Resource and correctness verification
|
||||
|
||||
All tasks ran serially under a one-CPU quota, 3 GiB early memory stop, 4 GiB hard
|
||||
limit, zero swap and 192-task ceiling. Whole process-family CPU and kernel memory
|
||||
peak include client/server, runtime fingerprint reads, serialization, source
|
||||
export and cleanup; kernel peak includes charged file cache. Independent oracle
|
||||
generation and tokenization are outside timed tasks. Payload exits, remaining
|
||||
owned processes and memory/PID-limit events were checked: all task payloads
|
||||
exited 0, cleanup verified, no guard stop/OOM. Final whole-scope counters were
|
||||
unavailable after transient-scope removal and are not represented as verified.
|
||||
|
||||
Executed additional checks:
|
||||
|
||||
- Six real CLI calls (three cases in both lanes): every oracle fact, complete
|
||||
CLI Evidence equality and matching MCP parent identity passed.
|
||||
- 36 tests in four exact source/public-boundary files passed, including short
|
||||
reads, unaligned windows across buffers, partial tails at minimum/intermediate/
|
||||
maximum block sizes, empty windows, cancellation through finalization,
|
||||
identity changes, capacity refusal, retained summaries and cleanup retry.
|
||||
- Six serial static tasks (typecheck, lint including module boundaries, format,
|
||||
knip, metadata and required compilation), plus architecture guards passed.
|
||||
Static family peak was 1.96 GiB; focused-test/preparation peak was 2.15 GiB.
|
||||
- An initial typecheck failure in the new short-read test's injected callback
|
||||
was fixed with an explicitly typed positional boundary. Its failed receipt is
|
||||
retained, not counted as a successful run. Initial local formatter path and
|
||||
early profile verification attempts also failed before their intended checks;
|
||||
corrected checks are recorded separately.
|
||||
|
||||
The public experiments qualify static byte inspection on these Linux files.
|
||||
They do not establish behavior on every filesystem or operating system and do
|
||||
not execute LLVM or ls. CLI/MCP contracts, Evidence fields, input limits and
|
||||
provider authority remain unchanged; generated catalogs and builds are ignored.
|
||||
|
||||
After integrating main `21a20306b`, the six serial static tasks, architecture
|
||||
guards and all 36 focused cases passed again. Three fresh real MCP/source-export
|
||||
tasks and three CLI calls matched every oracle fact and the frozen cohort's
|
||||
report/parent identity. The integrated test/preparation family peaked at
|
||||
2.23 GiB with verified payload exit/cleanup and no guard stop/OOM. These are
|
||||
integration qualification runs; they do not replace the fixed performance pins.
|
||||
The committed cost verifier also reproduced the complete measurement JSON.
|
||||
|
||||
## Reproduction
|
||||
|
||||
Build both pinned checkouts using their locked toolchain, serially. Supply an
|
||||
independent installed dependency tree matching the lock, without upgrading host
|
||||
software. Use the exact artifact digests above. The oracle accepts explicit
|
||||
`--path`, `--block-size`, `--offset`, `--length` and `--output`; omit length for
|
||||
the two whole-file controls. For example:
|
||||
|
||||
```sh
|
||||
python3 experiments/analysis-efficiency/byte-entropy-oracle.py \
|
||||
--path /usr/lib/x86_64-linux-gnu/libLLVM-16.so.1 \
|
||||
--block-size 256 --offset 3 --length 65539 --output /absolute/oracle.json
|
||||
python3 experiments/analysis-efficiency/resource-run.py --observe-payload-exit -- \
|
||||
node experiments/analysis-efficiency/byte-entropy-reads.mjs \
|
||||
--runtime /absolute/pinned-checkout --oracle /absolute/oracle.json \
|
||||
--identity /absolute/runtime-identity.json --output /absolute/new-task-directory
|
||||
```
|
||||
|
||||
Runtime identity JSON contains `source_revision`, `source_clean`,
|
||||
`node_sha256` and `files: [{path, sha256}]` with absolute paths. Fingerprint the
|
||||
entire compiled runtime and catalogs, scripts, lock/toolchain files, actual SDK
|
||||
JavaScript dependencies and the driver/helpers. Both sessions must use the same
|
||||
Node and dependency contents. Use the same one-CPU/memory policy for the harness;
|
||||
the historical runs additionally pinned the payload to CPUs 0–1 and the initial
|
||||
observer to CPU 15. Do not compare an unguarded default-concurrency run as an
|
||||
equivalent resource measurement.
|
||||
|
||||
Retained raw receipts are on the investigation host at
|
||||
`/tmp/rea-perf-byte-entropy-20261024/`. This includes `cohort/index.json`,
|
||||
per-task catalog/transcript/report/source export/guard logs, runtime identities,
|
||||
three oracle JSONs, CLI stdout/stderr, boundary profiles and verification logs.
|
||||
The cost verifier takes that receipt directory and a separate output:
|
||||
|
||||
```sh
|
||||
python3 experiments/analysis-efficiency/byte-entropy-costs.py \
|
||||
--receipts /absolute/receipt-directory --output /absolute/costs.json
|
||||
```
|
||||
|
||||
It requires all 18 successful verified samples, matching parent/report/catalog
|
||||
identity across lanes, successful source exports and oracle/runtime receipts.
|
||||
Failed or stopped attempts cannot enter the published cohort.
|
||||
@@ -1,154 +0,0 @@
|
||||
/** Complete entropy question through production stdio, with full source export. */
|
||||
import assert from "node:assert/strict";
|
||||
import { createHash } from "node:crypto";
|
||||
import { createWriteStream } from "node:fs";
|
||||
import { mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join, resolve } from "node:path";
|
||||
import { finished } from "node:stream/promises";
|
||||
import { parseArgs } from "node:util";
|
||||
import { Client } from "@modelcontextprotocol/client";
|
||||
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
|
||||
import { createRetainedQueryCaller } from "./retained-query-calls.mjs";
|
||||
import { finalizeRetainedQueryReceipt } from "./retained-query-finalization.mjs";
|
||||
|
||||
const { values } = parseArgs({
|
||||
options: Object.fromEntries(
|
||||
["runtime", "oracle", "identity", "output"].map((name) => [
|
||||
name,
|
||||
{ type: "string" },
|
||||
]),
|
||||
),
|
||||
});
|
||||
for (const value of Object.values(values)) assert.equal(value, resolve(value));
|
||||
const sha256 = (value) => createHash("sha256").update(value).digest("hex");
|
||||
const oracleBytes = await readFile(values.oracle);
|
||||
const oracle = JSON.parse(oracleBytes);
|
||||
const identityBytes = await readFile(values.identity);
|
||||
const identity = JSON.parse(identityBytes);
|
||||
await mkdir(values.output, { mode: 0o700 });
|
||||
const account = await mkdtemp(join(tmpdir(), "rea-entropy-task-"));
|
||||
const client = new Client({ name: "entropy-complete-task", version: "1" });
|
||||
const transport = new StdioClientTransport({
|
||||
command: process.execPath,
|
||||
args: [join(values.runtime, "scripts/rea.mjs"), "mcp"],
|
||||
maxBufferSize: 10485760,
|
||||
env: {
|
||||
HOME: account,
|
||||
TMPDIR: account,
|
||||
PATH: process.env.PATH,
|
||||
REA_LOG_LEVEL: "silent",
|
||||
NODE_OPTIONS: "--max-old-space-size=2048",
|
||||
NODE_DISABLE_COMPILE_CACHE: "1",
|
||||
UV_THREADPOOL_SIZE: "2",
|
||||
},
|
||||
stderr: "pipe",
|
||||
});
|
||||
const serverLog = createWriteStream(join(values.output, "server.log"));
|
||||
const serverLogFinished = finished(serverLog);
|
||||
serverLogFinished.catch(() => {});
|
||||
transport.stderr?.pipe(serverLog);
|
||||
const report = {
|
||||
status: "running",
|
||||
question:
|
||||
"Return the file SHA-256, file-wide entropy, every entropy block and adjacent high/low run in the requested byte window; preserve the complete source Evidence.",
|
||||
request: oracle.input,
|
||||
calls: [],
|
||||
phases: {},
|
||||
client:
|
||||
"Production stdio SDK, exact discovered toolDefinition and default validator",
|
||||
runtime_identity_sha256: sha256(identityBytes),
|
||||
oracle_sha256: sha256(oracleBytes),
|
||||
driver_sha256: sha256(await readFile(new URL(import.meta.url))),
|
||||
};
|
||||
const transcript = [];
|
||||
let definitions;
|
||||
const call = createRetainedQueryCaller(
|
||||
{
|
||||
callTool(params, options) {
|
||||
return client.callTool(params, {
|
||||
...options,
|
||||
toolDefinition: definitions.get(params.name),
|
||||
});
|
||||
},
|
||||
},
|
||||
(receipt, event) => {
|
||||
report.calls.push(receipt);
|
||||
transcript.push(event);
|
||||
},
|
||||
);
|
||||
const observe = async (name, args, phase) => {
|
||||
const result = await call(name, args, phase);
|
||||
assert.notEqual(result.isError, true, JSON.stringify(result));
|
||||
assert.ok(result.structuredContent);
|
||||
return result.structuredContent;
|
||||
};
|
||||
const verifyRuntime = async () => {
|
||||
assert.equal(sha256(await readFile(values.oracle)), sha256(oracleBytes));
|
||||
assert.equal(sha256(await readFile(values.identity)), sha256(identityBytes));
|
||||
assert.equal(sha256(await readFile(process.execPath)), identity.node_sha256);
|
||||
assert.equal(
|
||||
sha256(await readFile(new URL(import.meta.url))),
|
||||
report.driver_sha256,
|
||||
);
|
||||
for (const item of identity.files)
|
||||
assert.equal(sha256(await readFile(item.path)), item.sha256, item.path);
|
||||
};
|
||||
try {
|
||||
await verifyRuntime();
|
||||
const start = performance.now();
|
||||
await client.connect(transport);
|
||||
report.server_instructions = client.getInstructions() ?? "";
|
||||
const catalog = await client.listTools();
|
||||
definitions = new Map(catalog.tools.map((tool) => [tool.name, tool]));
|
||||
await writeFile(join(values.output, "catalog.json"), JSON.stringify(catalog));
|
||||
report.phases.discovery_ms = performance.now() - start;
|
||||
const parent = await observe("inspect_byte_entropy", oracle.input, "inspect");
|
||||
for (const [key, value] of Object.entries(oracle.expected))
|
||||
assert.deepEqual(parent.normalized_result[key], value, key);
|
||||
report.parent_evidence_id = parent.evidence_id;
|
||||
report.source_report_sha256 = sha256(
|
||||
JSON.stringify(parent.normalized_result),
|
||||
);
|
||||
report.artifact = parent.normalized_result.artifact;
|
||||
await observe(
|
||||
"export_evidence_bundle",
|
||||
{ path: join(values.output, "source.evidence.json") },
|
||||
"export",
|
||||
);
|
||||
const bundle = JSON.parse(
|
||||
await readFile(join(values.output, "source.evidence.json")),
|
||||
);
|
||||
assert.deepEqual(
|
||||
bundle.records.find((item) => item.evidence_id === parent.evidence_id),
|
||||
parent,
|
||||
);
|
||||
report.complete_source_export_verified = true;
|
||||
report.status = "passed";
|
||||
} catch (cause) {
|
||||
report.status = "failed";
|
||||
report.error = String(cause);
|
||||
} finally {
|
||||
try {
|
||||
await verifyRuntime();
|
||||
} catch (cause) {
|
||||
report.status = "failed";
|
||||
report.identity_failure = String(cause);
|
||||
}
|
||||
await finalizeRetainedQueryReceipt({
|
||||
client,
|
||||
transport,
|
||||
account,
|
||||
serverLog,
|
||||
serverLogFinished,
|
||||
report,
|
||||
transcript,
|
||||
output: values.output,
|
||||
});
|
||||
if (
|
||||
report.status !== "passed" ||
|
||||
report.cleanup_failures.length ||
|
||||
report.receipt_failures.length
|
||||
)
|
||||
process.exitCode = 1;
|
||||
}
|
||||
@@ -1,32 +0,0 @@
|
||||
"""Count all actual client JSON inputs and preserve bounded family resource receipts."""
|
||||
from pathlib import Path
|
||||
import argparse, json, hashlib
|
||||
import tiktoken
|
||||
parser=argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--root',type=Path,required=True)
|
||||
parser.add_argument('--output',type=Path,required=True)
|
||||
args=parser.parse_args();lab=args.root;enc=tiktoken.get_encoding('cl100k_base')
|
||||
compact=lambda value:json.dumps(value,ensure_ascii=False,separators=(',',':'),allow_nan=False)
|
||||
count=lambda value:len(enc.encode(compact(value),disallowed_special=()))
|
||||
def resources(root):
|
||||
scope=json.loads(Path(str(root)+'.log').read_text().splitlines()[-1]);o=scope['payload_resource_observation']
|
||||
assert scope['abort'] is None and scope['payload_resource_observation_verified'] and o['payload_exit_code']==0 and o['payload_cleanup_verified']
|
||||
for event in o['resource_events'].values():assert not any(event.values())
|
||||
return {'wall_seconds':scope['elapsed_seconds'],'cpu_seconds':o['cpu_stat']['usage_usec']/1e6,'family_peak_mib':o['memory_peak_bytes']/1048576,'payload_cleanup_verified':True,'final_whole_scope_counters_verified':scope['resource_counters_verified']}
|
||||
wire=[];sdk=[]
|
||||
for root in sorted(lab.glob('final-wire-*')):
|
||||
if not root.is_dir():continue
|
||||
report=json.loads((root/'report.json').read_text());requests=json.loads((root/'model-requests.json').read_text());assert report['status'] in ['passed','expected-incomplete-task'] and not report['cleanup_failures'] and report['appserver_exit_code']==0 and report['owned_scope_drain']['remaining_processes']==[]
|
||||
rows=[]
|
||||
for index,captured in enumerate(requests):
|
||||
body=captured['body'];namespace=[n for n in body.get('tools',[]) if n.get('type')=='namespace' and n.get('name')=='mcp__rea_audit'];compaction=not namespace
|
||||
if compaction:assert 'CONTEXT CHECKPOINT COMPACTION' in compact(body['input'][-1])
|
||||
else:assert len(namespace)==1 and len(namespace[0]['tools'])==210
|
||||
rows.append({'index':index+1,'kind':'compaction' if compaction else 'normal','whole_json_tokens_proxy':count(body),'rea_namespace_tokens_proxy':0 if not namespace else count(namespace[0]),'rea_function_count':0 if not namespace else len(namespace[0]['tools'])})
|
||||
wire.append({'case':root.name,'task_completed':report['task_completed'],'mode':report['mode'],'budget':report['observations']['tool_output_budget'],'requests':len(rows),'compactions':sum(r['kind']=='compaction' for r in rows),'whole_json_tokens_proxy':sum(r['whole_json_tokens_proxy'] for r in rows),'request_details':rows,**resources(root)})
|
||||
for root in sorted(lab.glob('final-sdk-*')):
|
||||
if not root.is_dir():continue
|
||||
report=json.loads((root/'report.json').read_text());assert report['status']=='passed' and not report['cleanup_failures'] and not report['receipt_failures']
|
||||
sdk.append({'case':root.name,'provider':report['provider'],'symbol':report['symbol'],'detail':report['detail'],'warm':report['warm'],'tool_calls':len(report['calls']),'exported_bundle_bytes':(root/'evidence.json').stat().st_size,'disassembly_calls':[r for r in report['calls'] if r['phase'].startswith('disassembly:')],**resources(root)})
|
||||
receipt={'measurement_scope':'Actual public SDK and Codex0.163 app-server with real GDB/BFD. Local deterministic SSE fixture and observed-ID checkpoint; no AI inference, model usage/quality/acceptance or billing claim. Whole compact JSON serialization tokens are proxies and include all schemas/replays/compaction. Incomplete default baseline is excluded from completed-workflow gains. Fresh processes have unflushed OS caches; warm SDK cases repeat the selected disassembly in the same server. Native target never executed.','tokenizer':{'package':tiktoken.__version__,'encoding':enc.name},'runtime_pins':{variant:{k:v for k,v in json.loads((lab/(variant+'-manifest.json')).read_text()).items() if k!='files'} for variant in ['baseline','candidate']},'native':json.loads((lab/'native-prerequisite-manifest.json').read_text()),'wire':wire,'sdk':sdk}
|
||||
args.output.write_text(json.dumps(receipt,indent=2)+'\n');print(json.dumps({'wire':[{k:v for k,v in row.items() if k!='request_details'} for row in wire],'sdk':[{k:v for k,v in row.items() if k!='disassembly_calls'} for row in sdk]},indent=2))
|
||||
@@ -1,22 +0,0 @@
|
||||
from pathlib import Path
|
||||
import struct,hashlib,json,argparse
|
||||
parser=argparse.ArgumentParser(description='Independent ELF64 little-endian native-command byte oracle');parser.add_argument('--input',type=Path,required=True);parser.add_argument('--output',type=Path,required=True);args=parser.parse_args();path=args.input;b=path.read_bytes();assert b[:6]==b'\x7fELF\x02\x01'
|
||||
shoff=struct.unpack_from('<Q',b,40)[0];entsize,n,strings=struct.unpack_from('<HHH',b,58)
|
||||
assert entsize==64 and shoff+n*entsize<=len(b)
|
||||
sections=[struct.unpack_from('<IIQQQQIIQQ',b,shoff+i*entsize) for i in range(n)]
|
||||
rows=[]
|
||||
for si,s in enumerate(sections):
|
||||
if s[1]!=11:continue
|
||||
text=sections[s[6]];names=b[text[4]:text[4]+text[5]]
|
||||
assert s[9]==24 and s[5]%s[9]==0
|
||||
for ordinal,off in enumerate(range(s[4],s[4]+s[5],24)):
|
||||
name,info,other,section,value,size=struct.unpack_from('<IBBHQQ',b,off)
|
||||
end=names.find(b'\0',name);assert end>=name
|
||||
if names[name:end] not in [b'execute_command',b'execute_command_internal'] or info&15!=2:continue
|
||||
target=sections[section];file_offset=target[4]+value-target[3]
|
||||
assert target[4]<=file_offset and file_offset+size<=target[4]+target[5]
|
||||
rows.append({'symbol_table_section':si,'symbol_ordinal':ordinal,'name':names[name:end].decode('ascii'),'address':value,'bytes':size,'file_offset':file_offset,'first_16_bytes_hex':b[file_offset:file_offset+16].hex()})
|
||||
assert rows
|
||||
result={'path':str(path),'artifact_sha256':hashlib.sha256(b).hexdigest(),'artifact_bytes':len(b),'elf_class':64,'byte_order':'little','elf_machine':struct.unpack_from('<H',b,18)[0],'symbols':rows,'scope':'Independent ELF64 little-endian dynamic symbol/section mapping; no target execution or REA parser.'}
|
||||
args.output.write_text(json.dumps(result,indent=2)+'\n')
|
||||
print(json.dumps(result))
|
||||
@@ -1,212 +0,0 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { createHash } from "node:crypto";
|
||||
import { createWriteStream } from "node:fs";
|
||||
import { mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises";
|
||||
import { createRequire } from "node:module";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join, resolve } from "node:path";
|
||||
import { finished } from "node:stream/promises";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { parseArgs } from "node:util";
|
||||
import { createRetainedQueryCaller } from "./retained-query-calls.mjs";
|
||||
import { finalizeRetainedQueryReceipt } from "./retained-query-finalization.mjs";
|
||||
|
||||
const { values } = parseArgs({
|
||||
options: {
|
||||
runtime: { type: "string" },
|
||||
input: { type: "string" },
|
||||
oracle: { type: "string" },
|
||||
output: { type: "string" },
|
||||
provider: { type: "string" },
|
||||
symbol: { type: "string" },
|
||||
detail: { type: "string", default: "complete" },
|
||||
warm: { type: "boolean", default: false },
|
||||
},
|
||||
});
|
||||
for (const key of ["runtime", "input", "oracle", "output"])
|
||||
assert.equal(values[key], resolve(values[key]));
|
||||
assert.ok(["gdb", "objdump"].includes(values.provider));
|
||||
assert.ok(["complete", "summary"].includes(values.detail));
|
||||
const oracle = JSON.parse(await readFile(values.oracle, "utf8"));
|
||||
assert.ok(oracle.symbols.some((row) => row.name === values.symbol));
|
||||
const inputBytes = await readFile(values.input);
|
||||
assert.equal(
|
||||
createHash("sha256").update(inputBytes).digest("hex"),
|
||||
oracle.artifact_sha256,
|
||||
);
|
||||
const require = createRequire(join(values.runtime, "package.json"));
|
||||
const { Client } = await import(
|
||||
pathToFileURL(require.resolve("@modelcontextprotocol/client"))
|
||||
);
|
||||
const { StdioClientTransport } = await import(
|
||||
pathToFileURL(require.resolve("@modelcontextprotocol/client/stdio"))
|
||||
);
|
||||
await mkdir(values.output, { mode: 0o700 });
|
||||
const account = await mkdtemp(join(tmpdir(), "rea-command-views-"));
|
||||
const client = new Client({ name: "command-observation-task", version: "1" });
|
||||
const transport = new StdioClientTransport({
|
||||
command: process.execPath,
|
||||
args: [join(values.runtime, "scripts/rea.mjs"), "mcp"],
|
||||
env: {
|
||||
HOME: account,
|
||||
PATH: process.env.PATH,
|
||||
NODE_OPTIONS: "--max-old-space-size=2048",
|
||||
UV_THREADPOOL_SIZE: "1",
|
||||
REA_LOG_LEVEL: "silent",
|
||||
},
|
||||
stderr: "pipe",
|
||||
});
|
||||
const serverLog = createWriteStream(join(values.output, "server.log"));
|
||||
const serverLogFinished = finished(serverLog);
|
||||
serverLogFinished.catch(() => {});
|
||||
transport.stderr?.pipe(serverLog);
|
||||
const transcript = [];
|
||||
const report = {
|
||||
question:
|
||||
"Read architecture and the complete byte-annotated disassembly of one selected native function; export complete Evidence. No inferior execution.",
|
||||
provider: values.provider,
|
||||
symbol: values.symbol,
|
||||
detail: values.detail,
|
||||
warm: values.warm,
|
||||
calls: [],
|
||||
observations: [],
|
||||
};
|
||||
const call = createRetainedQueryCaller(client, (receipt, event) => {
|
||||
report.calls.push(receipt);
|
||||
transcript.push(event);
|
||||
});
|
||||
let sessionId;
|
||||
try {
|
||||
await client.connect(transport);
|
||||
const catalog = await client.listTools();
|
||||
await writeFile(join(values.output, "catalog.json"), JSON.stringify(catalog));
|
||||
const definitions = new Map(catalog.tools.map((tool) => [tool.name, tool]));
|
||||
const originalCall = client.callTool.bind(client);
|
||||
client.callTool = (input, options) =>
|
||||
originalCall(input, {
|
||||
...options,
|
||||
timeout: 60000,
|
||||
toolDefinition: definitions.get(input.name),
|
||||
});
|
||||
const observe = async (name, input, phase = name) => {
|
||||
const response = await call(name, input, phase);
|
||||
assert.ok(!response.isError, JSON.stringify(response));
|
||||
report.observations.push({
|
||||
name,
|
||||
phase,
|
||||
evidence: response.structuredContent,
|
||||
});
|
||||
return response.structuredContent;
|
||||
};
|
||||
const completeParents = [];
|
||||
const disassemble = async (phase) => {
|
||||
const selection = values.detail === "summary" ? { detail: "summary" } : {};
|
||||
const result =
|
||||
values.provider === "gdb"
|
||||
? await observe(
|
||||
"gdb_console",
|
||||
{
|
||||
session_id: sessionId,
|
||||
command: `disassemble /r ${values.symbol}`,
|
||||
...selection,
|
||||
},
|
||||
phase,
|
||||
)
|
||||
: await observe(
|
||||
"inspect_with_objdump",
|
||||
{
|
||||
path: values.input,
|
||||
operation: "disassemble",
|
||||
symbol: values.symbol,
|
||||
...selection,
|
||||
},
|
||||
phase,
|
||||
);
|
||||
completeParents.push(result);
|
||||
};
|
||||
if (values.provider === "gdb") {
|
||||
const started = await observe("start_gdb_session", {});
|
||||
sessionId = started.session_id;
|
||||
for (const command of [
|
||||
"set auto-load off",
|
||||
"set debuginfod enabled off",
|
||||
`file ${values.input}`,
|
||||
"show architecture",
|
||||
])
|
||||
await observe("gdb_console", { session_id: sessionId, command });
|
||||
} else
|
||||
await observe("inspect_with_objdump", {
|
||||
path: values.input,
|
||||
operation: "file_headers",
|
||||
});
|
||||
await disassemble("disassembly:fresh");
|
||||
if (values.warm) await disassemble("disassembly:repeat");
|
||||
if (sessionId) {
|
||||
await observe("close_gdb_session", { session_id: sessionId });
|
||||
sessionId = undefined;
|
||||
}
|
||||
await observe("export_evidence_bundle", {
|
||||
path: join(values.output, "evidence.json"),
|
||||
});
|
||||
const bundle = JSON.parse(
|
||||
await readFile(join(values.output, "evidence.json"), "utf8"),
|
||||
);
|
||||
for (const delivered of completeParents) {
|
||||
const parent =
|
||||
values.detail === "complete"
|
||||
? delivered
|
||||
: bundle.records.find(
|
||||
(record) =>
|
||||
record.evidence_id ===
|
||||
delivered.normalized_result.parent_evidence_id,
|
||||
);
|
||||
assert.ok(parent);
|
||||
assert.equal(parent.normalized_result.completion_status, "complete");
|
||||
assert.equal(parent.normalized_result.output_truncated, false);
|
||||
if (values.provider === "gdb") assert.equal(parent.subject, null);
|
||||
else assert.equal(parent.subject.digest.sha256, oracle.artifact_sha256);
|
||||
if (values.detail === "summary") {
|
||||
const selected = delivered.normalized_result.summary;
|
||||
const { available_counts: counts, ...facts } = selected;
|
||||
const { console, target, log, ...records } = parent.normalized_result;
|
||||
assert.deepEqual(
|
||||
values.provider === "gdb" ? facts : selected,
|
||||
values.provider === "gdb" ? records : parent.normalized_result,
|
||||
);
|
||||
if (values.provider === "gdb")
|
||||
assert.equal(counts.console_utf16_units, console.length);
|
||||
assert.equal(delivered.raw_result, null);
|
||||
}
|
||||
}
|
||||
report.status = "passed";
|
||||
} catch (error) {
|
||||
report.status = "failed";
|
||||
report.failure = String(error?.stack ?? error);
|
||||
process.exitCode = 1;
|
||||
} finally {
|
||||
if (sessionId)
|
||||
try {
|
||||
const closed = await call(
|
||||
"close_gdb_session",
|
||||
{ session_id: sessionId },
|
||||
"cleanup:gdb",
|
||||
);
|
||||
assert.ok(!closed.isError);
|
||||
} catch (error) {
|
||||
report.gdb_cleanup_failure = String(error);
|
||||
process.exitCode = 1;
|
||||
}
|
||||
if (
|
||||
!(await finalizeRetainedQueryReceipt({
|
||||
report,
|
||||
client,
|
||||
transport,
|
||||
account,
|
||||
serverLog,
|
||||
serverLogFinished,
|
||||
output: values.output,
|
||||
transcript,
|
||||
}))
|
||||
)
|
||||
process.exitCode = 1;
|
||||
}
|
||||
@@ -1,28 +0,0 @@
|
||||
from pathlib import Path
|
||||
import json,re,hashlib,argparse
|
||||
parser=argparse.ArgumentParser(description='Match every native instruction byte against independently mapped ELF file ranges');parser.add_argument('--root',type=Path,required=True);args=parser.parse_args();lab=args.root;oracle=json.loads((lab/'elf-oracle.json').read_text());image=Path(oracle['path']).read_bytes();assert hashlib.sha256(image).hexdigest()==oracle['artifact_sha256'];rows=[]
|
||||
for directory in sorted(lab.iterdir()):
|
||||
bundle=directory/'evidence.json'
|
||||
if not bundle.is_file():continue
|
||||
for parent in json.loads(bundle.read_text())['records']:
|
||||
if parent['operation'] not in ['gdb_console','inspect_with_objdump']:continue
|
||||
params=parent['parameters'];gdb=parent['operation']=='gdb_console'
|
||||
symbol=params.get('command','').removeprefix('disassemble /r ') if gdb else params.get('symbol')
|
||||
if symbol not in {s['name'] for s in oracle['symbols']}:continue
|
||||
spec=next(s for s in oracle['symbols'] if s['name']==symbol)
|
||||
report=parent['normalized_result'];assert report['completion_status']=='complete' and not report['output_truncated']
|
||||
if gdb:
|
||||
assert parent['subject'] is None
|
||||
console='\n'.join(line for line in report['records'] if line.startswith('~'));assert console==report['console']
|
||||
text=''.join(json.loads(line[1:]) for line in report['records'] if line.startswith('~'))
|
||||
pattern=r'^\s*(?:=>\s*)?0x([0-9a-f]+)[^:]*:\s*((?:[0-9a-f]{2}(?:\s+|$))+)'
|
||||
else:
|
||||
assert parent['subject']['digest']['sha256']==oracle['artifact_sha256'];text=report['stdout'];pattern=r'^\s*([0-9a-f]+):\s*((?:[0-9a-f]{2}(?:\s+|$))+)'
|
||||
address=spec['address'];collected=bytearray();count=0
|
||||
for line in text.splitlines():
|
||||
m=re.match(pattern,line)
|
||||
if not m:continue
|
||||
actual=int(m[1],16);raw=bytes.fromhex(m[2]);assert actual==address+len(collected),(directory,actual,address+len(collected));collected.extend(raw);count+=1
|
||||
expected=image[spec['file_offset']:spec['file_offset']+spec['bytes']];assert collected==expected,(directory,len(collected),len(expected));rows.append({'case':directory.name,'mode':'gdb' if gdb else 'objdump','symbol':symbol,'byte_bearing_lines':count,'bytes':len(collected),'exact_file_bytes':True,'parent_evidence_id':parent['evidence_id']})
|
||||
assert rows
|
||||
(lab/'native-final-verification.json').write_text(json.dumps({'status':'passed','artifact_sha256':oracle['artifact_sha256'],'records':rows},indent=2)+'\n');print(json.dumps(rows))
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,139 +0,0 @@
|
||||
# Retained command observations
|
||||
|
||||
Addresses [#5584](https://github.com/morluto/rea/issues/5584). GDB command Evidence
|
||||
repeats ordered MI records in stream groups and a raw mirror; objdump repeats
|
||||
stdout/stderr in its raw mirror. Explicit `detail: summary` retains the complete
|
||||
parent and projects existing owner facts once. Default complete Evidence is
|
||||
unchanged. This adds no tool, MI decoder, live-state cache or command authority.
|
||||
|
||||
GDB summaries preserve **every ordered record**, including asynchronous and
|
||||
unrecognized records, correlated MI result, command, completion/truncation,
|
||||
invalid-UTF-8 and process-exit facts. They omit the convenience console/target/log
|
||||
groupings and report their original UTF-16 sizes. Objdump summaries preserve
|
||||
opaque stdout/stderr, exit/signal, completion and pre/post digest-change facts.
|
||||
Both carry source parameters/provider/profile, parent identity, limitations and
|
||||
unknowns. Subjectless observations stay subjectless. Historical observations do
|
||||
not establish current target state or global coverage. The existing
|
||||
`inspect_analysis_view` summary operation works after debugger close.
|
||||
|
||||
The domain codec now owns the command fact shapes used by contracts and GDB's
|
||||
capture type; protocol interpretation stays with its provider. MCP requires
|
||||
retention before command execution. CLI uses the existing shared projection and
|
||||
nonoverwriting bundle publisher after provider cleanup:
|
||||
|
||||
```sh
|
||||
rea inspect-with-objdump /usr/bin/bash disassemble --symbol execute_command \
|
||||
--detail summary --evidence_output /tmp/bash-command-evidence.json --json
|
||||
```
|
||||
|
||||
## Real input and correctness
|
||||
|
||||
Linux x86-64, Node 24.18.0/npm 11.16.0; GNU GDB 13.1 (Debian 13.1-3),
|
||||
GNU objdump 2.40, installed Bash ELF 1,265,648 bytes, SHA-256
|
||||
`55b89ab22bee4792a210f493a53fb066accd5d30b69837c28d98be5ff863efcf`.
|
||||
The [receipt](command-observation-views.json) pins native binaries, baseline main
|
||||
`c2858d2aa0916f0dd2d073d848b8c2195d9bf6b3`, source patch and packaged runtime digests.
|
||||
No target execution, attach, GUI or database mutation occurred.
|
||||
|
||||
An [independent ELF reader](command-observation-oracle.py) maps dynamic symbols
|
||||
to exact file ranges: `execute_command`, 239 bytes at `0x4ba50`, and
|
||||
`execute_command_internal`, 15,428 bytes at `0x47e00`. The
|
||||
[verifier](command-observation-verify.py) checks every disassembled byte and its
|
||||
contiguous address against those ranges. GDB reports 66/3,365 instructions;
|
||||
objdump byte lines also include continuations, so their count is not an
|
||||
instruction count. Ordered summary facts match each exported complete parent.
|
||||
GDB Evidence has a null subject; the external oracle does not manufacture a
|
||||
target binding. Actual CLI default objdump Evidence matches the baseline exactly,
|
||||
including identity; symbol/pathless summaries preserve it in bundles. CLI checks
|
||||
also cover rejected overwrite, missing destination before dispatch, and a real
|
||||
GDB command with a caller-owned PTY that remains available after debugger exit.
|
||||
|
||||
## Whole tasks and resource costs
|
||||
|
||||
The [SDK driver](command-observation-queries.mjs) discovers all 210 tools, reads
|
||||
architecture and symbol-scoped disassembly, closes GDB where applicable, exports
|
||||
complete Evidence and finalizes owned resources. GDB uses 8 calls; objdump uses 3.
|
||||
Warm cases repeat disassembly in the same process, adding one call. Fresh-process
|
||||
cases have **unflushed OS caches**, not disk-cold inputs. Native commands are
|
||||
rerun; results are not reused as current live state.
|
||||
|
||||
The [actual Codex 0.163 app-server driver](command-observation-wire.py) uses a
|
||||
loopback deterministic SSE fixture, no AI inference or paid service. All 210
|
||||
function schemas are present in every normal model-facing request. Matched tasks
|
||||
use full input/output profiles; only disassembly selects summary. The
|
||||
[counter](command-observation-costs.py) counts compact JSON serialization with
|
||||
`tiktoken 0.12.0 / cl100k_base` across **every** request, including schemas,
|
||||
replayed results and the actual client's local compaction request. The fixture
|
||||
answers compaction with observed identifiers and the retained-body location.
|
||||
These are serialization-token proxies, not model rendering, acceptance, answer
|
||||
quality, actual usage or billing. Compaction behavior is qualified only for this
|
||||
client/configuration; no inference quality is established.
|
||||
|
||||
| Completed GDB task | JSON token proxy before → after | Requests | Wall seconds | CPU seconds | Family peak MiB |
|
||||
| ------------------------------ | ------------------------------: | -------: | ------------: | ------------: | --------------: |
|
||||
| Small, 30,000 per-tool budget | 605,133 → 580,070 | 9 → 9 | 7.906 → 8.618 | 7.195 → 7.832 | 788.9 → 785.9 |
|
||||
| Large, 350,000 per-tool budget | 1,269,405 → 1,091,290 | 10 → 9 | 9.551 → 9.127 | 8.832 → 8.370 | 805.2 → 770.8 |
|
||||
|
||||
Whole-task context proxies decrease **4.14% / 14.03%**, rather than the much
|
||||
larger reply-byte reduction. The large baseline makes one compaction request;
|
||||
the candidate makes none. The small task is slower. These single paired samples
|
||||
do not establish a general speedup. With the client's default output budget,
|
||||
the small baseline delivers truncated invalid inner JSON and does **not**
|
||||
complete the question; summary completes it. That incomplete baseline is
|
||||
excluded from the percentage comparison.
|
||||
|
||||
| SDK task | Reply bytes before → after | Whole wall seconds | CPU seconds | Family peak MiB |
|
||||
| ------------- | -------------------------: | -----------------: | ------------: | --------------: |
|
||||
| GDB small | 54,123 → 18,009 | 6.631 → 5.447 | 6.102 → 5.240 | 536.3 → 535.8 |
|
||||
| GDB large | 2,763,983 → 700,453 | 6.965 → 6.660 | 6.729 → 6.451 | 595.0 → 545.8 |
|
||||
| Objdump small | 22,467 → 15,990 | 7.599 → 7.699 | 7.110 → 7.264 | 527.8 → 532.5 |
|
||||
| Objdump large | 964,889 → 487,219 | 7.737 → 7.499 | 7.334 → 6.935 | 537.0 → 536.6 |
|
||||
|
||||
Same-process repeat disassembly takes 1,100 → 1,016 ms for GDB and
|
||||
1,070 → 939 ms for objdump in these observations. Whole warm-task CPU improves
|
||||
7.613 → 7.162 s for GDB but increases 7.819 → 7.859 s for objdump. Timings include
|
||||
existing ownership checks; no check was removed or cached.
|
||||
|
||||
Retaining a summary also retains an additional derived record: large GDB bundles
|
||||
grow 1,310,073 → 1,640,022 bytes; objdump 474,288 → 712,656. This trades retained
|
||||
storage for smaller delivery; long-session heap growth is unmeasured. Retained-data lifetime follows the existing investigation ledger and cleanup
|
||||
owners. Output still scales
|
||||
with the requested complete disassembly; summary is not a fixed-size preview.
|
||||
|
||||
Each experiment runs in one owned systemd scope, one CPU, no swap, 4 GiB hard /
|
||||
3 GiB early memory thresholds and host-memory admission/stop thresholds. Recorded
|
||||
payload exit and cleanup are verified, with zero OOM/resource events. Final
|
||||
whole-scope counters disappear after teardown and remain explicitly unverified.
|
||||
The receipt includes CPU time, wall time, peak family memory, warm cases and
|
||||
bundle growth. Build/check scopes use separate bounded budgets. Host-memory
|
||||
check stops and initial fixture failures are preserved locally and are not passes.
|
||||
|
||||
## Coverage and reproduction
|
||||
|
||||
320 focused tests cover the new SDK schemas/views plus existing GDB queue,
|
||||
malformed output, cancellation, process exit, target identity and cleanup cases,
|
||||
objdump symbol selection and command admission. Real provider evidence is from
|
||||
the CLI/SDK/client runs above, not simulated-provider tests.
|
||||
|
||||
Run the oracle for the selected Bash image, then use the SDK driver with
|
||||
`--runtime`, `--input`, `--oracle`, `--output`, `--provider gdb|objdump`,
|
||||
`--symbol`, `--detail complete|summary` and optional `--warm`. The wire driver
|
||||
also takes the fixed Codex/Node paths, `--mode`, `--symbol` and `--budget`; it always performs the
|
||||
export (eight tool calls). Run each through `resource-run.py --observe-payload-exit`, preserving
|
||||
its sibling `.log`; the verifier and counter accept `--root` for those results.
|
||||
|
||||
Default Rizin execution returned ENOENT with no configured alternative. Cutter
|
||||
reported no live bridge; no configured selector or default bridge directory was
|
||||
present. Real Rizin/Cutter performance and GUI workflows remain unverified.
|
||||
Their command semantics were not changed. Existing objdump native symbol
|
||||
selection was reused. No claim covers arbitrary GDB target execution or another
|
||||
host client's output budget.
|
||||
|
||||
Review follow-up adds nine CLI publication regressions (16 CLI tests including the
|
||||
existing cleanup cases), three deterministic delivery-oracle tests, and a focused
|
||||
strict typecheck of the new CLI boundary and its transitive source. Real client
|
||||
small/default and large/generous summary runs pass the strengthened oracle and
|
||||
export complete parents; only the explicit complete/default baseline may finish
|
||||
with an incomplete answer. The unused export switch was removed: export is
|
||||
mandatory in this question. Product source and the original paired measurements
|
||||
are unchanged; the receipt records these additional validation runs separately.
|
||||
@@ -1,301 +0,0 @@
|
||||
"""Actual Codex wire capture with a deterministic local SSE backend; no AI inference."""
|
||||
import argparse
|
||||
import http.server
|
||||
import threading
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import time
|
||||
import traceback
|
||||
import subprocess
|
||||
from command_observation_delivery import verify_command_delivery
|
||||
|
||||
|
||||
async def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument("--runtime", type=Path, required=True)
|
||||
parser.add_argument("--codex", type=Path, required=True)
|
||||
parser.add_argument("--node", type=Path, required=True)
|
||||
parser.add_argument("--input-profile", choices=["full", "compact"], default="full")
|
||||
parser.add_argument("--output-profile", choices=["full", "off"], default="full")
|
||||
parser.add_argument("--input", type=Path, required=True)
|
||||
parser.add_argument("--oracle", type=Path, required=True)
|
||||
parser.add_argument("--expected-tools", type=int, default=210)
|
||||
parser.add_argument("--mode", choices=["complete", "summary"], required=True)
|
||||
parser.add_argument("--question", choices=["disassembly"], default="disassembly")
|
||||
parser.add_argument("--symbol", choices=["execute_command", "execute_command_internal"], required=True)
|
||||
parser.add_argument("--budget", type=int, default=30000)
|
||||
args = parser.parse_args()
|
||||
oracle=json.loads(args.oracle.read_text())
|
||||
target=str(args.input)
|
||||
assert args.input.is_absolute() and args.oracle.is_absolute() and args.output.is_absolute()
|
||||
assert hashlib.sha256(args.input.read_bytes()).hexdigest()==oracle["artifact_sha256"]
|
||||
def delivered(pieces):
|
||||
assert isinstance(pieces,str) and pieces.startswith("Wall time: "), pieces[:300]
|
||||
_,sep,payload=pieces.partition("Output:\n");assert sep
|
||||
try: value=json.loads(payload)
|
||||
except json.JSONDecodeError: return {"_invalid_json_payload":payload}
|
||||
return value["result"] if "result" in value and "normalized_result" not in value else value
|
||||
def outputs_of(body):
|
||||
return {item["call_id"]:delivered(item["output"]) for item in body.get("input",[]) if item.get("type")=="function_call_output"}
|
||||
steps=8
|
||||
args.output.mkdir(mode=0o700)
|
||||
audit_home = args.output / "account"
|
||||
codex_state = audit_home / ".codex"
|
||||
mcp_home = audit_home / "mcp"
|
||||
workspace = audit_home / "workspace"
|
||||
for directory in [codex_state, mcp_home, workspace]:
|
||||
directory.mkdir(parents=True)
|
||||
captured = []
|
||||
known_outputs = {}
|
||||
state = {"normal_requests":0,"compactions":0}
|
||||
class Capture(http.server.BaseHTTPRequestHandler):
|
||||
def log_message(self, *_):
|
||||
pass
|
||||
def do_POST(self):
|
||||
size = int(self.headers["Content-Length"])
|
||||
assert 0 < size <= 16 * 1024 * 1024
|
||||
assert self.headers.get("Content-Encoding", "identity") == "identity"
|
||||
body = json.loads(self.rfile.read(size))
|
||||
captured.append({"path": self.path, "body": body})
|
||||
(args.output / "model-requests.json").write_text(json.dumps(captured) + "\n")
|
||||
http_index = len(captured)
|
||||
known_outputs.update(outputs_of(body))
|
||||
names = [t.get("name", "") for ns in body.get("tools", []) if ns.get("type") == "namespace" and ns.get("name") == "mcp__rea_audit" for t in ns.get("tools", [])]
|
||||
if not names:
|
||||
last=body.get('input',[])[-1]
|
||||
assert 'CONTEXT CHECKPOINT COMPACTION' in json.dumps(last)
|
||||
state['compactions']+=1;assert state['compactions']<=1
|
||||
evidence=known_outputs['call_local_6'];assert 'normalized_result' in evidence
|
||||
captured_result=evidence['normalized_result'];command_report=captured_result.get('summary',captured_result)
|
||||
checkpoint={'fixture':'deterministic protocol checkpoint; no AI inference or model-quality claim','task':'Full native Bash function disassembly has been delivered to this compaction request. Retain complete Evidence and finish owned cleanup.','path':target,'artifact_sha256_external_oracle':oracle['artifact_sha256'],'symbol':args.symbol,'session_id':known_outputs['call_local_1']['session_id'],'parent_evidence_id':captured_result.get('parent_evidence_id',evidence['evidence_id']),'subject':evidence['subject'],'completion_status':command_report['completion_status'],'output_truncated':command_report['output_truncated'],'native_console_utf16_units':command_report.get('available_counts',{}).get('console_utf16_units',len(command_report.get('console','').encode('utf-16-le'))//2),'full_body_location':'same-session retained Evidence; export next to '+str(args.output/'evidence.json'),'remaining':['close_gdb_session','export_evidence_bundle']}
|
||||
text=json.dumps(checkpoint)
|
||||
item={"type":"message","id":f"msg_checkpoint_{state['compactions']}","role":"assistant","status":"completed","content":[{"type":"output_text","text":text,"annotations":[]}]}
|
||||
captured[-1]['kind']='deterministic-compaction'
|
||||
else:
|
||||
state['normal_requests']+=1;index=state['normal_requests'];assert index<=steps+1
|
||||
captured[-1]['kind']='tool-planning-or-final'
|
||||
if index<=steps:
|
||||
if index==1:
|
||||
operation="start_gdb_session";arguments={}
|
||||
else:
|
||||
session=known_outputs["call_local_1"]["session_id"]
|
||||
if index<=6:
|
||||
command=['set auto-load off','set debuginfod enabled off',f'file {target}','show architecture',f'disassemble /r {args.symbol}'][index-2]
|
||||
operation="gdb_console";arguments={"session_id":session,"command":command}
|
||||
if index==6 and args.mode=="summary":arguments["detail"]="summary"
|
||||
elif index==7:
|
||||
operation="close_gdb_session";arguments={"session_id":session}
|
||||
else:
|
||||
operation="export_evidence_bundle";arguments={"path":str(args.output/'evidence.json')}
|
||||
selected=[name for name in names if name.endswith(operation)];assert len(selected)==1
|
||||
item={"type":"function_call","id":f"fc_local_{index}","call_id":f"call_local_{index}","name":selected[0],"namespace":"mcp__rea_audit","arguments":json.dumps(arguments),"status":"completed"}
|
||||
else:
|
||||
item={"type":"message","id":"msg_local_done","role":"assistant","status":"completed","content":[{"type":"output_text","text":"Scripted local GDB disassembly task complete.","annotations":[]}]}
|
||||
response_id=f"resp_local_{http_index}"
|
||||
base = {"id": response_id, "object": "response", "model": body.get("model"), "status": "completed", "output": [item]}
|
||||
events = [
|
||||
{"type": "response.created", "response": {**base, "status": "in_progress", "output": []}},
|
||||
{"type": "response.output_item.added", "output_index": 0, "item": {**item, "status": "in_progress"}},
|
||||
{"type": "response.output_item.done", "output_index": 0, "item": item},
|
||||
{"type": "response.completed", "response": base},
|
||||
]
|
||||
wire = "".join("event: " + e["type"] + "\ndata: " + json.dumps(e) + "\n\n" for e in events).encode()
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "text/event-stream")
|
||||
self.send_header("Content-Length", str(len(wire)))
|
||||
self.end_headers()
|
||||
self.wfile.write(wire)
|
||||
endpoint = http.server.HTTPServer(("127.0.0.1", 0), Capture)
|
||||
endpoint_thread = threading.Thread(target=endpoint.serve_forever, daemon=True)
|
||||
endpoint_thread.start()
|
||||
task_path = f"{args.node.parent}:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
|
||||
mcp_env = {"HOME": str(mcp_home), "PATH": task_path,
|
||||
"NODE_OPTIONS": "--max-old-space-size=2048", "UV_THREADPOOL_SIZE": "1",
|
||||
"REA_LOG_LEVEL": "silent", "REA_MCP_INPUT_SCHEMA_PROFILE": args.input_profile,
|
||||
"REA_MCP_OUTPUT_SCHEMA_PROFILE": args.output_profile}
|
||||
config = 'model_provider = "rea_capture"\nmodel = "rea-wire-probe"\n[analytics]\nenabled = false\n[mcp_servers.rea_audit]\n'
|
||||
config += f'command = {json.dumps(str(args.node))}\n'
|
||||
config += f'args = {json.dumps([str(args.runtime / "scripts/rea.mjs"), "mcp"])}\n'
|
||||
config += 'startup_timeout_sec = 60\ntool_timeout_sec = 60\n'
|
||||
config += '[mcp_servers.rea_audit.env]\n'
|
||||
config += ''.join(f'{key} = {json.dumps(value)}\n' for key, value in mcp_env.items())
|
||||
for name in ['start_gdb_session','gdb_console','close_gdb_session','export_evidence_bundle']:
|
||||
config += f'[mcp_servers.rea_audit.tools.{name}]\napproval_mode = "approve"\n'
|
||||
if args.budget: config += f'output_token_limit = {args.budget}\n'
|
||||
config += f'\n[model_providers.rea_capture]\nname = "Local scripted capture"\nbase_url = "http://127.0.0.1:{endpoint.server_port}/v1"\nwire_api = "responses"\nrequires_openai_auth = false\nsupports_websockets = false\nrequest_max_retries = 0\nstream_max_retries = 0\n'
|
||||
(codex_state / "config.toml").write_text(config)
|
||||
(args.output / "declared-config.toml").write_text(config)
|
||||
environment = {"HOME": str(audit_home), "CODEX_HOME": str(codex_state), "PATH": task_path,
|
||||
"NODE_OPTIONS": "--max-old-space-size=2048", "UV_THREADPOOL_SIZE": "1"}
|
||||
report = {"question": "Return architecture and complete byte-annotated disassembly for one selected native Bash function and export complete Evidence. No target execution.",
|
||||
"client": "Codex app-server 0.163.0", "model_turns_started": 1, "backend": "local deterministic SSE fixture; no AI model or paid service",
|
||||
"actual_model_usage": None, "input_profile": args.input_profile,
|
||||
"output_profile": args.output_profile, "calls": [], "observations": [],
|
||||
"unexpected_server_requests": [], "cleanup_failures": []}
|
||||
client_version = subprocess.check_output([str(args.codex), "--version"], env=environment, cwd=workspace, text=True, timeout=10).strip()
|
||||
assert client_version == "codex-cli 0.163.0", client_version
|
||||
transcript = []
|
||||
pending = {}
|
||||
completed_turn = asyncio.Event()
|
||||
next_id = 0
|
||||
stderr = (args.output / "server.log").open("wb")
|
||||
process = await asyncio.create_subprocess_exec(str(args.codex), "app-server", "--stdio",
|
||||
stdin=asyncio.subprocess.PIPE, stdout=asyncio.subprocess.PIPE, stderr=stderr,
|
||||
cwd=workspace, env=environment, limit=16 * 1024 * 1024)
|
||||
|
||||
async def send(message):
|
||||
encoded = (json.dumps(message, separators=(",", ":")) + "\n").encode()
|
||||
transcript.append({"direction": "client", "bytes": len(encoded), "message": message})
|
||||
process.stdin.write(encoded)
|
||||
await process.stdin.drain()
|
||||
|
||||
async def read_messages():
|
||||
try:
|
||||
while line := await process.stdout.readline():
|
||||
value = json.loads(line)
|
||||
transcript.append({"direction": "server", "bytes": len(line), "message": value})
|
||||
if value.get("method") == "turn/completed":
|
||||
report["turn"] = value["params"]["turn"]
|
||||
completed_turn.set()
|
||||
if "id" in value and "method" not in value:
|
||||
future = pending.pop(value["id"], None)
|
||||
if future and not future.done():
|
||||
future.set_result(value)
|
||||
elif "id" in value:
|
||||
report["unexpected_server_requests"].append(value["method"])
|
||||
await send({"id": value["id"], "error": {"code": -32601, "message": "Unexpected request in scripted metadata probe"}})
|
||||
raise RuntimeError("app-server stdout closed")
|
||||
except Exception as error:
|
||||
for future in pending.values():
|
||||
if not future.done():
|
||||
future.set_exception(error)
|
||||
|
||||
reader = asyncio.create_task(read_messages())
|
||||
|
||||
async def request(method, params):
|
||||
nonlocal next_id
|
||||
assert method in ["initialize", "thread/start", "mcpServerStatus/list", "mcpServer/tool/call", "turn/start"]
|
||||
next_id += 1
|
||||
message = {"id": next_id, "method": method, "params": params}
|
||||
future = asyncio.get_running_loop().create_future()
|
||||
pending[next_id] = future
|
||||
start = time.monotonic()
|
||||
await send(message)
|
||||
reply = await asyncio.wait_for(future, 90)
|
||||
report["calls"].append({"method": method, "elapsed_ms": (time.monotonic() - start) * 1000,
|
||||
"error": reply.get("error")})
|
||||
if "error" in reply:
|
||||
raise RuntimeError(json.dumps(reply["error"])[:4000])
|
||||
return reply["result"]
|
||||
|
||||
try:
|
||||
await request("initialize", {"clientInfo": {"name": "rea_host_probe", "title": "REA host compatibility probe", "version": "1"},
|
||||
"capabilities": {"experimentalApi": True, "explicitGatewayOauth": True}})
|
||||
await send({"method": "initialized"})
|
||||
created = await request("thread/start", {"cwd": str(workspace), "ephemeral": True,
|
||||
"approvalPolicy": "never", "sandbox": "read-only"})
|
||||
thread_id = created["thread"]["id"]
|
||||
assert not created["thread"].get("turns", [])
|
||||
inventory = await request("mcpServerStatus/list", {"threadId": thread_id,
|
||||
"serverName": "rea_audit", "detail": "toolsAndAuthOnly", "limit": 10})
|
||||
assert inventory.get("nextCursor") is None
|
||||
assert len(inventory["data"]) == 1
|
||||
server = inventory["data"][0]
|
||||
assert server["name"] == "rea_audit" and server.get("toolsError") is None
|
||||
tools = list(server["tools"].values())
|
||||
names = [tool["name"] for tool in tools]
|
||||
assert len(names) == len(set(names)) == args.expected_tools
|
||||
assert "gdb_console" in names
|
||||
output_schemas = sum(tool.get("outputSchema") is not None for tool in tools)
|
||||
assert output_schemas == (args.expected_tools if args.output_profile == "full" else 0)
|
||||
raw = json.dumps(inventory, separators=(",", ":"), ensure_ascii=False).encode()
|
||||
(args.output / "catalog.json").write_bytes(raw)
|
||||
report["catalog"] = {"tools": len(tools), "output_schemas": output_schemas,
|
||||
"bytes": len(raw), "sha256": hashlib.sha256(raw).hexdigest(),
|
||||
"server_info": server.get("serverInfo"), "runtime_status": server.get("runtimeStatus")}
|
||||
await request("turn/start", {"threadId":thread_id,"input":[{"type":"text","text":f"Read architecture and the complete byte-annotated disassembly of {args.symbol} in the selected Bash executable. Retain and export complete Evidence, then close the owned debugger. The local fixture scripts the required tools; no AI inference or target execution is used."}]})
|
||||
await asyncio.wait_for(completed_turn.wait(), 90)
|
||||
assert report["turn"]["status"] == "completed", report["turn"]
|
||||
assert state["normal_requests"]==steps+1
|
||||
assert len(captured)==steps+1+state["compactions"]
|
||||
outputs=known_outputs
|
||||
assert len(outputs)==steps
|
||||
bundle=json.loads((args.output/'evidence.json').read_text())
|
||||
evidence=next(e for e in bundle['records'] if e['operation']=='gdb_console' and e['parameters']['command']==f'disassemble /r {args.symbol}')
|
||||
assert evidence['subject'] is None
|
||||
assert evidence['normalized_result']['completion_status']=='complete' and not evidence['normalized_result']['output_truncated']
|
||||
delivered_result=outputs['call_local_6']
|
||||
delivery_complete=verify_command_delivery(delivered_result,evidence,args.mode,args.budget)
|
||||
if delivery_complete and args.mode=='summary':
|
||||
assert next(e for e in bundle['records'] if e['evidence_id']==delivered_result['evidence_id'])==delivered_result
|
||||
report['task_completed']=delivery_complete
|
||||
report['observations']={'parent_evidence_id':evidence['evidence_id'],'exported':True,'native_console_utf16_units':len(evidence['normalized_result']['console'].encode('utf-16-le'))//2,'client_delivered_full_evidence':delivery_complete,'selected_symbol':args.symbol,'tool_output_budget':args.budget or 'client default','subject':evidence['subject']}
|
||||
report["captured_requests"]=len(captured)
|
||||
report["compaction_requests"]=state["compactions"]
|
||||
report["compaction_backend"]="Deterministic checkpoint of observed identifiers and retained-body location; no AI inference or quality claim."
|
||||
report["mode"]=args.mode
|
||||
report["request_tool_counts"] = [len(c["body"].get("tools", [])) for c in captured]
|
||||
assert not report["unexpected_server_requests"]
|
||||
assert hashlib.sha256(args.input.read_bytes()).hexdigest()==oracle["artifact_sha256"]
|
||||
report["status"] = "passed" if report["task_completed"] else "expected-incomplete-task"
|
||||
except Exception:
|
||||
report["status"] = "failed"
|
||||
report["failure"] = traceback.format_exc()[-6000:]
|
||||
finally:
|
||||
process.stdin.close()
|
||||
try:
|
||||
await asyncio.wait_for(process.wait(), 15)
|
||||
except asyncio.TimeoutError:
|
||||
report["cleanup_failures"].append("app-server did not exit after stdin close; terminated owned process")
|
||||
process.terminate()
|
||||
try:
|
||||
await asyncio.wait_for(process.wait(), 5)
|
||||
except asyncio.TimeoutError:
|
||||
process.kill()
|
||||
await process.wait()
|
||||
await reader
|
||||
# app-server can acknowledge exit before its MCP child finishes EOF
|
||||
# cleanup. Wait on the allocated resource scope; never target a PID.
|
||||
try:
|
||||
groups = [line[3:] for line in Path("/proc/self/cgroup").read_text().splitlines()
|
||||
if line.startswith("0::")]
|
||||
assert len(groups) == 1
|
||||
group = Path("/sys/fs/cgroup") / groups[0].lstrip("/")
|
||||
assert re.fullmatch(r"rea-efficiency-[0-9a-f]{32}\.scope", group.name)
|
||||
observer = os.getppid()
|
||||
observer_args = Path(f"/proc/{observer}/cmdline").read_bytes().split(b"\0")
|
||||
assert any(Path(os.fsdecode(value)).name == "resource_payload.py" for value in observer_args if value)
|
||||
expected = {os.getpid(), observer}
|
||||
until = time.monotonic() + 10
|
||||
while True:
|
||||
actors = {int(value) for value in (group / "cgroup.procs").read_text().splitlines()}
|
||||
assert expected <= actors
|
||||
remaining = sorted(actors - expected)
|
||||
if not remaining or time.monotonic() >= until:
|
||||
break
|
||||
await asyncio.sleep(0.05)
|
||||
report["owned_scope_drain"] = {"remaining_processes": remaining}
|
||||
if remaining:
|
||||
report["cleanup_failures"].append("owned MCP child did not settle after app-server exit")
|
||||
except (OSError, ValueError, AssertionError) as error:
|
||||
report["owned_scope_drain"] = {"status": "unavailable", "reason": type(error).__name__}
|
||||
report["cleanup_failures"].append("allocated resource-scope cleanup could not be verified")
|
||||
endpoint.shutdown()
|
||||
endpoint.server_close()
|
||||
endpoint_thread.join(5)
|
||||
stderr.close()
|
||||
report["appserver_exit_code"] = process.returncode
|
||||
if process.returncode != 0:
|
||||
report["cleanup_failures"].append(f"app-server exit code: {process.returncode}")
|
||||
(args.output / "transcript.json").write_text(json.dumps(transcript) + "\n")
|
||||
(args.output / "report.json").write_text(json.dumps(report, indent=2) + "\n")
|
||||
print(json.dumps({key: report.get(key) for key in ["status", "failure", "catalog", "observations", "cleanup_failures", "appserver_exit_code"]}))
|
||||
return 0 if report["status"] in ["passed","expected-incomplete-task"] and not report["cleanup_failures"] else 1
|
||||
|
||||
|
||||
raise SystemExit(asyncio.run(main()))
|
||||
@@ -1,52 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
import assert from "node:assert/strict";
|
||||
import { createHash } from "node:crypto";
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { isDeepStrictEqual } from "node:util";
|
||||
import canonicalize from "canonicalize";
|
||||
|
||||
const [baselinePath, candidatePath, recordPath] = process.argv.slice(2);
|
||||
assert.ok(baselinePath && candidatePath && recordPath);
|
||||
const record = JSON.parse(await readFile(recordPath, "utf8"));
|
||||
const fingerprinted = record.trials.at(-1);
|
||||
const digest = (value) =>
|
||||
createHash("sha256").update(canonicalize(value)).digest("hex");
|
||||
const readSemantic = async (path, label) => {
|
||||
const evidence = JSON.parse(await readFile(path, "utf8"));
|
||||
const analysis = evidence.normalized_result;
|
||||
assert.equal(analysis.root_artifact_sha256, record.input.asar_sha256);
|
||||
const semantic = analysis.semantic_graph;
|
||||
assert.equal(semantic.application_graph_id, analysis.graph.graph_id);
|
||||
const rawSha = digest(semantic);
|
||||
assert.equal(
|
||||
rawSha,
|
||||
fingerprinted[label].profile.analysis_fingerprints.semantic_graph_sha256,
|
||||
);
|
||||
const { application_graph_id, graph_id, ...content } = semantic;
|
||||
return { content, rawSha, application_graph_id, graph_id };
|
||||
};
|
||||
|
||||
// Read sequentially; retain semantic content rather than complete parent graphs.
|
||||
const before = await readSemantic(baselinePath, "baseline");
|
||||
const after = await readSemantic(candidatePath, "candidate");
|
||||
assert.ok(
|
||||
isDeepStrictEqual(before.content, after.content),
|
||||
"Semantic content differs after excluding only the two parent commitments",
|
||||
);
|
||||
console.log(
|
||||
JSON.stringify({
|
||||
excluded_top_level_fields: ["application_graph_id", "graph_id"],
|
||||
content_deep_equal: true,
|
||||
baseline_raw_sha256: before.rawSha,
|
||||
candidate_raw_sha256: after.rawSha,
|
||||
baseline_content_sha256: digest(before.content),
|
||||
candidate_content_sha256: digest(after.content),
|
||||
baseline_application_graph_id: before.application_graph_id,
|
||||
candidate_application_graph_id: after.application_graph_id,
|
||||
baseline_graph_id: before.graph_id,
|
||||
candidate_graph_id: after.graph_id,
|
||||
compared_fields: Object.keys(before.content).sort(),
|
||||
serialization:
|
||||
"RFC 8785 canonical JSON; arrays and all remaining fields retained",
|
||||
}),
|
||||
);
|
||||
@@ -1,59 +0,0 @@
|
||||
# Workflow coverage reconciliation, 2026-10-21
|
||||
|
||||
The checked snapshot is main `264862bf2`, with **46 first-level `src/`
|
||||
directories and 212 production stdio MCP tools**. Directory counts and tool
|
||||
counts are different inventories: one application workflow can cross several
|
||||
directories, and one provider can expose many independently costly operations.
|
||||
This is a source/history review, not a new runtime qualification of all tools.
|
||||
|
||||
The search includes this experiment directory, owning contracts and source
|
||||
history, original feature PRs and related open/closed issues. Missing experiment
|
||||
files do not establish missing audits. In particular, the model and IL2CPP
|
||||
feature PRs contain substantial real-task evidence absent from this directory.
|
||||
Historical measurements retain their original revisions, platforms and limits.
|
||||
|
||||
| Directories / workflow | Evidence already present | Remaining coverage boundary |
|
||||
| ----------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `kuna`, `ghidra` | Earlier native entry work and [Ghidra projection](ghidra-evidence-projection.md) | Selected questions only; database reuse is a separate [research issue #3136](https://github.com/morluto/rea/issues/3136). |
|
||||
| `ida` | [Windows native correctness and completed-task costs](ida-native-windows.md), including real `whoami.exe` | Warm selected-task gains do not establish cold analysis, all string formats, all operations or other IDA platforms. |
|
||||
| `hopper`, `jeb` | [Hopper string records](hopper-string-records.md), [Linux runtime](hopper-linux-python-runtime.md), [real JEB CE](jeb-linux-native.md) | Preserve negative whole-task results; other editions, platforms and workflows are separate scopes. |
|
||||
| `rizin`, `cutter` | Existing command/session adapters and lifecycle fixes | No comparable dedicated real completed-task performance record located in the searched records. Actual engine/GUI availability must be established before selecting a task. |
|
||||
| `gdb`, `objdump` | [Real Bash command observations](command-observation-views.md) | Selected read-only disassembly is measured; other commands, large outputs and running-target observation are separate tasks. |
|
||||
| `adb`, `frida`, `inspector` | [Frida native status/history experiment](frida-status-views.md) | Frida status does not qualify arbitrary scripts, ADB/device transfers, browser Inspector events or live target freshness. |
|
||||
| `go`, `rust` | [Go build/dependency selection on real binaries](go-build-views.md); Rust cargo-auditable support [#2456](https://github.com/morluto/rea/pull/2456) | Go is already measured. No comparable dedicated Rust dependency-investigation cost record located in this review. |
|
||||
| `javascript`, `browser`, `electron` | [Agent fact workflow](javascript-facts-agent-history.json), lifecycle experiments and [#4415](https://github.com/morluto/rea/issues/4415) | Real Joplin preparation remains exploratory. Individual JS features and browser/runtime workflows need their own evidence. |
|
||||
| `android`, `apktool` | [Package summary](android-package-summary.md), [three-APK shared resource decoding](apktool-string-tables.md) | These do not qualify all JADX class/method/reference operations or physical-device workflows. #4777 is already fixed. |
|
||||
| `dotnet`, `il2cpp` | [Managed summaries/pages](managed-artifact-summary.md), IL2CPP metadata [#3807](https://github.com/morluto/rea/pull/3807) and native links/seeds [#5327](https://github.com/morluto/rea/pull/5327) | IL2CPP already has real Windows/Android row comparisons, task costs and Ghidra import. Unsupported versions/generics/platforms remain separate correctness and coverage work. |
|
||||
| `model` | GGUF/safetensors [#5603](https://github.com/morluto/rea/pull/5603): real models, independent readers and complete-task costs | Existing delivery is already audited. [#5774](https://github.com/morluto/rea/issues/5774) isolates omitted-string validation cost within that producer. |
|
||||
| `native`, `artifacts` | [Native inventory](native-inventory-views.md), [entry views](native-entry-views.md); real NIB output reductions [#1460](https://github.com/morluto/rea/pull/1460), [#1477](https://github.com/morluto/rea/pull/1477) | Inventory does not qualify Apple UI/resource, signatures, dyld or runtime traces. NIB byte reductions alone do not establish token, time, CPU or memory savings for a completed question. |
|
||||
| `firmware` | Region pilot plus real full OpenWrt extraction/publication [#4602](https://github.com/morluto/rea/pull/4602) | Extraction is already measured; noisy timings establish no general speedup. Other image layouts, extraction trees and publication filesystems remain unqualified. |
|
||||
| `flutter`, `sqlite`, `wasm`, `evm`, `pcap`, `signals` | [Dart AOT](flutter-aot-summary.md), [SQLite controls](sqlite-existing-controls.md), [WAT framing](wasm-string-framing.md), [EVM](evm-interface-summary.md), [PCAP](pcap-small-captures.md), [signals](signal-selection-workflows.md) | Extend a specific question. Preserve cases where existing controls suffice and small tasks regress; these records do not qualify each format or tool exhaustively. |
|
||||
| `application`, `domain`, `server`, `cli`, `contracts` | [Mixed retained investigations](retained-investigation-workflows.md), [immutable parent validation](immutable-parent-validation.md), [capture validation](retained-capture-validation.md) | Shared selected-query paths are already optimized. Reconstruction/environment composition and new fact families require their own end-to-end questions. |
|
||||
| `browser` / recorded captures | [SAZ selection](saz-selection-workflows.md) and [real qlog producer/retained reads](retained-capture-validation.md) | [#3133](https://github.com/morluto/rea/issues/3133) still includes actual model/task questions; producer measurements do not close those requirements. |
|
||||
| `filesystem`, `process`, `reference`, `providers`, `captures`, `windows`, `main`, `composition`, `config`, `evaluation` | Cross-workflow resource/cleanup checks, [process traces](process-trace-validation.md), [MCP profile](mcp-profile-identity.md) and Windows lifecycle checkpoints | Infrastructure coverage follows the exercised workflow. No blanket startup/resource claim; reference imports and `compare_source_to_bundle` need a dedicated completed source-tree question. |
|
||||
|
||||
Every one of the 46 directories appears above; repeated `browser` highlights a
|
||||
distinct workflow, not another directory. The table records located evidence,
|
||||
not proof that every historical PR was inspected. Open issue state by itself
|
||||
does not prove missing implementation or absent measurements.
|
||||
|
||||
## Three bounded follow-up questions
|
||||
|
||||
1. **Rust dependency identity:** inspect a public cargo-auditable binary, find
|
||||
exact dependency/version edges and retain complete source Evidence. Compare
|
||||
existing delivery first; independently decode its auditable section. Include
|
||||
a small binary and account for all discovery, queries and serialization.
|
||||
2. **Apple UI graph lookup:** on macOS, locate a specific control/outlet in a
|
||||
real Transmission/XLD NIB graph using existing summary/document pages.
|
||||
Compare equivalent complete questions and independently check archive
|
||||
records. Historical byte reductions alone are not the completion criterion.
|
||||
3. **Reference/source comparison:** import a pinned public source tree and
|
||||
answer one source-to-bundle identity question through existing workflows.
|
||||
Independently check file manifests, symlink/unknown semantics and selected
|
||||
matches; measure repeated traversal and retained representation before
|
||||
proposing an index or cache.
|
||||
|
||||
Each lane stops with a measured benefit, an evidenced no-change decision, or an
|
||||
explicit unavailable prerequisite. Rizin/Cutter and device/live workflows stay
|
||||
visible as additional gaps; no new execution, injection or mutation authority is
|
||||
implied by this review. No all-provider run or new API is justified by counts.
|
||||
@@ -1,149 +0,0 @@
|
||||
# Directory decompilation: capacity, resume and source Evidence
|
||||
|
||||
The directory-export feature requested in #6090 already shipped in #6170.
|
||||
This follow-up verifies the current public workflow and fixes a source-provenance
|
||||
gap found while combining it with existing Ghidra projects (#1048).
|
||||
|
||||
## Completed task and correctness
|
||||
|
||||
The analyst asks for all functions as local pseudocode files, without putting
|
||||
their bodies in agent context. The committed `decompile-directory.mjs` driver
|
||||
uses the production stdio MCP entrypoint and installed SDK. It discovers the
|
||||
catalog, exports an initial chunk, closes/reopens the provider, resumes all
|
||||
remaining work, and resumes again to verify settled files. It authenticates an
|
||||
explicitly exported Evidence bundle and checks the original target/project.
|
||||
|
||||
Every written file is checked against its index SHA-256 and address/profile
|
||||
header. Existing file hashes and modification times must survive both resumes.
|
||||
Comparison facts include every address, name, status, body range and pseudocode
|
||||
digest. An independent fresh-import investigation from #1048 additionally checks
|
||||
the complete inventory and two nonempty pseudocode observations. This establishes
|
||||
export preservation of provider observations; it does not establish discovery of
|
||||
every actual function or equivalence to original source.
|
||||
|
||||
| Real artifact | Provider inventory | Written | External | Failed |
|
||||
| ----------------------- | -----------------: | ------: | -------: | -----: |
|
||||
| `/usr/bin/ls` | 652 | 541 | 111 | 0 |
|
||||
| system `libcrypto.so.3` | 14,338 | 14,194 | 143 | 1 |
|
||||
|
||||
Target SHA-256 values are respectively
|
||||
`cb30d69b24245bf2ecdc9e7f53bbad19159999970b6d82c0c00c7d32d9e37aa4` and
|
||||
`aa01e614af634bc7c8b678c9250b5c4f613c61989bacf0a5d6aed6570e7dcabd`.
|
||||
These installed ELF x86-64 binaries were analyzed statically, never executed.
|
||||
The large run explicitly selected a five-second per-function deadline; function
|
||||
`0x387f00` exceeded it, remained a recorded failure, and was retried on resume.
|
||||
Later functions were still exported. No disappearance of that failure is claimed.
|
||||
The resulting large tree occupies about 107 MiB on this filesystem.
|
||||
|
||||
## Source-provenance correction
|
||||
|
||||
Before this change, `BinarySession` retained provider `sourceEvidence` only inside
|
||||
its snapshot-eligible branch. Ghidra procedure reads without a document selector
|
||||
are not snapshot eligible. A composed directory export therefore lost saved
|
||||
project reports even though the provider supplied them. The workflow also ignored
|
||||
source-record IDs, and the function-dossier adapter copied observation fields
|
||||
manually without those links.
|
||||
|
||||
The session now authenticates/retains source records independently of replay
|
||||
eligibility. Plain application helpers also return authenticated immutable source
|
||||
records with their outcome, including late cancellation. Both adapters explicitly
|
||||
commit them before delivering links; an adapter without a retention owner refuses
|
||||
the response. Source collection borrows ledger records without constructing or
|
||||
hashing an intermediate bundle. The workflow links distinct source IDs without copying their payloads.
|
||||
The function adapter uses the common observation-to-Evidence constructor,
|
||||
preserving sources through normal and late-cancelled completion. Queries with no
|
||||
source records avoid the extra recording path. Complete provider bodies remain
|
||||
in the exported function files; this does not add per-function payload retention.
|
||||
|
||||
A real test project contains historical seed and disabled-analyzer reports saved
|
||||
by REA's actual Ghidra pre-scripts. The unmodified runtime completed the export
|
||||
but failed the provenance oracle: its bundle had two workflow records and no
|
||||
historical source. The fixed runtime retains exactly one historical source and
|
||||
links the workflow to it. All 652 inventory entries and the sampled pseudocode
|
||||
still match fresh import; source project content stays unchanged. The report is
|
||||
historical, with remaining original settings/completeness explicitly unknown.
|
||||
|
||||
## Measured costs
|
||||
|
||||
Baseline runtime: `19447516a4bdbc9437d112ebd3f12a5fd062db83`.
|
||||
Initial fixed runtime: `f50763fd21527bdf3bab944633617b9df61dded4`, rebased on
|
||||
main `f02df45a4`. Both use Node 24.22.1, locked dependencies, actual Ghidra 12.1.4
|
||||
and JDK 21.0.12.1+1. The fixed driver later delegates transcript publication to
|
||||
the existing cleanup/receipt owner; the protocol and task are unchanged.
|
||||
|
||||
| Completed task | Wall s | Family CPU s | Peak family GiB | Full-wire token estimate |
|
||||
| --------------------------------- | ------: | -----------: | --------------: | -----------------------: |
|
||||
| ls, before | 51.411 | 78.984 | 0.943 | 57,082 |
|
||||
| ls, initial fix | 50.937 | 79.891 | 1.201 | 57,072 |
|
||||
| ls with historical reports, fixed | 52.827 | 80.236 | 1.097 | 57,207 |
|
||||
| libcrypto, baseline capacity run | 506.387 | 459.818 | 1.502 | 57,322 |
|
||||
|
||||
The plain ls pair has identical complete comparison facts. Runtime is similar;
|
||||
the higher sampled memory in the fixed run is reported rather than treated as an
|
||||
improvement. These sequential shared-host samples establish no generic speed or
|
||||
memory ratio. [PR #7250](https://github.com/morluto/rea/pull/7250) records the final
|
||||
large-run comparison and CI/review results.
|
||||
The failed historical baseline is excluded from completed-task comparisons.
|
||||
|
||||
The complete MCP replies stayed around 11–12 KiB, including repeated textual and
|
||||
structured representations, profile and caveats. The normalized result is only
|
||||
the manifest. For example, the large continuation costs about 3,159 estimated
|
||||
wire tokens; its complete task is 57,322 because discovery alone costs 47,049.
|
||||
Both numbers include actual protocol content, rather than only pseudocode bytes.
|
||||
|
||||
Estimates use `cl100k_base` on the actual JSON-RPC messages, including discovery,
|
||||
requests, responses and notifications, excluding only transport framing and the
|
||||
driver's local transcript wrapper. They are not actual model billing. Host
|
||||
projection, prompts/reasoning and later agent file reads remain unmeasured.
|
||||
The historical source increases response token cost slightly while preserving
|
||||
provenance. No token-reduction claim is made for this correctness fix.
|
||||
|
||||
CPU and memory cover the driver, REA and provider process family. Each run uses
|
||||
CPU affinity 0,1 and an 8-GiB sampled family-RSS guard; Java heaps are 1536M for ls
|
||||
and 3072M for crypto. RSS is sampled every 100 ms and summed: shared pages may be
|
||||
counted repeatedly and short peaks may be missed. CPU is waited child usage.
|
||||
Reported completed runs had no resource stop, OOM or cleanup failure.
|
||||
Project preparation is excluded and documented separately in
|
||||
[project reuse](ghidra-project-reuse.md). Provider starts are process-cold, with
|
||||
host filesystem caches unflushed; verified resume occurs in the second session.
|
||||
|
||||
The index and inventory still scale with function count. This 14,338-item run
|
||||
does not establish a larger capacity guarantee. Real Windows/macOS filesystems,
|
||||
other decompilers and native DWARF directory layout remain unverified. The current
|
||||
workflow reports unavailable DWARF layout explicitly instead of inventing it.
|
||||
|
||||
Actual CLI `decompile-all` calls additionally selected only `0x10a420`, reported
|
||||
651 excluded entries and `debug_layout_unavailable` for requested dwarf layout,
|
||||
then resumed in a new process. The existing file's hash and modification time
|
||||
stayed unchanged; the returned historical source ID stayed stable. This CLI check
|
||||
verifies the returned manifest/link and local function file, not offline retrieval
|
||||
of the separately retained source record after the CLI process exits.
|
||||
|
||||
## Reproduce and inspect
|
||||
|
||||
Prepare an owned project using the documented
|
||||
[existing-project route](../../docs/installation.md#reuse-an-existing-ghidra-project).
|
||||
Use fresh receipt/output paths; the driver refuses existing output directories.
|
||||
|
||||
```sh
|
||||
node experiments/analysis-efficiency/decompile-directory.mjs \
|
||||
--runtime /absolute/rea --output /absolute/new-receipt \
|
||||
--target /usr/bin/ls --project /absolute/ls-analysis.gpr --program ls \
|
||||
--ghidra "$GHIDRA_INSTALL_DIR" --java "$JAVA_HOME"
|
||||
```
|
||||
|
||||
Use `--chunk 512 --heap 3072M --decompile-seconds 5` for the crypto case.
|
||||
`--control /absolute/previous-receipt` compares all exported facts;
|
||||
`--golden /absolute/fresh-import-receipt` checks the #1048 driver's inventory and
|
||||
pseudocode oracle. `--history required` additionally requires a linked historical
|
||||
source. The driver allows failed functions to succeed on a later retry, verifies only
|
||||
previously written files remain unchanged, and reloads the final index before
|
||||
comparison. Complete local receipts include the transcript, resource report, index,
|
||||
function files, comparison facts and authenticated Evidence export. The resource
|
||||
report is supplied by the external process-family sampler, not the driver.
|
||||
|
||||
The current nine-file suite passed 144 tests. Against unmodified main source,
|
||||
the three source-retention/constructor suites reproduce seven failures and 70
|
||||
passes; the fixed full suite passes. This includes invalid source commitments,
|
||||
live and cursor-dependent reads, shared source deduplication, selected/null/provider
|
||||
subjects, and source links on late-cancelled completion.
|
||||
@@ -1,328 +0,0 @@
|
||||
/** Real-provider directory export, cross-session resume and retained provenance. */
|
||||
import assert from "node:assert/strict";
|
||||
import { createHash } from "node:crypto";
|
||||
import { execFile } from "node:child_process";
|
||||
import { createWriteStream } from "node:fs";
|
||||
import { mkdir, mkdtemp, readFile, stat, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join, resolve } from "node:path";
|
||||
import { finished } from "node:stream/promises";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { parseArgs, promisify } from "node:util";
|
||||
import { Client } from "@modelcontextprotocol/client";
|
||||
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
|
||||
import { finalizeRetainedQueryReceipt } from "./retained-query-finalization.mjs";
|
||||
|
||||
const { values } = parseArgs({
|
||||
options: Object.fromEntries(
|
||||
[
|
||||
"runtime",
|
||||
"output",
|
||||
"target",
|
||||
"project",
|
||||
"program",
|
||||
"ghidra",
|
||||
"java",
|
||||
"control",
|
||||
"heap",
|
||||
"chunk",
|
||||
"decompile-seconds",
|
||||
"history",
|
||||
"golden",
|
||||
].map((key) => [key, { type: "string" }]),
|
||||
),
|
||||
});
|
||||
for (const key of [
|
||||
"runtime",
|
||||
"output",
|
||||
"target",
|
||||
"project",
|
||||
"ghidra",
|
||||
"java",
|
||||
]) {
|
||||
assert.ok(values[key], `${key} is required`);
|
||||
values[key] = resolve(values[key]);
|
||||
}
|
||||
const chunk = Number(values.chunk ?? 32);
|
||||
assert.ok(Number.isSafeInteger(chunk) && chunk > 0);
|
||||
await mkdir(values.output);
|
||||
const digest = (bytes) => createHash("sha256").update(bytes).digest("hex");
|
||||
const report = {
|
||||
calls: [],
|
||||
runtime: values.runtime,
|
||||
target: values.target,
|
||||
project: values.project,
|
||||
};
|
||||
let client, transport, account, serverLog, serverLogFinished;
|
||||
// This workflow has a fixed number of public calls; pseudocode stays on disk.
|
||||
const transcript = [];
|
||||
let phase = "initialize";
|
||||
const started = performance.now();
|
||||
const tree = join(values.output, "tree");
|
||||
const readIndex = async () =>
|
||||
JSON.parse(await readFile(join(tree, "decompile-index.json"), "utf8"));
|
||||
const fileStates = async (index) => {
|
||||
const states = {};
|
||||
for (const row of index.functions.filter((row) => row.status === "written")) {
|
||||
const path = join(tree, row.relative_path);
|
||||
const metadata = await stat(path);
|
||||
states[row.address] = {
|
||||
sha256: digest(await readFile(path)),
|
||||
mtime_ms: metadata.mtimeMs,
|
||||
};
|
||||
assert.equal(states[row.address].sha256, row.sha256);
|
||||
}
|
||||
return states;
|
||||
};
|
||||
try {
|
||||
report.revision = (
|
||||
await promisify(execFile)("git", [
|
||||
"-C",
|
||||
values.runtime,
|
||||
"rev-parse",
|
||||
"HEAD",
|
||||
])
|
||||
).stdout.trim();
|
||||
report.target_sha256 = digest(await readFile(values.target));
|
||||
const { resolveGhidraProject } = await import(
|
||||
pathToFileURL(join(values.runtime, "dist/ghidra/GhidraProjectSnapshot.js"))
|
||||
.href
|
||||
);
|
||||
const { ArtifactResourceScope } = await import(
|
||||
pathToFileURL(
|
||||
join(values.runtime, "dist/artifacts/ArtifactResourceScope.js"),
|
||||
).href
|
||||
);
|
||||
const fingerprint = async () => {
|
||||
const resources = new ArtifactResourceScope();
|
||||
try {
|
||||
return await resolveGhidraProject(
|
||||
{
|
||||
path: values.project,
|
||||
...(values.program ? { programName: values.program } : {}),
|
||||
},
|
||||
{ path: values.target },
|
||||
resources,
|
||||
);
|
||||
} finally {
|
||||
await resources.close();
|
||||
}
|
||||
};
|
||||
const original = await fingerprint();
|
||||
report.project_sha256 = original.sha256;
|
||||
account = await mkdtemp(join(tmpdir(), "rea-directory-investigation-"));
|
||||
transport = new StdioClientTransport({
|
||||
command: process.execPath,
|
||||
args: [join(values.runtime, "scripts/rea.mjs"), "mcp"],
|
||||
cwd: values.runtime,
|
||||
maxBufferSize: 10 * 1024 * 1024,
|
||||
env: {
|
||||
PATH: process.env.PATH,
|
||||
HOME: account,
|
||||
USERPROFILE: account,
|
||||
JAVA_HOME: values.java,
|
||||
GHIDRA_INSTALL_DIR: values.ghidra,
|
||||
GHIDRA_HEADLESS_MAXMEM: values.heap ?? "1536M",
|
||||
JAVA_TOOL_OPTIONS: "-XX:ActiveProcessorCount=2",
|
||||
NODE_OPTIONS: process.env.NODE_OPTIONS ?? "--max-old-space-size=3072",
|
||||
REA_GHIDRA_PROJECT_PATH: values.project,
|
||||
...(values.program ? { REA_GHIDRA_PROGRAM_NAME: values.program } : {}),
|
||||
REA_GHIDRA_STARTUP_TIMEOUT_MS: "600000",
|
||||
...(values["decompile-seconds"]
|
||||
? { REA_GHIDRA_DECOMPILE_TIMEOUT_SECONDS: values["decompile-seconds"] }
|
||||
: {}),
|
||||
REA_MCP_TOOL_FAMILIES: "direct,enhanced,application,session",
|
||||
REA_MCP_INPUT_SCHEMA_PROFILE: "compact",
|
||||
REA_MCP_OUTPUT_SCHEMA_PROFILE: "compact",
|
||||
},
|
||||
stderr: "pipe",
|
||||
});
|
||||
const start = transport.start.bind(transport);
|
||||
transport.start = async () => {
|
||||
const receive = transport.onmessage;
|
||||
transport.onmessage = (message) => {
|
||||
transcript.push({ phase, direction: "response", message });
|
||||
receive?.(message);
|
||||
};
|
||||
await start();
|
||||
};
|
||||
const send = transport.send.bind(transport);
|
||||
transport.send = async (message) => {
|
||||
transcript.push({ phase, direction: "request", message });
|
||||
await send(message);
|
||||
};
|
||||
serverLog = createWriteStream(join(values.output, "server.log"));
|
||||
serverLogFinished = finished(serverLog);
|
||||
serverLogFinished.catch(() => {});
|
||||
transport.stderr.pipe(serverLog);
|
||||
client = new Client({ name: "rea-directory-investigation", version: "1" });
|
||||
await client.connect(transport);
|
||||
phase = "discovery";
|
||||
const catalog = await client.listTools({}, { timeout: 15000 });
|
||||
await writeFile(join(values.output, "catalog.json"), JSON.stringify(catalog));
|
||||
report.tools = catalog.tools.length;
|
||||
const definitions = new Map(catalog.tools.map((tool) => [tool.name, tool]));
|
||||
const call = async (name, args, label) => {
|
||||
phase = label;
|
||||
const at = performance.now();
|
||||
const result = await client.callTool(
|
||||
{ name, arguments: args },
|
||||
{ timeout: 900000, toolDefinition: definitions.get(name) },
|
||||
);
|
||||
report.calls.push({
|
||||
name,
|
||||
phase,
|
||||
elapsed_ms: performance.now() - at,
|
||||
response_bytes: Buffer.byteLength(JSON.stringify(result)),
|
||||
is_error: result.isError ?? false,
|
||||
});
|
||||
assert.notEqual(
|
||||
result.isError,
|
||||
true,
|
||||
`${name}: ${JSON.stringify(result).slice(0, 800)}`,
|
||||
);
|
||||
return result.structuredContent;
|
||||
};
|
||||
const open = () =>
|
||||
call("open_binary", { path: values.target, provider_id: "ghidra" }, "open");
|
||||
await open();
|
||||
const first = await call(
|
||||
"decompile_all",
|
||||
{ output_directory: tree, limit: chunk },
|
||||
"first-chunk",
|
||||
);
|
||||
report.first = first.normalized_result;
|
||||
const firstFiles = await fileStates(await readIndex());
|
||||
await call("close_binary", {}, "close-first-session");
|
||||
await open();
|
||||
const remaining = await call(
|
||||
"decompile_all",
|
||||
{ output_directory: tree, resume: true },
|
||||
"cross-session-resume",
|
||||
);
|
||||
report.resume = remaining.normalized_result;
|
||||
report.resume_evidence_links = remaining.evidence_links?.length ?? 0;
|
||||
let index = await readIndex();
|
||||
const allFiles = await fileStates(index);
|
||||
for (const [address, state] of Object.entries(firstFiles))
|
||||
assert.deepEqual(allFiles[address], state);
|
||||
report.first_files_unchanged = true;
|
||||
const settled = await call(
|
||||
"decompile_all",
|
||||
{ output_directory: tree, resume: true },
|
||||
"verified-resume",
|
||||
);
|
||||
report.verified_resume = settled.normalized_result;
|
||||
if (report.resume.counts.failed === 0)
|
||||
assert.equal(report.verified_resume.counts.written, 0);
|
||||
assert.equal(report.verified_resume.counts.remaining, 0);
|
||||
index = await readIndex();
|
||||
const settledFiles = await fileStates(index);
|
||||
for (const [address, state] of Object.entries(allFiles))
|
||||
assert.deepEqual(settledFiles[address], state);
|
||||
assert.equal(index.functions.length, report.resume.counts.selected);
|
||||
const facts = [];
|
||||
for (const row of index.functions) {
|
||||
const fact = {
|
||||
address: row.address,
|
||||
name: row.name,
|
||||
status: row.status,
|
||||
body: row.body,
|
||||
};
|
||||
if (row.status === "written") {
|
||||
const text = await readFile(join(tree, row.relative_path), "utf8");
|
||||
assert.ok(text.includes(`// address: ${row.address}\n`));
|
||||
assert.ok(
|
||||
text.includes(
|
||||
`// analysis_profile: ${index.analysis_profile_digest}\n`,
|
||||
),
|
||||
);
|
||||
fact.pseudocode_sha256 = digest(text.slice(text.indexOf("\n\n") + 2));
|
||||
}
|
||||
facts.push(fact);
|
||||
}
|
||||
report.function_count = facts.length;
|
||||
report.written = facts.filter((row) => row.status === "written").length;
|
||||
report.failed = facts.filter((row) => row.status === "failed").length;
|
||||
report.external = facts.filter((row) => row.status === "external").length;
|
||||
report.facts_sha256 = digest(JSON.stringify(facts));
|
||||
await writeFile(join(values.output, "facts.json"), JSON.stringify(facts));
|
||||
const evidencePath = join(values.output, "complete-evidence.json");
|
||||
await call(
|
||||
"export_evidence_bundle",
|
||||
{ path: evidencePath },
|
||||
"complete-evidence-export",
|
||||
);
|
||||
await call("close_binary", {}, "close");
|
||||
await client.close();
|
||||
await transport.close();
|
||||
const { parseEvidenceBundle } = await import(
|
||||
pathToFileURL(join(values.runtime, "dist/domain/evidenceBundle.js")).href
|
||||
);
|
||||
const bundle = parseEvidenceBundle(
|
||||
JSON.parse(await readFile(evidencePath, "utf8")),
|
||||
);
|
||||
report.evidence_records = bundle.records.length;
|
||||
const retained = bundle.records.find(
|
||||
(record) => record.evidence_id === remaining.evidence_id,
|
||||
);
|
||||
assert.ok(retained);
|
||||
report.retained_links = retained.evidence_links.length;
|
||||
const byId = new Set(bundle.records.map((record) => record.evidence_id));
|
||||
for (const id of retained.evidence_links) assert.ok(byId.has(id));
|
||||
report.all_source_links_resolve = true;
|
||||
if (values.history === "required") {
|
||||
const historical = bundle.records.filter(
|
||||
(record) =>
|
||||
record.operation === "health" &&
|
||||
record.parameters.facet === "historical-project-analysis-reports",
|
||||
);
|
||||
assert.equal(historical.length, 1);
|
||||
assert.equal(historical[0].normalized_result.origin, "caller-project");
|
||||
assert.ok(retained.evidence_links.includes(historical[0].evidence_id));
|
||||
report.historical_source_retained_and_linked = true;
|
||||
}
|
||||
if (values.golden) {
|
||||
const golden = JSON.parse(
|
||||
await readFile(join(values.golden, "facts.json"), "utf8"),
|
||||
);
|
||||
const labels = (rows) =>
|
||||
rows
|
||||
.map((row) => [row.address, row.name ?? row.value])
|
||||
.sort((a, b) => a[0].localeCompare(b[0], "en"));
|
||||
assert.deepEqual(labels(facts), labels(golden.procedures));
|
||||
for (const item of golden.pseudocode) {
|
||||
const fact = facts.find((row) => row.address === item.procedure);
|
||||
assert.equal(fact?.pseudocode_sha256, digest(item.result));
|
||||
}
|
||||
report.fresh_import_inventory_and_sampled_pseudocode_match = true;
|
||||
}
|
||||
if (values.control) {
|
||||
const control = JSON.parse(
|
||||
await readFile(join(values.control, "facts.json"), "utf8"),
|
||||
);
|
||||
assert.deepEqual(facts, control);
|
||||
report.control_facts_match = true;
|
||||
}
|
||||
assert.deepEqual(await fingerprint(), original);
|
||||
assert.equal(digest(await readFile(values.target)), report.target_sha256);
|
||||
report.source_unchanged = true;
|
||||
report.outcome = "passed";
|
||||
} catch (cause) {
|
||||
report.outcome = "failed";
|
||||
report.failure = String(cause);
|
||||
process.exitCode = 1;
|
||||
} finally {
|
||||
report.elapsed_ms = performance.now() - started;
|
||||
const published = await finalizeRetainedQueryReceipt({
|
||||
report,
|
||||
client,
|
||||
transport,
|
||||
account,
|
||||
serverLog,
|
||||
serverLogFinished,
|
||||
output: values.output,
|
||||
transcript,
|
||||
});
|
||||
if (!published) process.exitCode = 1;
|
||||
}
|
||||
@@ -1,115 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Check verifier result/exit semantics with controlled replies, not real apps."""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
import textwrap
|
||||
|
||||
|
||||
def write(root, name, source):
|
||||
path = root / name
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(textwrap.dedent(source))
|
||||
|
||||
|
||||
def check(arguments, root, case):
|
||||
shutil.copyfile(arguments.verifier, root / "probe.mjs")
|
||||
write(root, "package.json", '{"type":"module"}')
|
||||
write(root, "node_modules/playwright-core/package.json",
|
||||
'{"type":"module","exports":"./index.js"}')
|
||||
write(root, "node_modules/playwright-core/index.js", """
|
||||
let reads = 0;
|
||||
const page = {
|
||||
locator: () => ({ waitFor: async () => {} }),
|
||||
getByText: () => ({ waitFor: async () => {} }),
|
||||
async evaluate() {
|
||||
const after = reads++ > 0;
|
||||
return { title: 'controlled oracle', invalid_file_dialog: !after,
|
||||
unsaved_changes: after,
|
||||
diagram_svg_elements: after && process.env.REA_DRAWIO_ORACLE_CASE !== 'insertion-failure' ? 2 : 0,
|
||||
cells: 0 };
|
||||
},
|
||||
};
|
||||
export const _electron = { async launch() {
|
||||
return { process: () => ({exitCode: 0, signalCode: null}),
|
||||
firstWindow: async () => page, close: async () => {} };
|
||||
}};
|
||||
""")
|
||||
write(root, "dist/browser/PlaywrightElectronActiveProvider.js", """
|
||||
export class PlaywrightElectronActiveProvider {
|
||||
constructor(environment, launch) { this.launch = launch; }
|
||||
}
|
||||
""")
|
||||
write(root, "dist/browser/PlaywrightElectronActiveActions.js", """
|
||||
export const runWithExecutionLimits = operation => operation;
|
||||
""")
|
||||
write(root, "dist/application/javascript/ElectronActiveObservationService.js", """
|
||||
export async function captureElectronScenario(provider, input) {
|
||||
const app = await provider.launch({});
|
||||
await app.close();
|
||||
const name = process.env.REA_DRAWIO_ORACLE_CASE;
|
||||
const capture = { actions: input.actions.map(action => ({...action,
|
||||
status: name === 'action-failure' ? 'failed' : 'completed'})),
|
||||
coverage: {}, retention: {}, ipc: {events: [], observed: 0, dropped: 0} };
|
||||
if (name === 'no-capture') return {ok: false, error: {reason: 'startup_failed'}};
|
||||
if (name === 'partial') return {ok: false, error: {reason: 'cleanup_failed',
|
||||
partialObservation: {capture}}};
|
||||
return {ok: true, value: {evidence_id: 'ev_controlled',
|
||||
normalized_result: capture}};
|
||||
}
|
||||
""")
|
||||
result = subprocess.run(
|
||||
[arguments.node, str(root / "probe.mjs"), "--runtime-root", str(root),
|
||||
"--executable", arguments.node, "--application", str(root),
|
||||
"--profile-root", str(root), "--dismiss-error", "--selector", "controlled"],
|
||||
env={**os.environ, "REA_DRAWIO_ORACLE_CASE": case},
|
||||
capture_output=True, text=True, timeout=15,
|
||||
)
|
||||
replies = []
|
||||
for line in result.stdout.splitlines():
|
||||
try:
|
||||
replies.append(json.loads(line))
|
||||
except ValueError:
|
||||
pass
|
||||
receipt = next((value for value in replies if "workflow_status" in value), None)
|
||||
expected_success = case == "complete"
|
||||
expected_provider = case not in ("partial", "no-capture")
|
||||
expected_actions = case not in ("action-failure", "no-capture")
|
||||
expected_inserted = case != "insertion-failure"
|
||||
matched = (
|
||||
(result.returncode == 0) is expected_success
|
||||
and receipt is not None
|
||||
and receipt["provider_ok"] is expected_provider
|
||||
and receipt["cleanup_verified"] is expected_provider
|
||||
and receipt["actions_completed"] is expected_actions
|
||||
and receipt["shape_inserted"] is expected_inserted
|
||||
and (receipt["workflow_status"] == "verified") is expected_success
|
||||
)
|
||||
return {"case": case, "exit_code": result.returncode,
|
||||
"expected_success": expected_success, "summary_present": receipt is not None,
|
||||
"matched_expected": matched}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--node", required=True)
|
||||
parser.add_argument("--verifier", type=Path,
|
||||
default=Path(__file__).with_name("drawio-electron-lifecycle.mjs"))
|
||||
arguments = parser.parse_args()
|
||||
receipts = []
|
||||
with tempfile.TemporaryDirectory(prefix="rea-drawio-oracles-") as directory:
|
||||
for case in ("complete", "partial", "action-failure", "insertion-failure", "no-capture"):
|
||||
root = Path(directory) / case
|
||||
root.mkdir()
|
||||
receipts.append(check(arguments, root, case))
|
||||
print(json.dumps({"oracle_cases": receipts, "real_provider_verified": False}))
|
||||
assert all(receipt["matched_expected"] for receipt in receipts), receipts
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,191 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
// Real draw.io ASAR, renderer and declared actions; DOM reads are independent
|
||||
// verification probes. No fabricated IPC channels or replacement app callbacks.
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdtemp, writeFile } from "node:fs/promises";
|
||||
import { join, resolve } from "node:path";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { parseArgs } from "node:util";
|
||||
import { _electron } from "playwright-core";
|
||||
|
||||
const { values } = parseArgs({
|
||||
options: {
|
||||
"runtime-root": { type: "string" },
|
||||
executable: { type: "string" },
|
||||
application: { type: "string" },
|
||||
"profile-root": { type: "string" },
|
||||
selector: { type: "string" },
|
||||
"dismiss-error": { type: "boolean", default: false },
|
||||
},
|
||||
});
|
||||
for (const key of ["runtime-root", "executable", "application", "profile-root"])
|
||||
assert.ok(values[key], `Missing --${key}`);
|
||||
const runtime = resolve(values["runtime-root"]);
|
||||
const { PlaywrightElectronActiveProvider } = await import(
|
||||
pathToFileURL(
|
||||
join(runtime, "dist/browser/PlaywrightElectronActiveProvider.js"),
|
||||
)
|
||||
);
|
||||
const { captureElectronScenario } = await import(
|
||||
pathToFileURL(
|
||||
join(
|
||||
runtime,
|
||||
"dist/application/javascript/ElectronActiveObservationService.js",
|
||||
),
|
||||
)
|
||||
);
|
||||
const { runWithExecutionLimits } = await import(
|
||||
pathToFileURL(
|
||||
join(runtime, "dist/browser/PlaywrightElectronActiveActions.js"),
|
||||
)
|
||||
);
|
||||
const profile = await mkdtemp(
|
||||
join(resolve(values["profile-root"]), "drawio-ui-"),
|
||||
);
|
||||
const controller = new AbortController();
|
||||
const timer = setTimeout(
|
||||
() => controller.abort(new Error("Verifier deadline")),
|
||||
40_000,
|
||||
);
|
||||
let application;
|
||||
let child;
|
||||
let page;
|
||||
let before;
|
||||
let after;
|
||||
let checkedFinalState = false;
|
||||
const inspect = () =>
|
||||
page.evaluate(() => {
|
||||
const diagram = document.querySelector(".geDiagramContainer svg");
|
||||
return {
|
||||
title: document.title,
|
||||
invalid_file_dialog:
|
||||
document.body.innerText.includes("Invalid file data"),
|
||||
unsaved_changes: document.body.innerText.includes("Unsaved changes"),
|
||||
diagram_svg_elements:
|
||||
diagram?.querySelectorAll("rect,path,ellipse,text").length ?? null,
|
||||
cells: document.querySelectorAll(".geDiagramContainer [data-cell-id]")
|
||||
.length,
|
||||
};
|
||||
});
|
||||
const started = performance.now();
|
||||
try {
|
||||
const actions = values["dismiss-error"]
|
||||
? [
|
||||
{
|
||||
step_id: "dismiss-invalid-file",
|
||||
kind: "click",
|
||||
window_index: 0,
|
||||
selector: 'button:has-text("OK"):visible',
|
||||
},
|
||||
]
|
||||
: [];
|
||||
if (values.selector !== undefined)
|
||||
actions.push({
|
||||
step_id: "insert-shape",
|
||||
kind: "click",
|
||||
window_index: 0,
|
||||
selector: values.selector,
|
||||
});
|
||||
actions.push({
|
||||
step_id: "settle-renderer",
|
||||
kind: "wait",
|
||||
window_index: 0,
|
||||
duration_ms: 500,
|
||||
});
|
||||
const result = await captureElectronScenario(
|
||||
new PlaywrightElectronActiveProvider(process.env, async (options) => {
|
||||
application = await _electron.launch(options);
|
||||
child = application.process();
|
||||
page = await application.firstWindow({ timeout: 15_000 });
|
||||
await page
|
||||
.locator(".geDiagramContainer")
|
||||
.waitFor({ state: "visible", timeout: 15_000 });
|
||||
if (values["dismiss-error"])
|
||||
await page
|
||||
.getByText("Invalid file data", { exact: true })
|
||||
.waitFor({ state: "visible", timeout: 15_000 });
|
||||
before = await inspect();
|
||||
return new Proxy(application, {
|
||||
get(target, key) {
|
||||
if (key === "close")
|
||||
return async () => {
|
||||
if (!checkedFinalState) {
|
||||
checkedFinalState = true;
|
||||
after = await inspect();
|
||||
}
|
||||
return target.close();
|
||||
};
|
||||
const value = Reflect.get(target, key, target);
|
||||
return typeof value === "function" ? value.bind(target) : value;
|
||||
},
|
||||
});
|
||||
}),
|
||||
{
|
||||
executable_path: resolve(values.executable),
|
||||
application_path: resolve(values.application),
|
||||
args: ["--no-sandbox", "--disable-gpu", `--user-data-dir=${profile}`],
|
||||
actions,
|
||||
},
|
||||
{ signal: controller.signal },
|
||||
);
|
||||
const capture = result.ok
|
||||
? result.value.normalized_result
|
||||
: result.error.partialObservation?.capture;
|
||||
await writeFile(join(profile, "result.json"), JSON.stringify(result));
|
||||
const actionsCompleted =
|
||||
capture?.actions.every((action) => action.status === "completed") === true;
|
||||
const inserted =
|
||||
values.selector === undefined
|
||||
? null
|
||||
: after?.diagram_svg_elements > before?.diagram_svg_elements;
|
||||
console.log(
|
||||
JSON.stringify({
|
||||
profile,
|
||||
runtime,
|
||||
elapsed_ms: performance.now() - started,
|
||||
deadline_triggered: controller.signal.aborted,
|
||||
selected_shape: values.selector ?? null,
|
||||
before,
|
||||
after,
|
||||
actions_completed: actionsCompleted,
|
||||
shape_inserted: inserted,
|
||||
provider_ok: result.ok,
|
||||
cleanup_verified: result.ok,
|
||||
reason: result.ok ? null : result.error.reason,
|
||||
coverage: capture?.coverage,
|
||||
retention: capture?.retention,
|
||||
ipc: capture?.ipc,
|
||||
actions: capture?.actions,
|
||||
workflow_status:
|
||||
result.ok && actionsCompleted && inserted !== false
|
||||
? "verified"
|
||||
: "partial-or-failed",
|
||||
}),
|
||||
);
|
||||
assert.ok(actionsCompleted, "Declared real-app actions did not complete");
|
||||
assert.ok(before && after, "Real renderer state was not verified");
|
||||
assert.notEqual(inserted, false, "Click did not insert a diagram shape");
|
||||
if (values["dismiss-error"]) assert.equal(after.invalid_file_dialog, false);
|
||||
assert.equal(
|
||||
controller.signal.aborted,
|
||||
false,
|
||||
"Verifier deadline was reached",
|
||||
);
|
||||
assert.ok(result.ok, "Provider cleanup remains unverified");
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
if (
|
||||
application !== undefined &&
|
||||
child?.exitCode === null &&
|
||||
child?.signalCode === null
|
||||
)
|
||||
await runWithExecutionLimits(
|
||||
application.close(),
|
||||
undefined,
|
||||
performance.now() + 5_000,
|
||||
).catch(() => {
|
||||
process.exitCode = 1;
|
||||
});
|
||||
// The resource scope owns any unresolved descendants; no PID/path-only kill.
|
||||
if (child?.exitCode === null && child?.signalCode === null) process.exit(1);
|
||||
}
|
||||
@@ -1,154 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Check verifier exit semantics with controlled replies, not real providers/apps.
|
||||
|
||||
The JS case owns a short-lived Node child so the main process has exited while
|
||||
the provider reply still reports incomplete cleanup. The CLI cases substitute
|
||||
only Popen. These are negative-oracle regressions, not runtime compatibility or
|
||||
performance evidence; real Joplin trials remain separately required.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import contextlib
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
import textwrap
|
||||
from unittest.mock import patch
|
||||
|
||||
|
||||
def write(root, name, text):
|
||||
path = root / name
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(textwrap.dedent(text))
|
||||
|
||||
|
||||
def check_js(arguments, root, complete):
|
||||
shutil.copyfile(arguments.javascript, root / "probe.mjs")
|
||||
write(root, "package.json", '{"type":"module"}')
|
||||
write(root, "node_modules/playwright-core/package.json",
|
||||
'{"type":"module","exports":"./index.js"}')
|
||||
write(root, "node_modules/playwright-core/index.js", """
|
||||
import { spawn } from 'node:child_process';
|
||||
import { once } from 'node:events';
|
||||
export const _electron = { async launch() {
|
||||
const child = spawn(process.execPath, ['-e', 'setTimeout(() => {}, 30000)'],
|
||||
{ stdio: 'ignore' });
|
||||
await once(child, 'spawn');
|
||||
return { process: () => child, async close() {
|
||||
if (child.exitCode !== null || child.signalCode !== null) return;
|
||||
const exited = once(child, 'exit'); child.kill(); await exited;
|
||||
}};
|
||||
}};
|
||||
""")
|
||||
write(root, "dist/browser/PlaywrightElectronActiveProvider.js", """
|
||||
export class PlaywrightElectronActiveProvider {
|
||||
constructor(environment, launch) { this.launch = launch; }
|
||||
}
|
||||
""")
|
||||
write(root, "dist/browser/PlaywrightElectronActiveActions.js", """
|
||||
export const runWithExecutionLimits = operation => operation;
|
||||
""")
|
||||
write(root, "dist/application/javascript/ElectronActiveObservationService.js", f"""
|
||||
import {{ once }} from 'node:events';
|
||||
export async function captureElectronScenario(provider, input, options) {{
|
||||
const app = await provider.launch({{}});
|
||||
try {{
|
||||
void app.firstWindow();
|
||||
await once(options.signal, 'abort');
|
||||
}} finally {{ await app.close(); }}
|
||||
const error = Object.assign(new Error('controlled cancellation'), {{
|
||||
_tag: 'BrowserObservationError', reason: 'cancelled',
|
||||
userCategory: 'cancelled', cleanupIncomplete: {str(not complete).lower()},
|
||||
partialObservation: {{capture: {{application: {{}},
|
||||
actions: [{{status: 'cancelled'}}], coverage: {{}}, windows: [], limitations: []}}}}
|
||||
}});
|
||||
return {{ok: false, error}};
|
||||
}}
|
||||
""")
|
||||
result = subprocess.run(
|
||||
[arguments.node, str(root / "probe.mjs"), "--runtime-root", str(root),
|
||||
"--executable", arguments.node, "--application", str(root),
|
||||
"--profile-root", str(root), "--case", "window-wait"],
|
||||
capture_output=True, text=True, timeout=15,
|
||||
)
|
||||
receipts = [json.loads(line) for line in result.stdout.splitlines()]
|
||||
receipt = next(value for value in receipts if "cleanup_phase" in value)
|
||||
assert receipt["launched_process_exited"] is True
|
||||
assert receipt["cleanup_phase"] == ("verified" if complete else "unverified")
|
||||
assert (result.returncode == 0) is complete, (result.returncode, result.stdout, result.stderr)
|
||||
return {"verifier": "shared", "controlled_cleanup_complete": complete,
|
||||
"main_exited": True, "exit_code": result.returncode}
|
||||
|
||||
|
||||
def check_cli(arguments, root, code, category, cleanup):
|
||||
spec = importlib.util.spec_from_file_location("cli_cancellation_probe", arguments.python)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
error = {"code": code, "category": category, "details": {"partial_observation": {
|
||||
"kind": "electron-active-observation", "capture": {
|
||||
"application": {"cleanup": cleanup}, "actions": [{"status": "cancelled"}]}}}}
|
||||
|
||||
class ExitedCLI:
|
||||
def __init__(self, command, **options):
|
||||
request = json.loads(command[3])
|
||||
(Path(request["args"][-1]) / "log-main-process.txt").touch()
|
||||
self.returncode = None
|
||||
self.output = options["stdout"]
|
||||
|
||||
def poll(self):
|
||||
return self.returncode
|
||||
|
||||
def wait(self, timeout):
|
||||
raise subprocess.TimeoutExpired("controlled CLI", timeout)
|
||||
|
||||
def send_signal(self, signal):
|
||||
self.output.write(json.dumps(error).encode())
|
||||
self.output.flush()
|
||||
self.returncode = 143
|
||||
|
||||
supplied = argparse.Namespace(profile_root=root, node=arguments.node,
|
||||
runtime_root=root, executable=Path(arguments.node), application=root)
|
||||
output = io.StringIO()
|
||||
passed = True
|
||||
with patch.object(module.subprocess, "Popen", ExitedCLI), contextlib.redirect_stdout(output):
|
||||
try:
|
||||
module.run(supplied)
|
||||
except AssertionError:
|
||||
passed = False
|
||||
expected = code == "cancelled" and category == "cancelled" and cleanup == "terminated-owned-process"
|
||||
assert passed is expected, (code, category, cleanup, output.getvalue())
|
||||
receipt = json.loads(output.getvalue())
|
||||
assert receipt["exit_code"] == 143
|
||||
return {"verifier": "cli", "code": code, "category": category,
|
||||
"application_cleanup": cleanup, "accepted": passed}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--node", required=True)
|
||||
parser.add_argument("--javascript", type=Path, default=Path(__file__).with_name("electron-cancellation.mjs"))
|
||||
parser.add_argument("--python", type=Path, default=Path(__file__).with_name("joplin-cli-cancellation.py"))
|
||||
arguments = parser.parse_args()
|
||||
receipts = []
|
||||
with tempfile.TemporaryDirectory(prefix="rea-cancellation-oracles-") as directory:
|
||||
root = Path(directory)
|
||||
for complete in (False, True):
|
||||
case = root / str(complete)
|
||||
case.mkdir()
|
||||
receipts.append(check_js(arguments, case, complete))
|
||||
for code, category, cleanup in [
|
||||
("cleanup_incomplete", "cancelled", "unverified"),
|
||||
("cleanup_incomplete", "cancelled", "terminated-owned-process"),
|
||||
("cancelled", "execution_failure", "terminated-owned-process"),
|
||||
("cancelled", "cancelled", "terminated-owned-process"),
|
||||
]:
|
||||
receipts.append(check_cli(arguments, root, code, category, cleanup))
|
||||
print(json.dumps({"oracle_cases": receipts, "real_provider_verified": False}))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,273 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
// Real applications run unchanged. Fault cases hold a REA-side Playwright reply;
|
||||
// they do not establish that the application naturally exhibits that failure.
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdtemp, readFile } from "node:fs/promises";
|
||||
import { join, resolve } from "node:path";
|
||||
import { fileURLToPath, pathToFileURL } from "node:url";
|
||||
import { parseArgs } from "node:util";
|
||||
import { _electron } from "playwright-core";
|
||||
|
||||
const { values } = parseArgs({
|
||||
options: {
|
||||
executable: { type: "string" },
|
||||
application: { type: "string" },
|
||||
"profile-root": { type: "string" },
|
||||
"runtime-root": { type: "string" },
|
||||
case: { type: "string", default: "normal-wait" },
|
||||
},
|
||||
});
|
||||
const cases = ["normal-wait", "window-wait", "initial-state", "final-state"];
|
||||
assert.equal(
|
||||
process.platform,
|
||||
"linux",
|
||||
"This verifier checks process identity through Linux /proc",
|
||||
);
|
||||
assert.ok(cases.includes(values.case));
|
||||
assert.ok(values.executable && values.application && values["profile-root"]);
|
||||
const runtime = resolve(
|
||||
values["runtime-root"] ?? fileURLToPath(new URL("../..", import.meta.url)),
|
||||
);
|
||||
const { PlaywrightElectronActiveProvider } = await import(
|
||||
pathToFileURL(
|
||||
join(runtime, "dist/browser/PlaywrightElectronActiveProvider.js"),
|
||||
)
|
||||
);
|
||||
const { captureElectronScenario } = await import(
|
||||
pathToFileURL(
|
||||
join(
|
||||
runtime,
|
||||
"dist/application/javascript/ElectronActiveObservationService.js",
|
||||
),
|
||||
)
|
||||
);
|
||||
const { runWithExecutionLimits } = await import(
|
||||
pathToFileURL(
|
||||
join(runtime, "dist/browser/PlaywrightElectronActiveActions.js"),
|
||||
)
|
||||
);
|
||||
const profile = await mkdtemp(join(resolve(values["profile-root"]), "joplin-"));
|
||||
const controller = new AbortController();
|
||||
const reason = new Error(`real-application cancellation: ${values.case}`);
|
||||
const started = performance.now();
|
||||
let application;
|
||||
let child;
|
||||
let initialBirth;
|
||||
let evaluateCalls = 0;
|
||||
let injected = false;
|
||||
let cancelledAt;
|
||||
let cancelTimer;
|
||||
let watchdog;
|
||||
let rejectDeadline;
|
||||
const deadline = new Promise((_, reject) => {
|
||||
rejectDeadline = reject;
|
||||
});
|
||||
const birth = async (pid) => {
|
||||
try {
|
||||
const text = await readFile(`/proc/${pid}/stat`, "utf8");
|
||||
const fields = text.slice(text.lastIndexOf(")") + 2).split(" ");
|
||||
return fields[0] === "Z" ? null : fields[19];
|
||||
} catch (cause) {
|
||||
if (cause.code === "ENOENT") return null;
|
||||
throw cause;
|
||||
}
|
||||
};
|
||||
const armCancellation = () => {
|
||||
if (cancelTimer !== undefined) return;
|
||||
cancelTimer = setTimeout(
|
||||
() => {
|
||||
cancelledAt = performance.now();
|
||||
controller.abort(reason);
|
||||
watchdog = setTimeout(
|
||||
() =>
|
||||
rejectDeadline(
|
||||
new Error("Cancellation did not settle within 20 seconds"),
|
||||
),
|
||||
20_000,
|
||||
);
|
||||
},
|
||||
values.case === "normal-wait" ? 1_000 : 100,
|
||||
);
|
||||
};
|
||||
const launch = async (options) => {
|
||||
application = await _electron.launch(options);
|
||||
child = application.process();
|
||||
initialBirth = await birth(child.pid);
|
||||
assert.ok(initialBirth, "Launched process identity is unavailable");
|
||||
return new Proxy(application, {
|
||||
get(target, key) {
|
||||
if (key === "evaluate")
|
||||
return (...args) => {
|
||||
evaluateCalls += 1;
|
||||
if (
|
||||
(values.case === "initial-state" && evaluateCalls === 1) ||
|
||||
(values.case === "final-state" && evaluateCalls === 5)
|
||||
) {
|
||||
injected = true;
|
||||
armCancellation();
|
||||
return new Promise(() => {});
|
||||
}
|
||||
const result = target.evaluate(...args);
|
||||
if (values.case === "normal-wait" && evaluateCalls === 4)
|
||||
void result.then(armCancellation, () => undefined);
|
||||
return result;
|
||||
};
|
||||
if (key === "firstWindow" && values.case === "window-wait")
|
||||
return () => {
|
||||
injected = true;
|
||||
armCancellation();
|
||||
return new Promise(() => {});
|
||||
};
|
||||
const value = Reflect.get(target, key, target);
|
||||
return typeof value === "function" ? value.bind(target) : value;
|
||||
},
|
||||
});
|
||||
};
|
||||
let record;
|
||||
try {
|
||||
const input = {
|
||||
executable_path: resolve(values.executable),
|
||||
application_path: resolve(values.application),
|
||||
args: [
|
||||
"--no-sandbox",
|
||||
"--disable-gpu",
|
||||
`--user-data-dir=${profile}/chromium`,
|
||||
"--profile",
|
||||
profile,
|
||||
],
|
||||
actions: values.case.endsWith("state")
|
||||
? []
|
||||
: [{ step_id: "settle", kind: "wait", duration_ms: 60_000 }],
|
||||
};
|
||||
const result = await Promise.race([
|
||||
captureElectronScenario(
|
||||
new PlaywrightElectronActiveProvider(process.env, launch),
|
||||
input,
|
||||
{ signal: controller.signal },
|
||||
),
|
||||
deadline,
|
||||
]);
|
||||
assert.equal(result.ok, false, "Expected a typed cancellation");
|
||||
const error = result.error;
|
||||
const partial = error.partialObservation?.capture;
|
||||
const causes = [];
|
||||
const seen = new Set();
|
||||
for (
|
||||
let cause = error;
|
||||
cause instanceof Error && !seen.has(cause);
|
||||
cause = cause.cause
|
||||
) {
|
||||
seen.add(cause);
|
||||
causes.push({
|
||||
tag: cause._tag ?? null,
|
||||
reason: cause.reason ?? null,
|
||||
message: cause.message,
|
||||
diagnostics: cause.diagnostics ?? null,
|
||||
});
|
||||
}
|
||||
record = {
|
||||
case: values.case,
|
||||
fault_injected: injected,
|
||||
input,
|
||||
profile,
|
||||
evaluate_calls: evaluateCalls,
|
||||
elapsed_ms: performance.now() - started,
|
||||
cancellation_to_result_ms:
|
||||
cancelledAt === undefined ? null : performance.now() - cancelledAt,
|
||||
tag: error._tag,
|
||||
reason: error.reason,
|
||||
user_category: error.userCategory,
|
||||
diagnostics: error.diagnostics ?? null,
|
||||
message: error.message,
|
||||
user_message: error.userMessage ?? null,
|
||||
cleanup: error.cleanup ?? null,
|
||||
cause: error.cause instanceof Error ? error.cause.message : null,
|
||||
causes,
|
||||
cleanup_incomplete: error.cleanupIncomplete,
|
||||
partial:
|
||||
partial === undefined
|
||||
? null
|
||||
: {
|
||||
application: partial.application,
|
||||
actions: partial.actions,
|
||||
coverage: partial.coverage,
|
||||
window_count: partial.windows.length,
|
||||
limitations: partial.limitations,
|
||||
},
|
||||
launched_process_exited:
|
||||
child !== undefined && (await birth(child.pid)) !== initialBirth,
|
||||
};
|
||||
assert.equal(controller.signal.aborted, true);
|
||||
assert.equal(error.userCategory, "cancelled");
|
||||
if (values.case !== "normal-wait") assert.equal(injected, true);
|
||||
if (values.case.endsWith("wait"))
|
||||
assert.ok(partial?.actions.some((action) => action.status === "cancelled"));
|
||||
if (values.case === "initial-state") assert.equal(partial, undefined);
|
||||
if (values.case === "final-state") {
|
||||
assert.ok(
|
||||
partial?.limitations.some((text) =>
|
||||
text.includes("initial pre-action snapshot"),
|
||||
),
|
||||
);
|
||||
assert.ok(
|
||||
causes.some(
|
||||
(cause) =>
|
||||
cause.diagnostics?.final_state_read === "cancellation_grace_expired",
|
||||
),
|
||||
);
|
||||
}
|
||||
// Cleanup is a separate oracle. A known Linux lineage limitation must remain
|
||||
// visible, rather than turning cancellation coverage into a cleanup claim.
|
||||
record.cancellation_phase = "verified";
|
||||
record.cleanup_phase =
|
||||
record.launched_process_exited && !error.cleanupIncomplete
|
||||
? "verified"
|
||||
: "unverified";
|
||||
record.workflow_status =
|
||||
record.cleanup_phase === "verified" ? "verified" : "partial";
|
||||
console.log(JSON.stringify(record));
|
||||
assert.equal(
|
||||
record.cleanup_phase,
|
||||
"verified",
|
||||
"Provider cleanup remains unverified; cancellation alone is a partial result",
|
||||
);
|
||||
} catch (cause) {
|
||||
console.log(
|
||||
JSON.stringify({
|
||||
case: values.case,
|
||||
fault_injected: injected,
|
||||
evaluate_calls: evaluateCalls,
|
||||
elapsed_ms: performance.now() - started,
|
||||
cancellation_to_failure_ms:
|
||||
cancelledAt === undefined ? null : performance.now() - cancelledAt,
|
||||
completed: false,
|
||||
observed: record ?? null,
|
||||
verifier_error: cause instanceof Error ? cause.message : String(cause),
|
||||
}),
|
||||
);
|
||||
throw cause;
|
||||
} finally {
|
||||
clearTimeout(cancelTimer);
|
||||
clearTimeout(watchdog);
|
||||
if (
|
||||
application !== undefined &&
|
||||
child?.exitCode === null &&
|
||||
child?.signalCode === null
|
||||
)
|
||||
await runWithExecutionLimits(
|
||||
application.close(),
|
||||
undefined,
|
||||
performance.now() + 5_000,
|
||||
).catch((cause) => {
|
||||
console.error(
|
||||
JSON.stringify({
|
||||
verifier_cleanup: "unverified",
|
||||
message: cause.message,
|
||||
}),
|
||||
);
|
||||
process.exitCode = 1;
|
||||
});
|
||||
// The external resource scope owns any leftovers; do not leave a failed
|
||||
// verifier awaiting an application whose ownership the provider cannot prove.
|
||||
if (child?.exitCode === null && child?.signalCode === null) process.exit(1);
|
||||
}
|
||||
@@ -1,89 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Serial native/hooked producer controls; not real-application evidence."""
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
|
||||
|
||||
def sha256(path):
|
||||
with path.open("rb") as stream:
|
||||
return hashlib.file_digest(stream, "sha256").hexdigest()
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--hook-root", type=Path, required=True)
|
||||
parser.add_argument("--node", type=Path, action="append", required=True)
|
||||
parser.add_argument("--electron", type=Path)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
arguments = parser.parse_args()
|
||||
arguments.output.mkdir(parents=True, exist_ok=True)
|
||||
packet = Path(tempfile.mkdtemp(prefix="native-once-", dir=arguments.output))
|
||||
stage = packet / "staged"
|
||||
module = stage / "node_modules/electron"
|
||||
module.mkdir(parents=True)
|
||||
fixture = module / "index.js"
|
||||
fixture.write_text(
|
||||
'const { EventEmitter } = require("node:events");\n'
|
||||
"module.exports = { app: new EventEmitter(), ipcMain: new EventEmitter() };\n"
|
||||
)
|
||||
sources = [
|
||||
arguments.hook_root / "scripts/electron-active-hook.cjs",
|
||||
arguments.hook_root / "scripts/electron-active-hook-boundaries.cjs",
|
||||
Path(__file__).with_name("electron-once-runtime.mjs"),
|
||||
]
|
||||
pins = {path.name: sha256(path) for path in sources}
|
||||
pins["electron-fixture"] = sha256(fixture)
|
||||
for source in sources:
|
||||
shutil.copyfile(source, stage / source.name)
|
||||
engines = [(engine, False) for engine in arguments.node]
|
||||
if arguments.electron is not None:
|
||||
engines.append((arguments.electron, True))
|
||||
results = []
|
||||
for index, (engine, run_as_node) in enumerate(engines):
|
||||
selected = engine.absolute()
|
||||
environment = dict(os.environ)
|
||||
if run_as_node:
|
||||
environment["ELECTRON_RUN_AS_NODE"] = "1"
|
||||
execution = subprocess.run(
|
||||
[str(selected), str(stage / sources[-1].name), str(stage / sources[0].name)],
|
||||
env=environment,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
timeout=15,
|
||||
check=False,
|
||||
)
|
||||
(packet / f"engine-{index}.stdout").write_bytes(execution.stdout)
|
||||
(packet / f"engine-{index}.stderr").write_bytes(execution.stderr)
|
||||
receipt = {
|
||||
"selected_binary": str(engine),
|
||||
"absolute_binary": str(selected),
|
||||
"binary_sha256": sha256(selected),
|
||||
"electron_run_as_node": run_as_node,
|
||||
"exit_code": execution.returncode,
|
||||
"stdout_sha256": hashlib.sha256(execution.stdout).hexdigest(),
|
||||
"stderr_sha256": hashlib.sha256(execution.stderr).hexdigest(),
|
||||
}
|
||||
if execution.returncode == 0:
|
||||
receipt["result"] = json.loads(execution.stdout)
|
||||
results.append(receipt)
|
||||
report = {"hook_root": str(arguments.hook_root), "source_sha256": pins,
|
||||
"engines": results, "scope": __doc__}
|
||||
(packet / "report.json").write_text(json.dumps(report, indent=2) + "\n")
|
||||
if execution.returncode != 0:
|
||||
raise RuntimeError(f"Engine failed; preserved receipt at {packet}")
|
||||
print(json.dumps({"packet": str(packet), "engines": [
|
||||
{"node": row["result"]["node"], "electron": row["result"]["electron"],
|
||||
"cases": row["result"]["cases"],
|
||||
"disagreements": row["result"]["disagreements"]} for row in results
|
||||
]}))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,111 +0,0 @@
|
||||
"""Verify completed BEAM selection/export tasks and count a declared token projection."""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import statistics
|
||||
|
||||
import tiktoken
|
||||
|
||||
|
||||
def compact(value):
|
||||
return json.dumps(value, ensure_ascii=False, separators=(',', ':'), allow_nan=False)
|
||||
|
||||
|
||||
def digest(path):
|
||||
return hashlib.sha256(path.read_bytes()).hexdigest()
|
||||
|
||||
|
||||
def inspect(path, encoding):
|
||||
report = json.loads((path / 'report.json').read_text())
|
||||
catalog = json.loads((path / 'catalog.json').read_text())
|
||||
transcript = json.loads((path / 'transcript.json').read_text())
|
||||
scope = next(json.loads(line) for line in reversed(path.with_suffix('.log').read_text().splitlines())
|
||||
if line.startswith('{"resource_scope":'))
|
||||
observed = scope.get('payload_resource_observation') or {}
|
||||
events = observed.get('resource_events', {})
|
||||
assert report['status'] == 'passed' and report['runtime_identity_sha256']
|
||||
assert report['complete_source_export_verified'] is True
|
||||
assert not report['cleanup_failures'] and not report['receipt_failures']
|
||||
assert scope['abort'] is None and scope['payload_resource_observation_verified'] is True
|
||||
assert observed['payload_exit_code'] == 0 and observed['payload_cleanup_verified'] is True
|
||||
assert all(events.get('memory.events', {}).get(key) == 0 for key in ['high', 'max', 'oom', 'oom_kill'])
|
||||
assert events.get('pids.events', {}).get('max') == 0
|
||||
assert len(catalog['tools']) == report['catalog_tools'] == 221
|
||||
assert transcript[0]['name'] == 'inspect_erlang_beam'
|
||||
assert transcript[-1]['name'] == 'export_evidence_bundle'
|
||||
assert all(event['name'] == 'inspect_analysis_view' for event in transcript[1:-1])
|
||||
assert all('reply' in event and not event['reply'].get('isError', False) for event in transcript)
|
||||
count = lambda text: len(encoding.encode(text, disallowed_special=()))
|
||||
directory = [{key: tool[key] for key in ['name', 'description', 'inputSchema']} for tool in catalog['tools']]
|
||||
request = transcript[0]['input']
|
||||
question = report['question']
|
||||
assert ('detail' in request) == (report['lane'] == 'selected')
|
||||
arguments = sum(count(compact({'name': event['name'], 'arguments': event['input']})) for event in transcript)
|
||||
replies = sum(count(block['text']) for event in transcript for block in event['reply']['content'] if block['type'] == 'text')
|
||||
inputs, prompt = count(compact(directory)), count(question)
|
||||
instructions = count(report['server_instructions'])
|
||||
return dict(case=path.name, lane=report['lane'], artifact=report['artifact'],
|
||||
answer_count=report['answer_count'], answer_sha256=report['answer_sha256'],
|
||||
parent_evidence_id=report['parent_evidence_id'], source_report_sha256=report['source_report_sha256'],
|
||||
runtime_identity_sha256=report['runtime_identity_sha256'], catalog_sha256=digest(path / 'catalog.json'),
|
||||
calls=report['calls'], phases=report['phases'], wall_seconds=scope['elapsed_seconds'],
|
||||
cpu_seconds=observed['cpu_stat']['usage_usec'] / 1e6,
|
||||
kernel_family_peak_bytes=observed['memory_peak_bytes'],
|
||||
events_at_payload_exit=events, payload_cleanup_verified=True,
|
||||
final_whole_scope_events=scope.get('resource_events_final'),
|
||||
token_proxy=dict(question=prompt, server_instructions=instructions, input_catalog_once=inputs, arguments=arguments,
|
||||
text_replies=replies, combined=prompt + instructions + inputs + arguments + replies,
|
||||
complete_transport_catalog=count(compact(catalog))),
|
||||
receipt_sha256={file.name: digest(file) for file in [path / 'report.json', path / 'catalog.json',
|
||||
path / 'transcript.json', path / 'evidence.json', path.with_suffix('.log')]})
|
||||
|
||||
|
||||
def summarize(rows):
|
||||
result = {}
|
||||
for artifact in sorted({row['artifact']['sha256'] for row in rows}):
|
||||
cases = [row for row in rows if row['artifact']['sha256'] == artifact]
|
||||
for field in ['answer_count', 'answer_sha256', 'parent_evidence_id', 'source_report_sha256']:
|
||||
assert len({row[field] for row in cases}) == 1, field
|
||||
result[artifact] = {}
|
||||
for lane in ['complete', 'selected']:
|
||||
samples = [row for row in cases if row['lane'] == lane]
|
||||
assert len(samples) == 3
|
||||
for field in ['runtime_identity_sha256', 'catalog_sha256']:
|
||||
assert len({row[field] for row in samples}) == 1, field
|
||||
metrics = {key: [row[key] for row in samples] for key in ['wall_seconds', 'cpu_seconds', 'kernel_family_peak_bytes']}
|
||||
metrics['projected_tokens'] = [row['token_proxy']['combined'] for row in samples]
|
||||
metrics['mcp_reply_bytes'] = [sum(call['reply_bytes'] for call in row['calls']) for row in samples]
|
||||
result[artifact][lane] = {key: dict(median=statistics.median(values), minimum=min(values), maximum=max(values))
|
||||
for key, values in metrics.items()}
|
||||
return result
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--cohort', type=Path, required=True)
|
||||
parser.add_argument('--output', type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
manifest = json.loads((args.cohort / 'manifest.json').read_text())
|
||||
index = json.loads((args.cohort / 'index.json').read_text())
|
||||
assert len(index) == 18 and all(case['exit'] == 0 for case in index)
|
||||
for file, expected in manifest['driver_files'].items():
|
||||
assert digest(Path(file)) == expected, file
|
||||
identities = {}
|
||||
for lane, path in manifest['runtime_identities'].items():
|
||||
identity = json.loads(Path(path).read_text())
|
||||
assert identity['source_clean'] is True
|
||||
identities[lane] = dict(sha256=digest(Path(path)), source_revision=identity['source_revision'])
|
||||
encoding = tiktoken.get_encoding('cl100k_base')
|
||||
rows = [inspect(args.cohort / case['name'], encoding) for case in index]
|
||||
assert all(row['runtime_identity_sha256'] == identities[row['lane']]['sha256'] for row in rows)
|
||||
receipt = dict(tiktoken=tiktoken.__version__, encoding=encoding.name, actual_model_usage=None,
|
||||
cohort=manifest, commands=index, runtime_identities=identities,
|
||||
projection='Scripted fresh SDK tasks, not model runs. One fixed question, server instructions and complete input directory (name/description/inputSchema), every argument and MCP text reply including export. Raw catalog including output schemas reported separately. No framing, host injection, history replay, inference quality or billable usage measured.',
|
||||
resources='CPU quota100%, payload affinity0-1, initial observer15. Early stop3GiB, hard4GiB, swap0. Fresh HOME/client/server; Node compile cache disabled, OS caches unflushed. Whole process-family CPU and kernel memory peak include file cache, runtime identity checks, hashing, assertions, SDK/server serialization and cleanup. Oracle generation and tokenization outside timed tasks. Payload-exit counters verified; final whole-scope counters unavailable.',
|
||||
summary=summarize(rows), recordings=rows)
|
||||
args.output.write_text(json.dumps(receipt, ensure_ascii=False, indent=2) + '\n')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -1,108 +0,0 @@
|
||||
"""Independent struct oracle for the fixed public OTP 27 benchmark artifacts.
|
||||
|
||||
Reads bytes only. It deliberately refuses compact atom layouts; format-wide
|
||||
coverage belongs to the production reader and its separate regressions.
|
||||
"""
|
||||
from pathlib import Path
|
||||
import getopt
|
||||
import hashlib
|
||||
import json
|
||||
import struct
|
||||
import sys
|
||||
|
||||
|
||||
def decode(path):
|
||||
data = path.read_bytes()
|
||||
assert len(data) <= 64 * 1024 * 1024
|
||||
u32 = lambda at: struct.unpack_from('>I', data, at)[0]
|
||||
assert data[:4] == b'FOR1' and data[8:12] == b'BEAM'
|
||||
assert u32(4) + 8 == len(data)
|
||||
chunks, selected, at = [], {}, 12
|
||||
while at < len(data):
|
||||
name = data[at:at + 4].decode('ascii')
|
||||
size = u32(at + 4)
|
||||
end = at + 8 + size
|
||||
padded = (end + 3) & ~3
|
||||
assert end <= padded <= len(data)
|
||||
chunks.append(dict(index=len(chunks), name=name,
|
||||
location=dict(offset=at, bytes=padded - at),
|
||||
payload=dict(location=dict(offset=at + 8, bytes=size),
|
||||
bytes=size, sha256=hashlib.sha256(data[at + 8:end]).hexdigest())))
|
||||
selected[name] = (at + 8, end)
|
||||
at = padded
|
||||
assert at == len(data)
|
||||
atom_name = 'AtU8' if 'AtU8' in selected else 'Atom'
|
||||
start, end = selected[atom_name]
|
||||
n = u32(start)
|
||||
assert n < 65536, 'Fixed OTP 27 oracle does not decode compact atom tables'
|
||||
at, atoms = start + 4, []
|
||||
for i in range(n):
|
||||
size = data[at]
|
||||
at += 1
|
||||
name = data[at:at + size].decode('utf-8' if atom_name == 'AtU8' else 'latin1')
|
||||
atoms.append(dict(index=i + 1, name=name))
|
||||
at += size
|
||||
assert at == end
|
||||
|
||||
def atom(i):
|
||||
return atoms[i - 1]['name'] if 0 < i <= len(atoms) else None
|
||||
|
||||
def table(name, keys):
|
||||
if name not in selected:
|
||||
return None
|
||||
start, end = selected[name]
|
||||
if name == 'FunT' and start == end:
|
||||
return None
|
||||
n, width = u32(start), len(keys)
|
||||
assert start + 4 + n * 4 * width == end
|
||||
rows = []
|
||||
for i in range(n):
|
||||
values = struct.unpack_from('>' + 'I' * width, data, start + 4 + i * width * 4)
|
||||
row = dict(zip(keys, values))
|
||||
if 'module_index' in row:
|
||||
row['module'] = atom(row['module_index'])
|
||||
row['function'] = atom(row['function_index'])
|
||||
rows.append(row)
|
||||
return rows
|
||||
|
||||
exports = table('ExpT', ['function_index', 'arity', 'label'])
|
||||
imports = table('ImpT', ['module_index', 'function_index', 'arity'])
|
||||
lambdas = table('FunT', ['function_index', 'arity', 'label', 'index', 'free_vars', 'old_unique'])
|
||||
code = None
|
||||
if 'Code' in selected:
|
||||
start, end = selected['Code']
|
||||
size = u32(start)
|
||||
assert size >= 16 and start + 4 + size <= end
|
||||
code = dict(zip(['instruction_set', 'opcode_max', 'labels', 'functions'],
|
||||
struct.unpack_from('>IIII', data, start + 4)))
|
||||
code.update(extra_header_bytes=size - 16,
|
||||
instructions=dict(offset=start + 4 + size, bytes=end - start - 4 - size))
|
||||
return dict(artifact=dict(path=str(path), sha256=hashlib.sha256(data).hexdigest(), bytes=len(data)),
|
||||
form=dict(location=dict(offset=0, bytes=len(data)), size=len(data) - 8),
|
||||
chunks=chunks, atoms=atoms, atom_encoding='utf8' if atom_name == 'AtU8' else 'latin1',
|
||||
exports=exports, imports=imports, lambdas=lambdas, code=code,
|
||||
module=atoms[0]['name'],
|
||||
coverage=dict(status='complete', examined_chunks=len(chunks), atoms=len(atoms)))
|
||||
|
||||
|
||||
def main():
|
||||
options, remaining = getopt.getopt(sys.argv[1:], '', ['input=', 'output=', 'name='])
|
||||
assert not remaining
|
||||
inputs = [v for k, v in options if k == '--input']
|
||||
outputs = [v for k, v in options if k == '--output']
|
||||
names = [v for k, v in options if k == '--name']
|
||||
assert len(inputs) == len(outputs) == 1 and names
|
||||
path = Path(inputs[0])
|
||||
assert path.is_absolute()
|
||||
expected = decode(path)
|
||||
answer = [dict(source_index=i, **row) for i, row in enumerate(expected['exports'] or [])
|
||||
if row['function'] in names]
|
||||
result = dict(oracle='Independent Python struct decoding of fixed OTP27 bytes; no BEAM execution. '
|
||||
'Every table, chunk range/hash and Code header checked. Other encodings/malformed inputs unqualified.',
|
||||
source='https://www.erlang.org/doc/apps/stdlib/beam_lib.html',
|
||||
expected=expected, names=names, answer=answer)
|
||||
Path(outputs[0]).write_text(json.dumps(result, ensure_ascii=False, indent=2) + '\n')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -1,534 +0,0 @@
|
||||
{
|
||||
"method": "3alternating freshsessions perlane; SDK cachesentirecatalog thenusesdefaultvalidator withouttoolDefinition override. Diagnostic client startup control, separate from18exactnamedtasks. Samequestion/fullsourceexports/oracle; allpayloadexit/cleanupverified.",
|
||||
"summary": {
|
||||
"716d78394b7c3da983e36aac7aadd6673e5041677ba7de20413cc2704d958d55": {
|
||||
"complete": {
|
||||
"wall_seconds": {
|
||||
"median": 19.764,
|
||||
"minimum": 17.257,
|
||||
"maximum": 21.577
|
||||
},
|
||||
"cpu_seconds": {
|
||||
"median": 16.276557,
|
||||
"minimum": 15.287268,
|
||||
"maximum": 16.561897
|
||||
},
|
||||
"kernel_family_peak_bytes": {
|
||||
"median": 1073950720,
|
||||
"minimum": 1041948672,
|
||||
"maximum": 1078980608
|
||||
},
|
||||
"projected_tokens": {
|
||||
"median": 76907,
|
||||
"minimum": 76907,
|
||||
"maximum": 76907
|
||||
},
|
||||
"mcp_reply_bytes": {
|
||||
"median": 147003,
|
||||
"minimum": 147003,
|
||||
"maximum": 147003
|
||||
}
|
||||
},
|
||||
"selected": {
|
||||
"wall_seconds": {
|
||||
"median": 19.375,
|
||||
"minimum": 18.052,
|
||||
"maximum": 22.397
|
||||
},
|
||||
"cpu_seconds": {
|
||||
"median": 16.867381,
|
||||
"minimum": 16.139353,
|
||||
"maximum": 17.069666
|
||||
},
|
||||
"kernel_family_peak_bytes": {
|
||||
"median": 1105092608,
|
||||
"minimum": 1059520512,
|
||||
"maximum": 1117036544
|
||||
},
|
||||
"projected_tokens": {
|
||||
"median": 58090,
|
||||
"minimum": 58090,
|
||||
"maximum": 58090
|
||||
},
|
||||
"mcp_reply_bytes": {
|
||||
"median": 7469,
|
||||
"minimum": 7469,
|
||||
"maximum": 7469
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"recordings": [
|
||||
{
|
||||
"case": "erl_lint-1-complete",
|
||||
"lane": "complete",
|
||||
"artifact": {
|
||||
"path": "/tmp/rea-perf-static-tool-audit-20261024/real-beam/erl_lint.beam",
|
||||
"sha256": "716d78394b7c3da983e36aac7aadd6673e5041677ba7de20413cc2704d958d55",
|
||||
"bytes": 233968
|
||||
},
|
||||
"answer_count": 4,
|
||||
"answer_sha256": "34ce3091dbea9ac8cd20831cf6443c7a1de6e02b753246cafa3e0e22b83907c9",
|
||||
"parent_evidence_id": "ev_df4bfa8b8f0391ad6291542fdccf9062ae3bb8dbabf01c2b166ca6f3493051fc",
|
||||
"source_report_sha256": "db64350ae25bff9e77f5e167f4a4ac095ed872e09c15de5ce3b2728c078a16d2",
|
||||
"runtime_identity_sha256": "1b4b40330a2e21d31c06a07d10937cb6a3b37cd0bcbf3d129380341d486b916e",
|
||||
"catalog_sha256": "f4a683209df5bed1ed25ae42c690fdd803348742132e88f77ba2e4e8333ca562",
|
||||
"calls": [
|
||||
{
|
||||
"name": "inspect_erlang_beam",
|
||||
"input": {
|
||||
"path": "/tmp/rea-perf-static-tool-audit-20261024/real-beam/erl_lint.beam"
|
||||
},
|
||||
"phase": "inspect",
|
||||
"elapsed_ms": 18092.161941,
|
||||
"reply_bytes": 146627,
|
||||
"is_error": false
|
||||
},
|
||||
{
|
||||
"name": "export_evidence_bundle",
|
||||
"input": {
|
||||
"path": "/tmp/rea-perf-static-tool-audit-20261024/sdk-cache-cohort/erl_lint-1-complete/evidence.json"
|
||||
},
|
||||
"phase": "export",
|
||||
"elapsed_ms": 52.40052199999991,
|
||||
"reply_bytes": 376,
|
||||
"is_error": false
|
||||
}
|
||||
],
|
||||
"phases": {
|
||||
"discovery_ms": 1018.838796
|
||||
},
|
||||
"wall_seconds": 21.577,
|
||||
"cpu_seconds": 16.561897,
|
||||
"kernel_family_peak_bytes": 1073950720,
|
||||
"events_at_payload_exit": {
|
||||
"memory.events": {
|
||||
"low": 0,
|
||||
"high": 0,
|
||||
"max": 0,
|
||||
"oom": 0,
|
||||
"oom_kill": 0,
|
||||
"oom_group_kill": 0
|
||||
},
|
||||
"pids.events": {
|
||||
"max": 0
|
||||
}
|
||||
},
|
||||
"payload_cleanup_verified": true,
|
||||
"final_whole_scope_events": null,
|
||||
"token_proxy": {
|
||||
"question": 32,
|
||||
"server_instructions": 513,
|
||||
"input_catalog_once": 55409,
|
||||
"arguments": 78,
|
||||
"text_replies": 20875,
|
||||
"combined": 76907,
|
||||
"complete_transport_catalog": 829075
|
||||
},
|
||||
"receipt_sha256": {
|
||||
"report.json": "3717e681ac50a70db8bd7303ac18d87bd2028d4321104a793421695a82579290",
|
||||
"catalog.json": "f4a683209df5bed1ed25ae42c690fdd803348742132e88f77ba2e4e8333ca562",
|
||||
"transcript.json": "c718ac6e9b6c63533bc28d18c90c73a03abc1697516ae216a444b6dd55827255",
|
||||
"evidence.json": "ca66367cd1bb325d7d4e47f958c86234d01aaf749b68778ac5212cac246aacb3",
|
||||
"erl_lint-1-complete.log": "a90d1345cd09e90865c8a75f732abe2561c2299d0084718e91205f2cb904f38d"
|
||||
}
|
||||
},
|
||||
{
|
||||
"case": "erl_lint-1-selected",
|
||||
"lane": "selected",
|
||||
"artifact": {
|
||||
"path": "/tmp/rea-perf-static-tool-audit-20261024/real-beam/erl_lint.beam",
|
||||
"sha256": "716d78394b7c3da983e36aac7aadd6673e5041677ba7de20413cc2704d958d55",
|
||||
"bytes": 233968
|
||||
},
|
||||
"answer_count": 4,
|
||||
"answer_sha256": "34ce3091dbea9ac8cd20831cf6443c7a1de6e02b753246cafa3e0e22b83907c9",
|
||||
"parent_evidence_id": "ev_df4bfa8b8f0391ad6291542fdccf9062ae3bb8dbabf01c2b166ca6f3493051fc",
|
||||
"source_report_sha256": "db64350ae25bff9e77f5e167f4a4ac095ed872e09c15de5ce3b2728c078a16d2",
|
||||
"runtime_identity_sha256": "a74d203a5a12d7eb82aaa1e0fda6595b9647af2255f318bf116953777052460f",
|
||||
"catalog_sha256": "9c09aa2944f10c806df78d1cfa2e1daeddc0a487c432701cb67fa65e14213b0c",
|
||||
"calls": [
|
||||
{
|
||||
"name": "inspect_erlang_beam",
|
||||
"input": {
|
||||
"path": "/tmp/rea-perf-static-tool-audit-20261024/real-beam/erl_lint.beam",
|
||||
"detail": "summary",
|
||||
"view": {
|
||||
"kind": "beam",
|
||||
"collection": "exports",
|
||||
"names": ["module", "exprs", "expr"],
|
||||
"fields": ["function_index", "function", "arity", "label"],
|
||||
"offset": 0,
|
||||
"limit": 64
|
||||
}
|
||||
},
|
||||
"phase": "inspect",
|
||||
"elapsed_ms": 18394.333454,
|
||||
"reply_bytes": 7093,
|
||||
"is_error": false
|
||||
},
|
||||
{
|
||||
"name": "export_evidence_bundle",
|
||||
"input": {
|
||||
"path": "/tmp/rea-perf-static-tool-audit-20261024/sdk-cache-cohort/erl_lint-1-selected/evidence.json"
|
||||
},
|
||||
"phase": "export",
|
||||
"elapsed_ms": 58.59494199999972,
|
||||
"reply_bytes": 376,
|
||||
"is_error": false
|
||||
}
|
||||
],
|
||||
"phases": {
|
||||
"discovery_ms": 1056.9951270000001
|
||||
},
|
||||
"wall_seconds": 22.397,
|
||||
"cpu_seconds": 17.069666,
|
||||
"kernel_family_peak_bytes": 1117036544,
|
||||
"events_at_payload_exit": {
|
||||
"memory.events": {
|
||||
"low": 0,
|
||||
"high": 0,
|
||||
"max": 0,
|
||||
"oom": 0,
|
||||
"oom_kill": 0,
|
||||
"oom_group_kill": 0
|
||||
},
|
||||
"pids.events": {
|
||||
"max": 0
|
||||
}
|
||||
},
|
||||
"payload_cleanup_verified": true,
|
||||
"final_whole_scope_events": null,
|
||||
"token_proxy": {
|
||||
"question": 32,
|
||||
"server_instructions": 513,
|
||||
"input_catalog_once": 56359,
|
||||
"arguments": 120,
|
||||
"text_replies": 1066,
|
||||
"combined": 58090,
|
||||
"complete_transport_catalog": 834792
|
||||
},
|
||||
"receipt_sha256": {
|
||||
"report.json": "c3c17495f33ee4ea9daabd7a353f698fa0a804451c4bbdbe9eb1adf905ab483a",
|
||||
"catalog.json": "9c09aa2944f10c806df78d1cfa2e1daeddc0a487c432701cb67fa65e14213b0c",
|
||||
"transcript.json": "fd8c589a6e0dbcd0783ef5a526b51db7a3c208cd4067986ed99804bda06c4498",
|
||||
"evidence.json": "2373bc2e57e40aac881f3ef91fa77187b098dd61d6cf44d37a4b8fe56d022b40",
|
||||
"erl_lint-1-selected.log": "fdd55def718ffa7c972b3cb9b6be1b7f66a7fa5fb1c371cc492051ffb0d08f62"
|
||||
}
|
||||
},
|
||||
{
|
||||
"case": "erl_lint-2-selected",
|
||||
"lane": "selected",
|
||||
"artifact": {
|
||||
"path": "/tmp/rea-perf-static-tool-audit-20261024/real-beam/erl_lint.beam",
|
||||
"sha256": "716d78394b7c3da983e36aac7aadd6673e5041677ba7de20413cc2704d958d55",
|
||||
"bytes": 233968
|
||||
},
|
||||
"answer_count": 4,
|
||||
"answer_sha256": "34ce3091dbea9ac8cd20831cf6443c7a1de6e02b753246cafa3e0e22b83907c9",
|
||||
"parent_evidence_id": "ev_df4bfa8b8f0391ad6291542fdccf9062ae3bb8dbabf01c2b166ca6f3493051fc",
|
||||
"source_report_sha256": "db64350ae25bff9e77f5e167f4a4ac095ed872e09c15de5ce3b2728c078a16d2",
|
||||
"runtime_identity_sha256": "a74d203a5a12d7eb82aaa1e0fda6595b9647af2255f318bf116953777052460f",
|
||||
"catalog_sha256": "9c09aa2944f10c806df78d1cfa2e1daeddc0a487c432701cb67fa65e14213b0c",
|
||||
"calls": [
|
||||
{
|
||||
"name": "inspect_erlang_beam",
|
||||
"input": {
|
||||
"path": "/tmp/rea-perf-static-tool-audit-20261024/real-beam/erl_lint.beam",
|
||||
"detail": "summary",
|
||||
"view": {
|
||||
"kind": "beam",
|
||||
"collection": "exports",
|
||||
"names": ["module", "exprs", "expr"],
|
||||
"fields": ["function_index", "function", "arity", "label"],
|
||||
"offset": 0,
|
||||
"limit": 64
|
||||
}
|
||||
},
|
||||
"phase": "inspect",
|
||||
"elapsed_ms": 15135.096698,
|
||||
"reply_bytes": 7093,
|
||||
"is_error": false
|
||||
},
|
||||
{
|
||||
"name": "export_evidence_bundle",
|
||||
"input": {
|
||||
"path": "/tmp/rea-perf-static-tool-audit-20261024/sdk-cache-cohort/erl_lint-2-selected/evidence.json"
|
||||
},
|
||||
"phase": "export",
|
||||
"elapsed_ms": 37.63045200000124,
|
||||
"reply_bytes": 376,
|
||||
"is_error": false
|
||||
}
|
||||
],
|
||||
"phases": {
|
||||
"discovery_ms": 1335.57118
|
||||
},
|
||||
"wall_seconds": 19.375,
|
||||
"cpu_seconds": 16.139353,
|
||||
"kernel_family_peak_bytes": 1059520512,
|
||||
"events_at_payload_exit": {
|
||||
"memory.events": {
|
||||
"low": 0,
|
||||
"high": 0,
|
||||
"max": 0,
|
||||
"oom": 0,
|
||||
"oom_kill": 0,
|
||||
"oom_group_kill": 0
|
||||
},
|
||||
"pids.events": {
|
||||
"max": 0
|
||||
}
|
||||
},
|
||||
"payload_cleanup_verified": true,
|
||||
"final_whole_scope_events": null,
|
||||
"token_proxy": {
|
||||
"question": 32,
|
||||
"server_instructions": 513,
|
||||
"input_catalog_once": 56359,
|
||||
"arguments": 120,
|
||||
"text_replies": 1066,
|
||||
"combined": 58090,
|
||||
"complete_transport_catalog": 834792
|
||||
},
|
||||
"receipt_sha256": {
|
||||
"report.json": "d25a255b106e1bb232237474915e2900e313ba354aded0d27dad81ea74cfe0c6",
|
||||
"catalog.json": "9c09aa2944f10c806df78d1cfa2e1daeddc0a487c432701cb67fa65e14213b0c",
|
||||
"transcript.json": "2c0f92c25f97bb825b99133b84d792c7aac315b4c100986d75cb01939af6ce77",
|
||||
"evidence.json": "2373bc2e57e40aac881f3ef91fa77187b098dd61d6cf44d37a4b8fe56d022b40",
|
||||
"erl_lint-2-selected.log": "011553e956860fb3ce9d4e54ca7d4a2baf88a260786d82f60b1d0b70974fdbb7"
|
||||
}
|
||||
},
|
||||
{
|
||||
"case": "erl_lint-2-complete",
|
||||
"lane": "complete",
|
||||
"artifact": {
|
||||
"path": "/tmp/rea-perf-static-tool-audit-20261024/real-beam/erl_lint.beam",
|
||||
"sha256": "716d78394b7c3da983e36aac7aadd6673e5041677ba7de20413cc2704d958d55",
|
||||
"bytes": 233968
|
||||
},
|
||||
"answer_count": 4,
|
||||
"answer_sha256": "34ce3091dbea9ac8cd20831cf6443c7a1de6e02b753246cafa3e0e22b83907c9",
|
||||
"parent_evidence_id": "ev_df4bfa8b8f0391ad6291542fdccf9062ae3bb8dbabf01c2b166ca6f3493051fc",
|
||||
"source_report_sha256": "db64350ae25bff9e77f5e167f4a4ac095ed872e09c15de5ce3b2728c078a16d2",
|
||||
"runtime_identity_sha256": "1b4b40330a2e21d31c06a07d10937cb6a3b37cd0bcbf3d129380341d486b916e",
|
||||
"catalog_sha256": "f4a683209df5bed1ed25ae42c690fdd803348742132e88f77ba2e4e8333ca562",
|
||||
"calls": [
|
||||
{
|
||||
"name": "inspect_erlang_beam",
|
||||
"input": {
|
||||
"path": "/tmp/rea-perf-static-tool-audit-20261024/real-beam/erl_lint.beam"
|
||||
},
|
||||
"phase": "inspect",
|
||||
"elapsed_ms": 15133.678018999999,
|
||||
"reply_bytes": 146627,
|
||||
"is_error": false
|
||||
},
|
||||
{
|
||||
"name": "export_evidence_bundle",
|
||||
"input": {
|
||||
"path": "/tmp/rea-perf-static-tool-audit-20261024/sdk-cache-cohort/erl_lint-2-complete/evidence.json"
|
||||
},
|
||||
"phase": "export",
|
||||
"elapsed_ms": 44.53432299999986,
|
||||
"reply_bytes": 376,
|
||||
"is_error": false
|
||||
}
|
||||
],
|
||||
"phases": {
|
||||
"discovery_ms": 1000.9708420000002
|
||||
},
|
||||
"wall_seconds": 19.764,
|
||||
"cpu_seconds": 15.287268,
|
||||
"kernel_family_peak_bytes": 1041948672,
|
||||
"events_at_payload_exit": {
|
||||
"memory.events": {
|
||||
"low": 0,
|
||||
"high": 0,
|
||||
"max": 0,
|
||||
"oom": 0,
|
||||
"oom_kill": 0,
|
||||
"oom_group_kill": 0
|
||||
},
|
||||
"pids.events": {
|
||||
"max": 0
|
||||
}
|
||||
},
|
||||
"payload_cleanup_verified": true,
|
||||
"final_whole_scope_events": null,
|
||||
"token_proxy": {
|
||||
"question": 32,
|
||||
"server_instructions": 513,
|
||||
"input_catalog_once": 55409,
|
||||
"arguments": 78,
|
||||
"text_replies": 20875,
|
||||
"combined": 76907,
|
||||
"complete_transport_catalog": 829075
|
||||
},
|
||||
"receipt_sha256": {
|
||||
"report.json": "61aa7d7c78db145af4399b22d57aa39c2008c479530b28a49d401a6d564e46f7",
|
||||
"catalog.json": "f4a683209df5bed1ed25ae42c690fdd803348742132e88f77ba2e4e8333ca562",
|
||||
"transcript.json": "3729bcc5547ab06f80b578c24bbb7937a438b0c190ee6680d0d7868cb5885159",
|
||||
"evidence.json": "ca66367cd1bb325d7d4e47f958c86234d01aaf749b68778ac5212cac246aacb3",
|
||||
"erl_lint-2-complete.log": "7bb8d637df3cb7ba542345c9d93ea09fe967d11c90cf5383083db7cd2c5fafb8"
|
||||
}
|
||||
},
|
||||
{
|
||||
"case": "erl_lint-3-complete",
|
||||
"lane": "complete",
|
||||
"artifact": {
|
||||
"path": "/tmp/rea-perf-static-tool-audit-20261024/real-beam/erl_lint.beam",
|
||||
"sha256": "716d78394b7c3da983e36aac7aadd6673e5041677ba7de20413cc2704d958d55",
|
||||
"bytes": 233968
|
||||
},
|
||||
"answer_count": 4,
|
||||
"answer_sha256": "34ce3091dbea9ac8cd20831cf6443c7a1de6e02b753246cafa3e0e22b83907c9",
|
||||
"parent_evidence_id": "ev_df4bfa8b8f0391ad6291542fdccf9062ae3bb8dbabf01c2b166ca6f3493051fc",
|
||||
"source_report_sha256": "db64350ae25bff9e77f5e167f4a4ac095ed872e09c15de5ce3b2728c078a16d2",
|
||||
"runtime_identity_sha256": "1b4b40330a2e21d31c06a07d10937cb6a3b37cd0bcbf3d129380341d486b916e",
|
||||
"catalog_sha256": "f4a683209df5bed1ed25ae42c690fdd803348742132e88f77ba2e4e8333ca562",
|
||||
"calls": [
|
||||
{
|
||||
"name": "inspect_erlang_beam",
|
||||
"input": {
|
||||
"path": "/tmp/rea-perf-static-tool-audit-20261024/real-beam/erl_lint.beam"
|
||||
},
|
||||
"phase": "inspect",
|
||||
"elapsed_ms": 13658.772345000001,
|
||||
"reply_bytes": 146627,
|
||||
"is_error": false
|
||||
},
|
||||
{
|
||||
"name": "export_evidence_bundle",
|
||||
"input": {
|
||||
"path": "/tmp/rea-perf-static-tool-audit-20261024/sdk-cache-cohort/erl_lint-3-complete/evidence.json"
|
||||
},
|
||||
"phase": "export",
|
||||
"elapsed_ms": 48.42420299999867,
|
||||
"reply_bytes": 376,
|
||||
"is_error": false
|
||||
}
|
||||
],
|
||||
"phases": {
|
||||
"discovery_ms": 1121.564358
|
||||
},
|
||||
"wall_seconds": 17.257,
|
||||
"cpu_seconds": 16.276557,
|
||||
"kernel_family_peak_bytes": 1078980608,
|
||||
"events_at_payload_exit": {
|
||||
"memory.events": {
|
||||
"low": 0,
|
||||
"high": 0,
|
||||
"max": 0,
|
||||
"oom": 0,
|
||||
"oom_kill": 0,
|
||||
"oom_group_kill": 0
|
||||
},
|
||||
"pids.events": {
|
||||
"max": 0
|
||||
}
|
||||
},
|
||||
"payload_cleanup_verified": true,
|
||||
"final_whole_scope_events": null,
|
||||
"token_proxy": {
|
||||
"question": 32,
|
||||
"server_instructions": 513,
|
||||
"input_catalog_once": 55409,
|
||||
"arguments": 78,
|
||||
"text_replies": 20875,
|
||||
"combined": 76907,
|
||||
"complete_transport_catalog": 829075
|
||||
},
|
||||
"receipt_sha256": {
|
||||
"report.json": "d683aab80d34c4f0fb474981c5715a4ba1ee706bf4e74be10bf69c40358ec011",
|
||||
"catalog.json": "f4a683209df5bed1ed25ae42c690fdd803348742132e88f77ba2e4e8333ca562",
|
||||
"transcript.json": "12380e1e67368b8df7787d54be828c953d4e2ef63007c04cd9fc912b42cc42b5",
|
||||
"evidence.json": "ca66367cd1bb325d7d4e47f958c86234d01aaf749b68778ac5212cac246aacb3",
|
||||
"erl_lint-3-complete.log": "404f609e4dc10bbf0d8f1d3076751b2cb8b271ff401b00ea161a00944a55d558"
|
||||
}
|
||||
},
|
||||
{
|
||||
"case": "erl_lint-3-selected",
|
||||
"lane": "selected",
|
||||
"artifact": {
|
||||
"path": "/tmp/rea-perf-static-tool-audit-20261024/real-beam/erl_lint.beam",
|
||||
"sha256": "716d78394b7c3da983e36aac7aadd6673e5041677ba7de20413cc2704d958d55",
|
||||
"bytes": 233968
|
||||
},
|
||||
"answer_count": 4,
|
||||
"answer_sha256": "34ce3091dbea9ac8cd20831cf6443c7a1de6e02b753246cafa3e0e22b83907c9",
|
||||
"parent_evidence_id": "ev_df4bfa8b8f0391ad6291542fdccf9062ae3bb8dbabf01c2b166ca6f3493051fc",
|
||||
"source_report_sha256": "db64350ae25bff9e77f5e167f4a4ac095ed872e09c15de5ce3b2728c078a16d2",
|
||||
"runtime_identity_sha256": "a74d203a5a12d7eb82aaa1e0fda6595b9647af2255f318bf116953777052460f",
|
||||
"catalog_sha256": "9c09aa2944f10c806df78d1cfa2e1daeddc0a487c432701cb67fa65e14213b0c",
|
||||
"calls": [
|
||||
{
|
||||
"name": "inspect_erlang_beam",
|
||||
"input": {
|
||||
"path": "/tmp/rea-perf-static-tool-audit-20261024/real-beam/erl_lint.beam",
|
||||
"detail": "summary",
|
||||
"view": {
|
||||
"kind": "beam",
|
||||
"collection": "exports",
|
||||
"names": ["module", "exprs", "expr"],
|
||||
"fields": ["function_index", "function", "arity", "label"],
|
||||
"offset": 0,
|
||||
"limit": 64
|
||||
}
|
||||
},
|
||||
"phase": "inspect",
|
||||
"elapsed_ms": 14405.966256000002,
|
||||
"reply_bytes": 7093,
|
||||
"is_error": false
|
||||
},
|
||||
{
|
||||
"name": "export_evidence_bundle",
|
||||
"input": {
|
||||
"path": "/tmp/rea-perf-static-tool-audit-20261024/sdk-cache-cohort/erl_lint-3-selected/evidence.json"
|
||||
},
|
||||
"phase": "export",
|
||||
"elapsed_ms": 31.32568199999878,
|
||||
"reply_bytes": 376,
|
||||
"is_error": false
|
||||
}
|
||||
],
|
||||
"phases": {
|
||||
"discovery_ms": 1107.032897
|
||||
},
|
||||
"wall_seconds": 18.052,
|
||||
"cpu_seconds": 16.867381,
|
||||
"kernel_family_peak_bytes": 1105092608,
|
||||
"events_at_payload_exit": {
|
||||
"memory.events": {
|
||||
"low": 0,
|
||||
"high": 0,
|
||||
"max": 0,
|
||||
"oom": 0,
|
||||
"oom_kill": 0,
|
||||
"oom_group_kill": 0
|
||||
},
|
||||
"pids.events": {
|
||||
"max": 0
|
||||
}
|
||||
},
|
||||
"payload_cleanup_verified": true,
|
||||
"final_whole_scope_events": null,
|
||||
"token_proxy": {
|
||||
"question": 32,
|
||||
"server_instructions": 513,
|
||||
"input_catalog_once": 56359,
|
||||
"arguments": 120,
|
||||
"text_replies": 1066,
|
||||
"combined": 58090,
|
||||
"complete_transport_catalog": 834792
|
||||
},
|
||||
"receipt_sha256": {
|
||||
"report.json": "732e5b4f880d3a981d5362ece88578a2da9e68e80e97eca2dec2ae5856ebbdee",
|
||||
"catalog.json": "9c09aa2944f10c806df78d1cfa2e1daeddc0a487c432701cb67fa65e14213b0c",
|
||||
"transcript.json": "e62dc35d939b21418989093a8addadbe533d358a046feb7d99cecab10cedc2d4",
|
||||
"evidence.json": "2373bc2e57e40aac881f3ef91fa77187b098dd61d6cf44d37a4b8fe56d022b40",
|
||||
"erl_lint-3-selected.log": "2f35909a9ad5efa9e8ddeffe665887e8a9e9c97c13b4a030b765dd1ceecb14ef"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,125 +0,0 @@
|
||||
# Selected BEAM exports: completed-task measurements
|
||||
|
||||
Issue #6938. The analyst asks for every exported function with a small set of
|
||||
exact names, preserving arities, original table/atom indexes and labels. Both
|
||||
lanes export and validate the **complete** parent report, not only the answer.
|
||||
|
||||
Baseline `e09b06369d923e9cccedd91f6839ba643dfbd893`; selected delivery
|
||||
`bcc3f7611` (full SHA and source-clean runtime identities in the JSON receipt).
|
||||
The reader, format budgets and complete Evidence identity are unchanged.
|
||||
Node 24.22.1, SDK client/core/server 2.3.1, npm 11.16.0; Linux x86-64. The
|
||||
18-task cohort alternates three fresh complete/selected sessions per artifact.
|
||||
Fresh HOME/client/server and disabled Node compile cache; OS caches unflushed.
|
||||
This is a scripted client task, not an actual model investigation.
|
||||
|
||||
## Results
|
||||
|
||||
Medians, before → after. Seconds and kernel process-family MiB (includes file
|
||||
cache), not parent-only RSS. Ranges and every receipt hash are in
|
||||
[the 18-task JSON](erlang-beam-views.json).
|
||||
|
||||
| Real artifact | Whole-task estimated tokens | Wall s | Family CPU s | Family peak MiB |
|
||||
| ------------- | --------------------------: | ------------: | ------------: | --------------: |
|
||||
| erl_lint | 76,907 → 58,090 | 6.773 → 6.870 | 5.581 → 5.312 | 667.0 → 630.9 |
|
||||
| erl_parse | 72,499 → 58,087 | 6.367 → 6.974 | 5.627 → 5.555 | 666.1 → 657.5 |
|
||||
| pg2 | 57,733 → 58,018 | 6.365 → 7.576 | 5.295 → 5.615 | 648.2 → 626.9 |
|
||||
|
||||
For erl_lint the estimate drops 24.5%; erl_parse drops 19.9%. Small pg2 grows 0.5%
|
||||
(285 tokens), so selected delivery does not benefit every input. Complete stays
|
||||
the default. No stable general elapsed/CPU/memory improvement is established;
|
||||
the small selected task has a higher wall/CPU median. Full parsing/retention and
|
||||
cold client setup remain. Matching source-report, parent-ID and answer digests
|
||||
agree across all six tasks for each artifact.
|
||||
|
||||
The projection counts one question, server instructions, all 221 input-tool
|
||||
names/descriptions/schemas once, and every argument/text reply including export.
|
||||
`tiktoken 0.14.0/cl100k_base` serialization estimates are not actual model usage.
|
||||
Input directory cost grows 55,409→56,359tokens; the complete transport catalog
|
||||
including output schemas grows 829,075→834,792tokens and is reported separately.
|
||||
Protocol framing, host injection/history replay, output-schema presentation to
|
||||
a model and inference quality are unmeasured. Reported context reduction is
|
||||
specific to this declared client projection.
|
||||
|
||||
## SDK cached-catalog control
|
||||
|
||||
Six additional alternating fresh tasks use the SDK's cached catalog/default
|
||||
validator without an explicit `toolDefinition`. They answer/export the same
|
||||
question and pass the same independent oracle and cleanup checks. They are
|
||||
separate from the 18-task primary cohort; do not pool their startup costs.
|
||||
|
||||
Median wall 19.764→19.375s (ranges 17.257–21.577 / 18.052–22.397);
|
||||
family CPU 16.277→16.867s (ranges 15.287–16.562 / 16.139–17.070);
|
||||
family peak 1024.2→1053.9MiB.
|
||||
No general cached-client startup improvement is claimed. See
|
||||
[all six control receipts](erlang-beam-sdk-cache.json).
|
||||
|
||||
## Correctness and resources
|
||||
|
||||
The unmodified Debian `erlang-base_27.3.4.1+dfsg-1+deb13u3_amd64.deb` supplies
|
||||
actual OTP 27 compiler/stdlib modules: erl_lint 233,968 bytes (915 atoms, 119 imports,
|
||||
112 lambdas), erl_parse 608,240 bytes (1,035 atoms), and pg2 1,152 bytes (4 atoms).
|
||||
The package URL/SHA256, every selected artifact digest, source revision, driver
|
||||
hash and runtime fingerprint are retained in the JSON records. The independent
|
||||
[Python struct oracle](erlang-beam-oracle.py) checks every decoded table, chunk
|
||||
range/payload digest and Code header, then the selected export answer. It reads
|
||||
bytes only and deliberately qualifies these fixed OTP 27 layouts, not every
|
||||
historical/compact format. No Erlang installation, emulator or target execution
|
||||
was used. Default CLI and selected CLI on all three inputs also matched every
|
||||
oracle fact and parent ID and published exactly the complete source Evidence.
|
||||
|
||||
Each of the 24 SDK tasks exited 0 with payload cleanup verified and no guard stop,
|
||||
OOM or task-cap event. Aggregate CPUQuota 100%, payload affinity0–1,3 GiB early
|
||||
stop/4 GiB hard limit, no swap, TasksMax 192. CPU/peak counters are observed after
|
||||
the payload exits while the observer lives; final whole-scope counters remain
|
||||
unavailable. Costs include SDK discovery/schema validation, runtime/dependency
|
||||
hashing, assertions, source retention/export and cleanup. Sample acquisition,
|
||||
oracle generation and tokenization occur outside timed tasks. Node, all compiled dist
|
||||
files, selected entrypoints, 59 actual SDK files and driver files are hashed
|
||||
before/after; the source revisions were clean when fingerprints were created.
|
||||
|
||||
Public regressions additionally exercise all five collections, duplicate names
|
||||
and arities, selectors/fields/pages, missing versus empty tables, unresolved atom
|
||||
names, unavailable retention, source deletion, offline bundles and refusal before
|
||||
file reads. They establish those boundaries; real new-view qualification here is
|
||||
OTP 27 on Linux, not every platform or provider.
|
||||
|
||||
## Reproducing the question
|
||||
|
||||
Build clean pinned checkouts with the repository's pinned toolchain and acquire
|
||||
the public package linked in the JSON. Keep extracted files outside the repo.
|
||||
Generate each oracle, for example:
|
||||
|
||||
```sh
|
||||
python3 experiments/analysis-efficiency/erlang-beam-oracle.py \
|
||||
--input /samples/erl_lint.beam --output /runs/erl_lint-oracle.json \
|
||||
--name module --name exprs --name expr
|
||||
node experiments/analysis-efficiency/erlang-beam-views.mjs \
|
||||
--runtime /built/rea --identity /runs/runtime-identity.json \
|
||||
--oracle /runs/erl_lint-oracle.json --output /runs/erl_lint-selected \
|
||||
--lane selected
|
||||
```
|
||||
|
||||
A runtime fingerprint supplies `compiled_files` and `entrypoints` with relative
|
||||
paths/SHA256, `driver_files` relative to this experiment directory, actual SDK
|
||||
`dependency_files` with absolute paths/SHA256, and `node_sha256`. Source revision
|
||||
and clean-state fields record preparation. Create fingerprints before timing;
|
||||
the driver verifies them before and after each task. The publication records
|
||||
fingerprint/receipt hashes and full commands; the original complete fingerprint
|
||||
files, SDK transcripts, complete Evidence bundles and resource logs remain at
|
||||
`/tmp/rea-perf-static-tool-audit-20261024/`. The local cohort controller is retained
|
||||
there too. These local receipts are not shipped binaries or installed providers.
|
||||
|
||||
Wrap each invocation with `resource-run.py --observe-payload-exit`, setting the
|
||||
pinned Node PATH/heap, process affinity and resource policy as in its recorded
|
||||
command. Use a fresh output directory/session each time, alternate lanes three
|
||||
times per artifact, and count with
|
||||
[erlang-beam-costs.py](erlang-beam-costs.py). `--page-size` can qualify extra
|
||||
retained-page calls; `--sdk-cache` selects the separate cached-catalog control.
|
||||
The18 published primary tasks use the exact discovered toolDefinition/default
|
||||
validator and 64-row pages, requiring two calls each including full-source export.
|
||||
|
||||
The initial exploratory baseline used earlier main `e2463372`; it is not mixed
|
||||
into this table. Failed development checks remain in the local record: an
|
||||
incorrect formatter path and copied service import, followed by exact complete
|
||||
contract narrowing in old test consumers; corrected type/format/public checks
|
||||
passed. These were not successful benchmark samples.
|
||||
@@ -1,255 +0,0 @@
|
||||
/** Compare a completed selected-export question through production stdio MCP. */
|
||||
import assert from "node:assert/strict";
|
||||
import { createHash } from "node:crypto";
|
||||
import { createWriteStream } from "node:fs";
|
||||
import { mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join, resolve } from "node:path";
|
||||
import { finished } from "node:stream/promises";
|
||||
import { parseArgs } from "node:util";
|
||||
import { Client } from "@modelcontextprotocol/client";
|
||||
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
|
||||
const { values } = parseArgs({
|
||||
options: {
|
||||
...Object.fromEntries(
|
||||
["runtime", "oracle", "output", "identity"].map((k) => [
|
||||
k,
|
||||
{ type: "string" },
|
||||
]),
|
||||
),
|
||||
lane: { type: "string", default: "complete" },
|
||||
"page-size": { type: "string", default: "64" },
|
||||
"sdk-cache": { type: "boolean", default: false },
|
||||
},
|
||||
});
|
||||
assert.ok(["complete", "selected"].includes(values.lane));
|
||||
const pageSize = Number(values["page-size"]);
|
||||
assert.ok(Number.isSafeInteger(pageSize) && pageSize > 0);
|
||||
for (const key of ["runtime", "oracle", "output", "identity"])
|
||||
assert.equal(values[key], resolve(values[key]));
|
||||
const { createRetainedQueryCaller } =
|
||||
await import("./retained-query-calls.mjs");
|
||||
const { finalizeRetainedQueryReceipt } =
|
||||
await import("./retained-query-finalization.mjs");
|
||||
const hash = (x) => createHash("sha256").update(x).digest("hex"),
|
||||
oracleBytes = await readFile(values.oracle),
|
||||
expected = JSON.parse(oracleBytes),
|
||||
identityBytes = await readFile(values.identity),
|
||||
identity = JSON.parse(identityBytes);
|
||||
assert.equal(
|
||||
hash(await readFile(expected.expected.artifact.path)),
|
||||
expected.expected.artifact.sha256,
|
||||
);
|
||||
await mkdir(values.output, { mode: 0o700 });
|
||||
const account = await mkdtemp(join(tmpdir(), "rea-beam-task-"));
|
||||
const client = new Client({ name: "beam-selected-exports", version: "1" });
|
||||
const transport = new StdioClientTransport({
|
||||
command: process.execPath,
|
||||
args: [join(values.runtime, "scripts/rea.mjs"), "mcp"],
|
||||
maxBufferSize: 10485760,
|
||||
env: {
|
||||
HOME: account,
|
||||
TMPDIR: account,
|
||||
PATH: process.env.PATH,
|
||||
REA_LOG_LEVEL: "silent",
|
||||
NODE_OPTIONS: "--max-old-space-size=2048",
|
||||
NODE_DISABLE_COMPILE_CACHE: "1",
|
||||
UV_THREADPOOL_SIZE: "2",
|
||||
},
|
||||
stderr: "pipe",
|
||||
});
|
||||
const serverLog = createWriteStream(join(values.output, "server.log")),
|
||||
serverLogFinished = finished(serverLog);
|
||||
serverLogFinished.catch(() => {});
|
||||
transport.stderr?.pipe(serverLog);
|
||||
const report = {
|
||||
status: "running",
|
||||
lane: values.lane,
|
||||
artifact: expected.expected.artifact,
|
||||
calls: [],
|
||||
phases: {},
|
||||
question: `Return every exported function named ${expected.names.join(", ")} with its original export-table index, function atom index, arity and code label. Preserve complete source Evidence.`,
|
||||
client: values["sdk-cache"]
|
||||
? "Production stdio SDK, cached catalog definitions and default validator"
|
||||
: "Production stdio SDK, exact discovered toolDefinition and default validator",
|
||||
runtime_identity_sha256: hash(identityBytes),
|
||||
oracle_sha256: hash(oracleBytes),
|
||||
driver_sha256: hash(await readFile(new URL(import.meta.url))),
|
||||
};
|
||||
const verifyRuntime = async () => {
|
||||
assert.equal(hash(await readFile(values.identity)), hash(identityBytes));
|
||||
assert.equal(hash(await readFile(values.oracle)), hash(oracleBytes));
|
||||
for (const r of [...identity.compiled_files, ...identity.entrypoints])
|
||||
assert.equal(
|
||||
hash(await readFile(join(values.runtime, r.path))),
|
||||
r.sha256,
|
||||
r.path,
|
||||
);
|
||||
for (const r of identity.dependency_files ?? [])
|
||||
assert.equal(hash(await readFile(r.path)), r.sha256, r.path);
|
||||
for (const r of identity.driver_files)
|
||||
assert.equal(
|
||||
hash(await readFile(new URL(r.path, import.meta.url))),
|
||||
r.sha256,
|
||||
r.path,
|
||||
);
|
||||
assert.equal(hash(await readFile(process.execPath)), identity.node_sha256);
|
||||
assert.equal(
|
||||
hash(await readFile(new URL(import.meta.url))),
|
||||
report.driver_sha256,
|
||||
);
|
||||
};
|
||||
const transcript = [];
|
||||
let definitions;
|
||||
const caller = createRetainedQueryCaller(
|
||||
{
|
||||
callTool(params, options) {
|
||||
return client.callTool(
|
||||
params,
|
||||
values["sdk-cache"]
|
||||
? options
|
||||
: {
|
||||
...options,
|
||||
toolDefinition: definitions.get(params.name),
|
||||
},
|
||||
);
|
||||
},
|
||||
},
|
||||
(receipt, event) => {
|
||||
report.calls.push(receipt);
|
||||
transcript.push(event);
|
||||
},
|
||||
);
|
||||
const observe = async (name, input, phase) => {
|
||||
const result = await caller(name, input, phase);
|
||||
assert.notEqual(result.isError, true, JSON.stringify(result));
|
||||
assert.ok(result.structuredContent);
|
||||
return result.structuredContent;
|
||||
};
|
||||
const verifySource = (source) => {
|
||||
for (const [key, value] of Object.entries(expected.expected))
|
||||
assert.deepEqual(source[key], value, key);
|
||||
assert.equal(source.coverage.status, "complete");
|
||||
};
|
||||
try {
|
||||
await verifyRuntime();
|
||||
const started = performance.now();
|
||||
await client.connect(transport);
|
||||
report.server_instructions = client.getInstructions() ?? "";
|
||||
const catalog = await client.listTools();
|
||||
definitions = new Map(catalog.tools.map((t) => [t.name, t]));
|
||||
report.catalog_tools = catalog.tools.length;
|
||||
await writeFile(join(values.output, "catalog.json"), JSON.stringify(catalog));
|
||||
report.phases.discovery_ms = performance.now() - started;
|
||||
const selected = values.lane === "selected";
|
||||
const selectedView = (offset) => ({
|
||||
kind: "beam",
|
||||
collection: "exports",
|
||||
names: expected.names,
|
||||
fields: ["function_index", "function", "arity", "label"],
|
||||
offset,
|
||||
limit: pageSize,
|
||||
});
|
||||
const response = await observe(
|
||||
"inspect_erlang_beam",
|
||||
{
|
||||
path: expected.expected.artifact.path,
|
||||
...(selected ? { detail: "summary", view: selectedView(0) } : {}),
|
||||
},
|
||||
"inspect",
|
||||
);
|
||||
report.parent_evidence_id = selected
|
||||
? response.normalized_result.parent_evidence_id
|
||||
: response.evidence_id;
|
||||
let answer;
|
||||
if (!selected) {
|
||||
verifySource(response.normalized_result);
|
||||
answer = (response.normalized_result.exports ?? [])
|
||||
.map((r, i) => ({ source_index: i, ...r }))
|
||||
.filter((r) => expected.names.includes(r.function));
|
||||
} else {
|
||||
answer = [];
|
||||
let offset = 0,
|
||||
page = response.normalized_result;
|
||||
for (;;) {
|
||||
assert.equal(page.kind, "beam");
|
||||
assert.equal(page.page.collection, "exports");
|
||||
assert.deepEqual(page.beam.artifact, expected.expected.artifact);
|
||||
assert.deepEqual(page.beam.coverage, expected.expected.coverage);
|
||||
assert.equal(page.beam.counts.exports, expected.expected.exports.length);
|
||||
assert.equal(page.coverage.total, expected.answer.length);
|
||||
answer.push(...page.page.items);
|
||||
if (page.coverage.next_offset === null) break;
|
||||
assert.ok(page.coverage.next_offset > offset);
|
||||
offset = page.coverage.next_offset;
|
||||
page = (
|
||||
await observe(
|
||||
"inspect_analysis_view",
|
||||
{
|
||||
source: {
|
||||
kind: "retained-evidence",
|
||||
evidence_id: report.parent_evidence_id,
|
||||
},
|
||||
view: selectedView(offset),
|
||||
},
|
||||
"selected-page",
|
||||
)
|
||||
).normalized_result;
|
||||
}
|
||||
}
|
||||
assert.deepEqual(answer, expected.answer);
|
||||
report.answer_count = answer.length;
|
||||
report.answer_sha256 = hash(
|
||||
JSON.stringify(
|
||||
answer.map((r) =>
|
||||
Object.fromEntries(
|
||||
Object.entries(r).sort(([a], [b]) => a.localeCompare(b)),
|
||||
),
|
||||
),
|
||||
),
|
||||
);
|
||||
await writeFile(join(values.output, "answer.json"), JSON.stringify(answer));
|
||||
await observe(
|
||||
"export_evidence_bundle",
|
||||
{ path: join(values.output, "evidence.json") },
|
||||
"export",
|
||||
);
|
||||
const bundle = JSON.parse(
|
||||
await readFile(join(values.output, "evidence.json")),
|
||||
);
|
||||
const parent = bundle.records.find(
|
||||
(r) => r.evidence_id === report.parent_evidence_id,
|
||||
);
|
||||
assert.ok(parent);
|
||||
assert.equal(parent.operation, "inspect_erlang_beam");
|
||||
verifySource(parent.normalized_result);
|
||||
report.source_report_sha256 = hash(JSON.stringify(parent.normalized_result));
|
||||
report.complete_source_export_verified = true;
|
||||
report.status = "passed";
|
||||
} catch (cause) {
|
||||
report.status = "failed";
|
||||
report.error = String(cause);
|
||||
} finally {
|
||||
try {
|
||||
await verifyRuntime();
|
||||
} catch (cause) {
|
||||
report.status = "failed";
|
||||
report.runtime_identity_failure = String(cause);
|
||||
}
|
||||
await finalizeRetainedQueryReceipt({
|
||||
client,
|
||||
transport,
|
||||
serverLog,
|
||||
serverLogFinished,
|
||||
account,
|
||||
report,
|
||||
output: values.output,
|
||||
transcript,
|
||||
});
|
||||
if (
|
||||
report.status !== "passed" ||
|
||||
report.cleanup_failures.length ||
|
||||
report.receipt_failures.length
|
||||
)
|
||||
process.exitCode = 1;
|
||||
}
|
||||
@@ -1,218 +0,0 @@
|
||||
# EVM descriptor and real-template checkpoint
|
||||
|
||||
Continues [#2016](https://github.com/morluto/rea/issues/2016) after the native
|
||||
checkpoint merged. The internal descriptor binds the existing exact EVM
|
||||
contract, CLI name and typed service lifetime. Existing complete inputs, output,
|
||||
provider selection and cleanup remain the baseline. Wiring alone does not establish
|
||||
token or latency savings.
|
||||
|
||||
The descriptor contains only the named contract, existing CLI name and existing
|
||||
factory. The legacy `createEvmInterfaceService` export remains usable. The MCP
|
||||
host selects one complete service instance, preserves injection identity, passes
|
||||
that same instance to registration and retains it for cleanup/retry. No bound
|
||||
service wrapper, generic registry, automatic dependency acquisition or new pack
|
||||
selection is introduced.
|
||||
|
||||
Five actual in-memory SDK boundary scenarios verify converted schemas, duplicate
|
||||
registration rejection before use, malformed inputs, explicit unavailable or
|
||||
incompatible-output failures with a healthy WASM peer, cleanup retry, and request
|
||||
cancellation followed by recovery. Their injected ports are recording seams, not
|
||||
real EVMole/WABT claims. Nine service and 26 provider-boundary cases also pass;
|
||||
two unsupported-host scenarios are skipped on this Linux host. The existing seven
|
||||
WASM MCP scenarios pass. Full `npm run check` passes; its approximately 3 GiB
|
||||
sampled peak and `memory.high` events are excluded from performance claims.
|
||||
Candidate real-artifact/catalog and model observations are recorded below.
|
||||
|
||||
## Public inputs and independent checks
|
||||
|
||||
[corpus.json](corpus.json) retains the original WETH historical example and adds
|
||||
Uniswap v3-core 1.0.1 Factory/Pool compiler runtime templates from the official npm
|
||||
package. Archive integrity, exact regular member, compiler JSON, original carrier
|
||||
and decoded-byte digests are pinned. The carrier is the original `deployedBytecode`
|
||||
field plus one trailing LF. No placeholder is rewritten and no contract is executed.
|
||||
|
||||
These are compiler templates, not live deployed instances. Their published Hardhat
|
||||
artifacts omit immutable-reference maps; that metadata remains explicitly unknown.
|
||||
The recorded empty link-reference maps do not establish absence of immutables.
|
||||
WETH remains a published historical RPC example, with unknown live authenticity.
|
||||
|
||||
Actual control CLI observations infer 7 Factory and 26 Pool functions. The
|
||||
[independent verifier](verify-evm-template.mjs) hashes each canonical published ABI
|
||||
signature with caller-supplied Ethereum Keccak and checks the complete ABI-dispatch
|
||||
selector multiset against the original REA response. Both sets match. This does not
|
||||
establish inferred argument/mutability equivalence, deployment authenticity or
|
||||
runtime semantics; all producer unknowns remain visible.
|
||||
|
||||
## Latest-main control
|
||||
|
||||
Production control: `1770f38cfc6cbe87491d0de8b8730c305c5cf8ea`, Node 24.18.0,
|
||||
npm 11.16.0, matching locked dependencies and unchanged EVMole 0.9.3.
|
||||
The frozen `dist/`/`bridge/` file-table SHA-256 is
|
||||
`7fd6c2c37b4e166df158ec29d48d31f3712b9b087d9807ee019ee997d06ea144`.
|
||||
Measurement driver SHA-256 is
|
||||
`f767c0c0232be1e11918e70e3197a6919680d16688076bef28d4320662f712f7`.
|
||||
Each public SDK run validates the selected named advertised schema and verifies
|
||||
original carrier and decoded identities, inference/unknown scope and unchanged
|
||||
input bytes. All three runs return the same complete 2,420,709-byte catalog.
|
||||
|
||||
| Question | Inferred functions | SDK tool-result JSON bytes | First workflow seconds |
|
||||
| ------------------------ | -----------------: | -------------------------: | ---------------------: |
|
||||
| WETH historical carrier | 11 | 26,760 | 11.432 |
|
||||
| Uniswap Factory template | 7 | 110,424 | 3.628 |
|
||||
| Uniswap Pool template | 26 | 110,998 | 11.211 |
|
||||
|
||||
These are initial observations, not paired performance conclusions or model-token
|
||||
counts. Each workflow includes connection, complete catalog discovery and one
|
||||
interface query. Full bytecode remains inline. Candidate/control repetitions must
|
||||
use the same source driver, inputs, provider settings and Node/worker limits.
|
||||
|
||||
Preparation against earlier frozen main `b1bf4a70d44191348ca20a91a52fb663da4daed0`
|
||||
also passed all 59 cases of the existing real CLI/MCP EVM verifier with privately
|
||||
supplied solc 0.8.30 and strace 6.1. Those cases cover compiler configurations,
|
||||
raw/hex identity, malformed/unsupported carriers, missing limiter and resource
|
||||
constraints. Its measured scope took 233.1 seconds, sampled peak 1,192,730,624
|
||||
bytes and last-observed CPU 181.018 seconds. This is a control, not candidate proof.
|
||||
|
||||
All heavy runs are serial under one CPU, 4 GiB hard memory/3 GiB early-stop, no swap
|
||||
and bounded tasks. Final cgroup counters disappear at scope teardown and remain
|
||||
unverified; last-observed zero limit events are not substituted for final counters.
|
||||
Raw replies, tools, compiler artifacts and detailed receipts remain outside Git.
|
||||
|
||||
## Reproduction
|
||||
|
||||
Build and freeze each pinned runtime using the repository toolchain, then run
|
||||
`measure.mjs --artifact uniswap-v3.factory.hex` or `uniswap-v3.pool.hex` with the
|
||||
exact derived carrier, `--schemas selected`, explicit runtime/revision and a fresh
|
||||
private output directory. Existing `weth.hex` answer fields and invocation remain
|
||||
unchanged. The same common workflow verifies all three offline carriers.
|
||||
|
||||
Run `verify-evm-template.mjs` with `--artifact` selecting the corpus ID,
|
||||
`--artifact-json` selecting its original published compiler JSON, `--evidence`
|
||||
selecting the recorded CLI Evidence or SDK structured reply, `--keccak-module`
|
||||
selecting the absolute `js-sha3/src/sha3.js` module from the private solc toolchain,
|
||||
and `--output` selecting a new private receipt. It acquires no dependencies and
|
||||
does not overwrite an existing receipt.
|
||||
|
||||
The [template model question](questions/evm-uniswap.txt) requires the same complete
|
||||
inferred interface facts and explicit unknowns. An ABI-count match is not used as
|
||||
a model grade.
|
||||
|
||||
## Candidate compatibility and real-provider verification
|
||||
|
||||
Candidate: `152984da771450048ee9d60db8a410994a125878`, frozen runtime file-table
|
||||
SHA-256 `725563640c27eb50ddb6f977d2dfbe1bdaddf7e05c02897318bceaa36fa602d7`.
|
||||
Two repetitions for each EVM input produce six passing paired fact/settings
|
||||
comparisons with the same driver and explicit limits. Each report preserves both
|
||||
original Evidence reference maps. The complete actual SDK catalog is byte-identical
|
||||
on every pair, including order, names, descriptions, schemas and annotations;
|
||||
catalog SHA-256 is
|
||||
`8664e42fd5225645cfd910f440f731c67ac198d3d53aa85b5c485374c74cf603`.
|
||||
Result byte counts remain unchanged. Candidate Factory/Pool selectors also match
|
||||
the independent published ABI oracle.
|
||||
|
||||
| Question | Control seconds, repetitions 1 / 2 | Candidate seconds, repetitions 1 / 2 |
|
||||
| -------- | ---------------------------------: | -----------------------------------: |
|
||||
| Factory | 3.628 / 3.650 | 3.708 / 3.605 |
|
||||
| Pool | 11.211 / 3.591 | 3.749 / 3.581 |
|
||||
| WETH | 11.432 / 3.694 | 3.572 / 3.677 |
|
||||
|
||||
First controls preceded wiring; each second pair ran candidate then control.
|
||||
Every measurement starts a new MCP process; host cache/page state is not fully
|
||||
controlled. Slower initial controls do not establish a descriptor speedup.
|
||||
Second-pair latency and last-observed CPU are similar. Sampled EVM SDK family peaks
|
||||
range from 510,255,104 to 527,622,144 bytes. No scope abort occurred; final counters
|
||||
remain unknown, so these measurements are descriptive.
|
||||
|
||||
The candidate passes all 59 existing real CLI/MCP verifier cases with solc 0.8.30
|
||||
and unchanged EVMole 0.9.3. Actual syscall traces report target execution and network
|
||||
sockets absent for those scenarios, and the verifier reports no remaining
|
||||
descendants. The scope took 177.456 seconds, sampled peak 1,237,020,672 bytes and
|
||||
last-observed CPU 144.913 seconds. Different totals from the earlier control are
|
||||
not attributed to unchanged provider algorithms.
|
||||
|
||||
Post-descriptor public MCP regression uses the original pinned shipped corpus:
|
||||
|
||||
| Domain / selected question | Actual provider / scope | Calls | Seconds |
|
||||
| ------------------------------------------- | -------------------------------------------------- | ----: | ------: |
|
||||
| SQLite WASM counts / exports | WABT 1.0.42; selected summary and retained read | 2 | 5.516 |
|
||||
| ShareX managed metadata / native boundaries | Static PE/CLI, no assembly execution | 2 | 3.710 |
|
||||
| Obsidian manifest / inventory | Complete inventory retained, no AST claim | 4 | 5.376 |
|
||||
| NewPipe package / MainActivity methods | Real headless JADX 0.7.1 | 3 | 21.669 |
|
||||
| OpenWrt regions | Real firmware signature scan, no device execution | 1 | 3.581 |
|
||||
| jq native entry | Ghidra 12.1.4, default analysis, ephemeral Program | 4 | 270.986 |
|
||||
|
||||
All six workflows pass named SDK output validation and original digest checks;
|
||||
input bytes remain unchanged. Maximum sampled family peak is 1,284,648,960 bytes.
|
||||
They establish those available Linux workflows, not paired performance or
|
||||
Windows/macOS, IDA/Hopper, Android-device, DOS COM, browser or JavaScript AST
|
||||
behavior. Browser/AST were not repeated for this narrow migration.
|
||||
|
||||
## Real model complete-output calibration
|
||||
|
||||
Six serial Codex CLI 0.161.0 trials request `gpt-5.6-terra`, high reasoning, with
|
||||
isolated temporary profiles pointing to the frozen candidate. Resolved server-side
|
||||
model identity is not inferred. Only the EVM inspection tool is enabled, matching
|
||||
the earlier WETH calibration; this is not full-catalog tool selection verification.
|
||||
Every trial makes one successful call and no failed call.
|
||||
|
||||
| Trial | Total input | Uncached input | Output | Seconds |
|
||||
| --------- | ----------: | -------------: | -----: | ------: |
|
||||
| Factory 1 | 66,158 | 36,974 | 1,329 | 43.483 |
|
||||
| Pool 1 | 65,600 | 36,416 | 2,307 | 69.546 |
|
||||
| WETH 1 | 54,700 | 25,516 | 1,144 | 56.372 |
|
||||
| Factory 2 | 65,964 | 36,780 | 1,453 | 26.412 |
|
||||
| Pool 2 | 65,753 | 36,569 | 1,939 | 74.706 |
|
||||
| WETH 2 | 54,502 | 25,318 | 1,208 | 35.137 |
|
||||
|
||||
Every trial reports 29,184 cached input tokens. Selected fact/reference/lifecycle
|
||||
grades all pass: original/decoded identities, every inferred selector, offset,
|
||||
dispatch, argument/mutability string including null, explicit unknowns, no runtime
|
||||
execution, observed Evidence references, unchanged bytes and removed profiles.
|
||||
Raw transcripts and grades remain private. Sampled peaks are
|
||||
594,853,888–776,335,360 bytes; last-observed CPU is 6.181–7.034 seconds. Strict
|
||||
overall grades remain unverified because final resource counters are unknown;
|
||||
they are not relabeled as wholly passing resource-qualified runs.
|
||||
|
||||
This calibrates the current complete path after compatible wiring, not a paired
|
||||
model performance experiment. The next token experiment should answer the same
|
||||
interface question through an explicitly selected precise view, retain full
|
||||
bytecode/original Evidence under the existing lifetime, and preserve complete
|
||||
defaults. Discover support before sending new options to an older server. Include
|
||||
discovery, follow-up, failures and retries in the complete task cost.
|
||||
|
||||
## Static contract admission verification
|
||||
|
||||
The follow-up from main `c462a4ebd3a10e59253bcf38353a0914b176738b` tests the
|
||||
pilot's actual static boundary, without adding a dynamic module registry or
|
||||
changing production code. Complete-project TypeScript checking now verifies
|
||||
that the inferred descriptor type rejects a WASM operation contract, a WASM
|
||||
CLI command, a WASM service factory, an unrelated input schema and an unrelated
|
||||
output schema. These are negative assignability assertions: their guarantee
|
||||
comes from the compiler, not from executing an incompatible descriptor.
|
||||
The existing named contract and CLI identity are also checked at runtime;
|
||||
separate factory calls produce distinct service/provider owners. Invalid target
|
||||
input remains an application error without launching a worker.
|
||||
|
||||
The actual public in-memory MCP SDK test additionally spies on the provider:
|
||||
duplicate EVM registration throws the named `already registered` error before
|
||||
any provider call. Converted input/output JSON Schemas validate before the
|
||||
first valid operation. Invalid client arguments are rejected before execution;
|
||||
incompatible provider reply data is validated **after** execution and is not
|
||||
presented as a pre-use contract guarantee. Existing healthy-peer, cancellation,
|
||||
retained summary/default and cleanup-retry scenarios remain covered.
|
||||
|
||||
Full project typecheck passes with the installed compiler's supported single
|
||||
thread scheduling; it includes all source/test files. The two exact focused
|
||||
files pass 12 runtime cases. The resource scopes sample 2,057,961,472 bytes for
|
||||
typecheck and 2,426,290,176 bytes for required build plus focused tests, below the
|
||||
unchanged 3 GiB early-stop/4 GiB hard ceiling. Final transient-cgroup counters
|
||||
remain unknown. No new real-provider or performance improvement is claimed by
|
||||
these compile/SDK checks; the earlier pinned real EVMole/WABT/lifecycle evidence
|
||||
retains its original scope and source identity.
|
||||
|
||||
This proves the one statically composed descriptor's named/schema/factory
|
||||
compatibility and duplicate-registration guard. It does not implement runtime
|
||||
admission of arbitrary JavaScript descriptors, third-party plugin ABI/version
|
||||
negotiation or a general plugin SDK. Those remain deferred by the original plan.
|
||||
Existing callers, full defaults, provider inputs, owned lifetimes and output
|
||||
identities are unchanged.
|
||||
@@ -1,208 +0,0 @@
|
||||
# EVM interface delivery and migration checkpoint
|
||||
|
||||
Continues [#2016](https://github.com/morluto/rea/issues/2016) after the static
|
||||
descriptor merged in [#2261](https://github.com/morluto/rea/pull/2261). Opt-in
|
||||
`inspect_evm_interface` MCP `detail: "summary"` returns every inferred interface
|
||||
candidate without bytecode hex or duplicate stdout. Complete Evidence remains
|
||||
retained under its original identity. Delivery bytes fall; the measured model
|
||||
cohort does **not** establish a stable token or task-time improvement.
|
||||
|
||||
## Frozen scope
|
||||
|
||||
Control production is `ad08510a113d7a9b8d92fb1e385757880e469869`; candidate
|
||||
production is `32c7548b9ae5c8cda348653d177b1747127c74ba`. Their compiled file-table
|
||||
SHA-256 values are respectively
|
||||
`f68e85030cb05082520ea9f0d7650298ae316ce7bba59a8942e0cc1318ae9850` and
|
||||
`978b3a014e3b2a1e1c6620364dcd793fa7cb40f19892d44cfb0312739a8c14a3`.
|
||||
The SDK driver SHA-256 is
|
||||
`a95d8d8e9f72c0b2b885c61907ed13054ebf7d1642f6f13b0126163840b1221a`.
|
||||
Driver commit `03bff83e70e09a917341c1fe3ebffbfdc67785c5` fixes only reading the
|
||||
public `get_evidence_bundle.result.records` envelope; production is unchanged.
|
||||
Both runtimes use Node 24.18.0, matching locked dependencies, SDK client/server
|
||||
2.3.1 and unchanged bundled EVMole 0.9.3.
|
||||
|
||||
Inputs are the digest-pinned WETH historical hex carrier and Uniswap v3-core
|
||||
1.0.1 Factory/Pool compiler runtime templates in [corpus.json](corpus.json).
|
||||
Original carrier and decoded-byte identities remain distinct. Templates are
|
||||
not authenticated deployed instances; constructor-dependent values and absent
|
||||
immutable-reference metadata stay unknown. No chain, target execution or
|
||||
signature lookup is introduced.
|
||||
|
||||
## Public SDK pairs
|
||||
|
||||
Two pairs per artifact run serially: complete control then selected candidate,
|
||||
followed by candidate then control. Each starts a fresh MCP process, discovers
|
||||
the catalog and validates the actual named advertised schema. Host caches are
|
||||
not independently controlled. All six fact/settings comparisons pass. Both
|
||||
answer projections omit unrequested hex; original complete replies remain intact.
|
||||
|
||||
| Input | Complete / summary result bytes | Reduction | Control seconds, 1 / 2 | Candidate seconds, 1 / 2 |
|
||||
| ------- | ------------------------------: | --------: | ---------------------: | -----------------------: |
|
||||
| Factory | 110,424 / 14,442 | 86.92% | 3.670 / 3.766 | 3.664 / 3.867 |
|
||||
| Pool | 110,998 / 19,922 | 82.05% | 4.110 / 3.740 | 3.977 / 3.870 |
|
||||
| WETH | 26,760 / 15,378 | 42.53% | 3.671 / 3.710 | 3.930 / 3.673 |
|
||||
|
||||
These measurements support smaller delivery, not faster interpretation. Analysis
|
||||
still runs and the full parent still incurs producer/retention costs. Catalog
|
||||
count and order stay at 184 tools. The complete catalog grows from 2,422,229 to
|
||||
2,429,938 compact JSON bytes (+7,709); all name/description/input definitions
|
||||
grow from 512,655 to 514,074 bytes. The EVM input definition grows from 1,443 to
|
||||
2,029 bytes. Only `inspect_analysis_view` and `inspect_evm_interface` definitions
|
||||
change. These discovery costs are visible and are not converted into token counts.
|
||||
|
||||
Nine separate measurements verify new-server complete defaults, old-server
|
||||
summary fallback and retained reads for all three inputs. The four-call retained
|
||||
proofs retrieve the complete canonical parent plus pages and cost 160,110,
|
||||
177,106 and 79,160 bytes for Factory, Pool and WETH respectively. They are
|
||||
verification workflows, not examples of one-call delivery savings.
|
||||
|
||||
## Real Codex pairs and answer qualification
|
||||
|
||||
Twelve serial Codex CLI 0.161.0 trials use the existing identical questions for
|
||||
each pair, the same artifact path, one enabled EVM tool, temporary MCP profiles
|
||||
and requested `gpt-5.6-terra` high. The requested alias is not proof of the
|
||||
resolved model identity. Trials include schema discovery, reasoning, one actual
|
||||
successful EVM call and final answer. No call fails or retries. Profiles are
|
||||
removed and input bytes remain unchanged. This fixed one-tool profile does not
|
||||
test selection from the full catalog.
|
||||
|
||||
| Input / repetition | Uncached input, control → candidate | Seconds, control → candidate | Candidate delivery |
|
||||
| ------------------ | ----------------------------------: | ---------------------------: | ------------------ |
|
||||
| Factory / 1 | 22,901 → 25,668 | 52.485 → 48.768 | summary |
|
||||
| Factory / 2 | 37,181 → 15,270 | 53.404 → 45.345 | summary |
|
||||
| Pool / 1 | 36,652 → 26,657 | 57.764 → 66.642 | summary |
|
||||
| Pool / 2 | 28,305 → 36,259 | 62.068 → 71.756 | complete |
|
||||
| WETH / 1 | 21,128 → 25,187 | 44.869 → 46.809 | summary |
|
||||
| WETH / 2 | 17,417 → 17,529 | 30.364 → 39.943 | summary |
|
||||
|
||||
Keep all trials, including the candidate that explicitly chose complete delivery.
|
||||
Five of six candidates select summary, but uncached input improves in only two
|
||||
pairs and time improves in only two. Aggregate control → candidate observations
|
||||
are 362,496 → 357,770 total input; 198,912 → 211,200 cached input;
|
||||
163,584 → 146,570 uncached input; 8,953 → 10,397 output; and
|
||||
300.954 → 319.263 seconds. Cache state and model/service variation are not
|
||||
separated; the aggregate does not establish a repeatable saving for an individual
|
||||
question or a qualified performance recommendation.
|
||||
|
||||
Every answer passes exact core-field checks: both identities/byte counts, every
|
||||
function selector/offset/dispatch/argument/mutability including null, inference,
|
||||
unknown completeness/kind/hardfork/authenticity, and absent runtime execution.
|
||||
All reported IDs occur in actual responses. However, the existing grader requires
|
||||
the final limitation list to equal the producer list verbatim and in order.
|
||||
Only three trials pass that additional check; nine fail. The questions request
|
||||
preserving limitations but do not explicitly request a verbatim ordered list.
|
||||
Four failures contain all original strings with extra correct scope statements
|
||||
or different ordering. The other five paraphrase or omit source caveats, including
|
||||
worker or upstream-metadata limitations. Do not silently waive those omissions
|
||||
or turn these into twelve fully acceptable answers. The strict reports and
|
||||
original transcripts remain intact; core-field success is a narrower result.
|
||||
|
||||
Model scope sampled peaks range from 562.5 to 710.5 MiB, with last-observed CPU
|
||||
5.945–6.878 seconds. CPU includes the local process family, not remote model
|
||||
computation. All heavy jobs use one CPU, a 4 GiB hard limit, 3 GiB early-stop,
|
||||
no swap and 192 tasks. No scope abort occurs in this cohort. Final cgroup
|
||||
counters are unavailable after teardown: final OOM/task/limit-event assertions
|
||||
remain **unverified**. Last-observed counters are not substituted for final ones.
|
||||
Consequently neither strict answer acceptance nor resource-qualified performance
|
||||
success is claimed for the whole cohort.
|
||||
|
||||
## Existing-user compatibility
|
||||
|
||||
Omitted and explicit-complete MCP requests, the existing CLI and provider inputs
|
||||
preserve their full behavior. `detail` is removed before the unchanged provider
|
||||
input; original full Evidence parameters, raw result and identity are retained.
|
||||
Summary receives its own identity linked to that parent. All candidate facts,
|
||||
original report limitations, stderr/truncation and explicit unknowns stay inline.
|
||||
Complete-within-view candidate coverage is not global discovery completeness.
|
||||
Function pages retain original order; text pages use UTF-16 code units, with
|
||||
hex characters distinguished from decoded-byte function offsets.
|
||||
|
||||
Actual new default replies for all three artifacts validate against the frozen
|
||||
old advertised input/output schemas, and structured/text payloads agree. Older
|
||||
servers reject neither request nor unsupported fields because the experimental
|
||||
client discovers support before selecting summary and otherwise calls complete.
|
||||
This is an explicit client fallback, not universal server forward compatibility.
|
||||
|
||||
Three authentic old-runtime records and old-created canonical bundles pass new
|
||||
MCP import, retained summary, CLI inline summary and export. Original records,
|
||||
IDs and bundle source bytes stay deeply equal. The reproducible verifier below
|
||||
also checks function/hex pages with the analysis limiter explicitly unavailable,
|
||||
so retained reads cannot silently start successful producer analysis. This
|
||||
proves the pinned old/new Linux workflows, not every historical client release.
|
||||
|
||||
The initial retained experiment failed because its driver read the bundle's
|
||||
wrong envelope. Correcting that consumer produced passing retained proofs; the
|
||||
failed receipt stays archived and is not recorded as a REA defect. Separately,
|
||||
one static-check attempt hit the unchanged 3 GiB early-stop. The bounded retry
|
||||
passes with Node heap 512 MiB/native checker memory controls; its roughly 3 GiB
|
||||
peak and soft-limit events are excluded from efficiency comparisons.
|
||||
|
||||
## Reproduction and checks
|
||||
|
||||
Use [the shared instructions](README.md), explicit frozen revisions/runtimes and
|
||||
fresh private output directories. Run `measure.mjs` for `weth.hex`,
|
||||
`uniswap-v3.factory.hex` and `uniswap-v3.pool.hex`, with `--schemas selected` and
|
||||
paired `--delivery complete` / `summary`; run `compare.mjs` on equivalent pairs.
|
||||
Add `--verify-retained` only for the separate four-call proof. Use the unchanged
|
||||
[WETH](questions/evm.txt) or [template](questions/evm-uniswap.txt) question and the
|
||||
same one-tool profile for `agent-trial.py`, preserving all usage and grader results.
|
||||
|
||||
Export complete Evidence with the old runtime's public ledger workflow, then run:
|
||||
|
||||
```sh
|
||||
node experiments/analysis-efficiency/verify-evm-retention.mjs \
|
||||
--runtime-root ABS_FROZEN_NEW_RUNTIME --revision NEW_SHA \
|
||||
--bundle ABS_OLD_BUNDLE_JSON --output NEW_PRIVATE_ABS_DIR
|
||||
```
|
||||
|
||||
The script acquires no dependencies and preserves the old bundle. Revision is
|
||||
caller-supplied; retain the frozen file-table receipt alongside it. Raw JSON,
|
||||
transcripts, profiles, bytecode, compiler inputs and generated artifacts stay
|
||||
outside Git. This checkpoint adds source verification and curated observations.
|
||||
|
||||
The production change passes 64 focused EVM/WASM/generic-view/domain/service
|
||||
checks and full static/docs gates. Recording provider seams establish exact
|
||||
schemas, old defaults, inference/null preservation, historical identity failures,
|
||||
peer availability, cancellation and cleanup retry; they are not real-provider
|
||||
claims. Real-provider and shared-domain checks are recorded separately.
|
||||
|
||||
The same frozen candidate also passes six actual public MCP artifact workflows:
|
||||
SQLite WASM with WABT summary, ShareX managed metadata, Obsidian inventory,
|
||||
NewPipe with JADX, an OpenWrt region inspection, and jq entry analysis with
|
||||
Ghidra. These are shared-contract regressions; they do not establish AST analysis
|
||||
of all Obsidian code, full firmware reconstruction, browser coverage, or new token
|
||||
savings in those domains. Native Windows PE and DOS coverage remains at the
|
||||
[earlier native checkpoint](native-entry-views.md), not a new Ghidra/Windows claim.
|
||||
The source retention verifier passes for all three authentic old EVM bundles,
|
||||
including exact summaries, function/hex pages, CLI inline views and unchanged
|
||||
exported originals with the limiter explicitly unavailable.
|
||||
|
||||
The frozen candidate passes all 59 existing real EVM CLI/MCP verifier cases with
|
||||
caller-supplied solc 0.8.30 and strace 6.1, including malformed/unsupported
|
||||
carriers, unavailable limiter and capacity failures. Its scope takes 96.076
|
||||
seconds with 1,345,052,672 sampled peak bytes and 95.130 last-observed CPU seconds.
|
||||
This is regression evidence, not an interpreter speedup comparison; final
|
||||
resource counters remain unverified. Fresh fast static and generated-document
|
||||
checks also pass after adding the migration verifier and this report.
|
||||
|
||||
## Latest-main integration
|
||||
|
||||
The branch rebases without conflict onto main
|
||||
`0043d9b134c0a353e8bf85c0bde4492275f1f720`. Production integration commit
|
||||
`3766749316c345ccb594b1b748747b1af9582b34` preserves the same EVM projection,
|
||||
provider and delivery semantics; intervening main adds unrelated capabilities.
|
||||
The above performance cohort stays pinned to its original frozen runtimes and
|
||||
is not relabeled as a measurement of the larger integrated catalog.
|
||||
|
||||
At `cda794fb9d4a5ca2bd661853a899d579fb8a47b2`, all 30 affected EVM/WASM/generic-view
|
||||
MCP tests pass again, as do full `npm run check` and `npm run docs:check`. A fresh
|
||||
real Factory SDK selected-summary/retained-parent/page workflow and the original
|
||||
Pool bundle's SDK/CLI migration verifier also pass against the rebuilt integrated
|
||||
runtime. The latter still has an explicitly unavailable analysis limiter.
|
||||
|
||||
The initial integrated static-check scope stops at 3,221,446,656 sampled bytes,
|
||||
just above the unchanged 3 GiB early-stop. Running the same typecheck through
|
||||
its single Turbo task with less launcher overhead passes at 3,219,353,600 sampled
|
||||
bytes; the subsequent full gate uses that valid cached typecheck and passes.
|
||||
No memory threshold or check requirement is relaxed. Preserve both receipts;
|
||||
these near-limit validation jobs are excluded from efficiency conclusions.
|
||||
@@ -1,159 +0,0 @@
|
||||
# EVM descriptor lifecycle acceptance
|
||||
|
||||
Continues [#2016](https://github.com/morluto/rea/issues/2016) after
|
||||
[#2349](https://github.com/morluto/rea/pull/2349) merged. The
|
||||
[public verifier](verify-evm-lifecycle.mjs) exercises actual compiled CLI and
|
||||
stdio MCP workflows with unchanged EVMole 0.9.3/WASM. This increment adds
|
||||
verification only; production contracts, factories, algorithms and defaults
|
||||
remain unchanged. It does not introduce a plugin SDK or database cache.
|
||||
|
||||
## Runtime and public artifacts
|
||||
|
||||
The verified runtime is explicitly frozen from
|
||||
`67f817e3653efad35f7044e68fbd6f163b86864b`, built with Node 24.18.0 and the
|
||||
matching locked dependencies. Its `dist/` and `bridge/` file-table SHA-256 is
|
||||
`cf03167870ac9def929cfaab536d6bd791089031660d1b35bff20672ac07cbbf`.
|
||||
The table is checked again after each workflow; source and target bytes must
|
||||
remain unchanged before a success receipt is published.
|
||||
|
||||
Inputs are the digest-pinned [corpus](corpus.json): published historical WETH
|
||||
and official Uniswap v3-core 1.0.1 Factory/Pool compiler runtime templates.
|
||||
They are not independently authenticated live deployments. Their deployment,
|
||||
hardfork, immutable-reference and inferred-interface limitations remain as
|
||||
recorded in [the descriptor checkpoint](evm-descriptor.md). The healthy peer
|
||||
is the shipped SQLite WASM 3.53.4-build1 module, inspected by real WABT 1.0.42.
|
||||
No contract, native target, chain/RPC query or transaction is executed.
|
||||
|
||||
## Actual cancellation and retained cleanup
|
||||
|
||||
The verifier supplies a private test launcher through the existing explicit
|
||||
`REA_EVM_PRLIMIT_COMMAND` setting. It passes the original arguments to real
|
||||
`/usr/bin/prlimit`, runs the unchanged Node/EVMole worker, then briefly gates
|
||||
launcher completion after that worker exits and writes its complete reply.
|
||||
This controls a real lifecycle boundary without replacing the analysis engine.
|
||||
It adds process overhead and is not a performance comparison.
|
||||
|
||||
Before cancellation or permission changes, production inspection primitives
|
||||
verify the live launcher's run token, parent/group relationship and native
|
||||
start identity. The complete worker reply must match the original selected
|
||||
bytecode and producer facts. Filesystem changes apply only to the verified,
|
||||
non-root user's temporary runtime directory.
|
||||
|
||||
Three scenarios are verified:
|
||||
|
||||
- **Late cancellation:** the real worker has completed, while the public SDK
|
||||
request remains pending. SDK cancellation reports `REQUEST_TIMEOUT` locally;
|
||||
token sweeps and directory removal independently confirm provider cleanup.
|
||||
A subsequent complete request returns identical analysis facts.
|
||||
- **Successful analysis followed by cleanup failure:** removing write permission
|
||||
produces actual `EACCES`. MCP reports `cleanup_incomplete` and retains the
|
||||
original resource path. A retry before permission repair fails on that same
|
||||
resource. Real SQLite WASM analysis still succeeds. Restoring permission lets
|
||||
the provider retire the old resource and perform a fresh complete EVM query.
|
||||
- **Format failure followed by cleanup failure:** a deliberately malformed
|
||||
odd-nibble carrier reaches the unchanged worker. The cleanup error preserves
|
||||
its original `invalid_input` category and exact producer diagnostic rather
|
||||
than overwriting the failure. The same repair/retry and healthy-peer checks
|
||||
succeed.
|
||||
|
||||
The existing CLI `inspect-evm-interface PATH hex --json` produces the same
|
||||
complete normalized facts, parameters and subject as MCP. Fresh Evidence IDs
|
||||
are not equated. Final verifier lineage is checked after client/transport close;
|
||||
an empty descendant list is a point-in-time observation, not proof that the
|
||||
workflow never launched children. Cleanup callbacks always run and preserve
|
||||
verification and cleanup failures together.
|
||||
|
||||
This proves cancellation after actual analysis, not interruption inside
|
||||
EVMole's synchronous WASM computation. Server-shutdown cleanup retry still has
|
||||
the separately labeled injected-owner SDK tests from the descriptor checkpoint.
|
||||
These Linux x64 observations do not establish Windows/macOS provider behavior.
|
||||
Earlier real CLI/MCP malformed/unavailable/capacity checks remain linked in the
|
||||
descriptor checkpoint; they are not silently rerun or attributed to this script.
|
||||
|
||||
## Measurements and compatibility scope
|
||||
|
||||
Initial three-carrier verifier source SHA-256 is
|
||||
`2fec069b53324ef82070d323b26dd613b6da0caa7f18100adbeb39b045cf98dc`.
|
||||
All three carriers pass all three scenarios and CLI parity with this same source.
|
||||
|
||||
| Carrier | Seconds | Sampled peak family bytes | Last-observed CPU seconds |
|
||||
| ------- | ------: | ------------------------: | ------------------------: |
|
||||
| WETH | 25.595 | 1,116,884,992 | 23.926 |
|
||||
| Factory | 23.979 | 1,132,994,560 | 23.419 |
|
||||
| Pool | 24.589 | 1,141,415,936 | 24.043 |
|
||||
|
||||
Runs are serial under the existing one-CPU quota, 4 GiB hard memory, unchanged
|
||||
3 GiB early-stop, no swap and 192-task ceiling. Resource receipts, complete
|
||||
producer replies, derived malformed input, launcher and runtime copies stay in
|
||||
the private lab outside Git. Final cgroup counters disappear at scope teardown
|
||||
and remain unknown; sampled zero events do not prove final no-OOM/no-limit
|
||||
results. This is compatibility and recovery evidence, not a model-token,
|
||||
whole-task latency or CPU saving claim.
|
||||
|
||||
Latest main `81cca2e0e3ebd98c21f41d7ad63d6a00e4cbc3f9` could not build during
|
||||
setup: its shared partial-error handler passes three arguments to an interface
|
||||
requiring four. Independently completed main CI at
|
||||
`fdd8f826023c568937d237c2f97d855de283a9b8` reports the same TypeScript failure
|
||||
([run 38309578208](https://github.com/morluto/rea/actions/runs/38309578208)).
|
||||
The unrelated regression is recorded in the maintainer-requested private bug
|
||||
handoff; this change does not repair it. The maintainer independently corrected
|
||||
the producing caller in main `8fc2043eb8bcedbf34dce064f977e455438635d6`.
|
||||
The frozen-runtime observations above remain pinned to their original revision;
|
||||
synchronizing the branch does not relabel them as latest-main verification.
|
||||
|
||||
After rebasing, a fresh build and all 10 focused EVM MCP boundary tests pass.
|
||||
The same unchanged source verifier also passes all three real Factory scenarios,
|
||||
CLI parity, healthy SQLite WASM checks and final cleanup on the newly built
|
||||
runtime `b3eadf147221a2fa997ac26667a183ca5bccb1f4` (parent is corrected main
|
||||
`8fc2043eb8bcedbf34dce064f977e455438635d6`). Its `dist/`/`bridge/` file-table
|
||||
SHA-256 is `7a4cec0fa4f7e929356b50a6b13472f80bd6f526221e427125370cca3ef98d00`.
|
||||
This related rerun takes 24.285 seconds, samples 1,146,167,296 peak family bytes
|
||||
and records 23.662 last-observed CPU seconds. The older failed build and earlier
|
||||
cohort receipts are preserved; wider domain/model matrices are not repeated.
|
||||
|
||||
Static composition continues to bind one named canonical contract and one
|
||||
typed factory. Existing SDK checks reject duplicate registration before use
|
||||
and reject malformed inputs before invoking the provider. Producer contract
|
||||
violations are detected when the producer returns a result; they cannot be
|
||||
validated in advance without that result. A dynamic third-party module ABI
|
||||
preflight is not implemented or claimed. The corresponding broader acceptance
|
||||
item remains open rather than creating the deferred generic plugin registry.
|
||||
|
||||
## Final launcher review
|
||||
|
||||
A missing gate marker while gating is disabled cannot establish absence of a
|
||||
new launch. The fixture now appends a launch record before starting the actual
|
||||
worker; both permission-failure retries must leave that log byte-identical.
|
||||
This independently verifies that retained cleanup failure stops a new EVM
|
||||
launch, while preserving the original resource and healthy WASM peer.
|
||||
|
||||
Final verifier source SHA-256 is
|
||||
`fc9a50e2e54e4c8eede4b8a19283e11ddaa1faadea52eebc390b55b5a1902668`.
|
||||
All three related real Factory scenarios, CLI parity and final cleanup pass
|
||||
again on the corrected-main runtime: 23.579 seconds, 1,163,685,888 peak sampled
|
||||
family bytes and 23.110 last-observed CPU seconds. The earlier three-carrier
|
||||
receipts remain pinned to their earlier source; wider artifacts/models are
|
||||
not repeated for this review fix.
|
||||
|
||||
## Reproduction
|
||||
|
||||
Build and freeze the explicit source revision as documented in the measurement
|
||||
README. Supply an object of explicit string-valued provider settings, including
|
||||
the caller-installed WABT paths. As a non-root Linux x64 user, run under the
|
||||
existing resource wrapper:
|
||||
|
||||
```bash
|
||||
python3 experiments/analysis-efficiency/resource-run.py -- \
|
||||
node experiments/analysis-efficiency/verify-evm-lifecycle.mjs \
|
||||
--runtime-root /absolute/frozen-runtime \
|
||||
--revision 67f817e3653efad35f7044e68fbd6f163b86864b \
|
||||
--path /absolute/UniswapV3Factory.runtime.hex \
|
||||
--peer-path /absolute/sqlite3.wasm \
|
||||
--providers /absolute/explicit-providers.json \
|
||||
--output /absolute/new-private-receipt-directory
|
||||
```
|
||||
|
||||
The verifier checks the carrier and SQLite module against the committed corpus,
|
||||
creates an owner-only receipt directory without overwriting existing receipts,
|
||||
and acquires no dependencies. Preserve failed attempts separately. A shorter
|
||||
reply or a successful cleanup verifier does not establish an agent-cost gain.
|
||||
@@ -1,22 +0,0 @@
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<manifest xmlns:android="http://schemas.android.com/apk/res/android" package="com.rea.components" android:versionCode="1" android:versionName="1">
|
||||
<uses-sdk android:minSdkVersion="21" android:targetSdkVersion="30"/>
|
||||
<application android:permission="com.rea.components.PRIVATE" android:enabled="true" android:process=":application">
|
||||
<activity android:name=".Main" android:exported="true" android:enabled="false" android:process=":activity">
|
||||
<intent-filter android:priority="7">
|
||||
<action android:name="android.intent.action.VIEW"/>
|
||||
<category android:name="android.intent.category.DEFAULT"/>
|
||||
<data android:scheme="https" android:host="example.test" android:pathPrefix="/open"/>
|
||||
</intent-filter>
|
||||
<intent-filter android:priority="@integer/filter_priority">
|
||||
<action android:name="android.intent.action.EDIT"/>
|
||||
<data android:scheme="content" android:mimeType="text/plain"/>
|
||||
</intent-filter>
|
||||
<meta-data android:name="public.configuration" android:value="retained-in-manifest"/>
|
||||
</activity>
|
||||
<receiver android:name=".SmsReceiver" android:exported="false" android:enabled="@bool/receiver_enabled" android:process=":sms" android:permission="android.permission.BROADCAST_SMS">
|
||||
<intent-filter android:priority="-10"><action android:name="android.provider.Telephony.SMS_RECEIVED"/></intent-filter>
|
||||
</receiver>
|
||||
<provider android:name=".DemoProvider" android:enabled="true" android:exported="false" android:authorities="com.rea.components.provider" android:grantUriPermissions="true"/>
|
||||
</application>
|
||||
</manifest>
|
||||
@@ -1,15 +0,0 @@
|
||||
import com.android.tools.smali.smali.Smali;
|
||||
import com.android.tools.smali.smali.SmaliOptions;
|
||||
import java.util.List;
|
||||
|
||||
/** Assemble the local fixture without Smali CLI's optional JCommander dependency. */
|
||||
class BuildDex {
|
||||
public static void main(String[] args) throws Exception {
|
||||
SmaliOptions options = new SmaliOptions();
|
||||
options.apiLevel = 30;
|
||||
options.jobs = 1;
|
||||
options.outputDexFile = args[1];
|
||||
if (!Smali.assemble(options, List.of(args[0])))
|
||||
throw new IllegalStateException("Manifest fixture assembly failed");
|
||||
}
|
||||
}
|
||||
@@ -1,3 +0,0 @@
|
||||
# A DEX container for a static manifest fixture; no Android app is executed.
|
||||
.class public Lcom/rea/components/ManifestFixture;
|
||||
.super Ljava/lang/Object;
|
||||
@@ -1,43 +0,0 @@
|
||||
# Static compiled-manifest fixture
|
||||
|
||||
These source-owned declarations cover URI data/filter association, exact resource
|
||||
references, metadata names, component process/enablement and provider URI grants.
|
||||
`ManifestFixture.smali` supplies a DEX container; it does not implement or execute
|
||||
the declared Android components. The verification uses existing AAPT2 36.1.0,
|
||||
Android 36 `android.jar`, JDK 21 and the digest-pinned JADX headless MCP 0.7.1 JAR.
|
||||
No Android device, emulator or app execution is involved.
|
||||
|
||||
From the repository root, choose existing `AAPT2`, `ANDROID_JAR`, `JAVA_HOME` and
|
||||
`REA_JADX_MCP_JAR` paths, then build into a fresh private directory:
|
||||
|
||||
```sh
|
||||
fixture=experiments/analysis-efficiency/fixtures/android-components
|
||||
work=$(mktemp -d)
|
||||
"$AAPT2" compile --dir "$fixture/res" -o "$work/resources.zip"
|
||||
"$AAPT2" link --manifest "$fixture/AndroidManifest.xml" -I "$ANDROID_JAR" \
|
||||
-o "$work/manifest.apk" "$work/resources.zip"
|
||||
"$JAVA_HOME/bin/java" -Xmx512m -XX:ActiveProcessorCount=2 \
|
||||
-cp "$REA_JADX_MCP_JAR" "$fixture/BuildDex.java" \
|
||||
"$fixture/ManifestFixture.smali" "$work/classes.dex"
|
||||
python3 - "$work" <<'PY'
|
||||
from pathlib import Path
|
||||
import sys, zipfile
|
||||
root = Path(sys.argv[1])
|
||||
with zipfile.ZipFile(root / "manifest.apk", "a") as apk:
|
||||
apk.write(root / "classes.dex", "classes.dex")
|
||||
PY
|
||||
"$AAPT2" dump xmltree --file AndroidManifest.xml "$work/manifest.apk" > "$work/aapt2-xmltree.txt"
|
||||
"$AAPT2" dump resources "$work/manifest.apk" > "$work/aapt2-resources.txt"
|
||||
REA_JADX_HEAP_MIB=1024 REA_JADX_ACTIVE_PROCESSOR_COUNT=2 \
|
||||
node scripts/rea.mjs inspect-android-package "$work/manifest.apk" \
|
||||
--detail summary --view '{"kind":"android","facet":"components"}' \
|
||||
--evidence_output "$work/evidence.json" --json > "$work/cli.json"
|
||||
python3 "$fixture/verify-fixture.py" "$work"
|
||||
```
|
||||
|
||||
`verify-fixture.py` cross-checks concrete compiled XML facts with the independent
|
||||
AAPT2 dump, structured CLI facts, parent APK identity and every retained manifest
|
||||
source location. It deliberately keeps resource declarations unresolved even
|
||||
though the resource table contains values. `manifest-oracle.py` is a separate
|
||||
stdlib ElementTree oracle used by the public MCP task driver. Generated binaries,
|
||||
Evidence and dumps stay outside Git.
|
||||
@@ -1,70 +0,0 @@
|
||||
"""Independent stdlib XML oracle over retained manifests in this verification corpus."""
|
||||
import json
|
||||
import sys
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
ANDROID = "{http://schemas.android.com/apk/res/android}"
|
||||
KINDS = {"activity", "activity-alias", "service", "receiver", "provider"}
|
||||
DATA_ATTRIBUTES = {
|
||||
"scheme": "scheme", "host": "host", "port": "port", "path": "path",
|
||||
"path_prefix": "pathPrefix", "path_pattern": "pathPattern",
|
||||
"path_advanced_pattern": "pathAdvancedPattern", "path_suffix": "pathSuffix",
|
||||
"mime_type": "mimeType",
|
||||
}
|
||||
|
||||
|
||||
def filter_facts(element):
|
||||
def names(tag):
|
||||
return [value.get(ANDROID + "name") for value in element.findall(tag)
|
||||
if value.get(ANDROID + "name") is not None]
|
||||
|
||||
return {
|
||||
"priority": element.get(ANDROID + "priority"),
|
||||
"actions": names("action"), "categories": names("category"),
|
||||
"data": [{key: value.get(ANDROID + attribute)
|
||||
for key, attribute in DATA_ATTRIBUTES.items()}
|
||||
for value in element.findall("data")],
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
with open(sys.argv[1], encoding="utf-8") as source:
|
||||
bundle = json.load(source)
|
||||
parent = next(record for record in bundle["records"] if record["evidence_id"] == sys.argv[2])
|
||||
root = ET.fromstring(parent["normalized_result"]["manifest"]["text"])
|
||||
package = root.get("package")
|
||||
application = root.find("application")
|
||||
assert application is not None and package, "Verification corpus requires a known package/application"
|
||||
inventory = []
|
||||
ordinal = 0
|
||||
for component in application:
|
||||
if component.tag not in KINDS:
|
||||
continue
|
||||
declared = component.get(ANDROID + "name")
|
||||
if declared is None:
|
||||
continue
|
||||
filters = [filter_facts(value) for value in component.findall("intent-filter")]
|
||||
enabled = component.get(ANDROID + "enabled")
|
||||
permission = component.get(ANDROID + "permission")
|
||||
if permission is None and component.tag != "activity-alias":
|
||||
permission = application.get(ANDROID + "permission")
|
||||
name = (package + declared if declared.startswith(".") else
|
||||
package + "." + declared if "." not in declared else declared)
|
||||
inventory.append({
|
||||
"source_index": ordinal, "kind": component.tag,
|
||||
"declared_name": declared, "name": name,
|
||||
"enabled": True if enabled == "true" else False if enabled == "false" else None,
|
||||
"enabled_declaration": enabled, "process": component.get(ANDROID + "process"),
|
||||
"permission": permission or None, "intent_filters": filters,
|
||||
"meta_data": [{"name": value.get(ANDROID + "name")}
|
||||
for value in component.findall("meta-data")],
|
||||
})
|
||||
ordinal += 1
|
||||
selected = [row for row in inventory
|
||||
if any(sys.argv[3] in value["actions"] for value in row["intent_filters"])]
|
||||
print(json.dumps({"components": selected, "inventory": inventory, "source_components": ordinal},
|
||||
ensure_ascii=False, separators=(",", ":")))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,4 +0,0 @@
|
||||
<resources>
|
||||
<bool name="receiver_enabled">false</bool>
|
||||
<integer name="filter_priority">80</integer>
|
||||
</resources>
|
||||
@@ -1,58 +0,0 @@
|
||||
"""Check real AAPT2/JADX/CLI facts against the source-built manifest corpus."""
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
root = Path(sys.argv[1])
|
||||
reply = json.loads((root / "cli.json").read_text())["summary"]
|
||||
view = reply["normalized_result"]
|
||||
bundle = json.loads((root / "evidence.json").read_text())
|
||||
parent = next(record for record in bundle["records"]
|
||||
if record["evidence_id"] == view["parent_evidence_id"])
|
||||
sha256 = hashlib.sha256((root / "manifest.apk").read_bytes()).hexdigest()
|
||||
assert parent["subject"]["digest"]["sha256"] == view["artifact"]["sha256"] == sha256
|
||||
assert reply["evidence_links"] == [parent["evidence_id"]]
|
||||
assert view["source_status"] == "complete" and view["source_components"] == 3
|
||||
assert view["coverage"]["exhausted"] and view["coverage"]["total"] == 3
|
||||
activity, receiver, provider = view["items"]
|
||||
assert [row["source_index"] for row in view["items"]] == [0, 1, 2]
|
||||
assert all(row["declaration_coverage"] == "complete" for row in view["items"])
|
||||
assert activity["enabled"] is False and activity["process"] == ":activity"
|
||||
first, second = activity["intent_filters"]
|
||||
assert first["actions"] == ["android.intent.action.VIEW"] and first["priority"] == "7"
|
||||
assert first["data"][0]["scheme"] == "https"
|
||||
assert first["data"][0]["host"] == "example.test" and first["data"][0]["path_prefix"] == "/open"
|
||||
assert second["actions"] == ["android.intent.action.EDIT"]
|
||||
assert second["priority"] == "@integer/filter_priority"
|
||||
assert second["data"][0]["scheme"] == "content" and second["data"][0]["mime_type"] == "text/plain"
|
||||
assert activity["meta_data"][0]["name"] == "public.configuration"
|
||||
assert "value" not in activity["meta_data"][0]
|
||||
assert receiver["enabled"] is None and receiver["enabled_declaration"] == "@bool/receiver_enabled"
|
||||
assert receiver["process"] == ":sms" and receiver["permission"] == "android.permission.BROADCAST_SMS"
|
||||
assert receiver["intent_filters"][0]["priority"] == "-10"
|
||||
assert provider["enabled"] is True and provider["grant_uri_permissions"] is True
|
||||
assert provider["grant_uri_permissions_declaration"] == "true" and provider["process"] is None
|
||||
manifest = parent["normalized_result"]["manifest"]["text"]
|
||||
assert manifest.isascii() and "retained-in-manifest" in manifest
|
||||
for row in view["items"]:
|
||||
nodes = [(row, row["kind"])]
|
||||
for element in row["intent_filters"]:
|
||||
nodes.append((element, "intent-filter"))
|
||||
nodes.extend((data, "data") for data in element["data"])
|
||||
nodes.extend((metadata, "meta-data") for metadata in row["meta_data"])
|
||||
for node, tag in nodes:
|
||||
location = node["location"]
|
||||
offset = location["offset"]
|
||||
assert manifest[offset:].startswith("<" + tag)
|
||||
assert manifest[:offset].count("\n") + 1 == location["line"]
|
||||
assert offset - manifest.rfind("\n", 0, offset) == location["column"]
|
||||
tree = (root / "aapt2-xmltree.txt").read_text()
|
||||
for fragment in [':scheme(0x01010027)="https"', ':host(0x01010028)="example.test"',
|
||||
':pathPrefix(0x0101002b)="/open"', ':priority(0x0101001c)=7',
|
||||
':priority(0x0101001c)=-10', ':priority(0x0101001c)=@0x7f020000',
|
||||
':enabled(0x0101000e)=@0x7f010000', ':grantUriPermissions(0x0101001b)=true']:
|
||||
assert fragment in tree, fragment
|
||||
print(json.dumps({"compiled_manifest_cli": "passed", "sha256": sha256,
|
||||
"components": 3, "filter_grouping": True, "locations": True,
|
||||
"resource_values_remain_declarations": True, "aapt2_oracle": True}))
|
||||
@@ -1,30 +0,0 @@
|
||||
<!doctype html>
|
||||
<meta charset="utf-8" />
|
||||
<title>Immediate saved-response follow-up fixture</title>
|
||||
<button id="run">Exchange and follow up</button>
|
||||
<output id="result">pending</output>
|
||||
<output id="sensitive"></output>
|
||||
<script>
|
||||
document.querySelector("#run").onclick = async () => {
|
||||
const source = await (
|
||||
await fetch("/escaped-values.json", { cache: "no-store" })
|
||||
).json();
|
||||
document.querySelector("#sensitive").textContent = source.token;
|
||||
console.log(source.token);
|
||||
const response = await fetch(
|
||||
"/api/collections/rea_items/records?token=" +
|
||||
encodeURIComponent(source.token),
|
||||
{
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
"X-Followup": source.token,
|
||||
},
|
||||
body: JSON.stringify({ token: source.token }),
|
||||
},
|
||||
);
|
||||
const result = document.querySelector("#result");
|
||||
result.textContent = String(response.status);
|
||||
result.dataset.ready = "yes";
|
||||
};
|
||||
</script>
|
||||
@@ -1,39 +0,0 @@
|
||||
<!doctype html><meta charset="utf-8" /><title>
|
||||
PocketBase saved value verifier
|
||||
</title>
|
||||
<label>Email<input id="email" /></label
|
||||
><label>Password<input type="password" id="password" /></label
|
||||
><button id="login">Sign in</button><button id="create">Create record</button>
|
||||
<pre id="result"></pre>
|
||||
<script>
|
||||
document.querySelector("#login").onclick = async () => {
|
||||
const r = await fetch("/api/collections/_superusers/auth-with-password", {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
identity: document.querySelector("#email").value,
|
||||
password: document.querySelector("#password").value,
|
||||
}),
|
||||
});
|
||||
const data = await r.json();
|
||||
localStorage.setItem("rea_auth", JSON.stringify(data));
|
||||
document.querySelector("#result").textContent = r.ok
|
||||
? "Signed in"
|
||||
: "Sign in failed";
|
||||
if (r.ok) document.querySelector("#result").dataset.ready = "yes";
|
||||
};
|
||||
document.querySelector("#create").onclick = async () => {
|
||||
const token = JSON.parse(localStorage.getItem("rea_auth") || "{}").token;
|
||||
const r = await fetch("/api/collections/rea_items/records", {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
...(token ? { Authorization: token } : {}),
|
||||
},
|
||||
body: JSON.stringify({ title: "rea-ui-record" }),
|
||||
});
|
||||
document.querySelector("#result").textContent = JSON.stringify(
|
||||
await r.json(),
|
||||
);
|
||||
};
|
||||
</script>
|
||||
@@ -1,15 +0,0 @@
|
||||
<!doctype html>
|
||||
<meta charset="utf-8" />
|
||||
<title>Declared JSON payload privacy fixture</title>
|
||||
<output id="result">pending</output>
|
||||
<script>
|
||||
fetch("/api/collections/rea_items/records", {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: localStorage.getItem("whole_json"),
|
||||
}).then((response) => {
|
||||
const result = document.querySelector("#result");
|
||||
result.textContent = String(response.status);
|
||||
result.dataset.ready = "yes";
|
||||
});
|
||||
</script>
|
||||
@@ -1,160 +0,0 @@
|
||||
# Dart AOT summary and retained string pools
|
||||
|
||||
Continues [#2016](https://github.com/morluto/rea/issues/2016) and
|
||||
[#2455](https://github.com/morluto/rea/pull/2455). The selected analyst question
|
||||
asks for shipped LocalSend target/ABI/library identity, every nullable snapshot
|
||||
observation, string-pool lower bounds and returned counts, and original coverage
|
||||
and limitations. It does not require individual observed strings.
|
||||
|
||||
## Compatibility and ownership
|
||||
|
||||
MCP `inspect_dart_aot` accepts explicit `detail: "summary"`. Omitted and complete
|
||||
requests preserve the complete result and original provider inputs. The CLI
|
||||
remains complete. ABI omission and explicit ABI selection retain their original
|
||||
meaning; the provider never receives the delivery field. Missing retention,
|
||||
invalid detail and pre-cancellation fail before provider work.
|
||||
|
||||
The existing application workflow retains the complete original Evidence, then
|
||||
returns a distinct derived summary linked to its parent ID. It preserves target
|
||||
and libapp digests, every snapshot section and nullable fact, original pool and
|
||||
overall coverage, lower-bound counts and limitations. `returned_counts` reports
|
||||
the lengths of the producer-returned arrays; it does not upgrade lower bounds
|
||||
to a complete snapshot inventory. The analysis and original source retention
|
||||
are unchanged. No new engine, persistent cache or plugin SDK is introduced.
|
||||
|
||||
Read exact retained strings using `inspect_analysis_view`, the parent ID, and
|
||||
`view: {kind: "dart-aot", collection: "package_uris", offset: 0, limit: 64}`.
|
||||
Other collections are `dart_uris` and `class_like_tokens`. Follow the returned
|
||||
next offset. Pages preserve source order, ABI/library identity and partial pool
|
||||
coverage without opening the APK again. Page exhaustion describes retained
|
||||
strings, not the entire snapshot. Strings remain byte-pattern observations,
|
||||
rather than recovered typed class/function semantics. Missing historical
|
||||
subjects remain explicitly unknown; conflicting recorded target digests fail.
|
||||
|
||||
## Frozen runtime and real SDK pairs
|
||||
|
||||
The official LocalSend 1.18.2 arm64 APK acquisition, size and SHA-256 are pinned
|
||||
in `corpus.json`. No APK installation, target execution or version-dataset
|
||||
lookup occurred. Source runtime: `811034a7283c9751a3ecbff33e8b2f28bec55793`;
|
||||
compiled dist/bridge file-table SHA-256:
|
||||
`dabbb63536c5862be839ecdc0f3eaad05b29c2a3816f57041528d30f754311be`.
|
||||
The previous runtime is `28c84e78cf2fbfde4aceed345b773c59eb889f68`, digest
|
||||
`5332dda13981f043e146b3601ebec1f380d9f698280ca53d75786b638a912e11`.
|
||||
Its production sources were verified byte-equal to main f254 at acquisition.
|
||||
Subsequent test-only changes and main's SAZ addition do not relabel these pins.
|
||||
|
||||
Node is 24.18.0 and the SDK client/server are 2.3.1. Each workflow starts a fresh
|
||||
server and validates the selected live advertised schema. Artifact bytes,
|
||||
provider environment/search-path fingerprints, driver and locked dependencies
|
||||
stay equal. Host page caches are not flushed. Heavy tasks run serially under
|
||||
the unchanged CPU, memory, task and host guards.
|
||||
|
||||
| Trial | AOT reply bytes | SDK workflow seconds | Payload CPU seconds | Kernel peak bytes |
|
||||
| -------------- | --------------: | -------------------: | ------------------: | ----------------: |
|
||||
| sdk-complete-1 | 151,341 | 5.139 | 6.819 | 691,441,664 |
|
||||
| sdk-summary-1 | 12,037 | 5.219 | 6.608 | 639,430,656 |
|
||||
| sdk-summary-2 | 12,037 | 5.064 | 6.405 | 640,868,352 |
|
||||
| sdk-complete-2 | 151,341 | 5.128 | 6.468 | 633,614,336 |
|
||||
|
||||
Both pairs use one successful public tool call and pass settings/fact comparison,
|
||||
including all snapshot fields, coverage, original limitations and all counts.
|
||||
Reply bytes fall 92.05%. Whole SDK time stays approximately 5.1 seconds; these
|
||||
trials do not establish engine speed, CPU or memory savings. Payload CPU includes
|
||||
driver/catalog work beyond the SDK workflow timer. Catalog JSON grows from
|
||||
2,506,213 bytes on the previous runtime to 2,514,896 (about 0.35%).
|
||||
|
||||
A fresh old-runtime complete request passes comparison against candidate
|
||||
complete, including identical 151,341-byte replies. This is scoped compatibility
|
||||
evidence, not a guarantee for every historical release or third-party client.
|
||||
|
||||
## Real old Evidence continuation
|
||||
|
||||
Fifteen successful public MCP calls import genuine old-runtime AOT Evidence,
|
||||
project retained summary, reconstruct all 500 package URIs, 216 Dart URIs and
|
||||
2,000 class-like tokens, compare inline portable summary, and export the original
|
||||
record. The APK is absent during all these operations. Original string order,
|
||||
lower bounds, partial coverage, nullable snapshot facts, subject and parent ID
|
||||
remain intact; the original exported record is deeply equal to its source.
|
||||
|
||||
The first private verification attempt used a nonexistent receipt filename and
|
||||
failed before import or artifact relocation. That driver error is retained;
|
||||
the corrected attempt selects the fresh old-runtime record. It is not a REA
|
||||
defect or a successful migration trial.
|
||||
|
||||
## Actual Codex task outcomes
|
||||
|
||||
Four serial trials use the same frozen candidate in complete/summary/summary/
|
||||
complete order. Requested configuration is Codex CLI 0.161.0, `gpt-5.6-terra`
|
||||
high, one exact enabled tool (`inspect_dart_aot`), and identical optional owned
|
||||
context capture/discovery guidance. This requested alias does not certify the
|
||||
server-resolved model. Temporary configuration points to the modified frozen
|
||||
REA MCP; all copied credential profiles are removed and target bytes remain
|
||||
unchanged. The source question is fixed; equal treatment instructions choose
|
||||
detail and explain the summary carrier and parent reference.
|
||||
|
||||
| Trial | Input | Cached | Uncached | Output | Seconds | MCP attempts | Failed attempts | Facts |
|
||||
| ---------------- | -----: | -----: | -------: | -----: | ------: | -----------: | --------------: | ----- |
|
||||
| model-complete-1 | 74,641 | 50,176 | 24,465 | 2,765 | 116.898 | 3 | 1 | pass |
|
||||
| model-summary-1 | 38,253 | 22,016 | 16,237 | 1,449 | 88.904 | 1 | 0 | pass |
|
||||
| model-summary-2 | 38,363 | 22,016 | 16,347 | 1,515 | 55.190 | 1 | 0 | pass |
|
||||
| model-complete-2 | 66,585 | 38,144 | 28,441 | 2,217 | 79.180 | 2 | 0 | pass |
|
||||
|
||||
Both pairs pass checks for every selected producer fact, exact nullable
|
||||
observations, target digest, observed response Evidence IDs, original limitations,
|
||||
artifact preservation and profile cleanup. The grader compares the reported
|
||||
facts with producer observations; it is not an independent Dart deserializer.
|
||||
Total input falls 48.75% / 42.38%, uncached input 33.63% / 42.52%, reported output
|
||||
47.59% / 31.66%, and whole-task time 23.95% / 30.30%. Usage is the final aggregate
|
||||
reported by the actual CLI, one usage record per trial; cached tokens are not
|
||||
subtracted from total input. Reported output includes reasoning output (1,262,
|
||||
321, 383 and 709 tokens respectively).
|
||||
|
||||
The first complete trial initially supplied `artifact_path` instead of the
|
||||
advertised `path`, corrected it, then repeated a successful complete call.
|
||||
The second complete trial made two successful calls. Both summary trials made
|
||||
one successful call. All attempts, discovery and correction costs remain
|
||||
included. The paired whole-task improvement therefore includes actual model
|
||||
decisions and the convenience of explicit returned counts; it does not isolate
|
||||
serialization as the sole cause or establish engine acceleration. Two pairs on
|
||||
one app/question/model configuration support this scope, not a universal gain.
|
||||
|
||||
Payload CPU seconds / kernel peak bytes are respectively 12.556 / 876,937,216;
|
||||
9.213 / 785,272,832; 9.108 / 793,939,968; and 13.164 / 803,209,216. Model payload
|
||||
observations verify owned cleanup and zero phase limit events, with no guard
|
||||
aborts. Whole-scope final counters remain unknown, so every strict all-selected
|
||||
resource grade remains false even though the separate factual grade passes.
|
||||
|
||||
## Resource and verification scope
|
||||
|
||||
SDK and migration payload-exit observations verify owned cleanup and zero
|
||||
observed memory/pids limit events, with peaks below 692 MB. Missing whole-scope
|
||||
terminal counters remain unknown; the strict whole-resource grader stays false.
|
||||
One-CPU quotas are experiment controls rather than CPU-efficiency claims.
|
||||
|
||||
Production-provider fixture tests cover explicit ABI, old complete defaults,
|
||||
actual SDK-converted schemas, exact pages after APK removal, portable records,
|
||||
nullable/partial facts, malformed inputs, target digest conflicts and early
|
||||
refusals. CLI/MCP acceptance covers original report equality, duplicate-member
|
||||
refusals and unsupported snapshot kinds. Fixture successes are distinguished
|
||||
from the real LocalSend SDK/migration workflows. Other Flutter apps, architectures,
|
||||
SDK layouts, typed class recovery and runtime behavior remain unverified here.
|
||||
Raw replies, binaries, model traces and temporary credentials stay outside Git.
|
||||
|
||||
## Integration with the newer main
|
||||
|
||||
The rebased verification runtime is `63984ce0898bd7997e6d9b138e6d96d0a24f2b09`
|
||||
on main `a2d8b94dd50dab8d966239dd63ab84718aa8b4b4`, with compiled file-table
|
||||
SHA-256 `19c8ccdd3e99bb46a5c54f4edd8c6b2d2eb4f3a4870b3c129061d38145382b68`.
|
||||
Targeted Flutter/SAZ/shared-view integration passes 35 tests; one opposite
|
||||
runtime-capacity branch is skipped because this host enforces decoder memory.
|
||||
Documentation checks and two README conformance tests pass.
|
||||
|
||||
Fresh real complete/summary SDK checks on this runtime match the earlier selected
|
||||
facts and settings, with unchanged 151,341 / 12,037-byte AOT replies. Fifteen
|
||||
old-Evidence migration calls again preserve every original string and record
|
||||
with the source APK absent. The real one-shot CLI complete result also matches
|
||||
the previous-runtime normalized report, subject, parameters and original ID.
|
||||
An initial CLI-only controller check lacked an owned resource receipt; it is
|
||||
retained and excluded from resource/performance claims, then repeated in the
|
||||
bounded owned scope. These checks establish integration, rather than relabeling
|
||||
the earlier paired model/SDK measurements as the rebased runtime.
|
||||
@@ -1,81 +0,0 @@
|
||||
"""Count complete actual-client requests, not only smaller tool-reply bytes."""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import tiktoken
|
||||
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--lab', type=Path, required=True)
|
||||
parser.add_argument('--output', type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
enc = tiktoken.get_encoding('cl100k_base')
|
||||
|
||||
|
||||
def tokens(value):
|
||||
return len(enc.encode(json.dumps(value, ensure_ascii=False, separators=(',', ':'), allow_nan=False), disallowed_special=()))
|
||||
|
||||
|
||||
def resources(root):
|
||||
receipt = json.loads(Path(str(root) + '.log').read_text().splitlines()[-1])
|
||||
payload = receipt['payload_resource_observation']
|
||||
assert receipt['abort'] is None and receipt['payload_resource_observation_verified']
|
||||
assert payload['payload_exit_code'] == 0 and payload['payload_cleanup_verified']
|
||||
for events in payload['resource_events'].values():
|
||||
assert not any(events.values())
|
||||
return {'wall_seconds': receipt['elapsed_seconds'], 'cpu_seconds': payload['cpu_stat']['usage_usec'] / 1e6,
|
||||
'family_peak_mib': payload['memory_peak_bytes'] / 1048576,
|
||||
'payload_cleanup_verified': True, 'final_whole_scope_counters_verified': receipt['resource_counters_verified']}
|
||||
|
||||
|
||||
wire, sdk = [], []
|
||||
for root in sorted(args.lab.glob('wire-v2-*')):
|
||||
if not root.is_dir():
|
||||
continue
|
||||
report = json.loads((root / 'report.json').read_text())
|
||||
assert report['status'] in ['passed', 'expected-incomplete-task'] and not report['cleanup_failures']
|
||||
assert report['appserver_exit_code'] == 0 and report['owned_scope_drain']['remaining_processes'] == []
|
||||
requests = json.loads((root / 'model-requests.json').read_text())
|
||||
details = []
|
||||
for captured in requests:
|
||||
body = captured['body']
|
||||
namespace = [n for n in body.get('tools', []) if n.get('type') == 'namespace' and n.get('name') == 'mcp__rea_audit']
|
||||
assert len(namespace) == 1 and len(namespace[0]['tools']) == 211, 'Unexpected compaction/catalog: audit before comparing'
|
||||
details.append({'whole_json_tokens_proxy': tokens(body), 'rea_namespace_tokens_proxy': tokens(namespace[0]), 'rea_functions': 211})
|
||||
wire.append({'case': root.name, 'task_completed': report['task_completed'], 'mode': report['mode'],
|
||||
'budget': report['observations']['tool_output_budget'], 'requests': len(details), 'compactions': report['compaction_requests'],
|
||||
'whole_json_tokens_proxy': sum(r['whole_json_tokens_proxy'] for r in details), 'request_details': details, **resources(root)})
|
||||
for root in sorted(args.lab.glob('sdk-*')):
|
||||
if not root.is_dir():
|
||||
continue
|
||||
report = json.loads((root / 'report.json').read_text())
|
||||
assert report['status'] == 'passed' and not report['cleanup_failures'] and not report['receipt_failures']
|
||||
transcript = json.loads((root / 'transcript.json').read_text())
|
||||
catalog = json.loads((root / 'catalog.json').read_text())
|
||||
assert len(catalog['tools']) == 211 and report['original_target_exited']
|
||||
sdk.append({'case': root.name, 'detail': report['detail'], 'capture': report['capture'], 'tool_calls': len(report['calls']),
|
||||
'exported_bundle_bytes': report['exported_bundle_bytes'], 'answer': report['answer'],
|
||||
'status_calls': [c for c in report['calls'] if c['name'] == 'frida_session_status'],
|
||||
'one_catalog_plus_all_sdk_calls_tokens_proxy': tokens(catalog) + sum(tokens({'name': t['name'], 'input': t['input'], 'reply': t['reply']}) for t in transcript), **resources(root)})
|
||||
assert len(sdk) == len(wire) == 4, 'A partial cohort cannot qualify a comparison'
|
||||
for capture in ['small', 'libc']:
|
||||
pair = [r for r in sdk if r['capture'] == capture]
|
||||
assert len(pair) == 2 and {r['detail'] for r in pair} == {'complete', 'summary'}
|
||||
assert {k: v for k, v in pair[0]['answer'].items() if k != 'pid'} == {k: v for k, v in pair[1]['answer'].items() if k != 'pid'}
|
||||
reports = [json.loads((args.lab / r['case'] / 'report.json').read_text()) for r in pair]
|
||||
assert reports[0]['load']['source_sha256'] == reports[1]['load']['source_sha256']
|
||||
completed = [r for r in wire if r['budget'] == 300000]
|
||||
assert len(completed) == 2 and all(r['task_completed'] for r in completed)
|
||||
answers = [json.loads((args.lab / r['case'] / 'report.json').read_text())['observations']['answers'] for r in completed]
|
||||
comparable_answers = [[{k: v for k, v in a.items() if k != 'parent_evidence_id'} for a in run] for run in answers]
|
||||
assert comparable_answers[0] == comparable_answers[1]
|
||||
result = {
|
||||
'scope': 'Real local Frida binding with controlled sleep target and caller-authored libc export inventory. Actual Codex0.163 app-server uses a deterministic local SSE fixture: no AI inference, model usage, quality or billing claim. Whole compact JSON token estimates include all 211 schemas and every request replay. Incomplete default baseline is excluded from completed-task gains. SDK exchange estimates are not model-input measurements. Fresh processes have unflushed OS caches; both status reads query live session observations. Timings include the eight-second target lifetime and cleanup; single pairs establish no general speedup. Payload-exit cgroup counters are verified; final whole-scope counters may be unavailable.',
|
||||
'tokenizer': {'package': tiktoken.__version__, 'encoding': enc.name},
|
||||
'runtime_pins': {v: {k: x for k, x in json.loads((args.lab / (v + '-manifest.json')).read_text()).items() if k != 'files'} for v in ['baseline', 'candidate']},
|
||||
'native': json.loads((args.lab / 'native-prerequisite-manifest.json').read_text()),
|
||||
'script_pins': {p.name: hashlib.sha256(p.read_bytes()).hexdigest() for p in Path(__file__).parent.glob('frida*') if p.suffix in ['.py', '.mjs']},
|
||||
'wire': wire, 'sdk': sdk,
|
||||
}
|
||||
args.output.write_text(json.dumps(result, indent=2) + '\n')
|
||||
print(json.dumps({'wire': [{k: v for k, v in r.items() if k != 'request_details'} for r in wire], 'sdk': [{k: v for k, v in r.items() if k not in ['status_calls', 'answer']} for r in sdk]}, indent=2))
|
||||
@@ -1,267 +0,0 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { createHash } from "node:crypto";
|
||||
import { createWriteStream } from "node:fs";
|
||||
import { mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises";
|
||||
import { createRequire } from "node:module";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { finished } from "node:stream/promises";
|
||||
import { parseArgs } from "node:util";
|
||||
import { createRetainedQueryCaller } from "./retained-query-calls.mjs";
|
||||
import { finalizeRetainedQueryReceipt } from "./retained-query-finalization.mjs";
|
||||
const { values } = parseArgs({
|
||||
options: {
|
||||
runtime: { type: "string" },
|
||||
output: { type: "string" },
|
||||
detail: { type: "string", default: "complete" },
|
||||
capture: { type: "string", default: "libc" },
|
||||
},
|
||||
});
|
||||
const { runtime, output, detail, capture } = values;
|
||||
assert.ok(runtime && output);
|
||||
assert.ok(["complete", "summary"].includes(detail));
|
||||
assert.ok(["libc", "small"].includes(capture));
|
||||
const require = createRequire(join(runtime, "package.json"));
|
||||
const { Client } = await import(
|
||||
pathToFileURL(require.resolve("@modelcontextprotocol/client"))
|
||||
);
|
||||
const { StdioClientTransport } = await import(
|
||||
pathToFileURL(require.resolve("@modelcontextprotocol/client/stdio"))
|
||||
);
|
||||
const { parseLinuxProcessStat } = await import(
|
||||
pathToFileURL(join(runtime, "dist/process/LinuxProcessStat.js"))
|
||||
);
|
||||
await mkdir(output, { mode: 0o700 });
|
||||
const account = await mkdtemp(join(tmpdir(), "rea-frida-real-"));
|
||||
const serverLog = createWriteStream(join(output, "server.log"));
|
||||
const serverLogFinished = finished(serverLog);
|
||||
serverLogFinished.catch(() => {});
|
||||
const client = new Client({ name: "rea-frida-audit", version: "1" });
|
||||
const transport = new StdioClientTransport({
|
||||
command: process.execPath,
|
||||
args: [join(runtime, "scripts/rea.mjs"), "mcp"],
|
||||
env: {
|
||||
PATH: process.env.PATH,
|
||||
HOME: account,
|
||||
NODE_OPTIONS: "--max-old-space-size=2048",
|
||||
UV_THREADPOOL_SIZE: "1",
|
||||
REA_LOG_LEVEL: "silent",
|
||||
},
|
||||
stderr: "pipe",
|
||||
});
|
||||
transport.stderr?.pipe(serverLog, { end: false });
|
||||
const report = {
|
||||
question:
|
||||
"Retain a complete caller-requested libc export inventory, then check only session identity/state and message capture coverage twice, preserving complete Evidence.",
|
||||
detail,
|
||||
capture,
|
||||
calls: [],
|
||||
status: "pending",
|
||||
};
|
||||
const transcript = [];
|
||||
const caller = createRetainedQueryCaller(client, (receipt, event) => {
|
||||
report.calls.push(receipt);
|
||||
transcript.push(event);
|
||||
});
|
||||
const call = async (name, input) => {
|
||||
const r = await caller(name, input, name);
|
||||
assert.ok(!r.isError, JSON.stringify(r));
|
||||
return r.structuredContent;
|
||||
};
|
||||
const delay = (ms) => new Promise((r) => setTimeout(r, ms));
|
||||
let session;
|
||||
let birth;
|
||||
const stat = async () => {
|
||||
try {
|
||||
return parseLinuxProcessStat(
|
||||
await readFile(`/proc/${session.pid}/stat`, "utf8"),
|
||||
);
|
||||
} catch (e) {
|
||||
if (e.code === "ENOENT") return null;
|
||||
throw e;
|
||||
}
|
||||
};
|
||||
try {
|
||||
await client.connect(transport);
|
||||
const catalog = await client.listTools();
|
||||
await writeFile(join(output, "catalog.json"), JSON.stringify(catalog));
|
||||
report.catalog_tools = catalog.tools.length;
|
||||
const defs = new Map(catalog.tools.map((t) => [t.name, t]));
|
||||
const orig = client.callTool.bind(client);
|
||||
client.callTool = (input, options) =>
|
||||
orig(input, {
|
||||
...options,
|
||||
timeout: 30000,
|
||||
toolDefinition: defs.get(input.name),
|
||||
});
|
||||
report.devices = await call("list_frida_devices", {});
|
||||
session = await call("start_frida_session", {
|
||||
mode: "spawn",
|
||||
device_id: "local",
|
||||
program: "/usr/bin/sleep",
|
||||
argv: ["8"],
|
||||
});
|
||||
report.session = session;
|
||||
birth = await stat();
|
||||
assert.ok(birth);
|
||||
report.target_birth = birth;
|
||||
report.same_owned_cgroup =
|
||||
(await readFile(`/proc/${session.pid}/cgroup`, "utf8")) ===
|
||||
(await readFile("/proc/self/cgroup", "utf8"));
|
||||
assert.ok(report.same_owned_cgroup);
|
||||
const source = `send({kind:'ready',pid:Process.id});
|
||||
setTimeout(() => {
|
||||
if (${JSON.stringify(capture)} === 'small') { send({kind:'complete',total:0}); return; }
|
||||
const module = Process.enumerateModules().find(m => /(^|\\/)libc\\.so\\.6$/.test(m.path));
|
||||
if (!module) { send({kind:'missing-libc'}); return; }
|
||||
const exports = module.enumerateExports().map(e => ({type:e.type,name:e.name,address:e.address.toString()}));
|
||||
send({kind:'module',pid:Process.id,path:module.path,base:module.base.toString(),size:module.size,total:exports.length});
|
||||
for (let offset = 0; offset < exports.length; offset += 64) send({kind:'exports',offset,items:exports.slice(offset,offset+64)});
|
||||
send({kind:'complete',total:exports.length});
|
||||
}, 300);`;
|
||||
await writeFile(join(output, "selected-script.js"), source);
|
||||
report.source_sha256 = createHash("sha256").update(source).digest("hex");
|
||||
const loaded = await call("load_frida_script", {
|
||||
session_id: session.session_id,
|
||||
source_kind: "inline",
|
||||
source,
|
||||
});
|
||||
assert.equal(loaded.normalized_result.source_sha256, report.source_sha256);
|
||||
report.load = loaded.normalized_result;
|
||||
await call("resume_frida_session", { session_id: session.session_id });
|
||||
await delay(750);
|
||||
const snapshots = [];
|
||||
for (let n = 0; n < 2; n++)
|
||||
snapshots.push(
|
||||
await call("frida_session_status", {
|
||||
session_id: session.session_id,
|
||||
...(detail === "summary" ? { detail } : {}),
|
||||
}),
|
||||
);
|
||||
await call("export_evidence_bundle", { path: join(output, "evidence.json") });
|
||||
const bundle = JSON.parse(
|
||||
await readFile(join(output, "evidence.json"), "utf8"),
|
||||
);
|
||||
for (const snapshot of snapshots) {
|
||||
assert.equal(snapshot.raw_result, null);
|
||||
const parent =
|
||||
detail === "summary"
|
||||
? bundle.records.find(
|
||||
(e) =>
|
||||
e.evidence_id === snapshot.normalized_result.parent_evidence_id,
|
||||
)
|
||||
: snapshot;
|
||||
assert.ok(parent);
|
||||
assert.equal(parent.authority, "external-service");
|
||||
assert.equal(parent.environment, null);
|
||||
assert.equal(parent.raw_result, null);
|
||||
assert.equal(parent.operation, "frida_session_status");
|
||||
assert.deepEqual(parent.parameters, { session_id: session.session_id });
|
||||
const r = parent.normalized_result;
|
||||
assert.equal(r.pid, session.pid);
|
||||
assert.equal(r.state, "running");
|
||||
assert.equal(r.messages_truncated, false);
|
||||
const payloads = r.messages
|
||||
.filter((m) => m.type === "send")
|
||||
.map((m) => m.payload);
|
||||
const ready = payloads.find((p) => p?.kind === "ready");
|
||||
const complete = payloads.find((p) => p?.kind === "complete");
|
||||
assert.equal(ready?.pid, session.pid);
|
||||
assert.ok(complete);
|
||||
let count = 0;
|
||||
let header;
|
||||
if (capture === "libc") {
|
||||
header = payloads.find((p) => p?.kind === "module");
|
||||
assert.ok(header);
|
||||
assert.equal(header.pid, session.pid);
|
||||
for (const chunk of payloads.filter((p) => p?.kind === "exports")) {
|
||||
assert.equal(chunk.offset, count);
|
||||
count += chunk.items.length;
|
||||
}
|
||||
assert.equal(count, header.total);
|
||||
assert.ok(count > 1000);
|
||||
const maps = await readFile(`/proc/${session.pid}/maps`, "utf8");
|
||||
assert.ok(maps.includes(header.path));
|
||||
}
|
||||
assert.equal(count, complete.total);
|
||||
if (detail === "summary") {
|
||||
const view = snapshot.normalized_result;
|
||||
assert.equal(view.parent_operation, parent.operation);
|
||||
assert.equal(view.parent_digest, parent.evidence_id.slice(3));
|
||||
assert.equal(view.artifact, null);
|
||||
assert.deepEqual(view.source_parameters, parent.parameters);
|
||||
assert.deepEqual(view.source_provider, parent.provider);
|
||||
const { messages, ...metadata } = r;
|
||||
assert.deepEqual(view.summary, {
|
||||
...metadata,
|
||||
messages_retained: messages.length,
|
||||
});
|
||||
assert.ok(view.unknowns.length > 0);
|
||||
assert.ok(view.limitations.some((m) => m.includes("operating system")));
|
||||
assert.deepEqual(
|
||||
bundle.records.find((e) => e.evidence_id === snapshot.evidence_id),
|
||||
snapshot,
|
||||
);
|
||||
}
|
||||
report.answer = {
|
||||
pid: r.pid,
|
||||
state: r.state,
|
||||
scripts: r.scripts.length,
|
||||
messages: r.messages.length,
|
||||
messages_truncated: r.messages_truncated,
|
||||
exports: count,
|
||||
module_path: header?.path ?? null,
|
||||
};
|
||||
}
|
||||
report.native_oracle =
|
||||
"Caller-authored script digest and contiguous chunk coverage; PID/birthtime/cgroup and libc path independently checked in procfs. Export addresses and native export completeness not independently qualified.";
|
||||
report.exported_bundle_bytes = Buffer.byteLength(
|
||||
await readFile(join(output, "evidence.json")),
|
||||
);
|
||||
await call("close_frida_session", { session_id: session.session_id });
|
||||
session.closed = true;
|
||||
const deadline = performance.now() + 12000;
|
||||
while (true) {
|
||||
const current = await stat();
|
||||
if (
|
||||
current === null ||
|
||||
current.startTime !== birth.startTime ||
|
||||
["zombie", "dead"].includes(current.lifecycle)
|
||||
) {
|
||||
report.original_target_exited = true;
|
||||
break;
|
||||
}
|
||||
assert.ok(
|
||||
performance.now() < deadline,
|
||||
"Original target exit not observed",
|
||||
);
|
||||
await delay(50);
|
||||
}
|
||||
report.status = "passed";
|
||||
} catch (error) {
|
||||
report.status = "failed";
|
||||
report.failure = String(error?.stack ?? error);
|
||||
process.exitCode = 1;
|
||||
} finally {
|
||||
if (session && !session.closed)
|
||||
try {
|
||||
await call("close_frida_session", { session_id: session.session_id });
|
||||
} catch (error) {
|
||||
report.session_cleanup_failure = String(error);
|
||||
process.exitCode = 1;
|
||||
}
|
||||
if (
|
||||
!(await finalizeRetainedQueryReceipt({
|
||||
report,
|
||||
client,
|
||||
transport,
|
||||
account,
|
||||
serverLog,
|
||||
serverLogFinished,
|
||||
output,
|
||||
transcript,
|
||||
}))
|
||||
)
|
||||
process.exitCode = 1;
|
||||
}
|
||||
@@ -1,468 +0,0 @@
|
||||
{
|
||||
"scope": "Real local Frida binding with controlled sleep target and caller-authored libc export inventory. Actual Codex0.163 app-server uses a deterministic local SSE fixture: no AI inference, model usage, quality or billing claim. Whole compact JSON token estimates include all 211 schemas and every request replay. Incomplete default baseline is excluded from completed-task gains. SDK exchange estimates are not model-input measurements. Fresh processes have unflushed OS caches; both status reads query live session observations. Timings include the eight-second target lifetime and cleanup; single pairs establish no general speedup. Payload-exit cgroup counters are verified; final whole-scope counters may be unavailable.",
|
||||
"tokenizer": {
|
||||
"package": "0.12.0",
|
||||
"encoding": "cl100k_base"
|
||||
},
|
||||
"runtime_pins": {
|
||||
"baseline": {
|
||||
"source_base": "17f581ff75d2b6cf643a997275e3e296f6c95760",
|
||||
"patch_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
|
||||
"archive_sha256": "cd16ca3e36633459bffc10bb90d1fe4ff5887fdd403729765b9a812969013807"
|
||||
},
|
||||
"candidate": {
|
||||
"source_base": "32a4690059907f09fb29e284ad492d94c1722278",
|
||||
"patch_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
|
||||
"archive_sha256": "4ac402f1de93f76f71b7ccdb8b8e36d8c8d0a87b1184812f06393bfc6fadccba"
|
||||
}
|
||||
},
|
||||
"native": {
|
||||
"host": "Linux-6.1.0-53-cloud-amd64-x86_64-with-glibc2.36",
|
||||
"toolchain": {
|
||||
"node": "24.18.0",
|
||||
"npm": "11.16.0"
|
||||
},
|
||||
"frida_binding_package_version": "17.23.2",
|
||||
"files": [
|
||||
{
|
||||
"path": "/usr/bin/sleep",
|
||||
"bytes": 43888,
|
||||
"sha256": "4add4bb89d8ca0e3b1bd861130ddd7ae0fd9617a8055de0a38c8d2ca1ac95723"
|
||||
},
|
||||
{
|
||||
"path": "/usr/lib/x86_64-linux-gnu/libc.so.6",
|
||||
"bytes": 1926232,
|
||||
"sha256": "6b4a45352fd0c540a9c7c718f35ce8c8e46a4e482f9d3885a910c32d1a0e1421"
|
||||
},
|
||||
{
|
||||
"path": "/home/yann/yann/rea-perf-next-audit-20261020/node_modules/frida/package.json",
|
||||
"bytes": 2252,
|
||||
"sha256": "7efb59d2050b7b6304e1617f463c4eb7b2f7cb7e3e80e677976825a76ee31e8d"
|
||||
},
|
||||
{
|
||||
"path": "/home/yann/yann/rea-perf-next-audit-20261020/node_modules/frida/build/frida_binding.node",
|
||||
"bytes": 112775824,
|
||||
"sha256": "5b2db5ae57c058ad0765e627d9d9c9c98e4e90c09746c6f727b7a80769eed7a5"
|
||||
}
|
||||
],
|
||||
"qualifications": {
|
||||
"Inspector": "real REA MCP target, concurrent native metadata only oracle, 776 rows matched hash/length; not global completeness or execution proof",
|
||||
"Frida": "local native binding and controlled sleep spawn; caller script digest and procfs PID/birthtime/cgroup/libc path verified; native export address/global completeness oracle unavailable",
|
||||
"ADB": "default executable unavailable in prior public call; no real device qualified"
|
||||
}
|
||||
},
|
||||
"script_pins": {
|
||||
"frida_status_delivery.py": "ed19bcd003165c7b522c8cdd4799bcf987178e4b79fc8d5c2442012f3bf60b8d",
|
||||
"frida-status-costs.py": "2f3196bcc3198a265f7f4842effc8dce31669723b42f07367f065afee0073906",
|
||||
"frida-status-queries.mjs": "c5919ae8bba046909858e92a945b4472645bbacbfcf88dd75c89ba959a986980",
|
||||
"frida-status-wire.py": "b4d20e64bf77b7815bd43b52569b3e4756983b254ab44c50ddae874d1964ef69"
|
||||
},
|
||||
"wire": [
|
||||
{
|
||||
"case": "wire-v2-baseline-0",
|
||||
"task_completed": false,
|
||||
"mode": "complete",
|
||||
"budget": "client default",
|
||||
"requests": 8,
|
||||
"compactions": 0,
|
||||
"whole_json_tokens_proxy": 540065,
|
||||
"request_details": [
|
||||
{
|
||||
"whole_json_tokens_proxy": 62729,
|
||||
"rea_namespace_tokens_proxy": 53518,
|
||||
"rea_functions": 211
|
||||
},
|
||||
{
|
||||
"whole_json_tokens_proxy": 62917,
|
||||
"rea_namespace_tokens_proxy": 53518,
|
||||
"rea_functions": 211
|
||||
},
|
||||
{
|
||||
"whole_json_tokens_proxy": 63599,
|
||||
"rea_namespace_tokens_proxy": 53518,
|
||||
"rea_functions": 211
|
||||
},
|
||||
{
|
||||
"whole_json_tokens_proxy": 63739,
|
||||
"rea_namespace_tokens_proxy": 53518,
|
||||
"rea_functions": 211
|
||||
},
|
||||
{
|
||||
"whole_json_tokens_proxy": 68261,
|
||||
"rea_namespace_tokens_proxy": 53518,
|
||||
"rea_functions": 211
|
||||
},
|
||||
{
|
||||
"whole_json_tokens_proxy": 72785,
|
||||
"rea_namespace_tokens_proxy": 53518,
|
||||
"rea_functions": 211
|
||||
},
|
||||
{
|
||||
"whole_json_tokens_proxy": 72926,
|
||||
"rea_namespace_tokens_proxy": 53518,
|
||||
"rea_functions": 211
|
||||
},
|
||||
{
|
||||
"whole_json_tokens_proxy": 73109,
|
||||
"rea_namespace_tokens_proxy": 53518,
|
||||
"rea_functions": 211
|
||||
}
|
||||
],
|
||||
"wall_seconds": 15.993,
|
||||
"cpu_seconds": 8.211179,
|
||||
"family_peak_mib": 830.6484375,
|
||||
"payload_cleanup_verified": true,
|
||||
"final_whole_scope_counters_verified": false
|
||||
},
|
||||
{
|
||||
"case": "wire-v2-baseline-300000",
|
||||
"task_completed": true,
|
||||
"mode": "complete",
|
||||
"budget": 300000,
|
||||
"requests": 8,
|
||||
"compactions": 0,
|
||||
"whole_json_tokens_proxy": 1049997,
|
||||
"request_details": [
|
||||
{
|
||||
"whole_json_tokens_proxy": 62715,
|
||||
"rea_namespace_tokens_proxy": 53518,
|
||||
"rea_functions": 211
|
||||
},
|
||||
{
|
||||
"whole_json_tokens_proxy": 62903,
|
||||
"rea_namespace_tokens_proxy": 53518,
|
||||
"rea_functions": 211
|
||||
},
|
||||
{
|
||||
"whole_json_tokens_proxy": 63577,
|
||||
"rea_namespace_tokens_proxy": 53518,
|
||||
"rea_functions": 211
|
||||
},
|
||||
{
|
||||
"whole_json_tokens_proxy": 63714,
|
||||
"rea_namespace_tokens_proxy": 53518,
|
||||
"rea_functions": 211
|
||||
},
|
||||
{
|
||||
"whole_json_tokens_proxy": 141108,
|
||||
"rea_namespace_tokens_proxy": 53518,
|
||||
"rea_functions": 211
|
||||
},
|
||||
{
|
||||
"whole_json_tokens_proxy": 218504,
|
||||
"rea_namespace_tokens_proxy": 53518,
|
||||
"rea_functions": 211
|
||||
},
|
||||
{
|
||||
"whole_json_tokens_proxy": 218646,
|
||||
"rea_namespace_tokens_proxy": 53518,
|
||||
"rea_functions": 211
|
||||
},
|
||||
{
|
||||
"whole_json_tokens_proxy": 218830,
|
||||
"rea_namespace_tokens_proxy": 53518,
|
||||
"rea_functions": 211
|
||||
}
|
||||
],
|
||||
"wall_seconds": 15.418,
|
||||
"cpu_seconds": 8.026944,
|
||||
"family_peak_mib": 824.28125,
|
||||
"payload_cleanup_verified": true,
|
||||
"final_whole_scope_counters_verified": false
|
||||
},
|
||||
{
|
||||
"case": "wire-v2-candidate-0",
|
||||
"task_completed": true,
|
||||
"mode": "summary",
|
||||
"budget": "client default",
|
||||
"requests": 8,
|
||||
"compactions": 0,
|
||||
"whole_json_tokens_proxy": 515539,
|
||||
"request_details": [
|
||||
{
|
||||
"whole_json_tokens_proxy": 62791,
|
||||
"rea_namespace_tokens_proxy": 53581,
|
||||
"rea_functions": 211
|
||||
},
|
||||
{
|
||||
"whole_json_tokens_proxy": 62978,
|
||||
"rea_namespace_tokens_proxy": 53581,
|
||||
"rea_functions": 211
|
||||
},
|
||||
{
|
||||
"whole_json_tokens_proxy": 63661,
|
||||
"rea_namespace_tokens_proxy": 53581,
|
||||
"rea_functions": 211
|
||||
},
|
||||
{
|
||||
"whole_json_tokens_proxy": 63803,
|
||||
"rea_namespace_tokens_proxy": 53581,
|
||||
"rea_functions": 211
|
||||
},
|
||||
{
|
||||
"whole_json_tokens_proxy": 64749,
|
||||
"rea_namespace_tokens_proxy": 53581,
|
||||
"rea_functions": 211
|
||||
},
|
||||
{
|
||||
"whole_json_tokens_proxy": 65697,
|
||||
"rea_namespace_tokens_proxy": 53581,
|
||||
"rea_functions": 211
|
||||
},
|
||||
{
|
||||
"whole_json_tokens_proxy": 65837,
|
||||
"rea_namespace_tokens_proxy": 53581,
|
||||
"rea_functions": 211
|
||||
},
|
||||
{
|
||||
"whole_json_tokens_proxy": 66023,
|
||||
"rea_namespace_tokens_proxy": 53581,
|
||||
"rea_functions": 211
|
||||
}
|
||||
],
|
||||
"wall_seconds": 15.889,
|
||||
"cpu_seconds": 7.939387,
|
||||
"family_peak_mib": 823.69921875,
|
||||
"payload_cleanup_verified": true,
|
||||
"final_whole_scope_counters_verified": false
|
||||
},
|
||||
{
|
||||
"case": "wire-v2-candidate-300000",
|
||||
"task_completed": true,
|
||||
"mode": "summary",
|
||||
"budget": 300000,
|
||||
"requests": 8,
|
||||
"compactions": 0,
|
||||
"whole_json_tokens_proxy": 515713,
|
||||
"request_details": [
|
||||
{
|
||||
"whole_json_tokens_proxy": 62804,
|
||||
"rea_namespace_tokens_proxy": 53581,
|
||||
"rea_functions": 211
|
||||
},
|
||||
{
|
||||
"whole_json_tokens_proxy": 62990,
|
||||
"rea_namespace_tokens_proxy": 53581,
|
||||
"rea_functions": 211
|
||||
},
|
||||
{
|
||||
"whole_json_tokens_proxy": 63677,
|
||||
"rea_namespace_tokens_proxy": 53581,
|
||||
"rea_functions": 211
|
||||
},
|
||||
{
|
||||
"whole_json_tokens_proxy": 63815,
|
||||
"rea_namespace_tokens_proxy": 53581,
|
||||
"rea_functions": 211
|
||||
},
|
||||
{
|
||||
"whole_json_tokens_proxy": 64772,
|
||||
"rea_namespace_tokens_proxy": 53581,
|
||||
"rea_functions": 211
|
||||
},
|
||||
{
|
||||
"whole_json_tokens_proxy": 65731,
|
||||
"rea_namespace_tokens_proxy": 53581,
|
||||
"rea_functions": 211
|
||||
},
|
||||
{
|
||||
"whole_json_tokens_proxy": 65870,
|
||||
"rea_namespace_tokens_proxy": 53581,
|
||||
"rea_functions": 211
|
||||
},
|
||||
{
|
||||
"whole_json_tokens_proxy": 66054,
|
||||
"rea_namespace_tokens_proxy": 53581,
|
||||
"rea_functions": 211
|
||||
}
|
||||
],
|
||||
"wall_seconds": 15.439,
|
||||
"cpu_seconds": 8.371664,
|
||||
"family_peak_mib": 813.6328125,
|
||||
"payload_cleanup_verified": true,
|
||||
"final_whole_scope_counters_verified": false
|
||||
}
|
||||
],
|
||||
"sdk": [
|
||||
{
|
||||
"case": "sdk-baseline-libc",
|
||||
"detail": "complete",
|
||||
"capture": "libc",
|
||||
"tool_calls": 8,
|
||||
"exported_bundle_bytes": 214214,
|
||||
"answer": {
|
||||
"pid": 517427,
|
||||
"state": "running",
|
||||
"scripts": 1,
|
||||
"messages": 50,
|
||||
"messages_truncated": false,
|
||||
"exports": 2963,
|
||||
"module_path": "/usr/lib/x86_64-linux-gnu/libc.so.6"
|
||||
},
|
||||
"status_calls": [
|
||||
{
|
||||
"name": "frida_session_status",
|
||||
"input": {
|
||||
"session_id": "ef8982d3-dbf2-43cb-8256-3a4f4bc83cc3"
|
||||
},
|
||||
"phase": "frida_session_status",
|
||||
"elapsed_ms": 197.58100300000024,
|
||||
"reply_bytes": 462831,
|
||||
"is_error": false
|
||||
},
|
||||
{
|
||||
"name": "frida_session_status",
|
||||
"input": {
|
||||
"session_id": "ef8982d3-dbf2-43cb-8256-3a4f4bc83cc3"
|
||||
},
|
||||
"phase": "frida_session_status",
|
||||
"elapsed_ms": 108.50572199999988,
|
||||
"reply_bytes": 462831,
|
||||
"is_error": false
|
||||
}
|
||||
],
|
||||
"one_catalog_plus_all_sdk_calls_tokens_proxy": 1186065,
|
||||
"wall_seconds": 13.759,
|
||||
"cpu_seconds": 6.097489,
|
||||
"family_peak_mib": 578.5625,
|
||||
"payload_cleanup_verified": true,
|
||||
"final_whole_scope_counters_verified": false
|
||||
},
|
||||
{
|
||||
"case": "sdk-baseline-small",
|
||||
"detail": "complete",
|
||||
"capture": "small",
|
||||
"tool_calls": 8,
|
||||
"exported_bundle_bytes": 2335,
|
||||
"answer": {
|
||||
"pid": 516970,
|
||||
"state": "running",
|
||||
"scripts": 1,
|
||||
"messages": 2,
|
||||
"messages_truncated": false,
|
||||
"exports": 0,
|
||||
"module_path": null
|
||||
},
|
||||
"status_calls": [
|
||||
{
|
||||
"name": "frida_session_status",
|
||||
"input": {
|
||||
"session_id": "3f1f778f-ca18-48ba-a7db-b58afbf4710d"
|
||||
},
|
||||
"phase": "frida_session_status",
|
||||
"elapsed_ms": 79.25232200000028,
|
||||
"reply_bytes": 2643,
|
||||
"is_error": false
|
||||
},
|
||||
{
|
||||
"name": "frida_session_status",
|
||||
"input": {
|
||||
"session_id": "3f1f778f-ca18-48ba-a7db-b58afbf4710d"
|
||||
},
|
||||
"phase": "frida_session_status",
|
||||
"elapsed_ms": 6.851230000000214,
|
||||
"reply_bytes": 2643,
|
||||
"is_error": false
|
||||
}
|
||||
],
|
||||
"one_catalog_plus_all_sdk_calls_tokens_proxy": 882323,
|
||||
"wall_seconds": 13.039,
|
||||
"cpu_seconds": 4.953337,
|
||||
"family_peak_mib": 577.140625,
|
||||
"payload_cleanup_verified": true,
|
||||
"final_whole_scope_counters_verified": false
|
||||
},
|
||||
{
|
||||
"case": "sdk-candidate-libc",
|
||||
"detail": "summary",
|
||||
"capture": "libc",
|
||||
"tool_calls": 8,
|
||||
"exported_bundle_bytes": 216931,
|
||||
"answer": {
|
||||
"pid": 517901,
|
||||
"state": "running",
|
||||
"scripts": 1,
|
||||
"messages": 50,
|
||||
"messages_truncated": false,
|
||||
"exports": 2963,
|
||||
"module_path": "/usr/lib/x86_64-linux-gnu/libc.so.6"
|
||||
},
|
||||
"status_calls": [
|
||||
{
|
||||
"name": "frida_session_status",
|
||||
"input": {
|
||||
"session_id": "9af7f188-05f8-4559-b570-98556196090e",
|
||||
"detail": "summary"
|
||||
},
|
||||
"phase": "frida_session_status",
|
||||
"elapsed_ms": 159.16798299999937,
|
||||
"reply_bytes": 5597,
|
||||
"is_error": false
|
||||
},
|
||||
{
|
||||
"name": "frida_session_status",
|
||||
"input": {
|
||||
"session_id": "9af7f188-05f8-4559-b570-98556196090e",
|
||||
"detail": "summary"
|
||||
},
|
||||
"phase": "frida_session_status",
|
||||
"elapsed_ms": 61.96135099999992,
|
||||
"reply_bytes": 5597,
|
||||
"is_error": false
|
||||
}
|
||||
],
|
||||
"one_catalog_plus_all_sdk_calls_tokens_proxy": 885626,
|
||||
"wall_seconds": 12.843,
|
||||
"cpu_seconds": 5.012703,
|
||||
"family_peak_mib": 579.0,
|
||||
"payload_cleanup_verified": true,
|
||||
"final_whole_scope_counters_verified": false
|
||||
},
|
||||
{
|
||||
"case": "sdk-candidate-small",
|
||||
"detail": "summary",
|
||||
"capture": "small",
|
||||
"tool_calls": 8,
|
||||
"exported_bundle_bytes": 5051,
|
||||
"answer": {
|
||||
"pid": 517162,
|
||||
"state": "running",
|
||||
"scripts": 1,
|
||||
"messages": 2,
|
||||
"messages_truncated": false,
|
||||
"exports": 0,
|
||||
"module_path": null
|
||||
},
|
||||
"status_calls": [
|
||||
{
|
||||
"name": "frida_session_status",
|
||||
"input": {
|
||||
"session_id": "1bcda0f5-e3cd-45ee-b82c-5886d8312bc6",
|
||||
"detail": "summary"
|
||||
},
|
||||
"phase": "frida_session_status",
|
||||
"elapsed_ms": 107.78244100000029,
|
||||
"reply_bytes": 5595,
|
||||
"is_error": false
|
||||
},
|
||||
{
|
||||
"name": "frida_session_status",
|
||||
"input": {
|
||||
"session_id": "1bcda0f5-e3cd-45ee-b82c-5886d8312bc6",
|
||||
"detail": "summary"
|
||||
},
|
||||
"phase": "frida_session_status",
|
||||
"elapsed_ms": 5.996030000000246,
|
||||
"reply_bytes": 5595,
|
||||
"is_error": false
|
||||
}
|
||||
],
|
||||
"one_catalog_plus_all_sdk_calls_tokens_proxy": 885676,
|
||||
"wall_seconds": 13.533,
|
||||
"cpu_seconds": 5.495048,
|
||||
"family_peak_mib": 576.48046875,
|
||||
"payload_cleanup_verified": true,
|
||||
"final_whole_scope_counters_verified": false
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,163 +0,0 @@
|
||||
# Frida status delivery and runtime-observation audit
|
||||
|
||||
Issue: [#5640](https://github.com/morluto/rea/issues/5640). Paired runtime base:
|
||||
`17f581ff75d2b6cf643a997275e3e296f6c95760`. Final candidate source: `32a4690059907f09fb29e284ad492d94c1722278` (clean tree).
|
||||
Exact package, patch, native input,
|
||||
script and resource pins are in [the measurement record](frida-status-views.json).
|
||||
|
||||
The analyst requests a complete libc export inventory as retained evidence,
|
||||
then checks the session's identity/state, script identity, queued message count
|
||||
and truncation twice. The caller-authored script enumerates actual native module
|
||||
exports in a controlled `/usr/bin/sleep` spawn, in chunks of 64. Both workflows
|
||||
retain all captured messages and export the complete Evidence bundle.
|
||||
|
||||
Start, status and one-shot contracts share one canonical session envelope.
|
||||
Message JSON is validated at the Evidence owner; the summary validates the
|
||||
envelope without reinterpreting opaque payloads.
|
||||
|
||||
The owning application service now accepts optional `detail=summary` for
|
||||
`frida_session_status`. It reads the provider on every call, retains the original
|
||||
complete Evidence, then uses the existing shared analysis-view projector.
|
||||
The summary omits arbitrary message payloads and preserves all other status
|
||||
fields, the retained message count, truncation, exact parent identity/digest,
|
||||
provider and source parameters. Default and explicit complete delivery keep the
|
||||
original Evidence identity and payload. Historical CLI/file projections use the
|
||||
same projector. Source script decoding/representation Evidence and session
|
||||
resource ownership are unchanged.
|
||||
|
||||
## Completed tasks
|
||||
|
||||
The native Frida Node binding package 17.23.2 captured 2,963 libc exports in 50 queued
|
||||
messages. Independent checks establish the test-owned process birthtime, cgroup,
|
||||
PID and libc mapping; the selected script digest and contiguous chunks are
|
||||
checked against the complete retained parent. This does not independently
|
||||
validate every native export address or establish global event coverage.
|
||||
|
||||
| Public SDK task | Complete | Summary |
|
||||
| ------------------------------------------ | -------: | ------: |
|
||||
| Large-history status reply, bytes per call | 462,831 | 5,597 |
|
||||
| Small-history status reply, bytes per call | 2,643 | 5,595 |
|
||||
| Large complete bundle, bytes | 214,214 | 216,931 |
|
||||
| Small complete bundle, bytes | 2,335 | 5,051 |
|
||||
| Large task elapsed, seconds | 13.759 | 12.843 |
|
||||
| Large task CPU, seconds | 6.097 | 5.013 |
|
||||
| Large family peak, MiB | 578.6 | 579.0 |
|
||||
| Small task elapsed, seconds | 13.039 | 13.533 |
|
||||
| Small task CPU, seconds | 4.953 | 5.495 |
|
||||
| Small family peak, MiB | 577.1 | 576.5 |
|
||||
|
||||
Each SDK task discovers all 211 tools and executes eight public calls, including
|
||||
two status reads and complete export. Small history contains two readiness/
|
||||
completion messages. Fresh processes retain unflushed OS caches. The second
|
||||
status read is a repeated live observation, not a cache hit. Every task includes
|
||||
the controlled target's eight-second lifetime and verified natural exit.
|
||||
Single pairs and this fixed wait establish no general latency/CPU advantage.
|
||||
Large status calls took 159.2/62.0 ms versus 197.6/108.5 ms. The first small
|
||||
summary status was slower (107.8 versus 79.3 ms).
|
||||
|
||||
## Actual client context
|
||||
|
||||
Codex app-server 0.163.0 performs the native Frida workflow through the actual
|
||||
REA MCP package. A deterministic local Responses/SSE fixture selects the calls;
|
||||
there is no AI inference, paid model, model-quality or billing claim. The capture
|
||||
contains all 211 tool input schemas on each request and all history replays. `tiktoken`
|
||||
0.12.0 `cl100k_base` counts compact JSON as a token estimate, not actual model
|
||||
usage. Both completed paths use the same 300,000 tool-output budget.
|
||||
|
||||
| Completed client task | Complete | Summary |
|
||||
| -------------------------------------- | --------: | ------: |
|
||||
| Requests / compactions | 8 / 0 | 8 / 0 |
|
||||
| Whole request JSON token estimate, sum | 1,049,997 | 515,713 |
|
||||
| Elapsed seconds | 15.418 | 15.439 |
|
||||
| CPU seconds | 8.027 | 8.372 |
|
||||
| Family peak MiB | 824.3 | 813.6 |
|
||||
|
||||
The completed task uses **50.9% fewer estimated request tokens**, including
|
||||
schemas and continuations. Client elapsed and CPU time increased in this pair. At the client's
|
||||
default output budget, complete status JSON is truncated and the task is
|
||||
incomplete; summary status is intact and completes. That failed delivery is
|
||||
excluded from completed-task gains. Three acceptance tests exercise invalid
|
||||
JSON, incorrect counts/state/source linkage, missing chunks and wrong sessions.
|
||||
|
||||
The complete catalog plus all SDK exchanges also shrinks for the large case
|
||||
(1,186,065 to 885,626 estimated tokens), but grows for the small case (882,323 to
|
||||
885,676). These SDK exchange estimates include advertised output schemas and
|
||||
are **not** actual model-input measurements. Summary delivery is optional
|
||||
because its provenance wrapper costs more than a small complete observation.
|
||||
|
||||
## Resource accounting and limits
|
||||
|
||||
Runtime cases use one CPU, a 4 GiB cgroup ceiling, a 3 GiB early stop, no swap,
|
||||
192 tasks, and host-memory admission/early-stop guards. Payload exit, descendant
|
||||
cleanup and zero resource-limit/OOM events are verified while the scope observer
|
||||
remains alive. Final whole-scope counters disappear after teardown and remain
|
||||
unknown. Runtime family peaks stay below 1 GiB. Build/typecheck use separately
|
||||
guarded larger limits; they are not runtime performance measurements.
|
||||
|
||||
Full queued messages still undergo validation, Evidence creation and retention;
|
||||
this change primarily reduces delivery and repeated context. The extra derived
|
||||
record increases retained bundle size. Long-lived heap retention, many-script
|
||||
sessions, remote devices, binary messages and saturated capture budgets are not
|
||||
performance-qualified by these native cases. Boundary tests preserve malformed
|
||||
input, truncation, cancellation and cleanup semantics; simulated providers do
|
||||
not qualify real remote behavior.
|
||||
|
||||
## Other providers in this audit
|
||||
|
||||
Inspector was measured against a test-owned real REA MCP process while it
|
||||
handled a verified GitHub CLI Go build-information request. A concurrent native
|
||||
CDP metadata subscription matched all 776 retained scripts by hash and reported
|
||||
length: 565 file locations and 211 built-ins. The reply was 608,719 bytes;
|
||||
capture retained 996/996 events with no reported drop/truncation. The qualification
|
||||
run took 20.956 seconds, 18.076 CPU seconds and 987.4 MiB peak family memory,
|
||||
including the independent oracle. This is not a paired performance cohort,
|
||||
global completeness proof, source-byte validation or evidence of script
|
||||
execution. Only `Runtime.enable` and `Debugger.enable` were sent. A prior
|
||||
post-capture oracle missed 25 historical rows and was rejected; it does not
|
||||
establish an Inspector regression. Existing raw-null delivery, grouped location
|
||||
validation and eight-worker projection were already present and were preserved.
|
||||
|
||||
ADB's configured default executable is unavailable on this host, and no real
|
||||
device is qualified. This audit makes no ADB device-performance claim. Inspector
|
||||
selection/caching and Frida queue/filter redesign remain unimplemented without
|
||||
an equivalent completed-task measurement that justifies them.
|
||||
|
||||
The existing bundled Frida verifier also returned functional success, but its
|
||||
1,500 ms fixture outlived the verifier's 500 ms observation. Its payload-exit
|
||||
cleanup receipt was therefore unverified and is excluded from qualified
|
||||
resource/performance results. That unchanged verifier has no process birthtime
|
||||
stamp to safely wait on an arbitrary later PID. The paired harness instead owns
|
||||
the paused spawn and records birthtime/cgroup before resume, then waits for the
|
||||
original target's natural exit. No PID-only cleanup or target-kill policy was
|
||||
introduced.
|
||||
|
||||
The shared-service import also exposed an existing module-mock load-order
|
||||
assumption in the Evidence-file cancellation test. That one mocked test now
|
||||
runs in the existing isolated `process-global` lane, with an assertion that the
|
||||
mocked read actually ran. Other file/service tests keep the shared module graph.
|
||||
The final exact ten-file gate passes 225 tests with file parallelism disabled
|
||||
under one CPU. An earlier parallel gate hit one existing 30-second MCP lifecycle
|
||||
timeout; its serial retry passed without increasing a timeout or changing the
|
||||
production lifecycle contract.
|
||||
|
||||
## Reproduction
|
||||
|
||||
Use the pinned Node/npm toolchain and native Frida dependency. Freeze separate
|
||||
baseline/candidate built packages and install the same locked dependencies.
|
||||
Run each case independently with the resource observer:
|
||||
|
||||
```sh
|
||||
python3 experiments/analysis-efficiency/resource-run.py --observe-payload-exit -- \
|
||||
node experiments/analysis-efficiency/frida-status-queries.mjs \
|
||||
--runtime /absolute/built-package --output /absolute/new-case \
|
||||
--capture libc --detail summary
|
||||
```
|
||||
|
||||
Repeat with `--detail complete` and `--capture small`. For actual-client captures,
|
||||
`frida-status-wire.py --help` names the runtime, pinned Codex and Node executables,
|
||||
output directory and selected script file (`selected-script.js` from the SDK
|
||||
case). Use both `--mode complete`/`summary` and `--budget 300000`/`0`.
|
||||
`frida-status-costs.py` checks task/resource receipts and counts complete requests.
|
||||
The first wire attempt used ambiguous suffix matching for load/unload names;
|
||||
the fixture now selects exact operation names. Failed attempts remain outside
|
||||
the performance cohort. No production behavior changed to accommodate them.
|
||||
@@ -1,299 +0,0 @@
|
||||
"""Actual Codex wire capture with a deterministic local SSE backend; no AI inference."""
|
||||
import argparse
|
||||
import http.server
|
||||
import threading
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import time
|
||||
import traceback
|
||||
import subprocess
|
||||
from frida_status_delivery import verify_frida_delivery
|
||||
|
||||
|
||||
|
||||
async def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument("--runtime", type=Path, required=True)
|
||||
parser.add_argument("--codex", type=Path, required=True)
|
||||
parser.add_argument("--node", type=Path, required=True)
|
||||
parser.add_argument("--input-profile", choices=["full", "compact"], default="full")
|
||||
parser.add_argument("--output-profile", choices=["full", "off"], default="full")
|
||||
parser.add_argument("--script", type=Path, required=True)
|
||||
parser.add_argument("--expected-tools", type=int, default=211)
|
||||
parser.add_argument("--mode", choices=["complete", "summary"], required=True)
|
||||
parser.add_argument("--budget", type=int, default=30000)
|
||||
args = parser.parse_args()
|
||||
source=args.script.read_text()
|
||||
assert args.runtime.is_absolute() and args.output.is_absolute() and args.script.is_absolute()
|
||||
def delivered(pieces):
|
||||
assert isinstance(pieces,str) and pieces.startswith("Wall time: "), pieces[:300]
|
||||
_,sep,payload=pieces.partition("Output:\n");assert sep
|
||||
try: value=json.loads(payload)
|
||||
except json.JSONDecodeError: return {"_invalid_json_payload":payload}
|
||||
return value["result"] if "result" in value and "normalized_result" not in value else value
|
||||
def outputs_of(body):
|
||||
return {item["call_id"]:delivered(item["output"]) for item in body.get("input",[]) if item.get("type")=="function_call_output"}
|
||||
steps=7
|
||||
args.output.mkdir(mode=0o700)
|
||||
audit_home = args.output / "account"
|
||||
codex_state = audit_home / ".codex"
|
||||
mcp_home = audit_home / "mcp"
|
||||
workspace = audit_home / "workspace"
|
||||
for directory in [codex_state, mcp_home, workspace]:
|
||||
directory.mkdir(parents=True)
|
||||
captured = []
|
||||
known_outputs = {}
|
||||
state = {"normal_requests":0,"compactions":0}
|
||||
class Capture(http.server.BaseHTTPRequestHandler):
|
||||
def log_message(self, *_):
|
||||
pass
|
||||
def do_POST(self):
|
||||
size = int(self.headers["Content-Length"])
|
||||
assert 0 < size <= 16 * 1024 * 1024
|
||||
assert self.headers.get("Content-Encoding", "identity") == "identity"
|
||||
body = json.loads(self.rfile.read(size))
|
||||
captured.append({"path": self.path, "body": body})
|
||||
(args.output / "model-requests.json").write_text(json.dumps(captured) + "\n")
|
||||
http_index = len(captured)
|
||||
known_outputs.update(outputs_of(body))
|
||||
names = [t.get("name", "") for ns in body.get("tools", []) if ns.get("type") == "namespace" and ns.get("name") == "mcp__rea_audit" for t in ns.get("tools", [])]
|
||||
if not names:
|
||||
last=body.get('input',[])[-1]
|
||||
assert 'CONTEXT CHECKPOINT COMPACTION' in json.dumps(last)
|
||||
state['compactions']+=1;assert state['compactions']<=1
|
||||
checkpoint={'fixture':'deterministic observed-ID checkpoint, no AI inference','task':'Check Frida session state, queued message count and truncation twice; retain/export complete observations and close instrumentation.','session_id':known_outputs['call_local_1']['session_id'],'delivered_call_ids':list(known_outputs),'remaining':'Continue scripted tools; source message bodies remain in retained Evidence.'}
|
||||
text=json.dumps(checkpoint)
|
||||
item={"type":"message","id":f"msg_checkpoint_{state['compactions']}","role":"assistant","status":"completed","content":[{"type":"output_text","text":text,"annotations":[]}]}
|
||||
captured[-1]['kind']='deterministic-compaction'
|
||||
else:
|
||||
state['normal_requests']+=1;index=state['normal_requests'];assert index<=steps+1
|
||||
captured[-1]['kind']='tool-planning-or-final'
|
||||
if index<=steps:
|
||||
if index==1:
|
||||
operation='start_frida_session';arguments={'mode':'spawn','device_id':'local','program':'/usr/bin/sleep','argv':['8']}
|
||||
else:
|
||||
session=known_outputs['call_local_1']['session_id']
|
||||
if index==2:
|
||||
operation='load_frida_script';arguments={'session_id':session,'source_kind':'inline','source':source}
|
||||
elif index==3:
|
||||
operation='resume_frida_session';arguments={'session_id':session}
|
||||
elif index in [4,5]:
|
||||
if index==4: time.sleep(0.75)
|
||||
operation='frida_session_status';arguments={'session_id':session,**({'detail':'summary'} if args.mode=='summary' else {})}
|
||||
elif index==6:
|
||||
operation='close_frida_session';arguments={'session_id':session}
|
||||
else:
|
||||
operation='export_evidence_bundle';arguments={'path':str(args.output/'evidence.json')}
|
||||
selected=[name for name in names if name == operation];assert len(selected)==1
|
||||
item={"type":"function_call","id":f"fc_local_{index}","call_id":f"call_local_{index}","name":selected[0],"namespace":"mcp__rea_audit","arguments":json.dumps(arguments),"status":"completed"}
|
||||
else:
|
||||
item={"type":"message","id":"msg_local_done","role":"assistant","status":"completed","content":[{"type":"output_text","text":"Scripted local Frida state and capture-coverage task complete.","annotations":[]}]}
|
||||
response_id=f"resp_local_{http_index}"
|
||||
base = {"id": response_id, "object": "response", "model": body.get("model"), "status": "completed", "output": [item]}
|
||||
events = [
|
||||
{"type": "response.created", "response": {**base, "status": "in_progress", "output": []}},
|
||||
{"type": "response.output_item.added", "output_index": 0, "item": {**item, "status": "in_progress"}},
|
||||
{"type": "response.output_item.done", "output_index": 0, "item": item},
|
||||
{"type": "response.completed", "response": base},
|
||||
]
|
||||
wire = "".join("event: " + e["type"] + "\ndata: " + json.dumps(e) + "\n\n" for e in events).encode()
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "text/event-stream")
|
||||
self.send_header("Content-Length", str(len(wire)))
|
||||
self.end_headers()
|
||||
self.wfile.write(wire)
|
||||
endpoint = http.server.HTTPServer(("127.0.0.1", 0), Capture)
|
||||
endpoint_thread = threading.Thread(target=endpoint.serve_forever, daemon=True)
|
||||
endpoint_thread.start()
|
||||
task_path = f"{args.node.parent}:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
|
||||
mcp_env = {"HOME": str(mcp_home), "PATH": task_path,
|
||||
"NODE_OPTIONS": "--max-old-space-size=2048", "UV_THREADPOOL_SIZE": "1",
|
||||
"REA_LOG_LEVEL": "silent", "REA_MCP_INPUT_SCHEMA_PROFILE": args.input_profile,
|
||||
"REA_MCP_OUTPUT_SCHEMA_PROFILE": args.output_profile}
|
||||
config = 'model_provider = "rea_capture"\nmodel = "rea-wire-probe"\n[analytics]\nenabled = false\n[mcp_servers.rea_audit]\n'
|
||||
config += f'command = {json.dumps(str(args.node))}\n'
|
||||
config += f'args = {json.dumps([str(args.runtime / "scripts/rea.mjs"), "mcp"])}\n'
|
||||
config += 'startup_timeout_sec = 60\ntool_timeout_sec = 60\n'
|
||||
config += '[mcp_servers.rea_audit.env]\n'
|
||||
config += ''.join(f'{key} = {json.dumps(value)}\n' for key, value in mcp_env.items())
|
||||
for name in ['start_frida_session','load_frida_script','resume_frida_session','frida_session_status','close_frida_session','export_evidence_bundle']:
|
||||
config += f'[mcp_servers.rea_audit.tools.{name}]\napproval_mode = "approve"\n'
|
||||
if args.budget: config += f'output_token_limit = {args.budget}\n'
|
||||
config += f'\n[model_providers.rea_capture]\nname = "Local scripted capture"\nbase_url = "http://127.0.0.1:{endpoint.server_port}/v1"\nwire_api = "responses"\nrequires_openai_auth = false\nsupports_websockets = false\nrequest_max_retries = 0\nstream_max_retries = 0\n'
|
||||
(codex_state / "config.toml").write_text(config)
|
||||
(args.output / "declared-config.toml").write_text(config)
|
||||
environment = {"HOME": str(audit_home), "CODEX_HOME": str(codex_state), "PATH": task_path,
|
||||
"NODE_OPTIONS": "--max-old-space-size=2048", "UV_THREADPOOL_SIZE": "1"}
|
||||
report = {"question": "Retain a caller-requested native libc export inventory, check session identity/state, queued message count and truncation twice, close instrumentation and export complete Evidence.",
|
||||
"client": "Codex app-server 0.163.0", "model_turns_started": 1, "backend": "local deterministic SSE fixture; no AI model or paid service",
|
||||
"actual_model_usage": None, "input_profile": args.input_profile,
|
||||
"output_profile": args.output_profile, "calls": [], "observations": [],
|
||||
"unexpected_server_requests": [], "cleanup_failures": []}
|
||||
client_version = subprocess.check_output([str(args.codex), "--version"], env=environment, cwd=workspace, text=True, timeout=10).strip()
|
||||
assert client_version == "codex-cli 0.163.0", client_version
|
||||
transcript = []
|
||||
pending = {}
|
||||
completed_turn = asyncio.Event()
|
||||
next_id = 0
|
||||
stderr = (args.output / "server.log").open("wb")
|
||||
process = await asyncio.create_subprocess_exec(str(args.codex), "app-server", "--stdio",
|
||||
stdin=asyncio.subprocess.PIPE, stdout=asyncio.subprocess.PIPE, stderr=stderr,
|
||||
cwd=workspace, env=environment, limit=16 * 1024 * 1024)
|
||||
|
||||
async def send(message):
|
||||
encoded = (json.dumps(message, separators=(",", ":")) + "\n").encode()
|
||||
transcript.append({"direction": "client", "bytes": len(encoded), "message": message})
|
||||
process.stdin.write(encoded)
|
||||
await process.stdin.drain()
|
||||
|
||||
async def read_messages():
|
||||
try:
|
||||
while line := await process.stdout.readline():
|
||||
value = json.loads(line)
|
||||
transcript.append({"direction": "server", "bytes": len(line), "message": value})
|
||||
if value.get("method") == "turn/completed":
|
||||
report["turn"] = value["params"]["turn"]
|
||||
completed_turn.set()
|
||||
if "id" in value and "method" not in value:
|
||||
future = pending.pop(value["id"], None)
|
||||
if future and not future.done():
|
||||
future.set_result(value)
|
||||
elif "id" in value:
|
||||
report["unexpected_server_requests"].append(value["method"])
|
||||
await send({"id": value["id"], "error": {"code": -32601, "message": "Unexpected request in scripted metadata probe"}})
|
||||
raise RuntimeError("app-server stdout closed")
|
||||
except Exception as error:
|
||||
for future in pending.values():
|
||||
if not future.done():
|
||||
future.set_exception(error)
|
||||
|
||||
reader = asyncio.create_task(read_messages())
|
||||
|
||||
async def request(method, params):
|
||||
nonlocal next_id
|
||||
assert method in ["initialize", "thread/start", "mcpServerStatus/list", "mcpServer/tool/call", "turn/start"]
|
||||
next_id += 1
|
||||
message = {"id": next_id, "method": method, "params": params}
|
||||
future = asyncio.get_running_loop().create_future()
|
||||
pending[next_id] = future
|
||||
start = time.monotonic()
|
||||
await send(message)
|
||||
reply = await asyncio.wait_for(future, 90)
|
||||
report["calls"].append({"method": method, "elapsed_ms": (time.monotonic() - start) * 1000,
|
||||
"error": reply.get("error")})
|
||||
if "error" in reply:
|
||||
raise RuntimeError(json.dumps(reply["error"])[:4000])
|
||||
return reply["result"]
|
||||
|
||||
try:
|
||||
await request("initialize", {"clientInfo": {"name": "rea_host_probe", "title": "REA host compatibility probe", "version": "1"},
|
||||
"capabilities": {"experimentalApi": True, "explicitGatewayOauth": True}})
|
||||
await send({"method": "initialized"})
|
||||
created = await request("thread/start", {"cwd": str(workspace), "ephemeral": True,
|
||||
"approvalPolicy": "never", "sandbox": "read-only"})
|
||||
thread_id = created["thread"]["id"]
|
||||
assert not created["thread"].get("turns", [])
|
||||
inventory = await request("mcpServerStatus/list", {"threadId": thread_id,
|
||||
"serverName": "rea_audit", "detail": "toolsAndAuthOnly", "limit": 10})
|
||||
assert inventory.get("nextCursor") is None
|
||||
assert len(inventory["data"]) == 1
|
||||
server = inventory["data"][0]
|
||||
assert server["name"] == "rea_audit" and server.get("toolsError") is None
|
||||
tools = list(server["tools"].values())
|
||||
names = [tool["name"] for tool in tools]
|
||||
assert len(names) == len(set(names)) == args.expected_tools
|
||||
assert "frida_session_status" in names
|
||||
output_schemas = sum(tool.get("outputSchema") is not None for tool in tools)
|
||||
assert output_schemas == (args.expected_tools if args.output_profile == "full" else 0)
|
||||
raw = json.dumps(inventory, separators=(",", ":"), ensure_ascii=False).encode()
|
||||
(args.output / "catalog.json").write_bytes(raw)
|
||||
report["catalog"] = {"tools": len(tools), "output_schemas": output_schemas,
|
||||
"bytes": len(raw), "sha256": hashlib.sha256(raw).hexdigest(),
|
||||
"server_info": server.get("serverInfo"), "runtime_status": server.get("runtimeStatus")}
|
||||
await request("turn/start", {"threadId":thread_id,"input":[{"type":"text","text":"Retain a complete libc export inventory using the selected caller-authored script. Check session state and queued message capture count/truncation twice, close Frida instrumentation and export complete Evidence. The local fixture scripts the tools; no AI inference."}]})
|
||||
await asyncio.wait_for(completed_turn.wait(), 90)
|
||||
assert report["turn"]["status"] == "completed", report["turn"]
|
||||
assert state["normal_requests"]==steps+1
|
||||
assert len(captured)==steps+1+state["compactions"]
|
||||
outputs=known_outputs
|
||||
assert len(outputs)==steps
|
||||
bundle=json.loads((args.output/'evidence.json').read_text())
|
||||
delivery_complete=True
|
||||
answers=[]
|
||||
for index in [4,5]:
|
||||
delivered_result=outputs[f'call_local_{index}']
|
||||
answer=verify_frida_delivery(delivered_result,bundle,args.mode,args.budget,outputs['call_local_1'])
|
||||
if answer is None: delivery_complete=False
|
||||
else: answers.append(answer)
|
||||
loaded=next(e for e in bundle['records'] if e['operation']=='load_frida_script')
|
||||
assert loaded['normalized_result']['source_sha256']==hashlib.sha256(source.encode()).hexdigest()
|
||||
report['task_completed']=delivery_complete
|
||||
report['observations']={'answers':answers,'exported':True,'exported_bundle_bytes':(args.output/'evidence.json').stat().st_size,'tool_output_budget':args.budget or 'client default','source_sha256':hashlib.sha256(source.encode()).hexdigest()}
|
||||
report["captured_requests"]=len(captured)
|
||||
report["compaction_requests"]=state["compactions"]
|
||||
report["compaction_backend"]="Deterministic checkpoint of observed identifiers and retained-body location; no AI inference or quality claim."
|
||||
report["mode"]=args.mode
|
||||
report["request_tool_counts"] = [len(c["body"].get("tools", [])) for c in captured]
|
||||
assert not report["unexpected_server_requests"]
|
||||
report["status"] = "passed" if report["task_completed"] else "expected-incomplete-task"
|
||||
except Exception:
|
||||
report["status"] = "failed"
|
||||
report["failure"] = traceback.format_exc()[-6000:]
|
||||
finally:
|
||||
process.stdin.close()
|
||||
try:
|
||||
await asyncio.wait_for(process.wait(), 15)
|
||||
except asyncio.TimeoutError:
|
||||
report["cleanup_failures"].append("app-server did not exit after stdin close; terminated owned process")
|
||||
process.terminate()
|
||||
try:
|
||||
await asyncio.wait_for(process.wait(), 5)
|
||||
except asyncio.TimeoutError:
|
||||
process.kill()
|
||||
await process.wait()
|
||||
await reader
|
||||
# app-server can acknowledge exit before its MCP child finishes EOF
|
||||
# cleanup. Wait on the allocated resource scope; never target a PID.
|
||||
try:
|
||||
groups = [line[3:] for line in Path("/proc/self/cgroup").read_text().splitlines()
|
||||
if line.startswith("0::")]
|
||||
assert len(groups) == 1
|
||||
group = Path("/sys/fs/cgroup") / groups[0].lstrip("/")
|
||||
assert re.fullmatch(r"rea-efficiency-[0-9a-f]{32}\.scope", group.name)
|
||||
observer = os.getppid()
|
||||
observer_args = Path(f"/proc/{observer}/cmdline").read_bytes().split(b"\0")
|
||||
assert any(Path(os.fsdecode(value)).name == "resource_payload.py" for value in observer_args if value)
|
||||
expected = {os.getpid(), observer}
|
||||
until = time.monotonic() + 10
|
||||
while True:
|
||||
actors = {int(value) for value in (group / "cgroup.procs").read_text().splitlines()}
|
||||
assert expected <= actors
|
||||
remaining = sorted(actors - expected)
|
||||
if not remaining or time.monotonic() >= until:
|
||||
break
|
||||
await asyncio.sleep(0.05)
|
||||
report["owned_scope_drain"] = {"remaining_processes": remaining}
|
||||
if remaining:
|
||||
report["cleanup_failures"].append("owned MCP child did not settle after app-server exit")
|
||||
except (OSError, ValueError, AssertionError) as error:
|
||||
report["owned_scope_drain"] = {"status": "unavailable", "reason": type(error).__name__}
|
||||
report["cleanup_failures"].append("allocated resource-scope cleanup could not be verified")
|
||||
endpoint.shutdown()
|
||||
endpoint.server_close()
|
||||
endpoint_thread.join(5)
|
||||
stderr.close()
|
||||
report["appserver_exit_code"] = process.returncode
|
||||
if process.returncode != 0:
|
||||
report["cleanup_failures"].append(f"app-server exit code: {process.returncode}")
|
||||
(args.output / "transcript.json").write_text(json.dumps(transcript) + "\n")
|
||||
(args.output / "report.json").write_text(json.dumps(report, indent=2) + "\n")
|
||||
print(json.dumps({key: report.get(key) for key in ["status", "failure", "catalog", "observations", "cleanup_failures", "appserver_exit_code"]}))
|
||||
return 0 if report["status"] in ["passed","expected-incomplete-task"] and not report["cleanup_failures"] else 1
|
||||
|
||||
|
||||
raise SystemExit(asyncio.run(main()))
|
||||
@@ -1,719 +0,0 @@
|
||||
{
|
||||
"issue": 5337,
|
||||
"initial_base_revision": "4e716b06078d725a53234503cbefbb340731e809",
|
||||
"integration_base_revision": "3fca2e0ca6e54ba700c286d9a85a9600dee6a19d",
|
||||
"environment": {
|
||||
"platform": "Linux x86_64",
|
||||
"node": "24.18.0",
|
||||
"npm": "11.16.0",
|
||||
"mcp_sdk": "2.3.1",
|
||||
"ghidra": "12.1.4",
|
||||
"java": "Eclipse Adoptium 21.0.12.1+1-LTS",
|
||||
"configured_java_heap": "768M",
|
||||
"configured_active_processors": 1,
|
||||
"query_cpu_quota_percent": 100,
|
||||
"query_cpu_affinity": [15],
|
||||
"query_hard_memory_bytes": 4294967296,
|
||||
"query_early_stop_bytes": 3221225472,
|
||||
"query_task_limit": 192,
|
||||
"swap_bytes": 0,
|
||||
"os_caches": "warm and unflushed; fresh servers, homes and Java analysis",
|
||||
"initial_startup_and_rpc_deadline_ms": 300000,
|
||||
"integrated_startup_and_rpc_deadline_ms": 600000,
|
||||
"product_default_deadlines_changed": false
|
||||
},
|
||||
"method": {
|
||||
"task": "Full observed procedure inventory, overview/load image and first internal non-thunk function facts; repeat inventory/pseudocode, export Evidence and close.",
|
||||
"calls": 15,
|
||||
"catalog_discoveries": 1,
|
||||
"fact_comparison_excludes_only": ["evidence_id", "raw_result"],
|
||||
"bytes": "Node JSON.stringify UTF-8 of complete SDK catalog, tool name/argument objects and reply envelopes; initialization, wire framing and model rendering unmeasured.",
|
||||
"actual_model_usage": null,
|
||||
"tokenizer_estimates": null,
|
||||
"memory": "Kernel cgroup family peak including file cache at payload exit; post-GC retained heap unmeasured.",
|
||||
"counter_scope": "Verified payload exit while the observer remains alive; final whole-scope counters after removal unavailable.",
|
||||
"latency_claim": "No whole-task latency, CPU or peak-memory improvement established.",
|
||||
"candidate_control_overhead": "Baseline fact-file parsing/comparison is included on the candidate side; its isolated cost is unmeasured."
|
||||
},
|
||||
"runtime_tables": {
|
||||
"baseline": {
|
||||
"files": 1841,
|
||||
"sha256_sorted_path_tab_digest_lines": "549bb479ed41e9c55505c169e726552b632f0e5341e63194380b8d0ded344658",
|
||||
"revision": "4e716b06078d725a53234503cbefbb340731e809"
|
||||
},
|
||||
"candidate": {
|
||||
"files": 1841,
|
||||
"sha256_sorted_path_tab_digest_lines": "0ebceab15311cc5ccdcfb92a0891022fc37736f2e7d01e2c1d22e88920df1ca8",
|
||||
"base_revision": "4e716b06078d725a53234503cbefbb340731e809",
|
||||
"source_diff_sha256": "7c460367e77daf2fad507800f1346dcd9e74553999fb049232b33b27684c8f49"
|
||||
},
|
||||
"integrated-baseline": {
|
||||
"files": 1847,
|
||||
"sha256_sorted_path_tab_digest_lines": "0d1672378dd3989663f7ef9ead3a91f56283dd06868f4574cae4d062dc83b041",
|
||||
"base_revision": "3fca2e0ca6e54ba700c286d9a85a9600dee6a19d"
|
||||
},
|
||||
"integrated-candidate": {
|
||||
"files": 1847,
|
||||
"sha256_sorted_path_tab_digest_lines": "d4348d25b4b18e371ac63375accd8f14e72197588934f66b8ab2bb7049ddb7d3",
|
||||
"base_revision": "3fca2e0ca6e54ba700c286d9a85a9600dee6a19d",
|
||||
"source_diff_sha256": "7c460367e77daf2fad507800f1346dcd9e74553999fb049232b33b27684c8f49",
|
||||
"driver_sha256": "b0463f8da110dcbae45769045a690dc5accf74791fa53f5086a60e8383f074de"
|
||||
}
|
||||
},
|
||||
"driver_versions": {
|
||||
"earlier": "b5bf4b378102de16f871c5f13e7302cbdcecbe0a14cc90b30ab80031920aaa06",
|
||||
"integrated": "b0463f8da110dcbae45769045a690dc5accf74791fa53f5086a60e8383f074de"
|
||||
},
|
||||
"completed_tasks": [
|
||||
{
|
||||
"name": "baseline-true-4",
|
||||
"expected_projection": "duplicate",
|
||||
"artifact_sha256": "c79bf44242829108e323378531f4ac839513ca1fba45efd6583643526e1e9fd2",
|
||||
"artifact_bytes": 35664,
|
||||
"procedure_count": 231,
|
||||
"selected_procedure": "0x102000",
|
||||
"facts_sha256": "5b826454c00709fb26777e3b6663aa69ce879f51597c54d6ec3021fc1984eb74",
|
||||
"call_count": 15,
|
||||
"catalog": {
|
||||
"elapsed_ms": 1132.755321,
|
||||
"bytes": 3255374,
|
||||
"sha256": "8a646b95cc5607ee57e5a3c0ef480b77e4cec7e5168498e72f511a8af40daf61"
|
||||
},
|
||||
"rpc_argument_bytes": 908,
|
||||
"rpc_reply_bytes": 473344,
|
||||
"catalog_arguments_replies_bytes": 3729626,
|
||||
"normalized_value_bytes_once_per_selected_reply": 94357,
|
||||
"raw_value_bytes_once_per_selected_reply": 90839,
|
||||
"bundle_bytes": 177922,
|
||||
"phases": {
|
||||
"open_binary": {
|
||||
"elapsed_ms": 913.6441110000001,
|
||||
"reply_bytes": 499
|
||||
},
|
||||
"list_documents": {
|
||||
"elapsed_ms": 27770.679519,
|
||||
"reply_bytes": 6029
|
||||
},
|
||||
"list_procedures": {
|
||||
"elapsed_ms": 156.53601400000116,
|
||||
"reply_bytes": 125181
|
||||
},
|
||||
"binary_overview": {
|
||||
"elapsed_ms": 101.32018699999753,
|
||||
"reply_bytes": 17387
|
||||
},
|
||||
"inspect_native_load_image": {
|
||||
"elapsed_ms": 265.31070799999725,
|
||||
"reply_bytes": 101123
|
||||
},
|
||||
"procedure_info": {
|
||||
"elapsed_ms": 80.39036699999997,
|
||||
"reply_bytes": 7643
|
||||
},
|
||||
"procedure_pseudo_code": {
|
||||
"elapsed_ms": 81.57453700000042,
|
||||
"reply_bytes": 6549
|
||||
},
|
||||
"procedure_assembly": {
|
||||
"elapsed_ms": 53.14257799999905,
|
||||
"reply_bytes": 6715
|
||||
},
|
||||
"procedure_callers": {
|
||||
"elapsed_ms": 22.954539999998815,
|
||||
"reply_bytes": 6709
|
||||
},
|
||||
"procedure_callees": {
|
||||
"elapsed_ms": 8.837930000001506,
|
||||
"reply_bytes": 6753
|
||||
},
|
||||
"analyze_function": {
|
||||
"elapsed_ms": 294.8276279999991,
|
||||
"reply_bytes": 56545
|
||||
},
|
||||
"warm-inventory": {
|
||||
"elapsed_ms": 49.338327999998,
|
||||
"reply_bytes": 125181
|
||||
},
|
||||
"warm-pseudocode": {
|
||||
"elapsed_ms": 16.670878999997512,
|
||||
"reply_bytes": 6549
|
||||
},
|
||||
"export_evidence_bundle": {
|
||||
"elapsed_ms": 51.058947999998054,
|
||||
"reply_bytes": 312
|
||||
},
|
||||
"close_binary": {
|
||||
"elapsed_ms": 494.6449689999936,
|
||||
"reply_bytes": 169
|
||||
}
|
||||
},
|
||||
"scope_seconds": 34.369,
|
||||
"payload_cpu_seconds": 33.150037,
|
||||
"payload_family_peak_bytes": 993923072,
|
||||
"payload_cleanup_verified": true,
|
||||
"zero_limit_events_at_payload_exit": true,
|
||||
"final_whole_scope_counters_verified": false
|
||||
},
|
||||
{
|
||||
"name": "candidate-true-1",
|
||||
"expected_projection": "compact",
|
||||
"artifact_sha256": "c79bf44242829108e323378531f4ac839513ca1fba45efd6583643526e1e9fd2",
|
||||
"artifact_bytes": 35664,
|
||||
"procedure_count": 231,
|
||||
"selected_procedure": "0x102000",
|
||||
"facts_sha256": "5b826454c00709fb26777e3b6663aa69ce879f51597c54d6ec3021fc1984eb74",
|
||||
"call_count": 15,
|
||||
"catalog": {
|
||||
"elapsed_ms": 1208.093488,
|
||||
"bytes": 3255374,
|
||||
"sha256": "8a646b95cc5607ee57e5a3c0ef480b77e4cec7e5168498e72f511a8af40daf61"
|
||||
},
|
||||
"rpc_argument_bytes": 909,
|
||||
"rpc_reply_bytes": 326642,
|
||||
"catalog_arguments_replies_bytes": 3582925,
|
||||
"normalized_value_bytes_once_per_selected_reply": 94357,
|
||||
"raw_value_bytes_once_per_selected_reply": 22114,
|
||||
"bundle_bytes": 137135,
|
||||
"phases": {
|
||||
"open_binary": {
|
||||
"elapsed_ms": 941.0834749999995,
|
||||
"reply_bytes": 499
|
||||
},
|
||||
"list_documents": {
|
||||
"elapsed_ms": 27897.447016,
|
||||
"reply_bytes": 5981
|
||||
},
|
||||
"list_procedures": {
|
||||
"elapsed_ms": 136.00869700000112,
|
||||
"reply_bytes": 65589
|
||||
},
|
||||
"binary_overview": {
|
||||
"elapsed_ms": 124.65224699999817,
|
||||
"reply_bytes": 17387
|
||||
},
|
||||
"inspect_native_load_image": {
|
||||
"elapsed_ms": 249.8859930000035,
|
||||
"reply_bytes": 101123
|
||||
},
|
||||
"procedure_info": {
|
||||
"elapsed_ms": 95.36772800000472,
|
||||
"reply_bytes": 6671
|
||||
},
|
||||
"procedure_pseudo_code": {
|
||||
"elapsed_ms": 84.8552570000029,
|
||||
"reply_bytes": 6355
|
||||
},
|
||||
"procedure_assembly": {
|
||||
"elapsed_ms": 97.34392800000205,
|
||||
"reply_bytes": 6305
|
||||
},
|
||||
"procedure_callers": {
|
||||
"elapsed_ms": 8.018889999999374,
|
||||
"reply_bytes": 6177
|
||||
},
|
||||
"procedure_callees": {
|
||||
"elapsed_ms": 6.999988999996276,
|
||||
"reply_bytes": 6199
|
||||
},
|
||||
"analyze_function": {
|
||||
"elapsed_ms": 271.9528430000064,
|
||||
"reply_bytes": 31929
|
||||
},
|
||||
"warm-inventory": {
|
||||
"elapsed_ms": 30.13376900000003,
|
||||
"reply_bytes": 65589
|
||||
},
|
||||
"warm-pseudocode": {
|
||||
"elapsed_ms": 7.71154899999965,
|
||||
"reply_bytes": 6355
|
||||
},
|
||||
"export_evidence_bundle": {
|
||||
"elapsed_ms": 71.93626799999765,
|
||||
"reply_bytes": 314
|
||||
},
|
||||
"close_binary": {
|
||||
"elapsed_ms": 620.6287440000015,
|
||||
"reply_bytes": 169
|
||||
}
|
||||
},
|
||||
"scope_seconds": 34.901,
|
||||
"payload_cpu_seconds": 33.834228,
|
||||
"payload_family_peak_bytes": 1025228800,
|
||||
"payload_cleanup_verified": true,
|
||||
"zero_limit_events_at_payload_exit": true,
|
||||
"final_whole_scope_counters_verified": false
|
||||
},
|
||||
{
|
||||
"name": "baseline-jq-1",
|
||||
"expected_projection": "duplicate",
|
||||
"artifact_sha256": "b1c22172dd303f3be49e935aa56aa48a8b7a46e0bc838b4997d3bb451495870f",
|
||||
"artifact_bytes": 2267912,
|
||||
"procedure_count": 2276,
|
||||
"selected_procedure": "0x401000",
|
||||
"facts_sha256": "f7b95167c93f2b9129b5a4332590ef6bbb9d7b646d554ba4c396db1055ef0d24",
|
||||
"call_count": 15,
|
||||
"catalog": {
|
||||
"elapsed_ms": 2761.8931860000002,
|
||||
"bytes": 3255374,
|
||||
"sha256": "8a646b95cc5607ee57e5a3c0ef480b77e4cec7e5168498e72f511a8af40daf61"
|
||||
},
|
||||
"rpc_argument_bytes": 965,
|
||||
"rpc_reply_bytes": 2372592,
|
||||
"catalog_arguments_replies_bytes": 5628931,
|
||||
"normalized_value_bytes_once_per_selected_reply": 536023,
|
||||
"raw_value_bytes_once_per_selected_reply": 533635,
|
||||
"bundle_bytes": 609212,
|
||||
"phases": {
|
||||
"open_binary": {
|
||||
"elapsed_ms": 1856.9621389999993,
|
||||
"reply_bytes": 617
|
||||
},
|
||||
"list_documents": {
|
||||
"elapsed_ms": 290349.418768,
|
||||
"reply_bytes": 6167
|
||||
},
|
||||
"list_procedures": {
|
||||
"elapsed_ms": 624.2473109999555,
|
||||
"reply_bytes": 1092959
|
||||
},
|
||||
"binary_overview": {
|
||||
"elapsed_ms": 214.38793299999088,
|
||||
"reply_bytes": 15727
|
||||
},
|
||||
"inspect_native_load_image": {
|
||||
"elapsed_ms": 432.1906869999948,
|
||||
"reply_bytes": 75633
|
||||
},
|
||||
"procedure_info": {
|
||||
"elapsed_ms": 106.62601700000232,
|
||||
"reply_bytes": 7813
|
||||
},
|
||||
"procedure_pseudo_code": {
|
||||
"elapsed_ms": 165.65909500000998,
|
||||
"reply_bytes": 6869
|
||||
},
|
||||
"procedure_assembly": {
|
||||
"elapsed_ms": 98.534897000005,
|
||||
"reply_bytes": 6855
|
||||
},
|
||||
"procedure_callers": {
|
||||
"elapsed_ms": 23.43981900002109,
|
||||
"reply_bytes": 6891
|
||||
},
|
||||
"procedure_callees": {
|
||||
"elapsed_ms": 9.403080000018235,
|
||||
"reply_bytes": 7479
|
||||
},
|
||||
"analyze_function": {
|
||||
"elapsed_ms": 327.3197410000139,
|
||||
"reply_bytes": 45277
|
||||
},
|
||||
"warm-inventory": {
|
||||
"elapsed_ms": 396.67419799999334,
|
||||
"reply_bytes": 1092959
|
||||
},
|
||||
"warm-pseudocode": {
|
||||
"elapsed_ms": 6.517379999975674,
|
||||
"reply_bytes": 6869
|
||||
},
|
||||
"export_evidence_bundle": {
|
||||
"elapsed_ms": 56.13109899999108,
|
||||
"reply_bytes": 308
|
||||
},
|
||||
"close_binary": {
|
||||
"elapsed_ms": 550.6349530000007,
|
||||
"reply_bytes": 169
|
||||
}
|
||||
},
|
||||
"scope_seconds": 304.543,
|
||||
"payload_cpu_seconds": 294.743485,
|
||||
"payload_family_peak_bytes": 1159921664,
|
||||
"payload_cleanup_verified": true,
|
||||
"zero_limit_events_at_payload_exit": true,
|
||||
"final_whole_scope_counters_verified": false
|
||||
},
|
||||
{
|
||||
"name": "integrated-baseline-jq",
|
||||
"expected_projection": "duplicate",
|
||||
"artifact_sha256": "b1c22172dd303f3be49e935aa56aa48a8b7a46e0bc838b4997d3bb451495870f",
|
||||
"artifact_bytes": 2267912,
|
||||
"procedure_count": 2279,
|
||||
"selected_procedure": "0x401000",
|
||||
"facts_sha256": "87722aab25011f1b047eb2bf8302cdbc5a4256c1b7f17d7bd68404a7d191e070",
|
||||
"call_count": 15,
|
||||
"catalog": {
|
||||
"elapsed_ms": 1411.786849,
|
||||
"bytes": 3263492,
|
||||
"sha256": "c754f774156866d70be754e050335d0e4d3a7a6cbb156746cf1f7b5ffa42b26b"
|
||||
},
|
||||
"rpc_argument_bytes": 974,
|
||||
"rpc_reply_bytes": 2375466,
|
||||
"catalog_arguments_replies_bytes": 5639932,
|
||||
"normalized_value_bytes_once_per_selected_reply": 536689,
|
||||
"raw_value_bytes_once_per_selected_reply": 534301,
|
||||
"bundle_bytes": 609878,
|
||||
"phases": {
|
||||
"open_binary": {
|
||||
"elapsed_ms": 983.5489280000002,
|
||||
"reply_bytes": 617
|
||||
},
|
||||
"list_documents": {
|
||||
"elapsed_ms": 270159.690556,
|
||||
"reply_bytes": 6167
|
||||
},
|
||||
"list_procedures": {
|
||||
"elapsed_ms": 717.9032449999941,
|
||||
"reply_bytes": 1094387
|
||||
},
|
||||
"binary_overview": {
|
||||
"elapsed_ms": 203.3595150000183,
|
||||
"reply_bytes": 15727
|
||||
},
|
||||
"inspect_native_load_image": {
|
||||
"elapsed_ms": 362.37314199999673,
|
||||
"reply_bytes": 75633
|
||||
},
|
||||
"procedure_info": {
|
||||
"elapsed_ms": 92.72515800001565,
|
||||
"reply_bytes": 7813
|
||||
},
|
||||
"procedure_pseudo_code": {
|
||||
"elapsed_ms": 85.43632900004741,
|
||||
"reply_bytes": 6869
|
||||
},
|
||||
"procedure_assembly": {
|
||||
"elapsed_ms": 78.87645799998427,
|
||||
"reply_bytes": 6855
|
||||
},
|
||||
"procedure_callers": {
|
||||
"elapsed_ms": 32.66119900002377,
|
||||
"reply_bytes": 6891
|
||||
},
|
||||
"procedure_callees": {
|
||||
"elapsed_ms": 8.895469999988563,
|
||||
"reply_bytes": 7479
|
||||
},
|
||||
"analyze_function": {
|
||||
"elapsed_ms": 343.2701520000119,
|
||||
"reply_bytes": 45277
|
||||
},
|
||||
"warm-inventory": {
|
||||
"elapsed_ms": 338.9290720000281,
|
||||
"reply_bytes": 1094387
|
||||
},
|
||||
"warm-pseudocode": {
|
||||
"elapsed_ms": 9.391289999999572,
|
||||
"reply_bytes": 6869
|
||||
},
|
||||
"export_evidence_bundle": {
|
||||
"elapsed_ms": 82.86341799999354,
|
||||
"reply_bytes": 326
|
||||
},
|
||||
"close_binary": {
|
||||
"elapsed_ms": 679.7842850000015,
|
||||
"reply_bytes": 169
|
||||
}
|
||||
},
|
||||
"scope_seconds": 279.699,
|
||||
"payload_cpu_seconds": 277.696079,
|
||||
"payload_family_peak_bytes": 1111494656,
|
||||
"payload_cleanup_verified": true,
|
||||
"zero_limit_events_at_payload_exit": true,
|
||||
"final_whole_scope_counters_verified": false
|
||||
},
|
||||
{
|
||||
"name": "integrated-candidate-jq",
|
||||
"expected_projection": "compact",
|
||||
"artifact_sha256": "b1c22172dd303f3be49e935aa56aa48a8b7a46e0bc838b4997d3bb451495870f",
|
||||
"artifact_bytes": 2267912,
|
||||
"procedure_count": 2279,
|
||||
"selected_procedure": "0x401000",
|
||||
"facts_sha256": "87722aab25011f1b047eb2bf8302cdbc5a4256c1b7f17d7bd68404a7d191e070",
|
||||
"call_count": 15,
|
||||
"catalog": {
|
||||
"elapsed_ms": 1213.2912340000003,
|
||||
"bytes": 3263492,
|
||||
"sha256": "c754f774156866d70be754e050335d0e4d3a7a6cbb156746cf1f7b5ffa42b26b"
|
||||
},
|
||||
"rpc_argument_bytes": 975,
|
||||
"rpc_reply_bytes": 1264884,
|
||||
"catalog_arguments_replies_bytes": 4529351,
|
||||
"normalized_value_bytes_once_per_selected_reply": 536689,
|
||||
"raw_value_bytes_once_per_selected_reply": 16143,
|
||||
"bundle_bytes": 345626,
|
||||
"phases": {
|
||||
"open_binary": {
|
||||
"elapsed_ms": 927.8219500000005,
|
||||
"reply_bytes": 617
|
||||
},
|
||||
"list_documents": {
|
||||
"elapsed_ms": 391135.781249,
|
||||
"reply_bytes": 6119
|
||||
},
|
||||
"list_procedures": {
|
||||
"elapsed_ms": 646.0353969999705,
|
||||
"reply_bytes": 550261
|
||||
},
|
||||
"binary_overview": {
|
||||
"elapsed_ms": 198.11136500001885,
|
||||
"reply_bytes": 15727
|
||||
},
|
||||
"inspect_native_load_image": {
|
||||
"elapsed_ms": 456.49483100004727,
|
||||
"reply_bytes": 75633
|
||||
},
|
||||
"procedure_info": {
|
||||
"elapsed_ms": 99.12928700004704,
|
||||
"reply_bytes": 6825
|
||||
},
|
||||
"procedure_pseudo_code": {
|
||||
"elapsed_ms": 114.82446700002765,
|
||||
"reply_bytes": 6584
|
||||
},
|
||||
"procedure_assembly": {
|
||||
"elapsed_ms": 80.73368800000753,
|
||||
"reply_bytes": 6444
|
||||
},
|
||||
"procedure_callers": {
|
||||
"elapsed_ms": 10.588890000013635,
|
||||
"reply_bytes": 6337
|
||||
},
|
||||
"procedure_callees": {
|
||||
"elapsed_ms": 19.5354899999802,
|
||||
"reply_bytes": 6631
|
||||
},
|
||||
"analyze_function": {
|
||||
"elapsed_ms": 352.812751999998,
|
||||
"reply_bytes": 26364
|
||||
},
|
||||
"warm-inventory": {
|
||||
"elapsed_ms": 197.59054599999217,
|
||||
"reply_bytes": 550261
|
||||
},
|
||||
"warm-pseudocode": {
|
||||
"elapsed_ms": 8.122649999975692,
|
||||
"reply_bytes": 6584
|
||||
},
|
||||
"export_evidence_bundle": {
|
||||
"elapsed_ms": 61.80356899998151,
|
||||
"reply_bytes": 328
|
||||
},
|
||||
"close_binary": {
|
||||
"elapsed_ms": 614.0456879999838,
|
||||
"reply_bytes": 169
|
||||
}
|
||||
},
|
||||
"scope_seconds": 399.097,
|
||||
"payload_cpu_seconds": 352.659548,
|
||||
"payload_family_peak_bytes": 1088741376,
|
||||
"payload_cleanup_verified": true,
|
||||
"zero_limit_events_at_payload_exit": true,
|
||||
"final_whole_scope_counters_verified": false
|
||||
}
|
||||
],
|
||||
"incomplete_attempts": [
|
||||
{
|
||||
"name": "baseline-true-1",
|
||||
"reason": "Harness selected subject.sha256 instead of subject.digest.sha256.",
|
||||
"reported_status": "failed",
|
||||
"payload_exit_code": 1,
|
||||
"scope_seconds": 41.401,
|
||||
"payload_cpu_seconds": 39.027824,
|
||||
"payload_family_peak_bytes": 1132675072,
|
||||
"observed_procedures": null,
|
||||
"payload_cleanup_verified": true,
|
||||
"resource_events_at_payload_exit": {
|
||||
"memory.events": {
|
||||
"low": 0,
|
||||
"high": 0,
|
||||
"max": 0,
|
||||
"oom": 0,
|
||||
"oom_kill": 0,
|
||||
"oom_group_kill": 0
|
||||
},
|
||||
"pids.events": {
|
||||
"max": 0
|
||||
}
|
||||
},
|
||||
"is_completed_performance_sample": false
|
||||
},
|
||||
{
|
||||
"name": "baseline-true-2",
|
||||
"reason": "Harness assumed the composed overview provider was Ghidra rather than rea-workflow.",
|
||||
"reported_status": "failed",
|
||||
"payload_exit_code": 1,
|
||||
"scope_seconds": 37.538,
|
||||
"payload_cpu_seconds": 36.599406,
|
||||
"payload_family_peak_bytes": 1001476096,
|
||||
"observed_procedures": 231,
|
||||
"payload_cleanup_verified": true,
|
||||
"resource_events_at_payload_exit": {
|
||||
"memory.events": {
|
||||
"low": 0,
|
||||
"high": 0,
|
||||
"max": 0,
|
||||
"oom": 0,
|
||||
"oom_kill": 0,
|
||||
"oom_group_kill": 0
|
||||
},
|
||||
"pids.events": {
|
||||
"max": 0
|
||||
}
|
||||
},
|
||||
"is_completed_performance_sample": false
|
||||
},
|
||||
{
|
||||
"name": "baseline-true-3",
|
||||
"reason": "Facts and cleanup completed, but the harness incorrectly read a status property from the boolean finalizer and exited 1.",
|
||||
"reported_status": "passed",
|
||||
"payload_exit_code": 1,
|
||||
"scope_seconds": 35.796,
|
||||
"payload_cpu_seconds": 34.736678,
|
||||
"payload_family_peak_bytes": 1001455616,
|
||||
"observed_procedures": 231,
|
||||
"payload_cleanup_verified": true,
|
||||
"resource_events_at_payload_exit": {
|
||||
"memory.events": {
|
||||
"low": 0,
|
||||
"high": 0,
|
||||
"max": 0,
|
||||
"oom": 0,
|
||||
"oom_kill": 0,
|
||||
"oom_group_kill": 0
|
||||
},
|
||||
"pids.events": {
|
||||
"max": 0
|
||||
}
|
||||
},
|
||||
"is_completed_performance_sample": false
|
||||
},
|
||||
{
|
||||
"name": "candidate-jq-1",
|
||||
"reason": "SDK five-minute first-query deadline expired while awaiting list_documents; cleanup completed.",
|
||||
"reported_status": "failed",
|
||||
"payload_exit_code": 1,
|
||||
"scope_seconds": 316.605,
|
||||
"payload_cpu_seconds": 293.28205,
|
||||
"payload_family_peak_bytes": 1065631744,
|
||||
"observed_procedures": null,
|
||||
"payload_cleanup_verified": true,
|
||||
"resource_events_at_payload_exit": {
|
||||
"memory.events": {
|
||||
"low": 0,
|
||||
"high": 0,
|
||||
"max": 0,
|
||||
"oom": 0,
|
||||
"oom_kill": 0,
|
||||
"oom_group_kill": 0
|
||||
},
|
||||
"pids.events": {
|
||||
"max": 0
|
||||
}
|
||||
},
|
||||
"is_completed_performance_sample": false
|
||||
},
|
||||
{
|
||||
"name": "integrated-baseline-jq-control",
|
||||
"reason": "Unmodified-main repeat differs by three observed procedure rows; strict comparison rejected it before export. Cause unestablished.",
|
||||
"reported_status": "failed",
|
||||
"payload_exit_code": 1,
|
||||
"scope_seconds": 276.557,
|
||||
"payload_cpu_seconds": 274.502165,
|
||||
"payload_family_peak_bytes": 1200726016,
|
||||
"observed_procedures": 2276,
|
||||
"payload_cleanup_verified": true,
|
||||
"resource_events_at_payload_exit": {
|
||||
"memory.events": {
|
||||
"low": 0,
|
||||
"high": 0,
|
||||
"max": 0,
|
||||
"oom": 0,
|
||||
"oom_kill": 0,
|
||||
"oom_group_kill": 0
|
||||
},
|
||||
"pids.events": {
|
||||
"max": 0
|
||||
}
|
||||
},
|
||||
"is_completed_performance_sample": false
|
||||
}
|
||||
],
|
||||
"capacity_control": {
|
||||
"baseline_revision": "3fca2e0ca6e54ba700c286d9a85a9600dee6a19d",
|
||||
"records": 30000,
|
||||
"baseline": "regression fails with resource_constraint for list_procedures at the unchanged 10485760-byte stdio budget",
|
||||
"candidate": "all 30000 records delivered exactly; raw_result null",
|
||||
"real_provider": false,
|
||||
"production_boundary": "GhidraProvider -> BinarySession -> public MCP"
|
||||
},
|
||||
"executed_validation": [
|
||||
{
|
||||
"check": "npm run check",
|
||||
"base": "4e716b060",
|
||||
"result": "6 tasks and architecture guards passed"
|
||||
},
|
||||
{
|
||||
"check": "npm run check",
|
||||
"base": "3fca2e0ca",
|
||||
"result": "6 tasks and architecture guards passed; before adding the capacity regression"
|
||||
},
|
||||
{
|
||||
"check": "Focused Ghidra inventory/function/provider and four MCP files",
|
||||
"base": "4e716b060",
|
||||
"result": "246 assertions passed across 7 files"
|
||||
},
|
||||
{
|
||||
"check": "Capacity regression alone",
|
||||
"base": "3fca2e0ca",
|
||||
"result": "candidate passes; unchanged production baseline fails with resource_constraint; six other tests filtered out"
|
||||
},
|
||||
{
|
||||
"check": "Existing real Ghidra large-result helper through a private stdio harness",
|
||||
"result": "passed",
|
||||
"successful_calls": 15,
|
||||
"expected_rejections": 4,
|
||||
"runtime_empty_after_provider_close": true,
|
||||
"scope": "oversized lookup/read/annotation/dossier retention and recovery, complete CLI readback, unchanged source bytes"
|
||||
},
|
||||
{
|
||||
"check": "node scripts/verify-real-ghidra-address-context.mjs",
|
||||
"result": "passed with real Ghidra/CLI/MCP and verified empty descendant set",
|
||||
"scope": "PIE load identity, file-offset decompilation/roles, address round trips, failure guidance/recovery, unchanged source and complete cleanup",
|
||||
"fixture_sha256": "093f8ba3aef80f3d7c0e8f22557e9a8af29505e286835578282b8e32ff804a02"
|
||||
},
|
||||
{
|
||||
"check": "Final focused Ghidra source and MCP regression selection",
|
||||
"base": "3fca2e0ca",
|
||||
"result": "247 assertions across 7 files passed"
|
||||
},
|
||||
{
|
||||
"check": "npm run docs:check",
|
||||
"result": "generated metadata/documentation checks and 5 README conformance assertions passed"
|
||||
}
|
||||
],
|
||||
"unmodified_main_control": {
|
||||
"same_target_digest_and_analysis_profile": true,
|
||||
"expected_procedures": 2279,
|
||||
"observed_procedures": 2276,
|
||||
"missing_relative_to_first_run": ["0x4f9930", "0x5260b0", "0x529640"],
|
||||
"cause": "unestablished; no engine/default-analysis repair included",
|
||||
"interpretation": "The matched pair is an observation, not a guarantee of cross-session procedure discovery or a latency distribution."
|
||||
},
|
||||
"limitations": [
|
||||
"No actual model-token or tokenizer measurement.",
|
||||
"Single completed pairs, not latency distributions.",
|
||||
"Unmodified-main procedure-inventory repeat failed the strict oracle; retained as an incomplete control.",
|
||||
"First three successful query receipts used the earlier driver; archived warm results were rechecked when explicit repeat assertions were added.",
|
||||
"Real Windows/macOS Ghidra, other engine versions and real NativeAOT overlays remain unqualified.",
|
||||
"Initial cached baseline build had an unverified payload-exit cleanup observation; a later systemd inspection found its scope inactive. It is not a query measurement."
|
||||
],
|
||||
"resource_stopped_validation": [
|
||||
{
|
||||
"check": "Final focused regression selection, first attempt",
|
||||
"abort": "host_available_memory_early_stop",
|
||||
"peak_family_bytes_sampled": 837332992,
|
||||
"outcome": "guard-stopped without final test results; not counted as passing",
|
||||
"retry_budget": "4 GiB hard/3 GiB early, based on the earlier focused run peak of about 1.3 GiB"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,165 +0,0 @@
|
||||
# Ghidra Evidence projection
|
||||
|
||||
Continues [#5337](https://github.com/morluto/rea/issues/5337). The bridge
|
||||
adapter supplied a complete `rawResult` for every successful operation, even
|
||||
when strict parsing preserved the same value in `normalized_result`. Function
|
||||
observation envelopes repeated their value and already-projected qualifications.
|
||||
The generic Evidence contract already permits `raw_result: null` in this case.
|
||||
|
||||
## Owning change
|
||||
|
||||
The Ghidra inventory and function parsers now declare whether they retain an
|
||||
independent upstream representation. Execution consumes those parsed facts and
|
||||
named metadata, replacing the positional helper argument that required every
|
||||
caller to supply another complete report. CLI and MCP use the same projection.
|
||||
|
||||
Ordinary inventories and function observations omit the redundant copy.
|
||||
Limitations, relationship coverage and locations remain canonical Evidence
|
||||
fields. The following representations remain available:
|
||||
|
||||
- The overview's upstream segment permissions, sections and provenance.
|
||||
- Load-image observations before local attestation and enrichment.
|
||||
- File-offset selection roles, which generic locations alone do not express.
|
||||
- The database type reply before adding a locally derived NativeAOT overlay.
|
||||
|
||||
No tool, schema, query default, provider protocol, cache, target execution or
|
||||
annotation authority is added. Existing source Evidence is not rewritten.
|
||||
New records have the identities appropriate to their new representation;
|
||||
artifact and procedure identities remain unchanged. Consumers should read
|
||||
function results from `normalized_result`, rather than `raw_result.value`.
|
||||
|
||||
## Real inputs and completed question
|
||||
|
||||
Both ELF x86-64 targets are read without executing them:
|
||||
|
||||
| Target | Bytes | SHA-256 | Provenance |
|
||||
| ---------------- | --------: | ------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `true` | 35,664 | `c79bf44242829108e323378531f4ac839513ca1fba45efd6583643526e1e9fd2` | Installed Debian coreutils 9.1-1; package metadata is a local observation. |
|
||||
| `jq-linux-amd64` | 2,267,912 | `b1c22172dd303f3be49e935aa56aa48a8b7a46e0bc838b4997d3bb451495870f` | [Official jq 1.8.2 asset](https://github.com/jqlang/jq/releases/tag/jq-1.8.2); matches both the corpus and the release API's asset digest/size. |
|
||||
|
||||
Each fresh production stdio MCP session discovers the complete catalog, opens
|
||||
the target, reads documents and the complete observed procedure inventory,
|
||||
selects the first internal non-thunk procedure, reads overview/load image and
|
||||
six function operations, repeats inventory/pseudocode, exports Evidence and
|
||||
closes. This is one catalog discovery and 15 RPCs, including cleanup.
|
||||
It qualifies this inventory/function-read question, rather than application
|
||||
feature reconstruction or complete native-code discovery.
|
||||
|
||||
The live named SDK contracts validate replies. Each selected record binds to
|
||||
the original target digest and Ghidra profile. The candidate's complete selected
|
||||
fact vector equals the baseline after excluding only `evidence_id` and
|
||||
`raw_result`: subject, parameters, normalized facts, provider/profile, authority,
|
||||
limitations, relationships, locations and Evidence links remain equal. Exported
|
||||
records match returned Evidence, repeated facts agree, and target hashes remain
|
||||
unchanged. This is representation equivalence, not an independent qualification
|
||||
of every Ghidra interpretation.
|
||||
|
||||
## Measurements and limits
|
||||
|
||||
Linux x86-64, Node 24.18.0/npm 11.16.0, MCP SDK 2.3.1, Ghidra 12.1.4 and
|
||||
Eclipse Adoptium 21.0.12.1+1-LTS. Sessions and Java analysis are fresh; OS caches
|
||||
are warm and unflushed. Configured Java heap is 768 MiB with one active processor.
|
||||
The whole process family uses one logical CPU, 4 GiB hard/3 GiB early-stop memory,
|
||||
no swap and 192 tasks. No completed query task has an OOM/limit event or cleanup
|
||||
failure at verified payload exit. Final whole-scope counters after scope removal
|
||||
remain unavailable. Post-GC retained heap is unmeasured.
|
||||
|
||||
The small pair uses main `4e716b06078d725a53234503cbefbb340731e809` and its
|
||||
candidate. Main advanced during the experiment; the large pair uses
|
||||
`3fca2e0ca6e54ba700c286d9a85a9600dee6a19d` on both sides. Existing timing
|
||||
samples keep their original runtime fingerprints. The [curated receipt](ghidra-evidence-projection.json)
|
||||
retains individual observations and incomplete attempts; raw data stays outside Git.
|
||||
|
||||
| Recording / metric | Before | After |
|
||||
| ----------------------------------------------- | ---------: | ---------: |
|
||||
| `true`: observed procedures | 231 | 231 |
|
||||
| `true`: all RPC result bytes | 473,344 | 326,642 |
|
||||
| `true`: catalog + request arguments + results | 3,729,626 | 3,582,925 |
|
||||
| `true`: complete scope wall | 34.369 s | 34.901 s |
|
||||
| `true`: payload-family CPU | 33.150 s | 33.834 s |
|
||||
| `true`: kernel family peak including file cache | 947.9 MiB | 977.7 MiB |
|
||||
| `jq`: observed procedures | 2,279 | 2,279 |
|
||||
| `jq`: all RPC result bytes | 2,375,466 | 1,264,884 |
|
||||
| `jq`: catalog + request arguments + results | 5,639,932 | 4,529,351 |
|
||||
| `jq`: complete scope wall | 279.699 s | 399.097 s |
|
||||
| `jq`: payload-family CPU | 277.696 s | 352.660 s |
|
||||
| `jq`: kernel family peak including file cache | 1060.0 MiB | 1038.3 MiB |
|
||||
|
||||
RPC result bytes include the SDK's text and structured representations. Request
|
||||
objects and the complete catalog are included in the combined byte count;
|
||||
initialization, wire framing and model rendering are unmeasured. Actual model
|
||||
tokens and tokenizer estimates are unmeasured. The observed byte reduction
|
||||
does not establish a token saving or better agent investigation outcome.
|
||||
|
||||
The large result envelopes shrink by 46.8%; combined serialization shrinks by
|
||||
19.7%, with unchanged catalog and call count. Small combined serialization
|
||||
shrinks by 3.9%. These are single pairs, not distributions. No whole-task latency,
|
||||
CPU or peak-memory improvement is established. The large candidate's first
|
||||
`list_documents` costs 391.136 s versus 270.160 s before; this phase includes
|
||||
startup, analysis wait and client work, not isolated engine time. Changed
|
||||
projection code executes after the bridge returns its observation, but the
|
||||
cause of the preparation-time variation is unestablished. One warm inventory
|
||||
read costs 338.93 → 197.59 ms; this is not a general latency qualification.
|
||||
|
||||
A subsequent unchanged-main repeat with the same target/profile returned 2,276
|
||||
procedures instead of 2,279, omitting `0x4f9930`, `0x5260b0` and `0x529640`
|
||||
relative to its earlier inventory. The strict oracle rejected that control
|
||||
before export. Its cause is unestablished; it is retained as an incomplete
|
||||
attempt, not added to performance averages or treated as a projection defect.
|
||||
|
||||
## Verification scope
|
||||
|
||||
The production-boundary capacity regression supplies 30,000 valid simulated
|
||||
Ghidra procedure rows through `GhidraProvider`, `BinarySession` and MCP. On
|
||||
unchanged production main it fails with `resource_constraint` at the existing
|
||||
10 MiB response budget. The candidate delivers every row exactly, without a
|
||||
budget increase or truncation. This qualifies response admission, not a real
|
||||
30,000-function Ghidra analysis or a universal maximum procedure count.
|
||||
|
||||
Source/parser tests cover strict rejection, sidecars, distinct file-offset
|
||||
mapping roles, overview details, NativeAOT enrichment and unchanged observations.
|
||||
Executed checks and real-provider verification are recorded in the receipt.
|
||||
NativeAOT overlay tests use producer fixtures; real Windows/macOS and other
|
||||
engine versions remain unqualified here.
|
||||
|
||||
The existing real large-result helper passed through a private stdio harness:
|
||||
oversized lookup/read/annotation/dossier replies retained complete Evidence,
|
||||
subsequent queries recovered, complete CLI readback matched the original bytes
|
||||
with `raw_result: null`, and generated target bytes remained unchanged.
|
||||
The public `verify-real-ghidra-address-context.mjs` also passed with real
|
||||
Ghidra/CLI/MCP, including file-offset selection, address round trips, recovery,
|
||||
unchanged source and verified cleanup. These capacity/address fixtures use a
|
||||
separate 6 GiB family cap; they are correctness checks, not latency pairs.
|
||||
|
||||
Three early `true` harness attempts used an incorrect subject-digest selector,
|
||||
assumed the composed overview's provider was Ghidra, or misread the existing
|
||||
finalizer's boolean result. The last completed its facts but exited nonzero;
|
||||
none is a completed performance sample. A first `jq` candidate hit the SDK's
|
||||
five-minute deadline while awaiting readiness, then cleaned up successfully.
|
||||
The final driver's startup/RPC deadlines are ten minutes on both sides; product
|
||||
defaults are unchanged. Earlier successful warm facts were rechecked from their
|
||||
archived production replies when explicit warm-result assertions were added.
|
||||
|
||||
## Reproduce
|
||||
|
||||
Build and retain separate baseline/candidate runtimes. Use the same absolute
|
||||
target path and caller-supplied Ghidra/JDK for both:
|
||||
|
||||
```sh
|
||||
python3 experiments/analysis-efficiency/resource-run.py --observe-payload-exit -- \
|
||||
node experiments/analysis-efficiency/ghidra-evidence-projection.mjs \
|
||||
--runtime /absolute/baseline --target /absolute/application \
|
||||
--ghidra /absolute/ghidra --java /absolute/jdk \
|
||||
--expect duplicate --output /absolute/new-before
|
||||
|
||||
python3 experiments/analysis-efficiency/resource-run.py --observe-payload-exit -- \
|
||||
node experiments/analysis-efficiency/ghidra-evidence-projection.mjs \
|
||||
--runtime /absolute/candidate --target /absolute/application \
|
||||
--ghidra /absolute/ghidra --java /absolute/jdk \
|
||||
--expect compact --control /absolute/new-before/facts.json \
|
||||
--output /absolute/new-after
|
||||
```
|
||||
|
||||
The candidate's independent baseline comparison is included in harness costs;
|
||||
its isolated verification overhead is unmeasured. Static checks use a separate
|
||||
8 GiB memory cap. All heavy operations run serially.
|
||||
@@ -1,216 +0,0 @@
|
||||
/** Real Ghidra inventory/function Evidence; raw inputs and receipts stay outside Git. */
|
||||
import assert from "node:assert/strict";
|
||||
import { createHash } from "node:crypto";
|
||||
import { createWriteStream } from "node:fs";
|
||||
import { mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join, resolve } from "node:path";
|
||||
import { finished } from "node:stream/promises";
|
||||
import { parseArgs } from "node:util";
|
||||
import { Client } from "@modelcontextprotocol/client";
|
||||
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
|
||||
import { createRetainedQueryCaller } from "./retained-query-calls.mjs";
|
||||
import { finalizeRetainedQueryReceipt } from "./retained-query-finalization.mjs";
|
||||
|
||||
const { values } = parseArgs({
|
||||
options: Object.fromEntries(
|
||||
["runtime", "target", "output", "ghidra", "java", "expect", "control"].map(
|
||||
(name) => [name, { type: "string" }],
|
||||
),
|
||||
),
|
||||
});
|
||||
for (const name of ["runtime", "target", "output", "ghidra", "java"])
|
||||
assert.ok(values[name] && values[name] === resolve(values[name]), name);
|
||||
assert.ok(["duplicate", "compact"].includes(values.expect));
|
||||
const hash = (value) => createHash("sha256").update(value).digest("hex");
|
||||
const output = values.output;
|
||||
await mkdir(output, { mode: 0o700 });
|
||||
const bytes = await readFile(values.target);
|
||||
const home = await mkdtemp(join(tmpdir(), "rea-ghidra-evidence-"));
|
||||
const client = new Client({ name: "ghidra-evidence-projection", version: "1" });
|
||||
const transport = new StdioClientTransport({
|
||||
command: process.execPath,
|
||||
args: [join(values.runtime, "scripts/rea.mjs"), "mcp"],
|
||||
env: {
|
||||
HOME: home,
|
||||
PATH: process.env.PATH,
|
||||
NODE_OPTIONS: "--max-old-space-size=2048",
|
||||
GHIDRA_INSTALL_DIR: values.ghidra,
|
||||
JAVA_HOME: values.java,
|
||||
GHIDRA_HEADLESS_MAXMEM: "768M",
|
||||
JAVA_TOOL_OPTIONS: "-XX:ActiveProcessorCount=1",
|
||||
REA_GHIDRA_STARTUP_TIMEOUT_MS: "600000",
|
||||
REA_LOG_LEVEL: "silent",
|
||||
},
|
||||
stderr: "pipe",
|
||||
});
|
||||
const serverLog = createWriteStream(join(output, "server.log"));
|
||||
const serverLogFinished = finished(serverLog);
|
||||
serverLogFinished.catch(() => {});
|
||||
transport.stderr?.pipe(serverLog);
|
||||
const report = {
|
||||
artifact: { path: values.target, bytes: bytes.length, sha256: hash(bytes) },
|
||||
expected_projection: values.expect,
|
||||
calls: [],
|
||||
};
|
||||
const transcript = [];
|
||||
const evidence = [];
|
||||
const call = createRetainedQueryCaller(client, (receipt, event) => {
|
||||
report.calls.push(receipt);
|
||||
transcript.push(event);
|
||||
});
|
||||
const result = async (name, input, phase = name) => {
|
||||
const reply = await call(name, input, phase);
|
||||
assert.notEqual(reply.isError, true, JSON.stringify(reply));
|
||||
assert.ok(reply.structuredContent);
|
||||
return reply.structuredContent;
|
||||
};
|
||||
const inspect = async (name, input, phase = name) => {
|
||||
const value = await result(name, input, phase);
|
||||
assert.equal(value.subject.digest.sha256, report.artifact.sha256);
|
||||
assert.equal(
|
||||
value.provider.id,
|
||||
name === "binary_overview" ? "rea-workflow" : "ghidra",
|
||||
);
|
||||
assert.equal(
|
||||
value.provider.version,
|
||||
name === "binary_overview" ? "1" : "12.1.4",
|
||||
);
|
||||
assert.ok(value.analysis_profile);
|
||||
evidence.push({ phase, value });
|
||||
return value;
|
||||
};
|
||||
const started = performance.now();
|
||||
let opened = false;
|
||||
try {
|
||||
await client.connect(transport);
|
||||
const discovery = performance.now();
|
||||
const catalog = await client.listTools();
|
||||
assert.equal(catalog.nextCursor, undefined);
|
||||
report.catalog = {
|
||||
elapsed_ms: performance.now() - discovery,
|
||||
bytes: Buffer.byteLength(JSON.stringify(catalog)),
|
||||
sha256: hash(JSON.stringify(catalog)),
|
||||
};
|
||||
await writeFile(join(output, "catalog.json"), JSON.stringify(catalog));
|
||||
const definitions = new Map(catalog.tools.map((tool) => [tool.name, tool]));
|
||||
const original = client.callTool.bind(client);
|
||||
client.callTool = (input, options) =>
|
||||
original(input, {
|
||||
...options,
|
||||
timeout: 600000,
|
||||
toolDefinition: definitions.get(input.name),
|
||||
});
|
||||
await result("open_binary", { path: values.target, provider_id: "ghidra" });
|
||||
opened = true;
|
||||
const documents = await inspect("list_documents", {});
|
||||
assert.equal(documents.normalized_result.length, 1);
|
||||
const procedures = await inspect("list_procedures", {});
|
||||
assert.ok(procedures.normalized_result.length > 0);
|
||||
const selected = procedures.normalized_result.find(
|
||||
(row) => row.procedure.external === false && row.procedure.thunk === false,
|
||||
);
|
||||
assert.ok(selected);
|
||||
report.procedure = selected.address;
|
||||
report.procedure_count = procedures.normalized_result.length;
|
||||
await inspect("binary_overview", {});
|
||||
await inspect("inspect_native_load_image", {});
|
||||
for (const name of [
|
||||
"procedure_info",
|
||||
"procedure_pseudo_code",
|
||||
"procedure_assembly",
|
||||
"procedure_callers",
|
||||
"procedure_callees",
|
||||
"analyze_function",
|
||||
])
|
||||
await inspect(name, { procedure: selected.address });
|
||||
const warmInventory = await inspect("list_procedures", {}, "warm-inventory");
|
||||
assert.deepEqual(
|
||||
warmInventory.normalized_result,
|
||||
procedures.normalized_result,
|
||||
);
|
||||
const warmPseudocode = await inspect(
|
||||
"procedure_pseudo_code",
|
||||
{ procedure: selected.address },
|
||||
"warm-pseudocode",
|
||||
);
|
||||
assert.deepEqual(
|
||||
warmPseudocode.normalized_result,
|
||||
evidence.find(({ phase }) => phase === "procedure_pseudo_code").value
|
||||
.normalized_result,
|
||||
);
|
||||
for (const { phase, value } of evidence) {
|
||||
if (phase === "binary_overview") {
|
||||
assert.equal(value.raw_result, null);
|
||||
continue;
|
||||
}
|
||||
const distinct = phase === "inspect_native_load_image";
|
||||
if (values.expect === "compact" && !distinct)
|
||||
assert.equal(value.raw_result, null, phase);
|
||||
else assert.notEqual(value.raw_result, null, phase);
|
||||
if (values.expect === "duplicate" && !distinct) {
|
||||
const raw = value.raw_result;
|
||||
const wrapped =
|
||||
raw !== null &&
|
||||
!Array.isArray(raw) &&
|
||||
Object.hasOwn(raw, "value") &&
|
||||
Object.hasOwn(raw, "limitations");
|
||||
assert.deepEqual(
|
||||
wrapped ? raw.value : raw,
|
||||
value.normalized_result,
|
||||
phase,
|
||||
);
|
||||
}
|
||||
}
|
||||
const facts = evidence.map(({ phase, value }) => {
|
||||
const { evidence_id: _id, raw_result: _raw, ...facts } = value;
|
||||
return { phase, facts };
|
||||
});
|
||||
report.facts_sha256 = hash(JSON.stringify(facts));
|
||||
await writeFile(join(output, "facts.json"), JSON.stringify(facts));
|
||||
if (values.control)
|
||||
assert.deepEqual(facts, JSON.parse(await readFile(values.control, "utf8")));
|
||||
const path = join(output, "evidence.json");
|
||||
await result("export_evidence_bundle", { path });
|
||||
const bundle = JSON.parse(await readFile(path, "utf8"));
|
||||
for (const { value } of evidence)
|
||||
assert.deepEqual(
|
||||
bundle.records.find((record) => record.evidence_id === value.evidence_id),
|
||||
value,
|
||||
);
|
||||
assert.equal(hash(await readFile(values.target)), report.artifact.sha256);
|
||||
report.status = "passed";
|
||||
} catch (cause) {
|
||||
report.status = "failed";
|
||||
report.failure = String(cause?.stack ?? cause);
|
||||
process.exitCode = 1;
|
||||
} finally {
|
||||
if (opened) {
|
||||
try {
|
||||
await result("close_binary", {});
|
||||
} catch (cause) {
|
||||
report.status = "failed";
|
||||
report.close_failure = String(cause?.stack ?? cause);
|
||||
}
|
||||
}
|
||||
report.elapsed_before_cleanup_ms = performance.now() - started;
|
||||
try {
|
||||
await writeFile(
|
||||
join(output, "transcript.json"),
|
||||
JSON.stringify(transcript),
|
||||
);
|
||||
} catch (cause) {
|
||||
report.status = "failed";
|
||||
report.transcript_failure = String(cause);
|
||||
}
|
||||
const finalized = await finalizeRetainedQueryReceipt({
|
||||
report,
|
||||
client,
|
||||
transport,
|
||||
account: home,
|
||||
serverLog,
|
||||
serverLogFinished,
|
||||
output,
|
||||
});
|
||||
if (report.status !== "passed" || !finalized) process.exitCode = 1;
|
||||
}
|
||||
@@ -1,142 +0,0 @@
|
||||
# Existing Ghidra project reuse: real workflow verification
|
||||
|
||||
The task counts the analyzed functions, reads one 32-record inventory window,
|
||||
reports the overview, decompiles two known local non-thunk functions, repeats the
|
||||
inventory query, exports complete Evidence and verifies it offline. Function
|
||||
bodies and the complete inventory are compared by stable function identity.
|
||||
The agent receives inventory summaries and the selected page; the complete
|
||||
inventory remains accessible through retained Evidence and its exported bundle.
|
||||
|
||||
## Executed scope
|
||||
|
||||
Linux x64, Ghidra 12.1.4, JDK 21.0.12.1, Node 24.22.1. Measurements follow main
|
||||
`e3066c170` with the feature core at `9746ec7d9`; receipt revisions distinguish
|
||||
that driver from subsequent `5d237eb0c` / `314b0e038` refinements. Subsequent core
|
||||
change only improves the setting named for an invalid derived program selection.
|
||||
The final source suites and actual CLI/MCP cases use the updated dependency lock.
|
||||
|
||||
- `/usr/bin/ls`: 652 function records; complete facts SHA-256
|
||||
`d2eb1ea6586b8d549663f98089290dbc020bcc9f9c867545f8ba132aa1a13195`.
|
||||
- `/usr/lib/x86_64-linux-gnu/libcrypto.so.3`: 14,338 function records; artifact SHA-256
|
||||
`aa01e614af634bc7c8b678c9250b5c4f613c61989bacf0a5d6aed6570e7dcabd`;
|
||||
complete facts SHA-256 `033eccbb258317b7d05a4d4a48f049f5d2649b276852623cef3c1fb93e9a029a`.
|
||||
|
||||
Both pairs return identical full function records and the same nonempty pseudocode
|
||||
for each selected address. Evidence IDs/profile identities differ intentionally:
|
||||
existing-project observations commit the caller project content and unknown
|
||||
original analysis history. Successful reuse verifies source project content
|
||||
unchanged after closing; original binary digests remain unchanged too.
|
||||
|
||||
## Matched completed tasks
|
||||
|
||||
Each row includes SDK initialization, one discovery, every call, full bundle
|
||||
export/authentication, connection cleanup and receipt publication. Both columns
|
||||
in a target pair use compact output schemas and the same families:
|
||||
`direct,enhanced,native,application,session`.
|
||||
|
||||
| Task | Functions | Wall s | Family CPU s | Peak family GiB | First / warm inventory s | Wire tokens, estimate |
|
||||
| ---------------- | --------: | -----: | -----------: | --------------: | -----------------------: | --------------------: |
|
||||
| current-ls-fresh | 652 | 63.62 | 84.87 | 1.276 | 55.83 / 0.416 | 85,151 |
|
||||
| current-ls-reuse | 652 | 31.69 | 46.61 | 0.938 | 20.58 / 0.151 | 85,448 |
|
||||
| crypto-fresh | 14,338 | 186.42 | 287.01 | 1.857 | 176.71 / 1.535 | 84,162 |
|
||||
| crypto-reuse | 14,338 | 25.45 | 40.23 | 1.258 | 17.29 / 1.256 | 84,516 |
|
||||
|
||||
The first query starts a new provider process; the warm query reuses that session.
|
||||
OS caches were not flushed. These are sequential samples on a shared host, not a
|
||||
general speed guarantee. Earlier completed small-target calibrations vary
|
||||
substantially (fresh 34.8–56.1 seconds; reuse 19.2 seconds), so elapsed time alone
|
||||
should not be treated as a stable ratio.
|
||||
|
||||
CPU is summed process-family user/system time. Memory sums family RSS sampled
|
||||
every 100 ms; shared pages can be double-counted and short peaks missed. Every
|
||||
run used 2-CPU affinity, an 8-GiB family RSS guard and a wall deadline. Fresh and
|
||||
reuse used the same heap request per target: 1,536 MiB for ls, 3,072 MiB for crypto.
|
||||
No measured workflow triggered OOM, a resource guard or a cleanup failure.
|
||||
|
||||
Token estimates encode every captured JSON-RPC message, including initialize,
|
||||
full discovery schemas, requests, result text/structured content and notifications,
|
||||
with `cl100k_base`. They exclude framing newlines and are not actual model usage.
|
||||
The model adapter, model tokenizer, hidden prompts and billing were not measured.
|
||||
Project provenance adds a small response cost; this feature primarily avoids
|
||||
repeated import/analysis rather than reducing tokens. Complete Evidence files
|
||||
are verified locally and are not inserted into the agent transcript.
|
||||
|
||||
## Preparation cost and scope refinement
|
||||
|
||||
The explicit pre-import step saves a new project outside REA; its time is not
|
||||
hidden in a warm result. With the same 2-CPU bound, ls preparation cost 36.17 s
|
||||
wall, 59.54 CPU s and 0.845 GiB peak family RSS. Crypto preparation cost 231.76 s,
|
||||
251.08 CPU s and 1.615 GiB peak. Both used default analysis followed by the same
|
||||
REA no-return repair that a fresh REA import uses, enabling an equivalent-data
|
||||
comparison. Ordinary caller projects may have different analysis histories.
|
||||
Preparation plus the first investigation can cost more than one fresh import;
|
||||
the benefit is repeated sessions or separately paid analysis that exceeds startup.
|
||||
|
||||
The task does not actually need the native family. A narrower executed ls pair
|
||||
uses `direct,enhanced,application,session`, advertises 97 tools, preserves the same
|
||||
complete facts and costs 67,523 / 67,820 estimated wire tokens for fresh / reuse,
|
||||
versus 85,151 / 85,448 above. Its observed wall/CPU/peak values are 37.35 s / 62.63 s /
|
||||
1.089 GiB fresh and 20.17 s / 32.21 s / 0.886 GiB reuse. These separate samples
|
||||
confirm the smaller catalog works; they do not establish a general speedup from
|
||||
scoping. Use the narrow scope when reproducing the task.
|
||||
|
||||
## Negative cases and verification limits
|
||||
|
||||
The actual production MCP path rejected `/usr/bin/true` against the ls project
|
||||
with the recorded-import digest mismatch. A separate owned source clone changed
|
||||
after open was rejected by the first query with its changed-input reason, then
|
||||
cleaned up; the test restores its own intentional marker edit. Successful files,
|
||||
source content and cleanup outcomes are verified separately. The actual compiled
|
||||
CLI `decompile` selected the same project/profile and returned the same pseudocode
|
||||
as MCP for `0x10a420`.
|
||||
|
||||
Final focused validation: 35 files, 958 passed / 4 platform skips; typecheck/lint,
|
||||
build, 9 documentation cases and full formatting passed. No real macOS, Windows,
|
||||
PE or Mach-O reuse claim is made. Windows and special fresh-import profiles remain
|
||||
explicitly unsupported for this mode. Very large project trees and interrupted
|
||||
cross-process database writes were not measured.
|
||||
|
||||
Initial driver calibrations caught external-first inventory ordering and
|
||||
Evidence-ID ordering across different profiles. The driver now selects known
|
||||
local functions from a bounded middle page and compares pseudocode by function
|
||||
identity. Those failed calibration receipts remain in the maintainer workspace;
|
||||
they are not counted as completed task successes.
|
||||
|
||||
## Reproduce
|
||||
|
||||
Build with the pinned toolchain and locked dependencies. Prepare a new owned
|
||||
project using the documented [pre-import route](../../docs/installation.md#reuse-an-existing-ghidra-project).
|
||||
The measurement used the same no-return repair during preparation for data parity.
|
||||
Run the committed driver first without `--project`, then with it and `--control`:
|
||||
|
||||
```sh
|
||||
node experiments/analysis-efficiency/ghidra-project-reuse.mjs \
|
||||
--runtime /absolute/rea --target /usr/bin/ls --output /absolute/new-receipt \
|
||||
--profile compact --families direct,enhanced,application,session \
|
||||
--ghidra "$GHIDRA_INSTALL_DIR" --java "$JAVA_HOME" \
|
||||
--project /absolute/projects/ls-analysis.gpr --program ls \
|
||||
--control /absolute/fresh-receipt
|
||||
```
|
||||
|
||||
Use `--heap 3072M` for the crypto case. `--expected mismatch` and `--expected stale`
|
||||
exercise negative cases; stale intentionally edits/restores an owned test marker,
|
||||
so use a disposable source copy. The driver retains full wire messages, catalog,
|
||||
report, exported Evidence and normalized comparison facts in its output directory.
|
||||
Resource receipts were collected with an external bounded process-family sampler;
|
||||
its method and limits are described above. Actual model-token usage remains unmeasured.
|
||||
|
||||
## Review boundary regressions
|
||||
|
||||
After the measurement runs, review coverage adds resolved-executable defaults for
|
||||
`.app` inputs (caller bundle paths remain provenance), early unavailable Windows
|
||||
existing-project discovery, exact cancellation reasons from both checkpoints and
|
||||
owned file reads, and selected-pair overlap checks that allow a project directly
|
||||
under `/tmp`. The ambiguous Windows properties fixture cases were removed; the
|
||||
test now covers its actual two distinct release-directory inputs.
|
||||
|
||||
A real Ghidra test copy saved seed and disabled-analyzer reports with REA's actual
|
||||
pre-scripts. Reopening it through production MCP preserved both as one immutable
|
||||
historical Evidence record, linked it from function observations, returned the
|
||||
same function/pseudocode facts, and left source content unchanged. Fresh-import
|
||||
checks still validate fresh requested work. No real macOS/Windows reuse claim is
|
||||
implied by simulated platform boundary tests.
|
||||
@@ -1,377 +0,0 @@
|
||||
/** Equivalent bounded Ghidra investigations with fresh import or an existing project. */
|
||||
import assert from "node:assert/strict";
|
||||
import { execFile } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
import { createWriteStream } from "node:fs";
|
||||
import { mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join, resolve } from "node:path";
|
||||
import { finished } from "node:stream/promises";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { parseArgs, promisify } from "node:util";
|
||||
import { Client } from "@modelcontextprotocol/client";
|
||||
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
|
||||
import { finalizeRetainedQueryReceipt } from "./retained-query-finalization.mjs";
|
||||
|
||||
const { values } = parseArgs({
|
||||
options: Object.fromEntries(
|
||||
[
|
||||
"runtime",
|
||||
"output",
|
||||
"profile",
|
||||
"target",
|
||||
"receive-bytes",
|
||||
"families",
|
||||
"ghidra",
|
||||
"java",
|
||||
"project",
|
||||
"program",
|
||||
"control",
|
||||
"expected",
|
||||
"heap",
|
||||
"history",
|
||||
].map((key) => [key, { type: "string" }]),
|
||||
),
|
||||
});
|
||||
for (const key of ["runtime", "output", "target", "ghidra", "java"]) {
|
||||
assert.ok(values[key], `${key} is required`);
|
||||
values[key] = resolve(values[key]);
|
||||
}
|
||||
assert.ok(["full", "compact"].includes(values.profile));
|
||||
const receiveBytes = Number(values["receive-bytes"] ?? 10 * 1024 * 1024);
|
||||
assert.ok(Number.isSafeInteger(receiveBytes) && receiveBytes > 0);
|
||||
await mkdir(values.output);
|
||||
const report = {
|
||||
profile: values.profile,
|
||||
families: values.families ?? null,
|
||||
receive_bytes: receiveBytes,
|
||||
runtime: values.runtime,
|
||||
target: values.target,
|
||||
project: values.project ?? null,
|
||||
expected: values.expected ?? "success",
|
||||
calls: [],
|
||||
};
|
||||
const transcript = [];
|
||||
let account, client, transport, serverLog, serverLogFinished;
|
||||
let phase = "initialize";
|
||||
const began = performance.now();
|
||||
try {
|
||||
account = await mkdtemp(join(tmpdir(), "rea-output-profile-"));
|
||||
report.target_sha256 = createHash("sha256")
|
||||
.update(await readFile(values.target))
|
||||
.digest("hex");
|
||||
report.revision = (
|
||||
await promisify(execFile)("git", [
|
||||
"-C",
|
||||
values.runtime,
|
||||
"rev-parse",
|
||||
"HEAD",
|
||||
])
|
||||
).stdout.trim();
|
||||
transport = new StdioClientTransport({
|
||||
command: process.execPath,
|
||||
args: [join(values.runtime, "scripts/rea.mjs"), "mcp"],
|
||||
cwd: values.runtime,
|
||||
maxBufferSize: receiveBytes,
|
||||
env: {
|
||||
PATH: process.env.PATH,
|
||||
GHIDRA_INSTALL_DIR: values.ghidra,
|
||||
JAVA_HOME: values.java,
|
||||
GHIDRA_HEADLESS_MAXMEM: values.heap ?? "1536M",
|
||||
JAVA_TOOL_OPTIONS: "-XX:ActiveProcessorCount=2",
|
||||
REA_GHIDRA_STARTUP_TIMEOUT_MS: "600000",
|
||||
...(values.project === undefined
|
||||
? {}
|
||||
: { REA_GHIDRA_PROJECT_PATH: values.project }),
|
||||
...(values.program === undefined
|
||||
? {}
|
||||
: { REA_GHIDRA_PROGRAM_NAME: values.program }),
|
||||
HOME: account,
|
||||
USERPROFILE: account,
|
||||
NODE_OPTIONS: process.env.NODE_OPTIONS ?? "--max-old-space-size=3072",
|
||||
REA_MCP_INPUT_SCHEMA_PROFILE: "compact",
|
||||
REA_MCP_OUTPUT_SCHEMA_PROFILE: values.profile,
|
||||
...(values.families === undefined
|
||||
? {}
|
||||
: { REA_MCP_TOOL_FAMILIES: values.families }),
|
||||
},
|
||||
stderr: "pipe",
|
||||
});
|
||||
const start = transport.start.bind(transport);
|
||||
transport.start = async () => {
|
||||
const onerror = transport.onerror;
|
||||
transport.onerror = (cause) => {
|
||||
(report.transport_errors ??= []).push(String(cause));
|
||||
onerror?.(cause);
|
||||
};
|
||||
const receive = transport.onmessage;
|
||||
transport.onmessage = (message) => {
|
||||
transcript.push({ direction: "response", phase, message });
|
||||
receive?.(message);
|
||||
};
|
||||
await start();
|
||||
};
|
||||
const send = transport.send.bind(transport);
|
||||
transport.send = async (message) => {
|
||||
transcript.push({ direction: "request", phase, message });
|
||||
await send(message);
|
||||
};
|
||||
serverLog = createWriteStream(join(values.output, "server.log"));
|
||||
serverLogFinished = finished(serverLog);
|
||||
serverLogFinished.catch(() => {});
|
||||
transport.stderr.pipe(serverLog);
|
||||
client = new Client({
|
||||
name: "rea-output-profile-investigation",
|
||||
version: "1",
|
||||
});
|
||||
await client.connect(transport);
|
||||
phase = "discovery-cold";
|
||||
const discoveryBegan = performance.now();
|
||||
const catalog = await client.listTools({}, { timeout: 15000 });
|
||||
report.discovery_ms = performance.now() - discoveryBegan;
|
||||
report.catalog_bytes = Buffer.byteLength(JSON.stringify(catalog));
|
||||
report.tools = catalog.tools.length;
|
||||
await writeFile(join(values.output, "catalog.json"), JSON.stringify(catalog));
|
||||
const definitions = new Map(catalog.tools.map((tool) => [tool.name, tool]));
|
||||
const call = async (name, args, label) => {
|
||||
phase = label;
|
||||
const at = performance.now();
|
||||
const result = await client.callTool(
|
||||
{ name, arguments: args },
|
||||
{ timeout: 900000, toolDefinition: definitions.get(name) },
|
||||
);
|
||||
assert.notEqual(
|
||||
result.isError,
|
||||
true,
|
||||
`${name}: ${JSON.stringify(result).slice(0, 500)}`,
|
||||
);
|
||||
report.calls.push({ name, phase, elapsed_ms: performance.now() - at });
|
||||
return result.structuredContent;
|
||||
};
|
||||
let sourceCommitment, projectResources, resolveProject;
|
||||
if (values.project !== undefined) {
|
||||
({ resolveGhidraProject: resolveProject } = await import(
|
||||
pathToFileURL(
|
||||
join(values.runtime, "dist/ghidra/GhidraProjectSnapshot.js"),
|
||||
).href
|
||||
));
|
||||
const { ArtifactResourceScope } = await import(
|
||||
pathToFileURL(
|
||||
join(values.runtime, "dist/artifacts/ArtifactResourceScope.js"),
|
||||
).href
|
||||
);
|
||||
projectResources = new ArtifactResourceScope();
|
||||
try {
|
||||
sourceCommitment = await resolveProject(
|
||||
{
|
||||
path: values.project,
|
||||
...(values.program === undefined
|
||||
? {}
|
||||
: { programName: values.program }),
|
||||
},
|
||||
{ path: values.target },
|
||||
projectResources,
|
||||
);
|
||||
} finally {
|
||||
await projectResources.close();
|
||||
}
|
||||
report.project_sha256 = sourceCommitment.sha256;
|
||||
}
|
||||
await call(
|
||||
"open_binary",
|
||||
{ path: values.target, provider_id: "ghidra" },
|
||||
"open",
|
||||
);
|
||||
if (["mismatch", "stale"].includes(values.expected)) {
|
||||
phase = "mismatched-project";
|
||||
const marker =
|
||||
values.expected === "stale" ? await readFile(values.project) : undefined;
|
||||
if (marker !== undefined)
|
||||
await writeFile(
|
||||
values.project,
|
||||
Buffer.concat([marker, Buffer.from("\n")]),
|
||||
);
|
||||
let rejected;
|
||||
try {
|
||||
rejected = await client.callTool(
|
||||
{ name: "list_procedures", arguments: { detail: "summary" } },
|
||||
{ timeout: 900000, toolDefinition: definitions.get("list_procedures") },
|
||||
);
|
||||
} finally {
|
||||
if (marker !== undefined) await writeFile(values.project, marker);
|
||||
}
|
||||
assert.equal(rejected.isError, true);
|
||||
assert.match(
|
||||
JSON.stringify(rejected),
|
||||
values.expected === "stale"
|
||||
? /changed while being read|artifact.changed/iu
|
||||
: /imported-byte digest does not match/u,
|
||||
);
|
||||
await writeFile(
|
||||
join(values.output, "rejection.json"),
|
||||
JSON.stringify(rejected),
|
||||
);
|
||||
report[`rejected_${values.expected}`] = true;
|
||||
} else {
|
||||
const summary = await call(
|
||||
"list_procedures",
|
||||
{ detail: "summary" },
|
||||
"inventory-cold",
|
||||
);
|
||||
const parentId = summary.normalized_result.parent_evidence_id;
|
||||
const count = summary.normalized_result.summary.row_count;
|
||||
assert.ok(count > 0);
|
||||
report.procedure_count = count;
|
||||
const page = await call(
|
||||
"inspect_analysis_view",
|
||||
{
|
||||
source: { kind: "retained-evidence", evidence_id: parentId },
|
||||
view: {
|
||||
kind: "native-inventory",
|
||||
offset: Math.floor(count / 2),
|
||||
limit: Math.min(count - Math.floor(count / 2), 32),
|
||||
fields: ["value", "procedure"],
|
||||
},
|
||||
},
|
||||
"selected-procedures",
|
||||
);
|
||||
const selected = page.normalized_result.rows
|
||||
.map((row) => row.record)
|
||||
.filter(
|
||||
(row) =>
|
||||
row.procedure?.external === false && row.procedure?.thunk === false,
|
||||
)
|
||||
.slice(0, 2);
|
||||
assert.equal(selected.length, 2);
|
||||
await call("binary_overview", {}, "overview");
|
||||
for (const row of selected)
|
||||
await call(
|
||||
"procedure_pseudo_code",
|
||||
{ procedure: row.address },
|
||||
`pseudocode:${row.address}`,
|
||||
);
|
||||
const warm = await call(
|
||||
"list_procedures",
|
||||
{ detail: "summary" },
|
||||
"inventory-warm",
|
||||
);
|
||||
assert.equal(warm.normalized_result.parent_evidence_id, parentId);
|
||||
const bundlePath = join(values.output, "complete-evidence.json");
|
||||
await call(
|
||||
"export_evidence_bundle",
|
||||
{ path: bundlePath },
|
||||
"complete-export",
|
||||
);
|
||||
await call("close_binary", {}, "close");
|
||||
await client.close();
|
||||
await transport.close();
|
||||
const { parseEvidenceBundle } = await import(
|
||||
pathToFileURL(join(values.runtime, "dist/domain/evidenceBundle.js")).href
|
||||
);
|
||||
const bundle = parseEvidenceBundle(
|
||||
JSON.parse(await readFile(bundlePath, "utf8")),
|
||||
);
|
||||
const parent = bundle.records.find(
|
||||
(record) => record.evidence_id === parentId,
|
||||
);
|
||||
assert.ok(parent);
|
||||
if (values.history === "required") {
|
||||
const historical = bundle.records.filter(
|
||||
(record) =>
|
||||
record.operation === "health" &&
|
||||
record.parameters.facet === "historical-project-analysis-reports",
|
||||
);
|
||||
assert.equal(historical.length, 1);
|
||||
assert.equal(historical[0].normalized_result.origin, "caller-project");
|
||||
assert.ok(historical[0].normalized_result.analysis_seeds);
|
||||
assert.ok(historical[0].normalized_result.disabled_analyzers);
|
||||
report.historical_evidence_id = historical[0].evidence_id;
|
||||
report.historical_evidence_linked =
|
||||
parent.evidence_links?.includes(historical[0].evidence_id) ?? false;
|
||||
}
|
||||
assert.equal(parent.normalized_result.length, count);
|
||||
const facts = {
|
||||
procedures: parent.normalized_result,
|
||||
pseudocode: bundle.records
|
||||
.filter((record) => record.operation === "procedure_pseudo_code")
|
||||
.map((record) => ({
|
||||
procedure: record.parameters.procedure,
|
||||
result: record.normalized_result,
|
||||
})),
|
||||
};
|
||||
facts.pseudocode.sort((left, right) =>
|
||||
left.procedure < right.procedure
|
||||
? -1
|
||||
: left.procedure > right.procedure
|
||||
? 1
|
||||
: 0,
|
||||
);
|
||||
report.facts_sha256 = createHash("sha256")
|
||||
.update(JSON.stringify(facts))
|
||||
.digest("hex");
|
||||
await writeFile(join(values.output, "facts.json"), JSON.stringify(facts));
|
||||
report.analysis_profile = parent.analysis_profile;
|
||||
assert.equal(parent.subject.digest.sha256, report.target_sha256);
|
||||
if (sourceCommitment !== undefined) {
|
||||
assert.equal(
|
||||
parent.analysis_profile.parameters.existing_project.sha256,
|
||||
sourceCommitment.sha256,
|
||||
);
|
||||
assert.ok(
|
||||
parent.limitations.some(
|
||||
(value) =>
|
||||
value.includes("completeness") && value.includes("unknown"),
|
||||
),
|
||||
);
|
||||
}
|
||||
if (values.control !== undefined) {
|
||||
const control = JSON.parse(
|
||||
await readFile(join(values.control, "facts.json"), "utf8"),
|
||||
);
|
||||
assert.deepEqual(facts, control);
|
||||
}
|
||||
}
|
||||
if (sourceCommitment !== undefined) {
|
||||
const { ArtifactResourceScope } = await import(
|
||||
pathToFileURL(
|
||||
join(values.runtime, "dist/artifacts/ArtifactResourceScope.js"),
|
||||
).href
|
||||
);
|
||||
const resources = new ArtifactResourceScope();
|
||||
let after;
|
||||
try {
|
||||
after = await resolveProject(
|
||||
{
|
||||
path: values.project,
|
||||
...(values.program === undefined
|
||||
? {}
|
||||
: { programName: values.program }),
|
||||
},
|
||||
{ path: values.target },
|
||||
resources,
|
||||
);
|
||||
} finally {
|
||||
await resources.close();
|
||||
}
|
||||
assert.deepEqual(after, sourceCommitment);
|
||||
report.original_project_unchanged = true;
|
||||
}
|
||||
report.outcome = "passed";
|
||||
} catch (cause) {
|
||||
report.outcome = "failed";
|
||||
report.failure = String(cause);
|
||||
process.exitCode = 1;
|
||||
} finally {
|
||||
report.elapsed_ms = performance.now() - began;
|
||||
const published = await finalizeRetainedQueryReceipt({
|
||||
report,
|
||||
client,
|
||||
transport,
|
||||
account,
|
||||
serverLog,
|
||||
serverLogFinished,
|
||||
output: values.output,
|
||||
transcript,
|
||||
});
|
||||
if (!published) process.exitCode = 1;
|
||||
}
|
||||
@@ -1,74 +0,0 @@
|
||||
"""Count verified Go tasks; tokenizer estimates are not model usage or billing."""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import tiktoken
|
||||
|
||||
|
||||
def compact(value):
|
||||
return json.dumps(value, ensure_ascii=False, separators=(",", ":"), allow_nan=False)
|
||||
|
||||
|
||||
def inspect(path, encoding):
|
||||
report = json.loads((path / "report.json").read_text())
|
||||
catalog = json.loads((path / "catalog.json").read_text())
|
||||
transcript = json.loads((path / "transcript.json").read_text())
|
||||
log = path.with_suffix(".log")
|
||||
scope = next(json.loads(line) for line in reversed(log.read_text().splitlines())
|
||||
if line.startswith('{"resource_scope":'))
|
||||
observed = scope.get("payload_resource_observation") or {}
|
||||
events = observed.get("resource_events", {})
|
||||
memory = events.get("memory.events", {})
|
||||
verified = (report["status"] == "observed"
|
||||
and not report["cleanup_failures"] and not report["receipt_failures"]
|
||||
and scope["abort"] is None
|
||||
and scope.get("payload_resource_observation_verified") is True
|
||||
and observed.get("payload_cleanup_verified") is True
|
||||
and observed.get("payload_exit_code") == 0
|
||||
and all(memory.get(key) == 0 for key in ["high", "max", "oom", "oom_kill"])
|
||||
and events.get("pids.events", {}).get("max") == 0)
|
||||
count = lambda value: len(encoding.encode(value, disallowed_special=()))
|
||||
inputs = count(compact([{key: tool[key] for key in ["name", "description", "inputSchema"]}
|
||||
for tool in catalog["tools"]]))
|
||||
arguments = sum(count(compact({"name": event["name"], "arguments": event["input"]}))
|
||||
for event in transcript)
|
||||
replies = sum(count(block["text"]) for event in transcript
|
||||
for block in event["reply"]["content"] if block["type"] == "text")
|
||||
cpu = observed.get("cpu_stat", {}).get("usage_usec")
|
||||
return {"case": path.name, "completed_with_verified_cleanup": verified,
|
||||
"mode": report["mode"], "warm_diagnostic": report["warm_diagnostic"],
|
||||
"answer": report["answer"], "parent_evidence_id": report["parent_evidence_id"],
|
||||
"artifact_sha256": report["artifact_sha256"],
|
||||
"complete_result_sha256": report["complete_result_file_sha256"],
|
||||
"calls": report["calls"], "wall_seconds": scope["elapsed_seconds"],
|
||||
"cpu_seconds": None if cpu is None else cpu / 1e6,
|
||||
"kernel_family_peak_bytes": observed.get("memory_peak_bytes"),
|
||||
"events_at_payload_exit": events,
|
||||
"final_whole_scope_events": scope.get("resource_events_final"),
|
||||
"token_proxy": {"input_catalog": inputs, "arguments": arguments,
|
||||
"text_replies": replies, "combined": inputs + arguments + replies,
|
||||
"complete_transport_catalog": count(compact(catalog))},
|
||||
"sha256": {file.name: hashlib.sha256(file.read_bytes()).hexdigest()
|
||||
for file in [path / "report.json", path / "catalog.json",
|
||||
path / "transcript.json", log]}}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument("cases", type=Path, nargs="+")
|
||||
args = parser.parse_args()
|
||||
encoding = tiktoken.get_encoding("cl100k_base")
|
||||
records = [inspect(path, encoding) for path in args.cases]
|
||||
receipt = {"tiktoken": tiktoken.__version__, "encoding": encoding.name,
|
||||
"actual_model_usage": None,
|
||||
"projection": "All discovered input schemas once, plus every call's arguments and text replies, including export. Full transport catalog counted separately; no model turn or host injection measured.",
|
||||
"recordings": records}
|
||||
args.output.write_text(json.dumps(receipt, ensure_ascii=False, indent=2) + "\n")
|
||||
assert all(record["completed_with_verified_cleanup"] for record in records)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,81 +0,0 @@
|
||||
"""Pin a completed Go report only after checking native build text and file bytes."""
|
||||
import argparse
|
||||
import base64
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--evidence", type=Path, required=True)
|
||||
parser.add_argument("--input", type=Path, required=True)
|
||||
parser.add_argument("--go", type=Path, required=True)
|
||||
parser.add_argument("--dependency", required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
source = args.input.read_bytes()
|
||||
evidence = json.loads(args.evidence.read_text())
|
||||
if "records" in evidence:
|
||||
evidence = next(record for record in evidence["records"]
|
||||
if record["operation"] == "inspect_go_binary"
|
||||
and record["parameters"] == {"path": str(args.input)})
|
||||
report = evidence["normalized_result"]
|
||||
info = report["build_info"]
|
||||
digest = hashlib.sha256(source).hexdigest()
|
||||
assert report["artifact"] == {"path": str(args.input), "sha256": digest,
|
||||
"bytes": len(source)}
|
||||
native = subprocess.check_output(
|
||||
[str(args.go), "version", "-m", str(args.input)], text=True)
|
||||
lines = native.splitlines()
|
||||
assert info["go_version"] == lines[0].rsplit(" ", 1)[1]
|
||||
text = "\n".join(line.removeprefix("\t") for line in lines[1:]) + "\n"
|
||||
assert info["module_text"] == text
|
||||
for name in ["version", "module"]:
|
||||
location = info[name + "_location"]
|
||||
key = ("go_version" if name == "version" else "module") + "_bytes_base64"
|
||||
assert source[location["offset"]:location["offset"] + location["bytes"]] == base64.b64decode(info[key])
|
||||
# Module identity columns are literal tab-separated fields. Preserve empty
|
||||
# sums, development versions, original ordering and local replacements.
|
||||
dependencies = []
|
||||
main_module = None
|
||||
previous = None
|
||||
for line in text.splitlines():
|
||||
columns = line.split("\t")
|
||||
tag = columns[0]
|
||||
if tag not in ["dep", "mod", "=>"]:
|
||||
continue
|
||||
assert len(columns) in [3, 4], columns
|
||||
item = {"path": columns[1], "version": columns[2],
|
||||
"sum": columns[3] if len(columns) == 4 else None}
|
||||
if tag == "=>":
|
||||
assert previous is not None
|
||||
previous["replacement"] = item
|
||||
else:
|
||||
item["replacement"] = None
|
||||
previous = item
|
||||
if tag == "mod":
|
||||
main_module = item
|
||||
else:
|
||||
dependencies.append(item)
|
||||
assert info["module"]["main"] == main_module
|
||||
assert info["module"]["dependencies"] == dependencies
|
||||
answer = {"go_version": info["go_version"], "main": main_module,
|
||||
"runtime_dependencies": "unknown",
|
||||
"dependencies": [{"index": index, **item}
|
||||
for index, item in enumerate(dependencies)
|
||||
if item["path"] == args.dependency]}
|
||||
complete = args.output.with_suffix(".complete-result.json")
|
||||
complete.write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n")
|
||||
receipt = {"artifact_sha256": digest, "dependency_path": args.dependency,
|
||||
"dependency_count": len(dependencies), "answer": answer,
|
||||
"complete_result_path": str(complete.resolve()),
|
||||
"native_text_sha256": hashlib.sha256(native.encode()).hexdigest(),
|
||||
"native_go_path": str(args.go),
|
||||
"native_go_sha256": hashlib.sha256(args.go.read_bytes()).hexdigest()}
|
||||
args.output.write_text(json.dumps(receipt, ensure_ascii=False, indent=2) + "\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,195 +0,0 @@
|
||||
# Selected Go build metadata
|
||||
|
||||
Addresses [#5499](https://github.com/morluto/rea/issues/5499). A compiler/main-module
|
||||
identity question previously required the complete Go report, including raw framed
|
||||
bytes, base64, module text and every parsed dependency. Opt-in `detail: summary`
|
||||
retains the same complete parent and returns identity, source ranges, counts and
|
||||
explicit unknowns. Default complete delivery remains compatible.
|
||||
|
||||
`inspect_analysis_view` reuses that parent for `kind: go`, collection
|
||||
`dependencies` or `settings`. Exact paths/keys and original indices intersect;
|
||||
fields select columns, while every row keeps its original index. Pagination
|
||||
counts matching rows in producer order, including duplicate module paths. The
|
||||
view never reopens a target or reparses module text. Unknown lines and original
|
||||
bytes remain in the complete parent. CLI summaries save that parent in a new
|
||||
Evidence bundle before returning their projection.
|
||||
|
||||
Only immutable authenticated Evidence results reuse schema validation, keyed
|
||||
by their object identity for the Evidence lifetime. Mutable inline results are
|
||||
validated on each call. Subject identity is checked separately on every view.
|
||||
There is no live-file cache. Only page rows are projected; inherited messages
|
||||
use the existing bounded, deduplicating view-message owner.
|
||||
|
||||
## Question and independent checks
|
||||
|
||||
The main artifact is the official GitHub CLI v2.102.0 Linux amd64 release:
|
||||
42,086,560 bytes, SHA-256
|
||||
`7469124f706944133d6a169691dd1c6c3511b12e85878d255e044e2948df4c9b`.
|
||||
The archive digest matches its official GitHub asset metadata. Its embedded
|
||||
compiler is `go1.27.1`, main module `github.com/cli/cli/[email protected]`, and it has
|
||||
165 parsed dependency records. The selected `golang.org/x/[email protected]` record
|
||||
keeps original index 152. These are embedded observations, not authentication
|
||||
of the compiler, supply chain or runtime behavior.
|
||||
|
||||
Native `go version -m` independently matches module text, main identity and all
|
||||
165 dependency identities in original order. Original file ranges match the
|
||||
reported compiler and framed module bytes. Before/after exports have identical
|
||||
complete normalized results and parent Evidence identity. The small real case
|
||||
is the installed 12,366,352-byte Go tool: embedded `go1.23.5`, no decoded main
|
||||
module and zero retained dependencies, with dependency coverage still unknown.
|
||||
REA reads both inputs without executing them. The installed Go tool also runs
|
||||
as the trusted native verifier; the downloaded GitHub CLI binary is never
|
||||
executed.
|
||||
|
||||
Timing baseline is main `ee12d517a25d34853d40e31fc039c7d1576c60e1`. Packaged runtimes
|
||||
and source patch are pinned in the [receipt](go-build-views.json); later changes
|
||||
do not relabel these timings. Node 24.18.0/npm 11.16.0, MCP SDK 2.3.1, Linux
|
||||
x86-64. Fresh processes use unflushed OS caches; the SDK controller pre-reads
|
||||
the input for verification. These are not disk-cold measurements.
|
||||
|
||||
## Actual client context, including every request
|
||||
|
||||
The [wire driver](go-build-wire.py) starts actual Codex app-server 0.163.0 with a
|
||||
fresh account/workspace and a loopback deterministic Responses SSE backend.
|
||||
It starts a scripted turn and real MCP calls, with **no AI inference or paid
|
||||
service**. All 210 tools remain discovered/advertised. Full input/output
|
||||
profiles are explicit, and each relevant tool has a 30,000-token output budget
|
||||
so complete JSON is delivered without retries or truncation. This does not
|
||||
qualify another client's defaults or a real model's reasoning/schema acceptance.
|
||||
|
||||
All original facts and parent identities match the native oracle. Export cases
|
||||
also verify the complete parent. The [counter](go-build-wire-costs.py) counts
|
||||
**every captured request**, including repeated schemas and accumulated tool
|
||||
outputs, using `tiktoken 0.12.0` / `cl100k_base` on compact request JSON. These are
|
||||
serialization-token proxies; actual model rendering, usage and cache billing
|
||||
remain unmeasured.
|
||||
|
||||
| Completed question | Requests before → after | Whole request JSON proxy before → after |
|
||||
| ------------------------------------------------------ | ----------------------: | --------------------------------------: |
|
||||
| Compiler, main module and dependency count | 2 → 2 | 158,529 → 123,368 (22.2% less) |
|
||||
| Compiler, main module and one selected dependency | 2 → 3 | 158,538 → 186,812 (17.8% more) |
|
||||
| Same selected dependency question plus Evidence export | 3 → 4 | 256,342 → 250,352 (2.3% less) |
|
||||
|
||||
The actual REA namespace costs 51,821 → 51,983 proxy tokens **per request**.
|
||||
An extra round can outweigh a smaller reply. Summary is useful when identity
|
||||
and counts already answer the question; complete delivery remains useful for
|
||||
an initial compound question. Retained views provide selected follow-up facts,
|
||||
without a claim that every such composition saves total context.
|
||||
|
||||
One recording per cell: identity wall/CPU/family peak is
|
||||
6.970 s / 6.331 s / 861.5 MiB before and 7.069 s / 6.457 s / 857.0 MiB after.
|
||||
Compound answer-only is 6.973 / 6.333 / 865.0 before and 6.969 / 6.348 / 861.6
|
||||
after. Export is 7.175 / 6.537 / 866.7 before and 7.274 / 6.568 / 872.6 after.
|
||||
These scripted-client observations establish no universal latency or memory
|
||||
gain, and do not measure model inference time.
|
||||
|
||||
## Direct SDK task and cold/warm costs
|
||||
|
||||
The [SDK driver](go-build-views.mjs) discovers all tools, answers the compound
|
||||
question, verifies exact facts and exports complete Evidence. Before: one
|
||||
complete inspection and export. After: summary, selected dependency and export.
|
||||
For the zero-dependency small case, the summary avoids an unnecessary empty
|
||||
query. Two repetitions alternate version/target order. All 208 other named
|
||||
catalog definitions agree; only Go inspection and the retained-view tool change.
|
||||
|
||||
| Input | Whole wall s before → after | Family CPU s before → after | Kernel family peak MiB before → after |
|
||||
| ---------- | --------------------------- | --------------------------- | ------------------------------------- |
|
||||
| GitHub CLI | 4.855–4.952 → 6.370–6.564 | 4.682–4.725 → 6.175–6.400 | 671.2–680.1 → 846.4–868.7 |
|
||||
| Go tool | 4.349–4.351 → 4.250–4.450 | 4.093–4.140 → 4.092–4.179 | 548.4–548.6 → 544.3–555.6 |
|
||||
|
||||
Cold selected-query calls take 1.719–1.751 s in the SDK task, including its
|
||||
output-schema validation. A separate warm diagnostic takes 1.712 s for the first
|
||||
query and 32.86 ms for the repeat in one session; both the client validator and
|
||||
retained facts are warm. This is not a paired whole-task speedup. The complete
|
||||
warm diagnostic is separately recorded, not mixed into the eight matched runs.
|
||||
|
||||
The [SDK counter](go-build-view-costs.py) counts the discovered input catalog
|
||||
once plus all arguments and text replies, including export: 229,183 → 195,643
|
||||
for GitHub CLI and 194,854 → 194,724 for the small case. Catalog cost grows
|
||||
193,192 → 193,652. This catalog-once projection is **not** the actual host
|
||||
exchange above; omitting repeated schemas would hide the compound-task penalty.
|
||||
|
||||
All qualified recordings use one CPU, a 4 GiB hard/3 GiB early family memory
|
||||
boundary, no swap and 192 tasks. Verified payload exits are zero, only the
|
||||
observer remains, and observed memory/task resource events are zero. Kernel
|
||||
family peaks include attributed file cache. Final whole-scope counters are
|
||||
unavailable. Release acquisition, oracle preparation and compilation are outside
|
||||
the timed tasks. An initial symlink input was correctly refused; qualification
|
||||
explicitly selects its regular target. One development static check was stopped
|
||||
by the host-available-memory guard, with owned cleanup verified and no observed
|
||||
OOM; the unchanged guard succeeded after headroom recovered.
|
||||
|
||||
## Coverage and reproduction
|
||||
|
||||
Executed: 10 files / 202 tests covering existing Go readers, mappings, provider,
|
||||
CLI/MCP and Rust readers, plus selected rows, duplicate paths, empty filters,
|
||||
identity-only fields, malformed/mutable parents, source conflicts, bounded
|
||||
messages and unavailable metadata. Actual CLI complete output is identical
|
||||
before/after; summary bundles retain that exact parent, and file-based CLI
|
||||
selected queries match the native oracle. Full `npm run check` passed.
|
||||
After rebasing onto main `c374c2fe7`, the 202 focused tests, full
|
||||
`npm run check` and `npm run docs:check` (including five README consistency
|
||||
tests) pass again. A real SDK selected query and complete export also pass
|
||||
against that rebuilt runtime, retaining the original parent identity and all
|
||||
165 dependencies. These checks do not relabel the frozen timing cohort.
|
||||
|
||||
A review found that a transcript-write failure could skip the SDK driver's
|
||||
cleanup. The shared finalizer now closes owned resources before publishing the
|
||||
optional transcript and records publication failures separately. All 11 receipt
|
||||
failure tests pass. A real selected SDK task and full export pass again; the
|
||||
same task with an injected transcript-write failure exits 1, preserves its
|
||||
answer and failure receipt, and verifies that the process family was cleaned.
|
||||
These qualification runs do not replace the frozen performance measurements.
|
||||
|
||||
A second review found that a valid external parent could disagree with its
|
||||
report's architecture. Go domain now owns the existing architecture-to-subject
|
||||
mapping; both Evidence production and every Go view use it. The view checks
|
||||
mapped architecture as well as path, digest and format, including `null` for
|
||||
unrecognized report names. All 213 focused tests pass, including authenticated
|
||||
inline/file conflicts and known-alias/unknown-architecture cases.
|
||||
|
||||
Initial CI also exposed two existing baseline failures. On unmodified `c374c2fe7`,
|
||||
`src/cliJsonPaths.test.ts` has the same three URL-message assertion failures.
|
||||
The duplication gate is already above its 3.5% threshold: 31,737 / 899,602 lines
|
||||
(3.52789%) on baseline versus 31,748 / 900,474 (3.52570%) on the initial PR head.
|
||||
These were reproduced separately with the pinned toolchain, bounded resources
|
||||
and verified cleanup; this PR leaves those unrelated failures unchanged.
|
||||
|
||||
Final qualification rebased onto main `cf5194f68`, preserving its new IL2CPP
|
||||
view contract alongside the Go additions. All 258 tests in 12 files pass,
|
||||
including 32 IL2CPP view tests and the now-fixed CLI path assertions. Full
|
||||
`npm run check`, `npm run docs:check` plus five README tests, and the real SDK
|
||||
selected query/full export pass again. Complete parent identity and all 165
|
||||
dependencies remain exact. Duplication now passes: baseline 31,644 / 904,778
|
||||
lines (3.49743%), PR 31,612 / 905,740 (3.49018%), against 3.5%.
|
||||
One latest-main build was guard-stopped during a host-memory dip; its owned
|
||||
scope was verified inactive with zero OOM events, then the unchanged guard's
|
||||
retry passed. These results qualify the rebased source, not new task timings.
|
||||
|
||||
The broader metadata audit also read three real Rust binaries. Alpine ripgrep
|
||||
14.1.1-r0's original `.dep-v0` matches native `readelf` coordinates and independent
|
||||
zlib/JSON decoding: 42 packages, 88 indexed edges, root index 33. The other two
|
||||
have no dependency section, which remains unknown coverage. Compiler-path
|
||||
inferences in all three match native read-only ranges and original byte offsets;
|
||||
they remain inferences. These fast metadata reads did not justify a Rust cache.
|
||||
The Alpine archive is pinned by observed SHA-256; APK signatures were not checked.
|
||||
Real provider verification here covers Linux ELF; PE/Mach-O use authored fixtures.
|
||||
|
||||
To reproduce, use built baseline/candidate runtimes and the pinned public input.
|
||||
Create a baseline complete CLI Evidence JSON, then pin the independent oracle:
|
||||
|
||||
```sh
|
||||
python3 experiments/analysis-efficiency/go-build-view-oracle.py \
|
||||
--evidence /tmp/baseline-evidence.json --input /absolute/gh \
|
||||
--go /absolute/go --dependency golang.org/x/crypto --output /tmp/go-oracle.json
|
||||
```
|
||||
|
||||
Run `go-build-views.mjs` with absolute `--runtime`, `--input`, `--oracle` and
|
||||
new `--output` directories, and `--mode complete` or `selected`, under
|
||||
`resource-run.py --observe-payload-exit`. `--warm` adds a separate diagnostic.
|
||||
The wire driver additionally requires pinned `--codex` and `--node`, and takes
|
||||
`--question identity --mode complete|summary` or
|
||||
`--question dependency --mode complete|selected`; `--export` adds export.
|
||||
All wire recordings require the same resource guard for owned-scope cleanup.
|
||||
Counter scripts take `--output` and recording directories, with their guard
|
||||
logs at the same path plus `.log`. Both counters require `tiktoken==0.12.0`.
|
||||
@@ -1,228 +0,0 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { createHash } from "node:crypto";
|
||||
import { createWriteStream } from "node:fs";
|
||||
import { mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises";
|
||||
import { createRequire } from "node:module";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join, resolve } from "node:path";
|
||||
import { finished } from "node:stream/promises";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { parseArgs } from "node:util";
|
||||
import { createRetainedQueryCaller } from "./retained-query-calls.mjs";
|
||||
import { finalizeRetainedQueryReceipt } from "./retained-query-finalization.mjs";
|
||||
|
||||
const { values } = parseArgs({
|
||||
options: {
|
||||
runtime: { type: "string" },
|
||||
input: { type: "string" },
|
||||
oracle: { type: "string" },
|
||||
output: { type: "string" },
|
||||
mode: { type: "string" },
|
||||
warm: { type: "boolean", default: false },
|
||||
},
|
||||
});
|
||||
for (const key of ["runtime", "input", "oracle", "output"])
|
||||
assert.equal(values[key], resolve(values[key]));
|
||||
assert.ok(["complete", "selected"].includes(values.mode));
|
||||
const oracle = JSON.parse(await readFile(values.oracle, "utf8"));
|
||||
const sha256 = (bytes) => createHash("sha256").update(bytes).digest("hex");
|
||||
const bytes = await readFile(values.input);
|
||||
assert.equal(sha256(bytes), oracle.artifact_sha256);
|
||||
const require = createRequire(join(values.runtime, "package.json"));
|
||||
const { Client } = await import(
|
||||
pathToFileURL(require.resolve("@modelcontextprotocol/client"))
|
||||
);
|
||||
const { StdioClientTransport } = await import(
|
||||
pathToFileURL(require.resolve("@modelcontextprotocol/client/stdio"))
|
||||
);
|
||||
await mkdir(values.output, { mode: 0o700 });
|
||||
const account = await mkdtemp(join(tmpdir(), "rea-go-build-views-"));
|
||||
const report = {
|
||||
question:
|
||||
"Report compiler, main module and the selected embedded dependency; preserve complete original Evidence.",
|
||||
mode: values.mode,
|
||||
warm_diagnostic: values.warm,
|
||||
calls: [],
|
||||
artifact_sha256: oracle.artifact_sha256,
|
||||
};
|
||||
const client = new Client({ name: "rea-go-build-views", version: "1" });
|
||||
const transport = new StdioClientTransport({
|
||||
command: process.execPath,
|
||||
args: [join(values.runtime, "scripts/rea.mjs"), "mcp"],
|
||||
env: {
|
||||
HOME: account,
|
||||
PATH: process.env.PATH,
|
||||
NODE_OPTIONS: "--max-old-space-size=2048",
|
||||
UV_THREADPOOL_SIZE: "1",
|
||||
REA_LOG_LEVEL: "silent",
|
||||
},
|
||||
stderr: "pipe",
|
||||
});
|
||||
const serverLog = createWriteStream(join(values.output, "server.log"));
|
||||
const serverLogFinished = finished(serverLog);
|
||||
serverLogFinished.catch(() => {});
|
||||
transport.stderr?.pipe(serverLog);
|
||||
const transcript = [];
|
||||
const call = createRetainedQueryCaller(client, (receipt, event) => {
|
||||
report.calls.push(receipt);
|
||||
transcript.push(event);
|
||||
});
|
||||
async function result(name, input, phase = name) {
|
||||
const reply = await call(name, input, phase);
|
||||
assert.ok(!reply.isError, JSON.stringify(reply));
|
||||
assert.ok(reply.structuredContent);
|
||||
assert.equal(reply.content.length, 1);
|
||||
assert.equal(reply.content[0].type, "text");
|
||||
assert.deepEqual(JSON.parse(reply.content[0].text), reply.structuredContent);
|
||||
return reply.structuredContent;
|
||||
}
|
||||
try {
|
||||
await client.connect(transport);
|
||||
const catalog = await client.listTools();
|
||||
assert.equal(catalog.nextCursor, undefined);
|
||||
report.catalog_tools = catalog.tools.length;
|
||||
await writeFile(join(values.output, "catalog.json"), JSON.stringify(catalog));
|
||||
const definitions = new Map(catalog.tools.map((tool) => [tool.name, tool]));
|
||||
const original = client.callTool.bind(client);
|
||||
client.callTool = (input, options) =>
|
||||
original(input, {
|
||||
...options,
|
||||
timeout: 60000,
|
||||
toolDefinition: definitions.get(input.name),
|
||||
});
|
||||
const inspected = await result(
|
||||
"inspect_go_binary",
|
||||
{
|
||||
path: values.input,
|
||||
...(values.mode === "selected" ? { detail: "summary" } : {}),
|
||||
},
|
||||
"inspection:first",
|
||||
);
|
||||
let answer;
|
||||
if (values.mode === "complete") {
|
||||
report.parent_evidence_id = inspected.evidence_id;
|
||||
const info = inspected.normalized_result.build_info;
|
||||
assert.ok(info);
|
||||
answer = {
|
||||
go_version: info.go_version,
|
||||
main: info.module.main,
|
||||
runtime_dependencies: "unknown",
|
||||
dependencies: info.module.dependencies.flatMap((row, index) =>
|
||||
row.path === oracle.dependency_path ? [{ index, ...row }] : [],
|
||||
),
|
||||
};
|
||||
} else {
|
||||
const summary = inspected.normalized_result;
|
||||
assert.equal(summary.kind, "summary");
|
||||
report.parent_evidence_id = summary.parent_evidence_id;
|
||||
assert.equal(summary.source_coverage.runtime_dependencies, "unknown");
|
||||
assert.equal(
|
||||
summary.summary.build_info.module.available_counts.dependencies,
|
||||
oracle.dependency_count,
|
||||
);
|
||||
const input = {
|
||||
source: {
|
||||
kind: "retained-evidence",
|
||||
evidence_id: report.parent_evidence_id,
|
||||
},
|
||||
view: {
|
||||
kind: "go",
|
||||
collection: "dependencies",
|
||||
paths: [oracle.dependency_path],
|
||||
fields: ["path", "version", "sum", "replacement"],
|
||||
offset: 0,
|
||||
limit: 64,
|
||||
},
|
||||
};
|
||||
let dependencies = [];
|
||||
if (summary.summary.build_info.module.available_counts.dependencies > 0) {
|
||||
const selected = await result(
|
||||
"inspect_analysis_view",
|
||||
input,
|
||||
"retained-query:first",
|
||||
);
|
||||
assert.equal(selected.normalized_result.coverage.exhausted, true);
|
||||
assert.equal(
|
||||
selected.normalized_result.source_coverage.runtime_dependencies,
|
||||
"unknown",
|
||||
);
|
||||
dependencies = selected.normalized_result.page.rows;
|
||||
if (values.warm) {
|
||||
const warm = await result(
|
||||
"inspect_analysis_view",
|
||||
input,
|
||||
"retained-query:warm-diagnostic",
|
||||
);
|
||||
assert.deepEqual(warm, selected);
|
||||
}
|
||||
}
|
||||
answer = {
|
||||
go_version: summary.summary.build_info.go_version,
|
||||
main: summary.summary.build_info.module.main,
|
||||
runtime_dependencies: "unknown",
|
||||
dependencies,
|
||||
};
|
||||
}
|
||||
assert.deepEqual(answer, oracle.answer);
|
||||
report.answer = answer;
|
||||
await result("export_evidence_bundle", {
|
||||
path: join(values.output, "evidence.json"),
|
||||
});
|
||||
const bundle = JSON.parse(
|
||||
await readFile(join(values.output, "evidence.json"), "utf8"),
|
||||
);
|
||||
const parent = bundle.records.find(
|
||||
(e) => e.evidence_id === report.parent_evidence_id,
|
||||
);
|
||||
assert.ok(parent);
|
||||
assert.equal(parent.operation, "inspect_go_binary");
|
||||
assert.deepEqual(parent.parameters, { path: values.input });
|
||||
assert.equal(
|
||||
parent.normalized_result.artifact.sha256,
|
||||
oracle.artifact_sha256,
|
||||
);
|
||||
// The pre-change normalized result was independently checked against native
|
||||
// go version -m and its file-backed bytes before pinning this exact JSON.
|
||||
const expected = JSON.parse(
|
||||
await readFile(oracle.complete_result_path, "utf8"),
|
||||
);
|
||||
assert.deepEqual(parent.normalized_result, expected);
|
||||
assert.equal(
|
||||
parent.normalized_result.build_info.module.dependencies.length,
|
||||
oracle.dependency_count,
|
||||
);
|
||||
for (const name of ["version", "module"]) {
|
||||
const info = parent.normalized_result.build_info;
|
||||
const location = info[name + "_location"];
|
||||
const encoded =
|
||||
info[(name === "version" ? "go_version" : "module") + "_bytes_base64"];
|
||||
assert.deepEqual(
|
||||
bytes.subarray(location.offset, location.offset + location.bytes),
|
||||
Buffer.from(encoded, "base64"),
|
||||
);
|
||||
}
|
||||
assert.equal(sha256(await readFile(values.input)), oracle.artifact_sha256);
|
||||
report.oracle_file_sha256 = sha256(await readFile(values.oracle));
|
||||
report.complete_result_file_sha256 = sha256(
|
||||
await readFile(oracle.complete_result_path),
|
||||
);
|
||||
report.status = "observed";
|
||||
} catch (cause) {
|
||||
report.status = "failed";
|
||||
report.failure = String(cause?.stack ?? cause);
|
||||
process.exitCode = 1;
|
||||
} finally {
|
||||
if (
|
||||
!(await finalizeRetainedQueryReceipt({
|
||||
report,
|
||||
client,
|
||||
transport,
|
||||
account,
|
||||
serverLog,
|
||||
serverLogFinished,
|
||||
output: values.output,
|
||||
transcript,
|
||||
}))
|
||||
)
|
||||
process.exitCode = 1;
|
||||
}
|
||||
@@ -1,86 +0,0 @@
|
||||
"""Count entire captured Codex request sequences, without claiming model usage."""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import tiktoken
|
||||
|
||||
|
||||
def compact(value):
|
||||
return json.dumps(value, ensure_ascii=False, separators=(",", ":"))
|
||||
|
||||
|
||||
def request_text(body):
|
||||
pieces = []
|
||||
if isinstance(body.get("instructions"), str):
|
||||
pieces.append(body["instructions"])
|
||||
for item in body.get("input", []):
|
||||
kind = item.get("type")
|
||||
if kind == "function_call_output":
|
||||
pieces.append(item["output"])
|
||||
elif kind == "function_call":
|
||||
pieces.append(item["arguments"])
|
||||
elif kind == "message" or "role" in item:
|
||||
for part in item.get("content", []):
|
||||
if isinstance(part, dict) and isinstance(part.get("text"), str):
|
||||
pieces.append(part["text"])
|
||||
return pieces
|
||||
|
||||
|
||||
def inspect(root, encoding):
|
||||
report = json.loads((root / "report.json").read_text())
|
||||
assert (report["status"] == "passed" and not report["cleanup_failures"]
|
||||
and report["appserver_exit_code"] == 0)
|
||||
requests = json.loads((root / "model-requests.json").read_text())
|
||||
whole, namespaces, texts = [], [], []
|
||||
count = lambda value: len(encoding.encode(value, disallowed_special=()))
|
||||
for event in requests:
|
||||
body = event["body"]
|
||||
whole.append(count(compact(body)))
|
||||
namespace = next(tool for tool in body["tools"]
|
||||
if tool.get("type") == "namespace"
|
||||
and tool["name"] == "mcp__rea_audit")
|
||||
namespaces.append(count(compact(namespace)))
|
||||
texts.append(sum(count(piece) for piece in request_text(body)))
|
||||
log = root.with_suffix(".log")
|
||||
scope = next(json.loads(line) for line in reversed(log.read_text().splitlines())
|
||||
if line.startswith('{"resource_scope":'))
|
||||
observed = scope["payload_resource_observation"]
|
||||
assert (scope["abort"] is None and scope["payload_resource_observation_verified"]
|
||||
and observed["payload_cleanup_verified"] and observed["payload_exit_code"] == 0)
|
||||
memory = observed["resource_events"]["memory.events"]
|
||||
assert all(memory[key] == 0 for key in ["high", "max", "oom", "oom_kill"])
|
||||
assert observed["resource_events"]["pids.events"]["max"] == 0
|
||||
return {"case": root.name, "question": report["question_kind"],
|
||||
"request_count": len(requests), "whole_request_json_proxy": whole,
|
||||
"whole_request_json_proxy_sum": sum(whole),
|
||||
"rea_namespace_json_proxy": namespaces,
|
||||
"rea_namespace_json_proxy_sum": sum(namespaces),
|
||||
"unescaped_instruction_and_input_text_proxy": texts,
|
||||
"rea_namespace_plus_text_proxy_sum": sum(namespaces) + sum(texts),
|
||||
"wall_seconds": scope["elapsed_seconds"],
|
||||
"family_cpu_seconds": observed["cpu_stat"]["usage_usec"] / 1e6,
|
||||
"family_peak_mib": observed["memory_peak_bytes"] / 2**20,
|
||||
"completed_with_verified_cleanup": True,
|
||||
"sha256": {file.name: hashlib.sha256(file.read_bytes()).hexdigest()
|
||||
for file in [root / "report.json", root / "model-requests.json",
|
||||
root / "transcript.json", log]}}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument("cases", type=Path, nargs="+")
|
||||
args = parser.parse_args()
|
||||
encoding = tiktoken.get_encoding("cl100k_base")
|
||||
receipt = {"actual_model_usage": None,
|
||||
"backend": "deterministic local Responses SSE; no AI inference or paid service; not a model-quality or billing test",
|
||||
"client": "Codex app-server 0.163.0", "tiktoken": tiktoken.__version__,
|
||||
"encoding": encoding.name,
|
||||
"records": [inspect(root, encoding) for root in args.cases]}
|
||||
args.output.write_text(json.dumps(receipt, ensure_ascii=False, indent=2) + "\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,292 +0,0 @@
|
||||
"""Actual Codex wire capture with a deterministic local SSE backend; no AI inference."""
|
||||
import argparse
|
||||
import http.server
|
||||
import threading
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import time
|
||||
import traceback
|
||||
import subprocess
|
||||
|
||||
|
||||
async def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument("--runtime", type=Path, required=True)
|
||||
parser.add_argument("--codex", type=Path, required=True)
|
||||
parser.add_argument("--node", type=Path, required=True)
|
||||
parser.add_argument("--input-profile", choices=["full", "compact"], default="full")
|
||||
parser.add_argument("--output-profile", choices=["full", "off"], default="full")
|
||||
parser.add_argument("--input", type=Path, required=True)
|
||||
parser.add_argument("--oracle", type=Path, required=True)
|
||||
parser.add_argument("--expected-tools", type=int, default=210)
|
||||
parser.add_argument("--mode", choices=["complete", "selected", "summary"], required=True)
|
||||
parser.add_argument("--export", action="store_true")
|
||||
parser.add_argument("--question", choices=["identity", "dependency"], default="dependency")
|
||||
args = parser.parse_args()
|
||||
assert args.question != "identity" or args.mode != "selected"
|
||||
assert args.mode != "summary" or args.question == "identity"
|
||||
oracle=json.loads(args.oracle.read_text())
|
||||
target=str(args.input)
|
||||
assert args.input.is_absolute() and args.oracle.is_absolute() and args.output.is_absolute()
|
||||
assert hashlib.sha256(args.input.read_bytes()).hexdigest()==oracle["artifact_sha256"]
|
||||
def delivered(pieces):
|
||||
assert isinstance(pieces,str) and pieces.startswith("Wall time: "), pieces[:300]
|
||||
_,sep,payload=pieces.partition("Output:\n");assert sep
|
||||
value=json.loads(payload)
|
||||
return value["result"] if "result" in value and "normalized_result" not in value else value
|
||||
def outputs_of(body):
|
||||
return {item["call_id"]:delivered(item["output"]) for item in body.get("input",[]) if item.get("type")=="function_call_output"}
|
||||
steps=1+(args.mode=="selected")+args.export
|
||||
args.output.mkdir(mode=0o700)
|
||||
audit_home = args.output / "account"
|
||||
codex_state = audit_home / ".codex"
|
||||
mcp_home = audit_home / "mcp"
|
||||
workspace = audit_home / "workspace"
|
||||
for directory in [codex_state, mcp_home, workspace]:
|
||||
directory.mkdir(parents=True)
|
||||
captured = []
|
||||
class Capture(http.server.BaseHTTPRequestHandler):
|
||||
def log_message(self, *_):
|
||||
pass
|
||||
def do_POST(self):
|
||||
size = int(self.headers["Content-Length"])
|
||||
assert 0 < size <= 16 * 1024 * 1024
|
||||
assert self.headers.get("Content-Encoding", "identity") == "identity"
|
||||
body = json.loads(self.rfile.read(size))
|
||||
captured.append({"path": self.path, "body": body})
|
||||
(args.output / "model-requests.json").write_text(json.dumps(captured) + "\n")
|
||||
index = len(captured)
|
||||
names = [t.get("name", "") for ns in body.get("tools", []) if ns.get("type") == "namespace" and ns.get("name") == "mcp__rea_audit" for t in ns.get("tools", [])]
|
||||
assert index <= steps+1
|
||||
response_id=f"resp_local_{index}"
|
||||
if index <= steps:
|
||||
previous=outputs_of(body)
|
||||
if index==1:
|
||||
operation="inspect_go_binary";arguments={"path":target}
|
||||
if args.mode!="complete":arguments["detail"]="summary"
|
||||
elif args.mode=="selected" and index==2:
|
||||
parent=previous["call_local_1"]["normalized_result"]["parent_evidence_id"]
|
||||
operation="inspect_analysis_view";arguments={"source":{"kind":"retained-evidence","evidence_id":parent},"view":{"kind":"go","collection":"dependencies","paths":[oracle["dependency_path"]],"fields":["path","version","sum","replacement"],"offset":0,"limit":64}}
|
||||
else:
|
||||
operation="export_evidence_bundle";arguments={"path":str(args.output/"evidence.json")}
|
||||
selected=[name for name in names if name.endswith(operation)];assert len(selected)==1
|
||||
item={"type":"function_call","id":f"fc_local_{index}","call_id":f"call_local_{index}","name":selected[0],"namespace":"mcp__rea_audit","arguments":json.dumps(arguments),"status":"completed"}
|
||||
else:
|
||||
item={"type":"message","id":"msg_local_done","role":"assistant","status":"completed","content":[{"type":"output_text","text":"Scripted local Go metadata query complete.","annotations":[]}]}
|
||||
base = {"id": response_id, "object": "response", "model": body.get("model"), "status": "completed", "output": [item]}
|
||||
events = [
|
||||
{"type": "response.created", "response": {**base, "status": "in_progress", "output": []}},
|
||||
{"type": "response.output_item.added", "output_index": 0, "item": {**item, "status": "in_progress"}},
|
||||
{"type": "response.output_item.done", "output_index": 0, "item": item},
|
||||
{"type": "response.completed", "response": base},
|
||||
]
|
||||
wire = "".join("event: " + e["type"] + "\ndata: " + json.dumps(e) + "\n\n" for e in events).encode()
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "text/event-stream")
|
||||
self.send_header("Content-Length", str(len(wire)))
|
||||
self.end_headers()
|
||||
self.wfile.write(wire)
|
||||
endpoint = http.server.HTTPServer(("127.0.0.1", 0), Capture)
|
||||
endpoint_thread = threading.Thread(target=endpoint.serve_forever, daemon=True)
|
||||
endpoint_thread.start()
|
||||
task_path = f"{args.node.parent}:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
|
||||
mcp_env = {"HOME": str(mcp_home), "PATH": task_path,
|
||||
"NODE_OPTIONS": "--max-old-space-size=2048", "UV_THREADPOOL_SIZE": "1",
|
||||
"REA_LOG_LEVEL": "silent", "REA_MCP_INPUT_SCHEMA_PROFILE": args.input_profile,
|
||||
"REA_MCP_OUTPUT_SCHEMA_PROFILE": args.output_profile}
|
||||
config = 'model_provider = "rea_capture"\nmodel = "rea-wire-probe"\n[analytics]\nenabled = false\n[mcp_servers.rea_audit]\n'
|
||||
config += f'command = {json.dumps(str(args.node))}\n'
|
||||
config += f'args = {json.dumps([str(args.runtime / "scripts/rea.mjs"), "mcp"])}\n'
|
||||
config += 'startup_timeout_sec = 60\ntool_timeout_sec = 60\n'
|
||||
config += '[mcp_servers.rea_audit.env]\n'
|
||||
config += ''.join(f'{key} = {json.dumps(value)}\n' for key, value in mcp_env.items())
|
||||
for name in ['inspect_go_binary','inspect_analysis_view','export_evidence_bundle']:
|
||||
config += f'[mcp_servers.rea_audit.tools.{name}]\napproval_mode = "approve"\noutput_token_limit = 30000\n'
|
||||
config += f'\n[model_providers.rea_capture]\nname = "Local scripted capture"\nbase_url = "http://127.0.0.1:{endpoint.server_port}/v1"\nwire_api = "responses"\nrequires_openai_auth = false\nsupports_websockets = false\nrequest_max_retries = 0\nstream_max_retries = 0\n'
|
||||
(codex_state / "config.toml").write_text(config)
|
||||
(args.output / "declared-config.toml").write_text(config)
|
||||
environment = {"HOME": str(audit_home), "CODEX_HOME": str(codex_state), "PATH": task_path,
|
||||
"NODE_OPTIONS": "--max-old-space-size=2048", "UV_THREADPOOL_SIZE": "1"}
|
||||
report = {"question": "Read embedded Go identity/counts or a selected dependency. Export is a separate task variant.",
|
||||
"client": "Codex app-server 0.163.0", "model_turns_started": 1, "backend": "local deterministic SSE fixture; no AI model or paid service",
|
||||
"actual_model_usage": None, "input_profile": args.input_profile,
|
||||
"output_profile": args.output_profile, "calls": [], "observations": [],
|
||||
"unexpected_server_requests": [], "cleanup_failures": []}
|
||||
client_version = subprocess.check_output([str(args.codex), "--version"], env=environment, cwd=workspace, text=True, timeout=10).strip()
|
||||
assert client_version == "codex-cli 0.163.0", client_version
|
||||
transcript = []
|
||||
pending = {}
|
||||
completed_turn = asyncio.Event()
|
||||
next_id = 0
|
||||
stderr = (args.output / "server.log").open("wb")
|
||||
process = await asyncio.create_subprocess_exec(str(args.codex), "app-server", "--stdio",
|
||||
stdin=asyncio.subprocess.PIPE, stdout=asyncio.subprocess.PIPE, stderr=stderr,
|
||||
cwd=workspace, env=environment, limit=16 * 1024 * 1024)
|
||||
|
||||
async def send(message):
|
||||
encoded = (json.dumps(message, separators=(",", ":")) + "\n").encode()
|
||||
transcript.append({"direction": "client", "bytes": len(encoded), "message": message})
|
||||
process.stdin.write(encoded)
|
||||
await process.stdin.drain()
|
||||
|
||||
async def read_messages():
|
||||
try:
|
||||
while line := await process.stdout.readline():
|
||||
value = json.loads(line)
|
||||
transcript.append({"direction": "server", "bytes": len(line), "message": value})
|
||||
if value.get("method") == "turn/completed":
|
||||
report["turn"] = value["params"]["turn"]
|
||||
completed_turn.set()
|
||||
if "id" in value and "method" not in value:
|
||||
future = pending.pop(value["id"], None)
|
||||
if future and not future.done():
|
||||
future.set_result(value)
|
||||
elif "id" in value:
|
||||
report["unexpected_server_requests"].append(value["method"])
|
||||
await send({"id": value["id"], "error": {"code": -32601, "message": "Unexpected request in scripted metadata probe"}})
|
||||
raise RuntimeError("app-server stdout closed")
|
||||
except Exception as error:
|
||||
for future in pending.values():
|
||||
if not future.done():
|
||||
future.set_exception(error)
|
||||
|
||||
reader = asyncio.create_task(read_messages())
|
||||
|
||||
async def request(method, params):
|
||||
nonlocal next_id
|
||||
assert method in ["initialize", "thread/start", "mcpServerStatus/list", "mcpServer/tool/call", "turn/start"]
|
||||
next_id += 1
|
||||
message = {"id": next_id, "method": method, "params": params}
|
||||
future = asyncio.get_running_loop().create_future()
|
||||
pending[next_id] = future
|
||||
start = time.monotonic()
|
||||
await send(message)
|
||||
reply = await asyncio.wait_for(future, 90)
|
||||
report["calls"].append({"method": method, "elapsed_ms": (time.monotonic() - start) * 1000,
|
||||
"error": reply.get("error")})
|
||||
if "error" in reply:
|
||||
raise RuntimeError(json.dumps(reply["error"])[:4000])
|
||||
return reply["result"]
|
||||
|
||||
try:
|
||||
await request("initialize", {"clientInfo": {"name": "rea_host_probe", "title": "REA host compatibility probe", "version": "1"},
|
||||
"capabilities": {"experimentalApi": True, "explicitGatewayOauth": True}})
|
||||
await send({"method": "initialized"})
|
||||
created = await request("thread/start", {"cwd": str(workspace), "ephemeral": True,
|
||||
"approvalPolicy": "never", "sandbox": "read-only"})
|
||||
thread_id = created["thread"]["id"]
|
||||
assert not created["thread"].get("turns", [])
|
||||
inventory = await request("mcpServerStatus/list", {"threadId": thread_id,
|
||||
"serverName": "rea_audit", "detail": "toolsAndAuthOnly", "limit": 10})
|
||||
assert inventory.get("nextCursor") is None
|
||||
assert len(inventory["data"]) == 1
|
||||
server = inventory["data"][0]
|
||||
assert server["name"] == "rea_audit" and server.get("toolsError") is None
|
||||
tools = list(server["tools"].values())
|
||||
names = [tool["name"] for tool in tools]
|
||||
assert len(names) == len(set(names)) == args.expected_tools
|
||||
assert "inspect_go_binary" in names
|
||||
output_schemas = sum(tool.get("outputSchema") is not None for tool in tools)
|
||||
assert output_schemas == (args.expected_tools if args.output_profile == "full" else 0)
|
||||
raw = json.dumps(inventory, separators=(",", ":"), ensure_ascii=False).encode()
|
||||
(args.output / "catalog.json").write_bytes(raw)
|
||||
report["catalog"] = {"tools": len(tools), "output_schemas": output_schemas,
|
||||
"bytes": len(raw), "sha256": hashlib.sha256(raw).hexdigest(),
|
||||
"server_info": server.get("serverInfo"), "runtime_status": server.get("runtimeStatus")}
|
||||
await request("turn/start", {"threadId": thread_id, "input": [{"type": "text", "text": ("Read the embedded Go compiler, main module and embedded dependency count; state runtime dependency coverage." if args.question=="identity" else "Read the embedded Go compiler, main module and golang.org/x/crypto dependency and state runtime dependency coverage.")+" The local fixture scripts the required tools; no AI inference is used."}]})
|
||||
await asyncio.wait_for(completed_turn.wait(), 90)
|
||||
assert report["turn"]["status"] == "completed", report["turn"]
|
||||
assert len(captured)==steps+1
|
||||
outputs=outputs_of(captured[-1]["body"])
|
||||
assert len(outputs)==steps
|
||||
initial=outputs["call_local_1"]
|
||||
assert initial["normalized_result"]["artifact"]["sha256"]==oracle["artifact_sha256"]
|
||||
if args.question=="identity":
|
||||
info=initial["normalized_result"]["build_info"] if args.mode=="complete" else initial["normalized_result"]["summary"]["build_info"]
|
||||
count=len(info["module"]["dependencies"]) if args.mode=="complete" else info["module"]["available_counts"]["dependencies"]
|
||||
facts={"go_version":info["go_version"],"main":info["module"]["main"],"runtime_dependencies":"unknown","dependency_count":count}
|
||||
parent=initial["evidence_id"] if args.mode=="complete" else initial["normalized_result"]["parent_evidence_id"]
|
||||
elif args.mode=="complete":
|
||||
info=initial["normalized_result"]["build_info"]
|
||||
facts={"go_version":info["go_version"],"main":info["module"]["main"],"runtime_dependencies":"unknown","dependencies":[{"index":i,**d} for i,d in enumerate(info["module"]["dependencies"]) if d["path"]==oracle["dependency_path"]]}
|
||||
parent=initial["evidence_id"]
|
||||
else:
|
||||
info=initial["normalized_result"]["summary"]["build_info"]
|
||||
selected=outputs["call_local_2"]["normalized_result"]
|
||||
assert selected["coverage"]["exhausted"] and selected["source_coverage"]["runtime_dependencies"]=="unknown"
|
||||
facts={"go_version":info["go_version"],"main":info["module"]["main"],"runtime_dependencies":"unknown","dependencies":selected["page"]["rows"]}
|
||||
parent=initial["normalized_result"]["parent_evidence_id"]
|
||||
expected=({key:oracle["answer"][key] for key in ["go_version","main","runtime_dependencies"]}|{"dependency_count":oracle["dependency_count"]}) if args.question=="identity" else oracle["answer"]
|
||||
assert facts==expected
|
||||
report["question_kind"]=args.question
|
||||
if args.export:
|
||||
bundle=json.loads((args.output/"evidence.json").read_text());evidence=next(e for e in bundle["records"] if e["evidence_id"]==parent)
|
||||
assert evidence["normalized_result"]==json.loads(Path(oracle["complete_result_path"]).read_text())
|
||||
report["observations"]={"answer":facts,"parent_evidence_id":parent,"exported":args.export}
|
||||
report["captured_requests"]=len(captured)
|
||||
report["mode"]=args.mode
|
||||
report["request_tool_counts"] = [len(c["body"].get("tools", [])) for c in captured]
|
||||
assert not report["unexpected_server_requests"]
|
||||
assert hashlib.sha256(args.input.read_bytes()).hexdigest()==oracle["artifact_sha256"]
|
||||
report["status"] = "passed"
|
||||
except Exception:
|
||||
report["status"] = "failed"
|
||||
report["failure"] = traceback.format_exc()[-6000:]
|
||||
finally:
|
||||
process.stdin.close()
|
||||
try:
|
||||
await asyncio.wait_for(process.wait(), 15)
|
||||
except asyncio.TimeoutError:
|
||||
report["cleanup_failures"].append("app-server did not exit after stdin close; terminated owned process")
|
||||
process.terminate()
|
||||
try:
|
||||
await asyncio.wait_for(process.wait(), 5)
|
||||
except asyncio.TimeoutError:
|
||||
process.kill()
|
||||
await process.wait()
|
||||
await reader
|
||||
# app-server can acknowledge exit before its MCP child finishes EOF
|
||||
# cleanup. Wait on the allocated resource scope; never target a PID.
|
||||
try:
|
||||
groups = [line[3:] for line in Path("/proc/self/cgroup").read_text().splitlines()
|
||||
if line.startswith("0::")]
|
||||
assert len(groups) == 1
|
||||
group = Path("/sys/fs/cgroup") / groups[0].lstrip("/")
|
||||
assert re.fullmatch(r"rea-efficiency-[0-9a-f]{32}\.scope", group.name)
|
||||
observer = os.getppid()
|
||||
observer_args = Path(f"/proc/{observer}/cmdline").read_bytes().split(b"\0")
|
||||
assert any(Path(os.fsdecode(value)).name == "resource_payload.py" for value in observer_args if value)
|
||||
expected = {os.getpid(), observer}
|
||||
until = time.monotonic() + 10
|
||||
while True:
|
||||
actors = {int(value) for value in (group / "cgroup.procs").read_text().splitlines()}
|
||||
assert expected <= actors
|
||||
remaining = sorted(actors - expected)
|
||||
if not remaining or time.monotonic() >= until:
|
||||
break
|
||||
await asyncio.sleep(0.05)
|
||||
report["owned_scope_drain"] = {"remaining_processes": remaining}
|
||||
if remaining:
|
||||
report["cleanup_failures"].append("owned MCP child did not settle after app-server exit")
|
||||
except (OSError, ValueError, AssertionError) as error:
|
||||
report["owned_scope_drain"] = {"status": "unavailable", "reason": type(error).__name__}
|
||||
report["cleanup_failures"].append("allocated resource-scope cleanup could not be verified")
|
||||
endpoint.shutdown()
|
||||
endpoint.server_close()
|
||||
endpoint_thread.join(5)
|
||||
stderr.close()
|
||||
report["appserver_exit_code"] = process.returncode
|
||||
if process.returncode != 0:
|
||||
report["cleanup_failures"].append(f"app-server exit code: {process.returncode}")
|
||||
(args.output / "transcript.json").write_text(json.dumps(transcript) + "\n")
|
||||
(args.output / "report.json").write_text(json.dumps(report, indent=2) + "\n")
|
||||
print(json.dumps({key: report.get(key) for key in ["status", "failure", "catalog", "observations", "cleanup_failures", "appserver_exit_code"]}))
|
||||
return 0 if report["status"] == "passed" and not report["cleanup_failures"] else 1
|
||||
|
||||
|
||||
raise SystemExit(asyncio.run(main()))
|
||||
Loaded 100 of 622 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user