diff --git a/.agents/skills/rea-changelog-update/SKILL.md b/.agents/skills/rea-changelog-update/SKILL.md index 027682cfa3..d45f226383 100644 --- a/.agents/skills/rea-changelog-update/SKILL.md +++ b/.agents/skills/rea-changelog-update/SKILL.md @@ -68,9 +68,9 @@ closing-issue reporters. Grouping preserves all those credits in one line. When Highlights, detailed entries or migrations reference the same work, put its thanks in one of those entries, not all of them. Include issue links. Direct-commit bullets carry the commit link and known human credit. -Exclude GitHub bots and known agent accounts. REA's observed Anthropic and -Cursor co-author trailers resolve to `@claude` and `@cursoragent` with GitHub -type `User`; the credit policy excludes them while retaining their metadata. +Exclude GitHub bots and known agent accounts. REA's observed Anthropic, Cursor +and Codex co-author trailers resolve to `@claude`, `@cursoragent` and `@codex` +with GitHub type `User`; the credit policy excludes them while retaining their metadata. At the REA maintainer's request, omit `@morluto` self-thanks while preserving authorship in the inventory. Other human contributors remain eligible. Use explicit GitHub links for cross-repository references to avoid ambiguous diff --git a/.agents/skills/reverse-engineer-anything/SKILL.md b/.agents/skills/reverse-engineer-anything/SKILL.md index 39f9ae20b2..426a05ae56 100644 --- a/.agents/skills/reverse-engineer-anything/SKILL.md +++ b/.agents/skills/reverse-engineer-anything/SKILL.md @@ -2,7 +2,7 @@ name: reverse-engineer-anything description: Reverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. metadata: - version: "41" + version: "42" --- # REA @@ -31,6 +31,49 @@ Diagnose the observed blocker, continue useful supported work, and resume the original task after repair. Client approval, shell PATH, registration and engine availability have different remedies; avoid repeating setup for all of them. +1. Diagnose without changing REA or client configuration. `npx` may download + the package and write to npm's cache. For Codex, run + `npx -y rea-agents@latest doctor --client codex --json`. Substitute the current + supported client: `claude_code`, `claude_desktop`, `codex`, `cursor`, + `gemini_cli`, `windsurf`, `devin`, `opencode`, `antigravity`, `copilot_cli`, + `commandcode`, `vscode`, `grok_build`, `omp`, or `grok_bot`. `grok_bot` has no local + file registration: doctor reports the Grok Bot chat step, and setup does not + write `mcp.json`. If the client is unknown, use `doctor --json` and + inspect its registration results before choosing a setup scope. +2. Distinguish the reason. Missing, malformed, or stale registration needs a + scoped configuration repair. An aligned registration with no tools in the + active session needs a restart/reconnection; doctor cannot prove that the + current agent has connected. A missing provider affects only tasks requiring + that provider: static JavaScript inspection needs neither Hopper nor Ghidra, + and Android inspection has separate bring-your-own JADX/Java prerequisites. +3. For a configuration repair, prepare the read-only plan: + `npx -y rea-agents@latest setup --client codex --dry-run --json`. + Use the current client's ID, show its exact proposed paths, backups, and + changes, and obtain approval before setup writes configuration or installs + Hopper. Setup normally installs the matching bundled skill too; include that + replacement in the reviewed plan. After approval, apply the same scope with + `npx -y rea-agents@latest setup --client codex --yes`. Add `--install-hopper` + only if that separate installation was needed and explicitly approved. +4. Restart/reconnect the affected agent. Verify that REA tools actually appear + in the session, then resume the original investigation. If they remain + absent, inspect the client's MCP launch error rather than repeating setup. + +REA setup never installs or upgrades Node.js, npm, Homebrew, Java, Ghidra, IDA, +JADX, Binwalk, or Unblob. Use existing prerequisites; do not install unrelated +software to repair +MCP registration. For an unsupported client, use manual stdio registration +with a version-pinned `rea-agents` package or continue through the CLI. + +A concrete CLI fallback for an operator-supplied JavaScript tree or ASAR is: + +```bash +npx -y rea-agents@latest analyze-javascript-application /absolute/path/to/app --json +``` + +No MCP registration or native engine is required. Read the returned Evidence, +graph, limitations, and unknowns with the same care as an MCP result; the CLI +returns the Evidence record directly. This fallback does not establish that +MCP is configured. Native CLI tasks still require their selected engine. Prefer an existing `rea` CLI. When none is installed and REA package acquisition is within the authorized scope, use `npx -y rea-agents@latest` in its place. Configuration repair starts with a read-only setup plan; show paths, backups and @@ -61,6 +104,10 @@ explicit path or endpoint and do not need it. consult the NativeAOT workflow in `references/native-and-artifacts.md` for optional metadata recovery and its supported host/layout boundary. - User-owned browser page already open: `list_browser_targets`. +- Browser action the agent must perform (click, type, navigate): + `capture_browser_scenario` with declared actions. See + [runtime observation](references/runtime-observation.md) for connect/launch + and the focused follow-up browser tools. - Fiddler Everywhere traffic or a native `.saz` capture: read the [Fiddler workflow](references/runtime-observation.md#fiddler-everywhere-traffic-debugging). Inspect saved SAZ with `inspect_web_network_capture` and `format: "saz"`; @@ -75,7 +122,7 @@ explicit path or endpoint and do not need it. pages; select one trace, stream ID and direction for contribution/range pages. Facts retain all-recorded-events-before-selection scope. Other qlog layouts and PCAP correlation are unsupported. -- User-owned Electron runtime already open: `list_electron_targets`. +- User-owned Electron runtime already open: `list_electron_targets`. Before an electron-main Inspector attempt, check packaged fuses with `inspect_electron_fuses`; disabled `EnableNodeCliInspectArguments` means the app ignores `--inspect`. - Packet-field discovery: `inspect_packet_fields` searches installed protocols and field names/types before choosing fields for `inspect_packet_capture`. Query/page the relevant records; registrations do not establish observed traffic. @@ -280,7 +327,8 @@ as complete while required questions remain open. [references/javascript-applications.md](references/javascript-applications.md) - Android APK declarations, classes, methods, and static references: [references/android-applications.md](references/android-applications.md) -- Passive browser/Electron observation and static/runtime reconciliation: +- Browser/Electron observation (passive and scenario) and static/runtime + reconciliation: [references/runtime-observation.md](references/runtime-observation.md) - Evidence paging, comparisons, residual unknowns, and verification: [references/evidence-workflows.md](references/evidence-workflows.md) diff --git a/.agents/skills/reverse-engineer-anything/references/evidence-workflows.md b/.agents/skills/reverse-engineer-anything/references/evidence-workflows.md index c651287f3e..539bfb6b2f 100644 --- a/.agents/skills/reverse-engineer-anything/references/evidence-workflows.md +++ b/.agents/skills/reverse-engineer-anything/references/evidence-workflows.md @@ -65,3 +65,7 @@ Use `inspect_analysis_view` with `view: {kind: "api-surface"}` on retained method, origin and path template. Format matches are inferences, not handler identity or schemas. Group pages cite parent record ordinals; consult record coverage and exclusions before claiming completeness. No requests are replayed. +When the connected schema advertises `include_query_parameter_names`, select +it in the view for distinct observed names with original record ordinals. Query +values are omitted; the names establish neither required parameters nor a +complete server schema. Keep the original parent Evidence for exact URLs. diff --git a/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md b/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md index 960e460945..789b7a89ea 100644 --- a/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md +++ b/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md @@ -169,6 +169,14 @@ repeat parsing. Returned-row exhaustion does not prove complete metadata or absence under partial/unavailable coverage; preserve build-local tokens and their artifact/MVID scope. +`inspect_managed_members` always returns the assembly-wide `member_refs`, even +for one selected type. For questions about a selected type, select +`detail: "summary"` with `type_names`; it returns the selection (with unmatched +names), coverage and per-collection counts. Page rows with +`view: {kind: "managed-members", collection: "call_edges", offset: 0, limit: 64}` +over `types`, `exported_types`, `fields`, `methods`, `member_refs`, `call_edges`, +`field_accesses` or `user_strings`. + ### .NET NativeAOT NativeAOT binaries contain native machine code, so ordinary CIL decompilers diff --git a/.agents/skills/reverse-engineer-anything/references/runtime-observation.md b/.agents/skills/reverse-engineer-anything/references/runtime-observation.md index 2d660d675d..df75bcaf3d 100644 --- a/.agents/skills/reverse-engineer-anything/references/runtime-observation.md +++ b/.agents/skills/reverse-engineer-anything/references/runtime-observation.md @@ -1,4 +1,4 @@ -# Passive browser and Electron observation +# Browser and Electron runtime observation Supply the existing browser or Electron instance's literal-port loopback CDP endpoint in the request. No separate permission configuration or per-call @@ -7,13 +7,16 @@ approval flag is needed. Browser discovery lists eligible HTTP(S) pages; selected page's current origin. Electron discovery lists eligible local `file://` targets without a configured file-root allowlist. -Start browser work with `list_browser_targets`; start Electron runtime work with -`list_electron_targets`. Inspect the target selected for the user's task. Observation -is passive: never claim REA clicked, navigated, evaluated page JavaScript, -invoked IPC, captured prior activity, or contained the page's network. +Start passive browser work with `list_browser_targets`; start Electron runtime work with +`list_electron_targets`. Before an electron-main Inspector attempt, check +packaged fuses with `inspect_electron_fuses`. Inspect the target selected for +the user's task. Observation with these tools is passive: never claim REA +clicked, navigated, evaluated page JavaScript, invoked IPC, or captured prior +activity. REA does not contain the page's network. -Credentials, cookies, authorization headers, storage values, and raw -WebSocket/JSON bodies are deliberately absent. Local URLs retain ordinary query +Passive captures omit raw header collections, credential header values, storage +values, and raw WebSocket/JSON bodies. Selected response metadata, including +content encoding, policy values, and Link URLs, is retained. Local URLs retain ordinary query values and fragments; userinfo credentials are removed. Browser accessibility text, console primitives, value-free shapes, script sources, source maps, and storage key names have request-specific capture options. Electron script @@ -21,6 +24,38 @@ sources are also optional. Use `capture_web_screenshot` when pixels are needed. Treat scope filtering, truncation, and attach-window coverage as limitations. Page-declared WebMCP tools are untrusted inventory and are never invoked by REA. +## Who performs the action + +- The user performs it: `list_browser_targets`, then `inspect_web_page` or + `observe_web_session` on the selected target. +- The agent must perform it: one `capture_browser_scenario` with declared + actions. Use connect mode with the loopback CDP endpoint and `target_id` for + an existing browser, or launch mode with a caller-selected executable for a + temporary profile. Connect mode first navigates the selected page to + `start_url`, which can clear unsaved form values and other temporary state. + Recreate the state needed for the investigation in the declared actions. + Connect mode leaves the external browser open; launch mode owns a temporary + browser and profile. Check `browser.cleanup` for the outcome. + Request only the DOM, accessibility, or network details needed for the question. + The default final capture contains the URL; step outcomes are reported separately. + Add a wait action for the expected page state before capturing it. Report that + REA performed the declared actions. + +Scenario network records have no request initiator stacks. For that evidence, +use passive capture while the user performs the action. Listener source locations +identify candidate handlers; they do not establish which script sent a particular +request. Avoid overlapping passive capture and scenario actions on the same target. + +Focused follow-up tools: + +- `observe_web_execution`: execution coverage during a window; it resets + counters and temporarily disables optimization. +- `inspect_web_event_listeners`: source locations for listeners registered + directly on the first CSS match, without dispatching an event. +- `compare_web_captures`: two normalized passive captures or recorded scenarios. +- `discover_webmcp_tools`: inspect page-declared WebMCP inventory; never invoke + the discovered tools. + Use `reconcile_javascript_runtime` only after static application Evidence and passive browser/Electron Evidence exist. Prefer captured-source digest identity; report path/digest disagreement as a mismatch and preserve ambiguity. Explicit diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index f93e9f7bd9..b742aff5f4 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -1,5 +1,5 @@ name: Bug report -description: Report a reproducible problem in REA +description: Report a problem in REA with observations or source evidence title: "[Bug] " labels: - bug @@ -8,8 +8,12 @@ body: - type: markdown attributes: value: | - Please report one primary behavior per issue. Keep observed behavior - separate from suspected causes, and remove credentials and private paths. + Report one primary behavior per issue. Keep answers concise and grounded + in available evidence. Distinguish observed behavior, source-derived + reasoning, and suspected causes; do not claim a reproduction you did not run. + If a narrative answer is not known, write `NOT_ENOUGH_INFO` and describe + what is missing. Intermittent failures and clearly labeled source-only + reports are welcome. Do not include credentials or sensitive target data. - type: checkboxes id: before-submitting @@ -42,7 +46,7 @@ body: id: observed-behavior attributes: label: Observed behavior - description: What happened? Include the command, MCP tool, provider, target type, and conditions that matter. + description: Describe the actual result, including the exact command or MCP request, relevant provider/target, and supporting evidence below. Label source-only findings instead of claiming execution. validations: required: true @@ -50,7 +54,7 @@ body: id: reproduction attributes: label: Reproduction - description: Give the smallest realistic command, configuration, input, or event sequence that demonstrates the problem. + description: Give the shortest steps you actually ran, with the relevant input or configuration. State how often they reproduced the problem when known. If reproduction is unavailable, say so and provide the observations or source reference you have. placeholder: | 1. Configure or start ... 2. Run ... @@ -62,7 +66,7 @@ body: id: expected-behavior attributes: label: Expected behavior - description: What result, contract, invariant, recovery action, or state transition should have occurred? + description: State the expected result, contract, or recovery action. Cite documentation, source, or a known-good version when available; mark an unverified expectation or unknown reference clearly. validations: required: true @@ -70,7 +74,7 @@ body: id: impact-scope attributes: label: Impact and scope - description: What is affected, what is not affected, and does this affect evidence integrity, process isolation, compatibility, or recovery? + description: Describe observed impact and frequency, including effects on evidence integrity, process isolation, compatibility, or recovery when known. Leave untested scope and severity unknown rather than guessing. validations: required: true @@ -79,7 +83,7 @@ body: attributes: label: REA version description: Use `rea --version` or provide the source revision. - placeholder: 3.1.0 or git revision + placeholder: Exact rea --version output or source commit SHA validations: required: true @@ -116,7 +120,7 @@ body: id: failure-stage attributes: label: Earliest failure stage - description: Select where the incorrect behavior first becomes observable. + description: Select where the incorrect behavior first becomes observable. Choose Unknown if this has not been established. options: - Request parsing or validation - Setup or provider admission @@ -133,8 +137,14 @@ body: id: relevant-evidence attributes: label: Relevant output or evidence - description: Paste the smallest useful CLI output, MCP result, error, trace, or evidence excerpt. Redact sensitive values. - render: text + description: Include the smallest useful output, MCP result, error, screenshot, artifact, or exact source reference. Say how it was obtained and which claim it supports. If unavailable, write NOT_ENOUGH_INFO and explain what is missing. Redact credentials and sensitive target data. + placeholder: | + Command or MCP request actually run: + Relevant result, error, or artifact: + For source-only findings, link the file/revision and relevant code: + Missing evidence or remaining uncertainty: + validations: + required: true - type: textarea id: typed-outcome @@ -146,10 +156,10 @@ body: id: analysis attributes: label: Analysis or suspected cause - description: Optional. Clearly mark source-derived reasoning or a hypothesis; do not present an unverified cause as fact. + description: Optional. Link supporting source and distinguish source-derived reasoning from a hypothesis. A root-cause analysis or proposed fix is not required. - type: textarea id: additional-context attributes: label: Additional context - description: Add links, reduced fixtures, screenshots, or a proposed regression test. + description: Add reduced fixtures, screenshots, or related issues. For observed regressions, include the last known good and first known bad versions when available. Label untested checks as proposals. diff --git a/.github/ISSUE_TEMPLATE/design_discussion.yml b/.github/ISSUE_TEMPLATE/design_discussion.yml index 38d33ac821..28aa67559f 100644 --- a/.github/ISSUE_TEMPLATE/design_discussion.yml +++ b/.github/ISSUE_TEMPLATE/design_discussion.yml @@ -41,7 +41,7 @@ body: id: context-evidence attributes: label: Context and evidence - description: Explain the problem or motivation. If you have evidence, distinguish observed behavior from ideas or guesses. + description: Explain the problem or motivation and link relevant evidence when available. Distinguish observed behavior, source-derived reasoning, proposed behavior, and unknowns. A question without measurements or a finished design is welcome. validations: required: true diff --git a/.github/ISSUE_TEMPLATE/documentation.yml b/.github/ISSUE_TEMPLATE/documentation.yml index fa2bc024ea..a522606a6e 100644 --- a/.github/ISSUE_TEMPLATE/documentation.yml +++ b/.github/ISSUE_TEMPLATE/documentation.yml @@ -5,6 +5,13 @@ labels: - documentation - triage body: + - type: markdown + attributes: + value: | + Report one documentation problem with the exact location and the smallest + useful supporting passage, example, or result. Keep answers concise. + If something is unknown, write `NOT_ENOUGH_INFO` and explain what is missing. + - type: checkboxes id: before-submitting attributes: @@ -28,7 +35,7 @@ body: id: problem attributes: label: Problem - description: What is unclear, missing, outdated, or incorrect, and who is affected? + description: Quote or describe the current content, explain what is unclear, missing, outdated, or incorrect, and say how to observe the problem. State who is affected when known. validations: required: true @@ -36,7 +43,7 @@ body: id: expected-content attributes: label: Expected content - description: Describe the accurate behavior or example that should be documented. + description: Describe the accurate behavior or example that should be documented, with a reference when available. Label a proposed clarification or unknown behavior clearly. validations: required: true @@ -44,7 +51,9 @@ body: id: evidence attributes: label: Supporting evidence - description: Add source, command output, provider documentation, or a link that supports the correction. + description: Include an exact passage, source reference, command output, provider documentation, or screenshot supporting the report. For missing or unclear guidance, point to the relevant section and explain the gap. If evidence is unavailable, write NOT_ENOUGH_INFO and explain what is missing. + validations: + required: true - type: textarea id: additional-context diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml index 4125f3db52..29ed544f21 100644 --- a/.github/ISSUE_TEMPLATE/feature_request.yml +++ b/.github/ISSUE_TEMPLATE/feature_request.yml @@ -11,6 +11,8 @@ body: Describe one problem and the outcome you would like. Short answers are welcome. Choose "Not sure yet" wherever needed; you do not need code, a finished design, or a complete test plan to propose an idea. + Use concrete examples when available, distinguish current limitations + from proposed behavior, and leave unmeasured impact or unknowns explicit. Read the [contribution scope](https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope). Maintainers can help resolve scope and technical questions. Unresolved @@ -58,7 +60,7 @@ body: id: use-cases attributes: label: Use cases - description: Optional additional examples of applications, targets, or callers. Leave blank if covered above. + description: Optional examples of affected workflows, targets, or callers and their practical impact. Include observed frequency or measurements when available; do not invent them. Leave blank if covered above. - type: textarea id: desired-outcome @@ -138,4 +140,4 @@ body: id: references attributes: label: References - description: Link relevant provider documentation, source, measurements, artifacts, issues, or discussions. + description: Optional supporting examples, provider documentation, source, measurements, artifacts, issues, or prior art. Distinguish observed results from expected benefits; an idea without evidence is welcome. diff --git a/.github/actions/npm-cache/action.yml b/.github/actions/npm-cache/action.yml new file mode 100644 index 0000000000..1af22da2b9 --- /dev/null +++ b/.github/actions/npm-cache/action.yml @@ -0,0 +1,42 @@ +name: Cache npm package downloads +description: Restore npm's download cache and save it only from trusted default-branch events. +outputs: + can-save: + description: Whether this run may update the shared npm download cache. + value: ${{ steps.context.outputs.can-save }} + path: + description: The npm download-cache directory. + value: ${{ steps.context.outputs.path }} +runs: + using: composite + steps: + - id: context + shell: bash + env: + DEFAULT_BRANCH: ${{ github.event.repository.default_branch }} + LOCKFILES_HASH: ${{ hashFiles('package-lock.json', 'npm-shrinkwrap.json', '.npmrc') }} + run: | + cache_path="$(npm config get cache)" + runtime_arch="$(node -p 'process.arch')" + can_save=false + if [[ -n "${DEFAULT_BRANCH}" && "${GITHUB_REF}" == "refs/heads/${DEFAULT_BRANCH}" && ( "${GITHUB_EVENT_NAME}" == "push" || "${GITHUB_EVENT_NAME}" == "schedule" || "${GITHUB_EVENT_NAME}" == "workflow_dispatch" ) ]]; then + can_save=true + fi + cache_key="npm-download-${RUNNER_OS}-${runtime_arch}-${LOCKFILES_HASH}" + { + printf 'can-save=%s\n' "${can_save}" + printf 'path=%s\n' "${cache_path}" + printf 'key=%s\n' "${cache_key}" + } >> "${GITHUB_OUTPUT}" + - name: Restore and save npm package downloads + if: steps.context.outputs.can-save == 'true' + uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 + with: + path: ${{ steps.context.outputs.path }} + key: ${{ steps.context.outputs.key }} + - name: Restore npm package downloads without saving + if: steps.context.outputs.can-save != 'true' + uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 + with: + path: ${{ steps.context.outputs.path }} + key: ${{ steps.context.outputs.key }} diff --git a/.github/actions/prepare-release-package/action.yml b/.github/actions/prepare-release-package/action.yml new file mode 100644 index 0000000000..7f0bc99383 --- /dev/null +++ b/.github/actions/prepare-release-package/action.yml @@ -0,0 +1,26 @@ +name: Prepare qualified release inputs +description: Restore and validate the exact qualified tarball and its verifier runtime +outputs: + version: + description: Package version validated against the source and qualified manifest + value: ${{ steps.package.outputs.version }} + tarball: + description: Absolute path to the validated qualified npm tarball + value: ${{ steps.package.outputs.tarball }} +runs: + using: composite + steps: + - name: Download the exact qualified package + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0 + with: + name: verified-release-package + path: .cache/verified-package/ + - name: Verify qualified package provenance and digest + id: package + shell: bash + run: node scripts/ci/verified-package.mjs validate .cache/verified-package + - name: Restore qualified verifier runtime + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0 + with: + name: rea-test-runtime + path: . diff --git a/.github/actions/publish-mcp/action.yml b/.github/actions/publish-mcp/action.yml new file mode 100644 index 0000000000..4655b7aa07 --- /dev/null +++ b/.github/actions/publish-mcp/action.yml @@ -0,0 +1,41 @@ +name: Publish MCP Registry metadata +description: Validate and publish one released server.json, preserving an identical existing version +inputs: + directory: + description: Directory containing the released server.json + default: . +runs: + using: composite + steps: + - name: Install MCP Registry publisher + shell: bash + working-directory: ${{ inputs.directory }} + env: + MCP_PUBLISHER_VERSION: v1.8.0 + MCP_PUBLISHER_SHA256_LINUX_AMD64: 1370446bbe74d562608e8005a6ccce02d146a661fbd78674e11cc70b9618d6cf + run: | + archive="mcp-publisher_linux_amd64.tar.gz" + curl -fsSLo "${archive}" \ + "https://github.com/modelcontextprotocol/registry/releases/download/${MCP_PUBLISHER_VERSION}/${archive}" + echo "${MCP_PUBLISHER_SHA256_LINUX_AMD64} ${archive}" | sha256sum --check --strict + tar xzf "${archive}" mcp-publisher + chmod +x mcp-publisher + - name: Validate MCP Registry metadata + shell: bash + working-directory: ${{ inputs.directory }} + run: ./mcp-publisher validate + - name: Check existing registry version + id: registry + shell: bash + working-directory: ${{ inputs.directory }} + run: node "${GITHUB_ACTION_PATH}/registry-state.mjs" + - name: Authenticate to MCP Registry + if: steps.registry.outputs.exists != 'true' + shell: bash + working-directory: ${{ inputs.directory }} + run: ./mcp-publisher login github-oidc + - name: Publish MCP Registry metadata + if: steps.registry.outputs.exists != 'true' + shell: bash + working-directory: ${{ inputs.directory }} + run: ./mcp-publisher publish diff --git a/.github/actions/publish-mcp/registry-state.d.mts b/.github/actions/publish-mcp/registry-state.d.mts new file mode 100644 index 0000000000..f26edc81b9 --- /dev/null +++ b/.github/actions/publish-mcp/registry-state.d.mts @@ -0,0 +1,4 @@ +export function registryPublicationState( + server: Readonly<{ name: string; version: string } & Record>, + request?: (url: URL, init: RequestInit) => Promise, +): Promise<"present" | "absent">; diff --git a/.github/actions/publish-mcp/registry-state.mjs b/.github/actions/publish-mcp/registry-state.mjs new file mode 100644 index 0000000000..e3a32867b4 --- /dev/null +++ b/.github/actions/publish-mcp/registry-state.mjs @@ -0,0 +1,89 @@ +import assert from "node:assert/strict"; +import { appendFile, readFile } from "node:fs/promises"; +import { resolve } from "node:path"; +import { fileURLToPath } from "node:url"; + +/** Exact-version routes currently return false 404s; enumerate the filtered listing. */ +export async function registryPublicationState(server, request = fetch) { + assert.equal(typeof server?.name, "string"); + assert.equal(typeof server?.version, "string"); + const cursors = new Set(); + let cursor; + const matches = []; + do { + const url = new URL( + "https://registry.modelcontextprotocol.io/v0.1/servers", + ); + url.search = new URLSearchParams({ + search: server.name, + version: server.version, + include_deleted: "true", + limit: "100", + }).toString(); + if (cursor !== undefined) url.searchParams.set("cursor", cursor); + const response = await request(url, { + signal: AbortSignal.timeout(30_000), + }); + if (!response.ok) + throw new Error(`Registry lookup failed: HTTP ${response.status}`); + const page = await response.json(); + assert(Array.isArray(page?.servers), "Registry response omitted servers"); + assert( + page.metadata && typeof page.metadata === "object", + "Registry response omitted metadata", + ); + for (const entry of page.servers) { + assert( + typeof entry?.server?.name === "string" && + typeof entry.server.version === "string", + "Malformed registry server record", + ); + if ( + entry.server.name === server.name && + entry.server.version === server.version + ) + matches.push(entry); + } + cursor = page.metadata.nextCursor; + if (cursor !== undefined) { + assert( + typeof cursor === "string" && cursor.length > 0 && !cursors.has(cursor), + "Registry returned a non-advancing cursor", + ); + cursors.add(cursor); + } + } while (cursor !== undefined); + assert( + matches.length <= 1, + "Registry returned ambiguous exact-version records", + ); + if (matches.length === 0) return "absent"; + assert.equal( + matches[0]._meta?.["io.modelcontextprotocol.registry/official"]?.status, + "active", + "Existing registry version is not active", + ); + assert.deepEqual( + matches[0].server, + server, + "Existing registry version differs from released metadata", + ); + return "present"; +} + +if ( + process.argv[1] !== undefined && + resolve(process.argv[1]) === fileURLToPath(import.meta.url) +) { + const state = await registryPublicationState( + JSON.parse(await readFile("server.json", "utf8")), + ); + assert(process.env.GITHUB_OUTPUT, "GITHUB_OUTPUT is required"); + await appendFile( + process.env.GITHUB_OUTPUT, + `exists=${state === "present"}\n`, + ); + process.stdout.write( + `Registry version is ${state}; ${state === "present" ? "preserving identical published metadata" : "publication may proceed"}.\n`, + ); +} diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 3dec52bd07..1c1ef2bd76 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -24,7 +24,7 @@ Unresolved scope, authority, side effects, or required verification block integration. Link the required maintainer scope decision for new providers or capabilities involving target execution, injection, mutation, or protection recovery; ordinary fixes within existing boundaries may state "not applicable". -Draft PRs may leave the scope decision pending and the final review checklist +Draft PRs may leave the scope decision pending and the readiness and merge checklists unchecked. Maintainers can help resolve open questions. ## Contract and boundary impact @@ -55,7 +55,10 @@ Unchecked items do not establish absence; leave uncertainties visible. ## Evidence and regression coverage +show the original failure and explain how the regression would fail on the affected +base revision when practical. Include the most useful proof inline; link long logs +or artifacts, or put them in optional details. Documentation-only changes can use +source, link, or rendered-document checks. --> - Tests added or updated: - Base reproduction or other evidence: @@ -71,7 +74,8 @@ For evidence-bearing changes: ## Validation performed +relevant real-provider or platform workflow, or explain why it was unavailable. +Keep these results current after changes; a command without its result is incomplete. --> - `command` — result @@ -90,6 +94,23 @@ limits, security/privacy/containment implications, and meaningful proof gaps. -- - Package or release metadata impact: - Security, privacy, process, or containment review: +## Author readiness + + + +Human owner: + +- [ ] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. +- [ ] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. +- [ ] I declare this PR ready for initial maintainer review. + ## Before merge # This is NOT the Turborepo you know diff --git a/CHANGELOG.md b/CHANGELOG.md index 8972c63f6f..4c7899ca14 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,118 @@ # Changelog +## [6.5.0](https://github.com/morluto/rea/compare/rea-agents-6.4.0...rea-agents-6.5.0) (2026-10-13) + +Provider prerequisites and recorded verification scope are listed in [the testing guide](docs/testing.md). + +### Highlights + +- Analyze selected JavaScript files and recover cross-bundle relationships without expanding the selected source scope. +- Read compact summaries and page retained Evidence across binary, managed, Android, Dart, WASM and network workflows. +- Inspect SQLite snapshots, packet and signal recordings, WebAssembly, Rust dependency metadata and Electron fuses offline. +- Keep completed Evidence and partial observations available when analysis or cleanup fails. + +### Changes + +- **JavaScript source selection:** analyze exact application paths without expanding imports into the selected scope. Unselected files retain identity and explicit unknown coverage; summary delivery remains separate from source selection. ([#1944](https://github.com/morluto/rea/pull/1944), [issue morluto/rea#1848](https://github.com/morluto/rea/issues/1848)). Thanks @kaoru0822-kitauji, @N0zoM1z0. +- **Code-split recovery:** recover related bundles from multiple selected inputs, retaining per-file provenance and downstream cross-chunk imports. Recorded CLI/MCP coverage is Linux x64 with Wakaru 1.14.0 and source-owned esbuild/webpack fixtures; other hosts, releases and arbitrary application equivalence remain unverified. ([#1686](https://github.com/morluto/rea/pull/1686), [issue morluto/rea#1605](https://github.com/morluto/rea/issues/1605)). Thanks @JingHao-Leon, @pionxzh. +- **Package imports:** resolve exact package import aliases using the selected loader and package ancestry, rather than guessing sibling files. ([#1862](https://github.com/morluto/rea/pull/1862), [issue morluto/rea#1838](https://github.com/morluto/rea/issues/1838)). Thanks @AWUQAQ, @mvanhorn. +- **Retained analysis views:** opt into compact inventory/WASM, managed metadata, Android manifest, Dart AOT, Mach-O and EVM summaries, then page retained Evidence without rerunning analysis. Summary delivery preserves the complete recorded result and its original coverage. ([#2122](https://github.com/morluto/rea/pull/2122), [#2193](https://github.com/morluto/rea/pull/2193), [#2413](https://github.com/morluto/rea/pull/2413), [#2438](https://github.com/morluto/rea/pull/2438), [#2455](https://github.com/morluto/rea/pull/2455), [#2634](https://github.com/morluto/rea/pull/2634), [#2308](https://github.com/morluto/rea/pull/2308), [9a18251](https://github.com/morluto/rea/commit/9a18251426f891e6a24856b715ef8002ab8b9610)). Thanks @kaoru0822-kitauji, @N0zoM1z0. +- **Managed assemblies:** Select declaring types when inspecting members, decode inspected `ldstr` literals with independent coverage, and page retained member summaries. Call and field targets expose `target_declaring_type` and `field_declaring_type`, including supported generic method targets; unresolved metadata remains unknown. ([#1706](https://github.com/morluto/rea/pull/1706), [#2125](https://github.com/morluto/rea/pull/2125), [issue morluto/rea#2121](https://github.com/morluto/rea/issues/2121), [#2162](https://github.com/morluto/rea/pull/2162), [issue morluto/rea#2158](https://github.com/morluto/rea/issues/2158), [#3061](https://github.com/morluto/rea/pull/3061), [issue morluto/rea#3010](https://github.com/morluto/rea/issues/3010), [#2163](https://github.com/morluto/rea/pull/2163), [issue morluto/rea#2160](https://github.com/morluto/rea/issues/2160)). Thanks @kaoru0822-kitauji, @N0zoM1z0, @Nelglor, @skoodster. +- **Packet captures:** inspect saved packets with bounded TShark output, compact capture summaries and retained pages; discover installed protocol/field definitions. Recorded real-engine coverage uses Linux and Wireshark 4.6.6; native Windows/macOS and Bun behavior remain unverified. ([#2318](https://github.com/morluto/rea/pull/2318), [#2340](https://github.com/morluto/rea/pull/2340), [#2496](https://github.com/morluto/rea/pull/2496)). Thanks @3QL, @N0zoM1z0, @Olorinm. +- **Recorded signals:** decode existing recordings and discover installed decoders/options through caller-supplied sigrok-cli 0.7.2. This is Linux-only and requires util-linux `prlimit`; it acquires no hardware samples. ([#2339](https://github.com/morluto/rea/pull/2339), [#2487](https://github.com/morluto/rea/pull/2487)). Thanks @N0zoM1z0, @Olorinm. +- **Web captures:** derive an inferred method/origin/path-template API surface from retained HAR data including observed query-parameter names while retaining unknowns; query values, headers and bodies remain excluded. ([#1891](https://github.com/morluto/rea/pull/1891), [#2982](https://github.com/morluto/rea/pull/2982), [issue morluto/rea#2913](https://github.com/morluto/rea/issues/2913)). Thanks @mvanhorn, @wunianze666-netizen, @zhangguiping-xydt. +- **QUIC captures:** inspect retained aioquic 1.5.0 JSON qlog 0.3 objects and page traces/streams/observations. Other qlog layouts remain unsupported; overlap and loss declarations do not prove retransmission or network loss. ([#2277](https://github.com/morluto/rea/pull/2277)). Thanks @LuisTaffelli, @N0zoM1z0. +- **Fiddler captures:** inspect native SAZ archives with retained HTTP provenance and bounded decoding. Recorded producer coverage is the pinned public encoding corpus; current Fiddler Everywhere variants and Bun-native execution remain unverified. ([#2409](https://github.com/morluto/rea/pull/2409)). Thanks @N0zoM1z0, @zahhak. +- **SQLite snapshots:** inspect an explicitly selected offline database with table/column selection and bounded Evidence. This operation requires Node 24.x >=24.15 or 26+ with `DatabaseSync.limits`; supported Node 22 and earlier 24 runtimes cannot use it. A live database/WAL copy does not establish an atomic snapshot. ([#2271](https://github.com/morluto/rea/pull/2271)). Thanks @N0zoM1z0, @Y0UW1SH. +- **WASM artifacts:** Inspect local modules with bring-your-own WABT 1.0.42 without executing them. Compare imports/exports from two complete core-module inspection Evidence records without requiring an active WABT installation; unsupported signatures and duplicate identities remain unknown, and comparisons do not establish runtime compatibility. Recorded real-tool inspection coverage is macOS arm64. ([#1533](https://github.com/morluto/rea/pull/1533), [issue morluto/rea#884](https://github.com/morluto/rea/issues/884), [#2464](https://github.com/morluto/rea/pull/2464), [issue morluto/rea#2356](https://github.com/morluto/rea/issues/2356)). Thanks @matbrgz, @N0zoM1z0, @thangldw. +- **Go binaries:** inspect embedded build metadata, including file-backed unmapped Mach-O debug segments. ([#1363](https://github.com/morluto/rea/pull/1363), [7fb097d](https://github.com/morluto/rea/commit/7fb097d72c9931dd5dfae481ad2b9d3bb6029f15)). Thanks @Y0UW1SH. +- **Flutter APKs:** identify snapshot/engine builds and inspect AOT structure without an external engine. Recorded real APK coverage uses the documented Flutter Gallery fixture; macOS/Windows and complete pool semantics remain unverified, and pool coverage stays partial. ([#1582](https://github.com/morluto/rea/pull/1582)). Thanks @3QL. +- **HarmonyOS packages:** inventory HAP/HSP/App Pack structures without executing code. The recorded real workflow covers one pinned HAP through repository-built CLI/MCP; installed-package and real HSP/App Pack coverage remain unverified. ([#1447](https://github.com/morluto/rea/pull/1447)). Thanks @BaaRai. +- **Ghidra raw inputs and references:** compatible Linux/macOS Ghidra profiles accept caller-selected raw architecture/entry information and expose typed incoming references at exact addresses. Windows Ghidra P0 admits the raw NMOS 6502 profile described below; other raw targets and the incoming-reference facet remain outside Windows P0. ([#1868](https://github.com/morluto/rea/pull/1868), [issue morluto/rea#1836](https://github.com/morluto/rea/issues/1836), [#1715](https://github.com/morluto/rea/pull/1715), [issue morluto/rea#1245](https://github.com/morluto/rea/issues/1245)). Thanks @bugstan, @casesolved-co-uk, @mvanhorn, @N0zoM1z0. +- **Ghidra MIPS:** Add bounded MIPS32r2/o32 relocatable objects, soft-float integer-register storage, admitted MIPS16 shared ELF, and a narrowly admitted recordless legacy MIPS32 executable profile. The latter pins an endian-matched 32-bit language and records inferred o32 as an analysis assumption. Admission follows each documented ABI/flag/relocation boundary. ([#1949](https://github.com/morluto/rea/pull/1949), [issue morluto/rea#1937](https://github.com/morluto/rea/issues/1937), [#1950](https://github.com/morluto/rea/pull/1950), [issue morluto/rea#1938](https://github.com/morluto/rea/issues/1938), [#1991](https://github.com/morluto/rea/pull/1991), [issue morluto/rea#1851](https://github.com/morluto/rea/issues/1851), [#1898](https://github.com/morluto/rea/pull/1898), [issue morluto/rea#1620](https://github.com/morluto/rea/issues/1620)). Thanks @AurelioB, @BaaRai, @bqw371, @kaoru0822-kitauji, @N0zoM1z0. +- **Ghidra RISC-V:** admit little-endian RV32/RV64 `ET_EXEC` ELF declaring the supported double-float ABI. Relocatable/shared, big-endian and other ABI profiles remain refused; see [the profile and recorded lane](docs/ghidra-riscv.md). ([#2151](https://github.com/morluto/rea/pull/2151)). Thanks @NIU-123370. +- **PSP static ELF profile:** the Ghidra profile is Linux x64, bounded little-endian ELF32 `ET_EXEC` with the documented PSP/ABI declarations and a compatible caller-installed Allegrex extension. PRX, shared/relocatable images, PBP/ISO/CSO and runtime emulation are outside scope. Historical extension experiments are separate from public-provider verification; real public Ghidra PSP coverage remains unverified. ([#1599](https://github.com/morluto/rea/pull/1599)). Thanks @jaelliot. +- **Ghidra analysis settings:** apply validated analyzer overrides before analysis while retaining the selected analysis profile. ([#1913](https://github.com/morluto/rea/pull/1913)). Thanks @akushonkamen, @kaoru0822-kitauji, @N0zoM1z0. +- **Attached native IDA MCP:** add an optional native bridge alongside existing legacy/headless profiles. Recorded native runs use Linux IDA Free 9.4 with a navigation correction and Windows IDA Pro 9.3 with a loopback-listener change; stock native switching, native macOS and Windows IDA Free remain unverified. Attached databases remain read-only and are never saved or closed. ([#1544](https://github.com/morluto/rea/pull/1544)). Thanks @imissapixel, @N0zoM1z0. +- **Frida instrumentation:** add explicit local/remote target selection, caller-supplied scripts and retained messages, with script-read/input diagnostics and bounded message eviction. Recorded real coverage is the Linux local Node marker workflow; macOS, Windows and remote devices/servers remain unverified. Instrumentation executes foreign code and is destructive, not read-only. ([#1359](https://github.com/morluto/rea/pull/1359), [#2211](https://github.com/morluto/rea/pull/2211), [issue morluto/rea#2209](https://github.com/morluto/rea/issues/2209), [#2217](https://github.com/morluto/rea/pull/2217), [issue morluto/rea#2213](https://github.com/morluto/rea/issues/2213), [#2628](https://github.com/morluto/rea/pull/2628)). Thanks @dandgabr, @kaoru0822-kitauji, @N0zoM1z0. +- **Client setup:** register Qoder, Trae, ZCode, DeepSeek Harness, Kimi Code, Factory Droid and WorkBuddy through the existing scoped setup flow. DeepSeek is included from 6.5.0; the historical npm 6.4 check did not cover it. Live Trae and DeepSeek API integration remain unverified. ([#1564](https://github.com/morluto/rea/pull/1564), [#1572](https://github.com/morluto/rea/pull/1572), [#1573](https://github.com/morluto/rea/pull/1573), [#1929](https://github.com/morluto/rea/pull/1929), [issue morluto/rea#1839](https://github.com/morluto/rea/issues/1839), [#2744](https://github.com/morluto/rea/pull/2744), [#2309](https://github.com/morluto/rea/pull/2309), [#3041](https://github.com/morluto/rea/pull/3041), [issue morluto/rea#3040](https://github.com/morluto/rea/issues/3040)). Thanks @0xRafie, @BaaRai, @cainiao1992, @IvenKooLab, @whoishaha, @ysntony. +- **Website:** Add maintained language editions, persistent/system-aware themes, keyboard styling and nested Evidence indentation, plus the reverse-engineering models essay. A simpler home page and beginner journey connect platform prerequisites, client/provider setup and a supplied first investigation. ([#1779](https://github.com/morluto/rea/pull/1779), [issue morluto/rea#1022](https://github.com/morluto/rea/issues/1022), [#1699](https://github.com/morluto/rea/pull/1699), [#1714](https://github.com/morluto/rea/pull/1714), [#1894](https://github.com/morluto/rea/pull/1894), [#2250](https://github.com/morluto/rea/pull/2250), [#2432](https://github.com/morluto/rea/pull/2432), [#2732](https://github.com/morluto/rea/pull/2732), [#3274](https://github.com/morluto/rea/pull/3274)). Thanks @CassieFox23, @kaoru0822-kitauji, @N0zoM1z0. + +- **Rust dependency metadata:** Inspect embedded cargo-auditable package/dependency records in ELF, PE and thin Mach-O without executing the file. Missing metadata and inferred compiler hashes retain explicit limitations; universal Mach-O requires a selected thin slice. ([#2456](https://github.com/morluto/rea/pull/2456)). Thanks @mvanhorn. +- **Electron fuses:** Read packaged fuse bytes and per-slice runtime implications without launching or modifying the app. Missing, truncated and unknown wires remain explicit; fuse observations do not prove other Inspector-enablement paths. ([#2399](https://github.com/morluto/rea/pull/2399)). Thanks @kaoru0822-kitauji, @mvanhorn, @N0zoM1z0. +- **Ghidra PowerPC:** Admit bounded ELF32 big-endian `ET_EXEC` PowerPC with an explicit language profile on supported Linux/macOS hosts. PPC64, DOL and other variants remain outside this lane; see [the profile and verification prerequisites](docs/ghidra-ppc.md). ([#1747](https://github.com/morluto/rea/pull/1747)). Thanks @arielfikru. + +- **Reference-source summaries:** Use CLI `import-reference-source --detail summary` for compact manifest, count, state and path-outline output. The complete graph remains available and is still the default; hidden outline entries retain separate coverage counts. ([#2768](https://github.com/morluto/rea/pull/2768), [issue morluto/rea#2442](https://github.com/morluto/rea/issues/2442)). Thanks @matbrgz, @To3akaRin. + +- **Recorded AArch64 crashes:** Decode 34 ABI register values per thread from supported Linux ELF64 little-endian core notes using the documented caller-provided Python dependencies. These are recorded facts, not proof of current process or library identity. ([#2505](https://github.com/morluto/rea/pull/2505), [issue morluto/rea#2353](https://github.com/morluto/rea/issues/2353)). Thanks @matbrgz, @NIU-123370. + +- **Ghidra byte search:** Search initialized analyzed memory using wildcards, bit masks, address bounds and paged results. Unloaded, overlay and uninitialized blocks retain skipped coverage; this is not a raw-file search. ([#2341](https://github.com/morluto/rea/pull/2341)). Thanks @mvanhorn. + +- **Raw NMOS 6502 images:** Analyze explicitly selected headerless images with Ghidra, including Windows P0, using a caller-selected entry and source mapping within the 16-bit address space. Cartridge parsing, reset-vector inference and 65C02/65816 are outside this profile. ([#2079](https://github.com/morluto/rea/pull/2079)). Thanks @Zhong-fan. + +- **Swift String observations:** Ghidra function dossiers derive supported AArch64 Swift String literals from constant-register pairs at recognized use sites, retaining instruction/register provenance and explicit omissions. This is a bounded derivation, not a complete string inventory. ([#2049](https://github.com/morluto/rea/pull/2049), [issue morluto/rea#2030](https://github.com/morluto/rea/issues/2030)). Thanks @thefolahan. + +- **Headless IDA procedure lookup:** Resolve addresses to canonical containing-procedure entries and identify the session’s admitted program document in the modern headless profile. Incomplete surveys remain errors; attached profiles do not advertise this containment capability. ([#2374](https://github.com/morluto/rea/pull/2374), [issue morluto/rea#2335](https://github.com/morluto/rea/issues/2335)). Thanks @nikolay-dementiev. + +- **Passive browser attachment:** Six existing passive browser tools accept an explicit-port loopback browser WebSocket endpoint when HTTP target discovery is unavailable. Existing consent and target/origin checks remain required; this does not launch a browser or enable runtime instrumentation. ([#2346](https://github.com/morluto/rea/pull/2346)). Thanks @all-black-493. + +- **Android DEX references:** Retain exact method descriptors and individual direct-invoke observations with source method, DEX entry, opcode and code-unit offset. These are static references; unavailable instruction facts remain separate from method-level usage totals. ([#2439](https://github.com/morluto/rea/pull/2439)). Thanks @LcokHcak. + +- **Offline EVM metadata:** Retain bounded, lossless facts from plausible terminal Solidity CBOR root-map candidates. Candidates do not prove Solidity origin or source/compiler identity, and linked metadata is not fetched. ([#2790](https://github.com/morluto/rea/pull/2790)). Thanks @matbrgz. + +### Fixes + +- **JavaScript loader and export identity:** Respect import/require conditions and unavailable package metadata, keep export links separate from writes, and resolve default identifier exports to the value captured at the export statement. ([7e5cffa](https://github.com/morluto/rea/commit/7e5cffa2ae9dfc9afaeca0715110d4dd41c0dcdd), [322506d](https://github.com/morluto/rea/commit/322506d0c0d39d3cd1e2e1c285c2cc23cfa73ee7), [81f7638](https://github.com/morluto/rea/commit/81f763820319a762f003cf95651ed14ce9326a4c), [dae1e55](https://github.com/morluto/rea/commit/dae1e55381f0d6cfee7f2082d5d27b938e55704c)). Thanks @paoValle, @VarunGore36, @xu91102. +- **JavaScript traces:** Retain nested bridge calls and surviving early-return predicates. Honor node-ID match options, match route seeds against source values, and keep ambiguous runtime candidates unknown. ([0351328](https://github.com/morluto/rea/commit/0351328e54ce70ea3f2814482e76b132774aff48), [5031099](https://github.com/morluto/rea/commit/503109995f9dcf8a955585d39a344fc846d2f59e), [44c9d29](https://github.com/morluto/rea/commit/44c9d2924d5b95d7a86595e8d7b7cf4fd4f0f6ac), [5e79d46](https://github.com/morluto/rea/commit/5e79d468e5864204cbe79a7ca2898541cee5e012)). Thanks @jpdefo. +- **Browser comparison coverage:** Compare only known redirect response fields and retain network-selection recovery guidance alongside independent event changes. Missing response facts do not prove equality or a change. ([2b60e09](https://github.com/morluto/rea/commit/2b60e097f281f7708c69f1f7ba9263694547bcc0)). +- **Evidence comparison and ledger input:** Report display-only bundle differences as unchanged while retaining digests, and classify noncanonical bundle records as invalid caller input. ([030c477](https://github.com/morluto/rea/commit/030c47732833c9cedcc31db1309eb3a1ca2c62e8), [3a34ad3](https://github.com/morluto/rea/commit/3a34ad37bdb7b8debba0f6515ba85b80141f2716)). +- **Capture and admission cleanup:** Keep deleted-file observations and original bytes when a parent becomes a file, preserve unrelated I/O failures during cancellation, and require explicit pre-launch proof before releasing an unknown LLDB target owner. LLDB failure controls used simulated boundaries. ([e67c364](https://github.com/morluto/rea/commit/e67c3642c8920a2a9c0e231ee2f16f50b245df31), [2daeff4](https://github.com/morluto/rea/commit/2daeff4b243e1cd137c591ca394e18a1c05c8bbc), [45b6a7f](https://github.com/morluto/rea/commit/45b6a7fe8f51a89951aaa823ffa627f6f5453a33)). Thanks @SII-YDD, @xu91102. +- **Client configuration:** Validate MCP output-schema profiles before setup writes, ignore untrusted Gemini workspace settings before parsing them, and identify installed skills independently of marker-only client roots. ([0930e94](https://github.com/morluto/rea/commit/0930e94e077ab7fb4d695f62785919a8a25c2e21), [aade9bb](https://github.com/morluto/rea/commit/aade9bbcb79aa8504ab6921b25f8ab082aba78e7), [ba33c70](https://github.com/morluto/rea/commit/ba33c706d26b63e43308be50c0249c72cff62cc6)). Thanks @CmsChase, @paoValle, @sx4im. +- **Host tool discovery:** Share macOS Hopper launcher discovery across setup and runtime, preserve explicit paths, and report validated ADB and Apktool availability. ([7582f0d](https://github.com/morluto/rea/commit/7582f0dff29ae0e80c97b885ddd386ac61a92570), [e5deb83](https://github.com/morluto/rea/commit/e5deb832cb09bb524877d0df40f49d5b5bb7dea4)). Thanks @Chessing234, @kaoru0822-kitauji, @viniciusdsandrade. +- **SAZ encoding validation:** Reject invalid XML bytes, unsupported encodings and declarations that contradict the archive bytes, including records outside the selected request. ([1250aae](https://github.com/morluto/rea/commit/1250aaedd8b4d1e471d8a382f76418bde65c05d2)). +- **Managed member matching:** Keep malformed TypeDef list ownership unknown, distinguish constants during structural method matching, and accept legacy small exception-section sizes. ([3f6b575](https://github.com/morluto/rea/commit/3f6b5755b3e699bd9373641acffe0f2c5d2ab094), [ee5cfbe](https://github.com/morluto/rea/commit/ee5cfbe05cd309a88003c711721e67a3bee36554)). +- **Bounded diagnostics:** Bound SDK validation messages and terminal normalization before allocation while retaining diagnostics and explicit unknown final output sizes. ([7fb0a0c](https://github.com/morluto/rea/commit/7fb0a0cc577f7c9033a2674d877d8242b5864556), [1d826f3](https://github.com/morluto/rea/commit/1d826f304f032acfe69dad3ff824b300aacaeb43)). Thanks @CRYPTONIKAV. + +- **Windows artifact classification:** identify `.exe`, `.com`, `.bat` and `.cmd` artifacts as launchable host executables. ([#1823](https://github.com/morluto/rea/pull/1823), [#1824](https://github.com/morluto/rea/pull/1824), [issue morluto/rea#1819](https://github.com/morluto/rea/issues/1819)). Thanks @dollarsmoney, @JasonBuildAI, @kaoru0822-kitauji. + +- **CLI/MCP delivery:** defer command-specific startup work, bundle runtime entries, retain field/input correction details across output formats and preserve shared optional-provider lifetime. MCP selectors, paging continuation and retained partial-error references remain explicit; session liveness/provider-attribution corrections retain their controlled-provider verification limits. ([#1691](https://github.com/morluto/rea/pull/1691), [issue morluto/rea#1685](https://github.com/morluto/rea/issues/1685), [#1705](https://github.com/morluto/rea/pull/1705), [issue morluto/rea#1695](https://github.com/morluto/rea/issues/1695), [#1755](https://github.com/morluto/rea/pull/1755), [issue morluto/rea#1746](https://github.com/morluto/rea/issues/1746), [#1773](https://github.com/morluto/rea/pull/1773), [issue morluto/rea#1771](https://github.com/morluto/rea/issues/1771), [#1780](https://github.com/morluto/rea/pull/1780), [#1814](https://github.com/morluto/rea/pull/1814), [#1910](https://github.com/morluto/rea/pull/1910), [issue morluto/rea#1849](https://github.com/morluto/rea/issues/1849), [#1996](https://github.com/morluto/rea/pull/1996), [#2140](https://github.com/morluto/rea/pull/2140), [issue morluto/rea#2135](https://github.com/morluto/rea/issues/2135), [#2205](https://github.com/morluto/rea/pull/2205), [issue morluto/rea#2195](https://github.com/morluto/rea/issues/2195), [#2230](https://github.com/morluto/rea/pull/2230), [issue morluto/rea#2180](https://github.com/morluto/rea/issues/2180), [#2294](https://github.com/morluto/rea/pull/2294), [#2482](https://github.com/morluto/rea/pull/2482), [a82d14e](https://github.com/morluto/rea/commit/a82d14e4db4dca989ffa2e7dbe78fd43551da256), [a4dd4e8](https://github.com/morluto/rea/commit/a4dd4e8e8700ce350e1a0c612ebd61b91ca5e585), [48c352b](https://github.com/morluto/rea/commit/48c352b6bfa34c820a530272cd36600158c37b1e), [07b36c1](https://github.com/morluto/rea/commit/07b36c148946f46b99a4ce04e55726e97d0ee03a), [570c40c](https://github.com/morluto/rea/commit/570c40cde55d31e455ef9bc484b7b908658ded58), [8fc2043](https://github.com/morluto/rea/commit/8fc2043eb8bcedbf34dce064f977e455438635d6), [7fc0b5e](https://github.com/morluto/rea/commit/7fc0b5ec9c2532bbf789d7783b7a0cab6fc054d8)). Thanks @369Apps, @chqcjq, @cold-atom, @coygeek, @JackOdegard, @kaoru0822-kitauji, @N0zoM1z0, @sx4im. +- **Evidence import/replay:** import bundles across session transitions while authenticating selected target and file identities, preserve literal keys, metadata/mutation intent and relationship coverage, and honor cancellation before session mutation. Reuse authenticated snapshots and comparisons within their owned lifetime; reject malformed/non-object snapshots and unsupported metadata without disguising uncertainty. ([#1694](https://github.com/morluto/rea/pull/1694), [#1716](https://github.com/morluto/rea/pull/1716), [issue morluto/rea#1680](https://github.com/morluto/rea/issues/1680), [#1721](https://github.com/morluto/rea/pull/1721), [issue morluto/rea#1718](https://github.com/morluto/rea/issues/1718), [#1733](https://github.com/morluto/rea/pull/1733), [issue morluto/rea#1723](https://github.com/morluto/rea/issues/1723), [issue morluto/rea#1724](https://github.com/morluto/rea/issues/1724), [#1776](https://github.com/morluto/rea/pull/1776), [#1873](https://github.com/morluto/rea/pull/1873), [#2001](https://github.com/morluto/rea/pull/2001), [#2013](https://github.com/morluto/rea/pull/2013), [issue morluto/rea#2009](https://github.com/morluto/rea/issues/2009), [#2083](https://github.com/morluto/rea/pull/2083), [#2303](https://github.com/morluto/rea/pull/2303), [#2685](https://github.com/morluto/rea/pull/2685), [issue morluto/rea#2675](https://github.com/morluto/rea/issues/2675), [c002b89](https://github.com/morluto/rea/commit/c002b89e2d8130a3c07e25e5fd604876be18c703), [9c85114](https://github.com/morluto/rea/commit/9c8511409c0706f9979c2b68dd56aaaf9f21e191), [aada0ca](https://github.com/morluto/rea/commit/aada0ca8126bf14c60750a4ccd4a79667770a1e4), [4a4d36f](https://github.com/morluto/rea/commit/4a4d36f881861bbab3c734b0dd459f805766caec), [1c64fe3](https://github.com/morluto/rea/commit/1c64fe3eff48679c6880f42d2f0a4af46686ade7), [6fb3a34](https://github.com/morluto/rea/commit/6fb3a345fe952b8e8f59221aec993acc37ccbf31), [3bcc68c](https://github.com/morluto/rea/commit/3bcc68c037e9415c78093cb286069c1fcca6a2f8), [6bae774](https://github.com/morluto/rea/commit/6bae774328724403255590aa2ddad4d7a2aa727f), [e997085](https://github.com/morluto/rea/commit/e9970855ba16ba711816ac30212c5ca17ae085cd), [d1dcebc](https://github.com/morluto/rea/commit/d1dcebce2c0820469ed63e33497a14258971b710), [b123b2e](https://github.com/morluto/rea/commit/b123b2e51cc7d6d2157525664be2df88fb1ba3bd), [049132a](https://github.com/morluto/rea/commit/049132ab43fa962d31774f2b258c15a71217148b), [042dcaa](https://github.com/morluto/rea/commit/042dcaa1ee66f1b8668f08b193f38ab01bceaacb), [acf75e6](https://github.com/morluto/rea/commit/acf75e6700ace28bb1db25e54e56673ac15e0ed3), [1f7f808](https://github.com/morluto/rea/commit/1f7f808c4f3bb5eca6b13bcc71d5cceb2ace6614), [26bffb9](https://github.com/morluto/rea/commit/26bffb94bf80355c35e22a98a8fc629c6210307e)). Thanks @coygeek, @DeryFerd, @jiangjiang248, @kaoru0822-kitauji, @khanukov, @N0zoM1z0, @NIU-123370, @rohitg00, @sx4im. +- **Ghidra setup and analysis:** preserve launcher/JVM diagnostics, Java-property and extension identity, stale-client recovery, UTF-8 bridge output and pending-startup progress. File-offset decompilation is Ghidra-specific; recorded coverage is Linux x64 with Ghidra 12.1.4. Startup/progress/compiler failure paths that use simulations remain unverified with a real engine. ([#1698](https://github.com/morluto/rea/pull/1698), [#1877](https://github.com/morluto/rea/pull/1877), [#1931](https://github.com/morluto/rea/pull/1931), [#2011](https://github.com/morluto/rea/pull/2011), [issue morluto/rea#1992](https://github.com/morluto/rea/issues/1992), [#2014](https://github.com/morluto/rea/pull/2014), [#2101](https://github.com/morluto/rea/pull/2101), [issue morluto/rea#2078](https://github.com/morluto/rea/issues/2078), [#2206](https://github.com/morluto/rea/pull/2206), [issue morluto/rea#2204](https://github.com/morluto/rea/issues/2204), [2187e16](https://github.com/morluto/rea/commit/2187e16f149a7a7b512a4cb45d5f468cc04091fe), [b953f96](https://github.com/morluto/rea/commit/b953f96c0aae27e1876c317a8f7db1d6f292b8ff)). Thanks @EDMND-SRC, @Ha1baraA11, @kaoru0822-kitauji, @lovecontroing, @lyly-bonbon, @N0zoM1z0, @nosmile99, @Petru101, @sx4im, @Zhong-fan. +- **Process and cleanup ownership:** retain private runtime/file owners and detailed incomplete-cleanup observations, bound trace cycles/order state, preserve caller paths, and use monotonic time for budgets while retaining wall dates. Windows affinity/admission ([#2202](https://github.com/morluto/rea/pull/2202), [issue morluto/rea#2181](https://github.com/morluto/rea/issues/2181)) and Bun memory-enforcement paths retain their recorded native-versus-controlled-fixture limits; these fixes are not a general process-ownership guarantee. ([#1700](https://github.com/morluto/rea/pull/1700), [issue morluto/rea#1697](https://github.com/morluto/rea/issues/1697), [#1730](https://github.com/morluto/rea/pull/1730), [#1756](https://github.com/morluto/rea/pull/1756), [issue morluto/rea#1753](https://github.com/morluto/rea/issues/1753), [#1760](https://github.com/morluto/rea/pull/1760), [issue morluto/rea#1757](https://github.com/morluto/rea/issues/1757), [#1826](https://github.com/morluto/rea/pull/1826), [#1907](https://github.com/morluto/rea/pull/1907), [issue morluto/rea#1856](https://github.com/morluto/rea/issues/1856), [#1999](https://github.com/morluto/rea/pull/1999), [issue morluto/rea#1993](https://github.com/morluto/rea/issues/1993), [#2232](https://github.com/morluto/rea/pull/2232), [issue morluto/rea#2170](https://github.com/morluto/rea/issues/2170), [#2252](https://github.com/morluto/rea/pull/2252), [issue morluto/rea#2218](https://github.com/morluto/rea/issues/2218), [#2467](https://github.com/morluto/rea/pull/2467), [issue morluto/rea#2257](https://github.com/morluto/rea/issues/2257), [#2468](https://github.com/morluto/rea/pull/2468), [#2488](https://github.com/morluto/rea/pull/2488), [09b6828](https://github.com/morluto/rea/commit/09b68285166bef2b7633402c8adf7eb25823b47e), [7285e15](https://github.com/morluto/rea/commit/7285e15839a4fecf9940683699b781a60ccc4775), [b5ededd](https://github.com/morluto/rea/commit/b5ededd0f9c28d0f696a205103066928117368ee), [d8c76b6](https://github.com/morluto/rea/commit/d8c76b629639bda4405afa9ae2f7676b694f2f10), [98d8470](https://github.com/morluto/rea/commit/98d8470d36cfbf7cd25820a4ed430aebae20c0c5), [c22111e](https://github.com/morluto/rea/commit/c22111eca2c4bc84a084f12711f405b3e7422a40), [cecc6ec](https://github.com/morluto/rea/commit/cecc6ecbffe84689a173d1634f342e046b0c8ceb), [10935b7](https://github.com/morluto/rea/commit/10935b7b1136accab2a43b308ab380749da0f7ae), [63e796d](https://github.com/morluto/rea/commit/63e796d50ea323ae8dcfef80f86c8154bb106790), [d9e9fb2](https://github.com/morluto/rea/commit/d9e9fb2273543caab4eb21d896c33d462ada6971), [9049dba](https://github.com/morluto/rea/commit/9049dbaf0c3df32357419f9899965df2b611f493), [ed881c6](https://github.com/morluto/rea/commit/ed881c6d2ef6a8adba0c2a241ebc9ab4603b7868)). Thanks @Ashen00001, @charan-rathore, @coygeek, @EDMND-SRC, @JasonBuildAI, @jimpablo, @kaoru0822-kitauji, @N0zoM1z0, @NIU-123370, @Rinkia, @rudycelekli, @sx4im, @xeropresence, @XiaoBinGan, @xthunder0, @hzh666kevin-hue. +- **Native image identity:** preserve universal Mach-O containers and selected slice/subtype identity, chained fixup locations and library ordinals, exact signing text, PE resource capacity and ELF/header framing. Mach-O summary selection and independent slice commands avoid unnecessary delivery/work. Controlled routing/parser fixtures do not establish every installed-provider path. ([#1703](https://github.com/morluto/rea/pull/1703), [issue morluto/rea#1702](https://github.com/morluto/rea/issues/1702), [#1708](https://github.com/morluto/rea/pull/1708), [#1709](https://github.com/morluto/rea/pull/1709), [#1713](https://github.com/morluto/rea/pull/1713), [#1772](https://github.com/morluto/rea/pull/1772), [issue morluto/rea#1765](https://github.com/morluto/rea/issues/1765), [#1784](https://github.com/morluto/rea/pull/1784), [issue morluto/rea#1775](https://github.com/morluto/rea/issues/1775), [#1982](https://github.com/morluto/rea/pull/1982), [#1995](https://github.com/morluto/rea/pull/1995), [issue morluto/rea#1990](https://github.com/morluto/rea/issues/1990), [#2040](https://github.com/morluto/rea/pull/2040), [#2107](https://github.com/morluto/rea/pull/2107), [#2161](https://github.com/morluto/rea/pull/2161), [issue morluto/rea#2105](https://github.com/morluto/rea/issues/2105), [#2248](https://github.com/morluto/rea/pull/2248), [issue morluto/rea#2176](https://github.com/morluto/rea/issues/2176), [#2281](https://github.com/morluto/rea/pull/2281), [issue morluto/rea#2274](https://github.com/morluto/rea/issues/2274), [#2292](https://github.com/morluto/rea/pull/2292), [#2466](https://github.com/morluto/rea/pull/2466), [#2506](https://github.com/morluto/rea/pull/2506), [#2733](https://github.com/morluto/rea/pull/2733), [issue morluto/rea#2661](https://github.com/morluto/rea/issues/2661), [f598bab](https://github.com/morluto/rea/commit/f598bab488e38c378d69346fa3c106f9e18529ed), [108b5f9](https://github.com/morluto/rea/commit/108b5f94254281639badd25b1f448cca35dc89a2), [f1924cf](https://github.com/morluto/rea/commit/f1924cf9c1910d5d29d11e7bf634b51af8d738cf), [8bd5ba2](https://github.com/morluto/rea/commit/8bd5ba2d234cbd9a5bb9dd5daeaa191c3a1cad24), [b38429c](https://github.com/morluto/rea/commit/b38429cb6a0de8f4bb5ae01633e24abe2d7cde46), [8d4d631](https://github.com/morluto/rea/commit/8d4d631606c701aac150cf009d3b80ae4a7ae7d5), [ac57c88](https://github.com/morluto/rea/commit/ac57c88efe64c3125850b54d3d833227e6b8ec51), [dc038d1](https://github.com/morluto/rea/commit/dc038d16bba9046afd28970de9c47f6259a43524), [34a4bb6](https://github.com/morluto/rea/commit/34a4bb6508006fa113463ef20ed799a2994e9ee8), [ebc22d9](https://github.com/morluto/rea/commit/ebc22d99d62225505c0a84f6e39823ba4ef2f829), [c02fb0e](https://github.com/morluto/rea/commit/c02fb0ea8055f98def2b12ad557aed0faefd3639)). Thanks @1NormalGuy, @Aliciawque, @baron, @coygeek, @dhl84, @hemanthrayuduu, @JasonBuildAI, @kaoru0822-kitauji, @N0zoM1z0, @thangldw, @viniciusdsandrade, @XiaoBinGan, @Zburgers. +- **Archive and artifact provenance:** keep physical ZIP occurrence identity, exact signatures/names and partial inventory coverage; bound decompression and reject malformed keys/integers. Directory/ASAR/DMG reads retain ownership through cleanup and publication cancellation. Duplicate ZIP names remain explicitly ambiguous where the public projection cannot represent them; synthetic archive/cleanup cases do not prove every real Unblob or DMG failure path. ([#1726](https://github.com/morluto/rea/pull/1726), [issue morluto/rea#1725](https://github.com/morluto/rea/issues/1725), [#1854](https://github.com/morluto/rea/pull/1854), [issue morluto/rea#1844](https://github.com/morluto/rea/issues/1844), [#1865](https://github.com/morluto/rea/pull/1865), [issue morluto/rea#1860](https://github.com/morluto/rea/issues/1860), [#2069](https://github.com/morluto/rea/pull/2069), [#2106](https://github.com/morluto/rea/pull/2106), [#2144](https://github.com/morluto/rea/pull/2144), [#2365](https://github.com/morluto/rea/pull/2365), [#2493](https://github.com/morluto/rea/pull/2493), [918713a](https://github.com/morluto/rea/commit/918713af376e307ecf24d75e6a0dd9d1b544545b), [eb97da3](https://github.com/morluto/rea/commit/eb97da33881e3d1472d97a7d788ee35944fb6b07), [2260887](https://github.com/morluto/rea/commit/2260887f2092be6a3d9c79509e4ae4a3e52fce7f), [74d7da6](https://github.com/morluto/rea/commit/74d7da6ced0f869936ee0316b00616931b088066), [ed485a3](https://github.com/morluto/rea/commit/ed485a3dc8cc3a4baf8cabe01acb7aa0490b73b3), [921bb95](https://github.com/morluto/rea/commit/921bb9575505065f6cfd57b14d97d80d0f238546), [d8ceb3f](https://github.com/morluto/rea/commit/d8ceb3f276a65c181b146f68c0729ff4b6f9fa61), [9435c7a](https://github.com/morluto/rea/commit/9435c7aae279ba184937c0dfeb713fd574ba3f29), [bb54082](https://github.com/morluto/rea/commit/bb540826d65316c164d2de8ba4ef10f81a8ad073), [a73a55f](https://github.com/morluto/rea/commit/a73a55f53cc2b2e5b7618effe6ecb5cee6e0cee2), [3c57e1a](https://github.com/morluto/rea/commit/3c57e1a8b21b379fca55032d177078e1d8426e64), [ec0b081](https://github.com/morluto/rea/commit/ec0b081371f118e6e5e042fbef3479359c672ddb), [e7882b3](https://github.com/morluto/rea/commit/e7882b34f4ac4a26f903c5f49264acc2c19bb562), [b955b82](https://github.com/morluto/rea/commit/b955b825fc002a4bb408f2910b78768c566c2cd1)). Thanks @1052326311, @1NormalGuy, @baron, @Bortlesboat, @Chaoqi31, @DeryFerd, @kaoru0822-kitauji, @N0zoM1z0, @XiaoBinGan. +- **Setup/uninstall recovery:** preserve unrelated YAML text, caller-selected roots, symlinked data and existing recovery snapshots; explain malformed configuration shapes and retain failed rollback/cleanup owners. Show terminal dry-run plans and each selected skill location. Bunx registration and fault-injected recovery cases remain simulated rather than live-client verification. ([#1731](https://github.com/morluto/rea/pull/1731), [#1754](https://github.com/morluto/rea/pull/1754), [#1821](https://github.com/morluto/rea/pull/1821), [issue morluto/rea#1816](https://github.com/morluto/rea/issues/1816), [#1983](https://github.com/morluto/rea/pull/1983), [#2118](https://github.com/morluto/rea/pull/2118), [issue morluto/rea#2111](https://github.com/morluto/rea/issues/2111), [#2154](https://github.com/morluto/rea/pull/2154), [issue morluto/rea#2152](https://github.com/morluto/rea/issues/2152), [#2157](https://github.com/morluto/rea/pull/2157), [#2182](https://github.com/morluto/rea/pull/2182), [issue morluto/rea#2159](https://github.com/morluto/rea/issues/2159), [#2481](https://github.com/morluto/rea/pull/2481), [bb50813](https://github.com/morluto/rea/commit/bb508134efbba392a404b651d6ec05be8b7eac1e), [c240768](https://github.com/morluto/rea/commit/c2407689cb04df082e8db0b6afa8f8d022f6bcd1), [dcfc5d4](https://github.com/morluto/rea/commit/dcfc5d403ac997c2b08bee4753e57a1a4a8a8538), [21eff3b](https://github.com/morluto/rea/commit/21eff3bae10f58711a35ad44f86e9f96f9bb4790), [94153f1](https://github.com/morluto/rea/commit/94153f1c757e07a6d092e00903aa8b125d4a0afa), [2397137](https://github.com/morluto/rea/commit/23971376158e7815edcbe6c04a8df7c9504650dc), [c339f9e](https://github.com/morluto/rea/commit/c339f9ebe7bd1f30608d6ccb7175da523dd3c485), [f6775a3](https://github.com/morluto/rea/commit/f6775a36694a6a9c620f4248638e8056a6cbcb66)). Thanks @BeqaKs, @Jax367, @kaoru0822-kitauji, @NIU-123370, @shoyann, @viniciusdsandrade, @xeropresence, @zaydiscold. +- **Bun updates:** locate global installations before changing directories and reuse parsed client activation during maintenance. The update flow is covered with package-manager/filesystem simulations; a real Bun global installation update remains unverified. ([#1735](https://github.com/morluto/rea/pull/1735), [e90acae](https://github.com/morluto/rea/commit/e90acaea674f5f7f9f17ea472104afdc1422ada6), [b045df1](https://github.com/morluto/rea/commit/b045df100e1489e08a36d6ff04cea7feecf1332c)). Thanks @xeropresence. +- **JavaScript relationships:** preserve loader-specific package/TypeScript/CommonJS boundaries, exact file-URL and source-map identity, Electron intrinsic receivers/IPC guards and chunk metadata. Index repeated lookups and retain completed facts through resource limits and cancellation. Real Electron integration, Bun worker integration and failure-path cleanup remain unverified where the recorded lanes use fixtures. ([#1740](https://github.com/morluto/rea/pull/1740), [#1759](https://github.com/morluto/rea/pull/1759), [issue morluto/rea#1748](https://github.com/morluto/rea/issues/1748), [#1762](https://github.com/morluto/rea/pull/1762), [issue morluto/rea#1761](https://github.com/morluto/rea/issues/1761), [#1767](https://github.com/morluto/rea/pull/1767), [#1788](https://github.com/morluto/rea/pull/1788), [issue morluto/rea#1781](https://github.com/morluto/rea/issues/1781), [issue morluto/rea#1782](https://github.com/morluto/rea/issues/1782), [#1789](https://github.com/morluto/rea/pull/1789), [#1799](https://github.com/morluto/rea/pull/1799), [issue morluto/rea#1793](https://github.com/morluto/rea/issues/1793), [#1827](https://github.com/morluto/rea/pull/1827), [issue morluto/rea#1802](https://github.com/morluto/rea/issues/1802), [#1833](https://github.com/morluto/rea/pull/1833), [issue morluto/rea#1831](https://github.com/morluto/rea/issues/1831), [#1857](https://github.com/morluto/rea/pull/1857), [issue morluto/rea#1842](https://github.com/morluto/rea/issues/1842), [#1871](https://github.com/morluto/rea/pull/1871), [#1874](https://github.com/morluto/rea/pull/1874), [#1905](https://github.com/morluto/rea/pull/1905), [#1912](https://github.com/morluto/rea/pull/1912), [issue morluto/rea#1909](https://github.com/morluto/rea/issues/1909), [#1915](https://github.com/morluto/rea/pull/1915), [issue morluto/rea#1911](https://github.com/morluto/rea/issues/1911), [#1916](https://github.com/morluto/rea/pull/1916), [#1921](https://github.com/morluto/rea/pull/1921), [#1934](https://github.com/morluto/rea/pull/1934), [#1939](https://github.com/morluto/rea/pull/1939), [issue morluto/rea#1881](https://github.com/morluto/rea/issues/1881), [issue morluto/rea#1882](https://github.com/morluto/rea/issues/1882), [issue morluto/rea#1883](https://github.com/morluto/rea/issues/1883), [#1941](https://github.com/morluto/rea/pull/1941), [#2062](https://github.com/morluto/rea/pull/2062), [issue morluto/rea#2058](https://github.com/morluto/rea/issues/2058), [#2104](https://github.com/morluto/rea/pull/2104), [#2422](https://github.com/morluto/rea/pull/2422), [#2475](https://github.com/morluto/rea/pull/2475), [#2737](https://github.com/morluto/rea/pull/2737), [issue morluto/rea#2663](https://github.com/morluto/rea/issues/2663), [d3088c3](https://github.com/morluto/rea/commit/d3088c3795f6c37cb3b6aec0bb0c371dd0640ee8), [1dfb933](https://github.com/morluto/rea/commit/1dfb933e47db0156cf1cb61f28715b81e740c57b), [f8513a4](https://github.com/morluto/rea/commit/f8513a42dc44451a621127f06edde5d38bc93f69), [b8fab74](https://github.com/morluto/rea/commit/b8fab7429771bb17a7ca0297661de3c9935212af), [42b0ed1](https://github.com/morluto/rea/commit/42b0ed126df7d5b908ac369715378d5e519051b7), [c0904b4](https://github.com/morluto/rea/commit/c0904b4c1aabfccf2f0ab12f44960a0a1fbbbfae), [9f67e44](https://github.com/morluto/rea/commit/9f67e44e5d451d705bae9a579ad9802d5c9f097c), [e4532b5](https://github.com/morluto/rea/commit/e4532b537663026c3e5b784380eecd009c959a36), [1c37914](https://github.com/morluto/rea/commit/1c37914306b3ff257d7a0360f11c9bf2a3f2cac7), [5ab0cbb](https://github.com/morluto/rea/commit/5ab0cbb48ea5a4f11542eac44479b9af48f1dabd), [b7169c8](https://github.com/morluto/rea/commit/b7169c8b747cbc032913b31f008a3b81c77b6219), [f43fc02](https://github.com/morluto/rea/commit/f43fc0247527aebb8efc7d9db04276fe2d489323), [3ae826c](https://github.com/morluto/rea/commit/3ae826ceba6f25b08bce50bcb51001209bc16b57), [1b95302](https://github.com/morluto/rea/commit/1b953021eadf4cac09e20be388d79536903633b3), [eaa9616](https://github.com/morluto/rea/commit/eaa961658578522cecfbba7c7be1acf1c64da65a), [c923fb0](https://github.com/morluto/rea/commit/c923fb0dc9462e302b84a15770131ff899f89455), [c7a890d](https://github.com/morluto/rea/commit/c7a890d12987d9b2d496119c1fbf1a88c0fc298e)). Thanks @Aurantium-47, @baron, @chqcjq, @coygeek, @cyanseek, @fengdwx, @flitzrrr, @immanuwell, @kaoru0822-kitauji, @N0zoM1z0, @Nelglor, @ranmori, @rudycelekli, @shoyann, @xu91102. +- **Browser observations:** preserve effective source-map URLs, empty query/fragment identity, response-body provenance, service-worker policies and explicit uncertainty for malformed events. Bound discovery lifetimes and retain cleanup ownership. Interrupt/retry/failure cases validated with controlled CDP/session fixtures do not establish those paths in real Chromium. ([#1742](https://github.com/morluto/rea/pull/1742), [issue morluto/rea#1729](https://github.com/morluto/rea/issues/1729), [#1829](https://github.com/morluto/rea/pull/1829), [#2074](https://github.com/morluto/rea/pull/2074), [#2099](https://github.com/morluto/rea/pull/2099), [#2239](https://github.com/morluto/rea/pull/2239), [issue morluto/rea#2169](https://github.com/morluto/rea/issues/2169), [#2477](https://github.com/morluto/rea/pull/2477), [486b567](https://github.com/morluto/rea/commit/486b56724c4eaba50719177bcb9eae84d6907297), [a82e37c](https://github.com/morluto/rea/commit/a82e37c8065cc200a0ca3e14e7c4f85e4e468580), [a8d4984](https://github.com/morluto/rea/commit/a8d4984404b611e60b88bc42289c0ba07eee4578), [966f7cc](https://github.com/morluto/rea/commit/966f7cc84b4a8bb0043641cb64d7a7aaf8715e05), [221c856](https://github.com/morluto/rea/commit/221c85648108e9846d9920b0964dafa7f108d581), [951b095](https://github.com/morluto/rea/commit/951b095e11adf9faa9c4a27e212260060aa6b095), [f09bb8a](https://github.com/morluto/rea/commit/f09bb8a99773f79c7306b8799c86f856fe2c0d4b), [6bd2b28](https://github.com/morluto/rea/commit/6bd2b28161411f70ed5dc72e42c2736a1aab758b)). Thanks @1052326311, @chAng-L19, @chqcjq, @coygeek, @evenafterallisaid, @fisforfaheem, @itsF4LCON, @N0zoM1z0, @shayanxcyber. +- **Managed metadata/CIL:** preserve ExportedType and pointer-table ownership, generic/array/function-pointer/modifier signatures, ReadyToRun host identity, metadata ExtraData and signed zero. Charge retained facts and decoded instruction counts instead of broad decoding bounds; malformed layouts remain caller input errors. ([#1743](https://github.com/morluto/rea/pull/1743), [issue morluto/rea#1736](https://github.com/morluto/rea/issues/1736), [#1768](https://github.com/morluto/rea/pull/1768), [#1822](https://github.com/morluto/rea/pull/1822), [issue morluto/rea#1801](https://github.com/morluto/rea/issues/1801), [#1828](https://github.com/morluto/rea/pull/1828), [issue morluto/rea#1812](https://github.com/morluto/rea/issues/1812), [#1835](https://github.com/morluto/rea/pull/1835), [issue morluto/rea#1832](https://github.com/morluto/rea/issues/1832), [#1886](https://github.com/morluto/rea/pull/1886), [#1893](https://github.com/morluto/rea/pull/1893), [#1897](https://github.com/morluto/rea/pull/1897), [issue morluto/rea#1878](https://github.com/morluto/rea/issues/1878), [#1943](https://github.com/morluto/rea/pull/1943), [issue morluto/rea#1940](https://github.com/morluto/rea/issues/1940), [#1985](https://github.com/morluto/rea/pull/1985), [#2194](https://github.com/morluto/rea/pull/2194), [issue morluto/rea#2191](https://github.com/morluto/rea/issues/2191), [#2437](https://github.com/morluto/rea/pull/2437), [52e76c5](https://github.com/morluto/rea/commit/52e76c5d39b58d1685bb5e22bd40d19524dfa2cb)). Thanks @ac1982, @chqcjq, @fisHarly0, @flitzrrr, @JasonBuildAI, @JayOfTheKeyboard, @jhuang-tw, @kaoru0822-kitauji, @N0zoM1z0, @Nelglor. +- **Hopper failure ownership:** preserve startup/process outcomes, retain cleanup leases across one-shot failures, wait for late bridges, and stop owned Linux demo children when their REA owner exits. Denied/crashed/late-bridge/queued-request failure paths use simulated process or bridge fixtures; they are not fresh installed-Hopper verification. ([#1763](https://github.com/morluto/rea/pull/1763), [issue morluto/rea#1751](https://github.com/morluto/rea/issues/1751), [#1817](https://github.com/morluto/rea/pull/1817), [#1866](https://github.com/morluto/rea/pull/1866), [#1927](https://github.com/morluto/rea/pull/1927), [#2474](https://github.com/morluto/rea/pull/2474), [#2499](https://github.com/morluto/rea/pull/2499), [issue morluto/rea#2459](https://github.com/morluto/rea/issues/2459)). Thanks @coygeek, @Joshwong1908, @kaoru0822-kitauji, @N0zoM1z0, @rudycelekli. +- **IDA session boundaries:** retain non-x86 call coverage and unresolved-procedure uncertainty, and bound private universal-Mach-O inputs to the exact selected slice. Transient headless busy recovery was exercised on macOS arm64 IDA Pro 9.1; Linux headless, attached GUI navigation and worker-reaped cleanup paths retain their documented verification gaps. ([#1764](https://github.com/morluto/rea/pull/1764), [issue morluto/rea#1712](https://github.com/morluto/rea/issues/1712), [#1778](https://github.com/morluto/rea/pull/1778), [issue morluto/rea#1766](https://github.com/morluto/rea/issues/1766), [#1785](https://github.com/morluto/rea/pull/1785), [issue morluto/rea#1769](https://github.com/morluto/rea/issues/1769), [#1906](https://github.com/morluto/rea/pull/1906), [issue morluto/rea#1794](https://github.com/morluto/rea/issues/1794), [#2201](https://github.com/morluto/rea/pull/2201), [issue morluto/rea#2139](https://github.com/morluto/rea/issues/2139), [#2291](https://github.com/morluto/rea/pull/2291), [issue morluto/rea#2285](https://github.com/morluto/rea/issues/2285), [#2324](https://github.com/morluto/rea/pull/2324), [12fa8fd](https://github.com/morluto/rea/commit/12fa8fd9f986dbd4ff58165c3ec71a46dc8a8a3b)). Thanks @akushonkamen, @coygeek, @kaoru0822-kitauji, @N0zoM1z0, @nikolay-dementiev, @zecaro. +- **Input and capability diagnostics:** identify invalid paths, format refusals, missing host tools and non-retryable errors without losing caller selections or retryable capability facts. Foreign execution/device mutation annotations remain destructive, analysis-target text is untrusted, and output ordering follows code points rather than host locale. ([#1770](https://github.com/morluto/rea/pull/1770), [#1867](https://github.com/morluto/rea/pull/1867), [#1869](https://github.com/morluto/rea/pull/1869), [issue morluto/rea#1855](https://github.com/morluto/rea/issues/1855), [#1872](https://github.com/morluto/rea/pull/1872), [#1879](https://github.com/morluto/rea/pull/1879), [issue morluto/rea#1853](https://github.com/morluto/rea/issues/1853), [#1899](https://github.com/morluto/rea/pull/1899), [#1902](https://github.com/morluto/rea/pull/1902), [#1904](https://github.com/morluto/rea/pull/1904), [#1908](https://github.com/morluto/rea/pull/1908), [#1925](https://github.com/morluto/rea/pull/1925), [#1987](https://github.com/morluto/rea/pull/1987), [issue morluto/rea#1979](https://github.com/morluto/rea/issues/1979), [#2004](https://github.com/morluto/rea/pull/2004), [issue morluto/rea#2002](https://github.com/morluto/rea/issues/2002), [#2026](https://github.com/morluto/rea/pull/2026), [#2091](https://github.com/morluto/rea/pull/2091), [#2133](https://github.com/morluto/rea/pull/2133), [issue morluto/rea#2129](https://github.com/morluto/rea/issues/2129), [#2145](https://github.com/morluto/rea/pull/2145), [issue morluto/rea#2143](https://github.com/morluto/rea/issues/2143), [#2472](https://github.com/morluto/rea/pull/2472), [#2478](https://github.com/morluto/rea/pull/2478), [issue morluto/rea#2454](https://github.com/morluto/rea/issues/2454), [#2494](https://github.com/morluto/rea/pull/2494), [#2507](https://github.com/morluto/rea/pull/2507), [#2509](https://github.com/morluto/rea/pull/2509), [#2708](https://github.com/morluto/rea/pull/2708), [3d83135](https://github.com/morluto/rea/commit/3d8313598785359c424d7305f503cb69fc8639a6), [c9fed50](https://github.com/morluto/rea/commit/c9fed50691ae014890e91a72534773d2fb58ce44), [8b18171](https://github.com/morluto/rea/commit/8b181716aca2cb467be499cba226abfb5ae19141), [7a2cd1e](https://github.com/morluto/rea/commit/7a2cd1ea0a63e8c44166860b7ee9c7a1aa43492b)). Thanks @astralyx0129, @CassieFox23, @DeryFerd, @dollarsmoney, @fisforfaheem, @flitzrrr, @Ha1baraA11, @jimpablo, @kaoru0822-kitauji, @larinyo, @N0zoM1z0, @rudycelekli, @SII-YDD, @viniciusdsandrade, @XiaoBinGan. +- **Android analysis:** retain decoded values, duplicate-class inventories and executable static initializers, reject malformed JADX UTF-8, and keep inline search provenance. Apktool accepts `--no-strings` and explains host requirements. Windows batch-launcher selection also resolves PATH/PATHEXT before choosing its launch mode. Recorded Apktool coverage is Linux/OpenJDK, and changed JADX/batch failure paths do not establish native Windows behavior. ([#1777](https://github.com/morluto/rea/pull/1777), [issue morluto/rea#1774](https://github.com/morluto/rea/issues/1774), [#1876](https://github.com/morluto/rea/pull/1876), [issue morluto/rea#1846](https://github.com/morluto/rea/issues/1846), [#1890](https://github.com/morluto/rea/pull/1890), [issue morluto/rea#1850](https://github.com/morluto/rea/issues/1850), [#1946](https://github.com/morluto/rea/pull/1946), [issue morluto/rea#1945](https://github.com/morluto/rea/issues/1945), [#2707](https://github.com/morluto/rea/pull/2707), [issue morluto/rea#2672](https://github.com/morluto/rea/issues/2672), [0677468](https://github.com/morluto/rea/commit/0677468877465c8dfbe25c72f142a7b5bc8a3212), [fe8d0bd](https://github.com/morluto/rea/commit/fe8d0bdd36fe12bc0836468d1a702a34fbdc219b), [#2802](https://github.com/morluto/rea/pull/2802)). Thanks @akushonkamen, @coygeek, @HeadmasterEggy, @kaoru0822-kitauji, @N0zoM1z0, @psipher. +- **ADB acquisition:** preserve device paths and directory identity, prevent host destination collisions/overwrites, count UTF-8 logcat bytes and restricted permissions, and evaluate the complete Activity Manager start report. Launch attempts, status, activity and raw failure output remain observations; real-device timeout/mixed-report and Android 8 directory-read paths remain unverified beyond the documented fixtures. ([#1962](https://github.com/morluto/rea/pull/1962), [#2212](https://github.com/morluto/rea/pull/2212), [issue morluto/rea#2175](https://github.com/morluto/rea/issues/2175), [#2221](https://github.com/morluto/rea/pull/2221), [issue morluto/rea#2220](https://github.com/morluto/rea/issues/2220), [#2359](https://github.com/morluto/rea/pull/2359), [c42232f](https://github.com/morluto/rea/commit/c42232f566d55fe7191763649755148c768eef9f), [55b6148](https://github.com/morluto/rea/commit/55b61485e9f7568b69597eef8696ff6abfe53cfd), [3bdc873](https://github.com/morluto/rea/commit/3bdc8730bd98e073522bcb8ba86527be402c0cbe), [c89972c](https://github.com/morluto/rea/commit/c89972c8777d0114566fe76a060849a18bdd2659), [e1cd292](https://github.com/morluto/rea/commit/e1cd292aa6e8f5f13441d408c66126782c8dbb0d)). Thanks @coygeek, @dakjdakd, @DeryFerd, @huiq777, @sx4im. +- **WASM names and diagnostics:** preserve debug/custom-section names including NUL and leading BOM, accept unpadded WABT section rows, and avoid irrelevant tool-remediation advice on unrelated runtime-root failures. ([#1963](https://github.com/morluto/rea/pull/1963), [issue morluto/rea#1956](https://github.com/morluto/rea/issues/1956), [#1967](https://github.com/morluto/rea/pull/1967), [#1981](https://github.com/morluto/rea/pull/1981), [#2084](https://github.com/morluto/rea/pull/2084), [#2102](https://github.com/morluto/rea/pull/2102)). Thanks @1052326311, @bugstan, @N0zoM1z0, @sathariels, @STiFLeR7, @viniciusdsandrade. + +- **Source-map and path handling:** retain source-map entries when embedded-content counts differ or an optional `sourceMappingURL` is used, and keep `file:` URLs out of operator-relative path resolution. ([#2873](https://github.com/morluto/rea/pull/2873), [#2886](https://github.com/morluto/rea/pull/2886), [#2882](https://github.com/morluto/rea/pull/2882), [issue morluto/rea#2821](https://github.com/morluto/rea/issues/2821), [issue morluto/rea#2883](https://github.com/morluto/rea/issues/2883), [issue morluto/rea#2577](https://github.com/morluto/rea/issues/2577)). Thanks @JFSAS, @rileygramlich, @shoyann, @shubhamahuja9999, @coygeek. +- **Native metadata:** accept declared facts in native dispatch inventories and share native verification input diagnostics. ([#2906](https://github.com/morluto/rea/pull/2906), [#2901](https://github.com/morluto/rea/pull/2901), [issue morluto/rea#2564](https://github.com/morluto/rea/issues/2564), [issue morluto/rea#2621](https://github.com/morluto/rea/issues/2621)). Thanks @ShitK, @coygeek. + +### ⚠ BREAKING CHANGES + +- **Runtime attachment state:** `attached` may be `null` in JavaScript, browser and Electron discovery/inspection results. Treat null as unknown; do not coerce it to false. ([e55ef72](https://github.com/morluto/rea/commit/e55ef7213515ba204c3d88a4487b4ee6a2773b06)). +- **Source maps:** original-source and mapping rows preserve a null source name when unavailable. Handle null explicitly instead of expecting an invented unknown-source string; new index fields remain optional when reading older Evidence. ([ee50ed1](https://github.com/morluto/rea/commit/ee50ed121e38cc6e1a2ff0dc8b789f15f71c5402)). +- **Streamed filtered JSON:** `--filter-output` now emits JSON `null` instead of `{}` when the selected value is null; consumers must distinguish a present null from a missing selection. Use `--full-output` when an object envelope is required. ([#2333](https://github.com/morluto/rea/pull/2333)). Thanks @Jackmeson1. +- **Call graphs:** `get_call_graph.normalized_result` is now `{ graph, limitations }`; read traversal-depth groups under `graph` and retain relationship-specific caveats. Contract/fixture validation does not establish new Ghidra or IDA engine coverage. ([5dfb9d0](https://github.com/morluto/rea/commit/5dfb9d050c2cc44a61b183dae765bef4d22a53b7)). +- **MCP discovery:** `binary_session` no longer repeats `server_identity.catalog.tools`. Use the connected server tool list, grouped `tool_availability` and provider capabilities; see [MCP contracts](docs/mcp-contracts.md). Origin: [6395e8d](https://github.com/morluto/rea/commit/6395e8d3007d5ee453477460e7ee1c51cff76be0). ([#1988](https://github.com/morluto/rea/pull/1988)). Thanks @kaoru0822-kitauji, @N0zoM1z0. + +- **Provider admission:** MIPS targets with unresolved GNU ABI attributes are refused. Supply supported ABI declarations. Saved Hopper databases on macOS are unavailable because the launcher cannot run the analysis bridge when opening a database; select an executable target instead. Database support requires a script-capable database launch route. ([d4bd65c](https://github.com/morluto/rea/commit/d4bd65c53c0254d0195cc7ab8da163dcadfd3b9b)). +- **Segment extents:** `binary_overview` segment length may be null when Hopper coordinates cannot establish a finite extent in one address space. Preserve unknown rather than converting it to zero or a saturated value. ([36db19c](https://github.com/morluto/rea/commit/36db19c59)). +- **Invocation identity:** `server.command_path` may be null, and registration/setup statuses or doctor classifications may be unknown. Preserve these states and establish the actual invocation before treating a registration as aligned. ([187f113](https://github.com/morluto/rea/commit/187f113f7)). Thanks @hemv-857, @qorvr. +- **ADB installation:** the optional `replace` input is unchanged. Results now require `replace_requested` to record the requested `adb install -r` mode; `replaced` is nullable because prior device package state was not observed. Update result readers to accept null instead of treating reinstall intent as proof of replacement. ([864314a](https://github.com/morluto/rea/commit/864314ac857924a7e6e65bfb37e8dc720fcc21e7)). +- **Managed/native verification:** unresolved P/Invoke results may retain `matched_native` and candidate facts when module identity is unknown. Interpret `status`, `basis` and `limitations`; `module-identity-unknown` does not verify correspondence, and unresolved no longer implies empty observations. ([3f6b575](https://github.com/morluto/rea/commit/3f6b5755b3e699bd9373641acffe0f2c5d2ab094)). + +Full contribution record: [893 commits and 529 merged PRs](https://github.com/morluto/rea/blob/rea-agents-6.5.0/docs/releases/6.5.0.contributions.json). + + ## [6.4.0](https://github.com/morluto/rea/compare/rea-agents-6.3.0...rea-agents-6.4.0) (2026-10-11) ### Highlights diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e80d903a39..10cede32b3 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -24,6 +24,82 @@ costs change. When adding or changing an MCP tool, follow the [tool design guide](docs/tool-design.md) and preserve the canonical contracts and generated catalog. +## Reporting issues + +Use the issue form that fits the report and keep it concise. For bugs, include +the actual result, the shortest reproduction you ran, the tested version and +environment, and the smallest useful output, artifact, screenshot, or source +reference. Ground expected behavior in documentation, source, or a known-good +version when available. State observed impact and frequency; keep untested scope +and suspected causes explicit. + +Incomplete reports and intermittent failures are welcome. If a narrative answer +or supporting evidence is unavailable, write `NOT_ENOUGH_INFO` and explain the +gap. Clearly label source-only findings and proposed checks; do not claim runtime +reproduction or results you did not observe. A root-cause analysis or fix is not +required to report a problem. Documentation reports should identify the relevant +passage or missing guidance and explain the correction or confusion. + +Feature and design proposals can start with a problem and desired outcome. +Examples, prior art, and measurements help when available; ideas without them +are welcome. Maintainers can investigate reports and help resolve open questions. +Human ownership and readiness declarations apply to PRs as described below. + +## Pull request ownership and readiness + +AI-assisted contributions are welcome. Every PR must have a human owner who +takes responsibility for the complete change, including code, tests, +documentation, and claims produced with AI assistance. The PR author is the +owner unless another human is named in the PR and explicitly accepts ownership. +These requirements also apply to maintainer-authored PRs. + +Before requesting maintainer review, the human owner must: + +- Review the final diff and relevant existing code. Be able to explain the + problem, the chosen approach, how it fits REA, and meaningful edge cases. +- Provide evidence that the change works. Summarize what was checked, the + observed results, and meaningful gaps in the PR body. Use focused tests, + representative CLI/MCP output, CI artifacts, screenshots, or other checks + appropriate to the change. For fixes, show the original failure and why the + regression coverage catches it when practical. Documentation-only changes + can use source, link, or rendered-document checks. Label source-derived + conclusions and proposed checks accurately; never report an unrun check as + passing. Follow the [verification requirements](#development-feedback-and-pr-verification) + for required checks and real-provider coverage. +- Complete the **Author readiness** declaration in the PR template, explicitly + stating that the PR is ready for initial review. + +Keep the PR description concise and current. Explain the problem, observable +outcome, approach, and useful evidence; link long logs or put them in optional +details. Address actionable automated review findings and explain disagreements. +Human ownership and review remain required when CI or an automated reviewer +reports success. Readiness requests maintainer review; it does not grant scope +approval or permission to merge. + +Draft PRs may request early feedback with open questions and incomplete checks. +Leave the readiness declaration unchecked until the human owner has completed +it. The declaration records initial readiness; later pushes do not invalidate +it or require another human sign-off. Keep the PR description and affected +validation current as work changes. AI agents must leave human declarations to +the human owner and must not attest to a human review on their behalf. + +The `PR readiness` check reports pending for drafts and requires a named human +owner and all three template declarations for non-drafts. It checks that the +declaration is filled, not whether the code or evidence is correct. Existing PRs +can copy the **Author readiness** section from the current template. +Open PRs must have distinct head commits because GitHub shares commit statuses. + +Maintainers enable merge enforcement after the workflow is merged: run +`gh workflow run pr-readiness.yml -f pull_request_number=NUMBER` for existing +open PRs, then add `PR readiness` from GitHub Actions to the branch's required +status checks alongside its existing checks. Deploy the workflow to each +protected target branch before requiring it there. The workflow reads PR +metadata using trusted default-branch code and writes only a commit status. + +Maintainers may return a PR to the author or close it when the problem is +unsupported, validation is missing, or the owner cannot explain the change. +Contributors are responsible for completing this work before requesting review. + ## Contribution scope REA focuses on software structure, runtime behavior, and evidence through diff --git a/README.md b/README.md index 8129813d3d..0cf0c68de2 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,7 @@
**English** · [简体中文](docs/readme/README_zh.md) · [繁體中文](docs/readme/README_zh-TW.md) · [日本語](docs/readme/README_ja.md) · [한국어](docs/readme/README_ko.md) · [Türkçe](docs/readme/README_tr.md) · [Русский](docs/readme/README_ru.md) · [Tiếng Việt](docs/readme/README_vi.md) · [ไทย](docs/readme/README_th.md) · [Deutsch](docs/readme/README_de.md) · [Español](docs/readme/README_es.md) · [Français](docs/readme/README_fr.md) · [Українська](docs/readme/README_uk.md) · [Polski](docs/readme/README_pl.md) · [Português (Brasil)](docs/readme/README_pt-BR.md) · [العربية](docs/readme/README_ar.md) · [فارسی](docs/readme/README_fa.md) · [Bahasa Indonesia](docs/readme/README_id.md) · [Italiano](docs/readme/README_it.md) +· [Ελληνικά](docs/readme/README_el.md) # REA: Reverse Engineer Anything @@ -23,6 +24,16 @@ npx rea-agents setup +
+ +> [!IMPORTANT] +> Get REA from [morluto/rea](https://github.com/morluto/rea), +> [rea.tools](https://rea.tools/), or the [rea-agents npm package](https://www.npmjs.com/package/rea-agents). +> Follow the installation and release links from these official sources. A similar +> repository name or copied README does not establish that a download is official. + +
+
REA launching its analysis bridge inside Hopper while inspecting a native binary @@ -57,6 +68,11 @@ Setup registers REA with your agent and installs matching workflow instructions. ## Quick start +On Linux, Ghidra, JADX, firmware tool launches and static JavaScript/Electron +analysis require a procps-compatible `ps` on REA's `PATH` for process ownership +checks. Recovery steps are in `docs/installation.md` under +"Linux owned-process prerequisite". + ### Set up your agent With Node.js and npm installed, run: @@ -152,7 +168,7 @@ Additional tools and host support depend on the target: | Target | What REA returns | Requirements and guide | | ---------------------- | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------- | -| Native binaries | Pseudocode, assembly, strings, symbols, calls and references | Hopper, Ghidra or IDA; [native analysis](https://rea.tools/guides/native/) | +| Native binaries | Pseudocode, assembly, strings, symbols, calls and references | Hopper, Ghidra or IDA; [native analysis](docs/native-investigation.md) | | Offline ELF layout | Sections, segments, original symbols/relocations and static mitigation candidates | Caller-supplied pwntools on Linux x64; [binary diagnostics](docs/binary-diagnostics.md) | | Go binaries | Embedded compiler version, modules, replacements and build settings | Built-in static ELF/PE/thin Mach-O inspection; no Go installation required | | Rust binaries | Embedded cargo-auditable crate names, versions, sources and dependency edges | Built-in static ELF/PE/thin Mach-O inspection; [Rust binaries](docs/rust-binaries.md) | @@ -160,8 +176,8 @@ Additional tools and host support depend on the target: | EVM bytecode | Dispatch selectors, byte offsets, inferred arguments and mutability | Local raw/hex carrier on Linux x64; [offline EVM guide](docs/evm-bytecode.md) | | SQLite databases | Tables, columns, indexes, views, triggers and selected ordinary-table rows | Local database/WAL snapshot; Node.js 24.x >=24.15 or 26+ with native SQLite limits | | Recorded Linux crashes | Raw notes, every recorded thread's registers/signals and optional mapping candidates | Caller-supplied pwntools on Linux x64; optional GDB/pwndbg; [recorded crashes](docs/recorded-crashes.md) | -| JavaScript / Electron | Modules, imports, source maps, routes, IPC and native add-on relationships | Node.js and npm; [application analysis](https://rea.tools/guides/javascript/) | -| Websites | Page structure, scripts, network observations and requested screenshots | A Chrome-family browser; [browser analysis](https://rea.tools/guides/browser/) | +| JavaScript / Electron | Modules, imports, source maps, routes, IPC and native add-on relationships | Node.js and npm; Linux: procps-compatible `ps`; [application analysis](docs/javascript-application-workflows.md) | +| Websites | Page structure, scripts, network observations and requested screenshots | A Chrome-family browser; [browser analysis](docs/browser-observation.md) | | Saved network captures | Requests, responses, exposed payloads and source locations | HAR or Fiddler Everywhere SAZ; mitmdump on Linux for native mitmproxy; [capture guide](docs/web-network-captures.md) | | .NET assemblies | Metadata, CIL instructions, declared native dependencies and build comparisons | None (static only); [managed-code guide](docs/managed-code-analysis.md) | | Android APKs | Manifest declarations, classes, decompiled methods and references | Headless JADX and a full JDK on Linux/macOS/Windows x64; [Android guide](docs/android-analysis.md) | @@ -180,6 +196,11 @@ Static JavaScript and .NET inspection read the supplied files without running the application. Runtime capture runs or interacts with the selected target using your user permissions; each runtime guide describes its effects. +For offline native-binary inspection, REA also supports caller-supplied GNU +objdump and Rizin. See the [reverse-engineering tools guide](docs/reverse-engineering-tools.md) +for provider setup, CLI commands, and Evidence limitations. REA does not install +or update either provider. + Native formats and host support vary by provider. See @@ -323,7 +344,7 @@ Report vulnerabilities through [SECURITY.md](SECURITY.md). ## Star history -🎉 **180,000 GitHub stars — thank you!** +🎉 **250,000 GitHub stars — thank you!** Thanks to everyone using REA, reporting bugs, requesting features, testing builds, and contributing fixes. diff --git a/bridge/cutter/rea_bridge.py b/bridge/cutter/rea_bridge.py new file mode 100644 index 0000000000..473f1d5fe6 --- /dev/null +++ b/bridge/cutter/rea_bridge.py @@ -0,0 +1,816 @@ +"""REA's experimental local bridge plugin for Cutter. + +Install this file in Cutter's user plugin directory. The bridge is REA-owned +IPC, not an upstream Cutter transport. It binds to loopback and requires a +per-process random token stored in a user-only descriptor file. +""" + +from __future__ import annotations + +import json +import hashlib +import hmac +import os +import secrets +import socket +import stat +import tempfile +import threading +import time +import uuid +from pathlib import Path +from typing import Any + +if os.name == "nt": + import ctypes + from ctypes import wintypes + +import cutter + +try: + from PySide6.QtCore import QObject, Signal +except ImportError: + from PySide2.QtCore import QObject, Signal + + +MAX_FRAME_BYTES = 16 * 1024 * 1024 +SOCKET_TIMEOUT_SECONDS = 20 +AUTH_FRAME_TIMEOUT_SECONDS = 1 +MAX_DESCRIPTOR_BYTES = 64 * 1024 +MAX_COMPLETION_RECEIPTS = 64 + + +def _windows_api() -> tuple[Any, Any]: + kernel32 = ctypes.WinDLL("kernel32", use_last_error=True) + advapi32 = ctypes.WinDLL("advapi32", use_last_error=True) + kernel32.GetCurrentProcess.restype = wintypes.HANDLE + kernel32.LocalFree.argtypes = [wintypes.HLOCAL] + kernel32.LocalFree.restype = wintypes.HLOCAL + kernel32.CloseHandle.argtypes = [wintypes.HANDLE] + kernel32.CloseHandle.restype = wintypes.BOOL + kernel32.GetFileAttributesW.argtypes = [wintypes.LPCWSTR] + kernel32.GetFileAttributesW.restype = wintypes.DWORD + kernel32.CreateDirectoryW.argtypes = [wintypes.LPCWSTR, wintypes.LPVOID] + kernel32.CreateDirectoryW.restype = wintypes.BOOL + kernel32.CreateFileW.argtypes = [wintypes.LPCWSTR, wintypes.DWORD, wintypes.DWORD, + wintypes.LPVOID, wintypes.DWORD, wintypes.DWORD, wintypes.HANDLE] + kernel32.CreateFileW.restype = wintypes.HANDLE + kernel32.WriteFile.argtypes = [wintypes.HANDLE, wintypes.LPVOID, wintypes.DWORD, + ctypes.POINTER(wintypes.DWORD), wintypes.LPVOID] + kernel32.WriteFile.restype = wintypes.BOOL + kernel32.FlushFileBuffers.argtypes = [wintypes.HANDLE] + kernel32.FlushFileBuffers.restype = wintypes.BOOL + kernel32.MoveFileExW.argtypes = [wintypes.LPCWSTR, wintypes.LPCWSTR, wintypes.DWORD] + kernel32.MoveFileExW.restype = wintypes.BOOL + kernel32.DeleteFileW.argtypes = [wintypes.LPCWSTR] + kernel32.DeleteFileW.restype = wintypes.BOOL + advapi32.OpenProcessToken.argtypes = [wintypes.HANDLE, wintypes.DWORD, ctypes.POINTER(wintypes.HANDLE)] + advapi32.OpenProcessToken.restype = wintypes.BOOL + advapi32.GetTokenInformation.argtypes = [wintypes.HANDLE, ctypes.c_int, wintypes.LPVOID, + wintypes.DWORD, ctypes.POINTER(wintypes.DWORD)] + advapi32.GetTokenInformation.restype = wintypes.BOOL + advapi32.ConvertSidToStringSidW.argtypes = [wintypes.LPVOID, ctypes.POINTER(wintypes.LPWSTR)] + advapi32.ConvertSidToStringSidW.restype = wintypes.BOOL + advapi32.GetNamedSecurityInfoW.argtypes = [wintypes.LPWSTR, ctypes.c_int, wintypes.DWORD, + ctypes.POINTER(wintypes.LPVOID), ctypes.POINTER(wintypes.LPVOID), + ctypes.POINTER(wintypes.LPVOID), ctypes.POINTER(wintypes.LPVOID), + ctypes.POINTER(wintypes.LPVOID)] + advapi32.GetNamedSecurityInfoW.restype = wintypes.DWORD + advapi32.GetSecurityDescriptorControl.argtypes = [wintypes.LPVOID, ctypes.POINTER(wintypes.WORD), + ctypes.POINTER(wintypes.DWORD)] + advapi32.GetSecurityDescriptorControl.restype = wintypes.BOOL + advapi32.EqualSid.argtypes = [wintypes.LPVOID, wintypes.LPVOID] + advapi32.EqualSid.restype = wintypes.BOOL + advapi32.GetAclInformation.argtypes = [wintypes.LPVOID, wintypes.LPVOID, wintypes.DWORD, ctypes.c_int] + advapi32.GetAclInformation.restype = wintypes.BOOL + advapi32.GetAce.argtypes = [wintypes.LPVOID, wintypes.DWORD, ctypes.POINTER(wintypes.LPVOID)] + advapi32.GetAce.restype = wintypes.BOOL + advapi32.IsValidSid.argtypes = [wintypes.LPVOID] + advapi32.IsValidSid.restype = wintypes.BOOL + advapi32.GetLengthSid.argtypes = [wintypes.LPVOID] + advapi32.GetLengthSid.restype = wintypes.DWORD + advapi32.ConvertStringSecurityDescriptorToSecurityDescriptorW.argtypes = [ + wintypes.LPCWSTR, wintypes.DWORD, ctypes.POINTER(wintypes.LPVOID), ctypes.POINTER(wintypes.DWORD), + ] + advapi32.ConvertStringSecurityDescriptorToSecurityDescriptorW.restype = wintypes.BOOL + return kernel32, advapi32 + + +def _windows_user_sid_text() -> str: + kernel32, advapi32 = _windows_api() + token = wintypes.HANDLE() + if not advapi32.OpenProcessToken(kernel32.GetCurrentProcess(), 0x0008, ctypes.byref(token)): + raise ctypes.WinError(ctypes.get_last_error()) + try: + required = wintypes.DWORD() + advapi32.GetTokenInformation(token, 1, None, 0, ctypes.byref(required)) + if required.value == 0: + raise ctypes.WinError(ctypes.get_last_error()) + buffer = ctypes.create_string_buffer(required.value) + if not advapi32.GetTokenInformation(token, 1, buffer, required, ctypes.byref(required)): + raise ctypes.WinError(ctypes.get_last_error()) + + class SID_AND_ATTRIBUTES(ctypes.Structure): + _fields_ = [("Sid", wintypes.LPVOID), ("Attributes", wintypes.DWORD)] + + class TOKEN_USER(ctypes.Structure): + _fields_ = [("User", SID_AND_ATTRIBUTES)] + + sid = ctypes.cast(buffer, ctypes.POINTER(TOKEN_USER)).contents.User.Sid + text = wintypes.LPWSTR() + if not advapi32.ConvertSidToStringSidW(sid, ctypes.byref(text)): + raise ctypes.WinError(ctypes.get_last_error()) + try: + return text.value + finally: + kernel32.LocalFree(text) + finally: + kernel32.CloseHandle(token) + + +def _verify_windows_private_path(path: Path, directory: bool) -> None: + kernel32, advapi32 = _windows_api() + attributes = kernel32.GetFileAttributesW(str(path)) + if attributes == 0xFFFFFFFF: + raise ctypes.WinError(ctypes.get_last_error()) + if attributes & 0x400 or bool(attributes & 0x10) != directory: + raise RuntimeError("Cutter bridge path is a reparse point or has the wrong object type") + + class ACL_SIZE_INFORMATION(ctypes.Structure): + _fields_ = [("AceCount", wintypes.DWORD), ("AclBytesInUse", wintypes.DWORD), ("AclBytesFree", wintypes.DWORD)] + + class ACE_HEADER(ctypes.Structure): + _fields_ = [("AceType", wintypes.BYTE), ("AceFlags", wintypes.BYTE), ("AceSize", wintypes.WORD)] + + class SID_HEADER(ctypes.Structure): + _fields_ = [("Revision", wintypes.BYTE), ("SubAuthorityCount", wintypes.BYTE), + ("IdentifierAuthority", wintypes.BYTE * 6)] + + class ACCESS_ALLOWED_ACE(ctypes.Structure): + _fields_ = [("Header", ACE_HEADER), ("Mask", wintypes.DWORD), ("SidStart", wintypes.DWORD)] + + owner = wintypes.LPVOID() + dacl = wintypes.LPVOID() + descriptor = wintypes.LPVOID() + result = advapi32.GetNamedSecurityInfoW( + str(path), 1, 0x1 | 0x4, ctypes.byref(owner), None, ctypes.byref(dacl), None, ctypes.byref(descriptor), + ) + if result != 0: + raise ctypes.WinError(result) + try: + control = wintypes.WORD() + revision = wintypes.DWORD() + if not advapi32.GetSecurityDescriptorControl(descriptor, ctypes.byref(control), ctypes.byref(revision)): + raise ctypes.WinError(ctypes.get_last_error()) + if not control.value & 0x1000 or not dacl: + raise RuntimeError("Cutter bridge ACL must be present and protected from inheritance") + + token = wintypes.HANDLE() + if not advapi32.OpenProcessToken(kernel32.GetCurrentProcess(), 0x0008, ctypes.byref(token)): + raise ctypes.WinError(ctypes.get_last_error()) + try: + required = wintypes.DWORD() + advapi32.GetTokenInformation(token, 1, None, 0, ctypes.byref(required)) + token_user = ctypes.create_string_buffer(required.value) + if not advapi32.GetTokenInformation(token, 1, token_user, required, ctypes.byref(required)): + raise ctypes.WinError(ctypes.get_last_error()) + + class SID_AND_ATTRIBUTES(ctypes.Structure): + _fields_ = [("Sid", wintypes.LPVOID), ("Attributes", wintypes.DWORD)] + + class TOKEN_USER(ctypes.Structure): + _fields_ = [("User", SID_AND_ATTRIBUTES)] + + current_sid = ctypes.cast(token_user, ctypes.POINTER(TOKEN_USER)).contents.User.Sid + if not advapi32.EqualSid(owner, current_sid): + raise RuntimeError("Cutter bridge path owner does not match the current user") + finally: + kernel32.CloseHandle(token) + + info = ACL_SIZE_INFORMATION() + if not advapi32.GetAclInformation(dacl, ctypes.byref(info), ctypes.sizeof(info), 2) or info.AceCount != 2: + raise RuntimeError("Cutter bridge ACL must allow only the current user and SYSTEM") + expected_user = _windows_user_sid_text() + user_allowed = False + system_allowed = False + for index in range(info.AceCount): + ace_pointer = wintypes.LPVOID() + if not advapi32.GetAce(dacl, index, ctypes.byref(ace_pointer)): + raise ctypes.WinError(ctypes.get_last_error()) + header = ctypes.cast(ace_pointer, ctypes.POINTER(ACE_HEADER)).contents + if header.AceType != 0 or header.AceSize < ctypes.sizeof(ACE_HEADER) + ctypes.sizeof(wintypes.DWORD) + ctypes.sizeof(SID_HEADER): + raise RuntimeError("Cutter bridge ACL contains a malformed or unsupported ACE") + ace_address = ctypes.cast(ace_pointer, ctypes.c_void_p).value + if ace_address is None: + raise RuntimeError("Cutter bridge ACL returned a null ACE") + sid_pointer = ctypes.c_void_p(ace_address + ctypes.sizeof(ACE_HEADER) + ctypes.sizeof(wintypes.DWORD)) + sid_header = ctypes.cast(sid_pointer, ctypes.POINTER(SID_HEADER)).contents + sid_size = ctypes.sizeof(SID_HEADER) + sid_header.SubAuthorityCount * ctypes.sizeof(wintypes.DWORD) + if sid_size > header.AceSize - ctypes.sizeof(ACE_HEADER) - ctypes.sizeof(wintypes.DWORD) or not advapi32.IsValidSid(sid_pointer) or advapi32.GetLengthSid(sid_pointer) != sid_size: + raise RuntimeError("Cutter bridge ACL contains an invalid SID") + ace = ctypes.cast(ace_pointer, ctypes.POINTER(ACCESS_ALLOWED_ACE)).contents + sid_text = wintypes.LPWSTR() + if ace.Mask != 0x001F01FF or ace.Header.AceFlags != 0x03: + raise RuntimeError("Cutter bridge ACL contains an unsupported ACE") + if not advapi32.ConvertSidToStringSidW(sid_pointer, ctypes.byref(sid_text)): + raise ctypes.WinError(ctypes.get_last_error()) + try: + if sid_text.value == expected_user: + user_allowed = True + elif sid_text.value == "S-1-5-18": + system_allowed = True + else: + raise RuntimeError("Cutter bridge ACL allows an unrelated principal") + finally: + kernel32.LocalFree(sid_text) + if not user_allowed or not system_allowed: + raise RuntimeError("Cutter bridge ACL must allow only the current user and SYSTEM") + finally: + kernel32.LocalFree(descriptor) + + +def _windows_security_attributes() -> tuple[Any, Any]: + kernel32, advapi32 = _windows_api() + sid = _windows_user_sid_text() + sddl = f"O:{sid}G:{sid}D:P(A;OICI;FA;;;{sid})(A;OICI;FA;;;SY)" + descriptor = wintypes.LPVOID() + if not advapi32.ConvertStringSecurityDescriptorToSecurityDescriptorW(sddl, 1, ctypes.byref(descriptor), None): + raise ctypes.WinError(ctypes.get_last_error()) + + class SECURITY_ATTRIBUTES(ctypes.Structure): + _fields_ = [("nLength", wintypes.DWORD), ("lpSecurityDescriptor", wintypes.LPVOID), ("bInheritHandle", wintypes.BOOL)] + + attributes = SECURITY_ATTRIBUTES(ctypes.sizeof(SECURITY_ATTRIBUTES), descriptor, False) + return kernel32, (descriptor, attributes) + + +def _ensure_windows_bridge_directory(directory: Path) -> None: + directory.parent.mkdir(parents=True, exist_ok=True) + if not directory.exists(): + kernel32, (descriptor, attributes) = _windows_security_attributes() + try: + if not kernel32.CreateDirectoryW(str(directory), ctypes.byref(attributes)): + error = ctypes.get_last_error() + if error != 183: + raise ctypes.WinError(error) + finally: + kernel32.LocalFree(descriptor) + _verify_windows_private_path(directory, True) + + +def _ensure_posix_bridge_directory(directory: Path) -> None: + directory.mkdir(mode=0o700, parents=True, exist_ok=True) + if not hasattr(os, "O_NOFOLLOW") or not hasattr(os, "O_DIRECTORY"): + raise RuntimeError("Cutter bridge directory requires no-follow directory handles") + descriptor = os.open(directory, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW) + try: + directory_info = os.fstat(descriptor) + if directory_info.st_uid != os.getuid(): + raise RuntimeError("Cutter bridge directory must be owned by the current user") + os.fchmod(descriptor, 0o700) + directory_info = os.fstat(descriptor) + if stat.S_IMODE(directory_info.st_mode) != 0o700: + raise RuntimeError("Cutter bridge directory must have mode 0700") + finally: + os.close(descriptor) + + +def _write_windows_descriptor(directory: Path, destination: Path, payload: bytes) -> None: + if len(payload) > MAX_DESCRIPTOR_BYTES: + raise RuntimeError("Cutter bridge descriptor exceeds its byte limit") + kernel32, (security_descriptor, attributes) = _windows_security_attributes() + temporary = directory / f".{destination.name}.{uuid.uuid4().hex}.tmp" + handle = wintypes.HANDLE() + try: + handle = kernel32.CreateFileW(str(temporary), 0x40000000 | 0x00020000 | 0x00040000, + 0x1 | 0x2 | 0x4, ctypes.byref(attributes), 1, 0x80, None) + if handle == wintypes.HANDLE(-1).value: + raise ctypes.WinError(ctypes.get_last_error()) + _verify_windows_private_path(temporary, False) + if payload: + data = ctypes.create_string_buffer(payload) + written = wintypes.DWORD() + if not kernel32.WriteFile(handle, data, len(payload), ctypes.byref(written), None) or written.value != len(payload): + raise ctypes.WinError(ctypes.get_last_error()) + if not kernel32.FlushFileBuffers(handle): + raise ctypes.WinError(ctypes.get_last_error()) + kernel32.CloseHandle(handle) + handle = wintypes.HANDLE() + if not kernel32.MoveFileExW(str(temporary), str(destination), 0x1 | 0x8): + raise ctypes.WinError(ctypes.get_last_error()) + finally: + if handle and handle != wintypes.HANDLE(-1).value: + kernel32.CloseHandle(handle) + kernel32.DeleteFileW(str(temporary)) + kernel32.LocalFree(security_descriptor) + + +def _bridge_directory() -> Path: + configured = os.environ.get("REA_CUTTER_BRIDGE_DIR") + if configured: + return Path(configured).expanduser() + local_app_data = os.environ.get("LOCALAPPDATA") + if os.name == "nt" and local_app_data: + return Path(local_app_data) / "REA" / "CutterBridge" + return Path.home() / ".cache" / "rea" / "cutter-bridge" + + +class _RequestDispatcher(QObject): + dispatch = Signal(object, object) + + def __init__(self) -> None: + super().__init__() + self.dispatch.connect(self._handle) + self.generation = 0 + self.completed_commands: dict[str, str] = {} + self._receipt_lock = threading.Lock() + self.current_file = self._read_current_file() + try: + self.cutter_version = str(cutter.core().getVersionInformation()) + except Exception: + self.cutter_version = None + + @staticmethod + def _read_current_file() -> str | None: + try: + value = cutter.core().currentlyOpenFile() + return str(value) if value else None + except Exception: + return None + + def _handle(self, request: dict[str, Any], reply: dict[str, Any]) -> None: + kind = request.get("kind") + request_id = request.get("request_id") + if kind in ("command", "ack"): + try: + valid_request_id = ( + isinstance(request_id, str) + and str(uuid.UUID(request_id)) == request_id.lower() + ) + except (ValueError, AttributeError): + valid_request_id = False + if not valid_request_id: + reply.update({ + "ok": False, + "error": "invalid-command-request-id", + "execution_state": "not_started", + }) + request["_event"].set() + return + if kind == "ack": + with self._receipt_lock: + acknowledged = self.completed_commands.pop(request_id, None) is not None + reply.update({"ok": True, "acknowledged": acknowledged}) + request["_event"].set() + return + if request["_cancelled"].is_set(): + reply.update({ + "ok": False, + "error": "request-cancelled-before-execution", + "execution_state": "not_started", + }) + if kind == "command": + self._record_command_completion(request_id, "not_started") + request["_event"].set() + return + self._refresh_document_identity() + if kind == "status": + with self._receipt_lock: + completed_commands = [ + {"request_id": receipt_id, "execution_state": state} + for receipt_id, state in self.completed_commands.items() + ] + reply.update({ + "ok": True, + "session_id": request.get("session_id"), + "current_file": self.current_file, + "document_generation": self.generation, + "cutter_version": self.cutter_version, + "identity_status": "partial", + "completed_commands": completed_commands, + }) + request["_event"].set() + return + if kind != "command": + reply.update({"ok": False, "error": "unsupported-request-kind"}) + request["_event"].set() + return + if request.get("expected_generation") != self.generation: + reply.update({ + "ok": False, + "error": "active-document-generation-changed", + "execution_state": "not_started", + }) + self._record_command_completion(request_id, "not_started") + request["_event"].set() + return + command = request.get("command") + if not isinstance(command, str) or not command.strip(): + reply.update({ + "ok": False, + "error": "invalid-command", + "execution_state": "not_started", + }) + self._record_command_completion(request_id, "not_started") + request["_event"].set() + return + reply["execution_state"] = "running" + try: + output = cutter.cmdj(command) if request.get("json") is True else cutter.cmd(command) + command_error = None + except Exception as error: + output = None + command_error = str(error) + self._refresh_document_identity() + if command_error is None: + reply.update({ + "ok": True, + "output": output, + "current_file": self.current_file, + "document_generation": self.generation, + "cutter_version": self.cutter_version, + "identity_status": "partial", + "execution_state": "complete", + }) + self._record_command_completion(request_id, "complete") + else: + reply.update({ + "ok": False, + "error": command_error, + "current_file": self.current_file, + "document_generation": self.generation, + "cutter_version": self.cutter_version, + "identity_status": "partial", + "execution_state": "unknown", + "message": "The command raised an exception; it may have produced partial effects. Do not retry automatically.", + }) + self._record_command_completion(request_id, "failed") + request["_event"].set() + + def _record_command_completion( + self, request_id: str, execution_state: str + ) -> None: + with self._receipt_lock: + if request_id in self.completed_commands or len(self.completed_commands) < MAX_COMPLETION_RECEIPTS: + self.completed_commands[request_id] = execution_state + + def can_accept_command(self) -> bool: + with self._receipt_lock: + return len(self.completed_commands) < MAX_COMPLETION_RECEIPTS + + def _refresh_document_identity(self) -> None: + current_file = self._read_current_file() + if current_file != self.current_file: + self.current_file = current_file + self.generation += 1 + + +class _BridgeServer(threading.Thread): + def __init__(self, dispatcher: _RequestDispatcher) -> None: + super().__init__(name="rea-cutter-bridge", daemon=True) + self.dispatcher = dispatcher + self.server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + self.server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + self.server.bind(("127.0.0.1", 0)) + self.server.listen(8) + self.server.settimeout(0.5) + self.port = self.server.getsockname()[1] + self.session_id = str(uuid.uuid4()) + self.token = secrets.token_urlsafe(32) + self.poisoned = False + self.connection_ordinal = 0 + self.directory = _bridge_directory() + self.descriptor = self.directory / f"cutter-{os.getpid()}-{self.session_id}.json" + + def publish(self) -> None: + if os.name == "nt": + _ensure_windows_bridge_directory(self.directory) + else: + _ensure_posix_bridge_directory(self.directory) + if os.name == "nt" and (self.directory.is_symlink() or not self.directory.is_dir()): + raise RuntimeError("Cutter bridge directory must be a real directory") + payload = { + "session_id": self.session_id, + "pid": os.getpid(), + "host": "127.0.0.1", + "port": self.port, + "token": self.token, + "document_generation": self.dispatcher.generation, + "current_file": self.dispatcher.current_file, + "cutter_version": self.dispatcher.cutter_version, + "identity_status": "partial", + } + if os.name == "nt": + _write_windows_descriptor(self.directory, self.descriptor, json.dumps(payload).encode("utf-8")) + return + descriptor, temporary_path = tempfile.mkstemp(prefix=f".{self.descriptor.name}.", dir=self.directory) + try: + os.fchmod(descriptor, 0o600) + descriptor_info = os.fstat(descriptor) + if descriptor_info.st_uid != os.getuid() or stat.S_IMODE(descriptor_info.st_mode) & 0o077: + raise RuntimeError("Cutter bridge descriptor must be owned by the current user with mode 0600") + with os.fdopen(descriptor, "w", encoding="utf-8") as stream: + stream.write(json.dumps(payload)) + stream.flush() + os.fsync(stream.fileno()) + os.replace(temporary_path, self.descriptor) + except Exception: + try: + os.close(descriptor) + except OSError: + pass + try: + os.unlink(temporary_path) + except OSError: + pass + raise + + def run(self) -> None: + try: + self.publish() + except Exception: + self.server.close() + return + while True: + try: + connection, _ = self.server.accept() + except socket.timeout: + if getattr(self, "stopping", False): + break + continue + except OSError: + break + self.connection_ordinal += 1 + connection_ordinal = self.connection_ordinal + with connection: + connection.settimeout(AUTH_FRAME_TIMEOUT_SECONDS) + authenticated = False + try: + hello = self._receive_frame(connection) + if not isinstance(hello, dict) or hello.get("kind") != "rea-auth-v2" or not isinstance(hello.get("nonce"), str) or len(hello["nonce"]) != 64: + self._send(connection, {"ok": False, "error": "authentication-failed"}) + continue + client_nonce = hello["nonce"] + server_nonce = f"{connection_ordinal:016x}{secrets.token_hex(24)}" + server_proof = hmac.new(self.token.encode(), f"rea-cutter-server-v2:{self.session_id}:{client_nonce}:{server_nonce}".encode(), hashlib.sha256).hexdigest() + self._send(connection, {"session_id": self.session_id, "nonce": server_nonce, "proof": server_proof}) + authenticated_frame = self._receive_frame(connection) + request_bytes = authenticated_frame.get("request") if isinstance(authenticated_frame, dict) else None + expected_proof = hmac.new(self.token.encode(), f"rea-cutter-client-v2:{self.session_id}:{client_nonce}:{server_nonce}:{request_bytes}".encode(), hashlib.sha256).hexdigest() + if not isinstance(request_bytes, str) or not isinstance(authenticated_frame.get("proof"), str) or not hmac.compare_digest(authenticated_frame["proof"], expected_proof): + self._send(connection, {"ok": False, "error": "authentication-failed"}) + continue + request = json.loads(request_bytes) + if not isinstance(request, dict) or request.get("session_id") != self.session_id: + self._send(connection, {"ok": False, "error": "authentication-failed"}) + continue + authenticated = True + connection.settimeout(SOCKET_TIMEOUT_SECONDS) + if self.poisoned and request.get("kind") not in ("status", "ack"): + self._send_authenticated(connection, { + "ok": False, + "error": "session-poisoned-after-ui-timeout", + "execution_state": "not_started", + }, client_nonce, server_nonce) + continue + if ( + request.get("kind") == "command" + and not self.dispatcher.can_accept_command() + ): + self._send_authenticated(connection, { + "ok": False, + "error": "completion-receipt-capacity-exceeded", + "execution_state": "not_started", + }, client_nonce, server_nonce) + continue + request["session_id"] = self.session_id + event = threading.Event() + cancelled = threading.Event() + reply: dict[str, Any] = {} + request["_event"] = event + request["_cancelled"] = cancelled + self.dispatcher.dispatch.emit(request, reply) + if not event.wait(SOCKET_TIMEOUT_SECONDS): + cancelled.set() + self.poisoned = True + self._send_authenticated(connection, { + "ok": False, + "error": "ui-thread-request-timeout", + "execution_state": "unknown", + "message": "The UI operation may still complete after this timeout; restart Cutter before issuing further commands.", + }, client_nonce, server_nonce) + else: + if request.get("kind") == "status" and reply.get("ok") is True: + reply["completed_connection"] = f"{connection_ordinal - 1:016x}" + self._send_authenticated(connection, reply, client_nonce, server_nonce) + except socket.timeout: + if authenticated: + self._send_authenticated(connection, { + "ok": False, + "error": "request-timeout", + "execution_state": "unknown", + "message": "The command may have produced partial or persistent effects. Do not retry automatically.", + }, client_nonce, server_nonce) + except Exception as error: + self._send(connection, {"ok": False, "error": str(error)}) + self.server.close() + try: + self.descriptor.unlink(missing_ok=True) + except OSError: + pass + + @staticmethod + def _receive_frame(connection: socket.socket) -> dict[str, Any]: + data = bytearray() + deadline = time.monotonic() + AUTH_FRAME_TIMEOUT_SECONDS + while b"\n" not in data: + if len(data) >= MAX_FRAME_BYTES: + raise ValueError("frame-limit-exceeded") + remaining = deadline - time.monotonic() + if remaining <= 0: + raise socket.timeout("authentication frame timed out") + connection.settimeout(remaining) + part = connection.recv(min(65536, MAX_FRAME_BYTES + 1 - len(data))) + if not part: + raise ValueError("incomplete authentication frame") + data.extend(part) + line, remainder = bytes(data).split(b"\n", 1) + if len(line) > MAX_FRAME_BYTES or remainder: + raise ValueError("invalid framed request") + value = json.loads(line) + return value if isinstance(value, dict) else {} + + def _send_authenticated(self, connection: socket.socket, value: dict[str, Any], client_nonce: str, server_nonce: str) -> None: + encoded = self._encode_authenticated_response(value, client_nonce, server_nonce) + if encoded is None: + value = self._truncated_response(value) + encoded = self._encode_authenticated_response(value, client_nonce, server_nonce) + if encoded is None: + # This fixed-size fallback is checked through the same signed framing path. + value = { + "ok": False, + "error": "response-limit-exceeded", + "execution_state": value.get("execution_state", "unknown"), + "output": None, + "output_truncated": True, + "message": "The result exceeded the bridge frame limit; command completion state is preserved.", + } + encoded = self._encode_authenticated_response(value, client_nonce, server_nonce) + if encoded is None: + return + try: + connection.sendall(encoded) + except OSError: + pass + + def _encode_authenticated_response(self, value: dict[str, Any], client_nonce: str, server_nonce: str) -> bytes | None: + if not _within_response_limit(value): + return None + prefix = b'{"response":"' + suffix = b'","proof":"' + (b"0" * 64) + b'"}\n' + frame_size = len(prefix) + len(suffix) + response_parts: list[bytes] = [] + escaped_parts: list[bytes] = [] + for fragment in json.JSONEncoder(ensure_ascii=False, separators=(",", ":")).iterencode(value): + raw = fragment.encode("utf-8") + escaped = json.dumps(fragment, ensure_ascii=False)[1:-1].encode("utf-8") + frame_size += len(escaped) + if frame_size > MAX_FRAME_BYTES: + return None + response_parts.append(raw) + escaped_parts.append(escaped) + proof = hmac.new(self.token.encode(), digestmod=hashlib.sha256) + proof.update(f"rea-cutter-response-v2:{self.session_id}:{client_nonce}:{server_nonce}:".encode()) + for part in response_parts: + proof.update(part) + return prefix + b"".join(escaped_parts) + b'","proof":"' + proof.hexdigest().encode() + b'"}\n' + + @staticmethod + def _truncated_response(value: dict[str, Any]) -> dict[str, Any]: + response: dict[str, Any] = { + "ok": value.get("ok") is True, + "output": None, + "execution_state": value.get("execution_state", "unknown"), + "output_truncated": True, + "message": "The command result exceeded the bridge frame limit; completion state is preserved and output was omitted.", + } + for key in ("current_file", "document_generation", "cutter_version", "identity_status"): + if key in value: + response[key] = value[key] + if value.get("ok") is not True and isinstance(value.get("error"), str): + response["error"] = value["error"] + return response + + @staticmethod + def _send(connection: socket.socket, value: dict[str, Any]) -> None: + if not _within_response_limit(value): + value = { + "ok": False, + "error": "response-limit-exceeded", + "execution_state": value.get("execution_state", "unknown"), + "output_truncated": True, + "message": "The command completed or may have completed, but its result exceeded the bridge response limit. Do not retry automatically.", + } + parts: list[bytes] = [] + encoded_size = 0 + for part in json.JSONEncoder(ensure_ascii=False).iterencode(value): + encoded_part = part.encode("utf-8") + encoded_size += len(encoded_part) + if encoded_size > MAX_FRAME_BYTES: + fallback = { + "ok": False, + "error": "response-limit-exceeded", + "execution_state": value.get("execution_state", "unknown"), + "output_truncated": True, + "message": "The command completed or may have completed, but its result exceeded the bridge response limit. Do not retry automatically.", + } + parts = [json.dumps(fallback, separators=(",", ":")).encode("utf-8")] + break + parts.append(encoded_part) + encoded = b"".join(parts) + try: + connection.sendall(encoded + b"\n") + except OSError: + # A disconnected caller must not terminate the bridge thread or + # prevent later commands from being served. + pass + + def stop(self) -> None: + self.stopping = True + try: + self.server.close() + except OSError: + pass + + +class ReaCutterPlugin(cutter.CutterPlugin): + name = "REA Local Bridge" + description = "Local REA analysis bridge for the active Cutter session" + version = "1.0" + author = "REA" + + def __init__(self) -> None: + super().__init__() + self._dispatcher: _RequestDispatcher | None = None + self._bridge: _BridgeServer | None = None + + def setupPlugin(self) -> None: + return + + def setupInterface(self, main: Any) -> None: + self._dispatcher = _RequestDispatcher() + self._bridge = _BridgeServer(self._dispatcher) + self._bridge.start() + + def terminate(self) -> None: + if self._bridge is not None: + self._bridge.stop() + self._bridge.join(timeout=2) + self._bridge = None + self._dispatcher = None + + +def create_cutter_plugin() -> ReaCutterPlugin: + return ReaCutterPlugin() + + +def _within_response_limit(value: Any) -> bool: + pending = [value] + seen: set[int] = set() + size = 0 + while pending: + current = pending.pop() + if isinstance(current, str): + size += 2 + if current.isascii(): + size += len(current) + size += current.count('"') + current.count("\\") + size += sum(5 for character in current if ord(character) < 0x20) + else: + size += 4 * len(current) + size += current.count('"') + current.count("\\") + size += sum(5 for character in current if ord(character) < 0x20) + elif isinstance(current, dict): + identity = id(current) + if identity in seen: + return False + seen.add(identity) + size += 2 + 4 * len(current) + if size > MAX_FRAME_BYTES: + return False + pending.extend(current.keys()) + pending.extend(current.values()) + elif isinstance(current, (list, tuple)): + identity = id(current) + if identity in seen: + return False + seen.add(identity) + size += 2 + 2 * len(current) + if size > MAX_FRAME_BYTES: + return False + pending.extend(current) + else: + size += 16 + if size > MAX_FRAME_BYTES: + return False + return True diff --git a/bridge/ghidra/ReaGhidraBridge.java b/bridge/ghidra/ReaGhidraBridge.java index 284eb7b531..c9057fc6e1 100644 --- a/bridge/ghidra/ReaGhidraBridge.java +++ b/bridge/ghidra/ReaGhidraBridge.java @@ -5,9 +5,11 @@ import java.io.BufferedReader; import java.io.BufferedWriter; import java.io.ByteArrayOutputStream; import java.io.IOException; +import java.io.InputStream; import java.math.BigInteger; import java.net.InetAddress; import java.net.InetSocketAddress; +import java.net.SocketTimeoutException; import java.net.StandardProtocolFamily; import java.net.UnixDomainSocketAddress; import java.nio.channels.Channels; @@ -81,6 +83,7 @@ import ghidra.program.model.listing.Function; import ghidra.program.model.listing.FunctionIterator; import ghidra.program.model.listing.Instruction; import ghidra.program.model.listing.InstructionIterator; +import ghidra.program.model.listing.Listing; import ghidra.program.model.listing.Variable; import ghidra.program.model.mem.MemoryBlock; import ghidra.program.model.mem.MemoryBlockSourceInfo; @@ -297,7 +300,7 @@ public final class ReaGhidraBridge extends HeadlessScript { Files.move(pendingPath, endpointPath); } } - acceptClient(server, descriptor); + acceptAuthenticatedLoopbackClient(server, descriptor); } finally { Files.deleteIfExists(pendingPath); @@ -309,6 +312,50 @@ public final class ReaGhidraBridge extends HeadlessScript { return System.getProperty("os.name", "").startsWith("Windows"); } + private static String readLoopbackHandshake( + SocketChannel client, int maxBytes, long timeoutMillis) throws IOException { + long deadline = System.nanoTime() + timeoutMillis * 1_000_000L; + InputStream input = client.socket().getInputStream(); + ByteArrayOutputStream bytes = new ByteArrayOutputStream(); + while (true) { + long remaining = deadline - System.nanoTime(); + if (remaining <= 0) throw new SocketTimeoutException("Bridge authentication timed out"); + client.socket().setSoTimeout((int) Math.max(1, + Math.min(Integer.MAX_VALUE, (remaining + 999_999L) / 1_000_000L))); + int value = input.read(); + if (value == -1) return null; + if (value == '\n') return bytes.toString(StandardCharsets.UTF_8); + if (bytes.size() == maxBytes) throw new IOException("Bridge authentication frame is too large"); + bytes.write(value); + } + } + + private void acceptAuthenticatedLoopbackClient( + ServerSocketChannel server, SessionDescriptor descriptor) throws Exception { + while (true) { + try (SocketChannel client = server.accept()) { + Request first; + try { + String line = readLoopbackHandshake(client, 64 * 1024, 1_000); + if (line == null) continue; + first = parseRequest(line, descriptor.token); + } + catch (IOException | RuntimeException rejected) { + // A rejected unauthenticated peer cannot consume the bridge session. + continue; + } + client.socket().setSoTimeout(0); + try (BufferedReader reader = new BufferedReader( + Channels.newReader(client, StandardCharsets.UTF_8)); + BufferedWriter writer = new BufferedWriter( + Channels.newWriter(client, StandardCharsets.UTF_8))) { + serveClient(descriptor, reader, writer, first); + } + return; + } + } + } + private void acceptClient( ServerSocketChannel server, SessionDescriptor descriptor) throws Exception { @@ -319,26 +366,28 @@ public final class ReaGhidraBridge extends HeadlessScript { BufferedWriter writer = new BufferedWriter( Channels.newWriter(client, StandardCharsets.UTF_8) )) { - serveClient(descriptor, reader, writer); + serveClient(descriptor, reader, writer, null); } } private void serveClient( SessionDescriptor descriptor, BufferedReader reader, - BufferedWriter writer) throws Exception { + BufferedWriter writer, + Request first) throws Exception { while (true) { - String line = reader.readLine(); - if (line == null) { - return; - } - Request request; - try { - request = parseRequest(line, descriptor.token); - } - catch (RuntimeException exception) { - writeFailure(writer, 1, "invalid_request", "Bridge request is invalid"); - return; + Request request = first; + first = null; + if (request == null) { + String line = reader.readLine(); + if (line == null) return; + try { + request = parseRequest(line, descriptor.token); + } + catch (RuntimeException exception) { + writeFailure(writer, 1, "invalid_request", "Bridge request is invalid"); + return; + } } try { if (request.method.equals("shutdown")) { @@ -405,7 +454,9 @@ public final class ReaGhidraBridge extends HeadlessScript { case "xrefs" -> xrefs(request.params); case "analyze_function" -> { FunctionAnalysisObservation observation = analyzeFunctionObservation(request.params); - yield functionObservation(observation.dossier(), observation.limitations()); + JsonObject result = functionObservation(observation.dossier(), observation.limitations()); + result.add("relationship_coverage", observation.relationshipCoverage()); + yield result; } case "set_address_name", "set_addresses_names" -> { if (!descriptor.transport.equals("unix-socket")) @@ -883,7 +934,8 @@ public final class ReaGhidraBridge extends HeadlessScript { String value = decompiledText(decompile(function)); JsonObject result = functionObservation( value == null ? JsonNull.INSTANCE : GSON.toJsonTree(value), - noReturnCallLimitations( + functionBoundaryLimitations( + function, currentProgram.getListing().getInstructions(function.getBody(), true) ) ); @@ -975,7 +1027,7 @@ public final class ReaGhidraBridge extends HeadlessScript { InstructionScan scan = scanInstructions(function, Integer.MAX_VALUE); return functionObservation( GSON.toJsonTree(renderAssembly(scan.instructions)), - noReturnCallLimitations(scan.instructions) + functionBoundaryLimitations(function, scan.instructions) ); } @@ -994,7 +1046,7 @@ public final class ReaGhidraBridge extends HeadlessScript { limitations.add( "The fast path does not invoke the decompiler or scan whole-program names and strings." ); - List boundaryLimitations = noReturnCallLimitations(scan.instructions); + List boundaryLimitations = functionBoundaryLimitations(function, scan.instructions); for (String limitation : boundaryLimitations) limitations.add(limitation); JsonObject result = new JsonObject(); result.add("procedure", procedureIdentity(function)); @@ -1016,12 +1068,19 @@ public final class ReaGhidraBridge extends HeadlessScript { for (Function item : ordered) { result.add(canonicalAddress(item.getEntryPoint())); } - return functionObservation( - result, - noReturnCallLimitations( + if (callers) { + JsonObject observation = functionObservation(result, functionBoundaryLimitations(function, currentProgram.getListing().getInstructions(function.getBody(), true) - ) + )); + observation.add("relationship_coverage", callRelationshipCoverage(null, + unrepresentedIncomingCalls(function, observed))); + monitor.checkCancelled(); + return observation; + } + CallBoundaryObservation calls = callBoundaryObservation( + function, currentProgram.getListing().getInstructions(function.getBody(), true), observed, false, null ); + return callRelationshipObservation(result, calls); } private JsonObject procedureInfo(JsonObject params) throws Exception { @@ -1040,7 +1099,8 @@ public final class ReaGhidraBridge extends HeadlessScript { result.add("classification", functionClassification(function)); return functionObservation( result, - noReturnCallLimitations( + functionBoundaryLimitations( + function, currentProgram.getListing().getInstructions(function.getBody(), true) ) ); @@ -1071,20 +1131,22 @@ public final class ReaGhidraBridge extends HeadlessScript { result.addProperty("direction", direction); result.add("references", edges); result.addProperty("reference_kinds_available", true); + List limitations; JsonArray unresolvedCalls = new JsonArray(); - if (direction.equals("outgoing")) for (Instruction instruction : scanInstructions(function, Integer.MAX_VALUE).instructions) { - if (!instruction.getFlowType().isCall()) continue; - boolean resolved = false; - for (Reference reference : instruction.getReferencesFrom()) if (reference.getReferenceType().isCall()) resolved = true; - if (!resolved) { - JsonObject site = new JsonObject(); - site.addProperty("address", canonicalAddress(instruction.getAddress())); - site.addProperty("reason", "Instruction decoder identifies a call but ReferenceManager supplies no call destination"); - unresolvedCalls.add(site); - } + if (direction.equals("outgoing")) { + CallBoundaryObservation calls = callBoundaryObservation( + function, currentProgram.getListing().getInstructions(function.getBody(), true), null, true, references + ); + unresolvedCalls = calls.unresolvedCalls(); + limitations = calls.limitations(); + } else { + limitations = functionBoundaryLimitations( + function, + currentProgram.getListing().getInstructions(function.getBody(), true) + ); } result.add("unresolved_calls", unresolvedCalls); - return result; + return functionObservation(result, limitations); } private JsonObject inspectNativeDataType(JsonObject params) throws Exception { @@ -1285,11 +1347,7 @@ public final class ReaGhidraBridge extends HeadlessScript { result.add("references", references); if (includeIncoming) { TreeMap incoming = new TreeMap<>(); - ReferenceIterator incomingIterator = currentProgram.getReferenceManager().getReferencesTo(address); - while (incomingIterator.hasNext()) { - monitor.checkCancelled(); - addReference(incoming, incomingIterator.next()); - } + visitObservedReferencesTo(address, reference -> addReference(incoming, reference)); List orderedIncoming = new ArrayList<>(incoming.values()); orderedIncoming.sort(REFERENCE_ORDER); JsonArray incomingReferences = new JsonArray(); @@ -1341,16 +1399,8 @@ public final class ReaGhidraBridge extends HeadlessScript { requireKeys(params, Set.of("document", "address")); requireDocument(params); Address address = requireAddress(params, "address"); - ReferenceIterator references = currentProgram.getReferenceManager().getReferencesTo(address); Set
sources = new HashSet<>(); - while (references.hasNext()) { - monitor.checkCancelled(); - Reference reference = references.next(); - if (reference.isEntryPointReference()) { - continue; - } - sources.add(reference.getFromAddress()); - } + visitObservedReferencesTo(address, reference -> sources.add(reference.getFromAddress())); List
ordered = new ArrayList<>(sources); ordered.sort(Address::compareTo); JsonArray result = new JsonArray(); @@ -1490,19 +1540,28 @@ public final class ReaGhidraBridge extends HeadlessScript { InstructionScan scan = scanInstructions(function, Integer.MAX_VALUE); DecompileResults decompilation = decompile(function); String pseudocode = decompiledText(decompilation); + JsonObject availability = new JsonObject(); + availability.add("pseudocode", facetAvailability( + pseudocode != null, + function.isExternal() ? "Ghidra does not decompile external functions." + : function.getBody().isEmpty() ? "Ghidra has no function body to decompile." + : "Ghidra completed decompilation without returning pseudocode." + )); if (pseudocode == null) { pseudocode = ""; } - List incomingReferences = collectReferences(function, "incoming") - .stream() + List bodyIncoming = collectReferences(function, "incoming"); + List incomingReferences = bodyIncoming.stream() .filter(reference -> !function.getBody().contains(reference.getFromAddress())) .toList(); List outgoingReferences = collectReferences(function, "outgoing"); JsonArray incoming = referenceEdges(incomingReferences); JsonArray outgoing = referenceEdges(outgoingReferences); - JsonArray comments = comments(scan.instructions); - JsonArray callers = procedureIdentities(function.getCallingFunctions(monitor)); - JsonArray callees = procedureIdentities(function.getCalledFunctions(monitor)); + JsonArray comments = comments(function); + Set callerFunctions = function.getCallingFunctions(monitor); + JsonArray callers = procedureIdentities(callerFunctions); + Set calleeFunctions = function.getCalledFunctions(monitor); + JsonArray callees = procedureIdentities(calleeFunctions); JsonArray referencedStrings = referencedStrings(outgoingReferences); JsonArray referencedNames = referencedNames(outgoingReferences); JsonArray blocks = basicBlockValues(function); @@ -1536,6 +1595,15 @@ public final class ReaGhidraBridge extends HeadlessScript { nativeApiBoundary(function, decompilation) ); result.add("native_value_flow", nativeValueFlow(decompilation)); + // Successful full queries distinguish collected empty sets from missing facts. + // Availability does not claim that Ghidra resolved every runtime relationship. + for (String facet : List.of("comments", "callers", "callees", "references", "strings_names", "cfg")) { + availability.add(facet, facetAvailability(true, null)); + } + availability.add("assembly", facetAvailability( + !scan.truncated(), "The function instruction scan was truncated." + )); + result.add("facet_availability", availability); JsonArray limitations = new JsonArray(); limitations.add( "Unresolved computed or indirect flows without target addresses are not represented as reference edges." @@ -1551,7 +1619,7 @@ public final class ReaGhidraBridge extends HeadlessScript { ); if (swift && aarch64) { limitations.add( - "Swift String literals are decoded from constant register pairs at calls, returns and paired stores; literals stored one word at a time, chosen by conditional selects, passed through unresolved jumps, or built at runtime are not reported." + "Swift String literals are decoded from constant register pairs at calls, returns and paired stores. Literals stored one word at a time, chosen by conditional selects, or built at runtime are not reported; candidate pairs at unresolved jumps are retained as unknown omissions." ); if (swiftLiterals.omissions().size() > 0) { limitations.add( @@ -1563,8 +1631,10 @@ public final class ReaGhidraBridge extends HeadlessScript { "Swift String literal decoding is implemented for AArch64 register pairs only; no Swift literals were decoded for this architecture." ); } - List boundaryLimitations = noReturnCallLimitations(scan.instructions); - for (String limitation : boundaryLimitations) { + CallBoundaryObservation calls = callBoundaryObservation( + function, scan.instructions, calleeFunctions, true, outgoingReferences); + result.add("unresolved_calls", calls.unresolvedCalls()); + for (String limitation : calls.limitations()) { limitations.add(limitation); } if (decompilation != null && decompilation.getErrorMessage() != null && @@ -1572,7 +1642,14 @@ public final class ReaGhidraBridge extends HeadlessScript { limitations.add("Ghidra decompiler diagnostic: " + decompilation.getErrorMessage()); } result.add("limitations", limitations); - return new FunctionAnalysisObservation(result, boundaryLimitations); + monitor.checkCancelled(); + int missingCallers = 0; + for (Reference reference : bodyIncoming) { + monitor.checkCancelled(); + if (unrepresentedCaller(reference, callerFunctions)) missingCallers++; + } + return new FunctionAnalysisObservation(result, calls.limitations(), + callRelationshipCoverage(calls, missingCallers)); } private JsonObject nativeValueFlow(DecompileResults decompilation) throws Exception { @@ -3276,52 +3353,278 @@ public final class ReaGhidraBridge extends HeadlessScript { return result; } - private List noReturnCallLimitations(Iterable instructions) + private static JsonObject facetAvailability(boolean available, String reason) { + JsonObject result = new JsonObject(); + result.addProperty("available", available); + if (!available) result.addProperty("reason", reason); + return result; + } + + private JsonObject callRelationshipObservation(JsonElement value, CallBoundaryObservation calls) throws Exception { - List limitations = new ArrayList<>(); - for (Instruction instruction : instructions) { + monitor.checkCancelled(); + JsonObject result = functionObservation(value, calls.limitations()); + result.add("relationship_coverage", callRelationshipCoverage(calls, null)); + return result; + } + + private static JsonObject callRelationshipCoverage( + CallBoundaryObservation calls, Integer missingCallers) { + JsonObject coverage = completeResolvedCallCoverage(); + Integer omitted = calls == null ? null : calls.unrepresentedCalleeCalls(); + JsonArray directions = new JsonArray(); + if (missingCallers != null && missingCallers > 0) directions.add("callers"); + String callerLimitation = missingCallers != null && missingCallers > 0 + ? "Ghidra's caller inventory omits sources of " + missingCallers + + " call reference(s) into this function's body." : null; + if ((calls != null && omitted == null) || (calls == null && missingCallers == null)) { + coverage.addProperty("status", "unknown"); + String reason = calls == null ? "No call inventory was examined." + : "The callee inventory was not examined alongside decoded call targets."; + coverage.addProperty("reason", reason + (callerLimitation == null ? "" : " " + callerLimitation)); + if (calls == null && missingCallers == null) directions.add("callers"); + directions.add("callees"); + coverage.add("limited_directions", directions); + return coverage; + } + JsonArray limitations = new JsonArray(); + if (callerLimitation != null) limitations.add(callerLimitation); + if (omitted != null && omitted > 0) { + limitations.add("Ghidra's callee inventory omits targets of " + omitted + + " call reference(s) or decoded call(s) without destinations."); + directions.add("callees"); + } + if (!limitations.isEmpty()) { + coverage.addProperty("status", "partial"); + coverage.add("limitations", limitations); + coverage.add("limited_directions", directions); + } + return coverage; + } + + private static JsonObject completeResolvedCallCoverage() { + JsonObject coverage = new JsonObject(); + coverage.addProperty("status", "complete"); + JsonArray diagnostics = new JsonArray(); + diagnostics.add( + "Only resolved Ghidra call references are returned; unresolved computed or indirect calls remain unknown, while function classifications distinguish thunks and externals." + ); + coverage.add("diagnostics", diagnostics); + return coverage; + } + + private boolean unrepresentedCaller(Reference reference, Set callers) { + if (!reference.getReferenceType().isCall()) return false; + Function caller = containingFunction(reference.getFromAddress()); + return caller == null || !callers.contains(caller); + } + + private int unrepresentedIncomingCalls(Function function, Set callers) + throws Exception { + int[] omitted = { 0 }; + visitReferences(function, "incoming", reference -> { + if (unrepresentedCaller(reference, callers)) omitted[0]++; + }); + return omitted[0]; + } + + private List functionBoundaryLimitations( + Function function, Iterable instructions) + throws Exception { + FunctionBoundaryLimitations boundary = new FunctionBoundaryLimitations(function); + for (Instruction instruction : instructions) boundary.observe(instruction); + return boundary.values(); + } + + private CallBoundaryObservation callBoundaryObservation( + Function function, Iterable instructions, Set returnedCallees, + boolean includeUnresolvedSites, List observedReferences) + throws Exception { + FunctionBoundaryLimitations boundary = new FunctionBoundaryLimitations(function); + CallBoundaryReferences calls = new CallBoundaryReferences(returnedCallees, includeUnresolvedSites); + if (observedReferences == null) visitReferences(function, "outgoing", calls::observe); + else for (Reference reference : observedReferences) { monitor.checkCancelled(); - if (!instruction.getFlowType().isCall() || !instruction.getFlowType().isTerminal()) { - continue; + calls.observe(reference); + } + for (Instruction instruction : instructions) { + boundary.observe(instruction); + if (!instruction.getFlowType().isCall()) continue; + boolean destination = false; + for (Reference reference : instruction.getReferencesFrom()) { + if (!reference.getReferenceType().isCall()) continue; + destination = true; + break; } - Address[] destinations = instruction.getFlows(); - StringBuilder limitation = new StringBuilder(); - limitation.append("Ghidra Listing reports a terminal call at ") - .append(canonicalAddress(instruction.getAddress())); - if (destinations.length == 0) { - limitation.append(" with no resolved callee; its no-return basis is unknown."); + if (!destination) { + calls.unrepresented++; + calls.unresolved(instruction.getAddress(), + "Instruction decoder identifies a call but ReferenceManager supplies no call destination"); } - else { - limitation.append(" to "); - for (int index = 0; index < destinations.length; index++) { - if (index > 0) limitation.append(", "); - Address destination = destinations[index]; - limitation.append(canonicalAddress(destination)); - Function target = currentProgram.getFunctionManager().getFunctionAt(destination); - if (target == null) { - limitation.append(" (callee identity unknown)"); - } - else { - limitation.append(" ").append(procedureName(target)); - limitation.append(" (external=").append(target.isExternal()); - limitation.append(", thunk=").append(target.isThunk()); - limitation.append(", hasNoReturn=").append(target.hasNoReturn()).append(")"); - } - } - limitation.append("; fallthrough is excluded by Ghidra's flow model."); - Function target = destinations.length == 1 - ? currentProgram.getFunctionManager().getFunctionAt(destinations[0]) - : null; - if (target != null && target.hasNoReturn()) { - limitation.append(" This records Ghidra's FunctionManager flag, not an independent verification that the callee cannot return."); + } + return new CallBoundaryObservation(boundary.values(), + returnedCallees == null ? null : calls.unrepresented, calls.unresolvedCalls()); + } + + private final class FunctionBoundaryLimitations { + private final Function function; + private final List limitations = new ArrayList<>(); + private Instruction beforePrevious; + private Instruction previous; + + private FunctionBoundaryLimitations(Function function) { + this.function = function; + } + + private void observe(Instruction instruction) throws Exception { + monitor.checkCancelled(); + String limitation = terminalCallLimitation(instruction); + if (limitation != null) limitations.add(limitation); + if (beforePrevious != null && previous != null) { + String seh = borlandSehHandlerLimitation( + function, beforePrevious, previous, instruction); + if (seh != null) limitations.add(seh); + } + beforePrevious = previous; + previous = instruction; + } + + private List values() { + return List.copyOf(limitations); + } + } + + private String borlandSehHandlerLimitation( + Function function, Instruction handlerPush, Instruction framePush, + Instruction frameLink) { + if (!immediatelyPrecedes(handlerPush, framePush) || + !immediatelyPrecedes(framePush, frameLink) || + !handlerPush.getMnemonicString().equalsIgnoreCase("PUSH") || + handlerPush.getNumOperands() != 1 || + !framePush.getMnemonicString().equalsIgnoreCase("PUSH") || + framePush.getNumOperands() != 1 || + !frameLink.getMnemonicString().equalsIgnoreCase("MOV") || + frameLink.getNumOperands() != 2 || + !sameFsMemoryOperand(framePush, frameLink) || + !hasRegisterOperand(frameLink, 1, "ESP")) + return null; + Object[] handlerOperand = handlerPush.getOpObjects(0); + if (handlerOperand.length != 1 || !(handlerOperand[0] instanceof Scalar scalar) || + scalar.bitLength() != 32) + return null; + Address handler = currentProgram.getAddressFactory().getDefaultAddressSpace() + .getAddress(scalar.getUnsignedValue()); + if (function.getBody().contains(handler)) return null; + return "Ghidra Listing shows a Borland/Delphi-style SEH frame registration at " + + canonicalAddress(frameLink.getAddress()) + " (push " + canonicalAddress(handler) + + " followed by fs-segment stack chaining); handler " + canonicalAddress(handler) + + " lies outside this procedure's body, so callees, strings, and cross-references may omit the handler region."; + } + + private static boolean immediatelyPrecedes(Instruction first, Instruction second) { + Address fallThrough = first.getFallThrough(); + return fallThrough != null && fallThrough.equals(second.getAddress()); + } + + private static boolean sameFsMemoryOperand(Instruction first, Instruction second) { + String firstOperand = normalizedFsMemoryOperand(first); + String secondOperand = normalizedFsMemoryOperand(second); + return firstOperand != null && firstOperand.equalsIgnoreCase(secondOperand); + } + + private static String normalizedFsMemoryOperand(Instruction instruction) { + String representation = instruction.getDefaultOperandRepresentation(0); + representation = representation.replaceAll("(?i)\\b(?:byte|word|dword|qword)\\s+ptr\\s+", "") + .replaceAll("\\s+", ""); + return representation.regionMatches(true, 0, "fs:", 0, 3) && + representation.indexOf('[') >= 0 ? representation : null; + } + + private static boolean hasRegisterOperand( + Instruction instruction, int operand, String name) { + for (Object object : instruction.getOpObjects(operand)) { + if (object instanceof Register register && register.getName().equalsIgnoreCase(name)) + return true; + } + return false; + } + + private final class CallBoundaryReferences { + private final Set returnedCallees; + private final TreeMap unresolved; + private int unrepresented; + + CallBoundaryReferences(Set returnedCallees, boolean includeUnresolvedSites) { + this.returnedCallees = returnedCallees; + unresolved = includeUnresolvedSites ? new TreeMap<>() : null; + } + + void observe(Reference reference) { + if (!reference.getReferenceType().isCall()) return; + // Reference identity follows the same containing-function owner as referenceEdge. + Function target = containingFunction(reference.getToAddress()); + if (returnedCallees != null && (target == null || !returnedCallees.contains(target))) + unrepresented++; + if (target == null) unresolved(reference.getFromAddress(), + "A call destination has no known containing Ghidra procedure"); + } + + void unresolved(Address address, String reason) { + if (unresolved != null) unresolved.put(address, reason); + } + + JsonArray unresolvedCalls() { + if (unresolved == null) return null; + JsonArray result = new JsonArray(); + for (var item : unresolved.entrySet()) { + JsonObject site = new JsonObject(); + site.addProperty("address", canonicalAddress(item.getKey())); + site.addProperty("reason", item.getValue()); + result.add(site); + } + return result; + } + } + + private String terminalCallLimitation(Instruction instruction) { + if (!instruction.getFlowType().isCall() || !instruction.getFlowType().isTerminal()) + return null; + Address[] destinations = instruction.getFlows(); + StringBuilder limitation = new StringBuilder(); + limitation.append("Ghidra Listing reports a terminal call at ") + .append(canonicalAddress(instruction.getAddress())); + if (destinations.length == 0) { + limitation.append(" with no resolved callee; its no-return basis is unknown."); + } + else { + limitation.append(" to "); + for (int index = 0; index < destinations.length; index++) { + if (index > 0) limitation.append(", "); + Address destination = destinations[index]; + limitation.append(canonicalAddress(destination)); + Function target = currentProgram.getFunctionManager().getFunctionAt(destination); + if (target == null) { + limitation.append(" (callee identity unknown)"); } else { - limitation.append(" The reason for the terminal flow is not established by this observation."); + limitation.append(" ").append(procedureName(target)); + limitation.append(" (external=").append(target.isExternal()); + limitation.append(", thunk=").append(target.isThunk()); + limitation.append(", hasNoReturn=").append(target.hasNoReturn()).append(")"); } } - limitations.add(limitation.toString()); + limitation.append("; fallthrough is excluded by Ghidra's flow model."); + Function target = destinations.length == 1 + ? currentProgram.getFunctionManager().getFunctionAt(destinations[0]) + : null; + if (target != null && target.hasNoReturn()) { + limitation.append(" This records Ghidra's FunctionManager flag, not an independent verification that the callee cannot return."); + } + else { + limitation.append(" The reason for the terminal flow is not established by this observation."); + } } - return limitations; + return limitation.toString(); } private static String renderAssembly(List instructions) { @@ -3347,43 +3650,62 @@ public final class ReaGhidraBridge extends HeadlessScript { private List collectReferences( Function function, String direction) throws Exception { - // References can target instruction interiors or originate in embedded - // data. Use actual body ownership, including an external's exact entry. - AddressSet scope = new AddressSet(function.getBody()); - scope.add(function.getEntryPoint()); - boolean outgoing = direction.equals("outgoing"); - AddressIterator addresses = outgoing - ? currentProgram.getReferenceManager().getReferenceSourceIterator(scope, true) - : currentProgram.getReferenceManager().getReferenceDestinationIterator(scope, true); TreeMap observed = new TreeMap<>(); - while (addresses.hasNext()) { - monitor.checkCancelled(); - Address address = addresses.next(); - if (outgoing) { - for (Reference reference : currentProgram.getReferenceManager() - .getReferencesFrom(address)) { - monitor.checkCancelled(); - addReference(observed, reference); - } - } - else { - ReferenceIterator iterator = currentProgram.getReferenceManager() - .getReferencesTo(address); - while (iterator.hasNext()) { - monitor.checkCancelled(); - addReference(observed, iterator.next()); - } - } - } + visitReferences(function, direction, reference -> addReference(observed, reference)); List result = new ArrayList<>(observed.values()); result.sort(REFERENCE_ORDER); return result; } - private static void addReference(TreeMap observed, Reference reference) { - if (reference.isEntryPointReference()) { - return; + @FunctionalInterface + private interface ReferenceVisitor { + void accept(Reference reference) throws Exception; + } + + private void visitReferences(Function function, String direction, ReferenceVisitor visitor) + throws Exception { + // References can target instruction interiors or originate in embedded + // data. Use actual body ownership, including an external's exact entry. + AddressSet scope = functionObservationScope(function); + boolean outgoing = direction.equals("outgoing"); + AddressIterator addresses = outgoing + ? currentProgram.getReferenceManager().getReferenceSourceIterator(scope, true) + : currentProgram.getReferenceManager().getReferenceDestinationIterator(scope, true); + while (addresses.hasNext()) { + monitor.checkCancelled(); + Address address = addresses.next(); + if (outgoing) { + visitObservedReferencesFrom(address, visitor); + } + else { + visitObservedReferencesTo(address, visitor); + } } + } + + private void visitObservedReferencesFrom(Address address, ReferenceVisitor visitor) + throws Exception { + for (Reference reference : currentProgram.getReferenceManager().getReferencesFrom(address)) { + monitor.checkCancelled(); + visitObservedReference(reference, visitor); + } + } + + private void visitObservedReferencesTo(Address address, ReferenceVisitor visitor) + throws Exception { + ReferenceIterator references = currentProgram.getReferenceManager().getReferencesTo(address); + while (references.hasNext()) { + monitor.checkCancelled(); + visitObservedReference(references.next(), visitor); + } + } + + private static void visitObservedReference(Reference reference, ReferenceVisitor visitor) + throws Exception { + if (!reference.isEntryPointReference()) visitor.accept(reference); + } + + private static void addReference(TreeMap observed, Reference reference) { String key = canonicalAddress(reference.getFromAddress()) + "\u0000" + canonicalAddress(reference.getToAddress()) + "\u0000" + reference.getReferenceType().getName() + "\u0000" + @@ -3392,6 +3714,12 @@ public final class ReaGhidraBridge extends HeadlessScript { observed.putIfAbsent(key, reference); } + private static AddressSet functionObservationScope(Function function) { + AddressSet scope = new AddressSet(function.getBody()); + scope.add(function.getEntryPoint()); + return scope; + } + private JsonArray referenceEdges(List references) { JsonArray result = new JsonArray(); for (Reference reference : references) { @@ -3545,23 +3873,28 @@ public final class ReaGhidraBridge extends HeadlessScript { return result; } - private static JsonArray comments(List instructions) { + private JsonArray comments(Function function) throws Exception { + Listing listing = currentProgram.getListing(); + AddressSet scope = functionObservationScope(function); + AddressIterator addresses = listing.getCommentAddressIterator(scope, true); TreeMap observed = new TreeMap<>(); - for (Instruction instruction : instructions) { + while (addresses.hasNext()) { + monitor.checkCancelled(); + Address address = addresses.next(); for (CommentType type : CommentType.values()) { - String text = instruction.getComment(type); + String text = listing.getComment(type, address); if (text == null || text.isEmpty()) { continue; } String kind = type == CommentType.EOL ? "inline" : "comment"; JsonObject item = new JsonObject(); - item.addProperty("address", canonicalAddress(instruction.getAddress())); + item.addProperty("address", canonicalAddress(address)); item.addProperty("kind", kind); item.addProperty("text", text); - String key = canonicalAddress(instruction.getAddress()) + "\u0000" + + String key = canonicalAddress(address) + "\u0000" + kind + "\u0000" + type.name() + "\u0000" + text; observed.put(key, item); - } + } } JsonArray result = new JsonArray(); observed.values().forEach(result::add); @@ -3635,8 +3968,11 @@ public final class ReaGhidraBridge extends HeadlessScript { return swiftStringLiterals(function, new SwiftLiteralBudget()); } - // The budget is owned by one decode run; an explicit budget also permits deterministic producer regressions. - private SwiftStringLiteralDecode swiftStringLiterals(Function function, SwiftLiteralBudget budget) throws Exception { + // An explicit budget makes capacity omissions deterministic in producer tests. + private SwiftStringLiteralDecode swiftStringLiterals( + Function function, + SwiftLiteralBudget budget + ) throws Exception { Register[] registers = new Register[31]; for (int index = 0; index < registers.length; index++) { registers[index] = currentProgram.getRegister("x" + index); @@ -3650,6 +3986,9 @@ public final class ReaGhidraBridge extends HeadlessScript { public boolean evaluateContextBefore(VarnodeContext context, Instruction instruction) { // Returning true ends only the current flow, so every queued flow checks the shared stop. if (budget.stopped()) return true; + FlowType flow = instruction.getFlowType(); + Address[] flowTargets = instruction.getFlows(); + boolean unresolvedJump = flow.isJump() && !flow.isConditional() && flowTargets.length == 0; for (Register[] pair : swiftStringPairs(function, instruction, registers)) { Long word = registerConstant(context, pair[0]); Long object = registerConstant(context, pair[1]); @@ -3699,12 +4038,14 @@ public final class ReaGhidraBridge extends HeadlessScript { JsonObject item = overCapacity ? swiftStringOmission(layout, word, object) : swiftStringItem(value, layout, word, object); + if (unresolvedJump && !overCapacity) + item = swiftStringOmission(layout, word, object, "unresolved_jump"); item.addProperty("source_address", canonicalAddress(instruction.getAddress())); JsonArray pairRegisters = new JsonArray(); pairRegisters.add(pair[0].getName()); pairRegisters.add(pair[1].getName()); item.add("registers", pairRegisters); - (overCapacity ? omitted : observed).put(key, item); + (overCapacity || unresolvedJump ? omitted : observed).put(key, item); if (overCapacity) { budget.stop(); return true; @@ -3740,11 +4081,12 @@ public final class ReaGhidraBridge extends HeadlessScript { if (operation.getOpcode() == PcodeOp.RETURN) returns = true; } // A jump is a proven tail call only when it has resolved targets and all leave the - // body. A targetless computed jump can still dispatch inside the function. + // body. For unresolved targetless jumps, candidate literals remain unknown omissions. Address[] targets = instruction.getFlows(); boolean tailCall = flow.isJump() && !flow.isConditional() && targets.length > 0 && Arrays.stream(targets).allMatch(target -> !function.getBody().contains(target)); - if (!flow.isCall() && !returns && !tailCall) return pairs; + boolean unresolvedJump = flow.isJump() && !flow.isConditional() && targets.length == 0; + if (!flow.isCall() && !returns && !tailCall && !unresolvedJump) return pairs; // A String occupies two consecutive registers among x0-x7, or x0 and x1 when returned. int lastFirstRegister = returns ? 0 : 6; for (int index = 0; index <= lastFirstRegister; index++) { @@ -3930,6 +4272,15 @@ public final class ReaGhidraBridge extends HeadlessScript { } private static JsonObject swiftStringOmission(SwiftLiteralLayout layout, long word, long object) { + return swiftStringOmission(layout, word, object, "capacity"); + } + + private static JsonObject swiftStringOmission( + SwiftLiteralLayout layout, + long word, + long object, + String reason + ) { JsonObject item = new JsonObject(); item.addProperty("representation", layout.representation()); if (layout.address() == null) { @@ -3938,7 +4289,7 @@ public final class ReaGhidraBridge extends HeadlessScript { item.addProperty("address", canonicalAddress(layout.address())); } item.addProperty("byte_length", layout.byteLength()); - item.addProperty("reason", "capacity"); + item.addProperty("reason", reason); item.add("words", swiftStringWords(word, object)); return item; } @@ -4506,7 +4857,8 @@ public final class ReaGhidraBridge extends HeadlessScript { private record InventoryItem(Address address, String value, JsonObject facts) {} private record AddressNameRequest(String inputAddress, Address address, String name) {} private record FunctionEntry(Function function, InventoryItem item) {} - private record FunctionAnalysisObservation(JsonObject dossier, List limitations) {} + private record FunctionAnalysisObservation(JsonObject dossier, List limitations, JsonObject relationshipCoverage) {} + private record CallBoundaryObservation(List limitations, Integer unrepresentedCalleeCalls, JsonArray unresolvedCalls) {} private record SwiftStringLiteralDecode(JsonArray literals, JsonArray omissions) {} private record SwiftLiteralLayout( String representation, diff --git a/bridge/hopper_bridge.py b/bridge/hopper_bridge.py index 92a473c061..0425cc0d91 100644 --- a/bridge/hopper_bridge.py +++ b/bridge/hopper_bridge.py @@ -644,7 +644,7 @@ def _analyze_function(document, params): "end": _hex(_basic_block_end(procedure, block)), "successors": sorted(set(successors), key=lambda value: int(value, 16)), }) - pseudo = _pseudocode(document, procedure) or "" + pseudo = _pseudocode(document, procedure) assembly_lines = _assembly(procedure).splitlines() callers = sorted((_procedure_identity(item) for item in procedure.getAllCallerProcedures()), key=lambda item: int(item["address"], 16)) callees = sorted((_procedure_identity(item) for item in procedure.getAllCalleeProcedures()), key=lambda item: int(item["address"], 16)) @@ -706,7 +706,7 @@ def _analyze_function(document, params): "signature": procedure.signatureString(), "locals": _procedure_locals(procedure), }, - "pseudocode": pseudo, + "pseudocode": pseudo or "", "assembly": assembly_lines, "comments": comments, "callers": callers, "callees": callees, @@ -718,6 +718,12 @@ def _analyze_function(document, params): "basic_blocks": blocks, "native_api": None, "native_value_flow": None, + "facet_availability": { + "pseudocode": {"available": True} if pseudo is not None else _unavailable("Hopper did not supply pseudocode."), + **{facet: {"available": True} for facet in ( + "assembly", "comments", "callers", "callees", "references", "strings_names", "cfg" + )}, + }, "limitations": [ "Native CallReference classifications are observed where available; detailed reference flags remain unknown.", "Hopper's public Python API does not expose equivalent external or thunk classification in this dossier.", @@ -730,7 +736,12 @@ def _analyze_function(document, params): def _dispatch(method, params): """Dispatch only the closed operation set implemented by REA's public tools.""" if method == "health": - return {"name": "REA Hopper bridge", "version": "1.0.0", "run_id": REA_RUN_ID} + return { + "name": "REA Hopper bridge", + "version": "1.0.0", + "run_id": REA_RUN_ID, + "process_id": os.getpid(), + } if method in ("shutdown", "shutdown_document"): document = _session_document() if document is None: diff --git a/bridge/mitmproxy/capture.py b/bridge/mitmproxy/capture.py index dcd90e745e..c5ca704f69 100644 --- a/bridge/mitmproxy/capture.py +++ b/bridge/mitmproxy/capture.py @@ -6,11 +6,13 @@ The containing mitmdump process and its configuration directory are REA-owned. import base64 import hashlib +import ipaddress import json import math import re import resource from pathlib import Path +from urllib.parse import unquote_to_bytes from mitmproxy import ctx, version from mitmproxy.io import tnetstring @@ -41,6 +43,167 @@ class BoundedReader: return self.handle.read(count) +SPECIAL_SCHEMES = {"ftp", "file", "http", "https", "ws", "wss"} +SCHEME_PREFIX = r"^[\x00-\x20]*([a-z][a-z0-9+.-]*):" +NETWORK_PREFIX = r"^[\x00-\x20]*[/\\]{2}" +URL_IGNORABLE_CONTROLS = "\t\n\r" +URL_CONTROL_TRANSLATION = {ord(char): None for char in URL_IGNORABLE_CONTROLS} +C0_CONTROL_OR_SPACE = "".join(chr(code) for code in range(0x21)) +FORBIDDEN_DOMAIN_CODE_POINTS = set(C0_CONTROL_OR_SPACE) | set("#%/:<>?@[\\]^|\x7f") + + +def authority_start(text, base_scheme=None): + """Return where an authority begins and whether it is special, or None. + + Follows WHATWG URL parsing as src/domain/recordedTransportUrl.ts does: + special schemes (and scheme-relative references) skip every leading "/" + or "\\" before the authority, so `https://\\user:pass@host` still has + userinfo. File URLs never carry userinfo: an "@" in a file host makes + WHATWG reject the whole URL, so there are no credentials to remove. + """ + scheme = re.match(SCHEME_PREFIX, text, flags=re.I) + name = scheme.group(1).lower() if scheme is not None else None + if name == "file" or (name is None and base_scheme == "file"): + return None + if name is not None and name in SPECIAL_SCHEMES: + # A same-scheme Location without two slash-equivalent separators is + # relative to the recorded request, so its @ belongs to the path. + following = text[scheme.end():scheme.end() + 2] + if name == base_scheme and (len(following) < 2 or any(char not in "/\\" for char in following)): + return None + start = scheme.end() + else: + if scheme is not None: + if text[scheme.end():scheme.end() + 2] != "//": + return None + return scheme.end() + 2, False + # A non-special base treats backslashes as authority data and only + # recognizes two forward slashes as a network reference. + if base_scheme is not None and base_scheme not in SPECIAL_SCHEMES: + prefix = re.match(r"^[\x00-\x20]*//", text) + return None if prefix is None else (prefix.end(), False) + prefix = re.match(NETWORK_PREFIX, text) + if prefix is None: + return None + start = prefix.end() + while start < len(text) and text[start] in "/\\": + start += 1 + return start, True + + +IPV4_DIGITS = {8: "01234567", 10: "0123456789", 16: "0123456789abcdefABCDEF"} + + +def ipv4_number(part): + """WHATWG IPv4 number parser: decimal, 0x hex or 0-prefixed octal, else None.""" + if part == "": + return None + radix, digits = 10, part + if part[:2] in ("0x", "0X"): + radix, digits = 16, part[2:] + elif len(part) > 1 and part[0] == "0": + radix, digits = 8, part[1:] + if digits == "": + return 0 + if any(char not in IPV4_DIGITS[radix] for char in digits): + return None + # Bound decimal text before int(): Python limits decimal conversions to + # 4300 digits, and anything past 10 significant digits exceeds IPv4. + significant = digits.lstrip("0") + if radix == 10 and len(significant) > 10: + return 2**32 + return int(digits, radix) + + +def domain_parses(domain): + """Apply WHATWG's ends-in-a-number check and IPv4 parser to a domain.""" + parts = domain.split(".") + if parts[-1] == "" and len(parts) > 1: + parts.pop() + last = parts[-1] + if not ((last != "" and last.isascii() and last.isdigit()) or ipv4_number(last) is not None): + return True + numbers = [ipv4_number(part) for part in parts] + if len(parts) > 4 or any(number is None for number in numbers): + return False + return all(number <= 255 for number in numbers[:-1]) and numbers[-1] < 256 ** (5 - len(numbers)) + + +def special_host_parses(host_port): + """Whether WHATWG could accept a special authority's host and port. + + A rejected host fails the whole URL, so it has no credentials to remove. + This mirrors the forbidden-code-point, empty-host, IPv4, IPv6 and port + rules for ASCII hosts. UTS #46 mapping is not modeled: a non-ASCII host + may map to an accepted one (fullwidth digits, ideographic full stops, + ignored code points), so it is treated as accepted and still redacts. + """ + if host_port.startswith("["): + host_end = host_port.find("]") + 1 + literal = host_port[1 : host_end - 1] + if host_end == 0 or "%" in literal: + return False + try: + ipaddress.IPv6Address(literal) + except ValueError: + return False + else: + colon = host_port.find(":") + host_end = len(host_port) if colon < 0 else colon + host = unquote_to_bytes(host_port[:host_end]).decode("utf-8", errors="replace") + # Mapping never removes ASCII, so these rejections hold either way. + if host == "" or any(char in FORBIDDEN_DOMAIN_CODE_POINTS for char in host): + return False + if host.isascii() and not domain_parses(host.lower()): + return False + port = host_port[host_end:] + if port == "": + return True + digits = port[1:] + if port[0] != ":" or not (digits == "" or (digits.isascii() and digits.isdigit())): + return False + # Bound the text first: int() refuses decimal strings over 4300 digits. + significant = digits.lstrip("0") + return len(significant) <= 5 and int(significant or "0") <= 65535 + + +def without_userinfo(value, base_scheme=None): + """Remove an authority's userinfo using the WHATWG authority boundary. + + Special schemes and scheme-relative references also end the authority + at a backslash, so later path text containing "@" is not userinfo. + """ + binary = isinstance(value, bytes) + text = value.decode("latin-1") if binary else value + # WHATWG removes these controls before interpreting scheme/separators. + # Keep that parse view separate from the exact source spelling and bytes. + # It also drops trailing C0 controls and spaces; offsets are unchanged. + parsed_text = text.translate(URL_CONTROL_TRANSLATION).rstrip(C0_CONTROL_OR_SPACE) + located = authority_start(parsed_text, base_scheme) + if located is None: + return value + start, special = located + terminators = "/?#\\" if special else "/?#" + end = start + while end < len(parsed_text) and parsed_text[end] not in terminators: + end += 1 + at = parsed_text.rfind("@", start, end) + if at < start or (special and not special_host_parses(parsed_text[at + 1 : end])): + return value + parsed_offset = 0 + raw_start = 0 + for raw_offset, char in enumerate(text): + if char in URL_IGNORABLE_CONTROLS: + continue + if parsed_offset == start: + raw_start = raw_offset + if parsed_offset == at: + safe = text[:raw_start] + text[raw_offset + 1 :] + return safe.encode("latin-1") if binary else safe + parsed_offset += 1 + raise CaptureFailure("decoder", "Normalized URL authority could not be bound to its source.") + + def project_record(state, secrets): binaries, numbers, redactions = [], [], [] byte_secrets = [] @@ -58,16 +221,23 @@ def project_record(state, secrets): return None return text - def transport_value(value, path, transport_url): + def transport_value(value, path, transport_url, base_scheme): if not (transport_url or re.fullmatch(r"/(?:backup/)*request/(?:path|authority)", path)): return value, False binary = isinstance(value, bytes) - pattern = rb"^([\x00-\x20]*(?:[a-z][a-z0-9+.-]*:)?//)[^/?#]*@" if binary else r"^([\x00-\x20]*(?:[a-z][a-z0-9+.-]*:)?//)[^/?#]*@" - replacement = rb"\1" if binary else r"\1" separator = b"@" if binary else "@" - safe_url = re.sub(pattern, replacement, value, flags=re.I) - if path.endswith("/authority") and separator in safe_url: - safe_url = safe_url.rsplit(separator, 1)[1] + if transport_url: + safe_url = without_userinfo(value, base_scheme) + elif path.endswith("/authority"): + safe_url = value.rsplit(separator, 1)[-1] + else: + # Native request.path is origin-form unless it explicitly carries + # a scheme (absolute-form). Leading separators are path evidence, + # not a header URL reference's network authority. + text = value.decode("latin-1") if binary else value + safe_url = without_userinfo(value) if re.match( + SCHEME_PREFIX, text.translate(URL_CONTROL_TRANSLATION), flags=re.I + ) else value changed = safe_url != value if changed: redactions.append({"pointer": path, "reason": "transport-credential"}) @@ -80,7 +250,7 @@ def project_record(state, secrets): del numbers[before[1]:] del redactions[before[2]:] - def visit(value, path, depth=0, credential=False, transport_url=False): + def visit(value, path, depth=0, credential=False, transport_url=False, base_scheme=None): if depth > 64: raise CaptureFailure("input-limit", "Native capture exceeds the 64-level evidence nesting budget.", path) if credential: @@ -90,7 +260,7 @@ def project_record(state, secrets): binaries.append({"pointer": path, "representation": "producer-bytes", "state": "redacted", "content_base64": None, "bytes": None, "sha256": None}) return None if isinstance(value, bytes): - safe_bytes, credential_url = transport_value(value, path, transport_url) + safe_bytes, credential_url = transport_value(value, path, transport_url, base_scheme) content_base64 = base64.b64encode(value).decode("ascii") digest = hashlib.sha256(value).hexdigest() hidden = any(secret in value or secret in safe_bytes for secret in byte_secrets) or any(secret in content_base64 or secret in digest or secret in str(len(value)) for secret in secrets) @@ -107,7 +277,7 @@ def project_record(state, secrets): return None return safe_text if isinstance(value, str): - safe_text, _ = transport_value(value, path, transport_url) + safe_text, _ = transport_value(value, path, transport_url, base_scheme) return redact(safe_text, path, value) if isinstance(value, bool) or value is None: return value @@ -125,10 +295,16 @@ def project_record(state, secrets): for index, field in enumerate(value): if not isinstance(field, (list, tuple)) or len(field) != 2 or not isinstance(field[0], bytes): raise CaptureFailure("format", "Native header/trailer is not an ordered byte pair.", pointer(path, index)) - result.append([visit(field[0], pointer(pointer(path, index), 0), depth + 2), visit(field[1], pointer(pointer(path, index), 1), depth + 2, field[0].lower() in CREDENTIALS, field[0].lower() in {b"location", b"referer", b"origin"})]) + result.append([visit(field[0], pointer(pointer(path, index), 0), depth + 2), visit(field[1], pointer(pointer(path, index), 1), depth + 2, field[0].lower() in CREDENTIALS, field[0].lower() in {b"location", b"referer", b"origin"}, base_scheme if field[0].lower() == b"location" else None)]) return result - return [visit(item, pointer(path, index), depth + 1) for index, item in enumerate(value)] + return [visit(item, pointer(path, index), depth + 1, base_scheme=base_scheme) for index, item in enumerate(value)] if isinstance(value, dict): + if path == "" or re.fullmatch(r"/backup(?:/backup)*", path): + request = value.get("request") + scheme = request.get("scheme") if isinstance(request, dict) else None + if isinstance(scheme, bytes): + scheme = scheme.decode("latin-1") + base_scheme = scheme.lower() if isinstance(scheme, str) and re.fullmatch(r"[a-z][a-z0-9+.-]*", scheme, flags=re.I) else None result = {} for key, item in value.items(): if not isinstance(key, str): @@ -139,7 +315,7 @@ def project_record(state, secrets): validate_omitted(item, pointer(path, key), depth + 1) redactions.append({"pointer": path, "reason": "explicit-sensitive-value", "scope": "property-name"}) continue - result[key] = visit(item, pointer(path, key), depth + 1) + result[key] = visit(item, pointer(path, key), depth + 1, base_scheme=base_scheme) return result raise CaptureFailure("decoder", "Native parser returned an unexpected value type: " + type(value).__name__ + ".", path) diff --git a/bridge/mitmproxy/test_capture.py b/bridge/mitmproxy/test_capture.py new file mode 100644 index 0000000000..3863312cb9 --- /dev/null +++ b/bridge/mitmproxy/test_capture.py @@ -0,0 +1,265 @@ +"""Source-owned tests for the mitmproxy adapter's record projection.""" + +import importlib.util +import os +import sys +import types +import unittest +from unittest import mock + + +def load_capture(): + """Load the adapter with fresh mitmproxy stand-ins scoped to its import.""" + fake_mitmproxy = types.ModuleType("mitmproxy") + fake_mitmproxy.ctx = types.SimpleNamespace() + fake_mitmproxy.version = types.SimpleNamespace(VERSION="12.2.3") + fake_io = types.ModuleType("mitmproxy.io") + fake_io.tnetstring = types.ModuleType("mitmproxy.io.tnetstring") + stand_ins = { + "mitmproxy": fake_mitmproxy, + "mitmproxy.io": fake_io, + "mitmproxy.io.tnetstring": fake_io.tnetstring, + } + spec = importlib.util.spec_from_file_location( + "rea_mitmproxy_capture", os.path.join(os.path.dirname(__file__), "capture.py") + ) + module = importlib.util.module_from_spec(spec) + with mock.patch.dict(sys.modules, stand_ins): + spec.loader.exec_module(module) + return module + + +capture = load_capture() + + +def location(value, scheme=None): + state = {"response": {"headers": [[b"Location", value]]}} + if scheme is not None: + state["request"] = {"scheme": scheme} + record = capture.project_record(state, []) + sidecar = next( + field for field in record["binary_fields"] if field["pointer"] == "/response/headers/0/1" + ) + return record["reported"]["response"]["headers"][0][1], sidecar, record["redactions"] + + +class TransportUserinfoTests(unittest.TestCase): + def test_ignorable_controls_cannot_hide_authority_credentials(self): + for control in [b"\t", b"\n", b"\r"]: + for value in [b"https://" + control + b"/user:pass@host/path", b"ht" + control + b"tps://user:" + control + b"pass@host/path"]: + with self.subTest(value=value): + text, sidecar, redactions = location(value, b"https") + self.assertNotIn("user", text) + self.assertNotIn("pass", text) + self.assertEqual(sidecar["state"], "redacted") + self.assertEqual(len(redactions), 1) + + def test_mixed_network_separators_remove_credentials(self): + for prefix in [b"\\\\", b"/\\", b"\\/"]: + with self.subTest(prefix=prefix): + text, sidecar, redactions = location(prefix + b"user:pass@host/path", b"https") + self.assertEqual(text, prefix.decode("ascii") + "host/path") + self.assertEqual(sidecar["state"], "redacted") + self.assertEqual(len(redactions), 1) + + def test_non_special_base_keeps_backslashes_inside_network_authorities(self): + text, sidecar, redactions = location(b"//user\\name:pass@host/x", b"foo") + self.assertEqual(text, "//host/x") + self.assertEqual(sidecar["state"], "redacted") + self.assertEqual(len(redactions), 1) + text, sidecar, redactions = location(b"/\\invoice@customer", b"foo") + self.assertEqual(text, "/\\invoice@customer") + self.assertEqual(sidecar["state"], "retained") + self.assertEqual(redactions, []) + + def test_native_request_paths_preserve_origin_form_and_redact_absolute_form(self): + for value in [b"/\\invoice@customer", b"//invoice@customer", b"\\\\invoice@customer"]: + with self.subTest(value=value): + record = capture.project_record({"request": {"path": value}}, []) + self.assertEqual(record["reported"]["request"]["path"], value.decode("ascii")) + self.assertEqual(record["binary_fields"][0]["state"], "retained") + self.assertEqual(record["redactions"], []) + record = capture.project_record({"request": {"path": b"https://user:pass@host/x"}}, []) + self.assertEqual(record["reported"]["request"]["path"], "https://host/x") + self.assertEqual(record["binary_fields"][0]["state"], "redacted") + + def test_same_scheme_locations_preserve_relative_path_text_and_bytes(self): + for value in [b"https:invoice@customer", b"https:/invoice@customer", b"https:\\invoice@customer"]: + with self.subTest(value=value): + text, sidecar, redactions = location(value, b"https") + self.assertEqual(text, value.decode("ascii")) + self.assertEqual(sidecar["state"], "retained") + self.assertEqual(redactions, []) + + def test_location_authorities_still_remove_credentials_with_a_base_scheme(self): + for value, scheme, expected in [ + (b"https://user:pass@host/path", b"https", "https://host/path"), + (b"https:\\\\user:pass@host/path", b"https", "https:\\\\host/path"), + (b"https:user:pass@host/path", b"http", "https:host/path"), + ]: + with self.subTest(value=value, scheme=scheme): + text, sidecar, redactions = location(value, scheme) + self.assertEqual(text, expected) + self.assertEqual(sidecar["state"], "redacted") + self.assertEqual(len(redactions), 1) + + def test_backup_location_uses_its_own_request_scheme(self): + record = capture.project_record({ + "request": {"scheme": b"http"}, + "backup": { + "request": {"scheme": b"https"}, + "response": {"headers": [[b"Location", b"https:invoice@customer"]]}, + }, + }, []) + self.assertEqual(record["reported"]["backup"]["response"]["headers"][0][1], "https:invoice@customer") + self.assertEqual(record["redactions"], []) + + def test_backslash_ends_a_special_scheme_authority(self): + text, sidecar, redactions = location(b"https://example.test\\part@tail") + self.assertEqual(text, "https://example.test\\part@tail") + self.assertEqual(sidecar["state"], "retained") + self.assertIsNotNone(sidecar["content_base64"]) + self.assertEqual(redactions, []) + + def test_userinfo_before_a_backslash_is_still_removed(self): + text, sidecar, redactions = location(b"https://user:pw@example.test\\path") + self.assertEqual(text, "https://example.test\\path") + self.assertEqual(sidecar["state"], "redacted") + self.assertEqual( + redactions, + [{"pointer": "/response/headers/0/1", "reason": "transport-credential"}], + ) + + def test_ordinary_userinfo_is_removed(self): + text, _sidecar, _redactions = location(b"https://user:pw@example.test/a@b") + self.assertEqual(text, "https://example.test/a@b") + + def test_empty_userinfo_marker_is_removed(self): + self.assertEqual( + capture.without_userinfo("https://@host"), "https://host" + ) + + def test_special_schemes_skip_leading_slashes_before_userinfo(self): + for value, expected in [ + (b"https://\\user:pass@host/path", b"https://\\host/path"), + (b"https:\\\\user:pass@host/path", b"https:\\\\host/path"), + (b"https:user:pass@host", b"https:host"), + (b"//\\user:pass@host", b"//\\host"), + ]: + with self.subTest(value=value): + self.assertEqual(capture.without_userinfo(value), expected) + text, sidecar, redactions = location(b"https://\\user:pass@host/path") + self.assertEqual(text, "https://\\host/path") + self.assertEqual(sidecar["state"], "redacted") + self.assertEqual(len(redactions), 1) + + def test_file_base_keeps_paths_after_the_fixed_network_boundary(self): + for value in [b"///u@h", b"////u@h"]: + with self.subTest(value=value): + text, sidecar, redactions = location(value, b"file") + self.assertEqual(text, value.decode("ascii")) + self.assertEqual(sidecar["state"], "retained") + self.assertIsNotNone(sidecar["content_base64"]) + self.assertEqual(redactions, []) + + def test_file_urls_keep_path_text(self): + self.assertEqual(capture.without_userinfo("file:///a@b"), "file:///a@b") + + def test_file_urls_reject_userinfo_and_keep_their_source(self): + # WHATWG rejects an "@" in a file host, so nothing parses as credentials. + self.assertEqual( + capture.without_userinfo("file://user:pass@host/x"), "file://user:pass@host/x" + ) + for value in [b"\\\\user:pass@host/x", b"/\\user:pass@host/x", b"//user:pass@host/x"]: + with self.subTest(value=value): + text, sidecar, redactions = location(value, b"file") + self.assertEqual(text, value.decode("ascii")) + self.assertEqual(sidecar["state"], "retained") + self.assertEqual(redactions, []) + + def test_rejected_special_hosts_keep_their_source(self): + # A rejected host or port fails the whole URL, so it has no userinfo. + for value in [ + "https:user@%", + "https://user@", + "https://user:pass@h^st/x", + "https://user@ho st/x", + "https://user@host:99999/x", + "https://user@host:12a", + "https://user@[::1", + ]: + with self.subTest(value=value): + self.assertEqual(capture.without_userinfo(value), value) + text, sidecar, redactions = location(b"https://user:pass@h^st/x", b"https") + self.assertEqual(text, "https://user:pass@h^st/x") + self.assertEqual(sidecar["state"], "retained") + self.assertEqual(redactions, []) + + def test_accepted_special_hosts_still_remove_userinfo(self): + for value, expected in [ + ("https:user@%41", "https:%41"), + ("https://user@[::1]:8443/x", "https://[::1]:8443/x"), + ("https://user@host:/x", "https://host:/x"), + ("wss://user:pass@host:65535", "wss://host:65535"), + ("https://user@0x7f.1/", "https://0x7f.1/"), + ("https://user@1.2.3.4./", "https://1.2.3.4./"), + ("https://user@4294967295/", "https://4294967295/"), + ("https://user@1.2.3.08a/", "https://1.2.3.08a/"), + ("https://user@[::ffff:1.2.3.4]/", "https://[::ffff:1.2.3.4]/"), + ]: + with self.subTest(value=value): + self.assertEqual(capture.without_userinfo(value), expected) + + def test_unicode_hosts_that_may_map_to_accepted_hosts_remove_userinfo(self): + # UTS #46 maps these to 1.2.3.4; mapping is not modeled, so redact. + for host in ["%EF%BC%91.2.3.4", "1。2.3.4", "%C2%AD1.2.3.4", "１.2.3.4"]: + with self.subTest(host=host): + self.assertEqual( + capture.without_userinfo(f"https://user:pw@{host}/"), f"https://{host}/" + ) + text, sidecar, redactions = location(b"https://user:pw@%EF%BC%91.2.3.4/", b"https") + self.assertEqual(text, "https://%EF%BC%91.2.3.4/") + self.assertEqual(sidecar["state"], "redacted") + self.assertEqual(len(redactions), 1) + # ASCII rejections still hold for non-ASCII hosts. + self.assertEqual(capture.without_userinfo("https://user@１^/"), "https://user@１^/") + + def test_trailing_controls_and_spaces_do_not_hide_userinfo(self): + # WHATWG strips trailing C0 controls and spaces before parsing. + for suffix in [b" ", b"\x01", b"\x00 \x1f"]: + with self.subTest(suffix=suffix): + value = b"https://user:pw@example.test" + suffix + text, sidecar, redactions = location(value, b"https") + self.assertEqual(text, "https://example.test" + suffix.decode("latin-1")) + self.assertEqual(sidecar["state"], "redacted") + self.assertEqual(len(redactions), 1) + + def test_rejected_ip_literals_keep_their_source(self): + for value in [ + "https://user:pass@256.256.256.256/path", + "https://user@1.2.3.4.5/", + "https://user@1.2.3.09/", + "https://user@4294967296/", + "https://user@a.0x/", + "https://user@1..2/", + "https://user@1." + "9" * 5000 + "/", + "https://user@[1::2::3]/", + "https://user@[::1%25eth0]/", + ]: + with self.subTest(value=value[:40]): + self.assertEqual(capture.without_userinfo(value), value) + + def test_overlong_ports_are_compared_without_numeric_overflow(self): + # Python 3.11+ refuses int() on decimal strings over 4300 digits. + accepted = "https://user@host:" + "0" * 5000 + "80/x" + self.assertEqual(capture.without_userinfo(accepted), "https://host:" + "0" * 5000 + "80/x") + rejected = "https://user@host:" + "9" * 5000 + "/x" + self.assertEqual(capture.without_userinfo(rejected), rejected) + + def test_non_special_schemes_keep_backslashes_in_the_authority(self): + self.assertEqual(capture.without_userinfo("foo://a\\b@host/x"), "foo://host/x") + self.assertEqual(capture.without_userinfo(b"//a\\b@host"), b"//a\\b@host") + + +if __name__ == "__main__": + unittest.main() diff --git a/bridge/native/rea_lldb_tracer.py b/bridge/native/rea_lldb_tracer.py index b99123bf4b..fcec3ea92b 100644 --- a/bridge/native/rea_lldb_tracer.py +++ b/bridge/native/rea_lldb_tracer.py @@ -567,18 +567,27 @@ def trace(debugger, config): before_digest = _hash_file(config["executable"]) before_identity = _file_identity(config["executable"]) except OSError as error: - return {"status": "target-integrity-error", "error": str(error)} + return { + "status": "target-integrity-error", + "error": str(error), + "target_process_created": False, + } if before_digest != config["expected_sha256"]: return { "status": "target-integrity-error", "error": "selected executable digest changed before LLDB target creation", + "target_process_created": False, } error = lldb.SBError() target = debugger.CreateTarget( config["executable"], config["architecture"], None, False, error ) if not target.IsValid(): - return {"status": "target-error", "error": error.GetCString()} + return { + "status": "target-error", + "error": error.GetCString(), + "target_process_created": False, + } module_path_before, module_file_identity_before, module_uuid_before = ( _module_identity(target) ) @@ -589,6 +598,7 @@ def trace(debugger, config): return { "status": "target-integrity-error", "error": "LLDB executable module resolved to a different pre-launch file identity", + "target_process_created": False, } created = [_create_breakpoints(target, spec) for spec in config["breakpoints"]] breakpoint_index = {} @@ -622,12 +632,24 @@ def trace(debugger, config): if not error.Success() or not process.IsValid(): stdout.close() stderr.close() - return {"status": "launch-error", "error": error.GetCString()} + return { + "status": "launch-error", + "error": error.GetCString(), + **( + {"target_process_created": True} + if process.IsValid() + else {} + ), + } launch_failure = _failed_launch_description(process) if launch_failure is not None: stdout.close(expected_exit=True) stderr.close(expected_exit=True) - return {"status": "launch-error", "error": launch_failure} + return { + "status": "launch-error", + "error": launch_failure, + "target_process_created": True, + } stdout.mark_launched() stderr.mark_launched() # Publish the PID before any post-launch file or module inspection. The @@ -644,6 +666,7 @@ def trace(debugger, config): return { "status": "target-identity-unavailable", "error": "host could not establish the launched process identity; target was stopped", + "target_process_created": True, } try: after_digest = _hash_file(config["executable"]) @@ -652,7 +675,11 @@ def trace(debugger, config): process.Kill() stdout.close(expected_exit=process.GetState() == lldb.eStateExited) stderr.close(expected_exit=process.GetState() == lldb.eStateExited) - return {"status": "target-integrity-error", "error": str(identity_error)} + return { + "status": "target-integrity-error", + "error": str(identity_error), + "target_process_created": True, + } if after_digest != config["expected_sha256"] or after_identity != before_identity: process.Kill() _wait_for_exit(listener, process, 5) @@ -661,6 +688,7 @@ def trace(debugger, config): return { "status": "target-integrity-error", "error": "selected executable digest or file identity changed during LLDB launch", + "target_process_created": True, } module_path_after, module_file_identity_after, module_uuid_after = ( _module_identity(target) @@ -676,6 +704,7 @@ def trace(debugger, config): return { "status": "target-integrity-error", "error": "LLDB executable module resolved to a different file identity", + "target_process_created": True, } if ( module_uuid_before is not None @@ -689,6 +718,7 @@ def trace(debugger, config): return { "status": "target-integrity-error", "error": "LLDB executable module identity changed during launch", + "target_process_created": True, } context = { "target": target, @@ -748,7 +778,11 @@ def trace(debugger, config): # exit code; preserve the debugger's concrete launch denial. launch_failure = _failed_launch_description(process) if launch_failure is not None: - return {"status": "launch-error", "error": launch_failure} + return { + "status": "launch-error", + "error": launch_failure, + "target_process_created": True, + } launch_identity_stable = ( module_path_before is not None and module_path_after == module_path_before diff --git a/bridge/native/test_rea_lldb_tracer.py b/bridge/native/test_rea_lldb_tracer.py index 7cd95d5038..308ed00336 100644 --- a/bridge/native/test_rea_lldb_tracer.py +++ b/bridge/native/test_rea_lldb_tracer.py @@ -8,6 +8,7 @@ import sys import tempfile import types import unittest +from unittest.mock import patch fake_lldb = sys.modules.setdefault("lldb", types.ModuleType("lldb")) @@ -22,6 +23,42 @@ spec.loader.exec_module(tracer) class NativeTraceBudgetTests(unittest.TestCase): + def test_pre_target_integrity_failure_reports_no_process_created(self): + original_debugger = getattr(tracer.lldb, "SBDebugger", None) + + class DebuggerApi: + @staticmethod + def SetInternalVariable(*_arguments): + return None + + class Debugger: + def SetAsync(self, _enabled): + return None + + def GetInstanceName(self): + return "fixture" + + def CreateTarget(self, *_arguments): + raise AssertionError("integrity failure precedes target creation") + + tracer.lldb.SBDebugger = DebuggerApi + try: + with patch.object(tracer, "_hash_file", return_value="changed"), patch.object( + tracer, "_file_identity", return_value=(1, 2) + ): + result = tracer.trace( + Debugger(), + {"executable": "fixture", "expected_sha256": "expected"}, + ) + finally: + if original_debugger is None: + del tracer.lldb.SBDebugger + else: + tracer.lldb.SBDebugger = original_debugger + + self.assertEqual(result["status"], "target-integrity-error") + self.assertIs(result["target_process_created"], False) + def test_journal_checks_exact_byte_budget_before_writing(self): record = {"kind": "other-stop", "reason": "signal"} encoded_size = len((tracer.json.dumps(record) + "\n").encode("utf-8")) diff --git a/docs/.vitepress/config.ts b/docs/.vitepress/config.ts index b4cf2e4ade..440338f987 100644 --- a/docs/.vitepress/config.ts +++ b/docs/.vitepress/config.ts @@ -44,6 +44,7 @@ export default defineConfig({ link: "/web-network-captures", }, { text: "Electron observation", link: "/electron-observation" }, + { text: "Electron fuses", link: "/electron-fuses" }, { text: "Process capture", link: "/process-capture" }, { text: ".NET assemblies", link: "/managed-code-analysis" }, { text: "Android applications", link: "/android-analysis" }, @@ -63,6 +64,7 @@ export default defineConfig({ { text: "Ghidra DOS analysis", link: "/ghidra-dos" }, { text: "Ghidra raw binary", link: "/raw-binary" }, { text: "Ghidra NativeAOT", link: "/ghidra-nativeaot" }, + { text: "Ghidra PowerPC", link: "/ghidra-ppc" }, { text: "Ghidra semantics", link: "/ghidra-provider" }, ], }, diff --git a/docs/adb-device-analysis.md b/docs/adb-device-analysis.md index 12fd77cd25..60fecc227b 100644 --- a/docs/adb-device-analysis.md +++ b/docs/adb-device-analysis.md @@ -26,6 +26,9 @@ Readiness is `rea inspect-adb-client`: it runs `adb version` and reports the resolved binary, its version, and where it was selected from. A missing binary reports `capability_unavailable` with the `REA_ADB_PATH` remediation rather than attempting an installation. +The `binary_session` inventory uses the same bounded client-version probe; it +does not query devices, start the adb server, or establish that a requested +serial is reachable. Device state is checked by each device operation. ## Answer an analyst question @@ -73,32 +76,37 @@ existing analysis families directly, for example ## Operations -| Tool | Kind | What it does | -| ----------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `inspect_adb_client` | read | Resolved binary, version, install path; never contacts the server | -| `list_adb_devices` | read | Serial, state, transport, model metadata, inferred emulator/physical kind with basis | -| `inspect_adb_device` | read | Fixed `getprop` whitelist: build, SDK, patch, fingerprint, product, ABIs, debuggable, `ro.kernel.qemu` | -| `inspect_adb_display` | read | `wm size` and `wm density` as declared | -| `list_adb_features` | read | `pm list features` (hardware/software features, hex GL versions) | -| `list_adb_services` | read | `service list` binder services and interfaces | -| `list_adb_processes` | read | `ps -A`: user, pid, ppid, rss, state, name | -| `inspect_adb_window` | read | `dumpsys window` focus declarations (what is on screen) | -| `read_adb_setting` | read | One `settings get` value; device-side null is reported as such | -| `read_adb_logcat` | read | Bounded `logcat -d -t` dump for one buffer, optional pid filter | -| `list_adb_packages` | read | `pm list packages -f` with scope; base APK device paths | -| `resolve_adb_packages` | read | Case-insensitive package-name substring search with exact-match flag | -| `inspect_adb_package` | read | `dumpsys package` projection: versions, install times, installer, flags, permission count | -| `list_adb_directory` | read | NUL-framed immediate children, including hidden entries: kinds, permissions, owner, group, sizes, symlink targets | -| `pull_adb_package` | read | Complete split APK set from `pm path`, each file digested | -| `pull_adb_file` | read | One absolute device path pulled and digested | -| `capture_adb_screen` | read | One `screencap -p` frame (exec-out, no device-side file) with dimensions | -| `collect_adb_bugreport` | read | Device-generated bugreport zip, digested | -| `push_adb_file` | **mutates device** | One local file to a device path (e.g. pushing tools into `/data/local/tmp`); refuses existing targets without explicit overwrite; verifies via the device's own sha256sum | -| `install_adb_package` | **mutates device** | Local APK via `adb install [-r]`; preserves the device's Success/Failure verdict | -| `uninstall_adb_package` | **mutates device** | Removes a package and its data for user 0 | -| `start_adb_app` | **mutates device** | Resolves the package's launcher activity (`cmd package resolve-activity --brief`) and starts it (`am start -n`) | -| `start_adb_activity` | **mutates device** | Starts an intent (`am start -a` with optional URI, component, and up to 16 typed extras); preserves the activity manager's verdict | -| `stop_adb_app` | **mutates device** | Force-stops every process of a package, ending a dynamic-analysis run | +| Tool | Kind | What it does | +| ----------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `inspect_adb_client` | read | Resolved binary, version, install path; never contacts the server | +| `list_adb_devices` | read | Serial, state, transport, model metadata, inferred emulator/physical kind with basis | +| `inspect_adb_device` | read | Fixed `getprop` whitelist: build, SDK, patch, fingerprint, product, ABIs, debuggable, `ro.kernel.qemu` | +| `inspect_adb_display` | read | `wm size` and `wm density` as declared | +| `list_adb_features` | read | `pm list features` (hardware/software features, hex GL versions) | +| `list_adb_services` | read | `service list` binder services and interfaces | +| `list_adb_processes` | read | `ps -A`: user, pid, ppid, rss, state, name | +| `inspect_adb_window` | read | `dumpsys window` focus declarations (what is on screen) | +| `read_adb_setting` | read | One `settings get` value; device-side null is reported as such | +| `read_adb_logcat` | read | Bounded `logcat -d -t` dump for one buffer, optional pid filter | +| `list_adb_packages` | read | `pm list packages -f` with scope; base APK device paths | +| `resolve_adb_packages` | read | Case-insensitive package-name substring search with exact-match flag | +| `inspect_adb_package` | read | `dumpsys package` projection: versions, install times, installer, flags, permission count | +| `list_adb_directory` | read | NUL-framed immediate children, including hidden entries: kinds, permissions, owner, group, sizes, symlink targets | +| `pull_adb_package` | read | Complete split APK set from `pm path`, each file digested | +| `pull_adb_file` | read | One absolute device path pulled and digested | +| `capture_adb_screen` | read | One `screencap -p` frame (exec-out, no device-side file) with dimensions | +| `collect_adb_bugreport` | read | Device-generated bugreport zip, digested | +| `push_adb_file` | **mutates device** | One local file to a device path (e.g. pushing tools into `/data/local/tmp`); checks for existing targets and refuses them without explicit overwrite (the check is not atomic); compares digests when device `sha256sum` is available | +| `install_adb_package` | **mutates device** | Local APK via `adb install [-r]`; preserves its Success/Failure verdict, records `replace_requested`, and leaves actual replacement `null` because prior package state is not observed | +| `uninstall_adb_package` | **mutates device** | Removes a package and its data for user 0 | +| `start_adb_app` | **mutates device** | Resolves the package's launcher activity (`cmd package resolve-activity --brief`) and starts it (`am start -n`) | +| `start_adb_activity` | **mutates device** | Starts an intent (`am start -a` with optional URI, component, and up to 16 typed extras); preserves the activity manager's verdict | +| `stop_adb_app` | **mutates device** | Force-stops every process of a package, ending a dynamic-analysis run | + +For `install_adb_package`, `replace_requested` records whether `-r` was sent. +The `replaced` result is `null`: the successful install response does not say +whether the package was installed fresh or replaced, and REA does not query the +device first. The install verdict and host APK digest remain available. ## Interpret the result @@ -118,6 +126,16 @@ existing analysis families directly, for example while POSIX hosts retain valid POSIX filenames such as names with spaces or colons. A package pull checks reported names against the destination filesystem before pulling, and refuses names that would share a local path. +- **Package path refusals are diagnosable.** Refusals from `pull_adb_package` + return `unreadable_output` with `retryable: false`, keep the `pm path` output + in `details.captured_output`, and name the refusal in `details.reason`. + `unsafe_device_path` lists refused entries in + `details.diagnostics.rejected_paths` as `{device_path, reason}`; + `duplicate_local_destination` reports the shared `file_name` and both + `device_paths`; `local_destination_unavailable` reports the `device_path`, + `file_name`, and filesystem `error_code`. Diagnostics retain raw device + strings, while the caller-facing message escapes invisible and bidirectional + characters so they cannot hide text or reorder the path. - **Splits are part of the answer.** `pm path` enumerates the base APK and every installed split; pulls cover the whole set. System packages whose APKs keep other names report the `unknown` role rather than guessing. @@ -127,8 +145,10 @@ existing analysis families directly, for example partial coverage instead of dropping rows. - **Mutations are explicit.** `push_adb_file`, `install_adb_package`, and `uninstall_adb_package` declare `mutatesTarget` and never run implicitly: - pushes refuse existing device files unless `overwrite` is set and verify - the transfer with the device's own `sha256sum`; installs preserve the + pushes check for existing device files before transfer and refuse them + unless `overwrite` is set. The check is not atomic, so a different process + can create the path before `adb push` writes it. When available, the device's + `sha256sum` verifies the transfer; installs preserve the device's `Failure [...]` reason verbatim. - **Producer identity stays attached.** Every result embeds the producing adb client (`binary_path`, `path_source`, `version`, `installed_path`). diff --git a/docs/android-analysis.md b/docs/android-analysis.md index 50087f4648..1bad043531 100644 --- a/docs/android-analysis.md +++ b/docs/android-analysis.md @@ -9,6 +9,10 @@ This family is present on repository main and in npm 4.1.0. Check the [released package boundary](installation.md#released-package-and-main) and the connected server's tool list before selecting it. +On Linux, JADX launches an REA-owned process and requires a procps-compatible +`ps` on REA's `PATH`; see the shared +[Linux process prerequisite](installation.md#linux-owned-process-prerequisite). + ## Optional package delivery The MCP `inspect_android_package` request keeps complete output when `detail` is @@ -33,10 +37,40 @@ unverified. ## Supply tools Use a **full JDK 17 or newer** (including its compiler module) and -**jadx-headless-mcp 0.7.1**. REA runs its packaged Java metadata bridge in -source-file mode against the unmodified engine JAR. REA does not install Java, -an Android SDK, JADX, a device service or an emulator. Set the path in the shell -or MCP server environment: +[**jadx-headless-mcp 0.7.1**](https://github.com/1013503897/jadx-headless-mcp/releases/download/v0.7.1/jadx-headless-mcp-0.7.1-all.jar). +Verify the downloaded JAR's SHA-256 against +[REA's pinned release metadata](https://github.com/morluto/rea/blob/main/src/android/JadxRelease.ts) +before using it. The following download and verification commands work for npm +package consumers as well as repository checkouts; the repository-only +`npm run fixtures:android` helper is not included in the npm package. + +On macOS or Linux, choose a destination and verify the pinned digest: + +```sh +mkdir -p "$HOME/.local/share/rea/android" && +cd "$HOME/.local/share/rea/android" && +curl -fL -o jadx-headless-mcp-0.7.1-all.jar https://github.com/1013503897/jadx-headless-mcp/releases/download/v0.7.1/jadx-headless-mcp-0.7.1-all.jar && +if command -v sha256sum >/dev/null; then + echo '6e5eacf500b64292bfb73c49797c1958f6ee44646e43e868039ae7feb573ff75 jadx-headless-mcp-0.7.1-all.jar' | sha256sum -c - +else + echo '6e5eacf500b64292bfb73c49797c1958f6ee44646e43e868039ae7feb573ff75 jadx-headless-mcp-0.7.1-all.jar' | shasum -a 256 -c - +fi && +export REA_JADX_MCP_JAR="$PWD/jadx-headless-mcp-0.7.1-all.jar" +``` + +On Windows, run the equivalent in PowerShell: + +```powershell +$jar = Join-Path $HOME '.local\share\rea\android\jadx-headless-mcp-0.7.1-all.jar' +New-Item -ItemType Directory -Force (Split-Path $jar) | Out-Null +Invoke-WebRequest -Uri 'https://github.com/1013503897/jadx-headless-mcp/releases/download/v0.7.1/jadx-headless-mcp-0.7.1-all.jar' -OutFile $jar +if ((Get-FileHash -Algorithm SHA256 $jar).Hash.ToLowerInvariant() -ne '6e5eacf500b64292bfb73c49797c1958f6ee44646e43e868039ae7feb573ff75') { throw 'JADX JAR SHA-256 does not match REA pinned metadata' } +$env:REA_JADX_MCP_JAR = $jar +``` + +REA runs its packaged Java metadata bridge in source-file mode against the +unmodified engine JAR. REA does not install Java, an Android SDK, JADX, a device +service or an emulator. Set the path in the shell or MCP server environment: ```sh export REA_JADX_MCP_JAR=/absolute/path/jadx-headless-mcp-0.7.1-all.jar diff --git a/docs/apktool-resource-analysis.md b/docs/apktool-resource-analysis.md index f29d7948ce..447d6ccfc5 100644 --- a/docs/apktool-resource-analysis.md +++ b/docs/apktool-resource-analysis.md @@ -21,6 +21,8 @@ Readiness is `rea inspect-apktool-client`: it runs `apktool --version` and reports the resolved launcher, its selection source, and its version. A missing launcher reports `capability_unavailable` with the `REA_APKTOOL_COMMAND` remediation rather than attempting an installation. +`binary_session` uses the same host probe for tool availability; it does not +validate a particular APK or guarantee that the selected input can be decoded. On Windows, a `.bat` or `.cmd` launcher (such as Chocolatey's `apktool.bat`) runs through an explicit `cmd.exe /d /s /c` call. That call has no AutoRun, diff --git a/docs/architecture.mermaid b/docs/architecture.mermaid index 753c57024e..90dd41ab17 100644 --- a/docs/architecture.mermaid +++ b/docs/architecture.mermaid @@ -38,6 +38,10 @@ graph TD JS_CODEC["src/javascript/sourceMaps/
pinned upstream map codec
owned heap-limited Node process, complete point evidence"] INSPECTOR["src/inspector/
passive V8 Inspector discovery and capture
script and target locations"] DOTNET["src/dotnet/
Managed static metadata provider
PE/CLI metadata inventory, members, IL"] + OBJDUMP["src/objdump/
GNU objdump command profile"] + RIZIN["src/rizin/
Static command profile and persistent debugger sessions"] + GDB["src/gdb/
Persistent GDB/MI sessions
owned process lifecycle and target state"] + FIRMWARE["src/firmware/
Binwalk / Unblob adapters
owned processes, report normalization, private extraction"] ANDROID["src/android/
Static APK adapter
owned SDK stdio, pinned headless JADX, normalization"] REFERENCE["src/reference/
Filesystem reference-source reader
bounded traversal, validation, errors"] @@ -50,7 +54,7 @@ graph TD end subgraph "Contracts Layer" - CONTRACTS["src/contracts/
Caller-visible schemas for analysis, managed, browser, Electron, application, and session tools"] + CONTRACTS["src/contracts/
Caller-visible schemas for analysis, reverse engineering, managed, browser, Electron, application, and session tools"] ERRORS["src/contracts/errorSchemas.ts
Shared analysis error schemas
CLI diagnostics, generated error catalog"] end @@ -122,6 +126,14 @@ graph TD INSPECTOR --> DOMAIN APP --> DOTNET DOTNET --> DOMAIN + COMPOSITION --> OBJDUMP + OBJDUMP --> DOMAIN + COMPOSITION --> RIZIN + RIZIN --> DOMAIN + CLI --> GDB + SERVER --> GDB + GDB --> PROCESS + GDB --> DOMAIN HOPPER --> HOPPER_BRIDGE GHIDRA --> GHIDRA_BRIDGE HOPPER --> PROCESS diff --git a/docs/browser-observation.md b/docs/browser-observation.md index 879d01520a..355d7f98ea 100644 --- a/docs/browser-observation.md +++ b/docs/browser-observation.md @@ -18,8 +18,19 @@ Inspect native SAZ or HAR 1.2 offline with REA, or connect Everywhere's separate - `inspect_web_page` / `rea inspect-web-page` captures DOM structure, accessibility nodes, scripts, resources, safe response/DOM metadata, attach-window network and console metadata, WebSocket frame sizes, workers, quota, and optionally selected storage key names, redacted storage-content fingerprints, or script sources. - `analyze_web_bundle` / `rea analyze-web-bundle` parses selected script artifacts without execution and derives chunk edges, route and endpoint candidates, vendor fingerprints, static WebMCP declarations, and optionally source-map/original-source evidence. - `observe_web_session` / `rea observe-web-session` arms an observation window of the requested duration for an external user action and records ordered reload, SPA navigation, redirect, failure, lifecycle, and target-termination metadata. -- `discover_webmcp_tools` / `rea discover-webmcp-tools` uses the experimental CDP WebMCP domain to return every in-scope registration and its complete observed `input_schema` declaration and canonical `input_schema_sha256` digest inline. Declarations are untrusted JSON data, including declared types, required fields, enums, examples and reference URLs. REA does not resolve schema references. These fields replace `input_schema_shape`. Registrations with the same name and URL in distinct observed frames retain separate identities. `tool_key` includes the observed frame ID; it is not a stable identity across navigation or browser sessions. Same-frame registration updates replace that owner only, and removal leaves other owners intact. REA never exposes `WebMCP.invokeTool`; malformed protocol payloads fail explicitly instead of returning incomplete tool metadata. +- `discover_webmcp_tools` / `rea discover-webmcp-tools` uses the experimental CDP WebMCP domain to return every in-scope registration and its complete observed `input_schema` declaration and canonical `input_schema_sha256` digest inline. Declarations are untrusted JSON data, including declared types, required fields, enums, examples and reference URLs. REA does not resolve schema references. These fields replace `input_schema_shape`. Registrations with the same name and URL in distinct observed frames retain separate identities. `tool_key` includes the observed frame ID; it is not a stable identity across navigation or browser sessions. Same-frame registration updates replace that owner only, and removal leaves other owners intact. Declared `readOnly`, `untrustedContent`, `consequential`, and `autosubmit` annotation hints are reported per registration: `true`/`false` are declared values, `null` means the declaration did not provide the hint in that capture, and a missing `consequential` key means the hint was not recorded, either in a historical capture or because the observed annotation was invalid. Consult discovery completeness and exclusions to distinguish these cases. Malformed annotation values retain the registration with incomplete coverage and an `invalid_protocol_value` exclusion. Other malformed protocol payloads can fail discovery explicitly. REA never exposes `WebMCP.invokeTool`. - `compare_web_captures` / `rea compare-web-captures` compares passive captures or exact-step-aligned browser scenarios. Scenario results commit reproducible literal normalization and expose alignment failures plus artifact-level action, screenshot, DOM, accessibility, URL, history, storage, and event differences. Missing or truncated evidence is never treated as equivalence. + +Passive DOM comparison checks complete node inventories and complete frame +parent/child structure independently. Frame IDs and sibling enumeration order +are capture-local and do not establish a change; duplicate IDs, missing parents, +and cyclic ancestry remain unknown. A complete frame difference can still prove +change when DOM nodes are incomplete, and vice versa. Partial DOM or metadata +collection digests do not prove modification: a missing node or link can result +from a protocol omission. Rejected DOM facts also mark derived URL and agent-hint +metadata incomplete, while valid facts from other nodes and documents remain +available in the capture. + - `capture_web_screenshot` / `rea capture-web-screenshot` returns a content-addressed visible-viewport PNG. - `compare_web_screenshots` / `rea compare-web-screenshots` performs bounded local PNG pixel comparison without OCR or external services. - The CLI comparison commands accept each capture, screenshot artifact or normalization policy as inline JSON or a JSON file path. Use files for screenshot artifacts and scenario captures, whose embedded PNG bytes can exceed the host's command-line length limit. @@ -190,6 +201,36 @@ rea discover-webmcp-tools http://127.0.0.1:9222 TARGET_ID \ --json ``` +### WebMCP discovery prerequisites + +`discover_webmcp_tools` reports only what the attached browser's experimental +`WebMCP` CDP domain exposes. When a browser rejects the unknown domain, the +result reports `status: "unavailable"`. Whether a page can register tools at +all depends on several conditions that are not interchangeable: + +- A site's own origin-trial token can enable the API for that origin + **provided the trial is still active, the browser version and platform + support it, and user settings permit it** — token presence alone is not a + guarantee. The trial is announced for Chrome 149 and later. +- The `WebMCPTesting` feature — `--enable-features=WebMCPTesting` at launch, or + `chrome://flags/#enable-webmcp-testing` in a normal profile — enables the + API page-side for that browser instance, independent of per-site tokens. + In one pilot run, a site whose registration code was present but whose + served token was scoped for third-party use registered 0 tools under a + default launch and 4 under the flag; that the token did not enroll the + first-party document is consistent with this but is an inference, not a + measured Chrome behavior. + +In Chrome 154 specifically (one tested version), `WebMCP.enable` answered +under a default launch — the debug domain was not flag-gated in that run. +Other versions may differ, so an unattributed zero-tool result should be +read as registration state only after domain availability is confirmed. + +Discovery is attach-only: the command needs a reachable loopback CDP endpoint +and a target ID from `list_browser_targets`, and it never navigates, reloads, +or evaluates the page. Observe the page in the state you want to measure; +registered tools reflect the document as loaded. + Accessibility names/descriptions, console primitive text, JSON body shapes, WebSocket shapes, script content, and storage key names are selected independently: ```bash @@ -240,6 +281,32 @@ REA retains local observation data and removes only URL userinfo credentials, ex - Accessibility structure and roles are retained by default, while names and descriptions require `include_accessibility_text: true`. - Network observations retain method, status, MIME type, size, type, initiator stack location, and complete URLs with userinfo credentials removed. Redirects reported by CDP are attached as ordered `redirects` on the one request ID, with the prior request URL, CDP response URL, method/type, response status/MIME/encoded length, request timestamp, and the next `requestWillBeSent` event timestamp. CDP delivers `redirectResponse` with that next request event, so the latter is an event boundary rather than an exact response timestamp. Redirect response headers and bodies are discarded; hops outside the selected origins are discarded with the request chain. Headers are discarded after an allowlisted projection of length/encoding, structured CSP/Link/policy fields, and untrusted agent hints. Cookies and authorization headers are never retained. - Request/response bodies are not requested or parsed by default. When selected, only allowed-origin JSON media types are read, converted immediately to complete value-free typed paths, types and observation counts, and discarded. Each path segment is either `{ "kind": "property", "name": "raw property name" }` or `{ "kind": "array-element" }`; array-element types include primitives and nested arrays. Literal `*` properties remain distinct from array elements. `Network.getResponseBody` is never sent unless JSON body shape capture is selected. Comparison uses body shapes only when both captures selected them, and compares request/response shapes only where both observations provide those facts. Unavailable or truncated selected coverage makes equivalence unknown; observed differences still report changed. + For normalized passive inputs with complete request inventories on both sides, + request additions and removals remain observable even when selected body shapes, + response metadata, or WebSocket coverage are unavailable. HTTP request and + WebSocket inventory coverage are evaluated separately; complete HTTP coverage + cannot prove a missing WebSocket connection. Limited request inventory still + prevents request-absence claims. This comparison rule does not make live CDP's + attach-limited inventory complete or establish body-shape equality. + An unavailable HTTP resource type also leaves request identity incomplete: + observations with the same method and URL remain possible matches across types, + and identical unknown types cannot establish complete equality. Ambiguous type + groups cannot prove membership or content changes either: a missing type may + redistribute repeated requests between buckets without changing the requests. + Their comparison remains unknown, while independently identified groups retain + proven changes. Different methods/URLs and complete empty peer inventories still + support absence claims. + Relevant dropped-event counts also limit inventory completeness even when no + section is listed as truncated: HTTP uses `network_requests`, while WebSocket + uses `websocket_connections` and `websocket_frames`. Drops in other event kinds + and the aggregate total do not by themselves limit HTTP inventory. Proven + modifications remain reportable despite incomplete inventory. + Repeated identical HTTP content samples establish multiplicity changes only + with complete request inventories; limited inventories retain known content + comparisons without interpreting duplicate sample counts as changes. + WebSocket stream inventories and connection counts likewise require complete + WebSocket coverage. Redirect resource types use only shared known metadata: + an omitted type leaves equality unknown rather than proving a modification. - Console observations with a stack source retain call type, argument types, timestamp, and source location. When console text capture is selected, already-delivered primitive values are retained verbatim; objects, getters, and remote properties are never expanded. - WebSocket observations retain direction, opcode, and payload byte length. When shape capture is selected, text frames are classified as text or complete value-free JSON shape; binary bytes, hashes, prefixes, and raw frames are never retained. - Storage observations always redact values. Key names, IndexedDB names, and cache names require `include_storage_keys`; stable content fingerprints require the additional `include_storage_fingerprints` selection. Cookie fingerprints cover cookies applicable to the current main-frame URL, including its path, rather than every cookie path on that origin. Other storage remains scoped to the origin. Cache bodies above 64 KiB and partial IndexedDB remote objects make fingerprint coverage incomplete, so identical observations remain `unknown`. @@ -264,8 +331,22 @@ validation and resource bounds. Omitted script metadata makes its affected section explicitly truncated. Disallowed-origin entries are filtered before retention. -CDP discovery, connection, and commands wait for the peer or caller cancellation; -a stalled peer can leave an operation waiting. WebSocket messages remain subject +Resource-tree inventories distinguish a valid empty `resources` array from a +missing or malformed collection. Invalid resource entries are omitted with +incomplete coverage, while valid resources from that frame and other frames +remain available. Comparing an incomplete resource inventory cannot prove +addition, removal, or equality; a complete empty inventory can prove removal. +Resource records require a nonempty type and a string MIME type; absent or +malformed required fields are recorded as omissions instead of invented defaults. +An explicitly empty MIME type is valid. The optional content size may be absent; +an invalid supplied size remains unknown with incomplete resource coverage while +the resource's other facts are retained. Zero and nonnegative fractional sizes +are preserved. + +CDP discovery has a 10-second deadline. Each WebSocket handshake and command +response has its own 20-second deadline, separate from the caller-selected +observation window. A stalled handshake or command reports `provider_timeout`; +caller cancellation still reports cancellation. WebSocket messages remain subject to the [64 MiB limit](#request-boundary). REA does not impose a message-count or pending-command quota. On cancellation, it closes only its own WebSocket, waiting up to one second before terminating it. Malformed unsolicited events diff --git a/docs/browser-scenario-contract.md b/docs/browser-scenario-contract.md index 403865be8c..1639faa61b 100644 --- a/docs/browser-scenario-contract.md +++ b/docs/browser-scenario-contract.md @@ -15,11 +15,17 @@ there is no separate origin allowlist or request-interception layer. Environment settings (including service workers blocked by default), empty storage, and a final sanitized URL capture are supplied by default. Set `environment.service_workers` to `allow` when the application requires them. +For attached browsers, `block` rejects captures when the selected page or its +frames are controlled by a service worker or already have a registration, +including after navigation. Existing caller registrations are never removed. +Navigation can run destination code before a newly discovered worker is rejected; +that rejection does not roll back browser effects. REA removes its owned +future-document scripts, restores live frame methods, then disconnects. Add secret declarations, storage seeds, and additional artifact or event capture only when needed. Action, secret, and storage counts are not capped. -Duration, action, and navigation -timeouts remain fixed provider-owned liveness limits; the request does not -accept a caller-controlled `limits` object. +Duration, action, and navigation timeouts remain fixed provider-owned liveness +limits. `capture.limits` controls retained event and network-content budgets; +it does not change those liveness deadlines. ## Minimal launch example @@ -173,12 +179,44 @@ declared action. Each step reports action status, elapsed time, sanitized URLs, event bounds, and independently typed capture states for screenshot, DOM, accessibility, URL, history, and storage. Console, page-error, network, WebSocket, frame, worker, popup, and cancelled-download events are attributed to -steps and all observed events and requested artifacts are returned inline. -There are no event, frame, worker, popup, WebSocket, DOM-node, -accessibility-node, storage-entry, screenshot-count, or cumulative -metadata-byte caps. Requested artifacts and observed events are returned -inline without application-defined size ceilings. Other missing sections -remain explicit and make the capture ineligible for equality claims. +steps. Retained events and requested artifacts are returned inline; omitted +events are counted by kind and identify affected network transactions. +Event and selected network-content retention have caller-selected limits under +`capture.limits`. Omitted values use the defaults below; every supplied value +must be a positive safe integer, and callers can raise or lower a default. +Byte limits are conservative allocation-cost estimates for retained event +values, including UTF-16 storage and JSON expansion, rather than serialized +bytes or a promise about exact process memory. `events.omissions.by_kind` +reports counts and reasons for omitted event kinds. Its +`network_transactions` rows identify affected transaction IDs and omitted +event kinds; `network_transaction_range` identifies the contiguous IDs beyond +the transaction limit. Omission metadata is bounded by the selected +transaction limit. Other missing sections remain explicit and make the capture +ineligible for equality claims. + +The default limits are 10,000 events, 4 MiB estimated allocation for one +event, 16 MiB estimated allocation across retained events, 1,024 network +transactions, 64 KiB of body bytes per transaction phase, 1 MiB estimated +allocation for body redaction/base64, 1 MiB for retained header values, and +1 MiB for network metadata. The same limits are included in the capture +Evidence. Playwright materializes response bodies before REA can inspect their +byte length; `max_network_body_bytes` bounds subsequent retained copies and +does not bound the browser API read itself. A body that exceeds its selected +limit has an explicit `body-size-limit` or `body-allocation-limit` unavailable +state with the observed and allowed values. + +Additional implementation bounds are 16 simultaneous network-content reads, +a 5-second deadline for each provider API read, and 2,048 live event-listener +owners. These are not caller-selectable. The concurrency bound queues admitted +content jobs and does not itself omit observations; a provider read timeout +produces a typed unavailable state, while exceeding listener capacity produces +event-kind omission counts. A preflight avoids body redaction and base64 copies +when the currently available event budget already rejects the body. +Concurrent reads may each pass that check before their events are retained; +their retained-copy work is bounded by 16 times the selected per-body +allocation limit. The Playwright response Buffer itself is materialized before +REA can apply that limit. Requested artifacts and retained events are returned +inline within the declared limits. Attach-mode captures also declare the unavoidable pre-attach event gap. Selected event families are observed on the root page and discovered descendant @@ -249,8 +287,12 @@ Each observed request object receives a capture-scoped `transaction_id`. Requests sharing a URL remain distinct. Redirects link to the preceding transaction only when it was observed; a null predecessor does not establish that no earlier hop existed. Request events require both transaction fields; -an absent ID is malformed capture data, while a null redirect predecessor is -an explicit unknown. +an absent ID is malformed capture data. `redirected_from_transaction_state` +records `root` when Playwright observed no predecessor and `unknown` when a +predecessor exists but was outside the capture. A linked predecessor is carried +by `redirected_from_transaction_id` with no state field. Older captures without +the state field retain the conservative meaning that a null predecessor is +unknown. `scenario.network_content` records the resolved content selection. Comparing captures with different selections reports network evidence as unknown with a @@ -278,14 +320,21 @@ retained representation after redaction. Only completed responses are read. At the end of the final step, observation stops; requests still in flight emit `request-unfinished`, and their observed -responses have explicitly unavailable bodies. This cutoff does not wait for -long-lived streaming connections. Already-started content reads settle before -browser cleanup, with a provider-owned five-second timeout per read and -cancellation support. Failed or unexposed reads retain per-content states and make selected -event coverage incomplete. Body events can arrive during later steps or final -drain; event bounds describe receipt during the step, while the source event -reference establishes the transaction phase. Comparison attributes content to -its source event's step and excludes the sequence reference from normalization. +responses have explicitly unavailable bodies. Already-admitted content reads +drain before browser cleanup. At most 16 reads run at once; queued work is +bounded by the selected network-transaction limit and by successful admission +of its request or response event. Each provider API read has its own five-second +timeout once it starts, so queue wait adds to total drain and cleanup time; this +is not a five-second total capture cutoff. Cancellation interrupts active reads +and queued jobs produce cancelled states without calling their provider APIs. +This drain does not wait for long-lived unfinished streaming connections and +does not refetch unfinished response bodies. Failed or unexposed reads retain +per-content states and make selected event coverage incomplete. Body events can +arrive during later steps or final drain; event bounds describe receipt during +the step, while the source event reference establishes the transaction phase. +Comparison attributes content, like every other event of its transaction, to +the step that sent the request (the receipt step when that request was not +observed) and excludes the sequence reference from normalization. Playwright scenario capture does not supply initiator stacks; passive CDP observation retains its own initiator evidence. @@ -300,7 +349,22 @@ an unchanged claim; observed differences may still prove a changed result. The comparison records its full normalization commitment and SHA-256 digest. Built-in rules exclude step elapsed time and event sequence/index fields, compare screenshots by content digest, and compare DOM/accessibility captures -by normalized text. Optional caller rules are bounded exact-literal +by normalized text. Network events are grouped by transaction without their +capture-local transaction IDs, attributed to the step that sent the request, +and compared as an unordered collection after caller rules apply, because +concurrent responses arrive in any order and can land in a later step; a +redirect retains the complete predecessor ancestry, including each hop's own +request, response outcome, and selected network content. Each transaction's +capture-local IDs, sequence/index fields, and content source-sequence anchor +are removed. Caller rules normalize retained strings before hashing. The +`preserve_redirect_transaction_ancestry` built-in rule fingerprints each hop +as canonical SHA-256 of `{ events, redirected_from }`: `events` is the sorted +list of its normalized event SHA-256 digests, and `redirected_from` is its +predecessor fingerprint or null for a root. Selected transaction fingerprints +are sorted for the step's network comparison. Temporary numeric graph indices +are lookup references only and do not enter the final digest. Missing or cyclic +predecessor evidence yields unknown comparability instead of an unchanged claim. +Other events keep receipt order. Optional caller rules are bounded exact-literal replacements over named artifact kinds. Rules and artifact-kind lists are sorted canonically before application, making the same policy reproducible regardless of input order. Rules operate only on already-redacted durable @@ -318,3 +382,7 @@ selection. Captures with different selections report event evidence as unknown with a recovery instruction, even when no selected family emitted an event. Captures written before this field existed omit it, so their event evidence is also unknown. + +Network normalization also commits the removal of capture-local transaction IDs, +attribution to the requesting step, and unordered transaction comparison. These +built-in rule identities participate in the normalization digest. diff --git a/docs/cli.md b/docs/cli.md index c801bf006d..0922f584a7 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -152,9 +152,13 @@ overrides it. In MCP, pass `provider_id` to `open_binary`: The selected provider remains bound to the session until an explicit switch or close. Provider failures are returned with their original reason. For an `ambiguous` selection error, choose from `details.candidate_ids`; for -`provider_unavailable`, run `rea doctor --provider ID --json` to diagnose the -selected engine. See [task readiness](installation.md#check-readiness-for-your-task) -and [provider selection](mcp-contracts.md#identity-and-discovery). +`provider_unavailable`, follow `message`, `remediation.action`, and the original +reasons in `details.rejections`. For Hopper, Ghidra, IDA, and JADX, run +`rea doctor --provider ID --json` to diagnose the selected engine. Other adapters, +such as `pwntools-elf`, report their configuration requirements directly and are +not accepted by `doctor --provider`. See +[task readiness](installation.md#check-readiness-for-your-task) and +[provider selection](mcp-contracts.md#identity-and-discovery). The session reports work still in progress through `analysis_activity`. A client timeout can end its wait while the provider continues analyzing. @@ -263,7 +267,12 @@ tree matches that commit remains unknown. It honors the source tree's `.gitignor and applies default exclusions for common generated and dependency paths such as `node_modules/`, `dist/`, and `*.log`. Pattern-based exclusions record the matched pattern and whether it came from project, default, caller, or sensitive -path policy. Set `REA_REFERENCE_SECRET_PATTERNS_JSON` to a JSON array of ignore +path policy. Links into excluded paths retain their internal target metadata; +the exclusion explains why the target has no inventory entry, and the inventory +remains partial. Exclusions record `entry_kind` when the importer observed it; +older saved exclusions without that field have unknown kind. An excluded +ancestor proves an internal descendant only when its kind is `directory`. +Set `REA_REFERENCE_SECRET_PATTERNS_JSON` to a JSON array of ignore patterns when you want to mark selected paths as sensitive; those patterns are reported separately as configured-secret exclusions. @@ -287,6 +296,7 @@ Choose the guide for the target and intended interaction: - [Browser observation](browser-observation.md): inspect a selected page through CDP. - [Browser scenarios](browser-scenario-contract.md): run declared interactions and capture their results. - [Electron observation](electron-observation.md): inspect an Electron renderer or capture an application scenario. +- [Electron fuses](electron-fuses.md): read packaged fuse bytes before expecting `--inspect` to reach an Electron main process. - [Node/Electron Inspector](javascript-runtime-observation.md): record script locations and execution contexts. - [Process capture](process-capture.md): run an executable and compare terminal, exit and filesystem observations. @@ -295,17 +305,65 @@ user permissions; consult the chosen guide for host requirements and effects. ## Output and exit status +`rea mcp doctor --json` checks production MCP startup and reports child logs in +`diagnostics`. It drains stderr while retaining at most its first 64 KiB of raw +bytes as text, omitting a final UTF-8 code point if it crosses that byte limit. +`stderr_bytes` counts all observed bytes, including omitted output; +`stderr_truncated` indicates that output exceeded the retention limit. The same +diagnostics appear inside the report when `--full-output` is selected. Log +truncation alone does not change the diagnostic health or exit status. + The default terminal format is TOON. Use `--json` when saving results for a -JSON consumer. `--json` is indented on a terminal and compact when piped or +JSON consumer. Valued output flags (`--format`, `--filter-output`, +`--token-limit`, `--token-offset`) accept `--flag value` or `--flag=value`; a +missing value fails with `MISSING_OUTPUT_VALUE` before the command runs. `--json` is indented on a terminal and compact when piped or redirected, which keeps large results smaller for agents and scripts; pipe through `jq .` for indented files. Output selection and formatting do not change operation status. -| Status | Meaning | -| --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | -| `0` | The operation completed. Its result may include partial evidence, warnings or unresolved questions. | -| `1` | The operation could not complete. Structured output identifies invalid input, permissions, cancellation, timeouts or another failure when available. | -| `128 + N` | Signal `N` ended the process, where the shell or runtime preserves the conventional signal-derived status. | +For `cutter-command`, `--json-output` requests command-specific JSON from +Cutter when available. This is separate from `--format json`, which selects the +CLI result representation; the global `--json` flag selects that same CLI +representation. + +Cutter still accepts deprecated `--json=true` and `--json=false` compatibility +values. When both forms are present, `--json-output` takes precedence in either +argument order. Bare `--json` retains its global meaning of selecting JSON CLI +output formatting. + +| Status | Meaning | +| --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `0` | The operation completed. Its result may include partial evidence, warnings or unresolved questions. | +| `1` | The operation could not complete. Structured output identifies invalid input, permissions, cancellation, timeouts or another failure when available. | +| `128 + N` | Signal-derived status. An unhandled signal may have ended the process, with the shell or runtime preserving the conventional status; a command may instead have handled the signal, waited for its operation and cleanup to settle, and assigned the same status (`130` for SIGINT, `143` for SIGTERM). In the handled case, it may also have printed a structured `cancelled` result. | + +A `cancelled` result does not by itself determine the exit status; it depends on +the command's signal handling. Commands routed through the shared +`withCommandCancellation` handler in the CLI set status `130` for SIGINT or +`143` for SIGTERM after their operation settles. Current source registrations +use that handler for process capture and comparison (`capture-process`, +`compare-process-captures`), JavaScript application analysis (`analyze` when it +selects the JavaScript lane, and `analyze-javascript-application`), selected +browser, web and Electron commands (including `analyze-web-bundle`, +`observe-web-execution`, `capture-browser-scenario` and +`capture-electron-scenario`), and the platform and provider command families +for Android/ADB, Apktool/JEB/Flutter, firmware, PCAP and signal analysis, +WASM/EVM, Go/Rust and binary diagnostics, Frida instrumentation, the Cutter +command, objdump, Rizin and GDB. In these paths, a returned structured +`cancelled` result can be printed before the wrapper assigns the +signal-derived status. This describes the current source routing; it is not a +claim that every command or provider runtime has been independently exercised. + +When the reader of piped output closes early, as `rea ... | head` does, the +CLI stops writing, prints no error and exits `141` (`128 + 13`, the status of a +SIGPIPE-terminated writer). The operation itself may have completed; only its +output was declined. + +`inspect-artifact` also returns a structured cancelled result, but its direct +analysis cancellation path does not set a signal-derived exit status, so it +exits `1`. Setup has its own contract: cancellation exits `0`. Treat the +structured result and process status as separate signals when scripting +cancellation. `setup --dry-run` returns `planned` and exits `0`; a cancelled setup also exits `0`. Setup returns `1` for `needs_confirmation` or `needs_human`. diff --git a/docs/client-configuration.md b/docs/client-configuration.md index 409bc7f196..add6ac0021 100644 --- a/docs/client-configuration.md +++ b/docs/client-configuration.md @@ -129,13 +129,46 @@ and Claude Code reports that autocompact is thrashing. To avoid this: The sizes above come from REA's own `tools/list`. A live Claude Code session against DeepSeek's API is unverified. +For a client whose stdio reader rejects the complete catalog frame, add +`"REA_MCP_OUTPUT_SCHEMA_PROFILE": "off"` to REA's `env` object and restart the +client. This omits only advertised output schemas; REA still validates complete +structured results against its canonical contracts. See [MCP output schema +profiles](mcp-contracts.md#output-schemas). + ## Kimi Code Setup registers `rea` in `~/.kimi-code/mcp.json`, or `$KIMI_CODE_HOME/mcp.json` when set, while preserving unrelated settings and creating a `.rea.backup` before updating an existing file. Kimi Code infers -stdio from the `command` entry, so setup does not add a `type` field. The -registration includes `startupTimeoutMs = 30000` and `toolTimeoutMs = 120000`. +stdio from the `command` entry, so setup does not add a `type` field. New +registrations include `startupTimeoutMs = 30000` and a `toolTimeoutMs` derived +from REA's known provider allowance plus 120 seconds of operation +headroom. The default is 450000 ms, covering the default 330-second Ghidra +startup; an explicit Ghidra startup timeout of 2700000 ms produces a 2820000 ms +tool timeout. The startup allowance follows an explicitly selected Ghidra or +Hopper provider, and otherwise uses the larger known Ghidra/Hopper deadline. +An explicit `REA_JEB_MCP_TIMEOUT_MS` request budget also contributes to that +maximum. For `ida` or `auto` selection, a valid IDA registration's parsed +`timeoutMs` also contributes to that maximum; its default is 300000 ms. +Setup forwards these explicitly selected REA deadlines and the input schema +profile into the registration; unrelated process values are omitted. +This headroom is a practical default, not a guarantee for every analysis +duration. Setup preserves positive timeout values already in the `rea` entry. +You can edit `startupTimeoutMs` and `toolTimeoutMs` there; doctor keeps positive +caller-selected values aligned and reports a non-blocking advisory when a tool +timeout is shorter than the known provider allowance. +Setup selects the compact input-schema advertisement for new or refreshed Kimi +registrations unless their stored configuration or setup environment explicitly +selects `REA_MCP_INPUT_SCHEMA_PROFILE=full` or `compact`. This keeps REA's +canonical tool validation unchanged while avoiding a Moonshot Kimi Code 2.2.0 +failure observed with the full catalog: its request was rejected because a +single tool schema exceeded the provider's size limit. The limit is +provider-specific and not an MCP protocol limit. `rea doctor --client kimi` +reports an older registration with no explicit profile as stale. To restore the +compact profile, run `rea setup --client kimi` or add +`REA_MCP_INPUT_SCHEMA_PROFILE=compact` to the registration's `env`, then restart +Kimi Code. An explicit `full` profile remains respected and can still exceed +Moonshot's limit. Kimi Code also discovers personal skills in `~/.agents/skills` and project-local `.agents/skills`, so no separate skill root is required; setup still installs and verifies the matching skill files through the normal shared-skill flow. @@ -201,8 +234,20 @@ standalone CLI check. ## Codex and Devin -Codex treats an unset or empty `CODEX_HOME` as `~/.codex`. Devin honors -`XDG_CONFIG_HOME` for its `devin/mcp_config.json` on non-Windows systems. +Codex treats an unset or empty `CODEX_HOME` as `~/.codex`. Its MCP server +registration supports `startup_timeout_sec` for connection initialization and +`tool_timeout_sec` for each tool call. New REA registrations use a 30-second +startup timeout and a tool timeout derived from the known provider +allowance plus 120 seconds of operation headroom. This yields 450 seconds for +the default Ghidra allowance; an explicit Ghidra startup timeout of 2700 +seconds yields 2820 seconds. Setup preserves positive caller-selected timeout +values in the REA entry, and doctor reports a non-blocking advisory when a tool +timeout is shorter than the known provider allowance. An explicitly selected +JEB request budget also contributes to the allowance. A configured IDA +registration's parsed `timeoutMs` contributes for `ida` or `auto` selection. +These values are client-side deadlines, so increase them in `~/.codex/config.toml` if a +workload needs more time. Devin honors `XDG_CONFIG_HOME` for its +`devin/mcp_config.json` on non-Windows systems. ## OMP @@ -265,6 +310,21 @@ The user-level MCP paths follow the editor's VS Code layout. Trae's official documentation describes the raw JSON editor without specifying those paths; loading this registration in Trae remains unverified. +## WorkBuddy + +For [WorkBuddy](https://www.workbuddy.cn/docs/workbuddy/From-Beginner-to-Expert-Guide/Function-Description/MCP-Guide), setup writes `mcpServers.rea` into the +global `~/.workbuddy/mcp.json` (a standard `mcpServers` JSON file at the user +home root on every platform). Entries carry `command`, `args`, and `env`; +stdio is the default transport. WorkBuddy discovers personal skills from +`~/.workbuddy/skills` (`%userprofile%\.workbuddy\skills` on Windows), so +setup installs its skill copy there. + +The global MCP path is a plain JSON file; setup preserves every other server +and top-level key when it upserts `rea`. WorkBuddy also reads a project-level +`/.workbuddy/mcp.json`, which setup does not touch: it registers the +user-level file only. Loading this registration and real REA tool calls +through WorkBuddy are verified. + ## ZCode For [ZCode](https://zcode.z.ai/en/docs/mcp-services), setup registers `rea` in the nested `mcp.servers` table of the @@ -279,7 +339,11 @@ the first native server into an empty table, it copies the fallback's `mcpServers` entries into the native table so they keep loading, and leaves the fallback file untouched. Existing REA environment overrides are preserved when refreshing its command. A malformed or unreadable fallback file fails setup instead of -being silently suppressed; repair or remove it and rerun. ZCode disables a +being silently suppressed; repair or remove it and rerun. Uninstall removes an +REA-owned `rea` entry from both the native table and the fallback, backing up +each changed file, so an emptied native table cannot load a leftover fallback +registration; other fallback servers, settings and a foreign `rea` entry are +kept. ZCode disables a server with `enable: false` on its entry; doctor reports such a registration as stale, and rerunning setup removes the flag to repair it. Native configuration uses strict JSON; setup refuses comments, trailing commas and a BOM rather than diff --git a/docs/client-verification.md b/docs/client-verification.md index 38d0bf3996..1eed82281d 100644 --- a/docs/client-verification.md +++ b/docs/client-verification.md @@ -6,6 +6,28 @@ and provider lanes. Each record below states its host, client version, model fixture, and remaining coverage; a native-client run with a loopback model does not establish live model-service acceptance. +## Published Pi code-mode renderer limit + +A separate probe of the published `@earendil-works/pi-codemode` 1.1.0 and 1.2.0 +declaration renderers rendered the full `compare_web_captures` input as +`args: unknown`: its generated TypeScript type exceeds Pi's 16,000-character +default. REA keeps the full advertisement's fields by inlining references when +the shared form would exceed Pi's 32-reference expansion limit; retaining that +shared form would leave later fields unknown instead. The existing +`REA_MCP_INPUT_SCHEMA_PROFILE=compact` profile lets callers stay below +its character limit by advertising reduced nested validation structure, while +REA continues to validate calls against the complete canonical input schema. +Compact applies to the server's advertised inputs and is not an automatic +full-profile fallback; see the [compact input schema profile](mcp-contracts.md#compact-input-schema-profile) +for details of its reduced forms. + +Custom renderer hosts can raise `inputMaxChars` through `renderToolSignature` or +`renderToolSample`; `renderDeclarations` does not expose that override. The +probe exercised the published renderer code, not a native Pi host or model, and +does not establish broader Pi compatibility. See the pinned upstream +[`declarations.ts`](https://github.com/earendil-works/pi/blob/b45e0ebcc5ae2fc42a6f2f23104d10e656e6bedb/packages/codemode/src/declarations.ts) +and [renderer usage notes](https://github.com/earendil-works/pi/blob/b45e0ebcc5ae2fc42a6f2f23104d10e656e6bedb/packages/codemode/README.md#declarations-for-the-model). + ## End-to-end, integration and golden evidence The optional `verify:qwen-client`, `verify:pi-client`, and `verify:hermes-client` @@ -377,6 +399,11 @@ Such a stall leaves the initial page unloaded with empty or placeholder CDP URLs it does not establish permission to capture that page. See [the keyring diagnosis](https://github.com/morluto/rea/issues/1384#issuecomment-6093781686). +On Windows, browser verifier lanes that launch an owned fixture require a +standard, non-elevated caller token. They inspect the verifier caller before +allocating fixture resources and refuse elevated or unobservable tokens; they do +not change caller authority or claim to inspect the browser process token. + `verify:browser` also captures a source-owned noise canvas as a real PNG above 8 MiB through the CLI and stdio MCP, with complete byte/digest parity and real PNG decoding. Its SDK client explicitly permits the larger inline JSON response; diff --git a/docs/electron-fuses.md b/docs/electron-fuses.md new file mode 100644 index 0000000000..4b02f73675 --- /dev/null +++ b/docs/electron-fuses.md @@ -0,0 +1,87 @@ +# Electron fuse state + +`inspect_electron_fuses` / `inspect-electron-fuses` answers whether a packaged +Electron app will honor `--inspect` and `--inspect-brk`, `ELECTRON_RUN_AS_NODE`, +embedded ASAR integrity validation, and loading application code only from +`app.asar`. It reads the fuse wire stored in the Electron binary. It does not +launch the app, modify fuse bytes, or use the network. + +```bash +rea inspect-electron-fuses /Applications/Example.app --json +``` + +The CLI resolves a relative path against the caller's working directory. MCP +requires an absolute filesystem path and preserves that exact selection: + +```json +{ + "name": "inspect_electron_fuses", + "arguments": { "path": "/Applications/Example.app" } +} +``` + +On POSIX hosts, the CLI preserves path components so symlinks are followed +before parent (`..`) steps. Bundle selections may end in `.` or parent steps +through real directories; a terminal parent step through a directory symlink +is rejected. Windows CLI paths retain native drive and separator resolution. + +Check this before `observe_javascript_runtime` with `electron-main`. When +`EnableNodeCliInspectArguments` is disabled or removed, Electron ignores +Inspector flags, and a later target listing can simply show no main-process +target. + +## What is read + +On macOS, pass the `.app` bundle or the Electron Framework binary. Bundle and +framework selection first tries the top-level `Electron Framework` binary, +then `Versions/Current/Electron Framework`, then +`Versions/A/Electron Framework`. Symlinks are followed only while they stay +inside that framework. +On Windows and Linux, pass the main Electron executable. The wire lives in +that executable rather than in `app.asar`. + +The inline Evidence result contains: + +- the selected path, the resolved regular file, its SHA-256, size, and container + format (`mach-o`, `mach-o-universal`, `pe`, or `elf`); +- one slice entry for a thin Mach-O, PE, or ELF file, and one entry per slice of + a universal Mach-O, with file-absolute `sentinel_offset` values; +- `wire_version`, `declared_length`, and ordered fuses `{ index, name, state, raw_byte }`; +- `sentinel_count`, including zero and more than one; +- `runtime_implications`, each marked `relationship: "derivation"` and tied to + the fuse index it came from. + +Version 1 names, in index order, are `RunAsNode`, `EnableCookieEncryption`, +`EnableNodeOptionsEnvironmentVariable`, `EnableNodeCliInspectArguments`, +`EnableEmbeddedAsarIntegrityValidation`, `OnlyLoadAppFromAsar`, +`LoadBrowserProcessSpecificV8Snapshot`, `GrantFileProtocolExtraPrivileges`, +and `WasmTrapHandlers`. An index past that table, or any index on another wire +version, keeps `name: null` and the raw byte. + +Wire bytes are `0x30` disabled, `0x31` enabled, and `0x72` removed. Any other +byte is `unknown`. A removed fuse is not enabled: Inspector flags and +`ELECTRON_RUN_AS_NODE` are not honored, and application code is not restricted +to `app.asar`. On macOS and Windows, a removed or disabled +`EnableEmbeddedAsarIntegrityValidation` byte means embedded ASAR integrity is +not enforced, and an enabled byte means it is enforced. Electron compiles that +validation only on macOS and Windows. On a Linux ELF executable the observed +byte is preserved and the enforcement outcome is `unknown`. + +## Unknown and multiple sentinels + +No sentinel means the fuse state is unknown. It does not mean the fuses are +disabled. A wire that continues past the slice is reported as truncated, with +only the fuse bytes that were present. Two or more sentinels in one slice are +all reported. REA does not pick one. + +Fuse bytes do not prove that the application enables Inspector some other way, +for example through `app.commandLine` or a remote-debugging switch in +application code. + +The selected file must be a regular file, or a bundle that resolves to one. +Symlinks are followed only inside `Electron Framework.framework`. The reader +admits at most 1 GiB, searches with 256 KiB windows, and keeps at most 32 +sentinels per slice. A real nine-fuse, one-slice result is about 1,200 CLI +tokens; a two-slice result is about 1,300 tokens. Exact size varies with path +lengths and limitations. This inspection creates no temporary files and does +not modify the selected bytes. diff --git a/docs/electron-observation.md b/docs/electron-observation.md index eb98a41f35..fb5e9be838 100644 --- a/docs/electron-observation.md +++ b/docs/electron-observation.md @@ -7,6 +7,10 @@ This passive runtime surface is distinct from the target-free static workflow. Static analysis reads a local ASAR or extracted directory without executing application code. Passive observation inspects an existing Electron page. The results keep static inferences and runtime observations distinct. +Packaged fuse state is a separate static read: +[`inspect_electron_fuses`](electron-fuses.md) reports whether this binary will +honor Inspector flags, `ELECTRON_RUN_AS_NODE`, embedded ASAR integrity, and +`app.asar`-only loading. Use the separate [`reconcile_javascript_runtime`](javascript-runtime-reconciliation.md) workflow when both Evidence sets already exist. @@ -112,6 +116,12 @@ If cleanup cannot be verified, the error retains collected observations in `details.partial_observation` and marks their application cleanup `unverified`. Cleanup uncertainty does not authorize signaling unrelated processes. +### Target fuse prerequisites + +Active scenario capture requires that the target Electron executable honors Node CLI inspector flags (`--inspect` or `--remote-debugging-port`) so that Playwright can establish its DevTools protocol session. + +When an Electron executable explicitly disables the `EnableNodeCliInspectArguments` fuse (or removes the capability), `capture-electron-scenario` fails immediately before process launch with an `AnalysisUnsupportedTargetError` ("unsupported_target") rather than waiting for the startup deadline. For targets with disabled inspector fuses, use passive CDP inspection via `inspect_electron_page` or run against an unrestricted development binary with the extracted `app.asar`. + The real verifier exercises compiled CLI and stdio MCP discovery, page/script inspection, static/runtime reconciliation, active observation, and CLI SIGTERM cleanup before checking the IPC, renderer crash/reload, utility/child-process, diff --git a/docs/firmware-analysis.md b/docs/firmware-analysis.md index 1f4ff14d60..152819177f 100644 --- a/docs/firmware-analysis.md +++ b/docs/firmware-analysis.md @@ -9,6 +9,10 @@ No device, root privilege, mount, emulation or execution of extracted programs is needed. Arbitrary firmware bytes do not pass native executable admission; select a returned ELF separately for supported Ghidra analysis. +On Linux, Binwalk and Unblob run as REA-owned processes and require a +procps-compatible `ps` on REA's `PATH`; see the shared +[Linux process prerequisite](installation.md#linux-owned-process-prerequisite). + ## Caller-supplied tools Provide existing tools using absolute executable paths, scoped to your shell: @@ -103,6 +107,23 @@ build's source revision, so that field remains unknown. engine finished its configured extraction without those reported conditions; default skip rules and format coverage still apply. It does not establish complete firmware reconstruction. +- `extraction_gaps` makes those conditions recoverable: a row identifies an + unextracted region or a failed task/stage. A chunk retains its identity and + byte range, plus its original-file range when it belongs to the selected input. + A multi-file stage retains its reported ID and handler with unknown byte ranges; + a standalone task failure keeps unavailable stage identity null. Rows retain exact dependency names from + the producer's missing-dependency reports; an accepted report with no names + remains a failure with unknown dependencies; no executable identity or specific installation step is inferred. Each row's `continuation_id` + references one instruction in `extraction_continuations`, so many regions + share recovery prose without losing any region identity. Instructions use + already configured capabilities where one exists. Every extraction rerun + requires a fresh absent `output_directory`, preserving the prior evidence. + Nested unknown chunks use `inspect_firmware_regions` on the actual published + child rather than repeating the same extraction with unchanged handlers. + Inspection requires separately configured Binwalk (`REA_BINWALK_COMMAND`); + without it, the inspection step is unavailable and the region remains unexamined. A listed gap does not + mean the target is corrupt or that a retained region is empty; REA installs + nothing. - The destination must be absent, absolute, with an existing parent. Publication uses exclusive, symlink-resistant writes and durable digest readback. Existing destinations are never overwritten. This is not an atomic directory rename: @@ -167,6 +188,12 @@ records gzip offset/length and extracted file hashes. Real verification checks CLI/MCP, full-file and selected-range extraction, unknown chunks, depth limits, content parity and unchanged source bytes. No binary fixture is committed. +The optional missing-extractor lane asserts dependency names and the owning +handler in `extraction_gaps`. Those assertions must be run against the pinned +Unblob release to establish real-provider coverage; synthetic producer fixtures +alone do not verify Unblob's report nesting or stage identities. The gap ownership +claims remain unverified against real Unblob until that lane runs. + Additional lanes are explicit, so the base lane stays small: ```sh diff --git a/docs/ghidra-mips.md b/docs/ghidra-mips.md index f010b4f4ca..45969f262b 100644 --- a/docs/ghidra-mips.md +++ b/docs/ghidra-mips.md @@ -52,13 +52,29 @@ section names or the record being inside the initial 4 KiB header probe. Program and section declarations must agree; duplicate, conflicting, truncated or out-of-file records are rejected. Stripped section tables and section-only records are supported. Extended table numbering is explicitly unsupported. -For carved ELFs whose section table extends beyond the captured file, REA -examines every complete captured section header and records the unavailable -headers as a limitation. A valid program-header ABI record can still establish -the supported declaration. Available section declarations must agree with it; -their duplicate, conflicting or unreadable records remain errors. A missing -program-header record does not make ABI declarations in uncaptured sections -absent or supply a default ABI. +GNU `SHT_GNU_ATTRIBUTES` sections are located by numeric section type; section +names never control detection. REA reads at most 4096 payload bytes and parses +format version `A`, the `gnu` vendor's file-scope `Tag_GNU_MIPS_ABI_FP` (4) and +`Tag_GNU_MIPS_ABI_MSA` (8) only. Unknown vendors, scopes or tags, repeated +declarations, malformed lengths, and out-of-file or oversized payloads remain +refusals. A nonzero FP tag must match the separate ABI flags `fp_abi` record; +value 0 (`ANY`) declares no requirement. The [GNU Object Attributes v1 +format](https://sourceware.org/binutils/docs/as/Object-Attributes-v1.html) and +[GNU MIPS tag definitions](https://sourceware.org/binutils/docs/as/GNU-Object-Attributes.html) +are the format references. The MSA tag describes the vector +argument ABI, while the ABI flags ASE field describes instruction-set +extensions; REA does not treat those as interchangeable. The 128-bit MSA +argument ABI and unknown nonzero MSA values remain unsupported. Without a +separate ABI flags record, FP attribute values are admitted only when absent, +`ANY` (0), double precision (1), or FPXX (5), the values covered by the +recordless profile. Parsed values are retained in the MIPS analysis profile. +For carved ELFs whose section table extends +beyond the captured file, REA examines every complete captured section header +and records the unavailable headers as a limitation. The provider refuses an +incomplete ABI declaration inspection even when a program-header ABI record is +valid, because an unavailable section header could carry an unsupported or +conflicting declaration. A missing program-header record does not make ABI +declarations in uncaptured sections absent or supply a default ABI. The field layout follows [`Elf_MIPS_ABIFlags_v0`](https://github.com/bminor/glibc/blob/master/elf/elf.h). diff --git a/docs/ghidra-ppc.md b/docs/ghidra-ppc.md new file mode 100644 index 0000000000..0341583785 --- /dev/null +++ b/docs/ghidra-ppc.md @@ -0,0 +1,90 @@ +# PowerPC ELF analysis through Ghidra + +This lane extends native ELF analysis to 32-bit big-endian PowerPC; it does +not execute PowerPC code. It is the first milestone of +[#1741](https://github.com/morluto/rea/issues/1741). GameCube/Wii DOL +executables, PPC64 and other PowerPC variants are separate follow-ups. + +## Admission boundary + +ELF `EM_PPC` (20) is retained as the provider-neutral `powerpc` family. The +target also retains class, byte order, `e_type` and raw `e_flags`. Recognition +does not mean every provider or PowerPC variant is supported. `EM_PPC64` (21) +is not recognized as this family. + +The Ghidra lane admits ELF32 `ET_EXEC`, big endian. ELF64, little-endian, +relocatable and shared objects are refused with a specific unsupported +constraint. `EF_PPC_EMB` declares an embedded variant and is refused; the +remaining `e_flags` bits only declare relocation conventions and do not change +instruction decoding. + +## Bound language + +Ghidra's ELF loader selects `PowerPC:BE:32:e500:default` for a plain `EM_PPC` +header with `e_flags` 0. e500 is the embedded Book-E variant whose SPE unit +replaces the classic floating-point unit, so it misinterprets ordinary +750-family code. Nothing in the header distinguishes the two. + +The lane therefore imports with `PowerPC:BE:32:default` and compiler spec +`default`, commits that choice in the analysis profile as +`language_selection: "provider-pinned-v1"`, and checks the language Ghidra +reports after import. A different language set through +`REA_GHIDRA_LANGUAGE_ID` (and optionally `REA_GHIDRA_COMPILER_SPEC_ID`) +replaces the lane default and is admitted against the installed language +catalog as usual; for example `PowerPC:BE:32:e500` for genuine e500 code. + +PowerPC header facts are committed in the analysis profile so interpretation +changes invalidate profile-bound snapshots. The support-lane identity is +versioned independently of file identity. Evidence, lifecycle outputs and saved +snapshots preserve the `powerpc` family. Hopper and IDA adapters do not +implicitly gain PowerPC support. Windows Ghidra P0 remains PE x86/x86-64 only. + +## Use + +With a caller-installed Ghidra and compatible JDK on a supported Linux/macOS host: + +```bash +rea analyze /absolute/path/to/ppc.elf --provider ghidra --json +rea function /absolute/path/to/ppc.elf function_name --provider ghidra --json +``` + +MCP uses the existing tools, including `open_binary` with an absolute path and +`provider_id: "ghidra"`. No PowerPC-specific tool catalog or emulator is +introduced. + +## Verification + +The source-owned freestanding fixture is `tests/conformance/c/ppc.c`. The +optional cross-target lane requires a caller-supplied Clang with the PowerPC +target, LLD, GNU `readelf`, Ghidra and its compatible JDK; ordinary host-native +checks do not acquire these tools. `REA_PPC_CLANG` selects Clang and +`REA_PPC_READELF` selects GNU readelf (for example +`/opt/homebrew/opt/binutils/bin/readelf` on macOS). These are test-time +prerequisites, not REA runtime providers. + +```bash +GHIDRA_INSTALL_DIR=/absolute/path/to/ghidra npm run verify:ghidra:ppc +``` + +The lane compiles the fixture for a 750-family core (`-mcpu=750`) and never +executes it. GNU readelf independently supplies header facts and symbol +addresses, compared with the production REA target resolver before provider +startup. Its version, bounded command output and selected tool path are +retained in the verification report. The parser deliberately recognizes only +this fixture's GNU output, rejecting missing, ambiguous or unexpected fields +rather than substituting defaults. + +Existing public MCP operations check Ghidra's **observed** language and +compiler specification and loaded-file hashes; an e500 import fails the lane. +The source-owned `ppc-probe.S` supplies fixed `addi` and conditional-branch +encodings: verification checks bytes, length, immediate and direct destination, +independent of decompiler variable names. Global and string bytes are checked at +independently reported symbol addresses. + +CLI/MCP discovery and decompilation, direct calls, target/profile/Evidence +identity, snapshot round-trip and final process cleanup remain covered. +Pseudocode presence is a liveness check, not proof of semantic equivalence. +Instruction decoding and reader agreement do not establish runtime or complete +floating-point semantics. Parser/profile tests are narrower checks, not +substitutes for this real-provider run. Report the actual lanes run and leave +unexecuted verification pending. diff --git a/docs/ghidra-provider.md b/docs/ghidra-provider.md index fd23dcd073..60434f8ee2 100644 --- a/docs/ghidra-provider.md +++ b/docs/ghidra-provider.md @@ -7,6 +7,8 @@ Executable bytes remain unchanged. Use [installation](installation.md#ghidra) for supported hosts, prerequisites, configuration, startup deadlines, resource controls, and cleanup recovery. +[Linux Ghidra launches](installation.md#linux-owned-process-prerequisite) also +require a procps-compatible `ps` on REA's `PATH` for process ownership checks. [Windows P0](windows-ghidra-p0.md) has its own admission and lifecycle boundary; [NativeAOT observations](ghidra-nativeaot.md) include a built-in read-only PE snapshot parser and a separate optional annotation extension. Check the @@ -151,15 +153,18 @@ overview merely because its source artifact is unchanged. Function extents and instruction completeness refer to Ghidra's analyzed body. An inferred no-return call can terminate that body before subsequent bytes. -Pseudocode, assembly, procedure info, instruction windows, callee and caller -reads, and dossiers report observed terminal call sites in Evidence limitations, +Pseudocode, assembly, procedure info, instruction windows, incoming and outgoing +references, callee and caller reads, and dossiers report observed terminal call sites in Evidence limitations, including resolved callee names, addresses, and FunctionManager flags when -available. Instruction windows -and dossiers also retain these warnings in their result limitations. An -unresolved callee or unestablished terminal-flow basis remains explicit. These -observations qualify the analysis; they do not prove that a no-return flag is -incorrect or recover excluded code. Analysis profiles version this observation -contract so older snapshots cannot replay unqualified results. +available. On x86-32, the same views also report the exact Borland/Delphi-style +FS exception-registration pattern when its handler address falls outside the +function body. This is a static registration observation; it does not establish +handler execution or change the body. Instruction windows and dossiers also +retain these warnings in their result limitations. An unresolved callee or +unestablished terminal-flow basis remains explicit. These observations qualify +the analysis; they do not prove that a no-return flag is incorrect or recover +excluded code. Analysis profiles version this observation contract so older +snapshots cannot replay unqualified results. On non-Windows hosts, the ephemeral import repair clears no-return flags on the known returning external ABI names `malloc`, `__tls_get_addr`, `memcpy`, `memset`, @@ -189,7 +194,7 @@ host/provider acceptance run with only the host compiler. `npm run verify:ghidra:windows` checks source-owned native x86-64 PE applications; `-- --x86` selects x86 and `-- --dll` selects DLLs, and the flags can be combined. -These lanes check all 25 read-only operations, target/snapshot/import SHA-256 linkage, +These lanes check all 26 read-only operations, target/snapshot/import SHA-256 linkage, and project, endpoint, process, and runtime cleanup. The independent native lane checks DACLs, handle admission, cancellation, and Job Object lifecycle. `npm run verify:ghidra:windows:package` packs and installs REA into an isolated diff --git a/docs/go-binaries.md b/docs/go-binaries.md index 155780984b..134a8c74d4 100644 --- a/docs/go-binaries.md +++ b/docs/go-binaries.md @@ -58,6 +58,12 @@ original stable file snapshot, including pointer-backed strings. It does not scan arbitrary byte strings as compiler metadata or interpret a fat Mach-O container as one selected architecture. +For ELF images without a named build-info section, discovery checks each +nonempty writable, non-executable load segment in program-header order. This +includes sectionless PIE images with metadata in a later data segment. Named +sections retain priority, and alignment, file mapping and record validation +remain required. + Historical moduleless Go binaries can store their empty module-string header in declared zero-initialized virtual memory rather than file-backed bytes. The reader recognizes this only when the complete optional module header lies in that memory @@ -78,9 +84,13 @@ host permission denial, changed artifacts and cancellation retain distinct errors. The selected file is never modified; this inspection creates no temporary files, starts no subprocesses and accesses no network endpoint. -Resource guards bound the selected file to 256 MiB, aggregate embedded strings +Resource guards bound the selected file and cumulative header searches to 256 MiB +each, aggregate embedded strings to 1 MiB, structural table decoding to 16 MiB and the complete normalized metadata -report to 8 MiB. Oversized results fail without partial success. MCP also applies +report to 8 MiB. Identical search spans with the same virtual alignment are +searched once; partially overlapping spans consume the cumulative search budget +and fail with a CPU resource constraint when it is exhausted. Oversized results +fail without partial success. MCP also applies its [response transport budget](mcp-contracts.md), which counts both text and structured content; a report within the metadata limit can still require export from retained Evidence. These limits are not operating-system memory or CPU diff --git a/docs/ida-provider.md b/docs/ida-provider.md index 012f75bfb5..e32b9054f2 100644 --- a/docs/ida-provider.md +++ b/docs/ida-provider.md @@ -6,7 +6,9 @@ Choose one lifecycle mode in an upstream MCP JSON registration. Set `REA_IDA_MCP ## Supported upstream versions -Each mode accepts one upstream tool profile, checked against `tools/list` before any analysis runs. Upstream's `main` branch has since moved to 2.0.0, which renamed the GUI tools and dropped `idb_close`, so it serves neither profile. Installing from that branch fails at startup with the missing tool names in the error diagnostics. +Each mode accepts one upstream tool profile, checked against `tools/list` before any analysis runs. The attached profile requires the legacy GUI tool names; upstream 2.0.0 renamed those tools, and REA has not established an equivalent attached-mode lifecycle and ownership contract for that catalog. It therefore fails closed with the missing tool names in the error diagnostics. + +The headless profile is defined by the database-supervisor API, not by an upstream version string. The upstream source snapshot [`c133c385`](https://github.com/mrexodia/ida-pro-mcp/tree/c133c3853faa111a9b00ee615c013b720d0c4acd) includes `idb_close` (introduced by [upstream PR #495](https://github.com/mrexodia/ida-pro-mcp/pull/495)) alongside `idb_open` and `idb_list`. Earlier builds may report the same `2.0.0` version while lacking that tool, so REA checks the live catalog rather than treating a version number or branch name as a compatibility guarantee. | Mode | Upstream profile | Known working | | -------- | ------------------- | ------------------------------------------------------ | @@ -129,19 +131,19 @@ Use a package version that actually includes this adapter; repository main can b {"name":"close_binary","arguments":{}} ``` -Opening an MCP target starts its selected provider session; headless opening waits for auto-analysis. Set the agent's MCP request deadline long enough for the target. The registration's optional `timeoutMs` defaults to 300000 for upstream requests. A freshly opened headless worker can briefly report `status: "busy"` from `server_health`; REA re-reads the health every 100 ms until the worker answers, the caller cancels, or `timeoutMs` has elapsed since the first read. Only expiry while the worker is still busy produces a retryable `provider_timeout`; no further read is started once the deadline has passed. That error names the busy tool of the last observation, or `unknown` when the worker reported none or a value that is not tool-name-shaped. Cancellation during the wait produces `cancelled`. Any other unexpected health payload, including a busy payload with an invalid shape, remains `unreadable_output`. Cancellation drains the current upstream request before cleanup; it does not force-stop GUI analysis. An open request that times out without a completed response retains the private workspace because worker inventory alone cannot prove the request has stopped. `binary_session` reports actual tool availability. Repeated doctor/setup calls are unnecessary when the registration already works. +Opening an MCP target starts its selected provider session; headless opening waits for auto-analysis. Set the agent's MCP request deadline long enough for the target. The registration's optional `timeoutMs` defaults to 300000 for upstream requests. An upstream SDK request that reaches this limit returns `provider_timeout` with its operation and actual request budget. A freshly opened headless worker can briefly report `status: "busy"` from `server_health`; REA re-reads health every 100 ms until the worker answers, the caller cancels, or `timeoutMs` has elapsed since the first read. Each health request is capped by the time remaining in that overall wait, so a stalled read cannot extend the readiness deadline by another full request timeout. Expiry while the worker is still busy produces a retryable `provider_timeout`; no further read is started after the deadline. That error names the busy tool from the last observation, or `unknown` when the worker reported none or a value that is not tool-name-shaped. If a health request itself consumes the remaining wait budget, the same last-busy timeout is returned with the stalled request's captured output. Cancellation during the wait produces `cancelled`. Any other unexpected health payload, including a busy payload with an invalid shape, remains `unreadable_output`. Cancellation drains the current upstream request before cleanup; it does not force-stop GUI analysis. An open request that times out without a completed response retains the private workspace because worker inventory alone cannot prove the request has stopped. `binary_session` reports actual tool availability. Repeated doctor/setup calls are unnecessary when the registration already works. -For guided registration, set `REA_IDA_MCP_CONFIG` before `rea setup --client `. Setup includes that file reference in its reviewed plan and preserves it after approval; it does not copy upstream credentials into the agent configuration. `rea doctor --provider ida --json` validates the registration without starting IDA and explicitly leaves runtime compatibility and target binding for analysis. +For guided registration, set `REA_IDA_MCP_CONFIG` before `rea setup --client `. Setup includes that file reference in its reviewed plan and preserves it after approval; it does not copy upstream credentials into the agent configuration. For `ida` or `auto` selection, Codex/Kimi setup and doctor use the effective client registration's parsed `timeoutMs` (default 300000 ms) when recommending the agent's per-tool deadline. Existing positive client deadlines remain caller choices. `rea doctor --provider ida --json` validates the registration without starting IDA and explicitly leaves runtime compatibility and target binding for analysis. ## Coverage and evidence All profiles support procedure/string inventory and literal or regex search, procedure resolution, pseudocode, assembly, raw function instructions, resolved callees, incoming address references, and a function dossier. Inventories and modern pseudocode/assembly follow upstream pagination. Upstream limits on modern xrefs and callees are retained as explicit truncation limitations. In the modern headless profile, targets that are not proven x86-only use processor-neutral `callgraph` edges instead of the x86 instruction checks in upstream `callees` (mrexodia/ida-pro-mcp#549). REA requests depth zero: the pinned upstream emits the root's outgoing edges without scanning child functions, then REA batches function-name lookup for those addresses. The graph covers recognized function destinations only; imports and other non-function targets remain unknown. A server without `callgraph` returns an empty relationship list with an explicit coverage limitation. Graph and path workflows retain those limitations alongside observed edges. A failed or inconsistent graph reply remains an error. A transport-level truncated preview is rejected rather than reported as complete evidence; the error preserves upstream retrieval guidance. -The modern headless profile also supports `resolve_containing_procedure` and `list_documents`. An address resolves through upstream `lookup_funcs` to the canonical entry of the function that contains it; the body stays unavailable like other IDA identities. When no function contains it, REA reads the database-scoped minimal `survey_binary` and reports `not_in_procedure` for an address inside one of its segments (ends are exclusive) and `outside_segments` otherwise. A missing, truncated or inconsistent survey fails the read instead of guessing a reason. `list_documents` returns the single admitted program path, so `inspect_address_context` and `trace_feature` need no explicit document. `trace_feature` marks a trace truncated, with a residual unknown for that address, when upstream caps the references it returned. The legacy and native profiles do not declare these operations. +All profiles expose `list_documents` as the single target admitted for the session. Attached mode verifies its digest against IDA metadata before returning it; headless mode binds the digest-checked private input. `inspect_address_context`, `trace_feature`, and `binary_overview` can therefore resolve that identity without upstream document-listing tools. Segment enumeration is optional: when the selected profile cannot provide it, `binary_overview` still returns document and procedure/string counts and omits `segments` and `segment_count`; omission means unobserved, not empty. The modern headless profile also supports `resolve_containing_procedure`. An address resolves through upstream `lookup_funcs` to the canonical entry of the function that contains it; the body stays unavailable like other IDA identities. When no function contains it, REA reads the database-scoped minimal `survey_binary` and reports `not_in_procedure` for an address inside one of its segments (ends are exclusive) and `outside_segments` otherwise. A missing, truncated or inconsistent survey fails the read instead of guessing a reason. `trace_feature` marks a trace truncated, with a residual unknown for that address, when upstream caps the references it returned. Attached profiles do not declare address containment resolution. The legacy profile additionally supports direct callers. Its producer may report call-site or interior addresses; REA resolves the reported function symbol to its canonical entry. The modern and native profiles do not prove direct callers from its code/data xref labels, so that capability is unavailable and the dossier marks it unknown. -Complete body ranges, typed reference edges, CFG, referenced data, source locals, comment kinds, and database revision are unavailable. Empty unsupported dossier facets mean **unknown**, not observed absence. `binary_overview`, whole-symbol inventory, mutation, debugging, arbitrary Python forwarding, and requested GUI navigation are outside this adapter's coverage. Check session availability before composing a workflow that needs those facets. +Complete body ranges, typed reference edges, CFG, referenced data, source locals, comment kinds, and database revision are unavailable. Empty unsupported dossier facets mean **unknown**, not observed absence. Whole-symbol inventory, mutation, debugging, arbitrary Python forwarding, and requested GUI navigation are outside this adapter's coverage. Check session availability before composing a workflow that needs those facets. `provider.version` identifies the REA adapter; the profile explicitly records `version_scope: "rea-ida-adapter"`. The MCP handshake's `serverInfo` is separate and does not establish IDA or Python distribution versions. Raw observations preserve producer metadata and output. Attached hashes identify the recorded original input; headless identity uses the digest-verified private copy and observed worker input path. Neither proves source-code recovery or an unmodified analysis database. diff --git a/docs/index.md b/docs/index.md index 9e0912dd10..5acee1b55c 100644 --- a/docs/index.md +++ b/docs/index.md @@ -52,7 +52,7 @@ For runtime instrumentation, see the [Frida guide](./frida-instrumentation.md). | Inspect native functions, references, or types | [Native binaries](native-investigation.md), [Go metadata](go-binaries.md), [Rust dependencies](rust-binaries.md), [offline ELF layout](binary-diagnostics.md), [recorded crashes](recorded-crashes.md) | | Select a native provider or import format | [Ghidra semantics](ghidra-provider.md), [Windows](windows-ghidra-p0.md), [DOS](ghidra-dos.md), [raw bytes](raw-binary.md), [MIPS](ghidra-mips.md), [PSP](ghidra-psp.md), [NativeAOT](ghidra-nativeaot.md), [IDA](ida-provider.md), [JEB](jeb-analysis.md) | | Analyze JavaScript or Electron artifacts | [Static analysis](javascript-artifact-reconstruction.md), [tracing and comparisons](javascript-application-workflows.md), [application graph](javascript-application-graph.md), [source recovery](javascript-recovery.md) | -| Compare static JavaScript with runtime observations | [Runtime attribution](javascript-runtime-observation.md), [reconciliation](javascript-runtime-reconciliation.md), [Electron observation](electron-observation.md) | +| Compare static JavaScript with runtime observations | [Runtime attribution](javascript-runtime-observation.md), [reconciliation](javascript-runtime-reconciliation.md), [Electron observation](electron-observation.md), [Electron fuses](electron-fuses.md) | | Inspect assemblies, packages, or firmware | [.NET](managed-code-analysis.md), [Android](android-analysis.md), [Apktool resources](apktool-resource-analysis.md), [ADB acquisition](adb-device-analysis.md), [Apple](apple-application-analysis.md), [Harmony](harmony-analysis.md), [firmware](firmware-analysis.md), [WASM](wasm-artifacts.md), [EVM](evm-bytecode.md) | | Observe or compare a browser scenario | [CDP observation](browser-observation.md), [scenarios](browser-scenario-contract.md), [runtime evidence](web-runtime.md), [historical captures](web-network-captures.md) | | Trace captured website source | [Source locations](web-source-location.md), [module traces](website-module-trace.md), [script export](website-script-export.md) | diff --git a/docs/installation.md b/docs/installation.md index 3ad6b70717..527014e54f 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -64,11 +64,21 @@ preserves a verified `.rea.backup..recovery` copy of the current bytes. Both snapshots remain available; a failed rollback reports the current recovery copy's path and the restore failure. Setup also retains valid environment overrides in REA's existing server entry, -such as `REA_MCP_INPUT_SCHEMA_PROFILE` and `REA_MCP_MAX_RESPONSE_BYTES`. +such as `REA_MCP_INPUT_SCHEMA_PROFILE`, `REA_MCP_OUTPUT_SCHEMA_PROFILE`, and +`REA_MCP_MAX_RESPONSE_BYTES`. Detected provider settings replace only matching environment keys. Setup and doctor report a malformed REA environment map or invalid runtime setting before setup writes files. Repair the named setting in the client configuration; setup does not silently discard the rest of the map. +This includes malformed loader/exclusion JSON arrays and a Ghidra compiler +specification without its required language identifier. Valid overrides keep +their original strings, including empty Hopper arguments. DeepSeek Harness +`!!js` environment expressions remain unevaluated and preserved; registration +alignment does not verify the runtime values those expressions will produce. +For layered OpenCode files, each file's values are checked before merging, while +setting dependencies are checked against the effective merged environment. +Setup preserves each layer's overrides instead of copying inherited settings +into the file it updates. After setup, REA reports which agents, analysis tools, and workflow files passed its final checks. Restart any agent named in the completion message, then begin @@ -112,6 +122,16 @@ See [client configuration](client-configuration.md) for client-specific roots, profiles, and environment overrides. A mixed selection plans each distinct destination. Selecting the skill without a client uses `~/.agents/skills`. Setup leaves existing skill copies in other locations untouched. +Setup refuses symbolic links at the selected skill directory or its managed +`references` directory, matching uninstall and update's ownership rules. Use a +regular skill directory, or select `--skill=false` to configure the client +registration without installing the bundled skill. +The final selected directories are checked before approval and again before +setup changes any client or Hopper installation. Unselected client destinations +do not block other setup choices. Doctor reports linked or non-directory skill +paths as `unmanaged`, names the affected paths, and provides those alternatives; +an unreadable skill identity remains `unknown`. Uninstall retains and reports +files it cannot safely remove, including their ownership manifest. `rea setup --skill-client claude_code --yes --json` refreshes and verifies the selected skill destination independently of MCP registration selection. The @@ -127,7 +147,152 @@ identity remains available at `identity.catalog`. Setup pins package-runner MCP registrations to the exact installed REA version, installs the matching skill and on-demand references in the same plan, and adds -`startup_timeout_sec = 30` for Codex and Grok Build. +`startup_timeout_sec = 30` for Codex and Grok Build. Codex also receives a +`tool_timeout_sec` derived from known provider startup plus operation headroom. + +## Limit REA to one project or agent profile + +Setup registers REA for the whole client. For Claude Code it writes +`mcpServers.rea` in `~/.claude.json` and installs the bundled skill under +`~/.claude/skills`, so every project that Claude Code opens can call REA. To +offer REA's tools only where they're wanted, use one of the routes below. + +Both routes control which sessions discover REA's MCP server and skill. They +leave shell commands, filesystem access and network access available. An agent +under the same OS account can run `rea` from `PATH` or `npx` and read other +profiles and REA output. For clean-room separation, use an implementation +environment with filesystem and network restrictions that prevent reading or +fetching REA, the target and analysis output, including through shared +installations and package registries. A separate account or container must +enforce those restrictions. + +For Claude Code, check existing default-profile state before choosing either +route. Setup is additive: a previous user-level REA registration and personal +skill remain active in other projects. Back up the default configuration and +REA skill first, then, with `CLAUDE_CONFIG_DIR` unset, remove the REA user +registration with `claude mcp remove --scope user rea` and remove the known +REA-installed `~/.claude/skills/reverse-engineer-anything` directory. Preserve +unrelated registrations and skills. `rea uninstall` applies to other clients +as well, so it is broader than this Claude-only migration. + +**A separate Claude Code profile.** Claude Code keeps settings, skills and session +history under `CLAUDE_CONFIG_DIR`. Setup follows the same +variable: it writes the registration to `$CLAUDE_CONFIG_DIR/.claude.json` and +the skill to `$CLAUDE_CONFIG_DIR/skills`. + +```bash +CLAUDE_CONFIG_DIR=~/.claude-rea npx rea-agents setup --client claude_code +CLAUDE_CONFIG_DIR=~/.claude-rea claude +``` + +A new profile may need authentication. Console sign-ins without an API key are +stored outside `CLAUDE_CONFIG_DIR`; see +[Claude Code authentication](https://code.claude.com/docs/en/authentication#log-in-with-multiple-accounts). +Sessions started without the variable use the default profile's registration, +skill and history. + +Inline assignments apply only to that command. Keep the same prefix for every +profile check, setup rerun, update, or removal; bare commands use the current +environment and can otherwise inspect or recreate default-profile state: + +```bash +CLAUDE_CONFIG_DIR=~/.claude-rea claude mcp list +CLAUDE_CONFIG_DIR=~/.claude-rea npx rea-agents doctor --client claude_code --skill --json +CLAUDE_CONFIG_DIR=~/.claude-rea npx -y rea-agents@latest setup --client claude_code --dry-run +``` + +After reviewing an update plan, rerun that same prefixed setup command without +`--dry-run` and approve its exact changes. Profile removal likewise needs the +prefix on `claude mcp remove --scope user rea` and any `rea uninstall` command; +keep the broader client scope of uninstall in mind. Restart the selected +profile after changing its pinned registration. + +`claude mcp remove` removes only the registration. To remove REA from this +profile without running the broader uninstall flow, first back up the profile's +REA-managed skill directory, then remove +`$CLAUDE_CONFIG_DIR/skills/reverse-engineer-anything` as well. Keep other skills +in the profile intact. + +The commands above use POSIX shell assignment syntax. In native Windows +PowerShell, select the profile for the whole setup/launch/check session and +restore the previous selection afterward: + +```powershell +$previousClaudeConfigDir = $env:CLAUDE_CONFIG_DIR +try { + $env:CLAUDE_CONFIG_DIR = Join-Path $HOME ".claude-rea" + npx rea-agents setup --client claude_code + claude mcp list + npx rea-agents doctor --client claude_code --skill --json + npx -y rea-agents@latest setup --client claude_code --dry-run + # After reviewing the plan, repeat setup without --dry-run and approve it. + claude + # Profile removal, when intended, also belongs inside this selected scope: + # claude mcp remove --scope user rea +} finally { + if ($null -eq $previousClaudeConfigDir) { + Remove-Item Env:CLAUDE_CONFIG_DIR -ErrorAction SilentlyContinue + } else { + $env:CLAUDE_CONFIG_DIR = $previousClaudeConfigDir + } +} +``` + +Use PowerShell for these Windows examples rather than pasting POSIX assignments +into cmd.exe. Keep profile updates, readiness checks and removal inside the +same selected environment; the `finally` block restores the previous profile +or unsets the variable when it was absent. + +**One project.** Install without configuring Claude Code: use the curl +installer's `--no-setup`, or run setup only for other clients. Then register +REA from the project directory at Claude Code's local scope. + +This route uses the default Claude profile. Keep `CLAUDE_CONFIG_DIR` unset for +the entire local lifecycle: registration, backup, listing, version replacement +and removal. In POSIX shells run `unset CLAUDE_CONFIG_DIR`; in PowerShell run +`Remove-Item Env:CLAUDE_CONFIG_DIR -ErrorAction SilentlyContinue` before these +commands. Record an existing selection first if you intend to restore it after +finishing. The configuration path below is the default profile's +`~/.claude.json`; a different active profile uses its own config root instead. + +Register with the variable unset: + + + +```bash +claude mcp add --scope local rea -- npx -y rea-agents@6.5.0 mcp +``` + +On native Windows, use the same project directory and wrap the package-runner +shim so the stdio client can launch it: + +```powershell +claude mcp add --scope local rea -- cmd.exe /d /c npx -y rea-agents@6.5.0 mcp +``` + + + +Local scope stores the entry under that directory's path in `~/.claude.json`, +so other projects don't load it. This route doesn't install the bundled skill. +Avoid `--scope project`, which writes `.mcp.json` into the repository where it +can be committed or copied. `rea doctor` and `rea uninstall` read only the +user-level entry, so they don't report or remove a local-scope registration; +remove it with `claude mcp remove --scope local rea` from the same directory. +`rea update` doesn't refresh it either, so it stays on the version it names. +To move it to a newer release, first back up the project entry in +`~/.claude.json` and record its environment and other metadata. In particular, +preserve caller-selected overrides such as `REA_MCP_INPUT_SCHEMA_PROFILE` and +`REA_MCP_MAX_RESPONSE_BYTES`. Run `claude mcp remove --scope local rea` from the +same directory, then recreate it with the platform-appropriate command above +and the new version. Reapply every recorded environment value with Claude +Code's `mcp add --env NAME=value` options and restore any other retained entry +metadata before restarting the project session. Removal and re-addition do not +preserve these values automatically. Running setup for Claude Code instead +would add a user-level entry for every project. + +Check the separate profile with the prefixed `claude mcp list` above and compare +with the default profile with `CLAUDE_CONFIG_DIR` unset. For local scope, run +`claude mcp list` from the selected project and again from another project. ## Check readiness for your task @@ -136,6 +301,15 @@ configuration without changing them. Without options, it audits every detected registration, the installed skill and optional analysis engines; its overall `healthy` value can be false while your chosen workflow works. +Doctor's `providerInspections` are host-level engine and registration checks, +not a readiness row for every advertised MCP adapter surface. `tools/list` +remains the complete stable catalog. Call `binary_session` with `{}` before +choosing an operation; it works before a target is opened, and +`result.tool_availability` reports per-operation availability, reasons and +client requirements for the current session state. It does not guarantee that +a particular caller-supplied input will be valid. See +[MCP discovery](mcp-contracts.md#identity-and-discovery). + Select the part you want to check: | Task | Readiness check | @@ -157,6 +331,21 @@ on `open_binary`. A `capability_unavailable` failure carries `details.candidate_ids`, while `provider_unavailable` asks you to repair the selected engine. See [provider selection](cli.md#choose-a-provider). +### Linux owned-process prerequisite + +On Linux, REA's shared process supervisor requires a procps-compatible `ps` +on REA's `PATH` for ownership checks before launching a child. Ghidra, JADX, +firmware tools and static JavaScript/Electron analysis use these checks. +Install your distribution's `procps` or `procps-ng` package, or put another +procps-compatible `ps` on `PATH` before +running those workflows. A missing or incompatible command causes a refusal +before the supervised child starts. + +The public error code varies by caller: `capability_unavailable`, +`provider_unavailable` or `execution_failure`. Some provider adapters report +only a generic startup error. Check this prerequisite before following +provider-specific recovery advice. + ## Diagnose, update, and remove `rea doctor --json` is strictly read-only. `rea update` updates only the npm or Bun global installation that owns the running CLI. Source checkouts and package-runner copies must be updated through the mechanism that owns them; a fresh package-runner invocation can use `npx rea-agents@latest` or `bunx rea-agents@latest`. `rea uninstall` removes only REA-owned agent registrations and skill files; `--purge-data` additionally removes REA cache and state paths. @@ -209,9 +398,11 @@ running server or its registration. The npm release checked on **2026-10-11** was **6.4.0**, published from [`a56f7bb6`](https://github.com/morluto/rea/commit/a56f7bb668ab23fec8d33f7af353ac7ed2fe1895). The actual registry package accepted dry-run setup plans for the 17 client IDs -in that release, including Qwen Code, Pi and Hermes. DeepSeek Harness (`dsh`) -is on main and is not in npm 6.4.0; `rea setup --client dsh` against `@latest` -reports an unsupported client until a release containing it is published. +in that release, including Qwen Code, Pi and Hermes. Seven client IDs are +included from 6.5.0 and were absent from the checked npm 6.4.0 package: `droid`, +`dsh`, `kimi`, `qoder`, `trae`, `workbuddy` and `zcode`. +The 6.4.0 registry check does not establish 6.5.0 publication or live DeepSeek +API integration. OpenCode's plan selected `OPENCODE_CONFIG_DIR` even when that directory did not yet exist. These checks used an isolated home on macOS with Node 24; they verify planning and argument @@ -288,11 +479,12 @@ hosted connector; see [its configuration](client-configuration.md#other-client-s | Qoder | `qoder` | | Factory Droid | `droid` | | Trae | `trae` | +| WorkBuddy | `workbuddy` | | ZCode | `zcode` | | DeepSeek Harness | `dsh` | -`dsh` and `kimi` require a release containing them. The registry check above does -not cover them. +`droid`, `dsh`, `kimi`, `qoder`, `trae`, `workbuddy`, and `zcode` require +`rea-agents@6.5.0` or later. The 6.4.0 registry check above does not cover them. See [client configuration and troubleshooting](client-configuration.md) for configuration paths, profiles, trust policies, and schema-limit workarounds. @@ -334,8 +526,53 @@ document for the same target. REA does not currently identify, focus, or reuse that existing GUI document, and the supported launcher exposes no attach or reuse action for it. +REA currently declines saved Hopper database (`.hop`) targets on macOS because +the native `open database` launch route does not run REA's bridge script. To +analyze that program with REA, select its original executable as the target and +open it through Hopper; this starts the supported bridge route. This does not +recover database-only annotations or analysis changes. Saved-database analysis +on Linux continues to use its supported demo-launch route. + +Hopper lease directories must be readable and owned by the current user. +REA validates and secures an opened directory before reserving a lease. If +opening that directory fails with a permission error, the diagnostic identifies +the path and recovery steps while leaving its permissions unchanged. Check +ownership and parent-directory access, then restore owner-only read, write, and +search permissions (`0700`) on a directory you own before retrying. + On supported Linux distributions, approved setup verifies Hopper's official `.deb`, `.rpm`, or Arch package before invoking the native package manager. +The official release feed selects the current public demo without a version +pin. REA verifies the package's detached PGP signature against Hopper's stable +primary fingerprint `7C2D73BAFAF5B9A710A69DA1A26DF815099B1BCD`, then checks +that the signed package and installed launcher agree with `main_exe_hash`. +Rotating signing subkeys remain bound to that primary identity. +The feed row itself is not PGP-signed; doctor labels its display version as +`reported_release_version`, separately from the authenticated launcher identity. + +The approval plan includes the caller-selected cache location. Approved setup +retains the signed archive, public key and release metadata under +`${XDG_CACHE_HOME:-$HOME/.cache}/rea/hopper-releases/`. +An empty `XDG_CACHE_HOME` uses `$HOME/.cache`. Approved setup can repair a +damaged caller-owned entry after authenticating a fresh package; the previous +entry is preserved beside it as `.backup-`. Ordinary doctor +and launches reject damaged evidence and never repair it implicitly. +Approved publications of the same canonical cache entry share a process-backed +lease. Concurrent setup waits without moving another writer's evidence; a +failed replacement restores the previous directory when possible and reports +any retained backup path. Unavailable reads or native inspection tools do not +authorize replacing existing evidence. +Doctor and launches authenticate this evidence for offline use; an editable +digest receipt alone cannot authorize a launcher. Verification reuses at most +eight immutable package proofs per process, keyed by the complete package, +key, signature, package format and expected launcher digest. Cached file reads +still check identity and content. Without retained evidence, inspecting an +existing install requires the matching current official package and network +access. Successful live inspection can be reused within that process after +rehashing the launcher; it does not write a persistent cache or claim that a +previously observed release is still the latest. Older uncached builds require +an approved update while the feed exposes only its current release. + REA runs the supported demo build on a private Xvfb display and selects Hopper's offered demo mode for each analysis session; it does not require the user's desktop display. Unattended package-manager access requires @@ -346,6 +583,13 @@ host mount and the rest of `/tmp` remain unchanged; this fallback never invokes `sudo`. `rea doctor --provider hopper --json` reports the selected strategy and both host and effective mount facts. +When the package-manager step fails, setup reports the failing command and its +captured output, so a package conflict is distinguishable from an authorization +failure. On CachyOS, an unversioned compatibility package such as +`zlib-ng-compat` may conflict with Hopper's versioned `zlib` dependency. Resolve +the package conflict yourself or install Hopper's official package manually, +then rerun `rea setup`. + The supported Linux demo forwards additional launches into its existing application, even across private displays. REA therefore reserves one Linux Hopper application per user across cooperating REA processes. A competing @@ -367,11 +611,19 @@ sessions before removing that recovery directory and retrying. ### Launcher paths and troubleshooting -On macOS, REA uses -`/Applications/Hopper Disassembler.app/Contents/MacOS/hopper` by default. +On macOS, REA checks executable Hopper launchers in this order: the canonical +system application, the canonical application installed by `rea setup` under +`~/Applications`, then other `Hopper*.app` bundles in `/Applications` and +`~/Applications`. The Homebrew cask location is an optional `rea doctor` +diagnostic; to use a Homebrew-installed launcher for CLI/MCP startup, set +`HOPPER_LAUNCHER_PATH` to the path reported by doctor. REA does not run Homebrew +while loading runtime configuration. +Automatic discovery accepts executable regular files, including file symlinks, +and skips directories. On Linux, it prefers executable `/opt/hopper/bin/Hopper`, then executable -`~/.local/share/rea/hopper/bin/Hopper`; if neither exists, the former remains -the diagnostic fallback. `HOPPER_LAUNCHER_PATH` overrides those choices: +`~/.local/share/rea/hopper/bin/Hopper`, followed by `/opt/hopper-v4/bin/hopper` +and `/usr/bin/hopper`; if none exists, the former remains the diagnostic +fallback. `HOPPER_LAUNCHER_PATH` overrides those choices: ```bash export HOPPER_LAUNCHER_PATH=/absolute/path/to/Hopper @@ -415,6 +667,20 @@ If startup reports `Cannot launch Hopper! Connection is invalid.` or waits for bridge readiness, check Hopper for a pending demo, license, or loader dialog. Complete the dialog and retry. If REA reports `cleanup_incomplete`, follow the retained-resource guidance below before reopening the target. + +On macOS, Hopper can show an `Executing Python Script...` overlay while the REA +bridge serves an active MCP connection. Long-lived MCP clients can keep that +connection open across agent turns, so the overlay may remain visible after the +analysis calls finish; by itself, it does not establish a stalled startup. REA +does not automatically idle-close a still-connected target. When the +investigation is finished, export an Evidence bundle first if you need to keep +the session ledger, then call `close_binary`; set `snapshot_path` to save a +provider-neutral analysis snapshot as part of closing. A successful close +shuts down the REA bridge and clears its overlay; REA does not control the +macOS Hopper application lifetime. +The Hopper GUI Cancel button's behavior for this bridge has not been verified; +`close_binary` is the supported cleanup path. + Hopper serializes analysis requests. Cancelling a wait can leave provider work running, which the session reports. Successful decompilation is cached until a relevant rename or comment changes it. @@ -428,12 +694,18 @@ memory has no original file offset. For FAT Mach-O, offsets refer to the origina container file. Results retain Hopper's image-relative offset, the observed slice base, and the source executable path. Loader selection reads the Mach-O container header, including FAT files with a -single architecture. For FAT64, REA validates the architecture table and selected -Mach-O header, prepares a private thin image, and loads it with Hopper's native -Mach-O loader. It checks the entire source's SHA-256 while copying the slice; -source identity and reported offsets still refer to the original container. -An ambiguous architecture subtype requires an explicitly extracted thin image; -REA does not guess. Configured loader arguments remain explicit overrides. +single architecture. For FAT32 and FAT64 containers, REA validates the +architecture table and selected Mach-O header, prepares a private thin image, +and loads it with Hopper's native Mach-O loader instead of Hopper's implicit +FAT slice choice. For arm64 targets, REA ranks subtype bases 0 (plain arm64) and +2 (arm64e), ignoring capability bits for ranking while retaining the raw subtype +for slice identity. These slices' function-entry encodings decode under +`--aarch64`, ahead of Apple's arm64e.x1 variant (base subtype 12), which is +selected only when it is the arm64 family's only candidate. REA checks the +entire source's SHA-256 while copying the slice; source identity and reported +offsets still refer to the original container. Any other ambiguous architecture +subtype requires an explicitly extracted thin image; REA does not guess. +Configured loader arguments remain explicit overrides. FAT64 is distinct from the CPU architecture: FAT32 containers can contain 64-bit executable images. This preparation avoids the Raw Binary loader dialog observed with native FAT64 loading on Hopper 6.1.0-demo. @@ -493,7 +765,13 @@ copying license secrets into logs, or killing unrelated Hopper processes. REA connects to an existing Ghidra installation on Linux x64/arm64, macOS x64/arm64, or experimental Windows x64 P0. It accepts Ghidra 12.1.x and the 64-bit full JDK declared by that installation's -`application.java.min` and `application.java.max`. Current 12.1 releases require +`application.java.min` and `application.java.max`. The JDK's reported platform must also +map to the same native decompiler directory as the host: a JDK running under +emulation (for example x86_64 under Rosetta on Apple silicon) or reporting a +non-host `os.name` fails the doctor +even when it is an otherwise accepted 64-bit full JDK. Select a matching JDK +through `JAVA_HOME`; adding another decompiler in the JVM-selected directory +does not satisfy this host-validated admission rule. Current 12.1 releases require JDK 21 or newer and set no maximum. The bridge is verified with Ghidra 12.1.4 and JDK 21. Each installation must include the native decompiler for the host architecture in `Ghidra/Features/Decompiler/os//` or the corresponding @@ -505,7 +783,7 @@ reported as incompatible instead of being admitted from its directory name, and a header that cannot be read or recognized is reported as unknown. REA does not build or install native tools, or change Gatekeeper quarantine settings. -The adapter exposes 25 read-only operations: thirteen inventory/name/search +The adapter exposes 26 read-only operations: fourteen inventory/name/search operations and twelve function-analysis operations. These cover metadata, decompilation, assembly, resolved calls, typed references, xrefs, function dossiers, instructions, recovered data types, measured load mappings, loaded @@ -553,9 +831,22 @@ installation remains unavailable on Windows. Setup never installs Ghidra, Java, or Python. It preserves valid detected Ghidra/JDK settings in agent registrations. See [Windows Ghidra P0](windows-ghidra-p0.md) for provider diagnostics. +When `GHIDRA_INSTALL_DIR` is unset, doctor and the Ghidra provider also search +`PATH` for `ghidra` or `ghidraRun`, resolve the selected launcher through its +real path, and accept it only when that location belongs to a Ghidra tree with +the application metadata and headless launcher. An explicitly supplied +`GHIDRA_INSTALL_DIR` always takes precedence, including when it is invalid; +REA reports that error instead of silently selecting another installation. +This discovery is read-only and does not execute the Ghidra launcher. + Doctor validates the platform, architecture, Ghidra 12.1.x application version, `support/analyzeHeadless` or `support/analyzeHeadless.bat`, the installation's Java major range, 64-bit JDK bitness, and the presence of `javac`/`javac.exe`. +It also checks that the reported JVM `os.arch` and `os.name` map to the same native decompiler +directory validated for the host; unrecognized reported architectures or a +non-host operating system are refused. LaunchSupport overrides of either +property, including through helper-environment JVM option variables, are +rejected at launch. When Java is found through `PATH`, setup records its observed JDK home so GUI MCP clients do not depend on an incidental shell path. Setup shows every exact environment entry in its plan, writes only after approval, and never downloads, @@ -788,7 +1079,7 @@ On a controlled Windows x64 runner, use `npm run verify:ghidra:windows`. The verifier generates a deterministic native PE fixture from source bytes and requires the Windows native authority before opening the provider. The controls are implemented on main; see the -[release boundary](#released-package-and-main). The lane verifies all 25 admitted +[release boundary](#released-package-and-main). The lane verifies all 26 admitted read-only operations, digest identity, and runtime cleanup. The separate `verify:ghidra:windows:package` lane checks the packed CLI/MCP boundary with an ordinary user. It requires the built Windows native bundle and an existing @@ -813,7 +1104,7 @@ For a client that requires manual configuration, use: "mcpServers": { "rea": { "command": "npx", - "args": ["-y", "rea-agents@6.4.0", "mcp"] + "args": ["-y", "rea-agents@6.5.0", "mcp"] } } } diff --git a/docs/javascript-application-workflows.md b/docs/javascript-application-workflows.md index e9fed813fc..7b581e8ae2 100644 --- a/docs/javascript-application-workflows.md +++ b/docs/javascript-application-workflows.md @@ -60,6 +60,8 @@ provider implicitly. `trace_javascript_semantics` queries the authenticated companion graph without rereading mutable files. A seed may identify a semantic or application node, literal, function fingerprint, property, endpoint, event, or boundary field. +For a recovered parse, coercion, or validation boundary, its field value is the +static operation's method name (for example, `Number`). The query declares backward provenance, forward influence, callers, or ownership and can restrict admitted relation kinds. REA returns every matching seed and the complete reachable subgraph; callers do not choose node, relation, @@ -83,10 +85,22 @@ callee bodies and built-ins such as `Object.assign`, `Reflect.set`, and values. This uncertainty does not prove that a call actually mutated a value. Interprocedural aliasing beyond the supported call-site references remains outside this mutation pass. +For process configuration, only known environment value slots and canonical +argument indexes are treated as primitive when passed to calls. Unknown or +inherited selections can expose a mutable prototype, so later reads of that +configuration container retain uncertainty; earlier top-level reads and unrelated +configuration containers remain independent. Function fingerprints commit normalized syntax, control-flow shape, relation shape, literal sets, arity, and detected effects without using local names or -source offsets. Equal duplicate fingerprints remain ambiguous. Dynamic +source offsets. Function-local directive literals contribute their values while +preserving escape spelling: `"use\x20strict"` differs from `"use strict"`. +Quote style, formatting, and local names remain normalized. Equal duplicate +fingerprints remain ambiguous. +Fingerprints are static candidates, not behavioral-equivalence proofs or a +version-independent identity. Re-analyze affected sources and obtain new seeds +after analyzer corrections; previously retained Evidence remains queryable using +its original component commitments. Dynamic calls/properties/names, parser recovery, unresolved handles, and unsupported API shapes stay explicit unknowns; a missing relation is never reported as proven behavioral absence. diff --git a/docs/javascript-artifact-reconstruction.md b/docs/javascript-artifact-reconstruction.md index 090902ff89..e66d52d724 100644 --- a/docs/javascript-artifact-reconstruction.md +++ b/docs/javascript-artifact-reconstruction.md @@ -206,6 +206,9 @@ code unit, matching the original bytes identified by the artifact digest. Plain `.ts` artifact sources use TypeScript syntax without JSX, including angle-bracket type assertions. `.tsx` and `.jsx` retain JSX parsing. +JavaScript sources use JavaScript grammar with JSX, never TypeScript syntax, +so valid JavaScript that TypeScript reads differently keeps its JavaScript +meaning. Relative TypeScript imports can name their emitted `.js`, `.mjs`, or `.cjs` counterparts while resolving to inventoried source files. This substitution @@ -220,7 +223,10 @@ Selected bounded text is then parsed as inert data to recover: - Webpack and Rspack chunk registrations and module factories represented as AST literals; - static imports, dynamic imports, CommonJS `require` calls, workers, and - service workers; + service workers (only `new Worker(new URL(path, import.meta.url))` resolves + relative to the calling module; string worker and service-worker paths use + the document base URL or Node's working directory, which static analysis + does not establish, so they keep their expression and stay unresolved); - route, network endpoint, storage, and vendor-marker observations; - local source-map declarations and original source names and content digests; @@ -273,6 +279,12 @@ RPC client syntax also contributes network endpoint candidates, in both `OperationDefinition` document object. A graphql-tag `loc.source.body` copy is not reported twice. Anonymous operations are not reported. +Literal link options and precompiled document fields follow object property +order: the final definition owns the key. A later spread, dynamic computed key, +method, or accessor can make a field unknown, so an overwritten literal is not +reported as the endpoint or operation. A final explicit property after a spread +or dynamic key remains usable. + A procedure path is the server-side procedure name, not a resolved URL; the transport URL depends on the link configuration and batching. When the proxy is reached through a member expression, such as a webpack module binding @@ -282,9 +294,14 @@ suffix. Utility calls such as `useUtils().invoice.list.fetch()`, `useQueries`, and links whose callee was renamed by a minifier are not recognized. Each recovered bundle module retains the exact factory-source digest. A complete -bounded AST also receives a `babel-ast-v1` structural fingerprint that ignores +bounded AST also receives a `babel-ast-v2` structural fingerprint that ignores ordinary identifier names while retaining syntax, literals, operators, object -keys, and member properties. If fingerprint construction reaches its structural +keys, and member properties. Computed member identifiers retain their access +syntax without treating a local variable's spelling as a fixed property; template +literal contents remain part of the fingerprint. Version comparison keeps the +algorithm in its matching key, so retained `babel-ast-v1` fingerprints cannot match +these new fingerprints. Re-analyze both snapshots to compare their new structure. +If fingerprint construction reaches its structural bound, the fingerprint is unavailable and its status is `truncated`; a bounded prefix is never presented as a complete stable fingerprint. @@ -615,7 +632,19 @@ href are read without surrounding whitespace or embedded tabs and newlines, and their percent-encoded path bytes are decoded; encoded dot and separator bytes are still rejected. CommonJS module lookups retain extension and directory resolution. +ESM, HTML, and URL references apply URL dot segments to empty path segments: +`./assets//../app.js` selects `assets/app.js`, while a remaining empty segment +such as `./assets//app.js` stays unavailable. CommonJS keeps filesystem path +semantics and selects `app.js` for the same text. Unresolved HTML references retain their declaration, source range, and resolution reason in the renderer observations. HTML script source ranges follow the HTML parser across LF, CRLF, and bare CR line endings, preserving UTF-16 columns. +Inline JavaScript in local HTML, meaning classic or module `', []], @@ -35,17 +38,21 @@ it.each([ ).toEqual(expected); }); -it("retains exact opening-tag offsets rather than the script body", () => { +it("retains exact opening-tag coordinates rather than the script body", () => { const opening = ''; const prefix = "\n\n"; const text = `${prefix}${opening}ignored body`; expect(htmlArtifactReferences(text)).toEqual({ baseHref: null, + inlineScripts: [], + inlineScriptCount: 0, scripts: [ { scriptPath: "actual.js", - startOffset: prefix.length, - endOffset: prefix.length + opening.length, + location: { + start: { line: 3, column: 0 }, + end: { line: 4, column: ' title=">" src=actual.js>'.length }, + }, }, ], }); @@ -68,3 +75,71 @@ it("traverses deep valid document content without recursive stack growth", () => htmlArtifactReferences(text).scripts.map(({ scriptPath }) => scriptPath), ).toEqual(["actual.js"]); }); + +it.each([ + ["", ["classic"]], + ['', ["classic"]], + ['', ["classic"]], + ['', ["classic"]], + ['', ["classic"]], + ['', ["classic"]], + ['', ["module"]], + // Only the JavaScript MIME check strips whitespace; `module` must match. + ['', []], + ['', []], + ["", ["classic"]], + // A src attribute, even an empty one, means the browser ignores the body. + ['', []], + ['', []], + ["", []], + ['', []], + ['', []], + ['', []], + ['', []], + ['', []], + ['', []], + ["", []], + ["", []], + ["", []], +])("classifies inline JavaScript in %s", (text, expected) => { + expect( + htmlArtifactReferences(text).inlineScripts.map( + ({ scriptType }) => scriptType, + ), + ).toEqual(expected); +}); + +it("retains exact inline script body coordinates, including unclosed scripts", () => { + const prefix = "\n

after

`; + expect(htmlArtifactReferences(closed).inlineScripts).toEqual([ + { + scriptType: "classic", + location: { start: { line: 2, column: 8 }, end: { line: 4, column: 0 } }, + }, + ]); + const unclosed = `${prefix}${body}`; + expect(htmlArtifactReferences(unclosed).inlineScripts).toEqual([ + { + scriptType: "classic", + location: { start: { line: 2, column: 8 }, end: { line: 4, column: 0 } }, + }, + ]); +}); + +it("counts every inline body but retains bounded locations", () => { + const count = 150_000; + const text = "".repeat(count); + const { inlineScripts, inlineScriptCount } = htmlArtifactReferences(text); + expect(inlineScriptCount).toBe(count); + expect(inlineScripts).toHaveLength(HTML_INLINE_SCRIPT_LOCATION_LIMIT); + const column = 18 * (HTML_INLINE_SCRIPT_LOCATION_LIMIT - 1) + 8; + expect(inlineScripts.at(-1)).toEqual({ + scriptType: "classic", + location: { + start: { line: 1, column }, + end: { line: 1, column: column + 1 }, + }, + }); +}); diff --git a/src/domain/javascript/htmlArtifactReferences.ts b/src/domain/javascript/htmlArtifactReferences.ts index fa9945e365..10a52f21a9 100644 --- a/src/domain/javascript/htmlArtifactReferences.ts +++ b/src/domain/javascript/htmlArtifactReferences.ts @@ -4,23 +4,99 @@ import { type DefaultTreeAdapterTypes, } from "parse5"; +import type { + JavaScriptSourcePoint, + JavaScriptSourceRange, +} from "./javascriptStaticAnalysisTypes.js"; + interface HtmlScriptReference { readonly scriptPath: string; - readonly startOffset: number; - readonly endOffset: number; + /** The opening tag's range in parse5 coordinates. */ + readonly location: JavaScriptSourceRange; } -/** Extract inert HTML script references and the first document base href. */ +/** JavaScript written directly in a script element instead of loaded by URL. */ +export interface HtmlInlineScript { + readonly scriptType: "classic" | "module"; + /** The body's range, between the opening and closing tags. */ + readonly location: JavaScriptSourceRange; +} + +/** + * Inline bodies whose exact locations are retained per HTML file. Every body + * is still counted; locations beyond this bound are reported as omitted. + */ +export const HTML_INLINE_SCRIPT_LOCATION_LIMIT = 100; + +// The JavaScript MIME type essences from the WHATWG MIME Sniffing standard. +const JAVASCRIPT_MIME_TYPES = new Set([ + "application/ecmascript", + "application/javascript", + "application/x-ecmascript", + "application/x-javascript", + "text/ecmascript", + "text/javascript", + "text/javascript1.0", + "text/javascript1.1", + "text/javascript1.2", + "text/javascript1.3", + "text/javascript1.4", + "text/javascript1.5", + "text/jscript", + "text/livescript", + "text/x-ecmascript", + "text/x-javascript", +]); + +/** + * Classify a script element's type as HTML "prepare the script element" does. + * Only a type attribute is whitespace-stripped, and only for the JavaScript + * MIME check; `module` must match the type string exactly, ignoring ASCII + * case. Import maps, speculation rules and data blocks are not JavaScript. + */ +const inlineScriptType = ( + attrs: readonly { readonly name: string; readonly value: string }[], +): HtmlInlineScript["scriptType"] | null => { + const type = attrs.find(({ name }) => name === "type"); + const language = attrs.find(({ name }) => name === "language"); + const typeString = + type !== undefined + ? type.value + : language !== undefined && language.value !== "" + ? `text/${language.value}` + : ""; + if (typeString === "") return "classic"; + const mimeCandidate = + type === undefined + ? typeString + : typeString.replace(/^[\t\n\f\r ]+|[\t\n\f\r ]+$/gu, ""); + if (JAVASCRIPT_MIME_TYPES.has(asciiLowercase(mimeCandidate))) + return "classic"; + return asciiLowercase(typeString) === "module" ? "module" : null; +}; + +const asciiLowercase = (value: string): string => + value.replace(/[A-Z]/gu, (letter) => letter.toLowerCase()); + +/** + * Extract inert HTML script references, inline JavaScript bodies, and the + * first document base href. + */ export const htmlArtifactReferences = ( text: string, ): { readonly baseHref: string | null; readonly scripts: readonly HtmlScriptReference[]; + /** The first inline bodies in document order, up to the location limit. */ + readonly inlineScripts: readonly HtmlInlineScript[]; + readonly inlineScriptCount: number; } => { const pending: DefaultTreeAdapterTypes.Node[] = [ parse(text, { sourceCodeLocationInfo: true }), ]; const scripts: HtmlScriptReference[] = []; + const inlineScripts: HtmlInlineScript[] = []; + let inlineScriptCount = 0; let baseHref: string | null = null; while (pending.length > 0) { const node = pending.pop(); @@ -39,10 +115,20 @@ export const htmlArtifactReferences = ( if (src !== undefined && src.value !== "" && location !== undefined) { scripts.push({ scriptPath: src.value, - startOffset: location.startOffset, - endOffset: location.endOffset, + location: { + start: sourcePoint(location.startLine, location.startCol), + end: sourcePoint(location.endLine, location.endCol), + }, }); } + const inline = src === undefined ? inlineScriptBody(node) : undefined; + const scriptType = + inline === undefined ? null : inlineScriptType(node.attrs); + if (inline !== undefined && scriptType !== null) { + inlineScriptCount += 1; + if (inlineScripts.length < HTML_INLINE_SCRIPT_LOCATION_LIMIT) + inlineScripts.push({ scriptType, location: inline }); + } } } // Template content belongs to a separate inert document fragment. Only @@ -51,5 +137,33 @@ export const htmlArtifactReferences = ( for (const child of node.childNodes.toReversed()) pending.push(child); } } - return { baseHref, scripts }; + return { baseHref, scripts, inlineScripts, inlineScriptCount }; }; + +/** Locate a script element's non-blank body; browsers ignore it when src is set. */ +const inlineScriptBody = ( + node: DefaultTreeAdapterTypes.Element, +): JavaScriptSourceRange | undefined => { + const startTag = node.sourceCodeLocation?.startTag; + if (startTag === undefined) return undefined; + let blank = true; + let end: DefaultTreeAdapterTypes.TextNode["sourceCodeLocation"]; + for (const child of node.childNodes) { + if (!defaultTreeAdapter.isTextNode(child)) continue; + if (blank && /\S/u.test(child.value)) blank = false; + end = child.sourceCodeLocation ?? end; + } + // An unclosed script's element range ends at its opening tag, so the body's + // own text location bounds it. + if (blank || end == null) return undefined; + return { + start: sourcePoint(startTag.endLine, startTag.endCol), + end: sourcePoint(end.endLine, end.endCol), + }; +}; + +/** parse5 lines and UTF-16 columns are one-based; ranges use zero-based columns. */ +const sourcePoint = (line: number, column: number): JavaScriptSourcePoint => ({ + line, + column: column - 1, +}); diff --git a/src/domain/javascript/javascriptApplicationChangeGraph.ts b/src/domain/javascript/javascriptApplicationChangeGraph.ts index 134fff5200..3935038447 100644 --- a/src/domain/javascript/javascriptApplicationChangeGraph.ts +++ b/src/domain/javascript/javascriptApplicationChangeGraph.ts @@ -180,7 +180,12 @@ const comparisonEvidence = ( executable_sha256: null, }, coverage: complete - ? { status: "complete", truncated: false, omitted_count: 0, limits: [] } + ? { + status: "complete", + truncated: false, + omitted_count: 0, + limits: sourceCoverageLimits(input), + } : { status: "partial", truncated: @@ -210,7 +215,7 @@ const changeGraphCoverage = ( status: "complete", truncated: false, omitted_count: 0, - limits: [], + limits: sourceCoverageLimits(input), }; return { status: "partial", diff --git a/src/domain/javascript/javascriptApplicationEvidenceSchemas.ts b/src/domain/javascript/javascriptApplicationEvidenceSchemas.ts index 65bce595b5..73f3798952 100644 --- a/src/domain/javascript/javascriptApplicationEvidenceSchemas.ts +++ b/src/domain/javascript/javascriptApplicationEvidenceSchemas.ts @@ -6,7 +6,7 @@ const boundedTextSchema = z.string().min(1); const boundedKeySchema = z.string().min(1); const unsafePathSegments = new Set(["", ".", ".."]); const isSafeRelativePath = (path: string): boolean => { - if (path.startsWith("/") || path.includes("\\")) return false; + if (path.startsWith("/")) return false; return path.split("/").every((part) => !unsafePathSegments.has(part)); }; const relativePathSchema = z diff --git a/src/domain/javascript/javascriptApplicationVersionComparison.test.ts b/src/domain/javascript/javascriptApplicationVersionComparison.test.ts index d73dceaf6b..fdb36b5dc5 100644 --- a/src/domain/javascript/javascriptApplicationVersionComparison.test.ts +++ b/src/domain/javascript/javascriptApplicationVersionComparison.test.ts @@ -143,7 +143,7 @@ const nativeEvidence = (sha256: string | undefined, ordinal: number) => }, ); -describe("application version comparison result", () => { +describe("application version comparison Evidence links", () => { it("links every exact digest observation only to its paired or unmatched item", () => { const left = buildGraph(LEFT_SHA, 1); const right = buildGraph(RIGHT_SHA, 2); @@ -227,7 +227,9 @@ describe("application version comparison result", () => { ].sort(), ); }); +}); +describe("application version comparison graph ownership", () => { it("returns a sealed change graph that passes complete validation", () => { const result = compare(); expect( @@ -243,6 +245,58 @@ describe("application version comparison result", () => { ); }); + it("retains deduplicated complete input limits on the change graph and inferred edge", () => { + const leftLimits = [ + { name: "modules", value: 100, unit: "items" as const }, + ]; + const rightLimits = [ + { name: "source-bytes", value: 4096, unit: "bytes" as const }, + { name: "modules", value: 100, unit: "items" as const }, + ]; + const withLimits = ( + graph: JavaScriptApplicationGraph, + limits: JavaScriptApplicationGraph["coverage"]["limits"], + ): JavaScriptApplicationGraph => { + const { graph_id: _graphId, ...input } = graph; + return createJavaScriptApplicationGraph({ + ...input, + coverage: { ...graph.coverage, limits }, + }); + }; + const result = compareJavaScriptApplicationVersions({ + left: { + evidenceId: `ev_${"1".repeat(64)}`, + rootArtifactSha256: LEFT_SHA, + graph: withLimits(buildGraph(LEFT_SHA, 1), leftLimits), + }, + right: { + evidenceId: `ev_${"2".repeat(64)}`, + rootArtifactSha256: RIGHT_SHA, + graph: withLimits(buildGraph(RIGHT_SHA, 2), rightLimits), + }, + leftNativeEvidence: [], + rightNativeEvidence: [], + }); + const expectedLimits = [leftLimits[0]!, rightLimits[0]!]; + const changed = result.graph.edges.find( + ({ relation }) => relation === "changed_from", + ); + + expect(changed).toBeDefined(); + expect(result.graph.coverage).toMatchObject({ + status: "complete", + truncated: false, + omitted_count: 0, + limits: expectedLimits, + }); + expect(changed?.evidence.coverage).toMatchObject({ + status: "complete", + truncated: false, + omitted_count: 0, + limits: expectedLimits, + }); + }); + it("does not reuse a change graph without an owned proof", () => { const result = compare(); const copy = structuredClone(result.graph); diff --git a/src/domain/javascript/javascriptApplicationVersionComparisonSchemas.ts b/src/domain/javascript/javascriptApplicationVersionComparisonSchemas.ts index d9d54449ed..bd2c1a1678 100644 --- a/src/domain/javascript/javascriptApplicationVersionComparisonSchemas.ts +++ b/src/domain/javascript/javascriptApplicationVersionComparisonSchemas.ts @@ -70,7 +70,9 @@ export const compareApplicationVersionsInputSchema = z }) .superRefine(validateApplicationVersionIdentities); -const candidateNodesSchema = z.tuple([nodeIdSchema]).rest(nodeIdSchema); +const candidateNodesSchema = z + .tuple([nodeIdSchema], nodeIdSchema) + .check(z.minLength(1)); const exactMatchSchema = z.strictObject({ status: z.literal("matched"), basis: z.enum([ diff --git a/src/domain/javascript/javascriptArtifactFiles.ts b/src/domain/javascript/javascriptArtifactFiles.ts index 819c6ceae8..c7013bc331 100644 --- a/src/domain/javascript/javascriptArtifactFiles.ts +++ b/src/domain/javascript/javascriptArtifactFiles.ts @@ -1,4 +1,9 @@ import type { JavaScriptSourceSelection } from "./javascriptSourceSelection.js"; +import type { ArtifactOccurrence } from "../artifactGraph.js"; + +/** Whether an inventoried path has a source extension read by the analyzer. */ +export const isJavaScriptSourcePath = (path: string): boolean => + /\.(?:cjs|cts|mjs|mts|js|jsx|ts|tsx)$/iu.test(path); /** Relevant file categories projected from the complete artifact inventory. */ export type JavaScriptArtifactFileKind = @@ -29,6 +34,33 @@ export interface JavaScriptArtifactFile { }; } +/** + * One inventoried regular file not backed by analyzed source bytes. Its inventory + * occurrence remains available even when content is opaque or unavailable. + */ +export interface JavaScriptUnanalyzedArtifactFile { + readonly path: string; + readonly container_sha256: string; + readonly sha256: string | null; + readonly bytes: number | null; + readonly inventory_artifact_id: string | null; + readonly inventory_occurrence_id?: string; + readonly kind?: JavaScriptArtifactFileKind; + readonly artifact_kind?: ArtifactOccurrence["artifact_kind"]; + readonly hash_status?: + | "verified" + | "mismatched" + | "not-hashed" + | "unavailable"; + readonly limitations?: readonly string[]; +} + +/** Borrowed inventory lookup; metadata is materialized only for requested paths. */ +export interface JavaScriptUnanalyzedArtifactLookup { + get(path: string): JavaScriptUnanalyzedArtifactFile | undefined; + hasDirectory(path: string, containerSha256: string | undefined): boolean; +} + /** One filesystem-backed ASAR nested beneath a directory input. */ export interface JavaScriptArtifactContainer { readonly path: string; @@ -40,6 +72,10 @@ export interface JavaScriptArtifactContainer { /** Deterministic relevant-file projection plus explicit omissions. */ export interface JavaScriptArtifactFileSet { readonly files: readonly JavaScriptArtifactFile[]; + /** Borrowed inventory lookup for regular files the source projection does not read. */ + readonly unanalyzed_files?: JavaScriptUnanalyzedArtifactLookup; + /** Relevant inventoried files whose bytes were unavailable. */ + readonly unavailable_relevant_files: readonly JavaScriptUnanalyzedArtifactFile[]; readonly containers: readonly JavaScriptArtifactContainer[]; readonly text_bytes_read: number; readonly invalid_utf8_files: number; diff --git a/src/domain/javascript/javascriptAstFingerprint.test.ts b/src/domain/javascript/javascriptAstFingerprint.test.ts index ddbb85adec..4a12ac8e3c 100644 --- a/src/domain/javascript/javascriptAstFingerprint.test.ts +++ b/src/domain/javascript/javascriptAstFingerprint.test.ts @@ -2,6 +2,7 @@ import { parse } from "@babel/parser"; import { describe, expect, it } from "vitest"; import { fingerprintJavaScriptAst } from "./javascriptAstFingerprint.js"; +import { analyzeJavaScriptStaticSource } from "./javascriptStaticAnalysis.js"; describe("JavaScript AST fingerprints", () => { it("includes complete literal values in the fingerprint", () => { @@ -14,3 +15,62 @@ describe("JavaScript AST fingerprints", () => { ); }); }); + +it.each(["obj[key]", "obj?.[key]"])( + "ignores a local computed key binding in %s", + (expression) => { + const left = parse(`function read(obj, key) { return ${expression}; }`); + const renamed = expression + .replace("obj", "value") + .replace("key", "selector"); + const right = parse( + `function read(value, selector) { return ${renamed}; }`, + ); + expect(fingerprintJavaScriptAst(left)).toBe( + fingerprintJavaScriptAst(right), + ); + }, +); + +it("distinguishes computed reads from fixed named members", () => { + expect(fingerprintJavaScriptAst(parse("obj[key]"))).not.toBe( + fingerprintJavaScriptAst(parse("obj.key")), + ); + expect(fingerprintJavaScriptAst(parse("obj.key"))).not.toBe( + fingerprintJavaScriptAst(parse("obj.other")), + ); + expect(fingerprintJavaScriptAst(parse('obj["key"]'))).not.toBe( + fingerprintJavaScriptAst(parse('obj["other"]')), + ); +}); + +it.each([ + ["obj[`left`]", "obj[`right`]"], + ["obj[``]", "obj[`right`]"], + ["obj[true]", "obj[false]"], + ["obj[null]", "obj[false]"], + ['obj["" ]', 'obj["right"]'], + ["obj[`prefix-${key}`]", "obj[`changed-${key}`]"], +])("retains computed literal contents in %s", (left, right) => { + expect(fingerprintJavaScriptAst(parse(left))).not.toBe( + fingerprintJavaScriptAst(parse(right)), + ); +}); + +it("keeps renamed computed-key bundle modules structurally matchable", () => { + const source = (name: string) => `globalThis.webpackChunkApp.push([[1], { + 17(module, exports) { const ${name} = selectKey(); exports.read = value[${name}]; } + }]);`; + const before = analyzeJavaScriptStaticSource(source("key")) + .bundler_registrations[0]?.modules[0]; + const after = analyzeJavaScriptStaticSource(source("selector")) + .bundler_registrations[0]?.modules[0]; + expect(before).toBeDefined(); + expect(after).toBeDefined(); + expect(before?.structural_fingerprint_algorithm).toBe("babel-ast-v2"); + expect(before?.source_sha256).not.toBe(after?.source_sha256); + expect(before?.structural_fingerprint_sha256).toMatch(/^[a-f0-9]{64}$/u); + expect(before?.structural_fingerprint_sha256).toBe( + after?.structural_fingerprint_sha256, + ); +}); diff --git a/src/domain/javascript/javascriptAstFingerprint.ts b/src/domain/javascript/javascriptAstFingerprint.ts index c85700da92..1dee83fa05 100644 --- a/src/domain/javascript/javascriptAstFingerprint.ts +++ b/src/domain/javascript/javascriptAstFingerprint.ts @@ -7,6 +7,7 @@ import { traverseJavaScriptAst } from "./javascriptSemanticTraversal.js"; import { compareUnicodeCodePoints } from "../unicodeCodePointOrder.js"; import { propertyName, + runtimeFunctionParameters, semanticStaticPropertyKey, } from "./javascriptAstValues.js"; import { stringValue } from "./javascriptStaticAnalysisHelpers.js"; @@ -38,15 +39,13 @@ export const collectJavaScriptExports = ( bindings?: JavaScriptSemanticAnalysisState, ): StaticExports => { const exports = new Set(); + const parameter = t.isFunction(node) + ? runtimeFunctionParameters(node)[1] + : undefined; const factoryExports = - bindings !== undefined && - t.isFunction(node) && - t.isIdentifier(node.params[1]) - ? resolveSemanticBindingState( - bindings, - node.params[1], - node.params[1].name, - )?.bindingId + bindings !== undefined && t.isFunction(node) && t.isIdentifier(parameter) + ? resolveSemanticBindingState(bindings, parameter, parameter.name) + ?.bindingId : undefined; const isExportsAlias = (reference: t.Node | null): boolean => { if (!t.isIdentifier(reference)) return false; @@ -85,13 +84,19 @@ const semanticTokens = (node: t.Node): string[] => { tokens.push(`boolean:${String(node.value)}`); else if (t.isRegExpLiteral(node)) tokens.push(`regexp:${node.pattern}/${node.flags}`); + else if (t.isTemplateElement(node)) + tokens.push(`template:${JSON.stringify(node.value)}`); else if (t.isBinaryExpression(node) || t.isLogicalExpression(node)) tokens.push(`operator:${node.operator}`); else if (t.isUnaryExpression(node) || t.isUpdateExpression(node)) tokens.push(`operator:${node.operator}`); if (t.isMemberExpression(node) || t.isOptionalMemberExpression(node)) { - const property = propertyName(node.property); - if (property !== "") tokens.push(`property:${property}`); + // A computed identifier names a local binding, not a fixed member. + tokens.push(`computed:${String(node.computed)}`); + if (!node.computed) { + const property = propertyName(node.property); + if (property !== "") tokens.push(`property:${property}`); + } } if ((t.isObjectProperty(node) || t.isObjectMethod(node)) && !node.computed) { const key = propertyName(node.key); diff --git a/src/domain/javascript/javascriptAstValues.ts b/src/domain/javascript/javascriptAstValues.ts index cb3a533fe4..8be3a5e838 100644 --- a/src/domain/javascript/javascriptAstValues.ts +++ b/src/domain/javascript/javascriptAstValues.ts @@ -1,6 +1,14 @@ import * as t from "@babel/types"; import type { JavaScriptSemanticPrimitive } from "./javascriptSemanticValueTypes.js"; +/** TypeScript receiver annotations occupy no positional runtime parameter slot. */ +export const runtimeFunctionParameters = ( + node: t.Function, +): t.Function["params"] => + t.isIdentifier(node.params[0], { name: "this" }) + ? node.params.slice(1) + : node.params; + /** * Remove wrappers that affect TypeScript or Flow checking but not runtime * expression identity. The returned depth lets bounded evaluators account for @@ -73,6 +81,20 @@ export const semanticStaticPropertyKey = ( property = unwrapJavaScriptExpression(property).node; if (t.isStringLiteral(property) || t.isNumericLiteral(property)) return String(property.value); + if (t.isBigIntLiteral(property)) return String(property.value); + if ( + t.isUnaryExpression(property, { operator: "-" }) && + t.isBigIntLiteral(property.argument) + ) + return String(-BigInt(property.argument.value)); + if ( + t.isTemplateLiteral(property) && + property.expressions.length === 0 && + property.quasis.length === 1 + ) { + const cooked = property.quasis[0]?.value.cooked; + return cooked === null || cooked === undefined ? null : cooked; + } return !computed && t.isIdentifier(property) ? property.name : null; }; @@ -86,6 +108,41 @@ export const semanticObjectPatternKeys = ( return key === null ? [] : [key]; }); +/** Return the syntactic node whose target this construct writes. */ +export const semanticWriteSiteTarget = (node: t.Node): t.Node | null => { + if (t.isAssignmentExpression(node)) return node.left; + if (t.isUpdateExpression(node)) return node.argument; + if (t.isUnaryExpression(node, { operator: "delete" })) return node.argument; + if (t.isForOfStatement(node) || t.isForInStatement(node)) return node.left; + return null; +}; + +/** Whether a write site reads the previous target value before writing. */ +export const semanticWriteSiteReadsFirst = (site: t.Node): boolean => + (t.isAssignmentExpression(site) && site.operator !== "=") || + t.isUpdateExpression(site); + +/** Find the owning write site through inert TypeScript and parenthesis wrappers. */ +export const semanticWrappedWriteSite = ( + node: t.Node, + parent: t.Node, + ancestors: readonly t.Node[], +): t.Node | null => { + const chain = + ancestors.at(-1) === parent ? ancestors : [...ancestors, parent]; + for (let index = chain.length - 1; index >= 0; index -= 1) { + const candidate = chain[index]; + if (candidate === undefined) return null; + const target = semanticWriteSiteTarget(candidate); + if (target !== null) + return unwrapJavaScriptExpression(target).node === node + ? candidate + : null; + if (unwrapJavaScriptExpression(candidate).depth === 0) return null; + } + return null; +}; + /** Visit binding and member leaves in direct and destructuring write targets. */ export function* semanticWriteTargets( root: t.Node, diff --git a/src/domain/javascript/javascriptBootstrap.ts b/src/domain/javascript/javascriptBootstrap.ts index cdb33d0245..2cc20ffdf8 100644 --- a/src/domain/javascript/javascriptBootstrap.ts +++ b/src/domain/javascript/javascriptBootstrap.ts @@ -1,6 +1,9 @@ import * as t from "@babel/types"; -import { semanticStaticPropertyKey } from "./javascriptAstValues.js"; +import { + semanticStaticPropertyKey, + runtimeFunctionParameters, +} from "./javascriptAstValues.js"; import { moduleFactory } from "./javascriptStaticAnalysisHelpers.js"; import { traverseJavaScriptAst } from "./javascriptSemanticTraversal.js"; import { @@ -27,7 +30,7 @@ export const javascriptBootstrapAt = ( (!t.isFunctionExpression(call.callee) && !t.isArrowFunctionExpression(call.callee)) || !t.isBlockStatement(call.callee.body) || - call.callee.params.length !== 0 || + runtimeFunctionParameters(call.callee).length !== 0 || call.arguments.length !== 0 ) return undefined; @@ -63,10 +66,10 @@ export const javascriptBootstrapAt = ( !t.isFunctionDeclaration(loader) || loader.id === null || loader.id === undefined || - loader.params.length !== 1 + runtimeFunctionParameters(loader).length !== 1 ) continue; - const parameter = loader.params[0]; + const parameter = runtimeFunctionParameters(loader)[0]; if (!t.isIdentifier(parameter)) continue; const loaderName = loader.id.name; const loaderBindingId = bindingId(state, loader.id); diff --git a/src/domain/javascript/javascriptBootstrapIntegrity.ts b/src/domain/javascript/javascriptBootstrapIntegrity.ts new file mode 100644 index 0000000000..1e748df414 --- /dev/null +++ b/src/domain/javascript/javascriptBootstrapIntegrity.ts @@ -0,0 +1,527 @@ +import * as t from "@babel/types"; + +import type { javascriptBootstrapAt } from "./javascriptBootstrap.js"; +import { semanticStaticPropertyKey } from "./javascriptAstValues.js"; +import { + resolveSemanticBindingState, + semanticStableInitializer, + type JavaScriptSemanticAnalysisState, +} from "./javascriptSemanticState.js"; +import { traverseJavaScriptAst } from "./javascriptSemanticTraversal.js"; +import { argumentValue } from "./javascriptStaticAnalysisHelpers.js"; + +type Bootstrap = NonNullable>; + +/** Slots a visible literal write may have replaced; the rest stay as declared. */ +export interface JavaScriptBootstrapIntegrity { + readonly replacedKeys: ReadonlySet; +} + +interface LoaderShape { + readonly parameterId: string; + /** The single unconditional `table[key](...)` call. */ + readonly dispatch: t.CallExpression; + readonly dispatches: ReadonlySet; + readonly moduleId: string; + readonly cacheId: string | undefined; + /** Cache reads, cache writes and deletes the loader itself performs. */ + readonly cacheSites: ReadonlySet; +} + +/** + * Prove that `loader(key)` returns the exports of the factory declared at + * `table[key]`: the loader returns only the dispatched module's exports or a + * cache entry it created for that module, the table has only literal keys, + * and the table and cache are referenced only by the loader or through the + * runtime's `loader.m`/`loader.c` exposure. Literal-key writes through that + * exposure replace their slot. Returns undefined when any of this is unproven. + */ +export const proveJavaScriptBootstrapIntegrity = ( + call: t.CallExpression, + bootstrap: Bootstrap, + loaderIds: ReadonlySet, + state: JavaScriptSemanticAnalysisState, +): JavaScriptBootstrapIntegrity | undefined => { + if ( + bootstrap.table.properties.some((property) => + t.isSpreadElement(property) + ? true + : semanticStaticPropertyKey(property.key, property.computed) === null, + ) + ) + return undefined; + const loader = bootstrap.body.body.find( + (statement): statement is t.FunctionDeclaration => + t.isFunctionDeclaration(statement) && + statement.id !== null && + statement.id !== undefined && + bindingOf(state, statement.id) === bootstrap.loaderBindingId, + ); + const shape = + loader === undefined + ? undefined + : loaderShape(loader, bootstrap.tableBindingId, state); + if (shape === undefined) return undefined; + return referencesAreContained(call, { bootstrap, shape, loaderIds }, state); +}; + +const loaderShape = ( + loader: t.FunctionDeclaration, + tableId: string, + state: JavaScriptSemanticAnalysisState, +): LoaderShape | undefined => { + const [parameter] = loader.params; + const parameterId = t.isIdentifier(parameter) + ? bindingOf(state, parameter) + : undefined; + // A rewritten key would dispatch a different slot than the caller named. + if ( + parameterId === undefined || + state.bindingsById.get(parameterId)?.definitions.length !== 1 || + loader.async || + loader.generator + ) + return undefined; + const dispatches = new Set(); + const calls: t.CallExpression[] = []; + const modules = new Set(); + const returns: (t.Expression | null | undefined)[] = []; + traverseJavaScriptAst(loader.body, { + skip: (node) => t.isFunction(node), + enter: (node) => { + if (t.isReturnStatement(node)) returns.push(node.argument); + if (!t.isCallExpression(node)) return; + const dispatch = tableDispatch(node, tableId, parameterId, state); + if (dispatch === undefined) return; + dispatches.add(dispatch.table); + calls.push(node); + if (dispatch.moduleId !== undefined) modules.add(dispatch.moduleId); + }, + }); + const [moduleId] = modules; + const [dispatch] = calls; + if ( + calls.length !== 1 || + dispatch === undefined || + modules.size !== 1 || + moduleId === undefined || + !runsUnconditionally(dispatch, loader, state) + ) + return undefined; + const cache = moduleCache(moduleId, parameterId, state); + if ( + cache === null || + !moduleObjectContained(loader, moduleId, dispatch, state) + ) + return undefined; + const shape: LoaderShape = { + parameterId, + dispatch, + dispatches, + moduleId, + cacheId: cache?.cacheId, + cacheSites: new Set(cache === undefined ? [] : [cache.site]), + }; + return returns.every((argument) => + returnsModuleExports(argument, shape, state), + ) + ? withCacheReads(loader, shape, state) + : undefined; +}; + +/** `table[key](module, ...)` or `table[key].call(thisArg, module, ...)`. */ +const tableDispatch = ( + call: t.CallExpression, + tableId: string, + parameterId: string, + state: JavaScriptSemanticAnalysisState, +): { readonly table: t.Node; readonly moduleId?: string } | undefined => { + const viaCall = + t.isMemberExpression(call.callee) && + semanticStaticPropertyKey(call.callee.property, call.callee.computed) === + "call"; + const slot = + viaCall && t.isMemberExpression(call.callee) + ? call.callee.object + : call.callee; + if ( + !t.isMemberExpression(slot) || + !slot.computed || + !t.isIdentifier(slot.object) || + !t.isIdentifier(slot.property) || + bindingOf(state, slot.object) !== tableId || + bindingOf(state, slot.property) !== parameterId + ) + return undefined; + const module = call.arguments[viaCall ? 1 : 0]; + const moduleId = t.isIdentifier(module) + ? bindingOf(state, module) + : undefined; + return moduleId === undefined + ? { table: slot.object } + : { table: slot.object, moduleId }; +}; + +/** + * The module object is a fresh literal, optionally stored as `cache[key]`. + * Undefined means no cache; null means the module binding is not proven. + */ +const moduleCache = ( + moduleId: string, + parameterId: string, + state: JavaScriptSemanticAnalysisState, +): { readonly cacheId: string; readonly site: t.Node } | undefined | null => { + const binding = state.bindingsById.get(moduleId); + const initializer = semanticStableInitializer(binding, state); + if ( + binding?.definitions.length !== 1 || + initializer === undefined || + initializer.projection.length !== 0 + ) + return null; + const value = initializer.node; + if (isPlainObject(value)) return undefined; + if ( + !t.isAssignmentExpression(value, { operator: "=" }) || + !isPlainObject(value.right) + ) + return null; + const cacheId = keyedSlot(value.left, parameterId, state); + return cacheId !== undefined && startsEmpty(cacheId, state) + ? { cacheId, site: value.left } + : null; +}; + +/** A pre-populated cache could answer the early return without a dispatch. */ +const startsEmpty = ( + cacheId: string, + state: JavaScriptSemanticAnalysisState, +): boolean => { + const binding = state.bindingsById.get(cacheId); + const initializer = semanticStableInitializer(binding, state); + return ( + binding?.definitions.length === 1 && + initializer?.projection.length === 0 && + t.isObjectExpression(initializer.node) && + initializer.node.properties.length === 0 + ); +}; + +/** An object literal of ordinary data properties, so `exports` stores plainly. */ +const isPlainObject = (node: t.Node): boolean => + t.isObjectExpression(node) && + node.properties.every( + (property) => + t.isObjectProperty(property) && + !property.computed && + semanticStaticPropertyKey(property.key, false) !== "__proto__", + ); + +/** Every ancestor up to the loader body executes whenever the loader runs. */ +const runsUnconditionally = ( + dispatch: t.Node, + loader: t.FunctionDeclaration, + state: JavaScriptSemanticAnalysisState, +): boolean => { + let child: t.Node = dispatch; + let parent = state.parentsByNode.get(child); + while (parent !== undefined && child !== loader.body) { + const ordered = + t.isSequenceExpression(parent) || + t.isExpressionStatement(parent) || + t.isReturnStatement(parent) || + t.isBlockStatement(parent) || + (t.isMemberExpression(parent) && parent.object === child) || + (t.isCallExpression(parent) && parent.callee === child) || + (t.isTryStatement(parent) && parent.block === child); + if (!ordered) return false; + child = parent; + parent = state.parentsByNode.get(parent); + } + return child === loader.body; +}; + +/** + * The module object reaches only the dispatch, `.exports` reads after it, and + * writes to members other than `exports`, so nothing swaps its exports. + */ +const moduleObjectContained = ( + loader: t.FunctionDeclaration, + moduleId: string, + dispatch: t.CallExpression, + state: JavaScriptSemanticAnalysisState, +): boolean => { + let contained = true; + traverseJavaScriptAst(loader.body, { + enter: (node, parent, readAncestors) => { + if (!contained || !t.isIdentifier(node) || parent === null) return; + const ancestors = readAncestors(); + if (!t.isReferenced(node, parent, ancestors.at(-2))) return; + if (bindingOf(state, node) !== moduleId) return; + contained = + !ancestors.some((ancestor) => t.isFunction(ancestor)) && + moduleUse(node, [parent, ancestors.at(-2)], dispatch); + }, + }); + return contained; +}; + +const moduleUse = ( + node: t.Identifier, + [parent, grandparent]: readonly [t.Node, t.Node | undefined], + dispatch: t.CallExpression, +): boolean => { + if (parent === dispatch) return dispatch.arguments.includes(node); + if (!t.isMemberExpression(parent) || parent.object !== node) return false; + const key = semanticStaticPropertyKey(parent.property, parent.computed); + const written = + (t.isAssignmentExpression(grandparent) && grandparent.left === parent) || + t.isUpdateExpression(grandparent) || + t.isUnaryExpression(grandparent, { operator: "delete" }); + if (written) return key !== null && key !== "exports"; + if (key !== "exports") return false; + // `module.exports` passed to the factory, or read after the dispatch ran. + return ( + grandparent === dispatch || + (parent.start ?? -1) > (dispatch.end ?? Number.POSITIVE_INFINITY) + ); +}; + +/** Record `cache[key]` reads that initialize the loader's early-return binding. */ +const withCacheReads = ( + loader: t.FunctionDeclaration, + shape: LoaderShape, + state: JavaScriptSemanticAnalysisState, +): LoaderShape | undefined => { + if (shape.cacheId === undefined) return shape; + const sites = new Set(shape.cacheSites); + let contained = true; + traverseJavaScriptAst(loader.body, { + skip: (node) => t.isFunction(node), + enter: (node, parent) => { + if ( + !t.isMemberExpression(node) || + keyedSlot(node, shape.parameterId, state) !== shape.cacheId + ) + return; + const read = t.isVariableDeclarator(parent) && parent.init === node; + const deleted = t.isUnaryExpression(parent, { operator: "delete" }); + if (read || deleted || sites.has(node)) sites.add(node); + else contained = false; + }, + }); + return contained ? { ...shape, cacheSites: sites } : undefined; +}; + +/** Accept `module.exports` or `cache[key].exports` read through a stable binding. */ +const returnsModuleExports = ( + argument: t.Expression | null | undefined, + shape: LoaderShape, + state: JavaScriptSemanticAnalysisState, +): boolean => { + const value = t.isSequenceExpression(argument) + ? argument.expressions.at(-1) + : argument; + const before = t.isSequenceExpression(argument) + ? argument.expressions.slice(0, -1) + : []; + if ( + !before.every((expression) => + sideEffectBeforeReturn(expression, shape, state), + ) + ) + return false; + if ( + !t.isMemberExpression(value) || + !t.isIdentifier(value.object) || + semanticStaticPropertyKey(value.property, value.computed) !== "exports" + ) + return false; + const id = bindingOf(state, value.object); + if (id === shape.moduleId) return true; + if (id === undefined || shape.cacheId === undefined) return false; + const binding = state.bindingsById.get(id); + const initializer = semanticStableInitializer(binding, state); + return ( + binding?.definitions.length === 1 && + initializer?.projection.length === 0 && + keyedSlot(initializer.node, shape.parameterId, state) === shape.cacheId + ); +}; + +/** The dispatch itself, or a write to a module member other than `exports`. */ +const sideEffectBeforeReturn = ( + expression: t.Expression, + shape: LoaderShape, + state: JavaScriptSemanticAnalysisState, +): boolean => { + if (expression === shape.dispatch) return true; + if ( + !t.isAssignmentExpression(expression) || + !t.isMemberExpression(expression.left) || + !t.isIdentifier(expression.left.object) || + bindingOf(state, expression.left.object) !== shape.moduleId + ) + return false; + const key = semanticStaticPropertyKey( + expression.left.property, + expression.left.computed, + ); + return key !== null && key !== "exports"; +}; + +/** The binding of `owner` in `owner[key]` when `key` is the loader parameter. */ +const keyedSlot = ( + node: t.Node, + parameterId: string, + state: JavaScriptSemanticAnalysisState, +): string | undefined => + t.isMemberExpression(node) && + node.computed && + t.isIdentifier(node.object) && + t.isIdentifier(node.property) && + bindingOf(state, node.property) === parameterId + ? bindingOf(state, node.object) + : undefined; + +const referencesAreContained = ( + call: t.CallExpression, + { + bootstrap, + shape, + loaderIds, + }: { + readonly bootstrap: Bootstrap; + readonly shape: LoaderShape; + readonly loaderIds: ReadonlySet; + }, + state: JavaScriptSemanticAnalysisState, +): JavaScriptBootstrapIntegrity | undefined => { + const tracked = new Map([ + [bootstrap.tableBindingId, "table"], + ...(shape.cacheId === undefined ? [] : [[shape.cacheId, "cache"] as const]), + ...[...loaderIds].map((id) => [id, "loader"] as const), + ]); + const names = new Set( + [...tracked.keys()].flatMap((id) => state.bindingsById.get(id)?.name ?? []), + ); + const replacedKeys = new Set(); + let contained = true; + traverseJavaScriptAst(call, { + skip: () => !contained, + enter: (node, parent, readAncestors) => { + if (!t.isIdentifier(node) || !names.has(node.name) || parent === null) + return; + const ancestors = readAncestors(); + if (!t.isReferenced(node, parent, ancestors.at(-2))) return; + const id = bindingOf(state, node); + const role = id === undefined ? undefined : tracked.get(id); + if (role === undefined) return; + const use: Use = { node, parent, ancestors, shape, loaderIds, state }; + if (role !== "loader") contained = containedUse(role, use); + else if (exposedTableUse(use) === "escaped") contained = false; + else exposedSlotWrite(use, replacedKeys); + }, + }); + return contained ? { replacedKeys } : undefined; +}; + +interface Use { + readonly node: t.Identifier; + readonly parent: t.Node; + readonly ancestors: readonly t.Node[]; + readonly shape: LoaderShape; + readonly loaderIds: ReadonlySet; + readonly state: JavaScriptSemanticAnalysisState; +} + +/** The table is read only by the dispatch; the cache only by the loader's own slot operations. */ +const containedUse = (role: "table" | "cache", use: Use): boolean => { + if (role === "table" && use.shape.dispatches.has(use.node)) return true; + if (role === "cache" && use.shape.cacheSites.has(use.parent)) return true; + return isRuntimeExposure(use, role === "table" ? "m" : "c"); +}; + +/** `loader.m = table` or `loader.c = cache`, the runtime's own exposure. */ +const isRuntimeExposure = (use: Use, key: "m" | "c"): boolean => { + const { parent, node, loaderIds, state } = use; + return ( + t.isAssignmentExpression(parent, { operator: "=" }) && + parent.right === node && + t.isMemberExpression(parent.left) && + t.isIdentifier(parent.left.object) && + semanticStaticPropertyKey(parent.left.property, parent.left.computed) === + key && + loaderIds.has(bindingOf(state, parent.left.object) ?? "") + ); +}; + +/** + * `loader.m`/`loader.c` may only be assigned (the exposure) or indexed + * directly; any other use, such as `const a = loader.m`, lets later writes + * escape this proof. Writing a member of an indexed entry, such as + * `loader.c[id].exports = value`, can replace what the loader returns. + */ +const exposedTableUse = (use: Use): "other" | "contained" | "escaped" => { + const { parent, ancestors } = use; + if ( + !t.isMemberExpression(parent) || + parent.object !== use.node || + !["m", "c"].includes( + semanticStaticPropertyKey(parent.property, parent.computed) ?? "", + ) + ) + return "other"; + const outer = ancestors.at(-2); + const assigned = + t.isAssignmentExpression(outer, { operator: "=" }) && outer.left === parent; + const indexed = + t.isMemberExpression(outer) && outer.object === parent && outer.computed; + if (assigned) return "contained"; + if (!indexed) return "escaped"; + const member = ancestors.at(-3); + const target = ancestors.at(-4); + const entryWritten = + t.isMemberExpression(member) && + member.object === outer && + ((t.isAssignmentExpression(target) && target.left === member) || + t.isUpdateExpression(target) || + t.isUnaryExpression(target, { operator: "delete" })); + return entryWritten ? "escaped" : "contained"; +}; + +/** + * `loader.m[key] = factory` or `delete loader.c[key]` replaces a literal slot. + * Writes with runtime keys are chunk installation, which adds modules under + * their own ids; they are not treated as replacing a declared slot. + */ +const exposedSlotWrite = (use: Use, replacedKeys: Set): void => { + const { parent, ancestors } = use; + const slot = ancestors.at(-2); + const write = ancestors.at(-3); + if ( + !t.isMemberExpression(parent) || + parent.object !== use.node || + !["m", "c"].includes( + semanticStaticPropertyKey(parent.property, parent.computed) ?? "", + ) || + !t.isMemberExpression(slot) || + slot.object !== parent || + !( + (t.isAssignmentExpression(write) && write.left === slot) || + t.isUnaryExpression(write, { operator: "delete" }) || + t.isUpdateExpression(write) + ) + ) + return; + const key = slot.computed + ? argumentValue(slot.property) + : semanticStaticPropertyKey(slot.property, false); + if (key !== undefined && key !== null) replacedKeys.add(key); +}; + +const bindingOf = ( + state: JavaScriptSemanticAnalysisState, + identifier: t.Identifier, +): string | undefined => + resolveSemanticBindingState(state, identifier, identifier.name)?.bindingId; diff --git a/src/domain/javascript/javascriptBuiltinResourceMethods.ts b/src/domain/javascript/javascriptBuiltinResourceMethods.ts new file mode 100644 index 0000000000..8034b902fe --- /dev/null +++ b/src/domain/javascript/javascriptBuiltinResourceMethods.ts @@ -0,0 +1,11 @@ +/** Supported native resource receivers, shared by acquisition and validity owners. */ +export const BUILTIN_RESOURCE_METHODS = { + fs: ["open", "openSync", "createReadStream", "createWriteStream"], + "fs/promises": ["open"], + net: ["connect", "createConnection"], + tls: ["connect", "createConnection"], +} as const; + +export const BUILTIN_RESOURCE_SPECIFIERS = Object.keys( + BUILTIN_RESOURCE_METHODS, +); diff --git a/src/domain/javascript/javascriptBundledRequireOrigin.ts b/src/domain/javascript/javascriptBundledRequireOrigin.ts new file mode 100644 index 0000000000..dbfe873e08 --- /dev/null +++ b/src/domain/javascript/javascriptBundledRequireOrigin.ts @@ -0,0 +1,451 @@ +import * as t from "@babel/types"; + +import { javascriptBootstrapAt } from "./javascriptBootstrap.js"; +import { proveJavaScriptBootstrapIntegrity } from "./javascriptBootstrapIntegrity.js"; +import { + semanticStaticPropertyKey, + unwrapJavaScriptExpression, +} from "./javascriptAstValues.js"; +import { collectJavaScriptIdentityWritesSteps } from "./javascriptElectronMemberWrites.js"; +import { createSemanticModuleInterop } from "./javascriptModuleInterop.js"; +import { + semanticMemberChain, + semanticRequireOrigin, + type JavaScriptRequireOrigin, +} from "./javascriptSemanticRequireOrigin.js"; +import { + resolveSemanticBindingState, + semanticResolutionBlocked, + type JavaScriptSemanticAnalysisState, +} from "./javascriptSemanticState.js"; +import { + argumentValue, + moduleFactory, + modulePropertyName, +} from "./javascriptStaticAnalysisHelpers.js"; +import { + completeSemanticSteps, + traverseJavaScriptAst, +} from "./javascriptSemanticTraversal.js"; + +/** Loader bindings and literal externals proven for one standalone bootstrap. */ +interface BootstrapExternals { + /** The runtime loader and every factory parameter it is passed as. */ + readonly loaderBindingIds: ReadonlySet; + /** Loader bindings with the canonical n.n getter installed before dispatch. */ + readonly interopGetterBindingIds: ReadonlySet; + /** Table keys whose factory only re-exports one unshadowed literal require; null when a key repeats. */ + readonly externals: ReadonlyMap; +} + +const proofs = new WeakMap< + JavaScriptSemanticAnalysisState, + WeakMap +>(); + +/** + * Recover a literal require reached through a proven Webpack bootstrap. + * + * Webpack emits externals as `{45840: e => { e.exports = require("electron") }}` + * and consumers call the runtime loader, `n(45840)`, instead of `require`. The + * origin is accepted only when the loader binding is the bootstrap's own loader + * or a factory parameter that the bootstrap proves it passes the loader as, and + * the keyed factory consists solely of that re-export. + */ +export const semanticBundledRequireOrigin = ( + node: t.Node | null | undefined, + state: JavaScriptSemanticAnalysisState, +): JavaScriptRequireOrigin | undefined => { + const chain = semanticMemberChain(node); + const current = chain?.root; + if ( + chain?.members.length === 0 && + t.isCallExpression(current) && + t.isMemberExpression(current.callee) && + semanticStaticPropertyKey( + current.callee.property, + current.callee.computed, + ) === "n" && + t.isIdentifier(current.callee.object) && + current.arguments.length === 1 + ) { + // Webpack's `n.n(module)` returns its default-export getter. For a + // CommonJS external, invoking that getter yields the exact exports object; + // bind its identity to the proven registry result while retaining the + // nested loader call as the source of provenance. + const nested = semanticBundledRequireOrigin(current.arguments[0], state); + const loaderBinding = resolveSemanticBindingState( + state, + current.callee.object, + current.callee.object.name, + ); + const proof = + nested === undefined || loaderBinding === undefined + ? undefined + : bootstrapProofForLoaderCall( + nested.call, + loaderBinding.bindingId, + state, + ); + const nestedCallee = nested?.call.callee; + const loaderBindingId = loaderBinding?.bindingId; + if ( + nested !== undefined && + nested.origin.importedPath.length === 0 && + t.isIdentifier(nestedCallee) && + loaderBindingId !== undefined && + loaderBindingId === + resolveSemanticBindingState(state, nestedCallee, nestedCallee.name) + ?.bindingId && + proof?.loaderBindingIds.has(loaderBindingId) === true && + proof.interopGetterBindingIds.has(loaderBindingId) + ) + return nested; + } + if ( + chain === undefined || + !t.isCallExpression(current) || + !t.isIdentifier(current.callee) || + current.arguments.length !== 1 + ) + return undefined; + const key = argumentValue(current.arguments[0]); + if (key === undefined) return undefined; + const loader = current.callee; + if (semanticResolutionBlocked(state, loader, loader.name)) return undefined; + const binding = resolveSemanticBindingState(state, loader, loader.name); + if ( + binding === undefined || + binding.definitions.length !== 1 || + binding.definitions[0]?.kind === "assignment" + ) + return undefined; + for ( + let ancestor = state.parentsByNode.get(current); + ancestor !== undefined; + ancestor = state.parentsByNode.get(ancestor) + ) { + if (!t.isCallExpression(ancestor)) continue; + const proof = bootstrapExternals(ancestor, state); + if (proof === null || !proof.loaderBindingIds.has(binding.bindingId)) + continue; + const specifier = proof.externals.get(key); + return typeof specifier === "string" + ? { + origin: { + specifier, + importedPath: chain.members, + resolutionMode: "require", + }, + call: current, + } + : undefined; + } + return undefined; +}; + +/** Prove the source-local n.n getter and reject visible replacements. */ +const canonicalWebpackInteropBindings = ( + loaderBindingIds: ReadonlySet, + bootstrapBody: t.BlockStatement, + state: JavaScriptSemanticAnalysisState, +): ReadonlySet => { + // The shared identity-write owner covers assignment, delete, destructuring, + // and Object/Reflect mutators; any write that may select `n` counts. + const writes = new Set(); + const interop = createSemanticModuleInterop(state); + completeSemanticSteps( + collectJavaScriptIdentityWritesSteps( + state.program, + (root, kind) => { + const node = unwrapJavaScriptExpression(root).node; + return kind === "member" && + t.isIdentifier(node) && + loaderBindingIds.has( + resolveSemanticBindingState(state, node, node.name)?.bindingId ?? + "", + ) + ? [["loader"]] + : []; + }, + interop.intrinsicInvocation, + interop.staticString, + interop.isFreshObject, + (node, path) => { + if (path.length < 2 || path[1] === "n" || path[1] === null) + writes.add(node); + }, + ), + ); + const proven = new Set(); + const [write] = writes; + if ( + writes.size === 1 && + t.isAssignmentExpression(write) && + isInstalledBeforeEntryDispatch( + write, + bootstrapBody, + loaderBindingIds, + state, + ) && + isCanonicalGetter(write, state) + ) + for (const bindingId of loaderBindingIds) proven.add(bindingId); + return proven; +}; + +const isCanonicalGetter = ( + write: t.AssignmentExpression, + state: JavaScriptSemanticAnalysisState, +): boolean => { + if ( + !t.isArrowFunctionExpression(write.right) || + write.right.params.length !== 1 || + !t.isIdentifier(write.right.params[0]) + ) + return false; + const parameter = write.right.params[0]; + const parameterBinding = resolveSemanticBindingState( + state, + parameter, + parameter.name, + ); + const body = write.right.body; + return ( + parameterBinding !== undefined && + t.isConditionalExpression(body) && + t.isLogicalExpression(body.test, { operator: "&&" }) && + sameBinding(body.test.left, parameterBinding.bindingId, state) && + t.isMemberExpression(body.test.right) && + semanticStaticPropertyKey( + body.test.right.property, + body.test.right.computed, + ) === "__esModule" && + sameBinding(body.test.right.object, parameterBinding.bindingId, state) && + getterReturnsDefault(body.consequent, parameterBinding.bindingId, state) && + getterReturnsModule(body.alternate, parameterBinding.bindingId, state) + ); +}; + +const isInstalledBeforeEntryDispatch = ( + write: t.AssignmentExpression, + body: t.BlockStatement, + loaderBindingIds: ReadonlySet, + state: JavaScriptSemanticAnalysisState, +): boolean => { + const statement = state.parentsByNode.get(write); + if (!t.isExpressionStatement(statement)) return false; + const writeIndex = body.body.indexOf(statement); + if (writeIndex < 0) return false; + let entryIndex = -1; + for (let index = 0; index < body.body.length; index += 1) { + const topLevel = body.body[index]; + if (topLevel === undefined) continue; + traverseJavaScriptAst(topLevel, { + enter: (node, _parent, ancestors) => { + if ( + entryIndex < 0 && + t.isCallExpression(node) && + t.isIdentifier(node.callee) && + ancestors().every((ancestor) => !t.isFunction(ancestor)) && + loaderBindingIds.has( + resolveSemanticBindingState(state, node.callee, node.callee.name) + ?.bindingId ?? "", + ) + ) + entryIndex = index; + }, + }); + if (entryIndex >= 0) break; + } + return entryIndex > writeIndex; +}; + +const bootstrapProofForLoaderCall = ( + call: t.CallExpression, + loaderBindingId: string, + state: JavaScriptSemanticAnalysisState, +): BootstrapExternals | undefined => { + for ( + let ancestor = state.parentsByNode.get(call); + ancestor !== undefined; + ancestor = state.parentsByNode.get(ancestor) + ) { + if (!t.isCallExpression(ancestor)) continue; + const proof = bootstrapExternals(ancestor, state); + if (proof?.loaderBindingIds.has(loaderBindingId)) return proof; + } + return undefined; +}; + +const sameBinding = ( + node: t.Node, + bindingId: string, + state: JavaScriptSemanticAnalysisState, +): boolean => + t.isIdentifier(node) && + resolveSemanticBindingState(state, node, node.name)?.bindingId === bindingId; + +const getterReturnsDefault = ( + node: t.Node, + bindingId: string, + state: JavaScriptSemanticAnalysisState, +): boolean => + t.isArrowFunctionExpression(node) && + node.params.length === 0 && + t.isMemberExpression(node.body) && + semanticStaticPropertyKey(node.body.property, node.body.computed) === + "default" && + sameBinding(node.body.object, bindingId, state); + +const getterReturnsModule = ( + node: t.Node, + bindingId: string, + state: JavaScriptSemanticAnalysisState, +): boolean => + t.isArrowFunctionExpression(node) && + node.params.length === 0 && + sameBinding(node.body, bindingId, state); + +const bootstrapExternals = ( + call: t.CallExpression, + state: JavaScriptSemanticAnalysisState, +): BootstrapExternals | null => { + let byCall = proofs.get(state); + if (byCall === undefined) { + byCall = new WeakMap(); + proofs.set(state, byCall); + } + const cached = byCall.get(call); + if (cached !== undefined) return cached; + const proof = proveBootstrapExternals(call, state); + byCall.set(call, proof); + return proof; +}; + +const proveBootstrapExternals = ( + call: t.CallExpression, + state: JavaScriptSemanticAnalysisState, +): BootstrapExternals | null => { + const bootstrap = javascriptBootstrapAt(call, state); + // A replaced loader no longer dispatches the declared table. + if ( + bootstrap === undefined || + !singleDefinition(state, bootstrap.loaderBindingId, "function") + ) + return null; + const loaderBindingIds = new Set([bootstrap.loaderBindingId]); + const factories = bootstrap.table.properties.flatMap((property) => { + const factory = moduleFactory(property); + return factory === undefined ? [] : [{ property, factory }]; + }); + for (const { factory } of factories) { + // The bootstrap proves `table[key](module, module.exports, loader)`. + const loaderParameter = factory.params[2]; + const bindingId = t.isIdentifier(loaderParameter) + ? resolveSemanticBindingState( + state, + loaderParameter, + loaderParameter.name, + )?.bindingId + : undefined; + if ( + bindingId !== undefined && + singleDefinition(state, bindingId, "parameter") + ) + loaderBindingIds.add(bindingId); + } + const integrity = proveJavaScriptBootstrapIntegrity( + call, + bootstrap, + loaderBindingIds, + state, + ); + if (integrity === undefined) return null; + const externals = new Map(); + for (const { property, factory } of factories) { + const key = modulePropertyName(property); + const specifier = integrity.replacedKeys.has(key) + ? undefined + : externalSpecifier(factory, state); + if (externals.has(key)) externals.set(key, null); + else externals.set(key, specifier ?? null); + } + return { + loaderBindingIds, + interopGetterBindingIds: canonicalWebpackInteropBindings( + loaderBindingIds, + bootstrap.body, + state, + ), + externals, + }; +}; + +/** + * Accept only `module.exports = require("specifier")` as the whole body of an + * ordinary factory; generator, async and accessor factories do not run it. + */ +const externalSpecifier = ( + factory: t.FunctionExpression | t.ArrowFunctionExpression | t.ObjectMethod, + state: JavaScriptSemanticAnalysisState, +): string | undefined => { + const moduleParameter = factory.params[0]; + const names = factory.params.map((parameter) => + t.isIdentifier(parameter) ? parameter.name : null, + ); + if ( + !t.isIdentifier(moduleParameter) || + // A duplicated name binds the later argument, not the module object. + new Set(names).size !== names.length || + factory.generator || + factory.async || + (t.isObjectMethod(factory) && factory.kind !== "method") + ) + return undefined; + let expression: t.Node | undefined; + if (t.isBlockStatement(factory.body)) { + const [statement] = factory.body.body; + if (factory.body.body.length !== 1 || !t.isExpressionStatement(statement)) + return undefined; + expression = statement.expression; + } else expression = factory.body; + if ( + !t.isAssignmentExpression(expression, { operator: "=" }) || + !t.isMemberExpression(expression.left) || + !t.isIdentifier(expression.left.object) || + semanticStaticPropertyKey( + expression.left.property, + expression.left.computed, + ) !== "exports" + ) + return undefined; + const moduleBinding = resolveSemanticBindingState( + state, + moduleParameter, + moduleParameter.name, + ); + const target = resolveSemanticBindingState( + state, + expression.left.object, + expression.left.object.name, + ); + if ( + moduleBinding === undefined || + target?.bindingId !== moduleBinding.bindingId + ) + return undefined; + const required = semanticRequireOrigin(expression.right, state)?.origin; + return required !== undefined && required.importedPath.length === 0 + ? required.specifier + : undefined; +}; + +const singleDefinition = ( + state: JavaScriptSemanticAnalysisState, + bindingId: string, + kind: "function" | "parameter", +): boolean => { + const definitions = state.bindingsById.get(bindingId)?.definitions; + return definitions?.length === 1 && definitions[0]?.kind === kind; +}; diff --git a/src/domain/javascript/javascriptCommonJsExportReceivers.ts b/src/domain/javascript/javascriptCommonJsExportReceivers.ts new file mode 100644 index 0000000000..9962a42d8e --- /dev/null +++ b/src/domain/javascript/javascriptCommonJsExportReceivers.ts @@ -0,0 +1,242 @@ +import * as t from "@babel/types"; + +import { + semanticStaticPropertyKey, + semanticWriteTargets, + unwrapJavaScriptExpression, +} from "./javascriptAstValues.js"; +import { + isUnshadowedGlobal, + type JavaScriptSemanticAnalysisState, +} from "./javascriptSemanticState.js"; +import { traverseJavaScriptAstSteps } from "./javascriptSemanticTraversal.js"; +import { semanticPropertyPathKeyMatches } from "./javascriptSemanticPropertyPaths.js"; + +export type CommonJsExportReceiverEffect = + | "connected" + | "detached" + | "uncertain"; + +const isCommonJsModuleExports = ( + node: t.Node, + state: JavaScriptSemanticAnalysisState, +): boolean => + (t.isMemberExpression(node) || t.isOptionalMemberExpression(node)) && + isUnshadowedGlobal(node.object, state, "module") && + semanticStaticPropertyKey(node.property, node.computed) === "exports"; + +/** One syntax owner for CJS export names and their receiver identities. */ +export const commonJsExportName = ( + node: t.Node, + state: JavaScriptSemanticAnalysisState, +): string | undefined => { + if (!t.isMemberExpression(node) && !t.isOptionalMemberExpression(node)) + return undefined; + const key = semanticStaticPropertyKey(node.property, node.computed); + if ( + isUnshadowedGlobal(node.object, state, "exports") || + isCommonJsModuleExports(node.object, state) + ) + return key === null ? "*" : key; + return isCommonJsModuleExports(node, state) ? "default" : undefined; +}; + +type Identity = t.Node | null; +type Execution = "ordered" | "conditional" | "deferred"; + +/** Pattern evaluation follows the RHS, unlike Babel's child visit order. */ +const hasDestructuringEvaluationOrder = (node: t.Node): boolean => { + const pattern = t.isAssignmentExpression(node) + ? unwrapJavaScriptExpression(node.left).node + : t.isVariableDeclarator(node) + ? node.id + : undefined; + return t.isObjectPattern(pattern) || t.isArrayPattern(pattern); +}; + +/** + * Capture receivers before their RHS, then compare with the final public + * identity. Fresh allocations prove distinctness; explicit wrapper reads + * preserve even opaque identities. Branches and deferred execution remain + * uncertain, as do local aliases. Facts live for one analysis and the + * cooperative traversal bounds work to the source tree size. + */ +export function* indexCommonJsExportReceiversSteps( + state: JavaScriptSemanticAnalysisState, +): Generator< + void, + ReadonlyMap +> { + const initial = state.program; + let publicIdentity: Identity = initial; + let exportsIdentity: Identity = initial; + let moduleIntact = true; + let deferredReplacement = false; + const fresh = new WeakSet([initial]); + const values = new WeakMap(); + const captures = new Map(); + const execution: Execution[] = []; + const publicCallWrites = new WeakSet(); + // Consume the shared write owner's resolved aliases, intrinsic invocation + // and keyed mutator facts instead of reinterpreting call names here. + for (const { node, path } of state.ambientWriteEffects) { + if ( + (t.isCallExpression(node) || t.isOptionalCallExpression(node)) && + path[0] === "commonjs-module" && + path.length <= 2 && + (path.length === 1 || + semanticPropertyPathKeyMatches(path[1] ?? null, "exports")) + ) + publicCallWrites.add(node); + } + const valueOf = (node: t.Node): Identity => { + const value = unwrapJavaScriptExpression(node).node; + // An opaque result may alias another object, but reading the current + // module.exports and explicitly reconnecting exports preserves that result. + return values.has(value) ? (values.get(value) ?? null) : value; + }; + + const write = (target: t.Node, identity: Identity, mode: Execution): void => { + target = unwrapJavaScriptExpression(target).node; + const possiblePublicWrite = + (t.isMemberExpression(target) || t.isOptionalMemberExpression(target)) && + isUnshadowedGlobal(target.object, state, "module") && + semanticStaticPropertyKey(target.property, target.computed) === null; + const publicWrite = + isCommonJsModuleExports(target, state) || possiblePublicWrite; + const exportsWrite = isUnshadowedGlobal(target, state, "exports"); + const moduleWrite = isUnshadowedGlobal(target, state, "module"); + if (!publicWrite && !exportsWrite && !moduleWrite) return; + if (mode === "deferred") { + // A source offset cannot order an invocation against module completion. + deferredReplacement = true; + return; + } + const selected = + mode === "ordered" && !possiblePublicWrite ? identity : null; + if (moduleWrite) moduleIntact = false; + if (publicWrite || moduleWrite) + publicIdentity = moduleIntact ? selected : null; + if (exportsWrite) exportsIdentity = selected; + }; + + yield* traverseJavaScriptAstSteps(state.program, { + enter: (node, parent) => { + const outer = execution.at(-1) ?? "ordered"; + const mode: Execution = + outer === "deferred" || t.isFunction(node) || t.isClass(node) + ? "deferred" + : outer === "conditional" || + // Mark the whole operation: a later tree-visited RHS must not + // restore certainty after an earlier tree-visited pattern write. + hasDestructuringEvaluationOrder(node) || + t.isIfStatement(node) || + t.isSwitchStatement(node) || + t.isLoop(node) || + t.isTryStatement(node) || + t.isObjectPattern(node) || + t.isArrayPattern(node) || + (t.isLogicalExpression(parent) && parent.right === node) || + (t.isAssignmentPattern(parent) && parent.right === node) || + (t.isAssignmentExpression(parent) && + ["&&=", "||=", "??="].includes(parent.operator) && + parent.right === node) || + t.isOptionalCallExpression(parent) || + (t.isOptionalMemberExpression(parent) && + parent.property === node) || + (t.isConditionalExpression(parent) && parent.test !== node) + ? "conditional" + : "ordered"; + execution.push(mode); + if ( + t.isAssignmentExpression(node) && + commonJsExportName(node.left, state) !== undefined && + !isCommonJsModuleExports(node.left, state) + ) { + const receiver = + t.isMemberExpression(node.left) || + t.isOptionalMemberExpression(node.left) + ? node.left.object + : null; + captures.set( + node, + mode !== "ordered" + ? null + : receiver !== null && + isUnshadowedGlobal(receiver, state, "exports") + ? exportsIdentity + : publicIdentity, + ); + } + }, + exit: (node) => { + const mode = execution.pop() ?? "conditional"; + if (publicCallWrites.has(node)) { + if (mode === "deferred") deferredReplacement = true; + else publicIdentity = null; + } + if (t.isAssignmentExpression(node)) { + const left = unwrapJavaScriptExpression(node.left).node; + const identity = + node.operator === "=" && + (t.isIdentifier(left) || + t.isMemberExpression(left) || + t.isOptionalMemberExpression(left)) + ? valueOf(node.right) + : null; + for (const target of semanticWriteTargets(node.left)) + write(target, identity, mode); + values.set(node, identity); + } else if (t.isVariableDeclarator(node) && node.init != null) { + for (const target of semanticWriteTargets(node.id)) + write( + target, + t.isIdentifier(node.id) ? valueOf(node.init) : null, + mode, + ); + } else if (t.isForOfStatement(node) || t.isForInStatement(node)) { + if (!t.isVariableDeclaration(node.left)) + for (const target of semanticWriteTargets(node.left)) + write(target, null, mode); + } else if ( + t.isUpdateExpression(node) || + t.isUnaryExpression(node, { operator: "delete" }) + ) { + write(node.argument, null, mode); + } else if (isUnshadowedGlobal(node, state, "exports")) { + values.set(node, exportsIdentity); + } else if (isCommonJsModuleExports(node, state)) { + values.set(node, publicIdentity); + } else if ( + t.isObjectExpression(node) || + t.isArrayExpression(node) || + t.isFunctionExpression(node) || + t.isArrowFunctionExpression(node) || + t.isClassExpression(node) + ) { + values.set(node, node); + fresh.add(node); + } else if (t.isSequenceExpression(node)) { + const last = node.expressions.at(-1); + values.set(node, last === undefined ? null : valueOf(last)); + } + }, + }); + const effects = new Map< + t.AssignmentExpression, + CommonJsExportReceiverEffect + >(); + for (const [node, receiver] of captures) { + effects.set( + node, + deferredReplacement || receiver === null || publicIdentity === null + ? "uncertain" + : receiver === publicIdentity + ? "connected" + : fresh.has(receiver) && fresh.has(publicIdentity) + ? "detached" + : "uncertain", + ); + } + return effects; +} diff --git a/src/domain/javascript/javascriptElectronMemberWrites.ts b/src/domain/javascript/javascriptElectronMemberWrites.ts index 14741af36d..62a921b87d 100644 --- a/src/domain/javascript/javascriptElectronMemberWrites.ts +++ b/src/domain/javascript/javascriptElectronMemberWrites.ts @@ -2,6 +2,7 @@ import * as t from "@babel/types"; import { semanticStaticPropertyKey, + semanticWriteSiteTarget, semanticWriteTargets, unwrapJavaScriptExpression, } from "./javascriptAstValues.js"; @@ -62,6 +63,7 @@ export function* collectJavaScriptIdentityWritesSteps( origin: ( root: t.Node, kind: "binding" | "member" | "opaque-call", + writeSite: t.Node, ) => readonly (readonly (string | null)[])[], intrinsicInvocation: ( node: t.CallExpression | t.OptionalCallExpression, @@ -88,7 +90,8 @@ export function* collectJavaScriptIdentityWritesSteps( if (t.isCallExpression(node) || t.isOptionalCallExpression(node)) { const invocation = intrinsicInvocation(node); if (invocation?.kind === "opaque") { - for (const scope of origin(node, "opaque-call")) retain(node, scope); + for (const scope of origin(node, "opaque-call", node)) + retain(node, scope); return; } const callee = invocation?.path; @@ -117,38 +120,32 @@ export function* collectJavaScriptIdentityWritesSteps( // A known mutator with an opaque argument list can overwrite any // captured intrinsic. Do not turn an unexamined target into proof. if (target === undefined || t.isSpreadElement(target)) { - for (const scope of origin(node, "opaque-call")) + for (const scope of origin(node, "opaque-call", node)) retain(node, scope); } if (t.isNode(target) && !t.isSpreadElement(target)) { - const identities = origin(target, "member"); + const identities = origin(target, "member", node); if (identities.length > 0) { const key = keyedMutators.has(method ?? "") ? [staticString(arguments_?.[1]) ?? null] - : []; + : // Mutating a target's contents does not replace the target. + // Retain an unknown child key, rather than its own identity. + [null]; for (const identity of identities) retain(node, [...identity, ...key]); } else if (!freshObject(target)) { - for (const scope of origin(node, "opaque-call")) + for (const scope of origin(node, "opaque-call", node)) retain(node, scope); } } } } } - const target = t.isAssignmentExpression(node) - ? node.left - : t.isUpdateExpression(node) - ? node.argument - : t.isUnaryExpression(node, { operator: "delete" }) - ? node.argument - : t.isForOfStatement(node) || t.isForInStatement(node) - ? node.left - : undefined; - if (target === undefined) return; + const target = semanticWriteSiteTarget(node); + if (target === null) return; for (const leaf of semanticWriteTargets(target)) { if (t.isIdentifier(leaf)) { - for (const identity of origin(leaf, "binding")) + for (const identity of origin(leaf, "binding", node)) retain(node, identity); continue; } @@ -159,7 +156,7 @@ export function* collectJavaScriptIdentityWritesSteps( ) continue; const key = semanticStaticPropertyKey(member.property, member.computed); - for (const identity of origin(member.object, "member")) + for (const identity of origin(member.object, "member", node)) retain(node, [...identity, key]); } }, diff --git a/src/domain/javascript/javascriptExportShapeComparisonSchemas.test.ts b/src/domain/javascript/javascriptExportShapeComparisonSchemas.test.ts index 067452b7eb..82e826d3bd 100644 --- a/src/domain/javascript/javascriptExportShapeComparisonSchemas.test.ts +++ b/src/domain/javascript/javascriptExportShapeComparisonSchemas.test.ts @@ -1,10 +1,31 @@ import { expect, it } from "vitest"; import { + compareJavaScriptExportShapesInputSchema, javaScriptExportShapeComparisonChangeSchema, projectedExportReturnShapesSchema, } from "./javascriptExportShapeComparisonSchemas.js"; +it("accepts exact empty export names in selectors and projections", () => { + expect( + compareJavaScriptExportShapesInputSchema.shape.left_export_name.safeParse( + "", + ).success, + ).toBe(true); + expect( + projectedExportReturnShapesSchema.safeParse({ + ...projectionWithField({ + path: "/kind", + presence: "present", + state: "literal", + value: "ok", + reason: null, + }), + exported_name: "", + }).success, + ).toBe(true); +}); + const projectionWithField = (field: unknown) => ({ semantic_role: "export-return-shapes", module_path: "index.js", diff --git a/src/domain/javascript/javascriptExportShapeComparisonSchemas.ts b/src/domain/javascript/javascriptExportShapeComparisonSchemas.ts index 5f94250157..b7ac7ee4f5 100644 --- a/src/domain/javascript/javascriptExportShapeComparisonSchemas.ts +++ b/src/domain/javascript/javascriptExportShapeComparisonSchemas.ts @@ -7,6 +7,7 @@ const evidenceIdSchema = prefixedDigestSchema("ev"); const graphIdSchema = prefixedDigestSchema("jag"); const nodeIdSchema = prefixedDigestSchema("jag_node"); const textSchema = z.string().min(1); +const exportNameSchema = z.string(); const semanticPrimitiveSchema = z.union([ z.string(), z.number().finite(), @@ -85,9 +86,9 @@ export const compareJavaScriptExportShapesInputSchema = z left: evidenceSchema, right: evidenceSchema, left_module_path: textSchema, - left_export_name: textSchema, + left_export_name: exportNameSchema, right_module_path: textSchema, - right_export_name: textSchema, + right_export_name: exportNameSchema, }) .superRefine(validateJavaScriptExportShapeIdentities); @@ -170,7 +171,7 @@ export const projectedExportReturnShapesSchema = z .strictObject({ semantic_role: z.literal("export-return-shapes"), module_path: textSchema, - exported_name: textSchema, + exported_name: exportNameSchema, callable_id: textSchema, callable_kind: z.enum(["function", "class", "method"]), static_return_shapes: z.array(projectedReturnShapeSchema), @@ -198,7 +199,7 @@ export const projectedExportReturnShapesSchema = z const exportCandidateSchema = z.strictObject({ node_id: nodeIdSchema, module_path: textSchema, - export_name: textSchema, + export_name: exportNameSchema, matches_requested_module: z.boolean(), matches_requested_export: z.boolean(), }); @@ -207,7 +208,7 @@ const selectorResultShape = { evidence_id: evidenceIdSchema, graph_id: graphIdSchema, requested_module_path: textSchema, - requested_export_name: textSchema, + requested_export_name: exportNameSchema, candidates: z.array(exportCandidateSchema), omitted_candidates: z.number().int().min(0), }; diff --git a/src/domain/javascript/javascriptExportShapeVariants.ts b/src/domain/javascript/javascriptExportShapeVariants.ts index bedf57cbe2..c4f0839acf 100644 --- a/src/domain/javascript/javascriptExportShapeVariants.ts +++ b/src/domain/javascript/javascriptExportShapeVariants.ts @@ -98,7 +98,7 @@ export const buildJavaScriptExportShapeChanges = ( input.rightSelection.selection.status !== "selected" ) return [selectionUnknownChange(input)]; - if (input.leftShapes.length === 0 || input.rightShapes.length === 0) + if (input.leftShapes.length === 0 && input.rightShapes.length === 0) return [emptyShapeUnknownChange(input)]; const paired = input.pairing.pairs.flatMap((pair) => { const left = input.leftShapes[pair.leftIndex]; diff --git a/src/domain/javascript/javascriptFeatureSeed.array_members.test.ts b/src/domain/javascript/javascriptFeatureSeed.array_members.test.ts index 50f6ceb095..f8a57c53ea 100644 --- a/src/domain/javascript/javascriptFeatureSeed.array_members.test.ts +++ b/src/domain/javascript/javascriptFeatureSeed.array_members.test.ts @@ -165,7 +165,6 @@ describe("feature seed selection controls", () => { { kind: "module", nodeKind: "javascript-module", field: "original_source" }, { kind: "native-export", nodeKind: "native-export", field: "label" }, { kind: "native-export", nodeKind: "native-export", field: "key" }, - { kind: "route", nodeKind: "endpoint", field: "label" }, { kind: "route", nodeKind: "endpoint", field: "value" }, { kind: "route", nodeKind: "endpoint", field: "path" }, { kind: "channel", nodeKind: "ipc-channel", field: "label" }, @@ -192,6 +191,30 @@ describe("feature seed selection controls", () => { }, ); + it("excludes route labels while preserving generic scalar and array matching", () => { + const scalar = memberNode("endpoint", { + endpoint_kind: "route", + label: "route endpoint", + }); + const array = memberNode("endpoint", { + endpoint_kind: "route", + label: ["route endpoint"], + }); + + expect( + findApplicationFeatureSeeds([scalar], seed("route", "route endpoint")), + ).toEqual([]); + expect( + findApplicationFeatureSeeds([array], seed("route", "route endpoint")), + ).toEqual([]); + expect( + findApplicationFeatureSeeds([scalar], seed("string", "route endpoint")), + ).toEqual([propertyMatch(scalar, "label")]); + expect( + findApplicationFeatureSeeds([array], seed("string", "route endpoint")), + ).toEqual([propertyMatch(array, "label[0]")]); + }); + it("keeps generic matching and scalar label and identity selection", () => { const node = memberNode("context-bridge-api", { members: ["openProject"] }); expect(findApplicationFeatureSeeds([node], seed("string"))).toEqual([ diff --git a/src/domain/javascript/javascriptFeatureSeed.ts b/src/domain/javascript/javascriptFeatureSeed.ts index d5f06b8172..2cd238487d 100644 --- a/src/domain/javascript/javascriptFeatureSeed.ts +++ b/src/domain/javascript/javascriptFeatureSeed.ts @@ -40,17 +40,9 @@ const seedMatch = ( node: ApplicationNode, seed: ApplicationFeatureSeed, ): ApplicationFeatureSeedMatch[] => { - if (seed.kind === "node-id") - return node.node_id === seed.value - ? [ - { - node_id: node.node_id, - kind: node.kind, - basis: "node-id", - field: "node_id", - }, - ] - : []; + const mode = featureSeedMatchMode(seed); + const caseSensitive = seed.case_sensitive; + const query = normalized(seed.value, caseSensitive); if (!kindMatches(node, seed.kind)) return []; const requestedEntryPath = seed.kind === "api" && "entry_path" in seed ? seed.entry_path : undefined; @@ -69,9 +61,6 @@ const seedMatch = ( }) ) return []; - const mode = featureSeedMatchMode(seed); - const caseSensitive = seed.case_sensitive; - const query = normalized(seed.value, caseSensitive); const match = candidateFields(node, seed.kind).find((candidate) => { const value = normalized(candidate.value, caseSensitive); return mode === "exact" ? value === query : value.includes(query); @@ -90,9 +79,9 @@ const seedMatch = ( const kindMatches = ( node: ApplicationNode, - kind: Exclude, + kind: ApplicationFeatureSeed["kind"], ): boolean => { - if (kind === "string") return true; + if (kind === "string" || kind === "node-id") return true; if (kind === "route") return ( node.kind === "endpoint" && @@ -116,8 +105,25 @@ const kindMatches = ( const candidateFields = ( node: ApplicationNode, - seedKind: Exclude, + seedKind: ApplicationFeatureSeed["kind"], ): CandidateField[] => { + if (seedKind === "node-id") + return [{ basis: "node-id", field: "node_id", value: node.node_id }]; + if (seedKind === "route") + return node.observations.flatMap((observation, index) => + (["value", "path"] as const).flatMap((key) => { + const value = observation.properties[key]; + return typeof value === "string" + ? [ + { + basis: "property" as const, + field: `observations[${String(index)}].properties.${key}`, + value, + }, + ] + : []; + }), + ); const fields: CandidateField[] = [ ...identityFields(node), ...node.observations.flatMap((observation, index) => [ diff --git a/src/domain/javascript/javascriptIntrinsicInvocation.test.ts b/src/domain/javascript/javascriptIntrinsicInvocation.test.ts new file mode 100644 index 0000000000..66c58bff6f --- /dev/null +++ b/src/domain/javascript/javascriptIntrinsicInvocation.test.ts @@ -0,0 +1,54 @@ +import { expect, it } from "vitest"; +import * as t from "@babel/types"; + +import { createSemanticModuleInterop } from "./javascriptModuleInterop.js"; +import { parseJavaScriptSource } from "./javascriptSourceParser.js"; +import { indexJavaScriptBindingsSteps } from "./javascriptSemanticAnalysis.js"; +import { + completeSemanticSteps, + traverseJavaScriptAst, +} from "./javascriptSemanticTraversal.js"; + +const invocationKinds = (source: string) => { + const parsed = parseJavaScriptSource(source); + if (parsed === null) throw new Error("invocation fixture did not parse"); + const state = completeSemanticSteps( + indexJavaScriptBindingsSteps(parsed.program), + ); + const interop = createSemanticModuleInterop(state); + const invocations: (ReturnType | null)[] = + []; + traverseJavaScriptAst(parsed.program, { + enter: (node) => { + if (t.isCallExpression(node) || t.isOptionalCallExpression(node)) + invocations.push(interop.intrinsicInvocation(node) ?? null); + }, + }); + return invocations; +}; + +it.each([ + 'const change = Object.defineProperty.call; change(null,Object,"create",{value:customCreate});', + 'const {apply:change} = Object.defineProperty; change(null,[Object,"create",{value:customCreate}]);', + 'const change = Reflect.apply.call; change(null,Object.defineProperty,null,[Object,"create",{value:customCreate}]);', +])( + "keeps a bare invocation member alias unknown, not effect-free: %s", + (source) => { + expect(invocationKinds(source)).toEqual([ + expect.objectContaining({ kind: "opaque" }), + ]); + }, +); + +it("keeps a comma-expression intrinsic call unknown when its receiver is erased", () => { + expect( + invocationKinds( + '(0, Object.defineProperty.call)(null, Object, "create", {value: customCreate});', + ), + ).toEqual([ + expect.objectContaining({ + kind: "opaque", + path: ["intrinsic", "Object", "defineProperty", "call"], + }), + ]); +}); diff --git a/src/domain/javascript/javascriptIntrinsicInvocation.ts b/src/domain/javascript/javascriptIntrinsicInvocation.ts index b46f28b42b..6aeb5ab785 100644 --- a/src/domain/javascript/javascriptIntrinsicInvocation.ts +++ b/src/domain/javascript/javascriptIntrinsicInvocation.ts @@ -81,6 +81,15 @@ export const resolveJavaScriptIntrinsicInvocation = ( let explicitReceiver = false; for (;;) { callee = unwrapJavaScriptExpression(callee).node; + if (t.isSequenceExpression(callee)) { + const finalExpression = callee.expressions.at(-1); + if (finalExpression === undefined) return undefined; + // The comma operator returns the final value without its member receiver. + // Keep that lost receiver explicit so intrinsic call/apply stays unknown. + explicitReceiver = true; + callee = finalExpression; + continue; + } if (visited.has(callee)) return undefined; visited.add(callee); let path = intrinsicPath(callee); @@ -129,7 +138,8 @@ export const resolveJavaScriptIntrinsicInvocation = ( : t.isMemberExpression(callee) || t.isOptionalMemberExpression(callee) ? callee.object : undefined; - if (!t.isExpression(callable)) return undefined; + if (!t.isExpression(callable)) + return { kind: "opaque", path, arguments: undefined }; callee = callable; receiver = arguments_?.[0]; explicitReceiver = true; diff --git a/src/domain/javascript/javascriptModuleInterop.ts b/src/domain/javascript/javascriptModuleInterop.ts index c2e26d6fe1..5df0252f81 100644 --- a/src/domain/javascript/javascriptModuleInterop.ts +++ b/src/domain/javascript/javascriptModuleInterop.ts @@ -26,6 +26,7 @@ import { import { SemanticPropertyPathCoverage } from "./javascriptSemanticPropertyPaths.js"; import { readExactJavaScriptLiteral, + runtimeFunctionParameters, unwrapJavaScriptExpression, } from "./javascriptAstValues.js"; @@ -553,7 +554,7 @@ export const createSemanticModuleInterop = ( converted === undefined || initial.projection.length !== 0 || converted.projection.length !== 0 || - state.conditionalInitializers.has(initial.node) + state.conditionallySkippedExpressions.has(initial.node) ) return undefined; const call = converted.node; @@ -662,10 +663,10 @@ const identityFunction = ( !t.isFunction(node) || node.async || node.generator || - node.params.length !== 1 + runtimeFunctionParameters(node).length !== 1 ) return false; - const parameter = node.params[0]; + const parameter = runtimeFunctionParameters(node)[0]; const result = t.isBlockStatement(node.body) ? node.body.body.length === 1 && t.isReturnStatement(node.body.body[0]) ? node.body.body[0].argument diff --git a/src/domain/javascript/javascriptModuleResolutionMode.ts b/src/domain/javascript/javascriptModuleResolutionMode.ts index d14b91bd96..ee2f6edaaf 100644 --- a/src/domain/javascript/javascriptModuleResolutionMode.ts +++ b/src/domain/javascript/javascriptModuleResolutionMode.ts @@ -16,7 +16,7 @@ export const moduleResolutionModeForJavaScriptReference = ( if (referenceKind === "require") return "require"; if (referenceKind === "dynamic-import") return "import"; const extension = posix.extname(sourcePath).toLowerCase(); - if (extension === ".cts") return "require"; + if (extension === ".cjs" || extension === ".cts") return "require"; if (extension === ".mts") return "import"; if (extension === ".ts" || extension === ".tsx") return "compiler-dependent"; return "import"; diff --git a/src/domain/javascript/javascriptRpcEndpoints.test.ts b/src/domain/javascript/javascriptRpcEndpoints.test.ts index b086470f3e..e0f049a388 100644 --- a/src/domain/javascript/javascriptRpcEndpoints.test.ts +++ b/src/domain/javascript/javascriptRpcEndpoints.test.ts @@ -13,6 +13,56 @@ const rpcEndpoints = (source: string) => mechanism.startsWith("trpc:") || mechanism.startsWith("graphql:"), ); +describe("RPC object property ownership", () => { + it.each([ + ['url: "/trpc/old", url: "/trpc/live"', "/trpc/live"], + ['url: "/trpc/old", url: runtimeUrl', null], + ['url: "/trpc/old", ...options', null], + ['url: "/trpc/old", [key]: runtimeUrl', null], + ['url: "/trpc/old", get url() { return runtimeUrl; }', null], + ['url: "/trpc/old", url() {}', null], + ['...options, url: "/trpc/live", headers: {}', "/trpc/live"], + ['[key]: runtimeUrl, url: "/trpc/live"', "/trpc/live"], + ['url: "/trpc/old", ["url"]: "/trpc/live"', "/trpc/live"], + ['url: "/trpc/old", [`url`]: "/trpc/live"', "/trpc/live"], + ['url: "/trpc/live", [`headers`]: {}', "/trpc/live"], + ['url: "/trpc/live", [1n]: {}', "/trpc/live"], + ['url: "/trpc/live", [0x1n]: {}', "/trpc/live"], + ['url: "/trpc/live", [0b1n]: {}', "/trpc/live"], + ['url: "/trpc/live", [-1n]: {}', "/trpc/live"], + ] as const)( + "reads only the final known link URL from {%s}", + (options, value) => { + expect(rpcEndpoints(`httpBatchLink({ ${options} });`)).toEqual( + value === null + ? [] + : [{ kind: "network", value, mechanism: "trpc:link:httpBatchLink" }], + ); + }, + ); + + it.each([ + ['operation: "mutation"', "Old", "graphql:mutation"], + ['name: { kind: "Name", value: "Current" }', "Current", "graphql:query"], + ['kind: "FragmentDefinition"', null, null], + ["name: runtimeName", null, null], + ["get name() { return runtimeName; }", null, null], + ["...options", null, null], + ["[key]: runtimeValue", null, null], + ] as const)( + "honors final GraphQL document properties: %s", + (override, value, mechanism) => { + const source = `const doc = { + kind: "OperationDefinition", operation: "query", + name: { kind: "Name", value: "Old" }, ${override} + };`; + expect(rpcEndpoints(source)).toEqual( + value === null ? [] : [{ kind: "network", value, mechanism }], + ); + }, + ); +}); + describe("tRPC procedure recognition", () => { it("recognizes React hook procedures on a renamed proxy root", () => { expect( @@ -278,6 +328,63 @@ describe("GraphQL operation recognition", () => { ]); }); + it("recognizes operations after leading fragment interpolations only when prior quasis are ignored", () => { + expect( + rpcEndpoints(` + const one = gql\` \${CORE_FIELDS} + query CommentsForPost($postId: ID!) { comments { id } }\`; + const two = gql\`\${CORE_FIELDS} + # another fragment + ,\${MORE_FIELDS} + mutation UpdatePost($id: ID!) { updatePost(id: $id) { id } }\`; + const prose = gql\`fragment prose before interpolation + \${CORE_FIELDS} + query NotAnOperation { viewer { id } }\`; + `), + ).toEqual([ + { kind: "network", value: "CommentsForPost", mechanism: "graphql:query" }, + { kind: "network", value: "UpdatePost", mechanism: "graphql:mutation" }, + ]); + }); + + it.each([ + ["query", "query # header\nGetUser { user }"], + ["mutation", "mutation,# header\r\nGetUser { user }"], + ["subscription", "subscription# header\rGetUser { user }"], + ["query", "query GetUser # selection\n{ user }"], + ["query", "query GetUser { # field\nuser }"], + ["query", "query GetUser( # variable\n$id: ID!) { user }"], + ["query", "query GetUser # directive\n@skip(if: false) { user }"], + ["query", "query GetUser @ # directive name\nkeep { user }"], + [ + "query", + "# prefix\nquery # one\n, # two\nGetUser # three\n{ # four\nuser }", + ], + ])("recognizes %s header comments in %s", (type, text) => { + const expected = [ + { kind: "network", value: "GetUser", mechanism: `graphql:${type}` }, + ]; + expect(rpcEndpoints(`const doc = ${JSON.stringify(text)};`)).toEqual( + expected, + ); + expect(rpcEndpoints("const doc = gql`" + text + "`; ")).toEqual(expected); + }); + + it.each([ + "query # GetUser { user }", + "query GetUser # { user }", + "query GetUser { # user }", + "query # anonymous\n{ user }", + "query GetUser # prose\nparameters", + "query Get# split name\nUser { user }", + "query GetUser @ # missing directive name", + "query GetUser { # fragment\n...UserFields }", + "query\u00a0GetUser { user }", + "query\vGetUser { user }", + ])("keeps incomplete or unsupported header text unknown: %s", (text) => { + expect(rpcEndpoints(`const doc = ${JSON.stringify(text)};`)).toEqual([]); + }); + it("retains operations after long ignored prefixes and long headers", () => { const name = "Operation" + "x".repeat(512); const text = @@ -291,7 +398,9 @@ describe("GraphQL operation recognition", () => { { kind: "network", value: name, mechanism: "graphql:query" }, ]); }); +}); +describe("GraphQL document ownership", () => { it("recognizes precompiled GraphQL document objects", () => { expect( rpcEndpoints(` @@ -339,6 +448,36 @@ describe("GraphQL operation recognition", () => { { kind: "network", value: "GetUser", mechanism: "graphql:query" }, ]); }); + + it.each([ + `body: "query Old { user }", body: "query Current { user }", locationOffset: { line: 1, column: 1 }`, + `body: "query Old { user }", locationOffset: { line: 1, column: 1 }, ...sourceOptions`, + `body: "query Old { user }", locationOffset: { line: 1, column: 1 }, ...{ body: "query Current { user }" }`, + `...{ body: "query Old { user }" }, ...runtimeOptions, locationOffset: { line: 1, column: 1 }`, + ])( + "does not report overwritten or uncertain GraphQL source bodies: %s", + (sourceFields) => { + expect( + rpcEndpoints(` + const source = { ${sourceFields} }; + const operation = { + kind: "OperationDefinition", + operation: "query", + name: { kind: "Name", value: "Current" }, + loc: { source }, + }; + `), + ).toEqual([ + { kind: "network", value: "Current", mechanism: "graphql:query" }, + ]); + }, + ); + + it("keeps ordinary GraphQL string scans outside Source-shaped objects", () => { + expect(rpcEndpoints('const text = "query Standalone { user }";')).toEqual([ + { kind: "network", value: "Standalone", mechanism: "graphql:query" }, + ]); + }); }); it("keeps existing network endpoint recognition unchanged", () => { diff --git a/src/domain/javascript/javascriptRpcEndpoints.ts b/src/domain/javascript/javascriptRpcEndpoints.ts index 1ccf6d3e5a..08e6f3aa61 100644 --- a/src/domain/javascript/javascriptRpcEndpoints.ts +++ b/src/domain/javascript/javascriptRpcEndpoints.ts @@ -122,54 +122,101 @@ const objectKeys = (node: t.Node | undefined): readonly string[] => }) : []; -const objectStringProperty = ( - node: t.ObjectExpression, - key: string, -): string | undefined => { - for (const property of node.properties) - if ( - t.isObjectProperty(property) && - semanticStaticPropertyKey(property.key, property.computed) === key && - t.isStringLiteral(property.value) - ) - return property.value.value; - return undefined; -}; - +/** Later properties own a key; an intervening dynamic write leaves it unknown. */ const objectProperty = ( node: t.ObjectExpression, key: string, ): t.Node | undefined => { - for (const property of node.properties) - if ( - t.isObjectProperty(property) && - semanticStaticPropertyKey(property.key, property.computed) === key - ) - return property.value; + for (let index = node.properties.length - 1; index >= 0; index -= 1) { + const property = node.properties[index]; + if (property === undefined || t.isSpreadElement(property)) return undefined; + const name = semanticStaticPropertyKey(property.key, property.computed); + if (name === null) return undefined; + if (name === key) + return t.isObjectProperty(property) ? property.value : undefined; + } return undefined; }; +const objectStringProperty = ( + node: t.ObjectExpression, + key: string, +): string | undefined => { + const value = objectProperty(node, key); + return t.isStringLiteral(value) ? value.value : undefined; +}; + +/** + * Find source-body string values owned by a GraphQL Source-shaped literal. + * A later duplicate or spread can make the effective `body` unknown, but the + * earlier strings still belong to this object and must not become standalone + * operation findings during the string scan. + */ +const graphqlSourceBodyLiterals = ( + node: t.ObjectExpression, +): readonly t.StringLiteral[] | undefined => { + const bodyStrings = (object: t.ObjectExpression) => + object.properties.flatMap((property) => + t.isObjectProperty(property) && + semanticStaticPropertyKey(property.key, property.computed) === "body" && + t.isStringLiteral(property.value) + ? [property.value] + : [], + ); + const bodies = bodyStrings(node); + const hasLocationOffset = node.properties.some( + (property) => + t.isObjectProperty(property) && + semanticStaticPropertyKey(property.key, property.computed) === + "locationOffset", + ); + if (!hasLocationOffset) return undefined; + + // Inline object spreads can own additional body values which the surrounding + // Source object may select. Include only body-key string literals from those + // spread values; ordinary strings elsewhere in the object remain scannable. + const visited = new WeakSet(); + visited.add(node); + const pending = node.properties.flatMap((property) => { + if (!t.isSpreadElement(property)) return []; + const spread = unwrapJavaScriptExpression(property.argument).node; + return t.isObjectExpression(spread) ? [spread] : []; + }); + while (pending.length > 0) { + const current = pending.pop(); + if (current === undefined || visited.has(current)) continue; + visited.add(current); + bodies.push(...bodyStrings(current)); + for (const property of current.properties) { + if (t.isSpreadElement(property)) { + const spread = unwrapJavaScriptExpression(property.argument).node; + if (t.isObjectExpression(spread)) pending.push(spread); + } + } + } + return bodies.length === 0 ? undefined : bodies; +}; + const GRAPHQL_OPERATION_TYPES: ReadonlySet = new Set([ "query", "mutation", "subscription", ]); -// A named GraphQL operation header (GraphQL October 2021, OperationDefinition): -// type, name, then variable definitions starting with `$`, a directive, or a -// selection set starting with a field name. -const GRAPHQL_OPERATION_HEADER = - /^(query|mutation|subscription)[\s,]+([_A-Za-z][_0-9A-Za-z]*)[\s,]*(?:\([\s,]*\$|@[_A-Za-z]|\{[\s,]*[_A-Za-z])/u; +// GraphQL October 2021 §2.1.7: ignored BOM, tab/space, line endings, commas; +// comments are consumed separately below. const isGraphqlIgnoredCharacter = (character: string): boolean => - character === "," || character === "" || /\s/u.test(character); + character === "," || + character === "\uFEFF" || + character === "\t" || + character === " " || + character === "\n" || + character === "\r"; -/** Recover a named operation from GraphQL document text. */ -export const graphqlOperationFromText = ( - text: string, -): JavaScriptRpcEndpoint | undefined => { - // Skip GraphQL ignored tokens: whitespace, commas, BOM and # comments. +/** Return the first offset after GraphQL ignored tokens. */ +const skipGraphqlIgnored = (text: string, start: number): number => { const limit = text.length; - let index = 0; + let index = start; while (index < limit) { const character = text.charAt(index); if (isGraphqlIgnoredCharacter(character)) index += 1; @@ -177,13 +224,40 @@ export const graphqlOperationFromText = ( while (index < limit && !/[\n\r]/u.test(text.charAt(index))) index += 1; } else break; } - if (index >= limit) return undefined; - const header = GRAPHQL_OPERATION_HEADER.exec(text.slice(index)); - const type = header?.[1]; - const name = header?.[2]; - return type === undefined || name === undefined + return index; +}; + +const graphqlNameAt = ( + text: string, + index: number, +): { readonly value: string; readonly end: number } | undefined => { + const name = /^[_A-Za-z][_0-9A-Za-z]*/u.exec(text.slice(index))?.[0]; + return name === undefined ? undefined - : { value: name, mechanism: `graphql:${type}` }; + : { value: name, end: index + name.length }; +}; + +/** Recover a named operation from GraphQL document text. */ +export const graphqlOperationFromText = ( + text: string, +): JavaScriptRpcEndpoint | undefined => { + const type = graphqlNameAt(text, skipGraphqlIgnored(text, 0)); + if (type === undefined || !GRAPHQL_OPERATION_TYPES.has(type.value)) + return undefined; + const name = graphqlNameAt(text, skipGraphqlIgnored(text, type.end)); + if (name === undefined) return undefined; + const index = skipGraphqlIgnored(text, name.end); + const next = text.charAt(index); + // Keep the existing bounded recognition scope: a named operation followed + // by variable definitions, a directive, or a selection set with a field. + const hasHeader = + next === "(" + ? text.charAt(skipGraphqlIgnored(text, index + 1)) === "$" + : (next === "@" || next === "{") && + graphqlNameAt(text, skipGraphqlIgnored(text, index + 1)) !== undefined; + return hasHeader + ? { value: name.value, mechanism: `graphql:${type.value}` } + : undefined; }; /** Recover a named operation from a precompiled GraphQL document object. */ @@ -258,9 +332,10 @@ export const createJavaScriptRpcScan = ( let marker: boolean | undefined; const hasMarker = (): boolean => (marker ??= TRPC_SOURCE_MARKERS.some((value) => source.includes(value))); - // graphql-tag documents retain their source text in loc.source.body; that - // text repeats the OperationDefinition already recovered from the object. - const documentSourceBodies = new WeakSet(); + // graphql-tag documents retain source text in loc.source.body. Keep each + // direct body literal owned by a Source-shaped object out of the independent + // string scan, including duplicates that a later property or spread hides. + const graphqlDocumentSourceBodies = new WeakSet(); const procedure = ( node: t.CallExpression | t.OptionalCallExpression, @@ -295,26 +370,29 @@ export const createJavaScriptRpcScan = ( if (t.isCallExpression(node) || t.isOptionalCallExpression(node)) return trpcLinkUrl(node) ?? procedure(node); if (t.isObjectExpression(node)) { - const body = objectProperty(node, "body"); - if ( - t.isStringLiteral(body) && - objectProperty(node, "locationOffset") !== undefined - ) { - documentSourceBodies.add(body); + const bodies = graphqlSourceBodyLiterals(node); + if (bodies !== undefined) { + for (const body of bodies) graphqlDocumentSourceBodies.add(body); return undefined; } return graphqlOperationFromDocumentNode(node); } if (t.isStringLiteral(node)) - return documentSourceBodies.has(node) + return graphqlDocumentSourceBodies.has(node) ? undefined : graphqlOperationFromText(node.value); if (t.isTemplateLiteral(node)) { - // A template recovered without a cooked value stays dynamic. - const cooked = node.quasis[0]?.value.cooked; - return cooked === undefined || cooked === null - ? undefined - : graphqlOperationFromText(cooked); + // Interpolations may contain fragment definitions. Continue only + // across quasis that contain ignored tokens; prose or dynamic text + // before an operation prevents claiming a later operation belongs to + // this document prefix. + for (const quasi of node.quasis) { + const cooked = quasi.value.cooked; + if (cooked === undefined || cooked === null) return undefined; + if (skipGraphqlIgnored(cooked, 0) === cooked.length) continue; + return graphqlOperationFromText(cooked); + } + return undefined; } return undefined; }, diff --git a/src/domain/javascript/javascriptRuntimeLoadState.ts b/src/domain/javascript/javascriptRuntimeLoadState.ts index 838f606f64..d2d2cab43a 100644 --- a/src/domain/javascript/javascriptRuntimeLoadState.ts +++ b/src/domain/javascript/javascriptRuntimeLoadState.ts @@ -46,6 +46,7 @@ const classifyLayer = ( ): { readonly states: JavaScriptStaticLoadState[] } => { const nodes = layer.graph.nodes.filter(isLoadStateNode); const direct = new Map(); + const ambiguous = new Set(); for (const match of matches) if ( match.status === "matched" && @@ -53,6 +54,10 @@ const classifyLayer = ( match.static_node_id !== null ) append(direct, match.static_node_id, match.runtime_node_id); + else if (match.status === "ambiguous") + for (const candidate of match.candidate_static_nodes) + if (candidate.static_layer_id === layer.layerId) + ambiguous.add(candidate.static_node_id); const resident = residentNodes(layer, new Set(direct.keys())); const runtimeScope = createStaticRuntimeScope(layer, runtimeEntities); const scopedEntities = runtimeEntities.filter( @@ -77,6 +82,12 @@ const classifyLayer = ( runtimeNodeIds: [], reason: "containing-asset-was-loaded", }); + if (ambiguous.has(node.node_id)) + return state(layer, node, { + status: "unknown", + runtimeNodeIds: [], + reason: "ambiguous-runtime-candidate", + }); const nodeWithinScope = staticNodeWithinRuntimeScope(node, runtimeScope); if (layerComplete && nodeWithinScope) return state(layer, node, { diff --git a/src/domain/javascript/javascriptRuntimeReconciliationMatching.ts b/src/domain/javascript/javascriptRuntimeReconciliationMatching.ts index 96bbe2e599..0c3d6e80b1 100644 --- a/src/domain/javascript/javascriptRuntimeReconciliationMatching.ts +++ b/src/domain/javascript/javascriptRuntimeReconciliationMatching.ts @@ -51,6 +51,13 @@ interface CandidateIndex { string, readonly StaticRuntimeCandidate[] >; + readonly byNativeDeclarationPath: ReadonlyMap< + string, + readonly { + readonly candidate: StaticRuntimeCandidate; + readonly sourcePath: string | null; + }[] + >; readonly byLayerCategoryPath: ReadonlyMap< string, readonly StaticRuntimeCandidate[] @@ -170,9 +177,25 @@ const candidatesWithLocation = ( const located = new Map(); for (const layer of layers) for (const mapping of mappedRuntimePaths(layer, entity)) - for (const candidate of index.byLayerCategoryPath.get( - layerCategoryPathKey(layer.layerId, entity.staticKind, mapping.path), - ) ?? []) { + for (const candidate of [ + ...(index.byLayerCategoryPath.get( + layerCategoryPathKey(layer.layerId, entity.staticKind, mapping.path), + ) ?? []), + ...(mapping.fileStyle === null || mapping.nativeRelativePath === null + ? [] + : ( + index.byNativeDeclarationPath.get( + `${mapping.fileStyle}\0${layerCategoryPathKey(layer.layerId, entity.staticKind, mapping.nativeRelativePath)}`, + ) ?? [] + ) + .filter( + ({ sourcePath }) => + mapping.artifactPrefix === "" || + (sourcePath !== null && + sourcePath.startsWith(`${mapping.artifactPrefix}/`)), + ) + .map(({ candidate }) => candidate)), + ]) { const key = candidateKey(candidate); if (!located.has(key)) located.set(key, { candidate, mapping }); } @@ -189,6 +212,10 @@ const indexCandidates = ( ): CandidateIndex => { const byCategoryDigest = new Map(); const byLayerCategoryPath = new Map(); + const byNativeDeclarationPath = new Map< + string, + { candidate: StaticRuntimeCandidate; sourcePath: string | null }[] + >(); for (const candidate of candidates) { for (const { sha256 } of candidate.digests) appendIndex( @@ -202,14 +229,23 @@ const indexCandidates = ( layerCategoryPathKey(candidate.layer.layerId, candidate.category, path), candidate, ); + for (const style of ["posix", "windows"] as const) + for (const { path, sourcePath } of candidate.nativeDeclarationPaths[ + style + ]) + appendIndex( + byNativeDeclarationPath, + `${style}\0${layerCategoryPathKey(candidate.layer.layerId, candidate.category, path)}`, + { candidate, sourcePath }, + ); } - return { byCategoryDigest, byLayerCategoryPath }; + return { byCategoryDigest, byLayerCategoryPath, byNativeDeclarationPath }; }; -const appendIndex = ( - index: Map, +const appendIndex = ( + index: Map, key: string, - candidate: StaticRuntimeCandidate, + candidate: Value, ): void => { const values = index.get(key); if (values === undefined) index.set(key, [candidate]); diff --git a/src/domain/javascript/javascriptRuntimeReconciliationSchemas.ts b/src/domain/javascript/javascriptRuntimeReconciliationSchemas.ts index ada1c8048e..976b82dbe5 100644 --- a/src/domain/javascript/javascriptRuntimeReconciliationSchemas.ts +++ b/src/domain/javascript/javascriptRuntimeReconciliationSchemas.ts @@ -1,6 +1,7 @@ import { z } from "zod"; import { compareUnicodeCodePoints } from "../unicodeCodePointOrder.js"; +import { logicalPathEscapesRoot } from "../artifactIdentity.js"; import { evidenceSchema } from "../evidence.js"; import { javascriptRuntimeUnresolvedLocationSchema } from "./javascriptRuntimeObservation.js"; import { javascriptApplicationGraphSchema } from "./javascriptApplicationGraph.js"; @@ -16,8 +17,7 @@ const safePrefixSchema = z .refine( (value) => value === "" || - (!value.startsWith("/") && - !value.includes("\\") && + (!logicalPathEscapesRoot(value) && value.split("/").every((part) => !["", ".", ".."].includes(part))), "Artifact prefix must be empty or a safe relative POSIX path", ); @@ -228,6 +228,7 @@ const staticLoadStateSchema = z.strictObject({ "not-observed-in-bounded-capture", "layer-outside-runtime-scope", "static-or-runtime-coverage-incomplete", + "ambiguous-runtime-candidate", ]), }); diff --git a/src/domain/javascript/javascriptRuntimeStaticCandidates.ts b/src/domain/javascript/javascriptRuntimeStaticCandidates.ts index 810ab7f5e1..a7e1afa736 100644 --- a/src/domain/javascript/javascriptRuntimeStaticCandidates.ts +++ b/src/domain/javascript/javascriptRuntimeStaticCandidates.ts @@ -16,6 +16,12 @@ export interface StaticRuntimeCandidate { readonly node: ApplicationNode; readonly category: "renderer" | "javascript" | "worker" | "service-worker"; readonly paths: readonly string[]; + readonly nativeDeclarationPaths: Readonly< + Record< + "posix" | "windows", + readonly { readonly path: string; readonly sourcePath: string | null }[] + > + >; readonly digests: readonly { readonly sha256: string; readonly basis: "content-sha256" | "module-source-sha256"; @@ -24,6 +30,9 @@ export interface StaticRuntimeCandidate { export interface MappedRuntimePath { readonly path: string; + readonly fileStyle: "posix" | "windows" | null; + readonly nativeRelativePath: string | null; + readonly artifactPrefix: string; readonly basis: | "artifact-path" | "operator-file-mapping" @@ -54,9 +63,20 @@ export const mappedRuntimePaths = ( entity: RuntimeReconciliationEntity, ): MappedRuntimePath[] => { const paths: MappedRuntimePath[] = []; + const fileStyle = + entity.location.kind === "file" + ? absoluteFilePathStyle(entity.location.value) + : null; if (entity.location.kind === "file" && layer.result.format === "directory") { const path = pathBelowRoot(layer.result.input_path, entity.location.value); - if (path !== null) paths.push({ path, basis: "artifact-path" }); + if (path !== null) + paths.push({ + path, + fileStyle, + nativeRelativePath: path, + artifactPrefix: "", + basis: "artifact-path", + }); } for (const mapping of layer.runtimeMappings) { const suffix = @@ -70,6 +90,9 @@ export const mappedRuntimePaths = ( if (path === null) continue; paths.push({ path, + fileStyle, + nativeRelativePath: mapping.kind === "file-root" ? suffix : null, + artifactPrefix: normalizeArtifactPrefix(mapping.artifact_prefix), basis: mapping.kind === "file-root" ? "operator-file-mapping" @@ -111,6 +134,7 @@ const candidatesForLayer = ( node, category, paths: pathsByNodeId.get(node.node_id) ?? [], + nativeDeclarationPaths: nativeDeclarationPaths(node), digests: candidateDigests(node), }, ]; @@ -157,7 +181,9 @@ const indexStaticPaths = ( node.node_id, uniqueSorted([ ...nodePaths(node), - ...(relatedBySource.get(node.node_id) ?? []).flatMap(nodePaths), + ...(usesApplicationFileDeclaration(node) + ? [] + : (relatedBySource.get(node.node_id) ?? []).flatMap(nodePaths)), ]), ]), ); @@ -168,9 +194,21 @@ const nodePaths = (node: ApplicationNode): string[] => { if (node.identity.strategy === "canonical-path") paths.push(node.identity.path); for (const observation of node.observations) { - for (const property of ["path", "resolved_path", "declared_path"]) - if (typeof observation.properties[property] === "string") - paths.push(observation.properties[property]); + const rawApplicationFile = + node.kind === "electron-renderer" && + isRawApplicationFileObservation(observation); + if (typeof observation.properties.path === "string") + paths.push(observation.properties.path); + if (rawApplicationFile) continue; + if (typeof observation.properties.resolved_path === "string") + paths.push(observation.properties.resolved_path); + const declaration = observation.properties.declared_path; + if ( + !hasOwnedArtifactPath(node) && + typeof declaration === "string" && + !declaration.includes("\\") + ) + paths.push(declaration); if ( observation.evidence.location.available && observation.evidence.location.value.kind === "artifact-path" @@ -184,6 +222,95 @@ const nodePaths = (node: ApplicationNode): string[] => { ); }; +// Raw filesystem expressions have host syntax only when a captured file location +// establishes that format. Genuinely owned artifact facts take precedence. +const hasOwnedArtifactPath = (node: ApplicationNode): boolean => + node.identity.strategy === "canonical-path" || + node.observations.some( + ({ properties }) => typeof properties.path === "string", + ) || + node.observations.some( + (observation) => + !isRawApplicationFileObservation(observation) && + typeof observation.properties.resolved_path === "string", + ); + +// loadFile receives a raw literal here: the syntax producer does not retain +// computed __dirname expressions. Electron resolves this relative to the app +// root, so its source-relative inference is not an owned runtime target. +const isRawApplicationFileObservation = ( + observation: ApplicationNode["observations"][number], +): boolean => + observation.properties.resolution_context === "filesystem-expression" && + typeof observation.properties.mechanism === "string" && + observation.properties.mechanism.startsWith("call:") && + observation.properties.mechanism.endsWith("loadFile"); + +const usesApplicationFileDeclaration = (node: ApplicationNode): boolean => + node.kind === "electron-renderer" && + !hasOwnedArtifactPath(node) && + node.observations.some(isRawApplicationFileObservation); + +const nativeDeclarationPaths = ( + node: ApplicationNode, +): StaticRuntimeCandidate["nativeDeclarationPaths"] => { + if (hasOwnedArtifactPath(node)) return { posix: [], windows: [] }; + const declarations = node.observations.flatMap((observation) => { + const { properties } = observation; + return properties.resolution_context === "filesystem-expression" && + typeof properties.declared_path === "string" && + (isRawApplicationFileObservation(observation) || + properties.declared_path.includes("\\")) + ? [ + { + path: properties.declared_path, + sourcePath: nativeDeclarationSource(observation), + }, + ] + : []; + }); + const normalized = (windows: boolean) => { + const values = declarations.flatMap(({ path, sourcePath }) => { + if (windows && (win32.isAbsolute(path) || /^[a-z]:/iu.test(path))) + return []; + const normalizedPath = relativeLogicalPath( + windows ? path.replaceAll("\\", "/") : path, + ); + return normalizedPath === null + ? [] + : [{ path: normalizedPath, sourcePath }]; + }); + return [ + ...new Map( + values.map((value) => [JSON.stringify(value), value]), + ).values(), + ].sort( + (left, right) => + compareUnicodeCodePoints(left.path, right.path) || + compareUnicodeCodePoints(left.sourcePath ?? "", right.sourcePath ?? ""), + ); + }; + return { posix: normalized(false), windows: normalized(true) }; +}; + +// A source anchor constrains the selected application namespace; it does not +// supply a module-relative base for the raw filesystem declaration. +const nativeDeclarationSource = ( + observation: ApplicationNode["observations"][number], +): string | null => { + const location = observation.evidence.location; + if (!location.available) return null; + const source = + location.value.kind === "source-range" + ? location.value.source + : location.value.kind === "artifact-path" + ? location.value.path + : null; + return source === null || source === "artifact-root" + ? null + : relativeLogicalPath(source); +}; + const candidateDigests = ( node: ApplicationNode, ): StaticRuntimeCandidate["digests"] => { @@ -217,7 +344,8 @@ const pathBelowRoot = (root: string, value: string): string | null => { rootStyle === "windows" ? win32.relative(root, value) : posix.relative(root, value); - const portableRemainder = remainder.replaceAll("\\", "/"); + const portableRemainder = + rootStyle === "windows" ? remainder.replaceAll("\\", "/") : remainder; if ( remainder === "" || (rootStyle === "windows" && win32.isAbsolute(remainder)) || @@ -288,10 +416,13 @@ const activeArtifactPrefixes = ( : false, ) ) - prefixes.add(mapping.artifact_prefix); + prefixes.add(normalizeArtifactPrefix(mapping.artifact_prefix)); return [...prefixes].sort(compareUnicodeCodePoints); }; +const normalizeArtifactPrefix = (prefix: string): string => + prefix === "" ? "" : normalizeJoinedLogicalPath(prefix); + const pathWithinPrefix = (path: string, prefix: string): boolean => prefix === "" || path === prefix || path.startsWith(`${prefix}/`); @@ -299,6 +430,7 @@ const safeArtifactPath = (prefix: string, suffix: string): string | null => relativeLogicalPath(prefix === "" ? suffix : posix.join(prefix, suffix)); const relativeLogicalPath = (value: string): string | null => { + // These are artifact names or already parsed runtime paths, not declarations. const normalized = normalizeJoinedLogicalPath(value); return normalized !== "" && normalized !== "." && @@ -312,10 +444,24 @@ const uniqueMappedPaths = ( ): MappedRuntimePath[] => [ ...new Map( - values.map((value) => [`${value.basis}:${value.path}`, value]), + values.map((value) => [ + JSON.stringify([ + value.basis, + value.path, + value.fileStyle, + value.nativeRelativePath, + value.artifactPrefix, + ]), + value, + ]), ).values(), ].sort( (left, right) => compareUnicodeCodePoints(left.path, right.path) || - compareUnicodeCodePoints(left.basis, right.basis), + compareUnicodeCodePoints(left.basis, right.basis) || + compareUnicodeCodePoints( + left.nativeRelativePath ?? "", + right.nativeRelativePath ?? "", + ) || + compareUnicodeCodePoints(left.artifactPrefix, right.artifactPrefix), ); diff --git a/src/domain/javascript/javascriptSemanticAmbientWrites.test.ts b/src/domain/javascript/javascriptSemanticAmbientWrites.test.ts index 2eaec7a4b7..a9832ac959 100644 --- a/src/domain/javascript/javascriptSemanticAmbientWrites.test.ts +++ b/src/domain/javascript/javascriptSemanticAmbientWrites.test.ts @@ -1,6 +1,28 @@ import { expect, it } from "vitest"; import { analyzeJavaScriptStaticSource } from "./javascriptStaticAnalysis.js"; +import { indexJavaScriptBindingsSteps } from "./javascriptSemanticAnalysis.js"; +import { completeSemanticSteps } from "./javascriptSemanticTraversal.js"; +import { parseJavaScriptSource } from "./javascriptSourceParser.js"; + +it("keeps wrapper module aliases distinct from global-object module properties", () => { + const parsed = parseJavaScriptSource( + ` + const wrapper = module; + Object.defineProperty(wrapper, "exports", { value: exports }); + Object.defineProperty(globalThis.module, "exports", { value: exports }); + `, + "app.cjs", + ); + if (parsed === null) throw new Error("Expected parsed source"); + const state = completeSemanticSteps( + indexJavaScriptBindingsSteps(parsed.program), + ); + expect(state.ambientWriteEffects.map(({ path }) => path)).toEqual([ + ["commonjs-module", "exports"], + ["global", "module", "exports"], + ]); +}); it.each([ { diff --git a/src/domain/javascript/javascriptSemanticAmbientWrites.ts b/src/domain/javascript/javascriptSemanticAmbientWrites.ts index dd24f35410..3b52f2ccca 100644 --- a/src/domain/javascript/javascriptSemanticAmbientWrites.ts +++ b/src/domain/javascript/javascriptSemanticAmbientWrites.ts @@ -16,6 +16,7 @@ import { type JavaScriptSemanticAnalysisState, } from "./javascriptSemanticState.js"; import { isImportMetaUrl } from "./javascriptStaticAnalysisHelpers.js"; +import { semanticMutationInitializers } from "./javascriptSemanticMutationInitializers.js"; const GLOBAL_OBJECT_NAMES = ["globalThis", "global", "window", "self"]; @@ -51,13 +52,17 @@ const createAmbientPathResolver = ( possibleWrites: boolean, ) => { const cached = new WeakMap(); - return (node: t.Node): readonly AmbientPath[] => { - const prior = cached.get(node); + return (node: t.Node, capture?: t.Node): readonly AmbientPath[] => { + const prior = possibleWrites ? undefined : cached.get(node); if (prior !== undefined) return prior; const result = new Map(); - const pending: { node: t.Node; suffix: AmbientProjection }[] = [ - { node, suffix: [] }, - ]; + // The caller's write site selects initializers for the written target. + // Reads reached through an initializer occur at their own position. + const pending: { + node: t.Node; + suffix: AmbientProjection; + readsAtSelf?: true; + }[] = [{ node, suffix: [] }]; // Join projections at their producing expression, rather than enumerating // alias histories. Each key can only widen to a wildcard, and conflicting // depths widen to unknown, so diamonds and cycles have finite state. @@ -104,7 +109,11 @@ const createAmbientPathResolver = ( // A widened projection is a possible write, never proof of an intrinsic. if (suffix === undefined && !possibleWrites) continue; const continueWith = (node: t.Node, path: AmbientProjection = suffix) => - pending.push({ node, suffix: path }); + pending.push({ + node, + suffix: path, + ...(current.readsAtSelf && { readsAtSelf: true }), + }); if ( t.isMetaProperty(root) && root.meta.name === "import" && @@ -114,6 +123,12 @@ const createAmbientPathResolver = ( continue; } if (t.isIdentifier(root)) { + // The wrapper module object is distinct from globalThis.module. Keep + // its identity through the same alias/projection owner used for writes. + if (isUnshadowedGlobal(root, state, "module")) { + retain(["commonjs-module", ...(suffix ?? [])]); + continue; + } if (globalObjectName(root, state) !== undefined) { retain(["global", ...(suffix ?? [])]); continue; @@ -125,14 +140,19 @@ const createAmbientPathResolver = ( const binding = resolveSemanticBindingState(state, root, root.name); if (binding === undefined) continue; const stable = semanticStableInitializer(binding, state); + const readAt = current.readsAtSelf ? root : capture; const initializers = possibleWrites - ? binding.initializers + ? readAt === undefined + ? binding.initializers + : semanticMutationInitializers(binding, readAt, state.parentsByNode) + .initializers : stable === undefined ? [] : [stable]; for (const initializer of initializers) pending.push({ node: initializer.node, + readsAtSelf: true, suffix: suffix === undefined ? undefined @@ -221,7 +241,7 @@ const createAmbientPathResolver = ( } } const paths = [...result.values()]; - cached.set(node, paths); + if (!possibleWrites) cached.set(node, paths); return paths; }; }; @@ -248,7 +268,7 @@ export function* indexJavaScriptAmbientWritesSteps( }); yield* collectJavaScriptIdentityWritesSteps( program, - (root, kind) => { + (root, kind, writeSite) => { if (kind === "opaque-call") return []; // Reassigning a local alias changes the binding, not the referenced object. if (kind === "binding") @@ -256,13 +276,14 @@ export function* indexJavaScriptAmbientWritesSteps( isUnshadowedGlobal(root, state, root.name) ? [["global", root.name]] : []; - return ambientWrites(root); + return ambientWrites(root, writeSite); }, interop.intrinsicInvocation, interop.staticString, interop.isFreshObject, - (_node, path) => { + (node, path) => { state.ambientWrites.retain(path); + state.ambientWriteEffects.push({ path, node }); }, ); } diff --git a/src/domain/javascript/javascriptSemanticAnalysis.accessor_writes.test.ts b/src/domain/javascript/javascriptSemanticAnalysis.accessor_writes.test.ts new file mode 100644 index 0000000000..1b52c87fa7 --- /dev/null +++ b/src/domain/javascript/javascriptSemanticAnalysis.accessor_writes.test.ts @@ -0,0 +1,108 @@ +import { expect, it } from "vitest"; + +import { analyzeJavaScriptSemantics } from "./javascriptSemanticAnalysis.js"; + +const returned = (source: string) => { + const ir = analyzeJavaScriptSemantics(source); + const callable = ir.callables.find(({ name }) => name === "observe"); + return callable?.returnSites[0]?.value; +}; + +it("keeps members unknown after a write through a local setter", () => { + // The setter runs on assignment and may replace any member of `this`. + expect( + returned(` + export function observe() { + const state = { value: 1, set update(next) { this.value = next; } }; + state.update = 2; + return state; + } + `), + ).toMatchObject({ status: "unknown" }); +}); + +it("keeps sibling literals when the written key is an ordinary property", () => { + expect( + returned(` + export function observe() { + const state = { value: 1, update: 0 }; + state.update = 2; + return state; + } + `), + ).toMatchObject({ + status: "object", + properties: expect.arrayContaining([ + expect.objectContaining({ + name: "value", + value: { status: "literal", value: 1 }, + }), + ]), + }); +}); + +it("keeps nested members unknown after a write through a nested setter", () => { + const value = returned(` + export function observe() { + const state = { + label: "kept", + nested: { value: 1, set update(next) { this.value = next; } }, + }; + state.nested.update = 2; + return state; + } + `); + expect(value).toMatchObject({ + status: "object", + properties: expect.arrayContaining([ + expect.objectContaining({ + name: "label", + value: { status: "literal", value: "kept" }, + }), + expect.objectContaining({ + name: "nested", + value: expect.objectContaining({ status: "unknown" }), + }), + ]), + }); +}); + +it("keeps sibling literals when an accessor is deleted rather than assigned", () => { + expect( + returned(` + export function observe() { + const state = { value: 1, set update(next) {} }; + delete state.update; + return state; + } + `), + ).toMatchObject({ + status: "object", + properties: expect.arrayContaining([ + expect.objectContaining({ + name: "value", + value: { status: "literal", value: 1 }, + }), + ]), + }); +}); + +it("keeps sibling literals when a later data property replaces the setter", () => { + expect( + returned(` + export function observe() { + const state = { value: 1, set update(next) {}, update: 0 }; + state.update = 2; + return state; + } + `), + ).toMatchObject({ + status: "object", + properties: expect.arrayContaining([ + expect.objectContaining({ + name: "value", + value: { status: "literal", value: 1 }, + }), + ]), + }); +}); diff --git a/src/domain/javascript/javascriptSemanticAnalysis.builtin_defaults.test.ts b/src/domain/javascript/javascriptSemanticAnalysis.builtin_defaults.test.ts index 15a4619c3b..42c7cce6e4 100644 --- a/src/domain/javascript/javascriptSemanticAnalysis.builtin_defaults.test.ts +++ b/src/domain/javascript/javascriptSemanticAnalysis.builtin_defaults.test.ts @@ -2,6 +2,428 @@ import { expect, it } from "vitest"; import { analyzeJavaScriptSemantics } from "./javascriptSemanticAnalysis.js"; +it("shares required process writes with globals while keeping earlier observations", () => { + const ir = analyzeJavaScriptSemantics(`const proc=require('process'); + process.cwd(); const before=proc.env.BEFORE; + proc.env={}; const after=process.env.AFTER;`); + expect(ir.configurationOperations.map(({ key }) => key)).toEqual(["BEFORE"]); +}); + +it("does not propagate replacement-object writes from import-equals direct origins", () => { + const ir = analyzeJavaScriptSemantics(`import fs = require('fs'); + const other=require('fs'); fs={}; fs.readFileSync=custom; other.readFileSync('kept');`); + expect(ir.configurationOperations.map(({ key }) => key)).toEqual(["kept"]); +}); + +it("retains original import-equals aliases captured before reassignment", () => { + const ir = analyzeJavaScriptSemantics(`import fs = require('fs'); + const alias=fs; const other=require('fs'); fs={}; alias.readFileSync=custom; other.readFileSync('wrong');`); + expect(ir.configurationOperations).toEqual([]); +}); + +it("keeps computed namespace calls as unknown receiver escapes", () => { + const ir = analyzeJavaScriptSemantics( + `const fs=require('fs'); fs.hook=custom; fs[key](); fs.readFileSync('wrong');`, + ); + expect(ir.configurationOperations).toEqual([]); +}); + +it.each(["(0, fs.readFileSync)", "(assigned = fs.readFileSync)"])( + "checks captured function validity through %s", + (value) => { + const ir = analyzeJavaScriptSemantics(`const fs=require('fs'); let assigned; + const saved=${value}; fs.readFileSync=custom; const late=${value}; saved('kept'); late('wrong');`); + expect(ir.configurationOperations.map(({ key }) => key)).toEqual(["kept"]); + }, +); + +it.each(['process.env={MODE:"local"};', "process.argv=[];", "patch(process);"])( + "shares unbound process object effects with required aliases: %s", + (write) => { + const ir = + analyzeJavaScriptSemantics(`${write} const proc=require('process'); + ${write.includes("argv") ? "const arg=proc.argv[2];" : "const mode=proc.env.MODE;"}`); + expect(ir.configurationOperations).toEqual([]); + }, +); + +it.each([ + "function local(process) {process.env={};}", + "process={}; process.env={};", +])("keeps unrelated process replacements separate: %s", (write) => { + const ir = analyzeJavaScriptSemantics( + `${write} const proc=require('process'); const mode=proc.env.MODE;`, + ); + expect(ir.configurationOperations.map(({ key }) => key)).toEqual(["MODE"]); +}); + +it("preserves exports captured before a later loader overwrite", () => { + const ir = analyzeJavaScriptSemantics(` + const fs=require('node:fs'); fs.readFileSync('before'); + const read=fs.readFileSync; + require('fs').readFileSync('direct-before'); + require=replacement; + fs.readFileSync('after'); read('saved'); + const late=require('fs'); late.readFileSync('wrong'); + require('fs').readFileSync('also-wrong'); + `); + expect(ir.configurationOperations.map(({ key }) => key)).toEqual([ + "before", + "direct-before", + "after", + "saved", + ]); +}); + +it.each([ + `for (;;) { require('fs').readFileSync('wrong'); require=replacement; }`, + `function run() { require('fs').readFileSync('wrong'); require=replacement; }`, + `require=replacement; const fs=require('fs'); fs.readFileSync('wrong'); require=again;`, +])( + "keeps repeated or already replaced loader captures uncertain: %s", + (source) => { + expect(analyzeJavaScriptSemantics(source).configurationOperations).toEqual( + [], + ); + }, +); + +it.each([ + ["fs.readFileSync = custom;", "other.readFileSync('wrong');"], + ["other.readFileSync = custom;", "fs.readFileSync('wrong');"], + ["patch(fs);", "other.readFileSync('wrong');"], +])( + "shares require-mode import-equals effects in both directions: %s", + (write, call) => { + const ir = analyzeJavaScriptSemantics(`import fs = require('node:fs'); + const other = require('fs'); other.readFileSync('before'); ${write} ${call}`); + expect(ir.configurationOperations.map(({ key }) => key)).toEqual([ + "before", + ]); + }, +); + +it.each([ + "import {readFileSync as read} from 'node:fs';", + "const {readFileSync:read}=require('node:fs');", + "const fs=require('fs'); const read=fs.readFileSync;", +])( + "retains captured native function calls after argument escape: %s", + (capture) => { + const ir = analyzeJavaScriptSemantics( + `${capture} observe(read); read('kept');`, + ); + expect(ir.configurationOperations.map(({ key }) => key)).toEqual(["kept"]); + }, +); + +it("distinguishes escaping a method value from escaping its namespace", () => { + const ir = analyzeJavaScriptSemantics(`const fs=require('fs'); + observe(fs.readFileSync); fs.readFileSync('kept'); + observe(fs); fs.readFileSync('wrong');`); + expect(ir.configurationOperations.map(({ key }) => key)).toEqual(["kept"]); +}); + +it("still invalidates escaped process configuration containers", () => { + const ir = analyzeJavaScriptSemantics(`const proc=require('process'); + observe(proc.env); const mode=proc.env.MODE;`); + expect(ir.configurationOperations).toEqual([]); +}); + +it("retains direct require effects without poisoning later cached receivers", () => { + const ir = analyzeJavaScriptSemantics(` + require('node:fs').readFileSync('first'); + const fs = require('fs'); fs.readFileSync('second'); + require('node:fs').createReadStream('stream'); + require('node:child_process').spawn('tool', []); + require('node:timers').setTimeout(callback, 1); + require('node:http').get('http://example.test'); + require('fs').readFileSync = custom; + require('node:fs').readFileSync('wrong'); fs.readFileSync('wrong'); + `); + expect(ir.configurationOperations.map(({ key }) => key)).toEqual([ + "first", + "second", + ]); + expect(ir.resourceOperations).toHaveLength(1); + expect(ir.childProcessSpawns).toHaveLength(1); + expect(ir.timerOperations).toHaveLength(1); + expect(ir.requestOperations).toHaveLength(1); +}); + +it.each([ + "const fs = require('fs'); fs.promises.open = custom;", + "require('node:fs').promises.open = custom;", + "const fsp = require('fs/promises'); fsp.open = custom;", +])("shares fs.promises object writes with fs/promises: %s", (write) => { + const ir = analyzeJavaScriptSemantics(` + const before = require('node:fs/promises'); before.open('before'); + ${write} + const after = require('node:fs/promises'); after.open('wrong'); + `); + expect(ir.resourceOperations).toHaveLength(1); +}); + +it.each([ + "require = () => ({ readFileSync() {} });", + "({ loader: require } = other);", + "for (require of loaders) {}", +])("rejects provenance from an overwritten ambient loader: %s", (write) => { + const ir = analyzeJavaScriptSemantics(`${write} + const fs = require('node:fs'); fs.readFileSync('wrong'); + require('node:fs').readFileSync('also-wrong'); + `); + expect(ir.configurationOperations).toEqual([]); + expect( + ir.bindings.find(({ name }) => name === "fs")?.provenance.status, + ).not.toBe("module"); +}); + +it("keeps local loader reassignment separate from the ambient loader", () => { + const ir = analyzeJavaScriptSemantics(` + function local(require) { require = replacement; } + const fs = require('node:fs'); fs.readFileSync('kept'); + `); + expect(ir.configurationOperations.map(({ key }) => key)).toEqual(["kept"]); +}); + +it("keeps wrapper require separate from global-object property writes", () => { + const ir = analyzeJavaScriptSemantics(` + globalThis.require = replacement; + function polyfill(key) { globalThis[key] = replacement; } + require('node:fs').readFileSync('kept'); + `); + expect(ir.configurationOperations.map(({ key }) => key)).toEqual(["kept"]); +}); + +it.each(["import fs from 'node:fs';", "const fs = require('node:fs');"])( + "retains reads and saved methods before a later write: %s", + (declaration) => { + const ir = analyzeJavaScriptSemantics(`${declaration} + fs.readFileSync('before'); + const saved = fs.readFileSync; + const { readFileSync: destructured } = fs; + const namespace = fs; + fs.readFileSync = custom; + saved('saved'); destructured('destructured'); + namespace.readFileSync('wrong'); fs.readFileSync('wrong'); + `); + expect(ir.configurationOperations.map(({ key }) => key)).toEqual([ + "before", + "saved", + "destructured", + ]); + }, +); + +it.each([ + "const { readFileSync: read } = require('fs');", + "const read = require('fs').readFileSync;", +])("checks cached module writes at the projected capture: %s", (capture) => { + const ir = analyzeJavaScriptSemantics(` + const fs = require('node:fs'); + const saved = require('fs').readFileSync; + fs.readFileSync = custom; + ${capture} + read('wrong'); saved('saved'); + `); + expect(ir.configurationOperations.map(({ key }) => key)).toEqual(["saved"]); +}); + +it("retains captured methods across later escapes without freezing a namespace alias", () => { + const ir = analyzeJavaScriptSemantics(` + const fs = require('node:fs'); const saved = require('fs').readFileSync; + const namespace = require('fs'); patch(fs); + const late = require('fs').readFileSync; + saved('saved'); late('wrong'); namespace.readFileSync('wrong'); + `); + expect(ir.configurationOperations.map(({ key }) => key)).toEqual(["saved"]); +}); + +it("keeps writes in repeated callable and loop activations conservative", () => { + const ir = analyzeJavaScriptSemantics(` + const fs = require('node:fs'); + function run() { fs.readFileSync('wrong'); fs.readFileSync = custom; } + for (;;) { const read = require('fs').readFileSync; read('wrong'); fs.readFileSync = custom; } + `); + expect(ir.configurationOperations).toEqual([]); +}); + +it("does not propagate a reassigned local object's writes into cached exports", () => { + const ir = analyzeJavaScriptSemantics( + `let a = require('node:fs'); const b = require('node:fs'); a = {}; a.readFileSync = custom; b.readFileSync('kept');`, + ); + expect(ir.configurationOperations).toMatchObject([{ key: "kept" }]); +}); + +it("shares repeated require mutations for every built-in effect family", () => { + const ir = analyzeJavaScriptSemantics(` + const fs = require('fs'); const otherFs = require('node:fs'); fs.createReadStream = custom; otherFs.createReadStream('wrong'); + const cp = require('child_process'); const otherCp = require('node:child_process'); cp.spawn = custom; otherCp.spawn('wrong', []); + const timers = require('timers'); const otherTimers = require('node:timers'); timers.setTimeout = custom; otherTimers.setTimeout(callback, 1); + const http = require('http'); const otherHttp = require('node:http'); http.get = custom; otherHttp.get('http://wrong.test'); + const proc = require('process'); const otherProc = require('node:process'); proc.env = replacement; const mode = otherProc.env.MODE; + `); + expect(ir.configurationOperations).toEqual([]); + expect(ir.resourceOperations).toEqual([]); + expect(ir.childProcessSpawns).toEqual([]); + expect(ir.timerOperations).toEqual([]); + expect(ir.requestOperations).toEqual([]); +}); + +it.each(["node:fs", "fs"])( + "shares repeated built-in require identity across %s", + (specifier) => { + const ir = analyzeJavaScriptSemantics( + `const a = require('node:fs'); const b = require('${specifier}'); const alias = b; a.readFileSync = custom; b.createReadStream('kept'); alias.readFileSync('wrong');`, + ); + expect(ir.configurationOperations).toEqual([]); + expect(ir.resourceOperations).toMatchObject([ + { method: "createReadStream" }, + ]); + }, +); + +it("shares unknown escapes across repeated requires without losing earlier effects", () => { + const ir = analyzeJavaScriptSemantics( + `const a = require('node:fs'); const b = require('fs'); b.readFileSync('before'); patch(a); b.readFileSync('after');`, + ); + expect(ir.configurationOperations.map(({ key }) => key)).toEqual(["before"]); +}); + +it("preserves exact default ESM import mutation paths", () => { + const ir = analyzeJavaScriptSemantics( + `import fs from 'node:fs'; fs.writeFileSync = custom; fs.readFileSync('kept'); fs.writeFileSync('replaced');`, + ); + expect(ir.configurationOperations).toMatchObject([ + { kind: "file", key: "kept" }, + ]); +}); + +it("retains built-in effects before later top-level escapes", () => { + const ir = analyzeJavaScriptSemantics(` + const fs = require('node:fs'); const cp = require('node:child_process'); const timers = require('node:timers'); const http = require('node:http'); + fs.readFileSync('before'); fs.createReadStream('before'); cp.spawn('tool', []); timers.setTimeout(callback, 1); http.get('http://example.test'); + patch(fs); patch(cp); patch(timers); patch(http); + fs.readFileSync('after'); fs.createReadStream('after'); cp.spawn('wrong', []); timers.setTimeout(callback, 1); http.get('http://wrong.test'); + `); + expect(ir.configurationOperations).toMatchObject([{ key: "before" }]); + expect(ir.resourceOperations).toHaveLength(1); + expect(ir.childProcessSpawns).toMatchObject([{ command: "tool" }]); + expect(ir.timerOperations).toHaveLength(1); + expect(ir.requestOperations).toMatchObject([ + { endpoint: "http://example.test" }, + ]); +}); + +it.each(["env", "argv"])( + "does not treat a replaced process %s as configuration", + (family) => { + const ir = analyzeJavaScriptSemantics( + `const proc = require('node:process'); proc.${family} = replacement; const value = proc.${family}[${family === "env" ? "'MODE'" : "2"}];`, + ); + expect(ir.configurationOperations).toEqual([]); + }, +); + +it("does not infer temporal safety across repeated callable or loop activations", () => { + const ir = analyzeJavaScriptSemantics( + `const fs = require('node:fs'); function run() { fs.readFileSync('callable'); patch(fs); } for (;;) { fs.readFileSync('loop'); patch(fs); }`, + ); + expect(ir.configurationOperations).toEqual([]); +}); + +it("does not recover a replaced method through a subsequent alias", () => { + const ir = analyzeJavaScriptSemantics( + `const fs = require('node:fs'); fs.readFileSync = custom; const read = fs.readFileSync; read('config.json');`, + ); + expect(ir.configurationOperations).toEqual([]); +}); + +it.each([ + "fs.readFileSync = custom; fs.createReadStream = custom; cp.spawn = custom; timers.setTimeout = custom; http.get = custom;", + "patch(fs); patch(cp); patch(timers); patch(http);", + "const fsAlias = fs; fsAlias.readFileSync = custom; fsAlias.createReadStream = custom; const cpAlias = cp; cpAlias.spawn = custom; timers[key] = custom; delete http.get;", +])("excludes replaced or escaped CommonJS methods: %s", (mutation) => { + const ir = analyzeJavaScriptSemantics(` + const fs = require('node:fs'); const cp = require('node:child_process'); + const timers = require('node:timers'); const http = require('node:http'); + ${mutation} + fs.readFileSync('config.json'); fs.createReadStream('config.json'); + cp.spawn('tool', []); timers.setTimeout(callback, 1); http.get('http://example.test'); + `); + expect(ir.configurationOperations).toEqual([]); + expect(ir.resourceOperations).toEqual([]); + expect(ir.childProcessSpawns).toEqual([]); + expect(ir.timerOperations).toEqual([]); + expect(ir.requestOperations).toEqual([]); +}); + +it("preserves unaffected CommonJS methods after sibling property mutation", () => { + const ir = analyzeJavaScriptSemantics( + `const fs = require('node:fs'); fs.writeFileSync = custom; fs.readFileSync('config.json');`, + ); + expect(ir.configurationOperations).toMatchObject([ + { kind: "file", key: "config.json" }, + ]); +}); + +it.each([ + `const fs = require("node:fs"); const cp = require("node:child_process"); + const timers = require("node:timers"); const http = require("node:http"); + const read = fs.readFileSync; const launch = cp.spawn; + const schedule = timers.setTimeout; const get = http.get;`, + `const fs = require("fs"); const { readFileSync: read } = fs; + const { spawn: launch } = require("child_process"); + const { setTimeout: schedule } = require("timers"); + const { get } = require("http");`, +])("recovers builtin effects from CommonJS provenance: %s", (declarations) => { + const ir = analyzeJavaScriptSemantics(`${declarations} + const text = read("config.json", "utf8"); + const stream = fs.createReadStream("config.json"); stream.destroy(); + const child = launch("tool", []); child.kill("SIGTERM"); + const timer = schedule(callback, 10); clearTimeout(timer); + get("http://example.test"); + `); + expect(ir.configurationOperations).toMatchObject([ + { kind: "file", key: "config.json" }, + ]); + expect( + ir.resourceOperations.map(({ kind, method }) => ({ kind, method })), + ).toEqual([ + { kind: "acquire", method: "createReadStream" }, + { kind: "release", method: "destroy" }, + ]); + expect(ir.childProcessSpawns).toMatchObject([ + { method: "spawn", command: "tool" }, + ]); + expect(ir.childProcessInteractions).toMatchObject([ + { kind: "signal", signalName: "SIGTERM" }, + ]); + expect( + ir.timerOperations.map(({ kind, method }) => ({ kind, method })), + ).toEqual([ + { kind: "schedule", method: "setTimeout" }, + { kind: "cancel", method: "clearTimeout" }, + ]); + expect(ir.requestOperations).toMatchObject([ + { method: "get", endpoint: "http://example.test" }, + ]); +}); + +it("does not promote ambiguous or non-namespace CommonJS origins to builtin effects", () => { + const ir = analyzeJavaScriptSemantics(` + const fake = require("./fs.js"); fake.readFileSync("external"); + const absentDefault = require("node:fs").default; absentDefault.readFileSync("default"); + const { constants } = require("node:fs"); constants.createReadStream("nested"); + let replaced = require("node:fs"); replaced = other; replaced.readFileSync("replaced"); + const mixed = condition ? require("node:fs") : other; mixed.readFileSync("mixed"); + function shadowed(require) { const fs = require("node:fs"); fs.readFileSync("shadowed"); } + `); + expect(ir.configurationOperations).toEqual([]); + expect(ir.resourceOperations).toEqual([]); +}); + it("does not assign built-in APIs to unrelated or shadowed bindings", () => { const ir = analyzeJavaScriptSemantics(` import fs from "./fs.js"; @@ -13,3 +435,244 @@ it("does not assign built-in APIs to unrelated or shadowed bindings", () => { expect(ir.resourceOperations).toEqual([]); }); + +it.each([ + `const fs = require('fs'); class Reader { text = fs.readFileSync('wrong'); } fs.readFileSync = custom; new Reader();`, + `const fs = require('fs'); class Reader { #text = fs.readFileSync('wrong'); } fs.readFileSync = custom; new Reader();`, + `const fs = require('fs'); fs.__defineGetter__('readFileSync', () => custom); fs.readFileSync('wrong');`, + `import proc from 'node:process'; proc.env = {}; const value = process.env.MODE;`, + `import fs from 'node:fs'; fs.readFileSync = custom; const other = require('fs'); other.readFileSync('wrong');`, +])("keeps delayed or replaced built-in identities unknown: %s", (source) => { + expect(analyzeJavaScriptSemantics(source).configurationOperations).toEqual( + [], + ); +}); + +it.each([ + `const fs = require('fs'); const before = fs.readFileSync('kept'), patch = (fs.readFileSync = custom);`, + `globalThis.process = {env: {}}; process.env = {}; const proc = require('process'); const value = proc.env.kept;`, + `import fs from 'node:fs'; fs.writeFileSync = custom; const other = require('fs'); other.readFileSync('kept');`, +])("keeps earlier and independent built-in observations: %s", (source) => { + expect( + analyzeJavaScriptSemantics(source).configurationOperations.map( + ({ key }) => key, + ), + ).toEqual(["kept"]); +}); + +it("shares equivalent alias subproblems and repeated intact receiver facts", () => { + const aliases = Array.from({ length: 30 }, (_, index) => { + const previous = index === 0 ? "fs" : `a${index - 1}`; + return `const a${index} = flag ? ${previous} : ${previous};`; + }); + const calls = Array.from( + { length: 128 }, + (_, index) => `a29.readFileSync('file-${index}');`, + ); + const ir = analyzeJavaScriptSemantics( + ["const fs = require('fs');", ...aliases, ...calls].join("\n"), + ); + expect(ir.configurationOperations).toHaveLength(128); +}); + +it.each([ + `import tls from 'node:tls'; const socket = tls.connect(443, 'example.test'); socket.end();`, + `import * as tls from 'node:tls'; const socket = tls.connect(443, 'example.test'); socket.end();`, + `const tls = require('tls'); const socket = tls.connect(443, 'example.test'); socket.end();`, +])("retains TLS acquisitions and their release: %s", (source) => { + expect( + analyzeJavaScriptSemantics(source).resourceOperations.map( + ({ kind, method }) => ({ kind, method }), + ), + ).toEqual([ + { kind: "acquire", method: "connect" }, + { kind: "release", method: "end" }, + ]); +}); + +it.each([ + `globalThis.process.env = {}; const proc = require('process'); const value = proc.env.MODE;`, + `const host = globalThis; host.process.env = {}; const proc = require('process'); const value = proc.env.MODE;`, + `import * as procNs from 'node:process'; procNs.default.env = {}; const value = process.env.MODE;`, + `globalThis.process.env = { MODE: 'local' }; const proc = require('process'); const value = proc.env.MODE;`, + `const proc = require('process'); proc.env.MODE = 'local'; const value = proc.env.MODE;`, +])("shares writes through the original process object: %s", (source) => { + expect(analyzeJavaScriptSemantics(source).configurationOperations).toEqual( + [], + ); +}); + +it.each([ + [ + `const fs = require('fs'); const result = (fs.readFileSync('kept'), fs.readFileSync = custom);`, + "kept", + ], + [ + `const fs = require('fs'); const read = fs.readFileSync; const result = (read('kept'), fs.readFileSync = custom);`, + "kept", + ], + [ + `const fs = require('fs'); consume(fs.readFileSync('before'), fs.readFileSync = custom);`, + "before", + ], +])( + "retains reads before a later sequence operand: %s", + (source, expectedKey) => { + expect( + analyzeJavaScriptSemantics(source).configurationOperations.map( + ({ key }) => key, + ), + ).toEqual([expectedKey]); + }, +); + +it("does not use source offsets for destructuring assignment evaluation order", () => { + const ir = + analyzeJavaScriptSemantics(`const fs = require('fs'); const target = {}; + [target[fs.readFileSync('wrong')]] = (fs.readFileSync = custom, []);`); + expect(ir.configurationOperations).toEqual([]); +}); + +it("keeps process reads after an alias is reassigned away from the global", () => { + const ir = + analyzeJavaScriptSemantics(`let host = globalThis; host = {process: {env: {}}}; + host.process.env = {}; const proc = require('process'); const value = proc.env.MODE;`); + expect(ir.configurationOperations.map(({ key }) => key)).toEqual(["MODE"]); +}); + +it("rejects CommonJS require after the wrapper module replaces its delegate", () => { + const ir = + analyzeJavaScriptSemantics(`module.require = () => ({ readFileSync() {} }); + const fs = require('node:fs'); fs.readFileSync('unknown');`); + expect(ir.configurationOperations).toEqual([]); +}); + +it.each([ + `module.require.extra = true; const fs = require('fs'); fs.readFileSync('kept');`, + `const owner = module; owner.require = () => ({}); const fs = require('fs'); fs.readFileSync('wrong');`, + `Object.defineProperty(module, 'require', {value: () => ({})}); const fs = require('fs'); fs.readFileSync('wrong');`, +])("tracks only actual wrapper require replacements: %s", (source) => { + const ir = analyzeJavaScriptSemantics(source); + expect(ir.configurationOperations.map(({ key }) => key)).toEqual( + source.includes("extra") ? ["kept"] : [], + ); +}); + +it("preserves a saved promises object after deleting its parent export", () => { + const ir = + analyzeJavaScriptSemantics(`const fs = require('fs'); const fsp = fs.promises; + delete fs.promises; const handle = fsp.open('kept'); handle.close();`); + expect( + ir.resourceOperations.map(({ kind, method }) => ({ kind, method })), + ).toEqual([ + { kind: "acquire", method: "open" }, + { kind: "release", method: "close" }, + ]); +}); + +it("does not infer network acquisition methods from another built-in", () => { + const ir = analyzeJavaScriptSemantics( + `const { open } = require('node:net'); open();`, + ); + expect(ir.resourceOperations).toEqual([]); +}); + +it("recognizes nested ESM namespace receivers as original built-ins", () => { + const ir = + analyzeJavaScriptSemantics(`import * as ns from 'node:fs'; import fs from 'node:fs'; + ns.default.readFileSync('first'); fs.readFileSync('second');`); + expect(ir.configurationOperations.map(({ key }) => key)).toEqual([ + "first", + "second", + ]); +}); + +it("keeps intact fs/promises identity after deleting the fs.promises slot", () => { + const ir = + analyzeJavaScriptSemantics(`const fs=require('fs'); delete fs.promises; + const fsp=require('fs/promises'); const handle=fsp.open('kept'); handle.close();`); + expect( + ir.resourceOperations.map(({ kind, method }) => ({ kind, method })), + ).toEqual([ + { kind: "acquire", method: "open" }, + { kind: "release", method: "close" }, + ]); +}); + +it.each([ + `import fs from 'node:fs'; fs.promises = { open() {} }; fs.promises.open('wrong');`, + `const fs = require('fs'); patch(fs.promises); require('fs/promises').open('wrong');`, + `const fs = require('fs'); patch(fs.promises); require('node:fs/promises').open('wrong');`, +])("keeps replaced or escaped promises receivers unknown: %s", (source) => { + expect(analyzeJavaScriptSemantics(source).resourceOperations).toEqual([]); +}); + +it("follows final sequence expressions for direct built-in method calls", () => { + const ir = analyzeJavaScriptSemantics(`const fs = require('fs'); + (0, fs.readFileSync)('kept'); fs.readFileSync = custom; + (0, fs.readFileSync)('wrong');`); + expect(ir.configurationOperations.map(({ key }) => key)).toEqual(["kept"]); +}); + +it.each([ + `import { env } from 'node:process'; patch(env); process.env.MODE;`, + `import { argv } from 'node:process'; patch(argv); process.argv[2];`, +])( + "shares named process containers with their namespace aliases: %s", + (source) => { + expect(analyzeJavaScriptSemantics(source).configurationOperations).toEqual( + [], + ); + }, +); + +it("retains uncertainty after a computed write to module's require delegate", () => { + const ir = + analyzeJavaScriptSemantics(`const key = 'require'; module[key] = replacement; + const fs = require('fs'); fs.readFileSync('wrong');`); + expect(ir.configurationOperations).toEqual([]); +}); + +it("retains a module namespace captured before its binding is reassigned", () => { + const ir = + analyzeJavaScriptSemantics(`let fs = require('fs'); const saved = fs; + fs = {}; saved.readFileSync('kept');`); + expect(ir.configurationOperations.map(({ key }) => key)).toEqual(["kept"]); +}); + +it("does not let own fs probe calls hide later builtin reads", () => { + const ir = analyzeJavaScriptSemantics( + `const fs=require('fs'); fs.existsSync('probe'); fs.readFileSync('kept');`, + ); + expect(ir.configurationOperations.map(({ key }) => key)).toEqual(["kept"]); +}); + +it("resolves nested ESM default namespace methods", () => { + const ir = analyzeJavaScriptSemantics( + `import * as ns from 'node:fs'; ns.default.readFileSync('kept');`, + ); + expect(ir.configurationOperations.map(({ key }) => key)).toEqual(["kept"]); +}); + +it("tracks process objects passed through qualified global references", () => { + const ir = + analyzeJavaScriptSemantics(`patch(globalThis.process); const proc = require('process'); + const mode = proc.env.MODE;`); + expect(ir.configurationOperations).toEqual([]); +}); + +it("checks a saved child object's method when that function is selected", () => { + const ir = + analyzeJavaScriptSemantics(`const fs = require('fs'); const fsp = fs.promises; + fs.promises.open = custom; const open = fsp.open; open('wrong');`); + expect(ir.resourceOperations).toEqual([]); +}); + +it("retains an earlier captured function after its object method is replaced", () => { + const ir = + analyzeJavaScriptSemantics(`const fs = require('fs'); const fsp = fs.promises; + const open = fsp.open; fs.promises.open = custom; open('kept');`); + expect(ir.resourceOperations).toMatchObject([ + { kind: "acquire", method: "open" }, + ]); +}); diff --git a/src/domain/javascript/javascriptSemanticAnalysis.calls.test.ts b/src/domain/javascript/javascriptSemanticAnalysis.calls.test.ts index f5ac65b7ac..536281b30d 100644 --- a/src/domain/javascript/javascriptSemanticAnalysis.calls.test.ts +++ b/src/domain/javascript/javascriptSemanticAnalysis.calls.test.ts @@ -8,6 +8,28 @@ import { } from "./javascriptSemanticAnalysis.fixture.js"; describe("JavaScript semantic analysis: calls 1", () => { + it("does not assign computed method-name reads to method closures", () => { + const ir = analyzeJavaScriptSemantics(` + function outer() { + const name = "m"; + const object = { [name]() { return 1; }, ["literal"]() { return 2; } }; + class Example { + [name]() { return 3; } + ["literal"]() { return 4; } + } + } + `); + expect(ir.closureCaptures).toEqual([]); + + const bodyRead = analyzeJavaScriptSemantics(` + function outer() { + const value = 1; + const object = { method() { return value; } }; + } + `); + expect(bodyRead.closureCaptures).toHaveLength(1); + }); + it("links exact local calls to parameters, returns, and closure captures", () => { const ir = analyzeJavaScriptSemantics(` const prefix = "rea"; @@ -136,6 +158,121 @@ describe("JavaScript semantic analysis: calls 1", () => { }); }); +describe("JavaScript semantic analysis: computed method keys", () => { + it("evaluates computed method keys in the enclosing callable", () => { + const ir = analyzeJavaScriptSemantics(` + function outer() { + const name = "m"; + const pick = () => "n"; + const obj = { [name]() { return 1; }, [pick()]() { return name; } }; + class C { [name]() { return 2; } } + } + `); + + const outer = onlyCallable(ir, "outer"); + const pickCall = ir.callSites.find( + ({ calleeCallableIds }) => calleeCallableIds.length === 1, + ); + expect(pickCall).toMatchObject({ + callerCallableId: outer.callableId, + resolution: "exact", + }); + // Only the method body that reads `name` captures it; the computed keys + // are evaluated outside every method environment. + expect( + ir.closureCaptures.map(({ callableId, bindingId }) => ({ + owner: ir.callables.find( + (callable) => callable.callableId === callableId, + )?.kind, + binding: bindingById(ir, bindingId)?.name, + })), + ).toEqual([{ owner: "method", binding: "name" }]); + }); + + it("keeps callables nested in a computed method key in their own frames", () => { + const ir = analyzeJavaScriptSemantics(` + function outer() { + const name = "m"; + const pick = () => "n"; + return { [(() => pick())()]() { return name; } }; + } + `); + + const outer = onlyCallable(ir, "outer"); + const pick = onlyCallable(ir, "pick"); + const keyArrow = ir.callables.find( + ({ kind, name }) => kind === "function" && name === null, + ); + const method = ir.callables.find(({ kind }) => kind === "method"); + expect(keyArrow).toBeDefined(); + expect(method).toBeDefined(); + const callerOf = (calleeCallableId: string | undefined) => + ir.callSites.find(({ calleeCallableIds }) => + calleeCallableIds.includes(calleeCallableId ?? ""), + )?.callerCallableId; + expect(callerOf(pick.callableId)).toBe(keyArrow?.callableId); + expect(callerOf(keyArrow?.callableId)).toBe(outer.callableId); + expect( + ir.closureCaptures.map(({ callableId, bindingId }) => ({ + callableId, + binding: bindingById(ir, bindingId)?.name, + })), + ).toEqual( + expect.arrayContaining([ + { callableId: keyArrow?.callableId, binding: "pick" }, + { callableId: method?.callableId, binding: "name" }, + ]), + ); + }); + + it("attributes effects in computed method keys to the enclosing callable", () => { + const ir = analyzeJavaScriptSemantics(` + import { spawn } from "node:child_process"; + function outer(config) { + return { + [fetch("/x")]() {}, + [setTimeout(tick, 1)]() {}, + [Promise.resolve(1)]() {}, + [spawn("ls")]() {}, + [config.key]() {}, + }; + } + class C { [fetch("/y")]() {} } + `); + + const outer = onlyCallable(ir, "outer"); + const owners = ( + operations: readonly { readonly ownerCallableId: string | null }[], + ) => operations.map(({ ownerCallableId }) => ownerCallableId); + const methodIds = new Set( + ir.callables + .filter(({ kind }) => kind === "method") + .map(({ callableId }) => callableId), + ); + expect(methodIds.size).toBe(6); + const classId = onlyCallable(ir, "C").callableId; + expect(owners(ir.requestOperations).sort()).toEqual( + [outer.callableId, classId].sort(), + ); + expect(owners(ir.timerOperations)).toEqual([outer.callableId]); + expect(owners(ir.promiseOperations)).toEqual([outer.callableId]); + expect(owners(ir.childProcessSpawns)).toEqual([outer.callableId]); + expect( + ir.objectOperations.filter(({ propertyPath }) => + propertyPath?.includes("key"), + ), + ).toEqual([expect.objectContaining({ ownerCallableId: outer.callableId })]); + for (const owner of [ + ...owners(ir.requestOperations), + ...owners(ir.timerOperations), + ...owners(ir.promiseOperations), + ...owners(ir.childProcessSpawns), + ...owners(ir.objectOperations), + ]) + expect(methodIds.has(owner ?? "")).toBe(false); + }); +}); + describe("JavaScript semantic analysis: calls 2", () => { it("does not treat a shadowed Promise binding as the intrinsic", () => { const ir = analyzeJavaScriptSemantics(` diff --git a/src/domain/javascript/javascriptSemanticAnalysis.conditional_write.test.ts b/src/domain/javascript/javascriptSemanticAnalysis.conditional_write.test.ts new file mode 100644 index 0000000000..c7aa8a6c91 --- /dev/null +++ b/src/domain/javascript/javascriptSemanticAnalysis.conditional_write.test.ts @@ -0,0 +1,73 @@ +import { describe, expect, it } from "vitest"; +import { analyzeJavaScriptSemantics } from "./javascriptSemanticAnalysis.js"; + +const bindingValue = (source: string, name: string) => + analyzeJavaScriptSemantics(source).bindings.find( + (binding) => binding.name === name, + )?.value; + +describe("JavaScript semantic analysis: conditionally skipped writes", () => { + it("does not promote a skipped assignment through a later read", () => { + const ir = analyzeJavaScriptSemantics( + "let value; if (flag) value = 1; const observed = value;", + ); + + expect(ir.bindings.find(({ name }) => name === "value")?.value.status).toBe( + "unknown", + ); + expect( + ir.bindings.find(({ name }) => name === "observed")?.value.status, + ).toBe("unknown"); + }); + + it("keeps a conditional initializer uncertain while preserving branch joins", () => { + expect(bindingValue("if (flag) { var value = 1; }", "value")?.status).toBe( + "unknown", + ); + expect( + bindingValue("let value; if (flag) value = 1; else value = 2;", "value"), + ).toMatchObject({ status: "ambiguous" }); + }); + + it("preserves writes that execute on every path and definite writes", () => { + expect(bindingValue("let value; value = 1;", "value")).toEqual({ + status: "literal", + value: 1, + }); + expect( + bindingValue("let value; do { value = 1; } while (flag);", "value"), + ).toEqual({ status: "literal", value: 1 }); + expect(bindingValue("let value; if (value = 1) {}", "value")).toEqual({ + status: "literal", + value: 1, + }); + }); + + it("distinguishes for-loop initializers from potentially skipped updates", () => { + expect( + bindingValue("let value; for (value = 1; false;) {}", "value"), + ).toEqual({ status: "literal", value: 1 }); + expect( + bindingValue("let value; for (; false; value = 1) {}", "value")?.status, + ).toBe("unknown"); + }); + + it("tracks arguments skipped by optional chains in a call callee", () => { + for (const call of [ + "foo?.bar(value = 1)", + "foo?.[key](value = 1)", + "foo?.bar?.(value = 1)", + "getThing?.()(value = 1)", + ]) + expect(bindingValue(`let value; ${call};`, "value")?.status).toBe( + "unknown", + ); + }); + + it("keeps arguments after a grouped optional expression definite", () => { + expect(bindingValue("let value; (foo?.bar)(value = 1);", "value")).toEqual({ + status: "literal", + value: 1, + }); + }); +}); diff --git a/src/domain/javascript/javascriptSemanticAnalysis.dataflow.test.ts b/src/domain/javascript/javascriptSemanticAnalysis.dataflow.test.ts index 15738d5b48..2e32b0fe3d 100644 --- a/src/domain/javascript/javascriptSemanticAnalysis.dataflow.test.ts +++ b/src/domain/javascript/javascriptSemanticAnalysis.dataflow.test.ts @@ -210,6 +210,64 @@ describe("JavaScript semantic analysis: read-modify-write", () => { ]); }); + it.each([ + ["destructuring assignment", "[source.count] = [2];"], + ["destructuring assignment with default", "[source.count = 2] = [];"], + [ + "nested destructuring assignment", + "({ value: { count: source.count } } = {});", + ], + ["rest destructuring assignment", "[...source.items] = [];"], + ["for-of head", "for (source.count of [4]) {}"], + ["for-in head", "for (source.count in { key: true }) {}"], + ["delete", "delete source.count;"], + ])("records a member target in %s as a write", (_name, statement) => { + const ir = analyzeJavaScriptSemantics(` + const source = { count: 1, items: [] }; + ${statement} + `); + expect(ir.objectOperations.map(({ kind }) => kind)).toEqual(["write"]); + }); + + it.each([ + ["non-null update", "source.count!++;"], + ["non-null compound assignment", "source.count! += 1;"], + ["cast compound assignment", "(source.count as number) += 1;"], + ["parenthesized update", "(source.count)++;"], + ])( + "retains the property read in a wrapped read-modify-write: %s", + (_name, statement) => { + const ir = analyzeJavaScriptSemantics(` + const source = { count: 1 }; + ${statement} + `); + expect(ir.objectOperations.map(({ kind }) => kind)).toEqual([ + "read", + "write", + ]); + }, + ); + + it.each([ + ["cast plain assignment", "(source.count as number) = 2;"], + ["non-null delete", "delete source.count!;"], + ])("keeps a wrapped %s write-only", (_name, statement) => { + const ir = analyzeJavaScriptSemantics(` + const source = { count: 1 }; + ${statement} + `); + expect(ir.objectOperations.map(({ kind }) => kind)).toEqual(["write"]); + }); + + it("keeps a member in a destructuring default value a read", () => { + const ir = analyzeJavaScriptSemantics(` + const source = { count: 1 }; + let value; + [value = source.count] = []; + `); + expect(ir.objectOperations.map(({ kind }) => kind)).toEqual(["read"]); + }); + it("does not model a compound binding write as a competing replacement", () => { const ir = analyzeJavaScriptSemantics(` let count = 1; @@ -222,6 +280,37 @@ describe("JavaScript semantic analysis: read-modify-write", () => { }); }); +describe("JavaScript semantic analysis: wrapped binding targets", () => { + it("records a wrapped identifier update as a read before a write", () => { + const ir = analyzeJavaScriptSemantics(` + let count = 1; + count! += 1; + (count as number)++; + const copy = count!; + `); + expect( + ir.references + .filter(({ name }) => name === "count") + .map(({ role }) => role), + ).toEqual(["read", "write", "read", "write", "read"]); + expect(topLevelBinding(ir, "count").value).toMatchObject({ + status: "unknown", + }); + }); + + it("records a wrapped identifier assignment as a write of its value", () => { + const ir = analyzeJavaScriptSemantics(` + let count = 1; + count! = 2; + `); + expect( + ir.references + .filter(({ name }) => name === "count") + .map(({ role }) => role), + ).toEqual(["write"]); + }); +}); + describe("JavaScript semantic analysis: dataflow 2", () => { it("retains partial property coverage and empty returns", () => { const partial = analyzeJavaScriptSemantics(` diff --git a/src/domain/javascript/javascriptSemanticAnalysis.empty_keys.test.ts b/src/domain/javascript/javascriptSemanticAnalysis.empty_keys.test.ts index 1ef0d72af8..e97b9e284f 100644 --- a/src/domain/javascript/javascriptSemanticAnalysis.empty_keys.test.ts +++ b/src/domain/javascript/javascriptSemanticAnalysis.empty_keys.test.ts @@ -115,3 +115,18 @@ it("retains exact empty request field names without dynamic fields", () => { ), ).toEqual(["", "body"]); }); + +it("retains the exact empty CommonJS export name", () => { + const ir = analyzeJavaScriptSemantics(` + exports[''] = 1; + module.exports[''] = 2; + exports['ok'] = 3; + exports[key] = 4; + `); + expect(ir.moduleLinks.map(({ exportedName }) => exportedName)).toEqual([ + "", + "", + "ok", + "*", + ]); +}); diff --git a/src/domain/javascript/javascriptSemanticAnalysis.fingerprints.test.ts b/src/domain/javascript/javascriptSemanticAnalysis.fingerprints.test.ts new file mode 100644 index 0000000000..ce88236a50 --- /dev/null +++ b/src/domain/javascript/javascriptSemanticAnalysis.fingerprints.test.ts @@ -0,0 +1,65 @@ +import { expect, it } from "vitest"; + +import { analyzeJavaScriptSemantics } from "./javascriptSemanticAnalysis.js"; + +const components = (source: string) => { + const ir = analyzeJavaScriptSemantics(source); + expect(ir.functionFingerprints).toHaveLength(1); + const fingerprint = ir.functionFingerprints[0]; + if (fingerprint === undefined) + throw new Error("Expected a function fingerprint"); + expect(fingerprint.status).toBe("complete"); + return fingerprint.components; +}; + +it.each([ + ["recognized directives", '"use strict"', '"use asm"'], + ["ordinary directives", '"alpha"', '"beta"'], + ["empty directive", '""', '"alpha"'], + ["escaped strict spelling", '"use strict"', String.raw`"use\x20strict"`], +])("retains %s in the function literal commitment", (_name, left, right) => { + const source = (directive: string) => + `function sample(value) { ${directive}; return value; }`; + expect(components(source(left)).literalSetSha256).not.toBe( + components(source(right)).literalSetSha256, + ); +}); + +it.each([ + [ + 'function sample(value) { "use strict"; return value; }', + "function renamed(input) {\n 'use strict' ;\n return input;\n}", + ], + [ + String.raw`function sample(value) { "use\x20strict"; return value; }`, + String.raw`function renamed(input) { 'use\x20strict'; return input; }`, + ], +])( + "normalizes quote style, formatting and names for %s", + (original, renamed) => { + expect(components(renamed)).toEqual(components(original)); + }, +); + +it("keeps a nested function's directive out of its owner fingerprint", () => { + const analyze = (directive: string) => { + const ir = analyzeJavaScriptSemantics( + `function outer() { function inner(value) { ${directive}; return value; } return 1; }`, + ); + return new Map( + ir.functionFingerprints.map((fingerprint) => [ + ir.callables.find( + ({ callableId }) => callableId === fingerprint.callableId, + )?.name, + fingerprint.components, + ]), + ); + }; + const first = analyze('"use strict"'); + const second = analyze('"ordinary"'); + expect(first.size).toBe(2); + expect(second.get("outer")).toEqual(first.get("outer")); + expect(second.get("inner")?.literalSetSha256).not.toBe( + first.get("inner")?.literalSetSha256, + ); +}); diff --git a/src/domain/javascript/javascriptSemanticAnalysis.invalid_templates.test.ts b/src/domain/javascript/javascriptSemanticAnalysis.invalid_templates.test.ts new file mode 100644 index 0000000000..ea50159a9a --- /dev/null +++ b/src/domain/javascript/javascriptSemanticAnalysis.invalid_templates.test.ts @@ -0,0 +1,38 @@ +import { expect, it } from "vitest"; + +import { analyzeJavaScriptSemantics } from "./javascriptSemanticAnalysis.js"; + +const effects = (call: (value: string) => string, value: string) => { + const ir = analyzeJavaScriptSemantics(` + const { spawn } = require("child_process"); + const { EventEmitter } = require("events"); + ${call(value)} + `); + return { + commands: ir.childProcessSpawns.map(({ command }) => command), + endpoints: ir.requestOperations.map(({ endpoint }) => endpoint), + events: ir.eventOperations.map(({ eventName }) => eventName), + }; +}; + +const calls = (value: string) => + [ + `spawn(${value});`, + `fetch(${value});`, + `new EventEmitter().on(${value}, () => 1);`, + ].join("\n"); + +it("keeps recovered templates without a cooked value dynamic", () => { + // `\\unicode` is an invalid escape: Babel recovers it with `cooked: null`. + expect(effects(calls, "`\\unicode`")).toEqual({ + commands: [null], + endpoints: [null], + events: [null], + }); + // Valid templates keep their cooked value, including escapes. + expect(effects(calls, "`ready\\n`")).toEqual({ + commands: ["ready\n"], + endpoints: ["ready\n"], + events: ["ready\n"], + }); +}); diff --git a/src/domain/javascript/javascriptSemanticAnalysis.member_depth.test.ts b/src/domain/javascript/javascriptSemanticAnalysis.member_depth.test.ts index 964551a7d6..234f05d262 100644 --- a/src/domain/javascript/javascriptSemanticAnalysis.member_depth.test.ts +++ b/src/domain/javascript/javascriptSemanticAnalysis.member_depth.test.ts @@ -134,11 +134,19 @@ it("keeps a shadowed require local through a deep member initializer", () => { it.each([ ["root.first.last()", "root.first.last"], + ["root[`first`].last()", "root.first.last"], + ['root["first" as const].last()', "root.first.last"], ['root[""].last()', "root..last"], + ["root[``].last()", "root..last"], + ['root["" as const].last()', "root..last"], ['("")[""].last()', "last"], ['("")[""]()', ""], ["root[0].last()", "root.0.last"], + ["root[0x1n].last()", "root.1.last"], + ["root[-1n].last()", "root.-1.last"], ["root[key].last()", "root.[computed@5].last"], + ["root[`first${key}`].last()", "root.[computed@5].last"], + ["root[`\\unicode`].last()", "root.[computed@5].last"], ["root?.first.last()", "root.first.last"], [`root${ordinaryMemberChain}.last()`, `root${ordinaryMemberChain}.last`], ])("preserves exact callee syntax (case %#)", (source, expected) => { diff --git a/src/domain/javascript/javascriptSemanticAnalysis.member_mutations.test.ts b/src/domain/javascript/javascriptSemanticAnalysis.member_mutations.test.ts index ee4e602b12..a9145a92cd 100644 --- a/src/domain/javascript/javascriptSemanticAnalysis.member_mutations.test.ts +++ b/src/domain/javascript/javascriptSemanticAnalysis.member_mutations.test.ts @@ -164,6 +164,32 @@ describe("JavaScript semantic values after explicit property mutations", () => { }); }); +it("invalidates a shared value reached through an object-literal prototype getter", () => { + const value = resultValue(` + const shared = { x: 1 }; + const proto = { get v() { return shared; } }; + const box = { __proto__: proto }; + box.v.x = 2; + return shared.x; + `); + expect( + value?.status === "unknown" || + (value?.status === "literal" && value.value === 2), + ).toBe(true); +}); + +it("does not follow an object-literal prototype getter past an own property", () => { + expect( + resultValue(` + const shared = { x: 1 }; + const proto = { get v() { return shared; } }; + const box = { __proto__: proto, v: { x: 3 } }; + box.v.x = 2; + return shared.x; + `), + ).toEqual({ status: "literal", value: 1 }); +}); + describe("JavaScript semantic mutations through getter results (#1494)", () => { it.each([ [ diff --git a/src/domain/javascript/javascriptSemanticAnalysis.native_bindings.test.ts b/src/domain/javascript/javascriptSemanticAnalysis.native_bindings.test.ts index 3c137cc0a1..6083e2bfe7 100644 --- a/src/domain/javascript/javascriptSemanticAnalysis.native_bindings.test.ts +++ b/src/domain/javascript/javascriptSemanticAnalysis.native_bindings.test.ts @@ -170,3 +170,102 @@ describe("JavaScript semantic analysis: native bindings 2", () => { ); }); }); + +describe("JavaScript semantic analysis: child-process signals", () => { + it("keeps unresolved kill signals distinct from default and literal signals", () => { + const ir = analyzeJavaScriptSemantics(` + import { spawn } from "node:child_process"; + function run(signal) { + const child = spawn("/bin/tool"); + child.kill(signal); + child.kill(); + child.kill("SIGINT"); + child.kill(undefined); + child.kill(undefined as NodeJS.Signals | undefined); + child.kill(void 0); + child.kill(void signal); + } + function shadowed(undefined) { + const child = spawn("/bin/tool"); + child.kill(undefined); + } + `); + + expect(ir.childProcessSpawns).toHaveLength(2); + expect( + ir.childProcessInteractions.map(({ signalName }) => signalName), + ).toEqual([ + null, + "SIGTERM", + "SIGINT", + "SIGTERM", + "SIGTERM", + "SIGTERM", + "SIGTERM", + null, + ]); + }); +}); + +describe("JavaScript semantic analysis: configuration value escapes", () => { + const configurationReads = (body: string) => + analyzeJavaScriptSemantics(`function run() { ${body} }`) + .configurationOperations.filter(({ kind }) => kind !== "default") + .map(({ kind, key }) => [kind, key]); + + it.each([ + [ + "an environment value", + "return Number(process.env.PORT);", + "environment", + "PORT", + ], + ["an argv entry", "return Number(process.argv[2]);", "argv", "2"], + ])("keeps a read after passing %s to a call", (_name, body, kind, key) => { + expect(configurationReads(body)).toEqual([[kind, key]]); + }); + + it.each([ + "patch(process.argv[key]); return process.argv[3];", + "patch(process.argv.__proto__); return process.argv[3];", + "const args = process.argv; patch(args[key]); return args[3];", + ])( + "retains uncertainty after an object-valued argv selection escapes: %s", + (body) => { + expect(configurationReads(body)).not.toContainEqual(["argv", "3"]); + }, + ); + + it("preserves an earlier top-level read and unrelated configuration after an argv escape", () => { + const captured = analyzeJavaScriptSemantics( + "const saved = process.argv[3]; patch(process.argv[key]); globalThis.saved = saved;", + ); + expect( + captured.configurationOperations.map(({ kind, key }) => [kind, key]), + ).toContainEqual(["argv", "3"]); + expect( + configurationReads("patch(process.argv[key]); return process.env.PORT;"), + ).toContainEqual(["environment", "PORT"]); + }); + + it.each([ + [ + "a direct argv entry", + "process.argv[2] = {}; return Number(process.argv[2]);", + ], + [ + "an aliased argv entry", + "const argv = process.argv; argv[2] = {}; return Number(process.argv[2]);", + ], + [ + "the argv container", + "mutate(process.argv); return Number(process.argv[2]);", + ], + [ + "the env container", + "mutate(process.env); return Number(process.env.PORT);", + ], + ])("leaves a read unknown after replacing %s", (_name, body) => { + expect(configurationReads(body)).toEqual([]); + }); +}); diff --git a/src/domain/javascript/javascriptSemanticAnalysis.require_links.test.ts b/src/domain/javascript/javascriptSemanticAnalysis.require_links.test.ts index a0d93c3c31..eafb40255b 100644 --- a/src/domain/javascript/javascriptSemanticAnalysis.require_links.test.ts +++ b/src/domain/javascript/javascriptSemanticAnalysis.require_links.test.ts @@ -1,6 +1,11 @@ import { describe, expect, it } from "vitest"; -import { analyzeJavaScriptSemantics } from "./javascriptSemanticAnalysis.js"; +import { + analyzeJavaScriptSemantics, + analyzeParsedJavaScriptSemantics, +} from "./javascriptSemanticAnalysis.js"; +import { parseJavaScriptSource } from "./javascriptSourceParser.js"; +import { moduleResolutionModeForJavaScriptReference } from "./javascriptModuleResolutionMode.js"; import { origin, topLevelBinding, @@ -56,6 +61,91 @@ describe("TypeScript import-equals links", () => { }); }); +describe("CommonJS wrapper parameter redeclarations", () => { + it("does not infer a CommonJS wrapper from resolution mode in an explicit module", () => { + const path = "app/main.mjs"; + const parsed = parseJavaScriptSource( + 'var require; require("./dependency.cjs");', + path, + ); + expect(parsed).not.toBeNull(); + expect(parsed?.program.sourceType).toBe("module"); + const ir = analyzeParsedJavaScriptSemantics( + parsed!, + moduleResolutionModeForJavaScriptReference(path, "static-import"), + ); + expect(ir.moduleLinks).not.toContainEqual( + expect.objectContaining({ specifier: "./dependency.cjs" }), + ); + }); + + it("retains ambient exports and require through uninitialized top-level vars", () => { + const path = "app/main.cjs"; + const parsed = parseJavaScriptSource( + 'var module, exports, require; module.exports = require("./dependency.cjs");', + path, + ); + expect(parsed).not.toBeNull(); + const ir = analyzeParsedJavaScriptSemantics( + parsed!, + moduleResolutionModeForJavaScriptReference(path, "static-import"), + ); + expect(ir.moduleLinks).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + kind: "commonjs-export", + specifier: "./dependency.cjs", + localName: null, + exportedName: "default", + }), + ]), + ); + }); + + it("retains the Node loader call on an uninitialized wrapper var assignment RHS", () => { + const path = "app/main.cjs"; + const parsed = parseJavaScriptSource( + 'var require; require = require("./bootstrap.cjs");', + path, + ); + expect(parsed).not.toBeNull(); + const ir = analyzeParsedJavaScriptSemantics( + parsed!, + moduleResolutionModeForJavaScriptReference(path, "static-import"), + ); + expect(ir.moduleLinks).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + kind: "require", + specifier: "./bootstrap.cjs", + }), + ]), + ); + }); + + it.each([ + 'var require = local; require("./dependency.cjs");', + 'var require; require = replacement; require("./dependency.cjs");', + "var module; module = replacement; module.exports = 1;", + "var exports; exports = replacement; exports.value = 1;", + 'let require; require("./dependency.cjs");', + 'function nested() { var require; require("./dependency.cjs"); }', + ])("keeps non-wrapper declarations as shadows: %s", (source) => { + const path = "app/main.cjs"; + const parsed = parseJavaScriptSource(source, path); + expect(parsed).not.toBeNull(); + const ir = analyzeParsedJavaScriptSemantics( + parsed!, + moduleResolutionModeForJavaScriptReference(path, "static-import"), + ); + expect( + ir.moduleLinks.filter( + ({ kind }) => kind === "require" || kind === "commonjs-export", + ), + ).toEqual([]); + }); +}); + describe("TypeScript type-only module links", () => { it("retains declarations without runtime bindings or callable claims", () => { const ir = analyzeJavaScriptSemantics(` @@ -184,6 +274,7 @@ describe("CommonJS dependency links", () => { const bound = (require("bound") as unknown); const { left, right } = require("destructured"); module.exports = require("forwarded"); + exports = module.exports; exports.member = require("member").value; exports.first = exports.second = require("chained"); `); diff --git a/src/domain/javascript/javascriptSemanticAnalysis.resource_handles.test.ts b/src/domain/javascript/javascriptSemanticAnalysis.resource_handles.test.ts new file mode 100644 index 0000000000..2b0d5ea098 --- /dev/null +++ b/src/domain/javascript/javascriptSemanticAnalysis.resource_handles.test.ts @@ -0,0 +1,59 @@ +import { expect, it } from "vitest"; + +import { analyzeJavaScriptSemantics } from "./javascriptSemanticAnalysis.js"; + +const releases = (source: string) => + analyzeJavaScriptSemantics(source) + .resourceOperations.filter(({ kind }) => kind === "release") + .map(({ method, linkedResourceIds, resolution }) => ({ + method, + linked: linkedResourceIds.length, + resolution, + })); + +it.each([ + [ + "a declaration", + `import { createReadStream } from "node:fs"; + const stream = createReadStream("input.txt");`, + ], + [ + "a later assignment", + `import { createReadStream } from "node:fs"; + let stream; + stream = createReadStream("input.txt");`, + ], + [ + "an awaited handle", + `import { open } from "node:fs/promises"; + const stream = await open("input.txt");`, + ], + [ + "a nullish placeholder before the acquisition", + `import { open } from "node:fs/promises"; + let stream = null; + try { stream = await open("input.txt"); } finally {}`, + ], +])( + "keeps a release complete when the handle holds only %s", + (_name, acquire) => { + expect(releases(`${acquire}\nstream.close();`)).toEqual([ + { method: "close", linked: 1, resolution: "complete" }, + ]); + }, +); + +it.each([ + ["an unconditional replacement", "stream = { destroy() {} };"], + ["a conditional replacement", "if (flag) stream = { destroy() {} };"], + ["a destructuring replacement", "({ stream } = other);"], +])("keeps a release partial after %s", (_name, replacement) => { + expect( + releases(` + import { createReadStream } from "node:fs"; + let stream = createReadStream("input.txt"); + ${replacement} + stream.destroy(); + `), + ).toEqual([{ method: "destroy", linked: 1, resolution: "partial" }]); +}); diff --git a/src/domain/javascript/javascriptSemanticAnalysis.response_endpoints.test.ts b/src/domain/javascript/javascriptSemanticAnalysis.response_endpoints.test.ts new file mode 100644 index 0000000000..425653e6f5 --- /dev/null +++ b/src/domain/javascript/javascriptSemanticAnalysis.response_endpoints.test.ts @@ -0,0 +1,59 @@ +import { expect, it } from "vitest"; + +import { analyzeJavaScriptSemantics } from "./javascriptSemanticAnalysis.js"; + +const responses = (source: string) => + analyzeJavaScriptSemantics(source) + .requestOperations.filter(({ kind }) => kind === "response-consumer") + .map(({ method, endpoint, resolution }) => ({ + method, + endpoint, + resolution, + })); + +it("keeps the endpoint of a uniquely consumed request response", () => { + expect( + responses(` + async function load() { + const a = await fetch("https://example.test/a"); + const b = await fetch("https://example.test/b"); + return [await a.json(), await b.text()]; + } + `), + ).toEqual([ + { + method: "json", + endpoint: "https://example.test/a", + resolution: "complete", + }, + { + method: "text", + endpoint: "https://example.test/b", + resolution: "complete", + }, + ]); +}); + +it("leaves an ambiguous response consumer without an endpoint", () => { + expect( + responses(` + async function load(flag) { + let response = await fetch("https://example.test/a"); + if (flag) response = await fetch("https://example.test/b"); + return response.json(); + } + `), + ).toEqual([{ method: "json", endpoint: null, resolution: "partial" }]); +}); + +it("leaves a response replaced by a non-request value without an endpoint", () => { + expect( + responses(` + async function load(cachedResponse) { + let response = await fetch("https://example.test/a"); + response = cachedResponse; + return response.json(); + } + `), + ).toEqual([expect.objectContaining({ method: "json", endpoint: null })]); +}); diff --git a/src/domain/javascript/javascriptSemanticAnalysis.structure.test.ts b/src/domain/javascript/javascriptSemanticAnalysis.structure.test.ts index 2312eff5d9..b15c1a6814 100644 --- a/src/domain/javascript/javascriptSemanticAnalysis.structure.test.ts +++ b/src/domain/javascript/javascriptSemanticAnalysis.structure.test.ts @@ -10,6 +10,45 @@ import { origin, } from "./javascriptSemanticAnalysis.fixture.js"; +describe("JavaScript semantic analysis: ESM import origins", () => { + it.each(["", "a", "0"])( + "preserves exact ESM import name %j in binding origins", + (name) => { + const ir = analyzeJavaScriptSemantics( + "import { " + + JSON.stringify(name) + + ' as selected } from "./module.mjs"; const alias = selected; export { alias };', + ); + for (const binding of ["selected", "alias"]) + expect(origin(topLevelBinding(ir, binding))).toEqual({ + specifier: "./module.mjs", + importedPath: [name], + }); + expect(ir.moduleLinks).toContainEqual( + expect.objectContaining({ + kind: "import", + importedName: name, + localName: "selected", + }), + ); + }, + ); + + it("keeps default and namespace import origins distinct from an empty named import", () => { + const ir = analyzeJavaScriptSemantics( + 'import value from "./module.mjs"; import * as namespace from "./module.mjs";', + ); + expect(origin(topLevelBinding(ir, "value"))).toEqual({ + specifier: "./module.mjs", + importedPath: ["default"], + }); + expect(origin(topLevelBinding(ir, "namespace"))).toEqual({ + specifier: "./module.mjs", + importedPath: [], + }); + }); +}); + describe("JavaScript semantic analysis: structure 1", () => { it("resolves imports, require destructuring, aliases, assignments, and shadowing", () => { const ir = analyzeJavaScriptSemantics(` @@ -354,7 +393,105 @@ describe("JavaScript semantic analysis: structure 2", () => { ]); }, ); +}); +describe("CommonJS receiver lifetimes", () => { + it.each([ + 'Object.defineProperty(module.exports, "other", { value: exports });', + "Object.assign(module.exports, { other: exports });", + ])( + "keeps the receiver identity intact when mutating its contents: %s", + (mutation) => { + const ir = analyzeJavaScriptSemantics( + `exports.parse = () => 1; ${mutation}`, + ); + const exported = ir.moduleLinks.find( + ({ exportedName }) => exportedName === "parse", + ); + expect(exported?.callableId).toEqual(expect.any(String)); + }, + ); + + it.each([ + "module.exports = {}; exports.parse = () => 1;", + "exports = {}; exports.parse = () => 1;", + "exports.parse = () => 1; module.exports = {};", + "module.exports.parse = () => 1; module.exports = [];", + 'module.exports = {}; exports.status = "enabled";', + "module.exports = exports = {}; module.exports = {}; exports.parse = () => 1;", + 'module.exports = {}; Reflect.set(module, "other", exports); exports.parse = () => 1;', + 'module.exports = {}; Object.defineProperty(globalThis.module, "exports", { value: exports }); exports.parse = () => 1;', + "module.exports = {}; let wrapper = module; wrapper = {}; Object.assign(wrapper, { exports }); exports.parse = () => 1;", + ])( + "excludes a detached CommonJS receiver across statements: %s", + (source) => { + const ir = analyzeJavaScriptSemantics(source); + expect( + ir.moduleLinks.filter( + ({ exportedName }) => + exportedName === "parse" || exportedName === "status", + ), + ).toEqual([]); + }, + ); + + it.each([ + "exports.parse = () => 1;", + "module.exports = {}; module.exports.parse = () => 1;", + "module.exports = {}; exports = module.exports; exports.parse = () => 1;", + "module.exports = exports = {}; exports.parse = () => 1;", + "exports = module.exports = {}; exports.parse = () => 1;", + "module.exports = module.exports; exports.parse = () => 1;", + "module.exports = external; exports = module.exports; exports.parse = () => 1;", + "module.exports = external; module.exports.parse = () => 1;", + "exports.parse = (exports = {} , () => 1);", + "exports.parse = (module.exports = {}); module.exports = exports;", + "(module.exports = {}, exports = module.exports, exports.parse = () => 1);", + ])("retains a connected CommonJS receiver: %s", (source) => { + expect(analyzeJavaScriptSemantics(source).moduleLinks).toContainEqual( + expect.objectContaining({ + kind: "commonjs-export", + exportedName: "parse", + }), + ); + }); + + it.each([ + "module.exports = external; exports.parse = () => 1;", + "if (flag) module.exports = {}; exports.parse = () => 1;", + "flag && (module.exports = {}); exports.parse = () => 1;", + "holder ||= (module.exports = {}); exports.parse = () => 1;", + "invoke?.(module.exports = {}); exports.parse = () => 1;", + "const { value = (module.exports = {}) } = source; exports.parse = () => 1;", + "module[key] = {}; exports.parse = () => 1;", + "exports.parse = () => 1; if (flag) module.exports = {};", + "function replace() { module.exports = {}; } exports.parse = () => 1; replace();", + "function replace() { exports = {}; } replace(); exports.parse = () => 1;", + "module.exports = {}; function install() { exports.parse = () => 1; } install();", + "for (exports of values) {} exports.parse = () => 1;", + "({ exports } = external); exports.parse = () => 1;", + "module = {}; module.exports = {}; module.exports.parse = () => 1;", + 'const holder = {}; ({ [(module.exports = exports, "x")]: holder.x } = (module.exports = {})); exports.parse = () => 1;', + 'const { [(module.exports = exports, "x")]: value } = (module.exports = {}); exports.parse = () => 1;', + "const [value = (module.exports = exports)] = (module.exports = []); exports.parse = () => 1;", + "module.exports = {}; Object.assign(module, { exports }); exports.parse = () => 1;", + 'module.exports = {}; Object.defineProperty(module, "exports", { value: exports }); exports.parse = () => 1;', + 'module.exports = {}; Reflect.set(module, "exports", exports); exports.parse = () => 1;', + "module.exports = {}; const assign = Object.assign; assign.call(null, module, { exports }); exports.parse = () => 1;", + "module.exports = {}; const wrapper = module; Object.assign(wrapper, { exports }); exports.parse = () => 1;", + ])("keeps uncertain CommonJS receiver identity unresolved: %s", (source) => { + expect(analyzeJavaScriptSemantics(source).moduleLinks).toContainEqual( + expect.objectContaining({ + kind: "commonjs-export", + exportedName: "parse", + callableId: null, + localName: null, + }), + ); + }); +}); + +describe("JavaScript semantic analysis: write references", () => { it("classifies destructuring and loop targets as writes, not reads", () => { const ir = analyzeJavaScriptSemantics(` let a, b, c; diff --git a/src/domain/javascript/javascriptSemanticAnalysis.ts b/src/domain/javascript/javascriptSemanticAnalysis.ts index 119fbdd453..df6f4d21b0 100644 --- a/src/domain/javascript/javascriptSemanticAnalysis.ts +++ b/src/domain/javascript/javascriptSemanticAnalysis.ts @@ -18,8 +18,10 @@ import { } from "./javascriptSemanticProjection.js"; import { semanticObjectPatternKeys, + runtimeFunctionParameters, semanticStaticPropertyKey, semanticWriteTargets, + unwrapJavaScriptExpression, } from "./javascriptAstValues.js"; import type { JavaScriptSemanticAnalysisState, @@ -41,6 +43,7 @@ import { traverseJavaScriptAstSteps, } from "./javascriptSemanticTraversal.js"; import { semanticRequireOrigin } from "./javascriptSemanticRequireOrigin.js"; +import { semanticBundledRequireOrigin } from "./javascriptBundledRequireOrigin.js"; import { collectSemanticMemberMutationsSteps } from "./javascriptSemanticMemberMutations.js"; import { indexJavaScriptAmbientWritesSteps } from "./javascriptSemanticAmbientWrites.js"; import { hasUncertainTypeOnlyModuleSideEffect } from "./javascriptModuleDeclaration.js"; @@ -57,7 +60,6 @@ import { ELECTRON_MODULE, } from "./javascriptElectronMemberWrites.js"; import { range } from "./javascriptStaticAnalysisHelpers.js"; -import { propertyName } from "./javascriptAstValues.js"; import { semanticCoverage } from "./javascriptSemanticCoverage.js"; import { semanticResourceLimitsIn } from "./javascriptSemanticResourceLimits.js"; import { @@ -68,6 +70,7 @@ import { import { isJavaScriptRpcClientFactory } from "./javascriptRpcEndpoints.js"; import { createSemanticModuleInterop } from "./javascriptModuleInterop.js"; import { SemanticPropertyPathCoverage } from "./javascriptSemanticPropertyPaths.js"; +import { indexCommonJsExportReceiversSteps } from "./javascriptCommonJsExportReceivers.js"; import type { JavaScriptModuleResolutionMode } from "./javascriptModuleResolutionMode.js"; interface BindPatternInput { @@ -76,6 +79,7 @@ interface BindPatternInput { readonly scope: JavaScriptSemanticScopeState; readonly state: JavaScriptSemanticAnalysisState; readonly mutable: boolean; + readonly declarationKind?: "var" | "let" | "const" | "using" | "await using"; readonly kind?: JavaScriptSemanticDefinition["kind"]; readonly projection?: readonly (string | number | null)[]; readonly referenceOnly?: boolean; @@ -99,6 +103,7 @@ interface AddBindingInput { readonly name: string; readonly kind: JavaScriptSemanticDefinition["kind"]; readonly mutable: boolean; + readonly declarationKind?: "var" | "let" | "const" | "using" | "await using"; readonly definitionNode: t.Node; readonly initializer: t.Node | null; readonly directOrigin?: JavaScriptModuleOrigin; @@ -157,9 +162,11 @@ export function* analyzeParsedJavaScriptSemanticsSteps( ): Generator { const state = createState(file.program, staticImportResolutionMode); yield* collectDefinitionsSteps(file.program, state); + yield* indexJavaScriptAmbientWritesSteps(file.program, state); yield* collectSemanticMemberMutationsSteps(file.program, state); + const commonJsReceivers = yield* indexCommonJsExportReceiversSteps(state); yield* traverseJavaScriptAstSteps(file.program, { - enter: (node) => collectSemanticModuleLink(node, state), + enter: (node) => collectSemanticModuleLink(node, state, commonJsReceivers), }); yield; const references = yield* collectSemanticReferencesSteps(file.program, state); @@ -216,37 +223,6 @@ export function* analyzeParsedJavaScriptSemanticsSteps( }; } -/** Resolve tRPC client roots by lexical identity, excluding overwritten bindings. */ -export function* classifyParsedJavaScriptRpcClientRootsSteps( - file: ParsedJavaScriptSource, -): Generator> { - const state = createState(file.program); - yield* collectDefinitionsSteps(file.program, state); - const roots = new Set(); - const clients = new Map(); - yield* traverseJavaScriptAstSteps(file.program, { - enter: (node) => { - if (!t.isMemberExpression(node) && !t.isOptionalMemberExpression(node)) - return; - const root = node.object; - if (!t.isIdentifier(root)) return; - const binding = resolveSemanticBindingState(state, root, root.name); - if (binding === undefined) return; - let client = clients.get(binding.bindingId); - if (client === undefined) { - const initializer = semanticStableInitializer(binding, state); - client = - initializer !== undefined && - initializer.projection.length === 0 && - isJavaScriptRpcClientFactory(initializer.node); - clients.set(binding.bindingId, client); - } - if (client) roots.add(root); - }, - }); - return roots; -} - /** * Receiver identity facts for the overloaded `.open` syntax, plus * `configuration` for `.get` reads of a `getConfiguration(...)` result. @@ -258,109 +234,193 @@ export type JavaScriptOpenReceiverFact = | "local" | "local-indexed-db" | "local-cache-storage" - | "configuration"; + | "configuration" + | "extension-state"; + +/** Lexical receiver identities that select static endpoint interpretations. */ +export interface JavaScriptReceiverFacts { + readonly openReceivers: ReadonlyMap; + readonly rpcClientRoots: ReadonlySet; +} /** - * Resolve only the lexical receiver identity needed to distinguish browser - * `window.open` from XHR and global storage `.open` calls, and configuration - * `.get` reads from HTTP client calls. This deliberately builds definitions - * without running semantic value or provenance analysis. + * Resolve only the lexical receiver identities needed to distinguish browser + * `window.open` from XHR and global storage `.open` calls, configuration + * `.get` reads from HTTP client calls, and tRPC client roots (excluding + * overwritten bindings). This deliberately builds definitions without running + * semantic value or provenance analysis. */ -export const classifyParsedJavaScriptOpenReceivers = ( +export const classifyParsedJavaScriptReceivers = ( file: ParsedJavaScriptSource, -): ReadonlyMap => - completeSemanticSteps(classifyParsedJavaScriptOpenReceiversSteps(file)); +): JavaScriptReceiverFacts => + completeSemanticSteps(classifyParsedJavaScriptReceiversSteps(file)); -/** Classify `.open` receivers while yielding during lexical and AST scans. */ -export function* classifyParsedJavaScriptOpenReceiversSteps( +/** Classify receivers while yielding during lexical and AST scans. */ +export function* classifyParsedJavaScriptReceiversSteps( file: ParsedJavaScriptSource, -): Generator> { +): Generator { const state = createState(file.program); yield* collectDefinitionsSteps(file.program, state); + return yield* classifyDefinitionReceiversSteps(file.program, state); +} + +/** + * Index bindings and classify receivers from one definitions pass. Receiver + * facts read the definitions-only phase, so they are classified before the + * same state is extended with ambient writes. + */ +export function* indexJavaScriptBindingsWithReceiversSteps( + program: t.Program, +): Generator< + void, + { + readonly bindings: JavaScriptSemanticAnalysisState; + readonly receivers: JavaScriptReceiverFacts; + } +> { + const state = createState(program); + yield* collectDefinitionsSteps(program, state); + const receivers = yield* classifyDefinitionReceiversSteps(program, state); + yield* indexJavaScriptAmbientWritesSteps(program, state); + return { bindings: state, receivers }; +} + +function* classifyDefinitionReceiversSteps( + program: t.Program, + state: JavaScriptSemanticAnalysisState, +): Generator { const facts = new Map(); - yield* traverseJavaScriptAstSteps(file.program, { - enter: (node) => { - if ( - (!t.isCallExpression(node) && !t.isOptionalCallExpression(node)) || - (!t.isMemberExpression(node.callee) && - !t.isOptionalMemberExpression(node.callee)) - ) - return; - const method = semanticStaticPropertyKey( - node.callee.property, - node.callee.computed, + const extensionStateReads: number[] = []; + const roots = new Set(); + const clients = new Map(); + const classifyRpcRoot = (node: t.Node): void => { + if (!t.isMemberExpression(node) && !t.isOptionalMemberExpression(node)) + return; + const root = node.object; + if (!t.isIdentifier(root)) return; + const binding = resolveSemanticBindingState(state, root, root.name); + if (binding === undefined) return; + let client = clients.get(binding.bindingId); + if (client === undefined) { + const initializer = semanticStableInitializer(binding, state); + client = + initializer !== undefined && + initializer.projection.length === 0 && + isJavaScriptRpcClientFactory(initializer.node); + clients.set(binding.bindingId, client); + } + if (client) roots.add(root); + }; + const classifyOpenReceiver = (node: t.Node): void => { + if ( + (!t.isCallExpression(node) && !t.isOptionalCallExpression(node)) || + (!t.isMemberExpression(node.callee) && + !t.isOptionalMemberExpression(node.callee)) + ) + return; + const method = semanticStaticPropertyKey( + node.callee.property, + node.callee.computed, + ); + if (method === "get") { + if (isConfigurationReceiver(state, node.callee.object)) + facts.set(node.start ?? -1, "configuration"); + else if (isExtensionStateMember(node.callee.object)) + extensionStateReads.push(node.start ?? -1); + return; + } + if (method !== "open" || !t.isIdentifier(node.callee.object)) return; + const receiver = node.callee.object.name; + const binding = resolveSemanticBindingState(state, node, receiver); + if (binding === undefined) { + if (semanticResolutionBlocked(state, node, receiver)) return; + const fact = openGlobalFact(receiver); + if (fact !== undefined) facts.set(node.start ?? -1, fact); + return; + } + // A lexically bound receiver overrides ambient browser globals. Mark an + // otherwise unresolved local as local so name-based window.open + // heuristics do not hide its XHR-like overloads. + facts.set(node.start ?? -1, "local"); + if (receiver === "indexedDB" || receiver === "caches") { + facts.set( + node.start ?? -1, + receiver === "indexedDB" ? "local-indexed-db" : "local-cache-storage", ); - if (method === "get") { - if (isConfigurationReceiver(state, node.callee.object)) - facts.set(node.start ?? -1, "configuration"); - return; - } - if (method !== "open" || !t.isIdentifier(node.callee.object)) return; - const receiver = node.callee.object.name; - const binding = resolveSemanticBindingState(state, node, receiver); - if (binding === undefined) { - if (semanticResolutionBlocked(state, node, receiver)) return; - const fact = openGlobalFact(receiver); - if (fact !== undefined) facts.set(node.start ?? -1, fact); - return; - } - // A lexically bound receiver overrides ambient browser globals. Mark an - // otherwise unresolved local as local so name-based window.open - // heuristics do not hide its XHR-like overloads. - facts.set(node.start ?? -1, "local"); - if (receiver === "indexedDB" || receiver === "caches") { - facts.set( - node.start ?? -1, - receiver === "indexedDB" ? "local-indexed-db" : "local-cache-storage", - ); - return; - } - if (binding.mutable || binding.initializers.length !== 1) return; - const initializer = binding.initializers[0]?.node; - if (!t.isIdentifier(initializer)) return; - const initializerBinding = resolveSemanticBindingState( - state, - initializer, - initializer.name, - ); - if (initializerBinding !== undefined) { - if (initializer.name === "indexedDB" || initializer.name === "caches") { - const localFact = - initializer.name === "indexedDB" - ? "local-indexed-db" - : "local-cache-storage"; - facts.set(node.start ?? -1, localFact); - if ( - initializerBinding.mutable || - initializerBinding.initializers.length !== 1 - ) - return; - const nested = initializerBinding.initializers[0]?.node; - if (t.isIdentifier(nested)) { - const nestedFact = resolveGlobalAliasFact( - state, - initializer, - nested.name, - ); - if (nestedFact !== undefined) - facts.set(node.start ?? -1, nestedFact); - } - } else if (initializer.name === "window") { - facts.set(node.start ?? -1, "local"); + return; + } + if (binding.mutable || binding.initializers.length !== 1) return; + const initializer = binding.initializers[0]?.node; + if (!t.isIdentifier(initializer)) return; + const initializerBinding = resolveSemanticBindingState( + state, + initializer, + initializer.name, + ); + if (initializerBinding !== undefined) { + if (initializer.name === "indexedDB" || initializer.name === "caches") { + const localFact = + initializer.name === "indexedDB" + ? "local-indexed-db" + : "local-cache-storage"; + facts.set(node.start ?? -1, localFact); + if ( + initializerBinding.mutable || + initializerBinding.initializers.length !== 1 + ) + return; + const nested = initializerBinding.initializers[0]?.node; + if (t.isIdentifier(nested)) { + const nestedFact = resolveGlobalAliasFact( + state, + initializer, + nested.name, + ); + if (nestedFact !== undefined) facts.set(node.start ?? -1, nestedFact); } - return; + } else if (initializer.name === "window") { + facts.set(node.start ?? -1, "local"); } - const aliasFact = resolveGlobalAliasFact( - state, - initializer, - initializer.name, - ); - if (aliasFact !== undefined) facts.set(node.start ?? -1, aliasFact); + return; + } + const aliasFact = resolveGlobalAliasFact( + state, + initializer, + initializer.name, + ); + if (aliasFact !== undefined) facts.set(node.start ?? -1, aliasFact); + }; + yield* traverseJavaScriptAstSteps(program, { + enter: (node) => { + classifyRpcRoot(node); + classifyOpenReceiver(node); }, }); - return facts; + if (extensionStateReads.length > 0 && importsVsCodeModule(state)) + for (const start of extensionStateReads) + facts.set(start, "extension-state"); + return { openReceivers: facts, rpcClientRoots: roots }; } +/** VS Code passes extension state through ExtensionContext, not an import. */ +const isExtensionStateMember = (receiver: t.Node): boolean => + (t.isMemberExpression(receiver) || t.isOptionalMemberExpression(receiver)) && + (() => { + const key = semanticStaticPropertyKey(receiver.property, receiver.computed); + return key === "globalState" || key === "workspaceState"; + })(); + +/** Require module evidence before treating a familiar member as VS Code state. */ +const importsVsCodeModule = (state: JavaScriptSemanticAnalysisState): boolean => + [...state.bindingsById.values()].some( + (binding) => + binding.directOrigins.some(({ specifier }) => specifier === "vscode") || + semanticRequireOrigin( + semanticStableInitializer(binding, state)?.node, + state, + )?.origin.specifier === "vscode", + ); + /** * Whether a call spells VS Code's `workspace.getConfiguration(section)`, whose * result `.get` reads setting keys rather than request paths. The `workspace` @@ -522,7 +582,7 @@ export function* indexJavaScriptElectronIdentitySteps( const wrapped = interop.callOrigin(root); const origin = wrapped === undefined - ? semanticRequireOrigin(root, state)?.origin + ? electronRequireOrigin(root, state)?.origin : collectedWrites && interop.invalidatedCall(root) ? undefined : wrapped.origin; @@ -606,6 +666,13 @@ const electronExport = ( : null; }; +const electronRequireOrigin = ( + node: t.Node, + state: JavaScriptSemanticAnalysisState, +) => + semanticRequireOrigin(node, state) ?? + semanticBundledRequireOrigin(node, state); + const bindingOrigin = ( binding: JavaScriptSemanticBindingState, state: JavaScriptSemanticAnalysisState, @@ -616,7 +683,7 @@ const bindingOrigin = ( : undefined; const initializer = semanticStableInitializer(binding, state); if (initializer === undefined) return undefined; - const origin = semanticRequireOrigin(initializer.node, state)?.origin; + const origin = electronRequireOrigin(initializer.node, state)?.origin; const projection = initializer.projection; return origin === undefined || !projection.every((key): key is string => typeof key === "string") @@ -637,6 +704,7 @@ const createState = ( bindings: new Map(), }; return { + program, staticImportResolutionMode, parentsByNode: new WeakMap(), scopes: [root], @@ -648,8 +716,12 @@ const createState = ( moduleLinks: [], linkedRequireCalls: new WeakSet(), moduleLinkBindings: new WeakMap(), - conditionalInitializers: new WeakSet(), + defaultExportExpressions: new WeakMap(), + conditionallySkippedExpressions: new WeakSet(), + requiredModuleEffects: new Map(), + ambientBindingWrites: new Map(), ambientWrites: new SemanticPropertyPathCoverage(), + ambientWriteEffects: [], }; }; @@ -664,6 +736,7 @@ function* collectDefinitionsSteps( const assignments: { readonly node: t.Node; readonly scope: JavaScriptSemanticScopeState; + readonly conditionallySkipped: boolean; }[] = []; const functionBodies: { readonly body: t.BlockStatement; @@ -718,7 +791,7 @@ function* collectDefinitionsSteps( functionBodies.push({ body: node, scope: nested, - parameters: parent.params, + parameters: runtimeFunctionParameters(parent), }); } const scope = currentSemanticScope(stack); @@ -739,16 +812,23 @@ function* collectDefinitionsSteps( node.init != null && parent !== null && t.isVariableDeclaration(parent, { kind: "var" }) && - hasConditionalInitializer(readAncestors()) + writeMayBeSkipped(node, readAncestors()) ) - state.conditionalInitializers.add(node.init); + state.conditionallySkippedExpressions.add(node.init); if ( t.isAssignmentExpression(node) || t.isUpdateExpression(node) || t.isForOfStatement(node) || t.isForInStatement(node) ) - assignments.push({ node, scope }); + assignments.push({ + node, + scope, + conditionallySkipped: + t.isForOfStatement(node) || + t.isForInStatement(node) || + writeMayBeSkipped(node, readAncestors()), + }); }, exit: (node) => { for (let count = openedScopes.get(node) ?? 0; count > 0; count--) @@ -757,9 +837,12 @@ function* collectDefinitionsSteps( }); // All declarations must exist before resolving writes, including hoisted // functions and local declarations that shadow an outer binding. - for (const [index, { node, scope }] of assignments.entries()) { + for (const [ + index, + { node, scope, conditionallySkipped }, + ] of assignments.entries()) { if (index % 64 === 0) yield; - bindAssignment(node, scope, state); + bindAssignment(node, scope, state, conditionallySkipped); } for (const [index, { body, scope, parameters }] of functionBodies.entries()) { if (index % 64 === 0) yield; @@ -813,24 +896,72 @@ const copyParameterToBody = (input: { } }; -const hasConditionalInitializer = (ancestors: readonly t.Node[]): boolean => { - // A hoisted var can remain undefined when control flow skips its initializer. - // Conditions outside the owning callable/static block do not govern its body. +const writeMayBeSkipped = ( + node: t.Node, + ancestors: readonly t.Node[], +): boolean => { + let child = node; for (let index = ancestors.length - 1; index >= 0; index -= 1) { - const node = ancestors[index]; - if (t.isFunction(node) || t.isProgram(node) || t.isStaticBlock(node)) - return false; + const parent = ancestors[index]; if ( - t.isIfStatement(node) || - t.isLoop(node) || - t.isSwitchCase(node) || - t.isTryStatement(node) + parent === undefined || + t.isFunction(parent) || + t.isProgram(parent) || + t.isStaticBlock(parent) ) - return true; + return false; + if (isSkippedBranch(parent, child)) return true; + child = parent; } return false; }; +const isSkippedBranch = (parent: t.Node, child: t.Node): boolean => { + if (t.isIfStatement(parent)) + return child === parent.consequent || child === parent.alternate; + if (t.isSwitchCase(parent)) return true; + if (t.isTryStatement(parent)) + return child === parent.block || child === parent.handler; + if (t.isForStatement(parent)) + return child === parent.body || child === parent.update; + if (t.isWhileStatement(parent)) return child === parent.body; + if (t.isDoWhileStatement(parent)) return false; + if (t.isForInStatement(parent) || t.isForOfStatement(parent)) + return child === parent.left || child === parent.body; + if (t.isLogicalExpression(parent)) return child === parent.right; + if (t.isConditionalExpression(parent)) + return child === parent.consequent || child === parent.alternate; + if (t.isOptionalCallExpression(parent) || t.isCallExpression(parent)) + return ( + parent.arguments.includes(child as t.Expression) && + optionalCallArgumentsMayBeSkipped(parent) + ); + if (t.isOptionalMemberExpression(parent)) + return parent.optional && parent.computed && child === parent.property; + return false; +}; + +const optionalCallArgumentsMayBeSkipped = ( + call: t.CallExpression | t.OptionalCallExpression, +): boolean => { + if (t.isOptionalCallExpression(call) && call.optional) return true; + let callee: t.Node = call.callee; + while (true) { + // Parentheses terminate optional-chain short-circuiting. In particular, + // `(foo?.bar)(arg)` evaluates arg before attempting the call. + if (callee.extra?.parenthesized === true) return false; + if (t.isOptionalMemberExpression(callee)) { + if (callee.optional) return true; + callee = callee.object; + } else if (t.isMemberExpression(callee)) callee = callee.object; + else if (t.isOptionalCallExpression(callee)) { + if (callee.optional) return true; + callee = callee.callee; + } else if (t.isCallExpression(callee)) callee = callee.callee; + else return false; + } +}; + const bindOuterDeclaration = ( node: t.Node, scope: JavaScriptSemanticScopeState, @@ -877,6 +1008,9 @@ const bindInnerDeclaration = ( parent !== null && t.isVariableDeclaration(parent) ? parent.kind !== "const" : true, + ...(parent !== null && t.isVariableDeclaration(parent) + ? { declarationKind: parent.kind } + : {}), }); else if (t.isClassExpression(node) && t.isIdentifier(node.id)) addBinding({ @@ -938,32 +1072,37 @@ const bindAssignment = ( node: t.Node, scope: JavaScriptSemanticScopeState, state: JavaScriptSemanticAnalysisState, + conditionallySkipped: boolean, ): void => { if (t.isAssignmentExpression(node)) { - if (t.isIdentifier(node.left)) + const left = unwrapJavaScriptExpression(node.left).node; + if (t.isIdentifier(left)) addAssignment( - node.left, + left, node.operator === "=" || ((node.operator === "||=" || node.operator === "??=") && - !hasIncomingValue(node.left, scope, state)) + !hasIncomingValue(left, scope, state)) ? node.right : node, scope, state, + conditionallySkipped, ); else for (const identifier of semanticWriteTargets(node.left)) if (t.isIdentifier(identifier)) - addAssignment(identifier, node, scope, state); - } else if (t.isUpdateExpression(node) && t.isIdentifier(node.argument)) - addAssignment(node.argument, node, scope, state); - else if (t.isForOfStatement(node) || t.isForInStatement(node)) + addAssignment(identifier, node, scope, state, conditionallySkipped); + } else if (t.isUpdateExpression(node)) { + const argument = unwrapJavaScriptExpression(node.argument).node; + if (t.isIdentifier(argument)) + addAssignment(argument, node, scope, state, conditionallySkipped); + } else if (t.isForOfStatement(node) || t.isForInStatement(node)) for (const pattern of t.isVariableDeclaration(node.left) ? node.left.declarations.map(({ id }) => id) : [node.left]) { for (const identifier of semanticWriteTargets(pattern)) if (t.isIdentifier(identifier)) - addAssignment(identifier, node, scope, state); + addAssignment(identifier, node, scope, state, conditionallySkipped); if (t.isForOfStatement(node)) // Keep the loop value unknown, but preserve references yielded into // each bound slot so writes/escapes can reach their iterable origins. @@ -1077,7 +1216,7 @@ const scopeKind = ( }; const functionParametersHaveExpressions = (node: t.Function): boolean => { - const pending: t.Node[] = [...node.params]; + const pending: t.Node[] = [...runtimeFunctionParameters(node)]; while (pending.length > 0) { const parameter = pending.pop(); if (t.isAssignmentPattern(parameter)) return true; @@ -1109,7 +1248,7 @@ const bindImports = ( ? ["default"] : t.isImportNamespaceSpecifier(specifier) ? [] - : [propertyName(specifier.imported) || "[dynamic]"]; + : [semanticStaticPropertyKey(specifier.imported, false) ?? "[dynamic]"]; addBinding({ state, scope, @@ -1134,7 +1273,7 @@ const bindFunctionLocals = ( scope: JavaScriptSemanticScopeState, state: JavaScriptSemanticAnalysisState, ): void => { - for (const parameter of node.params) + for (const parameter of runtimeFunctionParameters(node)) bindPattern({ pattern: t.isTSParameterProperty(parameter) ? parameter.parameter @@ -1154,6 +1293,7 @@ const bindPattern = (input: BindPatternInput): void => { scope, state, mutable, + declarationKind, kind = "variable", projection = [], referenceOnly = false, @@ -1196,6 +1336,7 @@ const bindPattern = (input: BindPatternInput): void => { name: pattern.name, kind, mutable, + ...(declarationKind === undefined ? {} : { declarationKind }), definitionNode: pattern, initializer, projection, @@ -1298,6 +1439,7 @@ const addBinding = (input: AddBindingInput): void => { name, kind, mutable, + declarationKind, definitionNode, initializer, directOrigin, @@ -1312,6 +1454,8 @@ const addBinding = (input: AddBindingInput): void => { state.bindingsById.set(binding.bindingId, binding); } binding.mutable ||= mutable; + if (declarationKind !== undefined) + binding.declarationKinds.add(declarationKind); binding.definitions.push({ kind, location: range(definitionNode) }); if (initializer !== null) binding.initializers.push({ node: initializer, projection }); @@ -1339,7 +1483,8 @@ const createBinding = ( name, kind, mutable, - mutatedPaths: [], + declarationKinds: new Set(), + memberWrites: [], escapedPaths: [], definitions: [], initializers: [], @@ -1353,14 +1498,24 @@ const addAssignment = ( initializer: t.Node, scope: JavaScriptSemanticScopeState, state: JavaScriptSemanticAnalysisState, + conditionallySkipped: boolean, ): void => { const binding = resolveSemanticBindingFromScope( scope, identifier.name, state, ); - if (binding === undefined) return; + if (binding === undefined) { + // Unbound identifiers can denote host wrapper bindings (such as require), + // which are separate from properties written through the global object. + const writes = state.ambientBindingWrites.get(identifier.name) ?? []; + writes.push(identifier); + state.ambientBindingWrites.set(identifier.name, writes); + return; + } binding.mutable = true; binding.definitions.push({ kind: "assignment", location: range(identifier) }); binding.initializers.push({ node: initializer, projection: [] }); + if (conditionallySkipped) + state.conditionallySkippedExpressions.add(initializer); }; diff --git a/src/domain/javascript/javascriptSemanticAnalysisSteps.test.ts b/src/domain/javascript/javascriptSemanticAnalysisSteps.test.ts index 633cca1216..1afd5e48bd 100644 --- a/src/domain/javascript/javascriptSemanticAnalysisSteps.test.ts +++ b/src/domain/javascript/javascriptSemanticAnalysisSteps.test.ts @@ -5,8 +5,8 @@ import { analyzeParsedJavaScriptSemanticsSteps, classifyParsedJavaScriptElectronBindings, classifyParsedJavaScriptElectronBindingsSteps, - classifyParsedJavaScriptOpenReceivers, - classifyParsedJavaScriptOpenReceiversSteps, + classifyParsedJavaScriptReceivers, + classifyParsedJavaScriptReceiversSteps, } from "./javascriptSemanticAnalysis.js"; import { parseJavaScriptSource } from "./javascriptSourceParser.js"; import { @@ -110,15 +110,15 @@ new NativeWindow(); `; const parsed = parseJavaScriptSource(text, "receivers.js"); if (parsed === null) throw new Error("Expected parsed source"); - const openSteps = classifyParsedJavaScriptOpenReceiversSteps(parsed); + const openSteps = classifyParsedJavaScriptReceiversSteps(parsed); let openStep = openSteps.next(); expect(openStep.done).toBe(false); while (openStep.done !== true) openStep = openSteps.next(); - expect([...openStep.value]).toEqual([ + expect([...openStep.value.openReceivers]).toEqual([ [text.indexOf('ambient.open("/local")'), "local"], [text.indexOf('ambient.open("/global")'), "window"], ]); - expect(openStep.value).toEqual(classifyParsedJavaScriptOpenReceivers(parsed)); + expect(openStep.value).toEqual(classifyParsedJavaScriptReceivers(parsed)); const electronSteps = classifyParsedJavaScriptElectronBindingsSteps(parsed); let electronStep = electronSteps.next(); diff --git a/src/domain/javascript/javascriptSemanticAsyncEffects.ts b/src/domain/javascript/javascriptSemanticAsyncEffects.ts index 5b3068101c..a34bfb7500 100644 --- a/src/domain/javascript/javascriptSemanticAsyncEffects.ts +++ b/src/domain/javascript/javascriptSemanticAsyncEffects.ts @@ -1,13 +1,11 @@ import * as t from "@babel/types"; -import { semanticBuiltinNamespacePath } from "./javascriptSemanticProvenance.js"; - import type { JavaScriptSemanticCallable, JavaScriptSemanticEventOperation, JavaScriptSemanticTimerOperation, } from "./javascriptSemanticIr.js"; -import { semanticCallableIdForNode } from "./javascriptSemanticProjection.js"; +import { createSemanticCallableIdStack } from "./javascriptSemanticCallableStack.js"; import { semanticStaticPropertyName } from "./javascriptAstValues.js"; import { resolveSemanticBindingState, @@ -18,6 +16,7 @@ import { import { traverseJavaScriptAst } from "./javascriptSemanticTraversal.js"; import { range } from "./javascriptStaticAnalysisHelpers.js"; import { + builtinDataEffectMethod, dataEffectLiteralString, dataEffectMemberCallee, assignedSemanticResultBindings, @@ -95,15 +94,13 @@ export const collectJavaScriptSemanticAsyncEffects = ( const events: JavaScriptSemanticEventOperation[] = []; const timers: TimerCandidate[] = []; const callableIds = new Set(callables.map(({ callableId }) => callableId)); - const callableStack: string[] = []; + const callableStack = createSemanticCallableIdStack(callableIds); const ancestors: t.Node[] = []; traverseJavaScriptAst(program, { - enter: (node) => { - const callableId = semanticCallableIdForNode(node); - if (callableId !== null && callableIds.has(callableId)) - callableStack.push(callableId); + enter: (node, parent) => { + callableStack.enter(node, parent); if (t.isCallExpression(node) || t.isOptionalCallExpression(node)) { - const ownerCallableId = callableStack.at(-1) ?? null; + const ownerCallableId = callableStack.callables.at(-1) ?? null; collectEvent(node, ownerCallableId, state, events); collectTimer({ node, @@ -116,10 +113,8 @@ export const collectJavaScriptSemanticAsyncEffects = ( ancestors.push(node); }, exit: (node) => { - const callableId = semanticCallableIdForNode(node); ancestors.pop(); - if (callableId !== null && callableStack.at(-1) === callableId) - callableStack.pop(); + callableStack.exit(node); }, }); return { @@ -308,22 +303,11 @@ const timerMethod = ( ? null : method; } - return timerMethodFromBinding(binding); } - if (!t.isMemberExpression(callee) && !t.isOptionalMemberExpression(callee)) - return null; - const method = admittedTimerMethod( - semanticStaticPropertyName(callee.property, callee.computed), + return admittedTimerMethod( + builtinDataEffectMethod(callee, state, ["timers", "node:timers"]) ?? "", ); - return method !== null && - t.isIdentifier(callee.object) && - timerNamespaceBinding( - resolveSemanticBindingState(state, callee.object, callee.object.name), - ) - ? method - : null; }; - const admittedTimerMethod = ( value: string, ): JavaScriptSemanticTimerOperation["method"] | null => @@ -331,28 +315,6 @@ const admittedTimerMethod = ( (candidate) => candidate === value, ) ?? null; -const timerMethodFromBinding = ( - binding: JavaScriptSemanticBindingState, -): JavaScriptSemanticTimerOperation["method"] | null => { - for (const { specifier, importedPath } of binding.directOrigins) { - if (!timerModule(specifier)) continue; - const method = admittedTimerMethod(importedPath.at(-1) ?? ""); - if (method !== null) return method; - } - return null; -}; - -const timerNamespaceBinding = ( - binding: JavaScriptSemanticBindingState | undefined, -): boolean => - binding?.directOrigins.some( - ({ specifier, importedPath }) => - timerModule(specifier) && semanticBuiltinNamespacePath(importedPath), - ) ?? false; - -const timerModule = (specifier: string): boolean => - specifier === "timers" || specifier === "node:timers"; - const bindingId = ( node: | t.Expression diff --git a/src/domain/javascript/javascriptSemanticCallResolution.ts b/src/domain/javascript/javascriptSemanticCallResolution.ts index 9292ea2658..3ed09dca0c 100644 --- a/src/domain/javascript/javascriptSemanticCallResolution.ts +++ b/src/domain/javascript/javascriptSemanticCallResolution.ts @@ -6,7 +6,10 @@ import type { } from "./javascriptSemanticIr.js"; import { compareUnicodeCodePoints } from "../unicodeCodePointOrder.js"; import { semanticCallableIdForNode } from "./javascriptSemanticProjection.js"; -import { unwrapJavaScriptExpression } from "./javascriptAstValues.js"; +import { + unwrapJavaScriptExpression, + runtimeFunctionParameters, +} from "./javascriptAstValues.js"; import { resolveSemanticBindingState, type JavaScriptSemanticAnalysisState, @@ -194,14 +197,15 @@ const parameterAcceptsArgument = ( ): boolean => { const node = state.callableNodesById.get(callable.callableId); if (!t.isFunction(node)) return false; - const parameterIndex = node.params.findIndex((parameter) => + const parameters = runtimeFunctionParameters(node); + const parameterIndex = parameters.findIndex((parameter) => rangeContains(range(parameter), location), ); if (parameterIndex === argumentIndex) return true; return ( parameterIndex >= 0 && parameterIndex < argumentIndex && - t.isRestElement(node.params[parameterIndex]) + t.isRestElement(parameters[parameterIndex]) ); }; diff --git a/src/domain/javascript/javascriptSemanticCallableStack.ts b/src/domain/javascript/javascriptSemanticCallableStack.ts new file mode 100644 index 0000000000..0718d4aadd --- /dev/null +++ b/src/domain/javascript/javascriptSemanticCallableStack.ts @@ -0,0 +1,85 @@ +import * as t from "@babel/types"; + +import { semanticCallableIdForNode } from "./javascriptSemanticProjection.js"; + +/** + * Innermost-last stack of admitted callables during one enter/exit traversal. + * + * Single owner for semantic callable ownership. A computed object/class method + * key is evaluated before the method's parameter and body environments, so the + * method is set aside while its key is traversed and the enclosing callable + * owns every fact recovered from the key. Collectors share this tracker so + * calls, captures, effects, promises, and object operations agree. + */ +export interface SemanticCallableStack { + /** Admitted callables, outermost first; the innermost owner is last. */ + readonly callables: readonly Callable[]; + readonly enter: (node: t.Node, parent: t.Node | null) => void; + readonly exit: (node: t.Node) => void; +} + +interface CallableFrame { + readonly callableId: string; + readonly callable: Callable; +} + +/** + * Track callables admitted by `admit`, which maps a callable ID to the value + * kept on the stack or `undefined` to ignore it. A caller-owned `callables` + * array is updated in place, so shared contexts can keep reading it. + */ +export const createSemanticCallableStack = ( + admit: (callableId: string) => Callable | undefined, + callables: Callable[] = [], +): SemanticCallableStack => { + const frames: CallableFrame[] = []; + const suspendedMethods = new Map>(); + callables.length = 0; + const push = (frame: CallableFrame): void => { + frames.push(frame); + callables.push(frame.callable); + }; + const pop = (): CallableFrame | undefined => { + callables.pop(); + return frames.pop(); + }; + return { + callables, + enter: (node, parent) => { + if ( + parent !== null && + (t.isObjectMethod(parent) || t.isClassMethod(parent)) && + parent.computed && + parent.key === node && + frames.at(-1)?.callableId === semanticCallableIdForNode(parent) + ) { + const method = pop(); + if (method !== undefined) suspendedMethods.set(node, method); + } + const callableId = semanticCallableIdForNode(node); + if (callableId === null) return; + const callable = admit(callableId); + if (callable !== undefined) push({ callableId, callable }); + }, + exit: (node) => { + const callableId = semanticCallableIdForNode(node); + if (callableId !== null && frames.at(-1)?.callableId === callableId) + pop(); + const method = suspendedMethods.get(node); + if (method !== undefined) { + suspendedMethods.delete(node); + push(method); + } + }, + }; +}; + +/** Track admitted callable IDs, keeping each ID as its stack value. */ +export const createSemanticCallableIdStack = ( + admitted: ReadonlySet, + callables: string[] = [], +): SemanticCallableStack => + createSemanticCallableStack( + (callableId) => (admitted.has(callableId) ? callableId : undefined), + callables, + ); diff --git a/src/domain/javascript/javascriptSemanticCalls.ts b/src/domain/javascript/javascriptSemanticCalls.ts index 53ada16541..fc0816635c 100644 --- a/src/domain/javascript/javascriptSemanticCalls.ts +++ b/src/domain/javascript/javascriptSemanticCalls.ts @@ -9,10 +9,8 @@ import type { JavaScriptSemanticClosureCapture, JavaScriptSemanticFrontier, } from "./javascriptSemanticIr.js"; -import { - semanticCallableIdForNode, - semanticReferenceRole, -} from "./javascriptSemanticProjection.js"; +import { semanticReferenceRole } from "./javascriptSemanticProjection.js"; +import { createSemanticCallableStack } from "./javascriptSemanticCallableStack.js"; import { semanticStaticPropertyKey } from "./javascriptAstValues.js"; import { resolveSemanticBindingState, @@ -85,14 +83,13 @@ export const collectJavaScriptSemanticCalls = ( parameterBindingCache: new Map(), ancestors: [], }; - const callableStack: JavaScriptSemanticCallable[] = []; + const callableStack = createSemanticCallableStack((callableId) => + callableById.get(callableId), + ); traverseJavaScriptAst(program, { enter: (node, parent) => { - const callableId = semanticCallableIdForNode(node); - const callable = - callableId === null ? undefined : callableById.get(callableId); - if (callable !== undefined) callableStack.push(callable); - const owner = enclosingFunction(callableStack); + callableStack.enter(node, parent); + const owner = enclosingFunction(callableStack.callables); collectCapture(node, parent, owner, context); collectDynamicProperty(node, owner, state, output); collectDynamicScope(node, owner, state, output); @@ -106,12 +103,7 @@ export const collectJavaScriptSemanticCalls = ( }, exit: (node) => { context.ancestors.pop(); - const callableId = semanticCallableIdForNode(node); - if ( - callableId !== null && - callableStack.at(-1)?.callableId === callableId - ) - callableStack.pop(); + callableStack.exit(node); }, }); return { diff --git a/src/domain/javascript/javascriptSemanticChildProcesses.ts b/src/domain/javascript/javascriptSemanticChildProcesses.ts index 1cffb51642..45d15f5719 100644 --- a/src/domain/javascript/javascriptSemanticChildProcesses.ts +++ b/src/domain/javascript/javascriptSemanticChildProcesses.ts @@ -1,15 +1,17 @@ import * as t from "@babel/types"; -import { semanticBuiltinNamespacePath } from "./javascriptSemanticProvenance.js"; - import type { JavaScriptSemanticCallable, JavaScriptSemanticChildProcessInteraction, JavaScriptSemanticChildProcessSpawn, } from "./javascriptSemanticIr.js"; -import { semanticCallableIdForNode } from "./javascriptSemanticProjection.js"; -import { semanticStaticPropertyName } from "./javascriptAstValues.js"; +import { createSemanticCallableIdStack } from "./javascriptSemanticCallableStack.js"; import { + semanticStaticPropertyName, + unwrapJavaScriptExpression, +} from "./javascriptAstValues.js"; +import { + isUnshadowedGlobal, resolveSemanticBindingState, type JavaScriptSemanticAnalysisState, type JavaScriptSemanticBindingState, @@ -18,6 +20,7 @@ import { compareUnicodeCodePoints } from "../unicodeCodePointOrder.js"; import { traverseJavaScriptAst } from "./javascriptSemanticTraversal.js"; import { range } from "./javascriptStaticAnalysisHelpers.js"; import { + builtinDataEffectMethod, dataEffectLiteralString, dataEffectMemberCallee, assignedSemanticResultBindings, @@ -81,13 +84,11 @@ const collectSpawns = ( const admittedCallables = new Set( callables.map(({ callableId }) => callableId), ); - const callableStack: string[] = []; + const callableStack = createSemanticCallableIdStack(admittedCallables); const ancestors: t.Node[] = []; traverseJavaScriptAst(program, { - enter: (node) => { - const callableId = semanticCallableIdForNode(node); - if (callableId !== null && admittedCallables.has(callableId)) - callableStack.push(callableId); + enter: (node, parent) => { + callableStack.enter(node, parent); if (t.isCallExpression(node) || t.isOptionalCallExpression(node)) { const method = childProcessMethod(node.callee, state); if (method !== null) { @@ -99,7 +100,7 @@ const collectSpawns = ( output.push({ node, method, - ownerCallableId: callableStack.at(-1) ?? null, + ownerCallableId: callableStack.callables.at(-1) ?? null, resultBindingId: scalarAssignedResultBinding( projectedResultBindings, ), @@ -115,7 +116,7 @@ const collectSpawns = ( }, exit: (node) => { ancestors.pop(); - popCallable(node, callableStack); + callableStack.exit(node); }, }); return output; @@ -149,7 +150,7 @@ const collectInteractions = ( ): JavaScriptSemanticChildProcessInteraction[] => { const output: JavaScriptSemanticChildProcessInteraction[] = []; const callableIds = new Set(callables.map(({ callableId }) => callableId)); - const callableStack: string[] = []; + const callableStack = createSemanticCallableIdStack(callableIds); const spawnByNode = new WeakMap(); const spawnsByBinding = new Map(); for (const spawn of spawns) { @@ -170,21 +171,19 @@ const collectInteractions = ( spawnsByBinding, }; traverseJavaScriptAst(program, { - enter: (node) => { - const callableId = semanticCallableIdForNode(node); - if (callableId !== null && callableIds.has(callableId)) - callableStack.push(callableId); + enter: (node, parent) => { + callableStack.enter(node, parent); if (!t.isCallExpression(node) && !t.isOptionalCallExpression(node)) return; const interaction = childInteraction( node, - callableStack.at(-1) ?? null, + callableStack.callables.at(-1) ?? null, context, ); if (interaction === null) return; output.push(interaction); }, - exit: (node) => popCallable(node, callableStack), + exit: (node) => callableStack.exit(node), }); return output; }; @@ -222,9 +221,7 @@ const childInteraction = ( .sort(compareUnicodeCodePoints), eventName: kind === "listener" ? eventName : null, signalName: - kind === "signal" - ? (dataEffectLiteralString(node.arguments[0]) ?? "SIGTERM") - : null, + kind === "signal" ? killSignalName(node.arguments, state) : null, listenerLocation: kind === "listener" && t.isNode(listener) ? range(listener) : null, resolution: @@ -237,6 +234,23 @@ const childInteraction = ( }; }; +// Only an omitted argument or the unshadowed global undefined selects the +// SIGTERM default; an unresolved argument leaves the signal unknown. +const killSignalName = ( + args: t.CallExpression["arguments"], + state: JavaScriptSemanticAnalysisState, +): string | null => { + const [signal] = args; + if (signal === undefined) return "SIGTERM"; + const expression = unwrapJavaScriptExpression(signal).node; + if ( + t.isUnaryExpression(expression, { operator: "void" }) || + isUnshadowedGlobal(expression, state, "undefined") + ) + return "SIGTERM"; + return dataEffectLiteralString(signal); +}; + const classifyInteraction = ( method: string, argument: @@ -266,52 +280,15 @@ const childProcessMethod = ( callee: t.Node, state: JavaScriptSemanticAnalysisState, ): JavaScriptSemanticChildProcessSpawn["method"] | null => { - if (t.isIdentifier(callee)) { - const binding = resolveSemanticBindingState(state, callee, callee.name); - return binding === undefined ? null : methodFromBinding(binding); - } - if (!t.isMemberExpression(callee) && !t.isOptionalMemberExpression(callee)) - return null; - const method = admittedChildMethod( - semanticStaticPropertyName(callee.property, callee.computed), + const method = builtinDataEffectMethod(callee, state, [ + "child_process", + "node:child_process", + ]); + return ( + CHILD_PROCESS_METHODS.find((candidate) => candidate === method) ?? null ); - if (method === null || !t.isIdentifier(callee.object)) return null; - const binding = resolveSemanticBindingState( - state, - callee.object, - callee.object.name, - ); - return childProcessNamespace(binding) ? method : null; }; -const methodFromBinding = ( - binding: JavaScriptSemanticBindingState, -): JavaScriptSemanticChildProcessSpawn["method"] | null => { - for (const { specifier, importedPath } of binding.directOrigins) { - if (!childProcessModule(specifier)) continue; - const method = admittedChildMethod(importedPath.at(-1) ?? ""); - if (method !== null) return method; - } - return null; -}; - -const admittedChildMethod = ( - value: string, -): JavaScriptSemanticChildProcessSpawn["method"] | null => - CHILD_PROCESS_METHODS.find((candidate) => candidate === value) ?? null; - -const childProcessNamespace = ( - binding: JavaScriptSemanticBindingState | undefined, -): boolean => - binding?.directOrigins.some( - ({ specifier, importedPath }) => - childProcessModule(specifier) && - semanticBuiltinNamespacePath(importedPath), - ) ?? false; - -const childProcessModule = (specifier: string): boolean => - specifier === "child_process" || specifier === "node:child_process"; - const resolveChildCandidates = ( node: t.Node, binding: JavaScriptSemanticBindingState | undefined, @@ -378,10 +355,5 @@ const objectBinding = ( ? resolveSemanticBindingState(state, node, node.name) : undefined; -const popCallable = (node: t.Node, stack: string[]): void => { - const callableId = semanticCallableIdForNode(node); - if (callableId !== null && stack.at(-1) === callableId) stack.pop(); -}; - const spawnId = (candidate: SpawnCandidate): string => `child:spawn:${String(candidate.node.start ?? -1)}:${String(candidate.node.end ?? -1)}`; diff --git a/src/domain/javascript/javascriptSemanticDataEffectHelpers.ts b/src/domain/javascript/javascriptSemanticDataEffectHelpers.ts index 0f7a6d942e..5c3ed9ddd4 100644 --- a/src/domain/javascript/javascriptSemanticDataEffectHelpers.ts +++ b/src/domain/javascript/javascriptSemanticDataEffectHelpers.ts @@ -1,19 +1,37 @@ import * as t from "@babel/types"; +import { BUILTIN_RESOURCE_METHODS } from "./javascriptBuiltinResourceMethods.js"; import { semanticBuiltinNamespacePath } from "./javascriptSemanticProvenance.js"; +import { + semanticRequireOrigin, + semanticRequireExportsIdentity, + semanticGlobalProcessOrigin, + semanticMutableBuiltinOrigin, +} from "./javascriptSemanticRequireOrigin.js"; +import type { JavaScriptModuleOrigin } from "./javascriptSemanticIr.js"; +import { + evaluateSemanticProvenance, + evaluateSemanticProvenanceAtRead, +} from "./javascriptSemanticValues.js"; +import { + semanticPropertyPathKeyMatches, + type JavaScriptSemanticPropertyPath, +} from "./javascriptSemanticPropertyPaths.js"; import { semanticCallableIdForNode } from "./javascriptSemanticProjection.js"; +import { semanticTopLevelEffectFollowsCapture } from "./javascriptSemanticMutationInitializers.js"; +import { createSemanticCallableIdStack } from "./javascriptSemanticCallableStack.js"; import { resolveSemanticBindingState, - semanticResolutionBlocked, type JavaScriptSemanticAnalysisState, type JavaScriptSemanticBindingState, + type JavaScriptSemanticMemberEffect, + type JavaScriptSemanticRequiredModuleEffect, } from "./javascriptSemanticState.js"; import { traverseJavaScriptAst } from "./javascriptSemanticTraversal.js"; import { readExactJavaScriptLiteral, semanticStaticPropertyKey, - semanticStaticPropertyName, unwrapJavaScriptExpression, } from "./javascriptAstValues.js"; @@ -137,59 +155,715 @@ export interface DataEffectTraversalContext { readonly ancestors: t.Node[]; } -/** Resolve a direct or namespace-imported built-in method. */ +/** Resolve built-in methods from the shared, unambiguous binding provenance. */ export const builtinDataEffectMethod = ( callee: t.Node, state: JavaScriptSemanticAnalysisState, specifiers: readonly string[], ): string | null => { + const expression = unwrapJavaScriptExpression(callee).node; + if (expression !== callee) + return builtinDataEffectMethod(expression, state, specifiers); + if (t.isSequenceExpression(expression)) { + const final = expression.expressions.at(-1); + return final === undefined + ? null + : builtinDataEffectMethod(final, state, specifiers); + } + const required = semanticRequireOrigin(callee, state); + if (required !== undefined) { + const origin = builtinNamespaceOrigin(required.origin); + return origin.importedPath.length === 1 && + specifiers.some( + (specifier) => specifier.replace(/^node:/u, "") === origin.specifier, + ) && + !requiredModuleChanged( + required.origin.specifier, + required.origin.importedPath, + callee, + state, + true, + ) + ? (origin.importedPath[0] ?? null) + : null; + } if (t.isIdentifier(callee)) { const binding = resolveSemanticBindingState(state, callee, callee.name); - return moduleMethod(binding, specifiers); + const path = builtinBindingPath( + binding, + state, + specifiers, + callee, + [], + true, + ); + return path?.length === 1 ? (path[0] ?? null) : null; } if (!t.isMemberExpression(callee) && !t.isOptionalMemberExpression(callee)) return null; - if (!t.isIdentifier(callee.object)) return null; + if (!t.isIdentifier(callee.object)) { + const method = semanticStaticPropertyKey(callee.property, callee.computed); + const receiver = semanticBoundMemberPath(callee.object, state); + if (method !== null && receiver !== undefined) { + const provenance = evaluateSemanticProvenance(receiver.binding, state); + if (provenance.status === "module") { + const origin = builtinNamespaceOrigin({ + ...provenance.origins[0], + importedPath: [ + ...provenance.origins[0].importedPath, + ...receiver.path, + ], + }); + if ( + specifiers.some( + (specifier) => + specifier.replace(/^node:/u, "") === origin.specifier, + ) + ) + return builtinBindingPath( + receiver.binding, + state, + specifiers, + callee, + [...receiver.path, method], + true, + )?.length === 0 + ? method + : null; + } + } + const source = semanticExpressionModuleOrigin(callee.object, state); + if (method === null || source === undefined) return null; + const origin = builtinNamespaceOrigin(source); + return origin.importedPath.length === 0 && + specifiers.some( + (specifier) => specifier.replace(/^node:/u, "") === origin.specifier, + ) && + !requiredModuleChanged( + origin.specifier, + [...origin.importedPath, method], + callee, + state, + true, + callee.object, + ) + ? method + : null; + } const binding = resolveSemanticBindingState( state, callee.object, callee.object.name, ); - const namespace = binding?.directOrigins.some( - ({ specifier, importedPath }) => - specifiers.includes(specifier) && - semanticBuiltinNamespacePath(importedPath), + const method = semanticStaticPropertyKey(callee.property, callee.computed); + if (method === null) return null; + const resolvedPath = builtinBindingPath( + binding, + state, + specifiers, + callee, + [method], + true, ); - return namespace - ? semanticStaticPropertyName(callee.property, callee.computed) + if (resolvedPath?.length === 0) return method; + const initializer = binding?.initializers[0]; + if ( + initializer === undefined || + binding?.initializers.length !== 1 || + (!t.isMemberExpression(initializer.node) && + !t.isOptionalMemberExpression(initializer.node)) + ) { + return null; + } + const provenance = evaluateSemanticProvenance(binding, state); + if (provenance.status !== "module") return null; + const origin = builtinNamespaceOrigin(provenance.origins[0]); + const selected = [...origin.importedPath, method]; + return specifiers.some( + (specifier) => specifier.replace(/^node:/u, "") === origin.specifier, + ) && + builtinBindingPath( + binding, + state, + specifiers, + initializer.node, + [method], + true, + )?.length === 0 && + !requiredModuleChanged( + provenance.origins[0].specifier, + selected, + callee, + state, + true, + initializer.node, + ) + ? method : null; }; -const moduleMethod = ( - binding: JavaScriptSemanticBindingState | undefined, - specifiers: readonly string[], -): string | null => { - for (const { specifier, importedPath } of binding?.directOrigins ?? []) { - if (!specifiers.includes(specifier)) continue; - return importedPath.at(-1) ?? null; +const semanticBoundMemberPath = ( + node: t.Node, + state: JavaScriptSemanticAnalysisState, +): + | { + readonly binding: JavaScriptSemanticBindingState; + readonly path: string[]; + } + | undefined => { + const path: string[] = []; + let current = unwrapJavaScriptExpression(node).node; + while ( + t.isMemberExpression(current) || + t.isOptionalMemberExpression(current) + ) { + const key = semanticStaticPropertyKey(current.property, current.computed); + if (key === null || !t.isNode(current.object)) return undefined; + path.unshift(key); + current = unwrapJavaScriptExpression(current.object).node; } - return null; + if (!t.isIdentifier(current)) return undefined; + const binding = resolveSemanticBindingState(state, current, current.name); + return binding === undefined ? undefined : { binding, path }; +}; + +type BuiltinBindingMemo = Map>; + +const builtinBindingPath = ( + binding: JavaScriptSemanticBindingState | undefined, + state: JavaScriptSemanticAnalysisState, + specifiers: readonly string[], + capture: t.Node, + selectedPath: readonly string[] = [], + callableValue = false, + visited: ReadonlySet = new Set(), + memo: BuiltinBindingMemo = new Map(), + provenanceCapture?: t.Identifier, + savedObjectCapture?: t.Node, +): readonly string[] | null => { + if (binding === undefined) return null; + const key = JSON.stringify([ + binding.bindingId, + selectedPath, + callableValue, + provenanceCapture?.start, + provenanceCapture?.end, + savedObjectCapture?.start, + savedObjectCapture?.end, + ]); + let captured = memo.get(capture); + if (captured?.has(key)) return captured.get(key) ?? null; + if (visited.has(binding.bindingId)) return null; + if (captured === undefined) { + captured = new Map(); + memo.set(capture, captured); + } + const result = computeBuiltinBindingPath( + binding, + state, + specifiers, + capture, + selectedPath, + callableValue, + visited, + memo, + provenanceCapture, + savedObjectCapture, + ); + captured.set(key, result); + return result; +}; + +const computeBuiltinBindingPath = ( + binding: JavaScriptSemanticBindingState | undefined, + state: JavaScriptSemanticAnalysisState, + specifiers: readonly string[], + capture: t.Node, + selectedPath: readonly string[] = [], + callableValue = false, + visited: ReadonlySet, + memo: BuiltinBindingMemo, + provenanceCapture?: t.Identifier, + savedObjectCapture?: t.Node, +): readonly string[] | null => { + if (binding === undefined || visited.has(binding.bindingId)) return null; + // Provenance identifies the module, independently of whether a selected slot + // still refers to its original built-in after member mutation or escape. + if ( + binding.memberWrites.some( + ({ path, node }) => + affectsBuiltinPath(path, selectedPath) && + !( + semanticTopLevelEffectFollowsCapture( + path.length === 0 && provenanceCapture !== undefined + ? provenanceCapture + : capture, + node, + state.parentsByNode, + ) || + (savedObjectCapture !== undefined && + path.length < selectedPath.length && + semanticTopLevelEffectFollowsCapture( + savedObjectCapture, + node, + state.parentsByNode, + )) + ), + ) || + builtinEscapes(binding, state).some( + ({ path, node }) => + affectsBuiltinPath(path, selectedPath) && + // A function-value escape cannot replace its native [[Call]]. An + // escaped containing object can still replace the selected method. + (!callableValue || path.length < selectedPath.length) && + !semanticTopLevelEffectFollowsCapture( + capture, + node, + state.parentsByNode, + ), + ) + ) + return null; + const provenance = + provenanceCapture === undefined + ? evaluateSemanticProvenance(binding, state) + : evaluateSemanticProvenanceAtRead(binding, provenanceCapture, state); + if (provenance.status !== "module") return null; + const { importedPath } = provenance.origins[0]; + const namespace = builtinNamespaceOrigin(provenance.origins[0]); + if ( + !specifiers.some((item) => { + const name = item.replace(/^node:/u, ""); + return ( + name === namespace.specifier || + (namespace.specifier === "fs" && + selectedPath[0] === "promises" && + name === "fs/promises") + ); + }) + ) + return null; + let resolvedPath: readonly string[] = importedPath; + for (const origin of binding.directOrigins) { + const shared = semanticMutableBuiltinOrigin(origin, selectedPath); + if ( + shared !== undefined && + requiredModuleChanged( + shared.specifier, + shared.path, + capture, + state, + callableValue, + ) + ) + return null; + } + const ancestors = new Set([...visited, binding.bindingId]); + for (const initializer of binding.initializers) { + const pending = [ + { + node: initializer.node, + path: [...initializer.projection.map(String), ...selectedPath], + savedObjectCapture: + selectedPath.length > 0 && + (t.isMemberExpression(initializer.node) || + t.isOptionalMemberExpression(initializer.node)) + ? initializer.node + : savedObjectCapture, + capture: + callableValue && + selectedPath.length === 0 && + initializer.projection.length > 0 + ? initializer.node + : capture, + }, + ]; + while (pending.length > 0) { + const current = pending.pop()!; + const node = unwrapJavaScriptExpression(current.node).node; + if (t.isIdentifier(node)) { + const source = resolveSemanticBindingState(state, node, node.name); + const sourcePath = builtinBindingPath( + source, + state, + specifiers, + current.capture, + current.path, + callableValue, + ancestors, + memo, + node, + current.savedObjectCapture, + ); + if (sourcePath === null) return null; + // Normalize ESM default namespaces at their owner before projecting an alias. + resolvedPath = [...sourcePath, ...current.path].slice( + 0, + sourcePath.length + current.path.length - selectedPath.length, + ); + } else if ( + t.isMemberExpression(node) || + t.isOptionalMemberExpression(node) + ) { + const key = semanticStaticPropertyKey(node.property, node.computed); + if (key === null) return null; + pending.push({ + node: node.object, + path: [key, ...current.path], + savedObjectCapture: current.savedObjectCapture, + // Freeze a selected function, not later member reads through a saved object. + capture: + callableValue && selectedPath.length === 0 ? node : current.capture, + }); + } else if (t.isSequenceExpression(node)) { + const final = node.expressions.at(-1); + if (final === undefined) return null; + pending.push({ ...current, node: final }); + } else if (t.isAssignmentExpression(node) && node.operator === "=") { + pending.push({ ...current, node: node.right }); + } else if (t.isConditionalExpression(node)) { + pending.push( + { ...current, node: node.consequent }, + { ...current, node: node.alternate }, + ); + } else if (t.isLogicalExpression(node)) { + pending.push( + { ...current, node: node.left }, + { ...current, node: node.right }, + ); + } else { + const required = semanticRequireOrigin(node, state); + if ( + required !== undefined && + requiredModuleChanged( + required.origin.specifier, + [...required.origin.importedPath, ...current.path], + current.capture, + state, + callableValue, + current.savedObjectCapture, + ) + ) + return null; + } + } + } + // Only a direct ESM default import denotes the built-in namespace; + // require("node:fs").default is a literal property, not interop syntax. + return binding.directOrigins.length > 0 && + semanticBuiltinNamespacePath(importedPath) + ? [] + : builtinNamespaceOrigin({ + ...provenance.origins[0], + importedPath: resolvedPath, + }).importedPath; +}; + +/** Interpret known Node namespace aliases without changing source provenance. */ +const builtinNamespaceOrigin = ( + origin: JavaScriptModuleOrigin, +): JavaScriptModuleOrigin => { + const specifier = origin.specifier.replace(/^node:/u, ""); + const importedPath = + origin.resolutionMode !== "require" && origin.importedPath[0] === "default" + ? origin.importedPath.slice(1) + : origin.importedPath; + return specifier === "fs" && importedPath[0] === "promises" + ? { + ...origin, + specifier: "fs/promises", + importedPath: importedPath.slice(1), + } + : { ...origin, specifier, importedPath }; +}; + +const affectsBuiltinPath = ( + path: JavaScriptSemanticPropertyPath, + selected: readonly string[], +): boolean => + path.length <= selected.length && + path.every((key, index) => + semanticPropertyPathKeyMatches(key, selected[index]!), + ); + +const BUILTIN_RECEIVER_METHODS: Readonly>> = + { + fs: new Set([ + "readFile", + "readFileSync", + "writeFile", + "writeFileSync", + "close", + "closeSync", + "existsSync", + ...BUILTIN_RESOURCE_METHODS.fs, + ]), + "fs/promises": new Set([ + "readFile", + "writeFile", + ...BUILTIN_RESOURCE_METHODS["fs/promises"], + ]), + net: new Set(BUILTIN_RESOURCE_METHODS.net), + tls: new Set(BUILTIN_RESOURCE_METHODS.tls), + http: new Set(["request", "get"]), + https: new Set(["request", "get"]), + child_process: new Set(["spawn", "exec", "execFile", "fork"]), + process: new Set(["cwd", "uptime", "hrtime", "nextTick"]), + timers: new Set([ + "setTimeout", + "setInterval", + "setImmediate", + "clearTimeout", + "clearInterval", + "clearImmediate", + ]), + }; + +interface BuiltinEffectIndex { + readonly writesByModule: Map< + string, + readonly JavaScriptSemanticRequiredModuleEffect[] + >; + readonly moduleEffects: Map< + string, + readonly JavaScriptSemanticRequiredModuleEffect[] + >; + readonly bindingEscapes: Map< + string, + readonly JavaScriptSemanticMemberEffect[] + >; + readonly receivers: WeakMap; +} + +// Derived passes run after mutation collection. These indexes share that state's +// lifetime and never cache facts while its mutation owner is still producing them. +const builtinEffectIndexes = new WeakMap< + JavaScriptSemanticAnalysisState, + BuiltinEffectIndex +>(); + +const builtinEffectIndex = ( + state: JavaScriptSemanticAnalysisState, +): BuiltinEffectIndex => { + let index = builtinEffectIndexes.get(state); + if (index === undefined) { + index = { + writesByModule: new Map( + [...state.requiredModuleEffects].map(([name, effects]) => [ + name, + effects.filter(({ kind }) => kind === "write"), + ]), + ), + moduleEffects: new Map(), + bindingEscapes: new Map(), + receivers: new WeakMap(), + }; + builtinEffectIndexes.set(state, index); + } + return index; +}; + +const builtinEscapes = ( + binding: JavaScriptSemanticBindingState, + state: JavaScriptSemanticAnalysisState, +): readonly JavaScriptSemanticMemberEffect[] => { + const cache = builtinEffectIndex(state).bindingEscapes; + let effects = cache.get(binding.bindingId); + if (effects === undefined) { + effects = binding.escapedPaths.filter( + ({ node }) => !originalBuiltinReceiver(node, state), + ); + cache.set(binding.bindingId, effects); + } + return effects; +}; + +const builtinModuleEffects = ( + specifier: string, + state: JavaScriptSemanticAnalysisState, +): readonly JavaScriptSemanticRequiredModuleEffect[] => { + const cache = builtinEffectIndex(state).moduleEffects; + let effects = cache.get(specifier); + if (effects === undefined) { + effects = (state.requiredModuleEffects.get(specifier) ?? []).filter( + ({ kind, node }) => + kind === "write" || !originalBuiltinReceiver(node, state), + ); + if (specifier === "fs/promises") { + const sharedFsEscapes = (state.requiredModuleEffects.get("fs") ?? []) + .filter( + ({ kind, path, node }) => + kind === "escape" && + (path.length === 0 || path[0] === "promises") && + !originalBuiltinReceiver(node, state), + ) + .map((effect) => ({ + ...effect, + path: + effect.path[0] === "promises" ? effect.path.slice(1) : effect.path, + })); + effects = [...effects, ...sharedFsEscapes]; + } + cache.set(specifier, effects); + } + return effects; +}; + +/** Check shared exports at the use or method-capture point, not at every later call. */ +const requiredModuleChanged = ( + specifier: string, + selected: readonly string[], + capture: t.Node, + state: JavaScriptSemanticAnalysisState, + callableValue = false, + savedObjectCapture?: t.Node, +): boolean => { + const identity = semanticRequireExportsIdentity(specifier, selected); + return builtinModuleEffects(identity.specifier, state).some( + ({ path, node, kind }) => + !( + savedObjectCapture !== undefined && + path.length < identity.path.length && + semanticTopLevelEffectFollowsCapture( + savedObjectCapture, + node, + state.parentsByNode, + ) + ) && + affectsBuiltinPath(path, identity.path) && + !semanticTopLevelEffectFollowsCapture( + capture, + node, + state.parentsByNode, + ) && + (kind === "write" || + !callableValue || + path.length < identity.path.length), + ); +}; + +/** Calling an unchanged built-in as a member is not an escape to unknown code. */ +const originalBuiltinReceiver = ( + node: t.Node | undefined, + state: JavaScriptSemanticAnalysisState, +): boolean => { + if (node === undefined) return false; + const receivers = builtinEffectIndex(state).receivers; + const previous = receivers.get(node); + if (previous !== undefined) return previous; + const result = computeOriginalBuiltinReceiver(node, state); + receivers.set(node, result); + return result; +}; + +const computeOriginalBuiltinReceiver = ( + node: t.Node | undefined, + state: JavaScriptSemanticAnalysisState, +): boolean => { + if (!t.isCallExpression(node) && !t.isOptionalCallExpression(node)) + return false; + const callee = node.callee; + if (!t.isMemberExpression(callee) && !t.isOptionalMemberExpression(callee)) + return false; + const receiver = t.isIdentifier(callee.object) + ? resolveSemanticBindingState(state, callee.object, callee.object.name) + : undefined; + const provenance = + receiver === undefined + ? undefined + : evaluateSemanticProvenance(receiver, state); + const sourceOrigin = + provenance?.status === "module" + ? provenance.origins[0] + : (semanticExpressionModuleOrigin(callee.object, state) ?? + semanticRequireOrigin(callee.object, state)?.origin ?? + semanticGlobalProcessOrigin(callee.object, state)); + if (sourceOrigin === undefined) return false; + const origin = builtinNamespaceOrigin(sourceOrigin); + if (origin.importedPath.length !== 0) return false; + const method = semanticStaticPropertyKey(callee.property, callee.computed); + const identity = semanticRequireExportsIdentity(origin.specifier, [ + method ?? "", + ]); + if ( + method === null || + !BUILTIN_RECEIVER_METHODS[origin.specifier]?.has(method) || + receiver?.memberWrites.some( + ({ path, node: write }) => + affectsBuiltinPath(path, [method]) && + !semanticTopLevelEffectFollowsCapture(node, write, state.parentsByNode), + ) || + ( + builtinEffectIndex(state).writesByModule.get(identity.specifier) ?? [] + ).some( + ({ path, node: write }) => + affectsBuiltinPath(path, identity.path) && + !semanticTopLevelEffectFollowsCapture(node, write, state.parentsByNode), + ) + ) + return false; + return !node.arguments.some((argument) => { + if (!t.isIdentifier(argument)) return false; + const passed = resolveSemanticBindingState(state, argument, argument.name); + if (passed === undefined) return false; + const passedOrigin = evaluateSemanticProvenance(passed, state); + return ( + passedOrigin.status === "module" && + builtinNamespaceOrigin(passedOrigin.origins[0]).specifier === + origin.specifier && + builtinNamespaceOrigin(passedOrigin.origins[0]).importedPath.length === 0 + ); + }); +}; + +const semanticExpressionModuleOrigin = ( + expression: t.Node, + state: JavaScriptSemanticAnalysisState, + visited = new Set(), +): JavaScriptModuleOrigin | undefined => { + const node = unwrapJavaScriptExpression(expression).node; + const required = semanticRequireOrigin(node, state); + if (required !== undefined) return required.origin; + if (t.isIdentifier(node)) { + const binding = resolveSemanticBindingState(state, node, node.name); + if (binding === undefined || visited.has(binding.bindingId)) + return undefined; + const provenance = evaluateSemanticProvenance(binding, state); + return provenance.status === "module" ? provenance.origins[0] : undefined; + } + if (t.isMemberExpression(node) || t.isOptionalMemberExpression(node)) { + const base = semanticExpressionModuleOrigin(node.object, state, visited); + const key = semanticStaticPropertyKey(node.property, node.computed); + return base === undefined || key === null + ? undefined + : { ...base, importedPath: [...base.importedPath, key] }; + } + return undefined; }; /** Recognize global or node:process aliases without accepting shadowing. */ export const isSemanticProcessObject = ( node: t.Expression | t.Super, state: JavaScriptSemanticAnalysisState, + family: "env" | "argv", + key?: string | null, ): boolean => { if (!t.isIdentifier(node)) return false; const binding = resolveSemanticBindingState(state, node, node.name); + const selected = key === undefined || key === null ? [family] : [family, key]; if (binding === undefined) return ( - node.name === "process" && - !semanticResolutionBlocked(state, node, node.name) + semanticGlobalProcessOrigin(node, state) !== undefined && + !requiredModuleChanged("process", selected, node, state) ); - return binding.directOrigins.some( - ({ specifier }) => specifier === "process" || specifier === "node:process", + return ( + builtinBindingPath( + binding, + state, + ["process", "node:process"], + node, + selected, + )?.length === 0 ); }; @@ -209,21 +883,20 @@ export const traverseDataEffects = ( context: DataEffectTraversalContext, visit: (node: t.Node, parent: t.Node | null) => void, ): void => { - context.callableStack.length = 0; + const callableStack = createSemanticCallableIdStack( + context.callableIds, + context.callableStack, + ); context.ancestors.length = 0; traverseJavaScriptAst(program, { enter: (node, parent) => { - const callableId = semanticCallableIdForNode(node); - if (callableId !== null && context.callableIds.has(callableId)) - context.callableStack.push(callableId); + callableStack.enter(node, parent); visit(node, parent); context.ancestors.push(node); }, exit: (node) => { context.ancestors.pop(); - const callableId = semanticCallableIdForNode(node); - if (callableId !== null && context.callableStack.at(-1) === callableId) - context.callableStack.pop(); + callableStack.exit(node); }, }); }; @@ -251,7 +924,11 @@ export const dataEffectPrimitive = ( return null; }; -/** Read one static string or expression-free template. */ +/** + * Read one static string or expression-free template. A template recovered + * with an invalid escape has no cooked value; its raw spelling is not a + * JavaScript string, so it stays dynamic. + */ export const dataEffectLiteralString = ( node: | t.Expression @@ -262,9 +939,8 @@ export const dataEffectLiteralString = ( ): string | null => { if (node === undefined) return null; const literal = readExactJavaScriptLiteral(node); - if (literal.found && typeof literal.value === "string") return literal.value; - return t.isTemplateLiteral(node) && node.expressions.length === 0 - ? (node.quasis[0]?.value.raw ?? null) + return literal.found && typeof literal.value === "string" + ? literal.value : null; }; diff --git a/src/domain/javascript/javascriptSemanticDataEffects.ts b/src/domain/javascript/javascriptSemanticDataEffects.ts index 607843aabe..6d1de8325e 100644 --- a/src/domain/javascript/javascriptSemanticDataEffects.ts +++ b/src/domain/javascript/javascriptSemanticDataEffects.ts @@ -1,7 +1,5 @@ import * as t from "@babel/types"; -import { semanticBuiltinNamespacePath } from "./javascriptSemanticProvenance.js"; - import type { JavaScriptSemanticBoundaryOperation, JavaScriptSemanticCallable, @@ -140,12 +138,12 @@ const processConfigurationMember = ( const object = node.object; if (!t.isMemberExpression(object) && !t.isOptionalMemberExpression(object)) return null; - if (!isProcessObject(object.object, state)) return null; const family = semanticStaticPropertyName(object.property, object.computed); if (family !== "env" && family !== "argv") return null; // `process.env[""]` names an exact (empty) key; only a dynamic key is unknown. const key = semanticStaticPropertyKey(node.property, node.computed); if (family === "argv" && key !== null && !isArrayIndexKey(key)) return null; + if (!isProcessObject(object.object, state, family, key)) return null; return { kind: family === "env" ? "environment" : "argv", key, @@ -272,6 +270,18 @@ const collectResponseConsumers = ( ? [] : (requestsByBinding.get(binding.bindingId) ?? []); if (linked.length === 0) return; + // A consumer of exactly one request through its own result binding + // reads that request's response, so it keeps the request's endpoint. + const uniqueRequest = + linked.length === 1 && + !linked[0]?.operation.projectedResultBindings.some( + ({ bindingId }) => bindingId === binding?.bindingId, + ) + ? linked[0] + : undefined; + // Any other write (assignment, update, loop or destructuring target) can + // replace the response before this read, so its endpoint stays unknown. + const soleWrite = binding?.initializers.length === 1; output.push({ requestId: `response:${String(node.start ?? -1)}:${String(node.end ?? -1)}`, kind: "response-consumer", @@ -289,15 +299,9 @@ const collectResponseConsumers = ( linkedRequestIds: linked .map(({ operation }) => operation.requestId) .sort(compareUnicodeCodePoints), - endpoint: null, + endpoint: soleWrite ? (uniqueRequest?.operation.endpoint ?? null) : null, fields: [], - resolution: - linked.length === 1 && - !linked[0]?.operation.projectedResultBindings.some( - ({ bindingId }) => bindingId === binding?.bindingId, - ) - ? "complete" - : "partial", + resolution: uniqueRequest === undefined ? "partial" : "complete", }); }); return output; @@ -375,22 +379,13 @@ const requestMethod = ( if (t.isNewExpression(node) && unbound && name === "WebSocket") return "WebSocket"; } - const member = memberCallee(node); - if (member === null || !t.isIdentifier(member.object)) return null; - const method = semanticStaticPropertyName(member.property, member.computed); - if (method !== "request" && method !== "get") return null; - const binding = resolveSemanticBindingState( - state, - member.object, - member.object.name, - ); - return binding?.directOrigins.some( - ({ specifier, importedPath }) => - ["http", "https", "node:http", "node:https"].includes(specifier) && - semanticBuiltinNamespacePath(importedPath), - ) - ? method - : null; + const method = builtinMethod(node.callee, state, [ + "http", + "https", + "node:http", + "node:https", + ]); + return method === "request" || method === "get" ? method : null; }; const requestFields = ( diff --git a/src/domain/javascript/javascriptSemanticFingerprints.test.ts b/src/domain/javascript/javascriptSemanticFingerprints.test.ts new file mode 100644 index 0000000000..48a88d7353 --- /dev/null +++ b/src/domain/javascript/javascriptSemanticFingerprints.test.ts @@ -0,0 +1,31 @@ +import { expect, it } from "vitest"; + +import { analyzeJavaScriptSemantics } from "./javascriptSemanticAnalysis.js"; + +const normalizedSyntaxFingerprint = (source: string): string => { + const [fingerprint] = analyzeJavaScriptSemantics(source).functionFingerprints; + expect(fingerprint?.status).toBe("complete"); + return fingerprint?.components.normalizedAstSha256 ?? ""; +}; + +it("distinguishes yielding an iterator result from delegating to the iterator", () => { + expect( + normalizedSyntaxFingerprint("function* sample(gen) { yield* gen(); }"), + ).not.toBe( + normalizedSyntaxFingerprint("function* sample(gen) { yield gen(); }"), + ); +}); + +it("distinguishes returned prefix and postfix updates", () => { + expect( + normalizedSyntaxFingerprint("function sample(i) { return ++i; }"), + ).not.toBe(normalizedSyntaxFingerprint("function sample(i) { return i++; }")); +}); + +it("keeps formatting and local renames normalized", () => { + expect( + normalizedSyntaxFingerprint( + "function renamed(value) {\n return value++ ;\n }", + ), + ).toBe(normalizedSyntaxFingerprint("function sample(i) { return i++; }")); +}); diff --git a/src/domain/javascript/javascriptSemanticFingerprints.ts b/src/domain/javascript/javascriptSemanticFingerprints.ts index 8b6edc0365..d46305362f 100644 --- a/src/domain/javascript/javascriptSemanticFingerprints.ts +++ b/src/domain/javascript/javascriptSemanticFingerprints.ts @@ -1,6 +1,7 @@ import { createHash } from "node:crypto"; import * as t from "@babel/types"; +import { runtimeFunctionParameters } from "./javascriptAstValues.js"; import { compareUnicodeCodePoints } from "../unicodeCodePointOrder.js"; import type { JavaScriptSemanticCallAnalysis } from "./javascriptSemanticCalls.js"; @@ -197,6 +198,8 @@ const normalizedNodeToken = (node: t.Node, parent: t.Node | null): string => { primitiveAttribute(node, "optional"), primitiveAttribute(node, "async"), primitiveAttribute(node, "generator"), + primitiveAttribute(node, "delegate"), + primitiveAttribute(node, "prefix"), ].filter((value): value is string => value !== null); return [node.type, ...attributes].join(":"); }; @@ -249,6 +252,9 @@ const controlFlowToken = (node: t.Node): string | null => { }; const semanticLiteral = (node: t.Node): string | null => { + // Babel retains directive escapes in value; decoding them would erase the + // distinction between a strict-mode directive and an escaped lookalike. + if (t.isDirectiveLiteral(node)) return `directive:${node.value}`; if (t.isStringLiteral(node)) return `string:${node.value}`; if (t.isNumericLiteral(node)) return `number:${String(node.value)}`; if (t.isBooleanLiteral(node)) return `boolean:${String(node.value)}`; @@ -279,7 +285,7 @@ const relationShape = ( ]; const callableParameterArity = (node: t.Node): number => - t.isFunction(node) ? node.params.length : 0; + t.isFunction(node) ? runtimeFunctionParameters(node).length : 0; const isAsyncCallable = (node: t.Node): boolean => t.isFunction(node) && node.async; diff --git a/src/domain/javascript/javascriptSemanticGraph.ts b/src/domain/javascript/javascriptSemanticGraph.ts index cdaa8c28b8..4b77391f9e 100644 --- a/src/domain/javascript/javascriptSemanticGraph.ts +++ b/src/domain/javascript/javascriptSemanticGraph.ts @@ -306,9 +306,11 @@ function* checkCoverageSteps( relationsByFamily.set(family, (relationsByFamily.get(family) ?? 0) + 1); } const unknownsById = new Map(); + const unknownFamilies = new Set(); for (const [index, unknown] of graph.unknowns.entries()) { if (index % 256 === 0) yield; unknownsById.set(unknown.unknown_id, unknown); + unknownFamilies.add(unknown.family); } for (const [index, family] of graph.coverage.families.entries()) { if (family.retained_relations !== relationsByFamily.get(family.family)) @@ -329,7 +331,9 @@ function* checkCoverageSteps( }); if ( family.status === "complete" && - (family.omitted_relations !== 0 || family.unknown_ids.length > 0) + (family.omitted_relations !== 0 || + family.unknown_ids.length > 0 || + unknownFamilies.has(family.family)) ) context.addIssue({ code: "custom", diff --git a/src/domain/javascript/javascriptSemanticGraphQuery.test.ts b/src/domain/javascript/javascriptSemanticGraphQuery.test.ts index 8b3e0e1112..b24e105f33 100644 --- a/src/domain/javascript/javascriptSemanticGraphQuery.test.ts +++ b/src/domain/javascript/javascriptSemanticGraphQuery.test.ts @@ -2,6 +2,7 @@ import { semanticFixtureGraph, semanticFixtureInput, semanticGraphTestEvidence, + unknownSemanticGraphTestEvidence, } from "../../../tests/support/javascriptSemanticGraphFixture.js"; import { expect, it } from "vitest"; import { z } from "zod"; @@ -10,6 +11,7 @@ import { digestCanonicalValue } from "../canonicalDigest.js"; import type { JsonValue } from "../jsonValue.js"; import type { ApplicationGraphEvidence } from "./javascriptApplicationEvidenceSchemas.js"; import { + JAVASCRIPT_SEMANTIC_RELATION_FAMILIES, JAVASCRIPT_SEMANTIC_RELATION_FAMILY, JAVASCRIPT_SEMANTIC_NODE_KINDS, JAVASCRIPT_SEMANTIC_RELATIONS, @@ -244,6 +246,86 @@ it("matches scalar literal seeds and skips non-scalar literal metadata", () => { expect(query("unmatched")).toEqual([]); }); +it("scopes callers unknowns to call relations while preserving graph uncertainty", () => { + const contexts = new JavaScriptSemanticEvidenceContextRegistry(); + const caller = node("function", "caller", {}, contexts); + const seed = node("function", "seed", {}, contexts); + const call = createJavaScriptSemanticGraphRelation( + { + source_node_id: caller.node_id, + target_node_id: seed.node_id, + relation: "calls", + resolution: "resolved", + properties: {}, + evidence: semanticGraphTestEvidence("inferred"), + }, + contexts, + ); + const readUnknown = createJavaScriptSemanticGraphUnknown( + { + node_id: caller.node_id, + family: "data-flow", + relation_kinds: ["reads"], + reason: "dynamic-property", + detail: "Unresolved property read at the caller.", + candidate_node_ids: [], + evidence: unknownSemanticGraphTestEvidence(), + }, + contexts, + ); + const graph = createJavaScriptSemanticGraph({ + schema: "JavaScriptSemanticRelationGraph", + root_artifact_sha256: SHA, + application_graph_id: `jag_${"b".repeat(64)}`, + root_node_ids: [seed.node_id], + evidence_contexts: contexts.contexts, + nodes: [caller, seed], + relations: [call], + fingerprints: [], + unknowns: [readUnknown], + coverage: { + status: "partial", + truncated: false, + omitted_nodes: null, + omitted_relations: null, + limits: [], + families: JAVASCRIPT_SEMANTIC_RELATION_FAMILIES.map((family) => ({ + family, + status: + family === "data-flow" ? ("unknown" as const) : ("complete" as const), + retained_relations: + JAVASCRIPT_SEMANTIC_RELATION_FAMILY[call.relation] === family ? 1 : 0, + omitted_relations: family === "data-flow" ? null : 0, + unknown_ids: family === "data-flow" ? [readUnknown.unknown_id] : [], + })), + }, + limitations: ["One property read remains unresolved."], + }); + const query = ( + direction: "callers" | "backward-provenance", + allowed_relations?: ["calls"], + ) => + queryJavaScriptSemanticGraph(graph, { + seed: { kind: "semantic-node", node_id: seed.node_id }, + direction, + ...(allowed_relations === undefined ? {} : { allowed_relations }), + }); + const callers = query("callers"); + const filtered = query("callers", ["calls"]); + const backward = query("backward-provenance"); + expect(callers.summary.traversed_nodes).toBe(2); + expect(callers.unknowns).toEqual([]); + expect(callers.summary.relevant_unknowns).toBe(0); + expect(callers.limitations).not.toContain( + "Relevant dynamic, unsupported, incomplete, or ambiguous semantics remain unknown.", + ); + expect(filtered.unknowns).toEqual(callers.unknowns); + expect(backward.unknowns.map(({ unknown_id }) => unknown_id)).toEqual([ + readUnknown.unknown_id, + ]); + expect(callers.coverage.status).toBe("partial"); +}); + it("counts candidate ownership cycles once and traverses them finitely", () => { const graph = graphWithCandidateRelations(); const binding = graph.nodes.find(({ label }) => label === "binding"); @@ -336,6 +418,69 @@ it("rejects stale identities, dangling endpoints, and incomplete coverage claims ); }); +it("rejects complete family coverage with an unlisted retained frontier", () => { + const input = semanticFixtureInput(true); + input.coverage = { + ...input.coverage, + status: "complete", + truncated: false, + omitted_nodes: 0, + omitted_relations: 0, + families: input.coverage.families.map((family) => + family.family === "call-flow" + ? { + ...family, + status: "complete", + omitted_relations: 0, + unknown_ids: [], + } + : family, + ), + }; + input.limitations = []; + expect(() => createJavaScriptSemanticGraph(input)).toThrow( + /Complete family coverage cannot omit or retain unknown facts/u, + ); + + const valid = createJavaScriptSemanticGraph(semanticFixtureInput(true)); + const coverage = { + ...valid.coverage, + status: "complete" as const, + truncated: false, + omitted_nodes: 0, + omitted_relations: 0, + families: valid.coverage.families.map((family) => + family.family === "call-flow" + ? { + ...family, + status: "complete" as const, + omitted_relations: 0, + unknown_ids: [], + } + : family, + ), + }; + const { graph_id: _previousId, ...semantic } = valid; + const changed = { ...semantic, coverage, limitations: [] }; + const invalid = { + ...changed, + graph_id: `jsrg_${digestCanonicalValue( + changed, + "JavaScript semantic graph", + )}`, + }; + const parsed = javaScriptSemanticGraphSchema.safeParse(invalid); + expect(parsed.success).toBe(false); + if (parsed.success) throw new Error("Expected inconsistent graph rejection"); + expect(parsed.error.issues.map(({ message }) => message)).toContain( + "Complete family coverage cannot omit or retain unknown facts", + ); + expect( + javaScriptSemanticGraphSchema.safeParse(JSON.parse(JSON.stringify(invalid))) + .success, + ).toBe(false); +}); + it("validates evidence context ownership and references", () => { const graph = semanticFixtureGraph(); const alteredContext = { diff --git a/src/domain/javascript/javascriptSemanticGraphSchemas.ts b/src/domain/javascript/javascriptSemanticGraphSchemas.ts index 4048ae698e..115d3762ec 100644 --- a/src/domain/javascript/javascriptSemanticGraphSchemas.ts +++ b/src/domain/javascript/javascriptSemanticGraphSchemas.ts @@ -184,7 +184,6 @@ const relativePathSchema = z .refine( (path) => !path.startsWith("/") && - !path.includes("\\") && path.split("/").every((part) => !["", ".", ".."].includes(part)), "Expected a normalized relative POSIX path without traversal", ); diff --git a/src/domain/javascript/javascriptSemanticIterationSources.ts b/src/domain/javascript/javascriptSemanticIterationSources.ts index ea3708e515..0521f76b8d 100644 --- a/src/domain/javascript/javascriptSemanticIterationSources.ts +++ b/src/domain/javascript/javascriptSemanticIterationSources.ts @@ -2,6 +2,7 @@ import * as t from "@babel/types"; import { semanticObjectPatternKeys, + runtimeFunctionParameters, semanticStaticPropertyKey, } from "./javascriptAstValues.js"; import { @@ -212,7 +213,8 @@ export const semanticIterationSources = ( else if (t.isYieldExpression(node) && node.delegate && node.argument != null) add(sources, { node: node.argument, projection: [] }, node, {}); else if (t.isFunction(node)) - for (const parameter of node.params) visit(context, parameter, undefined); + for (const parameter of runtimeFunctionParameters(node)) + visit(context, parameter, undefined); return sources; }; diff --git a/src/domain/javascript/javascriptSemanticMemberMutations.ts b/src/domain/javascript/javascriptSemanticMemberMutations.ts index 7b988af889..ce7310e367 100644 --- a/src/domain/javascript/javascriptSemanticMemberMutations.ts +++ b/src/domain/javascript/javascriptSemanticMemberMutations.ts @@ -1,5 +1,14 @@ import * as t from "@babel/types"; +import { + semanticGlobalProcessOrigin, + semanticGlobalProcessPathOrigin, + semanticOriginalGlobalProcessAt, + semanticMutableBuiltinOrigin, + semanticRequireExportsIdentity, + semanticRequireOrigin, +} from "./javascriptSemanticRequireOrigin.js"; + import { resolveSemanticBindingState, type JavaScriptSemanticBindingState, @@ -35,6 +44,7 @@ import { compareUnicodeCodePoints } from "../unicodeCodePointOrder.js"; import { traverseJavaScriptAstSteps } from "./javascriptSemanticTraversal.js"; import { semanticStaticPropertyKey, + semanticWriteSiteTarget, semanticWriteTargets, unwrapJavaScriptExpression, } from "./javascriptAstValues.js"; @@ -43,6 +53,82 @@ type PropertyPath = JavaScriptSemanticPropertyPath; type ValueEffect = "write" | "escape"; /** Preserve uncertainty from property writes and references exposed to calls, yielding during traversal and fixpoint iteration. */ + +/** Setter keys of one object literal; a computed setter key may match any key. */ +interface SetterKeys { + readonly names: ReadonlySet; + readonly computed: boolean; +} + +// Writes to one literal repeat; scan its properties once, including the +// common result that it defines no setter. +const setterKeysByObject = new WeakMap(); + +const setterKeys = (object: t.ObjectExpression): SetterKeys | null => { + const cached = setterKeysByObject.get(object); + if (cached !== undefined) return cached; + const names = new Set(); + let computed = false; + // Follow definition order: a later data property or method with the same + // static name replaces the accessor, so its setter no longer runs. Spreads + // and unknown computed keys may or may not replace it and keep it. + for (const property of object.properties) { + if (t.isSpreadElement(property)) continue; + const name = semanticStaticPropertyKey(property.key, property.computed); + if (t.isObjectMethod(property) && property.kind === "set") { + if (name === null) computed = true; + else names.add(name); + } else if ( + name !== null && + !(t.isObjectMethod(property) && property.kind === "get") + ) + names.delete(name); + } + const keys = names.size === 0 && !computed ? null : { names, computed }; + setterKeysByObject.set(object, keys); + return keys; +}; + +/** + * The receiver path whose own object-literal setter an assignment to `path` + * runs, following nested literal members from the binding's initializer. + */ +const localSetterReceiver = ( + initializers: readonly { + readonly node: t.Node; + readonly projection: readonly unknown[]; + }[], + path: readonly unknown[], +): readonly string[] | undefined => { + const key = path.at(-1); + if (typeof key !== "string") return undefined; + const receiver = path.slice(0, -1); + if (!receiver.every((segment) => typeof segment === "string")) + return undefined; + for (const { node, projection } of initializers) { + if (projection.length !== 0) continue; + let object: t.Node | undefined = unwrapJavaScriptExpression(node).node; + for (const segment of receiver) { + if (!t.isObjectExpression(object)) break; + const member: t.ObjectExpression["properties"][number] | undefined = + object.properties.findLast( + (property) => + t.isObjectProperty(property) && + semanticStaticPropertyKey(property.key, property.computed) === + segment, + ); + object = t.isObjectProperty(member) + ? unwrapJavaScriptExpression(member.value).node + : undefined; + } + if (!t.isObjectExpression(object)) continue; + const keys = setterKeys(object); + if (keys !== null && (keys.computed || keys.names.has(key))) + return receiver as readonly string[]; + } + return undefined; +}; + export function* collectSemanticMemberMutationsSteps( program: t.Program, state: JavaScriptSemanticAnalysisState, @@ -50,10 +136,114 @@ export function* collectSemanticMemberMutationsSteps( const parents = state.parentsByNode; const observablePaths = observableMutationPaths(state); const returnedReferences = createSemanticReturnedReferences(state); - const recordedEffects = new Set(); + const recordedRequiredEffects = new Set(); + let recordedEffectCount = 0; + const recordRequiredEffect = ( + specifier: string, + path: PropertyPath, + effect: ValueEffect, + node: t.Node | undefined, + ): void => { + const identity = semanticRequireExportsIdentity(specifier, path); + let effectPath = identity.path; + // Only known value slots are primitive. Unknown or inherited selections + // can expose the container prototype and affect any subsequently read slot. + if ( + effect === "escape" && + identity.specifier === "process" && + effectPath.length === 2 && + (effectPath[0] === "env" || effectPath[0] === "argv") + ) { + const selected = effectPath[1]; + if ( + (typeof selected === "string" || typeof selected === "number") && + (effectPath[0] === "env" + ? !Object.hasOwn(Object.prototype, selected) + : selected === "length" || semanticArrayIndex(selected) !== null) + ) + return; + effectPath = [effectPath[0], null]; + } + const key = JSON.stringify([ + "required", + identity.specifier, + effect, + effectPath, + node?.start, + node?.end, + ]); + if (recordedRequiredEffects.has(key)) return; + recordedRequiredEffects.add(key); + recordedEffectCount += 1; + const effects = state.requiredModuleEffects.get(identity.specifier) ?? []; + effects.push({ path: effectPath, node, kind: effect }); + state.requiredModuleEffects.set(identity.specifier, effects); + }; + // Ambient indexing owns aliases of global objects. Consume its canonical + // writes instead of reparsing qualified or aliased process assignments. + for (const { path, node } of state.ambientWriteEffects) + if ( + path[0] === "global" && + path[1] === "process" && + path.length > 2 && + node !== undefined && + semanticOriginalGlobalProcessAt(node, state) + ) + recordRequiredEffect("node:process", path.slice(2), "write", node); const selections = new Map(); const coveringPaths = new Map(); let arrayIterationUnknown = false; + // Definition collection finishes before this phase, so a scope's bindings no + // longer change and one node resolves to one binding. Repeated visits of the + // same node otherwise repeat the same scope walk. + const resolvedBindings = new WeakMap< + t.Node, + JavaScriptSemanticBindingState | null + >(); + const resolveBinding = ( + node: t.Node, + name: string, + ): JavaScriptSemanticBindingState | undefined => { + const cached = resolvedBindings.get(node); + if (cached !== undefined) return cached ?? undefined; + const resolved = resolveSemanticBindingState(state, node, name); + resolvedBindings.set(node, resolved ?? null); + return resolved; + }; + // One capture tuple recurs across every alias it reaches. Re-encoding it + // would rebuild the same long key and force a fresh string hash per lookup. + const captureKeys = new WeakMap< + JavaScriptSemanticBindingState, + Map + >(); + const captureKeyOf = ( + binding: JavaScriptSemanticBindingState, + originAt: t.Node | undefined, + ownerInitializer: t.Node | undefined, + ): string => { + let byPosition = captureKeys.get(binding); + if (byPosition === undefined) { + byPosition = new Map(); + captureKeys.set(binding, byPosition); + } + const positionKey = `${originAt?.start ?? "u"}:${originAt?.end ?? "u"}:${ + ownerInitializer?.start ?? "u" + }:${ownerInitializer?.end ?? "u"}`; + const cached = byPosition.get(positionKey); + if (cached !== undefined) return cached; + const created = JSON.stringify([ + binding.bindingId, + originAt?.start, + originAt?.end, + ownerInitializer?.start, + ownerInitializer?.end, + ]); + byPosition.set(positionKey, created); + return created; + }; + // A capture key already ends the array; a path stays one nested element. + const lifetimeKey = (captureKey: string, path: PropertyPath): string => + `${captureKey.slice(0, -1)},${JSON.stringify(path)}]`; const pendingReferences: { readonly initializer: JavaScriptSemanticBindingState["referenceInitializers"][number]; readonly path: PropertyPath; @@ -120,13 +310,37 @@ export function* collectSemanticMemberMutationsSteps( const current = pending[cursor]; if (current === undefined) break; const expression = unwrapJavaScriptExpression(current.node).node; - if (t.isIdentifier(expression)) { - const binding = resolveSemanticBindingState( - state, - expression, - expression.name, + const required = semanticRequireOrigin(expression, state); + if (required !== undefined) { + recordRequiredEffect( + required.origin.specifier, + [...required.origin.importedPath, ...current.path], + effect, + mutation, ); + continue; + } + const globalProcess = semanticGlobalProcessPathOrigin(expression, state); + if (globalProcess !== undefined) { + recordRequiredEffect( + globalProcess.specifier, + [...globalProcess.importedPath, ...current.path], + effect, + mutation, + ); + continue; + } + if (t.isIdentifier(expression)) { + const binding = resolveBinding(expression, expression.name); if (binding === undefined) { + const processOrigin = semanticGlobalProcessOrigin(expression, state); + if (processOrigin !== undefined) + recordRequiredEffect( + processOrigin.specifier, + current.path, + effect, + mutation, + ); if ( (expression.name === "Array" && (current.path[0] === "prototype" || current.path[0] === null)) || @@ -142,21 +356,18 @@ export function* collectSemanticMemberMutationsSteps( // An alias restored from a saved reference can revisit the same binding // at an earlier capture or a shorter getter-receiver path. Only a // repeated capture and path closes a cycle. - const capture = [ - binding.bindingId, - current.originAt?.start, - current.originAt?.end, - current.ownerInitializer?.start, - current.ownerInitializer?.end, - ]; - const lifetime = JSON.stringify([...capture, current.path]); + const captureKey = captureKeyOf( + binding, + current.originAt, + current.ownerInitializer, + ); + const lifetime = lifetimeKey(captureKey, current.path); if (current.bindings.has(lifetime)) continue; // A self alias such as `node = node[key]` revisits its capture with the // same prefix added on every cycle, so exact paths never repeat. When // that prefix can never be pruned, escaping it once covers every longer // effect, including writes below shared children. // Only a capture already on this alias chain can repeat a prefix. - const captureKey = JSON.stringify(capture); const revisited = current.bindings.has(captureKey); let repeatedPrefix: PropertyPath | undefined; for ( @@ -165,8 +376,7 @@ export function* collectSemanticMemberMutationsSteps( start++ ) { const suffix = current.path.slice(start); - if (!current.bindings.has(JSON.stringify([...capture, suffix]))) - continue; + if (!current.bindings.has(lifetimeKey(captureKey, suffix))) continue; repeatedPrefix = current.path.slice(0, start); break; } @@ -227,52 +437,6 @@ export function* collectSemanticMemberMutationsSteps( index === selectorIndex ? { ...selector, excludedKeys } : key, ); } - // Keep an antichain of exact paths. Widening distinct member keys would - // also invalidate unrelated siblings that never reach this effect. - const pathState = JSON.stringify([ - binding.bindingId, - effect, - // Short writes stop at owned literal slots, whereas longer writes - // reach shared children. Only equal-depth writes cover one another. - effect === "write" ? path.length : null, - mutation?.start, - mutation?.end, - current.originAt?.start, - current.originAt?.end, - current.ownerInitializer?.start, - current.ownerInitializer?.end, - ]); - let covering = coveringPaths.get(pathState); - if (covering === undefined) { - covering = new SemanticPropertyPathCoverage(); - coveringPaths.set(pathState, covering); - } - if (!covering.retain(path)) continue; - const identity = JSON.stringify([ - binding.bindingId, - effect, - path, - mutation?.start, - mutation?.end, - current.originAt?.start, - current.originAt?.end, - current.ownerInitializer?.start, - current.ownerInitializer?.end, - ]); - if (recordedEffects.has(identity)) continue; - // A recorded write invalidates the binding to a non-primitive value, so - // only its first write can target a primitive. Re-evaluating it would - // replay every earlier write and make repeated writes superlinear. - const value = - effect === "write" && binding.mutatedPaths.length === 0 - ? evaluateSemanticBinding(binding, state) - : undefined; - const primitiveWrite = - value?.status === "literal" || value?.status === "union"; - recordedEffects.add(identity); - const nested = new Set([...current.bindings, lifetime, captureKey]); - // Returning a literal's receiver reaches that allocation, not the - // binding's older values read while evaluating its initializer. const origins = current.ownerInitializer !== undefined ? { @@ -280,6 +444,7 @@ export function* collectSemanticMemberMutationsSteps( ({ node }) => node === current.ownerInitializer, ), referenceInitializers: [], + directOrigins: [], } : current.originAt === undefined ? binding @@ -288,10 +453,67 @@ export function* collectSemanticMemberMutationsSteps( current.originAt, parents, ); + // A setter on the binding's own literal runs on assignment and may + // replace any member of its receiver, so the written key is unknown. + // Deleting the accessor never calls it. + if ( + effect === "write" && + !t.isUnaryExpression(mutation, { operator: "delete" }) + ) { + const receiver = localSetterReceiver(origins.initializers, path); + if (receiver !== undefined) path = [...receiver, null]; + } + // Keep an antichain of exact paths. Widening distinct member keys would + // also invalidate unrelated siblings that never reach this effect. + // This map is local to the file scan. Escaping the binding id keeps a + // field separator injective while avoiding a fresh JSON array encode. + const pathState = `${JSON.stringify(binding.bindingId)}\u0001${ + effect + }\u0001${ + // Short writes stop at owned literal slots, whereas longer writes + // reach shared children. Only equal-depth writes cover one another. + effect === "write" ? path.length : -1 + }\u0001${mutation?.start ?? -1}\u0001${mutation?.end ?? -1}\u0001${ + current.originAt?.start ?? -1 + }\u0001${current.originAt?.end ?? -1}\u0001${ + current.ownerInitializer?.start ?? -1 + }\u0001${current.ownerInitializer?.end ?? -1}`; + let covering = coveringPaths.get(pathState); + if (covering === undefined) { + covering = new SemanticPropertyPathCoverage(); + coveringPaths.set(pathState, covering); + } + if (!covering.retain(path)) continue; + // A recorded write invalidates the binding to a non-primitive value, so + // only its first write can target a primitive. Re-evaluating it would + // replay every earlier write and make repeated writes superlinear. + const value = + effect === "write" && binding.memberWrites.length === 0 + ? evaluateSemanticBinding(binding, state) + : undefined; + const primitiveWrite = + value?.status === "literal" || value?.status === "union"; + // The coverage index already owns binding-effect deduplication. + recordedEffectCount += 1; + const nested = new Set(current.bindings); + nested.add(lifetime); + nested.add(captureKey); + // Returning a literal's receiver reaches that allocation, not the + // binding's older values read while evaluating its initializer. if (!primitiveWrite) { + for (const origin of origins.directOrigins) { + const shared = semanticMutableBuiltinOrigin(origin, path); + if (shared !== undefined) + recordRequiredEffect( + shared.specifier, + shared.path, + effect, + mutation, + ); + } if (effect === "escape") binding.escapedPaths.push({ path, node: mutation }); - else binding.mutatedPaths.push(path); + else binding.memberWrites.push({ path, node: mutation }); clearSemanticValueCaches(state); for (const initializer of origins.initializers) pending.push({ @@ -408,7 +630,7 @@ export function* collectSemanticMemberMutationsSteps( if (!t.isNode(node)) continue; const pathKey = JSON.stringify(value.path); if (t.isIdentifier(node) && value.iterableFallbackPath === undefined) { - const binding = resolveSemanticBindingState(state, node, node.name); + const binding = resolveBinding(node, node.name); const identity = JSON.stringify([ binding === undefined ? null : binding.bindingId, binding === undefined ? node.name : null, @@ -471,12 +693,8 @@ export function* collectSemanticMemberMutationsSteps( originAt: source.mutation, iterationOnly: source.kind === "iteration", }); - if (t.isAssignmentExpression(node)) markTarget(node.left, node); - else if (t.isUpdateExpression(node)) markTarget(node.argument, node); - else if (t.isUnaryExpression(node, { operator: "delete" })) - markTarget(node.argument, node); - else if (t.isForOfStatement(node) || t.isForInStatement(node)) - markTarget(node.left, node); + const writeTarget = semanticWriteSiteTarget(node); + if (writeTarget !== null) markTarget(writeTarget, node); else if ( t.isCallExpression(node) || t.isOptionalCallExpression(node) || @@ -509,7 +727,7 @@ export function* collectSemanticMemberMutationsSteps( // A later effect can make a previously defined destructuring source uncertain. // Reconsider defaults and iterable origins until their effects are stable. while (pendingReferences.length > 0 || iterableReferences.length > 0) { - const effectsBefore = recordedEffects.size; + const effectsBefore = recordedEffectCount; const iterablesBefore = iterableReferences.length; const iterationBefore: boolean = arrayIterationUnknown; for (const [index, reference] of pendingReferences.splice(0).entries()) { @@ -588,7 +806,7 @@ export function* collectSemanticMemberMutationsSteps( ); } if ( - recordedEffects.size === effectsBefore && + recordedEffectCount === effectsBefore && iterableReferences.length === iterablesBefore && arrayIterationUnknown === iterationBefore ) @@ -838,6 +1056,15 @@ const observableMutationPaths = ( for (const binding of state.bindingsById.values()) { const observer = bindingNodes.get(binding.bindingId); if (observer === undefined) continue; + // Imported values are external objects with observable member slots even + // when their syntax has no local object-allocation node (for example a + // CommonJS namespace from `require("node:fs")`). Keep those writes at their + // exact member paths instead of widening the first path to a wildcard. + const hasParsedRequireOrigin = binding.initializers.some( + ({ node }) => semanticRequireOrigin(node, state) !== undefined, + ); + if (binding.directOrigins.length > 0 || hasParsedRequireOrigin) + (observer.origins ??= new Set()).add("opaque"); for (const initializer of binding.initializers) { link( project(expressionNode(initializer.node), initializer.projection), diff --git a/src/domain/javascript/javascriptSemanticMutationInitializers.test.ts b/src/domain/javascript/javascriptSemanticMutationInitializers.test.ts index 8b5364578a..8a61792432 100644 --- a/src/domain/javascript/javascriptSemanticMutationInitializers.test.ts +++ b/src/domain/javascript/javascriptSemanticMutationInitializers.test.ts @@ -47,8 +47,9 @@ describe("parameter references copied into function body bindings", () => { scopeId: "body", name: "arg", kind: "variable", + declarationKinds: new Set(), mutable: true, - mutatedPaths: [], + memberWrites: [], escapedPaths: [], definitions: [], initializers: [entry, assigned], diff --git a/src/domain/javascript/javascriptSemanticMutationInitializers.ts b/src/domain/javascript/javascriptSemanticMutationInitializers.ts index 6de7217c1a..fba3882261 100644 --- a/src/domain/javascript/javascriptSemanticMutationInitializers.ts +++ b/src/domain/javascript/javascriptSemanticMutationInitializers.ts @@ -12,6 +12,37 @@ interface StatementPosition { readonly declaratorIndex: number; } +/** + * Initializer positions depend only on a node and its parent map, which one + * analysis builds once and never changes. Different mutations and initializers + * re-derive the same positions along shared ancestry, so retain the first + * result for as long as its parent map is reachable. + */ +interface InitializerPositionCache { + readonly direct: WeakMap; + readonly mutation: WeakMap; + readonly reference: WeakMap; +} + +const initializerPositionCaches = new WeakMap< + WeakMap, + InitializerPositionCache +>(); + +const initializerPositionCache = ( + parents: WeakMap, +): InitializerPositionCache => { + const cached = initializerPositionCaches.get(parents); + if (cached !== undefined) return cached; + const created: InitializerPositionCache = { + direct: new WeakMap(), + mutation: new WeakMap(), + reference: new WeakMap(), + }; + initializerPositionCaches.set(parents, created); + return created; +}; + const statementPosition = ( statement: t.Statement, parents: WeakMap, @@ -42,7 +73,95 @@ const positionPrecedes = ( left.declaratorIndex >= 0 && left.declaratorIndex < right.declaratorIndex)); -const directInitializerPosition = ( +/** Order effects only in a non-repeating top-level activation. */ +export const semanticTopLevelEffectFollowsCapture = ( + capture: t.Node, + effect: t.Node | undefined, + parents: WeakMap, +): boolean => { + const position = (node: t.Node | undefined): StatementPosition | null => { + let selected: StatementPosition | null = null; + let declarator: t.VariableDeclarator | undefined; + let child: t.Node | undefined; + while (node !== undefined) { + if (t.isFunction(node) || t.isLoop(node)) return null; + if ( + (t.isClassProperty(node) || + t.isClassPrivateProperty(node) || + t.isClassAccessorProperty(node)) && + !node.static && + node.value === child + ) + return null; + if (t.isVariableDeclarator(node)) declarator = node; + if (selected === null && t.isStatement(node)) + selected = statementPosition(node, parents, declarator); + if (t.isProgram(node)) return selected; + child = node; + node = parents.get(node); + } + return null; + }; + const before = position(capture); + const after = position(effect); + if (before === null || after === null) return false; + if (positionPrecedes(before, after)) return true; + if ( + before.body !== after.body || + before.index !== after.index || + before.declaratorIndex !== after.declaratorIndex + ) + return false; + return expressionEvaluationPrecedes(capture, effect, parents); +}; + +/** Prove ordering only where JavaScript evaluates sibling expressions left-to-right. */ +const expressionEvaluationPrecedes = ( + earlier: t.Node, + later: t.Node | undefined, + parents: WeakMap, +): boolean => { + if (later === undefined) return false; + const earlierAncestors = new Set(); + let current: t.Node | undefined = earlier; + while (current !== undefined) { + earlierAncestors.add(current); + current = parents.get(current); + } + const laterPath: t.Node[] = []; + current = later; + while (current !== undefined && !earlierAncestors.has(current)) { + laterPath.push(current); + current = parents.get(current); + } + const common = current; + if (common === undefined || common === earlier || common === later) + return false; + let earlierChild: t.Node = earlier; + while (parents.get(earlierChild) !== common) { + const parent = parents.get(earlierChild); + if (parent === undefined) return false; + earlierChild = parent; + } + const laterChild = laterPath.at(-1); + if (laterChild === undefined) return false; + const orderedChildren = ((): readonly t.Node[] | null => { + if (t.isSequenceExpression(common)) return common.expressions; + if (t.isCallExpression(common) || t.isOptionalCallExpression(common)) + return [common.callee, ...common.arguments]; + if (t.isNewExpression(common)) return [common.callee, ...common.arguments]; + if (t.isBinaryExpression(common) || t.isLogicalExpression(common)) + return [common.left, common.right]; + if (t.isConditionalExpression(common)) return [common.test]; + return null; + })(); + if (orderedChildren === null) return false; + const earlierIndex = orderedChildren.indexOf(earlierChild); + const laterIndex = orderedChildren.indexOf(laterChild); + return earlierIndex >= 0 && laterIndex > earlierIndex; +}; + +const computeDirectInitializerPosition = ( node: t.Node, parents: WeakMap, ): StatementPosition | null => { @@ -67,7 +186,19 @@ const directInitializerPosition = ( return null; }; -const directMutationPosition = ( +const directInitializerPosition = ( + node: t.Node, + parents: WeakMap, +): StatementPosition | null => { + const cache = initializerPositionCache(parents); + const cached = cache.direct.get(node); + if (cached !== undefined) return cached; + const computed = computeDirectInitializerPosition(node, parents); + cache.direct.set(node, computed); + return computed; +}; + +const computeDirectMutationPosition = ( node: t.Node, parents: WeakMap, ): StatementPosition | null => { @@ -115,7 +246,19 @@ const directMutationPosition = ( } }; -const referenceInitializerPosition = ( +const directMutationPosition = ( + node: t.Node, + parents: WeakMap, +): StatementPosition | null => { + const cache = initializerPositionCache(parents); + const cached = cache.mutation.get(node); + if (cached !== undefined) return cached; + const computed = computeDirectMutationPosition(node, parents); + cache.mutation.set(node, computed); + return computed; +}; + +const computeReferenceInitializerPosition = ( node: t.Node, parents: WeakMap, ): StatementPosition | null => { @@ -137,6 +280,18 @@ const referenceInitializerPosition = ( } }; +const referenceInitializerPosition = ( + node: t.Node, + parents: WeakMap, +): StatementPosition | null => { + const cache = initializerPositionCache(parents); + const cached = cache.reference.get(node); + if (cached !== undefined) return cached; + const computed = computeReferenceInitializerPosition(node, parents); + cache.reference.set(node, computed); + return computed; +}; + /** Resolve origins at their use or capture, never at a later transitive effect. */ export const semanticMutationInitializers = ( binding: JavaScriptSemanticBindingState, @@ -144,12 +299,64 @@ export const semanticMutationInitializers = ( parents: WeakMap, ): Pick< JavaScriptSemanticBindingState, - "initializers" | "referenceInitializers" + "initializers" | "referenceInitializers" | "directOrigins" > => { const mutationPosition = directMutationPosition(mutation, parents) ?? referenceInitializerPosition(mutation, parents); - return selectInitializers(binding, mutationPosition, parents); + if (mutationPosition === null) return binding; + const origins = [ + ...binding.initializers.map((initializer) => ({ + initializer, + position: + initializer.entryBody === undefined + ? directInitializerPosition(initializer.node, parents) + : { + body: initializer.entryBody.body, + index: -1, + declaratorIndex: -1, + }, + })), + ...binding.referenceInitializers.map((initializer) => ({ + initializer, + position: referenceInitializerPosition(initializer.node, parents), + })), + ]; + if ( + origins.some( + ({ position }) => + position === null || !positionPrecedes(position, mutationPosition), + ) + ) + return binding; + const latest = origins.reduce( + (latest, { position }) => + position !== null && + (latest === null || positionPrecedes(latest, position)) + ? position + : latest, + null, + ); + const retained = new Set( + origins + .filter( + ({ position }) => + position?.index === latest?.index && + position?.declaratorIndex === latest?.declaratorIndex, + ) + .map(({ initializer }) => initializer), + ); + return { + // A proven earlier assignment replaces even an import-equals value. A + // capture before or inside an uncertain assignment retains the full binding. + directOrigins: origins.length === 0 ? binding.directOrigins : [], + initializers: binding.initializers.filter((initializer) => + retained.has(initializer), + ), + referenceInitializers: binding.referenceInitializers.filter((initializer) => + retained.has(initializer), + ), + }; }; /** Select a local alias at its read, before any later straight-line reassignment. */ @@ -242,6 +449,16 @@ const selectInitializers = ( ) .map(({ initializer }) => initializer), ); + // Preserve the owner's identity when selection removes no facts. Consumers + // can then reuse the binding's dependency-aware provenance memo rather than + // treating every equivalent read as a new temporal snapshot. + if ( + binding.initializers.every((initializer) => retained.has(initializer)) && + binding.referenceInitializers.every((initializer) => + retained.has(initializer), + ) + ) + return binding; return { initializers: binding.initializers.filter((initializer) => retained.has(initializer), diff --git a/src/domain/javascript/javascriptSemanticObjectReferences.test.ts b/src/domain/javascript/javascriptSemanticObjectReferences.test.ts new file mode 100644 index 0000000000..f9867ed197 --- /dev/null +++ b/src/domain/javascript/javascriptSemanticObjectReferences.test.ts @@ -0,0 +1,73 @@ +import * as t from "@babel/types"; +import { expect, it } from "vitest"; + +import { indexJavaScriptBindingsSteps } from "./javascriptSemanticAnalysis.js"; +import { semanticObjectPropertyReferences } from "./javascriptSemanticObjectReferences.js"; +import { completeSemanticSteps } from "./javascriptSemanticTraversal.js"; +import { parseJavaScriptSource } from "./javascriptSourceParser.js"; + +const objectAndState = (source: string) => { + const parsed = parseJavaScriptSource(source, "object.js"); + if (parsed === null) throw new Error("Expected a parsed fixture"); + const declaration = parsed.program.body[0]; + if (!t.isVariableDeclaration(declaration)) + throw new Error("Expected a declaration"); + const init = declaration.declarations[0]?.init; + if (!t.isObjectExpression(init)) throw new Error("Expected an object"); + return { + object: init, + source, + state: completeSemanticSteps(indexJavaScriptBindingsSteps(parsed.program)), + }; +}; + +const spelled = ( + fixture: ReturnType, + key: string | number | null, +) => + semanticObjectPropertyReferences(fixture.object, [key], fixture.state).map( + ({ node, path }) => [ + fixture.source.slice(node.start ?? 0, node.end ?? 0), + JSON.stringify(path), + ], + ); + +it("resolves exact keys with source-order overrides, spreads and accessors", () => { + const fixture = objectAndState( + `const o = { a: 1, ...base, b: 2, a: 3, [dynamic]: 4, __proto__: proto, ...{ b: 5 }, get c() { return 6; }, c: 7 };`, + ); + // Expectations match the full source-order scan these exact keys replace. + expect(spelled(fixture, "a")).toEqual([ + ["{ b: 5 }", '["a"]'], + ["4", "[]"], + ["3", "[]"], + ]); + expect(spelled(fixture, "b")).toEqual([ + ["{ b: 5 }", '["b"]'], + ["4", "[]"], + ]); + expect(spelled(fixture, "c")).toEqual([ + ["7", "[]"], + ["4", "[]"], + ]); + expect(spelled(fixture, "missing")).toEqual([ + ["{ b: 5 }", '["missing"]'], + ["proto", '["missing"]'], + ["4", "[]"], + ["base", '["missing"]'], + ]); + // A wildcard key still scans every property. + expect(spelled(fixture, null)).toEqual([ + ["7", "[]"], + ["{ b: 5 }", '[{"excludedKeys":["c"]}]'], + ["proto", "[null]"], + ["4", "[]"], + ["3", "[]"], + ["2", "[]"], + ["base", '[{"excludedKeys":["a","b","c"]}]'], + ]); + // Numeric keys use the same exact spelling as string keys. + expect( + spelled(objectAndState(`const o = { 1: "one", 2: "two" };`), 1), + ).toEqual([['"one"', "[]"]]); +}); diff --git a/src/domain/javascript/javascriptSemanticObjectReferences.ts b/src/domain/javascript/javascriptSemanticObjectReferences.ts index 425e5fea5d..111417b5a7 100644 --- a/src/domain/javascript/javascriptSemanticObjectReferences.ts +++ b/src/domain/javascript/javascriptSemanticObjectReferences.ts @@ -739,6 +739,75 @@ const getterResultReferences = ( return references; }; +type ObjectLiteralMember = t.ObjectExpression["properties"][number]; + +/** Property positions grouped by whether an exact key can reach them. */ +interface ObjectPropertyIndex { + /** Spreads, computed keys and `__proto__` data entries reach every key. */ + readonly keyIndependent: readonly number[]; + readonly byName: ReadonlyMap; +} + +// Mutation tracking resolves one key of the same literal per write, so the +// index removes the per-write scan of every property. +const propertyIndexes = new WeakMap(); + +const objectPropertyIndex = (node: t.ObjectExpression): ObjectPropertyIndex => { + const cached = propertyIndexes.get(node); + if (cached !== undefined) return cached; + const keyIndependent: number[] = []; + const byName = new Map(); + node.properties.forEach((property, index) => { + const name = t.isSpreadElement(property) + ? null + : semanticStaticPropertyKey(property.key, property.computed); + if ( + name === null || + (t.isObjectProperty(property) && + !property.computed && + !property.shorthand && + name === "__proto__") + ) { + keyIndependent.push(index); + return; + } + const positions = byName.get(name); + if (positions === undefined) byName.set(name, [index]); + else positions.push(index); + }); + const index = { keyIndependent, byName }; + propertyIndexes.set(node, index); + return index; +}; + +/** + * Properties in reverse source order that can affect `key`. For an exact key, + * another static name only updates bookkeeping for that other name, which + * this key never reads, so only same-name and key-independent entries remain. + */ +const relevantPropertiesInReverse = ( + node: t.ObjectExpression, + key: PropertyPath[number] | undefined, +): readonly ObjectLiteralMember[] => { + if (typeof key !== "string" && typeof key !== "number") + return [...node.properties].reverse(); + const { keyIndependent, byName } = objectPropertyIndex(node); + const named = byName.get(String(key)) ?? []; + const relevant: ObjectLiteralMember[] = []; + let left = keyIndependent.length - 1; + let right = named.length - 1; + while (left >= 0 || right >= 0) { + const fromLeft = + right < 0 || + (left >= 0 && (keyIndependent[left] ?? -1) > (named[right] ?? -1)); + const position = fromLeft ? keyIndependent[left--] : named[right--]; + const property = + position === undefined ? undefined : node.properties[position]; + if (property !== undefined) relevant.push(property); + } + return relevant; +}; + export const semanticObjectPropertyReferences = ( node: t.ObjectExpression, path: PropertyPath, @@ -751,7 +820,7 @@ export const semanticObjectPropertyReferences = ( const accessorNames = new Set(); const getterNames = new Set(); const setterNames = new Set(); - for (const property of [...node.properties].reverse()) { + for (const property of relevantPropertiesInReverse(node, key)) { if (t.isSpreadElement(property)) { const selected = key ?? null; if (typeof selected !== "object" && overwritten.has(String(selected))) @@ -785,6 +854,33 @@ export const semanticObjectPropertyReferences = ( continue; } const name = semanticStaticPropertyKey(property.key, property.computed); + if ( + t.isObjectProperty(property) && + !property.computed && + !property.shorthand && + name === "__proto__" + ) { + const selected = key ?? null; + if ( + (selected === null || typeof selected === "object") && + t.isExpression(property.value) + ) { + references.push({ + node: property.value, + path: [selected, ...remaining], + }); + } else if ( + typeof selected === "string" && + !objectExpressionDefinesKey(node, selected) && + t.isExpression(property.value) + ) { + references.push({ + node: property.value, + path: [selected, ...remaining], + }); + } + continue; + } const isGetter = t.isObjectMethod(property) && property.kind === "get"; const isSetter = t.isObjectMethod(property) && property.kind === "set"; if ( diff --git a/src/domain/javascript/javascriptSemanticObjects.ts b/src/domain/javascript/javascriptSemanticObjects.ts index 08acf25f8a..6e4dbb0a1e 100644 --- a/src/domain/javascript/javascriptSemanticObjects.ts +++ b/src/domain/javascript/javascriptSemanticObjects.ts @@ -4,12 +4,12 @@ import type { JavaScriptSemanticCallable, JavaScriptSemanticObjectOperation, } from "./javascriptSemanticIr.js"; -import { - semanticCallableIdForNode, - semanticReadsBeforeWrite, -} from "./javascriptSemanticProjection.js"; +import { createSemanticCallableIdStack } from "./javascriptSemanticCallableStack.js"; import { semanticStaticPropertyKey, + semanticWriteSiteReadsFirst, + semanticWriteSiteTarget, + semanticWriteTargets, unwrapJavaScriptExpression, } from "./javascriptAstValues.js"; import { @@ -27,14 +27,20 @@ export const collectJavaScriptSemanticObjects = ( ): JavaScriptSemanticObjectOperation[] => { const output: JavaScriptSemanticObjectOperation[] = []; const callableIds = new Set(callables.map(({ callableId }) => callableId)); - const callableStack: string[] = []; - const context: ObjectCollectionContext = { state, output }; + const callableStack = createSemanticCallableIdStack(callableIds); + const context: ObjectCollectionContext = { + state, + output, + writeTargets: new Map(), + }; traverseJavaScriptAst(program, { enter: (node, parent) => { - const callableId = semanticCallableIdForNode(node); - if (callableId !== null && callableIds.has(callableId)) - callableStack.push(callableId); - const ownerCallableId = callableStack.at(-1) ?? null; + callableStack.enter(node, parent); + const ownerCallableId = callableStack.callables.at(-1) ?? null; + const writeTarget = semanticWriteSiteTarget(node); + if (writeTarget !== null) + for (const leaf of semanticWriteTargets(writeTarget)) + if (!t.isIdentifier(leaf)) context.writeTargets.set(leaf, node); if (t.isMemberExpression(node) || t.isOptionalMemberExpression(node)) collectMember(node, parent, ownerCallableId, context); else if ( @@ -61,11 +67,7 @@ export const collectJavaScriptSemanticObjects = ( ) collectDestructuring(node, ownerCallableId, state, output); }, - exit: (node) => { - const callableId = semanticCallableIdForNode(node); - if (callableId !== null && callableStack.at(-1) === callableId) - callableStack.pop(); - }, + exit: (node) => callableStack.exit(node), }); return output; }; @@ -73,6 +75,7 @@ export const collectJavaScriptSemanticObjects = ( interface ObjectCollectionContext { readonly state: JavaScriptSemanticAnalysisState; readonly output: JavaScriptSemanticObjectOperation[]; + readonly writeTargets: Map; } const collectMember = ( @@ -90,10 +93,11 @@ const collectMember = ( ) return; const identity = expressionObjectIdentity(node, state); - const write = - (t.isAssignmentExpression(parent) && parent.left === node) || - (t.isUpdateExpression(parent) && parent.argument === node); - const readsBeforeWrite = write && semanticReadsBeforeWrite(node, parent); + const writeSite = context.writeTargets.get(node); + context.writeTargets.delete(node); + const write = writeSite !== undefined; + const readsBeforeWrite = + writeSite !== undefined && semanticWriteSiteReadsFirst(writeSite); if (readsBeforeWrite) addObjectOperation( { diff --git a/src/domain/javascript/javascriptSemanticProjection.ts b/src/domain/javascript/javascriptSemanticProjection.ts index b477bb5af4..9389427d8f 100644 --- a/src/domain/javascript/javascriptSemanticProjection.ts +++ b/src/domain/javascript/javascriptSemanticProjection.ts @@ -9,7 +9,6 @@ import type { import { resolveSemanticBindingState, semanticResolutionBlocked, - isUnshadowedGlobal, type JavaScriptSemanticAnalysisState, type JavaScriptSemanticScopeState, } from "./javascriptSemanticState.js"; @@ -27,6 +26,10 @@ import { range } from "./javascriptStaticAnalysisHelpers.js"; import { propertyName, semanticStaticPropertyKey, + semanticWrappedWriteSite, + semanticWriteSiteReadsFirst, + semanticWriteSiteTarget, + unwrapJavaScriptExpression, } from "./javascriptAstValues.js"; import { isTypeOnlyExportDeclaration, @@ -34,6 +37,10 @@ import { isTypeOnlyImportDeclaration, isTypeOnlyImportSpecifier, } from "./javascriptModuleDeclaration.js"; +import { + commonJsExportName, + type CommonJsExportReceiverEffect, +} from "./javascriptCommonJsExportReceivers.js"; interface CollectCallableInput { readonly node: t.Node; @@ -74,6 +81,10 @@ export const collectSemanticCallable = (input: CollectCallableInput): void => { export const collectSemanticModuleLink = ( node: t.Node, state: JavaScriptSemanticAnalysisState, + commonJsReceivers: ReadonlyMap< + t.AssignmentExpression, + CommonJsExportReceiverEffect + >, ): void => { if (t.isTSImportEqualsDeclaration(node)) collectImportEquals(node, state); else if (t.isImportDeclaration(node)) collectImports(node, state); @@ -98,7 +109,7 @@ export const collectSemanticModuleLink = ( kind: "export", specifier: null, importedName: null, - localName: defaultDeclarationName(node.declaration), + localName: defaultExportLocalName(node.declaration), exportedName: "default", callableId: semanticCallableIdForNode(node.declaration), location: range(node), @@ -106,7 +117,8 @@ export const collectSemanticModuleLink = ( node, ); else if (t.isVariableDeclarator(node)) collectRequireLink(node, state); - else if (t.isAssignmentExpression(node)) collectCommonJsExport(node, state); + else if (t.isAssignmentExpression(node)) + collectCommonJsExport(node, state, commonJsReceivers.get(node)); else if (t.isCallExpression(node)) collectUnboundRequireLink(node, state); }; @@ -175,12 +187,17 @@ export function* collectSemanticReferencesSteps( !patterns.some((pattern) => bindsIdentifier(pattern, parent, node))) ) return; - const role = semanticIdentifierRole(node, parent, readAncestors()); + const ancestors = readAncestors(); + const role = semanticIdentifierRole(node, parent, ancestors); if (role === null) return; // A compound assignment or update reads before it writes; emit both so // reference consumers see the read they actually execute. const roles = - role === "write" && semanticReadsBeforeWrite(node, parent) + role === "write" && + semanticReadsBeforeWrite( + node, + semanticWrappedWriteSite(node, parent, ancestors) ?? parent, + ) ? (["read", "write"] as const) : ([role] as const); for (const resolvedRole of roles) { @@ -460,12 +477,10 @@ const requirePatternBindings = ( const collectCommonJsExport = ( node: t.AssignmentExpression, state: JavaScriptSemanticAnalysisState, + receiverEffect: CommonJsExportReceiverEffect | undefined, ): void => { const exportedName = commonJsExportName(node.left, state); if (exportedName === undefined) return; - const receiverEffect = isCommonJsModuleExports(node.left, state) - ? null - : commonJsExportReceiverEffect(node.right, state); if (receiverEffect === "detached") return; // Simple assignment chains yield their rightmost assigned value. Compound // and logical writes also depend on the previous value, so remain unknown. @@ -492,40 +507,6 @@ const collectCommonJsExport = ( if (required !== undefined) state.linkedRequireCalls.add(required.call); }; -/** - * An outer property assignment captures its receiver before evaluating the - * RHS. When that chain assigns a freshly created identity to `module.exports`, - * the outer write targets the old exports object and cannot define an export - * on the value ultimately returned by the CommonJS module. Reassignments from - * existing bindings may preserve identity, so this guard leaves them as - * export candidates. - */ -const commonJsExportReceiverEffect = ( - node: t.Node, - state: JavaScriptSemanticAnalysisState, -): "detached" | "uncertain" | null => { - let value = node; - while (t.isAssignmentExpression(value, { operator: "=" })) { - if (isCommonJsModuleExports(value.left, state)) - return isFreshAssignmentResult(value.right) ? "detached" : "uncertain"; - value = value.right; - } - return null; -}; - -const isFreshAssignmentResult = (node: t.Node): boolean => { - let value = node; - while (t.isAssignmentExpression(value, { operator: "=" })) - value = value.right; - return ( - t.isArrowFunctionExpression(value) || - t.isFunctionExpression(value) || - t.isClassExpression(value) || - t.isObjectExpression(value) || - t.isArrayExpression(value) - ); -}; - const addModuleLink = ( state: JavaScriptSemanticAnalysisState, link: Omit & { @@ -548,6 +529,11 @@ const addModuleLink = ( callableId: link.callableId ?? null, }; state.moduleLinks.push(collected); + if ( + t.isExportDefaultDeclaration(referenceNode) && + t.isExpression(referenceNode.declaration) + ) + state.defaultExportExpressions.set(collected, referenceNode.declaration); if (link.localName !== null) { const binding = resolveSemanticBindingState( state, @@ -594,12 +580,15 @@ const declarationCallableId = ( */ export const semanticReadsBeforeWrite = ( node: t.Node, - parent: t.Node, -): boolean => - (t.isAssignmentExpression(parent) && - parent.left === node && - parent.operator !== "=") || - (t.isUpdateExpression(parent) && parent.argument === node); + site: t.Node, +): boolean => { + const target = semanticWriteSiteTarget(site); + return ( + target !== null && + unwrapJavaScriptExpression(target).node === node && + semanticWriteSiteReadsFirst(site) + ); +}; /** * Role of one identifier occurrence, given the node's ancestry. @@ -620,6 +609,8 @@ export const semanticIdentifierRole = ( if (isPatternOwnerDeclaration(patternAncestor)) return null; return "write"; } + const site = semanticWrappedWriteSite(node, parent, ancestors); + if (site !== null && site !== parent && !isDeleteSite(site)) return "write"; return semanticReferenceRole(node, parent); }; @@ -684,6 +675,9 @@ const isPatternNode = (node: t.Node): boolean => t.isAssignmentPattern(node) || t.isRestElement(node); +const isDeleteSite = (site: t.Node): boolean => + t.isUnaryExpression(site, { operator: "delete" }); + const isPatternOwnerDeclaration = (node: t.Node): boolean => (t.isVariableDeclarator(node) && t.isPattern(node.id)) || (t.isFunction(node) && @@ -799,40 +793,13 @@ const declarationNames = (declaration: t.Declaration): string[] => { return []; }; -const defaultDeclarationName = ( +const defaultExportLocalName = ( declaration: t.ExportDefaultDeclaration["declaration"], -): string | null => - (t.isFunctionDeclaration(declaration) || t.isClassDeclaration(declaration)) && - t.isIdentifier(declaration.id) - ? declaration.id.name +): string | null => { + const value = unwrapJavaScriptExpression(declaration).node; + if (t.isIdentifier(value)) return value.name; + return (t.isFunctionDeclaration(value) || t.isClassDeclaration(value)) && + t.isIdentifier(value.id) + ? value.id.name : null; - -const commonJsExportName = ( - node: t.Node, - state: JavaScriptSemanticAnalysisState, -): string | undefined => { - // Rebinding the CommonJS `exports` alias does not replace `module.exports`. - if (!t.isMemberExpression(node) && !t.isOptionalMemberExpression(node)) - return undefined; - const key = semanticStaticPropertyKey(node.property, node.computed); - // `exports[key]`/`module.exports[key]` assign an export whose name is not - // knowable. Report the wildcard rather than the variable name, and never - // collapse it into `default`, which would claim a real default export. - if (isUnshadowedGlobal(node.object, state, "exports")) - return key === null ? "*" : key || "*"; - if ( - t.isMemberExpression(node.object) && - isCommonJsModuleExports(node.object, state) - ) - return key === null ? "*" : key || "default"; - if (isCommonJsModuleExports(node, state)) return "default"; - return undefined; }; - -const isCommonJsModuleExports = ( - node: t.Node, - state: JavaScriptSemanticAnalysisState, -): boolean => - (t.isMemberExpression(node) || t.isOptionalMemberExpression(node)) && - isUnshadowedGlobal(node.object, state, "module") && - semanticStaticPropertyKey(node.property, node.computed) === "exports"; diff --git a/src/domain/javascript/javascriptSemanticPromises.ts b/src/domain/javascript/javascriptSemanticPromises.ts index 524d3d0ff3..062e593560 100644 --- a/src/domain/javascript/javascriptSemanticPromises.ts +++ b/src/domain/javascript/javascriptSemanticPromises.ts @@ -5,6 +5,7 @@ import type { JavaScriptSemanticPromiseOperation, } from "./javascriptSemanticIr.js"; import { semanticCallableIdForNode } from "./javascriptSemanticProjection.js"; +import { createSemanticCallableIdStack } from "./javascriptSemanticCallableStack.js"; import { semanticStaticPropertyName, unwrapJavaScriptExpression, @@ -106,20 +107,18 @@ const collectCandidates = ( const admittedCallables = new Set( callables.map(({ callableId }) => callableId), ); - const callableStack: string[] = []; + const callableStack = createSemanticCallableIdStack(admittedCallables); const ancestors: t.Node[] = []; traverseJavaScriptAst(program, { - enter: (node) => { - const callableId = semanticCallableIdForNode(node); - if (callableId !== null && admittedCallables.has(callableId)) - callableStack.push(callableId); + enter: (node, parent) => { + callableStack.enter(node, parent); const details = promiseCandidateDetails(node, state); if (details !== null) { output.push({ node, ...details, promiseId: semanticPromiseId(details.kind, node), - ownerCallableId: callableStack.at(-1) ?? null, + ownerCallableId: callableStack.callables.at(-1) ?? null, ancestors: [...ancestors], }); } @@ -127,9 +126,7 @@ const collectCandidates = ( }, exit: (node) => { ancestors.pop(); - const callableId = semanticCallableIdForNode(node); - if (callableId !== null && callableStack.at(-1) === callableId) - callableStack.pop(); + callableStack.exit(node); }, }); return output; diff --git a/src/domain/javascript/javascriptSemanticQuery.ts b/src/domain/javascript/javascriptSemanticQuery.ts index a101cd934c..7b883521aa 100644 --- a/src/domain/javascript/javascriptSemanticQuery.ts +++ b/src/domain/javascript/javascriptSemanticQuery.ts @@ -250,11 +250,24 @@ const relevantUnknownFrontiers = ( return graph.unknowns.filter( (unknown) => (unknown.node_id === null || nodeIds.has(unknown.node_id)) && - (allowed === null || - unknown.relation_kinds.some((relation) => allowed.has(relation))), + unknown.relation_kinds.some( + (relation) => + directionAllowsRelation(input.direction, relation) && + (allowed === null || allowed.has(relation)), + ), ); }; +const directionAllowsRelation = ( + direction: JavaScriptSemanticQueryInput["direction"], + relation: JavaScriptSemanticGraphRelation["relation"], +): boolean => + direction === "callers" + ? relation === "calls" + : direction === "ownership" + ? ownershipRelation(relation) + : true; + const relevantCandidateRelationCount = ( graph: JavaScriptSemanticGraph, nodeIds: ReadonlySet, diff --git a/src/domain/javascript/javascriptSemanticRequireOrigin.ts b/src/domain/javascript/javascriptSemanticRequireOrigin.ts index 63fde3a959..d903020207 100644 --- a/src/domain/javascript/javascriptSemanticRequireOrigin.ts +++ b/src/domain/javascript/javascriptSemanticRequireOrigin.ts @@ -1,27 +1,408 @@ import * as t from "@babel/types"; +import { traverseJavaScriptAst } from "./javascriptSemanticTraversal.js"; +import { BUILTIN_RESOURCE_METHODS } from "./javascriptBuiltinResourceMethods.js"; import type { JavaScriptModuleOrigin } from "./javascriptSemanticIr.js"; import { resolveSemanticBindingState, + isUnmodifiedCommonJsWrapperBinding, semanticResolutionBlocked, + semanticStableInitializer, type JavaScriptSemanticAnalysisState, } from "./javascriptSemanticState.js"; import { semanticStaticPropertyKey, unwrapJavaScriptExpression, } from "./javascriptAstValues.js"; +import { semanticPropertyPathKeyMatches } from "./javascriptSemanticPropertyPaths.js"; import { stringValue } from "./javascriptStaticAnalysisHelpers.js"; +import { semanticTopLevelEffectFollowsCapture } from "./javascriptSemanticMutationInitializers.js"; +import { createSemanticModuleInterop } from "./javascriptModuleInterop.js"; -interface JavaScriptRequireOrigin { +export interface JavaScriptRequireOrigin { readonly origin: JavaScriptModuleOrigin; readonly call: t.CallExpression; } -/** Recover an unshadowed literal require origin and its exact member path. */ -export const semanticRequireOrigin = ( - node: t.Node | null | undefined, +export type JavaScriptRequireLoaderFact = + | "commonjs-loader" + | "factory-loader" + | "create-require" + | "local" + | "unknown"; + +/** Classify a bare `require(...)` using the shared lexical binding owner. */ +export const semanticRequireLoaderAt = ( + node: t.CallExpression | t.OptionalCallExpression | t.NewExpression, state: JavaScriptSemanticAnalysisState, -): JavaScriptRequireOrigin | undefined => { + factoryBindingIds: ReadonlySet, +): JavaScriptRequireLoaderFact => { + if (!t.isIdentifier(node.callee, { name: "require" })) return "local"; + const binding = resolveSemanticBindingState(state, node.callee, "require"); + if (binding === undefined) + return semanticResolutionBlocked(state, node.callee, "require") + ? "unknown" + : "commonjs-loader"; + if (factoryBindingIds.has(binding.bindingId)) + return binding.initializers.length === 0 ? "factory-loader" : "unknown"; + if (isWrapperRequireInitializerCall(binding, node, state)) + return "commonjs-loader"; + if (isUnmodifiedCommonJsWrapperBinding(node.callee, "require", state)) + return "commonjs-loader"; + if (isSemanticCreateRequireBinding(binding, state)) return "create-require"; + const initializer = semanticStableInitializer(binding, state)?.node; + if ( + (binding.kind === "function" || binding.kind === "class") && + initializer !== undefined + ) + return "local"; + if ( + binding.declarationKinds.size === 1 && + binding.declarationKinds.has("const") && + (t.isFunctionExpression(initializer) || + t.isArrowFunctionExpression(initializer)) + ) + return "local"; + return "unknown"; +}; + +/** Classify a call through an exact recovered factory-loader binding. */ +export const semanticFactoryRequireLoaderAt = ( + node: t.CallExpression | t.OptionalCallExpression | t.NewExpression, + state: JavaScriptSemanticAnalysisState, + bindingIds: ReadonlySet, +): JavaScriptRequireLoaderFact | undefined => { + const callee = unwrapJavaScriptExpression(node.callee).node; + if (t.isIdentifier(callee)) + return semanticFactoryLoaderMemberAt(node, state, bindingIds, null); + if ( + (t.isMemberExpression(callee) || t.isOptionalMemberExpression(callee)) && + semanticStaticPropertyKey(callee.property, callee.computed) === "bind" + ) + return semanticFactoryLoaderMemberAt(node, state, bindingIds, "bind"); + return undefined; +}; + +/** Classify an exact loader call by binding identity and expected member. */ +export const semanticFactoryLoaderMemberAt = ( + node: t.CallExpression | t.OptionalCallExpression | t.NewExpression, + state: JavaScriptSemanticAnalysisState, + bindingIds: ReadonlySet, + member: string | null, +): JavaScriptRequireLoaderFact | undefined => { + const callee = unwrapJavaScriptExpression(node.callee).node; + let loader: t.Identifier | undefined; + if (member === null) { + if (t.isIdentifier(callee)) loader = callee; + } else if ( + (t.isMemberExpression(callee) || t.isOptionalMemberExpression(callee)) && + semanticStaticPropertyKey(callee.property, callee.computed) === member + ) { + const root = unwrapJavaScriptExpression(callee.object).node; + if (t.isIdentifier(root)) loader = root; + } + if (loader === undefined) return undefined; + const binding = resolveSemanticBindingState(state, loader, loader.name); + if (binding === undefined || !bindingIds.has(binding.bindingId)) + return undefined; + return binding.initializers.length === 0 ? "factory-loader" : "unknown"; +}; + +const isSemanticCreateRequireBinding = ( + binding: NonNullable>, + state: JavaScriptSemanticAnalysisState, +): boolean => { + if (binding.initializers.length !== 1) return false; + const initializer = semanticStableInitializer(binding, state)?.node; + if (!t.isCallExpression(initializer)) return false; + const callee = unwrapJavaScriptExpression(initializer.callee).node; + if (t.isIdentifier(callee)) { + const origins = resolveSemanticBindingState( + state, + callee, + callee.name, + )?.directOrigins; + if (origins?.length !== 1) return false; + const [origin] = origins; + return ( + origin?.importedPath.at(-1) === "createRequire" && + ["module", "node:module"].includes(origin.specifier) + ); + } + const chain = semanticMemberChain(callee); + if (chain === undefined || chain.members.at(-1) !== "createRequire") + return false; + if (!t.isNode(chain.root)) return false; + const root = unwrapJavaScriptExpression(chain.root).node; + if (!t.isIdentifier(root)) return false; + const origins = resolveSemanticBindingState( + state, + root, + root.name, + )?.directOrigins; + if (origins?.length !== 1) return false; + const [origin] = origins; + return ( + origin?.importedPath.length === 0 && + ["module", "node:module"].includes(origin.specifier) + ); +}; + +/** The RHS call runs before its assignment replaces Node's wrapper loader. */ +const isWrapperRequireInitializerCall = ( + binding: NonNullable>, + node: t.CallExpression | t.OptionalCallExpression | t.NewExpression, + state: JavaScriptSemanticAnalysisState, +): boolean => { + const scope = state.scopesById.get(binding.scopeId); + if ( + scope?.kind !== "program" || + binding.kind !== "variable" || + binding.declarationKinds.size !== 1 || + !binding.declarationKinds.has("var") || + binding.initializers.length !== 1 || + binding.initializers[0]?.node !== node + ) + return false; + const assignment = state.parentsByNode.get(node); + if ( + !t.isAssignmentExpression(assignment) || + assignment.right !== node || + state.conditionallySkippedExpressions.has(node) + ) + return false; + const statement = state.parentsByNode.get(assignment); + if (!t.isExpressionStatement(statement)) return false; + if (state.parentsByNode.get(statement) !== state.program) return false; + const target = unwrapJavaScriptExpression(assignment.left).node; + return ( + t.isIdentifier(target, { name: "require" }) && + resolveSemanticBindingState(state, target, target.name) === binding + ); +}; + +// Ambient indexing finishes before these derived facts are consumed. Keep only +// replacement sites, rather than rescanning unrelated global member writes. +const globalProcessReplacements = new WeakMap< + JavaScriptSemanticAnalysisState, + readonly (t.Node | undefined)[] +>(); +const moduleRequireWrites = new WeakMap< + JavaScriptSemanticAnalysisState, + readonly t.Node[] +>(); + +const writesModuleRequire = ( + state: JavaScriptSemanticAnalysisState, +): readonly t.Node[] => { + const cached = moduleRequireWrites.get(state); + if (cached !== undefined) return cached; + // Interop invocation resolution may ask for require provenance while this + // index is being built. Publish a temporary empty result to break that cycle. + moduleRequireWrites.set(state, []); + const writes: t.Node[] = []; + const interop = createSemanticModuleInterop(state); + const isModuleObject = ( + node: t.Node, + visited = new Set(), + ): boolean => { + const value = unwrapJavaScriptExpression(node).node; + if (!t.isIdentifier(value)) return false; + if (value.name === "module") + return resolveSemanticBindingState(state, value, "module") === undefined + ? !semanticResolutionBlocked(state, value, "module") + : isUnmodifiedCommonJsWrapperBinding(value, "module", state); + const binding = resolveSemanticBindingState(state, value, value.name); + if (binding === undefined || visited.has(binding.bindingId)) return false; + const next = new Set(visited).add(binding.bindingId); + return ( + binding.initializers.length > 0 && + binding.initializers.every(({ node: init }) => isModuleObject(init, next)) + ); + }; + traverseJavaScriptAst(state.program, { + enter: (node) => { + const target = t.isAssignmentExpression(node) + ? node.left + : t.isUpdateExpression(node) || + t.isUnaryExpression(node, { operator: "delete" }) + ? node.argument + : undefined; + if (target !== undefined) { + const members: string[] = []; + let root: t.Node = target; + while ( + t.isMemberExpression(root) || + t.isOptionalMemberExpression(root) + ) { + const key = semanticStaticPropertyKey(root.property, root.computed); + if (key === null) break; + members.unshift(key); + root = root.object; + } + if ( + members.length === 1 && + members[0] === "require" && + isModuleObject(root) + ) + writes.push(node); + else if ( + members.length === 0 && + (t.isMemberExpression(target) || + t.isOptionalMemberExpression(target)) && + target.computed && + semanticStaticPropertyKey(target.property, true) === null && + isModuleObject(target.object) + ) + // An unknown computed key may be "require"; retain uncertainty rather + // than treating this direct delegate write as a proven unrelated slot. + writes.push(node); + } + if (t.isCallExpression(node) || t.isOptionalCallExpression(node)) { + const invocation = interop.intrinsicInvocation(node); + const owner = + invocation?.path?.length === 3 ? invocation.path[1] : undefined; + const method = invocation?.path?.at(-1); + const args = invocation?.arguments; + if ( + (owner === "Object" || owner === "Reflect") && + method === "defineProperty" && + isModuleObject(args?.[0] ?? t.nullLiteral()) + ) { + const key = interop.staticString(args?.[1]); + if (key === "require" || key === undefined) writes.push(node); + } else if ( + owner === "Object" && + method === "defineProperties" && + isModuleObject(args?.[0] ?? t.nullLiteral()) + ) { + const descriptors = args?.[1]; + if ( + !t.isObjectExpression(descriptors) || + descriptors.properties.some( + (property) => + t.isSpreadElement(property) || + (t.isObjectProperty(property) && + semanticStaticPropertyKey(property.key, property.computed) === + "require"), + ) + ) + writes.push(node); + } else if ( + owner === "Object" && + method === "assign" && + args?.some((arg) => isModuleObject(arg)) + ) { + const target = args?.[0]; + if (target !== undefined && isModuleObject(target)) { + const sourceHasRequire = args + ?.slice(1) + .some( + (arg) => + t.isObjectExpression(arg) && + arg.properties.some( + (property) => + t.isObjectProperty(property) && + semanticStaticPropertyKey( + property.key, + property.computed, + ) === "require", + ), + ); + if ( + sourceHasRequire || + args?.slice(1).some((arg) => !t.isObjectExpression(arg)) + ) + writes.push(node); + } + } + } + }, + }); + moduleRequireWrites.set(state, writes); + return writes; +}; + +/** Whether the original global process object can still own effects at this source site. */ +export const semanticOriginalGlobalProcessAt = ( + node: t.Node, + state: JavaScriptSemanticAnalysisState, +): boolean => { + let replacements = globalProcessReplacements.get(state); + if (replacements === undefined) { + replacements = [ + ...state.ambientWriteEffects + .filter( + ({ path }) => + path.length <= 2 && + path.every((key, index) => + semanticPropertyPathKeyMatches( + key, + ["global", "process"][index]!, + ), + ), + ) + .map(({ node: write }) => write), + ...(state.ambientBindingWrites.get("process") ?? []), + ]; + globalProcessReplacements.set(state, replacements); + } + return !replacements.some( + (write) => + !semanticTopLevelEffectFollowsCapture(node, write, state.parentsByNode), + ); +}; + +/** Node's unbound process reference shares the cached process exports object. */ +export const semanticGlobalProcessOrigin = ( + node: t.Node, + state: JavaScriptSemanticAnalysisState, +): JavaScriptModuleOrigin | undefined => + t.isIdentifier(node, { name: "process" }) && + resolveSemanticBindingState(state, node, "process") === undefined && + !semanticResolutionBlocked(state, node, "process") && + semanticOriginalGlobalProcessAt(node, state) + ? { specifier: "node:process", importedPath: [], resolutionMode: "require" } + : undefined; + +/** Resolve qualified global-object process references, preserving their selected path. */ +export const semanticGlobalProcessPathOrigin = ( + node: t.Node, + state: JavaScriptSemanticAnalysisState, +): JavaScriptModuleOrigin | undefined => { + const members: string[] = []; + let current = unwrapJavaScriptExpression(node).node; + while ( + t.isMemberExpression(current) || + t.isOptionalMemberExpression(current) + ) { + const key = semanticStaticPropertyKey(current.property, current.computed); + if (key === null) return undefined; + members.unshift(key); + current = unwrapJavaScriptExpression(current.object).node; + } + if ( + !t.isIdentifier(current) || + !["globalThis", "global", "window", "self"].includes(current.name) || + resolveSemanticBindingState(state, current, current.name) !== undefined || + semanticResolutionBlocked(state, current, current.name) || + members[0] !== "process" || + !semanticOriginalGlobalProcessAt(node, state) + ) + return undefined; + return { + specifier: "node:process", + importedPath: members.slice(1), + resolutionMode: "require", + }; +}; + +/** Strip a static member chain, returning its root expression and member path. */ +export const semanticMemberChain = ( + node: t.Node | null | undefined, +): + | { readonly root: t.Node | null | undefined; readonly members: string[] } + | undefined => { const members: string[] = []; let current = node; while (current !== null && current !== undefined) { @@ -39,15 +420,52 @@ export const semanticRequireOrigin = ( members.push(member); current = current.object; } + return { root: current, members: members.reverse() }; +}; + +/** Recover an unshadowed literal require origin and its exact member path. */ +export const semanticRequireOrigin = ( + node: t.Node | null | undefined, + state: JavaScriptSemanticAnalysisState, +): JavaScriptRequireOrigin | undefined => { + const chain = semanticMemberChain(node); + const current = chain?.root; if ( + chain === undefined || !t.isCallExpression(current) || !t.isIdentifier(current.callee, { name: "require" }) ) return undefined; if ( - resolveSemanticBindingState(state, current.callee, "require") !== - undefined || - semanticResolutionBlocked(state, current.callee, "require") + writesModuleRequire(state).some( + (write) => + !semanticTopLevelEffectFollowsCapture( + current, + write, + state.parentsByNode, + ), + ) + ) + return undefined; + const requireBinding = resolveSemanticBindingState( + state, + current.callee, + "require", + ); + if ( + (requireBinding !== undefined && + !isUnmodifiedCommonJsWrapperBinding(current.callee, "require", state) && + !isWrapperRequireInitializerCall(requireBinding, current, state)) || + (requireBinding === undefined && + semanticResolutionBlocked(state, current.callee, "require")) || + (state.ambientBindingWrites.get("require") ?? []).some( + (write) => + !semanticTopLevelEffectFollowsCapture( + current, + write, + state.parentsByNode, + ), + ) ) return undefined; const specifier = stringValue(current.arguments[0]); @@ -56,9 +474,50 @@ export const semanticRequireOrigin = ( : { origin: { specifier, - importedPath: members.reverse(), + importedPath: chain.members, resolutionMode: "require", }, call: current, }; }; + +/** Node's fs.promises export and fs/promises require share one object. */ +export const semanticRequireExportsIdentity = ( + specifier: string, + path: readonly Key[], +): { readonly specifier: string; readonly path: readonly (Key | string)[] } => { + const name = specifier.replace(/^node:/u, ""); + if (name === "fs/promises") return { specifier: name, path }; + return name === "fs" && path[0] === "promises" && path.length > 1 + ? { specifier: "fs/promises", path: path.slice(1) } + : { specifier: name, path }; +}; + +const MUTABLE_BUILTIN_DEFAULTS = new Set([ + ...Object.keys(BUILTIN_RESOURCE_METHODS), + "http", + "https", + "child_process", + "timers", + "process", +]); + +/** Mutable default/object selections share CommonJS identity; namespace slots do not. */ +export const semanticMutableBuiltinOrigin = ( + origin: JavaScriptModuleOrigin, + selected: readonly Key[] = [], +): + | { readonly specifier: string; readonly path: readonly (Key | string)[] } + | undefined => { + const path = [...origin.importedPath, ...selected]; + if (origin.resolutionMode === "require") + return { specifier: origin.specifier, path }; + const specifier = origin.specifier.replace(/^node:/u, ""); + if (!MUTABLE_BUILTIN_DEFAULTS.has(specifier)) return undefined; + if (path[0] === "default") return { specifier, path: path.slice(1) }; + if (specifier === "process" && (path[0] === "env" || path[0] === "argv")) + return { specifier, path }; + return specifier === "fs" && path[0] === "promises" + ? { specifier, path } + : undefined; +}; diff --git a/src/domain/javascript/javascriptSemanticResources.ts b/src/domain/javascript/javascriptSemanticResources.ts index 0a080fc90d..7ca9420076 100644 --- a/src/domain/javascript/javascriptSemanticResources.ts +++ b/src/domain/javascript/javascriptSemanticResources.ts @@ -1,18 +1,25 @@ import * as t from "@babel/types"; -import { semanticBuiltinNamespacePath } from "./javascriptSemanticProvenance.js"; - +import { + BUILTIN_RESOURCE_SPECIFIERS, + BUILTIN_RESOURCE_METHODS, +} from "./javascriptBuiltinResourceMethods.js"; import type { JavaScriptSemanticCallable, JavaScriptSemanticResourceOperation, } from "./javascriptSemanticIr.js"; -import { semanticStaticPropertyName } from "./javascriptAstValues.js"; +import { + semanticStaticPropertyName, + unwrapJavaScriptExpression, +} from "./javascriptAstValues.js"; +import { isUnmodifiedGlobalReference } from "./javascriptSemanticAmbientWrites.js"; import { resolveSemanticBindingState, type JavaScriptSemanticAnalysisState, type JavaScriptSemanticBindingState, } from "./javascriptSemanticState.js"; import { + builtinDataEffectMethod, dataEffectMemberCallee, outerDataEffectBinding, traverseDataEffects, @@ -20,14 +27,6 @@ import { } from "./javascriptSemanticDataEffectHelpers.js"; import { range } from "./javascriptStaticAnalysisHelpers.js"; -const ACQUIRE_METHODS = [ - "open", - "openSync", - "createReadStream", - "createWriteStream", - "connect", - "createConnection", -] as const; const RELEASE_METHODS = ["close", "destroy", "end"] as const; interface ResourceCandidate { @@ -105,21 +104,58 @@ const collectReleases = ( ); const linked = binding === undefined ? [] : (byBinding.get(binding.bindingId) ?? []); - if (linked.length === 0) return; + if (binding === undefined || linked.length === 0) return; output.push({ resourceId: `resource:release:${String(node.start ?? -1)}:${String(node.end ?? -1)}`, kind: "release", method, location: range(node), ownerCallableId: context.callableStack.at(-1) ?? null, - resultBindingId: binding?.bindingId ?? null, + resultBindingId: binding.bindingId, linkedResourceIds: linked.map(resourceCandidateId).sort(), - resolution: linked.length === 1 ? "complete" : "partial", + resolution: + linked.length === 1 && + holdsOnlyAcquisitions(binding, linked, context.state) + ? "complete" + : "partial", }); }); return output; }; +/** + * A lexical handle identifies its acquisition only while every value written + * to the binding is that acquisition itself. Any replacement, even a + * conditional or later one, leaves the released resource a candidate. A + * nullish placeholder, as in `let h = null; try { h = open() } finally { + * h?.close() }`, releases nothing else: the call is skipped or throws. + */ +const holdsOnlyAcquisitions = ( + binding: JavaScriptSemanticBindingState, + linked: readonly ResourceCandidate[], + state: JavaScriptSemanticAnalysisState, +): boolean => + [...binding.initializers, ...binding.referenceInitializers].every( + ({ node, projection }) => { + if (projection.length > 0) return false; + let value = unwrapJavaScriptExpression(node).node; + if (t.isAwaitExpression(value)) + value = unwrapJavaScriptExpression(value.argument).node; + return ( + isNullishPlaceholder(value, state) || + linked.some((acquisition) => acquisition.node === value) + ); + }, + ); + +const isNullishPlaceholder = ( + value: t.Node, + state: JavaScriptSemanticAnalysisState, +): boolean => + t.isNullLiteral(value) || + t.isUnaryExpression(value, { operator: "void" }) || + isUnmodifiedGlobalReference(value, state, "undefined"); + const immutableAcquisition = ( candidate: ResourceCandidate, ): JavaScriptSemanticResourceOperation => ({ @@ -137,58 +173,19 @@ const acquisitionMethod = ( callee: t.Node, state: JavaScriptSemanticAnalysisState, ): JavaScriptSemanticResourceOperation["method"] | null => { - if (t.isIdentifier(callee)) { - const binding = resolveSemanticBindingState(state, callee, callee.name); - return methodFromBinding(binding); - } - if (!t.isMemberExpression(callee) && !t.isOptionalMemberExpression(callee)) - return null; - const method = ACQUIRE_METHODS.find( - (candidate) => - candidate === - semanticStaticPropertyName(callee.property, callee.computed), - ); - if (method === undefined || !t.isIdentifier(callee.object)) return null; - const binding = resolveSemanticBindingState( - state, - callee.object, - callee.object.name, - ); - return resourceNamespace(binding) ? method : null; -}; - -const methodFromBinding = ( - binding: JavaScriptSemanticBindingState | undefined, -): JavaScriptSemanticResourceOperation["method"] | null => { - for (const { specifier, importedPath } of binding?.directOrigins ?? []) { - if (!resourceModule(specifier)) continue; - const method = ACQUIRE_METHODS.find( - (candidate) => candidate === importedPath.at(-1), - ); - if (method !== undefined) return method; + for (const specifier of BUILTIN_RESOURCE_SPECIFIERS) { + const method = builtinDataEffectMethod(callee, state, [specifier]); + if ( + method !== null && + ( + BUILTIN_RESOURCE_METHODS[ + specifier as keyof typeof BUILTIN_RESOURCE_METHODS + ] as readonly string[] + ).includes(method) + ) + return method as JavaScriptSemanticResourceOperation["method"]; } return null; }; - -const resourceNamespace = ( - binding: JavaScriptSemanticBindingState | undefined, -): boolean => - binding?.directOrigins.some( - ({ specifier, importedPath }) => - resourceModule(specifier) && semanticBuiltinNamespacePath(importedPath), - ) ?? false; - -const resourceModule = (specifier: string): boolean => - [ - "fs", - "fs/promises", - "net", - "node:fs", - "node:fs/promises", - "node:net", - "node:tls", - "tls", - ].includes(specifier); - const resourceCandidateId = (candidate: ResourceCandidate): string => `resource:acquire:${String(candidate.node.start ?? -1)}:${String(candidate.node.end ?? -1)}`; diff --git a/src/domain/javascript/javascriptSemanticReturnedReferences.ts b/src/domain/javascript/javascriptSemanticReturnedReferences.ts index fbf1dbb56c..bdaf2af589 100644 --- a/src/domain/javascript/javascriptSemanticReturnedReferences.ts +++ b/src/domain/javascript/javascriptSemanticReturnedReferences.ts @@ -74,10 +74,35 @@ export const createSemanticReturnedReferences = ( const pending: Candidate[] = [ { node: callee, path: [], bindings: new Set() }, ]; + // A shared initializer can be reached through both arms of every alias + // diamond. Expand an expression/path again only when its cycle context + // permits a binding that an earlier traversal blocked. + const expanded = new WeakMap< + t.Node, + Map[]> + >(); for (let cursor = 0; cursor < pending.length; cursor++) { const item = pending[cursor]; if (item === undefined) break; const node = unwrapJavaScriptExpression(item.node).node; + const pathKey = JSON.stringify(item.path); + const paths = + expanded.get(node) ?? new Map[]>(); + const contexts = paths.get(pathKey) ?? []; + if ( + contexts.some((context) => + [...context].every((binding) => item.bindings.has(binding)), + ) + ) + continue; + paths.set(pathKey, [ + ...contexts.filter( + (context) => + ![...item.bindings].every((binding) => context.has(binding)), + ), + item.bindings, + ]); + expanded.set(node, paths); if (t.isIdentifier(node)) { const binding = resolveSemanticBindingState(state, node, node.name); if (binding === undefined || item.bindings.has(binding.bindingId)) diff --git a/src/domain/javascript/javascriptSemanticReturns.ts b/src/domain/javascript/javascriptSemanticReturns.ts index 27bb030d01..50adf38ef1 100644 --- a/src/domain/javascript/javascriptSemanticReturns.ts +++ b/src/domain/javascript/javascriptSemanticReturns.ts @@ -28,6 +28,7 @@ import { evaluateSemanticExpression } from "./javascriptSemanticValues.js"; import { range } from "./javascriptStaticAnalysisHelpers.js"; import { semanticReturnCoverage } from "./javascriptSemanticCoverage.js"; import { semanticCallableResultExpressions } from "./javascriptSemanticCallableResults.js"; +import { semanticReadInitializers } from "./javascriptSemanticMutationInitializers.js"; interface ReturnExpression { readonly node: t.Node | null; @@ -104,15 +105,18 @@ export const resolveSemanticModuleCallables = ( if (link.kind === "type-import") return { ...link, callableId: null }; // Named exports follow their lexical binding, including every assignment. // A declaration's ID or a callable's display name cannot prove that link. - if (link.localName === null) + const captured = state.defaultExportExpressions.get(link); + if (link.localName === null && captured === undefined) return link.callableId !== null && callableIds.has(link.callableId) ? link : { ...link, callableId: null }; const binding = state.moduleLinkBindings.get(link); const resolved = - binding === undefined - ? [] - : callableIdsForBinding(binding, state, callableIds, new Set()); + captured !== undefined + ? callableIdsForNode(captured, state, callableIds, new Set(), true) + : binding === undefined + ? [] + : callableIdsForBinding(binding, state, callableIds, new Set()); const provenance = binding === undefined || link.specifier !== null ? undefined @@ -139,23 +143,31 @@ const callableIdsForBinding = ( state: JavaScriptSemanticAnalysisState, admitted: ReadonlySet, seen: ReadonlySet, + read?: t.Node, ): string[] => { // A single initializer does not prove an assignment executes or replace // the unknown incoming value of a parameter/catch binding. if ( seen.has(binding.bindingId) || - binding.initializers.length !== 1 || + (read === undefined && binding.initializers.length !== 1) || binding.definitions.some( ({ kind }) => - kind === "assignment" || kind === "parameter" || kind === "catch", + (read === undefined && kind === "assignment") || + kind === "parameter" || + kind === "catch", ) ) return []; - const initializer = binding.initializers[0]; + const selected = + read === undefined + ? binding + : semanticReadInitializers(binding, read, state.parentsByNode); + const initializer = selected.initializers[0]; if ( initializer === undefined || + selected.initializers.length !== 1 || initializer.projection.length > 0 || - state.conditionalInitializers.has(initializer.node) + state.conditionallySkippedExpressions.has(initializer.node) ) return []; return callableIdsForNode( @@ -163,6 +175,7 @@ const callableIdsForBinding = ( state, admitted, new Set([...seen, binding.bindingId]), + read !== undefined, ); }; @@ -171,17 +184,24 @@ const callableIdsForNode = ( state: JavaScriptSemanticAnalysisState, admitted: ReadonlySet, seen: ReadonlySet, + snapshot = false, ): string[] => { const unwrapped = unwrapJavaScriptExpression(node).node; if (unwrapped !== node) - return callableIdsForNode(unwrapped, state, admitted, seen); + return callableIdsForNode(unwrapped, state, admitted, seen, snapshot); const direct = semanticCallableIdForNode(node); if (direct !== null && admitted.has(direct)) return [direct]; if (t.isIdentifier(node)) { const binding = resolveSemanticBindingState(state, node, node.name); return binding === undefined ? [] - : callableIdsForBinding(binding, state, admitted, seen); + : callableIdsForBinding( + binding, + state, + admitted, + seen, + snapshot ? node : undefined, + ); } return []; }; diff --git a/src/domain/javascript/javascriptSemanticState.ts b/src/domain/javascript/javascriptSemanticState.ts index d01a4be522..96fa37d679 100644 --- a/src/domain/javascript/javascriptSemanticState.ts +++ b/src/domain/javascript/javascriptSemanticState.ts @@ -32,17 +32,25 @@ interface JavaScriptSemanticReferenceInitializer extends JavaScriptSemanticIniti } /** Mutable binding state used only while constructing the immutable IR. */ +export interface JavaScriptSemanticMemberEffect { + readonly path: JavaScriptSemanticPropertyPath; + readonly node: t.Node | undefined; +} + +export interface JavaScriptSemanticRequiredModuleEffect extends JavaScriptSemanticMemberEffect { + readonly kind: "write" | "escape"; +} + export interface JavaScriptSemanticBindingState { readonly bindingId: string; readonly scopeId: string; readonly name: string; kind: JavaScriptSemanticDefinition["kind"]; + /** Source declaration form retained internally for CommonJS wrapper bindings. */ + declarationKinds: Set<"var" | "let" | "const" | "using" | "await using">; mutable: boolean; - readonly mutatedPaths: JavaScriptSemanticPropertyPath[]; - readonly escapedPaths: { - readonly path: JavaScriptSemanticPropertyPath; - readonly node: t.Node | undefined; - }[]; + readonly memberWrites: JavaScriptSemanticMemberEffect[]; + readonly escapedPaths: JavaScriptSemanticMemberEffect[]; readonly definitions: JavaScriptSemanticDefinition[]; readonly initializers: JavaScriptSemanticInitializer[]; readonly referenceInitializers: JavaScriptSemanticReferenceInitializer[]; @@ -63,6 +71,7 @@ export interface JavaScriptSemanticScopeState { /** Shared state for semantic collection and evaluation. */ export interface JavaScriptSemanticAnalysisState { + readonly program: t.Program; /** Effective loader for static imports in this source artifact. */ readonly staticImportResolutionMode: JavaScriptModuleResolutionMode; readonly parentsByNode: WeakMap; @@ -71,6 +80,11 @@ export interface JavaScriptSemanticAnalysisState { /** Scope boundaries and descendant scopes resolved on demand. */ readonly scopeByNode: WeakMap; readonly bindingsById: Map; + /** Effects on a cached require exports object, independent of local projections. */ + readonly requiredModuleEffects: Map< + string, + JavaScriptSemanticRequiredModuleEffect[] + >; readonly callables: JavaScriptSemanticCallable[]; readonly callableNodesById: Map; readonly moduleLinks: JavaScriptSemanticModuleLink[]; @@ -80,9 +94,19 @@ export interface JavaScriptSemanticAnalysisState { JavaScriptSemanticModuleLink, JavaScriptSemanticBindingState >; - readonly conditionalInitializers: WeakSet; - /** Source-local writes, tagged by global-object or import.meta identity. */ + /** Default expressions capture their value at the export statement. */ + readonly defaultExportExpressions: WeakMap< + JavaScriptSemanticModuleLink, + t.Node + >; + /** Initializer expressions whose write can be skipped by enclosing control flow. */ + readonly conditionallySkippedExpressions: WeakSet; + /** Unbound identifier writes retain their source sites for capture ordering. */ + readonly ambientBindingWrites: Map; + /** Source-local writes, tagged by global, import.meta or CJS module identity. */ readonly ambientWrites: SemanticPropertyPathCoverage; + /** Source sites distinguish replacement before an ambient identity capture from later writes. */ + readonly ambientWriteEffects: JavaScriptSemanticMemberEffect[]; } /** @@ -104,7 +128,7 @@ export const semanticStableInitializer = ( (definition?.kind === "function" || definition?.kind === "class"))) && binding.initializers.length === 1 && initializer !== undefined && - !state.conditionalInitializers.has(initializer.node) + !state.conditionallySkippedExpressions.has(initializer.node) ? initializer : undefined; }; @@ -234,5 +258,30 @@ export const isUnshadowedGlobal = ( name: string, ): boolean => t.isIdentifier(node, { name }) && - resolveSemanticBindingState(state, node, name) === undefined && - !semanticResolutionBlocked(state, node, name); + (resolveSemanticBindingState(state, node, name) === undefined + ? !semanticResolutionBlocked(state, node, name) + : isUnmodifiedCommonJsWrapperBinding(node, name, state)); + +/** A program-level `var` redeclaration still denotes its untouched CJS wrapper parameter. */ +export const isUnmodifiedCommonJsWrapperBinding = ( + node: t.Node, + name: string, + state: JavaScriptSemanticAnalysisState, +): boolean => { + // Parser sourceType separates explicit modules from script/CommonJS source; + // specifier resolution mode does not establish that distinction. + if ( + state.program.sourceType !== "script" || + !["module", "exports", "require"].includes(name) || + !t.isIdentifier(node, { name }) + ) + return false; + const binding = resolveSemanticBindingState(state, node, name); + if (binding === undefined || binding.initializers.length !== 0) return false; + const scope = state.scopesById.get(binding.scopeId); + return ( + scope?.kind === "program" && + binding.declarationKinds.size > 0 && + [...binding.declarationKinds].every((kind) => kind === "var") + ); +}; diff --git a/src/domain/javascript/javascriptSemanticValues.memo.test.ts b/src/domain/javascript/javascriptSemanticValues.memo.test.ts index 8f32a723c3..5a9395cdbf 100644 --- a/src/domain/javascript/javascriptSemanticValues.memo.test.ts +++ b/src/domain/javascript/javascriptSemanticValues.memo.test.ts @@ -86,7 +86,7 @@ it("invalidates retained alias values at the producing mutation boundary", () => expect( property(property(evaluateSemanticBinding(root, state), "alias"), "value"), ).toEqual({ status: "literal", value: 1 }); - target.mutatedPaths.push(["value"]); + target.memberWrites.push({ path: ["value"], node: undefined }); clearSemanticValueCaches(state); expect( property(property(evaluateSemanticBinding(root, state), "alias"), "value") diff --git a/src/domain/javascript/javascriptSemanticValues.ts b/src/domain/javascript/javascriptSemanticValues.ts index d15c5edfc3..2c4addefd6 100644 --- a/src/domain/javascript/javascriptSemanticValues.ts +++ b/src/domain/javascript/javascriptSemanticValues.ts @@ -9,6 +9,7 @@ import { semanticCapturePosition, semanticEscapeFollowsCapture, semanticBindingPrecedesCapture, + semanticReadInitializers, } from "./javascriptSemanticMutationInitializers.js"; import { semanticSlotAtPath } from "./javascriptSemanticSlots.js"; @@ -228,6 +229,21 @@ export const evaluateSemanticProvenance = ( memo: createEvaluationMemo(state), }); +/** Follow a namespace binding as it was read at a specific source position. */ +export const evaluateSemanticProvenanceAtRead = ( + binding: JavaScriptSemanticBindingState, + read: t.Identifier, + state: JavaScriptSemanticAnalysisState, +): JavaScriptBindingProvenance => + provenanceForBindingAtRead(binding, read, { + state, + bindings: new Map(), + expressionDepth: 0, + primitiveBindingValues: primitiveBindingValuesFor(state), + dependence: { lowest: CALLER_INDEPENDENT }, + memo: createEvaluationMemo(state), + }); + const readMemoizedEvaluation = ( entry: MemoizedEvaluation | undefined, context: EvaluationContext, @@ -344,6 +360,11 @@ const evaluateBinding = ( const initializer = binding.initializers[0]; if (initializer === undefined) return { status: "unknown", reason: "Missing binding initializer." }; + if (context.state.conditionallySkippedExpressions.has(initializer.node)) + return { + status: "unknown", + reason: `Binding ${binding.name} may remain uninitialized.`, + }; const position = context.bindings.size; const callerDependence = context.dependence.lowest; context.dependence.lowest = CALLER_INDEPENDENT; @@ -383,7 +404,10 @@ const evaluateBinding = ( ? callerDependence : Math.min(callerDependence, context.dependence.lowest); } - let projected = invalidateSemanticMutationPaths(value, binding.mutatedPaths); + let projected = invalidateSemanticMutationPaths( + value, + binding.memberWrites.map(({ path }) => path), + ); for (const escape of binding.escapedPaths) { // Escapes only invalidate object references; other lattice values survive. if (projected.status !== "object" && projected.status !== "array") break; @@ -798,7 +822,7 @@ const provenanceForBindingUncached = ( ({ kind }) => kind === "assignment", ); const conditionallyInitialized = binding.initializers.some(({ node }) => - context.state.conditionalInitializers.has(node), + context.state.conditionallySkippedExpressions.has(node), ); if ( provenance.status !== "module" || @@ -809,7 +833,7 @@ const provenanceForBindingUncached = ( provenance.origins, assigned ? `Binding ${binding.name} may be reassigned.` - : `Binding ${binding.name} is conditionally initialized.`, + : `Binding ${binding.name} may remain uninitialized.`, ); }; @@ -912,7 +936,7 @@ const provenanceForExpression = ( "unknown", `Unbound identifier ${node.name}.`, ) - : provenanceForBinding(binding, nestedContext(context)); + : provenanceForBindingAtRead(binding, node, nestedContext(context)); } if (t.isMemberExpression(node) || t.isOptionalMemberExpression(node)) { const members: string[] = []; @@ -983,6 +1007,40 @@ const provenanceForExpression = ( ); }; +/** Resolve an alias from the value visible at this read, before later rebinds. */ +const provenanceForBindingAtRead = ( + binding: JavaScriptSemanticBindingState, + read: t.Identifier, + context: EvaluationContext, +): JavaScriptBindingProvenance => { + const selected = semanticReadInitializers( + binding, + read, + context.state.parentsByNode, + ); + if (selected.initializers === binding.initializers) + return provenanceForBinding(binding, context); + if (context.bindings.has(binding.bindingId)) + return semanticUnresolvedProvenance( + "cycle", + `Alias cycle at ${binding.name}.`, + ); + const snapshot: JavaScriptSemanticBindingState = { + ...binding, + ...selected, + // Assignment facts after the read cannot change the captured value. + definitions: binding.definitions.filter( + ({ kind }) => kind !== "assignment", + ), + }; + context.bindings.set(binding.bindingId, context.bindings.size); + try { + return provenanceForBindingValue(snapshot, context); + } finally { + context.bindings.delete(binding.bindingId); + } +}; + const isLocallyCreatedValue = (node: t.Node): boolean => readExactJavaScriptLiteral(node).found || t.isObjectExpression(node) || diff --git a/src/domain/javascript/javascriptSourceMapPaths.ts b/src/domain/javascript/javascriptSourceMapPaths.ts index 06b5f82656..5966f89e6b 100644 --- a/src/domain/javascript/javascriptSourceMapPaths.ts +++ b/src/domain/javascript/javascriptSourceMapPaths.ts @@ -93,10 +93,7 @@ const resolveJavaScriptSourceMapPath = ( const windowsPath = /^[a-z]:\//iu.test(portable) || source.startsWith("\\\\"); // Preserve parent traversal rather than letting URL resolution clamp at '/'. const anchor = `/${"source/".repeat(portable.split("/").length + 1)}`; - const directory = posix.join( - anchor, - posix.dirname(mapPath.replaceAll("\\", "/")), - ); + const directory = posix.join(anchor, posix.dirname(mapPath)); const base = pathToFileURL(`${directory}/`, { windows: false }); const absolute = windowsPath ? windowsFileUrl(source) : URL.parse(source); const resolved = windowsPath ? absolute : URL.parse(source, base); @@ -116,8 +113,11 @@ const resolveJavaScriptSourceMapPath = ( kind: "unresolved", reason: "Source reference names a directory rather than a file.", }; - // Encoded separators must not manufacture a different filesystem hierarchy. - if (/%(?:2f|5c)/iu.test(resolved.pathname)) + // Reject separators in declarations, not escaped literal names in the owned base. + const declaredPath = windowsPath + ? (absolute?.pathname ?? "") + : (URL.parse(source, "file:///")?.pathname ?? ""); + if (/%(?:2f|5c)/iu.test(declaredPath)) return { kind: "unresolved", reason: "Source reference contains an encoded path separator.", @@ -170,10 +170,11 @@ const windowsFileUrl = (path: string): URL | null => { const decodedUrlPath = (url: URL): string | null => { try { - if (url.protocol === "file:") - return fileURLToPath(url, { - windows: url.hostname !== "" && url.hostname !== "localhost", - }).replaceAll("\\", "/"); + if (url.protocol === "file:") { + const windows = url.hostname !== "" && url.hostname !== "localhost"; + const path = fileURLToPath(url, { windows }); + return windows ? path.replaceAll("\\", "/") : path; + } return `${url.host}${decodeURIComponent(url.pathname)}`; } catch (cause: unknown) { if (cause instanceof URIError || cause instanceof TypeError) return null; diff --git a/src/domain/javascript/javascriptSourceParser.test.ts b/src/domain/javascript/javascriptSourceParser.test.ts new file mode 100644 index 0000000000..e5f829a356 --- /dev/null +++ b/src/domain/javascript/javascriptSourceParser.test.ts @@ -0,0 +1,42 @@ +import { expect, it } from "vitest"; + +import { parseJavaScriptSource } from "./javascriptSourceParser.js"; +import { analyzeParsedJavaScriptStaticSource } from "./javascriptStaticAnalysis.js"; + +const analyze = (source: string, path: string) => { + const parsed = parseJavaScriptSource(source, path); + return parsed === null + ? null + : { + errors: parsed.errors.length, + facts: analyzeParsedJavaScriptStaticSource(source, parsed), + }; +}; + +it("parses conditional arrow bodies in JavaScript sources with JavaScript grammar", () => { + // Bundlers emit this shape; TypeScript reads `):H=>` as a return type. + const source = `const {ipcMain}=require("electron"); +const pick=n===0?H=>({start:H.startY}):H=>({start:H.startX}); +ipcMain.handle("layout:ready",()=>pick);`; + for (const path of ["main.js", "main.cjs", "main.mjs", "main.jsx"]) { + const result = analyze(source, path); + expect(result?.errors).toBe(0); + expect(result?.facts.electron.ipc).toContainEqual( + expect.objectContaining({ channel: "layout:ready", operation: "handle" }), + ); + } +}); + +it("keeps JavaScript comparison meaning for angle brackets in JavaScript sources", () => { + const source = `const a=fetch("/api/compare");`; + expect(analyze(source, "main.js")?.facts.endpoints).toEqual([]); + expect(analyze(source, "main.ts")?.facts.endpoints).toContainEqual( + expect.objectContaining({ value: "/api/compare" }), + ); +}); + +it("keeps TypeScript syntax available for TypeScript and unnamed sources", () => { + const source = "let count: number = 1; export type Id = string;"; + expect(parseJavaScriptSource(source, "value.ts")?.errors).toEqual([]); + expect(parseJavaScriptSource(source)?.errors).toEqual([]); +}); diff --git a/src/domain/javascript/javascriptSourceParser.ts b/src/domain/javascript/javascriptSourceParser.ts index 630d5e9bf3..d1141a2434 100644 --- a/src/domain/javascript/javascriptSourceParser.ts +++ b/src/domain/javascript/javascriptSourceParser.ts @@ -9,17 +9,21 @@ export type ParsedJavaScriptSource = ReturnType; * * - `dts:true` only for `.d.ts/.d.mts/.d.cts` ambient declarations; * ordinary `.ts` keeps the missing-initializer diagnostic. - * - JSX enabled for `.tsx`, `.jsx`, `.js`, and unknown paths; disabled for - * plain `.ts/.cts/.mts` (angle-bracket assertions, not JSX) and for all + * - `.js/.jsx/.mjs/.cjs` use JavaScript grammar with JSX. TypeScript reads + * some valid JavaScript differently (`c?x=>(a):y=>(b)` as a return type, + * `a(c)` as a generic call), so it must not select their meaning. + * - JSX enabled for `.tsx` and unknown paths; disabled for plain + * `.ts/.cts/.mts` (angle-bracket assertions, not JSX) and for all * declaration files. `.mts` (including `.d.mts`) sets * `disallowAmbiguousJSXLike`, matching tsc's `.mts` mode. * - `decorators-legacy` always present: without it decorated sources fail * hard instead of reporting recovered syntax, losing every derived fact. - * - Unknown paths retain the existing JSX-capable contract (best-effort - * parse; failures surface via `errorRecovery`, never silent misparse). + * - Unknown paths retain the existing JSX- and TypeScript-capable contract + * (best-effort parse; failures surface via `errorRecovery`). */ export const parserPluginsForPath = (sourcePath?: string): ParserPlugin[] => { const path = sourcePath ?? ""; + if (/\.(?:js|jsx|mjs|cjs)$/iu.test(path)) return ["decorators-legacy", "jsx"]; const dts = /\.d\.(?:ts|mts|cts)$/iu.test(path); const isTsLike = /\.(?:ts|cts|mts)$/iu.test(path); const isTsx = /\.tsx$/iu.test(path); diff --git a/src/domain/javascript/javascriptStaticAnalysis.event_source.test.ts b/src/domain/javascript/javascriptStaticAnalysis.event_source.test.ts new file mode 100644 index 0000000000..0fa5d65432 --- /dev/null +++ b/src/domain/javascript/javascriptStaticAnalysis.event_source.test.ts @@ -0,0 +1,40 @@ +import { describe, expect, it } from "vitest"; + +import { analyzeJavaScriptStaticSource } from "./javascriptStaticAnalysis.js"; + +const endpoints = (source: string) => + analyzeJavaScriptStaticSource(source).endpoints.map( + ({ kind, value, mechanism }) => ({ kind, value, mechanism }), + ); + +describe("server-sent event endpoints", () => { + it("reports EventSource URLs like WebSocket URLs", () => { + expect( + endpoints(` + new WebSocket("wss://chat.example/ws"); + new EventSource("/api/chat/stream"); + new window.EventSource("https://chat.example/events", { withCredentials: true }); + `), + ).toEqual([ + { + kind: "network", + value: "wss://chat.example/ws", + mechanism: "call:WebSocket", + }, + { + kind: "network", + value: "/api/chat/stream", + mechanism: "call:EventSource", + }, + { + kind: "network", + value: "https://chat.example/events", + mechanism: "call:window.EventSource", + }, + ]); + }); + + it("keeps a dynamic EventSource URL unreported", () => { + expect(endpoints("new EventSource(url);")).toEqual([]); + }); +}); diff --git a/src/domain/javascript/javascriptStaticAnalysis.routeProperties.test.ts b/src/domain/javascript/javascriptStaticAnalysis.routeProperties.test.ts new file mode 100644 index 0000000000..cc8d60a623 --- /dev/null +++ b/src/domain/javascript/javascriptStaticAnalysis.routeProperties.test.ts @@ -0,0 +1,32 @@ +import { expect, it } from "vitest"; + +import { analyzeJavaScriptStaticSource } from "./javascriptStaticAnalysis.js"; + +it("reads exact route-property templates and leaves interpolated values unknown", () => { + const analysis = analyzeJavaScriptStaticSource(` + const routes = [ + { path: \`settings\` }, + { route: \`users/\${userId}\` }, + { path: "/login" }, + ]; + `); + + expect( + analysis.endpoints.map(({ value, mechanism, interpretation }) => ({ + value, + mechanism, + interpretation, + })), + ).toEqual([ + { + value: "settings", + mechanism: "property:path", + interpretation: undefined, + }, + { + value: "/login", + mechanism: "property:path", + interpretation: undefined, + }, + ]); +}); diff --git a/src/domain/javascript/javascriptStaticAnalysis.send_beacon.test.ts b/src/domain/javascript/javascriptStaticAnalysis.send_beacon.test.ts new file mode 100644 index 0000000000..fd26b88b55 --- /dev/null +++ b/src/domain/javascript/javascriptStaticAnalysis.send_beacon.test.ts @@ -0,0 +1,34 @@ +import { describe, expect, it } from "vitest"; + +import { analyzeJavaScriptStaticSource } from "./javascriptStaticAnalysis.js"; + +const endpoints = (source: string) => + analyzeJavaScriptStaticSource(source).endpoints.map( + ({ kind, value, mechanism }) => ({ kind, value, mechanism }), + ); + +describe("beacon endpoints", () => { + it("reports navigator.sendBeacon URLs", () => { + expect( + endpoints(` + navigator.sendBeacon("/api/telemetry", payload); + window.navigator.sendBeacon("https://metrics.example/collect"); + `), + ).toEqual([ + { + kind: "network", + value: "/api/telemetry", + mechanism: "call:navigator.sendBeacon", + }, + { + kind: "network", + value: "https://metrics.example/collect", + mechanism: "call:window.navigator.sendBeacon", + }, + ]); + }); + + it("keeps a dynamic beacon URL unreported", () => { + expect(endpoints("navigator.sendBeacon(url, payload);")).toEqual([]); + }); +}); diff --git a/src/domain/javascript/javascriptStaticAnalysis.ts b/src/domain/javascript/javascriptStaticAnalysis.ts index c344624980..0b87da8645 100644 --- a/src/domain/javascript/javascriptStaticAnalysis.ts +++ b/src/domain/javascript/javascriptStaticAnalysis.ts @@ -25,6 +25,7 @@ import { inspectRouteProperty, inspectRoleProperty, inspectRpcNode, + indexVitePreloadDependencyTable, } from "./javascriptStaticAnalysisCalls.js"; import { inspectBundlerRegistration, @@ -41,10 +42,7 @@ import type { JavaScriptStaticAnalysis } from "./javascriptStaticAnalysisTypes.j import type { JavaScriptSemanticAnalysisState } from "./javascriptSemanticState.js"; import { indexJavaScriptElectronIdentitySteps, - indexJavaScriptBindingsSteps, - classifyParsedJavaScriptRpcClientRootsSteps, - classifyParsedJavaScriptOpenReceivers, - type JavaScriptOpenReceiverFact, + indexJavaScriptBindingsWithReceiversSteps, } from "./javascriptSemanticAnalysis.js"; import { parseJavaScriptSource, @@ -66,44 +64,38 @@ export const analyzeJavaScriptStaticSource = ( const file = parseJavaScriptSource(source); return file === null ? failedJavaScriptStaticAnalysis() - : analyzeParsedJavaScriptStaticSource( - source, - file, - classifyParsedJavaScriptOpenReceivers(file), - ); + : analyzeParsedJavaScriptStaticSource(source, file); }; /** Recover static structure from an already parsed JavaScript artifact. */ export const analyzeParsedJavaScriptStaticSource = ( source: string, file: ParsedJavaScriptSource, - openReceiverFacts?: ReadonlyMap, ): JavaScriptStaticAnalysis => - completeSemanticSteps( - analyzeParsedJavaScriptStaticSourceSteps(source, file, openReceiverFacts), - ); + completeSemanticSteps(analyzeParsedJavaScriptStaticSourceSteps(source, file)); /** Static analysis that yields during its source traversal; results are unchanged. */ export function* analyzeParsedJavaScriptStaticSourceSteps( source: string, file: ParsedJavaScriptSource, - openReceiverFacts?: ReadonlyMap, ): Generator { const accumulator = createJavaScriptAnalysisAccumulator(source.length); - const bindings = yield* indexJavaScriptBindingsSteps(file.program); + // One definitions pass serves receiver classification and binding lookup. + const { bindings, receivers } = + yield* indexJavaScriptBindingsWithReceiversSteps(file.program); const electronIdentity = yield* indexJavaScriptElectronIdentitySteps( file.program, bindings, ); - const clientRoots = yield* classifyParsedJavaScriptRpcClientRootsSteps(file); yield* traverseStaticSourceSteps( file, { source, accumulator, - ...(openReceiverFacts === undefined ? {} : { openReceiverFacts }), + vitePreloadDependencyTable: indexVitePreloadDependencyTable(file.program), + openReceiverFacts: receivers.openReceivers, electronBindings: electronIdentity.bindings, - rpcScan: createJavaScriptRpcScan(source, clientRoots), + rpcScan: createJavaScriptRpcScan(source, receivers.rpcClientRoots), }, bindings, ); diff --git a/src/domain/javascript/javascriptStaticAnalysisBundler.ts b/src/domain/javascript/javascriptStaticAnalysisBundler.ts index 773f88d7d2..11d127401d 100644 --- a/src/domain/javascript/javascriptStaticAnalysisBundler.ts +++ b/src/domain/javascript/javascriptStaticAnalysisBundler.ts @@ -1,9 +1,11 @@ import * as t from "@babel/types"; +import { runtimeFunctionParameters } from "./javascriptAstValues.js"; import { javascriptBootstrapAt } from "./javascriptBootstrap.js"; import { resolveSemanticBindingState, type JavaScriptSemanticAnalysisState, } from "./javascriptSemanticState.js"; +import { semanticFactoryLoaderMemberAt } from "./javascriptSemanticRequireOrigin.js"; import { traverseJavaScriptAst } from "./javascriptSemanticTraversal.js"; @@ -31,6 +33,7 @@ import type { JavaScriptBundlerModule, JavaScriptBundlerRegistration, } from "./javascriptStaticAnalysisTypes.js"; +import { JAVASCRIPT_AST_FINGERPRINT_ALGORITHM } from "./javascriptStaticAnalysisTypes.js"; /** Inspect a bundler registration call and recover its module table. */ export const inspectBundlerRegistration = ( @@ -44,7 +47,11 @@ export const inspectBundlerRegistration = ( if (runtime === undefined || !t.isArrayExpression(entry)) return; const chunkIds = entry.elements[0]; const table = entry.elements[1]; - const runtimeValue = runtimeMetadata(entry.elements[2]); + const runtimeValue = runtimeMetadata( + entry.elements[2], + bindings, + accumulator, + ); if (!t.isArrayExpression(chunkIds) || !t.isObjectExpression(table)) return; const recovered = recoverBundlerModules(source, table, accumulator, { runtime, @@ -251,7 +258,21 @@ const recoverBundlerModule = ( const fingerprint = fingerprintJavaScriptAst(factory); const exportsValue = collectJavaScriptExports(factory, bindings); const requireName = factoryRequireName(factory); - const asyncChunks = collectBundlerAsyncChunkKeys(factory, requireName); + const requireParameter = runtimeFunctionParameters(factory)[2]; + const requireBindingId = t.isIdentifier(requireParameter) + ? (resolveSemanticBindingState( + bindings, + requireParameter, + requireParameter.name, + )?.bindingId ?? null) + : null; + if (requireBindingId !== null) + accumulator.factoryRequireBindingIds.add(requireBindingId); + const asyncChunks = collectBundlerAsyncChunkKeys( + factory, + requireBindingId, + bindings, + ); if (typeof factory.start === "number" && typeof factory.end === "number") { const moduleRange = { start: factory.start, @@ -268,7 +289,7 @@ const recoverBundlerModule = ( factory_require_name: requireName, source_sha256: sha256Text(sourceSlice(source, factory)), structural_fingerprint_sha256: fingerprint, - structural_fingerprint_algorithm: "babel-ast-v1", + structural_fingerprint_algorithm: JAVASCRIPT_AST_FINGERPRINT_ALGORITHM, exports: exportsValue.values, location: range(factory), }, @@ -295,7 +316,11 @@ type BundlerFunction = | t.ArrowFunctionExpression | t.ObjectMethod; -const runtimeMetadata = (node: t.Node | null | undefined): RuntimeMetadata => { +const runtimeMetadata = ( + node: t.Node | null | undefined, + bindings: JavaScriptSemanticAnalysisState, + accumulator: AnalysisAccumulator, +): RuntimeMetadata => { if (!isBundlerFunction(node)) return { requireName: null, @@ -304,8 +329,14 @@ const runtimeMetadata = (node: t.Node | null | undefined): RuntimeMetadata => { asyncChunkKeys: [], unknownAsyncChunkKeys: 0, }; - const parameter = node.params[0]; + const parameter = runtimeFunctionParameters(node)[0]; const requireName = t.isIdentifier(parameter) ? parameter.name : null; + const requireBindingId = t.isIdentifier(parameter) + ? (resolveSemanticBindingState(bindings, parameter, parameter.name) + ?.bindingId ?? null) + : null; + if (requireBindingId !== null) + accumulator.factoryRequireBindingIds.add(requireBindingId); if (requireName === null) return { requireName, @@ -314,8 +345,18 @@ const runtimeMetadata = (node: t.Node | null | undefined): RuntimeMetadata => { asyncChunkKeys: [], unknownAsyncChunkKeys: 0, }; - const entries = collectBundlerCallArgumentValues(node, requireName); - const asyncChunks = collectBundlerAsyncChunkKeys(node, requireName); + const entries = collectBundlerCallArgumentValues( + node, + requireBindingId, + bindings, + null, + ); + const asyncChunks = collectBundlerCallArgumentValues( + node, + requireBindingId, + bindings, + "e", + ); return { requireName, entryModuleKeys: entries.values, @@ -327,23 +368,43 @@ const runtimeMetadata = (node: t.Node | null | undefined): RuntimeMetadata => { const collectBundlerAsyncChunkKeys = ( factory: BundlerFunction, - requireName: string | null, + requireBindingId: string | null, + bindings: JavaScriptSemanticAnalysisState, ): StaticValues => - requireName === null + requireBindingId === null ? { values: [], unknown: 0 } - : collectBundlerCallArgumentValues(factory, `${requireName}.e`); + : collectBundlerCallArgumentValues( + factory, + requireBindingId, + bindings, + "e", + ); const collectBundlerCallArgumentValues = ( factory: BundlerFunction, - callee: string, + bindingId: string | null, + bindings: JavaScriptSemanticAnalysisState, + member: string | null, ): StaticValues => { + if (bindingId === null) return { values: [], unknown: 0 }; const values: string[] = []; let unknown = 0; + const bindingIds = new Set([bindingId]); traverseJavaScriptAst(factory, { enter: (node) => { if (!t.isCallExpression(node) && !t.isNewExpression(node)) return undefined; - if (calleeName(node.callee) !== callee) return undefined; + const loader = semanticFactoryLoaderMemberAt( + node, + bindings, + bindingIds, + member, + ); + if (loader === undefined) return undefined; + if (loader === "unknown") { + unknown += 1; + return undefined; + } const value = argumentValue(node.arguments[0]); if (value === undefined) unknown += 1; else values.push(value); diff --git a/src/domain/javascript/javascriptStaticAnalysisCalleeBoundary.test.ts b/src/domain/javascript/javascriptStaticAnalysisCalleeBoundary.test.ts new file mode 100644 index 0000000000..ae396b39c6 --- /dev/null +++ b/src/domain/javascript/javascriptStaticAnalysisCalleeBoundary.test.ts @@ -0,0 +1,209 @@ +import { describe, expect, it } from "vitest"; + +import { analyzeJavaScriptStaticSource } from "./javascriptStaticAnalysis.js"; + +it("retains endpoint observations for exact template and TypeScript member keys", () => { + for (const [get, beacon] of [ + ['"get"', '"sendBeacon"'], + ["`get`", "`sendBeacon`"], + ['"get" as const', '"sendBeacon" as const'], + ]) { + const analysis = analyzeJavaScriptStaticSource(` + import * as vscode from "vscode"; + client[${get}]("/api/items"); + navigator[${beacon}]("/api/telemetry", payload); + client[${get}]("setting"); + new Headers()[${get}]("content-type"); + new URLSearchParams()[${get}]("q"); + vscode.workspace.getConfiguration("editor")[${get}]("tabSize"); + `); + + expect.soft(analysis.parse_error_count).toBe(0); + expect.soft(analysis.parse_status).toBe("complete"); + expect + .soft( + analysis.endpoints.map(({ kind, value, mechanism, interpretation }) => [ + kind, + value, + mechanism, + interpretation ?? null, + ]), + ) + .toEqual([ + ["network", "/api/items", "call:client.get", null], + ["network", "/api/telemetry", "call:navigator.sendBeacon", null], + ["network", "setting", "call:client.get", "candidate"], + ]); + } +}); + +it("preserves preload and utility paths through exact computed join members", () => { + for (const key of ['"join"', "`join`", '"join" as const']) { + const analysis = analyzeJavaScriptStaticSource(` + const { BrowserWindow, utilityProcess } = require("electron"); + const path = require("node:path"); + new BrowserWindow({ webPreferences: { + preload: path[${key}](__dirname, "preload.js") + } }); + utilityProcess.fork(path[${key}](__dirname, "worker.js"), [], { + serviceName: "kept" + }); + `); + + expect.soft(analysis.parse_error_count).toBe(0); + expect.soft(analysis.parse_status).toBe("complete"); + expect + .soft( + analysis.role_paths.map( + ({ role, path, resolution_context, mechanism }) => [ + role, + path, + resolution_context, + mechanism, + ], + ), + ) + .toEqual([ + ["preload", "preload.js", "filesystem-expression", "property:preload"], + ]); + expect + .soft( + analysis.electron.browser_windows.map( + ({ location: _location, ...finding }) => finding, + ), + ) + .toEqual([ + { + options_status: "object-literal", + web_preferences_status: "object-literal", + web_preferences: [ + { + name: "preload", + value: { + status: "literal", + value: "preload.js", + expression: null, + }, + }, + ], + preload_path: "preload.js", + preload_resolution_context: "filesystem-expression", + module_key: null, + }, + ]); + expect + .soft( + analysis.electron.utility_processes.map( + ({ location: _location, ...finding }) => finding, + ), + ) + .toEqual([ + { + module_path: "worker.js", + module_resolution_context: "filesystem-expression", + module_expression: null, + service_name: "kept", + module_key: null, + }, + ]); + } +}); + +it("keeps dynamic, interpolated, and invalid template keys unresolved", () => { + for (const { key, errors } of [ + { key: "method", errors: 0 }, + { key: "`join${suffix}`", errors: 0 }, + { key: ["`", String.raw`\unicode`, "`"].join(""), errors: 4 }, + ]) { + const preload = `path[${key}](__dirname, "preload.js")`; + const worker = `path[${key}](__dirname, "worker.js")`; + const analysis = analyzeJavaScriptStaticSource(` + const { BrowserWindow, utilityProcess } = require("electron"); + const path = require("node:path"); + client[${key}]("/api/items"); + navigator[${key}]("/api/telemetry", payload); + new BrowserWindow({ webPreferences: { preload: ${preload} } }); + utilityProcess.fork(${worker}, [], { serviceName: "kept" }); + `); + + expect.soft(analysis.parse_error_count).toBe(errors); + expect.soft(analysis.parse_status).toBe("partial"); + expect.soft(analysis.endpoints).toEqual([]); + expect.soft(analysis.role_paths).toEqual([]); + expect + .soft( + analysis.electron.browser_windows.map( + ({ location: _location, ...finding }) => finding, + ), + ) + .toEqual([ + { + options_status: "object-literal", + web_preferences_status: "object-literal", + web_preferences: [ + { + name: "preload", + value: { status: "dynamic", value: null, expression: preload }, + }, + ], + preload_path: null, + preload_resolution_context: null, + module_key: null, + }, + ]); + expect + .soft( + analysis.electron.utility_processes.map( + ({ location: _location, ...finding }) => finding, + ), + ) + .toEqual([ + { + module_path: null, + module_resolution_context: null, + module_expression: worker, + service_name: "kept", + module_key: null, + }, + ]); + } +}); + +describe("static callee boundary matching", () => { + it("matches complete callee members without suffix collisions", () => { + const analysis = analyzeJavaScriptStaticSource(` + win.loadFile("./window.html"); + BrowserWindow.loadURL("https://app.example.test"); + navigator.serviceWorker.register("./worker.js"); + loadFile("./bare.html"); + win?.loadURL("https://optional.example.test"); + + uploadFile("./upload.html"); + preloadFile("./preload.html"); + downloadURL("https://download.example.test"); + reloadURL("https://reload.example.test"); + notloadURL("https://other.example.test"); + myserviceWorker.register("./not-worker.js"); + `); + + expect( + analysis.role_paths.map(({ path, mechanism }) => [path, mechanism]), + ).toEqual([ + ["./window.html", "call:win.loadFile"], + ["https://app.example.test", "call:BrowserWindow.loadURL"], + ["./bare.html", "call:loadFile"], + ["https://optional.example.test", "call:win.loadURL"], + ]); + // The document base URL is unknown, so the script path stays unresolved. + expect( + analysis.references + .filter(({ kind }) => kind === "service-worker") + .map(({ specifier, expression }) => [specifier, expression]), + ).toEqual([[null, 'navigator.serviceWorker.register("./worker.js")']]); + expect( + analysis.endpoints + .filter(({ mechanism }) => mechanism.endsWith("loadURL")) + .map(({ value }) => value), + ).toEqual(["https://app.example.test", "https://optional.example.test"]); + }); +}); diff --git a/src/domain/javascript/javascriptStaticAnalysisCalls.ts b/src/domain/javascript/javascriptStaticAnalysisCalls.ts index b098b3c26d..16a123b8b5 100644 --- a/src/domain/javascript/javascriptStaticAnalysisCalls.ts +++ b/src/domain/javascript/javascriptStaticAnalysisCalls.ts @@ -1,4 +1,5 @@ import * as t from "@babel/types"; +import { traverseJavaScriptAst } from "./javascriptSemanticTraversal.js"; import { htmlUrlText, stripQueryAndFragment } from "../artifactPathSyntax.js"; import { sanitizeEndpointCandidate } from "../browserObservation.js"; @@ -39,6 +40,11 @@ import { isUnmodifiedGlobalReference, isUnmodifiedImportMetaUrl, } from "./javascriptSemanticAmbientWrites.js"; +import { + semanticFactoryLoaderMemberAt, + semanticFactoryRequireLoaderAt, + semanticRequireLoaderAt, +} from "./javascriptSemanticRequireOrigin.js"; type JavaScriptCallLike = | t.CallExpression @@ -55,23 +61,26 @@ export const inspectCall = ( const { accumulator } = context; const name = calleeName(node.callee); const first = stringValue(node.arguments[0]); - const module = context.accumulator.moduleRangeIndex.find(node.start); - const moduleRequireName = module?.requireName ?? null; - const requireCall = requireCallSpecifier(node, name, moduleRequireName); + const requireCall = requireCallSpecifier(node, bindings, context); if (requireCall !== null) addReference(context, { node, kind: "require", specifier: requireCall.specifier, }); - const chunkLoad = chunkLoadSpecifier(node, name, moduleRequireName); + const chunkLoad = chunkLoadSpecifier(node, bindings, context); if (chunkLoad !== null) addReference(context, { node, kind: "dynamic-import", specifier: chunkLoad.specifier, }); - for (const specifier of vitePreloadDependencySpecifiers(source, node, name)) + for (const specifier of vitePreloadDependencySpecifiers( + node, + name, + context.vitePreloadDependencyTable, + bindings, + )) addReference(context, { node, kind: "dynamic-import", specifier }); const worker = t.isNewExpression(node) ? node : undefined; const workerConstructor = @@ -86,21 +95,33 @@ export const inspectCall = ( addReference(context, { node, kind: "worker", - // Node string paths use the process cwd or filesystem root, neither of - // which is established by the analyzed source's artifact location. + // A string path is relative to the document base URL in a browser and + // to the process cwd in Node; the analyzed source's artifact location + // establishes neither. Only `new URL(path, import.meta.url)` names a + // location relative to this module. specifier: - (workerConstructor === "node" ? undefined : first) ?? - (workerConstructor === undefined || worker === undefined + workerConstructor === undefined || worker === undefined ? undefined - : workerUrlSpecifier(worker, bindings)), + : workerUrlSpecifier(worker, bindings), + }); + // Service worker script URLs are also relative to the document base URL. + if ( + name === "serviceWorker.register" || + name.endsWith(".serviceWorker.register") + ) + addReference(context, { + node, + kind: "service-worker", + specifier: undefined, }); - if (name.endsWith("serviceWorker.register")) - addReference(context, { node, kind: "service-worker", specifier: first }); inspectEndpointCall(node, name, first, context); inspectStorageCall(node, name, first, context); if (first === "audio" || first === "video") noteMediaElementPreloads(node, accumulator); - if (name.endsWith("loadFile") && first !== undefined) + if ( + (name === "loadFile" || name.endsWith(".loadFile")) && + first !== undefined + ) addLocatedFinding(context, { collection: accumulator.roles, key: `role\0renderer\0${first}`, @@ -114,7 +135,7 @@ export const inspectCall = ( location: range(node), }, }); - if (name.endsWith("loadURL") && first !== undefined) + if ((name === "loadURL" || name.endsWith(".loadURL")) && first !== undefined) addLocatedFinding(context, { collection: accumulator.roles, key: `role\0renderer\0${first}`, @@ -212,57 +233,185 @@ const noteMediaElementPreloads = ( const requireCallSpecifier = ( node: JavaScriptCallLike, - name: string, - moduleRequireName: string | null, + bindings: JavaScriptSemanticAnalysisState, + context: FindingContext, ): { readonly specifier: string | undefined } | null => { - if (name === "require" || name.endsWith(".require")) - return { specifier: argumentValue(node.arguments[0]) }; - if (name === "__webpack_require__" || name === moduleRequireName) - return { specifier: argumentValue(node.arguments[0]) }; - if ( - name === "__webpack_require__.bind" || - (moduleRequireName !== null && name === `${moduleRequireName}.bind`) - ) - return { specifier: argumentValue(node.arguments[1]) }; + const bareRequire = t.isIdentifier(node.callee, { name: "require" }); + if (bareRequire) { + const loader = semanticRequireLoaderAt( + node, + bindings, + context.accumulator.factoryRequireBindingIds, + ); + if (loader === "local") return null; + if (loader === "unknown") context.accumulator.unknownFindings += 1; + return { + specifier: + loader === "unknown" ? undefined : argumentValue(node.arguments[0]), + }; + } + const factoryLoader = semanticFactoryRequireLoaderAt( + node, + bindings, + context.accumulator.factoryRequireBindingIds, + ); + if (factoryLoader === "factory-loader") + return { + specifier: argumentValue( + node.arguments[ + t.isMemberExpression(unwrapJavaScriptExpression(node.callee).node) || + t.isOptionalMemberExpression( + unwrapJavaScriptExpression(node.callee).node, + ) + ? 1 + : 0 + ], + ), + }; + if (factoryLoader === "unknown") { + context.accumulator.unknownFindings += 1; + return { specifier: undefined }; + } return null; }; const chunkLoadSpecifier = ( node: JavaScriptCallLike, - name: string, - moduleRequireName: string | null, + bindings: JavaScriptSemanticAnalysisState, + context: FindingContext, ): { readonly specifier: string | undefined } | null => { - if ( - name !== "__webpack_require__.e" && - (moduleRequireName === null || name !== `${moduleRequireName}.e`) - ) - return null; + const loader = semanticFactoryLoaderMemberAt( + node, + bindings, + context.accumulator.factoryRequireBindingIds, + "e", + ); + if (loader === undefined) return null; + if (loader === "unknown") return { specifier: undefined }; return { specifier: prefixedArgument("chunk:", node.arguments[0]) }; }; const vitePreloadDependencySpecifiers = ( - source: string, node: JavaScriptCallLike, name: string, -): readonly string[] => { - if (name !== "__vite__mapDeps" || !t.isArrayExpression(node.arguments[0])) + table: readonly (string | undefined)[] | null, + bindings: JavaScriptSemanticAnalysisState, +): readonly (string | undefined)[] => { + if ( + name !== "__vite__mapDeps" || + table === null || + !t.isArrayExpression(node.arguments[0]) + ) return []; - const indexes = node.arguments[0].elements.flatMap((element) => - t.isNumericLiteral(element) ? [element.value] : [], - ); - if (indexes.length !== node.arguments[0].elements.length) return []; - const table = vitePreloadDependencyTable(source); - return indexes.flatMap((index) => table[index] ?? []); + const indexes = node.arguments[0].elements; + // Explicit cache overrides change which table the helper reads. Retain the + // call slots as unknown unless every override is statically undefined. + if ( + !node.arguments + .slice(1) + .every((argument) => isExplicitDefaultArgument(argument, bindings)) + ) + return indexes.map(() => undefined); + return indexes.map((element) => { + if (element === null || !t.isNumericLiteral(element)) return undefined; + const index = element.value; + return Number.isSafeInteger(index) && index >= 0 ? table[index] : undefined; + }); }; -const vitePreloadDependencyTable = (source: string): readonly string[] => { - const match = - /\.\s*f\s*\|\|\s*\(\s*[^)]*?\.f\s*=\s*\[([\s\S]*?)\]\s*\)/u.exec(source); - const body = match?.[1]; - if (body === undefined) return []; - return [ - ...body.matchAll(/"([^"\\]*(?:\\.[^"\\]*)*)"|'([^'\\]*(?:\\.[^'\\]*)*)'/gu), - ].map((literal) => literal[1] ?? literal[2] ?? ""); +const isExplicitDefaultArgument = ( + argument: t.CallExpression["arguments"][number], + bindings: JavaScriptSemanticAnalysisState, +): boolean => { + if (!t.isNode(argument)) return false; + const value = unwrapJavaScriptExpression(argument).node; + return ( + (t.isUnaryExpression(value) && value.operator === "void") || + (t.isIdentifier(value, { name: "undefined" }) && + isUnmodifiedGlobalReference(value, bindings, "undefined")) + ); +}; + +type ViteMapDepsFunction = + | t.FunctionDeclaration + | t.FunctionExpression + | t.ArrowFunctionExpression; + +/** Index the canonical Vite helper's cache initializer from parsed syntax. */ +export const indexVitePreloadDependencyTable = ( + program: t.Program, +): readonly (string | undefined)[] | null => { + const helpers: ViteMapDepsFunction[] = []; + traverseJavaScriptAst(program, { + enter: (node) => { + if (t.isFunctionDeclaration(node) && node.id?.name === "__vite__mapDeps") + helpers.push(node); + else if ( + t.isVariableDeclarator(node) && + t.isIdentifier(node.id, { name: "__vite__mapDeps" }) + ) { + const init = node.init; + if (t.isFunctionExpression(init) || t.isArrowFunctionExpression(init)) + helpers.push(init); + } + }, + }); + if (helpers.length !== 1) return null; + const helper = helpers[0]; + if (helper === undefined) return null; + const mapParameter = helper.params[1]; + const mapIdentifier = t.isAssignmentPattern(mapParameter) + ? mapParameter.left + : mapParameter; + if (!t.isIdentifier(mapIdentifier)) return []; + const tables: t.ArrayExpression[] = []; + traverseJavaScriptAst(helper, { + skip: (node) => t.isFunction(node), + enter: (node) => { + if ( + t.isAssignmentExpression(node) && + node.operator === "=" && + (t.isMemberExpression(node.left) || + t.isOptionalMemberExpression(node.left)) && + t.isIdentifier(node.left.object, { name: mapIdentifier.name }) && + semanticStaticPropertyName(node.left.property, node.left.computed) === + "f" && + t.isArrayExpression(node.right) + ) + tables.push(node.right); + }, + }); + if (tables.length !== 1) return []; + const table = tables[0]; + if (table === undefined) return []; + const dependencies = table.elements.map((element) => + element === null || t.isSpreadElement(element) + ? undefined + : stringValue(element), + ); + let escapedOrMutated = false; + traverseJavaScriptAst(program, { + skip: (node) => node === helper, + enter: (node, parent) => { + if (!t.isIdentifier(node, { name: "__vite__mapDeps" })) return; + if ( + (parent === helper && + t.isFunctionDeclaration(parent) && + parent.id === node) || + (t.isVariableDeclarator(parent) && + parent.init === helper && + parent.id === node) + ) + return; + if ( + (t.isCallExpression(parent) || t.isOptionalCallExpression(parent)) && + parent.callee === node + ) + return; + escapedOrMutated = true; + }, + }); + return escapedOrMutated ? dependencies.map(() => undefined) : dependencies; }; const inspectEndpointCall = ( @@ -292,8 +441,9 @@ const inspectEndpointCall = ( addEndpoint(context, { node, kind: "network", - value: endpoint, + value: endpoint.value, mechanism: `call:${name}`, + ...(endpoint.kind === "candidate" ? { interpretation: "candidate" } : {}), }); }; @@ -309,11 +459,12 @@ export const inspectRouteProperty = ( context: FindingContext, ): void => { const key = semanticStaticPropertyName(node.key, node.computed); - if ((key === "path" || key === "route") && t.isStringLiteral(node.value)) + const value = stringValue(node.value); + if ((key === "path" || key === "route") && value !== undefined) addEndpoint(context, { node, kind: "route", - value: node.value.value, + value, mechanism: `property:${key}`, }); }; @@ -401,12 +552,15 @@ export const addReference = ( const addEndpoint = (context: FindingContext, input: EndpointInput): void => addLocatedFinding(context, { collection: context.accumulator.endpoints, - key: `endpoint\0${input.kind}\0${input.value}`, + key: `endpoint\0${input.kind}\0${input.interpretation ?? "literal"}\0${input.value}`, node: input.node, value: { kind: input.kind, value: sanitizeEndpointCandidate(input.value), mechanism: input.mechanism, + ...(input.interpretation === undefined + ? {} + : { interpretation: input.interpretation }), module_key: null, location: range(input.node), }, diff --git a/src/domain/javascript/javascriptStaticAnalysisHelpers.ts b/src/domain/javascript/javascriptStaticAnalysisHelpers.ts index 71628bc8ce..fcb835ec45 100644 --- a/src/domain/javascript/javascriptStaticAnalysisHelpers.ts +++ b/src/domain/javascript/javascriptStaticAnalysisHelpers.ts @@ -13,6 +13,7 @@ import type { } from "./javascriptStaticAnalysisTypes.js"; import { readExactJavaScriptLiteral, + runtimeFunctionParameters, semanticStaticPropertyKey, semanticStaticPropertyName, unwrapJavaScriptExpression, @@ -147,7 +148,7 @@ export const modulePropertyName = ( export const factoryRequireName = ( factory: t.FunctionExpression | t.ArrowFunctionExpression | t.ObjectMethod, ): string | null => { - const parameter = factory.params[2]; + const parameter = runtimeFunctionParameters(factory)[2]; return t.isIdentifier(parameter) ? parameter.name : null; }; @@ -222,7 +223,7 @@ const staticCallPath = ( ? stringValue(node.arguments[0]) : undefined; if (name === "fileURLToPath" || name.endsWith(".fileURLToPath")) { - return staticFileUrlPath(argumentNode(node.arguments[0])); + return staticFileUrlPath(node); } if (name.endsWith(".resolve")) return name === "win32.resolve" || name.endsWith(".win32.resolve") @@ -245,12 +246,34 @@ const staticCallPath = ( }; const staticFileUrlPath = ( - argument: t.Node | undefined, + node: t.CallExpression | t.NewExpression, ): string | undefined => { + if (node.arguments.length !== 1) return undefined; + const argument = argumentNode(node.arguments[0]); if (argument === undefined) return undefined; - return isSourceFileUrl(argument) - ? sourceRelativeFileUrlPath(argument) - : staticPath(argument); + if (isSourceFileUrl(argument)) return sourceRelativeFileUrlPath(argument); + const value = staticPath(argument); + const rawPath = value?.split(/[?#]/u, 1)[0] ?? ""; + if ( + value === undefined || + !/^file:/iu.test(value) || + value !== value.trim() || + value.includes("\\") || + [...value].some((character) => character.charCodeAt(0) < 32) || + /%(?:2f|5c)/iu.test(rawPath) + ) + return undefined; + try { + const url = new URL(value); + if (url.protocol !== "file:") return undefined; + // Canonical analysis paths use POSIX spelling so results do not depend on + // the analyzer host. Drive and UNC paths need an explicit target platform. + const path = fileURLToPath(url, { windows: false }); + return /^\/[a-z]:/iu.test(path) ? undefined : path; + } catch (cause: unknown) { + void cause; + return undefined; + } }; /** Project a known source-relative URL through the actual file URL decoder. */ @@ -338,10 +361,13 @@ const isFilesystemPathExpression = (node: t.Node): boolean => { if (name === "URL" || name.endsWith(".URL")) return isFileIdentity(argumentNode(node.arguments[1])); if (name === "fileURLToPath" || name.endsWith(".fileURLToPath")) - return node.arguments.some((argument) => { - const value = argumentNode(argument); - return value !== undefined && isFilesystemPathExpression(value); - }); + return ( + staticFileUrlPath(node) !== undefined || + node.arguments.some((argument) => { + const value = argumentNode(argument); + return value !== undefined && isFilesystemPathExpression(value); + }) + ); return node.arguments.some((argument) => { const value = argumentNode(argument); return value !== undefined && isFilesystemPathExpression(value); @@ -409,13 +435,27 @@ const XHR_METHODS = new Set([ "TRACE", ]); -/** An absolute HTTP(S)/WebSocket URL, or an absolute or relative path. */ -const isRequestUrlLiteral = (value: string): boolean => +/** Legacy URL/path shapes that bypass the overloaded lookup exclusions. */ +const isRequestUrlOrPathLiteral = (value: string): boolean => /^(?:https?|wss?):\/\//iu.test(value) || value.startsWith("/") || value.startsWith("./") || value.startsWith("../"); +/** Explicit request references, classified after receiver-specific exclusions. */ +const isRequestUrlReference = (value: string): boolean => { + if ( + isRequestUrlOrPathLiteral(value) || + value.startsWith("?") || + value.startsWith("#") + ) + return true; + // WHATWG accepts HTTP/WS scheme forms without authority slashes. Parse + // without a base so bare keys stay ambiguous, and keep the source spelling. + if (!/^(?:https?|wss?):/iu.test(value)) return false; + return URL.parse(value) !== null; +}; + /** * Browsing-context and document `open` take a URL or type first and a target * name second. The name is only a spelling, since `parent` or `self` may be a @@ -537,10 +577,20 @@ const isKeyedCollectionReceiver = (callee: t.Node): boolean => { receiver.property, receiver.computed, ); - return property === "headers" || property === "searchParams"; + return ( + property === "headers" || + property === "headersList" || + property === "searchParams" + ); }; -/** Select the literal URL argument for recognized network callees. */ +/** A literal endpoint shape or an ambiguous argument of an HTTP-method-named call. */ +export interface JavaScriptEndpointArgument { + readonly kind: "endpoint" | "candidate"; + readonly value: string; +} + +/** Classify one literal without treating a generic method name as network identity. */ export const endpointArgument = ( name: string, args: readonly ( @@ -551,12 +601,31 @@ export const endpointArgument = ( )[], callee: t.Node, receiverFact?: JavaScriptOpenReceiverFact, -): string | undefined => { +): JavaScriptEndpointArgument | undefined => { + const literal = ( + value: string | undefined, + ): JavaScriptEndpointArgument | undefined => + value === undefined ? undefined : { kind: "endpoint", value }; if (receiverFact === "window") return undefined; - if (name === "fetch" || name.endsWith(".fetch") || name === "WebSocket") - return stringValue(args[0]); - if (name.endsWith(".open")) - return xhrOpenUrl(name, args[0], args[1], receiverFact); + if ( + name === "fetch" || + name.endsWith(".fetch") || + name === "WebSocket" || + name === "EventSource" || + name.endsWith(".EventSource") + ) + return literal(stringValue(args[0])); + if (name.endsWith(".open")) { + const value = xhrOpenUrl(name, args[0], args[1], receiverFact); + if (value === undefined) return undefined; + return { + kind: + stringValue(args[0]) !== undefined || isRequestUrlReference(value) + ? "endpoint" + : "candidate", + value, + }; + } const method = ["get", "post", "put", "patch", "delete", "request"].find( (candidate) => name === candidate || name.endsWith(`.${candidate}`), ); @@ -565,15 +634,26 @@ export const endpointArgument = ( // Only get and delete also belong to the keyed-collection APIs, whose // keys are header, field, and parameter names rather than URLs or paths. // Configuration `get` reads setting keys. - return value !== undefined && + if (value === undefined) return undefined; + if ( (method === "get" || method === "delete") && - !isRequestUrlLiteral(value) && + !isRequestUrlOrPathLiteral(value) && (isKeyedCollectionReceiver(callee) || - (method === "get" && receiverFact === "configuration")) - ? undefined - : value; + (method === "get" && + (receiverFact === "configuration" || + receiverFact === "extension-state"))) + ) + return undefined; + return { + kind: isRequestUrlReference(value) ? "endpoint" : "candidate", + value, + }; } - if (name.endsWith("loadURL")) return stringValue(args[0]); + // Beacon URLs are endpoint observations, retaining the upstream API support. + if (name === "sendBeacon" || name.endsWith(".sendBeacon")) + return literal(stringValue(args[0])); + if (name === "loadURL" || name.endsWith(".loadURL")) + return literal(stringValue(args[0])); return undefined; }; @@ -613,11 +693,8 @@ export const calleeName = (node: t.Node): string => { const memberPropertyName = ( node: t.MemberExpression | t.OptionalMemberExpression, ): string => - node.computed && - !t.isStringLiteral(node.property) && - !t.isNumericLiteral(node.property) - ? `[computed@${String(node.property.start ?? -1)}]` - : semanticStaticPropertyName(node.property, node.computed); + semanticStaticPropertyKey(node.property, node.computed) ?? + (node.computed ? `[computed@${String(node.property.start ?? -1)}]` : ""); /** * Return a string literal value without evaluating an expression. A template diff --git a/src/domain/javascript/javascriptStaticAnalysisModuleIndex.test.ts b/src/domain/javascript/javascriptStaticAnalysisModuleIndex.test.ts index cb15159476..f22aa55f2d 100644 --- a/src/domain/javascript/javascriptStaticAnalysisModuleIndex.test.ts +++ b/src/domain/javascript/javascriptStaticAnalysisModuleIndex.test.ts @@ -62,6 +62,210 @@ it("does not treat a same-named shadow as the factory exports parameter", () => ]); }); +it("does not resolve a parameter named require as a module dependency", () => { + const analysis = analyzeJavaScriptStaticSource( + `function run(require) { return require("./ghost.cjs"); }`, + ); + expect(analysis.references).not.toContainEqual( + expect.objectContaining({ specifier: "./ghost.cjs" }), + ); + expect(analysis.references).toContainEqual( + expect.objectContaining({ + kind: "require", + specifier: null, + expression: 'require("./ghost.cjs")', + }), + ); +}); + +it("keeps a top-level wrapper require redeclaration", () => { + const analysis = analyzeJavaScriptStaticSource( + `var require; require("./wrapper.cjs");`, + ); + expect(analysis.references).toContainEqual( + expect.objectContaining({ specifier: "./wrapper.cjs" }), + ); +}); + +it("retains the wrapper dependency evaluated inside its replacing assignment", () => { + const analysis = analyzeJavaScriptStaticSource( + `var require; require = require("./before-replacement.cjs");`, + ); + expect(analysis.references).toContainEqual( + expect.objectContaining({ specifier: "./before-replacement.cjs" }), + ); +}); + +it.each([ + [ + "a deferred function assignment", + `var require; function load() { require = require("./maybe.cjs"); } load(); load();`, + ], + [ + "an assignment repeated by a loop", + `var require; while (condition) { require = require("./maybe.cjs"); }`, + ], +])("keeps %s unknown", (_label, source) => { + const analysis = analyzeJavaScriptStaticSource(source); + expect(analysis.references).not.toContainEqual( + expect.objectContaining({ specifier: "./maybe.cjs" }), + ); + expect(analysis.references).toContainEqual( + expect.objectContaining({ kind: "require", specifier: null }), + ); +}); + +it.each([ + [ + "a skipped wrapper write", + `var require; if (false) require = localLoader; require("./ghost.cjs");`, + "./ghost.cjs", + ], + [ + "a skipped createRequire write", + `import { createRequire } from "node:module"; let require = () => "local"; if (false) require = createRequire(import.meta.url); require("./ghost.cjs");`, + "./ghost.cjs", + ], + [ + "a class static field write before its computed key in source order", + `var require; class C { static value = (require = () => "local"); [require("./computed-key.cjs")]() {} }`, + "./computed-key.cjs", + ], + [ + "a loop condition write before a later call", + `var require; while ((require = () => "local", false)) {} require("./after-loop.cjs");`, + "./after-loop.cjs", + ], +])("preserves uncertainty for %s", (_label, source, specifier) => { + const analysis = analyzeJavaScriptStaticSource(source); + expect(analysis.references).not.toContainEqual( + expect.objectContaining({ specifier }), + ); + expect(analysis.references).toContainEqual( + expect.objectContaining({ kind: "require", specifier: null }), + ); +}); + +it("keeps a proven createRequire binding", () => { + const analysis = analyzeJavaScriptStaticSource(` + import { createRequire as makeRequire } from "node:module"; + const require = makeRequire(import.meta.url); + require("./created.cjs"); + `); + expect(analysis.references).toEqual( + expect.arrayContaining([ + expect.objectContaining({ specifier: "./created.cjs" }), + ]), + ); +}); + +it("recognizes only factory loader calls and supported bind references", () => { + const analysis = analyzeJavaScriptStaticSource(` + globalThis.webpackChunkApp.push([[1], { + "factory": function (module, exports, loader) { + loader.resolve("./not-a-dependency.cjs"); + loader("./dependency.cjs"); + loader.bind(module, "./bound-dependency.cjs"); + } + }]); + `); + expect(analysis.references.map(({ specifier }) => specifier)).toEqual( + expect.arrayContaining(["./dependency.cjs", "./bound-dependency.cjs"]), + ); + expect(analysis.references).not.toContainEqual( + expect.objectContaining({ specifier: "./not-a-dependency.cjs" }), + ); +}); + +it("keeps reassigned factory loaders unknown instead of resolving imports", () => { + const analysis = analyzeJavaScriptStaticSource(` + globalThis.webpackChunkApp.push([[1], { + "factory": function (module, exports, loader) { + loader = userInput; + loader("./uncertain.cjs"); + } + }]); + `); + expect(analysis.references).not.toContainEqual( + expect.objectContaining({ specifier: "./uncertain.cjs" }), + ); + expect(analysis.references).toContainEqual( + expect.objectContaining({ kind: "require", specifier: null }), + ); +}); + +it("keeps a reassigned function named require unknown", () => { + const analysis = analyzeJavaScriptStaticSource(` + function require() { return userInput; } + require = replacement; + require("./uncertain.cjs"); + `); + expect(analysis.references).not.toContainEqual( + expect.objectContaining({ specifier: "./uncertain.cjs" }), + ); + expect(analysis.references).toContainEqual( + expect.objectContaining({ kind: "require", specifier: null }), + ); +}); + +it("does not treat a shadowed webpack loader name as bundler ownership", () => { + const analysis = analyzeJavaScriptStaticSource(` + function local(__webpack_require__) { + __webpack_require__("./ghost.cjs"); + __webpack_require__.e("ghost-chunk"); + } + `); + expect(analysis.references).not.toContainEqual( + expect.objectContaining({ specifier: "./ghost.cjs" }), + ); + expect(analysis.references).not.toContainEqual( + expect.objectContaining({ specifier: "chunk:ghost-chunk" }), + ); +}); + +it("uses recovered factory and runtime loader identities for webpack imports and chunks", () => { + const analysis = analyzeJavaScriptStaticSource(` + globalThis.webpackChunkApp.push([[1], { + "factory": function (module, exports, __webpack_require__) { + __webpack_require__("./factory-dependency.cjs"); + __webpack_require__.e("factory-chunk"); + } + }, function (__webpack_require__) { + __webpack_require__("factory"); + __webpack_require__.e("runtime-chunk"); + }]); + `); + expect(analysis.references.map(({ specifier }) => specifier)).toEqual( + expect.arrayContaining([ + "./factory-dependency.cjs", + "chunk:factory-chunk", + "chunk:runtime-chunk", + ]), + ); + expect(analysis.bundler_registrations[0]?.async_chunk_keys).toEqual( + expect.arrayContaining(["factory-chunk", "runtime-chunk"]), + ); +}); + +it("does not attribute a shadowed factory loader parameter to async chunks", () => { + const analysis = analyzeJavaScriptStaticSource(` + globalThis.webpackChunkApp.push([[1], { + "factory": function (module, exports, loader) { + function local(__webpack_require__) { + __webpack_require__.e("ghost-chunk"); + } + loader.e("real-chunk"); + } + }]); + `); + expect(analysis.bundler_registrations[0]?.async_chunk_keys).toEqual([ + "real-chunk", + ]); + expect(analysis.references).not.toContainEqual( + expect.objectContaining({ specifier: "chunk:ghost-chunk" }), + ); +}); + it("does not confuse a shadowed dispatch table with the outer module table", () => { const analysis = analyzeJavaScriptStaticSource(`(() => { const modules = { 17(module) { module.exports = require("electron"); } }; diff --git a/src/domain/javascript/javascriptStaticAnalysisState.ts b/src/domain/javascript/javascriptStaticAnalysisState.ts index 5173362faf..70e1eb0860 100644 --- a/src/domain/javascript/javascriptStaticAnalysisState.ts +++ b/src/domain/javascript/javascriptStaticAnalysisState.ts @@ -53,6 +53,8 @@ export interface JavaScriptAnalysisAccumulator { readonly utilityProcesses: LocatedJavaScriptFinding[]; readonly nativeAddonBindings: LocatedJavaScriptFinding[]; readonly moduleRangeIndex: JavaScriptModuleRangeIndex; + /** Binding identities proven as third-parameter loaders of recovered factories. */ + readonly factoryRequireBindingIds: Set; /** `preload` attributes of audio and video elements, which are not scripts. */ readonly mediaElementPreloads: WeakSet; readonly seen: Set; @@ -65,6 +67,7 @@ export interface JavaScriptFindingContext { readonly rpcScan: JavaScriptRpcScan; readonly source: string; readonly accumulator: JavaScriptAnalysisAccumulator; + readonly vitePreloadDependencyTable: readonly (string | undefined)[] | null; readonly openReceiverFacts?: ReadonlyMap; /** Proven Electron export path segments by root offset. */ readonly electronBindings?: ReadonlyMap; @@ -83,6 +86,7 @@ export interface JavaScriptEndpointInput { readonly kind: JavaScriptStaticEndpoint["kind"]; readonly value: string; readonly mechanism: string; + readonly interpretation?: "candidate"; } /** Values required to append one module-attributable static finding. */ @@ -112,6 +116,7 @@ export const createJavaScriptAnalysisAccumulator = ( utilityProcesses: [], nativeAddonBindings: [], moduleRangeIndex: new JavaScriptModuleRangeIndex(sourceLength), + factoryRequireBindingIds: new Set(), mediaElementPreloads: new WeakSet(), seen: new Set(), visitedNodes: 0, diff --git a/src/domain/javascript/javascriptStaticAnalysisTypes.ts b/src/domain/javascript/javascriptStaticAnalysisTypes.ts index 1e72deb1d7..887277950a 100644 --- a/src/domain/javascript/javascriptStaticAnalysisTypes.ts +++ b/src/domain/javascript/javascriptStaticAnalysisTypes.ts @@ -1,5 +1,8 @@ import type { ElectronStaticFindings } from "./electronStaticAnalysisTypes.js"; +/** Token semantics shared by bundle producers and worker admission. */ +export const JAVASCRIPT_AST_FINGERPRINT_ALGORITHM = "babel-ast-v2"; + /** Static syntax context retained for later artifact path resolution. */ export type JavaScriptStaticPathContext = | "module-specifier" @@ -42,6 +45,8 @@ export interface JavaScriptStaticEndpoint { readonly kind: "route" | "network"; readonly value: string; readonly mechanism: string; + /** An overloaded callee name suggests this value; its network receiver is unproven. */ + readonly interpretation?: "candidate"; readonly module_key: string | null; /** Recovered factory runtime; absent in older or unbundled findings. */ readonly module_runtime?: string; @@ -75,7 +80,7 @@ interface JavaScriptBundlerModuleState { /** One exact module factory recovered from a Webpack/Rspack registration. */ export interface JavaScriptBundlerModule extends JavaScriptBundlerModuleState { readonly structural_fingerprint_sha256: string; - readonly structural_fingerprint_algorithm: "babel-ast-v1"; + readonly structural_fingerprint_algorithm: typeof JAVASCRIPT_AST_FINGERPRINT_ALGORITHM; } /** One statically recognized bundler chunk or module registration. */ diff --git a/src/domain/javascript/javascriptStaticPathResolution.test.ts b/src/domain/javascript/javascriptStaticPathResolution.test.ts index 30bbc317ff..8d1a58ff12 100644 --- a/src/domain/javascript/javascriptStaticPathResolution.test.ts +++ b/src/domain/javascript/javascriptStaticPathResolution.test.ts @@ -1,4 +1,5 @@ import { posix } from "node:path"; +import { fileURLToPath } from "node:url"; import { parseExpression } from "@babel/parser"; import { describe, expect, it } from "vitest"; @@ -22,6 +23,7 @@ const absoluteParts = [ const anchoredExpressions = [ ['path.resolve(__dirname, "preload.cjs")', "preload.cjs"], + ['path[`resolve`](__dirname, "preload.js")', "preload.js"], ['path.posix.resolve("ignored", __dirname, "preload.js")', "preload.js"], ['path.resolve("/ignored", __dirname, "preload.js")', "preload.js"], [ @@ -44,6 +46,8 @@ const anchoredExpressions = [ ] as const; const unknownExpressions = [ + 'path[method](__dirname, "preload.js")', + 'path[`join${suffix}`](__dirname, "preload.js")', "path.resolve()", "path.posix.resolve()", 'path.resolve("sub", "../preload.js")', @@ -105,6 +109,11 @@ describe("inert JavaScript path resolution", () => { "preload.js", "filesystem-expression", ], + [ + 'path[`join`](__dirname, "preload.js")', + "preload.js", + "filesystem-expression", + ], [ 'path.join(__dirname, "/preload.js")', "preload.js", @@ -207,6 +216,32 @@ describe("inert JavaScript path resolution", () => { ); }); +it.each([ + "file:///app/preload%20file.js", + "file:///app/%E2%9C%93.js", + "file:///app/preload.js?query=1#fragment", +])("decodes literal file URL %s as a filesystem expression", (url) => { + assertPathProjection( + `fileURLToPath(${JSON.stringify(url)})`, + fileURLToPath(url, { windows: false }), + "filesystem-expression", + ); +}); + +it.each([ + 'fileURLToPath("file:///app/%2Fpreload.js")', + 'fileURLToPath("file:///app/%5Cpreload.js")', + 'fileURLToPath("file:///app/%")', + 'fileURLToPath("file:///C:/app/preload.js")', + 'fileURLToPath("file://remote/share/preload.js")', + 'fileURLToPath("file:///app/preload.js", options)', + "fileURLToPath(selectedUrl)", +])("leaves platform-dependent or invalid %s unknown", (expression) => + expect( + staticPath(parseExpression(expression, { sourceType: "module" })), + ).toBeUndefined(), +); + const analyzePathConsumers = (expression: string) => { const property = `preload: ${expression}`; const genericPrefix = "const options = { "; diff --git a/src/domain/javascript/javascriptSymbolicBindings.test.ts b/src/domain/javascript/javascriptSymbolicBindings.test.ts index a4248e30b8..b78bd543a1 100644 --- a/src/domain/javascript/javascriptSymbolicBindings.test.ts +++ b/src/domain/javascript/javascriptSymbolicBindings.test.ts @@ -170,6 +170,34 @@ it("invalidates a bootstrap module slot mutated through an alias", () => { ]); }); +it.each([ + [ + "destructuring assignment", + `[table[1]] = [function(module) { module.exports = require("other"); }];`, + ], + ["for-of head", `for (table[1] of replacements) {}`], + [ + "wrapped assignment", + `(table as never)[1] = function(module) { module.exports = require("other"); };`, + ], +])( + "invalidates a bootstrap module slot overwritten through %s", + (_name, write) => { + const values = inspectedValues(`(() => { + const modules = { + 1(module) { module.exports = require("electron"); }, + 2(module, exports, load) { inspect(load(1)); } + }; + const table = modules; + function load(id) { const m = {exports:{}}; modules[id](m,m.exports,load); return m.exports; } + ${write} + load(2); + })();`); + expect(values).toHaveLength(1); + expect(values[0]).toEqual(expect.objectContaining({ kind: "unknown" })); + }, +); + it.each([ `const electron=require("electron"); const ipc=electron.ipcMain; ipc.handle=()=>null; inspect(electron.ipcMain.handle);`, `require("electron").ipcMain.handle=()=>null; inspect(require("electron").ipcMain.handle);`, @@ -361,6 +389,62 @@ it("composes fallthrough guards across successive conditional returns", () => { ]); }); +it.each([ + ["while", 'inspect((() => { while (flag) { return "loop"; } })());'], + ["for", 'inspect((() => { for (;;) { return "forever"; } })());'], + ["try", 'inspect((() => { try { return "try"; } finally {} })());'], + [ + "switch", + 'inspect((() => { switch (x) { default: return "switch"; } })());', + ], +])( + "keeps unsupported %s return control flow explicitly unknown", + (_name, source) => { + const value = inspectOne(source); + expect( + symbolicVariants(value).some((variant) => variant.kind === "unknown"), + ).toBe(true); + expect(value).not.toEqual({ kind: "literal", value: undefined }); + }, +); + +it("preserves later returns with unresolved conditions after unsupported return flow", () => { + const variants = symbolicVariants( + inspectOne(`inspect((() => { + while (flag) { return "inside"; } + return "after"; + })());`), + ); + + expect(variants.some((variant) => variant.kind === "unknown")).toBe(true); + expect(variants).toContainEqual( + expect.objectContaining({ + kind: "literal", + value: "after", + conditionUnresolved: true, + }), + ); +}); + +it("does not treat loops without owned returns as unknown return flow", () => { + expect( + inspectOne(`inspect((() => { + while (flag) { work(); } + return "after"; + })());`), + ).toEqual({ kind: "literal", value: "after" }); +}); + +it("does not treat nested-function returns as wrapper return flow", () => { + expect( + inspectOne(`inspect((() => { + try { function nested() { return "inner"; } } + catch {} + return "outer"; + })());`), + ).toEqual({ kind: "literal", value: "outer" }); +}); + it("marks return guards unresolved when a nested join cannot be represented", () => { const value = inspectOne(` function choose() { @@ -438,3 +522,55 @@ it("propagates branch guards through object properties and function calls", () = ]); } }); + +it("completes a loaded module past a very large array without stack overflow", () => { + const parsed = parseJavaScriptSource(`(() => { + const modules = { + 4(module) { + module.exports = { table: [0] }; + if (module) { return; } + return; + }, + 8(module, exports, load) { const dep = load(4); inspect(dep); } + }; + function load(id) { const m = {exports:{}}; modules[id](m,m.exports,load); return m.exports; } + load(8); + })();`); + if (parsed === null) throw new Error("symbolic source fixture did not parse"); + const arrays: t.ArrayExpression[] = []; + const bareReturns: t.ReturnStatement[] = []; + let inspectCall: t.CallExpression | undefined; + traverseJavaScriptAst(parsed.program, { + enter: (node) => { + if (t.isArrayExpression(node)) arrays.push(node); + if (t.isReturnStatement(node) && node.argument == null) + bareReturns.push(node); + if ( + t.isCallExpression(node) && + t.isIdentifier(node.callee, { name: "inspect" }) + ) + inspectCall = node; + }, + }); + for (const array of arrays) + array.elements = Array.from({ length: 1_000_000 }, () => + t.numericLiteral(0), + ); + const earliestReturn = bareReturns.toSorted( + (left, right) => (left.start ?? 0) - (right.start ?? 0), + )[0]; + const inspected = inspectCall?.arguments[0]; + if (earliestReturn === undefined || inspected === undefined) + throw new Error("symbolic source fixture is missing its probes"); + + const state = completeSemanticSteps( + indexJavaScriptBindingsSteps(parsed.program), + ); + const resolver = new JavaScriptSymbolicBindings(state, parsed.program); + const namespace = symbolicVariants(resolver.evaluate(inspected)).find( + (variant) => variant.kind === "namespace", + ); + if (namespace?.kind !== "namespace") + throw new Error("loaded module namespace missing"); + expect(namespace.environment.completedAt).toBe(earliestReturn); +}); diff --git a/src/domain/javascript/javascriptSymbolicBindings.ts b/src/domain/javascript/javascriptSymbolicBindings.ts index 62c70f953e..42f629f097 100644 --- a/src/domain/javascript/javascriptSymbolicBindings.ts +++ b/src/domain/javascript/javascriptSymbolicBindings.ts @@ -1,6 +1,9 @@ import * as t from "@babel/types"; import { semanticStaticPropertyKey, + semanticWriteSiteTarget, + semanticWriteTargets, + runtimeFunctionParameters, unwrapJavaScriptExpression, } from "./javascriptAstValues.js"; import { javascriptBootstrapAt } from "./javascriptBootstrap.js"; @@ -15,7 +18,6 @@ import { modulePropertyName, } from "./javascriptStaticAnalysisHelpers.js"; import { - childNodes, completeSemanticSteps, traverseJavaScriptAst, } from "./javascriptSemanticTraversal.js"; @@ -56,6 +58,17 @@ const isObjectPrototypeSetter = ( !property.shorthand && name === "__proto__"; +const containsOwnedReturn = (root: t.Node): boolean => { + let found = false; + traverseJavaScriptAst(root, { + skip: (node) => found || t.isFunction(node) || t.isClass(node), + enter: (node) => { + if (t.isReturnStatement(node)) found = true; + }, + }); + return found; +}; + /** Demand-driven substitutions for local bindings, closures and literal module tables. */ export class JavaScriptSymbolicBindings { readonly #writes = new Map(); @@ -104,12 +117,8 @@ export class JavaScriptSymbolicBindings { }); traverseJavaScriptAst(program, { enter: (node) => { - if (t.isAssignmentExpression(node)) - this.#indexWrite(node.left, node.right, node); - if (t.isUpdateExpression(node)) - this.#indexWrite(node.argument, node, node); - if (t.isUnaryExpression(node, { operator: "delete" })) - this.#indexWrite(node.argument, node, node); + const target = semanticWriteSiteTarget(node); + if (target !== null) this.#indexWriteSite(target, node); if ( t.isCallExpression(node) && t.isMemberExpression(node.callee) && @@ -387,7 +396,7 @@ export class JavaScriptSymbolicBindings { invocationNode?: t.Node, ): SymbolicEnvironment { const values = new Map(); - fn.node.params.forEach((param, index) => { + runtimeFunctionParameters(fn.node).forEach((param, index) => { const target = t.isRestElement(param) ? param.argument : param; if (!t.isIdentifier(target)) return; const binding = resolveSemanticBindingState( @@ -548,7 +557,7 @@ export class JavaScriptSymbolicBindings { this.unresolved = true; return symbolicUnknown("initializer not available at this read"); } - if (this.state.conditionalInitializers.has(initializer.node)) + if (this.state.conditionallySkippedExpressions.has(initializer.node)) return symbolicUnknown("conditional initializer"); let value = this.evaluate(initializer.node, env); for (const key of initializer.projection) { @@ -635,20 +644,19 @@ export class JavaScriptSymbolicBindings { #moduleCompletionPoint(factory: t.Function): t.Node { let earliestReturn: t.ReturnStatement | undefined; - const pending: t.Node[] = [factory.body]; - while (pending.length > 0) { - const node = pending.pop(); - if (!node) continue; - if (node !== factory.body && (t.isFunction(node) || t.isClass(node))) - continue; - if ( - t.isReturnStatement(node) && - (earliestReturn === undefined || - (node.start ?? Infinity) < (earliestReturn.start ?? Infinity)) - ) - earliestReturn = node; - pending.push(...childNodes(node)); - } + // Use the shared traversal so source-sized child lists are pushed one at + // a time. Spreading array elements or call arguments overflows the stack. + traverseJavaScriptAst(factory.body, { + skip: (node) => t.isFunction(node) || t.isClass(node), + enter: (node) => { + if ( + t.isReturnStatement(node) && + (earliestReturn === undefined || + (node.start ?? Infinity) < (earliestReturn.start ?? Infinity)) + ) + earliestReturn = node; + }, + }); return earliestReturn ?? factory.body; } @@ -835,6 +843,24 @@ export class JavaScriptSymbolicBindings { unresolvedFallthrough = true; } } + if ( + t.isWhileStatement(statement) || + t.isDoWhileStatement(statement) || + t.isForStatement(statement) || + t.isForInStatement(statement) || + t.isForOfStatement(statement) || + t.isSwitchStatement(statement) || + t.isTryStatement(statement) + ) { + if (containsOwnedReturn(statement)) { + possible.push( + symbolicUnknown( + `unsupported return control flow: ${statement.type}`, + ), + ); + unresolvedFallthrough = true; + } + } } return { values: possible, fallsThrough: true }; } @@ -866,9 +892,7 @@ export class JavaScriptSymbolicBindings { ? {} : { scopeId: this.state.scopeByNode.get(factory)!.scopeId }), }; - const first = factory.params[0]; - const second = factory.params[1]; - const third = factory.params[2]; + const [first, second, third] = runtimeFunctionParameters(factory); const binding = (n: t.Node | undefined) => t.isIdentifier(n) ? resolveSemanticBindingState(this.state, n, n.name)?.bindingId @@ -944,15 +968,33 @@ export class JavaScriptSymbolicBindings { else for (const key of invalidSlots) modules.delete(key); } - #indexWrite(target: t.Node, value: t.Node, site: t.Node): void { + #indexWriteSite(target: t.Node, site: t.Node): void { + const direct = unwrapJavaScriptExpression(target).node; + if ( + (t.isIdentifier(direct) || t.isMemberExpression(direct)) && + !t.isForOfStatement(site) && + !t.isForInStatement(site) + ) { + this.#indexWrite( + direct, + t.isAssignmentExpression(site) ? site.right : site, + site, + ); + return; + } + for (const leaf of semanticWriteTargets(target)) + this.#indexWrite(leaf, site, site, false); + } + + #indexWrite(target: t.Node, value: t.Node, site: t.Node, exact = true): void { const path: string[] = []; - let root = target; - let dynamic = false; + let root = unwrapJavaScriptExpression(target).node; + let dynamic = !exact; while (t.isMemberExpression(root)) { const key = semanticStaticPropertyKey(root.property, root.computed); if (key === null) dynamic = true; else path.unshift(key); - root = root.object; + root = unwrapJavaScriptExpression(root.object).node; } if (dynamic && t.isIdentifier(root)) { const binding = resolveSemanticBindingState(this.state, root, root.name); diff --git a/src/domain/javascript/javascriptVitePreloadDependencies.test.ts b/src/domain/javascript/javascriptVitePreloadDependencies.test.ts new file mode 100644 index 0000000000..4995dd06c8 --- /dev/null +++ b/src/domain/javascript/javascriptVitePreloadDependencies.test.ts @@ -0,0 +1,86 @@ +import { describe, expect, it } from "vitest"; + +import { analyzeJavaScriptStaticSource } from "./javascriptStaticAnalysis.js"; + +describe("Vite preload dependency recovery", () => { + it("decodes helper table values and preserves sparse and dynamic slots", () => { + const source = String.raw` + const __vite__mapDeps = (indexes, map = __vite__mapDeps, dependencies = + (map.f || (map.f = ["./caf\u00e9.js", , getDependency(), "./last.js"]))) => + indexes.map((index) => dependencies[index]); + const dynamicIndex = chooseIndex(); + __vite__mapDeps([0, , 3, dynamicIndex]); + `; + const analysis = analyzeJavaScriptStaticSource(source); + const references = analysis.references.filter( + ({ kind }) => kind === "dynamic-import", + ); + + expect(references.map(({ specifier }) => specifier).sort()).toEqual([ + "./café.js", + "./last.js", + null, + ]); + expect(references.every(({ location }) => location.start.line === 6)).toBe( + true, + ); + expect(references.filter(({ specifier }) => specifier === null)).toEqual([ + expect.objectContaining({ + expression: expect.stringContaining( + "__vite__mapDeps([0, , 3, dynamicIndex])", + ), + }), + ]); + }); + + it("does not take dependency strings from comments or unrelated helpers", () => { + const analysis = analyzeJavaScriptStaticSource(` + // cache.f = ["./comment.js"] + const unrelated = (cache) => cache.f || (cache.f = ["./unrelated.js"]); + __vite__mapDeps([0]); + `); + + expect( + analysis.references.filter(({ kind }) => kind === "dynamic-import"), + ).toEqual([]); + }); + + it("keeps cache escapes and explicit cache overrides unknown", () => { + const helper = ` + const __vite__mapDeps = (indexes, map = __vite__mapDeps, dependencies = + (map.f || (map.f = ["./original.js"]))) => + indexes.map((index) => dependencies[index]); + `; + const escaped = analyzeJavaScriptStaticSource(` + ${helper} + const dependencies = __vite__mapDeps.f; + dependencies[0] = "./changed.js"; + __vite__mapDeps([0]); + `); + expect( + escaped.references + .filter(({ kind }) => kind === "dynamic-import") + .map(({ specifier }) => specifier), + ).toEqual([null]); + + const overridden = analyzeJavaScriptStaticSource(` + ${helper} + __vite__mapDeps([0], customMap); + `); + expect( + overridden.references + .filter(({ kind }) => kind === "dynamic-import") + .map(({ specifier }) => specifier), + ).toEqual([null]); + + const defaulted = analyzeJavaScriptStaticSource(` + ${helper} + __vite__mapDeps([0], undefined, void 0); + `); + expect( + defaulted.references + .filter(({ kind }) => kind === "dynamic-import") + .map(({ specifier }) => specifier), + ).toEqual(["./original.js"]); + }); +}); diff --git a/src/domain/javascript/sourceToBundleComparison.test.ts b/src/domain/javascript/sourceToBundleComparison.test.ts index d231cb5ec1..a16ea4c0c2 100644 --- a/src/domain/javascript/sourceToBundleComparison.test.ts +++ b/src/domain/javascript/sourceToBundleComparison.test.ts @@ -30,6 +30,58 @@ const HASH = { const EVIDENCE_ID = `ev_${"a".repeat(64)}`; describe("historical source to bundle comparison", () => { + it("matches an unchanged JSON source artifact by its exact content digest", () => { + const path = "src/data.json"; + const artifact = createJavaScriptApplicationNode({ + kind: "artifact", + identity: { + strategy: "content-digest", + stability: "global-exact", + sha256: HASH.unchanged, + }, + observations: [ + { + label: path, + properties: { path, file_kind: "json" }, + evidence: artifactEvidence(HASH.unchanged, path), + }, + ], + }); + const result = compareSourceToBundle({ + reference: historicalGraph("complete", [path]), + application: { + evidenceId: EVIDENCE_ID, + rootArtifactSha256: HASH.artifact, + graph: createJavaScriptApplicationGraph({ + schema: "JavaScriptApplicationGraph", + root_node_ids: [artifact.node_id], + nodes: [artifact], + edges: [], + coverage: { + status: "complete", + truncated: false, + omitted_count: 0, + limits: [], + }, + limitations: [], + }), + }, + }); + + expect(item(result, path)).toMatchObject({ + status: "unchanged", + confidence: "exact", + current_node_ids: [artifact.node_id], + candidates: [ + expect.objectContaining({ + signals: expect.arrayContaining([ + expect.objectContaining({ kind: "exact-source-digest" }), + ]), + }), + ], + }); + }); + it("classifies exact, changed, absent, split, and merged mappings stably", () => { const reference = historicalGraph("complete"); const graph = applicationGraph("complete"); @@ -120,7 +172,9 @@ describe("historical source to bundle comparison", () => { }); expect(result.coverage.status).toBe("partial"); }); +}); +describe("source to bundle candidate evaluation", () => { it("evaluates all indexed candidates for each source file", () => { const result = compareSourceToBundle({ reference: historicalGraph("complete", ["src/split.ts"]), @@ -330,11 +384,13 @@ describe("source to bundle path syntax", () => { }); }); - it.each([ - "https://example.test/src/modified.ts?revision=2#section", - "webpack://fixture/./src/modified.ts?revision=2#section", - ])("still strips URL query and fragment from %s", (path) => { - const node = moduleNode("url", { path }); + it.each( + [ + "https://example.test/src/modified.ts?revision=2#section", + "webpack://fixture/./src/modified.ts?revision=2#section", + ].flatMap((path) => ["path", "module_path"].map((key) => ({ key, path }))), + )("still strips URL query and fragment from $key $path", ({ key, path }) => { + const node = moduleNode("url", { [key]: path }); const graph = createJavaScriptApplicationGraph({ schema: "JavaScriptApplicationGraph", root_node_ids: [node.node_id], diff --git a/src/domain/javascript/sourceToBundleSignals.ts b/src/domain/javascript/sourceToBundleSignals.ts index b3546aecc6..62490bec86 100644 --- a/src/domain/javascript/sourceToBundleSignals.ts +++ b/src/domain/javascript/sourceToBundleSignals.ts @@ -45,6 +45,9 @@ export interface SourceToBundleCandidateIndex { } const RELEVANT_NODE_KINDS = new Set([ + // Non-JavaScript source files such as JSON are projected as artifacts. + // Their content digests and paths still identify historical source files. + "artifact", "javascript-asset", "javascript-module", "source-module", @@ -188,7 +191,7 @@ const projectCurrentNode = (node: ApplicationNode): CurrentProjection => { digests.add(node.identity.source_sha256); } if (node.identity.strategy === "canonical-path") - addPath(paths, "canonical-path", node.identity.path); + addPath(paths, "canonical-path", node.identity.path, "artifact"); for (const observation of node.observations) { if (node.identity.strategy === "source-map-original") { const resolution = observation.source_map_reference?.resolution; @@ -202,8 +205,26 @@ const projectCurrentNode = (node: ApplicationNode): CurrentProjection => { const sourceDigest = observation.properties.source_sha256; if (typeof sourceDigest === "string" && isDigest(sourceDigest)) digests.add(sourceDigest); - for (const key of PATH_PROPERTIES) - addJsonPath(paths, observation.properties[key]); + for (const key of PATH_PROPERTIES) { + // A resolved declaration already has an owning artifact-path fact. + if ( + key === "declared_path" && + typeof observation.properties.resolved_path === "string" + ) + continue; + addJsonPath( + paths, + observation.properties[key], + key === "resolved_path" || + ((key === "path" || key === "module_path") && + !( + typeof observation.properties[key] === "string" && + SCHEME_URL.test(observation.properties[key]) + )) + ? "artifact" + : "reference", + ); + } } } return { @@ -323,29 +344,40 @@ const matchingUnicodePaths = ( .map(({ value }) => value); }; -const addJsonPath = (paths: CurrentPath[], value: unknown): void => { - if (typeof value === "string") addPath(paths, "observation-path", value); +const addJsonPath = ( + paths: CurrentPath[], + value: unknown, + format: "artifact" | "reference", +): void => { + if (typeof value === "string") + addPath(paths, "observation-path", value, format); }; const addPath = ( paths: CurrentPath[], kind: CurrentPathKind, raw: string, + format: "artifact" | "reference", ): void => { - const value = normalizeCurrentPath(raw); + const value = normalizeCurrentPath(raw, format); if (value !== null) paths.push({ kind, value, allowSuffix: true }); }; const SCHEME_URL = /^[a-z][a-z0-9+.-]*:\/\//iu; const SCHEME_URL_PREFIX = /^[a-z][a-z0-9+.-]*:\/\/+/iu; -const normalizeCurrentPath = (raw: string): string | null => { - const withoutQuery = SCHEME_URL.test(raw) - ? (raw.split(/[?#]/u, 1)[0] ?? "") - : raw; - const portable = withoutQuery - .replace(SCHEME_URL_PREFIX, "") - .replaceAll("\\", "/"); +const normalizeCurrentPath = ( + raw: string, + format: "artifact" | "reference", +): string | null => { + const withoutQuery = + format === "reference" && SCHEME_URL.test(raw) + ? (raw.split(/[?#]/u, 1)[0] ?? "") + : raw; + const portable = + format === "artifact" + ? withoutQuery + : withoutQuery.replace(SCHEME_URL_PREFIX, "").replaceAll("\\", "/"); const parts: string[] = []; for (const part of portable.split("/")) { if (part === "" || part === ".") continue; diff --git a/src/domain/jsonValue.test.ts b/src/domain/jsonValue.test.ts index 2378b23b2c..faca71f535 100644 --- a/src/domain/jsonValue.test.ts +++ b/src/domain/jsonValue.test.ts @@ -40,6 +40,13 @@ it("still parses only JSON values at runtime", () => { expect(jsonValueSchema.safeParse(value).success).toBe(false); }); +it("rejects sparse arrays and enumerable symbol-keyed object members", () => { + const sparse = new Array(1); + const symbolKeyed = { [Symbol("caller data")]: "retain or reject" }; + expect(jsonValueSchema.safeParse(sparse).success).toBe(false); + expect(jsonValueSchema.safeParse(symbolKeyed).success).toBe(false); +}); + const deepObject = (depth: number): unknown => { let value: unknown = 1; for (let index = 0; index < depth; index += 1) value = { nested: value }; @@ -75,6 +82,18 @@ describe("jsonValueSchema depth bound", () => { }, ); + it("stops enclosing refinements before rejected JSON reaches consumers", () => { + let consumed = 0; + const envelope = z.object({ value: jsonValueSchema }).superRefine(() => { + consumed += 1; + }); + const result = envelope.safeParse({ value: deepObject(10_000) }); + expect(result.success).toBe(false); + if (result.success) return; + expect(result.error.issues[0]?.message).toContain("maximum nesting depth"); + expect(consumed).toBe(0); + }); + it("rejects cyclic values instead of recursing forever", () => { const cyclic: Record = {}; cyclic.self = cyclic; @@ -122,3 +141,55 @@ it.each([undefined, Number.NaN, Infinity, 1n, () => 0, new Date(0)])( } }, ); + +describe("inert JSON data boundaries", () => { + it.each(["member", "constructor", "array"])( + "rejects an accessor at %s without invoking it", + (kind) => { + let accessed = false; + const value = kind === "array" ? [] : {}; + Object.defineProperty(value, kind === "array" ? "0" : kind, { + enumerable: true, + get() { + accessed = true; + throw new Error("caller accessor must remain inert"); + }, + }); + expect(jsonValueSchema.safeParse(value).success).toBe(false); + expect(accessed).toBe(false); + }, + ); + + it("rejects proxies before reflection can invoke traps", () => { + let observed = false; + const value = new Proxy( + {}, + { + getPrototypeOf() { + observed = true; + throw new Error("caller trap must remain inert"); + }, + }, + ); + expect(jsonValueSchema.safeParse(value).success).toBe(false); + expect(observed).toBe(false); + const revoked = Proxy.revocable([], {}); + revoked.revoke(); + expect(jsonValueSchema.safeParse(revoked.proxy).success).toBe(false); + }); + + it("projects array data without invoking caller iteration or map methods", () => { + const value = [1, { nested: "preserved" }]; + Object.defineProperty(value, Symbol.iterator, { + value() { + throw new Error("caller iterator must remain inert"); + }, + }); + Object.defineProperty(value, "map", { + value() { + throw new Error("caller map must remain inert"); + }, + }); + expect(jsonValueSchema.parse(value)).toEqual([1, { nested: "preserved" }]); + }); +}); diff --git a/src/domain/jsonValue.ts b/src/domain/jsonValue.ts index adaf2017f4..5c954ba1f9 100644 --- a/src/domain/jsonValue.ts +++ b/src/domain/jsonValue.ts @@ -1,3 +1,4 @@ +import { types } from "node:util"; import { z } from "zod"; import { isImmutableJsonSnapshot } from "./immutableJson.js"; @@ -15,7 +16,6 @@ export type JsonValue = const isRecord = (value: unknown): value is Record => typeof value === "object" && value !== null && !Array.isArray(value); -const recordShape = z.record(z.string(), z.unknown()); const immutableValidation = new WeakMap(); /** Validate JSON meaning and nesting without cloning a trusted immutable value. */ @@ -57,6 +57,8 @@ export function* jsonValueValidationSteps( `JSON value exceeds maximum nesting depth of ${MAX_JSON_DEPTH}`, ); const value = item.value; + if (typeof value === "object" && value !== null && types.isProxy(value)) + return finish("JSON values cannot contain proxies"); if ( value === null || typeof value === "string" || @@ -66,12 +68,27 @@ export function* jsonValueValidationSteps( if (typeof value === "number") { if (!Number.isFinite(value)) return finish("JSON numbers must be finite"); } else if (Array.isArray(value)) { - for (const child of value) - pending.push({ value: child, depth: item.depth + 1 }); - } else if (isRecord(value) && recordShape.safeParse(value).success) { - // Inspect every own member, including keys Zod's record parser skips. - for (const child of Object.values(value)) - pending.push({ value: child, depth: item.depth + 1 }); + for (let index = 0; index < value.length; index += 1) { + const descriptor = Object.getOwnPropertyDescriptor(value, index); + if (descriptor === undefined || !Object.hasOwn(descriptor, "value")) + return finish( + "JSON arrays require dense data properties without accessors", + ); + pending.push({ value: descriptor.value, depth: item.depth + 1 }); + } + } else if (isRecord(value)) { + const prototype: unknown = Object.getPrototypeOf(value); + if (prototype !== Object.prototype && prototype !== null) + return finish("JSON objects require ordinary data properties"); + for (const key of Reflect.ownKeys(value)) { + const descriptor = Object.getOwnPropertyDescriptor(value, key); + if (descriptor?.enumerable !== true) continue; + if (typeof key !== "string") + return finish("JSON object keys must be strings"); + if (!Object.hasOwn(descriptor, "value")) + return finish("JSON objects cannot contain accessors"); + pending.push({ value: descriptor.value, depth: item.depth + 1 }); + } } else return finish("Invalid JSON value"); } return finish(undefined); @@ -80,7 +97,12 @@ export function* jsonValueValidationSteps( const preserveJsonValue = (value: JsonValue): JsonValue => { const immutableCandidate: unknown = value; if (isImmutableJsonSnapshot(immutableCandidate)) return value; - if (Array.isArray(value)) return value.map(preserveJsonValue); + if (Array.isArray(value)) { + const copy: JsonValue[] = []; + for (let index = 0; index < value.length; index += 1) + copy.push(preserveJsonValue(value[index]!)); + return copy; + } if (value !== null && typeof value === "object") return Object.fromEntries( Object.entries(value).map(([key, child]) => [ @@ -91,13 +113,14 @@ const preserveJsonValue = (value: JsonValue): JsonValue => { return value; }; -const validatedJsonValue = z.custom( - (value) => jsonValueValidationIssue(value) === undefined, - { - error: (issue) => - jsonValueValidationIssue(issue.input) ?? "Invalid JSON value", - }, -); +const validatedJsonValue = z + .unknown() + .superRefine((value, context) => { + const issue = jsonValueValidationIssue(value); + if (issue !== undefined) + context.addIssue({ code: "custom", message: issue, continue: false }); + }) + .transform((value) => value as JsonValue); // Keep both stages nonrecursive in public and Standard Schema projections. validatedJsonValue._zod.toJSONSchema = () => ({}); diff --git a/src/domain/managed/managedApplicationGraphNodes.ts b/src/domain/managed/managedApplicationGraphNodes.ts index 064b6f951e..117e8f6640 100644 --- a/src/domain/managed/managedApplicationGraphNodes.ts +++ b/src/domain/managed/managedApplicationGraphNodes.ts @@ -24,6 +24,13 @@ type PinvokeImport = ManagedNativeBoundaryInspection["pinvoke_imports"][number]; type NativeImplementation = ManagedNativeBoundaryInspection["native_implementations"][number]; +/** + * Display label for a metadata name. Metadata may hold an empty name; the + * graph then has no label, while the node's properties keep the source value. + */ +const metadataLabel = (name: string): string | null => + name === "" ? null : name; + /** Options for a managed contains relationship edge. */ export interface ContainsEdgeOptions { readonly kind: string; @@ -77,7 +84,7 @@ export const addArtifactIdentityNodes = ( ), observations: [ { - label: assemblyFacts.name, + label: metadataLabel(assemblyFacts.name), properties: { name: assemblyFacts.name, version: assemblyFacts.version, @@ -111,7 +118,7 @@ export const addArtifactIdentityNodes = ( ), observations: [ { - label: module.name, + label: metadataLabel(module.name), properties: { name: module.name, mvid: module.mvid, @@ -237,7 +244,7 @@ const typeNode = ( ), observations: [ { - label: type.full_name, + label: metadataLabel(type.full_name), properties: { token: type.token, namespace: type.namespace, @@ -266,10 +273,11 @@ const methodNode = ( ), observations: [ { - label: + label: metadataLabel( method.declaring_type === null ? method.name : `${method.declaring_type}.${method.name}`, + ), properties: { token: method.token, declaring_type_token: method.declaring_type_token, @@ -302,10 +310,11 @@ const fieldNode = ( ), observations: [ { - label: + label: metadataLabel( field.declaring_type === null ? field.name : `${field.declaring_type}.${field.name}`, + ), properties: { token: field.token, declaring_type_token: field.declaring_type_token, @@ -410,7 +419,7 @@ const pinvokeNode = ( ), observations: [ { - label: pinvoke.import_name, + label: metadataLabel(pinvoke.import_name), properties: { token: pinvoke.token, member_token: pinvoke.member_token, @@ -441,7 +450,7 @@ const nativeImplementationNode = ( ), observations: [ { - label: implementation.name, + label: metadataLabel(implementation.name), properties: { token: implementation.token, name: implementation.name, diff --git a/src/domain/managed/managedArtifact.ts b/src/domain/managed/managedArtifact.ts index f41173ba6a..1988050d13 100644 --- a/src/domain/managed/managedArtifact.ts +++ b/src/domain/managed/managedArtifact.ts @@ -79,6 +79,13 @@ const customAttributeSchema = z.object({ const managedSignatureSchema = z.object({ raw_length: z.number().int().min(0), raw_sha256: digestSchema, + // Evidence recorded before this field existed reads back as null. + resolved_sha256: digestSchema + .nullable() + .default(null) + .describe( + "SHA-256 of the raw signature with each TypeDef or TypeRef operand replaced by that type's namespace-qualified name; equals raw_sha256 when the signature names no types. Unlike raw_sha256 it does not depend on build-local metadata rows. Null when the signature could not be decoded or names a type REA could not resolve, such as a TypeSpec.", + ), kind: z.enum(["method", "field", "unknown"]), parse_status: z.enum(["decoded", "partial", "unsupported", "malformed"]), calling_convention: z.string().nullable(), @@ -96,8 +103,20 @@ const managedTypeSchema = z.object({ namespace: z.string(), name: z.string(), full_name: z.string(), + name_sha256: digestSchema + .nullable() + .default(null) + .describe( + "Framed nominal namespace/name/enclosing-name digest, independent of build-local rows. Null for historical Evidence or unresolved nesting; defining assemblies and CLR runtime identity are not established.", + ), flags: z.number().int().min(0).max(0xffff_ffff), extends_token: tokenSchema.nullable(), + enclosing_type_token: tokenSchema + .nullable() + .default(null) + .describe( + "Observed enclosing TypeDef from NestedClass; null when no enclosing entry was available, which alone does not establish top-level identity.", + ), field_list: z.object({ first_row: z.number().int().min(1).nullable(), last_row: z.number().int().min(0).nullable(), @@ -438,6 +457,13 @@ export const managedArtifactInspectionSchema = z.object({ /** Provider-neutral, execution-free metadata/signature/IL member inspection. */ export const managedMemberInspectionSchema = z.object({ + signature_resolution: z + .literal("type-names-v1") + .nullable() + .default(null) + .describe( + "Signature type-name resolution was examined, including unresolved operands; null means historical Evidence without this attestation.", + ), artifact: z.object({ path: z.string().min(1), sha256: digestSchema, @@ -459,6 +485,16 @@ export const managedMemberInspectionSchema = z.object({ requires_mvid: cliMetadataGuidSchema.nullable(), }), types: z.array(managedTypeSchema), + // Member Evidence recorded before NestedClass rows were read omits this + // field; null is historical uncertainty. False means a current NestedClass + // row was unreadable or invalid, not that no nested types exist. + nesting_read: z + .boolean() + .nullable() + .default(null) + .describe( + "Whether all NestedClass rows were read and valid. Null for historical Evidence; false means current nesting is incomplete. Individual name digests can remain unresolved for missing, cyclic or over-deep enclosing chains.", + ), // Member Evidence recorded before ExportedType rows were read omits this // field; null means the table was not examined, not that it was empty. exported_types: z.array(managedExportedTypeSchema).nullable().default(null), diff --git a/src/domain/managed/managedMemberComparison.ts b/src/domain/managed/managedMemberComparison.ts index 77e382dc70..4f58590a71 100644 --- a/src/domain/managed/managedMemberComparison.ts +++ b/src/domain/managed/managedMemberComparison.ts @@ -18,6 +18,7 @@ import { buildMethodItems, } from "./managedMemberComparisonItems.js"; import { keyMembers, sha256 } from "./managedMemberComparisonMatch.js"; +import { hasSignatureResolution } from "./managedMemberSignatureIdentity.js"; import { digestSchema } from "../digests.js"; import { prefixedDigestSchema } from "../digests.js"; @@ -231,6 +232,9 @@ export const compareManagedMembers = ( rightComplete: right.result.metadata.status === "complete", leftSelection: selectedTypes(left.result), rightSelection: selectedTypes(right.result), + signatureResolutionExamined: + hasSignatureResolution(left.result) && + hasSignatureResolution(right.result), }; const methodItems = buildMethodItems(methodMatches, itemContext); const fieldItems = buildFieldItems(fieldMatches, itemContext); diff --git a/src/domain/managed/managedMemberComparisonCoverage.ts b/src/domain/managed/managedMemberComparisonCoverage.ts index 85da3b4eba..ee6021f5c7 100644 --- a/src/domain/managed/managedMemberComparisonCoverage.ts +++ b/src/domain/managed/managedMemberComparisonCoverage.ts @@ -3,6 +3,7 @@ import type { ManagedMemberComparisonSide, } from "./managedMemberComparison.js"; import type { ManagedMemberInspection } from "./managedArtifact.js"; +import { hasSignatureResolution } from "./managedMemberSignatureIdentity.js"; interface ComparisonCoverageInput { readonly left: ManagedMemberInspection; @@ -107,9 +108,28 @@ export const comparisonLimitations = ( ): string[] => { const limitations: string[] = [ "Metadata tokens are build-local coordinates; matched pairs are remaps, not persistent identities.", - "Methods pair by exact CIL/signature, then declared type, name, and signature, then structural shape; names alone are never a matching basis.", - "Fields pair by exact signature, then declared type, name, and signature; an undecoded signature is compared only by its raw bytes, and an unmatched member with one stays unknown.", + "Methods pair within one declared type and name by exact CIL/signature, then signature, then structural shape; only then may exact CIL/signature or structural shape alone remap across declared types or names, never taking a member whose own declared type and name remain unpaired on the other side or whose declaring type is unresolved. Names alone are never a matching basis.", + "Fields pair by declared type, name, and signature, then remap by exact decoded signature alone under the same rule; an undecoded signature is compared only for an exact signature match, and an unmatched member with one stays unknown.", ]; + const namesTypes = (result: ManagedMemberInspection) => + result.methods.some( + ({ signature }) => + signature.resolved_sha256 !== null && + signature.resolved_sha256 !== signature.raw_sha256, + ) || + result.fields.some( + ({ signature }) => + signature.resolved_sha256 !== null && + signature.resolved_sha256 !== signature.raw_sha256, + ); + if ( + hasSignatureResolution(left) && + hasSignatureResolution(right) && + (namesTypes(left) || namesTypes(right)) + ) + limitations.push( + "Signature type operands compare by namespace, name, and enclosing names only; defining assemblies and CLR type identity are not established by this comparison.", + ); if (left.coverage.state !== "complete" || right.coverage.state !== "complete") limitations.push("At least one managed member observation is partial."); if (!sameSelection(left, right)) @@ -120,6 +140,14 @@ export const comparisonLimitations = ( limitations.push( `Both observations cover only the selected declaring types (${left.selection.type_names.join(", ")}); member totals and statuses describe those types, not the whole assembly.`, ); + if (!hasSignatureResolution(left) || !hasSignatureResolution(right)) + limitations.push( + "At least one managed member observation does not attest signature type-name resolution, so signatures compare by raw bytes, which name types by build-local metadata row.", + ); + if (left.nesting_read === null || right.nesting_read === null) + limitations.push( + "At least one managed member observation predates enclosing-name identity. The legacy bare-name comparison does not establish the enclosing owner of a nested type.", + ); if (left.exported_types === null || right.exported_types === null) limitations.push( "At least one managed member observation predates ExportedType inspection; its exported and forwarded types are unknown.", diff --git a/src/domain/managed/managedMemberComparisonItems.ts b/src/domain/managed/managedMemberComparisonItems.ts index 26331be083..c4bd1e65e2 100644 --- a/src/domain/managed/managedMemberComparisonItems.ts +++ b/src/domain/managed/managedMemberComparisonItems.ts @@ -7,6 +7,7 @@ import { type ManagedMethodMatches, sha256, } from "./managedMemberComparisonMatch.js"; +import { signatureIdentity } from "./managedMemberSignatureIdentity.js"; type Method = ManagedMemberInspection["methods"][number]; type Field = ManagedMemberInspection["fields"][number]; @@ -29,6 +30,8 @@ interface ComparisonItemContext { /** Declaring types each side selected; null when it inspected every type. */ readonly leftSelection: ReadonlySet | null; readonly rightSelection: ReadonlySet | null; + /** Whether matching keyed signatures by resolved digests, not raw bytes. */ + readonly signatureResolutionExamined: boolean; } /** Whether an inventory observed every member of this member's declaring type. */ @@ -41,10 +44,18 @@ const inventoryCovers = ( (selection === null || (member.declaring_type !== null && selection.has(member.declaring_type))); -/** Declared type and name of each unpaired member, per side, whose signature was not decoded. */ +/** + * Declared type and name of each unpaired member, per side, whose signature + * was not decoded, or was decoded but not resolved while matching keyed + * resolved signatures. + */ interface UndecodedOneSidedNames { - readonly left: ReadonlySet; - readonly right: ReadonlySet; + readonly resolved: boolean; + readonly undecoded: Readonly>>; + readonly unresolved: Readonly>>; + readonly unknownOwners: Readonly< + Record<"left" | "right", ReadonlySet> + >; } const methodDimensions = ( @@ -57,7 +68,7 @@ const methodDimensions = ( const dimensions: MethodItem["dimensions"] = []; if (left.flags !== right.flags || left.impl_flags !== right.impl_flags) dimensions.push("metadata"); - if (left.signature.raw_sha256 !== right.signature.raw_sha256) + if (signatureIdentity(left.signature) !== signatureIdentity(right.signature)) dimensions.push("signature"); const leftBodyComplete = left.body.status === "present" && left.body.truncated_instructions === 0; @@ -135,17 +146,20 @@ const unmatchedComparison = (): UnmatchedComparison => ({ const methodOnlyItem = ( member: OneSided, + identity: string, + identityUnavailable: boolean, context: ComparisonItemContext, undecoded: UndecodedOneSidedNames, ): MethodItem => { if (member.left !== undefined) { - const limitations = absenceLimitations(member.left, { + const limitations = absenceLimitations(member.left, identity, { oppositeComplete: inventoryCovers( context.rightComplete, context.rightSelection, member.left, ), - oppositeUndecodedNames: undecoded.right, + unpaired: undecoded, + ownIdentityUnavailable: identityUnavailable, opposite: "right", inventory: "method", }); @@ -161,13 +175,14 @@ const methodOnlyItem = ( }; } - const limitations = absenceLimitations(member.right, { + const limitations = absenceLimitations(member.right, identity, { oppositeComplete: inventoryCovers( context.leftComplete, context.leftSelection, member.right, ), - oppositeUndecodedNames: undecoded.left, + unpaired: undecoded, + ownIdentityUnavailable: identityUnavailable, opposite: "left", inventory: "method", }); @@ -189,17 +204,31 @@ const methodOnlyItem = ( * decoded, so no shape round could seek a changed counterpart, or the other * inventory has an unpaired (one-sided or ambiguous) same-named member whose * signature was not decoded, which may be this member with a changed - * signature. + * signature. A decoded signature that names a type REA could not name, while + * the comparison otherwise keys resolved signatures, is compared by raw bytes + * that name types by build-local row, so it is treated the same way. */ const absenceLimitations = ( member: Method | Field, + identity: string, opposite: { readonly oppositeComplete: boolean; - readonly oppositeUndecodedNames: ReadonlySet; + readonly ownIdentityUnavailable: boolean; + readonly unpaired: UndecodedOneSidedNames; readonly opposite: "left" | "right"; readonly inventory: "method" | "field"; }, ): string[] => [ + ...(opposite.ownIdentityUnavailable + ? [ + "declaring-type-name-not-resolved: Declared type identity could not be resolved; a bare display name cannot establish a counterpart or absence.", + ] + : []), + ...(opposite.unpaired.unknownOwners[opposite.opposite].has(member.name) + ? [ + `counterpart-declaring-type-name-not-resolved: The ${opposite.opposite} ${opposite.inventory} inventory has an unpaired same-named member with unresolved enclosing-name identity, so absence was not established.`, + ] + : []), ...(opposite.oppositeComplete ? [] : [ @@ -208,62 +237,126 @@ const absenceLimitations = ( ...(member.signature.parse_status === "decoded" ? [] : [ - `signature-not-decoded: The member's ${member.signature.parse_status} signature was compared only by exact declared type, name, and raw bytes, so a changed or renamed counterpart could not be sought.`, + `signature-not-decoded: The member's ${member.signature.parse_status} signature was compared only by exact declared type, name, and signature bytes, so a changed or renamed counterpart could not be sought.`, ]), ...(member.signature.parse_status === "decoded" && - opposite.oppositeUndecodedNames.has(declaredName(member)) + opposite.unpaired.undecoded[opposite.opposite].has(identity) ? [ `counterpart-signature-not-decoded: The ${opposite.opposite} ${opposite.inventory} inventory has an unpaired member with the same declared type and name whose signature was not decoded, so it may be this member with a changed signature.`, ] : []), + ...(unresolved(member, opposite.unpaired.resolved) + ? [ + `signature-not-resolved: The member's signature names a type REA could not name, so it was compared by raw bytes, which name types by build-local metadata row; a counterpart whose referenced type moved rows could not be found.`, + ] + : []), + ...(member.signature.parse_status === "decoded" && + !unresolved(member, opposite.unpaired.resolved) && + opposite.unpaired.unresolved[opposite.opposite].has(identity) + ? [ + `counterpart-signature-not-resolved: The ${opposite.opposite} ${opposite.inventory} inventory has an unpaired member with the same declared type and name whose signature names a type REA could not name, so it may be this member.`, + ] + : []), ]; -const declaredName = (member: Method | Field): string => - JSON.stringify([member.declaring_type, member.name]); +/** A known signature form whose type operands still compare by raw bytes. */ +const unresolved = (member: Method | Field, resolved: boolean): boolean => + resolved && + member.signature.kind !== "unknown" && + member.signature.resolved_sha256 === null; + +/** Equal build-local bytes cannot establish equal unresolved type operands. */ +const pairedSignatureLimitations = ( + left: Method | Field, + right: Method | Field, + examined: boolean, +): string[] => + unresolved(left, examined) || unresolved(right, examined) + ? [ + "signature-not-resolved: At least one signature names a type REA could not resolve; equal build-local row bytes do not establish equivalent signature types.", + ] + : []; /** * Declared names of unpaired members whose signature was not decoded: one-sided * members and candidates of ambiguous groups, which also remain unpaired. */ -const undecodedOneSidedNames = (members: { - readonly leftOnly: readonly { readonly item: Method | Field }[]; - readonly rightOnly: readonly { readonly item: Method | Field }[]; - readonly ambiguous: readonly { - readonly left: readonly { readonly item: Method | Field }[]; - readonly right: readonly { readonly item: Method | Field }[]; - }[]; -}): UndecodedOneSidedNames => { - const names = (side: readonly { readonly item: Method | Field }[]) => - new Set( - side - .filter(({ item }) => item.signature.parse_status !== "decoded") - .map(({ item }) => declaredName(item)), - ); - return { - left: names([ +interface Unpaired { + readonly item: Method | Field; + readonly identity: string; + readonly identityUnavailable: boolean; +} + +const undecodedOneSidedNames = ( + members: { + readonly leftOnly: readonly Unpaired[]; + readonly rightOnly: readonly Unpaired[]; + readonly ambiguous: readonly { + readonly left: readonly Unpaired[]; + readonly right: readonly Unpaired[]; + }[]; + }, + resolved: boolean, +): UndecodedOneSidedNames => { + const sides = { + left: [ ...members.leftOnly, ...members.ambiguous.flatMap(({ left }) => left), - ]), - right: names([ + ], + right: [ ...members.rightOnly, ...members.ambiguous.flatMap(({ right }) => right), - ]), + ], + }; + const names = ( + side: readonly Unpaired[], + keep: (item: Method | Field) => boolean, + ) => + new Set( + side.filter(({ item }) => keep(item)).map(({ identity }) => identity), + ); + const undecoded = (item: Method | Field) => + item.signature.parse_status !== "decoded"; + const notResolved = (item: Method | Field) => unresolved(item, resolved); + return { + resolved, + unknownOwners: { + left: new Set( + sides.left.filter((x) => x.identityUnavailable).map((x) => x.item.name), + ), + right: new Set( + sides.right + .filter((x) => x.identityUnavailable) + .map((x) => x.item.name), + ), + }, + undecoded: { + left: names(sides.left, undecoded), + right: names(sides.right, undecoded), + }, + unresolved: { + left: names(sides.left, notResolved), + right: names(sides.right, notResolved), + }, }; }; const fieldOnlyItem = ( member: OneSided, + identity: string, + identityUnavailable: boolean, context: ComparisonItemContext, undecoded: UndecodedOneSidedNames, ): FieldItem => { if (member.left !== undefined) { - const limitations = absenceLimitations(member.left, { + const limitations = absenceLimitations(member.left, identity, { oppositeComplete: inventoryCovers( context.rightComplete, context.rightSelection, member.left, ), - oppositeUndecodedNames: undecoded.right, + unpaired: undecoded, + ownIdentityUnavailable: identityUnavailable, opposite: "right", inventory: "field", }); @@ -278,13 +371,14 @@ const fieldOnlyItem = ( }; } - const limitations = absenceLimitations(member.right, { + const limitations = absenceLimitations(member.right, identity, { oppositeComplete: inventoryCovers( context.leftComplete, context.leftSelection, member.right, ), - oppositeUndecodedNames: undecoded.left, + unpaired: undecoded, + ownIdentityUnavailable: identityUnavailable, opposite: "left", inventory: "field", }); @@ -300,18 +394,23 @@ const fieldOnlyItem = ( }; /** - * Name the identity key an ambiguous group shares. The exact-signature key - * already includes the declared type and name, so names cannot separate it. + * Name only evidence shared by the whole group. Exact-signature ambiguities + * include both declared-identity collisions and unresolved signature choices. */ const ambiguityLimitation = ( members: "methods" | "fields", basis: MethodItem["match"]["basis"], + identityUnavailable: boolean, ): string => - basis === "exact-signature" - ? `Multiple managed ${members} share the same declared type, name, and raw signature; REA did not guess a token remap.` - : members === "methods" - ? "Multiple managed methods share the same non-name identity key; REA did not guess a token remap." - : "Multiple managed fields share the same signature; REA did not guess a token remap from names."; + identityUnavailable + ? "declaring-type-name-not-resolved: Declared type identity remains unresolved; REA did not guess a counterpart from display names, signatures or CIL." + : basis === "exact-signature" + ? `Multiple managed ${members} share the same signature; REA did not guess declared ownership or a token remap.` + : basis === "exact-il-signature" + ? "Multiple managed methods share the same CIL and signature; REA did not guess a token remap." + : members === "methods" + ? "Multiple managed methods share the same structural shape; REA did not guess a token remap." + : "Multiple managed fields share the same signature; REA did not guess a token remap."; export const buildMethodItems = ( matches: ManagedMethodMatches, @@ -319,6 +418,11 @@ export const buildMethodItems = ( ): MethodItem[] => { const items: MethodItem[] = []; for (const pair of matches.pairs) { + const signatureLimitations = pairedSignatureLimitations( + pair.left.item, + pair.right.item, + context.signatureResolutionExamined, + ); const { dimensions: observedDimensions, bodyUnknown } = methodDimensions( pair.left.item, pair.right.item, @@ -335,7 +439,7 @@ export const buildMethodItems = ( })}`, status: observedChange ? "changed" - : bodyUnknown + : bodyUnknown || signatureLimitations.length > 0 ? "unknown" : "unchanged", left: methodIdentity(pair.left.item), @@ -349,11 +453,14 @@ export const buildMethodItems = ( }, dimensions, evidence_links: [context.leftEvidenceId, context.rightEvidenceId], - limitations: bodyUnknown - ? [ - "Method body facets are unknown because at least one side has unavailable or partial body data.", - ] - : [], + limitations: [ + ...signatureLimitations, + ...(bodyUnknown + ? [ + "Method body facets are unknown because at least one side has unavailable or partial body data.", + ] + : []), + ], }); } for (const ambiguous of matches.ambiguous) { @@ -375,14 +482,40 @@ export const buildMethodItems = ( }, dimensions: ["availability"], evidence_links: [context.leftEvidenceId, context.rightEvidenceId], - limitations: [ambiguityLimitation("methods", ambiguous.basis)], + limitations: [ + ambiguityLimitation( + "methods", + ambiguous.basis, + ambiguous.left.some((item) => item.identityUnavailable) || + ambiguous.right.some((item) => item.identityUnavailable), + ), + ], }); } - const undecoded = undecodedOneSidedNames(matches); + const undecoded = undecodedOneSidedNames( + matches, + context.signatureResolutionExamined, + ); for (const item of matches.leftOnly) - items.push(methodOnlyItem({ left: item.item }, context, undecoded)); + items.push( + methodOnlyItem( + { left: item.item }, + item.identity, + item.identityUnavailable, + context, + undecoded, + ), + ); for (const item of matches.rightOnly) - items.push(methodOnlyItem({ right: item.item }, context, undecoded)); + items.push( + methodOnlyItem( + { right: item.item }, + item.identity, + item.identityUnavailable, + context, + undecoded, + ), + ); return items; }; @@ -392,16 +525,25 @@ export const buildFieldItems = ( ): FieldItem[] => { const items: FieldItem[] = []; for (const pair of matches.pairs) { + const signatureLimitations = pairedSignatureLimitations( + pair.left.item, + pair.right.item, + context.signatureResolutionExamined, + ); const changed = - pair.left.item.signature.raw_sha256 !== - pair.right.item.signature.raw_sha256 || + signatureIdentity(pair.left.item.signature) !== + signatureIdentity(pair.right.item.signature) || pair.left.item.flags !== pair.right.item.flags; items.push({ item_id: `mmc_field_${sha256({ left: pair.left.item.token, right: pair.right.item.token, })}`, - status: changed ? "changed" : "unchanged", + status: changed + ? "changed" + : signatureLimitations.length > 0 + ? "unknown" + : "unchanged", left: fieldIdentity(pair.left.item), right: fieldIdentity(pair.right.item), match: { @@ -412,7 +554,7 @@ export const buildFieldItems = ( candidate_right_tokens: [], }, evidence_links: [context.leftEvidenceId, context.rightEvidenceId], - limitations: [], + limitations: signatureLimitations, }); } for (const ambiguous of matches.ambiguous) @@ -432,12 +574,38 @@ export const buildFieldItems = ( candidate_right_tokens: ambiguous.right.map(({ item }) => item.token), }, evidence_links: [context.leftEvidenceId, context.rightEvidenceId], - limitations: [ambiguityLimitation("fields", ambiguous.basis)], + limitations: [ + ambiguityLimitation( + "fields", + ambiguous.basis, + ambiguous.left.some((item) => item.identityUnavailable) || + ambiguous.right.some((item) => item.identityUnavailable), + ), + ], }); - const undecoded = undecodedOneSidedNames(matches); + const undecoded = undecodedOneSidedNames( + matches, + context.signatureResolutionExamined, + ); for (const item of matches.leftOnly) - items.push(fieldOnlyItem({ left: item.item }, context, undecoded)); + items.push( + fieldOnlyItem( + { left: item.item }, + item.identity, + item.identityUnavailable, + context, + undecoded, + ), + ); for (const item of matches.rightOnly) - items.push(fieldOnlyItem({ right: item.item }, context, undecoded)); + items.push( + fieldOnlyItem( + { right: item.item }, + item.identity, + item.identityUnavailable, + context, + undecoded, + ), + ); return items; }; diff --git a/src/domain/managed/managedMemberComparisonMatch.ts b/src/domain/managed/managedMemberComparisonMatch.ts index 90106a17cd..aa932500f9 100644 --- a/src/domain/managed/managedMemberComparisonMatch.ts +++ b/src/domain/managed/managedMemberComparisonMatch.ts @@ -11,6 +11,10 @@ import type { ManagedMemberComparisonResult, ManagedMemberComparisonSide, } from "./managedMemberComparison.js"; +import { + comparableSides, + signatureIdentity, +} from "./managedMemberSignatureIdentity.js"; import type { JsonValue } from "../jsonValue.js"; export const sha256 = (value: JsonValue): string => { @@ -22,14 +26,21 @@ export const sha256 = (value: JsonValue): string => { type Method = ManagedMemberInspection["methods"][number]; type Field = ManagedMemberInspection["fields"][number]; + type MatchBasis = ManagedMemberComparisonResult["methods"][number]["match"]["basis"]; type ConcreteMatchBasis = Exclude; interface Keyed { readonly item: Item; + /** Declared type and name; an unresolved type is null, the name remains. */ + readonly identity: string; + /** Declared identity could not be resolved; display facts remain. */ + readonly identityUnavailable: boolean; readonly exactKey: string | null; readonly signatureKey: string | null; + /** The name-free exact signature, resolved where both sides resolve it. */ + readonly signatureId: string; readonly structuralKey: string | null; } @@ -46,45 +57,52 @@ interface Ambiguous { readonly basis: ConcreteMatchBasis; } -const keyMethod = (item: Method): Keyed => ({ +/** A declared type: a historical bare name or a tagged observed nominal digest. */ +type DeclaredName = string | string[] | null; +type DeclaredNameOf = (item: Method | Field) => DeclaredName; +const keyMethod = (item: Method, declared: DeclaredNameOf): Keyed => ({ item, + identity: identityOf({ declaring_type: declared(item), name: item.name }), + identityUnavailable: declared(item) === null, + signatureId: signatureIdentity(item.signature), exactKey: item.body.status === "present" && item.body.il_sha256 !== null && item.body.normalized_il_sha256 !== null ? sha256([ "method-exact", - item.signature.raw_sha256, + signatureIdentity(item.signature), item.body.il_sha256, item.body.normalized_il_sha256, ]) : null, - // The raw blob is the exact signature; decoding only feeds shape keys. - signatureKey: sha256([ - "method-signature", - item.declaring_type, - item.name, - item.signature.raw_sha256, - ]), + // The blob, with type operands named, is the exact signature; decoding + // only gates the shape keys. + signatureKey: + declared(item) === null + ? null + : sha256([ + "method-signature", + declared(item), + item.name, + signatureIdentity(item.signature), + ]), structuralKey: item.signature.parse_status === "decoded" && item.body.status === "present" ? sha256([ "method-structural", - item.signature.kind, - item.signature.calling_convention, - item.signature.generic_parameter_count, - item.signature.parameter_count, - item.signature.return_type, - item.signature.parameter_types, + // Decoded type strings carry build-local rows; the identity names them. + signatureIdentity(item.signature), item.body.status, item.body.header_format, item.body.il_size, item.body.max_stack, item.body.init_locals, item.body.opcode_counts, - item.body.anchors.map(({ opcode, operand_kind }) => [ + item.body.anchors.map(({ opcode, operand_kind, operand }) => [ opcode, operand_kind, + operand_kind === "constant" ? operand : null, ]), item.body.exception_regions.map((region) => [ region.flags, @@ -97,20 +115,26 @@ const keyMethod = (item: Method): Keyed => ({ : null, }); -const keyField = (item: Field): Keyed => ({ +const keyField = (item: Field, declared: DeclaredNameOf): Keyed => ({ item, + identity: identityOf({ declaring_type: declared(item), name: item.name }), + identityUnavailable: declared(item) === null, + signatureId: signatureIdentity(item.signature), exactKey: item.signature.parse_status === "decoded" - ? sha256(["field-exact", item.signature.raw_sha256]) + ? sha256(["field-exact", signatureIdentity(item.signature)]) : null, // Only an undecoded field reaches this round: a decoded field with the - // same raw signature already met its counterpart in the exact round. - signatureKey: sha256([ - "field-signature", - item.declaring_type, - item.name, - item.signature.raw_sha256, - ]), + // same signature already met its counterpart in the exact round. + signatureKey: + declared(item) === null + ? null + : sha256([ + "field-signature", + declared(item), + item.name, + signatureIdentity(item.signature), + ]), structuralKey: sha256([ "field-structural", item.signature.kind, @@ -126,15 +150,34 @@ const matchMethods = ( matchByKeys({ left, right, - exactBasis: "exact-il-signature", - fallbackBases: [ + rounds: [ + { + basis: "exact-il-signature", + key: declaredAs(({ exactKey }) => exactKey), + }, { basis: "exact-signature", key: ({ signatureKey }) => signatureKey, + competitionKey: ({ signatureId }) => signatureId, + }, + { + basis: "structural-method-shape", + // An unresolved owner has no declared shape match to prove. + key: declaredAs((keyed) => + keyed.identityUnavailable ? null : keyed.structuralKey, + ), + competitionKey: ({ structuralKey }) => structuralKey, + }, + { + basis: "exact-il-signature", + key: ({ exactKey }) => exactKey, + remap: true, }, { basis: "structural-method-shape", key: ({ structuralKey }) => structuralKey, + competitionKey: ({ structuralKey }) => structuralKey, + remap: true, }, ], }); @@ -146,11 +189,20 @@ const matchFields = ( matchByKeys({ left, right, - exactBasis: "field-signature", - fallbackBases: [ + rounds: [ + { + basis: "field-signature", + key: declaredAs(({ exactKey }) => exactKey), + }, { basis: "exact-signature", key: ({ signatureKey }) => signatureKey, + competitionKey: ({ signatureId }) => signatureId, + }, + { + basis: "field-signature", + key: ({ exactKey }) => exactKey, + remap: true, }, ], }); @@ -158,21 +210,49 @@ const matchFields = ( export type ManagedMethodMatches = ReturnType; export type ManagedFieldMatches = ReturnType; +interface MatchRound { + readonly basis: ConcreteMatchBasis; + readonly key: (item: Keyed) => string | null; + /** Name-free evidence paired with the retained member name for contesting. */ + readonly competitionKey?: (item: Keyed) => string | null; + /** + * The key omits the declared type, so a pair may join different declared + * identities; same-identity pairs were taken by earlier rounds. Contention + * still retains the member name, which unresolved owners keep as evidence. + * A member whose own identity is still unpaired on the other side does not + * take part, and a group with a member of an unresolved type stays ambiguous. + */ + readonly remap?: true; +} + interface MatchByKeysInput { readonly left: readonly Keyed[]; readonly right: readonly Keyed[]; - readonly exactBasis: ConcreteMatchBasis; - readonly fallbackBases: readonly { - readonly basis: ConcreteMatchBasis; - readonly key: (item: Keyed) => string | null; - }[]; + readonly rounds: readonly MatchRound[]; } -const matchByKeys = ({ +export const identityOf = ({ + declaring_type, + name, +}: { + readonly declaring_type: DeclaredName; + readonly name: string; +}): string => sha256([declaring_type, name]); + +/** Narrow a key to members of one declared type and name. */ +const declaredAs = + (key: (item: Keyed) => string | null) => + (item: Keyed): string | null => { + const value = key(item); + return value === null || item.identityUnavailable + ? null + : sha256([item.identity, value]); + }; + +const matchByKeys = ({ left, right, - exactBasis, - fallbackBases, + rounds, }: MatchByKeysInput): { readonly pairs: readonly MatchedPair[]; readonly ambiguous: readonly Ambiguous[]; @@ -181,33 +261,58 @@ const matchByKeys = ({ } => { const usedLeft = new Set>(); const usedRight = new Set>(); + const pairedLeft = new Set>(); + const pairedRight = new Set>(); const pairs: MatchedPair[] = []; const ambiguous: Ambiguous[] = []; - const rounds: readonly { - readonly basis: ConcreteMatchBasis; - readonly key: (item: Keyed) => string | null; - }[] = [ - { basis: exactBasis, key: ({ exactKey }) => exactKey }, - ...fallbackBases, - ]; for (const round of rounds) { + const competitionKey = round.competitionKey ?? ((item) => item.exactKey); + const unpairedLeft = unpairedOf(left, pairedLeft); + const unpairedRight = unpairedOf(right, pairedRight); + const declarable = declaredCandidates( + unpairedLeft, + unpairedRight, + competitionKey, + ); + const openLeft = left.filter((item) => !usedLeft.has(item)); + const openRight = right.filter((item) => !usedRight.has(item)); + // Only open candidates can justify overriding a held declared identity: + // consumed ambiguous rows remain identity barriers, but cannot join a + // later remap group to establish its ambiguity. + const remapContested = + round.remap === true + ? ownerlessContested([openLeft, openRight], competitionKey) + : new Set(); const leftGroups = groupBy( - left.filter((item) => !usedLeft.has(item)), + round.remap === true + ? remappable(openLeft, unpairedRight, competitionKey, remapContested) + : openLeft.filter(declarable), round.key, ); const rightGroups = groupBy( - right.filter((item) => !usedRight.has(item)), + round.remap === true + ? remappable(openRight, unpairedLeft, competitionKey, remapContested) + : openRight.filter(declarable), round.key, ); for (const [key, leftItems] of leftGroups) { const rightItems = rightGroups.get(key); if (rightItems === undefined) continue; - if (leftItems.length === 1 && rightItems.length === 1) { + // An unresolved owner may be either member's own counterpart, so a + // remap group it joins stays ambiguous. + const refused = + round.remap === true && + [...leftItems, ...rightItems].some( + ({ identityUnavailable }) => identityUnavailable, + ); + if (!refused && leftItems.length === 1 && rightItems.length === 1) { const [leftItem] = leftItems; const [rightItem] = rightItems; if (leftItem === undefined || rightItem === undefined) continue; usedLeft.add(leftItem); usedRight.add(rightItem); + pairedLeft.add(leftItem); + pairedRight.add(rightItem); pairs.push({ left: leftItem, right: rightItem, @@ -230,6 +335,22 @@ const matchByKeys = ({ } } } + // Contention can prevent a declared signature pair even when CIL/shape + // evidence cannot form a later remap. Held ambiguities still contest + // ownership; only proven pairs leave that evidence pool. Group only open + // candidates so held members retain their original ambiguity and provenance. + for (const group of unresolvedSignatureAmbiguities( + left.filter((item) => !usedLeft.has(item)), + right.filter((item) => !usedRight.has(item)), + ownerlessContested( + [unpairedOf(left, pairedLeft), unpairedOf(right, pairedRight)], + (member) => member.signatureId, + ), + )) { + for (const item of group.left) usedLeft.add(item); + for (const item of group.right) usedRight.add(item); + ambiguous.push(group); + } return { pairs, ambiguous, @@ -238,6 +359,111 @@ const matchByKeys = ({ }; }; +/** Open signature choices contested by unresolved owners, including held rows. */ +const unresolvedSignatureAmbiguities = ( + left: readonly Keyed[], + right: readonly Keyed[], + contested: ReadonlySet, +): Ambiguous[] => { + const key = (member: Keyed): string => + competitionIdentityKey(member, member.signatureId) ?? ""; + const leftGroups = groupBy(left, key); + const rightGroups = groupBy(right, key); + const groups: Ambiguous[] = []; + for (const [signature, leftItems] of leftGroups) { + const rightItems = rightGroups.get(signature); + if (!contested.has(signature) || rightItems === undefined) continue; + groups.push({ + left: leftItems, + right: rightItems, + basis: "exact-signature", + }); + } + return groups; +}; + +interface Declared { + readonly declaring_type: string | null; + readonly name: string; +} + +const unpairedOf = ( + items: readonly Keyed[], + paired: ReadonlySet>, +): Keyed[] => items.filter((item) => !paired.has(item)); + +/** + * Round-specific evidence keys an unresolved owner competes for, paired with + * its retained member name. Another same-name member on the same side can be + * the unresolved row's owner; a different name cannot. Contested candidates + * stay in remap rounds rather than pairing by declared identity. + */ +const competitionIdentityKey = ( + member: Keyed, + evidenceKey: string | null, +): string | null => + evidenceKey === null ? null : JSON.stringify([member.item.name, evidenceKey]); + +const ownerlessContested = ( + sides: readonly (readonly Keyed[])[], + keyOf: (item: Keyed) => string | null, +): ReadonlySet => { + const contested = new Set(); + for (const side of sides) { + const seen = new Map(); + const ownerless = new Set(); + for (const member of side) { + const key = competitionIdentityKey(member, keyOf(member)); + if (key === null) continue; + seen.set(key, (seen.get(key) ?? 0) + 1); + if (member.identityUnavailable) ownerless.add(key); + } + for (const key of ownerless) + if ((seen.get(key) ?? 0) > 1) contested.add(key); + } + return contested; +}; + +/** + * Members a remap round may group: an unresolved owner, which keeps its group + * ambiguous; a member whose selected key an unresolved owner competes for, so the + * group that preserved the base matcher's ambiguity can form; or no member of + * the same declared identity left unpaired on the other side, whether + * one-sided or in an ambiguous group. The sets are taken before grouping, so + * no group order decides a remap. + */ +const remappable = ( + items: readonly Keyed[], + others: readonly Keyed[], + keyOf: (item: Keyed) => string | null, + contested: ReadonlySet, +): Keyed[] => { + const kept = new Set(others.map(({ identity }) => identity)); + return items.filter((member) => { + const key = keyOf(member); + return ( + (key !== null && + contested.has(competitionIdentityKey(member, key) ?? "")) || + member.identityUnavailable || + !kept.has(member.identity) + ); + }); +}; + +/** A declared round needs a resolved owner and no unresolved contender. */ +const declaredCandidates = ( + left: readonly Keyed[], + right: readonly Keyed[], + keyOf: (item: Keyed) => string | null, +): ((member: Keyed) => boolean) => { + const contested = ownerlessContested([left, right], keyOf); + return (member) => { + if (member.identityUnavailable) return false; + const key = competitionIdentityKey(member, keyOf(member)); + return key === null || !contested.has(key); + }; +}; + const groupBy = ( items: readonly Item[], keyOf: (item: Item) => string | null, @@ -266,7 +492,7 @@ const observedBy = * the other side's selection stays one-sided, so a type selection never yields * a remap to a member of a type the other side did not inspect. */ -const matchObserved = ( +const matchObserved = ( left: readonly Item[], right: readonly Item[], sides: { @@ -277,47 +503,69 @@ const matchObserved = ( left: readonly Keyed[], right: readonly Keyed[], ) => ReturnType>, - key: (item: Item) => Keyed, + keyLeft: (item: Item) => Keyed, + keyRight: (item: Item) => Keyed, ): ReturnType> => { const matched = match( - left.filter(sides.inRight).map(key), - right.filter(sides.inLeft).map(key), + left.filter(sides.inRight).map(keyLeft), + right.filter(sides.inLeft).map(keyRight), ); return { ...matched, leftOnly: [ ...matched.leftOnly, - ...left.filter((item) => !sides.inRight(item)).map(key), + ...left.filter((item) => !sides.inRight(item)).map(keyLeft), ], rightOnly: [ ...matched.rightOnly, - ...right.filter((item) => !sides.inLeft(item)).map(key), + ...right.filter((item) => !sides.inLeft(item)).map(keyRight), ], }; }; export const keyMembers = ( - left: ManagedMemberComparisonSide, - right: ManagedMemberComparisonSide, + observedLeft: ManagedMemberComparisonSide, + observedRight: ManagedMemberComparisonSide, ): { readonly methodMatches: ReturnType; readonly fieldMatches: ReturnType; } => { + const [left, right] = comparableSides(observedLeft, observedRight); const sides = { inLeft: observedBy(left), inRight: observedBy(right) }; + const modern = + left.result.nesting_read !== null && right.result.nesting_read !== null; + const declared = (inspection: ManagedMemberInspection): DeclaredNameOf => { + const names = new Map( + inspection.types.map((type) => [type.token, type.name_sha256]), + ); + return (item) => { + if (inspection.nesting_read === false) return null; + const digest = names.get(item.declaring_type_token ?? ""); + // A historical counterpart cannot repair an observed current name gap. + if (inspection.nesting_read !== null && digest == null) return null; + // Historical observations retain their explicitly limited bare-name mode. + if (!modern) return item.declaring_type; + return digest == null ? null : ["nominal-name-sha256", digest]; + }; + }; + const declaredLeft = declared(left.result); + const declaredRight = declared(right.result); return { methodMatches: matchObserved( left.result.methods, right.result.methods, sides, matchMethods, - keyMethod, + (item) => keyMethod(item, declaredLeft), + (item) => keyMethod(item, declaredRight), ), fieldMatches: matchObserved( left.result.fields, right.result.fields, sides, matchFields, - keyField, + (item) => keyField(item, declaredLeft), + (item) => keyField(item, declaredRight), ), }; }; diff --git a/src/domain/managed/managedMemberSignatureIdentity.test.ts b/src/domain/managed/managedMemberSignatureIdentity.test.ts new file mode 100644 index 0000000000..a07c2c1c1c --- /dev/null +++ b/src/domain/managed/managedMemberSignatureIdentity.test.ts @@ -0,0 +1,27 @@ +import { describe, expect, it } from "vitest"; +import { signatureIdentity } from "./managedMemberSignatureIdentity.js"; + +type Signature = Parameters[0]; + +const digest = "ab".repeat(32); +const signature = (raw: string, resolved: string | null) => + ({ raw_sha256: raw, resolved_sha256: resolved }) as Signature; + +describe("managed signature identity", () => { + it("keeps a raw digest apart from an equal resolved digest", () => { + // An undecodable blob can hold another signature's resolved preimage, so + // its raw digest can equal that signature's resolved digest. + expect(signatureIdentity(signature(digest, null))).not.toBe( + signatureIdentity(signature("cd".repeat(32), digest)), + ); + }); + + it("matches resolved digests and raw digests among themselves", () => { + expect(signatureIdentity(signature("01".repeat(32), digest))).toBe( + signatureIdentity(signature("02".repeat(32), digest)), + ); + expect(signatureIdentity(signature(digest, null))).toBe( + signatureIdentity(signature(digest, null)), + ); + }); +}); diff --git a/src/domain/managed/managedMemberSignatureIdentity.ts b/src/domain/managed/managedMemberSignatureIdentity.ts new file mode 100644 index 0000000000..e683aa7c99 --- /dev/null +++ b/src/domain/managed/managedMemberSignatureIdentity.ts @@ -0,0 +1,64 @@ +import type { ManagedMemberInspection } from "./managedArtifact.js"; +import type { ManagedMemberComparisonSide } from "./managedMemberComparison.js"; + +type Method = ManagedMemberInspection["methods"][number]; +type Field = ManagedMemberInspection["fields"][number]; + +/** + * The exact signature across builds: raw bytes with type operands resolved to + * names. Raw bytes name types by build-local rows, so they stand in only when + * REA could not resolve the operands, or for Evidence recorded before it did. + * The kind is part of the key: an undecodable blob can hold another + * signature's resolved preimage, so its raw digest can equal that resolved + * digest without the two being the same signature. + */ +export const signatureIdentity = ({ + raw_sha256, + resolved_sha256, +}: Method["signature"]): string => + resolved_sha256 === null + ? `raw:${raw_sha256}` + : `resolved:${resolved_sha256}`; + +/** + * The producer attests examination independently of whether it could resolve + * any operands. A current inventory of only unresolved types is not legacy. + */ +export const hasSignatureResolution = ( + result: ManagedMemberInspection, +): boolean => + result.signature_resolution !== null || + (result.methods.length === 0 && result.fields.length === 0); + +const withoutResolution = ( + members: readonly Member[], +): Member[] => + members.map((member) => ({ + ...member, + signature: { ...member.signature, resolved_sha256: null }, + })); + +/** + * Compare raw signatures on both sides when either observation predates + * resolved digests; a resolved digest never equals a raw one for a signature + * that names a type, so mixing them would lose pairs a raw comparison finds. + */ +export const comparableSides = ( + left: ManagedMemberComparisonSide, + right: ManagedMemberComparisonSide, +): readonly [ManagedMemberComparisonSide, ManagedMemberComparisonSide] => { + if ( + hasSignatureResolution(left.result) && + hasSignatureResolution(right.result) + ) + return [left, right]; + const raw = (side: ManagedMemberComparisonSide) => ({ + ...side, + result: { + ...side.result, + methods: withoutResolution(side.result.methods), + fields: withoutResolution(side.result.fields), + }, + }); + return [raw(left), raw(right)]; +}; diff --git a/src/domain/managed/managedNativeVerificationMatch.ts b/src/domain/managed/managedNativeVerificationMatch.ts index bbe886a708..49cb0cdf34 100644 --- a/src/domain/managed/managedNativeVerificationMatch.ts +++ b/src/domain/managed/managedNativeVerificationMatch.ts @@ -124,13 +124,21 @@ const moduleCompatible = ( scope: string | null, symbol: NativeSymbol, ): boolean => { - if (scope === null) return true; + if (scope === null || scope.trim().length === 0) return false; const expected = normalizeModule(scope); return [symbol.module_name, symbol.module_path] - .filter((value): value is string => value !== null) + .filter( + (value): value is string => + value !== null && normalizeModule(value).length > 0, + ) .some((value) => nativeModuleIdentities(value).has(expected)); }; +const hasModuleIdentity = (symbol: NativeSymbol): boolean => + [symbol.module_name, symbol.module_path].some( + (value) => value !== null && normalizeModule(value).length > 0, + ); + const candidateNames = ( managed: Pick< PinvokeImport, @@ -190,6 +198,9 @@ const matchPinvoke = ( }; const candidates: [NativeSymbol, ...NativeSymbol[]] = [first, ...remaining]; + const scopeKnown = + managed.import_scope_name !== null && + managed.import_scope_name.trim().length > 0; const exact = candidates.find( (symbol) => managed.import_name === symbol.name && @@ -198,17 +209,13 @@ const matchPinvoke = ( const decorated = candidates.find((symbol) => moduleCompatible(managed.import_scope_name, symbol), ); - const selected = exact ?? decorated ?? first; - if (!moduleCompatible(managed.import_scope_name, selected)) - return { - match: { - status: "contradicted", - basis: "module-mismatch", - confidence: "unknown", - matched_native: selected, - candidates, - }, - }; + const selected = + exact ?? + decorated ?? + (scopeKnown + ? candidates.find((symbol) => !hasModuleIdentity(symbol)) + : undefined) ?? + first; if (exact !== undefined) return { match: { @@ -222,11 +229,31 @@ const matchPinvoke = ( candidates, }, }; + if (decorated !== undefined) + return { + match: { + basis: "decorated-name-candidate", + status: "inferred", + confidence: "inferred", + matched_native: selected, + candidates, + }, + }; + if (!scopeKnown || !hasModuleIdentity(selected)) + return { + match: { + status: "unresolved", + basis: "module-identity-unknown", + confidence: "unknown", + matched_native: selected, + candidates, + }, + }; return { match: { - basis: "decorated-name-candidate", - status: "inferred", - confidence: "inferred", + status: "contradicted", + basis: "module-mismatch", + confidence: "unknown", matched_native: selected, candidates, }, @@ -234,26 +261,31 @@ const matchPinvoke = ( }; const pinvokeLimitations = ( - status: PinvokeVerification["status"], + match: PinvokeMatch, managed: PinvokeImport, ): readonly string[] => [ - ...(managed.import_scope_name === null + ...(managed.import_scope_name === null || + managed.import_scope_name.trim().length === 0 ? ["Managed declaration does not name an import scope/module."] : []), - ...(status === "inferred" + ...(match.status === "inferred" ? [ "Native symbol matched a decorated candidate rather than the exact declared import name.", ] : []), - ...(status === "contradicted" + ...(match.status === "contradicted" ? [ "A symbol name candidate exists, but the supplied native Evidence module identity does not match the managed import scope.", ] : []), - ...(status === "unresolved" - ? [ - "No supplied native export/function Evidence matched this declared import.", - ] + ...(match.status === "unresolved" + ? match.basis === "module-identity-unknown" + ? [ + "A native name candidate exists, but a required module identity is unavailable; module correspondence is unknown.", + ] + : [ + "No supplied native export/function Evidence matched this declared import.", + ] : []), ]; @@ -283,7 +315,7 @@ export const verifyPinvoke = ({ declaration_verification: item.verification, }; const selection = matchPinvoke(item, native.symbols, native.accepted); - const limitations = pinvokeLimitations(selection.match.status, item); + const limitations = pinvokeLimitations(selection.match, item); return { verification: pinvokeVerificationSchema.parse({ item_id: `mnv_pinvoke_${digest({ diff --git a/src/domain/managed/managedNativeVerificationSchemas.ts b/src/domain/managed/managedNativeVerificationSchemas.ts index 16afe4048a..320842a75c 100644 --- a/src/domain/managed/managedNativeVerificationSchemas.ts +++ b/src/domain/managed/managedNativeVerificationSchemas.ts @@ -48,10 +48,10 @@ const pinvokeVerificationContextShape = { }; const observedCandidatesSchema = z - .tuple([nativeSymbolSchema]) - .rest(nativeSymbolSchema); + .tuple([nativeSymbolSchema], nativeSymbolSchema) + .check(z.minLength(1)); -export const pinvokeVerificationSchema = z.discriminatedUnion("status", [ +export const pinvokeVerificationSchema = z.union([ z.strictObject({ ...pinvokeVerificationContextShape, status: z.literal("verified"), @@ -76,6 +76,14 @@ export const pinvokeVerificationSchema = z.discriminatedUnion("status", [ matched_native: z.null(), candidates: emptyArraySchema, }), + z.strictObject({ + ...pinvokeVerificationContextShape, + status: z.literal("unresolved"), + basis: z.literal("module-identity-unknown"), + confidence: z.literal("unknown"), + matched_native: nativeSymbolSchema, + candidates: observedCandidatesSchema, + }), z.strictObject({ ...pinvokeVerificationContextShape, status: z.literal("contradicted"), diff --git a/src/domain/mobileApplicationGraphProjection.ts b/src/domain/mobileApplicationGraphProjection.ts index 13bbe9545c..b143e44034 100644 --- a/src/domain/mobileApplicationGraphProjection.ts +++ b/src/domain/mobileApplicationGraphProjection.ts @@ -32,6 +32,16 @@ export const projectedComponentSchema = z.strictObject({ format: z.string().min(1), }); +/** Inventoried paths whose content identity is unavailable, retained as unknown. */ +export const unavailableComponentSchema = z.strictObject({ + path: projectionPathSchema, + artifact_id: z.null(), + sha256: z.null(), + format: z.string().min(1), + encrypted: z.boolean(), + limitations: z.array(z.string()), +}); + export type ProjectedComponent = z.infer; /** Path-pair hypotheses connecting managed content to a native library. */ @@ -90,13 +100,24 @@ export const projectedComponents = ( }); }; -export { bridgeCandidateCoverageSchema, projectCartesianCandidates }; +export const unavailableComponents = ( + inventory: ParsedInventorySet["inventory"], +) => + inventory.occurrences + .filter( + (occurrence) => + occurrence.logical_path !== "." && occurrence.artifact_id === null, + ) + .map((occurrence) => ({ + path: occurrence.logical_path, + artifact_id: null, + sha256: null, + format: occurrence.artifact_format, + encrypted: occurrence.encrypted, + limitations: occurrence.limitations, + })); -/** Unicode code point ordering so projection arrays sort deterministically. */ -export const compareProjectionStrings = ( - left: string, - right: string, -): number => (left < right ? -1 : left > right ? 1 : 0); +export { bridgeCandidateCoverageSchema, projectCartesianCandidates }; /** Digest a canonical JSON projection payload for its stable identifier. */ export const projectionDigest = (value: unknown, label: string): string => { diff --git a/src/domain/native/objcSwiftMetadata.test.ts b/src/domain/native/objcSwiftMetadata.test.ts index c8a3492480..2babf2161c 100644 --- a/src/domain/native/objcSwiftMetadata.test.ts +++ b/src/domain/native/objcSwiftMetadata.test.ts @@ -38,4 +38,40 @@ describe("ObjC/Swift metadata", () => { }), ); }); + + it("projects every accepted Swift mangling prefix and counts it", () => { + const names = [ + "$s5Store7launchyyF", + "_$s5Store7launchyyF", + "$S5Store7launchyyF", + "_$S5Store7launchyyF", + "_T05Store7launchyyF", + "@__swiftmacro_5Store7launchfMf_", + "$e5Store7launchyyF", + "_$e5Store7launchyyF", + "_TtC5Store11StoreHandle", + "_Tprintf", + ]; + const result = inspectNativeDispatchMetadata( + names.map((name, index) => ({ + address: `ram:0x${(0x1000 + index * 0x100).toString(16)}`, + name, + })), + names.length, + ); + + expect(result.swift_symbols.map((entry) => entry.mangled_name)).toEqual( + names.slice(0, -1), + ); + expect( + result.swift_symbols.every((entry) => entry.decode.status === "partial"), + ).toBe(true); + expect(result.coverage).toContainEqual( + expect.objectContaining({ + facet: "swift_declarations", + examined: 9, + decoded: 0, + }), + ); + }); }); diff --git a/src/domain/native/objcSwiftMetadata.ts b/src/domain/native/objcSwiftMetadata.ts index a360c32029..851b9f1b7b 100644 --- a/src/domain/native/objcSwiftMetadata.ts +++ b/src/domain/native/objcSwiftMetadata.ts @@ -1,5 +1,6 @@ import { z } from "zod"; import { providerSchema } from "../evidence.js"; +import { isSwiftMangledSymbolName } from "../swiftSymbolClassification.js"; import { compareUnicodeCodePoints, @@ -368,7 +369,7 @@ export const inspectNativeDispatchMetadata = ( classNames.add(methodMatch[2]); } - if (/^(?:_?\$s|_Tt)/u.test(entry.name)) + if (isSwiftMangledSymbolName(entry.name) || entry.name.startsWith("_Tt")) swiftSymbols.push({ mangled_name: entry.name, address: entry.address, diff --git a/src/domain/process/processCapture.validation.test.ts b/src/domain/process/processCapture.validation.test.ts index db36b2c643..4e8ec81a58 100644 --- a/src/domain/process/processCapture.validation.test.ts +++ b/src/domain/process/processCapture.validation.test.ts @@ -62,6 +62,37 @@ it("requires producer coverage accounting in every parsed capture", () => { ); }); +it("does not infer normalized retention budget from missing scenario defaults", () => { + const capture = emptyCapture(); + const scenario = { + ...capture.manifest.scenario, + limits: { + files: 10_000, + file_bytes: 10_000_000, + processes: 1_000, + filesystem_depth: 16, + }, + }; + expect(() => + parseProcessCapture({ + ...capture, + manifest: { + ...capture.manifest, + scenario, + full_scenario_sha256: digestProcessCommitment(scenario), + }, + truncation_details: { + ...capture.truncation_details, + normalized_terminal: { + budget_bytes: 1_000_000, + retained_bytes: 0, + omitted_frames: 0, + }, + }, + }), + ).toThrow("manifest.scenario.limits.output_bytes"); +}); + it("rejects settlement and cleanup combinations that cannot occur", () => { const capture = emptyCapture(); expect(() => diff --git a/src/domain/process/processCaptureCoverage.ts b/src/domain/process/processCaptureCoverage.ts index 72bbb9fb1c..5a38e92409 100644 --- a/src/domain/process/processCaptureCoverage.ts +++ b/src/domain/process/processCaptureCoverage.ts @@ -8,6 +8,7 @@ const byteRetentionSchema = z.strictObject({ retained_bytes: z.number().int().nonnegative(), observed_frames: z.number().int().nonnegative(), retained_frames: z.number().int().nonnegative(), + unknown_size_frames: z.number().int().nonnegative().optional(), }); /** Filesystem enumeration and whole-file digest coverage for one checkpoint. */ @@ -16,7 +17,19 @@ export const filesystemCoverageSchema = z.strictObject({ depth_limit: z.number().int().nonnegative(), enumeration_truncated: z.boolean(), enumeration_reasons: z.array( - z.enum(["files_limit", "depth_limit", "identity_changed"]), + z.enum([ + "files_limit", + "depth_limit", + "identity_changed", + "directory_unavailable", + ]), + ), + enumeration_omissions: z.array( + z.strictObject({ + path: z.string(), + reason: z.literal("directory_unavailable"), + system_code: z.string().nullable(), + }), ), hash_budget_bytes: z.number().int().nonnegative(), hashed_bytes: z.number().int().nonnegative(), @@ -42,8 +55,18 @@ export const processCaptureTruncationDetailsSchema = z.strictObject({ "UTF-8 bytes of original PTY chunks; whole chunks are retained or omitted.", ), rendered_terminal: byteRetentionSchema.describe( - "Cumulative UTF-8 bytes of serialized state plus visible lines, evaluated from retained PTY input; whole frames are retained or omitted.", + "Cumulative UTF-8 bytes of serialized state plus visible lines, evaluated from retained PTY input; whole frames are retained or omitted. When normalization exceeds a bound before the frame size is known, observed_bytes contains the exact total for size-known frames and unknown_size_frames counts the other omissions.", ), + normalized_terminal: z + .strictObject({ + budget_bytes: z.number().int().nonnegative(), + retained_bytes: z.number().int().nonnegative(), + omitted_frames: z.number().int().nonnegative(), + }) + .optional() + .describe( + "UTF-8 bytes of normalized raw-chunk text retained under output_bytes; omitted frames can have unknown final size when normalization exceeded an intermediate bound.", + ), filesystem_before: filesystemCoverageSchema, filesystem_after: filesystemCoverageSchema, process: z.strictObject({ @@ -60,13 +83,18 @@ export type ProcessCaptureTruncationDetails = z.infer< typeof processCaptureTruncationDetailsSchema >; export type TerminalRetention = ProcessCaptureTruncationDetails["raw_terminal"]; +export type NormalizedTerminalRetention = NonNullable< + ProcessCaptureTruncationDetails["normalized_terminal"] +>; export type ProcessSamplingCoverage = ProcessCaptureTruncationDetails["process"]; const omitted = (retention: TerminalRetention): boolean => retention.observed_frames > retention.retained_frames; const filesystemTruncated = (coverage: FilesystemCoverage): boolean => - coverage.enumeration_truncated || coverage.hash_omissions.length > 0; + coverage.enumeration_truncated || + coverage.enumeration_omissions.length > 0 || + coverage.hash_omissions.length > 0; /** Derive the aggregate truncation flag from independent producer observations. */ export const hasCaptureTruncation = ( @@ -74,6 +102,8 @@ export const hasCaptureTruncation = ( ): boolean => omitted(details.raw_terminal) || omitted(details.rendered_terminal) || + (details.normalized_terminal?.omitted_frames ?? 0) > 0 || + (details.rendered_terminal.unknown_size_frames ?? 0) > 0 || filesystemTruncated(details.filesystem_before) || filesystemTruncated(details.filesystem_after) || details.process.sampling_partial; diff --git a/src/domain/process/processCaptureExample.ts b/src/domain/process/processCaptureExample.ts index c8dbff8975..8259f38501 100644 --- a/src/domain/process/processCaptureExample.ts +++ b/src/domain/process/processCaptureExample.ts @@ -79,6 +79,7 @@ export const EMPTY_PROCESS_CAPTURE_EXAMPLE = { depth_limit: 16, enumeration_truncated: false, enumeration_reasons: [], + enumeration_omissions: [], hash_budget_bytes: 10_000_000, hashed_bytes: 0, hash_omissions: [], @@ -88,6 +89,7 @@ export const EMPTY_PROCESS_CAPTURE_EXAMPLE = { depth_limit: 16, enumeration_truncated: false, enumeration_reasons: [], + enumeration_omissions: [], hash_budget_bytes: 10_000_000, hashed_bytes: 0, hash_omissions: [], diff --git a/src/domain/process/processCaptureValidation.ts b/src/domain/process/processCaptureValidation.ts index 537f6f92f5..2c76007abe 100644 --- a/src/domain/process/processCaptureValidation.ts +++ b/src/domain/process/processCaptureValidation.ts @@ -1,4 +1,7 @@ -import { digestProcessCommitment } from "./processScenario.js"; +import { + digestProcessCommitment, + processScenarioOutputBudgetProjectionSchema, +} from "./processScenario.js"; import { hasCaptureTruncation } from "./processCaptureCoverage.js"; import type { @@ -312,8 +315,31 @@ const validateCoverage = ( retention.observed_bytes >= retention.retained_bytes && (retention.observed_frames > count || retention.observed_bytes === retention.retained_bytes) && - retention.retained_bytes <= - retention.budget_bytes, `truncation_details.${name}`, "retained observations and budget accounting do not agree"); + retention.retained_bytes <= retention.budget_bytes && + (retention.unknown_size_frames ?? 0) <= + retention.observed_frames - + retention.retained_frames, `truncation_details.${name}`, "retained observations and budget accounting do not agree"); + } + const normalized = details.normalized_terminal; + if (normalized !== undefined) { + const committedOutputBudget = + processScenarioOutputBudgetProjectionSchema.safeParse( + capture.manifest.scenario["limits"], + ); + require(committedOutputBudget.success, "manifest.scenario.limits.output_bytes", "committed process scenario must include an explicit valid output_bytes budget"); + const retainedBytes = capture.frames.reduce( + (total, frame) => total + Buffer.byteLength(frame.data), + 0, + ); + if (committedOutputBudget.success) + require(normalized.budget_bytes === + committedOutputBudget.data.output_bytes && + normalized.retained_bytes === retainedBytes && + normalized.retained_bytes <= normalized.budget_bytes && + normalized.omitted_frames <= + details.raw_terminal.observed_frames - + details.raw_terminal + .retained_frames, "truncation_details.normalized_terminal", "normalized terminal retention must agree with the committed budget and retained frames"); } for (const [name, coverage, files, checkpoint] of [ [ @@ -334,6 +360,9 @@ const validateCoverage = ( .filter((file) => file.type === "file" && file.sha256 === null) .map((file) => [file.path, file]), ); + const directories = new Set( + files.filter((file) => file.type === "directory").map(({ path }) => path), + ); require(coverage.hash_omissions.length === unhashedFiles.size && new Set(coverage.hash_omissions.map(({ path }) => path)).size === unhashedFiles.size && @@ -345,6 +374,14 @@ const validateCoverage = ( ? omission.system_code !== null : omission.system_code === null), ), `truncation_details.${name}.hash_omissions`, "every retained regular file without a digest must have one matching omission reason"); + require(new Set(coverage.enumeration_omissions.map(({ path }) => path)) + .size === coverage.enumeration_omissions.length && + (coverage.enumeration_omissions.length === 0 || + coverage.enumeration_reasons.includes("directory_unavailable")) && + coverage.enumeration_omissions.every( + (omission) => + directories.has(omission.path) && omission.system_code !== null, + ), `truncation_details.${name}.enumeration_omissions`, "every directory enumeration omission must identify one retained directory and its system error"); require(coverage.hashed_bytes === files .filter((file) => file.type === "file" && file.sha256 !== null) @@ -356,6 +393,7 @@ const validateCoverage = ( coverage.enumeration_reasons.length > 0 && checkpoint?.truncated === (coverage.enumeration_truncated || + coverage.enumeration_omissions.length > 0 || coverage.hash_omissions.length > 0), `truncation_details.${name}`, "filesystem checkpoint truncation must match enumeration and digest coverage"); } diff --git a/src/domain/process/processComparison.ts b/src/domain/process/processComparison.ts index eafcd4fd83..06e1340616 100644 --- a/src/domain/process/processComparison.ts +++ b/src/domain/process/processComparison.ts @@ -1,7 +1,10 @@ import canonicalize from "canonicalize"; import { z } from "zod"; import { processSourceTruncated } from "./processCaptureCoverage.js"; -import type { ProcessObservationSource } from "./processObservation.js"; +import type { + ProcessObservationLocation, + ProcessObservationSource, +} from "./processObservation.js"; import type { ProcessCapture } from "./processCaptureParsing.js"; import { comparableTerminalFrame } from "./processObservation.js"; @@ -299,6 +302,18 @@ const differingContractFields = ( ) .sort(); +const LOCATION_SOURCES = { + frames: "terminal_raw", + rendered_frames: "terminal_rendered", + interaction_events: "interaction", + lifecycle: "lifecycle", + process_samples: "process", + filesystem_checkpoints: "filesystem", +} as const satisfies Record< + ProcessObservationLocation["collection"], + ProcessObservationSource +>; + const applyTraceVerdict = ( dimensions: ComparisonDimensions, specification: ProcessTraceSpecification, @@ -314,6 +329,20 @@ const applyTraceVerdict = ( : "changed"; const sources = new Set(specification.events.map(({ source }) => source)); const covered = dimensionsForTraceSources(sources); + // A different verdict names the failing events and observations; attribute + // it only to their dimensions so equal observations keep their exact status. + const diagnosticSources = new Set([ + ...specification.events + .filter(({ id }) => trace.diagnostic?.event_ids.includes(id) === true) + .map(({ source }) => source), + ...(trace.diagnostic?.locations ?? []).map( + ({ collection }) => LOCATION_SOURCES[collection], + ), + ]); + const attributed = + diagnosticSources.size > 0 + ? dimensionsForTraceSources(diagnosticSources) + : covered; const statusFor = ( dimension: (typeof PROCESS_COMPARISON_DIMENSIONS)[number], ): ComparisonStatus => { @@ -321,7 +350,10 @@ const applyTraceVerdict = ( if (!covered.has(dimension)) return exact; if (trace.verdict === "nonconforming") return exact; if (trace.verdict === "different") - return traceShowsDifference ? "changed" : exact; + return traceShowsDifference && + (exact !== "unchanged" || attributed.has(dimension)) + ? "changed" + : exact; if ( trace.verdict === "unknown" && ["changed", "added", "removed"].includes(exact) diff --git a/src/domain/process/processScenario.ts b/src/domain/process/processScenario.ts index e98d7a5376..bc91d5b16e 100644 --- a/src/domain/process/processScenario.ts +++ b/src/domain/process/processScenario.ts @@ -3,6 +3,10 @@ import canonicalize from "canonicalize"; import { z } from "zod"; const positiveBudget = z.number().int().safe().positive(); +/** Read the explicitly committed output budget without applying scenario defaults. */ +export const processScenarioOutputBudgetProjectionSchema = z + .object({ output_bytes: positiveBudget }) + .passthrough(); /** * Maximum xterm backing cells for one visible-plus-scrollback buffer. Each * xterm BufferLine stores three Uint32 words per cell (12 MB at this cap), diff --git a/src/domain/process/processTraceComparison.test.ts b/src/domain/process/processTraceComparison.test.ts index 2207352e37..5966e4ef10 100644 --- a/src/domain/process/processTraceComparison.test.ts +++ b/src/domain/process/processTraceComparison.test.ts @@ -1,17 +1,49 @@ import { describe, expect, it } from "vitest"; +import { jsonValueSchema } from "../jsonValue.js"; import { parseProcessCapture } from "./processCaptureParsing.js"; import type { ProcessTraceSpecification } from "./processTraceSpecification.js"; import { processTraceSpecificationSchema as specificationSchema } from "./processTraceSpecification.js"; import { processTraceComparisonResultSchema } from "./processTraceEvaluation.js"; import { capture, + emptyCapture, partialSpecification, processStarted, terminal, values, } from "./processTraceComparison.fixture.js"; +import { compareProcessCaptures } from "./processComparison.js"; import { compareProcessTraces } from "./processTraceComparison.js"; + +const traced = ( + left: ReturnType, + right: ReturnType, +) => + compareProcessCaptures(left, right, { + traceSpecification: partialSpecification(), + }); + +const withPayloads = (data: string, command: string) => + capture({ + ...values(["terminal", "process"]), + frames: [{ ...terminal, data }], + process_samples: [{ ...processStarted, command }], + }); + +type FiniteTraceLanguage = Extract< + ProcessTraceSpecification["language"], + { kind: "finite_traces" } +>; + +const finiteTraceSpecification = ( + events: ProcessTraceSpecification["events"], + variants: FiniteTraceLanguage["variants"], +): ProcessTraceSpecification => ({ + events, + language: { kind: "finite_traces", variants }, +}); + describe("generic capture trace comparison", () => { it("requires explicit partial ordering and never infers causality from timestamps", () => { const specification = partialSpecification(); @@ -72,8 +104,8 @@ describe("generic capture trace comparison", () => { { collection: "frames", index: 1 }, ], }); - const specification: ProcessTraceSpecification = { - events: [ + const specification = finiteTraceSpecification( + [ { id: "tick1", source: "terminal_raw", @@ -87,14 +119,11 @@ describe("generic capture trace comparison", () => { cardinality: { kind: "optional" }, }, ], - language: { - kind: "finite_traces", - variants: [ - { id: "one", trace: ["tick1"] }, - { id: "two", trace: ["tick1", "tick2"] }, - ], - }, - }; + [ + { id: "one", trace: ["tick1"] }, + { id: "two", trace: ["tick1", "tick2"] }, + ], + ); expect(compareProcessTraces(one, two, specification)).toMatchObject({ verdict: "equivalent", left: { matched_variant: "one" }, @@ -107,8 +136,8 @@ describe("generic capture trace comparison", () => { // echoes those identifiers verbatim, so both must accept the same values // rather than handing the caller a raw schema error for its own input. const eventId = `e${"0".repeat(64)}`; - const specification: ProcessTraceSpecification = { - events: [ + const specification = finiteTraceSpecification( + [ { id: eventId, source: "terminal_raw", @@ -116,11 +145,8 @@ describe("generic capture trace comparison", () => { cardinality: { kind: "required" }, }, ], - language: { - kind: "finite_traces", - variants: [{ id: "only", trace: [eventId] }], - }, - }; + [{ id: "only", trace: [eventId] }], + ); expect(specificationSchema.safeParse(specification).success).toBe(true); const left = capture(values(["terminal", "process"])); expect(compareProcessTraces(left, left, specification)).toMatchObject({ @@ -130,6 +156,53 @@ describe("generic capture trace comparison", () => { }); }); +describe("finite trace variant mismatch attribution", () => { + it("reports tied best-prefix omissions across variants", () => { + const observed = capture({ + frames: [terminal], + process_samples: [], + event_journal: [{ collection: "frames", index: 0 }], + }); + const specification = finiteTraceSpecification( + [ + { + id: "ready", + source: "terminal_raw", + exact: terminal, + cardinality: { kind: "required" }, + }, + { + id: "worker", + source: "process", + exact: processStarted, + cardinality: { kind: "optional" }, + }, + { + id: "worker_alt", + source: "process", + exact: { ...processStarted, command: "alternate-worker" }, + cardinality: { kind: "optional" }, + }, + ], + [ + { id: "ready_worker", trace: ["ready", "worker"] }, + { id: "ready_worker_alt", trace: ["ready", "worker_alt"] }, + ], + ); + + expect( + compareProcessTraces(observed, observed, specification), + ).toMatchObject({ + verdict: "nonconforming", + diagnostic: { + kind: "trace", + event_ids: ["worker", "worker_alt"], + locations: [], + }, + }); + }); +}); + describe("process trace constraints and evidence sufficiency", () => { it("enforces exact and range cardinality for a repeated terminal observation", () => { const repeated = (count: number) => @@ -285,3 +358,330 @@ describe("unknown process trace evidence", () => { ).toThrow("event_journal"); }); }); + +describe("process trace dimension attribution", () => { + it("applies passing filesystem trace timestamp omissions to exact comparison", () => { + const base = capture(values(["terminal", "process"])); + const left = base; + const right = parseProcessCapture({ + ...base, + filesystem_checkpoints: base.filesystem_checkpoints.map((checkpoint) => + checkpoint.name === "after_settlement" + ? { ...checkpoint, at_ms: checkpoint.at_ms + 10 } + : checkpoint, + ), + }); + const specification = finiteTraceSpecification( + emptyCapture.filesystem_checkpoints.map( + ({ at_ms: _atMs, ...checkpoint }) => ({ + id: checkpoint.name, + source: "filesystem" as const, + exact: jsonValueSchema.parse(checkpoint), + ignore_fields: ["at_ms"], + cardinality: { kind: "required" as const }, + }), + ), + [{ id: "checkpoints", trace: ["before", "after_settlement"] }], + ); + + expect(compareProcessCaptures(left, right)).toMatchObject({ + filesystem: "changed", + first_divergence: { status: "found", dimension: "filesystem" }, + }); + expect( + compareProcessCaptures(left, right, { + traceSpecification: specification, + }), + ).toMatchObject({ + trace: { verdict: "equivalent" }, + filesystem: "unchanged", + first_divergence: { status: "none" }, + }); + + const finalFile = { + path: "root_0:created.txt", + type: "file" as const, + mode: 0o644, + size: 4, + sha256: "a".repeat(64), + symlink_target: null, + }; + const created = { + path: finalFile.path, + status: "created" as const, + before: null, + after: finalFile, + }; + const changedRight = parseProcessCapture({ + ...right, + filesystem_checkpoints: right.filesystem_checkpoints.map((checkpoint) => + checkpoint.name === "after_settlement" + ? { ...checkpoint, files: [finalFile], effects: [created] } + : checkpoint, + ), + files_after: [finalFile], + filesystem_effects: [created], + truncation_details: { + ...right.truncation_details, + filesystem_after: { + ...right.truncation_details.filesystem_after, + hashed_bytes: finalFile.size, + }, + }, + }); + expect( + compareProcessCaptures(left, changedRight, { + traceSpecification: specification, + }), + ).toMatchObject({ + trace: { verdict: "different" }, + filesystem: "changed", + first_divergence: { status: "found", dimension: "filesystem" }, + }); + }); + + it("keeps equal dimensions unchanged when one side fails a predicate", () => { + expect( + traced( + withPayloads("Ready", "worker"), + withPayloads("Different", "worker"), + ), + ).toMatchObject({ + trace: { verdict: "different" }, + terminal: "changed", + process: "unchanged", + }); + expect( + traced( + withPayloads("Ready", "worker"), + withPayloads("Ready", "other-worker"), + ), + ).toMatchObject({ + trace: { verdict: "different" }, + terminal: "unchanged", + process: "changed", + }); + }); +}); + +describe("finite trace dimension attribution", () => { + it("attributes a finite-trace omission to the omitted event dimension", () => { + const left = capture({ + frames: [terminal], + process_samples: [], + event_journal: [{ collection: "frames", index: 0 }], + }); + const right = capture(values(["terminal", "process"])); + const specification = finiteTraceSpecification( + [ + { + id: "ready", + source: "terminal_raw", + exact: terminal, + cardinality: { kind: "required" }, + }, + { + id: "worker", + source: "process", + exact: processStarted, + cardinality: { kind: "optional" }, + }, + ], + [{ id: "ready_worker", trace: ["ready", "worker"] }], + ); + + expect( + compareProcessCaptures(left, right, { + traceSpecification: specification, + }), + ).toMatchObject({ + status: "changed", + process: "changed", + terminal: "unchanged", + trace: { + verdict: "different", + diagnostic: { kind: "trace", event_ids: ["worker"], locations: [] }, + }, + first_divergence: { + status: "found", + dimension: "process", + index: 0, + }, + }); + }); +}); + +describe("unordered finite trace mismatch attribution", () => { + it("attributes only the missing member of an unordered group", () => { + const db = { sequence: 1, at_ms: 901, data: "DB" }; + const alert = { ...processStarted, pid: 2, command: "other-worker" }; + const left = capture({ + frames: [terminal], + process_samples: [processStarted], + event_journal: [ + { collection: "frames", index: 0 }, + { collection: "process_samples", index: 0 }, + ], + }); + const right = capture({ + frames: [terminal, db], + process_samples: [processStarted], + event_journal: [ + { collection: "frames", index: 0 }, + { collection: "process_samples", index: 0 }, + { collection: "frames", index: 1 }, + ], + }); + const specification = finiteTraceSpecification( + [ + { + id: "ready", + source: "terminal_raw", + exact: terminal, + cardinality: { kind: "required" }, + }, + { + id: "worker", + source: "process", + exact: processStarted, + cardinality: { kind: "optional" }, + }, + { + id: "db", + source: "terminal_raw", + exact: db, + cardinality: { kind: "optional" }, + }, + { + id: "alert", + source: "process", + exact: alert, + cardinality: { kind: "optional" }, + }, + ], + [ + { + id: "ready_then_work", + trace: ["ready", { unordered: ["worker", "db"] }], + }, + ], + ); + + expect( + compareProcessCaptures(left, right, { + traceSpecification: specification, + }), + ).toMatchObject({ + trace: { + verdict: "different", + diagnostic: { + side: "left", + event_ids: ["db"], + locations: [], + }, + }, + terminal: "changed", + process: "unchanged", + first_divergence: { status: "found", dimension: "terminal", index: 1 }, + }); + + const unexpected = capture({ + frames: [terminal], + process_samples: [processStarted, alert], + event_journal: [ + { collection: "frames", index: 0 }, + { collection: "process_samples", index: 0 }, + { collection: "process_samples", index: 1 }, + ], + }); + expect( + compareProcessTraces(unexpected, unexpected, specification), + ).toMatchObject({ + verdict: "nonconforming", + diagnostic: { + kind: "trace", + event_ids: ["db", "alert"], + locations: [{ collection: "process_samples", index: 1 }], + }, + }); + }); +}); + +describe("partial-order dimension attribution", () => { + it("attributes an ordering difference to both ordered dimensions", () => { + expect( + traced( + capture(values(["terminal", "process"])), + capture(values(["process", "terminal"])), + ), + ).toMatchObject({ + trace: { verdict: "different" }, + terminal: "changed", + process: "changed", + }); + }); + + it("attributes every independent failed ordering edge", () => { + const terminals = [ + terminal, + { ...terminal, sequence: 1, data: "Ready again" }, + ]; + const workers = [processStarted, { ...processStarted, at_ms: 2, pid: 2 }]; + const events = ["ready1", "worker1", "ready2", "worker2"] as const; + const make = (order: readonly (typeof events)[number][]) => + capture({ + frames: terminals, + process_samples: workers, + event_journal: order.map((event) => ({ + collection: event.startsWith("ready") + ? ("frames" as const) + : ("process_samples" as const), + index: Number(event.slice(-1)) - 1, + })), + }); + const left = make(events); + const right = make(["worker1", "ready1", "worker2", "ready2"]); + const specification: ProcessTraceSpecification = { + events: [ + ...terminals.map((frame, index) => ({ + id: `ready${String(index + 1)}`, + source: "terminal_raw" as const, + exact: frame, + cardinality: { kind: "required" as const }, + })), + ...workers.map((sample, index) => ({ + id: `worker${String(index + 1)}`, + source: "process" as const, + exact: sample, + cardinality: { kind: "required" as const }, + })), + ], + language: { + kind: "partial_order", + happens_before: [ + { before: "ready1", after: "worker1" }, + { before: "ready2", after: "worker2" }, + ], + not_before: [], + unordered_groups: [ + { events: ["ready1", "ready2"] }, + { events: ["worker1", "worker2"] }, + { events: ["ready1", "worker2"] }, + { events: ["worker1", "ready2"] }, + ], + prefix: [], + suffix: [], + }, + }; + + expect( + compareProcessCaptures(left, right, { + traceSpecification: specification, + }), + ).toMatchObject({ + trace: { verdict: "different" }, + terminal: "changed", + process: "changed", + }); + }); +}); diff --git a/src/domain/process/processTraceDimensionProjection.ts b/src/domain/process/processTraceDimensionProjection.ts index 8e8208d0ab..ed909acd82 100644 --- a/src/domain/process/processTraceDimensionProjection.ts +++ b/src/domain/process/processTraceDimensionProjection.ts @@ -46,7 +46,7 @@ export const traceCoversObservedDimension = ( case "exit": return sources.has("lifecycle"); case "filesystem": - return false; + return sources.has("filesystem"); case "process": return sources.has("process"); } diff --git a/src/domain/process/processTraceEvaluation.ts b/src/domain/process/processTraceEvaluation.ts index 7c232a4806..cf80a791df 100644 --- a/src/domain/process/processTraceEvaluation.ts +++ b/src/domain/process/processTraceEvaluation.ts @@ -123,29 +123,92 @@ const failure = (input: FailureInput): EvaluatedSide => ({ }, }); +type FiniteTraceEvaluation = + | { readonly status: "pass"; readonly variant: string } + | { + readonly status: "fail"; + readonly eventIds: readonly string[]; + readonly locationIndexes: readonly number[]; + }; + +/** Match variants once, retaining the best-prefix mismatch for attribution. */ const evaluateFiniteTrace = ( ids: readonly string[], - specification: ProcessTraceSpecification, -): string | null => { - if (specification.language.kind !== "finite_traces") return null; - for (const variant of specification.language.variants) { - let offset = 0; - let accepted = true; - for (const token of variant.trace) { - if (typeof token === "string") { - if (ids[offset] !== token) accepted = false; - offset += 1; - continue; - } - const actual = ids.slice(offset, offset + token.unordered.length).sort(); - const expected = [...token.unordered].sort(); - if (canonicalTraceJson(actual) !== canonicalTraceJson(expected)) - accepted = false; - offset += token.unordered.length; + language: Extract< + ProcessTraceSpecification["language"], + { kind: "finite_traces" } + >, +): FiniteTraceEvaluation => { + let bestPrefix = -1; + const bestEventIds = new Set(); + const bestLocationIndexes = new Set(); + + const recordMismatch = ( + offset: number, + missing: readonly string[], + unexpected: readonly { readonly id: string; readonly index: number }[], + ): void => { + if (offset > bestPrefix) { + bestPrefix = offset; + bestEventIds.clear(); + bestLocationIndexes.clear(); } - if (accepted && offset === ids.length) return variant.id; + if (offset !== bestPrefix) return; + for (const eventId of missing) bestEventIds.add(eventId); + for (const { id, index } of unexpected) { + bestEventIds.add(id); + bestLocationIndexes.add(index); + } + }; + + for (const variant of language.variants) { + let offset = 0; + let mismatched = false; + for (const token of variant.trace) { + const expected = typeof token === "string" ? [token] : token.unordered; + const actual = ids.slice(offset, offset + expected.length); + const remaining = new Map(); + for (const eventId of expected) + remaining.set(eventId, (remaining.get(eventId) ?? 0) + 1); + const unexpected: Array<{ readonly id: string; readonly index: number }> = + []; + for (const [index, eventId] of actual.entries()) { + const count = remaining.get(eventId) ?? 0; + if (count === 0) + unexpected.push({ id: eventId, index: offset + index }); + else if (count === 1) remaining.delete(eventId); + else remaining.set(eventId, count - 1); + } + const missing = [...remaining].flatMap(([eventId, count]) => + Array.from({ length: count }, () => eventId), + ); + if ( + missing.length > 0 || + unexpected.length > 0 || + actual.length < expected.length + ) { + recordMismatch(offset, missing, unexpected); + mismatched = true; + break; + } + offset += expected.length; + } + + if (mismatched) continue; + if (offset === ids.length) return { status: "pass", variant: variant.id }; + const unexpected = ids[offset]; + recordMismatch( + offset, + [], + unexpected === undefined ? [] : [{ id: unexpected, index: offset }], + ); } - return null; + + return { + status: "fail", + eventIds: [...bestEventIds], + locationIndexes: [...bestLocationIndexes], + }; }; type MatchedTrace = { @@ -248,38 +311,33 @@ const evaluateOrderingConstraints = ( language: PartialOrderLanguage, ): { readonly failure: EvaluatedSide | null; readonly satisfied: string[] } => { const satisfied: string[] = []; + const failures: FailureInput[] = []; for (const edge of language.happens_before) { const before = trace.locationsById.get(edge.before) ?? []; const after = trace.locationsById.get(edge.after) ?? []; const lastBefore = before.at(-1); const firstAfter = after[0]; if (lastBefore === undefined && firstAfter !== undefined) - return { - failure: failure({ - kind: "edge", - message: `${edge.before} is required when ${edge.after} occurs.`, - eventIds: [edge.before, edge.after], - locations: [firstAfter], - rawTrace: trace.rawTrace, - }), - satisfied, - }; - if ( + failures.push({ + kind: "edge", + message: `${edge.before} is required when ${edge.after} occurs.`, + eventIds: [edge.before, edge.after], + locations: [firstAfter], + rawTrace: trace.rawTrace, + }); + else if ( lastBefore !== undefined && firstAfter !== undefined && lastBefore.capture_order >= firstAfter.capture_order ) - return { - failure: failure({ - kind: "edge", - message: `${edge.before} must happen before ${edge.after}.`, - eventIds: [edge.before, edge.after], - locations: [lastBefore, firstAfter], - rawTrace: trace.rawTrace, - }), - satisfied, - }; - satisfied.push(`happens_before:${edge.before}:${edge.after}`); + failures.push({ + kind: "edge", + message: `${edge.before} must happen before ${edge.after}.`, + eventIds: [edge.before, edge.after], + locations: [lastBefore, firstAfter], + rawTrace: trace.rawTrace, + }); + else satisfied.push(`happens_before:${edge.before}:${edge.after}`); } for (const constraint of language.not_before) { const events = trace.locationsById.get(constraint.event) ?? []; @@ -287,34 +345,47 @@ const evaluateOrderingConstraints = ( const firstEvent = events[0]; const firstAnchor = anchors[0]; if (firstEvent !== undefined && firstAnchor === undefined) - return { - failure: failure({ - kind: "edge", - message: `${constraint.anchor} is required when ${constraint.event} occurs.`, - eventIds: [constraint.event, constraint.anchor], - locations: [firstEvent], - rawTrace: trace.rawTrace, - }), - satisfied, - }; - if ( + failures.push({ + kind: "edge", + message: `${constraint.anchor} is required when ${constraint.event} occurs.`, + eventIds: [constraint.event, constraint.anchor], + locations: [firstEvent], + rawTrace: trace.rawTrace, + }); + else if ( firstEvent !== undefined && firstAnchor !== undefined && firstEvent.capture_order < firstAnchor.capture_order ) - return { - failure: failure({ - kind: "edge", - message: `${constraint.event} must not occur before ${constraint.anchor}.`, - eventIds: [constraint.event, constraint.anchor], - locations: [firstEvent, firstAnchor], - rawTrace: trace.rawTrace, - }), - satisfied, - }; - satisfied.push(`not_before:${constraint.event}:${constraint.anchor}`); + failures.push({ + kind: "edge", + message: `${constraint.event} must not occur before ${constraint.anchor}.`, + eventIds: [constraint.event, constraint.anchor], + locations: [firstEvent, firstAnchor], + rawTrace: trace.rawTrace, + }); + else satisfied.push(`not_before:${constraint.event}:${constraint.anchor}`); } - return { failure: null, satisfied }; + if (failures.length === 0) return { failure: null, satisfied }; + const unique = (values: readonly T[], key: (value: T) => string): T[] => [ + ...new Map(values.map((value) => [key(value), value])).values(), + ]; + return { + failure: failure({ + kind: "edge", + message: failures.map(({ message }) => message).join(" "), + eventIds: unique( + failures.flatMap(({ eventIds }) => eventIds), + String, + ), + locations: unique( + failures.flatMap(({ locations }) => locations), + ({ collection, index }) => `${collection}:${String(index)}`, + ), + rawTrace: trace.rawTrace, + }), + satisfied, + }; }; const boundaryFailure = ( @@ -383,20 +454,23 @@ const evaluateMatchedTrace = ( if (specification.language.kind === "partial_order") return evaluatePartialOrder(trace, specification.language); const ids = trace.rawTrace.map(({ event_id }) => event_id); - const variant = evaluateFiniteTrace(ids, specification); - return variant === null + const evaluation = evaluateFiniteTrace(ids, specification.language); + return evaluation.status === "fail" ? failure({ kind: "trace", message: "Observed events do not match any finite trace variant.", - eventIds: ids, - locations: trace.rawTrace.map(({ location }) => location), + eventIds: evaluation.eventIds, + locations: evaluation.locationIndexes.flatMap((index) => { + const location = trace.rawTrace[index]?.location; + return location === undefined ? [] : [location]; + }), rawTrace: trace.rawTrace, }) : { result: { status: "pass", - matched_variant: variant, - satisfied_constraints: [`variant:${variant}`], + matched_variant: evaluation.variant, + satisfied_constraints: [`variant:${evaluation.variant}`], raw_trace: trace.rawTrace, }, diagnostic: null, diff --git a/src/domain/propertyListKeys.test.ts b/src/domain/propertyListKeys.test.ts index ded36daff8..d95f648572 100644 --- a/src/domain/propertyListKeys.test.ts +++ b/src/domain/propertyListKeys.test.ts @@ -1,4 +1,5 @@ -import { expect, it } from "vitest"; +import { describe, expect, it } from "vitest"; +import { parse } from "plist"; import { omitPrototypeKeys, parseXmlPropertyList } from "./propertyListKeys.js"; @@ -234,3 +235,55 @@ it("locates sparse omitted keys across newline-dense source", () => { omittedPrototypeKeys: 2, }); }); + +describe("OpenStep plist nesting preflight", () => { + const arrays = (depth: number, leaf = "x") => + "(".repeat(depth) + leaf + ")".repeat(depth); + + it.each([129, 3000])( + "rejects %s nested OpenStep values with the shared depth limit", + (depth) => { + expect(() => parseXmlPropertyList(arrays(depth))).toThrow( + "Plist exceeds 128 nesting levels", + ); + }, + ); + + it.each(["{}", "()"])( + "preserves the first %s root despite deeply nested trailing text", + (root) => { + expect(parseXmlPropertyList(root + arrays(3000))).toEqual({ + value: root === "{}" ? {} : [], + omittedPrototypeKeys: 0, + }); + }, + ); + + it("accepts the deepest supported OpenStep value and empty container", () => { + expect(() => parseXmlPropertyList(arrays(128))).not.toThrow(); + expect(() => parseXmlPropertyList(arrays(129, ""))).not.toThrow(); + }); + + it.each([ + "(x, y,)", + "{ key = x next = (y,); }", + "{ (compound, key) = x; { nested = key; } = y; }", + "{ key = (); next = {}; }", + ])("preserves pinned separator and compound-key semantics for %s", (text) => { + expect(parseXmlPropertyList(text)).toEqual({ + value: parse(text), + omittedPrototypeKeys: 0, + }); + }); + + it("ignores brackets in quoted strings, escaped quotes, comments, and data", () => { + expect( + parseXmlPropertyList( + '{ quote = "\\\"( )"; /* ((( */ data = <2829>; // (((\n values = (x); }', + ), + ).toEqual({ + value: { quote: '"( )', data: new Uint8Array([40, 41]), values: ["x"] }, + omittedPrototypeKeys: 0, + }); + }); +}); diff --git a/src/domain/propertyListKeys.ts b/src/domain/propertyListKeys.ts index 35aa0889d9..2dcab952ce 100644 --- a/src/domain/propertyListKeys.ts +++ b/src/domain/propertyListKeys.ts @@ -94,7 +94,10 @@ export const parseXmlPropertyList = ( // Preserve the pinned decoder's existing OpenStep autodetection, also used // by app Info.plist resolution. Only XML needs the key-element substitution. const first = text.trimStart()[0]; - if (first === "{" || first === "(") return omitPrototypeKeys(parse(text)); + if (first === "{" || first === "(") { + assertOpenStepDepthWithinLimit(text); + return omitPrototypeKeys(parse(text)); + } // Normalize only the literal line endings the pinned decoder normalizes. // Preserve character references in the source: serializing a DOM would turn // e.g. an encoded carriage return into a literal newline on the second parse. @@ -111,6 +114,133 @@ export const parseXmlPropertyList = ( return withoutKey(parse(substituted), placeholder); }; +interface OpenStepContainer { + end: string; + next: "key" | "equals" | "value" | "separator" | "after-value"; +} + +/** Consume only the punctuation expected between the pinned decoder's values. */ +const consumeOpenStepSeparator = ( + parent: OpenStepContainer | undefined, + character: string, +): "consumed" | "invalid" | "none" => { + if (parent?.next === "after-value") { + parent.next = "key"; + return character === ";" ? "consumed" : "none"; + } + if (parent?.next !== "equals" && parent?.next !== "separator") return "none"; + if (character !== (parent.next === "equals" ? "=" : ",")) return "invalid"; + parent.next = "value"; + return "consumed"; +}; + +/** An array may close after a comma; a dictionary still needs its assigned value. */ +const admitsOpenStepCloser = ( + parent: OpenStepContainer | undefined, + character: string, +): boolean => + parent?.end === character && + parent.next !== "equals" && + !(parent.end === "}" && parent.next === "value"); + +/** Bound OpenStep value nesting before the pinned decoder recursively reads it. */ +const assertOpenStepDepthWithinLimit = (text: string): void => { + const containers: OpenStepContainer[] = []; + const completeValue = (): void => { + const parent = containers.at(-1); + if (parent !== undefined) + parent.next = + parent.next === "key" + ? "equals" + : parent.end === "}" + ? "after-value" + : "separator"; + }; + const assertDepth = (): void => { + if (containers.length > MAX_PLIST_DEPTH) + throw new RangeError( + `Plist exceeds ${String(MAX_PLIST_DEPTH)} nesting levels`, + ); + }; + let cursor = 0; + while (cursor < text.length) { + const afterTrivia = skipOpenStepTrivia(text, cursor); + if (afterTrivia !== cursor) { + cursor = afterTrivia; + continue; + } + const character = text[cursor] ?? ""; + const parent = containers.at(-1); + // Stop at the first impossible token: only the pinned decoder owns syntax + // diagnostics. A later deep suffix cannot change that failure into a limit. + if (character === "}" || character === ")") { + if (!admitsOpenStepCloser(parent, character)) return; + containers.pop(); + // parseOpenStep returns the first root value without reading trailing text. + if (containers.length === 0) return; + completeValue(); + cursor += 1; + continue; + } + const separator = consumeOpenStepSeparator(parent, character); + if (separator === "invalid") return; + if (separator === "consumed") { + cursor += 1; + continue; + } + if (character === "{" || character === "(") { + assertDepth(); + containers.push({ + end: character === "{" ? "}" : ")", + next: character === "{" ? "key" : "value", + }); + cursor += 1; + continue; + } + const end = skipOpenStepScalar(text, cursor); + if (end === cursor) return; + if (parent?.next !== "key") assertDepth(); + cursor = end; + completeValue(); + } +}; + +/** Skip grammar trivia without interpreting delimiter examples inside comments. */ +const skipOpenStepTrivia = (text: string, cursor: number): number => { + if (/\s/u.test(text[cursor] ?? "")) return cursor + 1; + if (!text.startsWith("//", cursor) && !text.startsWith("/*", cursor)) + return cursor; + const line = text[cursor + 1] === "/"; + const end = text.indexOf(line ? "\n" : "*/", cursor + 2); + return end === -1 ? text.length : end + (line ? 1 : 2); +}; + +/** Read the extent of one opaque scalar; the pinned decoder still owns its value. */ +const skipOpenStepScalar = (text: string, start: number): number => { + let cursor = start; + const character = text[cursor]; + if (character === '"') { + cursor += 1; + while (cursor < text.length) { + const next = text[cursor]; + cursor += next === "\\" ? 2 : 1; + if (next === '"') break; + } + } else if (character === "<") { + const end = text.indexOf(">", cursor + 1); + cursor = end === -1 ? text.length : end + 1; + } else { + // Match the pinned OpenStep decoder's unquoted-string alphabet. Other + // syntax remains for the decoder to reject; preflight only owns depth. + while ( + cursor < text.length && + /[a-zA-Z0-9._/$:-]/u.test(text[cursor] ?? "") + ) + cursor += 1; + } + return cursor; +}; + /** Elements whose children the pinned plist value decoder visits recursively. */ const PLIST_CONTAINER_ELEMENTS = new Set(["plist", "dict", "array"]); diff --git a/src/domain/propertyListXmlText.test.ts b/src/domain/propertyListXmlText.test.ts new file mode 100644 index 0000000000..8cae86b390 --- /dev/null +++ b/src/domain/propertyListXmlText.test.ts @@ -0,0 +1,25 @@ +import { expect, it } from "vitest"; + +import { + decodeXmlPlistText, + XmlPlistTextDecodeError, +} from "./propertyListXmlText.js"; + +it.each([ + [ + Buffer.from(''), + "encoding-mismatch", + ], + [ + Buffer.from(''), + "unsupported-encoding", + ], + [Buffer.from([0xff, 0xfe, 0x3c]), "invalid-bytes"], +] as const)("tags XML decoding failures as %s", (bytes, reason) => { + expect(() => decodeXmlPlistText(bytes)).toThrow( + expect.objectContaining({ + name: "XmlPlistTextDecodeError", + reason, + } satisfies Partial), + ); +}); diff --git a/src/domain/propertyListXmlText.ts b/src/domain/propertyListXmlText.ts index 2d6820120f..8cf52d772d 100644 --- a/src/domain/propertyListXmlText.ts +++ b/src/domain/propertyListXmlText.ts @@ -1,4 +1,19 @@ -/** Decode BOM-aware XML plist text with fatal byte validation. */ +export class XmlPlistTextDecodeError extends TypeError { + constructor( + readonly reason: + | "invalid-bytes" + | "unsupported-encoding" + | "encoding-mismatch", + message: string, + readonly declaredEncoding?: string, + options?: ErrorOptions, + ) { + super(message, options); + this.name = "XmlPlistTextDecodeError"; + } +} + +/** Decode BOM-aware XML plist text and verify any declaration against its bytes. */ export const decodeXmlPlistText = (bytes: Uint8Array): string => { const encoding = bytes[0] === 0xff && bytes[1] === 0xfe @@ -6,7 +21,17 @@ export const decodeXmlPlistText = (bytes: Uint8Array): string => { : bytes[0] === 0xfe && bytes[1] === 0xff ? "utf-16be" : "utf-8"; - const text = new TextDecoder(encoding, { fatal: true }).decode(bytes); + let text: string; + try { + text = new TextDecoder(encoding, { fatal: true }).decode(bytes); + } catch (cause: unknown) { + throw new XmlPlistTextDecodeError( + "invalid-bytes", + "XML text bytes are invalid for the detected encoding", + undefined, + { cause }, + ); + } const declaration = /^<\?xml\s[^?]*\bencoding\s*=\s*["']([^"']+)["']/u.exec( text, )?.[1]; @@ -15,14 +40,20 @@ export const decodeXmlPlistText = (bytes: Uint8Array): string => { declared !== undefined && !["utf-8", "utf-16", "utf-16le", "utf-16be"].includes(declared) ) - throw new TypeError( + throw new XmlPlistTextDecodeError( + "unsupported-encoding", `Unsupported XML encoding declaration: ${declaration}. Supported encodings are UTF-8 and BOM-marked UTF-16 in either byte order.`, + declaration, ); if ( declared !== undefined && declared !== encoding && !(encoding !== "utf-8" && declared === "utf-16") ) - throw new TypeError("XML encoding declaration disagrees with its bytes"); + throw new XmlPlistTextDecodeError( + "encoding-mismatch", + "XML encoding declaration disagrees with its bytes", + declaration, + ); return text; }; diff --git a/src/domain/providerCleanupError.ts b/src/domain/providerCleanupError.ts index 4503b583ee..257d27e08e 100644 --- a/src/domain/providerCleanupError.ts +++ b/src/domain/providerCleanupError.ts @@ -4,7 +4,7 @@ import type { AnalysisPartialObservation } from "./analysisErrorBase.js"; /** Provider resources could not be proven closed after bounded cleanup. */ export class ProviderCleanupError extends ProviderAdapterError { - override readonly cleanupIncomplete = true; + override cleanupIncomplete = true; override readonly cleanupResources: readonly string[]; override readonly userMessage = "Provider cleanup could not be fully confirmed. Review the reported local resources before opening another target."; diff --git a/src/domain/providerSelectionError.ts b/src/domain/providerSelectionError.ts index e6520848f8..fb7a262c3f 100644 --- a/src/domain/providerSelectionError.ts +++ b/src/domain/providerSelectionError.ts @@ -23,6 +23,7 @@ export interface ProviderSelectionRejection { /** Complete typed context for one failed deep-provider selection. */ export interface ProviderSelectionErrorOptions { readonly capturedOutput?: AnalysisCapturedOutput; + readonly userMessage?: string; readonly operation?: string; readonly reason: ProviderSelectionFailureReason; readonly requestedProviderId: string; @@ -56,7 +57,8 @@ export class ProviderSelectionError extends AnalysisError { ...rejection, diagnostics: structuredClone(rejection.diagnostics), })); - this.userMessage = providerSelectionUserMessage(options); + this.userMessage = + options.userMessage ?? providerSelectionUserMessage(options); } } diff --git a/src/domain/reconstructionVerification.functionSources.test.ts b/src/domain/reconstructionVerification.functionSources.test.ts index 8f25ccb365..14220987de 100644 --- a/src/domain/reconstructionVerification.functionSources.test.ts +++ b/src/domain/reconstructionVerification.functionSources.test.ts @@ -323,11 +323,60 @@ describe("historical calls-only reconstruction changes", () => { }); describe("function reconstruction call coverage", () => { + it("preserves complete call diagnostics while passing equal call claims", () => { + const diagnostic = "Indirect calls remain unresolved."; + const leftWithDiagnostic = source("a", { + relationshipCoverage: { status: "complete", diagnostics: [diagnostic] }, + }); + const rightWithDiagnostic = source("b", { + relationshipCoverage: { status: "complete", diagnostics: [diagnostic] }, + }); + const observed = compareFunctions(leftWithDiagnostic, rightWithDiagnostic); + expect( + observed.dimensions.find(({ dimension }) => dimension === "calls"), + ).toMatchObject({ + status: "unchanged", + limitations: [`Left: ${diagnostic}`, `Right: ${diagnostic}`], + }); + const evidence = comparison( + { + left_evidence_id: leftWithDiagnostic.evidence_id, + right_evidence_id: rightWithDiagnostic.evidence_id, + }, + {}, + [leftWithDiagnostic, rightWithDiagnostic], + {}, + observed, + ); + expect( + admittedVerification( + evidence, + [leftWithDiagnostic, rightWithDiagnostic], + "calls", + )(), + ).toMatchObject({ + status: "pass", + claims: { + items: [ + expect.objectContaining({ + status: "pass", + observed_status: "unchanged", + limitations: expect.arrayContaining([ + `Source ${leftWithDiagnostic.evidence_id}: ${diagnostic}`, + `Source ${rightWithDiagnostic.evidence_id}: ${diagnostic}`, + ]), + }), + ], + }, + }); + }); + it("rechecks old unchanged comparisons without affecting unrelated finite dimensions", () => { const incomplete = source("7", { relationshipCoverage: { status: "unknown", reason: "Call graph unavailable.", + diagnostics: ["Indirect calls remain unresolved."], }, }); const incompleteSides = { @@ -352,6 +401,7 @@ describe("function reconstruction call coverage", () => { observed_status: "unchanged", limitations: expect.arrayContaining([ `Source ${incomplete.evidence_id}: Call graph unavailable.`, + `Source ${incomplete.evidence_id}: Indirect calls remain unresolved.`, ]), }), ], @@ -662,7 +712,7 @@ describe("function reconstruction admission controls", () => { ]); }); - it("rejects selected-dimension citations outside comparison closure", () => { + it("rejects any comparison finding link outside the top-level closure", () => { const evidence = comparison(singular, { result: jsonValueSchema.parse({ ...baselineResult, @@ -677,7 +727,7 @@ describe("function reconstruction admission controls", () => { }), }); expect(admittedVerification(evidence, [left, right, extra])).toThrow( - /dimension cites Evidence outside comparison closure/u, + /finding cites Evidence outside its top-level closure/u, ); }); }); diff --git a/src/domain/reconstructionVerification.test.ts b/src/domain/reconstructionVerification.test.ts index 664438d853..159a817296 100644 --- a/src/domain/reconstructionVerification.test.ts +++ b/src/domain/reconstructionVerification.test.ts @@ -21,7 +21,13 @@ const source = ( digit: string, authority: "controlled-replay" | "shipped-artifact", confidence: "observed" | "derived" = "observed", - captureResult: JsonValue = EMPTY_PROCESS_CAPTURE_EXAMPLE, + { + captureResult = EMPTY_PROCESS_CAPTURE_EXAMPLE, + replayEnvironment = environment, + }: { + readonly captureResult?: JsonValue; + readonly replayEnvironment?: typeof environment | null; + } = {}, ): Evidence => createEvidence( { @@ -43,7 +49,7 @@ const source = ( result: authority === "controlled-replay" ? captureResult : { digit }, confidence, authority, - environment: authority === "controlled-replay" ? environment : null, + environment: authority === "controlled-replay" ? replayEnvironment : null, }, ); @@ -113,6 +119,31 @@ const processComparison = ( }, ); +const multiSourceProcessComparison = ( + left: readonly Evidence[], + right: readonly Evidence[], +): Evidence => + createEvidence( + undefined, + { + id: "rea-process", + name: "REA deterministic process harness", + version: "3", + }, + { + predicateType: "rea.process-comparison", + operation: "compare_process_captures", + parameters: { + left_evidence_ids: left.map(({ evidence_id: id }) => id), + right_evidence_ids: right.map(({ evidence_id: id }) => id), + }, + result: processResult(), + confidence: "derived", + authority: "analyst-inference", + evidenceLinks: [...left, ...right].map(({ evidence_id: id }) => id), + }, + ); + const behavioralSpec = (comparison: Evidence) => ({ name: "CLI compatibility", claims: [ @@ -288,12 +319,9 @@ describe("reconstruction verification", () => { describe("reconstruction process capture input", () => { it("rejects saved captures without explicit executable identity", () => { - const left = source( - "1", - "controlled-replay", - "observed", - legacyV3Capture(), - ); + const left = source("1", "controlled-replay", "observed", { + captureResult: legacyV3Capture(), + }); const right = source("2", "controlled-replay"); const comparison = processComparison(left, right, processResult()); expect(() => @@ -306,6 +334,49 @@ describe("reconstruction process capture input", () => { }); describe("reconstruction verification outcomes", () => { + it("requires every selected behavioral source to share a replay environment", () => { + const replay = (digit: string, id: string | null) => + source(digit, "controlled-replay", "observed", { + replayEnvironment: id === null ? null : { ...environment, id }, + }); + const verify = (left: readonly Evidence[], right: readonly Evidence[]) => { + const comparison = multiSourceProcessComparison(left, right); + return verifyReconstruction( + behavioralSpec(comparison), + createEvidenceBundle([...left, ...right, comparison]), + ); + }; + const limitation = + "Controlled replay environments are unavailable or incompatible."; + const differentOnLeft = verify( + [replay("1", "env-a"), replay("2", "env-b")], + [replay("3", "env-a"), replay("4", "env-a")], + ); + expect(differentOnLeft.status).toBe("unknown"); + expect(differentOnLeft.claims.items[0]?.limitations).toContain(limitation); + + const differentOnRight = verify( + [replay("5", "env-a"), replay("6", "env-a")], + [replay("7", "env-a"), replay("8", "env-b")], + ); + expect(differentOnRight.status).toBe("unknown"); + expect(differentOnRight.claims.items[0]?.limitations).toContain(limitation); + + const unavailable = verify( + [replay("9", "env-a"), replay("a", "env-a")], + [replay("b", "env-a"), replay("c", null)], + ); + expect(unavailable.status).toBe("unknown"); + expect(unavailable.claims.items[0]?.limitations).toContain(limitation); + + expect( + verify( + [replay("d", "env-a"), replay("e", "env-a")], + [replay("f", "env-a"), replay("0", "env-a")], + ).status, + ).toBe("pass"); + }); + it("fails an authoritative observed disagreement", () => { const left = source("1", "controlled-replay"); const right = source("2", "controlled-replay"); diff --git a/src/domain/reconstructionVerification.ts b/src/domain/reconstructionVerification.ts index 6f693a1589..1467f1a66b 100644 --- a/src/domain/reconstructionVerification.ts +++ b/src/domain/reconstructionVerification.ts @@ -7,7 +7,10 @@ import { artifactComparisonResultSchema } from "./artifactComparison.js"; import { artifactInventoryResultSchema } from "./artifactGraph.js"; import { uniqueSorted } from "./canonicalOrdering.js"; import { compareUnicodeCodePoints } from "./unicodeCodePointOrder.js"; -import { comparisonSourceEvidenceSides } from "./comparisonSourceEvidence.js"; +import { + comparisonSourceEvidenceSides, + assertComparisonFindingLinks, +} from "./comparisonSourceEvidence.js"; import { parseEvidence, type Evidence } from "./evidence.js"; import { parseEvidenceBundle } from "./evidenceBundle.js"; import { functionComparisonResultSchema } from "./functionComparisonSchemas.js"; @@ -16,7 +19,7 @@ import { isProcessEvidenceProvider, } from "./process/processEvidenceProvider.js"; import { functionDossierSchema } from "./hopperValues.js"; -import { functionCallCoverageLimitations } from "./functionCallCoverage.js"; +import { assessFunctionCallCoverage } from "./functionCallCoverage.js"; import { deriveProcessComparisonStatus, PROCESS_COMPARISON_DIMENSIONS, @@ -61,7 +64,8 @@ const providers = { interface FunctionSourceFacts { readonly evidence_id: string; - readonly callCoverageLimitations: readonly string[]; + readonly callCoverageBlockers: readonly string[]; + readonly callCoverageDiagnostics: readonly string[]; } const acceptsPredicate = ( @@ -146,13 +150,14 @@ const evaluateClaim = ( const functionSourceFacts = readValidatedSourceFacts(claim, sources); const observation = comparisonObservation(claim, comparison); const observed = observation.status; - const limitations = [ + const callCoverage = sourceCallCoverage( + claim, + functionSourceFacts, + observation.hasIndependentFunctionChange === true, + ); + const blockers = [ ...authorityLimitations(claim, sources, sides), - ...sourceCallCoverageLimitations( - claim, - functionSourceFacts, - observation.hasIndependentFunctionChange === true, - ), + ...callCoverage.blockers, ]; const relatedUnknowns = reconstructionClaimUnknowns( heads, @@ -161,7 +166,7 @@ const evaluateClaim = ( claim.dimension, ); const relevantUnknowns = relatedUnknowns; - const status = classify(observed, limitations, relevantUnknowns); + const status = classify(observed, blockers, relevantUnknowns); const displayedUnknowns = relevantUnknowns; return reconstructionClaimResultSchema.parse({ claim_id: claim.claim_id, @@ -176,7 +181,9 @@ const evaluateClaim = ( unknown_ids: displayedUnknowns.map(({ unknown_id: id }) => id), limitations: uniqueSorted([ ...comparison.limitations, - ...limitations, + ...observation.limitations, + ...blockers, + ...callCoverage.diagnostics, ...displayedUnknowns.map( ({ question }) => `Residual unknown: ${question}`, ), @@ -210,12 +217,14 @@ const readValidatedSourceFacts = ( parseProcessCapture(source.normalized_result); else if (claim.kind === "structural-function") { const dossier = functionDossierSchema.parse(source.normalized_result); + const callCoverage = assessFunctionCallCoverage( + source.relationship_coverage, + dossier, + ); functionSources.push({ evidence_id: source.evidence_id, - callCoverageLimitations: functionCallCoverageLimitations( - source.relationship_coverage, - dossier, - ), + callCoverageBlockers: callCoverage.blockers, + callCoverageDiagnostics: callCoverage.diagnostics, }); } else artifactInventoryResultSchema.parse(source.normalized_result); } @@ -245,6 +254,7 @@ const comparisonObservation = ( evidence: Evidence, ): { readonly status: ObservedStatus; + readonly limitations: readonly string[]; readonly hasIndependentFunctionChange?: boolean; } => { if (claim.kind === "behavioral") { @@ -255,16 +265,26 @@ const comparisonObservation = ( return { status: claim.dimension === "overall" ? result.status : result[claim.dimension], + limitations: [], }; } if (claim.kind === "structural-function") { const result = functionComparisonResultSchema.parse( evidence.normalized_result, ); + assertComparisonFindingLinks( + evidence, + result.dimensions.flatMap(({ evidence_links }) => evidence_links), + ); if (claim.dimension === "overall") return { status: result.status, - // Historical overall changes may come solely from incomplete call sets. + limitations: result.dimensions.flatMap( + ({ limitations }) => limitations, + ), + // Historical overall changes can be caused only by incomplete call + // sets. Ignore that aggregate change when independently checking the + // source call-coverage limitations. hasIndependentFunctionChange: result.function_match.status === "mismatched" || (result.function_match.status === "matched" && @@ -279,28 +299,16 @@ const comparisonObservation = ( ); if (dimension === undefined) throw new TypeError("Function comparison dimension missing"); - if ( - dimension.evidence_links.some( - (id) => !evidence.evidence_links.includes(id), - ) - ) - throw new TypeError( - "Function dimension cites Evidence outside comparison closure", - ); - return { status: dimension.status }; + return { status: dimension.status, limitations: dimension.limitations }; } const result = artifactComparisonResultSchema.parse( evidence.normalized_result, ); - if ( - result.changes.some((item) => - item.evidence_links.some((id) => !evidence.evidence_links.includes(id)), - ) - ) - throw new TypeError( - "Artifact change cites Evidence outside comparison closure", - ); - return { status: result.status }; + assertComparisonFindingLinks( + evidence, + result.changes.flatMap(({ evidence_links }) => evidence_links), + ); + return { status: result.status, limitations: [] }; }; const validateProcessStatus = ( @@ -346,13 +354,15 @@ const authorityLimitations = ( ) limitations.push("Structural source artifact identity is unavailable."); if (claim.kind === "behavioral") { - const byId = new Map(sources.map((source) => [source.evidence_id, source])); - const left = byId.get(sides.left[0] ?? ""); - const right = byId.get(sides.right[0] ?? ""); + const referenceEnvironment = sources[0]?.environment; if ( - left?.environment === null || - right?.environment === null || - JSON.stringify(left?.environment) !== JSON.stringify(right?.environment) + referenceEnvironment === null || + referenceEnvironment === undefined || + sources.some( + ({ environment }) => + environment === null || + JSON.stringify(environment) !== JSON.stringify(referenceEnvironment), + ) ) limitations.push( "Controlled replay environments are unavailable or incompatible.", @@ -361,20 +371,28 @@ const authorityLimitations = ( return limitations; }; -const sourceCallCoverageLimitations = ( +const sourceCallCoverage = ( claim: Claim, functionSources: readonly FunctionSourceFacts[], hasIndependentFunctionChange: boolean, -): string[] => { +): { blockers: string[]; diagnostics: string[] } => { if ( claim.kind !== "structural-function" || (claim.dimension !== "calls" && !(claim.dimension === "overall" && !hasIndependentFunctionChange)) ) - return []; - return functionSources.flatMap(({ evidence_id, callCoverageLimitations }) => - callCoverageLimitations.map((reason) => `Source ${evidence_id}: ${reason}`), - ); + return { blockers: [], diagnostics: [] }; + return { + blockers: functionSources.flatMap(({ evidence_id, callCoverageBlockers }) => + callCoverageBlockers.map((reason) => `Source ${evidence_id}: ${reason}`), + ), + diagnostics: functionSources.flatMap( + ({ evidence_id, callCoverageDiagnostics }) => + callCoverageDiagnostics.map( + (reason) => `Source ${evidence_id}: ${reason}`, + ), + ), + }; }; const classify = ( diff --git a/src/domain/recordedTransportUrl.test.ts b/src/domain/recordedTransportUrl.test.ts new file mode 100644 index 0000000000..9c7b78ea0c --- /dev/null +++ b/src/domain/recordedTransportUrl.test.ts @@ -0,0 +1,11 @@ +import { expect, it } from "vitest"; + +import { parseRecordedTransportUrl } from "./recordedTransportUrl.js"; + +it("recognizes an empty userinfo marker at the authority start", () => { + expect(parseRecordedTransportUrl("https://@host")).toMatchObject({ + origin: "https://host", + path: "/", + credentials: { start: 8, end: 9 }, + }); +}); diff --git a/src/domain/recordedTransportUrl.ts b/src/domain/recordedTransportUrl.ts index c524ae1a87..429c0f6ca8 100644 --- a/src/domain/recordedTransportUrl.ts +++ b/src/domain/recordedTransportUrl.ts @@ -2,6 +2,8 @@ export interface RecordedTransportUrl { readonly origin: string | null; readonly path: string | null; + /** WHATWG query spelling, present only when explicitly selected for projection. */ + readonly query?: string; readonly credentials: | { readonly start: number; readonly end: number } | "ambiguous" @@ -15,6 +17,7 @@ export interface RecordedTransportUrl { */ export const parseRecordedTransportUrl = ( value: string, + options: { readonly includeQuery?: boolean } = {}, ): RecordedTransportUrl => { const prefix = /^[\x00-\x20]*(?:([A-Za-z][A-Za-z0-9+.-]*):)?\/\//u.exec( value, @@ -67,9 +70,22 @@ export const parseRecordedTransportUrl = ( (special && value[authorityEnd] === "\\") ) return { origin, path: null, credentials }; - if (value[authorityEnd] !== "/") return { origin, path: "/", credentials }; + const query = + options.includeQuery === true ? { query: absolute!.search } : {}; + if (value[authorityEnd] !== "/") + return { origin, path: "/", credentials, ...query }; let pathEnd = authorityEnd; while (pathEnd < value.length && !"?#".includes(value[pathEnd]!)) pathEnd += 1; - return { origin, path: value.slice(authorityEnd, pathEnd), credentials }; + return { + origin, + path: value.slice(authorityEnd, pathEnd), + credentials, + ...query, + }; }; + +/** WHATWG form-query name decoding; preserve order, repeats and empty names. */ +export const parseUrlQueryParameterNames = ( + query: string, +): IterableIterator => new URLSearchParams(query).keys(); diff --git a/src/domain/referenceSourceClassification.ts b/src/domain/referenceSourceClassification.ts index ea53a18a4e..fe8a6b28dd 100644 --- a/src/domain/referenceSourceClassification.ts +++ b/src/domain/referenceSourceClassification.ts @@ -27,6 +27,8 @@ const CODE_EXTENSIONS = new Set([ "rs", "go", "java", + "kt", + "kts", "c", "h", "cpp", @@ -54,6 +56,8 @@ const LANGUAGE_BY_EXTENSION = new Map([ ["rs", "Rust"], ["go", "Go"], ["java", "Java"], + ["kt", "Kotlin"], + ["kts", "Kotlin"], ["c", "C"], ["h", "C"], ["cpp", "C++"], @@ -213,7 +217,6 @@ const GENERATED_DIRECTORY_NAMES = new Set([ ".svelte-kit", ".astro", "storybook-static", - "docs/api", ".codex", ".devin", ".factory", @@ -225,6 +228,8 @@ const GENERATED_DIRECTORY_NAMES = new Set([ ".nyc_output", ]); +const GENERATED_DIRECTORY_PATHS = [["docs", "api"]] as const; + const VENDOR_DIRECTORY_NAMES = new Set([ "node_modules", "vendor", @@ -315,13 +320,18 @@ const basenameMatch = (filename: string, set: ReadonlySet): boolean => { return set.has(lower); }; -const isPathUnderAny = (path: string, names: ReadonlySet): boolean => { - for (const segment of path.split("/")) { - if (segment === "") continue; - if (names.has(segment.toLowerCase())) return true; - } - return false; -}; +const isPathUnderAny = ( + parts: readonly string[], + names: ReadonlySet, +): boolean => parts.some((segment) => names.has(segment)); + +const containsDirectoryPath = ( + parts: readonly string[], + directory: readonly string[], +): boolean => + parts.some((_, start) => + directory.every((segment, offset) => parts[start + offset] === segment), + ); /** Detect a source language from a POSIX path. */ export const detectReferenceSourceLanguage = (path: string): string | null => { @@ -344,33 +354,34 @@ export const classifyReferenceSourcePath = ( path: string, ): ReferenceSourceClassification[] => { const classifications: Set = new Set(); - const parts = path.split("/").filter((segment) => segment !== ""); + const parts = path + .toLowerCase() + .split("/") + .filter((segment) => segment !== ""); const filename = parts[parts.length - 1] ?? ""; - const lower = filename.toLowerCase(); - if (parts.length === 0) { classifications.add("source"); return [...classifications].sort(); } - addDirectoryClassifications(path, classifications); + addDirectoryClassifications(parts, classifications); addFilenameClassifications(filename, classifications); if ( - lower.endsWith(".json") || - lower.endsWith(".yaml") || - lower.endsWith(".yml") + filename.endsWith(".json") || + filename.endsWith(".yaml") || + filename.endsWith(".yml") ) { if (!classifications.has("manifest") && !classifications.has("config")) classifications.add("config"); } - if (isPathUnderAny(path, SOURCE_DIRECTORY_NAMES)) { + if (isPathUnderAny(parts, SOURCE_DIRECTORY_NAMES)) { if (!classifications.has("vendor") && !classifications.has("generated")) classifications.add("source"); } if (classifications.size === 0) { - if (lower === "license" || lower === "copying") + if (filename === "license" || filename === "copying") classifications.add("documentation"); else classifications.add("unknown"); } @@ -379,15 +390,20 @@ export const classifyReferenceSourcePath = ( }; const addDirectoryClassifications = ( - path: string, + parts: readonly string[], classifications: Set, ): void => { - if (isPathUnderAny(path, TEST_DIRECTORY_NAMES)) classifications.add("test"); - if (isPathUnderAny(path, DOCUMENTATION_DIRECTORY_NAMES)) + if (isPathUnderAny(parts, TEST_DIRECTORY_NAMES)) classifications.add("test"); + if (isPathUnderAny(parts, DOCUMENTATION_DIRECTORY_NAMES)) classifications.add("documentation"); - if (isPathUnderAny(path, VENDOR_DIRECTORY_NAMES)) + if (isPathUnderAny(parts, VENDOR_DIRECTORY_NAMES)) classifications.add("vendor"); - if (isPathUnderAny(path, GENERATED_DIRECTORY_NAMES)) + if ( + isPathUnderAny(parts, GENERATED_DIRECTORY_NAMES) || + GENERATED_DIRECTORY_PATHS.some((directory) => + containsDirectoryPath(parts, directory), + ) + ) classifications.add("generated"); }; diff --git a/src/domain/referenceSourceGraph.test.ts b/src/domain/referenceSourceGraph.test.ts index 1286cdde6b..5458b984ac 100644 --- a/src/domain/referenceSourceGraph.test.ts +++ b/src/domain/referenceSourceGraph.test.ts @@ -259,6 +259,30 @@ describe("historical source graph", () => { }); describe("partial historical source graphs", () => { + it("keeps unknown-entry diagnostics outside the root commitment", () => { + const base = graphInput(); + base.inventory_state = "partial"; + base.entries[2] = { + path: "src/package.json", + kind: "unknown", + classifications: ["unknown"], + limitations: ["Permission denied at /private/first"], + }; + const graph = createHistoricalSourceGraph(base); + const changedMessage = structuredClone(base); + changedMessage.entries[2] = { + path: "src/package.json", + kind: "unknown", + classifications: ["unknown"], + limitations: ["Permission denied at /private/second"], + }; + const revised = createHistoricalSourceGraph(changedMessage); + expect(revised.root_sha256).toBe(graph.root_sha256); + expect(computeHistoricalSourceGraphSha256(revised)).not.toBe( + computeHistoricalSourceGraphSha256(graph), + ); + }); + it("keeps an unreadable symlink target unknown and out of root identity", () => { const base = graphInput(); base.inventory_state = "partial"; @@ -434,6 +458,66 @@ describe("historical source graph validation", () => { }); }); +describe("historical source graph excluded symlink targets", () => { + it("retains internal symlinks into excluded trees without accepting sibling prefixes", () => { + const input = { ...graphInput(), inventory_state: "partial" as const }; + input.exclusions = [ + { + path: "dist", + entry_kind: "directory", + reason: "caller-excluded", + pattern: "dist/", + }, + ]; + input.entries[2] = symlinkEntry({ target: "../dist/generated.js" }); + const graph = createHistoricalSourceGraph(input); + expect(graph.entries[2]).toMatchObject({ + target: "../dist/generated.js", + target_state: "internal", + }); + + input.entries[2] = symlinkEntry({ target: "../dist-old/generated.js" }); + expect(() => createHistoricalSourceGraph(input)).toThrow( + /recorded exclusion/u, + ); + }); + + it("does not treat an excluded file or unknown-kind ancestor as a directory", () => { + const input = { ...graphInput(), inventory_state: "partial" as const }; + input.exclusions = [ + { + path: "secret.txt", + entry_kind: "file", + reason: "configured-secret", + pattern: "secret.txt", + }, + ]; + input.entries[2] = symlinkEntry({ target: "../secret.txt/child" }); + expect(() => createHistoricalSourceGraph(input)).toThrow( + /recorded exclusion/u, + ); + + input.exclusions[0] = { + path: "secret.txt", + entry_kind: "unknown", + reason: "configured-secret", + pattern: "secret.txt", + }; + expect(() => createHistoricalSourceGraph(input)).toThrow( + /recorded exclusion/u, + ); + + input.exclusions[0] = { + path: "secret.txt", + reason: "configured-secret", + pattern: "secret.txt", + }; + expect(() => createHistoricalSourceGraph(input)).toThrow( + /recorded exclusion/u, + ); + }); +}); + describe("historical source graph symlink path identity", () => { it("joins internal symlink targets with the inventory NFC normalizer", () => { // "café" spelled NFC, as the inventory produces it, and the same @@ -486,3 +570,54 @@ describe("historical source graph symlink path identity", () => { ); }); }); + +it.each([ + "\0", + "/absolute.js", + "../outside.js", + "a/../b.js", + "a//b.js", + "./a.js", + "a/", +])("rejects unsafe historical logical entry spelling %j", (path) => { + const input = graphInput(); + input.entries = [fileEntry({ path })]; + input.relationships = []; + expect(() => createHistoricalSourceGraph(input)).toThrow(); +}); + +it("rejects NUL in observed symlink targets before graph identity", () => { + const input = graphInput(); + input.entries[2] = symlinkEntry({ target: "literal\0name.js" }); + expect(() => createHistoricalSourceGraph(input)).toThrow( + /normalized POSIX symlink target/u, + ); +}); + +it.each(["C:\\outside\\file.js", "\\\\server\\share.js"])( + "preserves unowned recorded Windows absolute target policy %j", + (target) => { + const external = graphInput(); + external.inventory_state = "partial"; + external.entries[2] = symlinkEntry({ target, target_state: "external" }); + expect(createHistoricalSourceGraph(external).entries).toContainEqual( + expect.objectContaining({ target, target_state: "external" }), + ); + const internal = graphInput(); + internal.entries[2] = symlinkEntry({ target, target_state: "internal" }); + expect(() => createHistoricalSourceGraph(internal)).toThrow( + /target path does not match its target state/u, + ); + }, +); + +it("refuses an absolute POSIX internal target even when lexical joining would name an inventoried file", () => { + const input = graphInput(); + input.entries[2] = symlinkEntry({ + target: "/main.ts", + target_state: "internal", + }); + expect(() => createHistoricalSourceGraph(input)).toThrow( + /target path does not match its target state/u, + ); +}); diff --git a/src/domain/referenceSourceGraph.ts b/src/domain/referenceSourceGraph.ts index 197e473bfc..a10213e87e 100644 --- a/src/domain/referenceSourceGraph.ts +++ b/src/domain/referenceSourceGraph.ts @@ -18,7 +18,7 @@ const relativePathSchema = z .refine( (path) => !path.startsWith("/") && - !path.includes("\\") && + !path.includes("\0") && path .split("/") .every((part) => part !== "" && part !== "." && part !== ".."), @@ -85,10 +85,9 @@ const symlinkTargetSchema = z .min(1) .refine( (target) => - isPortableAbsoluteReferenceTarget(target) || - (!target.startsWith("/") && - !target.includes("\\") && - posix.normalize(target) === target), + !target.includes("\0") && + (isPortableAbsoluteReferenceTarget(target) || + (!target.startsWith("/") && posix.normalize(target) === target)), "Expected a normalized POSIX symlink target", ); @@ -111,6 +110,10 @@ const sourceEntrySchema = z.union([ sourceFileSchema, sourceDirectorySchema, sourceSymlinkSchema, + z.strictObject({ + ...entryBaseShape, + kind: z.literal("unknown"), + }), ]); const sourceRelationshipSchema = z.strictObject({ @@ -130,6 +133,9 @@ const parseFailureSchema = z.strictObject({ const exclusionSchema = z.discriminatedUnion("reason", [ z.strictObject({ path: relativePathSchema, + entry_kind: z + .enum(["file", "directory", "symlink", "other", "unknown"]) + .exactOptional(), reason: z.enum([ "configured-secret", "project-ignored", @@ -140,6 +146,9 @@ const exclusionSchema = z.discriminatedUnion("reason", [ }), z.strictObject({ path: relativePathSchema, + entry_kind: z + .enum(["file", "directory", "symlink", "other", "unknown"]) + .exactOptional(), reason: z.enum(["symlink-escape", "unreadable"]), }), ]); @@ -320,10 +329,50 @@ const checkSymlinks = ( context: z.RefinementCtx, ): void => { const lookupPath = createReferenceSourcePathLookup(paths); + const excludedPaths = createReferenceSourcePathLookup( + graph.exclusions.map(({ path }) => path), + ); + const excludedKinds = new Map< + string, + Set<"file" | "directory" | "symlink" | "other" | "unknown" | undefined> + >(); + for (const { path, entry_kind } of graph.exclusions) { + const kinds = excludedKinds.get(path) ?? new Set(); + kinds.add(entry_kind); + excludedKinds.set(path, kinds); + } for (const [index, entry] of graph.entries.entries()) { if (entry.kind !== "symlink") continue; if (entry.target === null) continue; - const absolute = isPortableAbsoluteReferenceTarget(entry.target); + const resolved = posix.normalize( + posix.join(posix.dirname(entry.path), entry.target), + ); + const escapesRoot = resolved.startsWith("../") || resolved === ".."; + // The root is implicit; every other internal target needs an exact or + // uniquely Unicode-equivalent recorded inventory identity. + const inventoried = + resolved === "." || lookupPath.resolve(resolved) !== undefined; + const exactExcluded = excludedPaths.resolve(resolved); + let excluded = exactExcluded !== undefined; + for ( + let candidate = posix.dirname(resolved); + candidate !== "." && candidate !== "/"; + candidate = posix.dirname(candidate) + ) { + const excludedAncestor = excludedPaths.resolve(candidate); + if (excludedAncestor === undefined) continue; + const kinds = excludedKinds.get(excludedAncestor); + if (kinds?.size !== 1 || !kinds.has("directory")) { + excluded = false; + break; + } + excluded = true; + } + const ownedRelativeTarget = + !entry.target.startsWith("/") && !escapesRoot && inventoried; + const absolute = + isPortableAbsoluteReferenceTarget(entry.target) && + !(entry.target_state === "internal" && ownedRelativeTarget); if ( (entry.target_state === "external" && !absolute) || (entry.target_state === "internal" && absolute) @@ -334,18 +383,11 @@ const checkSymlinks = ( path: ["entries", index, "target"], }); if (entry.target_state !== "internal") continue; - const resolved = posix.normalize( - posix.join(posix.dirname(entry.path), entry.target), - ); - if ( - resolved.startsWith("../") || - resolved === ".." || - // The inventory root is implicit rather than an entry with path ".". - (resolved !== "." && lookupPath.resolve(resolved) === undefined) - ) + if (escapesRoot || (!inventoried && !excluded)) context.addIssue({ code: "custom", - message: "Internal symlink target must resolve to an inventoried entry", + message: + "Internal symlink target must resolve to an inventoried entry or a recorded exclusion", path: ["entries", index, "target"], }); } @@ -361,7 +403,8 @@ const checkCompleteInventory = ( entry.limitations.length > 0 || (entry.kind === "file" && entry.content_state !== "hashed") || (entry.kind === "directory" && entry.tree_state !== "enumerated") || - (entry.kind === "symlink" && entry.target_state !== "internal"), + (entry.kind === "symlink" && entry.target_state !== "internal") || + entry.kind === "unknown", ); const partialRelationship = graph.relationships.some( ({ parse_state, resolution }) => @@ -478,6 +521,11 @@ const rootCommitment = ( target: entry.target, target_state: entry.target_state, }; + case "unknown": + return { + kind: entry.kind, + path: entry.path, + }; } }), exclusions, diff --git a/src/domain/referenceSourcePathIdentity.ts b/src/domain/referenceSourcePathIdentity.ts index cde64c71a3..4ac6963cf8 100644 --- a/src/domain/referenceSourcePathIdentity.ts +++ b/src/domain/referenceSourcePathIdentity.ts @@ -1,6 +1,10 @@ +import { posix } from "node:path"; + export interface ReferenceSourcePathLookup { readonly resolve: (path: string) => string | undefined; readonly resolveCandidates: (paths: readonly string[]) => string | undefined; + /** Resolve a path to the closest recorded parent identity. */ + readonly resolveAncestor: (path: string) => string | undefined; } /** @@ -18,11 +22,24 @@ export const createReferenceSourcePathLookup = ( } const equivalent = (path: string): string | undefined => canonical.get(path.normalize("NFC")) ?? undefined; + const resolve = (path: string): string | undefined => + exact.has(path) ? path : equivalent(path); return { - resolve: (path) => (exact.has(path) ? path : equivalent(path)), + resolve, resolveCandidates: (candidates) => candidates.find((path) => exact.has(path)) ?? candidates.map(equivalent).find((path) => path !== undefined), + resolveAncestor: (path) => { + for ( + let candidate = path; + candidate !== "." && candidate !== "/"; + candidate = posix.dirname(candidate) + ) { + const found = resolve(candidate); + if (found !== undefined) return found; + } + return undefined; + }, }; }; diff --git a/src/domain/referenceSourceRequireBindings.ts b/src/domain/referenceSourceRequireBindings.ts index f2f8d1ef54..234121a512 100644 --- a/src/domain/referenceSourceRequireBindings.ts +++ b/src/domain/referenceSourceRequireBindings.ts @@ -1,5 +1,11 @@ import * as t from "@babel/types"; +import { + semanticStaticPropertyKey, + semanticWriteSiteTarget, + semanticWriteTargets, + unwrapJavaScriptExpression, +} from "./javascript/javascriptAstValues.js"; import { analyzeParsedJavaScriptReferences } from "./javascript/javascriptSemanticAnalysis.js"; import { semanticBindingId, @@ -49,6 +55,22 @@ export const unshadowedReferenceSourceRequireCalls = ( (reference.resolution === "unbound" || reference.bindingId === programRequireBinding), ); + const isLoaderReference = (identifier: t.Identifier): boolean => { + const start = identifier.loc?.start; + const reference = + start === undefined + ? undefined + : references.get(`${start.line}:${start.column}`); + return ( + (allowsUnboundRequire && reference?.resolution === "unbound") || + (reference?.resolution === "resolved" && + reference.bindingId === programRequireBinding) + ); + }; + const loaderMembersIntact = requireLoaderMembersIntact( + file.program, + isLoaderReference, + ); const calls = new Set(); traverseJavaScriptAst(file.program, { @@ -64,6 +86,9 @@ export const unshadowedReferenceSourceRequireCalls = ( ) ) return; + // `require.resolve` and `require.main` denote Node's members only + // while no visible write or escape can have replaced them. + if (!t.isIdentifier(node.callee) && !loaderMembersIntact) return; const reference = references.get(`${start.line}:${start.column}`); if ( (allowsUnboundRequire && @@ -80,6 +105,68 @@ export const unshadowedReferenceSourceRequireCalls = ( return calls; }; +/** + * Whether the loader's `resolve` and `main` members keep Node's identity. A + * write to either member or to a computed member, or `require` used as a + * value other than a call or member read (where a mutator or alias can + * replace members), leaves that identity unknown. + */ +const requireLoaderMembersIntact = ( + program: t.Program, + isLoaderReference: (identifier: t.Identifier) => boolean, +): boolean => { + let intact = true; + const isLoaderMemberTarget = (target: t.Node): boolean => { + if (!t.isMemberExpression(target) && !t.isOptionalMemberExpression(target)) + return false; + const object = unwrapJavaScriptExpression(target.object).node; + if (!t.isIdentifier(object, { name: "require" })) return false; + if (!isLoaderReference(object)) return false; + const key = semanticStaticPropertyKey(target.property, target.computed); + return key === null || key === "resolve" || key === "main"; + }; + traverseJavaScriptAst(program, { + enter: (node, parent, readAncestors) => { + if (!intact) return; + const site = semanticWriteSiteTarget(node); + if (site !== null) { + for (const target of semanticWriteTargets(site)) + if (isLoaderMemberTarget(target)) intact = false; + return; + } + if ( + !t.isIdentifier(node, { name: "require" }) || + parent === null || + !isLoaderReference(node) + ) + return; + // Inert parentheses and TypeScript wrappers do not change how the + // loader value is used. + let used: t.Node = node; + let user: t.Node | undefined = parent; + const read = readAncestors(); + const ancestors = read.at(-1) === parent ? read : [...read, parent]; + for (let index = ancestors.length - 1; index >= 0; index -= 1) { + user = ancestors[index]; + if ( + user === undefined || + unwrapJavaScriptExpression(user).node !== node + ) + break; + used = user; + } + const usedAsLoader = + ((t.isCallExpression(user) || t.isOptionalCallExpression(user)) && + user.callee === used) || + ((t.isMemberExpression(user) || t.isOptionalMemberExpression(user)) && + user.object === used) || + t.isUnaryExpression(user, { operator: "typeof" }); + if (!usedAsLoader) intact = false; + }, + }); + return intact; +}; + const contains = (container: t.Node, node: t.Node): boolean => typeof container.start === "number" && typeof container.end === "number" && diff --git a/src/domain/referenceSourceSummary.test.ts b/src/domain/referenceSourceSummary.test.ts index a0d23b0e16..3f34a073f4 100644 --- a/src/domain/referenceSourceSummary.test.ts +++ b/src/domain/referenceSourceSummary.test.ts @@ -81,6 +81,26 @@ describe("committed historical-source summaries", () => { expect(graph).toEqual(original); }); + it("counts entries of unknown kind separately from symlinks", () => { + const graph = createHistoricalSourceGraph( + input([ + { + path: "deep", + kind: "unknown", + classifications: ["unknown"], + limitations: ["lstat: ENAMETOOLONG"], + }, + ]), + ); + expect(createHistoricalSourceSummary(graph, 1).counts).toMatchObject({ + entries: 1, + files: 0, + directories: 0, + symlinks: 0, + unknown: 1, + }); + }); + it("sums known bytes losslessly beyond the JSON safe integer range", () => { const graph = createHistoricalSourceGraph( input( diff --git a/src/domain/referenceSourceSummary.ts b/src/domain/referenceSourceSummary.ts index e7e99e3fa9..55d24025d2 100644 --- a/src/domain/referenceSourceSummary.ts +++ b/src/domain/referenceSourceSummary.ts @@ -29,6 +29,7 @@ export const createHistoricalSourceSummary = ( files: 0, directories: 0, symlinks: 0, + unknown: 0, relationships: graph.relationships.length, parse_failures: graph.parse_failures.length, exclusions: graph.exclusions.length, @@ -59,7 +60,8 @@ export const createHistoricalSourceSummary = ( } else if (entry.kind === "directory") { counts.directories += 1; treeStates[entry.tree_state] += 1; - } else counts.symlinks += 1; + } else if (entry.kind === "symlink") counts.symlinks += 1; + else counts.unknown += 1; // Canonical entries already use Unicode code-point order; filtering preserves it. if (entry.path.split("/").length <= outlineDepth) outline.push(outlineEntry(entry)); diff --git a/src/domain/relationshipCoverage.ts b/src/domain/relationshipCoverage.ts index 1de973669e..ec69dc5cc8 100644 --- a/src/domain/relationshipCoverage.ts +++ b/src/domain/relationshipCoverage.ts @@ -1,24 +1,51 @@ import { z } from "zod"; -const diagnostics = z.array(z.string().min(1)).min(1); -const completeCoverage = z.strictObject({ status: z.literal("complete") }); +import { fixedTupleSchema } from "./fixedTupleSchema.js"; + +const diagnostics = z.array(z.string().min(1)).min(1).exactOptional(); +const completeCoverage = z.strictObject({ + status: z.literal("complete"), + diagnostics, +}); +const limitedDirections = z.union([ + fixedTupleSchema([z.literal("callers")]), + fixedTupleSchema([z.literal("callees")]), + fixedTupleSchema([z.literal("callers"), z.literal("callees")]), +]); const partialCoverage = z.strictObject({ status: z.literal("partial"), limitations: z.array(z.string().min(1)).min(1), + limited_directions: limitedDirections.exactOptional(), + diagnostics, }); const unknownCoverage = z.strictObject({ status: z.literal("unknown"), reason: z.string().min(1), + limited_directions: limitedDirections.exactOptional(), + diagnostics, }); /** Explicit coverage state for a provider's observed caller/callee set. */ -export const relationshipCoverageSchema = z.union([ +export const relationshipCoverageSchema = z.discriminatedUnion("status", [ completeCoverage, - completeCoverage.extend({ diagnostics }), partialCoverage, - partialCoverage.extend({ diagnostics }), unknownCoverage, - unknownCoverage.extend({ diagnostics }), ]); export type RelationshipCoverage = z.infer; + +/** Reasons the observed relationship inventory cannot establish completeness. */ +export const relationshipCoverageLimitations = ( + coverage: RelationshipCoverage | null | undefined, +): readonly string[] => { + switch (coverage?.status) { + case undefined: + return ["Provider did not report relationship coverage."]; + case "complete": + return []; + case "partial": + return coverage.limitations; + case "unknown": + return [coverage.reason]; + } +}; diff --git a/src/domain/reverseEngineering.ts b/src/domain/reverseEngineering.ts new file mode 100644 index 0000000000..fb0cc9a08c --- /dev/null +++ b/src/domain/reverseEngineering.ts @@ -0,0 +1,40 @@ +import { z } from "zod"; + +const objdumpArtifactInputSchema = z + .object({ + path: z.string().min(1), + operation: z.enum([ + "file_headers", + "section_headers", + "symbols", + "relocations", + "dwarf", + "disassemble", + "disassemble_all", + ]), + follow_debug_links: z.boolean().default(false), + }) + .strict(); + +/** Caller-selected artifact facet or pathless provider capability query. */ +export const objdumpInputSchema = z.discriminatedUnion("operation", [ + objdumpArtifactInputSchema, + z.strictObject({ operation: z.literal("architectures") }), +]); + +/** Caller-selected Rizin command against one local artifact. */ +export const rizinInputSchema = z + .object({ + path: z.string().min(1), + command: z + .string() + .min(1) + .refine( + (value) => !/[\r\n\u0000]/u.test(value), + "A single command line is required", + ), + }) + .strict(); + +export type ObjdumpInput = z.input; +export type RizinInput = z.input; diff --git a/src/domain/rust/rustBinary.ts b/src/domain/rust/rustBinary.ts index f09cde222c..95cc60a807 100644 --- a/src/domain/rust/rustBinary.ts +++ b/src/domain/rust/rustBinary.ts @@ -2,7 +2,8 @@ import { z } from "zod"; import { digestSchema } from "../digests.js"; import { localPathStringSchema } from "../localPath.js"; -export const RUST_AUDITABLE_FORMAT_VERSION = 1; +/** Known revisions used to interpret compatible cargo-auditable documents. */ +export const RUST_AUDITABLE_FORMAT_VERSIONS = [0, 1, 8] as const; const index = z.number().int().nonnegative(); const rustcHash = z.string().regex(/^[0-9a-f]{40}$/u); @@ -29,7 +30,8 @@ export const rustAuditableMetadataSchema = z.strictObject({ section_bytes: z.number().int().positive(), compressed_bytes: z.number().int().positive(), uncompressed_bytes: z.number().int().positive(), - format_version: z.literal(RUST_AUDITABLE_FORMAT_VERSION).nullable(), + /** Original embedded revision; null means the source omitted the field. */ + format_version: z.number().int().nonnegative().max(0xffff_ffff).nullable(), packages: z.array(rustPackageSchema), root_package_index: index.nullable(), dependency_edges: z.array(dependencyEdgeSchema), diff --git a/src/domain/sqlite/sqliteDatabase.ts b/src/domain/sqlite/sqliteDatabase.ts index 60f3c6d3fa..cb126098e7 100644 --- a/src/domain/sqlite/sqliteDatabase.ts +++ b/src/domain/sqlite/sqliteDatabase.ts @@ -44,7 +44,7 @@ export const inspectSqliteDatabaseInputSchema = z .enum(["inventory", "selected_table", "database"]) .optional() .describe( - "Default: compact inventory without a table, or detailed selected-table schema with a table. Database explicitly expands all stored declarations and supported columns.", + "Default: compact inventory without a table, or detailed selected-table schema with a table. selected_table requires table. Database explicitly expands all stored declarations and supported columns.", ), row_limit: z .number() @@ -55,6 +55,7 @@ export const inspectSqliteDatabaseInputSchema = z "Selected-table sample size, default 20; subject to byte and worker resource limits", ), }) + .describe("When schema_scope is selected_table, also provide table.") .superRefine((input, context) => { if (input.row_limit !== undefined && input.table === undefined) context.addIssue({ diff --git a/src/domain/staticRuntimeCorrelation.test.ts b/src/domain/staticRuntimeCorrelation.test.ts index e2d4fff143..83051af7e6 100644 --- a/src/domain/staticRuntimeCorrelation.test.ts +++ b/src/domain/staticRuntimeCorrelation.test.ts @@ -2,7 +2,10 @@ import { describe, expect, it } from "vitest"; import { INVESTIGATION_EXAMPLES } from "../contracts/investigationExamples.js"; import { createEvidence, type Evidence } from "./evidence.js"; -import type { JsonValue } from "./jsonValue.js"; +import { jsonValueSchema, type JsonValue } from "./jsonValue.js"; +import { FUNCTION_COMPARISON_EXAMPLE } from "../contracts/functionComparisonExample.js"; +import { functionDossierSchema } from "./hopperValues.js"; +import { compareFunctions } from "./functionComparison.js"; import { correlateStaticAndRuntime, staticRuntimeCorrelationInputSchema, @@ -113,6 +116,70 @@ const mapping = ( ], }); +it("keeps text, assembly and unavailable CFG caveats on their selected correlations", () => { + const source = FUNCTION_COMPARISON_EXAMPLE.left; + const left = createEvidence( + { path: "/tmp/unavailable-cfg", sha256: "a".repeat(64), format: "mach-o" }, + source.provider, + { + operation: source.operation, + parameters: source.parameters, + result: jsonValueSchema.parse({ + ...functionDossierSchema.parse(source.normalized_result), + basic_blocks: [{ start: "0x1000", end: "0x1000", successors: [] }], + }), + confidence: source.confidence, + authority: source.authority, + }, + ); + const right = FUNCTION_COMPARISON_EXAMPLE.right; + const computed = compareFunctions(left, right); + const comparison = createEvidence(undefined, staticComparison.provider, { + predicateType: "rea.function-comparison", + operation: "compare_functions", + parameters: { + left_evidence_id: left.evidence_id, + right_evidence_id: right.evidence_id, + }, + result: jsonValueSchema.parse(computed), + confidence: "derived", + authority: "analyst-inference", + limitations: computed.limitations, + evidenceLinks: [left.evidence_id, right.evidence_id], + }); + const input = mapping(processComparison("unchanged"), "static_only"); + const result = correlateStaticAndRuntime({ + ...input, + static_comparisons: [comparison], + mappings: ["pseudocode", "assembly", "cfg"].map((dimension) => ({ + ...input.mappings[0], + static: { + comparison_evidence_id: comparison.evidence_id, + selector: { kind: "function", dimension }, + }, + })), + }); + expect(result.status).toBe("unknown"); + const caveats = new Map( + result.correlations.items.map((item) => [ + item.static.selector.kind === "function" + ? item.static.selector.dimension + : "artifact", + item.limitations.join(" "), + ]), + ); + expect(caveats.get("pseudocode")).toMatch(/generated-name normalization/u); + expect(caveats.get("pseudocode")).not.toMatch( + /Assembly is opaque|CFG addresses/u, + ); + expect(caveats.get("assembly")).toMatch(/Assembly is opaque/u); + expect(caveats.get("assembly")).not.toMatch(/CFG addresses/u); + expect(caveats.get("cfg")).toMatch(/CFG addresses/u); + expect(caveats.get("cfg")).not.toMatch( + /generated-name normalization|Assembly is opaque/u, + ); +}); + describe("static/runtime correlation", () => { it("accepts real runtime dimensions and rejects removed protocol dimensions", () => { const runtime = processComparison("unchanged"); @@ -154,6 +221,9 @@ describe("static/runtime correlation", () => { expect(result.limitations.join(" ")).toMatch( /does not establish causality/u, ); + expect(result.correlations.items[0]?.limitations.join(" ")).toMatch( + /generated-name normalization/u, + ); }); it("classifies an explicit expectation mismatch as contradiction", () => { diff --git a/src/domain/staticRuntimeCorrelation.ts b/src/domain/staticRuntimeCorrelation.ts index b00f5a3d5e..0362569a7f 100644 --- a/src/domain/staticRuntimeCorrelation.ts +++ b/src/domain/staticRuntimeCorrelation.ts @@ -6,7 +6,10 @@ import { z } from "zod"; import { artifactComparisonResultSchema } from "./artifactComparison.js"; import { uniqueSorted } from "./canonicalOrdering.js"; import { compareUnicodeCodePoints } from "./unicodeCodePointOrder.js"; -import { comparisonSourceEvidenceIds } from "./comparisonSourceEvidence.js"; +import { + comparisonSourceEvidenceIds, + assertComparisonFindingLinks, +} from "./comparisonSourceEvidence.js"; import { evidenceSchema, parseEvidence, type Evidence } from "./evidence.js"; import { functionComparisonResultSchema } from "./functionComparisonSchemas.js"; import { @@ -236,7 +239,10 @@ const correlate = ( mapping.runtime.comparison_evidence_id, "runtime comparison", ); - const selectedStatic = selectStatic(staticEvidence, mapping.static.selector); + const { limitations: selectedLimitations, ...selectedStatic } = selectStatic( + staticEvidence, + mapping.static.selector, + ); const process = processCaptureComparisonSchema.parse( runtimeEvidence.normalized_result, ); @@ -270,6 +276,7 @@ const correlate = ( ]), limitations: uniqueSorted([ ...staticEvidence.limitations, + ...selectedLimitations, ...runtimeEvidence.limitations, ...process.limitations, "This correlation tests an explicit hypothesis and does not establish causality.", @@ -281,7 +288,9 @@ const correlate = ( const selectStatic = ( evidence: Evidence, selector: z.infer, -): Pick => { +): Pick & { + readonly limitations: readonly string[]; +} => { if (selector.kind === "function") { if (evidence.operation !== "compare_functions") throw new TypeError( @@ -299,6 +308,7 @@ const selectStatic = ( return { status: match.status, conclusion_kind: match.conclusion_kind, + limitations: match.limitations, }; } if (evidence.operation !== "compare_artifacts") @@ -319,6 +329,7 @@ const selectStatic = ( const status = match.classification; return { status, + limitations: [], conclusion_kind: status === "contradiction" ? "contradiction" @@ -342,7 +353,7 @@ const parseStatic = (input: unknown): Evidence => { const result = artifactComparisonResultSchema.parse( evidence.normalized_result, ); - assertNestedLinks( + assertComparisonFindingLinks( evidence, result.changes.flatMap(({ evidence_links: links }) => links), ); @@ -350,7 +361,7 @@ const parseStatic = (input: unknown): Evidence => { const result = functionComparisonResultSchema.parse( evidence.normalized_result, ); - assertNestedLinks( + assertComparisonFindingLinks( evidence, result.dimensions.flatMap(({ evidence_links: links }) => links), ); @@ -358,17 +369,6 @@ const parseStatic = (input: unknown): Evidence => { return evidence; }; -const assertNestedLinks = ( - evidence: Evidence, - nestedLinks: readonly string[], -): void => { - const closure = new Set(evidence.evidence_links); - if (nestedLinks.some((evidenceId) => !closure.has(evidenceId))) - throw new TypeError( - "Comparison finding cites Evidence outside its top-level closure", - ); -}; - const parseRuntime = (input: unknown): Evidence => { const evidence = parseEvidence(input); if (evidence.operation !== "compare_process_captures") diff --git a/src/domain/swiftSymbolClassification.ts b/src/domain/swiftSymbolClassification.ts index 58b9b0d261..04910e5c58 100644 --- a/src/domain/swiftSymbolClassification.ts +++ b/src/domain/swiftSymbolClassification.ts @@ -55,6 +55,26 @@ export const classifySwiftSymbol = ( return { category }; }; +/** + * Prefixes accepted by Swift's demangler as manglings. This is deliberately + * narrower than visible Swift runtime class names (`_Tt…`): those names are + * useful metadata symbols but are not themselves demangler manglings. + */ +const SWIFT_MANGLING_PREFIXES = [ + "_T0", + "$S", + "_$S", + "$s", + "_$s", + "@__swiftmacro_", + "$e", + "_$e", +] as const; + +/** Recognize names the Swift demangler accepts as mangled symbols. */ +export const isSwiftMangledSymbolName = (name: string): boolean => + SWIFT_MANGLING_PREFIXES.some((prefix) => name.startsWith(prefix)); + const identifierEnd = ( mangling: string, offset: number, diff --git a/src/domain/symbolAnalysis.test.ts b/src/domain/symbolAnalysis.test.ts index c43bbfb764..2afa88bf29 100644 --- a/src/domain/symbolAnalysis.test.ts +++ b/src/domain/symbolAnalysis.test.ts @@ -94,4 +94,46 @@ describe("Swift alias ambiguity", () => { ], }); }); + + it("retains manglings whose prefix the demangler accepts", () => { + const names = [ + "$e4main7ServiceCMa", + "_$e4main7ServiceCMa", + "@__swiftmacro_4main7ServiceCMa", + ]; + const result = categorizeSwiftTypes( + names.map((name, index) => ({ address: `0x${String(index)}`, name })), + ); + expect(result.categories.other?.items.map((item) => item.name)).toEqual( + names, + ); + expect(result.unclassified.map((item) => item.name)).toEqual(names); + }); + + it("keeps legacy names with an optional leading underscore", () => { + expect( + categorizeSwiftTypes([{ address: "0x1", name: "__T04main4PairVMa" }], { + category: "structs", + }), + ).toMatchObject({ + total: 1, + categories: { structs: { count: 1 } }, + unclassified: [], + }); + }); + + it("does not let a demangler-only alias erase a decoded category", () => { + const result = categorizeSwiftTypes( + [{ address: "0x1", name: "$s4main7ServiceCMa" }], + {}, + [{ address: "0x1", name: "$e4main7ServiceCMa" }], + ); + expect(result.categories.classes?.items).toMatchObject([ + { + name: "$s4main7ServiceCMa", + mangled_names: ["$s4main7ServiceCMa", "$e4main7ServiceCMa"], + }, + ]); + expect(result.categories.other?.items).toEqual([]); + }); }); diff --git a/src/domain/symbolAnalysis.ts b/src/domain/symbolAnalysis.ts index 6667afcc6d..173dd99f9e 100644 --- a/src/domain/symbolAnalysis.ts +++ b/src/domain/symbolAnalysis.ts @@ -2,6 +2,7 @@ import type { JsonValue } from "./jsonValue.js"; import type { AddressedName } from "./hopperValues.js"; import { classifySwiftSymbol, + isSwiftMangledSymbolName, type SwiftTypeCategory, } from "./swiftSymbolClassification.js"; @@ -58,6 +59,16 @@ export const discoverObjcProtocols = ( }; }; +/** + * A Swift symbol carries either a demangler mangling or a name the category + * decoder already accepts, including visible runtime class names (`_Tt…`) and + * the optional leading underscore of legacy encodings (`__T0…`). Whether the + * nominal category can be decoded is a separate question, so it must not decide + * whether the symbol is Swift at all. + */ +const isSwiftSymbolName = (name: string): boolean => + isSwiftMangledSymbolName(name) || classifySwiftSymbol(name) !== undefined; + /** Categorize procedure names using Swift manglings at their exact addresses. */ export const categorizeSwiftTypes = ( procedures: readonly AddressedName[], @@ -94,7 +105,7 @@ export const categorizeSwiftTypes = ( if (seen.has(entry.name)) continue; const mangledNames = [ ...new Set([ - ...(classifySwiftSymbol(entry.name) === undefined ? [] : [entry.name]), + ...(isSwiftSymbolName(entry.name) ? [entry.name] : []), ...(aliases.get(entry.address) ?? []), ]), ]; @@ -104,8 +115,12 @@ export const categorizeSwiftTypes = ( ![entry.name, ...mangledNames].some((name) => name.includes(pattern)) ) continue; + // Only decoded names vote. Unresolved aliases remain evidence without + // erasing category evidence from a name the decoder did recognize. const kinds = new Set( - mangledNames.map((name) => classifySwiftSymbol(name)?.category ?? null), + mangledNames + .filter((name) => classifySwiftSymbol(name) !== undefined) + .map((name) => classifySwiftSymbol(name)?.category ?? null), ); const category = kinds.size === 1 ? ([...kinds][0] ?? null) : null; const item = { @@ -154,7 +169,7 @@ export const categorizeSwiftTypes = ( const swiftAliasesByAddress = (symbols: readonly AddressedName[]) => { const aliases = new Map(); for (const { address, name } of symbols) { - if (classifySwiftSymbol(name) === undefined) continue; + if (!isSwiftSymbolName(name)) continue; const existing = aliases.get(address); if (existing === undefined) aliases.set(address, [name]); else existing.push(name); diff --git a/src/domain/webBundleAnalysis.ts b/src/domain/webBundleAnalysis.ts index 27c79d1265..b4e4c11f8f 100644 --- a/src/domain/webBundleAnalysis.ts +++ b/src/domain/webBundleAnalysis.ts @@ -202,7 +202,12 @@ export const webBundleAnalysisSchema = z.object({ }), inferences: z.array( z.object({ - kind: z.enum(["vendor_fingerprint", "route_framework", "bundle_runtime"]), + kind: z.enum([ + "vendor_fingerprint", + "route_framework", + "bundle_runtime", + "endpoint_candidate", + ]), value: z.string(), confidence: z.enum(["low", "medium", "high"]), basis: z.array(basisSchema).min(1), diff --git a/src/domain/webBundleAnalyzer.test.ts b/src/domain/webBundleAnalyzer.test.ts index edcb37a209..ed0a151eb0 100644 --- a/src/domain/webBundleAnalyzer.test.ts +++ b/src/domain/webBundleAnalyzer.test.ts @@ -161,7 +161,9 @@ describe("web bundle analyzer", () => { trust: "page-declared-untrusted", }); }); +}); +describe("web bundle source-map coverage", () => { it("reports unavailable source maps as partial without dropping requested maps", () => { const analysis = analyzeCapturedWebBundle(inspection("export {};"), { status: "unavailable", @@ -191,6 +193,18 @@ describe("web bundle analyzer", () => { }); }); + it("keeps map-less captured source complete when no maps were requested", () => { + const analysis = analyzeCapturedWebBundle(inspection("export {};"), { + status: "not_requested", + requested: 0, + processed: 0, + items: [], + }); + + expect(analysis.completeness.status).toBe("complete"); + expect(analysis.unknowns).toEqual([]); + }); + it("rejects inconsistent source-map coverage counts", () => { const result = analyzeCapturedWebBundle(inspection("export {};")); expect( @@ -257,9 +271,15 @@ describe("web bundle static-analysis parity", () => { "/dynamic-method", "/shadowed-parent-xhr", "/xhr-get", - "api/relative", "https://xhr.example.test/submit", ]); + expect(result.inferences).toContainEqual( + expect.objectContaining({ + kind: "endpoint_candidate", + value: "api/relative", + confidence: "low", + }), + ); }); }); @@ -356,11 +376,252 @@ describe("web bundle API classification parity", () => { "/search", "/users", "/v1/headers", - "cache-key", "https://api.example.test/headers", "https://api.example.test/users", + ]); + expect( + result.inferences + .filter(({ kind }) => kind === "endpoint_candidate") + .map(({ value }) => value) + .sort(), + ).toEqual(["cache-key", "users"]); + }); + + it("keeps undici header-store reads out of endpoint observations", () => { + const result = analyzeCapturedWebBundle( + inspection(` + response.headersList.get("location", true); + request.headersList.delete("authorization"); + request.headersList.get("/header-list"); + api.get("/api/users"); + `), + ); + expect(result.observations.endpoints.map(({ value }) => value)).toEqual([ + "/header-list", + "/api/users", + ]); + expect( + result.inferences + .filter(({ kind }) => kind === "endpoint_candidate") + .map(({ value }) => value), + ).toEqual([]); + }); +}); + +describe("web bundle explicit request reference forms", () => { + it("recognizes URL, path, query and fragment forms after excluding lookup receivers", () => { + const references = [ + "http://api.example.test/users", + "https://api.example.test/users?q=1#part", + "ws://api.example.test/events", + "wss://api.example.test/events", + "http:api.example.test/users", + "https:api.example.test/users?q=1&tag=a&tag=b#part", + "https:api.example.test:443/a/../b?x=a+b&x=two#part", + "ws:api.example.test/events", + "wss:api.example.test/events", + "HtTpS:/api.example.test/users", + String.raw`https:\api.example.test\users`, + "https://[invalid", + "//api.example.test/users", + "/api/users", + "./users", + "../users", + "?", + "?page=2&tag=a&tag=b#section", + "#", + "#section", + "#frag?tag=a&tag=b", + ]; + const ambiguous = [ + "", + ".", + "..", + "users", + "%23section", + "%3Fpage=2", + " #section", + " https:api.example.test/users", + "https:", + "http:?page=2", + "ws:#section", + "https:[invalid", + "https:api.example.test:99999", + "ftp:api.example.test/users", + "file:/users", + "data:text/plain,users", + ]; + const source = ` + ${references.map((value) => `api.get(${JSON.stringify(value)});`).join("\n")} + ${ambiguous.map((value) => `api.get(${JSON.stringify(value)});`).join("\n")} + xhr.open(method, "?view=1"); + xhr.open(method, "#view=1"); + xhr.open("GET", "#literal=1"); + xhr.open(method, "https:api.example.test/xhr"); + request("POST", target); + snackBar.open(message, "X"); + new Map().get("?map-key"); + new Map().get("#map-key"); + new Map().get("https:api.example.test/map-key"); + new Headers().delete("?header-key"); + new Headers().delete("#header-key"); + new URLSearchParams().get("?parameter-key"); + new URLSearchParams().get("#parameter-key"); + response.headers.get("?response-key"); + response.headers.get("#response-key"); + const config = vscode.workspace.getConfiguration("git"); + config.get("?setting"); + config.get("#setting"); + config.get("https:api.example.test/setting"); + window.open(url, "?window-target"); + window.open(url, "#window-target"); + indexedDB.open(name, "?database-version"); + indexedDB.open(name, "#database-version"); + caches.open(name, "#cache-name"); + fs.open(path, "r"); + `; + const expected = [ + ...references, + "?view=1", + "#view=1", + "#literal=1", + "https:api.example.test/xhr", + ].sort(); + const expectedCandidates = [...ambiguous, "POST", "X"].sort(); + const web = analyzeCapturedWebBundle(inspection(source)); + expect(web.observations.endpoints.map(({ value }) => value).sort()).toEqual( + expected, + ); + expect( + web.inferences + .filter(({ kind }) => kind === "endpoint_candidate") + .map(({ value }) => value) + .sort(), + ).toEqual(expectedCandidates); + const analyzed = analyzeJavaScriptStaticSource(source); + expect( + analyzed.endpoints + .filter(({ interpretation }) => interpretation === undefined) + .map(({ value }) => value) + .sort(), + ).toEqual(expected); + expect( + analyzed.endpoints + .filter(({ interpretation }) => interpretation === "candidate") + .map(({ value }) => value) + .sort(), + ).toEqual(expectedCandidates); + }); +}); + +describe("web bundle endpoint candidate classification", () => { + it("keeps ambiguous HTTP-method arguments as candidates instead of endpoint observations", () => { + const source = ` + fetch("login"); + api.get("/api/users"); + api.get("users"); + form.get("passwordControl"); + translate.get("ADDRESS_ADDED"); + cookieService.put("token", value); + http.request("POST", target); + snackBar.open(message, "X"); + xhr.open(method, "login"); + xhr.open("GET", "ping"); + `; + const result = analyzeCapturedWebBundle(inspection(source)); + expect( + result.observations.endpoints.map(({ value }) => value).sort(), + ).toEqual(["/api/users", "login", "ping"]); + const candidates = result.inferences.filter( + ({ kind }) => kind === "endpoint_candidate", + ); + expect(candidates.map(({ value }) => value).sort()).toEqual([ + "ADDRESS_ADDED", + "POST", + "X", + "login", + "passwordControl", + "token", "users", ]); + expect( + candidates.every( + ({ confidence, basis }) => + confidence === "low" && + basis.length === 1 && + basis[0]?.artifact_sha256 === + createWebTextArtifact(source, "text/javascript").sha256 && + basis[0]?.line !== null && + basis[0]?.detector.startsWith("call:"), + ), + ).toBe(true); + const staticEndpoints = analyzeJavaScriptStaticSource(source).endpoints; + expect( + staticEndpoints + .filter(({ interpretation }) => interpretation === "candidate") + .map(({ value }) => value) + .sort(), + ).toEqual(candidates.map(({ value }) => value).sort()); + expect( + staticEndpoints + .filter(({ interpretation }) => interpretation === undefined) + .map(({ value }) => value) + .sort(), + ).toEqual(result.observations.endpoints.map(({ value }) => value).sort()); + expect(webBundleAnalysisSchema.parse(result)).toEqual(result); + }); + + it("deduplicates repeated candidate sites while keeping script, value and detector identity", () => { + const source = `${'form.get("name");\n'.repeat(5_000)}translate.get("name");\nform.get("other");`; + const captured = inspection(source); + const first = captured.scripts.items[0]; + if (first === undefined) throw new Error("Missing captured script fixture"); + const secondKey = `scr_${"2".repeat(64)}`; + const result = analyzeCapturedWebBundle({ + ...captured, + scripts: { + total: 2, + items: [ + first, + { + ...first, + script_key: secondKey, + url: `${origin}/assets/second.js`, + }, + ], + }, + }); + const candidates = result.inferences.filter( + ({ kind }) => kind === "endpoint_candidate", + ); + expect(candidates).toHaveLength(6); + for (const scriptKey of [first.script_key, secondKey]) { + const local = candidates.filter( + ({ basis }) => basis[0]?.script_key === scriptKey, + ); + expect( + local.map(({ value, basis }) => [value, basis[0]?.detector]), + ).toEqual([ + ["name", "call:form.get:unverified-network-receiver"], + ["name", "call:translate.get:unverified-network-receiver"], + ["other", "call:form.get:unverified-network-receiver"], + ]); + expect(local[0]?.basis[0]).toMatchObject({ + line: 1, + column: 0, + artifact_sha256: createWebTextArtifact(source, "text/javascript") + .sha256, + }); + } + expect(result.observations.endpoints).toEqual([]); + expect(result.capture.source_artifacts).toHaveLength(2); + expect(result.completeness).toMatchObject({ + status: "complete", + parsed_scripts: 2, + }); + expect(result.limitations).toContain( + 'Endpoint candidates are deduplicated by script, value and detector; basis retains only the first matching source coordinate. They are not a complete call-site inventory or a runtime occurrence count. completeness.status="complete" describes examination of captured source, not a call-site census.', + ); }); }); @@ -408,11 +669,16 @@ describe("web bundle configuration and navigation parity", () => { "shadowed", "users", ]; + const web = analyzeCapturedWebBundle(inspection(source)); + expect(web.observations.endpoints.map(({ value }) => value)).toEqual([ + "/v1/settings", + ]); expect( - analyzeCapturedWebBundle(inspection(source)) - .observations.endpoints.map(({ value }) => value) + web.inferences + .filter(({ kind }) => kind === "endpoint_candidate") + .map(({ value }) => value) .sort(), - ).toEqual(expected); + ).toEqual(expected.slice(1)); expect( analyzeJavaScriptStaticSource(source) .endpoints.map(({ value }) => value) @@ -420,6 +686,68 @@ describe("web bundle configuration and navigation parity", () => { ).toEqual(expected); }); + it("classifies proven VS Code extension state reads as non-endpoints", () => { + const reads = ` + context.globalState.get("lastVersion"); + this.workspaceState.get("closedRepositories", []); + context?.globalState.get("onboarded"); + context.globalState.get("/v1/state"); + context.secrets.get("token"); + api.get("users"); + `; + const analyze = (source: string) => { + const web = analyzeCapturedWebBundle(inspection(source)); + const staticAnalysis = analyzeJavaScriptStaticSource(source); + return { + web: { + endpoints: web.observations.endpoints + .map(({ value }) => value) + .sort(), + candidates: web.inferences + .filter(({ kind }) => kind === "endpoint_candidate") + .map(({ value }) => value) + .sort(), + }, + application: { + endpoints: staticAnalysis.endpoints + .filter(({ interpretation }) => interpretation === undefined) + .map(({ value }) => value) + .sort(), + candidates: staticAnalysis.endpoints + .filter(({ interpretation }) => interpretation === "candidate") + .map(({ value }) => value) + .sort(), + }, + }; + }; + const proven = { + endpoints: ["/v1/state"], + candidates: ["token", "users"], + }; + expect(analyze(`const vscode = require("vscode");${reads}`)).toEqual({ + web: proven, + application: proven, + }); + expect(analyze(`import * as vscode from "vscode";${reads}`)).toEqual({ + web: proven, + application: proven, + }); + const unproven = { + endpoints: ["/v1/state"], + candidates: [ + "closedRepositories", + "lastVersion", + "onboarded", + "token", + "users", + ], + }; + expect(analyze(`const vscode = require("./vscode");${reads}`)).toEqual({ + web: unproven, + application: unproven, + }); + }); + it("recognizes loadURL endpoints like static analysis", () => { const result = analyzeCapturedWebBundle( inspection('window.win.loadURL("https://embedded.test/app");'), @@ -588,6 +916,38 @@ const inspection = (source: string) => limitations: [], }); +it("recognizes expression-free template literals as route properties", () => { + const result = analyzeCapturedWebBundle( + inspection(` + const routes = [ + { path: \`score-board\`, component: ScoreBoard }, + { route: \`administration\`, component: Admin }, + { path: \`/user/\${name}\`, component: Dynamic }, + ]; + `), + ); + + expect(result.observations.routes.map(({ value }) => value)).toEqual([ + "score-board", + "administration", + ]); +}); + +it("uses the shared parser for decorated captured scripts", () => { + const result = analyzeCapturedWebBundle( + inspection(`@sealed class Widget {} fetch('/decorated');`), + ); + + expect(result.completeness).toMatchObject({ + status: "complete", + parsed_scripts: 1, + parse_failures: 0, + }); + expect(result.observations.endpoints).toEqual([ + expect.objectContaining({ value: "/decorated" }), + ]); +}); + it("keeps bare module locations unresolved without inventing URL-relative paths", () => { const result = analyzeCapturedWebBundle( inspection(` @@ -662,6 +1022,48 @@ describe("web metadata key semantics", () => { }); }); +it("retains WebMCP declarations through optional member and call chaining", () => { + const result = analyzeCapturedWebBundle( + inspection(` + document.modelContext?.registerTool({ name: "optional member" }); + document.modelContext.registerTool?.({ name: "optional call" }); + `), + ); + + expect(result.observations.webmcp_declarations).toEqual([ + expect.objectContaining({ + name: "optional member", + trust: "page-declared-untrusted", + }), + expect.objectContaining({ + name: "optional call", + trust: "page-declared-untrusted", + }), + ]); +}); + +it("reads expression-free template dynamic imports as chunk edges", () => { + const result = analyzeCapturedWebBundle( + inspection(` + import(\`./lazy.js\`); + import("./quoted.js"); + import(\`./\${name}.js\`); + `), + ); + + expect(result.observations.chunks.edges).toEqual([ + expect.objectContaining({ + kind: "dynamic_import", + specifier: "./lazy.js", + resolved_url: `${origin}/assets/lazy.js`, + }), + expect.objectContaining({ + kind: "dynamic_import", + specifier: "./quoted.js", + }), + ]); +}); + it("does not infer call names from dynamic computed identifiers", () => { const result = analyzeCapturedWebBundle( inspection(` diff --git a/src/domain/webBundleAnalyzerAst.test.ts b/src/domain/webBundleAnalyzerAst.test.ts new file mode 100644 index 0000000000..b3210d94fc --- /dev/null +++ b/src/domain/webBundleAnalyzerAst.test.ts @@ -0,0 +1,27 @@ +import * as t from "@babel/types"; +import { expect, it } from "vitest"; + +import { parseJavaScriptSource } from "./javascript/javascriptSourceParser.js"; +import { objectString } from "./webBundleAnalyzerAst.js"; + +it("reads WebMCP declaration strings through the exact-literal owner", () => { + const file = parseJavaScriptSource( + "const declaration = { name: `catalog`, description: `Browse catalog` }; const dynamic = { name: `catalog ${scope}` };", + ); + const declaration = file?.program.body[0]; + if (!t.isVariableDeclaration(declaration)) + throw new Error("Expected declaration fixture"); + const object = declaration.declarations[0]?.init; + if (!t.isObjectExpression(object)) + throw new Error("Expected object expression fixture"); + const dynamicDeclaration = file?.program.body[1]; + if (!t.isVariableDeclaration(dynamicDeclaration)) + throw new Error("Expected dynamic declaration fixture"); + const dynamic = dynamicDeclaration.declarations[0]?.init; + if (!t.isObjectExpression(dynamic)) + throw new Error("Expected dynamic object expression fixture"); + + expect(objectString(object, "name")).toBe("catalog"); + expect(objectString(object, "description")).toBe("Browse catalog"); + expect(objectString(dynamic, "name")).toBeUndefined(); +}); diff --git a/src/domain/webBundleAnalyzerAst.ts b/src/domain/webBundleAnalyzerAst.ts index 0decf4370a..ce98e1b959 100644 --- a/src/domain/webBundleAnalyzerAst.ts +++ b/src/domain/webBundleAnalyzerAst.ts @@ -1,7 +1,10 @@ import * as t from "@babel/types"; import { sanitizeBrowserUrl } from "./browserObservation.js"; -import { semanticStaticPropertyName } from "./javascript/javascriptAstValues.js"; +import { + readExactJavaScriptLiteral, + semanticStaticPropertyName, +} from "./javascript/javascriptAstValues.js"; /** Pick the endpoint argument for common network-call patterns. */ @@ -28,7 +31,11 @@ export const objectString = ( name: string, ): string | undefined => { const value = objectValue(object, name); - return t.isStringLiteral(value) ? value.value : undefined; + if (value === undefined) return undefined; + const literal = readExactJavaScriptLiteral(value); + return literal.found && typeof literal.value === "string" + ? literal.value + : undefined; }; /** Whether module syntax supplies a URL location without a package/import-map resolver. */ diff --git a/src/domain/webBundleAnalyzerInspection.ts b/src/domain/webBundleAnalyzerInspection.ts index 12c35109c8..93a4f9c708 100644 --- a/src/domain/webBundleAnalyzerInspection.ts +++ b/src/domain/webBundleAnalyzerInspection.ts @@ -1,18 +1,11 @@ -import { parse } from "@babel/parser"; import * as t from "@babel/types"; import { sanitizeEndpointCandidate } from "./browserObservation.js"; import { compositeKey } from "./unicodeCodePointOrder.js"; import type { WebPageInspection } from "./browserObservationSchemas.js"; import type { WebBundleAnalysis } from "./webBundleAnalysis.js"; -import { - completeSemanticSteps, - traverseJavaScriptAst, -} from "./javascript/javascriptSemanticTraversal.js"; -import { - classifyParsedJavaScriptOpenReceivers, - classifyParsedJavaScriptRpcClientRootsSteps, -} from "./javascript/javascriptSemanticAnalysis.js"; +import { traverseJavaScriptAst } from "./javascript/javascriptSemanticTraversal.js"; +import { classifyParsedJavaScriptReceivers } from "./javascript/javascriptSemanticAnalysis.js"; import type { JavaScriptOpenReceiverFact } from "./javascript/javascriptSemanticAnalysis.js"; import { semanticStaticPropertyName } from "./javascript/javascriptAstValues.js"; import { @@ -29,6 +22,7 @@ import { isUrlLikeModuleSpecifier, } from "./webBundleAnalyzerAst.js"; import { createJavaScriptRpcScan } from "./javascript/javascriptRpcEndpoints.js"; +import { parseJavaScriptSource } from "./javascript/javascriptSourceParser.js"; type BundleObservations = WebBundleAnalysis["observations"]; type ChunkEdge = BundleObservations["chunks"]["edges"][number]; @@ -58,16 +52,8 @@ export const analyzeScript = ( script: IncludedScript, accumulator: AnalysisAccumulator, ): void => { - let file: ReturnType; - try { - file = parse(script.source.artifact.text, { - sourceType: "unambiguous", - errorRecovery: true, - plugins: ["jsx", "typescript"], - }); - } catch (cause: unknown) { - // Unparseable scripts are counted; the failure needs no extra detail. - void cause; + const file = parseJavaScriptSource(script.source.artifact.text); + if (file === null) { accumulator.failedScripts.push(script.script_key); return; } @@ -75,10 +61,11 @@ export const analyzeScript = ( if ((file.errors?.length ?? 0) > 0) accumulator.recoveredScripts.push(script.script_key); detectVendorFingerprints(script, accumulator); - const openReceiverFacts = classifyParsedJavaScriptOpenReceivers(file); + const receivers = classifyParsedJavaScriptReceivers(file); + const openReceiverFacts = receivers.openReceivers; const rpc = createJavaScriptRpcScan( script.source.artifact.text, - completeSemanticSteps(classifyParsedJavaScriptRpcClientRootsSteps(file)), + receivers.rpcClientRoots, ); traverseJavaScriptAst(file, { enter: (node) => { @@ -123,14 +110,17 @@ const inspectNode = ( node, accumulator, }); - else if (t.isImportExpression(node) && t.isStringLiteral(node.source)) - addEdge({ - script, - specifier: node.source.value, - kind: "dynamic_import", - node, - accumulator, - }); + else if (t.isImportExpression(node)) { + const specifier = stringValue(node.source); + if (specifier !== undefined) + addEdge({ + script, + specifier, + kind: "dynamic_import", + node, + accumulator, + }); + } if (t.isObjectProperty(node)) inspectRouteProperty(script, node, accumulator); if ( t.isCallExpression(node) || @@ -189,19 +179,20 @@ const inspectCall = ( node.callee, openReceiverFacts.get(node.start ?? -1), ); - if (endpoint !== undefined) + if (endpoint?.kind === "candidate") + addEndpointCandidate(script, node, name, endpoint.value, accumulator); + else if (endpoint !== undefined) addFinding({ collection: accumulator.endpoints, script, - rawValue: endpoint, + rawValue: endpoint.value, mechanism: `call:${name}`, node, accumulator, }); if ( - (t.isCallExpression(node) || t.isNewExpression(node)) && - (name.endsWith("modelContext.registerTool") || - name === "modelContext.registerTool") + name.endsWith("modelContext.registerTool") || + name === "modelContext.registerTool" ) addWebMcpDeclaration(script, node, accumulator); }; @@ -212,11 +203,12 @@ const inspectRouteProperty = ( accumulator: AnalysisAccumulator, ): void => { const key = semanticStaticPropertyName(node.key, node.computed); - if ((key === "path" || key === "route") && t.isStringLiteral(node.value)) + const route = stringValue(node.value); + if ((key === "path" || key === "route") && route !== undefined) addFinding({ collection: accumulator.routes, script, - rawValue: node.value.value, + rawValue: route, mechanism: `property:${key}`, node, accumulator, @@ -257,6 +249,33 @@ interface AddFindingContext { readonly accumulator: AnalysisAccumulator; } +const addEndpointCandidate = ( + script: IncludedScript, + node: t.Node, + name: string, + rawValue: string, + accumulator: AnalysisAccumulator, +): void => { + const value = sanitizeEndpointCandidate(rawValue); + const detector = `call:${name}:unverified-network-receiver`; + const key = compositeKey([ + "endpoint-candidate", + script.script_key, + value, + detector, + ]); + // This is a per-script candidate inventory, not a callsite list. Keep the + // first representative source position without multiplying inline records. + addUnique(accumulator, key, () => + accumulator.inferences.push({ + kind: "endpoint_candidate", + value, + confidence: "low", + basis: [{ ...basis(script, detector), ...locationFields(node) }], + }), + ); +}; + const addFinding = (context: AddFindingContext): void => { const value = sanitizeEndpointCandidate(context.rawValue); const key = `finding\0${context.mechanism}\0${context.script.script_key}\0${value}`; @@ -271,7 +290,7 @@ const addFinding = (context: AddFindingContext): void => { const addWebMcpDeclaration = ( script: IncludedScript, - node: t.CallExpression | t.NewExpression, + node: t.CallExpression | t.OptionalCallExpression | t.NewExpression, accumulator: AnalysisAccumulator, ): void => { const declaration = node.arguments[0]; diff --git a/src/domain/webBundleAnalyzerResult.ts b/src/domain/webBundleAnalyzerResult.ts index 4d25096756..b34867f85e 100644 --- a/src/domain/webBundleAnalyzerResult.ts +++ b/src/domain/webBundleAnalyzerResult.ts @@ -201,6 +201,8 @@ const buildCompleteness = ( const bundleLimitations = (): WebBundleAnalysis["limitations"] => [ "Static bundle findings are observations or bounded inferences; REA does not execute captured JavaScript.", + "Bare string arguments of unbound HTTP-method-named calls are low-confidence endpoint candidates; form, translation, storage and other APIs can use the same syntax. URL-shaped findings still do not prove network execution.", + 'Endpoint candidates are deduplicated by script, value and detector; basis retains only the first matching source coordinate. They are not a complete call-site inventory or a runtime occurrence count. completeness.status="complete" describes examination of captured source, not a call-site census.', "String-built routes and endpoints, encrypted configuration, and server-side behavior may remain unknown.", "Page-declared WebMCP metadata is untrusted and is never registered or invoked as an REA tool.", ]; diff --git a/src/domain/webCaptureDiff.network.test.ts b/src/domain/webCaptureDiff.network.test.ts new file mode 100644 index 0000000000..cf1640a1c0 --- /dev/null +++ b/src/domain/webCaptureDiff.network.test.ts @@ -0,0 +1,679 @@ +import { describe, expect, it } from "vitest"; + +import { + passiveNetworkCapture, + withPassiveDroppedEvents, +} from "../../tests/fixtures/passiveNetworkCapture.js"; +import { + browserCaptureComparisonInputSchema, + compareBrowserCaptures, +} from "./browserCaptureComparison.js"; +import type { WebPageInspection } from "./browserObservationSchemas.js"; +import { webCaptureDiffSchema } from "./webCaptureDiffSchemas.js"; + +const compare = (before: WebPageInspection, after: WebPageInspection) => + webCaptureDiffSchema.parse( + compareBrowserCaptures( + browserCaptureComparisonInputSchema.parse({ + before: { inspection: before }, + after: { inspection: after }, + }), + ), + ).dimensions.network; + +const unselectedShapes = (capture: WebPageInspection): WebPageInspection => { + capture.completeness.excluded.push({ + section: "json_body_shapes", + reason: "not_approved", + count: null, + }); + for (const request of capture.network.requests) + request.body_shapes.status = "not_approved"; + return capture; +}; + +const redirectCapture = (resourceType: string | null): WebPageInspection => { + const capture = unselectedShapes(passiveNetworkCapture()); + capture.network.requests[0]!.redirects = [ + { + url: "https://app.example.test/start", + response_url: "https://app.example.test/start", + method: "GET", + resource_type: resourceType, + status: 302, + mime_type: "text/html", + encoded_data_length: 20, + request_timestamp: 1, + redirect_event_timestamp: 2, + }, + ]; + return capture; +}; + +describe("nullable redirect resource types", () => { + it.each([true, false])( + "retains uncertainty when a redirect type is omitted (reversed=%s)", + (reversed) => { + const before = redirectCapture("Document"); + const after = structuredClone(before); + after.network.requests[0]!.redirects![0]!.resource_type = null; + const [left, right] = reversed + ? ([after, before] as const) + : ([before, after] as const); + expect(compare(left, right)).toMatchObject({ + status: "unknown", + total_changes: 0, + changes: [], + }); + }, + ); + + it.each(["Document", "Fetch"])( + "keeps a known redirect type %s comparable", + (resourceType) => { + const before = redirectCapture("Document"); + const after = structuredClone(before); + after.network.requests[0]!.redirects![0]!.resource_type = resourceType; + expect(compare(before, after)).toMatchObject({ + status: resourceType === "Document" ? "unchanged" : "changed", + total_changes: resourceType === "Document" ? 0 : 1, + }); + }, + ); + + it("retains proven HTTP membership with incomplete redirect metadata", () => { + const before = unselectedShapes(passiveNetworkCapture([])); + const after = redirectCapture(null); + expect(compare(before, after)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ change: "added" }], + }); + }); +}); + +describe("passive network inventory changes", () => { + it.each(["websocket_connections", "websocket_frames"] as const)( + "does not infer duplicate WebSocket inventory changes from dropped %s", + (kind) => { + const before = unselectedShapes(passiveNetworkCapture()); + before.network.websocket_connections = [ + { + request_id: "ws-one", + url: "wss://app.example.test/live", + events: [], + }, + ]; + const after = structuredClone(before); + after.network.websocket_connections.push({ + request_id: "ws-two", + url: "wss://app.example.test/live", + events: [], + }); + const limited = withPassiveDroppedEvents(before, { [kind]: 1 }); + for (const [left, right] of [ + [limited, after], + [after, limited], + ] as const) + expect(compare(left, right)).toMatchObject({ + status: "unknown", + total_changes: 0, + changes: [], + }); + expect(compare(before, after)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ change: "modified" }], + }); + }, + ); + + it.each(["dropped", "truncated", "unavailable"] as const)( + "does not infer repeated-request count changes from %s HTTP inventory", + (limitation) => { + const after = unselectedShapes(passiveNetworkCapture(["/a", "/a"])); + let before = unselectedShapes(passiveNetworkCapture()); + if (limitation === "dropped") + before = withPassiveDroppedEvents(before, { network_requests: 1 }); + else if (limitation === "truncated") + before.completeness.truncated_sections.push("network_requests"); + else before.completeness.unavailable_sections.push("network_requests"); + for (const [left, right] of [ + [before, after], + [after, before], + ] as const) + expect(compare(left, right)).toMatchObject({ + status: "unknown", + total_changes: 0, + changes: [], + }); + }, + ); + + it.each(["unavailable", "partial", "truncated"] as const)( + "retains complete request additions/removals with %s selected shapes", + (coverage) => { + const before = passiveNetworkCapture(["/a"]); + const after = passiveNetworkCapture(["/a", "/b"]); + for (const capture of [before, after]) { + if (coverage === "truncated") + capture.completeness.truncated_sections.push("json_body_shapes"); + else capture.completeness.unavailable_sections.push("json_body_shapes"); + for (const request of capture.network.requests) { + const shape = { + root_type: "object" as const, + node_count: 1, + max_depth_observed: 0, + properties: [], + }; + if (coverage !== "unavailable") request.body_shapes.request = shape; + if (coverage === "truncated") request.body_shapes.response = shape; + request.body_shapes.status = + coverage === "truncated" ? "included" : coverage; + } + } + for (const [left, right, change] of [ + [before, after, "added"], + [after, before, "removed"], + ] as const) + expect(compare(left, right)).toEqual({ + status: "changed", + total_changes: 1, + changes: [{ identity: expect.stringMatching(/^net_/u), change }], + reason: null, + }); + expect(compare(after, after)).toMatchObject({ + status: "unknown", + total_changes: 0, + }); + }, + ); + + it("keeps request absence proof independent of WebSocket inventory and response metadata", () => { + const before = passiveNetworkCapture([]); + const after = passiveNetworkCapture(["/added"]); + for (const capture of [before, after]) { + capture.completeness.attach_limited_sections.push( + "websocket_connections", + "websocket_frames", + ); + for (const request of capture.network.requests) { + request.status = null; + request.mime_type = null; + request.encoded_data_length = null; + } + } + for (const [left, right, change] of [ + [before, after, "added"], + [after, before, "removed"], + ] as const) + expect(compare(left, right)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ change }], + }); + expect(compare(after, after)).toMatchObject({ + status: "unknown", + total_changes: 0, + }); + }); +}); + +describe("passive network coverage uncertainty", () => { + it.each([ + "attach_limited_sections", + "unavailable_sections", + "truncated_sections", + "policy_filtered_sections", + ] as const)( + "does not infer request absence when its inventory is %s", + (section) => { + const before = passiveNetworkCapture(); + const after = passiveNetworkCapture(["/a", "/b"]); + after.completeness[section].push("network_requests"); + for (const [left, right] of [ + [before, after], + [after, before], + ]) + expect(compare(left!, right!)).toMatchObject({ + status: "unknown", + total_changes: 0, + }); + after.network.requests[0]!.status = 503; + expect(compare(before, after)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ change: "modified" }], + }); + }, + ); + + it("does not promote unavailable shapes to a modification and preserves unselected-shape behavior", () => { + const before = passiveNetworkCapture(); + const after = structuredClone(before); + after.network.requests[0]!.body_shapes = { + status: "included", + request: { + root_type: "object", + node_count: 1, + max_depth_observed: 0, + properties: [], + }, + response: { + root_type: "string", + node_count: 1, + max_depth_observed: 0, + properties: [], + }, + }; + expect(compare(before, after)).toMatchObject({ + status: "unknown", + total_changes: 0, + }); + before.completeness.excluded.push({ + section: "json_body_shapes", + reason: "not_approved", + count: null, + }); + expect(compare(before, after)).toMatchObject({ + status: "unchanged", + total_changes: 0, + }); + }); +}); + +describe("independent WebSocket inventory evidence", () => { + it("does not use complete HTTP inventory to infer an unobserved WebSocket connection", () => { + const before = passiveNetworkCapture(); + const after = structuredClone(before); + after.network.websocket_connections.push({ + request_id: "ws", + url: "wss://app.example.test/live", + events: [], + }); + before.completeness.attach_limited_sections.push("websocket_connections"); + expect(compare(before, after)).toMatchObject({ + status: "unknown", + total_changes: 0, + }); + expect(compare(after, before)).toMatchObject({ + status: "unknown", + total_changes: 0, + }); + after.network.requests.push({ + ...passiveNetworkCapture(["/b"]).network.requests[0]!, + request_id: "request-b", + }); + expect(compare(before, after)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ identity: expect.stringMatching(/^net_/u), change: "added" }], + }); + }); + + it("retains complete WebSocket membership changes when HTTP inventory is limited", () => { + const before = passiveNetworkCapture(); + before.completeness.attach_limited_sections.push("network_requests"); + const after = structuredClone(before); + after.network.websocket_connections.push({ + request_id: "ws", + url: "wss://app.example.test/live", + events: [], + }); + for (const [left, right, change] of [ + [before, after, "added"], + [after, before, "removed"], + ] as const) + expect(compare(left, right)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ identity: expect.stringMatching(/^ws_/u), change }], + }); + }); + + it("does not turn missing WebSocket payload facts into membership changes", () => { + const before = passiveNetworkCapture(); + before.network.websocket_connections.push({ + request_id: "ws", + url: "wss://app.example.test/live", + events: [ + { + direction: "sent", + opcode: 1, + payload_bytes: 4, + payload_shape: { format: "text", json_shape: null }, + }, + ], + }); + const after = structuredClone(before); + const event = after.network.websocket_connections[0]!.events[0]!; + event.payload_bytes = null; + event.payload_shape = null; + expect(compare(before, after)).toMatchObject({ + status: "unknown", + total_changes: 0, + }); + after.network.requests.push({ + ...passiveNetworkCapture(["/b"]).network.requests[0]!, + request_id: "request-b", + }); + expect(compare(before, after)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ identity: expect.stringMatching(/^net_/u), change: "added" }], + }); + }); +}); + +describe("repeated HTTP request identity uncertainty", () => { + it.each([true, false])( + "keeps repeated-request regrouping unknown when one type is missing (shapes selected=%s)", + (selected) => { + const before = passiveNetworkCapture(["/a", "/a"]); + const after = structuredClone(before); + after.network.requests[1]!.resource_type = null; + if (!selected) + for (const capture of [before, after]) unselectedShapes(capture); + for (const [left, right] of [ + [before, after], + [after, before], + ] as const) + expect(compare(left, right)).toMatchObject({ + status: "unknown", + total_changes: 0, + changes: [], + }); + }, + ); + + it("retains a known repeated-request count change", () => { + const before = unselectedShapes(passiveNetworkCapture(["/a", "/a"])); + const after = unselectedShapes(passiveNetworkCapture()); + for (const [left, right] of [ + [before, after], + [after, before], + ] as const) + expect(compare(left, right)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ change: "modified" }], + }); + }); + + it("does not assign observed content to an ambiguous type bucket", () => { + const before = unselectedShapes(passiveNetworkCapture(["/a", "/a"])); + before.network.requests[1]!.resource_type = null; + before.network.requests[1]!.status = 503; + const after = structuredClone(before); + after.network.requests[0]!.status = 503; + after.network.requests[1]!.status = 200; + for (const [left, right] of [ + [before, after], + [after, before], + ] as const) + expect(compare(left, right)).toMatchObject({ + status: "unknown", + total_changes: 0, + changes: [], + }); + }); +}); + +describe("nullable HTTP request identities", () => { + it.each([true, false])( + "keeps known/unknown resource-type matches uncertain (shapes selected=%s)", + (selected) => { + const before = passiveNetworkCapture(); + const after = structuredClone(before); + after.network.requests[0]!.resource_type = null; + if (!selected) + for (const capture of [before, after]) unselectedShapes(capture); + for (const [left, right] of [ + [before, after], + [after, before], + ] as const) + expect(compare(left, right)).toMatchObject({ + status: "unknown", + total_changes: 0, + changes: [], + reason: expect.stringContaining("request identity"), + }); + }, + ); + + it("does not prove equality from two unknown request types", () => { + const capture = unselectedShapes(passiveNetworkCapture()); + capture.network.requests[0]!.resource_type = null; + expect(compare(capture, structuredClone(capture))).toMatchObject({ + status: "unknown", + total_changes: 0, + reason: expect.stringContaining("request identity"), + }); + }); + + it("does not assign an unknown type to one of multiple same-method/URL candidates", () => { + const before = unselectedShapes(passiveNetworkCapture(["/a", "/a"])); + before.network.requests[1]!.resource_type = "Fetch"; + const after = structuredClone(before); + after.network.requests[1]!.resource_type = null; + for (const [left, right] of [ + [before, after], + [after, before], + ] as const) + expect(compare(left, right)).toMatchObject({ + status: "unknown", + total_changes: 0, + }); + }); + + it.each([ + ["/b", "GET"], + ["/a", "POST"], + ])( + "retains a distinct %s %s request beside ambiguous identities", + (path, method) => { + const before = unselectedShapes(passiveNetworkCapture()); + const after = unselectedShapes(passiveNetworkCapture(["/a", path])); + after.network.requests[0]!.resource_type = null; + after.network.requests[1]!.method = method; + for (const [left, right, change] of [ + [before, after, "added"], + [after, before, "removed"], + ] as const) + expect(compare(left, right)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ identity: expect.stringMatching(/^net_/u), change }], + }); + }, + ); + + it("uses a complete empty peer inventory to prove absence despite an unknown type", () => { + const empty = unselectedShapes(passiveNetworkCapture([])); + const observed = unselectedShapes(passiveNetworkCapture()); + observed.network.requests[0]!.resource_type = null; + for (const [left, right, change] of [ + [empty, observed, "added"], + [observed, empty, "removed"], + ] as const) + expect(compare(left, right)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ change }], + }); + expect(compare(empty, empty)).toMatchObject({ + status: "unchanged", + total_changes: 0, + }); + }); + + it("retains definite type changes and independently observed metadata changes", () => { + const before = unselectedShapes(passiveNetworkCapture()); + const changedType = structuredClone(before); + changedType.network.requests[0]!.resource_type = "Fetch"; + expect(compare(before, changedType)).toMatchObject({ + status: "changed", + total_changes: 2, + changes: expect.arrayContaining([ + expect.objectContaining({ change: "added" }), + expect.objectContaining({ change: "removed" }), + ]), + }); + before.network.requests.push({ + ...passiveNetworkCapture(["/b"]).network.requests[0]!, + resource_type: null, + }); + const changedStatus = structuredClone(before); + changedStatus.network.requests[0]!.status = 503; + expect(compare(before, changedStatus)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ change: "modified" }], + }); + }); +}); + +describe("passive dropped-event inventory coverage", () => { + it("does not infer request absence from a capture with independently counted drops", () => { + const before = withPassiveDroppedEvents(passiveNetworkCapture(), { + network_requests: 1, + }); + const after = passiveNetworkCapture(["/a", "/b"]); + expect(before.completeness).toMatchObject({ + status: "truncated", + truncated_sections: [], + unavailable_sections: [], + attach_limited_sections: [], + policy_filtered_sections: [], + dropped_events: { network_requests: 1, total: 1 }, + }); + for (const [left, right] of [ + [before, after], + [after, before], + ] as const) + expect(compare(left, right)).toMatchObject({ + status: "unknown", + total_changes: 0, + }); + const changed = structuredClone(before); + changed.network.requests[0]!.status = 503; + expect(compare(before, changed)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ change: "modified" }], + }); + }); + + it.each([ + "network_requests", + "websocket_connections", + "websocket_frames", + ] as const)( + "cannot prove network equality after dropped %s even when shapes are unselected", + (kind) => { + const before = withPassiveDroppedEvents( + unselectedShapes(passiveNetworkCapture()), + { [kind]: 1 }, + ); + expect(before.completeness.truncated_sections).toEqual([]); + expect(compare(before, structuredClone(before))).toMatchObject({ + status: "unknown", + total_changes: 0, + }); + }, + ); + + it.each(["websocket_connections", "websocket_frames"] as const)( + "keeps %s drops separate from HTTP inventory", + (kind) => { + const before = withPassiveDroppedEvents(passiveNetworkCapture(), { + [kind]: 1, + }); + const after = structuredClone(before); + after.network.websocket_connections.push({ + request_id: "ws", + url: "wss://app.example.test/live", + events: [], + }); + for (const [left, right] of [ + [before, after], + [after, before], + ] as const) + expect(compare(left, right)).toMatchObject({ + status: "unknown", + total_changes: 0, + }); + after.network.requests.push({ + ...passiveNetworkCapture(["/b"]).network.requests[0]!, + request_id: "request-b", + }); + for (const [left, right, change] of [ + [before, after, "added"], + [after, before, "removed"], + ] as const) + expect(compare(left, right)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ identity: expect.stringMatching(/^net_/u), change }], + }); + }, + ); + + it("retains complete WebSocket membership despite HTTP drops", () => { + const before = withPassiveDroppedEvents(passiveNetworkCapture(), { + network_requests: 1, + }); + const after = structuredClone(before); + after.network.websocket_connections.push({ + request_id: "ws", + url: "wss://app.example.test/live", + events: [], + }); + expect(compare(before, after)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ identity: expect.stringMatching(/^ws_/u), change: "added" }], + }); + expect(compare(after, before)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [ + { identity: expect.stringMatching(/^ws_/u), change: "removed" }, + ], + }); + }); + + it.each([ + "console_events", + "scripts", + "timeline_events", + "webmcp_tools", + ] as const)( + "does not use the total or unrelated %s drops as HTTP absence limits", + (kind) => { + const before = withPassiveDroppedEvents( + unselectedShapes(passiveNetworkCapture()), + { [kind]: 2 }, + ); + expect(before.completeness).toMatchObject({ + status: "truncated", + dropped_events: { total: 2 }, + }); + expect(compare(before, structuredClone(before))).toMatchObject({ + status: "unchanged", + total_changes: 0, + }); + const after = unselectedShapes(passiveNetworkCapture(["/a", "/b"])); + expect(compare(before, after)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ change: "added" }], + }); + expect(compare(after, before)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ change: "removed" }], + }); + }, + ); +}); diff --git a/src/domain/webCaptureDiff.ts b/src/domain/webCaptureDiff.ts index d8ce6d9e21..de24a0fc3d 100644 --- a/src/domain/webCaptureDiff.ts +++ b/src/domain/webCaptureDiff.ts @@ -1,5 +1,8 @@ import { digestCanonicalValue } from "./canonicalDigest.js"; -import { compareUnicodeCodePoints } from "./unicodeCodePointOrder.js"; +import { + compareUnicodeCodePoints, + compositeKey, +} from "./unicodeCodePointOrder.js"; import type { WebPageInspection } from "./browserObservationSchemas.js"; import { webCaptureDiffSchema, @@ -9,6 +12,7 @@ import { type WebCaptureDimension, } from "./webCaptureDiffSchemas.js"; import type { WebMcpDiscovery } from "./webMcpDiscovery.js"; +import { fingerprintWebCaptureFrames } from "./webCaptureFrameComparison.js"; type Dimension = WebCaptureDimension; type Change = WebCaptureChange; @@ -70,11 +74,15 @@ const compareDimension = ( after: ReadonlyMap, complete: boolean, reason: string, - inventoryComplete = complete, + inventoryComplete: boolean | ((identity: string) => boolean) = complete, ): Dimension => { // Membership evidence can be complete even when selected content is unknown. const all = compareIdentities(before, after).filter( - ({ change }) => inventoryComplete || change === "modified", + ({ identity, change }) => + change === "modified" || + (typeof inventoryComplete === "boolean" + ? inventoryComplete + : inventoryComplete(identity)), ); if (all.length > 0) return { @@ -172,13 +180,7 @@ const compareWebCaptureDimensions = ( const before = input.before.inspection; const after = input.after.inspection; return { - dom_structure: compareDimension( - singleton("document", digestCanonicalValue(domProjection(before))), - singleton("document", digestCanonicalValue(domProjection(after))), - sectionsComplete(before, ["frames", "dom"]) && - sectionsComplete(after, ["frames", "dom"]), - "DOM or frame capture was incomplete in at least one observation.", - ), + dom_structure: domDimension(before, after), scripts: compareDimension( keyed( before.scripts.items.map(scriptProjection), @@ -200,19 +202,15 @@ const compareWebCaptureDimensions = ( "Resource inventory was incomplete in at least one observation.", ), network: networkDimension(before, after), - metadata: compareDimension( - singleton("metadata", digestCanonicalValue(metadataProjection(before))), - singleton("metadata", digestCanonicalValue(metadataProjection(after))), + metadata: compareAggregate( + "metadata", + metadataProjection(before), + metadataProjection(after), sectionsComplete(before, ["metadata"]) && sectionsComplete(after, ["metadata"]), "Safe metadata capture was incomplete in at least one observation.", ), - webmcp: compareDimension( - webMcpMap(input.before.webmcp), - webMcpMap(input.after.webmcp), - webMcpComplete(input.before.webmcp) && webMcpComplete(input.after.webmcp), - "WebMCP discovery was unavailable or incomplete in at least one capture.", - ), + webmcp: webMcpDimension(input.before.webmcp, input.after.webmcp), accessibility: accessibilityDimension(before, after), storage: storageDimension(before, after), }; @@ -252,40 +250,145 @@ const networkIdentity = ( ): string => `net_${digestCanonicalValue({ method: request.method, url: request.url, resource_type: request.resource_type })}`; +/** An unknown request type can match any observed type at the same method/URL. */ +const networkIdentityCoverage = ( + before: WebPageInspection, + after: WebPageInspection, +): { + readonly complete: boolean; + readonly ambiguousIdentities: ReadonlySet; +} => { + const complete = [before, after].every((inspection) => + inspection.network.requests.every( + ({ resource_type }) => resource_type !== null, + ), + ); + const ambiguousIdentities = new Set(); + if (complete) return { complete, ambiguousIdentities }; + type Request = WebPageInspection["network"]["requests"][number]; + const matchKey = (request: Request): string => + JSON.stringify([request.method, request.url]); + const unknownTypes = ( + inspection: WebPageInspection, + ): ReadonlyMap => { + const groups = new Map(); + for (const request of inspection.network.requests) { + const key = matchKey(request); + groups.set( + key, + groups.get(key) === true || request.resource_type === null, + ); + } + return groups; + }; + const retainPotentialMatches = ( + inspection: WebPageInspection, + peer: ReadonlyMap, + ): void => { + for (const request of inspection.network.requests) { + const peerUnknown = peer.get(matchKey(request)); + if ( + peerUnknown === true || + (peerUnknown === false && request.resource_type === null) + ) + ambiguousIdentities.add(networkIdentity(request)); + } + }; + retainPotentialMatches(before, unknownTypes(after)); + retainPotentialMatches(after, unknownTypes(before)); + return { complete, ambiguousIdentities }; +}; + const networkResponseFields = [ "status", "mime_type", "encoded_data_length", ] as const; -type NetworkResponseField = (typeof networkResponseFields)[number]; +const redirectMetadataFields = [ + ...networkResponseFields, + "resource_type", +] as const; +type NetworkMetadataField = (typeof redirectMetadataFields)[number]; -const sharedNetworkResponseFields = ( +/** Own both comparable fields and completeness instead of deriving it from set sizes. */ +const sharedNetworkMetadata = ( before: WebPageInspection, after: WebPageInspection, -): ReadonlyMap> => { - const shared = new Map>(); +): { + readonly fields: ReadonlyMap>; + readonly complete: boolean; +} => { + const shared = new Map>(); + let complete = true; + const retainKnownFields = ( + identity: string, + metadata: Pick< + WebPageInspection["network"]["requests"][number], + NetworkMetadataField + >, + required: readonly NetworkMetadataField[], + ): void => { + const fields = shared.get(identity) ?? new Set(required); + for (const field of required) + if (metadata[field] === null) { + fields.delete(field); + complete = false; + } + shared.set(identity, fields); + }; for (const inspection of [before, after]) for (const request of inspection.network.requests) { const identity = networkIdentity(request); - const fields = shared.get(identity) ?? new Set(networkResponseFields); - for (const field of networkResponseFields) - if (request[field] === null) fields.delete(field); - shared.set(identity, fields); + retainKnownFields(identity, request, networkResponseFields); + for (const [index, redirect] of (request.redirects ?? []).entries()) + retainKnownFields( + redirectNetworkIdentity(identity, index), + redirect, + redirectMetadataFields, + ); } - return shared; + return { fields: shared, complete }; }; +const redirectNetworkIdentity = (identity: string, index: number): string => + compositeKey([identity, "redirect", index]); +/** Section lists and per-kind dropped counts independently limit absence evidence. */ +const networkInventoryComplete = ( + inspection: WebPageInspection, + sections: readonly ( + | "network_requests" + | "websocket_connections" + | "websocket_frames" + )[], +): boolean => + sectionsComplete(inspection, sections) && + sections.every( + (section) => inspection.completeness.dropped_events[section] === 0, + ); + const networkDimension = ( before: WebPageInspection, after: WebPageInspection, ): Dimension => { - const responseFields = sharedNetworkResponseFields(before, after); - const responseFieldsComplete = [...responseFields.values()].every( - (fields) => fields.size === networkResponseFields.length, + const requestsComplete = [before, after].every((inspection) => + networkInventoryComplete(inspection, ["network_requests"]), ); + const webSocketInventoryComplete = [before, after].every((inspection) => + networkInventoryComplete(inspection, [ + "websocket_connections", + "websocket_frames", + ]), + ); + const requestIdentities = networkIdentityCoverage(before, after); + const metadata = sharedNetworkMetadata(before, after); const selected = bodyShapesSelected(before) && bodyShapesSelected(after); const webSocketShapes = webSocketShapesSelected(before) && webSocketShapesSelected(after); + const projection = { + metadataFields: metadata.fields, + ambiguousIdentities: requestIdentities.ambiguousIdentities, + inventoryComplete: requestsComplete, + }; const sources = new Map(); if (selected) { const observedSources = (inspection: WebPageInspection) => { @@ -345,74 +448,118 @@ const networkDimension = ( ); const compared = compareDimension( new Map([ - ...networkMap(before, sources, responseFields), - ...webSocketMetadataMap(beforeWebSockets), + ...networkMap(before, sources, projection), + ...(webSocketInventoryComplete + ? webSocketMetadataMap(beforeWebSockets) + : []), ...beforeWebSocketPayload, ]), new Map([ - ...networkMap(after, sources, responseFields), - ...webSocketMetadataMap(afterWebSockets), + ...networkMap(after, sources, projection), + ...(webSocketInventoryComplete + ? webSocketMetadataMap(afterWebSockets) + : []), ...afterWebSocketPayload, ]), - [before, after].every((inspection) => - sectionsComplete(inspection, [ - "network_requests", - "websocket_connections", - "websocket_frames", - ]), - ) && + requestsComplete && + requestIdentities.complete && + webSocketInventoryComplete && shapesComplete && webSocketPayloadBytesComplete && webSocketShapesComplete, - "Network capture or selected HTTP/WebSocket payload-shape coverage is attach-window limited, unavailable, or incomplete.", + requestIdentities.complete + ? "Network capture or selected HTTP/WebSocket payload-shape coverage is limited, unavailable, incomplete, or includes dropped network events." + : "HTTP request identity is incomplete because a resource type is unavailable; potentially matching type groups cannot establish membership/content changes or complete network equality.", + // HTTP and WebSocket identities have independent inventory coverage. Shape + // selection can prevent equality without erasing proven membership changes. + (identity) => + identity.startsWith("net_") + ? requestsComplete + : identity.startsWith("ws_") && webSocketInventoryComplete, ); - return compared.status === "unchanged" && !responseFieldsComplete - ? { - ...compared, - status: "unknown", - reason: - "HTTP response status, MIME type, or byte-count evidence is incomplete.", - } - : compared; + if (metadata.complete) return compared; + const responseEvidenceReason = + "HTTP response or redirect status, MIME type, byte count, or resource-type evidence is incomplete."; + if (compared.status === "changed") return compared; + return { + status: "unknown", + total_changes: 0, + changes: [], + reason: + compared.reason === null + ? responseEvidenceReason + : `${responseEvidenceReason} ${compared.reason}`, + }; }; const networkMap = ( inspection: WebPageInspection, sources: ReadonlyMap, - responseFields: ReadonlyMap>, + projection: { + readonly metadataFields: ReadonlyMap< + string, + ReadonlySet + >; + readonly ambiguousIdentities: ReadonlySet; + readonly inventoryComplete: boolean; + }, ): ReadonlyMap => { - const grouped = new Map(); + const grouped = new Map(); for (const request of inspection.network.requests) { const identity = networkIdentity(request); + // A missing type can move repeated requests between these buckets. Neither + // their counts nor fingerprints establish a change to that typed group. + if (projection.ambiguousIdentities.has(identity)) continue; const shapes = sources.get(identity); const values = grouped.get(identity) ?? []; const response: Record = {}; - for (const field of responseFields.get(identity) ?? []) + for (const field of projection.metadataFields.get(identity) ?? []) response[field] = request[field]; - values.push({ - ...response, - redirects: (request.redirects ?? []).map( - ({ - request_timestamp: _requestTimestamp, - redirect_event_timestamp: _redirectEventTimestamp, - ...redirect - }) => redirect, - ), - initiator: request.initiator, - ...(shapes?.request === true - ? { request_shape: request.body_shapes.request } - : {}), - ...(shapes?.response === true - ? { response_shape: request.body_shapes.response } - : {}), - }); + values.push( + digestCanonicalValue({ + ...response, + redirects: (request.redirects ?? []).map( + ( + { + request_timestamp: _requestTimestamp, + redirect_event_timestamp: _redirectEventTimestamp, + ...redirect + }, + index, + ) => { + const response: Partial< + Record + > = {}; + for (const field of projection.metadataFields.get( + redirectNetworkIdentity(identity, index), + ) ?? []) + if (redirect[field] !== null) response[field] = redirect[field]; + return { + url: redirect.url, + response_url: redirect.response_url, + method: redirect.method, + ...response, + }; + }, + ), + initiator: request.initiator, + ...(shapes?.request === true + ? { request_shape: request.body_shapes.request } + : {}), + ...(shapes?.response === true + ? { response_shape: request.body_shapes.response } + : {}), + }), + ); grouped.set(identity, values); } return new Map( [...grouped].map(([identity, values]) => [ identity, digestCanonicalValue( - values.map((value) => digestCanonicalValue(value)).sort(), + // Repeated samples do not establish multiplicity across omissions; + // keep their known content without interpreting missing rows as change. + (projection.inventoryComplete ? values : [...new Set(values)]).sort(), ), ]), ); @@ -532,12 +679,51 @@ const webSocketPayloadMaps = ( return [left, right]; }; -const webMcpMap = ( - discovery: WebMcpDiscovery | null, -): ReadonlyMap => - discovery === null - ? new Map() - : keyed(discovery.tools.items, (tool) => tool.tool_key); +const webMcpDimension = ( + before: WebMcpDiscovery | null, + after: WebMcpDiscovery | null, +): Dimension => { + const afterTools = new Map( + after?.tools.items.map((tool) => [tool.tool_key, tool]), + ); + const unknownHints = new Set( + (before?.tools.items ?? []) + .filter((tool) => { + const other = afterTools.get(tool.tool_key); + return ( + other !== undefined && + (tool.annotations.consequential === undefined || + other.annotations.consequential === undefined) + ); + }) + .map((tool) => tool.tool_key), + ); + const project = ( + discovery: WebMcpDiscovery | null, + ): ReadonlyMap => + keyed( + (discovery?.tools.items ?? []).map((tool) => { + if (!unknownHints.has(tool.tool_key)) return tool; + const { consequential: _unknown, ...annotations } = tool.annotations; + return { ...tool, annotations }; + }), + (tool) => tool.tool_key, + ); + const result = compareDimension( + project(before), + project(after), + webMcpComplete(before) && webMcpComplete(after), + "WebMCP discovery was unavailable or incomplete in at least one capture.", + ); + return result.status === "unchanged" && unknownHints.size > 0 + ? { + ...result, + status: "unknown", + reason: + "Consequential annotations were not recorded or were invalid in at least one capture.", + } + : result; +}; const webMcpComplete = (discovery: WebMcpDiscovery | null): boolean => discovery !== null && @@ -565,17 +751,64 @@ const incompleteSections = (completeness: { ...completeness.unavailable_sections, ]); -const domProjection = (inspection: WebPageInspection) => ({ - frames: inspection.frames - .map(({ url, origin }) => ({ url, origin })) - .sort( - (left, right) => - compareUnicodeCodePoints(left.url, right.url) || - compareUnicodeCodePoints(left.origin ?? "", right.origin ?? "") || - Number(left.origin !== null) - Number(right.origin !== null), - ), - nodes: inspection.dom.nodes.map(({ index: _index, ...node }) => node), -}); +/** Whole-collection digest differences cannot prove a change across omissions. */ +const compareAggregate = ( + identity: string, + before: unknown, + after: unknown, + complete: boolean, + reason: string, +): Dimension => + compareDimension( + complete ? singleton(identity, digestCanonicalValue(before)) : new Map(), + complete ? singleton(identity, digestCanonicalValue(after)) : new Map(), + complete, + reason, + ); + +const domDimension = ( + before: WebPageInspection, + after: WebPageInspection, +): Dimension => { + const beforeFrames = fingerprintWebCaptureFrames(before.frames); + const afterFrames = fingerprintWebCaptureFrames(after.frames); + const nodesComplete = [before, after].every( + (inspection) => + sectionsComplete(inspection, ["dom"]) && + inspection.dom.total_nodes === inspection.dom.nodes.length, + ); + const framesComplete = + beforeFrames !== null && + afterFrames !== null && + [before, after].every((inspection) => + sectionsComplete(inspection, ["frames"]), + ); + const changes: Change[] = []; + if ( + nodesComplete && + digestCanonicalValue(before.dom.nodes) !== + digestCanonicalValue(after.dom.nodes) + ) + changes.push({ identity: "document", change: "modified" }); + if (framesComplete && beforeFrames !== afterFrames) + changes.push({ identity: "frames", change: "modified" }); + if (changes.length > 0) + return { + status: "changed", + total_changes: changes.length, + changes, + reason: null, + }; + return nodesComplete && framesComplete + ? { status: "unchanged", total_changes: 0, changes: [], reason: null } + : { + status: "unknown", + total_changes: 0, + changes: [], + reason: + "DOM or frame capture was incomplete, or frame ancestry was unresolved in at least one observation.", + }; +}; const scriptProjection = ( script: WebPageInspection["scripts"]["items"][number], diff --git a/src/domain/webCaptureFrameComparison.test.ts b/src/domain/webCaptureFrameComparison.test.ts new file mode 100644 index 0000000000..bd70649a96 --- /dev/null +++ b/src/domain/webCaptureFrameComparison.test.ts @@ -0,0 +1,62 @@ +import { describe, expect, it } from "vitest"; + +import type { WebPageInspection } from "./browserObservationSchemas.js"; +import { fingerprintWebCaptureFrames } from "./webCaptureFrameComparison.js"; + +const frame = ( + id: string, + parent: string | null = null, + url = "https://app.example.test/frame", +): WebPageInspection["frames"][number] => ({ + frame_id: id, + parent_frame_id: parent, + url, + origin: "https://app.example.test", +}); + +describe("capture-local frame graph normalization", () => { + it.each([ + [frame("child", "missing")], + [frame("same"), frame("same")], + [frame("a", "b"), frame("b", "a")], + [frame("self", "self")], + ])("keeps unresolved ancestry unknown (%#)", (...frames) => { + expect(fingerprintWebCaptureFrames(frames)).toBeNull(); + }); + + it("preserves multiplicity while ignoring sibling order and frame IDs", () => { + const left = [frame("root"), frame("a", "root"), frame("b", "root")]; + const renamed = [frame("y", "new"), frame("new"), frame("x", "new")]; + expect(fingerprintWebCaptureFrames(left)).toBe( + fingerprintWebCaptureFrames(renamed), + ); + expect(fingerprintWebCaptureFrames(left)).not.toBe( + fingerprintWebCaptureFrames(left.slice(0, 2)), + ); + expect(fingerprintWebCaptureFrames(left)).not.toBe( + fingerprintWebCaptureFrames([ + frame("root"), + frame("a", "root"), + frame("b", "a"), + ]), + ); + }); + + it("handles deep ancestry without recursive serialization", () => { + const frames = Array.from({ length: 5000 }, (_, index) => + frame(String(index), index === 0 ? null : String(index - 1)), + ); + const renamed = frames + .map((item) => ({ + ...item, + frame_id: "new-" + item.frame_id, + parent_frame_id: + item.parent_frame_id === null ? null : "new-" + item.parent_frame_id, + })) + .reverse(); + expect(fingerprintWebCaptureFrames(frames)).toMatch(/^[a-f0-9]{64}$/u); + expect(fingerprintWebCaptureFrames(frames)).toBe( + fingerprintWebCaptureFrames(renamed), + ); + }); +}); diff --git a/src/domain/webCaptureFrameComparison.ts b/src/domain/webCaptureFrameComparison.ts new file mode 100644 index 0000000000..e85f03e2c1 --- /dev/null +++ b/src/domain/webCaptureFrameComparison.ts @@ -0,0 +1,49 @@ +import type { WebPageInspection } from "./browserObservationSchemas.js"; +import { digestCanonicalValue } from "./canonicalDigest.js"; + +/** Compare complete frame forests without treating capture-local IDs as identity. */ +export const fingerprintWebCaptureFrames = ( + frames: WebPageInspection["frames"], +): string | null => { + const byId = new Map(frames.map((frame) => [frame.frame_id, frame])); + if (byId.size !== frames.length) return null; + const remaining = new Map(frames.map((frame) => [frame.frame_id, 0])); + for (const frame of frames) { + if (frame.parent_frame_id === null) continue; + const count = remaining.get(frame.parent_frame_id); + if (count === undefined) return null; + remaining.set(frame.parent_frame_id, count + 1); + } + const ready = frames.filter((frame) => remaining.get(frame.frame_id) === 0); + const children = new Map(); + const roots: string[] = []; + let examined = 0; + for (let index = 0; index < ready.length; index += 1) { + const frame = ready[index]; + if (frame === undefined) throw new Error("Missing ready frame"); + const fingerprint = digestCanonicalValue({ + url: frame.url, + origin: frame.origin, + children: (children.get(frame.frame_id) ?? []).sort(), + }); + examined += 1; + const parentId = frame.parent_frame_id; + if (parentId === null) { + roots.push(fingerprint); + continue; + } + const siblings = children.get(parentId) ?? []; + siblings.push(fingerprint); + children.set(parentId, siblings); + const count = (remaining.get(parentId) ?? 0) - 1; + remaining.set(parentId, count); + if (count === 0) { + const parent = byId.get(parentId); + if (parent === undefined) + throw new Error("Missing admitted parent frame"); + ready.push(parent); + } + } + // Cyclic ancestry never reaches the leaf queue and cannot prove equality. + return examined === frames.length ? digestCanonicalValue(roots.sort()) : null; +}; diff --git a/src/domain/webMcpDiscovery.test.ts b/src/domain/webMcpDiscovery.test.ts new file mode 100644 index 0000000000..452038d6b5 --- /dev/null +++ b/src/domain/webMcpDiscovery.test.ts @@ -0,0 +1,128 @@ +import { describe, expect, it } from "vitest"; + +import { createEvidence } from "./evidence.js"; +import { + createEvidenceBundle, + parseEvidenceBundle, + serializeEvidenceBundle, +} from "./evidenceBundle.js"; +import { jsonValueSchema } from "./jsonValue.js"; +import { + webMcpDiscoverySchema, + type WebMcpDiscovery, +} from "./webMcpDiscovery.js"; + +const PROVIDER = { id: "fixture", name: "Fixture provider", version: "1" }; + +const discovery = (annotations: Record): WebMcpDiscovery => + webMcpDiscoverySchema.parse({ + browser: { + product: "Chrome", + protocol_version: "1.3", + revision: "test", + user_agent: "test", + js_version: "test", + }, + target: { + target_id: "page-1", + url: "http://127.0.0.1:3000/", + origin: "http://127.0.0.1:3000", + }, + status: "available", + tools: { + total: 1, + items: [ + { + tool_key: `webmcp_${"a".repeat(64)}`, + name: "book_flight", + description: "Book a flight", + frame_id: "frame-main", + frame_url: "http://127.0.0.1:3000/", + owner_origin: "http://127.0.0.1:3000", + declaration_kind: "imperative", + input_schema: null, + input_schema_sha256: null, + annotations, + registration_source: null, + trust: "page-declared-untrusted", + }, + ], + }, + completeness: { + status: "complete_within_window", + conditions: ["complete_within_window"], + policy_filtered_sections: [], + attach_limited_sections: [], + truncated_sections: [], + unavailable_sections: [], + excluded: [], + dropped_events: { + scripts: 0, + network_requests: 0, + console_events: 0, + websocket_connections: 0, + websocket_frames: 0, + webmcp_tools: 0, + timeline_events: 0, + total: 0, + }, + }, + limitations: [], + }); + +describe("WebMCP discovery consequential annotation", () => { + it.each([true, false, null])( + "preserves a declared consequential value of %s", + (consequential) => { + const parsed = discovery({ + read_only: null, + untrusted_content: null, + consequential, + autosubmit: null, + }); + expect(parsed.tools.items[0]?.annotations.consequential).toBe( + consequential, + ); + }, + ); + + it("admits historical captures recorded before consequential was surfaced", () => { + const parsed = discovery({ + read_only: true, + untrusted_content: null, + autosubmit: false, + }); + expect(parsed.tools.items[0]?.annotations.consequential).toBeUndefined(); + }); + + it("survives the Evidence bundle export and re-read round-trip", () => { + const result = discovery({ + read_only: null, + untrusted_content: null, + consequential: true, + autosubmit: null, + }); + const evidence = createEvidence(undefined, PROVIDER, { + predicateType: "rea.webmcp-discovery", + operation: "discover_webmcp_tools", + parameters: {}, + result: jsonValueSchema.parse(result), + }); + const bundle = createEvidenceBundle([evidence]); + const reread = parseEvidenceBundle( + JSON.parse(serializeEvidenceBundle(bundle)), + ); + const record = reread.records[0]; + expect(record?.normalized_result).toMatchObject({ + tools: { + items: [ + { + annotations: { + consequential: true, + }, + }, + ], + }, + }); + }); +}); diff --git a/src/domain/webMcpDiscovery.ts b/src/domain/webMcpDiscovery.ts index 0924f5b782..0edf81dd3c 100644 --- a/src/domain/webMcpDiscovery.ts +++ b/src/domain/webMcpDiscovery.ts @@ -33,6 +33,9 @@ const webMcpToolSchema = z.object({ annotations: z.object({ read_only: z.boolean().nullable(), untrusted_content: z.boolean().nullable(), + // Absent when this annotation was not recorded or its protocol value was invalid; null + // distinguishes a declaration that omitted it from a declared false. + consequential: z.boolean().nullable().optional(), autosubmit: z.boolean().nullable(), }), registration_source: z diff --git a/src/domain/webScriptExportError.ts b/src/domain/webScriptExportError.ts index 68067a3ec0..db01d50b21 100644 --- a/src/domain/webScriptExportError.ts +++ b/src/domain/webScriptExportError.ts @@ -4,7 +4,7 @@ import type { AnalysisErrorOptions } from "./analysisErrorBase.js"; /** Publication failure retaining its input/output target and cleanup state. */ export class WebScriptExportError extends ArtifactOperationError { override readonly userMessage: string; - override readonly cleanupIncomplete: boolean; + override cleanupIncomplete: boolean; override readonly cleanupResources: readonly string[]; constructor( diff --git a/src/domain/zipPackageFormat.ts b/src/domain/zipPackageFormat.ts index 04a73065cc..6d9fd4663b 100644 --- a/src/domain/zipPackageFormat.ts +++ b/src/domain/zipPackageFormat.ts @@ -52,6 +52,6 @@ export const zipPackageFormatForPath = ( return undefined; }; -/** Inventory cannot see bytes that are not central-directory entries. */ -export const ZIP_NON_ENTRY_TAIL_LIMITATION = - "ZIP inventory lists central-directory entries only. Bytes after the central directory, including an APK Signing Block or a ZIP comment, are not occurrences."; +/** Inventory lists entries, not all bytes surrounding the central directory. */ +export const ZIP_NON_ENTRY_LIMITATION = + "ZIP inventory lists central-directory entries only. Non-entry bytes, including an APK Signing Block before the central directory or a ZIP comment after it, are not occurrences."; diff --git a/src/dotnet/ManagedArtifactInspector.ts b/src/dotnet/ManagedArtifactInspector.ts index 625969aba8..e5e06f678c 100644 --- a/src/dotnet/ManagedArtifactInspector.ts +++ b/src/dotnet/ManagedArtifactInspector.ts @@ -10,6 +10,7 @@ import { } from "../domain/managed/managedArtifact.js"; import { managedCli32BitRequirement, + managedCliIsMixedClrNative, readManagedPeLayout, type ManagedCliHeader, type ManagedPeLayout, @@ -24,6 +25,7 @@ import { readManagedMetadataLayout, type ManagedMetadataLayout, } from "./ManagedMetadataLayout.js"; +import { readTypeMemberListFacts } from "./ManagedTypeMemberLists.js"; import { ManagedReaderFailure } from "./ManagedReaderFailure.js"; const flagNames = (flags: number): string[] => { @@ -163,7 +165,7 @@ const runtimeFamilyFromCliHeader = ( ): ManagedArtifactInspection["classification"]["runtime_family"] => cli?.readyToRunSignature === true ? "modern-dotnet" - : cli !== null && (cli.flags & 1) === 0 + : cli !== null && managedCliIsMixedClrNative(cli) ? "mixed-clr-native" : "unknown"; @@ -189,8 +191,6 @@ const runtimeFamilyFor = ( ); let runtimeFamily: ManagedArtifactInspection["classification"]["runtime_family"] = "unknown"; - // crossgen2 clears ILONLY on ReadyToRun output, so only a clear flag - // without the ReadyToRun signature indicates mixed CLR/native code. if (runtimeFamilyFromCliHeader(cli) === "mixed-clr-native") { runtimeFamily = "mixed-clr-native"; evidence.push({ @@ -384,10 +384,12 @@ export const inspectManagedArtifactBytes = ( resourceIssues, ); try { + const memberLists = readTypeMemberListFacts(bytes, metadata); const inventory = readManagedMetadataInventory( bytes, metadata, resourceDirectory, + memberLists, ); const issues = [...resourceIssues, ...inventory.issues]; const limitations = [ diff --git a/src/dotnet/ManagedCilDecoding.test.ts b/src/dotnet/ManagedCilDecoding.test.ts index 852762b2c1..8214d0e85d 100644 --- a/src/dotnet/ManagedCilDecoding.test.ts +++ b/src/dotnet/ManagedCilDecoding.test.ts @@ -7,6 +7,7 @@ import { buildManagedPeFixture, managedPeFixtureTarget, } from "./ManagedPe.fixture.js"; +import { parseExceptionRegions } from "./ManagedMemberInstructionDecoder.js"; const managedBodyWithSections = (...sections: Buffer[]) => { const header = Buffer.alloc(12); @@ -331,7 +332,9 @@ describe("managed operand extent diagnostics", () => { }); }, ); +}); +describe("managed complete operand decoding", () => { // The extent check must also reject nothing that fits, so every multi-byte // width is exercised at exactly its own size with a known value. Each body // ends at its operand, so a width declared too large turns a valid operand @@ -664,6 +667,59 @@ describe("managed exception region flags and filters", () => { }); describe("managed exception section decoding", () => { + it("reads legacy small EH DataSize without counting its header", () => { + const section = Buffer.alloc(16); + section[0] = 0x01; + section[1] = 12; + + expect(managedBodyWithSections(section)).toMatchObject({ + status: "present", + issue: null, + exception_regions: [expect.any(Object)], + normalized_il_sha256: expect.any(String), + }); + }); + + it("advances a chained legacy section by its physical size", () => { + const legacy = Buffer.alloc(16); + legacy[0] = 0x81; + legacy[1] = 12; + const conformant = Buffer.alloc(16); + conformant[0] = 0x01; + conformant[1] = 16; + + expect(managedBodyWithSections(legacy, conformant)).toMatchObject({ + status: "present", + issue: null, + exception_regions: [expect.any(Object), expect.any(Object)], + normalized_il_sha256: expect.any(String), + }); + }); + + it("rejects a legacy-sized clause whose physical bytes leave the artifact", () => { + const section = Buffer.alloc(12); + section[0] = 0x01; + section[1] = 12; + + expect(parseExceptionRegions(section, 0, section.length)).toMatchObject({ + status: "malformed", + regions: [], + issue: "Exception section size leaves artifact", + }); + }); + + it("still rejects a small EH DataSize that matches neither layout", () => { + const section = Buffer.alloc(17); + section[0] = 0x01; + section[1] = 13; + + expect(managedBodyWithSections(section)).toMatchObject({ + status: "malformed", + exception_regions: [], + issue: "Exception section size does not contain whole clauses", + }); + }); + it("marks an exception section with a partial clause as malformed", () => { const section = Buffer.alloc(17); section[0] = 0x01; @@ -693,6 +749,19 @@ describe("managed exception section decoding", () => { }); }); + it("keeps small section reserved bytes invalid", () => { + const section = Buffer.alloc(16); + section[0] = 0x01; + section[1] = 16; + section[2] = 1; + + expect(managedBodyWithSections(section)).toMatchObject({ + status: "malformed", + exception_regions: [], + issue: "Small exception section reserved bytes are nonzero", + }); + }); + it("preserves decoded clauses when a chained section has an unsupported kind", () => { const first = Buffer.alloc(16); first[0] = 0x81; diff --git a/src/dotnet/ManagedDecodeBudget.test.ts b/src/dotnet/ManagedDecodeBudget.test.ts index ebab4a99e9..b46f6a0a45 100644 --- a/src/dotnet/ManagedDecodeBudget.test.ts +++ b/src/dotnet/ManagedDecodeBudget.test.ts @@ -35,6 +35,35 @@ describe("managed projection admission", () => { ManagedReaderFailure, ); }); + + it("charges an admitted record once inside its containing projection", () => { + const budget = new ManagedDecodeBudget(); + const record = admitManagedProjection( + { name: "a".repeat(40 * 1024 * 1024) }, + budget, + ); + + expect(() => + admitManagedProjection({ records: [record] }, budget), + ).not.toThrow(); + // The containing projection spent the credit, so later reuse pays again. + expect(() => + admitManagedProjection({ records: [record, record] }, budget), + ).toThrow(ManagedReaderFailure); + }); + + it("charges a repeated occurrence of an admitted record again", () => { + const budget = new ManagedDecodeBudget(); + const record = admitManagedProjection( + { name: "a".repeat(40 * 1024 * 1024) }, + budget, + ); + + // The first occurrence spends the credit; the second serializes again. + expect(() => + admitManagedProjection({ records: [record, record] }, budget), + ).toThrow(ManagedReaderFailure); + }); }); describe("managed decoding bounds", () => { diff --git a/src/dotnet/ManagedDecodeBudget.ts b/src/dotnet/ManagedDecodeBudget.ts index d02fce0bc7..fba6229a27 100644 --- a/src/dotnet/ManagedDecodeBudget.ts +++ b/src/dotnet/ManagedDecodeBudget.ts @@ -18,6 +18,7 @@ export class ManagedDecodeBudget { readonly #rows = new Map>(); readonly #strings = new Map(); readonly #digests = new WeakMap>(); + readonly #prepaid = new WeakSet(); reserve(bytes: number, offset: number | null): void { this.#admit(bytes, offset); @@ -82,6 +83,16 @@ export class ManagedDecodeBudget { return digest; } + /** Record an admitted projection; its next serialized occurrence is already charged. */ + prepay(value: object): void { + this.#prepaid.add(value); + } + + /** Spend an admitted projection's charge once; repeated occurrences pay again. */ + consumePrepaid(value: object): boolean { + return this.#prepaid.delete(value); + } + cachedString(index: number): CachedString | undefined { return this.#strings.get(index); } @@ -141,12 +152,23 @@ const jsonBytes = (item: unknown): number => ? managedJsonStringBytes(item) : Buffer.byteLength(JSON.stringify(item) ?? "null"); -/** Admit repeated values too: shared heap strings still expand at the JSON boundary. */ +/** + * Admit repeated values too: shared heap strings still expand at the JSON + * boundary. An object admitted earlier was charged for one occurrence, so a + * containing projection spends that charge instead of counting it again. + * Admitted values must stay unmodified. + */ export const admitManagedProjection = ( value: Value, budget: ManagedDecodeBudget = new ManagedDecodeBudget(), ): Value => { const visit = (item: unknown): void => { + if ( + item !== null && + typeof item === "object" && + budget.consumePrepaid(item) + ) + return; if (Array.isArray(item)) { budget.reserve(item.length + 2, null); for (const child of item) visit(child); @@ -161,5 +183,6 @@ export const admitManagedProjection = ( } }; visit(value); + if (value !== null && typeof value === "object") budget.prepay(value); return value; }; diff --git a/src/dotnet/ManagedMemberInspector.test.ts b/src/dotnet/ManagedMemberInspector.test.ts index 3aee06e735..e059579b36 100644 --- a/src/dotnet/ManagedMemberInspector.test.ts +++ b/src/dotnet/ManagedMemberInspector.test.ts @@ -11,14 +11,17 @@ import { type ManagedMetadataLayout, } from "./ManagedMetadataLayout.js"; import { readManagedPeLayout } from "./ManagedPeReader.js"; +import { memberListIndexSize } from "./ManagedMemberPointers.js"; import { edges } from "./ManagedMemberRows.js"; import { parseUserStrings } from "./ManagedUserStrings.js"; import { signature, type ManagedMethod } from "./ManagedMemberInspectorCore.js"; import { ManagedReaderFailure } from "./ManagedReaderFailure.js"; import { managedMemberInspectionSchema } from "../domain/managed/managedArtifact.js"; import { comparisonLimitations } from "../domain/managed/managedMemberComparisonCoverage.js"; +import { buildManagedPeFixture as buildMultiMethodManagedPeFixture } from "../../scripts/fixtures/managed/pe.mjs"; import { + buildManagedNestedTypeFixture, buildManagedPeFixture, buildNativePeFixture, fixtureUserStringHeap, @@ -523,6 +526,111 @@ describe("managed MethodSpec call targets", () => { }); }); +describe("managed nested types", () => { + const inspectNested = ( + options: Parameters[0] = {}, + ) => { + const bytes = buildManagedNestedTypeFixture(options); + return inspectManagedMembersBytes( + bytes, + managedPeFixtureTarget(bytes, "/fixture/nested.dll"), + ); + }; + + it("reports the enclosing type of each nested type", () => { + const result = inspectNested(); + + expect( + result.types.map(({ full_name, enclosing_type_token }) => [ + full_name, + enclosing_type_token, + ]), + ).toEqual([ + ["", null], + ["Fixture.A", null], + ["Fixture.B", null], + ["Inner", "0x02000002"], + ["Inner", "0x02000003"], + ["Leaf", "0x02000004"], + ]); + expect(result.nesting_read).toBe(true); + expect(result.coverage.issues).toEqual([]); + }); + + it.each([2, 3, 4, 5, 6, 7])( + "keeps nested visibility %i unresolved when its parent entry is missing", + (visibility) => { + const bytes = buildManagedNestedTypeFixture({ nestedClassRows: [] }); + const target = () => managedPeFixtureTarget(bytes, "/fixture/nested.dll"); + const row = inspectManagedMembersBytes(bytes, target()).types.find( + (type) => type.token === "0x02000004", + )!.row_offset; + bytes.writeUInt32LE((bytes.readUInt32LE(row) & ~7) | visibility, row); + const result = inspectManagedMembersBytes(bytes, target()); + expect(result.nesting_read).toBe(true); + expect( + result.types.find((type) => type.token === "0x02000004"), + ).toMatchObject({ enclosing_type_token: null, name_sha256: null }); + expect(result.coverage.issues).toContainEqual( + expect.objectContaining({ + code: "invalid-row", + scope: "metadata.TypeDef:0x02000004", + offset: row, + }), + ); + }, + ); + + it("retains a selected child's contradictory parent issue exactly once", () => { + const bytes = buildManagedNestedTypeFixture(); + const target = () => managedPeFixtureTarget(bytes, "/fixture/nested.dll"); + const row = inspectManagedMembersBytes(bytes, target()).types.find( + (type) => type.token === "0x02000004", + )!.row_offset; + bytes.writeUInt32LE((bytes.readUInt32LE(row) & ~7) | 1, row); + const result = inspectManagedMembersBytes(bytes, target(), { + typeNames: ["Leaf"], + }); + expect(result.types).toHaveLength(1); + expect(result.types[0]?.name_sha256).toBeNull(); + expect(result.coverage.state).toBe("partial"); + expect( + result.coverage.issues.filter( + (issue) => issue.scope === "metadata.TypeDef:0x02000004", + ), + ).toEqual([expect.objectContaining({ code: "invalid-row", offset: row })]); + }); + + it("reports NestedClass rows that do not pair distinct existing types once", () => { + const result = inspectNested({ + nestedClassRows: [ + [4, 4], + [5, 3], + [5, 2], + [6, 9], + [0, 2], + [4, 0], + [9, 2], + ], + }); + + expect( + result.types.map(({ enclosing_type_token }) => enclosing_type_token), + ).toEqual([null, null, null, null, "0x02000003", null]); + expect(result.nesting_read).toBe(false); + expect( + result.coverage.issues.map(({ code, scope }) => [code, scope]), + ).toEqual([ + ["invalid-row", "metadata.NestedClass:0x29000001"], + ["invalid-row", "metadata.NestedClass:0x29000003"], + ["invalid-row", "metadata.NestedClass:0x29000004"], + ["invalid-row", "metadata.NestedClass:0x29000005"], + ["invalid-row", "metadata.NestedClass:0x29000006"], + ["invalid-row", "metadata.NestedClass:0x29000007"], + ]); + }); +}); + describe("managed exported types", () => { it("reports type forwarders and nested exported types with their implementation scope", () => { const bytes = buildManagedPeFixture({ @@ -1223,7 +1331,173 @@ describe("managed derived caller names", () => { }); }); -describe("managed member reference bounds", () => { +describe("managed TypeDef list ownership bounds", () => { + it("retains rows but leaves ownership unknown for zero TypeDef list starts", () => { + const bytes = buildManagedPeFixture(); + const pe = readManagedPeLayout(bytes); + if (pe.cli === null) throw new Error("fixture has no CLI header"); + const { rva, size } = pe.cli.metadata; + const layout = readManagedMetadataLayout( + bytes, + pe.rvaToOffset(rva, size, "cli.metadata"), + size, + ); + const listOffset = + metadataRowOffset(layout, 2, 1) + + 4 + + 2 * layout.stringIndexSize + + layout.codedIndexSize("TypeDefOrRef"); + const listIndexWidth = Math.max( + layout.tableIndexSize(3), + layout.tableIndexSize(4), + layout.tableIndexSize(5), + layout.tableIndexSize(6), + ); + const writeIndex = (offset: number) => + listIndexWidth === 2 + ? bytes.writeUInt16LE(0, offset) + : bytes.writeUInt32LE(0, offset); + writeIndex(listOffset); + writeIndex(listOffset + listIndexWidth); + + const result = inspectManagedMembersBytes( + bytes, + managedPeFixtureTarget(bytes), + ); + + expect(result.types[0]).toMatchObject({ + field_list: { first_row: null, last_row: null, count: 0 }, + method_list: { first_row: null, last_row: null, count: 0 }, + }); + expect(result.fields[0]).toMatchObject({ + token: "0x04000001", + name: "counter", + declaring_type: null, + }); + expect(result.methods[0]).toMatchObject({ + token: "0x06000001", + name: "Main", + declaring_type: null, + }); + expect(result.coverage).toMatchObject({ + state: "partial", + issues: [ + expect.objectContaining({ + code: "invalid-row", + detail: expect.stringContaining("TypeDef FieldList start 0"), + }), + expect.objectContaining({ + code: "invalid-row", + detail: expect.stringContaining("TypeDef MethodList start 0"), + }), + ], + }); + }); + + it("does not infer an earlier owner from an invalid next list endpoint", () => { + const bytes = buildManagedPeFixture({ additionalEmptyTypes: 1 }); + const pe = readManagedPeLayout(bytes); + if (pe.cli === null) throw new Error("fixture has no CLI header"); + const { rva, size } = pe.cli.metadata; + const layout = readManagedMetadataLayout( + bytes, + pe.rvaToOffset(rva, size, "cli.metadata"), + size, + ); + const methodStartOffset = + metadataRowOffset(layout, 2, 2) + + 4 + + 2 * layout.stringIndexSize + + layout.codedIndexSize("TypeDefOrRef") + + memberListIndexSize(layout, 4); + if (memberListIndexSize(layout, 6) === 2) + bytes.writeUInt16LE(0, methodStartOffset); + else bytes.writeUInt32LE(0, methodStartOffset); + + const result = inspectManagedMembersBytes( + bytes, + managedPeFixtureTarget(bytes), + ); + + expect(result.methods[0]).toMatchObject({ + token: "0x06000001", + declaring_type: null, + }); + expect(result.types[0]?.method_list).toMatchObject({ + first_row: null, + last_row: null, + count: 0, + }); + expect(result.coverage.state).toBe("partial"); + }); +}); + +describe("managed TypeDef list ordering", () => { + it("leaves adjacent ownership unknown when valid list starts decrease", () => { + const bytes = buildManagedPeFixture({ + additionalEmptyTypes: 1, + methodRowCount: 2, + }); + const pe = readManagedPeLayout(bytes); + if (pe.cli === null) throw new Error("fixture has no CLI header"); + const { rva, size } = pe.cli.metadata; + const layout = readManagedMetadataLayout( + bytes, + pe.rvaToOffset(rva, size, "cli.metadata"), + size, + ); + const methodOffset = + 4 + + 2 * layout.stringIndexSize + + layout.codedIndexSize("TypeDefOrRef") + + memberListIndexSize(layout, 4); + const writeMethodStart = (row: number, value: number): void => { + const offset = metadataRowOffset(layout, 2, row) + methodOffset; + if (memberListIndexSize(layout, 6) === 2) + bytes.writeUInt16LE(value, offset); + else bytes.writeUInt32LE(value, offset); + }; + writeMethodStart(1, 2); + writeMethodStart(2, 1); + + const result = inspectManagedMembersBytes( + bytes, + managedPeFixtureTarget(bytes), + ); + + expect(result.methods.map(({ declaring_type }) => declaring_type)).toEqual([ + null, + null, + ]); + expect(result.coverage).toMatchObject({ + state: "partial", + issues: [ + expect.objectContaining({ + code: "invalid-row", + detail: expect.stringContaining( + "MethodList start 1 decreases from preceding start 2", + ), + }), + ], + }); + }); + + it("accepts the one-based start for an empty ordinary member table", () => { + const bytes = buildManagedPeFixture({ methodRowCount: 0 }); + const result = inspectManagedMembersBytes( + bytes, + managedPeFixtureTarget(bytes), + ); + expect(result.types[0]?.method_list).toEqual({ + first_row: null, + last_row: null, + count: 0, + }); + expect(result.coverage).toMatchObject({ state: "complete", issues: [] }); + }); +}); + +describe("managed metadata reference bounds", () => { it("retains members and reports an out-of-range TypeDef coded index as partial", () => { const bytes = buildManagedPeFixture({ extendsRaw: 12 }); const result = inspectManagedMembersBytes( @@ -1358,11 +1632,15 @@ it("decodes generic instantiation member signatures as complete coverage", () => // shape, with TypeRef/TypeSpec coded tokens kept as encoded. const bytes = buildManagedPeFixture({ methodSignature: Buffer.from([ - 0x00, 0x01, 0x01, 0x15, 0x12, 0x71, 0x01, 0x0e, + 0x00, 0x01, 0x01, 0x15, 0x12, 0x09, 0x01, 0x0e, ]), fieldSignature: Buffer.from([ - 0x06, 0x15, 0x11, 0x0a, 0x02, 0x13, 0x00, 0x1e, 0x01, + 0x06, 0x15, 0x11, 0x0d, 0x02, 0x13, 0x00, 0x1e, 0x01, ]), + typeReferences: [ + { namespace: "System", name: "Func`1" }, + { namespace: "System", name: "ValueTuple`2" }, + ], }); const result = inspectManagedMembersBytes( bytes, @@ -1372,12 +1650,12 @@ it("decodes generic instantiation member signatures as complete coverage", () => expect(result.methods[0]?.signature).toMatchObject({ parse_status: "decoded", return_type: "void", - parameter_types: ["class:113"], + parameter_types: ["class:9"], issue: null, }); expect(result.fields[0]?.signature).toMatchObject({ parse_status: "decoded", - field_type: "valuetype:10", + field_type: "valuetype:13", }); expect(result.coverage).toMatchObject({ state: "complete", issues: [] }); }); @@ -1735,3 +2013,61 @@ describe("managed member type selection", () => { expect(result.member_refs).toEqual(everything.member_refs); }); }); + +describe("managed member method paging", () => { + it("returns only the requested MethodDef rows", () => { + const image = buildMultiMethodManagedPeFixture({ + methods: [ + { name: "First", body: Buffer.from([0x2a]) }, + { name: "Second", body: Buffer.from([0x2a]) }, + { name: "Third", body: Buffer.from([0x2a]) }, + ], + }); + const all = inspectManagedMembersBytes( + image, + managedPeFixtureTarget(image), + ); + expect(all.methods.map(({ name }) => name)).toEqual([ + "First", + "Second", + "Third", + ]); + + const page = inspectManagedMembersBytes( + image, + managedPeFixtureTarget(image), + { methodOffset: 1, methodLimit: 1 }, + ); + + expect(page.methods.map(({ name, token }) => ({ name, token }))).toEqual([ + { name: "Second", token: "0x06000002" }, + ]); + expect(page.coverage.state).toBe("complete"); + + const offsetOnly = inspectManagedMembersBytes( + image, + managedPeFixtureTarget(image), + { methodOffset: 1 }, + ); + expect(offsetOnly.methods.map(({ name }) => name)).toEqual([ + "Second", + "Third", + ]); + + const emptyPage = inspectManagedMembersBytes( + image, + managedPeFixtureTarget(image), + { methodOffset: 1, methodLimit: 0 }, + ); + expect(emptyPage.methods).toEqual([]); + }); + + it("rejects invalid method page bounds", () => { + const image = buildManagedPeFixture(); + expect(() => + inspectManagedMembersBytes(image, managedPeFixtureTarget(image), { + methodOffset: -1, + }), + ).toThrow(RangeError); + }); +}); diff --git a/src/dotnet/ManagedMemberInspector.ts b/src/dotnet/ManagedMemberInspector.ts index 1b018522ac..36f6137ba1 100644 --- a/src/dotnet/ManagedMemberInspector.ts +++ b/src/dotnet/ManagedMemberInspector.ts @@ -29,8 +29,11 @@ import { type TypeRange, } from "./ManagedMemberInspectorCore.js"; import { edgeTargets } from "./ManagedMethodSpecs.js"; +import { managedTypeNameDigests } from "./ManagedTypeNameDigests.js"; import { typeNameLookup } from "./ManagedTypeNames.js"; +import { managedTypeReferences } from "./ManagedTypeReferences.js"; import { parseUserStrings } from "./ManagedUserStrings.js"; +import { readTypeMemberListFacts } from "./ManagedTypeMemberLists.js"; import { edges, parseExportedTypes, @@ -71,6 +74,8 @@ const unavailable = ( requires_mvid: null, }, types: [], + signature_resolution: "type-names-v1", + nesting_read: false, exported_types: null, user_strings: null, fields: [], @@ -113,12 +118,14 @@ const readMemberInventory = ( rootOffset, cli.metadata.size, ); + const memberLists = readTypeMemberListFacts(bytes, layout); const inventory = readManagedMetadataInventory( bytes, layout, readManagedResourceDirectory(pe, issues), + memberLists, ); - return { layout, inventory }; + return { layout, inventory, memberLists }; }; const selectTypes = ( @@ -160,20 +167,38 @@ export const inspectManagedMembersBytes = ( target: BinaryTarget, { typeNames, + methodOffset = 0, + methodLimit = Number.MAX_SAFE_INTEGER, userStrings: readUserStrings = true, }: { readonly typeNames?: readonly string[]; + /** Zero-based offset into MethodDef rows. */ + readonly methodOffset?: number; + readonly methodLimit?: number; readonly userStrings?: boolean; } = {}, ): ManagedMemberInspection => { + if ( + !Number.isSafeInteger(methodOffset) || + methodOffset < 0 || + !Number.isSafeInteger(methodLimit) || + methodLimit < 0 + ) + throw new RangeError( + "methodOffset and methodLimit must be non-negative safe integers", + ); const pe = readManagedPeLayout(bytes); if (pe.cli === null) return unavailable(target, bytes, pe.cliIssue, typeNames); const issues: ManagedParseIssue[] = []; try { - const { layout, inventory } = readMemberInventory(bytes, pe, issues); + const { layout, inventory, memberLists } = readMemberInventory( + bytes, + pe, + issues, + ); appendManagedParseIssues(issues, inventory.issues); - const ranges = typeRanges(bytes, layout, issues); + const ranges = typeRanges(bytes, layout, issues, memberLists); const { tokens: selected, selection, @@ -183,11 +208,22 @@ export const inspectManagedMembersBytes = ( appendManagedParseIssues(issues, parsedTypes.issues); const exportedTypes = parseExportedTypes(bytes, layout); appendManagedParseIssues(issues, exportedTypes.issues); - const fields = parseFields(bytes, layout, ranges, selected); + const typeReferences = managedTypeReferences(bytes, layout, issues); + const signatureTypeName = managedTypeNameDigests( + layout, + issues, + parsedTypes, + typeReferences, + ); + const fields = parseFields(bytes, layout, ranges, memberLists, { + signatureTypeName, + selected, + }); const memberRefs = parseMemberRefs( bytes, layout, - typeNameLookup(bytes, layout, ranges, issues), + typeNameLookup(bytes, layout, ranges, issues, typeReferences), + signatureTypeName, ); appendManagedParseIssues(issues, memberRefs.issues); const methods = parseMethods({ @@ -195,7 +231,11 @@ export const inspectManagedMembersBytes = ( layout, pe, ranges, + signatureTypeName, + memberLists, selected, + methodOffset, + methodLimit, }); appendManagedParseIssues(issues, fields.issues, methods.issues); // #US literals sit beside the member inventory and carry their own @@ -212,7 +252,6 @@ export const inspectManagedMembersBytes = ( methods.core, ]); appendManagedParseIssues(issues, targets.issues); - const coverageIssues = uniqueManagedParseIssues(issues); const incompleteMembers = [...fields.fields, ...memberRefs.refs, ...methods.methods].some( (member) => member.signature.parse_status !== "decoded", @@ -227,6 +266,23 @@ export const inspectManagedMembersBytes = ( targets.members, managedDecodeBudget(layout), ); + const types = parsedTypes.types + .filter((type) => selected === null || selected.has(type.token)) + .map((type) => { + managedDecodeBudget(layout).reserve(512, type.row_offset); + return { + ...type, + name_sha256: + signatureTypeName( + Number.parseInt(type.token.slice(4), 16) * 4, + )?.toString("hex") ?? null, + }; + }); + const unnamedTypes = types.reduce( + (count, type) => count + Number(type.name_sha256 === null), + 0, + ); + const coverageIssues = uniqueManagedParseIssues(issues); return managedMemberInspectionSchema.parse( admitManagedProjection( { @@ -247,10 +303,9 @@ export const inspectManagedMembersBytes = ( requires_artifact_sha256: target.sha256, requires_mvid: inventory.module?.mvid ?? null, }, - types: - selected === null - ? parsedTypes.types - : parsedTypes.types.filter((type) => selected.has(type.token)), + types, + signature_resolution: "type-names-v1", + nesting_read: parsedTypes.nesting.complete, exported_types: exportedTypes.exportedTypes, fields: fields.fields, methods: methods.methods, @@ -267,6 +322,11 @@ export const inspectManagedMembersBytes = ( issues: coverageIssues, }, limitations: [ + ...(unnamedTypes === 0 + ? [] + : [ + `${String(unnamedTypes)} returned TypeDef name identity(s) remain unresolved; incomplete, missing, contradictory, cyclic or over-deep enclosing facts cannot be repaired by bare display names.`, + ]), "Metadata tokens are build-local coordinates and are only meaningful with the reported artifact SHA-256 and MVID.", "CIL instruction anchors are decoded from file-backed method bodies only; no target assembly is loaded or executed.", "Exported types are reported from ExportedType rows as declared; forwarded types are not resolved in their implementing assemblies.", diff --git a/src/dotnet/ManagedMemberInspectorCore.ts b/src/dotnet/ManagedMemberInspectorCore.ts index caac2e66eb..5e9a7ef40e 100644 --- a/src/dotnet/ManagedMemberInspectorCore.ts +++ b/src/dotnet/ManagedMemberInspectorCore.ts @@ -1,3 +1,5 @@ +import { createHash } from "node:crypto"; + import { managedFailure, ManagedReaderFailure, @@ -8,6 +10,10 @@ import type { ManagedParseIssue, } from "../domain/managed/managedArtifact.js"; import { managedDecodeBudget } from "./ManagedDecodeBudget.js"; +import { + readManagedTypeNesting, + type ManagedTypeNesting, +} from "./ManagedTypeNesting.js"; import type { ManagedMetadataLayout, MetadataTableLayout, @@ -20,10 +26,8 @@ import { readMetadataString, sha256Bytes, } from "./ManagedMetadataHeaps.js"; -import { - memberListIndexSize, - memberListTable, -} from "./ManagedMemberPointers.js"; +import { memberListTable } from "./ManagedMemberPointers.js"; +import type { TypeMemberListFacts } from "./ManagedTypeMemberLists.js"; export type ManagedType = ManagedMemberInspection["types"][number]; export type ManagedExportedType = NonNullable< @@ -47,10 +51,10 @@ export type ManagedExceptionRegion = export interface TypeRange { readonly token: string; readonly fullName: string | null; - readonly fieldStart: number; - readonly fieldEnd: number; - readonly methodStart: number; - readonly methodEnd: number; + readonly fieldStart: number | null; + readonly fieldEnd: number | null; + readonly methodStart: number | null; + readonly methodEnd: number | null; } /** @@ -80,18 +84,23 @@ export interface ParsedInstruction { const rowRange = ( table: MetadataTableLayout | undefined, - start: number, - nextStart: number, + start: number | null, + end: number | null, ): { readonly first: number | null; readonly last: number | null; readonly count: number; } => { const total = table?.rowCount ?? 0; - if (total === 0 || start === 0) return { first: null, last: null, count: 0 }; - const end = Math.min(nextStart === 0 ? total + 1 : nextStart, total + 1); - if (start >= end) return { first: null, last: null, count: 0 }; - return { first: start, last: end - 1, count: end - start }; + if (start === null || end === null) + return { first: null, last: null, count: 0 }; + const lastExclusive = Math.min(end, total + 1); + if (start >= lastExclusive) return { first: null, last: null, count: 0 }; + return { + first: start, + last: lastExclusive - 1, + count: lastExclusive - start, + }; }; export const fullName = (namespace: string, name: string): string => @@ -102,9 +111,8 @@ const readTypeRange = ( layout: ManagedMetadataLayout, row: number, issues: ManagedParseIssue[], + memberLists: TypeMemberListFacts, ): TypeRange => { - const methods = memberListTable(layout, 6); - const fields = memberListTable(layout, 4); const cursor = metadataRowCursor(bytes, layout, 2, row); cursor.readUInt32(); const nameIndex = cursor.readIndex(layout.stringIndexSize); @@ -124,27 +132,14 @@ const readTypeRange = ( issues, ) ?? null; cursor.readIndex(layout.codedIndexSize("TypeDefOrRef")); - const fieldStart = cursor.readIndex(memberListIndexSize(layout, 4)); - const methodStart = cursor.readIndex(memberListIndexSize(layout, 6)); - let nextField = (fields?.rowCount ?? 0) + 1; - let nextMethod = (methods?.rowCount ?? 0) + 1; - const typeTable = layout.table(2); - if (typeTable !== undefined && row < typeTable.rowCount) { - const next = metadataRowCursor(bytes, layout, 2, row + 1); - next.readUInt32(); - next.readIndex(layout.stringIndexSize); - next.readIndex(layout.stringIndexSize); - next.readIndex(layout.codedIndexSize("TypeDefOrRef")); - nextField = next.readIndex(memberListIndexSize(layout, 4)); - nextMethod = next.readIndex(memberListIndexSize(layout, 6)); - } + const current = memberLists.entries[row - 1]; return { token: metadataToken(2, row), fullName: declaredName, - fieldStart, - fieldEnd: nextField, - methodStart, - methodEnd: nextMethod, + fieldStart: current?.field.start.value ?? null, + fieldEnd: current?.field.end ?? null, + methodStart: current?.method.start.value ?? null, + methodEnd: current?.method.end ?? null, }; }; @@ -152,14 +147,20 @@ export const typeRanges = ( bytes: Buffer, layout: ManagedMetadataLayout, issues: ManagedParseIssue[], + memberLists: TypeMemberListFacts, ): readonly TypeRange[] => { + issues.push(...memberLists.issues); const typeTable = layout.table(2); const ranges: TypeRange[] = []; for (let row = 1; row <= (typeTable?.rowCount ?? 0); row += 1) - ranges.push(readTypeRange(bytes, layout, row, issues)); + ranges.push(readTypeRange(bytes, layout, row, issues, memberLists)); return ranges; }; +/** + * Enclosing TypeDef row of each nested TypeDef row, from the NestedClass + * table. Nesting is complete only when every row was read and valid. + */ export const parseTypes = ( bytes: Buffer, layout: ManagedMetadataLayout, @@ -167,12 +168,14 @@ export const parseTypes = ( ): { readonly types: readonly ManagedType[]; readonly issues: readonly ManagedParseIssue[]; + readonly nesting: ManagedTypeNesting; } => { const typeTable = layout.table(2); const fields = memberListTable(layout, 4); const methods = memberListTable(layout, 6); const items: ManagedType[] = []; const issues: ManagedParseIssue[] = []; + const nesting = readManagedTypeNesting(bytes, layout, issues); for (let row = 1; row <= (typeTable?.rowCount ?? 0); row += 1) { readManagedValue(() => { const cursor = metadataRowCursor(bytes, layout, 2, row); @@ -193,18 +196,16 @@ export const parseTypes = ( const extendsRaw = cursor.readIndex( layout.codedIndexSize("TypeDefOrRef"), ); - const fieldStart = cursor.readIndex(memberListIndexSize(layout, 4)); - const methodStart = cursor.readIndex(memberListIndexSize(layout, 6)); - const nextRange = ranges[row] ?? null; + const currentRange = ranges[row - 1] ?? null; const fieldRange = rowRange( fields, - fieldStart, - nextRange?.fieldStart ?? 0, + currentRange?.fieldStart ?? null, + currentRange?.fieldEnd ?? null, ); const methodRange = rowRange( methods, - methodStart, - nextRange?.methodStart ?? 0, + currentRange?.methodStart ?? null, + currentRange?.methodEnd ?? null, ); const extendsReason = metadataCodedTokenInvalidReason( extendsRaw, @@ -219,6 +220,7 @@ export const parseTypes = ( offset: extendsOffset, detail: `TypeDef Extends coded index 0x${extendsRaw.toString(16)} is invalid: ${extendsReason}`, }); + const enclosingRow = nesting.enclosing.get(row); items.push({ token: metadataToken(2, row), row_offset: cursor.start, @@ -232,6 +234,9 @@ export const parseTypes = ( [2, 1, 27], layout.rowCounts, ), + enclosing_type_token: + enclosingRow === undefined ? null : metadataToken(2, enclosingRow), + name_sha256: null, field_list: { first_row: fieldRange.first, last_row: fieldRange.last, @@ -245,7 +250,7 @@ export const parseTypes = ( }); }, issues); } - return { types: items, issues }; + return { types: items, issues, nesting }; }; export const readCompressed = ( @@ -503,11 +508,26 @@ const readTypeOperand = ( return token; }; +/** A TypeDefOrRefOrSpec coded index embedded in a signature blob. */ +interface TypeOperand { + readonly offset: number; + readonly next: number; + readonly value: number; +} + const readTypeSignature = ( blob: Buffer, offset: number, base: SignaturePosition, - layout?: ManagedMetadataLayout, + { + layout, + operands, + }: { + /** Validate each type operand against these tables, as specs require. */ + readonly layout?: ManagedMetadataLayout; + /** Collect each type operand, to resolve it by name. */ + readonly operands?: TypeOperand[]; + } = {}, ): { readonly value: string; readonly next: number } => { const stack: TypeContinuation[] = []; let cursor = offset; @@ -535,6 +555,7 @@ const readTypeSignature = ( cursor += 1; } else if (kind === 0x1f || kind === 0x20) { const token = readTypeOperand(blob, cursor + 1, layout); + operands?.push({ offset: cursor + 1, ...token }); const modifier = `${kind === 0x1f ? "modreq" : "modopt"}:${String(token.value)}`; // A modifier frame on top was pushed by the directly preceding byte; // keep an adjacent run together and in source order. @@ -544,6 +565,7 @@ const readTypeSignature = ( cursor = token.next; } else if (kind === 0x11 || kind === 0x12) { const token = readTypeOperand(blob, cursor + 1, layout); + operands?.push({ offset: cursor + 1, ...token }); value = `${kind === 0x11 ? "valuetype" : "class"}:${String(token.value)}`; cursor = token.next; } else if (kind === 0x1e || kind === 0x13) { @@ -564,6 +586,7 @@ const readTypeSignature = ( cursor + 1, ); const token = readTypeOperand(blob, cursor + 2, layout, [2, 1]); + operands?.push({ offset: cursor + 2, ...token }); const argumentCount = readCompressed(blob, token.next); if (argumentCount.value === 0) return malformedSignature( @@ -685,7 +708,7 @@ export const specSignature = ( const types: string[] = []; // Each Type consumes at least one byte, so a false count fails on truncation. for (let index = 0; index < count; index += 1) { - const type = readTypeSignature(blob, cursor, "type", layout); + const type = readTypeSignature(blob, cursor, "type", { layout }); types.push(type.value); cursor = type.next; } @@ -711,13 +734,19 @@ export const specSignature = ( } }; -type DecodedSignature = Omit; +type DecodedSignature = Omit< + ManagedSignature, + "raw_length" | "raw_sha256" | "resolved_sha256" +>; /** * Decode a field or member method signature. A malformed or unsupported blob * decodes as kind `unknown` with its reason. */ -const decodeSignature = (blob: Buffer): DecodedSignature => { +const decodeSignature = ( + blob: Buffer, + operands: TypeOperand[] = [], +): DecodedSignature => { try { if (blob.length === 0) throw managedFailure("invalid-blob", "signature", "empty signature", 0); @@ -725,7 +754,7 @@ const decodeSignature = (blob: Buffer): DecodedSignature => { // FieldSig (II.23.2.4) is exactly FIELD; a flagged FIELD header is not a // method header either, so it is malformed below. if (first === 0x06) { - const fieldType = readTypeSignature(blob, 1, "field"); + const fieldType = readTypeSignature(blob, 1, "field", { operands }); return { kind: "field", parse_status: fieldType.next === blob.length ? "decoded" : "partial", @@ -740,11 +769,15 @@ const decodeSignature = (blob: Buffer): DecodedSignature => { }; } const header = readMethodSignatureHeader(blob, 0, "member"); - const returnType = readTypeSignature(blob, header.next, "return"); + const returnType = readTypeSignature(blob, header.next, "return", { + operands, + }); let offset = returnType.next; const parameters: string[] = []; for (let index = 0; index < header.parameterCount; index += 1) { - const parameter = readTypeSignature(blob, offset, "parameter"); + const parameter = readTypeSignature(blob, offset, "parameter", { + operands, + }); parameters.push(parameter.value); offset = parameter.next; } @@ -812,11 +845,58 @@ export const signatureGenericArity = ( ? (decoded.generic_parameter_count ?? 0) : null; +/** + * SHA-256 of the namespace-qualified name of a TypeDefOrRefOrSpec coded index, + * or null when it has no name. + */ +export type SignatureTypeName = (codedIndex: number) => Buffer | null; + +/** + * Each type operand is replaced by 0xff, which never starts a compressed + * coded index, then the 32-byte name digest; a blob without type operands + * keeps its raw digest. So no decodable signature's resolved preimage is + * another's, but an undecodable blob can hold one, and its raw digest then + * equals that resolved digest. Comparison keys the two kinds apart. + */ +const OPERAND_MARKER = Buffer.from([0xff]); + +/** + * Digest the blob with each type operand replaced by its type's name digest, + * so the value does not depend on build-local row numbers. + */ +const resolvedSha256 = ( + blob: Buffer, + operands: readonly TypeOperand[], + typeName: SignatureTypeName, +): string | null => { + const hash = createHash("sha256"); + let copied = 0; + for (const operand of operands) { + const name = typeName(operand.value); + if (name === null) return null; + hash.update(blob.subarray(copied, operand.offset)); + hash.update(OPERAND_MARKER); + hash.update(name); + copied = operand.next; + } + hash.update(blob.subarray(copied)); + return hash.digest("hex"); +}; + export const signature = ( blob: Buffer, rawSha256?: string, -): ManagedSignature => ({ - raw_length: blob.length, - raw_sha256: rawSha256 ?? sha256Bytes(blob), - ...decodeSignature(blob), -}); + typeName: SignatureTypeName = () => null, +): ManagedSignature => { + const operands: TypeOperand[] = []; + const decoded = decodeSignature(blob, operands); + return { + raw_length: blob.length, + raw_sha256: rawSha256 ?? sha256Bytes(blob), + resolved_sha256: + decoded.kind === "unknown" + ? null + : resolvedSha256(blob, operands, typeName), + ...decoded, + }; +}; diff --git a/src/dotnet/ManagedMemberInstructionDecoder.ts b/src/dotnet/ManagedMemberInstructionDecoder.ts index 0c04b7a368..7fbe8810b0 100644 --- a/src/dotnet/ManagedMemberInstructionDecoder.ts +++ b/src/dotnet/ManagedMemberInstructionDecoder.ts @@ -543,11 +543,36 @@ export const parseExceptionRegions = ( "Exception section size leaves artifact", regions, ); - if ((size - 4) % clauseSize !== 0) + let count: number; + let sectionSize = size; + if (fat) { + if ((size - 4) % clauseSize !== 0) + return malformedExceptionRegions( + "Exception section size does not contain whole clauses", + regions, + ); + count = (size - 4) / clauseSize; + } else if ((size - 4) % clauseSize === 0) { + count = (size - 4) / clauseSize; + } else if (size >= clauseSize && size % clauseSize === 0) { + // CoreCLR accommodates legacy VB/MC++ sections whose DataSize omits the + // four-byte section header. DataSize is still not the physical extent. + count = size / clauseSize; + sectionSize = 4 + count * clauseSize; + } else { return malformedExceptionRegions( "Exception section size does not contain whole clauses", regions, ); + } + if ( + sectionOffset > methodEnd - sectionSize || + sectionOffset > bytes.length - sectionSize + ) + return malformedExceptionRegions( + "Exception section size leaves artifact", + regions, + ); if ( !fat && (bytes[sectionOffset + 2] !== 0 || bytes[sectionOffset + 3] !== 0) @@ -557,7 +582,6 @@ export const parseExceptionRegions = ( regions, ); - const count = (size - 4) / clauseSize; // Accumulates with the method IL bound; the body reserves what it retains. budget?.admitTransient(count * 256, sectionOffset); for (const region of readExceptionClauses( @@ -569,7 +593,7 @@ export const parseExceptionRegions = ( regions.push(region); hasMoreSections = (kind & 0x80) !== 0; - sectionOffset += size; + sectionOffset += sectionSize; } return { status: "complete", regions }; }; diff --git a/src/dotnet/ManagedMemberOwnership.test.ts b/src/dotnet/ManagedMemberOwnership.test.ts index 9e570a195e..cee5e985a7 100644 --- a/src/dotnet/ManagedMemberOwnership.test.ts +++ b/src/dotnet/ManagedMemberOwnership.test.ts @@ -6,6 +6,7 @@ import { inspectManagedMembersBytes } from "./ManagedMemberInspector.js"; import { managedDecodeBudget } from "./ManagedDecodeBudget.js"; import { memberDeclaringTypeRowLookup } from "./ManagedMemberOwnership.js"; import { memberListIndexSize } from "./ManagedMemberPointers.js"; +import { readTypeMemberListFacts } from "./ManagedTypeMemberLists.js"; import { readManagedMetadataLayout } from "./ManagedMetadataLayout.js"; import { buildManagedMemberPointerFixture, @@ -28,15 +29,15 @@ const metadataLayout = (bytes: Buffer) => { describe("managed declaring type ownership", () => { it.prop([ fc.tuple( - fc.tuple( - ...Array.from({ length: 3 }, () => fc.integer({ min: 0, max: 6 })), - ), - fc.tuple( - ...Array.from({ length: 3 }, () => fc.integer({ min: 0, max: 6 })), - ), + fc + .array(fc.integer({ min: 1, max: 3 }), { minLength: 3, maxLength: 3 }) + .map((starts) => starts.sort((left, right) => left - right)), + fc + .array(fc.integer({ min: 1, max: 3 }), { minLength: 3, maxLength: 3 }) + .map((starts) => starts.sort((left, right) => left - right)), ), ])( - "retains first-match field and method owners for arbitrary metadata lists", + "matches literal ownership ranges for valid nondecreasing list starts", ([fields, methods]) => { const bytes = buildManagedMemberPointerFixture(); const layout = metadataLayout(bytes); @@ -64,11 +65,16 @@ describe("managed declaring type ownership", () => { ] as const) { // The fixture's pointers reorder physical rows 1,2 to list positions 2,1. const expected = [2, 1].map((position) => { - const owner = starts.findIndex( - (start, index) => - position >= start && position < (starts[index + 1] ?? 3), - ); - return owner < 0 ? null : members.types[owner]?.full_name; + let owner: number | null = null; + for (let index = 0; index < starts.length; index += 1) { + const start = starts[index]; + const end = starts[index + 1] ?? 3; + if (start !== undefined && position >= start && position < end) { + owner = index; + break; + } + } + return owner === null ? null : members.types[owner]?.full_name; }); expect(members[table].map((member) => member.declaring_type)).toEqual( expected, @@ -80,10 +86,10 @@ describe("managed declaring type ownership", () => { }, ); - it("indexes large unordered member lists once per layout", () => { - const count = 100_000; + it("indexes wide valid member lists once per layout", () => { + const count = 65_538; const bytes = buildManagedPeFixture({ - additionalEmptyTypes: count - 1, + additionalEmptyTypes: count - 3, methodRowCount: count, }); const layout = metadataLayout(bytes); @@ -95,24 +101,25 @@ describe("managed declaring type ownership", () => { 2 * layout.stringIndexSize + layout.codedIndexSize("TypeDefOrRef") + memberListIndexSize(layout, 4); - // The first reversed range forces first-match semantics, with the owner last. - // 100,000 MethodDef rows require a four-byte method-list index. - bytes.writeUInt32LE(2, methodOffset); - const start = performance.now(); - const lookup = memberDeclaringTypeRowLookup(bytes, layout, 6); - expect( - Array.from({ length: count }, (_, row) => lookup(row + 1)).every( - (owner) => owner === count, - ), - ).toBe(true); - expect(performance.now() - start).toBeLessThan(2000); - expect(memberDeclaringTypeRowLookup(bytes, layout, 6)).toBe(lookup); + // Each preceding TypeDef owns one row; later TypeDefs have empty intervals. + // This exercises the four-byte list-index boundary without malformed starts. + for (let row = 0; row < count; row += 1) + bytes.writeUInt32LE( + Math.min(row + 1, 65_537), + methodOffset + row * types.rowSize, + ); + const facts = readTypeMemberListFacts(bytes, layout); + const lookup = memberDeclaringTypeRowLookup(bytes, layout, 6, facts); + expect(lookup(65_536)).toBe(65_536); + expect(lookup(65_535)).toBe(65_535); + expect(memberDeclaringTypeRowLookup(bytes, layout, 6, facts)).toBe(lookup); expect([lookup(0), lookup(count + 1)]).toEqual([0, 0]); }); it("admits ownership storage before allocation and retains failed admission", () => { const bytes = buildManagedPeFixture(); const layout = metadataLayout(bytes); + const facts = readTypeMemberListFacts(bytes, layout); const budget = managedDecodeBudget(layout); const capacityFailure = new ManagedReaderFailure({ code: "resource-limit", @@ -134,12 +141,12 @@ describe("managed declaring type ownership", () => { }), ); try { - expect(() => memberDeclaringTypeRowLookup(bytes, layout, 6)).toThrow( - capacityFailure, - ); - expect(() => memberDeclaringTypeRowLookup(bytes, layout, 6)).toThrow( - capacityFailure, - ); + expect(() => + memberDeclaringTypeRowLookup(bytes, layout, 6, facts), + ).toThrow(capacityFailure); + expect(() => + memberDeclaringTypeRowLookup(bytes, layout, 6, facts), + ).toThrow(capacityFailure); expect(admission).toHaveBeenCalledTimes(1); expect(allocations).toBe(0); } finally { diff --git a/src/dotnet/ManagedMemberOwnership.ts b/src/dotnet/ManagedMemberOwnership.ts index 0292f5e0ab..6119b214f5 100644 --- a/src/dotnet/ManagedMemberOwnership.ts +++ b/src/dotnet/ManagedMemberOwnership.ts @@ -1,33 +1,35 @@ import { managedDecodeBudget } from "./ManagedDecodeBudget.js"; -import { metadataRowCursor } from "./ManagedMetadataHeaps.js"; import type { ManagedMetadataLayout } from "./ManagedMetadataLayout.js"; import { - memberListIndexSize, memberListRowLookup, memberListTable, } from "./ManagedMemberPointers.js"; import { ManagedReaderFailure } from "./ManagedReaderFailure.js"; +import type { + MemberDefinitionTable, + TypeMemberListFacts, +} from "./ManagedTypeMemberLists.js"; -type MemberTable = 4 | 6; type OwnerLookup = (definitionRow: number) => number; const lookups = new WeakMap< - ManagedMetadataLayout, - Map + TypeMemberListFacts, + Map >(); /** One first-match TypeDef owner per physical member, shared by every inspection facet. */ export const memberDeclaringTypeRowLookup = ( bytes: Buffer, layout: ManagedMetadataLayout, - table: MemberTable, + table: MemberDefinitionTable, + facts: TypeMemberListFacts, ): OwnerLookup => { - let byTable = lookups.get(layout); + let byTable = lookups.get(facts); const cached = byTable?.get(table); if (cached instanceof ManagedReaderFailure) throw cached; if (cached !== undefined) return cached; if (byTable === undefined) { byTable = new Map(); - lookups.set(layout, byTable); + lookups.set(facts, byTable); } try { const positionOf = memberListRowLookup(bytes, layout, table); @@ -50,30 +52,26 @@ export const memberDeclaringTypeRowLookup = ( } return root; }; - const startForType = (row: number): number => { - const cursor = metadataRowCursor(bytes, layout, 2, row); - cursor.readUInt32(); - cursor.readIndex(layout.stringIndexSize); - cursor.readIndex(layout.stringIndexSize); - cursor.readIndex(layout.codedIndexSize("TypeDefOrRef")); - const fieldStart = cursor.readIndex(memberListIndexSize(layout, 4)); - return table === 4 - ? fieldStart - : cursor.readIndex(memberListIndexSize(layout, 6)); - }; - let start = types === 0 ? 0 : startForType(1); for (let row = 1; row <= types; row += 1) { - const end = row === types ? positions + 1 : startForType(row + 1); - const limit = Math.min(end, positions + 1); - for ( - let p = unowned(Math.min(Math.max(start, 1), positions + 1)); - p < limit; - p = unowned(p + 1) - ) { - owners[p] = row; - next[p] = p + 1; + const memberStart = + table === 4 + ? facts.entries[row - 1]?.field + : facts.entries[row - 1]?.method; + const start = memberStart?.start.value ?? null; + const end = memberStart?.end ?? null; + // A malformed endpoint makes this entire interval uncertain. In + // particular, zero must not become row one through owner-side clamping. + if (start !== null && end !== null) { + const limit = Math.min(end, positions + 1); + for ( + let p = unowned(Math.min(start, positions + 1)); + p < limit; + p = unowned(p + 1) + ) { + owners[p] = row; + next[p] = p + 1; + } } - start = end; } const definitions = layout.table(table)?.rowCount ?? 0; const lookup: OwnerLookup = (definition) => diff --git a/src/dotnet/ManagedMemberPointers.ts b/src/dotnet/ManagedMemberPointers.ts index 3517d0a111..876440c78b 100644 --- a/src/dotnet/ManagedMemberPointers.ts +++ b/src/dotnet/ManagedMemberPointers.ts @@ -1,5 +1,8 @@ import { managedDecodeBudget } from "./ManagedDecodeBudget.js"; -import type { ManagedMetadataLayout } from "./ManagedMetadataLayout.js"; +import type { + ManagedMetadataLayout, + MetadataTableLayout, +} from "./ManagedMetadataLayout.js"; import { managedFailure, ManagedReaderFailure, @@ -32,6 +35,17 @@ export const memberListIndexSize = ( ? 4 : 2; +/** Normalize a TypeDef list coordinate once for parsing and owner lookup. */ +export const validatedMemberListStart = ( + start: number, + table: MetadataTableLayout | undefined, +): number | null => { + const total = table?.rowCount ?? 0; + return Number.isSafeInteger(start) && start >= 1 && start <= total + 1 + ? start + : null; +}; + /** Resolve physical definition rows to list positions without changing token identity. */ export const memberListRowLookup = ( bytes: Buffer, diff --git a/src/dotnet/ManagedMemberRows.ts b/src/dotnet/ManagedMemberRows.ts index 700f723e67..1a5af8f0bd 100644 --- a/src/dotnet/ManagedMemberRows.ts +++ b/src/dotnet/ManagedMemberRows.ts @@ -17,6 +17,7 @@ import { type ManagedMemberRef, type ManagedMethod, type MemberCore, + type SignatureTypeName, type TypeRange, } from "./ManagedMemberInspectorCore.js"; import { @@ -30,6 +31,7 @@ import { import { readManagedValue } from "./ManagedReaderFailure.js"; import { methodBody } from "./ManagedMethodBodyReader.js"; import { memberDeclaringTypeRowLookup } from "./ManagedMemberOwnership.js"; +import type { TypeMemberListFacts } from "./ManagedTypeMemberLists.js"; /** * Declaring-type tokens whose members are decoded in full; null selects every @@ -47,7 +49,14 @@ export const parseFields = ( bytes: Buffer, layout: ManagedMetadataLayout, ranges: readonly TypeRange[], - selected: SelectedTypeTokens = null, + memberLists: TypeMemberListFacts, + { + signatureTypeName, + selected = null, + }: { + readonly signatureTypeName: SignatureTypeName; + readonly selected?: SelectedTypeTokens; + }, ): { readonly fields: readonly ManagedField[]; readonly core: ReadonlyMap; @@ -58,7 +67,7 @@ export const parseFields = ( const issues: ManagedParseIssue[] = []; const table = layout.table(4); const ownerRow = readManagedValue( - () => memberDeclaringTypeRowLookup(bytes, layout, 4), + () => memberDeclaringTypeRowLookup(bytes, layout, 4, memberLists), issues, ); for (let row = 1; row <= (table?.rowCount ?? 0); row += 1) { @@ -97,7 +106,7 @@ export const parseFields = ( declaring_type: declared?.fullName ?? null, name, flags, - signature: budgetedSignature(sig, layout), + signature: budgetedSignature(sig, layout, signatureTypeName), }); }, issues); } @@ -179,6 +188,7 @@ export const parseMemberRefs = ( bytes: Buffer, layout: ManagedMetadataLayout, typeName: (token: string | null) => string | null, + signatureTypeName: SignatureTypeName, ): { readonly refs: readonly ManagedMemberRef[]; readonly core: ReadonlyMap; @@ -230,7 +240,7 @@ export const parseMemberRefs = ( layout.rowCounts, ); const parentName = typeName(parentToken); - const decoded = budgetedSignature(sig, layout); + const decoded = budgetedSignature(sig, layout, signatureTypeName); refs.push({ token, row_offset: cursor.start, @@ -256,7 +266,11 @@ interface ParseMethodsInput { readonly layout: ManagedMetadataLayout; readonly pe: ManagedPeLayout; readonly ranges: readonly TypeRange[]; + readonly signatureTypeName: SignatureTypeName; + readonly memberLists: TypeMemberListFacts; readonly selected?: SelectedTypeTokens; + readonly methodOffset?: number; + readonly methodLimit?: number; } export const parseMethods = ({ @@ -264,7 +278,11 @@ export const parseMethods = ({ layout, pe, ranges, + signatureTypeName, + memberLists, selected = null, + methodOffset = 0, + methodLimit = Number.MAX_SAFE_INTEGER, }: ParseMethodsInput): { readonly methods: readonly ManagedMethod[]; readonly core: ReadonlyMap; @@ -281,7 +299,7 @@ export const parseMethods = ({ let unscannedMethods = 0; const table = layout.table(6); const ownerRow = readManagedValue( - () => memberDeclaringTypeRowLookup(bytes, layout, 6), + () => memberDeclaringTypeRowLookup(bytes, layout, 6, memberLists), issues, ); for (let row = 1; row <= (table?.rowCount ?? 0); row += 1) { @@ -292,12 +310,14 @@ export const parseMethods = ({ ? undefined : readManagedValue(() => ownerRow(row), issues); const declared = owner === undefined ? undefined : ranges[owner - 1]; + const inPage = row > methodOffset && row - methodOffset <= methodLimit; const mayBeSelected = - selected === null || - owner === undefined || - isSelected(selected, declared) || - // A type whose name is unreadable may be one the caller requested. - declared?.fullName === null; + inPage && + (selected === null || + owner === undefined || + isSelected(selected, declared) || + // A type whose name is unreadable may be one the caller requested. + declared?.fullName === null); const produced = methods.length; readManagedValue(() => { const cursor = metadataRowCursor(bytes, layout, 6, row); @@ -318,6 +338,7 @@ export const parseMethods = ({ kind: "method", } as const; core.set(token, member); + if (!inPage) return; if (!isSelected(selected, declared)) return; const sig = readMetadataBlob( bytes, @@ -326,7 +347,7 @@ export const parseMethods = ({ layout.blob.size, ); cursor.readIndex(layout.tableIndexSize(8)); - const decoded = budgetedSignature(sig, layout); + const decoded = budgetedSignature(sig, layout, signatureTypeName); core.set(token, { ...member, genericArity: signatureGenericArity(decoded), @@ -468,12 +489,13 @@ const callerNameBytes = (method: ManagedMethod): number => const budgetedSignature = ( bytes: Buffer, layout: ManagedMetadataLayout, + signatureTypeName: SignatureTypeName, ): ReturnType => { const budget = managedDecodeBudget(layout); budget.admitTransient(bytes.length * 256, layout.blob.offset); try { return admitManagedProjection( - signature(bytes, budget.digest(bytes, "sha256")), + signature(bytes, budget.digest(bytes, "sha256"), signatureTypeName), budget, ); } finally { diff --git a/src/dotnet/ManagedMetadataExtraData.test.ts b/src/dotnet/ManagedMetadataExtraData.test.ts index 38434390b9..012e8d3352 100644 --- a/src/dotnet/ManagedMetadataExtraData.test.ts +++ b/src/dotnet/ManagedMetadataExtraData.test.ts @@ -153,7 +153,8 @@ describe("CLR ExtraData index widths and boundaries", () => { it("reports a truncated row count before interpreting ExtraData", () => { const bytes = buildManagedPeFixture({ metadataExtraData: 0x1234_5678 }); - const { streamHeader, extraDataOffset } = fixtureTablesStream(bytes); + const { streamHeader, tablesOffset, extraDataOffset } = + fixtureTablesStream(bytes); bytes.writeUInt32LE(extraDataOffset - 1, streamHeader + 4); const result = inspectManagedMembersBytes( bytes, @@ -166,6 +167,8 @@ describe("CLR ExtraData index widths and boundaries", () => { expect.objectContaining({ code: "invalid-metadata-root", scope: "metadata.table-row-count", + // The last row count's artifact file offset, not its stream offset. + offset: tablesOffset + extraDataOffset - 4, }), ], }); @@ -173,6 +176,24 @@ describe("CLR ExtraData index widths and boundaries", () => { expect(result.methods).toEqual([]); }); + it("reports a stream extent beyond the metadata root at its file offset", () => { + const bytes = buildManagedPeFixture(); + const { streamHeader, tablesOffset } = fixtureTablesStream(bytes); + bytes.writeUInt32LE(0xffff_ffff, streamHeader + 4); + const result = inspectManagedMembersBytes( + bytes, + managedPeFixtureTarget(bytes), + ); + + expect(result.coverage.issues).toEqual([ + expect.objectContaining({ + code: "invalid-metadata-root", + scope: "metadata.#~", + offset: tablesOffset, + }), + ]); + }); + it.each([0, 1, 3])( "rejects an ExtraData word with only %i declared bytes", (availableBytes) => { diff --git a/src/dotnet/ManagedMetadataInventory.ts b/src/dotnet/ManagedMetadataInventory.ts index c31059d575..f2d5a90452 100644 --- a/src/dotnet/ManagedMetadataInventory.ts +++ b/src/dotnet/ManagedMetadataInventory.ts @@ -15,6 +15,7 @@ import { readResource, } from "./ManagedMetadataInventoryRows.js"; import { metadataToken } from "./ManagedMetadataHeaps.js"; +import type { TypeMemberListFacts } from "./ManagedTypeMemberLists.js"; import { readManagedValue, uniqueManagedParseIssues, @@ -149,13 +150,22 @@ const readAttributes = ( bytes: Buffer, layout: ManagedMetadataLayout, issues: ManagedParseIssue[], + memberLists: TypeMemberListFacts, ): { readonly attributes: readonly CustomAttribute[]; readonly targetFrameworks: readonly string[]; } => { const attributes = readRows(layout.table(12), (row) => readManagedValue( - () => readCustomAttribute(bytes, layout, row, heapExtent(layout), issues), + () => + readCustomAttribute( + bytes, + layout, + row, + heapExtent(layout), + issues, + memberLists, + ), issues, ), ); @@ -177,8 +187,9 @@ export const readManagedMetadataInventory = ( bytes: Buffer, layout: ManagedMetadataLayout, resourceDirectory: ManagedResourceDirectory | null, + memberLists: TypeMemberListFacts, ): ManagedMetadataInventory => { - const issues: ManagedParseIssue[] = []; + const issues: ManagedParseIssue[] = [...memberLists.issues]; validateIdentityTableCounts(layout, issues); const module = readManagedValue( @@ -201,6 +212,7 @@ export const readManagedMetadataInventory = ( bytes, layout, issues, + memberLists, ); return { module, diff --git a/src/dotnet/ManagedMetadataInventoryRows.ts b/src/dotnet/ManagedMetadataInventoryRows.ts index 30f9e8e5a3..3eb54cf50b 100644 --- a/src/dotnet/ManagedMetadataInventoryRows.ts +++ b/src/dotnet/ManagedMetadataInventoryRows.ts @@ -1,5 +1,6 @@ import { managedDecodeBudget } from "./ManagedDecodeBudget.js"; import { memberDeclaringTypeRowLookup } from "./ManagedMemberOwnership.js"; +import type { TypeMemberListFacts } from "./ManagedTypeMemberLists.js"; import type { ManagedArtifactInspection, @@ -248,8 +249,14 @@ const declaringTypeForMethod = ( layout: ManagedMetadataLayout, methodRow: number, heapExtent: number, + memberLists: TypeMemberListFacts, ): string | null => { - const owner = memberDeclaringTypeRowLookup(bytes, layout, 6)(methodRow); + const owner = memberDeclaringTypeRowLookup( + bytes, + layout, + 6, + memberLists, + )(methodRow); if (owner === 0) return null; return readTypeName({ bytes, layout, table: 2, row: owner, heapExtent }); }; @@ -259,11 +266,13 @@ const attributeTypeName = ( layout: ManagedMetadataLayout, rawType: number, heapExtent: number, + memberLists: TypeMemberListFacts, ): string | null => { const tag = rawType & 7; const row = Math.floor(rawType / 8); if (row === 0) return null; - if (tag === 2) return declaringTypeForMethod(bytes, layout, row, heapExtent); + if (tag === 2) + return declaringTypeForMethod(bytes, layout, row, heapExtent, memberLists); if (tag !== 3) return null; const memberRefs = layout.table(10); if (memberRefs === undefined || row > memberRefs.rowCount) return null; @@ -288,7 +297,13 @@ const attributeTypeName = ( heapExtent, }); if (parentTag === 3) - return declaringTypeForMethod(bytes, layout, parentRow, heapExtent); + return declaringTypeForMethod( + bytes, + layout, + parentRow, + heapExtent, + memberLists, + ); return null; }; @@ -349,6 +364,7 @@ export const readCustomAttribute = ( row: number, heapExtent: number, issues: ManagedParseIssue[], + memberLists: TypeMemberListFacts, ): CustomAttribute => { const cursor = metadataRowCursor(bytes, layout, 12, row); const parentRaw = cursor.readIndex( @@ -411,7 +427,8 @@ export const readCustomAttribute = ( if (constructor !== null) typeName = readManagedValue( - () => attributeTypeName(bytes, layout, typeRaw, heapExtent), + () => + attributeTypeName(bytes, layout, typeRaw, heapExtent, memberLists), issues, ) ?? null; return { diff --git a/src/dotnet/ManagedMetadataLayout.ts b/src/dotnet/ManagedMetadataLayout.ts index b388546453..9ad02a0aa7 100644 --- a/src/dotnet/ManagedMetadataLayout.ts +++ b/src/dotnet/ManagedMetadataLayout.ts @@ -116,8 +116,10 @@ const CODED_INDEXES = { const SUPPORTED_TABLE_MASK = (1n << BigInt(METADATA_TABLE_NAMES.length)) - 1n; +/** Check a range within `bytes`, which start at artifact file offset `base`. */ const requireRange = ( bytes: Buffer, + base: number, offset: number, length: number, scope: string, @@ -133,7 +135,7 @@ const requireRange = ( "invalid-metadata-root", scope, `${scope} leaves the metadata byte range`, - Number.isSafeInteger(offset) && offset >= 0 ? offset : null, + Number.isSafeInteger(base + offset) && offset >= 0 ? base + offset : null, ); }; @@ -224,7 +226,7 @@ const readMetadataStreams = ( rootOffset, ); const versionLength = bytes.readUInt32LE(12); - requireRange(bytes, 16, versionLength, "metadata.version"); + requireRange(bytes, rootOffset, 16, versionLength, "metadata.version"); const versionBytes = bytes.subarray(16, 16 + versionLength); const zero = versionBytes.indexOf(0); const decodedVersionLength = zero < 0 ? versionBytes.length : zero; @@ -233,7 +235,7 @@ const readMetadataStreams = ( .subarray(0, decodedVersionLength) .toString("utf8"); let cursor = align4(16 + versionLength); - requireRange(bytes, cursor, 4, "metadata.stream-count"); + requireRange(bytes, rootOffset, cursor, 4, "metadata.stream-count"); const streamCount = bytes.readUInt16LE(cursor + 2); if (streamCount === 0 || streamCount > 64) throw managedFailure( @@ -245,7 +247,7 @@ const readMetadataStreams = ( cursor += 4; const streams = new Map(); for (let index = 0; index < streamCount; index += 1) { - requireRange(bytes, cursor, 8, "metadata.stream-header"); + requireRange(bytes, rootOffset, cursor, 8, "metadata.stream-header"); const relativeOffset = bytes.readUInt32LE(cursor); const streamSize = bytes.readUInt32LE(cursor + 4); const named = readStreamName(bytes, cursor + 8); @@ -256,7 +258,13 @@ const readMetadataStreams = ( `Duplicate metadata stream ${named.name}`, rootOffset + cursor, ); - requireRange(bytes, relativeOffset, streamSize, `metadata.${named.name}`); + requireRange( + bytes, + rootOffset, + relativeOffset, + streamSize, + `metadata.${named.name}`, + ); streams.set(named.name, { name: named.name, offset: rootOffset + relativeOffset, @@ -295,7 +303,13 @@ const readTablesHeader = ( tablesStream.offset, tablesStream.offset + tablesStream.size, ); - requireRange(tableBytes, 0, 24, "metadata.tables-header"); + requireRange( + tableBytes, + tablesStream.offset, + 0, + 24, + "metadata.tables-header", + ); const heapSizes = tableBytes[6] ?? 0; const heaps = { string: (heapSizes & 0x01) === 0 ? (2 as const) : (4 as const), @@ -315,13 +329,20 @@ const readTablesHeader = ( const readRowCounts = ( tableBytes: Buffer, + tablesOffset: number, valid: bigint, ): readonly number[] => { const rowCounts = Array(METADATA_TABLE_NAMES.length).fill(0); let tableCursor = 24; for (let index = 0; index < METADATA_TABLE_NAMES.length; index += 1) { if ((valid & (1n << BigInt(index))) === 0n) continue; - requireRange(tableBytes, tableCursor, 4, "metadata.table-row-count"); + requireRange( + tableBytes, + tablesOffset, + tableCursor, + 4, + "metadata.table-row-count", + ); const count = tableBytes.readUInt32LE(tableCursor); rowCounts[index] = count; tableCursor += 4; @@ -384,7 +405,13 @@ const buildMetadataTables = ({ "Metadata table byte size overflowed", tablesOffset + tableCursor, ); - requireRange(tableBytes, tableCursor, byteLength, "metadata.table-data"); + requireRange( + tableBytes, + tablesOffset, + tableCursor, + byteLength, + "metadata.table-data", + ); tables.set(index, { index, name: METADATA_TABLE_NAMES[index] ?? `Table${String(index)}`, @@ -417,7 +444,7 @@ export const readManagedMetadataLayout = ( artifact, streams, ); - const rowCounts = readRowCounts(tableBytes, valid); + const rowCounts = readRowCounts(tableBytes, tablesOffset, valid); const tables = buildMetadataTables({ tableBytes, rowCounts, diff --git a/src/dotnet/ManagedNativeBoundaryHelpers.ts b/src/dotnet/ManagedNativeBoundaryHelpers.ts index e4b1a59600..d0d3dd768b 100644 --- a/src/dotnet/ManagedNativeBoundaryHelpers.ts +++ b/src/dotnet/ManagedNativeBoundaryHelpers.ts @@ -20,6 +20,8 @@ import { managedTableRowCounts } from "./ManagedMetadataInventory.js"; import type { ManagedMetadataInventory } from "./ManagedMetadataInventory.js"; import { managedCli32BitRequirement, + managedCliIsMixedClrNative, + type ManagedCliHeader, type ManagedPeLayout, } from "./ManagedPeReader.js"; import { readManagedValue } from "./ManagedReaderFailure.js"; @@ -378,36 +380,37 @@ const boundaryKind = ( }; export const cliNative = ( - pe: ManagedPeLayout, + cli: ManagedCliHeader | null, ): ManagedNativeBoundaryInspection["cli_native"] => { - if (pe.cli === null) return null; + if (cli === null) return null; return { - il_only: (pe.cli.flags & 0x0000_0001) !== 0, - requires_32bit: managedCli32BitRequirement(pe.cli.flags) === "required", - strong_name_signed: (pe.cli.flags & 0x0000_0008) !== 0, - native_entry_point: (pe.cli.flags & 0x0000_0010) !== 0, - ready_to_run_signature: pe.cli.readyToRunSignature, - managed_native_header_rva: pe.cli.managedNativeHeader.rva, - managed_native_header_size: pe.cli.managedNativeHeader.size, + il_only: (cli.flags & 0x0000_0001) !== 0, + requires_32bit: managedCli32BitRequirement(cli.flags) === "required", + strong_name_signed: (cli.flags & 0x0000_0008) !== 0, + native_entry_point: (cli.flags & 0x0000_0010) !== 0, + ready_to_run_signature: cli.readyToRunSignature, + managed_native_header_rva: cli.managedNativeHeader.rva, + managed_native_header_size: cli.managedNativeHeader.size, }; }; /** Summarize declaration counts with the native facets of the CLI header. */ export const nativeBoundarySummary = ( - native: ManagedNativeBoundaryInspection["cli_native"], + cli: ManagedCliHeader | null, counts: Pick< ManagedNativeBoundaryInspection["summary"], "module_ref_count" | "pinvoke_import_count" | "native_implementation_count" >, ): ManagedNativeBoundaryInspection["summary"] => ({ ...counts, - ready_to_run: native?.ready_to_run_signature ?? null, + ready_to_run: cli?.readyToRunSignature ?? null, mixed_mode_or_native_header: - native === null + cli === null ? null - : native.managed_native_header_rva !== 0 || - native.managed_native_header_size !== 0 || - native.native_entry_point, + : managedCliIsMixedClrNative(cli) || + cli.managedNativeHeader.rva !== 0 || + cli.managedNativeHeader.size !== 0 || + (cli.flags & 0x0000_0010) !== 0, }); interface BoundaryInspectionContext { @@ -419,19 +422,18 @@ interface BoundaryInspectionContext { readonly moduleRefs: readonly ModuleRef[]; readonly imports: readonly NativeImport[]; readonly implementations: readonly NativeImplementation[]; - readonly native: ManagedNativeBoundaryInspection["cli_native"]; readonly issues: readonly ManagedParseIssue[]; } export const buildNativeBoundaryInspection = ({ target, bytes, + pe, layout, inventory, moduleRefs, imports, implementations, - native, issues, }: BoundaryInspectionContext): ManagedNativeBoundaryInspection => managedNativeBoundaryInspectionSchema.parse( @@ -454,11 +456,11 @@ export const buildNativeBoundaryInspection = ({ requires_artifact_sha256: target.sha256, requires_mvid: inventory.module?.mvid ?? null, }, - cli_native: native, + cli_native: cliNative(pe.cli), module_refs: moduleRefs, pinvoke_imports: imports, native_implementations: implementations, - summary: nativeBoundarySummary(native, { + summary: nativeBoundarySummary(pe.cli, { module_ref_count: moduleRefs.length, pinvoke_import_count: imports.length, native_implementation_count: implementations.length, diff --git a/src/dotnet/ManagedNativeBoundaryInspector.test.ts b/src/dotnet/ManagedNativeBoundaryInspector.test.ts index 18ea8d9885..8be4d882c8 100644 --- a/src/dotnet/ManagedNativeBoundaryInspector.test.ts +++ b/src/dotnet/ManagedNativeBoundaryInspector.test.ts @@ -76,6 +76,35 @@ describe("managed native boundaries", () => { "P/Invoke rows prove managed import declarations only; this inspection does not verify that a native library, export, thunk, or provider-qualified function exists.", ); }); + + it.each([ + { + name: "C++/CLI with a native method body", + options: { + cliFlags: 0, + methods: [ + { name: "NativeMethod", body: Buffer.from([0xc3]), implFlags: 1 }, + ], + }, + mixed: true, + }, + { + name: "ILONLY clear with unreadable metadata", + options: { cliFlags: 0, corruptMetadataSignature: true }, + mixed: true, + }, + { name: "pure IL", options: {}, mixed: false }, + { name: "ReadyToRun", options: { readyToRun: true }, mixed: true }, + ])("agrees with artifact classification for $name", ({ options, mixed }) => { + const bytes = buildManagedPeFixture(options); + const target = managedPeFixtureTarget(bytes); + const artifact = inspectManagedArtifactBytes(bytes, target); + const boundaries = inspectManagedNativeBoundariesBytes(bytes, target); + + expect(boundaries.summary.mixed_mode_or_native_header).toBe(mixed); + if (artifact.classification.runtime_family === "mixed-clr-native") + expect(boundaries.summary.mixed_mode_or_native_header).toBe(true); + }); }); describe("managed native boundary coded indexes", () => { diff --git a/src/dotnet/ManagedNativeBoundaryInspector.ts b/src/dotnet/ManagedNativeBoundaryInspector.ts index d2b84be049..42e057e74c 100644 --- a/src/dotnet/ManagedNativeBoundaryInspector.ts +++ b/src/dotnet/ManagedNativeBoundaryInspector.ts @@ -13,8 +13,10 @@ import { readManagedMetadataLayout, type ManagedMetadataLayout, } from "./ManagedMetadataLayout.js"; +import { readTypeMemberListFacts } from "./ManagedTypeMemberLists.js"; import { readManagedPeLayout, + type ManagedCliHeader, type ManagedPeLayout, } from "./ManagedPeReader.js"; import { ManagedReaderFailure } from "./ManagedReaderFailure.js"; @@ -32,8 +34,8 @@ import { type Inventory = ReturnType; /** - * Report a PE whose CLI metadata is absent or cannot be admitted. `native` - * holds the CLI header facts when the header itself was admitted; otherwise + * Report a PE whose CLI metadata is absent or cannot be admitted. `cli` + * holds the CLI header when the header itself was admitted; otherwise * the header is absent (`not-managed`) or unreadable (`malformed`). */ const emptyInspection = ( @@ -41,10 +43,10 @@ const emptyInspection = ( bytes: Buffer, classification: "not-managed" | "malformed", { - native = null, + cli = null, issues = [], }: { - readonly native?: ManagedNativeBoundaryInspection["cli_native"] | null; + readonly cli?: ManagedCliHeader | null; readonly issues?: readonly ManagedParseIssue[]; } = {}, ): ManagedNativeBoundaryInspection => { @@ -71,11 +73,11 @@ const emptyInspection = ( requires_artifact_sha256: target.sha256, requires_mvid: null, }, - cli_native: native, + cli_native: cliNative(cli), module_refs: [], pinvoke_imports: [], native_implementations: [], - summary: nativeBoundarySummary(native, { + summary: nativeBoundarySummary(cli, { module_ref_count: 0, pinvoke_import_count: 0, native_implementation_count: 0, @@ -108,10 +110,12 @@ const readBoundaryInventory = ( metadataOffset, cli.metadata.size, ); + const memberLists = readTypeMemberListFacts(bytes, layout); const inventory = readManagedMetadataInventory( bytes, layout, readManagedResourceDirectory(pe, issues), + memberLists, ); return { layout, inventory }; }; @@ -138,7 +142,7 @@ export const inspectManagedNativeBoundariesBytes = ( } catch (cause: unknown) { if (!(cause instanceof ManagedReaderFailure)) throw cause; return emptyInspection(target, bytes, "malformed", { - native: cliNative(pe), + cli: pe.cli, issues: [cause.issue], }); } @@ -176,13 +180,12 @@ export const inspectManagedNativeBoundariesBytes = ( moduleRefs, imports, implementations, - native: cliNative(pe), issues, }); } catch (cause: unknown) { if (!(cause instanceof ManagedReaderFailure)) throw cause; return emptyInspection(target, bytes, "malformed", { - native: cliNative(pe), + cli: pe.cli, issues: [...issues, cause.issue], }); } diff --git a/src/dotnet/ManagedPe.fixture.ts b/src/dotnet/ManagedPe.fixture.ts index 0d4cf49786..7487544015 100644 --- a/src/dotnet/ManagedPe.fixture.ts +++ b/src/dotnet/ManagedPe.fixture.ts @@ -53,6 +53,15 @@ interface ManagedPeFixtureOptions { readonly targetFramework?: string | null; readonly typeName?: string; readonly typeNamespace?: string; + /** NestedClass rows as [nested, enclosing] TypeDef rows. */ + readonly nestedTypes?: readonly (readonly [number, number])[]; + /** TypeRef rows after the TargetFrameworkAttribute row, from row 2. */ + readonly typeReferences?: readonly { + readonly namespace: string; + readonly name: string; + /** TypeRef row of the enclosing type, for a nested type. */ + readonly enclosingRow?: number; + }[]; /** Raw #US stream bytes; omitted means the metadata has no #US stream. */ readonly userStringHeap?: Buffer; readonly virtualSectionSize?: number; @@ -590,6 +599,11 @@ interface FixtureMetadataRowContext { readonly name: number; readonly namespace: number; }[]; + readonly typeReferences: readonly { + readonly name: number; + readonly namespace: number; + readonly enclosingRow?: number; + }[]; readonly tokenBlob: number; readonly fieldSignature: number; readonly methodSignature: number; @@ -668,6 +682,52 @@ const exportedTypeTable = ( ], ]; +const typeRefTable = ( + options: ManagedPeFixtureOptions, + indexes: FixtureMetadataRowContext["indexes"], + heapSizes: FixtureMetadataRowContext["heapSizes"], + widths: FixtureIndexWidths, +): readonly [number, readonly Buffer[]] => [ + 1, + [ + typeRefRow( + indexes.attributeName, + indexes.attributeNamespace, + heapSizes.strings, + metadataIndex(options.typeRefScopeRaw ?? 0, widths.resolutionScope), + ), + ...indexes.typeReferences.map(({ name, namespace, enclosingRow }) => + typeRefRow( + name, + namespace, + heapSizes.strings, + metadataIndex( + enclosingRow === undefined ? 0 : (enclosingRow << 2) | 3, + widths.resolutionScope, + ), + ), + ), + ], +]; + +const nestedClassTable = ( + options: ManagedPeFixtureOptions, + widths: FixtureIndexWidths, +): readonly (readonly [number, readonly Buffer[]])[] => + (options.nestedTypes?.length ?? 0) === 0 + ? [] + : [ + [ + 41, + (options.nestedTypes ?? []).map(([nested, enclosing]) => + Buffer.concat([ + metadataIndex(nested, widths.table(2)), + metadataIndex(enclosing, widths.table(2)), + ]), + ), + ], + ]; + const specTables = ( options: ManagedPeFixtureOptions, indexes: FixtureMetadataRowContext["indexes"], @@ -706,7 +766,7 @@ const fixtureIndexWidths = ({ }: FixtureMetadataRowContext): FixtureIndexWidths => { const rowCounts = new Map([ [0, options.moduleRowCount ?? 1], - [1, 1], + [1, indexes.typeReferences.length + 1], [2, indexes.additionalTypeNames.length + 1], [4, 1], [6, options.methodRowCount ?? 1], @@ -725,6 +785,7 @@ const fixtureIndexWidths = ({ [35, referenceStringIndexes.length], [39, indexes.exportedTypes.length], [40, 1], + [41, options.nestedTypes?.length ?? 0], [43, options.methodSpecMethods?.length ?? 0], ]); const count = (table: number): number => rowCounts.get(table) ?? 0; @@ -765,6 +826,9 @@ const fixtureMetadataRows = ({ indexes, heapSizes, }); + const nestedRows = new Set( + (options.nestedTypes ?? []).map(([nested]) => nested), + ); const rows = new Map([ [ 0, @@ -772,22 +836,13 @@ const fixtureMetadataRows = ({ moduleRow(indexes.moduleName, heapSizes.strings, heapSizes.guids), ), ], - [ - 1, - [ - typeRefRow( - indexes.attributeName, - indexes.attributeNamespace, - heapSizes.strings, - metadataIndex(options.typeRefScopeRaw ?? 0, widths.resolutionScope), - ), - ], - ], + typeRefTable(options, indexes, heapSizes, widths), [ 2, - [...indexes.additionalTypeNames, indexes.typeName].map((name) => + [...indexes.additionalTypeNames, indexes.typeName].map((name, index) => typeDefRow({ name, + flags: nestedRows.has(index + 1) ? 0x0010_0002 : 0x0010_0001, namespace: indexes.typeNamespace, fieldList: 1, methodList: 1, @@ -887,6 +942,7 @@ const fixtureMetadataRows = ({ ], ...exportedTypeTable(options, indexes, heapSizes, widths), ...specTables(options, indexes, heapSizes, widths), + ...nestedClassTable(options, widths), [ 40, [ @@ -959,6 +1015,13 @@ const buildManagedFixtureMetadata = ( namespace: strings.add(namespace), }), ); + const typeReferences = (options.typeReferences ?? []).map( + ({ namespace, name, enclosingRow }) => ({ + name: strings.add(name), + namespace: strings.add(namespace), + ...(enclosingRow === undefined ? {} : { enclosingRow }), + }), + ); const referenceNames = options.references ?? ["System.Runtime"]; const referenceStringIndexes = referenceNames.map((name) => strings.add(name), @@ -1000,6 +1063,7 @@ const buildManagedFixtureMetadata = ( pinvokeImportName, resourceName, exportedTypes, + typeReferences, tokenBlob, fieldSignature, methodSignature, @@ -1266,6 +1330,133 @@ export const buildManagedMemberPointerFixture = ( return image; }; +/** + * Two top-level types A and B, each enclosing a nested type named Inner with + * one method M and one field F; the first Inner also encloses Leaf with a + * method M. Every method shares the fixture body. Swapping the enclosing + * types keeps every row but moves each Inner, with its Leaf, under the other + * top-level type. NestedClass rows pair a nested TypeDef row with its + * enclosing row and may be replaced to exercise malformed metadata. + */ +export const buildManagedNestedTypeFixture = ({ + swapEnclosing = false, + nestedClassRows, + literalTopLevelName, + topLevelTypes = false, +}: { + readonly swapEnclosing?: boolean; + readonly nestedClassRows?: readonly (readonly [number, number])[]; + readonly literalTopLevelName?: string; + readonly topLevelTypes?: boolean; +} = {}): Buffer => { + const image = buildManagedFixtureImage({}, (methodRva) => { + const strings = new StringHeap(); + const blobs = new BlobHeap(); + const moduleName = strings.add("Nested.dll"); + const assemblyName = strings.add("Fixture.Nested"); + const namespace = strings.add("Fixture"); + const typeNames = [ + "", + "A", + "B", + "Inner", + literalTopLevelName ?? (topLevelTypes ? "OtherInner" : "Inner"), + "Leaf", + ].map((name) => strings.add(name)); + const methodName = strings.add("M"); + const fieldName = strings.add("F"); + const objectName = strings.add("Object"); + const systemNamespace = strings.add("System"); + const runtimeName = strings.add("System.Runtime"); + const methodSignature = blobs.add(Buffer.from([0x20, 0, 0x08])); + const fieldSignature = blobs.add(Buffer.from([0x06, 0x08])); + // TypeDef rows 4 and 5 own one method and one field each; row 6 owns a method. + const memberList = [1, 1, 1, 1, 2, 3]; + const rows = new Map([ + [0, [moduleRow(moduleName, 2, 2)]], + [ + 1, + [ + Buffer.concat([ + u16((1 << 2) | 2), + u16(objectName), + u16(systemNamespace), + ]), + ], + ], + [ + 2, + typeNames.map((name, index) => + typeDefRow({ + name, + namespace: index === 1 || index === 2 ? namespace : 0, + fieldList: Math.min(memberList[index] ?? 1, 3), + methodList: memberList[index] ?? 1, + stringSize: 2, + typeDefOrRefSize: 2, + fieldListSize: 2, + methodListSize: 2, + extendsRaw: index === 0 ? 0 : (1 << 2) | 1, + flags: + index <= 2 || + topLevelTypes || + (index === 4 && literalTopLevelName !== undefined) + ? 0x0010_0001 + : 0x0010_0002, + }), + ), + ], + [4, [1, 2].map(() => fieldRow(fieldName, fieldSignature, 2, 2))], + [ + 6, + [1, 2, 3].map(() => + methodDefRow({ + name: methodName, + signature: methodSignature, + stringSize: 2, + blobSize: 2, + parameterSize: 2, + parameterList: 1, + rva: methodRva, + flags: 0x0086, + }), + ), + ], + [ + 41, + ( + nestedClassRows ?? + (topLevelTypes + ? [] + : [ + [4, swapEnclosing ? 3 : 2], + [5, swapEnclosing ? 2 : 3], + [6, 4], + ]) + ) + .filter( + ([nested]) => literalTopLevelName === undefined || nested !== 5, + ) + .map(([nested, enclosing]) => + Buffer.concat([u16(nested), u16(enclosing)]), + ), + ], + [32, [assemblyRow(assemblyName, 2, 2)]], + [35, [assemblyRefRow(runtimeName, 0, false, 2, 2)]], + ]); + return metadataRoot({ + tables: tablesStream(rows, 0n, 0), + strings: strings.toBuffer(), + guid: guidHeap(), + blob: blobs.toBuffer(), + }); + }); + // The shared PE shell defaults to an executable; this fixture is a DLL. + image.writeUInt16LE(image.readUInt16LE(0x84 + 18) | 0x2000, 0x84 + 18); + image.writeUInt32LE(0, 0x0200 + 20); + return image; +}; + /** Build a syntactically valid PE32 fixture with no CLI directory. */ export const buildNativePeFixture = (): Buffer => { const image = buildManagedPeFixture(); diff --git a/src/dotnet/ManagedPeFixtureLayout.test.ts b/src/dotnet/ManagedPeFixtureLayout.test.ts new file mode 100644 index 0000000000..99d30a77c6 --- /dev/null +++ b/src/dotnet/ManagedPeFixtureLayout.test.ts @@ -0,0 +1,142 @@ +import { describe, expect, it } from "vitest"; + +import { buildManagedPeFixture } from "../../scripts/fixtures/managed/pe.mjs"; +import { inspectManagedArtifactBytes } from "./ManagedArtifactInspector.js"; +import { inspectManagedMembersBytes } from "./ManagedMemberInspector.js"; +import { readManagedPeLayout } from "./ManagedPeReader.js"; +import { managedPeFixtureTarget } from "./ManagedPe.fixture.js"; + +const rawOffset = (rva: number) => 0x200 + (rva - 0x2000); + +describe("managed PE fixture layout", () => { + it("preserves bodyless native methods without allocating an IL extent", () => { + const image = buildManagedPeFixture({ + methods: [ + { name: "NativeMethod", flags: 0x2016, rva: 0 }, + { name: "ManagedMethod", body: Buffer.from([0x06, 0x2a]) }, + ], + }); + const inspected = inspectManagedMembersBytes( + image, + managedPeFixtureTarget(image), + ); + expect(inspected.methods[0]?.rva).toBe(0); + expect(inspected.methods[0]?.body.status).toBe("absent"); + expect(inspected.methods[0]?.body.opcode_counts).toEqual({}); + expect(inspected.methods[1]?.body.opcode_counts).toEqual({ ret: 1 }); + }); + + it("preserves complete adjacent method bodies", () => { + const firstBody = Buffer.concat([ + Buffer.from([0x9e]), + Buffer.alloc(38), + Buffer.from([0x2a]), + ]); + const secondBody = Buffer.from([0x06, 0x2a]); + const image = buildManagedPeFixture({ + methods: [ + { name: "LongBody", body: firstBody }, + { name: "NextBody", body: secondBody }, + ], + }); + + expect( + image.subarray(rawOffset(0x2800), rawOffset(0x2800) + firstBody.length), + ).toEqual(firstBody); + expect( + image.subarray(rawOffset(0x2840), rawOffset(0x2840) + secondBody.length), + ).toEqual(secondBody); + const inspected = inspectManagedMembersBytes( + image, + managedPeFixtureTarget(image), + ); + expect(inspected.methods.map(({ body }) => body.opcode_counts)).toEqual([ + { nop: 38, ret: 1 }, + { ret: 1 }, + ]); + }); + + it("keeps the ReadyToRun signature outside expanded metadata", () => { + const image = buildManagedPeFixture({ + readyToRun: true, + methods: Array.from({ length: 47 }, (_, index) => ({ + name: `Method${index}`, + })), + }); + const nativeHeaderRva = image.readUInt32LE(0x240); + const nativeHeaderSize = image.readUInt32LE(0x244); + + expect(nativeHeaderSize).toBe(4); + expect( + image + .subarray(rawOffset(nativeHeaderRva), rawOffset(nativeHeaderRva) + 4) + .toString("latin1"), + ).toBe("RTR\0"); + }); + + it("keeps all 2045 requested resource bytes file-backed and inspectable", () => { + const resourceData = Buffer.alloc(2045, 0x41); + const image = buildManagedPeFixture({ resourceData }); + const layout = readManagedPeLayout(image); + if (layout.cli === null) throw new Error("fixture omitted CLI header"); + const directoryOffset = layout.rvaToOffset( + layout.cli.resources.rva, + layout.cli.resources.size, + "cli.resources", + ); + const inspection = inspectManagedArtifactBytes( + image, + managedPeFixtureTarget(image), + ); + + expect(layout.cli.resources.size).toBe(resourceData.length + 4); + expect(image.readUInt32LE(directoryOffset)).toBe(resourceData.length); + expect( + image.subarray( + directoryOffset + 4, + directoryOffset + 4 + resourceData.length, + ), + ).toEqual(resourceData); + expect(inspection.classification.status).toBe("managed"); + expect(inspection.resources).toMatchObject([ + { + name: "Fixture.resources", + embedded: true, + data_length: resourceData.length, + }, + ]); + expect(inspection.coverage).toMatchObject({ + state: "complete", + issues: [], + }); + }); + + it("keeps metadata for all 60 requested assembly references file-backed", () => { + const references = Array.from( + { length: 60 }, + (_, index) => `System.Reference.Assembly.${index}`, + ); + const image = buildManagedPeFixture({ references }); + const layout = readManagedPeLayout(image); + if (layout.cli === null) throw new Error("fixture omitted CLI header"); + const metadataOffset = layout.rvaToOffset( + layout.cli.metadata.rva, + layout.cli.metadata.size, + "cli.metadata", + ); + const inspection = inspectManagedArtifactBytes( + image, + managedPeFixtureTarget(image), + ); + + expect(metadataOffset + layout.cli.metadata.size).toBeLessThanOrEqual( + image.length, + ); + expect(inspection.classification.status).toBe("managed"); + expect(inspection.references.map(({ name }) => name)).toEqual(references); + expect(inspection.coverage).toMatchObject({ + state: "complete", + issues: [], + }); + }); +}); diff --git a/src/dotnet/ManagedPeReader.ts b/src/dotnet/ManagedPeReader.ts index 9a30c007be..db5ce01d9a 100644 --- a/src/dotnet/ManagedPeReader.ts +++ b/src/dotnet/ManagedPeReader.ts @@ -45,6 +45,13 @@ export const managedCli32BitRequirement = ( return pair === 0x0002_0002 ? "preferred" : "none"; }; +/** + * crossgen2 clears ILONLY on ReadyToRun output, so only a clear ILONLY flag + * without the ReadyToRun signature marks mixed CLR/native code. + */ +export const managedCliIsMixedClrNative = (cli: ManagedCliHeader): boolean => + (cli.flags & 0x0000_0001) === 0 && !cli.readyToRunSignature; + export interface ManagedPeLayout { readonly machine: number; readonly architecture: "x86" | "x86_64" | "arm" | "arm64"; diff --git a/src/dotnet/ManagedSignatureResolution.test.ts b/src/dotnet/ManagedSignatureResolution.test.ts new file mode 100644 index 0000000000..4154b4fa90 --- /dev/null +++ b/src/dotnet/ManagedSignatureResolution.test.ts @@ -0,0 +1,332 @@ +import { createHash } from "node:crypto"; + +import { describe, expect, it, vi } from "vitest"; + +import { + attachManagedDecodeBudget, + ManagedDecodeBudget, +} from "./ManagedDecodeBudget.js"; +import { inspectManagedMembersBytes } from "./ManagedMemberInspector.js"; +import { + parseTypes, + signature, + typeRanges, +} from "./ManagedMemberInspectorCore.js"; +import { + metadataRowOffset, + readManagedMetadataLayout, +} from "./ManagedMetadataLayout.js"; +import { + buildManagedPeFixture, + managedPeFixtureTarget, +} from "./ManagedPe.fixture.js"; +import { readManagedPeLayout } from "./ManagedPeReader.js"; +import { managedTypeNameDigests } from "./ManagedTypeNameDigests.js"; +import { managedTypeReferences } from "./ManagedTypeReferences.js"; +import { readTypeMemberListFacts } from "./ManagedTypeMemberLists.js"; + +// A top-level type's name digest, as ManagedTypeNameDigests forms it. +const nameDigest = (name: string): Buffer => { + const dot = name.lastIndexOf("."); + return createHash("sha256") + .update(Buffer.from([0])) + .update(dot === -1 ? "" : name.slice(0, dot)) + .update(Buffer.from([0])) + .update(name.slice(dot + 1)) + .digest(); +}; + +const metadataLayout = (bytes: Buffer) => { + const pe = readManagedPeLayout(bytes); + if (pe.cli === null) throw new Error("Fixture has no CLI metadata"); + return readManagedMetadataLayout( + bytes, + pe.rvaToOffset(pe.cli.metadata.rva, pe.cli.metadata.size, "cli.metadata"), + pe.cli.metadata.size, + ); +}; + +const named = + (names: Readonly>) => + (codedIndex: number): Buffer | null => { + const name = names[codedIndex]; + return name === undefined ? null : nameDigest(name); + }; + +describe("resolved member signature digests", () => { + it("names modifier, class, and generic type operands instead of their rows", () => { + // modreq(M) G

on the left and the same types at other rows on the right. + const left = signature( + Buffer.from([0x06, 0x1f, 0x05, 0x15, 0x12, 0x09, 0x01, 0x11, 0x0d]), + undefined, + named({ 0x05: "N.Mod", 0x09: "N.G`1", 0x0d: "N.P" }), + ); + const right = signature( + Buffer.from([0x06, 0x1f, 0x15, 0x15, 0x12, 0x19, 0x01, 0x11, 0x1d]), + undefined, + named({ 0x15: "N.Mod", 0x19: "N.G`1", 0x1d: "N.P" }), + ); + + expect(left.raw_sha256).not.toBe(right.raw_sha256); + expect(left.resolved_sha256).toMatch(/^[0-9a-f]{64}$/u); + expect(left.resolved_sha256).toBe(right.resolved_sha256); + }); + + it("keeps the raw digest for a signature without type operands", () => { + const decoded = signature(Buffer.from([0x00, 0x01, 0x08, 0x0e])); + + expect(decoded.resolved_sha256).toBe(decoded.raw_sha256); + }); + + it("withholds the digest when an operand has no name", () => { + expect( + signature(Buffer.from([0x06, 0x12, 0x05]), undefined, named({})) + .resolved_sha256, + ).toBeNull(); + }); + + it("leaves a blob forged from a resolved preimage undecoded", () => { + // The resolved preimage of class "A": 0xff, then the name digest. 0xff + // never starts a compressed coded index, so the forged blob does not + // decode and has only a raw digest, equal to the resolved one; comparison + // tags the two kinds apart. + const forged = signature( + Buffer.concat([Buffer.from([0x06, 0x12, 0xff]), nameDigest("A")]), + ); + const resolved = signature( + Buffer.from([0x06, 0x12, 0x05]), + undefined, + named({ 0x05: "A" }), + ); + + expect(forged.parse_status).toBe("malformed"); + expect(forged.resolved_sha256).toBeNull(); + expect(forged.raw_sha256).toBe(resolved.resolved_sha256); + }); + + it.each([ + ["a TypeSpec, whose blob member inspection does not decode", (1 << 2) | 2], + ["a TypeRef row outside its table", (9 << 2) | 1], + ])("does not name %s", (_name, codedIndex) => { + const bytes = buildManagedPeFixture({ + fieldSignature: Buffer.from([0x06, 0x12, codedIndex]), + }); + const field = inspectManagedMembersBytes( + bytes, + managedPeFixtureTarget(bytes), + ).fields[0]; + + expect(field?.signature.parse_status).toBe("decoded"); + expect(field?.signature.resolved_sha256).toBeNull(); + }); + + it("names each type operand once, so repeated operands are not charged again", () => { + const bytes = buildManagedPeFixture({ + typeReferences: [{ namespace: "System", name: "Uri" }], + }); + const layout = metadataLayout(bytes); + const budget = new ManagedDecodeBudget(); + attachManagedDecodeBudget(layout, budget); + const typeName = managedTypeNameDigests( + layout, + [], + { + types: [], + nesting: { enclosing: new Map(), complete: true }, + }, + managedTypeReferences(bytes, layout, []), + ); + const uri = (2 << 2) | 1; + + expect(typeName(uri)).toEqual(nameDigest("System.Uri")); + const reserve = vi.spyOn(budget, "reserve"); + expect(typeName(uri)).toEqual(nameDigest("System.Uri")); + expect(reserve).not.toHaveBeenCalled(); + }); + + it("bounds name loading before resolving a deep real TypeRef scope chain", () => { + const name = "X".repeat(2048); + const rows = 10_000; + const bytes = buildManagedPeFixture({ + fieldSignature: Buffer.from([0x06, 0x12, (2 << 2) | 1]), + typeReferences: Array.from({ length: rows }, (_, index) => ({ + namespace: "", + name, + ...(index + 1 === rows ? {} : { enclosingRow: index + 3 }), + })), + }); + const layout = metadataLayout(bytes); + const first = + metadataRowOffset(layout, 1, 2) + + layout.codedIndexSize("ResolutionScope"); + const sharedName = + layout.stringIndexSize === 4 + ? bytes.readUInt32LE(first) + : bytes.readUInt16LE(first); + for (let row = 3; row <= rows + 1; row += 1) { + const offset = + metadataRowOffset(layout, 1, row) + + layout.codedIndexSize("ResolutionScope"); + if (layout.stringIndexSize === 4) bytes.writeUInt32LE(sharedName, offset); + else bytes.writeUInt16LE(sharedName, offset); + } + const issues: Parameters[2] = []; + const parsedTypes = parseTypes( + bytes, + layout, + typeRanges(bytes, layout, issues, readTypeMemberListFacts(bytes, layout)), + ); + const resolve = managedTypeNameDigests( + layout, + issues, + parsedTypes, + managedTypeReferences(bytes, layout, issues), + ); + expect(() => resolve((2 << 2) | 1)).not.toThrow(); + expect(resolve((2 << 2) | 1)).toBeNull(); + expect(issues).toEqual([]); + const inspection = inspectManagedMembersBytes( + bytes, + managedPeFixtureTarget(bytes), + ); + expect(inspection.coverage).toEqual({ state: "complete", issues: [] }); + expect(inspection.fields[0]).toBeDefined(); + expect(inspection.fields[0]?.signature.resolved_sha256).toBeNull(); + }); +}); + +const fieldOf = (options: Parameters[0]) => { + const bytes = buildManagedPeFixture(options); + return inspectManagedMembersBytes(bytes, managedPeFixtureTarget(bytes)); +}; + +describe("type operand names", () => { + it("keeps a TypeRef's namespace and name apart", () => { + const resolved = (namespace: string, name: string) => + fieldOf({ + typeReferences: [{ namespace, name }], + fieldSignature: Buffer.from([0x06, 0x12, (2 << 2) | 1]), + }).fields[0]?.signature.resolved_sha256; + + expect(resolved("A", "B.C")).toMatch(/^[0-9a-f]{64}$/u); + expect(resolved("A", "B.C")).not.toBe(resolved("A.B", "C")); + }); + + it("keeps a TypeDef's namespace and name apart", () => { + const resolved = (typeNamespace: string, typeName: string) => + fieldOf({ + typeNamespace, + typeName, + fieldSignature: Buffer.from([0x06, 0x12, 1 << 2]), + }).fields[0]?.signature.resolved_sha256; + + expect(resolved("A", "B.C")).toMatch(/^[0-9a-f]{64}$/u); + expect(resolved("A", "B.C")).not.toBe(resolved("A.B", "C")); + }); + + it("names a TypeDef and a TypeRef of one namespace and name alike", () => { + const definition = fieldOf({ + fieldSignature: Buffer.from([0x06, 0x12, 1 << 2]), + }).fields[0]?.signature.resolved_sha256; + const reference = fieldOf({ + typeReferences: [{ namespace: "Fixture", name: "Program" }], + fieldSignature: Buffer.from([0x06, 0x12, (2 << 2) | 1]), + }).fields[0]?.signature.resolved_sha256; + + expect(definition).toMatch(/^[0-9a-f]{64}$/u); + expect(definition).toBe(reference); + }); + + it("reports an unreadable TypeRef operand row once in coverage", () => { + const uri = (2 << 2) | 1; + const bytes = buildManagedPeFixture({ + typeReferences: [{ namespace: "System", name: "Uri" }], + fieldSignature: Buffer.from([0x06, 0x12, uri]), + methodSignature: Buffer.from([0x00, 0x00, 0x12, uri]), + }); + const layout = metadataLayout(bytes); + // An out-of-range name index makes TypeRef row 2 unreadable. + bytes.writeUInt16LE( + 0xffff, + metadataRowOffset(layout, 1, 2) + + layout.codedIndexSize("ResolutionScope"), + ); + const inspection = inspectManagedMembersBytes( + bytes, + managedPeFixtureTarget(bytes), + ); + + expect(inspection.fields[0]?.signature.resolved_sha256).toBeNull(); + expect(inspection.coverage.state).toBe("partial"); + expect(inspection.coverage.issues).toEqual([ + expect.objectContaining({ + code: "invalid-heap-index", + scope: "metadata.#Strings", + }), + ]); + }); + + it("retains an operand outside its type table as incomplete coverage", () => { + const inspection = fieldOf({ + fieldSignature: Buffer.from([0x06, 0x12, (9 << 2) | 1]), + }); + + expect(inspection.fields[0]?.signature.resolved_sha256).toBeNull(); + expect(inspection.coverage.state).toBe("partial"); + expect(inspection.coverage.issues).toEqual([ + expect.objectContaining({ code: "invalid-row" }), + ]); + }); + + it("preserves readable TypeRef names while withholding an invalid scope's signature identity", () => { + const inspection = fieldOf({ + typeRefScopeRaw: (99 << 2) | 2, + fieldSignature: Buffer.from([0x06, 0x12, (1 << 2) | 1]), + memberRefParentRaw: (1 << 3) | 1, + }); + expect(inspection.fields[0]?.signature.resolved_sha256).toBeNull(); + expect(inspection.member_refs[0]?.parent_name).toBe( + "System.Runtime.Versioning.TargetFrameworkAttribute", + ); + expect(inspection.coverage.state).toBe("partial"); + expect(inspection.coverage.issues).toEqual([ + expect.objectContaining({ + code: "invalid-row", + scope: "metadata.TypeRef:0x01000001", + }), + ]); + }); + + it.each([ + { + rows: [ + [1, 2], + [1, 2], + ], + }, + { + rows: [ + [1, 2], + [1, 3], + ], + }, + { rows: [[1, 1]] }, + { rows: [[1, 0]] }, + { rows: [[0, 2]] }, + { rows: [[1, 4]] }, + ] as const)( + "withholds TypeDef identity for invalid nesting $rows", + ({ rows }) => { + const inspection = fieldOf({ + additionalEmptyTypes: 2, + nestedTypes: rows, + fieldSignature: Buffer.from([0x06, 0x12, 1 << 2]), + }); + expect(inspection.fields[0]?.signature.resolved_sha256).toBeNull(); + expect(inspection.coverage.state).toBe("partial"); + expect(inspection.coverage.issues).toEqual([ + expect.objectContaining({ code: "invalid-row" }), + ]); + }, + ); +}); diff --git a/src/dotnet/ManagedTypeMemberLists.ts b/src/dotnet/ManagedTypeMemberLists.ts new file mode 100644 index 0000000000..1db5674c64 --- /dev/null +++ b/src/dotnet/ManagedTypeMemberLists.ts @@ -0,0 +1,147 @@ +import type { ManagedParseIssue } from "../domain/managed/managedArtifact.js"; +import { managedDecodeBudget } from "./ManagedDecodeBudget.js"; +import type { ManagedMetadataLayout } from "./ManagedMetadataLayout.js"; +import { metadataRowCursor, metadataToken } from "./ManagedMetadataHeaps.js"; +import { + memberListIndexSize, + memberListTable, + validatedMemberListStart, +} from "./ManagedMemberPointers.js"; + +export type MemberDefinitionTable = 4 | 6; + +export interface TypeMemberListStart { + readonly raw: number; + readonly value: number | null; + readonly offset: number; +} + +export interface TypeMemberListRange { + readonly start: TypeMemberListStart; + /** Next TypeDef's validated start, or the one-past-end sentinel. */ + readonly end: number | null; +} + +export interface TypeMemberListEntry { + readonly field: TypeMemberListRange; + readonly method: TypeMemberListRange; +} + +/** Parsed TypeDef list coordinates for one metadata-layout lifetime. */ +export interface TypeMemberListFacts { + readonly entries: readonly TypeMemberListEntry[]; + readonly issues: readonly ManagedParseIssue[]; +} + +export const readTypeMemberListFacts = ( + bytes: Buffer, + layout: ManagedMetadataLayout, +): TypeMemberListFacts => { + const fields = memberListTable(layout, 4); + const methods = memberListTable(layout, 6); + const rows: { field: TypeMemberListStart; method: TypeMemberListStart }[] = + []; + const issues: ManagedParseIssue[] = []; + const types = layout.table(2); + const typeCount = types?.rowCount ?? 0; + // Peak retained shape includes both the parsed-start and range arrays, their + // nested records, plus each eventual JSON object/key projection. Malformed + // row diagnostics are admitted separately before they are allocated. + managedDecodeBudget(layout).reserve( + typeCount * 512 + 256, + types?.offset ?? null, + ); + for (let row = 1; row <= typeCount; row += 1) { + const cursor = metadataRowCursor(bytes, layout, 2, row); + cursor.readUInt32(); + cursor.readIndex(layout.stringIndexSize); + cursor.readIndex(layout.stringIndexSize); + cursor.readIndex(layout.codedIndexSize("TypeDefOrRef")); + const field = { + offset: cursor.offset, + raw: cursor.readIndex(memberListIndexSize(layout, 4)), + }; + const method = { + offset: cursor.offset, + raw: cursor.readIndex(memberListIndexSize(layout, 6)), + }; + const entry = { + field: { ...field, value: validatedMemberListStart(field.raw, fields) }, + method: { + ...method, + value: validatedMemberListStart(method.raw, methods), + }, + }; + rows.push(entry); + for (const [name, start, table] of [ + ["FieldList", entry.field, fields], + ["MethodList", entry.method, methods], + ] as const) + if (start.value === null) { + managedDecodeBudget(layout).reserve(512, start.offset); + issues.push({ + code: "invalid-row", + scope: `metadata.TypeDef:${metadataToken(2, row)}`, + offset: start.offset, + detail: `TypeDef ${name} start ${String(start.raw)} is outside the one-based list range 1..${String((table?.rowCount ?? 0) + 1)}`, + }); + } + } + const nonmonotonicFields = new Set(); + const nonmonotonicMethods = new Set(); + for (let index = 0; index + 1 < rows.length; index += 1) { + const current = rows[index]; + const next = rows[index + 1]; + if (current === undefined || next === undefined) continue; + for (const [name, before, after, affected] of [ + ["FieldList", current.field, next.field, nonmonotonicFields], + ["MethodList", current.method, next.method, nonmonotonicMethods], + ] as const) + if ( + before.value !== null && + after.value !== null && + after.value < before.value + ) { + managedDecodeBudget(layout).reserve(512, after.offset); + affected.add(index); + affected.add(index + 1); + issues.push({ + code: "invalid-row", + scope: `metadata.TypeDef:${metadataToken(2, index + 2)}`, + offset: after.offset, + detail: `TypeDef ${name} start ${String(after.raw)} decreases from preceding start ${String(before.raw)}; adjacent member ranges are unknown`, + }); + } + } + const entries: TypeMemberListEntry[] = rows.map((entry, index) => { + const next = rows[index + 1]; + return { + field: { + start: { + ...entry.field, + value: nonmonotonicFields.has(index) ? null : entry.field.value, + }, + end: + next === undefined + ? (fields?.rowCount ?? 0) + 1 + : nonmonotonicFields.has(index) || nonmonotonicFields.has(index + 1) + ? null + : next.field.value, + }, + method: { + start: { + ...entry.method, + value: nonmonotonicMethods.has(index) ? null : entry.method.value, + }, + end: + next === undefined + ? (methods?.rowCount ?? 0) + 1 + : nonmonotonicMethods.has(index) || + nonmonotonicMethods.has(index + 1) + ? null + : next.method.value, + }, + }; + }); + return { entries, issues }; +}; diff --git a/src/dotnet/ManagedTypeNameDigests.test.ts b/src/dotnet/ManagedTypeNameDigests.test.ts new file mode 100644 index 0000000000..f309a722c3 --- /dev/null +++ b/src/dotnet/ManagedTypeNameDigests.test.ts @@ -0,0 +1,113 @@ +import { describe, expect, it } from "vitest"; + +import { + MAX_TYPE_NESTING_DEPTH, + nestedTypeDigests, +} from "./ManagedTypeNameDigests.js"; + +/** Rows 1..length, each nested in the row before it. */ +const chain = (length: number) => { + const calls: number[] = []; + const link = (row: number) => { + calls.push(row); + return row >= 1 && row <= length + ? { + namespace: "", + name: `T${String(row)}`, + enclosingRow: row > 1 ? row - 1 : null, + } + : null; + }; + return { calls, digest: nestedTypeDigests(link, () => undefined) }; +}; + +describe("nested signature type digests", () => { + it("reads each row of a chain within the nesting cap once", () => { + const { calls, digest } = chain(MAX_TYPE_NESTING_DEPTH); + for (let row = MAX_TYPE_NESTING_DEPTH; row >= 1; row -= 1) + expect(digest(row)).toBeInstanceOf(Buffer); + + expect(calls).toHaveLength(MAX_TYPE_NESTING_DEPTH); + }); + + it("does not name a type nested deeper than the cap, and stops at the cap", () => { + const { calls, digest } = chain(4_000); + + expect(digest(4_000)).toBeNull(); + expect(calls.length).toBeLessThanOrEqual(MAX_TYPE_NESTING_DEPTH + 1); + expect(digest(MAX_TYPE_NESTING_DEPTH)).toBeInstanceOf(Buffer); + expect(digest(MAX_TYPE_NESTING_DEPTH + 1)).toBeNull(); + }); + + it("reads shared parents once when many types are nested past the cap", () => { + // Rows 1..64 form a chain; rows 1001..2000 are each nested in row 64, one + // level past the cap. Naming the deepest types first must not reread the + // chain for each of them. + const calls: number[] = []; + const children = 1_000; + const link = (row: number) => { + calls.push(row); + if (row >= 1 && row <= MAX_TYPE_NESTING_DEPTH) + return { + namespace: "", + name: `T${String(row)}`, + enclosingRow: row > 1 ? row - 1 : null, + }; + return row > 1_000 && row <= 1_000 + children + ? { + namespace: "", + name: `C${String(row)}`, + enclosingRow: MAX_TYPE_NESTING_DEPTH, + } + : null; + }; + const digest = nestedTypeDigests(link, () => undefined); + for (let row = 1_001; row <= 1_000 + children; row += 1) + expect(digest(row)).toBeNull(); + + // No row is read twice; the walk can stop at the cap before the outermost. + expect(new Set(calls).size).toBe(calls.length); + expect(calls.length).toBeLessThanOrEqual(MAX_TYPE_NESTING_DEPTH + children); + }); + + it("names every type of a long chain in linear time", () => { + const { calls, digest } = chain(4_000); + for (let row = 1; row <= 4_000; row += 1) digest(row); + + // Rows past the cap may each walk at most the cap; none rewalks below it. + expect(calls.length).toBeLessThanOrEqual( + 4_000 * (MAX_TYPE_NESTING_DEPTH + 1), + ); + expect(calls.filter((row) => row === 1)).toHaveLength(1); + }); + + it("does not name a type whose nesting is cyclic", () => { + const digest = nestedTypeDigests( + (row) => ({ + namespace: "", + name: `T${String(row)}`, + enclosingRow: row === 1 ? 2 : 1, + }), + () => undefined, + ); + + expect(digest(1)).toBeNull(); + expect(digest(2)).toBeNull(); + }); + + it("tells a nested type from a top-level type with its joined name", () => { + const nested = nestedTypeDigests( + (row) => + row === 1 + ? { namespace: "", name: "Outer", enclosingRow: null } + : { namespace: "", name: "Inner", enclosingRow: 1 }, + () => undefined, + ); + const flat = nestedTypeDigests( + () => ({ namespace: "", name: "Outer/Inner", enclosingRow: null }), + () => undefined, + ); + + expect(nested(2)).not.toEqual(flat(1)); + }); +}); diff --git a/src/dotnet/ManagedTypeNameDigests.ts b/src/dotnet/ManagedTypeNameDigests.ts new file mode 100644 index 0000000000..dff47b5f4d --- /dev/null +++ b/src/dotnet/ManagedTypeNameDigests.ts @@ -0,0 +1,211 @@ +import { createHash } from "node:crypto"; + +import { managedDecodeBudget } from "./ManagedDecodeBudget.js"; +import type { ManagedParseIssue } from "../domain/managed/managedArtifact.js"; +import type { + parseTypes, + SignatureTypeName, +} from "./ManagedMemberInspectorCore.js"; +import { metadataToken } from "./ManagedMetadataHeaps.js"; +import type { ManagedTypeReferenceLookup } from "./ManagedTypeReferences.js"; +import type { ManagedMetadataLayout } from "./ManagedMetadataLayout.js"; +import { managedFailure, readManagedValue } from "./ManagedReaderFailure.js"; + +const TYPE_REF = 1; +const TYPE_DEF = 2; +/** One type's own namespace and name, and its enclosing type's row if nested. */ +interface TypeLink { + readonly namespace: string; + readonly name: string; + readonly enclosingRow: number | null; +} + +/** + * Deepest type nesting REA names, counting the outermost type as 1. Real + * assemblies nest a few levels; the cap bounds the walk a hostile chain of + * NestedClass rows or TypeRef ResolutionScopes can force per operand. + */ +export const MAX_TYPE_NESTING_DEPTH = 64; + +/** Digest prefixes that keep a top-level type apart from a nested one. */ +const TOP_LEVEL = Buffer.from([0]); +const NESTED = Buffer.from([1]); +/** + * Separates a namespace from a name. #Strings entries end at NUL, so neither + * holds one, and `A` + `B.C` stays apart from `A.B` + `C`. + */ +const NAME_SEPARATOR = Buffer.from([0]); + +interface NamedRow { + readonly digest: Buffer; + readonly depth: number; +} + +/** + * Digest each row's type name after its enclosing types' names: a top-level + * type hashes its namespace and name, each framed, and a nested type hashes + * its enclosing type's digest and its own namespace and name, as + * `Outer/Inner` would read. Each + * row is read and digested once, so naming a chain is linear in its length. + * A gap, a cycle, or nesting deeper than `MAX_TYPE_NESTING_DEPTH` has no + * digest. `retain` is charged once per row read and once per remembered row. + */ +export const nestedTypeDigests = ( + link: (row: number) => TypeLink | null, + retain: () => void, +): ((row: number) => Buffer | null) => { + const named = new Map(); + const remember = (row: number, value: NamedRow | null): void => { + retain(); + named.set(row, value); + }; + // Rows a walk read but could not name, such as the parents of a type nested + // past the cap, are still read only once. + const links = new Map(); + const readLink = (row: number): TypeLink | null => { + const known = links.get(row); + if (known !== undefined) return known; + const read = link(row); + retain(); + links.set(row, read); + return read; + }; + return (row) => { + // Walk out to a named row or the outermost type, at most past the cap. + const path: { readonly row: number; readonly link: TypeLink }[] = []; + const seen = new Set(); + let outer: NamedRow | null = null; + for (let current: number | null = row; current !== null;) { + const known = named.get(current); + if (known !== undefined) { + if (known === null) { + for (const step of path) remember(step.row, null); + return null; + } + outer = known; + break; + } + if (seen.has(current)) { + // Every row on a cycle, and every row nested in one, has no name. + for (const step of path) remember(step.row, null); + return null; + } + if (path.length === MAX_TYPE_NESTING_DEPTH) { + // The row the walk started from is nested past the cap. + remember(row, null); + return null; + } + const step = readLink(current); + if (step === null) { + remember(current, null); + for (const visited of path) remember(visited.row, null); + return null; + } + seen.add(current); + path.push({ row: current, link: step }); + current = step.enclosingRow; + } + // Name inward from the outermost row; a row past the cap, and every row + // nested in it, has no name. + let inner: NamedRow | null = outer; + let exceeded = false; + for (const step of path.reverse()) { + const depth = (inner?.depth ?? 0) + 1; + exceeded ||= depth > MAX_TYPE_NESTING_DEPTH; + if (exceeded) { + remember(step.row, null); + continue; + } + const hash = createHash("sha256"); + if (inner === null) hash.update(TOP_LEVEL); + else hash.update(NESTED).update(inner.digest); + hash + .update(step.link.namespace) + .update(NAME_SEPARATOR) + .update(step.link.name); + inner = { digest: hash.digest(), depth }; + remember(step.row, inner); + } + return exceeded ? null : (inner?.digest ?? null); + }; +}; + +/** + * Name signature type operands independently of their metadata rows: a TypeDef + * or TypeRef by namespace and name, after its enclosing types for a nested + * type, as in `Outer/Inner`. A TypeDef and a TypeRef with the same name are + * the same nominal name, and a TypeRef's resolution scope assembly is not part of its + * name, so a type forwarded between assemblies keeps its name. A TypeSpec, an + * unreadable row, cyclic nesting, or nesting deeper than the cap has no name, + * so its declaration and signature operands have no resolved digest; an unreadable row's failure is + * retained in `issues`, once per row. Each coded index and each row is + * named and digested once, so repeated operands cost a lookup. + */ +export const managedTypeNameDigests = ( + layout: ManagedMetadataLayout, + issues: ManagedParseIssue[], + parsedTypes: Pick, "types" | "nesting">, + reference: ManagedTypeReferenceLookup, +): SignatureTypeName => { + const retain = () => managedDecodeBudget(layout).reserve(128, null); + managedDecodeBudget(layout).reserve(parsedTypes.types.length * 128, null); + const types = new Map(); + for (const type of parsedTypes.types) types.set(type.token, type); + const definitions = nestedTypeDigests((row) => { + if (!parsedTypes.nesting.complete) return null; + const type = types.get(metadataToken(TYPE_DEF, row)); + if ( + type === undefined && + (row < 1 || row > (layout.table(TYPE_DEF)?.rowCount ?? 0)) + ) { + readManagedValue(() => { + throw managedFailure( + "invalid-row", + `metadata.signature.type:${TYPE_DEF}:${row}`, + `Signature references an unavailable TypeDef row ${row}`, + ); + }, issues); + return null; + } + if (type === undefined) return null; + const enclosingRow = parsedTypes.nesting.enclosing.get(row) ?? null; + // ECMA-335 TypeAttributes.VisibilityMask: 0/1 are top-level, 2..7 nested. + const nested = (type.flags & 0x7) >= 2; + if (nested !== (enclosingRow !== null)) { + managedDecodeBudget(layout).reserve(512, type.row_offset); + readManagedValue(() => { + throw managedFailure( + "invalid-row", + `metadata.TypeDef:${type.token}`, + nested + ? `Nested TypeDef ${type.token} has no NestedClass enclosing entry` + : `Top-level TypeDef ${type.token} has a NestedClass enclosing entry`, + type.row_offset, + ); + }, issues); + return null; + } + return { + namespace: type.namespace, + name: type.name, + enclosingRow, + }; + }, retain); + const references = nestedTypeDigests((row) => { + const value = reference(row); + return value?.scopeValid === true ? value : null; + }, retain); + const digests = new Map(); + return (codedIndex) => { + let digest = digests.get(codedIndex); + if (digest === undefined) { + managedDecodeBudget(layout).reserve(256, null); + const row = codedIndex >>> 2; + const table = codedIndex & 3; + digest = + table === 0 ? definitions(row) : table === 1 ? references(row) : null; + digests.set(codedIndex, digest); + } + return digest; + }; +}; diff --git a/src/dotnet/ManagedTypeNames.ts b/src/dotnet/ManagedTypeNames.ts index ab3b34c8f0..ef4a964725 100644 --- a/src/dotnet/ManagedTypeNames.ts +++ b/src/dotnet/ManagedTypeNames.ts @@ -1,4 +1,4 @@ -import { managedDecodeBudget } from "./ManagedDecodeBudget.js"; +import type { ManagedTypeReferenceLookup } from "./ManagedTypeReferences.js"; import type { ManagedParseIssue } from "../domain/managed/managedArtifact.js"; import { fullName, @@ -11,11 +11,9 @@ import { type ManagedMetadataLayout, } from "./ManagedMetadataLayout.js"; import { - metadataCodedTokenInvalidReason, metadataRowCursor, metadataToken, readMetadataBlob, - readMetadataString, } from "./ManagedMetadataHeaps.js"; import { managedFailure, readManagedValue } from "./ManagedReaderFailure.js"; @@ -31,62 +29,18 @@ export const typeNameLookup = ( layout: ManagedMetadataLayout, ranges: readonly TypeRange[], issues: ManagedParseIssue[], + references: ManagedTypeReferenceLookup, ): ((token: string | null) => string | null) => { const names = new Map(); - const checkedScopes = new Set(); - const validateScopeChain = (row: number): void => { - const path = new Set(); - let current = row; - while (!checkedScopes.has(current)) { - const cursor = metadataRowCursor(bytes, layout, 1, current); - if (path.has(current)) { - issues.push({ - code: "invalid-row", - scope: `metadata.TypeRef:${metadataToken(1, current)}`, - offset: cursor.start, - detail: "TypeRef ResolutionScope chain contains a cycle", - }); - break; - } - // Admit the chain's visited entry and retained validation-cache entry. - managedDecodeBudget(layout).reserve(128, cursor.start); - path.add(current); - const raw = cursor.readIndex(layout.codedIndexSize("ResolutionScope")); - const reason = metadataCodedTokenInvalidReason( - raw, - 2, - [0, 26, 35, 1], - layout.rowCounts, - ); - if (reason !== null) { - issues.push({ - code: "invalid-row", - scope: `metadata.TypeRef:${metadataToken(1, current)}`, - offset: cursor.start, - detail: `TypeRef ResolutionScope coded index 0x${raw.toString(16)} is invalid: ${reason}`, - }); - break; - } - // Null scopes are legal; non-TypeRef scopes terminate local validation. - if (raw === 0 || (raw & 3) !== 3) break; - current = Math.floor(raw / 4); - } - for (const visited of path) checkedScopes.add(visited); - }; const definitionName = (table: 1 | 2, row: number): string | null => { if (table === 2) { metadataRowOffset(layout, 2, row); return ranges[row - 1]?.fullName ?? null; } - const cursor = metadataRowCursor(bytes, layout, 1, row); - cursor.readIndex(layout.codedIndexSize("ResolutionScope")); - readManagedValue(() => validateScopeChain(row), issues); - const name = cursor.readIndex(layout.stringIndexSize); - const namespace = cursor.readIndex(layout.stringIndexSize); - return fullName( - readMetadataString(bytes, layout, namespace, layout.strings.size), - readMetadataString(bytes, layout, name, layout.strings.size), - ); + const reference = references(row); + return reference === null + ? null + : fullName(reference.namespace, reference.name); }; const specName = (row: number): string | null => { const cursor = metadataRowCursor(bytes, layout, 27, row); diff --git a/src/dotnet/ManagedTypeNesting.ts b/src/dotnet/ManagedTypeNesting.ts new file mode 100644 index 0000000000..7dfbbdd4d1 --- /dev/null +++ b/src/dotnet/ManagedTypeNesting.ts @@ -0,0 +1,45 @@ +import { managedDecodeBudget } from "./ManagedDecodeBudget.js"; +import { metadataRowCursor, metadataToken } from "./ManagedMetadataHeaps.js"; +import type { ManagedMetadataLayout } from "./ManagedMetadataLayout.js"; +import { managedFailure, readManagedValue } from "./ManagedReaderFailure.js"; +import type { ManagedParseIssue } from "../domain/managed/managedArtifact.js"; + +/** Parsed NestedClass facts shared by declaring types and signature operands. */ +export interface ManagedTypeNesting { + readonly enclosing: ReadonlyMap; + readonly complete: boolean; +} + +export const readManagedTypeNesting = ( + bytes: Buffer, + layout: ManagedMetadataLayout, + issues: ManagedParseIssue[], +): ManagedTypeNesting => { + const initialIssues = issues.length; + const typeCount = layout.table(2)?.rowCount ?? 0; + const enclosing = new Map(); + for (let row = 1; row <= (layout.table(41)?.rowCount ?? 0); row += 1) { + readManagedValue(() => { + const cursor = metadataRowCursor(bytes, layout, 41, row); + const nested = cursor.readIndex(layout.tableIndexSize(2)); + const outer = cursor.readIndex(layout.tableIndexSize(2)); + if ( + nested < 1 || + outer < 1 || + nested > typeCount || + outer > typeCount || + nested === outer || + enclosing.has(nested) + ) + throw managedFailure( + "invalid-row", + `metadata.NestedClass:${metadataToken(41, row)}`, + `NestedClass requires a unique nested type and distinct existing enclosing type; observed ${nested} -> ${outer}`, + cursor.start, + ); + managedDecodeBudget(layout).reserve(128, cursor.start); + enclosing.set(nested, outer); + }, issues); + } + return { enclosing, complete: issues.length === initialIssues }; +}; diff --git a/src/dotnet/ManagedTypeReferences.ts b/src/dotnet/ManagedTypeReferences.ts new file mode 100644 index 0000000000..6b436a410c --- /dev/null +++ b/src/dotnet/ManagedTypeReferences.ts @@ -0,0 +1,140 @@ +import { managedDecodeBudget } from "./ManagedDecodeBudget.js"; +import type { ManagedMetadataLayout } from "./ManagedMetadataLayout.js"; +import { + metadataCodedTokenInvalidReason, + metadataRowCursor, + metadataToken, + readMetadataString, +} from "./ManagedMetadataHeaps.js"; +import { readManagedValue } from "./ManagedReaderFailure.js"; +import { metadataRowOffset } from "./ManagedMetadataLayout.js"; +import type { ManagedParseIssue } from "../domain/managed/managedArtifact.js"; + +export interface ManagedTypeReference { + readonly namespace: string; + readonly name: string; + readonly enclosingRow: number | null; + readonly scopeValid: boolean; +} + +export type ManagedTypeReferenceLookup = ( + row: number, +) => ManagedTypeReference | null; + +interface TypeReferenceRow { + readonly nameIndex: number; + readonly namespaceIndex: number; + readonly enclosingRow: number | null; + readonly scopeValid: boolean; +} + +/** Own each TypeRef source tuple and scope validation once per inspection. */ +export const managedTypeReferences = ( + bytes: Buffer, + layout: ManagedMetadataLayout, + issues: ManagedParseIssue[], +): ManagedTypeReferenceLookup => { + const rows = new Map(); + const names = new Map(); + const checked = new Map(); + const read = (row: number): TypeReferenceRow | null => { + const cached = rows.get(row); + if (cached !== undefined) return cached; + // Admit the parsed row, validation entry and bounded walk bookkeeping. + managedDecodeBudget(layout).reserve(384, null); + const value = + readManagedValue(() => { + const cursor = metadataRowCursor(bytes, layout, 1, row); + const raw = cursor.readIndex(layout.codedIndexSize("ResolutionScope")); + const reason = metadataCodedTokenInvalidReason( + raw, + 2, + [0, 26, 35, 1], + layout.rowCounts, + ); + if (reason !== null) + issues.push({ + code: "invalid-row", + scope: `metadata.TypeRef:${metadataToken(1, row)}`, + offset: cursor.start, + detail: `TypeRef ResolutionScope coded index 0x${raw.toString(16)} is invalid: ${reason}`, + }); + return { + nameIndex: cursor.readIndex(layout.stringIndexSize), + namespaceIndex: cursor.readIndex(layout.stringIndexSize), + enclosingRow: + raw !== 0 && (raw & 3) === 3 ? Math.floor(raw / 4) : null, + scopeValid: reason === null, + }; + }, issues) ?? null; + rows.set(row, value); + return value; + }; + const validate = (row: number): boolean => { + const prior = checked.get(row); + if (prior !== undefined) return prior; + const path: number[] = []; + const seen = new Set(); + let current: number | null = row; + let valid = true; + while (current !== null) { + const known = checked.get(current); + if (known !== undefined) { + valid = known; + break; + } + if (seen.has(current)) { + issues.push({ + code: "invalid-row", + scope: `metadata.TypeRef:${metadataToken(1, current)}`, + offset: metadataRowOffset(layout, 1, current), + detail: "TypeRef ResolutionScope chain contains a cycle", + }); + valid = false; + break; + } + seen.add(current); + path.push(current); + const value = read(current); + if (value === null || !value.scopeValid) { + valid = false; + break; + } + current = value.enclosingRow; + } + for (const visited of path) checked.set(visited, valid); + return valid; + }; + return (row) => { + const cached = names.get(row); + if (cached !== undefined) return cached; + const value = read(row); + if (value === null) return null; + const scopeValid = validate(row); + // Scope validation loads only scalar row facts. The digest owner's nesting + // cap determines which names are requested, rather than preloading parents. + managedDecodeBudget(layout).reserve(128, null); + const named = + readManagedValue( + () => ({ + namespace: readMetadataString( + bytes, + layout, + value.namespaceIndex, + layout.strings.size, + ), + name: readMetadataString( + bytes, + layout, + value.nameIndex, + layout.strings.size, + ), + enclosingRow: value.enclosingRow, + scopeValid, + }), + issues, + ) ?? null; + names.set(row, named); + return named; + }; +}; diff --git a/src/electron/ElectronFusePath.ts b/src/electron/ElectronFusePath.ts new file mode 100644 index 0000000000..0836c961cb --- /dev/null +++ b/src/electron/ElectronFusePath.ts @@ -0,0 +1,353 @@ +import type { BigIntStats } from "node:fs"; +import { lstat, readlink } from "node:fs/promises"; +import { basename, dirname, isAbsolute, sep } from "node:path"; + +import { isPathWithinRoot } from "../domain/localPath.js"; + +const FRAMEWORK_NAME = "Electron Framework.framework"; +const BINARY_NAME = "Electron Framework"; +const MAX_SYMLINK_HOPS = 32; + +/** Filesystem facts owned by path admission and consumed unchanged by the reader. */ +export interface ElectronFusePathNode { + readonly path: string; + readonly metadata: BigIntStats; +} +export interface ResolvedElectronFuseFile { + readonly path: string; + readonly metadata: BigIntStats; + readonly nodes: readonly ElectronFusePathNode[]; +} +export interface ResolvedElectronFuseTarget { + readonly selectedPath: string; + readonly file: ResolvedElectronFuseFile; +} + +/** A selected path is not an admitted Electron bundle or regular file. */ +export class ElectronFusePathFailure extends Error { + constructor( + readonly reason: "invalid_value" | "invalid_format", + message: string, + ) { + super(message); + this.name = "ElectronFusePathFailure"; + } +} + +type WalkResult = + | ({ readonly kind: "file" } & ResolvedElectronFuseFile) + | { readonly kind: "absent" | "escapes" | "not-file" | "loop" }; + +/** + * Admit a regular executable, a macOS app bundle, or an Electron Framework + * binary reached through `Versions/Current` without leaving the framework. + */ +export const resolveElectronFuseTarget = async ( + selectedPath: string, + signal?: AbortSignal, +): Promise => { + signal?.throwIfAborted(); + const metadata = await lstat(selectedPath, { bigint: true }); + let file: ResolvedElectronFuseFile; + if (metadata.isSymbolicLink()) + file = await resolveFrameworkSymlink(selectedPath, signal); + else if (metadata.isFile()) + file = + frameworkRootOf(selectedPath) === null + ? { path: selectedPath, metadata, nodes: [] } + : await resolveFrameworkSymlink(selectedPath, signal); + else if (metadata.isDirectory()) { + file = await resolveBundleBinary(selectedPath, signal); + file = { + ...file, + nodes: [{ path: selectedPath, metadata }, ...file.nodes], + }; + } else + throw new ElectronFusePathFailure( + "invalid_format", + `Expected a regular file or Electron app bundle: ${selectedPath}`, + ); + return { selectedPath, file }; +}; + +const resolveFrameworkSymlink = async ( + selectedPath: string, + signal?: AbortSignal, +): Promise => { + const root = frameworkRootOf(selectedPath); + if (root === null) + throw new ElectronFusePathFailure( + "invalid_format", + `Expected a regular file without a symlink: ${selectedPath}`, + ); + const nodes = await requireFrameworkRoot(root, selectedPath); + const relativePath = relativeInside(root, selectedPath); + if (relativePath === null) + throw new ElectronFusePathFailure( + "invalid_value", + `Framework symlink escapes ${FRAMEWORK_NAME}: ${selectedPath}`, + ); + return requireFile( + await resolveInside(root, relativePath, signal), + selectedPath, + nodes, + ); +}; + +const resolveBundleBinary = async ( + selectedPath: string, + signal?: AbortSignal, +): Promise => { + const directory = await resolveDirectorySuffix(selectedPath, signal); + const name = basename(directory.path); + let file: ResolvedElectronFuseFile; + if (name === FRAMEWORK_NAME) + file = await resolveKnownBinary(directory.path, selectedPath, signal); + else if (name.endsWith(".app")) + file = await resolveKnownBinary( + appendPath(directory.path, "Contents", "Frameworks", FRAMEWORK_NAME), + selectedPath, + signal, + ); + else + throw new ElectronFusePathFailure( + "invalid_format", + `Expected an Electron .app bundle, ${FRAMEWORK_NAME} directory, or executable file: ${selectedPath}`, + ); + return { ...file, nodes: [...directory.nodes, ...file.nodes] }; +}; + +/** Reduce directory suffixes only when doing so preserves filesystem selection. */ +const resolveDirectorySuffix = async ( + selectedPath: string, + signal?: AbortSignal, +): Promise<{ + readonly path: string; + readonly nodes: readonly ElectronFusePathNode[]; +}> => { + let path = selectedPath; + const pending: string[] = []; + const nodes: ElectronFusePathNode[] = []; + while (true) { + signal?.throwIfAborted(); + const name = basename(path); + // lstat("link/") follows a link; inspect the named component itself. + if (name !== "") while (path.endsWith(sep)) path = path.slice(0, -1); + if (name === "." || name === "..") { + pending.push(name); + path = dirname(path); + continue; + } + const component = pending.pop(); + if (component === undefined) return { path, nodes }; + if (component === ".") continue; + const metadata = await lstat(path, { bigint: true }); + if (!metadata.isDirectory() || metadata.isSymbolicLink()) + throw new ElectronFusePathFailure( + "invalid_value", + `A bundle path ending in '..' requires a real directory before each parent segment: ${selectedPath}`, + ); + nodes.push({ path, metadata }); + path = dirname(path); + } +}; + +/** `root` is the framework directory for both an `.app` and a framework selection. */ +const resolveKnownBinary = async ( + root: string, + bundle: string, + signal: AbortSignal | undefined, +): Promise => { + const nodes = await requireFrameworkRoot(root, bundle); + for (const candidate of [ + BINARY_NAME, + `Versions/Current/${BINARY_NAME}`, + `Versions/A/${BINARY_NAME}`, + ]) { + const selected = await resolveInside(root, candidate, signal); + if (selected.kind === "file") return requireFile(selected, bundle, nodes); + if (selected.kind !== "absent") + throw pathWalkFailure(selected.kind, bundle); + } + throw new ElectronFusePathFailure( + "invalid_value", + `Electron Framework binary was not found inside ${bundle}.`, + ); +}; + +const resolveInside = async ( + root: string, + relativePath: string, + signal?: AbortSignal, +): Promise => { + const pending = relativePath.split("/").filter((segment) => segment !== ""); + const resolved: string[] = []; + const nodes: ElectronFusePathNode[] = []; + let fileMetadata: BigIntStats | undefined; + let hops = 0; + let last: "file" | "directory" | undefined; + while (pending.length > 0) { + signal?.throwIfAborted(); + const segment = pending.shift() ?? ""; + if (segment === ".") continue; + if (segment === "..") { + if (resolved.length === 0) return { kind: "escapes" }; + resolved.pop(); + last = "directory"; + continue; + } + const absolute = appendPath(root, ...resolved, segment); + if (!isPathWithinRoot(root, absolute)) return { kind: "escapes" }; + let metadata; + try { + metadata = await lstat(absolute, { bigint: true }); + } catch (cause: unknown) { + if (isMissing(cause)) return { kind: "absent" }; + throw cause; + } + if (metadata.isSymbolicLink()) { + nodes.push({ path: absolute, metadata }); + hops += 1; + if (hops > MAX_SYMLINK_HOPS) return { kind: "loop" }; + const target = await readlink(absolute); + if (isAbsolute(target)) return { kind: "escapes" }; + pending.unshift(...target.split("/").filter((part) => part !== "")); + continue; + } + if (metadata.isDirectory()) { + nodes.push({ path: absolute, metadata }); + resolved.push(segment); + last = "directory"; + continue; + } + if (!metadata.isFile() || pending.length > 0) return { kind: "not-file" }; + resolved.push(segment); + fileMetadata = metadata; + last = "file"; + } + if (last !== "file" || fileMetadata === undefined) + return { kind: "not-file" }; + const path = appendPath(root, ...resolved); + return isPathWithinRoot(root, path) + ? { kind: "file", path, metadata: fileMetadata, nodes } + : { kind: "escapes" }; +}; + +/** Validate the app-parent chain equally for app, framework, and binary selection. */ +const requireFrameworkRoot = async ( + root: string, + selectedPath: string, +): Promise => { + const frameworks = dirname(root); + const contents = dirname(frameworks); + const app = dirname(contents); + if ( + basename(frameworks) === "Frameworks" && + basename(contents) === "Contents" && + basename(app).endsWith(".app") + ) + return requireRealDirectories( + dirname(app), + [basename(app), "Contents", "Frameworks", FRAMEWORK_NAME], + selectedPath, + ); + else + return requireRealDirectories( + dirname(root), + [basename(root)], + selectedPath, + ); +}; + +/** + * Each named directory below `anchor` must be a real directory, so the + * framework root cannot redirect the walk outside the selected bundle. + */ +const requireRealDirectories = async ( + anchor: string, + segments: readonly string[], + selectedPath: string, +): Promise => { + let current = anchor; + const nodes: ElectronFusePathNode[] = []; + for (const segment of segments) { + current = appendPath(current, segment); + let metadata; + try { + metadata = await lstat(current, { bigint: true }); + } catch (cause: unknown) { + if (isMissing(cause)) + throw new ElectronFusePathFailure( + "invalid_value", + `Electron Framework binary was not found inside ${selectedPath}.`, + ); + throw cause; + } + nodes.push({ path: current, metadata }); + if (metadata.isSymbolicLink()) + throw pathWalkFailure("escapes", selectedPath); + if (!metadata.isDirectory()) + throw pathWalkFailure("not-file", selectedPath); + } + return nodes; +}; + +const requireFile = ( + walk: WalkResult, + selectedPath: string, + nodes: readonly ElectronFusePathNode[], +): ResolvedElectronFuseFile => { + if (walk.kind === "file") + return { + path: walk.path, + metadata: walk.metadata, + nodes: [...nodes, ...walk.nodes], + }; + if (walk.kind === "absent") + throw new ElectronFusePathFailure( + "invalid_value", + `Electron Framework symlink target was not found: ${selectedPath}`, + ); + throw pathWalkFailure(walk.kind, selectedPath); +}; + +const pathWalkFailure = ( + kind: "escapes" | "not-file" | "loop", + path: string, +): ElectronFusePathFailure => + new ElectronFusePathFailure( + "invalid_value", + kind === "escapes" + ? `Framework symlink escapes ${FRAMEWORK_NAME}: ${path}` + : kind === "loop" + ? `Framework symlink hop limit exceeded: ${path}` + : `Electron Framework path is not a regular file: ${path}`, + ); + +const frameworkRootOf = (path: string): string | null => { + let current = dirname(path); + while (basename(current) !== FRAMEWORK_NAME) { + const parent = dirname(current); + if (parent === current) return null; + current = parent; + } + return current; +}; + +const relativeInside = (root: string, path: string): string | null => { + // Keep dot segments until resolveInside has followed preceding symlinks. + // Lexical normalization would select a different file for link/../ paths. + const prefix = root.endsWith(sep) ? root : `${root}${sep}`; + if (!path.startsWith(prefix)) return null; + const value = path.slice(prefix.length); + return value === "" ? null : value.split(sep).join("/"); +}; + +/** Append known components without normalizing a caller-selected parent path. */ +const appendPath = (parent: string, ...segments: readonly string[]): string => + `${parent}${parent.endsWith(sep) ? "" : sep}${segments.join(sep)}`; + +const isMissing = (cause: unknown): boolean => + cause instanceof Error && + "code" in cause && + ["ENOENT", "ENOTDIR"].includes(String(cause.code)); diff --git a/src/electron/ElectronFuseProvider.test.ts b/src/electron/ElectronFuseProvider.test.ts new file mode 100644 index 0000000000..dd6227caaf --- /dev/null +++ b/src/electron/ElectronFuseProvider.test.ts @@ -0,0 +1,701 @@ +import { resolveElectronFuseTarget } from "./ElectronFusePath.js"; +import { RegularFileChangedError } from "../filesystem/RegularFile.js"; +import { mkdir, symlink, writeFile, truncate, rename } from "node:fs/promises"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; + +import { createTestTempDirectory } from "../../tests/fixtures/temporaryDirectory.js"; +import { + elfFuseImage, + fuseWire, + FUSE_SENTINEL, + OBSIDIAN_FUSE_BYTES, + peFuseImage, + sentinelFreeMacho, + thinFuseImage, + thinMacho, + truncatedFuseImage, + twoSentinelElf, + universalFuseImage, +} from "../../tests/fixtures/electron/fuseImage.js"; +import { ELECTRON_FUSE_STATE_BYTES } from "../domain/electron/electronFuses.js"; +import { ElectronFuseProvider } from "./ElectronFuseProvider.js"; +import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; +import { readElectronFuseImage } from "./ElectronFuseReader.js"; + +const provider = new ElectronFuseProvider(); + +const inspect = async (path: string) => { + const result = await provider.inspect({ path }); + if (!result.ok) throw result.error; + return result.value; +}; + +describe("Electron fuse wire reading", () => { + it("reads thin Mach-O, universal Mach-O, PE, and ELF wires", async () => { + const root = await createTestTempDirectory("rea-electron-fuses-"); + const thin = thinFuseImage(OBSIDIAN_FUSE_BYTES); + const universal = universalFuseImage(); + const pe = peFuseImage(OBSIDIAN_FUSE_BYTES); + const elf = elfFuseImage([ + ELECTRON_FUSE_STATE_BYTES.enabled, + ELECTRON_FUSE_STATE_BYTES.removed, + 0x99, + ]); + await writeFile(join(root, "thin"), thin.bytes); + await writeFile(join(root, "universal"), universal.bytes); + await writeFile(join(root, "app.exe"), pe.bytes); + await writeFile(join(root, "app.elf"), elf.bytes); + + const thinReport = await inspect(join(root, "thin")); + expect(thinReport.artifact).toMatchObject({ + format: "mach-o", + bytes: thin.bytes.length, + }); + expect(thinReport.slices).toMatchObject([ + { + architecture: "arm64", + sentinel_count: 1, + sentinels: [ + { + sentinel_offset: thin.sentinelOffset, + wire_version: 1, + declared_length: 8, + complete: true, + }, + ], + }, + ]); + expect( + thinReport.slices[0]?.sentinels[0]?.fuses.map((fuse) => fuse.state), + ).toEqual([ + "disabled", + "enabled", + "disabled", + "disabled", + "disabled", + "disabled", + "disabled", + "enabled", + ]); + + const universalReport = await inspect(join(root, "universal")); + expect(universalReport.artifact.format).toBe("mach-o-universal"); + expect(universalReport.slices).toMatchObject([ + { + file_offset: universal.left.fileOffset, + architecture: "x86_64", + sentinels: [{ sentinel_offset: universal.left.sentinelOffset }], + }, + { + file_offset: universal.right.fileOffset, + architecture: "arm64", + sentinels: [{ sentinel_offset: universal.right.sentinelOffset }], + }, + ]); + expect( + universalReport.runtime_implications.filter( + (item) => item.subject === "node_cli_inspect_arguments", + ), + ).toEqual([ + expect.objectContaining({ + slice_index: 0, + outcome: "not_honored", + sentinel_offset: universal.left.sentinelOffset, + }), + expect.objectContaining({ + slice_index: 1, + outcome: "honored", + sentinel_offset: universal.right.sentinelOffset, + }), + ]); + + const peReport = await inspect(join(root, "app.exe")); + expect(peReport.artifact.format).toBe("pe"); + expect(peReport.slices[0]?.sentinels[0]?.sentinel_offset).toBe( + pe.sentinelOffset, + ); + const elfReport = await inspect(join(root, "app.elf")); + expect(elfReport.slices[0]?.sentinels[0]?.fuses).toMatchObject([ + { name: "RunAsNode", state: "enabled", raw_byte: 0x31 }, + { name: "EnableCookieEncryption", state: "removed", raw_byte: 0x72 }, + { + name: "EnableNodeOptionsEnvironmentVariable", + state: "unknown", + raw_byte: 0x99, + }, + ]); + expect(elfReport.runtime_implications).toEqual([ + expect.objectContaining({ + subject: "electron_run_as_node", + outcome: "honored", + }), + ]); + }); + + it("keeps a truncated wire, duplicate sentinels, and a missing sentinel", async () => { + const root = await createTestTempDirectory("rea-electron-fuses-partial-"); + await writeFile(join(root, "truncated"), truncatedFuseImage()); + const duplicated = twoSentinelElf(); + await writeFile(join(root, "duplicated"), duplicated.bytes); + await writeFile(join(root, "empty"), sentinelFreeMacho()); + + const truncated = await inspect(join(root, "truncated")); + expect(truncated.slices[0]?.sentinels[0]).toMatchObject({ + wire_version: 1, + declared_length: 4, + complete: false, + fuses: [{ index: 0, state: "enabled", raw_byte: 0x31 }], + }); + expect(truncated.limitations.join(" ")).toContain("truncated"); + + const duplicates = await inspect(join(root, "duplicated")); + expect(duplicates.slices[0]?.sentinel_count).toBe(2); + expect( + duplicates.slices[0]?.sentinels.map((item) => item.sentinel_offset), + ).toEqual([...duplicated.offsets]); + expect(duplicates.limitations.join(" ")).toContain("none is selected"); + + const missing = await inspect(join(root, "empty")); + expect(missing.slices[0]?.sentinel_count).toBe(0); + expect(missing.runtime_implications).toEqual([]); + expect(missing.limitations.join(" ")).toContain("unknown, not disabled"); + }); +}); + +describe("ELF embedded ASAR integrity applicability", () => { + it("preserves an enabled ASAR integrity fuse without claiming enforcement on ELF", async () => { + const root = await createTestTempDirectory("rea-electron-fuses-elf-asar-"); + const fuses = [ + ELECTRON_FUSE_STATE_BYTES.disabled, + ELECTRON_FUSE_STATE_BYTES.disabled, + ELECTRON_FUSE_STATE_BYTES.disabled, + ELECTRON_FUSE_STATE_BYTES.disabled, + ELECTRON_FUSE_STATE_BYTES.enabled, + ]; + const image = elfFuseImage(fuses); + const path = join(root, "electron"); + await writeFile(path, image.bytes); + const report = await inspect(path); + expect(report.artifact.format).toBe("elf"); + expect(report.slices[0]?.sentinels[0]?.fuses[4]).toMatchObject({ + index: 4, + name: "EnableEmbeddedAsarIntegrityValidation", + state: "enabled", + raw_byte: ELECTRON_FUSE_STATE_BYTES.enabled, + }); + expect( + report.runtime_implications.find( + (item) => item.subject === "embedded_asar_integrity_validation", + ), + ).toMatchObject({ + relationship: "derivation", + fuse_index: 4, + outcome: "unknown", + sentinel_offset: image.sentinelOffset, + }); + expect(report.limitations.join(" ")).toContain("only on macOS and Windows"); + }); +}); + +describe("Electron fuse reader resource boundaries", () => { + it("rejects invalid scan window sizes before allocating or scanning", async () => { + const root = await createTestTempDirectory( + "rea-electron-fuses-window-limit-", + ); + const path = join(root, "window"); + await writeFile(path, thinFuseImage(OBSIDIAN_FUSE_BYTES).bytes); + for (const chunkBytes of [0, -1, 1.5, 256 * 1024 + 1]) + await expect(readElectronFuseImage(path, { chunkBytes })).rejects.toThrow( + /chunkBytes must be an integer between/, + ); + }); + + it("rejects overlapping universal Mach-O slices before repeated searching", async () => { + const root = await createTestTempDirectory("rea-electron-fuses-overlap-"); + const image = universalFuseImage().bytes; + const firstOffset = image.readUInt32BE(16); + const firstSize = image.readUInt32BE(20); + image.writeUInt32BE(0x01000007, 28); + image.writeUInt32BE(firstOffset, 36); + image.writeUInt32BE(firstSize, 40); + const path = join(root, "overlapping"); + await writeFile(path, image); + + const result = await provider.inspect({ path }); + expect(result.ok).toBe(false); + if (!result.ok) + expect(result.error).toMatchObject({ + _tag: "AnalysisInputError", + issues: [ + expect.objectContaining({ + path: ["path"], + reason: "invalid_format", + message: expect.stringContaining("slices 0 and 1 overlap"), + }), + ], + }); + }); +}); + +describe("Electron fuse path and container admission", () => { + it("finds a sentinel that crosses a read window", async () => { + const root = await createTestTempDirectory("rea-electron-fuses-window-"); + const image = thinFuseImage(OBSIDIAN_FUSE_BYTES); + const path = join(root, "window"); + await writeFile(path, image.bytes); + const read = await readElectronFuseImage(path, { chunkBytes: 20 }); + expect(read.slices[0]?.sentinels[0]?.sentinel_offset).toBe( + image.sentinelOffset, + ); + expect(read.slices[0]?.sentinels[0]?.fuses).toHaveLength(8); + }); + + it("resolves an app bundle and a framework symlink inside the bundle", async () => { + const root = await createTestTempDirectory("rea-electron-fuses-app-"); + const image = thinFuseImage(OBSIDIAN_FUSE_BYTES); + const app = join(root, "Fixture.app"); + const versions = join( + app, + "Contents", + "Frameworks", + "Electron Framework.framework", + "Versions", + ); + await mkdir(join(versions, "A"), { recursive: true }); + const binary = join(versions, "A", "Electron Framework"); + await writeFile(binary, image.bytes); + await symlink("A", join(versions, "Current")); + await symlink( + "Versions/Current/Electron Framework", + join(versions, "..", "Electron Framework"), + ); + + const fromApp = await inspect(app); + expect(fromApp.artifact.selected_path).toBe(app); + expect(fromApp.artifact.resolved_path).toBe(binary); + expect(fromApp.slices[0]?.sentinel_count).toBe(1); + + const link = join(versions, "..", "Electron Framework"); + const fromLink = await inspect(link); + expect(fromLink.artifact.resolved_path).toBe(binary); + expect(fromLink.artifact.sha256).toBe(fromApp.artifact.sha256); + }); + + it("rejects an app bundle whose Versions/Current symlink leaves the framework", async () => { + const root = await createTestTempDirectory( + "rea-electron-fuses-framework-root-", + ); + const app = join(root, "Fixture.app"); + const framework = join( + app, + "Contents", + "Frameworks", + "Electron Framework.framework", + ); + await mkdir(join(framework, "Versions"), { recursive: true }); + await mkdir(join(app, "Contents", "Other"), { recursive: true }); + await writeFile( + join(app, "Contents", "Other", "Electron Framework"), + elfFuseImage([ELECTRON_FUSE_STATE_BYTES.enabled]).bytes, + ); + await symlink("../../../Other", join(framework, "Versions", "Current")); + + for (const path of [app, framework]) { + const result = await provider.inspect({ path }); + expect(result.ok, path).toBe(false); + if (result.ok) continue; + expect(result.error).toMatchObject({ + _tag: "AnalysisInputError", + issues: [ + expect.objectContaining({ + path: ["path"], + message: expect.stringContaining("escapes"), + }), + ], + }); + } + }); + + it("refuses a framework symlink that leaves the bundle", async () => { + const root = await createTestTempDirectory("rea-electron-fuses-escape-"); + const versions = join( + root, + "Fixture.app", + "Contents", + "Frameworks", + "Electron Framework.framework", + "Versions", + ); + await mkdir(versions, { recursive: true }); + await symlink("/etc", join(versions, "Current")); + const result = await provider.inspect({ + path: join(root, "Fixture.app"), + }); + expect(result.ok).toBe(false); + if (result.ok) return; + expect(result.error).toMatchObject({ + _tag: "AnalysisInputError", + issues: [ + expect.objectContaining({ + path: ["path"], + message: expect.stringContaining("escapes"), + }), + ], + }); + }); + + it("rejects an unrecognized container and a big-endian Mach-O image", async () => { + const root = await createTestTempDirectory("rea-electron-fuses-reject-"); + const text = join(root, "notes.txt"); + const bigEndian = join(root, "big"); + const java = join(root, "Main.class"); + await writeFile(text, "This is not an executable container."); + const header = Buffer.alloc(32); + header.writeUInt32BE(0xfeedfacf, 0); + await writeFile(bigEndian, header); + await writeFile( + java, + Buffer.from([0xca, 0xfe, 0xba, 0xbe, 0x00, 0x00, 0x00, 0x37]), + ); + for (const path of [text, bigEndian, java]) { + const result = await provider.inspect({ path }); + expect(result.ok, path).toBe(false); + if (!result.ok) + expect(result.error._tag).toBe("AnalysisUnsupportedTargetError"); + } + }); + + it("does not invent a sentinel from bytes that only straddle two slices", async () => { + const root = await createTestTempDirectory("rea-electron-fuses-span-"); + const left = thinMacho(0x01000007, Buffer.alloc(16)); + const right = thinMacho(0x0100000c, Buffer.alloc(16)); + const overhang = 10; + FUSE_SENTINEL.copy(left, left.length - overhang); + const gap = Buffer.alloc(FUSE_SENTINEL.length - overhang); + FUSE_SENTINEL.copy(gap, 0, overhang); + const tableEnd = 48; + const rightOffset = tableEnd + left.length + gap.length; + const table = Buffer.alloc(tableEnd); + table.writeUInt32BE(0xcafebabe, 0); + table.writeUInt32BE(2, 4); + table.writeUInt32BE(0x01000007, 8); + table.writeUInt32BE(tableEnd, 16); + table.writeUInt32BE(left.length, 20); + table.writeUInt32BE(0x0100000c, 28); + table.writeUInt32BE(rightOffset, 36); + table.writeUInt32BE(right.length, 40); + const path = join(root, "split"); + await writeFile(path, Buffer.concat([table, left, gap, right])); + const report = await inspect(path); + expect(report.slices.map((item) => item.sentinel_count)).toEqual([0, 0]); + }); +}); + +describe("direct Electron Framework binary selection", () => { + const frameworkFixture = async (prefix: string) => { + const root = await createTestTempDirectory(prefix); + const app = join(root, "Fixture.app"); + const framework = join( + app, + "Contents", + "Frameworks", + "Electron Framework.framework", + ); + await mkdir(join(framework, "Versions"), { recursive: true }); + return { app, framework }; + }; + + it("resolves a binary selected through Versions/Current to its target", async () => { + const { framework } = await frameworkFixture("rea-electron-fuses-direct-"); + await mkdir(join(framework, "Versions", "A"), { recursive: true }); + const binary = join(framework, "Versions", "A", "Electron Framework"); + await writeFile(binary, thinFuseImage(OBSIDIAN_FUSE_BYTES).bytes); + await symlink("A", join(framework, "Versions", "Current")); + + const selected = join( + framework, + "Versions", + "Current", + "Electron Framework", + ); + const report = await inspect(selected); + expect(report.artifact.selected_path).toBe(selected); + expect(report.artifact.resolved_path).toBe(binary); + }); + + it("follows symlinks before parent segments in a selected binary path", async () => { + const { framework } = await frameworkFixture("rea-electron-fuses-parent-"); + const versions = join(framework, "Versions"); + await mkdir(join(versions, "nested", "B"), { recursive: true }); + await mkdir(join(versions, "nested", "A"), { recursive: true }); + await mkdir(join(versions, "A"), { recursive: true }); + const binary = join(versions, "nested", "A", "Electron Framework"); + await writeFile(binary, thinFuseImage(OBSIDIAN_FUSE_BYTES).bytes); + await writeFile( + join(versions, "A", "Electron Framework"), + sentinelFreeMacho(), + ); + await symlink("nested/B", join(versions, "Current")); + const selected = `${versions}/Current/../A/Electron Framework`; + const report = await inspect(selected); + expect(report.artifact.selected_path).toBe(selected); + expect(report.artifact.resolved_path).toBe(binary); + expect(report.slices[0]?.sentinel_count).toBe(1); + }); + + it("rejects a binary selected through a Versions/Current link that leaves the framework", async () => { + const { app, framework } = await frameworkFixture( + "rea-electron-fuses-direct-escape-", + ); + await mkdir(join(app, "Contents", "Other"), { recursive: true }); + await writeFile( + join(app, "Contents", "Other", "Electron Framework"), + elfFuseImage([ELECTRON_FUSE_STATE_BYTES.enabled]).bytes, + ); + await symlink("../../../Other", join(framework, "Versions", "Current")); + + const result = await provider.inspect({ + path: join(framework, "Versions", "Current", "Electron Framework"), + }); + expect(result.ok).toBe(false); + if (result.ok) return; + expect(result.error).toMatchObject({ + _tag: "AnalysisInputError", + issues: [ + expect.objectContaining({ + path: ["path"], + message: expect.stringContaining("escapes"), + }), + ], + }); + }); + + it("rejects a framework directory that is a symlink outside the bundle", async () => { + const root = await createTestTempDirectory( + "rea-electron-fuses-framework-link-", + ); + const outside = join(root, "outside"); + await mkdir(join(outside, "Versions", "A"), { recursive: true }); + await writeFile( + join(outside, "Versions", "A", "Electron Framework"), + thinFuseImage(OBSIDIAN_FUSE_BYTES).bytes, + ); + const app = join(root, "Example.app"); + const frameworks = join(app, "Contents", "Frameworks"); + await mkdir(frameworks, { recursive: true }); + const framework = join(frameworks, "Electron Framework.framework"); + await symlink(outside, framework); + + for (const path of [ + app, + join(framework, "Versions", "A", "Electron Framework"), + ]) { + const result = await provider.inspect({ path }); + expect(result.ok, path).toBe(false); + if (result.ok) continue; + expect(result.error).toMatchObject({ + _tag: "AnalysisInputError", + issues: [ + expect.objectContaining({ + path: ["path"], + message: expect.stringContaining("escapes"), + }), + ], + }); + } + }); +}); + +it.each(["Contents", "Frameworks"])( + "rejects a redirected %s for every app selection form", + async (segment) => { + const root = await createTestTempDirectory("rea-electron-parent-link-"); + const app = join(root, "Fixture.app"); + const outside = join(root, "outside"); + const actualFramework = join( + outside, + ...(segment === "Contents" ? ["Frameworks"] : []), + "Electron Framework.framework", + ); + await mkdir(join(actualFramework, "Versions", "A"), { recursive: true }); + await writeFile( + join(actualFramework, "Versions", "A", "Electron Framework"), + thinFuseImage(OBSIDIAN_FUSE_BYTES).bytes, + ); + await symlink("A", join(actualFramework, "Versions", "Current")); + await mkdir(segment === "Contents" ? app : join(app, "Contents"), { + recursive: true, + }); + await symlink( + outside, + join( + app, + "Contents", + ...(segment === "Frameworks" ? ["Frameworks"] : []), + ), + ); + const framework = join( + app, + "Contents", + "Frameworks", + "Electron Framework.framework", + ); + for (const path of [ + app, + framework, + join(framework, "Versions", "Current", "Electron Framework"), + ]) { + const result = await provider.inspect({ path }); + expect(result.ok, path).toBe(false); + if (!result.ok) expect(result.error._tag).toBe("AnalysisInputError"); + } + }, +); + +it("enforces the 32 sentinel boundary with typed and public limit details", async () => { + const root = await createTestTempDirectory("rea-electron-sentinel-limit-"); + const wire = Buffer.concat([FUSE_SENTINEL, fuseWire(1, [0x31])]); + for (const count of [32, 33]) { + const path = join(root, String(count)); + await writeFile( + path, + thinMacho( + 0x0100000c, + Buffer.concat(Array.from({ length: count }, () => wire)), + ), + ); + const result = await provider.inspect({ path }); + if (count === 32) { + expect(result.ok).toBe(true); + if (result.ok) expect(result.value.slices[0]?.sentinel_count).toBe(32); + } else { + expect(result.ok).toBe(false); + if (!result.ok) { + expect(result.error._tag).toBe("AnalysisResourceConstraintError"); + expect(projectAnalysisError(result.error)).toMatchObject({ + code: "resource_constraint", + details: { reported_limits: { maximum_sentinels_per_slice: 32 } }, + }); + } + } + } +}); + +it("rejects a sparse artifact over 1 GiB before scanning", async () => { + const root = await createTestTempDirectory("rea-electron-input-limit-"); + const path = join(root, "large"); + await writeFile(path, thinFuseImage(OBSIDIAN_FUSE_BYTES).bytes); + await truncate(path, 1024 * 1024 * 1024 + 1); + const result = await provider.inspect({ path }); + expect(result.ok).toBe(false); + if (!result.ok) { + expect(result.error._tag).toBe("AnalysisResourceConstraintError"); + expect(projectAnalysisError(result.error)).toMatchObject({ + code: "resource_constraint", + details: { reported_limits: { maximum_input_bytes: 1024 * 1024 * 1024 } }, + }); + } +}); + +it("follows the active framework binary while an inactive A version remains", async () => { + const root = await createTestTempDirectory("rea-electron-current-version-"); + const app = join(root, "Fixture.app"); + const framework = join( + app, + "Contents", + "Frameworks", + "Electron Framework.framework", + ); + for (const [version, state] of [ + ["A", 0x30], + ["B", 0x31], + ] as const) { + await mkdir(join(framework, "Versions", version), { recursive: true }); + await writeFile( + join(framework, "Versions", version, "Electron Framework"), + thinFuseImage([state]).bytes, + ); + } + await symlink("B", join(framework, "Versions", "Current")); + await symlink( + "Versions/Current/Electron Framework", + join(framework, "Electron Framework"), + ); + for (const path of [app, framework, join(framework, "Electron Framework")]) { + const report = await inspect(path); + expect(report.artifact.resolved_path).toBe( + join(framework, "Versions", "B", "Electron Framework"), + ); + expect(report.slices[0]?.sentinels[0]?.fuses[0]?.state).toBe("enabled"); + } +}); + +it("rejects a checked version directory swapped before the reader opens it", async () => { + const root = await createTestTempDirectory("rea-electron-directory-swap-"); + const app = join(root, "Fixture.app"); + const framework = join( + app, + "Contents", + "Frameworks", + "Electron Framework.framework", + ); + const version = join(framework, "Versions", "A"); + const outside = join(root, "outside"); + await mkdir(version, { recursive: true }); + await mkdir(outside); + await writeFile( + join(version, "Electron Framework"), + thinFuseImage([0x30]).bytes, + ); + await writeFile( + join(outside, "Electron Framework"), + thinFuseImage([0x31]).bytes, + ); + const located = await resolveElectronFuseTarget(app); + await rename(version, `${version}.saved`); + await symlink(outside, version); + await expect( + readElectronFuseImage(located.file.path, { selection: located.file }), + ).rejects.toBeInstanceOf(RegularFileChangedError); +}); + +it("reports a checked version directory removed before open as changed", async () => { + const root = await createTestTempDirectory("rea-electron-directory-missing-"); + const app = join(root, "Fixture.app"); + const version = join( + app, + "Contents", + "Frameworks", + "Electron Framework.framework", + "Versions", + "A", + ); + await mkdir(version, { recursive: true }); + await writeFile( + join(version, "Electron Framework"), + thinFuseImage([0x30]).bytes, + ); + const located = await resolveElectronFuseTarget(app); + await rename(version, `${version}.saved`); + await expect( + readElectronFuseImage(located.file.path, { selection: located.file }), + ).rejects.toBeInstanceOf(RegularFileChangedError); +}); + +it("retains the identity of a directory removed by a terminal parent segment", async () => { + const root = await createTestTempDirectory("rea-electron-parent-identity-"); + const framework = join(root, "Electron Framework.framework"); + const versions = join(framework, "Versions"); + await mkdir(versions, { recursive: true }); + await writeFile( + join(framework, "Electron Framework"), + thinFuseImage([0x31]).bytes, + ); + const located = await resolveElectronFuseTarget(`${versions}/..`); + await rename(versions, `${versions}.saved`); + await mkdir(versions); + await expect( + readElectronFuseImage(located.file.path, { selection: located.file }), + ).rejects.toBeInstanceOf(RegularFileChangedError); +}); diff --git a/src/electron/ElectronFuseProvider.ts b/src/electron/ElectronFuseProvider.ts new file mode 100644 index 0000000000..d05baff43d --- /dev/null +++ b/src/electron/ElectronFuseProvider.ts @@ -0,0 +1,170 @@ +import type { ExecutionOptions } from "../application/AnalysisProvider.js"; +import type { ElectronFusePort } from "../application/electron/ElectronFusePort.js"; +import { AnalysisError } from "../domain/analysisErrorBase.js"; +import { + AnalysisAccessDeniedError, + AnalysisArtifactChangedError, + AnalysisCancelledError, + AnalysisInputError, + AnalysisOutputError, + AnalysisResourceConstraintError, + AnalysisUnsupportedTargetError, +} from "../domain/analysisErrorCore.js"; +import { + describeElectronFuseFindings, + electronFuseReportSchema, + type ElectronFuseReport, + type InspectElectronFusesInput, +} from "../domain/electron/electronFuses.js"; +import { ProviderAdapterError } from "../domain/providerAdapterError.js"; +import { err, ok, type Result } from "../domain/result.js"; +import { ZodError } from "zod"; +import { RegularFileChangedError } from "../filesystem/RegularFile.js"; +import { NonRegularFileReadError } from "../filesystem/RegularFile.js"; +import { + ElectronFusePathFailure, + resolveElectronFuseTarget, +} from "./ElectronFusePath.js"; +import { + electronFuseReadLimitation, + ElectronFuseFormatFailure, + ElectronFuseResourceFailure, + readElectronFuseImage, +} from "./ElectronFuseReader.js"; + +/** Bundled fuse-wire reader. It does not flip fuses or launch the application. */ +export const ELECTRON_FUSE_PROVIDER_IDENTITY = { + id: "electron-fuses", + name: "REA Electron fuse reader", + version: "1", +} as const; + +const OPERATION = "inspect_electron_fuses"; + +/** Read packaged Electron fuse wires from one selected local file or app bundle. */ +export class ElectronFuseProvider implements ElectronFusePort { + readonly identity = ELECTRON_FUSE_PROVIDER_IDENTITY; + + async inspect( + input: InspectElectronFusesInput, + options?: ExecutionOptions, + ): Promise> { + try { + options?.signal?.throwIfAborted(); + const located = await resolveElectronFuseTarget( + input.path, + options?.signal, + ); + options?.signal?.throwIfAborted(); + const image = await readElectronFuseImage(located.file.path, { + selection: located.file, + ...(options?.signal === undefined ? {} : { signal: options.signal }), + }); + const findings = describeElectronFuseFindings(image.slices, image.format); + const parsed = electronFuseReportSchema.safeParse({ + artifact: { + selected_path: located.selectedPath, + resolved_path: located.file.path, + sha256: image.sha256, + bytes: image.bytes, + format: image.format, + }, + slices: image.slices, + runtime_implications: findings.runtime_implications, + limitations: [...findings.limitations, electronFuseReadLimitation()], + }); + if (!parsed.success) + throw new AnalysisOutputError( + OPERATION, + "Electron fuse reader produced a report outside its schema.", + { cause: parsed.error }, + ); + options?.signal?.throwIfAborted(); + return ok(parsed.data); + } catch (cause: unknown) { + return err(failureFor(cause, input.path, options?.signal)); + } + } +} + +const failureFor = ( + cause: unknown, + path: string, + signal?: AbortSignal, +): AnalysisError => { + if (cause instanceof AnalysisError) return cause; + if (signal?.aborted) return new AnalysisCancelledError(OPERATION); + if (cause instanceof ElectronFusePathFailure) + return new AnalysisInputError(OPERATION, { cause }, [ + { path: ["path"], reason: cause.reason, message: cause.message }, + ]); + if (cause instanceof ElectronFuseResourceFailure) + return new AnalysisResourceConstraintError( + OPERATION, + cause.boundary === "file-size" ? "file-size" : "capacity", + cause.message, + cause.boundary === "file-size" + ? { maximum_input_bytes: 1024 * 1024 * 1024 } + : { maximum_sentinels_per_slice: 32 }, + { cause }, + ); + if (cause instanceof ElectronFuseFormatFailure) + return cause.kind === "unsupported" + ? new AnalysisUnsupportedTargetError(OPERATION, path, cause.message, { + cause, + }) + : new AnalysisInputError(OPERATION, { cause }, [ + { + path: ["path"], + reason: "invalid_format", + message: `Selected artifact ${path}: ${cause.message}`, + }, + ]); + if (cause instanceof RegularFileChangedError) + return new AnalysisArtifactChangedError(OPERATION, path, cause.message, { + cause, + }); + if (cause instanceof NonRegularFileReadError) + return new AnalysisInputError(OPERATION, { cause }, [ + { + path: ["path"], + reason: "invalid_format", + message: cause.message, + }, + ]); + if (cause instanceof ZodError) + return new AnalysisOutputError( + OPERATION, + "Electron fuse reader produced a report outside its schema.", + { cause }, + ); + if (cause instanceof Error && "code" in cause) { + if (cause.code === "EACCES" || cause.code === "EPERM") + return new AnalysisAccessDeniedError(OPERATION, path, cause.code, { + cause, + }); + if ( + ["ENOENT", "ENOTDIR", "ELOOP", "ENAMETOOLONG"].includes( + String(cause.code), + ) + ) + return new AnalysisInputError(OPERATION, { cause }, [ + { + path: ["path"], + reason: "invalid_value", + message: `Selected artifact could not be read (${String(cause.code)}): ${path}.`, + }, + ]); + } + return new ProviderAdapterError( + ELECTRON_FUSE_PROVIDER_IDENTITY.id, + OPERATION, + { + cause, + diagnostics: { + path, + reason: cause instanceof Error ? cause.message : String(cause), + }, + }, + ); +}; diff --git a/src/electron/ElectronFuseReader.ts b/src/electron/ElectronFuseReader.ts new file mode 100644 index 0000000000..4db52b9c7d --- /dev/null +++ b/src/electron/ElectronFuseReader.ts @@ -0,0 +1,480 @@ +import type { ResolvedElectronFuseFile } from "./ElectronFusePath.js"; +import { createHash } from "node:crypto"; +import { lstat } from "node:fs/promises"; + +import { + hasMachoMagic, + readMachoImage, +} from "../artifacts/apple/MachoLoadCommandReader.js"; +import type { ReadAt } from "../artifacts/apple/MachoLoadCommandReader.js"; +import { + electronFuseName, + electronFuseState, + ELECTRON_FUSE_SENTINEL, + type ElectronFuseSentinel, + type ElectronFuseSlice, +} from "../domain/electron/electronFuses.js"; +import { + openRegularFile, + RegularFileChangedError, + sameRegularFileState, +} from "../filesystem/RegularFile.js"; + +/** Admission and search bounds for one Electron executable. */ +export const ELECTRON_FUSE_LIMITS = { + maximumFileBytes: 1024 * 1024 * 1024, + chunkBytes: 256 * 1024, + maximumSentinelsPerSlice: 32, + maximumPeHeaderOffset: 1024 * 1024, +} as const; + +const SENTINEL = Buffer.from(ELECTRON_FUSE_SENTINEL, "ascii"); +const FAT_MAGICS = new Set([0xcafebabe, 0xcafebabf, 0xbebafeca, 0xbfbafeca]); + +/** Container identity and per-slice fuse wires. Offsets are file-absolute. */ +export interface ElectronFuseImage { + readonly sha256: string; + readonly bytes: number; + readonly format: "mach-o" | "mach-o-universal" | "pe" | "elf"; + readonly slices: readonly ElectronFuseSlice[]; +} + +type ElectronFuseContainerFormat = ElectronFuseImage["format"]; + +interface SliceRange { + readonly index: number; + readonly fileOffset: number; + readonly size: number; + readonly architecture: string | null; +} + +/** The selected container is malformed or outside the supported formats. */ +export class ElectronFuseFormatFailure extends Error { + constructor( + readonly kind: "malformed" | "unsupported", + message: string, + ) { + super(message); + this.name = "ElectronFuseFormatFailure"; + } +} + +/** A measured bound was hit before a complete fuse report could be returned. */ +export class ElectronFuseResourceFailure extends Error { + constructor( + readonly boundary: "file-size" | "capacity", + message: string, + ) { + super(message); + this.name = "ElectronFuseResourceFailure"; + } +} + +/** + * Hash one admitted regular file and search each slice for the fuse sentinel. + * The search keeps only a sentinel-sized overlap between chunks. + */ +export const readElectronFuseImage = async ( + path: string, + options?: { + readonly signal?: AbortSignal; + readonly chunkBytes?: number; + readonly selection?: ResolvedElectronFuseFile; + }, +): Promise => { + const signal = options?.signal; + signal?.throwIfAborted(); + const selection = options?.selection; + const recheckPath = async (candidate: string) => { + signal?.throwIfAborted(); + try { + const metadata = await lstat(candidate, { bigint: true }); + signal?.throwIfAborted(); + return metadata; + } catch (cause: unknown) { + if (selection !== undefined && isMissingPath(cause)) + throw new RegularFileChangedError(path); + throw cause; + } + }; + const assertPathNodes = async (): Promise => { + if (selection === undefined) return; + if (selection.path !== path) throw new RegularFileChangedError(path); + for (const node of selection.nodes) { + const current = await recheckPath(node.path); + if ( + current.dev !== node.metadata.dev || + current.ino !== node.metadata.ino || + current.isDirectory() !== node.metadata.isDirectory() || + current.isSymbolicLink() !== node.metadata.isSymbolicLink() + ) + throw new RegularFileChangedError(path); + } + }; + await assertPathNodes(); + const before = await recheckPath(path); + if ( + selection !== undefined && + !sameRegularFileState(selection.metadata, before) + ) + throw new RegularFileChangedError(path); + if (!before.isFile() || before.isSymbolicLink()) + throw new ElectronFuseFormatFailure( + "unsupported", + `Expected a regular file without a symlink: ${path}`, + ); + if (before.size > BigInt(ELECTRON_FUSE_LIMITS.maximumFileBytes)) + throw new ElectronFuseResourceFailure( + "file-size", + `Selected artifact exceeds ${ELECTRON_FUSE_LIMITS.maximumFileBytes} bytes: ${path}`, + ); + const size = Number(before.size); + const chunkBytes = options?.chunkBytes ?? ELECTRON_FUSE_LIMITS.chunkBytes; + if ( + !Number.isSafeInteger(chunkBytes) || + chunkBytes < 1 || + chunkBytes > ELECTRON_FUSE_LIMITS.chunkBytes + ) + throw new TypeError( + `chunkBytes must be an integer between 1 and ${ELECTRON_FUSE_LIMITS.chunkBytes}.`, + ); + const handle = await openRegularFile(path, { symlinks: "reject", signal }); + try { + const opened = await handle.stat({ bigint: true }); + if (!sameRegularFileState(before, opened)) + throw new RegularFileChangedError(path); + await assertPathNodes(); + const readAt = readerFor(handle, size, path, signal); + const container = await identifyContainer(readAt, size, path); + const scanned = await scanSlices( + handle, + path, + size, + container.slices, + chunkBytes, + signal, + ); + const current = await recheckPath(path); + if (!sameRegularFileState(before, current)) + throw new RegularFileChangedError(path); + await assertPathNodes(); + return { + sha256: scanned.sha256, + bytes: size, + format: container.format, + slices: scanned.slices, + }; + } finally { + await handle.close(); + } +}; + +/** Sentence describing the file and sentinel bounds enforced by this reader. */ +export const electronFuseReadLimitation = (): string => + `The selected file is admitted as one regular file of at most 1 GiB. Sentinel search uses ${ELECTRON_FUSE_LIMITS.chunkBytes}-byte windows, and each slice keeps at most ${ELECTRON_FUSE_LIMITS.maximumSentinelsPerSlice} sentinels.`; + +const readerFor = + ( + handle: Awaited>, + size: number, + path: string, + signal: AbortSignal | undefined, + ): ReadAt => + async (offset, length) => { + signal?.throwIfAborted(); + if (length <= 0 || offset < 0 || offset >= size) return new Uint8Array(); + const bounded = Math.min(length, size - offset); + const buffer = Buffer.alloc(bounded); + const read = await handle.read(buffer, 0, bounded, offset); + if (read.bytesRead !== bounded) throw new RegularFileChangedError(path); + return buffer.subarray(0, read.bytesRead); + }; + +const identifyContainer = async ( + readAt: ReadAt, + size: number, + path: string, +): Promise<{ + readonly format: ElectronFuseContainerFormat; + readonly slices: readonly SliceRange[]; +}> => { + const head = await readAt(0, Math.min(size, 8)); + if (hasMachoMagic(head)) return identifyMacho(readAt, size, head, path); + if (isElf(head)) + return { + format: "elf", + slices: [{ index: 0, fileOffset: 0, size, architecture: null }], + }; + if (await isPe(readAt, size)) + return { + format: "pe", + slices: [{ index: 0, fileOffset: 0, size, architecture: null }], + }; + throw new ElectronFuseFormatFailure( + "unsupported", + `Selected artifact ${path} does not contain a Mach-O, ELF, or PE container.`, + ); +}; + +const identifyMacho = async ( + readAt: ReadAt, + size: number, + head: Uint8Array, + path: string, +): Promise<{ + readonly format: ElectronFuseContainerFormat; + readonly slices: readonly SliceRange[]; +}> => { + const facts = await readMachoImage(readAt, size); + if (facts.status !== "parsed") { + const reason = + facts.status === "not-mach-o" + ? `Selected artifact ${path} does not contain a Mach-O, ELF, or PE container.` + : facts.reason; + throw new ElectronFuseFormatFailure( + facts.status === "malformed" ? "malformed" : "unsupported", + reason, + ); + } + const format = isFat(head) ? "mach-o-universal" : "mach-o"; + const slices = facts.slices.map((slice, index) => ({ + index, + fileOffset: slice.slice_offset, + size: slice.slice_size, + architecture: slice.architecture, + })); + assertNonOverlappingSlices(slices); + return { + format, + slices, + }; +}; + +/** A FAT architecture owns a distinct byte range; overlap is malformed. */ +const assertNonOverlappingSlices = (ranges: readonly SliceRange[]): void => { + const ordered = [...ranges].sort( + (left, right) => left.fileOffset - right.fileOffset, + ); + for (let index = 1; index < ordered.length; index += 1) { + const previous = ordered[index - 1]; + const current = ordered[index]; + if ( + previous !== undefined && + current !== undefined && + current.fileOffset < previous.fileOffset + previous.size + ) + throw new ElectronFuseFormatFailure( + "malformed", + `Mach-O slices ${previous.index} and ${current.index} overlap.`, + ); + } +}; + +const scanSlices = async ( + handle: Awaited>, + path: string, + size: number, + ranges: readonly SliceRange[], + chunkBytes: number, + signal: AbortSignal | undefined, +): Promise<{ + readonly sha256: string; + readonly slices: readonly ElectronFuseSlice[]; +}> => { + const states = ranges.map((slice) => ({ + slice, + carry: Buffer.alloc(0), + hits: [] as number[], + })); + const hash = createHash("sha256"); + const chunk = Buffer.allocUnsafe(chunkBytes); + let position = 0; + while (position < size) { + signal?.throwIfAborted(); + const length = Math.min(chunkBytes, size - position); + const read = await handle.read(chunk, 0, length, position); + if (read.bytesRead !== length) throw new RegularFileChangedError(path); + const piece = chunk.subarray(0, length); + hash.update(piece); + absorbChunk(states, piece, position); + position += length; + } + if (position !== size) throw new RegularFileChangedError(path); + const slices: ElectronFuseSlice[] = []; + for (const state of states) { + const sentinels: ElectronFuseSentinel[] = []; + for (const offset of state.hits) + sentinels.push( + await parseWire( + handle, + path, + offset, + state.slice.fileOffset + state.slice.size, + signal, + ), + ); + slices.push({ + index: state.slice.index, + file_offset: state.slice.fileOffset, + size: state.slice.size, + architecture: state.slice.architecture, + sentinel_count: sentinels.length, + sentinels, + }); + } + return { sha256: hash.digest("hex"), slices }; +}; + +const absorbChunk = ( + states: readonly { + readonly slice: SliceRange; + carry: Buffer; + readonly hits: number[]; + }[], + chunk: Buffer, + position: number, +): void => { + const chunkEnd = position + chunk.length; + for (const state of states) { + const start = Math.max(position, state.slice.fileOffset); + const end = Math.min(chunkEnd, state.slice.fileOffset + state.slice.size); + if (start >= end) continue; + const piece = chunk.subarray(start - position, end - position); + const carryLength = state.carry.length; + const window = + carryLength === 0 ? piece : Buffer.concat([state.carry, piece]); + const windowFileStart = start - carryLength; + let from = 0; + while (from <= window.length - SENTINEL.length) { + const found = window.indexOf(SENTINEL, from); + if (found < 0) break; + const sentinelOffset = windowFileStart + found; + if (!state.hits.includes(sentinelOffset)) { + if (state.hits.length >= ELECTRON_FUSE_LIMITS.maximumSentinelsPerSlice) + throw new ElectronFuseResourceFailure( + "capacity", + `Slice ${state.slice.index} contains more than ${ELECTRON_FUSE_LIMITS.maximumSentinelsPerSlice} Electron fuse sentinels.`, + ); + state.hits.push(sentinelOffset); + } + from = found + 1; + } + const tail = SENTINEL.length - 1; + state.carry = + window.length <= tail + ? Buffer.from(window) + : Buffer.from(window.subarray(window.length - tail)); + } +}; + +const parseWire = async ( + handle: Awaited>, + path: string, + sentinelOffset: number, + sliceEnd: number, + signal: AbortSignal | undefined, +): Promise => { + const wireStart = sentinelOffset + SENTINEL.length; + const available = sliceEnd - wireStart; + if (available < 1) + return { + sentinel_offset: sentinelOffset, + wire_version: null, + declared_length: null, + fuses: [], + complete: false, + }; + const header = await readExact( + handle, + path, + wireStart, + Math.min(available, 2), + signal, + ); + const wireVersion = header[0]; + if (wireVersion === undefined || header.length < 2) + return { + sentinel_offset: sentinelOffset, + wire_version: wireVersion ?? null, + declared_length: null, + fuses: [], + complete: false, + }; + const declared = header[1] ?? 0; + const fuseLength = Math.min(declared, available - 2); + const fuseBytes = + fuseLength === 0 + ? Buffer.alloc(0) + : await readExact(handle, path, wireStart + 2, fuseLength, signal); + return { + sentinel_offset: sentinelOffset, + wire_version: wireVersion, + declared_length: declared, + fuses: [...fuseBytes].map((raw, index) => ({ + index, + name: electronFuseName(wireVersion, index), + state: electronFuseState(raw), + raw_byte: raw, + })), + complete: fuseBytes.length === declared, + }; +}; + +const readExact = async ( + handle: Awaited>, + path: string, + offset: number, + length: number, + signal: AbortSignal | undefined, +): Promise => { + signal?.throwIfAborted(); + const buffer = Buffer.alloc(length); + const read = await handle.read(buffer, 0, length, offset); + if (read.bytesRead !== length) throw new RegularFileChangedError(path); + return buffer; +}; + +const isElf = (head: Uint8Array): boolean => + head.length >= 4 && + head[0] === 0x7f && + head[1] === 0x45 && + head[2] === 0x4c && + head[3] === 0x46; + +const isFat = (head: Uint8Array): boolean => { + if (head.length < 4) return false; + const magic = Buffer.from(head.buffer, head.byteOffset, 4).readUInt32BE(0); + return FAT_MAGICS.has(magic); +}; + +const isPe = async (readAt: ReadAt, size: number): Promise => { + if (size < 64) return false; + const header = await readAt(0, 64); + if (header.length < 64 || header[0] !== 0x4d || header[1] !== 0x5a) + return false; + const offset = Buffer.from( + header.buffer, + header.byteOffset, + header.byteLength, + ).readUInt32LE(0x3c); + if ( + !Number.isSafeInteger(offset) || + offset < 64 || + offset > ELECTRON_FUSE_LIMITS.maximumPeHeaderOffset || + offset > size - 4 + ) + return false; + const signature = await readAt(offset, 4); + return ( + signature.length === 4 && + signature[0] === 0x50 && + signature[1] === 0x45 && + signature[2] === 0 && + signature[3] === 0 + ); +}; + +const isMissingPath = (cause: unknown): boolean => + cause instanceof Error && + "code" in cause && + ["ENOENT", "ENOTDIR"].includes(String(cause.code)); diff --git a/src/evaluation/AgentEvaluationFixtures.ts b/src/evaluation/AgentEvaluationFixtures.ts index b0535a96e9..83b22db415 100644 --- a/src/evaluation/AgentEvaluationFixtures.ts +++ b/src/evaluation/AgentEvaluationFixtures.ts @@ -8,6 +8,11 @@ import type { JsonValue } from "../domain/jsonValue.js"; import { javascriptApplicationAnalysisResultSchema } from "../domain/javascript/javascriptApplicationAnalysis.js"; import type { ApplicationNode } from "../domain/javascript/javascriptApplicationGraphSchemas.js"; import { javaScriptExportShapeComparisonResultSchema } from "../domain/javascript/javascriptExportShapeComparisonSchemas.js"; +import { + applicationFeatureTraceResultSchema, + featureSeedMatchMode, +} from "../domain/javascript/javascriptFeatureTraceSchemas.js"; +import { javaScriptSemanticTraceResultSchema } from "../domain/javascript/javascriptSemanticTraceSchemas.js"; import { managedArtifactInspectionSchema, managedMemberInspectionSchema, @@ -28,6 +33,8 @@ export interface AgentEvaluationFixtureTargets { readonly javascriptSha256: string; readonly javascriptShapeLeft: string; readonly javascriptShapeRight: string; + readonly notesElectron: string; + readonly notesElectronSha256: string; readonly managed?: string; readonly managedSha256?: string; } @@ -50,6 +57,8 @@ export const createAgentEvaluationFixtures = async ( const javascript = join(root, "desktop-app.asar"); const javascriptShapeLeft = join(root, "parser-v1"); const javascriptShapeRight = join(root, "parser-v2"); + const notesDirectory = join(root, "notes-electron"); + const notesElectron = join(root, "notes-electron.asar"); await Promise.all([ mkdir(join(desktop, "renderer"), { recursive: true }), mkdir(javascriptShapeLeft, { recursive: true }), @@ -67,8 +76,14 @@ export const createAgentEvaluationFixtures = async ( join(repositoryRoot, "tests/fixtures/replay/parser-v2.mjs"), join(javascriptShapeRight, "parser.mjs"), ), + cp( + join(repositoryRoot, "website/public/examples/notes-electron"), + notesDirectory, + { recursive: true }, + ), ]); await createPackage(desktop, javascript); + await createPackage(notesDirectory, notesElectron); return { javascript, javascriptSha256: createHash("sha256") @@ -76,6 +91,10 @@ export const createAgentEvaluationFixtures = async ( .digest("hex"), javascriptShapeLeft, javascriptShapeRight, + notesElectron, + notesElectronSha256: createHash("sha256") + .update(await readFile(notesElectron)) + .digest("hex"), }; }; @@ -189,21 +208,116 @@ const runtimeSemantics = (value: JsonValue): JsonValue | undefined => { : undefined; }; -const asarClaims = ( +const featureModules = (value: JsonValue): JsonValue | undefined => { + const result = applicationFeatureTraceResultSchema.safeParse(value); + if (!result.success || result.data.graph === null) return undefined; + if ( + result.data.seed.kind !== "channel" || + result.data.seed.value !== "notes:export" || + featureSeedMatchMode(result.data.seed) !== "exact" + ) + return undefined; + const paths = result.data.graph.nodes + .filter((node) => node.kind === "javascript-module") + .flatMap((node) => node.observations) + .filter(({ properties }) => properties.semantic_role === "source-module") + .map(({ properties }) => properties.path); + if (paths.length === 0 || !paths.every((path) => typeof path === "string")) + return undefined; + return { + paths: [...new Set(paths)].sort(compareUnicodeCodePoints), + coverage: result.data.coverage.status, + }; +}; + +const semanticBoundary = (value: JsonValue): JsonValue | undefined => { + const result = javaScriptSemanticTraceResultSchema.safeParse(value); + if (!result.success) return undefined; + return { + status: result.data.status, + coverage: result.data.coverage.status, + relevant_unknowns: result.data.summary.relevant_unknowns, + }; +}; + +const notesClaims = ( targets: AgentEvaluationFixtureTargets, +): readonly FixtureClaimExpectation[] => { + const subject = { + path: targets.notesElectron, + sha256: targets.notesElectronSha256, + }; + const derived = { + authority: "analyst-inference", + confidence: "inferred", + linkedSubjects: [ + { operation: "analyze_javascript_application", ...subject }, + ], + } satisfies Pick< + FixtureClaimExpectation["source"], + "authority" | "confidence" | "linkedSubjects" + >; + // Fixture source establishes bridge/module facts; the semantic claim checks + // the bounded static-query contract, not execution or CSV formatter behavior. + return [ + ...bridgeClaims(subject, "notes", ["exportCsv"], "notes:export"), + { + id: "feature_modules", + expectedValue: { + paths: ["csv.js", "main.js", "preload.js", "renderer.js"], + coverage: "complete-within-source", + }, + source: { + ...derived, + operation: "trace_application_feature", + parameters: { + direction: "both", + }, + select: featureModules, + }, + }, + { + id: "semantic_boundary", + expectedValue: { + status: "ambiguous", + coverage: "partial", + relevant_unknowns: 1, + }, + source: { + ...derived, + operation: "trace_javascript_semantics", + parameters: { + query: { + seed: { kind: "property", name: "invoke" }, + direction: "forward-influence", + include_ambiguous_dynamic_edges: false, + expected: null, + }, + }, + select: semanticBoundary, + }, + }, + ]; +}; + +const bridgeClaims = ( + subject: { readonly path: string; readonly sha256: string }, + key: string, + members: readonly string[], + channel: string, ): readonly FixtureClaimExpectation[] => { const source = { operation: "analyze_javascript_application", authority: "shipped-artifact", confidence: "derived", - subject: { path: targets.javascript, sha256: targets.javascriptSha256 }, + subject, } satisfies Omit; return [ { id: "renderer_api", expectedValue: { - key: "profileApi", - members: ["read"], + key, + members: [...members], world: "main", source: "preload.js", }, @@ -212,7 +326,7 @@ const asarClaims = ( { id: "renderer_ipc", expectedValue: { - channel: "profile:read", + channel, operation: "invoke", source: "preload.js", }, @@ -221,7 +335,7 @@ const asarClaims = ( { id: "main_ipc", expectedValue: { - channel: "profile:read", + channel, operation: "handle", source: "main.js", }, @@ -235,6 +349,16 @@ const asarClaims = ( ]; }; +const asarClaims = ( + targets: AgentEvaluationFixtureTargets, +): readonly FixtureClaimExpectation[] => + bridgeClaims( + { path: targets.javascript, sha256: targets.javascriptSha256 }, + "profileApi", + ["read"], + "profile:read", + ); + const parserClaims = ( targets: AgentEvaluationFixtureTargets, ): readonly FixtureClaimExpectation[] => { @@ -399,25 +523,33 @@ export const agentFixtureClaims = ( if (scenarioId === "asar") return asarClaims(targets); if (scenarioId === "managed") return managedClaims(targets); if (scenarioId === "javascript-export-shape") return parserClaims(targets); + if (scenarioId === "notes-export") return notesClaims(targets); return undefined; }; +const bridgeAnswerDescriptions = [ + 'renderer_api: {"key": string, "members": string[], "world": string, "source": string}, describing the exposed bridge API, its members, world, and source file.', + 'renderer_ipc: {"channel": string, "operation": string, "source": string}, describing the renderer-side IPC operation and its source file.', + 'main_ipc: {"channel": string, "operation": string, "source": string}, describing the main-side IPC operation and its source file.', + 'preload: {"path": string, "resolution_status": string}, describing the BrowserWindow-declared preload resolved within the shipped artifact.', +]; + const answerValueDescriptions: Readonly> = { managed: [ "public_types: string[], listing the exact public top-level type full names in Unicode code-point order from unfiltered member Evidence (omit type_names).", 'entry_point: {"declaring_type": string, "name": string}, resolving the CLI metadata entry-point token against the member inventory from the same artifact SHA-256 and module MVID. Cite member Evidence; both artifact and member Evidence must be delivered.', ], - asar: [ - 'renderer_api: {"key": string, "members": string[], "world": string, "source": string}, describing the exposed bridge API, its members, world, and source file.', - 'renderer_ipc: {"channel": string, "operation": string, "source": string}, describing the renderer-side IPC operation and its source file.', - 'main_ipc: {"channel": string, "operation": string, "source": string}, describing the main-side IPC operation and its source file.', - 'preload: {"path": string, "resolution_status": string}, describing the BrowserWindow-declared preload resolved within the shipped artifact.', - ], + asar: bridgeAnswerDescriptions, "javascript-export-shape": [ 'heading_change: {"status": string, "path": string, "discriminant": object or null, "left": object, "right": object}, copying the exact heading-change fields from the comparison result.', 'comparison_coverage: {"status": string, "paired_variants": number, "added": number, "removed": number, "changed": number, "unknown": number}, copying comparison coverage and summary counts.', 'runtime_semantics: "established" or "not_established", stating whether the cited comparison establishes runtime behavior.', ], + "notes-export": [ + ...bridgeAnswerDescriptions, + 'feature_modules: {"paths": string[], "coverage": string}, listing the unique source-module observation paths in the feature trace in Unicode code-point order, and its coverage status.', + 'semantic_boundary: {"status": string, "coverage": string, "relevant_unknowns": number}, copying the semantic trace status, coverage status, and summary.relevant_unknowns. Static traces do not establish runtime execution.', + ], }; /** Describe the strict answer contract without revealing any golden fixture values. */ diff --git a/src/evaluation/CodexAgentEval.ts b/src/evaluation/CodexAgentEval.ts index 55ec8333e6..7445240671 100644 --- a/src/evaluation/CodexAgentEval.ts +++ b/src/evaluation/CodexAgentEval.ts @@ -205,7 +205,7 @@ export const evaluateCodexEvents = ( ); const reaCalls = state.calls.filter( - ({ server }) => server === null || server.toLowerCase() === "rea", + ({ server }) => server?.toLowerCase() === "rea", ); const signatures = new Map(); let targetRevision = 0; diff --git a/src/evm/EvmoleFailures.ts b/src/evm/EvmoleFailures.ts index 63c6578aae..da1bae8d85 100644 --- a/src/evm/EvmoleFailures.ts +++ b/src/evm/EvmoleFailures.ts @@ -183,6 +183,9 @@ export const evmInterfaceFailure = ( { cause, ...cleanupOptions, + ...(cause instanceof OwnedCommandFailure && cause.snapshot !== null + ? { capturedOutput: capturedEvmOutput(cause.snapshot) } + : {}), diagnostics: { phase, path, @@ -198,13 +201,6 @@ export const evmInterfaceFailure = ( failure_kind: cause.reason, exit_code: cause.snapshot?.exitCode ?? null, signal: cause.snapshot?.signal ?? null, - stdout: cause.snapshot?.stdout.text ?? null, - stderr: cause.snapshot?.stderr.text ?? null, - ...(cause.snapshot === null - ? {} - : { - captured_output: { ...capturedEvmOutput(cause.snapshot) }, - }), } : {}), }, diff --git a/src/evm/EvmoleInterfaceProvider.ts b/src/evm/EvmoleInterfaceProvider.ts index cbcb316155..1868945a98 100644 --- a/src/evm/EvmoleInterfaceProvider.ts +++ b/src/evm/EvmoleInterfaceProvider.ts @@ -15,7 +15,10 @@ import type { import { readStableArtifact } from "../artifacts/readStableArtifact.js"; import { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js"; import { ArtifactReaderFailure } from "../artifacts/ArtifactReader.js"; -import { OwnedCommandFailure } from "../process/OwnedCommand.js"; +import { + OwnedCommandFailure, + ownedCommandResources, +} from "../process/OwnedCommand.js"; import type { AnalysisError, AnalysisCapturedOutput, @@ -28,6 +31,7 @@ import { AnalysisOutputError, } from "../domain/analysisErrorCore.js"; import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; +import { createEvidence } from "../domain/evidence.js"; import type { InspectEvmInterfaceInput } from "../domain/evm/evmInterface.js"; import { evmInterfaceSchema } from "../domain/evm/evmInterface.js"; import { solidityMetadataSchema } from "../domain/evm/solidityMetadata.js"; @@ -79,6 +83,63 @@ const replySchema = z.discriminatedUnion("ok", [ }), ]); type Launcher = NonNullable[2]>["launcher"]; +type EvmOutcome = Result; + +const compactErrorProjection = (error: AnalysisError) => { + const projection = projectAnalysisError(error); + if (error.capturedOutput === undefined) return projection; + // The cleanup result owns this exact captured stream. Other diagnostics and + // partial Evidence retain their original fields, including stdout/stderr. + const details = { ...projection.details }; + delete details.captured_output; + return { ...projection, details }; +}; + +const evmExecutionEvidence = ( + input: InspectEvmInterfaceInput, + execution: EvmInterfaceExecution, +) => + createEvidence(execution.subject ?? undefined, execution.provider, { + operation: OPERATION, + parameters: input, + result: execution.result, + ...(execution.rawResult === null ? {} : { rawResult: execution.rawResult }), + confidence: "inferred", + locations: execution.locations, + limitations: execution.limitations, + }); + +const evmCleanupFailure = ( + input: InspectEvmInterfaceInput, + result: EvmOutcome, + resources: readonly string[], + reason: string, + output?: AnalysisCapturedOutput, + cause?: unknown, +): ProviderCleanupError => { + const partialObservation = result.ok + ? evmExecutionEvidence(input, result.value) + : result.error.partialObservation; + const capturedOutput = result.ok + ? output + : (result.error.capturedOutput ?? output); + return new ProviderCleanupError( + EVMOLE_PROVIDER_IDENTITY.id, + resources, + { + reason, + previous_error: result.ok ? null : compactErrorProjection(result.error), + ...(capturedOutput === undefined + ? {} + : { captured_output: { ...capturedOutput } }), + }, + { + operation: OPERATION, + ...(cause === undefined ? {} : { cause }), + ...(partialObservation === undefined ? {} : { partialObservation }), + }, + ); +}; /** Run unchanged EVMole/WASM in an owned worker; no target, chain or network execution. */ export class EvmoleInterfaceProvider implements EvmInterfacePort { @@ -130,6 +191,12 @@ export class EvmoleInterfaceProvider implements EvmInterfacePort { if (pending !== undefined) return err(pending); let root: Pick | undefined; let processOwner: ProviderProcessSupervisor | undefined; + let processCleanup: + | { + readonly reason: string; + readonly resources: readonly string[]; + } + | undefined; let retainedOutput: AnalysisCapturedOutput | undefined; let phase: "configuration" | "artifact-read" | "worker" = "configuration"; let workerLimits: EvmWorkerLimits | undefined; @@ -245,6 +312,16 @@ export class EvmoleInterfaceProvider implements EvmInterfacePort { { ...(options?.signal === undefined ? {} : { signal: options.signal }), ...(this.launcher === undefined ? {} : { launcher: this.launcher }), + retainCleanup: (owner, cleanup) => { + processOwner = owner; + processCleanup = { + reason: cleanup.reason, + resources: + owner.launch.ownership === undefined + ? ["owned-worker"] + : ownedCommandResources(owner.launch.ownership, root?.path), + }; + }, }, ); const capturedOutput = capturedEvmOutput(execution); @@ -377,7 +454,10 @@ export class EvmoleInterfaceProvider implements EvmInterfacePort { }), ); } catch (cause: unknown) { - if (cause instanceof OwnedCommandFailure) + if ( + cause instanceof OwnedCommandFailure && + cause.cleanupOwner !== undefined + ) processOwner = cause.cleanupOwner; let fileFailure: EvmFileSizeFailureEvidence | undefined; let markerCleanupFailure: ArtifactReaderFailure | undefined; @@ -452,9 +532,26 @@ export class EvmoleInterfaceProvider implements EvmInterfacePort { ); } if (processOwner !== undefined) { - this.#pendingCleanup = { root, process: processOwner }; - // The command failure already owns this cleanup diagnostic and its resources. - return result; + this.#pendingCleanup = { + root, + process: processOwner, + }; + if (processCleanup === undefined) return result; + return err( + evmCleanupFailure( + input, + result, + [ + ...new Set([ + ...(root === undefined ? [] : [root.path]), + ...processCleanup.resources, + ]), + ], + processCleanup.reason, + retainedOutput, + processOwner, + ), + ); } if (root !== undefined) { try { @@ -464,19 +561,13 @@ export class EvmoleInterfaceProvider implements EvmInterfacePort { this.#pendingCleanup = { root: retainedRoot, process: undefined }; root = undefined; return err( - new ProviderCleanupError( - EVMOLE_PROVIDER_IDENTITY.id, + evmCleanupFailure( + input, + result, [retainedRoot.path], - { - reason: cause instanceof Error ? cause.message : String(cause), - previous_error: result.ok - ? null - : projectAnalysisError(result.error), - ...(retainedOutput === undefined - ? {} - : { captured_output: { ...retainedOutput } }), - }, - { operation: OPERATION }, + cause instanceof Error ? cause.message : String(cause), + retainedOutput, + cause, ), ); } @@ -553,7 +644,12 @@ export class EvmoleInterfaceProvider implements EvmInterfacePort { if (stopped.status === "incomplete") return new ProviderCleanupError( EVMOLE_PROVIDER_IDENTITY.id, - [pending.process.launch.ownership?.runId ?? "owned-worker"], + [ + ...(pending.root === undefined ? [] : [pending.root.path]), + ...(pending.process.launch.ownership === undefined + ? ["owned-worker"] + : ownedCommandResources(pending.process.launch.ownership)), + ], { reason: stopped.reason }, { operation: OPERATION }, ); diff --git a/src/evm/SolidityMetadata.test.ts b/src/evm/SolidityMetadata.test.ts index 1ca056f186..bf1eb5385d 100644 --- a/src/evm/SolidityMetadata.test.ts +++ b/src/evm/SolidityMetadata.test.ts @@ -67,6 +67,46 @@ it("preserves a UTF-8 BOM inside a CBOR text value rather than stripping it", () ).toBeNull(); }); +it.each([ + [ + "value", + "a1617860", + { kind: "text", value: "x" }, + { kind: "text", value: "" }, + ], + [ + "key", + "a1606178", + { kind: "text", value: "" }, + { kind: "text", value: "x" }, + ], + [ + "long-form value", + "a161787800", + { kind: "text", value: "x" }, + { kind: "text", value: "" }, + ], +])( + "decodes an empty CBOR text %s like any other text", + (_name, hex, key, value) => { + expect(inspectSolidityMetadata(bytecode(hex))).toMatchObject({ + status: "decoded", + entries: [{ key, value }], + }); + }, +); + +it("keeps the compiler version beside a shortest-form empty text", () => { + const result = inspectSolidityMetadata( + bytecode("a264736f6c634300081e617860"), + ); + expect(result).toMatchObject({ + status: "decoded", + compiler_version: { representation: "release-bytes", value: "0.8.30" }, + }); + expect(result.entries).toHaveLength(2); +}); + it.each([ ["invalid text", "a1617861ff"], ["truncated value", "a16178"], diff --git a/src/evm/SolidityMetadata.ts b/src/evm/SolidityMetadata.ts index 5b24ce2453..8a82bceed4 100644 --- a/src/evm/SolidityMetadata.ts +++ b/src/evm/SolidityMetadata.ts @@ -36,11 +36,14 @@ class BoundedMetadataTokenizer extends Tokenizer { "CBOR tags and floating-point values are outside this metadata profile.", ); if (Type.equals(token.type, Type.string)) { - if (token.byteValue === undefined) + if (token.byteValue !== undefined) + token.value = utf8.decode(token.byteValue); + // cborg answers the shortest-form empty text (0x60) with a shared token that has no + // bytes; it can only mean "", which is already its value. + else if (token.value !== "" || token.encodedLength !== 1) throw new Error( "CBOR text bytes were unavailable for strict UTF-8 validation.", ); - token.value = utf8.decode(token.byteValue); } const parent = this.remaining.length - 1; if (parent >= 0) this.remaining[parent]!--; diff --git a/src/firmware/FirmwareProvider.ts b/src/firmware/FirmwareProvider.ts index 31fac4df1f..b4c2badeb5 100644 --- a/src/firmware/FirmwareProvider.ts +++ b/src/firmware/FirmwareProvider.ts @@ -97,6 +97,7 @@ const limitations = [ "Signature matches and reported sizes do not establish partition validity, encryption or runtime mappings. Binwalk may infer unknown lengths before emitting its report.", "Only Linux is supported by this integration. One worker, inherited per-process address-space/file-size/CPU limits and a deadline constrain execution; staging byte and entry budgets are polled and may transiently overshoot. Limits are not aggregate process-memory or filesystem quotas.", "Unblob's default sandbox and skip rules remain enabled. Missing external extractors, unknown chunks, encrypted chunks and depth limits yield partial coverage. Extracted code is never executed; only regular files are published, without executable permission bits.", + "extraction_gaps lists regions that were carved or left unextracted and names extractor dependencies the producer reported as missing. A listed gap does not mean the target is corrupt or that a retained region is empty.", "The actual launcher executable is hashed. Interpreter, shared libraries and installed Python package dependencies are not independently attested; source revision remains unknown for caller-supplied builds.", ] as const; @@ -381,6 +382,7 @@ export class FirmwareProvider implements FirmwareAnalysisPort { entries, normalized, selection: target.selection, + maxDepth: request.input.max_depth, engine, signal, retainCleanup: (tree) => { diff --git a/src/firmware/FirmwarePublication.ts b/src/firmware/FirmwarePublication.ts index b2bc4ddd51..274a884b5c 100644 --- a/src/firmware/FirmwarePublication.ts +++ b/src/firmware/FirmwarePublication.ts @@ -1,3 +1,4 @@ +import { createHash } from "node:crypto"; import { createReadStream } from "node:fs"; import { join } from "node:path"; import type { z } from "zod"; @@ -9,6 +10,74 @@ import { jsonValueSchema } from "../domain/jsonValue.js"; import { firmwareResultSchemas } from "../domain/firmware/firmwareAnalysis.js"; import { hashFirmwareFile, type FirmwareEntry } from "./FirmwareFiles.js"; import type { normalizeUnblobReport } from "./FirmwareReports.js"; +import type { extractionGap } from "../domain/firmware/firmwareAnalysis.js"; + +type ExtractionGap = z.infer; + +/** Static, truthful continuations that use already configured capabilities. */ +const GAP_CONTINUATIONS: Record = { + "missing-extractor-dependency": + "REA installs nothing. Add the named dependency to your own environment and rerun extract_firmware on the same immutable input with a fresh output_directory to re-evaluate the affected region; otherwise it stays unextracted.", + "unknown-chunk-handler": + "If Binwalk is configured with REA_BINWALK_COMMAND, run inspect_firmware_regions on the same input for signature hypotheses. Without configured Binwalk that inspection is unavailable. Alternatively, select a materially narrower range with extract_firmware and a fresh output_directory; the region is not decoded content.", + "encrypted-chunk": + "No static extraction continues past encryption; treat the region as unexamined rather than empty or corrupt.", + "extractor-failure": + "The producer retained this chunk's extraction problem in diagnostics. Review the reported failure, then rerun extract_firmware on the same immutable input with a fresh output_directory; the region is not decoded content.", + "depth-limited": + "Rerun extract_firmware with a higher max_depth (up to 10) and a fresh output_directory to descend into this path.", + "unpublished-entry": + "The entry is retained as metadata; only regular files are published, without executable permission bits.", +}; + +const UNNAMED_DEPENDENCY_CONTINUATION = + "The producer reported a missing extractor dependency without naming it; its identity is unavailable in this evidence. No specific installation or rerun can be prescribed from this report. Retain the gap as unextracted unless additional provider evidence establishes the dependency. REA installs nothing."; + +import { FIRMWARE_LIMITS } from "./FirmwareRelease.js"; + +/** The depth ceiling cannot be raised past the input schema maximum. */ +const MAX_EXTRACTION_DEPTH = 10; + +const depthContinuation = ( + maxDepth: number, + publishedChildPaths: readonly string[], +): string => + maxDepth < MAX_EXTRACTION_DEPTH + ? GAP_CONTINUATIONS["depth-limited"] + : publishedChildPaths.length > 0 + ? `This extraction already ran at the lane's maximum depth. To continue, rerun extract_firmware with path set to any eligible published child below this path and a fresh output directory; each child listed here restarts the depth budget: ${publishedChildPaths.map((path) => JSON.stringify(path)).join(", ")}.` + : `This extraction already ran at the lane's maximum depth, and no published child under this path is both nonempty and within the ${FIRMWARE_LIMITS.inputBytes}-byte firmware input limit, so the depth budget cannot be restarted from this output. Treat the region below this path as unexamined.`; + +// Use a different analyzer on the actual published child; repeating the same +// whole-child extraction with unchanged handlers would reproduce this gap. +const nestedUnknownContinuation = ( + publishedFile: { readonly path: string; readonly size: number } | undefined, +): string => { + const base = + "This unknown chunk was carved from a nested extraction, so its range is relative to the published child named by this gap's input_path, not to the original input."; + return publishedFile !== undefined && + publishedFile.size > 0 && + publishedFile.size <= FIRMWARE_LIMITS.inputBytes + ? `${base} Run inspect_firmware_regions with path set to that published child for signature hypotheses only if Binwalk is configured with REA_BINWALK_COMMAND. This inspection step is unavailable without configured Binwalk; retain the region as unexamined. Repeating the same whole-child extraction with unchanged handlers would not advance analysis.` + : `${base} That published child is not a nonempty file within the ${FIRMWARE_LIMITS.inputBytes}-byte firmware input limit, so inspect_firmware_regions cannot accept it as a fresh input; treat the region as unexamined.`; +}; + +const emptyGap = { + relative_path: null, + handler: null, + provider_id: null, + range: null, + root_file_range: null, + dependencies: [], +}; + +const gapSortKey = (gap: ExtractionGap): string => + [ + gap.input_path, + gap.kind, + gap.provider_id ?? "", + gap.relative_path ?? "", + ].join("\u0000"); /** Publish verified regular files, preserving links as metadata and rolling back failed writes. */ export const publishFirmwareExtraction = async (context: { @@ -16,6 +85,8 @@ export const publishFirmwareExtraction = async (context: { entries: readonly FirmwareEntry[]; normalized: ReturnType; selection: { offset: number; length: number; sha256: string }; + /** Caller-selected extraction depth, for boundary-aware continuations. */ + maxDepth: number; engine: z.infer["engine"]; signal: AbortSignal; retainCleanup: (tree: SafeOutputTree) => void; @@ -93,6 +164,12 @@ export const publishFirmwareExtraction = async (context: { derivations: context.normalized.derivations, depth_limited_paths: context.normalized.depth_limited_paths, diagnostics: context.normalized.diagnostics, + ...extractionGaps( + unpublished, + files, + context.normalized, + context.maxDepth, + ), published_bytes: files.reduce((sum, file) => sum + file.size, 0), }); context.signal.throwIfAborted(); @@ -135,3 +212,239 @@ export const publishFirmwareExtraction = async (context: { throw cause; } }; + +/** Derive explicit recovery rows from retained report facts; no new claims. */ +/** Gap rows for missing-dependency reports: chunk-nested rows keep their owning chunk. */ +const dependencyExtractionGaps = ( + normalized: ReturnType, + continuation: (instruction: string) => string, +): ExtractionGap[] => { + // Chunk-nested dependency reports name the chunk that could not extract; + // task-wide reports (standalone or multi-file) attach to the task. + // One index over the chunk list keeps dependency-row resolution linear. + const chunksByKey = new Map( + normalized.chunks.map((chunk) => [ + JSON.stringify([chunk.input_path, chunk.provider_id]), + chunk, + ]), + ); + return [ + ...normalized.chunk_missing_dependencies.flatMap( + ({ input_path, provider_id, dependencies }): ExtractionGap[] => { + const affected = chunksByKey.get( + JSON.stringify([input_path, provider_id]), + ); + return [ + { + kind: "missing-extractor-dependency", + input_path, + ...emptyGap, + ...(affected === undefined + ? {} + : { + handler: affected.handler, + provider_id, + range: affected.range, + root_file_range: affected.root_file_range, + }), + dependencies: [...dependencies], + continuation_id: continuation( + dependencies.length > 0 + ? GAP_CONTINUATIONS["missing-extractor-dependency"] + : UNNAMED_DEPENDENCY_CONTINUATION, + ), + }, + ]; + }, + ), + ...normalized.failed_extraction_stages + .filter(({ kind }) => kind === "missing-extractor-dependency") + .map( + ({ + input_path, + provider_id, + handler, + dependencies, + }): ExtractionGap => ({ + kind: "missing-extractor-dependency", + input_path, + ...emptyGap, + handler, + provider_id, + dependencies: [...dependencies], + continuation_id: continuation( + dependencies.length > 0 + ? GAP_CONTINUATIONS["missing-extractor-dependency"] + : UNNAMED_DEPENDENCY_CONTINUATION, + ), + }), + ), + // Standalone reports name only the task; do not borrow another stage's id. + ...normalized.missing_dependencies.map( + ({ input_path, dependencies }): ExtractionGap => ({ + kind: "missing-extractor-dependency", + input_path, + ...emptyGap, + dependencies: [...dependencies], + continuation_id: continuation( + dependencies.length > 0 + ? GAP_CONTINUATIONS["missing-extractor-dependency"] + : UNNAMED_DEPENDENCY_CONTINUATION, + ), + }), + ), + ]; +}; + +const extractionGaps = ( + unpublished: z.infer< + typeof firmwareResultSchemas.extract_firmware + >["unpublished_entries"], + files: readonly z.infer< + typeof firmwareResultSchemas.extract_firmware + >["files"][number][], + normalized: ReturnType, + maxDepth: number, +) => { + // Intern before row construction: N gaps retain N identities, while K unique + // recovery instructions consume prose only K times in the public result. + const instructions = new Map(); + const continuation = (instruction: string): string => { + const known = instructions.get(instruction); + if (known !== undefined) return known; + const id = createHash("sha256").update(instruction).digest("hex"); + instructions.set(instruction, id); + return id; + }; + const dependencyGaps = dependencyExtractionGaps(normalized, continuation); + const publishedFiles = new Map( + files.map((file) => [`$output/${file.relative_path}`, file]), + ); + const nestedContinuationId = (inputPath: string): string => { + return continuation( + nestedUnknownContinuation(publishedFiles.get(inputPath)), + ); + }; + const chunkGaps = ( + kind: ExtractionGap["kind"], + matches: (chunk: (typeof normalized.chunks)[number]) => boolean, + ): ExtractionGap[] => + normalized.chunks.filter(matches).map((chunk) => ({ + kind, + input_path: chunk.input_path, + ...emptyGap, + handler: chunk.handler, + provider_id: chunk.provider_id, + range: chunk.range, + root_file_range: chunk.root_file_range, + dependencies: [], + continuation_id: + kind === "unknown-chunk-handler" && + chunk.input_path.startsWith("$output/") + ? nestedContinuationId(chunk.input_path) + : continuation(GAP_CONTINUATIONS[kind]), + })); + // Chunks whose failure a dependency row already covers stay out of the + // generic extractor-failure emission. + const dependencyChunkKeys = new Set( + normalized.chunk_missing_dependencies.map(({ input_path, provider_id }) => + JSON.stringify([input_path, provider_id]), + ), + ); + // A depth ceiling can only be re-rooted through a published child that a + // fresh firmware input could actually accept; the index is only needed + // when a depth row will actually consult it. + const ceilingReached = + maxDepth >= MAX_EXTRACTION_DEPTH && + normalized.depth_limited_paths.length > 0; + const reRootablePaths = new Map>(); + if (ceilingReached) + for (const [filePath, file] of [...publishedFiles].sort( + ([left], [right]) => (left < right ? -1 : left > right ? 1 : 0), + )) { + const size = file.size; + if (size <= 0 || size > FIRMWARE_LIMITS.inputBytes) continue; + // The published file itself can re-root a fresh run; so can every + // directory containing it. + const segments = filePath.split("/"); + while (segments.length > 1) { + const ancestor = segments.join("/"); + const childPaths = reRootablePaths.get(ancestor) ?? new Set(); + childPaths.add(file.path); + reRootablePaths.set(ancestor, childPaths); + segments.pop(); + } + } + const depthContinuationIds = new Map(); + const depthContinuationId = (inputPath: string): string => { + const existing = depthContinuationIds.get(inputPath); + if (existing !== undefined) return existing; + const id = continuation( + depthContinuation( + maxDepth, + [...(reRootablePaths.get(inputPath) ?? [])].sort(), + ), + ); + depthContinuationIds.set(inputPath, id); + return id; + }; + const gaps: ExtractionGap[] = [ + ...dependencyGaps, + ...normalized.failed_extraction_stages + .filter(({ kind }) => kind === "extractor-failure") + .map(({ input_path, provider_id, handler }): ExtractionGap => ({ + kind: "extractor-failure", + input_path, + ...emptyGap, + handler, + provider_id, + dependencies: [], + continuation_id: continuation( + "The producer retained this extraction stage's failure in diagnostics. Review the stage report and correct the reported cause before rerunning extract_firmware on the same immutable input with a fresh output_directory; this report establishes no byte-region ownership.", + ), + })), + ...chunkGaps("unknown-chunk-handler", (chunk) => chunk.handler === null), + ...chunkGaps("encrypted-chunk", (chunk) => chunk.encrypted === true), + ...chunkGaps( + "extractor-failure", + (chunk) => + chunk.extraction_failed === true && + chunk.handler !== null && + chunk.encrypted !== true && + // The chunk's own dependency row already carries its recovery; a + // task-wide dependency names no region and replaces nothing. + !dependencyChunkKeys.has( + JSON.stringify([chunk.input_path, chunk.provider_id]), + ), + ), + ...normalized.depth_limited_paths.map((input_path): ExtractionGap => ({ + kind: "depth-limited", + input_path, + ...emptyGap, + dependencies: [], + continuation_id: depthContinuationId(input_path), + })), + ...unpublished.map((entry): ExtractionGap => ({ + kind: "unpublished-entry", + input_path: `$output/${entry.relative_path}`, + ...emptyGap, + relative_path: entry.relative_path, + dependencies: [], + continuation_id: continuation(GAP_CONTINUATIONS["unpublished-entry"]), + })), + ]; + gaps.sort((left, right) => + gapSortKey(left) < gapSortKey(right) + ? -1 + : gapSortKey(left) > gapSortKey(right) + ? 1 + : 0, + ); + return { + extraction_gaps: gaps, + extraction_continuations: [...instructions].map(([instruction, id]) => ({ + id, + instruction, + })), + }; +}; diff --git a/src/firmware/FirmwareReports.test.ts b/src/firmware/FirmwareReports.test.ts index b924180ba2..1a491d5073 100644 --- a/src/firmware/FirmwareReports.test.ts +++ b/src/firmware/FirmwareReports.test.ts @@ -42,7 +42,28 @@ const root = () => ({ subtasks: [], }); -describe("firmware producer boundaries", () => { +/** A chunk whose extraction failed on a missing dependency, when asked for it. */ +const nestedDependencyChunk = (id: string, offset: number) => ({ + __typename__: "ChunkReport", + id, + start_offset: offset, + end_offset: offset + 10, + size: 10, + handler_name: `handler-${id}`, + is_encrypted: false, + extraction_reports: + id === "gzip-id" + ? [ + { + __typename__: "ExtractorDependencyNotFoundReport", + severity: "ERROR", + dependencies: ["sasquatch"], + }, + ] + : [], +}); + +describe("firmware producer chunk observations", () => { it("retains unknown chunks and their selected original-file range", () => { const normalized = normalizeUnblobReport([root()], context); expect(normalized.coverage).toBe("partial"); @@ -51,6 +72,412 @@ describe("firmware producer boundaries", () => { range: { offset: 0, length: 20 }, root_file_range: { offset: 64, length: 20 }, }); + expect(normalized.missing_dependencies).toEqual([]); + }); +}); + +describe("firmware producer dependency ownership", () => { + it("derives missing extractor dependencies from chunk and multi-file reports", () => { + const record = root(); + const normalized = normalizeUnblobReport( + [ + { + ...record, + reports: record.reports.filter( + (item) => item.__typename__ !== "UnknownChunkReport", + ), + subtasks: [ + { + path: `${context.outputRoot}/rootfs`, + depth: 1, + blob_id: "sq-id", + is_multi_file: false, + }, + ], + }, + { + task: { + path: `${context.outputRoot}/rootfs`, + depth: 1, + blob_id: "sq-id", + is_multi_file: false, + }, + reports: [ + { + __typename__: "StatReport", + path: `${context.outputRoot}/rootfs`, + size: 0, + is_file: false, + is_dir: true, + is_link: false, + link_target: null, + }, + { + __typename__: "MultiFileReport", + id: "sq-id", + handler_name: "squashfs", + name: "rootfs", + paths: [], + extraction_reports: [ + { + __typename__: "ExtractorDependencyNotFoundReport", + severity: "ERROR", + dependencies: ["sasquatch"], + }, + ], + }, + ], + subtasks: [], + }, + ], + context, + ); + expect(normalized.missing_dependencies).toEqual([]); + expect(normalized.failed_extraction_stages).toEqual([ + { + input_path: "$output/rootfs", + provider_id: "sq-id", + handler: "squashfs", + dependencies: ["sasquatch"], + kind: "missing-extractor-dependency", + }, + ]); + expect(normalized.chunk_missing_dependencies).toEqual([]); + }); + it("orders chunk dependencies totally across provider ids", () => { + const record = root(); + const chunkWithDependency = (id: string, offset: number) => ({ + __typename__: "ChunkReport", + id, + start_offset: offset, + end_offset: offset + 10, + size: 10, + handler_name: `handler-${id}`, + is_encrypted: false, + extraction_reports: [ + { + __typename__: "ExtractorDependencyNotFoundReport", + severity: "ERROR", + dependencies: [`${id}-tool`], + }, + ], + }); + const normalized = normalizeUnblobReport( + [ + { + ...record, + reports: [ + ...record.reports.filter( + (item) => item.__typename__ !== "UnknownChunkReport", + ), + chunkWithDependency("b-id", 0), + chunkWithDependency("a-id", 10), + ], + }, + ], + context, + ); + expect(normalized.chunk_missing_dependencies).toEqual([ + { + input_path: "$input", + provider_id: "a-id", + dependencies: ["a-id-tool"], + }, + { + input_path: "$input", + provider_id: "b-id", + dependencies: ["b-id-tool"], + }, + ]); + }); + it("keeps chunk-nested dependency reports bound to their owning chunk", () => { + const record = root(); + const normalized = normalizeUnblobReport( + [ + { + ...record, + reports: [ + ...record.reports.filter( + (item) => item.__typename__ !== "UnknownChunkReport", + ), + nestedDependencyChunk("gzip-id", 0), + nestedDependencyChunk("other-id", 10), + ], + }, + ], + context, + ); + expect(normalized.missing_dependencies).toEqual([]); + expect(normalized.chunk_missing_dependencies).toEqual([ + { + input_path: "$input", + provider_id: "gzip-id", + dependencies: ["sasquatch"], + }, + ]); + }); +}); + +describe("firmware dependency failures with unavailable identities", () => { + it.each([{ dependencies: [] }, { dependencies: ["", ""] }])( + "retains unnamed chunk dependency ownership ($dependencies)", + ({ dependencies }) => { + const record = root(); + const normalized = normalizeUnblobReport( + [ + { + ...record, + reports: [ + ...record.reports.filter( + (item) => item.__typename__ !== "UnknownChunkReport", + ), + { + ...nestedDependencyChunk("gzip-id", 0), + extraction_reports: [ + { + __typename__: "ExtractorDependencyNotFoundReport", + severity: "ERROR", + dependencies, + }, + ], + }, + ], + }, + ], + context, + ); + expect(normalized.missing_dependencies).toEqual([]); + expect(normalized.chunk_missing_dependencies).toEqual([ + { + input_path: "$input", + provider_id: "gzip-id", + dependencies: [], + }, + ]); + }, + ); + it("retains the missing-dependency reason on an unnamed multi-file stage", () => { + const record = root(); + const normalized = normalizeUnblobReport( + [ + { + ...record, + reports: [ + ...record.reports, + { + __typename__: "MultiFileReport", + id: "multi-id", + handler_name: "multipart-firmware", + name: "parts", + paths: [], + extraction_reports: [ + { + __typename__: "ExtractorDependencyNotFoundReport", + severity: "ERROR", + dependencies: [], + }, + ], + }, + ], + }, + ], + context, + ); + expect(normalized.failed_extraction_stages).toEqual([ + { + input_path: "$input", + provider_id: "multi-id", + handler: "multipart-firmware", + dependencies: [], + kind: "missing-extractor-dependency", + }, + ]); + }); + it("retains named and unnamed dependency reports on one multi-file stage", () => { + const record = root(); + const normalized = normalizeUnblobReport( + [ + { + ...record, + reports: [ + ...record.reports, + { + __typename__: "MultiFileReport", + id: "multi-id", + handler_name: "multipart-firmware", + name: "parts", + paths: [], + extraction_reports: [ + { + __typename__: "ExtractorDependencyNotFoundReport", + severity: "ERROR", + dependencies: ["sasquatch"], + }, + { + __typename__: "ExtractorDependencyNotFoundReport", + severity: "ERROR", + dependencies: [], + }, + ], + }, + ], + }, + ], + context, + ); + expect(normalized.failed_extraction_stages).toEqual([ + { + input_path: "$input", + provider_id: "multi-id", + handler: "multipart-firmware", + dependencies: ["sasquatch"], + kind: "missing-extractor-dependency", + }, + { + input_path: "$input", + provider_id: "multi-id", + handler: "multipart-firmware", + dependencies: [], + kind: "missing-extractor-dependency", + }, + ]); + }); +}); + +describe("firmware producer chunk failures", () => { + it("marks chunks whose extraction reported problems", () => { + const record = root(); + const failing = { + __typename__: "ChunkReport", + id: "zip-id", + start_offset: 0, + end_offset: 20, + size: 20, + handler_name: "zip", + is_encrypted: false, + extraction_reports: [ + { __typename__: "SomeProducerFailure", severity: "ERROR" }, + ], + }; + const normalized = normalizeUnblobReport( + [ + { + ...record, + reports: [ + ...record.reports.filter( + (item) => item.__typename__ !== "UnknownChunkReport", + ), + failing, + ], + }, + ], + context, + ); + expect(normalized.chunks).toHaveLength(1); + expect(normalized.chunks[0]).toMatchObject({ + handler: "zip", + extraction_failed: true, + }); + expect(normalized.coverage).toBe("partial"); + }); +}); + +describe("firmware task dependency failures", () => { + it.each([{ dependencies: [] }, { dependencies: ["", ""] }])( + "retains unnamed dependency failures ($dependencies)", + ({ dependencies }) => { + const record = root(); + const normalized = normalizeUnblobReport( + [ + { + ...record, + reports: [ + ...record.reports, + { + __typename__: "ExtractorDependencyNotFoundReport", + severity: "ERROR", + dependencies, + }, + ], + }, + ], + context, + ); + expect(normalized.missing_dependencies).toEqual([ + { input_path: "$input", dependencies: [] }, + ]); + expect(normalized.diagnostics).toHaveLength(1); + }, + ); + it("retains named and unnamed task dependency reports together", () => { + const record = root(); + const normalized = normalizeUnblobReport( + [ + { + ...record, + reports: [ + ...record.reports, + { + __typename__: "ExtractorDependencyNotFoundReport", + severity: "ERROR", + dependencies: ["sasquatch"], + }, + { + __typename__: "ExtractorDependencyNotFoundReport", + severity: "ERROR", + dependencies: [], + }, + ], + }, + ], + context, + ); + expect(normalized.missing_dependencies).toEqual([ + { input_path: "$input", dependencies: ["sasquatch"] }, + { input_path: "$input", dependencies: [] }, + ]); + }); +}); + +describe("firmware chunk dependency failures", () => { + it("retains named and unnamed dependency reports on one chunk", () => { + const record = root(); + const normalized = normalizeUnblobReport( + [ + { + ...record, + reports: [ + ...record.reports.filter( + (item) => item.__typename__ !== "UnknownChunkReport", + ), + { + ...nestedDependencyChunk("gzip-id", 0), + extraction_reports: [ + { + __typename__: "ExtractorDependencyNotFoundReport", + severity: "ERROR", + dependencies: ["sasquatch"], + }, + { + __typename__: "ExtractorDependencyNotFoundReport", + severity: "ERROR", + dependencies: [], + }, + ], + }, + ], + }, + ], + context, + ); + expect(normalized.chunk_missing_dependencies).toEqual([ + { + input_path: "$input", + provider_id: "gzip-id", + dependencies: ["sasquatch"], + }, + { input_path: "$input", provider_id: "gzip-id", dependencies: [] }, + ]); }); it("preserves unfamiliar report types as explicit diagnostics", () => { const record = root(); @@ -237,6 +664,77 @@ describe("Unblob task derivation identities", () => { report: multiFileReport, }, ]); + expect(normalized.failed_extraction_stages).toEqual([ + { + input_path: "$input", + provider_id: "multi-id", + handler: "multipart-firmware", + dependencies: ["sasquatch"], + kind: "missing-extractor-dependency", + }, + ]); + }); +}); + +describe("failed multi-file extraction stages", () => { + it("records a failed multi-file stage for a recovery row without chunks", () => { + const outputPath = `${context.outputRoot}/multi`; + const childTask = { + path: outputPath, + depth: 1, + blob_id: "multi-id", + is_multi_file: true, + }; + const stageFailure = { + __typename__: "ExtractorExecutionErrorReport", + severity: "ERROR", + handler: "multipart-firmware", + }; + const normalized = normalizeUnblobReport( + [ + { + ...root(), + reports: root().reports.filter( + (item) => item.__typename__ !== "UnknownChunkReport", + ), + subtasks: [childTask], + }, + { + task: childTask, + reports: [ + { + __typename__: "StatReport", + path: outputPath, + size: 0, + is_dir: true, + is_file: false, + is_link: false, + link_target: null, + }, + { + __typename__: "MultiFileReport", + id: "multi-id", + handler_name: "multipart-firmware", + name: "rootfs", + paths: [`${outputPath}/part-a`], + extraction_reports: [stageFailure], + }, + ], + subtasks: [], + }, + ], + context, + ); + expect(normalized.failed_extraction_stages).toEqual([ + { + input_path: "$output/multi", + provider_id: "multi-id", + handler: "multipart-firmware", + dependencies: [], + kind: "extractor-failure", + }, + ]); + expect(normalized.missing_dependencies).toEqual([]); }); }); diff --git a/src/firmware/FirmwareReports.ts b/src/firmware/FirmwareReports.ts index 90857de848..f06129aa28 100644 --- a/src/firmware/FirmwareReports.ts +++ b/src/firmware/FirmwareReports.ts @@ -2,6 +2,7 @@ import { z } from "zod"; import { AnalysisOutputError } from "../domain/analysisErrorCore.js"; import { jsonValueSchema, type JsonValue } from "../domain/jsonValue.js"; import { digestSchema } from "../domain/digests.js"; +import { uniqueSorted } from "../domain/canonicalOrdering.js"; import { firmwareLogicalPath } from "./FirmwareFiles.js"; const integer = z.number().int().min(0).max(Number.MAX_SAFE_INTEGER); @@ -51,6 +52,15 @@ const multiFile = z.object({ extraction_reports: z.array(jsonValueSchema), }); const hashes = z.object({ sha256: digestSchema }); +/** + * The provider's structured missing-dependency report. Non-strict on purpose: + * unfamiliar extra fields stay in retained diagnostics, only the dependency + * names are derived here. + */ +const missingDependency = z.object({ + __typename__: z.literal("ExtractorDependencyNotFoundReport"), + dependencies: z.array(z.string()), +}); const region = z.object({ id: z.string(), start_offset: integer, @@ -79,7 +89,9 @@ const dependencyFailure = z.object({ /** Collect the named extractors from any dependency report nested in one value. */ const missingDependencies = (value: unknown): string[] => { - const found: string[] = []; + // Producer-controlled name volume must not make deduplication quadratic; + // a Set keeps collection linear and uniqueSorted restores canonical order. + const found = new Set(); const visit = (node: unknown): void => { if (Array.isArray(node)) { for (const item of node) visit(item); @@ -88,14 +100,13 @@ const missingDependencies = (value: unknown): string[] => { if (typeof node !== "object" || node === null) return; const failure = dependencyFailure.safeParse(node); if (failure.success) { - for (const dependency of failure.data.dependencies) - if (!found.includes(dependency)) found.push(dependency); + for (const dependency of failure.data.dependencies) found.add(dependency); return; } for (const child of Object.values(node)) visit(child); }; visit(value); - return found; + return [...found]; }; /** @@ -194,6 +205,13 @@ export const normalizeUnblobReport = ( >(); const sizes = new Map(); const depths: string[] = []; + const failedExtractionStages: { + input_path: string; + provider_id: string | null; + handler: string | null; + dependencies: readonly string[]; + kind: "missing-extractor-dependency" | "extractor-failure"; + }[] = []; const diagnostics: JsonValue[] = []; const blobOrigins = new Map(); const addBlobOrigin = ( @@ -216,8 +234,74 @@ export const normalizeUnblobReport = ( handler: string | null; encrypted: boolean | null; root_file_range: { offset: number; length: number } | null; + extraction_failed: boolean; }[] = []; const seen = new Set(); + const taskFailures = new Set(); + type DependencyFacts = { readonly names: Set; unnamed: boolean }; + const missingDependencies = new Map(); + /** Dependencies reported by one chunk's extraction, keyed with its chunk id. */ + const chunkMissingDependencies = new Map(); + const dependencyNames = ( + reports: readonly unknown[], + ): DependencyFacts | null => { + const names = new Set(); + let unnamed = false; + let reported = false; + for (const item of reports) { + const parsed = missingDependency.safeParse(item); + if (!parsed.success) continue; + reported = true; + if ( + parsed.data.dependencies.length === 0 || + parsed.data.dependencies.includes("") + ) + unnamed = true; + for (const name of parsed.data.dependencies) + if (name !== "") names.add(name); + } + return reported ? { names, unnamed } : null; + }; + const dependencyRows = ( + facts: DependencyFacts, + ): readonly { dependencies: readonly string[] }[] => [ + ...(facts.names.size > 0 + ? [{ dependencies: uniqueSorted([...facts.names]) }] + : []), + ...(facts.unnamed ? [{ dependencies: [] }] : []), + ]; + const noteMissingDependencies = ( + path: string, + reports: readonly unknown[], + ): void => { + const names = dependencyNames(reports); + // null means no accepted failure; [] means a definite dependency failure + // whose executable identity the producer left unavailable. + if (names === null) return; + const known = missingDependencies.get(path) ?? { + names: new Set(), + unnamed: false, + }; + for (const name of names.names) known.names.add(name); + known.unnamed ||= names.unnamed; + missingDependencies.set(path, known); + }; + const noteChunkMissingDependencies = ( + path: string, + chunkId: string, + reports: readonly unknown[], + ): void => { + const names = dependencyNames(reports); + if (names === null) return; + const key = JSON.stringify([path, chunkId]); + const known = chunkMissingDependencies.get(key) ?? { + names: new Set(), + unnamed: false, + }; + for (const name of names.names) known.names.add(name); + known.unnamed ||= names.unnamed; + chunkMissingDependencies.set(key, known); + }; for (const record of records) { const path = logical(record.task.path); if (seen.has(path)) @@ -273,6 +357,20 @@ export const normalizeUnblobReport = ( diagnostics.push( jsonValueSchema.parse({ input_path: path, report: item }), ); + noteMissingDependencies(path, [item]); + if ( + item.__typename__ === "ExtractorExecutionErrorReport" && + !taskFailures.has(path) + ) { + taskFailures.add(path); + failedExtractionStages.push({ + input_path: path, + provider_id: null, + handler: null, + dependencies: [], + kind: "extractor-failure", + }); + } if ( item.__typename__ === "ChunkReport" || item.__typename__ === "UnknownChunkReport" @@ -297,8 +395,10 @@ export const normalizeUnblobReport = ( }) .parse(item) : null; + const extractionFailed = + detail !== null && detail.extraction_reports.length > 0; if (detail !== null) - for (const problem of detail.extraction_reports) + for (const problem of detail.extraction_reports) { diagnostics.push( extractionDiagnostic( path, @@ -306,6 +406,9 @@ export const normalizeUnblobReport = ( jsonValueSchema.parse(problem), ), ); + // Nested under one chunk: keep the owning chunk, not the whole task. + noteChunkMissingDependencies(path, parsed.id, [problem]); + } addBlobOrigin(path, parsed.id, { handler: detail?.handler_name ?? null, range: { offset: parsed.start_offset, length: parsed.size }, @@ -316,6 +419,7 @@ export const normalizeUnblobReport = ( range: { offset: parsed.start_offset, length: parsed.size }, handler: detail?.handler_name ?? null, encrypted: detail?.is_encrypted ?? null, + extraction_failed: extractionFailed, root_file_range: path === "$input" ? { @@ -331,7 +435,7 @@ export const normalizeUnblobReport = ( handler: parsed.handler_name, range: null, }); - if (parsed.extraction_reports.length > 0) + if (parsed.extraction_reports.length > 0) { diagnostics.push( extractionDiagnostic( path, @@ -339,6 +443,28 @@ export const normalizeUnblobReport = ( jsonValueSchema.parse(item), ), ); + // The report owns its failure and dependencies. A produced subtask + // is a derivation, not a new owner for the parent's report. + const dependencies = dependencyNames(parsed.extraction_reports); + const rows = + dependencies === null + ? [{ dependencies: [] as readonly string[] }] + : dependencyRows(dependencies); + for (const row of rows) + failedExtractionStages.push({ + input_path: path, + provider_id: parsed.id, + handler: parsed.handler_name, + dependencies: + dependencies === null + ? [] + : uniqueSorted([...row.dependencies]), + kind: + dependencies === null + ? "extractor-failure" + : "missing-extractor-dependency", + }); + } } } } @@ -403,7 +529,48 @@ export const normalizeUnblobReport = ( chunks, derivations, depth_limited_paths: depths.sort(), + failed_extraction_stages: failedExtractionStages.sort((left, right) => + `${left.input_path}\u0000${left.provider_id ?? ""}` < + `${right.input_path}\u0000${right.provider_id ?? ""}` + ? -1 + : `${left.input_path}\u0000${left.provider_id ?? ""}` > + `${right.input_path}\u0000${right.provider_id ?? ""}` + ? 1 + : 0, + ), diagnostics, + missing_dependencies: [...missingDependencies.entries()] + .flatMap(([input_path, facts]) => + dependencyRows(facts).map(({ dependencies }) => ({ + input_path, + dependencies, + })), + ) + .sort((left, right) => + left.input_path < right.input_path + ? -1 + : left.input_path > right.input_path + ? 1 + : 0, + ), + chunk_missing_dependencies: [...chunkMissingDependencies.entries()] + .flatMap(([key, facts]) => { + const [input_path, provider_id] = JSON.parse(key) as [string, string]; + return dependencyRows(facts).map(({ dependencies }) => ({ + input_path, + provider_id, + dependencies, + })); + }) + .sort((left, right) => + `${left.input_path}\u0000${left.provider_id}` < + `${right.input_path}\u0000${right.provider_id}` + ? -1 + : `${left.input_path}\u0000${left.provider_id}` > + `${right.input_path}\u0000${right.provider_id}` + ? 1 + : 0, + ), coverage: depths.length > 0 || diagnostics.length > 0 || diff --git a/src/flutter/DartElf.test.ts b/src/flutter/DartElf.test.ts index ff2724695b..da1557138a 100644 --- a/src/flutter/DartElf.test.ts +++ b/src/flutter/DartElf.test.ts @@ -3,6 +3,7 @@ import { describe, expect, it } from "vitest"; import { readElfSymbols, readSnapshotHeader } from "./DartElf.js"; import { REFERENCE_HASH, + buildElf32Image, buildElf64Image, buildStrippedElf64Image, snapshotSection, @@ -299,6 +300,106 @@ describe("ELF symbol-table ownership", () => { }); }); +/** Section-header field positions for each ELF class. */ +const ELF_CLASSES = { + elf64: { + build: buildElf64Image, + word: { + read: (image: Buffer, at: number) => Number(image.readBigUInt64LE(at)), + write: (image: Buffer, value: number, at: number) => + image.writeBigUInt64LE(BigInt(value), at), + }, + shoff: 0x28, + shent: 64, + flags: 8, + address: 0x10, + offset: 0x18, + }, + elf32: { + build: buildElf32Image, + word: { + read: (image: Buffer, at: number) => image.readUInt32LE(at), + write: (image: Buffer, value: number, at: number) => + image.writeUInt32LE(value, at), + }, + shoff: 0x20, + shent: 40, + flags: 8, + address: 0x0c, + offset: 0x10, + }, +} as const; + +describe.each(Object.entries(ELF_CLASSES))( + "allocated %s symbol tables", + (_, layout) => { + const { word } = layout; + /** Snapshot sections, with dynamic tables allocated at their offsets. */ + const allocatedTables = (): { + image: Buffer; + headers: number[]; + tables: number[]; + } => { + const elf = layout.build([ + { name: "_kDartVmSnapshotData", value: 0x400, size: 52 }, + ]); + const section = snapshotSection(GALLERY_HASH); + const image = Buffer.concat([ + elf, + Buffer.alloc(0x400 - elf.length), + section, + ]); + const shoff = word.read(image, layout.shoff); + const headers = [shoff + layout.shent, shoff + 2 * layout.shent]; + const tables = headers.map((header) => { + const offset = word.read(image, header + layout.offset); + word.write(image, 2, header + layout.flags); + word.write(image, offset, header + layout.address); + return offset; + }); + return { image, headers, tables }; + }; + + it("accepts tables whose mapping matches their sections", async () => { + const { image } = allocatedTables(); + expect(await readElfSymbols(image, ["_kDartVmSnapshotData"])).toEqual({ + symbols: [{ name: "_kDartVmSnapshotData", offset: 0x400, size: 52 }], + failures: [], + }); + }); + + it.each([ + [0, "Dynamic symbol table has conflicting PT_LOAD mappings"], + [1, "Dynamic symbol string table has conflicting PT_LOAD mappings"], + ])( + "refuses table %i under a conflicting PT_LOAD interpretation", + async (table, reason) => { + const { image, tables } = allocatedTables(); + const conflict = appendElfRanges(image, "program", [ + { type: 1, address: tables[table]!, offset: 0x300, size: 4 }, + ]); + expect( + await readElfSymbols(conflict, ["_kDartVmSnapshotData"]), + ).toEqual({ symbols: [], failures: [{ reason }] }); + }, + ); + + it("refuses a table whose section offset disagrees with its mapping", async () => { + const { image, headers } = allocatedTables(); + word.write(image, 0x300, headers[0]! + layout.address); + expect(await readElfSymbols(image, ["_kDartVmSnapshotData"])).toEqual({ + symbols: [], + failures: [ + { + reason: + "Dynamic symbol table section offset disagrees with its PT_LOAD mapping", + }, + ], + }); + }); + }, +); + describe("readSnapshotHeader", () => { it("reads a synthesized data header", async () => { const header = readSnapshotHeader(snapshotSection(GALLERY_HASH), 0); diff --git a/src/flutter/DartElf.ts b/src/flutter/DartElf.ts index b1c6fc584f..981f689a62 100644 --- a/src/flutter/DartElf.ts +++ b/src/flutter/DartElf.ts @@ -59,6 +59,7 @@ export interface ElfParseFailure { } const SHT_DYNSYM = 11; +const SHF_ALLOC = 2; const PT_LOAD = 1; const PT_DYNAMIC = 2; /** Position-independent executable images carry PT_DYNAMIC. */ @@ -225,6 +226,28 @@ export const readElfSymbols = async ( stringOffset: number; stringSize: number; }[] = []; + /** + * The loader reads an allocated section through PT_LOAD, so its section + * offset must be the single file interpretation of its address range. + */ + const sectionMappingFailure = ( + header: number, + offset: number, + size: number, + ): string | undefined => { + const flags = is64 ? reader.u64(header + 8) : reader.u32(header + 8); + const address = is64 + ? reader.u64(header + 0x10) + : reader.u32(header + 0x0c); + if (flags === null || address === null) + return "has an unrepresentable section header"; + if ((flags & SHF_ALLOC) === 0) return undefined; + const mapped = toOffset(address, size); + if (!mapped.ok) return mapped.error.reason; + return mapped.value === offset + ? undefined + : "section offset disagrees with its PT_LOAD mapping"; + }; if (!dynamicFallback) { for (let index = 0; index < e_shnum; index += 1) { if (index % 4096 === 0) { @@ -269,6 +292,16 @@ export const readElfSymbols = async ( return failure( "Dynamic symbol strings are out of bounds or not a string table", ); + const symbolMapping = sectionMappingFailure(header, offset, size); + if (symbolMapping !== undefined) + return failure(`Dynamic symbol table ${symbolMapping}`); + const stringMapping = sectionMappingFailure( + stringHeader, + stringOffset, + stringSize, + ); + if (stringMapping !== undefined) + return failure(`Dynamic symbol string table ${stringMapping}`); tables.push({ offset, count: size / entrySize, diff --git a/src/flutter/FlutterBuild.fixture.ts b/src/flutter/FlutterBuild.fixture.ts index 27c43c3bb2..65ce3323da 100644 --- a/src/flutter/FlutterBuild.fixture.ts +++ b/src/flutter/FlutterBuild.fixture.ts @@ -115,6 +115,63 @@ export const buildElf64Image = ( return image; }; +/** The ELF32 (ARM) counterpart of {@link buildElf64Image}, with the same layout roles. */ +export const buildElf32Image = ( + symbols: ReadonlyArray<{ name: string; value: number; size: number }>, +): Buffer => { + const ELF_HEADER = 52; + const PHENT = 32; + const SHENT = 40; + const SYMENT = 16; + const phoff = ELF_HEADER; + const shoff = phoff + PHENT; + const dynsymOff = shoff + 3 * SHENT + 16; + const dynstrOff = dynsymOff + symbols.length * SYMENT; + const image = Buffer.alloc(dynstrOff + 256); + image.writeUInt32LE(0x464c457f, 0); + image[4] = 1; // ELFCLASS32 + image[5] = 1; // little-endian + image[6] = 1; + image.writeUInt16LE(3, 0x10); + image.writeUInt16LE(0x0028, 0x12); // e_machine ARM + image.writeUInt32LE(1, 0x14); + image.writeUInt32LE(phoff, 0x1c); + image.writeUInt32LE(shoff, 0x20); + image.writeUInt16LE(ELF_HEADER, 0x28); + image.writeUInt16LE(PHENT, 0x2a); + image.writeUInt16LE(1, 0x2c); + image.writeUInt16LE(SHENT, 0x2e); + image.writeUInt16LE(3, 0x30); // null, dynsym, dynstr + // One identity PT_LOAD covering the materialized payload. + image.writeUInt32LE(1, phoff); + image.writeUInt32LE( + Math.max(image.length, ...symbols.map(({ value, size }) => value + size)), + phoff + 0x10, + ); + const dyn = shoff + SHENT; + image.writeUInt32LE(11, dyn + 4); // SHT_DYNSYM + image.writeUInt32LE(dynsymOff, dyn + 0x10); + image.writeUInt32LE(symbols.length * SYMENT, dyn + 0x14); + image.writeUInt32LE(2, dyn + 0x18); // sh_link -> strtab + image.writeUInt32LE(SYMENT, dyn + 0x24); + const str = dyn + SHENT; + image.writeUInt32LE(3, str + 4); // SHT_STRTAB + image.writeUInt32LE(dynstrOff, str + 0x10); + image.writeUInt32LE(256, str + 0x14); + let strCursor = dynstrOff + 1; + symbols.forEach((symbol, index) => { + image.write(symbol.name, strCursor, "latin1"); + const base = dynsymOff + index * SYMENT; + image.writeUInt32LE(strCursor - dynstrOff, base); + image.writeUInt32LE(symbol.value, base + 4); + image.writeUInt32LE(symbol.size, base + 8); + image[base + 0xc] = 0x10; // GLOBAL + image.writeUInt16LE(1, base + 0xe); // Defined export. + strCursor += symbol.name.length + 1; + }); + return image; +}; + /** * Build a section-stripped ELF64 image (e_shnum 0, ET_DYN) whose exported * symbols are reachable only through the PT_DYNAMIC tag-value pairs, the diff --git a/src/frida/FridaInstrumentationManager.ts b/src/frida/FridaInstrumentationManager.ts index b93cffae5d..7ef5d115f9 100644 --- a/src/frida/FridaInstrumentationManager.ts +++ b/src/frida/FridaInstrumentationManager.ts @@ -92,6 +92,7 @@ interface ManagedSession extends ManagedRemoteConnection { resumeAfterDetachPending: boolean; capturedBytes: number; messagesTruncated: boolean; + readonly truncatedScripts: Set; } export interface FridaInstrumentationManagerOptions { @@ -115,6 +116,7 @@ export class FridaInstrumentationManager implements FridaInstrumentationPort { readonly #remoteQueues = new Map>(); readonly #deviceManagerLoader: () => Promise; #deviceManager: DeviceManager | undefined; + #deviceManagerLoading: Promise | undefined; #availabilityFailure: OptionalProviderLoadFailure | undefined; readonly #operations = new Set>(); #closing = false; @@ -273,9 +275,14 @@ export class FridaInstrumentationManager implements FridaInstrumentationPort { let pid: number; let target: string; if (input.mode === "attach") { - const process = await device.getProcessByPid(input.pid); - pid = process.pid; - target = `${process.name} (pid ${process.pid})`; + pid = input.pid; + target = `pid ${pid}`; + try { + const process = await device.getProcessByPid(pid); + target = `${process.name} (pid ${pid})`; + } catch { + // Frida can attach to processes absent from its enumeration snapshot. + } } else { pid = await device.spawn( input.program, @@ -307,10 +314,15 @@ export class FridaInstrumentationManager implements FridaInstrumentationPort { resumeAfterDetachPending: false, capturedBytes: 0, messagesTruncated: false, + truncatedScripts: new Set(), }; session.detached.connect(() => { managed.state = "detached"; }); + // A target can exit while metadata lookup/attach is pending, before + // Frida installs this listener. Subscribe first, then reconcile the + // current provider state so an already-emitted detach is not lost. + if (session.isDetached()) managed.state = "detached"; this.#sessions.set(id, managed); return ok(sessionObservation(managed)); } catch (cause: unknown) { @@ -432,7 +444,7 @@ export class FridaInstrumentationManager implements FridaInstrumentationPort { sourcePath: source.sourceKind === "file" ? source.path : null, sourceSha256, messages: managed.messagesByScript.get(scriptId)?.toArray() ?? [], - messagesTruncated: managed.messagesTruncated, + messagesTruncated: managed.truncatedScripts.has(scriptId), }; if (signal?.aborted) { const cleanupReason = await this.#releaseScript(managed, entry); @@ -598,6 +610,7 @@ export class FridaInstrumentationManager implements FridaInstrumentationPort { } managed.scripts.delete(entry.id); managed.messagesByScript.delete(entry.id); + managed.truncatedScripts.delete(entry.id); return undefined; } @@ -916,7 +929,7 @@ export class FridaInstrumentationManager implements FridaInstrumentationPort { bytes: Buffer.byteLength('{"payload_unavailable":true}'), }; } - if (payloadResult.truncated) managed.messagesTruncated = true; + if (payloadResult.truncated) markMessageTruncated(managed, scriptId); } let binaryResult: Record = {}; if (data !== null) { @@ -939,7 +952,7 @@ export class FridaInstrumentationManager implements FridaInstrumentationPort { data_truncated: true, data_bytes: data.byteLength, }; - managed.messagesTruncated = true; + markMessageTruncated(managed, scriptId); } else { binaryResult = { data_base64: data.toString("base64") }; } @@ -961,12 +974,21 @@ export class FridaInstrumentationManager implements FridaInstrumentationPort { message.stack === undefined ? undefined : boundedRedactedString(message.stack, MAX_MESSAGE_TEXT_BYTES, secrets); + const fileName = + message.fileName === undefined + ? undefined + : boundedRedactedString( + message.fileName, + MAX_MESSAGE_TEXT_BYTES, + secrets, + ); if ( type.truncated || description?.truncated === true || - stack?.truncated === true + stack?.truncated === true || + fileName?.truncated === true ) - managed.messagesTruncated = true; + markMessageTruncated(managed, scriptId); const observation: JsonValue = { script_id: scriptId, type: type.value, @@ -989,11 +1011,27 @@ export class FridaInstrumentationManager implements FridaInstrumentationPort { stack: stack.value, ...(stack.truncated ? { stack_truncated: true } : {}), }), + ...(fileName === undefined + ? {} + : { + file_name: fileName.value, + ...(fileName.truncated ? { file_name_truncated: true } : {}), + }), + ...(typeof message.lineNumber === "number" && + Number.isSafeInteger(message.lineNumber) && + message.lineNumber >= 0 + ? { line_number: message.lineNumber } + : {}), + ...(typeof message.columnNumber === "number" && + Number.isSafeInteger(message.columnNumber) && + message.columnNumber >= 0 + ? { column_number: message.columnNumber } + : {}), ...binaryResult, }; const charge = Buffer.byteLength(JSON.stringify(observation)); if (charge > MAX_SESSION_CAPTURE_BYTES) { - managed.messagesTruncated = true; + markMessageTruncated(managed, scriptId); return; } while ( @@ -1008,8 +1046,8 @@ export class FridaInstrumentationManager implements FridaInstrumentationPort { const scriptMessages = managed.messagesByScript.get(removed.scriptId); if (scriptMessages?.peek() === removed.observation) scriptMessages.shift(); + markMessageTruncated(managed, removed.scriptId); } - managed.messagesTruncated = true; } managed.messages.push({ observation, scriptId, charge }); managed.messagesByScript.get(scriptId)?.push(observation); @@ -1017,18 +1055,26 @@ export class FridaInstrumentationManager implements FridaInstrumentationPort { } async #getDeviceManager(): Promise { - if (this.#deviceManager === undefined) { - try { - this.#deviceManager = await this.#deviceManagerLoader(); - } catch (cause: unknown) { - this.#availabilityFailure = { - providerId: "frida", - reason: cause instanceof Error ? cause.message : String(cause), - }; - throw cause; - } + if (this.#deviceManager !== undefined) return this.#deviceManager; + if (this.#deviceManagerLoading !== undefined) + return this.#deviceManagerLoading; + const loading = Promise.resolve().then(this.#deviceManagerLoader); + this.#deviceManagerLoading = loading; + try { + this.#deviceManager = await loading; + if (this.#availabilityFailure?.providerId === "frida") + this.#availabilityFailure = undefined; + return this.#deviceManager; + } catch (cause: unknown) { + this.#availabilityFailure = { + providerId: "frida", + reason: boundedRedactedString(errorMessage(cause), 4096, []).value, + }; + throw cause; + } finally { + if (this.#deviceManagerLoading === loading) + this.#deviceManagerLoading = undefined; } - return this.#deviceManager; } async #selectDevice( @@ -1054,7 +1100,14 @@ export class FridaInstrumentationManager implements FridaInstrumentationPort { let reference = this.#remoteDevices.get(remote.address); if (reference !== undefined) { if (reference.optionsDigest !== optionsDigest) - throw new AnalysisInputError("frida_remote_connection"); + throw new AnalysisInputError("frida_remote_connection", undefined, [ + { + path: ["remote"], + reason: "invalid_value", + message: + "This endpoint is already retained with different connection options. Reuse the existing options or close its sessions before changing them.", + }, + ]); reference.references += 1; } else { const device = manager.addRemoteDevice( @@ -1361,6 +1414,8 @@ const providerError = ( remote?: FridaRemoteConnection, cleanupReason?: string, ): AnalysisError => { + if (cause instanceof AnalysisInputError) + return new AnalysisInputError(operation, { cause }, cause.issues); if (cause instanceof AnalysisCancelledError) return new AnalysisCancelledError(operation, { cause: cause.cause, @@ -1403,6 +1458,15 @@ interface CapturedMessage { readonly charge: number; } +const markMessageTruncated = ( + managed: ManagedSession, + scriptId: string, +): void => { + managed.messagesTruncated = true; + if (managed.messagesByScript.has(scriptId)) + managed.truncatedScripts.add(scriptId); +}; + /** * Arrival-ordered queue whose eviction advances a head offset. A full capture * budget evicts on every message, and `Array#shift` would copy every retained diff --git a/src/frida/frida-bindings.d.ts b/src/frida/frida-bindings.d.ts index 9ccdb89538..280e8c2701 100644 --- a/src/frida/frida-bindings.d.ts +++ b/src/frida/frida-bindings.d.ts @@ -4,6 +4,9 @@ declare module "frida" { readonly payload?: unknown; readonly description?: string; readonly stack?: string; + readonly fileName?: string; + readonly lineNumber?: number; + readonly columnNumber?: number; } export type SignalHandler = (...args: never[]) => void; diff --git a/src/gdb/GdbSessionManager.ts b/src/gdb/GdbSessionManager.ts new file mode 100644 index 0000000000..5102ad84e5 --- /dev/null +++ b/src/gdb/GdbSessionManager.ts @@ -0,0 +1,2057 @@ +import { randomUUID } from "node:crypto"; +import { constants as fsConstants } from "node:fs"; +import { isUtf8 } from "node:buffer"; +import { readFile, readlink, realpath, stat } from "node:fs/promises"; + +import { + AnalysisCapabilityUnavailableError, + AnalysisCancelledError, + AnalysisInputError, + AnalysisResourceConstraintError, + AnalysisTimeoutError, +} from "../domain/analysisErrorCore.js"; +import { + AnalysisError, + type AnalysisErrorOptions, +} from "../domain/analysisErrorBase.js"; +import { createEvidence, type Evidence } from "../domain/evidence.js"; +import { err, ok, type Result } from "../domain/result.js"; +import { + ProviderProcessSupervisor, + spawnOwnedProviderProcess, + type OwnedProviderProcessSpawnOptions, + type SpawnedOwnedProviderProcess, +} from "../process/ProviderProcess.js"; +import { + closeOwnedDebuggerSessions, + stopOwnedDebuggerProcess, + type OwnedDebuggerDetachResult, + type OwnedInferiorIdentity, + type ProcessStartTimeReader, + verifyOwnedInferiorsEnded, +} from "../process/ownedDebuggerSession.js"; +import { parseLinuxProcessStartTime } from "../process/ProcessOwnershipObservation.js"; + +const STARTUP_TIMEOUT_MS = 10_000; +const COMMAND_TIMEOUT_MS = 120_000; +const MAX_COMMAND_OUTPUT_BYTES = 16 * 1024 * 1024; +const MAX_COMMAND_OUTPUT_RECORDS = 16_384; +const MAX_MI_LINE_BYTES = 16 * 1024 * 1024; +const MAX_RECENT_MI_LINE_BYTES = 64 * 1024; +const MAX_DIAGNOSTIC_BYTES = 64 * 1024; +const DEFAULT_MAX_SESSIONS = 8; +const MAX_PENDING_GDB_COMMANDS = 16; +const RECENT_RECORD_TRUNCATION_MARKER = "[recent record truncated]"; + +export const GDB_PROVIDER_IDENTITY = { + id: "gnu.gdb", + name: "GNU GDB", + version: null, +} as const; + +interface PendingCommand { + readonly token: number; + readonly command: string; + readonly evidenceCommand: string; + readonly lines: string[]; + outputBytes: number; + outputTruncated: boolean; + outputInvalidUtf8: boolean; + terminalRecord: string | undefined; + awaitingInferiorStop: boolean; + readonly resolve: (result: GdbCommandResult) => void; + readonly reject: (error: Error) => void; + readonly cleanup: () => void; + timer: ReturnType; +} + +export interface GdbCommandResult { + readonly command: string; + readonly mi: string; + readonly records: readonly string[]; + readonly console: string; + readonly target: string; + readonly log: string; + readonly process_exit_observed?: boolean; + readonly process_exit_code?: number | null; + readonly process_exit_signal?: NodeJS.Signals | null; + readonly output_truncated: boolean; + readonly completion_status: "complete" | "unknown"; + readonly output_invalid_utf8?: true; +} + +class GdbCommandTimeoutError extends AnalysisTimeoutError { + constructor( + readonly partial: GdbCommandResult, + readonly evidence: Evidence, + timeoutMs: number, + ) { + super("gdb_console", timeoutMs); + } +} + +const truncateUtf8RecordPrefix = (line: string): string => { + const budget = + MAX_RECENT_MI_LINE_BYTES - + Buffer.byteLength(RECENT_RECORD_TRUNCATION_MARKER, "utf8"); + let prefix = ""; + let retainedBytes = 0; + for (const codePoint of line) { + const codePointBytes = Buffer.byteLength(codePoint, "utf8"); + if (retainedBytes + codePointBytes > budget) break; + prefix += codePoint; + retainedBytes += codePointBytes; + } + return `${prefix}${RECENT_RECORD_TRUNCATION_MARKER}`; +}; + +/** @internal Pure parser for live PIDs in recursive GDB MI thread groups. */ +export const gdbThreadGroupPids = ( + mi: string, +): readonly string[] | undefined => { + const marker = mi.indexOf("groups=["); + if (marker < 0) return undefined; + const start = marker + "groups=[".length; + let squareDepth = 1; + let objectDepth = 0; + let quoted = false; + let escaped = false; + const pids: string[] = []; + let fieldStart = start; + let pidStart: number | undefined; + let hasLivePid = false; + let currentPid: string | undefined; + for (let index = start; index < mi.length; index += 1) { + const character = mi[index]; + if (quoted) { + if (escaped) escaped = false; + else if (character === "\\") escaped = true; + else if (character === '"') { + quoted = false; + if (pidStart !== undefined) { + const pid = mi.slice(pidStart, index); + if (!/^[1-9]\d*$/u.test(pid)) return undefined; + hasLivePid = true; + currentPid = pid; + pidStart = undefined; + } + } + continue; + } + if (character === '"') { + if ( + squareDepth === 1 && + objectDepth === 1 && + mi.slice(fieldStart, index) === "pid=" + ) { + if (hasLivePid) return undefined; + pidStart = index + 1; + } + quoted = true; + continue; + } + if (character === "[") squareDepth += 1; + else if (character === "]") { + squareDepth -= 1; + if (squareDepth === 0) { + if (objectDepth !== 0) return undefined; + const body = mi.slice(start, index); + if (body.includes("groups=[")) return undefined; + return pids; + } + } else if (character === "{") { + if (squareDepth === 1 && objectDepth === 0) { + hasLivePid = false; + currentPid = undefined; + fieldStart = index + 1; + } + objectDepth += 1; + } else if (character === "}") { + if (squareDepth === 1 && objectDepth === 1) { + const field = mi.slice(fieldStart, index); + if (field.startsWith("pid=") && !/^pid="[1-9]\d*"$/u.test(field)) + return undefined; + } + objectDepth -= 1; + if (objectDepth < 0) return undefined; + if (squareDepth === 1 && objectDepth === 0 && hasLivePid) { + if (currentPid === undefined) return undefined; + pids.push(currentPid); + } + } else if (character === "," && squareDepth === 1 && objectDepth === 1) { + const field = mi.slice(fieldStart, index); + if (field.startsWith("pid=") && !/^pid="[1-9]\d*"$/u.test(field)) + return undefined; + fieldStart = index + 1; + } + } + return undefined; +}; + +/** @internal Pure count of recursive GDB MI groups with live PIDs. */ +export const gdbThreadGroupCount = (mi: string): number | undefined => { + const pids = gdbThreadGroupPids(mi); + return pids?.length; +}; + +const verifyGdbInferiorOutputSink = async ( + pid: string, + selectedTty: string, + rememberIdentity: (identity: OwnedInferiorIdentity) => void, +): Promise => { + if (process.platform !== "linux") + throw new Error( + "GDB cannot confirm live inferior output descriptors outside Linux /proc.", + ); + const statPath = `/proc/${pid}/stat`; + const before = parseLinuxProcessStartTime(await readFile(statPath, "utf8")); + if (before === undefined) + throw new Error("GDB could not establish the live inferior identity."); + const ttyPath = await realpath(selectedTty); + const tty = await stat(ttyPath); + if ((tty.mode & fsConstants.S_IFMT) !== fsConstants.S_IFCHR) + throw new Error("The selected GDB output sink is not a character device."); + for (const descriptor of ["1", "2"]) { + const descriptorPath = `/proc/${pid}/fd/${descriptor}`; + const destination = await readlink(descriptorPath); + const observed = await stat(descriptorPath); + if ( + observed.dev !== tty.dev || + observed.ino !== tty.ino || + (observed.mode & fsConstants.S_IFMT) !== fsConstants.S_IFCHR || + (await realpath(destination)) !== ttyPath + ) + throw new Error( + `GDB inferior descriptor ${descriptor} no longer points to the selected TTY.`, + ); + } + const after = parseLinuxProcessStartTime(await readFile(statPath, "utf8")); + if (after !== before) + throw new Error( + "The GDB inferior identity changed during descriptor readback.", + ); + rememberIdentity({ pid, startTime: before }); +}; + +class GdbCommandProcessExitError extends Error { + constructor(readonly partial: GdbCommandResult) { + super( + `GDB exited before returning the active MI command result (exit code ${String(partial.process_exit_code)}, signal ${String(partial.process_exit_signal)}).`, + ); + } +} + +class GdbProtocolFailureError extends Error { + constructor( + message: string, + readonly partial: GdbCommandResult, + ) { + super(message); + } +} + +/** @internal Convert an unknown boundary failure without dropping non-Error values. */ +export const gdbErrorMessage = (cause: unknown): string => + cause instanceof Error ? cause.message : String(cause); + +interface GdbSession { + readonly id: string; + readonly miVersion: "mi3" | "mi2"; + readonly launched: SpawnedOwnedProviderProcess; + readonly supervisor: ProviderProcessSupervisor; + /** Confirms target disposition and any durable sink before process cleanup. */ + readonly detachInferior: () => Promise; + readonly inferiorStateUnknown: () => boolean; + readonly verifyInferiorsEndedAfterExit: () => Promise; + readonly knownInferiorIdentities: Map; + readonly inferiorTty: string | undefined; + readonly maxCommandOutputBytes: number; + readonly stdoutSettled: Promise; + readonly settleStdout: () => void; + inferiorTtyVerified: boolean; + noInferiorVerified: boolean; + stopAttempt: ReturnType | undefined; + readonly pending: Map; + readonly cleanupFailures: Set; + readonly lines: string[]; + readonly bufferChunks: Buffer[]; + bufferTail: Buffer | undefined; + bufferTailBytes: number; + bufferBytes: number; + invalidUtf8Observed: boolean; + promptTail: string; + ready: boolean; + exited: boolean; + exitCode: number | null; + exitSignal: NodeJS.Signals | null; + nextToken: number; + startupResolve: (() => void) | undefined; + startupReject: ((error: Error) => void) | undefined; + commandTail: Promise; + pendingCommandCount: number; + protocolError: string | undefined; + recentRecordsTruncated: boolean; +} + +/** Owns persistent GDB/MI sessions independently from the active binary target. */ +export class GdbSessionManager { + readonly #sessions = new Map(); + readonly #activeStarts = new Map< + AbortController, + Promise< + Result< + { readonly session_id: string; readonly mi_version: "mi3" | "mi2" }, + AnalysisError + > + > + >(); + readonly #command: string; + readonly #environment: NodeJS.ProcessEnv; + readonly #commandTimeoutMs: number; + readonly #startupTimeoutMs: number; + readonly #platform: NodeJS.Platform; + readonly #stopProcess: typeof stopOwnedDebuggerProcess; + readonly #spawnProcess: typeof spawnOwnedProviderProcess; + readonly #verifyInferiorOutputSink: ( + pid: string, + selectedTty: string, + rememberIdentity: (identity: OwnedInferiorIdentity) => void, + ) => Promise; + readonly #readInferiorProcessStartTime: ProcessStartTimeReader | undefined; + readonly #maxSessions: number; + readonly #maxCommandOutputBytes: number; + #sessionReservations = 0; + #closing = false; + #closePromise: Promise | undefined; + + constructor( + options: { + readonly environment?: Readonly; + readonly commandTimeoutMs?: number; + readonly startupTimeoutMs?: number; + readonly platform?: NodeJS.Platform; + readonly stopProcess?: typeof stopOwnedDebuggerProcess; + readonly spawnProcess?: typeof spawnOwnedProviderProcess; + /** @internal Tests may replace only the /proc and TTY readback boundary. */ + readonly verifyInferiorOutputSink?: ( + pid: string, + selectedTty: string, + rememberIdentity: (identity: OwnedInferiorIdentity) => void, + ) => Promise; + /** @internal Read-only process identity observer for lifecycle tests. */ + readonly readInferiorProcessStartTime?: ProcessStartTimeReader; + readonly maxSessions?: number; + /** @internal Test seam; production callers should use the 16 MiB default. */ + readonly maxCommandOutputBytes?: number; + } = {}, + ) { + this.#environment = { ...(options.environment ?? process.env) }; + this.#command = this.#environment.REA_GDB_COMMAND ?? "gdb"; + this.#commandTimeoutMs = options.commandTimeoutMs ?? COMMAND_TIMEOUT_MS; + this.#startupTimeoutMs = options.startupTimeoutMs ?? STARTUP_TIMEOUT_MS; + this.#platform = options.platform ?? process.platform; + this.#stopProcess = options.stopProcess ?? stopOwnedDebuggerProcess; + this.#spawnProcess = options.spawnProcess ?? spawnOwnedProviderProcess; + this.#verifyInferiorOutputSink = + options.verifyInferiorOutputSink ?? verifyGdbInferiorOutputSink; + this.#readInferiorProcessStartTime = options.readInferiorProcessStartTime; + this.#maxSessions = options.maxSessions ?? DEFAULT_MAX_SESSIONS; + this.#maxCommandOutputBytes = + options.maxCommandOutputBytes ?? MAX_COMMAND_OUTPUT_BYTES; + } + + async start( + signal?: AbortSignal, + options: { readonly inferiorTty?: string } = {}, + ): Promise< + Result< + { readonly session_id: string; readonly mi_version: "mi3" | "mi2" }, + AnalysisError + > + > { + if ( + options.inferiorTty !== undefined && + (options.inferiorTty.length === 0 || options.inferiorTty.includes("\0")) + ) + return err( + new AnalysisInputError("start_gdb_session", undefined, [ + { + path: ["inferior_tty"], + reason: "invalid_value", + message: + "The inferior TTY path must be non-empty and must not contain NUL.", + }, + ]), + ); + if (this.#closing || signal?.aborted) + return err(new AnalysisCancelledError("start_gdb_session")); + if (this.#platform !== "linux") + return err( + new AnalysisCapabilityUnavailableError( + "gdb", + "start_gdb_session", + this.#platform === "win32" + ? "Persistent Windows GDB sessions are unavailable because owned Job Object cleanup can terminate debugger inferiors." + : "Persistent GDB sessions are available only on Linux because REA verifies live inferior output descriptors through /proc.", + ), + ); + const controller = new AbortController(); + const relayAbort = (): void => controller.abort(signal?.reason); + signal?.addEventListener("abort", relayAbort, { once: true }); + if (signal?.aborted) relayAbort(); + const tracked = Promise.resolve() + .then(() => this.#startWithReservation(controller.signal, options)) + .finally(() => { + signal?.removeEventListener("abort", relayAbort); + this.#activeStarts.delete(controller); + }); + this.#activeStarts.set(controller, tracked); + return tracked; + } + + async #startWithReservation( + signal: AbortSignal, + options: { readonly inferiorTty?: string } = {}, + ): Promise< + Result< + { readonly session_id: string; readonly mi_version: "mi3" | "mi2" }, + AnalysisError + > + > { + if (signal?.aborted) + return err(new AnalysisCancelledError("start_gdb_session")); + const occupied = this.#sessions.size + this.#sessionReservations; + if (occupied >= this.#maxSessions) + return err( + new AnalysisResourceConstraintError( + "start_gdb_session", + "memory", + "GDB session capacity reached.", + { + max_sessions: this.#maxSessions, + active_or_reserved_sessions: occupied, + }, + ), + ); + this.#sessionReservations += 1; + let reservationHeld = true; + const releaseReservation = (): void => { + if (!reservationHeld) return; + reservationHeld = false; + this.#sessionReservations -= 1; + }; + try { + return await this.#startReserved(releaseReservation, signal, options); + } finally { + releaseReservation(); + } + } + + async #startReserved( + releaseReservation: () => void, + signal: AbortSignal, + options: { readonly inferiorTty?: string } = {}, + ): Promise< + Result< + { readonly session_id: string; readonly mi_version: "mi3" | "mi2" }, + AnalysisError + > + > { + const id = randomUUID(); + const mi3 = await this.#startProtocol( + id, + "mi3", + releaseReservation, + signal, + options.inferiorTty, + ); + return mi3.ok || signal.aborted || this.#sessions.has(id) + ? mi3 + : this.#startProtocol( + id, + "mi2", + releaseReservation, + signal, + options.inferiorTty, + ); + } + + async #startProtocol( + id: string, + miVersion: "mi3" | "mi2", + releaseReservation: () => void, + signal: AbortSignal, + inferiorTty?: string, + ): Promise< + Result< + { readonly session_id: string; readonly mi_version: "mi3" | "mi2" }, + AnalysisError + > + > { + let launched: SpawnedOwnedProviderProcess; + try { + const spawnOptions: OwnedProviderProcessSpawnOptions = { + command: this.#command, + arguments: ["--nx", "--quiet", `--interpreter=${miVersion}`], + runId: randomUUID(), + stdin: "pipe", + hostEnvironment: this.#environment, + signal, + }; + launched = await this.#spawnProcess(spawnOptions); + } catch (cause) { + if (signal?.aborted || cause instanceof AnalysisCancelledError) + return err(new AnalysisCancelledError("start_gdb_session", { cause })); + return err( + new AnalysisCapabilityUnavailableError( + "gdb", + "start_gdb_session", + cause instanceof Error ? cause.message : "Unable to start GDB", + ), + ); + } + const supervisor = new ProviderProcessSupervisor( + { process: launched.process, ownsProcessLifetime: false }, + { captureStdout: false, maxDiagnosticBytes: MAX_DIAGNOSTIC_BYTES }, + ); + let settleStdout!: () => void; + const stdoutSettled = new Promise((resolve) => { + settleStdout = resolve; + }); + const session: GdbSession = { + id, + miVersion, + launched, + supervisor, + detachInferior: () => this.#detachInferior(session), + inferiorStateUnknown: () => !session.noInferiorVerified, + verifyInferiorsEndedAfterExit: async () => { + const ended = await verifyOwnedInferiorsEnded( + session.knownInferiorIdentities.values(), + this.#readInferiorProcessStartTime, + ); + if (ended) session.knownInferiorIdentities.clear(); + return ended; + }, + knownInferiorIdentities: new Map(), + inferiorTty, + maxCommandOutputBytes: this.#maxCommandOutputBytes, + stdoutSettled, + settleStdout, + inferiorTtyVerified: false, + // GDB is launched with --nx and receives no target-execution commands + // during startup, so no inferior can exist until execute() is called. + noInferiorVerified: true, + stopAttempt: undefined, + pending: new Map(), + cleanupFailures: new Set(), + lines: [], + bufferChunks: [], + bufferTail: undefined, + bufferTailBytes: 0, + bufferBytes: 0, + invalidUtf8Observed: false, + promptTail: "", + ready: false, + exited: false, + exitCode: null, + exitSignal: null, + nextToken: 1, + startupResolve: undefined, + startupReject: undefined, + commandTail: Promise.resolve(), + pendingCommandCount: 0, + protocolError: undefined, + recentRecordsTruncated: false, + }; + launched.process.stdout?.on("data", (chunk: string | Buffer) => + this.#receive( + session, + typeof chunk === "string" ? Buffer.from(chunk, "utf8") : chunk, + ), + ); + if (launched.process.stdout === null) settleStdout(); + else launched.process.stdout.once("end", settleStdout); + launched.process.stdin?.on("error", (cause: Error) => + this.#failProtocol(session, `GDB stdin stream failed: ${cause.message}`), + ); + launched.process.stdin?.once("close", () => + this.#failProtocol(session, "GDB stdin stream closed unexpectedly."), + ); + launched.process.once("exit", (code, signalCode) => { + session.exited = true; + session.exitCode = code; + session.exitSignal = signalCode; + void this.#waitForStdoutSettlement(session).then(() => + session.startupReject?.( + new Error( + `GDB exited during startup (exit code ${String(code)}, signal ${String(signalCode)}).`, + ), + ), + ); + void this.#settleExitedCommands(session, code, signalCode); + }); + let startupTimer!: ReturnType; + let startupAbortListener!: () => void; + try { + await new Promise((resolve, reject) => { + session.startupResolve = resolve; + session.startupReject = reject; + startupAbortListener = () => + reject(new AnalysisCancelledError("start_gdb_session")); + signal?.addEventListener("abort", startupAbortListener, { + once: true, + }); + if (signal?.aborted) startupAbortListener(); + startupTimer = setTimeout( + () => + reject( + new AnalysisTimeoutError( + "start_gdb_session", + this.#startupTimeoutMs, + { + cause: new Error( + "GDB did not emit the MI prompt before the startup deadline", + ), + }, + ), + ), + this.#startupTimeoutMs, + ); + startupTimer.unref(); + }); + session.ready = true; + if (signal?.aborted) + throw new AnalysisCancelledError("start_gdb_session"); + const autoLoad = await this.#request( + session, + "-gdb-set auto-load off", + signal, + "start_gdb_session", + ); + if (!/^\d+\^done(?:,|$)/u.test(autoLoad.mi)) { + throw new AnalysisCapabilityUnavailableError( + "gdb", + "start_gdb_session", + `GDB rejected the required automatic-loading disable command (${autoLoad.mi}); the session was not started.`, + ); + } + if (inferiorTty !== undefined) { + const configured = await this.#request( + session, + `-inferior-tty-set ${miQuote(inferiorTty)}`, + signal, + "start_gdb_session", + ); + if ( + configured.completion_status !== "complete" || + configured.output_truncated || + !/^\d+\^done(?:,|$)/u.test(configured.mi) + ) + throw new AnalysisCapabilityUnavailableError( + "gdb", + "start_gdb_session", + "GDB did not confirm the inferior TTY configuration.", + ); + const observed = await this.#request( + session, + "-inferior-tty-show", + signal, + "start_gdb_session", + ); + const reportedTty = + observed.completion_status === "complete" && + !observed.output_truncated + ? miStringField(observed.mi, "inferior_tty_terminal") + : undefined; + if ( + !/^\d+\^done(?:,|$)/u.test(observed.mi) || + reportedTty !== inferiorTty + ) + throw new AnalysisCapabilityUnavailableError( + "gdb", + "start_gdb_session", + "GDB did not report the exact caller-selected inferior TTY after configuration.", + ); + session.inferiorTtyVerified = true; + } + if (signal?.aborted) + throw new AnalysisCancelledError("start_gdb_session"); + // Give queued exit notifications a turn before publishing startup. + await new Promise((resolve) => setImmediate(resolve)); + if (signal?.aborted) + throw new AnalysisCancelledError("start_gdb_session"); + if (session.exited || session.protocolError !== undefined) + throw new Error( + session.protocolError ?? "GDB exited during startup readiness.", + ); + // Before this point no target command can run. Once a usable session is + // published, the inferior state is no longer inferred from startup phase. + session.noInferiorVerified = false; + this.#sessions.set(id, session); + releaseReservation(); + return ok({ session_id: id, mi_version: miVersion }); + } catch (cause) { + const startupExited = + cause instanceof GdbCommandProcessExitError || session.exited; + const processExit = + cause instanceof GdbCommandProcessExitError + ? cause.partial + : startupExited + ? { + command: "GDB startup", + mi: "", + records: [...session.lines], + console: session.lines + .filter((line) => line.startsWith("~")) + .join("\n"), + target: session.lines + .filter((line) => line.startsWith("@")) + .join("\n"), + log: session.lines + .filter((line) => line.startsWith("&")) + .join("\n"), + process_exit_observed: true, + process_exit_code: session.exitCode, + process_exit_signal: session.exitSignal, + output_truncated: session.recentRecordsTruncated, + completion_status: "unknown" as const, + } + : undefined; + const startupEvidenceResult = + processExit ?? + (session.lines.length > 0 + ? { + command: "GDB startup", + mi: "", + records: [...session.lines], + console: session.lines + .filter((line) => line.startsWith("~")) + .join("\n"), + target: session.lines + .filter((line) => line.startsWith("@")) + .join("\n"), + log: session.lines + .filter((line) => line.startsWith("&")) + .join("\n"), + output_truncated: session.recentRecordsTruncated, + completion_status: "unknown" as const, + } + : undefined); + const startupEvidencePayload = + startupEvidenceResult === undefined + ? undefined + : { + ...startupEvidenceResult, + ...(session.invalidUtf8Observed + ? { output_invalid_utf8: true as const } + : {}), + }; + const startupEvidence = + startupEvidencePayload === undefined + ? undefined + : createEvidence(undefined, GDB_PROVIDER_IDENTITY, { + predicateType: "rea.debugger.startup-observation", + operation: "start_gdb_session", + parameters: { + mi_version: miVersion, + command: startupEvidencePayload.command, + }, + result: { + ...startupEvidencePayload, + records: [...startupEvidencePayload.records], + }, + rawResult: { + ...startupEvidencePayload, + records: [...startupEvidencePayload.records], + }, + subjectUnavailableReason: + "No usable debugger session became ready during GDB startup.", + limitations: [ + "GDB startup failed before a usable session could be established; captured startup records are incomplete.", + ...(session.invalidUtf8Observed + ? [ + "GDB startup output contained invalid UTF-8 bytes; decoded records contain replacement characters.", + ] + : []), + ], + }); + const processOutput = session.supervisor.snapshot(); + const capturedOutput = { + stdout: processOutput.stdout.text, + stderr: processOutput.stderr.text, + truncated: + processOutput.diagnosticTruncated === true || + processOutput.stdout.observedBytes > processOutput.stdout.bytes || + processOutput.stderr.observedBytes > processOutput.stderr.bytes, + stdout_bytes: processOutput.stdout.observedBytes, + stderr_bytes: processOutput.stderr.observedBytes, + exit_code: processOutput.exitCode ?? null, + signal: processOutput.signal ?? null, + }; + const stopped = await this.#stopDebuggerOnly(session); + if (stopped.status === "incomplete") { + this.#sessions.set(id, session); + releaseReservation(); + const cleanup = { reason: stopped.reason, resources: [session.id] }; + const details = { + cause, + capturedOutput, + cleanup, + ...(startupEvidence === undefined + ? {} + : { partialObservation: startupEvidence }), + }; + if (cause instanceof AnalysisTimeoutError) + return err( + new AnalysisTimeoutError(cause.operation, cause.timeoutMs, details), + ); + if (cause instanceof AnalysisCancelledError) + return err(new AnalysisCancelledError(cause.operation, details)); + const startupFailureMessage = gdbErrorMessage(cause); + return err( + new AnalysisCapabilityUnavailableError( + "gdb", + "start_gdb_session", + `GDB startup failed (${startupFailureMessage}) and its owned process could not be confirmed stopped: ${stopped.reason}`, + details, + ), + ); + } + const errorDetails = { + cause, + capturedOutput, + ...(startupEvidence === undefined + ? {} + : { partialObservation: startupEvidence }), + }; + const error = + signal?.aborted || cause instanceof AnalysisCancelledError + ? new AnalysisCancelledError("start_gdb_session", errorDetails) + : cause instanceof AnalysisTimeoutError + ? new AnalysisTimeoutError( + cause instanceof GdbCommandTimeoutError + ? "start_gdb_session" + : cause.operation, + cause.timeoutMs, + errorDetails, + ) + : cause instanceof AnalysisCapabilityUnavailableError + ? new AnalysisCapabilityUnavailableError( + "gdb", + "start_gdb_session", + cause.message, + errorDetails, + ) + : session.protocolError !== undefined + ? new AnalysisCapabilityUnavailableError( + "gdb", + "start_gdb_session", + session.protocolError, + errorDetails, + ) + : new AnalysisCapabilityUnavailableError( + "gdb", + "start_gdb_session", + `GDB startup failed (${gdbErrorMessage(cause)}).`, + errorDetails, + ); + return err(error); + } finally { + clearTimeout(startupTimer); + signal?.removeEventListener("abort", startupAbortListener); + } + } + + async execute( + sessionId: string, + command: string, + signal?: AbortSignal, + ): Promise< + Result< + { readonly result: GdbCommandResult; readonly evidence: Evidence }, + AnalysisError + > + > { + const session = this.#sessions.get(sessionId); + if ( + this.#closing || + session === undefined || + !session.ready || + session.exited || + session.protocolError !== undefined + ) + return err( + new AnalysisInputError("gdb_console", undefined, [ + { + path: ["session_id"], + reason: "invalid_value", + message: "GDB session is unavailable or closed.", + }, + ]), + ); + if (signal?.aborted) return err(new AnalysisCancelledError("gdb_console")); + try { + const completed = await this.#serialize( + session, + async () => { + if (session.exited || session.protocolError !== undefined) + return err(new AnalysisInputError("gdb_console")); + session.noInferiorVerified = false; + session.knownInferiorIdentities.clear(); + if (commandMayChangeInferiorOutput(command)) + session.inferiorTtyVerified = false; + try { + const result = await this.#request( + session, + `-interpreter-exec console ${miQuote(command)}`, + signal, + "gdb_console", + command, + ); + const evidence = createEvidence( + undefined, + { id: "gnu.gdb", name: "GNU GDB", version: null }, + { + predicateType: "rea.debugger.console-observation", + operation: "gdb_console", + parameters: { session_id: sessionId, command }, + result: { + command, + mi: result.mi, + records: [...result.records], + console: result.console, + target: result.target, + log: result.log, + output_truncated: result.output_truncated, + ...(result.output_invalid_utf8 + ? { output_invalid_utf8: true as const } + : {}), + completion_status: result.completion_status, + }, + rawResult: { + command, + mi: result.mi, + records: [...result.records], + console: result.console, + target: result.target, + log: result.log, + output_truncated: result.output_truncated, + ...(result.output_invalid_utf8 + ? { output_invalid_utf8: true as const } + : {}), + completion_status: result.completion_status, + }, + subjectUnavailableReason: + "The active GDB inferior identity is not established by the debugger session identifier.", + limitations: [ + "MI records are returned as decoded text; their semantics depend on GDB version and target.", + ...(result.output_invalid_utf8 + ? [ + "MI stream bytes were not valid UTF-8; returned text contains replacement characters.", + ] + : []), + "The unrestricted GDB console can access local shell, filesystem, network, and target controls.", + ], + }, + ); + return ok({ result, evidence }); + } catch (cause) { + if (cause instanceof AnalysisCancelledError) return err(cause); + if (cause instanceof AnalysisCapabilityUnavailableError) + return err(cause as AnalysisError); + if (cause instanceof GdbCommandTimeoutError) { + const result = cause.partial; + return ok({ result, evidence: cause.evidence }); + } + if (session.protocolError !== undefined) + return err(await this.#stopAfterProtocolFailure(session, cause)); + if (cause instanceof GdbCommandProcessExitError) { + const processExit = cause as GdbCommandProcessExitError; + const result: GdbCommandResult = { + ...processExit.partial, + command, + }; + const evidence = createEvidence( + undefined, + { id: "gnu.gdb", name: "GNU GDB", version: null }, + { + predicateType: "rea.debugger.console-observation", + operation: "gdb_console", + parameters: { session_id: sessionId, command }, + result: { ...result, records: [...result.records] }, + rawResult: { ...result, records: [...result.records] }, + subjectUnavailableReason: + "The active GDB inferior identity is not established by the debugger session identifier.", + limitations: [ + "GDB exited before returning a correlated MI result record; only records retained for the active command are returned raw.", + ...(result.output_truncated + ? [ + "Retained MI records are a truncated prefix; output completeness is unknown.", + ] + : []), + ], + }, + ); + const stopped = await this.#stopDebuggerOnly(session); + if (stopped.status === "incomplete") + return err( + this.#retainCleanupFailure( + session, + "gdb_console", + stopped.reason, + { cause, partialObservation: evidence }, + ), + ); + return ok({ result, evidence }); + } + return err( + new AnalysisTimeoutError("gdb_console", this.#commandTimeoutMs, { + cause, + }), + ); + } + }, + signal, + "gdb_console", + ); + if (!completed.ok && completed.error.cleanupIncomplete) + session.cleanupFailures.add(completed.error); + return completed; + } catch (cause) { + const error = + cause instanceof AnalysisError + ? cause + : new AnalysisCancelledError("gdb_console", { cause }); + if (error.cleanupIncomplete) session.cleanupFailures.add(error); + return err(error); + } + } + + status(sessionId: string): + | { + readonly session_id: string; + readonly mi_version: "mi3" | "mi2"; + readonly state: "ready" | "closed"; + readonly recent_mi_records: readonly string[]; + readonly recent_mi_records_truncated: boolean; + readonly mi_invalid_utf8: boolean; + readonly stderr: string; + readonly stderr_observed_bytes: number; + readonly stderr_truncated: boolean; + readonly stderr_invalid_utf8: boolean; + readonly diagnostics_truncated: boolean; + readonly inferior_tty: string | null; + readonly inferior_tty_verified: boolean; + } + | undefined { + const session = this.#sessions.get(sessionId); + if (session === undefined) return undefined; + const diagnostics = session.supervisor.snapshot(); + return { + session_id: session.id, + mi_version: session.miVersion, + state: + session.exited || session.protocolError !== undefined || !session.ready + ? "closed" + : "ready", + recent_mi_records: [...session.lines], + recent_mi_records_truncated: session.recentRecordsTruncated, + mi_invalid_utf8: session.invalidUtf8Observed, + stderr: diagnostics.stderr.text, + stderr_observed_bytes: diagnostics.stderr.observedBytes, + stderr_truncated: + diagnostics.stderr.observedBytes > diagnostics.stderr.bytes, + stderr_invalid_utf8: diagnostics.stderr.invalidUtf8 === true, + diagnostics_truncated: diagnostics.diagnosticTruncated === true, + inferior_tty: session.inferiorTty ?? null, + inferior_tty_verified: session.inferiorTtyVerified, + }; + } + + async close( + sessionId: string, + ): Promise< + Result< + { readonly target_state: "none_observed" | "unknown" }, + AnalysisError + > + > { + if (this.#closing) + return err(new AnalysisCancelledError("close_gdb_session")); + const session = this.#sessions.get(sessionId); + if (session === undefined) + return err(new AnalysisInputError("close_gdb_session")); + if (session.exited) { + const stopped = await this.#stopDebuggerOnly(session); + if (stopped.status === "incomplete") + return err( + this.#retainCleanupFailure( + session, + "close_gdb_session", + stopped.reason, + ), + ); + this.#sessions.delete(sessionId); + return ok({ target_state: "unknown" }); + } + try { + return await this.#serialize( + session, + async () => { + if (session.exited) { + const stopped = await this.#stopDebuggerOnly(session); + if (stopped.status === "incomplete") + return err( + this.#retainCleanupFailure( + session, + "close_gdb_session", + stopped.reason, + ), + ); + this.#sessions.delete(sessionId); + return ok({ target_state: "unknown" }); + } + let targetState: "none_observed" | "unknown" = "unknown"; + try { + const groups = await this.#request( + session, + "-list-thread-groups --recurse 1", + ); + if ( + groups.completion_status !== "complete" || + groups.output_truncated || + !/^\d+\^done(?:,|$)/u.test(groups.mi) + ) + throw new Error( + "GDB could not confirm its complete, untruncated inferior list before shutdown.", + ); + const groupCount = gdbThreadGroupCount(groups.mi); + if (groupCount === undefined) + throw new Error( + "GDB returned an unknown inferior list before shutdown.", + ); + if (groupCount > 0 && !session.inferiorTtyVerified) + return err( + new AnalysisInputError("close_gdb_session", undefined, [ + { + path: ["session_id"], + reason: "invalid_value", + message: + "Live GDB thread groups remain and no verified durable inferior TTY is configured. Select a safe disposition through the GDB console or restart with inferior_tty before closing.", + }, + ]), + ); + if (groupCount === 0) { + session.noInferiorVerified = true; + targetState = "none_observed"; + await this.#request(session, "-gdb-exit"); + } else { + session.noInferiorVerified = false; + const disposition = await this.#detachInferior(session); + targetState = + disposition.status === "no-inferior" + ? "none_observed" + : "unknown"; + } + } catch { + // If MI synchronization is lost, stop only the owned debugger process. + } + const stopped = await this.#stopDebuggerOnly(session); + if (stopped.status === "incomplete") + return err( + this.#retainCleanupFailure( + session, + "close_gdb_session", + stopped.reason, + ), + ); + this.#sessions.delete(sessionId); + return ok({ target_state: targetState }); + }, + undefined, + "close_gdb_session", + ); + } catch (cause) { + if (cause instanceof AnalysisError) return err(cause); + throw cause; + } + } + + async retryCleanupFailures(): Promise> { + for (const session of this.#sessions.values()) { + if (session.cleanupFailures.size === 0) continue; + const stopped = await this.#stopDebuggerOnly(session); + if (stopped.status === "incomplete") + return err( + [...session.cleanupFailures].at(-1) ?? + this.#retainCleanupFailure( + session, + "close_gdb_session", + stopped.reason, + ), + ); + this.#sessions.delete(session.id); + } + return ok(null); + } + + async closeAll(): Promise { + this.#closing = true; + this.#closePromise ??= this.#drainStartsAndCloseSessions().catch( + (cause: unknown) => { + this.#closePromise = undefined; + throw cause; + }, + ); + return this.#closePromise; + } + + async #drainStartsAndCloseSessions(): Promise { + const active = [...this.#activeStarts.entries()]; + for (const [controller] of active) controller.abort(); + await Promise.allSettled(active.map(([, start]) => start)); + await Promise.all( + [...this.#sessions.values()].map((session) => session.commandTail), + ); + await closeOwnedDebuggerSessions( + this.#sessions, + (session) => this.#stopDebuggerOnly(session), + "GDB", + ); + } + + async #serialize( + session: GdbSession, + operation: () => Promise, + signal?: AbortSignal, + operationName: "gdb_console" | "close_gdb_session" = "gdb_console", + ): Promise { + if (session.pendingCommandCount >= MAX_PENDING_GDB_COMMANDS) + throw new AnalysisResourceConstraintError( + operationName, + "transport", + "GDB command queue capacity reached.", + { + max_pending_commands: MAX_PENDING_GDB_COMMANDS, + active_or_queued_commands: session.pendingCommandCount, + }, + ); + session.pendingCommandCount += 1; + const previous = session.commandTail; + let release: (() => void) | undefined; + let acquired = false; + session.commandTail = new Promise((resolve) => { + release = resolve; + }); + try { + await waitForGdbCommandTurn(previous, signal); + acquired = true; + return await operation(); + } finally { + if (acquired) { + session.pendingCommandCount -= 1; + release?.(); + } else { + // The cancelled placeholder still occupies commandTail until its turn. + // Keep its reservation until then so repeated cancellations cannot + // bypass the per-session queue bound while an earlier command stalls. + void previous.then(() => { + session.pendingCommandCount -= 1; + release?.(); + }); + } + } + } + + #receive(session: GdbSession, data: Buffer): void { + if (session.startupResolve !== undefined) { + const prompt = session.promptTail + data.toString("utf8"); + if (prompt.includes("(gdb)")) { + const resolve = session.startupResolve; + session.startupResolve = undefined; + resolve(); + } + session.promptTail = prompt.slice(-4); + } + const lines: { readonly value: string; readonly invalidUtf8: boolean }[] = + []; + let offset = 0; + while (offset < data.byteLength) { + const newline = data.indexOf(0x0a, offset); + const piece = data.subarray( + offset, + newline < 0 ? data.byteLength : newline, + ); + session.bufferBytes += piece.byteLength; + const lineBytes = + session.bufferBytes - (newline >= 0 && piece.at(-1) === 0x0d ? 1 : 0); + if (lineBytes > MAX_MI_LINE_BYTES) { + this.#failProtocol( + session, + "GDB emitted an MI line exceeding the 16 MiB protocol limit.", + ); + return; + } + this.#appendLineBytes(session, piece); + if (newline < 0) break; + const record = Buffer.concat( + [ + ...session.bufferChunks, + ...(session.bufferTail === undefined + ? [] + : [session.bufferTail.subarray(0, session.bufferTailBytes)]), + ], + session.bufferBytes, + ); + const withoutCarriageReturn = + record.at(-1) === 0x0d ? record.subarray(0, -1) : record; + const invalidUtf8 = !isUtf8(withoutCarriageReturn); + const line = withoutCarriageReturn.toString("utf8"); + session.bufferChunks.length = 0; + session.bufferTail = undefined; + session.bufferTailBytes = 0; + session.bufferBytes = 0; + if (invalidUtf8) session.invalidUtf8Observed = true; + if (line.length > 0) lines.push({ value: line, invalidUtf8 }); + offset = newline + 1; + } + for (const { value: line } of lines) { + if (Buffer.byteLength(line, "utf8") > MAX_RECENT_MI_LINE_BYTES) { + session.lines.push(truncateUtf8RecordPrefix(line)); + session.recentRecordsTruncated = true; + } else session.lines.push(line); + } + if (session.lines.length > 256) { + session.lines.splice(0, session.lines.length - 256); + session.recentRecordsTruncated = true; + } + for (const { value: line, invalidUtf8 } of lines) { + for (const pending of session.pending.values()) { + const lineBytes = Buffer.byteLength(line, "utf8") + 1; + if ( + pending.outputBytes + lineBytes <= session.maxCommandOutputBytes && + pending.outputBytes + lineBytes <= MAX_COMMAND_OUTPUT_BYTES && + pending.lines.length < MAX_COMMAND_OUTPUT_RECORDS + ) { + pending.lines.push(line); + pending.outputBytes += lineBytes; + pending.outputInvalidUtf8 ||= invalidUtf8; + } else pending.outputTruncated = true; + } + if (line.startsWith("*stopped")) { + for (const [token, pending] of session.pending) { + if (!pending.awaitingInferiorStop) continue; + clearTimeout(pending.timer); + session.pending.delete(token); + pending.cleanup(); + const records = [...pending.lines]; + pending.resolve({ + command: "", + mi: pending.terminalRecord ?? "", + records, + console: records + .filter((record) => record.startsWith("~")) + .join("\n"), + target: records + .filter((record) => record.startsWith("@")) + .join("\n"), + log: records.filter((record) => record.startsWith("&")).join("\n"), + output_truncated: pending.outputTruncated, + ...(pending.outputInvalidUtf8 + ? { output_invalid_utf8: true as const } + : {}), + completion_status: "complete", + }); + } + continue; + } + const match = /^(\d+)\^(done|running|connected|error|exit)(?:,|$)/u.exec( + line, + ); + if (match === null) continue; + const token = Number(match[1]); + const pending = session.pending.get(token); + if (pending === undefined) continue; + if (match[2] === "running") { + pending.terminalRecord = line; + pending.awaitingInferiorStop = true; + continue; + } + clearTimeout(pending.timer); + session.pending.delete(token); + pending.cleanup(); + const records = [...pending.lines]; + pending.resolve({ + command: "", + mi: line, + records, + console: records.filter((record) => record.startsWith("~")).join("\n"), + target: records.filter((record) => record.startsWith("@")).join("\n"), + log: records.filter((record) => record.startsWith("&")).join("\n"), + output_truncated: pending.outputTruncated, + ...(pending.outputInvalidUtf8 + ? { output_invalid_utf8: true as const } + : {}), + completion_status: "complete", + }); + } + } + + #appendLineBytes(session: GdbSession, piece: Buffer): void { + const slabBytes = 64 * 1024; + let offset = 0; + while (offset < piece.length) { + session.bufferTail ??= Buffer.allocUnsafe(slabBytes); + const remaining = slabBytes - session.bufferTailBytes; + const copied = Math.min(remaining, piece.length - offset); + piece.copy( + session.bufferTail, + session.bufferTailBytes, + offset, + offset + copied, + ); + session.bufferTailBytes += copied; + offset += copied; + if (session.bufferTailBytes === slabBytes) { + session.bufferChunks.push(session.bufferTail); + session.bufferTail = undefined; + session.bufferTailBytes = 0; + } + } + } + + #failProtocol(session: GdbSession, message: string): void { + if (session.exited || session.protocolError !== undefined) return; + session.protocolError = message; + for (const pending of session.pending.values()) { + clearTimeout(pending.timer); + pending.cleanup(); + const records = [...pending.lines]; + pending.reject( + new GdbProtocolFailureError(message, { + command: pending.command, + mi: "", + records, + console: records.filter((line) => line.startsWith("~")).join("\n"), + target: records.filter((line) => line.startsWith("@")).join("\n"), + log: records.filter((line) => line.startsWith("&")).join("\n"), + output_truncated: pending.outputTruncated, + ...(pending.outputInvalidUtf8 + ? { output_invalid_utf8: true as const } + : {}), + completion_status: "unknown", + }), + ); + } + session.pending.clear(); + session.startupReject?.(new Error(message)); + } + + async #stopAfterProtocolFailure( + session: GdbSession, + cause: unknown, + ): Promise { + const stopped = await this.#stopDebuggerOnly(session); + const partial = + cause instanceof GdbProtocolFailureError ? cause.partial : undefined; + const partialObservation = + partial === undefined + ? undefined + : createEvidence(undefined, GDB_PROVIDER_IDENTITY, { + predicateType: "rea.debugger.console-observation", + operation: "gdb_console", + parameters: { + session_id: session.id, + command: partial.command, + }, + result: { ...partial, records: [...partial.records] }, + rawResult: { ...partial, records: [...partial.records] }, + subjectUnavailableReason: + "The active GDB inferior identity is not established by the debugger session identifier.", + limitations: [ + "The GDB MI protocol failed; retained records are partial and command completion is unknown.", + "The unrestricted GDB console can access local shell, filesystem, network, and target controls.", + ], + }); + return new AnalysisCapabilityUnavailableError( + "gdb", + "gdb_console", + stopped.status === "incomplete" + ? `${session.protocolError} The owned process could not be confirmed stopped: ${stopped.reason}` + : session.protocolError!, + { + cause, + ...(stopped.status === "incomplete" + ? { + cleanup: { + reason: stopped.reason, + resources: [session.id], + }, + } + : {}), + ...(partialObservation === undefined ? {} : { partialObservation }), + }, + ); + } + + #stopDebuggerOnly( + session: GdbSession, + ): ReturnType { + if (session.stopAttempt !== undefined) return session.stopAttempt; + const attempt = this.#stopProcess(session, "GDB") + .then((stopped) => { + if (stopped.status !== "incomplete") { + for (const failure of session.cleanupFailures) + failure.resolveCleanup(); + session.cleanupFailures.clear(); + } + return stopped; + }) + .finally(() => { + if (session.stopAttempt === attempt) session.stopAttempt = undefined; + }); + session.stopAttempt = attempt; + return attempt; + } + + #retainCleanupFailure( + session: GdbSession, + operation: string, + reason: string, + options: AnalysisErrorOptions = {}, + ): AnalysisError { + const failure = new AnalysisCapabilityUnavailableError( + "gdb", + operation, + `The owned GDB process could not be confirmed stopped: ${reason}`, + { ...options, cleanup: { reason, resources: [session.id] } }, + ); + session.cleanupFailures.add(failure); + return failure; + } + + async #detachInferior( + session: GdbSession, + ): Promise { + if (session.noInferiorVerified) return { status: "no-inferior" }; + if (session.exited) + throw new Error("GDB exited before inferior state could be verified."); + // A new MI inspection replaces the old snapshot. If this read is + // incomplete, malformed, or reports multiple inferiors, no earlier PID + // can stand in for the current unknown inventory. + session.knownInferiorIdentities.clear(); + if ( + session.protocolError !== undefined || + session.pending.size > 0 || + session.pendingCommandCount > 1 + ) + throw new Error( + "GDB MI state is not synchronized; inferior state is unknown.", + ); + const groups = await this.#request( + session, + "-list-thread-groups --recurse 1", + ); + if ( + groups.completion_status !== "complete" || + groups.output_truncated || + !/^\d+\^done(?:,|$)/u.test(groups.mi) + ) + throw new Error("GDB could not confirm its inferior state."); + const inferiorPids = gdbThreadGroupPids(groups.mi); + if (inferiorPids?.length === 0) { + session.knownInferiorIdentities.clear(); + session.noInferiorVerified = true; + return { status: "no-inferior" }; + } + session.noInferiorVerified = false; + if (inferiorPids?.length !== 1) + throw new Error( + inferiorPids === undefined + ? "GDB returned an unknown recursive inferior list." + : "GDB reports multiple inferiors; automatic detach cannot be confirmed safely.", + ); + if (!session.inferiorTtyVerified) + throw new Error( + "GDB cannot detach the live inferior because no durable output sink was verified at session startup.", + ); + const observedTty = await this.#request(session, "-inferior-tty-show"); + if ( + observedTty.completion_status !== "complete" || + observedTty.output_truncated || + !/^\d+\^done(?:,|$)/u.test(observedTty.mi) || + miStringField(observedTty.mi, "inferior_tty_terminal") !== + session.inferiorTty + ) + throw new Error( + "GDB no longer reports the caller-selected inferior TTY; detachment cannot be confirmed safely.", + ); + await this.#verifyInferiorOutputSink( + inferiorPids[0]!, + session.inferiorTty!, + (identity) => + session.knownInferiorIdentities.set( + `${identity.pid}:${identity.startTime}`, + identity, + ), + ); + const detached = await this.#request(session, "-target-detach"); + if ( + detached.completion_status !== "complete" || + detached.output_truncated || + !/^\d+\^done(?:,|$)/u.test(detached.mi) + ) + throw new Error("GDB did not confirm inferior detachment."); + session.knownInferiorIdentities.clear(); + return { status: "detached", outputSink: "durable" }; + } + + async #settleExitedCommands( + session: GdbSession, + code: number | null, + signalCode: NodeJS.Signals | null, + ): Promise { + await this.#waitForStdoutSettlement(session); + for (const pending of session.pending.values()) { + clearTimeout(pending.timer); + pending.reject( + new GdbCommandProcessExitError({ + command: pending.command, + mi: "", + records: [...pending.lines], + console: pending.lines + .filter((line) => line.startsWith("~")) + .join("\n"), + target: pending.lines + .filter((line) => line.startsWith("@")) + .join("\n"), + log: pending.lines.filter((line) => line.startsWith("&")).join("\n"), + process_exit_observed: true, + process_exit_code: code, + process_exit_signal: signalCode, + output_truncated: pending.outputTruncated, + ...(pending.outputInvalidUtf8 + ? { output_invalid_utf8: true as const } + : {}), + completion_status: "unknown", + }), + ); + } + session.pending.clear(); + } + + async #waitForStdoutSettlement(session: GdbSession): Promise { + let timer!: ReturnType; + try { + await Promise.race([ + session.stdoutSettled, + new Promise((resolve) => { + timer = setTimeout(resolve, 1_000); + timer.unref?.(); + }), + ]); + } finally { + clearTimeout(timer); + } + } + + #request( + session: GdbSession, + command: string, + signal?: AbortSignal, + operation: "gdb_console" | "start_gdb_session" = "gdb_console", + evidenceCommand: string = command, + ): Promise { + const stdin = session.launched.process.stdin; + if (session.exited) + return Promise.reject( + new Error( + `GDB MI process has exited (exit code ${String(session.exitCode)}, signal ${String(session.exitSignal)}).`, + ), + ); + if ( + stdin === null || + stdin === undefined || + stdin.destroyed || + !stdin.writable + ) { + const message = "GDB MI input stream is closed"; + this.#failProtocol(session, message); + return Promise.reject(new Error(message)); + } + const token = session.nextToken++; + return new Promise((resolve, reject) => { + let abortListener!: () => void; + const cleanup = (): void => { + signal?.removeEventListener("abort", abortListener); + }; + let pending!: PendingCommand; + const timer = setTimeout(() => { + session.pending.delete(token); + session.protocolError = + "A GDB MI command timed out; untagged stream output can no longer be safely correlated with later commands."; + pending.cleanup(); + const partial: GdbCommandResult = { + command: pending.evidenceCommand, + mi: "", + records: [...pending.lines], + console: pending.lines + .filter((line) => line.startsWith("~")) + .join("\n"), + target: pending.lines + .filter((line) => line.startsWith("@")) + .join("\n"), + log: pending.lines.filter((line) => line.startsWith("&")).join("\n"), + output_truncated: pending.outputTruncated, + ...(pending.outputInvalidUtf8 + ? { output_invalid_utf8: true as const } + : {}), + completion_status: "unknown", + }; + const timeout = new AnalysisTimeoutError( + operation, + this.#commandTimeoutMs, + { cause: new Error(`GDB MI command timed out: ${command}`) }, + ); + const evidence = createEvidence(undefined, GDB_PROVIDER_IDENTITY, { + predicateType: + operation === "start_gdb_session" + ? "rea.debugger.startup-observation" + : "rea.debugger.console-observation", + operation, + parameters: { + session_id: session.id, + command: pending.evidenceCommand, + }, + result: { ...partial, records: [...partial.records] }, + rawResult: { ...partial, records: [...partial.records] }, + subjectUnavailableReason: + "The active GDB inferior identity is not established by the debugger session identifier.", + limitations: [ + "The GDB command exceeded its deadline; retained MI and stream records are partial and command completion is unknown.", + "Cleanup of the owned GDB process was requested because output correlation could no longer be guaranteed.", + "The unrestricted GDB console can access local shell, filesystem, network, and target controls.", + ], + }); + void this.#stopDebuggerOnly(session).then( + (stopped) => { + if ( + stopped.status === "incomplete" && + operation === "start_gdb_session" + ) { + pending.reject( + new AnalysisTimeoutError(operation, this.#commandTimeoutMs, { + cause: timeout, + cleanup: { reason: stopped.reason, resources: [session.id] }, + partialObservation: evidence, + }), + ); + return; + } + pending.reject( + stopped.status === "incomplete" + ? new AnalysisCapabilityUnavailableError( + "gdb", + "gdb_console", + `GDB command output could not be correlated and its owned process could not be confirmed stopped: ${stopped.reason}`, + { + cause: timeout, + cleanup: { + reason: stopped.reason, + resources: [session.id], + }, + partialObservation: evidence, + }, + ) + : new GdbCommandTimeoutError( + partial, + evidence, + this.#commandTimeoutMs, + ), + ); + }, + (cause: unknown) => + pending.reject( + new AnalysisCapabilityUnavailableError( + "gdb", + "gdb_console", + "GDB command output could not be correlated and owned process shutdown failed.", + { + cause: new AggregateError( + [timeout, cause], + "GDB timeout and cleanup both failed.", + ), + cleanup: { + reason: gdbErrorMessage(cause), + resources: [session.id], + }, + partialObservation: evidence, + }, + ), + ), + ); + }, this.#commandTimeoutMs); + pending = { + token, + command, + evidenceCommand, + lines: [], + resolve: (result) => { + cleanup(); + resolve(result); + }, + reject: (error) => { + cleanup(); + reject(error); + }, + cleanup, + timer, + outputBytes: 0, + outputTruncated: false, + outputInvalidUtf8: false, + terminalRecord: undefined, + awaitingInferiorStop: false, + }; + session.pending.set(token, pending); + abortListener = (): void => { + clearTimeout(pending.timer); + const partial: GdbCommandResult = { + command: pending.evidenceCommand, + mi: "", + records: [...pending.lines], + console: pending.lines + .filter((line) => line.startsWith("~")) + .join("\n"), + target: pending.lines + .filter((line) => line.startsWith("@")) + .join("\n"), + log: pending.lines.filter((line) => line.startsWith("&")).join("\n"), + output_truncated: pending.outputTruncated, + ...(pending.outputInvalidUtf8 + ? { output_invalid_utf8: true as const } + : {}), + completion_status: "unknown", + }; + const evidence = createEvidence(undefined, GDB_PROVIDER_IDENTITY, { + predicateType: "rea.debugger.console-observation", + operation: "gdb_console", + parameters: { + session_id: session.id, + command: pending.evidenceCommand, + }, + result: { ...partial, records: [...partial.records] }, + rawResult: { ...partial, records: [...partial.records] }, + subjectUnavailableReason: + "The active GDB inferior identity is not established by the debugger session identifier.", + limitations: [ + "The GDB command was cancelled; retained MI records are partial and command completion is unknown.", + "Shutdown of the owned GDB process was requested because output correlation could no longer be guaranteed.", + "The unrestricted GDB console can access local shell, filesystem, network, and target controls.", + ], + }); + session.pending.delete(token); + session.protocolError = + "A GDB MI command was cancelled; owned-session shutdown was requested because output correlation can no longer be guaranteed."; + void this.#stopDebuggerOnly(session).then( + (stopped) => { + pending.reject( + stopped.status === "incomplete" + ? new AnalysisCancelledError("gdb_console", { + cause: new Error(stopped.reason), + cleanup: { + reason: stopped.reason, + resources: [session.id], + }, + partialObservation: evidence, + }) + : new AnalysisCancelledError("gdb_console", { + partialObservation: evidence, + }), + ); + }, + (cause: unknown) => + pending.reject( + new AnalysisCancelledError("gdb_console", { + cause, + partialObservation: evidence, + }), + ), + ); + }; + signal?.addEventListener("abort", abortListener, { once: true }); + if (signal?.aborted) { + abortListener(); + return; + } + try { + session.launched.process.stdin?.write( + `${token}${command}\n`, + "utf8", + (cause?: Error | null) => { + if (cause) + this.#failProtocol( + session, + `GDB stdin write failed: ${cause.message}`, + ); + }, + ); + } catch (cause) { + this.#failProtocol( + session, + `GDB stdin write failed: ${cause instanceof Error ? cause.message : "unknown error"}`, + ); + } + }).then((result) => ({ ...result, command })); + } +} + +const miQuote = (value: string): string => + `"${value.replaceAll("\\", "\\\\").replaceAll('"', '\\"').replaceAll("\r", "\\r").replaceAll("\n", "\\n").replaceAll("\t", "\\t")}"`; + +/** @internal Decode one unambiguous GDB MI C-string field. */ +export const miStringField = ( + record: string, + field: string, +): string | undefined => { + const escapedField = field.replace(/[.*+?^${}()|[\]\\]/gu, "\\$&"); + const valuePattern = `((?:\\\\.|[^"\\\\])*)`; + const fieldPattern = new RegExp( + `(?:^|,)${escapedField}="${valuePattern}"`, + "gu", + ); + const matches = [...record.matchAll(fieldPattern)]; + if (matches.length !== 1) return undefined; + const match = new RegExp( + `^\\d+\\^done,${escapedField}="${valuePattern}"$`, + "u", + ).exec(record); + const raw = match?.[1]; + if (raw === undefined) return undefined; + const bytes: number[] = []; + for (let index = 0; index < raw.length;) { + const character = raw[index]!; + if (character !== "\\") { + const codePoint = raw.codePointAt(index); + if ( + codePoint === undefined || + (codePoint >= 0xd800 && codePoint <= 0xdfff) + ) + return undefined; + const value = String.fromCodePoint(codePoint); + bytes.push(...Buffer.from(value, "utf8")); + index += value.length; + continue; + } + const escaped = raw[++index]!; + const simple: Readonly> = { + a: 7, + b: 8, + f: 12, + n: 10, + r: 13, + t: 9, + v: 11, + "\\": 92, + '"': 34, + "'": 39, + "?": 63, + }; + const simpleByte = simple[escaped]; + if (simpleByte !== undefined) { + bytes.push(simpleByte); + index += 1; + continue; + } + if (escaped === "x") { + const hex = /^[0-9a-f]+/iu.exec(raw.slice(index + 1))?.[0]; + if (hex === undefined) return undefined; + bytes.push(Number.parseInt(hex, 16) & 0xff); + index += hex.length; + index += 1; + continue; + } + if (/[0-7]/u.test(escaped)) { + const octal = `${escaped}${/^[0-7]{0,2}/u.exec(raw.slice(index + 1))![0]}`; + bytes.push(Number.parseInt(octal, 8) & 0xff); + index += octal.length; + continue; + } + return undefined; + } + const decoded = Buffer.from(bytes).toString("utf8"); + return Buffer.from(decoded, "utf8").equals(Buffer.from(bytes)) + ? decoded + : undefined; +}; + +/** @internal Classify commands that can invalidate the configured output sink. */ +export const gdbOneShotCommandMayInvalidateDetachProof = ( + command: string, +): boolean => { + if (commandMayChangeInferiorOutput(command)) return true; + const firstToken = /^\s*([^\s;]+)/u.exec(command)?.[1]?.toLowerCase(); + if ( + firstToken !== undefined && + (["source", "define", "alias"].some((name) => + name.startsWith(firstToken), + ) || + firstToken === "py" || + "python".startsWith(firstToken)) + ) + return true; + // One-shot execution cannot establish which shell/GDB context interprets + // redirections, nor safely track multiline command definitions. + return /[<>]{1,2}|&>|[\r\n;]/u.test(command); +}; + +/** @internal Classify commands that can invalidate the configured output sink. */ +export const commandMayChangeInferiorOutput = (command: string): boolean => { + const setting = /^\s*(?:set|with)\s+(\S+)/iu.exec(command)?.[1]; + const maySetInferiorTty = + setting !== undefined && "inferior-tty".startsWith(setting.toLowerCase()); + const firstToken = /^\s*([^\s;]+)/u.exec(command)?.[1]?.toLowerCase(); + const mayBeTtyCommand = + firstToken !== undefined && "tty".startsWith(firstToken); + const mayDefineAlias = + firstToken !== undefined && "alias".startsWith(firstToken); + const mayRunWithOutputRedirection = + firstToken !== undefined && + ["run", "start", "starti"].some((name) => name.startsWith(firstToken)) && + /[<>]{1,2}|&>/u.test(command.slice(firstToken.length)); + return ( + maySetInferiorTty || + mayBeTtyCommand || + mayDefineAlias || + /\binferior-tty\b/iu.test(command) || + /^\s*(?:wit(?:h)?(?:\s|$)|tty(?:\s|$)|source\b|py\w*|define\b)/iu.test( + command, + ) || + mayRunWithOutputRedirection + ); +}; + +/** Terminate only the directly owned debugger process, never its process group or inferiors. */ +export const waitForGdbCommandTurn = ( + previous: Promise, + signal?: AbortSignal, +): Promise => { + if (signal === undefined) return previous; + if (signal.aborted) + return Promise.reject(new AnalysisCancelledError("gdb_console")); + return new Promise((resolve, reject) => { + const onAbort = (): void => + reject(new AnalysisCancelledError("gdb_console")); + signal.addEventListener("abort", onAbort, { once: true }); + previous.then( + () => { + signal.removeEventListener("abort", onAbort); + resolve(); + }, + (cause: unknown) => { + signal.removeEventListener("abort", onAbort); + reject(cause); + }, + ); + }); +}; diff --git a/src/generatedPackageMetadata.ts b/src/generatedPackageMetadata.ts index bada606633..baa7bf1434 100644 --- a/src/generatedPackageMetadata.ts +++ b/src/generatedPackageMetadata.ts @@ -1,9 +1,9 @@ /** Generated from package.json and package-lock.json; do not edit. */ export const PACKAGE_METADATA = { name: "rea-agents", - version: "6.4.0", // x-release-please-version + version: "6.5.0", // x-release-please-version serverSdkVersion: "2.3.1", clientSdkVersion: "2.3.1", coreSdkVersion: "2.3.1", - skillVersion: "41", + skillVersion: "42", } as const; diff --git a/src/ghidra/GhidraAnalysisProfile.ts b/src/ghidra/GhidraAnalysisProfile.ts index d4bd5a4330..9c7f0b254e 100644 --- a/src/ghidra/GhidraAnalysisProfile.ts +++ b/src/ghidra/GhidraAnalysisProfile.ts @@ -14,6 +14,7 @@ import type { AnalysisError } from "../domain/analysisErrorBase.js"; import { err, ok, type Result } from "../domain/result.js"; import type { GhidraInstallationInspection } from "./GhidraInstallation.js"; import { + isLegacyMipsAdmission, ghidraMipsProfileParameters, ghidraMipsLanguageOverride, } from "./GhidraMipsProfile.js"; @@ -21,6 +22,10 @@ import { ghidraRiscvLanguageOverride, ghidraRiscvProfileParameters, } from "./GhidraRiscvProfile.js"; +import { + ghidraPowerPcLanguage, + ghidraPowerPcProfileParameters, +} from "./GhidraPowerPcProfile.js"; import { ghidraRawBinaryLanguage } from "./GhidraRawBinaryLanguage.js"; import { @@ -36,6 +41,11 @@ import { NATIVE_AOT_PROFILE_CONTRACT_REVISION, } from "../domain/native/nativeAotPe.js"; +const callRelationshipCoverage = "function-body-call-coverage-v5"; +const functionBoundaryObservations = "ghidra-function-boundary-limitations-v4"; +const instructionReferenceCoverage = + "exact-address-incoming-without-synthetic-entry-v2"; + /** * The processor language REA pins for this target, or undefined when Ghidra may * select from the header. The committed profile and the session readback both @@ -44,7 +54,9 @@ import { export const ghidraPinnedLanguage = ( target: BinaryTarget, ): GhidraLanguageOverride | undefined => - ghidraMipsLanguageOverride(target) ?? ghidraRiscvLanguageOverride(target); + ghidraMipsLanguageOverride(target) ?? + ghidraRiscvLanguageOverride(target) ?? + ghidraPowerPcLanguage(target); /** Resolve version-bound, deterministic semantics before Ghidra imports a target. */ export const resolveGhidraAnalysisProfile = async ( @@ -119,14 +131,16 @@ export const resolveGhidraAnalysisProfile = async ( load_image_observations: "source-mappings-entry-context-v2", function_body_evidence: "complete-inclusive-ranges-v1", function_references: "complete-body-and-entry-reference-manager-v2", - call_relationship_coverage: "resolved-call-reference-diagnostics-v1", + call_relationship_coverage: callRelationshipCoverage, + function_dossier_coverage: + "observed-facets-and-call-relationships-v1", string_inventory_evidence: "defined-data-coverage-v1", location_resolution: "explicit-address-exact-entry-symbol-first-v3", procedure_file_offset_resolution: "address-first-unique-target-source-mapping-v2", instruction_flow_evidence: "decoded-return-pcode-v1", - instruction_references: "exact-address-incoming-v1", - function_boundary_observations: "ghidra-terminal-call-limitations-v2", + instruction_references: instructionReferenceCoverage, + function_boundary_observations: functionBoundaryObservations, no_return_repair: installation.platform === "win32" ? "disabled-windows-p0" @@ -153,8 +167,12 @@ export const resolveGhidraAnalysisProfile = async ( const dosCom = target.format === "dos-com"; const dos = dosMz || dosCom; const pinnedLanguage = ghidraPinnedLanguage(target); + const callerMaySelectLegacyMipsLanguage = + target.architecture === "mips" && isLegacyMipsAdmission(target); if ( pinnedLanguage !== undefined && + target.architecture !== "powerpc" && + !callerMaySelectLegacyMipsLanguage && languageOverride !== undefined && (languageOverride.languageId !== pinnedLanguage.languageId || (languageOverride.compilerSpecId !== undefined && @@ -207,20 +225,21 @@ export const resolveGhidraAnalysisProfile = async ( ].sort(), import_mode: "ephemeral-source-immutable", ...(target.format === "pe" - ? { pdb_observations: "program-info-v1" } + ? { pdb_observations: "program-info-v5" } : {}), annotation_policy: "atomic-function-entry-metadata-v1", load_image_observations: "source-mappings-entry-context-v2", function_body_evidence: "complete-inclusive-ranges-v1", function_references: "complete-body-and-entry-reference-manager-v2", - call_relationship_coverage: "resolved-call-reference-diagnostics-v1", + call_relationship_coverage: callRelationshipCoverage, + function_dossier_coverage: "observed-facets-and-call-relationships-v1", string_inventory_evidence: "defined-data-coverage-v1", location_resolution: "explicit-address-exact-entry-symbol-first-v3", procedure_file_offset_resolution: "address-first-unique-target-source-mapping-v2", instruction_flow_evidence: "decoded-return-pcode-v1", - instruction_references: "exact-address-incoming-v1", - function_boundary_observations: "ghidra-terminal-call-limitations-v2", + instruction_references: instructionReferenceCoverage, + function_boundary_observations: functionBoundaryObservations, no_return_repair: installation.platform === "win32" ? "disabled-windows-p0" @@ -277,6 +296,7 @@ export const resolveGhidraAnalysisProfile = async ( ...analyzerParameters, ...ghidraMipsProfileParameters(target, languageOverride), ...ghidraRiscvProfileParameters(target), + ...ghidraPowerPcProfileParameters(target), ...(pspExtension?.ok === true ? { mips_support_lane: "psp-elf32-exec-eabi32-allegrex-v2", diff --git a/src/ghidra/GhidraAnalysisSeeds.test.ts b/src/ghidra/GhidraAnalysisSeeds.test.ts index 37437ebf7c..7693573732 100644 --- a/src/ghidra/GhidraAnalysisSeeds.test.ts +++ b/src/ghidra/GhidraAnalysisSeeds.test.ts @@ -23,7 +23,7 @@ import { type GhidraSeedCommitment, type GhidraSeedReport, } from "./GhidraAnalysisSeeds.js"; -import type { GhidraInstallationHost } from "./GhidraInstallation.js"; +import { ghidraProfileInstallationHost as host } from "./GhidraInstallation.fixture.js"; import { GhidraProvider } from "./GhidraProvider.js"; const directories: string[] = []; @@ -193,19 +193,6 @@ describe("Ghidra seed session verification", () => { }); describe("Ghidra seed profile", () => { - const host: GhidraInstallationHost = { - platform: "linux", - architecture: "x64", - readText: () => "application.version=12.1.4\napplication.java.min=21\n", - executable: () => true, - probeJava: () => ({ - version: "21.0.11", - major: 21, - home: "/jdk-21", - bits: 64, - runtime: "jdk", - }), - }; const pe: BinaryTarget = { path: "/fixture.exe", sha256: "a".repeat(64), diff --git a/src/ghidra/GhidraClient.ts b/src/ghidra/GhidraClient.ts index d1dee3f01a..acbd58ee68 100644 --- a/src/ghidra/GhidraClient.ts +++ b/src/ghidra/GhidraClient.ts @@ -58,6 +58,7 @@ import { sameRegularFileState, } from "../filesystem/RegularFile.js"; import { OwnedFileHandle } from "../filesystem/OwnedFileHandle.js"; +import { redactCapturedTransportCredential } from "../process/ProviderDiagnosticRedaction.js"; export type GhidraTargetSnapshot = | { readonly kind: "captured"; readonly bytes: Buffer } @@ -119,7 +120,7 @@ export class GhidraClient { #closePromise: Promise | undefined; readonly #failure = bindGhidraSessionFailure( () => this.#diagnostics(), - (value) => this.#redactAuthentication(value), + (value, incomplete) => this.#redactAuthentication(value, incomplete), ); readonly #wire: GhidraWire; readonly #requestQueue: GhidraRequestQueue; @@ -662,25 +663,47 @@ export class GhidraClient { maxDiagnosticBytes: GHIDRA_PROCESS_DIAGNOSTIC_BYTES, onDiagnostic: (event) => this.#onProcessDiagnostic(event), }); - const connected = await this.#connect(endpoint, deadline); - if (!connected.ok) { - const failure = connected.error; - return this.#startupFailure(failure); - } - const completed = await completeGhidraStartupHandshake({ - deadline, - request: (method, params, requestOptions) => - this.#wire.request(method, params, requestOptions), - parseSessionInfo: (value) => this.#parseSessionInfo(value), - failure: this.#failure, - startupTimeoutMs: this.#options.startupTimeoutMs, - }); + const completed = await this.#connectAndHandshake(endpoint, deadline); if (!completed.ok) return this.#startupFailure(completed.error); await this.#lineage.observe(this.#launch); this.#requestQueue.reopen(); return completed; } + async #connectAndHandshake( + endpoint: GhidraEndpoint, + deadline: ProviderStartupDeadline, + ): Promise { + // Retry only a dropped loopback transport, never identity or protocol failures. + for (let attempt = 0; ; attempt += 1) { + const connected = await this.#connect(endpoint, deadline); + if (!connected.ok) return connected; + const completed = await completeGhidraStartupHandshake({ + deadline, + request: (method, params, requestOptions) => + this.#wire.request(method, params, requestOptions), + parseSessionInfo: (value) => this.#parseSessionInfo(value), + failure: this.#failure, + startupTimeoutMs: this.#options.startupTimeoutMs, + }); + if ( + completed.ok || + completed.error.kind !== "process" || + endpoint.transport !== "authenticated-loopback-tcp" || + attempt >= 2 || + deadline.signal.aborted || + this.#closing || + this.#process?.observedExit() !== undefined + ) + return completed; + this.#detachSocket?.(); + this.#detachSocket = undefined; + this.#socket?.destroy(); + this.#socket = undefined; + this.#responseBuffer.reset(); + } + } + async #connect( endpoint: GhidraEndpoint, deadline: ProviderStartupDeadline, @@ -880,9 +903,15 @@ export class GhidraClient { ); } - #redactAuthentication(value: string): string { - for (const token of this.#authenticationTokens) - value = value.replaceAll(token, "[REDACTED]"); + #redactAuthentication(value: string, incomplete = false): string { + for (const token of this.#authenticationTokens) { + const bytes = Buffer.byteLength(value); + value = redactCapturedTransportCredential( + { text: value, bytes, observedBytes: bytes + (incomplete ? 1 : 0) }, + token, + "[REDACTED]", + ); + } return value; } diff --git a/src/ghidra/GhidraDoctor.test.ts b/src/ghidra/GhidraDoctor.test.ts index 844438408a..0a98bcb759 100644 --- a/src/ghidra/GhidraDoctor.test.ts +++ b/src/ghidra/GhidraDoctor.test.ts @@ -4,7 +4,7 @@ import { runDoctor, type DoctorHost } from "../application/Doctor.js"; import { createDoctorHostFixture } from "../application/Doctor.fixture.js"; import { providerRegistrationEnvironment } from "../application/SetupRegistrationEnvironment.js"; import { WINDOWS_NATIVE_AUTHORITY_UNAVAILABLE_REASON } from "../process/WindowsAuthority.js"; -import * as windowsAuthority from "../process/WindowsAuthority.js"; +import * as nativeLoader from "../windows/WindowsNativeLoader.js"; import { inspectGhidraInstallation, type GhidraInstallationHost, @@ -15,10 +15,13 @@ import { projectGhidraDoctorInspection } from "./GhidraDoctor.js"; const INSTALL = "/opt/ghidra_12.1.4_PUBLIC"; const installationHost = ( overrides: Partial = {}, + platform: NodeJS.Platform = "linux", ): GhidraInstallationHost => ({ readText: () => "application.version=12.1.4\napplication.java.min=21\n", executable: () => true, probeJava: () => ({ + osArch: "amd64", + osName: platform === "win32" ? "Windows 11" : "Linux", version: "21.0.11", major: 21, home: "/usr/lib/jvm/jdk-21", @@ -45,7 +48,7 @@ const inspection = ( ? {} : { installDir: options.installDir }), }, - installationHost(overrides), + installationHost(overrides, options.platform ?? "linux"), ); const doctorHost = ( @@ -66,7 +69,6 @@ const doctorHost = ( validTarget: () => Promise.resolve(true), executable: (path) => Promise.resolve(hopperAvailable && path.includes("Hopper")), - supportedLinuxHopper: () => Promise.resolve(true), linuxDemoRuntimeCheck: () => Promise.resolve({ name: "hopper-demo-runtime", @@ -129,6 +131,8 @@ describe("Ghidra doctor integration", () => { { installDir: INSTALL }, { probeJava: () => ({ + osArch: "amd64", + osName: "Linux", version: "17.0.12", major: 17, home: "/usr/lib/jvm/jdk-17", @@ -175,13 +179,11 @@ describe("Ghidra doctor integration", () => { describe("Windows Ghidra doctor authority", () => { beforeEach(() => { - vi.spyOn(windowsAuthority, "hasWindowsNativeAuthority").mockReturnValue( - false, - ); - vi.spyOn( - windowsAuthority, - "windowsNativeAuthorityUnavailableReason", - ).mockReturnValue(WINDOWS_NATIVE_AUTHORITY_UNAVAILABLE_REASON); + vi.spyOn(nativeLoader, "systemWindowsNativeAuthority").mockReturnValue({ + available: false, + failureStage: "artifact", + reason: WINDOWS_NATIVE_AUTHORITY_UNAVAILABLE_REASON, + }); }); afterEach(() => vi.restoreAllMocks()); it("distinguishes a valid Windows installation from unavailable native authority", async () => { @@ -223,9 +225,9 @@ describe("Windows Ghidra doctor authority", () => { result.checks.find(({ name }) => name === "ghidra-native_authority"), ).toMatchObject({ ok: false, - classification: "unsupported_host", + classification: "missing_dependency", detail: WINDOWS_NATIVE_AUTHORITY_UNAVAILABLE_REASON, - remediation: expect.stringContaining("cannot enable"), + remediation: expect.stringContaining("matching Windows x64 REA package"), }); }); @@ -247,7 +249,109 @@ describe("Windows Ghidra doctor authority", () => { expect(result.scope_checks).toContainEqual( expect.objectContaining({ name: "ghidra-native_authority", - classification: "unsupported_host", + classification: "missing_dependency", + detail: WINDOWS_NATIVE_AUTHORITY_UNAVAILABLE_REASON, + }), + ); + }); +}); + +describe("Windows native doctor failure diagnostics", () => { + beforeEach(() => { + vi.spyOn(nativeLoader, "systemWindowsNativeAuthority").mockReturnValue({ + available: false, + failureStage: "artifact", + reason: WINDOWS_NATIVE_AUTHORITY_UNAVAILABLE_REASON, + }); + }); + afterEach(() => vi.restoreAllMocks()); + + it.each([ + ["host", "unsupported_host", "unsupported_host", "Windows x64 host"], + [ + "artifact", + "runtime_missing", + "missing_dependency", + "native/windows/README.md", + ], + [ + "self-test", + "runtime_unavailable", + "runtime_failure", + "preserved native self-test error", + ], + [ + "inspection", + "runtime_unavailable", + "runtime_failure", + "every required security control", + ], + ] as const)( + "projects %s failure without blaming Ghidra", + async (failureStage, code, classification, remediation) => { + const reason = + "Windows native failure: remove_owned_directory: directory is not empty (145)"; + const load = vi.mocked(nativeLoader.systemWindowsNativeAuthority); + load.mockReturnValue({ available: false, failureStage, reason }); + const result = await runDoctor( + undefined, + { + ...doctorHost(inspection({ installDir: INSTALL, platform: "win32" })), + platform: "win32", + linuxDistribution: () => Promise.resolve(undefined), + }, + { providers: ["ghidra"] }, + ); + expect(load).toHaveBeenCalledTimes(1); + expect(result.healthy).toBe(false); + expect(result.providerInspections?.[0]).toMatchObject({ + available: false, + registrationEnvironment: { GHIDRA_INSTALL_DIR: INSTALL }, + checks: expect.arrayContaining([ + expect.objectContaining({ + name: "native_authority", + ok: false, + code, + detail: reason, + }), + ]), + }); + expect(result.scope_checks).toContainEqual( + expect.objectContaining({ + name: "ghidra-native_authority", + classification, + detail: reason, + remediation: expect.stringContaining(remediation), + }), + ); + }, + ); + + it("keeps one failure observation even when the next query would differ", () => { + const load = vi.mocked(nativeLoader.systemWindowsNativeAuthority); + load.mockReturnValueOnce({ + available: false, + failureStage: "self-test", + retryable: true, + reason: "cleanup failed (145)", + }); + const ghidra = inspection({ installDir: INSTALL, platform: "win32" }); + const failed = projectGhidraDoctorInspection(ghidra); + expect(load).toHaveBeenCalledTimes(1); + expect(failed.checks).toContainEqual( + expect.objectContaining({ + name: "native_authority", + ok: false, + code: "runtime_unavailable", + detail: "cleanup failed (145)", + }), + ); + const later = projectGhidraDoctorInspection(ghidra); + expect(load).toHaveBeenCalledTimes(2); + expect(later.checks).toContainEqual( + expect.objectContaining({ + name: "native_authority", + code: "runtime_missing", detail: WINDOWS_NATIVE_AUTHORITY_UNAVAILABLE_REASON, }), ); diff --git a/src/ghidra/GhidraDoctor.ts b/src/ghidra/GhidraDoctor.ts index 5e74a91995..1590ea14b8 100644 --- a/src/ghidra/GhidraDoctor.ts +++ b/src/ghidra/GhidraDoctor.ts @@ -2,10 +2,7 @@ import type { DoctorProviderCheck, DoctorProviderInspection, } from "../application/Doctor.js"; -import { - hasWindowsNativeAuthority, - windowsNativeAuthorityUnavailableReason, -} from "../process/WindowsAuthority.js"; +import { inspectWindowsNativeReadiness } from "../process/WindowsAuthority.js"; import { inspectGhidraInstallation, type GhidraInstallationCheck, @@ -16,10 +13,11 @@ import { export const projectGhidraDoctorInspection = ( inspection: GhidraInstallationInspection, ): DoctorProviderInspection => { - const windowsAuthorityUnavailable = - inspection.status === "available" && - inspection.platform === "win32" && - !hasWindowsNativeAuthority(inspection.platform); + const readiness = + inspection.status === "available" && inspection.platform === "win32" + ? inspectWindowsNativeReadiness(inspection.platform) + : undefined; + const windowsAuthorityUnavailable = readiness?.available === false; return { id: "ghidra", configured: inspection.installDir !== null, @@ -37,19 +35,22 @@ export const projectGhidraDoctorInspection = ( ...selectedChecks(inspection).map((check) => projectCheck(check, inspection.installDir !== null), ), - ...(inspection.status === "available" && inspection.platform === "win32" + ...(readiness !== undefined ? [ { name: "native_authority", ok: !windowsAuthorityUnavailable, - code: windowsAuthorityUnavailable ? "unsupported_host" : null, - detail: windowsAuthorityUnavailable - ? windowsNativeAuthorityUnavailableReason(inspection.platform) + code: !readiness.available ? readiness.code : null, + detail: !readiness.available + ? readiness.reason : "Verified Windows native Job Object, private DACL, and reparse-safe handle controls; local NTFS only.", - remediation: windowsAuthorityUnavailable - ? "Use the matching Windows x64 REA package with its bundled native addon and a writable local NTFS temporary directory. The native failure above identifies the failed constraint; Ghidra installation or registration changes cannot enable a missing native backend." - : null, - classification: "unsupported_host" as const, + remediation: !readiness.available ? readiness.remediation : null, + classification: + !readiness.available && readiness.code === "runtime_unavailable" + ? ("runtime_failure" as const) + : !readiness.available && readiness.code === "runtime_missing" + ? ("missing_dependency" as const) + : ("unsupported_host" as const), }, ] : []), diff --git a/src/ghidra/GhidraFunctionValues.test.ts b/src/ghidra/GhidraFunctionValues.test.ts index e7e5d8e9df..71c8ddb504 100644 --- a/src/ghidra/GhidraFunctionValues.test.ts +++ b/src/ghidra/GhidraFunctionValues.test.ts @@ -27,6 +27,7 @@ const observationOperations: ReadonlySet = new Set([ "procedure_callers", "procedure_info", "procedure_pseudo_code", + "procedure_references", "read_function_instructions", ]); @@ -43,7 +44,152 @@ const parseFunctionValueFixture = ( return parsed.ok ? { ok: true as const, value: parsed.value.value } : parsed; }; +describe("Ghidra unresolved call locations", () => { + it("rejects a noncanonical unresolved call site rather than retaining unverifiable location evidence", () => { + expect( + parseGhidraFunctionResult("analyze_function", { + value: { + ...functionDossierSchema.parse(ghidraFunctionDossier()), + unresolved_calls: [ + { address: "not-an-address", reason: "No call destination." }, + ], + }, + limitations: [], + relationship_coverage: { + status: "partial", + limitations: ["An unresolved call remains."], + }, + }), + ).toMatchObject({ ok: false, error: { _tag: "AnalysisOutputError" } }); + }); +}); + describe("Ghidra function-analysis result values", () => { + it("preserves unresolved Swift jump omissions", () => { + const dossier = { + ...functionDossierSchema.parse(ghidraFunctionDossier()), + swift_string_literals: [], + swift_string_literal_omissions: [ + { + source_address: "0x401004", + representation: "small", + address: null, + byte_length: 5, + reason: "unresolved_jump", + registers: ["x0", "x1"], + words: ["0x5000000000000005", "0x0000000000000000"], + }, + ], + }; + expect( + parseGhidraFunctionResult("analyze_function", { + value: dossier, + limitations: [], + }), + ).toMatchObject({ + ok: true, + value: { + value: { + swift_string_literal_omissions: + dossier.swift_string_literal_omissions, + }, + }, + }); + }); + + it("retains unavailable pseudocode instead of treating its empty representation as observed source", () => { + const availability = { + pseudocode: { + available: false, + reason: "Ghidra does not decompile external functions.", + }, + comments: { available: true }, + }; + const parsed = parseGhidraFunctionResult("analyze_function", { + value: { + ...functionDossierSchema.parse(ghidraFunctionDossier()), + pseudocode: "", + facet_availability: availability, + }, + limitations: [], + }); + expect(parsed).toMatchObject({ + ok: true, + value: { value: { pseudocode: "", facet_availability: availability } }, + }); + }); + it.each([ + { status: "complete", diagnostics: ["Resolved static references only."] }, + { + status: "partial", + limitations: ["Only the first instruction range was examined."], + }, + { status: "unknown", reason: "Relationship inventory was unavailable." }, + { + status: "partial", + limitations: ["Missing callers."], + limited_directions: ["callers"], + }, + { + status: "unknown", + reason: "Missing callees.", + limited_directions: ["callees"], + }, + ])("preserves observed $status relationship coverage", (coverage) => { + const parsed = parseGhidraFunctionResult("analyze_function", { + value: ghidraFunctionDossier(), + limitations: [], + relationship_coverage: coverage, + }); + expect(parsed).toMatchObject({ + ok: true, + value: { relationshipCoverage: coverage }, + }); + }); + + it("preserves omitted relationship coverage for legacy observations", () => { + const parsed = parseGhidraFunctionResult("analyze_function", { + value: ghidraFunctionDossier(), + limitations: [], + }); + if (!parsed.ok) throw parsed.error; + expect(parsed.value).not.toHaveProperty("relationshipCoverage"); + }); +}); + +describe("Ghidra relationship metadata validation", () => { + it.each([ + { status: "partial", limitations: [] }, + { status: "unknown", reason: "" }, + { status: "complete", limitations: ["Truncated."] }, + { status: "partial", limitations: ["Truncated."], limited_directions: [] }, + { + status: "partial", + limitations: ["Truncated."], + limited_directions: ["callers", "callers"], + }, + { status: "unknown", reason: "Missing.", limited_directions: ["incoming"] }, + { status: "complete", limited_directions: ["callers"] }, + ])("rejects malformed relationship coverage %j", (coverage) => { + expect( + parseGhidraFunctionResult("analyze_function", { + value: ghidraFunctionDossier(), + limitations: [], + relationship_coverage: coverage, + }), + ).toMatchObject({ ok: false, error: { _tag: "AnalysisOutputError" } }); + }); + + it("rejects relationship metadata on unrelated function operations", () => { + expect( + parseGhidraFunctionResult("procedure_assembly", { + value: "ret", + limitations: [], + relationship_coverage: { status: "complete" }, + }), + ).toMatchObject({ ok: false, error: { _tag: "AnalysisOutputError" } }); + }); + it("preserves file-offset resolution in Evidence locations and rejects inexact offsets", () => { const selection = { file_offset: 0x210, @@ -95,6 +241,43 @@ describe("Ghidra function-analysis result values", () => { }); }); + it("preserves handler-region limitations on procedure references", () => { + const limitation = + "Ghidra Listing shows a Borland/Delphi-style SEH frame registration at 0x401008 (push 0x4010a0 followed by fs-segment stack chaining); handler 0x4010a0 lies outside this procedure's body, so callees, strings, and cross-references may omit the handler region."; + const parsed = parseGhidraFunctionResult("procedure_references", { + value: { + procedure: ghidraFunctionIdentity(), + direction: "outgoing", + references: [ghidraReferenceEdge()], + reference_kinds_available: true, + unresolved_calls: [], + }, + limitations: [limitation], + }); + expect(parsed).toEqual({ + ok: true, + value: { + value: { + procedure: ghidraFunctionIdentity(), + direction: "outgoing", + references: [ghidraReferenceEdge()], + reference_kinds_available: true, + unresolved_calls: [], + }, + limitations: [limitation], + }, + }); + expect( + parseGhidraFunctionResult("procedure_references", { + procedure: ghidraFunctionIdentity(), + direction: "outgoing", + references: [], + reference_kinds_available: true, + unresolved_calls: [], + }), + ).toMatchObject({ ok: false, error: { _tag: "AnalysisOutputError" } }); + }); + it.each(["procedure_callees", "procedure_callers"] as const)( "requires the function-observation envelope to carry terminal-call limitations for %s", (operation) => { diff --git a/src/ghidra/GhidraFunctionValues.ts b/src/ghidra/GhidraFunctionValues.ts index a6e62c93cf..c8890aae8b 100644 --- a/src/ghidra/GhidraFunctionValues.ts +++ b/src/ghidra/GhidraFunctionValues.ts @@ -27,6 +27,10 @@ import { import { nativeApiBoundarySchema } from "../domain/native/nativeApiBoundary.js"; import { nativeValueFlowSchema } from "../domain/native/nativeValueFlow.js"; import type { EvidenceLocation } from "../domain/evidence.js"; +import { + relationshipCoverageSchema, + type RelationshipCoverage, +} from "../domain/relationshipCoverage.js"; import { err, ok, type Result } from "../domain/result.js"; import { analysisInputErrorFromIssues } from "../domain/inputIssueProjection.js"; import { @@ -307,6 +311,7 @@ const ghidraFunctionDossier = functionDossierSchema ...value.comments.map(({ address }) => address), ...value.callers.map(({ address }) => address), ...value.callees.map(({ address }) => address), + ...(value.unresolved_calls ?? []).map(({ address }) => address), ...value.incoming_references.flatMap(referenceAddresses), ...value.outgoing_references.flatMap(referenceAddresses), ...value.referenced_strings.flatMap(({ address, source_address }) => [ @@ -413,12 +418,14 @@ const functionBoundaryObservationOperations: ReadonlySet["selection"]; @@ -450,13 +459,17 @@ export const parseGhidraFunctionResult = ( const envelope = functionObservationEnvelope.safeParse(value); if ( !envelope.success || + (operation !== "analyze_function" && + operation !== "procedure_callers" && + operation !== "procedure_callees" && + envelope.data.relationship_coverage !== undefined) || (operation !== "procedure_pseudo_code" && envelope.data.selection !== undefined) ) return err( new AnalysisOutputError( operation, - "Ghidra bridge omitted the function observation envelope", + "Ghidra bridge returned an invalid function observation envelope for this operation", envelope.success ? undefined : { cause: envelope.error }, ), ); @@ -469,6 +482,9 @@ export const parseGhidraFunctionResult = ( ? ok({ value: jsonValueSchema.parse(parsed.data), limitations: observation.limitations, + ...(observation.relationship_coverage === undefined + ? {} + : { relationshipCoverage: observation.relationship_coverage }), ...(observation.selection === undefined ? {} : { diff --git a/src/ghidra/GhidraInstallation.fixture.ts b/src/ghidra/GhidraInstallation.fixture.ts new file mode 100644 index 0000000000..1becec6c8b --- /dev/null +++ b/src/ghidra/GhidraInstallation.fixture.ts @@ -0,0 +1,18 @@ +import type { GhidraInstallationHost } from "./GhidraInstallation.js"; + +/** Simulated valid installation for profile tests; never starts Ghidra or Java. */ +export const ghidraProfileInstallationHost: GhidraInstallationHost = { + platform: "linux", + architecture: "x64", + readText: () => "application.version=12.1.4\napplication.java.min=21\n", + executable: () => true, + probeJava: () => ({ + version: "21.0.11", + major: 21, + home: "/jdk-21", + bits: 64, + runtime: "jdk", + osArch: "amd64", + osName: "Linux", + }), +}; diff --git a/src/ghidra/GhidraInstallation.test.ts b/src/ghidra/GhidraInstallation.test.ts index 776c10308c..a60e7b4b10 100644 --- a/src/ghidra/GhidraInstallation.test.ts +++ b/src/ghidra/GhidraInstallation.test.ts @@ -1,9 +1,14 @@ +import { chmod, mkdir, symlink, writeFile } from "node:fs/promises"; +import { join } from "node:path"; import { afterEach, describe, expect, it, vi } from "vitest"; +import { spawnSync } from "node:child_process"; import { elf, fat, thinMach } from "../domain/binaryTarget.fixture.js"; import { projectGhidraDoctorInspection } from "./GhidraDoctor.js"; import { + ghidraInstallationDiagnostics, ghidraJavaEnvironment, + discoverGhidraInstallationOnPath, inspectGhidraInstallation, type GhidraExecutableHeaderProbe, type GhidraInstallationHost, @@ -15,6 +20,15 @@ import { ghidraApplicationProperties, ghidraJavaRequirement, } from "./GhidraInstallationPolicy.js"; +import { createTestTempDirectory } from "../../tests/fixtures/temporaryDirectory.js"; + +// The real Java probe shells out via spawnSync; stub it so one test can run +// the real probeJava against captured `-XshowSettings:properties` output. +// Every other test overrides probeJava, so the stub never fires for them. +vi.mock("node:child_process", async (importOriginal) => { + const original = await importOriginal(); + return { ...original, spawnSync: vi.fn() }; +}); const INSTALL = "/opt/ghidra"; const PROPERTIES = `${INSTALL}/Ghidra/application.properties`; @@ -32,6 +46,8 @@ const headerProbe = (bytes: Buffer): GhidraExecutableHeaderProbe => ({ }); const JAVA: GhidraJavaObservation = { + osArch: "amd64", + osName: "Linux", version: "21.0.11", major: SUPPORTED_GHIDRA_JAVA_MAJOR, home: "/usr/lib/jvm/jdk-21", @@ -56,6 +72,100 @@ afterEach(() => { vi.unstubAllEnvs(); }); +it.skipIf(process.platform === "win32")( + "resolves a PATH launcher only when its real target is a complete installation", + async () => { + const root = await createTestTempDirectory("rea-ghidra-path-"); + const installation = join(root, "installation"); + const pathDirectory = join(root, "bin"); + const launcher = join(installation, "ghidraRun"); + await mkdir(join(installation, "Ghidra"), { recursive: true }); + await mkdir(join(installation, "support"), { recursive: true }); + await mkdir(pathDirectory, { recursive: true }); + await writeFile( + join(installation, "Ghidra", "application.properties"), + "application.version=12.1.4\n", + ); + await writeFile(launcher, "#!/bin/sh\nexit 0\n"); + await writeFile( + join(installation, "support", "analyzeHeadless"), + "#!/bin/sh\nexit 0\n", + ); + await chmod(launcher, 0o700); + await chmod(join(installation, "support", "analyzeHeadless"), 0o700); + await symlink(launcher, join(pathDirectory, "ghidra")); + + expect( + discoverGhidraInstallationOnPath( + { PATH: pathDirectory }, + process.platform, + ), + ).toBe(installation); + + const incomplete = join(root, "incomplete"); + const incompletePath = join(root, "incomplete-bin"); + await mkdir(join(incomplete, "Ghidra"), { recursive: true }); + await mkdir(incompletePath, { recursive: true }); + await writeFile( + join(incomplete, "Ghidra", "application.properties"), + "application.version=12.1.4\n", + ); + await writeFile(join(incomplete, "ghidraRun"), "launcher\n"); + await chmod(join(incomplete, "ghidraRun"), 0o700); + await symlink( + join(incomplete, "ghidraRun"), + join(incompletePath, "ghidra"), + ); + expect( + discoverGhidraInstallationOnPath( + { PATH: incompletePath }, + process.platform, + ), + ).toBeUndefined(); + + const nonExecutable = join(root, "non-executable"); + const nonExecutablePath = join(root, "non-executable-bin"); + await mkdir(join(nonExecutable, "Ghidra"), { recursive: true }); + await mkdir(join(nonExecutable, "support"), { recursive: true }); + await mkdir(nonExecutablePath, { recursive: true }); + await writeFile( + join(nonExecutable, "Ghidra", "application.properties"), + "application.version=12.1.4\n", + ); + await writeFile( + join(nonExecutable, "support", "analyzeHeadless"), + "headless\n", + ); + await chmod(join(nonExecutable, "support", "analyzeHeadless"), 0o700); + await writeFile(join(nonExecutablePath, "ghidra"), "not executable\n"); + expect( + discoverGhidraInstallationOnPath( + { PATH: nonExecutablePath }, + process.platform, + ), + ).toBeUndefined(); + }, +); + +it("does not fall back to PATH when an explicit Ghidra installation path is malformed", () => { + const inspection = inspectGhidraInstallation( + { + environment: { PATH: "/valid-ghidra-bin" }, + installDir: "relative/ghidra", + platform: "linux", + architecture: "x64", + }, + host({ discoverInstallDirOnPath: () => INSTALL }), + ); + expect(inspection.status).toBe("unavailable"); + expect(inspection.installDir).toBe("relative/ghidra"); + expect(inspection.checks[0]).toMatchObject({ + name: "configuration", + status: "failed", + detail: "GHIDRA_INSTALL_DIR must be absolute", + }); +}); + it("probes the selected Java environment without reading conflicting ambient settings", () => { vi.stubEnv("REA_GHIDRA_CONTEXT", "ambient"); vi.stubEnv("REA_GHIDRA_AMBIENT", "ambient-only"); @@ -174,6 +284,11 @@ describe("Ghidra installation inspection", () => { { environment: {}, installDir: INSTALL, platform, architecture }, host({ executable: (path) => path === HEADLESS || path === decompiler, + probeJava: () => ({ + ...JAVA, + osArch: architecture === "arm64" ? "aarch64" : "amd64", + osName: platform === "darwin" ? "Mac OS X" : "Linux", + }), }), ), ).toMatchObject({ @@ -202,6 +317,7 @@ describe("Ghidra installation inspection", () => { return { ...JAVA, home: "C:\\Java\\jdk-21", + osName: "Windows 11", }; }, }; @@ -721,7 +837,10 @@ describe("Ghidra FAT native decompiler admission", () => { platform: "darwin", architecture: "arm64", }, - host({ executableHeader: () => headerProbe(bytes) }), + host({ + probeJava: () => ({ ...JAVA, osArch: "aarch64", osName: "Mac OS X" }), + executableHeader: () => headerProbe(bytes), + }), ); expect(result.status).toBe(status); expect( @@ -739,7 +858,10 @@ describe("Ghidra FAT native decompiler admission", () => { platform: "darwin", architecture: "arm64", }, - host({ executableHeader: () => headerProbe(bytes) }), + host({ + probeJava: () => ({ ...JAVA, osArch: "aarch64", osName: "Mac OS X" }), + executableHeader: () => headerProbe(bytes), + }), ); expect(result.status).toBe("available"); expect( @@ -788,7 +910,14 @@ describe("Ghidra native decompiler admission", () => { ])("distinguishes $name", ({ options, executableHeader }) => { const result = inspectGhidraInstallation( options, - host({ executableHeader }), + host({ + probeJava: () => ({ + ...JAVA, + osArch: options.architecture === "arm64" ? "aarch64" : "amd64", + osName: options.platform === "darwin" ? "Mac OS X" : "Linux", + }), + executableHeader, + }), ); expect(result.status).toBe("unavailable"); expect( @@ -849,6 +978,7 @@ describe("Ghidra native decompiler admission", () => { architecture: "arm64", }, host({ + probeJava: () => ({ ...JAVA, osArch: "aarch64", osName: "Mac OS X" }), executableHeader: (path: string) => path === MAC_DECOMPILER ? headerProbe(bytes) : undefined, }), @@ -872,6 +1002,7 @@ describe("Ghidra native decompiler admission", () => { architecture: "arm64", }, host({ + probeJava: () => ({ ...JAVA, osArch: "aarch64", osName: "Mac OS X" }), executableHeader: (path: string) => path === MAC_DECOMPILER ? headerProbe(elf(2, 1, 62)) : undefined, }), @@ -889,3 +1020,205 @@ describe("Ghidra native decompiler admission", () => { }); }); }); + +describe("Ghidra JVM native directory admission", () => { + it("rejects a JDK whose os.arch selects a different native decompiler directory", () => { + const result = inspectGhidraInstallation( + { + environment: {}, + installDir: INSTALL, + platform: "darwin", + architecture: "arm64", + }, + host({ + probeJava: () => ({ ...JAVA, osArch: "x86_64", osName: "Mac OS X" }), + }), + ); + expect(result.status).toBe("unavailable"); + if (result.status !== "unavailable") return; + expect( + result.checks.find(({ name }) => name === "native_decompiler"), + ).toMatchObject({ + status: "failed", + code: "executable_missing", + detail: expect.stringContaining( + "JVM reports os.arch=x86_64, so Ghidra selects its native decompiler from os/mac_x86_64", + ), + remediation: expect.stringContaining("JAVA_HOME"), + }); + }); + + it("accepts a JDK whose os.arch selects the validated native decompiler directory", () => { + const result = inspectGhidraInstallation( + { + environment: {}, + installDir: INSTALL, + platform: "darwin", + architecture: "arm64", + }, + host({ + probeJava: () => ({ ...JAVA, osArch: "aarch64", osName: "Mac OS X" }), + }), + ); + expect(result.status).toBe("available"); + }); + + it("rejects a successful Java probe that omits os.arch", () => { + const { osArch: _observedArchitecture, ...withoutArchitecture } = JAVA; + const result = inspectGhidraInstallation( + { + environment: {}, + installDir: INSTALL, + platform: "darwin", + architecture: "arm64", + }, + host({ probeJava: () => withoutArchitecture }), + ); + expect(result.status).toBe("unavailable"); + expect( + result.checks.find(({ name }) => name === "native_decompiler"), + ).toMatchObject({ + status: "failed", + detail: expect.stringContaining("did not report os.arch"), + }); + expect(projectGhidraDoctorInspection(result).available).toBe(false); + }); + + it.each(["riscv64", "sparcv9", ""])( + "refuses an unrecognized JVM os.arch (%s) instead of assuming alignment", + (osArch) => { + const result = inspectGhidraInstallation( + { + environment: {}, + installDir: INSTALL, + platform: "darwin", + architecture: "arm64", + }, + host({ + probeJava: () => ({ ...JAVA, osArch }), + }), + ); + expect(result.status).toBe("unavailable"); + if (result.status !== "unavailable") return; + expect( + result.checks.find(({ name }) => name === "native_decompiler"), + ).toMatchObject({ + status: "failed", + detail: expect.stringContaining("unrecognized os.arch="), + }); + }, + ); + + it("rejects a JDK whose os.name selects a different host OS", () => { + const result = inspectGhidraInstallation( + { + environment: {}, + installDir: INSTALL, + platform: "darwin", + architecture: "arm64", + }, + host({ + probeJava: () => ({ + ...JAVA, + osArch: "aarch64", + osName: "Windows 11", + }), + }), + ); + expect(result.status).toBe("unavailable"); + if (result.status !== "unavailable") return; + expect( + result.checks.find(({ name }) => name === "native_decompiler"), + ).toMatchObject({ + status: "failed", + code: "executable_missing", + detail: expect.stringContaining("JVM reports os.name=Windows 11"), + remediation: expect.stringContaining("JAVA_HOME"), + }); + }); +}); + +describe("Ghidra JVM OS admission", () => { + it("rejects a successful Java probe that omits os.name", () => { + const { osName: _observedOsName, ...withoutOsName } = JAVA; + const result = inspectGhidraInstallation( + { + environment: {}, + installDir: INSTALL, + platform: "linux", + architecture: "x64", + }, + host({ probeJava: () => withoutOsName }), + ); + expect(result.status).toBe("unavailable"); + expect( + result.checks.find(({ name }) => name === "native_decompiler"), + ).toMatchObject({ + status: "failed", + detail: expect.stringContaining("did not report os.name"), + }); + expect(projectGhidraDoctorInspection(result).available).toBe(false); + }); + + it.each(["SunOS", "FreeBSD", ""])( + "refuses an unrecognized JVM os.name (%s) instead of assuming alignment", + (osName) => { + const result = inspectGhidraInstallation( + { + environment: {}, + installDir: INSTALL, + platform: "linux", + architecture: "x64", + }, + host({ + probeJava: () => ({ ...JAVA, osName }), + }), + ); + expect(result.status).toBe("unavailable"); + if (result.status !== "unavailable") return; + expect( + result.checks.find(({ name }) => name === "native_decompiler"), + ).toMatchObject({ + status: "failed", + detail: expect.stringContaining("os.name="), + }); + }, + ); +}); + +describe("Ghidra Java probe os.arch extraction", () => { + it("extracts os.arch through the real Java probe into the installation diagnostics", () => { + vi.mocked(spawnSync).mockReturnValue({ + status: 0, + stdout: "", + stderr: [ + "Property settings:", + " file.encoding = UTF-8", + " java.home = /usr/lib/jvm/jdk-21", + " java.version = 21.0.11", + " java.vm.name = OpenJDK 64-Bit Server VM", + " os.arch = x86_64", + " os.name = Linux", + " sun.arch.data.model = 64", + "", + ].join("\n"), + } as never); + try { + // No host override: the real system host runs the real probeJava + // against the stubbed spawnSync. + const inspection = inspectGhidraInstallation({ + environment: {}, + installDir: INSTALL, + javaHome: "/usr/lib/jvm/jdk-21", + platform: "linux", + architecture: "x64", + }); + expect(ghidraInstallationDiagnostics(inspection)).toMatchObject({ + java_os_arch: "x86_64", + java_os_name: "Linux", + }); + } finally { + vi.mocked(spawnSync).mockReset(); + } + }); +}); diff --git a/src/ghidra/GhidraInstallation.ts b/src/ghidra/GhidraInstallation.ts index 9675d133fb..6a77ee1af4 100644 --- a/src/ghidra/GhidraInstallation.ts +++ b/src/ghidra/GhidraInstallation.ts @@ -8,9 +8,10 @@ import { openSync, readFileSync, readSync, + realpathSync, statSync, } from "node:fs"; -import { posix, win32 } from "node:path"; +import { delimiter as pathDelimiter, dirname, posix, win32 } from "node:path"; import type { ProviderRejectionCode } from "../contracts/providerSelection.js"; import { @@ -203,6 +204,17 @@ interface GhidraInstallationObservation { readonly javaCommand: string; /** Observed java.vm.name; omitted when the probe cannot establish VM identity. */ readonly javaVmName?: string; + /** + * Observed JVM os.arch; omitted when the probe cannot establish it. Ghidra + * selects its native decompiler directory from this value. + */ + readonly javaOsArch?: string; + /** + * Observed JVM os.name; omitted when the probe cannot establish it. Ghidra + * selects its native decompiler OS directory from the JVM OS/architecture + * pair, so a non-host value selects an unvalidated directory. + */ + readonly javaOsName?: string; readonly checks: readonly GhidraInstallationCheck[]; } @@ -241,9 +253,22 @@ export interface GhidraJavaObservation { readonly version: string; readonly major: number; readonly home: string; - readonly bits: number; + readonly bits: number | null; + /** Present when Java reported a bitness value that could not be parsed exactly. */ + readonly unparsedBits?: string; readonly runtime: "jdk" | "jre"; readonly vmName?: string; + /** + * Observed JVM os.arch property, which Ghidra uses to select its native + * decompiler directory. Omitted when the probe cannot establish it. + */ + readonly osArch?: string; + /** + * Observed JVM os.name property, which Ghidra uses with os.arch to select + * its native decompiler OS directory. Omitted when the probe cannot + * establish it. + */ + readonly osName?: string; } /** A bounded executable-header probe: a small prefix plus the whole file size. */ @@ -271,6 +296,11 @@ export interface GhidraInstallationHost { command: string, environment: NodeJS.ProcessEnv, ): GhidraJavaObservation | undefined; + /** Resolve a PATH launcher to a validated installation without executing it. */ + discoverInstallDirOnPath?( + environment: Readonly, + platform: NodeJS.Platform, + ): string | undefined; } interface GhidraInstallationCoordinates { @@ -324,6 +354,12 @@ export const inspectGhidraInstallation = ( ...(coordinates.java?.vmName === undefined ? {} : { javaVmName: coordinates.java.vmName }), + ...(coordinates.java?.osArch === undefined + ? {} + : { javaOsArch: coordinates.java.osArch }), + ...(coordinates.java?.osName === undefined + ? {} + : { javaOsName: coordinates.java.osName }), checks, }; if (failed !== undefined) @@ -375,7 +411,12 @@ const installationCoordinates = ( host: GhidraInstallationHost, ): GhidraInstallationCoordinates => { const path = platform === "win32" ? win32 : posix; - const installDir = options.installDir ?? null; + // An explicit setting remains authoritative even when malformed or stale; + // automatic discovery is only a fallback when the caller supplied none. + const installDir = + options.installDir ?? + host.discoverInstallDirOnPath?.(options.environment, platform) ?? + null; const applicationPropertiesPath = installDir === null ? null @@ -555,6 +596,8 @@ export const ghidraInstallationDiagnostics = ( java_home: inspection.javaHome, java_version: inspection.javaVersion, java_vm_name: inspection.javaVmName ?? null, + java_os_arch: inspection.javaOsArch ?? null, + java_os_name: inspection.javaOsName ?? null, platform: inspection.platform, architecture: inspection.architecture, checks: inspection.checks.map((check) => ({ @@ -632,6 +675,61 @@ const nativeDecompilerCheck = ({ architecture, host.executableHeader?.(path), ); + // Ghidra resolves its native decompiler directory from the JVM's os.arch, + // not the host process architecture. A JDK running under emulation (for + // example x86_64 under Rosetta on Apple silicon) selects a directory that + // was never validated, so every analysis fails after a passing doctor. + // Ghidra resolves the OS directory from the JVM os.name/os.arch pair, so a + // JVM reporting a non-host os.name (for example through -Dos.name=Windows) + // selects a different OS directory even when os.arch matches. + const osArch = coordinates.java?.osArch; + const osName = coordinates.java?.osName; + const selectedDirectory = jvmOsArchNativeDirectory(platform, osArch); + const validatedDirectory = NATIVE_PLATFORMS[platform]?.[architecture]; + // Without a JVM observation the java check reports the missing runtime; + // the decompiler directory cannot be selected, so this check defers to it. + if (coordinates.java !== undefined) { + if (selectedDirectory === undefined) + return installationCheck({ + name: "native_decompiler", + passed: false, + code: "executable_missing", + detail: + osArch === undefined + ? "The Java probe did not report os.arch; the JVM-selected native decompiler directory is unknown and cannot be matched to the validated host directory" + : `JVM reports unrecognized os.arch=${osArch}; its selected native decompiler directory is unknown and cannot be matched to the validated host directory`, + remediation: `Point JAVA_HOME at a JDK whose reported architecture maps to the validated Ghidra decompiler (${validatedDirectory ?? "unsupported host"}). REA does not build or install native tools.`, + }); + if ( + validatedDirectory !== undefined && + selectedDirectory !== validatedDirectory && + path !== null + ) + return installationCheck({ + name: "native_decompiler", + passed: false, + code: "executable_missing", + detail: `JVM reports os.arch=${osArch}, so Ghidra selects its native decompiler from os/${selectedDirectory}, but the validated decompiler targets ${validatedDirectory} (host ${platform}/${architecture})`, + remediation: `Point JAVA_HOME at a JDK whose architecture matches the built decompiler (${validatedDirectory}). REA does not build or install native tools.`, + }); + if (osName === undefined) + return installationCheck({ + name: "native_decompiler", + passed: false, + code: "executable_missing", + detail: + "The Java probe did not report os.name; the JVM-selected native decompiler OS directory is unknown and cannot be matched to the validated host platform", + remediation: `Point JAVA_HOME at a JDK whose reported operating system matches the host platform (${platform}). REA does not build or install native tools.`, + }); + if (!jvmOsNameMatchesPlatform(platform, osName) && path !== null) + return installationCheck({ + name: "native_decompiler", + passed: false, + code: "executable_missing", + detail: `JVM reports os.name=${osName}, which does not match the validated host platform ${platform} (host ${platform}/${architecture})`, + remediation: `Point JAVA_HOME at a JDK whose reported operating system matches the host platform (${platform}). REA does not build or install native tools.`, + }); + } return installationCheck({ name: "native_decompiler", passed: path !== null && compatibility?.status !== "incompatible", @@ -646,6 +744,43 @@ const nativeDecompilerCheck = ({ }); }; +/** + * Whether a JVM os.name value selects the host platform's native OS directory. + * Unknown values never align with the validated host directory; admission + * must not treat them as matching. + */ +const jvmOsNameMatchesPlatform = ( + platform: NodeJS.Platform, + osName: string, +): boolean => { + const normalized = osName.toLowerCase(); + if (platform === "win32") return normalized.startsWith("windows"); + if (platform === "darwin") + return normalized.startsWith("mac") || normalized.startsWith("darwin"); + if (platform === "linux") return normalized.startsWith("linux"); + return false; +}; + +/** + * Map a JVM os.arch value to the Ghidra native directory that JVM selects. + * Unknown values stay unknown; admission must not treat them as aligned with + * the validated host directory. + */ +const jvmOsArchNativeDirectory = ( + platform: NodeJS.Platform, + osArch: string | undefined, +): string | undefined => { + if (osArch === undefined) return undefined; + const normalized = osArch.toLowerCase(); + const node: NodeJS.Architecture | undefined = + normalized === "x86_64" || normalized === "amd64" || normalized === "x64" + ? "x64" + : normalized === "aarch64" || normalized === "arm64" + ? "arm64" + : undefined; + return node === undefined ? undefined : NATIVE_PLATFORMS[platform]?.[node]; +}; + const versionCheck = ( properties: GhidraApplicationProperties | undefined, ): GhidraInstallationCheck => { @@ -681,12 +816,23 @@ const javaCheck = ( ghidraJavaMajorAccepted(observation.major, requirement) && observation.bits === 64 && observation.runtime === "jdk"; + const bitness = + observation === undefined + ? undefined + : observation.bits === null + ? `bitness unknown${observation.unparsedBits === undefined ? "" : ` (sun.arch.data.model = ${JSON.stringify(observation.unparsedBits)})`}` + : `${String(observation.bits)}-bit`; + const knownMismatch = + observation !== undefined && + (!ghidraJavaMajorAccepted(observation.major, requirement) || + (observation.bits !== null && observation.bits !== 64) || + observation.runtime !== "jdk"); const detail = observation === undefined ? requirement.unresolved === null ? command : `${command}; ${requirement.unresolved}` - : `${observation.version}; ${String(observation.bits)}-bit; ${observation.runtime === "jdk" ? "JDK" : "runtime only"}; ${observation.home}; accepted ${range}`; + : `${observation.version}; ${bitness}; ${observation.runtime === "jdk" ? "JDK" : "runtime only"}; ${observation.home}; accepted ${range}`; return installationCheck({ name: "java", passed: supported, @@ -695,7 +841,9 @@ const javaCheck = ( ? "version_unresolved" : observation === undefined ? "runtime_missing" - : "unsupported_version", + : knownMismatch + ? "unsupported_version" + : "version_unresolved", detail, remediation: javaRemediation(requirement, configuredHome, range), }); @@ -781,17 +929,35 @@ const systemGhidraInstallationHost = (): GhidraInstallationHost => ({ const version = javaProperty(output, "java.version"); const home = javaProperty(output, "java.home"); const vmName = javaProperty(output, "java.vm.name"); - const bits = Number.parseInt( - javaProperty(output, "sun.arch.data.model") ?? "0", - 10, - ); - if (version === null || home === null) return undefined; + const osArch = javaProperty(output, "os.arch"); + const osName = javaProperty(output, "os.name"); + const reportedBits = javaProperty(output, "sun.arch.data.model"); + const parsedBits = + reportedBits !== null && /^\d+$/.test(reportedBits) + ? Number(reportedBits) + : null; + const bits = + parsedBits !== null && Number.isSafeInteger(parsedBits) + ? parsedBits + : null; + if ( + version === null || + version.length === 0 || + home === null || + home.length === 0 + ) + return undefined; return { version, major: javaMajor(version), home, ...(vmName === null ? {} : { vmName }), + ...(osArch === null ? {} : { osArch }), + ...(osName === null ? {} : { osName }), bits, + ...(reportedBits === null || bits !== null + ? {} + : { unparsedBits: reportedBits }), runtime: executablePath( (process.platform === "win32" ? win32 : posix).join( home, @@ -803,12 +969,56 @@ const systemGhidraInstallationHost = (): GhidraInstallationHost => ({ : "jre", }; }, + discoverInstallDirOnPath: discoverGhidraInstallationOnPath, }); +/** Find a PATH launcher whose real target is inside a complete Ghidra tree. */ +export const discoverGhidraInstallationOnPath = ( + environment: Readonly, + platform: NodeJS.Platform, +): string | undefined => { + if (platform !== process.platform) return undefined; + const path = platform === "win32" ? win32 : posix; + const pathValue = environment.PATH; + if (pathValue === undefined || pathValue.length === 0) return undefined; + const launchers = + platform === "win32" + ? ["ghidra.bat", "ghidraRun.bat", "ghidra.exe", "ghidraRun.exe"] + : ["ghidra", "ghidraRun"]; + for (const directory of pathValue.split(pathDelimiter)) { + if (directory.length === 0) continue; + for (const launcher of launchers) { + let resolved: string; + try { + const candidate = path.resolve(directory, launcher); + if (!executablePath(candidate)) continue; + resolved = realpathSync(candidate); + } catch (cause: unknown) { + void cause; + continue; + } + const root = dirname(resolved); + const properties = path.join(root, "Ghidra", "application.properties"); + const headless = path.join( + root, + "support", + platform === "win32" ? "analyzeHeadless.bat" : "analyzeHeadless", + ); + try { + if (statSync(properties).isFile() && executablePath(headless)) + return root; + } catch (cause: unknown) { + void cause; + } + } + } + return undefined; +}; + const executablePath = (path: string): boolean => { try { accessSync(path, constants.X_OK); - return true; + return statSync(path).isFile(); } catch (cause: unknown) { // best-effort cleanup: optional executable probing; failure means missing. void cause; @@ -818,7 +1028,7 @@ const executablePath = (path: string): boolean => { const javaProperty = (output: string, name: string): string | null => new RegExp( - `^\\s*${name.replaceAll(".", "\\.")}\\s*=\\s*(.+?)\\s*$`, + `^[ \\t]*${name.replaceAll(".", "\\.")}[ \\t]*=[ \\t]*([^\\r\\n]*?)[ \\t]*\\r?$`, "mu", ).exec(output)?.[1] ?? null; diff --git a/src/ghidra/GhidraJavaLaunch.ts b/src/ghidra/GhidraJavaLaunch.ts index 1e34eeb082..598a152c2a 100644 --- a/src/ghidra/GhidraJavaLaunch.ts +++ b/src/ghidra/GhidraJavaLaunch.ts @@ -1,12 +1,77 @@ import { snapshotEnvironment } from "../process/snapshotEnvironment.js"; import { dirname, join } from "node:path"; -import { execFileOutput } from "../process/ExecFileOutput.js"; +import { + execFileOutput, + execFileOutputFailure, + type ExecFileOutputFailure, +} from "../process/ExecFileOutput.js"; + +export const GHIDRA_LAUNCH_SUPPORT_CAPTURE_BYTES = 64 * 1024; +type LaunchSupportPhase = "-envvars" | "-vmargs"; +type LaunchSupportOutcome = Omit< + ExecFileOutputFailure, + "killed" | "outputTruncated" +> & { + readonly killed: boolean | null; + readonly outputTruncated: boolean | null; +}; + +export class GhidraLaunchSupportError extends Error { + readonly outcome: LaunchSupportOutcome | undefined; + + constructor(phase: LaunchSupportPhase, cause: unknown) { + super(`Ghidra LaunchSupport ${phase} failed`, { cause }); + this.name = "GhidraLaunchSupportError"; + const outcome = execFileOutputFailure(cause); + this.outcome = + outcome === undefined + ? undefined + : { + ...outcome, + killed: outcome.code === "ABORT_ERR" ? null : outcome.killed, + outputTruncated: + outcome.code === "ABORT_ERR" ? null : outcome.outputTruncated, + }; + } +} + +/** JVM option environments remove quotes, including quotes inside an option. */ +const jvmEnvironmentArguments = (value: string): readonly string[] => { + const arguments_: string[] = []; + let argument = ""; + let quote: "'" | '"' | undefined; + for (const character of value) { + if (quote !== undefined) { + if (character === quote) quote = undefined; + else argument += character; + } else if (character === "'" || character === '"') quote = character; + else if (/\s/u.test(character)) { + if (argument !== "") arguments_.push(argument); + argument = ""; + } else argument += character; + } + if (quote !== undefined) + throw new Error( + "Invalid quoted Ghidra LaunchSupport JVM environment option", + ); + if (argument !== "") arguments_.push(argument); + return arguments_; +}; + +const rejectJvmPlatformOverrides = (arguments_: readonly string[]): void => { + if ( + arguments_.some((argument) => /^-Dos\.(?:arch|name)(?:=|$)/u.test(argument)) + ) + throw new Error( + "Ghidra LaunchSupport may not override os.arch or os.name; the native decompiler was validated for the inspected JVM platform.", + ); +}; /** Launch coordinates obtained from Ghidra's own release configuration helper. */ export const ghidraJavaLaunch = async (options: { readonly platform: NodeJS.Platform; readonly analyzeHeadlessPath: string; - readonly javaHome: string; + readonly javaHome?: string; readonly javaVmName?: string; readonly homeRoot: string; readonly tempRoot: string; @@ -21,7 +86,17 @@ export const ghidraJavaLaunch = async (options: { }> => { const supportRoot = dirname(options.analyzeHeadlessPath); const installRoot = dirname(supportRoot); - const command = join(options.javaHome, "bin", "java"); + const selectedJavaHome = options.javaHome ?? options.environment.JAVA_HOME; + const command = + selectedJavaHome === undefined + ? options.platform === "win32" + ? "java.exe" + : "java" + : join( + selectedJavaHome, + "bin", + options.platform === "win32" ? "java.exe" : "java", + ); const isolatedProperties = [ `-Duser.home=${options.homeRoot}`, `-Djava.io.tmpdir=${options.tempRoot}`, @@ -43,14 +118,20 @@ export const ghidraJavaLaunch = async (options: { const environment = { ...snapshotEnvironment(options.environment, options.platform), }; - const configuredEnvironment = await execFileOutput( - command, - [...supportArguments, "-envvars"], - { - env: environment, - ...(options.signal === undefined ? {} : { signal: options.signal }), - }, - ); + const supportOutput = async (phase: LaunchSupportPhase) => { + try { + return await execFileOutput(command, [...supportArguments, phase], { + env: environment, + maxBuffer: GHIDRA_LAUNCH_SUPPORT_CAPTURE_BYTES, + // Wait for the helper's exit on abort so no probe outlives its owner. + stopSignal: "SIGTERM", + ...(options.signal === undefined ? {} : { signal: options.signal }), + }); + } catch (cause: unknown) { + throw new GhidraLaunchSupportError(phase, cause); + } + }; + const configuredEnvironment = await supportOutput("-envvars"); options.onStderr("-envvars", configuredEnvironment.stderr); for (const line of configuredEnvironment.stdout.split(/\r?\n/u)) { if (line === "") continue; @@ -65,14 +146,18 @@ export const ghidraJavaLaunch = async (options: { if ((environment[key] ?? "") === "") environment[key] = line.slice(separator + 1); } - const configuredArguments = await execFileOutput( - command, - [...supportArguments, "-vmargs"], - { - env: environment, - ...(options.signal === undefined ? {} : { signal: options.signal }), - }, - ); + // The merged environment becomes the analysis JVM's environment, and the + // JVM applies these options variables automatically. An injected + // -Dos.arch/-Dos.name there selects an unvalidated native directory without + // appearing in vmArguments, so reject the effective values as well. + for (const key of [ + "JAVA_TOOL_OPTIONS", + "_JAVA_OPTIONS", + "JDK_JAVA_OPTIONS", + ] as const) { + rejectJvmPlatformOverrides(jvmEnvironmentArguments(environment[key] ?? "")); + } + const configuredArguments = await supportOutput("-vmargs"); options.onStderr("-vmargs", configuredArguments.stderr); const vmArguments = configuredArguments.stdout .split(/\r?\n/u) @@ -83,6 +168,7 @@ export const ghidraJavaLaunch = async (options: { ) ) throw new Error("Invalid Ghidra LaunchSupport JVM argument"); + rejectJvmPlatformOverrides(vmArguments); return { command, environment, diff --git a/src/ghidra/GhidraLanguageCatalog.ts b/src/ghidra/GhidraLanguageCatalog.ts index 6373f187f3..1cde8769f4 100644 --- a/src/ghidra/GhidraLanguageCatalog.ts +++ b/src/ghidra/GhidraLanguageCatalog.ts @@ -1,9 +1,12 @@ -import { readdir, readFile } from "node:fs/promises"; +import { open, readdir } from "node:fs/promises"; +import type { FileHandle } from "node:fs/promises"; import { join } from "node:path"; import { DOMParser, onWarningStopParsing } from "@xmldom/xmldom"; import type { GhidraLanguageOverride } from "../config/ghidraLanguageOverride.js"; import { ConfigurationError } from "../domain/configurationErrors.js"; +import { OwnedFileHandle } from "../filesystem/OwnedFileHandle.js"; +import { readBoundedFileBytes } from "../process/BoundedFileBytes.js"; /** One language definition and the compiler specs it declares. */ export interface GhidraLanguageDefinition { @@ -16,6 +19,62 @@ const MAX_LDEFS_BYTES = 4 * 1024 * 1024; // Processor modules ship under Processors; installed extensions may add more. const LANGUAGE_ROOTS = ["Ghidra/Processors", "Ghidra/Extensions"] as const; +interface RetainedLanguageFile { + readonly path: string; + readonly owner: OwnedFileHandle; +} + +/** Language definition handles whose close is unconfirmed but retryable. */ +const pendingLanguageFileCleanup = new Set(); + +/** + * Cleanup of one or more language definition handles is unconfirmed. Any + * read failure that preceded it is kept as the primary failure. + */ +export class GhidraLanguageCatalogCleanupError extends Error { + constructor( + readonly paths: readonly string[], + readonly causes: readonly unknown[], + readonly readFailure?: unknown, + ) { + super( + `Language definition cleanup is unconfirmed: ${paths.join(", ")}`, + readFailure === undefined ? undefined : { cause: readFailure }, + ); + } +} + +/** Close one owner; retain it only while a later close can still succeed. */ +const closeLanguageFile = async ( + file: RetainedLanguageFile, +): Promise => { + try { + await file.owner.close(); + pendingLanguageFileCleanup.delete(file); + return undefined; + } catch (cause: unknown) { + // Descriptor invalidation does not prove that this owner confirmed close. + // Keep even sticky failures in the lifecycle scope so later reads surface + // the unresolved cleanup instead of silently abandoning ownership. + pendingLanguageFileCleanup.add(file); + return cause; + } +}; + +/** Retry every retained owner before reporting the failures together. */ +const retryLanguageFileCleanup = async (): Promise => { + const failed: { readonly path: string; readonly cause: unknown }[] = []; + for (const file of pendingLanguageFileCleanup) { + const cause = await closeLanguageFile(file); + if (cause !== undefined) failed.push({ path: file.path, cause }); + } + if (failed.length > 0) + throw new GhidraLanguageCatalogCleanupError( + failed.map(({ path }) => path), + failed.map(({ cause }) => cause), + ); +}; + /** Parse the `` elements of one `.ldefs` document. */ export const parseGhidraLanguageDefinitions = ( text: string, @@ -59,10 +118,40 @@ const languageDirectories = async (installDir: string): Promise => { return directories; }; +/** Read at most the catalog limit; undefined marks an oversized file. */ +const readLanguageDefinitionFile = async ( + path: string, +): Promise => { + await retryLanguageFileCleanup(); + const file = { path, owner: new OwnedFileHandle(await open(path, "r")) }; + let read: + | { readonly ok: true; readonly bytes: Buffer | undefined } + | { readonly ok: false; readonly cause: unknown }; + try { + read = { + ok: true, + bytes: await readBoundedFileBytes(file.owner.handle, MAX_LDEFS_BYTES), + }; + } catch (cause: unknown) { + read = { ok: false, cause }; + } + const cleanup = await closeLanguageFile(file); + if (cleanup !== undefined) + throw new GhidraLanguageCatalogCleanupError( + [path], + [cleanup], + read.ok ? undefined : read.cause, + ); + if (!read.ok) throw read.cause; + return read.bytes; +}; + /** Read every language definition shipped by one Ghidra installation. */ export const readGhidraLanguageCatalog = async ( installDir: string, ): Promise => { + // Surface retained cleanup even when this catalog has no definitions to read. + await retryLanguageFileCleanup(); const definitions: GhidraLanguageDefinition[] = []; for (const directory of await languageDirectories(installDir)) { let entries: string[]; @@ -73,8 +162,8 @@ export const readGhidraLanguageCatalog = async ( } for (const entry of entries.sort()) { if (!entry.endsWith(".ldefs")) continue; - const bytes = await readFile(join(directory, entry)); - if (bytes.length > MAX_LDEFS_BYTES) continue; + const bytes = await readLanguageDefinitionFile(join(directory, entry)); + if (bytes === undefined) continue; // XML 1.0 section 4.3.3: a leading UTF-8 BOM is an encoding signature, // not document content; TextDecoder consumes it, Buffer#toString does not. definitions.push( diff --git a/src/ghidra/GhidraLanguageOverride.test.ts b/src/ghidra/GhidraLanguageOverride.test.ts index 02c5bf245f..6a07445d2c 100644 --- a/src/ghidra/GhidraLanguageOverride.test.ts +++ b/src/ghidra/GhidraLanguageOverride.test.ts @@ -11,7 +11,7 @@ import { ghidraProfileLanguageOverride, sameGhidraLanguageOverride, } from "./GhidraAnalysisProfile.js"; -import type { GhidraInstallationHost } from "./GhidraInstallation.js"; +import { ghidraProfileInstallationHost as host } from "./GhidraInstallation.fixture.js"; import { admitGhidraLanguageOverride, parseGhidraLanguageDefinitions, @@ -39,20 +39,15 @@ const X86_LDEFS = ` `; -// Fake installation inspection; the language catalog is a real directory. -const host: GhidraInstallationHost = { - platform: "linux", - architecture: "x64", - readText: () => "application.version=12.1.4\napplication.java.min=21\n", - executable: () => true, - probeJava: () => ({ - version: "21.0.11", - major: 21, - home: "/jdk-21", - bits: 64, - runtime: "jdk", - }), -}; +const MIPS_LDEFS = ` + + + MIPS 32-bit little endian + + + +`; const pe: BinaryTarget = { path: "/fixture.exe", @@ -65,6 +60,23 @@ const pe: BinaryTarget = { managed: false, }; +const legacyMips: BinaryTarget = { + path: "/fixture-mips.elf", + sha256: "b".repeat(64), + kind: "executable", + format: "elf", + architecture: "mips", + availableArchitectures: ["mips"], + mips: { + elfClass: 32, + byteOrder: "little", + type: 2, + flags: 0x50000001, + abiFlags: null, + gnuAttributes: null, + }, +}; + let installDir = ""; beforeAll(async () => { installDir = await mkdtemp(join(tmpdir(), "rea-ghidra-languages-")); @@ -291,6 +303,30 @@ describe("Ghidra language override profile", () => { }); }); + it("accepts the caller-selected language on the legacy MIPS lane", async () => { + const mipsLanguages = join( + installDir, + "Ghidra/Processors/MIPS/data/languages", + ); + await mkdir(mipsLanguages, { recursive: true }); + const definition = join(mipsLanguages, "mips.ldefs"); + await writeFile(definition, MIPS_LDEFS); + try { + const resolved = await provider({ + REA_GHIDRA_LANGUAGE_ID: "MIPS:LE:32:default", + REA_GHIDRA_COMPILER_SPEC_ID: "default", + }).resolveAnalysisProfile(legacyMips); + if (!resolved.ok) throw resolved.error; + expect(resolved.value.profile?.parameters).toMatchObject({ + language_id: "MIPS:LE:32:default", + compiler_spec_id: "default", + language_selection: "configured-v1", + }); + } finally { + await rm(mipsLanguages, { recursive: true, force: true }); + } + }); + it("records the language default when no compiler spec is configured", async () => { const resolved = await provider({ REA_GHIDRA_LANGUAGE_ID: "x86:LE:32:default", diff --git a/src/ghidra/GhidraLaunchFailure.test.ts b/src/ghidra/GhidraLaunchFailure.test.ts index 246ec82d0f..d25cd9932c 100644 --- a/src/ghidra/GhidraLaunchFailure.test.ts +++ b/src/ghidra/GhidraLaunchFailure.test.ts @@ -56,7 +56,8 @@ const launchOutcomeOf = (error: Error): Record | undefined => { string, unknown >; - const outcome = cause["launch_outcome"] as Record; + const nested = cause["cause"] as Record | undefined; + const outcome = nested?.["command_outcome"] as Record; return outcome === undefined ? undefined : outcome; }; @@ -78,12 +79,16 @@ describe("Ghidra launch failure diagnostics", () => { launched.error instanceof GhidraLaunchError ? launched.error.capturedOutput : undefined; - expect(captured).toBeDefined(); - expect(captured?.exit_code).toBe(3); - expect(captured?.signal).toBeNull(); - expect(captured?.stdout).toBe(""); - expect(captured?.stderr).toContain("jvm probe exploded"); - expect(captured?.truncated).toBe(false); + expect(captured).toBeUndefined(); + const outcome = launchOutcomeOf(launched.error); + expect(outcome).toMatchObject({ + killed: false, + outputTruncated: false, + stdoutTruncated: false, + stderrTruncated: false, + stdout: "", + }); + expect(outcome?.["stderr"]).toContain("jvm probe exploded"); } finally { await rm(javaHome, { recursive: true, force: true }); await rm(runtimeRoot, { recursive: true, force: true }); @@ -103,10 +108,11 @@ describe("Ghidra launch failure diagnostics", () => { if (launched.ok) throw new Error("expected the launch to fail"); const outcome = launchOutcomeOf(launched.error); expect(outcome).toBeDefined(); - expect(outcome?.["exit_code"]).toBe(3); - expect(outcome?.["signal"]).toBeNull(); + expect(outcome?.["killed"]).toBe(false); + expect(outcome?.["outputTruncated"]).toBe(false); + expect(outcome?.["stderrTruncated"]).toBe(false); expect(outcome?.["stderr"]).toContain("jvm probe exploded"); - expect(outcome?.["truncated"]).toBe(false); + expect(outcome?.["outputTruncated"]).toBe(false); } finally { await rm(javaHome, { recursive: true, force: true }); await rm(runtimeRoot, { recursive: true, force: true }); @@ -126,9 +132,9 @@ describe("Ghidra launch failure diagnostics", () => { expect(launched.ok).toBe(false); if (launched.ok) throw new Error("expected the launch to fail"); const outcome = launchOutcomeOf(launched.error); - expect(outcome?.["truncated"]).toBe(true); - expect(outcome?.["exit_code"]).toBe(1); - expect(outcome?.["stderr_bytes"]).toBe(oversized.length); + expect(outcome?.["stderrTruncated"]).toBe(true); + expect(outcome?.["outputTruncated"]).toBe(false); + expect(outcome?.["stderrBytes"]).toBe(oversized.length); expect((outcome?.["stderr"] as string).length).toBeLessThanOrEqual( EXPECTED_OUTPUT_BOUND_BYTES, ); diff --git a/src/ghidra/GhidraLauncher.ts b/src/ghidra/GhidraLauncher.ts index e1527ea818..21990183bc 100644 --- a/src/ghidra/GhidraLauncher.ts +++ b/src/ghidra/GhidraLauncher.ts @@ -172,6 +172,8 @@ export interface GhidraHeadlessLauncherOptions { readonly mipsSoftFloat?: true; /** Spawn seam for provider-boundary lifecycle tests. */ readonly spawnProcess?: typeof spawnOwnedProviderProcess; + /** LaunchSupport probe seam for Java-free launcher fixtures. */ + readonly javaLaunch?: typeof ghidraJavaLaunch; } /** Launch Ghidra without copying scripts into or modifying its installation. */ @@ -303,43 +305,46 @@ export class GhidraHeadlessLauncher implements GhidraLauncher { throw new GhidraLaunchError( "POSIX Ghidra requires its inspected JDK home", ); + const inspectedJavaLaunch = await ( + this.options.javaLaunch ?? ghidraJavaLaunch + )({ + platform, + analyzeHeadlessPath: this.options.analyzeHeadlessPath, + ...(this.options.javaHome === undefined + ? {} + : { javaHome: this.options.javaHome }), + ...(this.options.javaVmName === undefined + ? {} + : { javaVmName: this.options.javaVmName }), + homeRoot: paths.homeRoot, + tempRoot: paths.tempRoot, + arguments: headlessArguments, + environment, + onStderr: (mode, stderr) => { + if (stderr === "") return; + const bytes = Buffer.byteLength(stderr); + const redacted = redactCapturedTransportCredential( + { text: stderr, bytes, observedBytes: bytes }, + session.token, + "[REDACTED]", + ); + const bounded = boundedUtf8Text( + redacted, + GHIDRA_LAUNCH_COMMAND_OUTPUT_BYTES, + ); + launchSupport.push({ + mode, + stderr: bounded.text, + stderr_bytes: bytes, + stderr_truncated: bounded.truncated, + }); + }, + ...(options.signal === undefined ? {} : { signal: options.signal }), + }); const command = - platform !== "win32" && this.options.javaHome !== undefined - ? await ghidraJavaLaunch({ - platform, - analyzeHeadlessPath: this.options.analyzeHeadlessPath, - javaHome: this.options.javaHome, - ...(this.options.javaVmName === undefined - ? {} - : { javaVmName: this.options.javaVmName }), - homeRoot: paths.homeRoot, - tempRoot: paths.tempRoot, - arguments: headlessArguments, - environment, - onStderr: (mode, stderr) => { - if (stderr === "") return; - const bytes = Buffer.byteLength(stderr); - const redacted = redactCapturedTransportCredential( - { text: stderr, bytes, observedBytes: bytes }, - session.token, - "[REDACTED]", - ); - const bounded = boundedUtf8Text( - redacted, - GHIDRA_LAUNCH_COMMAND_OUTPUT_BYTES, - ); - launchSupport.push({ - mode, - stderr: bounded.text, - stderr_bytes: bytes, - stderr_truncated: bounded.truncated, - }); - }, - ...(options.signal === undefined - ? {} - : { signal: options.signal }), - }) - : { ...scriptCommand, environment }; + platform === "win32" + ? { ...scriptCommand, environment } + : inspectedJavaLaunch; started = await (this.options.spawnProcess ?? spawnOwnedProviderProcess)({ command: command.command, arguments: command.arguments, diff --git a/src/ghidra/GhidraMipsProfile.test.ts b/src/ghidra/GhidraMipsProfile.test.ts index 800de9490d..1e64df9eb5 100644 --- a/src/ghidra/GhidraMipsProfile.test.ts +++ b/src/ghidra/GhidraMipsProfile.test.ts @@ -14,7 +14,7 @@ import { ghidraMipsProfileParameters, ghidraPinnedMipsLanguage, } from "./GhidraMipsProfile.js"; -import type { GhidraInstallationHost } from "./GhidraInstallation.js"; +import { ghidraProfileInstallationHost as host } from "./GhidraInstallation.fixture.js"; import { GHIDRA_SESSION_CAPABILITIES } from "./GhidraSessionValues.js"; import type { GhidraMipsSoftFloatReport } from "./GhidraMipsProfile.js"; import { @@ -22,20 +22,6 @@ import { ghidraProfileLanguageOverride, } from "./GhidraAnalysisProfile.js"; -// Fake installation inspection tests admission/profile contracts, not Ghidra. -const host: GhidraInstallationHost = { - platform: "linux", - architecture: "x64", - readText: () => "application.version=12.1.4\napplication.java.min=21\n", - executable: () => true, - probeJava: () => ({ - version: "21.0.11", - major: 21, - home: "/jdk-21", - bits: 64, - runtime: "jdk", - }), -}; const parsed = parseConfig({ GHIDRA_INSTALL_DIR: "/ghidra" }); if (!parsed.ok) throw parsed.error; const config = parsed.value; @@ -51,6 +37,7 @@ const target = (change: Partial = {}): BinaryTarget => ({ byteOrder: "little", type: 2, flags: 0x70001001, + gnuAttributes: null, abiFlags: { version: 0, isaLevel: 32, @@ -109,6 +96,7 @@ describe("bounded Ghidra MIPS admission", () => { flags1: 0, flags2: 0, }, + gnu_attributes: null, }); expect(left.value.profile?.digest).not.toBe(right.value.profile?.digest); const windows = new GhidraProvider( @@ -155,7 +143,13 @@ describe("bounded Ghidra MIPS admission", () => { reason: expect.stringContaining(reason), }); const resolved = await ghidra.resolveAnalysisProfile(value); - expect(resolved.ok).toBe(false); + expect(resolved).toMatchObject({ + ok: false, + error: { + _tag: "AnalysisUnsupportedTargetError", + reason: expect.stringContaining(reason), + }, + }); }, ); @@ -180,7 +174,13 @@ describe("recordless legacy MIPS32 admission", () => { status: "unsupported", }); const resolved = await ghidra.resolveAnalysisProfile(object); - expect(resolved.ok).toBe(false); + expect(resolved).toMatchObject({ + ok: false, + error: { + _tag: "AnalysisUnsupportedTargetError", + reason: expect.stringContaining("ET_EXEC"), + }, + }); }); it.each([ @@ -472,7 +472,7 @@ describe("MIPS16 shared-image profile", () => { ); }, ); - it.each(["sharedLoad", "abiFlags"] as const)( + it.each(["sharedLoad", "abiFlags", "gnuAttributes"] as const)( "refuses an uninspected %s declaration", (declaration) => { const value = shared(); @@ -486,9 +486,11 @@ describe("MIPS16 shared-image profile", () => { ).toMatchObject({ status: "unsupported", reason: expect.stringContaining( - declaration === "abiFlags" - ? "inspected ABI flags" - : "have not been inspected", + { + abiFlags: "inspected ABI flags", + gnuAttributes: "GNU MIPS ABI attribute inspection is unavailable", + sharedLoad: "have not been inspected", + }[declaration], ), }); }, diff --git a/src/ghidra/GhidraMipsProfile.ts b/src/ghidra/GhidraMipsProfile.ts index 76a753ccb6..83eee37ff0 100644 --- a/src/ghidra/GhidraMipsProfile.ts +++ b/src/ghidra/GhidraMipsProfile.ts @@ -1,5 +1,8 @@ import type { GhidraLanguageOverride } from "../config/ghidraLanguageOverride.js"; -import type { BinaryTarget } from "../domain/binaryTargetTypes.js"; +import type { + BinaryTarget, + MipsElfMetadata, +} from "../domain/binaryTargetTypes.js"; import type { JsonValue } from "../domain/jsonValue.js"; const MIPS_FP_ABI_SOFT = 3; @@ -65,6 +68,41 @@ const hasLegacyMipsHeaderShape = (target: BinaryTarget): boolean => { ); }; +/** Return a refusal for GNU ABI attributes this existing MIPS lane cannot model. */ +/** + * Every MIPS lane, including PSP, needs every ABI declaration source examined: + * headers outside the captured file may hold declarations that would refuse it. + */ +export const ghidraMipsIncompleteDeclarationsReason = ( + metadata: MipsElfMetadata, +): string | null => + (metadata.limitations?.length ?? 0) > 0 + ? [ + "MIPS ABI declaration inspection is incomplete; unsupported attributes or conflicting declaration sources may remain unknown.", + ...(metadata.limitations ?? []), + ].join(" ") + : null; + +export const ghidraMipsGnuAttributesUnsupportedReason = ( + metadata: MipsElfMetadata, +): string | null => { + const attributes = metadata.gnuAttributes; + if (attributes === undefined) + return "GNU MIPS ABI attribute inspection is unavailable; provider selection cannot treat it as absent."; + if (attributes === null) return null; + if (attributes.msa !== null && attributes.msa !== 0) + return "The GNU MIPS MSA ABI attribute declares an unsupported or unknown vector argument ABI."; + if ( + metadata.abiFlags === null && + attributes.fpAbi !== null && + attributes.fpAbi !== 0 && + attributes.fpAbi !== 1 && + attributes.fpAbi !== 5 + ) + return "Without a MIPS ABI flags record, the GNU floating-point ABI attribute must be absent, any, double-precision, or FPXX for the verified recordless lane."; + return null; +}; + /** Keep family recognition separate from the deliberately bounded Ghidra lane. */ export const ghidraMipsUnsupportedReason = ( target: BinaryTarget, @@ -77,6 +115,9 @@ export const ghidraMipsUnsupportedReason = ( const shared = isGhidraMips16SharedTarget(target); if (metadata.elfClass !== 32) return "This Ghidra MIPS lane supports ELF32 only; MIPS64 and n64 remain unsupported."; + const gnuAttributesReason = + ghidraMipsGnuAttributesUnsupportedReason(metadata); + if (gnuAttributesReason !== null) return gnuAttributesReason; if ( metadata.type !== MIPS_ET_EXEC && metadata.type !== MIPS_ET_REL && @@ -137,6 +178,8 @@ export const ghidraMipsUnsupportedReason = ( if (load.symbolRelativeCount !== 0) return "Symbol-bearing R_MIPS_REL32 is unsupported: Ghidra normalizes MIPS16 function addresses and can lose the source function-pointer mode tag. Symbol-index-zero REL32 is verified separately."; } + const incomplete = ghidraMipsIncompleteDeclarationsReason(metadata); + if (incomplete !== null) return incomplete; const abi = metadata.abiFlags; if (shared && abi === null && (metadata.limitations?.length ?? 0) === 0) return null; // Inspected absence leaves FP semantics unknown, never inferred. @@ -292,6 +335,17 @@ export const ghidraMipsProfileParameters = ( flags1: abi.flags1, flags2: abi.flags2, }, + ...(target.mips.gnuAttributes === undefined + ? {} + : { + gnu_attributes: + target.mips.gnuAttributes === null + ? null + : { + fp_abi: target.mips.gnuAttributes.fpAbi, + msa: target.mips.gnuAttributes.msa, + }, + }), ...(target.mips.sharedLoad === undefined ? {} : { diff --git a/src/ghidra/GhidraPeProfile.test.ts b/src/ghidra/GhidraPeProfile.test.ts index 561aad7012..b8c869564f 100644 --- a/src/ghidra/GhidraPeProfile.test.ts +++ b/src/ghidra/GhidraPeProfile.test.ts @@ -1,95 +1,297 @@ -import { describe, expect, it } from "vitest"; +import { expect, it } from "vitest"; import type { BinaryTarget } from "../domain/binaryTargetTypes.js"; -import { ghidraPeCodeViewLimitations } from "./GhidraPeProfile.js"; +import { ghidraPePdbLimitations } from "./GhidraPeProfile.js"; +import { + ghidraPdbLimitations, + type GhidraSessionInfo, +} from "./GhidraSessionValues.js"; -describe("ghidraPeCodeViewLimitations", () => { - it("returns empty limitations for non-PE targets", () => { - const elfTarget: BinaryTarget = { - path: "/test/file", - sha256: "abc", - kind: "executable", - format: "elf", - architecture: "x86_64", - availableArchitectures: ["x86_64"], +const target: BinaryTarget = { + path: "/fixture/public.exe", + sha256: "a".repeat(64), + kind: "executable", + format: "pe", + architecture: "x86_64", + availableArchitectures: ["x86_64"], + executableRole: "application", + managed: false, + codeView: { + pdbPath: "public.pdb", + pdbFileName: "public.pdb", + guid: "01234567-89ab-cdef-0123-456789abcdef", + age: 10, + }, +}; +const absentAttributes = { + file: null, + guid: null, + age: null, + signature: null, + version: null, + loaded: null, +}; + +it.each(["file", "guid", "age"] as const)( + "retains declared identity when only reported %s is omitted", + (attribute) => { + const pdb: GhidraSessionInfo["pdb"] = { + file: "program.pdb", + guid: "reported-guid", + age: "a", + signature: null, + version: "RSDS", + loaded: null, + resource_limit: { omitted_attributes: [attribute], utf16_budget: 4096 }, + [attribute]: null, }; - expect(ghidraPeCodeViewLimitations(elfTarget)).toEqual([]); - }); + const caveats = ghidraPePdbLimitations(target, pdb).join(" "); + expect(caveats).toContain("omitted"); + expect(caveats).toContain('PDB "public.pdb"'); + expect(caveats).toContain("01234567-89ab-cdef-0123-456789abcdef"); + expect(caveats).toContain("load state unknown"); + }, +); - it("does not claim CodeView symbols were unapplied when the session loaded a PDB", () => { - const peTarget: BinaryTarget = { - path: "/test/file.exe", - sha256: "abc", - kind: "executable", - architecture: "x86_64", - availableArchitectures: ["x86_64"], - executableRole: "application", - managed: false, - format: "pe", +it.each([undefined, absentAttributes])( + "retains declared identity and unknown load state when reported attributes are %j", + (pdb) => { + const caveats = ghidraPePdbLimitations(target, pdb); + expect(caveats).toHaveLength(1); + expect(caveats[0]).toContain( + 'PDB "public.pdb" (GUID 01234567-89ab-cdef-0123-456789abcdef, age (decimal) 10)', + ); + expect(caveats[0]).toContain("load state unknown"); + expect(caveats[0]).not.toContain("were not applied"); + expect(caveats[0]).not.toContain("names are analysis-generated"); + }, +); + +it.each([true, false, null])( + "uses reported PDB observations once with loaded = %s", + (loaded) => { + const pdb = { + file: "public.pdb", + guid: "{01234567-89AB-CDEF-0123-456789ABCDEF}", + age: "0X000A", + signature: null, + version: "RSDS", + loaded, + }; + const caveats = ghidraPePdbLimitations(target, pdb); + expect(caveats).toEqual(ghidraPdbLimitations(pdb)); + expect(caveats).toHaveLength(1); + expect(caveats.join(" ")).not.toContain('PDB "public.pdb"'); + }, +); + +it.each([ + { file: "different.pdb", guid: "0123456789abcdef0123456789abcdef", age: "a" }, + { file: "public.pdb", guid: "1123456789abcdef0123456789abcdef", age: "a" }, + { file: "public.pdb", guid: "0123456789abcdef0123456789abcdef", age: "10" }, + { file: "public.pdb", guid: "unrecognized-guid", age: "a" }, + { file: "public.pdb", guid: "0123456789abcdef0123456789abcdef", age: "ten" }, +])("retains non-equivalent declared PDB identity for %j", (identity) => { + for (const loaded of [false, null, true]) { + const pdb = { ...absentAttributes, ...identity, version: "RSDS", loaded }; + const caveats = ghidraPePdbLimitations(target, pdb); + expect(caveats[0]).toBe(ghidraPdbLimitations(pdb)[0]); + expect(caveats[0]).toContain(identity.file); + expect(caveats[0]).toContain(identity.guid); + expect(caveats[0]).toContain(`age (hex) ${identity.age}`); + expect(caveats).toHaveLength(2); + expect(caveats[1]).toContain('PDB "public.pdb"'); + expect(caveats[1]).toContain("01234567-89ab-cdef-0123-456789abcdef"); + expect(caveats[1]).toContain("age (decimal) 10"); + expect(caveats[1]).toContain("equivalence is not established"); + expect(pdb).toMatchObject({ ...identity, loaded }); + } +}); + +it.each([ + "0123456789ABCDEF0123456789ABCDEF", + "01234567-89ab-cdef-0123-456789abcdef", +])("avoids duplicate identity for normalized matching GUID %s", (guid) => { + const pdb = { ...absentAttributes, file: "public.pdb", guid, age: "a" }; + expect(ghidraPePdbLimitations(target, pdb)).toEqual( + ghidraPdbLimitations(pdb), + ); +}); + +it("retains explicit failure to load without asserting all names are generated", () => { + const caveats = ghidraPePdbLimitations(target, { + ...absentAttributes, + loaded: false, + }); + expect(caveats[0]).toContain("reports not loaded"); + expect(caveats[0]).not.toContain("names are analysis-generated"); +}); + +it("retains a loaded-state attestation caveat even when reference attributes are absent", () => { + const caveats = ghidraPePdbLimitations(target, { + ...absentAttributes, + loaded: true, + }); + expect(caveats[0]).toEqual( + ghidraPdbLimitations({ ...absentAttributes, loaded: true })[0], + ); + expect(caveats).toHaveLength(2); + expect(caveats[1]).toContain('PDB "public.pdb"'); + expect(caveats[0]).toContain("not an independent PDB identity"); + expect(caveats[0]).not.toContain("reports not loaded"); +}); + +it("preserves omitted-attribute resource coverage with declared fallback identity", () => { + const pdb: GhidraSessionInfo["pdb"] = { + ...absentAttributes, + resource_limit: { omitted_attributes: ["file"], utf16_budget: 4096 }, + }; + const caveats = ghidraPePdbLimitations(target, pdb); + expect(caveats).toHaveLength(2); + expect(caveats[0]).toContain("omitted"); + expect(caveats[1]).toContain("public.pdb"); +}); + +it("reports NB10 declared age without inventing a GUID", () => { + const caveats = ghidraPePdbLimitations( + { + ...target, codeView: { - pdbPath: "C:\\symbols.pdb", - pdbFileName: "symbols.pdb", - guid: "12345678-1234-1234-1234-123456789abc", - age: 1, + pdbPath: "legacy.pdb", + pdbFileName: "legacy.pdb", + guid: null, + age: 2, }, - }; - expect(ghidraPeCodeViewLimitations(peTarget, true)).toEqual([]); - }); + }, + undefined, + ); + expect(caveats[0]).toContain('PDB "legacy.pdb" (age (decimal) 2)'); + expect(caveats[0]).not.toContain("GUID"); +}); - it("returns empty limitations for PE target without codeView", () => { - const peTarget: BinaryTarget = { - path: "/test/file.exe", - sha256: "def", - kind: "executable", - format: "pe", - architecture: "x86_64", - availableArchitectures: ["x86_64"], - executableRole: "application", - managed: false, - }; - expect(ghidraPeCodeViewLimitations(peTarget)).toEqual([]); - }); - - it("returns limitation with GUID and age for RSDS CodeView records", () => { - const peTarget: BinaryTarget = { - path: "/test/file.exe", - sha256: "def", - kind: "executable", - format: "pe", - architecture: "x86_64", - availableArchitectures: ["x86_64"], - executableRole: "application", - managed: false, +it.each([false, null, true])( + "keeps an omitted NB10 GUID unknown with loaded = %s", + (loaded) => { + const pdbPath = "a".repeat(4070) + "\\legacy.pdb"; + const legacy = { + ...target, codeView: { - pdbPath: "C:\\build\\app.pdb", - pdbFileName: "app.pdb", - guid: "01234567-89ab-cdef-0123-456789abcdef", - age: 1, - }, - }; - expect(ghidraPeCodeViewLimitations(peTarget)).toEqual([ - 'The target references debug symbols "app.pdb" (GUID 01234567-89ab-cdef-0123-456789abcdef, age 1) that were not applied; procedure names are analysis-generated.', - ]); - }); - - it("returns limitation without GUID for NB10 CodeView records", () => { - const peTarget: BinaryTarget = { - path: "/test/file.exe", - sha256: "def", - kind: "executable", - format: "pe", - architecture: "x86", - availableArchitectures: ["x86"], - executableRole: "application", - managed: false, - codeView: { - pdbPath: "D:\\test\\legacy.pdb", + pdbPath, pdbFileName: "legacy.pdb", guid: null, age: 2, }, }; - expect(ghidraPeCodeViewLimitations(peTarget)).toEqual([ - 'The target references debug symbols "legacy.pdb" (age 2) that were not applied; procedure names are analysis-generated.', + const pdb = { + ...absentAttributes, + file: pdbPath, + age: "2", + signature: "01234567", + version: "NB10", + loaded, + }; + expect(ghidraPePdbLimitations(legacy, pdb)).toEqual( + ghidraPdbLimitations(pdb), + ); + const omitted: GhidraSessionInfo["pdb"] = { + ...pdb, + resource_limit: { omitted_attributes: ["guid"], utf16_budget: 4096 }, + }; + const caveats = ghidraPePdbLimitations(legacy, omitted); + expect(caveats).toEqual([ + ...ghidraPdbLimitations(omitted), + expect.stringContaining('PDB "legacy.pdb" (age (decimal) 2)'), ]); - }); + expect(caveats.at(-1)).toContain("equivalence is not established"); + }, +); + +it.each(["signature", "version"] as const)( + "keeps matching NB10 identity when only %s is omitted", + (attribute) => { + const legacy = { + ...target, + codeView: { + pdbPath: "legacy.pdb", + pdbFileName: "legacy.pdb", + guid: null, + age: 2, + }, + }; + const pdb: GhidraSessionInfo["pdb"] = { + ...absentAttributes, + file: "legacy.pdb", + age: "2", + resource_limit: { omitted_attributes: [attribute], utf16_budget: 4096 }, + }; + expect(ghidraPePdbLimitations(legacy, pdb)).toEqual( + ghidraPdbLimitations(pdb), + ); + }, +); + +it("retains missing-report uncertainty without a declared CodeView record", () => { + const { codeView: _, ...withoutReference } = target; + expect(ghidraPePdbLimitations(withoutReference, undefined)).toEqual( + ghidraPdbLimitations(undefined), + ); +}); + +it("adds no PDB warning for a non-PE target", () => { + const elf: BinaryTarget = { + path: "/fixture/program", + sha256: "a".repeat(64), + kind: "executable", + format: "elf", + architecture: "x86_64", + availableArchitectures: ["x86_64"], + }; + expect(ghidraPePdbLimitations(elf, undefined)).toEqual([]); +}); + +it("bounds an escaped inline locator while preserving its declared source value", () => { + const name = '"'.repeat(5000) + ".pdb"; + const selected = { + ...target, + codeView: { ...target.codeView!, pdbFileName: name }, + }; + const caveats = ghidraPePdbLimitations(selected, undefined).join(" "); + expect(Buffer.byteLength(caveats)).toBeLessThan(1024); + expect(caveats).toContain("locator omitted"); + expect(caveats).toContain("01234567-89ab-cdef-0123-456789abcdef"); + expect(selected.codeView.pdbFileName).toBe(name); +}); + +it.each([ + ["long ASCII", "a".repeat(171)], + ["ASCII at the byte budget", "a".repeat(1022)], + ["ASCII beyond the byte budget", "a".repeat(1023)], + ["quotes at the byte budget", '"'.repeat(511)], + ["quotes beyond the byte budget", '"'.repeat(512)], + ["short control escapes", "\n".repeat(511)], + ["short control escapes beyond the byte budget", "\n".repeat(512)], + ["six-byte control escapes", "\u0001".repeat(170)], + ["six-byte control escapes beyond the byte budget", "\u0001".repeat(171)], + ["UTF-8 BMP text", "中".repeat(340)], + ["UTF-8 BMP text beyond the byte budget", "中".repeat(341)], + ["surrogate pairs", "😀".repeat(255)], + ["surrogate pairs beyond the byte budget", "😀".repeat(256)], + ["unpaired high surrogates", "\ud800".repeat(170)], + ["unpaired high surrogates beyond the byte budget", "\ud800".repeat(171)], + ["unpaired low surrogates", "\udfff".repeat(170)], + ["unpaired low surrogates beyond the byte budget", "\udfff".repeat(171)], +])("uses actual JSON UTF-8 size for %s", (_, name) => { + const selected = { + ...target, + codeView: { ...target.codeView!, pdbFileName: name }, + }; + const caveat = ghidraPePdbLimitations(selected, undefined).join(" "); + const encoded = JSON.stringify(name); + if (Buffer.byteLength(encoded) <= 1024) { + expect(caveat).toContain(`PDB ${encoded} (`); + expect(caveat).not.toContain("locator omitted"); + } else { + expect(caveat).toContain("locator omitted"); + expect(caveat).not.toContain(encoded); + } + expect(selected.codeView.pdbFileName).toBe(name); }); diff --git a/src/ghidra/GhidraPeProfile.ts b/src/ghidra/GhidraPeProfile.ts index 217b3256f3..7f60edb912 100644 --- a/src/ghidra/GhidraPeProfile.ts +++ b/src/ghidra/GhidraPeProfile.ts @@ -1,24 +1,113 @@ -import type { BinaryTarget } from "../domain/binaryTargetTypes.js"; +import type { + BinaryTarget, + PeCodeViewRecord, +} from "../domain/binaryTargetTypes.js"; +import { + ghidraPdbLimitations, + type GhidraSessionInfo, +} from "./GhidraSessionValues.js"; -/** - * Report unapplied PE companion PDB symbols when CodeView debug information is present. - * - * Ghidra does not automatically locate or apply sidecar PDBs in REA's isolated - * runtime directories without companion admission. When the PE target header - * references debug symbols, surface this limitation so callers understand that - * procedure names remain analysis-generated. - */ -export const ghidraPeCodeViewLimitations = ( - target: BinaryTarget, - pdbLoaded = false, -): readonly string[] => { - if (target.format !== "pe" || target.codeView === undefined || pdbLoaded) { - return []; - } - const { pdbFileName, guid, age } = target.codeView; - return [ - guid !== null - ? `The target references debug symbols "${pdbFileName}" (GUID ${guid}, age ${age}) that were not applied; procedure names are analysis-generated.` - : `The target references debug symbols "${pdbFileName}" (age ${age}) that were not applied; procedure names are analysis-generated.`, - ]; +const PDB_LOCATOR_INLINE_BYTE_BUDGET = 1024; + +/** Establish equivalence only for the declared fields and recognized formats. */ +const reportedPdbMatchesDeclaration = ( + declared: PeCodeViewRecord, + reported: NonNullable, +): boolean => { + if ( + reported.resource_limit?.omitted_attributes.some( + (attribute) => + attribute === "file" || attribute === "guid" || attribute === "age", + ) || + (reported.file !== declared.pdbPath && + reported.file !== declared.pdbFileName) || + reported.age === null || + !/^(?:0x)?[0-9a-f]+$/iu.test(reported.age) || + Number.parseInt(reported.age, 16) !== declared.age + ) + return false; + if (declared.guid === null) return reported.guid === null; + if (reported.guid === null) return false; + let guid = reported.guid; + if (guid.startsWith("{") && guid.endsWith("}")) guid = guid.slice(1, -1); + if ( + !/^(?:[0-9a-f]{32}|[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})$/iu.test( + guid, + ) + ) + return false; + return ( + guid.replaceAll("-", "").toLowerCase() === + declared.guid.replaceAll("-", "").toLowerCase() + ); +}; + +/** Measure JSON quotes, escapes and UTF-8 without allocating the encoded string. */ +const pdbLocatorFitsInlineBudget = (value: string): boolean => { + if (value.length * 6 + 2 <= PDB_LOCATOR_INLINE_BYTE_BUDGET) return true; + if (value.length + 2 > PDB_LOCATOR_INLINE_BYTE_BUDGET) return false; + let bytes = 2; + for (let index = 0; index < value.length; index += 1) { + const point = value.codePointAt(index) ?? 0; + if (point > 0xffff) { + bytes += 4; + index += 1; + } else if (point >= 0xd800 && point <= 0xdfff) { + // JSON.stringify escapes unpaired UTF-16 surrogates as six ASCII bytes. + bytes += 6; + } else if (point < 0x20) { + bytes += + point === 8 || + point === 9 || + point === 10 || + point === 12 || + point === 13 + ? 2 + : 6; + } else if (point === 0x22 || point === 0x5c) { + bytes += 2; + } else { + bytes += point < 0x80 ? 1 : point < 0x800 ? 2 : 3; + } + if (bytes > PDB_LOCATOR_INLINE_BYTE_BUDGET) return false; + } + return true; +}; + +/** Use reported Program facts first, retaining declared PE facts as a fallback. */ +export const ghidraPePdbLimitations = ( + target: BinaryTarget, + pdb: GhidraSessionInfo["pdb"], +): readonly string[] => { + if (target.format !== "pe") return []; + const reported = ghidraPdbLimitations(pdb); + if ( + target.codeView === undefined || + (pdb !== undefined && reportedPdbMatchesDeclaration(target.codeView, pdb)) + ) + return reported; + const { pdbFileName, guid, age } = target.codeView; + // Include JSON escaping in the inline budget before allocating that representation. + const locator = pdbLocatorFitsInlineBudget(pdbFileName) + ? JSON.stringify(pdbFileName) + : `(declared locator omitted from this inline caveat: ${PDB_LOCATOR_INLINE_BYTE_BUDGET}-byte resource budget)`; + const identity = + guid === null + ? `age (decimal) ${age}` + : `GUID ${guid}, age (decimal) ${age}`; + const state = + pdb === undefined + ? "Ghidra did not report PDB attributes; load state unknown" + : pdb.loaded === false + ? "Ghidra reports not loaded" + : pdb.loaded === true + ? "Ghidra reports a loaded PDB; this declared identity is not an independent matching-PDB attestation" + : "Ghidra reports load state unknown"; + const comparison = + pdb !== undefined && + [pdb.file, pdb.guid, pdb.age].some((value) => value !== null) + ? " Reported and declared PDB observations are retained separately; equivalence is not established." + : ""; + const declared = `The PE CodeView record references PDB ${locator} (${identity}); ${state}.${comparison} Original PDB names and types may be missing. Use list_procedures and address-based analysis to inspect recovered functions. REA does not admit a caller-selected PDB companion.`; + return pdb === undefined ? [declared] : [...reported, declared]; }; diff --git a/src/ghidra/GhidraPowerPcProfile.test.ts b/src/ghidra/GhidraPowerPcProfile.test.ts new file mode 100644 index 0000000000..0be7aa898e --- /dev/null +++ b/src/ghidra/GhidraPowerPcProfile.test.ts @@ -0,0 +1,159 @@ +import { mkdir, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { createTestTempDirectory } from "../../tests/fixtures/temporaryDirectory.js"; +import { parseConfig } from "../config/parseConfig.js"; +import type { + BinaryTarget, + PowerPcElfMetadata, +} from "../domain/binaryTargetTypes.js"; +import { silentLogger } from "../logger.js"; +import { HopperProvider } from "../hopper/HopperProvider.js"; +import { hopperLoaderArgsForTarget } from "../hopper/HopperAnalysisProfile.js"; +import { IdaProvider } from "../ida/IdaProvider.js"; +import { + ghidraPinnedLanguage, + ghidraProfileLanguageOverride, +} from "./GhidraAnalysisProfile.js"; +import { GhidraProvider } from "./GhidraProvider.js"; +import { ghidraProfileInstallationHost as host } from "./GhidraInstallation.fixture.js"; + +// Abbreviated from Ghidra 12.1.4 Processors/PowerPC/data/languages/ppc.ldefs. +const PPC_LDEFS = ` + + + + + + + + +`; + +const configFor = (environment: NodeJS.ProcessEnv) => { + const parsed = parseConfig({ GHIDRA_INSTALL_DIR: "/ghidra", ...environment }); + if (!parsed.ok) throw parsed.error; + return parsed.value; +}; +const config = configFor({}); +const target = (change: Partial = {}): BinaryTarget => ({ + path: "/fixture.elf", + sha256: "a".repeat(64), + kind: "executable", + format: "elf", + architecture: "powerpc", + availableArchitectures: ["powerpc"], + powerpc: { elfClass: 32, byteOrder: "big", type: 2, flags: 0, ...change }, +}); + +describe("bounded Ghidra PowerPC admission", () => { + it("binds the generic language instead of the loader's e500 choice", async () => { + const ghidra = new GhidraProvider(config, silentLogger, {}, host); + expect(ghidra.inspectAvailability()).toMatchObject({ status: "available" }); + expect(ghidra.inspectTargetSupport(target()).status).toBe("supported"); + const resolved = await ghidra.resolveAnalysisProfile(target()); + if (!resolved.ok) throw resolved.error; + const parameters = resolved.value.profile?.parameters ?? {}; + expect(parameters).toMatchObject({ + architecture: "powerpc", + language_id: "PowerPC:BE:32:default", + compiler_spec_id: "default", + language_selection: "provider-pinned-v1", + powerpc_elf: { elf_class: 32, byte_order: "big", type: 2, flags: 0 }, + powerpc_support_lane: "elf32-be-exec-generic-v1", + }); + expect(ghidraPinnedLanguage(target())).toEqual({ + languageId: "PowerPC:BE:32:default", + compilerSpecId: "default", + }); + expect(ghidraProfileLanguageOverride(parameters)).toEqual( + ghidraPinnedLanguage(target()), + ); + }); + + it("lets a configured language replace the lane default", async () => { + const installDir = await createTestTempDirectory("rea-ghidra-ppc-"); + const languages = join( + installDir, + "Ghidra/Processors/PowerPC/data/languages", + ); + await mkdir(languages, { recursive: true }); + await writeFile(join(languages, "ppc.ldefs"), PPC_LDEFS); + const ghidra = new GhidraProvider( + configFor({ + GHIDRA_INSTALL_DIR: installDir, + REA_GHIDRA_LANGUAGE_ID: "PowerPC:BE:32:e500", + }), + silentLogger, + {}, + host, + ); + const resolved = await ghidra.resolveAnalysisProfile(target()); + if (!resolved.ok) throw resolved.error; + const parameters = resolved.value.profile?.parameters ?? {}; + expect(parameters).toMatchObject({ + language_id: "PowerPC:BE:32:e500", + language_selection: "configured-v1", + }); + expect(ghidraProfileLanguageOverride(parameters)).toEqual({ + languageId: "PowerPC:BE:32:e500", + }); + }); + + it("commits header flags so a changed interpretation changes the profile", async () => { + const ghidra = new GhidraProvider(config, silentLogger, {}, host); + const left = await ghidra.resolveAnalysisProfile(target()); + const right = await ghidra.resolveAnalysisProfile( + target({ flags: 0x00010000 }), + ); + if (!left.ok) throw left.error; + if (!right.ok) throw right.error; + expect(left.value.profile?.digest).not.toBe(right.value.profile?.digest); + }); + + it("does not widen Windows P0", () => { + const windows = new GhidraProvider( + config, + silentLogger, + {}, + { ...host, platform: "win32" }, + ); + expect(windows.inspectTargetSupport(target()).status).toBe("unsupported"); + }); + + it.each([ + [{ elfClass: 64 }, "ELF32"], + [{ byteOrder: "little" }, "big-endian"], + [{ type: 1 }, "ET_EXEC"], + [{ type: 3 }, "ET_EXEC"], + [{ flags: 0x80000000 }, "EF_PPC_EMB"], + ] satisfies [Partial, string][])( + "refuses an unverified PowerPC interpretation %j", + async (change, reason) => { + const ghidra = new GhidraProvider(config, silentLogger, {}, host); + const value = target(change); + expect(ghidra.inspectTargetSupport(value)).toMatchObject({ + status: "unsupported", + reason: expect.stringContaining(reason), + }); + const resolved = await ghidra.resolveAnalysisProfile(value); + expect(resolved).toMatchObject({ + ok: false, + error: { + _tag: "AnalysisUnsupportedTargetError", + reason: expect.stringContaining(reason), + }, + }); + }, + ); + + it("keeps PowerPC out of unverified Hopper and IDA adapter lanes", () => { + const hopper = new HopperProvider(config, silentLogger, {}, "linux"); + expect(hopper.inspectTargetSupport(target()).status).toBe("unsupported"); + expect(hopperLoaderArgsForTarget(target()).ok).toBe(false); + const ida = new IdaProvider(config, {}); + expect(ida.inspectTargetSupport(target()).status).toBe("unsupported"); + }); +}); diff --git a/src/ghidra/GhidraPowerPcProfile.ts b/src/ghidra/GhidraPowerPcProfile.ts new file mode 100644 index 0000000000..c1655f2523 --- /dev/null +++ b/src/ghidra/GhidraPowerPcProfile.ts @@ -0,0 +1,67 @@ +import type { GhidraLanguageOverride } from "../config/ghidraLanguageOverride.js"; +import type { BinaryTarget } from "../domain/binaryTargetTypes.js"; +import type { JsonValue } from "../domain/jsonValue.js"; + +/** + * Generic 32-bit big-endian PowerPC. Ghidra's ELF loader selects the e500 + * (Book-E/SPE) variant for a plain EM_PPC header, so the lane binds this + * language instead of accepting the loader's choice. + */ +export const GHIDRA_POWERPC_LANGUAGE: GhidraLanguageOverride = { + languageId: "PowerPC:BE:32:default", + compilerSpecId: "default", +}; + +/** Keep family recognition separate from the deliberately bounded Ghidra lane. */ +export const ghidraPowerPcUnsupportedReason = ( + target: BinaryTarget, +): string | null => { + if (target.kind !== "executable" || target.architecture !== "powerpc") + return null; + const metadata = target.powerpc; + if (target.format !== "elf" || metadata === undefined) + return "PowerPC analysis requires ELF header metadata; a family name alone does not select a processor variant."; + if (metadata.elfClass !== 32) + return "This Ghidra PowerPC lane supports ELF32 only; PPC64 remains unsupported."; + if (metadata.byteOrder !== "big") + return "This Ghidra PowerPC lane supports big-endian ELF32 only; little-endian PowerPC needs separate verification."; + if (metadata.type !== 2) + return "This Ghidra PowerPC lane supports ET_EXEC only; relocatable objects and shared images need separate load verification."; + // EF_PPC_EMB (0x80000000) marks embedded variants; other bits only declare + // relocation conventions that do not change instruction decoding. + if ((metadata.flags & 0x80000000) !== 0) + return "EF_PPC_EMB declares an embedded PowerPC variant that needs a separately supported processor profile."; + return null; +}; + +/** + * The language the lane imports with, unless REA_GHIDRA_LANGUAGE_ID already + * selected one through the committed profile. + */ +export const ghidraPowerPcLanguage = ( + target: BinaryTarget, +): GhidraLanguageOverride | undefined => + target.kind === "executable" && target.architecture === "powerpc" + ? GHIDRA_POWERPC_LANGUAGE + : undefined; + +/** Commit source ELF interpretation without confusing target ISA with host CPU. */ +export const ghidraPowerPcProfileParameters = ( + target: BinaryTarget, +): Readonly> => { + if ( + target.kind !== "executable" || + target.architecture !== "powerpc" || + target.powerpc === undefined + ) + return {}; + return { + powerpc_elf: { + elf_class: target.powerpc.elfClass, + byte_order: target.powerpc.byteOrder, + type: target.powerpc.type, + flags: target.powerpc.flags, + }, + powerpc_support_lane: "elf32-be-exec-generic-v1", + }; +}; diff --git a/src/ghidra/GhidraProvider.behavior.test.ts b/src/ghidra/GhidraProvider.behavior.test.ts index b485f6ab31..5c585df5b3 100644 --- a/src/ghidra/GhidraProvider.behavior.test.ts +++ b/src/ghidra/GhidraProvider.behavior.test.ts @@ -8,7 +8,9 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { fileURLToPath } from "node:url"; import { promisify } from "node:util"; -import { describe, expect, it, onTestFinished, vi } from "vitest"; +import { afterEach, describe, expect, it, onTestFinished, vi } from "vitest"; +import * as nativeLoader from "../windows/WindowsNativeLoader.js"; +import { WINDOWS_NATIVE_AUTHORITY_UNAVAILABLE_REASON } from "../process/WindowsAuthority.js"; import { fixtureDosLoadImage } from "./GhidraLoadImage.fixture.js"; import { fixtureRawBinaryLoadImage } from "./GhidraRawBinaryLoadImage.fixture.js"; import { @@ -30,19 +32,27 @@ import { err, ok } from "../domain/result.js"; import type { Result } from "../domain/result.js"; import { GhidraSessionError } from "./GhidraSessionError.js"; import { silentLogger } from "../logger.js"; -import * as WindowsNativeLoader from "../windows/WindowsNativeLoader.js"; -import { WINDOWS_NATIVE_AUTHORITY_UNAVAILABLE_REASON } from "../process/WindowsAuthority.js"; import { ProviderCleanupError } from "../domain/providerCleanupError.js"; import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; import { ProviderStartupDeadline } from "../process/ProviderDeadline.js"; +import { createAnalysisProfile } from "../domain/analysisProfile.js"; +import { createAnalysisExecution } from "../application/AnalysisProvider.js"; +import { AnalysisSnapshotCache } from "../application/binary/AnalysisSnapshotCache.js"; +import { createEvidence } from "../domain/evidence.js"; +import { createEvidenceBundle } from "../domain/evidenceBundle.js"; const INSTALL = "/opt/ghidra_12.1.4_PUBLIC"; -const installationHost = (): GhidraInstallationHost => ({ - platform: "linux", +afterEach(() => vi.restoreAllMocks()); +const installationHost = ( + platform: NodeJS.Platform = "linux", +): GhidraInstallationHost => ({ + platform, architecture: "x64", readText: () => "application.version=12.1.4\napplication.java.min=21\n", executable: () => true, probeJava: () => ({ + osArch: "amd64", + osName: platform === "win32" ? "Windows 11" : "Linux", version: "21.0.11", major: 21, home: "/usr/lib/jvm/jdk-21", @@ -76,6 +86,69 @@ const createElfClient = async (ghidra: GhidraProvider) => { return ghidra.createClient(target, resolved.value.profile); }; +it.each([ + "program-info-v1", + "program-info-v2", + "program-info-v3", + "program-info-v4", +])( + "retains historical PE PDB Evidence for %s without replaying its old projection", + async (pdbObservations) => { + const target = peTarget("x86_64"); + const resolved = await provider().resolveAnalysisProfile(target); + if (!resolved.ok || resolved.value.profile === null) + throw new Error("Expected admitted PE profile"); + const current = resolved.value.profile; + const prior = createAnalysisProfile(current.provider, { + ...current.parameters, + pdb_observations: pdbObservations, + }); + const operation = "search_procedures"; + const parameters = { pattern: "wmain" }; + const limitations = [ + pdbObservations === "program-info-v4" + ? "Legacy projection discarded NB10 declaration when the reported GUID was omitted." + : "Legacy projection collapsed unknown PDB load state.", + ]; + const cache = new AnalysisSnapshotCache(); + cache.record({ + target, + profile: prior, + operation, + parameters, + execution: createAnalysisExecution([], prior.provider, { + subject: target, + analysisProfile: prior, + limitations, + }), + }); + const evidence = createEvidence(target, prior.provider, { + operation, + parameters, + result: [], + analysisProfile: prior, + limitations, + }); + const exported = cache.export( + target, + prior, + createEvidenceBundle([evidence]), + ); + if (!exported.ok) throw exported.error; + const imported = new AnalysisSnapshotCache(); + expect(imported.import(exported.value, undefined, () => ok(0)).ok).toBe( + true, + ); + expect( + imported.lookup(target, current, operation, parameters), + ).toBeUndefined(); + expect( + imported.lookup(target, prior, operation, parameters)?.limitations, + ).toEqual(expect.arrayContaining(limitations)); + expect(exported.value.evidence_bundle.records).toContainEqual(evidence); + }, +); + describe("Ghidra provider", () => { it("commits DOS loader and real-mode language while retaining the CPU family", async () => { const ghidra = provider(); @@ -102,7 +175,7 @@ describe("Ghidra provider", () => { }, }, }); - const windows = provider({ ...installationHost(), platform: "win32" }); + const windows = provider(installationHost("win32")); expect(windows.inspectTargetSupport(target).status).toBe("unsupported"); }); it("discovers the exact installation once without launching Ghidra", () => { @@ -112,6 +185,8 @@ describe("Ghidra provider", () => { probeJava: () => { probeCount += 1; return { + osArch: "amd64", + osName: "Linux", version: "21.0.11", major: 21, home: "/usr/lib/jvm/jdk-21", @@ -193,7 +268,7 @@ describe("Ghidra provider", () => { describe("Ghidra raw-binary profile", () => { it("admits raw 6502 on Windows and binds its 16-bit language without mutation authority", async () => { - const ghidra = provider({ ...installationHost(), platform: "win32" }); + const ghidra = provider(installationHost("win32")); const target: BinaryTarget = { path: "/tmp/6502.bin", sha256: "a".repeat(64), @@ -260,7 +335,7 @@ describe("Ghidra raw-binary profile", () => { expect(shifted.value.profile?.digest).not.toBe( resolved.value.profile.digest, ); - const windows = provider({ ...installationHost(), platform: "win32" }); + const windows = provider(installationHost("win32")); expect(windows.inspectTargetSupport(target).status).toBe("unsupported"); }); }); @@ -302,7 +377,7 @@ describe("Ghidra Mach-O slice support", () => { describe("Ghidra platform support", () => { it("commits the PE role and managed classification to profile identity", async () => { - const ghidra = provider({ ...installationHost(), platform: "win32" }); + const ghidra = provider(installationHost("win32")); const application = peTarget("x86_64"); const library = { ...application, @@ -343,7 +418,7 @@ describe("Ghidra platform support", () => { }); it("keeps Windows annotation mutation unavailable independently of native controls", () => { - const ghidra = provider({ ...installationHost(), platform: "win32" }); + const ghidra = provider(installationHost("win32")); expect( ghidra .capabilities() @@ -354,18 +429,20 @@ describe("Ghidra platform support", () => { availabilityCode: "unsupported_host", }); }); +}); +describe("Ghidra Windows platform support", () => { it("keeps Windows P0 unavailable until native isolation authority exists", () => { const native = vi - .spyOn(WindowsNativeLoader, "systemWindowsNativeAuthority") + .spyOn(nativeLoader, "systemWindowsNativeAuthority") .mockReturnValue({ available: false, + failureStage: "artifact", reason: WINDOWS_NATIVE_AUTHORITY_UNAVAILABLE_REASON, }); onTestFinished(() => native.mockRestore()); - const ghidra = provider({ ...installationHost(), platform: "win32" }); + const ghidra = provider(installationHost("win32")); const nativeApplication = peTarget("x86_64"); - for (const architecture of ["x86", "x86_64"] as const) expect(ghidra.inspectTargetSupport(peTarget(architecture))).toMatchObject( { @@ -380,11 +457,15 @@ describe("Ghidra platform support", () => { ); expect(ghidra.inspectAvailability()).toMatchObject({ status: "unavailable", - code: "unsupported_host", + code: "runtime_missing", reason: expect.stringContaining( "Windows native controls are unavailable", ), diagnostics: { + windows_native_failure: { + stage: "artifact", + remediation: expect.any(String), + }, windows_security: { job_object_process_ownership: expect.objectContaining({ available: false, @@ -406,7 +487,7 @@ describe("Ghidra platform support", () => { expect.objectContaining({ operation: "list_procedures", available: false, - availabilityCode: "unsupported_host", + availabilityCode: "runtime_missing", limitations: expect.arrayContaining([ expect.stringContaining( "matching packaged Windows x64 native addon", @@ -448,13 +529,132 @@ describe("Ghidra platform support", () => { }); }); +describe("Windows Ghidra native failure diagnostics", () => { + it.each([ + ["artifact", "runtime_missing"], + ["self-test", "runtime_unavailable"], + ["inspection", "runtime_unavailable"], + ] as const)( + "keeps Windows P0 unavailable after a %s failure", + async (failureStage, code) => { + const load = vi + .spyOn(nativeLoader, "systemWindowsNativeAuthority") + .mockReturnValue({ + available: false, + failureStage, + reason: "Windows native controls are unavailable: missing manifest", + }); + const ghidra = provider(installationHost("win32")); + const nativeApplication = peTarget("x86_64"); + + for (const architecture of ["x86", "x86_64"] as const) + expect( + ghidra.inspectTargetSupport(peTarget(architecture)), + ).toMatchObject({ + status: "supported", + diagnostics: { + host_platform: "win32", + architecture, + executable_role: "application", + managed: false, + }, + }); + expect(ghidra.inspectAvailability()).toMatchObject({ + status: "unavailable", + code, + reason: expect.stringContaining( + "Windows native controls are unavailable", + ), + diagnostics: { + windows_native_failure: { + stage: failureStage, + remediation: expect.any(String), + }, + windows_security: { + job_object_process_ownership: expect.objectContaining({ + available: false, + proof: "not-proven", + }), + private_runtime_dacl: expect.objectContaining({ + available: false, + proof: "not-proven", + }), + reparse_safe_path_admission: expect.objectContaining({ + available: false, + proof: "not-proven", + }), + }, + }, + }); + expect(load).toHaveBeenCalledTimes(1); + expect(ghidra.capabilities()).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + operation: "list_procedures", + available: false, + availabilityCode: code, + limitations: expect.arrayContaining([ + expect.stringContaining( + "matching packaged Windows x64 native addon", + ), + expect.stringContaining( + "private DACLs, and Job Object ownership automatically", + ), + ]), + }), + ]), + ); + expect( + ghidra.inspectTargetSupport({ + ...nativeApplication, + executableRole: "non-executable", + }), + ).toMatchObject({ + status: "unsupported", + code: "target_role_unsupported", + }); + expect( + ghidra.inspectTargetSupport({ ...nativeApplication, managed: true }), + ).toMatchObject({ + status: "unsupported", + code: "managed_target_unsupported", + }); + expect( + ghidra.inspectTargetSupport(executableTarget("elf", "x86_64")), + ).toMatchObject({ + status: "unsupported", + code: "target_format_unsupported", + }); + expect( + ghidra.inspectTargetSupport(executableTarget("pe", "arm64")), + ).toMatchObject({ + status: "unsupported", + code: "architecture_unsupported", + }); + const client = ghidra.createClient(nativeApplication); + const health = await client.execute("health", {}); + if (health.ok) + throw new Error( + "Failed native admission must not create a ready client", + ); + expect(projectAnalysisError(health.error)).toMatchObject({ + code: "capability_unavailable", + details: { + reason: "Windows native controls are unavailable: missing manifest", + }, + }); + expect(load).toHaveBeenCalledTimes(3); + }, + ); +}); + describe("Ghidra Windows PE fixture admission", () => { it("admits generated PE32 and PE32+ DLLs and applications while preserving native-only Windows admission", async () => { const root = fileURLToPath(new URL("../../", import.meta.url)); await promisify(execFile)(process.execPath, [ join(root, "scripts/create-ghidra-windows-fixture.mjs"), ]); - const ghidra = provider({ ...installationHost(), platform: "win32" }); + const ghidra = provider(installationHost("win32")); for (const architecture of ["x86", "x86_64"] as const) { for (const dll of [false, true]) { const path = join( @@ -627,7 +827,8 @@ describe("Ghidra client projection", () => { expect(resolved.value.profile.parameters).toMatchObject({ language_id: "auto-from-header", compiler_spec_id: "auto-default", - call_relationship_coverage: "resolved-call-reference-diagnostics-v1", + call_relationship_coverage: "function-body-call-coverage-v5", + function_dossier_coverage: "observed-facets-and-call-relationships-v1", }); const result = await ghidra @@ -823,36 +1024,70 @@ describe("Ghidra result projection", () => { }); describe("Ghidra PE CodeView debug symbol limitations", () => { - it("projects unapplied companion PDB limitation for PE target with CodeView record", async () => { - const ghidra = provider(installationHost(), () => ({ - start: () => Promise.resolve(ok(sessionInfo())), - callTool: () => Promise.resolve(ok(null)), - close: () => Promise.resolve(ok(null)), - })); - const target = { - ...peTarget("x86_64"), - codeView: { - pdbPath: "C:\\projects\\app\\build\\test.pdb", - pdbFileName: "test.pdb", - guid: "01234567-89ab-cdef-0123-456789abcdef", - age: 1, - }, - }; - const profile = await ghidra.resolveAnalysisProfile(target); - if (!profile.ok || profile.value.profile === null) - throw new Error("expected PE profile"); - const client = ghidra.createClient(target, profile.value.profile); - const health = await client.execute("health", {}); - expect(health).toMatchObject({ - ok: true, - value: { - limitations: expect.arrayContaining([ - 'The target references debug symbols "test.pdb" (GUID 01234567-89ab-cdef-0123-456789abcdef, age 1) that were not applied; procedure names are analysis-generated.', - ]), - }, - }); - await client.close(); - }); + it.each([ + ["absent report", undefined, "load state unknown"], + ["unknown load", null, "load state unknown"], + ["not loaded", false, "reports not loaded"], + ["loaded", true, "reports a loaded PDB"], + ] as const)( + "preserves %s in operation and health Evidence without duplicate caveats", + async (_, loaded, expected) => { + const ghidra = provider(installationHost(), () => ({ + start: () => + Promise.resolve( + ok({ + ...sessionInfo(), + ...(loaded === undefined + ? {} + : { + pdb: { + file: "test.pdb", + guid: "01234567-89ab-cdef-0123-456789abcdef", + age: "1", + signature: null, + version: "RSDS", + loaded, + }, + }), + }), + ), + callTool: () => Promise.resolve(ok([])), + close: () => Promise.resolve(ok(null)), + })); + const target = { + ...peTarget("x86_64"), + codeView: { + pdbPath: "test.pdb", + pdbFileName: "test.pdb", + guid: "01234567-89ab-cdef-0123-456789abcdef", + age: 1, + }, + }; + const profile = await ghidra.resolveAnalysisProfile(target); + if (!profile.ok || profile.value.profile === null) + throw new Error("expected PE profile"); + const client = ghidra.createClient(target, profile.value.profile); + try { + const search = await client.execute("search_procedures", { + pattern: "add_numbers", + }); + if (!search.ok) throw search.error; + const health = await client.execute("health", {}); + if (!health.ok) throw health.error; + for (const observation of [search.value, health.value]) { + const caveats = observation.limitations.filter((limit) => + limit.includes("PDB"), + ); + expect(caveats).toHaveLength(1); + expect(caveats[0]).toContain(expected); + expect(caveats[0]).not.toContain("names are analysis-generated"); + expect(caveats[0]).not.toContain("that were not applied"); + } + } finally { + await client.close(); + } + }, + ); }); describe("Ghidra relationship coverage", () => { @@ -863,7 +1098,16 @@ describe("Ghidra relationship coverage", () => { Promise.resolve( ok( operation === "procedure_callees" - ? { value: ["0x2000"], limitations: [] } + ? { + value: ["0x2000"], + limitations: [], + relationship_coverage: { + status: "complete", + diagnostics: [ + "Only resolved Ghidra call references are returned; unresolved computed or indirect calls remain unknown, while function classifications distinguish thunks and externals.", + ], + }, + } : { value: [], limitations: [] }, ), ), @@ -1817,6 +2061,8 @@ describe("compatible Ghidra builds", () => { readText: () => "application.version=12.1.2\napplication.java.min=21\napplication.java.max=\n", probeJava: () => ({ + osArch: "amd64", + osName: "Linux", version: "27", major: 27, home: "/usr/lib/jvm/java-27-openjdk", @@ -2219,3 +2465,104 @@ describe("Ghidra disabled analyzer options", () => { } }); }); + +describe("Ghidra queries after the owned provider is lost", () => { + const lostProviderSession = async ( + state: "exited" | "unreachable" | "idle", + failureKind: "process" | "protocol" | "remote" = "process", + answeredBeforeLoss = true, + ) => { + let calls = 0; + const ghidra = provider(installationHost(), () => ({ + start: async () => + err(new GhidraSessionError("start", "startup is not exercised")), + callTool: async () => { + calls += 1; + if (answeredBeforeLoss && calls === 1) return ok(null); + return err( + new GhidraSessionError( + failureKind, + "Ghidra bridge socket closed", + { failure_kind: failureKind }, + failureKind === "remote" ? { remoteCode: "internal" } : {}, + ), + ); + }, + close: async () => ok(null), + operationHealthSnapshot: () => + state === "idle" + ? undefined + : { + state, + stage: null, + retryAction: "restart_provider", + exitCode: state === "exited" ? 137 : null, + requests: [], + }, + })); + const target = executableTarget("elf", "x86_64"); + const profile = await ghidra.resolveAnalysisProfile(target); + if (!profile.ok || profile.value.profile === null) + throw new Error("Expected Ghidra analysis profile"); + const client = ghidra.createClient(target, profile.value.profile); + if (answeredBeforeLoss) await client.execute("list_documents", {}); + return client; + }; + + it("tells the caller to reopen the target when Ghidra exited", async () => { + const client = await lostProviderSession("exited"); + const result = await client.execute("list_documents", {}); + expect(result.ok).toBe(false); + if (result.ok) return; + const projected = projectAnalysisError(result.error); + expect(projected).toMatchObject({ + code: "execution_failure", + retryable: false, + details: { provider_id: "ghidra", operation: "list_documents" }, + }); + expect(projected.message).toContain("exited with code 137"); + expect(projected.message).toContain("Call open_binary"); + expect(projected.remediation.action).toContain("Call open_binary"); + expect(projected.remediation.action).not.toContain("rea doctor"); + }); + + it("tells the caller to reopen the target when the bridge became unreachable", async () => { + const client = await lostProviderSession("unreachable", "protocol"); + const result = await client.execute("list_procedures", {}); + expect(result.ok).toBe(false); + if (result.ok) return; + const projected = projectAnalysisError(result.error); + expect(projected.message).toContain("became unreachable"); + expect(projected.remediation.action).toContain("Call open_binary"); + }); + + it("keeps installation guidance when Ghidra dies before it ever answered", async () => { + const client = await lostProviderSession("exited", "process", false); + const result = await client.execute("list_documents", {}); + expect(result.ok).toBe(false); + if (result.ok) return; + const projected = projectAnalysisError(result.error); + expect(projected.message).not.toContain("Call open_binary"); + expect(projected.remediation.action).toContain("rea doctor"); + }); + + it("keeps the generic failure when the provider is not known to be lost", async () => { + const client = await lostProviderSession("idle"); + const result = await client.execute("list_documents", {}); + expect(result.ok).toBe(false); + if (result.ok) return; + expect(projectAnalysisError(result.error).remediation.action).toContain( + "rea doctor", + ); + }); + + it("does not reinterpret a bridge-reported failure as a lost provider", async () => { + const client = await lostProviderSession("exited", "remote"); + const result = await client.execute("list_documents", {}); + expect(result.ok).toBe(false); + if (result.ok) return; + expect(projectAnalysisError(result.error).message).not.toContain( + "Call open_binary", + ); + }); +}); diff --git a/src/ghidra/GhidraProvider.ts b/src/ghidra/GhidraProvider.ts index 0a1cde0f07..3cee8bacec 100644 --- a/src/ghidra/GhidraProvider.ts +++ b/src/ghidra/GhidraProvider.ts @@ -4,6 +4,7 @@ import type { AnalysisClientContext, AnalysisProfileResolutionOptions, AnalysisProviderCandidate, + AnalysisProviderReadinessSnapshot, CapabilityDescriptor, ProviderAvailability, ProviderIdentity, @@ -25,6 +26,7 @@ import { } from "../domain/jsonValue.js"; import { err, ok } from "../domain/result.js"; import { ProviderAdapterError } from "../domain/providerAdapterError.js"; +import { ProviderCleanupError } from "../domain/providerCleanupError.js"; import type { Logger } from "pino"; import { GhidraClient } from "./GhidraClient.js"; import { @@ -38,6 +40,7 @@ import { AnalysisCancelledError } from "../domain/analysisErrorCore.js"; import { resolveGhidraAnalysisSeeds } from "./GhidraAnalysisSeeds.js"; import { admitGhidraLanguageOverride, + GhidraLanguageCatalogCleanupError, readGhidraLanguageCatalog, } from "./GhidraLanguageCatalog.js"; import { @@ -55,8 +58,8 @@ import { type GhidraProviderClientFactory, } from "./GhidraProviderClient.js"; import { - windowsNativeAuthorityUnavailableReason, - hasWindowsNativeAuthority, + inspectWindowsNativeReadiness, + type WindowsNativeReadiness, windowsNativeCapabilities, } from "../process/WindowsAuthority.js"; @@ -68,6 +71,7 @@ const SUPPORTED_ARCHITECTURES = new Set([ "mips", "6502", "riscv", + "powerpc", ]); /** Ghidra candidate backed by an isolated ephemeral headless import. */ @@ -100,22 +104,43 @@ export class GhidraProvider implements AnalysisProviderCandidate { } inspectAvailability(): ProviderAvailability { + return this.inspectReadinessSnapshot().availability; + } + + inspectReadinessSnapshot(): AnalysisProviderReadinessSnapshot { const installation = this.#inspectInstallation(); + const readiness = + installation.platform === "win32" + ? inspectWindowsNativeReadiness(installation.platform) + : undefined; + return { + availability: this.#availability(installation, readiness), + capabilities: + readiness === undefined + ? CAPABILITIES + : windowsP0Capabilities(readiness), + }; + } + + #availability( + installation: GhidraInstallationInspection, + readiness: WindowsNativeReadiness | undefined, + ): ProviderAvailability { const diagnostics = ghidraInstallationDiagnostics(installation); - if ( - installation.status === "available" && - installation.platform === "win32" && - !hasWindowsNativeAuthority(installation.platform) - ) + if (installation.status === "available" && readiness?.available === false) return { status: "unavailable", - code: "unsupported_host", - reason: windowsNativeAuthorityUnavailableReason(installation.platform), + code: readiness.code, + reason: readiness.reason, diagnostics: { ...diagnostics, windows_security: jsonObjectSchema.parse( - windowsNativeCapabilities(installation.platform), + windowsNativeCapabilities(installation.platform, readiness), ), + windows_native_failure: { + stage: readiness.failureStage, + remediation: readiness.remediation, + }, }, }; return installation.status === "available" @@ -216,6 +241,22 @@ export class GhidraProvider implements AnalysisProviderCandidate { try { catalog = await readGhidraLanguageCatalog(installation.installDir); } catch (cause: unknown) { + if (cause instanceof GhidraLanguageCatalogCleanupError) + return err( + new ProviderCleanupError( + "ghidra", + cause.paths, + { + reason: cause.message, + paths: [...cause.paths], + close_failures: cause.causes.map(providerFailureText), + ...(cause.readFailure === undefined + ? {} + : { read_failure: providerFailureText(cause.readFailure) }), + }, + { operation: "resolve_analysis_profile", cause }, + ), + ); return err( new ProviderAdapterError("ghidra", "resolve_analysis_profile", { cause, @@ -301,6 +342,9 @@ export class GhidraProvider implements AnalysisProviderCandidate { } } +const providerFailureText = (cause: unknown): string => + cause instanceof Error ? cause.message : String(cause); + const inspectWindowsP0TargetSupport = ( target: BinaryTarget, diagnostics: Readonly>, diff --git a/src/ghidra/GhidraProviderCapabilities.ts b/src/ghidra/GhidraProviderCapabilities.ts index 0714d1ce04..6272802d30 100644 --- a/src/ghidra/GhidraProviderCapabilities.ts +++ b/src/ghidra/GhidraProviderCapabilities.ts @@ -2,10 +2,9 @@ import type { CapabilityDescriptor, ProviderIdentity, } from "../application/AnalysisProvider.js"; -import type { RelationshipCoverage } from "../domain/relationshipCoverage.js"; import { - hasWindowsNativeAuthority, - windowsNativeAuthorityUnavailableReason, + inspectWindowsNativeReadiness, + type WindowsNativeReadiness, } from "../process/WindowsAuthority.js"; import { GHIDRA_FUNCTION_OPERATIONS } from "./GhidraFunctionValues.js"; import { GHIDRA_INVENTORY_OPERATIONS } from "./GhidraInventoryValues.js"; @@ -44,20 +43,6 @@ const stringInventoryLimitation = const searchLimitation = "Regex mode uses Java Pattern semantics. Matcher or compiler stack exhaustion returns a resource constraint without closing the session; literal mode avoids regex recursion and returns complete matching values."; -const resolvedCallReferenceLimitation = - "Only resolved Ghidra call references are returned; unresolved computed or indirect calls remain unknown, while function classifications distinguish thunks and externals."; - -/** Relationship-only caveats belong to each observed caller/callee result. */ -export const relationshipCoverageFor = ( - operation: string, -): RelationshipCoverage | undefined => - operation === "procedure_callers" || operation === "procedure_callees" - ? { - status: "complete", - diagnostics: [resolvedCallReferenceLimitation], - } - : undefined; - /** Limitation text for one admitted operation, including the common base. */ export const limitationsFor = (operation: string): readonly string[] => { const common = [ @@ -212,8 +197,9 @@ export const CAPABILITIES: readonly CapabilityDescriptor[] = Object.freeze( ); /** Capabilities advertised for the experimental Windows x64 P0 boundary. */ -export const windowsP0Capabilities = (): readonly CapabilityDescriptor[] => { - const available = hasWindowsNativeAuthority("win32"); +export const windowsP0Capabilities = ( + readiness: WindowsNativeReadiness = inspectWindowsNativeReadiness("win32"), +): readonly CapabilityDescriptor[] => { return Object.freeze( CAPABILITIES.map((capability): CapabilityDescriptor => { const limitations = Object.freeze([ @@ -228,13 +214,13 @@ export const windowsP0Capabilities = (): readonly CapabilityDescriptor[] => { availabilityCode: "unsupported_host", limitations, }); - return available + return readiness.available ? Object.freeze({ ...capability, limitations }) : Object.freeze({ ...capability, available: false, - reason: windowsNativeAuthorityUnavailableReason("win32"), - availabilityCode: "unsupported_host", + reason: readiness.reason, + availabilityCode: readiness.code, limitations, }); }), diff --git a/src/ghidra/GhidraProviderClient.ts b/src/ghidra/GhidraProviderClient.ts index 9db1b0c5f6..da81fca433 100644 --- a/src/ghidra/GhidraProviderClient.ts +++ b/src/ghidra/GhidraProviderClient.ts @@ -34,6 +34,11 @@ import { } from "../domain/analysisErrorCore.js"; import type { AnalysisError } from "../domain/analysisErrorBase.js"; import type { EvidenceLocation } from "../domain/evidence.js"; +import type { RelationshipCoverage } from "../domain/relationshipCoverage.js"; +import { + providerRetryAction, + type ProviderOperationHealth, +} from "../domain/providerOperationHealth.js"; import { jsonValueSchema, type JsonValue } from "../domain/jsonValue.js"; import { ProviderAdapterError } from "../domain/providerAdapterError.js"; import { err, ok, type Result } from "../domain/result.js"; @@ -68,7 +73,7 @@ import { isGhidraMipsSoftFloatTarget, } from "./GhidraMipsProfile.js"; import { ghidraRiscvLimitations } from "./GhidraRiscvProfile.js"; -import { ghidraPeCodeViewLimitations } from "./GhidraPeProfile.js"; +import { ghidraPePdbLimitations } from "./GhidraPeProfile.js"; import { unverifiedGhidraBuildLimitation } from "./GhidraInstallationPolicy.js"; import { GhidraHeadlessLauncher } from "./GhidraLauncher.js"; import { attestGhidraNativeLoadImage } from "./GhidraLoadImageAttest.js"; @@ -86,23 +91,16 @@ import { healthLimitations, windowsP0Limitations, limitationsFor, - relationshipCoverageFor, } from "./GhidraProviderCapabilities.js"; import type { GhidraSessionError } from "./GhidraSessionError.js"; -import { - ghidraPdbLimitations, - type GhidraSessionInfo, -} from "./GhidraSessionValues.js"; +import type { GhidraSessionInfo } from "./GhidraSessionValues.js"; import { ghidraExtensionFailure } from "./extensions/GhidraExtensionFailures.js"; import { ghidraExtensionSchema, validateGhidraExtensionProfile, ghidraExtensionLimitations, } from "./extensions/GhidraExtensions.js"; -import { - windowsNativeAuthorityUnavailableReason, - hasWindowsNativeAuthority, -} from "../process/WindowsAuthority.js"; +import { inspectWindowsNativeReadiness } from "../process/WindowsAuthority.js"; import { ghidraRawBinaryLanguage } from "./GhidraRawBinaryLanguage.js"; /** Production seam for exercising provider projection without a real process. */ @@ -531,7 +529,6 @@ export const createGhidraProviderClient = (input: { // Replaced by the reported outcome once the session handshake is checked. const seedLimitations: string[] = [...ghidraSeedLimitations(seeds)]; const pdbLimitations: string[] = []; - let pdbLoaded = false; // Replaced by the reported storage outcome once the handshake is checked. const softFloatLimitations: string[] = [ ...ghidraMipsSoftFloatLimitations(softFloat), @@ -652,16 +649,20 @@ export const createGhidraProviderClient = (input: { ...(context === undefined ? {} : { runId: context.runId }), logger: logger.child({ layer: "ghidra-bridge" }), }); + // Startup failures retain installation guidance; only a previously running + // provider can be described as lost and reopened. + let providerStarted = false; + const lostProviderHealth = (): ProviderOperationHealth | undefined => + providerStarted ? client.operationHealthSnapshot?.() : undefined; const checkSessionObservations = async ( operation: AnalysisOperation, info: GhidraSessionInfo, ): Promise => { if (target.format === "pe") { - pdbLoaded = info.pdb?.loaded === true; pdbLimitations.splice( 0, pdbLimitations.length, - ...ghidraPdbLimitations(info.pdb), + ...ghidraPePdbLimitations(target, info.pdb), ); } const softFloatReason = ghidraMipsSoftFloatFailure( @@ -716,7 +717,6 @@ export const createGhidraProviderClient = (input: { ...ghidraExtensionLimitations(extensions), ...seedLimitations, ...pdbLimitations, - ...ghidraPeCodeViewLimitations(target, pdbLoaded), ...(releaseLimitation === undefined ? [] : [releaseLimitation]), ]; const nativeAotOwner = @@ -733,8 +733,8 @@ export const createGhidraProviderClient = (input: { rawResult: JsonValue, observationLimitations: readonly string[] = [], locations: readonly EvidenceLocation[] = [], + relationshipCoverage?: RelationshipCoverage, ) => { - const relationshipCoverage = relationshipCoverageFor(operation); return createAnalysisExecution(normalized, committedProfile.provider, { rawResult, locations, @@ -757,6 +757,7 @@ export const createGhidraProviderClient = (input: { operation, called.error, config.ghidraStartupTimeoutMs, + lostProviderHealth(), ), ); const parsed = parseGhidraInventoryResult(operation, called.value); @@ -780,12 +781,14 @@ export const createGhidraProviderClient = (input: { ); if (operation === "health") { const started = await client.start(options?.signal, options?.progress); + if (started.ok) providerStarted = true; if (!started.ok) return err( projectSessionError( operation, started.error, config.ghidraStartupTimeoutMs, + lostProviderHealth(), ), ); const failed = await checkSessionObservations(operation, started.value); @@ -808,12 +811,14 @@ export const createGhidraProviderClient = (input: { softFloat ) { const started = await client.start(options?.signal, options?.progress); + if (started.ok) providerStarted = true; if (!started.ok) return err( projectSessionError( operation, started.error, config.ghidraStartupTimeoutMs, + lostProviderHealth(), ), ); const failed = await checkSessionObservations(operation, started.value); @@ -825,6 +830,7 @@ export const createGhidraProviderClient = (input: { ? {} : { progress: options.progress }), }); + if (called.ok) providerStarted = true; if (operation === "list_documents" || operation === "list_procedures") return inventoryExecution(operation, called); if (!called.ok) @@ -833,17 +839,20 @@ export const createGhidraProviderClient = (input: { operation, called.error, config.ghidraStartupTimeoutMs, + lostProviderHealth(), ), ); let normalized: JsonValue; let observationLimitations: readonly string[] = []; let observationLocations: readonly EvidenceLocation[] = []; + let relationshipCoverage: RelationshipCoverage | undefined; if (isGhidraFunctionOperation(operation)) { const result = parseGhidraFunctionResult(operation, called.value); if (!result.ok) return result; normalized = result.value.value; observationLimitations = result.value.limitations; observationLocations = result.value.locations ?? []; + relationshipCoverage = result.value.relationshipCoverage; } else { const result = parseGhidraInventoryResult(operation, called.value); if (!result.ok) return result; @@ -863,6 +872,7 @@ export const createGhidraProviderClient = (input: { operation, started.error, config.ghidraStartupTimeoutMs, + lostProviderHealth(), ), ); const derived = await nativeAotOwner.derive( @@ -925,6 +935,7 @@ export const createGhidraProviderClient = (input: { operation, failure, config.ghidraStartupTimeoutMs, + lostProviderHealth(), ), nativeAotResult: derived, signal: options?.signal, @@ -939,6 +950,7 @@ export const createGhidraProviderClient = (input: { called.value, observationLimitations, observationLocations, + relationshipCoverage, ), ); }, @@ -997,15 +1009,16 @@ const ghidraClientPrerequisites = ( diagnostics: ghidraInstallationDiagnostics(installation), }), ); - if ( - installation.platform === "win32" && - !hasWindowsNativeAuthority(installation.platform) - ) + const readiness = + installation.platform === "win32" + ? inspectWindowsNativeReadiness(installation.platform) + : undefined; + if (readiness?.available === false) return err( new AnalysisCapabilityUnavailableError( "ghidra", "health", - windowsNativeAuthorityUnavailableReason(installation.platform), + readiness.reason, ), ); if (profile === undefined || profile.provider.id !== "ghidra") @@ -1027,11 +1040,13 @@ const projectSessionError = ( operation: AnalysisOperation, failure: GhidraSessionError, startupTimeoutMs: number, + health?: ProviderOperationHealth, ): AnalysisError => { const primary = projectPrimarySessionError( operation, failure, startupTimeoutMs, + health, ); return failure.cleanupFailure === undefined ? primary @@ -1046,6 +1061,7 @@ const projectPrimarySessionError = ( operation: AnalysisOperation, failure: GhidraSessionError, startupTimeoutMs: number, + health?: ProviderOperationHealth, ): AnalysisError => { const context = { cause: failure, diagnostics: failure.diagnostics }; if (failure.cause instanceof AnalysisAccessDeniedError) @@ -1112,5 +1128,31 @@ const projectPrimarySessionError = ( failure.message, context, ); + const exited = exitedProviderMessage(operation, failure, health); + if (exited !== undefined) + return new ProviderAdapterError("ghidra", operation, { + ...context, + userMessage: exited, + }); return new ProviderAdapterError("ghidra", operation, context); }; + +/** A provider already observed as running can be restarted by reopening. */ +const exitedProviderMessage = ( + operation: AnalysisOperation, + failure: GhidraSessionError, + health: ProviderOperationHealth | undefined, +): string | undefined => { + if (failure.kind !== "process" && failure.kind !== "protocol") + return undefined; + if ( + health === undefined || + providerRetryAction(health.state) !== "restart_provider" + ) + return undefined; + const observed = + health.state === "exited" + ? `exited${health.exitCode === null ? "" : ` with code ${String(health.exitCode)}`}` + : "became unreachable"; + return `The Ghidra provider for this session ${observed} before ${operation} could complete. Call open_binary for the same target to start a new Ghidra session, then retry; session-only annotations from the previous provider are not retained.`; +}; diff --git a/src/ghidra/GhidraPspProfile.ts b/src/ghidra/GhidraPspProfile.ts index 17260d038e..e119ec9a56 100644 --- a/src/ghidra/GhidraPspProfile.ts +++ b/src/ghidra/GhidraPspProfile.ts @@ -1,5 +1,10 @@ import type { BinaryTarget } from "../domain/binaryTargetTypes.js"; -import { ghidraMipsUnsupportedReason } from "./GhidraMipsProfile.js"; +import { + ghidraMipsGnuAttributesUnsupportedReason, + ghidraMipsIncompleteDeclarationsReason, + ghidraMipsUnsupportedReason, +} from "./GhidraMipsProfile.js"; +import { ghidraPowerPcUnsupportedReason } from "./GhidraPowerPcProfile.js"; import { ghidraRiscvUnsupportedReason } from "./GhidraRiscvProfile.js"; /** Allegrex's ELF machine declaration, not a filename or inferred instruction. */ @@ -14,6 +19,8 @@ export const isGhidraPspTarget = (target: BinaryTarget): boolean => export const ghidraProcessorUnsupportedReason = ( target: BinaryTarget, ): string | null => { + if (target.kind === "executable" && target.architecture === "powerpc") + return ghidraPowerPcUnsupportedReason(target); if (target.kind === "executable" && target.architecture === "riscv") return ghidraRiscvUnsupportedReason(target); if ( @@ -24,6 +31,11 @@ export const ghidraProcessorUnsupportedReason = ( return ghidraMipsUnsupportedReason(target); const metadata = target.mips; if (metadata === undefined) return "PSP analysis requires ELF metadata."; + const gnuAttributesReason = + ghidraMipsGnuAttributesUnsupportedReason(metadata); + if (gnuAttributesReason !== null) return gnuAttributesReason; + const incomplete = ghidraMipsIncompleteDeclarationsReason(metadata); + if (incomplete !== null) return incomplete; if (metadata.elfClass !== 32 || metadata.byteOrder !== "little") return "The PSP Allegrex profile requires little-endian ELF32."; if (metadata.type !== 2) diff --git a/src/ghidra/GhidraRiscvProfile.test.ts b/src/ghidra/GhidraRiscvProfile.test.ts index 718fe8d46d..3ada4d15b9 100644 --- a/src/ghidra/GhidraRiscvProfile.test.ts +++ b/src/ghidra/GhidraRiscvProfile.test.ts @@ -13,7 +13,7 @@ import { hopperLoaderArgsForTarget } from "../hopper/HopperAnalysisProfile.js"; import { IdaProvider } from "../ida/IdaProvider.js"; import { GhidraProvider } from "./GhidraProvider.js"; import { ghidraRiscvLimitations } from "./GhidraRiscvProfile.js"; -import type { GhidraInstallationHost } from "./GhidraInstallation.js"; +import { ghidraProfileInstallationHost as host } from "./GhidraInstallation.fixture.js"; import { ghidraPinnedLanguage, ghidraProfileLanguageOverride, @@ -49,20 +49,6 @@ const RISCV_LDEFS = ` `; -// Fake installation inspection tests admission/profile contracts, not Ghidra. -const host: GhidraInstallationHost = { - platform: "linux", - architecture: "x64", - readText: () => "application.version=12.1.4\napplication.java.min=21\n", - executable: () => true, - probeJava: () => ({ - version: "21.0.11", - major: 21, - home: "/jdk-21", - bits: 64, - runtime: "jdk", - }), -}; const configOf = ( installDir: string, environment: Record = {}, diff --git a/src/ghidra/GhidraSessionError.ts b/src/ghidra/GhidraSessionError.ts index ca3b4f6aca..9f4abfb52d 100644 --- a/src/ghidra/GhidraSessionError.ts +++ b/src/ghidra/GhidraSessionError.ts @@ -2,6 +2,7 @@ import type { AnalysisError } from "../domain/analysisErrorBase.js"; import type { JsonValue } from "../domain/jsonValue.js"; import { DarwinProcessOwnershipInspectionError } from "../process/DarwinProcessRunTokenReader.js"; import { boundedUtf8Text } from "../process/boundedUtf8Text.js"; +import { GhidraLaunchSupportError } from "./GhidraJavaLaunch.js"; import { GhidraLaunchError, GHIDRA_LAUNCH_COMMAND_OUTPUT_BYTES, @@ -11,6 +12,7 @@ const MAX_CAUSE_DEPTH = 3; const MAX_CAUSE_MESSAGE_BYTES = 2 * 1024; const MAX_CAUSE_FIELD_BYTES = 1024; const MAX_COMPILER_STDERR_BYTES = 8 * 1024; +const MAX_COMMAND_OUTPUT_BYTES = 8 * 1024; /** Stable failure phase for one private Ghidra headless session. */ export type GhidraSessionFailureKind = @@ -59,7 +61,7 @@ const createGhidraSessionError = (failure: { readonly cause?: unknown; readonly timeoutMs?: number; readonly remoteCode?: string; - readonly redact: (value: string) => string; + readonly redact: (value: string, incomplete?: boolean) => string; }): GhidraSessionError => new GhidraSessionError( failure.kind, @@ -90,7 +92,7 @@ const createGhidraSessionError = (failure: { export const bindGhidraSessionFailure = ( diagnostics: () => Readonly>, - redact: (value: string) => string = (value) => value, + redact: (value: string, incomplete?: boolean) => string = (value) => value, ) => ( kind: GhidraSessionFailureKind, @@ -114,7 +116,7 @@ export const bindGhidraSessionFailure = const describeCause = ( cause: unknown, - redact: (value: string) => string, + redact: (value: string, incomplete?: boolean) => string, depth = 0, ): JsonValue => { if (cause instanceof Error) { @@ -137,7 +139,7 @@ const describeCause = ( const describeErrorCause = ( cause: Error, - redact: (value: string) => string, + redact: (value: string, incomplete?: boolean) => string, depth: number, ): JsonValue => { const message = boundedUtf8Text( @@ -152,15 +154,45 @@ const describeErrorCause = ( ...(message.truncated ? { message_truncated: true } : {}), ...causeCode(cause, redact), ...causeSignal(cause, redact), + ...causeCommandOutcome(cause, redact), ...causeCompilerOutcome(cause, redact), ...causeLaunchOutcome(cause, redact), ...nestedErrorCause(cause, redact, depth), }; }; +const causeCommandOutcome = ( + cause: Error, + redact: (value: string, incomplete?: boolean) => string, +): Record => { + if (!(cause instanceof GhidraLaunchSupportError)) return {}; + const outcome = cause.outcome; + if (outcome === undefined) return {}; + const stdout = boundedUtf8Text( + redact(outcome.stdout, outcome.outputTruncated !== false), + MAX_COMMAND_OUTPUT_BYTES, + ); + const stderr = boundedUtf8Text( + redact(outcome.stderr, outcome.outputTruncated !== false), + MAX_COMMAND_OUTPUT_BYTES, + ); + return { + command_outcome: { + stdout: stdout.text, + stderr: stderr.text, + stdoutBytes: outcome.stdoutBytes ?? Buffer.byteLength(outcome.stdout), + stderrBytes: outcome.stderrBytes ?? Buffer.byteLength(outcome.stderr), + stdoutTruncated: stdout.truncated, + stderrTruncated: stderr.truncated, + outputTruncated: outcome.outputTruncated, + killed: outcome.killed, + }, + }; +}; + const causeCode = ( cause: Error, - redact: (value: string) => string, + redact: (value: string, incomplete?: boolean) => string, ): Record => { const code = "code" in cause ? cause.code : undefined; if (typeof code === "string") { @@ -176,7 +208,7 @@ const causeCode = ( const causeSignal = ( cause: Error, - redact: (value: string) => string, + redact: (value: string, incomplete?: boolean) => string, ): Record => { const signal = "signal" in cause ? cause.signal : undefined; if (typeof signal === "string") { @@ -192,7 +224,7 @@ const causeSignal = ( const causeCompilerOutcome = ( cause: Error, - redact: (value: string) => string, + redact: (value: string, incomplete?: boolean) => string, ): Record => { if (!(cause instanceof DarwinProcessOwnershipInspectionError)) return {}; const outcome = cause.compilerOutcome; @@ -218,7 +250,7 @@ const causeCompilerOutcome = ( const boundedCompilerField = ( value: string | number | null, - redact: (value: string) => string, + redact: (value: string, incomplete?: boolean) => string, ): { readonly value: string | number | null; readonly truncated: boolean } => { if (typeof value !== "string") return { value, truncated: false }; const bounded = boundedUtf8Text(redact(value), MAX_CAUSE_FIELD_BYTES); @@ -228,7 +260,7 @@ const boundedCompilerField = ( /** Retain a launch failure's launcher command outcome in its cause envelope. */ const causeLaunchOutcome = ( cause: Error, - redact: (value: string) => string, + redact: (value: string, incomplete?: boolean) => string, ): Record => { if (!(cause instanceof GhidraLaunchError)) return {}; const launchSupport = @@ -240,6 +272,9 @@ const causeLaunchOutcome = ( stderr: redact(output.stderr), })), }; + // A failed LaunchSupport helper reports its own command outcome in the + // nested cause; earlier helper warnings still belong to this launch. + if (cause.cause instanceof GhidraLaunchSupportError) return launchSupport; const output = cause.capturedOutput; if (output === undefined) return launchSupport; const stdout = boundedUtf8Text( @@ -273,7 +308,7 @@ const causeLaunchOutcome = ( const nestedErrorCause = ( cause: Error, - redact: (value: string) => string, + redact: (value: string, incomplete?: boolean) => string, depth: number, ): Record => { if (!("cause" in cause)) return {}; diff --git a/src/ghidra/GhidraSessionValues.ts b/src/ghidra/GhidraSessionValues.ts index e370892f3e..5d7bda0d27 100644 --- a/src/ghidra/GhidraSessionValues.ts +++ b/src/ghidra/GhidraSessionValues.ts @@ -121,21 +121,26 @@ export const ghidraPdbLimitations = ( limitations.push( `PDB attributes ${pdb.resource_limit.omitted_attributes.join(", ")} were omitted because the handshake's ${pdb.resource_limit.utf16_budget}-UTF-16-unit PDB attribute budget was exhausted. Omitted values remain unknown; they were not truncated or reported as absent.`, ); + const hasReference = [ + pdb.file, + pdb.guid, + pdb.age, + pdb.signature, + pdb.version, + ].some((value) => value !== null); + const reference = hasReference + ? `The Ghidra Program references PDB ${pdb.file ?? "(file unknown)"} (GUID ${pdb.guid ?? "unknown"}, age (hex) ${pdb.age ?? "unknown"}, signature ${pdb.signature ?? "unknown"}, version ${pdb.version ?? "unknown"}); ` + : ""; if (pdb.loaded === true) return [ ...limitations, - "Ghidra reports a loaded PDB; this is not an independent PDB identity or complete symbol/type-coverage attestation. Original names and types may still be missing; an empty name search does not prove a procedure is absent. Use list_procedures and address-based analysis to inspect recovered functions.", + `${reference}Ghidra reports a loaded PDB; this is not an independent PDB identity or complete symbol/type-coverage attestation. Original names and types may still be missing; an empty name search does not prove a procedure is absent. Use list_procedures and address-based analysis to inspect recovered functions.`, ]; - if ( - [pdb.file, pdb.guid, pdb.age, pdb.signature, pdb.version].every( - (value) => value === null, - ) - ) - return limitations; + if (!hasReference) return limitations; const state = pdb.loaded === false ? "not loaded" : "load state unknown"; return [ ...limitations, - `The target references PDB ${pdb.file ?? "(file unknown)"} (GUID ${pdb.guid ?? "unknown"}, age (hex) ${pdb.age ?? "unknown"}, signature ${pdb.signature ?? "unknown"}, version ${pdb.version ?? "unknown"}); Ghidra reports ${state}. Original names and types may be missing; an empty name search does not prove a procedure is absent. Use list_procedures and address-based analysis to inspect recovered functions. REA does not admit a caller-selected PDB companion.`, + `${reference}Ghidra reports ${state}. Original names and types may be missing; an empty name search does not prove a procedure is absent. Use list_procedures and address-based analysis to inspect recovered functions. REA does not admit a caller-selected PDB companion.`, ]; }; diff --git a/src/ghidra/GhidraSnapshotCache.test.ts b/src/ghidra/GhidraSnapshotCache.test.ts index 9cf5c24ad4..5ed8c319f9 100644 --- a/src/ghidra/GhidraSnapshotCache.test.ts +++ b/src/ghidra/GhidraSnapshotCache.test.ts @@ -11,6 +11,7 @@ import { silentLogger } from "../logger.js"; import { GhidraProvider } from "./GhidraProvider.js"; import { GhidraSessionError } from "./GhidraSessionError.js"; import { GHIDRA_SESSION_CAPABILITIES } from "./GhidraSessionValues.js"; +import type { RelationshipCoverage } from "../domain/relationshipCoverage.js"; const TARGET: BinaryTarget = { path: "/tmp/ghidra-cache-fixture", @@ -23,7 +24,7 @@ const TARGET: BinaryTarget = { // Exercise the real provider descriptors, profile, input/output projection and // session cache; only the installed Ghidra process is replaced at its seam. -const fixture = (version = "12.1.4") => { +const fixture = (version = "12.1.4", coverage?: RelationshipCoverage) => { const config = parseConfig({ GHIDRA_INSTALL_DIR: "/opt/ghidra" }); if (!config.ok) throw config.error; const calls: string[] = []; @@ -43,6 +44,8 @@ const fixture = (version = "12.1.4") => { `application.version=${version}\napplication.java.min=21\n`, executable: () => true, probeJava: () => ({ + osArch: "amd64", + osName: "Linux", version: "21.0.11", major: 21, home: "/usr/lib/jvm/jdk-21", @@ -111,7 +114,13 @@ const fixture = (version = "12.1.4") => { return ok( operation === "procedure_pseudo_code" || operation === "analyze_function" - ? { value, limitations: [] } + ? { + value, + limitations: [], + ...(operation === "analyze_function" && coverage !== undefined + ? { relationship_coverage: coverage } + : {}), + } : value, ); }, @@ -131,42 +140,129 @@ const open = async (session: ReturnType["session"]) => { expect((await session.openResolvedTarget(preview.value)).ok).toBe(true); }; -describe("Ghidra snapshot cache", () => { - it("refuses snapshots whose profile predates typed incoming references", async () => { - const initial = fixture(); - await open(initial.session); - const snapshot = initial.session.exportAnalysisSnapshot(); - if (!snapshot.ok) throw snapshot.error; - await initial.session.close(); - const profile = snapshot.value.binding.analysis_profile; - expect(profile.parameters.instruction_references).toBe( - "exact-address-incoming-v1", - ); - const legacy = { - ...snapshot.value, - binding: { - ...snapshot.value.binding, - analysis_profile: createAnalysisProfile( - profile.provider, - Object.fromEntries( - Object.entries(profile.parameters).filter( - ([key]) => key !== "instruction_references", - ), - ), - ), - }, - }; - const replay = fixture(); - const imported = replay.session.importAnalysisSnapshot(legacy); - if (!imported.ok) throw imported.error; - expect(await replay.session.previewTarget(TARGET)).toMatchObject({ - ok: false, - error: { _tag: "EvidenceIntegrityError" }, - }); - expect(replay.calls).toEqual([]); - await replay.session.close(); - }); +describe("Ghidra relationship coverage replay", () => { + it.each([ + undefined, + { status: "complete", diagnostics: ["Resolved static references only."] }, + { status: "partial", limitations: ["Instruction coverage was truncated."] }, + { status: "unknown", reason: "Relationship inventory was unavailable." }, + { + status: "partial", + limitations: ["Missing callers."], + limited_directions: ["callers"], + }, + ] satisfies (RelationshipCoverage | undefined)[])( + "preserves relationship coverage %j through live execution, cache and snapshot replay", + async (coverage) => { + const initial = fixture("12.1.4", coverage); + await open(initial.session); + const parameters = { procedure: "0x1000" }; + for (let i = 0; i < 2; i++) { + const execution = await initial.session.execute( + "analyze_function", + parameters, + ); + if (!execution.ok) throw execution.error; + expect(execution.value.relationshipCoverage).toEqual(coverage); + } + expect(initial.calls).toEqual(["analyze_function"]); + const snapshot = initial.session.exportAnalysisSnapshot(); + if (!snapshot.ok) throw snapshot.error; + await initial.session.close(); + const replay = fixture(); + expect(replay.session.importAnalysisSnapshot(snapshot.value).ok).toBe( + true, + ); + await open(replay.session); + const execution = await replay.session.execute( + "analyze_function", + parameters, + ); + if (!execution.ok) throw execution.error; + expect(execution.value.relationshipCoverage ?? undefined).toEqual( + coverage, + ); + expect(replay.calls).toEqual([]); + await replay.session.close(); + }, + ); +}); +describe("Ghidra snapshot profile admission", () => { + it.each([ + { + parameter: "function_boundary_observations", + current: "ghidra-function-boundary-limitations-v4", + legacy: "ghidra-function-boundary-limitations-v3", + }, + { + parameter: "instruction_references", + current: "exact-address-incoming-without-synthetic-entry-v2", + legacy: "exact-address-incoming-v1", + }, + { + parameter: "instruction_references", + current: "exact-address-incoming-without-synthetic-entry-v2", + legacy: undefined, + }, + { + parameter: "call_relationship_coverage", + current: "function-body-call-coverage-v5", + legacy: "resolved-call-reference-diagnostics-v1", + }, + { + parameter: "call_relationship_coverage", + current: "function-body-call-coverage-v5", + legacy: "decoded-call-target-coverage-v2", + }, + { + parameter: "call_relationship_coverage", + current: "function-body-call-coverage-v5", + legacy: "decoded-call-target-coverage-v3", + }, + { + parameter: "call_relationship_coverage", + current: "function-body-call-coverage-v5", + legacy: "function-body-call-coverage-v4", + }, + ])( + "refuses snapshots with outdated $parameter semantics", + async ({ parameter, current, legacy: prior }) => { + const initial = fixture(); + await open(initial.session); + const snapshot = initial.session.exportAnalysisSnapshot(); + if (!snapshot.ok) throw snapshot.error; + await initial.session.close(); + const profile = snapshot.value.binding.analysis_profile; + expect(profile.parameters[parameter]).toBe(current); + const legacy = { + ...snapshot.value, + binding: { + ...snapshot.value.binding, + analysis_profile: createAnalysisProfile(profile.provider, { + ...Object.fromEntries( + Object.entries(profile.parameters).filter( + ([key]) => key !== parameter, + ), + ), + ...(prior === undefined ? {} : { [parameter]: prior }), + }), + }, + }; + const replay = fixture(); + const imported = replay.session.importAnalysisSnapshot(legacy); + if (!imported.ok) throw imported.error; + expect(await replay.session.previewTarget(TARGET)).toMatchObject({ + ok: false, + error: { _tag: "EvidenceIntegrityError" }, + }); + expect(replay.calls).toEqual([]); + await replay.session.close(); + }, + ); +}); + +describe("Ghidra snapshot cache", () => { it.each([ { operation: "analyze_function", parameters: { procedure: "0x1000" } }, { diff --git a/src/go/GoBinaryContainer.ts b/src/go/GoBinaryContainer.ts index 96b4cb1469..11a3c3adae 100644 --- a/src/go/GoBinaryContainer.ts +++ b/src/go/GoBinaryContainer.ts @@ -12,7 +12,7 @@ export class GoBinaryFormatFailure extends Error { /** Resource boundary exceeded while decoding metadata or native container structure. */ export class GoBinaryResourceFailure extends Error { constructor( - readonly boundary: "build-info" | "structure", + readonly boundary: "build-info" | "structure" | "search", readonly maximum_bytes: number, message: string, ) { @@ -44,7 +44,7 @@ export interface GoImageSection { readonly compressed: boolean; } -/** Native container facts and its producer-defined build-info search area. */ +/** Native container facts and its ordered producer-defined build-info search ranges. */ export interface GoBinaryContainer { readonly format: "elf" | "pe" | "macho"; readonly architecture: string; @@ -53,7 +53,7 @@ export interface GoBinaryContainer { readonly mappings: readonly GoFileMapping[]; readonly zero_fills: readonly GoZeroFillMapping[]; readonly sections: readonly GoImageSection[]; - readonly search: GoFileMapping | null; + readonly search: readonly GoFileMapping[]; } /** Reader that checks every structural range before decoding numeric fields. */ diff --git a/src/go/GoBinaryImage.ts b/src/go/GoBinaryImage.ts index e94a80ab65..c9e2366c8c 100644 --- a/src/go/GoBinaryImage.ts +++ b/src/go/GoBinaryImage.ts @@ -25,6 +25,7 @@ const MAGIC = Buffer.from("ff20476f206275696c64696e663a", "hex"); const MODULE_START = Buffer.from("3077af0c9274080241e1c107e6d618e6", "hex"); const MODULE_END = Buffer.from("f932433186182072008242104116d8f2", "hex"); const MAX_BUILD_INFO_BYTES = 1024 * 1024; +const MAX_HEADER_SEARCH_BYTES = 256 * 1024 * 1024; /** Exact file span of one embedded string's contents, excluding its length/header. */ export interface GoStringLocation { @@ -57,10 +58,31 @@ export interface GoBinaryImage { export const readGoBinaryImage = (bytes: Buffer): GoBinaryImage => { const container = readGoBinaryContainer(bytes); const { format, architecture, bits, byte_order } = container; - const search = container.search; - const header = findHeader(bytes, search); - if (header === null || search === null) + let selected: { search: GoFileMapping; header: number } | null = null; + const searched = new Set(); + let remainingSearchBytes = MAX_HEADER_SEARCH_BYTES; + for (const search of container.search) { + // Identical file spans and alignment inspect the same candidate bytes. + // Partially overlapping spans still consume a cumulative work budget. + const identity = `${search.offset}:${search.size}:${search.address % 16n}`; + if (searched.has(identity)) continue; + searched.add(identity); + if (search.size > remainingSearchBytes) + throw new GoBinaryResourceFailure( + "search", + MAX_HEADER_SEARCH_BYTES, + "Go build-info discovery exceeds the cumulative 256 MiB header-search budget", + ); + remainingSearchBytes -= search.size; + const header = findHeader(bytes, search); + if (header !== null) { + selected = { search, header }; + break; + } + } + if (selected === null) return { format, architecture, bits, byte_order, build_info: null }; + const { search, header } = selected; const verifyMapping = (size: number): void => { const address = search.address + BigInt(header - search.offset); if ( @@ -222,11 +244,8 @@ export const readGoBinaryContainer = (bytes: Buffer): GoBinaryContainer => { ); }; -const findHeader = ( - bytes: Buffer, - search: GoFileMapping | null, -): number | null => { - if (search === null || search.size === 0) return null; +const findHeader = (bytes: Buffer, search: GoFileMapping): number | null => { + if (search.size === 0) return null; const end = search.offset + search.size; const selected = bytes.subarray(search.offset, end); let offset = Number((16n - (search.address % 16n)) % 16n); diff --git a/src/go/GoBinaryProvider.ts b/src/go/GoBinaryProvider.ts index 466136f3f6..9e2ac1f301 100644 --- a/src/go/GoBinaryProvider.ts +++ b/src/go/GoBinaryProvider.ts @@ -55,7 +55,7 @@ const failureForPrimary = ( if (cause instanceof GoBinaryResourceFailure) return new AnalysisResourceConstraintError( OPERATION, - "memory", + cause.boundary === "search" ? "cpu" : "memory", `Selected artifact ${path}: ${cause.message}`, { boundary: cause.boundary, maximum_bytes: cause.maximum_bytes }, { cause }, @@ -194,7 +194,7 @@ export class GoBinaryProvider implements GoBinaryPort { limitations: [ "Embedded build metadata is observed file content, not proof of compiler identity, dependency authenticity or runtime behavior.", "Only Go build information is inspected; function names, source mappings, type metadata and decompiled code are not recovered.", - "Resource guards bound stable artifact reads to 256 MiB, aggregate embedded strings to 1 MiB, structural table decoding to 16 MiB and complete metadata payloads to 8 MiB; these are not operating-system memory or CPU limits.", + "Resource guards bound stable artifact reads and cumulative header searches to 256 MiB each, aggregate embedded strings to 1 MiB, structural table decoding to 16 MiB and complete metadata payloads to 8 MiB; these are not operating-system memory or CPU limits.", ...(info !== null && (info.go_version === null || info.module_text === null) ? [ diff --git a/src/go/GoElfImage.ts b/src/go/GoElfImage.ts index de43881141..1ba0c30584 100644 --- a/src/go/GoElfImage.ts +++ b/src/go/GoElfImage.ts @@ -103,7 +103,7 @@ export const readGoElfImage = (bytes: Buffer): GoBinaryContainer => { const mappings: GoFileMapping[] = []; const zeroFills: GoZeroFillMapping[] = []; const sections: GoImageSection[] = []; - let fallback: GoFileMapping | null = null; + const fallback: GoFileMapping[] = []; for (let index = 0; index < phCount; index++) { const offset = phStart + index * phWidth; if (reader.u32(offset) !== 1) continue; @@ -126,7 +126,7 @@ export const readGoElfImage = (bytes: Buffer): GoBinaryContainer => { if (mapping.size !== 0) mappings.push(mapping); if (memorySize > size) zeroFills.push({ address: address + size, size: memorySize - size }); - if (fallback === null && (flags & 3) === 2) fallback = mapping; + if (mapping.size !== 0 && (flags & 3) === 2) fallback.push(mapping); } let search: GoFileMapping | null = null; if (shCount !== 0) { @@ -226,7 +226,7 @@ export const readGoElfImage = (bytes: Buffer): GoBinaryContainer => { mappings, zero_fills: zeroFills, sections, - search: search ?? fallback, + search: search === null ? fallback : [search], }; }; diff --git a/src/go/GoMachoImage.ts b/src/go/GoMachoImage.ts index 5184a5829d..cba13bf2db 100644 --- a/src/go/GoMachoImage.ts +++ b/src/go/GoMachoImage.ts @@ -193,7 +193,7 @@ export const readGoMachoImage = (bytes: Buffer): GoBinaryContainer => { mappings, zero_fills: zeroFills, sections, - search: named[0] ?? fallback, + search: named.length > 0 ? named : fallback === null ? [] : [fallback], }; }; diff --git a/src/go/GoPeImage.ts b/src/go/GoPeImage.ts index 0787bc93d7..304a703974 100644 --- a/src/go/GoPeImage.ts +++ b/src/go/GoPeImage.ts @@ -157,7 +157,7 @@ export const readGoPeImage = (bytes: Buffer): GoBinaryContainer => { mappings, zero_fills: zeroFills, sections, - search, + search: search === null ? [] : [search], }; }; diff --git a/src/hopper/BridgeLauncher.ts b/src/hopper/BridgeLauncher.ts index 043d394a4b..1cef1cf7d2 100644 --- a/src/hopper/BridgeLauncher.ts +++ b/src/hopper/BridgeLauncher.ts @@ -1,5 +1,5 @@ import { execFile } from "node:child_process"; -import { chmod, writeFile } from "node:fs/promises"; +import { chmod, rm, writeFile } from "node:fs/promises"; import { basename, dirname, extname, join } from "node:path"; import { promisify } from "node:util"; @@ -9,6 +9,8 @@ import { HopperStartError, } from "../domain/hopperErrors.js"; import { err, ok, type Result } from "../domain/result.js"; +import { ProviderCleanupError } from "../domain/providerCleanupError.js"; +import { ArtifactReaderFailure } from "../artifacts/ArtifactReader.js"; import { cleanupOwnedProcessGroup } from "../process/ProcessOwnership.js"; import { execFileOutput } from "../process/ExecFileOutput.js"; import { @@ -30,9 +32,12 @@ import { import { acquireHopperTargetLease, acquireLinuxHopperApplicationLease, -} from "./HopperTargetLease.js"; -import type { HopperTargetLease } from "./HopperTargetLease.js"; +} from "../process/HopperLease.js"; +import type { HopperTargetLease } from "../process/HopperLease.js"; import { HOPPER_SHUTDOWN_TIMEOUT_MS } from "./HopperSessionValues.js"; +import { verifyMacHopperNativeHelper } from "./HopperMacPublisher.js"; +import { inspectLinuxHopperLauncher } from "../application/LinuxHopper.js"; +import { retryLinuxHopperPublisherCleanup } from "../application/LinuxHopperPublisher.js"; const execFileAsync = promisify(execFile); const HOPPER_BACKGROUND_STARTUP_MS = 5_000; @@ -47,6 +52,8 @@ export interface BridgeSession { /** Process handle returned by a bridge launcher. */ export type BridgeLaunch = { + /** Verified publisher and sealed native-helper identity; omission is unknown. */ + readonly nativeHelperVerified?: true; /** Only literal native-helper arguments support a pre-handoff denial conclusion. */ readonly authorizationDenialBeforeHandoff?: true; /** Release application/target exclusivity only after verified document and process cleanup. */ @@ -70,6 +77,8 @@ export type BridgeLaunch = { /** Application-owned capability that starts the in-Hopper bridge. */ export interface BridgeLauncher { + /** Retry owned pre-launch resources even when no application was started. */ + cleanup?(): Promise>; launch( session: BridgeSession, options?: { readonly signal?: AbortSignal }, @@ -106,6 +115,9 @@ export type HopperApplicationLauncherOptions = }); export interface HopperApplicationLauncherDependencies { + readonly verifyNativeHelper?: typeof verifyMacHopperNativeHelper; + readonly verifyLinuxLauncher?: typeof inspectLinuxHopperLauncher; + readonly retryPublisherCleanup?: typeof retryLinuxHopperPublisherCleanup; readonly platform?: NodeJS.Platform; readonly acquireTargetLease?: typeof acquireHopperTargetLease; readonly acquireApplicationLease?: typeof acquireLinuxHopperApplicationLease; @@ -136,6 +148,32 @@ export class HopperApplicationLauncher implements BridgeLauncher { readonly dependencies: HopperApplicationLauncherDependencies = {}, ) {} + async cleanup(): Promise> { + if (!usesLinuxDemo(this.options)) return ok(null); + try { + await ( + this.dependencies.retryPublisherCleanup ?? + retryLinuxHopperPublisherCleanup + )(); + return ok(null); + } catch (cause: unknown) { + const observation = ArtifactReaderFailure.cleanupObservation( + cause, + "hopper-publisher-verification", + ); + return err( + new ProviderCleanupError( + "hopper", + observation.resources, + { + reason: observation.reason, + }, + { cause }, + ), + ); + } + } + async launch( session: BridgeSession, options: { readonly signal?: AbortSignal } = {}, @@ -209,10 +247,38 @@ export class HopperApplicationLauncher implements BridgeLauncher { action, input.loadedPath, ]; + let launched = false; try { if (signal?.aborted === true) return err(new HopperCancelledError()); const display = await this.#privateDisplay(signal); if (!display.ok) return err(display.error); + let expectedLinuxSha256: string | undefined; + if (usesLinuxDemo(this.options)) { + const inspected = await ( + this.dependencies.verifyLinuxLauncher ?? inspectLinuxHopperLauncher + )(this.options.launcherPath, this.options.environment, signal); + if (!inspected.ok) + return inspected.error.reason === "cancelled" + ? err(new HopperCancelledError()) + : err( + new HopperStartError({ + userMessage: inspected.error.message, + cause: inspected.error, + launchRejection: { + boundary: "publisher_verification", + reason: inspected.error.reason, + }, + }), + ); + expectedLinuxSha256 = inspected.value.sha256; + } + const nativeHelperVerified = + (this.dependencies.platform ?? process.platform) === "darwin" && + this.options.launchMode === "native" && + (await ( + this.dependencies.verifyNativeHelper ?? verifyMacHopperNativeHelper + )(this.options.launcherPath, this.options.environment, signal)) === + "verified"; const prepared = await prepareHopperApplication( this.options.launcherPath, this.options.environment, @@ -224,6 +290,7 @@ export class HopperApplicationLauncher implements BridgeLauncher { session, argumentsForTarget, display.strategy, + expectedLinuxSha256, ); const started = await launchHopperProcess({ options: this.options, @@ -235,6 +302,7 @@ export class HopperApplicationLauncher implements BridgeLauncher { signal, }); if (started === null) return err(new HopperCancelledError()); + launched = true; const cleanup = () => cleanupOwnedProcessGroup(started.ownership); const ownedLauncher = { ...(this.options.preparedImage === undefined @@ -257,6 +325,9 @@ export class HopperApplicationLauncher implements BridgeLauncher { ...ownedLauncher, providerLifetime: "external-application", shutdownMode: "bridge-request", + ...(nativeHelperVerified + ? { nativeHelperVerified: true as const } + : {}), // The vendor helper interpolates these values into AppleScript // literals without escaping. Preserve them, but withhold the // pre-handoff conclusion when they can alter that script. @@ -269,6 +340,12 @@ export class HopperApplicationLauncher implements BridgeLauncher { }); } catch (cause: unknown) { return err(hopperLaunchFailure(cause, signal)); + } finally { + // Remove the token-bearing bootstrap file on every non-success path. + // Only the launched process needs the file; all error returns, early + // cancellations, and thrown exceptions must not leave it on disk. + if (!launched) + await rm(bootstrapPath, { force: true }).catch(() => undefined); } } @@ -333,6 +410,8 @@ export class HopperApplicationLauncher implements BridgeLauncher { await chmod(bootstrapPath, 0o600); return ok(bootstrapPath); } catch (cause: unknown) { + // Clean up any partially written bootstrap file that may contain the token. + await rm(bootstrapPath, { force: true }).catch(() => undefined); return err(new HopperStartError({ cause })); } } @@ -425,6 +504,7 @@ export const linuxDemoLaunch = ( session: BridgeSession, hopperArgs: readonly string[], strategy: LinuxPrivateDisplayRunnableStrategy | undefined, + expectedSha256?: string, ): | { readonly command: string; @@ -434,6 +514,10 @@ export const linuxDemoLaunch = ( | undefined => { if (!usesLinuxDemo(options) || options.demoHelperPath === undefined) return undefined; + if (expectedSha256 === undefined || !/^[a-f0-9]{64}$/u.test(expectedSha256)) + throw new TypeError( + "Linux Hopper launch requires an authenticated launcher SHA-256", + ); const helperArguments = [ options.demoHelperPath, "--owner-stdin", @@ -442,6 +526,8 @@ export const linuxDemoLaunch = ( ...(strategy === "user-mount-namespace" ? ["--mount-private-x11"] : []), "--hopper", options.launcherPath, + "--expected-launcher-sha256", + expectedSha256, "--socket", session.socketPath, "--", @@ -477,7 +563,7 @@ const hopperLaunchFailure = ( ? new HopperCancelledError() : new HopperStartError({ cause }); -/** Select the version-pinned adapter from the caller's explicit launch mode. */ +/** Select the publisher-verified demo adapter from the caller's explicit launch mode. */ export const usesLinuxDemo = ( options: HopperApplicationLauncherOptions, ): options is HopperApplicationLauncherOptions & { diff --git a/src/hopper/HopperAnalysisProfile.test.ts b/src/hopper/HopperAnalysisProfile.test.ts index 7c3c715f23..e5b5745049 100644 --- a/src/hopper/HopperAnalysisProfile.test.ts +++ b/src/hopper/HopperAnalysisProfile.test.ts @@ -1,6 +1,14 @@ +import { createHash } from "node:crypto"; +import { writeFile } from "node:fs/promises"; +import { join } from "node:path"; + import { expect, it } from "vitest"; -import { hopperLoaderArgsForTarget } from "./HopperAnalysisProfile.js"; +import { createTestTempDirectory } from "../../tests/fixtures/temporaryDirectory.js"; +import { + hopperLoaderArgsForTarget, + resolveHopperAnalysisProfile, +} from "./HopperAnalysisProfile.js"; it("routes physical universal Mach-O containers through Hopper's FAT loader", () => { const result = hopperLoaderArgsForTarget( @@ -21,3 +29,65 @@ it("routes physical universal Mach-O containers through Hopper's FAT loader", () value: ["-l", "FAT", "--aarch64", "-l", "Mach-O"], }); }); + +it("prepares a verified thin arm64e slice for a FAT32 system container", async () => { + const directory = await createTestTempDirectory("rea-hopper-fat32-profile-"); + const writeUInt32 = (buffer: Buffer, offset: number, value: number): void => { + buffer.writeUInt32BE(value, offset); + }; + const slices = [ + { cpuType: 0x01000007, cpuSubtype: 3 }, + { cpuType: 0x0100000c, cpuSubtype: 0x80000002 }, + { cpuType: 0x0100000c, cpuSubtype: 0x8000000c }, + ]; + const container = Buffer.alloc(3 * 4096 + 64); + writeUInt32(container, 0, 0xcafebabe); + writeUInt32(container, 4, slices.length); + slices.forEach((slice, index) => { + const offset = 4096 * (index + 1); + const entry = 8 + index * 20; + writeUInt32(container, entry, slice.cpuType); + writeUInt32(container, entry + 4, slice.cpuSubtype); + writeUInt32(container, entry + 8, offset); + writeUInt32(container, entry + 12, 64); + writeUInt32(container, entry + 16, 12); + writeUInt32(container, offset, 0xfeedfacf); + writeUInt32(container, offset + 4, slice.cpuType); + writeUInt32(container, offset + 8, slice.cpuSubtype); + }); + const arm64eOffset = 8192; + const path = join(directory, "system.fat"); + await writeFile(path, container); + const result = await resolveHopperAnalysisProfile( + { + path, + sha256: createHash("sha256").update(container).digest("hex"), + kind: "executable", + format: "mach-o-universal", + architecture: "arm64", + availableArchitectures: ["x86_64", "arm64"], + universalSlice: { offset: arm64eOffset, size: 64 }, + }, + { + launcherPath: path, + loaderArgsOverride: [], + provider: { id: "hopper", name: "Hopper Disassembler", version: null }, + }, + ); + if (!result.ok) throw new Error(result.error.message); + const parameters = result.value.profile?.parameters; + expect(parameters).toMatchObject({ + macho_container: "fat32", + prepared_image: { + method: "fat32_thin_slice", + source_offset: arm64eOffset, + byte_length: 64, + cpu_type: 0x0100000c, + cpu_subtype: 0x80000002, + }, + loader: { + source: "derived", + arguments: ["-l", "Mach-O", "--aarch64"], + }, + }); +}); diff --git a/src/hopper/HopperAnalysisProfile.ts b/src/hopper/HopperAnalysisProfile.ts index a4848533f3..85159ee315 100644 --- a/src/hopper/HopperAnalysisProfile.ts +++ b/src/hopper/HopperAnalysisProfile.ts @@ -54,7 +54,7 @@ export const resolveHopperAnalysisProfile = async ( let preparedImage: HopperMachOImage | undefined; if ( target.kind === "executable" && - machoContainer === "fat64" && + (machoContainer === "fat64" || machoContainer === "fat32") && options.loaderArgsOverride.length === 0 ) { const resolved = await resolveHopperMachOImage(target, options.signal); @@ -82,6 +82,7 @@ export const resolveHopperAnalysisProfile = async ( profile: createAnalysisProfile(provider, { target_kind: target.kind, target_format: target.format, + function_dossier_coverage: "observed-facets-and-call-relationships-v1", architecture: target.architecture ?? null, available_architectures: [ ...(target.availableArchitectures ?? []), @@ -110,6 +111,7 @@ const HOPPER_DECLINED_ARCHITECTURE_LABELS: Record< > = { mips: "MIPS", riscv: "RISC-V", + powerpc: "PowerPC", }; /** Derive Hopper's complete non-interactive CLI loader selection. */ diff --git a/src/hopper/HopperCleanup.test.ts b/src/hopper/HopperCleanup.test.ts index 8e078b13df..2e56864af7 100644 --- a/src/hopper/HopperCleanup.test.ts +++ b/src/hopper/HopperCleanup.test.ts @@ -7,11 +7,15 @@ import { EventEmitter } from "node:events"; import { Socket } from "node:net"; import pino from "pino"; -import { describe, expect, it } from "vitest"; +import { describe, expect, it, vi } from "vitest"; import { ok } from "../domain/result.js"; import type { BridgeLaunch } from "./BridgeLauncher.js"; -import { cleanupHopperSession } from "./HopperCleanup.js"; +import { + cleanupHopperSession, + confirmExternalApplicationExit, + type HopperOwnedResources, +} from "./HopperCleanup.js"; const loggerHarness = () => { const logs: unknown[] = []; @@ -47,6 +51,61 @@ const processCleanupLaunch = (): BridgeLaunch => ({ const failure = (message: string) => Object.assign(new TypeError(message), { code: "ECONNRESET" }); +const externalApplicationResources = (): HopperOwnedResources => ({ + launch: { + process: Object.assign(new EventEmitter(), { + stdout: null, + stderr: null, + exitCode: null, + signalCode: null, + kill: () => true, + }), + ownsProcessLifetime: false, + providerLifetime: "external-application", + shutdownMode: "bridge-request", + }, + processSupervisor: undefined, + runtimeRoot: undefined, + shutdownConfirmed: false, + externalApplication: { + pid: 123, + identity: { state: "readable", identity: "captured-start" }, + }, +}); + +describe("external Hopper process exit confirmation", () => { + it("confirms only the captured GUI process exit and releases its transport", async () => { + const resources = externalApplicationResources(); + const released: unknown[] = []; + const observeExit = vi.fn().mockResolvedValue("exited"); + await confirmExternalApplicationExit( + resources, + undefined, + (socket) => released.push(socket), + observeExit, + ); + expect(observeExit).toHaveBeenCalledWith(123, "captured-start"); + expect(resources.shutdownConfirmed).toBe(true); + expect(released).toEqual([undefined]); + }); + + it.each(["present", "unverified"] as const)( + "does not confirm an external application while its exact identity is %s", + async (observation) => { + const resources = externalApplicationResources(); + const releaseTransport = vi.fn(); + await confirmExternalApplicationExit( + resources, + undefined, + releaseTransport, + async () => observation, + ); + expect(resources.shutdownConfirmed).toBe(false); + expect(releaseTransport).not.toHaveBeenCalled(); + }, + ); +}); + describe("Hopper shutdown rejection diagnostics", () => { it("retains the primary shutdown operation and rejection cause", async () => { const { logger, logs } = loggerHarness(); @@ -57,6 +116,7 @@ describe("Hopper shutdown rejection diagnostics", () => { processSupervisor: undefined, runtimeRoot: undefined, shutdownConfirmed: false, + externalApplication: undefined, }, activeRequest: null, retainDocument: false, @@ -96,6 +156,7 @@ describe("Hopper shutdown rejection diagnostics", () => { processSupervisor: undefined, runtimeRoot: undefined, shutdownConfirmed: false, + externalApplication: undefined, }, activeRequest: null, retainDocument: false, @@ -152,6 +213,7 @@ it("keeps a prepared backing image when native document closure cannot be confir runtimeRoot, processSupervisor: undefined, shutdownConfirmed: false, + externalApplication: undefined, }, activeRequest: null, retainDocument: true, diff --git a/src/hopper/HopperCleanup.ts b/src/hopper/HopperCleanup.ts index eb155a9fb5..ac4cce6594 100644 --- a/src/hopper/HopperCleanup.ts +++ b/src/hopper/HopperCleanup.ts @@ -10,8 +10,12 @@ import { ProviderCleanupError } from "../domain/providerCleanupError.js"; import { err, ok, type Result } from "../domain/result.js"; import type { Logger } from "pino"; import type { PrivateRuntimeRoot } from "../process/PrivateRuntimeRoot.js"; -import type { ProcessCleanupResult } from "../process/ProcessOwnership.js"; +import type { + ProcessCleanupResult, + ProcessIdentityObservation, +} from "../process/ProcessOwnership.js"; import type { ProviderProcessSupervisor } from "../process/ProviderProcess.js"; +import { observeProcessStartIdentityExit } from "../process/ProcessOwnershipObservation.js"; import type { BridgeLaunch } from "./BridgeLauncher.js"; import { createOwnedHopperShutdownDiagnostic, @@ -42,6 +46,12 @@ export interface HopperOwnedResources { processSupervisor: ProviderProcessSupervisor | undefined; runtimeRoot: PrivateRuntimeRoot | undefined; shutdownConfirmed: boolean; + externalApplication: + | { + readonly pid: number; + readonly identity: ProcessIdentityObservation; + } + | undefined; } export interface HopperCleanupInput { @@ -68,6 +78,11 @@ export const cleanupHopperSession = async ( cleanupResult: undefined, processStopped: input.resources.processSupervisor === undefined, }; + await confirmExternalApplicationExit( + input.resources, + input.socket, + input.releaseTransport, + ); await report( input.progress, 0, @@ -119,6 +134,28 @@ export const cleanupHopperSession = async ( return cleanupOutcome(input, state); }; +export const confirmExternalApplicationExit = async ( + resources: HopperOwnedResources, + socket: Socket | undefined, + releaseTransport: (socket: Socket | undefined) => void, + observeExit: typeof observeProcessStartIdentityExit = observeProcessStartIdentityExit, +): Promise => { + const application = resources.externalApplication; + if ( + resources.shutdownConfirmed || + resources.launch?.providerLifetime !== "external-application" || + application?.identity.state !== "readable" + ) + return; + if ( + (await observeExit(application.pid, application.identity.identity)) === + "exited" + ) { + resources.shutdownConfirmed = true; + releaseTransport(socket); + } +}; + const requestShutdown = async ( input: HopperCleanupInput, state: CleanupState, diff --git a/src/hopper/HopperClient.ts b/src/hopper/HopperClient.ts index 8196adcee7..a0668c3a77 100644 --- a/src/hopper/HopperClient.ts +++ b/src/hopper/HopperClient.ts @@ -29,6 +29,7 @@ import { type ProviderOperationRequest, } from "../domain/providerOperationHealth.js"; import { err, ok, type Result } from "../domain/result.js"; +import { ProviderCleanupError } from "../domain/providerCleanupError.js"; import type { JsonValue } from "../domain/jsonValue.js"; import type { ProgressReporter } from "../application/ProgressReporter.js"; import { silentLogger } from "../logger.js"; @@ -44,11 +45,13 @@ import { type ProviderProcessDiagnostic, ProviderProcessSupervisor, } from "../process/ProviderProcess.js"; +import { observeProcessStartIdentity } from "../process/ProcessOwnershipObservation.js"; import type { BridgeLaunch, BridgeLauncher } from "./BridgeLauncher.js"; import { hopperOperationStage } from "./HopperOperationStage.js"; import type { HopperDiagnostic } from "./HopperDiagnostics.js"; import { cleanupHopperSession, + confirmExternalApplicationExit, type HopperOwnedResources, } from "./HopperCleanup.js"; import { @@ -59,6 +62,7 @@ import { import { connectHopperSocketOnce } from "./HopperSocketConnection.js"; import { hopperLauncherFailureDiagnostic, + hopperLauncherReportedApplicationNotFound, nativeHopperLaunchWasDenied, } from "./HopperProcessDiagnostic.js"; import { DEFAULT_HOPPER_STARTUP_TIMEOUT_MS } from "../config/providerStartupTimeout.js"; @@ -112,6 +116,7 @@ export class HopperClient { processSupervisor: undefined, runtimeRoot: undefined, shutdownConfirmed: false, + externalApplication: undefined, }; #token: string | undefined; readonly #lineage = new ProviderRunLineage(); @@ -324,7 +329,8 @@ export class HopperClient { if ( this.#socket !== undefined || this.#resources.launch !== undefined || - this.#resources.runtimeRoot !== undefined + this.#resources.runtimeRoot !== undefined || + this.#startupFailureResult?.cleanupIncomplete === true ) { return err( this.#startupFailureResult ?? @@ -433,6 +439,27 @@ export class HopperClient { } const parsed = parseHopperServerInfo(health.value, runId); if (!parsed.ok) return this.#startupFailure(parsed.error); + if (this.#resources.launch?.providerLifetime === "external-application") { + let identity; + try { + identity = await observeProcessStartIdentity(parsed.value.processId); + } catch (cause: unknown) { + identity = { + state: "unavailable" as const, + reason: + cause instanceof Error + ? cause.message + : "process identity observation failed", + }; + } + this.#resources.externalApplication = { + pid: parsed.value.processId, + identity: identity ?? { + state: "unavailable", + reason: "process identity was not returned while Hopper was running", + }, + }; + } await this.#lineage.observe(this.#resources.launch); return parsed; } @@ -534,8 +561,13 @@ export class HopperClient { // best-effort cleanup: a rejected startup must not mask close/cleanup; // #cleanup reports the authoritative close result. await starting?.catch(() => undefined); + await confirmExternalApplicationExit(this.#resources, this.#socket, () => + this.#releaseTransport(), + ); await this.#reconnectForCleanup(); - return await this.#cleanup(options); + const closed = await this.#cleanup(options); + if (closed.ok) this.#startupFailureResult = undefined; + return closed; } finally { if (this.#startPromise === starting) this.#startPromise = undefined; if (this.#startupController === controller) { @@ -613,7 +645,7 @@ export class HopperClient { this.#closing = true; this.#ready = false; try { - return await cleanupHopperSession({ + const closed = await cleanupHopperSession({ socket: this.#socket, resources: this.#resources, activeRequest: this.#requests.activity(), @@ -625,6 +657,26 @@ export class HopperClient { retainDocument: options.retainDocument === true, releaseTransport: () => this.#releaseTransport(), }); + const launcherClosed = await this.#options.launcher + .cleanup?.() + .catch((cause: unknown) => + err( + new ProviderCleanupError( + "hopper", + ["hopper-launcher"], + { + reason: cause instanceof Error ? cause.message : String(cause), + }, + { cause }, + ), + ), + ); + if (launcherClosed === undefined || launcherClosed.ok) return closed; + return closed.ok + ? launcherClosed + : err( + analysisErrorWithCleanupFailure(closed.error, launcherClosed.error), + ); } finally { if (this.#socket?.destroyed === true) this.#socket = undefined; @@ -768,6 +820,17 @@ export class HopperClient { launcherFailure, }); } + if (hopperLauncherReportedApplicationNotFound(launcherFailure)) { + // Only the verified native helper's Spotlight exit precedes any target + // handoff. Wrappers can forward the same message after opening a document. + if (this.#resources.launch?.nativeHelperVerified === true) + this.#resources.shutdownConfirmed = true; + return new HopperStartError({ + userMessage: + "Hopper's launcher could not find Hopper because it relies on Spotlight. Enable Spotlight indexing for the volume that contains Hopper (for example: sudo mdutil -i on /System/Volumes/Data), wait for indexing to finish, then retry opening the target.", + launcherFailure, + }); + } return new HopperStartError({ userMessage: "Hopper's launcher helper failed before bridge readiness. Review details.launcher and complete Hopper setup before retrying.", diff --git a/src/hopper/HopperMacPublisher.ts b/src/hopper/HopperMacPublisher.ts new file mode 100644 index 0000000000..f60852915e --- /dev/null +++ b/src/hopper/HopperMacPublisher.ts @@ -0,0 +1,36 @@ +import { HOPPER_PUBLISHER } from "../domain/hopperPublisher.js"; +import { macDeveloperIdRequirement } from "../process/MacCodeSignature.js"; +import { execFileOutput } from "../process/ExecFileOutput.js"; + +/** Failure or unavailable verification leaves custom/native provenance unknown. */ +export const verifyMacHopperNativeHelper = async ( + launcherPath: string, + environment: Readonly, + signal?: AbortSignal, +): Promise<"verified" | "unknown"> => { + // codesign treats a bare name as a local file, while spawn searches PATH. + // Without an explicit path they need not identify the same executable. + if (!launcherPath.includes("/")) return "unknown"; + try { + await execFileOutput( + "/usr/bin/codesign", + [ + "--verify", + "--strict", + `-R${macDeveloperIdRequirement("hopper", HOPPER_PUBLISHER.teamIdentifier)}`, + launcherPath, + ], + { + env: environment, + ...(signal === undefined ? {} : { signal }), + timeout: 5_000, + stopSignal: "SIGKILL", + }, + ); + return "verified"; + } catch (cause: unknown) { + // Optional provenance: a failed check must never authorize lease release. + void cause; + return "unknown"; + } +}; diff --git a/src/hopper/HopperMachOImage.test.ts b/src/hopper/HopperMachOImage.test.ts index d503a055b5..e4f81ec6a2 100644 --- a/src/hopper/HopperMachOImage.test.ts +++ b/src/hopper/HopperMachOImage.test.ts @@ -1,11 +1,109 @@ -import { chmod, writeFile } from "node:fs/promises"; +import { createHash } from "node:crypto"; +import { chmod, readFile, writeFile } from "node:fs/promises"; import { join } from "node:path"; import { expect, it } from "vitest"; import { createTestTempDirectory } from "../../tests/fixtures/temporaryDirectory.js"; import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; -import { resolveHopperMachOImage } from "./HopperMachOImage.js"; +import { + prepareHopperMachOImage, + resolveHopperMachOImage, +} from "./HopperMachOImage.js"; + +const CPU_TYPES = { x86_64: 0x01000007, arm64: 0x0100000c } as const; +const FAT32_ALIGNMENT = 12; +const FAT32_SLICE_SIZE = 64; +const ARM64E = 2; +const ARM64E_X1 = 12; +const ARM64E_WITH_CAPABILITIES = 0x80000002; +const ARM64E_X1_WITH_CAPABILITIES = 0x8000000c; + +interface FatFixtureSlice { + readonly cpuType: number; + readonly cpuSubtype: number; + /** Thin-slice byte order, independent of the container's table byte order. */ + readonly little?: boolean; +} + +interface Fat32Fixture { + readonly container: Buffer; + readonly sliceOffsets: readonly number[]; +} + +/** Synthesize a FAT32 container whose slices are 4096-aligned thin headers. */ +const buildFat32Container = ( + slices: readonly FatFixtureSlice[], + options: { readonly littleEndianTable?: boolean } = {}, +): Fat32Fixture => { + const littleTable = options.littleEndianTable === true; + const writeTableUInt32 = littleTable + ? (buffer: Buffer, offset: number, value: number): number => + buffer.writeUInt32LE(value, offset) + : (buffer: Buffer, offset: number, value: number): number => + buffer.writeUInt32BE(value, offset); + const unit = 2 ** FAT32_ALIGNMENT; + const sliceOffsets = slices.map((_slice, index) => unit * (index + 1)); + const lastOffset = sliceOffsets[sliceOffsets.length - 1] ?? unit; + const container = Buffer.alloc(lastOffset + FAT32_SLICE_SIZE, 0); + // The magic VALUE is 0xcafebabe in the table's byte order: little-endian + // storage yields the FAT_CIGAM magic a big-endian reader observes. + writeTableUInt32(container, 0, 0xcafebabe); + writeTableUInt32(container, 4, slices.length); + for (const [index, slice] of slices.entries()) { + const entry = 8 + index * 20; + const offset = sliceOffsets[index]!; + writeTableUInt32(container, entry, slice.cpuType); + writeTableUInt32(container, entry + 4, slice.cpuSubtype); + writeTableUInt32(container, entry + 8, offset); + writeTableUInt32(container, entry + 12, FAT32_SLICE_SIZE); + writeTableUInt32(container, entry + 16, FAT32_ALIGNMENT); + const writeThinUInt32 = + slice.little === true + ? (buffer: Buffer, fieldOffset: number, value: number): number => + buffer.writeUInt32LE(value, fieldOffset) + : (buffer: Buffer, fieldOffset: number, value: number): number => + buffer.writeUInt32BE(value, fieldOffset); + writeThinUInt32(container, offset, 0xfeedfacf); + writeThinUInt32(container, offset + 4, slice.cpuType); + writeThinUInt32(container, offset + 8, slice.cpuSubtype); + } + return { container, sliceOffsets }; +}; + +const writeFixture = async ( + directory: string, + name: string, + fixture: Fat32Fixture, +): Promise => { + const path = join(directory, name); + await writeFile(path, fixture.container); + return path; +}; + +const fat32Target = ( + path: string, + fixture: Fat32Fixture, + architecture: "arm64" | "x86_64", +) => ({ + path, + sha256: createHash("sha256").update(fixture.container).digest("hex"), + kind: "executable" as const, + format: "mach-o-universal" as const, + architecture, + availableArchitectures: ["x86_64", "arm64"] as const, + universalSlice: { offset: fixture.sliceOffsets[0]!, size: FAT32_SLICE_SIZE }, +}); + +const requireResolved = ( + result: Awaited>, +) => { + if (!result.ok) + throw new Error( + `FAT fixture resolution failed: ${JSON.stringify(projectAnalysisError(result.error))}`, + ); + return result.value; +}; it.skipIf(process.platform === "win32" || process.getuid?.() === 0)( "reports a FAT64 source that became unreadable as a host read denial", @@ -36,7 +134,7 @@ it.skipIf(process.platform === "win32" || process.getuid?.() === 0)( }, ); -it("rejects an ELF target before attempting FAT64 source preparation", async () => { +it("rejects an ELF target before attempting FAT source preparation", async () => { const directory = await createTestTempDirectory("rea-hopper-mips-boundary-"); // No file is created: format validation must precede any filesystem lookup. const path = join(directory, "mips.elf"); @@ -54,6 +152,235 @@ it("rejects an ELF target before attempting FAT64 source preparation", async () expect(result.error).toMatchObject({ _tag: "BinaryTargetError", path, - reason: "Hopper FAT64 image preparation requires a Mach-O target", + reason: "Hopper FAT image preparation requires a Mach-O target", }); }); + +it("selects the plain-arm64-decodable arm64e slice from an Apple-style FAT32 container", async () => { + const directory = await createTestTempDirectory("rea-hopper-fat32-arm64e-"); + // /bin/* system binaries: x86_64, arm64e (cpusubtype 2), arm64e.x1 (subtype 12). + const fixture = buildFat32Container([ + { cpuType: CPU_TYPES.x86_64, cpuSubtype: 3 }, + { cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E }, + { cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E_X1 }, + ]); + const path = await writeFixture(directory, "system.fat", fixture); + const result = requireResolved( + await resolveHopperMachOImage(fat32Target(path, fixture, "arm64")), + ); + expect(result).toMatchObject({ + method: "fat32_thin_slice", + architecture_count: 3, + source_offset: fixture.sliceOffsets[1], + byte_length: FAT32_SLICE_SIZE, + cpu_type: CPU_TYPES.arm64, + cpu_subtype: ARM64E, + }); +}); + +it("ranks capability-bearing arm64 subtypes by base and retains raw identity", async () => { + const directory = await createTestTempDirectory( + "rea-hopper-fat32-capabilities-", + ); + const fixture = buildFat32Container([ + { + cpuType: CPU_TYPES.arm64, + cpuSubtype: ARM64E_X1_WITH_CAPABILITIES, + }, + { + cpuType: CPU_TYPES.arm64, + cpuSubtype: ARM64E_WITH_CAPABILITIES, + }, + ]); + const path = await writeFixture(directory, "system.fat", fixture); + const selected = requireResolved( + await resolveHopperMachOImage(fat32Target(path, fixture, "arm64")), + ); + expect(selected).toMatchObject({ + source_offset: fixture.sliceOffsets[1], + cpu_subtype: ARM64E_WITH_CAPABILITIES, + }); +}); + +it("fails closed when capability variants share the preferred arm64e base", async () => { + const directory = await createTestTempDirectory( + "rea-hopper-fat32-capability-ambiguity-", + ); + const fixture = buildFat32Container([ + { cpuType: CPU_TYPES.arm64, cpuSubtype: 0x80000002 }, + { cpuType: CPU_TYPES.arm64, cpuSubtype: 0x81000002 }, + ]); + const path = await writeFixture(directory, "ambiguous.fat", fixture); + const result = await resolveHopperMachOImage( + fat32Target(path, fixture, "arm64"), + ); + if (result.ok || result.error._tag !== "BinaryTargetError") + throw new Error( + "Expected ambiguous FAT selection to return a BinaryTargetError", + ); + expect(result.error.reason).toContain("selection for arm64 is ambiguous"); +}); + +it("prefers cpusubtype 2 over 12 regardless of FAT32 table order", async () => { + const directory = await createTestTempDirectory("rea-hopper-fat32-order-"); + const fixture = buildFat32Container([ + { cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E_X1 }, + { cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E }, + ]); + const path = await writeFixture(directory, "reordered.fat", fixture); + const result = requireResolved( + await resolveHopperMachOImage(fat32Target(path, fixture, "arm64")), + ); + expect(result).toMatchObject({ + cpu_subtype: ARM64E, + source_offset: fixture.sliceOffsets[1], + }); +}); + +it("prefers a plain arm64 slice over arm64e in a FAT32 container", async () => { + const directory = await createTestTempDirectory("rea-hopper-fat32-plain-"); + const fixture = buildFat32Container([ + { cpuType: CPU_TYPES.arm64, cpuSubtype: 0 }, + { cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E }, + ]); + const path = await writeFixture(directory, "plain.fat", fixture); + const result = requireResolved( + await resolveHopperMachOImage(fat32Target(path, fixture, "arm64")), + ); + expect(result).toMatchObject({ + cpu_subtype: 0, + source_offset: fixture.sliceOffsets[0], + }); +}); + +it("keeps a lone arm64e.x1 FAT32 slice selectable to preserve today's open behavior", async () => { + const directory = await createTestTempDirectory("rea-hopper-fat32-lone-"); + const fixture = buildFat32Container([ + { cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E_X1 }, + ]); + const path = await writeFixture(directory, "lone.fat", fixture); + const result = requireResolved( + await resolveHopperMachOImage(fat32Target(path, fixture, "arm64")), + ); + expect(result).toMatchObject({ cpu_subtype: ARM64E_X1 }); +}); + +it("still fails closed when two arm64e.x1 FAT32 slices compete", async () => { + const directory = await createTestTempDirectory("rea-hopper-fat32-ambig-"); + const fixture = buildFat32Container([ + { cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E_X1 }, + { cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E_X1 }, + ]); + const path = await writeFixture(directory, "ambiguous.fat", fixture); + const result = await resolveHopperMachOImage( + fat32Target(path, fixture, "arm64"), + ); + if (result.ok) throw new Error("Ambiguous FAT32 selection succeeded"); + expect( + projectAnalysisError(result.error).details?.reason ?? result.error, + ).toMatch(/selection.*ambiguous/u); +}); + +it("selects the single x86_64 slice of a FAT32 container for an x86_64 target", async () => { + const directory = await createTestTempDirectory("rea-hopper-fat32-intel-"); + const fixture = buildFat32Container([ + { cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E }, + { cpuType: CPU_TYPES.x86_64, cpuSubtype: 3 }, + ]); + const path = await writeFixture(directory, "universal.fat", fixture); + const result = requireResolved( + await resolveHopperMachOImage(fat32Target(path, fixture, "x86_64")), + ); + expect(result).toMatchObject({ + cpu_type: CPU_TYPES.x86_64, + cpu_subtype: 3, + source_offset: fixture.sliceOffsets[1], + }); +}); + +it("parses a little-endian FAT32 table without tying it to slice byte order", async () => { + const directory = await createTestTempDirectory("rea-hopper-fat32-le-"); + const fixture = buildFat32Container( + [ + { cpuType: CPU_TYPES.x86_64, cpuSubtype: 3 }, + { cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E }, + { cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E_X1 }, + ], + { littleEndianTable: true }, + ); + const path = await writeFixture(directory, "little.fat", fixture); + const result = requireResolved( + await resolveHopperMachOImage(fat32Target(path, fixture, "arm64")), + ); + expect(result).toMatchObject({ + method: "fat32_thin_slice", + cpu_subtype: ARM64E, + source_offset: fixture.sliceOffsets[1], + }); +}); + +it("rejects a FAT32 slice whose declared range extends beyond the file", async () => { + const directory = await createTestTempDirectory("rea-hopper-fat32-range-"); + const fixture = buildFat32Container([ + { cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E }, + ]); + fixture.container.writeUInt32BE(fixture.container.length + 16, 8 + 12); + const path = await writeFixture(directory, "overrange.fat", fixture); + const result = await resolveHopperMachOImage( + fat32Target(path, fixture, "arm64"), + ); + if (result.ok) throw new Error("An overranging FAT32 slice was accepted"); + expect( + projectAnalysisError(result.error).details?.reason ?? result.error, + ).toMatch(/extends beyond the file/u); +}); + +it("copies exactly the selected FAT32 slice bytes and verifies the source", async () => { + const directory = await createTestTempDirectory("rea-hopper-fat32-copy-"); + const fixture = buildFat32Container([ + { cpuType: CPU_TYPES.x86_64, cpuSubtype: 3 }, + { cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E }, + { cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E_X1 }, + ]); + const path = await writeFixture(directory, "system.fat", fixture); + const target = fat32Target(path, fixture, "arm64"); + const resolved = requireResolved(await resolveHopperMachOImage(target)); + const destination = join(directory, "image.macho"); + const prepared = await prepareHopperMachOImage( + path, + target.sha256, + resolved, + destination, + ); + if (!prepared.ok) throw new Error(prepared.error.userMessage); + const arm64eOffset = fixture.sliceOffsets[1]!; + const copied = fixture.container.subarray( + arm64eOffset, + arm64eOffset + FAT32_SLICE_SIZE, + ); + expect(await readFile(destination)).toEqual(copied); +}); + +it("rejects FAT32 preparation when the source table changed after resolution", async () => { + const directory = await createTestTempDirectory("rea-hopper-fat32-race-"); + const fixture = buildFat32Container([ + { cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E }, + { cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E_X1 }, + ]); + const path = await writeFixture(directory, "system.fat", fixture); + const target = fat32Target(path, fixture, "arm64"); + const resolved = requireResolved(await resolveHopperMachOImage(target)); + const mutated = Buffer.from(fixture.container); + mutated.writeUInt32BE(0x01000007, 8); + await writeFile(path, mutated); + const destination = join(directory, "image.macho"); + const prepared = await prepareHopperMachOImage( + path, + target.sha256, + resolved, + destination, + ); + if (prepared.ok) + throw new Error("A mutated FAT32 source passed verification"); + expect(prepared.error.userMessage).toMatch(/no longer agrees/u); +}); diff --git a/src/hopper/HopperMachOImage.ts b/src/hopper/HopperMachOImage.ts index ed81ea7495..4b0a48cf51 100644 --- a/src/hopper/HopperMachOImage.ts +++ b/src/hopper/HopperMachOImage.ts @@ -8,7 +8,10 @@ import { AnalysisCancelledError } from "../domain/analysisErrorCore.js"; import { readFatSliceDeclarations, validateFatSlice, + type FatSliceDeclaration, } from "../domain/apple/machoContainer.js"; +import { machoCpuSubtypeBase } from "../domain/apple/machoCpuIdentity.js"; +import type { HeaderArchitecture } from "../domain/binaryTargetTypes.js"; import { HopperCancelledError, HopperStartError, @@ -17,7 +20,10 @@ import { err, ok, type Result } from "../domain/result.js"; /** Stable source coordinates committed before preparing an owned thin image. */ export const hopperMachOImageSchema = z.strictObject({ - method: z.literal("fat64_thin_slice"), + method: z.union([ + z.literal("fat32_thin_slice"), + z.literal("fat64_thin_slice"), + ]), architecture_count: z.number().int().min(1).max(128), container_table_sha256: z.string().regex(/^[a-f0-9]{64}$/u), source_offset: z.number().int().nonnegative().max(Number.MAX_SAFE_INTEGER), @@ -28,7 +34,7 @@ export const hopperMachOImageSchema = z.strictObject({ export type HopperMachOImage = z.infer; -/** Validate the selected FAT64 slice without depending on Apple's lipo tool. */ +/** Validate the selected FAT slice without depending on Apple's lipo tool. */ export const resolveHopperMachOImage = async ( target: BinaryTarget & { readonly kind: "executable" }, signal?: AbortSignal, @@ -45,14 +51,14 @@ export const resolveHopperMachOImage = async ( ); if (signal?.aborted) return err(new AnalysisCancelledError("open_binary")); if (target.format !== "mach-o" && target.format !== "mach-o-universal") - return invalid("Hopper FAT64 image preparation requires a Mach-O target"); + return invalid("Hopper FAT image preparation requires a Mach-O target"); let file: FileHandle; try { file = await open(target.path, "r"); } catch (cause: unknown) { // Keep the OS error so a read-permission denial reports access_denied. return invalid( - `Cannot read FAT64 source for Hopper image preparation: ${cause instanceof Error ? cause.message : String(cause)}`, + `Cannot read FAT source for Hopper image preparation: ${cause instanceof Error ? cause.message : String(cause)}`, cause, ); } @@ -64,24 +70,29 @@ export const resolveHopperMachOImage = async ( }; const header = await read(0, 8); const magic = header.length === 8 ? header.readUInt32BE(0) : 0; - if (magic !== 0xcafebabf && magic !== 0xbfbafeca) + const isFatMagic = [ + 0xcafebabe, 0xcafebabf, 0xbebafeca, 0xbfbafeca, + ].includes(magic); + if (!isFatMagic) return invalid( - "FAT64 source header changed before Hopper image preparation", + "FAT source header changed before Hopper image preparation", ); - const littleEndian = magic === 0xbfbafeca; + const wide = magic === 0xcafebabf || magic === 0xbfbafeca; + const littleEndian = magic === 0xbebafeca || magic === 0xbfbafeca; + const entrySize = wide ? 32 : 20; const count = littleEndian ? header.readUInt32LE(4) : header.readUInt32BE(4); const size = (await file.stat()).size; - const tableEnd = 8 + count * 32; + const tableEnd = 8 + count * entrySize; if (count === 0 || count > 128 || tableEnd > size) return invalid( - "FAT64 architecture table is truncated or exceeds the supported 128 entries", + "FAT architecture table is truncated or exceeds the supported 128 entries", ); - const table = await read(8, count * 32); + const table = await read(8, count * entrySize); const parsed = readFatSliceDeclarations(table, { count, - wide: true, + wide, littleEndian, }); if (parsed.status === "malformed") return invalid(parsed.reason); @@ -93,24 +104,26 @@ export const resolveHopperMachOImage = async ( return err(new AnalysisCancelledError("open_binary")); const rangeError = validateFatSlice(slice, size, tableEnd); if (rangeError !== null) return invalid(rangeError); - const reserved = littleEndian - ? table.readUInt32LE(slice.index * 32 + 28) - : table.readUInt32BE(slice.index * 32 + 28); - if (reserved !== 0) - return invalid( - `FAT64 architecture ${slice.index} has a nonzero reserved field`, - ); + if (wide) { + const reserved = littleEndian + ? table.readUInt32LE(slice.index * 32 + 28) + : table.readUInt32BE(slice.index * 32 + 28); + if (reserved !== 0) + return invalid( + `FAT64 architecture ${slice.index} has a nonzero reserved field`, + ); + } if (slice.size < 12) return invalid( - `FAT64 architecture ${slice.index} has a truncated Mach-O header`, + `FAT architecture ${slice.index} has a truncated Mach-O header`, ); const embedded = await read(slice.offset, 12); if (embedded.length !== 12) - return invalid(`FAT64 architecture ${slice.index} became truncated`); + return invalid(`FAT architecture ${slice.index} became truncated`); const thinMagic = embedded.readUInt32BE(0); if (![0xfeedface, 0xfeedfacf, 0xcefaedfe, 0xcffaedfe].includes(thinMagic)) return invalid( - `FAT64 architecture ${slice.index} does not contain a Mach-O image`, + `FAT architecture ${slice.index} does not contain a Mach-O image`, ); const little = thinMagic === 0xcefaedfe || thinMagic === 0xcffaedfe; const identity = { @@ -133,15 +146,15 @@ export const resolveHopperMachOImage = async ( current !== undefined && current.offset < previous.offset + previous.size ) - return invalid("FAT64 slices overlap"); + return invalid("FAT slices overlap"); } - const selected = matches.length === 1 ? matches[0] : undefined; + const selected = selectHopperSlice(matches, target.architecture); if (selected === undefined) return invalid( - `FAT64 selection for ${target.architecture} is ${matches.length === 0 ? "missing" : "ambiguous"}; extract the exact architecture into a thin image before opening it`, + `FAT selection for ${target.architecture} is ${matches.length === 0 ? "missing" : "ambiguous"}; extract the exact architecture into a thin image before opening it`, ); return ok({ - method: "fat64_thin_slice", + method: wide ? "fat64_thin_slice" : "fat32_thin_slice", architecture_count: count, container_table_sha256: createHash("sha256") .update(header) @@ -154,13 +167,35 @@ export const resolveHopperMachOImage = async ( }); } catch (cause: unknown) { return invalid( - `Cannot validate FAT64 image: ${cause instanceof Error ? cause.message : String(cause)}`, + `Cannot validate FAT image: ${cause instanceof Error ? cause.message : String(cause)}`, ); } finally { await file.close(); } }; +/** + * Plain arm64 (cpusubtype 0) and arm64e (cpusubtype 2) function-entry encodings + * decode under Hopper's --aarch64; Apple's arm64e.x1 variant (cpusubtype 12) + * does not, so it is selected only when it is the arm64 family's only candidate. + * Every other ambiguity stays fail-closed. + */ +const selectHopperSlice = ( + matches: readonly FatSliceDeclaration[], + architecture: HeaderArchitecture, +): FatSliceDeclaration | undefined => { + if (matches.length === 1) return matches[0]; + if (architecture !== "arm64") return undefined; + for (const cpuSubtype of [0, 2]) { + const preferred = matches.filter( + (slice) => machoCpuSubtypeBase(slice.cpuSubtype) === cpuSubtype, + ); + if (preferred.length === 1) return preferred[0]; + if (preferred.length > 1) return undefined; + } + return undefined; +}; + /** Copy a selected slice and verify the entire original source in the same pass. */ export const prepareHopperMachOImage = async ( sourcePath: string, @@ -170,6 +205,7 @@ export const prepareHopperMachOImage = async ( signal?: AbortSignal, ): Promise> => { if (signal?.aborted) return err(new HopperCancelledError()); + const tableEntrySize = image.method === "fat64_thin_slice" ? 32 : 20; try { const source = await open(sourcePath, "r"); try { @@ -177,7 +213,7 @@ export const prepareHopperMachOImage = async ( try { const hash = createHash("sha256"); const tableHash = createHash("sha256"); - const tableEnd = 8 + image.architecture_count * 32; + const tableEnd = 8 + image.architecture_count * tableEntrySize; const thinHeader: Buffer[] = []; let position = 0; let copied = 0; @@ -223,13 +259,13 @@ export const prepareHopperMachOImage = async ( ) return err( new HopperStartError({ - userMessage: `FAT64 source ${sourcePath} no longer agrees with its committed architecture table or selected Mach-O header. Open the stable source again.`, + userMessage: `FAT source ${sourcePath} no longer agrees with its committed architecture table or selected Mach-O header. Open the stable source again.`, }), ); if (observed !== sourceSha256 || copied !== image.byte_length) return err( new HopperStartError({ - userMessage: `FAT64 source ${sourcePath} changed before launch: expected SHA-256 ${sourceSha256}, observed ${observed}; copied ${copied} of ${image.byte_length} selected bytes. Open the stable source again.`, + userMessage: `FAT source ${sourcePath} changed before launch: expected SHA-256 ${sourceSha256}, observed ${observed}; copied ${copied} of ${image.byte_length} selected bytes. Open the stable source again.`, }), ); return ok(destination); @@ -245,7 +281,7 @@ export const prepareHopperMachOImage = async ( : err( new HopperStartError({ cause, - userMessage: `Cannot prepare the selected FAT64 image from ${sourcePath}: ${cause instanceof Error ? cause.message : String(cause)}`, + userMessage: `Cannot prepare the selected FAT image from ${sourcePath}: ${cause instanceof Error ? cause.message : String(cause)}`, }), ); } diff --git a/src/hopper/HopperProcessDiagnostic.test.ts b/src/hopper/HopperProcessDiagnostic.test.ts index 2fe1c4cc9f..6828c7762c 100644 --- a/src/hopper/HopperProcessDiagnostic.test.ts +++ b/src/hopper/HopperProcessDiagnostic.test.ts @@ -1,7 +1,10 @@ import { expect, it } from "vitest"; import type { HopperLauncherOutcome } from "../domain/hopperErrors.js"; -import { nativeHopperLaunchWasDenied } from "./HopperProcessDiagnostic.js"; +import { + hopperLauncherReportedApplicationNotFound, + nativeHopperLaunchWasDenied, +} from "./HopperProcessDiagnostic.js"; // Captured from Hopper 6.1.0-demo and 6.6.0-demo's unmodified helpers with native // AppleScript error -1743 before open dispatch. The target script never ran. @@ -23,6 +26,8 @@ it("recognizes the native helper's complete authorization rejection", () => { }); it.each([ + { stdout: { text: "wrapper output\n", bytes: 15, retained_bytes: 15 } }, + { stdout: { text: "", bytes: 1, retained_bytes: 0 } }, { output_closed: false }, { diagnostic_truncated: true }, { signal: "SIGTERM", exit_code: null }, @@ -52,3 +57,41 @@ it.each([ expect(nativeHopperLaunchWasDenied({ ...denied, ...change })).toBe(false); }, ); + +// Captured from Hopper 6's unmodified helper with Spotlight indexing disabled. +const notFound: HopperLauncherOutcome = { + exit_code: 3, + signal: null, + stdout: { text: "", bytes: 0, retained_bytes: 0 }, + stderr: { + text: "Hopper has not been found.\nPlease make sure that Spotlight is active in order for this tool to find Hopper's executable.\n", + bytes: 121, + retained_bytes: 121, + }, + output_closed: true, + diagnostic_truncated: false, +}; + +it("recognizes the native helper's complete Spotlight not-found report", () => { + expect(hopperLauncherReportedApplicationNotFound(notFound)).toBe(true); + expect(nativeHopperLaunchWasDenied(notFound)).toBe(false); +}); + +it.each([ + { stdout: { text: "wrapper output\n", bytes: 15, retained_bytes: 15 } }, + { stdout: { text: "", bytes: 1, retained_bytes: 0 } }, + { output_closed: false }, + { diagnostic_truncated: true }, + { signal: "SIGTERM", exit_code: null }, + { exit_code: 2 }, + { + stderr: { + ...notFound.stderr, + text: `${notFound.stderr.text}another failure\n`, + }, + }, +])("rejects other not-found diagnostic shapes: %j", (change) => { + expect( + hopperLauncherReportedApplicationNotFound({ ...notFound, ...change }), + ).toBe(false); +}); diff --git a/src/hopper/HopperProcessDiagnostic.ts b/src/hopper/HopperProcessDiagnostic.ts index cf16fb9bb8..47348d8817 100644 --- a/src/hopper/HopperProcessDiagnostic.ts +++ b/src/hopper/HopperProcessDiagnostic.ts @@ -6,6 +6,14 @@ import type { ProviderProcessDiagnostic } from "../process/ProviderProcess.js"; import { parseLinuxPrivateDisplayDiagnostic } from "./LinuxPrivateDisplayDiagnostic.js"; import type { HopperLauncherOutcome } from "../domain/hopperErrors.js"; +/** Recognize launcher stderr signatures only with complete, empty stdout. */ +const hasCompleteLauncherOutput = (outcome: HopperLauncherOutcome): boolean => + outcome.output_closed && + !outcome.diagnostic_truncated && + outcome.stdout.text === "" && + outcome.stdout.bytes === 0 && + outcome.stdout.retained_bytes === 0; + /** * Hopper's native helper sends activate before its open-document Apple event. * Recognize its explicit authorization rejection. The launcher must separately @@ -18,12 +26,27 @@ export const nativeHopperLaunchWasDenied = ( outcome !== undefined && outcome.exit_code === 2 && outcome.signal === null && - outcome.output_closed && - !outcome.diagnostic_truncated && + hasCompleteLauncherOutput(outcome) && /^Cannot launch Hopper!\r?\nNot authorized to send Apple events to [^\r\n]+\.\r?\n?$/u.test( outcome.stderr.text, ); +/** + * Recognize the launcher's complete Spotlight diagnostic for actionable setup + * guidance. Custom launchers may forward this report after a handoff, so the + * report does not establish whether a document exists or release ownership. + */ +export const hopperLauncherReportedApplicationNotFound = ( + outcome: HopperLauncherOutcome | undefined, +): outcome is HopperLauncherOutcome => + outcome !== undefined && + outcome.exit_code === 3 && + outcome.signal === null && + hasCompleteLauncherOutput(outcome) && + /^Hopper has not been found\.\r?\nPlease make sure that Spotlight is active in order for this tool to find Hopper's executable\.\r?\n?$/u.test( + outcome.stderr.text, + ); + /** Extract only a matching Linux adapter failure record. */ export const hopperLauncherFailureDiagnostic = ( event: Extract, diff --git a/src/hopper/HopperProvider.ts b/src/hopper/HopperProvider.ts index aec8c20829..684c8bb08d 100644 --- a/src/hopper/HopperProvider.ts +++ b/src/hopper/HopperProvider.ts @@ -114,6 +114,14 @@ export class HopperProvider implements AnalysisProviderCandidate { reason: `Hopper cannot directly analyze ${target.kind} targets.`, diagnostics, }; + if (target.kind === "database" && this.platform === "darwin") + return { + status: "unsupported", + code: "target_kind_unsupported", + reason: + "This Hopper macOS launcher cannot start REA's analysis bridge when opening a saved Hopper database. Open an executable target instead; saved-database analysis is unavailable until Hopper exposes a database launch route that runs the bridge script.", + diagnostics, + }; if (target.kind === "database") return { status: "supported", @@ -126,7 +134,11 @@ export class HopperProvider implements AnalysisProviderCandidate { status: "unsupported", code: "architecture_unsupported", reason: `REA's Hopper adapter does not admit ${ - target.architecture === "mips" ? "MIPS" : "RISC-V" + target.architecture === "mips" + ? "MIPS" + : target.architecture === "riscv" + ? "RISC-V" + : "PowerPC" } targets.`, diagnostics, }; @@ -268,14 +280,15 @@ export class HopperProvider implements AnalysisProviderCandidate { rawResult, ...(profile === undefined ? {} : { analysisProfile: profile }), ...(operation === "procedure_callers" || - operation === "procedure_callees" + operation === "procedure_callees" || + operation === "analyze_function" ? { relationshipCoverage: { status: "complete" as const } } : {}), limitations: [ ...(preparedImage === undefined ? [] : [ - "REA loaded a verified thin Mach-O slice from the original FAT64 source. Identity and file offsets refer to the original file. The owned temporary image and its Hopper document close together, including on MCP shutdown.", + "REA loaded a verified thin Mach-O slice from the original FAT source. Identity and file offsets refer to the original file. The owned temporary image and its Hopper document close together, including on MCP shutdown.", ]), ...(CAPABILITIES.find( (descriptor) => descriptor.operation === operation, diff --git a/src/hopper/HopperProviderDiscovery.test.ts b/src/hopper/HopperProviderDiscovery.test.ts index 82acf8888a..5df7eacad0 100644 --- a/src/hopper/HopperProviderDiscovery.test.ts +++ b/src/hopper/HopperProviderDiscovery.test.ts @@ -46,7 +46,7 @@ describe("Hopper provider discovery", () => { }); it("separates target kind, format, and architecture support", () => { - const hopper = provider(process.execPath); + const hopper = provider(process.execPath, "linux"); expect(hopper.inspectTargetSupport(databaseTarget())).toMatchObject({ status: "supported", }); @@ -59,12 +59,31 @@ describe("Hopper provider discovery", () => { diagnostics: { target_kind: "archive", target_format: "asar" }, }); }); + + it("fails closed for saved Hopper databases on macOS without a bridge launch route", () => { + expect( + provider(process.execPath, "darwin").inspectTargetSupport( + databaseTarget(), + ), + ).toMatchObject({ + status: "unsupported", + code: "target_kind_unsupported", + reason: expect.stringContaining("cannot start REA's analysis bridge"), + diagnostics: { + target_kind: "database", + target_format: "analysis-database", + }, + }); + }); }); -const provider = (launcherPath: string): HopperProvider => { +const provider = ( + launcherPath: string, + platform: NodeJS.Platform = process.platform, +): HopperProvider => { const config = parseConfig({ HOPPER_LAUNCHER_PATH: launcherPath }); if (!config.ok) throw config.error; - return new HopperProvider(config.value, silentLogger, {}); + return new HopperProvider(config.value, silentLogger, {}, platform); }; const databaseTarget = (): BinaryTarget => ({ diff --git a/src/hopper/HopperSessionValues.ts b/src/hopper/HopperSessionValues.ts index 86d15b1351..abee06e047 100644 --- a/src/hopper/HopperSessionValues.ts +++ b/src/hopper/HopperSessionValues.ts @@ -9,6 +9,7 @@ export const HOPPER_SHUTDOWN_TIMEOUT_MS = 30_000; export interface HopperServerInfo { readonly name: string; readonly version: string; + readonly processId: number; } /** Validate the bridge identity and exact per-launch run identifier. */ @@ -22,9 +23,16 @@ export const parseHopperServerInfo = ( !Array.isArray(value) && value.name === "REA Hopper bridge" && typeof value.version === "string" && + typeof value.process_id === "number" && + Number.isSafeInteger(value.process_id) && + value.process_id > 0 && value.run_id === expectedRunId ) - return ok({ name: value.name, version: value.version }); + return ok({ + name: value.name, + version: value.version, + processId: value.process_id, + }); return err(new HopperProtocolError("Hopper bridge health result is invalid")); }; diff --git a/src/hopper/HopperTargetLease.test.ts b/src/hopper/HopperTargetLease.test.ts index 2c72909fa2..87d8355029 100644 --- a/src/hopper/HopperTargetLease.test.ts +++ b/src/hopper/HopperTargetLease.test.ts @@ -1,6 +1,7 @@ import { execFile } from "node:child_process"; import { once } from "node:events"; import { + chmod, lstat, mkdir, mkdtemp, @@ -19,7 +20,7 @@ import { afterEach, describe, expect, it } from "vitest"; import { acquireHopperTargetLease, acquireLinuxHopperApplicationLease, -} from "./HopperTargetLease.js"; +} from "../process/HopperLease.js"; const directories: string[] = []; @@ -31,6 +32,84 @@ afterEach(async () => { ); }); +describe.skipIf(process.platform === "win32")( + "Hopper lease directory admission", + () => { + it("secures an existing owned directory without replacing it", async () => { + const directory = await temporaryDirectory(); + await chmod(directory, 0o755); + const before = await lstat(directory); + const acquired = await acquireLinuxHopperApplicationLease({ + directory, + runId: "permission-hardening", + }); + if (!acquired.acquired) + throw new Error("Expected the owned directory lease"); + try { + const after = await lstat(directory); + expect(after.dev).toBe(before.dev); + expect(after.ino).toBe(before.ino); + expect(after.mode & 0o777).toBe(0o700); + } finally { + await acquired.lease.release(); + } + }); + + it("rejects a directory symlink without changing its target", async () => { + const directory = await temporaryDirectory(); + const realDirectory = join(directory, "actual"); + const alias = join(directory, "selected"); + await mkdir(realDirectory); + await chmod(realDirectory, 0o755); + await symlink(realDirectory, alias); + await expect( + acquireLinuxHopperApplicationLease({ + directory: alias, + runId: "alias", + }), + ).rejects.toMatchObject({ _tag: "HopperStartError" }); + expect(await readdir(realDirectory)).toEqual([]); + expect((await lstat(realDirectory)).mode & 0o777).toBe(0o755); + expect((await lstat(alias)).isSymbolicLink()).toBe(true); + }); + + it.skipIf(process.getuid?.() === 0)( + "explains unreadable directory recovery without changing unverified permissions", + async () => { + const directory = await temporaryDirectory(); + await chmod(directory, 0o333); + try { + await expect( + acquireLinuxHopperApplicationLease({ + directory, + runId: "unreadable", + }), + ).rejects.toMatchObject({ + _tag: "HopperStartError", + cause: expect.objectContaining({ code: "EACCES" }), + userMessage: expect.stringContaining( + "restore owner-only read, write, and search permissions (0700)", + ), + }); + expect((await lstat(directory)).mode & 0o777).toBe(0o333); + // This fixture owns the directory. Apply the documented recovery, + // then exercise the same production acquisition again. + await chmod(directory, 0o700); + const recovered = await acquireLinuxHopperApplicationLease({ + directory, + runId: "recovered-access", + }); + if (!recovered.acquired) throw new Error("Expected recovered access"); + await recovered.lease.release(); + expect(await readdir(directory)).toEqual([]); + } finally { + await chmod(directory, 0o700); + } + }, + ); + }, +); + describe("Hopper target leases", () => { it("reports the owning REA session for a duplicate target and profile", async () => { const directory = await temporaryDirectory(); diff --git a/src/ida/IdaBusyHealth.test.ts b/src/ida/IdaBusyHealth.test.ts index b997b25916..47d59883e4 100644 --- a/src/ida/IdaBusyHealth.test.ts +++ b/src/ida/IdaBusyHealth.test.ts @@ -61,6 +61,34 @@ it("sleeps the fixed retry delay between busy reads and returns the settled answ expect(sleeps, "two fixed 100 ms pauses").toEqual([100, 100]); }); +it("returns an expired readiness timeout without starting a health read", async () => { + const timeoutMs = 10; + const { connection, reads } = scripted([ok]); + let clockReads = 0; + const clock = { + now: () => (clockReads++ === 0 ? 0 : 11), + delay: async () => { + throw new Error("An expired wait must not schedule a pause"); + }, + }; + + const failure = await readSettledHealth(connection, { + database: "db", + timeoutMs, + clock, + }).catch((cause: unknown) => cause); + + expect(failure).toMatchObject({ + _tag: "AnalysisTimeoutError", + operation: "server_health", + timeoutMs, + }); + expect( + reads, + "an expired budget never becomes an SDK timeout of zero", + ).toHaveLength(0); +}); + it("clamps the last pause to the remaining budget and stops without another read at expiry", async () => { const { connection, reads } = scripted([busy("config_json_get")]); const { clock, sleeps } = virtualClock(); diff --git a/src/ida/IdaBusyHealth.ts b/src/ida/IdaBusyHealth.ts index 144aa8e526..fdb02da474 100644 --- a/src/ida/IdaBusyHealth.ts +++ b/src/ida/IdaBusyHealth.ts @@ -29,6 +29,11 @@ const isBusyStatus = (raw: JsonValue): boolean => !Array.isArray(raw) && raw["status"] === "busy"; +interface LastBusyHealth { + readonly raw: JsonValue; + readonly operation: string; +} + /** * Reads `server_health` until the worker stops reporting a transient busy * state. The wait ends when the worker answers, the caller aborts, or the @@ -53,22 +58,62 @@ export const readSettledHealth = async ( ): Promise => { const started = clock.now(); const remaining = (): number => timeoutMs - (clock.now() - started); + let lastBusy: LastBusyHealth | undefined; for (;;) { - const raw = await connection.call("server_health", { database }); + const budgetMs = remaining(); + if (budgetMs <= 0) + throw lastBusy === undefined + ? new AnalysisTimeoutError("server_health", timeoutMs) + : busyTimeout(lastBusy, timeoutMs); + const requestTimeoutMs = Math.min(timeoutMs, Math.ceil(budgetMs)); + let raw: JsonValue; + try { + raw = await connection.call( + "server_health", + { database }, + { timeoutMs: requestTimeoutMs }, + ); + } catch (cause: unknown) { + if ( + lastBusy !== undefined && + cause instanceof AnalysisTimeoutError && + cause.timeoutMs === requestTimeoutMs && + requestTimeoutMs < timeoutMs + ) { + throw new AnalysisTimeoutError(lastBusy.operation, timeoutMs, { + cause: lastBusy.raw, + capturedOutput: cause.capturedOutput, + diagnostics: { + health_request_operation: cause.operation, + health_request_timeout_ms: cause.timeoutMs, + }, + }); + } + throw cause; + } const busy = busyHealthSchema.safeParse(raw); if (!busy.success) { if (isBusyStatus(raw)) throw busy.error; return raw; } + const observedBusy = { + raw, + operation: `server_health: headless worker still busy in ${busyToolName(busy.data.busy_tool)}`, + }; + lastBusy = observedBusy; const expired = (): AnalysisTimeoutError => - new AnalysisTimeoutError( - `server_health: headless worker still busy in ${busyToolName(busy.data.busy_tool)}`, - timeoutMs, - { cause: raw }, - ); + busyTimeout(observedBusy, timeoutMs); if (remaining() <= 0) throw expired(); await clock.delay(Math.min(BUSY_HEALTH_RETRY_MS, remaining()), signal); signal?.throwIfAborted(); if (remaining() <= 0) throw expired(); } }; + +const busyTimeout = ( + lastBusy: LastBusyHealth, + timeoutMs: number, +): AnalysisTimeoutError => + new AnalysisTimeoutError(lastBusy.operation, timeoutMs, { + cause: lastBusy.raw, + }); diff --git a/src/ida/IdaContainment.test.ts b/src/ida/IdaContainment.test.ts index 069388df82..f81d7b9cdf 100644 --- a/src/ida/IdaContainment.test.ts +++ b/src/ida/IdaContainment.test.ts @@ -148,7 +148,7 @@ const found = (query: string, fn: typeof function_) => ({ }); describe("IDA containment capabilities", () => { - it("declares both operations for the headless profile only", () => { + it("declares document identity in both profiles and address resolution only in headless", () => { const declared = (mode: "attached" | "headless", native = false) => idaCapabilities(mode, false, native) .map(({ operation }) => operation) @@ -172,35 +172,30 @@ describe("IDA containment capabilities", () => { "resolve_containing_procedure", "list_documents", ]); - expect(declared("attached")).toEqual([]); - expect(declared("attached", true)).toEqual([]); + expect(declared("attached")).toEqual(["list_documents"]); + expect(declared("attached", true)).toEqual(["list_documents"]); expect( idaCapabilities("attached", false).map(({ operation }) => operation), - "attached profiles do not even declare them", - ).not.toContain("list_documents"); + "attached profiles declare the bound document but not address resolution", + ).toContain("list_documents"); }); - it.each(["resolve_containing_procedure", "list_documents"] as const)( - "refuses %s on the legacy attached profile without calling upstream", - async (operation) => { - const { client, producer } = await session({}, "attached"); - try { - const result = await client.execute( - operation, - operation === "list_documents" ? {} : { address: "0x1000" }, - ); - - expect(result.ok).toBe(false); - if (!result.ok) - expect(result.error._tag).toBe("AnalysisCapabilityUnavailableError"); - expect(producer.calls.some(({ name }) => name === "lookup_funcs")).toBe( - false, - ); - } finally { - expect((await client.close()).ok).toBe(true); - } - }, - ); + it("refuses address resolution on the legacy attached profile", async () => { + const { client, producer } = await session({}, "attached"); + try { + const result = await client.execute("resolve_containing_procedure", { + address: "0x1000", + }); + expect(result.ok).toBe(false); + if (!result.ok) + expect(result.error._tag).toBe("AnalysisCapabilityUnavailableError"); + expect(producer.calls.some(({ name }) => name === "lookup_funcs")).toBe( + false, + ); + } finally { + expect((await client.close()).ok).toBe(true); + } + }); }); describe("headless IDA containing procedure", () => { diff --git a/src/ida/IdaDiagnosticRedactor.test.ts b/src/ida/IdaDiagnosticRedactor.test.ts new file mode 100644 index 0000000000..0c65e37d31 --- /dev/null +++ b/src/ida/IdaDiagnosticRedactor.test.ts @@ -0,0 +1,563 @@ +import { describe, expect, it } from "vitest"; +import { + createIdaDiagnosticRedactor, + IDA_DIAGNOSTIC_SOURCE_BYTES, +} from "./IdaDiagnosticRedactor.js"; + +describe("IDA diagnostic credential redaction", () => { + it.each([false, true])( + "redacts literal credentials once regardless of header order (reversed: %s)", + (reversed) => { + const entries = [ + ["Authorization", "Bearer fixture"], + ["proxy-AUTHORIZATION", "bAsIc fixture-long.$[token]"], + ["cOoKiE", "session=fixture-long.$[token]; marker=redacted"], + ["X-Evidence", "caller-selected-evidence"], + ]; + const redact = createIdaDiagnosticRedactor({ + url: "http://127.0.0.1:12345/mcp", + headers: Object.fromEntries(reversed ? entries.reverse() : entries), + mode: "attached", + timeoutMs: 1000, + }).text; + expect( + redact( + "Bearer fixture / fixture / bAsIc fixture-long.$[token] / fixture-long.$[token] / session=fixture-long.$[token]; marker=redacted /caller/input?token=visible API_KEY=visible caller-selected-evidence", + ), + ).toBe( + "␟ / ␟ / ␟ / ␟ / ␟ /caller/input?token=visible API_KEY=visible caller-selected-evidence", + ); + }, + ); + it("redacts configured and HTTP-normalized header values", () => { + const redact = createIdaDiagnosticRedactor({ + url: "http://127.0.0.1:12345/mcp", + headers: { + aUtHoRiZaTiOn: "\r\n \tBearer fixture-secret \t\r\n", + "pRoXy-AuThOrIzAtIoN": "\tBasic proxy-secret ", + CoOkIe: " session=cookie-secret\t", + }, + mode: "attached", + timeoutMs: 1000, + }).text; + expect( + redact( + "raw=\r\n \tBearer fixture-secret \t\r\n; wire=Bearer fixture-secret; token=fixture-secret; proxy=Basic proxy-secret; bare=proxy-secret; cookie=session=cookie-secret", + ), + ).toBe( + "raw=[redacted transport credential]; wire=[redacted transport credential]; token=[redacted transport credential]; proxy=[redacted transport credential]; bare=[redacted transport credential]; cookie=[redacted transport credential]", + ); + }); + it("does not treat empty HTTP authentication as diagnostic content", () => { + const redact = createIdaDiagnosticRedactor({ + url: "http://127.0.0.1:12345/mcp", + headers: { + Authorization: "\r\n \t \r\n", + "Proxy-Authorization": "", + Cookie: "\t", + }, + mode: "attached", + timeoutMs: 1000, + }).text; + const message = "Failed at /caller/input \t with ordinary whitespace"; + expect(redact(message)).toBe(message); + }); + it("does not reprocess the replacement marker as another credential", () => { + const redact = createIdaDiagnosticRedactor({ + url: "http://127.0.0.1:12345/mcp", + headers: { Authorization: "Bearer alpha", Cookie: "redacted" }, + mode: "attached", + timeoutMs: 1000, + }).text; + expect(redact("alpha redacted")).toBe("␟ ␟"); + }); +}); + +const credentialCases: readonly { + readonly name: string; + readonly headers: Readonly>; + readonly message: string; + readonly expected: string; +}[] = [ + { + name: "duplicate cookie names, quoted values, and equals padding", + headers: { + Cookie: + 'session=first-secret; session=second-secret; signed="padded=value=="', + }, + message: "first-secret / second-secret / padded=value== / session signed", + expected: + "[redacted transport credential] / [redacted transport credential] / [redacted transport credential] / session signed", + }, + { + name: "empty cookie and auth-param values", + headers: { Cookie: 'a=; b=""', Authorization: 'Custom empty=""' }, + message: 'Ordinary spaces and "" remain; a b empty are names', + expected: 'Ordinary spaces and "" remain; a b empty are names', + }, + { + name: "opaque token68 and explicit one-character credentials", + headers: { Authorization: "Negotiate opaque+/==", Cookie: "a=Q" }, + message: "opaque+/== Q Negotiate a", + expected: + "[redacted transport credential] [redacted transport credential] Negotiate a", + }, + { + name: "auth-param values retain quoted commas and decode quoted pairs", + headers: { + Authorization: String.raw`Digest first="nonce,part", response="quote\"part", opaque="back\\part", tail=last-secret`, + }, + message: String.raw`nonce,part / quote"part / quote\"part / back\part / back\\part / last-secret / Digest first response opaque tail`, + expected: + "[redacted transport credential] / [redacted transport credential] / [redacted transport credential] / [redacted transport credential] / [redacted transport credential] / [redacted transport credential] / Digest first response opaque tail", + }, + { + name: "complete parameters before a malformed quoted member", + headers: { + Authorization: + 'Digest first=known-secret, broken="unterminated, second=visible', + }, + message: "known-secret visible broken second", + expected: "[redacted transport credential] visible broken second", + }, + { + name: "unclosed quotes do not create parameters from their contents", + headers: { Authorization: 'Digest broken="unterminated, second=visible' }, + message: "unterminated visible broken second", + expected: "unterminated visible broken second", + }, + { + name: "malformed cookie members do not hide valid later pairs", + headers: { + Cookie: 'broken; =not-a-value; malformed="unclosed; session=valid-secret', + }, + message: "valid-secret broken not-a-value unclosed session", + expected: + "[redacted transport credential] broken not-a-value unclosed session", + }, + { + name: "cookie octets remain opaque without URL decoding", + headers: { Cookie: "session=encoded%2Fcredential" }, + message: "encoded%2Fcredential / encoded/credential", + expected: "[redacted transport credential] / encoded/credential", + }, +]; +it.each(credentialCases)( + "parses $name at the configured credential boundary", + ({ headers, message, expected }) => { + const redact = createIdaDiagnosticRedactor({ + url: "http://127.0.0.1:12345/mcp", + headers, + mode: "attached", + timeoutMs: 1000, + }).text; + expect(redact(message)).toBe(expected); + }, +); + +it("redacts full configured malformed headers without guessing their internal values", () => { + const header = 'Digest broken="unterminated, second=visible'; + const redactor = createIdaDiagnosticRedactor({ + url: "http://127.0.0.1:12345/mcp", + headers: { Authorization: header }, + mode: "attached", + timeoutMs: 1000, + }); + expect(redactor.text(header)).toBe("[redacted transport credential]"); + expect(redactor.json(JSON.stringify(header))).toBe( + '"[redacted transport credential]"', + ); +}); + +const marker = "[redacted transport credential]"; +const redactorFor = (header: string, name = "Authorization") => + createIdaDiagnosticRedactor({ + url: "http://127.0.0.1:12345/mcp", + headers: { [name]: header }, + mode: "attached", + timeoutMs: 1000, + }); + +describe("IDA Basic authentication payload components", () => { + it.each(["aUtHoRiZaTiOn", "pRoXy-AuThOrIzAtIoN"])( + "protects decoded Basic pairs and passwords while preserving user names for %s", + (name) => { + const pair = "fixture-user:fixture-password:tail"; + const encoded = Buffer.from(pair, "utf8").toString("base64"); + const redactor = redactorFor("bAsIc " + encoded, name); + const reason = + "pair=" + + pair + + "; user=fixture-user; password=fixture-password:tail; wire=" + + encoded; + const safe = + "pair=" + + marker + + "; user=fixture-user; password=" + + marker + + "; wire=" + + marker; + expect(redactor.text(reason + "; path=/caller/input?token=visible")).toBe( + safe + "; path=/caller/input?token=visible", + ); + expect( + redactor.json(JSON.stringify({ reason, path: "/caller/input" })), + ).toBe(JSON.stringify({ reason: safe, path: "/caller/input" })); + }, + ); + + it.each([ + [ + ":fixture-password", + "pair=:fixture-password; password=fixture-password; empty values leave spaces", + "pair=" + marker + "; password=" + marker + "; empty values leave spaces", + ], + [ + "fixture-user:", + "pair=fixture-user:; user=fixture-user; empty values leave spaces", + "pair=" + marker + "; user=fixture-user; empty values leave spaces", + ], + [ + ":", + "pair=:; empty values leave spaces", + "pair=" + marker + "; empty values leave spaces", + ], + ])( + "protects a decoded pair with empty components: %s", + (pair, reason, safe) => { + const encoded = Buffer.from(pair, "utf8").toString("base64"); + expect(redactorFor("Basic " + encoded).text(reason)).toBe(safe); + }, + ); +}); + +describe("IDA Basic authentication charset and invalid data", () => { + it("retains both strict UTF-8 and exact octet-compatible interpretations when charset is unknown", () => { + const pair = "tést-user:päss£word"; + const bytes = Buffer.from(pair, "utf8"); + const octets = bytes.toString("latin1"); + const redactor = redactorFor("Basic " + bytes.toString("base64")); + const reason = + "utf8=" + + pair + + "; octets=" + + octets + + "; user=tést-user; password=päss£word; octet-user=tést-user; octet-password=päss£word; caller=/caller/input"; + expect(redactor.text(reason)).toBe( + "utf8=" + + marker + + "; octets=" + + marker + + "; user=tést-user; password=" + + marker + + "; octet-user=tést-user; octet-password=" + + marker + + "; caller=/caller/input", + ); + }); + + it("preserves exact Latin-1 octets without inventing lossy UTF-8 replacement credentials", () => { + const bytes = Buffer.from("latin-user:pässword", "latin1"); + const redactor = redactorFor("Basic " + bytes.toString("base64")); + expect( + redactor.text( + "latin-user:pässword / latin-user / pässword / p\ufffdssword /caller/input", + ), + ).toBe( + marker + " / latin-user / " + marker + " / p\ufffdssword /caller/input", + ); + }); + + it("preserves a UTF-8 BOM as credential data rather than dropping it", () => { + const pair = "\ufefffixture-user:fixture-password"; + const encoded = Buffer.from(pair, "utf8").toString("base64"); + expect( + redactorFor("Basic " + encoded).text(pair + " / \ufefffixture-user"), + ).toBe(marker + " / \ufefffixture-user"); + }); + + it("keeps decoded Basic values opaque without URL decoding", () => { + const encoded = Buffer.from("fixture-user:p%2Fsecret", "utf8").toString( + "base64", + ); + expect(redactorFor("Basic " + encoded).text("p%2Fsecret / p/secret")).toBe( + marker + " / p/secret", + ); + }); + + const canonical = Buffer.from( + "fixture-user:fixture-password", + "utf8", + ).toString("base64"); + const alphabet = + "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; + const nonzeroPadBits = canonical.replace( + /(.)(=)$/u, + (_match, symbol: string, padding: string) => + (alphabet[alphabet.indexOf(symbol) + 1] ?? "!") + padding, + ); + it.each([ + { name: "unpadded", payload: canonical.slice(0, -1) }, + { name: "at most two padding characters", payload: canonical + "=" }, + ])( + "accepts the supported normalized Basic Base64 spelling: $name", + ({ payload }) => { + const redactor = redactorFor("Basic " + payload); + expect( + redactor.text( + "fixture-user:fixture-password / fixture-password / fixture-user", + ), + ).toBe(marker + " / " + marker + " / fixture-user"); + }, + ); + + it.each([ + { + name: "non-alphabet character", + payload: canonical.slice(0, 4) + "!" + canonical.slice(4), + }, + { + name: "embedded whitespace", + payload: canonical.slice(0, 4) + " " + canonical.slice(4), + }, + { name: "more than two pad characters", payload: canonical + "==" }, + { name: "nonzero pad bits", payload: nonzeroPadBits }, + { + name: "missing colon separator", + payload: Buffer.from("fixture-user", "utf8").toString("base64"), + }, + { + name: "ASCII control in user-pass", + payload: Buffer.from("fixture-user:fixture-password\n", "utf8").toString( + "base64", + ), + }, + { + name: "auth-param-shaped malformed Basic data", + payload: "credential=fixture-password", + }, + ])("retains only opaque configured Basic data for $name", ({ payload }) => { + const header = "Basic " + payload; + const redactor = redactorFor(header); + expect(redactor.text(header + " / " + payload)).toBe( + marker + " / " + marker, + ); + expect(redactor.text("fixture-user / fixture-password /caller/input")).toBe( + "fixture-user / fixture-password /caller/input", + ); + }); + + it("does not decode another scheme's base64-shaped opaque payload", () => { + const encoded = Buffer.from( + "fixture-user:fixture-password", + "utf8", + ).toString("base64"); + const redactor = redactorFor("Custom " + encoded); + expect(redactor.text(encoded + " / fixture-user / fixture-password")).toBe( + marker + " / fixture-user / fixture-password", + ); + }); +}); + +it("redacts protocol-decoded Basic credentials and password without treating ordinary names as secrets", () => { + const redactor = createIdaDiagnosticRedactor({ + url: "http://127.0.0.1:12345/mcp", + mode: "attached", + timeoutMs: 1000, + headers: { + Authorization: `Basic ${Buffer.from("user:secret").toString("base64")}`, + }, + }); + expect(redactor.text("user:secret / secret / user")).toBe( + "[redacted transport credential] / [redacted transport credential] / user", + ); + expect(redactor.json('"\\u0073ecret"')).toBe( + '"[redacted transport credential]"', + ); +}); + +it.each(["text", "json"] as const)( + "bounds short-secret diagnostic expansion before %s redaction", + (format) => { + const redactor = createIdaDiagnosticRedactor({ + url: "http://127.0.0.1:12345/mcp", + mode: "attached", + timeoutMs: 1000, + headers: { Cookie: "a=Q" }, + }); + const result = redactor[format]("Q".repeat(2 * 1024 * 1024)); + expect(result).toBe("[IDA diagnostic omitted: exceeds 65536 UTF-8 bytes]"); + }, +); + +it.each(["text", "json"] as const)( + "applies the UTF-8 source-byte budget before %s processing even without credentials", + (format) => { + const atBudget = "£".repeat(IDA_DIAGNOSTIC_SOURCE_BYTES / 2); + const overBudget = atBudget + "£"; + const configuredHeaders: readonly Readonly>[] = [ + {}, + { Cookie: "a=Q" }, + ]; + for (const headers of configuredHeaders) { + const redactor = createIdaDiagnosticRedactor({ + url: "http://127.0.0.1:12345/mcp", + mode: "attached", + timeoutMs: 1000, + headers, + }); + expect(redactor[format](atBudget)).toBe(atBudget); + expect(redactor[format](overBudget)).toBe( + "[IDA diagnostic omitted: exceeds 65536 UTF-8 bytes]", + ); + } + }, +); + +it.each(["dXNlcjpzZWNyZXQ=", "dXNlcjpzZWNyZXQ", "dXNlcjpzZWNyZXQ=="])( + "redacts accepted canonical Basic spellings for %s", + (payload) => { + const redactor = createIdaDiagnosticRedactor({ + url: "http://127.0.0.1:12345/mcp", + mode: "attached", + timeoutMs: 1000, + headers: { Authorization: `Basic ${payload}` }, + }); + const marker = "[redacted transport credential]"; + expect( + redactor.text( + "dXNlcjpzZWNyZXQ= / dXNlcjpzZWNyZXQ / user:secret / secret", + ), + ).toBe(`${marker} / ${marker} / ${marker} / ${marker}`); + }, +); + +describe("overlapping IDA credential source spans", () => { + it.each([false, true])( + "unions all overlapping plain credentials before writing (reversed: %s)", + (reversed) => { + const entries = [ + ["Authorization", "Custom X]"], + ["Cookie", "a=]SECRET"], + ]; + const redactor = createIdaDiagnosticRedactor({ + url: "http://127.0.0.1:12345/mcp", + headers: Object.fromEntries(reversed ? entries.reverse() : entries), + mode: "attached", + timeoutMs: 1000, + }); + const result = redactor.text( + "Rejected X]SECRET at /caller/input?cursor=visible", + ); + expect(result).toBe("Rejected ␟ at /caller/input?cursor=visible"); + expect(result).not.toContain("]SECRET"); + }, + ); + + it("unions a chain of different overlapping credentials", () => { + const redactor = createIdaDiagnosticRedactor({ + url: "http://127.0.0.1:12345/mcp", + headers: { Authorization: "Custom abc", Cookie: "a=cde; b=def" }, + mode: "attached", + timeoutMs: 1000, + }); + expect(redactor.text("prefix abcdefg suffix")).toBe( + "prefix " + marker + "g suffix", + ); + }); + + it("merges dense same-credential overlaps without retaining a per-match collection", () => { + const redactor = redactorFor("Custom AA"); + const result = redactor.text("A".repeat(IDA_DIAGNOSTIC_SOURCE_BYTES)); + expect(result.length).toBe(marker.length); + expect(result).toBe(marker); + }); + + it("retains adjacent disjoint credential occurrences", () => { + expect(redactorFor("Custom AA").text("AA-AA")).toBe(marker + "-" + marker); + }); +}); + +describe("credential-safe redaction markers", () => { + it.each([ + { secret: "]SECRET", source: "XSECRET", expected: "␟SECRET" }, + { secret: "SECRET[", source: "SECRETX", expected: "SECRET␟" }, + { + secret: "a[redacted transport credential]b", + source: "aXb", + expected: "a␟b", + }, + { secret: "redacted", source: "redacted", expected: "␟" }, + ])( + "does not create $secret from replacement text or boundaries", + ({ secret, source, expected }) => { + const redactor = createIdaDiagnosticRedactor({ + url: "http://127.0.0.1:12345/mcp", + mode: "attached", + timeoutMs: 1000, + headers: { Authorization: "Custom " + secret, Cookie: "a=X" }, + }); + expect(redactor.text(source)).toBe(expected); + expect( + redactor.json( + JSON.stringify({ reason: source, context: "/caller/input" }), + ), + ).toBe(JSON.stringify({ reason: expected, context: "/caller/input" })); + expect(redactor.json('"' + source + '",x')).toBe('"' + expected + '",x'); + }, + ); + it("chooses a different graphic scalar when a decoded Basic credential contains the first marker", () => { + const redactor = createIdaDiagnosticRedactor({ + url: "http://127.0.0.1:12345/mcp", + mode: "attached", + timeoutMs: 1000, + headers: { + Authorization: + "Basic " + Buffer.from("user:␟", "utf8").toString("base64"), + Cookie: "a=]SECRET", + }, + }); + expect(redactor.text("␟ at /caller/input")).toBe("␠ at /caller/input"); + expect(redactor.json(String.raw`{"reason":"␟","context":"kept"}`)).toBe( + '{"reason":"␠","context":"kept"}', + ); + }); +}); + +describe("unclassified diagnostic blocks and message ownership", () => { + it.each>>([{}, { Cookie: "a=Q" }])( + "budgets the entire source block list before redaction: %s", + (headers) => { + const redactor = createIdaDiagnosticRedactor({ + url: "http://127.0.0.1:12345/mcp", + mode: "attached", + timeoutMs: 1000, + headers, + }); + const first = "£".repeat(IDA_DIAGNOSTIC_SOURCE_BYTES / 4); + expect(redactor.blocks([first, first])).toBe( + "[IDA diagnostic omitted: exceeds 65536 UTF-8 bytes]", + ); + expect(redactor.blocks([first, first.slice(1)])).toBe( + first + "\n" + first.slice(1), + ); + }, + ); + it("keeps complete quoted plain credentials while preserving explicit JSON semantics", () => { + const redactor = redactorFor(String.raw`Custom folder\tmp`); + const source = String.raw`"folder\tmp at /caller/input"`; + expect(redactor.blocks([source])).toBe( + '"[redacted transport credential] at /caller/input"', + ); + expect(redactor.json(source)).toBe(source); + }); + it("sanitizes the omission notice as diagnostic text", () => { + const redactor = redactorFor("Custom omitted"); + expect( + redactor.blocks(["Q".repeat(IDA_DIAGNOSTIC_SOURCE_BYTES + 1)]), + ).toContain("[redacted transport credential]"); + expect( + redactor.text("Q".repeat(IDA_DIAGNOSTIC_SOURCE_BYTES + 1)), + ).not.toContain("omitted"); + }); +}); diff --git a/src/ida/IdaDiagnosticRedactor.ts b/src/ida/IdaDiagnosticRedactor.ts new file mode 100644 index 0000000000..d332d15488 --- /dev/null +++ b/src/ida/IdaDiagnosticRedactor.ts @@ -0,0 +1,243 @@ +import { + redactIdaJsonDiagnostic, + redactIdaUnclassifiedDiagnostic, +} from "./IdaJsonDiagnostic.js"; +import { + IDA_REDACTION_MARKER, + joinIdaDiagnosticParts, + writeIdaDiagnostic, + type IdaDiagnosticMatch, +} from "./IdaDiagnosticWriter.js"; +import type { IdaConfiguration } from "./IdaConfiguration.js"; +import { AnalysisProtocolError } from "../domain/analysisErrorCore.js"; + +// Diagnostic text is bounded before decoding or replacement expansion. +export const IDA_DIAGNOSTIC_SOURCE_BYTES = 64 * 1024; +const omittedDiagnostic = "[IDA diagnostic omitted: exceeds 65536 UTF-8 bytes]"; + +const boundedDiagnostic = + (redact: (message: string) => string) => + (message: string): string => + Buffer.byteLength(message, "utf8") > IDA_DIAGNOSTIC_SOURCE_BYTES + ? redact(omittedDiagnostic) + : redact(message); + +/** RFC 7617 user-pass candidates; the configured request carries no charset. */ +const basicPayloads = (payload: string): readonly string[] => { + if (!/^[A-Za-z\d+/]+={0,2}$/u.test(payload)) return []; + const octets = Buffer.from(payload, "base64"); + const canonical = octets.toString("base64"); + // Match the supported padded/unpadded policy without accepting Buffer's + // permissive alphabet or discarding nonzero pad bits. + if ( + canonical.replace(/=+$/u, "") !== payload.replace(/=+$/u, "") || + octets.indexOf(0x3a) < 0 + ) + return []; + // Basic user-id and password exclude ASCII control characters. + if (octets.some((value) => value < 0x20 || value === 0x7f)) return []; + const candidates = [octets.toString("latin1")]; + try { + candidates.push( + new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }).decode(octets), + ); + } catch { + // Invalid UTF-8 has no decoded Unicode candidate. Exact Latin-1 preserves + // each octet; do not invent a replacement-character credential. + } + // The default charset is unknown. Retain both supported interpretations + // rather than selecting one or normalizing their configured source bytes. + return [ + canonical, + canonical.replace(/=+$/u, ""), + ...candidates.flatMap((pair) => { + const separator = pair.indexOf(":"); + // The complete pair and password are credentials; a bare user-id is not. + return [pair, pair.slice(separator + 1)].filter( + (value) => value.length > 0, + ); + }), + ]; +}; + +/** RFC 9110 credentials: any scheme, followed by opaque data or auth-params. */ +const authorizationPayloads = (header: string): readonly string[] => { + const credentials = header.match( + /^([!#$%&'*+.^_`|~\dA-Za-z-]+)[ \t]+([\s\S]+)$/u, + ); + const payload = credentials?.[2]; + if (payload === undefined) return []; + if (credentials?.[1]?.toLowerCase() === "basic") + return [payload, ...basicPayloads(payload)]; + const values = [payload]; + // A sticky parse cannot reinterpret commas inside an incomplete quoted value. + // Keep complete parameters preceding malformed input; the opaque payload above + // remains a known credential even when the parameter grammar is not satisfied. + const parameter = + /[ \t,]*[!#$%&'*+.^_`|~\dA-Za-z-]+[ \t]*=[ \t]*("(?:[^"\\\r\n]|\\[^\r\n])*"|[!#$%&'*+.^_`|~\dA-Za-z-]+)[ \t]*(?:,|$)/uy; + let cursor = 0; + while (cursor < payload.length) { + parameter.lastIndex = cursor; + const match = parameter.exec(payload); + if (match === null) break; + cursor = parameter.lastIndex; + const value = match[1] ?? ""; + if (!value.startsWith('"')) { + values.push(value); + continue; + } + const quoted = value.slice(1, -1); + if (quoted.length > 0) + values.push(value, quoted, quoted.replace(/\\([^\r\n])/gu, "$1")); + } + return values; +}; + +/** RFC 6265 cookie-pairs: retain duplicates and '=' inside values. */ +const cookiePayloads = (header: string): readonly string[] => + header.split(";").flatMap((pair) => { + const value = pair.match( + /^[ \t]*[!#$%&'*+.^_`|~\dA-Za-z-]+[ \t]*=[ \t]*(.*?)[ \t]*$/u, + )?.[1]; + if (value === undefined) return []; + const payload = + value.startsWith('"') && value.endsWith('"') ? value.slice(1, -1) : value; + // Cookie quotes wrap cookie-octets; they do not introduce escape syntax. + if (!/^[\x21\x23-\x2b\x2d-\x3a\x3c-\x5b\x5d-\x7e]+$/u.test(payload)) + return []; + return [value, payload]; + }); + +/** Choose one marker for this credential owner, including replacement boundaries. */ +const redactionMarker = (secrets: readonly string[]): string => { + // Crossing either edge of the default marker requires a square bracket. + // A credential wholly inside the marker must also be excluded. + if ( + secrets.every( + (secret) => + !secret.includes("[") && + !secret.includes("]") && + !IDA_REDACTION_MARKER.includes(secret), + ) + ) + return IDA_REDACTION_MARKER; + const characters = new Set(); + for (const secret of secrets) + for (const character of secret) characters.add(character); + // A graphic scalar absent from every credential cannot contain or connect + // a credential, and can be written verbatim inside a JSON string. + const graphic = /[\p{L}\p{N}\p{P}\p{S}]/u; + for (let point = 0x241f; point <= 0x10ffff; point++) { + if (point >= 0xd800 && point <= 0xdfff) continue; + const character = String.fromCodePoint(point); + if (graphic.test(character) && !characters.has(character)) return character; + } + throw new AnalysisProtocolError( + "Configured IDA credentials leave no available diagnostic redaction character.", + ); +}; + +/** Own configured credential interpretation for all IDA error diagnostics. */ +export const createIdaDiagnosticRedactor = ( + config?: IdaConfiguration, +): Readonly<{ + text(message: string): string; + json(message: string): string; + blocks(messages: readonly string[]): string; + message(prefix: string, ownedBody: string): string; +}> => { + const headers = + config !== undefined && "headers" in config + ? Object.entries(config.headers) + : []; + const values = headers.flatMap(([name, value]) => { + if (!/^(?:authorization|proxy-authorization|cookie)$/iu.test(name)) + return []; + // Fetch normalizes surrounding HTTP whitespace before sending headers. + const normalized = value.replace(/^[ \t\r\n]+|[ \t\r\n]+$/gu, ""); + if (normalized.length === 0) return []; + return [ + value, + normalized, + ...(name.toLowerCase() === "cookie" + ? cookiePayloads(normalized) + : authorizationPayloads(normalized)), + ]; + }); + const secrets = [...new Set(values.filter((value) => value.length > 0))].sort( + (left, right) => right.length - left.length, + ); + const replacement = + secrets.length === 0 ? IDA_REDACTION_MARKER : redactionMarker(secrets); + const alternatives = secrets + .map((secret) => secret.replace(/[.*+?^${}()|[\]\\]/gu, "\\$&")) + .join("|"); + // Look ahead at each source position; a global consuming match loses overlap. + // Longest-first alternatives cover every shorter match at the same position. + const pattern = + alternatives.length === 0 + ? undefined + : new RegExp("(?=(" + alternatives + "))", "gu"); + const matches = function* (value: string): Iterable { + if (pattern === undefined) return; + let pending: IdaDiagnosticMatch | undefined; + for (const match of value.matchAll(pattern)) { + const credential = match[1]; + if (credential === undefined || credential.length === 0) + throw new AnalysisProtocolError( + "IDA diagnostic matching omitted its credential span.", + ); + const span = { start: match.index, end: match.index + credential.length }; + if (pending !== undefined && span.start < pending.end) { + pending = { + start: pending.start, + end: Math.max(pending.end, span.end), + }; + } else { + if (pending !== undefined) yield pending; + pending = span; + } + } + if (pending !== undefined) yield pending; + }; + const text = boundedDiagnostic((message) => + writeIdaDiagnostic(message, matches(message), replacement), + ); + const json = boundedDiagnostic((message) => + secrets.length === 0 + ? message + : redactIdaJsonDiagnostic(message, matches, replacement), + ); + return { + text, + json, + blocks(messages) { + let bytes = 0; + let sourceUnits = 0; + for (let index = 0; index < messages.length; index++) { + const value = messages[index]!; + bytes += Buffer.byteLength(value, "utf8") + (index === 0 ? 0 : 1); + sourceUnits += value.length + (index === 0 ? 0 : 1); + if (bytes > IDA_DIAGNOSTIC_SOURCE_BYTES) return text(omittedDiagnostic); + } + function* parts(): Generator { + for (let index = 0; index < messages.length; index++) { + if (index > 0) yield "\n"; + const value = messages[index]!; + yield secrets.length === 0 || value.length === 0 + ? value + : redactIdaUnclassifiedDiagnostic(value, matches, replacement); + } + } + return joinIdaDiagnosticParts(parts(), sourceUnits); + }, + message(prefix, ownedBody) { + // Normalized wire headers and supported decoded candidates have no LF. + // Preserve the already-owned body without inspecting its replacement text. + return joinIdaDiagnosticParts( + [text(prefix), "\n", ownedBody], + prefix.length + 1 + ownedBody.length, + ); + }, + }; +}; diff --git a/src/ida/IdaDiagnosticWriter.ts b/src/ida/IdaDiagnosticWriter.ts new file mode 100644 index 0000000000..7507405074 --- /dev/null +++ b/src/ida/IdaDiagnosticWriter.ts @@ -0,0 +1,85 @@ +import { constants } from "node:buffer"; +import { AnalysisResourceConstraintError } from "../domain/analysisErrorCore.js"; +import { bufferedJsonParts } from "../domain/jsonSerialization.js"; + +export const IDA_REDACTION_MARKER = "[redacted transport credential]"; +export type IdaDiagnosticMatch = Readonly<{ start: number; end: number }>; + +type DiagnosticRepresentation = "redacted-output" | "json-character-run"; + +/** An owned diagnostic cannot be represented as one runtime string. */ +export class IdaDiagnosticResourceError extends AnalysisResourceConstraintError { + constructor( + inputCodeUnits: number, + representedCodeUnits: number, + representation: DiagnosticRepresentation = "redacted-output", + ) { + super( + "ida_diagnostic_redaction", + "memory", + representation === "redacted-output" + ? "Redacted IDA diagnostics exceed the runtime's maximum string length." + : "Temporary IDA diagnostic JSON character decoding exceeds the runtime's maximum string length.", + { + boundary: "ida-diagnostic-redaction", + representation, + input_code_units: inputCodeUnits, + ...(representation === "redacted-output" + ? { output_code_units_at_least: representedCodeUnits } + : { json_decoding_code_units_at_least: representedCodeUnits }), + max_string_code_units: constants.MAX_STRING_LENGTH, + }, + ); + } +} + +/** Account for the complete representation before allocating its final string. */ +export const joinIdaDiagnosticParts = ( + parts: Iterable, + inputCodeUnits: number, +): string => { + function* checkedParts(): Generator { + let outputCodeUnits = 0; + for (const part of parts) { + outputCodeUnits += part.length; + if (outputCodeUnits > constants.MAX_STRING_LENGTH) + throw new IdaDiagnosticResourceError(inputCodeUnits, outputCodeUnits); + yield part; + } + } + return [...bufferedJsonParts(checkedParts())].join(""); +}; + +/** Write ordered source spans without retaining a per-match collection. */ +export const writeIdaDiagnostic = ( + text: string, + matches: Iterable, + replacement: string = IDA_REDACTION_MARKER, +): string => { + function* parts(): Generator { + let cursor = 0; + let pending: IdaDiagnosticMatch | undefined; + for (const match of matches) { + if (pending !== undefined && match.start < pending.end) { + pending = { + start: pending.start, + end: Math.max(pending.end, match.end), + }; + continue; + } + if (pending !== undefined) { + yield text.slice(cursor, pending.start); + yield replacement; + cursor = pending.end; + } + pending = match; + } + if (pending !== undefined) { + yield text.slice(cursor, pending.start); + yield replacement; + cursor = pending.end; + } + yield text.slice(cursor); + } + return joinIdaDiagnosticParts(parts(), text.length); +}; diff --git a/src/ida/IdaDoctor.ts b/src/ida/IdaDoctor.ts index 750ca757ec..8a711aff14 100644 --- a/src/ida/IdaDoctor.ts +++ b/src/ida/IdaDoctor.ts @@ -52,13 +52,16 @@ const inspectPrivateWorkspace = (): DoctorProviderCheck => { return { name: "private_workspace", ok: privacy.available, - code: privacy.available ? null : "unsupported_host", + code: privacy.available ? null : privacy.code, detail: privacy.available ? `Private workspace authority is available (${privacy.proof}).` : privacy.reason, - remediation: privacy.available - ? null - : "Use the matching Windows x64 REA package and a writable local NTFS temporary directory.", - classification: "unsupported_host", + remediation: privacy.available ? null : privacy.remediation, + classification: + !privacy.available && privacy.code === "runtime_missing" + ? "missing_dependency" + : !privacy.available && privacy.code === "runtime_unavailable" + ? "runtime_failure" + : "unsupported_host", }; }; diff --git a/src/ida/IdaJsonDiagnostic.test.ts b/src/ida/IdaJsonDiagnostic.test.ts new file mode 100644 index 0000000000..3929551d33 --- /dev/null +++ b/src/ida/IdaJsonDiagnostic.test.ts @@ -0,0 +1,337 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; +import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; +import { + IdaDiagnosticResourceError, + writeIdaDiagnostic, +} from "./IdaDiagnosticWriter.js"; +import { createIdaDiagnosticRedactor } from "./IdaDiagnosticRedactor.js"; +import { redactIdaJsonDiagnostic } from "./IdaJsonDiagnostic.js"; + +const runtime = vi.hoisted(() => ({ limit: undefined as number | undefined })); +vi.mock("node:buffer", async (importOriginal) => { + const original = await importOriginal(); + return { + ...original, + constants: { + ...original.constants, + get MAX_STRING_LENGTH() { + return runtime.limit ?? original.constants.MAX_STRING_LENGTH; + }, + }, + }; +}); +afterEach(() => { + runtime.limit = undefined; +}); + +const marker = "[redacted transport credential]"; +const forCredential = (credential: string) => + createIdaDiagnosticRedactor({ + url: "http://127.0.0.1:12345/mcp", + headers: { Authorization: `Custom ${credential}` }, + mode: "attached", + timeoutMs: 1000, + }); + +describe("IDA JSON diagnostic source preservation", () => { + it("matches equivalent JSON encodings without enumerating credential spellings", () => { + const redactor = forCredential(""); + const encodings = [ + ["<", String.raw`\u003c`, String.raw`\u003C`], + ["s", String.raw`\u0073`], + ["/", String.raw`\/`, String.raw`\u002f`, String.raw`\u002F`], + [">", String.raw`\u003e`, String.raw`\u003E`], + ].reduce( + (prefixes, units) => + prefixes.flatMap((prefix) => units.map((unit) => prefix + unit)), + [""], + ); + for (const encoded of encodings) + expect(redactor.json(`{"reason":"${encoded}"}`)).toBe( + `{"reason":"${marker}"}`, + ); + }); + + it("preserves unrelated spelling inside a matching string", () => { + const source = String.raw`{"reason":"\u003cpublic\u003e\/asset \u003csecret\u003e \ud83d\uDE00"}`; + expect(forCredential("").json(source)).toBe( + String.raw`{"reason":"\u003cpublic\u003e\/asset ` + + marker + + String.raw` \ud83d\uDE00"}`, + ); + }); + + it("retains duplicate fields, whitespace and exact numeric lexemes", () => { + const source = + '{\n "\\u003csecret\\u003e": 9007199254740993, "n": -0, "n": 1.00e+03, "value": ""\n}'; + expect(forCredential("").json(source)).toBe( + `{\n "${marker}": 9007199254740993, "n": -0, "n": 1.00e+03, "value": "${marker}"\n}`, + ); + }); + + it.each([ + { credential: "a😀b", encoded: String.raw`a\ud83d\uDE00b` }, + { credential: "a\ud800b", encoded: String.raw`a\uD800b` }, + { credential: "a\nb", encoded: String.raw`a\nb` }, + { credential: 'a"b', encoded: String.raw`a\u0022b` }, + ])("maps UTF-16 source spans for $encoded", ({ credential, encoded }) => { + expect(forCredential(credential).json(`"before ${encoded} after"`)).toBe( + `"before ${marker} after"`, + ); + }); + + it.each([ + { + source: String.raw`{broken: "\u003csecret\u003e",`, + expected: `{broken: "${marker}",`, + }, + { + source: String.raw`{"x":"\u003csecret\u003e`, + expected: `{"x":"${marker}`, + }, + { + source: String.raw`{"x":"\u003csecret\u003e\q"}`, + expected: `{"x":"${marker}\\q"}`, + }, + ])( + "preserves malformed source around known complete characters: $source", + ({ source, expected }) => { + expect(forCredential("").json(source)).toBe(expected); + }, + ); + + it("does not join decoded fragments across invalid escapes", () => { + const source = String.raw`{"x":""}`; + expect(forCredential("").json(source)).toBe(source); + }); + + it("retains literal matches crossing invalid JSON units", () => { + expect( + forCredential(String.raw`foo\qbar`).json(String.raw`{"x":"foo\qbar"}`), + ).toBe(`{"x":"${marker}"}`); + }); + + it.each( + ["b", "f", "n", "r", "t", "/", '"', "\\", "u0061"].map( + (escape) => "folder\\" + escape + "tmp", + ), + )( + "retains literal credentials in ambiguous malformed fragments: %s", + (credential) => { + const redactor = forCredential(credential); + for (const [before, after] of [ + ["Rejected ", ""], + ['Rejected "', '"'], + ['{"reason":"Rejected ', ""], + ['{"reason":"Rejected ', String.raw`\q"}`], + ]) + expect(redactor.json(before + credential + after)).toBe( + before + marker + after, + ); + expect(redactor.json('"' + credential + '"')).toBe( + '"' + credential + '"', + ); + expect(redactor.json(JSON.stringify("Rejected " + credential))).toBe( + JSON.stringify("Rejected " + marker), + ); + }, + ); + + it("uses decoded values instead of matching unrelated escape syntax", () => { + const source = String.raw`{"x":"\u003cpublic\u003e","y":"u003c"}`; + expect(forCredential("u003c").json(source)).toBe( + String.raw`{"x":"\u003cpublic\u003e","y":"` + marker + '"}', + ); + }); + + it("retains literal opaque matches spanning quoted fragments in plain bodies", () => { + const credential = 'prefix"quoted"suffix'; + expect( + forCredential(credential).json( + `Auth Custom ${credential}; payload ${credential}; path /caller/input`, + ), + ).toBe(`Auth ${marker}; payload ${marker}; path /caller/input`); + }); + + it("replaces adjacent matches once without rescanning the marker", () => { + expect( + forCredential("redacted").json(String.raw`"redacted\u0072edacted"`), + ).toBe('"␟␟"'); + }); + + it("handles unquoted JSON fragments while retaining ordinary text semantics", () => { + const source = String.raw`\u003csecret\u003e`; + const redactor = forCredential(""); + expect(redactor.text(source)).toBe(source); + expect(redactor.json(source)).toBe(marker); + const invalid = String.raw``; + expect(redactor.json(invalid)).toBe(invalid); + const unrelated = String.raw`"\u003cpublic\u003e"`; + expect(forCredential("u003c").json(unrelated)).toBe(unrelated); + expect(forCredential("u003c").json(unrelated.slice(1, -1))).toBe( + "\\" + marker + String.raw`public\u003e`, + ); + }); + it("leaves ordinary text diagnostics and credential-free configurations unchanged", () => { + const source = String.raw`{"x":"\u003csecret\u003e"}`; + expect(forCredential("").text(source)).toBe(source); + const redactor = createIdaDiagnosticRedactor({ + headers: {}, + url: "http://127.0.0.1:12345/mcp", + mode: "attached", + timeoutMs: 1000, + }); + expect(redactor.json(source)).toBe(source); + }); +}); + +describe("IDA diagnostic output resource bounds", () => { + it.each(["text", "json"] as const)( + "bounds %s expansion using the runtime string limit", + (format) => { + runtime.limit = 64; + const redactor = forCredential("Q"); + const prefix = format === "json" ? '"' : ""; + const suffix = prefix; + expect(redactor[format](prefix + "QQ" + suffix)).toBe( + prefix + marker + marker + suffix, + ); + let failure: unknown; + try { + redactor[format](prefix + "QQQ" + suffix); + } catch (cause: unknown) { + failure = cause; + } + expect(failure).toBeInstanceOf(IdaDiagnosticResourceError); + if (!(failure instanceof IdaDiagnosticResourceError)) + throw new Error("Expected an owned diagnostic resource failure"); + expect(failure.reportedLimits).toMatchObject({ + boundary: "ida-diagnostic-redaction", + max_string_code_units: 64, + }); + expect(projectAnalysisError(failure)).toMatchObject({ + code: "resource_constraint", + details: { resource: "memory" }, + }); + }, + ); + + it("bounds temporary JSON character decoding before wrapping an oversized run", () => { + runtime.limit = 64; + const redactor = forCredential("Z"); + expect(redactor.json("Q".repeat(62))).toBe("Q".repeat(62)); + let failure: unknown; + try { + redactor.json("Q".repeat(63)); + } catch (cause: unknown) { + failure = cause; + } + expect(failure).toBeInstanceOf(IdaDiagnosticResourceError); + if (!(failure instanceof IdaDiagnosticResourceError)) + throw new Error("Expected a temporary decoding resource failure"); + expect(failure.message).toContain("Temporary"); + expect(failure.reportedLimits).toEqual({ + boundary: "ida-diagnostic-redaction", + representation: "json-character-run", + input_code_units: 63, + json_decoding_code_units_at_least: 65, + max_string_code_units: 64, + }); + }); + + it("closes the first match stream if priming the second stream fails", () => { + runtime.limit = 64; + const source = "Q".repeat(63); + let closed = false; + function* matches() { + try { + yield { start: 0, end: 1 }; + } finally { + closed = true; + } + } + expect(() => redactIdaJsonDiagnostic(source, matches)).toThrow( + IdaDiagnosticResourceError, + ); + expect(closed).toBe(true); + }); + + it("stops consuming a lazy match stream before oversized output is allocated", () => { + runtime.limit = 32; + let consumed = 0; + function* matches() { + for (let index = 0; index < 10000; index++) { + consumed++; + yield { start: index, end: index + 1 }; + } + } + expect(() => writeIdaDiagnostic("Q".repeat(10000), matches())).toThrow( + IdaDiagnosticResourceError, + ); + expect(consumed).toBeLessThan(10); + }); +}); + +describe("overlapping raw and decoded JSON credential spans", () => { + const overlappingMarker = "␟"; + const overlappingRedactor = () => + createIdaDiagnosticRedactor({ + url: "http://127.0.0.1:12345/mcp", + headers: { Authorization: "Custom X]", Cookie: "a=]SECRET" }, + mode: "attached", + timeoutMs: 1000, + }); + it.each([ + String.raw`{"reason":"X]SECRET","n":1.00e+03}`, + String.raw`{"reason":"X\u005dSECRET","n":1.00e+03}`, + ])("unions decoded overlaps before monotone source mapping: %s", (source) => { + expect(overlappingRedactor().json(source)).toBe( + '{"reason":"' + overlappingMarker + '","n":1.00e+03}', + ); + }); + it.each(["X]SECRET", String.raw`X\u005dSECRET`])( + "unions raw/recovered overlaps in a malformed document: %s", + (source) => { + expect( + overlappingRedactor().json("Rejected " + source + " at /caller/input"), + ).toBe("Rejected " + overlappingMarker + " at /caller/input"); + }, + ); + it("maps a merged overlap containing an escaped surrogate pair", () => { + const encoded = Buffer.from("fixture-user:😀X]", "utf8").toString("base64"); + const redactor = createIdaDiagnosticRedactor({ + url: "http://127.0.0.1:12345/mcp", + headers: { Authorization: "Basic " + encoded, Cookie: "a=]SECRET" }, + mode: "attached", + timeoutMs: 1000, + }); + expect( + redactor.json( + String.raw`{"reason":"\ud83d\ude00X\u005dSECRET","context":"kept"}`, + ), + ).toBe('{"reason":"' + overlappingMarker + '","context":"kept"}'); + }); + it("unions dense escaped decoded overlaps lazily", () => { + const source = '"' + String.raw`\u0041`.repeat(4096) + '"'; + const result = forCredential("AA").json(source); + expect(result.length).toBe(marker.length + 2); + expect(result).toBe('"' + marker + '"'); + }); +}); + +it("bounds the final multi-block diagnostic representation", () => { + runtime.limit = 64; + const redactor = forCredential("Q"); + expect(() => redactor.blocks(["QQ", "QQ"])).toThrow( + IdaDiagnosticResourceError, + ); +}); + +it("bounds context plus the already-owned diagnostic body before joining", () => { + runtime.limit = 64; + const redactor = forCredential("Q"); + const body = redactor.text("QQ"); + expect(() => redactor.message("ctx", body)).toThrow( + IdaDiagnosticResourceError, + ); +}); diff --git a/src/ida/IdaJsonDiagnostic.ts b/src/ida/IdaJsonDiagnostic.ts new file mode 100644 index 0000000000..4f0823b44f --- /dev/null +++ b/src/ida/IdaJsonDiagnostic.ts @@ -0,0 +1,216 @@ +import { constants } from "node:buffer"; +import { createScanner, type ScanError, type SyntaxKind } from "jsonc-parser"; +import { + IDA_REDACTION_MARKER, + IdaDiagnosticResourceError, + writeIdaDiagnostic, + type IdaDiagnosticMatch, +} from "./IdaDiagnosticWriter.js"; + +// jsonc-parser declares ambient const enums, which cannot be imported as values +// under verbatimModuleSyntax. Keep its scanner codes local and type-checked. +const JSON_STRING_LITERAL = 10 satisfies SyntaxKind; +const JSON_EOF = 17 satisfies SyntaxKind; +const JSON_NO_ERROR = 0 satisfies ScanError; + +/** Matches are non-overlapping and ordered by their UTF-16 source offsets. */ +type Matches = (value: string) => Iterable; + +// Complete JSON character runs remain meaningful even in an unfinished or +// malformed string. Invalid units separate runs instead of joining their text. +const JSON_CHARACTER_RUN = + /(?:[^"\\\u0000-\u001f]|\\(?:["\\/bfnrt]|u[\dA-Fa-f]{4}))+/gu; + +function* stringMatches( + encoded: string, + decoded: string, + offset: number, + matches: Matches, +): Generator { + // Match offsets and JSON escape widths are both UTF-16 units. No source bytes + // outside a matched value need to be normalized or serialized again. + let sourceCursor = 0; + let decodedCursor = 0; + const sourceOffset = (boundary: number): number => { + while (decodedCursor < boundary) { + sourceCursor += + encoded[sourceCursor] === "\\" + ? encoded[sourceCursor + 1] === "u" + ? 6 + : 2 + : 1; + decodedCursor++; + } + return offset + sourceCursor; + }; + for (const match of matches(decoded)) + yield { + start: sourceOffset(match.start), + end: sourceOffset(match.end), + }; +} + +function* characterRunMatches( + encoded: string, + offset: number, + matches: Matches, + inputCodeUnits: number, +): Generator { + for (const run of encoded.matchAll(JSON_CHARACTER_RUN)) { + // The run grammar admits only complete JSON characters. JSON.parse owns + // their decoded meaning; malformed units remain verbatim between runs. + const decodingCodeUnits = run[0].length + 2; + if (decodingCodeUnits > constants.MAX_STRING_LENGTH) + throw new IdaDiagnosticResourceError( + inputCodeUnits, + decodingCodeUnits, + "json-character-run", + ); + const decoded: unknown = JSON.parse('"' + run[0] + '"'); + if (typeof decoded !== "string") + throw new TypeError("A JSON character run did not decode to a string."); + yield* stringMatches(run[0], decoded, offset + run.index, matches); + } +} + +function* decodedMatches( + text: string, + matches: Matches, +): Generator { + let scannedEnd = 0; + const scanner = createScanner(text); + while (scanner.scan() !== JSON_EOF) { + if (scanner.getToken() !== JSON_STRING_LITERAL) continue; + const start = scanner.getTokenOffset(); + const end = start + scanner.getTokenLength(); + if (scannedEnd < start) + yield* characterRunMatches( + text.slice(scannedEnd, start), + scannedEnd, + matches, + text.length, + ); + if (scanner.getTokenError() === JSON_NO_ERROR) + yield* stringMatches( + text.slice(start + 1, end - 1), + scanner.getTokenValue(), + start + 1, + matches, + ); + else + yield* characterRunMatches( + text.slice(start + 1, end), + start + 1, + matches, + text.length, + ); + scannedEnd = end; + } + yield* characterRunMatches( + text.slice(scannedEnd), + scannedEnd, + matches, + text.length, + ); +} + +function* literalMatches( + text: string, + matches: Matches, + validDocument: boolean, +): Generator { + if (!validDocument) { + // Recovery cannot establish a unique meaning for a malformed document. + yield* matches(text); + return; + } + const scanner = createScanner(text); + let complete: IdaDiagnosticMatch | undefined; + let ended = false; + for (const match of matches(text)) { + while (!ended && (complete === undefined || complete.end <= match.start)) { + complete = undefined; + while (scanner.scan() !== JSON_EOF) { + if (scanner.getToken() !== JSON_STRING_LITERAL) continue; + const start = scanner.getTokenOffset(); + complete = { start, end: start + scanner.getTokenLength() }; + break; + } + if (complete === undefined) ended = true; + } + // Only a complete string in a valid document owns decoded-only semantics. + // Raw matches crossing tokens remain visible to the credential boundary. + if ( + complete === undefined || + match.start < complete.start || + match.end > complete.end + ) + yield match; + } +} + +function* mergeMatches( + left: Iterable, + right: Iterable, +): Generator { + const leftIterator = left[Symbol.iterator](); + const rightIterator = right[Symbol.iterator](); + try { + let leftMatch = leftIterator.next(); + let rightMatch = rightIterator.next(); + while (!leftMatch.done || !rightMatch.done) { + if ( + rightMatch.done || + (!leftMatch.done && leftMatch.value.start <= rightMatch.value.start) + ) { + if (leftMatch.done) break; + yield leftMatch.value; + leftMatch = leftIterator.next(); + } else { + yield rightMatch.value; + rightMatch = rightIterator.next(); + } + } + } finally { + try { + leftIterator.return?.(); + } finally { + rightIterator.return?.(); + } + } +} + +/** Redact complete or malformed JSON diagnostics without rewriting unrelated source. */ +export const redactIdaJsonDiagnostic = ( + text: string, + matches: Matches, + replacement: string = IDA_REDACTION_MARKER, +): string => { + let validDocument = false; + try { + JSON.parse(text); + validDocument = true; + } catch { + // Keep both raw and decoded interpretations when the document is incomplete. + } + return writeIdaDiagnostic( + text, + mergeMatches( + literalMatches(text, matches, validDocument), + decodedMatches(text, matches), + ), + replacement, + ); +}; + +/** Untyped producer text retains both literal and JSON character interpretations. */ +export const redactIdaUnclassifiedDiagnostic = ( + text: string, + matches: Matches, + replacement: string = IDA_REDACTION_MARKER, +): string => + writeIdaDiagnostic( + text, + mergeMatches(matches(text), decodedMatches(text, matches)), + replacement, + ); diff --git a/src/ida/IdaMcpConnection.test.ts b/src/ida/IdaMcpConnection.test.ts index a582ed902c..f7e7a93755 100644 --- a/src/ida/IdaMcpConnection.test.ts +++ b/src/ida/IdaMcpConnection.test.ts @@ -1,31 +1,70 @@ -import { describe, expect, it } from "vitest"; +import { inspect } from "node:util"; +import { SdkErrorCode, SdkHttpError } from "@modelcontextprotocol/client"; +import { afterEach, describe, expect, it, vi } from "vitest"; import { + createIdaMcpConnection, decodeIdaToolResult, - redactIdaTransportFailure, } from "./IdaMcpConnection.js"; +import { createIdaDiagnosticRedactor } from "./IdaDiagnosticRedactor.js"; +import { AnalysisTimeoutError } from "../domain/analysisErrorCore.js"; +import { readSettledHealth } from "./IdaBusyHealth.js"; describe("upstream IDA tool result decoding", () => { - it("redacts configured transport authentication without redacting caller evidence", () => { - const error = redactIdaTransportFailure( - new Error( - "Invalid Authorization value Bearer fixture-secret at /caller/input", - ), - { - url: "http://127.0.0.1:12345/mcp", - headers: { Authorization: "Bearer fixture-secret" }, - mode: "attached", - timeoutMs: 1000, - }, - ); - expect(error.message).not.toContain("fixture-secret"); - expect(error.message).toContain("/caller/input"); + it("leaves successful legacy, structured, and native evidence unchanged", () => { + const redact = createIdaDiagnosticRedactor({ + url: "http://127.0.0.1:12345/mcp", + headers: { Authorization: "Bearer fixture-secret" }, + mode: "attached", + timeoutMs: 1000, + }); + const evidence = "Caller-selected fixture-secret evidence"; expect( - decodeIdaToolResult({ - content: [ - { type: "text", text: "Caller-selected fixture-secret evidence" }, - ], - }), - ).toBe("Caller-selected fixture-secret evidence"); + decodeIdaToolResult( + { content: [{ type: "text", text: evidence }] }, + undefined, + redact, + ), + ).toBe(evidence); + expect( + decodeIdaToolResult( + { content: [], structuredContent: { evidence } }, + undefined, + redact, + ), + ).toEqual({ evidence }); + expect( + decodeIdaToolResult( + { content: [{ type: "text", text: JSON.stringify({ evidence }) }] }, + "native", + redact, + ), + ).toEqual({ evidence }); + }); + it("redacts credentials in truncated-output diagnostics before constructing the error", () => { + const redact = createIdaDiagnosticRedactor({ + url: "http://127.0.0.1:12345/mcp", + headers: { Authorization: "Bearer fixture-secret" }, + mode: "attached", + timeoutMs: 1000, + }); + expect(() => + decodeIdaToolResult( + { + content: [], + _meta: { + ida_mcp: { + output_truncated: true, + download_hint: + "Download /caller/output?cursor=kept using Bearer fixture-secret", + }, + }, + }, + undefined, + redact, + ), + ).toThrow( + "Download /caller/output?cursor=kept using [redacted transport credential]", + ); }); it("decodes scalar wrappers and legacy text without dropping structured dictionaries", () => { expect( @@ -79,3 +118,388 @@ describe("upstream IDA tool result decoding", () => { ).toThrow("Download the full upstream output"); }); }); + +afterEach(() => { + vi.unstubAllGlobals(); +}); + +const failingHttpConnection = ( + credential: string, + response: Response | Error, +) => { + // Exercise the real SDK consumer with an explicitly simulated Fetch boundary. + vi.stubGlobal( + "fetch", + response instanceof Error + ? vi.fn().mockRejectedValue(response) + : vi.fn().mockResolvedValue(response), + ); + return createIdaMcpConnection( + { + url: "http://127.0.0.1:12345/mcp", + headers: { Authorization: "Custom " + credential }, + mode: "attached", + timeoutMs: 1000, + }, + {}, + ); +}; + +interface MockedIdaHttpRequest { + readonly id?: string | number; + readonly method: string; + readonly params?: Record; +} + +const jsonRpcResponse = (value: unknown): Response => + new Response(JSON.stringify(value), { + headers: { "Content-Type": "application/json" }, + }); + +const stalledResponse = (signal: AbortSignal | null | undefined) => + new Promise((_resolve, reject) => { + const aborted = () => + reject(new DOMException("The request was aborted", "AbortError")); + if (signal?.aborted === true) { + aborted(); + return; + } + signal?.addEventListener("abort", aborted, { once: true }); + }); + +const mockedHttpBoundary = ({ + serverName, + tools, + call, +}: { + readonly serverName: string; + readonly tools: readonly { + readonly name: string; + readonly inputSchema: object; + }[]; + readonly call: ( + request: MockedIdaHttpRequest, + signal: AbortSignal | null | undefined, + ) => Response | Promise; +}): void => { + vi.stubGlobal( + "fetch", + vi.fn(async (_input: Parameters[0], init?: RequestInit) => { + const request = JSON.parse( + String(init?.body ?? ""), + ) as MockedIdaHttpRequest; + if (request.method === "initialize") + return jsonRpcResponse({ + jsonrpc: "2.0", + id: request.id, + result: { + protocolVersion: request.params?.protocolVersion, + capabilities: { tools: {} }, + serverInfo: { name: serverName, version: "1" }, + }, + }); + if ( + request.method === "notifications/initialized" || + request.method === "notifications/cancelled" + ) + return new Response(null, { status: 202 }); + if (request.method === "tools/list") + return jsonRpcResponse({ + jsonrpc: "2.0", + id: request.id, + result: { tools }, + }); + if (request.method === "tools/call") return call(request, init?.signal); + throw new Error(`Unexpected fixture request: ${request.method}`); + }), + ); +}; + +it("preserves the SDK request timeout as a typed IDA timeout", async () => { + const timeoutMs = 1000; + const requestTimeoutMs = 40; + const pendingCall = vi.fn(); + mockedHttpBoundary({ + serverName: "ida-timeout-fixture", + tools: [], + call: (_request, signal) => { + pendingCall(); + return stalledResponse(signal); + }, + }); + const connection = createIdaMcpConnection( + { + url: "http://127.0.0.1:12345/mcp", + headers: { Authorization: "Bearer fixture-secret" }, + mode: "attached", + timeoutMs, + }, + {}, + ); + try { + await expect(connection.connect()).resolves.toEqual([]); + const failure = await connection + .call("decompile", {}, { timeoutMs: requestTimeoutMs }) + .catch((cause: unknown) => cause); + expect(pendingCall).toHaveBeenCalledOnce(); + expect(failure).toBeInstanceOf(AnalysisTimeoutError); + expect(failure).toMatchObject({ + operation: "decompile", + timeoutMs: requestTimeoutMs, + }); + if (!(failure instanceof AnalysisTimeoutError)) + throw new Error("Expected a typed IDA request timeout"); + expect(failure.cause).toBeInstanceOf(Error); + expect(inspectRetainedError(failure)).not.toContain("fixture-secret"); + } finally { + await connection.close(); + } +}); + +it("bounds a stalled health poll by the remaining readiness deadline", async () => { + const timeoutMs = 800; + const pendingCall = vi.fn(); + let clockNow = 0; + const clock = { + now: () => clockNow, + delay: async (ms: number) => { + clockNow += ms; + }, + }; + let pendingCalls = 0; + const busy = { + status: "busy", + busy_tool: "config_json_get", + busy_sec: 1, + queued_calls: 0, + }; + mockedHttpBoundary({ + serverName: "ida-health-deadline-fixture", + tools: [{ name: "server_health", inputSchema: { type: "object" } }], + call: (request, signal) => { + pendingCalls += 1; + if (pendingCalls === 1) { + clockNow += 450; + const result = { + content: [{ type: "text", text: JSON.stringify(busy) }], + structuredContent: busy, + }; + return jsonRpcResponse({ jsonrpc: "2.0", id: request.id, result }); + } + return stalledResponse(signal); + }, + }); + const connection = createIdaMcpConnection( + { + url: "http://127.0.0.1:12345/mcp", + headers: {}, + mode: "headless", + timeoutMs, + }, + {}, + ); + try { + await connection.connect(); + const failure = await readSettledHealth(connection, { + database: "fixture-db", + timeoutMs, + clock, + }).catch((cause: unknown) => cause); + + expect(pendingCalls).toBe(2); + expect(failure).toBeInstanceOf(AnalysisTimeoutError); + expect(failure).toMatchObject({ + operation: "server_health: headless worker still busy in config_json_get", + timeoutMs, + cause: busy, + diagnostics: { + health_request_operation: "server_health", + health_request_timeout_ms: 250, + }, + }); + if (!(failure instanceof AnalysisTimeoutError)) + throw new Error("Expected the readiness deadline to expire"); + expect(failure.capturedOutput).toBeDefined(); + } finally { + await connection.close(); + } +}); + +const inspectRetainedError = (error: unknown): string => + inspect(error, { depth: null, maxStringLength: null, breakLength: Infinity }); + +describe("retained IDA diagnostic errors", () => { + it.each([ + { + credential: "fixture-secret", + body: '{"reason":"fixture-secret at /caller/input",', + forbidden: "fixture-secret", + }, + { + credential: "", + body: String.raw`["\u003ctoken\u003e",x]`, + forbidden: "u003ctoken", + }, + ])( + "retains only sanitized malformed HTTP JSON: $credential", + async ({ credential, body, forbidden }) => { + const connection = failingHttpConnection( + credential, + new Response(body, { headers: { "Content-Type": "application/json" } }), + ); + const failure: unknown = await connection + .connect() + .catch((cause: unknown) => cause); + expect(failure).toBeInstanceOf(Error); + if (!(failure instanceof Error)) + throw new Error("Expected a malformed HTTP JSON failure"); + expect(failure.message).toContain("[redacted transport credential]"); + expect(inspectRetainedError(failure)).not.toContain(forbidden); + expect(inspectRetainedError(failure)).not.toContain(credential); + await connection.close(); + }, + ); + + it("does not rescan an owned malformed JSON replacement marker", async () => { + const connection = failingHttpConnection( + "redacted", + new Response("Rejected redacted", { + headers: { "Content-Type": "application/json" }, + }), + ); + const failure: unknown = await connection + .connect() + .catch((cause: unknown) => cause); + expect(failure).toBeInstanceOf(Error); + if (!(failure instanceof Error)) + throw new Error("Expected a malformed HTTP JSON failure"); + expect(failure.message).toBe( + "IDA MCP returned malformed HTTP JSON. Response body: \nRejected ␟", + ); + expect(inspectRetainedError(failure)).not.toContain("[[redacted"); + await connection.close(); + }); + + it("replaces an unknown network error graph with a safe owned reason", async () => { + const upstream = Object.assign( + new Error("fixture-secret refused at /caller/input", { + cause: new Error("nested fixture-secret"), + }), + { headers: { Authorization: "Custom fixture-secret" } }, + ); + const connection = failingHttpConnection("fixture-secret", upstream); + const failure: unknown = await connection + .connect() + .catch((cause: unknown) => cause); + expect(failure).toBeInstanceOf(Error); + if (!(failure instanceof Error)) + throw new Error("Expected a network failure"); + expect(failure.message).toContain("refused at /caller/input"); + expect(failure.cause).not.toBe(upstream); + expect(inspectRetainedError(failure)).not.toContain("fixture-secret"); + await connection.close(); + }); + + it("retains meaningful HTTP status without unsafe SDK metadata", async () => { + const connection = failingHttpConnection( + "fixture-secret", + new Response("unavailable fixture-secret at /caller/input", { + status: 500, + statusText: "fixture-secret", + headers: { "Content-Type": "text/plain" }, + }), + ); + const failure: unknown = await connection + .connect() + .catch((cause: unknown) => cause); + expect(failure).toBeInstanceOf(Error); + if (!(failure instanceof Error)) + throw new Error("Expected an HTTP failure"); + expect(failure.cause).toMatchObject({ data: { status: 500 } }); + expect(failure.message).toContain( + "unavailable [redacted transport credential] at /caller/input", + ); + expect(inspectRetainedError(failure)).not.toContain("fixture-secret"); + await connection.close(); + }); +}); + +it.each([ + { + code: SdkErrorCode.ClientHttpNotImplemented, + message: "Changed SDK context: fixture-secret", + }, + { + code: SdkErrorCode.ClientHttpFailedToOpenStream, + message: "Error POSTing to endpoint: fixture-secret", + }, +])( + "does not infer owned body spans from an unknown SDK shape: $code / $message", + async ({ code, message }) => { + const connection = failingHttpConnection( + "fixture-secret", + new SdkHttpError(code, message, { status: 500, text: "fixture-secret" }), + ); + try { + const failure = await connection + .connect() + .catch((cause: unknown) => cause); + if (!(failure instanceof Error)) throw new Error("Expected SDK failure"); + expect(inspectRetainedError(failure)).not.toContain("fixture-secret"); + expect(failure.message).toContain("[redacted transport credential]"); + expect(failure.cause).toMatchObject({ code, data: { status: 500 } }); + } finally { + await connection.close(); + } + }, +); + +it("redacts JSON character candidates in unclassified SDK errors", async () => { + const connection = failingHttpConnection( + 'q"uote', + new Error(String.raw`network context {"error":"q\"uote"} at /caller/input`), + ); + try { + const failure = await connection.connect().catch((cause: unknown) => cause); + if (!(failure instanceof Error)) throw new Error("Expected SDK failure"); + expect(failure.message).toContain( + '{"error":"[redacted transport credential]"} at /caller/input', + ); + expect(inspectRetainedError(failure)).not.toContain(String.raw`q\"uote`); + } finally { + await connection.close(); + } +}); + +it("owns unclassified truncated-output hints separately from their context", () => { + const diagnostic = createIdaDiagnosticRedactor({ + url: "http://127.0.0.1:12345/mcp", + mode: "attached", + timeoutMs: 1000, + headers: { Authorization: 'Custom q"uote' }, + }); + let failure: unknown; + try { + decodeIdaToolResult( + { + content: [], + _meta: { + ida_mcp: { + output_truncated: true, + download_hint: String.raw`{"error":"q\"uote","path":"/caller/output"}`, + }, + }, + }, + undefined, + diagnostic, + ); + } catch (cause: unknown) { + failure = cause; + } + if (!(failure instanceof Error)) + throw new Error("Expected truncation failure"); + expect(failure.message).toContain( + '{"error":"[redacted transport credential]","path":"/caller/output"}', + ); + expect(inspectRetainedError(failure)).not.toContain(String.raw`q\"uote`); +}); diff --git a/src/ida/IdaMcpConnection.ts b/src/ida/IdaMcpConnection.ts index 8137edcc96..d427d1fe45 100644 --- a/src/ida/IdaMcpConnection.ts +++ b/src/ida/IdaMcpConnection.ts @@ -1,8 +1,17 @@ +import { + createIdaDiagnosticRedactor, + IDA_DIAGNOSTIC_SOURCE_BYTES, +} from "./IdaDiagnosticRedactor.js"; +import { IdaDiagnosticResourceError } from "./IdaDiagnosticWriter.js"; import { ChildProcess } from "node:child_process"; import { Socket } from "node:net"; import { snapshotEnvironment } from "../process/snapshotEnvironment.js"; import { Client, + ProtocolError, + SdkError, + SdkErrorCode, + SdkHttpError, StreamableHTTPClientTransport, } from "@modelcontextprotocol/client"; import { @@ -12,7 +21,12 @@ import { import { z } from "zod"; import type { IdaConfiguration } from "./IdaConfiguration.js"; import { jsonValueSchema, type JsonValue } from "../domain/jsonValue.js"; -import { AnalysisProtocolError } from "../domain/analysisErrorCore.js"; +import { err, ok, type Result } from "../domain/result.js"; +import { + AnalysisTimeoutError, + AnalysisProtocolError, + AnalysisResourceConstraintError, +} from "../domain/analysisErrorCore.js"; import { ProviderAdapterError } from "../domain/providerAdapterError.js"; /** Injectable MCP boundary used by the provider and transport conformance tests. */ @@ -21,6 +35,7 @@ export interface IdaMcpConnection { call( name: string, args: Readonly>, + options?: { readonly timeoutMs?: number | undefined }, ): Promise; readResource?(uri: string): Promise; serverInfo(): JsonValue; @@ -33,6 +48,10 @@ export interface IdaMcpConnection { detach(): void; } +export type IdaMcpConnectionDependencies = Readonly<{ + createDiagnosticRedactor?: typeof createIdaDiagnosticRedactor; +}>; + const toolResultSchema = z.object({ isError: z.boolean().optional(), structuredContent: z.record(z.string(), z.unknown()).optional(), @@ -53,14 +72,14 @@ const toolResultSchema = z.object({ .optional(), }); +const IDA_STDIO_MAX_MESSAGE_BYTES = 64 * 1024 * 1024; + const parseIdaJsonText = (text: string): JsonValue => { let value: unknown; try { value = JSON.parse(text); - } catch (cause: unknown) { - throw new AnalysisProtocolError("IDA MCP returned malformed JSON text.", { - cause, - }); + } catch { + throw new AnalysisProtocolError("IDA MCP returned malformed JSON text."); } return jsonValueSchema.parse(value); }; @@ -69,18 +88,30 @@ const parseIdaJsonText = (text: string): JsonValue => { export const decodeIdaToolResult = ( input: unknown, protocol?: "native", + diagnostic: IdaDiagnosticRedactor = createIdaDiagnosticRedactor(), ): JsonValue => { const result = toolResultSchema.parse(input); const blocks = result.content.flatMap((block) => block.type === "text" && block.text !== undefined ? [block.text] : [], ); - if (result.isError === true) + if (result.isError === true) { + const reason = diagnostic.blocks(blocks); throw new AnalysisProtocolError( - `IDA MCP tool failed: ${blocks.join("\n")}`, + diagnostic.message("IDA MCP tool failed: ", reason), + { + diagnostics: { reason }, + }, ); + } if (result._meta?.ida_mcp?.output_truncated === true) throw new AnalysisProtocolError( - `IDA MCP truncated its output; the preview cannot establish a complete observation. ${result._meta.ida_mcp.download_hint ?? "Use the upstream output download workflow to retrieve the full result."}`, + diagnostic.message( + "IDA MCP truncated its output; the preview cannot establish a complete observation. ", + diagnostic.blocks([ + result._meta.ida_mcp.download_hint ?? + "Use the upstream output download workflow to retrieve the full result.", + ]), + ), ); // Native MCP retains the complete JSON text while structuredContent may // contain a shortened preview. Parse that producer-owned representation once. @@ -113,59 +144,273 @@ export const decodeIdaToolResult = ( return values[0] ?? null; }; -/** Redact configured transport authentication values from SDK diagnostics only. */ -export const redactIdaTransportFailure = ( - cause: unknown, - config: IdaConfiguration, -): Error => { - let message = - cause instanceof Error - ? cause.message - : "Unknown IDA MCP transport failure"; - if ("headers" in config) { - for (const [name, value] of Object.entries(config.headers)) { - if (!/^(?:authorization|proxy-authorization|cookie)$/iu.test(name)) - continue; - const credential = value.match(/^(?:Bearer|Basic)\s+([\s\S]+)$/iu)?.[1]; - for (const secret of [value, credential]) - if (secret !== undefined && secret.length > 0) - message = message - .split(secret) - .join("[redacted transport credential]"); - } +// Retain only producer-sanitized JSON diagnostics, never a native parser cause. +class IdaHttpJsonError extends Error { + constructor(text: string, redactor: IdaDiagnosticRedactor) { + super( + redactor.message( + "IDA MCP returned malformed HTTP JSON. Response body: ", + text, + ), + ); } - return new Error(message, { cause }); +} + +type IdaDiagnosticRedactor = ReturnType; + +type IdaContentType = Readonly<{ + essence: string; + subtype: string; + charset: string | undefined; + parameters: ReadonlyMap; +}>; + +const httpQuotedOctet = (code: number): boolean => + code === 9 || + code === 32 || + (code >= 33 && code <= 126) || + (code >= 128 && code <= 255); + +/** Consume one complete RFC 9110 media type; never discard a malformed parameter. */ +const parseIdaContentType = ( + text: string, +): Result => { + let cursor = 0; + const token = new RegExp("[!#$%&'*+.^_`|~0-9A-Za-z-]+", "y"); + const readToken = (): string | undefined => { + token.lastIndex = cursor; + const match = token.exec(text); + if (match === null) return undefined; + cursor = token.lastIndex; + return match[0]; + }; + const readOws = (): void => { + while (text[cursor] === " " || text[cursor] === "\t") cursor++; + }; + const readQuoted = (): string | undefined => { + cursor++; + let value = ""; + while (cursor < text.length) { + let code = text.charCodeAt(cursor++); + if (code === 34) return value; + if (code === 92) { + code = text.charCodeAt(cursor++); + if (!httpQuotedOctet(code)) return undefined; + } else if (!httpQuotedOctet(code)) return undefined; + value += String.fromCharCode(code); + } + return undefined; + }; + readOws(); + const type = readToken()?.toLowerCase(); + if (type === undefined || text[cursor++] !== "/") return err("malformed"); + const subtype = readToken()?.toLowerCase(); + if (subtype === undefined) return err("malformed"); + const parameters = new Map(); + let ambiguous = false; + while (true) { + readOws(); + if (cursor === text.length) break; + if (text[cursor++] !== ";") return err("malformed"); + readOws(); + // RFC 9110 parameters admit an empty item after a semicolon. + if (cursor === text.length || text[cursor] === ";") continue; + const name = readToken()?.toLowerCase(); + if (name === undefined || text[cursor++] !== "=") return err("malformed"); + const value = text[cursor] === '"' ? readQuoted() : readToken(); + if (value === undefined) return err("malformed"); + // Repetition is ambiguous for this owner, even if both values agree. + // Keep parsing so syntax errors anywhere in the field cannot be discarded. + if (parameters.has(name)) ambiguous = true; + else parameters.set(name, value); + } + return ambiguous + ? err("ambiguous") + : ok({ + essence: type + "/" + subtype, + subtype, + charset: parameters.get("charset"), + parameters, + }); +}; +const httpErrorBody = ( + response: Response, + text: string, + redactor: IdaDiagnosticRedactor, +): string => { + const omitted = redactor.text( + "IDA MCP response body omitted: unsupported or unknown format or character encoding.", + ); + const contentType = response.headers.get("content-type"); + if (contentType === null) return omitted; + const mediaType = parseIdaContentType(contentType); + if (!mediaType.ok) return omitted; + const charset = mediaType.value.charset?.toLowerCase(); + if ( + charset !== undefined && + !["utf-8", "utf8", "us-ascii", "ascii"].includes(charset) + ) + return omitted; + if ( + mediaType.value.subtype === "json" || + mediaType.value.subtype.endsWith("+json") + ) + return redactor.json(text); + if (mediaType.value.essence === "text/plain") return redactor.text(text); + return omitted; }; +/** Keep error-body materialization within the diagnostic source-byte budget. */ +const readBoundedResponseText = async ( + response: Response, +): Promise => { + if (response.body === null) return ""; + const reader = response.body.getReader(); + const chunks: Buffer[] = []; + let total = 0; + try { + for (;;) { + const { done, value } = await reader.read(); + if (done) return Buffer.concat(chunks, total).toString("utf8"); + total += value.byteLength; + if (total > IDA_DIAGNOSTIC_SOURCE_BYTES) { + await reader.cancel().catch(() => undefined); + return undefined; + } + chunks.push(Buffer.from(value)); + } + } finally { + reader.releaseLock(); + } +}; + +const diagnosticFetch = + (redactor: IdaDiagnosticRedactor): typeof fetch => + async (input, init) => { + const response = await fetch(input, init); + const readText = response.text.bind(response); + let safeBody: string | undefined; + if (!response.ok) { + // Normalize before the SDK can swallow a body-consumer failure or drop + // its format metadata. Both consumers use this one owned diagnostic. + const sourceBody = await readBoundedResponseText(response); + safeBody = + sourceBody === undefined + ? redactor.text( + `IDA MCP response body omitted: exceeds ${IDA_DIAGNOSTIC_SOURCE_BYTES} UTF-8 bytes.`, + ) + : httpErrorBody(response, sourceBody, redactor); + const body = safeBody; + Object.defineProperty(response, "text", { + value: async (): Promise => body, + }); + } + // Successful JSON remains lazy, and SSE keeps its original body stream. + Object.defineProperty(response, "json", { + value: async (): Promise => { + const text = safeBody ?? (await readText()); + try { + const value: unknown = JSON.parse(text); + return value; + } catch { + // Parsing failure owns diagnostics even when the HTTP status succeeded. + throw new IdaHttpJsonError( + safeBody ?? httpErrorBody(response, text, redactor), + redactor, + ); + } + }, + }); + return response; + }; + +const retainedTransportCause = (cause: unknown, message: string): Error => { + // Do not clone arbitrary upstream fields, stacks, data or nested causes. + if (cause instanceof SdkHttpError) + return new SdkHttpError(cause.code, message, { + status: cause.data.status, + statusText: "", + }); + if (cause instanceof ProtocolError) + return new ProtocolError(cause.code, message); + return new Error(message); +}; /** Create a standard SDK connection to an existing upstream registration. */ export const createIdaMcpConnection = ( config: IdaConfiguration, environment: Readonly, + dependencies: IdaMcpConnectionDependencies = {}, ): IdaMcpConnection => { const client = new Client({ name: "rea-ida-provider", version: "1" }); + const redactor = ( + dependencies.createDiagnosticRedactor ?? createIdaDiagnosticRedactor + )(config); let stderr = Buffer.alloc(0); let stderrBytes = 0; - const transportFailure = (cause: unknown): never => { - const failure = redactIdaTransportFailure(cause, config); - if (stderrBytes === 0) throw failure; + const transportFailure = ( + cause: unknown, + operation = "ida_mcp_response", + requestTimeoutMs = config.timeoutMs, + ): never => { + const bufferLimit = + cause instanceof Error + ? /ReadBuffer exceeded maximum size of (\d+) bytes/u.exec(cause.message) + : null; + if (cause instanceof IdaDiagnosticResourceError) throw cause; + let failure: Error; + if (cause instanceof IdaHttpJsonError) + failure = new Error(cause.message, { cause }); + else { + const message = + cause instanceof Error + ? cause.message + : "Unknown IDA MCP transport failure"; + const body = cause instanceof SdkHttpError ? cause.data.text : undefined; + // Only the pinned SDK POST producer places this owned body at a known + // span. Equal substrings elsewhere remain untrusted SDK context. + const prefix = "Error POSTing to endpoint: "; + const ownedPostBody = + cause instanceof SdkHttpError && + cause.code === SdkErrorCode.ClientHttpNotImplemented && + typeof body === "string" && + message === prefix + body; + // Normalized wire headers and supported decoded candidates have no LF. + // Separating owners prevents their concatenation from spelling a credential. + const safeMessage = ownedPostBody + ? redactor.message(prefix, body) + : redactor.blocks([message]); + failure = new Error(safeMessage, { + cause: retainedTransportCause(cause, safeMessage), + }); + } let start = 0; - // The byte tail may start inside a UTF-8 code point. Retain whole units. if (stderrBytes > stderr.length) while (start < stderr.length && (stderr[start]! & 0xc0) === 0x80) start++; - const diagnostic = redactIdaTransportFailure( - new Error(stderr.subarray(start).toString("utf8")), - config, - ).message; + const capturedOutput = { + stdout: "", + stderr: redactor.blocks([stderr.subarray(start).toString("utf8")]), + truncated: stderrBytes > stderr.length - start, + stdout_bytes: 0, + stderr_bytes: stderrBytes, + }; + if (cause instanceof SdkError && cause.code === SdkErrorCode.RequestTimeout) + throw new AnalysisTimeoutError(operation, requestTimeoutMs, { + cause: failure, + capturedOutput, + }); + if (bufferLimit !== null) + throw new AnalysisResourceConstraintError( + operation, + "transport", + `Upstream IDA MCP reply exceeded the ${bufferLimit[1]}-byte transport limit; no partial Evidence was returned.`, + { max_response_bytes: Number(bufferLimit[1]) }, + { cause, capturedOutput }, + ); + if (stderrBytes === 0) throw failure; throw new ProviderAdapterError("ida", "transport", { cause: failure, diagnostics: { reason: failure.message }, - capturedOutput: { - stdout: "", - stderr: diagnostic, - truncated: stderrBytes > stderr.length - start, - stdout_bytes: 0, - stderr_bytes: stderrBytes, - }, + capturedOutput, }); }; const transport = @@ -188,9 +433,11 @@ export const createIdaMcpConnection = ( ...snapshotEnvironment(config.env), }, stderr: "pipe", + maxBufferSize: IDA_STDIO_MAX_MESSAGE_BYTES, }) : new StreamableHTTPClientTransport(new URL(config.url), { requestInit: { headers: config.headers, redirect: "error" }, + fetch: diagnosticFetch(redactor), }); // Keep a bounded diagnostic tail for genuine failures; never persist ambient variables. if (transport instanceof StdioClientTransport) @@ -205,26 +452,35 @@ export const createIdaMcpConnection = ( async connect() { await client .connect(transport, { timeout: config.timeoutMs }) - .catch(transportFailure); + .catch((cause: unknown) => transportFailure(cause, "ida_mcp_connect")); const catalog = await client .listTools(undefined, { timeout: config.timeoutMs, }) - .catch(transportFailure); + .catch((cause: unknown) => transportFailure(cause, "ida_mcp_catalog")); return catalog.tools.map(({ name }) => name); }, - async call(name, args) { + async call(name, args, options) { + const requestTimeoutMs = Math.min( + config.timeoutMs, + options?.timeoutMs ?? config.timeoutMs, + ); return decodeIdaToolResult( await client - .callTool({ name, arguments: args }, { timeout: config.timeoutMs }) - .catch(transportFailure), + .callTool({ name, arguments: args }, { timeout: requestTimeoutMs }) + .catch((cause: unknown) => + transportFailure(cause, name, requestTimeoutMs), + ), config.protocol, + redactor, ); }, async readResource(uri) { const response = await client .readResource({ uri }, { timeout: config.timeoutMs }) - .catch(transportFailure); + .catch((cause: unknown) => + transportFailure(cause, "ida_mcp_read_resource"), + ); const resource = z .object({ contents: z diff --git a/src/ida/IdaMcpEnvironment.test.ts b/src/ida/IdaMcpEnvironment.test.ts index b744779ae8..5e7a746d0c 100644 --- a/src/ida/IdaMcpEnvironment.test.ts +++ b/src/ida/IdaMcpEnvironment.test.ts @@ -43,12 +43,23 @@ input.on('line', (line) => { HOME: "/selected-home", PATH: "/selected-path", }; + const registrationEnvironment: Record = { + REA_REGISTRATION_CONTEXT: "registration", + }; + if (process.platform === "win32") { + const systemRoot = process.env.SYSTEMROOT; + if (systemRoot === undefined || systemRoot.length === 0) + throw new Error( + "The Windows Node fixture requires its actual SYSTEMROOT.", + ); + registrationEnvironment.SYSTEMROOT = systemRoot; + } for (const selected of [environment, {}]) { const connection = createIdaMcpConnection( { command: process.execPath, args: [script], - env: { REA_REGISTRATION_CONTEXT: "registration" }, + env: registrationEnvironment, mode: "attached", timeoutMs: 5000, }, diff --git a/src/ida/IdaNativeProtocol.test.ts b/src/ida/IdaNativeProtocol.test.ts index 466a09ad56..a1ba9fc0ce 100644 --- a/src/ida/IdaNativeProtocol.test.ts +++ b/src/ida/IdaNativeProtocol.test.ts @@ -779,18 +779,18 @@ describe("native IDA attached operation admission", () => { await client.close(); }); - it("does not admit the headless-only containment and document operations", async () => { - const { client, producer } = await fixture(); - for (const [operation, parameters] of [ - ["resolve_containing_procedure", { address: "0x1000" }], - ["list_documents", {}], - ] as const) { - const result = await client.execute(operation, parameters); + it("reports the bound document but not headless-only address resolution", async () => { + const { client, producer, target } = await fixture(); + const containing = await client.execute("resolve_containing_procedure", { + address: "0x1000", + }); + expect(containing.ok).toBe(false); + if (!containing.ok) + expect(containing.error._tag).toBe("AnalysisCapabilityUnavailableError"); - expect(result.ok).toBe(false); - if (!result.ok) - expect(result.error._tag).toBe("AnalysisCapabilityUnavailableError"); - } + const documents = await client.execute("list_documents", {}); + if (!documents.ok) throw documents.error; + expect(documents.value.result).toEqual([target.path]); expect( producer.calls.some(({ name }) => name === "lookup_funcs"), "nothing is dispatched upstream", diff --git a/src/ida/IdaOperationFailures.test.ts b/src/ida/IdaOperationFailures.test.ts index 1f8a83c6b4..8886f68611 100644 --- a/src/ida/IdaOperationFailures.test.ts +++ b/src/ida/IdaOperationFailures.test.ts @@ -1,5 +1,7 @@ -import { rm } from "node:fs/promises"; +import { rm, stat } from "node:fs/promises"; +import { dirname } from "node:path"; import { afterEach, describe, expect, it } from "vitest"; +import { z } from "zod"; import { createIdaTarget, RecordingIdaMcp, @@ -7,6 +9,7 @@ import { import type { JsonValue } from "../domain/jsonValue.js"; import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; import { AnalysisInputError } from "../domain/analysisErrorCore.js"; +import { createIdaDiagnosticRedactor } from "./IdaDiagnosticRedactor.js"; import { decodeIdaToolResult } from "./IdaMcpConnection.js"; import type { AnalysisOperation } from "../application/AnalysisProvider.js"; import { IdaSessionClient } from "./IdaSessionClient.js"; @@ -17,6 +20,193 @@ afterEach(async () => { await rm(root, { recursive: true, force: true }); }); +it("preserves a rejected headless decompile diagnostic with completed lookup evidence and releases its owned worker", async () => { + const { root, target } = await createIdaTarget(); + roots.push(root); + const producer = new RecordingIdaMcp(target, "headless"); + const reason = + "Invalid params: unexpected parameters: ['offset', 'max_lines']"; + const config = { + url: "http://127.0.0.1:12345/mcp", + headers: { + Authorization: "Token custom-fixture-token", + Cookie: "session=cookie-fixture-secret", + }, + mode: "headless" as const, + timeoutMs: 1000, + workspaceRoot: root, + }; + const redactor = createIdaDiagnosticRedactor(config); + producer.beforeCall = async (name) => { + if (name === "decompile") + decodeIdaToolResult( + { + isError: true, + content: [ + { + type: "text", + text: reason + " (custom-fixture-token / cookie-fixture-secret)", + }, + ], + }, + undefined, + redactor, + ); + }; + const safeReason = + reason + + " ([redacted transport credential] / [redacted transport credential])"; + const client = new IdaSessionClient(config, target, producer); + try { + const result = await client.execute("procedure_pseudo_code", { + procedure: "main", + }); + if (result.ok) throw new Error("Expected the upstream rejection"); + expect(projectAnalysisError(result.error)).toMatchObject({ + code: "unreadable_output", + details: { + diagnostics: { reason: safeReason }, + partial_observation: { + provider: { id: "ida" }, + operation: "procedure_pseudo_code", + normalized_result: { + status: "partial", + observations: [expect.objectContaining({ tool: "lookup_funcs" })], + target_binding: { + status: "not_verified", + reason: "IDA MCP tool failed: \n" + safeReason, + }, + }, + }, + }, + }); + expect(JSON.stringify(projectAnalysisError(result.error))).not.toContain( + "custom-fixture-token", + ); + expect(JSON.stringify(projectAnalysisError(result.error))).not.toContain( + "cookie-fixture-secret", + ); + expect( + producer.calls.find(({ name }) => name === "decompile"), + ).toMatchObject({ + args: { addr: "0x1000", offset: 0, max_lines: 100 }, + }); + } finally { + const database = producer.database; + expect((await client.close()).ok).toBe(true); + expect(producer.calls.filter(({ name }) => name === "idb_close")).toEqual([ + { name: "idb_close", args: { database, save: false } }, + ]); + expect(producer.closes).toBe(1); + await expect(stat(dirname(producer.inputPath))).rejects.toMatchObject({ + code: "ENOENT", + }); + } +}); + +describe("headless IDA tool diagnostics in partial evidence", () => { + const callerPath = "/caller/input?token=visible"; + const marker = "[redacted transport credential]"; + it.each([ + { + name: "a decoded Basic password matching the tool failure prefix", + password: "failed", + upstream: "Rejected decompile request for " + callerPath, + safeReason: "Rejected decompile request for " + callerPath, + forbiddenInBinding: ["failed"], + }, + { + name: "a JSON-encoded tool explanation", + password: "", + upstream: String.raw`{"reason":"\u003csecret\u003e","user":"user","path":"/caller/input?token=visible"}`, + safeReason: + '{"reason":"' + + marker + + '","user":"user","path":"/caller/input?token=visible"}', + forbiddenInBinding: ["", String.raw`\u003csecret\u003e`], + }, + ])( + "redacts $name before retaining the public target-binding reason", + async ({ password, upstream, safeReason, forbiddenInBinding }) => { + const { root, target } = await createIdaTarget(); + roots.push(root); + const producer = new RecordingIdaMcp(target, "headless"); + const config = { + url: "http://127.0.0.1:12345/mcp", + headers: { + Authorization: + "Basic " + + Buffer.from("user:" + password, "utf8").toString("base64"), + }, + mode: "headless" as const, + timeoutMs: 1000, + workspaceRoot: root, + }; + const redactor = createIdaDiagnosticRedactor(config); + producer.beforeCall = async (name) => { + if (name === "decompile") + decodeIdaToolResult( + { isError: true, content: [{ type: "text", text: upstream }] }, + undefined, + redactor, + ); + }; + const client = new IdaSessionClient(config, target, producer); + try { + const result = await client.execute("procedure_pseudo_code", { + procedure: "main", + }); + if (result.ok) throw new Error("Expected the producer's tool error"); + const projected = projectAnalysisError(result.error); + expect(projected).toMatchObject({ + code: "unreadable_output", + details: { + partial_observation: { + provider: { id: "ida" }, + operation: "procedure_pseudo_code", + normalized_result: { + status: "partial", + observations: [ + expect.objectContaining({ tool: "lookup_funcs" }), + ], + target_binding: { + status: "not_verified", + reason: expect.any(String), + }, + }, + }, + }, + }); + const details = z + .object({ + diagnostics: z.object({ reason: z.string() }), + partial_observation: z.object({ + normalized_result: z.object({ + target_binding: z.object({ reason: z.string() }), + }), + }), + }) + .parse(projected.details); + const bindingReason = + details.partial_observation.normalized_result.target_binding.reason; + for (const credential of forbiddenInBinding) + expect.soft(bindingReason).not.toContain(credential); + expect.soft(bindingReason).toContain(marker); + expect(bindingReason).toContain(callerPath); + expect.soft(details.diagnostics.reason).toBe(safeReason); + } finally { + expect((await client.close()).ok).toBe(true); + expect( + producer.calls.filter(({ name }) => name === "idb_close"), + ).toHaveLength(1); + await expect(stat(dirname(producer.inputPath))).rejects.toMatchObject({ + code: "ENOENT", + }); + } + }, + ); +}); + describe("headless IDA partial operation evidence", () => { it("retains completed rows when post-operation target identity changes", async () => { const { root, target } = await createIdaTarget(); diff --git a/src/ida/IdaOperationRunner.ts b/src/ida/IdaOperationRunner.ts index 576dac1e71..252bdbe3a1 100644 --- a/src/ida/IdaOperationRunner.ts +++ b/src/ida/IdaOperationRunner.ts @@ -311,14 +311,12 @@ export class IdaOperationRunner { const current: number = offset; if (this.protocol === "legacy") { const page = legacyPageSchema.parse( - await this.call("list_strings", { offset, count: 100 }), - ); - result.push( - ...page.data.map((item) => { - const s = legacyStringSchema.parse(item); - return { address: s.address, value: s.string }; - }), + await this.call("list_strings", { offset, count: 0 }), ); + for (const item of page.data) { + const s = legacyStringSchema.parse(item); + result.push({ address: s.address, value: s.string }); + } offset = page.next_offset; } else { const page = ( diff --git a/src/ida/IdaProvider.ts b/src/ida/IdaProvider.ts index 09011caa57..d2bf970600 100644 --- a/src/ida/IdaProvider.ts +++ b/src/ida/IdaProvider.ts @@ -68,13 +68,17 @@ export class IdaProvider implements AnalysisProviderCandidate { if (!privacy.available) return { status: "unavailable", - code: "unsupported_host", + code: privacy.code, reason: privacy.reason, diagnostics: { configured: true, mode: "headless", live_connection_probed: false, private_workspace_available: false, + windows_native_failure: { + stage: privacy.failureStage, + remediation: privacy.remediation, + }, }, }; } @@ -113,7 +117,11 @@ export class IdaProvider implements AnalysisProviderCandidate { status: "unsupported", code: "architecture_unsupported", reason: `REA's IDA adapter has no verified ${ - target.architecture === "mips" ? "MIPS" : "RISC-V" + target.architecture === "mips" + ? "MIPS" + : target.architecture === "riscv" + ? "RISC-V" + : "PowerPC" } lane. Select a supported Ghidra profile instead.`, diagnostics, }; diff --git a/src/ida/IdaProviderCapabilities.ts b/src/ida/IdaProviderCapabilities.ts index 0ccc016d25..1394310d93 100644 --- a/src/ida/IdaProviderCapabilities.ts +++ b/src/ida/IdaProviderCapabilities.ts @@ -32,7 +32,6 @@ export type IdaOperation = (typeof IDA_OPERATIONS)[number]; /** Operations only the modern headless profile implements. */ const HEADLESS_ONLY_OPERATIONS: ReadonlySet = new Set([ "resolve_containing_procedure", - "list_documents", ]); /** Whether the configured profile admits an operation. */ diff --git a/src/ida/IdaSessionClient.test.ts b/src/ida/IdaSessionClient.test.ts index 1a35dd7c78..c6bffd2e09 100644 --- a/src/ida/IdaSessionClient.test.ts +++ b/src/ida/IdaSessionClient.test.ts @@ -1,6 +1,6 @@ import { readFile, readdir, rm } from "node:fs/promises"; import { sep } from "node:path"; -import { afterEach, describe, expect, it } from "vitest"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { z } from "zod"; import { createIdaTarget, @@ -11,11 +11,34 @@ import { functionDossierSchema } from "../domain/hopperValues.js"; import { jsonValueSchema, type JsonValue } from "../domain/jsonValue.js"; import { ProviderAdapterError } from "../domain/providerAdapterError.js"; import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; +import { WindowsPrivateRuntime } from "../windows/WindowsPrivateRuntime.js"; const roots: string[] = []; +const windowsRuntimes = new Set(); +beforeEach(() => { + if (process.platform !== "win32") return; + const create = WindowsPrivateRuntime.create; + vi.spyOn(WindowsPrivateRuntime, "create").mockImplementation( + (parent, prefix) => { + const runtime = create(parent, prefix); + windowsRuntimes.add(runtime); + return runtime; + }, + ); +}); afterEach(async () => { - for (const root of roots.splice(0)) - await rm(root, { recursive: true, force: true }); + try { + // These pure producer fixtures spawn no IDA worker. After the retention + // assertions, release only the actual native owners this test created. + for (const runtime of windowsRuntimes) { + await runtime.close(); + windowsRuntimes.delete(runtime); + } + for (const root of roots.splice(0)) + await rm(root, { recursive: true, force: true }); + } finally { + vi.restoreAllMocks(); + } }); /** A failed close whose diagnostic names exactly these unverified sessions. */ const reportsSessions = (...sessions: readonly object[]) => ({ @@ -130,7 +153,28 @@ describe("headless IDA producer pagination", () => { }); }); -describe("attached IDA observation semantics", () => { +describe("attached IDA readiness and queued cancellation", () => { + it("allows ready-only work after an attached provider has initialized", async () => { + const { producer, client } = await fixture("attached"); + await expect(client.executeIfReady("list_procedures", {})).resolves.toBe( + undefined, + ); + + const startup = await client.execute("list_procedures", {}); + if (!startup.ok) throw startup.error; + const result = await client.executeIfReady("list_procedures", {}); + if (result === undefined) + throw new Error("expected ready attached IDA dispatch"); + if (!result.ok) throw result.error; + expect(result.value.result).toEqual([ + expect.objectContaining({ address: "0x1000", value: "main" }), + ]); + expect( + producer.calls.filter(({ name }) => name === "idb_open"), + ).toHaveLength(0); + await client.close(); + }); + it("returns cold and pending startup calls immediately, then cancels queued work before dispatch", async () => { const { producer, client } = await fixture("headless"); await expect(client.executeIfReady("list_procedures", {})).resolves.toBe( @@ -210,7 +254,9 @@ describe("attached IDA observation semantics", () => { ); await client.close(); }); +}); +describe("attached IDA input identity and validation", () => { it("accepts a normalized spelling of the bound document path", async () => { const { target, producer, client } = await fixture(); const equivalent = @@ -1495,6 +1541,38 @@ it("binds retained transport diagnostics to the requested operation and recovery await client.close(); }); +describe("attached IDA legacy string inventory", () => { + it("fetches legacy strings and literal searches in single upstream calls", async () => { + const { producer, client } = await fixture(); + const all = Array.from({ length: 250 }, (_, i) => ({ + address: `0x${(0x3000 + i).toString(16)}`, + length: 5, + string: `hello-${i}`, + })); + producer.overrideCall = (name, args) => { + if (name !== "list_strings") return undefined; + const offset = Number(args.offset ?? 0); + const count = Number(args.count ?? 100); + const data = count === 0 ? all : all.slice(offset, offset + count); + return { + data, + next_offset: + count === 0 || offset + count >= all.length ? null : offset + count, + }; + }; + const listed = await client.execute("list_strings", {}); + expect(listed.ok && listed.value.result).toHaveLength(250); + expect( + producer.calls.filter(({ name }) => name === "list_strings")[0]?.args + .count, + ).toBe(0); + expect( + producer.calls.filter(({ name }) => name === "list_strings"), + ).toHaveLength(1); + await client.close(); + }); +}); + it("retains transport diagnostics on a cancellation regardless of stderr", async () => { const { producer, client } = await fixture("attached"); await client.execute("health", {}); diff --git a/src/ida/IdaSessionClient.ts b/src/ida/IdaSessionClient.ts index d157b5209c..3dac1112ec 100644 --- a/src/ida/IdaSessionClient.ts +++ b/src/ida/IdaSessionClient.ts @@ -162,6 +162,7 @@ export class IdaSessionClient implements AnalysisClient { #databaseCloseCompleted = false; #openAttempted = false; #openCompleted = false; + #ready = false; #blockingSessions: readonly UpstreamSession[] | undefined; #openRejected = false; #callGraphAdvertised = false; @@ -197,11 +198,10 @@ export class IdaSessionClient implements AnalysisClient { parameters: Readonly>, options?: ExecutionOptions, ) { - if (!this.#openCompleted || this.#closed || this.#closing) + if (!this.#ready || this.#closed || this.#closing) return Promise.resolve(undefined); const call = this.#tail.then(() => { - if (!this.#openCompleted || this.#closed || this.#closing) - return undefined; + if (!this.#ready || this.#closed || this.#closing) return undefined; if (wasCancelled(options)) return err(new AnalysisCancelledError(operation)); return this.#execute(operation, parameters, options, true); @@ -601,6 +601,7 @@ export class IdaSessionClient implements AnalysisClient { ); this.#database = opened.session.session_id; } + this.#ready = true; } async #nativeMetadata(): Promise { diff --git a/src/inspector/V8InspectorEndpoint.test.ts b/src/inspector/V8InspectorEndpoint.test.ts new file mode 100644 index 0000000000..83a0d20fcb --- /dev/null +++ b/src/inspector/V8InspectorEndpoint.test.ts @@ -0,0 +1,69 @@ +import { createServer } from "node:http"; + +import { expect, it } from "vitest"; + +import { V8InspectorProvider } from "./V8InspectorProvider.js"; + +it("keeps connectable siblings when a target omits its direct socket", async () => { + const server = createServer((request, response) => { + response.setHeader("content-type", "application/json"); + if (request.url === "/json/version") { + response.end( + JSON.stringify({ + Browser: "Node.js/v24.18.0", + "Protocol-Version": "1.3", + }), + ); + return; + } + if (request.url === "/json/list") { + const address = server.address(); + if (address === null || typeof address === "string") { + response.writeHead(500).end(); + return; + } + response.end( + JSON.stringify([ + { + id: "connectable", + type: "node", + url: "https://example.test/entry.js", + webSocketDebuggerUrl: `ws://127.0.0.1:${String(address.port)}/connectable`, + }, + { + id: "anonymous", + type: "node", + url: "https://example.test/eval.js", + }, + ]), + ); + return; + } + response.writeHead(404).end(); + }); + + try { + await new Promise((resolve) => + server.listen(0, "127.0.0.1", resolve), + ); + const address = server.address(); + if (address === null || typeof address === "string") + throw new Error("Inspector fixture did not bind a TCP port"); + + const result = await new V8InspectorProvider().listTargets({ + inspector_endpoint: `http://127.0.0.1:${String(address.port)}`, + }); + if (!result.ok) + throw new Error(result.error.message, { cause: result.error }); + + expect(result).toMatchObject({ + ok: true, + value: { + targets: [{ target_id: "connectable" }], + excluded: { unconnectable: 1 }, + }, + }); + } finally { + await new Promise((resolve) => server.close(() => resolve())); + } +}); diff --git a/src/inspector/V8InspectorEndpoint.ts b/src/inspector/V8InspectorEndpoint.ts index 53f15e0736..a3dcd2357e 100644 --- a/src/inspector/V8InspectorEndpoint.ts +++ b/src/inspector/V8InspectorEndpoint.ts @@ -23,7 +23,10 @@ const targetSchema = z type: z.string().min(1), url: z.string(), attached: z.boolean().nullable().optional(), - webSocketDebuggerUrl: z.string().min(1), + // Node may publish anonymous/evaluated targets without a direct socket. + // Keep malformed values local to their target so one row cannot discard + // otherwise usable siblings from the endpoint inventory. + webSocketDebuggerUrl: z.unknown().optional(), }) .passthrough(); const targetsSchema = z.array(targetSchema); @@ -33,7 +36,7 @@ export interface V8InspectorTarget { readonly type: string; readonly url: string; readonly attached: boolean | null; - readonly webSocketUrl: string; + readonly webSocketUrl: string | null; } export interface V8InspectorDiscovery { @@ -68,7 +71,6 @@ export const discoverV8Inspector = async ( endpoint, target.id, target.webSocketDebuggerUrl, - operation, ), })), }; @@ -77,16 +79,14 @@ export const discoverV8Inspector = async ( const validatedInspectorWebSocket = ( endpoint: string, targetId: string, - reported: string, - operation: BrowserObservationOperation, -): string => { + reported: unknown, +): string | null => { + if (typeof reported !== "string" || reported.length === 0) return null; let socket: URL; try { socket = new URL(reported); - } catch (cause: unknown) { - throw new BrowserObservationError(operation, "invalid_endpoint_response", { - cause, - }); + } catch { + return null; } const trusted = new URL(endpoint); const allowedPaths = new Set([ @@ -102,12 +102,16 @@ const validatedInspectorWebSocket = ( urlDeclaresQueryOrFragment(socket) || !allowedPaths.has(socket.pathname) ) - throw new BrowserObservationError(operation, "invalid_endpoint_response"); + return null; socket.hostname = trusted.hostname; return socket.href; }; /** Authorized target plus its durable location. */ -export interface AuthorizedV8InspectorTarget extends V8InspectorTarget { +export interface AuthorizedV8InspectorTarget extends Omit< + V8InspectorTarget, + "webSocketUrl" +> { + readonly webSocketUrl: string; readonly location: JavaScriptRuntimeTargetLocation; } diff --git a/src/inspector/V8InspectorProvider.partial.test.ts b/src/inspector/V8InspectorProvider.partial.test.ts index 0ee635249b..6b1ac23c85 100644 --- a/src/inspector/V8InspectorProvider.partial.test.ts +++ b/src/inspector/V8InspectorProvider.partial.test.ts @@ -4,7 +4,7 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { pathToFileURL } from "node:url"; -import { expect, it } from "vitest"; +import { expect, it, vi } from "vitest"; import { WebSocketServer } from "ws"; import { z } from "zod"; @@ -14,9 +14,116 @@ import { V8InspectorProvider } from "./V8InspectorProvider.js"; const failureCases = [ ["disconnect", "disconnected"], ["protocol failure", "protocol_error"], + ["idle protocol failure", "protocol_error"], ["cancellation", "cancelled"], ] as const; +const createPartialInspectorServer = ( + failure: (typeof failureCases)[number][0], + scriptUrl: string, + reportedScriptUrl: string, +) => { + let signalScriptSent: (() => void) | undefined; + const scriptSent = new Promise((resolve) => { + signalScriptSent = resolve; + }); + const server = createServer((request, response) => { + response.setHeader("content-type", "application/json"); + if (request.url === "/json/version") { + response.end( + JSON.stringify({ + Browser: "Node.js/v24.18.0", + "Protocol-Version": "1.3", + "V8-Version": "12.0", + }), + ); + return; + } + if (request.url === "/json/list") { + const address = server.address(); + if (address === null || typeof address === "string") { + response.writeHead(500).end(); + return; + } + response.end( + JSON.stringify([ + { + id: "target-1", + type: "node", + url: scriptUrl, + attached: false, + webSocketDebuggerUrl: `ws://127.0.0.1:${String(address.port)}/target-1`, + }, + ]), + ); + return; + } + response.writeHead(404).end(); + }); + const sockets = new WebSocketServer({ server, path: "/target-1" }); + sockets.on("connection", (socket) => { + socket.on("message", (raw) => { + const parsed = safeParseJson(raw.toString()); + if (!parsed.ok) throw parsed.cause; + const command = z + .object({ id: z.number(), method: z.string() }) + .parse(parsed.value); + if (command.method === "Runtime.enable") { + socket.send( + JSON.stringify({ + method: "Runtime.executionContextCreated", + params: { + context: { id: 1, name: "main", origin: reportedScriptUrl }, + }, + }), + ); + socket.send( + JSON.stringify({ + method: "Debugger.scriptParsed", + params: { + scriptId: "1", + url: reportedScriptUrl, + executionContextId: 1, + hash: "captured-before-failure", + length: 24, + isModule: false, + }, + }), + ); + socket.send(JSON.stringify({ id: command.id, result: {} })); + return; + } + if (command.method === "Debugger.enable") { + // Receiving the next command proves the preceding events and reply + // were consumed; cancellation needs no timing assumption. + signalScriptSent?.(); + if (failure === "protocol failure") { + socket.send(JSON.stringify({ id: command.id, result: "malformed" })); + } else { + socket.send(JSON.stringify({ id: command.id, result: {} })); + if (failure === "disconnect") socket.close(); + if (failure === "idle protocol failure") { + setTimeout(() => socket.send("{malformed-json"), 20); + setTimeout( + () => + socket.send( + JSON.stringify({ + method: "Debugger.scriptParsed", + params: { scriptId: "late", url: scriptUrl }, + }), + ), + 40, + ); + } + } + return; + } + socket.send(JSON.stringify({ id: command.id, result: {} })); + }); + }); + return { server, sockets, scriptSent }; +}; + it.each(failureCases)( "preserves captured scripts and contexts after %s", async (failure, reason) => { @@ -26,93 +133,11 @@ it.each(failureCases)( const reportedScriptUrl = failure === "cancellation" ? "" : scriptUrl; await writeFile(scriptPath, "globalThis.ready = true;\n"); - let signalScriptSent: (() => void) | undefined; - const scriptSent = new Promise((resolve) => { - signalScriptSent = resolve; - }); - const server = createServer((request, response) => { - response.setHeader("content-type", "application/json"); - if (request.url === "/json/version") { - response.end( - JSON.stringify({ - Browser: "Node.js/v24.18.0", - "Protocol-Version": "1.3", - "V8-Version": "12.0", - }), - ); - return; - } - if (request.url === "/json/list") { - const address = server.address(); - if (address === null || typeof address === "string") { - response.writeHead(500).end(); - return; - } - response.end( - JSON.stringify([ - { - id: "target-1", - type: "node", - url: scriptUrl, - attached: false, - webSocketDebuggerUrl: `ws://127.0.0.1:${String(address.port)}/target-1`, - }, - ]), - ); - return; - } - response.writeHead(404).end(); - }); - const sockets = new WebSocketServer({ server, path: "/target-1" }); - sockets.on("connection", (socket) => { - socket.on("message", (raw) => { - const parsed = safeParseJson(raw.toString()); - if (!parsed.ok) throw parsed.cause; - const command = z - .object({ id: z.number(), method: z.string() }) - .parse(parsed.value); - if (command.method === "Runtime.enable") { - socket.send( - JSON.stringify({ - method: "Runtime.executionContextCreated", - params: { - context: { id: 1, name: "main", origin: reportedScriptUrl }, - }, - }), - ); - socket.send( - JSON.stringify({ - method: "Debugger.scriptParsed", - params: { - scriptId: "1", - url: reportedScriptUrl, - executionContextId: 1, - hash: "captured-before-failure", - length: 24, - isModule: false, - }, - }), - ); - socket.send(JSON.stringify({ id: command.id, result: {} })); - return; - } - if (command.method === "Debugger.enable") { - // Receiving the next command proves the preceding events and reply - // were consumed; cancellation needs no timing assumption. - signalScriptSent?.(); - if (failure === "protocol failure") { - socket.send( - JSON.stringify({ id: command.id, result: "malformed" }), - ); - } else { - socket.send(JSON.stringify({ id: command.id, result: {} })); - if (failure === "disconnect") socket.close(); - } - return; - } - socket.send(JSON.stringify({ id: command.id, result: {} })); - }); - }); + const { server, sockets, scriptSent } = createPartialInspectorServer( + failure, + scriptUrl, + reportedScriptUrl, + ); try { await new Promise((resolve) => @@ -123,13 +148,19 @@ it.each(failureCases)( throw new Error("Inspector fixture did not bind a TCP port"); const controller = new AbortController(); + const timerSpy = + failure === "disconnect" ? vi.spyOn(globalThis, "setTimeout") : null; + const clearSpy = + failure === "disconnect" ? vi.spyOn(globalThis, "clearTimeout") : null; const executionOptions = - failure === "cancellation" ? { signal: controller.signal } : {}; + failure === "cancellation" || failure === "disconnect" + ? { signal: controller.signal } + : {}; const observing = new V8InspectorProvider().observe( { inspector_endpoint: `http://127.0.0.1:${String(address.port)}`, target_id: "target-1", - observation_ms: 10_000, + observation_ms: failure === "disconnect" ? 12_345 : 10_000, }, executionOptions, ); @@ -139,6 +170,17 @@ it.each(failureCases)( } const result = await observing; + if (failure === "disconnect") { + const observationTimer = timerSpy?.mock.results.find( + (result, index) => timerSpy.mock.calls[index]?.[1] === 12_345, + )?.value; + expect(observationTimer).toBeDefined(); + expect( + clearSpy?.mock.calls.some(([timer]) => timer === observationTimer), + ).toBe(true); + expect(controller.signal.aborted).toBe(false); + } + expect(result.ok).toBe(false); if (result.ok) throw new Error("Expected the Inspector operation to fail"); @@ -167,6 +209,7 @@ it.each(failureCases)( }, }); } finally { + vi.restoreAllMocks(); await new Promise((resolve) => sockets.close(() => resolve())); await new Promise((resolve) => server.close(() => resolve())); await rm(root, { recursive: true, force: true }); diff --git a/src/inspector/V8InspectorProvider.ts b/src/inspector/V8InspectorProvider.ts index 06a413871d..bf89e227e4 100644 --- a/src/inspector/V8InspectorProvider.ts +++ b/src/inspector/V8InspectorProvider.ts @@ -94,6 +94,7 @@ export class V8InspectorProvider implements JavaScriptRuntimeObservationPort { ); const allowed: AuthorizedV8InspectorTarget[] = []; const excluded = createInspectorExclusionCounts(); + let unconnectable = 0; for (const target of discovery.targets) { const decision = await authorizeRuntimeTargetLocation(target.url, { type: target.type, @@ -107,7 +108,18 @@ export class V8InspectorProvider implements JavaScriptRuntimeObservationPort { excluded.unsupported_location += 1; continue; } - allowed.push({ ...target, location: decision.location }); + if (target.webSocketUrl === null) { + unconnectable += 1; + continue; + } + allowed.push({ + id: target.id, + type: target.type, + url: target.url, + attached: target.attached, + webSocketUrl: target.webSocketUrl, + location: decision.location, + }); } allowed.sort((left, right) => left.id < right.id ? -1 : left.id > right.id ? 1 : 0, @@ -116,7 +128,7 @@ export class V8InspectorProvider implements JavaScriptRuntimeObservationPort { javascriptRuntimeTargetListSchema.parse({ runtime: discovery.runtime, targets: allowed.map(projectTarget), - excluded: { ...excluded, unconnectable: 0 }, + excluded: { ...excluded, unconnectable }, limitations: describeInspectorTargetLimitations( allowed.some((target) => target.attached === null), ), @@ -367,7 +379,23 @@ const authorizedTarget = async ( "observe_javascript_runtime", "target_not_allowed", ); - return { ...target, location: decision.location }; + if (target.webSocketUrl === null) + throw new BrowserObservationError( + "observe_javascript_runtime", + "invalid_endpoint_response", + { + detail: + "The selected Inspector target has no validated direct WebSocket endpoint.", + }, + ); + return { + id: target.id, + type: target.type, + url: target.url, + attached: target.attached, + webSocketUrl: target.webSocketUrl, + location: decision.location, + }; }; const assertRuntimeKind = ( @@ -515,20 +543,35 @@ const waitForCapture = async ( signal?: AbortSignal, ): Promise => { let removeDisconnect = (): void => undefined; + let removeProtocolFailure = (): void => undefined; const disconnected = new Promise((_resolve, reject) => { removeDisconnect = connection.onDisconnect((error) => reject(error)); }); + const protocolFailure = new Promise((_resolve, reject) => { + removeProtocolFailure = connection.onProtocolFailure((error) => + reject(error), + ); + }); + const timerOwner = new AbortController(); + const timerSignal = + signal === undefined + ? timerOwner.signal + : AbortSignal.any([signal, timerOwner.signal]); try { await Promise.race([ delayWithCancellation( observationMs, "observe_javascript_runtime", - signal, + timerSignal, ), disconnected, + protocolFailure, ]); } finally { + // Cancel only this wait's timer. The caller owns and retains its signal. + timerOwner.abort(); removeDisconnect(); + removeProtocolFailure(); } }; diff --git a/src/javascript/analysis/JavaScriptAnalysisWorker.ts b/src/javascript/analysis/JavaScriptAnalysisWorker.ts index de7ac6c42d..b9471abb96 100644 --- a/src/javascript/analysis/JavaScriptAnalysisWorker.ts +++ b/src/javascript/analysis/JavaScriptAnalysisWorker.ts @@ -278,7 +278,7 @@ export class JavaScriptAnalysisWorker { error: new AnalysisResourceConstraintError( OPERATION, response.resource ?? "memory", - `${file.path}: ${response.message}`, + response.message, { ...this.#limits(), ...response.limits }, ), }); diff --git a/src/javascript/analysis/JavaScriptAnalysisWorkerSchemas.test.ts b/src/javascript/analysis/JavaScriptAnalysisWorkerSchemas.test.ts new file mode 100644 index 0000000000..6a461031da --- /dev/null +++ b/src/javascript/analysis/JavaScriptAnalysisWorkerSchemas.test.ts @@ -0,0 +1,75 @@ +import { expect, it } from "vitest"; + +import { analyzeJavaScriptStaticSource } from "../../domain/javascript/javascriptStaticAnalysis.js"; +import { javaScriptWorkerStaticAnalysisSchema } from "./JavaScriptAnalysisWorkerSchemas.js"; + +it("retains candidate endpoint interpretation across the worker boundary", () => { + const facts = analyzeJavaScriptStaticSource(` + client.get("settingKey"); + fetch("/api"); + `); + expect(facts.endpoints).toEqual([ + expect.objectContaining({ + value: "settingKey", + interpretation: "candidate", + }), + expect.objectContaining({ value: "/api" }), + ]); + expect(facts.endpoints[1]).not.toHaveProperty("interpretation"); + const serialized = JSON.parse(JSON.stringify(facts)) as unknown; + expect(javaScriptWorkerStaticAnalysisSchema.parse(serialized)).toEqual(facts); + expect( + javaScriptWorkerStaticAnalysisSchema.safeParse({ + ...facts, + endpoints: facts.endpoints.map((endpoint) => ({ + ...endpoint, + interpretation: "confirmed", + })), + }).success, + ).toBe(false); +}); + +it("versions fingerprints through the worker schema and rejects v1", () => { + const facts = + analyzeJavaScriptStaticSource(`globalThis.webpackChunkApp.push([[1], { + 17(module, exports) { const key = selectKey(); exports.read = value[key]; } + }]);`); + expect(facts.bundler_registrations[0]?.modules[0]).toMatchObject({ + structural_fingerprint_algorithm: "babel-ast-v2", + structural_fingerprint_sha256: expect.stringMatching(/^[a-f0-9]{64}$/u), + }); + expect( + javaScriptWorkerStaticAnalysisSchema.safeParse( + JSON.parse(JSON.stringify(facts)) as unknown, + ).success, + ).toBe(true); + const old = { + ...facts, + bundler_registrations: facts.bundler_registrations.map((registration) => ({ + ...registration, + modules: registration.modules.map((module) => ({ + ...module, + structural_fingerprint_algorithm: "babel-ast-v1", + })), + })), + }; + expect(javaScriptWorkerStaticAnalysisSchema.safeParse(old).success).toBe( + false, + ); +}); + +it("accepts candidate endpoint interpretations from static analysis", () => { + const facts = analyzeJavaScriptStaticSource(`http.request("POST", target);`); + expect(facts.endpoints).toContainEqual( + expect.objectContaining({ + kind: "network", + value: "POST", + interpretation: "candidate", + }), + ); + expect( + javaScriptWorkerStaticAnalysisSchema.safeParse( + JSON.parse(JSON.stringify(facts)) as unknown, + ).success, + ).toBe(true); +}); diff --git a/src/javascript/analysis/JavaScriptAnalysisWorkerSchemas.ts b/src/javascript/analysis/JavaScriptAnalysisWorkerSchemas.ts index 40073f1ae9..63ad24b629 100644 --- a/src/javascript/analysis/JavaScriptAnalysisWorkerSchemas.ts +++ b/src/javascript/analysis/JavaScriptAnalysisWorkerSchemas.ts @@ -1,6 +1,7 @@ import { z } from "zod"; import { digestSchema } from "../../domain/digests.js"; import type { JavaScriptStaticAnalysis } from "../../domain/javascript/javascriptStaticAnalysisTypes.js"; +import { JAVASCRIPT_AST_FINGERPRINT_ALGORITHM } from "../../domain/javascript/javascriptStaticAnalysisTypes.js"; import type { JavaScriptSemanticProperty, JavaScriptSemanticValue, @@ -327,6 +328,7 @@ export const javaScriptWorkerStaticAnalysisSchema = z kind: z.enum(["route", "network"]), value: z.string(), mechanism: z.string(), + interpretation: z.literal("candidate").exactOptional(), }), ), storage: z.array( @@ -363,7 +365,9 @@ export const javaScriptWorkerStaticAnalysisSchema = z exports: strings, location: range, structural_fingerprint_sha256: digestSchema, - structural_fingerprint_algorithm: z.literal("babel-ast-v1"), + structural_fingerprint_algorithm: z.literal( + JAVASCRIPT_AST_FINGERPRINT_ALGORITHM, + ), }), ), location: range, diff --git a/src/javascript/analysis/JavaScriptAnalysisWorkerServer.ts b/src/javascript/analysis/JavaScriptAnalysisWorkerServer.ts index 0e9f15b59a..e3d0a8bc2e 100644 --- a/src/javascript/analysis/JavaScriptAnalysisWorkerServer.ts +++ b/src/javascript/analysis/JavaScriptAnalysisWorkerServer.ts @@ -189,7 +189,13 @@ const analyzeJob = async ( cause instanceof AnalysisResourceConstraintError ? cause.reportedLimits : null, - message: cause instanceof Error ? cause.message : String(cause), + // The parent owns resource-error formatting; send its canonical reason. + message: + cause instanceof AnalysisResourceConstraintError + ? cause.reason + : cause instanceof Error + ? cause.message + : String(cause), }); } }; diff --git a/src/javascript/recovery/RecoveryFiles.test.ts b/src/javascript/recovery/RecoveryFiles.test.ts new file mode 100644 index 0000000000..574699abd5 --- /dev/null +++ b/src/javascript/recovery/RecoveryFiles.test.ts @@ -0,0 +1,55 @@ +import { execFile } from "node:child_process"; +import { writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { promisify } from "node:util"; +import { expect, it, vi } from "vitest"; +import { AnalysisOutputError } from "../../domain/analysisErrorCore.js"; +import { OwnedFileHandle } from "../../filesystem/OwnedFileHandle.js"; +import { + openRegularFile, + RegularFileAdmissionFailure, +} from "../../filesystem/RegularFile.js"; +import { readWithoutFifoWriter } from "../../../tests/fixtures/fifoInput.js"; +import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js"; +import { readRecoveryFile } from "./RecoveryFiles.js"; + +vi.mock("../../filesystem/RegularFile.js", async (importOriginal) => { + const actual = + await importOriginal(); + return { ...actual, openRegularFile: vi.fn(actual.openRegularFile) }; +}); + +it.skipIf(process.platform === "win32")( + "rejects a named pipe without waiting for a writer", + async () => { + const root = await createTestTempDirectory("rea-recovery-fifo-"); + const regular = join(root, "regular.js"); + await writeFile(regular, "module.exports = 1;\n"); + await expect(readRecoveryFile(regular, 1024)).resolves.toHaveLength(20); + + const fifoPath = join(root, "sample.js"); + await promisify(execFile)("mkfifo", [fifoPath]); + const outcome = await readWithoutFifoWriter(fifoPath, () => + readRecoveryFile(fifoPath, 1024).catch((cause: unknown) => cause), + ); + if (outcome.state !== "completed") + throw new Error("Recovery input read waited for a FIFO writer"); + expect(outcome.result).toBeInstanceOf(AnalysisOutputError); + }, +); + +it("keeps a failed admission close owned when cancellation is also requested", async () => { + const owner = new OwnedFileHandle({ fd: -1 } as never); + const failure = new RegularFileAdmissionFailure( + "/selected.js", + new Error("cancelled during admission"), + new Error("close failed"), + owner, + ); + vi.mocked(openRegularFile).mockRejectedValueOnce(failure); + const controller = new AbortController(); + controller.abort(); + await expect( + readRecoveryFile("/selected.js", 1024, controller.signal), + ).rejects.toBe(failure); +}); diff --git a/src/javascript/recovery/RecoveryFiles.ts b/src/javascript/recovery/RecoveryFiles.ts index c64f011b48..6cea1b81b6 100644 --- a/src/javascript/recovery/RecoveryFiles.ts +++ b/src/javascript/recovery/RecoveryFiles.ts @@ -1,6 +1,5 @@ import { createHash } from "node:crypto"; -import { constants } from "node:fs"; -import { lstat, mkdir, open, opendir, writeFile } from "node:fs/promises"; +import { lstat, mkdir, opendir, writeFile } from "node:fs/promises"; import { dirname, isAbsolute, join, relative, resolve, sep } from "node:path"; import { TextDecoder } from "node:util"; import { normalizeArtifactPath } from "../../artifacts/ArtifactPaths.js"; @@ -10,6 +9,11 @@ import { AnalysisOutputError, AnalysisTimeoutError, } from "../../domain/analysisErrorCore.js"; +import { + NonRegularFileReadError, + openRegularFile, + RegularFileAdmissionFailure, +} from "../../filesystem/RegularFile.js"; import { RECOVERY_LIMITS } from "./WakaruRelease.js"; const OPERATION = "recover_javascript_sources"; @@ -25,11 +29,26 @@ const scanRecoveryFile = async ( consume: (chunk: Buffer) => void, signal?: AbortSignal, ) => { - const handle = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW); + // A FIFO without a writer must be rejected before it can block the open. + const handle = await openRegularFile(path, { + symlinks: "reject", + signal, + }).catch((cause: unknown) => { + // A descriptor whose close failed stays owned by its error, even when + // cancellation was also requested. + if (cause instanceof RegularFileAdmissionFailure) throw cause; + checkRecoveryCancellation(signal); + if (cause instanceof NonRegularFileReadError) + throw new AnalysisOutputError( + OPERATION, + `Expected a regular file no larger than ${String(maximum)} bytes: ${path}`, + ); + throw cause; + }); const hash = createHash("sha256"); try { const before = await handle.stat(); - if (!before.isFile() || before.size > maximum) + if (before.size > maximum) throw new AnalysisOutputError( OPERATION, `Expected a regular file no larger than ${String(maximum)} bytes: ${path}`, @@ -167,6 +186,11 @@ export const snapshotRecoveryInputs = async ( ? "bundle.js" : relative(commonDirectory, resolve(path)).split(sep).join("/"); try { + // Wakaru's multi-input public paths interpret backslashes as separators. + if (relativePath.includes("\\")) + throw new TypeError( + "Wakaru cannot preserve literal backslashes in input paths", + ); if (normalizeArtifactPath(relativePath) !== relativePath) throw new TypeError( "Input names must already use NFC Unicode spelling", diff --git a/src/jeb/JebMcpConnection.test.ts b/src/jeb/JebMcpConnection.test.ts index 284f04f07d..b5bc163d46 100644 --- a/src/jeb/JebMcpConnection.test.ts +++ b/src/jeb/JebMcpConnection.test.ts @@ -5,7 +5,171 @@ import { decodeJebToolResult, createStreamableHttpJebMcpConnection, } from "./JebMcpConnection.js"; -import { AnalysisProtocolError } from "../domain/analysisErrorCore.js"; +import { + AnalysisCancelledError, + AnalysisProtocolError, + AnalysisTimeoutError, +} from "../domain/analysisErrorCore.js"; + +const jebTools = [ + "get_client_information", + "open_project", + "list_units", + "decompile_code_item", +] as const; + +const waitForRequestCount = async ( + readCount: () => number, + expected: number, + method: string, +): Promise => { + const deadline = performance.now() + 1_000; + while (readCount() < expected && performance.now() < deadline) + await new Promise((wake) => setTimeout(wake, 5)); + if (readCount() < expected) + throw new Error(`Expected ${expected} ${method} requests`); +}; + +const startJebHttpFixture = ( + responseDelayMs: number, + catalogRefreshDelayMs = 0, +) => { + let callCount = 0; + let catalogCount = 0; + const pendingTimers = new Set(); + const server = createServer((request, response) => { + const chunks: Buffer[] = []; + request.on("data", (chunk: Buffer) => chunks.push(chunk)); + request.on("end", () => { + if (chunks.length === 0) { + response.writeHead(202).end(); + return; + } + const message: { + readonly id?: string | number; + readonly method: string; + readonly params?: Record; + } = JSON.parse(Buffer.concat(chunks).toString("utf8")); + if (message.method === "notifications/initialized") { + response.writeHead(202).end(); + return; + } + if (message.method === "notifications/cancelled") { + response.writeHead(202).end(); + return; + } + if (message.method === "initialize") { + response.writeHead(200, { "Content-Type": "application/json" }).end( + JSON.stringify({ + jsonrpc: "2.0", + id: message.id, + result: { + protocolVersion: message.params?.protocolVersion, + capabilities: { tools: {} }, + serverInfo: { name: "jeb-timeout-fixture", version: "1" }, + }, + }), + ); + return; + } + if (message.method === "tools/list") { + catalogCount++; + let responseClosed = false; + response.on("close", () => { + if (!response.writableEnded) responseClosed = true; + }); + const sendCatalog = () => { + if (responseClosed) return; + response.writeHead(200, { "Content-Type": "application/json" }).end( + JSON.stringify({ + jsonrpc: "2.0", + id: message.id, + result: { + tools: jebTools.map((name) => ({ + name, + inputSchema: { type: "object", properties: {} }, + })), + }, + }), + ); + }; + if (catalogCount > 1 && catalogRefreshDelayMs > 0) { + const timer = setTimeout(() => { + pendingTimers.delete(timer); + sendCatalog(); + }, catalogRefreshDelayMs); + pendingTimers.add(timer); + } else { + sendCatalog(); + } + return; + } + if (message.method !== "tools/call") { + response.writeHead(202).end(); + return; + } + callCount++; + let responseClosed = false; + response.on("close", () => { + if (!response.writableEnded) { + responseClosed = true; + } + }); + const timer = setTimeout(() => { + pendingTimers.delete(timer); + if (responseClosed) return; + response.writeHead(200, { "Content-Type": "application/json" }).end( + JSON.stringify({ + jsonrpc: "2.0", + id: message.id, + result: { + structuredContent: { success: true, text: "finished" }, + content: [{ type: "text", text: '{"success":true}' }], + }, + }), + ); + }, responseDelayMs); + pendingTimers.add(timer); + }); + }); + + return new Promise<{ + readonly endpoint: URL; + readonly callCount: () => number; + readonly waitForCall: (count?: number) => Promise; + readonly waitForCatalog: (count?: number) => Promise; + readonly close: () => Promise; + }>((resolve, reject) => { + server.once("error", reject); + server.listen(0, "127.0.0.1", () => { + const address = server.address(); + if (address === null || typeof address === "string") { + reject(new Error("Expected TCP listener")); + return; + } + server.removeListener("error", reject); + resolve({ + endpoint: new URL(`http://127.0.0.1:${address.port}/mcp`), + callCount: () => callCount, + waitForCall: (count = 1) => + waitForRequestCount(() => callCount, count, "tools/call"), + waitForCatalog: (count = 1) => + waitForRequestCount(() => catalogCount, count, "tools/list"), + close: async () => { + for (const timer of pendingTimers) clearTimeout(timer); + pendingTimers.clear(); + const closed = new Promise((done, fail) => + server.close((cause) => + cause === undefined ? done() : fail(cause), + ), + ); + server.closeAllConnections(); + await closed; + }, + }); + }); + }); +}; describe("decodeJebToolResult", () => { it("passes refusal envelopes through so the provider preserves the engine reason", () => { @@ -118,3 +282,93 @@ it("refuses HTTP redirects before sending requests to another endpoint", async ( ); } }); + +it("applies the selected timeout to real MCP tool calls", async () => { + const fixture = await startJebHttpFixture(180); + const short = createStreamableHttpJebMcpConnection(fixture.endpoint, { + requestTimeoutMs: 50, + }); + const extended = createStreamableHttpJebMcpConnection(fixture.endpoint, { + requestTimeoutMs: 500, + }); + try { + await short.connect(); + await expect(short.call("decompile_code_item", {})).rejects.toBeInstanceOf( + AnalysisTimeoutError, + ); + expect(fixture.callCount()).toBe(1); + + await extended.connect(); + await expect( + extended.call("decompile_code_item", {}), + ).resolves.toMatchObject({ success: true, text: "finished" }); + expect(fixture.callCount()).toBe(2); + } finally { + try { + await Promise.all([short.close(), extended.close()]); + } finally { + await fixture.close(); + } + } +}); + +it("keeps caller abort and connection close as cancellations for active HTTP calls", async () => { + const fixture = await startJebHttpFixture(1_000); + const connection = createStreamableHttpJebMcpConnection(fixture.endpoint, { + requestTimeoutMs: 2_000, + }); + try { + await connection.connect(); + const alreadyAborted = new AbortController(); + alreadyAborted.abort(); + await expect( + connection.call("decompile_code_item", {}, alreadyAborted.signal), + ).rejects.toBeInstanceOf(AnalysisCancelledError); + expect(fixture.callCount()).toBe(0); + + const caller = new AbortController(); + const callerCall = connection.call( + "decompile_code_item", + {}, + caller.signal, + ); + await fixture.waitForCall(1); + caller.abort(); + await expect(callerCall).rejects.toBeInstanceOf(AnalysisCancelledError); + + const closingCall = connection.call("decompile_code_item", {}); + await fixture.waitForCall(2); + await connection.close(); + await expect(closingCall).rejects.toBeInstanceOf(AnalysisCancelledError); + } finally { + try { + await connection.close(); + } finally { + await fixture.close(); + } + } +}); + +it("keeps caller abort and connection close as cancellations during catalog refresh", async () => { + const fixture = await startJebHttpFixture(0, 1_000); + const connection = createStreamableHttpJebMcpConnection(fixture.endpoint); + try { + await connection.connect(); + const caller = new AbortController(); + const callerRefresh = connection.listTools(caller.signal); + await fixture.waitForCatalog(2); + caller.abort(); + await expect(callerRefresh).rejects.toBeInstanceOf(AnalysisCancelledError); + + const closingRefresh = connection.listTools(); + await fixture.waitForCatalog(3); + await connection.close(); + await expect(closingRefresh).rejects.toBeInstanceOf(AnalysisCancelledError); + } finally { + try { + await connection.close(); + } finally { + await fixture.close(); + } + } +}); diff --git a/src/jeb/JebMcpConnection.ts b/src/jeb/JebMcpConnection.ts index 137d0f32d7..41c4a4fc36 100644 --- a/src/jeb/JebMcpConnection.ts +++ b/src/jeb/JebMcpConnection.ts @@ -1,5 +1,8 @@ import { Client, + DEFAULT_REQUEST_TIMEOUT_MSEC, + SdkError, + SdkErrorCode, StreamableHTTPClientTransport, } from "@modelcontextprotocol/client"; import { z } from "zod"; @@ -7,12 +10,26 @@ import { jsonValueSchema, type JsonValue } from "../domain/jsonValue.js"; import { AnalysisCancelledError, AnalysisProtocolError, + AnalysisTimeoutError, } from "../domain/analysisErrorCore.js"; +import type { JebOperation } from "../domain/jeb/jebAnalysis.js"; +import { ProviderStartupDeadline } from "../process/ProviderDeadline.js"; + +/** Upstream protocol names for the operations REA exposes. */ +export const JEB_ENGINE_TOOLS = { + inspect_jeb_client: "get_client_information", + open_jeb_project: "open_project", + list_jeb_units: "list_units", + decompile_jeb_item: "decompile_code_item", +} as const satisfies Readonly>; +const RELEVANT_ENGINE_TOOLS = new Set(Object.values(JEB_ENGINE_TOOLS)); /** Injectable MCP boundary used by the provider and conformance tests. */ export interface JebMcpConnection { - /** Connect lazily and return the advertised tool names. */ + /** Connect lazily and return advertised names used by REA. */ connect(signal?: AbortSignal): Promise; + /** Refresh support for REA operations from the live engine catalog. */ + listTools(signal?: AbortSignal): Promise; /** Invoke one JEB MCP tool and decode its single structured result. */ call( name: string, @@ -67,12 +84,27 @@ const isJebFailureEnvelope = (value: JsonValue): boolean => !Array.isArray(value) && value.success === false; -export type JebMcpConnectionFactory = (endpoint: URL) => JebMcpConnection; +export interface JebMcpConnectionTimeouts { + /** Absolute catalog connection and pagination deadline. */ + readonly discoveryTimeoutMs?: number; + /** Absolute timeout for each upstream tools/call request. */ + readonly requestTimeoutMs?: number; +} + +export type JebMcpConnectionFactory = ( + endpoint: URL, + timeouts?: JebMcpConnectionTimeouts, +) => JebMcpConnection; /** Serve JEB requests from a running engine client over streamable HTTP. */ export const createStreamableHttpJebMcpConnection: JebMcpConnectionFactory = ( endpoint, + timeouts, ) => { + const discoveryTimeoutMs = + timeouts?.discoveryTimeoutMs ?? DEFAULT_REQUEST_TIMEOUT_MSEC; + const requestTimeoutMs = + timeouts?.requestTimeoutMs ?? DEFAULT_REQUEST_TIMEOUT_MSEC; interface ConnectionState { readonly client: Client; readonly transport: StreamableHTTPClientTransport; @@ -164,29 +196,37 @@ export const createStreamableHttpJebMcpConnection: JebMcpConnectionFactory = ( }; state = owner; const acquisition = (async () => { + const deadline = new ProviderStartupDeadline( + discoveryTimeoutMs, + owner.signal.signal, + ); try { await owner.client.connect(owner.transport, { - signal: owner.signal.signal, + signal: deadline.signal, }); - const listing = await owner.client.listTools( - {}, - { signal: owner.signal.signal }, - ); - const tools = listing.tools.map((tool) => tool.name); + const tools = await readToolNames(owner.client, deadline.signal); owner.tools = tools; return tools; } catch (cause: unknown) { - owner.signal.abort(cause); + const failure = + deadline.interruption === "timeout" + ? new AnalysisTimeoutError("jeb_mcp_catalog", discoveryTimeoutMs, { + cause, + }) + : cause; + owner.signal.abort(failure); try { await dispose(owner); } catch (cleanupCause: unknown) { throw new AggregateError( - [cause, cleanupCause], + [failure, cleanupCause], "JEB MCP connection acquisition and cleanup failed", - { cause }, + { cause: failure }, ); } - throw cause; + throw failure; + } finally { + deadline.dispose(); } })(); owner.acquisition = acquisition; @@ -203,16 +243,74 @@ export const createStreamableHttpJebMcpConnection: JebMcpConnectionFactory = ( return { connect, + async listTools(signal) { + const existing = state?.tools !== undefined; + // Initial acquisition owns its deadline and cleanup. A second timer + // could detach this waiter before that owner has disposed a failed client. + const tools = await connect(signal); + if (!existing) return tools; + const owner = state; + if (owner === undefined) + throw new AnalysisProtocolError("JEB MCP client failed to connect"); + const discoverySignal = + signal === undefined + ? owner.signal.signal + : AbortSignal.any([signal, owner.signal.signal]); + const deadline = new ProviderStartupDeadline( + discoveryTimeoutMs, + discoverySignal, + ); + try { + const refreshed = await readToolNames(owner.client, deadline.signal); + owner.tools = refreshed; + return refreshed; + } catch (cause: unknown) { + if (deadline.interruption === "timeout") + throw new AnalysisTimeoutError( + "jeb_mcp_catalog", + discoveryTimeoutMs, + { cause }, + ); + if (deadline.interruption === "cancelled") + throw new AnalysisCancelledError("jeb_mcp_catalog", { cause }); + throw cause; + } finally { + deadline.dispose(); + } + }, async call(name, args, signal) { await connect(signal); const owner = state; if (owner === undefined || owner.tools === undefined) throw new AnalysisProtocolError("JEB MCP client failed to connect"); - const result = await owner.client.callTool( - { name, arguments: args }, - signal === undefined ? {} : { signal }, + const requestSignal = + signal === undefined + ? owner.signal.signal + : AbortSignal.any([signal, owner.signal.signal]); + const deadline = new ProviderStartupDeadline( + requestTimeoutMs, + requestSignal, ); - return decodeJebToolResult(result); + try { + const result = await owner.client.callTool( + { name, arguments: args }, + { timeout: deadline.remainingMs(), signal: deadline.signal }, + ); + return decodeJebToolResult(result); + } catch (cause: unknown) { + if (deadline.interruption === "timeout") + throw new AnalysisTimeoutError(name, requestTimeoutMs, { cause }); + if (requestSignal.aborted) + throw new AnalysisCancelledError(name, { cause }); + if ( + cause instanceof SdkError && + cause.code === SdkErrorCode.RequestTimeout + ) + throw new AnalysisTimeoutError(name, requestTimeoutMs, { cause }); + throw cause; + } finally { + deadline.dispose(); + } }, async close() { closed = true; @@ -235,3 +333,32 @@ export const createStreamableHttpJebMcpConnection: JebMcpConnectionFactory = ( }, }; }; + +const readToolNames = async ( + client: Client, + signal: AbortSignal, +): Promise => { + const names = new Set(); + let cursor: string | undefined; + do { + signal.throwIfAborted(); + // listTools without a cursor aggregates and caches every page in SDK v2. + // This owner needs single pages to bound retained names and the whole walk. + const page = await client.request( + { + method: "tools/list", + ...(cursor === undefined ? {} : { params: { cursor } }), + }, + { signal }, + ); + signal.throwIfAborted(); + for (const tool of page.tools) + if (RELEVANT_ENGINE_TOOLS.has(tool.name)) names.add(tool.name); + if (names.size === RELEVANT_ENGINE_TOOLS.size) return [...names]; + const next = page.nextCursor; + if (next !== undefined && next === cursor) + throw new AnalysisProtocolError("JEB MCP repeated a tool catalog cursor"); + cursor = next; + } while (cursor !== undefined); + return [...names]; +}; diff --git a/src/jeb/JebMcpEndpoint.test.ts b/src/jeb/JebMcpEndpoint.test.ts index 0252632d86..d07ad0d586 100644 --- a/src/jeb/JebMcpEndpoint.test.ts +++ b/src/jeb/JebMcpEndpoint.test.ts @@ -5,6 +5,7 @@ import { parseJebMcpEndpoint } from "./JebMcpEndpoint.js"; describe("JEB MCP endpoint", () => { it("rejects an explicit empty query or fragment instead of dropping it", () => { expect(parseJebMcpEndpoint("http://127.0.0.1:8425/mcp").ok).toBe(true); + expect(parseJebMcpEndpoint("http://127.0.0.1:8425/mcp/").ok).toBe(true); for (const value of [ "http://127.0.0.1:8425/mcp?", "http://127.0.0.1:8425/mcp#", @@ -12,4 +13,12 @@ describe("JEB MCP endpoint", () => { ]) expect(parseJebMcpEndpoint(value).ok, value).toBe(false); }); + + it("accepts only the declared MCP route", () => { + expect(parseJebMcpEndpoint("http://127.0.0.1:8425/mcp").ok).toBe(true); + for (const path of ["/mcp/extra", "/not-mcp", "/"]) + expect(parseJebMcpEndpoint(`http://127.0.0.1:8425${path}`).ok, path).toBe( + false, + ); + }); }); diff --git a/src/jeb/JebMcpEndpoint.ts b/src/jeb/JebMcpEndpoint.ts index bae4a00a9c..97bca88f0f 100644 --- a/src/jeb/JebMcpEndpoint.ts +++ b/src/jeb/JebMcpEndpoint.ts @@ -47,6 +47,12 @@ export const parseJebMcpEndpoint = ( "Embedding credentials in the JEB MCP endpoint is not supported; configure authentication with the engine", }), ); + if (url.pathname !== "/mcp" && url.pathname !== "/mcp/") + return err( + new AnalysisInputError("jeb_mcp_endpoint", { + cause: "JEB MCP endpoint must use the /mcp route", + }), + ); if (urlDeclaresQueryOrFragment(url)) return err( new AnalysisInputError("jeb_mcp_endpoint", { diff --git a/src/jeb/JebProvider.test.ts b/src/jeb/JebProvider.test.ts index c6bd8c6ec2..e8b46d7a66 100644 --- a/src/jeb/JebProvider.test.ts +++ b/src/jeb/JebProvider.test.ts @@ -1,7 +1,11 @@ import { describe, expect, it } from "vitest"; +import { createJebAnalysisProvider } from "../composition/jeb.js"; import { JebProvider } from "./JebProvider.js"; -import type { JebMcpConnection } from "./JebMcpConnection.js"; +import type { + JebMcpConnection, + JebMcpConnectionTimeouts, +} from "./JebMcpConnection.js"; import { AnalysisCapabilityUnavailableError, AnalysisInputError, @@ -9,6 +13,7 @@ import { AnalysisCancelledError, } from "../domain/analysisErrorCore.js"; import type { JsonValue } from "../domain/jsonValue.js"; +import { jebResultSchemas } from "../domain/jeb/jebAnalysis.js"; const scriptedConnection = ( responses: Readonly>, @@ -17,7 +22,10 @@ const scriptedConnection = ( return { calls, async connect() { - return []; + return Object.keys(responses); + }, + async listTools() { + return Object.keys(responses); }, async call(name, args) { calls.push([name, args]); @@ -37,7 +45,78 @@ const providerWith = (connection: JebMcpConnection) => connectionFactory: () => connection, }); -describe("JebProvider boundary", () => { +describe("JebProvider live availability", () => { + it("passes its validated environment timeout to the production connection", async () => { + let configured: JebMcpConnectionTimeouts | undefined; + const provider = createJebAnalysisProvider( + { REA_JEB_MCP_TIMEOUT_MS: "900000" }, + (_endpoint, timeouts) => { + configured = timeouts; + return scriptedConnection({}); + }, + ); + expect(configured).toEqual({ requestTimeoutMs: 900_000 }); + await provider.close(); + }); + + it("rejects an invalid JEB timeout before constructing its MCP connection", async () => { + let constructed = false; + const provider = createJebAnalysisProvider( + { REA_JEB_MCP_TIMEOUT_MS: "0" }, + () => { + constructed = true; + return scriptedConnection({}); + }, + ); + expect(constructed).toBe(false); + await expect(provider.inspectAvailability()).resolves.toMatchObject({ + status: "unavailable", + diagnostics: { setting: "REA_JEB_MCP_TIMEOUT_MS" }, + }); + await provider.close(); + }); + + it("reports only the operations advertised by the live engine", async () => { + const connection = scriptedConnection({ + get_client_information: { + success: true, + version: "5.48.0", + gui_client: false, + }, + list_units: { success: true, units: [] }, + decompile_code_item: { success: true, text: "" }, + }); + const provider = providerWith(connection); + expect(await provider.inspectAvailability()).toMatchObject({ + status: "available", + supportedOperations: [ + "inspect_jeb_client", + "list_jeb_units", + "decompile_jeb_item", + ], + }); + expect(connection.calls).toEqual([["get_client_information", {}]]); + }); + + it("refreshes live operation support rather than reusing a readiness snapshot", async () => { + const connection = scriptedConnection({ + get_client_information: { success: true, version: "5.48.0" }, + }); + let tools = ["get_client_information", "open_project"]; + connection.listTools = async () => tools; + const provider = providerWith(connection); + expect(await provider.inspectAvailability()).toMatchObject({ + supportedOperations: ["inspect_jeb_client", "open_jeb_project"], + }); + tools = ["get_client_information"]; + expect(await provider.inspectAvailability()).toMatchObject({ + supportedOperations: ["inspect_jeb_client"], + }); + expect(connection.calls).toHaveLength(2); + }); +}); + +describe("JebProvider observations and failures", () => { it("normalizes client information and caches engine identity", async () => { const connection = scriptedConnection({ get_client_information: { @@ -52,12 +131,10 @@ describe("JebProvider boundary", () => { operation: "inspect_jeb_client", input: {}, }); - expect(result.ok).toBe(true); - if (!result.ok) return; - const normalized = result.value.result as { - readonly engine: { readonly version: string | null }; - readonly startup_ts: number | null; - }; + if (!result.ok) throw result.error; + const normalized = jebResultSchemas.inspect_jeb_client.parse( + result.value.result, + ); expect(normalized.engine.version).toBe("5.48.0.202610071539"); expect(normalized.startup_ts).toBe(1791623439); expect(result.value.provider.version).toBe("5.48.0.202610071539"); @@ -150,18 +227,18 @@ describe("JebProvider boundary", () => { operation: "list_jeb_units", input: { count: 3, index: 0 }, }); + if (!full.ok) throw full.error; expect( - full.ok && - (full.value.result as { coverage: string }).coverage === "partial", - ).toBe(true); + jebResultSchemas.list_jeb_units.parse(full.value.result).coverage, + ).toBe("partial"); const last = await provider.execute({ operation: "list_jeb_units", input: { count: 4, index: 0 }, }); + if (!last.ok) throw last.error; expect( - last.ok && - (last.value.result as { coverage: string }).coverage === "complete", - ).toBe(true); + jebResultSchemas.list_jeb_units.parse(last.value.result).coverage, + ).toBe("complete"); }); }); @@ -306,7 +383,10 @@ it("forwards in-flight cancellation to the MCP call", async () => { }); const provider = providerWith({ async connect() { - return []; + return ["get_client_information"]; + }, + async listTools() { + return ["get_client_information"]; }, async close() {}, async call(_name, _args, signal) { diff --git a/src/jeb/JebProvider.ts b/src/jeb/JebProvider.ts index bf62598a5d..797060d597 100644 --- a/src/jeb/JebProvider.ts +++ b/src/jeb/JebProvider.ts @@ -1,9 +1,11 @@ import { z } from "zod"; -import type { JebAnalysisPort } from "../application/jeb/JebAnalysisPort.js"; +import type { + JebAnalysisPort, + JebAnalysisAvailability, +} from "../application/jeb/JebAnalysisPort.js"; import type { AnalysisExecution, ExecutionOptions, - ProviderAvailability, ProviderIdentity, } from "../application/AnalysisProvider.js"; import { createAnalysisExecution } from "../application/AnalysisProvider.js"; @@ -23,6 +25,7 @@ import { } from "./JebMcpEndpoint.js"; import { createStreamableHttpJebMcpConnection, + JEB_ENGINE_TOOLS, type JebMcpConnection, type JebMcpConnectionFactory, } from "./JebMcpConnection.js"; @@ -36,6 +39,8 @@ export const JEB_PROVIDER_IDENTITY: ProviderIdentity = { export interface JebProviderOptions { /** Caller environment carrying REA_JEB_MCP_URL. */ readonly environment?: Readonly>; + /** Per-tools/call budget selected by REA_JEB_MCP_TIMEOUT_MS. */ + readonly requestTimeoutMs?: number; /** Injectable MCP boundary; defaults to the streamable HTTP client. */ readonly connectionFactory?: JebMcpConnectionFactory; } @@ -108,21 +113,29 @@ export class JebProvider implements JebAnalysisPort { const endpoint = parseJebMcpEndpoint(environment.REA_JEB_MCP_URL); if (!endpoint.ok) throw endpoint.error; this.#endpoint = endpoint.value; - this.#connection = ( - options.connectionFactory ?? - ((url: URL) => createStreamableHttpJebMcpConnection(url)) - )(this.#endpoint); + const createConnection = + options.connectionFactory ?? createStreamableHttpJebMcpConnection; + this.#connection = + options.requestTimeoutMs === undefined + ? createConnection(this.#endpoint) + : createConnection(this.#endpoint, { + requestTimeoutMs: options.requestTimeoutMs, + }); } async inspectAvailability( signal?: AbortSignal, - ): Promise { + ): Promise { try { + const tools = new Set(await this.#connection.listTools(signal)); await this.#refreshClientInformation(signal); return { status: "available", code: null, reason: null, + supportedOperations: ( + Object.keys(JEB_ENGINE_TOOLS) as JebOperation[] + ).filter((operation) => tools.has(JEB_ENGINE_TOOLS[operation])), diagnostics: { endpoint: this.#endpoint.toString(), engine_version: this.#identity.version, diff --git a/src/kuna/KunaConfiguration.ts b/src/kuna/KunaConfiguration.ts new file mode 100644 index 0000000000..48fe418645 --- /dev/null +++ b/src/kuna/KunaConfiguration.ts @@ -0,0 +1,144 @@ +import { isAbsolute, join } from "node:path"; + +import { homeDirectoryFromEnvironment } from "../config/homeDirectory.js"; +import { providerStartupTimeoutSchema } from "../config/providerStartupTimeout.js"; +import { err, ok, type Result } from "../domain/result.js"; +import { + KUNA_COMMAND_REMEDIATION, + KUNA_UNSUPPORTED_HOST_REMEDIATION, +} from "./KunaRelease.js"; + +/** Kuna caps its own decode lanes and worker pool at 32. */ +const MAX_KUNA_JOBS = 32; + +/** + * A 1.4 MB sqlite3 shell exported serially in 66-108 s on Apple M-series and + * Linux hosts; thirty minutes admits images an order of magnitude larger + * without an unbounded run. Larger images need a raised deadline or more jobs. + */ +const DEFAULT_PROJECT_TIMEOUT_MS = 30 * 60 * 1000; + +/** + * A serial (`--jobs 1`) export is Kuna's reference answer: it decompiles + * callees before callers, which recovers prototypes that parallel workers + * cannot. Measured on coreutils sort, 40-60% of functions differ otherwise. + */ +const DEFAULT_JOBS = 1; + +/** Measured sqlite3 exports occupy ~25 MiB; four GiB retains many projects. */ +const DEFAULT_CACHE_MAX_BYTES = 4 * 1024 ** 3; + +/** One project needs its .c, .h, .asm, inventory and manifest; refuse smaller caps. */ +const MIN_CACHE_MAX_BYTES = 1024 ** 2; + +/** Caller-selected Kuna installation and REA-owned project cache policy. */ +export interface KunaConfiguration { + /** Caller-selected executable path, preserved exactly as configured. */ + readonly command: string; + /** REA-owned cache root; Kuna's own spec cache lives beneath it too. */ + readonly cacheRoot: string; + readonly cacheMaxBytes: number; + readonly projectTimeoutMs: number; + readonly jobs: number; +} + +/** Why the configured environment cannot select a Kuna installation. */ +export interface KunaConfigurationFailure { + readonly code: "not_configured" | "open_options_invalid" | "unsupported_host"; + readonly reason: string; + readonly setting: string | null; +} + +const failure = ( + code: KunaConfigurationFailure["code"], + reason: string, + setting: string | null = null, +): Result => err({ code, reason, setting }); + +const positiveInteger = (value: string): number | undefined => + /^[1-9]\d*$/u.test(value) && Number.isSafeInteger(Number(value)) + ? Number(value) + : undefined; + +/** Per-user cache directory that REA owns for Kuna on this host. */ +const defaultCacheRoot = ( + environment: Readonly, + platform: NodeJS.Platform, +): string => { + const home = homeDirectoryFromEnvironment(environment, platform); + if (platform === "darwin") + return join(home, "Library", "Caches", "rea", "kuna"); + const xdg = environment.XDG_CACHE_HOME; + return join( + xdg !== undefined && isAbsolute(xdg) ? xdg : join(home, ".cache"), + "rea", + "kuna", + ); +}; + +/** Parse the Kuna settings without touching the filesystem or the executable. */ +export const readKunaConfiguration = ( + environment: Readonly, + platform: NodeJS.Platform = process.platform, +): Result => { + const command = environment.REA_KUNA_COMMAND; + if (command === undefined || command.length === 0) + return failure("not_configured", KUNA_COMMAND_REMEDIATION); + if (!isAbsolute(command)) + return failure( + "open_options_invalid", + `REA_KUNA_COMMAND must be an absolute path; received ${command}. ${KUNA_COMMAND_REMEDIATION}`, + "REA_KUNA_COMMAND", + ); + if (platform !== "darwin" && platform !== "linux") + return failure("unsupported_host", KUNA_UNSUPPORTED_HOST_REMEDIATION); + + const cacheRoot = environment.REA_KUNA_CACHE_DIR; + if (cacheRoot !== undefined && !isAbsolute(cacheRoot)) + return failure( + "open_options_invalid", + `REA_KUNA_CACHE_DIR must be an absolute directory; received ${cacheRoot}.`, + "REA_KUNA_CACHE_DIR", + ); + + const rawMaxBytes = environment.REA_KUNA_CACHE_MAX_BYTES; + const cacheMaxBytes = + rawMaxBytes === undefined + ? DEFAULT_CACHE_MAX_BYTES + : positiveInteger(rawMaxBytes); + if (cacheMaxBytes === undefined || cacheMaxBytes < MIN_CACHE_MAX_BYTES) + return failure( + "open_options_invalid", + `REA_KUNA_CACHE_MAX_BYTES must be a decimal byte count of at least ${String(MIN_CACHE_MAX_BYTES)}; received ${String(rawMaxBytes)}.`, + "REA_KUNA_CACHE_MAX_BYTES", + ); + + const rawTimeout = environment.REA_KUNA_PROJECT_TIMEOUT_MS; + const timeout = + rawTimeout === undefined + ? undefined + : providerStartupTimeoutSchema.safeParse(rawTimeout); + if (timeout !== undefined && !timeout.success) + return failure( + "open_options_invalid", + `REA_KUNA_PROJECT_TIMEOUT_MS must be a decimal millisecond count from 1 to 2147483647; received ${String(rawTimeout)}.`, + "REA_KUNA_PROJECT_TIMEOUT_MS", + ); + + const rawJobs = environment.REA_KUNA_JOBS; + const jobs = rawJobs === undefined ? DEFAULT_JOBS : positiveInteger(rawJobs); + if (jobs === undefined || jobs > MAX_KUNA_JOBS) + return failure( + "open_options_invalid", + `REA_KUNA_JOBS must be an integer from 1 to ${String(MAX_KUNA_JOBS)}; received ${String(rawJobs)}.`, + "REA_KUNA_JOBS", + ); + + return ok({ + command, + cacheRoot: cacheRoot ?? defaultCacheRoot(environment, platform), + cacheMaxBytes, + projectTimeoutMs: timeout?.data ?? DEFAULT_PROJECT_TIMEOUT_MS, + jobs, + }); +}; diff --git a/src/kuna/KunaDoctor.ts b/src/kuna/KunaDoctor.ts new file mode 100644 index 0000000000..4c3488dfb2 --- /dev/null +++ b/src/kuna/KunaDoctor.ts @@ -0,0 +1,211 @@ +import { constants } from "node:fs"; +import { access, stat } from "node:fs/promises"; +import { dirname } from "node:path"; + +import type { + DoctorProviderCheck, + DoctorProviderInspection, +} from "../application/Doctor.js"; +import { readKunaConfiguration } from "./KunaConfiguration.js"; +import { + inspectKunaInstallation, + type KunaReadinessFailure, +} from "./KunaInstallation.js"; +import { + KUNA_COMMAND_REMEDIATION, + KUNA_PROVIDER_ID, + KUNA_UNSUPPORTED_HOST_REMEDIATION, + KUNA_VERIFIED_VERSION, + KUNA_VERSION_PROBE_REMEDIATION, +} from "./KunaRelease.js"; + +/** Settings a registration may carry; only those the caller set explicitly. */ +const REGISTRATION_SETTINGS = [ + "REA_KUNA_COMMAND", + "REA_KUNA_CACHE_DIR", + "REA_KUNA_CACHE_MAX_BYTES", + "REA_KUNA_PROJECT_TIMEOUT_MS", + "REA_KUNA_JOBS", +] as const; + +/** Project Kuna readiness into the doctor contract without writing anything. */ +export const inspectKunaProvider = async ( + environment: Readonly, + platform: NodeJS.Platform = process.platform, +): Promise => { + const installation = await inspectKunaInstallation(environment, { + platform, + }); + const configured = + environment.REA_KUNA_COMMAND !== undefined && + environment.REA_KUNA_COMMAND.length > 0; + if (!installation.ok) + return { + id: KUNA_PROVIDER_ID, + configured, + available: false, + providerVersion: null, + registrationEnvironment: {}, + checks: [installationFailure(installation.error, configured)], + }; + const { value } = installation; + return { + id: KUNA_PROVIDER_ID, + configured, + available: true, + providerVersion: value.version, + registrationEnvironment: Object.fromEntries( + REGISTRATION_SETTINGS.flatMap((name) => { + const setting = environment[name]; + return setting === undefined ? [] : [[name, setting]]; + }), + ), + checks: [ + { + name: "installation", + ok: true, + code: null, + detail: `${value.executable} reports Kuna ${value.version} (sha256 ${value.sha256})${ + value.verified + ? "." + : `; REA verified ${KUNA_VERIFIED_VERSION}, so this release's project layout is unverified.` + }`, + remediation: null, + classification: "config_drift", + details: { + executable: value.executable, + version: value.version, + verified_version: KUNA_VERIFIED_VERSION, + sha256: value.sha256, + }, + }, + await cacheCheck(environment, platform), + ], + }; +}; + +const installationFailure = ( + failure: KunaReadinessFailure, + configured: boolean, +): DoctorProviderCheck => ({ + name: "installation", + ok: false, + code: failure.code, + detail: failure.reason, + remediation: + failure.code === "unsupported_host" + ? KUNA_UNSUPPORTED_HOST_REMEDIATION + : failure.code === "version_unresolved" + ? KUNA_VERSION_PROBE_REMEDIATION + : KUNA_COMMAND_REMEDIATION, + classification: + failure.code === "unsupported_host" + ? "unsupported_host" + : configured + ? "config_drift" + : "missing_analysis_engine", + details: failure.diagnostics, +}); + +/** The cache root may not exist yet; doctor reports it without creating it. */ +const cacheCheck = async ( + environment: Readonly, + platform: NodeJS.Platform, +): Promise => { + const configuration = readKunaConfiguration(environment, platform); + if (!configuration.ok) + return { + name: "cache", + ok: false, + code: configuration.error.code, + detail: configuration.error.reason, + remediation: KUNA_COMMAND_REMEDIATION, + classification: "config_drift", + }; + const { cacheRoot, cacheMaxBytes } = configuration.value; + const details = { cache_root: cacheRoot, cache_max_bytes: cacheMaxBytes }; + try { + if (!(await stat(cacheRoot)).isDirectory()) + return { + name: "cache", + ok: false, + code: "open_options_invalid", + detail: `The Kuna cache root ${cacheRoot} exists but is not a directory.`, + remediation: + "Set REA_KUNA_CACHE_DIR to a directory REA may own, or remove the conflicting file.", + classification: "config_drift", + details, + }; + await access(cacheRoot, constants.W_OK | constants.X_OK); + } catch (cause: unknown) { + if (cause instanceof Error && "code" in cause && cause.code === "ENOENT") { + const creatable = await nearestDirectoryWritable(cacheRoot); + return creatable.ok + ? { + name: "cache", + ok: true, + code: null, + detail: `REA creates the Kuna project cache at ${cacheRoot} on first use.`, + remediation: null, + classification: "config_drift", + details, + } + : { + name: "cache", + ok: false, + code: "open_options_invalid", + detail: `REA cannot create the Kuna cache root ${cacheRoot}: ${creatable.reason}`, + remediation: + "Set REA_KUNA_CACHE_DIR to a directory the REA process user can create or write.", + classification: "config_drift", + details, + }; + } + return { + name: "cache", + ok: false, + code: "open_options_invalid", + detail: `REA cannot write the Kuna cache root ${cacheRoot}: ${cause instanceof Error ? cause.message : String(cause)}`, + remediation: + "Set REA_KUNA_CACHE_DIR to a directory the REA process user can write.", + classification: "config_drift", + details, + }; + } + return { + name: "cache", + ok: true, + code: null, + detail: `Kuna project exports are cached in ${cacheRoot} (at most ${String(cacheMaxBytes)} bytes).`, + remediation: null, + classification: "config_drift", + details, + }; +}; + +/** Whether the nearest existing ancestor is a directory REA may create entries in. */ +const nearestDirectoryWritable = async ( + path: string, +): Promise< + { readonly ok: true } | { readonly ok: false; readonly reason: string } +> => { + let candidate = dirname(path); + for (;;) { + try { + if (!(await stat(candidate)).isDirectory()) + return { ok: false, reason: `${candidate} is not a directory.` }; + await access(candidate, constants.W_OK | constants.X_OK); + return { ok: true }; + } catch (cause: unknown) { + const missing = + cause instanceof Error && "code" in cause && cause.code === "ENOENT"; + const parent = dirname(candidate); + if (!missing || parent === candidate) + return { + ok: false, + reason: `${candidate} is not writable: ${cause instanceof Error ? cause.message : String(cause)}`, + }; + candidate = parent; + } + } +}; diff --git a/src/kuna/KunaInstallation.ts b/src/kuna/KunaInstallation.ts new file mode 100644 index 0000000000..5ba0c58b18 --- /dev/null +++ b/src/kuna/KunaInstallation.ts @@ -0,0 +1,191 @@ +import { constants } from "node:fs"; +import { access, open, realpath, stat } from "node:fs/promises"; +import { join } from "node:path"; + +import type { JsonValue } from "../domain/jsonValue.js"; +import { err, ok, type Result } from "../domain/result.js"; +import { hashStableFile } from "../filesystem/StableFileHash.js"; +import { + execFileOutput, + execFileOutputFailure, +} from "../process/ExecFileOutput.js"; +import { + readKunaConfiguration, + type KunaConfiguration, + type KunaConfigurationFailure, +} from "./KunaConfiguration.js"; +import { + KUNA_COMMAND_REMEDIATION, + KUNA_VERIFIED_VERSION, + KUNA_VERSION_PROBE_REMEDIATION, +} from "./KunaRelease.js"; + +/** `kuna --version` exits before loading specs or an input; it needs no budget beyond this. */ +const VERSION_PROBE_TIMEOUT_MS = 10_000; +const VERSION_PROBE_MAX_OUTPUT_BYTES = 64 * 1024; + +/** Release CI prints exactly `kuna `; source builds print the Cargo version. */ +const VERSION_LINE = /^kuna (\d+\.\d+(?:\.\d+)?)\r?\n?$/u; + +/** One observed Kuna executable, identified by its bytes as well as its version. */ +export interface KunaInstallation { + readonly configuration: KunaConfiguration; + /** Canonical path of the probed executable. */ + readonly executable: string; + readonly version: string; + /** Whether this release is the one REA verified. */ + readonly verified: boolean; + readonly sha256: string; + readonly bytes: number; + /** + * Whether the executable is a `#!` script, such as a package-manager shim. + * Its process then reports the interpreter as its command, so ownership + * checks cannot match the configured path by argv prefix. + */ + readonly script: boolean; +} + +/** Why the selected Kuna installation cannot be used. */ +export interface KunaReadinessFailure { + readonly code: + | KunaConfigurationFailure["code"] + | "executable_missing" + | "version_unresolved"; + readonly reason: string; + readonly diagnostics: Readonly>; +} + +/** + * Environment for every Kuna child. Kuna unpacks its built-in SLEIGH tree into + * the per-user cache under HOME (macOS) or XDG_CACHE_HOME (Linux), so both + * point inside REA's own cache root instead of the user's. + */ +export const kunaChildEnvironment = ( + configuration: KunaConfiguration, +): NodeJS.ProcessEnv => { + const home = join(configuration.cacheRoot, "home"); + return { + HOME: home, + XDG_CACHE_HOME: join(home, ".cache"), + NO_COLOR: "1", + }; +}; + +const failure = ( + code: KunaReadinessFailure["code"], + reason: string, + diagnostics: Readonly>, +): Result => err({ code, reason, diagnostics }); + +const message = (cause: unknown): string => + cause instanceof Error ? cause.message : String(cause); + +/** Resolve, version-probe and fingerprint the configured executable. */ +export const inspectKunaInstallation = async ( + environment: Readonly, + options: { + readonly platform?: NodeJS.Platform; + readonly signal?: AbortSignal; + /** Probe deadline; tests shorten it. */ + readonly probeTimeoutMs?: number; + } = {}, +): Promise> => { + const configuration = readKunaConfiguration(environment, options.platform); + if (!configuration.ok) + return failure(configuration.error.code, configuration.error.reason, { + configured: configuration.error.code !== "not_configured", + ...(configuration.error.setting === null + ? {} + : { setting: configuration.error.setting }), + }); + const { command } = configuration.value; + + let executable: string; + try { + executable = await realpath(command); + await access(executable, constants.X_OK); + if (!(await stat(executable)).isFile()) + return failure( + "executable_missing", + `REA_KUNA_COMMAND is not a regular file: ${command}. ${KUNA_COMMAND_REMEDIATION}`, + { command }, + ); + } catch (cause: unknown) { + options.signal?.throwIfAborted(); + return failure( + "executable_missing", + `Cannot execute REA_KUNA_COMMAND ${command}: ${message(cause)}. ${KUNA_COMMAND_REMEDIATION}`, + { command }, + ); + } + + const deadline = options.probeTimeoutMs ?? VERSION_PROBE_TIMEOUT_MS; + let stdout: string; + try { + ({ stdout } = await execFileOutput(executable, ["--version"], { + env: kunaChildEnvironment(configuration.value), + timeout: deadline, + maxBuffer: VERSION_PROBE_MAX_OUTPUT_BYTES, + // Escalate to SIGKILL so an executable that ignores SIGTERM cannot hang doctor. + stopSignal: "SIGTERM", + ...(options.signal === undefined ? {} : { signal: options.signal }), + })); + } catch (cause: unknown) { + options.signal?.throwIfAborted(); + const output = execFileOutputFailure(cause); + return failure( + "version_unresolved", + `${executable} --version failed: ${output?.stderr.trim() || message(cause)}. ${KUNA_VERSION_PROBE_REMEDIATION}`, + { + command, + executable, + exit_code: typeof output?.code === "number" ? output.code : null, + signal: output?.signal ?? null, + killed: output?.killed ?? false, + output_truncated: output?.outputTruncated ?? false, + deadline_ms: deadline, + }, + ); + } + const version = VERSION_LINE.exec(stdout)?.[1]; + if (version === undefined) + return failure( + "version_unresolved", + `${executable} --version printed no recognizable Kuna version. ${KUNA_VERSION_PROBE_REMEDIATION}`, + { command, executable, stdout: stdout.slice(0, 256) }, + ); + + let fingerprint: { readonly sha256: string; readonly bytes: number }; + let script: boolean; + try { + fingerprint = await hashStableFile(executable, options.signal); + script = await startsWithShebang(executable); + } catch (cause: unknown) { + options.signal?.throwIfAborted(); + return failure( + "executable_missing", + `Cannot fingerprint ${executable}: ${message(cause)}`, + { command, executable }, + ); + } + + return ok({ + configuration: configuration.value, + executable, + version, + verified: version === KUNA_VERIFIED_VERSION, + ...fingerprint, + script, + }); +}; + +const startsWithShebang = async (path: string): Promise => { + const handle = await open(path, "r"); + try { + const prefix = Buffer.alloc(2); + const { bytesRead } = await handle.read(prefix, 0, 2, 0); + return bytesRead === 2 && prefix.toString("latin1") === "#!"; + } finally { + await handle.close(); + } +}; diff --git a/src/kuna/KunaProjectCache.ts b/src/kuna/KunaProjectCache.ts new file mode 100644 index 0000000000..9bdcb406f4 --- /dev/null +++ b/src/kuna/KunaProjectCache.ts @@ -0,0 +1,690 @@ +import { createHash, randomUUID } from "node:crypto"; +import { + mkdir, + mkdtemp, + readdir, + readFile, + rename, + rm, + stat, + utimes, +} from "node:fs/promises"; +import { join } from "node:path"; + +import canonicalize from "canonicalize"; +import writeFileAtomic from "write-file-atomic"; +import { z } from "zod"; + +import type { AnalysisError } from "../domain/analysisErrorBase.js"; +import { + AnalysisCancelledError, + AnalysisResourceConstraintError, +} from "../domain/analysisErrorCore.js"; +import { ProviderCleanupError } from "../domain/providerCleanupError.js"; +import { err, ok, type Result } from "../domain/result.js"; +import { hashStableFile } from "../filesystem/StableFileHash.js"; +import type { KunaInstallation } from "./KunaInstallation.js"; +import { + exportKunaProject, + KUNA_EXPORT_OPERATION, + kunaExportFailure, + readKunaExport, + type KunaExport, + type KunaExportStatus, + type KunaProcessLauncher, +} from "./KunaProjectExport.js"; +import type { + KunaInventoryFunction, + KunaProjectFunction, +} from "./KunaProjectFiles.js"; +import { KUNA_PROVIDER_ID } from "./KunaRelease.js"; + +/** Bumped whenever the entry layout or manifest meaning changes. */ +const CACHE_LAYOUT = 1; + +/** A manifest names five files and their digests; anything larger is not ours. */ +const MANIFEST_MAX_BYTES = 64 * 1024; + +const KEY_PATTERN = /^[0-9a-f]{64}$/u; + +const SHA256_PATTERN = /^[0-9a-f]{64}$/u; + +/** Inputs that determine an export's content. Each one is part of the cache key. */ +export interface KunaProjectKeyInputs { + readonly targetSha256: string; + readonly kunaSha256: string; + readonly kunaVersion: string; + /** + * `--jobs 1` is Kuna's reference answer; more jobs decompile without + * callee-first prototypes and change some bodies. + */ + readonly jobs: number; +} + +/** Content address of one export: every input that can change its bytes. */ +export const kunaProjectKey = (inputs: KunaProjectKeyInputs): string => + createHash("sha256") + .update( + canonicalize({ + layout: CACHE_LAYOUT, + target_sha256: inputs.targetSha256, + kuna_sha256: inputs.kunaSha256, + kuna_version: inputs.kunaVersion, + export: { command: "decompile-project", jobs: inputs.jobs }, + }) ?? "", + ) + .digest("hex"); + +const fileSchema = z.strictObject({ + path: z.string().min(1), + bytes: z.number().int().nonnegative(), + sha256: z.string().regex(SHA256_PATTERN), +}); + +const manifestSchema = z.strictObject({ + layout: z.literal(CACHE_LAYOUT), + key: z.string().regex(KEY_PATTERN), + inputs: z.strictObject({ + target_sha256: z.string().regex(SHA256_PATTERN), + kuna_sha256: z.string().regex(SHA256_PATTERN), + kuna_version: z.string().min(1), + jobs: z.number().int().positive(), + }), + files: z.strictObject({ + c: fileSchema, + h: fileSchema, + asm: fileSchema, + readme: fileSchema, + inventory: fileSchema, + }), + summary: z.strictObject({ + decompiled: z.number().int().nonnegative(), + failed: z.number().int().nonnegative(), + }), + created_at: z.string().datetime(), +}); + +/** Immutable description of one published export, written once at publish. */ +export type KunaProjectManifest = z.infer; + +/** One verified, published export. */ +export interface KunaProject { + readonly key: string; + /** Directory holding the export's `.c`, `.h`, `.asm`, README and inventory. */ + readonly directory: string; + readonly manifest: KunaProjectManifest; + /** Decompiled function blocks in `.c` order. */ + readonly functions: readonly KunaProjectFunction[]; + /** Every callable entry Kuna found, including imports it does not decompile. */ + readonly inventory: readonly KunaInventoryFunction[]; +} + +/** One request to produce or reuse the export for a target. */ +export interface KunaProjectRequest { + readonly installation: KunaInstallation; + /** Caller-selected target path; REA decompiles a verified copy of it. */ + readonly targetPath: string; + readonly targetSha256: string; + readonly signal?: AbortSignal; + readonly onStatus?: (status: KunaExportStatus) => void; +} + +interface StagedResult { + readonly result: Result; + /** Set when an owned process may still write into staging. */ + readonly retainStaging: boolean; +} + +interface InflightBuild { + readonly key: string; + readonly promise: Promise>; + readonly controller: AbortController; + readonly listeners: Set<(status: KunaExportStatus) => void>; + waiters: number; +} + +const isMissing = (cause: unknown): boolean => + cause instanceof Error && + "code" in cause && + (cause.code === "ENOENT" || cause.code === "ENOTDIR"); + +/** A file an entry's manifest names is gone, replaced or changed under it. */ +const isDamaged = (cause: unknown): boolean => + cause instanceof Error && + "code" in cause && + ["ENOENT", "ENOTDIR", "EISDIR", "ENOTFILE", "ESTALE"].includes( + String(cause.code), + ); + +const isPublishedByAnother = (cause: unknown): boolean => + cause instanceof Error && + "code" in cause && + (cause.code === "EEXIST" || cause.code === "ENOTEMPTY"); + +const parseManifest = (text: string): KunaProjectManifest | undefined => { + try { + const parsed = manifestSchema.safeParse(JSON.parse(text)); + return parsed.success ? parsed.data : undefined; + } catch { + return undefined; + } +}; + +const manifestBytes = (manifest: KunaProjectManifest): number => + Object.values(manifest.files).reduce((sum, file) => sum + file.bytes, 0); + +/** + * REA-owned on-disk cache of `kuna decompile-project` exports. + * + * Layout under the root: `projects//` holds published entries + * (`manifest.json` plus the export and inventory in `project/`), `staging/` + * holds builds whose processes may still be live, `trash/` holds retired + * workspaces and evicted entries awaiting removal, + * and `home/` is Kuna's own HOME so its spec cache stays inside REA's + * ownership. + * + * Entries are content-addressed and immutable. A build runs in private + * staging and is published by one directory rename, so a reader sees a + * complete entry or none, and concurrent publishers of one key keep the first. + * File digests are recorded once at publish and checked on every first read. + * Recency is the entry directory's mtime; publishing evicts the least recently + * used entries above the size cap, never an entry pinned by this process. + */ +export class KunaProjectCache { + readonly #root: string; + readonly #maxBytes: number; + readonly #launcher: KunaProcessLauncher | undefined; + /** Builds new requests may join, at most one per key. */ + readonly #inflight = new Map(); + /** Every build not yet settled, including abandoned ones a retry replaced. */ + readonly #unfinished = new Set(); + readonly #verified = new Map(); + readonly #pins = new Map(); + /** One eviction decision at a time; exports themselves remain concurrent. */ + #eviction: Promise = Promise.resolve(); + #capacity: "within_limit" | "pinned" | "building" = "within_limit"; + #maintenance: Promise | undefined; + #maintenanceRequested = false; + readonly #retainedProcesses: ProviderCleanupError[] = []; + /** Only resources whose process cleanup is proven may be retried as files. */ + readonly #retainedFiles = new Map(); + + constructor( + root: string, + options: { + readonly maxBytes: number; + readonly launcher?: KunaProcessLauncher; + }, + ) { + this.#root = root; + this.#maxBytes = options.maxBytes; + this.#launcher = options.launcher; + } + + /** Keep an entry from eviction while this process serves it. */ + pin(key: string): () => void { + this.#pins.set(key, (this.#pins.get(key) ?? 0) + 1); + let released = false; + return () => { + if (released) return; + released = true; + const count = (this.#pins.get(key) ?? 1) - 1; + if (count === 0) { + this.#pins.delete(key); + if (this.#capacity !== "within_limit") this.#scheduleEviction(); + } else this.#pins.set(key, count); + }; + } + + /** + * Return the verified published entry for a key, or undefined when none is + * usable. A corrupt or tampered entry is removed and reported as absent. + */ + async lookup( + key: string, + signal?: AbortSignal, + ): Promise { + const verified = this.#verified.get(key); + if (verified !== undefined) { + // Another process may have evicted it since this one verified it. + if (await this.#touch(key)) return verified; + this.#verified.delete(key); + return undefined; + } + const entry = join(this.#root, "projects", key); + let text: string; + try { + text = await this.#readBounded(join(entry, "manifest.json")); + } catch (cause: unknown) { + if (isMissing(cause)) return undefined; + if (!(cause instanceof RangeError)) throw cause; + await this.#discard(key); + return undefined; + } + const manifest = parseManifest(text); + if (manifest?.key !== key) { + await this.#discard(key); + return undefined; + } + const directory = join(entry, "project"); + for (const file of Object.values(manifest.files)) { + let observed: { readonly sha256: string; readonly bytes: number }; + try { + observed = await hashStableFile(join(directory, file.path), signal); + } catch (cause: unknown) { + signal?.throwIfAborted(); + if (!isDamaged(cause)) throw cause; + await this.#discard(key); + return undefined; + } + if (observed.sha256 !== file.sha256 || observed.bytes !== file.bytes) { + await this.#discard(key); + return undefined; + } + } + const parsed = await readKunaExport( + directory, + manifest.files, + manifest.summary, + signal, + ); + if (!parsed.ok) { + await this.#discard(key); + return undefined; + } + const project = { key, directory, manifest, ...parsed.value }; + if (!(await this.#touch(key))) return undefined; + this.#verified.set(key, project); + return project; + } + + /** + * Return the entry for a request, building and publishing it when absent. + * Concurrent requests for one key share one Kuna run; the run is cancelled + * only when every waiting request has been cancelled. + */ + async build( + request: KunaProjectRequest, + ): Promise> { + if (request.signal?.aborted === true) + return err(new AnalysisCancelledError(KUNA_EXPORT_OPERATION)); + const key = kunaProjectKey({ + targetSha256: request.targetSha256, + kunaSha256: request.installation.sha256, + kunaVersion: request.installation.version, + jobs: request.installation.configuration.jobs, + }); + const running = this.#inflight.get(key); + // A build whose waiters all left is winding down; a new request starts afresh. + if (running !== undefined && !running.controller.signal.aborted) + return this.#await(running, request); + const controller = new AbortController(); + const listeners = new Set<(status: KunaExportStatus) => void>(); + const promise = this.#produce(key, request, controller.signal, (status) => { + for (const listener of listeners) listener(status); + }); + const created: InflightBuild = { + key, + promise, + controller, + listeners, + waiters: 0, + }; + this.#inflight.set(key, created); + this.#unfinished.add(created); + void promise.then(() => { + this.#unfinished.delete(created); + if (this.#inflight.get(key) === created) this.#inflight.delete(key); + if (this.#unfinished.size === 0 && this.#capacity === "building") + this.#scheduleEviction(); + }); + return this.#await(created, request); + } + + /** Wait for builds, retry safe file removal, and report unresolved cleanup. */ + async close(): Promise> { + await Promise.allSettled( + [...this.#unfinished].map((entry) => entry.promise), + ); + while (this.#maintenance !== undefined) await this.#maintenance; + for (const path of [...this.#retainedFiles.keys()]) + await this.#removeRetired(path).catch(() => undefined); + const retained = [ + ...this.#retainedProcesses, + ...this.#retainedFiles.values(), + ]; + if (retained.length === 0) return ok(null); + return err( + new ProviderCleanupError( + KUNA_PROVIDER_ID, + retained.flatMap((entry) => entry.cleanupResources), + { + reason: retained + .map((entry) => + String(entry.diagnostics?.["reason"] ?? entry.message), + ) + .join("; "), + }, + { operation: KUNA_EXPORT_OPERATION }, + ), + ); + } + + #await( + entry: InflightBuild, + request: KunaProjectRequest, + ): Promise> { + entry.waiters += 1; + const onStatus = request.onStatus; + // Each waiter owns its registration, even when callers reuse a callback. + const listener = + onStatus === undefined + ? undefined + : (status: KunaExportStatus) => onStatus(status); + if (listener !== undefined) entry.listeners.add(listener); + return new Promise((resolve) => { + let settled = false; + const leave = () => { + if (listener !== undefined) entry.listeners.delete(listener); + entry.waiters -= 1; + }; + const onAbort = () => { + if (settled) return; + settled = true; + leave(); + if (entry.waiters === 0) entry.controller.abort(); + resolve(err(new AnalysisCancelledError(KUNA_EXPORT_OPERATION))); + }; + request.signal?.addEventListener("abort", onAbort, { once: true }); + void entry.promise.then((result) => { + request.signal?.removeEventListener("abort", onAbort); + if (settled) return; + settled = true; + leave(); + resolve(result); + }); + }); + } + + async #produce( + key: string, + request: KunaProjectRequest, + signal: AbortSignal, + onStatus: (status: KunaExportStatus) => void, + ): Promise> { + let staging: string; + try { + const existing = await this.lookup(key, signal); + if (existing !== undefined) return ok(existing); + await Promise.all( + ["projects", "staging", "trash", "home"].map((name) => + mkdir(join(this.#root, name), { recursive: true, mode: 0o700 }), + ), + ); + await this.#sweep(signal); + staging = await mkdtemp(join(this.#root, "staging", "build-")); + } catch (cause: unknown) { + return err(kunaExportFailure(cause, signal)); + } + const built = await this.#exportAndPublish( + key, + staging, + request, + signal, + onStatus, + ).catch((cause: unknown): StagedResult => ({ + result: err(kunaExportFailure(cause, signal)), + retainStaging: false, + })); + if (built.retainStaging) return built.result; + let cleanupPath = staging; + try { + // Only a proven-stopped build may enter trash. A failed write, crash or + // lost cleanup identity therefore never makes live staging reclaimable. + const retired = join(this.#root, "trash", `build-${randomUUID()}`); + await rename(staging, retired); + cleanupPath = retired; + await this.#removeRetired(retired); + } catch (cause: unknown) { + if (cause instanceof ProviderCleanupError) return err(cause); + return err(this.#retainFileCleanup(cleanupPath, cause)); + } + if (built.result.ok) + // The entry is published either way; a failed eviction retries on the next publish. + await this.#evict(key).catch(() => undefined); + return built.result; + } + + async #exportAndPublish( + key: string, + staging: string, + request: KunaProjectRequest, + signal: AbortSignal, + onStatus: (status: KunaExportStatus) => void, + ): Promise { + const outcome = await exportKunaProject({ + installation: request.installation, + targetPath: request.targetPath, + targetSha256: request.targetSha256, + workspace: staging, + output: join(staging, "entry", "project"), + signal, + onStatus, + launcher: this.#launcher, + }); + if (outcome.kind === "retained") { + this.#retainedProcesses.push(outcome.error); + return { result: err(outcome.error), retainStaging: true }; + } + return { + result: outcome.result.ok + ? await this.#publish( + key, + staging, + request, + outcome.result.value, + signal, + ) + : outcome.result, + retainStaging: false, + }; + } + + /** Record the manifest and publish the staged export under its key. */ + async #publish( + key: string, + staging: string, + request: KunaProjectRequest, + exported: KunaExport, + signal: AbortSignal, + ): Promise> { + const manifest: KunaProjectManifest = { + layout: CACHE_LAYOUT, + key, + inputs: { + target_sha256: request.targetSha256, + kuna_sha256: request.installation.sha256, + kuna_version: request.installation.version, + jobs: request.installation.configuration.jobs, + }, + files: exported.files, + summary: exported.summary, + created_at: new Date().toISOString(), + }; + const total = manifestBytes(manifest); + if (total > this.#maxBytes) + return err( + new AnalysisResourceConstraintError( + KUNA_EXPORT_OPERATION, + "capacity", + `This export occupies ${String(total)} bytes, more than the whole Kuna cache may hold.`, + { cache_max_bytes: this.#maxBytes, export_bytes: total }, + { + remediationAction: + "Raise REA_KUNA_CACHE_MAX_BYTES above the export size, or analyze this target with another provider.", + }, + ), + ); + await writeFileAtomic( + join(staging, "entry", "manifest.json"), + `${JSON.stringify(manifest, null, 2)}\n`, + { mode: 0o600 }, + ); + const destination = join(this.#root, "projects", key); + try { + await rename(join(staging, "entry"), destination); + } catch (cause: unknown) { + if (!isPublishedByAnother(cause)) throw cause; + const winner = await this.lookup(key, signal); + if (winner !== undefined) return ok(winner); + throw cause; + } + const project = { + key, + directory: join(destination, "project"), + manifest, + functions: exported.functions, + inventory: exported.inventory, + }; + this.#verified.set(key, project); + return ok(project); + } + + /** + * Remove retired resources only. Age and a missing marker cannot establish + * that a process stopped, so unresolved staging is never swept. + */ + async #sweep(signal: AbortSignal): Promise { + for (const name of await readdir(join(this.#root, "trash"))) { + signal.throwIfAborted(); + await this.#removeRetired(join(this.#root, "trash", name)).catch( + () => undefined, + ); + } + } + + /** Evict least recently used entries above the cap, keeping pinned ones. */ + #evict(keep: string | null): Promise { + const eviction = this.#eviction.then(() => this.#evictUnpinned(keep)); + this.#eviction = eviction.catch(() => undefined); + return eviction; + } + + /** Coalesce deferred reclamation after callers have had a turn to pin results. */ + #scheduleEviction(): void { + this.#maintenanceRequested = true; + if (this.#maintenance !== undefined) return; + const maintenance = (async () => { + while (this.#maintenanceRequested) { + await new Promise((resolve) => setImmediate(resolve)); + this.#maintenanceRequested = false; + await this.#evict(null).catch(() => undefined); + } + })(); + this.#maintenance = maintenance; + void maintenance.then(() => { + if (this.#maintenance === maintenance) this.#maintenance = undefined; + if (this.#maintenanceRequested) this.#scheduleEviction(); + }); + } + + async #evictUnpinned(keep: string | null): Promise { + // A successful build must reach its callers before they can pin its entry. + // Protect every unfinished build while reading this eviction snapshot. + const building = new Set([...this.#unfinished].map(({ key }) => key)); + const entries: { key: string; bytes: number; mtimeMs: number }[] = []; + for (const name of await readdir(join(this.#root, "projects"))) { + if (!KEY_PATTERN.test(name)) continue; + const path = join(this.#root, "projects", name); + const status = await stat(path).catch(() => undefined); + if (status === undefined) continue; + const manifest = await this.#readBounded(join(path, "manifest.json")) + .then(parseManifest) + .catch(() => undefined); + // An owned entry without a valid manifest is unusable; evict it first. + entries.push( + manifest === undefined + ? { key: name, bytes: 0, mtimeMs: -1 } + : { + key: name, + bytes: manifestBytes(manifest), + mtimeMs: status.mtimeMs, + }, + ); + } + let total = entries.reduce((sum, entry) => sum + entry.bytes, 0); + let deferredByBuild = false; + for (const entry of entries.sort((a, b) => a.mtimeMs - b.mtimeMs)) { + if (total <= this.#maxBytes && entry.mtimeMs >= 0) break; + if (building.has(entry.key) || this.#inflight.has(entry.key)) { + deferredByBuild = true; + continue; + } + if (entry.key === keep || this.#pins.has(entry.key)) continue; + await this.#discard(entry.key); + total -= entry.bytes; + } + this.#capacity = + total <= this.#maxBytes + ? "within_limit" + : deferredByBuild + ? "building" + : "pinned"; + } + + /** Atomically retire one owned entry, then remove it. */ + async #discard(key: string): Promise { + this.#verified.delete(key); + const trash = join(this.#root, "trash", `${key}-${randomUUID()}`); + try { + await mkdir(join(this.#root, "trash"), { recursive: true, mode: 0o700 }); + await rename(join(this.#root, "projects", key), trash); + } catch (cause: unknown) { + if (isMissing(cause)) return; + throw cause; + } + await this.#removeRetired(trash); + } + + /** Keep each failed removal owned until that path is successfully removed. */ + async #removeRetired(path: string): Promise { + try { + await rm(path, { recursive: true, force: true }); + this.#retainedFiles.delete(path); + } catch (cause: unknown) { + throw this.#retainFileCleanup(path, cause); + } + } + + /** Mark an entry recently used; false when it is no longer published. */ + async #touch(key: string): Promise { + const now = new Date(); + try { + await utimes(join(this.#root, "projects", key), now, now); + return true; + } catch (cause: unknown) { + // Recency is advisory; only a missing entry changes the answer. + return !isMissing(cause); + } + } + + async #readBounded(path: string): Promise { + const status = await stat(path); + if (status.size > MANIFEST_MAX_BYTES) + throw new RangeError( + `${path} exceeds ${String(MANIFEST_MAX_BYTES)} bytes`, + ); + return readFile(path, "utf8"); + } + + #retainFileCleanup(path: string, cause: unknown): ProviderCleanupError { + const error = new ProviderCleanupError( + KUNA_PROVIDER_ID, + [path], + { reason: cause instanceof Error ? cause.message : String(cause) }, + { operation: KUNA_EXPORT_OPERATION, cause }, + ); + this.#retainedFiles.set(path, error); + return error; + } +} diff --git a/src/kuna/KunaProjectExport.ts b/src/kuna/KunaProjectExport.ts new file mode 100644 index 0000000000..7b45f1c871 --- /dev/null +++ b/src/kuna/KunaProjectExport.ts @@ -0,0 +1,466 @@ +import { createHash, randomUUID } from "node:crypto"; +import { createReadStream, createWriteStream } from "node:fs"; +import { lstat, mkdir, readFile, writeFile } from "node:fs/promises"; +import { basename, join } from "node:path"; +import { Transform } from "node:stream"; +import { pipeline } from "node:stream/promises"; + +import { AnalysisError } from "../domain/analysisErrorBase.js"; +import { + AnalysisArtifactChangedError, + AnalysisCancelledError, + AnalysisResourceConstraintError, + AnalysisTimeoutError, +} from "../domain/analysisErrorCore.js"; +import { ProviderAdapterError } from "../domain/providerAdapterError.js"; +import { ProviderCleanupError } from "../domain/providerCleanupError.js"; +import { err, ok, type Result } from "../domain/result.js"; +import { hashStableFile } from "../filesystem/StableFileHash.js"; +import { + OwnedCommandFailure, + runOwnedCommand, +} from "../process/OwnedCommand.js"; +import { prepareProcessOwnershipInspection } from "../process/ProcessOwnershipObservation.js"; +import type { + OwnedProviderProcessSpawnOptions, + ProviderProcessSnapshot, + SpawnedOwnedProviderProcess, +} from "../process/ProviderProcess.js"; +import { + kunaChildEnvironment, + type KunaInstallation, +} from "./KunaInstallation.js"; +import { + parseKunaExportSummary, + parseKunaInventory, + scanKunaProjectSource, + type KunaExportSummary, + type KunaInventoryFunction, + type KunaProjectFunction, +} from "./KunaProjectFiles.js"; +import { KUNA_PROVIDER_ID } from "./KunaRelease.js"; + +/** Operation name every export and cache failure reports. */ +export const KUNA_EXPORT_OPERATION = "kuna:decompile-project"; + +/** Kuna prints one summary line per export; notes on a large image stay far below this. */ +const DIAGNOSTIC_BYTES = 8 * 1024 * 1024; + +/** + * `kuna functions --json` costs ~210 bytes per function; this admits about + * 600,000 functions, above the 392,814 of the largest image Kuna documents. + */ +const INVENTORY_BYTES = 128 * 1024 * 1024; + +const INVENTORY_FILE = "functions.json"; + +/** Identity of one file an export consists of. */ +export interface KunaExportFile { + readonly path: string; + readonly bytes: number; + readonly sha256: string; +} + +/** The files of one export, by role. */ +export interface KunaExportFiles { + readonly c: KunaExportFile; + readonly h: KunaExportFile; + readonly asm: KunaExportFile; + readonly readme: KunaExportFile; + readonly inventory: KunaExportFile; +} + +/** One validated export: its files, Kuna's counts and the facts parsed from them. */ +export interface KunaExport { + readonly files: KunaExportFiles; + readonly summary: KunaExportSummary; + /** Decompiled function blocks in `.c` order. */ + readonly functions: readonly KunaProjectFunction[]; + /** Every callable entry Kuna found, including imports it does not decompile. */ + readonly inventory: readonly KunaInventoryFunction[]; +} + +/** The step an export is in; Kuna itself reports no progress until it finishes. */ +export interface KunaExportStatus { + readonly phase: "inventory" | "decompiling" | "sealing"; + /** Callable entries found by the inventory, once known. */ + readonly inventoryFunctions: number | null; +} + +/** Test seam for the owned process launcher; production uses the default. */ +export type KunaProcessLauncher = ( + options: OwnedProviderProcessSpawnOptions, +) => Promise; + +/** + * How an export ended. `retained` means an owned Kuna process could not be + * proven stopped, so it may still write into the workspace, which therefore + * stays with the caller's cleanup owner. + */ +export type KunaExportOutcome = + | { + readonly kind: "done"; + readonly result: Result; + } + | { readonly kind: "retained"; readonly error: ProviderCleanupError }; + +/** One export of a target into a private workspace. */ +export interface KunaExportRequest { + readonly installation: KunaInstallation; + /** Caller-selected target path; Kuna decompiles a verified copy of it. */ + readonly targetPath: string; + readonly targetSha256: string; + /** Private directory for the copy, Kuna's temporary files and its output. */ + readonly workspace: string; + /** Directory, inside the workspace, that receives the export's files. */ + readonly output: string; + readonly signal: AbortSignal; + readonly onStatus: (status: KunaExportStatus) => void; + readonly launcher?: KunaProcessLauncher | undefined; +} + +const message = (cause: unknown): string => + cause instanceof Error ? cause.message : String(cause); + +const isMissing = (cause: unknown): boolean => + cause instanceof Error && + "code" in cause && + (cause.code === "ENOENT" || cause.code === "ENOTDIR"); + +/** Map an unexpected failure to a typed one, preferring cancellation. */ +export const kunaExportFailure = ( + cause: unknown, + signal: AbortSignal, +): AnalysisError => { + if (signal.aborted) + return new AnalysisCancelledError(KUNA_EXPORT_OPERATION, { cause }); + if (cause instanceof AnalysisError) return cause; + return new ProviderAdapterError(KUNA_PROVIDER_ID, KUNA_EXPORT_OPERATION, { + cause, + userMessage: `Kuna project cache failed: ${message(cause)}`, + }); +}; + +const unusable = (reason: string): KunaExportOutcome => ({ + kind: "done", + result: err( + new ProviderAdapterError(KUNA_PROVIDER_ID, KUNA_EXPORT_OPERATION, { + userMessage: `Kuna's project export is not usable: ${reason}`, + }), + ), +}); + +/** Verify the selected installation at each execution and publication boundary. */ +const verifyKunaExecutable = async ( + installation: KunaInstallation, + signal: AbortSignal, +): Promise> => { + const executable = await hashStableFile(installation.executable, signal); + if (executable.sha256 === installation.sha256) return ok(null); + return err( + new ProviderAdapterError(KUNA_PROVIDER_ID, KUNA_EXPORT_OPERATION, { + userMessage: `${installation.executable} changed after readiness inspection; reopen the target so REA probes the current Kuna executable.`, + diagnostics: { + expected_sha256: installation.sha256, + observed_sha256: executable.sha256, + }, + }), + ); +}; + +/** + * Inventory and export one target with Kuna, then validate the result. + * + * Kuna runs on a copy hashed while copying, so the export always describes + * the digest the caller selected. Both Kuna runs share one deadline. + */ +export const exportKunaProject = async ( + request: KunaExportRequest, +): Promise => { + const { installation, signal } = request; + const { configuration } = installation; + const deadline = Date.now() + configuration.projectTimeoutMs; + const input = join(request.workspace, "input", basename(request.targetPath)); + await mkdir(join(request.workspace, "input"), { mode: 0o700 }); + await mkdir(request.output, { recursive: true, mode: 0o700 }); + + const copied = await copyAndHash(request.targetPath, input, signal); + if (copied !== request.targetSha256) + return { + kind: "done", + result: err( + new AnalysisArtifactChangedError( + KUNA_EXPORT_OPERATION, + request.targetPath, + "the target no longer matches the digest selected for analysis", + ), + ), + }; + await prepareProcessOwnershipInspection(signal); + const run = (arguments_: readonly string[], diagnosticBytes: number) => + runKuna(request, arguments_, { deadline, diagnosticBytes }); + + request.onStatus({ phase: "inventory", inventoryFunctions: null }); + const listed = await run(["functions", input, "--json"], INVENTORY_BYTES); + if (!listed.ok) return listed.outcome; + const inventory = parseKunaInventory(listed.snapshot.stdout.text); + if (!inventory.ok) return unusable(inventory.error); + if (listed.snapshot.stdout.invalidUtf8 === true) + return unusable("the function inventory is not valid UTF-8"); + await writeFile( + join(request.output, INVENTORY_FILE), + listed.snapshot.stdout.text, + { mode: 0o600 }, + ); + + request.onStatus({ + phase: "decompiling", + inventoryFunctions: inventory.value.length, + }); + const exported = await run( + [ + "decompile-project", + input, + "-o", + request.output, + "--jobs", + String(configuration.jobs), + ], + DIAGNOSTIC_BYTES, + ); + if (!exported.ok) return exported.outcome; + + request.onStatus({ + phase: "sealing", + inventoryFunctions: inventory.value.length, + }); + const summary = parseKunaExportSummary(exported.snapshot.stdout.text); + if (summary === undefined) + return unusable( + "Kuna did not print its `functions: N ok, M failed` summary.", + ); + const stem = basename(request.targetPath); + const names = { + c: `${stem}.c`, + h: `${stem}.h`, + asm: `${stem}.asm`, + readme: "README.md", + inventory: INVENTORY_FILE, + } as const; + const files: Partial> = {}; + for (const [role, name] of Object.entries(names) as [ + keyof KunaExportFiles, + string, + ][]) { + const path = join(request.output, name); + const status = await lstat(path).catch((cause: unknown) => { + if (isMissing(cause)) return undefined; + throw cause; + }); + if (status === undefined || !status.isFile()) + return unusable(`Kuna's export has no regular ${name}.`); + files[role] = { path: name, ...(await hashStableFile(path, signal)) }; + } + const complete = files as KunaExportFiles; + const parsed = await readKunaExport( + request.output, + complete, + summary, + signal, + ); + if (!parsed.ok) return unusable(parsed.error); + const identity = await verifyKunaExecutable(installation, signal); + if (!identity.ok) return { kind: "done", result: identity }; + return { + kind: "done", + result: ok({ files: complete, summary, ...parsed.value }), + }; +}; + +/** Parse an export's function blocks and inventory from its own files. */ +export const readKunaExport = async ( + directory: string, + files: KunaExportFiles, + summary: KunaExportSummary, + signal?: AbortSignal, +): Promise< + Result< + { + readonly functions: readonly KunaProjectFunction[]; + readonly inventory: readonly KunaInventoryFunction[]; + }, + string + > +> => { + const functions = await scanKunaProjectSource( + createReadStream( + join(directory, files.c.path), + signal === undefined ? {} : { signal }, + ), + summary, + ); + if (!functions.ok) return functions; + const inventory = parseKunaInventory( + await readFile(join(directory, files.inventory.path), { + encoding: "utf8", + ...(signal === undefined ? {} : { signal }), + }), + ); + if (!inventory.ok) return inventory; + return ok({ functions: functions.value, inventory: inventory.value }); +}; + +const runKuna = async ( + request: KunaExportRequest, + arguments_: readonly string[], + limits: { readonly deadline: number; readonly diagnosticBytes: number }, +): Promise< + | { readonly ok: true; readonly snapshot: ProviderProcessSnapshot } + | { readonly ok: false; readonly outcome: KunaExportOutcome } +> => { + const { installation, signal } = request; + try { + const identity = await verifyKunaExecutable(installation, signal); + if (!identity.ok) + return { ok: false, outcome: { kind: "done", result: identity } }; + const snapshot = await runOwnedCommand( + { + command: installation.executable, + arguments: arguments_, + runId: `rea-kuna-${randomUUID()}`, + // A script's process reports its interpreter, not the configured path. + ...(installation.script ? { expectedCommand: null } : {}), + cwd: request.workspace, + hostEnvironment: kunaChildEnvironment(installation.configuration), + env: { TMPDIR: request.workspace }, + }, + { + timeoutMs: Math.max(1, limits.deadline - Date.now()), + diagnosticBytes: limits.diagnosticBytes, + }, + { + signal, + ...(request.launcher === undefined + ? {} + : { launcher: request.launcher }), + }, + ); + return { ok: true, snapshot }; + } catch (cause: unknown) { + return { + ok: false, + outcome: commandFailure(cause, request, arguments_[0] ?? "", limits), + }; + } +}; + +/** Copy the target into the workspace and return the SHA-256 of exactly the bytes copied. */ +const copyAndHash = async ( + source: string, + destination: string, + signal: AbortSignal, +): Promise => { + const hash = createHash("sha256"); + await pipeline( + createReadStream(source), + new Transform({ + transform(chunk: Buffer, _encoding, callback) { + hash.update(chunk); + callback(null, chunk); + }, + }), + createWriteStream(destination, { flags: "wx", mode: 0o600 }), + { signal }, + ); + return hash.digest("hex"); +}; + +/** Map an owned-command failure to its typed error, keeping unproven cleanup owned. */ +const commandFailure = ( + cause: unknown, + request: KunaExportRequest, + subcommand: string, + limits: { readonly diagnosticBytes: number }, +): KunaExportOutcome => { + const { installation, signal } = request; + if (!(cause instanceof OwnedCommandFailure)) + return { + kind: "done", + result: err( + signal.aborted + ? new AnalysisCancelledError(KUNA_EXPORT_OPERATION, { cause }) + : new ProviderAdapterError(KUNA_PROVIDER_ID, KUNA_EXPORT_OPERATION, { + cause, + userMessage: `Cannot run ${installation.executable}: ${message(cause)}`, + }), + ), + }; + const snapshot = cause.snapshot; + const capturedOutput = + snapshot === null + ? undefined + : { + // An inventory is evidence-sized, not diagnostic-sized; keep only its tail. + stdout: snapshot.stdout.text.slice(-4096), + stderr: snapshot.stderr.text, + truncated: + snapshot.diagnosticTruncated === true || + snapshot.outputIncomplete === true || + snapshot.stdout.text.length > 4096, + exit_code: snapshot.exitCode ?? null, + signal: snapshot.signal ?? null, + }; + const details = { + cause, + ...(capturedOutput === undefined ? {} : { capturedOutput }), + }; + if (cause.cleanupFailure !== null) + return { + kind: "retained", + error: new ProviderCleanupError( + KUNA_PROVIDER_ID, + [...cause.resources, request.workspace], + { reason: cause.cleanupFailure }, + { operation: KUNA_EXPORT_OPERATION, cause }, + ), + }; + if (cause.reason === "cancelled") + return { + kind: "done", + result: err(new AnalysisCancelledError(KUNA_EXPORT_OPERATION, details)), + }; + if (cause.reason === "timeout") + return { + kind: "done", + result: err( + new AnalysisTimeoutError( + KUNA_EXPORT_OPERATION, + installation.configuration.projectTimeoutMs, + details, + ), + ), + }; + if (cause.reason === "output-limit") + return { + kind: "done", + result: err( + new AnalysisResourceConstraintError( + KUNA_EXPORT_OPERATION, + "capacity", + `kuna ${subcommand} wrote more than ${String(limits.diagnosticBytes)} bytes of output; the export was not retained.`, + { output_limit_bytes: limits.diagnosticBytes }, + details, + ), + ), + }; + const exitCode = capturedOutput?.exit_code; + return { + kind: "done", + result: err( + new ProviderAdapterError(KUNA_PROVIDER_ID, KUNA_EXPORT_OPERATION, { + ...details, + userMessage: `kuna ${subcommand} failed${exitCode === undefined || exitCode === null ? "" : ` with exit code ${String(exitCode)}`}: ${capturedOutput?.stderr.trim().split("\n").at(-1) ?? cause.message}`, + }), + ), + }; +}; diff --git a/src/kuna/KunaProjectFiles.ts b/src/kuna/KunaProjectFiles.ts new file mode 100644 index 0000000000..9f0714f58b --- /dev/null +++ b/src/kuna/KunaProjectFiles.ts @@ -0,0 +1,189 @@ +import { z } from "zod"; + +import { err, ok, type Result } from "../domain/result.js"; + +/** + * One function block in a `kuna decompile-project` `.c` file. The span covers + * its `// Function:` header through the blank line that ends the block, which + * is how REA serves one function without reading the whole file. + */ +export interface KunaProjectFunction { + readonly name: string; + /** Kuna's display address, a lowercase `0x…` virtual address. */ + readonly address: string; + readonly cOffset: number; + readonly cLength: number; + /** Kuna's own failure for this function; its block is then the header alone. */ + readonly error: string | null; +} + +// Rendered by kuna-console's `render_c`: `// Function: {name} @ 0x{addr:x}`, +// or `... @ 0x{addr:x} (error: {message})` for a failed function. A name +// can itself contain " @ ", so the address is the last such field. +const HEADER = /^\/\/ Function: (.+) @ (0x[0-9a-f]+)(?: {2}\(error: (.*)\))?$/u; +const HEADER_PREFIX = "// Function: "; + +/** Kuna's completed-export summary on stdout: `functions: N ok, M failed`. */ +const SUMMARY = /functions: (\d+) ok, (\d+) failed/u; + +/** Function counts Kuna reported for a finished export. */ +export interface KunaExportSummary { + readonly decompiled: number; + readonly failed: number; +} + +/** Read the completed-file summary Kuna prints after finalizing an export. */ +export const parseKunaExportSummary = ( + stdout: string, +): KunaExportSummary | undefined => { + const match = SUMMARY.exec(stdout); + if (match?.[1] === undefined || match[2] === undefined) return undefined; + return { decompiled: Number(match[1]), failed: Number(match[2]) }; +}; + +/** A header line longer than this is not one Kuna wrote. */ +const MAX_HEADER_BYTES = 64 * 1024; +const PREFIX_BYTES = Buffer.from(HEADER_PREFIX); + +interface Header { + readonly name: string; + readonly address: string; + readonly error: string | null; + readonly offset: number; +} + +/** + * Split a finished export's `.c` into its function blocks, reading it as a + * stream so a very large export is never held in memory. + * + * The `#include` preamble precedes the first header. The header count must + * equal the functions Kuna reported, and every address must be distinct; an + * export that fails either check is not a usable function inventory. + */ +export const scanKunaProjectSource = async ( + chunks: AsyncIterable, + summary: KunaExportSummary, +): Promise> => { + const headers: Header[] = []; + let offset = 0; + let lineStart = 0; + // Bytes of the current line, kept only while it may still be a header. + let line: Buffer[] | undefined = []; + let lineBytes = 0; + const endLine = (): string | undefined => { + if (line === undefined) return undefined; + const text = Buffer.concat(line).toString("utf8"); + if (!text.startsWith(HEADER_PREFIX)) return undefined; + const match = HEADER.exec(text); + if (match?.[1] === undefined || match[2] === undefined) + return `unrecognized function header at byte ${String(lineStart)}`; + headers.push({ + name: match[1], + address: match[2], + error: match[3] ?? null, + offset: lineStart, + }); + return undefined; + }; + for await (const chunk of chunks) { + let position = 0; + while (position < chunk.length) { + const newline = chunk.indexOf(0x0a, position); + const end = newline === -1 ? chunk.length : newline; + if (line !== undefined) { + const segment = chunk.subarray(position, end); + line.push(segment); + lineBytes += segment.length; + const seen = Buffer.concat(line).subarray(0, PREFIX_BYTES.length); + if (!PREFIX_BYTES.subarray(0, seen.length).equals(seen)) + line = undefined; + else if (lineBytes > MAX_HEADER_BYTES) + return err( + `function header at byte ${String(lineStart)} is too long`, + ); + } + if (newline === -1) break; + const failure = endLine(); + if (failure !== undefined) return err(failure); + position = newline + 1; + lineStart = offset + position; + line = []; + lineBytes = 0; + } + offset += chunk.length; + } + const failure = endLine(); + if (failure !== undefined) return err(failure); + const total = offset; + const expected = summary.decompiled + summary.failed; + if (headers.length !== expected) + return err( + `the .c holds ${String(headers.length)} function headers but Kuna reported ${String(expected)} functions`, + ); + const failed = headers.filter(({ error }) => error !== null).length; + if (failed !== summary.failed) + return err( + `the .c marks ${String(failed)} failed functions but Kuna reported ${String(summary.failed)}`, + ); + const addresses = new Set(); + const functions: KunaProjectFunction[] = []; + for (const [index, header] of headers.entries()) { + if (addresses.has(header.address)) + return err(`the .c repeats address ${header.address}`); + addresses.add(header.address); + const end = headers[index + 1]?.offset ?? total; + functions.push({ + name: header.name, + address: header.address, + cOffset: header.offset, + cLength: end - header.offset, + error: header.error, + }); + } + return ok(functions); +}; + +const inventorySchema = z.object({ + functions: z.array( + z.object({ + name: z.string(), + address_hex: z.string().regex(/^0x[0-9a-f]+$/u), + aliases: z.array(z.string()), + size: z.number().int().nonnegative().max(Number.MAX_SAFE_INTEGER), + }), + ), +}); + +/** One callable entry from `kuna functions --json`, including imports Kuna does not decompile. */ +export interface KunaInventoryFunction { + readonly name: string; + readonly address: string; + readonly aliases: readonly string[]; + /** Inventory extent: an upper bound clipped at the next entry, or 0 when unmeasured. */ + readonly size: number; +} + +/** Parse the callable-symbol inventory `kuna functions --json` writes. */ +export const parseKunaInventory = ( + text: string, +): Result => { + let value: unknown; + try { + value = JSON.parse(text); + } catch { + return err("the function inventory is not JSON"); + } + const parsed = inventorySchema.safeParse(value); + if (!parsed.success) + return err( + `the function inventory does not match Kuna's document: ${parsed.error.issues[0]?.message ?? "invalid"}`, + ); + return ok( + parsed.data.functions.map((entry) => ({ + name: entry.name, + address: entry.address_hex, + aliases: entry.aliases, + size: entry.size, + })), + ); +}; diff --git a/src/kuna/KunaRelease.ts b/src/kuna/KunaRelease.ts new file mode 100644 index 0000000000..e1ff9b8311 --- /dev/null +++ b/src/kuna/KunaRelease.ts @@ -0,0 +1,16 @@ +/** Kuna release whose CLI and project layout REA verified; others report as unverified. */ +export const KUNA_VERIFIED_VERSION = "1.833"; + +/** Stable provider ID shared by readiness, doctor and Evidence. */ +export const KUNA_PROVIDER_ID = "kuna"; + +/** Remediation when `REA_KUNA_COMMAND` is missing or not an absolute path. */ +export const KUNA_COMMAND_REMEDIATION = `Set REA_KUNA_COMMAND to the absolute path of a caller-supplied kuna executable (verified with ${KUNA_VERIFIED_VERSION}). REA does not download, install or update Kuna.`; + +/** Remediation when the host cannot supervise a Kuna process tree. */ +export const KUNA_UNSUPPORTED_HOST_REMEDIATION = + "Use Linux or macOS for the Kuna provider. Windows process-tree ownership for Kuna is not verified."; + +/** Remediation when `kuna --version` cannot establish the installed release. */ +export const KUNA_VERSION_PROBE_REMEDIATION = + "Run the selected executable with --version: it must exit promptly and print `kuna .`."; diff --git a/src/main.ts b/src/main.ts index 3d32798b70..0743843ba0 100644 --- a/src/main.ts +++ b/src/main.ts @@ -1,6 +1,7 @@ #!/usr/bin/env node import { snapshotEnvironment } from "./process/snapshotEnvironment.js"; +import { CURRENT_ENTRYPOINT_PATH } from "./process/CurrentEntrypointPath.js"; import { pathToFileURL } from "node:url"; import { realpathSync } from "node:fs"; @@ -109,10 +110,19 @@ export const runEntrypoint = async ( } }; -const entryPath = process.argv[1]; -if ( - entryPath !== undefined && - pathToFileURL(realpathSync(entryPath)).href === import.meta.url -) { +/** Return false when the runtime's argv path is synthetic or no longer exists. */ +const isDirectEntrypoint = ( + entryPath: string | undefined, + moduleUrl: string, +): boolean => { + if (entryPath === undefined) return false; + try { + return pathToFileURL(realpathSync(entryPath)).href === moduleUrl; + } catch { + return false; + } +}; + +if (isDirectEntrypoint(CURRENT_ENTRYPOINT_PATH, import.meta.url)) { void runEntrypoint(); } diff --git a/src/main/transport.ts b/src/main/transport.ts index eb7b157d75..4846bedd9d 100644 --- a/src/main/transport.ts +++ b/src/main/transport.ts @@ -115,12 +115,17 @@ export const startMcpTransport = async ( }, ); } catch (cause: unknown) { - const [binaryCleanup, androidCleanup, serverCleanup] = - await Promise.allSettled([ - session.close(), - closeAndroid(), - closeServers(), - ]); + const [ + binaryCleanup, + androidCleanup, + serverCleanup, + optionalProviderCleanup, + ] = await Promise.allSettled([ + session.close(), + closeAndroid(), + closeServers(), + closeOptionalProviders(), + ]); serverLogger.error( { error: cause instanceof Error ? cause.message : String(cause), @@ -136,7 +141,12 @@ export const startMcpTransport = async ( ); dependencies.writeStderr(`${cleanupError.message}\n`); } - for (const cleanup of [binaryCleanup, androidCleanup, serverCleanup]) { + for (const cleanup of [ + binaryCleanup, + androidCleanup, + serverCleanup, + optionalProviderCleanup, + ]) { if (cleanup.status === "rejected") { const reason = cleanup.reason instanceof Error diff --git a/src/mcpDoctor.ts b/src/mcpDoctor.ts index d7bf1e366f..d1dc0d34a7 100644 --- a/src/mcpDoctor.ts +++ b/src/mcpDoctor.ts @@ -3,8 +3,14 @@ import { StdioClientTransport } from "@modelcontextprotocol/client/stdio"; import { Formatter, Help, z } from "incur"; import { performance } from "node:perf_hooks"; import { resolve } from "node:path"; +import { StringDecoder } from "node:string_decoder"; -import { CATALOG_IDENTITY } from "./catalogIdentity.js"; +import { createCatalogIdentity } from "./catalogIdentity.js"; +import { + compactPresentationForMcpInputSchemaProfile, + parseMcpInputSchemaProfile, +} from "./config/mcpInputSchemaProfile.js"; +import { parseMcpOutputSchemaProfile } from "./config/mcpOutputSchemaProfile.js"; import { PROMPT_CONTRACTS } from "./contracts/promptContracts.js"; import { TOOL_CONTRACTS } from "./contracts/toolContracts.js"; import { err, ok } from "./domain/result.js"; @@ -74,6 +80,7 @@ interface McpDoctorCheck { /** Diagnose the actual production stdio adapter with one startup-bounded SDK session. */ export const runProductionMcpDoctor = async (options: McpDoctorOptions) => { + const catalog = catalogIdentityForEnvironment(options.environment); const deadline = performance.now() + (options.deadlineMs ?? MCP_STARTUP_POLICY.doctorDeadlineMs); @@ -103,6 +110,7 @@ export const runProductionMcpDoctor = async (options: McpDoctorOptions) => { client, deadline, controller.signal, + catalog, ); } catch (cause: unknown) { result = { @@ -188,6 +196,7 @@ const inspectProductionMcpSession = async ( client: Client, deadline: number, signal: AbortSignal, + catalog: ReturnType, ) => { const request = requestOptions(deadline, signal); const [tools, prompts] = await Promise.all([ @@ -237,8 +246,7 @@ const inspectProductionMcpSession = async ( name: "server-identity", ok: observedIdentity.packageVersion === PRODUCT_IDENTITY.packageVersion && - observedIdentity.catalogDigest === - CATALOG_IDENTITY.digests.combined_sha256, + observedIdentity.catalogDigest === catalog.digests.combined_sha256, detail: `package=${observedIdentity.packageVersion ?? "missing"}; catalog=${observedIdentity.catalogDigest ?? "missing"}`, }, { @@ -261,7 +269,7 @@ const inspectProductionMcpSession = async ( identity: { expected_package_version: PRODUCT_IDENTITY.packageVersion, observed_package_version: observedIdentity.packageVersion, - expected_catalog_digest: CATALOG_IDENTITY.digests.combined_sha256, + expected_catalog_digest: catalog.digests.combined_sha256, observed_catalog_digest: observedIdentity.catalogDigest, }, request_flow: { @@ -280,6 +288,7 @@ export const runProductionMcpDoctorCli = async ( arguments_: readonly string[], input: { readonly dispatcherPath: string; readonly packageRoot: string }, ): Promise<{ readonly output: string; readonly exitCode: number }> => { + const started = performance.now(); const parsed = parseOutputArguments(arguments_); if (!parsed.ok) { if (parsed.help) @@ -298,8 +307,18 @@ export const runProductionMcpDoctorCli = async ( cwd: input.packageRoot, environment: process.env, }); + const outputValue = parsed.fullOutput + ? { + ok: result.healthy, + ...(result.healthy ? { data: result } : { error: result }), + meta: { + command: MCP_DOCTOR_COMMAND, + duration: `${Math.round(performance.now() - started)}ms`, + }, + } + : result; return { - output: `${Formatter.format(result, parsed.format)}\n`, + output: `${Formatter.format(outputValue, parsed.format)}\n`, exitCode: result.healthy ? 0 : 1, }; }; @@ -369,7 +388,11 @@ const parseIdentityToolResult = ( * flag is an output selector declared in `MCP_DOCTOR_OPTIONS`. */ type McpDoctorArguments = - | { readonly ok: true; readonly format: OutputFormat } + | { + readonly ok: true; + readonly format: OutputFormat; + readonly fullOutput: boolean; + } | { readonly ok: false; readonly help: true } | { readonly ok: false; @@ -387,6 +410,7 @@ const parseOutputArguments = ( // masking the typo with usage. if (hasHelpInOptionPosition(arguments_)) return { ok: false, help: true }; let format: OutputFormat = "toon"; + let fullOutput = false; for (let index = 0; index < arguments_.length; index += 1) { const argument = arguments_[index]; if (argument === "--json") { @@ -409,7 +433,10 @@ const parseOutputArguments = ( format = candidate; continue; } - if (argument === "--full-output") continue; + if (argument === "--full-output") { + fullOutput = true; + continue; + } if (HELP_FLAGS.has(argument ?? "")) return { ok: false, help: true }; return { ok: false, @@ -418,7 +445,7 @@ const parseOutputArguments = ( message: `Unknown mcp doctor option: ${argument ?? "missing"}`, }; } - return { ok: true, format }; + return { ok: true, format, fullOutput }; }; /** @@ -463,20 +490,48 @@ const targetFreeEnvironment = ( return result; }; +const catalogIdentityForEnvironment = ( + environment: Readonly>, +): ReturnType => { + const inputProfile = parseMcpInputSchemaProfile( + environment.REA_MCP_INPUT_SCHEMA_PROFILE, + ); + const outputProfile = parseMcpOutputSchemaProfile( + environment.REA_MCP_OUTPUT_SCHEMA_PROFILE, + ); + return createCatalogIdentity( + compactPresentationForMcpInputSchemaProfile( + inputProfile.ok ? inputProfile.value : undefined, + ), + outputProfile.ok ? (outputProfile.value ?? "full") : "full", + ); +}; + const captureStderr = () => { - const chunks: Buffer[] = []; + const retained = Buffer.allocUnsafe(MCP_STARTUP_POLICY.doctorStderrBytes); + let retainedBytes = 0; let bytes = 0; + let truncated = false; return { append(chunk: unknown) { const buffer = Buffer.isBuffer(chunk) ? chunk : Buffer.from(String(chunk), "utf8"); bytes += buffer.byteLength; - chunks.push(Buffer.from(buffer)); + const remaining = retained.byteLength - retainedBytes; + const copied = Math.min(remaining, buffer.byteLength); + if (copied > 0) { + buffer.copy(retained, retainedBytes, 0, copied); + retainedBytes += copied; + } + if (copied < buffer.byteLength) truncated = true; }, value: () => ({ - stderr: Buffer.concat(chunks, bytes).toString("utf8"), + stderr: truncated + ? new StringDecoder("utf8").write(retained.subarray(0, retainedBytes)) + : retained.toString("utf8", 0, retainedBytes), stderr_bytes: bytes, + stderr_truncated: truncated, }), }; }; diff --git a/src/mcpStartupPolicy.ts b/src/mcpStartupPolicy.ts index dc31114c1d..96feb8a061 100644 --- a/src/mcpStartupPolicy.ts +++ b/src/mcpStartupPolicy.ts @@ -1,16 +1,61 @@ +import type { AnalysisProviderSelector } from "./contracts/providerSelection.js"; + /** - * Supported MCP startup deadlines and regression budgets. + * Supported MCP startup deadlines and operation headroom. * - * The Codex timeout is caller-facing configuration. The smaller phase budgets + * Client timeouts are caller-facing configuration. The smaller phase budgets * preserve operational headroom, while the doctor deadline covers additional * inventory requests after initialization. */ export const MCP_STARTUP_POLICY = { codexStartupTimeoutSeconds: 30, kimiStartupTimeoutMs: 30_000, - kimiToolTimeoutMs: 120_000, + toolOperationHeadroomMs: 120_000, initializeBudgetMs: 15_000, toolsListBudgetMs: 5_000, firstCatalogBudgetMs: 20_000, doctorDeadlineMs: 30_000, + doctorStderrBytes: 64 * 1024, } as const; + +/** Parsed startup and upstream request budgets affecting a client's tool deadline. */ +export interface McpToolTimeoutInputs { + readonly analysisProvider: AnalysisProviderSelector; + readonly ghidraStartupTimeoutMs: number; + readonly hopperStartupTimeoutMs: number; + readonly jebMcpTimeoutMs?: number; + readonly providerRequestTimeoutMs?: number; +} + +/** Allow for known startup phases and caller-selected upstream request budgets. */ +export const mcpProviderDeadlineAllowanceMs = ( + inputs: McpToolTimeoutInputs, +): number => { + const selectedProvider = inputs.analysisProvider; + const selectedTimeout = + selectedProvider === "ghidra" + ? inputs.ghidraStartupTimeoutMs + : selectedProvider === "hopper" + ? inputs.hopperStartupTimeoutMs + : Math.max( + inputs.ghidraStartupTimeoutMs, + inputs.hopperStartupTimeoutMs, + ); + return Math.max( + selectedTimeout, + inputs.jebMcpTimeoutMs ?? 0, + inputs.providerRequestTimeoutMs ?? 0, + ); +}; + +/** Add operation headroom to the parsed provider deadline allowance. */ +export const recommendedMcpToolTimeoutMs = ( + inputs: McpToolTimeoutInputs, +): number => + mcpProviderDeadlineAllowanceMs(inputs) + + MCP_STARTUP_POLICY.toolOperationHeadroomMs; + +/** Round up so Codex's whole-second deadline still covers the millisecond budget. */ +export const recommendedMcpToolTimeoutSeconds = ( + inputs: McpToolTimeoutInputs, +): number => Math.ceil(recommendedMcpToolTimeoutMs(inputs) / 1000); diff --git a/src/native/AppleDispatchCoverage.ts b/src/native/AppleDispatchCoverage.ts index a6c68ca512..44c6b9f369 100644 --- a/src/native/AppleDispatchCoverage.ts +++ b/src/native/AppleDispatchCoverage.ts @@ -27,6 +27,8 @@ export const pushDispatchCoverage = (input: { readonly examined: number; readonly categoriesExamined: number; readonly truncated: boolean; + readonly objcListScanTruncated: boolean; + readonly objcClassListScanTruncated: boolean; readonly fixups: PointerFixups; readonly vtables: FacetDecodeFacts; readonly swift: FacetDecodeFacts; @@ -38,6 +40,8 @@ export const pushDispatchCoverage = (input: { examined, categoriesExamined, truncated, + objcListScanTruncated, + objcClassListScanTruncated, fixups, vtables, swift, @@ -78,7 +82,7 @@ export const pushDispatchCoverage = (input: { const classReasons = [ ...failures, ...classRecords, - ...(truncated ? ["max_records_reached"] : []), + ...(truncated || objcListScanTruncated ? ["max_records_reached"] : []), ]; const categoryMethods = result.objc_dispatch_implementations.filter( ({ category }) => category !== undefined, @@ -99,20 +103,18 @@ export const pushDispatchCoverage = (input: { ({ decode }) => decode?.status === "decoded", ).length, }); + const relativePointerReasons = [ + ...(truncated || objcClassListScanTruncated ? ["max_records_reached"] : []), + ...(result.relative_pointers.some( + (item) => item.decode.status !== "decoded", + ) + ? ["relative_pointer_targets_unresolved"] + : []), + ]; result.coverage.push({ facet: "binary_relative_pointers", - status: - truncated || - result.relative_pointers.some((item) => item.decode.status !== "decoded") - ? "partial" - : "complete", - reason: truncated - ? "max_records_reached" - : result.relative_pointers.some( - (item) => item.decode.status !== "decoded", - ) - ? "relative_pointer_targets_unresolved" - : null, + status: relativePointerReasons.length > 0 ? "partial" : "complete", + reason: relativePointerReasons.join("; ") || null, examined: result.relative_pointers.length, decoded: result.relative_pointers.filter( (item) => item.decode.status === "decoded", @@ -126,6 +128,7 @@ export const pushDispatchCoverage = (input: { facet: "objc_properties_categories", status: truncated || + objcClassListScanTruncated || categoryFailures.length > 0 || result.objc_categories.some( ({ decode }) => decode.status !== "decoded", @@ -137,7 +140,9 @@ export const pushDispatchCoverage = (input: { [ ...categoryFailures, ...categoryMethodReasons, - ...(truncated ? ["max_records_reached"] : []), + ...(truncated || objcClassListScanTruncated + ? ["max_records_reached"] + : []), ].join("; ") || null, examined: categoriesExamined, decoded: result.objc_categories.filter( diff --git a/src/native/AppleDispatchMetadata.test.ts b/src/native/AppleDispatchMetadata.test.ts index 244a35aeb3..f1af48e436 100644 --- a/src/native/AppleDispatchMetadata.test.ts +++ b/src/native/AppleDispatchMetadata.test.ts @@ -1,4 +1,5 @@ import { describe, expect, it } from "vitest"; +import { machoWithPropertyAttributes } from "../../tests/fixtures/objcPropertyMacho.js"; import { decodeAppleDispatchMetadata } from "./AppleDispatchMetadata.js"; const base = 0x100000000n; @@ -115,6 +116,50 @@ const fixture = ( return bytes; }; const provenance = { path: "/fixture/app", sha256: "a".repeat(64) }; + +const partialSiblingsFixture = ( + options: { + readonly emptyMiddleClass?: boolean; + } = {}, +) => { + const bytes = Buffer.alloc(4096); + const imageBase = 0x100000000n; + bytes.writeUInt32LE(0xfeedfacf, 0); + bytes.writeUInt32LE(0x0100000c, 4); + bytes.writeUInt32LE(2, 12); + bytes.writeUInt32LE(1, 16); + bytes.writeUInt32LE(232, 20); + bytes.writeUInt32LE(0x19, 32); + bytes.writeUInt32LE(232, 36); + bytes.write("__DATA", 40); + bytes.writeBigUInt64LE(imageBase, 56); + bytes.writeBigUInt64LE(4096n, 64); + bytes.writeBigUInt64LE(4096n, 80); + bytes.writeUInt32LE(5, 92); + bytes.writeUInt32LE(2, 96); + bytes.write("__objc_classlist", 104); + bytes.writeBigUInt64LE(imageBase + 0x200n, 136); + bytes.writeBigUInt64LE(24n, 144); + bytes.write("__objc_protolist", 184); + bytes.writeBigUInt64LE(imageBase + 0x300n, 216); + bytes.writeBigUInt64LE(8n, 224); + for (const [slot, record, ro, name] of [ + [0x200, 0x400, 0x500, 0x600], + [0x208, 0x440, 0x540, 0x620], + [0x210, 0x480, 0x580, 0x640], + ] as const) { + bytes.writeBigUInt64LE(imageBase + BigInt(record), slot); + bytes.writeBigUInt64LE(imageBase + BigInt(ro), record + 32); + bytes.writeUInt32LE(2, ro); + bytes.writeUInt32LE(16, ro + 8); + bytes.writeBigUInt64LE(imageBase + BigInt(name), ro + 24); + } + bytes.write("KeptClass\0", 0x600); + if (!options.emptyMiddleClass) bytes.write("OtherClass\0", 0x620); + bytes.write("LaterClass\0", 0x640); + return bytes; +}; + describe("Apple dispatch pointer fixup metadata", () => { it("keeps decoded bindings and their coverage when an independent stream fails", () => { const bytes = fixture(); @@ -279,6 +324,19 @@ describe("Apple dispatch binary metadata", () => { }); }, ); + + it.each([ + [0xffffffff, 8], + [3, 8], + [0xfffffffe, null], + ])("decodes ivar alignment raw value %i as %s", (raw, alignment) => { + const bytes = fixture(); + bytes.writeUInt32LE(raw, 0x520); + const result = decodeAppleDispatchMetadata(bytes, 100, provenance); + expect(result.objc_ivars).toMatchObject([ + { name: "state", alignment, decode: { status: "decoded" } }, + ]); + }); }); describe("Apple dispatch coverage ownership", () => { @@ -366,7 +424,56 @@ describe("Apple dispatch coverage ownership", () => { }); }); +describe("malformed Objective-C property records", () => { + it.each([ + ["an unmapped name", { secondNameOffset: 0x2000 }, "first", "0x100000518"], + ["an empty name", { firstName: "" }, "second", "0x100000508"], + ] as const)( + "retains the valid sibling of a property with %s", + (_, layout, kept, location) => { + const result = decodeAppleDispatchMetadata( + machoWithPropertyAttributes("Tq,N", layout), + 100, + provenance, + ); + expect( + result.objc_classes.map(({ name, properties }) => ({ + name, + properties: properties.map((property) => property.name), + })), + ).toEqual([{ name: "Fixture", properties: [kept] }]); + expect( + result.coverage.find( + ({ facet }) => facet === "objc_properties_categories", + ), + ).toMatchObject({ + status: "partial", + reason: expect.stringContaining( + `property_decode_failed at ${location}: Fixture: property`, + ), + }); + }, + ); +}); + describe("malformed Apple dispatch metadata", () => { + it("retains valid class siblings when one class has an empty name", () => { + const result = decodeAppleDispatchMetadata( + partialSiblingsFixture({ emptyMiddleClass: true }), + 100, + provenance, + ); + expect(result.objc_classes.map(({ name }) => name)).toEqual([ + "KeptClass", + "LaterClass", + ]); + expect( + result.coverage.find( + ({ facet }) => facet === "objc_class_method_ivar_metadata", + ), + ).toMatchObject({ status: "partial", examined: 3 }); + }); + it("rejects malformed command and section boundaries", () => { const bytes = fixture(); bytes.writeUInt32LE(7, 36); @@ -530,3 +637,218 @@ describe("byte-swapped universal Apple dispatch metadata", () => { ); }); }); + +const unmappedProtocolSectionBase = 0x100000000n; +const unmappedProtocolImage = (bad: "protocol" | "class" | "neither") => { + const bytes = Buffer.alloc(4096); + bytes.writeUInt32LE(0xfeedfacf, 0); + bytes.writeUInt32LE(0x0100000c, 4); + bytes.writeUInt32LE(2, 12); + bytes.writeUInt32LE(1, 16); + bytes.writeUInt32LE(232, 20); + bytes.writeUInt32LE(0x19, 32); + bytes.writeUInt32LE(232, 36); + bytes.write("__DATA", 40); + bytes.writeBigUInt64LE(unmappedProtocolSectionBase, 56); + bytes.writeBigUInt64LE(4096n, 64); + bytes.writeBigUInt64LE(4096n, 80); + bytes.writeUInt32LE(5, 92); + bytes.writeUInt32LE(2, 96); + bytes.write("__objc_classlist", 104); + bytes.writeBigUInt64LE( + unmappedProtocolSectionBase + (bad === "class" ? 0x9000n : 0x200n), + 136, + ); + bytes.writeBigUInt64LE(8n, 144); + bytes.write("__objc_protolist", 184); + bytes.writeBigUInt64LE( + unmappedProtocolSectionBase + (bad === "protocol" ? 0x9000n : 0x300n), + 216, + ); + bytes.writeBigUInt64LE(8n, 224); + bytes.writeBigUInt64LE(unmappedProtocolSectionBase + 0x400n, 0x200); + bytes.writeBigUInt64LE(unmappedProtocolSectionBase + 0x500n, 0x420); + bytes.writeUInt32LE(2, 0x500); + bytes.writeUInt32LE(16, 0x508); + bytes.writeBigUInt64LE(unmappedProtocolSectionBase + 0x600n, 0x518); + bytes.write("RetainedClass\0", 0x600); + return bytes; +}; +const unmappedProtocolClassCoverage = (result: { + coverage: { facet: string; status: string; reason: string | null }[]; +}) => + result.coverage.find( + ({ facet }) => facet === "objc_class_method_ivar_metadata", + ); + +describe("Apple dispatch unmapped protocol-list slot", () => { + it("returns the class when every list slot is mapped", () => { + const result = decodeAppleDispatchMetadata( + unmappedProtocolImage("neither"), + 100, + provenance, + ); + expect(result.objc_classes).toMatchObject([{ name: "RetainedClass" }]); + expect(unmappedProtocolClassCoverage(result)).toMatchObject({ + status: "complete", + }); + }); + + it("records partial coverage when the class-list slot is unmapped", () => { + const result = decodeAppleDispatchMetadata( + unmappedProtocolImage("class"), + 100, + provenance, + ); + expect(unmappedProtocolClassCoverage(result)).toMatchObject({ + status: "partial", + reason: expect.stringContaining("0x100009000"), + }); + }); + + it("retains the class and partial coverage when the protocol-list slot is unmapped", () => { + const result = decodeAppleDispatchMetadata( + unmappedProtocolImage("protocol"), + 100, + provenance, + ); + expect(result.objc_classes).toMatchObject([{ name: "RetainedClass" }]); + expect(unmappedProtocolClassCoverage(result)).toMatchObject({ + status: "partial", + reason: expect.stringContaining("0x100009000"), + }); + }); +}); + +describe("Apple dispatch list scan budget", () => { + it("bounds failed protocol-list slots by the record budget", () => { + // The section size is file-declared, so a crafted image can claim an + // arbitrarily large unmapped protocol list. Failed slots must spend the + // record budget like decoded ones, otherwise the loop runs unbounded. + const bytes = unmappedProtocolImage("protocol"); + bytes.writeBigUInt64LE(80000n, 224); + const result = decodeAppleDispatchMetadata(bytes, 100, provenance); + const coverage = unmappedProtocolClassCoverage(result); + expect(coverage).toMatchObject({ status: "partial" }); + expect(coverage?.reason).toContain("max_records_reached"); + const failures = + coverage?.reason?.match(/Unmapped or ambiguous metadata pointer/g) ?? []; + expect(failures.length).toBeLessThanOrEqual(100); + }); + + it("keeps independent facets complete when only zero-pointer list scanning is capped", () => { + const bytes = unmappedProtocolImage("neither"); + bytes.writeBigUInt64LE(0n, 144); + bytes.writeBigUInt64LE(16n, 224); + const result = decodeAppleDispatchMetadata(bytes, 1, provenance); + expect(unmappedProtocolClassCoverage(result)).toMatchObject({ + status: "partial", + reason: expect.stringContaining("max_records_reached"), + }); + for (const facet of [ + "objc_properties_categories", + "binary_relative_pointers", + ]) + expect( + result.coverage.find((item) => item.facet === facet), + ).toMatchObject({ + status: "complete", + reason: null, + }); + }); + + it("marks class-dependent coverage partial when zero slots hide a later class", () => { + const bytes = unmappedProtocolImage("neither"); + bytes.writeBigUInt64LE(0n, 224); + bytes.writeBigUInt64LE(16n, 144); + bytes.writeBigUInt64LE(0n, 0x200); + bytes.writeBigUInt64LE(unmappedProtocolSectionBase + 0x400n, 0x208); + const result = decodeAppleDispatchMetadata(bytes, 1, provenance); + expect(result.objc_classes).toEqual([]); + for (const facet of [ + "objc_class_method_ivar_metadata", + "objc_properties_categories", + "binary_relative_pointers", + ]) + expect( + result.coverage.find((item) => item.facet === facet), + ).toMatchObject({ + status: "partial", + reason: expect.stringContaining("max_records_reached"), + }); + for (const facet of [ + "swift_class_vtable_descriptors", + "swift_conformances_static_witness_slots", + ]) + expect( + result.coverage.find((item) => item.facet === facet), + ).toMatchObject({ status: "complete" }); + }); + + it("bounds failed class-list slots by the record budget", () => { + // Same file-declared-size hazard as the protocol list: a crafted image + // can declare an arbitrarily large unmapped class list. Failed class + // slots must spend the record budget like decoded ones, otherwise the + // loop runs unbounded (each failed read produced an unbounded failure + // string with no budget spend). + const bytes = unmappedProtocolImage("class"); + bytes.writeBigUInt64LE(80000n, 144); + const result = decodeAppleDispatchMetadata(bytes, 100, provenance); + const coverage = unmappedProtocolClassCoverage(result); + expect(coverage).toMatchObject({ status: "partial" }); + expect(coverage?.reason).toContain("max_records_reached"); + const failures = + coverage?.reason?.match(/Unmapped or ambiguous metadata pointer/g) ?? []; + expect(failures.length).toBeLessThanOrEqual(100); + }); + + it("reports the real failure for a mapped-but-malformed class at the budget boundary", () => { + // The class-list slot itself is mapped, so readClass admits the record + // budget and only then throws on the unmapped class_ro pointer. The + // failure must be recorded as-is: it already spent one budget unit, so + // admitting again would break before the reason is appended and report + // only max_records_reached. + const bytes = unmappedProtocolImage("neither"); + bytes.writeBigUInt64LE(unmappedProtocolSectionBase + 0x700n, 0x200); + bytes.writeBigUInt64LE(unmappedProtocolSectionBase + 0x9000n, 0x720); + const result = decodeAppleDispatchMetadata(bytes, 1, provenance); + const coverage = unmappedProtocolClassCoverage(result); + expect(coverage).toMatchObject({ status: "partial" }); + expect(coverage?.reason).toContain( + "Unmapped or ambiguous metadata pointer 0x100009018", + ); + expect(coverage?.reason).not.toContain("max_records_reached"); + }); + + it("does not skip a valid class after a mapped-but-malformed class", () => { + // With two slots and a budget of two, the corrupt first slot must spend + // exactly one unit: admitting twice would exhaust the budget and skip + // the valid class behind it. + const bytes = unmappedProtocolImage("neither"); + bytes.writeBigUInt64LE(16n, 144); + bytes.writeBigUInt64LE(unmappedProtocolSectionBase + 0x700n, 0x200); + bytes.writeBigUInt64LE(unmappedProtocolSectionBase + 0x9000n, 0x720); + bytes.writeBigUInt64LE(unmappedProtocolSectionBase + 0x400n, 0x208); + const result = decodeAppleDispatchMetadata(bytes, 2, provenance); + expect(result.objc_classes).toMatchObject([{ name: "RetainedClass" }]); + expect(unmappedProtocolClassCoverage(result)?.reason).toContain( + "Unmapped or ambiguous metadata pointer 0x100009018", + ); + }); + + it("reports the real failure for a mapped-but-truncated protocol slot", () => { + // The protocol slot is mapped, so protocolReader.record admits the + // budget itself before throwing on the truncated declaration. The loop + // must not admit a second time; the recorded reason stays the + // declaration failure, not max_records_reached. + const bytes = unmappedProtocolImage("neither"); + bytes.writeBigUInt64LE(unmappedProtocolSectionBase + 0x700n, 0x300); + const result = decodeAppleDispatchMetadata(bytes, 100, provenance); + const coverage = unmappedProtocolClassCoverage(result); + expect(coverage).toMatchObject({ status: "partial" }); + expect(coverage?.reason).toContain( + "Truncated Objective-C protocol declaration", + ); + expect(coverage?.reason).not.toContain("max_records_reached"); + }); +}); diff --git a/src/native/AppleDispatchMetadata.ts b/src/native/AppleDispatchMetadata.ts index 26ff8caaf5..dfc5aec6a7 100644 --- a/src/native/AppleDispatchMetadata.ts +++ b/src/native/AppleDispatchMetadata.ts @@ -105,15 +105,22 @@ export const decodeAppleDispatchMetadata = ( evidence, offset, }; - const { failures, categoryIssues, examined, categoriesExamined } = - decodeObjcDispatchFacets({ - bytes, - sections, - segments, - readers, - budget, - result, - }); + const { + failures, + categoryIssues, + examined, + categoriesExamined, + objcListScanTruncated, + objcClassListScanTruncated, + } = decodeObjcDispatchFacets({ + bytes, + sections, + segments, + readers, + budget, + maxListSlots: maxRecords, + result, + }); const swiftRelative = createSwiftRelativeReader({ bytes, readers, @@ -163,6 +170,8 @@ export const decodeAppleDispatchMetadata = ( categoryIssues, examined, categoriesExamined, + objcListScanTruncated, + objcClassListScanTruncated, truncated: budget.truncated, fixups, vtables, diff --git a/src/native/AppleMachoFixups.test.ts b/src/native/AppleMachoFixups.test.ts index bf4ae348f2..f3f84a76c1 100644 --- a/src/native/AppleMachoFixups.test.ts +++ b/src/native/AppleMachoFixups.test.ts @@ -907,6 +907,47 @@ describe("legacy pointer fixup streams", () => { target: 0x99n, }); }); + + it("wraps a negative DO_BIND_ADD_ADDR_ULEB step and keeps decoding", () => { + // The unsigned 64-bit encoding of -8 revisits the current pointer slot. + const back = [0xf8, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0x01]; + const opcodes = Buffer.from([ + 0x11, // SET_DYLIB_ORDINAL_IMM 1 + 0x40, + ...Buffer.from("__ZdlPv\0"), + 0x71, + 0x08, // SET_SEGMENT_AND_OFFSET segment 1 + 8 + 0xa0, + ...back, // DO_BIND_ADD_ADDR_ULEB -8 + 0x90, // DO_BIND the same slot again + 0x12, // SET_DYLIB_ORDINAL_IMM 2 + 0x40, + ...Buffer.from("_OBJC_CLASS_$_NSObject\0"), + 0x90, // DO_BIND the next slot + 0x00, + ]); + const fixups = parsePointerFixups( + opcodes, + { + chained: null, + binds: [{ offset: 0, size: opcodes.length, stream: "bind" }], + rebases: [], + dylibs: ["/usr/lib/libc++.1.dylib", "/usr/lib/libobjc.A.dylib"], + }, + SEGMENTS, + BASE, + ); + expect(fixups.failures).toEqual([]); + expect(fixups.decode(BASE + 0x4010n, 0n)).toMatchObject({ + kind: "bind", + symbol: "_OBJC_CLASS_$_NSObject", + library: "/usr/lib/libobjc.A.dylib", + }); + expect(fixups.decode(BASE + 0x4018n, 0x99n)).toEqual({ + kind: "rebase", + target: 0x99n, + }); + }); }); describe("Objective-C property attributes", () => { @@ -942,4 +983,17 @@ describe("Objective-C property attributes", () => { expect(parsePropertyAttributes("count", "").atomicity).toBeNull(); expect(parsePropertyAttributes("count", "R").atomicity).toBeNull(); }); + + it("establishes no defaults from an unfinished quoted class name", () => { + // `,N,R` here lie inside the unterminated quote, so they are not flags. + expect(parsePropertyAttributes("title", 'T@"NSString,N,R')).toMatchObject({ + type_encoding: '@"NSString,N,R', + atomicity: null, + is_readonly: null, + }); + expect(parsePropertyAttributes("title", 'N,R,T@"NSString')).toMatchObject({ + atomicity: "nonatomic", + is_readonly: true, + }); + }); }); diff --git a/src/native/AppleMachoFixups.ts b/src/native/AppleMachoFixups.ts index 09080ec1d6..bafb676f87 100644 --- a/src/native/AppleMachoFixups.ts +++ b/src/native/AppleMachoFixups.ts @@ -652,7 +652,9 @@ const readBindStream = ( } else { binds.push({ start: state.address, count, stride, binding }); } - state.address += count * stride; + // The dyld bind stream advances addresses modulo 64 bits; ld uses -8 to + // bind the current pointer slot again with DO_BIND_ADD_ADDR_ULEB. + state.address = (state.address + count * stride) & 0xffffffffffffffffn; }; while (cursor.at < end) { const byte = bytes[cursor.at++] ?? 0; diff --git a/src/native/AppleObjcDispatchFacets.ts b/src/native/AppleObjcDispatchFacets.ts index b525458408..21cda550fe 100644 --- a/src/native/AppleObjcDispatchFacets.ts +++ b/src/native/AppleObjcDispatchFacets.ts @@ -1,3 +1,4 @@ +import { err, ok, type Result } from "../domain/result.js"; import type { ObjcSwiftMetadata } from "../domain/native/objcSwiftMetadata.js"; import type { Segment, Section } from "./AppleMachoSelection.js"; import { boundClassName, decodeObjcCategories } from "./AppleObjcCategories.js"; @@ -67,14 +68,25 @@ export const decodeObjcDispatchFacets = (input: { segments: readonly Segment[]; readers: ObjcDispatchReaders; budget: DispatchRecordBudget; + maxListSlots: number; result: ObjcSwiftMetadata; }): { failures: string[]; categoryIssues: DecodeIssue[]; examined: number; categoriesExamined: number; + objcListScanTruncated: boolean; + objcClassListScanTruncated: boolean; } => { - const { bytes, sections, segments, readers: read, budget, result } = input; + const { + bytes, + sections, + segments, + readers: read, + budget, + maxListSlots, + result, + } = input; const failures: string[] = []; const categoryIssues: DecodeIssue[] = []; let examined = 0; @@ -190,6 +202,33 @@ export const decodeObjcDispatchFacets = (input: { result, failures, }); + // Each list scan has its own slot bound: zero and duplicate pointers do + // not produce records, but still consume pointer-reading work. The bound + // is per list so one list's slots cannot starve the other list's scan. + // Record decoders retain their existing admission ownership. + let objcListScanTruncated = false; + let objcClassListScanTruncated = false; + const createListSlotReader = () => { + let listSlotsExamined = 0; + return (field: bigint): Result => { + if (listSlotsExamined >= maxListSlots) { + objcListScanTruncated = true; + return err("exhausted"); + } + listSlotsExamined++; + try { + return ok(read.pointer(field)); + } catch (cause: unknown) { + const admitted = budget.admit(); + failures.push( + `${hex(field)}: ${cause instanceof Error ? cause.message : String(cause)}`, + ); + return err(admitted ? "unmapped" : "exhausted"); + } + }; + }; + const readProtocolListSlot = createListSlotReader(); + const readClassListSlot = createListSlotReader(); for (const section of sections.filter( ({ name }) => name === "__objc_protolist", )) { @@ -200,7 +239,21 @@ export const decodeObjcDispatchFacets = (input: { budget.markTruncated(); break; } - protocolReader.record(read.pointer(section.address + BigInt(index * 8))); + const field = section.address + BigInt(index * 8); + const pointer = readProtocolListSlot(field); + if (!pointer.ok) { + if (pointer.error === "exhausted") break; + continue; + } + try { + protocolReader.record(pointer.value); + } catch (cause) { + // record() admits the slot itself before decoding, so admitting + // again here would count one protocol twice. + failures.push( + `${hex(field)}: ${cause instanceof Error ? cause.message : String(cause)}`, + ); + } } } const visited = new Set(); @@ -209,6 +262,10 @@ export const decodeObjcDispatchFacets = (input: { visited.add(hex(address)); const ro = read.pointer(address + 32n) & ~7n; const name = read.string(read.pointer(ro + 24n)); + if (name.length === 0) + throw new TypeError( + `Objective-C class at ${hex(address)} has an empty name`, + ); const root = (read.u32(ro) & 2) !== 0; const superclass = resolveSuperclass(read, address, name, failures); if (superclass.name === null && superclass.address === null && !root) @@ -235,14 +292,20 @@ export const decodeObjcDispatchFacets = (input: { void cause; failures.push(`Unresolved ivar offset pointer ${hex(encodedOffset)}`); } + const alignment = read.u32(entry + 24n); result.objc_ivars.push({ class_name: name, name: read.string(read.pointer(entry + 8n)), type_encoding: read.string(read.pointer(entry + 16n)), offset: value, size: read.u32(entry + 28n), + // objc4 ivar_t::alignment(): ~0 means word alignment (8 for 32-byte entries). alignment: - read.u32(entry + 24n) < 31 ? 2 ** read.u32(entry + 24n) : null, + alignment === 0xffffffff + ? 8 + : alignment < 31 + ? 2 ** alignment + : null, location: read.location(entry), decode: value === null @@ -320,11 +383,24 @@ export const decodeObjcDispatchFacets = (input: { budget.markTruncated(); break; } - examined++; const field = section.address + BigInt(index * 8); + const pointer = readClassListSlot(field); + if (pointer.ok || pointer.error === "unmapped") examined++; + if (!pointer.ok) { + if (pointer.error === "exhausted") { + objcClassListScanTruncated = true; + break; + } + continue; + } try { - readClass(read.pointer(field)); + readClass(pointer.value); } catch (cause) { + // The slot was already admitted inside readClass before anything + // could throw (its admit runs before the first pointer read), so + // admitting again here would count one class twice: at the budget + // boundary it would break before appending the failure reason, and + // with larger limits malformed classes would starve valid ones. failures.push( `${hex(field)}: ${cause instanceof Error ? cause.message : String(cause)}`, ); @@ -341,5 +417,12 @@ export const decodeObjcDispatchFacets = (input: { protocols: (list) => protocolReader.list(list), localClassName: (address) => localClassName(read, address), }); - return { failures, categoryIssues, examined, categoriesExamined }; + return { + failures, + categoryIssues, + examined, + categoriesExamined, + objcListScanTruncated, + objcClassListScanTruncated, + }; }; diff --git a/src/native/AppleObjcProperties.ts b/src/native/AppleObjcProperties.ts index cd9c0774d8..266d5ead62 100644 --- a/src/native/AppleObjcProperties.ts +++ b/src/native/AppleObjcProperties.ts @@ -19,8 +19,21 @@ export interface ObjcPropertyReaders { export const parsePropertyAttributes = ( name: string, attributes: string, -): ObjcProperty => { - const tokens = attributes.length === 0 ? [] : splitAttributes(attributes); +): ObjcProperty => decodePropertyAttributes(name, attributes).property; + +/** + * Decode one attribute string. A quoted class name that never closes leaves + * the string incomplete: later attributes may be inside the unfinished type, + * so a missing `N` or `R` does not establish the Objective-C defaults. + */ +const decodePropertyAttributes = ( + name: string, + attributes: string, +): { readonly property: ObjcProperty; readonly complete: boolean } => { + const { tokens, complete } = + attributes.length === 0 + ? { tokens: [], complete: true } + : splitAttributes(attributes); const typeAttributes = tokens.filter((token) => token[0] === "T"); const typeAttribute = typeAttributes.length === 1 ? typeAttributes[0] : undefined; @@ -30,7 +43,8 @@ export const parsePropertyAttributes = ( : null; const value = (code: string): string | null => tokens.find((token) => token[0] === code)?.slice(1) ?? null; - return { + const defaultsEstablished = typeEncoding !== null && complete; + const property: ObjcProperty = { name, type_encoding: typeEncoding, attributes: tokens.map((token) => ({ @@ -39,21 +53,27 @@ export const parsePropertyAttributes = ( })), atomicity: tokens.includes("N") ? "nonatomic" - : typeEncoding === null - ? null - : "atomic", + : defaultsEstablished + ? "atomic" + : null, is_readonly: tokens.includes("R") ? true - : typeEncoding === null - ? null - : false, + : defaultsEstablished + ? false + : null, getter: value("G"), setter: value("S"), }; + return { property, complete }; }; -/** Split on commas that are outside a quoted class name in the type encoding. */ -const splitAttributes = (attributes: string): string[] => { +/** + * Split on commas that are outside a quoted class name in the type encoding, + * and report whether every quote closed. + */ +const splitAttributes = ( + attributes: string, +): { readonly tokens: string[]; readonly complete: boolean } => { const tokens: string[] = []; let current = ""; let quoted = false; @@ -65,34 +85,68 @@ const splitAttributes = (attributes: string): string[] => { } else current += character; } tokens.push(current); - return tokens.filter((token) => token.length > 0); + return { + tokens: tokens.filter((token) => token.length > 0), + complete: !quoted, + }; }; -/** Decode `property_list_t { entsize_and_flags, count, property_t[] }`. */ +/** One property-level decode problem, relative to the list's owner. */ +interface PropertyListIssue { + readonly code: "property_decode_failed" | "property_attributes_incomplete"; + readonly message: string; + readonly location: string; +} + +/** + * Decode `property_list_t { entsize_and_flags, count, property_t[] }`. Each + * entry is decoded on its own, so one malformed entry is reported without + * discarding the properties decoded before or after it. + */ export const readObjcPropertyList = ( read: ObjcPropertyReaders, list: bigint, admit: () => boolean, -): ObjcProperty[] => { - if (list === 0n) return []; +): { + readonly properties: ObjcProperty[]; + readonly issues: PropertyListIssue[]; +} => { + const properties: ObjcProperty[] = []; + const issues: PropertyListIssue[] = []; + if (list === 0n) return { properties, issues }; const entrySize = read.u32(list) & ~3; const count = read.u32(list + 4n); if (entrySize !== 16) throw new TypeError( `Unsupported Objective-C property entry size ${entrySize}`, ); - const properties: ObjcProperty[] = []; for (let index = 0; index < count; index++) { if (!admit()) break; const entry = list + 8n + BigInt(index * entrySize); - properties.push( - parsePropertyAttributes( - read.string(read.pointer(entry)), + const location = `0x${entry.toString(16)}`; + try { + const name = read.string(read.pointer(entry)); + if (name.length === 0) throw new TypeError("the property name is empty"); + const decoded = decodePropertyAttributes( + name, read.string(read.pointer(entry + 8n)), - ), - ); + ); + properties.push(decoded.property); + if (!decoded.complete) + issues.push({ + code: "property_attributes_incomplete", + message: `property ${name} has an unterminated quoted class name in its type attribute, so atomicity and unestablished read-only state are unknown`, + location, + }); + } catch (cause: unknown) { + issues.push({ + code: "property_decode_failed", + message: `property ${String(index)} could not be decoded: ${cause instanceof Error ? cause.message : String(cause)}`, + location, + }); + } } - return properties; + return { properties, issues }; }; /** Decode a property list, recording a failure for `owner` instead of throwing. */ @@ -107,7 +161,21 @@ export const readObjcPropertiesOf = ( }, ): ObjcProperty[] => { try { - return readObjcPropertyList(read, list, context.admit); + const { properties, issues } = readObjcPropertyList( + read, + list, + context.admit, + ); + for (const item of issues) + context.issues.push( + issue( + "objc_properties_categories", + item.code, + `${context.owner}: ${item.message}`, + item.location, + ), + ); + return properties; } catch (cause: unknown) { context.issues.push( issue( diff --git a/src/native/AppleSwiftVtables.test.ts b/src/native/AppleSwiftVtables.test.ts index fb16158205..fe79b955e3 100644 --- a/src/native/AppleSwiftVtables.test.ts +++ b/src/native/AppleSwiftVtables.test.ts @@ -60,6 +60,8 @@ describe("Swift vtable coverage facts: unresolved Swift dispatch slots", () => { examined: 0, categoriesExamined: 0, truncated: false, + objcListScanTruncated: false, + objcClassListScanTruncated: false, fixups: { kind: "none", formats: [], @@ -165,6 +167,8 @@ describe("Swift vtable coverage facts: unsupported pointer fixup coverage", () = examined: 0, categoriesExamined: 0, truncated: false, + objcListScanTruncated: false, + objcClassListScanTruncated: false, fixups: { kind: "chained", formats: ["format-99"], diff --git a/src/native/LldbCallTracer.ts b/src/native/LldbCallTracer.ts index 9fa5813b86..126852c7e2 100644 --- a/src/native/LldbCallTracer.ts +++ b/src/native/LldbCallTracer.ts @@ -123,6 +123,7 @@ const tracerOutputSchema = z.discriminatedUnion("status", [ "tracer-error", ]), error: z.string().nullable(), + target_process_created: z.boolean().optional(), }), ]); @@ -163,7 +164,7 @@ export interface NativeCallTracer { /** Remediation for a target LLDB may not debug. */ const ATTACH_REMEDIATION = - "Allow debugging for this user (Developer Tools access: `DevToolsSecurity -enable`, or membership in the _developer group). A hardened-runtime target is debuggable only with the com.apple.security.get-task-allow entitlement; see inspect_signature's debugger-attach facet."; + "Allow debugging for this user (Developer Tools access: `DevToolsSecurity -enable`, or membership in the _developer group). A hardened-runtime target is debuggable only with the com.apple.security.get-task-allow entitlement; inspect_signature reports hardened_runtime and entitlements."; /** Production tracer: `lldb --batch` running the REA LLDB bridge. */ export class LldbCallTracer implements NativeCallTracer { @@ -423,11 +424,12 @@ export class LldbCallTracer implements NativeCallTracer { let targetTermination: TargetTermination = exited.targetLaunched ? await ensureTerminated(paths.pid, exited.targetIdentity, true) : "terminated"; - // These two bridge outcomes are emitted before a target process exists. - // Other missing-PID cases, including a missing result, remain unknown. + // Only an explicit bridge phase fact may prove that no target process was + // created. A missing PID or an integrity-error status alone is ambiguous. if ( output !== undefined && - (output.status === "target-error" || output.status === "launch-error") && + output.status !== "traced" && + output.target_process_created === false && (await readTargetPid(paths.pid)) === undefined ) targetTermination = "terminated"; diff --git a/src/native/NativeMacOSProvider.ts b/src/native/NativeMacOSProvider.ts index f4d06ae840..4fb002a92a 100644 --- a/src/native/NativeMacOSProvider.ts +++ b/src/native/NativeMacOSProvider.ts @@ -7,7 +7,7 @@ import { inspectAppleDispatchMetadata } from "./AppleDispatchMetadata.js"; import { observeNativeUi } from "./NativeUiObservation.js"; import { LldbCallTracer, type NativeCallTracer } from "./LldbCallTracer.js"; import { observeNativeCalls } from "./NativeCallObservation.js"; -import { dirname, isAbsolute, resolve } from "node:path"; +import { basename, dirname, isAbsolute, resolve } from "node:path"; import { open, realpath, stat } from "node:fs/promises"; import { z } from "zod"; @@ -1094,7 +1094,7 @@ const unreadablePlist = ( ): AnalysisError => { const sentence = reason.endsWith(".") ? reason : `${reason}.`; // Name the default that is missing and the selection that still works. - const missingDefault = `The target has no readable Contents/Info.plist at ${path}: ${sentence} Select another plist in the target with the path input (CLI: --relative-path).`; + const missingDefault = `The target has no readable default plist at ${path}: ${sentence} Select another plist in the target with the path input (CLI: --relative-path).`; return requested === undefined ? new AnalysisCapabilityUnavailableError( IDENTITY.id, @@ -1150,6 +1150,15 @@ const requireRegularPlist = async ( } }; +/** The bundle whose Contents/MacOS directory directly holds `path`. */ +const macOSBundleRoot = (path: string): string | undefined => { + const programs = dirname(path); + const contents = dirname(programs); + return basename(programs) === "MacOS" && basename(contents) === "Contents" + ? dirname(contents) + : undefined; +}; + const resolvePlistPath = async ( target: BinaryTarget, requested: string | undefined, @@ -1159,8 +1168,13 @@ const resolvePlistPath = async ( if (requested !== undefined && isAbsolute(requested)) return ok(resolve(requested)); const sourceMetadata = await stat(source); - const root = sourceMetadata.isDirectory() ? source : dirname(target.path); + const root = sourceMetadata.isDirectory() + ? source + : (macOSBundleRoot(target.path) ?? dirname(target.path)); if (requested !== undefined) return ok(resolve(root, requested)); + // A selected plist file is itself the plist to inspect. + if (target.kind === "artifact" && target.format === "plist") + return ok(target.path); // A bundle target records the Info.plist its program file was declared // in: Contents/Info.plist, or the root plist of an iOS-style bundle. return ok(target.bundleInfoPlist ?? resolve(root, "Contents/Info.plist")); diff --git a/src/native/NativeMachoInspection.test.ts b/src/native/NativeMachoInspection.test.ts index 640e540be9..9e9ef8dd33 100644 --- a/src/native/NativeMachoInspection.test.ts +++ b/src/native/NativeMachoInspection.test.ts @@ -53,7 +53,6 @@ const inspect = async (lipo: string, architecture: "x86" | "arm64") => { run: (tool, args) => { commands.push({ tool, args }); if (tool === "lipo") return Promise.resolve(ok(capture(tool, lipo))); - if (tool === "file") return Promise.resolve(ok(capture(tool, "Mach-O"))); return Promise.resolve(err(new AnalysisCancelledError("inspect_native"))); }, invocation: () => { @@ -72,7 +71,7 @@ describe("Mach-O tool architecture selection", () => { _tag: "AnalysisOutputError", reason: "lipo did not list i386; refusing to read another slice", }); - expect(commands.map(({ tool }) => tool)).toEqual(["file", "lipo"]); + expect(commands.map(({ tool }) => tool)).toEqual(["lipo"]); }); it("passes the arm64e slice lipo listed for an arm64 target", async () => { @@ -264,7 +263,7 @@ describe("Mach-O slice command scheduling", () => { let maxInFlight = 0; let launched = 0; const run = async (tool: string, args: readonly string[]) => { - if (tool === "file" || tool === "lipo") return fixtures.run(tool, args); + if (tool === "lipo") return fixtures.run(tool, args); const order = launched++; inFlight += 1; maxInFlight = Math.max(maxInFlight, inFlight); @@ -300,7 +299,7 @@ describe("Mach-O slice command scheduling", () => { if (!result.ok) throw result.error; expect(runner.maxInFlight()).toBe(SLICE_TOOLS.length + 1); expect(result.value.result).toMatchObject({ - provenance: ["file", "lipo", ...SLICE_TOOLS, "vtool"].map((tool) => ({ + provenance: ["lipo", ...SLICE_TOOLS, "vtool"].map((tool) => ({ tool, })), }); diff --git a/src/native/NativeMachoInspection.ts b/src/native/NativeMachoInspection.ts index 8453f38a8d..c6a05ad98f 100644 --- a/src/native/NativeMachoInspection.ts +++ b/src/native/NativeMachoInspection.ts @@ -39,14 +39,17 @@ interface NativeMachoInspectionContext { ) => NativeCommandInvocation; } -const DISCOVERY_COMMANDS = [ - ["file", ["-b"]], - ["lipo", ["-detailed_info"]], -] as const; +const DISCOVERY_COMMANDS = [["lipo", ["-detailed_info"]]] as const; + +// llvm-nm's default dyld-info pass fakes up symbols the symbol table lacks +// from dyld metadata. It takes seconds on large apps (Keynote arm64: 5.8 s +// versus 0.01 s without it), and under -gU it can only add export-trie names. +const SYMBOL_TABLE_EXPORTS = ["nm", ["-gjU", "--no-dyldinfo"]] as const; +const DYLD_INFO_EXPORTS = ["nm", ["-gjU"]] as const; const REQUIRED_SLICE_COMMANDS = [ ["otool", ["-h", "-l"]], - ["nm", ["-gjU"]], + SYMBOL_TABLE_EXPORTS, ["dyld_info", ["-imports"]], ["dyld_info", ["-exports"]], ["dwarfdump", ["--uuid"]], @@ -98,22 +101,42 @@ export const inspectNativeMacho = async ( ], context.signal, ); + // vtool knows no variant subtypes, so no spelling selects an arm64e.x1 slice. + const vtoolSelectsSlice = + selectedArchitecture === null || + !toolArchitectureName("vtool", selectedArchitecture).includes("."); // Slice commands only read the selected artifact, so launch them together. // Every launch settles before return; declared order fixes provenance and // which failure is reported. const [required, optional] = await Promise.all([ Promise.all(REQUIRED_SLICE_COMMANDS.map(runSliceCommand)), - runSliceCommand(OPTIONAL_VTOOL_COMMAND), + vtoolSelectsSlice ? runSliceCommand(OPTIONAL_VTOOL_COMMAND) : null, ]); for (const captured of required) { if (!captured.ok) return err(captured.error); captures.push(captured.value); } const limitations: string[] = []; - if (optional.ok) captures.push(optional.value); + if (optional === null) + limitations.push( + `vtool cannot select the ${selectedArchitecture} slice; build metadata is normalized from otool only.`, + ); + else if (optional.ok) captures.push(optional.value); else if (optional.error._tag === "AnalysisCapabilityUnavailableError") limitations.push(VTOOL_UNAVAILABLE_LIMITATION); else return err(optional.error); + const nmIndex = captures.findIndex(({ tool }) => tool === "nm"); + const dyldExports = captures.filter(({ tool }) => tool === "dyld_info")[1]; + if (nmIndex < 0 || dyldExports === undefined) + throw new TypeError("Missing nm or dyld_info export capture"); + // A stripped symbol table (strip -T) can omit names the export trie keeps. + // Only then does llvm-nm's dyld-info pass add names, and its exact names + // disambiguate dyld_info rows, so rerun it rather than lose them. + if (!namesEveryExport(captures[nmIndex]!.stdout, dyldExports)) { + const withDyldInfo = await runSliceCommand(DYLD_INFO_EXPORTS); + if (!withDyldInfo.ok) return err(withDyldInfo.error); + captures[nmIndex] = withDyldInfo.value; + } const otool = captures.find(({ tool }) => tool === "otool"); if (otool === undefined) throw new TypeError("Missing otool capture"); let load: ReturnType; @@ -247,8 +270,8 @@ const normalizeMacho = ( return inspectMachoSchema.parse({ format: "mach-o", selected_architecture: selectedArchitecture, - endian: parseEndian(byTool("file").stdout), - word_size: parseWordSize(byTool("file").stdout), + endian: load.endian, + word_size: load.wordSize, file_type: load.fileType, flags: load.flags, uuid: uuid.value, @@ -296,6 +319,34 @@ const captureLookup = return capture; }; +/** + * Whether every dyld_info export row names a symbol-table export. A row this + * parser cannot read counts as uncovered, so an unfamiliar format falls back. + */ +const namesEveryExport = ( + nmOutput: string, + dyldExports: NativeCommandCapture, +): boolean => { + const names = new Set(parseNmExports(nmOutput).map(({ name }) => name)); + return sliceOutput(dyldExports) + .split(/\r?\n/u) + .every((rawLine, index) => { + const line = rawLine.trim(); + if ( + line.length === 0 || + (index === 0 && line.endsWith(":")) || + /^-?exports:$/u.test(line) || + /^offset\s+symbol$/u.test(line) + ) + return true; + const rows = parseDyldSymbols(line, "exports", null, names); + // The parser withholds a row only when nm names both of its readings. + if (rows.length === 0) + return names.has(/^0x[\da-f]+\s+(.+)$/iu.exec(line)?.[1] ?? ""); + return rows.length === 1 && names.has(rows[0]!.name); + }); +}; + const parseNmExports = (output: string) => output .split(/\r?\n/u) @@ -308,16 +359,6 @@ const parseNmExports = (output: string) => source: "nm", })); -const parseEndian = (output: string): "little" | "big" | null => - /little-endian/iu.test(output) - ? "little" - : /big-endian/iu.test(output) - ? "big" - : null; - -const parseWordSize = (output: string): 32 | 64 | null => - /64-bit/iu.test(output) ? 64 : /32-bit/iu.test(output) ? 32 : null; - const covered = ( items: readonly Value[], exhaustive: boolean, @@ -386,16 +427,21 @@ export const architectureLocations = ( return locations; }; +/** + * lipo appends the pointer-auth ABI version to arm64e (`arm64e.v1`). Only + * dyld_info accepts that spelling; the other Apple tools select the slice by + * its CPU-family name. Other variants are distinct subtypes (`arm64e.x1`) + * that the tools address only by their exact name. + */ +const toolArchitectureName = (tool: string, architecture: string): string => + tool === "dyld_info" ? architecture : architecture.replace(/\.v\d+$/u, ""); + const selectedArchitectureArguments = ( tool: string, architecture: string | null, ): string[] => { if (architecture === null) return []; - const baseArchitecture = architecture.split(".")[0] ?? architecture; - // dyld_info distinguishes arm64e ABI variants; the other Apple tools use - // the base CPU-family spelling for the same slice. - const toolArchitecture = - tool === "dyld_info" ? architecture : baseArchitecture; + const toolArchitecture = toolArchitectureName(tool, architecture); return tool === "dwarfdump" ? [`--arch=${toolArchitecture}`] : ["-arch", toolArchitecture]; diff --git a/src/native/parsers/demangle.test.ts b/src/native/parsers/demangle.test.ts new file mode 100644 index 0000000000..dd17e34aad --- /dev/null +++ b/src/native/parsers/demangle.test.ts @@ -0,0 +1,92 @@ +import { describe, expect, it } from "vitest"; + +import { parseDemangledSymbols } from "./demangle.js"; +import { swiftSymbolsSchema } from "../../contracts/native/nativeToolContracts.js"; + +describe("parseDemangledSymbols", () => { + it("returns an empty array when given an empty input list", () => { + expect(swiftSymbolsSchema.safeParse([]).success).toBe(true); + expect(parseDemangledSymbols([], "")).toEqual([]); + expect(parseDemangledSymbols([], "\n")).toEqual([]); + }); + + it("rejects demangler output when the input batch is empty", () => { + expect(() => parseDemangledSymbols([], "unexpected\n")).toThrow( + "swift-demangle emitted output for an empty input", + ); + }); + + it("parses demangled symbols successfully", () => { + const inputs = ["$s4main4testyyF"]; + const output = "main.test() -> ()\n"; + expect(parseDemangledSymbols(inputs, output)).toEqual([ + { + input: "$s4main4testyyF", + output: "main.test() -> ()", + status: "demangled", + }, + ]); + }); + + it("identifies unchanged non-Swift symbols", () => { + const inputs = ["_printf"]; + const output = "_printf\n"; + expect(parseDemangledSymbols(inputs, output)).toEqual([ + { + input: "_printf", + output: "_printf", + status: "unchanged", + }, + ]); + }); + + it("identifies invalid Swift symbols for every demangler mangling prefix", () => { + const inputs = [ + "_T0Bogus", + "$SBogus", + "_$SBogus", + "$sBogus", + "_$sBogus", + "@__swiftmacro_Bogus", + "$eBogus", + "_$eBogus", + "_Tprintf", + ]; + expect( + parseDemangledSymbols(inputs, `${inputs.join("\n")}\n`).map( + ({ status }) => status, + ), + ).toEqual([...Array(8).fill("invalid"), "unchanged"]); + }); + + it("parses mixed batches preserving ordering", () => { + const inputs = ["$s4main4testyyF", "_printf", "$sInvalidMangle"]; + const output = "main.test() -> ()\n_printf\n$sInvalidMangle\n"; + expect(parseDemangledSymbols(inputs, output)).toEqual([ + { + input: "$s4main4testyyF", + output: "main.test() -> ()", + status: "demangled", + }, + { + input: "_printf", + output: "_printf", + status: "unchanged", + }, + { + input: "$sInvalidMangle", + output: "$sInvalidMangle", + status: "invalid", + }, + ]); + }); + + it("throws TypeError when output count does not match input count", () => { + expect(() => + parseDemangledSymbols(["sym1", "sym2"], "demangled1\n"), + ).toThrow(TypeError); + expect(() => + parseDemangledSymbols(["sym1"], "demangled1\ndemangled2\n"), + ).toThrow("swift-demangle output count does not match input"); + }); +}); diff --git a/src/native/parsers/demangle.ts b/src/native/parsers/demangle.ts index ba960be567..3c036d37b7 100644 --- a/src/native/parsers/demangle.ts +++ b/src/native/parsers/demangle.ts @@ -1,11 +1,15 @@ import { z } from "zod"; +import { isSwiftMangledSymbolName } from "../../domain/swiftSymbolClassification.js"; + const resultSchema = z.object({ input: z.string(), output: z.string(), status: z.enum(["demangled", "unchanged", "invalid"]), }); +export type DemangledSymbol = z.infer; + /** * Preserve input ordering while parsing `swift-demangle --compact` lines. * The demangler ends each record with LF, so a carriage return belongs to the @@ -14,7 +18,11 @@ const resultSchema = z.object({ export const parseDemangledSymbols = ( inputs: readonly string[], output: string, -) => { +): DemangledSymbol[] => { + if (inputs.length === 0) { + if (output === "" || output === "\n") return []; + throw new TypeError("swift-demangle emitted output for an empty input"); + } const lines = output.replace(/\n$/u, "").split("\n"); if (lines.length !== inputs.length) throw new TypeError("swift-demangle output count does not match input"); @@ -27,7 +35,7 @@ export const parseDemangledSymbols = ( output: value, status: value === input - ? input.startsWith("$s") || input.startsWith("_$s") + ? isSwiftMangledSymbolName(input) ? "invalid" : "unchanged" : "demangled", diff --git a/src/native/parsers/otool.ts b/src/native/parsers/otool.ts index 8f1ed9dda9..10804b126c 100644 --- a/src/native/parsers/otool.ts +++ b/src/native/parsers/otool.ts @@ -134,6 +134,8 @@ export const parseOtoolLoadCommands = (raw: string) => { parseLoadCommand(block, state); validateLoadCommandFraming(header, state.commands); return { + wordSize: WORD_SIZES.get(header.magic ?? "") ?? null, + endian: header.cpuType === null ? null : cpuEndian(header.cpuType), fileType: header.fileType, flags: [...state.flags].sort(), uuid: state.uuid, @@ -145,10 +147,27 @@ export const parseOtoolLoadCommands = (raw: string) => { }; }; +/** otool prints the selected slice's magic in host order, symbolic with -v. */ +const WORD_SIZES: ReadonlyMap = new Map([ + ["0xfeedface", 32], + ["0xfeedfacf", 64], + ["MH_MAGIC", 32], + ["MH_MAGIC_64", 64], +]); + +/** Mach-O byte order by CPU family, ignoring the ABI bits; others stay unknown. */ +const cpuEndian = (cpuType: number): "little" | "big" | null => { + const family = cpuType & 0x00ffffff; + if (family === 7 || family === 12) return "little"; + return family === 18 ? "big" : null; +}; + const parseHeader = ( output: string, ): | { + magic: string | null; + cpuType: number | null; fileType: string | null; flags: string[]; commandCount: number; @@ -175,7 +194,10 @@ const parseHeader = ( : fileTypeIndex; const flagsEnd = flagsIndex >= 0 ? values.length - (columns.length - flagsIndex - 1) : 0; + const magicIndex = columns.indexOf("magic"); return { + magic: magicIndex < 0 ? null : (values[magicIndex] ?? null), + cpuType: headerValue(columns, values, "cputype"), commandCount, commandBytes, fileType: diff --git a/src/native/parsers/plist.ts b/src/native/parsers/plist.ts index b27f67b1ee..a3b6c4fa5e 100644 --- a/src/native/parsers/plist.ts +++ b/src/native/parsers/plist.ts @@ -4,6 +4,7 @@ import { AnalysisOutputError } from "../../domain/analysisErrorCore.js"; import type { JsonValue } from "../../domain/jsonValue.js"; import { projectPlistValue, + unknownDateLimitation, type ProjectedPlistValue, } from "../../domain/apple/plistValue.js"; import { @@ -178,6 +179,9 @@ export const parsePlistXml = ( : [ `${String(projected.unknownRealCount)} non-finite real value(s) are reported as { "$plist_type": "real", "value": null } because the XML decoder does not distinguish NaN from infinity.`, ]), + ...(projected.unknownDateCount === 0 + ? [] + : [unknownDateLimitation(projected.unknownDateCount)]), ...numberLimitations(exactIntegerCount, ambiguousNumberCount), ...prototypeKeyLimitations(omittedPrototypeKeys), ], diff --git a/src/native/pwntools/PwntoolsFailures.ts b/src/native/pwntools/PwntoolsFailures.ts index e36ae95032..17e3a8296e 100644 --- a/src/native/pwntools/PwntoolsFailures.ts +++ b/src/native/pwntools/PwntoolsFailures.ts @@ -265,6 +265,7 @@ export const pwntoolsUnavailable = ( ): ProviderSelectionError => new ProviderSelectionError({ ...(capturedOutput === undefined ? {} : { capturedOutput }), + userMessage: reason, operation: context.operation, reason: "provider_unavailable", requestedProviderId: context.providerId, diff --git a/src/objdump/ObjdumpCommand.ts b/src/objdump/ObjdumpCommand.ts new file mode 100644 index 0000000000..bca39bff46 --- /dev/null +++ b/src/objdump/ObjdumpCommand.ts @@ -0,0 +1,24 @@ +import { resolve } from "node:path"; + +import type { ObjdumpInput } from "../domain/reverseEngineering.js"; + +export const OBJDUMP_PROVIDER_IDENTITY = { + id: "gnu.objdump", + name: "GNU objdump", + version: null, +} as const; + +/** Build GNU objdump argv without invoking a shell or following DWARF links by default. */ +export const objdumpCommand = (input: ObjdumpInput): readonly string[] => { + if (input.operation === "architectures") return ["-i"]; + const argumentsByOperation = { + file_headers: ["-f"], + section_headers: ["-h"], + symbols: ["-t"], + relocations: ["-r"], + dwarf: ["-W", input.follow_debug_links ? "-WK" : "-WN"], + disassemble: ["-d"], + disassemble_all: ["-D"], + } satisfies Record; + return [...argumentsByOperation[input.operation], resolve(input.path)]; +}; diff --git a/src/pcap/PcapCommand.ts b/src/pcap/PcapCommand.ts index 0d28a3d752..c5d61fef1f 100644 --- a/src/pcap/PcapCommand.ts +++ b/src/pcap/PcapCommand.ts @@ -9,6 +9,7 @@ import { } from "../domain/analysisErrorCore.js"; import { OwnedCommandFailure, + ownedCommandResources, runOwnedCommand, } from "../process/OwnedCommand.js"; import type { ProviderProcessSupervisor } from "../process/ProviderProcess.js"; @@ -20,6 +21,26 @@ import { } from "../process/WindowsBatchCommand.js"; import { pcapToolUnavailable } from "./PcapConfiguration.js"; +export type PcapCommandLauncher = NonNullable< + Parameters[2] +>["launcher"]; + +export interface PcapCommandCleanup { + readonly reason: string; + readonly resources: readonly string[]; + readonly capturedOutput: { + readonly stdout: string; + readonly stderr: string; + readonly truncated: boolean; + }; +} + +export interface PcapCommandOutput { + readonly stdout: string; + readonly stderr: string; + readonly cleanup?: PcapCommandCleanup; +} + export interface PcapCommandLimits { readonly timeoutMs: number; readonly maxOutputBytes: number; @@ -38,11 +59,14 @@ export const runPcapCommand = async (options: { readonly limiter?: { command: string; arguments: readonly string[] }; readonly memoryLimitBytes?: number; }; + readonly launcher?: PcapCommandLauncher; readonly retain: ( owner: ProviderProcessSupervisor, resources: readonly string[], + reason: string, + capturedOutput: PcapCommandCleanup["capturedOutput"], ) => void; -}): Promise<{ stdout: string; stderr: string }> => { +}): Promise => { const { operation, command, limits, signal } = options; if (signal?.aborted) throw new AnalysisCancelledError(operation, { cause: signal.reason }); @@ -52,6 +76,7 @@ export const runPcapCommand = async (options: { LC_ALL: "C", LANG: "C", }; + let cleanup: PcapCommandCleanup | undefined; try { const batch = process.platform === "win32" && isWindowsBatchFile(command); const invocation = batch @@ -87,26 +112,64 @@ export const runPcapCommand = async (options: { : { memoryLimitBytes: options.nativeBound.memoryLimitBytes }), }, { timeoutMs: limits.timeoutMs, diagnosticBytes: limits.maxOutputBytes }, - signal === undefined ? {} : { signal }, + { + ...(signal === undefined ? {} : { signal }), + ...(options.launcher === undefined + ? {} + : { launcher: options.launcher }), + retainCleanup: (owner, stopped) => { + const resources = + owner.launch.ownership === undefined + ? ["owned-process"] + : ownedCommandResources(owner.launch.ownership); + const snapshot = owner.snapshot(); + const capturedOutput = { + stdout: snapshot.stdout.text, + stderr: snapshot.stderr.text, + truncated: + snapshot.diagnosticTruncated === true || + snapshot.outputIncomplete === true, + }; + options.retain(owner, resources, stopped.reason, capturedOutput); + cleanup = { reason: stopped.reason, resources, capturedOutput }; + }, + }, ); - return { stdout: output.stdout.text, stderr: output.stderr.text }; + return { + stdout: output.stdout.text, + stderr: output.stderr.text, + ...(cleanup === undefined ? {} : { cleanup }), + }; } catch (cause) { if (cause instanceof AnalysisError) throw cause; if ( cause instanceof OwnedCommandFailure && cause.cleanupOwner !== undefined ) - options.retain(cause.cleanupOwner, cause.resources); + options.retain( + cause.cleanupOwner, + cause.resources, + cause.cleanupFailure ?? cause.message, + cause.snapshot === null + ? { stdout: "", stderr: "", truncated: true } + : { + stdout: cause.snapshot.stdout.text, + stderr: cause.snapshot.stderr.text, + truncated: + cause.snapshot.diagnosticTruncated === true || + cause.snapshot.outputIncomplete === true, + }, + ); const details = { cause, ...(cause instanceof OwnedCommandFailure && cause.snapshot !== null ? { capturedOutput: { - stdout: "", + stdout: cause.snapshot.stdout.text, stderr: cause.snapshot.stderr.text, truncated: - cause.snapshot.stdout.observedBytes > 0 || - (cause.snapshot.diagnosticTruncated ?? false), + cause.snapshot.diagnosticTruncated === true || + cause.snapshot.outputIncomplete === true, exit_code: cause.snapshot.exitCode ?? null, signal: cause.snapshot.signal ?? null, }, diff --git a/src/pcap/PcapProvider.ts b/src/pcap/PcapProvider.ts index be2c12ed2b..5f1e4e3225 100644 --- a/src/pcap/PcapProvider.ts +++ b/src/pcap/PcapProvider.ts @@ -17,6 +17,7 @@ import { AnalysisProtocolError, } from "../domain/analysisErrorCore.js"; import { AnalysisError } from "../domain/analysisErrorBase.js"; +import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; import { analysisErrorWithCleanupFailure } from "../domain/analysisErrorCleanup.js"; import { err, ok, type Result } from "../domain/result.js"; import type { ProviderProcessSupervisor } from "../process/ProviderProcess.js"; @@ -33,8 +34,14 @@ import { parseTsharkJsonOutput, TSHARK_JSON_MAX_BYTES, } from "./PcapOutput.js"; -import { runPcapCommand, type PcapCommandLimits } from "./PcapCommand.js"; +import { + runPcapCommand, + type PcapCommandCleanup, + type PcapCommandLimits, + type PcapCommandLauncher, +} from "./PcapCommand.js"; import { ProviderCleanupError } from "../domain/providerCleanupError.js"; +import { jsonValueSchema } from "../domain/jsonValue.js"; import { createPcapEvidence } from "../domain/pcap/pcapEvidence.js"; import { preparePcapNativeBound } from "./PcapNativeBound.js"; import { PcapSnapshot } from "./PcapSnapshot.js"; @@ -62,8 +69,62 @@ const FIELDS = [ "http.request.full_uri", ]; +const pcapPartialEvidence = ( + request: PcapRequest, + provider: + | typeof PCAP_PROVIDER_IDENTITY + | { + readonly id: string; + readonly name: string; + readonly version: string | null; + }, + path: string, + identity: { readonly bytes: number; readonly sha256: string }, + result: unknown, + limitations: readonly string[], +) => + createPcapEvidence(request, { + subject: { path, format: "file", sha256: identity.sha256 }, + provider, + result: jsonValueSchema.parse(result), + rawResult: null, + limitations, + locations: [{ kind: "artifact-path", path }], + }); + +const pcapCommandCleanupError = ( + operation: PcapOperation, + snapshotPath: string | undefined, + cleanup: PcapCommandCleanup, + capturedOutput: { readonly stdout: string; readonly stderr: string }, + partialObservation?: ReturnType, +) => + new ProviderCleanupError( + "pcap", + [ + ...new Set([ + ...(snapshotPath === undefined ? [] : [snapshotPath]), + ...cleanup.resources, + ]), + ], + { + reason: cleanup.reason, + previous_error: null, + captured_output: { + stdout: capturedOutput.stdout, + stderr: capturedOutput.stderr, + truncated: cleanup.capturedOutput.truncated, + }, + }, + { + operation, + ...(partialObservation === undefined ? {} : { partialObservation }), + }, + ); + export interface PcapProviderOptions { readonly environment?: Readonly>; + readonly launcher?: PcapCommandLauncher; } /** Inspect saved captures through bounded, owned tool processes and one stable input copy. */ @@ -79,17 +140,24 @@ export class PcapProvider { }, ); private readonly environment: Readonly>; + private readonly launcher: PcapCommandLauncher | undefined; private readonly snapshots = new Set(); private readonly commands = new Map< ProviderProcessSupervisor, - { snapshot: PcapSnapshot | undefined; resources: readonly string[] } + { + snapshot: PcapSnapshot | undefined; + resources: readonly string[]; + reason: string; + capturedOutput: PcapCommandCleanup["capturedOutput"]; + } >(); constructor(options: PcapProviderOptions = {}) { this.environment = options.environment ?? process.env; + this.launcher = options.launcher; } - private run( + private async run( operation: PcapOperation, command: string, args: readonly string[], @@ -98,7 +166,8 @@ export class PcapProvider { snapshot?: PcapSnapshot, inputPath?: string, ) { - return runPcapCommand({ + let cleanup: PcapCommandCleanup | undefined; + const output = await runPcapCommand({ operation, command, arguments: args, @@ -115,10 +184,33 @@ export class PcapProvider { ...(inputPath === undefined || this.nativeBound === undefined ? {} : { nativeBound: this.nativeBound }), - retain: (owner, resources) => { - this.commands.set(owner, { snapshot, resources }); + ...(this.launcher === undefined ? {} : { launcher: this.launcher }), + retain: (owner, resources, reason, capturedOutput) => { + this.commands.set(owner, { + snapshot, + resources, + reason, + capturedOutput, + }); + cleanup = { reason, resources, capturedOutput }; }, }); + if (inputPath === undefined && cleanup !== undefined) + throw new ProviderCleanupError( + "pcap", + cleanup.resources, + { + reason: cleanup.reason, + previous_error: null, + captured_output: { + stdout: output.stdout, + stderr: output.stderr, + truncated: cleanup.capturedOutput.truncated, + }, + }, + { operation }, + ); + return output; } private nativeBound: Parameters[0]["nativeBound"]; @@ -203,6 +295,7 @@ export class PcapProvider { diagnostics: client, }; } catch (cause) { + if (cause instanceof ProviderCleanupError) throw cause; if (signal?.aborted || cause instanceof AnalysisCancelledError) throw new AnalysisCancelledError("inspect_pcap_client", { cause }); if (cause instanceof PcapConfigurationFailure) @@ -324,6 +417,9 @@ export class PcapProvider { request.input.path, identity, signal, + request, + provider, + client, ); const data = pcapResultSchemas.inspect_pcap_capture.parse({ client, @@ -363,7 +459,13 @@ export class PcapProvider { partialObservation: createPcapEvidence(request, result.value), }), ); - if (snapshot !== undefined) { + const pendingCommand = + snapshot === undefined + ? undefined + : [...this.commands.values()].find( + (command) => command.snapshot === snapshot, + ); + if (snapshot !== undefined && pendingCommand === undefined) { const cleanup = await this.discard(snapshot); if (signal?.aborted && result.ok) result = err( @@ -387,6 +489,43 @@ export class PcapProvider { ), ); } + if ( + snapshot !== undefined && + pendingCommand !== undefined && + (result.ok || !(result.error instanceof ProviderCleanupError)) + ) { + const partialObservation = result.ok + ? createPcapEvidence(request, result.value) + : result.error.partialObservation; + const capturedOutput = result.ok + ? undefined + : result.error.capturedOutput; + result = err( + new ProviderCleanupError( + "pcap", + [...new Set([snapshot.path, ...pendingCommand.resources])], + { + reason: pendingCommand.reason, + previous_error: result.ok + ? null + : projectAnalysisError(result.error), + ...(capturedOutput === undefined && + pendingCommand.capturedOutput.stdout === "" && + pendingCommand.capturedOutput.stderr === "" + ? {} + : { + captured_output: { + ...(capturedOutput ?? pendingCommand.capturedOutput), + }, + }), + }, + { + operation: request.operation, + ...(partialObservation === undefined ? {} : { partialObservation }), + }, + ), + ); + } return result; } @@ -396,6 +535,13 @@ export class PcapProvider { path: string, identity: { bytes: number; sha256: string }, signal: AbortSignal | undefined, + request: PcapRequest, + provider: { + readonly id: string; + readonly name: string; + readonly version: string | null; + }, + client: unknown, ) { const info = await this.run( "inspect_pcap_capture", @@ -411,6 +557,34 @@ export class PcapProvider { throw new AnalysisProtocolError( "capinfos SHA256 disagrees with the owned capture snapshot", ); + if (info.cleanup !== undefined) + throw pcapCommandCleanupError( + request.operation, + snapshot.path, + info.cleanup, + info, + pcapPartialEvidence( + request, + provider, + path, + identity, + { + client, + target: { path, ...identity }, + stage: "capinfos-complete-tshark-not-run", + file_type: metadata.fileType, + encapsulation: metadata.encapsulation, + packet_count: metadata.packetCount, + capture_start: metadata.captureStart, + capture_end: metadata.captureEnd, + capture_duration_seconds: metadata.captureDurationSeconds, + snapshot_length: metadata.snapshotLength, + }, + [ + "capinfos output was parsed from a complete bounded stream, but process-tree cleanup remains unconfirmed; tshark was not run", + ], + ), + ); const dissected = await this.run( "inspect_pcap_capture", selection.tshark.command, @@ -435,7 +609,7 @@ export class PcapProvider { throw new AnalysisProtocolError( `capinfos counted ${String(metadata.packetCount)} packets but tshark returned ${String(fields.packetCount)}`, ); - return { + const captured = { result: { target: { path, ...identity }, file_type: metadata.fileType, @@ -487,5 +661,26 @@ export class PcapProvider { : []), ], }; + if (dissected.cleanup !== undefined) { + const full = pcapResultSchemas.inspect_pcap_capture.parse({ + client, + ...captured.result, + }); + throw pcapCommandCleanupError( + request.operation, + snapshot.path, + dissected.cleanup, + dissected, + pcapPartialEvidence( + request, + provider, + path, + identity, + full, + captured.limitations, + ), + ); + } + return captured; } } diff --git a/src/process/ArtifactAnalysisResources.test.ts b/src/process/ArtifactAnalysisResources.test.ts new file mode 100644 index 0000000000..84ce0e6d7d --- /dev/null +++ b/src/process/ArtifactAnalysisResources.test.ts @@ -0,0 +1,85 @@ +import { EventEmitter } from "node:events"; +import { PassThrough } from "node:stream"; +import { expect, it } from "vitest"; +import { readStableArtifact } from "../artifacts/readStableArtifact.js"; +import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; +import { ProviderAdapterError } from "../domain/providerAdapterError.js"; +import { err } from "../domain/result.js"; +import { ArtifactAnalysisResources } from "./ArtifactAnalysisResources.js"; +import { readLinuxResourceLimits } from "./LinuxResourceLimits.js"; +import { ProviderProcessSupervisor } from "./ProviderProcess.js"; + +it("projects the specific reason and member evidence for retained cleanup", async () => { + const producer = Object.assign(new EventEmitter(), { + stdout: new PassThrough(), + stderr: new PassThrough(), + exitCode: null, + signalCode: null, + kill: () => true, + }); + const owner = new ProviderProcessSupervisor({ + process: producer, + ownsProcessLifetime: true, + ownership: { + runId: "rea-tshark-run-1", + leaderPid: 4242, + processGroupId: 4242, + }, + cleanup: async () => ({ + cleaned: false as const, + reason: "process ownership token could not be read for 1 live process", + failures: [ + { + pid: 101, + reason: "environment-unreadable" as const, + diagnostic: "sysctl_errno_22", + }, + ], + unverified: [ + { pid: 900, diagnostic: "platform_binary_environment_withheld" }, + ], + }), + }); + const resources = new ArtifactAnalysisResources( + "fixture-provider", + "fixture-operation", + readStableArtifact, + readLinuxResourceLimits, + { message: "busy", details: {} }, + ); + resources.retain({ path: "runtime-root", close: async () => {} }, owner); + + const result = await resources.finish( + err(new ProviderAdapterError("fixture-provider", "fixture-operation")), + () => { + throw new Error("unexpected success observation"); + }, + ); + owner.dispose(); + + if (result.ok) throw new Error("expected cleanup failure"); + expect(projectAnalysisError(result.error)).toMatchObject({ + code: "cleanup_incomplete", + details: { + resources: [ + "rea-tshark-run-1", + "runtime-root", + "pid:4242", + "process-group:4242", + ], + diagnostics: { + reason: "process ownership token could not be read for 1 live process", + failures: [ + { + pid: 101, + reason: "environment-unreadable", + diagnostic: "sysctl_errno_22", + }, + ], + unverified: [ + { pid: 900, diagnostic: "platform_binary_environment_withheld" }, + ], + }, + }, + }); +}); diff --git a/src/process/ArtifactAnalysisResources.ts b/src/process/ArtifactAnalysisResources.ts index ae3d9f8034..ff7b630488 100644 --- a/src/process/ArtifactAnalysisResources.ts +++ b/src/process/ArtifactAnalysisResources.ts @@ -24,7 +24,7 @@ import { } from "./LinuxResourceLimits.js"; import type { PrivateRuntimeRoot } from "./PrivateRuntimeRoot.js"; import type { ProviderProcessSupervisor } from "./ProviderProcess.js"; -import { OwnedCommandFailure } from "./OwnedCommand.js"; +import { OwnedCommandFailure, ownedCommandResources } from "./OwnedCommand.js"; export type ArtifactAnalysisRuntime = Pick< PrivateRuntimeRoot, @@ -129,22 +129,67 @@ export class ArtifactAnalysisResources { return [...this.#pending.keys()].map((root) => root.path); } + #resourcesFor(root: ArtifactAnalysisRuntime): readonly string[] { + const ownership = this.#pending.get(root)?.owner?.launch.ownership; + return ownership === undefined + ? [root.path] + : ownedCommandResources(ownership, root.path); + } + + get #pendingResources(): readonly string[] { + return [...this.#pending.keys()].flatMap((root) => + this.#resourcesFor(root), + ); + } + async release(): Promise { for (const [root, retained] of this.#pending) { try { const { owner } = retained; if (owner !== undefined) { const stopped = await owner.stop(); - if (stopped.status === "incomplete") throw new Error(stopped.reason); + if (stopped.status === "incomplete") { + const cleanup = stopped.cleanupResult; + throw new ProviderCleanupError( + this.providerId, + this.#resourcesFor(root), + { + reason: stopped.reason, + ...(cleanup?.failures === undefined + ? {} + : { + failures: cleanup.failures.map((failure) => ({ + ...failure, + })), + }), + ...(cleanup?.unverified === undefined + ? {} + : { + unverified: cleanup.unverified.map((process) => ({ + ...process, + })), + }), + }, + { operation: retained.operation }, + ); + } this.#pending.set(root, { ...retained, owner: undefined }); } await root.close(); this.#pending.delete(root); } catch (cause: unknown) { + const diagnostics = + cause instanceof ProviderCleanupError ? cause.diagnostics : undefined; throw new ProviderCleanupError( this.providerId, - [root.path], - { reason: String(cause) }, + this.#resourcesFor(root), + { + ...diagnostics, + reason: + cause instanceof ProviderCleanupError + ? String(diagnostics?.reason ?? cause.message) + : String(cause), + }, { operation: retained.operation, cause }, ); } @@ -236,12 +281,18 @@ export class ArtifactAnalysisResources { const partialObservation = result.ok ? observation(result.value) : result.error.partialObservation; + const cleanupDiagnostics = + cause instanceof ProviderCleanupError ? cause.diagnostics : undefined; return err( new ProviderCleanupError( this.providerId, - this.pendingPaths, + this.#pendingResources, { - reason: String(cause), + ...cleanupDiagnostics, + reason: + cause instanceof ProviderCleanupError + ? String(cleanupDiagnostics?.reason ?? cause.message) + : String(cause), previous_error: result.ok ? null : projectAnalysisError(result.error), diff --git a/src/process/CurrentEntrypointPath.ts b/src/process/CurrentEntrypointPath.ts new file mode 100644 index 0000000000..8005e1aab0 --- /dev/null +++ b/src/process/CurrentEntrypointPath.ts @@ -0,0 +1,11 @@ +import { resolve } from "node:path"; + +/** Absolute argv entrypoint, or undefined when the process did not supply one. */ +export const normalizeEntrypointPath = ( + entrypoint: string | undefined, +): string | undefined => + entrypoint === undefined || entrypoint.length === 0 + ? undefined + : resolve(entrypoint); + +export const CURRENT_ENTRYPOINT_PATH = normalizeEntrypointPath(process.argv[1]); diff --git a/src/process/DetachedPgpSignature.test.ts b/src/process/DetachedPgpSignature.test.ts new file mode 100644 index 0000000000..ee240089fa --- /dev/null +++ b/src/process/DetachedPgpSignature.test.ts @@ -0,0 +1,128 @@ +import { beforeAll, describe, expect, it } from "vitest"; +import { createMessage, generateKey, readPrivateKey, sign } from "openpgp"; + +import { verifyDetachedPgpSignature } from "./DetachedPgpSignature.js"; + +const bytes = Buffer.from("binary package fixture\0\xff", "latin1"); +let publicKey: string; +let fingerprint: string; +let signatures: Uint8Array[]; +let signingFingerprints: string[]; + +beforeAll(async () => { + const generated = await generateKey({ + type: "ecc", + curve: "curve25519Legacy", + userIDs: [{ name: "Fixture publisher" }], + subkeys: [{ sign: true }, { sign: true }], + format: "armored", + }); + const privateKey = await readPrivateKey({ armoredKey: generated.privateKey }); + publicKey = generated.publicKey; + fingerprint = privateKey.getFingerprint().toUpperCase(); + signingFingerprints = privateKey.subkeys.map((subkey) => + subkey.getFingerprint().toUpperCase(), + ); + signatures = []; + for (const subkey of privateKey.subkeys) + signatures.push( + await sign({ + message: await createMessage({ binary: bytes }), + signingKeys: privateKey, + signingKeyIDs: [subkey.getKeyID()], + detached: true, + format: "binary", + }), + ); +}); + +describe("detached binary PGP publisher verification", () => { + it("accepts independently selected signing subkeys through one pinned primary", async () => { + for (const [index, signature] of signatures.entries()) + expect( + await verifyDetachedPgpSignature( + bytes, + signature, + publicKey, + fingerprint, + ), + ).toEqual({ + ok: true, + value: { + primaryFingerprint: fingerprint, + signingFingerprint: signingFingerprints[index], + }, + }); + }); + + it("rejects a valid signature belonging to another primary", async () => { + expect( + await verifyDetachedPgpSignature( + bytes, + signatures[0]!, + publicKey, + "0".repeat(40), + ), + ).toMatchObject({ ok: false, error: { reason: "wrong_publisher" } }); + }); + + it("rejects tampered payload bytes", async () => { + const tampered = Buffer.from(bytes); + tampered[0] = (tampered[0] ?? 0) ^ 1; + expect( + await verifyDetachedPgpSignature( + tampered, + signatures[0]!, + publicKey, + fingerprint, + ), + ).toMatchObject({ ok: false, error: { reason: "invalid_signature" } }); + }); + + it("rejects malformed signatures and public keys", async () => { + expect( + await verifyDetachedPgpSignature( + bytes, + new Uint8Array([1, 2, 3]), + publicKey, + fingerprint, + ), + ).toMatchObject({ ok: false, error: { reason: "invalid_signature" } }); + expect( + await verifyDetachedPgpSignature( + bytes, + signatures[0]!, + "malformed", + fingerprint, + ), + ).toMatchObject({ ok: false, error: { reason: "invalid_signature" } }); + }); + + it("preserves cancellation", async () => { + expect( + await verifyDetachedPgpSignature( + bytes, + signatures[0]!, + publicKey, + fingerprint, + AbortSignal.abort(), + ), + ).toMatchObject({ ok: false, error: { reason: "cancelled" } }); + }); + + it("preserves cancellation while loading the publisher key, before classifying its identity", async () => { + const controller = new AbortController(); + const verification = verifyDetachedPgpSignature( + bytes, + signatures[0]!, + publicKey, + "0".repeat(40), + controller.signal, + ); + queueMicrotask(() => controller.abort()); + expect(await verification).toMatchObject({ + ok: false, + error: { reason: "cancelled" }, + }); + }); +}); diff --git a/src/process/DetachedPgpSignature.ts b/src/process/DetachedPgpSignature.ts new file mode 100644 index 0000000000..8d1a215cd1 --- /dev/null +++ b/src/process/DetachedPgpSignature.ts @@ -0,0 +1,101 @@ +import { err, ok, type Result } from "../domain/result.js"; + +export class DetachedPgpSignatureError extends Error { + readonly _tag = "DetachedPgpSignatureError"; + constructor( + readonly reason: "invalid_signature" | "wrong_publisher" | "cancelled", + message: string, + options?: ErrorOptions, + ) { + super(message, options); + this.name = this._tag; + } +} + +/** Verify the complete binary payload and its signing subkey's primary identity. */ +export const verifyDetachedPgpSignature = async ( + bytes: Uint8Array, + detachedSignature: Uint8Array, + armoredPublicKey: string, + primaryFingerprint: string, + signal?: AbortSignal, +): Promise< + Result< + { + readonly primaryFingerprint: string; + readonly signingFingerprint: string; + }, + DetachedPgpSignatureError + > +> => { + const cancelled = () => + err( + new DetachedPgpSignatureError( + "cancelled", + "Package verification was cancelled.", + ), + ); + const isAborted = () => signal?.aborted === true; + if (isAborted()) return cancelled(); + try { + // Keep the cryptographic implementation out of unrelated CLI cold starts. + const pgp = await import("openpgp"); + const key = await pgp.readKey({ armoredKey: armoredPublicKey }); + if (isAborted()) return cancelled(); + const fingerprint = key.getFingerprint().toUpperCase(); + if (fingerprint !== primaryFingerprint) + return err( + new DetachedPgpSignatureError( + "wrong_publisher", + "The package signing key does not belong to the pinned publisher.", + ), + ); + const signature = await pgp.readSignature({ + binarySignature: detachedSignature, + }); + if (isAborted()) return cancelled(); + if ( + signature.packets.length !== 1 || + signature.packets[0]?.signatureType !== pgp.enums.signature.binary + ) + return err( + new DetachedPgpSignatureError( + "invalid_signature", + "Expected one detached binary package signature.", + ), + ); + const verified = await pgp.verify({ + message: await pgp.createMessage({ binary: bytes }), + signature, + verificationKeys: key, + format: "binary", + }); + const signed = verified.signatures[0]; + if (signed === undefined || verified.signatures.length !== 1) + return err( + new DetachedPgpSignatureError( + "invalid_signature", + "The package has no unique verified signature.", + ), + ); + await signed.verified; + const signingKey = await key.getSigningKey( + signed.keyID, + signature.packets[0].created, + ); + if (isAborted()) return cancelled(); + return ok({ + primaryFingerprint: fingerprint, + signingFingerprint: signingKey.getFingerprint().toUpperCase(), + }); + } catch (cause: unknown) { + if (isAborted()) return cancelled(); + return err( + new DetachedPgpSignatureError( + "invalid_signature", + "The package signature could not be verified.", + { cause }, + ), + ); + } +}; diff --git a/src/process/ExecFileBoundedOutput.test.ts b/src/process/ExecFileBoundedOutput.test.ts new file mode 100644 index 0000000000..c055bc0028 --- /dev/null +++ b/src/process/ExecFileBoundedOutput.test.ts @@ -0,0 +1,50 @@ +import { describe, expect, it } from "vitest"; + +import { execFileBoundedOutput } from "./ExecFileBoundedOutput.js"; + +describe("execFileBoundedOutput", () => { + it("lets a command finish after its output exceeds the capture limit", async () => { + const result = await execFileBoundedOutput( + process.execPath, + [ + "-e", + "process.stdout.write('x'.repeat(70000)); process.stderr.write('failure detail');", + ], + { env: process.env, limit: 64 * 1024 }, + ); + expect(result.exitCode).toBe(0); + expect(Buffer.byteLength(result.stdout)).toBeLessThan(64 * 1024 + 50); + expect(result.stdout).toContain("[output omitted]"); + expect(result.stderr).toBe("failure detail"); + expect(result.outputTruncated).toBe(true); + }); + + it("retains a failing command's status and stderr after verbose stdout", async () => { + const result = await execFileBoundedOutput( + process.execPath, + [ + "-e", + "process.stdout.write('x'.repeat(70000)); process.stderr.write('package conflict'); process.exitCode = 1;", + ], + { env: process.env, limit: 64 * 1024 }, + ); + expect(result.exitCode).toBe(1); + expect(result.stderr).toBe("package conflict"); + expect(result.outputTruncated).toBe(true); + }); + + it("retains a final error after a long prefix in the same stream", async () => { + const result = await execFileBoundedOutput( + process.execPath, + [ + "-e", + "process.stderr.write('warning '.repeat(10000)); process.stderr.write('error: final conflict'); process.exitCode = 1;", + ], + { env: process.env, limit: 64 * 1024 }, + ); + expect(result.exitCode).toBe(1); + expect(result.stderr).toContain("[output omitted]"); + expect(result.stderr).toContain("error: final conflict"); + expect(result.outputTruncated).toBe(true); + }); +}); diff --git a/src/process/ExecFileBoundedOutput.ts b/src/process/ExecFileBoundedOutput.ts new file mode 100644 index 0000000000..fad51e2bce --- /dev/null +++ b/src/process/ExecFileBoundedOutput.ts @@ -0,0 +1,73 @@ +import { spawn } from "node:child_process"; + +export interface BoundedCommandOutput { + readonly exitCode: string | number | null; + readonly stdout: string; + readonly stderr: string; + readonly outputTruncated: boolean; +} + +/** Drain both streams while retaining bounded head and tail evidence. */ +export const execFileBoundedOutput = ( + executable: string, + args: readonly string[], + options: { + readonly env: Readonly; + readonly limit: number; + }, +): Promise => + new Promise((resolve) => { + const child = spawn(executable, [...args], { + env: options.env, + stdio: ["ignore", "pipe", "pipe"], + }); + const headLimit = Math.ceil(options.limit / 2); + const tailLimit = options.limit - headLimit; + const stdout = { head: Buffer.alloc(0), tail: Buffer.alloc(0), total: 0 }; + const stderr = { head: Buffer.alloc(0), tail: Buffer.alloc(0), total: 0 }; + let outputTruncated = false; + const capture = + (stream: typeof stdout) => + (chunk: Buffer): void => { + stream.total += chunk.length; + const headCount = Math.min( + chunk.length, + Math.max(0, headLimit - stream.head.length), + ); + if (headCount > 0) + stream.head = Buffer.concat([ + stream.head, + chunk.subarray(0, headCount), + ]); + const rest = chunk.subarray(headCount); + if (rest.length > 0 && tailLimit > 0) + stream.tail = Buffer.concat([stream.tail, rest]).subarray(-tailLimit); + if (stream.total > options.limit) outputTruncated = true; + }; + child.stdout.on("data", capture(stdout)); + child.stderr.on("data", capture(stderr)); + let spawnError: NodeJS.ErrnoException | undefined; + child.on("error", (error: NodeJS.ErrnoException) => { + spawnError = error; + }); + child.on("close", (code, signal) => { + resolve({ + exitCode: spawnError?.code ?? code ?? signal, + stdout: retainedOutput(stdout, options.limit), + stderr: retainedOutput(stderr, options.limit), + outputTruncated, + }); + }); + }); + +const retainedOutput = ( + stream: { + readonly head: Buffer; + readonly tail: Buffer; + readonly total: number; + }, + limit: number, +): string => + stream.total > limit + ? `${stream.head.toString("utf8")} … [output omitted] … ${stream.tail.toString("utf8")}` + : Buffer.concat([stream.head, stream.tail]).toString("utf8"); diff --git a/src/process/ExecFileOutput.test.ts b/src/process/ExecFileOutput.test.ts index 78d87800a3..7d7905cc1f 100644 --- a/src/process/ExecFileOutput.test.ts +++ b/src/process/ExecFileOutput.test.ts @@ -1,13 +1,49 @@ -import { access, readFile } from "node:fs/promises"; +import { access, readFile, rm } from "node:fs/promises"; import { join } from "node:path"; import { describe, expect, it } from "vitest"; import { createTestTempDirectory } from "../../tests/fixtures/temporaryDirectory.js"; -import { execFileBytes, execFileOutput } from "./ExecFileOutput.js"; +import { + execFileBytes, + execFileOutput, + execFileOutputFailure, +} from "./ExecFileOutput.js"; + +const waitForFile = async (path: string): Promise => { + for (let attempt = 0; attempt < 500; attempt += 1) { + if ( + await access(path).then( + () => true, + () => false, + ) + ) + return true; + await new Promise((resolve) => setTimeout(resolve, 10)); + } + return access(path).then( + () => true, + () => false, + ); +}; describe("execFileOutput", () => { + it("ignores non-Error and incomplete failure metadata", () => { + expect(execFileOutputFailure("failure")).toBeUndefined(); + expect(execFileOutputFailure(new Error("failure"))).toBeUndefined(); + }); + + it("recognizes the execFile output limit code as truncated output", () => { + const failure = Object.assign(new Error("output limit"), { + stdout: "partial stdout", + stderr: "partial stderr", + code: "ERR_CHILD_PROCESS_STDIO_MAXBUFFER", + }); + + expect(execFileOutputFailure(failure)?.outputTruncated).toBe(true); + }); + it("preserves captured bytes for callers that own the output format", async () => { const result = await execFileBytes(process.execPath, [ "-e", @@ -18,6 +54,20 @@ describe("execFileOutput", () => { expect(result.stderr).toEqual(Buffer.from([0xfe])); }); + it("reports raw byte counts when UTF-8 decoding changes the captured text", async () => { + const result = await execFileOutput(process.execPath, [ + "-e", + "process.stdout.write(Buffer.from([0x61, 0xff])); process.stderr.write(Buffer.from([0xe2, 0x82, 0xac]));", + ]); + + expect(result).toMatchObject({ + stdout: "a�", + stderr: "€", + stdoutBytes: 2, + stderrBytes: 3, + }); + }); + it("captures output larger than Node's default execFile limit", async () => { const marker = "tail-marker"; const result = await execFileOutput(process.execPath, [ @@ -48,6 +98,14 @@ describe("execFileOutput", () => { ).rejects.toMatchObject({ code: 7, stdout: "out", stderr: "err" }); }); + it("uses the built-in execFile timeout when no stop signal is requested", async () => { + await expect( + execFileOutput(process.execPath, ["-e", "setInterval(() => {}, 1000)"], { + timeout: 100, + }), + ).rejects.toMatchObject({ killed: true }); + }); + it.skipIf(process.platform === "win32")( "stops an aborted process with the caller's signal and settles after it exits", async () => { @@ -75,25 +133,48 @@ describe("execFileOutput", () => { ), }), ); - for (let attempt = 0; attempt < 500; attempt += 1) { - if ( - await access(ready).then( - () => true, - () => false, - ) - ) - break; - await new Promise((resolve) => setTimeout(resolve, 10)); - } - + const readyObserved = await waitForFile(ready); controller.abort(); const outcome = await execution; + expect(readyObserved).toBe(true); expect(outcome?.cause).toMatchObject({ name: "AbortError" }); expect(outcome?.stoppedBeforeSettling).toBe("SIGINT"); }, ); + it.skipIf(process.platform === "win32")( + "keeps the first stop reason when timeout arrives during abort cleanup", + async () => { + const directory = await createTestTempDirectory("rea-exec-double-stop-"); + const ready = join(directory, "ready"); + const script = [ + 'const fs = require("node:fs");', + 'for (const name of ["SIGINT", "SIGTERM"]) process.on(name, () => {});', + `fs.writeFileSync(${JSON.stringify(ready)}, "");`, + "setInterval(() => {}, 1000);", + ].join("\n"); + const controller = new AbortController(); + const execution = execFileOutput(process.execPath, ["-e", script], { + signal: controller.signal, + timeout: 10_000, + stopSignal: "SIGINT", + }); + const startedAt = Date.now(); + expect(await waitForFile(ready)).toBe(true); + const remainingBeforeDeadline = Math.max( + 0, + 9_000 - (Date.now() - startedAt), + ); + await new Promise((resolve) => + setTimeout(resolve, remainingBeforeDeadline), + ); + controller.abort(); + + await expect(execution).rejects.toMatchObject({ name: "AbortError" }); + }, + ); + it("rejects an already aborted stoppable command without starting it", async () => { const controller = new AbortController(); controller.abort(); @@ -149,6 +230,33 @@ describe.skipIf(process.platform === "win32")( }, ); + it("preserves a child failure's missing signal when reporting timeout", async () => { + const directory = await createTestTempDirectory( + "rea-exec-missing-signal-", + ); + const ready = join(directory, "ready"); + const script = [ + 'const fs = require("node:fs");', + 'for (const name of ["SIGINT", "SIGTERM"])', + " process.on(name, () => process.exit(1));", + `fs.writeFileSync(${JSON.stringify(ready)}, "");`, + "setInterval(() => {}, 1000);", + ].join("\n"); + try { + const execution = execFileOutput(process.execPath, ["-e", script], { + timeout: 10_000, + stopSignal: "SIGINT", + }); + expect(await waitForFile(ready)).toBe(true); + await expect(execution).rejects.toMatchObject({ + code: "ETIMEDOUT", + signal: null, + }); + } finally { + await rm(directory, { recursive: true, force: true }); + } + }); + it("kills a timed-out process that ignores the stop signal", async () => { const script = [ 'for (const name of ["SIGINT", "SIGTERM"]) process.on(name, () => {});', @@ -168,3 +276,40 @@ describe.skipIf(process.platform === "win32")( }); }, ); + +describe("execFileOutput failure metadata", () => { + it("preserves raw byte counts on a failed command with invalid UTF-8", async () => { + const failure: unknown = await execFileOutput(process.execPath, [ + "-e", + "process.stdout.write(Buffer.from([0x61, 0xff])); process.stderr.write(Buffer.from([0xe2, 0x82, 0xac])); process.exitCode = 7;", + ]).then( + () => undefined, + (cause: unknown) => cause, + ); + + expect(execFileOutputFailure(failure)).toMatchObject({ + stdout: "a�", + stderr: "€", + stdoutBytes: 2, + stderrBytes: 3, + code: 7, + }); + }); + it("preserves explicit truncation metadata from an owned process failure", () => { + const failure = Object.assign(new Error("owned output was truncated"), { + stdout: "partial stdout", + stderr: "partial stderr", + code: 7, + outputTruncated: true, + }); + + expect(execFileOutputFailure(failure)).toEqual({ + stdout: "partial stdout", + stderr: "partial stderr", + code: 7, + signal: null, + killed: false, + outputTruncated: true, + }); + }); +}); diff --git a/src/process/ExecFileOutput.ts b/src/process/ExecFileOutput.ts index 08046d83d3..b51f6e6ae1 100644 --- a/src/process/ExecFileOutput.ts +++ b/src/process/ExecFileOutput.ts @@ -76,6 +76,9 @@ export interface ExecFileOutputFailure { readonly signal: string | null; readonly killed: boolean; readonly outputTruncated: boolean; + readonly stdoutBytes?: number; + readonly stderrBytes?: number; + readonly collectionFailure?: boolean; } /** Read captured output metadata from a failed execFileOutput invocation. */ @@ -88,15 +91,31 @@ export const execFileOutputFailure = ( const code = Reflect.get(cause, "code"); const signal = Reflect.get(cause, "signal"); const killed = Reflect.get(cause, "killed"); + const outputTruncated = Reflect.get(cause, "outputTruncated"); + const stdoutBytes = Reflect.get(cause, "stdoutBytes"); + const stderrBytes = Reflect.get(cause, "stderrBytes"); + const collectionFailure = Reflect.get(cause, "collectionFailure"); if (typeof stdout !== "string" || typeof stderr !== "string") return undefined; return { stdout, stderr, + ...(typeof stdoutBytes === "number" && + Number.isSafeInteger(stdoutBytes) && + stdoutBytes >= 0 + ? { stdoutBytes } + : {}), + ...(typeof stderrBytes === "number" && + Number.isSafeInteger(stderrBytes) && + stderrBytes >= 0 + ? { stderrBytes } + : {}), code: typeof code === "string" || typeof code === "number" ? code : null, signal: typeof signal === "string" ? signal : null, killed: killed === true, - outputTruncated: code === "ERR_CHILD_PROCESS_STDIO_MAXBUFFER", + outputTruncated: + outputTruncated === true || code === "ERR_CHILD_PROCESS_STDIO_MAXBUFFER", + ...(collectionFailure === true ? { collectionFailure: true } : {}), }; }; @@ -150,6 +169,8 @@ export const execFileBytes = ( if (failure !== null) { Reflect.set(failure, "stdout", stdout.toString("utf8")); Reflect.set(failure, "stderr", stderr.toString("utf8")); + Reflect.set(failure, "stdoutBytes", stdout.byteLength); + Reflect.set(failure, "stderrBytes", stderr.byteLength); reject(failure); return; } @@ -179,8 +200,15 @@ export const execFileOutput = ( command: string, arguments_: readonly string[], options: ExecFileOutputOptions = {}, -): Promise<{ readonly stdout: string; readonly stderr: string }> => +): Promise<{ + readonly stdout: string; + readonly stderr: string; + readonly stdoutBytes: number; + readonly stderrBytes: number; +}> => execFileBytes(command, arguments_, options).then(({ stdout, stderr }) => ({ stdout: stdout.toString("utf8"), stderr: stderr.toString("utf8"), + stdoutBytes: stdout.byteLength, + stderrBytes: stderr.byteLength, })); diff --git a/src/hopper/HopperTargetLease.ts b/src/process/HopperLease.ts similarity index 75% rename from src/hopper/HopperTargetLease.ts rename to src/process/HopperLease.ts index 1507002670..51f1b66407 100644 --- a/src/hopper/HopperTargetLease.ts +++ b/src/process/HopperLease.ts @@ -1,5 +1,6 @@ import { createHash } from "node:crypto"; -import { chmod, lstat, mkdir, realpath, rm, rmdir } from "node:fs/promises"; +import { chmod, mkdir, open, realpath, rm, rmdir } from "node:fs/promises"; +import { constants as fsConstants } from "node:fs"; import { createConnection, createServer, @@ -16,6 +17,8 @@ interface HopperTargetLeaseOwner { } export interface HopperTargetLease { + /** Bound socket identity for cleanup diagnostics; omission remains unknown. */ + readonly socketPath?: string; readonly release: () => Promise; } @@ -65,6 +68,21 @@ export const acquireLinuxHopperApplicationLease = (input: { }): Promise => acquireLease({ ...input, identity: { application: "linux-hopper-demo" } }); +/** Serialize approved publication/repair of one canonical signed-package entry. */ +export const acquireHopperReleaseCacheLease = async (input: { + readonly cacheDirectory: string; + readonly launcherSha256: string; + readonly runId: string; +}): Promise => + acquireLease({ + runId: input.runId, + identity: { + cacheDirectory: await realpath(input.cacheDirectory), + launcherSha256: input.launcherSha256, + operation: "signed-package-retention", + }, + }); + const acquireLease = async (input: { readonly identity: Readonly>; readonly runId: string; @@ -152,6 +170,7 @@ const tryAcquireLease = async ( return { acquired: true, lease: { + socketPath, release: async () => { if (released) return; // Node removes a bound Unix socket on close. A second unlink here @@ -164,27 +183,69 @@ const tryAcquireLease = async ( }; const ensureLeaseDirectory = async (directory: string): Promise => { - let directoryStat: Awaited>; try { await mkdir(directory, { recursive: true, mode: 0o700 }); - directoryStat = await lstat(directory); } catch (cause: unknown) { throw new HopperStartError({ cause, - userMessage: - "REA could not create or inspect its private Hopper lease directory.", + userMessage: "REA could not create its private Hopper lease directory.", }); } - if ( - directoryStat.isSymbolicLink() || - directoryStat.isDirectory() === false || - (process.getuid !== undefined && directoryStat.uid !== process.getuid()) - ) + // Use O_DIRECTORY|O_NOFOLLOW to open the directory without following symlinks, + // then inspect and secure that descriptor. Later socket operations still use + // the path and are not made atomic by this directory validation. + let handle; + try { + handle = await open( + directory, + fsConstants.O_RDONLY | + (fsConstants.O_DIRECTORY ?? 0) | + (fsConstants.O_NOFOLLOW ?? 0), + ); + const directoryStat = await handle.stat(); + if ( + directoryStat.isSymbolicLink() || + directoryStat.isDirectory() === false || + (process.getuid !== undefined && directoryStat.uid !== process.getuid()) + ) + throw new HopperStartError({ + userMessage: + "REA's Hopper lease directory is not a private directory owned by this user.", + }); + // POSIX permission hardening: chmod on the already-opened descriptor avoids + // a second TOCTOU window where the directory could be swapped between check + // and permission fix. Windows DACL-based security is handled separately. + const posixPermissionsApply = process.getuid !== undefined; + if (posixPermissionsApply && (directoryStat.mode & 0o077) !== 0) { + await handle.chmod(0o700); + // Re-verify permissions on the same opened directory. + const afterChmod = await handle.stat(); + if ( + afterChmod.isSymbolicLink() || + afterChmod.isDirectory() === false || + (afterChmod.mode & 0o077) !== 0 + ) + throw new HopperStartError({ + userMessage: + "REA's Hopper lease directory permissions could not be secured.", + }); + } + } catch (cause: unknown) { + if (cause instanceof HopperStartError) throw cause; + const accessDenied = + process.getuid !== undefined && + cause instanceof Error && + "code" in cause && + cause.code === "EACCES"; throw new HopperStartError({ - userMessage: - "REA's Hopper lease directory is not a private directory owned by this user.", + cause, + userMessage: accessDenied + ? `REA could not access its Hopper lease directory at ${JSON.stringify(directory)}. Check its ownership and parent-directory access. For a directory you own, restore owner-only read, write, and search permissions (0700), then retry.` + : "REA could not inspect its private Hopper lease directory.", }); - if ((directoryStat.mode & 0o077) !== 0) await chmod(directory, 0o700); + } finally { + await handle?.close(); + } }; const bind = (server: Server, socketPath: string): Promise => diff --git a/src/process/MacCodeSignature.ts b/src/process/MacCodeSignature.ts new file mode 100644 index 0000000000..92cd7f86ff --- /dev/null +++ b/src/process/MacCodeSignature.ts @@ -0,0 +1,7 @@ +/** Require a sealed identifier and Apple Developer ID signing team together. */ +export const macDeveloperIdRequirement = ( + identifier: string, + teamIdentifier: string, +): string => + `= identifier "${identifier}" and anchor apple generic` + + ` and certificate leaf[subject.OU] = "${teamIdentifier}"`; diff --git a/src/process/OwnedCommand.ts b/src/process/OwnedCommand.ts index d375971ae9..3a77c73425 100644 --- a/src/process/OwnedCommand.ts +++ b/src/process/OwnedCommand.ts @@ -1,6 +1,7 @@ import type { OwnedProviderProcessSpawnOptions, ProviderProcessSnapshot, + ProviderProcessStopResult, SpawnedOwnedProviderProcess, } from "./ProviderProcess.js"; import { @@ -20,18 +21,23 @@ import { /** A short-lived command failed its lifecycle or complete-output contract. */ export class OwnedCommandFailure extends Error { readonly cleanupResult: ProcessCleanupFailure | undefined; + readonly collectionFailure: boolean; constructor( readonly reason: "cancelled" | "timeout" | "output-limit" | "process", message: string, readonly snapshot: ProviderProcessSnapshot | null = null, readonly cleanupFailure: string | null = null, - options?: ErrorOptions & { readonly cleanupResult?: ProcessCleanupFailure }, + options?: ErrorOptions & { + readonly cleanupResult?: ProcessCleanupFailure; + readonly collectionFailure?: boolean; + }, readonly resources: readonly string[] = [], readonly cleanupOwner?: ProviderProcessSupervisor, ) { super(message, options); this.cleanupResult = options?.cleanupResult; + this.collectionFailure = options?.collectionFailure ?? false; } } @@ -56,9 +62,12 @@ export const runOwnedCommand = async ( readonly acceptNonZeroExit?: boolean; }, options: { - readonly signal?: AbortSignal; + readonly signal?: AbortSignal | undefined; /** Synchronously accept unresolved cleanup ownership; execution still returns its original outcome. Must not throw. */ - readonly retainCleanup?: (owner: ProviderProcessSupervisor) => void; + readonly retainCleanup?: ( + owner: ProviderProcessSupervisor, + result: Extract, + ) => void; readonly launcher?: ( input: OwnedProviderProcessSpawnOptions, ) => Promise; @@ -97,7 +106,10 @@ const collectOwnedCommand = async ( ) => Promise, deadline: ProviderStartupDeadline, callerSignal?: AbortSignal, - retainCleanup?: (owner: ProviderProcessSupervisor) => void, + retainCleanup?: ( + owner: ProviderProcessSupervisor, + result: Extract, + ) => void, ): Promise => { const signal = deadline.signal; let launched: SpawnedOwnedProviderProcess; @@ -156,7 +168,9 @@ const collectOwnedCommand = async ( "Command diagnostic output exceeded its complete-output budget.", ); if (processFailure !== undefined) - throw new OwnedCommandFailure("process", processFailure); + throw new OwnedCommandFailure("process", processFailure, null, null, { + collectionFailure: true, + }); await waitForAbortableDelay(25, signal); } const snapshot = supervisor.snapshot(); @@ -191,6 +205,8 @@ const collectOwnedCommand = async ( processFailure ?? "Command failed; inspect its exit status and diagnostics.", snapshot, + null, + { collectionFailure: processFailure !== undefined }, ); } catch (cause: unknown) { failure = @@ -207,7 +223,7 @@ const collectOwnedCommand = async ( const stopped = await supervisor.stop(); const snapshot = supervisor.snapshot(); if (stopped.status === "incomplete" && retainCleanup !== undefined) - retainCleanup(supervisor); + retainCleanup(supervisor, stopped); else if (stopped.status === "incomplete") throw new OwnedCommandFailure( failure?.reason ?? "process", @@ -219,6 +235,9 @@ const collectOwnedCommand = async ( ...(stopped.cleanupResult === undefined ? {} : { cleanupResult: stopped.cleanupResult }), + ...(failure?.collectionFailure === true + ? { collectionFailure: true } + : {}), }, ownedCommandResources(launched.ownership, spawn.cwd), supervisor, @@ -234,6 +253,7 @@ const collectOwnedCommand = async ( ...(failure.cleanupResult === undefined ? {} : { cleanupResult: failure.cleanupResult }), + ...(failure.collectionFailure ? { collectionFailure: true } : {}), }, failure.resources, ); diff --git a/src/process/ProcessOwnership.test.ts b/src/process/ProcessOwnership.test.ts index de233698c2..fafa399508 100644 --- a/src/process/ProcessOwnership.test.ts +++ b/src/process/ProcessOwnership.test.ts @@ -209,6 +209,185 @@ describe("opaque process neighbors during cleanup", () => { }); describe("related sweep candidates during cleanup", () => { + it("counts each unreadable live member once", async () => { + const processes = [ + { + pid: 100, + parentPid: 1, + processGroupId: 100, + state: "S", + command: "capture", + }, + { + pid: 101, + parentPid: 100, + processGroupId: 100, + state: "S", + command: "child-a", + }, + { + pid: 102, + parentPid: 100, + processGroupId: 100, + state: "S", + command: "child-b", + }, + ]; + const signalGroup = vi.fn(); + const adapter: ProcessOwnershipHost = { + listProcesses: async () => processes, + environment: async () => ({}), + runTokens: async (members) => + new Map( + members.map(({ pid }) => [ + pid, + pid === 100 + ? { state: "readable" as const, runId: ownership.runId } + : { + state: "unavailable" as const, + reason: "environment_unavailable", + }, + ]), + ), + signalGroup, + }; + const result = await cleanupOwnedProcessGroup( + { ...ownership, sweepTokenOwnedProcesses: true, captureBaseline: [] }, + adapter, + ); + if (result.cleaned) throw new Error("expected cleanup failure"); + expect(signalGroup).not.toHaveBeenCalled(); + expect(result.failures?.map(({ pid }) => pid).sort()).toEqual([101, 102]); + expect(result.reason).toContain("2 live process(es)"); + }); + + it("upgrades unreadable evidence when later group validation finds a foreign token", async () => { + const processes = [ + { + pid: 100, + parentPid: 1, + processGroupId: 100, + state: "S", + command: "capture", + }, + { + pid: 200, + parentPid: 100, + processGroupId: 200, + state: "S", + command: "detached-child", + }, + ]; + const signalGroup = vi.fn(); + let childReads = 0; + const adapter: ProcessOwnershipHost = { + listProcesses: async () => processes, + processIdentities: async (members) => + new Map( + members.map(({ pid }) => [ + pid, + { state: "readable" as const, identity: `start-${String(pid)}` }, + ]), + ), + environment: async () => ({}), + runTokens: async (members) => + new Map( + members.map(({ pid }) => [ + pid, + pid === 100 + ? { state: "readable" as const, runId: ownership.runId } + : ++childReads <= 4 + ? { + state: "unavailable" as const, + reason: "environment_unavailable", + } + : { state: "readable" as const, runId: "foreign-token" }, + ]), + ), + signalGroup, + }; + const result = await cleanupOwnedProcessGroup( + { ...ownership, sweepTokenOwnedProcesses: true, captureBaseline: [] }, + adapter, + ); + expect(result).toMatchObject({ + cleaned: false, + reason: "process ownership token mismatch for PID(s): 200", + failures: [{ pid: 200, reason: "run-token-mismatch" }], + }); + expect(signalGroup).not.toHaveBeenCalledWith(200, "SIGKILL"); + }); +}); + +describe("withheld process evidence during cleanup", () => { + it.each(["revalidation", "signal-error"] as const)( + "retains withheld-process evidence after %s failure", + async (mode) => { + const processes = [ + { + pid: 100, + parentPid: 1, + processGroupId: 100, + state: "S", + command: "fixture", + }, + { + pid: 900, + parentPid: 1, + processGroupId: 900, + state: "S", + command: "opaque", + }, + ]; + let rootReads = 0; + const signalGroup = vi.fn(() => { + if (mode === "signal-error") + throw Object.assign(new Error("signal denied"), { code: "EACCES" }); + }); + const adapter: ProcessOwnershipHost = { + listProcesses: async () => processes, + environment: async () => ({}), + runTokens: async (members) => + new Map( + members.map(({ pid }) => [ + pid, + pid === 900 + ? { + state: "unavailable" as const, + reason: "platform_binary_environment_withheld", + } + : { + state: "readable" as const, + runId: + mode === "revalidation" && ++rootReads > 2 + ? "other-run" + : "run-token", + }, + ]), + ), + signalGroup, + }; + + const result = await cleanupOwnedProcessGroup( + { ...ownership, sweepTokenOwnedProcesses: true, captureBaseline: [] }, + adapter, + ); + + expect(result.unverified).toEqual([ + { pid: 900, diagnostic: "platform_binary_environment_withheld" }, + ]); + if (result.cleaned) throw new Error("expected cleanup failure"); + if (mode === "revalidation") + expect(result.reason).toBe( + "process ownership token mismatch for PID(s): 100", + ); + else + expect(result.reason).toBe( + "owned process group signal failed: signal denied", + ); + }, + ); + it.each([ "environment_unavailable", "platform_binary_environment_withheld", diff --git a/src/process/ProcessOwnership.ts b/src/process/ProcessOwnership.ts index 399b9cd063..91d681f79b 100644 --- a/src/process/ProcessOwnership.ts +++ b/src/process/ProcessOwnership.ts @@ -227,6 +227,15 @@ export const readProcessRunId = async ( ): Promise => (await host.environment(pid, signal)).REA_PROCESS_RUN_ID; +/** Keep unrelated-process uncertainty attached to every outcome of this cleanup attempt. */ +const withUnverifiedObservations = ( + result: ProcessCleanupFailure, + unverified: readonly OwnershipSweepUnverifiedProcess[], +): ProcessCleanupFailure => + unverified.length === 0 || (result.unverified?.length ?? 0) > 0 + ? result + : { ...result, unverified }; + /** Token-validate every rooted POSIX process group before signaling any. */ export const cleanupOwnedProcessGroup = async ( ownership: OwnedProcessGroup, @@ -252,7 +261,16 @@ export const cleanupOwnedProcessGroup = async ( host, plan.tokenOwnedIdentities, ); - if ("cleaned" in revalidation) return revalidation; + if ("cleaned" in revalidation) { + if (!revalidation.cleaned) + return withUnverifiedObservations(revalidation, plan.unverified); + return { + ...revalidation, + ...(revalidation.unverified?.length || plan.unverified.length === 0 + ? {} + : { unverified: plan.unverified }), + }; + } if (revalidation.empty) continue; try { host.signalGroup(processGroupId, "SIGKILL"); @@ -261,10 +279,13 @@ export const cleanupOwnedProcessGroup = async ( const code = cause instanceof Error && "code" in cause ? cause.code : undefined; if (code !== "ESRCH") - return { - cleaned: false, - reason: `owned process group signal failed: ${errorMessage(cause)}`, - }; + return withUnverifiedObservations( + { + cleaned: false, + reason: `owned process group signal failed: ${errorMessage(cause)}`, + }, + plan.unverified, + ); } } if (plan.unresolved.length > 0) @@ -507,7 +528,8 @@ const createOwnedCleanupPlan = async ( const scanFailures = tokenOwned.failures.filter(({ pid }) => members.some((member) => member.pid === pid), ); - if (failures.length > 0) unresolved.push(...failures); + for (const failure of failures) + mergeProcessOwnershipFailure(unresolved, failure); if ( failures.length === 0 && scanFailures.length === 0 && @@ -533,6 +555,36 @@ const createOwnedCleanupPlan = async ( }; }; +/** Keep one current, most useful ownership diagnosis for each live PID. */ +const mergeProcessOwnershipFailure = ( + failures: ProcessOwnershipValidationFailure[], + next: ProcessOwnershipValidationFailure, +): void => { + const index = failures.findIndex(({ pid }) => pid === next.pid); + if (index < 0) { + failures.push(next); + return; + } + const previous = failures[index]; + if ( + previous !== undefined && + ownershipFailurePriority(next.reason) >= + ownershipFailurePriority(previous.reason) + ) + failures[index] = next; +}; + +const ownershipFailurePriority = ( + reason: ProcessOwnershipValidationFailure["reason"], +): number => + reason === "run-token-mismatch" + ? 3 + : reason === "process-identity-unavailable" + ? 2 + : reason === "environment-unreadable" + ? 1 + : 0; + const revalidateOwnedProcessGroup = async ( processGroupId: number, ownership: OwnedProcessGroup, @@ -768,9 +820,27 @@ const recheckOwnershipFailures = async ( if (!failures.some(({ reason }) => reason !== "run-token-mismatch")) return failures; try { - const live = new Set( - liveProcesses(await host.listProcesses()).map(({ pid }) => pid), - ); + let processes = await host.listProcesses(); + const isDarwin = (host.platform ?? process.platform) === "darwin"; + for ( + let attempt = 0; + isDarwin && + attempt < TOKEN_READ_RETRY_ATTEMPTS && + processes.some( + ({ pid, state }) => + failures.some( + (failure) => + failure.pid === pid && failure.reason !== "run-token-mismatch", + ) && state.slice(1).includes("E"), + ); + attempt += 1 + ) { + await new Promise((resolve) => + setTimeout(resolve, TOKEN_READ_RETRY_DELAY_MS), + ); + processes = await host.listProcesses(); + } + const live = new Set(liveProcesses(processes).map(({ pid }) => pid)); return failures.filter( ({ pid, reason }) => reason === "run-token-mismatch" || live.has(pid), ); diff --git a/src/process/ProcessOwnership.validation.test.ts b/src/process/ProcessOwnership.validation.test.ts index c718f4b936..ed6179a77e 100644 --- a/src/process/ProcessOwnership.validation.test.ts +++ b/src/process/ProcessOwnership.validation.test.ts @@ -21,6 +21,40 @@ const launcher: ProcessTableEntry = { }; describe("owned process-group cleanup validation: ownership and lineage", () => { + it("settles unreadable observations for Darwin processes exiting in the kernel", async () => { + let snapshots = 0; + const child = { + pid: 101, + parentPid: 100, + processGroupId: 100, + state: "?E", + command: "fixture-child", + }; + const adapter: ProcessOwnershipHost = { + platform: "darwin", + listProcesses: () => { + snapshots += 1; + return Promise.resolve( + snapshots === 1 + ? [launcher, child] + : snapshots === 2 + ? [launcher, child] + : [launcher, { ...child, state: "Z" }], + ); + }, + environment: (pid) => + pid === 100 + ? Promise.resolve({ REA_PROCESS_RUN_ID: "run-token" }) + : Promise.reject(new Error("sysctl_errno_22")), + signalGroup: vi.fn(), + }; + + await expect(cleanupOwnedProcessGroup(ownership, adapter)).resolves.toEqual( + { cleaned: true, signaled: true }, + ); + expect(snapshots).toBeGreaterThanOrEqual(3); + }); + it("fails closed when a descendant in another process group lacks the token", async () => { const adapter: ProcessOwnershipHost = { listProcesses: () => diff --git a/src/process/ProcessOwnershipObservation.ts b/src/process/ProcessOwnershipObservation.ts index 5747cded80..3b4311c94e 100644 --- a/src/process/ProcessOwnershipObservation.ts +++ b/src/process/ProcessOwnershipObservation.ts @@ -510,6 +510,22 @@ export const observeProcessStartIdentity = async ( }; }; +/** Confirm whether the process that held an authenticated bridge has exited. */ +export const observeProcessStartIdentityExit = async ( + pid: number, + expectedIdentity: string, + host: ProcessOwnershipHost = systemProcessOwnershipHost, +): Promise<"exited" | "present" | "unverified"> => { + try { + const observed = await observeProcessStartIdentity(pid, host); + if (observed === undefined) return "exited"; + if (observed.state !== "readable") return "unverified"; + return observed.identity === expectedIdentity ? "present" : "exited"; + } catch { + return "unverified"; + } +}; + /** Revalidate a launch-time start identity immediately before signaling a PID. */ export const signalProcessWithStartIdentity = async ( pid: number, diff --git a/src/process/ProcessStartIdentity.test.ts b/src/process/ProcessStartIdentity.test.ts index 20b2862eec..8d2f532618 100644 --- a/src/process/ProcessStartIdentity.test.ts +++ b/src/process/ProcessStartIdentity.test.ts @@ -3,6 +3,7 @@ import { describe, expect, it, vi } from "vitest"; import type { ProcessOwnershipHost } from "./ProcessOwnership.js"; import { observeProcessStartIdentity, + observeProcessStartIdentityExit, signalProcessWithStartIdentity, } from "./ProcessOwnershipObservation.js"; @@ -37,6 +38,26 @@ describe("process start identity lease", () => { }); }); + it("confirms exit or PID replacement without treating unreadable state as exit", async () => { + await expect( + observeProcessStartIdentityExit(42, "start-1", host(undefined)), + ).resolves.toBe("exited"); + await expect( + observeProcessStartIdentityExit(42, "start-1", host("replacement-start")), + ).resolves.toBe("exited"); + await expect( + observeProcessStartIdentityExit(42, "start-1", host("start-1")), + ).resolves.toBe("present"); + const unavailable: ProcessOwnershipHost = { + ...host("start-1"), + processIdentities: async () => + new Map([[42, { state: "unavailable", reason: "reader unavailable" }]]), + }; + await expect( + observeProcessStartIdentityExit(42, "start-1", unavailable), + ).resolves.toBe("unverified"); + }); + it("does not adopt a replacement PID's first identity for an owned launch", async () => { const replacement: ProcessOwnershipHost = { ...host("replacement-start"), diff --git a/src/process/ProviderDeadline.test.ts b/src/process/ProviderDeadline.test.ts index 6f26c870e1..7271328999 100644 --- a/src/process/ProviderDeadline.test.ts +++ b/src/process/ProviderDeadline.test.ts @@ -279,9 +279,9 @@ it("releases the timer and listener on dispose with a pending wait", async () => } }); -// Preserve the authored fractional-clock control while following the current -// owning deadline's re-arm contract for an early native timer callback. -it("reports zero after the re-armed deadline reaches a fractional early callback", async () => { +// Retain the authored terminal-budget regression under the newer monotonic +// expiry policy and re-arm contract for an early native timer callback. +it("reports no remaining time after terminal expiry near the monotonic boundary", async () => { vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] }); const now = vi.spyOn(performance, "now").mockReturnValue(1_000); const deadline = new ProviderStartupDeadline(40); @@ -294,6 +294,9 @@ it("reports zero after the re-armed deadline reaches a fractional early callback await vi.advanceTimersByTimeAsync(1); expect(deadline.signal.aborted).toBe(true); expect(deadline.interruption).toBe("timeout"); + // A terminal timeout stays terminal even if the monotonic source is + // observed below its deadline after the abort timer has fired. + now.mockReturnValue(1_039); expect(deadline.remainingMs()).toBe(0); } finally { deadline.dispose(); diff --git a/src/process/ProviderDeadline.ts b/src/process/ProviderDeadline.ts index 7681ecf62f..26e1b0fb02 100644 --- a/src/process/ProviderDeadline.ts +++ b/src/process/ProviderDeadline.ts @@ -80,7 +80,8 @@ export class ProviderStartupDeadline { /** Milliseconds still available to the next startup phase. */ remainingMs(): number { // Node timers can fire a fraction of a millisecond before the monotonic - // deadline, so an elapsed timer must not report time left to spend. + // deadline, so an elapsed timer must not report time left to spend. A + // terminal timeout also cannot regain budget after later clock queries. if (this.#interruption === "timeout") return 0; return Math.max(0, Math.ceil(this.#deadlineAt - performance.now())); } diff --git a/src/process/ProviderProcess.ts b/src/process/ProviderProcess.ts index 8e650898bd..17eda48d7b 100644 --- a/src/process/ProviderProcess.ts +++ b/src/process/ProviderProcess.ts @@ -6,10 +6,14 @@ import { WindowsOwnedProcess } from "../windows/WindowsOwnedProcess.js"; import type { OwnedProcessGroup, + ProcessOwnershipBaseline, ProcessCleanupFailure, ProcessCleanupResult, } from "./ProcessOwnership.js"; -import { prepareProcessOwnershipInspection } from "./ProcessOwnershipObservation.js"; +import { + prepareProcessOwnershipInspection, + systemProcessOwnershipHost, +} from "./ProcessOwnershipObservation.js"; const DEFAULT_TERMINATION_GRACE_MS = 250; const DEFAULT_KILL_GRACE_MS = 1_000; @@ -36,6 +40,8 @@ export interface OwnedProviderProcessSpawnOptions { * Windows Job Object. Other launchers cannot enforce it and refuse it. */ readonly memoryLimitBytes?: number; + /** Sweep exact run-token descendants during POSIX cleanup. */ + readonly sweepTokenOwnedProcesses?: boolean; /** Cancels native ownership preparation before provider process creation. */ readonly signal?: AbortSignal; } @@ -60,6 +66,8 @@ export interface ProviderProcessHandle extends Pick< readonly signalCode: NodeJS.Signals | null; /** True once an enforced process memory limit reported an allocation failure. */ readonly memoryLimitReached?: boolean; + /** True if a Windows backpressured stream was destroyed before output drained. */ + readonly outputIncomplete?: boolean; kill(signal?: NodeJS.Signals | number): boolean; unref?(): void; ref?(): void; @@ -91,10 +99,12 @@ export interface ProviderProcessExitObservation { /** Detached diagnostics captured for one supervised provider process. */ export interface ProviderProcessSnapshot { - /** True only when a configured diagnostic budget excluded output bytes. */ + /** True when retained diagnostics omit bytes due to the budget or provider output loss. */ readonly diagnosticTruncated?: boolean; readonly stdout: { readonly text: string; + /** True when retained bytes cannot be represented exactly as UTF-8 text. */ + readonly invalidUtf8?: true; /** Bytes retained in text, limited by maxDiagnosticBytes when configured. */ readonly bytes: number; /** All bytes drained from this stream since construction or resetOutput. */ @@ -102,6 +112,8 @@ export interface ProviderProcessSnapshot { }; readonly stderr: { readonly text: string; + /** True when retained bytes cannot be represented exactly as UTF-8 text. */ + readonly invalidUtf8?: true; /** Bytes retained in text, limited by maxDiagnosticBytes when configured. */ readonly bytes: number; readonly observedBytes: number; @@ -110,6 +122,8 @@ export interface ProviderProcessSnapshot { readonly signal: NodeJS.Signals | null | undefined; /** Present only when the process tree reached its enforced memory limit. */ readonly memoryLimitReached?: true; + /** Present when a provider stream was destroyed before all bytes were observed. */ + readonly outputIncomplete?: true; } /** Resource-safe events emitted without imposing provider protocol semantics. */ @@ -174,6 +188,16 @@ export const spawnOwnedProviderProcess = async ( await prepareProcessOwnershipInspection(options.signal); options.signal?.throwIfAborted(); const platform = options.platform ?? process.platform; + const sweepTokenOwnedProcesses = + options.sweepTokenOwnedProcesses === true && platform !== "win32"; + let captureBaseline: ProcessOwnershipBaseline | undefined; + if (sweepTokenOwnedProcesses) { + const capture = systemProcessOwnershipHost.captureBaseline; + if (capture === undefined) + throw new Error("Process ownership baseline is unavailable on this host"); + captureBaseline = await capture(options.signal); + options.signal?.throwIfAborted(); + } const hostEnvironment = options.hostEnvironment ?? process.env; const environment = { ...hostEnvironment, @@ -226,6 +250,12 @@ export const spawnOwnedProviderProcess = async ( leaderPid: pid, processGroupId: pid, expectedParentPid: process.pid, + ...(sweepTokenOwnedProcesses + ? { + sweepTokenOwnedProcesses: true, + captureBaseline: captureBaseline!, + } + : {}), ...(options.expectedCommand === null ? {} : { expectedCommand: options.expectedCommand ?? options.command }), @@ -282,12 +312,14 @@ export class ProviderProcessSupervisor { #exitObservation: ProviderProcessExitObservation | undefined; #stopPromise: Promise | undefined; #disposed = false; + #maxDiagnosticBytes: number | undefined; constructor( readonly launch: ProviderProcessLaunch, options: ProviderProcessSupervisorOptions = {}, ) { this.#options = options; + this.#maxDiagnosticBytes = options.maxDiagnosticBytes; if (options.captureStdout !== false) this.#attach(launch.process.stdout, "stdout"); this.#attach(launch.process.stderr, "stderr"); @@ -310,8 +342,11 @@ export class ProviderProcessSupervisor { /** Latest retained output and exit observation, with explicit configured truncation. */ snapshot(): ProviderProcessSnapshot { + const outputIncomplete = this.launch.process.outputIncomplete === true; return { - ...(this.#diagnosticTruncated ? { diagnosticTruncated: true } : {}), + ...(this.#diagnosticTruncated || outputIncomplete + ? { diagnosticTruncated: true } + : {}), stdout: this.#stdout.snapshot(this.#stdoutObservedBytes), stderr: this.#stderr.snapshot(this.#stderrObservedBytes), exitCode: this.#exitObservation?.code, @@ -319,6 +354,7 @@ export class ProviderProcessSupervisor { ...(this.launch.process.memoryLimitReached === true ? { memoryLimitReached: true } : {}), + ...(outputIncomplete ? { outputIncomplete: true } : {}), }; } @@ -340,7 +376,7 @@ export class ProviderProcessSupervisor { ); } - /** Wait until the producer has closed its output streams; exit alone does not prove complete output. */ + /** Wait until output streams close; WindowsOwnedProcess closes only after its whole Job Object settles. */ async waitForOutputClose(timeoutMs: number): Promise { return waitForProcessEvent( this.#outputClosed, @@ -349,6 +385,35 @@ export class ProviderProcessSupervisor { ); } + /** Bound additional retained bytes while still draining all later output. */ + limitAdditionalOutputBytes(maxBytes: number): void { + if (!Number.isSafeInteger(maxBytes) || maxBytes < 0) + throw new RangeError( + "Additional output limit must be a non-negative safe integer.", + ); + const retained = this.#stdout.bytes + this.#stderr.bytes; + const requestedLimit = Math.min( + Number.MAX_SAFE_INTEGER, + retained + maxBytes, + ); + this.#maxDiagnosticBytes = + this.#maxDiagnosticBytes === undefined + ? requestedLimit + : Math.min(this.#maxDiagnosticBytes, requestedLimit); + } + + /** Unreference inherited output pipes without closing streams shared with descendants. */ + unrefOutputStreams(): void { + for (const stream of [ + this.launch.process.stdout, + this.launch.process.stderr, + ]) { + const unref = (stream as (Readable & { unref?: () => void }) | null) + ?.unref; + unref?.call(stream); + } + } + /** Share each stop attempt; incomplete cleanup retains ownership and permits retry. */ stop( options: ProviderProcessStopOptions = {}, @@ -509,13 +574,11 @@ export class ProviderProcessSupervisor { if (stream === "stdout") this.#stdoutObservedBytes += bytes.byteLength; else this.#stderrObservedBytes += bytes.byteLength; const remaining = - this.#options.maxDiagnosticBytes === undefined + this.#maxDiagnosticBytes === undefined ? bytes.byteLength : Math.max( 0, - this.#options.maxDiagnosticBytes - - this.#stdout.bytes - - this.#stderr.bytes, + this.#maxDiagnosticBytes - this.#stdout.bytes - this.#stderr.bytes, ); const retained = Math.min(bytes.byteLength, remaining); if (retained < bytes.byteLength) this.#diagnosticTruncated = true; @@ -564,11 +627,19 @@ class ProcessOutputCapture { } snapshot(observedBytes: number): ProviderProcessSnapshot["stdout"] { - const text = Buffer.concat(this.#chunks, this.#bytes).toString("utf8"); + const buffer = Buffer.concat(this.#chunks, this.#bytes); + const text = buffer.toString("utf8"); + let invalidUtf8 = false; + try { + new TextDecoder("utf-8", { fatal: true }).decode(buffer); + } catch { + invalidUtf8 = true; + } return { text, bytes: this.#bytes, observedBytes, + ...(invalidUtf8 ? { invalidUtf8: true } : {}), }; } } diff --git a/src/process/ProviderProcessOutput.test.ts b/src/process/ProviderProcessOutput.test.ts index 3ca2621f06..993386e79e 100644 --- a/src/process/ProviderProcessOutput.test.ts +++ b/src/process/ProviderProcessOutput.test.ts @@ -122,6 +122,30 @@ it("reports a reached process memory limit only when the handle observed one", ( } }); +it("projects provider output loss as truncated diagnostics without inventing a byte count", () => { + const producer = Object.assign(new EventEmitter(), { + stdout: new PassThrough(), + stderr: new PassThrough(), + exitCode: null, + signalCode: null, + kill: () => false, + }); + Object.defineProperty(producer, "outputIncomplete", { value: true }); + const supervisor = new ProviderProcessSupervisor({ + process: producer, + ownsProcessLifetime: false, + }); + try { + expect(supervisor.snapshot()).toMatchObject({ + diagnosticTruncated: true, + outputIncomplete: true, + }); + expect(supervisor.snapshot()).not.toHaveProperty("discardedOutputBytes"); + } finally { + supervisor.dispose(); + } +}); + it("unreferences process and stream handles when stop reports incomplete cleanup", async () => { const unrefProcess = vi.fn(); const unrefStdin = vi.fn(); diff --git a/src/process/WindowsAuthority.ts b/src/process/WindowsAuthority.ts index b25a22863d..3c83fe6885 100644 --- a/src/process/WindowsAuthority.ts +++ b/src/process/WindowsAuthority.ts @@ -1,4 +1,7 @@ -import { systemWindowsNativeAuthority } from "../windows/WindowsNativeLoader.js"; +import { + systemWindowsNativeAuthority, + type WindowsNativeLoadResult, +} from "../windows/WindowsNativeLoader.js"; /** Windows controls are proven by the packaged native boundary, not POSIX flags. */ export type WindowsNativeCapability = @@ -12,6 +15,15 @@ export type WindowsNativeCapability = readonly reason: string; readonly proof: "not-proven" | "not-applicable"; readonly retryable?: true; + readonly code: + | "unsupported_host" + | "runtime_missing" + | "runtime_unavailable"; + readonly failureStage: Exclude< + WindowsNativeLoadResult, + { available: true } + >["failureStage"]; + readonly remediation: string; }; /** Security controls required before REA may claim Windows isolation. */ @@ -25,64 +37,87 @@ export interface WindowsNativeCapabilitySet { export const WINDOWS_NATIVE_AUTHORITY_UNAVAILABLE_REASON = "Windows native controls are unavailable in this runtime; use the Windows x64 REA package with its matching bundled native artifact."; +/** One readiness observation, shared by diagnostics and security controls. */ +export type WindowsNativeReadiness = + | { readonly available: true } + | { + readonly available: false; + readonly reason: string; + readonly failureStage: Exclude< + WindowsNativeLoadResult, + { available: true } + >["failureStage"]; + readonly code: + | "unsupported_host" + | "runtime_missing" + | "runtime_unavailable"; + readonly retryable?: true; + readonly remediation: string; + readonly proof: "not-proven" | "not-applicable"; + }; + +/** Interpret the loader's tagged failure without guessing from OS error text. */ +export const inspectWindowsNativeReadiness = ( + platform: NodeJS.Platform, + observation?: WindowsNativeLoadResult, +): WindowsNativeReadiness => { + if (platform !== "win32") + return { + available: false, + failureStage: "host", + reason: "Windows native authority is not applicable on this host", + proof: "not-applicable", + code: "unsupported_host", + remediation: "Use a Windows x64 host for Windows native controls.", + }; + const loaded = observation ?? systemWindowsNativeAuthority(); + if (loaded.available) return { available: true }; + const failureStage = loaded.failureStage; + return { + available: false, + failureStage, + reason: loaded.reason, + ...(loaded.retryable === true ? { retryable: true as const } : {}), + proof: "not-proven", + code: + failureStage === "host" + ? "unsupported_host" + : failureStage === "artifact" + ? "runtime_missing" + : "runtime_unavailable", + remediation: + failureStage === "host" + ? "Use a Windows x64 host and a supported Node.js runtime with Node-API 8 or newer." + : failureStage === "artifact" + ? "Use the matching Windows x64 REA package, or restore native/windows/build/manifest.json and native/windows/build/rea-windows-x64.node from the same-version rea-agents tarball or build them with native/windows/README.md. The loader checks version, ABI, architecture, and digest before granting authority; no download or install is performed automatically." + : failureStage === "inspection" + ? "Install the matching Windows x64 REA package; its native inspection must prove every required security control." + : "Inspect the preserved native self-test error. Check that the process's temporary directory is writable local NTFS and that its filesystem, DACL, Job Object, and cleanup operations are permitted. Resolve the reported constraint, then rerun rea doctor; reinstalling Ghidra or Java cannot fix this self-test.", + }; +}; + /** * Describe native Windows controls without inferring them from taskkill, * chmod, symlink creation, or a numeric POSIX open flag. */ export const windowsNativeCapabilities = ( platform: NodeJS.Platform, + readiness: WindowsNativeReadiness = inspectWindowsNativeReadiness(platform), ): WindowsNativeCapabilitySet => { - if (platform !== "win32") { - const notApplicable = { - available: false, - reason: "Windows native authority is not applicable on this host", - proof: "not-applicable", - } as const; - return { - job_object_process_ownership: notApplicable, - private_runtime_dacl: notApplicable, - reparse_safe_path_admission: notApplicable, - }; - } - - if (process.platform === "win32") { - const loaded = systemWindowsNativeAuthority(); - const capability = loaded.available - ? ({ available: true, reason: null, proof: "native-authority" } as const) - : ({ - available: false, - reason: loaded.reason, - proof: "not-proven", - ...(loaded.retryable === true ? { retryable: true as const } : {}), - } as const); - return { - job_object_process_ownership: capability, - private_runtime_dacl: capability, - reparse_safe_path_admission: capability, - }; - } - - const notProven = { - available: false, - reason: WINDOWS_NATIVE_AUTHORITY_UNAVAILABLE_REASON, - proof: "not-proven", - } as const; + const capability = readiness.available + ? ({ available: true, reason: null, proof: "native-authority" } as const) + : ({ + available: false, + reason: readiness.reason, + proof: readiness.proof, + code: readiness.code, + failureStage: readiness.failureStage, + remediation: readiness.remediation, + ...(readiness.retryable === true ? { retryable: true as const } : {}), + } as const); return { - job_object_process_ownership: notProven, - private_runtime_dacl: notProven, - reparse_safe_path_admission: notProven, + job_object_process_ownership: capability, + private_runtime_dacl: capability, + reparse_safe_path_admission: capability, }; }; - -/** Whether every native control needed for Windows isolation is proven. */ -export const hasWindowsNativeAuthority = (platform: NodeJS.Platform): boolean => - Object.values(windowsNativeCapabilities(platform)).every( - ({ available }) => available, - ); - -/** Preserve the actual loader or operating-system constraint in provider errors. */ -export const windowsNativeAuthorityUnavailableReason = ( - platform: NodeJS.Platform, -): string => - windowsNativeCapabilities(platform).job_object_process_ownership.reason ?? - WINDOWS_NATIVE_AUTHORITY_UNAVAILABLE_REASON; diff --git a/src/process/capture/FilesystemCoverage.test.ts b/src/process/capture/FilesystemCoverage.test.ts index baa5f5c942..a2ea110798 100644 --- a/src/process/capture/FilesystemCoverage.test.ts +++ b/src/process/capture/FilesystemCoverage.test.ts @@ -100,6 +100,45 @@ it("distinguishes unavailable content hashes from completed path enumeration", a ).toMatchObject({ status: "deleted", before: { sha256: null }, after: null }); }); +it.skipIf(process.platform === "win32" || process.getuid?.() === 0)( + "retains readable entries and marks an unreadable directory subtree unknown", + async () => { + const root = await createTestTempDirectory("rea-fs-unreadable-directory-"); + const visible = join(root, "a-visible.txt"); + const blocked = join(root, "z-blocked"); + await writeFile(visible, "visible"); + await mkdir(blocked); + await chmod(blocked, 0); + const scenario = parseProcessScenario({ + executable: process.execPath, + working_directory: root, + filesystem_observation_paths: [root], + }); + + try { + const snapshot = await snapshotRoots(scenario); + expect(snapshot.files.map(({ path }) => path)).toContain( + "root_0:a-visible.txt", + ); + expect(snapshot.files.map(({ path }) => path)).toContain( + "root_0:z-blocked", + ); + expect(snapshot.completeRoots).toEqual([]); + expect(snapshot.coverage.enumeration_omissions).toEqual([ + { + path: "root_0:z-blocked", + reason: "directory_unavailable", + system_code: "EACCES", + }, + ]); + expect(snapshot.truncated).toBe(true); + } finally { + await chmod(blocked, 0o700); + await rm(root, { recursive: true, force: true }); + } + }, +); + it.skipIf(process.platform === "win32" || process.getuid?.() === 0)( "retains unreadable-file and sibling observations in a complete snapshot", async () => { @@ -300,6 +339,38 @@ it.skipIf(process.platform === "win32")( }, ); +it.skipIf(process.platform === "win32")( + "keeps the observed symlink target when dot segments resolve to the same path", + async () => { + const root = await createTestTempDirectory("rea-fs-symlink-spelling-"); + await mkdir(join(root, "aaa")); + await writeFile(join(root, "target"), "content"); + // Equal length, so only the target spelling distinguishes the links; + // "bbb" does not exist, so the replacement link is broken. + await symlink("aaa/../target", join(root, "link")); + const scenario = parseProcessScenario({ + executable: process.execPath, + working_directory: root, + filesystem_observation_paths: [root], + limits: { file_bytes: 1 }, + }); + const before = await snapshotRoots(scenario); + await rm(join(root, "link")); + await symlink("bbb/../target", join(root, "link")); + const after = await snapshotRoots(scenario); + + const link = (snapshot: typeof before) => + snapshot.files.find(({ path }) => path === "root_0:link"); + expect(link(before)?.symlink_target).toBe("aaa/../target"); + expect(link(after)?.symlink_target).toBe("bbb/../target"); + expect( + classifyFilesystemEffects(before, after).find( + ({ path }) => path === "root_0:link", + )?.status, + ).toBe("modified"); + }, +); + it("preserves creation when the selected root was observed missing before capture", async () => { const root = await createTestTempDirectory("rea-fs-missing-root-"); const file = join(root, "created.txt"); diff --git a/src/process/capture/FilesystemSnapshot.test.ts b/src/process/capture/FilesystemSnapshot.test.ts index 16799b9738..3aeab0391b 100644 --- a/src/process/capture/FilesystemSnapshot.test.ts +++ b/src/process/capture/FilesystemSnapshot.test.ts @@ -1,6 +1,8 @@ import { fstatSync } from "node:fs"; import { mkdtemp, + mkdir, + symlink, lstat, open, readdir, @@ -35,6 +37,27 @@ const countOpenDescriptorsFor = async (path: string): Promise => { return matches.filter(Boolean).length; }; +it.skipIf(process.platform === "win32")( + "omits a captured regular file replaced by a symlink", + async () => { + const directory = await mkdtemp(join(tmpdir(), "rea-snapshot-symlink-")); + const selected = join(directory, "selected"); + const replacement = join(directory, "replacement"); + try { + await writeFile(selected, "captured bytes"); + await writeFile(replacement, "unselected bytes"); + const expected = await lstat(selected); + await rm(selected); + await symlink(replacement, selected); + const outcome = await hashFile(selected, expected, 1_000); + expect(outcome.state).toBe("omitted"); + expect(outcome).not.toHaveProperty("sha256"); + } finally { + await rm(directory, { recursive: true, force: true }); + } + }, +); + it("does not hash a replacement file using the earlier path metadata", async () => { const directory = await mkdtemp(join(tmpdir(), "rea-snapshot-identity-")); const path = join(directory, "observed-file"); @@ -154,3 +177,43 @@ it.skipIf(process.platform === "win32")( } }, ); + +for (const variant of [ + "unchanged", + "case-alias", + "ancestor-junction", + "replaced-junction", +] as const) { + it.runIf(process.platform === "win32")( + `hashes only the captured Windows file through ${variant}`, + async () => { + const directory = await mkdtemp(join(tmpdir(), "rea-snapshot-windows-")); + try { + const original = join(directory, "original"); + const replacement = join(directory, "replacement"); + await mkdir(original); + await mkdir(replacement); + await writeFile(join(original, "TestFile.txt"), "captured bytes"); + await writeFile(join(replacement, "TestFile.txt"), "replaced bytes"); + const alias = join(directory, "alias"); + if (variant === "ancestor-junction" || variant === "replaced-junction") + await symlink(original, alias, "junction"); + const path = join( + variant.includes("junction") ? alias : original, + variant === "case-alias" ? "testfile.txt" : "TestFile.txt", + ); + const expected = await lstat(path); + if (variant === "replaced-junction") { + await rm(alias); + await symlink(replacement, alias, "junction"); + } + const result = await hashFile(path, expected, 1_000); + expect(result.state).toBe( + variant === "replaced-junction" ? "omitted" : "hashed", + ); + } finally { + await rm(directory, { recursive: true, force: true }); + } + }, + ); +} diff --git a/src/process/capture/FilesystemSnapshot.ts b/src/process/capture/FilesystemSnapshot.ts index 5740f65255..60bee8fd5b 100644 --- a/src/process/capture/FilesystemSnapshot.ts +++ b/src/process/capture/FilesystemSnapshot.ts @@ -1,7 +1,7 @@ import { createHash } from "node:crypto"; import { constants as fsConstants } from "node:fs"; import { lstat, open, opendir, readlink } from "node:fs/promises"; -import { dirname, join, relative, resolve } from "node:path"; +import { join, relative } from "node:path"; import type { FileHandle } from "node:fs/promises"; import type { ProcessScenario } from "../../domain/process/processScenario.js"; import type { FilesystemCoverage } from "../../domain/process/processCaptureCoverage.js"; @@ -47,12 +47,26 @@ const fileUnavailableSystemCode = (cause: unknown): string | undefined => { : undefined; }; -const lstatIfPresent = async (path: string): Promise => { +const lstatIfPresent = async ( + path: string, + previouslyObservedRoot = false, + signal?: AbortSignal, +): Promise => { + signal?.throwIfAborted(); try { - return await lstat(path); + const stats = await lstat(path); + signal?.throwIfAborted(); + return stats; } catch (cause: unknown) { - if (cause instanceof Error && "code" in cause && cause.code === "ENOENT") + if ( + cause instanceof Error && + "code" in cause && + (cause.code === "ENOENT" || + (previouslyObservedRoot && cause.code === "ENOTDIR")) + ) { + signal?.throwIfAborted(); return undefined; + } throw cause; } }; @@ -74,12 +88,14 @@ export const hashFile = async ( signal?: AbortSignal, ): Promise => { signal?.throwIfAborted(); + const noFollowAvailable = + typeof fsConstants.O_NOFOLLOW === "number" && fsConstants.O_NOFOLLOW !== 0; let handle: FileHandle; try { handle = await open( path, fsConstants.O_RDONLY | - (fsConstants.O_NOFOLLOW ?? 0) | + (noFollowAvailable ? fsConstants.O_NOFOLLOW : 0) | (fsConstants.O_NONBLOCK ?? 0), ); } catch (cause: unknown) { @@ -94,6 +110,18 @@ export const hashFile = async ( } try { signal?.throwIfAborted(); + // On hosts without O_NOFOLLOW, compare the leaf's identity with the + // captured regular file. Ancestor aliases do not change file identity. + if (!noFollowAvailable) { + const named = await lstat(path); + signal?.throwIfAborted(); + if (!hasSameFileState(expected, named)) + return { + state: "omitted", + reason: "file_changed_or_short_read", + system_code: null, + }; + } const stats = await handle.stat(); signal?.throwIfAborted(); if (!hasSameFileState(expected, stats) || stats.size > maxBytes) @@ -124,7 +152,13 @@ export const hashFile = async ( } const after = await handle.stat(); signal?.throwIfAborted(); - if (!hasSameFileState(stats, after)) + let unchanged = hasSameFileState(stats, after); + if (unchanged && !noFollowAvailable) { + const namedAfter = await lstat(path); + signal?.throwIfAborted(); + unchanged = hasSameFileState(after, namedAfter); + } + if (!unchanged) return { state: "omitted", reason: "file_changed_or_short_read", @@ -149,7 +183,11 @@ export const hashFile = async ( export const snapshotRoots = async ( scenario: ProcessScenario, signal?: AbortSignal, + previous?: ProcessFilesystemSnapshot, ): Promise => { + // A prior root entry establishes same-run admission. Preserve initial + // obstruction errors; a later non-directory ancestor makes it absent. + const previousPaths = new Set(previous?.files.map(({ path }) => path)); const entries: FileState[] = []; const completeRoots: string[] = []; let remainingBytes = scenario.limits.file_bytes; @@ -158,6 +196,7 @@ export const snapshotRoots = async ( const enumerationReasons = new Set< FilesystemCoverage["enumeration_reasons"][number] >(); + const enumerationOmissions: FilesystemCoverage["enumeration_omissions"] = []; const hashOmissions: FilesystemCoverage["hash_omissions"] = []; const visit = async ( root: string, @@ -177,11 +216,16 @@ export const snapshotRoots = async ( enumerationReasons.add("depth_limit"); return false; } - const stats = await lstatIfPresent(path); + const stats = await lstatIfPresent( + path, + previousPaths.has(`${rootAlias}:.`), + signal, + ); if (stats === undefined) return true; const relativePath = relative(root, path) || "."; if (stats.isSymbolicLink()) { - const target = resolve(dirname(path), await readlink(path)); + // Keep the target as stored: lexical resolution can merge distinct targets. + const target = await readlink(path); const afterRead = await lstat(path); if (!hasSameIdentity(stats, afterRead, "symlink")) { truncated = true; @@ -252,7 +296,23 @@ export const snapshotRoots = async ( let complete = true; const remainingEntries = scenario.limits.files - entries.length - pendingNames; - const directory = await opendir(path); + let directory; + try { + directory = await opendir(path); + } catch (cause: unknown) { + signal?.throwIfAborted(); + const systemCode = fileUnavailableSystemCode(cause); + if (systemCode === undefined) throw cause; + truncated = true; + complete = false; + enumerationReasons.add("directory_unavailable"); + enumerationOmissions.push({ + path: `${rootAlias}:${relativePath}`, + reason: "directory_unavailable", + system_code: systemCode, + }); + return false; + } for await (const child of directory) { signal?.throwIfAborted(); if ( @@ -289,7 +349,10 @@ export const snapshotRoots = async ( }; for (const [index, root] of scenario.filesystem_observation_paths.entries()) { const alias = `root_${String(index)}`; - if ((await lstatIfPresent(root)) === undefined) { + if ( + (await lstatIfPresent(root, previousPaths.has(`${alias}:.`), signal)) === + undefined + ) { completeRoots.push(alias); continue; } @@ -305,6 +368,7 @@ export const snapshotRoots = async ( enumeration_truncated: completeRoots.length !== scenario.filesystem_observation_paths.length, enumeration_reasons: [...enumerationReasons], + enumeration_omissions: enumerationOmissions, hash_budget_bytes: scenario.limits.file_bytes, hashed_bytes: scenario.limits.file_bytes - remainingBytes, hash_omissions: hashOmissions, diff --git a/src/process/capture/ProcessCaptureLifecycle.ts b/src/process/capture/ProcessCaptureLifecycle.ts index ab9271ddac..d7ab6afbf7 100644 --- a/src/process/capture/ProcessCaptureLifecycle.ts +++ b/src/process/capture/ProcessCaptureLifecycle.ts @@ -38,12 +38,13 @@ import { assertNotCancelled } from "./ProcessScenarioRuntimeValidation.js"; import { normalizeProcessElapsedTime, normalizeProcessSamples, - normalizeProcessText, + type BoundedProcessTextNormalization, } from "./ProcessNormalization.js"; import { PROCESS_PROVIDER } from "../../domain/process/processEvidenceProvider.js"; import { TerminalRenderer } from "./TerminalRenderer.js"; import { hasCaptureTruncation, + type NormalizedTerminalRetention, type TerminalRetention, type ProcessCaptureTruncationDetails, } from "../../domain/process/processCaptureCoverage.js"; @@ -409,7 +410,9 @@ const captureCoverageUnknowns = ( unknowns.push({ scope: "terminal", reason: - "Raw PTY chunks exceeded output_bytes and were omitted; rendered states also lack those inputs. See truncation_details.raw_terminal.", + (details.normalized_terminal?.omitted_frames ?? 0) > 0 + ? "Raw PTY chunks were omitted because original input exceeded output_bytes or an ordered normalization stage exceeded its remaining bound; final normalized sizes for the latter omissions are unknown. Rendered states also lack those inputs. See raw_terminal and normalized_terminal retention details, including the normalized omission count." + : "Raw PTY chunks exceeded output_bytes and were omitted; rendered states also lack those inputs. See truncation_details.raw_terminal.", }); if ( details.rendered_terminal.observed_frames > @@ -418,7 +421,9 @@ const captureCoverageUnknowns = ( unknowns.push({ scope: "terminal", reason: - "Cumulative rendered state and visible-line bytes exceeded output_bytes; whole rendered frames were omitted. Raw PTY coverage is reported separately. See truncation_details.rendered_terminal.", + (details.rendered_terminal.unknown_size_frames ?? 0) > 0 + ? "Rendered frames were omitted because a normalization stage exceeded the remaining output_bytes budget; final rendered byte totals are unknown for those frames. See truncation_details.rendered_terminal." + : "Cumulative rendered state and visible-line bytes exceeded output_bytes; whole rendered frames were omitted. Raw PTY coverage is reported separately. See truncation_details.rendered_terminal.", }); for (const [name, coverage] of [ ["before", details.filesystem_before], @@ -1302,52 +1307,70 @@ export const captureTerminalFrames = (options: { readonly scenario: ProcessScenario; readonly frames: TerminalFrame[]; readonly started: number; - readonly temporaryRoot: string; readonly onOutput: () => void; readonly renderer: TerminalRenderer; + readonly normalizeText: ( + value: string, + maxBytes: number, + ) => BoundedProcessTextNormalization; readonly recordEvent: RecordProcessCaptureEvent; -}): (() => TerminalRetention) => { - let outputBytes = 0; +}): (() => TerminalCaptureRetention) => { + let rawBytes = 0; + let normalizedBytes = 0; let observedBytes = 0; let observedFrames = 0; + let normalizedOmittedFrames = 0; options.terminal.onData((data) => { options.onOutput(); const bytes = Buffer.byteLength(data); observedBytes += bytes; observedFrames += 1; - if (outputBytes + bytes > options.scenario.limits.output_bytes) { + const budget = options.scenario.limits.output_bytes; + if (bytes > budget - rawBytes) { return; } - outputBytes += bytes; + const normalized = options.normalizeText(data, budget - normalizedBytes); + if (normalized.state === "over_budget") { + normalizedOmittedFrames += 1; + return; + } + rawBytes += bytes; + normalizedBytes += normalized.utf8Bytes; const atMs = normalizeProcessElapsedTime( monotonicProcessCaptureClock() - options.started, options.scenario.normalization.time_bucket_ms, ); - const normalized = normalizeProcessText( - data, - options.scenario, - options.temporaryRoot, - options.terminal.pid, - ); const sequence = options.frames.length; options.frames.push({ sequence, at_ms: atMs, - data: normalized, - ...(normalized === data ? {} : { raw_data: data }), + data: normalized.value, + ...(normalized.value === data ? {} : { raw_data: data }), }); options.renderer.write(data, atMs); options.recordEvent("frames", sequence); }); return () => ({ - budget_bytes: options.scenario.limits.output_bytes, - observed_bytes: observedBytes, - retained_bytes: outputBytes, - observed_frames: observedFrames, - retained_frames: options.frames.length, + raw_terminal: { + budget_bytes: options.scenario.limits.output_bytes, + observed_bytes: observedBytes, + retained_bytes: rawBytes, + observed_frames: observedFrames, + retained_frames: options.frames.length, + }, + normalized_terminal: { + budget_bytes: options.scenario.limits.output_bytes, + retained_bytes: normalizedBytes, + omitted_frames: normalizedOmittedFrames, + }, }); }; +export interface TerminalCaptureRetention { + readonly raw_terminal: TerminalRetention; + readonly normalized_terminal: NormalizedTerminalRetention; +} + export const resolveProcessResult = ( capture: PendingProcessCapture | ProcessCapture | undefined, executionFailure: unknown, diff --git a/src/process/capture/ProcessCaptureObservation.test.ts b/src/process/capture/ProcessCaptureObservation.test.ts index ddf0ffd428..51ebe20fa9 100644 --- a/src/process/capture/ProcessCaptureObservation.test.ts +++ b/src/process/capture/ProcessCaptureObservation.test.ts @@ -66,7 +66,12 @@ it("returns detached terminal observations", async () => { rows: 12, scrollback: 100, maxBytes: 100_000, - normalize: (value) => value, + normalize: (value, maxBytes) => { + const utf8Bytes = Buffer.byteLength(value); + return utf8Bytes <= maxBytes + ? { state: "normalized", value, utf8Bytes } + : { state: "over_budget" }; + }, recordEvent: (collection, index) => observed.push(`${collection}:${String(index)}`), }); diff --git a/src/process/capture/ProcessCaptureProgress.test.ts b/src/process/capture/ProcessCaptureProgress.test.ts index 69349bef9e..d3cc98b2c1 100644 --- a/src/process/capture/ProcessCaptureProgress.test.ts +++ b/src/process/capture/ProcessCaptureProgress.test.ts @@ -110,4 +110,31 @@ it("still delivers terminal status after an observer rejects an update", async ( completed: 2, terminal: true, }); + expect(delivered.at(-1)?.message).toContain( + "owned_process_group=unverified terminal_renderer=unverified temporary_root=unverified", + ); +}); + +it("preserves known cleanup states in terminal progress", async () => { + const delivered: Parameters[0][] = []; + const tracker = createProcessCaptureProgressTracker({ + async report(update) { + delivered.push(update); + }, + }); + await tracker.report({ + phase: "cleanup", + counts: { frames: 0, samples: 0, interactions: 0 }, + disposition: "failed", + terminal: true, + cleanup: { + owned_process_group: { state: "not_required", reason: null }, + terminal_renderer: { state: "not_required", reason: null }, + temporary_root: { state: "cleaned", reason: null }, + }, + }); + + expect(delivered[0]?.message).toContain( + "owned_process_group=not_required terminal_renderer=not_required temporary_root=cleaned", + ); }); diff --git a/src/process/capture/ProcessCaptureProgress.ts b/src/process/capture/ProcessCaptureProgress.ts index 205334db78..5cbe34cdc2 100644 --- a/src/process/capture/ProcessCaptureProgress.ts +++ b/src/process/capture/ProcessCaptureProgress.ts @@ -53,9 +53,11 @@ const formatProgressMessage = ( const dispositionText = disposition === undefined ? "" : ` disposition=${disposition}`; const cleanupText = - cleanup === undefined - ? "" - : ` owned_process_group=${cleanup.owned_process_group.state} terminal_renderer=${cleanup.terminal_renderer.state} temporary_root=${cleanup.temporary_root.state}`; + cleanup !== undefined + ? ` owned_process_group=${cleanup.owned_process_group.state} terminal_renderer=${cleanup.terminal_renderer.state} temporary_root=${cleanup.temporary_root.state}` + : disposition === undefined + ? "" + : " owned_process_group=unverified terminal_renderer=unverified temporary_root=unverified"; return `${status}${dispositionText}${cleanupText}`; }; diff --git a/src/process/capture/ProcessCaptureTerminalRetention.test.ts b/src/process/capture/ProcessCaptureTerminalRetention.test.ts new file mode 100644 index 0000000000..e3bde2264c --- /dev/null +++ b/src/process/capture/ProcessCaptureTerminalRetention.test.ts @@ -0,0 +1,63 @@ +import { expect, it } from "vitest"; +import type { IPty } from "@lydell/node-pty"; + +import { parseProcessScenario } from "../../domain/process/processScenario.js"; +import type { TerminalFrame } from "../../domain/process/processCapture.js"; +import { captureTerminalFrames } from "./ProcessCaptureLifecycle.js"; +import { createBoundedProcessTextNormalizer } from "./ProcessNormalization.js"; + +it("omits an expanding chunk whole and retains a later smaller normalized chunk", () => { + let onData: ((data: string) => void) | undefined; + const terminal = { + onData: (listener: (data: string) => void) => { + onData = listener; + return { dispose: () => undefined }; + }, + } as unknown as IPty; + const scenario = parseProcessScenario({ + executable: "unused", + limits: { output_bytes: 1_000 }, + normalization: { + paths: false, + pids: false, + ports: false, + patterns: [{ pattern: "a", replacement: "x".repeat(500) }], + }, + }); + const frames: TerminalFrame[] = []; + const renderedChunks: string[] = []; + const journal: Array<[string, number]> = []; + const captureRetention = captureTerminalFrames({ + terminal, + scenario, + frames, + started: 0, + onOutput: () => undefined, + renderer: { write: (data: string) => renderedChunks.push(data) } as never, + normalizeText: createBoundedProcessTextNormalizer(scenario, "/tmp", -1), + recordEvent: (collection, index) => journal.push([collection, index]), + }); + + onData?.("aaa"); + onData?.("z"); + + expect(frames).toEqual([ + { sequence: 0, at_ms: expect.any(Number), data: "z" }, + ]); + expect(renderedChunks).toEqual(["z"]); + expect(journal).toEqual([["frames", 0]]); + expect(captureRetention()).toEqual({ + raw_terminal: { + budget_bytes: 1_000, + observed_bytes: 4, + retained_bytes: 1, + observed_frames: 2, + retained_frames: 1, + }, + normalized_terminal: { + budget_bytes: 1_000, + retained_bytes: 1, + omitted_frames: 1, + }, + }); +}); diff --git a/src/process/capture/ProcessHarness.ts b/src/process/capture/ProcessHarness.ts index 847e512f8b..f411718aab 100644 --- a/src/process/capture/ProcessHarness.ts +++ b/src/process/capture/ProcessHarness.ts @@ -1,5 +1,4 @@ import type { IPty } from "@lydell/node-pty"; -import type { TerminalRetention } from "../../domain/process/processCaptureCoverage.js"; import type { ProcessCapture } from "../../domain/process/processCaptureParsing.js"; import type { ProcessScenario } from "../../domain/process/processScenario.js"; import { parseProcessCapture } from "../../domain/process/processCaptureParsing.js"; @@ -24,7 +23,10 @@ import { startProcessSampler } from "./ProcessSampling.js"; import { snapshotRoots } from "./FilesystemSnapshot.js"; import { TerminalRenderer } from "./TerminalRenderer.js"; import type { ProcessTimer } from "./ProcessTimer.js"; -import { normalizeProcessText } from "./ProcessNormalization.js"; +import { + createBoundedProcessTextNormalizer, + type BoundedProcessTextNormalization, +} from "./ProcessNormalization.js"; import { selectCapturedProcessGroupIds } from "../ProcessOwnershipProcessTree.js"; import { awaitTerminalExit, @@ -46,6 +48,7 @@ import { describeUnverifiedCause, type ProcessCaptureResourceCleanupOptions, ProcessCaptureResourceScope, + type TerminalCaptureRetention, } from "./ProcessCaptureLifecycle.js"; import { assertNotCancelled, @@ -79,7 +82,7 @@ interface StartedCaptureRuntime { readonly startedAt: Date; readonly executableIdentity: ReturnType; readonly lastOutput: () => number; - readonly rawTerminalRetention: () => TerminalRetention; + readonly terminalRetention: () => TerminalCaptureRetention; readonly stopSampler: ReturnType; /** Signals the captured root only while its launch-time start identity still matches. */ readonly signalRoot: (signal: "SIGTERM" | "SIGKILL") => Promise<{ @@ -222,8 +225,10 @@ const cleanupFailedStartup = async (options: { const createTerminalRenderer = ( scenario: ProcessScenario, - temporaryRoot: string, - terminalPid: () => number, + normalizeText: ( + value: string, + maxBytes: number, + ) => BoundedProcessTextNormalization, recordEvent: RecordProcessCaptureEvent, ): TerminalRenderer => new TerminalRenderer({ @@ -231,8 +236,7 @@ const createTerminalRenderer = ( rows: scenario.terminal.rows, scrollback: scenario.terminal.scrollback, maxBytes: scenario.limits.output_bytes, - normalize: (value) => - normalizeProcessText(value, scenario, temporaryRoot, terminalPid()), + normalize: normalizeText, recordEvent, }); @@ -271,10 +275,14 @@ const startCaptureRuntime = async ( const startedAt = new Date(); const started = monotonicProcessCaptureClock(); let lastOutput = started; - renderer = createTerminalRenderer( + const normalizeText = createBoundedProcessTextNormalizer( scenario, options.temporaryRoot, () => terminal?.pid ?? -1, + ); + renderer = createTerminalRenderer( + scenario, + normalizeText, options.recordEvent, ); terminal = spawn(scenario.executable, [...scenario.arguments], { @@ -293,12 +301,13 @@ const startCaptureRuntime = async ( scenario.executable, selectedExecutable, ); - const rawTerminalRetention = captureTerminalFrames({ + const terminalRetention = captureTerminalFrames({ ...options, terminal, started, onOutput: () => (lastOutput = monotonicProcessCaptureClock()), renderer, + normalizeText, }); scheduleScenarioInteractions({ ...options, @@ -321,7 +330,7 @@ const startCaptureRuntime = async ( startedAt, executableIdentity, lastOutput: () => lastOutput, - rawTerminalRetention, + terminalRetention, stopSampler, signalRoot: retainRootSignaller( terminal.pid, @@ -499,7 +508,11 @@ const completeCapture = async (options: { assertNotCancelled(options.signal); let after: Awaited>; try { - after = await options.captureSnapshot(scenario, options.signal); + after = await options.captureSnapshot( + scenario, + options.signal, + options.before, + ); } catch (cause: unknown) { options.observationBuffer.filesystem_snapshots = { ...options.observationBuffer.filesystem_snapshots, @@ -572,7 +585,7 @@ const completeCapture = async (options: { before: options.before, after, truncationDetails: { - raw_terminal: runtime.rawTerminalRetention(), + ...runtime.terminalRetention(), rendered_terminal: runtime.renderer.retention(), filesystem_before: options.before.coverage, filesystem_after: after.coverage, @@ -659,6 +672,10 @@ const runProcessScenario = async ( counts: progressCounts(), terminal: true, disposition: signal?.aborted === true ? "cancelled" : "failed", + ...(cause instanceof ProcessCaptureError && + cause.cleanupReport !== undefined + ? { cleanup: cause.cleanupReport } + : {}), }); throw cause; } diff --git a/src/process/capture/ProcessNormalization.test.ts b/src/process/capture/ProcessNormalization.test.ts index e5ef61062d..e23ced1ca9 100644 --- a/src/process/capture/ProcessNormalization.test.ts +++ b/src/process/capture/ProcessNormalization.test.ts @@ -1,6 +1,8 @@ import { expect, it } from "vitest"; import { + createBoundedProcessTextNormalizer, + createProcessTextNormalizer, normalizeProcessSamples, normalizeProcessText, } from "./ProcessNormalization.js"; @@ -94,6 +96,214 @@ it("replaces caller-declared patterns literally, without expanding replacement p ).toBe("listening on $&$&; $$; $1=alpha; tail $1"); }); +it("bounds ordered normalization before replacement expansion and preserves later smaller chunks", () => { + const normalize = createBoundedProcessTextNormalizer( + scenario({ + paths: false, + ports: false, + pids: false, + patterns: [{ pattern: "a", replacement: "x".repeat(500) }], + }), + "/temporary", + rootPid, + ); + + expect(normalize("a".repeat(100), 1_000)).toEqual({ state: "over_budget" }); + expect(normalize("z", 1_000)).toEqual({ + state: "normalized", + value: "z", + utf8Bytes: 1, + }); +}); + +it("keeps ordered intermediate expansion explicit even when a later rule could shrink it", () => { + const normalize = createBoundedProcessTextNormalizer( + scenario({ + paths: false, + ports: false, + pids: false, + patterns: [ + { pattern: "a", replacement: "x".repeat(20) }, + { pattern: "x", replacement: "y" }, + ], + }), + "/temporary", + rootPid, + ); + + expect(normalize("a", 10)).toEqual({ state: "over_budget" }); +}); + +it("counts UTF-8 bytes across replacement boundaries like the final JavaScript string", () => { + const splitPair = scenario({ + paths: false, + ports: false, + pids: false, + patterns: [{ pattern: "\ude00", replacement: "\ude01" }], + }); + const splitPairResult = createBoundedProcessTextNormalizer( + splitPair, + "/temporary", + rootPid, + )("😀", 4); + expect(splitPairResult).toEqual({ + state: "normalized", + value: normalizeProcessText("😀", splitPair, "/temporary", rootPid), + utf8Bytes: 4, + }); + expect( + splitPairResult.state === "normalized" && splitPairResult.utf8Bytes, + ).toBe( + Buffer.byteLength( + normalizeProcessText("😀", splitPair, "/temporary", rootPid), + ), + ); + + const joinedPair = scenario({ + paths: false, + ports: false, + pids: false, + patterns: [{ pattern: "\ude00\ud83d", replacement: "" }], + }); + const joinedInput = "😀😀"; + const joinedExpected = normalizeProcessText( + joinedInput, + joinedPair, + "/temporary", + rootPid, + ); + expect(joinedExpected).toBe("😀"); + expect( + createBoundedProcessTextNormalizer( + joinedPair, + "/temporary", + rootPid, + )(joinedInput, Buffer.byteLength(joinedExpected)), + ).toEqual({ + state: "normalized", + value: joinedExpected, + utf8Bytes: Buffer.byteLength(joinedExpected), + }); + + const emptyPattern = scenario({ + paths: false, + ports: false, + pids: false, + patterns: [{ pattern: "", replacement: "\ude00" }], + }); + const input = "\ud83d"; + const expected = normalizeProcessText( + input, + emptyPattern, + "/temporary", + rootPid, + ); + expect( + createBoundedProcessTextNormalizer( + emptyPattern, + "/temporary", + rootPid, + )(input, Buffer.byteLength(expected)), + ).toEqual({ + state: "normalized", + value: expected, + utf8Bytes: Buffer.byteLength(expected), + }); + expect( + createBoundedProcessTextNormalizer( + emptyPattern, + "/temporary", + rootPid, + )(input, Buffer.byteLength(expected) - 1), + ).toEqual({ state: "over_budget" }); + + const loneSurrogate = scenario({ + paths: false, + ports: false, + pids: false, + patterns: [{ pattern: "x", replacement: "y" }], + }); + const loneInput = "\ud83dx"; + const loneExpected = normalizeProcessText( + loneInput, + loneSurrogate, + "/temporary", + rootPid, + ); + expect( + createBoundedProcessTextNormalizer( + loneSurrogate, + "/temporary", + rootPid, + )(loneInput, Buffer.byteLength(loneExpected)), + ).toEqual({ + state: "normalized", + value: loneExpected, + utf8Bytes: Buffer.byteLength(loneExpected), + }); +}); + +it("lets a later pattern shrink text when an earlier pattern does not match", () => { + const declared = scenario({ + paths: false, + ports: false, + pids: false, + patterns: [ + { pattern: "z", replacement: "x" }, + { pattern: "aa", replacement: "b" }, + ], + }); + const expected = normalizeProcessText("aa", declared, "/temporary", rootPid); + expect(expected).toBe("b"); + expect( + createBoundedProcessTextNormalizer( + declared, + "/temporary", + rootPid, + )("aa", Buffer.byteLength(expected)), + ).toEqual({ + state: "normalized", + value: expected, + utf8Bytes: Buffer.byteLength(expected), + }); +}); + +it("reuses matchers while observing the terminal PID after launch", () => { + let pid = -1; + const normalize = createProcessTextNormalizer( + scenario({ ports: false, patterns: [] }), + "/temporary", + () => pid, + ); + expect(normalize("root 41010 /workspace/item")).toBe( + "root 41010 /item", + ); + pid = 41_010; + expect(normalize("root 41010 /workspace/item")).toBe( + "root /item", + ); + pid = 42_020; + expect(normalize("root 41010 peer 42020")).toBe("root 41010 peer "); +}); + +it("keeps equal-path priority and replacement aliases local to each capture", () => { + const declared = parseProcessScenario({ + executable: "/selected/tool", + working_directory: "/selected", + filesystem_observation_paths: ["/selected", ""], + normalization: { paths: true, pids: false, ports: false, patterns: [] }, + }); + const normalize = createProcessTextNormalizer(declared, "/selected", -1); + const other = createProcessTextNormalizer(declared, "/other", -1); + const text = "/selected/item /source"; + expect(normalize(text)).toBe( + "/item /source", + ); + expect(other(text)).toBe( + "/item /source", + ); +}); + it("preserves empty-pattern boundaries, including empty text", () => { const declared = scenario({ patterns: [{ pattern: "", replacement: "$$" }], diff --git a/src/process/capture/ProcessNormalization.ts b/src/process/capture/ProcessNormalization.ts index 74db31b1c7..1f5402993e 100644 --- a/src/process/capture/ProcessNormalization.ts +++ b/src/process/capture/ProcessNormalization.ts @@ -3,42 +3,296 @@ import { isIP } from "node:net"; import type { ProcessScenario } from "../../domain/process/processScenario.js"; import type { ProcessSample } from "../../domain/process/processCapture.js"; + +export type BoundedProcessTextNormalization = + | { + readonly state: "normalized"; + readonly value: string; + readonly utf8Bytes: number; + } + | { readonly state: "over_budget" }; + +type RegexReplacement = (match: string, ...groups: string[]) => string; + +type BoundedStageResult = + | { + readonly state: "bounded"; + readonly value: string; + readonly utf8Bytes: number; + } + | { readonly state: "unchanged"; readonly value: string } + | { readonly state: "over_budget" }; + +const createUtf8ByteCounter = (maxBytes: number) => { + let bytes = 0; + let pendingHighSurrogate = false; + const addBytes = (added: number): void => { + if (bytes > maxBytes || added > maxBytes - bytes) { + bytes = Number.POSITIVE_INFINITY; + return; + } + bytes += added; + }; + return { + add: (value: string): void => { + if (bytes > maxBytes || value.length === 0) return; + let index = 0; + if (pendingHighSurrogate) { + if (value.length > 0 && isLowSurrogate(value.charCodeAt(0))) { + addBytes(4); + index = 1; + } else { + addBytes(3); + } + pendingHighSurrogate = false; + } + for (; index < value.length; index += 1) { + const code = value.charCodeAt(index); + if (code <= 0x7f) addBytes(1); + else if (code <= 0x7ff) addBytes(2); + else if (isHighSurrogate(code)) { + if (index + 1 === value.length) { + pendingHighSurrogate = true; + } else if (isLowSurrogate(value.charCodeAt(index + 1))) { + addBytes(4); + index += 1; + } else addBytes(3); + } else addBytes(3); + if (bytes > maxBytes) return; + } + }, + finish: (): number => { + if (pendingHighSurrogate) addBytes(3); + return bytes; + }, + }; +}; + +const isHighSurrogate = (code: number): boolean => + code >= 0xd800 && code <= 0xdbff; +const isLowSurrogate = (code: number): boolean => + code >= 0xdc00 && code <= 0xdfff; + +const boundedRegexReplace = ( + value: string, + matcher: RegExp, + replacement: RegexReplacement, + maxBytes: number, +): BoundedStageResult => { + const byteCounter = createUtf8ByteCounter(maxBytes); + let cursor = 0; + let changed = false; + matcher.lastIndex = 0; + let match: RegExpExecArray | null; + while ((match = matcher.exec(value)) !== null) { + const index = match.index; + const matched = match[0] ?? ""; + const nextCursor = index + matched.length; + const replacementValue = replacement(matched, ...match.slice(1)); + changed ||= replacementValue !== matched; + byteCounter.add(value.slice(cursor, index)); + byteCounter.add(replacementValue); + cursor = nextCursor; + } + byteCounter.add(value.slice(cursor)); + const bytes = byteCounter.finish(); + if (bytes > maxBytes) { + return changed ? { state: "over_budget" } : { state: "unchanged", value }; + } + if (!changed) return { state: "unchanged", value }; + matcher.lastIndex = 0; + const normalized = value.replace(matcher, replacement); + return { + state: "bounded", + value: normalized, + utf8Bytes: bytes, + }; +}; + +const boundedLiteralReplace = ( + value: string, + pattern: string, + replacement: string, + maxBytes: number, +): BoundedStageResult => { + const byteCounter = createUtf8ByteCounter(maxBytes); + let cursor = 0; + if (pattern === "") { + if (replacement === "") return { state: "unchanged", value }; + for (let index = 0; index < value.length; index += 1) { + byteCounter.add(replacement); + byteCounter.add(value.slice(index, index + 1)); + } + byteCounter.add(replacement); + const bytes = byteCounter.finish(); + if (bytes > maxBytes) return { state: "over_budget" }; + return { + state: "bounded", + value: value.replaceAll(pattern, () => replacement), + utf8Bytes: bytes, + }; + } + if (pattern === replacement) return { state: "unchanged", value }; + + let index = value.indexOf(pattern, cursor); + if (index === -1) return { state: "unchanged", value }; + while (index !== -1) { + byteCounter.add(value.slice(cursor, index)); + byteCounter.add(replacement); + cursor = index + pattern.length; + index = value.indexOf(pattern, cursor); + } + byteCounter.add(value.slice(cursor)); + const bytes = byteCounter.finish(); + if (bytes > maxBytes) return { state: "over_budget" }; + return { + state: "bounded", + value: value.replaceAll(pattern, () => replacement), + utf8Bytes: bytes, + }; +}; + +const boundedIdentity = ( + value: string, + maxBytes: number, +): BoundedStageResult => { + const utf8Bytes = Buffer.byteLength(value); + return utf8Bytes <= maxBytes + ? { state: "bounded", value, utf8Bytes } + : { state: "unchanged", value }; +}; + /** Bucket one elapsed process-capture timestamp under scenario normalization. */ export const normalizeProcessElapsedTime = ( elapsedMs: number, timeBucketMs: number, ): number => Math.floor(elapsedMs / timeBucketMs) * timeBucketMs; -/** Normalize one terminal/protocol payload under explicitly selected scenario rules. */ +/** Prepare capture-local matchers while retaining a live terminal PID lookup. */ +export const createProcessTextNormalizer = ( + scenario: ProcessScenario, + temporaryRoot: string, + pid: number | (() => number), +): ((value: string) => string) => { + const normalizePaths = scenario.normalization.paths + ? createPathNormalizer(scenario, temporaryRoot) + : (value: string): string => value; + let previousPid: number | undefined; + let normalizePids = (value: string): string => value; + return (value) => { + let normalized = normalizePaths(value); + if (scenario.normalization.ports) + normalized = normalizePortTokens(normalized); + if (scenario.normalization.pids) { + const currentPid = typeof pid === "number" ? pid : pid(); + if (currentPid !== previousPid) { + previousPid = currentPid; + normalizePids = createPidNormalizer([currentPid]); + } + normalized = normalizePids(normalized); + } + for (const pattern of scenario.normalization.patterns) + normalized = normalized.replaceAll( + pattern.pattern, + () => pattern.replacement, + ); + return normalized; + }; +}; + +/** Normalize terminal text without materializing an intermediate over its byte budget. */ +export const createBoundedProcessTextNormalizer = ( + scenario: ProcessScenario, + temporaryRoot: string, + pid: number | (() => number), +): ((value: string, maxBytes: number) => BoundedProcessTextNormalization) => { + const pathRule = scenario.normalization.paths + ? createPathRule(scenario, temporaryRoot) + : undefined; + let previousPid: number | undefined; + let pidMatcher: RegExp | undefined; + return (value, maxBytes) => { + let normalized = value; + const apply = (result: BoundedStageResult): boolean => { + if (result.state === "over_budget") return false; + if (result.state === "bounded") normalized = result.value; + return true; + }; + if (pathRule !== undefined) { + if ( + !apply( + boundedRegexReplace( + normalized, + pathRule.matcher, + pathRule.replace, + maxBytes, + ), + ) + ) + return { state: "over_budget" }; + } + if (scenario.normalization.ports) { + if (!apply(normalizePortsBounded(normalized, maxBytes))) + return { state: "over_budget" }; + } + if (scenario.normalization.pids) { + const currentPid = typeof pid === "number" ? pid : pid(); + if (currentPid !== previousPid) { + previousPid = currentPid; + pidMatcher = createPidMatcher([currentPid]); + } + if (pidMatcher !== undefined) { + if ( + !apply( + boundedRegexReplace( + normalized, + pidMatcher, + () => "", + maxBytes, + ), + ) + ) + return { state: "over_budget" }; + } + } + for (const pattern of scenario.normalization.patterns) { + if ( + !apply( + boundedLiteralReplace( + normalized, + pattern.pattern, + pattern.replacement, + maxBytes, + ), + ) + ) + return { state: "over_budget" }; + } + return boundedIdentity(normalized, maxBytes).state === "bounded" + ? { + state: "normalized", + value: normalized, + utf8Bytes: Buffer.byteLength(normalized), + } + : { state: "over_budget" }; + }; +}; + +/** Normalize one payload when its caller does not retain a capture-local matcher. */ export const normalizeProcessText = ( value: string, scenario: ProcessScenario, temporaryRoot: string, pid: number, -): string => { - let normalized = value; - if (scenario.normalization.paths) - normalized = normalizePathTokens(normalized, scenario, temporaryRoot); - if (scenario.normalization.ports) - normalized = normalizePortTokens(normalized); - if (scenario.normalization.pids) - normalized = normalizePidTokens(normalized, [pid]); - // A callback keeps replacement text literal while preserving replaceAll's - // empty-pattern boundaries, which the process scenario contract permits. - for (const pattern of scenario.normalization.patterns) - normalized = normalized.replaceAll( - pattern.pattern, - () => pattern.replacement, - ); - return normalized; -}; +): string => createProcessTextNormalizer(scenario, temporaryRoot, pid)(value); /** Match original paths once, preferring the most specific selected identity. */ -const normalizePathTokens = ( - value: string, +const createPathRule = ( scenario: ProcessScenario, temporaryRoot: string, -): string => { +): + | { readonly matcher: RegExp; readonly replace: RegexReplacement } + | undefined => { const aliases = new Map(); const selected: readonly (readonly [string, string])[] = [ [temporaryRoot, ""], @@ -56,14 +310,24 @@ const normalizePathTokens = ( const paths = [...aliases.keys()].sort( (left, right) => right.length - left.length, ); - if (paths.length === 0) return value; + if (paths.length === 0) return undefined; const pattern = paths .map((path) => path.replace(/[.*+?^${}()|[\]\\]/gu, "\\$&")) .join("|"); - return value.replace( - new RegExp(pattern, "gu"), - (path) => aliases.get(path) ?? path, - ); + const matcher = new RegExp(pattern, "gu"); + return { + matcher, + replace: (path) => aliases.get(path) ?? path, + }; +}; + +const createPathNormalizer = ( + scenario: ProcessScenario, + temporaryRoot: string, +): ((value: string) => string) => { + const rule = createPathRule(scenario, temporaryRoot); + if (rule === undefined) return (value) => value; + return (value) => value.replace(rule.matcher, rule.replace); }; // A port may end a sentence, but a period followed by a letter, digit, or @@ -72,21 +336,45 @@ const normalizePathTokens = ( // PTY chunks are normalized independently, so a period that ends the chunk // stays unnormalized: its continuation may arrive in the next chunk. const normalizePortTokens = (value: string): string => { - const endpointNormalized = value.replaceAll( - /(\b[a-z][a-z0-9+.-]*:\/\/[^\s/?#"'<>]+|\blocalhost|\b(?:\d{1,3}\.){3}\d{1,3}|\[[\da-f:.%]+\]):(\d{1,5})(?![\p{L}\p{N}_@:]|\.(?:[\p{L}\p{N}_]|$))/giu, - (match: string, endpoint: string, port: string) => { - if (Number(port) > 65_535) return match; - const isEndpoint = endpoint.includes("://") - ? URL.canParse(`${endpoint}:${port}`) - : endpoint.toLowerCase() === "localhost" || - isIP(endpoint.replace(/^\[|\]$/gu, "")) !== 0; - return isEndpoint ? `${endpoint}:` : match; - }, + return value + .replaceAll(PORT_ENDPOINT_MATCHER, portEndpointReplacement) + .replaceAll(PORT_FIELD_MATCHER, portFieldReplacement); +}; + +const PORT_ENDPOINT_MATCHER = + /(\b[a-z][a-z0-9+.-]*:\/\/[^\s/?#"'<>]+|\blocalhost|\b(?:\d{1,3}\.){3}\d{1,3}|\[[\da-f:.%]+\]):(\d{1,5})(?![\p{L}\p{N}_@:]|\.(?:[\p{L}\p{N}_]|$))/giu; +const PORT_FIELD_MATCHER = + /(\b(?:port|tcp_port|udp_port|listen)\b["']?\s*[:=]\s*)(\d{1,5})(?![\p{L}\p{N}_]|\.(?:[\p{L}\p{N}_]|$))/giu; + +const portEndpointReplacement: RegexReplacement = (match, endpoint, port) => { + if (Number(port) > 65_535) return match; + const isEndpoint = endpoint.includes("://") + ? URL.canParse(`${endpoint}:${port}`) + : endpoint.toLowerCase() === "localhost" || + isIP(endpoint.replace(/^\[|\]$/gu, "")) !== 0; + return isEndpoint ? `${endpoint}:` : match; +}; + +const portFieldReplacement: RegexReplacement = (match, prefix, port) => + Number(port) <= 65_535 ? `${prefix}` : match; + +const normalizePortsBounded = ( + value: string, + maxBytes: number, +): BoundedStageResult => { + const endpoint = boundedRegexReplace( + value, + PORT_ENDPOINT_MATCHER, + portEndpointReplacement, + maxBytes, ); - return endpointNormalized.replaceAll( - /(\b(?:port|tcp_port|udp_port|listen)\b["']?\s*[:=]\s*)(\d{1,5})(?![\p{L}\p{N}_]|\.(?:[\p{L}\p{N}_]|$))/giu, - (match: string, prefix: string, port: string) => - Number(port) <= 65_535 ? `${prefix}` : match, + if (endpoint.state === "over_budget") return endpoint; + const afterEndpoint = endpoint.state === "bounded" ? endpoint.value : value; + return boundedRegexReplace( + afterEndpoint, + PORT_FIELD_MATCHER, + portFieldReplacement, + maxBytes, ); }; @@ -109,16 +397,17 @@ export const normalizeProcessSamples = ( for (const identifier of identifiers) if (identifier > 0 && !mapping.has(identifier)) mapping.set(identifier, mapping.size + 1); + const normalizeText = createProcessTextNormalizer( + scenario, + "", + rootPid, + ); + const normalizeSamplePids = scenario.normalization.pids + ? createPidNormalizer(identifiers) + : undefined; const normalizeCommand = (command: string): string => { - const normalized = normalizeProcessText( - command, - scenario, - "", - rootPid, - ); - const withNormalizedPids = scenario.normalization.pids - ? normalizePidTokens(normalized, identifiers) - : normalized; + const normalized = normalizeText(command); + const withNormalizedPids = normalizeSamplePids?.(normalized) ?? normalized; return withNormalizedPids.trim(); }; return samples.map((sample) => ({ @@ -144,17 +433,21 @@ export const normalizeProcessSamples = ( })); }; -const normalizePidTokens = ( - value: string, +const createPidNormalizer = ( identifiers: readonly number[], -): string => { +): ((value: string) => string) => { + const matcher = createPidMatcher(identifiers); + if (matcher === undefined) return (value) => value; + return (value) => value.replace(matcher, ""); +}; + +const createPidMatcher = ( + identifiers: readonly number[], +): RegExp | undefined => { const tokens = [...new Set(identifiers)] .filter((identifier) => Number.isSafeInteger(identifier) && identifier > 0) .sort((left, right) => right - left) .map(String); - if (tokens.length === 0) return value; - return value.replace( - new RegExp(`(?", - ); + if (tokens.length === 0) return undefined; + return new RegExp(`(? string, + ): ((value: string, maxBytes: number) => BoundedProcessTextNormalization) => + (value, maxBytes) => { + const normalized = normalize(value); + const utf8Bytes = Buffer.byteLength(normalized); + return utf8Bytes <= maxBytes + ? { state: "normalized", value: normalized, utf8Bytes } + : { state: "over_budget" }; + }; + const renderFrames = async ( data: string, normalize: (value: string) => string = (value) => value, @@ -22,7 +37,7 @@ const renderFrames = async ( rows: 4, scrollback: 10, maxBytes: 100_000, - normalize, + normalize: boundedNormalize(normalize), }); renderer.write(data, 0); const frames = await renderer.frames(); @@ -56,7 +71,7 @@ it("applies rendered retention cumulatively across sparse frames", async () => { rows: 1, scrollback: 0, maxBytes: 5, - normalize: (value) => value, + normalize: boundedNormalize((value) => value), }); try { renderer.write("a", 0); @@ -66,10 +81,11 @@ it("applies rendered retention cumulatively across sparse frames", async () => { expect(frames.map(({ lines }) => lines)).toEqual([["a"]]); expect(renderer.retention()).toEqual({ budget_bytes: 5, - observed_bytes: 6, + observed_bytes: 2, retained_bytes: 2, observed_frames: 2, retained_frames: 1, + unknown_size_frames: 1, }); expect(renderer.truncated()).toBe(true); } finally { @@ -84,6 +100,44 @@ it("trims only after normalization sees the full-width row", async () => { expect(frames[0]?.lines[0]).toBe("pid "); }); +it("omits rendered frames with unknown expanded size and keeps later bounded states", async () => { + const renderer = new TerminalRenderer({ + columns: 10, + rows: 1, + scrollback: 0, + maxBytes: 1_000, + normalize: createBoundedProcessTextNormalizer( + parseProcessScenario({ + executable: "unused", + normalization: { + paths: false, + pids: false, + ports: false, + patterns: [{ pattern: "a", replacement: "x".repeat(2_000) }], + }, + }), + "/tmp", + -1, + ), + }); + try { + renderer.write("a", 0); + renderer.write("\u001b[2J\rz", 1); + const frames = await renderer.frames(); + + expect(frames).toHaveLength(1); + expect(frames[0]?.lines[0]).toBe("z"); + expect(renderer.retention()).toMatchObject({ + observed_frames: 2, + retained_frames: 1, + unknown_size_frames: 1, + }); + expect(renderer.truncated()).toBe(true); + } finally { + await renderer.dispose(); + } +}); + it.each(["write", "resize"] as const)( "releases terminal resources after a queued %s observation fails", async (operation) => { @@ -95,9 +149,9 @@ it.each(["write", "resize"] as const)( rows: 4, scrollback: 10, maxBytes: 100_000, - normalize: () => { + normalize: boundedNormalize(() => { throw failure; - }, + }), }); const scope = new ProcessCaptureResourceScope(); const temporaryRoot = await mkdtemp( @@ -147,7 +201,7 @@ it("retains disposal failures for a later cleanup retry", async () => { rows: 4, scrollback: 10, maxBytes: 100_000, - normalize: (value) => value, + normalize: boundedNormalize((value) => value), }); const scope = new ProcessCaptureResourceScope(); const temporaryRoot = await mkdtemp(join(tmpdir(), "rea-terminal-retry-")); diff --git a/src/process/capture/TerminalRenderer.ts b/src/process/capture/TerminalRenderer.ts index 7ec2e10e5c..06598bf86c 100644 --- a/src/process/capture/TerminalRenderer.ts +++ b/src/process/capture/TerminalRenderer.ts @@ -1,5 +1,6 @@ import { createRequire } from "node:module"; import type { TerminalRetention } from "../../domain/process/processCaptureCoverage.js"; +import type { BoundedProcessTextNormalization } from "./ProcessNormalization.js"; import type { RecordProcessCaptureEvent, @@ -21,7 +22,10 @@ interface TerminalRendererOptions { readonly rows: number; readonly scrollback: number; readonly maxBytes: number; - readonly normalize: (value: string) => string; + readonly normalize: ( + value: string, + maxBytes: number, + ) => BoundedProcessTextNormalization; readonly recordEvent?: RecordProcessCaptureEvent; } @@ -42,6 +46,7 @@ export class TerminalRenderer { #observedBytes = 0; #observedFrames = 0; #truncated = false; + #unknownSizeFrames = 0; constructor(private readonly options: TerminalRendererOptions) { this.#terminal = new HeadlessPackage.Terminal({ @@ -97,6 +102,9 @@ export class TerminalRenderer { retained_bytes: this.#capturedBytes, observed_frames: this.#observedFrames, retained_frames: this.#frames.length, + ...(this.#unknownSizeFrames === 0 + ? {} + : { unknown_size_frames: this.#unknownSizeFrames }), }; } @@ -118,32 +126,50 @@ export class TerminalRenderer { #capture(atMs: number): void { const buffer = this.#terminal.buffer.active; const lines: string[] = []; + this.#observedFrames += 1; + let remainingBytes = this.options.maxBytes - this.#capturedBytes; + let bytes = 0; for (let row = 0; row < this.#terminal.rows; row += 1) { const line = buffer.getLine(buffer.viewportY + row); + const paddedLine = ( + line?.translateToString(false, 0, this.#terminal.cols) ?? "" + ).padEnd(this.#terminal.cols, " "); + const trailingSpaceBytes = Buffer.byteLength( + paddedLine.match(TRAILING_SPACES)?.[0] ?? "", + ); + const normalized = this.options.normalize( + paddedLine, + Math.min(Number.MAX_SAFE_INTEGER, remainingBytes + trailingSpaceBytes), + ); + if (normalized.state === "over_budget") { + this.#unknownSizeFrames += 1; + this.#truncated = true; + return; + } + const normalizedLine = normalized.value.replace(TRAILING_SPACES, ""); + const lineBytes = Buffer.byteLength(normalizedLine); + if (lineBytes > remainingBytes) { + this.#unknownSizeFrames += 1; + this.#truncated = true; + return; + } + remainingBytes -= lineBytes; + bytes += lineBytes; // Trailing blank cells carry no information beyond `columns`; dropping // them keeps sparse screens from costing a full row per blank line. - lines.push( - this.options - .normalize( - ( - line?.translateToString(false, 0, this.#terminal.cols) ?? "" - ).padEnd(this.#terminal.cols, " "), - ) - .replace(TRAILING_SPACES, ""), - ); + lines.push(normalizedLine); } - const serializedState = this.options.normalize( + const normalizedState = this.options.normalize( this.#serializeAddon.serialize(), + remainingBytes, ); - const bytes = - Buffer.byteLength(serializedState) + - lines.reduce((total, line) => total + Buffer.byteLength(line), 0); - this.#observedBytes += bytes; - this.#observedFrames += 1; - if (this.#capturedBytes + bytes > this.options.maxBytes) { + if (normalizedState.state === "over_budget") { + this.#unknownSizeFrames += 1; this.#truncated = true; return; } + bytes += normalizedState.utf8Bytes; + this.#observedBytes += bytes; this.#capturedBytes += bytes; const sequence = this.#frames.length; this.#frames.push({ @@ -155,7 +181,7 @@ export class TerminalRenderer { cursor_y: buffer.cursorY, active_buffer: buffer.type, lines, - serialized_state: serializedState, + serialized_state: normalizedState.value, }); this.options.recordEvent?.("rendered_frames", sequence); } diff --git a/src/process/ownedDebuggerSession.ts b/src/process/ownedDebuggerSession.ts new file mode 100644 index 0000000000..7fc2c0b4f1 --- /dev/null +++ b/src/process/ownedDebuggerSession.ts @@ -0,0 +1,311 @@ +import { readFile } from "node:fs/promises"; + +import { parseLinuxProcessStartTime } from "./ProcessOwnershipObservation.js"; + +/** Result of terminating one debugger process owned by this application. */ +export type OwnedDebuggerStopResult = + | { readonly status: "stopped" | "already_exited" } + | { readonly status: "incomplete"; readonly reason: string }; + +const DETACH_TIMEOUT_MS = 1_000; +const MAX_LATE_DEBUGGER_OUTPUT_BYTES = 64 * 1024; +export type OwnedDebuggerDetachResult = + | { readonly status: "no-inferior" } + | { readonly status: "detached"; readonly outputSink: "durable" }; +export interface OwnedInferiorIdentity { + readonly pid: string; + readonly startTime: string; +} + +export type ProcessStartTimeReader = (pid: string) => Promise; + +const readLinuxProcessStartTime: ProcessStartTimeReader = async (pid) => { + if (process.platform !== "linux") + throw new Error( + "Inferior process identity verification requires Linux /proc.", + ); + let stat: string; + try { + stat = await readFile(`/proc/${pid}/stat`, "utf8"); + } catch (cause: unknown) { + if ( + typeof cause === "object" && + cause !== null && + "code" in cause && + cause.code === "ENOENT" + ) + return null; + throw cause; + } + const startTime = parseLinuxProcessStartTime(stat); + if (startTime === undefined) + throw new Error(`Inferior PID ${pid} has an unreadable process identity.`); + return startTime; +}; + +/** Confirm only that every previously identified inferior process has ended. */ +export const verifyOwnedInferiorsEnded = async ( + identities: Iterable, + readStartTime: ProcessStartTimeReader = readLinuxProcessStartTime, +): Promise => { + let foundIdentity = false; + for (const identity of identities) { + foundIdentity = true; + const observed = await readStartTime(identity.pid); + if (observed === identity.startTime) return false; + } + return foundIdentity; +}; + +const detachAttempts = new WeakMap< + object, + Promise +>(); +const detachOutcomes = new WeakMap< + object, + "confirmed" | "unconfirmed" | "inferior-ended" +>(); + +const isVerifiedDetachResult = ( + result: unknown, +): result is OwnedDebuggerDetachResult => + typeof result === "object" && + result !== null && + (("status" in result && result.status === "no-inferior") || + ("status" in result && + result.status === "detached" && + "outputSink" in result && + result.outputSink === "durable")); + +const waitForInferiorDetach = async ( + session: object, + detach: () => Promise, +): Promise => { + let attempt = detachAttempts.get(session); + if (attempt === undefined) { + // Providers may use `this` to access their command transport. + const pending = Promise.resolve().then(() => detach.call(session)); + const tracked = pending.then( + (result) => { + detachAttempts.delete(session); + detachOutcomes.set( + session, + isVerifiedDetachResult(result) ? "confirmed" : "unconfirmed", + ); + return result; + }, + (cause: unknown) => { + detachAttempts.delete(session); + detachOutcomes.set(session, "unconfirmed"); + throw cause; + }, + ); + attempt = tracked; + detachAttempts.set(session, tracked); + } + let timer!: NodeJS.Timeout; + try { + return await Promise.race([ + attempt, + new Promise((resolve) => { + timer = setTimeout(() => resolve(undefined), DETACH_TIMEOUT_MS); + }), + ]); + } finally { + clearTimeout(timer); + } +}; + +interface OwnedDebuggerProcess { + readonly id: string; + readonly exited: boolean; + readonly inferiorStateUnknown?: () => boolean; + readonly verifyInferiorsEndedAfterExit?: () => Promise; + readonly launched: { + readonly process: { kill(signal: NodeJS.Signals): boolean }; + }; + /** Must return only after the provider verifies detachment or no inferior. */ + /** Confirms no inferior, or detaches only after verifying a durable output sink. */ + readonly detachInferior?: () => Promise; + readonly supervisor: { + waitForExit(timeoutMs: number): Promise; + /** WindowsOwnedProcess closes output only after the whole Job Object settles. */ + waitForOutputClose(timeoutMs: number): Promise; + unrefOutputStreams(): void; + /** Cap bytes retained after debugger shutdown while continuing to drain. */ + limitAdditionalOutputBytes(maxBytes: number): void; + dispose(): void; + }; +} + +const finishExitedDebuggerSession = async ( + session: OwnedDebuggerProcess, + debuggerName: string, + platform: NodeJS.Platform, + status: "stopped" | "already_exited", +): Promise => { + try { + session.supervisor.limitAdditionalOutputBytes( + MAX_LATE_DEBUGGER_OUTPUT_BYTES, + ); + if (!(await session.supervisor.waitForOutputClose(1_000))) { + return { + status: "incomplete", + reason: + platform === "win32" + ? `Owned ${debuggerName} process leader exited, but its Windows Job Object still has active descendants or has not reported settlement.` + : `Owned ${debuggerName} process leader exited, but inherited output streams are still open or have not reported settlement.`, + }; + } + } catch (cause: unknown) { + return { + status: "incomplete", + reason: + platform === "win32" + ? `Owned ${debuggerName} Windows Job Object settlement could not be verified: ${cause instanceof Error ? cause.message : String(cause)}` + : `Owned ${debuggerName} inherited output stream settlement could not be verified: ${cause instanceof Error ? cause.message : String(cause)}`, + }; + } + if (platform !== "win32") session.supervisor.unrefOutputStreams(); + session.supervisor.dispose(); + return { status }; +}; + +/** Stop only the directly owned debugger process, never its process group or target. */ +export const stopOwnedDebuggerProcess = async ( + session: OwnedDebuggerProcess, + debuggerName: string, + options: { readonly platform?: NodeJS.Platform } = {}, +): Promise => { + const platform = options.platform ?? process.platform; + if (session.exited) { + if (detachAttempts.has(session)) + return { + status: "incomplete", + reason: `Owned ${debuggerName} inferior detach is still pending after the process exited.`, + }; + const detachOutcome = detachOutcomes.get(session); + if ( + detachOutcome === "unconfirmed" || + (detachOutcome === undefined && session.inferiorStateUnknown?.() === true) + ) { + try { + if ((await session.verifyInferiorsEndedAfterExit?.()) !== true) + return { + status: "incomplete", + reason: `Owned ${debuggerName} inferior disposition was not verified before the debugger exited, and no owner proof confirms the identified inferior processes have ended.`, + }; + detachOutcomes.set(session, "inferior-ended"); + } catch (cause: unknown) { + return { + status: "incomplete", + reason: `Owned ${debuggerName} inferior process identity could not be verified after debugger exit: ${cause instanceof Error ? cause.message : String(cause)}`, + }; + } + } + return finishExitedDebuggerSession( + session, + debuggerName, + platform, + "already_exited", + ); + } + if (platform === "win32") + return { + status: "incomplete", + reason: `Owned ${debuggerName} process cannot be stopped safely on Windows because job termination can also terminate its inferior processes.`, + }; + try { + if (session.detachInferior === undefined) { + detachOutcomes.set(session, "unconfirmed"); + return { + status: "incomplete", + reason: `Owned ${debuggerName} inferior detach cannot be verified because the provider has no detach capability.`, + }; + } + const detached = await waitForInferiorDetach( + session, + session.detachInferior, + ); + if (detached === undefined) + return { + status: "incomplete", + reason: `Owned ${debuggerName} inferior detach was not verified within ${DETACH_TIMEOUT_MS} ms.`, + }; + if (!isVerifiedDetachResult(detached)) { + detachOutcomes.set(session, "unconfirmed"); + return { + status: "incomplete", + reason: `Owned ${debuggerName} provider did not confirm no inferior or detachment with a durable output sink.`, + }; + } + detachOutcomes.set(session, "confirmed"); + } catch (cause: unknown) { + return { + status: "incomplete", + reason: `Owned ${debuggerName} inferior detach could not be verified: ${cause instanceof Error ? cause.message : String(cause)}`, + }; + } + session.launched.process.kill("SIGTERM"); + if (!(await session.supervisor.waitForExit(300))) { + session.launched.process.kill("SIGKILL"); + const exitedAfterKill = await session.supervisor.waitForExit(1_000); + if (exitedAfterKill) + return finishExitedDebuggerSession( + session, + debuggerName, + platform, + "stopped", + ); + return { + status: "incomplete", + reason: `Owned ${debuggerName} process did not exit after direct termination.`, + }; + } + return finishExitedDebuggerSession( + session, + debuggerName, + platform, + "stopped", + ); +}; + +/** + * Stop every owned session and retain failed sessions for possible recovery. + * Callers must block replacement admission and drain in-flight starts until + * this settles; incomplete cleanup leaves the old owner in the map for retry. + */ +export const closeOwnedDebuggerSessions = async < + Session extends { id: string }, +>( + sessions: Map, + stop: (session: Session) => Promise, + debuggerName: string, +): Promise => { + const results = await Promise.allSettled( + [...sessions.entries()].map(async ([key, session]) => ({ + key, + session, + result: await stop(session), + })), + ); + const failures: Error[] = []; + for (const settled of results) { + if (settled.status === "rejected") { + failures.push( + settled.reason instanceof Error + ? settled.reason + : new Error(String(settled.reason)), + ); + continue; + } + const { key, session, result } = settled.value; + if (result.status === "incomplete") failures.push(new Error(result.reason)); + else if (sessions.get(key) === session) sessions.delete(key); + } + if (failures.length > 0) + throw new AggregateError( + failures, + `One or more owned ${debuggerName} processes did not stop cleanly`, + ); +}; diff --git a/src/process/snapshotEnvironment.test.ts b/src/process/snapshotEnvironment.test.ts index 9dfbf1e0a6..4012262993 100644 --- a/src/process/snapshotEnvironment.test.ts +++ b/src/process/snapshotEnvironment.test.ts @@ -25,19 +25,24 @@ it("canonicalizes selected Windows variables without inheriting ambient values", expect(Object.isFrozen(snapshot)).toBe(true); }); -it("uses Node's first lexicographic Windows key even when it is undefined", () => { - expect( - snapshotEnvironment( - { - path: "lower", - Path: "mixed", - PATH: "upper", - ComSpec: "mixed-cmd", - COMSPEC: "upper-cmd", - Temp: "mixed-temp", - TEMP: undefined, - }, - "win32", - ), - ).toEqual({ PATH: "upper", COMSPEC: "upper-cmd", TEMP: undefined }); +it("keeps the last caller value for Windows aliases, including undefined", () => { + const snapshot = snapshotEnvironment( + { + PATH: "upper", + path: "lower", + COMSPEC: "upper-cmd", + ComSpec: "mixed-cmd", + TEMP: "ambient-temp", + temp: undefined, + }, + "win32", + ); + + expect(snapshot).toEqual({ + COMSPEC: "mixed-cmd", + PATH: "lower", + TEMP: undefined, + }); + expect(Object.keys(snapshot)).toEqual(["COMSPEC", "PATH", "TEMP"]); + expect(Object.isFrozen(snapshot)).toBe(true); }); diff --git a/src/process/snapshotEnvironment.ts b/src/process/snapshotEnvironment.ts index 2bb54302f3..bfb57cd7d6 100644 --- a/src/process/snapshotEnvironment.ts +++ b/src/process/snapshotEnvironment.ts @@ -1,16 +1,16 @@ -/** Snapshot caller-selected variables using Node child-process key semantics. */ +/** Snapshot selected variables while preserving platform and caller precedence. */ export const snapshotEnvironment = ( environment: Readonly, platform: NodeJS.Platform = process.platform, ): Readonly => { if (platform !== "win32") return Object.freeze({ ...environment }); + const selected = new Map(); + // Fold aliases in caller order so an explicit later override wins before + // this canonical environment crosses a Windows process boundary. + for (const [key, value] of Object.entries(environment)) + selected.set(key.toUpperCase(), value); const snapshot: NodeJS.ProcessEnv = {}; - // Node sorts keys lexicographically and forwards the first case-insensitive - // match on Windows. Canonical keys also make subsequent lookups deterministic. - for (const key of Object.keys(environment).sort()) { - const canonicalKey = key.toUpperCase(); - if (!Object.hasOwn(snapshot, canonicalKey)) - snapshot[canonicalKey] = environment[key]; - } + for (const key of [...selected.keys()].sort()) + snapshot[key] = selected.get(key); return Object.freeze(snapshot); }; diff --git a/src/reference/ReferenceSourceReaderEntries.ts b/src/reference/ReferenceSourceReaderEntries.ts index e684a2b5cb..a7ae837e82 100644 --- a/src/reference/ReferenceSourceReaderEntries.ts +++ b/src/reference/ReferenceSourceReaderEntries.ts @@ -25,7 +25,7 @@ import { import type { PendingDirectory, ReferenceSourceEntry, - ReferenceSourceEntryKind, + ReferenceSourceExclusionKind, ReferenceSourceResult, TraversalState, } from "./ReferenceSourceReaderTypes.js"; @@ -201,6 +201,13 @@ const processEntry = async ( const metadata = await readMetadata(absolute); if (isAborted(state.signal)) return { ok: false, error: cancelled() }; if (!metadata.ok) { + const excluded = applyExclusion(state.shouldExclude, path, "unknown"); + if (!excluded.ok) + return { + ok: false, + error: failure("io", "Reference source exclusion check failed"), + }; + if (excluded.value) return { ok: true, value: undefined }; state.entries.push(entryFailure(path, "unknown", "io", metadata.message)); return { ok: true, value: undefined }; } @@ -350,7 +357,7 @@ const readSymlinkTarget = async ( } catch (cause: unknown) { const code = cause instanceof Error ? Reflect.get(cause, "code") : undefined; - if (code !== "ENOENT") throw cause; + if (code !== "ENOENT" && code !== "ENOTDIR") throw cause; return { status: "read", kind: "symlink", @@ -371,10 +378,10 @@ const readSymlinkTarget = async ( const applyExclusion = ( shouldExclude: - | ((path: string, kind: ReferenceSourceEntryKind) => boolean) + | ((path: string, kind: ReferenceSourceExclusionKind) => boolean) | undefined, path: string, - kind: ReferenceSourceEntryKind, + kind: ReferenceSourceExclusionKind, ): { readonly ok: true; readonly value: boolean } | { readonly ok: false } => { try { return { ok: true, value: shouldExclude?.(path, kind) === true }; diff --git a/src/reference/ReferenceSourceReaderErrors.ts b/src/reference/ReferenceSourceReaderErrors.ts index 6ddd02a3f2..2c826b4c96 100644 --- a/src/reference/ReferenceSourceReaderErrors.ts +++ b/src/reference/ReferenceSourceReaderErrors.ts @@ -29,6 +29,9 @@ const FILESYSTEM_ERROR_CODES = new Set([ "EISDIR", "ELOOP", "EMFILE", + // A traversed child can exceed the platform's absolute path limit even + // though the selected root and every component are valid. + "ENAMETOOLONG", "ENFILE", "ENOENT", "ENOTDIR", @@ -44,6 +47,8 @@ export const filesystemFailureDetail = ( ): string | undefined => { if (!(cause instanceof Error)) return undefined; const code: unknown = Reflect.get(cause, "code"); + if (code === "ENAMETOOLONG") + return `${operation}: ENAMETOOLONG (entry path exceeds the operating system path limit)`; return typeof code === "string" && FILESYSTEM_ERROR_CODES.has(code) ? `${operation}: ${cause.message}` : undefined; diff --git a/src/reference/ReferenceSourceReaderTypes.ts b/src/reference/ReferenceSourceReaderTypes.ts index 3b0a368b51..1fec383131 100644 --- a/src/reference/ReferenceSourceReaderTypes.ts +++ b/src/reference/ReferenceSourceReaderTypes.ts @@ -10,11 +10,17 @@ export type ReferenceSourceEntryKind = | "symlink" | "other"; +/** + * Kind offered to exclusion predicates. `unknown` marks an entry whose + * metadata could not be read, so its concrete kind cannot be claimed. + */ +export type ReferenceSourceExclusionKind = ReferenceSourceEntryKind | "unknown"; + export interface ReferenceSourceReaderOptions { readonly signal?: AbortSignal; readonly shouldExclude?: ( path: string, - kind: ReferenceSourceEntryKind, + kind: ReferenceSourceExclusionKind, ) => boolean; } @@ -84,7 +90,7 @@ export type TraversalState = { readonly signal?: AbortSignal; readonly shouldExclude?: ( path: string, - kind: ReferenceSourceEntryKind, + kind: ReferenceSourceExclusionKind, ) => boolean; readonly entries: ReferenceSourceEntry[]; readonly pending: PendingDirectory[]; diff --git a/src/rizin/RizinCommand.ts b/src/rizin/RizinCommand.ts new file mode 100644 index 0000000000..e2431da632 --- /dev/null +++ b/src/rizin/RizinCommand.ts @@ -0,0 +1,18 @@ +import { resolve } from "node:path"; + +import type { RizinInput } from "../domain/reverseEngineering.js"; + +export const RIZIN_PROVIDER_IDENTITY = { + id: "rizin", + name: "Rizin", + version: null, +} as const; + +/** Build a one-shot rz-pipe compatible command while retaining plugin availability. */ +export const rizinCommand = (input: RizinInput): readonly string[] => [ + "-N", + "-q", + "-c", + input.command, + resolve(input.path), +]; diff --git a/src/rizin/RizinDebugSessionManager.ts b/src/rizin/RizinDebugSessionManager.ts new file mode 100644 index 0000000000..fbad4a9e61 --- /dev/null +++ b/src/rizin/RizinDebugSessionManager.ts @@ -0,0 +1,1590 @@ +import { randomUUID } from "node:crypto"; +import { isUtf8 } from "node:buffer"; +import { open, readFile, readlink, realpath, stat } from "node:fs/promises"; +import { constants as fsConstants } from "node:fs"; +import { isatty } from "node:tty"; +import { resolve } from "node:path"; + +import { + AnalysisCancelledError, + AnalysisCapabilityUnavailableError, + AnalysisInputError, + AnalysisResourceConstraintError, + AnalysisTimeoutError, +} from "../domain/analysisErrorCore.js"; +import type { + AnalysisError, + AnalysisErrorOptions, +} from "../domain/analysisErrorBase.js"; +import { createEvidence, type Evidence } from "../domain/evidence.js"; +import { err, ok, type Result } from "../domain/result.js"; +import { + ProviderProcessSupervisor, + spawnOwnedProviderProcess, + type SpawnedOwnedProviderProcess, +} from "../process/ProviderProcess.js"; +import { + closeOwnedDebuggerSessions, + stopOwnedDebuggerProcess, + verifyOwnedInferiorsEnded, + type OwnedInferiorIdentity, + type OwnedDebuggerDetachResult, +} from "../process/ownedDebuggerSession.js"; + +const MAX_SESSIONS = 8; +const MAX_PENDING_COMMANDS = 16; +const FRAME_TIMEOUT_MS = 120_000; +const CLEANUP_COMMAND_TIMEOUT_MS = 1_000; +const STARTUP_TIMEOUT_MS = 30_000; +const MAX_FRAME_BYTES = 16 * 1024 * 1024; +const MAX_COMMAND_OUTPUT_BYTES = 16 * 1024 * 1024; +const MAX_QUEUED_FRAME_BYTES = 4 * 1024 * 1024; +const MAX_QUEUED_FRAMES = 4096; +const MIN_QUEUED_FRAME_CHARGE_BYTES = 64; +const MAX_COMMAND_FRAMES = 4096; +const MAX_RECENT_FRAME_BYTES = 16 * 1024; +const MAX_DIAGNOSTIC_BYTES = 64 * 1024; + +export const RIZIN_DEBUGGER_PROVIDER_IDENTITY = { + id: "rizin.debugger", + name: "Rizin debugger", + version: null, +} as const; + +interface RizinLaunchProvenance { + readonly caller_selected_path: string; + readonly resolved_path: string; +} + +const RIZIN_LAUNCH_PROVENANCE_LIMITATION = + "Launch provenance records the original and resolved startup paths; it does not establish the current target after unrestricted debugger commands."; + +interface RizinCommandEvidenceInput { + readonly sessionId: string; + readonly command: string; + readonly result: RizinDebugCommandResult; + readonly inferiorTty: string; + readonly launchProvenance: RizinLaunchProvenance; + readonly limitations: readonly string[]; +} + +const createRizinCommandEvidence = ({ + sessionId, + command, + result, + inferiorTty, + launchProvenance, + limitations, +}: RizinCommandEvidenceInput): Evidence => + createEvidence(undefined, RIZIN_DEBUGGER_PROVIDER_IDENTITY, { + predicateType: "rea.debugger.command-observation", + operation: "rizin_debug_command", + parameters: { + session_id: sessionId, + command, + inferior_tty: inferiorTty, + launch_provenance: { ...launchProvenance }, + }, + result: { ...result }, + rawResult: { ...result }, + subjectUnavailableReason: + "The current debugger target identity is not established by the Rizin session identifier.", + limitations: [ + ...limitations, + ...(result.output_invalid_utf8 + ? [ + "Collected command frames contained invalid UTF-8 bytes; decoded text may contain replacement characters.", + ] + : []), + RIZIN_LAUNCH_PROVENANCE_LIMITATION, + ], + }); + +export class RizinCommandTimeoutError extends Error { + constructor( + readonly partialOutput: string, + readonly outputInvalidUtf8 = false, + ) { + super("Timed out waiting for Rizin's command-completion marker"); + } +} + +export const isRizinCommandDeadlineExpired = (remainingMs: number): boolean => + remainingMs <= 0; + +export const createRizinCommandTimeoutError = ( + partialFrames: readonly string[], + outputInvalidUtf8 = false, +): RizinCommandTimeoutError => + new RizinCommandTimeoutError(partialFrames.join("\n"), outputInvalidUtf8); + +interface RizinCommandTimeoutEvidenceInput { + readonly sessionId: string; + readonly command: string; + readonly result: RizinDebugCommandResult; + readonly inferiorTty: string; + readonly cleanupFailure: string | undefined; + readonly launchProvenance: RizinLaunchProvenance; +} + +export const createRizinCommandTimeoutEvidence = ({ + sessionId, + command, + result, + inferiorTty, + cleanupFailure, + launchProvenance, +}: RizinCommandTimeoutEvidenceInput): Evidence => + createRizinCommandEvidence({ + sessionId, + command, + result, + inferiorTty, + launchProvenance, + limitations: [ + "The command timed out before its completion marker; retained output is partial and completion is unknown.", + ...(result.output_invalid_utf8 + ? [ + "Collected command frame bytes did not form valid UTF-8; displayed output contains replacement characters.", + ] + : []), + ...(cleanupFailure === undefined + ? [ + "The owned debugger session was stopped because output correlation could no longer be guaranteed.", + ] + : [ + `The owned debugger session remains retained because cleanup could not be verified: ${cleanupFailure}`, + ]), + "The command is unrestricted and may access local files, the shell, network, or mutate the target.", + ], + }); + +interface RizinDebugSession { + readonly id: string; + readonly launched: SpawnedOwnedProviderProcess; + readonly supervisor: ProviderProcessSupervisor; + readonly frames: Array<{ + readonly frame: RizinFrame; + readonly queueChargeBytes: number; + }>; + readonly frameWaiters: Array<{ + readonly resolve: (frame: RizinFrame) => void; + readonly reject: (cause: Error) => void; + }>; + readonly backend: string | null; + readonly launchProvenance: RizinLaunchProvenance; + readonly inferiorTty: string; + readonly inferiorTtyDevice: number; + readonly inferiorTtyInode: number; + readonly knownInferiorIdentities: Map; + inferiorInventoryComplete: boolean; + inferiorStateUnknown: () => boolean; + verifyInferiorsEndedAfterExit: () => Promise; + readonly detachInferior: () => Promise; + cleanupCommandTurnHeld: boolean; + stopAttempt: ReturnType | undefined; + readonly cleanupFailures: Set; + pendingCommandCount: number; + invalidUtf8Observed: boolean; + closing: boolean; + readonly history: string[]; + queuedFrameBytes: number; + historyTruncated: boolean; + frameChunks: Buffer[]; + bufferedFrameBytes: number; + ready: boolean; + exited: boolean; + commandTail: Promise; + protocolError: string | undefined; + commandCaptureActive: boolean; + discardingOversizedFrame: boolean; +} + +interface RizinFrame { + readonly text: string; + readonly truncated: boolean; + readonly invalidUtf8: boolean; +} + +class RizinProcessExitError extends Error { + constructor( + readonly partialOutput: string, + readonly outputInvalidUtf8 = false, + ) { + super("Rizin process exited before completing the active command"); + } +} + +class RizinCommandCancelledError extends AnalysisCancelledError { + constructor( + readonly partialOutput: string, + readonly outputInvalidUtf8: boolean, + cause: unknown, + ) { + super("rizin_debug_command", { + cause, + capturedOutput: { + stdout: partialOutput, + stderr: "", + truncated: false, + }, + }); + } +} + +class RizinFrameTimeoutError extends Error { + constructor() { + super("Timed out waiting for the next Rizin NUL frame"); + } +} + +/** Owns persistent Rizin debugger processes independently from BinarySession. */ +export class RizinDebugSessionManager { + readonly #sessions = new Map(); + #activeStarts = 0; + #sessionReservations = 0; + readonly #environment: Readonly; + #closingAll = false; + readonly #startDrainWaiters: Array<() => void> = []; + readonly #command: string; + readonly #frameTimeoutMs: number; + readonly #startupTimeoutMs: number; + readonly #maxCommandOutputBytes: number; + readonly #platform: NodeJS.Platform; + readonly #stopProcess: typeof stopOwnedDebuggerProcess; + readonly #spawnProcess: typeof spawnOwnedProviderProcess; + + constructor( + environment: Readonly = process.env, + options: { + readonly frameTimeoutMs?: number; + /** @internal Test seam; production callers should use the 30 s default. */ + readonly startupTimeoutMs?: number; + /** @internal Test seam; production callers should use the 16 MiB default. */ + readonly maxCommandOutputBytes?: number; + readonly platform?: NodeJS.Platform; + readonly stopProcess?: typeof stopOwnedDebuggerProcess; + readonly spawnProcess?: typeof spawnOwnedProviderProcess; + } = {}, + ) { + this.#environment = { ...environment }; + this.#command = environment.REA_RIZIN_COMMAND ?? "rizin"; + this.#frameTimeoutMs = options.frameTimeoutMs ?? FRAME_TIMEOUT_MS; + this.#startupTimeoutMs = options.startupTimeoutMs ?? STARTUP_TIMEOUT_MS; + this.#maxCommandOutputBytes = + options.maxCommandOutputBytes ?? MAX_COMMAND_OUTPUT_BYTES; + this.#platform = options.platform ?? process.platform; + this.#stopProcess = options.stopProcess ?? stopOwnedDebuggerProcess; + this.#spawnProcess = options.spawnProcess ?? spawnOwnedProviderProcess; + } + + async start( + input: { + readonly path: string; + readonly inferior_tty: string; + readonly backend?: string; + }, + signal?: AbortSignal, + ): Promise< + Result< + { readonly session_id: string; readonly backend: string | null }, + AnalysisError + > + > { + if (this.#closingAll) + return err(new AnalysisInputError("start_rizin_debug_session")); + const occupied = this.#sessions.size + this.#sessionReservations; + if (occupied >= MAX_SESSIONS) + return err( + new AnalysisResourceConstraintError( + "start_rizin_debug_session", + "memory", + "Rizin session capacity reached.", + { max_sessions: MAX_SESSIONS, active_or_reserved_sessions: occupied }, + ), + ); + this.#activeStarts += 1; + this.#sessionReservations += 1; + let reserved = true; + const releaseReservation = () => { + if (!reserved) return; + reserved = false; + this.#sessionReservations -= 1; + }; + try { + return await this.#start(input, signal, releaseReservation); + } finally { + releaseReservation(); + this.#activeStarts -= 1; + if (this.#activeStarts === 0) + for (const resolve of this.#startDrainWaiters.splice(0)) resolve(); + } + } + + async #start( + input: { + readonly path: string; + readonly inferior_tty: string; + readonly backend?: string; + }, + signal: AbortSignal | undefined, + releaseReservation: () => void, + ): Promise< + Result< + { readonly session_id: string; readonly backend: string | null }, + AnalysisError + > + > { + if (signal?.aborted) + return err(new AnalysisCancelledError("start_rizin_debug_session")); + if (this.#platform !== "linux") + return err( + new AnalysisCapabilityUnavailableError( + "rizin", + "start_rizin_debug_session", + "Persistent Rizin debugger sessions require Linux /proc descriptor readback to verify inferior detachment and output routing.", + ), + ); + if (input.inferior_tty === undefined || input.inferior_tty.length === 0) + return err(new AnalysisInputError("start_rizin_debug_session")); + let ttyPath: string; + let ttyDevice: number; + let ttyInode: number; + try { + ttyPath = await realpath(input.inferior_tty); + const ttyHandle = await open( + ttyPath, + fsConstants.O_RDWR | fsConstants.O_NOCTTY | fsConstants.O_NONBLOCK, + ); + try { + const ttyStat = await ttyHandle.stat(); + if ( + !isatty(ttyHandle.fd) || + (ttyStat.mode & fsConstants.S_IFMT) !== fsConstants.S_IFCHR + ) + throw new Error("The selected inferior_tty is not a character TTY."); + ttyDevice = ttyStat.dev; + ttyInode = ttyStat.ino; + } finally { + await ttyHandle.close(); + } + } catch (cause) { + return err( + new AnalysisInputError("start_rizin_debug_session", { cause }), + ); + } + // TTY identity validation is asynchronous. Cancellation may arrive while + // realpath/open/stat are pending, and must prevent the debugger spawn. + if (signal?.aborted) + return err(new AnalysisCancelledError("start_rizin_debug_session")); + const id = randomUUID(); + const resolvedPath = resolve(input.path); + const launchProvenance: RizinLaunchProvenance = { + caller_selected_path: input.path, + resolved_path: resolvedPath, + }; + const arguments_ = [ + "-N", + "-0", + "-d", + "-R", + `stdout=${ttyPath}`, + "-R", + `stderr=${ttyPath}`, + "-e", + "dbg.exitkills=false", + ...(input.backend === undefined ? [] : ["-D", input.backend]), + resolvedPath, + ]; + let launched: SpawnedOwnedProviderProcess; + try { + launched = await this.#spawnProcess({ + command: this.#command, + arguments: arguments_, + hostEnvironment: this.#environment, + runId: id, + stdin: "pipe", + ...(signal === undefined ? {} : { signal }), + }); + } catch (cause) { + if (signal?.aborted) + return err( + new AnalysisCancelledError("start_rizin_debug_session", { cause }), + ); + return err( + new AnalysisCapabilityUnavailableError( + "rizin", + "start_rizin_debug_session", + cause instanceof Error + ? cause.message + : "Unable to start Rizin debugger", + ), + ); + } + const session: RizinDebugSession = { + id, + launched, + launchProvenance, + backend: input.backend ?? null, + inferiorTty: ttyPath, + inferiorTtyDevice: ttyDevice, + inferiorTtyInode: ttyInode, + knownInferiorIdentities: new Map(), + inferiorInventoryComplete: false, + inferiorStateUnknown: () => !session.inferiorInventoryComplete, + verifyInferiorsEndedAfterExit: async () => { + if (!session.inferiorInventoryComplete) return false; + const ended = await verifyOwnedInferiorsEnded( + session.knownInferiorIdentities.values(), + ); + if (ended) session.knownInferiorIdentities.clear(); + return ended; + }, + detachInferior: () => + session.cleanupCommandTurnHeld + ? this.#detachInferiorUnlocked(session) + : this.#detachInferior(session), + cleanupCommandTurnHeld: false, + stopAttempt: undefined, + cleanupFailures: new Set(), + pendingCommandCount: 0, + invalidUtf8Observed: false, + closing: false, + supervisor: new ProviderProcessSupervisor( + { process: launched.process, ownsProcessLifetime: false }, + { + captureStdout: false, + maxDiagnosticBytes: MAX_DIAGNOSTIC_BYTES, + }, + ), + frames: [], + frameWaiters: [], + history: [], + queuedFrameBytes: 0, + historyTruncated: false, + frameChunks: [], + bufferedFrameBytes: 0, + ready: false, + exited: false, + commandTail: Promise.resolve(), + protocolError: undefined, + commandCaptureActive: false, + discardingOversizedFrame: false, + }; + this.#sessions.set(id, session); + releaseReservation(); + launched.process.stdout?.on("data", (chunk: Buffer | string) => + this.#receive(session, Buffer.from(chunk)), + ); + launched.process.stdin?.on("error", (cause: Error) => + this.#failProtocol( + session, + `Rizin stdin stream failed: ${cause.message}`, + ), + ); + launched.process.stdin?.once("close", () => { + if (!session.exited && session.protocolError === undefined) + this.#failProtocol(session, "Rizin stdin stream closed unexpectedly."); + }); + launched.process.once("exit", () => { + session.exited = true; + for (const waiter of session.frameWaiters) + waiter.reject(new RizinProcessExitError("")); + session.frameWaiters.length = 0; + }); + try { + await waitForFrame(session, this.#startupTimeoutMs, signal); + // Let process and stream lifecycle notifications queued alongside the + // initial frame run before publishing the session as ready. + await new Promise((resolve) => setImmediate(resolve)); + if (signal?.aborted) + throw new AnalysisCancelledError("start_rizin_debug_session"); + if (session.exited) + throw new RizinProcessExitError("Rizin exited during startup."); + if (session.protocolError !== undefined) + throw new Error(session.protocolError); + session.ready = true; + return ok({ session_id: id, backend: input.backend ?? null }); + } catch (cause) { + const stopped = await this.#stopDebuggerOnly(session); + const capturedOutput = this.#startupCapturedOutput(session); + if (stopped.status !== "incomplete") this.#sessions.delete(id); + if (stopped.status === "incomplete") + return err( + this.#retainCleanupFailure( + session, + "start_rizin_debug_session", + stopped.reason, + { cause, capturedOutput }, + ), + ); + if (signal?.aborted || cause instanceof AnalysisCancelledError) + return err( + new AnalysisCancelledError("start_rizin_debug_session", { + cause, + capturedOutput, + }), + ); + if (session.protocolError !== undefined) + return err( + new AnalysisCapabilityUnavailableError( + "rizin", + "start_rizin_debug_session", + session.protocolError, + { cause, capturedOutput }, + ), + ); + if (cause instanceof RizinProcessExitError) { + return err( + new AnalysisCapabilityUnavailableError( + "rizin", + "start_rizin_debug_session", + cause.message, + { + cause, + capturedOutput, + }, + ), + ); + } + return err( + new AnalysisTimeoutError( + "start_rizin_debug_session", + this.#startupTimeoutMs, + { cause, capturedOutput }, + ), + ); + } + } + + async execute( + sessionId: string, + command: string, + signal?: AbortSignal, + ): Promise< + Result< + { readonly result: RizinDebugCommandResult; readonly evidence: Evidence }, + AnalysisError + > + > { + const session = this.#sessions.get(sessionId); + if ( + session === undefined || + !session.ready || + session.exited || + session.protocolError !== undefined || + session.closing || + this.#closingAll + ) + return err(new AnalysisInputError("rizin_debug_command")); + if (signal?.aborted) + return err(new AnalysisCancelledError("rizin_debug_command")); + if (session.pendingCommandCount >= MAX_PENDING_COMMANDS) + return err( + new AnalysisResourceConstraintError( + "rizin_debug_command", + "transport", + "Rizin command queue capacity reached.", + { + max_pending_commands: MAX_PENDING_COMMANDS, + active_or_queued_commands: session.pendingCommandCount, + }, + ), + ); + session.pendingCommandCount += 1; + let release: (() => void) | undefined; + let acquired = false; + let commandSent = false; + const previous = session.commandTail; + session.commandTail = new Promise((resolve) => { + release = resolve; + }); + try { + await waitForTurn(previous, signal); + acquired = true; + if (signal?.aborted) + return err(new AnalysisCancelledError("rizin_debug_command")); + if ( + !session.ready || + session.exited || + session.closing || + this.#closingAll || + session.protocolError !== undefined + ) + return err(new AnalysisInputError("rizin_debug_command")); + if (session.frames.length > 0) { + const message = + "Unsolicited Rizin frames arrived before a command; command output can no longer be safely correlated."; + this.#failProtocol(session, message); + return err( + await this.#stopAfterProtocolFailure(session, undefined, true), + ); + } + const marker = `__REA_COMMAND_COMPLETE_${randomUUID().replaceAll("-", "")}__`; + // Arbitrary debugger commands can create or replace inferiors. Discard + // the prior inventory before any command bytes reach the provider. + session.inferiorInventoryComplete = false; + session.knownInferiorIdentities.clear(); + commandSent = true; + const stdin = session.launched.process.stdin; + if ( + stdin === null || + stdin === undefined || + stdin.destroyed || + !stdin.writable + ) { + this.#failProtocol(session, "Rizin stdin stream is closed."); + } else { + try { + stdin.write( + `${command}\n!echo ${marker}\n`, + "utf8", + (cause?: Error | null) => { + if (cause) + this.#failProtocol( + session, + `Rizin stdin write failed: ${cause.message}`, + ); + }, + ); + } catch (cause) { + this.#failProtocol( + session, + `Rizin stdin write failed: ${cause instanceof Error ? cause.message : "unknown error"}`, + ); + } + } + if (session.protocolError !== undefined) + throw new Error(session.protocolError); + session.commandCaptureActive = true; + const captured = await waitForCommandFrames( + session, + marker, + this.#frameTimeoutMs, + signal, + this.#maxCommandOutputBytes, + ); + if (signal?.aborted) + throw new AnalysisCancelledError("rizin_debug_command"); + let cleanupFailure: string | undefined; + if (captured.outputTruncated) { + // Stryker disable next-line BooleanLiteral -- `waitForCommandFrames` sets `protocolError` before reporting truncation; both detach wrappers therefore fail closed before a queued command can be awaited, so the held-tail flag cannot change this public path. + const stopped = await this.#stopDebuggerOnly(session, true); + if (stopped.status === "incomplete") cleanupFailure = stopped.reason; + } + const result = { + command, + output: captured.output, + output_scope: "command_and_interleaved_session_output" as const, + output_truncated: captured.outputTruncated, + completion_status: captured.outputTruncated + ? ("unknown" as const) + : ("complete" as const), + backend: session.backend, + ...(captured.outputInvalidUtf8 + ? { output_invalid_utf8: true as const } + : {}), + }; + const evidence = createRizinCommandEvidence({ + sessionId, + command, + result, + inferiorTty: session.inferiorTty, + launchProvenance: session.launchProvenance, + limitations: [ + "Rizin commands and debugger effects depend on the loaded IO plugin and target.", + "Output contains all NUL frames through an explicit command-completion marker; backend output interleaved before that marker cannot be distinguished from command output.", + ...(captured.outputTruncated + ? [ + `The Rizin command exceeded the output or frame limit. Retained output is partial and command completion is unknown.${cleanupFailure === undefined ? " The owned debugger session was stopped." : ` The owned debugger session remains retained because cleanup could not be verified: ${cleanupFailure}`}`, + ] + : []), + "The command is unrestricted and may access local files, the shell, network, or mutate the target.", + ], + }); + if (cleanupFailure !== undefined) + return err( + this.#retainCleanupFailure( + session, + "rizin_debug_command", + cleanupFailure, + { + partialObservation: evidence, + }, + ), + ); + return ok({ result, evidence }); + } catch (cause) { + if (signal?.aborted || cause instanceof AnalysisCancelledError) { + if (!commandSent) + return err( + new AnalysisCancelledError("rizin_debug_command", { cause }), + ); + const partialOutput = + cause instanceof RizinCommandCancelledError + ? cause.partialOutput + : undefined; + const commandInvalidUtf8 = + cause instanceof RizinCommandCancelledError && + cause.outputInvalidUtf8; + const partialEvidence = + partialOutput === undefined + ? undefined + : createRizinCommandEvidence({ + sessionId, + command, + result: { + command, + output: partialOutput, + output_scope: "command_and_interleaved_session_output", + output_truncated: false, + completion_status: "unknown", + backend: session.backend, + ...(commandInvalidUtf8 + ? { output_invalid_utf8: true as const } + : {}), + }, + inferiorTty: session.inferiorTty, + launchProvenance: session.launchProvenance, + limitations: [ + "The Rizin command was cancelled; collected frame output is partial and command completion is unknown.", + "The owned debugger session was stopped because command output can no longer be correlated safely.", + "The command is unrestricted and may access local files, the shell, network, or mutate the target.", + ], + }); + session.protocolError = + "A Rizin command was cancelled; the owned session was stopped because command output can no longer be correlated safely."; + const stopped = await this.#stopDebuggerOnly(session, true); + if (stopped.status === "incomplete") + return err( + this.#retainCleanupFailure( + session, + "rizin_debug_command", + stopped.reason, + { + cause: new AnalysisCancelledError("rizin_debug_command", { + cause, + ...(cause instanceof RizinCommandCancelledError + ? { capturedOutput: cause.capturedOutput } + : {}), + }), + ...(partialEvidence === undefined + ? {} + : { partialObservation: partialEvidence }), + }, + ), + ); + return err( + new AnalysisCancelledError("rizin_debug_command", { + cause, + ...(cause instanceof RizinCommandCancelledError + ? { capturedOutput: cause.capturedOutput } + : {}), + ...(partialEvidence === undefined + ? {} + : { partialObservation: partialEvidence }), + }), + ); + } + if (session.protocolError !== undefined) + return err(await this.#stopAfterProtocolFailure(session, cause, true)); + if (cause instanceof RizinCommandTimeoutError) { + session.protocolError = + "Rizin command timed out before its completion marker; output correlation is no longer safe."; + const stopped = await this.#stopDebuggerOnly(session, true); + const result: RizinDebugCommandResult = { + command, + output: cause.partialOutput, + output_scope: "command_and_interleaved_session_output", + output_truncated: true, + completion_status: "unknown", + backend: session.backend, + ...(cause.outputInvalidUtf8 + ? { output_invalid_utf8: true as const } + : {}), + }; + const evidence = createRizinCommandTimeoutEvidence({ + sessionId, + command, + result, + inferiorTty: session.inferiorTty, + cleanupFailure: + stopped.status === "incomplete" ? stopped.reason : undefined, + launchProvenance: session.launchProvenance, + }); + if (stopped.status === "incomplete") + return err( + this.#retainCleanupFailure( + session, + "rizin_debug_command", + stopped.reason, + { + cause, + partialObservation: evidence, + }, + ), + ); + return ok({ result, evidence }); + } + if (cause instanceof RizinProcessExitError) { + const result: RizinDebugCommandResult = { + command, + output: cause.partialOutput, + output_scope: "command_and_interleaved_session_output", + output_truncated: false, + completion_status: "unknown", + backend: session.backend, + ...(cause.outputInvalidUtf8 + ? { output_invalid_utf8: true as const } + : {}), + }; + const evidence = createRizinCommandEvidence({ + sessionId, + command, + result, + inferiorTty: session.inferiorTty, + launchProvenance: session.launchProvenance, + limitations: [ + "Rizin exited before the command-completion marker; retained frame output is partial and command completion is unknown.", + "The command is unrestricted and may access local files, the shell, network, or mutate the target.", + ], + }); + const stopped = await this.#stopDebuggerOnly(session, true); + if (stopped.status === "incomplete") + return err( + this.#retainCleanupFailure( + session, + "rizin_debug_command", + stopped.reason, + { cause, partialObservation: evidence }, + ), + ); + return ok({ result, evidence }); + } + session.protocolError = + "Rizin did not return the current command-completion marker; the session was stopped to prevent late output from being attributed to a later command."; + const stopped = await this.#stopDebuggerOnly(session, true); + if (stopped.status === "incomplete") + return err( + this.#retainCleanupFailure( + session, + "rizin_debug_command", + stopped.reason, + { cause }, + ), + ); + return err( + new AnalysisTimeoutError("rizin_debug_command", this.#frameTimeoutMs, { + cause, + }), + ); + } finally { + if (acquired) { + session.commandCaptureActive = false; + session.pendingCommandCount -= 1; + release?.(); + } else + void previous.then(() => { + session.pendingCommandCount -= 1; + release?.(); + }); + } + } + + status(sessionId: string): + | { + readonly session_id: string; + readonly state: "ready" | "closed"; + readonly inferior_tty: string; + readonly recent_output: readonly string[]; + readonly recent_output_truncated: boolean; + readonly diagnostics_truncated: boolean; + readonly output_invalid_utf8: boolean; + } + | undefined { + const session = this.#sessions.get(sessionId); + return session === undefined + ? undefined + : { + session_id: session.id, + state: + session.exited || + session.protocolError !== undefined || + session.closing || + !session.ready + ? "closed" + : "ready", + inferior_tty: session.inferiorTty, + recent_output: [...session.history], + recent_output_truncated: session.historyTruncated, + diagnostics_truncated: + session.supervisor.snapshot().diagnosticTruncated === true, + output_invalid_utf8: session.invalidUtf8Observed, + }; + } + + async close( + sessionId: string, + ): Promise> { + if (this.#closingAll) + return err(new AnalysisInputError("close_rizin_debug_session")); + const session = this.#sessions.get(sessionId); + if (session === undefined) + return err(new AnalysisInputError("close_rizin_debug_session")); + session.closing = true; + const stopped = await this.#stopDebuggerOnly(session); + if (stopped.status === "incomplete") + return err( + this.#retainCleanupFailure( + session, + "close_rizin_debug_session", + stopped.reason, + ), + ); + this.#sessions.delete(sessionId); + return ok({ target_state: "unknown" }); + } + + async retryCleanupFailures(): Promise> { + for (const session of this.#sessions.values()) { + if (session.cleanupFailures.size === 0) continue; + const stopped = await this.#stopDebuggerOnly(session); + if (stopped.status === "incomplete") + return err( + [...session.cleanupFailures].at(-1) ?? + this.#retainCleanupFailure( + session, + "close_rizin_debug_session", + stopped.reason, + ), + ); + this.#sessions.delete(session.id); + } + return ok(null); + } + + async closeAll(): Promise { + this.#closingAll = true; + if (this.#activeStarts > 0) + await new Promise((resolve) => + this.#startDrainWaiters.push(resolve), + ); + for (const session of this.#sessions.values()) session.closing = true; + return closeOwnedDebuggerSessions( + this.#sessions, + (session) => this.#stopDebuggerOnly(session), + "Rizin", + ); + } + + #receive(session: RizinDebugSession, chunk: Buffer): void { + let offset = 0; + while (offset < chunk.length) { + const boundary = chunk.indexOf(0, offset); + const end = boundary < 0 ? chunk.length : boundary; + const nextOffset = boundary + 1; + if (boundary >= 0 && nextOffset <= offset) { + this.#failProtocol( + session, + "Rizin frame parser failed to advance past a NUL delimiter.", + ); + return; + } + const piece = chunk.subarray(offset, end); + if (session.discardingOversizedFrame) { + if (boundary < 0) return; + session.discardingOversizedFrame = false; + this.#deliverBufferedFrame(session, true); + offset = nextOffset; + continue; + } + if (session.bufferedFrameBytes + piece.length > MAX_FRAME_BYTES) { + const message = + "Rizin emitted an NUL frame exceeding the 16 MiB protocol limit."; + if (!session.commandCaptureActive) { + this.#failProtocol(session, message); + return; + } + session.protocolError = message; + session.historyTruncated = true; + const remainingBytes = MAX_FRAME_BYTES - session.bufferedFrameBytes; + if (remainingBytes > 0) { + session.frameChunks.push( + Buffer.from(piece.subarray(0, remainingBytes)), + ); + session.bufferedFrameBytes += remainingBytes; + } + if (boundary < 0) { + session.discardingOversizedFrame = true; + return; + } + this.#deliverBufferedFrame(session, true); + offset = nextOffset; + continue; + } + if (piece.length > 0) { + session.frameChunks.push(Buffer.from(piece)); + session.bufferedFrameBytes += piece.length; + } + if (boundary < 0) return; + this.#deliverBufferedFrame(session, false); + offset = nextOffset; + } + } + + #deliverBufferedFrame(session: RizinDebugSession, truncated: boolean): void { + const frameBytes = Buffer.concat( + session.frameChunks, + session.bufferedFrameBytes, + ); + session.frameChunks = []; + session.bufferedFrameBytes = 0; + const invalidUtf8 = !isUtf8(frameBytes); + session.invalidUtf8Observed ||= invalidUtf8; + this.#deliverFrame( + session, + { text: frameBytes.toString("utf8"), truncated, invalidUtf8 }, + frameBytes.length, + ); + } + + #startupCapturedOutput(session: RizinDebugSession) { + const output = session.supervisor.snapshot(); + const partialFrame = + session.bufferedFrameBytes === 0 + ? "" + : Buffer.concat( + session.frameChunks, + session.bufferedFrameBytes, + ).toString("utf8"); + const protocolOutput = [...session.history, partialFrame] + .filter((frame) => frame.length > 0) + .join("\n"); + const protocolBytes = Buffer.from(protocolOutput, "utf8"); + let stdoutBytes = Math.min(protocolBytes.length, MAX_DIAGNOSTIC_BYTES); + while ( + stdoutBytes > 0 && + stdoutBytes < protocolBytes.length && + (protocolBytes[stdoutBytes]! & 0xc0) === 0x80 + ) + stdoutBytes -= 1; + return { + stdout: protocolBytes.subarray(0, stdoutBytes).toString("utf8"), + stderr: output.stderr.text, + truncated: + session.historyTruncated || + protocolBytes.length > stdoutBytes || + output.diagnosticTruncated === true || + output.stdout.observedBytes > output.stdout.bytes || + output.stderr.observedBytes > output.stderr.bytes, + }; + } + + #deliverFrame( + session: RizinDebugSession, + frame: RizinFrame, + frameBytes: number, + ): void { + const historyFrame = + frameBytes > MAX_RECENT_FRAME_BYTES || frame.truncated + ? `${frame.text.slice(0, MAX_RECENT_FRAME_BYTES)}[recent frame truncated]` + : frame.text; + if (frameBytes > MAX_RECENT_FRAME_BYTES || frame.truncated) + session.historyTruncated = true; + session.history.push(historyFrame); + if (session.history.length > 64) { + session.history.splice(0, session.history.length - 64); + session.historyTruncated = true; + } + const waiter = session.frameWaiters.shift(); + if (waiter !== undefined) { + waiter.resolve(frame); + return; + } + const queueChargeBytes = Math.max( + frameBytes, + MIN_QUEUED_FRAME_CHARGE_BYTES, + ); + if ( + session.frames.length >= MAX_QUEUED_FRAMES || + session.queuedFrameBytes + queueChargeBytes > MAX_QUEUED_FRAME_BYTES + ) { + this.#failProtocol( + session, + "Rizin produced unsolicited frames exceeding the bounded queue limit.", + ); + return; + } + session.frames.push({ frame, queueChargeBytes }); + session.queuedFrameBytes += queueChargeBytes; + } + + #failProtocol(session: RizinDebugSession, message: string): void { + session.protocolError = message; + session.frameChunks = []; + session.bufferedFrameBytes = 0; + for (const waiter of session.frameWaiters) + waiter.reject(new Error(message)); + session.frameWaiters.length = 0; + } + + #stopDebuggerOnly(session: RizinDebugSession, commandTurnHeld = false) { + session.closing = true; + if (session.stopAttempt !== undefined) return session.stopAttempt; + session.cleanupCommandTurnHeld = commandTurnHeld; + session.stopAttempt = Promise.resolve() + .then(() => + this.#stopProcess(session, "Rizin debugger", { + platform: this.#platform, + }), + ) + .then((stopped) => { + if (stopped.status !== "incomplete") { + for (const failure of session.cleanupFailures) + failure.resolveCleanup(); + session.cleanupFailures.clear(); + } + return stopped; + }) + .finally(() => { + session.stopAttempt = undefined; + }); + return session.stopAttempt; + } + + async #detachInferior( + session: RizinDebugSession, + ): Promise< + | { readonly status: "no-inferior" } + | { readonly status: "detached"; readonly outputSink: "durable" } + > { + if (session.exited || session.protocolError !== undefined) + throw new Error( + "Rizin protocol state is ambiguous; inferior detach is unverified.", + ); + const previous = session.commandTail; + let release!: () => void; + session.commandTail = new Promise((resolve) => (release = resolve)); + await previous; + try { + return await this.#detachInferiorUnlocked(session); + } finally { + release(); + } + } + + async #detachInferiorUnlocked( + session: RizinDebugSession, + ): Promise< + | { readonly status: "no-inferior" } + | { readonly status: "detached"; readonly outputSink: "durable" } + > { + if (session.exited || session.protocolError !== undefined) + throw new Error( + "Rizin protocol state is ambiguous; inferior detach is unverified.", + ); + { + session.inferiorInventoryComplete = false; + session.knownInferiorIdentities.clear(); + const initial = parseRizinPtraceRows( + await this.#runInternalCommand(session, "olj"), + ); + const inferior = initial; + if (inferior.length === 0) { + session.knownInferiorIdentities.clear(); + session.inferiorInventoryComplete = true; + return { status: "no-inferior" }; + } + session.knownInferiorIdentities.clear(); + for (const [, pid] of inferior) { + if (session.knownInferiorIdentities.has(pid)) continue; + const startTime = await readProcStartTime(pid); + session.knownInferiorIdentities.set(pid, { + pid, + startTime, + }); + } + session.inferiorInventoryComplete = true; + if (inferior.length !== 1) + throw new Error( + "Rizin reported multiple or ambiguous ptrace inferiors.", + ); + const [fd, pid] = inferior[0]!; + const processStartTime = await readProcStartTime(pid); + await this.#verifyInferiorOutputConfiguration(session, pid); + await this.#runInternalCommand(session, `o- ${fd}`); + const after = parseRizinPtraceRows( + await this.#runInternalCommand(session, "olj"), + ); + if ( + after.some( + ([activeFd, activePid]) => activeFd === fd || activePid === pid, + ) || + after.length !== 0 + ) + throw new Error( + "Rizin did not confirm that the selected inferior FD was detached.", + ); + assertRizinProcessStartTimeStable( + processStartTime, + await readProcStartTime(pid), + ); + await this.#verifyInferiorOutputConfiguration(session, pid); + return { status: "detached", outputSink: "durable" }; + } + } + + async #verifyInferiorOutputConfiguration( + session: RizinDebugSession, + pid: string, + ): Promise { + const exitKills = ( + await this.#runInternalCommand(session, "e dbg.exitkills") + ).trim(); + if (exitKills !== "false") + throw new Error("Rizin dbg.exitkills readback did not confirm false."); + for (const descriptor of ["1", "2"]) { + const descriptorPath = `/proc/${pid}/fd/${descriptor}`; + const destination = await readlink(descriptorPath); + const destinationStat = await stat(descriptorPath); + if ( + destinationStat.dev !== session.inferiorTtyDevice || + destinationStat.ino !== session.inferiorTtyInode || + (destinationStat.mode & fsConstants.S_IFMT) !== fsConstants.S_IFCHR || + (await realpath(destination)) !== session.inferiorTty + ) + throw new Error( + `Inferior descriptor ${descriptor} is not routed to the selected TTY.`, + ); + } + } + + async #runInternalCommand( + session: RizinDebugSession, + command: string, + ): Promise { + const marker = `__REA_CLEANUP_COMPLETE_${randomUUID().replaceAll("-", "")}__`; + const stdin = session.launched.process.stdin; + if ( + stdin === null || + stdin === undefined || + stdin.destroyed || + !stdin.writable + ) + throw new Error("Rizin stdin is unavailable during cleanup."); + stdin.write(`${command}\n!echo ${marker}\n`); + const captured = await waitForCommandFrames( + session, + marker, + CLEANUP_COMMAND_TIMEOUT_MS, + undefined, + this.#maxCommandOutputBytes, + ); + if (captured.outputTruncated) + throw new Error("Rizin cleanup readback was truncated."); + return captured.output; + } + + #retainCleanupFailure( + session: RizinDebugSession, + operation: string, + reason: string, + options: AnalysisErrorOptions = {}, + ): AnalysisError { + const failure = new AnalysisCapabilityUnavailableError( + "rizin", + operation, + `The owned Rizin process could not be confirmed stopped: ${reason}`, + { ...options, cleanup: { reason, resources: [session.id] } }, + ); + session.cleanupFailures.add(failure); + return failure; + } + + async #stopAfterProtocolFailure( + session: RizinDebugSession, + cause: unknown, + commandTurnHeld = false, + ): Promise { + const stopped = await this.#stopDebuggerOnly(session, commandTurnHeld); + return stopped.status === "incomplete" + ? this.#retainCleanupFailure( + session, + "rizin_debug_command", + stopped.reason, + { cause }, + ) + : new AnalysisCapabilityUnavailableError( + "rizin", + "rizin_debug_command", + session.protocolError!, + { cause }, + ); + } +} + +export interface RizinDebugCommandResult { + readonly command: string; + readonly output: string; + readonly output_scope: "command_and_interleaved_session_output"; + readonly output_truncated: boolean; + readonly output_invalid_utf8?: true; + readonly completion_status: "complete" | "unknown"; + readonly backend: string | null; +} + +const parseRizinPtraceRows = (output: string): Array<[string, string]> => { + let parsed: unknown; + try { + parsed = JSON.parse(output); + } catch (cause) { + throw new Error("Rizin returned malformed JSON for `olj`.", { cause }); + } + if (!Array.isArray(parsed)) + throw new Error("Rizin `olj` response is not an array."); + const rows: Array<[string, string]> = []; + for (const row of parsed) { + if ( + typeof row !== "object" || + row === null || + typeof (row as { fd?: unknown }).fd !== "number" || + !Number.isSafeInteger((row as { fd: number }).fd) || + (row as { fd: number }).fd < 0 || + typeof (row as { raised?: unknown }).raised !== "boolean" || + typeof (row as { uri?: unknown }).uri !== "string" + ) + throw new Error( + "Rizin `olj` response contains a malformed descriptor row.", + ); + const uri = (row as { uri: string }).uri; + const match = /^ptrace:\/\/(\d+)$/u.exec(uri); + if (match !== null) { + if ((row as { raised: boolean }).raised !== true) + throw new Error("Rizin `olj` ptrace descriptor is not active."); + rows.push([String((row as { fd: number }).fd), match[1]!]); + } else if (uri.startsWith("ptrace://")) + throw new Error("Rizin `olj` returned a malformed ptrace URI."); + } + return rows; +}; + +/** @internal Pure parser kept separately testable from live `/proc` access. */ +export const parseRizinProcStartTime = (statContents: string): string => { + const commandEnd = statContents.lastIndexOf(")"); + if (commandEnd < 0) + throw new Error("Inferior process identity is malformed."); + const fieldsAfterCommand = statContents + .slice(commandEnd + 1) + .trim() + .split(/\s+/u); + const startTime = fieldsAfterCommand[19]; + if (startTime === undefined || !/^\d+$/u.test(startTime)) + throw new Error("Inferior process start time could not be verified."); + return startTime; +}; + +/** @internal Reject a PID that changed identity during inferior detach. */ +export const assertRizinProcessStartTimeStable = ( + before: string, + after: string, +): void => { + if (after !== before) + throw new Error( + "The inferior PID disappeared or was reused during detach verification.", + ); +}; + +const readProcStartTime = async (pid: string): Promise => + parseRizinProcStartTime(await readFile(`/proc/${pid}/stat`, "utf8")); + +const waitForCommandFrames = async ( + session: RizinDebugSession, + marker: string, + timeoutMs: number, + signal?: AbortSignal, + maxCommandOutputBytes = MAX_COMMAND_OUTPUT_BYTES, +): Promise<{ + readonly output: string; + readonly outputTruncated: boolean; + readonly outputInvalidUtf8: boolean; +}> => { + const deadline = Date.now() + timeoutMs; + const output: string[] = []; + let outputInvalidUtf8 = false; + const commandTimeoutError = (): RizinCommandTimeoutError => + createRizinCommandTimeoutError(output, outputInvalidUtf8); + let outputBytes = 0; + let frameCount = 0; + for (;;) { + const remaining = deadline - Date.now(); + if (isRizinCommandDeadlineExpired(remaining)) throw commandTimeoutError(); + let received: RizinFrame; + try { + received = await waitForFrame( + session, + remaining, + signal, + commandTimeoutError, + ); + } catch (cause) { + if (cause instanceof AnalysisCancelledError) + throw new RizinCommandCancelledError( + output.join("\n"), + outputInvalidUtf8, + cause, + ); + if (cause instanceof RizinProcessExitError) + throw new RizinProcessExitError(output.join("\n"), outputInvalidUtf8); + throw cause; + } + outputInvalidUtf8 ||= received.invalidUtf8; + frameCount += 1; + if (frameCount > MAX_COMMAND_FRAMES) { + const message = + "Rizin emitted more than 4096 frames for one command; the session was stopped because output correlation is ambiguous."; + session.protocolError = message; + return { + output: output.join("\n"), + outputTruncated: true, + outputInvalidUtf8, + }; + } + if (received.truncated) { + const separatorBytes = output.length > 0 ? 1 : 0; + output.push( + truncateUtf8( + received.text, + maxCommandOutputBytes - outputBytes - separatorBytes, + ), + ); + return { + output: output.join("\n"), + outputTruncated: true, + outputInvalidUtf8, + }; + } + const frame = received.text; + const markerIndex = frame.indexOf(marker); + if (markerIndex >= 0) { + const beforeMarker = frame.slice(0, markerIndex).trimEnd(); + if (beforeMarker.length > 0) { + const beforeMarkerBytes = Buffer.byteLength(beforeMarker, "utf8"); + const separatorBytes = output.length > 0 ? 1 : 0; + if ( + outputBytes + separatorBytes + beforeMarkerBytes > + maxCommandOutputBytes + ) { + const message = + "Rizin command output exceeded the 16 MiB command limit."; + session.protocolError = message; + const retained = truncateUtf8( + beforeMarker, + maxCommandOutputBytes - outputBytes - separatorBytes, + ); + if (retained.length > 0) output.push(retained); + return { + output: output.join("\n"), + outputTruncated: true, + outputInvalidUtf8, + }; + } + outputBytes += separatorBytes + beforeMarkerBytes; + output.push(beforeMarker); + } + return { + output: output.join("\n"), + outputTruncated: false, + outputInvalidUtf8, + }; + } + const frameBytes = Buffer.byteLength(frame, "utf8"); + const separatorBytes = output.length > 0 ? 1 : 0; + if (outputBytes + separatorBytes + frameBytes > maxCommandOutputBytes) { + const message = "Rizin command output exceeded the 16 MiB command limit."; + session.protocolError = message; + output.push( + truncateUtf8( + frame, + maxCommandOutputBytes - outputBytes - separatorBytes, + ), + ); + return { + output: output.join("\n"), + outputTruncated: true, + outputInvalidUtf8, + }; + } + outputBytes += separatorBytes + frameBytes; + output.push(frame); + } +}; + +const truncateUtf8 = (value: string, maximumBytes: number): string => { + if (maximumBytes <= 0) return ""; + const bytes = Buffer.from(value, "utf8").subarray(0, maximumBytes); + return bytes.toString("utf8").replace(/\uFFFD$/u, ""); +}; + +const waitForFrame = ( + session: RizinDebugSession, + timeoutMs: number, + signal?: AbortSignal, + timeoutError?: () => Error, +): Promise => { + if (signal?.aborted) + return Promise.reject(new AnalysisCancelledError("rizin_debug_command")); + const queued = session.frames.shift(); + if (queued !== undefined) { + session.queuedFrameBytes -= queued.queueChargeBytes; + return Promise.resolve(queued.frame); + } + if (session.exited) return Promise.reject(new RizinProcessExitError("")); + return new Promise((resolve, reject) => { + // Settlement removes all three trigger paths before resolving or rejecting. + const cleanup = (): void => { + clearTimeout(timer); + signal?.removeEventListener("abort", onAbort); + const index = session.frameWaiters.indexOf(waiter); + if (index >= 0) session.frameWaiters.splice(index, 1); + }; + const onAbort = (): void => { + cleanup(); + reject(new AnalysisCancelledError("rizin_debug_command")); + }; + const timer = setTimeout(() => { + cleanup(); + reject(timeoutError?.() ?? new RizinFrameTimeoutError()); + }, timeoutMs); + const waiter = { + resolve: (frame: RizinFrame): void => { + cleanup(); + resolve(frame); + }, + reject: (cause: Error): void => { + cleanup(); + reject(cause); + }, + }; + session.frameWaiters.push(waiter); + signal?.addEventListener("abort", onAbort, { once: true }); + if (signal?.aborted) onAbort(); + }); +}; + +const waitForTurn = ( + previous: Promise, + signal?: AbortSignal, +): Promise => { + if (signal === undefined) return previous; + if (signal.aborted) + return Promise.reject(new AnalysisCancelledError("rizin_debug_command")); + return new Promise((resolve, reject) => { + const onAbort = (): void => + reject(new AnalysisCancelledError("rizin_debug_command")); + signal.addEventListener("abort", onAbort, { once: true }); + previous.then(() => { + signal.removeEventListener("abort", onAbort); + resolve(); + }); + }); +}; diff --git a/src/rust/RustAuditableReader.test.ts b/src/rust/RustAuditableReader.test.ts index cb281185a3..3edfdf2890 100644 --- a/src/rust/RustAuditableReader.test.ts +++ b/src/rust/RustAuditableReader.test.ts @@ -1,5 +1,6 @@ import { deflateSync } from "node:zlib"; import { expect, it } from "vitest"; +import { rustAuditableMetadataSchema } from "../domain/rust/rustBinary.js"; import { RUST_SECTION_BYTES, readAuditableSection, @@ -153,14 +154,54 @@ it("stops an inflate that exceeds the section ratio", () => { ).toThrow(/1 MiB/); }); -it.each([{ packages: [] }, { packages: [{ future_package_field: true }] }])( - "rejects an unknown format before interpreting its packages: %j", - ({ packages }) => { - expect(() => section({ format: 99, packages })).toThrow( +it.each([0, 8])("reads documented cargo-auditable revision %i", (format) => { + const parsed = section({ + format, + packages: [ + { name: "sample", version: "0.1.0", source: "local", root: true }, + ], + }); + expect(parsed).toMatchObject({ + format_version: format, + root_package_index: 0, + }); + expect(rustAuditableMetadataSchema.parse(parsed)).toEqual(parsed); +}); + +it.each([{ source: 4 }, { source: 9 }, { source: 99 }])( + "preserves unknown revision $source for the documented preceding-revision interpretation", + ({ source }) => { + const parsed = section({ + format: source, + packages: [ + { name: "sample", version: "0.1.0", source: "local", root: true }, + ], + }); + expect(parsed).toMatchObject({ + format_version: source, + }); + expect(rustAuditableMetadataSchema.parse(parsed)).toEqual(parsed); + }, +); + +it("rejects unknown package fields even for a forward-compatible revision", () => { + expect(() => + section({ + format: 99, + packages: [ + { name: "sample", version: "0.1.0", source: "local", future: true }, + ], + }), + ).toThrow(/package schema/); +}); + +it.each([-1, 1.5, 0x1_0000_0000])( + "rejects invalid cargo-auditable format value %s", + (format) => { + expect(() => section({ format, packages: [] })).toThrow( expect.objectContaining({ name: "RustAuditableFormatFailure", - kind: "unsupported", - message: "Unsupported cargo-auditable format version 99", + kind: "malformed", }), ); }, diff --git a/src/rust/RustAuditableReader.ts b/src/rust/RustAuditableReader.ts index d28b957047..c8805864a7 100644 --- a/src/rust/RustAuditableReader.ts +++ b/src/rust/RustAuditableReader.ts @@ -1,7 +1,6 @@ import { deflateSync, inflateSync } from "node:zlib"; import { z } from "zod"; import { - RUST_AUDITABLE_FORMAT_VERSION, cyclicDependencyIndices, type RustAuditableMetadata, type RustPackage, @@ -28,11 +27,10 @@ const packageSchema = z.strictObject({ root: z.boolean().optional(), }); const documentVersionSchema = z.object({ - format: z.number().int().nonnegative().optional(), + format: z.number().int().nonnegative().max(0xffff_ffff).optional(), }); const documentSchema = documentVersionSchema .extend({ - format: z.literal(RUST_AUDITABLE_FORMAT_VERSION).optional(), packages: z.array(packageSchema), }) .strict(); @@ -239,14 +237,6 @@ const parseDocument = (bytes: Buffer): z.infer => { "Dependency JSON does not match the format schema", { cause: version.error }, ); - if ( - version.data.format !== undefined && - version.data.format !== RUST_AUDITABLE_FORMAT_VERSION - ) - throw new RustAuditableFormatFailure( - "unsupported", - `Unsupported cargo-auditable format version ${version.data.format}`, - ); admitGraphExpansion(value); // One malformed package must fail before Zod retains an issue for every field. rejectMalformedPackages(value); diff --git a/src/rust/RustBinaryProvider.ts b/src/rust/RustBinaryProvider.ts index d0ce690f01..9808b615c2 100644 --- a/src/rust/RustBinaryProvider.ts +++ b/src/rust/RustBinaryProvider.ts @@ -13,6 +13,7 @@ import { AnalysisUnsupportedTargetError, } from "../domain/analysisErrorCore.js"; import { + RUST_AUDITABLE_FORMAT_VERSIONS, rustBinarySchema, type InspectRustBinaryInput, type RustBinary, @@ -97,6 +98,7 @@ const limitationsFor = ( "No .dep-v0 section was found. Absence of .dep-v0 does not mean the binary has no dependencies.", ] : []), + ...unknownFormatLimitation(image.auditable_metadata?.format_version), ...(image.auditable_metadata?.packages.some((pkg) => pkg.kind === null) ? [ "An omitted package kind is left null. The embedded format treats an omitted kind as runtime, and that default was not stored in the section.", @@ -111,6 +113,20 @@ const limitationsFor = ( ...rustcLimitation(image.rustc_commit), ]; +const unknownFormatLimitation = ( + format: number | null | undefined, +): string[] => { + if (format === null || format === undefined) return []; + let preceding = 0; + for (const known of RUST_AUDITABLE_FORMAT_VERSIONS) { + if (known === format) return []; + if (known < format) preceding = known; + } + return [ + `cargo-auditable format revision ${format} is unknown. Its schema directs readers to interpret it using the highest known preceding revision (${preceding}); this specific revision has not been independently verified.`, + ]; +}; + const rustcLimitation = ( commit: ReturnType["rustc_commit"], ): string[] => { diff --git a/src/server/EvidenceMcpServer.test.ts b/src/server/EvidenceMcpServer.test.ts new file mode 100644 index 0000000000..ce3e7a2746 --- /dev/null +++ b/src/server/EvidenceMcpServer.test.ts @@ -0,0 +1,146 @@ +import { describe, expect, it } from "vitest"; +import { Client, InMemoryTransport } from "@modelcontextprotocol/client"; +import { z } from "zod"; + +import { InvestigationRecords } from "../application/investigation/InvestigationRecords.js"; +import { + EvidenceMcpServer, + forgetPendingMcpRequest, +} from "./EvidenceMcpServer.js"; +import { ToolResultDelivery } from "./toolResult.js"; + +const firstText = (result: { + readonly content: readonly { + readonly type: string; + readonly text?: string; + }[]; +}): string | undefined => { + for (const item of result.content) { + if (item.type === "text") return item.text; + } + return undefined; +}; + +describe("MCP request bookkeeping", () => { + it("releases list, tool, and completion state when cancellation arrives", () => { + const completion = new AbortController(); + const pendingTools = new Map([[7, "inspect_artifact"]]); + const pendingToolLists = new Set([7]); + const pendingCompletions = new Map([[7, completion]]); + + forgetPendingMcpRequest( + pendingTools, + pendingToolLists, + pendingCompletions, + 7, + ); + + expect(pendingTools.has(7)).toBe(false); + expect(pendingToolLists.has(7)).toBe(false); + expect(completion.signal.aborted).toBe(true); + expect(pendingCompletions.has(7)).toBe(false); + }); +}); + +describe("SDK validation error delivery", () => { + it("bounds oversized errors, retains their source, and preserves fitting and valid calls", async () => { + const delivery = new ToolResultDelivery(4096); + const records = new InvestigationRecords(); + const server = new EvidenceMcpServer( + { name: "sdk-validation-budget-test", version: "1" }, + { capabilities: {} }, + (evidence) => records.recordEvidence(evidence), + delivery, + ); + server.registerTool( + "procedure_pseudo_code", + { inputSchema: z.strictObject({ required: z.string() }) }, + async () => ({ content: [{ type: "text", text: "valid call" }] }), + ); + const client = new Client({ + name: "sdk-validation-budget-client", + version: "1", + }); + const [clientTransport, serverTransport] = + InMemoryTransport.createLinkedPair(); + const oversizedKey = `unexpected_${"x".repeat(6000)}`; + try { + await server.connect(serverTransport); + await client.connect(clientTransport); + + const oversized = await client.callTool({ + name: "procedure_pseudo_code", + arguments: { required: "present", [oversizedKey]: true }, + }); + expect(oversized.isError).toBe(true); + expect(oversized.structuredContent).toBeUndefined(); + expect(oversized._meta).toEqual({ + provenance: { untrusted_content: true, source: "analysis_target" }, + }); + expect(Buffer.byteLength(JSON.stringify(oversized))).toBeLessThan(4096); + const oversizedError = z + .object({ + error: z.object({ + code: z.string(), + details: z.object({ + reported_limits: z.record(z.string(), z.unknown()), + }), + }), + }) + .parse(JSON.parse(firstText(oversized) ?? "null")); + expect(oversizedError.error.code).toBe("resource_constraint"); + const reportedLimits = oversizedError.error.details.reported_limits; + expect(reportedLimits).toMatchObject({ + evidence_reference: { + kind: "retained-evidence", + evidence_id: expect.any(String), + }, + diagnostic_truncated: true, + }); + expect(reportedLimits).not.toHaveProperty("retention"); + const bundle = records.exportEvidenceBundle(); + expect(bundle.records).toHaveLength(1); + expect(bundle.records[0]?.operation).toBe("mcp_tool_error"); + expect(bundle.records[0]?.raw_result).toMatchObject({ + isError: true, + content: [ + { type: "text", text: expect.stringContaining(oversizedKey) }, + ], + }); + const evidenceReference = z + .object({ evidence_id: z.string() }) + .parse(reportedLimits.evidence_reference); + expect(bundle.records[0]?.evidence_id).toBe( + evidenceReference.evidence_id, + ); + + const fitting = await client.callTool({ + name: "procedure_pseudo_code", + arguments: { unexpected: true }, + }); + expect(fitting.isError).toBe(true); + expect(firstText(fitting)).toContain("Input validation error"); + + const quotedKey = '"\\'.repeat(350); + const quoted = await client.callTool({ + name: "procedure_pseudo_code", + arguments: { required: "present", [quotedKey]: true }, + }); + expect(quoted.isError).toBe(true); + expect(firstText(quoted)).toContain("Input validation error"); + expect(Buffer.byteLength(JSON.stringify(quoted))).toBeLessThan(4096); + expect(records.exportEvidenceBundle().records).toHaveLength(1); + + const valid = await client.callTool({ + name: "procedure_pseudo_code", + arguments: { required: "present" }, + }); + expect(valid.content).toContainEqual({ + type: "text", + text: "valid call", + }); + } finally { + await Promise.all([client.close(), server.close()]); + } + }); +}); diff --git a/src/server/EvidenceMcpServer.ts b/src/server/EvidenceMcpServer.ts index f42fe9de67..cbf694bd7f 100644 --- a/src/server/EvidenceMcpServer.ts +++ b/src/server/EvidenceMcpServer.ts @@ -21,7 +21,11 @@ import { } from "@modelcontextprotocol/server"; import { analysisErrorProjectionSchema } from "../contracts/errorSchemas.js"; -import { compactAdvertisedInputSchema } from "../contracts/compactInputSchema.js"; +import { + presentAdvertisedInputSchema, + type CompactInputSchemaPresentation, +} from "../contracts/compactInputSchema.js"; +import { CATALOG_IDENTITY, createCatalogIdentity } from "../catalogIdentity.js"; import { transformAdvertisedInputJsonSchema } from "../contracts/toolSchemaMetadata.js"; import type { EvidenceWriter } from "../application/investigation/InvestigationRecordPort.js"; import { AnalysisResourceConstraintError } from "../domain/analysisErrorCore.js"; @@ -33,13 +37,11 @@ import { toolReturnsTargetContent } from "../contracts/toolEffects.js"; import { carriesTargetContentMarker, encodeToolResult, + measureMcpResultWithinBudget, TARGET_CONTENT_RESULT_META, } from "./toolResultEncoding.js"; -/** Compact advertised input schemas within this per-schema byte budget. */ -export interface CompactInputSchemaPresentation { - readonly budgetBytes: number; -} +export type { CompactInputSchemaPresentation } from "../contracts/compactInputSchema.js"; const currentRequestSignal = new AsyncLocalStorage(); @@ -60,8 +62,54 @@ const hasAdvertisedInputJsonSchema = (value: unknown): boolean => { ); }; +const isRecord = (value: unknown): value is Record => + typeof value === "object" && value !== null && !Array.isArray(value); + +/** Remove only client-facing output schemas; SDK registrations stay canonical. */ +const omitOutputSchemasFromToolsList = ( + message: JSONRPCMessage, + requestId: RequestId, +): JSONRPCMessage => { + if ( + !isJSONRPCResultResponse(message) || + message.id !== requestId || + !isRecord(message.result) || + !Array.isArray(message.result.tools) + ) + return message; + return { + ...message, + result: { + ...message.result, + tools: message.result.tools.map((tool: unknown) => { + if (!isRecord(tool)) return tool; + const { outputSchema: _outputSchema, ...advertisedTool } = tool; + return advertisedTool; + }), + }, + } as JSONRPCMessage; +}; + +/** Release every transport-owned reference for a cancelled/completed request. */ +export const forgetPendingMcpRequest = ( + pendingTools: Map, + pendingToolLists: Set, + pendingCompletions: Map, + id: RequestId, +): void => { + pendingTools.delete(id); + pendingToolLists.delete(id); + pendingCompletions.get(id)?.abort(); + pendingCompletions.delete(id); +}; + /** Bind oversized MCP error recovery to this server's Evidence ledger. */ export class EvidenceMcpServer extends McpServer { + readonly catalogIdentity: ReturnType; + private readonly compactPresentation: + | CompactInputSchemaPresentation + | undefined; + constructor( info: Implementation, options: McpServerOptions, @@ -69,18 +117,27 @@ export class EvidenceMcpServer extends McpServer { | EvidenceWriter["recordEvidence"] | undefined, readonly delivery: ToolResultDelivery, - private readonly compactPresentation: - | CompactInputSchemaPresentation - | undefined = undefined, + compactPresentation: CompactInputSchemaPresentation | undefined = undefined, + private readonly omitOutputSchemas = false, ) { super(info, options); + this.compactPresentation = + compactPresentation === undefined + ? undefined + : { budgetBytes: compactPresentation.budgetBytes }; + this.catalogIdentity = + this.compactPresentation === undefined && !this.omitOutputSchemas + ? CATALOG_IDENTITY + : createCatalogIdentity( + this.compactPresentation, + this.omitOutputSchemas ? "off" : "full", + ); } /** - * Compact the advertised input schema of every registered tool when the - * compact profile is active. The registered canonical schema, its parser, - * and output schemas are untouched; only the advertised JSON Schema - * projection is rendered within the provider's per-schema size budget. + * Compact advertised input schemas and optionally omit output-schema + * advertisements. Canonical parsers and contracts remain registered; the + * output-schema-off path validates successful structured results locally. * The signatures mirror the SDK's registrations so call sites keep their * inferred handler argument types. */ @@ -111,14 +168,14 @@ export class EvidenceMcpServer extends McpServer { typeof transformAdvertisedInputJsonSchema >[0], (projected) => - compactAdvertisedInputSchema(projected, budget).schema, + presentAdvertisedInputSchema(projected, { budgetBytes: budget }), ) : undefined; + const registeredConfig = + inputSchema === undefined ? config : { ...config, inputSchema }; return super.registerTool( name, - (inputSchema === undefined - ? config - : { ...config, inputSchema }) as never, + registeredConfig as never, callback as never, ); } @@ -126,7 +183,12 @@ export class EvidenceMcpServer extends McpServer { /** Preserve the SDK transport lifecycle while retaining oversized errors. */ override connect(transport: Transport): Promise { return super.connect( - new ErrorEvidenceTransport(transport, this.recordEvidence, this.delivery), + new ErrorEvidenceTransport( + transport, + this.recordEvidence, + this.delivery, + this.omitOutputSchemas, + ), ); } } @@ -137,6 +199,7 @@ class ErrorEvidenceTransport implements Transport { onmessage: Transport["onmessage"]; readonly hasPerRequestStream?: boolean; private readonly pendingTools = new Map(); + private readonly pendingToolLists = new Set(); private readonly pendingCompletions = new Map(); constructor( @@ -145,6 +208,7 @@ class ErrorEvidenceTransport implements Transport { | EvidenceWriter["recordEvidence"] | undefined, private readonly delivery: ToolResultDelivery, + private readonly omitOutputSchemas: boolean, ) { if (transport.hasPerRequestStream !== undefined) this.hasPerRequestStream = transport.hasPerRequestStream; @@ -155,6 +219,8 @@ class ErrorEvidenceTransport implements Transport { typeof message.params?.["name"] === "string" ) this.pendingTools.set(message.id, message.params["name"]); + if (isJSONRPCRequest(message) && message.method === "tools/list") + this.pendingToolLists.add(message.id); if (isJSONRPCRequest(message) && message.method === "completion/complete") this.pendingCompletions.set(message.id, new AbortController()); // The SDK suppresses replies after cancellation, so send() cannot release @@ -164,11 +230,13 @@ class ErrorEvidenceTransport implements Transport { message.method === "notifications/cancelled" ) { const id = message.params?.["requestId"]; - if (typeof id === "string" || typeof id === "number") { - this.pendingTools.delete(id); - this.pendingCompletions.get(id)?.abort(); - this.pendingCompletions.delete(id); - } + if (typeof id === "string" || typeof id === "number") + forgetPendingMcpRequest( + this.pendingTools, + this.pendingToolLists, + this.pendingCompletions, + id, + ); } const completion = isJSONRPCRequest(message) && message.method === "completion/complete" @@ -183,6 +251,7 @@ class ErrorEvidenceTransport implements Transport { transport.onerror = (error) => this.onerror?.(error); transport.onclose = () => { this.pendingTools.clear(); + this.pendingToolLists.clear(); for (const completion of this.pendingCompletions.values()) completion.abort(); this.pendingCompletions.clear(); @@ -206,6 +275,7 @@ class ErrorEvidenceTransport implements Transport { await this.transport.close(); } finally { this.pendingTools.clear(); + this.pendingToolLists.clear(); for (const completion of this.pendingCompletions.values()) completion.abort(); this.pendingCompletions.clear(); @@ -230,17 +300,24 @@ class ErrorEvidenceTransport implements Transport { ? message.id : undefined; const tool = id === undefined ? undefined : this.pendingTools.get(id); + const omitToolSchemas = + this.omitOutputSchemas && + id !== undefined && + this.pendingToolLists.has(id); try { const recovered = this.recoverError(message, tool); + const advertised = omitToolSchemas + ? omitOutputSchemasFromToolsList(recovered, id) + : recovered; if ( - isJSONRPCResultResponse(recovered) && - isCallToolResult(recovered.result) && - recovered.result.isError === true + isJSONRPCResultResponse(advertised) && + isCallToolResult(advertised.result) && + advertised.result.isError === true ) { // Keep the complete typed JSON in content. Error projections are not // success outputSchema data; oversized recovery must consume their // private structured carrier before it is removed from the wire. - const { structuredContent: _diagnostic, ...result } = recovered.result; + const { structuredContent: _diagnostic, ...result } = advertised.result; // Errors the SDK builds itself (input validation, a thrown handler) // bypass ToolResultDelivery; mark them for target-content tools too. const marked = @@ -252,12 +329,16 @@ class ErrorEvidenceTransport implements Transport { _meta: { ...result._meta, ...TARGET_CONTENT_RESULT_META }, } : result; - await this.transport.send({ ...recovered, result: marked }, options); - } else await this.transport.send(recovered, options); + await this.transport.send({ ...advertised, result: marked }, options); + } else await this.transport.send(advertised, options); } finally { if (!isJSONRPCRequest(message) && id !== undefined) { - this.pendingTools.delete(id); - this.pendingCompletions.delete(id); + forgetPendingMcpRequest( + this.pendingTools, + this.pendingToolLists, + this.pendingCompletions, + id, + ); } } } @@ -269,38 +350,65 @@ class ErrorEvidenceTransport implements Transport { if ( !isJSONRPCResultResponse(message) || !isCallToolResult(message.result) || - message.result.isError !== true || - message.result.structuredContent === undefined + message.result.isError !== true ) return message; - const structured = jsonObjectSchema.parse(message.result.structuredContent); const budget = this.delivery.resultBudgetBytes; // Count the provenance marker exactly as the projected error did, and keep // it on the replacement: the retained original may hold target content. - const targetContent = carriesTargetContentMarker(message.result); - // The delivered error omits structuredContent, so its text and error flag count. - const encoded = encodeToolResult( - structured, - budget, - "error-text", - targetContent, - ); + const targetContent = + carriesTargetContentMarker(message.result) || + (tool !== undefined && toolReturnsTargetContent(tool)); + // SDK validation failures have no structured carrier. Size their existing + // text-only result without serializing the complete JSON-RPC message. + const isStructured = message.result.structuredContent !== undefined; + const structured = isStructured + ? jsonObjectSchema.parse(message.result.structuredContent) + : undefined; + const sdkResultWithMarker = + !isStructured && + targetContent && + !carriesTargetContentMarker(message.result) + ? { + ...message.result, + _meta: { + ...message.result._meta, + ...TARGET_CONTENT_RESULT_META, + }, + } + : message.result; + const encoded = + structured !== undefined + ? encodeToolResult(structured, budget, "error-text", targetContent) + : measureMcpResultWithinBudget(sdkResultWithMarker, budget); if (encoded.ok) return message; const operation = tool ?? "mcp_tool_error"; - const originalError = analysisErrorProjectionSchema.safeParse( - structured["error"], - ); - // Retain the already projected diagnostic. Transport authentication and raw - // request arguments are deliberately excluded: the application owns their - // explicit sensitivity policy, and its error projection is authoritative. + const originalResult = jsonObjectSchema.parse(message.result); + const sdkDiagnostic = !isStructured + ? message.result.content.find((item) => item.type === "text")?.text + : undefined; + const retainedResult = structured ?? originalResult; + const originalError = isStructured + ? analysisErrorProjectionSchema.safeParse(structured?.["error"]) + : undefined; + // Retain the complete original response, including SDK validation text. + // The SDK error contains its rendered issue, not a separate argument bag; + // transport authentication and caller arguments are never added here. const evidence = createEvidence( undefined, { id: "rea-mcp", name: "REA MCP delivery", version: "1" }, { operation: "mcp_tool_error", parameters: { tool_name: operation }, - result: structured, - rawResult: jsonObjectSchema.parse(message.result), + result: + sdkDiagnostic === undefined + ? retainedResult + : { + boundary: "sdk-input-validation", + diagnostic_preview: sdkDiagnostic.slice(0, 256), + diagnostic_truncated: sdkDiagnostic.length > 256, + }, + rawResult: originalResult, }, ); const recorded = this.recordEvidence?.(evidence); @@ -311,12 +419,18 @@ class ErrorEvidenceTransport implements Transport { "The tool failed, but its complete error response exceeds the MCP response budget.", { boundary: "mcp-response", - ...(originalError.success + ...(originalError?.success ? { original_error_code: originalError.data.code } : {}), result_budget_bytes: budget, response_bytes_at_least: encoded.bytesAtLeast, constraint: encoded.constraint, + ...(sdkDiagnostic === undefined + ? {} + : { + diagnostic_preview: sdkDiagnostic.slice(0, 256), + diagnostic_truncated: sdkDiagnostic.length > 256, + }), ...(retained ? { evidence_reference: { diff --git a/src/server/createServer.ts b/src/server/createServer.ts index 8acc5ac849..0195bcee8c 100644 --- a/src/server/createServer.ts +++ b/src/server/createServer.ts @@ -8,6 +8,7 @@ import type { WasmArtifactService } from "../application/wasm/WasmArtifactServic import { wasmArtifactModule } from "../composition/wasm.js"; import { registerWasmTools } from "./registerWasmTools.js"; import { snapshotEnvironment } from "../process/snapshotEnvironment.js"; +import { captureProviderEnvironment } from "./providerEnvironment.js"; import { createPeResourcesService } from "../composition/binaryDiagnostics.js"; import type { PeResourcesService } from "../application/binaryDiagnostics/PeResourcesService.js"; import { registerPeResourcesTool } from "./registerPeResourcesTool.js"; @@ -21,6 +22,8 @@ import { registerRecordedCrashTools } from "./registerRecordedCrashTools.js"; import { createRecordedCrashService } from "../composition/binaryDiagnostics.js"; import { createGoBinaryService } from "../composition/go.js"; import type { GoBinaryService } from "../application/go/GoBinaryService.js"; +import { createElectronFuseService } from "../composition/electronFuses.js"; +import type { ElectronFuseService } from "../application/electron/ElectronFuseService.js"; import { registerGoTools } from "./registerGoTools.js"; import { createRustBinaryService } from "../composition/rust.js"; import type { RustBinaryService } from "../application/rust/RustBinaryService.js"; @@ -38,8 +41,9 @@ import { type ToolResultDelivery, } from "./toolResult.js"; import { parseMcpResponseBudget } from "../config/mcpResponseBudget.js"; +import { parseMcpOutputSchemaProfile } from "../config/mcpOutputSchemaProfile.js"; import { - COMPACT_INPUT_SCHEMA_BUDGET_BYTES, + compactPresentationForMcpInputSchemaProfile, parseMcpInputSchemaProfile, } from "../config/mcpInputSchemaProfile.js"; import type { CompactInputSchemaPresentation } from "./EvidenceMcpServer.js"; @@ -122,9 +126,18 @@ import { type ServerAnalysisSource, } from "./analysisAdmission.js"; import type { WithAdmittedAnalysis } from "./analysisAdmission.js"; +import { createReverseEngineeringService } from "../composition/reverseEngineering.js"; +import { registerReverseEngineeringTools } from "./registerReverseEngineeringTools.js"; +import { CutterBridgeClient } from "../cutter/CutterBridgeClient.js"; +import { CutterBridgeService } from "../cutter/CutterBridgeService.js"; +import { registerCutterTools } from "./registerCutterTools.js"; +import { GdbSessionManager } from "../gdb/GdbSessionManager.js"; +import { registerGdbTools } from "./registerGdbTools.js"; +import { RizinDebugSessionManager } from "../rizin/RizinDebugSessionManager.js"; +import { registerRizinDebugTools } from "./registerRizinDebugTools.js"; const TARGET_FREE_INSTRUCTIONS = - "REA provides reverse-engineering tools for local artifacts, native binaries, managed code, browser pages, and runtimes. Use the tool that directly answers the question; discover targets or inspect inventory only when needed. Tool results include inline Evidence and report their coverage and limitations."; + "REA provides reverse-engineering tools for local artifacts, native binaries, managed code, browser pages, and runtimes. Use the tool that directly answers the question; discover targets or inspect inventory only when needed. Tool results include inline Evidence and report their coverage and limitations. An opened target persists across turns. When finished, export needed Evidence and call close_binary; its snapshot_path can retain analysis."; const ACTIVE_TARGET_INSTRUCTIONS = "REA analyzes the active reverse-engineering target. Use the analysis tool that answers the question directly. Search or list symbols when discovery is needed; analyze_function provides a function dossier, and focused procedure tools return individual facets."; @@ -139,12 +152,14 @@ export interface CreateServerOptions { readonly packetCapture?: PacketCaptureService; readonly signalRecording?: SignalRecordingService; readonly wasmArtifact?: WasmArtifactService; + readonly providerEnvironment?: Readonly; readonly evmInterface?: EvmInterfaceService; readonly sqliteDatabase?: SqliteDatabaseService; readonly logger?: Logger; readonly binaryLayout?: BinaryLayoutService; readonly goBinary?: GoBinaryService; readonly rustBinary?: RustBinaryService; + readonly electronFuses?: ElectronFuseService; readonly recordedCrash?: RecordedCrashService; readonly firmwareAnalysis?: FirmwareAnalysisPort; readonly javascriptRecovery?: JavaScriptRecoveryPort; @@ -239,6 +254,7 @@ const createMcpServer = ( session: BinarySessionPort | undefined, delivery: ToolResultDelivery, compactPresentation: CompactInputSchemaPresentation | undefined, + omitOutputSchemas: boolean, ): EvidenceMcpServer => new EvidenceMcpServer( { @@ -258,6 +274,7 @@ const createMcpServer = ( : (evidence) => session.recordEvidence(evidence), delivery, compactPresentation, + omitOutputSchemas, ); /** @@ -272,6 +289,11 @@ export const createServer = ( const session = source.kind === "session" ? source.session : undefined; const analysis = source.kind === "session" ? source.session : source.analysis; const environment = snapshotEnvironment(options.environment ?? process.env); + const omitOutputSchemas = outputSchemasOmitted(environment); + const providerEnvironment = captureProviderEnvironment( + options.providerEnvironment, + environment, + ); const delivery = selectToolResultDelivery(environment, options.delivery); const evmInterface = options.evmInterface ?? evmInterfaceModule.create(environment); @@ -292,6 +314,8 @@ export const createServer = ( const javascriptRecovery = options.javascriptRecovery ?? createJavaScriptRecoveryProvider(environment); const artifactResources = new ArtifactResourceScope(); + const reverseEngineering = + createReverseEngineeringService(providerEnvironment); const firmwareAnalysis = new FirmwareAnalysisService( options.firmwareAnalysis ?? createFirmwareAnalysisProvider(environment), ); @@ -313,7 +337,12 @@ export const createServer = ( session, delivery, selectCompactInputSchemaPresentation(environment), + omitOutputSchemas, ); + const gdbSessions = new GdbSessionManager({ + environment: providerEnvironment, + }); + const rizinDebugSessions = new RizinDebugSessionManager(providerEnvironment); const android = options.androidAnalysis ?? createAndroidAnalysisProvider(environment); const pcap = options.pcapAnalysis ?? createPcapAnalysisProvider(environment); @@ -359,7 +388,10 @@ export const createServer = ( webNetworkCapture.close(), javascriptRecovery.close?.() ?? Promise.resolve(), artifactResources.close(), + reverseEngineering.close(), firmwareAnalysis.close(), + gdbSessions.closeAll(), + rizinDebugSessions.closeAll(), ]) .then((results) => { const failures = results.flatMap((result) => @@ -405,6 +437,28 @@ export const createServer = ( webNetworkCapture, }; registerBinaryAnalysisTools(toolContext); + registerReverseEngineeringTools( + server, + reverseEngineering, + toolLogger, + delivery, + session === undefined + ? undefined + : (evidence) => session.recordEvidence(evidence), + analysisAdmission, + session === undefined + ? undefined + : () => session.invalidateSnapshotForExternalMutation(), + session === undefined + ? undefined + : (path, signal, operation, rebindFailure) => + session.withAdmittedExternalMutation( + path, + signal, + operation, + rebindFailure, + ), + ); registerFridaTools( server, fridaInstrumentation, @@ -412,7 +466,37 @@ export const createServer = ( delivery, session === undefined ? undefined - : (input) => session.recordEvidence(input), + : (evidence) => session.recordEvidence(evidence), + ); + registerCutterTools( + server, + new CutterBridgeService(new CutterBridgeClient(providerEnvironment)), + toolLogger, + delivery, + session === undefined + ? undefined + : (evidence) => session.recordEvidence(evidence), + session, + ); + registerGdbTools( + server, + gdbSessions, + toolLogger, + delivery, + session === undefined + ? undefined + : (evidence) => session.recordEvidence(evidence), + session, + ); + registerRizinDebugTools( + server, + rizinDebugSessions, + toolLogger, + delivery, + session === undefined + ? undefined + : (evidence) => session.recordEvidence(evidence), + session, ); const previousOnclose = server.server.onclose; server.server.onclose = () => { @@ -493,6 +577,11 @@ export const createServer = ( : { availabilityPolicy: availability.policy }), androidAnalysisAvailability: (signal) => android.inspectAvailability(signal), + adbAnalysisAvailability: (signal) => + adbDevice.inspectAvailability(signal), + apktoolAnalysisAvailability: (signal) => + apktool.inspectAvailability(signal), + jebAnalysisAvailability: (signal) => jeb.inspectAvailability(signal), startedAt, }); } @@ -511,6 +600,16 @@ const selectToolResultDelivery = ( return createToolResultDelivery(configured.value); }; +const outputSchemasOmitted = ( + environment: Readonly, +): boolean => { + const configured = parseMcpOutputSchemaProfile( + environment.REA_MCP_OUTPUT_SCHEMA_PROFILE, + ); + if (!configured.ok) throw configured.error; + return configured.value === "off"; +}; + const selectCompactInputSchemaPresentation = ( environment: Readonly, ): CompactInputSchemaPresentation | undefined => { @@ -518,9 +617,7 @@ const selectCompactInputSchemaPresentation = ( environment.REA_MCP_INPUT_SCHEMA_PROFILE, ); if (!configured.ok) throw configured.error; - return configured.value === "compact" - ? { budgetBytes: COMPACT_INPUT_SCHEMA_BUDGET_BYTES } - : undefined; + return compactPresentationForMcpInputSchemaProfile(configured.value); }; const registerConfiguredAnalysisTools = ( @@ -793,6 +890,7 @@ const registerObservationTools = ({ evidenceById, electron: options.electronObservation, electronActive: options.electronActiveObservation, + electronFuses: options.electronFuses ?? createElectronFuseService(), observationLoadFailure: options.optionalProviderLoadFailures?.electronObservation, activeLoadFailure: diff --git a/src/server/externalMutationAdmission.ts b/src/server/externalMutationAdmission.ts new file mode 100644 index 0000000000..b4ebd98e66 --- /dev/null +++ b/src/server/externalMutationAdmission.ts @@ -0,0 +1,96 @@ +import type { BinarySessionPort } from "../application/binary/BinarySessionPort.js"; +import type { AnalysisError } from "../domain/analysisErrorBase.js"; +import { AnalysisOutputError } from "../domain/analysisErrorCore.js"; +import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; +import type { Evidence } from "../domain/evidence.js"; +import { err, type Result } from "../domain/result.js"; + +export type ExternalMutationAdmission = Pick< + BinarySessionPort, + "withAdmittedExternalMutation" +>; + +/** Shared MCP admission and failure preservation for unrestricted commands. */ +export const runWithExternalMutationAdmission = async ( + session: ExternalMutationAdmission | undefined, + operationName: string, + signal: AbortSignal | undefined, + operation: () => Promise>, + evidenceOf: ((value: Value) => Evidence) | undefined, + cleanupRecovery?: ( + failure: AnalysisError | undefined, + ) => Promise>, + retainedDetails?: (value: Value) => Readonly>, +): Promise> => { + if (session === undefined) { + const recovered = await cleanupRecovery?.(undefined); + if (recovered !== undefined && !recovered.ok) return recovered; + return operation(); + } + const admitted = await session.withAdmittedExternalMutation( + undefined, + signal, + operation, + (value, cause) => + value.ok + ? new AnalysisOutputError( + operationName, + "The external command may have changed the bound artifact, but the active analysis session could not be retired.", + { + cause, + ...(evidenceOf === undefined + ? {} + : { partialObservation: evidenceOf(value.value) }), + ...(cause.cleanup === undefined + ? {} + : { cleanup: cause.cleanup }), + ...(cause.capturedOutput === undefined + ? {} + : { capturedOutput: cause.capturedOutput }), + ...(retainedDetails === undefined + ? {} + : { diagnostics: retainedDetails(value.value) }), + }, + ) + : value.error.retainSecondaryFailure(cause, { + retirement_error: projectAnalysisError(cause), + }), + operationName, + cleanupRecovery, + ); + return admitted.ok ? admitted.value : err(admitted.error); +}; + +/** Admit an owned close once, including when it is recovering its own blocker. */ +export const runOwnedCleanupWithExternalMutationAdmission = async ( + session: ExternalMutationAdmission | undefined, + operationName: string, + signal: AbortSignal | undefined, + resourceId: string, + cleanup: () => Promise>, +): Promise> => { + let completed: Result | undefined; + const cleanupOnce = async (): Promise> => { + completed ??= await cleanup(); + return completed; + }; + const admitted = await runWithExternalMutationAdmission( + session, + operationName, + signal, + cleanupOnce, + undefined, + async (failure) => { + if ( + failure === undefined || + !failure.cleanup?.resources.includes(resourceId) + ) + return failure === undefined + ? { ok: true, value: null } + : { ok: false, error: failure }; + const recovered = await cleanupOnce(); + return recovered.ok ? { ok: true, value: null } : recovered; + }, + ); + return admitted; +}; diff --git a/src/server/promptCompletion.ts b/src/server/promptCompletion.ts index c4a4c028ab..21d67ededc 100644 --- a/src/server/promptCompletion.ts +++ b/src/server/promptCompletion.ts @@ -7,8 +7,8 @@ import type { AnalysisOperationPort } from "../application/AnalysisProvider.js"; import type { BinarySessionPort } from "../application/binary/BinarySessionPort.js"; import type { PromptCompletionKind } from "../contracts/promptContracts.js"; import { currentMcpRequestSignal } from "./EvidenceMcpServer.js"; -import { artifactInventoryResultSchema } from "../domain/artifactGraph.js"; -import { artifactInspectionResultSchema } from "../domain/artifactInspection.js"; +import { parseArtifactInventoryEvidence } from "../domain/artifactInventoryEvidence.js"; +import { EvidenceIntegrityError } from "../domain/evidenceErrors.js"; import { parseProcessCapture } from "../domain/process/processCaptureParsing.js"; const documentListSchema = z.array(z.string().min(1)); @@ -180,24 +180,32 @@ const evidenceValues = ( if (kind === "evidence") return [evidence.evidence_id]; if (kind === "capture") return isProcessCaptureEvidence(evidence) ? [evidence.evidence_id] : []; - const inventoryResult = - evidence.operation === "inventory_artifact" - ? evidence.normalized_result - : evidence.operation === "inspect_artifact" - ? (() => { - const inspection = artifactInspectionResultSchema.safeParse( - evidence.normalized_result, - ); - return inspection.success - ? inspection.data.substeps[0]?.evidence.normalized_result - : undefined; - })() - : undefined; - if (inventoryResult === undefined) return []; - const inventory = artifactInventoryResultSchema.safeParse(inventoryResult); - if (!inventory.success) return []; - if (kind === "manifest") return [inventory.data.manifest.manifest_id]; - return inventory.data.occurrences.map(({ occurrence_id: id }) => id); + if ( + evidence.operation === "inventory_artifact" || + evidence.operation === "inspect_artifact" + ) + return artifactInventoryValues(kind, evidence); + return []; +}; + +const artifactInventoryValues = ( + kind: "manifest" | "occurrence", + evidence: unknown, +): readonly string[] => { + try { + const { inventory } = parseArtifactInventoryEvidence(evidence); + return kind === "manifest" + ? [inventory.manifest.manifest_id] + : inventory.occurrences.map(({ occurrence_id: id }) => id); + } catch (cause: unknown) { + if ( + cause instanceof TypeError || + cause instanceof z.ZodError || + cause instanceof EvidenceIntegrityError + ) + return []; + throw cause; + } }; const isProcessCaptureEvidence = (evidence: LedgerEvidence): boolean => { diff --git a/src/server/providerEnvironment.ts b/src/server/providerEnvironment.ts new file mode 100644 index 0000000000..3d0d2c78a1 --- /dev/null +++ b/src/server/providerEnvironment.ts @@ -0,0 +1,8 @@ +import { snapshotEnvironment } from "../process/snapshotEnvironment.js"; + +/** Capture the selected provider environment at server construction time. */ +export const captureProviderEnvironment = ( + providerEnvironment: Readonly | undefined, + serverEnvironment: Readonly, +): Readonly => + snapshotEnvironment(providerEnvironment ?? serverEnvironment); diff --git a/src/server/registerArtifactComparisonTool.ts b/src/server/registerArtifactComparisonTool.ts index f9839ac1d4..3e0661f7c4 100644 --- a/src/server/registerArtifactComparisonTool.ts +++ b/src/server/registerArtifactComparisonTool.ts @@ -5,6 +5,7 @@ import { toolContract } from "../contracts/toolContracts.js"; import { compareArtifacts } from "../domain/artifactComparison.js"; import { createEvidence, parseEvidence } from "../domain/evidence.js"; import { jsonValueSchema } from "../domain/jsonValue.js"; +import { resolveArtifactComparisonEvidence } from "../application/artifacts/ArtifactComparisonEvidence.js"; import type { RecordUnknownInput } from "../domain/residualUnknown.js"; import { recordDerivedEvidence } from "./recordDerivedEvidence.js"; import { runDerivedOperation } from "./runDerivedOperation.js"; @@ -20,6 +21,9 @@ export const registerArtifactComparisonTool = ( contract: ReturnType>, ): void => { const admission = withAdmittedAnalysis({ kind: "session", session }); + const evidenceById = (evidenceId: string) => + session.evidenceForAnalysis?.(evidenceId) ?? + session.evidenceById(evidenceId); server.registerTool( contract.name, toolRegistrationOptions(contract), @@ -34,8 +38,20 @@ export const registerArtifactComparisonTool = ( context, contract.name, () => { - const left = parseEvidence(input.left); - const right = parseEvidence(input.right); + const suppliedLeft = parseEvidence(input.left); + const suppliedRight = parseEvidence(input.right); + const resolvedLeft = resolveArtifactComparisonEvidence( + suppliedLeft, + evidenceById, + ); + if (!resolvedLeft.ok) throw resolvedLeft.error; + const resolvedRight = resolveArtifactComparisonEvidence( + suppliedRight, + evidenceById, + ); + if (!resolvedRight.ok) throw resolvedRight.error; + const left = resolvedLeft.value; + const right = resolvedRight.value; return { left, right, comparison: compareArtifacts(left, right) }; }, ); diff --git a/src/server/registerArtifactTools.ts b/src/server/registerArtifactTools.ts index 1217664e62..d2df1da9e0 100644 --- a/src/server/registerArtifactTools.ts +++ b/src/server/registerArtifactTools.ts @@ -1,39 +1,36 @@ import type { EvidenceMcpServer } from "./EvidenceMcpServer.js"; -import type { EvidenceWriter } from "../application/investigation/InvestigationRecordPort.js"; import { ARTIFACT_TOOL_CONTRACTS, artifactInspectionInputSchema, } from "../contracts/artifactToolContracts.js"; -import type { EvidenceLookup } from "../application/EvidenceInputResolver.js"; -import { - requireAnalysisViewRetention, - summarizeRetainedAnalysis, -} from "../application/analysisView/AnalysisViewService.js"; import { createArtifactExtractionDestination } from "../application/artifacts/ArtifactExtractionDestination.js"; -import { ok } from "../domain/result.js"; -import type { BinaryTarget } from "../domain/binaryTargetTypes.js"; -import type { Logger } from "pino"; import { registerEvidenceTools } from "./registerEvidenceTools.js"; import { artifactInspectionResultSchema } from "../domain/artifactInspection.js"; +import { + retainedSummaryDelivery, + withoutDeliveryDetail, + type EvidenceToolFamilyOptions, +} from "./retainedSummaryDelivery.js"; + +/** Operations whose MCP delivery may be a retained summary view. */ +const SUMMARY_DELIVERY: ReadonlySet = new Set([ + "inspect_artifact", + "decode_interface_builder", +]); +/** inspect_artifact keeps an explicit detail in its Evidence parameters. */ +const decodeWithoutDelivery = withoutDeliveryDetail( + new Set(["decode_interface_builder"]), +); /** Register deterministic artifact inventory and safe extraction operations. */ export const registerArtifactTools = ( server: EvidenceMcpServer, - options: { - readonly logger: Logger; - readonly activeTarget: (() => BinaryTarget | undefined) | undefined; - readonly recordEvidence: EvidenceWriter["recordEvidence"] | undefined; - readonly evidenceById?: EvidenceLookup | undefined; - readonly withAdmittedAnalysis: import("./analysisAdmission.js").WithAdmittedAnalysis; - }, + options: EvidenceToolFamilyOptions, ): void => { - const retention = { - recordEvidence: options.recordEvidence, - evidenceById: options.evidenceById, - }; registerEvidenceTools(server, ARTIFACT_TOOL_CONTRACTS, { ...options, + ...retainedSummaryDelivery(SUMMARY_DELIVERY, options), parametersForExecution: (operation, parameters) => { if (operation === "inspect_artifact") { const { detail: _detail, ...execution } = @@ -42,16 +39,9 @@ export const registerArtifactTools = ( } return operation === "extract_artifact" ? { ...parameters, output_root: createArtifactExtractionDestination() } - : parameters; + : decodeWithoutDelivery(operation, parameters); }, - preflight: (operation, parameters) => - operation === "inspect_artifact" && parameters.detail === "summary" - ? requireAnalysisViewRetention(operation, retention) - : ok(null), - projectEvidence: (operation, parameters, evidence) => - operation === "inspect_artifact" && parameters.detail === "summary" - ? summarizeRetainedAnalysis(evidence, retention) - : ok(evidence), + parametersForEvidence: decodeWithoutDelivery, sourceEvidence: (operation, result) => operation === "inspect_artifact" ? artifactInspectionResultSchema diff --git a/src/server/registerCutterTools.ts b/src/server/registerCutterTools.ts new file mode 100644 index 0000000000..ab23feca91 --- /dev/null +++ b/src/server/registerCutterTools.ts @@ -0,0 +1,79 @@ +import type { McpServer } from "@modelcontextprotocol/server"; +import type { Logger } from "pino"; + +import type { CutterBridgeService } from "../cutter/CutterBridgeService.js"; +import type { EvidenceWriter } from "../application/investigation/InvestigationRecordPort.js"; +import { toolContract } from "../contracts/toolContracts.js"; +import { ok } from "../domain/result.js"; +import { evidenceSchema } from "../domain/evidence.js"; +import { + runWithExternalMutationAdmission, + type ExternalMutationAdmission, +} from "./externalMutationAdmission.js"; +import { logToolExecution } from "./toolLogging.js"; +import { toolRegistrationOptions } from "./toolRegistrationOptions.js"; +import type { ToolResultDelivery } from "./toolResult.js"; + +/** Register Cutter discovery and command tools against their named contracts. */ +export const registerCutterTools = ( + server: McpServer, + service: CutterBridgeService, + logger: Logger, + delivery: ToolResultDelivery, + recordEvidence?: EvidenceWriter["recordEvidence"], + session?: ExternalMutationAdmission, +): void => { + const list = toolContract("list_cutter_sessions"); + const command = toolContract("cutter_command"); + server.registerTool(list.name, toolRegistrationOptions(list), async () => { + const result = await logToolExecution(logger, list.name, async () => + ok(await service.listSessions()), + ); + if (!result.ok) return delivery.toCallToolResult(result, list); + return delivery.toCallToolResult( + { + ok: true, + value: { ...result.value, sessions: [...result.value.sessions] }, + }, + list, + ); + }); + server.registerTool( + command.name, + toolRegistrationOptions(command), + async (input, context) => { + const result = await runWithExternalMutationAdmission( + session, + command.name, + context.mcpReq.signal, + () => + logToolExecution(logger, command.name, () => + service.execute(input, { signal: context.mcpReq.signal }), + ), + (value) => value, + () => service.retryCleanup(), + ); + if (!result.ok) { + const partial = evidenceSchema.safeParse( + result.error.partialObservation, + ); + if (partial.success) { + const recorded = recordEvidence?.(partial.data); + if (recorded !== undefined) + return delivery.toRecordedPartialErrorToolResult( + result.error, + partial.data.evidence_id, + recorded, + command.returnsTargetContent, + ); + } + return delivery.toCallToolResult(result, command); + } + return delivery.toEvidenceToolResult( + result.value, + command, + recordEvidence?.(result.value), + ); + }, + ); +}; diff --git a/src/server/registerElectronTools.ts b/src/server/registerElectronTools.ts index e8008177da..96a518e0f2 100644 --- a/src/server/registerElectronTools.ts +++ b/src/server/registerElectronTools.ts @@ -21,6 +21,7 @@ import { analyzeJavaScriptApplicationValidated } from "../application/javascript import type { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js"; import { reconcileJavaScriptRuntimeEvidenceValidated } from "../application/javascript/JavaScriptRuntimeReconciliationService.js"; import type { ProgressReporter } from "../application/ProgressReporter.js"; +import type { ElectronFuseService } from "../application/electron/ElectronFuseService.js"; import { toolContract } from "../contracts/toolContracts.js"; import type { ToolContract } from "../contracts/toolContractTypes.js"; import type { AnalysisError } from "../domain/analysisErrorBase.js"; @@ -32,6 +33,7 @@ import { logToolExecution } from "./toolLogging.js"; import { toolRegistrationOptions } from "./toolRegistrationOptions.js"; import type { WithAdmittedAnalysis } from "./analysisAdmission.js"; import { runAdmittedToolOperation } from "./admittedToolOperation.js"; +import { createEvidenceToolHandler } from "./contractToolHandler.js"; interface ElectronToolRegistration { readonly artifactResources: ArtifactResourceScope; @@ -40,6 +42,7 @@ interface ElectronToolRegistration { readonly activeLoadFailure?: OptionalProviderLoadFailure | undefined; readonly electron: ElectronObservationPort | undefined; readonly electronActive: ElectronActiveObservationPort | undefined; + readonly electronFuses: ElectronFuseService; readonly recordEvidence: EvidenceWriter["recordEvidence"] | undefined; readonly evidenceById: EvidenceLookup | undefined; readonly withAdmittedAnalysis?: WithAdmittedAnalysis; @@ -57,6 +60,7 @@ export const registerElectronTools = ( options: ElectronToolRegistration, ): void => { const registration = { ...options, delivery: server.delivery, server }; + registerElectronFuseTool(server, options); const listContract = toolContract("list_electron_targets"); const inspectContract = toolContract("inspect_electron_page"); const analyzeContract = toolContract("analyze_javascript_application"); @@ -141,6 +145,25 @@ export const registerElectronTools = ( ); }; +const registerElectronFuseTool = ( + server: EvidenceMcpServer, + options: ElectronToolRegistration, +): void => { + const contract = toolContract("inspect_electron_fuses"); + const handler = createEvidenceToolHandler( + server, + (input, execution) => options.electronFuses.inspect(input, execution), + options.logger, + options.recordEvidence, + options.withAdmittedAnalysis, + ); + server.registerTool( + contract.name, + toolRegistrationOptions(contract), + handler(contract), + ); +}; + const runElectronTool = async ( options: ElectronToolRegistration & { readonly delivery: ToolResultDelivery; diff --git a/src/server/registerEnhancedTools.ts b/src/server/registerEnhancedTools.ts index e1e27c1271..32cc4b6a1f 100644 --- a/src/server/registerEnhancedTools.ts +++ b/src/server/registerEnhancedTools.ts @@ -14,6 +14,7 @@ import type { BinarySessionPort } from "../application/binary/BinarySessionPort. import { EnhancedTools } from "../application/EnhancedTools.js"; import type { ValidatedEnhancedCall } from "../application/EnhancedToolTypes.js"; import { + executeComposedWorkflow, createWorkflowEvidence, workflowSnapshotRecord, recordWorkflowUnknowns, @@ -231,15 +232,20 @@ const executeEnhancedTool = async ( context, parameters, }); - const services = new EnhancedTools({ + const analysis: AnalysisOperationPort = { execute: (operation, operationParameters, executionOptions) => admittedAnalysis.execute(operation, operationParameters, { ...executionOptions, progress, }), - }); + }; const result = await logToolExecution(registration.logger, name, () => - services.executeValidated(validatedCall, context.mcpReq.signal), + executeComposedWorkflow(analysis, (observed) => + new EnhancedTools(observed).executeValidated( + validatedCall, + context.mcpReq.signal, + ), + ), ); await progress.report({ phase: name, @@ -253,8 +259,9 @@ const executeEnhancedTool = async ( target: registration.activeTarget?.(), operation: name, parameters, - result: result.value, + result: result.value.result, upstreamProfile: registration.analysisProfile?.(), + upstreamLimitations: result.value.limitations, }); const recorded = registration.recordEvidence?.(evidence); if (recorded !== undefined && !recorded.ok) @@ -273,7 +280,7 @@ const executeEnhancedTool = async ( } const unknowns = recordWorkflowUnknowns({ name, - result: result.value, + result: result.value.result, evidenceId: evidence.evidence_id, recordUnknown: registration.recordUnknown, }); diff --git a/src/server/registerFridaTools.ts b/src/server/registerFridaTools.ts index fb2e06bb4f..a56ec95785 100644 --- a/src/server/registerFridaTools.ts +++ b/src/server/registerFridaTools.ts @@ -6,7 +6,9 @@ import { FridaInstrumentationService } from "../application/frida/FridaInstrumen import type { FridaInstrumentationPort } from "../application/frida/FridaInstrumentationPort.js"; import type { FridaRemoteConnection } from "../application/frida/FridaInstrumentationPort.js"; import { toolContract } from "../contracts/toolContracts.js"; +import type { ToolContract } from "../contracts/toolContractTypes.js"; import { evidenceSchema } from "../domain/evidence.js"; +import type { AnalysisError } from "../domain/analysisErrorBase.js"; import { AnalysisInputError } from "../domain/analysisErrorCore.js"; import { jsonValueSchema } from "../domain/jsonValue.js"; import { logToolExecution } from "./toolLogging.js"; @@ -121,8 +123,12 @@ export const registerFridaTools = ( }), ); if (!result.ok) { - recordPartialEvidence(result.error, recordEvidence); - return delivery.toCallToolResult(result, start); + return deliverPartialEvidenceError( + result, + start, + delivery, + recordEvidence, + ); } return delivery.toCallToolResult( { @@ -162,8 +168,12 @@ export const registerFridaTools = ( }), ); if (!result.ok) { - recordPartialEvidence(result.error, recordEvidence); - return delivery.toCallToolResult(result, load); + return deliverPartialEvidenceError( + result, + load, + delivery, + recordEvidence, + ); } return delivery.toEvidenceToolResult( result.value, @@ -295,8 +305,12 @@ export const registerFridaTools = ( () => operation, ); if (!result.ok) { - recordPartialEvidence(result.error, recordEvidence); - return delivery.toCallToolResult(result, instrument); + return deliverPartialEvidenceError( + result, + instrument, + delivery, + recordEvidence, + ); } return delivery.toEvidenceToolResult( result.value.evidence, @@ -307,13 +321,21 @@ export const registerFridaTools = ( ); }; -const recordPartialEvidence = ( - error: { readonly partialObservation?: unknown }, +const deliverPartialEvidenceError = ( + result: { readonly ok: false; readonly error: AnalysisError }, + contract: ToolContract, + delivery: ToolResultDelivery, recordEvidence: EvidenceWriter["recordEvidence"] | undefined, -): void => { - if (error.partialObservation === undefined) return; - const parsed = evidenceSchema.safeParse(error.partialObservation); - if (parsed.success) recordEvidence?.(parsed.data); +): ReturnType => { + const parsed = evidenceSchema.safeParse(result.error.partialObservation); + if (!parsed.success || recordEvidence === undefined) + return delivery.toCallToolResult(result, contract); + return delivery.toRecordedPartialErrorToolResult( + result.error, + parsed.data.evidence_id, + recordEvidence(parsed.data), + contract.returnsTargetContent, + ); }; const remoteConnection = ( diff --git a/src/server/registerGdbTools.ts b/src/server/registerGdbTools.ts new file mode 100644 index 0000000000..3b8fa44777 --- /dev/null +++ b/src/server/registerGdbTools.ts @@ -0,0 +1,156 @@ +import type { McpServer, ServerContext } from "@modelcontextprotocol/server"; +import type { Logger } from "pino"; + +import type { GdbSessionManager } from "../gdb/GdbSessionManager.js"; +import type { EvidenceWriter } from "../application/investigation/InvestigationRecordPort.js"; +import { toolContract } from "../contracts/toolContracts.js"; +import { evidenceSchema } from "../domain/evidence.js"; +import { err } from "../domain/result.js"; +import { AnalysisInputError } from "../domain/analysisErrorCore.js"; +import { + runOwnedCleanupWithExternalMutationAdmission, + runWithExternalMutationAdmission, + type ExternalMutationAdmission, +} from "./externalMutationAdmission.js"; +import { logToolExecution } from "./toolLogging.js"; +import { toolRegistrationOptions } from "./toolRegistrationOptions.js"; +import type { ToolResultDelivery } from "./toolResult.js"; + +export type GdbToolManager = Pick< + GdbSessionManager, + "start" | "execute" | "status" | "close" | "retryCleanupFailures" +>; + +/** Register persistent GDB session operations on the existing REA MCP server. */ +export const registerGdbTools = ( + server: McpServer, + manager: GdbToolManager, + logger: Logger, + delivery: ToolResultDelivery, + recordEvidence?: EvidenceWriter["recordEvidence"], + session?: ExternalMutationAdmission, +): void => { + const start = toolContract("start_gdb_session"); + const consoleContract = toolContract("gdb_console"); + const status = toolContract("gdb_session_status"); + const close = toolContract("close_gdb_session"); + server.registerTool( + start.name, + toolRegistrationOptions(start), + async (input, context: ServerContext) => { + const result = await logToolExecution(logger, start.name, () => + manager.start( + context.mcpReq.signal, + input.inferior_tty === undefined + ? {} + : { inferiorTty: input.inferior_tty }, + ), + ); + if (result.ok) + return delivery.toCallToolResult( + { ok: true, value: result.value }, + start, + ); + const partial = result.error.partialObservation; + const parsed = evidenceSchema.safeParse(partial); + if (parsed.success) { + const recorded = recordEvidence?.(parsed.data); + if (recorded !== undefined) + return delivery.toRecordedPartialErrorToolResult( + result.error, + parsed.data.evidence_id, + recorded, + start.returnsTargetContent, + ); + } + return delivery.toCallToolResult(result, start); + }, + ); + server.registerTool( + consoleContract.name, + toolRegistrationOptions(consoleContract), + async (input, context: ServerContext) => { + const execute = () => + logToolExecution(logger, consoleContract.name, () => + manager.execute( + input.session_id, + input.command, + context.mcpReq.signal, + ), + ); + const result = await runWithExternalMutationAdmission( + session, + consoleContract.name, + context.mcpReq.signal, + execute, + (value) => value.evidence, + () => manager.retryCleanupFailures(), + ); + if (!result.ok) { + const partial = result.error.partialObservation; + const parsed = evidenceSchema.safeParse(partial); + if (parsed.success) { + const recorded = recordEvidence?.(parsed.data); + if (recorded !== undefined) + return delivery.toRecordedPartialErrorToolResult( + result.error, + parsed.data.evidence_id, + recorded, + consoleContract.returnsTargetContent, + ); + } + return delivery.toCallToolResult(result, consoleContract); + } + const recorded = recordEvidence?.(result.value.evidence); + return delivery.toEvidenceToolResult( + result.value.evidence, + consoleContract, + recorded, + ); + }, + ); + server.registerTool( + status.name, + toolRegistrationOptions(status), + async (input) => { + const result = manager.status(input.session_id); + if (result === undefined) + return delivery.toCallToolResult( + err(invalidSession("gdb_session_status")), + status, + ); + return delivery.toCallToolResult( + { + ok: true, + value: { + ...result, + recent_mi_records: [...result.recent_mi_records], + }, + }, + status, + ); + }, + ); + server.registerTool( + close.name, + toolRegistrationOptions(close), + async (input, context: ServerContext) => { + const result = await runOwnedCleanupWithExternalMutationAdmission( + session, + close.name, + context.mcpReq.signal, + input.session_id, + () => + logToolExecution(logger, close.name, () => + manager.close(input.session_id), + ), + ); + return result.ok + ? delivery.toCallToolResult({ ok: true, value: result.value }, close) + : delivery.toCallToolResult(result, close); + }, + ); +}; + +const invalidSession = (operation: string): AnalysisInputError => + new AnalysisInputError(operation); diff --git a/src/server/registerManagedTools.ts b/src/server/registerManagedTools.ts index 5ff513d67a..f5929243f8 100644 --- a/src/server/registerManagedTools.ts +++ b/src/server/registerManagedTools.ts @@ -1,50 +1,36 @@ import type { EvidenceMcpServer } from "./EvidenceMcpServer.js"; -import type { EvidenceWriter } from "../application/investigation/InvestigationRecordPort.js"; -import type { EvidenceLookup } from "../application/EvidenceInputResolver.js"; -import { - requireAnalysisViewRetention, - summarizeRetainedAnalysis, -} from "../application/analysisView/AnalysisViewService.js"; - import type { AnalysisOperationPort } from "../application/AnalysisProvider.js"; import { MANAGED_TOOL_CONTRACTS } from "../contracts/managed/managedToolContracts.js"; -import type { BinaryTarget } from "../domain/binaryTargetTypes.js"; import { AnalysisInputError } from "../domain/analysisErrorCore.js"; -import { err, ok } from "../domain/result.js"; -import type { Logger } from "pino"; +import { err } from "../domain/result.js"; import { registerEvidenceTools } from "./registerEvidenceTools.js"; +import { + retainedSummaryDelivery, + withoutDeliveryDetail, + type EvidenceToolFamilyOptions, +} from "./retainedSummaryDelivery.js"; import { runManagedProviderExecution } from "../composition/directAnalysis.js"; import { isManagedToolName } from "../contracts/managed/managedToolContracts.js"; +/** Operations whose MCP delivery may be a retained summary view. */ +const SUMMARY_DELIVERY: ReadonlySet = new Set([ + "inspect_managed_artifact", + "inspect_managed_members", +]); + /** Register execution-free managed PE/CLI inspection. */ export const registerManagedTools = ( server: EvidenceMcpServer, - options: { - readonly logger: Logger; - readonly activeTarget: (() => BinaryTarget | undefined) | undefined; - readonly recordEvidence: EvidenceWriter["recordEvidence"] | undefined; - readonly evidenceById?: EvidenceLookup | undefined; - readonly withAdmittedAnalysis: import("./analysisAdmission.js").WithAdmittedAnalysis; - }, + options: EvidenceToolFamilyOptions, ): void => { let selectedManagedPath: string | undefined; + const withoutDelivery = withoutDeliveryDetail(SUMMARY_DELIVERY); const analysisForAdmission = ( analysis: AnalysisOperationPort, ): AnalysisOperationPort => ({ execute: async (operation, parameters, executionOptions) => { if (!isManagedToolName(operation)) return analysis.execute(operation, parameters, executionOptions); - const { detail: _detail, ...artifactParameters } = parameters; - if ( - operation === "inspect_managed_artifact" && - parameters.detail === "summary" - ) { - const ready = requireAnalysisViewRetention(operation, { - recordEvidence: options.recordEvidence, - evidenceById: options.evidenceById, - }); - if (!ready.ok) return ready; - } const requestedPath = parameters.path; const activeTarget = options.activeTarget?.(); const path = @@ -69,9 +55,7 @@ export const registerManagedTools = ( path, operation, executionOptions?.signal, - operation === "inspect_managed_artifact" - ? artifactParameters - : parameters, + parameters, ); if (execution.ok && typeof requestedPath === "string") selectedManagedPath = requestedPath; @@ -80,14 +64,9 @@ export const registerManagedTools = ( }); registerEvidenceTools(server, MANAGED_TOOL_CONTRACTS, { ...options, + ...retainedSummaryDelivery(SUMMARY_DELIVERY, options), + parametersForExecution: withoutDelivery, + parametersForEvidence: withoutDelivery, analysisForAdmission, - projectEvidence: (operation, parameters, evidence) => - operation === "inspect_managed_artifact" && - parameters.detail === "summary" - ? summarizeRetainedAnalysis(evidence, { - recordEvidence: options.recordEvidence, - evidenceById: options.evidenceById, - }) - : ok(evidence), }); }; diff --git a/src/server/registerNativeTools.ts b/src/server/registerNativeTools.ts index 036389d16e..62668daa14 100644 --- a/src/server/registerNativeTools.ts +++ b/src/server/registerNativeTools.ts @@ -1,54 +1,25 @@ import type { EvidenceMcpServer } from "./EvidenceMcpServer.js"; -import type { EvidenceWriter } from "../application/investigation/InvestigationRecordPort.js"; import { NATIVE_TOOL_CONTRACTS } from "../contracts/native/nativeToolContracts.js"; -import type { EvidenceLookup } from "../application/EvidenceInputResolver.js"; -import { - requireAnalysisViewRetention, - summarizeRetainedAnalysis, -} from "../application/analysisView/AnalysisViewService.js"; -import type { JsonValue } from "../domain/jsonValue.js"; -import { ok } from "../domain/result.js"; -import type { BinaryTarget } from "../domain/binaryTargetTypes.js"; -import type { Logger } from "pino"; import { registerEvidenceTools } from "./registerEvidenceTools.js"; +import { + retainedSummaryDelivery, + withoutDeliveryDetail, + type EvidenceToolFamilyOptions, +} from "./retainedSummaryDelivery.js"; -/** Delivery detail selects a response shape; it never reaches the tools or Evidence identity. */ -const withoutDelivery = ( - operation: string, - parameters: Readonly>, -): Readonly> => { - if (operation !== "inspect_macho") return parameters; - const { detail: _detail, ...selected } = parameters; - return selected; -}; +const SUMMARY_DELIVERY: ReadonlySet = new Set(["inspect_macho"]); /** Register macOS native contracts; Mach-O inspection can deliver a retained summary. */ export const registerNativeTools = ( server: EvidenceMcpServer, - options: { - readonly logger: Logger; - readonly activeTarget: (() => BinaryTarget | undefined) | undefined; - readonly recordEvidence: EvidenceWriter["recordEvidence"] | undefined; - readonly evidenceById?: EvidenceLookup | undefined; - readonly withAdmittedAnalysis: import("./analysisAdmission.js").WithAdmittedAnalysis; - }, + options: EvidenceToolFamilyOptions, ): void => { - const retention = { - recordEvidence: options.recordEvidence, - evidenceById: options.evidenceById, - }; + const withoutDelivery = withoutDeliveryDetail(SUMMARY_DELIVERY); registerEvidenceTools(server, NATIVE_TOOL_CONTRACTS, { ...options, + ...retainedSummaryDelivery(SUMMARY_DELIVERY, options), parametersForExecution: withoutDelivery, parametersForEvidence: withoutDelivery, - preflight: (operation, parameters) => - operation === "inspect_macho" && parameters.detail === "summary" - ? requireAnalysisViewRetention(operation, retention) - : ok(null), - projectEvidence: (operation, parameters, evidence) => - operation === "inspect_macho" && parameters.detail === "summary" - ? summarizeRetainedAnalysis(evidence, retention) - : ok(evidence), }); }; diff --git a/src/server/registerReverseEngineeringTools.ts b/src/server/registerReverseEngineeringTools.ts new file mode 100644 index 0000000000..49ccbb1ff3 --- /dev/null +++ b/src/server/registerReverseEngineeringTools.ts @@ -0,0 +1,207 @@ +import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; +import type { + CallToolResult, + McpServer, + ServerContext, +} from "@modelcontextprotocol/server"; +import type { Logger } from "pino"; + +import type { ReverseEngineeringService } from "../application/reverse/ReverseEngineeringService.js"; +import type { EvidenceWriter } from "../application/investigation/InvestigationRecordPort.js"; +import type { Evidence } from "../domain/evidence.js"; +import { toolContract } from "../contracts/toolContracts.js"; +import type { ToolContract } from "../contracts/toolContractTypes.js"; +import { logToolExecution } from "./toolLogging.js"; +import { toolRegistrationOptions } from "./toolRegistrationOptions.js"; +import type { ToolResultDelivery } from "./toolResult.js"; +import type { WithAdmittedAnalysis } from "./analysisAdmission.js"; +import { rizinInputSchema } from "../domain/reverseEngineering.js"; +import { AnalysisOutputError } from "../domain/analysisErrorCore.js"; +import type { AnalysisError } from "../domain/analysisErrorBase.js"; +import type { Result } from "../domain/result.js"; + +/** Register objdump and Rizin operations against their exact named contracts. */ +export const registerReverseEngineeringTools = ( + server: McpServer, + service: ReverseEngineeringService, + logger: Logger, + delivery: ToolResultDelivery, + recordEvidence?: EvidenceWriter["recordEvidence"], + withAdmittedAnalysis?: WithAdmittedAnalysis, + invalidateSnapshotForExternalMutation?: () => void, + withAdmittedExternalMutation?: ( + path: string | undefined, + signal: AbortSignal | undefined, + operation: () => Promise>, + retirementFailure: ( + value: Result, + error: AnalysisError, + ) => AnalysisError, + ) => Promise, AnalysisError>>, +): void => { + const objdump = toolContract("inspect_with_objdump"); + const rizin = toolContract("execute_rizin_command"); + server.registerTool( + objdump.name, + toolRegistrationOptions(objdump), + (input, context: ServerContext) => + runAdmitted( + objdump, + context, + withAdmittedAnalysis, + delivery, + async () => { + const result = await logToolExecution(logger, objdump.name, () => + service.inspectWithObjdump(input, { + signal: context.mcpReq.signal, + }), + ); + if (!result.ok) + return deliverFailure( + result.error, + objdump, + delivery, + recordEvidence, + ); + return delivery.toEvidenceToolResult( + result.value, + objdump, + recordEvidence?.(result.value), + ); + }, + ), + ); + server.registerTool( + rizin.name, + toolRegistrationOptions(rizin), + async (input, context: ServerContext) => { + const parsed = rizinInputSchema.safeParse(input); + if (parsed.success && withAdmittedExternalMutation !== undefined) { + const cleanup = await service.retryCleanup(rizin.name); + if (!cleanup.ok) + return deliverFailure(cleanup.error, rizin, delivery, recordEvidence); + const admitted = await withAdmittedExternalMutation( + parsed.data.path, + context.mcpReq.signal, + async () => { + invalidateSnapshotForExternalMutation?.(); + return logToolExecution(logger, rizin.name, () => + service.executeRizinCommand(parsed.data, { + signal: context.mcpReq.signal, + }), + ); + }, + (value, cause) => { + if (!value.ok) + return value.error.retainSecondaryFailure(cause, { + retirement_error: projectAnalysisError(cause), + }); + const observation = value.value; + return new AnalysisOutputError( + rizin.name, + "Rizin may have changed the bound artifact, but the active analysis session could not be retired.", + { + cause, + ...(cause.cleanup === undefined + ? {} + : { cleanup: cause.cleanup }), + ...(cause.capturedOutput === undefined + ? {} + : { capturedOutput: cause.capturedOutput }), + ...(observation === undefined + ? {} + : { partialObservation: observation }), + }, + ); + }, + ); + if (!admitted.ok) + return deliverFailure( + admitted.error, + rizin, + delivery, + recordEvidence, + ); + const result = admitted.value; + if (!result.ok) + return deliverFailure(result.error, rizin, delivery, recordEvidence); + return delivery.toEvidenceToolResult( + result.value, + rizin, + recordEvidence?.(result.value), + ); + } + return runAdmitted( + rizin, + context, + withAdmittedAnalysis, + delivery, + async () => { + if ( + invalidateSnapshotForExternalMutation !== undefined && + rizinInputSchema.safeParse(input).success + ) + invalidateSnapshotForExternalMutation(); + const result = await logToolExecution(logger, rizin.name, () => + service.executeRizinCommand(input, { + signal: context.mcpReq.signal, + }), + ); + if (!result.ok) + return deliverFailure( + result.error, + rizin, + delivery, + recordEvidence, + ); + return delivery.toEvidenceToolResult( + result.value, + rizin, + recordEvidence?.(result.value), + ); + }, + ); + }, + ); +}; + +const runAdmitted = async ( + contract: ToolContract, + context: ServerContext, + admission: WithAdmittedAnalysis | undefined, + delivery: ToolResultDelivery, + operation: () => Promise, +): Promise => { + if (admission === undefined) return operation(); + const admitted = await admission( + contract.name, + context.mcpReq.signal, + async () => operation(), + ); + return admitted.ok + ? admitted.value + : delivery.toErrorToolResult(admitted.error, contract.returnsTargetContent); +}; + +const deliverFailure = ( + error: import("../domain/analysisErrorBase.js").AnalysisError, + contract: ToolContract, + delivery: ToolResultDelivery, + recordEvidence?: EvidenceWriter["recordEvidence"], +) => { + const partial = error.partialObservation; + if (isEvidence(partial) && recordEvidence !== undefined) + return delivery.toRecordedPartialErrorToolResult( + error, + partial.evidence_id, + recordEvidence(partial), + contract.returnsTargetContent, + ); + return delivery.toErrorToolResult(error, contract.returnsTargetContent); +}; + +const isEvidence = (value: unknown): value is Evidence => + typeof value === "object" && + value !== null && + "evidence_id" in value && + typeof value.evidence_id === "string"; diff --git a/src/server/registerRizinDebugTools.ts b/src/server/registerRizinDebugTools.ts new file mode 100644 index 0000000000..1c6b2b8a69 --- /dev/null +++ b/src/server/registerRizinDebugTools.ts @@ -0,0 +1,146 @@ +import type { McpServer, ServerContext } from "@modelcontextprotocol/server"; +import type { Logger } from "pino"; + +import type { EvidenceWriter } from "../application/investigation/InvestigationRecordPort.js"; +import { toolContract } from "../contracts/toolContracts.js"; +import { AnalysisInputError } from "../domain/analysisErrorCore.js"; +import type { RizinDebugSessionManager } from "../rizin/RizinDebugSessionManager.js"; +import { evidenceSchema } from "../domain/evidence.js"; +import { + runOwnedCleanupWithExternalMutationAdmission, + runWithExternalMutationAdmission, + type ExternalMutationAdmission, +} from "./externalMutationAdmission.js"; +import { logToolExecution } from "./toolLogging.js"; +import { toolRegistrationOptions } from "./toolRegistrationOptions.js"; +import type { ToolResultDelivery } from "./toolResult.js"; + +export type RizinDebugToolManager = Pick< + RizinDebugSessionManager, + "start" | "execute" | "status" | "close" | "retryCleanupFailures" +>; + +/** Register persistent Rizin debugger operations on the existing REA MCP server. */ +export const registerRizinDebugTools = ( + server: McpServer, + manager: RizinDebugToolManager, + logger: Logger, + delivery: ToolResultDelivery, + recordEvidence?: EvidenceWriter["recordEvidence"], + session?: ExternalMutationAdmission, +): void => { + const start = toolContract("start_rizin_debug_session"); + const command = toolContract("rizin_debug_command"); + const status = toolContract("rizin_debug_session_status"); + const close = toolContract("close_rizin_debug_session"); + server.registerTool( + start.name, + toolRegistrationOptions(start), + async (input, context: ServerContext) => { + const result = await runWithExternalMutationAdmission( + session, + start.name, + context.mcpReq.signal, + () => + logToolExecution(logger, start.name, () => + manager.start( + { + path: input.path, + inferior_tty: input.inferior_tty, + ...(input.backend === undefined + ? {} + : { backend: input.backend }), + }, + context.mcpReq.signal, + ), + ), + undefined, + () => manager.retryCleanupFailures(), + (value) => ({ session_id: value.session_id }), + ); + return result.ok + ? delivery.toCallToolResult({ ok: true, value: result.value }, start) + : delivery.toCallToolResult(result, start); + }, + ); + server.registerTool( + command.name, + toolRegistrationOptions(command), + async (input, context: ServerContext) => { + const result = await runWithExternalMutationAdmission( + session, + command.name, + context.mcpReq.signal, + () => + logToolExecution(logger, command.name, () => + manager.execute( + input.session_id, + input.command, + context.mcpReq.signal, + ), + ), + (value) => value.evidence, + () => manager.retryCleanupFailures(), + ); + if (!result.ok) { + const partial = evidenceSchema.safeParse( + result.error.partialObservation, + ); + if (partial.success) { + const recorded = recordEvidence?.(partial.data); + if (recorded !== undefined) + return delivery.toRecordedPartialErrorToolResult( + result.error, + partial.data.evidence_id, + recorded, + command.returnsTargetContent, + ); + } + return delivery.toCallToolResult(result, command); + } + return delivery.toEvidenceToolResult( + result.value.evidence, + command, + recordEvidence?.(result.value.evidence), + ); + }, + ); + server.registerTool( + status.name, + toolRegistrationOptions(status), + async (input) => { + const result = manager.status(input.session_id); + if (result === undefined) + return delivery.toCallToolResult( + { ok: false, error: new AnalysisInputError(status.name) }, + status, + ); + return delivery.toCallToolResult( + { + ok: true, + value: { ...result, recent_output: [...result.recent_output] }, + }, + status, + ); + }, + ); + server.registerTool( + close.name, + toolRegistrationOptions(close), + async (input, context: ServerContext) => { + const result = await runOwnedCleanupWithExternalMutationAdmission( + session, + close.name, + context.mcpReq.signal, + input.session_id, + () => + logToolExecution(logger, close.name, () => + manager.close(input.session_id), + ), + ); + return result.ok + ? delivery.toCallToolResult({ ok: true, value: result.value }, close) + : delivery.toCallToolResult(result, close); + }, + ); +}; diff --git a/src/server/registerSessionStatusTool.ts b/src/server/registerSessionStatusTool.ts index 563e7528b9..08587b0a7b 100644 --- a/src/server/registerSessionStatusTool.ts +++ b/src/server/registerSessionStatusTool.ts @@ -2,6 +2,7 @@ import type { EvidenceMcpServer } from "./EvidenceMcpServer.js"; import type { BinarySessionPort } from "../application/binary/BinarySessionPort.js"; import type { ProviderAvailability } from "../application/AnalysisProvider.js"; +import type { JebAnalysisAvailability } from "../application/jeb/JebAnalysisPort.js"; import { AnalysisCancelledError } from "../domain/analysisErrorCore.js"; import { buildCapabilityInventory, @@ -18,17 +19,31 @@ import { mcpClientMetadata } from "./mcpClientMetadata.js"; import type { AvailabilityPolicy } from "../application/CapabilityInventory.js"; import { toolRegistrationOptions } from "./toolRegistrationOptions.js"; import { err } from "../domain/result.js"; +import { CURRENT_ENTRYPOINT_PATH } from "../process/CurrentEntrypointPath.js"; + +/** Provider-owned live probes used by session discovery. */ +export interface SessionAvailabilityProbes { + readonly androidAnalysisAvailability: ( + signal: AbortSignal, + ) => Promise; + readonly adbAnalysisAvailability?: ( + signal: AbortSignal, + ) => Promise; + readonly apktoolAnalysisAvailability?: ( + signal: AbortSignal, + ) => Promise; + readonly jebAnalysisAvailability?: ( + signal: AbortSignal, + ) => Promise; +} /** Inputs required to register the binary session status tool. */ -export interface SessionStatusToolOptions { +export interface SessionStatusToolOptions extends SessionAvailabilityProbes { readonly server: EvidenceMcpServer; readonly session: BinarySessionPort; readonly contract: ReturnType>; readonly startedAt: string; readonly availabilityPolicy: () => AvailabilityPolicy; - readonly androidAnalysisAvailability: ( - signal: AbortSignal, - ) => Promise; } /** Register the read-only provider and target status operation. */ @@ -42,6 +57,9 @@ export const registerSessionStatusTool = ( startedAt, availabilityPolicy, androidAnalysisAvailability, + adbAnalysisAvailability, + apktoolAnalysisAvailability, + jebAnalysisAvailability, } = options; server.registerTool( contract.name, @@ -52,10 +70,20 @@ export const registerSessionStatusTool = ( server.server, ); let androidAvailability: ProviderAvailability; + let adbAvailability: ProviderAvailability | undefined; + let apktoolAvailability: ProviderAvailability | undefined; + let jebAvailability: JebAnalysisAvailability | undefined; try { - androidAvailability = await androidAnalysisAvailability( - context.mcpReq.signal, - ); + const [android, adb, apktool, jeb] = await Promise.all([ + androidAnalysisAvailability(context.mcpReq.signal), + adbAnalysisAvailability?.(context.mcpReq.signal) ?? undefined, + apktoolAnalysisAvailability?.(context.mcpReq.signal) ?? undefined, + jebAnalysisAvailability?.(context.mcpReq.signal) ?? undefined, + ] as const); + androidAvailability = android; + adbAvailability = adb; + apktoolAvailability = apktool; + jebAvailability = jeb; } catch (cause) { if (!context.mcpReq.signal.aborted) throw cause; return server.delivery.toCallToolResult( @@ -70,11 +98,22 @@ export const registerSessionStatusTool = ( { ...availabilityPolicy(), androidAnalysisAvailability: androidAvailability, + ...(adbAvailability === undefined + ? {} + : { adbAnalysisAvailability: adbAvailability }), + ...(apktoolAvailability === undefined + ? {} + : { apktoolAnalysisAvailability: apktoolAvailability }), + ...(jebAvailability === undefined + ? {} + : { jebAnalysisAvailability: jebAvailability }), }, clientFeatures, ); const serverIdentity = createServerIdentity({ startedAt, + commandPath: CURRENT_ENTRYPOINT_PATH, + catalog: server.catalogIdentity, expected: { ...(input.expected_package_version === undefined ? {} diff --git a/src/server/registerSessionTools.ts b/src/server/registerSessionTools.ts index 16b9e64084..8053abf660 100644 --- a/src/server/registerSessionTools.ts +++ b/src/server/registerSessionTools.ts @@ -12,7 +12,6 @@ import { captureProcessScenario } from "../process/capture/ProcessHarness.js"; import { ProcessCaptureResourceScope } from "../process/capture/ProcessCaptureLifecycle.js"; import { ProcessCaptureError } from "../process/capture/ProcessCaptureError.js"; import { toolContract } from "../contracts/toolContracts.js"; -import type { AnalysisSnapshot } from "../domain/analysisSnapshot.js"; import { binaryTargetSummary } from "../application/binary/BinaryTargetSummary.js"; import { UnknownRegistryError } from "../domain/unknownRegistryError.js"; import type { AnalysisError } from "../domain/analysisErrorBase.js"; @@ -32,7 +31,10 @@ import { registerEvidenceTools, registerUnknownTools, } from "./registerSessionRecordTools.js"; -import { registerSessionStatusTool } from "./registerSessionStatusTool.js"; +import { + registerSessionStatusTool, + type SessionAvailabilityProbes, +} from "./registerSessionStatusTool.js"; import { sessionAvailabilityPolicy } from "./sessionAvailabilityPolicy.js"; import type { AvailabilityPolicy } from "../application/CapabilityInventory.js"; import { logToolExecution } from "./toolLogging.js"; @@ -143,7 +145,7 @@ const registerProcessTools = ({ ); }; -export interface LifecycleToolRegistration { +export interface LifecycleToolRegistration extends SessionAvailabilityProbes { readonly server: EvidenceMcpServer; readonly session: BinarySessionPort; readonly logger: Logger; @@ -152,9 +154,6 @@ export interface LifecycleToolRegistration { readonly statusContract: ReturnType>; readonly startedAt: string; readonly availabilityPolicy: () => AvailabilityPolicy; - readonly androidAnalysisAvailability: ( - signal: AbortSignal, - ) => Promise; readonly closeProcessResources: () => Promise>; } @@ -162,23 +161,11 @@ const registerLifecycleTools = ( registration: LifecycleToolRegistration, artifactResources: ArtifactResourceScope, ): void => { - const { - server, - session, - statusContract, - startedAt, - availabilityPolicy, - androidAnalysisAvailability, - } = registration; registerOpenLifecycleTool(registration, artifactResources); registerCloseLifecycleTool(registration); registerSessionStatusTool({ - server, - session, - contract: statusContract, - startedAt, - availabilityPolicy, - androidAnalysisAvailability, + ...registration, + contract: registration.statusContract, }); }; @@ -190,17 +177,7 @@ const registerOpenLifecycleTool = ( openContract.name, toolRegistrationOptions(openContract), async (input, context) => { - let snapshot: AnalysisSnapshot | undefined; - if (input.snapshot_path !== undefined) { - const loaded = await readAnalysisSnapshot( - input.snapshot_path, - context.mcpReq.signal, - artifactResources, - ); - if (!loaded.ok) - return server.delivery.toCallToolResult(loaded, openContract); - snapshot = loaded.value; - } + const snapshotPath = input.snapshot_path; const opened = await logToolExecution(logger, openContract.name, () => session.open(input.path, { signal: context.mcpReq.signal, @@ -214,7 +191,18 @@ const registerOpenLifecycleTool = ( ...(input.provider_id === undefined ? {} : { providerId: input.provider_id }), - ...(snapshot === undefined ? {} : { snapshot }), + ...(snapshotPath === undefined + ? {} + : { + // Read inside the open's turn: calls issued after it must not + // overtake a file read that precedes its place in the order. + loadSnapshot: () => + readAnalysisSnapshot( + snapshotPath, + context.mcpReq.signal, + artifactResources, + ), + }), }), ); return opened.ok @@ -231,13 +219,10 @@ const registerOpenLifecycleTool = ( }; /** Register MCP-only target lifecycle operations on a long-lived session. */ -export interface SessionToolOptions { +export interface SessionToolOptions extends Partial { readonly artifactResources: ArtifactResourceScope; readonly startedAt?: string; readonly availabilityPolicy?: () => AvailabilityPolicy; - readonly androidAnalysisAvailability?: ( - signal: AbortSignal, - ) => Promise; } const registerContextTools = ( @@ -304,6 +289,7 @@ export const registerSessionTools = ( const processResources = new ProcessCaptureResourceScope(); registerLifecycleTools( { + ...options, server, session, logger, diff --git a/src/server/retainedSummaryDelivery.ts b/src/server/retainedSummaryDelivery.ts new file mode 100644 index 0000000000..0bc703a7c1 --- /dev/null +++ b/src/server/retainedSummaryDelivery.ts @@ -0,0 +1,62 @@ +import type { EvidenceWriter } from "../application/investigation/InvestigationRecordPort.js"; +import type { EvidenceLookup } from "../application/EvidenceInputResolver.js"; +import { + requireAnalysisViewRetention, + summarizeRetainedAnalysis, +} from "../application/analysisView/AnalysisViewService.js"; +import type { BinaryTarget } from "../domain/binaryTargetTypes.js"; +import type { Evidence } from "../domain/evidence.js"; +import type { JsonValue } from "../domain/jsonValue.js"; +import { ok } from "../domain/result.js"; +import type { Logger } from "pino"; +import type { WithAdmittedAnalysis } from "./analysisAdmission.js"; + +/** Registration inputs shared by provider-backed Evidence tool families. */ +export interface EvidenceToolFamilyOptions { + readonly logger: Logger; + readonly activeTarget: (() => BinaryTarget | undefined) | undefined; + readonly recordEvidence: EvidenceWriter["recordEvidence"] | undefined; + readonly evidenceById?: EvidenceLookup | undefined; + readonly withAdmittedAnalysis: WithAdmittedAnalysis; +} + +type Parameters = Readonly>; + +/** + * Summary delivery for the named operations: refuse it before analysis when + * this server retains no Evidence, then return the retained summary view. + */ +export const retainedSummaryDelivery = ( + operations: ReadonlySet, + options: Pick, +) => { + const retention = { + recordEvidence: options.recordEvidence, + evidenceById: options.evidenceById, + }; + const selected = (operation: string, parameters: Parameters): boolean => + operations.has(operation) && parameters.detail === "summary"; + return { + preflight: (operation: string, parameters: Parameters) => + selected(operation, parameters) + ? requireAnalysisViewRetention(operation, retention) + : ok(null), + projectEvidence: ( + operation: string, + parameters: Parameters, + evidence: Evidence, + ) => + selected(operation, parameters) + ? summarizeRetainedAnalysis(evidence, retention) + : ok(evidence), + }; +}; + +/** Delivery detail selects a response shape; it never reaches tools or Evidence identity. */ +export const withoutDeliveryDetail = + (operations: ReadonlySet) => + (operation: string, parameters: Parameters): Parameters => { + if (!operations.has(operation)) return parameters; + const { detail: _detail, ...selected } = parameters; + return selected; + }; diff --git a/src/server/toolResult.ts b/src/server/toolResult.ts index 5d40afb5b1..63a45f872f 100644 --- a/src/server/toolResult.ts +++ b/src/server/toolResult.ts @@ -1,12 +1,10 @@ +import { INSPECT_ANALYSIS_VIEW_OPERATION } from "../domain/analysisView/analysisViewErrors.js"; import type { CallToolResult } from "@modelcontextprotocol/server"; import { STDIO_DEFAULT_MAX_BUFFER_SIZE } from "@modelcontextprotocol/server"; import type { ToolContract } from "../contracts/toolContractTypes.js"; import type { Evidence } from "../domain/evidence.js"; -import { - INSPECT_ANALYSIS_VIEW_OPERATION, - inspectAnalysisViewInputSchema, -} from "../domain/analysisView/analysisView.js"; +import { inspectAnalysisViewInputSchema } from "../domain/analysisView/analysisView.js"; import { projectAnalysisError, type AnalysisErrorProjection, @@ -48,7 +46,6 @@ const nativeEvidenceContinuation = ( /** Immutable encoding and recovery policy shared by one MCP server. */ export class ToolResultDelivery { readonly resultBudgetBytes: number; - constructor(maximumResponseBytes: number) { this.resultBudgetBytes = maximumResponseBytes - 1024; Object.freeze(this); diff --git a/src/server/toolResultEncoding.ts b/src/server/toolResultEncoding.ts index 73ec2c717e..dc563e1054 100644 --- a/src/server/toolResultEncoding.ts +++ b/src/server/toolResultEncoding.ts @@ -63,6 +63,26 @@ export type ToolResultEncoding = readonly constraint: "receive-buffer" | "string-length"; }; +/** Measure an already-formed MCP result object without allocating its JSON text. */ +export const measureMcpResultWithinBudget = ( + value: unknown, + budgetBytes: number, +): + | { readonly ok: true; readonly bytes: number } + | { + readonly ok: false; + readonly bytesAtLeast: number; + readonly constraint: "receive-buffer"; + } => { + let bytes = 0; + for (const part of bufferedJsonParts(jsonParts(value))) { + bytes += Buffer.byteLength(part); + if (bytes > budgetBytes) + return { ok: false, bytesAtLeast: bytes, constraint: "receive-buffer" }; + } + return { ok: true, bytes }; +}; + /** Account for the delivered JSON representations before allocating the complete text. */ export const encodeToolResult = ( value: JsonValue, diff --git a/src/serverIdentity.ts b/src/serverIdentity.ts index 55a4f1a3b4..12d88b5f47 100644 --- a/src/serverIdentity.ts +++ b/src/serverIdentity.ts @@ -1,7 +1,10 @@ import { dirname, basename, resolve } from "node:path"; import { fileURLToPath } from "node:url"; -import { CATALOG_IDENTITY } from "./catalogIdentity.js"; +import { + CATALOG_IDENTITY, + type createCatalogIdentity, +} from "./catalogIdentity.js"; import { PRODUCT_IDENTITY, SDK_IDENTITY } from "./identity.js"; /** Optional observations supplied by a client comparing its expected server. */ @@ -27,11 +30,14 @@ const packageRoot = (): string => { /** Build one runtime identity without claiming alignment from absent observations. */ export const createServerIdentity = (input: { readonly startedAt: string; + readonly commandPath: string | undefined; readonly expected?: ExpectedServerIdentity; readonly client?: ConnectedClientIdentity; readonly protocolVersion?: string | undefined; + readonly catalog?: ReturnType; }) => { - const serverPath = resolve(process.argv[1] ?? "unknown"); + const catalog = input.catalog ?? CATALOG_IDENTITY; + const serverPath = input.commandPath || undefined; const reasons: string[] = []; if ( input.expected?.package_version !== undefined && @@ -40,17 +46,20 @@ export const createServerIdentity = (input: { reasons.push("package_version_mismatch"); if ( input.expected?.catalog_digest !== undefined && - input.expected.catalog_digest !== CATALOG_IDENTITY.digests.combined_sha256 + input.expected.catalog_digest !== catalog.digests.combined_sha256 ) reasons.push("catalog_digest_mismatch"); if ( input.expected?.server_path !== undefined && + serverPath !== undefined && resolve(input.expected.server_path) !== serverPath ) reasons.push("registration_path_mismatch"); const observedExpectation = input.expected !== undefined && Object.values(input.expected).some((value) => value !== undefined); + const pathComparable = + input.expected?.server_path === undefined || serverPath !== undefined; return { package: { name: PRODUCT_IDENTITY.packageName, @@ -62,7 +71,7 @@ export const createServerIdentity = (input: { name: PRODUCT_IDENTITY.mcpServerKey, version: PRODUCT_IDENTITY.packageVersion, started_at: input.startedAt, - command_path: serverPath, + command_path: serverPath ?? null, }, sdk: SDK_IDENTITY, negotiated_protocol_version: input.protocolVersion ?? null, @@ -74,7 +83,10 @@ export const createServerIdentity = (input: { name: PRODUCT_IDENTITY.skillName, expected_version: PRODUCT_IDENTITY.skillVersion, }, - catalog: CATALOG_IDENTITY, + catalog: { + counts: catalog.counts, + digests: catalog.digests, + }, protocol_features: { progress: true, cancellation: true, @@ -84,16 +96,19 @@ export const createServerIdentity = (input: { state: reasons.length > 0 ? "mcp_server_restart_required" - : observedExpectation + : observedExpectation && pathComparable ? "aligned" : "unknown", reasons, remediation: reasons.length > 0 ? "Restart the registered MCP server/client, then compare identity again." - : observedExpectation + : observedExpectation && pathComparable ? null - : "Supply expected package, catalog, or registration identity to compare the live server.", + : input.expected?.server_path !== undefined && + serverPath === undefined + ? "The server entrypoint path is unavailable, so registration identity cannot be compared." + : "Supply expected package, catalog, or registration identity to compare the live server.", }, } as const; }; diff --git a/src/setupEnvironment.test.ts b/src/setupEnvironment.test.ts index ce1c7d7d8b..15d57a1319 100644 --- a/src/setupEnvironment.test.ts +++ b/src/setupEnvironment.test.ts @@ -5,7 +5,9 @@ import { join } from "node:path"; import { afterEach, expect, it, vi } from "vitest"; import { createDoctorHostFixture } from "./application/Doctor.fixture.js"; +import * as macHopperInstaller from "./application/MacHopper.js"; import { runSetup } from "./application/Setup.js"; +import { discoverSetupState } from "./application/SetupPlan.js"; import { options } from "./application/Setup.fixture.js"; import { systemSetupHost } from "./application/SetupHost.js"; import { systemUninstallHost } from "./application/Uninstall.js"; @@ -92,6 +94,143 @@ it("composes provider diagnostics from selected configuration rather than ambien expect(JSON.stringify(inspections)).not.toContain("/ambient/"); }); +it("includes the macOS Hopper installer diagnostic in setup remediation", async () => { + const install = vi + .spyOn(macHopperInstaller, "installMacHopper") + .mockResolvedValue({ + status: "failed", + reason: "mount", + diagnostic: "hdiutil attach exited 1:\n\u001b[31mresource busy\u001b[0m", + }); + const host = systemSetupHost( + createDoctorHostFixture({ platform: "darwin" }), + { HOME: "/fixture/home", USERPROFILE: "/fixture/home" }, + ); + try { + const result = await host.installHopper(false); + expect(result).toMatchObject({ + status: "failed", + code: "mount_failed", + remediation: expect.stringContaining( + "Installer detail: hdiutil attach exited 1: resource busy", + ), + }); + if (result.status !== "failed") throw new Error("expected setup failure"); + expect(result.remediation).not.toContain("\u001b"); + } finally { + await host.close?.(); + install.mockRestore(); + } +}); + +it("preserves a validated selected JEB endpoint in setup registrations without claiming live readiness", async () => { + const root = await mkdtemp(join(tmpdir(), "rea-jeb-setup-")); + const endpoint = "http://127.0.0.1:8425/mcp/"; + const ambientEndpoint = "http://127.0.0.1:8426/mcp"; + vi.stubEnv("REA_JEB_MCP_URL", ambientEndpoint); + try { + const environment = { + HOME: root, + USERPROFILE: root, + CODEX_HOME: join(root, "codex"), + REA_JEB_MCP_URL: endpoint, + }; + const doctorHost = createSystemDoctorHost(environment); + const setupHost = systemSetupHost(doctorHost, environment); + const state = await setupHost.initialSetupState(); + + expect(state.providerEnvironment).toHaveProperty( + "REA_JEB_MCP_URL", + endpoint, + ); + const planned = await runSetup( + { + ...options(false), + dryRun: true, + clientIds: ["codex"], + installSkill: false, + }, + setupHost, + ); + expect(planned.status).toBe("planned"); + expect(JSON.stringify(planned.plannedActions)).toContain(endpoint); + expect(JSON.stringify(planned.plannedActions)).not.toContain( + ambientEndpoint, + ); + + expect( + createSystemDoctorHost({}).providerRegistrationEnvironment?.(), + ).toEqual({}); + expect( + createSystemDoctorHost({ + REA_JEB_MCP_URL: "http://127.0.0.1:8425/mcp", + }).providerRegistrationEnvironment?.(), + ).toEqual({ REA_JEB_MCP_URL: "http://127.0.0.1:8425/mcp" }); + expect( + createSystemDoctorHost({ + REA_JEB_MCP_URL: "https://example.com/mcp", + }).providerRegistrationEnvironment?.(), + ).toEqual({}); + } finally { + vi.unstubAllEnvs(); + await rm(root, { recursive: true, force: true }); + } +}); + +it.each([ + { + name: "registration-only settings", + environment: { + REA_ANALYSIS_PROVIDER: "auto", + REA_GHIDRA_STARTUP_TIMEOUT_MS: "2700000", + REA_MCP_INPUT_SCHEMA_PROFILE: "full", + }, + providerConfigured: false, + installHopper: true, + }, + { + name: "an explicit unready provider selection", + environment: { REA_ANALYSIS_PROVIDER: "ghidra" }, + providerConfigured: true, + installHopper: false, + }, +])( + "derives Hopper proposal behavior from setup-host facts for $name", + async ({ environment, providerConfigured, installHopper }) => { + const doctorHost = createDoctorHostFixture({ + platform: "darwin", + validTarget: () => Promise.resolve(false), + executable: () => Promise.resolve(false), + }); + const setupHost = systemSetupHost(doctorHost, { + HOME: "/fixture/home", + USERPROFILE: "/fixture/home", + ...environment, + }); + try { + const initial = await setupHost.initialSetupState(); + expect(initial.providerConfigured).toBe(providerConfigured); + if ("REA_MCP_INPUT_SCHEMA_PROFILE" in environment) { + expect(initial.providerEnvironment).toMatchObject({ + REA_ANALYSIS_PROVIDER: "auto", + REA_GHIDRA_STARTUP_TIMEOUT_MS: "2700000", + REA_MCP_INPUT_SCHEMA_PROFILE: "full", + }); + } + const discovery = await discoverSetupState({ + host: setupHost, + providerConfigured: initial.providerConfigured, + forceHopperInstall: false, + proposeHopper: true, + initialDoctor: initial.doctor, + }); + expect(discovery.installHopper).toBe(installHopper); + } finally { + await setupHost.close?.(); + } + }, +); + it("passes only selected environment to updater subprocesses and drops npm invocation identity", async () => { vi.stubEnv("REA_UPDATE_MARKER", "ambient"); const result = await runUpdateCommand( @@ -128,3 +267,31 @@ it.each([ }); }, ); + +it("registers the selected Windows package runner after canonicalizing environment aliases", async () => { + const environment = snapshotEnvironment( + { + NPM_COMMAND: "run", + npm_command: "exec", + NPM_CONFIG_USER_AGENT: "npm/10.9.0 node/v24.11.0 win32 x64", + }, + "win32", + ); + expect(environment).toEqual({ + NPM_COMMAND: "exec", + NPM_CONFIG_USER_AGENT: "npm/10.9.0 node/v24.11.0 win32 x64", + }); + expect(invokingPackageRunner(environment, "win32")).toBe("npx"); + + const host = systemSetupHost( + createDoctorHostFixture({ platform: "win32" }), + environment, + ); + try { + expect(host.registrationCommand).toEqual( + packageRunnerRegistrationCommand("npx", "win32"), + ); + } finally { + await host.close?.(); + } +}); diff --git a/src/signals/SigrokRecordingProvider.ts b/src/signals/SigrokRecordingProvider.ts index 541138a6a4..816556c62a 100644 --- a/src/signals/SigrokRecordingProvider.ts +++ b/src/signals/SigrokRecordingProvider.ts @@ -162,6 +162,7 @@ export class SigrokRecordingProvider implements SignalRecordingPort { this.#resources.retain(root, undefined); const snapshotPath = join(root.path, "recording"); const execute = async (args: readonly string[]) => { + capturedOutput = undefined; const output = await runOwnedCommand( { command: limiter, diff --git a/src/sqlite/SqliteDatabaseFailures.ts b/src/sqlite/SqliteDatabaseFailures.ts index 06bf4e71ed..09a1c08677 100644 --- a/src/sqlite/SqliteDatabaseFailures.ts +++ b/src/sqlite/SqliteDatabaseFailures.ts @@ -7,6 +7,7 @@ import { import { AnalysisAccessDeniedError, AnalysisArtifactChangedError, + AnalysisCapabilityUnavailableError, AnalysisCancelledError, AnalysisInputError, AnalysisOutputError, @@ -29,6 +30,15 @@ export const capturedSqliteOutput = ( truncated: snapshot.diagnosticTruncated === true, }); +/** The file-set member a filesystem error names, if it is the database, its WAL or its journal. */ +const busyMember = (cause: Error, path: string): string => { + const failed = "path" in cause ? cause.path : undefined; + return typeof failed === "string" && + [path, `${path}-wal`, `${path}-journal`].includes(failed) + ? failed + : path; +}; + /** Preserve artifact, caller, process, resource and output failure meanings. */ export const sqliteDatabaseFailure = ( cause: unknown, @@ -85,6 +95,21 @@ export const sqliteDatabaseFailure = ( return new AnalysisAccessDeniedError(operation, path, cause.code, { cause, }); + // Windows reports a sharing violation as EBUSY: another process holds the file without read + // sharing. The unchanged request succeeds once that process releases it. Name the member of the + // file set that was busy (the database, its WAL or its journal) when the error identifies it. + if (cause.code === "EBUSY") + return new AnalysisCapabilityUnavailableError( + SQLITE_PROVIDER_IDENTITY.id, + operation, + `Selected database file set is in use by another process (EBUSY): ${busyMember(cause, path)}`, + { + cause, + retryable: true, + userMessage: + "The selected SQLite database is held open by another process without read sharing. Close that application or inspect a copy, then retry.", + }, + ); if (cause.code === "ENOENT" || cause.code === "ENOTDIR") return new AnalysisInputError(operation, { cause }, [ { diff --git a/src/sqlite/SqliteDatabaseProvider.ts b/src/sqlite/SqliteDatabaseProvider.ts index c44185616f..7572034d87 100644 --- a/src/sqlite/SqliteDatabaseProvider.ts +++ b/src/sqlite/SqliteDatabaseProvider.ts @@ -17,6 +17,7 @@ import { AnalysisResourceConstraintError, } from "../domain/analysisErrorCore.js"; import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; +import type { JsonValue } from "../domain/jsonValue.js"; import { createEvidence } from "../domain/evidence.js"; import { ProviderCleanupError } from "../domain/providerCleanupError.js"; import { err, ok, type Result } from "../domain/result.js"; @@ -29,8 +30,14 @@ import { } from "../domain/sqlite/sqliteDatabase.js"; import { isAbsolute } from "node:path"; import { PrivateRuntimeRoot } from "../process/PrivateRuntimeRoot.js"; -import { OwnedCommandFailure } from "../process/OwnedCommand.js"; -import type { ProviderProcessSupervisor } from "../process/ProviderProcess.js"; +import { + OwnedCommandFailure, + ownedCommandResources, +} from "../process/OwnedCommand.js"; +import type { + ProviderProcessStopResult, + ProviderProcessSupervisor, +} from "../process/ProviderProcess.js"; import { sqliteDatabaseFailure } from "./SqliteDatabaseFailures.js"; import { captureSqliteDatabaseSnapshot } from "./SqliteDatabaseSnapshot.js"; import { SQLITE_DATABASE_LIMITS } from "./SqliteDatabaseLimits.js"; @@ -228,6 +235,9 @@ export class SqliteDatabaseProvider implements SqliteDatabasePort { ): Promise { let root: Pick | undefined; let processOwner: ProviderProcessSupervisor | undefined; + let processCleanup: + | Extract + | undefined; let retainedOutput: AnalysisCapturedOutput | undefined; let phase = "configuration"; let result: Result; @@ -258,6 +268,10 @@ export class SqliteDatabaseProvider implements SqliteDatabasePort { captured: (output) => { retainedOutput = output; }, + retainCleanup: (owner, result) => { + processOwner = owner; + processCleanup = result; + }, }); const validated = sqliteDatabaseSchema.safeParse({ artifact: snapshot.artifact, @@ -274,7 +288,12 @@ export class SqliteDatabaseProvider implements SqliteDatabasePort { ); result = ok(createSqliteObservation(validated.data, input.path)); } catch (cause: unknown) { - if (cause instanceof OwnedCommandFailure) + // A failure rethrown after retainCleanup carries no owner of its own; + // keep the retained worker instead of discarding it. + if ( + cause instanceof OwnedCommandFailure && + cause.cleanupOwner !== undefined + ) processOwner = cause.cleanupOwner; const failure = sqliteDatabaseFailure( cause, @@ -293,6 +312,39 @@ export class SqliteDatabaseProvider implements SqliteDatabasePort { output: retainedOutput, process: processOwner, }); + if (processCleanup !== undefined) { + const ownership = processOwner.launch.ownership; + return err( + sqliteCleanupFailure( + ownership === undefined + ? ["owned-sqlite-worker", root.path] + : ownedCommandResources(ownership, root.path), + processCleanup.reason, + result, + input, + retainedOutput, + processCleanup.cleanupResult === undefined + ? undefined + : { + ...(processCleanup.cleanupResult.failures === undefined + ? {} + : { + failures: processCleanup.cleanupResult.failures.map( + (failure) => ({ ...failure }), + ), + }), + ...(processCleanup.cleanupResult.unverified === undefined + ? {} + : { + unverified: + processCleanup.cleanupResult.unverified.map( + (process) => ({ ...process }), + ), + }), + }, + ), + ); + } return result; } const cleaned = await cleanupSqliteRoot( @@ -360,41 +412,55 @@ const cleanupSqliteRoot = async ( return ok(null); } catch (cause: unknown) { return err( - new ProviderCleanupError( - SQLITE_PROVIDER_IDENTITY.id, + sqliteCleanupFailure( [root.path], - { - reason: cause instanceof Error ? cause.message : String(cause), - previous_error: result.ok ? null : projectAnalysisError(result.error), - ...(output === undefined ? {} : { captured_output: { ...output } }), - }, - { - operation, - cause, - ...(result.ok - ? { - partialObservation: createEvidence( - result.value.subject ?? undefined, - result.value.provider, - { - operation, - parameters: sqliteInspectionParameters(input), - result: result.value.result, - confidence: "observed", - locations: result.value.locations, - limitations: result.value.limitations, - }, - ), - } - : result.error.partialObservation === undefined - ? {} - : { partialObservation: result.error.partialObservation }), - }, + cause instanceof Error ? cause.message : String(cause), + result, + input, + output, + undefined, + cause, ), ); } }; +const sqliteCleanupFailure = ( + resources: readonly string[], + reason: string, + result: SqliteOutcome, + input: InspectSqliteDatabaseInput, + output?: AnalysisCapturedOutput, + details?: Readonly>, + cause?: unknown, +): ProviderCleanupError => { + const partialObservation = result.ok + ? createEvidence(result.value.subject ?? undefined, result.value.provider, { + operation, + parameters: sqliteInspectionParameters(input), + result: result.value.result, + confidence: "observed", + locations: result.value.locations, + limitations: result.value.limitations, + }) + : result.error.partialObservation; + return new ProviderCleanupError( + SQLITE_PROVIDER_IDENTITY.id, + resources, + { + ...details, + reason, + previous_error: result.ok ? null : projectAnalysisError(result.error), + ...(output === undefined ? {} : { captured_output: { ...output } }), + }, + { + operation, + ...(cause === undefined ? {} : { cause }), + ...(partialObservation === undefined ? {} : { partialObservation }), + }, + ); +}; + const validateSqliteInput = (input: InspectSqliteDatabaseInput): void => { const parsed = inspectSqliteDatabaseInputSchema.safeParse(input); if (!parsed.success || !isAbsolute(input.path)) diff --git a/src/sqlite/SqliteDatabaseWorkerExecution.ts b/src/sqlite/SqliteDatabaseWorkerExecution.ts index cba180399f..930a871a46 100644 --- a/src/sqlite/SqliteDatabaseWorkerExecution.ts +++ b/src/sqlite/SqliteDatabaseWorkerExecution.ts @@ -19,6 +19,10 @@ import { readStableArtifact } from "../artifacts/readStableArtifact.js"; import { ArtifactReaderFailure } from "../artifacts/ArtifactReader.js"; import type { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js"; import { runOwnedCommand } from "../process/OwnedCommand.js"; +import type { + ProviderProcessSupervisor, + ProviderProcessStopResult, +} from "../process/ProviderProcess.js"; import { capturedSqliteOutput } from "./SqliteDatabaseFailures.js"; import { SQLITE_DATABASE_LIMITS } from "./SqliteDatabaseLimits.js"; @@ -109,6 +113,10 @@ export const executeSqliteDatabaseWorker = async (request: { readonly launcher?: SqliteWorkerLauncher; readonly signal?: AbortSignal; readonly captured: (output: AnalysisCapturedOutput) => void; + readonly retainCleanup?: ( + owner: ProviderProcessSupervisor, + result: Extract, + ) => void; }): Promise> => { const requestPath = join(request.root, "request.json"); const replyPath = join(request.root, "reply.json"); @@ -143,6 +151,9 @@ export const executeSqliteDatabaseWorker = async (request: { { ...(request.signal === undefined ? {} : { signal: request.signal }), ...(request.launcher === undefined ? {} : { launcher: request.launcher }), + ...(request.retainCleanup === undefined + ? {} + : { retainCleanup: request.retainCleanup }), }, ); const capturedOutput = capturedSqliteOutput(execution); diff --git a/src/wasm/WabtArtifactProvider.test.ts b/src/wasm/WabtArtifactProvider.test.ts index 161e542d2e..e7dc2a515d 100644 --- a/src/wasm/WabtArtifactProvider.test.ts +++ b/src/wasm/WabtArtifactProvider.test.ts @@ -1,8 +1,10 @@ +import { createHash } from "node:crypto"; import { access, chmod, readFile, writeFile } from "node:fs/promises"; import { join } from "node:path"; import { expect, it, vi } from "vitest"; import { createTestTempDirectory } from "../../tests/fixtures/temporaryDirectory.js"; import { WabtArtifactProvider } from "./WabtArtifactProvider.js"; +import { wasmArtifactSchema } from "../domain/wasm/wasmArtifact.js"; import { PrivateRuntimeRoot } from "../process/PrivateRuntimeRoot.js"; import { PrivateRuntimeRootUnavailableError } from "../process/PrivateRuntimeRootUnavailableError.js"; import { spawnOwnedProviderProcess } from "../process/ProviderProcess.js"; @@ -271,6 +273,35 @@ it.each([false, true])( }); }, ); +const assertGlueObservations = ( + value: unknown, + inputs: readonly { readonly path: string; readonly source: string }[], + selectedPath: string, +): void => { + const glue = wasmArtifactSchema.parse(value).glue; + expect(glue).toHaveLength(inputs.length); + for (const [index, { path, source }] of inputs.entries()) { + expect(glue[index]).toMatchObject({ + artifact: { + path, + sha256: createHash("sha256").update(source).digest("hex"), + bytes: Buffer.byteLength(source), + }, + parse_status: "complete", + references: [ + { + value: "./selected.wasm", + line: source.includes("\n") ? 2 : 1, + column: source.startsWith("\uFEFFfetch") ? 7 : 6, + candidate_paths: [selectedPath], + association: "local-path-candidate", + evidence_kind: "static-literal", + }, + ], + }); + } +}; + it.each([ "success", "invalid", @@ -371,10 +402,34 @@ it.each([ }; }, ); + const literal = 'fetch("./selected.wasm");'; + const glueInputs = ( + mode === "success" + ? [ + literal, + `\uFEFF${literal}`, + `\uFEFF\n${literal}`, + `#!/usr/bin/env node\n${literal}`, + `\uFEFF#!/usr/bin/env node\n${literal}`, + ] + : [] + ).map((source, index) => ({ + path: join(selectedRoot, `glue-${index}.js`), + source, + })); + await Promise.all( + glueInputs.map(({ path, source }) => writeFile(path, source)), + ); const result = await provider.inspect( - { path, glue_paths: [], candidate_paths: [] }, + { + path, + glue_paths: glueInputs.map(({ path }) => path), + candidate_paths: [], + }, { signal: controller.signal }, ); + if (mode === "success" && result.ok) + assertGlueObservations(result.value.result, glueInputs, path); assertModeOutcome(mode, result, path); if (mode === "cleanup") { await assertCleanupRecovery({ @@ -489,6 +544,49 @@ it("reports WABT output beyond the budget as a resource constraint with a bounde signal: "SIGKILL", }); }); +it("does not attribute wasm-objdump output to a wasm2wat launch failure", async () => { + const selectedRoot = await createTestTempDirectory("rea-wabt-launch-"); + const path = join(selectedRoot, "selected.wasm"); + await writeFile(path, Buffer.alloc(76, 1)); + for (const tool of ["wasm-validate", "wasm-objdump", "wasm2wat"]) + await writeFile( + join(selectedRoot, tool + (process.platform === "win32" ? ".exe" : "")), + "source-owned executable seam", + { mode: 0o700 }, + ); + const dump = await readFile( + new URL("../../tests/fixtures/wasm/objdump.txt", import.meta.url), + "utf8", + ); + const provider = new WabtArtifactProvider( + { REA_WABT_BIN_DIRECTORY: selectedRoot }, + (input) => { + const version = input.arguments[0] === "--version"; + if (!version && input.command.includes("wasm2wat")) + throw Object.assign(new Error("spawn E2BIG"), { code: "E2BIG" }); + const script = version + ? 'process.stdout.write("1.0.42\\n")' + : input.command.includes("objdump") + ? `process.stdout.write(${JSON.stringify(dump)})` + : ""; + return spawnOwnedProviderProcess({ + ...input, + command: process.execPath, + arguments: ["-e", script], + expectedCommand: null, + }); + }, + ); + const result = await provider.inspect({ + path, + glue_paths: [], + candidate_paths: [], + }); + if (result.ok) throw new Error("Expected the wasm2wat launch to fail"); + expect(result.error.message).toContain("spawn E2BIG"); + expect(result.error.capturedOutput).toBeUndefined(); +}); + it("bounds an oversized stderr to an excerpt", () => { expect( projectAnalysisError( diff --git a/src/wasm/WabtArtifactProvider.ts b/src/wasm/WabtArtifactProvider.ts index 3be21b2a11..fae455c244 100644 --- a/src/wasm/WabtArtifactProvider.ts +++ b/src/wasm/WabtArtifactProvider.ts @@ -180,6 +180,7 @@ export class WabtArtifactProvider implements WasmArtifactPort { args: readonly string[], acceptNonZeroExit = false, ) => { + capturedOutput = undefined; await verifySnapshot(); const snapshot = await runOwnedCommand( { @@ -268,9 +269,11 @@ export class WabtArtifactProvider implements WasmArtifactPort { undefined, this.#resources, ); - const source = new TextDecoder("utf-8", { fatal: true }).decode( - bytes.bytes, - ); + // Keep source characters intact so locations match the hashed artifact. + const source = new TextDecoder("utf-8", { + fatal: true, + ignoreBOM: true, + }).decode(bytes.bytes); glue.push({ artifact: { path, sha256: bytes.sha256, bytes: bytes.bytes.length }, ...associateWasmGlue( diff --git a/src/wasm/WasmGlueAdapter.ts b/src/wasm/WasmGlueAdapter.ts index 1f8713042f..91c5242707 100644 --- a/src/wasm/WasmGlueAdapter.ts +++ b/src/wasm/WasmGlueAdapter.ts @@ -8,7 +8,11 @@ export const associateWasmGlue = ( path: string, candidates: readonly string[], ) => { - const file = parseJavaScriptSource(source); + // Babel requires a hashbang at offset zero. Removing only its preceding BOM + // leaves reference coordinates intact: all code starts on a later line. + const file = parseJavaScriptSource( + source.startsWith("\uFEFF#!") ? source.slice(1) : source, + ); const references: { value: string; line: number; diff --git a/src/windows/WindowsNativeLoader.test.ts b/src/windows/WindowsNativeLoader.test.ts index c0fee8273a..eada66fa8b 100644 --- a/src/windows/WindowsNativeLoader.test.ts +++ b/src/windows/WindowsNativeLoader.test.ts @@ -8,7 +8,11 @@ import { } from "./WindowsNativeLoader.js"; const fixture = ( - capabilities: { protocolStdin?: boolean; jobMemoryLimit?: unknown } = {}, + capabilities: { + protocolStdin?: boolean; + jobMemoryLimit?: unknown; + privateBridgeDescriptorAccess?: unknown; + } = {}, ) => { const bytes = Buffer.alloc(128); bytes.write("MZ"); @@ -51,6 +55,7 @@ const fixture = ( architecture: "x64", filesystem: identity, privateDacl: true, + privateBridgeDescriptorAccess: true, atomicJobAssignment: true, killOnOwnerClose: true, ...capabilities, @@ -61,7 +66,7 @@ const fixture = ( return { host, manifest, bytes, loads: () => loads }; }; -describe("packaged Windows native admission", () => { +describe("packaged Windows native compatibility", () => { it.each([true, false])( "preserves protocol stdin capability %s alongside existing job controls", (protocolStdin) => { @@ -106,24 +111,52 @@ describe("packaged Windows native admission", () => { }); it.each([ - { platform: "linux" as const }, - { architecture: "arm64" }, - { nodeApiVersion: "7" }, - { nodeApiVersion: undefined }, - { packageVersion: "2.0.0" }, - ])("rejects incompatible host metadata before loading: %j", (override) => { + [{ platform: "linux" as const }, "host"], + [{ architecture: "arm64" }, "host"], + [{ nodeApiVersion: "7" }, "host"], + [{ nodeApiVersion: undefined }, "host"], + [{ packageVersion: "2.0.0" }, "artifact"], + ] as const)( + "rejects incompatible host metadata before loading: %j", + (override, failureStage) => { + const value = fixture(); + expect( + loadWindowsNativeAuthority({ ...value.host, ...override }), + ).toMatchObject({ available: false, failureStage }); + expect(value.loads()).toBe(0); + }, + ); + + it("names the absent packaged manifest and matching-version recovery", () => { const value = fixture(); + const missing = Object.assign(new Error("file absent"), { + code: "ENOENT", + }); expect( - loadWindowsNativeAuthority({ ...value.host, ...override }).available, - ).toBe(false); + loadWindowsNativeAuthority({ + ...value.host, + readManifest: () => { + throw missing; + }, + }), + ).toMatchObject({ + available: false, + failureStage: "artifact", + reason: expect.stringMatching( + /Missing packaged Windows native file native\/windows\/build\/manifest\.json \(ENOENT\).*native\/windows\/build\/rea-windows-x64\.node.*native\/windows\/README\.md/u, + ), + }); expect(value.loads()).toBe(0); }); +}); +describe("packaged Windows native artifact validation", () => { it("rejects tampering even when the original file was a valid PE image", () => { const value = fixture(); value.bytes[127] = 1; expect(loadWindowsNativeAuthority(value.host)).toMatchObject({ available: false, + failureStage: "artifact", reason: expect.stringContaining("SHA-256"), }); expect(value.loads()).toBe(0); @@ -185,6 +218,7 @@ describe("packaged Windows native admission", () => { }), ).toMatchObject({ available: false, + failureStage: "self-test", retryable: true, reason: expect.stringContaining("private DACL: access denied"), }); @@ -197,6 +231,15 @@ describe("packaged Windows native admission", () => { loadArtifact: () => ({ call: () => ({ privateDacl: true }) }), }); expect(loaded.available).toBe(false); + expect(loaded).toMatchObject({ failureStage: "inspection" }); expect(loaded).not.toHaveProperty("retryable"); }); + + it("rejects Windows controls without handle-bound Cutter descriptor access", () => { + const value = fixture({ privateBridgeDescriptorAccess: undefined }); + expect(loadWindowsNativeAuthority(value.host)).toMatchObject({ + available: false, + reason: expect.stringContaining("privateBridgeDescriptorAccess"), + }); + }); }); diff --git a/src/windows/WindowsNativeLoader.ts b/src/windows/WindowsNativeLoader.ts index 2ab08e1ed6..4c4289635c 100644 --- a/src/windows/WindowsNativeLoader.ts +++ b/src/windows/WindowsNativeLoader.ts @@ -41,6 +41,7 @@ const inspectionSchema = z.strictObject({ architecture: z.literal("x64"), filesystem: windowsFileIdentitySchema, privateDacl: z.literal(true), + privateBridgeDescriptorAccess: z.literal(true), atomicJobAssignment: z.literal(true), killOnOwnerClose: z.literal(true), protocolStdin: z.boolean().optional(), @@ -70,6 +71,7 @@ export type WindowsNativeLoadResult = | { readonly available: false; readonly reason: string; + readonly failureStage: "host" | "artifact" | "self-test" | "inspection"; /** Only a failed native invocation can recover on a later query. */ readonly retryable?: true; }; @@ -85,6 +87,24 @@ export interface WindowsNativeLoadHost { loadArtifact(): unknown; } +const missingWindowsNativeFileReason = (file: string): string => + `Missing packaged Windows native file native/windows/build/${file} (ENOENT). Restore both native/windows/build/manifest.json and native/windows/build/rea-windows-x64.node from the rea-agents@${PACKAGE_METADATA.version} tarball or build them with native/windows/README.md; no native artifact was loaded.`; + +const readPackagedWindowsNativeFile = (file: string, read: () => T): T => { + try { + return read(); + } catch (cause: unknown) { + if ( + typeof cause === "object" && + cause !== null && + "code" in cause && + cause.code === "ENOENT" + ) + throw new Error(missingWindowsNativeFileReason(file), { cause }); + throw cause; + } +}; + /** Validate compatibility and artifact identity before invoking the native addon. */ export const loadWindowsNativeAuthority = ( host: WindowsNativeLoadHost, @@ -92,16 +112,25 @@ export const loadWindowsNativeAuthority = ( if (host.platform !== "win32" || host.architecture !== "x64") return { available: false, + failureStage: "host", reason: "REA Windows native controls require a Windows x64 host.", }; + const nodeApiVersion = Number(host.nodeApiVersion); + if (!Number.isSafeInteger(nodeApiVersion) || nodeApiVersion < 8) + return { + available: false, + failureStage: "host", + reason: "REA Windows native controls require Node-API 8 or newer.", + }; try { - const manifest = windowsNativeManifestSchema.parse(host.readManifest()); + const manifest = windowsNativeManifestSchema.parse( + readPackagedWindowsNativeFile("manifest.json", () => host.readManifest()), + ); if (manifest.packageVersion !== host.packageVersion) throw new Error("Windows native artifact belongs to another REA version"); - const nodeApiVersion = Number(host.nodeApiVersion); - if (!Number.isSafeInteger(nodeApiVersion) || nodeApiVersion < 8) - throw new Error("Windows native artifact requires Node-API 8 or newer"); - const artifact = host.readArtifact(); + const artifact = readPackagedWindowsNativeFile("rea-windows-x64.node", () => + host.readArtifact(), + ); if ( createHash("sha256").update(artifact).digest("hex") !== manifest.artifactSha256 @@ -117,15 +146,24 @@ export const loadWindowsNativeAuthority = ( } catch (cause: unknown) { return { available: false, + failureStage: "self-test", retryable: true, reason: `REA Windows native self-test failed: ${cause instanceof Error ? cause.message : String(cause)}`, }; } - const inspection = inspectionSchema.parse(observed); + const parsed = inspectionSchema.safeParse(observed); + if (!parsed.success) + return { + available: false, + failureStage: "inspection", + reason: `REA Windows native inspection is invalid: ${parsed.error.message}`, + }; + const inspection = parsed.data; return { available: true, authority: { call: binding.call, inspection } }; } catch (cause: unknown) { return { available: false, + failureStage: "artifact", reason: `REA Windows native controls are unavailable: ${cause instanceof Error ? cause.message : String(cause)}`, }; } diff --git a/src/windows/WindowsOwnedProcess.test.ts b/src/windows/WindowsOwnedProcess.test.ts index e9cf106fd4..0950b93939 100644 --- a/src/windows/WindowsOwnedProcess.test.ts +++ b/src/windows/WindowsOwnedProcess.test.ts @@ -1,5 +1,5 @@ import { once } from "node:events"; -import { describe, expect, it } from "vitest"; +import { describe, expect, it, vi } from "vitest"; import type { WindowsNativeAuthority } from "./WindowsNativeLoader.js"; import { @@ -40,6 +40,7 @@ const authority = ( readonly jobMemoryLimit?: true; readonly polls?: readonly unknown[]; readonly appliedLimit?: (requested: unknown) => unknown; + readonly failClose?: () => boolean; } = {}, ) => { const calls: { operation: string; arguments_: readonly unknown[] }[] = []; @@ -59,6 +60,7 @@ const authority = ( directory: true, }, privateDacl: true, + privateBridgeDescriptorAccess: true, atomicJobAssignment: true, killOnOwnerClose: true, protocolStdin: true, @@ -81,7 +83,11 @@ const authority = ( : {}), }; if (operation === "process_poll") return polls.shift(); - if (operation === "process_close") return null; + if (operation === "process_terminate") return true; + if (operation === "process_close") { + if (options.failClose?.()) throw new Error("native close failed"); + return null; + } throw new Error(`Unmodeled native operation ${operation}`); }, }; @@ -104,7 +110,70 @@ const launch = ( }, ); +describe("Windows native release retry", () => { + it("ends output streams and retains ownership for a failed close retry", async () => { + vi.useFakeTimers(); + let shouldFailClose = true; + try { + const { native, calls } = authority({ + polls: [{ ...settledPoll }, { ...settledPoll }], + failClose: () => shouldFailClose, + }); + const child = launch(native); + const failures: string[] = []; + child.on("error", (error: Error) => failures.push(error.message)); + const closed = new Promise((resolve) => { + child.once("close", () => resolve()); + }); + + await vi.advanceTimersByTimeAsync(25); + expect(child.stdout.writableEnded).toBe(true); + expect(child.stderr.writableEnded).toBe(true); + expect(failures).toContain("native close failed"); + expect(calls.some(({ operation }) => operation === "process_close")).toBe( + true, + ); + + shouldFailClose = false; + const cleanup = child.cleanup(); + await vi.advanceTimersByTimeAsync(25); + expect(await cleanup).toMatchObject({ cleaned: true }); + await closed; + expect(child.stdout.writableEnded).toBe(true); + expect(child.stderr.writableEnded).toBe(true); + } finally { + vi.useRealTimers(); + } + }); +}); + describe("Windows Job Object memory limits", () => { + it("emits close only after descendants leave the Job Object", async () => { + const { native, calls } = authority({ + polls: [ + { + ...settledPoll, + stdoutEnded: false, + stderrEnded: false, + activeProcesses: 1, + }, + settledPoll, + ], + }); + const child = launch(native); + const events: string[] = []; + child.once("exit", () => events.push("exit")); + child.once("close", () => events.push("close")); + + await once(child, "close"); + + expect(events).toEqual(["exit", "close"]); + expect( + calls.filter(({ operation }) => operation === "process_poll"), + ).toHaveLength(2); + expect(calls.at(-1)?.operation).toBe("process_close"); + }); + it("refuses a requested limit before launch when the artifact lacks the control", () => { const { native, calls } = authority(); expect(() => launch(native, LIMIT)).toThrow( @@ -178,3 +247,223 @@ describe("Windows Job Object memory limits", () => { expect(() => launch(native, LIMIT)).toThrow(); }); }); + +describe("Windows output capture drain", () => { + it("repolls without the 25ms idle delay while the child keeps producing output", async () => { + vi.useFakeTimers(); + try { + const chunk = Buffer.alloc(8192, 0x61); + const polls: Array<{ + stdout: Buffer; + stderr: Buffer; + stdoutEnded: boolean; + stderrEnded: boolean; + exitCode: number | null; + activeProcesses: number; + }> = Array.from({ length: 10 }, () => ({ + stdout: chunk, + stderr: Buffer.alloc(0), + stdoutEnded: false, + stderrEnded: false, + exitCode: null, + activeProcesses: 1, + })); + polls.push({ ...settledPoll }); + const { native, calls } = authority({ polls }); + const child = launch(native); + child.stdout.resume(); + const closed = once(child, "close"); + // 100ms of simulated time: the fixed 25ms poll cadence can complete + // at most ~4 polls, so all 10 data polls plus the settle poll must + // still be pending. Draining repolls immediately while data flows. + await vi.advanceTimersByTimeAsync(100); + expect( + calls.filter((call) => call.operation === "process_poll"), + ).toHaveLength(11); + await closed; + } finally { + vi.useRealTimers(); + } + }); + + it("keeps the 25ms idle delay after a poll with no output", async () => { + vi.useFakeTimers(); + try { + const idle = { + stdout: Buffer.alloc(0), + stderr: Buffer.alloc(0), + stdoutEnded: false, + stderrEnded: false, + exitCode: null, + activeProcesses: 1, + }; + const { native, calls } = authority({ + polls: [idle, { ...settledPoll }], + }); + const child = launch(native); + const closed = once(child, "close"); + const pollCount = () => + calls.filter((call) => call.operation === "process_poll").length; + // The constructor schedules the first poll 25ms out. + await vi.advanceTimersByTimeAsync(25); + expect(pollCount()).toBe(1); + // An idle child must not be repolled early: 10ms of simulated time + // must not reach the next poll, the remaining 15ms must. + await vi.advanceTimersByTimeAsync(10); + expect(pollCount()).toBe(1); + await vi.advanceTimersByTimeAsync(15); + expect(pollCount()).toBe(2); + await closed; + } finally { + vi.useRealTimers(); + } + }); + + it("repolls without the 25ms idle delay while only stderr carries output", async () => { + vi.useFakeTimers(); + try { + const chunk = Buffer.alloc(4096, 0x62); + const { native, calls } = authority({ + polls: [ + { + stdout: Buffer.alloc(0), + stderr: chunk, + stdoutEnded: false, + stderrEnded: false, + exitCode: null, + activeProcesses: 1, + }, + { ...settledPoll }, + ], + }); + const child = launch(native); + const closed = once(child, "close"); + const pollCount = () => + calls.filter((call) => call.operation === "process_poll").length; + await vi.advanceTimersByTimeAsync(25); + expect(pollCount()).toBe(1); + // stderr-only output still means the child is producing: the next + // poll must be immediate, not 25ms out. + await vi.advanceTimersByTimeAsync(1); + expect(pollCount()).toBe(2); + await closed; + } finally { + vi.useRealTimers(); + } + }); +}); + +describe("Windows output backpressure and cleanup", () => { + it("stops native reads until both paused output consumers drain", async () => { + vi.useFakeTimers(); + try { + const chunk = Buffer.alloc(65536, 0x63); + const { native, calls } = authority({ + polls: [ + { + stdout: chunk, + stderr: chunk, + stdoutEnded: false, + stderrEnded: false, + exitCode: null, + activeProcesses: 1, + }, + { ...settledPoll, stderr: Buffer.alloc(0) }, + ], + }); + const child = launch(native); + const closed = once(child, "close"); + child.stdout.pause(); + child.stderr.pause(); + const pollCount = () => + calls.filter((call) => call.operation === "process_poll").length; + await vi.advanceTimersByTimeAsync(25); + expect(child.stdout.writableNeedDrain).toBe(true); + expect(child.stderr.writableNeedDrain).toBe(true); + await vi.advanceTimersByTimeAsync(5000); + expect(pollCount()).toBe(1); + const stdoutDrained = once(child.stdout, "drain"); + child.stdout.resume(); + await stdoutDrained; + await vi.advanceTimersByTimeAsync(25); + expect(pollCount()).toBe(1); + const stderrDrained = once(child.stderr, "drain"); + child.stderr.resume(); + await stderrDrained; + await vi.advanceTimersByTimeAsync(1); + await closed; + expect(pollCount()).toBe(2); + } finally { + vi.useRealTimers(); + } + }); + + it("discards blocked output during cleanup so the owned job can settle", async () => { + vi.useFakeTimers(); + try { + const chunk = Buffer.alloc(65536, 0x64); + const { native, calls } = authority({ + polls: [ + { + stdout: chunk, + stderr: Buffer.alloc(0), + stdoutEnded: false, + stderrEnded: false, + exitCode: null, + activeProcesses: 1, + }, + { ...settledPoll, stderr: Buffer.alloc(0) }, + ], + }); + const child = launch(native); + child.stdout.pause(); + await vi.advanceTimersByTimeAsync(25); + expect(child.stdout.writableNeedDrain).toBe(true); + const closed = once(child, "close"); + const cleanup = child.cleanup(); + await vi.advanceTimersByTimeAsync(25); + expect(await cleanup).toEqual({ cleaned: true, signaled: true }); + await closed; + expect( + calls.filter((call) => call.operation === "process_poll"), + ).toHaveLength(2); + expect(child.stdout.destroyed).toBe(true); + expect(child.outputIncomplete).toBe(true); + } finally { + vi.useRealTimers(); + } + }); + + it("resumes polling when a blocked output stream is destroyed", async () => { + vi.useFakeTimers(); + try { + const chunk = Buffer.alloc(65536, 0x65); + const { native, calls } = authority({ + polls: [ + { + stdout: chunk, + stderr: Buffer.alloc(0), + stdoutEnded: false, + stderrEnded: false, + exitCode: null, + activeProcesses: 1, + }, + { ...settledPoll, stderr: Buffer.alloc(0) }, + ], + }); + const child = launch(native); + const closed = once(child, "close"); + await vi.advanceTimersByTimeAsync(25); + expect(child.stdout.writableNeedDrain).toBe(true); + child.stdout.destroy(); + await vi.advanceTimersByTimeAsync(0); + await closed; + expect( + calls.filter((call) => call.operation === "process_poll"), + ).toHaveLength(2); + expect(child.outputIncomplete).toBe(true); + } finally { + vi.useRealTimers(); + } + }); +}); diff --git a/src/windows/WindowsOwnedProcess.ts b/src/windows/WindowsOwnedProcess.ts index b6a5304613..73ac7aefdd 100644 --- a/src/windows/WindowsOwnedProcess.ts +++ b/src/windows/WindowsOwnedProcess.ts @@ -67,11 +67,14 @@ export class WindowsOwnedProcess extends EventEmitter { #closed = false; #failure: string | undefined; #timer: NodeJS.Timeout | undefined; + #blockedWake: NodeJS.Timeout | undefined; + readonly #blockedOutput = new Set(); #unreferenced = false; #cleanup: Promise | undefined; readonly #memoryLimitBytes: number | undefined; #memoryLimitReached = false; #peakJobMemoryBytes: number | undefined; + #outputIncomplete = false; constructor( command: string, @@ -179,6 +182,11 @@ export class WindowsOwnedProcess extends EventEmitter { return this.#peakJobMemoryBytes; } + /** True when cleanup destroyed a backpressured stream before all output was observed. */ + get outputIncomplete(): boolean { + return this.#outputIncomplete; + } + async #writeInput(chunk: unknown): Promise { const bytes = z.instanceof(Buffer).parse(chunk); const written = z @@ -227,12 +235,14 @@ export class WindowsOwnedProcess extends EventEmitter { unref(): void { this.#unreferenced = true; this.#timer?.unref(); + this.#blockedWake?.unref(); } /** Restore event-loop references when retrying process lifecycle observation. */ ref(): void { this.#unreferenced = false; this.#timer?.ref(); + this.#blockedWake?.ref(); } async #stop(): Promise { @@ -240,6 +250,7 @@ export class WindowsOwnedProcess extends EventEmitter { let timer: NodeJS.Timeout | undefined; try { const signaled = this.kill(); + this.#discardBlockedOutput(); this.#schedule(); const settled = await Promise.race([ this.#settled.then(() => true), @@ -265,12 +276,50 @@ export class WindowsOwnedProcess extends EventEmitter { } } - #schedule(): void { - if (this.#closed || this.#timer !== undefined) return; - this.#timer = setTimeout(() => this.#poll(), 25); + #schedule(delay = 25): void { + if ( + this.#closed || + this.#timer !== undefined || + this.#blockedOutput.size > 0 + ) + return; + this.#timer = setTimeout(() => this.#poll(), delay); if (this.#unreferenced) this.#timer.unref(); } + #scheduleBlockedWake(): void { + if ( + this.#closed || + this.#blockedOutput.size === 0 || + this.#blockedWake !== undefined + ) + return; + this.#blockedWake = setTimeout(() => { + this.#blockedWake = undefined; + this.#scheduleBlockedWake(); + }, 1_000); + if (this.#unreferenced) this.#blockedWake.unref(); + } + + #clearBlockedWake(): void { + if (this.#blockedWake !== undefined) clearTimeout(this.#blockedWake); + this.#blockedWake = undefined; + } + + #unblockOutput(stream: PassThrough): void { + if (stream.destroyed) this.#outputIncomplete = true; + this.#blockedOutput.delete(stream); + if (this.#blockedOutput.size === 0) this.#clearBlockedWake(); + this.#schedule(0); + } + + #discardBlockedOutput(): void { + if (this.#blockedOutput.size > 0) this.#outputIncomplete = true; + for (const stream of this.#blockedOutput) stream.destroy(); + this.#blockedOutput.clear(); + this.#clearBlockedWake(); + } + #poll(): void { this.#timer = undefined; if (this.#closed) return; @@ -289,8 +338,11 @@ export class WindowsOwnedProcess extends EventEmitter { state = limited; } this.#failure = undefined; - if (state.stdout.length > 0) this.stdout.write(state.stdout); - if (state.stderr.length > 0) this.stderr.write(state.stderr); + // Stop native reads while either consumer is backpressured. A slower + // cadence still accumulates unbounded unread output in the parent heap. + const idle = state.stdout.length === 0 && state.stderr.length === 0; + this.#writeOutput(this.stdout, state.stdout); + this.#writeOutput(this.stderr, state.stderr); if (state.exitCode !== null && this.exitCode === null) { this.exitCode = state.exitCode; this.emit("exit", state.exitCode, null); @@ -302,7 +354,7 @@ export class WindowsOwnedProcess extends EventEmitter { state.activeProcesses === 0 ) { this.#release(); - } else this.#schedule(); + } else this.#schedule(idle ? 25 : 0); } catch (cause: unknown) { const failure = cause instanceof Error ? cause : new Error(String(cause)); this.#failure = failure.message; @@ -313,14 +365,35 @@ export class WindowsOwnedProcess extends EventEmitter { } } + #writeOutput(stream: PassThrough, bytes: Buffer): void { + if (bytes.length === 0 || stream.writableEnded) return; + if (stream.destroyed) { + this.#outputIncomplete = true; + return; + } + if (stream.write(bytes)) return; + this.#blockedOutput.add(stream); + const unblock = () => { + stream.off("drain", unblock); + stream.off("close", unblock); + this.#unblockOutput(stream); + }; + stream.once("drain", unblock); + stream.once("close", unblock); + this.#scheduleBlockedWake(); + } + #release(): void { if (this.#closed) return; if (this.#timer !== undefined) clearTimeout(this.#timer); + this.#clearBlockedWake(); + this.stdin?.destroy(); + if (!this.stdout.destroyed) this.stdout.end(); + if (!this.stderr.destroyed) this.stderr.end(); + // End local streams before releasing the native handle. If release fails, + // the retained job can be retried without leaving output streams open. this.#authority.call("process_close", [this.#handle]); this.#closed = true; - this.stdin?.destroy(); - this.stdout.end(); - this.stderr.end(); this.#settle?.(); this.emit("close", this.exitCode, null); } diff --git a/stryker.config.mjs b/stryker.config.mjs new file mode 100644 index 0000000000..0fe7f37455 --- /dev/null +++ b/stryker.config.mjs @@ -0,0 +1,237 @@ +import { mutationRanges } from "./scripts/lib/reverse-tools-mutation-targets.mjs"; + +const config = { + $schema: "./node_modules/@stryker-mutator/core/schema/stryker-schema.json", + testRunner: "vitest", + plugins: [ + "@stryker-mutator/typescript-checker", + "@stryker-mutator/vitest-runner", + ], + vitest: { + configFile: "vitest.reverse-tools.config.ts", + related: true, + }, + checkers: ["typescript"], + tsconfigFile: "tsconfig.reverse-tools-mutation.json", + typescriptChecker: { + prioritizePerformanceOverAccuracy: true, + }, + coverageAnalysis: "perTest", + thresholds: { break: null }, + ignorePatterns: ["/.cache/rea-command-locks/**", "/coverage/**"], + concurrency: 2, + timeoutMS: 30000, + reporters: ["clear-text", "progress", "html", "json"], + jsonReporter: { fileName: "reports/mutation/reverse-tools.json" }, + htmlReporter: { fileName: "reports/mutation/reverse-tools.html" }, +}; + +export default { + ...config, + mutate: [ + ...mutationRanges("src/application/reverse/ReverseEngineeringService.ts", [ + { + kind: "method", + owner: "ReverseEngineeringService", + name: "inspectWithObjdump", + }, + { + kind: "method", + owner: "ReverseEngineeringService", + name: "executeRizinCommand", + }, + { + kind: "method", + owner: "ReverseEngineeringService", + name: "#observe", + }, + { + kind: "method", + owner: "ReverseEngineeringService", + name: "retryCleanup", + }, + { + kind: "guard", + owner: "ReverseEngineeringService", + method: "#observeProvider", + condition: "signal?.aborted", + direct: true, + }, + { + kind: "guard", + owner: "ReverseEngineeringService", + method: "#observeProvider", + condition: "!cleanup.ok", + }, + { + kind: "declaration", + owner: "ReverseEngineeringService", + method: "#observeProvider", + name: "absolutePath", + }, + { + kind: "declaration", + owner: "ReverseEngineeringService", + method: "#observeProvider", + name: "cleanup", + }, + { + kind: "try", + owner: "ReverseEngineeringService", + method: "#observeProvider", + assignment: "digest", + }, + { + kind: "try", + owner: "ReverseEngineeringService", + method: "#observeProvider", + callee: "this.#run", + }, + { + kind: "expression", + text: "snapshot.signal !== null && snapshot.signal !== undefined", + }, + ]), + ...mutationRanges("src/process/ExecFileOutput.ts", [ + { + kind: "declaration", + name: "execFileOutputFailure", + }, + ]), + ...mutationRanges("src/server/providerEnvironment.ts", [ + { + kind: "declaration", + name: "captureProviderEnvironment", + }, + ]), + ...mutationRanges("src/server/registerReverseEngineeringTools.ts", [ + { + kind: "declaration", + name: "registerReverseEngineeringTools", + }, + ]), + ...mutationRanges("src/gdb/GdbSessionManager.ts", [ + { + kind: "constructor", + owner: "GdbCommandTimeoutError", + }, + { + kind: "constructor", + owner: "GdbCommandProcessExitError", + }, + { + kind: "constructor", + owner: "GdbSessionManager", + }, + { + kind: "declaration", + name: "DEFAULT_MAX_SESSIONS", + }, + { + kind: "declaration", + owner: "GdbSessionManager", + method: "#startProtocol", + name: "session", + }, + { + kind: "declaration", + owner: "GdbSessionManager", + method: "#startProtocol", + name: "stdoutSettled", + }, + { + kind: "call", + owner: "GdbSessionManager", + method: "#startProtocol", + callee: "launched.process.stdout?.on", + event: "data", + }, + { + kind: "call", + owner: "GdbSessionManager", + method: "#startProtocol", + callee: "launched.process.stdin?.on", + event: "error", + }, + { + kind: "call", + owner: "GdbSessionManager", + method: "#startProtocol", + callee: "launched.process.stdin?.once", + event: "close", + }, + { + kind: "call", + owner: "GdbSessionManager", + method: "#startProtocol", + callee: "launched.process.once", + event: "exit", + }, + { + kind: "guard", + owner: "GdbSessionManager", + method: "#startProtocol", + condition: "launched.process.stdout === null", + }, + { + kind: "method", + owner: "GdbSessionManager", + name: "close", + }, + ]), + ...mutationRanges("src/process/ownedDebuggerSession.ts", [ + { + kind: "declaration", + name: "waitForInferiorDetach", + }, + ]), + ...mutationRanges("src/rizin/RizinDebugSessionManager.ts", [ + { + kind: "guard", + owner: "RizinDebugSessionManager", + method: "execute", + condition: "captured.outputTruncated", + }, + { + kind: "declaration", + owner: "RizinDebugSessionManager", + method: "execute", + name: "result", + withinBlockWithBinding: "captured", + }, + { + kind: "declaration", + owner: "RizinDebugSessionManager", + method: "execute", + name: "evidence", + withinBlockWithBinding: "captured", + }, + { + kind: "declaration", + owner: "RizinDebugSessionManager", + method: "#receive", + name: "nextOffset", + }, + ]), + ...mutationRanges("src/cli/reverseEngineeringCommands.ts", [ + { + kind: "command-handler", + command: "debugWithGdb", + }, + { + kind: "command-handler", + command: "debugWithRizin", + }, + ]), + ...mutationRanges("src/cutter/CutterBridgeClient.ts", [ + { + kind: "expression", + text: "status.data.current_file === undefined", + }, + { + kind: "expression", + text: "response.current_file === undefined", + }, + ]), + ], +}; diff --git a/stryker.semantics.config.mjs b/stryker.semantics.config.mjs new file mode 100644 index 0000000000..81c58dc40e --- /dev/null +++ b/stryker.semantics.config.mjs @@ -0,0 +1,53 @@ +import { mutationRanges } from "./scripts/lib/reverse-tools-mutation-targets.mjs"; + +const config = { + $schema: "./node_modules/@stryker-mutator/core/schema/stryker-schema.json", + testRunner: "command", + commandRunner: { + command: "npx vitest run --config vitest.reverse-tools-semantics.config.ts", + }, + plugins: ["@stryker-mutator/typescript-checker"], + checkers: ["typescript"], + tsconfigFile: "tsconfig.reverse-tools-mutation.json", + typescriptChecker: { + prioritizePerformanceOverAccuracy: true, + }, + coverageAnalysis: "off", + thresholds: { break: null }, + ignorePatterns: ["/.cache/rea-command-locks/**", "/coverage/**"], + concurrency: 2, + timeoutMS: 30000, + reporters: ["clear-text", "progress", "html", "json"], + jsonReporter: { fileName: "reports/mutation/reverse-tools-semantics.json" }, + htmlReporter: { fileName: "reports/mutation/reverse-tools-semantics.html" }, +}; + +export default { + ...config, + mutate: [ + ...mutationRanges("src/cutter/CutterBridgeClient.ts", [ + { + kind: "expression", + text: "status.data.current_file === undefined", + }, + { + kind: "expression", + text: "response.current_file === undefined", + }, + ]), + ...mutationRanges("src/rizin/RizinDebugSessionManager.ts", [ + { + kind: "declaration", + name: "isRizinCommandDeadlineExpired", + }, + { + kind: "declaration", + name: "createRizinCommandTimeoutError", + }, + { + kind: "declaration", + name: "createRizinCommandTimeoutEvidence", + }, + ]), + ], +}; diff --git a/tests/acceptance/applications/JavaScriptModuleLookupCli.test.ts b/tests/acceptance/applications/JavaScriptModuleLookupCli.test.ts index 3a7fad35ff..24dc748b79 100644 --- a/tests/acceptance/applications/JavaScriptModuleLookupCli.test.ts +++ b/tests/acceptance/applications/JavaScriptModuleLookupCli.test.ts @@ -18,6 +18,9 @@ const createModuleLookupFixture = async () => { type: "commonjs", }), "esm/entry.mjs": [ + 'import "./a%2eb.mjs";', + 'import "./sub/%2e%2e/x.mjs";', + 'import "./sub%2fx.mjs";', 'import "./extensionless";', 'import "./directory";', 'import "./explicit.js";', @@ -43,6 +46,8 @@ const createModuleLookupFixture = async () => { "esm/directory/index.js": "module.exports = 2;", "esm/explicit.js.js": "module.exports = 3;", "esm/exact.js": "module.exports = 4;", + "esm/a.b.mjs": "export default 'encoded dot';", + "esm/x.mjs": "export default 'confined parent';", "esm/actual.cjs": "module.exports = 5;", "esm/.hidden.js": "module.exports = 'local hidden';", "esm/..pkg.js": "module.exports = 'local dot package';", @@ -51,6 +56,8 @@ const createModuleLookupFixture = async () => { "esm/typed.mts": 'import value from "./typed-dep.js"; void value;', "esm/typed-dep.ts": "export default 42;", "cjs/entry.cjs": [ + 'require("./lit%2eb");', + 'require("./x%2Fy");', 'require("./extensionless");', 'require("./directory");', 'require("./feature");', @@ -76,6 +83,8 @@ const createModuleLookupFixture = async () => { "index.js": "module.exports = 'parent directory';", "cjs/.hidden.js": "module.exports = 'local hidden';", "cjs/extensionless.cjs": "module.exports = 1;", + "cjs/lit%2eb.js": "module.exports = 'literal dot bytes';", + "cjs/x%2Fy.js": "module.exports = 'literal slash bytes';", "cjs/feature.cjs": "module.exports = 2;", "cjs/module-only.mjs": "export default 42;", "cjs/valid-directory/index.js": "module.exports = 42;", @@ -98,7 +107,7 @@ const createModuleLookupFixture = async () => { "node_modules/ancestor-blocked/index.js": "module.exports = 9;", "node_modules/ancestor-sub/parser.js": "module.exports = 10;", "esm/runtime-control.mjs": [ - 'const specifiers = ["./extensionless", "./directory", "./explicit.js", "./exact.js", "./actual.cjs?cache=1", "empty-main", "nested-main", "file-only", "ancestor-empty", "ancestor-blocked", "ancestor-sub/parser", "./.hidden.js", "./..pkg.js", "./exact.js/.", "./exact.js/child/..", ".hidden.js", "..pkg.js", ".", ".."];', + 'const specifiers = ["./a%2eb.mjs", "./sub/%2e%2e/x.mjs", "./sub%2fx.mjs", "./extensionless", "./directory", "./explicit.js", "./exact.js", "./actual.cjs?cache=1", "empty-main", "nested-main", "file-only", "ancestor-empty", "ancestor-blocked", "ancestor-sub/parser", "./.hidden.js", "./..pkg.js", "./exact.js/.", "./exact.js/child/..", ".hidden.js", "..pkg.js", ".", ".."];', "const results = [];", "for (const specifier of specifiers) {", " try { const value = (await import(specifier)).default; results.push({ specifier, ok: true, value }); }", @@ -107,7 +116,7 @@ const createModuleLookupFixture = async () => { "console.log(JSON.stringify(results));", ].join("\n"), "cjs/runtime-control.cjs": [ - 'const specifiers = ["./extensionless", "./directory", "./feature", "./feature.cjs", "./module-only", "./valid-directory", "./default-js", "./default-json", "empty-main", "nested-main", "file-only", "ancestor-empty", "ancestor-blocked", "ancestor-sub/parser", ".hidden.js", "./.hidden.js", ".", ".."];', + 'const specifiers = ["./lit%2eb", "./x%2Fy", "./extensionless", "./directory", "./feature", "./feature.cjs", "./module-only", "./valid-directory", "./default-js", "./default-json", "empty-main", "nested-main", "file-only", "ancestor-empty", "ancestor-blocked", "ancestor-sub/parser", ".hidden.js", "./.hidden.js", ".", ".."];', "const results = [];", "for (const specifier of specifiers) {", " try { const value = require(specifier); results.push({ specifier, ok: true, value }); }", @@ -142,6 +151,13 @@ const expectNodeRuntimeControls = ( const esmResults = JSON.parse(esmOutput) as RuntimeControlResult[]; const cjsResults = JSON.parse(cjsOutput) as RuntimeControlResult[]; expect(esmResults).toEqual([ + { specifier: "./a%2eb.mjs", ok: true, value: "encoded dot" }, + { specifier: "./sub/%2e%2e/x.mjs", ok: true, value: "confined parent" }, + { + specifier: "./sub%2fx.mjs", + ok: false, + code: "ERR_INVALID_MODULE_SPECIFIER", + }, { specifier: "./extensionless", ok: false, @@ -199,6 +215,8 @@ const expectNodeRuntimeControls = ( { specifier: "..", ok: false, code: "ERR_UNSUPPORTED_DIR_IMPORT" }, ]); expect(cjsResults).toEqual([ + { specifier: "./lit%2eb", ok: true, value: "literal dot bytes" }, + { specifier: "./x%2Fy", ok: true, value: "literal slash bytes" }, { specifier: "./extensionless", ok: false, @@ -250,6 +268,9 @@ cliTest.for(["CLI", "stdio MCP"] as const)( const { graph } = await analyzeJavaScriptDirectory(surface, cli, root); const expectations = [ + ["import", "./a%2eb.mjs", "esm/a.b.mjs", "resolved"], + ["import", "./sub/%2e%2e/x.mjs", "esm/x.mjs", "resolved"], + ["import", "./sub%2fx.mjs", null, "rejected"], ["import", "./extensionless", null, "not-found"], ["import", "./directory", null, "not-found"], ["import", "./explicit.js", null, "not-found"], @@ -271,6 +292,8 @@ cliTest.for(["CLI", "stdio MCP"] as const)( ["import", "./exact.js/.", null, "not-found"], ["import", "./exact.js/child/..", null, "not-found"], ["require", "./extensionless", null, "not-found"], + ["require", "./lit%2eb", "cjs/lit%2eb.js", "resolved"], + ["require", "./x%2Fy", "cjs/x%2Fy.js", "resolved"], ["require", "./directory", null, "not-found"], ["require", "./feature", null, "not-found"], ["require", "./feature.cjs", "cjs/feature.cjs", "resolved"], diff --git a/tests/acceptance/applications/applicationVersionMatchingCli.test.ts b/tests/acceptance/applications/applicationVersionMatchingCli.test.ts index 64e438ed36..17cc61eaab 100644 --- a/tests/acceptance/applications/applicationVersionMatchingCli.test.ts +++ b/tests/acceptance/applications/applicationVersionMatchingCli.test.ts @@ -96,3 +96,85 @@ cliTest( } }, ); + +cliTest( + "matches an unchanged source module after a sibling file changes", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-source-module-version-"); + const versions = await Promise.all( + ["left", "right"].map(async (name) => { + const application = join(root, name); + await mkdir(join(application, "src"), { recursive: true }); + await writeFile( + join(application, "package.json"), + JSON.stringify({ + name: "source-module-fixture", + version: "1.0.0", + main: "src/main.js", + }), + ); + await writeFile( + join(application, "src/main.js"), + 'require("./util.js");\n', + ); + await writeFile( + join(application, "src/util.js"), + `module.exports = ${JSON.stringify(name)};\n`, + ); + const response = await cli.run({ + arguments: ["analyze-javascript-application", application, "--json"], + environment: { REA_LOG_LEVEL: "silent" }, + }); + expect(response.exitCode, response.stderr).toBe(0); + return parseEvidence(response.json); + }), + ); + const [left, right] = versions; + if (left === undefined || right === undefined) + throw new Error("Missing source module Evidence"); + const analyses = versions.map(({ normalized_result }) => + javascriptApplicationAnalysisResultSchema.parse(normalized_result), + ); + const sourceModules = analyses.map(({ graph }) => { + const node = graph.nodes.find( + ({ kind, observations }) => + kind === "javascript-module" && + observations.some( + ({ properties }) => + properties.semantic_role === "source-module" && + properties.path === "src/main.js", + ), + ); + if (node === undefined) throw new Error("Missing main source module"); + expect( + node.observations.some( + ({ properties }) => typeof properties.source_sha256 === "string", + ), + ).toBe(true); + return node; + }); + const inputPath = join(root, "comparison.json"); + await writeFile(inputPath, JSON.stringify({ left, right })); + const response = await cli.run({ + arguments: ["compare-application-versions", inputPath, "--json"], + environment: { REA_LOG_LEVEL: "silent" }, + }); + expect(response.exitCode, response.stderr).toBe(0); + const result = applicationVersionComparisonResultSchema.parse( + parseEvidence(response.json).normalized_result, + ); + + expect(result.items).toContainEqual( + expect.objectContaining({ + left_node_id: sourceModules[0]?.node_id, + right_node_id: sourceModules[1]?.node_id, + status: "unchanged", + match: expect.objectContaining({ + status: "matched", + basis: "exact-module-source-digest", + confidence: "exact", + }), + }), + ); + }, +); diff --git a/tests/acceptance/applications/applicationWorkflowCli.test.ts b/tests/acceptance/applications/applicationWorkflowCli.test.ts index 6548d210bf..da514f2921 100644 --- a/tests/acceptance/applications/applicationWorkflowCli.test.ts +++ b/tests/acceptance/applications/applicationWorkflowCli.test.ts @@ -485,6 +485,7 @@ it("preserves complete line-comment source-map URLs through CLI and MCP", async declared_url: declaredUrl, declared_url_sha256: digest, resolved_path: "app.js.map", + resolution_status: "resolved", available: true, }, evidence: expect.objectContaining({ @@ -507,6 +508,7 @@ it("preserves complete line-comment source-map URLs through CLI and MCP", async properties: { declared_url_sha256: digest, resolved_path: "app.js.map", + resolution_status: "resolved", }, }), ); @@ -539,7 +541,7 @@ it("resolves source-map URL paths once without changing their declarations", asy await writeFile( join(root, "app.js"), cases.map(([url]) => `//# sourceMappingURL=${url}\n`).join("") + - 'new Worker("./worker%20file.js");\n' + + 'new Worker(new URL("./worker%20file.js", import.meta.url));\n' + 'navigator.serviceWorker.register("./service%20worker.js");\n', ); await writeFile(join(root, "worker file.js"), "export const worker = 1;\n"); @@ -565,6 +567,7 @@ it("resolves source-map URL paths once without changing their declarations", asy declared_url: declaredUrl, declared_url_sha256: digest, resolved_path: path, + resolution_status: "resolved", available: true, }, evidence: expect.objectContaining({ @@ -587,31 +590,49 @@ it("resolves source-map URL paths once without changing their declarations", asy expect.objectContaining({ relation: "maps_to", target_node_id: map?.node_id, - properties: { declared_url_sha256: digest, resolved_path: path }, - }), - ); - } - for (const [kind, specifier, path] of [ - ["worker", "./worker%20file.js", "worker file.js"], - ["service-worker", "./service%20worker.js", "service worker.js"], - ]) { - const worker = graph.nodes.find((node) => node.kind === kind); - expect(worker?.observations).toContainEqual( - expect.objectContaining({ - properties: expect.objectContaining({ - declared_specifier: specifier, + properties: { + declared_url_sha256: digest, resolved_path: path, - }), - }), - ); - expect(graph.edges).toContainEqual( - expect.objectContaining({ - source_node_id: worker?.node_id, - relation: "maps_to", - properties: { resolved_path: path }, + resolution_status: "resolved", + }, }), ); } + const worker = graph.nodes.find((node) => node.kind === "worker"); + expect(worker?.observations).toContainEqual( + expect.objectContaining({ + properties: expect.objectContaining({ + declared_specifier: "./worker%20file.js", + resolved_path: "worker file.js", + }), + }), + ); + expect(graph.edges).toContainEqual( + expect.objectContaining({ + source_node_id: worker?.node_id, + relation: "maps_to", + properties: { resolved_path: "worker file.js" }, + }), + ); + // A service worker script URL is relative to the unestablished document base. + const serviceWorker = graph.nodes.find( + (node) => node.kind === "service-worker", + ); + expect(serviceWorker?.observations).toContainEqual( + expect.objectContaining({ + properties: expect.objectContaining({ + declared_specifier: null, + expression: 'navigator.serviceWorker.register("./service%20worker.js")', + resolved_path: null, + }), + }), + ); + expect( + graph.edges.some( + ({ source_node_id, relation }) => + source_node_id === serviceWorker?.node_id && relation === "maps_to", + ), + ).toBe(false); expect((await analyzeThroughStdioMcp(root)).graph).toEqual(graph); }, 20_000); @@ -873,8 +894,9 @@ describe("rest parameter semantic trace CLI", () => { kind === "expression" && label === `argument ${String(index)}`, ); if (argument === undefined) throw new Error("Missing argument node"); - const traced = await runCli([ - "trace-javascript-semantics", + const requestPath = join(root, `trace-request-${index}.json`); + await writeFile( + requestPath, JSON.stringify({ application, query: { @@ -884,6 +906,10 @@ describe("rest parameter semantic trace CLI", () => { expected: { role: "sink", classes: ["parameter"] }, }, }), + ); + const traced = await runCli([ + "trace-javascript-semantics", + requestPath, "--json", ]); expect(traced).toMatchObject({ @@ -1571,8 +1597,11 @@ it("retains bundled isolated-world contextBridge APIs through dedicated and gene world: "isolated", world_id: { status: "literal", value: 1001, expression: null }, api_key: "isolatedApi", + api_status: "object-literal", members: ["ping"], - resolution_basis: "wrapper-substitution", + unknown_members: 0, + // The proven Webpack bootstrap resolves `load(2)` directly. + resolution_basis: "direct-syntax", }), evidence: expect.objectContaining({ location: expect.objectContaining({ diff --git a/tests/acceptance/applications/artifactComparisonCoverage.test.ts b/tests/acceptance/applications/artifactComparisonCoverage.test.ts index db86603114..42ab385205 100644 --- a/tests/acceptance/applications/artifactComparisonCoverage.test.ts +++ b/tests/acceptance/applications/artifactComparisonCoverage.test.ts @@ -105,7 +105,6 @@ const assertCoverageControls = async ({ derive({ ...inventory, coverage: { scope: "root", status: "unknown" }, - coverage_status: "partial", limitations: [], }), ); @@ -125,26 +124,14 @@ const assertCoverageControls = async ({ limitations: expect.arrayContaining(["Artifact inventory scopes differ."]), }); - const { coverage, coverage_status: _coverageStatus, ...legacy } = inventory; - expect(coverage).toBeDefined(); - const retained = await compare(left, derive(legacy)); - expect(retained.normalized_result).toMatchObject({ - status: "unchanged", - summary: { unchanged: 1, unknown: 0 }, - }); - for (const limitation of [ - "Child inventory is unavailable.", - "Missing child observations; 1 integrity contradiction(s) were recorded.", - ]) { - const unfamiliarLegacy = await compare( - left, - derive({ ...legacy, limitations: [limitation] }), - ); - expect(unfamiliarLegacy.normalized_result).toMatchObject({ - status: "unknown", - summary: { unchanged: 0, unknown: 1 }, - }); - } + const { coverage: _coverage, ...withoutCoverage } = inventory; + expect(() => artifactInventoryResultSchema.parse(withoutCoverage)).toThrow(); + expect(() => + artifactInventoryResultSchema.parse({ + ...inventory, + coverage_status: "complete", + }), + ).toThrow(); }; cliTest( diff --git a/tests/acceptance/applications/blockSourceMapCli.test.ts b/tests/acceptance/applications/blockSourceMapCli.test.ts index 957cd49c26..4e7cb7e115 100644 --- a/tests/acceptance/applications/blockSourceMapCli.test.ts +++ b/tests/acceptance/applications/blockSourceMapCli.test.ts @@ -73,6 +73,7 @@ cliTest.for(["CLI", "stdio MCP"] as const)( declared_url: url, declared_url_sha256: digest, resolved_path: url, + resolution_status: "resolved", available: true, }, evidence: expect.objectContaining({ @@ -92,7 +93,11 @@ cliTest.for(["CLI", "stdio MCP"] as const)( expect.objectContaining({ relation: "maps_to", target_node_id: map?.node_id, - properties: { declared_url_sha256: digest, resolved_path: url }, + properties: { + declared_url_sha256: digest, + resolved_path: url, + resolution_status: "resolved", + }, }), ); }, diff --git a/tests/acceptance/applications/electronInvalidPaths.test.ts b/tests/acceptance/applications/electronInvalidPaths.test.ts index e7d5935e21..d1b2d54876 100644 --- a/tests/acceptance/applications/electronInvalidPaths.test.ts +++ b/tests/acceptance/applications/electronInvalidPaths.test.ts @@ -1,4 +1,4 @@ -import { writeFile } from "node:fs/promises"; +import { chmod, mkdir, writeFile } from "node:fs/promises"; import { join } from "node:path"; import { expect } from "vitest"; @@ -68,6 +68,130 @@ cliTest( }, ); +const isWindows = process.platform === "win32"; + +const errorBody = (result: { + readonly content: readonly { readonly type: string }[]; +}) => { + const content = result.content.find((item) => item.type === "text"); + if (content?.type !== "text" || !("text" in content)) + throw new Error("Missing error response"); + return JSON.parse(String(content.text)); +}; + +const createElectronApplication = async (root: string) => { + const application = join(root, "app"); + await mkdir(application); + await writeFile( + join(application, "package.json"), + JSON.stringify({ name: "fixture", main: "main.js" }), + ); + await writeFile(join(application, "main.js"), ""); + return application; +}; + +cliTest.skipIf(isWindows)( + "rejects a non-executable Electron executable_path before launch through CLI and MCP", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-electron-not-executable-"); + const application = await createElectronApplication(root); + const executable = join(root, "not-an-executable.txt"); + await writeFile(executable, "plain text"); + await chmod(executable, 0o644); + const request = { + executable_path: executable, + application_path: application, + actions: [], + }; + const requestPath = join(root, "input.json"); + await writeFile(requestPath, JSON.stringify(request)); + + const response = await cli.run({ + arguments: ["capture-electron-scenario", requestPath, "--format", "json"], + environment: { REA_LOG_LEVEL: "silent" }, + }); + expect(response.exitCode).toBe(1); + expect(response.json).toMatchObject({ + code: "invalid_request", + details: { + issues: [{ path: ["executable_path"], reason: "invalid_value" }], + }, + }); + + const { call } = await connectLocalToolsMcp(); + const result = await call("capture_electron_scenario", request); + expect(result.isError).toBe(true); + expect(errorBody(result)).toMatchObject({ + error: { + code: "invalid_request", + details: { + issues: [{ path: ["executable_path"], reason: "invalid_value" }], + }, + }, + }); + const session = await call("binary_session", {}); + expect(session.isError, JSON.stringify(session)).not.toBe(true); + }, +); + +cliTest.skipIf(isWindows)( + "returns a typed failure when an executable exits before Electron starts and keeps the MCP server and its Evidence", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-electron-early-exit-"); + const application = await createElectronApplication(root); + const executable = join(root, "exits-immediately"); + await writeFile(executable, "#!/bin/sh\nexit 0\n"); + await chmod(executable, 0o755); + const request = { + executable_path: executable, + application_path: application, + actions: [], + }; + const requestPath = join(root, "input.json"); + await writeFile(requestPath, JSON.stringify(request)); + + const response = await cli.run({ + arguments: ["capture-electron-scenario", requestPath, "--format", "json"], + environment: { REA_LOG_LEVEL: "silent" }, + }); + expect(response.exitCode).toBe(1); + expect(response.json).toMatchObject({ + code: "execution_failure", + details: { operation: "capture_electron_scenario" }, + }); + expect(response.stdout).toContain("Process failed to launch"); + + const { call } = await connectLocalToolsMcp(); + const analysis = await call("analyze_javascript_application", { + input_path: application, + }); + expect(analysis.isError, JSON.stringify(analysis)).not.toBe(true); + const retained = async () => { + const bundle = await call("get_evidence_bundle", {}); + expect(bundle.isError, JSON.stringify(bundle)).not.toBe(true); + const structured = bundle.structuredContent as { + result: { records: { evidence_id: string }[] }; + }; + return structured.result.records.map((record) => record.evidence_id); + }; + const before = await retained(); + expect(before).toHaveLength(1); + + const result = await call("capture_electron_scenario", request); + expect(result.isError).toBe(true); + expect(errorBody(result)).toMatchObject({ + error: { + code: "execution_failure", + details: { operation: "capture_electron_scenario" }, + }, + }); + const session = await call("binary_session", {}); + expect(session.isError, JSON.stringify(session)).not.toBe(true); + expect(await retained()).toEqual(before); + }, + 180_000, +); + cliTest( "rejects a file URL path field without resolving an invented local path", async ({ cli }) => { diff --git a/tests/acceptance/applications/emptyImportNameCli.test.ts b/tests/acceptance/applications/emptyImportNameCli.test.ts new file mode 100644 index 0000000000..1805d8d6c0 --- /dev/null +++ b/tests/acceptance/applications/emptyImportNameCli.test.ts @@ -0,0 +1,70 @@ +import { writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { expect } from "vitest"; + +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { cliTest } from "../../support/cli/cliFixture.js"; +import { analyzeJavaScriptDirectory } from "../../support/cli/javascriptApplication.js"; + +cliTest.for(["CLI", "stdio MCP"] as const)( + "preserves empty ESM import names on local re-export origins through %s", + async (surface, { cli, processes }) => { + const root = await createTestTempDirectory("rea-empty-import-name-"); + const modulePath = join(root, "module.mjs"); + const appPath = join(root, "app.mjs"); + await writeFile( + modulePath, + 'const value = 1; export { value as "", value as "a", value as "0" };', + ); + await writeFile( + appPath, + [ + 'import { "" as empty, "a" as letter, "0" as zero } from "./module.mjs";', + "export { empty, letter, zero };", + 'export { "" as direct } from "./module.mjs";', + ].join("\n"), + ); + for (const path of [modulePath, appPath]) { + const checked = await processes.run(process.execPath, ["--check", path]); + expect(checked.exitCode, checked.stderr).toBe(0); + } + const result = await analyzeJavaScriptDirectory(surface, cli, root); + expect(result.statistics.parse_failures).toBe(0); + const imports = result.graph.edges.filter( + ({ relation, properties }) => + relation === "imports" && properties.specifier === "./module.mjs", + ); + for (const [name, local] of [ + ["", "empty"], + ["a", "letter"], + ["0", "zero"], + ]) { + for (const kind of ["import", "export"]) { + expect(imports).toContainEqual( + expect.objectContaining({ + properties: expect.objectContaining({ + module_link_kind: kind, + ...(kind === "export" + ? { exported_name: local } + : { local_name: local }), + imported_name: name, + imported_path: [name], + resolved_path: "module.mjs", + resolution_status: "resolved", + }), + }), + ); + } + } + expect(imports).toContainEqual( + expect.objectContaining({ + properties: expect.objectContaining({ + exported_name: "direct", + imported_name: "", + imported_path: [""], + resolution_status: "resolved", + }), + }), + ); + }, +); diff --git a/tests/acceptance/applications/exportedFunctionBindingsCli.test.ts b/tests/acceptance/applications/exportedFunctionBindingsCli.test.ts index d7fac06987..c9ee1aeb10 100644 --- a/tests/acceptance/applications/exportedFunctionBindingsCli.test.ts +++ b/tests/acceptance/applications/exportedFunctionBindingsCli.test.ts @@ -195,6 +195,121 @@ describe("exported function binding resolution through stdio MCP and CLI", () => ); }); +describe("CommonJS receiver lifetimes through stdio MCP and CLI", () => { + it.each([ + "module.exports = {}; exports.parse = VALUE;", + "exports = {}; exports.parse = VALUE;", + "exports.parse = VALUE; module.exports = {};", + "module.exports.parse = VALUE; module.exports = {};", + "module.exports = {}; exports = module.exports; module.exports = {}; exports.parse = VALUE;", + 'module.exports = {}; exports.parse = "enabled";', + 'module.exports = {}; Reflect.set(module, "other", exports); exports.parse = VALUE;', + "module.exports = {}; let wrapper = module; wrapper = {}; Object.assign(wrapper, { exports }); exports.parse = VALUE;", + ])("excludes detached public export receivers: %s", async (source) => { + const fixture = await analyzeVersions( + (count) => + source.replace( + "VALUE", + `() => ({ kind: "result", count: ${String(count)} })`, + ), + "parse", + "app.cjs", + ); + for (const path of [fixture.leftPath, fixture.rightPath]) { + const value = requireFixture(path); + expect(Object.hasOwn(value, "parse")).toBe(false); + expect(value.parse).toBeUndefined(); + } + for (const evidence of [fixture.input.left, fixture.input.right]) { + const { graph } = applicationEnvelope.parse(evidence).normalized_result; + expect( + graph.nodes + .flatMap(({ observations }) => observations) + .filter( + ({ properties }) => + properties.exported_name === "parse" && + ["export-binding", "export-return-shapes"].includes( + String(properties.semantic_role), + ), + ), + ).toEqual([]); + } + await expectCliAnalysisParity(fixture); + const comparison = await compareThroughMcp(fixture.input); + expect(comparison).toMatchObject({ + left: { status: "missing" }, + right: { status: "missing" }, + summary: { added: 0, removed: 0, changed: 0 }, + }); + expect(await compareThroughCli(fixture)).toEqual(comparison); + }); + + it.each([ + "exports.parse = VALUE;", + "module.exports = {}; module.exports.parse = VALUE;", + "module.exports = {}; exports = module.exports; exports.parse = VALUE;", + "exports.parse = VALUE; Object.assign(module.exports, { other: exports });", + ])("retains connected public export receivers: %s", async (source) => { + const fixture = await analyzeVersions( + (count) => + source.replace( + "VALUE", + `() => ({ kind: "result", count: ${String(count)} })`, + ), + "parse", + "app.cjs", + ); + for (const [path, count] of [ + [fixture.leftPath, 1], + [fixture.rightPath, 2], + ] as const) + expect(requireFixture(path).parse()).toEqual({ kind: "result", count }); + await expectCliAnalysisParity(fixture); + const comparison = await compareThroughMcp(fixture.input); + expectCountChange(comparison); + expect(await compareThroughCli(fixture)).toEqual(comparison); + }); + + it.each([ + "if (false) module.exports = {}; exports.parse = VALUE;", + "exports.parse = VALUE; if (false) module.exports = {};", + "function replace() { module.exports = {}; } exports.parse = VALUE;", + "function replace() { exports = {}; } exports.parse = VALUE;", + 'const holder = {}; ({ [(module.exports = exports, "x")]: holder.x } = (module.exports = {})); exports.parse = VALUE;', + 'const { [(module.exports = exports, "x")]: value } = (module.exports = {}); exports.parse = VALUE;', + "const [value = (module.exports = exports)] = (module.exports = []); exports.parse = VALUE;", + "module.exports = {}; Object.assign(module, { exports }); exports.parse = VALUE;", + 'module.exports = {}; Object.defineProperty(module, "exports", { value: exports }); exports.parse = VALUE;', + 'module.exports = {}; Reflect.set(module, "exports", exports); exports.parse = VALUE;', + "module.exports = {}; const assign = Object.assign; assign.call(null, module, { exports }); exports.parse = VALUE;", + "module.exports = {}; const wrapper = module; Object.assign(wrapper, { exports }); exports.parse = VALUE;", + ])( + "keeps execution-dependent receiver identity uncertain: %s", + async (source) => { + const fixture = await analyzeVersions( + (count) => + source.replace( + "VALUE", + `() => ({ kind: "result", count: ${String(count)} })`, + ), + "parse", + "app.cjs", + ); + for (const [path, count] of [ + [fixture.leftPath, 1], + [fixture.rightPath, 2], + ] as const) + expect(requireFixture(path).parse()).toEqual({ kind: "result", count }); + for (const evidence of [fixture.input.left, fixture.input.right]) + expectUnresolvedExportBinding(evidence, "parse"); + await expectCliAnalysisParity(fixture); + const comparison = await compareThroughMcp(fixture.input); + expectUncertainComparison(comparison); + expect(await compareThroughCli(fixture)).toEqual(comparison); + }, + ); +}); + describe("CommonJS assignment-chain receiver identity through stdio MCP", () => { it("does not export a named write whose captured module.exports receiver is replaced", async () => { const fixture = await analyzeVersions( @@ -462,6 +577,48 @@ describe("stable exported function bindings through stdio MCP", () => { const fixture = await analyzeVersions(source, "default"); expectCountChange(await compareThroughMcp(fixture.input)); }); + + it.each([ + { + name: "a local callable captured by an identifier default export", + source: (count: number) => + `const run = () => ({ kind: "result", count: ${String(count)} });\nexport default run;`, + runtimeValue: (count: number) => ({ kind: "result", count }), + expected: "known", + }, + { + name: "the latest callable assigned before the default export", + source: (count: number) => + `let run = () => ({ count: 0 });\nrun = () => ({ kind: "result", count: ${String(count)} });\nexport default run;`, + runtimeValue: (count: number) => ({ kind: "result", count }), + expected: "known", + }, + { + name: "a callable assigned only after the default export capture", + source: (count: number) => + `var run;\nexport default run;\nrun = () => ({ kind: "result", count: ${String(count)} });`, + runtimeValue: () => undefined, + expected: "unknown", + }, + { + name: "a reassignment after the default export capture", + source: (count: number) => + `let run = () => ({ kind: "result", count: ${String(count)} });\nexport default run;\nrun = () => ({ count: 99 });`, + runtimeValue: (count: number) => ({ kind: "result", count }), + expected: "known", + }, + ] as const)( + "uses the value at $name", + async ({ source, runtimeValue, expected }) => { + const fixture = await analyzeVersions(source, "default"); + await expectRuntimeExports(fixture, runtimeValue); + const mcp = await compareThroughMcp(fixture.input); + const cli = await compareThroughCli(fixture); + expect(cli).toEqual(mcp); + if (expected === "known") expectCountChange(mcp); + else expectUncertainComparison(mcp); + }, + ); }); describe("export bindings with loop and assignment-only writes", () => { diff --git a/tests/acceptance/applications/interfaceBuilderHierarchyLimits.test.ts b/tests/acceptance/applications/interfaceBuilderHierarchyLimits.test.ts new file mode 100644 index 0000000000..af508592cf --- /dev/null +++ b/tests/acceptance/applications/interfaceBuilderHierarchyLimits.test.ts @@ -0,0 +1,174 @@ +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { build } from "plist"; +import { expect, onTestFinished } from "vitest"; + +import { interfaceBuilderAnalysisSchema } from "../../../src/domain/apple/interfaceBuilderGraph.js"; +import { thinMach } from "../../../src/domain/binaryTarget.fixture.js"; +import { parseEvidence } from "../../../src/domain/evidence.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { cliTest } from "../../support/cli/cliFixture.js"; +import { createTestMcp } from "../../support/mcp/mcpFixture.js"; +import { openArtifactTarget } from "../../support/mcp/artifactSession.js"; + +const verifyMcpParity = async ( + app: string, + expected: ReturnType, +): Promise => { + const mcp = createTestMcp(); + onTestFinished(async () => mcp.close()); + const client = await openArtifactTarget(mcp, onTestFinished, app); + const complete = await client.callTool({ + name: "decode_interface_builder", + arguments: { max_objects: 2 }, + }); + expect(complete.isError, JSON.stringify(complete)).not.toBe(true); + const evidence = parseEvidence(complete.structuredContent); + expect( + interfaceBuilderAnalysisSchema.parse(evidence.normalized_result), + ).toEqual(expected); + + const summarized = await client.callTool({ + name: "decode_interface_builder", + arguments: { max_objects: 2, detail: "summary" }, + }); + expect(summarized.isError, JSON.stringify(summarized)).not.toBe(true); + expect( + parseEvidence(summarized.structuredContent).normalized_result, + ).toMatchObject({ + kind: "summary", + parent_evidence_id: evidence.evidence_id, + summary: { + documents: expected.documents, + truncated: true, + coverage_statuses: { partial: expect.any(Number) }, + }, + }); +}; + +const archive = (objects: readonly string[], hierarchy: readonly string[]) => + build({ + "com.apple.ibtool.document.objects": Object.fromEntries( + objects.map((id) => [id, { class: "UIView" }]), + ), + "com.apple.ibtool.document.hierarchy": hierarchy.map((objectID) => ({ + objectID, + })), + }); + +const cases = [ + { + name: "a later document's root is omitted", + archives: [ + ["First.nib", archive(["first"], ["first"])], + ["Second.nib", archive(["second"], ["second"])], + ] as const, + hierarchyComplete: [true, false], + document: "Second.nib", + omittedEdges: 1, + unresolvedEdge: false, + }, + { + name: "a retained hierarchy edge has an omitted endpoint", + archives: [ + ["Single.nib", archive(["kept", "omitted"], ["omitted"])], + ] as const, + hierarchyComplete: [false], + document: "Single.nib", + omittedEdges: 0, + unresolvedEdge: true, + }, +]; + +cliTest.for(cases)( + "reports partial hierarchy through the CLI when $name", + async ( + { archives, hierarchyComplete, document, omittedEdges, unresolvedEdge }, + { cli }, + ) => { + const root = await createTestTempDirectory("rea-ib-hierarchy-limits-"); + const app = join(root, "Fixture.app"); + const contents = join(app, "Contents"); + const resources = join(contents, "Resources"); + await mkdir(resources, { recursive: true }); + await mkdir(join(contents, "MacOS")); + await writeFile( + join(contents, "MacOS", "Fixture"), + thinMach(0xfeedfacf, 0x0100000c), + ); + await writeFile( + join(contents, "Info.plist"), + build({ CFBundleExecutable: "Fixture" }), + ); + for (const [name, contents] of archives) + await writeFile(join(resources, name), contents); + + const output = await cli.run({ + arguments: [ + "decode-interface-builder", + app, + "--max-objects", + "2", + "--json", + ], + environment: { REA_LOG_LEVEL: "silent", REA_ANALYSIS_PROVIDER: "auto" }, + }); + expect(output.exitCode, output.stdout + output.stderr).toBe(0); + const evidence = parseEvidence(output.json); + const report = interfaceBuilderAnalysisSchema.parse( + evidence.normalized_result, + ); + await verifyMcpParity(app, report); + expect( + report.documents.map(({ hierarchy_complete }) => hierarchy_complete), + ).toEqual(hierarchyComplete); + expect(report.graph.truncated).toBe(true); + expect(report.graph.nodes).toHaveLength(2); + expect(report.graph.edges).toHaveLength(1); + expect(report.graph.coverage).toContainEqual( + expect.objectContaining({ + facet: `hierarchy:Contents/Resources/${document}`, + status: "partial", + reason: "graph_limit_reached", + examined: 1, + omitted: omittedEdges, + }), + ); + if (unresolvedEdge) + expect(report.graph.edges[0]).toMatchObject({ + relation: "contains", + resolution: "unresolved", + to: null, + reason: "endpoint_omitted_by_object_limit", + }); + + const viewPath = join(root, "summary.json"); + await writeFile( + viewPath, + JSON.stringify({ + source: { kind: "inline", evidence: output.json }, + view: { kind: "summary" }, + }), + ); + const summaryOutput = await cli.run({ + arguments: ["inspect-analysis-view", viewPath, "--json"], + environment: { REA_LOG_LEVEL: "silent" }, + }); + expect( + summaryOutput.exitCode, + summaryOutput.stdout + summaryOutput.stderr, + ).toBe(0); + expect(parseEvidence(summaryOutput.json).normalized_result).toMatchObject({ + kind: "summary", + parent_evidence_id: evidence.evidence_id, + summary: { + documents: report.documents, + truncated: true, + coverage_statuses: { partial: expect.any(Number) }, + }, + }); + for (const [name, contents] of archives) + expect(await readFile(join(resources, name), "utf8")).toBe(contents); + }, +); diff --git a/tests/acceptance/applications/javascriptBuiltinDefaultsCli.test.ts b/tests/acceptance/applications/javascriptBuiltinDefaultsCli.test.ts index 6dbf855fc4..36cb537aee 100644 --- a/tests/acceptance/applications/javascriptBuiltinDefaultsCli.test.ts +++ b/tests/acceptance/applications/javascriptBuiltinDefaultsCli.test.ts @@ -8,9 +8,252 @@ import { expect, it } from "vitest"; import { parseEvidence } from "../../../src/domain/evidence.js"; import { javascriptApplicationAnalysisResultSchema } from "../../../src/domain/javascript/javascriptApplicationAnalysis.js"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { connectLocalToolsMcp } from "../../fixtures/localToolsMcp.js"; const execute = promisify(execFile); +it("preserves earlier and sibling effects without inventing replaced cached-module or process effects", async () => { + const root = await createTestTempDirectory("rea-builtin-identity-"); + await writeFile( + join(root, "app.cjs"), + ` + throw new Error('parse only'); + const a = require('node:fs'); const b = require('fs'); + b.readFileSync('before.json'); const stream = b.createReadStream('before.json'); stream.destroy(); + patch(a); b.readFileSync('wrong-after.json'); + const proc = require('node:process'); proc.env = {}; proc.argv = []; const mode = proc.env.MODE; const arg = proc.argv[2]; + `, + ); + await writeFile( + join(root, "sibling.mjs"), + `import fs from 'node:fs'; fs.writeFileSync = custom; fs.readFileSync('sibling.json'); fs.readFileSync = custom;`, + ); + await writeFile( + join(root, "captures.cjs"), + `const fs = require('node:fs'); + const saved = require('fs').readFileSync; + const { readFileSync: savedDestructured } = fs; + fs.readFileSync('before-write.json'); fs.readFileSync = custom; + const { readFileSync: late } = require('fs'); + const lateMember = require('fs').readFileSync; + saved('saved.json'); savedDestructured('saved-destructured.json'); + late('wrong-late.json'); lateMember('wrong-member.json');`, + ); + await writeFile( + join(root, "direct.cjs"), + `require('node:fs').readFileSync('direct.json'); + const fs = require('fs'); fs.readFileSync('later.json'); + require('fs/promises').open('before'); + fs.promises.open = custom; require('node:fs/promises').open('wrong');`, + ); + await writeFile( + join(root, "loader.cjs"), + `require = () => ({ readFileSync() {} }); + const fs = require('node:fs'); fs.readFileSync('wrong-loader.json'); + require('node:fs').readFileSync('wrong-direct-loader.json');`, + ); + await writeFile( + join(root, "imports.ts"), + `import fs = require('node:fs'); const other = require('fs'); + const read = other.readFileSync; observe(read); read('escaped.json'); + fs.readFileSync = custom; other.readFileSync('wrong-import-equals.json');`, + ); + await writeFile( + join(root, "late-loader.cjs"), + `const fs=require('fs'); fs.readFileSync('loader-before.json'); + require=replacement; fs.readFileSync('loader-after.json'); + require('fs').readFileSync('wrong-late-loader.json');`, + ); + await writeFile( + join(root, "replacement.ts"), + `import fs = require('fs'); const other=require('fs'); fs={}; fs.readFileSync=custom; other.readFileSync('replacement-kept.json');`, + ); + await writeFile( + join(root, "dynamic.cjs"), + `const fs=require('fs'); fs.hook=custom; fs[key](); fs.readFileSync('wrong-dynamic.json');`, + ); + await writeFile( + join(root, "wrapped.cjs"), + `const fs=require('fs'); let assigned; const saved=(0, fs.readFileSync); fs.readFileSync=custom; const late=(assigned=fs.readFileSync); saved('wrapped-kept.json'); late('wrong-wrapped.json');`, + ); + await writeFile( + join(root, "global-process.cjs"), + `process.env={MODE:'local'}; const proc=require('process'); const mode=proc.env.MODE;`, + ); + const { stdout } = await execute( + process.execPath, + ["scripts/rea.mjs", "analyze-javascript-application", root, "--json"], + { cwd: process.cwd(), maxBuffer: 16 * 1024 * 1024 }, + ); + const evidence = parseEvidence(JSON.parse(stdout)); + const result = javascriptApplicationAnalysisResultSchema.parse( + evidence.normalized_result, + ); + expect( + result.semantic_graph.nodes + .filter(({ kind }) => kind === "config-source") + .map(({ properties }) => properties.key) + .sort(), + ).toEqual([ + "before-write.json", + "before.json", + "direct.json", + "escaped.json", + "later.json", + "loader-after.json", + "loader-before.json", + "replacement-kept.json", + "saved-destructured.json", + "saved.json", + "sibling.json", + "wrapped-kept.json", + ]); + expect( + result.semantic_graph.nodes.filter(({ kind }) => kind === "resource"), + ).toHaveLength(2); + expect( + result.semantic_graph.relations.filter( + ({ relation }) => relation === "releases", + ), + ).toHaveLength(1); + const { client, call } = await connectLocalToolsMcp(); + const response = await call("analyze_javascript_application", { + input_path: root, + format: "auto", + }); + expect(response.isError, JSON.stringify(response)).not.toBe(true); + expect(response.structuredContent).toEqual(evidence); + await client.ping(); +}, 60000); + +it("excludes monkey-patched CommonJS effects from CLI and MCP evidence", async () => { + const root = await createTestTempDirectory("rea-builtin-mutated-"); + await writeFile( + join(root, "app.cjs"), + ` + throw new Error("parse only"); + const fs = require("node:fs"); fs.readFileSync = custom; fs.createReadStream = custom; + const cp = require("node:child_process"); cp.spawn = custom; + const timers = require("node:timers"); patch(timers); + const http = require("node:http"); delete http.get; + fs.readFileSync("config.json"); const stream = fs.createReadStream("config.json"); stream.destroy(); + const child = cp.spawn("tool", []); child.kill(); + const timer = timers.setTimeout(callback, 1); clearTimeout(timer); + http.get("http://example.test"); + `, + ); + const { stdout } = await execute( + process.execPath, + ["scripts/rea.mjs", "analyze-javascript-application", root, "--json"], + { cwd: process.cwd(), maxBuffer: 16 * 1024 * 1024 }, + ); + const evidence = parseEvidence(JSON.parse(stdout)); + const result = javascriptApplicationAnalysisResultSchema.parse( + evidence.normalized_result, + ); + expect( + result.semantic_graph.nodes.filter(({ kind }) => + [ + "config-source", + "resource", + "child-process", + "timer", + "request", + ].includes(kind), + ), + ).toEqual([]); + expect( + result.semantic_graph.relations.filter(({ relation }) => + ["releases", "forwards-signal", "cancels-timer"].includes(relation), + ), + ).toEqual([]); + const { client, call } = await connectLocalToolsMcp(); + const response = await call("analyze_javascript_application", { + input_path: root, + format: "auto", + }); + expect(response.isError, JSON.stringify(response)).not.toBe(true); + expect(response.structuredContent).toEqual(evidence); + await client.ping(); +}, 60000); + +it("retains CommonJS builtin effects and handle links through CLI and MCP", async () => { + const root = await createTestTempDirectory("rea-builtin-commonjs-"); + await writeFile( + join(root, "app.cjs"), + ` + throw new Error("This source must only be parsed, never executed"); + const fs = require("node:fs"); + const {spawn: launch} = require("node:child_process"); + const timers = require("timers"); + const {get} = require("http"); + const proc = require("node:process"); + const text = fs.readFileSync("config.json", "utf8"); + const stream = fs.createReadStream("config.json"); stream.destroy(); + const child = launch("tool", []); child.kill("SIGTERM"); + const timer = timers.setTimeout(callback, 10); timers.clearTimeout(timer); + get("http://example.test"); + const mode = proc.env.MODE; + const absent = require("node:fs").default; absent.readFileSync("wrong-default"); + const external = require("./fs.js"); external.readFileSync("wrong-external"); + function shadowed(require) { const fs = require("node:fs"); fs.readFileSync("wrong"); } + `, + ); + const { stdout } = await execute( + process.execPath, + ["scripts/rea.mjs", "analyze-javascript-application", root, "--json"], + { cwd: process.cwd(), maxBuffer: 16 * 1024 * 1024 }, + ); + const evidence = parseEvidence(JSON.parse(stdout)); + const result = javascriptApplicationAnalysisResultSchema.parse( + evidence.normalized_result, + ); + const nodes = result.semantic_graph.nodes; + expect( + nodes.filter(({ kind }) => + [ + "config-source", + "resource", + "child-process", + "timer", + "request", + ].includes(kind), + ), + ).toHaveLength(6); + expect(nodes).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + kind: "config-source", + properties: expect.objectContaining({ key: "config.json" }), + }), + expect.objectContaining({ + kind: "config-source", + properties: expect.objectContaining({ key: "MODE" }), + }), + ...["createReadStream", "spawn", "setTimeout", "get"].map((method) => + expect.objectContaining({ + properties: expect.objectContaining({ method }), + }), + ), + ]), + ); + const relations = result.semantic_graph.relations; + for (const relation of ["releases", "forwards-signal", "cancels-timer"]) + expect(relations).toEqual( + expect.arrayContaining([ + expect.objectContaining({ relation, resolution: "resolved" }), + ]), + ); + const { client, call } = await connectLocalToolsMcp(); + const response = await call("analyze_javascript_application", { + input_path: root, + format: "auto", + }); + expect(response.isError, JSON.stringify(response)).not.toBe(true); + expect(response.structuredContent).toEqual(evidence); + await client.ping(); +}, 60000); + it.each([ "import fs from 'node:fs';", "import {default as fs} from 'node:fs';", diff --git a/tests/acceptance/applications/javascriptCancellationCli.test.ts b/tests/acceptance/applications/javascriptCancellationCli.test.ts index c270e678c2..473ae34362 100644 --- a/tests/acceptance/applications/javascriptCancellationCli.test.ts +++ b/tests/acceptance/applications/javascriptCancellationCli.test.ts @@ -58,13 +58,39 @@ it.skipIf(process.platform === "win32").each([ parsedFiles: 1, }, ])( - "returns typed cancellation and exit 130 after SIGINT through $command during $phase", + "returns typed cancellation and exit 130 through $command during $phase", async ({ command, phase, parsedFiles }) => { const root = await createTestTempDirectory("rea-cli-js-cancellation-"); await writeFile(join(root, "main.js"), "export const observed = 1;\n"); + const nodeArguments: string[] = []; + if (parsedFiles !== null) { + // A tiny graph/hash phase may finish before the parent receives stderr. + // Invoke the real signal handler synchronously in the owned CLI child at + // that progress boundary; the two early-phase cases still use OS SIGINT. + const controls = await createTestTempDirectory("rea-cli-cancel-control-"); + const preload = join(controls, "cancel-at-phase.mjs"); + await writeFile( + preload, + ` + const write = process.stderr.write.bind(process.stderr); + let requested = false; + process.stderr.write = function(chunk, ...args) { + const written = write(chunk, ...args); + let update; + try { update = JSON.parse(String(chunk)); } catch {} + if (!requested && update?.rea_progress?.phase === ${JSON.stringify(phase)}) { + requested = true; + process.emit("SIGINT"); + } + return written; + }; + `, + ); + nodeArguments.push("--import", preload); + } const child = spawn( process.execPath, - ["scripts/rea.mjs", command, root, "--json"], + [...nodeArguments, "scripts/rea.mjs", command, root, "--json"], { stdio: ["ignore", "pipe", "pipe"], }, @@ -95,7 +121,7 @@ it.skipIf(process.platform === "win32").each([ !requested ) { requested = true; - child.kill("SIGINT"); + if (parsedFiles === null) child.kill("SIGINT"); } } }); diff --git a/tests/acceptance/applications/javascriptConfigurationReadsCli.test.ts b/tests/acceptance/applications/javascriptConfigurationReadsCli.test.ts index 8b9ba1abc9..db7af4df5c 100644 --- a/tests/acceptance/applications/javascriptConfigurationReadsCli.test.ts +++ b/tests/acceptance/applications/javascriptConfigurationReadsCli.test.ts @@ -76,3 +76,24 @@ cliTest.for(["CLI", "stdio MCP"] as const)( } }, ); + +cliTest.for(["CLI", "stdio MCP"] as const)( + "does not claim an argv value after an unknown object selection escapes through %s", + async (surface, { cli }) => { + const root = await createTestTempDirectory("rea-config-escape-"); + await writeFixtureFiles(root, { + "entry.mjs": [ + 'throw new Error("static fixture must never execute");', + "export function run(key) { patch(process.argv[key]); return process.argv[3]; }", + "export function independent() { return process.env.PORT; }", + ].join("\n"), + }); + const graph = (await analyzeJavaScriptDirectory(surface, cli, root)) + .semantic_graph; + const reads = graph.nodes + .filter((node) => node.kind === "config-source") + .map((node) => [node.properties.source_kind, node.properties.key]); + expect(reads).not.toContainEqual(["argv", "3"]); + expect(reads).toContainEqual(["environment", "PORT"]); + }, +); diff --git a/tests/acceptance/applications/javascriptEndpointCandidateCli.test.ts b/tests/acceptance/applications/javascriptEndpointCandidateCli.test.ts new file mode 100644 index 0000000000..e741971adc --- /dev/null +++ b/tests/acceptance/applications/javascriptEndpointCandidateCli.test.ts @@ -0,0 +1,40 @@ +import { execFile } from "node:child_process"; +import { writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { promisify } from "node:util"; + +import { expect, it } from "vitest"; + +import { parseEvidence } from "../../../src/domain/evidence.js"; +import { javascriptApplicationAnalysisResultSchema } from "../../../src/domain/javascript/javascriptApplicationAnalysis.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +const execute = promisify(execFile); + +it("reports a candidate endpoint through the worker boundary in the public CLI", async () => { + const root = await createTestTempDirectory("rea-endpoint-candidate-cli-"); + // A method-like call whose first argument is neither a URL nor a path. + await writeFile(join(root, "main.js"), 'client.post("session");\n'); + const { stdout } = await execute( + process.execPath, + ["scripts/rea.mjs", "analyze-javascript-application", root, "--json"], + { cwd: process.cwd(), maxBuffer: 16 * 1024 * 1024 }, + ); + const result = javascriptApplicationAnalysisResultSchema.parse( + parseEvidence(JSON.parse(stdout)).normalized_result, + ); + const endpoint = result.graph.nodes.find(({ kind }) => kind === "endpoint"); + expect(endpoint?.observations[0]).toMatchObject({ + properties: { + endpoint_kind: "network", + value: "session", + mechanism: "call:client.post", + interpretation: "candidate", + }, + evidence: { state: "inferred", confidence: "low" }, + }); + const edge = result.graph.edges.find( + ({ target_node_id }) => target_node_id === endpoint?.node_id, + ); + expect(edge?.properties).toMatchObject({ interpretation: "candidate" }); +}, 20_000); diff --git a/tests/acceptance/applications/javascriptEndpointPayload.test.ts b/tests/acceptance/applications/javascriptEndpointPayload.test.ts index 6a980aa7ad..75b54e26e4 100644 --- a/tests/acceptance/applications/javascriptEndpointPayload.test.ts +++ b/tests/acceptance/applications/javascriptEndpointPayload.test.ts @@ -108,7 +108,9 @@ cliTest( ), ).toHaveLength(1); expect(repeatedNode.observations).toHaveLength(2); - expect(repeatedNode.identity).toMatchObject({ key: "/ok" }); + expect(repeatedNode.identity).toMatchObject({ + key: JSON.stringify(["literal", "/ok"]), + }); expect( repeatedNode.observations.every(({ label }) => label === "/ok"), ).toBe(true); diff --git a/tests/acceptance/applications/javascriptFeatureTraceSeedSelectionCli.test.ts b/tests/acceptance/applications/javascriptFeatureTraceSeedSelectionCli.test.ts new file mode 100644 index 0000000000..a0438556d7 --- /dev/null +++ b/tests/acceptance/applications/javascriptFeatureTraceSeedSelectionCli.test.ts @@ -0,0 +1,256 @@ +import { Client } from "@modelcontextprotocol/client"; +import { StdioClientTransport } from "@modelcontextprotocol/client/stdio"; +import { writeFile } from "node:fs/promises"; +import { join, resolve } from "node:path"; +import { expect } from "vitest"; +import { z } from "zod"; + +import { toolContract } from "../../../src/contracts/toolContracts.js"; +import { + applicationFeatureSeedSchema, + applicationFeatureTraceResultSchema, +} from "../../../src/domain/javascript/javascriptFeatureTraceSchemas.js"; +import { javascriptApplicationAnalysisResultSchema } from "../../../src/domain/javascript/javascriptApplicationAnalysis.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { cliTest, type TestCli } from "../../support/cli/cliFixture.js"; + +type FeatureSeed = z.input; +type Trace = z.infer; +type SeedScenario = { + readonly name: string; + readonly seed: FeatureSeed; + readonly expected: { + readonly nodeId: string; + readonly basis: "node-id" | "property"; + } | null; +}; + +const createSeedScenarios = ( + graph: z.infer["graph"], +): SeedScenario[] => { + const notificationRoute = graph.nodes.find( + (node) => + node.kind === "endpoint" && + node.observations.some( + ({ properties }) => + properties.endpoint_kind === "route" && + properties.value === "/accounts/preferences/notifications", + ), + ); + const tinyRoute = graph.nodes.find( + (node) => + node.kind === "endpoint" && + node.observations.some( + ({ properties }) => + properties.endpoint_kind === "route" && properties.value === "/tiny", + ), + ); + const labeledNode = graph.nodes.find((node) => + node.observations.some( + ({ label }) => label !== null && label !== node.node_id, + ), + ); + if (notificationRoute === undefined || tinyRoute === undefined) + throw new Error( + "Expected literal route endpoints in the application graph", + ); + if (labeledNode === undefined) + throw new Error("Expected a labeled node in the application graph"); + const label = labeledNode.observations.find( + ({ label: value }) => value !== null && value !== labeledNode.node_id, + )?.label; + if (label === undefined || label === null) + throw new Error("Expected the selected graph node to have a display label"); + + const id = labeledNode.node_id; + return [ + { + name: "node ID contains prefix", + seed: { kind: "node-id", value: id.slice(0, 24), match: "contains" }, + expected: { nodeId: id, basis: "node-id" }, + }, + { + name: "node ID exact ignores case by default", + seed: { kind: "node-id", value: id.toUpperCase(), match: "exact" }, + expected: { nodeId: id, basis: "node-id" }, + }, + { + name: "node ID exact can be case sensitive", + seed: { + kind: "node-id", + value: id.toUpperCase(), + match: "exact", + case_sensitive: true, + }, + expected: null, + }, + { + name: "node ID never falls back to a display label", + seed: { kind: "node-id", value: label, match: "exact" }, + expected: null, + }, + { + name: "generated route label exact", + seed: { + kind: "route", + value: "route endpoint", + match: "exact", + case_sensitive: true, + }, + expected: null, + }, + { + name: "generated route label substring", + seed: { kind: "route", value: "route", match: "contains" }, + expected: null, + }, + { + name: "preserved long route exact", + seed: { + kind: "route", + value: "/accounts/preferences/notifications", + match: "exact", + }, + expected: { nodeId: notificationRoute.node_id, basis: "property" }, + }, + { + name: "preserved route substring", + seed: { + kind: "route", + value: "/accounts/preferences", + match: "contains", + }, + expected: { nodeId: notificationRoute.node_id, basis: "property" }, + }, + { + name: "short preserved route exact", + seed: { kind: "route", value: "/tiny", match: "exact" }, + expected: { nodeId: tinyRoute.node_id, basis: "property" }, + }, + ]; +}; + +const analyzeFixture = async (cli: TestCli, root: string) => { + const response = await cli.run({ + arguments: [ + "analyze-javascript-application", + root, + "--artifact-format", + "directory", + "--json", + ], + }); + expect(response.exitCode).toBe(0); + const evidence = toolContract( + "analyze_javascript_application", + ).outputSchema.parse(response.json); + return { + evidence, + result: javascriptApplicationAnalysisResultSchema.parse( + evidence.normalized_result, + ), + }; +}; + +const traceThroughCli = async ( + cli: TestCli, + application: unknown, + scenario: SeedScenario, +): Promise => { + const response = await cli.run({ + arguments: [ + "trace-application-feature", + JSON.stringify({ + application, + seed: scenario.seed, + direction: "both", + }), + "--json", + ], + }); + expect(response.exitCode, `${scenario.name} via CLI`).toBe(0); + const evidence = toolContract("trace_application_feature").outputSchema.parse( + response.json, + ); + return applicationFeatureTraceResultSchema.parse(evidence.normalized_result); +}; + +const traceThroughMcp = async ( + client: Client, + application: unknown, + scenario: SeedScenario, +): Promise => { + const response = await client.callTool({ + name: "trace_application_feature", + arguments: { + application, + seed: scenario.seed, + direction: "both", + }, + }); + expect(response.isError, `${scenario.name} via MCP`).not.toBe(true); + const evidence = toolContract("trace_application_feature").outputSchema.parse( + response.structuredContent, + ); + return applicationFeatureTraceResultSchema.parse(evidence.normalized_result); +}; + +const expectTrace = (trace: Trace, scenario: SeedScenario): void => { + if (scenario.expected === null) { + expect(trace.seed_matches).toEqual([]); + expect(trace.summary.matched_seeds).toBe(0); + expect(trace.coverage.status).toBe("no-match"); + return; + } + expect(trace.seed_matches).toHaveLength(1); + expect(trace.seed_matches[0]).toMatchObject({ + node_id: scenario.expected.nodeId, + basis: scenario.expected.basis, + }); + expect(trace.summary.matched_seeds).toBe(1); + expect(trace.coverage.status).toBe("complete-within-source"); + if (scenario.seed.kind === "node-id") + expect(trace.seed_matches[0]).toMatchObject({ + basis: "node-id", + field: "node_id", + }); + else + expect(trace.seed_matches[0]?.field).toMatch( + /\.properties\.(value|path)$/u, + ); +}; + +cliTest( + "matches feature seeds against node IDs and preserved route values through CLI and MCP", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-feature-seed-selection-"); + await writeFile( + join(root, "routes.js"), + `const notificationRoute = { path: "/accounts/preferences/notifications" };\nconst tinyRoute = { path: "/tiny" };\nmodule.exports = { notificationRoute, tinyRoute };\n`, + ); + const { evidence, result } = await analyzeFixture(cli, root); + const client = new Client({ name: "feature-seed-selection", version: "1" }); + const transport = new StdioClientTransport({ + command: process.execPath, + args: [resolve("scripts/rea.mjs"), "mcp"], + cwd: process.cwd(), + env: { PATH: process.env.PATH ?? "", REA_LOG_LEVEL: "silent" }, + stderr: "pipe", + }); + transport.stderr?.on("data", () => undefined); + try { + await client.connect(transport); + for (const scenario of createSeedScenarios(result.graph)) { + const cliTrace = await traceThroughCli(cli, evidence, scenario); + const mcpTrace = await traceThroughMcp(client, evidence, scenario); + expectTrace(cliTrace, scenario); + expectTrace(mcpTrace, scenario); + expect(mcpTrace.seed_matches).toEqual(cliTrace.seed_matches); + expect(mcpTrace.coverage).toEqual(cliTrace.coverage); + } + } finally { + await client.close(); + await transport.close(); + } + }, +); diff --git a/tests/acceptance/applications/javascriptFunctionFingerprintsCli.test.ts b/tests/acceptance/applications/javascriptFunctionFingerprintsCli.test.ts new file mode 100644 index 0000000000..f66c08366a --- /dev/null +++ b/tests/acceptance/applications/javascriptFunctionFingerprintsCli.test.ts @@ -0,0 +1,127 @@ +import { mkdir, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { expect } from "vitest"; + +import { toolContract } from "../../../src/contracts/toolContracts.js"; +import { javascriptApplicationAnalysisResultSchema } from "../../../src/domain/javascript/javascriptApplicationAnalysis.js"; +import { javaScriptSemanticTraceResultSchema } from "../../../src/domain/javascript/javascriptSemanticTraceSchemas.js"; +import { connectLocalToolsMcp } from "../../fixtures/localToolsMcp.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { cliTest } from "../../support/cli/cliFixture.js"; + +cliTest( + "function fingerprint seeds retain directive values and normalized duplicates through CLI and MCP", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-function-fingerprints-"); + const application = join(root, "application"); + await mkdir(application); + await writeFile( + join(application, "app.cjs"), + String.raw` + throw new Error("This analysis fixture must not execute"); + function strict(value) { "use strict"; return value; } + function ordinary(value) { "ordinary"; return value; } + function asm(value) { "use asm"; return value; } + function alpha(value) { "alpha"; return value; } + function beta(value) { "beta"; return value; } + function escaped(value) { "use\x20strict"; return value; } + function renamed(input) { 'use strict'; return input; } + `, + ); + const analyzed = await cli.run({ + arguments: [ + "analyze-javascript-application", + application, + "--artifact-format", + "directory", + "--json", + ], + environment: { REA_LOG_LEVEL: "silent" }, + }); + expect(analyzed.exitCode, analyzed.stderr).toBe(0); + const evidence = toolContract( + "analyze_javascript_application", + ).outputSchema.parse(analyzed.json); + const { semantic_graph: graph } = + javascriptApplicationAnalysisResultSchema.parse( + evidence.normalized_result, + ); + const { client, call } = await connectLocalToolsMcp(); + const response = await call("analyze_javascript_application", { + input_path: application, + format: "directory", + }); + expect(response.isError).not.toBe(true); + const retained = toolContract( + "analyze_javascript_application", + ).outputSchema.parse(response.structuredContent); + expect(retained.evidence_id).toBe(evidence.evidence_id); + const fingerprint = (name: string) => { + const node = graph.nodes.find( + (candidate) => + candidate.kind === "function" && candidate.properties.name === name, + ); + if (node === undefined) throw new Error(`Missing function ${name}`); + const result = graph.fingerprints.find( + ({ function_node_id }) => function_node_id === node.node_id, + ); + if (result === undefined) throw new Error(`Missing fingerprint ${name}`); + expect(result.status).toBe("complete"); + return result; + }; + for (const [selected, expectedNames] of [ + ["strict", ["strict", "renamed"]], + ["ordinary", ["ordinary"]], + ["asm", ["asm"]], + ["alpha", ["alpha"]], + ["beta", ["beta"]], + ["escaped", ["escaped"]], + ] as const) { + const query = { + seed: { + kind: "function", + fingerprint_sha256: fingerprint(selected).fingerprint_sha256, + }, + direction: "forward-influence", + }; + const traced = await call("trace_javascript_semantics", { + application: { + kind: "retained-evidence", + evidence_id: retained.evidence_id, + }, + query, + }); + expect(traced.isError, JSON.stringify(traced)).not.toBe(true); + const traceEvidence = toolContract( + "trace_javascript_semantics", + ).outputSchema.parse(traced.structuredContent); + const trace = javaScriptSemanticTraceResultSchema.parse( + traceEvidence.normalized_result, + ); + expect(trace.seed_node_ids).toEqual( + expectedNames.map((name) => fingerprint(name).function_node_id).sort(), + ); + expect(trace.status).toBe( + expectedNames.length > 1 ? "ambiguous" : "found", + ); + expect(trace.source_graph_id).toBe(graph.graph_id); + if (selected === "strict") { + const requestPath = join(root, "trace-request.json"); + await writeFile( + requestPath, + JSON.stringify({ application: evidence, query }), + ); + const cliTrace = await cli.run({ + arguments: ["trace-javascript-semantics", requestPath, "--json"], + environment: { REA_LOG_LEVEL: "silent" }, + }); + expect(cliTrace.exitCode, cliTrace.stderr).toBe(0); + const cliEvidence = toolContract( + "trace_javascript_semantics", + ).outputSchema.parse(cliTrace.json); + expect(cliEvidence.evidence_id).toBe(traceEvidence.evidence_id); + } + } + await client.ping(); + }, +); diff --git a/tests/acceptance/applications/javascriptHtmlUrlSuffixesCli.test.ts b/tests/acceptance/applications/javascriptHtmlUrlSuffixesCli.test.ts new file mode 100644 index 0000000000..88e94144ff --- /dev/null +++ b/tests/acceptance/applications/javascriptHtmlUrlSuffixesCli.test.ts @@ -0,0 +1,136 @@ +import { expect } from "vitest"; + +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { cliTest } from "../../support/cli/cliFixture.js"; +import { analyzeJavaScriptDirectory } from "../../support/cli/javascriptApplication.js"; +import { writeFixtureFiles } from "../../support/javascriptApplicationFixture.js"; + +cliTest.for(["CLI", "stdio MCP"] as const)( + "admits HTML script pathnames without treating URL suffixes as paths through %s", + async (surface, { cli }) => { + const root = await createTestTempDirectory("rea-html-url-suffixes-"); + const cases = [ + { + source: "query.html", + script: "./app.js?redirect=%2Ffoo", + resolved: "app.js", + }, + { + source: "fragment.html", + script: "./app.js#route=%2Ffoo", + resolved: "app.js", + }, + { + source: "suffix-bytes.html", + script: "./app.js?value=%2e%5c%00%ff\\tail", + resolved: "app.js", + }, + { + source: "url-text.html", + script: " \n./a\tpp.js?redirect=%2\nfassets ", + resolved: "app.js", + }, + { + source: "self.html", + script: "?redirect=%2Ffoo", + resolved: "self.html", + }, + { + source: "self-fragment.html", + script: "#route=%5Cfoo", + resolved: "self-fragment.html", + }, + { + source: "base.html", + base: "assets/?redirect=%2Froot", + script: "app.js#route=%2Ffoo", + resolved: "assets/app.js", + }, + { + source: "base-self.html", + base: "assets/app.js?redirect=%5Croot", + script: "?redirect=%2Ffoo", + resolved: "assets/app.js", + }, + { + source: "path-separator.html", + script: "./assets%2Fapp.js?valid=1", + resolved: null, + }, + { source: "path-dot.html", script: "./%2e/app.js#valid", resolved: null }, + { + source: "path-backslash.html", + script: "./assets\\app.js?valid=1", + resolved: null, + }, + { + source: "path-nul.html", + script: "./app%00.js?valid=1", + resolved: null, + }, + { + source: "split-separator.html", + script: "./assets%2\nfapp.js?valid=1", + resolved: null, + }, + { + source: "base-separator.html", + base: "assets%2F?valid=1", + script: "app.js?redirect=%2Ffoo", + resolved: null, + }, + ]; + await writeFixtureFiles(root, { + ...Object.fromEntries( + cases.map(({ source, base, script }) => [ + source, + `${base === undefined ? "" : ``}`, + ]), + ), + "app.js": "globalThis.rootApp = true;", + "assets/app.js": "globalThis.nestedApp = true;", + }); + + const result = await analyzeJavaScriptDirectory(surface, cli, root); + const loads = result.graph.edges.filter( + ({ relation }) => relation === "loads", + ); + const unresolved = result.graph.nodes + .flatMap(({ observations }) => observations) + .filter( + ({ properties }) => properties.mechanism === "html-script-reference", + ); + for (const { source, base, script, resolved } of cases) { + const load = loads.find( + ({ evidence }) => + evidence.location.available && + evidence.location.value.kind === "source-range" && + evidence.location.value.source === source, + ); + if (resolved === null) { + expect( + unresolved.find( + ({ properties }) => properties.script_path === script, + ), + ).toMatchObject({ + properties: { + script_path: script, + resolved_path: null, + resolution_status: "rejected", + }, + }); + expect(load).toBeUndefined(); + continue; + } + expect(load).toMatchObject({ + properties: { + script_path: script, + base_href: base ?? null, + resolved_path: resolved, + resolution_context: "html-reference", + resolution_status: "resolved", + }, + }); + } + }, +); diff --git a/tests/acceptance/applications/javascriptRpcPropertyOwnershipCli.test.ts b/tests/acceptance/applications/javascriptRpcPropertyOwnershipCli.test.ts new file mode 100644 index 0000000000..a369e7519d --- /dev/null +++ b/tests/acceptance/applications/javascriptRpcPropertyOwnershipCli.test.ts @@ -0,0 +1,57 @@ +import { expect } from "vitest"; + +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { cliTest } from "../../support/cli/cliFixture.js"; +import { + analyzeJavaScriptDirectory, + runNodeJsonControl, +} from "../../support/cli/javascriptApplication.js"; +import { writeFixtureFiles } from "../../support/javascriptApplicationFixture.js"; + +cliTest.for(["CLI", "stdio MCP"] as const)( + "retains only final known RPC object properties through %s", + async (surface, { cli }) => { + const root = await createTestTempDirectory("rea-rpc-properties-"); + await writeFixtureFiles(root, { + "entry.mjs": ` + function httpBatchLink(options) { return options.url; } + const options = { url: "/trpc/runtime" }; + const key = "url"; + const links = [ + httpBatchLink({ url: "/trpc/old", url: "/trpc/live" }), + httpBatchLink({ url: "/trpc/spread-old", ...options }), + httpBatchLink({ url: "/trpc/computed-old", [key]: options.url }), + httpBatchLink({ ...options, url: "/trpc/final", headers: {} }), + ]; + const doc = { + kind: "OperationDefinition", operation: "query", operation: "mutation", + name: { kind: "Name", value: "Old" }, + name: { kind: "Name", value: "Current" }, + }; + console.log(JSON.stringify([links, doc.operation, doc.name.value])); + `, + }); + // This control checks JavaScript object semantics, not a live tRPC client. + expect(await runNodeJsonControl(root, "entry.mjs")).toEqual([ + ["/trpc/live", "/trpc/runtime", "/trpc/runtime", "/trpc/final"], + "mutation", + "Current", + ]); + const { graph } = await analyzeJavaScriptDirectory(surface, cli, root); + const endpoints = graph.nodes + .filter((node) => node.kind === "endpoint") + .flatMap((node) => node.observations) + .filter(({ properties }) => + /^(trpc:link:|graphql:)/u.test(String(properties.mechanism)), + ) + .map(({ properties }) => [properties.value, properties.mechanism]); + expect(endpoints).toEqual( + expect.arrayContaining([ + ["/trpc/live", "trpc:link:httpBatchLink"], + ["/trpc/final", "trpc:link:httpBatchLink"], + ["Current", "graphql:mutation"], + ]), + ); + expect(endpoints).toHaveLength(3); + }, +); diff --git a/tests/acceptance/applications/javascriptWorkerUrlCli.test.ts b/tests/acceptance/applications/javascriptWorkerUrlCli.test.ts index 6d9b62a63d..1039f5d3a1 100644 --- a/tests/acceptance/applications/javascriptWorkerUrlCli.test.ts +++ b/tests/acceptance/applications/javascriptWorkerUrlCli.test.ts @@ -279,12 +279,6 @@ cliTest( specifier: "./worker.js", resolved: "bundled/worker.js", }, - { - source: "nested/literal.mjs", - expression: 'new Worker("./literal-worker.js");', - specifier: "./literal-worker.js", - resolved: "nested/literal-worker.js", - }, ]; await writeFixtureFiles(root, { ...Object.fromEntries( @@ -390,6 +384,9 @@ cliTest( ]), ), "dynamic-base.mjs": 'new Worker(new URL("./worker.js", base));', + "empty-worker-url.mjs": 'new Worker("");', + "query-only-worker-url.mjs": 'new Worker("?cache=1");', + "fragment-only-worker-url.mjs": 'new Worker("#worker");', "shadowed.mjs": 'function start(URL) { return new Worker(new URL("./worker.js", import.meta.url)); }', "typed-shadowed.ts": @@ -472,6 +469,57 @@ cliTest( }, ); +cliTest( + "keeps string Worker paths unresolved because the document base is unknown", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-worker-document-base-"); + // A browser resolves these strings against the document base URL, not the + // calling script; Node resolves them against the working directory. + const expressions = { + "scripts/main.js": 'new Worker("./worker.js")', + "scripts/service.js": 'navigator.serviceWorker.register("./sw.js")', + }; + await writeFixtureFiles(root, { + "index.html": '', + ...Object.fromEntries( + Object.entries(expressions).map(([path, code]) => [path, `${code};`]), + ), + "worker.js": "self.postMessage('document-relative worker');", + "scripts/worker.js": "self.postMessage('script-relative decoy');", + "scripts/sw.js": "self.postMessage('script-relative decoy');", + }); + + const result = await analyzeJavaScriptDirectory("CLI", cli, root); + for (const [source, expression] of Object.entries(expressions)) { + const node = result.graph.nodes.find( + ({ kind, observations }) => + (kind === "worker" || kind === "service-worker") && + observations.some( + ({ evidence }) => + evidence.location.available && + evidence.location.value.kind === "source-range" && + evidence.location.value.source === source, + ), + ); + expect(node?.observations).toContainEqual( + expect.objectContaining({ + properties: expect.objectContaining({ + declared_specifier: null, + expression, + resolved_path: null, + }), + }), + ); + expect( + result.graph.edges.some( + ({ relation, source_node_id }) => + relation === "maps_to" && source_node_id === node?.node_id, + ), + ).toBe(false); + } + }, +); + cliTest( "bounds repeated ambient alias branches through the compiled CLI", async ({ cli }) => { diff --git a/tests/acceptance/applications/packageMainLookupCli.test.ts b/tests/acceptance/applications/packageMainLookupCli.test.ts new file mode 100644 index 0000000000..efdef8a552 --- /dev/null +++ b/tests/acceptance/applications/packageMainLookupCli.test.ts @@ -0,0 +1,284 @@ +import { expect } from "vitest"; + +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { cliTest } from "../../support/cli/cliFixture.js"; +import { + analyzeJavaScriptDirectory, + runNodeJsonControl, +} from "../../support/cli/javascriptApplication.js"; +import { writeFixtureFiles } from "../../support/javascriptApplicationFixture.js"; + +const cases = [ + { name: "null-main", metadata: { main: null } }, + { name: "false-main", metadata: { main: false } }, + { name: "zero-main", metadata: { main: 0 } }, + { name: "true-main", metadata: { main: true } }, + { name: "object-main", metadata: { main: {} } }, + { name: "empty-main", metadata: { main: "" } }, + { name: "missing-main", metadata: {} }, + { name: "string-main", metadata: { main: "./index.js" } }, +] as const; + +cliTest.for(["CLI", "stdio MCP"] as const)( + "matches Node package main interpretation through %s", + async (surface, { cli }) => { + const root = await createTestTempDirectory("rea-package-main-"); + const files: Record = { + "package.json": JSON.stringify({ type: "commonjs" }), + "entry.cjs": cases + .map(({ name }) => `require(${JSON.stringify(name)});`) + .join("\n"), + "entry.mjs": cases + .map(({ name }) => `import ${JSON.stringify(name)};`) + .join("\n"), + "control.cjs": [ + `const specifiers = ${JSON.stringify(cases.map(({ name }) => name))};`, + "console.log(JSON.stringify(specifiers.map((specifier) => require(specifier))));", + ].join("\n"), + "control.mjs": [ + `const specifiers = ${JSON.stringify(cases.map(({ name }) => name))};`, + "console.log(JSON.stringify(await Promise.all(specifiers.map(async (specifier) => (await import(specifier)).default))));", + ].join("\n"), + }; + for (const { name, metadata } of cases) { + files[`node_modules/${name}/package.json`] = JSON.stringify(metadata); + files[`node_modules/${name}/index.js`] = + `module.exports = ${JSON.stringify(`node_modules/${name}/index.js`)};`; + } + await writeFixtureFiles(root, files); + + for (const filename of ["control.cjs", "control.mjs"]) { + expect(await runNodeJsonControl(root, filename)).toEqual( + cases.map(({ name }) => `node_modules/${name}/index.js`), + ); + } + + const { graph } = await analyzeJavaScriptDirectory(surface, cli, root); + for (const kind of ["require", "import"]) + for (const { name } of cases) + expect( + graph.edges + .filter(({ relation }) => relation === "imports") + .map(({ properties }) => properties), + ).toContainEqual( + expect.objectContaining({ + module_link_kind: kind, + specifier: name, + resolved_path: `node_modules/${name}/index.js`, + resolution_status: "resolved", + }), + ); + }, +); + +cliTest.for(["CLI", "stdio MCP"] as const)( + "preserves loader-specific package main probes and finalization through %s", + async (surface, { cli }) => { + const root = await createTestTempDirectory("rea-package-main-path-"); + const packages = [ + { + name: "dot-slash-main", + main: "./lib/", + files: ["lib.js", "lib/index.js", "index.js"], + require: "lib.js", + import: null, + }, + { + name: "slash-main", + main: "lib/", + files: ["lib.js", "lib/index.js", "index.js"], + require: "lib.js", + import: null, + }, + { + name: "plain-main", + main: "./lib", + files: ["lib.js", "lib/index.js"], + require: "lib.js", + import: "lib.js", + }, + { + name: "directory-main", + main: "./lib/", + files: ["lib/index.js", "index.js"], + require: "lib/index.js", + import: "lib/index.js", + }, + { + name: "fallback-main", + main: "./missing/", + files: ["index.js"], + require: "index.js", + import: "index.js", + }, + { + name: "exact-main", + main: "./lib.js/", + files: ["lib.js", "index.js"], + require: "lib.js", + import: "lib.js", + }, + { + name: "finalized-directory-main", + main: "./lib/", + files: ["lib.js", "lib/.js/child.js"], + require: "lib.js", + import: null, + importError: "ERR_UNSUPPORTED_DIR_IMPORT", + }, + { + name: "finalized-main", + main: "./lib/", + files: ["lib.js", "lib/.js", "lib/index.js"], + require: "lib.js", + import: "lib/.js", + }, + ] as const; + const files: Record = { + "package.json": JSON.stringify({ type: "commonjs" }), + "entry.cjs": [ + ...packages.map(({ name }) => `require(${JSON.stringify(name)});`), + 'require("./local/");', + ].join("\n"), + "entry.mjs": packages + .map(({ name }) => `import ${JSON.stringify(name)};`) + .join("\n"), + "local.js": 'module.exports = "local.js";', + "local/index.js": 'module.exports = "local/index.js";', + }; + for (const item of packages) { + files[`node_modules/${item.name}/package.json`] = JSON.stringify({ + main: item.main, + }); + for (const path of item.files) { + const target = `node_modules/${item.name}/${path}`; + files[target] = `module.exports = ${JSON.stringify(target)};`; + } + } + for (const kind of ["require", "import"] as const) { + const filename = `control.${kind === "require" ? "cjs" : "mjs"}`; + files[filename] = [ + `const specifiers = ${JSON.stringify(packages.map(({ name }) => name))};`, + "const results = [];", + "for (const specifier of specifiers) {", + ` try { results.push({ path: ${kind === "require" ? "require(specifier)" : "(await import(specifier)).default"} }); }`, + " catch (error) { results.push({ code: error.code }); }", + "}", + "console.log(JSON.stringify(results));", + ].join("\n"); + } + files["local-control.cjs"] = + 'console.log(JSON.stringify(require("./local/")));'; + await writeFixtureFiles(root, files); + for (const kind of ["require", "import"] as const) { + const control = await runNodeJsonControl( + root, + `control.${kind === "require" ? "cjs" : "mjs"}`, + ); + expect(control).toEqual( + packages.map((item) => + item[kind] === null + ? { + code: + "importError" in item + ? item.importError + : "ERR_MODULE_NOT_FOUND", + } + : { path: `node_modules/${item.name}/${item[kind]}` }, + ), + ); + } + expect(await runNodeJsonControl(root, "local-control.cjs")).toBe( + "local/index.js", + ); + const { graph } = await analyzeJavaScriptDirectory(surface, cli, root); + const relationships = graph.edges + .filter(({ relation }) => relation === "imports") + .map(({ properties }) => properties); + for (const kind of ["require", "import"] as const) + for (const item of packages) + expect(relationships).toContainEqual( + expect.objectContaining({ + module_link_kind: kind, + specifier: item.name, + resolved_path: + item[kind] === null + ? null + : `node_modules/${item.name}/${item[kind]}`, + resolution_status: + item[kind] === null + ? "importError" in item + ? "rejected" + : "not-found" + : "resolved", + }), + ); + expect(relationships).toContainEqual( + expect.objectContaining({ + module_link_kind: "require", + specifier: "./local/", + resolved_path: "local/index.js", + resolution_status: "resolved", + }), + ); + }, +); + +cliTest.for(["CLI", "stdio MCP"] as const)( + "resolves package mains that name the artifact root through %s", + async (surface, { cli }) => { + const mains = ["../..", "../../", "../../."]; + const names = mains.map((_, index) => `root-main-${index}`); + const requires = names.map((name) => `require("${name}")`); + for (const rootIndex of ["index.js", "index.json", null]) { + const root = await createTestTempDirectory("rea-package-main-root-"); + const importOptions = + rootIndex === "index.json" ? ', { with: { type: "json" } }' : ""; + const imports = names.map( + (name) => + `import("${name}"${importOptions}).then(module => module.default)`, + ); + const files: Record = { + "package.json": '{"type":"commonjs"}', + "entry.cjs": requires.join(";\n"), + "entry.mjs": names.map((name) => `import "${name}";`).join("\n"), + ".js": 'module.exports = ".js decoy";', + ".json": '".json decoy"', + "control.cjs": `console.log(JSON.stringify([${requires.join(",")}]));`, + "control.mjs": `console.log(JSON.stringify(await Promise.all([${imports.join(",")}])));`, + }; + if (rootIndex !== null) + files[rootIndex] = + rootIndex === "index.js" + ? 'module.exports = "index.js";' + : '"index.json"'; + for (const [index, main] of mains.entries()) { + const directory = `node_modules/${names[index]}`; + files[`${directory}/package.json`] = JSON.stringify({ main }); + files[`${directory}/index.js`] = + `module.exports = "${directory}/index.js";`; + } + await writeFixtureFiles(root, files); + const expected = names.map( + (name) => rootIndex ?? `node_modules/${name}/index.js`, + ); + for (const filename of ["control.cjs", "control.mjs"]) + expect(await runNodeJsonControl(root, filename)).toEqual(expected); + + const { graph } = await analyzeJavaScriptDirectory(surface, cli, root); + const relationships = graph.edges + .filter(({ relation }) => relation === "imports") + .map(({ properties }) => properties); + for (const kind of ["require", "import"]) + for (const [index, name] of names.entries()) + expect(relationships).toContainEqual( + expect.objectContaining({ + module_link_kind: kind, + specifier: name, + resolved_path: expected[index], + resolution_status: "resolved", + }), + ); + } + }, +); diff --git a/tests/acceptance/applications/packageSpecifierSuffixCli.test.ts b/tests/acceptance/applications/packageSpecifierSuffixCli.test.ts index ad175006e0..ee319ff7e0 100644 --- a/tests/acceptance/applications/packageSpecifierSuffixCli.test.ts +++ b/tests/acceptance/applications/packageSpecifierSuffixCli.test.ts @@ -18,6 +18,23 @@ type PackageSpecifierCase = { readonly error: string | null; }; +const directoryFormSpecifiers = [ + "legacy/feature.js/.", + "legacy/feature.js/child/..", + "legacy/feature.js/", + "legacy/feature.js/.?cache=1#entry", + "legacy/with%20space.js/.", + "legacy/missing/.", + "legacy/feature.js/.\t", + "legacy/feature.js/.\n?cache=1#entry", + "legacy/feature.js/ch\til\rd/..", + "legacy/feature.js/. ", +]; +const encodedDotSpecifiers = ["\t", "\n", "\r"].flatMap((control) => [ + `legacy/child/%2${control}e%2${control}e/feature.js`, + `legacy/child/.%2${control}E/feature.js`, +]); + const requests = [ { specifier: "rea-self-fixture", @@ -131,6 +148,74 @@ const requests = [ value: "legacy spaced feature", error: null, }, + ...directoryFormSpecifiers.map((specifier) => ({ + specifier, + target: null, + status: "rejected" as const, + value: null, + error: "ERR_UNSUPPORTED_DIR_IMPORT", + })), + ...encodedDotSpecifiers.map((specifier) => ({ + specifier, + target: null, + status: "rejected" as const, + value: "legacy feature", + error: null, + })), + { + specifier: "legacy/feature.js/%2e", + target: null, + status: "rejected", + value: null, + error: "ERR_UNSUPPORTED_DIR_IMPORT", + }, + { + specifier: "legacy/feature.js/.%09", + target: null, + status: "not-found", + value: null, + error: "ERR_MODULE_NOT_FOUND", + }, + { + specifier: "legacy/feature.js/%2f", + target: null, + status: "rejected", + value: null, + error: "ERR_INVALID_MODULE_SPECIFIER", + }, + ...[ + "legacy/feature.js?cache=/..", + "legacy/feature.js#entry/.", + "legacy/fea\tture.js", + "legacy/feature.js?cache=%2\te", + "legacy/feature.js#entry=%2\nf", + "legacy/child/../feature.js", + ].map((specifier) => ({ + specifier, + target: "node_modules/legacy/feature.js", + status: "resolved" as const, + value: "legacy feature", + error: null, + })), + ...[ + "fixture/feature.js/.", + "fixture/feature.js/.\t", + "fixture/child/%2\te%2\te/feature", + "legacy/../fixture/literal.cjs", + ].map((specifier) => ({ + specifier, + target: "node_modules/fixture/literal.cjs", + status: "resolved" as const, + value: "literal", + error: null, + })), + { + specifier: "legacy/literal%252e.js", + target: "node_modules/legacy/literal%2e.js", + status: "resolved", + value: "legacy literal percent", + error: null, + }, { specifier: "legacy/feature", target: null, @@ -184,6 +269,9 @@ const createPackageSpecifierFixture = async (): Promise<{ exports: { "./feature": "./feature.cjs", "./literal?cache=1": "./literal.cjs", + "./feature.js/.": "./literal.cjs", + "./feature.js/.\t": "./literal.cjs", + "./child/%2\te%2\te/feature": "./literal.cjs", }, }), ); @@ -206,6 +294,10 @@ const createPackageSpecifierFixture = async (): Promise<{ join(legacyPackagePath, "with space.js"), 'module.exports = "legacy spaced feature";', ), + writeFile( + join(legacyPackagePath, "literal%2e.js"), + 'module.exports = "legacy literal percent";', + ), writeFile( join(legacyPackagePath, "nested", "package.json"), JSON.stringify({ main: "./main.cjs" }), @@ -258,6 +350,20 @@ cliTest.for(["CLI", "stdio MCP"] as const)( resolution_status: status, }), evidence: expect.objectContaining({ + ...(encodedDotSpecifiers.includes(specifier) + ? { + limitations: expect.arrayContaining([ + "Encoded dot or separator bytes are rejected before artifact path resolution.", + ]), + } + : {}), + ...(directoryFormSpecifiers.includes(specifier) + ? { + limitations: expect.arrayContaining([ + `ESM package subpath ${JSON.stringify(specifier)} has URL directory form; directory imports are unsupported and no exact-file, package main or index lookup was attempted.`, + ]), + } + : {}), ...(specifier === "legacy/feature" ? { limitations: expect.arrayContaining([ diff --git a/tests/acceptance/applications/packageUrlUnicodeCli.test.ts b/tests/acceptance/applications/packageUrlUnicodeCli.test.ts new file mode 100644 index 0000000000..a50b4c837d --- /dev/null +++ b/tests/acceptance/applications/packageUrlUnicodeCli.test.ts @@ -0,0 +1,268 @@ +import { createPackageWithOptions } from "@electron/asar"; +import { createHash } from "node:crypto"; +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { expect } from "vitest"; + +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { cliTest } from "../../support/cli/cliFixture.js"; +import { analyzeJavaScriptDirectory } from "../../support/cli/javascriptApplication.js"; +import { writeFixtureFiles } from "../../support/javascriptApplicationFixture.js"; + +const escapingTarget = "./%2\te%2\te/outside.js"; +const targets = { + feature: { value: "./feature.js", imports: "relative", exports: "relative" }, + encodedSeparator: { + value: "./feature%2\tf.js", + imports: "rejected", + exports: "rejected", + }, + malformedEncoding: { + value: "./feature%ZZ.js", + imports: "rejected", + exports: "rejected", + }, + blocked: { value: null, imports: "external", exports: "external" }, + empty: { value: [], imports: "external", exports: "external" }, + inactive: { + value: { browser: "./feature.js" }, + imports: "external", + exports: "external", + }, + external: { + value: "another-package", + imports: "external", + exports: "rejected", + }, + recursive: { value: "#another", imports: "external", exports: "rejected" }, + invalid: { value: "../feature.js", imports: "rejected", exports: "rejected" }, + scalar: { value: false, imports: "rejected", exports: "rejected" }, + numeric: { value: { 0: null }, imports: "rejected", exports: "rejected" }, + arrayNull: { value: [false, null], imports: "external", exports: "external" }, + arrayInactive: { + value: [{ browser: "./feature.js" }], + imports: "external", + exports: "external", + }, + arrayExternal: { + value: [false, "another-package"], + imports: "external", + exports: "rejected", + }, + arrayRelative: { + value: [false, "./feature.js"], + imports: "relative", + exports: "relative", + }, + relativeFirst: { + value: ["./feature.js", null], + imports: "relative", + exports: "relative", + }, + escapeExternal: { + value: [escapingTarget, "another-package"], + imports: "external", + exports: "rejected", + }, + escapeNull: { + value: [escapingTarget, null], + imports: "external", + exports: "external", + }, + escapeRelative: { + value: [escapingTarget, "./feature.js"], + imports: "relative", + exports: "relative", + }, + escapeCondition: { + value: { node: escapingTarget, default: "./feature.js" }, + imports: "rejected", + exports: "rejected", + }, + escapeNested: { + value: [{ node: escapingTarget }, "another-package"], + imports: "external", + exports: "rejected", + }, + escapeConfig: { + value: [{ 0: escapingTarget }, "another-package"], + imports: "rejected", + exports: "rejected", + }, + wtf8Reentry: { + value: ["./%2\te%2\te/%ED%A0%80/feature.js", "another-package"], + imports: "external", + exports: "rejected", + }, + opaqueReentry: { + value: ["./%2\te%2\te/rea-op\taque-0/feature.js", "another-package"], + imports: "external", + exports: "rejected", + }, + structuralReentry: { + value: "./%2\te%2\te/mapped/feature.js", + imports: "rejected", + exports: "relative", + }, +} as const; +const requests = [ + { + specifier: "legacy/feature.js", + path: "node_modules/legacy/feature.js", + status: "legacy" as const, + }, + { + specifier: "legacy/feature.js/.", + path: "", + status: "rejected" as const, + }, + { + specifier: "legacy/../mapped/feature.js", + path: "node_modules/mapped/feature.js", + status: "legacy" as const, + }, + { + specifier: "legacy/../mapped/feature.js/.", + path: "", + status: "rejected" as const, + }, + { + specifier: "legacy/../../../outside.js", + path: "outside.js", + status: "root-legacy" as const, + }, + { + specifier: "legacy/../../../absent.js", + path: "", + status: "not-found" as const, + }, + ...Object.entries(targets).flatMap(([name, target]) => [ + { specifier: `#${name}`, path: "feature.js", status: target.imports }, + { + specifier: `mapped/${name}`, + path: "node_modules/mapped/feature.js", + status: target.exports, + }, + ]), + { specifier: "#missing", path: "", status: "external" as const }, + { specifier: "mapped/missing", path: "", status: "external" as const }, +]; +const sources = ["import", "require"].map((mode) => ({ + mode, + path: mode === "import" ? "main.mjs" : "main.cjs", + source: requests + .filter(({ specifier }) => mode === "import" || !specifier.endsWith("/.")) + .map(({ specifier }) => + mode === "import" + ? `import ${JSON.stringify(specifier)};` + : `require(${JSON.stringify(specifier)});`, + ) + .join("\n"), +})); + +cliTest.for(["CLI", "stdio MCP"] as const)( + "preserves package target facts under unrepresentable URL paths through %s", + async (surface, { cli }) => { + const root = await createTestTempDirectory("rea-package-url-unicode-"); + const application = join(root, "application"); + await mkdir(application); + const fixture = join(root, "source"); + await writeFixtureFiles(fixture, { "outside.js": "export default 4;" }); + const prefixes = ["highxx", "lowxxx", "valid\ud83d\ude00"]; + for (const prefix of prefixes) + await writeFixtureFiles(join(fixture, prefix), { + ...Object.fromEntries( + sources.map(({ path, source }) => [path, source]), + ), + "package.json": JSON.stringify({ + imports: Object.fromEntries( + Object.entries(targets).map(([name, { value }]) => [ + `#${name}`, + value, + ]), + ), + }), + "feature.js": "export default 1;", + "node_modules/legacy/package.json": "{}", + "node_modules/legacy/feature.js": "export default 2;", + "node_modules/mapped/package.json": JSON.stringify({ + exports: Object.fromEntries( + Object.entries(targets).map(([name, { value }]) => [ + `./${name}`, + value, + ]), + ), + }), + "node_modules/mapped/feature.js": "export default 3;", + }); + const archive = join(application, "app.asar"); + await createPackageWithOptions(fixture, archive, {}); + const bytes = await readFile(archive); + const header = bytes.subarray(16, 16 + bytes.readUInt32LE(12)); + const escaped = Buffer.from( + header + .toString("utf8") + .replace('"highxx":', '"\\ud800":') + .replace('"lowxxx":', '"\\udc00":'), + ); + expect(escaped.length).toBe(header.length); + expect(escaped.equals(header)).toBe(false); + escaped.copy(bytes, 16); + await writeFile(archive, bytes); + + const { graph } = await analyzeJavaScriptDirectory( + surface, + cli, + application, + ); + for (const prefix of ["\ud800", "\udc00", "valid\ud83d\ude00"]) + for (const { mode, path, source } of sources) + for (const request of requests) { + if (mode === "require" && request.specifier.endsWith("/.")) continue; + const base = `app.asar/${prefix}`; + const available = + prefix.startsWith("valid") || + request.status === "root-legacy" || + (mode === "require" && request.status === "legacy"); + const fileStatus = available ? "resolved" : "unavailable"; + const status = + request.status === "relative" || + request.status === "legacy" || + request.status === "root-legacy" + ? fileStatus + : request.status; + expect(graph.edges).toContainEqual( + expect.objectContaining({ + relation: "imports", + properties: expect.objectContaining({ + module_link_kind: mode, + specifier: request.specifier, + resolution_status: status, + resolved_path: + status === "resolved" + ? `${request.status === "root-legacy" ? "app.asar" : base}/${request.path}` + : null, + }), + evidence: expect.objectContaining({ + limitations: + status === "unavailable" + ? expect.arrayContaining([ + expect.stringContaining("unpaired UTF-16 surrogate"), + ]) + : expect.not.arrayContaining([ + expect.stringContaining("unpaired UTF-16 surrogate"), + ]), + artifact: expect.objectContaining({ + sha256: createHash("sha256").update(source).digest("hex"), + }), + location: expect.objectContaining({ + value: expect.objectContaining({ source: `${base}/${path}` }), + }), + }), + }), + ); + } + expect(await readFile(archive)).toEqual(bytes); + }, +); diff --git a/tests/acceptance/applications/runtime.test.ts b/tests/acceptance/applications/runtime.test.ts index 80989478ce..d085d27713 100644 --- a/tests/acceptance/applications/runtime.test.ts +++ b/tests/acceptance/applications/runtime.test.ts @@ -13,6 +13,7 @@ import { toolAvailabilityInventory, } from "../../../src/contracts/toolOutputSchemaPrimitives.js"; import { parseEvidence } from "../../../src/domain/evidence.js"; +import { parseMcpToolError } from "../../fixtures/mcpToolError.js"; const mainPath = fileURLToPath( new URL("../../../dist/main.js", import.meta.url), @@ -69,10 +70,13 @@ const completeStderrRecords = (stderr: string): unknown[] => .filter((line) => line.length > 0) .map((line): unknown => JSON.parse(line)); -const callCurrentDocument = async (target?: { - readonly path: string; - readonly kind?: "database"; -}) => { +const callCurrentDocument = async ( + target?: { + readonly path: string; + readonly kind?: "database"; + }, + inspectSession?: (client: Client) => Promise, +) => { const transport = new StdioClientTransport({ command: process.execPath, args: [mainPath], @@ -105,6 +109,7 @@ const callCurrentDocument = async (target?: { name: "current_document", arguments: {}, }); + await inspectSession?.(client); } finally { await client.close(); await transport.close(); @@ -240,18 +245,62 @@ describe("production stdio runtime", () => { ); expect(fatal).toEqual([]); }); +}); +describe("production stdio saved-database admission", () => { it.runIf(process.platform === "darwin")( - "serves a database-kind initial target without a fatal record on macOS", + "refuses database analysis on macOS while keeping the session usable without a fatal record", async () => { - const { result, records } = await callCurrentDocument({ - path: fileURLToPath(import.meta.url), - kind: "database", + const path = fileURLToPath(import.meta.url); + const { result, records } = await callCurrentDocument( + { path, kind: "database" }, + async (client) => { + const status = await expectAvailableToolInventory( + client, + await client.listTools(), + ); + expect(status.structuredContent).toMatchObject({ + result: { + open: true, + path, + kind: "database", + format: "analysis-database", + analysis_provider_binding: null, + }, + }); + }, + ); + expect(parseMcpToolError(result)).toMatchObject({ + error: { + code: "capability_unavailable", + category: "unsupported_provider", + retryable: false, + details: { + operation: "current_document", + selection_reason: "target_unsupported", + rejections: expect.arrayContaining([ + expect.objectContaining({ + provider_id: "hopper", + code: "target_kind_unsupported", + reason: expect.stringContaining( + "cannot start REA's analysis bridge", + ), + diagnostics: { + target_kind: "database", + target_format: "analysis-database", + architecture: null, + }, + }), + ]), + }, + }, }); - expect(result.isError).not.toBe(true); - evidenceResultOf(z.literal("fixture")).parse(result.structuredContent); expect(records).toContainEqual( - expect.objectContaining({ tool: "current_document", status: "ok" }), + expect.objectContaining({ + tool: "current_document", + status: "error", + level: 40, + }), ); expect( records.some( diff --git a/tests/acceptance/applications/typescriptThisParametersCli.test.ts b/tests/acceptance/applications/typescriptThisParametersCli.test.ts new file mode 100644 index 0000000000..a90dddec5c --- /dev/null +++ b/tests/acceptance/applications/typescriptThisParametersCli.test.ts @@ -0,0 +1,140 @@ +import { join } from "node:path"; + +import { expect } from "vitest"; + +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { cliTest } from "../../support/cli/cliFixture.js"; +import { analyzeJavaScriptDirectory } from "../../support/cli/javascriptApplication.js"; +import { writeFixtureFiles } from "../../support/javascriptApplicationFixture.js"; + +cliTest.for(["CLI", "stdio MCP"] as const)( + "maps arguments to runtime TypeScript parameters through %s", + async (surface, { cli, processes }) => { + const root = await createTestTempDirectory("rea-this-parameters-"); + await writeFixtureFiles(root, { + "bundle.mts": [ + "self.webpackChunkApp.push([[1], { 7: function(this: void, module: unknown, exports: Record, require: (id: number) => unknown) { exports.answer = 42; } }, function(this: void, require: (id: number) => unknown) { require(7); }]);", + ].join("\n"), + "interop.cts": [ + "function identity(this: void, value: T) { return value; }", + 'const electron = identity(require("electron"));', + 'electron.ipcRenderer.send("typed-identity");', + ].join("\n"), + "ipc-wrapper.cts": [ + 'const { ipcRenderer } = require("electron");', + "function send(this: void, channel: string) { ipcRenderer.send(channel); }", + "function sendRest(this: void, ...channels: string[]) { ipcRenderer.send(...channels); }", + 'send("typed-channel");', + 'sendRest("typed-rest");', + ].join("\n"), + "entry.mts": [ + "function first(this: void, value: number) { return value; }", + "function rest(this: void, ...values: number[]) { return values; }", + "function destructured({ left }: { left: number }, right: number) { return left + right; }", + "function mixed(this: void, head: number, ...tail: number[]) { return [head, ...tail]; }", + "const a = first(7);", + "const b = rest(1, 2);", + "const c = destructured({ left: 3 }, 4);", + "const d = mixed(9, 10, 11);", + "console.log(JSON.stringify([a, b, c, d, first.length, rest.length, mixed.length]));", + ].join("\n"), + }); + const control = await processes.run( + process.execPath, + [join(root, "entry.mts")], + { timeoutMs: 15_000 }, + ); + expect(control.exitCode, control.stderr).toBe(0); + expect(JSON.parse(control.stdout)).toEqual([ + 7, + [1, 2], + 7, + [9, 10, 11], + 1, + 0, + 1, + ]); + const analysis = await analyzeJavaScriptDirectory(surface, cli, root); + const graph = analysis.semantic_graph; + const parameters = graph.nodes.filter( + (node) => + node.kind === "parameter" && node.identity.module_path === "entry.mts", + ); + expect(parameters.map((node) => node.label).sort()).toEqual([ + "head", + "left", + "right", + "tail", + "value", + "values", + ]); + const parameterIds = new Set(parameters.map((node) => node.node_id)); + const flows = graph.relations.filter( + (relation) => + relation.relation === "argument-to-parameter" && + parameterIds.has(relation.target_node_id), + ); + expect( + flows + .map((flow) => { + const source = graph.nodes.find( + (node) => node.node_id === flow.source_node_id, + ); + const target = parameters.find( + (node) => node.node_id === flow.target_node_id, + ); + const owner = graph.nodes.find( + (node) => node.node_id === target?.function_node_id, + ); + return [ + source?.identity.source_range?.start.line, + source?.label, + owner?.label, + target?.label, + ]; + }) + .sort(), + ).toEqual([ + [5, "argument 0", "first", "value"], + [6, "argument 0", "rest", "values"], + [6, "argument 1", "rest", "values"], + [7, "argument 0", "destructured", "left"], + [7, "argument 1", "destructured", "right"], + [8, "argument 0", "mixed", "head"], + [8, "argument 1", "mixed", "tail"], + [8, "argument 2", "mixed", "tail"], + ]); + expect( + analysis.graph.nodes + .flatMap((node) => node.observations) + .some( + (observation) => + Array.isArray(observation.properties.exports) && + observation.properties.exports.includes("answer"), + ), + ).toBe(true); + const channels = analysis.graph.nodes + .filter((node) => node.kind === "ipc-channel") + .flatMap((node) => + node.observations.map((observation) => observation.properties.channel), + ); + expect(channels).toEqual( + expect.arrayContaining(["typed-channel", "typed-rest", "typed-identity"]), + ); + for (const [name, arity] of [ + ["first", 1], + ["rest", 1], + ["destructured", 2], + ["mixed", 2], + ] as const) { + const callable = graph.nodes.find( + (node) => node.kind === "function" && node.label === name, + ); + expect(callable, name).toBeDefined(); + const fingerprint = graph.fingerprints.find( + (value) => value.function_node_id === callable?.node_id, + ); + expect(fingerprint?.components.parameter_arity, name).toBe(arity); + } + }, +); diff --git a/tests/acceptance/artifactComparisonPlist.test.ts b/tests/acceptance/artifactComparisonPlist.test.ts index 3781eb6c71..a6cb31bd25 100644 --- a/tests/acceptance/artifactComparisonPlist.test.ts +++ b/tests/acceptance/artifactComparisonPlist.test.ts @@ -34,7 +34,7 @@ it("compares identical complete plist inventories across CLI and MCP and creates if (cliInventory === undefined) throw new Error("CLI inspection did not retain inventory Evidence"); expect(cliInventory.normalized_result).toMatchObject({ - coverage_status: "complete", + coverage: { scope: "root", status: "complete" }, limitations: ["Artifact has no child container entries."], }); @@ -50,7 +50,7 @@ it("compares identical complete plist inventories across CLI and MCP and creates if (mcpInventory === undefined) throw new Error("MCP inspection did not retain inventory Evidence"); expect(mcpInventory.normalized_result).toMatchObject({ - coverage_status: "complete", + coverage: { scope: "root", status: "complete" }, limitations: ["Artifact has no child container entries."], }); diff --git a/tests/acceptance/artifactInventoryJvmClass.test.ts b/tests/acceptance/artifactInventoryJvmClass.test.ts index 408888ce71..51f3e01ebf 100644 --- a/tests/acceptance/artifactInventoryJvmClass.test.ts +++ b/tests/acceptance/artifactInventoryJvmClass.test.ts @@ -107,8 +107,8 @@ it("keeps valid JVM classes from being classified as Mach-O while identifying FA }, { logical_path: "pkg/Big.class", - artifact_kind: "unknown", - artifact_format: "unknown", + artifact_kind: "resource", + artifact_format: "file", }, ]); const jarClassIds = new Set( @@ -121,7 +121,7 @@ it("keeps valid JVM classes from being classified as Mach-O while identifying FA .filter(({ artifact_id }) => jarClassIds.has(artifact_id)) .map(({ format }) => format) .sort(), - ).toEqual(["file", "unknown"]); + ).toEqual(["file", "file"]); const mixedPath = join(directory, "mixed.zip"); await writeFile(mixedPath, await archive([...javaEntries, ...fatControls])); @@ -135,11 +135,10 @@ it("keeps valid JVM classes from being classified as Mach-O while identifying FA ), ); for (const [name] of javaEntries) - expect(formatsByPath.get(name)).toEqual( - name === "Hello.class" - ? { artifact_kind: "resource", artifact_format: "file" } - : { artifact_kind: "unknown", artifact_format: "unknown" }, - ); + expect(formatsByPath.get(name)).toEqual({ + artifact_kind: "resource", + artifact_format: "file", + }); for (const [name] of fatControls) expect(formatsByPath.get(name)).toEqual({ artifact_kind: "executable", diff --git a/tests/acceptance/electronFusesCli.test.ts b/tests/acceptance/electronFusesCli.test.ts new file mode 100644 index 0000000000..e1e86f2767 --- /dev/null +++ b/tests/acceptance/electronFusesCli.test.ts @@ -0,0 +1,347 @@ +import { mkdir, realpath, symlink, writeFile } from "node:fs/promises"; +import { dirname, join, parse, sep } from "node:path"; +import { createHash } from "node:crypto"; +import { expect } from "vitest"; + +import { toolContract } from "../../src/contracts/toolContracts.js"; +import { + OBSIDIAN_FUSE_BYTES, + sentinelFreeMacho, + thinFuseImage, + universalFuseImage, +} from "../fixtures/electron/fuseImage.js"; +import { createTestTempDirectory } from "../fixtures/temporaryDirectory.js"; +import { cliTest } from "../support/cli/cliFixture.js"; + +// Windows resolves dot components before directory links in Win32 paths. +cliTest.skipIf(process.platform === "win32").for([ + { kind: "binary", mode: "absolute" }, + { kind: "binary", mode: "relative" }, + { kind: "bundle", mode: "absolute" }, + { kind: "bundle", mode: "relative" }, +] as const)( + "preserves $mode $kind selections containing a symlink before a parent segment", + async ({ kind, mode }, { cli }) => { + const root = await createTestTempDirectory("rea-electron-fuses-cli-path-"); + const framework = "Contents/Frameworks/Electron Framework.framework"; + const directory = + kind === "binary" + ? join(root, "Fixture.app", framework, "Versions") + : root; + const selectedTail = + kind === "binary" ? "A/Electron Framework" : "Fixture.app"; + const binaryTail = + kind === "binary" + ? selectedTail + : `Fixture.app/${framework}/Versions/A/Electron Framework`; + const binary = join(directory, "nested", binaryTail); + const sibling = join(directory, binaryTail); + const image = thinFuseImage([0x31, 0x30, 0x30, 0x31]); + const otherImage = thinFuseImage([0x30, 0x30, 0x30, 0x30]); + await mkdir(dirname(binary), { recursive: true }); + await mkdir(dirname(sibling), { recursive: true }); + await mkdir(join(directory, "nested", "B"), { recursive: true }); + await writeFile(binary, image.bytes); + await writeFile(sibling, otherImage.bytes); + await symlink("nested/B", join(directory, "Current")); + const selected = `${directory}/Current/../${selectedTail}`; + expect(await realpath(selected)).toBe( + kind === "binary" ? binary : join(root, "nested", "Fixture.app"), + ); + + const result = await cli.run({ + arguments: [ + "inspect-electron-fuses", + mode === "absolute" ? selected : selected.slice(root.length + 1), + "--json", + ], + cwd: root, + }); + expect(result.exitCode, result.stderr).toBe(0); + const evidence = toolContract("inspect_electron_fuses").outputSchema.parse( + result.json, + ); + const digest = createHash("sha256").update(image.bytes).digest("hex"); + expect(evidence.normalized_result.artifact).toMatchObject({ + selected_path: selected, + resolved_path: + kind === "binary" + ? binary + : `${selected}/${framework}/Versions/A/Electron Framework`, + sha256: digest, + }); + expect( + await realpath(evidence.normalized_result.artifact.resolved_path), + ).toBe(binary); + expect(evidence.subject?.local_path).toBe( + evidence.normalized_result.artifact.resolved_path, + ); + expect(evidence.subject?.digest.sha256).toBe(digest); + expect(evidence.normalized_result.runtime_implications).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + subject: "electron_run_as_node", + outcome: "honored", + }), + expect.objectContaining({ + subject: "node_cli_inspect_arguments", + outcome: "honored", + }), + ]), + ); + + const control = await cli.run({ + arguments: ["inspect-electron-fuses", sibling, "--json"], + }); + expect(control.exitCode, control.stderr).toBe(0); + const siblingEvidence = toolContract( + "inspect_electron_fuses", + ).outputSchema.parse(control.json); + expect(siblingEvidence.normalized_result.artifact.sha256).toBe( + createHash("sha256").update(otherImage.bytes).digest("hex"), + ); + expect(siblingEvidence.normalized_result.runtime_implications).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + subject: "electron_run_as_node", + outcome: "not_honored", + }), + expect.objectContaining({ + subject: "node_cli_inspect_arguments", + outcome: "not_honored", + }), + ]), + ); + }, +); + +cliTest.skipIf(process.platform !== "win32")( + "retains Windows drive, root, and mixed-separator path resolution", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-electron-fuses-win-path-"); + const tail = + "Fixture.app/Contents\\Frameworks/Electron Framework.framework/Versions\\A/Electron Framework"; + const binary = join(root, tail); + const image = thinFuseImage([0x31]); + await mkdir(dirname(binary), { recursive: true }); + await writeFile(binary, image.bytes); + const volume = parse(root).root; + for (const input of [ + binary.replaceAll("\\", "/"), + tail, + `${volume.slice(0, -1)}${tail}`, + binary.slice(volume.length - 1), + ]) { + const result = await cli.run({ + arguments: ["inspect-electron-fuses", input, "--json"], + cwd: root, + }); + expect(result.exitCode, result.stderr).toBe(0); + const evidence = toolContract( + "inspect_electron_fuses", + ).outputSchema.parse(result.json); + expect(evidence.normalized_result.artifact).toMatchObject({ + selected_path: binary, + resolved_path: binary, + sha256: createHash("sha256").update(image.bytes).digest("hex"), + }); + } + }, +); + +cliTest( + "reads thin and universal Electron fuse wires through the public CLI", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-electron-fuses-cli-"); + const thin = thinFuseImage(OBSIDIAN_FUSE_BYTES); + const universal = universalFuseImage(); + await writeFile(join(root, "thin"), thin.bytes); + await writeFile(join(root, "universal"), universal.bytes); + + const thinResult = await cli.run({ + arguments: ["inspect-electron-fuses", "thin", "--json"], + cwd: root, + }); + expect(thinResult.exitCode, thinResult.stderr).toBe(0); + const thinEvidence = toolContract( + "inspect_electron_fuses", + ).outputSchema.parse(thinResult.json); + expect(thinEvidence.subject?.local_path).toBe(join(root, "thin")); + expect(thinEvidence.subject?.digest.sha256).toBe( + createHash("sha256").update(thin.bytes).digest("hex"), + ); + expect(thinEvidence.subject?.architecture).toBe("arm64"); + expect( + thinEvidence.normalized_result.slices[0]?.sentinels[0], + ).toMatchObject({ + sentinel_offset: thin.sentinelOffset, + wire_version: 1, + declared_length: 8, + }); + expect( + thinEvidence.normalized_result.runtime_implications.find( + (item) => item.subject === "node_cli_inspect_arguments", + ), + ).toMatchObject({ outcome: "not_honored", fuse_index: 3 }); + + const universalResult = await cli.run({ + arguments: ["inspect-electron-fuses", join(root, "universal"), "--json"], + }); + expect(universalResult.exitCode, universalResult.stderr).toBe(0); + const universalEvidence = toolContract( + "inspect_electron_fuses", + ).outputSchema.parse(universalResult.json); + expect(universalEvidence.normalized_result.artifact.format).toBe( + "mach-o-universal", + ); + expect(universalEvidence.subject?.architecture).toBeNull(); + expect( + universalEvidence.normalized_result.slices.map((slice) => ({ + file_offset: slice.file_offset, + sentinel_offset: slice.sentinels[0]?.sentinel_offset, + })), + ).toEqual([ + { + file_offset: universal.left.fileOffset, + sentinel_offset: universal.left.sentinelOffset, + }, + { + file_offset: universal.right.fileOffset, + sentinel_offset: universal.right.sentinelOffset, + }, + ]); + }, +); + +cliTest( + "reports a missing sentinel as unknown fuse state through the public CLI", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-electron-fuses-cli-empty-"); + const path = join(root, "empty"); + await writeFile(path, sentinelFreeMacho()); + const result = await cli.run({ + arguments: ["inspect-electron-fuses", path, "--json"], + }); + expect(result.exitCode, result.stderr).toBe(0); + const evidence = toolContract("inspect_electron_fuses").outputSchema.parse( + result.json, + ); + expect(evidence.normalized_result.slices[0]?.sentinel_count).toBe(0); + expect(evidence.normalized_result.runtime_implications).toEqual([]); + expect(evidence.limitations.join(" ")).toContain( + "unknown, not that every fuse is disabled", + ); + }, +); + +cliTest( + "resolves an Electron app bundle to its framework binary through the public CLI", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-electron-fuses-cli-app-"); + const image = thinFuseImage(OBSIDIAN_FUSE_BYTES); + const app = join(root, "Fixture.app"); + const binary = join( + app, + "Contents", + "Frameworks", + "Electron Framework.framework", + "Versions", + "A", + "Electron Framework", + ); + await mkdir(join(binary, ".."), { recursive: true }); + await writeFile(binary, image.bytes); + await symlink("A", join(binary, "..", "..", "Current")); + for (const { input, cwd, selected } of [ + { input: app, cwd: root, selected: app }, + { input: `${app}/./`, cwd: root, selected: `${app}/./` }, + { input: "Fixture.app/.", cwd: root, selected: `${app}/.` }, + { input: ".", cwd: app, selected: `${app}${sep}.` }, + { + input: "..", + cwd: join(app, "Contents"), + selected: `${app}${sep}Contents${sep}..`, + }, + { + input: ".././..", + cwd: join(app, "Contents", "Frameworks"), + selected: `${app}${sep}Contents${sep}Frameworks${sep}.././..`, + }, + ]) { + const result = await cli.run({ + arguments: ["inspect-electron-fuses", input, "--json"], + cwd, + }); + expect(result.exitCode, result.stderr).toBe(0); + const evidence = toolContract( + "inspect_electron_fuses", + ).outputSchema.parse(result.json); + expect(evidence.normalized_result.artifact).toMatchObject({ + selected_path: process.platform === "win32" ? app : selected, + resolved_path: binary, + format: "mach-o", + }); + expect(evidence.subject?.local_path).toBe(binary); + } + + const empty = await cli.run({ + arguments: ["inspect-electron-fuses", "", "--json"], + cwd: app, + }); + expect(empty.exitCode, empty.stderr).toBe(1); + expect(empty.json).toMatchObject({ + code: "invalid_request", + details: { operation: "inspect_electron_fuses" }, + }); + }, +); + +cliTest.skipIf(process.platform === "win32")( + "refuses a terminal bundle parent segment after a directory symlink", + async ({ cli }) => { + const root = await createTestTempDirectory( + "rea-electron-fuses-cli-parent-", + ); + const app = join(root, "Fixture.app"); + const binary = join( + app, + "Contents", + "Frameworks", + "Electron Framework.framework", + "Electron Framework", + ); + await mkdir(dirname(binary), { recursive: true }); + await writeFile(binary, thinFuseImage([0x31]).bytes); + await symlink("Contents", join(app, "linked")); + const result = await cli.run({ + arguments: ["inspect-electron-fuses", `${app}/linked//..`, "--json"], + }); + expect(result.exitCode, result.stderr).toBe(1); + expect(result.json).toMatchObject({ + code: "invalid_request", + details: { + operation: "inspect_electron_fuses", + issues: [ + expect.objectContaining({ path: ["path"], reason: "invalid_value" }), + ], + }, + }); + }, +); + +cliTest( + "rejects an unrecognized container through the public CLI", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-electron-fuses-cli-text-"); + const path = join(root, "notes.txt"); + await writeFile(path, "This is not an executable container."); + const result = await cli.run({ + arguments: ["inspect-electron-fuses", path, "--json"], + }); + expect(result.exitCode).toBe(1); + expect(result.json).toMatchObject({ + code: "unsupported_target", + details: { operation: "inspect_electron_fuses", path }, + }); + }, +); diff --git a/tests/acceptance/goBinaryCli.test.ts b/tests/acceptance/goBinaryCli.test.ts index fd6cd648e8..22cb9fed0c 100644 --- a/tests/acceptance/goBinaryCli.test.ts +++ b/tests/acceptance/goBinaryCli.test.ts @@ -7,11 +7,74 @@ import { createGoBinaryByteStringFixture, createGoBinaryDiagnosticFixtures, createGoBinaryFixture, + createGoSplitElfFixture, + createGoOverlappingElfFixture, createGoBinaryPathFailureFixture, GO_MODULE_TEXT, } from "../fixtures/go/image.js"; import { createTestTempDirectory } from "../fixtures/temporaryDirectory.js"; import { cliTest } from "../support/cli/cliFixture.js"; +import { connectLocalToolsMcp } from "../fixtures/localToolsMcp.js"; +import { parseMcpToolError } from "../fixtures/mcpToolError.js"; + +cliTest( + "bounds overlapping ELF scans through CLI and keeps MCP responsive", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-go-overlap-"); + const path = join(root, "overlapping.elf"); + await writeFile(path, createGoOverlappingElfFixture()); + const result = await cli.run({ + arguments: ["inspect-go-binary", path, "--json"], + }); + expect(result.exitCode).toBe(1); + const expected = { + code: "resource_constraint", + details: { + operation: "inspect_go_binary", + resource: "cpu", + reported_limits: { + boundary: "search", + maximum_bytes: 256 * 1024 * 1024, + }, + }, + }; + expect(result.json).toMatchObject(expected); + const { client, call } = await connectLocalToolsMcp(); + const response = await call("inspect_go_binary", { path }); + expect(parseMcpToolError(response).error).toMatchObject(expected); + await client.ping(); + }, + 30000, +); + +cliTest( + "retains later-load sectionless ELF evidence through public CLI and stdio MCP", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-go-split-elf-"); + const { client, call } = await connectLocalToolsMcp(); + const infos = []; + for (const sectionless of [false, true]) { + const fixture = createGoSplitElfFixture({ sectionless }); + const path = join(root, sectionless ? "sectionless.elf" : "original.elf"); + await writeFile(path, fixture.bytes); + const result = await cli.run({ + arguments: ["inspect-go-binary", path, "--json"], + }); + expect(result.exitCode, result.stderr).toBe(0); + const evidence = toolContract("inspect_go_binary").outputSchema.parse( + result.json, + ); + expect(evidence.normalized_result.build_info).not.toBeNull(); + infos.push(evidence.normalized_result.build_info); + const response = await call("inspect_go_binary", { path }); + expect(response.isError, JSON.stringify(response)).not.toBe(true); + expect(response.structuredContent).toEqual(evidence); + } + expect(infos[1]).toEqual(infos[0]); + await client.ping(); + }, + 30000, +); for (const expectedErrno of ["ELOOP", "ENAMETOOLONG"] as const) { cliTest.skipIf(expectedErrno === "ELOOP" && process.platform === "win32")( diff --git a/tests/acceptance/providers/hopperStartupRecovery.test.ts b/tests/acceptance/providers/hopperStartupRecovery.test.ts index 7fb039d4d9..2cf22c9b90 100644 --- a/tests/acceptance/providers/hopperStartupRecovery.test.ts +++ b/tests/acceptance/providers/hopperStartupRecovery.test.ts @@ -10,7 +10,7 @@ import { z } from "zod"; import { parseMcpToolError } from "../../fixtures/mcpToolError.js"; import { workspaceCliTest } from "../../support/cli/workspaceCliFixture.js"; -import { acquireHopperTargetLease } from "../../../src/hopper/HopperTargetLease.js"; +import { acquireHopperTargetLease } from "../../../src/process/HopperLease.js"; const statusSchema = z.object({ result: z.object({ open: z.boolean() }) }); const test = workspaceCliTest.skipIf(process.platform !== "darwin"); @@ -50,7 +50,42 @@ test( }); const mcp = new Client({ name: "hopper-literal-boundary", version: "1" }); let runtime: string | undefined; - onTestFinished(async () => { + try { + await mcp.connect(transport); + const call = (name: string, arguments_: Record = {}) => + mcp.callTool({ name, arguments: arguments_ }); + expect((await call("open_binary", { path: target })).isError).not.toBe( + true, + ); + const first = parseMcpToolError(await call("current_document")); + runtime = dirname(await readFile(receipt, "utf8")); + expect(first.error).toMatchObject({ + code: "cleanup_incomplete", + details: { + diagnostics: { + primary_error: { + code: "provider_unavailable", + details: { + launcher: { + exit_code: 2, + stderr: { + text: expect.stringContaining( + "Not authorized to send Apple events", + ), + }, + }, + }, + }, + }, + }, + }); + await access(`${receipt}.document-created`); + await access(runtime); + expect(parseMcpToolError(await call("list_procedures"))).toEqual(first); + expect(parseMcpToolError(await call("close_binary")).error.code).toBe( + "cleanup_incomplete", + ); + } finally { // SDK close can return immediately after SIGKILL. Wait for its actual // child-close event before recovering this process's retained lease. let serverExited: Promise | undefined; @@ -77,41 +112,7 @@ test( }); if (!acquired.acquired) throw new Error("Fixture lease remained live"); await acquired.lease.release(); - }); - await mcp.connect(transport); - const call = (name: string, arguments_: Record = {}) => - mcp.callTool({ name, arguments: arguments_ }); - expect((await call("open_binary", { path: target })).isError).not.toBe( - true, - ); - const first = parseMcpToolError(await call("current_document")); - runtime = dirname(await readFile(receipt, "utf8")); - expect(first.error).toMatchObject({ - code: "cleanup_incomplete", - details: { - diagnostics: { - primary_error: { - code: "provider_unavailable", - details: { - launcher: { - exit_code: 2, - stderr: { - text: expect.stringContaining( - "Not authorized to send Apple events", - ), - }, - }, - }, - }, - }, - }, - }); - await access(`${receipt}.document-created`); - await access(runtime); - expect(parseMcpToolError(await call("list_procedures"))).toEqual(first); - expect(parseMcpToolError(await call("close_binary")).error.code).toBe( - "cleanup_incomplete", - ); + } } }, ); @@ -201,13 +202,13 @@ test("recovers a denied Hopper launch through the same stdio MCP session and CLI ).toBe(false); }); -test( - "preserves a timed-out handoff and closes only a late bridge with matching run identity", +test.for(["handoff", "not-found"] as const)( + "preserves an unconfirmed %s handoff and closes only a late bridge with matching run identity", { timeout: 60_000 }, - async ({ workspace }) => { + async (launchMode, { workspace }) => { const target = workspace.path("target"); await copyFile("/usr/bin/true", target); - const mode = await workspace.write("mode", "handoff"); + const mode = await workspace.write("mode", launchMode); const receipt = workspace.path("bootstrap-path"); const transport = new StdioClientTransport({ command: process.execPath, @@ -257,17 +258,35 @@ test( details: { diagnostics: { primary_error: { - code: "provider_timeout", - details: { timeout_ms: 2000, launcher: { exit_code: 0 } }, + code: + launchMode === "handoff" + ? "provider_timeout" + : "provider_unavailable", + details: + launchMode === "handoff" + ? { timeout_ms: 2000, launcher: { exit_code: 0 } } + : { stage: "launch", launcher: { exit_code: 3 } }, }, }, }, }); + if (launchMode === "not-found") { + await access(`${receipt}.document-created`); + expect(initial.error.details).toMatchObject({ + diagnostics: { + primary_error: { message: expect.stringContaining("Spotlight") }, + }, + }); + } expect(parseMcpToolError(await call("list_procedures"))).toEqual(initial); const status = z.object({ result: z.object({ provider_operation_health: z.object({ - state: z.literal("not_started"), + // A timeout observes incomplete startup; a failed launcher without + // confirmed ownership leaves the provider's state unknown. + state: z.literal( + launchMode === "handoff" ? "not_started" : "unknown", + ), stage: z.literal("launch"), }), }), @@ -289,7 +308,7 @@ test( const token = setting("REA_TOKEN"); const runId = setting("REA_RUN_ID"); runtime = dirname(socket); - const lateBridge = async (identity: string, credential = token) => { + const lateBridge = (identity: string, credential = token) => { const child = spawn( process.execPath, [ @@ -301,14 +320,14 @@ test( { stdio: "ignore" }, ); children.push(child); - await vi.waitFor(() => access(socket)); return child; }; for (const [identity, credential] of [ ["different-run", token], [runId, "different-token"], ] as const) { - const wrong = await lateBridge(identity, credential); + const wrong = lateBridge(identity, credential); + await vi.waitFor(() => access(socket)); expect(parseMcpToolError(await call("close_binary")).error.code).toBe( "cleanup_incomplete", ); @@ -319,7 +338,9 @@ test( } const closing = call("close_binary"); await delay(150); - const owned = await lateBridge(runId); + // A verified close may remove the socket before the fixture can poll it. + // The successful close and child exit establish the owned bridge outcome. + const owned = lateBridge(runId); expect((await closing).isError).not.toBe(true); await vi.waitFor(() => expect(owned.exitCode).toBe(0)); await expect(access(runtime)).rejects.toMatchObject({ code: "ENOENT" }); diff --git a/tests/acceptance/rustBinaryCli.test.ts b/tests/acceptance/rustBinaryCli.test.ts index 2f984c618d..d1d86c3114 100644 --- a/tests/acceptance/rustBinaryCli.test.ts +++ b/tests/acceptance/rustBinaryCli.test.ts @@ -3,7 +3,10 @@ import { writeFile } from "node:fs/promises"; import { join } from "node:path"; import { expect } from "vitest"; import { toolContract } from "../../src/contracts/toolContracts.js"; -import { createRustImage } from "../fixtures/rust/image.js"; +import { + createRustImage, + rustDependencyPayload, +} from "../fixtures/rust/image.js"; import { createTestTempDirectory } from "../fixtures/temporaryDirectory.js"; import { cliTest } from "../support/cli/cliFixture.js"; @@ -49,6 +52,37 @@ cliTest("returns null metadata when .dep-v0 is absent", async ({ cli }) => { ); }); +cliTest( + "preserves an unknown cargo-auditable revision and reports its fallback", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-rust-cli-unknown-format-"); + const path = join(root, "sample"); + const payload = rustDependencyPayload({ + format: 4, + packages: [ + { name: "sample", version: "0.1.0", source: "local", root: true }, + ], + }); + await writeFile(path, createRustImage({ format: "elf", payload })); + const result = await cli.run({ + arguments: ["inspect-rust-binary", path, "--json"], + }); + expect(result.exitCode, result.stderr).toBe(0); + const evidence = toolContract("inspect_rust_binary").outputSchema.parse( + result.json, + ); + expect(evidence.normalized_result.auditable_metadata?.format_version).toBe( + 4, + ); + expect(evidence.normalized_result.limitations.join("\n")).toContain( + "highest known preceding revision (1)", + ); + expect(evidence.normalized_result.limitations.join("\n")).toContain( + "this specific revision has not been independently verified", + ); + }, +); + cliTest("reports malformed zlib as invalid input", async ({ cli }) => { const root = await createTestTempDirectory("rea-rust-cli-zlib-"); const path = join(root, "broken.elf"); diff --git a/tests/acceptance/setup/geminiWorkspaceTrust.test.ts b/tests/acceptance/setup/geminiWorkspaceTrust.test.ts new file mode 100644 index 0000000000..a1d19c0125 --- /dev/null +++ b/tests/acceptance/setup/geminiWorkspaceTrust.test.ts @@ -0,0 +1,220 @@ +import { access, mkdir, readFile, writeFile } from "node:fs/promises"; +import { dirname, join } from "node:path"; +import { expect } from "vitest"; + +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { cliTest, type TestCli } from "../../support/cli/cliFixture.js"; + +const setupArguments = ["setup", "--client", "gemini_cli", "--yes", "--json"]; +const doctorArguments = ["doctor", "--client", "gemini_cli", "--json"]; + +const geminiWorkspace = async (cli: TestCli) => { + const home = await createTestTempDirectory("rea-gemini-workspace-trust-"); + const project = join(home, "project"); + const workspace = join(project, ".gemini/settings.json"); + const user = join(home, ".gemini/settings.json"); + await mkdir(dirname(workspace), { recursive: true }); + const environment: NodeJS.ProcessEnv = { + HOME: home, + USERPROFILE: home, + GEMINI_CLI_HOME: home, + GEMINI_CLI_SYSTEM_SETTINGS_PATH: join(home, "system.json"), + GEMINI_CLI_SYSTEM_DEFAULTS_PATH: join(home, "defaults.json"), + GEMINI_CLI_TRUSTED_FOLDERS_PATH: join(home, "trust.json"), + }; + const initial = await cli.run({ + arguments: setupArguments, + cwd: home, + environment, + }); + expect(initial.exitCode, initial.stdout).toBe(0); + const originalUser = await readFile(user); + const run = (arguments_: readonly string[]) => + cli.run({ arguments: arguments_, cwd: project, environment }); + return { workspace, user, originalUser, environment, run }; +}; + +cliTest( + "Gemini does not require workspace trust until workspace settings exist", + async ({ cli }) => { + const fixture = await geminiWorkspace(cli); + const trust = join(dirname(fixture.user), "trustedFolders.json"); + fixture.environment.GEMINI_CLI_TRUSTED_FOLDERS_PATH = trust; + for (const text of ["{}", "{ not json"]) { + await writeFile(trust, text); + const setup = await fixture.run(setupArguments); + expect(setup.exitCode, setup.stdout).toBe(0); + expect(setup.json).toMatchObject({ status: "ready", appliedActions: [] }); + const doctor = await fixture.run(doctorArguments); + expect(doctor.exitCode, doctor.stdout).toBe(0); + expect(doctor.json).toMatchObject({ + healthy: true, + identity: { + registrations: expect.arrayContaining([ + expect.objectContaining({ + client: "gemini_cli", + state: "aligned", + remediation: null, + }), + ]), + }, + }); + await expect(access(fixture.workspace)).rejects.toMatchObject({ + code: "ENOENT", + }); + expect(await readFile(trust, "utf8")).toBe(text); + expect(await readFile(fixture.user)).toEqual(fixture.originalUser); + } + + const workspaceText = "{ not json"; + await writeFile(fixture.workspace, workspaceText); + const setup = await fixture.run(setupArguments); + expect(setup.exitCode, setup.stdout).toBe(1); + expect(setup.json).toMatchObject({ + status: "needs_human", + plannedActions: [], + appliedActions: [], + remediation: expect.stringContaining(trust), + }); + const doctor = await fixture.run(doctorArguments); + expect(doctor.exitCode, doctor.stdout).toBe(1); + expect(doctor.json).toMatchObject({ + healthy: false, + identity: { + registrations: expect.arrayContaining([ + expect.objectContaining({ + client: "gemini_cli", + state: "invalid", + remediation: expect.stringContaining(trust), + }), + ]), + }, + }); + expect(await readFile(fixture.workspace, "utf8")).toBe(workspaceText); + expect(await readFile(trust, "utf8")).toBe("{ not json"); + expect(await readFile(fixture.user)).toEqual(fixture.originalUser); + }, +); + +for (const scenario of [ + { name: "an unlisted workspace", environment: {} }, + { + name: "an explicitly untrusted workspace", + environment: { GEMINI_CLI_TRUST_WORKSPACE: "false" }, + }, + { + name: "a restricted workspace with an explicit trust opt-in", + environment: { + GEMINI_CLI_TRUST_WORKSPACE: "true", + GEMINI_RESTRICTED_MODE: "true", + }, + }, +]) + cliTest( + `Gemini setup and doctor ignore malformed settings in ${scenario.name}`, + async ({ cli }) => { + const fixture = await geminiWorkspace(cli); + Object.assign(fixture.environment, scenario.environment); + const malformed = Buffer.from("{ not json"); + await writeFile(fixture.workspace, malformed); + + const setup = await fixture.run(setupArguments); + expect(setup.exitCode, setup.stdout).toBe(0); + expect(setup.json).toMatchObject({ status: "ready", appliedActions: [] }); + const doctor = await fixture.run(doctorArguments); + expect(doctor.exitCode, doctor.stdout).toBe(0); + expect(doctor.json).toMatchObject({ + healthy: true, + identity: { + registrations: expect.arrayContaining([ + expect.objectContaining({ + client: "gemini_cli", + state: "aligned", + remediation: null, + }), + ]), + }, + }); + expect(await readFile(fixture.workspace)).toEqual(malformed); + expect(await readFile(fixture.user)).toEqual(fixture.originalUser); + await expect( + access(`${fixture.workspace}.rea.backup`), + ).rejects.toMatchObject({ + code: "ENOENT", + }); + + if (scenario.name === "an explicitly untrusted workspace") { + const excluded = Buffer.from( + JSON.stringify({ mcp: { excluded: ["rea"] } }), + ); + await writeFile(fixture.workspace, excluded); + const excludedSetup = await fixture.run(setupArguments); + expect(excludedSetup.exitCode, excludedSetup.stdout).toBe(0); + const excludedDoctor = await fixture.run(doctorArguments); + expect(excludedDoctor.exitCode, excludedDoctor.stdout).toBe(0); + expect(excludedDoctor.json).toMatchObject({ + identity: { + registrations: expect.arrayContaining([ + expect.objectContaining({ + client: "gemini_cli", + state: "aligned", + }), + ]), + }, + }); + expect(await readFile(fixture.workspace)).toEqual(excluded); + } + }, + ); + +cliTest( + "Gemini still rejects malformed settings and exclusions in a trusted workspace", + async ({ cli }) => { + const fixture = await geminiWorkspace(cli); + fixture.environment.GEMINI_CLI_TRUST_WORKSPACE = "true"; + for (const scenario of [ + { + text: "{ not json", + state: "invalid", + diagnostic: "Invalid JSON/JSONC", + }, + { + text: JSON.stringify({ mcp: { excluded: ["rea"] } }), + state: "stale", + diagnostic: "mcp.excluded", + }, + ]) { + const original = Buffer.from(scenario.text); + await writeFile(fixture.workspace, original); + const setup = await fixture.run(setupArguments); + expect(setup.exitCode, setup.stdout).toBe(1); + expect(setup.json).toMatchObject({ + status: "needs_human", + plannedActions: [], + appliedActions: [], + remediation: expect.stringContaining(scenario.diagnostic), + }); + const doctor = await fixture.run(doctorArguments); + expect(doctor.exitCode, doctor.stdout).toBe(1); + expect(doctor.json).toMatchObject({ + healthy: false, + identity: { + registrations: expect.arrayContaining([ + expect.objectContaining({ + client: "gemini_cli", + state: scenario.state, + remediation: expect.stringContaining(scenario.diagnostic), + }), + ]), + }, + }); + expect(await readFile(fixture.workspace)).toEqual(original); + expect(await readFile(fixture.user)).toEqual(fixture.originalUser); + await expect( + access(`${fixture.workspace}.rea.backup`), + ).rejects.toMatchObject({ + code: "ENOENT", + }); + } + }, +); diff --git a/tests/acceptance/setup/hermesClientSetup.test.ts b/tests/acceptance/setup/hermesClientSetup.test.ts index 5a7baa3334..3b2a146a0b 100644 --- a/tests/acceptance/setup/hermesClientSetup.test.ts +++ b/tests/acceptance/setup/hermesClientSetup.test.ts @@ -6,6 +6,102 @@ import { expect } from "vitest"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; import { cliTest } from "../../support/cli/cliFixture.js"; +for (const [version, keyStyle] of [ + ["1.1", "quoted"], + ["1.2", "quoted"], + ["1.2", "plain"], +] as const) { + cliTest( + `Hermes setup preserves string environment values under YAML ${version} with ${keyStyle} keys`, + async ({ cli }) => { + const home = await createTestTempDirectory("rea-hermes-yaml-strings-"); + const profile = join(home, "hermes"); + const config = join(profile, "config.yaml"); + const environment = { + REA_MCP_OUTPUT_SCHEMA_PROFILE: "off", + ON: "on", + YES: "yes", + NO: "no", + SHORT_TRUE: "y", + SHORT_FALSE: "n", + DATE: "2026-10-14", + TIME: "12:34:56", + FLOAT: "1.0e-7", + OCTAL: "0600", + NULL_WORD: "NULL", + NULL_MARKER: "~", + }; + const directive = version === "1.2" ? "%YAML 1.2\n---\n" : ""; + const settings = + directive + + "# User settings\nnotifications: on\ndate: 2026-10-14\n" + + "timestamp: 2026-10-14T00:00:00.123456+08:00\n"; + const original = + settings + + "mcp_servers:\n rea:\n command: rea-old\n args: [mcp]\n env:\n" + + Object.entries(environment) + .map( + ([key, value]) => + ` ${keyStyle === "plain" ? key : JSON.stringify(key)}: ${JSON.stringify(value)}\n`, + ) + .join(""); + await mkdir(profile); + await writeFile(config, original); + const run = (arguments_: readonly string[]) => + cli.run({ + arguments: arguments_, + cwd: home, + environment: { USERPROFILE: home, HERMES_HOME: profile }, + timeoutMs: 20_000, + }); + const setup = [ + "setup", + "--client", + "hermes", + "--skill=false", + "--yes", + "--json", + ]; + expect((await run(setup)).json).toMatchObject({ status: "ready" }); + const configured = await readFile(config, "utf8"); + expect(parse(configured, { version })).toMatchObject({ + mcp_servers: { rea: { env: environment, enabled: true } }, + }); + // Hermes's C loader applies YAML 1.1 even with the explicit directive. + // Independently model that schema; real Hermes loader verification is + // recorded in the PR rather than requiring Python dependencies in CI. + if (keyStyle === "plain") { + // The real C loader rejects these YAML 1.2 string keys as duplicate + // boolean keys. Preserve that fallback rather than changing how + // Hermes interprets unrelated user settings. + for (const text of [original, configured]) + expect(() => + parse(text.slice(directive.length), { version: "1.1" }), + ).toThrow("Map keys must be unique"); + } else { + expect( + parse(configured.slice(directive.length), { version: "1.1" }), + ).toMatchObject({ + mcp_servers: { rea: { env: environment, enabled: true } }, + }); + } + expect(configured.startsWith(settings)).toBe(true); + expect(await readFile(`${config}.rea.backup`, "utf8")).toBe(original); + expect( + (await run(["doctor", "--client", "hermes", "--json"])).json, + ).toMatchObject({ + healthy: true, + identity: { registrations: [{ client: "hermes", state: "aligned" }] }, + }); + expect((await run(setup)).json).toMatchObject({ + status: "ready", + appliedActions: [], + }); + expect(await readFile(config, "utf8")).toBe(configured); + }, + ); +} + cliTest( "Hermes setup, doctor and uninstall use the expanded profile and its personal skill", async ({ cli }) => { diff --git a/tests/acceptance/setup/piClientSetup.test.ts b/tests/acceptance/setup/piClientSetup.test.ts index 736e6a5088..afc24ae7c7 100644 --- a/tests/acceptance/setup/piClientSetup.test.ts +++ b/tests/acceptance/setup/piClientSetup.test.ts @@ -11,9 +11,14 @@ import { COMPACT_INPUT_SCHEMA_BUDGET_BYTES } from "../../../src/config/mcpInputS import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; import { cliTest } from "../../support/cli/cliFixture.js"; -for (const profile of ["", "bogus"]) +for (const [setting, profile] of [ + ["REA_MCP_INPUT_SCHEMA_PROFILE", ""], + ["REA_MCP_INPUT_SCHEMA_PROFILE", "bogus"], + ["REA_MCP_OUTPUT_SCHEMA_PROFILE", ""], + ["REA_MCP_OUTPUT_SCHEMA_PROFILE", "bogus"], +] as const) cliTest( - `setup and doctor refuse an invalid retained schema profile ${JSON.stringify(profile)}`, + `setup and doctor refuse an invalid retained ${setting} value ${JSON.stringify(profile)}`, async ({ cli }) => { const home = await createTestTempDirectory("rea-pi-invalid-profile-cli-"); const agent = join(home, ".pi", "agent"); @@ -24,7 +29,7 @@ for (const profile of ["", "bogus"]) type: "stdio", command: "rea", args: ["mcp"], - env: { REA_MCP_INPUT_SCHEMA_PROFILE: profile }, + env: { [setting]: profile }, }, }, }); @@ -52,7 +57,7 @@ for (const profile of ["", "bogus"]) status: "needs_human", plannedActions: [], appliedActions: [], - remediation: expect.stringContaining("REA_MCP_INPUT_SCHEMA_PROFILE"), + remediation: expect.stringContaining(setting), }); const doctor = await cli.run({ arguments: ["doctor", "--client", "pi", "--json"], @@ -83,6 +88,7 @@ cliTest( const config = join(agent, "mcp.json"); const custom = { REA_MCP_INPUT_SCHEMA_PROFILE: "compact", + REA_MCP_OUTPUT_SCHEMA_PROFILE: "off", REA_LOG_LEVEL: "silent", }; const original = JSON.stringify({ diff --git a/tests/acceptance/setup/registrationRuntimeEnvironmentCli.test.ts b/tests/acceptance/setup/registrationRuntimeEnvironmentCli.test.ts new file mode 100644 index 0000000000..e0597f1fff --- /dev/null +++ b/tests/acceptance/setup/registrationRuntimeEnvironmentCli.test.ts @@ -0,0 +1,514 @@ +import { access, mkdir, readFile, writeFile } from "node:fs/promises"; +import { dirname, join, resolve } from "node:path"; +import { expect } from "vitest"; +import { stringify } from "yaml"; + +import { + effectiveClientServer, + parseClientConfiguration, + clientRegistrationEntry, +} from "../../../src/application/ClientConfigurationDocument.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { cliTest } from "../../support/cli/cliFixture.js"; + +const scenarios = [ + { + client: "cursor", + format: "json", + path: ".cursor/mcp.json", + setting: "HOPPER_LOADER_ARGS_JSON", + invalid: { HOPPER_LOADER_ARGS_JSON: "{" }, + repaired: { HOPPER_LOADER_ARGS_JSON: '["", "--loader", "Mach-O"]' }, + }, + { + client: "dsh", + format: "dsh", + path: ".dsh/cordis.patch.yml", + setting: "REA_GHIDRA_COMPILER_SPEC_ID", + invalid: { REA_GHIDRA_COMPILER_SPEC_ID: "borlandcpp" }, + repaired: { + REA_GHIDRA_COMPILER_SPEC_ID: "borlandcpp", + REA_GHIDRA_LANGUAGE_ID: "x86:LE:32:default", + }, + }, +] as const; + +for (const scenario of scenarios) + cliTest( + `setup and doctor reject ${scenario.client} overrides rejected by MCP startup`, + async ({ cli }) => { + const home = await createTestTempDirectory("rea-registration-runtime-"); + const config = join(home, scenario.path); + await mkdir(dirname(config), { recursive: true }); + const environment = { + HOME: home, + USERPROFILE: home, + XDG_CONFIG_HOME: join(home, ".config"), + DSH_HOME: join(home, ".dsh"), + }; + const registration = (env: Readonly>) => { + const server = { + command: "stale-launcher", + args: ["mcp"], + env: { CALLER_SETTING: "keep exactly", ...env }, + }; + return scenario.client === "dsh" + ? stringify([ + { + insert: [ + { + id: "rea-mcp", + name: "@deepseek-ai/dsh-mcp-client", + config: { + serverName: "rea", + transport: "stdio", + ...server, + }, + }, + ], + }, + ]) + : JSON.stringify({ mcpServers: { rea: server } }); + }; + const original = registration(scenario.invalid); + await writeFile(config, original); + const startup = await cli.run({ + arguments: ["mcp"], + cwd: home, + environment: { ...environment, ...scenario.invalid }, + }); + expect(startup.exitCode).toBe(1); + expect(startup.stderr).toContain(scenario.setting); + expect(startup.stdout).toBe(""); + const run = (args: readonly string[]) => + cli.run({ arguments: [...args, "--json"], cwd: home, environment }); + const setup = await run([ + "setup", + "--client", + scenario.client, + "--skill=false", + "--yes", + ]); + expect(setup.exitCode).toBe(1); + expect(setup.json).toMatchObject({ + status: "needs_human", + plannedActions: [], + appliedActions: [], + remediation: expect.stringContaining(scenario.setting), + }); + const doctor = await run(["doctor", "--client", scenario.client]); + expect(doctor.exitCode).toBe(1); + expect(doctor.json).toMatchObject({ + identity: { + registrations: expect.arrayContaining([ + expect.objectContaining({ + client: scenario.client, + state: "invalid", + remediation: expect.stringContaining(scenario.setting), + }), + ]), + }, + }); + expect(await readFile(config, "utf8")).toBe(original); + await expect(access(`${config}.rea.backup`)).rejects.toMatchObject({ + code: "ENOENT", + }); + + // Model the caller repairing the named setting before rerunning setup. + const repaired = registration(scenario.repaired); + await writeFile(config, repaired); + const configured = await run([ + "setup", + "--client", + scenario.client, + "--skill=false", + "--yes", + ]); + expect(configured.exitCode, configured.stdout).toBe(0); + const text = await readFile(config, "utf8"); + const parsed = parseClientConfiguration(text, scenario.format); + expect(effectiveClientServer(parsed, "rea")).toMatchObject({ + env: { CALLER_SETTING: "keep exactly", ...scenario.repaired }, + }); + expect(await readFile(`${config}.rea.backup`, "utf8")).toBe(repaired); + const repeated = await run([ + "setup", + "--client", + scenario.client, + "--skill=false", + "--yes", + ]); + expect(repeated.json).toMatchObject({ + status: "ready", + appliedActions: [], + plannedActions: [], + }); + expect(await readFile(config, "utf8")).toBe(text); + }, + ); + +cliTest( + "OpenCode CLI validates dependent settings after merging JSON and JSONC", + async ({ cli }) => { + const home = await createTestTempDirectory("rea-layered-runtime-cli-"); + const directory = join(home, ".config/opencode"); + await mkdir(directory, { recursive: true }); + const lower = join(directory, "opencode.json"); + const upper = join(directory, "opencode.jsonc"); + const content = (env: Record) => + JSON.stringify({ + mcp: { + rea: clientRegistrationEntry( + "opencode", + [resolve("scripts/rea.mjs"), "mcp"], + env, + ), + }, + }); + const lowerContent = content({ + REA_GHIDRA_LANGUAGE_ID: "x86:LE:32:default", + }); + const upperContent = content({ REA_GHIDRA_COMPILER_SPEC_ID: "borlandcpp" }); + await writeFile(lower, lowerContent); + await writeFile(upper, upperContent); + const environment = { + HOME: home, + USERPROFILE: home, + XDG_CONFIG_HOME: join(home, ".config"), + // Select an absent launcher so host-installed Hopper cannot make this + // configuration merge test plan an unrelated provider-environment edit. + HOPPER_LAUNCHER_PATH: join(home, "missing-hopper"), + }; + const run = (args: readonly string[]) => + cli.run({ arguments: [...args, "--json"], cwd: home, environment }); + const setup = await run([ + "setup", + "--client", + "opencode", + "--skill=false", + "--yes", + ]); + expect(setup.exitCode, setup.stdout).toBe(0); + expect(setup.json).toMatchObject({ + status: "ready", + plannedActions: [], + appliedActions: [], + }); + const doctor = await run(["doctor", "--client", "opencode"]); + expect(doctor.exitCode, doctor.stdout).toBe(0); + expect(doctor.json).toMatchObject({ + identity: { + registrations: expect.arrayContaining([ + expect.objectContaining({ client: "opencode", state: "aligned" }), + ]), + }, + }); + expect(await readFile(lower, "utf8")).toBe(lowerContent); + expect(await readFile(upper, "utf8")).toBe(upperContent); + await writeFile(lower, content({ CALLER_SETTING: "retained" })); + const missing = await run([ + "setup", + "--client", + "opencode", + "--skill=false", + "--yes", + ]); + expect(missing.exitCode).toBe(1); + expect(missing.json).toMatchObject({ + status: "needs_human", + plannedActions: [], + appliedActions: [], + remediation: expect.stringContaining("REA_GHIDRA_COMPILER_SPEC_ID"), + }); + const invalidDoctor = await run(["doctor", "--client", "opencode"]); + expect(invalidDoctor.exitCode).toBe(1); + expect(invalidDoctor.json).toMatchObject({ + identity: { + registrations: expect.arrayContaining([ + expect.objectContaining({ client: "opencode", state: "invalid" }), + ]), + }, + }); + expect(await readFile(upper, "utf8")).toBe(upperContent); + for (const path of [lower, upper]) + await expect(access(`${path}.rea.backup`)).rejects.toMatchObject({ + code: "ENOENT", + }); + }, +); + +cliTest( + "public setup propagates selected provider deadlines and input profile to Codex and Kimi", + async ({ cli }) => { + const home = await createTestTempDirectory( + "rea-selected-registration-env-", + ); + const codexHome = join(home, "codex"); + const kimiHome = join(home, "kimi"); + await Promise.all( + [codexHome, kimiHome].map((path) => mkdir(path, { recursive: true })), + ); + const environment = { + HOME: home, + USERPROFILE: home, + XDG_CONFIG_HOME: join(home, ".config"), + CODEX_HOME: codexHome, + KIMI_CODE_HOME: kimiHome, + HOPPER_LAUNCHER_PATH: join(home, "missing-hopper"), + REA_ANALYSIS_PROVIDER: "auto", + REA_GHIDRA_STARTUP_TIMEOUT_MS: "2700001", + REA_HOPPER_STARTUP_TIMEOUT_MS: "2800002", + REA_JEB_MCP_TIMEOUT_MS: "2900003", + REA_MCP_INPUT_SCHEMA_PROFILE: "full", + REA_UNRELATED_MARKER: "do-not-persist", + }; + const run = (arguments_: readonly string[]) => + cli.run({ + arguments: [...arguments_, "--json"], + cwd: home, + environment, + }); + + for (const client of ["codex", "kimi"] as const) { + const setup = await run([ + "setup", + "--client", + client, + "--skill=false", + "--yes", + ]); + expect(setup.exitCode, setup.stderr).toBe(0); + expect(setup.json).toMatchObject({ status: "ready" }); + } + + const codexConfig = await readFile(join(codexHome, "config.toml"), "utf8"); + const codexRegistration = effectiveClientServer( + parseClientConfiguration(codexConfig, "toml"), + "rea", + ); + expect(codexRegistration).toMatchObject({ tool_timeout_sec: 3021 }); + expect(codexConfig).toContain('REA_JEB_MCP_TIMEOUT_MS = "2900003"'); + expect(codexConfig).not.toContain("REA_UNRELATED_MARKER"); + + const kimiPath = join(kimiHome, "mcp.json"); + const kimiRegistration = effectiveClientServer( + parseClientConfiguration(await readFile(kimiPath, "utf8"), "kimi"), + "rea", + ); + expect(kimiRegistration).toMatchObject({ + toolTimeoutMs: 3_020_003, + env: { + REA_ANALYSIS_PROVIDER: "auto", + REA_GHIDRA_STARTUP_TIMEOUT_MS: "2700001", + REA_HOPPER_STARTUP_TIMEOUT_MS: "2800002", + REA_JEB_MCP_TIMEOUT_MS: "2900003", + REA_MCP_INPUT_SCHEMA_PROFILE: "full", + }, + }); + expect(await readFile(kimiPath, "utf8")).not.toContain( + "REA_UNRELATED_MARKER", + ); + + const doctor = await run(["doctor", "--client", "kimi"]); + expect(doctor.exitCode, doctor.stderr).toBe(0); + expect(doctor.json).toMatchObject({ + identity: { + registrations: expect.arrayContaining([ + expect.objectContaining({ client: "kimi", state: "aligned" }), + ]), + }, + }); + }, + 120_000, +); + +for (const [client, provider] of [ + ["codex", "ida"], + ["kimi", "ida"], + ["kimi", "auto"], +] as const) + cliTest( + `public ${client} setup and doctor use the effective IDA registration request budget with ${provider} selection`, + async ({ cli }) => { + const home = await createTestTempDirectory( + "rea-ida-registration-budget-", + ); + const clientHome = join(home, client); + await mkdir(clientHome, { recursive: true }); + const idaConfig = join(home, "ida.json"); + const writeRegistration = (timeoutMs: number) => + writeFile( + idaConfig, + JSON.stringify({ + url: "http://127.0.0.1:12345/mcp", + headers: { Authorization: "Bearer fixture-credential" }, + timeoutMs, + }), + ); + await writeRegistration(1_800_001); + const environment = { + HOME: home, + USERPROFILE: home, + XDG_CONFIG_HOME: join(home, ".config"), + CODEX_HOME: clientHome, + KIMI_CODE_HOME: clientHome, + HOPPER_LAUNCHER_PATH: join(home, "missing-hopper"), + }; + const setupArguments = [ + "setup", + "--client", + client, + "--skill=false", + "--yes", + "--json", + ]; + const configured = await cli.run({ + arguments: setupArguments, + cwd: home, + environment: { + ...environment, + REA_ANALYSIS_PROVIDER: provider, + REA_IDA_MCP_CONFIG: idaConfig, + }, + }); + expect(configured.exitCode, configured.stdout).toBe(0); + const config = join( + clientHome, + client === "codex" ? "config.toml" : "mcp.json", + ); + const text = await readFile(config, "utf8"); + expect( + effectiveClientServer( + parseClientConfiguration(text, client === "codex" ? "toml" : "kimi"), + "rea", + ), + ).toMatchObject({ + ...(client === "codex" + ? { tool_timeout_sec: 1921 } + : { toolTimeoutMs: 1_920_001 }), + env: { REA_ANALYSIS_PROVIDER: provider, REA_IDA_MCP_CONFIG: idaConfig }, + }); + expect(text).not.toContain("fixture-credential"); + const run = (arguments_: readonly string[]) => + cli.run({ arguments: arguments_, cwd: home, environment }); + const healthy = await run(["doctor", "--client", client, "--json"]); + expect(healthy.exitCode, healthy.stdout).toBe(0); + expect(JSON.stringify(healthy.json)).not.toContain('"advisory"'); + + // Only the stored client's provider path identifies this changed budget; + // the doctor/setup process has no IDA configuration in its environment. + await writeRegistration(2_400_001); + const doctor = await run(["doctor", "--client", client, "--json"]); + expect(doctor.exitCode, doctor.stdout).toBe(0); + expect(doctor.json).toMatchObject({ + identity: { + registrations: expect.arrayContaining([ + expect.objectContaining({ + client, + state: "aligned", + advisory: expect.stringContaining( + client === "codex" + ? "tool_timeout_sec = 2521 seconds" + : "toolTimeoutMs = 2520001 ms", + ), + }), + ]), + }, + }); + const repeated = await run(setupArguments); + expect(repeated.exitCode, repeated.stdout).toBe(0); + expect(repeated.json).toMatchObject({ + status: "ready", + plannedActions: [], + appliedActions: [], + }); + expect(await readFile(config, "utf8")).toBe(text); + await expect(access(`${config}.rea.backup`)).rejects.toMatchObject({ + code: "ENOENT", + }); + if (provider === "ida") { + await writeRegistration(0); + const invalidDoctor = await run([ + "doctor", + "--client", + client, + "--json", + ]); + expect(invalidDoctor.exitCode).toBe(1); + expect(invalidDoctor.json).toMatchObject({ + identity: { + registrations: expect.arrayContaining([ + expect.objectContaining({ + client, + state: "invalid", + remediation: expect.stringContaining("REA_IDA_MCP_CONFIG"), + }), + ]), + }, + }); + const invalidSetup = await run(setupArguments); + expect(invalidSetup.exitCode).toBe(1); + expect(invalidSetup.stdout).toContain("REA_IDA_MCP_CONFIG"); + expect(await readFile(config, "utf8")).toBe(text); + await expect(access(`${config}.rea.backup`)).rejects.toMatchObject({ + code: "ENOENT", + }); + } + }, + 120_000, + ); + +for (const invalidSetting of [ + { setting: "REA_MCP_INPUT_SCHEMA_PROFILE", value: "unknown" }, + { setting: "REA_GHIDRA_STARTUP_TIMEOUT_MS", value: "0" }, + { setting: "REA_JEB_MCP_TIMEOUT_MS", value: "1.5" }, +] as const) + cliTest( + `setup rejects malformed selected ${invalidSetting.setting} before writing`, + async ({ cli }) => { + const home = await createTestTempDirectory( + "rea-invalid-selected-registration-env-", + ); + const kimiHome = join(home, "kimi"); + await mkdir(kimiHome, { recursive: true }); + const config = join(kimiHome, "mcp.json"); + const environment = { + HOME: home, + USERPROFILE: home, + XDG_CONFIG_HOME: join(home, ".config"), + KIMI_CODE_HOME: kimiHome, + HOPPER_LAUNCHER_PATH: join(home, "missing-hopper"), + [invalidSetting.setting]: invalidSetting.value, + }; + const result = await cli.run({ + arguments: [ + "setup", + "--client", + "kimi", + "--skill=false", + "--yes", + "--json", + ], + cwd: home, + environment, + }); + expect(result.exitCode).toBe(1); + expect(result.json).toMatchObject({ + code: "configuration_invalid", + category: "invalid_input", + details: { + settings: [ + expect.objectContaining({ setting: invalidSetting.setting }), + ], + }, + }); + expect(`${result.stdout}\n${result.stderr}`).toContain( + invalidSetting.setting, + ); + await expect(access(config)).rejects.toMatchObject({ code: "ENOENT" }); + await expect(access(`${config}.rea.backup`)).rejects.toMatchObject({ + code: "ENOENT", + }); + }, + 120_000, + ); diff --git a/tests/acceptance/setup/selectedSkillSetup.test.ts b/tests/acceptance/setup/selectedSkillSetup.test.ts index bc73c85b3d..8c76e1ac68 100644 --- a/tests/acceptance/setup/selectedSkillSetup.test.ts +++ b/tests/acceptance/setup/selectedSkillSetup.test.ts @@ -1,11 +1,190 @@ -import { access, readFile, readdir, writeFile } from "node:fs/promises"; -import { join, relative, resolve } from "node:path"; +import { + access, + cp, + lstat, + mkdir, + readFile, + readdir, + symlink, + writeFile, +} from "node:fs/promises"; +import { dirname, join, relative, resolve } from "node:path"; import { expect } from "vitest"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; import { cliTest } from "../../support/cli/cliFixture.js"; +cliTest.for(["root", "references"] as const)( + "rejects a linked skill %s during planning before configuring a client", + async (kind, { cli }) => { + const directory = await createTestTempDirectory("rea-skill-preflight-"); + const selected = join( + directory, + ".claude/skills/reverse-engineer-anything", + ); + const managed = join(directory, "user-managed"); + const source = resolve("skills/reverse-engineer-anything"); + await mkdir(kind === "root" ? dirname(selected) : selected, { + recursive: true, + }); + await cp(kind === "root" ? source : join(source, "references"), managed, { + recursive: true, + }); + if (kind === "references") + await cp(join(source, "SKILL.md"), join(selected, "SKILL.md")); + const link = kind === "root" ? selected : join(selected, "references"); + await symlink( + managed, + link, + process.platform === "win32" ? "junction" : "dir", + ); + const run = (arguments_: readonly string[]) => + cli.run({ + arguments: [...arguments_, "--json"], + cwd: directory, + environment: { USERPROFILE: directory }, + }); + const preview = await run([ + "setup", + "--client", + "claude_code", + "--dry-run", + ]); + const applied = await run(["setup", "--client", "claude_code", "--yes"]); + await expect(access(join(directory, ".claude.json"))).rejects.toMatchObject( + { + code: "ENOENT", + }, + ); + for (const result of [preview, applied]) { + expect(result.exitCode, result.stdout).toBe(1); + expect(result.json).toMatchObject({ + status: "needs_human", + code: "skill_install_failed", + appliedActions: [], + }); + expect(result.stdout).toContain(link); + expect(result.stdout).toContain("--skill=false"); + } + const doctor = await run(["doctor", "--client", "claude_code", "--skill"]); + expect(doctor.exitCode, doctor.stdout).toBe(1); + expect(doctor.json).toMatchObject({ + healthy: false, + identity: { skill: { state: "unmanaged" } }, + }); + expect(doctor.stdout).toContain(link); + expect((await lstat(link)).isSymbolicLink()).toBe(true); + }, +); + +cliTest( + "setup refuses a linked skill while allowing registration-only setup", + async ({ cli }) => { + const directory = await createTestTempDirectory("rea-linked-skill-cli-"); + const managed = join(directory, "managed"); + const selected = join( + directory, + ".claude/skills/reverse-engineer-anything", + ); + await mkdir(managed); + await mkdir(join(directory, ".claude/skills"), { recursive: true }); + await cp(resolve("skills/reverse-engineer-anything"), managed, { + recursive: true, + }); + const original = await readFile(join(managed, "SKILL.md"), "utf8"); + await symlink( + managed, + selected, + process.platform === "win32" ? "junction" : "dir", + ); + const run = (arguments_: readonly string[]) => + cli.run({ + arguments: [...arguments_, "--json"], + cwd: directory, + environment: { USERPROFILE: directory }, + }); + + const refused = await run([ + "setup", + "--skill-client", + "claude_code", + "--yes", + ]); + expect(refused.exitCode, refused.stdout).toBe(1); + expect(refused.json).toMatchObject({ + status: "needs_human", + code: "skill_install_failed", + }); + expect(refused.stdout).toContain("symbolic link"); + const doctor = await run(["doctor", "--client", "claude_code", "--skill"]); + expect(doctor.exitCode, doctor.stdout).toBe(1); + expect(doctor.json).toMatchObject({ + healthy: false, + identity: { skill: { state: "unmanaged" } }, + }); + const registration = await run([ + "setup", + "--client", + "claude_code", + "--skill=false", + "--yes", + ]); + expect(registration.exitCode, registration.stdout).toBe(0); + expect(registration.json).toMatchObject({ status: "ready" }); + await access(join(directory, ".claude.json")); + expect((await lstat(selected)).isSymbolicLink()).toBe(true); + expect((await readdir(managed)).sort()).toEqual(["SKILL.md", "references"]); + expect(await readFile(join(managed, "SKILL.md"), "utf8")).toBe(original); + }, +); + +cliTest( + "uninstall reports retained references and preserves their ownership manifest", + async ({ cli }) => { + const directory = await createTestTempDirectory("rea-skill-retained-cli-"); + const selected = join( + directory, + ".claude/skills/reverse-engineer-anything", + ); + const managed = join(directory, "user-managed-references"); + const source = resolve("skills/reverse-engineer-anything"); + await mkdir(selected, { recursive: true }); + await cp(join(source, "SKILL.md"), join(selected, "SKILL.md")); + await cp(join(source, "references"), managed, { recursive: true }); + await symlink( + managed, + join(selected, "references"), + process.platform === "win32" ? "junction" : "dir", + ); + const before = await readFile(join(managed, "native-and-artifacts.md")); + const result = await cli.run({ + arguments: ["uninstall", "--json"], + cwd: directory, + environment: { USERPROFILE: directory }, + }); + expect(result.exitCode, result.stdout).toBe(0); + expect(result.json).toMatchObject({ + items: expect.arrayContaining([ + expect.objectContaining({ + name: "skill", + status: "retained", + detail: expect.stringContaining("retained"), + }), + ]), + }); + expect(await readFile(join(selected, "SKILL.md"))).toEqual( + await readFile(join(source, "SKILL.md")), + ); + expect(await readFile(join(managed, "native-and-artifacts.md"))).toEqual( + before, + ); + expect((await lstat(join(selected, "references"))).isSymbolicLink()).toBe( + true, + ); + }, +); + cliTest( "skill-only setup verifies its selected client copy on install, rerun and repair", async ({ cli }) => { diff --git a/tests/acceptance/setup/unqualifiedDoctorSkill.test.ts b/tests/acceptance/setup/unqualifiedDoctorSkill.test.ts new file mode 100644 index 0000000000..014467dc3f --- /dev/null +++ b/tests/acceptance/setup/unqualifiedDoctorSkill.test.ts @@ -0,0 +1,121 @@ +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { dirname, join } from "node:path"; +import { expect } from "vitest"; + +import { PRODUCT_IDENTITY } from "../../../src/identity.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { cliTest } from "../../support/cli/cliFixture.js"; + +cliTest( + "unqualified doctor checks the installed managed skill and preserves its files", + async ({ cli }) => { + const home = await createTestTempDirectory("rea-unqualified-doctor-skill-"); + const skill = join( + home, + ".claude/skills/reverse-engineer-anything/SKILL.md", + ); + const run = (arguments_: readonly string[]) => + cli.run({ + arguments: [...arguments_, "--json"], + cwd: home, + environment: { HOME: home, USERPROFILE: home }, + }); + + const install = await run([ + "setup", + "--skill-client", + "claude_code", + "--yes", + ]); + expect(install.exitCode, install.stdout).toBe(0); + const original = await readFile(skill); + + const skillCheck = await run(["doctor", "--skill"]); + expect(skillCheck.exitCode, skillCheck.stdout).toBe(0); + expect(skillCheck.json).toMatchObject({ + healthy: true, + identity: { + skill: { + state: "aligned", + installed_version: PRODUCT_IDENTITY.skillVersion, + installed_tool_count: expect.any(Number), + }, + }, + }); + expect(await readFile(skill)).toEqual(original); + + const audit = await run(["doctor"]); + expect(audit.json).toMatchObject({ + identity: { + skill: { + state: "aligned", + installed_version: PRODUCT_IDENTITY.skillVersion, + installed_tool_count: expect.any(Number), + }, + }, + }); + expect(await readFile(skill)).toEqual(original); + + await mkdir(join(home, ".cursor")); + const withMarker = await run(["doctor"]); + expect(withMarker.json).toMatchObject({ + identity: { + skill: { + state: "aligned", + installed_version: PRODUCT_IDENTITY.skillVersion, + installed_tool_count: expect.any(Number), + }, + }, + }); + expect(await readFile(skill)).toEqual(original); + + await writeFile( + skill, + Buffer.concat([original, Buffer.from("\nlocal edit\n")]), + ); + const changed = await run(["doctor", "--skill"]); + expect(changed.exitCode).toBe(1); + expect(changed.json).toMatchObject({ + healthy: false, + identity: { skill: { state: "stale" } }, + }); + + const emptyHome = await createTestTempDirectory( + "rea-unqualified-doctor-empty-", + ); + const empty = await cli.run({ + arguments: ["doctor", "--skill", "--json"], + cwd: emptyHome, + environment: { HOME: emptyHome, USERPROFILE: emptyHome }, + }); + expect(empty.exitCode).toBe(1); + expect(empty.json).toMatchObject({ + healthy: false, + identity: { skill: { state: "missing" } }, + }); + + const foreignHome = await createTestTempDirectory( + "rea-unqualified-doctor-foreign-skill-", + ); + const foreignSkill = join( + foreignHome, + ".claude/skills/reverse-engineer-anything/SKILL.md", + ); + const foreignBytes = Buffer.from( + "---\nname: another-skill\n---\nA caller-owned workflow.\n", + ); + await mkdir(dirname(foreignSkill), { recursive: true }); + await writeFile(foreignSkill, foreignBytes); + const foreign = await cli.run({ + arguments: ["doctor", "--skill", "--json"], + cwd: foreignHome, + environment: { HOME: foreignHome, USERPROFILE: foreignHome }, + }); + expect(foreign.exitCode).toBe(1); + expect(foreign.json).toMatchObject({ + healthy: false, + identity: { skill: { state: "stale", installed_version: null } }, + }); + expect(await readFile(foreignSkill)).toEqual(foreignBytes); + }, +); diff --git a/tests/acceptance/tools/localToolContracts.test.ts b/tests/acceptance/tools/localToolContracts.test.ts index 7b911bd27e..9f3a092ba9 100644 --- a/tests/acceptance/tools/localToolContracts.test.ts +++ b/tests/acceptance/tools/localToolContracts.test.ts @@ -156,6 +156,38 @@ const pathRequests: readonly [string, Record][] = [ import_map: { path: nul, base_url: "http://127.0.0.1/a" }, }, ], + ["start_frida_session", { mode: "spawn", device_id: "local", program: nul }], + [ + "load_frida_script", + { + session_id: "00000000-0000-4000-8000-000000000001", + source_kind: "file", + path: nul, + }, + ], + [ + "instrument_with_frida", + { + mode: "spawn", + device_id: "local", + program: nul, + source_kind: "inline", + source: "send(1);", + }, + ], + [ + "instrument_with_frida", + { + mode: "spawn", + device_id: "local", + program: "/bin/true", + source_kind: "file", + path: nul, + }, + ], + ["start_rizin_debug_session", { path: nul, inferior_tty: "/dev/pts/7" }], + ["start_rizin_debug_session", { path: "/tmp/program", inferior_tty: nul }], + ["start_gdb_session", { inferior_tty: nul }], ]; it("rejects OS-invalid local inputs through the real advertised MCP contracts", async () => { diff --git a/tests/acceptance/tools/passiveNetworkInventory.test.ts b/tests/acceptance/tools/passiveNetworkInventory.test.ts new file mode 100644 index 0000000000..8e5e7a51ff --- /dev/null +++ b/tests/acceptance/tools/passiveNetworkInventory.test.ts @@ -0,0 +1,215 @@ +import { readFile, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { expect } from "vitest"; + +import { parseEvidence } from "../../../src/domain/evidence.js"; +import { webCaptureDiffSchema } from "../../../src/domain/webCaptureDiffSchemas.js"; +import { connectLocalToolsMcp } from "../../fixtures/localToolsMcp.js"; +import { + passiveNetworkCapture, + withPassiveDroppedEvents, +} from "../../fixtures/passiveNetworkCapture.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { cliTest } from "../../support/cli/cliFixture.js"; + +const comparisonCases = () => { + const before = passiveNetworkCapture(); + const after = passiveNetworkCapture(["/a", "/b"]); + const incomplete = structuredClone(before); + incomplete.completeness.attach_limited_sections.push("network_requests"); + const wsBefore = structuredClone(before); + const wsAfter = structuredClone(after); + for (const capture of [wsBefore, wsAfter]) + capture.completeness.attach_limited_sections.push( + "websocket_connections", + "websocket_frames", + ); + wsAfter.network.websocket_connections.push({ + request_id: "ws", + url: "wss://app.example.test/live", + events: [], + }); + const repeated = passiveNetworkCapture(["/a", "/a"]); + const repeatedUnknown = structuredClone(repeated); + repeatedUnknown.network.requests[1]!.resource_type = null; + const unknownIdentity = structuredClone(before); + unknownIdentity.network.requests[0]!.resource_type = null; + const unknownIdentityAdded = structuredClone(after); + unknownIdentityAdded.network.requests[0]!.resource_type = null; + const unselectedUnknownIdentity = structuredClone(unknownIdentity); + unselectedUnknownIdentity.completeness.excluded.push({ + section: "json_body_shapes", + reason: "not_approved", + count: null, + }); + unselectedUnknownIdentity.network.requests[0]!.body_shapes.status = + "not_approved"; + const droppedRequest = withPassiveDroppedEvents(before, { + network_requests: 1, + }); + const droppedWebSocket = withPassiveDroppedEvents(before, { + websocket_connections: 1, + websocket_frames: 1, + }); + const unrelatedDrop = withPassiveDroppedEvents(before, { + console_events: 1, + }); + const droppedUnselected = withPassiveDroppedEvents(structuredClone(before), { + network_requests: 1, + }); + droppedUnselected.completeness.excluded.push({ + section: "json_body_shapes", + reason: "not_approved", + count: null, + }); + droppedUnselected.network.requests[0]!.body_shapes.status = "not_approved"; + const droppedRepeated = withPassiveDroppedEvents(before, { + network_requests: 1, + }); + const redirectKnown = structuredClone(before); + redirectKnown.network.requests[0]!.redirects = [ + { + url: "https://app.example.test/start", + response_url: "https://app.example.test/start", + method: "GET", + resource_type: "Document", + status: 302, + mime_type: "text/html", + encoded_data_length: 20, + request_timestamp: 1, + redirect_event_timestamp: 2, + }, + ]; + const redirectUnknown = structuredClone(redirectKnown); + redirectUnknown.network.requests[0]!.redirects![0]!.resource_type = null; + return [ + ["added", before, after, "added"], + ["removed", after, before, "removed"], + ["equal unavailable", before, before, null], + ["incomplete inventory", incomplete, after, null], + ["limited WebSocket inventory", wsBefore, wsAfter, "added"], + ["repeated request type uncertainty", repeated, repeatedUnknown, null], + [ + "repeated request type uncertainty reversed", + repeatedUnknown, + repeated, + null, + ], + ["unknown request identity", before, unknownIdentity, null], + ["unknown request identity reversed", unknownIdentity, before, null], + [ + "unknown identity without selected shapes", + unselectedUnknownIdentity, + unselectedUnknownIdentity, + null, + ], + [ + "different URL beside unknown identity", + before, + unknownIdentityAdded, + "added", + ], + ["dropped request count", droppedRequest, after, null], + ["dropped request count reversed", after, droppedRequest, null], + [ + "dropped request with shapes unselected", + droppedUnselected, + droppedUnselected, + null, + ], + [ + "WebSocket drops do not hide HTTP additions", + droppedWebSocket, + after, + "added", + ], + [ + "unrelated drops do not hide HTTP additions", + unrelatedDrop, + after, + "added", + ], + [ + "duplicate samples with dropped inventory", + droppedRepeated, + repeated, + null, + ], + [ + "duplicate samples with dropped inventory reversed", + repeated, + droppedRepeated, + null, + ], + ["unknown redirect type", redirectKnown, redirectUnknown, null], + ["unknown redirect type reversed", redirectUnknown, redirectKnown, null], + ] as const; +}; + +const cases = comparisonCases().map(([name, left, right, change]) => ({ + name, + left, + right, + change, +})); + +const expectComparison = ( + document: unknown, + name: string, + change: "added" | "removed" | null, +): void => { + const evidence = parseEvidence(document); + expect(evidence.operation).toBe("compare_web_captures"); + const result = webCaptureDiffSchema.parse(evidence.normalized_result); + expect(result.dimensions.network, name).toMatchObject({ + status: change === null ? "unknown" : "changed", + total_changes: change === null ? 0 : 1, + changes: + change === null + ? [] + : [{ identity: expect.stringMatching(/^net_/u), change }], + }); + expect(result.overall_status).toBe(change === null ? "unknown" : "changed"); +}; + +cliTest.for(cases)( + "CLI preserves passive inventory evidence: $name", + async ({ name, left, right, change }, { cli }) => { + const root = await createTestTempDirectory("rea-passive-inventory-"); + const paths = [ + join(root, "before.json"), + join(root, "after.json"), + ] as const; + const serialized = [ + JSON.stringify({ inspection: left }), + JSON.stringify({ inspection: right }), + ]; + await Promise.all( + paths.map((path, index) => writeFile(path, serialized[index]!)), + ); + const response = await cli.run({ + arguments: ["compare-web-captures", ...paths, "--json"], + environment: { REA_LOG_LEVEL: "silent" }, + }); + expect(response.exitCode, response.stderr).toBe(0); + expectComparison(response.json, name, change); + expect( + await Promise.all(paths.map((path) => readFile(path, "utf8"))), + ).toEqual(serialized); + }, +); + +cliTest( + "stdio MCP preserves passive inventory evidence on one owned connection", + async () => { + const mcp = await connectLocalToolsMcp(); + for (const { name, left, right, change } of cases) { + const response = await mcp.call("compare_web_captures", { + before: { inspection: left }, + after: { inspection: right }, + }); + expect(response.isError, JSON.stringify(response)).not.toBe(true); + expectComparison(response.structuredContent, name, change); + } + }, +); diff --git a/tests/acceptance/webRuntimeCli.test.ts b/tests/acceptance/webRuntimeCli.test.ts index 084687dc2e..2b37fecddd 100644 --- a/tests/acceptance/webRuntimeCli.test.ts +++ b/tests/acceptance/webRuntimeCli.test.ts @@ -111,17 +111,124 @@ const cancellationSignals = process.platform === "win32" ? (["SIGINT"] as const) : (["SIGINT", "SIGTERM"] as const); -const webRuntimeCancellationCases = ( - ["observe-web-execution", "inspect-web-event-listeners"] as const -).flatMap((command) => - cancellationSignals.map( - (signal) => [command, signal, signal === "SIGINT" ? 130 : 143] as const, - ), + +interface CliCancellationCommand { + readonly command: string; + readonly operation: string; + readonly buildArgs: (endpoint: string) => readonly string[]; +} + +const cancellationCommands: readonly CliCancellationCommand[] = [ + { + command: "observe-web-execution", + operation: "observe_web_execution", + buildArgs: (endpoint) => [ + "observe-web-execution", + endpoint, + "page-fixture", + "--observation-ms", + "30000", + "--json", + ], + }, + { + command: "inspect-web-event-listeners", + operation: "inspect_web_event_listeners", + buildArgs: (endpoint) => [ + "inspect-web-event-listeners", + endpoint, + "page-fixture", + "body", + "--json", + ], + }, + { + command: "list-browser-targets", + operation: "list_browser_targets", + buildArgs: (endpoint) => ["list-browser-targets", endpoint, "--json"], + }, + { + command: "inspect-web-page", + operation: "inspect_web_page", + buildArgs: (endpoint) => [ + "inspect-web-page", + endpoint, + "page-fixture", + "--json", + ], + }, + { + command: "analyze-web-bundle", + operation: "analyze_web_bundle", + buildArgs: (endpoint) => [ + "analyze-web-bundle", + endpoint, + "page-fixture", + "--json", + ], + }, + { + command: "observe-web-session", + operation: "observe_web_session", + buildArgs: (endpoint) => [ + "observe-web-session", + endpoint, + "page-fixture", + "--json", + ], + }, + { + command: "discover-webmcp-tools", + operation: "discover_webmcp_tools", + buildArgs: (endpoint) => [ + "discover-webmcp-tools", + endpoint, + "page-fixture", + "--json", + ], + }, + { + command: "capture-web-screenshot", + operation: "capture_web_screenshot", + buildArgs: (endpoint) => [ + "capture-web-screenshot", + endpoint, + "page-fixture", + "--json", + ], + }, + { + command: "list-javascript-runtime-targets", + operation: "list_javascript_runtime_targets", + buildArgs: (endpoint) => [ + "list-javascript-runtime-targets", + endpoint, + "--json", + ], + }, + { + command: "observe-javascript-runtime", + operation: "observe_javascript_runtime", + buildArgs: (endpoint) => [ + "observe-javascript-runtime", + endpoint, + "target-fixture", + "--json", + ], + }, +]; + +const cancellationCases = cancellationCommands.flatMap((command) => + cancellationSignals.map((signal) => ({ + ...command, + signal, + expectedExitCode: signal === "SIGINT" ? 130 : 143, + })), ); -it.each(webRuntimeCancellationCases)( - "%s preserves %s status when cancellation interrupts CDP discovery", - async (command, signal, expectedExitCode) => { +it.each(cancellationCases)( + "$command preserves $signal status and awaited cleanup during CDP discovery", + async ({ command, operation, buildArgs, signal, expectedExitCode }) => { const sockets = new Set(); const server = createServer(() => undefined); server.on("connection", (socket) => { @@ -136,21 +243,14 @@ it.each(webRuntimeCancellationCases)( const endpoint = `http://127.0.0.1:${String(address.port)}`; const contacted = once(server, "request"); - const args = - command === "observe-web-execution" - ? [ - command, - endpoint, - "page-fixture", - "--observation-ms", - "30000", - "--json", - ] - : [command, endpoint, "page-fixture", "body", "--json"]; - const child = spawn(process.execPath, [entrypoint, ...args], { - env: environment(), - stdio: ["ignore", "pipe", "pipe"], - }); + const child = spawn( + process.execPath, + [entrypoint, ...buildArgs(endpoint)], + { + env: environment(), + stdio: ["ignore", "pipe", "pipe"], + }, + ); const closed = once(child, "close"); let stdout = ""; child.stdout.setEncoding("utf8"); @@ -180,7 +280,7 @@ it.each(webRuntimeCancellationCases)( expect(JSON.parse(stdout)).toMatchObject({ code: "cancelled", category: "cancelled", - details: { cleanup: "complete" }, + details: { cleanup: "complete", operation }, }); } finally { clearTimeout(timeout); diff --git a/tests/boundary/android/JadxAnalysis.test.ts b/tests/boundary/android/JadxAnalysis.test.ts index 36b0ab9d9c..10d451cad9 100644 --- a/tests/boundary/android/JadxAnalysis.test.ts +++ b/tests/boundary/android/JadxAnalysis.test.ts @@ -4,7 +4,10 @@ import { analyzeJadxRequest, type JadxToolPort, } from "../../../src/android/JadxAnalysis.js"; -import { androidInputSchemas } from "../../../src/domain/android/androidAnalysis.js"; +import { + androidInputSchemas, + androidResultSchemas, +} from "../../../src/domain/android/androidAnalysis.js"; const summary = { full_name: "fixture.Target", @@ -41,6 +44,43 @@ const engine = { heap_limit_mib: null, }; +it("requires explicit direct invoke coverage state in result evidence", () => { + const result = { + engine, + class_name: "fixture.Target", + method_name: null, + overload_index: null, + reported_signature: null, + dex_descriptor: null, + total: 1, + references: [ + { + kind: "method", + name: "caller", + reported_full_name: "fixture.Other.caller", + containing_class: "fixture.Other", + top_class: "fixture.Other", + source_line: null, + dex_offset: null, + direct_invokes: null, + }, + ], + coverage: "complete", + reference_unit: "using_node", + }; + expect( + androidResultSchemas.trace_android_references.safeParse(result).success, + ).toBe(false); + expect( + androidResultSchemas.trace_android_references.safeParse({ + ...result, + references: [ + { ...result.references[0], direct_invokes_exhaustive: null }, + ], + }).success, + ).toBe(true); +}); + it("publishes exact DEX prototypes instead of discarding bridge metadata", async () => { const tools: JadxToolPort = { json: () => Promise.resolve(summary), @@ -246,7 +286,9 @@ it.each([ async (fields, expected) => { const result = await traceWithReference(fields as Record); expect(result).toMatchObject({ - references: [{ direct_invokes: expected }], + references: [ + { direct_invokes: expected, direct_invokes_exhaustive: null }, + ], }); }, ); @@ -279,26 +321,44 @@ it("preserves bounded instruction coverage separately from using-node coverage", }); }); -it("rejects malformed method descriptors at the Android adapter boundary", async () => { - const invalidSummary = { - ...summary, - methods: summary.methods.map((method) => ({ - ...method, - dex_descriptor: "not-a-dex-descriptor", - })), - }; - await expect( - analyzeJadxRequest( - { - json: () => Promise.resolve(invalidSummary), - text: () => Promise.reject(new Error("unexpected text call")), +it.each([ + "not-a-dex-descriptor", + "(Lbad(name;)V", + "(Lbad)name;)V", + "()Lbad(name;", + "()Lbad)name;", +])( + "rejects malformed DEX prototype %s at the adapter boundary", + async (descriptor) => { + const invalidSummary = { + ...summary, + methods: summary.methods.map((method) => ({ + ...method, + dex_descriptor: descriptor, + })), + }; + await expect( + analyzeJadxRequest( + { + json: () => Promise.resolve(invalidSummary), + text: () => Promise.reject(new Error("unexpected text call")), + }, + { + operation: "inspect_android_class", + input: { path: "/fixture.apk", class_name: "fixture.Target" }, + }, + {} as never, + engine, + ), + ).rejects.toMatchObject({ + _tag: "AnalysisOutputError", + cause: { + issues: expect.arrayContaining([ + expect.objectContaining({ + path: ["methods", 0, "dex_descriptor"], + }), + ]), }, - { - operation: "inspect_android_class", - input: { path: "/fixture.apk", class_name: "fixture.Target" }, - }, - {} as never, - engine, - ), - ).rejects.toThrow(); -}); + }); + }, +); diff --git a/tests/boundary/android/jadxConfiguration.test.ts b/tests/boundary/android/jadxConfiguration.test.ts index 3ebba32718..7600ce44ff 100644 --- a/tests/boundary/android/jadxConfiguration.test.ts +++ b/tests/boundary/android/jadxConfiguration.test.ts @@ -380,3 +380,10 @@ const createFakeJdk = async ( await chmod(java, 0o700); return javaHome; }; + +it("tags unavailable diagnostics with the boundary failure kind", async () => { + const availability = await inspectJadxAvailability({}); + expect(availability.status).toBe("unavailable"); + if (availability.status !== "unavailable") throw new Error("unreachable"); + expect(availability.diagnostics["failure_kind"]).toBe("jar"); +}); diff --git a/tests/boundary/android/jadxDoctor.test.ts b/tests/boundary/android/jadxDoctor.test.ts new file mode 100644 index 0000000000..e053f66b5f --- /dev/null +++ b/tests/boundary/android/jadxDoctor.test.ts @@ -0,0 +1,396 @@ +import { chmod, mkdir, realpath, symlink, writeFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { expect, it as test } from "vitest"; + +import { inspectJadxProvider } from "../../../src/android/JadxDoctor.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { writeJadxJarInventory } from "../../fixtures/android/jadxJar.js"; +import { writeOrderedZip } from "../../fixtures/artifactEntryOrder.js"; + +const it = test.skipIf(process.platform === "win32"); + +it("publishes REA_JADX_MCP_JAR as registration environment when the configured JAR validates", async () => { + const root = await createTestTempDirectory("rea-jadx-doctor-"); + const jar = join(root, "jadx-headless-mcp.jar"); + const javaHome = await createFakeJdk(root); + await writeJadxJarInventory(jar); + + const inspection = await inspectJadxProvider({ + REA_JADX_MCP_JAR: jar, + JAVA_HOME: javaHome, + }); + + expect(inspection).toMatchObject({ + id: "jadx", + configured: true, + available: true, + providerVersion: null, + registrationEnvironment: { REA_JADX_MCP_JAR: await realpath(jar) }, + checks: [ + { + name: "configuration", + ok: true, + code: null, + classification: "config_drift", + }, + ], + }); +}); + +it("publishes JAVA_HOME alongside the JAR when the configured JAR validates", async () => { + const root = await createTestTempDirectory("rea-jadx-doctor-javahome-"); + const jar = join(root, "jadx-headless-mcp.jar"); + const javaHome = await createFakeJdk(root); + await writeJadxJarInventory(jar); + + const inspection = await inspectJadxProvider({ + REA_JADX_MCP_JAR: jar, + JAVA_HOME: javaHome, + }); + + expect(inspection.registrationEnvironment).toEqual({ + REA_JADX_MCP_JAR: jar, + JAVA_HOME: javaHome, + }); +}); + +it("preserves the caller-selected JAR path when it is a symlink", async () => { + const root = await createTestTempDirectory("rea-jadx-doctor-symlink-"); + const jar = join(root, "jadx-headless-mcp.jar"); + const link = join(root, "current.jar"); + const javaHome = await createFakeJdk(root); + await writeJadxJarInventory(jar); + await symlink(jar, link); + + const inspection = await inspectJadxProvider({ + REA_JADX_MCP_JAR: link, + JAVA_HOME: javaHome, + }); + + expect(inspection.available).toBe(true); + expect(inspection.registrationEnvironment["REA_JADX_MCP_JAR"]).toBe(link); +}); + +it("guides Java selection when the Java runtime is missing", async () => { + const root = await createTestTempDirectory("rea-jadx-doctor-remediation-"); + const jar = join(root, "jadx-headless-mcp.jar"); + await writeJadxJarInventory(jar); + + const inspection = await inspectJadxProvider({ + REA_JADX_MCP_JAR: jar, + JAVA_HOME: join(root, "missing-jdk"), + }); + + expect(inspection.checks[0]?.code).toBe("runtime_missing"); + expect(inspection.checks[0]?.remediation).toContain("JAVA_HOME"); + expect(inspection.checks[0]?.remediation).not.toContain("REA_JADX_MCP_JAR"); +}); + +it("guides Java selection when JAVA_HOME is not an absolute path", async () => { + const root = await createTestTempDirectory("rea-jadx-doctor-options-"); + const jar = join(root, "jadx-headless-mcp.jar"); + await writeJadxJarInventory(jar); + + const inspection = await inspectJadxProvider({ + REA_JADX_MCP_JAR: jar, + JAVA_HOME: "relative/path", + }); + + expect(inspection.checks[0]?.code).toBe("open_options_invalid"); + expect(inspection.checks[0]?.remediation).toContain("JAVA_HOME"); + expect(inspection.checks[0]?.remediation).not.toContain("REA_JADX_MCP_JAR"); +}); + +it("publishes no registration environment when REA_JADX_MCP_JAR is unset", async () => { + const inspection = await inspectJadxProvider({}); + + expect(inspection).toMatchObject({ + id: "jadx", + configured: false, + available: false, + providerVersion: null, + registrationEnvironment: {}, + checks: [ + { + name: "configuration", + ok: false, + code: "not_configured", + classification: "missing_analysis_engine", + }, + ], + }); +}); + +it("publishes no registration environment when the configured JAR fails validation", async () => { + const root = await createTestTempDirectory("rea-jadx-doctor-invalid-"); + const jar = join(root, "engine.jar"); + const javaHome = await createFakeJdk(root); + await writeFile(jar, "not a Java archive"); + + const inspection = await inspectJadxProvider({ + REA_JADX_MCP_JAR: jar, + JAVA_HOME: javaHome, + }); + + expect(inspection).toMatchObject({ + id: "jadx", + configured: true, + available: false, + registrationEnvironment: {}, + checks: [ + { + name: "configuration", + ok: false, + code: "version_unresolved", + classification: "config_drift", + }, + ], + }); + expect(inspection.checks[0]?.detail).toContain("REA_JADX_MCP_JAR"); +}); + +it("classifies a missing Java runtime as a missing dependency", async () => { + const root = await createTestTempDirectory("rea-jadx-doctor-runtime-"); + const jar = join(root, "jadx-headless-mcp.jar"); + await writeJadxJarInventory(jar); + + const inspection = await inspectJadxProvider({ + REA_JADX_MCP_JAR: jar, + JAVA_HOME: join(root, "missing-jdk"), + }); + + expect(inspection).toMatchObject({ + configured: true, + available: false, + registrationEnvironment: {}, + checks: [ + { + name: "configuration", + ok: false, + code: "runtime_missing", + classification: "missing_dependency", + }, + ], + }); +}); + +it("guides JVM option correction when REA_JADX_HEAP_MIB is invalid", async () => { + const root = await createTestTempDirectory("rea-jadx-doctor-heap-"); + const jar = join(root, "jadx-headless-mcp.jar"); + const javaHome = await createFakeJdk(root); + await writeJadxJarInventory(jar); + + const inspection = await inspectJadxProvider({ + REA_JADX_MCP_JAR: jar, + JAVA_HOME: javaHome, + REA_JADX_HEAP_MIB: "not-a-number", + }); + + expect(inspection.checks[0]?.code).toBe("open_options_invalid"); + expect(inspection.checks[0]?.remediation).toContain("REA_JADX_HEAP_MIB"); + expect(inspection.checks[0]?.remediation).not.toContain("JAVA_HOME"); +}); + +it("publishes validated JVM resource overrides in the registration environment", async () => { + const root = await createTestTempDirectory("rea-jadx-doctor-overrides-"); + const jar = join(root, "jadx-headless-mcp.jar"); + const javaHome = await createFakeJdk(root); + await writeJadxJarInventory(jar); + + const inspection = await inspectJadxProvider({ + REA_JADX_MCP_JAR: jar, + JAVA_HOME: javaHome, + REA_JADX_HEAP_MIB: "4096", + REA_JADX_ACTIVE_PROCESSOR_COUNT: "8", + }); + + expect(inspection.available).toBe(true); + expect(inspection.registrationEnvironment).toEqual({ + REA_JADX_MCP_JAR: jar, + JAVA_HOME: javaHome, + REA_JADX_HEAP_MIB: "4096", + REA_JADX_ACTIVE_PROCESSOR_COUNT: "8", + }); +}); + +it("guides JDK selection when the Java runtime is older than 17", async () => { + const root = await createTestTempDirectory("rea-jadx-doctor-oldjdk-"); + const jar = join(root, "jadx-headless-mcp.jar"); + const javaHome = await createFakeJdk(root, "11"); + await writeJadxJarInventory(jar); + + const inspection = await inspectJadxProvider({ + REA_JADX_MCP_JAR: jar, + JAVA_HOME: javaHome, + }); + + expect(inspection.checks[0]?.code).toBe("unsupported_version"); + expect(inspection.checks[0]?.remediation).toContain("JDK"); + expect(inspection.checks[0]?.remediation).not.toContain("REA_JADX_MCP_JAR"); +}); + +it("guides JAR replacement when the JAR misses bridge classes", async () => { + const root = await createTestTempDirectory("rea-jadx-doctor-classes-"); + const jar = join(root, "jadx-headless-mcp.jar"); + const javaHome = await createFakeJdk(root); + await writeOrderedZip(jar, ["com/atxx/jhmcp/JadxSession.class"]); + + const inspection = await inspectJadxProvider({ + REA_JADX_MCP_JAR: jar, + JAVA_HOME: javaHome, + }); + + expect(inspection.checks[0]?.code).toBe("unsupported_version"); + expect(inspection.checks[0]?.remediation).toContain("REA_JADX_MCP_JAR"); +}); + +it("remediates an unsupported host with platform guidance", async () => { + const originalPlatform = process.platform; + Object.defineProperty(process, "platform", { + value: "sunos", + configurable: true, + }); + try { + const inspection = await inspectJadxProvider({ + REA_JADX_MCP_JAR: "/opt/jadx-headless-mcp.jar", + }); + + expect(inspection.checks[0]?.code).toBe("unsupported_host"); + expect(inspection.checks[0]?.remediation).toContain("Linux"); + } finally { + Object.defineProperty(process, "platform", { + value: originalPlatform, + configurable: true, + }); + } +}); + +it("guides Windows native package correction when process ownership is unavailable", async () => { + const originalPlatform = process.platform; + const originalArch = process.arch; + Object.defineProperty(process, "platform", { + value: "win32", + configurable: true, + }); + Object.defineProperty(process, "arch", { value: "x64", configurable: true }); + try { + const inspection = await inspectJadxProvider({ + REA_JADX_MCP_JAR: "/opt/jadx-headless-mcp.jar", + }); + + expect(inspection.checks[0]?.code).toBe("runtime_missing"); + expect(inspection.checks[0]?.remediation).toContain( + "Windows x64 REA package", + ); + expect(inspection.checks[0]?.remediation).not.toContain("JAVA_HOME"); + } finally { + Object.defineProperty(process, "platform", { + value: originalPlatform, + configurable: true, + }); + Object.defineProperty(process, "arch", { + value: originalArch, + configurable: true, + }); + } +}); + +it("guides Java probe correction when the module listing is unparseable", async () => { + const root = await createTestTempDirectory("rea-jadx-doctor-unparseable-"); + const jar = join(root, "jadx-headless-mcp.jar"); + await writeJadxJarInventory(jar); + // Fake JDK whose module listing lacks a parseable java.base version. + const javaHome = join(root, "fake-jdk"); + const bin = join(javaHome, "bin"); + const java = join(bin, "java"); + await mkdir(bin, { recursive: true }); + await writeFile(java, "#!/bin/sh\nprintf 'jdk.compiler@21\\n'\n"); + await chmod(java, 0o700); + + const inspection = await inspectJadxProvider({ + REA_JADX_MCP_JAR: jar, + JAVA_HOME: javaHome, + }); + + expect(inspection.checks[0]?.code).toBe("version_unresolved"); + expect(inspection.checks[0]?.remediation).toContain("java --list-modules"); + expect(inspection.checks[0]?.remediation).not.toContain("REA_JADX_MCP_JAR"); +}); + +const createFakeJdk = async (root: string, major = "21"): Promise => { + const javaHome = join(root, "fake-jdk"); + const bin = join(javaHome, "bin"); + const java = join(bin, "java"); + await mkdir(bin, { recursive: true }); + await writeFile( + java, + `#!/bin/sh\nprintf 'java.base@${major}\\njdk.compiler@${major}\\n'\n`, + ); + await chmod(java, 0o700); + return javaHome; +}; + +it("classifies missing Windows native artifacts as missing dependencies", async () => { + const originalPlatform = process.platform; + const originalArch = process.arch; + Object.defineProperty(process, "platform", { + value: "win32", + configurable: true, + }); + Object.defineProperty(process, "arch", { value: "x64", configurable: true }); + try { + const inspection = await inspectJadxProvider({ + REA_JADX_MCP_JAR: "/opt/jadx-headless-mcp.jar", + }); + + expect(inspection.checks[0]?.code).toBe("runtime_missing"); + expect(inspection.checks[0]?.classification).toBe("missing_dependency"); + } finally { + Object.defineProperty(process, "platform", { + value: originalPlatform, + configurable: true, + }); + Object.defineProperty(process, "arch", { + value: originalArch, + configurable: true, + }); + } +}); + +it("classifies malformed JAVA_HOME as configuration drift", async () => { + const root = await createTestTempDirectory("rea-jadx-doctor-javahome-"); + const jar = join(root, "jadx-headless-mcp.jar"); + await writeJadxJarInventory(jar); + + const inspection = await inspectJadxProvider({ + REA_JADX_MCP_JAR: jar, + JAVA_HOME: "relative/path", + }); + + expect(inspection.checks[0]?.code).toBe("open_options_invalid"); + expect(inspection.checks[0]?.classification).toBe("config_drift"); +}); + +it("preserves Java probe diagnostics in doctor output", async () => { + const root = await createTestTempDirectory("rea-jadx-doctor-probe-"); + const jar = join(root, "jadx-headless-mcp.jar"); + await writeJadxJarInventory(jar); + const javaHome = join(root, "fake-jdk-unparseable"); + const bin = join(javaHome, "bin"); + const java = join(bin, "java"); + await mkdir(bin, { recursive: true }); + // Emits jdk.compiler but no parseable java.base version. + await writeFile(java, `#!/bin/sh\nprintf 'jdk.compiler@21\\n'\n`); + await chmod(java, 0o700); + + const inspection = await inspectJadxProvider({ + REA_JADX_MCP_JAR: jar, + JAVA_HOME: javaHome, + }); + + expect(inspection.checks[0]?.code).toBe("version_unresolved"); + expect(inspection.checks[0]?.details).toMatchObject({ + java_executable: expect.any(String), + }); +}); diff --git a/tests/boundary/browser/browserProcessStartup.test.ts b/tests/boundary/browser/browserProcessStartup.test.ts index b1cfaeafee..3bf88d1027 100644 --- a/tests/boundary/browser/browserProcessStartup.test.ts +++ b/tests/boundary/browser/browserProcessStartup.test.ts @@ -14,8 +14,31 @@ import { BrowserStartupError, waitForBrowserDevtoolsPort, } from "../../../src/browser/BrowserProcessStartup.js"; +import { assertOrdinaryWindowsVerifierCaller } from "../../../scripts/lib/windows-ordinary-user-verifier.mjs"; describe("browser process startup", () => { + it("requires a positively observed ordinary Windows verifier caller", () => { + expect(() => + assertOrdinaryWindowsVerifierCaller({ + administrator: false, + elevated: false, + integrityLevel: 8192, + }), + ).not.toThrow(); + + for (const token of [ + { administrator: true, elevated: false }, + { administrator: false, elevated: true }, + { administrator: false }, + { administrator: false, elevated: null }, + null, + ]) { + expect(() => assertOrdinaryWindowsVerifierCaller(token)).toThrow( + /ordinary, non-elevated user token/u, + ); + } + }); + it("returns a delayed valid DevToolsActivePort", async ({ processes }) => { const root = await temporaryRoot(); const portPath = join(root, "DevToolsActivePort"); diff --git a/tests/boundary/browser/browserVerifierFixture.test.ts b/tests/boundary/browser/browserVerifierFixture.test.ts new file mode 100644 index 0000000000..e171d14f87 --- /dev/null +++ b/tests/boundary/browser/browserVerifierFixture.test.ts @@ -0,0 +1,30 @@ +import { createConnection } from "node:net"; +import { it, expect } from "vitest"; +import { startBrowserVerifierSite } from "../../../scripts/fixtures/browser-verifier-site.mjs"; + +it("closes an accepted HTTP socket that has not sent request headers", async () => { + const site = await startBrowserVerifierSite(); + const { port } = new URL(site.origin); + const socket = createConnection(Number(port), "127.0.0.1"); + let close: Promise | undefined; + let timeout: ReturnType | undefined; + try { + await new Promise((resolve, reject) => { + socket.once("connect", resolve); + socket.once("error", reject); + }); + const closing = site.close(); + close = closing; + const closedBeforeDeadline = await Promise.race([ + closing.then(() => true), + new Promise((resolve) => { + timeout = setTimeout(() => resolve(false), 1_000); + }), + ]); + expect(closedBeforeDeadline).toBe(true); + } finally { + if (timeout !== undefined) clearTimeout(timeout); + socket.destroy(); + await (close ?? site.close()); + } +}); diff --git a/tests/boundary/browser/cdpBrowserProvider.document_script.test.ts b/tests/boundary/browser/cdpBrowserProvider.document_script.test.ts index 5fd1e061bb..8127879871 100644 --- a/tests/boundary/browser/cdpBrowserProvider.document_script.test.ts +++ b/tests/boundary/browser/cdpBrowserProvider.document_script.test.ts @@ -256,63 +256,77 @@ describe("CdpBrowserProvider: transient WebMCP frames", () => { }); describe("CdpBrowserProvider: WebMCP discovery and completeness", () => { - it("discovers untrusted WebMCP declarations without registering or invoking them", async () => { - const browser = await startFakeCdpBrowser({ webMcpTools: true }); - trackBrowser(browser); - const result = await new CdpBrowserProvider().discoverWebMcpTools( - discoverWebMcpToolsInputSchema.parse({ - cdp_endpoint: browser.endpoint, - allowed_origins: [browser.allowedOrigin], - target_id: "allowed-page", - observation_ms: 0, - }), - ); + it.each([true, false])( + "discovers untrusted WebMCP declarations with consequential=%s without invoking them", + async (consequential) => { + const browser = await startFakeCdpBrowser({ + webMcpTools: true, + webMcpConsequential: consequential, + }); + trackBrowser(browser); + const result = await new CdpBrowserProvider().discoverWebMcpTools( + discoverWebMcpToolsInputSchema.parse({ + cdp_endpoint: browser.endpoint, + allowed_origins: [browser.allowedOrigin], + target_id: "allowed-page", + observation_ms: 0, + }), + ); - if (!result.ok) throw result.error; - expect(result.value).toMatchObject({ - status: "available", - tools: { - total: 1, - items: [ - { - name: "search_orders", - description: "Search orders; authorization=Bearer tool-secret", - declaration_kind: "declarative", - owner_origin: browser.allowedOrigin, - annotations: { - read_only: true, - untrusted_content: true, - autosubmit: false, + if (!result.ok) throw result.error; + expect(result.value).toMatchObject({ + status: "available", + tools: { + total: 1, + items: [ + { + name: "search_orders", + description: "Search orders; authorization=Bearer tool-secret", + declaration_kind: "declarative", + owner_origin: browser.allowedOrigin, + annotations: { + read_only: true, + untrusted_content: true, + consequential, + autosubmit: false, + }, + trust: "page-declared-untrusted", + registration_source: { + url: `${browser.allowedOrigin}/app.js?token=tool-source-secret`, + line: 12, + column: 4, + }, }, - trust: "page-declared-untrusted", - registration_source: { - url: `${browser.allowedOrigin}/app.js?token=tool-source-secret`, - line: 12, - column: 4, - }, - }, - ], - }, - }); - expect(result.value.completeness.excluded).toContainEqual( - expect.objectContaining({ - section: "webmcp_tools", - reason: "out_of_target_scope", - }), - ); - expect(result.value.completeness.unavailable_sections).not.toContain( - "webmcp_tools", - ); - const serialized = JSON.stringify(result.value); - expect(serialized).toContain("tool-secret"); - expect(serialized).toContain("tool-source-secret"); - expect(serialized).toContain("schema-secret"); - expect(serialized).not.toContain("private-tool-secret"); - const methods = browser.commands.map(({ method }) => method); - expect(methods).toContain("WebMCP.enable"); - expect(methods).not.toContain("WebMCP.invokeTool"); - expect(methods).not.toContain("Runtime.evaluate"); - }); + ], + }, + }); + expect(result.value.completeness.excluded).toContainEqual( + expect.objectContaining({ + section: "webmcp_tools", + reason: "out_of_target_scope", + }), + ); + expect(result.value.completeness.unavailable_sections).not.toContain( + "webmcp_tools", + ); + const serialized = JSON.stringify(result.value); + expect(serialized).toContain("tool-secret"); + expect(serialized).toContain("tool-source-secret"); + expect(serialized).toContain("schema-secret"); + expect(serialized).not.toContain("private-tool-secret"); + const methods = browser.commands.map(({ method }) => method); + expect(methods).toContain("WebMCP.enable"); + expect(methods).toContain("WebMCP.disable"); + expect(methods.indexOf("WebMCP.enable")).toBeLessThan( + methods.indexOf("WebMCP.disable"), + ); + expect(methods.indexOf("WebMCP.disable")).toBeLessThan( + methods.indexOf("Target.detachFromTarget"), + ); + expect(methods).not.toContain("WebMCP.invokeTool"); + expect(methods).not.toContain("Runtime.evaluate"); + }, + ); it("reports WebMCP unavailable when the experimental domain is absent", async () => { const browser = await startFakeCdpBrowser({ @@ -756,6 +770,15 @@ describe("CdpBrowserProvider: WebMCP registration owners", () => { const tools = result.value.tools.items; expect(tools).toHaveLength(mode === "retain" ? 2 : 1); expect(result.value.tools.total).toBe(tools.length); + // Declarations without annotations report absent hints as null, never + // collapsed to a declared false. + for (const tool of tools) + expect(tool.annotations).toEqual({ + read_only: null, + untrusted_content: null, + consequential: null, + autosubmit: null, + }); expect(tools).toContainEqual( expect.objectContaining({ name: "read_item", diff --git a/tests/boundary/browser/cdpBrowserProvider.lifecycle_failures.test.ts b/tests/boundary/browser/cdpBrowserProvider.lifecycle_failures.test.ts index 7c0d07445e..01b1f54da6 100644 --- a/tests/boundary/browser/cdpBrowserProvider.lifecycle_failures.test.ts +++ b/tests/boundary/browser/cdpBrowserProvider.lifecycle_failures.test.ts @@ -1,10 +1,11 @@ -import { describe, expect, it } from "vitest"; +import { describe, expect, it, vi } from "vitest"; import { CdpBrowserProvider } from "../../../src/browser/CdpBrowserProvider.js"; import { inspectWebPageInputSchema, listBrowserTargetsInputSchema, } from "../../../src/domain/browserObservation.js"; +import { AnalysisInputError } from "../../../src/domain/analysisErrorCore.js"; import { analyzeWebBundleInputSchema } from "../../../src/domain/webBundleAnalysis.js"; import { observeWebSessionInputSchema } from "../../../src/domain/browserSession.js"; import { @@ -514,6 +515,71 @@ describe("CdpBrowserProvider: navigation commit and target identity", () => { }); }); +describe("CdpBrowserProvider: source-map cancellation", () => { + it("classifies an AnalysisError abort reason as cancellation", async () => { + const browser = await startFakeCdpBrowser({ + sourceMapBody: JSON.stringify({ + version: 3, + sources: ["app.ts"], + names: [], + mappings: "", + }), + }); + trackBrowser(browser); + const controller = new AbortController(); + const reason = new AnalysisInputError("caller_operation"); + let markSourceMapRequested!: () => void; + const sourceMapRequested = new Promise((resolve) => { + markSourceMapRequested = resolve; + }); + const originalFetch = globalThis.fetch; + const fetchMock = vi + .spyOn(globalThis, "fetch") + .mockImplementation((resource, init) => { + if (String(resource).includes("/app.js.map")) { + markSourceMapRequested(); + return new Promise((_resolve, reject) => { + const signal = init?.signal; + if (signal?.aborted === true) { + reject(signal.reason); + return; + } + signal?.addEventListener("abort", () => reject(signal.reason), { + once: true, + }); + }); + } + return originalFetch(resource, init); + }); + try { + const pending = new CdpBrowserProvider().analyzeBundle( + analyzeWebBundleInputSchema.parse({ + cdp_endpoint: browser.endpoint, + allowed_origins: [browser.allowedOrigin], + target_id: "allowed-page", + observation_ms: 0, + fetch_source_maps: true, + }), + { signal: controller.signal }, + ); + await sourceMapRequested; + controller.abort(reason); + const result = await pending; + expect(result).toMatchObject({ + ok: false, + error: { + _tag: "AnalysisCancelledError", + operation: "analyze_web_bundle", + }, + }); + if (result.ok) throw new Error("Expected source-map cancellation"); + expect(result.error.cause).toBe(reason); + } finally { + fetchMock.mockRestore(); + } + }); +}); + const waitForCommand = async ( browser: FakeCdpBrowser, method: string, diff --git a/tests/boundary/browser/cdpCleanupCancellation.test.ts b/tests/boundary/browser/cdpCleanupCancellation.test.ts index 0d4c17a0fa..8495313132 100644 --- a/tests/boundary/browser/cdpCleanupCancellation.test.ts +++ b/tests/boundary/browser/cdpCleanupCancellation.test.ts @@ -73,7 +73,13 @@ it("releases cleanup when cancellation arrives after capture completes", async ( }); controller.abort(); const result = await pending; - expect(result.ok).toBe(true); + expect(result).toMatchObject({ + ok: false, + error: { + _tag: "AnalysisCancelledError", + operation: "observe_web_session", + }, + }); expect(browser.commands.map(({ method }) => method)).toContain( "Target.detachFromTarget", ); diff --git a/tests/boundary/browser/cdpConnection.test.ts b/tests/boundary/browser/cdpConnection.test.ts index 993a44e6d6..71ef6a2456 100644 --- a/tests/boundary/browser/cdpConnection.test.ts +++ b/tests/boundary/browser/cdpConnection.test.ts @@ -28,6 +28,7 @@ const INVALID_REPLIES = [ }), ], ["invalid result", (id: number) => ({ id, result: [] })], + ["missing session id", (id: number) => ({ id, result: {} })], ["invalid session id", (id: number) => ({ id, result: {}, sessionId: 7 })], [ "foreign session id", diff --git a/tests/boundary/browser/cdpDomComparisonCompleteness.test.ts b/tests/boundary/browser/cdpDomComparisonCompleteness.test.ts new file mode 100644 index 0000000000..d8c5386c32 --- /dev/null +++ b/tests/boundary/browser/cdpDomComparisonCompleteness.test.ts @@ -0,0 +1,266 @@ +import { describe, expect, it } from "vitest"; +import { writeFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { CdpBrowserProvider } from "../../../src/browser/CdpBrowserProvider.js"; +import { inspectWebPageInputSchema } from "../../../src/domain/browserObservation.js"; +import type { WebPageInspection } from "../../../src/domain/browserObservationSchemas.js"; +import { compareWebCaptures } from "../../../src/domain/webCaptureDiff.js"; +import { compareWebCapturesInputSchema } from "../../../src/domain/webCaptureDiffSchemas.js"; +import { startFakeCdpBrowser } from "../../fixtures/fakeCdpBrowser.js"; +import { trackBrowser } from "./cdpBrowserProvider.support.js"; +import { runCliJson } from "../../fixtures/cliJsonProcess.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +type SnapshotMode = + | "valid" + | "omitted-node" + | "changed-node" + | "missing-nodes" + | "null-nodes" + | "missing-documents" + | "invalid-documents" + | "empty-documents" + | "invalid-document" + | "mixed-documents" + | "missing-strings"; + +const snapshot = (origin: string, mode: SnapshotMode) => { + const document = { + documentURL: 0, + nodes: { + nodeType: [9, 1], + nodeName: [1, mode === "changed-node" ? 6 : 3], + nodeValue: [2, 2], + parentIndex: [-1, 0], + attributes: [[], mode === "omitted-node" ? null : [4, 5]], + }, + }; + let documents: unknown = [document]; + if (mode === "missing-nodes") documents = [{ documentURL: 0 }]; + if (mode === "null-nodes") documents = [{ documentURL: 0, nodes: null }]; + if (mode === "missing-documents") documents = undefined; + if (mode === "invalid-documents") documents = {}; + if (mode === "empty-documents") documents = []; + if (mode === "invalid-document") documents = [null]; + if (mode === "mixed-documents") documents = [document, null]; + return { + strings: + mode === "missing-strings" + ? undefined + : [origin + "/app", "#document", "", "A", "href", "/target", "SPAN"], + documents, + }; +}; + +const startBrowser = async () => { + const state = { + mode: "valid" as SnapshotMode, + nested: false, + renamed: false, + parentOverride: undefined as { value: unknown } | undefined, + }; + const browser = await startFakeCdpBrowser({ + commandResult(command, origin) { + if (command.method === "DOMSnapshot.captureSnapshot") + return snapshot(origin, state.mode); + if (command.method !== "Page.getFrameTree") return undefined; + const id = (name: string) => (state.renamed ? "new-" + name : name); + const child = { + frame: { + id: id("b"), + parentId: + state.parentOverride === undefined + ? id(state.nested ? "a" : "main") + : state.parentOverride.value, + loaderId: "loader-b", + url: origin + "/b", + }, + }; + const parent = { + frame: { + id: id("a"), + parentId: id("main"), + loaderId: "loader-a", + url: origin + "/a", + }, + childFrames: state.nested ? [child] : [], + }; + return { + frameTree: { + frame: { + id: id("main"), + loaderId: "loader-main", + url: origin + "/app", + }, + childFrames: state.nested ? [parent] : [parent, child], + }, + }; + }, + }); + trackBrowser(browser); + const inspect = async () => { + const result = await new CdpBrowserProvider().inspectPage( + inspectWebPageInputSchema.parse({ + cdp_endpoint: browser.endpoint, + allowed_origins: [browser.allowedOrigin], + target_id: "allowed-page", + observation_ms: 0, + }), + ); + if (!result.ok) throw result.error; + return result.value; + }; + return { state, inspect }; +}; + +const compare = (before: WebPageInspection, after: WebPageInspection) => + compareWebCaptures( + compareWebCapturesInputSchema.parse({ + before: { inspection: before }, + after: { inspection: after }, + }), + ); + +describe("DOM protocol omissions at the production capture boundary", () => { + it.each([ + "missing-nodes", + "null-nodes", + "missing-documents", + "invalid-documents", + "empty-documents", + "invalid-document", + "mixed-documents", + "missing-strings", + ] as const)("retains unknown coverage for %s", async (mode) => { + const { state, inspect } = await startBrowser(); + state.mode = mode; + const capture = await inspect(); + for (const section of ["dom", "metadata"]) + expect(capture.completeness.unavailable_sections).toContain(section); + if (mode === "mixed-documents") { + expect(capture.dom.nodes.map(({ node_name }) => node_name)).toEqual([ + "#document", + "A", + ]); + expect(capture.metadata.dom_urls).toHaveLength(1); + } + const comparison = compare(capture, capture); + expect(comparison.dimensions.dom_structure.status).toBe("unknown"); + expect(comparison.dimensions.metadata.status).toBe("unknown"); + }); + + it("does not turn a rejected node and its URL into a page change", async () => { + const { state, inspect } = await startBrowser(); + const valid = await inspect(); + state.mode = "omitted-node"; + const partial = await inspect(); + expect(partial.dom.total_nodes).toBe(2); + expect(partial.dom.nodes).toHaveLength(1); + for (const [before, after] of [ + [valid, partial], + [partial, valid], + ]) { + if (before === undefined || after === undefined) + throw new Error("Missing pair"); + const comparison = compare(before, after); + expect(comparison.dimensions.dom_structure.status).toBe("unknown"); + expect(comparison.dimensions.metadata.status).toBe("unknown"); + expect(comparison.overall_status).toBe("unknown"); + } + }); +}); + +describe("complete frame and DOM comparison", () => { + it.each([undefined, null, 17, ""])( + "keeps missing or invalid child parent evidence unknown (%#)", + async (parent) => { + const { state, inspect } = await startBrowser(); + const before = await inspect(); + state.parentOverride = { value: parent }; + const after = await inspect(); + expect(after.frames).toHaveLength(3); + expect(after.completeness.unavailable_sections).toContain("frames"); + expect(compare(before, after).dimensions.dom_structure.status).toBe( + "unknown", + ); + expect(compare(after, after).dimensions.dom_structure.status).toBe( + "unknown", + ); + }, + ); + it("retains a complete observed node change", async () => { + const { state, inspect } = await startBrowser(); + const before = await inspect(); + expect(compare(before, before).dimensions.dom_structure.status).toBe( + "unchanged", + ); + state.mode = "changed-node"; + expect( + compare(before, await inspect()).dimensions.dom_structure.status, + ).toBe("changed"); + }); + + it("compares frame ancestry independently of volatile IDs", async () => { + const { state, inspect } = await startBrowser(); + const before = await inspect(); + state.renamed = true; + expect( + compare(before, await inspect()).dimensions.dom_structure.status, + ).toBe("unchanged"); + state.nested = true; + const nested = await inspect(); + expect(nested.frames[2]?.parent_frame_id).toBe("new-a"); + expect(compare(before, nested).dimensions.dom_structure.status).toBe( + "changed", + ); + }); + + it("retains proven frame changes when DOM node evidence is incomplete", async () => { + const { state, inspect } = await startBrowser(); + const before = await inspect(); + state.mode = "omitted-node"; + state.nested = true; + const comparison = compare(before, await inspect()); + expect(comparison.dimensions.dom_structure.status).toBe("changed"); + expect(comparison.dimensions.metadata.status).toBe("unknown"); + }); +}); + +describe("DOM evidence through the public CLI", () => { + it("preserves omissions and independently proven frame changes", async () => { + const { state, inspect } = await startBrowser(); + const root = await createTestTempDirectory("rea-dom-comparison-"); + const before = join(root, "before.json"); + const after = join(root, "after.json"); + await writeFile(before, JSON.stringify({ inspection: await inspect() })); + state.mode = "omitted-node"; + for (const nested of [false, true]) { + state.nested = nested; + await writeFile(after, JSON.stringify({ inspection: await inspect() })); + expect( + await runCliJson(["compare-web-captures", before, after, "--json"]), + ).toMatchObject({ + operation: "compare_web_captures", + normalized_result: { + overall_status: nested ? "changed" : "unknown", + dimensions: { + dom_structure: { status: nested ? "changed" : "unknown" }, + metadata: { status: "unknown" }, + }, + }, + }); + } + state.mode = "missing-nodes"; + await writeFile(after, JSON.stringify({ inspection: await inspect() })); + expect( + await runCliJson(["compare-web-captures", after, after, "--json"]), + ).toMatchObject({ + operation: "compare_web_captures", + normalized_result: { + overall_status: "unknown", + dimensions: { dom_structure: { status: "unknown" } }, + }, + }); + }); +}); diff --git a/tests/boundary/browser/cdpDomSvgIdentity.test.ts b/tests/boundary/browser/cdpDomSvgIdentity.test.ts index 7d42ae3366..eeaebf2e7a 100644 --- a/tests/boundary/browser/cdpDomSvgIdentity.test.ts +++ b/tests/boundary/browser/cdpDomSvgIdentity.test.ts @@ -254,6 +254,7 @@ describe("CDP SVG query concurrency", () => { browser.emitRawMessage( JSON.stringify({ id: command.id, + sessionId: command.sessionId, result: { node: { backendNodeId: id, @@ -295,6 +296,7 @@ describe("CDP SVG query concurrency", () => { browser.emitRawMessage( JSON.stringify({ id: batch.at(-1)?.id, + sessionId: batch.at(-1)?.sessionId, error: { code: -32_602, message: "Invalid parameters" }, }), ); diff --git a/tests/boundary/browser/cdpElectronProvider.test.ts b/tests/boundary/browser/cdpElectronProvider.test.ts index f115846109..c8a3241f4b 100644 --- a/tests/boundary/browser/cdpElectronProvider.test.ts +++ b/tests/boundary/browser/cdpElectronProvider.test.ts @@ -258,7 +258,62 @@ describe("CdpElectronProvider packaged pages", () => { }); }); -describe("CdpElectronProvider capture", () => { +describe("CdpElectronProvider frame capture", () => { + it.each([7, undefined])( + "reports a malformed child frame ID as unavailable protocol evidence: %s", + async (childId) => { + const root = await electronFixture(); + const main = pathToFileURL(join(root, "index.html")).href; + const child = pathToFileURL(join(root, "child.html")).href; + const browser = await startFakeCdpBrowser({ + electronFileUrl: main, + commandResult: (command) => + command.method === "Page.getFrameTree" + ? { + frameTree: { + frame: { id: "frame-main", url: main }, + childFrames: [ + { + frame: { + ...(childId === undefined ? {} : { id: childId }), + url: child, + }, + }, + ], + }, + } + : undefined, + }); + browsers.push(browser); + const result = await new CdpElectronProvider().inspectPage( + inspectElectronPageInputSchema.parse({ + cdp_endpoint: browser.endpoint, + target_id: "electron-page", + observation_ms: 0, + }), + ); + + if (!result.ok) throw result.error; + expect(result.value.frames.map((frame) => frame.frame_id)).toEqual([ + "frame-main", + ]); + expect(result.value.completeness).toMatchObject({ + unavailable_sections: ["frames"], + }); + expect(result.value.completeness.excluded).toContainEqual({ + section: "frames", + reason: "invalid_protocol_value", + count: 1, + }); + expect(result.value.completeness.policy_filtered_sections).not.toContain( + "frames", + ); + expect(result.value.completeness.conditions).toContain("attach_limited"); + }, + ); +}); + +describe("CdpElectronProvider malformed source replies", () => { it.each([{}, { scriptSource: null }, { scriptSource: 123 }])( "rejects malformed Electron source replies instead of inventing source: %j", async (reply) => { @@ -291,7 +346,66 @@ describe("CdpElectronProvider capture", () => { ); }, ); +}); +describe("CdpElectronProvider malformed inventory replies", () => { + const inspectWith = async ( + method: string, + reply: (fileUrl: string) => Readonly>, + ) => { + const root = await electronFixture(); + const fileUrl = pathToFileURL(join(root, "index.html")).href; + const browser = await startFakeCdpBrowser({ + electronFileUrl: fileUrl, + commandResult: (command) => + command.method === method ? reply(fileUrl) : undefined, + }); + browsers.push(browser); + const result = await new CdpElectronProvider().inspectPage( + inspectElectronPageInputSchema.parse({ + cdp_endpoint: browser.endpoint, + target_id: "electron-page", + observation_ms: 0, + }), + ); + if (!result.ok) throw result.error; + return result.value; + }; + + it.each([ + ["dom", "DOMSnapshot.captureSnapshot", () => ({ strings: [] })], + [ + "dom", + "DOMSnapshot.captureSnapshot", + (fileUrl: string) => ({ + strings: [fileUrl], + documents: [{ documentURL: 0, nodes: { nodeType: null } }], + }), + ], + [ + "resources", + "Page.getResourceTree", + (fileUrl: string) => ({ + frameTree: { frame: { id: "main", url: fileUrl }, resources: null }, + }), + ], + ["workers", "Target.getTargets", () => ({ targetInfos: null })], + ] as const)( + "keeps a malformed %s inventory incomplete (%s)", + async (section, method, reply) => { + const inspected = await inspectWith(method, reply); + expect(inspected.completeness.status).not.toBe("complete_within_window"); + expect(inspected.completeness.excluded).toContainEqual( + expect.objectContaining({ + section, + reason: "invalid_protocol_value", + }), + ); + }, + ); +}); + +describe("CdpElectronProvider script source capture", () => { it("retains an explicitly empty Electron producer script", async () => { const root = await electronFixture(); const browser = await startFakeCdpBrowser({ diff --git a/tests/boundary/browser/cdpObservationResourceTypes.test.ts b/tests/boundary/browser/cdpObservationResourceTypes.test.ts index f7c97d3ddc..2dece6f965 100644 --- a/tests/boundary/browser/cdpObservationResourceTypes.test.ts +++ b/tests/boundary/browser/cdpObservationResourceTypes.test.ts @@ -50,8 +50,80 @@ it.each(["Document", undefined] as const)( ); if (!result.ok) throw result.error; expect(result.value.window.end_reason).toBe("target_left_scope"); + expect(result.value.target.final_url).toBeNull(); expect(result.value.timeline.some(({ type }) => type === "redirect")).toBe( true, ); }, ); + +it("keeps final URL unknown when a redirect races the final frame-tree read", async () => { + let frameTreeReads = 0; + const browser = await startFakeCdpBrowser({ + eventsBeforeReply: ["Page.getFrameTree"], + commandEvents: (command) => { + if (command.method !== "Page.getFrameTree") return undefined; + frameTreeReads += 1; + if (frameTreeReads !== 2 || command.sessionId === undefined) + return undefined; + return [ + { + method: "Network.requestWillBeSent", + sessionId: command.sessionId, + params: { + requestId: "late-main-document", + type: "Document", + frameId: "frame-main", + request: { + url: "https://private.example.test/redirected", + method: "GET", + }, + redirectResponse: { status: 302 }, + }, + }, + ]; + }, + }); + onTestFinished(async () => { + await browser.close(); + }); + const result = await new CdpBrowserProvider().observeSession( + observeWebSessionInputSchema.parse({ + cdp_endpoint: browser.endpoint, + allowed_origins: [browser.allowedOrigin], + target_id: "allowed-page", + observation_ms: 1, + }), + ); + + if (!result.ok) throw result.error; + expect(result.value.window.end_reason).toBe("target_left_scope"); + expect(result.value.target.final_url).toBeNull(); + expect(result.value.timeline.at(-1)).toMatchObject({ + type: "redirect", + url: null, + destination_scope: "outside_policy", + }); + expect(JSON.stringify(result.value)).not.toContain("private.example.test"); +}); + +it("retains the approved URL returned by the final frame-tree read", async () => { + const browser = await startFakeCdpBrowser(); + onTestFinished(async () => { + await browser.close(); + }); + const result = await new CdpBrowserProvider().observeSession( + observeWebSessionInputSchema.parse({ + cdp_endpoint: browser.endpoint, + allowed_origins: [browser.allowedOrigin], + target_id: "allowed-page", + observation_ms: 1, + }), + ); + + if (!result.ok) throw result.error; + expect(result.value.window.end_reason).toBe("window_elapsed"); + expect(result.value.target.final_url).toBe( + `${browser.allowedOrigin}/app?token=frame-secret`, + ); +}); diff --git a/tests/boundary/browser/cdpResourceCompleteness.test.ts b/tests/boundary/browser/cdpResourceCompleteness.test.ts new file mode 100644 index 0000000000..d40e838cc2 --- /dev/null +++ b/tests/boundary/browser/cdpResourceCompleteness.test.ts @@ -0,0 +1,322 @@ +import { writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; + +import { CdpBrowserProvider } from "../../../src/browser/CdpBrowserProvider.js"; +import { inspectWebPageInputSchema } from "../../../src/domain/browserObservation.js"; +import type { WebPageInspection } from "../../../src/domain/browserObservationSchemas.js"; +import { compareWebCaptures } from "../../../src/domain/webCaptureDiff.js"; +import { compareWebCapturesInputSchema } from "../../../src/domain/webCaptureDiffSchemas.js"; +import { startFakeCdpBrowser } from "../../fixtures/fakeCdpBrowser.js"; +import { runCliJson } from "../../fixtures/cliJsonProcess.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { trackBrowser } from "./cdpBrowserProvider.support.js"; + +type ResourceMode = + | "valid" + | "missing" + | "null" + | "wrong-type" + | "invalid-entry" + | "child-missing" + | "child-invalid-entry" + | "child-wrong-type" + | "empty"; + +const startBrowser = async () => { + const state = { + mode: "valid" as ResourceMode, + peerFields: undefined as Record | undefined, + }; + const browser = await startFakeCdpBrowser({ + commandResult(command, origin) { + const malformedChildren = + state.mode === "child-invalid-entry" || + state.mode === "child-wrong-type"; + if (command.method === "Page.getFrameTree" && malformedChildren) + return { + frameTree: { + frame: { id: "frame-main", url: origin + "/app" }, + childFrames: + state.mode === "child-wrong-type" + ? {} + : [ + { + frame: { + id: "child", + parentId: "frame-main", + url: origin + "/child", + }, + }, + null, + ], + }, + }; + if (command.method !== "Page.getResourceTree") return undefined; + const resource = (name: string) => ({ + url: origin + "/" + name, + type: "Script", + mimeType: "text/javascript", + contentSize: 128, + ...(name === "peer.js" ? state.peerFields : undefined), + }); + let resources: unknown = [resource("keep.js"), resource("peer.js")]; + if (state.mode === "missing") resources = undefined; + if (state.mode === "null") resources = null; + if (state.mode === "wrong-type") resources = {}; + if (state.mode === "invalid-entry") + resources = [resource("keep.js"), null]; + if (state.mode === "empty") resources = []; + const malformedChildFrames = + state.mode === "child-wrong-type" + ? {} + : state.mode === "child-invalid-entry" + ? [ + { + frame: { + id: "child", + parentId: "frame-main", + url: origin + "/child", + }, + resources: [resource("child.js")], + }, + null, + ] + : undefined; + return { + frameTree: { + frame: { id: "frame-main", url: origin + "/app" }, + resources, + ...(malformedChildFrames === undefined + ? {} + : { childFrames: malformedChildFrames }), + ...(state.mode === "child-missing" + ? { + childFrames: [ + { + frame: { + id: "child", + parentId: "frame-main", + url: origin + "/child", + }, + }, + ], + } + : {}), + }, + }; + }, + }); + trackBrowser(browser); + const inspect = async () => { + const result = await new CdpBrowserProvider().inspectPage( + inspectWebPageInputSchema.parse({ + cdp_endpoint: browser.endpoint, + allowed_origins: [browser.allowedOrigin], + target_id: "allowed-page", + observation_ms: 0, + }), + ); + if (!result.ok) throw result.error; + return result.value; + }; + return { state, inspect, origin: browser.allowedOrigin }; +}; + +const compareResources = ( + before: WebPageInspection, + after: WebPageInspection, +) => + compareWebCaptures( + compareWebCapturesInputSchema.parse({ + before: { inspection: before }, + after: { inspection: after }, + }), + ).dimensions.resources; + +describe("resource inventory omissions through CDP", () => { + it.each([ + { fields: { type: undefined }, retained: false }, + { fields: { type: null }, retained: false }, + { fields: { type: 17 }, retained: false }, + { fields: { type: "" }, retained: false }, + { fields: { mimeType: undefined }, retained: false }, + { fields: { mimeType: null }, retained: false }, + { fields: { mimeType: 17 }, retained: false }, + { fields: { contentSize: -1 }, retained: true }, + { fields: { contentSize: null }, retained: true }, + { fields: { contentSize: "128" }, retained: true }, + ])( + "keeps malformed resource fields uncertain: $fields", + async ({ fields, retained }) => { + const { state, inspect, origin } = await startBrowser(); + const before = await inspect(); + state.peerFields = fields; + const after = await inspect(); + for (const [left, right] of [ + [before, after], + [after, before], + [after, after], + ] as const) + expect(compareResources(left, right)).toMatchObject({ + status: "unknown", + total_changes: 0, + changes: [], + }); + expect(after.completeness.excluded).toContainEqual({ + section: "resources", + reason: "invalid_protocol_value", + count: 1, + }); + expect(after.resources.map(({ url }) => url).sort()).toEqual( + retained + ? [origin + "/keep.js", origin + "/peer.js"] + : [origin + "/keep.js"], + ); + if (retained) + expect( + after.resources.find(({ url }) => url.endsWith("/peer.js")), + ).toMatchObject({ + type: "Script", + mime_type: "text/javascript", + content_size: null, + }); + }, + ); + + it.each([ + { + fields: { type: "Other", mimeType: "", contentSize: undefined }, + size: null, + }, + { fields: { contentSize: 0 }, size: 0 }, + { fields: { contentSize: 1.5 }, size: 1.5 }, + ])( + "accepts valid optional and empty fields: $fields", + async ({ fields, size }) => { + const { state, inspect } = await startBrowser(); + state.peerFields = fields; + const capture = await inspect(); + expect(capture.resources).toHaveLength(2); + expect( + capture.resources.find(({ url }) => url.endsWith("/peer.js")) + ?.content_size, + ).toBe(size); + expect(capture.completeness.unavailable_sections).not.toContain( + "resources", + ); + expect(compareResources(capture, capture).status).toBe("unchanged"); + }, + ); + + it.each([ + "missing", + "null", + "wrong-type", + "invalid-entry", + "child-missing", + "child-invalid-entry", + "child-wrong-type", + ] as const)( + "does not claim a resource removal or equality for %s", + async (mode) => { + const { state, inspect, origin } = await startBrowser(); + const before = await inspect(); + expect(before.resources).toHaveLength(2); + expect(before.completeness.unavailable_sections).not.toContain( + "resources", + ); + state.mode = mode; + const after = await inspect(); + for (const [left, right] of [ + [before, after], + [after, before], + [after, after], + ] as const) + expect(compareResources(left, right)).toMatchObject({ + status: "unknown", + total_changes: 0, + changes: [], + }); + expect(after.completeness.unavailable_sections).toContain("resources"); + expect(after.completeness.excluded).toContainEqual({ + section: "resources", + reason: "invalid_protocol_value", + count: mode === "invalid-entry" ? 1 : null, + }); + if (mode === "child-invalid-entry" || mode === "child-wrong-type") { + expect(after.completeness.excluded).toContainEqual({ + section: "frames", + reason: "invalid_protocol_value", + count: null, + }); + expect(after.frames.map(({ frame_id }) => frame_id)).toContain( + "frame-main", + ); + if (mode === "child-invalid-entry") + expect(after.frames.map(({ frame_id }) => frame_id)).toContain( + "child", + ); + } + const expected = + mode === "invalid-entry" + ? [origin + "/keep.js"] + : mode === "child-invalid-entry" + ? [origin + "/child.js", origin + "/keep.js", origin + "/peer.js"] + : mode === "child-missing" + ? [origin + "/keep.js", origin + "/peer.js"] + : mode === "child-wrong-type" + ? [origin + "/keep.js", origin + "/peer.js"] + : []; + expect(after.resources.map(({ url }) => url).sort()).toEqual(expected); + }, + ); + + it("distinguishes a complete empty inventory from an unavailable collection", async () => { + const { state, inspect } = await startBrowser(); + const before = await inspect(); + state.mode = "empty"; + const after = await inspect(); + expect(after.resources).toEqual([]); + expect(after.completeness.unavailable_sections).not.toContain("resources"); + expect(compareResources(before, after)).toMatchObject({ + status: "changed", + total_changes: 2, + changes: [{ change: "removed" }, { change: "removed" }], + }); + expect(compareResources(after, after).status).toBe("unchanged"); + }); +}); + +describe("resource uncertainty in public CLI Evidence", () => { + it("preserves partial membership while complete inventories still prove removal", async () => { + const { state, inspect } = await startBrowser(); + const root = await createTestTempDirectory("rea-resource-comparison-"); + const before = join(root, "before.json"); + const after = join(root, "after.json"); + await writeFile(before, JSON.stringify({ inspection: await inspect() })); + for (const { mode, fields } of [ + { mode: "invalid-entry", fields: undefined }, + { mode: "valid", fields: { type: undefined } }, + { mode: "valid", fields: { contentSize: -1 } }, + { mode: "empty", fields: undefined }, + ] as const) { + state.mode = mode; + state.peerFields = fields; + await writeFile(after, JSON.stringify({ inspection: await inspect() })); + expect( + await runCliJson(["compare-web-captures", before, after, "--json"]), + ).toMatchObject({ + operation: "compare_web_captures", + normalized_result: { + dimensions: { + resources: { + status: mode === "empty" ? "changed" : "unknown", + total_changes: mode === "empty" ? 2 : 0, + }, + }, + }, + }); + } + }); +}); diff --git a/tests/boundary/browser/cdpScreenshotDocumentIdentity.test.ts b/tests/boundary/browser/cdpScreenshotDocumentIdentity.test.ts index b3fcbc592e..b768485989 100644 --- a/tests/boundary/browser/cdpScreenshotDocumentIdentity.test.ts +++ b/tests/boundary/browser/cdpScreenshotDocumentIdentity.test.ts @@ -3,6 +3,30 @@ import { CdpBrowserProvider } from "../../../src/browser/CdpBrowserProvider.js"; import { captureWebScreenshotInputSchema } from "../../../src/domain/webScreenshot.js"; import { startFakeCdpBrowser } from "../../fixtures/fakeCdpBrowser.js"; describe("screenshot committed document identity", () => { + it.each([ + ["missing screenshot data", {}], + ["invalid base64 screenshot data", { data: "not base64%%%" }], + ])("reports %s as a protocol error", async (_label, result) => { + const browser = await startFakeCdpBrowser({ + commandResult: ({ method }) => + method === "Page.captureScreenshot" ? result : undefined, + }); + onTestFinished(async () => { + await browser.close(); + }); + const observation = await new CdpBrowserProvider().captureScreenshot( + captureWebScreenshotInputSchema.parse({ + cdp_endpoint: browser.endpoint, + allowed_origins: [browser.allowedOrigin], + target_id: "allowed-page", + }), + ); + expect(observation).toMatchObject({ + ok: false, + error: { reason: "protocol_error" }, + }); + }); + it.each([ { offset: 8, value: 14 }, { offset: 16, value: 0x8000_0000 }, diff --git a/tests/boundary/browser/cdpTargetDeadlines.test.ts b/tests/boundary/browser/cdpTargetDeadlines.test.ts new file mode 100644 index 0000000000..965c867fff --- /dev/null +++ b/tests/boundary/browser/cdpTargetDeadlines.test.ts @@ -0,0 +1,277 @@ +import { createServer, type Socket } from "node:net"; + +import { afterEach, beforeEach, expect, it, vi } from "vitest"; + +import { CdpBrowserProvider } from "../../../src/browser/CdpBrowserProvider.js"; +import { CdpConnection } from "../../../src/browser/CdpConnection.js"; +import { projectAnalysisError } from "../../../src/domain/analysisErrorProjection.js"; +import { inspectWebPageInputSchema } from "../../../src/domain/browserObservation.js"; +import { WEB_RUNTIME_LIMITS } from "../../../src/domain/webRuntime.js"; +import { V8InspectorProvider } from "../../../src/inspector/V8InspectorProvider.js"; +import { startFakeCdpBrowser } from "../../fixtures/fakeCdpBrowser.js"; +import { startFakeV8Inspector } from "../../fixtures/inspector/fakeV8Inspector.js"; + +// Keep real socket scheduling; expire only a timer whose owning phase is ready. +beforeEach(() => { + vi.spyOn(globalThis, "setTimeout"); + vi.spyOn(globalThis, "clearTimeout"); +}); + +afterEach(() => vi.restoreAllMocks()); + +const activeTimers = (milliseconds: number) => { + const scheduled = vi.mocked(globalThis.setTimeout); + const cleared = vi.mocked(globalThis.clearTimeout); + return scheduled.mock.calls.flatMap(([callback, delay], index) => { + const result = scheduled.mock.results[index]; + if ( + delay !== milliseconds || + result?.type !== "return" || + cleared.mock.calls.some(([timer]) => timer === result.value) + ) + return []; + return [{ callback, timer: result.value }]; + }); +}; + +const expireTimer = (milliseconds: number): void => { + const timers = activeTimers(milliseconds); + expect(timers).toHaveLength(1); + const timer = timers[0]; + if (timer === undefined) throw new Error("Expected one active timer"); + try { + timer.callback(); + if (milliseconds === WEB_RUNTIME_LIMITS.commandTimeoutMs) + expect(globalThis.clearTimeout).toHaveBeenCalledWith(timer.timer); + } finally { + // The observation callback resolves naturally without cancelling its timer. + clearTimeout(timer.timer); + } +}; + +it("bounds a stalled WebSocket handshake and releases its socket", async () => { + const sockets = new Set(); + const server = createServer((socket) => { + sockets.add(socket); + socket.on("close", () => sockets.delete(socket)); + socket.resume(); + }); + await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") + throw new Error("Expected a TCP address"); + try { + const rejection = expect( + CdpConnection.connect( + `ws://127.0.0.1:${String(address.port)}/target`, + "inspect_web_page", + AbortSignal.timeout(10_000), + ), + ).rejects.toMatchObject({ reason: "timeout" }); + await vi.waitFor(() => expect(sockets.size).toBe(1), { timeout: 5_000 }); + expireTimer(WEB_RUNTIME_LIMITS.commandTimeoutMs); + await rejection; + await vi.waitFor(() => expect(sockets.size).toBe(0)); + } finally { + for (const socket of sockets) socket.destroy(); + await new Promise((resolve) => server.close(() => resolve())); + } +}); + +it.each(["Target.attachToTarget", "Page.enable", "Runtime.enable"])( + "times out %s after HTTP discovery and closes the target session", + async (hangOnMethod) => { + const browser = await startFakeCdpBrowser({ hangOnMethod }); + try { + const pending = new CdpBrowserProvider().inspectPage( + inspectWebPageInputSchema.parse({ + cdp_endpoint: browser.endpoint, + target_id: "allowed-page", + observation_ms: 0, + }), + { signal: AbortSignal.timeout(10_000) }, + ); + await vi.waitFor( + () => + expect(browser.commands.map(({ method }) => method)).toContain( + hangOnMethod, + ), + { timeout: 5_000 }, + ); + expireTimer(WEB_RUNTIME_LIMITS.commandTimeoutMs); + const result = await pending; + if (result.ok) throw new Error("Expected a target command timeout"); + expect(projectAnalysisError(result.error)).toMatchObject({ + code: "provider_timeout", + category: "timeout", + }); + expect(browser.commands.map(({ method }) => method)).toContain( + hangOnMethod, + ); + await vi.waitFor(() => expect(browser.activeConnectionCount()).toBe(0)); + } finally { + await browser.close(); + } + }, +); + +it("ignores a late timed-out reply and keeps following commands correlated", async () => { + const browser = await startFakeCdpBrowser({ hangOnMethod: "Page.enable" }); + const connection = await CdpConnection.connect( + browser.browserWebSocketUrl, + "inspect_web_page", + ); + try { + const rejection = expect( + connection.send( + "Page.enable", + {}, + undefined, + AbortSignal.timeout(10_000), + ), + ).rejects.toMatchObject({ reason: "timeout" }); + await vi.waitFor(() => expect(browser.commands).toHaveLength(1), { + timeout: 5_000, + }); + expireTimer(WEB_RUNTIME_LIMITS.commandTimeoutMs); + await rejection; + const command = browser.commands[0]; + if (command === undefined) + throw new Error("Expected the timed-out command"); + browser.emitRawMessage(JSON.stringify({ id: command.id, result: {} })); + await expect(connection.send("Runtime.enable")).resolves.toEqual({}); + } finally { + await connection.close(); + await browser.close(); + } +}); + +it.each(["success", "rejection", "abort", "disconnect"])( + "releases handshake and command timers on %s", + async (outcome) => { + const cleared = vi.mocked(globalThis.clearTimeout); + const browser = await startFakeCdpBrowser({ + ...(outcome === "abort" || outcome === "success" + ? { hangOnMethod: "Runtime.enable" } + : {}), + ...(outcome === "disconnect" ? { closeOnMethod: "Runtime.enable" } : {}), + }); + const controller = new AbortController(); + const removeAbort = vi.spyOn(controller.signal, "removeEventListener"); + const connection = await CdpConnection.connect( + browser.browserWebSocketUrl, + "inspect_web_page", + controller.signal, + ); + try { + const pending = connection.send( + outcome === "rejection" ? "Fixture.unmodeledMethod" : "Runtime.enable", + {}, + undefined, + controller.signal, + ); + const assertion = + outcome === "success" + ? expect(pending).resolves.toEqual({}) + : expect(pending).rejects.toMatchObject({ + _tag: + outcome === "abort" + ? "AnalysisCancelledError" + : "BrowserObservationError", + }); + if (outcome === "success") { + await vi.waitFor(() => expect(browser.commands).toHaveLength(1), { + timeout: 5_000, + }); + // A healthy reply may take longer than the former 100 ms test budget. + await new Promise((resolve) => setTimeout(resolve, 250)); + const command = browser.commands[0]; + if (command === undefined) + throw new Error("Expected a pending command"); + browser.emitRawMessage(JSON.stringify({ id: command.id, result: {} })); + } + if (outcome === "abort") controller.abort(); + await assertion; + const scheduled = vi.mocked(globalThis.setTimeout); + const deadlines = scheduled.mock.calls.flatMap((call, index) => + call[1] === WEB_RUNTIME_LIMITS.commandTimeoutMs + ? [scheduled.mock.results[index]?.value] + : [], + ); + expect(deadlines).toHaveLength(2); + for (const timer of deadlines) + expect(cleared).toHaveBeenCalledWith(timer); + expect(removeAbort).toHaveBeenCalledTimes(2); + } finally { + await connection.close(); + await browser.close(); + } + }, +); + +it.each(["Runtime.enable", "Debugger.enable"])( + "times out Inspector %s and preserves observations received before expiry", + async (hangOnMethod) => { + const inspector = await startFakeV8Inspector({ + targetUrl: "https://fixture.example/app.js", + hangOnMethod, + }); + try { + const pending = new V8InspectorProvider().observe( + { + inspector_endpoint: inspector.endpoint, + target_id: inspector.targetId, + observation_ms: 0, + }, + { signal: AbortSignal.timeout(10_000) }, + ); + await vi.waitFor( + () => + expect(inspector.commands.map(({ method }) => method)).toContain( + hangOnMethod, + ), + { timeout: 5_000 }, + ); + expireTimer(WEB_RUNTIME_LIMITS.commandTimeoutMs); + const result = await pending; + if (result.ok) throw new Error("Expected an Inspector command timeout"); + expect(projectAnalysisError(result.error)).toMatchObject({ + code: "provider_timeout", + category: "timeout", + }); + expect(result.error.partialObservation).toMatchObject({ + capture: { + events_observed: hangOnMethod === "Debugger.enable" ? 1 : 0, + }, + }); + } finally { + await inspector.close(); + } + }, +); + +it("keeps the chosen observation window independent of command deadlines", async () => { + const browser = await startFakeCdpBrowser(); + const observationMs = WEB_RUNTIME_LIMITS.commandTimeoutMs + 250; + try { + const pending = new CdpBrowserProvider().inspectPage( + inspectWebPageInputSchema.parse({ + cdp_endpoint: browser.endpoint, + target_id: "allowed-page", + observation_ms: observationMs, + }), + ); + await vi.waitFor( + () => expect(activeTimers(observationMs)).toHaveLength(1), + { + timeout: 5_000, + }, + ); + expect(activeTimers(WEB_RUNTIME_LIMITS.commandTimeoutMs)).toHaveLength(0); + expireTimer(observationMs); + const result = await pending; + expect(result.ok).toBe(true); + } finally { + await browser.close(); + } +}); diff --git a/tests/boundary/browser/electronActiveHook.test.ts b/tests/boundary/browser/electronActiveHook.test.ts index 99d4a0a3da..2709dc174b 100644 --- a/tests/boundary/browser/electronActiveHook.test.ts +++ b/tests/boundary/browser/electronActiveHook.test.ts @@ -1,6 +1,8 @@ -import { rm, writeFile } from "node:fs/promises"; -import { join } from "node:path"; +import { copyFile, mkdir, rm, writeFile } from "node:fs/promises"; +import { dirname, join } from "node:path"; import { spawn } from "node:child_process"; +import { createRequire } from "node:module"; +import { fileURLToPath } from "node:url"; import { afterEach, expect, it } from "vitest"; @@ -131,11 +133,10 @@ afterEach(async () => { it("retains active hook evidence within its event-retention budget", async () => { const root = await createTestTempDirectory("rea-electron-hook-"); temporary.push(root); - await writeFile(join(root, "electron.js"), electronModuleSource); + await writeElectronFixture(root); const script = join(root, "exercise.cjs"); await writeFile(script, activeCaptureSource); - const hook = join(process.cwd(), "scripts/electron-active-hook.cjs"); - const execution = await runNode(hook, script, root, root); + const execution = await runNode(script, root); expect(execution.code, execution.stderr).toBe(0); const result = JSON.parse( execution.stdout.trim().split("\n").at(-1) ?? "null", @@ -218,35 +219,138 @@ it("retains active hook evidence within its event-retention budget", async () => ); }); +it("classifies a dropped browser preload as preload configuration at capture", async () => { + const root = await createTestTempDirectory( + "rea-electron-hook-browser-preload-", + ); + temporary.push(root); + await writeElectronFixture(root); + const script = join(root, "browser-preload.cjs"); + await writeFile( + script, + `Object.defineProperty(process, "type", { value: "browser" }); +const electron = require("electron"); +new electron.BrowserWindow({ webPreferences: { preload: "p".repeat(6 * 1024 * 1024) } }); +const snapshot = globalThis.__reaElectronActiveSnapshot(); +process.stdout.write(JSON.stringify(snapshot) + "\\n"); +`, + ); + const execution = await runNode(script, root); + expect(execution.code, execution.stderr).toBe(0); + const snapshot = JSON.parse( + execution.stdout.trim().split("\n").at(-1) ?? "null", + ) as { + readonly dropped_event_families: readonly { + readonly family: string; + readonly count: number; + }[]; + readonly dropped_event_roles: readonly { + readonly role: string; + readonly count: number; + }[]; + }; + expect(snapshot.dropped_event_families).toContainEqual({ + family: "preload-configuration", + count: 1, + }); + expect(snapshot.dropped_event_roles).toContainEqual({ + role: "browser", + count: 1, + }); +}); + +it("isolates hook and exercise from ancestor Electron packages", async () => { + const root = await createTestTempDirectory("rea-electron-hook-ancestor-"); + temporary.push(root); + await writeElectronFixture(root, 'module.exports = "/ambient/Electron";'); + const ambientHook = await stageHook(join(root, "producer", "scripts")); + expect(createRequire(ambientHook)("electron")).toBe("/ambient/Electron"); + + const fixture = join(root, "fixture"); + await writeElectronFixture(fixture); + const script = join(fixture, "exercise.cjs"); + await writeFile( + script, + `const { BrowserWindow } = require("electron"); +const window = new BrowserWindow({ webPreferences: {} }); +process.stdout.write(JSON.stringify({ + hook_error: globalThis.__reaElectronActiveSnapshot().hook_error, + window_open_handler: typeof window.webContents.windowOpenHandler, +}) + "\\n"); +`, + ); + const result = await runNode(script, fixture, ambientHook); + expect(result.code, result.stderr).toBe(0); + expect(JSON.parse(result.stdout)).toEqual({ + hook_error: false, + window_open_handler: "function", + }); +}); + it("preserves fatal process termination after recording uncaught exceptions", async () => { const root = await createTestTempDirectory("rea-electron-hook-fatal-"); temporary.push(root); + await writeElectronFixture( + root, + 'throw new Error("controlled Electron unavailable");', + ); const script = join(root, "fatal.cjs"); - await writeFile(script, 'throw new Error("fatal-hook-secret");\n'); - const hook = join(process.cwd(), "scripts/electron-active-hook.cjs"); - const result = await runNode(hook, script, root, root); + await writeFile( + script, + 'process.stdout.write(String(globalThis.__reaElectronActiveSnapshot().hook_error) + "\\n");\nthrow new Error("fatal-hook-secret");\n', + ); + const result = await runNode(script, root); expect(result.code).not.toBe(0); + expect(result.stdout.trim()).toBe("true"); expect(result.stderr).toContain("fatal-hook-secret"); }); -const runNode = ( - hook: string, +const writeElectronFixture = async ( + root: string, + source = electronModuleSource, +) => { + const directory = join(root, "node_modules", "electron"); + await mkdir(directory, { recursive: true }); + await writeFile( + join(directory, "package.json"), + JSON.stringify({ name: "electron", main: "index.cjs" }), + ); + await writeFile(join(directory, "index.cjs"), source); +}; + +const stageHook = async ( + root: string, + source = fileURLToPath( + new URL("../../../scripts/electron-active-hook.cjs", import.meta.url), + ), +) => { + await mkdir(root, { recursive: true }); + const hook = join(root, "electron-active-hook.cjs"); + await Promise.all([ + copyFile(source, hook), + copyFile( + join(dirname(source), "electron-active-hook-boundaries.cjs"), + join(root, "electron-active-hook-boundaries.cjs"), + ), + ]); + return hook; +}; + +const runNode = async ( script: string, - moduleRoot: string, - cwd: string, + root: string, + sourceHook?: string, ): Promise<{ readonly code: number | null; readonly stdout: string; readonly stderr: string; -}> => - new Promise((resolve, reject) => { +}> => { + const hook = await stageHook(root, sourceHook); + return new Promise((resolve, reject) => { const child = spawn(process.execPath, ["-r", hook, script], { - cwd, - env: { - ...process.env, - NODE_PATH: moduleRoot, - }, + cwd: root, + env: process.env, stdio: ["ignore", "pipe", "pipe"], }); const stdout: Buffer[] = []; @@ -262,3 +366,4 @@ const runNode = ( }), ); }); +}; diff --git a/tests/boundary/browser/historicalCaptureDecoder.test.ts b/tests/boundary/browser/historicalCaptureDecoder.test.ts index e02524f8d4..240c04d43a 100644 --- a/tests/boundary/browser/historicalCaptureDecoder.test.ts +++ b/tests/boundary/browser/historicalCaptureDecoder.test.ts @@ -365,6 +365,7 @@ it("cancels after snapshot acquisition before launch and removes private input d it.each([ { reason: "input-limit", tag: "AnalysisInputError" }, + { reason: "expanded-limit", tag: "AnalysisInputError" }, { reason: "resource-limit", tag: "AnalysisResourceConstraintError" }, { reason: "decoder", tag: "ProviderAdapterError" }, { reason: "limit", tag: "AnalysisOutputError" }, @@ -425,6 +426,22 @@ it.each([ ], }, }); + } else if (reason === "expanded-limit") { + expect(projected).toMatchObject({ + category: "invalid_input", + details: { + issues: [ + { + path: ["capture_path", "/extension/child"], + reason: "out_of_range", + expected: { + maximum_expanded_capture_bytes: + WEB_NETWORK_CAPTURE_LIMITS.inputBytes, + }, + }, + ], + }, + }); } else if (tag === "ProviderAdapterError") { expect(projected).toMatchObject({ details: { diff --git a/tests/boundary/browser/historicalWebCapture.test.ts b/tests/boundary/browser/historicalWebCapture.test.ts index 781172cd34..ee0b9d14cb 100644 --- a/tests/boundary/browser/historicalWebCapture.test.ts +++ b/tests/boundary/browser/historicalWebCapture.test.ts @@ -1,5 +1,5 @@ import { createHash } from "node:crypto"; -import { mkdtemp, writeFile, rm } from "node:fs/promises"; +import { mkdtemp, writeFile, rm, readFile } from "node:fs/promises"; import { join } from "node:path"; import { tmpdir } from "node:os"; import { expect, it, onTestFinished } from "vitest"; @@ -44,6 +44,105 @@ const cli = async ( }; import { historicalHar } from "../../fixtures/historicalHar.js"; +it.each([ + { encoded: "1e3", literal: "1000" }, + { encoded: "0.0000001", literal: "1e-7" }, +])( + "excludes normalized HAR numbers through CLI and MCP: $encoded", + async ({ encoded, literal }) => { + const root = await createTestTempDirectory("rea-har-normalized-number-"); + const path = join(root, "capture.har"); + const original = JSON.stringify(historicalHar()) + .replace('"log":{', `"_number":${encoded},"log":{`) + .replace('"cache":{}', `"cache":{},"_number":${encoded}`); + await writeFile(path, original); + const { call } = await connectLocalToolsMcp(); + for (const marked of [false, true]) { + const fromCli = await cli( + path, + "har", + marked ? ["--sensitive-value", literal] : [], + ); + expect(fromCli.ok, fromCli.stdout).toBe(true); + const evidence = parseEvidence(JSON.parse(fromCli.stdout)); + const response = await call("inspect_web_network_capture", { + capture_path: path, + format: "har", + ...(marked ? { sensitive_values: [literal] } : {}), + }); + expect(response.isError, JSON.stringify(response)).not.toBe(true); + const mcp = parseEvidence(response.structuredContent); + expect(mcp.evidence_id).toBe(evidence.evidence_id); + expect(mcp.normalized_result).toEqual(evidence.normalized_result); + const capture = webNetworkCaptureSchema.parse(evidence.normalized_result); + for (const projection of [capture.container, ...capture.records]) { + expect(projection.reported).toMatchObject({ + _number: marked ? null : Number(encoded), + }); + const numbers = projection.numeric_literals.filter( + (number) => number.pointer === "/_number", + ); + expect(numbers).toEqual( + marked + ? [] + : [ + { + pointer: "/_number", + producer_type: "json-number", + literal: encoded, + }, + ], + ); + if (marked) + expect(projection.redactions).toContainEqual({ + pointer: "/_number", + reason: "explicit-sensitive-value", + }); + } + } + expect(await readFile(path, "utf8")).toBe(original); + }, +); + +it("retains Playwright IPv6 spelling through public CLI and MCP workflows", async () => { + const root = await createTestTempDirectory("rea-har-ipv6-"); + const path = join(root, "capture.har"); + const original = JSON.stringify(historicalHar()) + .replace( + '"name":"rea-source-owned-fixture","version":"1"', + '"name":"Playwright","version":"1.58.2"', + ) + .replace( + '"startedDateTime":', + '"serverIPAddress":"[2001:db8::1]","startedDateTime":', + ); + await writeFile(path, original); + const { call } = await connectLocalToolsMcp(); + const fromCli = await cli(path, "har"); + expect(fromCli.ok, fromCli.stdout).toBe(true); + const cliEvidence = parseEvidence(JSON.parse(fromCli.stdout)); + const response = await call("inspect_web_network_capture", { + capture_path: path, + format: "har", + }); + expect(response.isError, JSON.stringify(response)).not.toBe(true); + const mcpEvidence = parseEvidence(response.structuredContent); + expect(mcpEvidence.evidence_id).toBe(cliEvidence.evidence_id); + expect(mcpEvidence.normalized_result).toEqual(cliEvidence.normalized_result); + const capture = webNetworkCaptureSchema.parse(cliEvidence.normalized_result); + expect(capture.records[0]).toMatchObject({ + reported: { + serverIPAddress: "[2001:db8::1]", + }, + }); + expect( + capture.records[0]?.limitations.some((limitation) => + limitation.includes("bracketed IPv6 serverIPAddress"), + ), + ).toBe(true); + expect(await readFile(path, "utf8")).toBe(original); +}); + it("excludes declared binary lengths through CLI, MCP and retained capture pages", async () => { const root = await createTestTempDirectory("rea-har-byte-length-"); const path = join(root, "capture.har"); diff --git a/tests/boundary/browser/runtimeFixtureCleanup.test.ts b/tests/boundary/browser/runtimeFixtureCleanup.test.ts index 4aafa317f9..49dafe3567 100644 --- a/tests/boundary/browser/runtimeFixtureCleanup.test.ts +++ b/tests/boundary/browser/runtimeFixtureCleanup.test.ts @@ -44,35 +44,3 @@ it.skipIf(process.platform === "win32")( expect(await readdir(root)).toEqual([]); }, ); - -it("attempts every fixture cleanup and retains the primary capture failure", async () => { - const helper = new URL( - "../../../scripts/lib/browser-runtime-fixture-lifecycle.mjs", - import.meta.url, - ); - const script = ` - const { closeRuntimeFixtureResources } = await import(${JSON.stringify(helper.href)}); - const seen = []; - try { - await closeRuntimeFixtureResources([ - async () => { seen.push('first'); throw new Error('first cleanup failed'); }, - async () => { seen.push('second'); throw new Error('second cleanup failed'); }, - async () => { seen.push('third'); }, - ], new Error('capture failed')); - throw new Error('Expected aggregate cleanup failure'); - } catch (error) { - if (!(error instanceof AggregateError)) throw error; - console.log(JSON.stringify({ seen, errors: error.errors.map(item => item.message) })); - } - `; - const { stdout } = await promisify(execFile)( - process.execPath, - ["--input-type=module", "-e", script], - { timeout: 10_000 }, - ); - const result: unknown = JSON.parse(stdout); - expect(result).toEqual({ - seen: ["first", "second", "third"], - errors: ["capture failed", "first cleanup failed", "second cleanup failed"], - }); -}); diff --git a/tests/boundary/browser/sazWebCapture.test.ts b/tests/boundary/browser/sazWebCapture.test.ts index 69eca33b84..9a53ff65a8 100644 --- a/tests/boundary/browser/sazWebCapture.test.ts +++ b/tests/boundary/browser/sazWebCapture.test.ts @@ -10,8 +10,12 @@ import { Ajv2020 } from "ajv/dist/2020.js"; import { expect, it as runtimeTest } from "vitest"; import { z } from "zod"; import { parseEvidence } from "../../../src/domain/evidence.js"; -import { webNetworkCaptureSchema } from "../../../src/domain/webNetworkCapture.js"; +import { + WEB_NETWORK_CAPTURE_LIMITS, + webNetworkCaptureSchema, +} from "../../../src/domain/webNetworkCapture.js"; import { connectLocalToolsMcp } from "../../fixtures/localToolsMcp.js"; +import { parseMcpToolError } from "../../fixtures/mcpToolError.js"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; import { @@ -339,6 +343,51 @@ it("keeps CRLF separators inside a body after LF-only HTTP headers", async () => expect(await readFile(path)).toEqual(bytes); }); +it("validates SAZ XML declarations against archive member bytes at the CLI boundary", async () => { + const withMetadata = (metadata: Buffer): [string, Buffer][] => + members().map(([name, bytes]) => + name === "raw/0042_m.xml" ? [name, metadata] : [name, bytes], + ); + const utf8 = Buffer.from( + '', + ); + const utf16le = Buffer.concat([ + Buffer.from([0xff, 0xfe]), + Buffer.from( + '', + "utf16le", + ), + ]); + for (const metadata of [utf8, utf16le]) { + const result = await cli( + await fixture(await archive(withMetadata(metadata))), + ); + expect( + webNetworkCaptureSchema.parse(result.normalized_result).records, + ).toHaveLength(2); + } + const mismatch = Buffer.from( + '', + ); + const rejected = await cliResponse( + await fixture(await archive(withMetadata(mismatch))), + ["--record", "1"], + ); + expect(rejected.ok).toBe(false); + expect(JSON.parse(rejected.stdout)).toMatchObject({ + category: "invalid_input", + details: { + issues: [ + expect.objectContaining({ + message: expect.stringContaining( + "SAZ metadata encoding declaration disagrees with its bytes", + ), + }), + ], + }, + }); +}); + it("excludes explicit literals from XML, HTTP byte views, member coordinates and binary sidecars", async () => { const path = await fixture(await archive()); const result = await cli(path, [ @@ -478,12 +527,42 @@ it.each([ expect(JSON.parse(result.stdout).category).toBe( ["encrypted", "unknown-member", "dtd"].includes(variant) ? "unsupported_provider" - : variant === "expanded-limit" - ? "execution_failure" - : "invalid_input", + : "invalid_input", ); - if (variant === "expanded-limit") - expect(JSON.parse(result.stdout).code).toBe("unreadable_output"); + // Expanded size is a caller-correctable input bound, not a decoder failure. + if (variant === "expanded-limit") { + const error = JSON.parse(result.stdout); + expect(error.code).toBe("invalid_request"); + expect(error.details.issues).toEqual([ + expect.objectContaining({ + path: ["capture_path", ""], + reason: "out_of_range", + expected: { + maximum_expanded_capture_bytes: WEB_NETWORK_CAPTURE_LIMITS.inputBytes, + }, + }), + ]); + const { call } = await connectLocalToolsMcp(); + const response = await call("inspect_web_network_capture", { + capture_path: path, + format: "saz", + record_ordinals: [0], + }); + expect(response.isError).toBe(true); + const mcpFailure = parseMcpToolError(response); + expect(mcpFailure).toMatchObject({ + error: { + code: error.code, + category: error.category, + details: error.details, + }, + }); + expect(error).not.toHaveProperty("normalized_result"); + expect(error.details).not.toHaveProperty("partial_observation"); + expect(mcpFailure).not.toHaveProperty("normalized_result"); + expect(mcpFailure.error.details).not.toHaveProperty("partial_observation"); + expect(await readFile(path)).toEqual(bytes); + } expect(JSON.stringify(result)).not.toContain("malformed-private-input"); }); diff --git a/tests/boundary/browser/webCaptureDiff.test.ts b/tests/boundary/browser/webCaptureDiff.test.ts index 99f347c815..955b63714d 100644 --- a/tests/boundary/browser/webCaptureDiff.test.ts +++ b/tests/boundary/browser/webCaptureDiff.test.ts @@ -3,6 +3,7 @@ import { afterEach, describe, expect, it } from "vitest"; import { CdpBrowserProvider } from "../../../src/browser/CdpBrowserProvider.js"; import { inspectWebPageInputSchema } from "../../../src/domain/browserObservation.js"; import type { WebPageInspection } from "../../../src/domain/browserObservationSchemas.js"; +import { discoverWebMcpToolsInputSchema } from "../../../src/domain/webMcpDiscovery.js"; import { compareWebCaptures } from "../../../src/domain/webCaptureDiff.js"; import { compareWebCapturesInputSchema, @@ -202,6 +203,64 @@ afterEach(async () => { }); describe("web capture diff", () => { + it.each([ + { missing: "before", changed: false, expected: "unknown" }, + { missing: "after", changed: false, expected: "unknown" }, + { missing: "before", changed: true, expected: "changed" }, + { missing: "after", changed: true, expected: "changed" }, + { missing: "neither", changed: false, expected: "changed" }, + ] as const)( + "preserves consequential uncertainty: $missing, other change $changed", + async ({ missing, changed, expected }) => { + const browser = await startFakeCdpBrowser({ webMcpTools: true }); + browsers.push(browser); + const provider = new CdpBrowserProvider(); + const request = { + cdp_endpoint: browser.endpoint, + allowed_origins: [browser.allowedOrigin], + target_id: "allowed-page", + observation_ms: 0, + }; + const inspection = await provider.inspectPage( + inspectWebPageInputSchema.parse(request), + ); + const discovery = await provider.discoverWebMcpTools( + discoverWebMcpToolsInputSchema.parse(request), + ); + if (!inspection.ok) throw inspection.error; + if (!discovery.ok) throw discovery.error; + const before = structuredClone(discovery.value); + const after = structuredClone(discovery.value); + markSectionsComplete(before, ["webmcp_tools"]); + markSectionsComplete(after, ["webmcp_tools"]); + const left = before.tools.items[0]; + const right = after.tools.items[0]; + if (left === undefined || right === undefined) + throw new Error("missing WebMCP registration fixture"); + left.annotations.consequential = true; + right.annotations.consequential = false; + if (missing === "before") delete left.annotations.consequential; + if (missing === "after") delete right.annotations.consequential; + if (changed) right.description += " changed"; + const result = compareWebCaptures( + compareWebCapturesInputSchema.parse({ + before: { inspection: inspection.value, webmcp: before }, + after: { inspection: inspection.value, webmcp: after }, + }), + ); + expect(result.dimensions.webmcp).toMatchObject({ + status: expected, + total_changes: expected === "changed" ? 1 : 0, + }); + if (expected === "changed") + expect(result.dimensions.webmcp.changes[0]).toMatchObject({ + change: "modified", + }); + }, + ); +}); + +describe("web capture dimension uncertainty", () => { it("reports stable observed changes while preserving unknown dimensions", async () => { const browser = await startFakeCdpBrowser(); browsers.push(browser); @@ -625,6 +684,44 @@ describe("web capture WebSocket payload byte coverage", () => { }); describe("web capture diff", () => { + it("reports request membership changes when selected body shapes are unavailable", async () => { + const browser = await startFakeCdpBrowser(); + browsers.push(browser); + const captured = await new CdpBrowserProvider().inspectPage( + inspectWebPageInputSchema.parse({ + cdp_endpoint: browser.endpoint, + allowed_origins: [browser.allowedOrigin], + target_id: "allowed-page", + observation_ms: 0, + }), + ); + if (!captured.ok) throw captured.error; + const before = structuredClone(captured.value); + const after = structuredClone(captured.value); + const original = before.network.requests[0]; + if (original === undefined) throw new Error("Missing request fixture"); + after.network.requests.push({ + ...structuredClone(original), + request_id: `${original.request_id}-added`, + url: `${browser.allowedOrigin}/added`, + }); + for (const inspection of [before, after]) { + markSectionsComplete(inspection, [ + "network_requests", + "websocket_connections", + "websocket_frames", + ]); + for (const request of inspection.network.requests) + request.body_shapes.status = "unavailable"; + } + + expect(compareNetwork(before, after)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [expect.objectContaining({ change: "added" })], + }); + }); + it.each(["status", "mime_type"] as const)( "keeps an independently unavailable HTTP %s unknown", async (field) => { @@ -772,6 +869,89 @@ describe("web capture redirect comparison", () => { changes: [expect.objectContaining({ change: "modified" })], }); }); + + it.each([ + ["status", 302], + ["mime_type", "text/plain"], + ["encoded_data_length", 12], + ] as const)( + "keeps unavailable redirect %s evidence unknown without suppressing known changes", + async (field, knownValue) => { + const browser = await startFakeCdpBrowser(); + browsers.push(browser); + const captured = await new CdpBrowserProvider().inspectPage( + inspectWebPageInputSchema.parse({ + cdp_endpoint: browser.endpoint, + allowed_origins: [browser.allowedOrigin], + target_id: "allowed-page", + observation_ms: 0, + }), + ); + if (!captured.ok) throw captured.error; + const before = structuredClone(captured.value); + const beforeRequest = before.network.requests[0]; + if (beforeRequest === undefined) + throw new Error("Missing network request"); + beforeRequest.redirects = [ + { + url: `${browser.allowedOrigin}/prior`, + response_url: `${browser.allowedOrigin}/prior`, + method: "GET", + resource_type: "Fetch", + status: 302, + mime_type: "text/plain", + encoded_data_length: 12, + request_timestamp: 1, + redirect_event_timestamp: 2, + }, + ]; + markSectionsComplete(before, [ + "network_requests", + "websocket_connections", + "websocket_frames", + ]); + + const compare = (left: typeof before, right: typeof before) => + compareWebCaptures( + compareWebCapturesInputSchema.parse({ + before: { inspection: left }, + after: { inspection: right }, + }), + ).dimensions.network; + for (const missingOnRight of [false, true]) { + const known = structuredClone(before); + const missing = structuredClone(before); + const knownHop = known.network.requests[0]?.redirects?.[0]; + const missingHop = missing.network.requests[0]?.redirects?.[0]; + if (knownHop === undefined || missingHop === undefined) + throw new Error("Missing redirect fixture"); + Reflect.set(knownHop, field, knownValue); + Reflect.set(missingHop, field, null); + const result = missingOnRight + ? compare(known, missing) + : compare(missing, known); + expect(result).toMatchObject({ + status: "unknown", + total_changes: 0, + reason: expect.stringContaining("redirect"), + }); + expect(compare(missing, missing)).toMatchObject({ + status: "unknown", + total_changes: 0, + reason: expect.stringContaining("redirect"), + }); + } + + const changed = structuredClone(before); + const changedHop = changed.network.requests[0]?.redirects?.[0]; + if (changedHop === undefined) throw new Error("Missing redirect fixture"); + changedHop.status = 307; + expect(compare(before, changed)).toMatchObject({ + status: "changed", + changes: [expect.objectContaining({ change: "modified" })], + }); + }, + ); }); describe("web capture diff incomplete inventories", () => { diff --git a/tests/boundary/browser/webSourceMapFetcher.test.ts b/tests/boundary/browser/webSourceMapFetcher.test.ts index 5fab66f588..782ff8cf07 100644 --- a/tests/boundary/browser/webSourceMapFetcher.test.ts +++ b/tests/boundary/browser/webSourceMapFetcher.test.ts @@ -988,6 +988,41 @@ describe("source-map original dependency syntax", () => { result.items[0]?.original_module_edges.map(({ specifier }) => specifier), ).toEqual(["./real.js", "./dynamic.js", "./common.cjs"]); }); + + it("reads expression-free template module specifiers as literals", async () => { + const text = [ + "import(`./dep.js`);", + "import(`./dep.${name}.js`);", + "require(`./common.cjs`);", + "require(`./${name}.cjs`);", + ].join("\n"); + const result = await fetchWebSourceMaps([request], input(), undefined, { + fetch: () => + Promise.resolve( + new Response( + JSON.stringify({ + version: 3, + names: [], + sources: ["main.ts"], + sourcesContent: [text], + mappings: "AAAA", + }), + ), + ), + }); + const item = result.items[0]; + if (item?.status !== "included") throw new Error("map not included"); + expect(item.original_module_edges).toEqual([ + expect.objectContaining({ + kind: "dynamic_import", + specifier: "./dep.js", + resolved_source: `${origin}/assets/dep.js`, + }), + expect.objectContaining({ kind: "require", specifier: "./common.cjs" }), + ]); + expect(item.original_sources[0]?.artifact?.text).toBe(text); + expect(item.mappings).toHaveLength(1); + }); }); describe("source-map original syntax boundaries", () => { @@ -1067,6 +1102,21 @@ describe("source-map dependency coverage of hard-to-parse sources", () => { 'import lib = require("./lib.js");', ["./lib.js"], ], + [ + "whole-statement type-only import", + 'import type { Value } from "./types.js";', + [], + ], + [ + "whole-statement type-only export", + 'export type { Value } from "./types.js";', + [], + ], + [ + "inline type import retains module side effects", + 'import { type Value } from "./types.js";', + ["./types.js"], + ], [ "a local import alias", 'import lib = localAlias;\nimport "./a.js";', @@ -1080,6 +1130,34 @@ describe("source-map dependency coverage of hard-to-parse sources", () => { ).toEqual(expected); }); + it("selects parser mode from the original source URL pathname", async () => { + const result = await fetchWebSourceMaps([request], input(), undefined, { + fetch: () => + Promise.resolve( + new Response( + JSON.stringify({ + version: 3, + names: [], + sources: ["main.ts?revision=2#view.tsx"], + sourcesContent: [`import "./dep.ts"; const value = 1;`], + mappings: "AAAA", + }), + ), + ), + }); + expect(result.items[0]?.status).toBe("included"); + expect(result.items[0]?.limitation).toBeNull(); + expect(result.items[0]?.original_sources[0]?.source).toBe( + `${origin}/assets/main.ts?revision=2#view.tsx`, + ); + expect(result.items[0]?.original_module_edges).toEqual([ + expect.objectContaining({ + specifier: "./dep.ts", + resolved_source: `${origin}/assets/dep.ts`, + }), + ]); + }); + it("keeps recovered edges from a partly recovered source and reports them as partial", async () => { const result = await fetchWithSources([ 'with (scope) { require("./c.js"); }\nrequire("./b.js");\nimport "./a.js";', diff --git a/tests/boundary/captures/captureProvider.test.ts b/tests/boundary/captures/captureProvider.test.ts index 881b43e63e..30281bed3e 100644 --- a/tests/boundary/captures/captureProvider.test.ts +++ b/tests/boundary/captures/captureProvider.test.ts @@ -53,6 +53,8 @@ else { if(readFileSync('capture', 'utf8') !== 'recording bytes') throw Error('wr }, async (input) => { calls++; + if (mode === "launch-failure" && !input.arguments.includes("--version")) + throw Object.assign(new Error("spawn E2BIG"), { code: "E2BIG" }); if (mode === "file-limit" && !input.arguments.includes("--version")) { const stdout = { text: "", bytes: 0, observedBytes: 0 }; const stderr = { @@ -255,6 +257,21 @@ it.each([ }, ); +it.each(["capture", "fields"] as const)( + "does not attribute the version probe's output to a %s command that fails to launch", + async (operation) => { + const f = await fixture("launch-failure"); + const result = + operation === "capture" + ? await f.service.inspect({ path: f.path, fields: ["ip.addr"] }) + : await f.service.inspectFields({ kind: "fields" }); + if (result.ok) throw new Error("Expected launch failure"); + expect(f.calls()).toBe(2); + expect(result.error.message).toContain("spawn E2BIG"); + expect(result.error.capturedOutput).toBeUndefined(); + }, +); + it("retains a completed observation and cleanup ownership for retry", async () => { const f = await fixture("ok", true); const result = await f.service.inspect({ path: f.path, fields: ["ip.addr"] }); diff --git a/tests/boundary/cli/adbCli.test.ts b/tests/boundary/cli/adbCli.test.ts index 71309e8787..cda871ae34 100644 --- a/tests/boundary/cli/adbCli.test.ts +++ b/tests/boundary/cli/adbCli.test.ts @@ -1,6 +1,8 @@ -import { chmod, lstat, readFile } from "node:fs/promises"; +import { chmod, lstat, readFile, unlink } from "node:fs/promises"; import { expect, it, onTestFinished } from "vitest"; +import { z } from "zod"; +import { toolContract } from "../../../src/contracts/toolContracts.js"; import { expectConfigurationRefusal, serveCli as serve, @@ -42,6 +44,158 @@ it("validates device serials before contacting any device", async () => { }); }); +it.each(["string:key", ":", "string"])( + "classifies malformed activity extra %j before contacting adb", + async (extra) => { + const result = await serve( + [ + "start-adb-activity", + "emulator-5554", + "android.intent.action.VIEW", + "--extra", + extra, + "--format", + "json", + ], + { REA_ADB_PATH: "/path/that/does/not/exist" }, + ); + + const error = JSON.parse(result.stdout); + expect(error).toMatchObject({ + code: "invalid_request", + category: "invalid_input", + retryable: true, + remediation: { action: expect.any(String) }, + }); + expect(error.remediation.action).not.toBe(""); + expect(error.message).not.toBe(""); + }, +); + +it.each([ + ["bogus:key:value", "invalid_request"], + ["string:key:value", "capability_unavailable"], +] as const)( + "preserves downstream validation for well-delimited activity extra %j", + async (extra, code) => { + const result = await serve( + [ + "start-adb-activity", + "emulator-5554", + "android.intent.action.VIEW", + "--extra", + extra, + "--format", + "json", + ], + { REA_ADB_PATH: "/path/that/does/not/exist" }, + ); + + expect(JSON.parse(result.stdout)).toMatchObject({ code }); + }, +); + +it.skipIf(process.platform === "win32")( + "preserves requested reinstall mode without claiming prior package state", + async () => { + const workspace = await createTestWorkspace("rea-adb-install-state-"); + onTestFinished(() => removeTestWorkspace(workspace.root)); + const apk = await workspace.write("app.apk", "apk"); + const adb = await workspace.write( + "adb", + `#!/bin/sh +root=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) +printf '%s\\n' "$*" >> "$root/calls" +if [ "$1" = version ]; then + printf 'Android Debug Bridge version 1.0.41\\nVersion 34.0.5-cli\\nInstalled as %s\\n' "$0" + exit 0 +fi +if [ "$1" = -s ] && [ "$3" = install ]; then + if [ -f "$root/installed" ]; then + if [ "$4" != -r ]; then + printf 'Failure [INSTALL_FAILED_ALREADY_EXISTS]\\n' + exit 0 + fi + printf 'replacement\\n' >> "$root/events" + else + printf 'fresh-install\\n' >> "$root/events" + fi + : > "$root/installed" + printf 'Performing Streamed Install\\nSuccess\\n' + exit 0 +fi +exit 1 +`, + ); + await chmod(adb, 0o755); + + const installEnvelope = z.object({ + ok: z.literal(true), + data: toolContract("install_adb_package").outputSchema, + }); + const install = async (replace: boolean) => { + const result = await serve( + [ + "install-adb-package", + "emulator-5554", + apk, + ...(replace ? ["--replace"] : []), + "--format", + "json", + "--full-output", + ], + { REA_ADB_PATH: adb }, + ); + expect(result.exitCode).toBe(0); + return installEnvelope.parse(JSON.parse(result.stdout)); + }; + + const freshReplace = await install(true); + expect(freshReplace).toMatchObject({ + ok: true, + data: { + parameters: { replace: true }, + normalized_result: { replace_requested: true, replaced: null }, + confidence: "observed", + }, + }); + expect(freshReplace.data.limitations.join("\n")).toContain( + "Whether an existing package was replaced is unknown", + ); + + const existingReplace = await install(true); + expect(existingReplace).toMatchObject({ + ok: true, + data: { + parameters: { replace: true }, + normalized_result: { replace_requested: true, replaced: null }, + }, + }); + + await unlink(workspace.path("installed")); + const freshDefault = await install(false); + expect(freshDefault).toMatchObject({ + ok: true, + data: { + parameters: { replace: false }, + normalized_result: { replace_requested: false, replaced: null }, + }, + }); + expect(await readFile(workspace.path("events"), "utf8")).toBe( + "fresh-install\nreplacement\nfresh-install\n", + ); + const calls = await readFile(workspace.path("calls"), "utf8"); + expect(calls.split("\n").filter((line) => line !== "")).toEqual([ + "version", + `-s emulator-5554 install -r ${apk}`, + "version", + `-s emulator-5554 install -r ${apk}`, + "version", + `-s emulator-5554 install ${apk}`, + ]); + }, +); + it("requires the output directory option on pull", async () => { const result = await serve([ "pull-adb-package", diff --git a/tests/boundary/cli/analysisViewCli.test.ts b/tests/boundary/cli/analysisViewCli.test.ts index 4e63358d7d..d4b1f32224 100644 --- a/tests/boundary/cli/analysisViewCli.test.ts +++ b/tests/boundary/cli/analysisViewCli.test.ts @@ -20,6 +20,56 @@ import { ghidraFunctionDossier } from "../../../src/domain/ghidraValues.fixture. import { createEvidenceBundle } from "../../../src/domain/evidenceBundle.js"; import { inspectEvidenceBundle } from "../../../src/application/investigation/InspectEvidenceBundle.js"; +cliTest( + "projects opted-in query names through compiled CLI without query values", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-query-names-cli-"); + const path = join(root, "view.json"); + const capture = analysisViewCaptureFixture(); + capture.records = capture.records.slice(0, 2); + capture.total_records = 2; + capture.records.forEach((record, index) => { + record.reported = { + request: { + method: "GET", + url: + index === 0 + ? "https://api.test/products?category=books&page=1&category=repeat" + : "https://api.test/products?category=games&sort=price#ignored", + }, + }; + }); + await writeFile( + path, + JSON.stringify({ + source: { + kind: "inline", + evidence: analysisViewCaptureEvidence(capture), + }, + view: { kind: "api-surface", include_query_parameter_names: true }, + }), + ); + const result = await cli.run({ + arguments: ["inspect-analysis-view", path, "--json"], + }); + expect(result.exitCode).toBe(0); + const evidence = parseEvidence(result.json); + expect(evidence.normalized_result).toMatchObject({ + groups: [ + { + query_parameter_names: [ + { name: "category", record_ordinals: [0, 1] }, + { name: "page", record_ordinals: [0] }, + { name: "sort", record_ordinals: [1] }, + ], + }, + ], + }); + for (const value of ["books", "games", "repeat", "price", "ignored"]) + expect(result.stdout).not.toContain(value); + }, +); + cliTest( "projects portable artifact inventory after the original ZIP is removed", async ({ cli }) => { diff --git a/tests/boundary/cli/analyzeAppBundlePathCli.test.ts b/tests/boundary/cli/analyzeAppBundlePathCli.test.ts new file mode 100644 index 0000000000..eb4d020f0c --- /dev/null +++ b/tests/boundary/cli/analyzeAppBundlePathCli.test.ts @@ -0,0 +1,52 @@ +import { mkdir, writeFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { describe, expect } from "vitest"; + +import { thinMach } from "../../../src/domain/binaryTarget.fixture.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { cliTest } from "../../support/cli/cliFixture.js"; + +// An unknown standing provider stops native analysis before any engine runs. +const environment = { + REA_LOG_LEVEL: "silent", + REA_ANALYSIS_PROVIDER: "missing-provider", +} as const; + +const fixtureApp = async () => { + const directory = await createTestTempDirectory("rea-analyze-app-path-"); + const contents = join(directory, "Sample.app", "Contents"); + await mkdir(join(contents, "MacOS"), { recursive: true }); + await writeFile( + join(contents, "MacOS/Sample"), + thinMach(0xfeedfacf, 0x100000c), + ); + await writeFile( + join(contents, "Info.plist"), + "CFBundleExecutableSample", + ); + return directory; +}; + +describe("analyze app bundle path spellings", () => { + cliTest.for(["Sample.app", "Sample.app/", "Sample.app/.", "Sample.app//"])( + "routes %s to native provider selection", + async (path, { cli }) => { + const cwd = await fixtureApp(); + const result = await cli.run({ + arguments: ["analyze", path, "--json"], + cwd, + environment, + }); + expect(result.exitCode).toBe(1); + expect(result.json).toMatchObject({ + code: "capability_unavailable", + details: { + operation: "open_binary", + selection_reason: "unknown_provider", + requested_provider_id: "missing-provider", + }, + }); + }, + ); +}); diff --git a/tests/boundary/cli/browserCli.test.ts b/tests/boundary/cli/browserCli.test.ts index 351d075125..86a9db75d5 100644 --- a/tests/boundary/cli/browserCli.test.ts +++ b/tests/boundary/cli/browserCli.test.ts @@ -1,6 +1,7 @@ import { afterEach, describe, expect, it } from "vitest"; import { runCliJson as runCli } from "../../fixtures/cliJsonProcess.js"; +import { DEFAULT_BROWSER_SCENARIO_CAPTURE_LIMITS } from "../../../src/domain/browserScenarioCaptureLimits.js"; import { startFakeCdpBrowser, type FakeCdpBrowser, @@ -183,6 +184,51 @@ it( INTEGRATION_TEST_TIMEOUT_MS, ); +it( + "compares redirect metadata uncertainty through the public CLI", + async () => { + const browser = await startFakeCdpBrowser(); + browsers.push(browser); + const inspected = await runCli( + [ + "inspect-web-page", + browser.endpoint, + "allowed-page", + "--observation-ms", + "0", + "--json", + ], + process.env, + ); + const inspection = normalizedResult(inspected); + const missingVsKnown = await compareCapturesCli( + redirectCapture(inspection, browser.allowedOrigin, null), + redirectCapture(inspection, browser.allowedOrigin, 302), + ); + expect(missingVsKnown).toMatchObject({ + normalized_result: { + dimensions: { + network: { + status: "unknown", + total_changes: 0, + reason: expect.stringContaining("redirect"), + }, + }, + }, + }); + const knownDifference = await compareCapturesCli( + redirectCapture(inspection, browser.allowedOrigin, 302), + redirectCapture(inspection, browser.allowedOrigin, 307), + ); + expect(knownDifference).toMatchObject({ + normalized_result: { + dimensions: { network: { status: "changed", total_changes: 1 } }, + }, + }); + }, + INTEGRATION_TEST_TIMEOUT_MS, +); + it( "browser CLI attaches through a browser socket and records it redacted", async () => { @@ -344,6 +390,68 @@ describe("scenario comparison CLI diagnostics", () => { ); }); +describe("scenario comparison recovery guidance", () => { + it("reports independent network recovery alongside console differences", async () => { + const base = completeScenarioCapture(); + const matchingSelection = scenarioCaptureWithConsole( + "before", + base.scenario.network_content, + ); + const differentSelection = scenarioCaptureWithConsole("after", { + ...base.scenario.network_content, + response_body: true, + }); + const changed = await compareCapturesCli( + matchingSelection, + differentSelection, + ); + expect(changed).toMatchObject({ + normalized_result: { + overall_status: "changed", + steps: expect.arrayContaining([ + expect.objectContaining({ + step_id: "open-settings", + artifact_diffs: expect.arrayContaining([ + expect.objectContaining({ + artifact: "events", + status: "changed", + reason: expect.stringContaining( + "Network content selections differ", + ), + }), + ]), + }), + ]), + }, + }); + const quietMismatch = await compareCapturesCli( + matchingSelection, + scenarioCaptureWithConsole( + "before", + differentSelection.scenario.network_content, + ), + ); + expect(quietMismatch).toMatchObject({ + normalized_result: { + steps: expect.arrayContaining([ + expect.objectContaining({ + step_id: "open-settings", + artifact_diffs: expect.arrayContaining([ + expect.objectContaining({ + artifact: "events", + status: "unknown", + reason: expect.stringContaining( + "Network content selections differ", + ), + }), + ]), + }), + ]), + }, + }); + }); +}); + describe("browser CLI capture parity", () => { it( "runs analysis, session, WebMCP, screenshot, and comparison from explicit request scope", @@ -411,7 +519,14 @@ describe("browser CLI capture parity", () => { operation: "discover_webmcp_tools", normalized_result: { tools: { - items: [expect.objectContaining({ name: "search_orders" })], + items: [ + expect.objectContaining({ + name: "search_orders", + annotations: expect.objectContaining({ + consequential: true, + }), + }), + ], }, }, }); @@ -461,6 +576,43 @@ const normalizedResult = (value: unknown): Record => { return value.normalized_result; }; +const redirectCapture = ( + inspection: Record, + origin: string, + status: number | null, +) => { + const copy = structuredClone(inspection); + if (!isRecord(copy.network) || !Array.isArray(copy.network.requests)) + throw new Error("Missing network capture"); + const request = copy.network.requests[0]; + if (!isRecord(request)) throw new Error("Missing network request"); + request.redirects = [ + { + url: `${origin}/prior`, + response_url: `${origin}/prior`, + method: "GET", + resource_type: "Fetch", + status, + mime_type: "text/plain", + encoded_data_length: 12, + request_timestamp: 1, + redirect_event_timestamp: 2, + }, + ]; + return { inspection: copy }; +}; + +const compareCapturesCli = (before: unknown, after: unknown) => + runCli( + [ + "compare-web-captures", + JSON.stringify(before), + JSON.stringify(after), + "--json", + ], + process.env, + ); + const completeScenarioCapture = () => { const url = { url: "https://app.example.test/", @@ -514,17 +666,65 @@ const completeScenarioCapture = () => { header_values: false, }, event_families: [], + limits: DEFAULT_BROWSER_SCENARIO_CAPTURE_LIMITS, }, duration_ms: 1, steps: [ step(0, "scenario_start", "scenario_start"), step(1, "open-settings", "click"), ], - events: { retained: 0, dropped: 0, items: [] }, + events: { + retained: 0, + dropped: 0, + items: [], + omissions: { + by_kind: [], + network_transactions: [], + network_transaction_range: null, + }, + }, completeness, limitations: [], }; }; +const scenarioCaptureWithConsole = ( + text: string, + networkContent: { + readonly request_body: boolean; + readonly response_body: boolean; + readonly header_values: boolean; + }, +) => { + const capture = completeScenarioCapture(); + return { + ...capture, + scenario: { + ...capture.scenario, + network_content: networkContent, + event_families: ["console"], + }, + events: { + retained: 1, + dropped: 0, + items: [ + { + sequence: 1, + step_index: 1, + kind: "console", + level: "info", + text, + url: null, + }, + ], + omissions: { + by_kind: [], + network_transactions: [], + network_transaction_range: null, + }, + }, + }; +}; + const isRecord = (value: unknown): value is Record => typeof value === "object" && value !== null && !Array.isArray(value); diff --git a/tests/boundary/cli/bundleComparisonCli.test.ts b/tests/boundary/cli/bundleComparisonCli.test.ts new file mode 100644 index 0000000000..620cd0eaf2 --- /dev/null +++ b/tests/boundary/cli/bundleComparisonCli.test.ts @@ -0,0 +1,105 @@ +import { writeFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { expect } from "vitest"; + +import { createEvidence } from "../../../src/domain/evidence.js"; +import { bundleComparisonResultSchema } from "../../../src/domain/bundleComparison.js"; +import { + createEvidenceBundle, + serializeEvidenceBundle, +} from "../../../src/domain/evidenceBundle.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { cliTest } from "../../support/cli/cliFixture.js"; + +cliTest( + "compares Evidence bundles without unexplained display-only changes", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-bundle-comparison-cli-"); + const leftPath = join(root, "left.json"); + const displayPath = join(root, "display.json"); + const changedPath = join(root, "changed.json"); + const target = { + path: join(root, "left.elf"), + sha256: "a".repeat(64), + format: "elf" as const, + architecture: "x86_64" as const, + }; + const record = createEvidence( + target, + { id: "fixture", name: "Fixture", version: "1" }, + { + operation: "observe", + parameters: {}, + result: { observed: true }, + confidence: "observed", + authority: "shipped-artifact", + }, + ); + const left = createEvidenceBundle([record]); + const displayOnly = { + ...left, + records: [ + { + ...record, + subject: { + ...record.subject!, + name: "renamed.elf", + local_path: join(root, "renamed.elf"), + }, + }, + ], + }; + const different = createEvidenceBundle([ + createEvidence( + target, + { id: "fixture", name: "Fixture", version: "1" }, + { + operation: "observe", + parameters: {}, + result: { observed: false }, + confidence: "observed", + authority: "shipped-artifact", + }, + ), + ]); + await Promise.all([ + writeFile(leftPath, serializeEvidenceBundle(left)), + writeFile(displayPath, serializeEvidenceBundle(displayOnly)), + writeFile(changedPath, serializeEvidenceBundle(different)), + ]); + const compare = async (path: string) => + cli.run({ arguments: ["compare", leftPath, path, "--json"] }); + const same = await compare(leftPath); + expect(same.exitCode).toBe(0); + expect(bundleComparisonResultSchema.parse(same.json)).toMatchObject({ + status: "unchanged", + changes: [], + }); + const display = await compare(displayPath); + expect(display.exitCode).toBe(0); + const displayResult = bundleComparisonResultSchema.parse(display.json); + expect(displayResult).toMatchObject({ status: "unchanged", changes: [] }); + expect(displayResult.left_bundle_sha256).not.toBe( + displayResult.right_bundle_sha256, + ); + const differentResult = await compare(changedPath); + expect(differentResult.exitCode).toBe(0); + expect( + bundleComparisonResultSchema.parse(differentResult.json), + ).toMatchObject({ + status: "unknown", + summary: { unresolved: 2 }, + changes: [ + expect.objectContaining({ + entity: "evidence", + conclusion_kind: "unresolved_branch", + }), + expect.objectContaining({ + entity: "evidence", + conclusion_kind: "unresolved_branch", + }), + ], + }); + }, +); diff --git a/tests/boundary/cli/cliInputSchemaRequired.test.ts b/tests/boundary/cli/cliInputSchemaRequired.test.ts index 82f72af10c..efd72f10df 100644 --- a/tests/boundary/cli/cliInputSchemaRequired.test.ts +++ b/tests/boundary/cli/cliInputSchemaRequired.test.ts @@ -200,4 +200,27 @@ describe("web module CLI advertised input schema", () => { expect(options({ importMapBaseUrl: "" })).toBe(false); }); }); + +describe("reference-source outline-depth input schema", () => { + it("aligns advertised bounds with the parser's accepted values", async () => { + const cli = createCli({}); + const advertised = await advertisedSchema(cli, "import-reference-source"); + const outlineDepth = advertised.options?.properties?.outlineDepth; + expect(outlineDepth).toMatchObject({ type: "integer", minimum: 0 }); + + const parser = parserSchema(cli, "import-reference-source", "options"); + expect(parser?.safeParse({ outlineDepth: -1 }).success).toBe(false); + expect(parser?.safeParse({ outlineDepth: 0 }).success).toBe(true); + expect(parser?.safeParse({ outlineDepth: 2 }).success).toBe(true); + + const ajv = new Ajv2020({ strict: false }); + const validate = ajv.compile({ + ...advertised.options, + $schema: "https://json-schema.org/draft/2020-12/schema", + }); + expect(validate({ outlineDepth: -1 })).toBe(false); + expect(validate({ outlineDepth: 0 })).toBe(true); + expect(validate({ outlineDepth: 2 })).toBe(true); + }); +}); import { resolve } from "node:path"; diff --git a/tests/boundary/cli/cliOutput.test.ts b/tests/boundary/cli/cliOutput.test.ts index 44b3ac2346..351a49892c 100644 --- a/tests/boundary/cli/cliOutput.test.ts +++ b/tests/boundary/cli/cliOutput.test.ts @@ -7,6 +7,7 @@ import { describe, expect, it } from "vitest"; import { workspaceCliTest } from "../../support/cli/workspaceCliFixture.js"; import { + normalizeCliOutputArguments, renderCliOutputArgumentError, renderEmptyFilteredCliOutput, validateCliOutputArguments, @@ -58,6 +59,50 @@ describe("CLI output argument boundary", () => { }); }); +describe("CLI output flag value forms", () => { + it("splits only the equals forms of valued output flags", () => { + expect( + normalizeCliOutputArguments([ + "inspect-sqlite-database", + "--table=a=b", + "--format=json", + "--filter-output=a.b,c[0,1]", + "--token-limit=5", + "--token-offset=", + "--json", + "--format-hint=x", + ]), + ).toEqual([ + "inspect-sqlite-database", + "--table=a=b", + "--format", + "json", + "--filter-output", + "a.b,c[0,1]", + "--token-limit", + "5", + "--token-offset", + "", + "--json", + "--format-hint=x", + ]); + }); + + it("reports a missing value only for a flag the parser did not consume", () => { + expect( + validateCliOutputArguments([ + "capabilities", + "--json", + "--filter-output", + "--format", + ]), + ).toEqual({ ok: true }); + expect( + validateCliOutputArguments(["capabilities", "--json", "--format"]), + ).toMatchObject({ ok: false, code: "MISSING_OUTPUT_VALUE" }); + }); +}); + describe("unclosed CLI output filter slices", () => { it.each([ "providers[", @@ -153,7 +198,7 @@ describe("CLI output valued-flag parsing", () => { async ({ cli }) => { for (const arguments_ of [ ["providers", "--", "--json", "--token-limit", "5"], - ["providers", "--json", "--format=toon", "--token-limit", "5"], + ["providers", "--format=json", "--token-limit=5"], ]) { const result = await cli.run({ arguments: arguments_ }); expect(result.exitCode).toBe(1); @@ -183,22 +228,45 @@ describe("CLI output valued-flag parsing", () => { ); workspaceCliTest( - "does not treat empty or equals-style output values as active builtins", + "names missing output values and honors equals-style output values", async ({ cli }) => { - for (const arguments_ of [ - ["capabilities", "--json", "--format", ""], - ["capabilities", "--json", "--filter-output", ""], - ["capabilities", "--json", "--token-limit", ""], - ["capabilities", "--json", "--token-limit=5"], - ]) { + for (const [arguments_, flag] of [ + [["capabilities", "--json", "--format", ""], "--format"], + [["capabilities", "--json", "--filter-output", ""], "--filter-output"], + [["capabilities", "--json", "--token-limit", ""], "--token-limit"], + [["capabilities", "--json", "--token-offset="], "--token-offset"], + [["capabilities", "--format"], "--format"], + ] as const) { const result = await cli.run({ arguments: arguments_ }); expect(result.exitCode).toBe(1); - expect(result.json).toBeDefined(); expect(result.stdout).not.toContain("[truncated:"); - expect(JSON.stringify(result.json)).not.toContain( - "UNSUPPORTED_OUTPUT_COMBINATION", - ); + expect(result.json).toMatchObject({ + ok: false, + error: { + code: "MISSING_OUTPUT_VALUE", + message: expect.stringContaining(`${flag} requires a value`), + }, + }); } + const equalsForms = await cli.run({ + arguments: [ + "capabilities", + "--format=json", + "--filter-output=capabilities[0,1]", + ], + }); + expect(equalsForms.exitCode).toBe(0); + expect(equalsForms.json).toMatchObject({ + capabilities: [expect.any(Object)], + }); + const equalsTokenWindow = await cli.run({ + arguments: ["capabilities", "--json", "--token-limit=5"], + }); + expect(equalsTokenWindow.exitCode).toBe(1); + expect(equalsTokenWindow.json).toMatchObject({ + ok: false, + error: { code: "UNSUPPORTED_OUTPUT_COMBINATION" }, + }); const validTokenWindow = await cli.run({ arguments: ["capabilities", "--json", "--token-limit", "5"], }); diff --git a/tests/boundary/cli/cliSetup.test.ts b/tests/boundary/cli/cliSetup.test.ts index 38bc24bb3e..15cea67f50 100644 --- a/tests/boundary/cli/cliSetup.test.ts +++ b/tests/boundary/cli/cliSetup.test.ts @@ -1,6 +1,15 @@ import { execFile } from "node:child_process"; -import { access, rm } from "node:fs/promises"; -import { join } from "node:path"; +import { + access, + lstat, + mkdir, + readFile, + readdir, + rm, + symlink, + writeFile, +} from "node:fs/promises"; +import { dirname, join } from "node:path"; import { promisify } from "node:util"; import { spawn } from "@lydell/node-pty"; @@ -74,48 +83,53 @@ const clientStates = [ const agentActions = actions.filter(({ kind }) => kind !== "install_hopper"); +const runSetupDryRunInTty = async ( + home: string, + arguments_: readonly string[], +): Promise<{ readonly exitCode: number; readonly output: string }> => { + const terminal = spawn( + process.execPath, + ["scripts/rea.mjs", "setup", "--dry-run", ...arguments_], + { + cwd: process.cwd(), + env: { + ...process.env, + HOME: home, + USERPROFILE: home, + XDG_CONFIG_HOME: join(home, ".config"), + REA_LOG_LEVEL: "silent", + NO_COLOR: "1", + }, + name: "xterm-256color", + cols: 120, + rows: 40, + }, + ); + let output = ""; + terminal.onData((data) => { + output += data; + }); + const exitCode = await new Promise((resolvePromise, reject) => { + const timeout = setTimeout(() => { + terminal.kill(); + reject(new Error(`TTY dry run timed out after output:\n${output}`)); + }, 15_000); + terminal.onExit(({ exitCode: code }) => { + clearTimeout(timeout); + resolvePromise(code); + }); + }); + return { exitCode, output }; +}; + describe("TTY setup dry run", () => { it("prints the plan and no-mutation notice for a TTY dry run", async () => { const home = await createTestTempDirectory("rea-cli-setup-dry-run-"); - const terminal = spawn( - process.execPath, - [ - "scripts/rea.mjs", - "setup", - "--dry-run", - "--client", - "codex", - "--skill=false", - ], - { - cwd: process.cwd(), - env: { - ...process.env, - HOME: home, - USERPROFILE: home, - XDG_CONFIG_HOME: join(home, ".config"), - REA_LOG_LEVEL: "silent", - NO_COLOR: "1", - }, - name: "xterm-256color", - cols: 120, - rows: 40, - }, - ); - let output = ""; - terminal.onData((data) => { - output += data; - }); - const exitCode = await new Promise((resolvePromise, reject) => { - const timeout = setTimeout(() => { - terminal.kill(); - reject(new Error(`TTY dry run timed out after output:\n${output}`)); - }, 15_000); - terminal.onExit(({ exitCode: code }) => { - clearTimeout(timeout); - resolvePromise(code); - }); - }); + const { exitCode, output } = await runSetupDryRunInTty(home, [ + "--client", + "codex", + "--skill=false", + ]); expect(exitCode, output).toBe(0); expect(output).toContain("CREATE Codex"); @@ -127,12 +141,48 @@ describe("TTY setup dry run", () => { code: "ENOENT", }); }); + + it("shows the refusal instead of a blocked skill plan for a TTY dry run", async () => { + const home = await createTestTempDirectory("rea-cli-setup-linked-dry-run-"); + const managed = join(home, "managed-skill"); + const linked = join(home, ".claude/skills/reverse-engineer-anything"); + const original = "user-managed skill bytes\n"; + await mkdir(managed); + await writeFile(join(managed, "SKILL.md"), original); + await mkdir(dirname(linked), { recursive: true }); + await symlink( + managed, + linked, + process.platform === "win32" ? "junction" : "dir", + ); + + const { exitCode, output } = await runSetupDryRunInTty(home, [ + "--client", + "claude_code", + ]); + + expect(exitCode, output).toBe(1); + expect(output).toContain("Skill directory is a symbolic link."); + expect(output).toContain(linked); + expect(output).toContain("--skill=false"); + expect(output).not.toContain("REA setup plan"); + expect(output).not.toContain("Install or update the REA skill"); + expect((await lstat(linked)).isSymbolicLink()).toBe(true); + expect(await readFile(join(managed, "SKILL.md"), "utf8")).toBe(original); + expect(await readdir(managed)).toEqual(["SKILL.md"]); + await expect(access(join(home, ".claude.json"))).rejects.toMatchObject({ + code: "ENOENT", + }); + }); }); describe("interactive setup journey", () => { it("shows detected and absent clients without preselecting detected clients", async () => { const result = await runJourney( - [step("Which agents should use REA?", "\u001b[B\r")], + [ + step("Which agents should use REA?", "\u001b[B"), + step("Not detected · configuration is still available", "\r"), + ], false, [actions[0]], ); @@ -312,7 +362,14 @@ describe("interactive setup completion", () => { { cwd: process.cwd(), env: { - ...process.env, + ...Object.fromEntries( + Object.entries(process.env).filter( + ([key]) => + !["npm_command", "npm_config_user_agent"].includes( + key.toLowerCase(), + ), + ), + ), npm_command: npmCommand, npm_config_user_agent: userAgent, NO_COLOR: "1", @@ -380,6 +437,7 @@ const runJourney = async ( env: { ...process.env, HOME: isolatedHome, + USERPROFILE: isolatedHome, NO_COLOR: "1", TERM: "xterm-256color", }, @@ -417,9 +475,13 @@ const runJourney = async ( finish(new Error(`Setup journey returned no decision:\n${output}`)); return; } - const serialized = output - .slice(markerIndex + decisionMarker.length) - .split(/\r?\n/u, 1)[0]; + const raw = output.slice(markerIndex + decisionMarker.length); + const jsonStart = raw.indexOf("{"); + const jsonEnd = raw.lastIndexOf("}"); + const serialized = + jsonStart !== -1 && jsonEnd !== -1 && jsonEnd > jsonStart + ? raw.slice(jsonStart, jsonEnd + 1) + : raw.split(/\r?\n/u, 1)[0]; if (serialized === undefined) { finish( new Error(`Setup journey returned an empty decision:\n${output}`), diff --git a/tests/boundary/cli/clientProfileLifecycle.test.ts b/tests/boundary/cli/clientProfileLifecycle.test.ts index 1ceaf49cdf..e2bca171f2 100644 --- a/tests/boundary/cli/clientProfileLifecycle.test.ts +++ b/tests/boundary/cli/clientProfileLifecycle.test.ts @@ -163,31 +163,33 @@ for (const group of groups) { environment, timeoutMs: 20_000, }); - for (const { client, config, skillDirectory } of profiles) { - const planned = await run([ - "setup", - "--client", - client, - "--dry-run", - "--json", - ]); - expect(planned.exitCode, client).toBe(0); - expect(planned.json, client).toMatchObject({ - status: "planned", - plannedActions: expect.arrayContaining([ + const planned = await run([ + "setup", + ...clientArguments, + "--dry-run", + "--json", + ]); + expect(planned.exitCode, planned.stderr).toBe(0); + expect(planned.json).toMatchObject({ + status: "planned", + plannedActions: expect.arrayContaining([ + ...profiles.map(({ client, config }) => expect.objectContaining({ id: `configure_client:${client}`, target: config, }), - expect.objectContaining({ target: skillDirectory }), - ]), - }); + ), + ...[ + ...new Set(profiles.map(({ skillDirectory }) => skillDirectory)), + ].map((target) => expect.objectContaining({ target })), + ]), + }); + for (const { config, skillDirectory } of profiles) { + await expect(access(config)).rejects.toMatchObject({ code: "ENOENT" }); await expect(access(skillDirectory)).rejects.toMatchObject({ code: "ENOENT", }); } - for (const { config } of profiles) - await expect(access(config)).rejects.toMatchObject({ code: "ENOENT" }); const setup = await run(["setup", ...clientArguments, "--yes", "--json"]); expect(setup.exitCode).toBe(0); expect(setup.json).toMatchObject({ status: "ready" }); diff --git a/tests/boundary/cli/clientProfiles.test.ts b/tests/boundary/cli/clientProfiles.test.ts index bab6dedd8a..092ce65d0c 100644 --- a/tests/boundary/cli/clientProfiles.test.ts +++ b/tests/boundary/cli/clientProfiles.test.ts @@ -262,6 +262,91 @@ cliTest( }, ); +cliTest( + "Gemini matches allowlist spellings independently of their policy order", + async ({ cli }) => { + const home = await createTestTempDirectory("rea-gemini-case-"); + const system = join(home, "system.json"); + const defaults = join(home, "defaults.json"); + await writeFile(system, JSON.stringify({ mcp: { allowed: ["REA"] } })); + const environment = { + HOME: home, + USERPROFILE: home, + GEMINI_CLI_SYSTEM_SETTINGS_PATH: system, + GEMINI_CLI_SYSTEM_DEFAULTS_PATH: defaults, + }; + const run = () => + cli.run({ + arguments: ["setup", "--client", "gemini_cli", "--yes", "--json"], + cwd: home, + environment, + }); + // The intersection compares names case-insensitively, so the spelling that + // survives must not decide the outcome: the same pair of policies in + // either arrangement has to consolidate to the same decision. + for (const [systemSpelling, defaultSpelling] of [ + ["REA", "rea"], + ["rea", "REA"], + ]) { + await writeFile( + system, + JSON.stringify({ mcp: { allowed: [systemSpelling] } }), + ); + await writeFile( + defaults, + JSON.stringify({ mcp: { allowed: [defaultSpelling] } }), + ); + const result = await run(); + expect(result.exitCode, `${systemSpelling}+${defaultSpelling}`).toBe(0); + expect(result.json, `${systemSpelling}+${defaultSpelling}`).toMatchObject( + { + status: "ready", + }, + ); + } + }, +); + +cliTest( + "Gemini compares exclusions and padded allowlist entries through one normalization", + async ({ cli }) => { + const home = await createTestTempDirectory("rea-gemini-normalized-"); + const system = join(home, "system.json"); + const defaults = join(home, "defaults.json"); + const environment = { + HOME: home, + USERPROFILE: home, + GEMINI_CLI_SYSTEM_SETTINGS_PATH: system, + GEMINI_CLI_SYSTEM_DEFAULTS_PATH: defaults, + }; + const run = () => + cli.run({ + arguments: ["setup", "--client", "gemini_cli", "--yes", "--json"], + cwd: home, + environment, + }); + await writeFile(defaults, JSON.stringify({ mcp: {} })); + // An exclusion is a membership test over the same names the intersection + // compares, so a differently spelled and padded entry blocks setup rather + // than slipping past the raw comparison. + await writeFile(system, JSON.stringify({ mcp: { excluded: [" REA "] } })); + const blocked = await run(); + expect(blocked.exitCode).toBe(1); + expect(blocked.json).toMatchObject({ + status: "needs_human", + plannedActions: [], + appliedActions: [], + remediation: expect.stringContaining("mcp.excluded"), + }); + // Padding is presentation, not identity: a padded allowlist entry permits + // the same server its exact spelling permits. + await writeFile(system, JSON.stringify({ mcp: { allowed: [" REA "] } })); + const allowed = await run(); + expect(allowed.exitCode).toBe(0); + expect(allowed.json).toMatchObject({ status: "ready" }); + }, +); + cliTest( "OpenCode doctor sees a global registration when the custom directory is absent", async ({ cli }) => { diff --git a/tests/boundary/cli/closedStdout.test.ts b/tests/boundary/cli/closedStdout.test.ts new file mode 100644 index 0000000000..8227d0b16a --- /dev/null +++ b/tests/boundary/cli/closedStdout.test.ts @@ -0,0 +1,84 @@ +import { join, resolve } from "node:path"; +import { DatabaseSync } from "node:sqlite"; + +import { describe, expect } from "vitest"; + +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { cliTest } from "../../support/cli/cliFixture.js"; +import type { TestProcesses } from "../../support/process/processFixture.js"; + +/** Run the CLI, closing its stdout reader after `bytes` (0: before any write). */ +const runWithClosedReader = async ( + processes: TestProcesses, + arguments_: readonly string[], + bytes: number, +) => { + const child = processes.spawn( + process.execPath, + [resolve("scripts/rea.mjs"), ...arguments_], + { env: { PATH: process.env.PATH ?? "", REA_LOG_LEVEL: "silent" } }, + ); + let received = 0; + if (bytes === 0) child.stdout.destroy(); + else + child.stdout.on("data", (chunk: Buffer) => { + received += chunk.length; + if (received >= bytes) child.stdout.destroy(); + }); + child.stdin.end(); + let stderr = ""; + child.stderr.setEncoding("utf8"); + child.stderr.on("data", (chunk: string) => (stderr += chunk)); + const code = await new Promise((done) => + child.once("close", done), + ); + return { code, stderr, received }; +}; + +// Windows reports a closed pipe reader through a different error path. +describe.skipIf(process.platform === "win32")( + "CLI with a closed stdout", + () => { + cliTest.for([ + ["toon", ["capabilities", "--format", "toon"]], + ["json", ["capabilities", "--format", "json"]], + ["mcp doctor", ["mcp", "doctor", "--json"]], + ] as const)( + "stops quietly with the SIGPIPE status when the reader closes first ($0)", + async ([, arguments_], { processes }) => { + const result = await runWithClosedReader(processes, arguments_, 0); + expect(result).toMatchObject({ code: 141, stderr: "" }); + }, + ); + + cliTest( + "stops quietly when the reader closes during streamed JSON", + async ({ processes }) => { + const directory = await createTestTempDirectory("rea-closed-stdout-"); + const database = join(directory, "rows.db"); + const writer = new DatabaseSync(database); + writer.exec("CREATE TABLE t (v TEXT)"); + const insert = writer.prepare("INSERT INTO t VALUES (?)"); + // 4 MiB of rows outlasts any pipe buffer, so writes continue after close. + for (let row = 0; row < 64; row += 1) insert.run("x".repeat(65_536)); + writer.close(); + const result = await runWithClosedReader( + processes, + [ + "inspect-sqlite-database", + database, + "--table", + "t", + "--row-limit", + "64", + "--format", + "json", + ], + 1, + ); + expect(result.received).toBeGreaterThan(0); + expect(result).toMatchObject({ code: 141, stderr: "" }); + }, + ); + }, +); diff --git a/tests/boundary/cli/cutterCommands.test.ts b/tests/boundary/cli/cutterCommands.test.ts new file mode 100644 index 0000000000..6a588cb4b8 --- /dev/null +++ b/tests/boundary/cli/cutterCommands.test.ts @@ -0,0 +1,204 @@ +import { createServer } from "node:net"; +import { createHmac } from "node:crypto"; +import { mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { afterEach, describe, expect, it } from "vitest"; + +import { serveCli } from "./cliServeHarness.js"; + +const SESSION_ID = "27d3e3f1-f1e5-49ae-91ec-95af1f343a5a"; +const listen = async ( + server: ReturnType, +): Promise => { + await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (address === null || typeof address === "string") + throw new Error("Expected an ephemeral TCP listener"); + return address.port; +}; + +describe("Cutter CLI commands", () => { + let directory: string | undefined; + let server: ReturnType | undefined; + + afterEach(async () => { + if (server !== undefined) + await new Promise((resolve) => server?.close(() => resolve())); + if (directory !== undefined) + await rm(directory, { recursive: true, force: true }); + directory = undefined; + server = undefined; + }); + + it("lists sessions from the selected bridge directory", async () => { + directory = await mkdtemp(join(tmpdir(), "rea-cutter-cli-test-")); + + const result = await serveCli(["list-cutter-sessions", "--json"], { + REA_CUTTER_BRIDGE_DIR: directory, + }); + + expect(result.exitCode).toBe(0); + expect(JSON.parse(result.stdout)).toMatchObject({ + sessions: [], + discovery_status: "no_live_bridge_found", + bridge_directory: directory, + }); + }); + + it("reports an unavailable selected session for a command", async () => { + directory = await mkdtemp(join(tmpdir(), "rea-cutter-cli-test-")); + + const result = await serveCli( + ["cutter-command", SESSION_ID, "0", "pdi", "--json"], + { REA_CUTTER_BRIDGE_DIR: directory }, + ); + + expect(result.exitCode).toBe(0); + expect(result.stdout).toContain("invalid_request"); + }); +}); + +describe("Cutter CLI command output flags", () => { + let directory: string | undefined; + let server: ReturnType | undefined; + + afterEach(async () => { + if (server !== undefined) + await new Promise((resolve) => server?.close(() => resolve())); + if (directory !== undefined) + await rm(directory, { recursive: true, force: true }); + directory = undefined; + server = undefined; + }); + + it("returns Evidence when the selected Cutter session completes a command", async () => { + directory = await mkdtemp(join(tmpdir(), "rea-cutter-cli-test-")); + const token = "synthetic-cutter-cli-token-for-test"; + let nextConnectionOrdinal = 0; + const commandJsonModes: boolean[] = []; + server = createServer((connection) => { + const ordinal = ++nextConnectionOrdinal; + connection.setEncoding("utf8"); + let input = ""; + let clientNonce = ""; + let serverNonce = ""; + connection.on("data", (chunk: string) => { + input += chunk; + const newline = input.indexOf("\n"); + if (newline < 0) return; + const frame = JSON.parse(input.slice(0, newline)) as Record< + string, + unknown + >; + input = input.slice(newline + 1); + if (frame.kind === "rea-auth-v2") { + clientNonce = String(frame.nonce); + serverNonce = `${ordinal.toString(16).padStart(16, "0")}${"b".repeat(48)}`; + connection.write( + `${JSON.stringify({ session_id: SESSION_ID, nonce: serverNonce, proof: createHmac("sha256", token).update(`rea-cutter-server-v2:${SESSION_ID}:${clientNonce}:${serverNonce}`).digest("hex") })}\n`, + ); + return; + } + const requestBytes = String(frame.request); + const request = JSON.parse(requestBytes) as Record; + if (request.kind === "command") + commandJsonModes.push(request.json === true); + if ( + frame.proof !== + createHmac("sha256", token) + .update( + `rea-cutter-client-v2:${SESSION_ID}:${clientNonce}:${serverNonce}:${requestBytes}`, + ) + .digest("hex") + ) { + connection.destroy(); + return; + } + const status = request.kind === "status"; + const response = JSON.stringify({ + ok: true, + session_id: request.session_id, + ...(status + ? { + current_file: null, + document_generation: 0, + cutter_version: "Cutter fixture", + identity_status: "partial", + completed_connection: (ordinal - 1) + .toString(16) + .padStart(16, "0"), + completed_commands: [], + } + : request.kind === "ack" + ? { acknowledged: false } + : { + output: { answer: 42 }, + current_file: null, + document_generation: 0, + cutter_version: "Cutter fixture", + identity_status: "partial", + execution_state: "complete", + output_truncated: false, + }), + }); + const proof = createHmac("sha256", token) + .update( + `rea-cutter-response-v2:${SESSION_ID}:${clientNonce}:${serverNonce}:${response}`, + ) + .digest("hex"); + connection.end(`${JSON.stringify({ response, proof })}\n`); + }); + }); + const port = await listen(server); + await writeFile( + join(directory, `cutter-${process.pid}-${SESSION_ID}.json`), + JSON.stringify({ + session_id: SESSION_ID, + pid: process.pid, + host: "127.0.0.1", + port, + token, + document_generation: 0, + current_file: null, + cutter_version: "Cutter fixture", + identity_status: "partial", + }), + { mode: 0o600 }, + ); + + const result = await serveCli( + ["cutter-command", SESSION_ID, "0", "pdi", "--json-output"], + { REA_CUTTER_BRIDGE_DIR: directory }, + ); + + expect(result.exitCode).toBe(0); + expect(commandJsonModes).toEqual([true]); + expect(result.stdout).toContain("answer: 42"); + + const formatted = await serveCli( + ["cutter-command", SESSION_ID, "0", "pdi", "--json"], + { REA_CUTTER_BRIDGE_DIR: directory }, + ); + expect(formatted.exitCode).toBe(0); + expect(commandJsonModes).toEqual([true, false]); + expect(() => JSON.parse(formatted.stdout)).not.toThrow(); + + for (const [flags, expected] of [ + [["--json=true"], true], + [["--json=false"], false], + [["--json=true", "--json-output=false"], false], + [["--json=false", "--json-output=true"], true], + [["--json-output=false", "--json=true"], false], + [["--json-output=true", "--json=false"], true], + ] as const) { + const compatible = await serveCli( + ["cutter-command", SESSION_ID, "0", "pdi", ...flags], + { REA_CUTTER_BRIDGE_DIR: directory }, + ); + expect(compatible.exitCode).toBe(0); + expect(commandJsonModes.at(-1)).toBe(expected); + } + }); +}); diff --git a/tests/boundary/cli/developmentTests.test.ts b/tests/boundary/cli/developmentTests.test.ts index 2987206c87..93e3d9ccca 100644 --- a/tests/boundary/cli/developmentTests.test.ts +++ b/tests/boundary/cli/developmentTests.test.ts @@ -1,5 +1,6 @@ import { copyFile, + lstat, mkdir, readFile, symlink, @@ -12,12 +13,18 @@ import { describe, expect, it } from "vitest"; import { execFileOutput } from "../../../src/process/ExecFileOutput.js"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; -const run = (root: string, command: string, arguments_: readonly string[]) => +const run = ( + root: string, + command: string, + arguments_: readonly string[], + env?: NodeJS.ProcessEnv, +) => execFileOutput(command, arguments_, { cwd: root, stopSignal: "SIGTERM", timeout: 30_000, maxBuffer: 1024 * 1024, + ...(env === undefined ? {} : { env }), }); const git = (root: string, ...arguments_: string[]) => @@ -44,7 +51,9 @@ const fixture = async () => { await mkdir(join(root, "src")); for (const path of [ "scripts/run-development-tests.mjs", + "scripts/run-exact-tests.mjs", "scripts/lib/development-tests.mjs", + "scripts/lib/test-projects.mjs", ]) await copyFile(join(process.cwd(), path), join(root, path)); await symlink( @@ -69,11 +78,214 @@ const fixture = async () => { return root; }; +// Assertions read the child's plain-text summary. An inherited color setting +// styled the counts in the CI lane; NO_COLOR takes precedence over FORCE_COLOR. const testCommand = (root: string, ...arguments_: string[]) => - run(root, process.execPath, [ - "scripts/run-development-tests.mjs", - ...arguments_, - ]); + run( + root, + process.execPath, + ["scripts/run-development-tests.mjs", ...arguments_], + { ...process.env, NO_COLOR: "1" }, + ); + +describe("development test runner symlink aliases", () => { + it("rejects excluded file and directory aliases without running their targets", async ({ + skip, + }) => { + for (const { kind, mode } of [ + { kind: "file", mode: "focused" }, + { kind: "directory", mode: "local" }, + { kind: "directory", mode: "focused" }, + ] as const) { + const root = await fixture(); + const included = + kind === "file" ? join(root, "src") : join(root, "src", "domain"); + const includedTest = join(included, "value.test.ts"); + const alias = + kind === "file" + ? join(root, "src", "excluded.test.ts") + : join(root, "src", "contracts"); + if (kind === "directory") { + await mkdir(included); + for (const name of ["value.ts", "value.test.ts"]) + await copyFile(join(root, "src", name), join(included, name)); + } + try { + await symlink( + kind === "file" ? includedTest : included, + alias, + kind === "directory" + ? process.platform === "win32" + ? "junction" + : "dir" + : undefined, + ); + } catch (cause) { + if ( + process.platform === "win32" && + cause instanceof Error && + "code" in cause && + (cause.code === "EPERM" || cause.code === "EACCES") + ) + skip("Windows symlink creation requires unavailable privileges"); + throw cause; + } + const selectedAlias = + kind === "file" + ? "src/excluded.test.ts" + : "src/contracts/value.test.ts"; + const includedSelection = + kind === "file" ? "src/value.test.ts" : "src/domain/value.test.ts"; + await writeFile( + join(root, "vitest.config.mjs"), + `export default { test: { projects: [{ test: { name: "domain", include: ["${includedSelection}"] } }] } };\n`, + ); + + await expect( + testCommand(root, mode, selectedAlias), + ).rejects.toMatchObject({ + code: 1, + stderr: expect.stringContaining( + `Selected test files were not discovered: ${selectedAlias}`, + ), + }); + await expect( + readFile(join(root, "selected.txt"), "utf8"), + ).rejects.toMatchObject({ code: "ENOENT" }); + await testCommand(root, mode, includedSelection); + expect(await readFile(join(root, "selected.txt"), "utf8")).toBe("1"); + } + }); +}); + +describe("development test runner exact selection", () => { + it("matches selected path casing to the host filesystem", async () => { + const root = await fixture(); + try { + await lstat(join(root, "src/VALUE.test.ts")); + } catch (cause) { + if ( + !(cause instanceof Error) || + !("code" in cause) || + cause.code !== "ENOENT" + ) + throw cause; + await expect( + testCommand(root, "focused", "src/VALUE.test.ts"), + ).rejects.toMatchObject({ + code: 1, + stderr: expect.stringContaining("ENOENT"), + }); + return; + } + await testCommand(root, "focused", "src/VALUE.test.ts"); + expect(await readFile(join(root, "selected.txt"), "utf8")).toBe("1"); + }); + + it("matches Unicode-normalized aliases by filesystem identity", async () => { + const root = await fixture(); + const composed = "caf\u00e9.test.ts"; + const decomposed = composed.normalize("NFD"); + await writeFile( + join(root, "src", composed), + 'import { it } from "vitest"; it("checks the selected Unicode path", () => {});\n', + ); + const selected = join(root, "src", decomposed); + try { + await lstat(selected); + } catch (cause) { + if ( + !(cause instanceof Error) || + !("code" in cause) || + cause.code !== "ENOENT" + ) + throw cause; + await expect( + testCommand(root, "focused", `src/${decomposed}`), + ).rejects.toMatchObject({ + code: 1, + stderr: expect.stringContaining("ENOENT"), + }); + return; + } + const result = await testCommand(root, "focused", `src/${decomposed}`); + expect(result.stdout).toContain("1 passed"); + }); + + it.each(["local", "focused"])( + "runs only the exact selected file in %s mode", + async (mode) => { + const root = await fixture(); + await mkdir(join(root, "src", "nested", "src"), { recursive: true }); + await writeFile( + join(root, "src", "nested", "src", "value.test.ts"), + 'import { it } from "vitest"; it("must not run", () => { throw new Error("unselected nested test ran"); });\n', + ); + const result = await testCommand(root, mode, "src/value.test.ts"); + expect(result.stdout).toContain("1 passed"); + expect(result.stdout).not.toContain("nested/src/value.test.ts"); + expect(await readFile(join(root, "selected.txt"), "utf8")).toBe("1"); + }, + ); + + it("fails when any selected existing test file is excluded from discovery", async () => { + const root = await fixture(); + await writeFile( + join(root, "src", "excluded.test.ts"), + 'import { it } from "vitest"; it("excluded", () => {});\n', + ); + await writeFile( + join(root, "vitest.config.mjs"), + 'export default { test: { projects: [{ test: { name: "domain", include: ["src/value.test.ts"] } }] } };\n', + ); + await expect( + testCommand(root, "focused", "src/value.test.ts", "src/excluded.test.ts"), + ).rejects.toMatchObject({ + code: 1, + stderr: expect.stringContaining( + "Selected test files were not discovered: src/excluded.test.ts", + ), + }); + }); + + it("runs every selected file and preserves selected test failures", async () => { + const root = await fixture(); + await writeFile( + join(root, "src", "failure.test.ts"), + 'import { it } from "vitest"; it("selected failure", () => { throw new Error("selected regression"); });\n', + ); + await expect( + testCommand(root, "focused", "src/value.test.ts", "src/failure.test.ts"), + ).rejects.toMatchObject({ + code: 1, + stdout: expect.stringContaining("1 failed | 1 passed"), + stderr: expect.stringContaining("selected regression"), + }); + expect(await readFile(join(root, "selected.txt"), "utf8")).toBe("1"); + }); + + it("preserves all configured projects for an exact file", async () => { + const root = await fixture(); + await writeFile( + join(root, "vitest.config.mjs"), + 'export default { test: { projects: ["domain", "evaluation"].map(name => ({ test: { name, include: ["src/value.test.ts"] } })) } };\n', + ); + const result = await testCommand(root, "focused", "src/value.test.ts"); + expect(result.stdout).toMatch(/Test Files\s+2 passed/u); + }); + + it("fails an exact file with no tests even when another selected file passes", async () => { + const root = await fixture(); + await writeFile(join(root, "src", "empty.test.ts"), "export {};\n"); + await expect( + testCommand(root, "focused", "src/value.test.ts", "src/empty.test.ts"), + ).rejects.toMatchObject({ + code: 1, + stderr: expect.stringContaining("No test suite found"), + }); + expect(await readFile(join(root, "selected.txt"), "utf8")).toBe("1"); + }); +}); describe("development test runner public CLI", () => { it("runs an explicitly selected source test on a clean tree without build artifacts", async () => { diff --git a/tests/boundary/cli/dispatcher.test.ts b/tests/boundary/cli/dispatcher.test.ts index e1cd84a0e7..68d8fb39de 100644 --- a/tests/boundary/cli/dispatcher.test.ts +++ b/tests/boundary/cli/dispatcher.test.ts @@ -29,7 +29,7 @@ describe("executable dispatcher", () => { ), workspace.write( "dist/cli-runtime/cliOutput.js", - "export const validateCliOutputArguments = () => ({ ok: true }); export const renderCliOutputArgumentError = () => ''; export const renderEmptyFilteredCliOutput = () => undefined;\n", + "export const normalizeCliOutputArguments = (args) => args; export const validateCliOutputArguments = () => ({ ok: true }); export const renderCliOutputArgumentError = () => ''; export const renderEmptyFilteredCliOutput = () => undefined;\n", ), workspace.write( "dist/cli-runtime/cliResultOutput.js", diff --git a/tests/boundary/cli/doctorHopperDiscovery.test.ts b/tests/boundary/cli/doctorHopperDiscovery.test.ts new file mode 100644 index 0000000000..db4e442600 --- /dev/null +++ b/tests/boundary/cli/doctorHopperDiscovery.test.ts @@ -0,0 +1,93 @@ +import { access, chmod, mkdir, readFile, writeFile } from "node:fs/promises"; +import { delimiter, join } from "node:path"; + +import { expect } from "vitest"; + +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { cliTest } from "../../support/cli/cliFixture.js"; + +cliTest.skipIf(!["darwin", "linux"].includes(process.platform))( + "retains a configured Hopper path in host diagnostics without launching it", + async ({ cli }) => { + const home = await createTestTempDirectory("rea-doctor-hopper-home-"); + const launcher = `${home}/bin/../bin/hopper`; + const launchMarker = join(home, "hopper-was-launched"); + await mkdir(join(home, "bin"), { recursive: true }); + await writeFile( + launcher, + '#!/bin/sh\nprintf launched > "$REA_HOPPER_TEST_MARKER"\nexit 99\n', + ); + await chmod(launcher, 0o700); + const loaderMarker = join(home, "loader-checked-launcher"); + if (process.platform === "linux") { + // A shell fixture needs controlled loader diagnostics on Linux. This + // checks discovery and path preservation, not actual Hopper compatibility. + const loader = join(home, "bin", "ldd"); + await writeFile( + loader, + [ + "#!/bin/sh", + 'test "$#" -eq 1 || exit 87', + 'test "$1" = "$HOPPER_LAUNCHER_PATH" || exit 88', + 'printf "%s" "$1" > "$REA_HOPPER_LDD_TEST_MARKER"', + 'printf "%s\\n" "libc.so.6 => /fixture/libc.so.6 (0x1)"', + "", + ].join("\n"), + ); + await chmod(loader, 0o700); + } + + const result = await cli.run({ + arguments: ["doctor", "--provider", "hopper", "--json"], + cwd: home, + environment: { + ...(process.platform === "linux" + ? { + PATH: [join(home, "bin"), process.env.PATH ?? ""].join(delimiter), + REA_HOPPER_LDD_TEST_MARKER: loaderMarker, + NODE_OPTIONS: `--import=${new URL("../../fixtures/hopperRejectionFeed.mjs", import.meta.url).href}`, + } + : {}), + HOME: home, + USERPROFILE: home, + HOPPER_LAUNCHER_PATH: launcher, + REA_HOPPER_TEST_MARKER: launchMarker, + GHIDRA_INSTALL_DIR: join(home, "missing-ghidra"), + }, + timeoutMs: 20_000, + }); + + const hopperReady = process.platform === "darwin"; + expect(result.signal).toBeNull(); + expect(result.exitCode).toBe(hopperReady ? 0 : 1); + expect(result.json).toMatchObject({ + healthy: hopperReady, + environment_healthy: false, + hopperPath: launcher, + checks: expect.arrayContaining([ + expect.objectContaining({ + name: "hopper", + ok: true, + detail: launcher, + classification: "healthy", + }), + ...(process.platform === "linux" + ? [ + expect.objectContaining({ + name: "hopper-version", + ok: false, + detail: launcher, + classification: "config_drift", + remediation: expect.stringContaining("HOPPER_LAUNCHER_PATH"), + }), + ] + : []), + ]), + }); + await expect(access(launchMarker)).rejects.toMatchObject({ + code: "ENOENT", + }); + if (process.platform === "linux") + expect(await readFile(loaderMarker, "utf8")).toBe(launcher); + }, +); diff --git a/tests/boundary/cli/dylibResolutionCli.test.ts b/tests/boundary/cli/dylibResolutionCli.test.ts index 21ea99a9c6..128c51d161 100644 --- a/tests/boundary/cli/dylibResolutionCli.test.ts +++ b/tests/boundary/cli/dylibResolutionCli.test.ts @@ -106,6 +106,57 @@ const withBrokenHelper = async (app: string): Promise => { await writeFiles(app, { "Contents/Helpers/broken": broken }); }; +cliTest( + "does not turn U+FEFF-prefixed strings into dyld path tokens", + async ({ cli }) => { + const app = await fixtureApp(); + await writeFiles(app, { + "Contents/MacOS/App": machoImage({ + commands: [ + buildVersionCommand(1), + rpathCommand("@executable_path/../Frameworks"), + dylibCommand(LC.LOAD_DYLIB, "@rpath/libcore.dylib"), + dylibCommand(LC.LOAD_DYLIB, "\uFEFF@rpath/libcore.dylib"), + ], + }), + "Contents/Helpers/tool": machoImage({ + commands: [ + buildVersionCommand(1), + rpathCommand("\uFEFF@loader_path/../Frameworks"), + dylibCommand(LC.LOAD_DYLIB, "@rpath/libcore.dylib"), + ], + }), + }); + const output = await cli.run({ + arguments: ["trace-dylib-resolution", app, "--json"], + environment: ENVIRONMENT, + }); + expect(output.exitCode, output.stderr).toBe(0); + const trace = dylibResolutionResultSchema.parse( + parseEvidence(output.json).normalized_result, + ); + const ordinary = trace.edges.find( + ({ loader, install_name }) => + loader === "Contents/MacOS/App" && + install_name === "@rpath/libcore.dylib", + ); + expect(ordinary?.resolution).toMatchObject({ + status: "resolved", + image: "Contents/Frameworks/libcore.dylib", + }); + const marked = trace.edges.find( + ({ install_name }) => install_name === "\uFEFF@rpath/libcore.dylib", + ); + expect(marked).toBeDefined(); + expect(marked?.resolution.image).toBeNull(); + const helper = trace.edges.find( + ({ loader }) => loader === "Contents/Helpers/tool", + ); + expect(helper).toBeDefined(); + expect(helper?.resolution.image).toBeNull(); + }, +); + describe("trace-dylib-resolution CLI", () => { cliTest("traces every executable root of an app bundle", async ({ cli }) => { const app = await fixtureApp(); @@ -634,3 +685,79 @@ cliTest( } }, ); + +/** An app bundle whose roots sit among more files than one batch of magic reads. */ +const crowdedApp = async (): Promise => { + const app = await fixtureApp(); + await withBrokenHelper(app); + const resources: Record = {}; + for (let index = 0; index < 40; index += 1) + resources[`Contents/Resources/r${index % 4}/file${index}.dat`] = "x".repeat( + index % 5, + ); + for (const index of [0, 1, 2]) + resources[`Contents/Resources/r${index}/tool${index}`] = machoImage({ + commands: [buildVersionCommand(1)], + }); + resources["Contents/Resources/r3/libextra.dylib"] = machoImage({ + fileType: FILE_TYPE.dylib, + }); + await writeFiles(app, resources); + return app; +}; + +describe("trace-dylib-resolution root enumeration", () => { + cliTest( + "classifies every Mach-O among more files than one batch of magic reads", + async ({ cli }) => { + const app = await crowdedApp(); + const result = await cli.run({ + arguments: ["trace-dylib-resolution", app, "--json"], + environment: ENVIRONMENT, + }); + expect(result.exitCode, JSON.stringify(result.json)).toBe(0); + const trace = dylibResolutionResultSchema.parse( + (result.json as { normalized_result: unknown }).normalized_result, + ); + expect(trace.roots.map(({ image }) => image)).toEqual([ + "Contents/Helpers/tool", + "Contents/MacOS/App", + "Contents/Resources/r0/tool0", + "Contents/Resources/r1/tool1", + "Contents/Resources/r2/tool2", + ]); + expect(trace.coverage).toMatchObject({ + status: "partial", + unparsed_images: ["Contents/Helpers/broken"], + }); + }, + ); + + // This permission fixture requires POSIX mode bits and a non-root caller. + cliTest.skipIf(process.platform === "win32" || process.getuid?.() === 0)( + "reports an unreadable bundle file found while enumerating roots", + async ({ cli }) => { + const app = await crowdedApp(); + const secret = join(app, "Contents/Resources/r2/secret.dat"); + await writeFile(secret, "x"); + await chmod(secret, 0o000); + try { + const result = await cli.run({ + arguments: ["trace-dylib-resolution", app, "--json"], + environment: ENVIRONMENT, + }); + expect(result.exitCode).toBe(1); + expect(result.json).toMatchObject({ + details: { + reason: "unavailable", + detail: expect.stringContaining( + "Permission denied (EACCES) reading Contents/Resources/r2/secret.dat", + ), + }, + }); + } finally { + await chmod(secret, 0o644); + } + }, + ); +}); diff --git a/tests/boundary/cli/electronCli.test.ts b/tests/boundary/cli/electronCli.test.ts index 6ba962e9b4..8898ad9004 100644 --- a/tests/boundary/cli/electronCli.test.ts +++ b/tests/boundary/cli/electronCli.test.ts @@ -13,6 +13,12 @@ import { type FakeCdpBrowser, } from "../../fixtures/fakeCdpBrowser.js"; import { writeElectronBoundaryFixture } from "../../fixtures/electronBoundaryApplication.js"; +import { + SOURCE, + applicationFixture, + analyzeFixture, + electronRuntimeEvidence, +} from "../../fixtures/javascriptRuntimeReconciliation.js"; import { JAVASCRIPT_RUNTIME_RECONCILIATION_EXAMPLE } from "../../../src/contracts/javascript/javascriptRuntimeReconciliationExample.js"; const execute = promisify(execFile); @@ -219,8 +225,123 @@ describe("Electron CLI runtime reconciliation", () => { }, INTEGRATION_TEST_TIMEOUT_MS, ); + + it( + "keeps ambiguous runtime candidates unknown while unrelated scoped assets remain absent", + async () => { + const application = await applicationFixture(); + const assets = await createTestTempDirectory("rea-runtime-cli-assets-"); + const runtime = await createTestTempDirectory("rea-runtime-cli-live-"); + const inputPath = join(assets, "reconciliation.json"); + temporary.push(application, assets, runtime); + await Promise.all([ + writeFile(join(assets, "app.js"), SOURCE), + writeFile(join(assets, "absent.js"), "export const absent = true;\n"), + writeFile(join(runtime, "app.js"), SOURCE), + ]); + const rootMapping = [{ kind: "file-root", root: runtime }] as const; + const input = { + static_layers: [ + { + role: "application", + analysis: await analyzeFixture(application), + runtime_mappings: rootMapping, + }, + { + role: "assets", + analysis: await analyzeFixture(assets), + runtime_mappings: rootMapping, + }, + ], + runtime_observations: [ + electronRuntimeEvidence(runtime, SOURCE, { includeWorker: false }), + ], + }; + await writeFile(inputPath, JSON.stringify(input)); + const reconciled = await runCli( + ["reconcile-javascript-runtime", inputPath, "--json"], + process.env, + ); + + const normalized = + isRecord(reconciled) && isRecord(reconciled.normalized_result) + ? reconciled.normalized_result + : undefined; + const reconciliations = normalized?.reconciliations; + const loadStates = normalized?.static_load_states; + if (!Array.isArray(reconciliations) || !Array.isArray(loadStates)) + throw new Error("Missing runtime reconciliation output"); + const ambiguous = reconciliations.find( + (item) => + isRecord(item) && + item.status === "ambiguous" && + item.entity_kind === "script", + ); + if ( + !isRecord(ambiguous) || + !Array.isArray(ambiguous.candidate_static_nodes) + ) + throw new Error("Missing ambiguous runtime script"); + const candidateIds = ambiguous.candidate_static_nodes.flatMap((item) => + isRecord(item) && typeof item.static_node_id === "string" + ? [item.static_node_id] + : [], + ); + expect(candidateIds).toHaveLength(2); + const candidateStates = loadStates.filter( + (item) => + isRecord(item) && + typeof item.static_node_id === "string" && + candidateIds.includes(item.static_node_id), + ); + expect(candidateStates).toHaveLength(2); + expect( + candidateStates.every( + (item) => + isRecord(item) && + item.status === "unknown" && + item.reason === "ambiguous-runtime-candidate", + ), + ).toBe(true); + expect(loadStates).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + status: "not-observed-in-capture", + reason: "not-observed-in-bounded-capture", + }), + ]), + ); + + const singleLayerPath = join(application, "single-reconciliation.json"); + await writeFile( + singleLayerPath, + JSON.stringify({ + static_layers: [input.static_layers[0]], + runtime_observations: input.runtime_observations, + }), + ); + const uniquelyMatched = await runCli( + ["reconcile-javascript-runtime", singleLayerPath, "--json"], + process.env, + ); + expect(uniquelyMatched).toMatchObject({ + normalized_result: { + static_load_states: expect.arrayContaining([ + expect.objectContaining({ + status: "loaded", + reason: "runtime-script-correspondence", + }), + ]), + }, + }); + }, + INTEGRATION_TEST_TIMEOUT_MS, + ); }); +const isRecord = (value: unknown): value is Record => + typeof value === "object" && value !== null && !Array.isArray(value); + const runCli = async ( arguments_: readonly string[], environment: NodeJS.ProcessEnv, diff --git a/tests/boundary/cli/electronWrappedCallPaths.test.ts b/tests/boundary/cli/electronWrappedCallPaths.test.ts index 4dd0d02a85..b7ed2b67c5 100644 --- a/tests/boundary/cli/electronWrappedCallPaths.test.ts +++ b/tests/boundary/cli/electronWrappedCallPaths.test.ts @@ -9,6 +9,19 @@ import { javascriptApplicationAnalysisResultSchema } from "../../../src/domain/j import { applicationFeatureTraceResultSchema } from "../../../src/domain/javascript/javascriptFeatureTraceSchemas.js"; import { analyzeJavaScriptDirectory } from "../../support/cli/javascriptApplication.js"; +const sourcePositionSchema = z.object({ line: z.number(), column: z.number() }); +const callPathConditionSchema = z.object({ + expression: z.string(), + location: z.object({ + start: sourcePositionSchema, + end: sourcePositionSchema, + }), +}); +const callPathTerminalObservationSchema = z.object({ + properties: z.object({ conditions: z.array(callPathConditionSchema) }), + evidence: z.object({ coverage: z.object({ status: z.string() }) }), +}); + const analyze = async (cli: TestCli, extra = "") => { const root = await createTestTempDirectory("rea-electron-wrapped-paths-"); await writeFixtureFiles(root, { @@ -55,7 +68,8 @@ cliTest( expect( byId.get(edge.target_node_id)?.observations[0]?.properties.channel, ).toBe(edge.properties.channel); - expect(edge.properties.conditions).toEqual([ + expect(edge.properties.bridge_conditions).toEqual([]); + expect(edge.properties.handler_conditions).toEqual([ expect.objectContaining({ expression: "allowed(event)" }), ]); } @@ -122,6 +136,80 @@ cliTest( }, ); +cliTest( + "preserves early-return predicates on recovered bridge paths through CLI and stdio MCP", + async ({ cli }) => { + const cases = [ + { + name: "conditional early return", + body: 'if(x)return;e.contextBridge.exposeInMainWorld("api",{copy:()=>e.ipcRenderer.send("x")});', + conditions: ["!(x)"], + coverage: "partial", + }, + { + name: "equivalent nested branch", + body: 'if(!x){e.contextBridge.exposeInMainWorld("api",{copy:()=>e.ipcRenderer.send("x")});}', + conditions: ["!x"], + coverage: "partial", + }, + { + name: "definite early return", + body: 'return;e.contextBridge.exposeInMainWorld("api",{copy:()=>e.ipcRenderer.send("x")});', + conditions: [], + coverage: "complete", + }, + { + name: "unguarded exposure", + body: 'e.contextBridge.exposeInMainWorld("api",{copy:()=>e.ipcRenderer.send("x")});', + conditions: [], + coverage: "complete", + }, + ] as const; + + for (const surface of ["CLI", "stdio MCP"] as const) + for (const fixture of cases) { + const root = await createTestTempDirectory( + "rea-electron-early-return-", + ); + await writeFixtureFiles(root, { + "index.js": `const e=require("electron");function f(x){${fixture.body}}f(globalThis.blocked);`, + }); + const application = await analyzeJavaScriptDirectory( + surface, + cli, + root, + ); + const observations = application.graph.nodes.flatMap( + (node) => node.observations, + ); + const terminal = observations.find( + (observation) => + observation.properties.entry === "api.copy" && + observation.properties.step_kind === "terminal", + ); + if (fixture.name === "definite early return") { + expect(terminal).toBeUndefined(); + continue; + } + const terminalEvidence = + callPathTerminalObservationSchema.parse(terminal); + expect( + terminalEvidence.properties.conditions.map( + ({ expression }) => expression, + ), + ).toEqual(fixture.conditions); + expect(terminalEvidence.evidence.coverage.status).toBe( + fixture.coverage, + ); + expect(application.graph.coverage.status).toBe(fixture.coverage); + const source = observations.find( + (observation) => observation.properties.file_kind === "javascript", + ); + expect(source?.properties.parse_status).toBe(fixture.coverage); + } + }, +); + cliTest( "traces a nested contextBridge member through a unique IPC channel", async ({ cli }) => { @@ -307,6 +395,47 @@ cliTest( ); for (const surface of ["CLI", "stdio MCP"] as const) { + cliTest( + `retains explicit nested contextBridge members after an unknown spread through ${surface}`, + async ({ cli }) => { + const analyzeBridge = async (api: string) => { + const root = await createTestTempDirectory( + "rea-electron-spread-bridge-members-", + ); + await writeFixtureFiles(root, { + "preload.js": `const { contextBridge } = require("electron"); + const extra = getExtra(); + contextBridge.exposeInMainWorld("api", ${api});`, + }); + return analyzeJavaScriptDirectory(surface, cli, root); + }; + const withSpread = await analyzeBridge("{ ...extra, a: { b: 1 }, c: 2 }"); + const control = await analyzeBridge("{ a: { b: 1 }, c: 2 }"); + + const bridgeObservation = (application: typeof withSpread) => + application.graph.nodes + .filter((node) => node.kind === "context-bridge-api") + .flatMap((node) => node.observations) + .find((observation) => observation.properties.api_key === "api"); + expect(bridgeObservation(withSpread)).toMatchObject({ + properties: { + api_status: "dynamic", + members: ["a", "a.b", "c"], + unknown_members: 1, + }, + }); + expect(bridgeObservation(control)).toMatchObject({ + properties: { + api_status: "object-literal", + members: ["a", "a.b", "c"], + unknown_members: 0, + }, + }); + expect(withSpread.summary.exposed_api_members).toBe(3); + expect(control.summary.exposed_api_members).toBe(3); + }, + ); + cliTest( `preserves handler ambiguity and short-circuit factory paths through ${surface}`, async ({ cli }) => { @@ -495,3 +624,44 @@ cliTest( ).toEqual(["empty:key"]); }, ); + +cliTest( + "retains bridge and handler conditions on Electron IPC join edges", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-electron-join-conditions-"); + await writeFixtureFiles(root, { + "preload.js": `const e=require("electron"); + e.contextBridge.exposeInMainWorld("desktop", { + choose: flag + ? () => e.ipcRenderer.send("test:copy") + : () => e.ipcRenderer.send("test:cut") + });`, + "main.js": `const e=require("electron"); + if(!flag)e.ipcMain.on("test:copy",event=>allowed(event)&&event.sender.copy()); + if(flag)e.ipcMain.on("test:cut",event=>allowed(event)&&event.sender.cut());`, + }); + const { graph } = await analyzeJavaScriptDirectory("CLI", cli, root); + const joins = graph.edges.filter( + (edge) => + edge.evidence.extractor.operation === "join-electron-wrapper-paths", + ); + expect(joins).toHaveLength(2); + for (const [channel, bridgeCondition] of [ + ["test:copy", "flag"], + ["test:cut", "!(flag)"], + ]) { + const edge = joins.find( + (candidate) => candidate.properties.channel === channel, + ); + expect(edge?.properties.bridge_conditions).toEqual([ + expect.objectContaining({ expression: bridgeCondition }), + ]); + expect(edge?.properties.handler_conditions).toEqual([ + expect.objectContaining({ + expression: channel === "test:copy" ? "!flag" : "flag", + }), + expect.objectContaining({ expression: "allowed(event)" }), + ]); + } + }, +); diff --git a/tests/boundary/cli/fridaCli.test.ts b/tests/boundary/cli/fridaCli.test.ts index 317ca62152..0751e28d42 100644 --- a/tests/boundary/cli/fridaCli.test.ts +++ b/tests/boundary/cli/fridaCli.test.ts @@ -246,6 +246,11 @@ it.each([ "script", "Provide exactly one of --script or --script-file.", ], + [ + ["attach", "71", "--script", "send(1)", "--argument", "ignored"], + "argument", + "--argument is only valid in spawn mode; remove it when attaching to a process.", + ], ])( "names the rejected instrument-with-frida argument for %j", async (argv, path, message) => { diff --git a/tests/boundary/cli/gdbStartupCleanupCli.test.ts b/tests/boundary/cli/gdbStartupCleanupCli.test.ts new file mode 100644 index 0000000000..a241f91634 --- /dev/null +++ b/tests/boundary/cli/gdbStartupCleanupCli.test.ts @@ -0,0 +1,132 @@ +import pino from "pino"; +import { expect, it, vi } from "vitest"; + +import { AnalysisOutputError } from "../../../src/domain/analysisErrorCore.js"; +import { err } from "../../../src/domain/result.js"; +import { registerReverseEngineeringCommands } from "../../../src/cli/reverseEngineeringCommands.js"; +import { CLI_COMMANDS } from "../../../src/cliCommandNames.js"; +import type { CliInstance } from "../../../src/cli/types.js"; +import type { GdbSessionManager } from "../../../src/gdb/GdbSessionManager.js"; + +const registerGdbCommand = ( + createManager: () => GdbSessionManager, +): { + run(input: { + args: { command: string }; + options: { inferiorTty: string }; + }): Promise; +} => { + const commands = new Map< + string, + { + run(input: { + args: { command: string }; + options: { inferiorTty: string }; + }): Promise; + } + >(); + const cli = { + command: ( + name: string, + definition: { + run(input: { + args: { command: string }; + options: { inferiorTty: string }; + }): Promise; + }, + ) => { + commands.set(name, definition); + return cli; + }, + } as unknown as CliInstance; + registerReverseEngineeringCommands(cli, pino({ enabled: false }), {}, () => + createManager(), + ); + const command = commands.get(CLI_COMMANDS.debugWithGdb); + if (command === undefined) + throw new Error("GDB CLI command was not registered"); + return command; +}; + +it.each([ + "source /tmp/gdbinit", + "python print('x')", + "def custom-command", + "run > /tmp/out", + "set inferior-t /dev/null", + "with inferior-t /dev/null -- run", +])("rejects one-shot command %s before starting GDB", async (inputCommand) => { + const manager = { + start: vi.fn(), + closeAll: vi.fn(), + } as unknown as GdbSessionManager; + const command = registerGdbCommand(() => manager); + const result = await command.run({ + args: { command: inputCommand }, + options: { inferiorTty: "/dev/pts/42" }, + }); + expect(result).toMatchObject({ + code: "invalid_request", + details: { + issues: [ + expect.objectContaining({ + message: expect.stringContaining("persistent GDB session workflow"), + }), + ], + }, + }); + expect(manager.start).not.toHaveBeenCalled(); + expect(manager.closeAll).not.toHaveBeenCalled(); +}); + +it("retries retained GDB cleanup when CLI session startup fails", async () => { + const closeAll = vi.fn(async () => undefined); + const manager = { + start: vi.fn(async () => + err(new AnalysisOutputError("start_gdb_session", "startup failed")), + ), + closeAll, + } as unknown as GdbSessionManager; + const originalExitCode = process.exitCode; + const command = registerGdbCommand(() => manager); + + try { + const result = await command.run({ + args: { command: "info files" }, + options: { inferiorTty: "/dev/pts/42" }, + }); + expect(result).toMatchObject({ code: "unreadable_output" }); + expect(manager.start).toHaveBeenCalledOnce(); + expect(manager.start).toHaveBeenCalledWith(expect.anything(), { + inferiorTty: "/dev/pts/42", + }); + expect(closeAll).toHaveBeenCalledOnce(); + } finally { + process.exitCode = originalExitCode; + } +}); + +it("marks the CLI failed when retained GDB cleanup remains incomplete", async () => { + const manager = { + start: vi.fn(async () => + err(new AnalysisOutputError("start_gdb_session", "startup failed")), + ), + closeAll: vi.fn(async () => { + throw new Error("process still owned"); + }), + } as unknown as GdbSessionManager; + const originalExitCode = process.exitCode; + const command = registerGdbCommand(() => manager); + + try { + const result = await command.run({ + args: { command: "info files" }, + options: { inferiorTty: "/dev/pts/42" }, + }); + expect(result).toMatchObject({ code: "unreadable_output" }); + expect(manager.closeAll).toHaveBeenCalledOnce(); + expect(process.exitCode).toBe(1); + } finally { + process.exitCode = originalExitCode; + } +}); diff --git a/tests/boundary/cli/hopperLaunchFailureCli.test.ts b/tests/boundary/cli/hopperLaunchFailureCli.test.ts index 476f58f6be..29e132b8a0 100644 --- a/tests/boundary/cli/hopperLaunchFailureCli.test.ts +++ b/tests/boundary/cli/hopperLaunchFailureCli.test.ts @@ -1,11 +1,11 @@ import { chmod, copyFile, rm } from "node:fs/promises"; -import { describe, expect, onTestFinished } from "vitest"; +import { describe, expect } from "vitest"; import { z } from "zod"; import { workspaceCliTest } from "../../support/cli/workspaceCliFixture.js"; -import { acquireHopperTargetLease } from "../../../src/hopper/HopperTargetLease.js"; +import { acquireHopperTargetLease } from "../../../src/process/HopperLease.js"; const FAILED_LAUNCHER = `#!/bin/sh printf 'Cannot launch Hopper!\\nDocument state is unknown.\\n' >&2 @@ -39,7 +39,57 @@ describe("Hopper launcher failure through the one-shot CLI", () => { const targetPath = workspace.path("probe"); await copyFile("/usr/bin/true", targetPath); const loaderArgs = ["--probe"]; - onTestFinished(async () => { + try { + for (const attempt of ["first", "retry"]) { + const result = await cli.run({ + arguments: [ + "analyze", + targetPath, + "--provider", + "hopper", + "--json", + ], + environment: { + ...workspace.environment, + HOPPER_LAUNCHER_PATH: launcherPath, + HOPPER_LOADER_ARGS_JSON: JSON.stringify(loaderArgs), + REA_LOG_LEVEL: "silent", + }, + timeoutMs: 45_000, + }); + const retained = RetainedRuntime.safeParse(result.json); + if (retained.success) + await rm( + retained.data.details.diagnostics.cleanup_error.details + .diagnostics.runtime_root, + { recursive: true, force: true }, + ); + + expect(result.exitCode, attempt).toBe(1); + expect(result.json, attempt).toMatchObject({ + code: "cleanup_incomplete", + details: { + cleanup: "incomplete", + diagnostics: { + primary_error: { + code: "provider_unavailable", + details: { + stage: "launch", + launcher: { + exit_code: 2, + stderr: { + text: expect.stringContaining( + "Document state is unknown", + ), + }, + }, + }, + }, + }, + }, + }); + } + } finally { const recovered = await acquireHopperTargetLease({ targetPath, targetKind: "executable", @@ -48,50 +98,6 @@ describe("Hopper launcher failure through the one-shot CLI", () => { }); expect(recovered.acquired).toBe(true); if (recovered.acquired) await recovered.lease.release(); - }); - - for (const attempt of ["first", "retry"]) { - const result = await cli.run({ - arguments: ["analyze", targetPath, "--provider", "hopper", "--json"], - environment: { - ...workspace.environment, - HOPPER_LAUNCHER_PATH: launcherPath, - HOPPER_LOADER_ARGS_JSON: JSON.stringify(loaderArgs), - REA_LOG_LEVEL: "silent", - }, - timeoutMs: 45_000, - }); - const retained = RetainedRuntime.safeParse(result.json); - if (retained.success) - await rm( - retained.data.details.diagnostics.cleanup_error.details.diagnostics - .runtime_root, - { recursive: true, force: true }, - ); - - expect(result.exitCode, attempt).toBe(1); - expect(result.json, attempt).toMatchObject({ - code: "cleanup_incomplete", - details: { - cleanup: "incomplete", - diagnostics: { - primary_error: { - code: "provider_unavailable", - details: { - stage: "launch", - launcher: { - exit_code: 2, - stderr: { - text: expect.stringContaining( - "Document state is unknown", - ), - }, - }, - }, - }, - }, - }, - }); } }, ); diff --git a/tests/boundary/cli/idaRetainedCleanupCli.test.ts b/tests/boundary/cli/idaRetainedCleanupCli.test.ts index 826bb7f1d9..cca51531ca 100644 --- a/tests/boundary/cli/idaRetainedCleanupCli.test.ts +++ b/tests/boundary/cli/idaRetainedCleanupCli.test.ts @@ -65,11 +65,16 @@ describe("headless IDA timeout through the one-shot CLI", () => { expect(result.json).toMatchObject({ code: "cleanup_incomplete", details: { - provider_id: "ida", cleanup: "incomplete", + execution_failure: "provider_timeout", + operation: "idb_open", + timeout_ms: 500, resources: [expect.stringContaining(workspaceRoot)], cleanup_error: { + code: "cleanup_incomplete", details: { + provider_id: "ida", + operation: "close", diagnostics: { upstream_sessions: { state: "unverified", diff --git a/tests/boundary/cli/javascriptIntegrityPolicyCli.test.ts b/tests/boundary/cli/javascriptIntegrityPolicyCli.test.ts index 4883ac741b..1a79942bdb 100644 --- a/tests/boundary/cli/javascriptIntegrityPolicyCli.test.ts +++ b/tests/boundary/cli/javascriptIntegrityPolicyCli.test.ts @@ -15,6 +15,19 @@ const run = async (arguments_: readonly string[]): Promise => { return JSON.parse(stdout); }; +const runFailure = async (arguments_: readonly string[]): Promise => { + try { + await run(arguments_); + } catch (cause: unknown) { + if (typeof cause !== "object" || cause === null || !("stdout" in cause)) + throw cause; + const stdout = cause.stdout; + if (typeof stdout !== "string") throw cause; + return JSON.parse(stdout); + } + throw new Error("Expected the CLI command to fail"); +}; + it.each(["analyze-javascript-application", "analyze"])( "%s carries the integrity policy into shared JavaScript analysis", async (command) => { @@ -46,3 +59,48 @@ it("rejects the JavaScript integrity policy on binary-only inspection", async () stdout: expect.stringContaining("Unknown flag: --integrityPolicy"), }); }); + +it.each(["fail", "record-and-continue"] as const)( + "rejects the explicitly selected %s integrity policy on a binary analyze target", + async (policy) => { + const failure = await runFailure([ + "analyze", + "unused.bin", + "--integrity-policy", + policy, + ]); + expect(failure).toMatchObject({ + code: "invalid_request", + details: { + issues: [ + { + path: ["integrity_policy"], + reason: "invalid_value", + message: + "An integrity policy requires the static JavaScript ASAR/directory analysis lane; binary_overview accepts no input.", + }, + ], + }, + }); + }, +); + +it("still routes a binary analyze target without an integrity policy", async () => { + await expect(run(["analyze", "unused.bin"])).rejects.toMatchObject({ + code: 1, + stdout: expect.stringContaining("target does not exist"), + }); +}); + +it("advertises the optional integrity policy without an undefined union", async () => { + const schema = (await run(["analyze", "--schema"])) as { + options?: { properties?: { integrityPolicy?: unknown } }; + }; + const policy = schema.options?.properties?.integrityPolicy; + expect(policy).toMatchObject({ + type: "string", + enum: ["fail", "record-and-continue"], + description: "Behavior when declared application integrity does not match", + }); + expect(JSON.stringify(policy)).not.toContain("undefined"); +}); diff --git a/tests/boundary/cli/javascriptSelfImportRelationships.test.ts b/tests/boundary/cli/javascriptSelfImportRelationships.test.ts index 44b741d4cd..97d542fdd8 100644 --- a/tests/boundary/cli/javascriptSelfImportRelationships.test.ts +++ b/tests/boundary/cli/javascriptSelfImportRelationships.test.ts @@ -1,8 +1,13 @@ +import { createPackageWithOptions } from "@electron/asar"; +import { rm, writeFile } from "node:fs/promises"; +import { join } from "node:path"; + import { expect } from "vitest"; import { parseEvidence } from "../../../src/domain/evidence.js"; import { javascriptApplicationAnalysisResultSchema } from "../../../src/domain/javascript/javascriptApplicationAnalysis.js"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { createStrippedAsarAddon } from "../../fixtures/strippedAsarAddon.js"; import { cliTest } from "../../support/cli/cliFixture.js"; import { findExportNode, @@ -11,6 +16,161 @@ import { writeFixtureFiles, } from "../../support/javascriptApplicationFixture.js"; +cliTest( + "keeps missing same-container package manifests unavailable until restored", + async ({ cli }) => { + const root = await createTestTempDirectory( + "rea-missing-package-manifests-", + ); + const sourceRoot = join(root, "source"); + const archive = join(root, "app.asar"); + const relativeManifest = '{"main":"public.js"}'; + const exportsManifest = + '{"exports":{"import":"./import.js","require":"./require.js"}}'; + await writeFixtureFiles(sourceRoot, { + "main.cjs": 'require("./relative");\nrequire("fixture");\n', + "main.mjs": 'import "fixture";\n', + "relative/package.json": relativeManifest, + "relative/index.js": "export const fallback = true;\n", + "relative/public.js": "export const publicEntry = true;\n", + "node_modules/fixture/package.json": exportsManifest, + "node_modules/fixture/index.js": "export const fallback = true;\n", + "node_modules/fixture/import.js": "export const imported = true;\n", + "node_modules/fixture/require.js": "module.exports = true;\n", + }); + await createPackageWithOptions(sourceRoot, archive, { + unpack: "package.json", + }); + const unpacked = `${archive}.unpacked`; + const relativeManifestPath = join(unpacked, "relative/package.json"); + const exportsManifestPath = join( + unpacked, + "node_modules/fixture/package.json", + ); + await Promise.all([rm(relativeManifestPath), rm(exportsManifestPath)]); + + const analyze = async () => { + const output = await cli.run({ + arguments: ["analyze-javascript-application", archive, "--json"], + }); + expect(output.exitCode, output.stderr).toBe(0); + return javascriptApplicationAnalysisResultSchema.parse( + parseEvidence(output.json).normalized_result, + ); + }; + const unavailable = await analyze(); + const cjs = findSourceModule(unavailable.graph, "main.cjs"); + const mjs = findSourceModule(unavailable.graph, "main.mjs"); + for (const [source, specifier] of [ + [cjs, "./relative"], + [cjs, "fixture"], + [mjs, "fixture"], + ] as const) + expect( + unavailable.graph.edges.find( + (edge) => + edge.source_node_id === source?.node_id && + edge.properties.specifier === specifier, + )?.properties, + ).toMatchObject({ + resolution_status: "unavailable", + resolved_path: null, + }); + expect(unavailable.graph.coverage.status).toBe("partial"); + + await Promise.all([ + writeFile(relativeManifestPath, relativeManifest), + writeFile(exportsManifestPath, exportsManifest), + ]); + const restored = await analyze(); + for (const [sourcePath, specifier, target] of [ + ["main.cjs", "./relative", "relative/public.js"], + ["main.cjs", "fixture", "node_modules/fixture/require.js"], + ["main.mjs", "fixture", "node_modules/fixture/import.js"], + ] as const) + expect( + restored.graph.edges.find( + (edge) => + edge.source_node_id === + findSourceModule(restored.graph, sourcePath)?.node_id && + edge.properties.specifier === specifier, + )?.properties, + ).toMatchObject({ + resolution_status: "resolved", + resolved_path: target, + }); + }, +); + +cliTest( + "resolves ESM specifiers containing whitespace using URL text semantics", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-esm-whitespace-path-"); + const rawSpecifier = "./feature\tname.js"; + const encodedSpecifier = "./feature%09name.js"; + await writeFixtureFiles(root, { + "main.mjs": `import ${JSON.stringify(rawSpecifier)};\nimport ${JSON.stringify(encodedSpecifier)};\n`, + "featurename.js": "export const plain = 1;\n", + "feature\tname.js": "export const value = 1;\n", + }); + const output = await cli.run({ + arguments: ["analyze-javascript-application", root, "--json"], + }); + expect(output.exitCode, output.stderr).toBe(0); + const result = javascriptApplicationAnalysisResultSchema.parse( + parseEvidence(output.json).normalized_result, + ); + expect( + findGraphEdge( + result.graph, + findSourceModule(result.graph, "main.mjs"), + findSourceModule(result.graph, "featurename.js"), + "imports", + )?.properties, + ).toMatchObject({ + specifier: rawSpecifier, + resolved_path: "featurename.js", + }); + expect( + findGraphEdge( + result.graph, + findSourceModule(result.graph, "main.mjs"), + findSourceModule(result.graph, "feature\tname.js"), + "imports", + )?.properties, + ).toMatchObject({ + specifier: encodedSpecifier, + resolved_path: "feature\tname.js", + }); + }, +); + +cliTest( + "retains an inventoried native addon when its ASAR companion is missing", + async ({ cli }) => { + const { archive, addon } = await createStrippedAsarAddon(); + await rm(join(`${archive}.unpacked`, addon)); + const output = await cli.run({ + arguments: ["analyze-javascript-application", archive, "--json"], + }); + expect(output.exitCode, output.stderr).toBe(0); + const result = javascriptApplicationAnalysisResultSchema.parse( + parseEvidence(output.json).normalized_result, + ); + expect(result.graph.coverage.status).toBe("partial"); + expect( + result.graph.nodes.some( + (node) => + node.kind === "native-addon" && + node.observations.some( + ({ label, properties }) => + label === addon && properties.availability === "unavailable", + ), + ), + ).toBe(true); + }, +); + cliTest( "resolves emitted module imports without substituting Electron runtime paths", async ({ cli }) => { diff --git a/tests/boundary/cli/jsonInputSize.test.ts b/tests/boundary/cli/jsonInputSize.test.ts index 0a4aad8eed..33996b42f2 100644 --- a/tests/boundary/cli/jsonInputSize.test.ts +++ b/tests/boundary/cli/jsonInputSize.test.ts @@ -8,14 +8,14 @@ import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; import { cliTest } from "../../support/cli/cliFixture.js"; cliTest( - "rejects malformed oversized JSON through both CLI and Evidence readers", + "rejects malformed oversized input through CLI and Evidence readers", async ({ cli, processes }) => { const root = await createTestTempDirectory("rea-json-string-limit-"); const input = join(root, "oversized.json"); const file = await open(input, "wx"); try { - // Sparse zero bytes are valid UTF-8 but invalid JSON. Both readers - // stream the input and reject the syntax before assembling a value. + // Sparse zero bytes are valid UTF-8. Both streaming readers reject + // their invalid JSON syntax without assembling a whole-document string. await file.truncate(constants.MAX_STRING_LENGTH + 1); } finally { await file.close(); @@ -57,7 +57,7 @@ cliTest( cliTest( "traces valid Evidence beyond the whole-document string limit", - async ({ cli }) => { + async ({ cli, processes }) => { const root = await createTestTempDirectory("rea-json-streamed-evidence-"); const source = join(root, "application"); await mkdir(source); @@ -175,7 +175,7 @@ cliTest( cliTest( "returns a typed string assembly constraint under a bounded heap", - async ({ cli }) => { + async ({ cli, processes }) => { const root = await createTestTempDirectory("rea-json-string-headroom-"); const input = join(root, "long-string.json"); const file = await open(input, "wx"); @@ -207,5 +207,21 @@ cliTest( action: expect.stringContaining("re-analyze a smaller selection"), }, }); + const evidence = await processes.run(process.execPath, [ + "--max-old-space-size=768", + "--input-type=module", + "-e", + `import { readJsonFile } from "./dist/application/JsonFiles.js"; + import { projectAnalysisError } from "./dist/domain/analysisErrorProjection.js"; + const result = await readJsonFile(process.argv[1]); + console.log(JSON.stringify(result.ok ? result : projectAnalysisError(result.error)));`, + input, + ]); + expect(JSON.parse(evidence.stdout)).toMatchObject({ + code: "resource_constraint", + details: { operation: "read_evidence_file", resource: "memory" }, + }); + expect(evidence.stderr).toBe(""); + expect(evidence.exitCode).toBe(0); }, ); diff --git a/tests/boundary/cli/keyedArchiveXmlExtensionCli.test.ts b/tests/boundary/cli/keyedArchiveXmlExtensionCli.test.ts new file mode 100644 index 0000000000..6ed0292284 --- /dev/null +++ b/tests/boundary/cli/keyedArchiveXmlExtensionCli.test.ts @@ -0,0 +1,38 @@ +import { writeFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { build } from "plist"; +import { expect } from "vitest"; + +import { parseEvidence } from "../../../src/domain/evidence.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { cliTest } from "../../support/cli/cliFixture.js"; + +cliTest( + "compiled CLI inspects an XML keyed archive with a non-plist suffix", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-keyed-xml-extension-cli-"); + const path = join(root, "Model.archive"); + await writeFile( + path, + build({ + $archiver: "NSKeyedArchiver", + $version: 100000, + $objects: ["$null", "leaf"], + $top: { root: { UID: 1 } }, + }), + ); + const output = await cli.run({ + arguments: ["inspect-keyed-archive", path, "--json"], + environment: { REA_LOG_LEVEL: "silent", REA_ANALYSIS_PROVIDER: "auto" }, + }); + expect(output.exitCode, output.stderr).toBe(0); + expect(parseEvidence(output.json).normalized_result).toMatchObject({ + archive_path: "Model.archive", + archive_format: "xml-plist", + objects: expect.arrayContaining([ + expect.objectContaining({ value: "leaf" }), + ]), + }); + }, +); diff --git a/tests/boundary/cli/nativeDemangleCli.test.ts b/tests/boundary/cli/nativeDemangleCli.test.ts index fe905cbbf7..431cf02eff 100644 --- a/tests/boundary/cli/nativeDemangleCli.test.ts +++ b/tests/boundary/cli/nativeDemangleCli.test.ts @@ -18,10 +18,10 @@ const evidenceTarget = async () => { return path; }; -describe.skipIf(process.platform !== "darwin")( - "native Swift demangling CLI with line-terminator characters", - () => { - cliTest("keeps a carriage return in the symbol", async ({ cli }) => { +describe("Swift demangling CLI with line-terminator characters", () => { + cliTest.skipIf(process.platform !== "darwin")( + "keeps a carriage return in the symbol", + async ({ cli }) => { const result = await cli.run({ arguments: [ "demangle-swift", @@ -41,26 +41,32 @@ describe.skipIf(process.platform !== "darwin")( ], }, }); - }); + }, + ); - cliTest("rejects a symbol spanning lines as input", async ({ cli }) => { - const missingTarget = join( - await createTestTempDirectory("rea-demangle-invalid-input-"), - "absent", - ); - const result = await cli.run({ - arguments: [ - "demangle-swift", - missingTarget, - "$s4main3FooV", - "$s4main\n3FooV", - "--json", - ], - environment, - timeoutMs: 20_000, - }); - expect(result.exitCode).not.toBe(0); - expect(result.json).toMatchObject({ code: "VALIDATION_ERROR" }); + cliTest("rejects a symbol spanning lines as input", async ({ cli }) => { + const missingTarget = join( + await createTestTempDirectory("rea-demangle-invalid-input-"), + "absent", + ); + const result = await cli.run({ + arguments: [ + "demangle-swift", + missingTarget, + "$s4main3FooV", + "$s4main\n3FooV", + "--json", + ], + environment, + timeoutMs: 20_000, }); - }, -); + expect(result.signal).toBeNull(); + expect(result.exitCode).toBe(1); + expect(result.json).toMatchObject({ + code: "VALIDATION_ERROR", + fieldErrors: expect.arrayContaining([ + expect.objectContaining({ path: "symbols.1", missing: false }), + ]), + }); + }); +}); diff --git a/tests/boundary/cli/nativeDispatchPropertyAttributes.test.ts b/tests/boundary/cli/nativeDispatchPropertyAttributes.test.ts new file mode 100644 index 0000000000..7aa614afc4 --- /dev/null +++ b/tests/boundary/cli/nativeDispatchPropertyAttributes.test.ts @@ -0,0 +1,114 @@ +import { writeFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { describe, expect } from "vitest"; + +import { machoWithPropertyAttributes } from "../../fixtures/objcPropertyMacho.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { cliTest } from "../../support/cli/cliFixture.js"; + +describe.skipIf(process.platform !== "darwin")( + "native dispatch property attributes", + () => { + cliTest.for([ + ["complete nonatomic", "Tq,N", "nonatomic", false, "complete"], + ["complete default", 'T@"NSString",C', "atomic", false, "complete"], + ["unfinished quote", 'T@"NSString', null, null, "partial"], + ["unfinished quote after R", 'R,T@"NSString', null, true, "partial"], + ] as const)( + "reports established property flags only for $0 attributes", + async ([, attributes, atomicity, readonly, facet], { cli }) => { + const directory = await createTestTempDirectory("rea-objc-property-"); + const path = join(directory, "fixture.bin"); + await writeFile(path, machoWithPropertyAttributes(attributes)); + const output = await cli.run({ + arguments: ["inspect-native-dispatch-metadata", path, "--json"], + environment: { REA_LOG_LEVEL: "silent" }, + }); + expect(output.exitCode).toBe(0); + const property = { + atomicity, + is_readonly: readonly, + attributes: expect.arrayContaining([ + { name: "T", value: expect.any(String) }, + ]), + }; + expect(output.json).toMatchObject({ + normalized_result: { + result: { + objc_classes: [ + { + name: "Fixture", + properties: [ + { name: "first", ...property }, + { name: "second", ...property }, + ], + }, + ], + coverage: expect.arrayContaining([ + expect.objectContaining({ + facet: "objc_properties_categories", + status: facet, + reason: + facet === "partial" + ? expect.stringContaining( + "property_attributes_incomplete", + ) + : null, + }), + ]), + }, + }, + }); + }, + ); + }, +); + +describe.skipIf(process.platform !== "darwin")( + "native dispatch property record isolation", + () => { + cliTest.for([ + [ + "an unmapped name pointer", + { secondNameOffset: 0x2000 }, + "first", + "0x100000518", + ], + ["an empty name", { firstName: "" }, "second", "0x100000508"], + ] as const)( + "retains the valid sibling of a property with $0", + async ([, layout, kept, location], { cli }) => { + const directory = await createTestTempDirectory("rea-objc-property-"); + const path = join(directory, "fixture.bin"); + await writeFile(path, machoWithPropertyAttributes("Tq,N", layout)); + const output = await cli.run({ + arguments: ["inspect-native-dispatch-metadata", path, "--json"], + environment: { REA_LOG_LEVEL: "silent" }, + }); + expect(output.exitCode).toBe(0); + expect(output.json).toMatchObject({ + normalized_result: { + result: { + objc_classes: [ + { + name: "Fixture", + properties: [{ name: kept, atomicity: "nonatomic" }], + }, + ], + coverage: expect.arrayContaining([ + expect.objectContaining({ + facet: "objc_properties_categories", + status: "partial", + reason: expect.stringContaining( + `property_decode_failed at ${location}: Fixture: property`, + ), + }), + ]), + }, + }, + }); + }, + ); + }, +); diff --git a/tests/boundary/cli/nativePlistCli.test.ts b/tests/boundary/cli/nativePlistCli.test.ts index 50ed030f33..62ec9ce985 100644 --- a/tests/boundary/cli/nativePlistCli.test.ts +++ b/tests/boundary/cli/nativePlistCli.test.ts @@ -12,13 +12,22 @@ import { cliTest } from "../../support/cli/cliFixture.js"; describe.skipIf(process.platform !== "darwin")( "native plist CLI selection", () => { + // Targets are app-relative: the bundle, its program file, or a plist file. cliTest.for([ - ["default", undefined, "default"], - ["alternate", "Contents/Alternate.plist", "alternate"], - ["missing", "Contents/Missing.plist", undefined], + ["default", "", undefined, "default"], + ["alternate", "", "Contents/Alternate.plist", "alternate"], + ["missing", "", "Contents/Missing.plist", undefined], + ["program default", "Contents/MacOS/App", undefined, "default"], + [ + "program alternate", + "Contents/MacOS/App", + "Contents/Alternate.plist", + "alternate", + ], + ["plist file", "Contents/Alternate.plist", undefined, "alternate"], ] as const)( "honors the $0 plist selection", - async ([, selected, identifier], { cli }) => { + async ([, target, selected, identifier], { cli }) => { const directory = await createTestTempDirectory("rea-plist-cli-"); const app = join(directory, "Example.app"); const contents = join(app, "Contents"); @@ -39,7 +48,7 @@ describe.skipIf(process.platform !== "darwin")( const result = await cli.run({ arguments: [ "inspect-plist", - app, + join(app, target), ...(selected === undefined ? [] : ["--relative-path", selected]), "--json", ], @@ -55,7 +64,11 @@ describe.skipIf(process.platform !== "darwin")( expect(result.exitCode).toBe(0); expect(result.json).toMatchObject({ normalized_result: { - source_path: join(app, selected ?? "Contents/Info.plist"), + source_path: join( + app, + selected ?? + (target.endsWith(".plist") ? target : "Contents/Info.plist"), + ), value: { CFBundleIdentifier: identifier }, }, }); @@ -83,6 +96,8 @@ describe.skipIf(process.platform !== "darwin")( "CFBundleIdentifiercom.example.types", "BlobAAEC", "Built2020-01-01T00:00:00Z", + // plutil writes years past 9999 unsigned, which the decoder rejects. + "Far11507-08-17T05:20:00Z", "Rationan", "Nested", "Ratio0.5", @@ -117,6 +132,7 @@ describe.skipIf(process.platform !== "darwin")( $plist_type: "date", iso8601: "2020-01-01T00:00:00.000Z", }, + Far: { $plist_type: "date", iso8601: null }, Ratio: { $plist_type: "real", value: null }, Nested: [ { @@ -135,6 +151,7 @@ describe.skipIf(process.platform !== "darwin")( limitations: [ expect.stringContaining("XML conversion"), expect.stringContaining("2 non-finite real"), + expect.stringContaining("1 date value(s)"), ], }, }); diff --git a/tests/boundary/cli/nativePlistSelectionFailureCli.test.ts b/tests/boundary/cli/nativePlistSelectionFailureCli.test.ts index e14ca4cc38..640e12b635 100644 --- a/tests/boundary/cli/nativePlistSelectionFailureCli.test.ts +++ b/tests/boundary/cli/nativePlistSelectionFailureCli.test.ts @@ -122,7 +122,7 @@ describe.skipIf(process.platform !== "darwin")( details: { operation: "inspect_plist", reason: expect.stringContaining( - `no readable Contents/Info.plist at ${join(directory, "Contents/Info.plist")}: no file exists there.`, + `no readable default plist at ${join(directory, "Contents/Info.plist")}: no file exists there.`, ), }, }); diff --git a/tests/boundary/cli/reconstructionObligationLedgerCli.test.ts b/tests/boundary/cli/reconstructionObligationLedgerCli.test.ts index 63f3e94c41..eee6b3df1a 100644 --- a/tests/boundary/cli/reconstructionObligationLedgerCli.test.ts +++ b/tests/boundary/cli/reconstructionObligationLedgerCli.test.ts @@ -3,11 +3,86 @@ import { promisify } from "node:util"; import { describe, expect, it } from "vitest"; +import { createEvidence } from "../../../src/domain/evidence.js"; import { createEvidenceBundle } from "../../../src/domain/evidenceBundle.js"; const execute = promisify(execFile); describe("reconstruction obligation ledger CLI parity", () => { + it("reports non-canonical authenticated bundles as caller input", async () => { + const records = ["first", "second"].map((label) => + createEvidence( + undefined, + { id: "fixture", name: "Fixture", version: "1" }, + { + operation: "observe", + parameters: { label }, + result: { label }, + }, + ), + ); + const canonical = createEvidenceBundle(records); + const input = { + evidence_bundle: canonical, + reviewed_obligations: [], + manifest: { bindings: [], contradictions: [] }, + }; + const accepted = await execute( + process.execPath, + [ + "scripts/rea.mjs", + "build-reconstruction-obligation-ledger", + JSON.stringify(input), + "--json", + ], + { cwd: process.cwd(), env: process.env, maxBuffer: 16 * 1_024 * 1_024 }, + ); + expect(JSON.parse(accepted.stdout)).toMatchObject({ + normalized_result: { status: "unknown", summary: { total: 0 } }, + }); + input.evidence_bundle = { + ...canonical, + records: [...canonical.records].reverse(), + }; + let stdout: string; + try { + await execute( + process.execPath, + [ + "scripts/rea.mjs", + "build-reconstruction-obligation-ledger", + JSON.stringify(input), + "--json", + ], + { cwd: process.cwd(), env: process.env, maxBuffer: 16 * 1_024 * 1_024 }, + ); + throw new Error("Expected non-canonical bundle input to fail"); + } catch (cause: unknown) { + if ( + !(cause instanceof Error) || + !("stdout" in cause) || + typeof cause.stdout !== "string" + ) + throw cause; + stdout = cause.stdout; + } + expect(JSON.parse(stdout)).toMatchObject({ + code: "invalid_request", + category: "invalid_input", + retryable: true, + details: { + operation: "build_reconstruction_obligation_ledger", + issues: [ + expect.objectContaining({ + message: expect.stringContaining( + "Evidence bundle manifests are not canonical", + ), + }), + ], + }, + }); + }, 20_000); + it("builds the same Evidence-backed empty ledger through the CLI", async () => { const input = { evidence_bundle: createEvidenceBundle([]), diff --git a/tests/boundary/cli/referenceSourceDirectoryImports.test.ts b/tests/boundary/cli/referenceSourceDirectoryImports.test.ts index 6270855cbc..0d2f72421c 100644 --- a/tests/boundary/cli/referenceSourceDirectoryImports.test.ts +++ b/tests/boundary/cli/referenceSourceDirectoryImports.test.ts @@ -19,6 +19,7 @@ describe.skipIf(process.platform === "win32")( "rea-reference-directory-imports-", ); const root = join(workspace, "source"); + const outside = join(workspace, "outside", "index.js"); const imports = [ ["slash.cjs", "./lib/", "lib/index.js"], ["dot.cjs", "./lib/.", "lib/index.js"], @@ -29,11 +30,18 @@ describe.skipIf(process.platform === "win32")( ["direct.cjs", "./lib.js", "lib.js"], ["fallback.cjs", "./fallback", "fallback/index.js"], ["legacy-order.cjs", "./dep", "dep/index.js"], + ["bare-index.cjs", "./bare", "bare.js"], + ["package-main.cjs", "./pkg", "pkg/actual.js"], + ["package-main-fallback.cjs", "./stale", "stale/index.js"], + ["package-main-slash.cjs", "./slashed", "slashed/lib.js"], ] as const; const unresolved = [ ["missing.cjs", "./missing/", "missing/"], ["escape.cjs", "../outside/", "../outside/"], ["main.ts", "./folder.js/.", "folder.js"], + ["only-bare.cjs", "./only/", "only/"], + ["invalid-manifest.cjs", "./broken", "broken"], + ["absolute-main.cjs", "./absolute", outside], ] as const; const files = new Map([ ["index.js", 'module.exports = "root index";\n'], @@ -48,6 +56,25 @@ describe.skipIf(process.platform === "win32")( 'export const value = "emitted filename counterpart";\n', ], ["missing.js", 'module.exports = "not a directory";\n'], + ["bare.js", 'module.exports = "file before directory";\n'], + ["bare/index", 'module.exports = "extensionless index";\n'], + ["only/index", 'module.exports = "extensionless index";\n'], + // A package main outranks the index; a missing main falls back to it. + ["pkg/package.json", '{"main":"actual.js"}\n'], + ["pkg/actual.js", 'module.exports = "declared main";\n'], + ["pkg/index.js", 'module.exports = "ignored index";\n'], + ["stale/package.json", '{"main":"gone.js"}\n'], + ["stale/index.js", 'module.exports = "legacy fallback";\n'], + ["broken/package.json", '{"main":\n'], + ["broken/index.js", 'module.exports = "unknown selection";\n'], + // Node resolves main with path.resolve: no trailing-slash directory + // intent, and an absolute main leaves the imported root. + ["slashed/package.json", '{"main":"lib/"}\n'], + ["slashed/lib.js", 'module.exports = "file main";\n'], + ["slashed/lib/index.js", 'module.exports = "not selected";\n'], + ["slashed/index.js", 'module.exports = "not selected";\n'], + ["absolute/package.json", `${JSON.stringify({ main: outside })}\n`], + ["absolute/index.js", 'module.exports = "not selected";\n'], ...[...imports, ...unresolved].map( ([path, specifier]) => [path, `require(${JSON.stringify(specifier)});\n`] as const, @@ -60,7 +87,6 @@ describe.skipIf(process.platform === "win32")( await writeFile(target, contents); }), ); - const outside = join(workspace, "outside", "index.js"); await mkdir(dirname(outside), { recursive: true }); await writeFile(outside, 'module.exports = "outside import root";\n'); @@ -75,9 +101,18 @@ describe.skipIf(process.platform === "win32")( expect(() => createRequire(join(root, "main.ts")).resolve("./folder.js/."), ).toThrow(); + expect(() => + createRequire(join(root, "only-bare.cjs")).resolve("./only/"), + ).toThrow(); + expect(() => + createRequire(join(root, "invalid-manifest.cjs")).resolve("./broken"), + ).toThrow(); expect( createRequire(join(root, "escape.cjs")).resolve("../outside/"), ).toBe(outside); + expect( + createRequire(join(root, "absolute-main.cjs")).resolve("./absolute"), + ).toBe(outside); const result = await cli.run({ arguments: ["import-reference-source", root, "--json"], diff --git a/tests/boundary/cli/referenceSourceExclusions.test.ts b/tests/boundary/cli/referenceSourceExclusions.test.ts index cbc0632856..87b8541166 100644 --- a/tests/boundary/cli/referenceSourceExclusions.test.ts +++ b/tests/boundary/cli/referenceSourceExclusions.test.ts @@ -1,4 +1,10 @@ -import { mkdir, writeFile } from "node:fs/promises"; +import { + mkdir, + readFile, + readlink, + symlink, + writeFile, +} from "node:fs/promises"; import { join } from "node:path"; import { describe, expect } from "vitest"; @@ -10,6 +16,56 @@ import { cliTest } from "../../support/cli/cliFixture.js"; describe.skipIf(process.platform === "win32")( "compiled reference-source exclusion provenance", () => { + cliTest( + "imports source beside a link into a default-excluded directory", + async ({ cli }) => { + const root = await createTestTempDirectory( + "rea-reference-link-exclusion-", + ); + await mkdir(join(root, "dist")); + const source = "export const value = 1;\n"; + await writeFile(join(root, "source.ts"), source); + await symlink("dist", join(root, "alias")); + const result = await cli.run({ + arguments: ["import-reference-source", root, "--json"], + cwd: root, + environment: { + HOME: root, + USERPROFILE: root, + XDG_CONFIG_HOME: root, + XDG_CACHE_HOME: root, + }, + }); + expect(result.exitCode).toBe(0); + const graph = historicalSourceGraphSchema.parse(result.json); + expect(graph.inventory_state).toBe("partial"); + expect(graph.exclusions).toContainEqual({ + path: "dist", + entry_kind: "directory", + reason: "default-ignored", + pattern: "dist/", + }); + expect(graph.entries).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + path: "alias", + kind: "symlink", + target: "dist", + target_state: "internal", + }), + expect.objectContaining({ + path: "source.ts", + kind: "file", + content_state: "hashed", + }), + ]), + ); + expect(graph.entries.some(({ path }) => path === "dist")).toBe(false); + expect(await readFile(join(root, "source.ts"), "utf8")).toBe(source); + expect(await readlink(join(root, "alias"))).toBe("dist"); + }, + ); + cliTest( "reports project, default, and explicit secret exclusions", async ({ cli }) => { @@ -42,16 +98,19 @@ describe.skipIf(process.platform === "win32")( expect.arrayContaining([ { path: "project.tmp", + entry_kind: "file", reason: "project-ignored", pattern: "project.tmp", }, { path: "logs/application.log", + entry_kind: "file", reason: "default-ignored", pattern: "*.log", }, { path: "private.txt", + entry_kind: "file", reason: "configured-secret", pattern: "private.txt", }, diff --git a/tests/boundary/cli/referenceSourceUtf8.test.ts b/tests/boundary/cli/referenceSourceUtf8.test.ts index 06093f427e..0a1243a913 100644 --- a/tests/boundary/cli/referenceSourceUtf8.test.ts +++ b/tests/boundary/cli/referenceSourceUtf8.test.ts @@ -1,5 +1,11 @@ import { createHash } from "node:crypto"; -import { writeFile } from "node:fs/promises"; +import { + mkdir, + readFile, + readlink, + symlink, + writeFile, +} from "node:fs/promises"; import { join } from "node:path"; import { expect } from "vitest"; @@ -39,7 +45,9 @@ cliTest( arguments: ["import-reference-source", root, "--json"], environment: { HOME: root, XDG_CONFIG_HOME: root, XDG_CACHE_HOME: root }, }); - expect(result.exitCode).toBe(0); + expect(result.exitCode, result.stderr + JSON.stringify(result.json)).toBe( + 0, + ); const graph = historicalSourceGraphSchema.parse(result.json); expect(graph.parse_failures).toEqual([ { @@ -75,3 +83,139 @@ cliTest( ); }, ); + +cliTest.skipIf(process.platform === "win32")( + "preserves historical literal-backslash entries and their exact owning import base", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-reference-literal-paths-"); + const config = await createTestTempDirectory( + "rea-reference-literal-config-", + ); + await mkdir(join(root, "literal")); + await mkdir(join(root, "literal\\dir")); + const sources = new Map([ + ["literal\\name.js", "export const value = 'literal';\n"], + ["literal/name.js", "export const value = 'slash';\n"], + ["literal\\dir/main.js", 'import "./dep.js";\n'], + ["literal\\dir/dep.js", "export const value = 'owned';\n"], + ]); + for (const [path, bytes] of sources) + await writeFile(join(root, path), bytes); + const result = await cli.run({ + arguments: ["import-reference-source", root, "--json"], + environment: { + HOME: config, + XDG_CONFIG_HOME: config, + XDG_CACHE_HOME: config, + }, + }); + expect(result.exitCode, result.stderr + JSON.stringify(result.json)).toBe( + 0, + ); + const graph = historicalSourceGraphSchema.parse(result.json); + for (const [path, bytes] of sources) + expect(graph.entries).toContainEqual( + expect.objectContaining({ + path, + sha256: createHash("sha256").update(bytes).digest("hex"), + content_state: "hashed", + }), + ); + for (const [path, bytes] of sources) + expect(await readFile(join(root, path), "utf8")).toBe(bytes); + expect(graph.relationships).toContainEqual({ + from_path: "literal\\dir/main.js", + to: "literal\\dir/dep.js", + kind: "imports", + resolution: "internal", + parse_state: "parsed", + }); + }, +); + +cliTest.skipIf(process.platform === "win32")( + "preserves an observed historical symlink target with a literal backslash", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-reference-literal-link-"); + const config = await createTestTempDirectory( + "rea-reference-literal-link-config-", + ); + await writeFile( + join(root, "literal\\name.js"), + "export const value = 42;\n", + ); + await symlink("literal\\name.js", join(root, "link.js")); + const result = await cli.run({ + arguments: ["import-reference-source", root, "--json"], + environment: { + HOME: config, + XDG_CONFIG_HOME: config, + XDG_CACHE_HOME: config, + }, + }); + expect(result.exitCode, result.stderr + JSON.stringify(result.json)).toBe( + 0, + ); + const graph = historicalSourceGraphSchema.parse(result.json); + expect(await readlink(join(root, "link.js"))).toBe("literal\\name.js"); + expect(await readFile(join(root, "literal\\name.js"), "utf8")).toBe( + "export const value = 42;\n", + ); + expect(graph.entries).toContainEqual( + expect.objectContaining({ + kind: "symlink", + path: "link.js", + target: "literal\\name.js", + target_state: "internal", + }), + ); + }, +); + +cliTest.skipIf(process.platform === "win32")( + "imports an inventory-owned POSIX UNC-looking symlink target", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-reference-unc-literal-"); + const config = await createTestTempDirectory("rea-reference-unc-config-"); + const literal = "\\\\server\\share.js"; + const literalBytes = "export const value = 'literal';\n"; + const slashBytes = "export const value = 'slash';\n"; + await mkdir(join(root, "server")); + await writeFile(join(root, literal), literalBytes); + await writeFile(join(root, "server", "share.js"), slashBytes); + await symlink(literal, join(root, "link.js")); + const result = await cli.run({ + arguments: ["import-reference-source", root, "--json"], + environment: { + HOME: config, + XDG_CONFIG_HOME: config, + XDG_CACHE_HOME: config, + }, + }); + expect(result.exitCode, result.stderr + JSON.stringify(result.json)).toBe( + 0, + ); + const graph = historicalSourceGraphSchema.parse(result.json); + expect(graph.entries).toContainEqual( + expect.objectContaining({ + kind: "symlink", + path: "link.js", + target: literal, + target_state: "internal", + }), + ); + for (const [path, bytes] of [ + [literal, literalBytes], + ["server/share.js", slashBytes], + ]) { + expect(graph.entries).toContainEqual( + expect.objectContaining({ + path, + sha256: createHash("sha256").update(bytes!).digest("hex"), + }), + ); + expect(await readFile(join(root, path!), "utf8")).toBe(bytes); + } + expect(await readlink(join(root, "link.js"))).toBe(literal); + }, +); diff --git a/tests/boundary/cli/reverseEngineeringCommands.test.ts b/tests/boundary/cli/reverseEngineeringCommands.test.ts new file mode 100644 index 0000000000..9af7217706 --- /dev/null +++ b/tests/boundary/cli/reverseEngineeringCommands.test.ts @@ -0,0 +1,429 @@ +import pino from "pino"; +import { Cli, z } from "incur"; +import { afterEach, expect, it, vi } from "vitest"; +import { rm } from "node:fs/promises"; + +import { registerReverseEngineeringCommands } from "../../../src/cli/reverseEngineeringCommands.js"; +import { CLI_COMMANDS } from "../../../src/cliCommandNames.js"; +import { + AnalysisCancelledError, + AnalysisTimeoutError, + AnalysisOutputError, +} from "../../../src/domain/analysisErrorCore.js"; +import { projectAnalysisError } from "../../../src/domain/analysisErrorProjection.js"; +import { + createEvidence, + evidenceSchema, +} from "../../../src/domain/evidence.js"; +import { err, ok } from "../../../src/domain/result.js"; +import { RizinDebugSessionManager } from "../../../src/rizin/RizinDebugSessionManager.js"; +import { ReverseEngineeringService as CoreReverseEngineeringService } from "../../../src/application/reverse/ReverseEngineeringService.js"; +import { createReverseEngineeringCommandPort } from "../../../src/composition/reverseEngineering.js"; +import { OwnedCommandFailure } from "../../../src/process/OwnedCommand.js"; +import type { ProviderProcessSupervisor } from "../../../src/process/ProviderProcess.js"; +import { + createArtifact, + providerSnapshot, +} from "../../support/reverseEngineeringCompositionFixtures.js"; +import type { CliInstance } from "../../../src/cli/types.js"; +import type { ReverseEngineeringService } from "../../../src/application/reverse/ReverseEngineeringService.js"; + +const originalExitCode = process.exitCode; + +afterEach(() => { + process.exitCode = originalExitCode; +}); + +const observation = (completionStatus: "complete" | "unknown") => + createEvidence( + undefined, + { id: "gnu.objdump", name: "GNU objdump", version: null }, + { + predicateType: "rea.reverse-engineering.command-observation", + operation: "inspect_with_objdump", + parameters: { operation: "file_headers" }, + result: { completion_status: completionStatus }, + rawResult: { completion_status: completionStatus }, + }, + ); + +const runObjdump = async ( + service: Partial, + path = "/tmp/sample", +) => { + const commands = new Map< + string, + { + run(input: { + args: { path: string; operation: string }; + options: { followDebugLinks: boolean }; + }): Promise; + } + >(); + const cli = { + command: ( + name: string, + definition: { + run(input: { + args: { path: string; operation: string }; + options: { followDebugLinks: boolean }; + }): Promise; + }, + ) => { + commands.set(name, definition); + return cli; + }, + } as unknown as CliInstance; + registerReverseEngineeringCommands( + cli, + pino({ enabled: false }), + {}, + undefined, + () => service as ReverseEngineeringService, + ); + const command = commands.get(CLI_COMMANDS.inspectWithObjdump); + if (command === undefined) + throw new Error("objdump CLI command was not registered"); + return command.run({ + args: { path, operation: "file_headers" }, + options: { followDebugLinks: false }, + }); +}; + +it("lists objdump architectures without inventing an artifact path", async () => { + const evidence = observation("complete"); + const inspectWithObjdump = vi.fn(async (_input: unknown) => ok(evidence)); + const commands = new Map< + string, + { + readonly args?: { safeParse(input: unknown): { success: boolean } }; + run(input: { args?: unknown; options?: unknown }): Promise; + } + >(); + const cli = { + command: (_name: string, definition: unknown) => { + commands.set( + _name, + definition as { + readonly args?: { + safeParse(input: unknown): { success: boolean }; + }; + run(input: { args?: unknown; options?: unknown }): Promise; + }, + ); + return cli; + }, + } as unknown as CliInstance; + registerReverseEngineeringCommands( + cli, + pino({ enabled: false }), + {}, + undefined, + () => + ({ + inspectWithObjdump, + close: vi.fn(async () => undefined), + }) as unknown as ReverseEngineeringService, + ); + const command = commands.get(CLI_COMMANDS.listObjdumpArchitectures); + if (command === undefined) + throw new Error("architecture command was not registered"); + + const result = await command.run({}); + + expect(result).toEqual(evidence); + expect(inspectWithObjdump).toHaveBeenCalledWith( + { operation: "architectures" }, + expect.objectContaining({ signal: expect.any(AbortSignal) }), + ); + const artifactCommand = commands.get(CLI_COMMANDS.inspectWithObjdump); + expect( + artifactCommand?.args?.safeParse({ + path: "/tmp/sample", + operation: "dwarf", + }).success, + ).toBe(true); + expect( + artifactCommand?.args?.safeParse({ + path: "/tmp/sample", + operation: "architectures", + }).success, + ).toBe(false); +}); + +it("promotes only complete production cleanup-only Evidence after CLI cleanup retry", async () => { + const { directory, path } = await createArtifact(); + const owner = { + stop: vi.fn(async () => ({ status: "verified-cleanup" as const })), + } as unknown as ProviderProcessSupervisor; + const service = new CoreReverseEngineeringService({ + commands: createReverseEngineeringCommandPort({}), + runOwned: vi.fn(async () => { + throw new OwnedCommandFailure( + "process", + "Cleanup after process completion was incomplete.", + providerSnapshot(), + "provider process still running", + undefined, + ["owned-run"], + owner, + ); + }), + }); + const inspect = service.inspectWithObjdump.bind(service); + let producedCleanupFailure: AnalysisOutputError | undefined; + let cleanupIncompleteBeforeClose: boolean | undefined; + let completionBeforeClose: unknown; + vi.spyOn(service, "inspectWithObjdump").mockImplementation( + async (...args) => { + const result = await inspect(...args); + if (!result.ok && result.error instanceof AnalysisOutputError) { + producedCleanupFailure = result.error; + cleanupIncompleteBeforeClose = result.error.cleanupIncomplete; + const partialEvidence = evidenceSchema.safeParse( + result.error.partialObservation, + ); + if (partialEvidence.success) + completionBeforeClose = partialEvidence.data.normalized_result; + } + return result; + }, + ); + + try { + const result = await runObjdump(service, path); + + expect(result).toMatchObject({ + provider: { id: "gnu.objdump" }, + normalized_result: { completion_status: "complete" }, + }); + expect(producedCleanupFailure?.cleanupOnly).toBe(true); + expect(cleanupIncompleteBeforeClose).toBe(true); + expect(completionBeforeClose).toMatchObject({ + completion_status: "complete", + }); + expect(producedCleanupFailure).toMatchObject({ + cleanupIncomplete: false, + cleanup: { resources: ["owned-run"] }, + partialObservation: { + normalized_result: { completion_status: "complete" }, + }, + }); + expect(owner.stop).toHaveBeenCalledOnce(); + } finally { + await service.close().catch(() => undefined); + await rm(directory, { recursive: true, force: true }); + } +}); + +it("preserves an operation timeout and unknown partial Evidence after cleanup retry", async () => { + const { directory, path } = await createArtifact(); + const owner = { + stop: vi.fn(async () => ({ status: "verified-cleanup" as const })), + } as unknown as ProviderProcessSupervisor; + const service = new CoreReverseEngineeringService({ + commands: createReverseEngineeringCommandPort({}), + runOwned: vi.fn(async () => { + throw new OwnedCommandFailure( + "timeout", + "Provider command timed out.", + providerSnapshot({ exitCode: null }), + "provider process still running", + undefined, + ["owned-run"], + owner, + ); + }), + }); + let timeout: AnalysisTimeoutError | undefined; + const inspect = service.inspectWithObjdump.bind(service); + vi.spyOn(service, "inspectWithObjdump").mockImplementation( + async (...args) => { + const result = await inspect(...args); + if (!result.ok && result.error instanceof AnalysisTimeoutError) + timeout = result.error; + return result; + }, + ); + + try { + const result = await runObjdump(service, path); + + expect(result).toMatchObject({ + code: "provider_timeout", + details: { + partial_observation: { + normalized_result: { completion_status: "unknown" }, + }, + }, + }); + expect(timeout).toBeInstanceOf(AnalysisTimeoutError); + expect(timeout?.cleanupIncomplete).toBe(false); + expect(owner.stop).toHaveBeenCalledOnce(); + } finally { + await service.close().catch(() => undefined); + await rm(directory, { recursive: true, force: true }); + } +}); + +it("preserves an independent timeout after cleanup succeeds", async () => { + const evidence = observation("unknown"); + const error = new AnalysisTimeoutError("inspect_with_objdump", 1_000, { + cleanup: { reason: "first stop incomplete", resources: ["owned-run"] }, + partialObservation: evidence, + }); + const service = { + inspectWithObjdump: vi.fn(async () => err(error)), + close: vi.fn(async () => { + error.resolveCleanup(); + }), + }; + + const result = await runObjdump(service); + + expect(result).toMatchObject({ + code: "provider_timeout", + details: { + partial_observation: { + normalized_result: { completion_status: "unknown" }, + }, + }, + }); + expect(error.cleanupIncomplete).toBe(false); + expect(projectAnalysisError(error).details).not.toHaveProperty("cleanup"); +}); + +it("does not project cancellation cleanup as incomplete after a successful retry", async () => { + const error = new AnalysisCancelledError("inspect_with_objdump", { + cleanup: { reason: "first stop incomplete", resources: ["owned-run"] }, + partialObservation: observation("unknown"), + }); + const service = { + inspectWithObjdump: vi.fn(async () => err(error)), + close: vi.fn(async () => { + error.resolveCleanup(); + }), + }; + + await runObjdump(service); + + expect(error.cleanupIncomplete).toBe(false); + expect(projectAnalysisError(error).details).toMatchObject({ + cleanup: "complete", + }); +}); + +it("parses the Rizin inferior TTY as an option and passes it to the manager", async () => { + const commands = new Map< + string, + { + run(input: { + args: Record; + options: Record; + }): Promise; + } + >(); + const cli = { + command: ( + name: string, + definition: { + run(input: { + args: Record; + options: Record; + }): Promise; + }, + ) => { + commands.set(name, definition); + return cli; + }, + } as unknown as CliInstance; + const evidence = createEvidence( + undefined, + { id: "rizin.debugger", name: "Rizin debugger", version: null }, + { + predicateType: "rea.debugger.command-observation", + operation: "rizin_debug_command", + parameters: { + session_id: "session-1", + command: "dr", + inferior_tty: "/dev/pts/7", + }, + result: { output: "registers" }, + rawResult: { output: "registers" }, + }, + ); + const start = vi + .spyOn(RizinDebugSessionManager.prototype, "start") + .mockResolvedValue(ok({ session_id: "session-1", backend: null })); + const execute = vi + .spyOn(RizinDebugSessionManager.prototype, "execute") + .mockResolvedValue( + ok({ + evidence, + result: { + command: "dr", + output: "registers", + output_scope: "command_and_interleaved_session_output", + output_truncated: false, + completion_status: "complete", + backend: null, + }, + }), + ); + vi.spyOn(RizinDebugSessionManager.prototype, "close").mockResolvedValue( + ok({ target_state: "unknown" }), + ); + vi.spyOn(RizinDebugSessionManager.prototype, "closeAll").mockResolvedValue( + undefined, + ); + registerReverseEngineeringCommands(cli, pino({ enabled: false }), {}); + const command = commands.get(CLI_COMMANDS.debugWithRizin); + if (command === undefined) + throw new Error("Rizin CLI command was not registered"); + + await command.run({ + args: { path: "./sample.bin", command: "dr" }, + options: { inferiorTty: "/dev/pts/7" }, + }); + + expect(start).toHaveBeenCalledWith( + { path: "./sample.bin", command: "dr", inferior_tty: "/dev/pts/7" }, + expect.any(AbortSignal), + ); + expect(execute).toHaveBeenCalledWith( + "session-1", + "dr", + expect.any(AbortSignal), + ); + + const parserCli = Cli.create("rea-test", { version: "1.0.0", sync: false }); + registerReverseEngineeringCommands( + parserCli as unknown as CliInstance, + pino({ enabled: false }), + {}, + ); + const commandContract = Cli.toCommands + .get(parserCli) + ?.get(CLI_COMMANDS.debugWithRizin); + expect(commandContract).toBeDefined(); + const argsSchema = + commandContract === undefined + ? undefined + : Reflect.get(commandContract, "args"); + const optionsSchema = + commandContract === undefined + ? undefined + : Reflect.get(commandContract, "options"); + if ( + !(argsSchema instanceof z.ZodObject) || + !(optionsSchema instanceof z.ZodObject) + ) + throw new Error("Rizin CLI args/options schemas were not registered"); + expect(argsSchema.parse({ path: "./sample.bin", command: "dr" })).toEqual({ + path: "./sample.bin", + command: "dr", + }); + expect(optionsSchema.parse({ inferiorTty: "/dev/pts/7" })).toMatchObject({ + inferiorTty: "/dev/pts/7", + }); +}); diff --git a/tests/boundary/evm/evmoleInterfaceProvider.test.ts b/tests/boundary/evm/evmoleInterfaceProvider.test.ts index 8956c5b0f8..51b5cb54ad 100644 --- a/tests/boundary/evm/evmoleInterfaceProvider.test.ts +++ b/tests/boundary/evm/evmoleInterfaceProvider.test.ts @@ -112,12 +112,12 @@ it.runIf(!unsupportedHost).each(["missing", "wrong"])( if (result.ok) throw new Error("Expected launcher failure"); expect(projectAnalysisError(result.error).code).toBe("execution_failure"); expect(result.error).toMatchObject({ + capturedOutput: { stdout: "launcher failure", truncated: false }, diagnostics: { file_size_failure_marker: { verified: false, failure: expect.any(String), }, - captured_output: { stdout: "launcher failure", truncated: false }, }, }); await expect(access(ownedPath)).rejects.toMatchObject({ code: "ENOENT" }); @@ -163,16 +163,12 @@ it.runIf(!unsupportedHost)( const cleanup = z .object({ reason: z.string(), - previous_error: z - .object({ - failure_kind: z.string(), - stdout: z.unknown().optional(), - stderr: z.unknown().optional(), - }) - .transform(({ failure_kind, stdout, stderr }) => ({ - failure_kind, - output_repeated: stdout !== undefined || stderr !== undefined, - })), + previous_error: z.object({ + code: z.string(), + category: z.string(), + message: z.string(), + details: z.record(z.string(), z.unknown()).optional(), + }), captured_output: z.object({ truncated: z.boolean(), stderr: z.literal(""), @@ -182,12 +178,13 @@ it.runIf(!unsupportedHost)( .parse(projectAnalysisError(result.error).details?.["diagnostics"]); expect(cleanup).toEqual({ reason: "injected post-cleanup reporting failure", - previous_error: { - failure_kind: "output-limit", - output_repeated: false, - }, + previous_error: expect.objectContaining({ code: "unreadable_output" }), captured_output: { truncated: true, stderr: "", stdout: 1048576 }, }); + expect(cleanup.previous_error.details?.["captured_output"]).toBeUndefined(); + expect(JSON.stringify(cleanup.previous_error)).not.toContain( + "x".repeat(256), + ); expect(result.error.cleanupResources).toContain(ownedPath); await access(ownedPath); const blocked = await provider.inspect({ path, encoding: "hex" }); @@ -202,6 +199,92 @@ it.runIf(!unsupportedHost)( }, ); +it.runIf(!unsupportedHost).each(["completed", "malformed-reply"] as const)( + "retains worker facts or its primary reply failure while process cleanup is unresolved: %s", + async (scenario) => { + const { path } = await fixture(); + let ownedRoot = ""; + let launches = 0; + let mayClean = false; + const provider = new EvmoleInterfaceProvider({}, async (spawn) => { + launches += 1; + ownedRoot = spawn.cwd ?? ""; + const requestPath = spawn.arguments.at(-1); + if (requestPath === undefined) throw new Error("Request missing"); + const request = requestSchema.parse( + JSON.parse(await readFile(requestPath, "utf8")), + ); + if (scenario === "malformed-reply") + await writeFile(request.reply_path, "malformed worker reply"); + else { + const bytes = Buffer.from("60006000f3", "hex"); + await writeFile( + request.reply_path, + JSON.stringify({ + ok: true, + version: "0.9.3", + bytecode: { + sha256: createHash("sha256").update(bytes).digest("hex"), + bytes: bytes.length, + hex: bytes.toString("hex"), + }, + raw: { functions: [] }, + }), + ); + } + const launched = await spawnOwnedProviderProcess({ + ...spawn, + command: process.execPath, + arguments: ["-e", 'process.stderr.write("worker completed")'], + }); + const cleanup = + launched.cleanup ?? + (() => cleanupOwnedProcessGroup(launched.ownership)); + return { + ...launched, + cleanup: async () => { + const stopped = await cleanup(); + if (!mayClean) throw new Error("injected retained cleanup failure"); + return stopped; + }, + }; + }); + onTestFinished(async () => { + mayClean = true; + await provider.close(); + }); + + const result = await provider.inspect({ path, encoding: "hex" }); + if (result.ok) + throw new Error("Expected process cleanup to remain visible"); + expect(result.error).toMatchObject({ cleanupIncomplete: true }); + const projected = projectAnalysisError(result.error); + expect(projected.details?.["diagnostics"]).toMatchObject({ + reason: "injected retained cleanup failure", + captured_output: { stderr: "worker completed", truncated: false }, + ...(scenario === "malformed-reply" + ? { previous_error: { code: "unreadable_output" } } + : { previous_error: null }), + }); + if (scenario === "completed") + expect(result.error.partialObservation).toMatchObject({ + normalized_result: { + bytecode: { hex: "60006000f3" }, + evidence_kind: "inferred", + }, + }); + else expect(result.error.partialObservation).toBeUndefined(); + await access(ownedRoot); + + const blocked = await provider.inspect({ path, encoding: "hex" }); + expect(blocked.ok).toBe(false); + expect(launches).toBe(1); + mayClean = true; + await provider.close(); + await expect(access(ownedRoot)).rejects.toMatchObject({ code: "ENOENT" }); + }, +); + it.runIf(!unsupportedHost).each([1048576, 1048577])( "keeps complete diagnostics or rejects a valid worker reply with %s diagnostic bytes", async (size) => { diff --git a/tests/boundary/evm/solidityMetadataWorker.test.ts b/tests/boundary/evm/solidityMetadataWorker.test.ts index 0577983c45..df72002b5d 100644 --- a/tests/boundary/evm/solidityMetadataWorker.test.ts +++ b/tests/boundary/evm/solidityMetadataWorker.test.ts @@ -1,6 +1,7 @@ import { execFile } from "node:child_process"; import { readFile, writeFile } from "node:fs/promises"; import { join } from "node:path"; +import { fileURLToPath } from "node:url"; import { promisify } from "node:util"; import { expect, it } from "vitest"; import { solidityMetadataSchema } from "../../../src/domain/evm/solidityMetadata.js"; @@ -38,8 +39,12 @@ it.each([ [ "--max-old-space-size=256", "--disable-wasm-trap-handler", - new URL("../../../dist/evm/EvmoleInterfaceWorker.js", import.meta.url) - .pathname, + fileURLToPath( + new URL( + "../../../dist/evm/EvmoleInterfaceWorker.js", + import.meta.url, + ), + ), request, ], { timeout: 30_000, maxBuffer: 1024 * 1024 }, diff --git a/tests/boundary/filesystem/android/application.test.ts b/tests/boundary/filesystem/android/application.test.ts index 01a5901c80..eb5efb950a 100644 --- a/tests/boundary/filesystem/android/application.test.ts +++ b/tests/boundary/filesystem/android/application.test.ts @@ -205,6 +205,42 @@ describe("APK family without a root manifest", () => { }); }); +it("retains encrypted native library paths as unknown component evidence", async () => { + const root = await createTestTempDirectory("rea-android-encrypted-"); + const path = join(root, "Encrypted.apk"); + const writer = new ZipWriter(new Uint8ArrayWriter()); + await writer.add( + "lib/arm64-v8a/libentry.so", + new TextReader("encrypted native library"), + { password: "fixture-only" }, + ); + await writeFile(path, await writer.close()); + const inventory = parseEvidence( + await runProviderAnalysis(path, "inventory_artifact", {}), + ); + const projection = androidApplicationProjectionResultSchema.parse( + requireSuccessfulProjection( + projectAndroidApplicationEvidence({ inventory_evidence: [inventory] }), + ).normalized_result, + ); + + expect(projection.unavailable_components).toContainEqual( + expect.objectContaining({ + path: "lib/arm64-v8a/libentry.so", + artifact_id: null, + sha256: null, + encrypted: true, + }), + ); + expect(projection.coverage).toMatchObject({ + status: "partial", + inventory_complete: true, + }); + expect(projection.limitations).toContainEqual( + expect.stringContaining("component absence is unknown"), + ); +}); + describe("case-distinct APK resource names", () => { it("inventories and projects an APK whose resource names differ only in case", async () => { const root = await createTestTempDirectory("rea-android-case-"); @@ -228,7 +264,7 @@ describe("case-distinct APK resource names", () => { await runProviderAnalysis(path, "inventory_artifact", {}), ); expect(inventory.normalized_result).toMatchObject({ - coverage_status: "complete", + coverage: { scope: "container-entries", status: "complete" }, }); const result = projectAndroidApplicationEvidence({ inventory_evidence: [inventory], diff --git a/tests/boundary/filesystem/appleApplication.test.ts b/tests/boundary/filesystem/appleApplication.test.ts index 31c395b554..b067342d3e 100644 --- a/tests/boundary/filesystem/appleApplication.test.ts +++ b/tests/boundary/filesystem/appleApplication.test.ts @@ -93,7 +93,7 @@ describe("Apple application projection", () => { await runProviderAnalysis(path, "inventory_artifact", {}), ); expect(inventory.normalized_result).toMatchObject({ - coverage_status: "complete", + coverage: { scope: "container-entries", status: "complete" }, }); const result = projectAppleApplicationEvidence({ inventory_evidence: [inventory], diff --git a/tests/boundary/filesystem/appleXmlDeclarations.test.ts b/tests/boundary/filesystem/appleXmlDeclarations.test.ts index e066655eba..56763d8b00 100644 --- a/tests/boundary/filesystem/appleXmlDeclarations.test.ts +++ b/tests/boundary/filesystem/appleXmlDeclarations.test.ts @@ -1,5 +1,7 @@ import { writeFile } from "node:fs/promises"; +import { execFile } from "node:child_process"; import { join } from "node:path"; +import { promisify } from "node:util"; import { build } from "plist"; import { expect, it } from "vitest"; @@ -14,6 +16,32 @@ const archive = build({ $top: { root: { CF$UID: 1 } }, }); +it("preflights repeated unterminated quoted declarations without suffix rescans", async () => { + const preflightUrl = new URL( + "../../../dist/artifacts/apple/InterfaceBuilderDecodeBudget.js", + import.meta.url, + ).href; + const { stdout } = await promisify(execFile)( + process.execPath, + [ + "--input-type=module", + "-e", + ` + import { preflightPropertyListDecode } from ${JSON.stringify(preflightUrl)}; + const xml = 'x" '.repeat(70000); + const observed = preflightPropertyListDecode(Buffer.from(xml), 16 * 1024 * 1024); + process.stdout.write(JSON.stringify({ observed, expected: xml.length * 4 })); + `, + ], + { timeout: 5000 }, + ); + const result = JSON.parse(stdout) as { + observed: { estimatedBytes: number }; + expected: number; + }; + expect(result.observed.estimatedBytes).toBe(result.expected); +}, 10000); + const fixture = async (declaration: string) => { const bundlePath = await createTestTempDirectory("rea-xml-declaration-"); await writeFile( diff --git a/tests/boundary/filesystem/artifactComparison.test.ts b/tests/boundary/filesystem/artifactComparison.test.ts index f298c28dd8..603201c468 100644 --- a/tests/boundary/filesystem/artifactComparison.test.ts +++ b/tests/boundary/filesystem/artifactComparison.test.ts @@ -2,18 +2,20 @@ import { chmod, mkdir, writeFile } from "node:fs/promises"; import { join } from "node:path"; import { describe, expect, it } from "vitest"; +import { TextReader, Uint8ArrayWriter, ZipWriter } from "@zip.js/zip.js"; import { fragmentInventoryEvidence } from "../../fixtures/artifactEvidence.js"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; import { inventoryArtifact } from "../../fixtures/artifactInventory.js"; import { artifactInventoryResultSchema } from "../../../src/domain/artifactGraph.js"; +import { ARTIFACT_COMPARISON_EXAMPLE } from "../../../src/contracts/artifactComparisonExample.js"; import { artifactComparisonResultSchema, compareArtifacts, } from "../../../src/domain/artifactComparison.js"; import { parseArtifactInventoryEvidence } from "../../../src/domain/artifactInventoryEvidence.js"; -import { createEvidence } from "../../../src/domain/evidence.js"; +import { createEvidence, type Evidence } from "../../../src/domain/evidence.js"; import { jsonValueSchema } from "../../../src/domain/jsonValue.js"; const PROVIDER = { @@ -41,6 +43,66 @@ const observe = async (path: string) => { ); }; +const inspectPages = ( + pages: readonly Evidence[], + substepsCompleted = pages.length, +) => { + const first = pages[0]; + if (first === undefined) throw new Error("Missing inspection page"); + const inventory = artifactInventoryResultSchema.parse( + first.normalized_result, + ); + return createEvidence( + { + path: "artifact-inspection-example", + sha256: inventory.manifest.root_sha256, + format: inventory.manifest.root_format, + }, + PROVIDER, + { + operation: "inspect_artifact", + parameters: {}, + result: jsonValueSchema.parse({ + inspection_id: `ai_${"f".repeat(64)}`, + subject: { + manifest_id: inventory.manifest.manifest_id, + root_artifact_id: inventory.manifest.root_artifact_id, + root_sha256: inventory.manifest.root_sha256, + root_format: inventory.manifest.root_format, + }, + substeps: pages.map((evidence, index) => ({ + substep_id: `ais_${(index + 1).toString(16).padStart(64, "0")}`, + operation: "inventory_artifact", + status: "completed", + evidence_id: evidence.evidence_id, + evidence, + })), + observations: [], + hypotheses: [], + contradictions: [], + unexplored_branches: [], + next_probes: [], + coverage: { + status: "complete-within-substeps", + substeps_completed: substepsCompleted, + observations_retained: 0, + observations_omitted: 0, + hypotheses_retained: 0, + hypotheses_omitted: 0, + unexplored_branches_retained: 0, + unexplored_branches_omitted: 0, + next_probes_retained: 0, + next_probes_omitted: 0, + }, + evidence_links: pages.map(({ evidence_id }) => evidence_id), + limitations: [], + }), + confidence: "observed", + authority: "shipped-artifact", + }, + ); +}; + const changedArtifactComparison = async () => { const parent = await createTestTempDirectory("rea-artifact-compare-"); const leftPath = join(parent, "left.app"); @@ -63,26 +125,16 @@ const changedArtifactComparison = async () => { }; describe("leaf artifact comparison", () => { - it("accepts persisted complete leaf inventory Evidence without coverage_status", async () => { + it("rejects inventory Evidence without required producer coverage", async () => { const root = await createTestTempDirectory("rea-artifact-plist-compare-"); const leftPath = join(root, "left.plist"); - const rightPath = join(root, "right.plist"); const xml = 'labelhello'; - await Promise.all([writeFile(leftPath, xml), writeFile(rightPath, xml)]); - const [left, right] = await Promise.all([ - observe(leftPath), - observe(rightPath), - ]); - // Persisted inventory Evidence predates coverage_status. The known leaf - // notice remains complete when normalized at the inventory boundary. + await writeFile(leftPath, xml); + const left = await observe(leftPath); const parsed = artifactInventoryResultSchema.parse(left.normalized_result); - const { - coverage: _coverage, - coverage_status: _coverageStatus, - ...legacyInventory - } = parsed; - const legacy = createEvidence( + const { coverage: _coverage, ...withoutCoverage } = parsed; + const malformed = createEvidence( { path: left.subject!.local_path, sha256: left.subject!.digest.sha256, @@ -92,16 +144,12 @@ describe("leaf artifact comparison", () => { { operation: "inventory_artifact", parameters: {}, - result: jsonValueSchema.parse(legacyInventory), + result: jsonValueSchema.parse(withoutCoverage), confidence: "observed", authority: "shipped-artifact", }, ); - expect(compareArtifacts(legacy, right)).toMatchObject({ - status: "unchanged", - summary: { unchanged: 1, unknown: 0 }, - changes: [], - }); + expect(() => parseArtifactInventoryEvidence(malformed)).toThrow(); }); it("keeps content and executable changes visible for complete plist roots", async () => { @@ -149,8 +197,95 @@ describe("leaf artifact comparison", () => { }); }); +describe("artifact inspection inventory substeps", () => { + it("assembles every paged inventory substep and retains its outer Evidence once", async () => { + const root = await createTestTempDirectory("rea-inspect-pages-"); + const path = join(root, "fixture.app"); + await mkdir(path); + await Promise.all([ + writeFile(join(path, "first.js"), "first();"), + writeFile(join(path, "second.js"), "second();"), + ]); + const source = await observe(path); + const pages = fragmentInventoryEvidence(source, 2); + const inspection = inspectPages(pages); + const parsed = parseArtifactInventoryEvidence(inspection); + + expect(parsed.evidence).toEqual([inspection]); + expect(parsed.inventory.complete).toBe(true); + expect(parsed.inventory.nodes).toHaveLength( + artifactInventoryResultSchema.parse(source.normalized_result).nodes + .length, + ); + }); + + it("rejects inspection substeps with different inventory manifests", async () => { + const root = await createTestTempDirectory("rea-inspect-mixed-pages-"); + const firstPath = join(root, "first.plist"); + const secondPath = join(root, "second.plist"); + await Promise.all([ + writeFile( + firstPath, + 'a1', + ), + writeFile( + secondPath, + 'b2', + ), + ]); + + const inspection = inspectPages([ + await observe(firstPath), + await observe(secondPath), + ]); + expect(() => parseArtifactInventoryEvidence(inspection)).toThrow( + "Artifact inventory pages do not share one manifest", + ); + }); + + it("rejects inspection coverage counts that disagree with retained substeps", async () => { + const root = await createTestTempDirectory("rea-inspect-count-mismatch-"); + const path = join(root, "fixture.app"); + await mkdir(path); + await Promise.all([ + writeFile(join(path, "first.js"), "first();"), + writeFile(join(path, "second.js"), "second();"), + ]); + const pages = fragmentInventoryEvidence(await observe(path), 2); + const inspection = inspectPages(pages, 1); + + expect(() => parseArtifactInventoryEvidence(inspection)).toThrow( + "Artifact inspection substep count does not match its coverage declaration", + ); + }); + + it("authenticates later inventory substeps instead of silently ignoring them", async () => { + const root = await createTestTempDirectory("rea-inspect-invalid-page-"); + const path = join(root, "fixture.app"); + await mkdir(path); + await writeFile(join(path, "main.js"), "main();"); + const source = await observe(path); + const [first, later] = fragmentInventoryEvidence(source, 2); + if (first === undefined || later === undefined) + throw new Error("Missing inventory page"); + const graph = artifactInventoryResultSchema.parse(later.normalized_result); + const tamperedLater: Evidence = { + ...later, + normalized_result: jsonValueSchema.parse({ + ...graph, + limitations: ["Changed after this Evidence identifier was computed."], + }), + }; + const inspection = inspectPages([first, tamperedLater]); + + expect(() => parseArtifactInventoryEvidence(inspection)).toThrow( + "Evidence semantic identifier does not match its record", + ); + }); +}); + describe("artifact inventory coverage", () => { - it("keeps explicit and legacy unknown coverage fail-closed", async () => { + it("uses current unknown coverage without deriving it from limitation prose", async () => { const root = await createTestTempDirectory("rea-artifact-coverage-"); const path = join(root, "complete.plist"); await writeFile( @@ -161,7 +296,7 @@ describe("artifact inventory coverage", () => { const graph = artifactInventoryResultSchema.parse( complete.normalized_result, ); - const contradictory = createEvidence( + const unknownCoverage = createEvidence( { path: complete.subject!.local_path, sha256: complete.subject!.digest.sha256, @@ -171,63 +306,19 @@ describe("artifact inventory coverage", () => { { operation: "inventory_artifact", parameters: {}, - result: jsonValueSchema.parse({ ...graph, coverage_status: "partial" }), + result: jsonValueSchema.parse({ + ...graph, + coverage: { scope: "root", status: "unknown" }, + limitations: ["No extra child entries were observed."], + }), confidence: "derived", authority: "shipped-artifact", }, ); - expect(() => parseArtifactInventoryEvidence(contradictory)).toThrow( - "Artifact inventory has conflicting coverage declarations", - ); - - const { coverage: _coverage, ...legacyStatusGraph } = graph; - const partial = createEvidence( - { - path: complete.subject!.local_path, - sha256: complete.subject!.digest.sha256, - format: complete.subject!.format, - }, - PROVIDER, - { - operation: "inventory_artifact", - parameters: {}, - result: jsonValueSchema.parse({ - ...legacyStatusGraph, - coverage_status: "partial", - }), - confidence: "observed", - authority: "shipped-artifact", - }, - ); - expect(parseArtifactInventoryEvidence(partial).inventory.complete).toBe( - true, - ); - expect(compareArtifacts(partial, partial)).toMatchObject({ - status: "unknown", - summary: { unchanged: 0, unknown: 1 }, - }); - - const { coverage_status: _coverageStatus, ...legacyGraph } = - legacyStatusGraph; - const unknownLegacy = createEvidence( - { - path: complete.subject!.local_path, - sha256: complete.subject!.digest.sha256, - format: complete.subject!.format, - }, - PROVIDER, - { - operation: "inventory_artifact", - parameters: {}, - result: jsonValueSchema.parse({ - ...legacyGraph, - limitations: ["Producer omitted unexamined child entries."], - }), - confidence: "observed", - authority: "shipped-artifact", - }, - ); - expect(compareArtifacts(unknownLegacy, unknownLegacy)).toMatchObject({ + expect( + parseArtifactInventoryEvidence(unknownCoverage).inventory.complete, + ).toBe(true); + expect(compareArtifacts(unknownCoverage, unknownCoverage)).toMatchObject({ status: "unknown", summary: { unchanged: 0, unknown: 1 }, }); @@ -247,7 +338,6 @@ describe("artifact inventory coverage", () => { const fragmentGraph = artifactInventoryResultSchema.parse( fragment.normalized_result, ); - const { coverage: _coverage, ...legacyFragmentGraph } = fragmentGraph; const mixed = createEvidence( { path: fragment.subject!.local_path, @@ -259,8 +349,8 @@ describe("artifact inventory coverage", () => { operation: "inventory_artifact", parameters: fragment.parameters, result: jsonValueSchema.parse({ - ...legacyFragmentGraph, - coverage_status: "partial", + ...fragmentGraph, + coverage: { ...fragmentGraph.coverage, status: "unknown" }, }), confidence: "derived", authority: "shipped-artifact", @@ -273,6 +363,42 @@ describe("artifact inventory coverage", () => { }); describe("artifact comparison", () => { + it("keeps the canonical public example valid and fully comparable", () => { + const left = parseArtifactInventoryEvidence( + ARTIFACT_COMPARISON_EXAMPLE.left, + ); + const right = parseArtifactInventoryEvidence( + ARTIFACT_COMPARISON_EXAMPLE.right, + ); + + expect(left.inventory).toMatchObject({ + complete: true, + coverage: { scope: "root", status: "complete" }, + }); + expect(right.inventory).toMatchObject({ + complete: true, + coverage: { scope: "root", status: "complete" }, + }); + expect( + artifactComparisonResultSchema.parse( + compareArtifacts( + ARTIFACT_COMPARISON_EXAMPLE.left, + ARTIFACT_COMPARISON_EXAMPLE.right, + ), + ), + ).toMatchObject({ + status: "changed", + summary: { changed: 1, unknown: 0 }, + changes: [ + expect.objectContaining({ + logical_path: ".", + classification: "changed", + dimensions: expect.arrayContaining(["content"]), + }), + ], + }); + }); + it("classifies deterministic path changes and cites both inventories", async () => { const firstObservation = await changedArtifactComparison(); const secondObservation = await changedArtifactComparison(); @@ -442,7 +568,182 @@ describe("artifact comparison completeness", () => { changes: [], }); }, 15_000); +}); +describe("artifact source-map relationship coverage", () => { + it("preserves unknown source-map relationships for duplicate ZIP paths", async () => { + const root = await createTestTempDirectory("rea-duplicate-source-map-"); + const path = join(root, "duplicate-source-map.zip"); + const writer = new ZipWriter(new Uint8ArrayWriter()); + await writer.add("main.js", new TextReader("first();"), { level: 0 }); + await writer.add("your.js", new TextReader("second();"), { level: 0 }); + await writer.add("main.js.map", new TextReader("{}"), { level: 0 }); + const bytes = Buffer.from(await writer.close()); + for (const [signature, headerSize] of [ + [Buffer.from([0x50, 0x4b, 0x03, 0x04]), 30], + [Buffer.from([0x50, 0x4b, 0x01, 0x02]), 46], + ] as const) { + const first = bytes.indexOf(signature); + const second = bytes.indexOf(signature, first + 4); + if (first < 0 || second <= first) + throw new Error("Expected both source ZIP member headers"); + expect( + bytes.subarray(second + headerSize, second + headerSize + 7).toString(), + ).toBe("your.js"); + bytes.write("main.js", second + headerSize); + } + await writeFile(path, bytes); + + const evidence = await observe(path); + const parsed = parseArtifactInventoryEvidence(evidence); + const { inventory } = parsed; + const map = inventory.occurrences.find( + ({ logical_path }) => logical_path === "main.js.map", + ); + if (map === undefined) throw new Error("Expected source-map occurrence"); + const mapEdge = inventory.edges.find( + ({ occurrence_id }) => occurrence_id === map.occurrence_id, + ); + + expect(inventory.coverage.status).toBe("unknown"); + expect(map.limitations).toContain( + 'Source-map target is unknown because multiple occurrences share "main.js".', + ); + expect(mapEdge).toMatchObject({ relation: "contains" }); + expect(mapEdge?.relation).not.toBe("maps-source"); + expect(inventory.integrityContradictions).toEqual([]); + expect( + inventory.limitations.some((limitation) => + limitation.toLowerCase().includes("integrity"), + ), + ).toBe(false); + }); +}); + +describe("duplicate archive occurrence ownership", () => { + it("keeps directory ancestry unknown when ZIP directory paths repeat", async () => { + const root = await createTestTempDirectory( + "rea-artifact-duplicate-directory-", + ); + const path = join(root, "duplicate-directories.zip"); + const writer = new ZipWriter(new Uint8ArrayWriter()); + await writer.add("x/", new TextReader("")); + await writer.add("y/", new TextReader("")); + await writer.add("x/file.txt", new TextReader("member")); + const bytes = Buffer.from(await writer.close()); + for (const [signature, headerSize] of [ + [Buffer.from([0x50, 0x4b, 0x03, 0x04]), 30], + [Buffer.from([0x50, 0x4b, 0x01, 0x02]), 46], + ] as const) { + const first = bytes.indexOf(signature); + const second = bytes.indexOf(signature, first + 4); + if (first < 0 || second <= first) + throw new Error("Expected both ZIP directory headers"); + expect( + bytes.subarray(second + headerSize, second + headerSize + 2).toString(), + ).toBe("y/"); + bytes.write("x/", second + headerSize); + } + await writeFile(path, bytes); + + const evidence = await observe(path); + const inventory = parseArtifactInventoryEvidence(evidence).inventory; + const directory = inventory.occurrences.filter( + ({ logical_path }) => logical_path === "x", + ); + const child = inventory.occurrences.find( + ({ logical_path }) => logical_path === "x/file.txt", + ); + expect(directory).toHaveLength(2); + expect(directory.every(({ artifact_id }) => artifact_id === null)).toBe( + true, + ); + expect(child?.parent_occurrence_id).toBeNull(); + expect(child?.limitations).toContain( + 'The parent occurrence for "x/file.txt" is unknown because multiple observed directory or container occurrences share "x".', + ); + expect(inventory.coverage.status).toBe("unknown"); + }); + + it("retains duplicate path occurrences across pages and leaves pairing unknown", async () => { + const root = await createTestTempDirectory("rea-artifact-duplicate-pages-"); + const archives = [join(root, "left.zip"), join(root, "right.zip")] as const; + for (const [index, path] of archives.entries()) { + const writer = new ZipWriter(new Uint8ArrayWriter()); + await writer.add("x.txt", new TextReader("first")); + await writer.add( + "y.txt", + new TextReader(index === 0 ? "second-left" : "second-right"), + ); + await writer.add("neighbor.txt", new TextReader("neighbor")); + const bytes = Buffer.from(await writer.close()); + // Produce a duplicate path the same way as real malformed/ambiguous ZIP + // inputs: retain each member record and rewrite only the equal-length name. + for (const [signature, headerSize] of [ + [Buffer.from([0x50, 0x4b, 0x03, 0x04]), 30], + [Buffer.from([0x50, 0x4b, 0x01, 0x02]), 46], + ] as const) { + const first = bytes.indexOf(signature); + const second = bytes.indexOf(signature, first + 4); + if (first < 0 || second <= first) + throw new Error("Expected both ZIP member headers"); + expect( + bytes + .subarray(second + headerSize, second + headerSize + 5) + .toString(), + ).toBe("y.txt"); + bytes.write("x.txt", second + headerSize); + } + await writeFile(path, bytes); + } + + const [left, right] = await Promise.all([ + observe(archives[0]), + observe(archives[1]), + ]); + const parsed = parseArtifactInventoryEvidence(left); + const repeated = parsed.inventory.occurrences.filter( + ({ logical_path }) => logical_path === "x.txt", + ); + expect(repeated).toHaveLength(2); + expect( + new Set(repeated.map(({ occurrence_id }) => occurrence_id)).size, + ).toBe(2); + + const leftPages = fragmentInventoryEvidence(left, 2); + const parsedPages = parseArtifactInventoryEvidence(leftPages); + expect( + parsedPages.inventory.occurrences.filter( + ({ logical_path }) => logical_path === "x.txt", + ), + ).toHaveLength(2); + + const comparison = compareArtifacts( + leftPages, + fragmentInventoryEvidence(right, 2), + ); + expect(comparison).toMatchObject({ + status: "unknown", + summary: { unknown: expect.any(Number) }, + changes: expect.arrayContaining([ + expect.objectContaining({ + logical_path: "x.txt", + classification: "unknown", + dimensions: ["availability"], + left_occurrence_id: null, + right_occurrence_id: null, + left_artifact_id: null, + right_artifact_id: null, + }), + ]), + }); + expect(comparison.limitations).toContain( + "Duplicate logical paths cannot be paired across inventories; comparisons for those paths are unknown.", + ); + }); +}); + +describe("artifact comparison fragment integrity", () => { it("assembles real inventory fragments and retains every citation", async () => { const parent = await createTestTempDirectory("rea-artifact-fragments-"); const leftPath = join(parent, "left.app"); @@ -553,3 +854,52 @@ describe("artifact comparison completeness", () => { ); }); }); + +describe("ZIP non-entry coverage", () => { + it("retains pre-directory APK block changes under current ZIP coverage", async () => { + const root = await createTestTempDirectory("rea-apk-block-compare-"); + const writer = new ZipWriter(new Uint8ArrayWriter()); + await writer.add("classes.dex", new TextReader("unchanged entry")); + const zip = Buffer.from(await writer.close()); + const eocd = zip.length - 22; + expect(zip.readUInt32LE(eocd)).toBe(0x06054b50); + const directory = zip.readUInt32LE(eocd + 16); + // APK Signing Block framing with one opaque padding pair, not a verified signature. + // https://source.android.com/docs/security/features/apksigning/v2#apk-signing-block + const block = Buffer.alloc(48); + block.writeBigUInt64LE(40n, 0); + block.writeBigUInt64LE(8n, 8); + block.writeUInt32LE(0x42726577, 16); + block.writeBigUInt64LE(40n, 24); + block.write("APK Sig Block 42", 32, "ascii"); + const paths = [join(root, "before.apk"), join(root, "after.apk")]; + for (const [index, path] of paths.entries()) { + block.writeUInt32LE(index, 20); + const apk = Buffer.concat([ + zip.subarray(0, directory), + block, + zip.subarray(directory), + ]); + apk.writeUInt32LE(directory + block.length, eocd + block.length + 16); + await writeFile(path, apk); + } + const left = await observe(paths[0]!); + const right = await observe(paths[1]!); + const inventory = artifactInventoryResultSchema.parse( + left.normalized_result, + ); + expect(inventory.limitations).toContain( + "ZIP inventory lists central-directory entries only. Non-entry bytes, including an APK Signing Block before the central directory or a ZIP comment after it, are not occurrences.", + ); + expect( + inventory.occurrences.map(({ logical_path }) => logical_path).sort(), + ).toEqual([".", "classes.dex"]); + expect(compareArtifacts(left, right)).toMatchObject({ + status: "changed", + summary: { changed: 1, unchanged: 1, unknown: 0 }, + changes: [ + expect.objectContaining({ logical_path: ".", dimensions: ["content"] }), + ], + }); + }); +}); diff --git a/tests/boundary/filesystem/artifactLiteralBackslash.test.ts b/tests/boundary/filesystem/artifactLiteralBackslash.test.ts new file mode 100644 index 0000000000..02105ad604 --- /dev/null +++ b/tests/boundary/filesystem/artifactLiteralBackslash.test.ts @@ -0,0 +1,222 @@ +import { createHash } from "node:crypto"; +import { access, mkdir, readFile, readdir, writeFile } from "node:fs/promises"; +import { dirname, join } from "node:path"; +import { Readable } from "node:stream"; + +import { createPackage } from "@electron/asar"; +import { expect, it } from "vitest"; + +import { extractArtifact } from "../../../src/artifacts/extraction/ArtifactExtraction.js"; +import { SafeOutputTree } from "../../../src/artifacts/SafeOutputTree.js"; +import { + inventoryArtifact, + withArtifactResourceScope, +} from "../../fixtures/artifactInventory.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { analyzeJavaScriptApplication } from "../../support/javascriptApplicationScope.js"; +import { javascriptApplicationAnalysisResultSchema } from "../../../src/domain/javascript/javascriptApplicationAnalysis.js"; + +const NAMES = [ + "literal\\name.txt", + "\\leading.txt", + "\\\\server\\share.txt", + "ordinary.txt", + "literal/name.txt", + "nested/literal\\name.txt", +]; +const CONTENT = Buffer.from("exact member bytes\n"); +const DIGEST = createHash("sha256").update(CONTENT).digest("hex"); + +it.skipIf(process.platform === "win32").each(["directory", "asar"] as const)( + "inventories and extracts literal POSIX backslashes from %s without changing names", + async (format) => { + const root = await createTestTempDirectory("rea-literal-backslash-"); + const source = join(root, "source"); + await mkdir(source); + for (const name of NAMES) { + await mkdir(dirname(join(source, name)), { recursive: true }); + await writeFile(join(source, name), CONTENT); + } + const input = format === "directory" ? source : join(root, "input.asar"); + if (format === "asar") await createPackage(source, input); + + const inventory = await inventoryArtifact(input); + for (const name of NAMES) { + const occurrence = inventory.occurrences.find( + ({ logical_path }) => logical_path === name, + ); + expect(occurrence).toMatchObject({ + entry_kind: "file", + hash_status: "verified", + }); + expect( + inventory.nodes.find( + ({ artifact_id }) => artifact_id === occurrence?.artifact_id, + ), + ).toMatchObject({ sha256: DIGEST, size: CONTENT.length }); + } + + const output = join(root, "output"); + const extracted = await withArtifactResourceScope((resourceScope) => + extractArtifact({ + inputPath: input, + // The owner checks lstat first and selects the directory reader; + // BinaryTarget formats have no separate directory tag. + inputFormat: "asar", + outputRoot: output, + environment: {}, + integrityPolicy: "fail", + resourceScope, + }), + ); + expect( + extracted.artifacts.map(({ relative_path }) => relative_path).sort(), + ).toEqual([...NAMES].sort()); + expect((await readdir(output)).sort()).toEqual( + [...new Set(NAMES.map((name) => name.split("/")[0]))].sort(), + ); + for (const name of NAMES) + expect(await readFile(join(output, name))).toEqual(CONTENT); + }, +); + +it.each(NAMES.filter((name) => name.includes("\\")))( + "rejects Windows separator interpretation before materializing %j", + async (name) => { + const root = await createTestTempDirectory("rea-backslash-output-"); + const output = join(root, "output"); + const tree = await SafeOutputTree.create(output, "win32"); + const stream = Readable.from(CONTENT); + try { + await expect( + tree.write(name, stream, { sha256: DIGEST, bytes: CONTENT.length }), + ).rejects.toMatchObject({ + reason: "path", + message: expect.stringContaining( + "Windows destination cannot materialize artifact path", + ), + }); + expect(stream.destroyed).toBe(true); + expect(await readdir(output)).toEqual([]); + } finally { + expect(await tree.rollback()).toMatchObject({ status: "complete" }); + } + await expect(access(output)).rejects.toMatchObject({ code: "ENOENT" }); + }, +); + +it.each([ + "../outside", + "safe/../outside", + "/outside", + "//server/share", + "C:\\outside", + "C:/outside", + "safe/\0bad", +])("keeps output containment rejection for %j", async (name) => { + const root = await createTestTempDirectory("rea-backslash-containment-"); + const output = join(root, "output"); + const tree = await SafeOutputTree.create(output); + try { + await expect( + tree.write(name, Readable.from(CONTENT), { + sha256: DIGEST, + bytes: CONTENT.length, + }), + ).rejects.toMatchObject({ reason: "path" }); + expect(await readdir(output)).toEqual([]); + } finally { + expect(await tree.rollback()).toMatchObject({ status: "complete" }); + } +}); + +it.skipIf(process.platform === "win32").each(["directory", "asar"] as const)( + "retains literal POSIX JavaScript names in application and semantic graphs from %s", + async (format) => { + const root = await createTestTempDirectory("rea-backslash-js-graph-"); + const source = join(root, "source"); + await mkdir(source); + const names = [ + "literal\\name.js", + "\\leading.js", + "\\\\server\\share.js", + "nested/literal\\name.js", + ]; + for (const name of names) { + await mkdir(dirname(join(source, name)), { recursive: true }); + await writeFile(join(source, name), "export const marker = 42;\n"); + } + const input = format === "directory" ? source : join(root, "input.asar"); + if (format === "asar") await createPackage(source, input); + const result = await analyzeJavaScriptApplication({ + input_path: input, + format, + }); + if (!result.ok) throw result.error; + const parsed = javascriptApplicationAnalysisResultSchema.parse( + result.value.normalized_result, + ); + const observed = parsed.graph.nodes.flatMap(({ observations }) => + observations.map(({ properties }) => properties.path), + ); + expect(observed).toEqual(expect.arrayContaining(names)); + const semantic = parsed.semantic_graph?.nodes.map( + ({ identity }) => identity.module_path, + ); + expect(semantic).toEqual(expect.arrayContaining(names)); + }, +); + +it.skipIf(process.platform === "win32").each(["directory", "asar"] as const)( + "resolves ordinary relative imports inside a literal POSIX directory from %s", + async (format) => { + const root = await createTestTempDirectory("rea-backslash-import-base-"); + const source = join(root, "source"); + const literal = "literal\\dir"; + await mkdir(join(source, literal), { recursive: true }); + await mkdir(join(source, "literal", "dir"), { recursive: true }); + const body = "export const marker = 'literal';\n"; + await writeFile( + join(source, literal, "a.js"), + "import { marker } from './b.js'; export { marker };\n", + ); + await writeFile(join(source, literal, "b.js"), body); + await writeFile( + join(source, "literal", "dir", "b.js"), + "export const marker = 'decoy';\n", + ); + const input = format === "directory" ? source : join(root, "input.asar"); + if (format === "asar") await createPackage(source, input); + const result = await analyzeJavaScriptApplication({ + input_path: input, + format, + }); + if (!result.ok) throw result.error; + const parsed = javascriptApplicationAnalysisResultSchema.parse( + result.value.normalized_result, + ); + const imports = parsed.graph.edges.filter( + ({ properties }) => properties.specifier === "./b.js", + ); + expect(imports.length).toBeGreaterThan(0); + expect(imports.map(({ properties }) => properties.resolved_path)).toEqual( + imports.map(() => `${literal}/b.js`), + ); + expect(parsed.graph.nodes).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + identity: { + strategy: "content-digest", + stability: "global-exact", + sha256: createHash("sha256").update(body).digest("hex"), + }, + observations: expect.arrayContaining([ + expect.objectContaining({ + properties: expect.objectContaining({ path: `${literal}/b.js` }), + }), + ]), + }), + ]), + ); + }, +); diff --git a/tests/boundary/filesystem/artifactZipIntegrity.test.ts b/tests/boundary/filesystem/artifactZipIntegrity.test.ts index cf7dcd5957..e02c22674e 100644 --- a/tests/boundary/filesystem/artifactZipIntegrity.test.ts +++ b/tests/boundary/filesystem/artifactZipIntegrity.test.ts @@ -39,6 +39,7 @@ it("completes an empty ZIP extraction with an empty identity", async () => { }); const legacyControlNames = [ + ["unflagged UTF-8-looking legacy bytes", Buffer.from("café"), false, "caf├⌐"], ["legacy ASCII control", Buffer.from("Icon\r"), false, "Icon\r"], [ "legacy CP437 control", @@ -311,10 +312,13 @@ it("retains duplicate ZIP members without replacing their source identities", as const target = await parseBinaryTarget(path); if (!target.ok) throw target.error; await withArtifactClient(target.value, async (client) => { - expect(await client.execute("inventory_artifact", {})).toMatchObject({ - ok: false, - error: { _tag: "ArtifactOperationError", reason: "path" }, - }); + const result = await client.execute("inventory_artifact", {}); + if (!result.ok) throw result.error; + expect( + artifactInventoryResultSchema + .parse(result.value.result) + .occurrences.filter(({ logical_path }) => logical_path === "x.txt"), + ).toHaveLength(2); }); }); diff --git a/tests/boundary/filesystem/asarHeaderAdmission.test.ts b/tests/boundary/filesystem/asarHeaderAdmission.test.ts new file mode 100644 index 0000000000..e16e2ea2de --- /dev/null +++ b/tests/boundary/filesystem/asarHeaderAdmission.test.ts @@ -0,0 +1,115 @@ +import { mkdir, open, readFile, truncate, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { createPackage } from "@electron/asar"; +import { expect, it, vi } from "vitest"; + +import { + admitAsarHeader, + MAX_ASAR_HEADER_BYTES, +} from "../../../src/artifacts/AsarHeader.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +const createFixture = async () => { + const root = await createTestTempDirectory("rea-asar-header-admission-"); + const source = join(root, "source"); + const archive = join(root, "fixture.asar"); + await mkdir(source); + await writeFile(join(source, "main.js"), "export const answer = 42;\n"); + await createPackage(source, archive); + return { archive, original: await readFile(archive) }; +}; + +const admit = async (path: string) => { + const handle = await open(path, "r"); + const read = vi.spyOn(handle, "read"); + try { + return await admitAsarHeader(handle); + } finally { + // Admission must inspect only the fixed prefix, even for forged extents. + const calls = [...read.mock.calls]; + read.mockRestore(); + await handle.close(); + expect(handle.fd).toBe(-1); + expect(calls).toEqual([[expect.any(Buffer), 0, 16, 0]]); + } +}; + +it("admits a header produced by the upstream ASAR serializer", async () => { + const { archive, original } = await createFixture(); + await expect(admit(archive)).resolves.toEqual({ + archiveBytes: original.length, + headerBytes: original.readUInt32LE(4), + jsonBytes: original.readUInt32LE(12), + }); + expect(await readFile(archive)).toEqual(original); +}); + +it("rejects every truncated prefix of a real serialized header", async () => { + const { archive, original } = await createFixture(); + const headerEnd = 8 + original.readUInt32LE(4); + for (let length = 0; length < headerEnd; length++) { + await writeFile(archive, original.subarray(0, length)); + await expect( + admit(archive), + `truncated at byte ${length}`, + ).rejects.toMatchObject({ + reason: "format", + }); + } + // Member data is outside header admission; an intact header still passes. + await writeFile(archive, original.subarray(0, headerEnd)); + await expect(admit(archive)).resolves.toMatchObject({ + archiveBytes: headerEnd, + }); +}); + +it("rejects contradictory and unsigned-boundary extents before decoding", async () => { + const { archive, original } = await createFixture(); + const mutations = [ + { offset: 0, values: [0, 1, 3, 5, 0xffffffff] }, + { offset: 4, values: [0, 4, 7, original.length, 0x80000000, 0xffffffff] }, + { offset: 8, values: [0, 1, original.readUInt32LE(8) + 4, 0xffffffff] }, + { offset: 12, values: [original.readUInt32LE(8), 0x80000000, 0xffffffff] }, + ]; + for (const { offset, values } of mutations) { + for (const value of values) { + const bytes = Buffer.from(original); + bytes.writeUInt32LE(value, offset); + await writeFile(archive, bytes); + await expect( + admit(archive), + `uint32 at ${offset} = ${value}`, + ).rejects.toMatchObject({ + reason: "format", + }); + } + } +}); + +it("separates the header decode budget from container validity at its exact boundary", async () => { + const root = await createTestTempDirectory("rea-asar-header-budget-"); + const archive = join(root, "budget.asar"); + expect(MAX_ASAR_HEADER_BYTES).toBe(16 * 1024 * 1024); + for (const headerBytes of [ + MAX_ASAR_HEADER_BYTES, + MAX_ASAR_HEADER_BYTES + 4, + ]) { + const prefix = Buffer.alloc(16); + prefix.writeUInt32LE(4, 0); + prefix.writeUInt32LE(headerBytes, 4); + prefix.writeUInt32LE(headerBytes - 4, 8); + prefix.writeUInt32LE(2, 12); + await writeFile(archive, prefix); + // Supply the declared extent without allocating a header-sized JS buffer. + // This tests extent admission only; the padding is not valid archive JSON. + await truncate(archive, headerBytes + 8); + if (headerBytes === MAX_ASAR_HEADER_BYTES) + await expect(admit(archive)).resolves.toEqual({ + archiveBytes: headerBytes + 8, + headerBytes, + jsonBytes: 2, + }); + else + await expect(admit(archive)).rejects.toMatchObject({ reason: "limit" }); + } +}); diff --git a/tests/boundary/filesystem/binaryTarget.test.ts b/tests/boundary/filesystem/binaryTarget.test.ts index eb5977a541..93ef57afa4 100644 --- a/tests/boundary/filesystem/binaryTarget.test.ts +++ b/tests/boundary/filesystem/binaryTarget.test.ts @@ -26,6 +26,7 @@ import { thinMach, } from "../../../src/domain/binaryTarget.fixture.js"; import { projectAnalysisError } from "../../../src/domain/analysisErrorProjection.js"; +import { hasXmlPlistSignature } from "../../../src/domain/apple/appleArchiveFormat.js"; import { parseExecutableHeader } from "../../../src/domain/binaryTarget.js"; import { createEvidence, parseEvidence } from "../../../src/domain/evidence.js"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; @@ -258,6 +259,46 @@ describe("binary target I/O: package classification", () => { }); }); + it("classifies XML plists by content while preserving named formats", async () => { + const directory = await createTestTempDirectory("rea-xml-plist-target-"); + const xml = Buffer.from( + '$archiverNSKeyedArchiver', + ); + const archivePath = join(directory, "Model.archive"); + const javascriptPath = join(directory, "Model.js"); + await writeFile(archivePath, xml); + await writeFile(javascriptPath, xml); + + const archive = await parseBinaryTarget(archivePath); + const javascript = await parseBinaryTarget(javascriptPath); + expect(archive.ok && archive.value).toMatchObject({ + kind: "artifact", + format: "plist", + }); + expect(javascript.ok && javascript.value).toMatchObject({ + kind: "artifact", + format: "javascript", + }); + }); + + it("continues a bounded XML plist probe beyond the initial file window", async () => { + const directory = await createTestTempDirectory("rea-xml-long-prolog-"); + const path = join(directory, "Model.archive"); + await writeFile( + path, + `${" ".repeat(8192)}`, + ); + + const result = await parseBinaryTarget(path); + expect(result.ok && result.value).toMatchObject({ + kind: "artifact", + format: "plist", + }); + expect(hasXmlPlistSignature(Buffer.from(``))).toBe( + false, + ); + }); + it("does not trust a ZIP-family extension without ZIP magic", async () => { const directory = await createTestTempDirectory("rea-fake-archive-"); const path = join(directory, "fake.zip"); diff --git a/tests/boundary/filesystem/ciCachePolicy.test.ts b/tests/boundary/filesystem/ciCachePolicy.test.ts new file mode 100644 index 0000000000..17bd6c9747 --- /dev/null +++ b/tests/boundary/filesystem/ciCachePolicy.test.ts @@ -0,0 +1,267 @@ +import { execFileSync } from "node:child_process"; +import { chmod, readFile, readdir, writeFile } from "node:fs/promises"; +import { delimiter, join } from "node:path"; + +import { describe, expect, it } from "vitest"; +import { parse } from "yaml"; +import { z } from "zod"; + +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +const repository = process.cwd(); +const actionPath = join(repository, ".github/actions/npm-cache/action.yml"); +const readAction = async () => + z + .object({ + outputs: z.record(z.string(), z.object({ value: z.string() })), + runs: z.object({ + steps: z.array( + z.object({ + id: z.string().optional(), + if: z.string().optional(), + uses: z.string().optional(), + with: z.record(z.string(), z.string()).optional(), + run: z.string().optional(), + env: z.record(z.string(), z.string()).optional(), + }), + ), + }), + }) + .parse(parse(await readFile(actionPath, "utf8"))); + +const readWorkflows = async () => { + const directory = join(repository, ".github/workflows"); + const files = (await readdir(directory)).filter((file) => + /\.ya?ml$/u.test(file), + ); + return Promise.all( + files.map(async (file) => ({ + file, + workflow: z + .object({ + jobs: z + .record( + z.string(), + z.object({ + steps: z.array(z.record(z.string(), z.unknown())).optional(), + }), + ) + .optional(), + }) + .parse(parse(await readFile(join(directory, file), "utf8"))), + })), + ); +}; + +describe("hosted npm download-cache policy", () => { + it("only allows shared writes for trusted original default-branch events", async () => { + const action = await readAction(); + const policy = action.runs.steps.find((step) => step.id === "context"); + const command = z.string().parse(policy?.run); + + const outputDirectory = await createTestTempDirectory( + "rea-ci-cache-policy-", + ); + const npmStub = join(outputDirectory, "npm"); + await writeFile( + npmStub, + String.raw`#!/bin/sh +printf '%s\n' 'C:\Users\runner with spaces\AppData\Local\npm-cache' +`, + ); + await chmod(npmStub, 0o755); + + for (const [event, ref, defaultBranch, expected] of [ + ["push", "refs/heads/main", "main", "true"], + ["schedule", "refs/heads/main", "main", "true"], + ["workflow_dispatch", "refs/heads/main", "main", "true"], + ["push", "refs/heads/stable", "stable", "true"], + ["pull_request", "refs/heads/main", "main", "false"], + ["pull_request_target", "refs/heads/main", "main", "false"], + ["workflow_call", "refs/heads/main", "main", "false"], + ["push", "refs/heads/feature/cache", "main", "false"], + ["push", "refs/heads/main", "stable", "false"], + ["push", "refs/heads/main", "", "false"], + ]) { + const outputPath = join( + outputDirectory, + `policy-${Date.now()}-${Math.random()}`, + ); + execFileSync("bash", ["-euo", "pipefail", "-c", command], { + env: { + ...process.env, + PATH: `${outputDirectory}${delimiter}${process.env.PATH ?? ""}`, + GITHUB_EVENT_NAME: event, + GITHUB_REF: ref, + GITHUB_OUTPUT: outputPath, + DEFAULT_BRANCH: defaultBranch, + LOCKFILES_HASH: "lock-hash", + RUNNER_OS: "Windows", + }, + }); + const output = await readFile(outputPath, "utf8"); + expect(output).toContain(`can-save=${expected}\n`); + expect(output).toContain( + `path=${String.raw`C:\Users\runner with spaces\AppData\Local\npm-cache`}\n`, + ); + expect(output).toContain( + `key=npm-download-Windows-${process.arch}-lock-hash\n`, + ); + } + }); + + it("uses a lockfile-, npmrc-, OS-, and runtime-architecture-scoped download cache", async () => { + const action = await readAction(); + const policy = action.runs.steps.find((step) => step.id === "context"); + const restoreAndSave = action.runs.steps.find((step) => + step.uses?.startsWith("actions/cache@"), + ); + const restoreOnly = action.runs.steps.find((step) => + step.uses?.startsWith("actions/cache/restore@"), + ); + + expect(action.outputs["can-save"]?.value).toBe( + "${{ steps.context.outputs.can-save }}", + ); + expect(action.outputs.path?.value).toBe( + "${{ steps.context.outputs.path }}", + ); + expect(policy?.env?.DEFAULT_BRANCH).toBe( + "${{ github.event.repository.default_branch }}", + ); + expect(policy?.env?.LOCKFILES_HASH).toBe( + "${{ hashFiles('package-lock.json', 'npm-shrinkwrap.json', '.npmrc') }}", + ); + expect(policy?.run).toContain( + 'cache_key="npm-download-${RUNNER_OS}-${runtime_arch}-${LOCKFILES_HASH}"', + ); + expect(restoreAndSave?.uses).toBe( + "actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830", + ); + expect(restoreAndSave?.if).toBe("steps.context.outputs.can-save == 'true'"); + expect(restoreOnly?.uses).toBe( + "actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830", + ); + expect(restoreOnly?.if).toBe("steps.context.outputs.can-save != 'true'"); + expect(restoreAndSave?.with?.path).toBe( + "${{ steps.context.outputs.path }}", + ); + expect(restoreOnly?.with?.path).toBe(restoreAndSave?.with?.path); + expect(restoreAndSave?.with?.key).toBe("${{ steps.context.outputs.key }}"); + expect(restoreOnly?.with?.key).toBe(restoreAndSave?.with?.key); + expect(restoreAndSave?.with).not.toHaveProperty("restore-keys"); + expect(restoreOnly?.with).not.toHaveProperty("restore-keys"); + }); + + it("replaces every inline setup-node npm cache immediately and preserves conditions", async () => { + const workflows = await readWorkflows(); + let npmCacheSteps = 0; + + for (const { file, workflow } of workflows) { + const source = await readFile( + join(repository, ".github/workflows", file), + "utf8", + ); + expect(source).not.toMatch(/^\s+cache:\s*npm\s*$/mu); + for (const [jobName, job] of Object.entries(workflow.jobs ?? {})) { + const steps = job.steps ?? []; + for (let index = 0; index < steps.length; index++) { + const step = steps[index]; + if (step?.uses !== "./.github/actions/npm-cache") continue; + npmCacheSteps++; + const setup = steps[index - 1]; + expect( + typeof setup?.uses === "string" && + setup.uses.startsWith("actions/setup-node@"), + `${file}:${jobName} npm cache action must immediately follow setup-node`, + ).toBe(true); + expect(step.if).toBe(setup?.if); + expect(setup?.with).not.toHaveProperty("cache", "npm"); + const npmCiIndex = steps.findIndex( + (candidate, candidateIndex) => + candidateIndex > index && candidate.run === "npm ci", + ); + expect( + npmCiIndex, + `${file}:${jobName} retains npm ci after cache`, + ).toBeGreaterThan(index); + } + } + } + + expect(npmCacheSteps).toBeGreaterThan(0); + }); +}); + +it("shares static task snapshots only after validation using the npm cache owner's write authority", async () => { + const workflows = await readWorkflows(); + const ci = workflows.find(({ file }) => file === "ci.yml"); + const steps = ci?.workflow.jobs?.static?.steps ?? []; + expect( + steps.find((step) => step.uses === "./.github/actions/npm-cache")?.id, + ).toBe("npm-cache"); + const restore = steps.find((step) => step.id === "static-cache"); + const save = steps.find( + (step) => step.name === "Save validated static task cache", + ); + expect(restore?.uses).toBe( + "actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830", + ); + expect(restore?.if).toBe("needs.changes.outputs.code == 'true'"); + expect(restore?.with).toMatchObject({ path: ".turbo/cache/" }); + expect(save?.uses).toBe( + "actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830", + ); + expect(save?.if).toContain("success()"); + expect(save?.if).toContain("steps.npm-cache.outputs.can-save == 'true'"); + expect(save?.with).toEqual({ + path: ".turbo/cache/", + key: "${{ steps.static-cache.outputs.cache-primary-key }}", + }); + expect(steps.indexOf(save!)).toBeGreaterThan( + steps.findIndex((step) => step.run === "npm run test:python"), + ); +}); + +it("passes the npm cache owner's selected path to package installation", async () => { + const workflows = await readWorkflows(); + const ci = workflows.find(({ file }) => file === "ci.yml"); + const steps = ci?.workflow.jobs?.["package-e2e"]?.steps ?? []; + const cacheIndex = steps.findIndex( + (step) => step.uses === "./.github/actions/npm-cache", + ); + expect(steps[cacheIndex]?.id).toBe("npm-cache"); + const consumerIndex = steps.findIndex( + (step) => step.run === "npm run verify:package", + ); + expect(consumerIndex).toBeGreaterThan(cacheIndex); + expect( + z.record(z.string(), z.unknown()).parse(steps[consumerIndex]?.env) + .npm_config_cache, + ).toBe("${{ steps.npm-cache.outputs.path }}"); +}); + +it("bounds the rolling static snapshot with Turbo age and size eviction", async () => { + const turbo = z + .object({ + cacheMaxAge: z.string(), + cacheMaxSize: z.string(), + }) + .parse(JSON.parse(await readFile(join(repository, "turbo.json"), "utf8"))); + // The static job restores the previous snapshot by prefix and saves the + // whole .turbo/cache directory again. Without local eviction every main + // commit carries all historical entries into a new immutable GitHub cache, + // growing transfers toward the 10 GB repository limit. + expect(turbo.cacheMaxAge).toMatch(/^[1-9]\d*(s|m|h|d|w)$/u); + expect(turbo.cacheMaxAge).not.toBe("0"); + const size = /^(\d+(?:\.\d+)?)(MB|GB|TB)$/iu.exec(turbo.cacheMaxSize); + expect(size, "cacheMaxSize must use an explicit MB/GB/TB budget").not.toBe( + null, + ); + const value = Number(size?.[1] ?? "0"); + const unit = (size?.[2] ?? "MB").toUpperCase(); + const megabytes = + unit === "TB" ? value * 1024 * 1024 : unit === "GB" ? value * 1024 : value; + expect(megabytes).toBeGreaterThan(0); + expect(megabytes).toBeLessThanOrEqual(1024); +}); diff --git a/tests/boundary/filesystem/ciChangeScope.test.ts b/tests/boundary/filesystem/ciChangeScope.test.ts index fd7abf65b2..e56c894ba3 100644 --- a/tests/boundary/filesystem/ciChangeScope.test.ts +++ b/tests/boundary/filesystem/ciChangeScope.test.ts @@ -144,6 +144,51 @@ it("keeps every translated README and authored guide on the documentation lane", }); }); +it.each(["cjs", "cts", "js", "jsx", "mjs", "mts", "ts", "tsx"])( + "selects static checks for authored documentation code (%s) without provider lanes", + async (extension) => { + const { directory, base } = await fixture(); + const head = await change(directory, [ + `docs/.vitepress/helpers/navigation.${extension}`, + ]); + const plan = await classify(directory, base, head); + const { CI_FLAGS } = await loadScopes(); + expect(plan).toMatchObject({ tier: "fast", code: true, docs: true }); + for (const flag of CI_FLAGS) + if (flag !== "code" && flag !== "docs") expect(plan[flag]).toBe(false); + }, +); + +it("retains code checks when documentation code is removed or renamed to Markdown", async () => { + const { directory } = await fixture(); + const base = await change(directory, ["docs/.vitepress/config.ts"]); + await git(directory, [ + "mv", + "docs/.vitepress/config.ts", + "docs/configuration.md", + ]); + const head = await commit(directory); + expect(await classify(directory, base, head)).toMatchObject({ + code: true, + docs: true, + runtime: false, + package: false, + }); +}); + +it("selects code checks for documentation filenames containing line separators", async () => { + const { directory, base } = await fixture(); + const head = await change(directory, [ + "docs/.vitepress/navigation\u2028menu.ts", + ]); + expect(await classify(directory, base, head)).toMatchObject({ + code: true, + docs: true, + runtime: false, + package: false, + }); +}); + it.each([ "job-timings.mjs", "job-timings-batch.mjs", @@ -215,6 +260,84 @@ it("does not charge a documentation PR for unrelated implementation updates on m }); }); +it.each([ + "src/gdb/GdbSessionManager.ts", + "src/rizin/RizinDebugSessionManager.ts", + "src/cutter/CutterBridgeClient.ts", + "src/objdump/ObjdumpCommand.ts", + "tests/boundary/providers/GdbSessionManagerCoverage.test.ts", + "tests/boundary/providers/GdbCommandQueue.test.ts", + "tests/boundary/providers/RizinDebugSessionCoverage.part3.test.ts", + "tests/boundary/providers/RizinCommandTimeout.test.ts", + "tests/boundary/providers/RizinTimeoutEvidence.test.ts", + "tests/boundary/providers/GdbSessionReviewRegressions.test.ts", + "tests/boundary/providers/gdbTestHelpers.ts", + "tests/boundary/providers/RizinDebugSessionReviewRegressions.test.ts", + "tests/boundary/mcp/gdbCancellationMcp.test.ts", + "tests/boundary/mcp/rizinDebuggerCancellationMcp.test.ts", + "tests/composition/reverseEngineeringAdmission.test.ts", + "tests/support/reverseEngineeringMcpHarness.ts", + "tests/support/reverseEngineeringCompositionEdgeCases.ts", + "tests/support/reverseEngineeringCompositionFixtures.ts", + "tests/support/reverseEngineeringService.ts", + "tests/support/ownedRizinCommandEvidenceTests.ts", + "tests/support/reverseDebuggerMcpTestHelpers.ts", + "tests/boundary/cli/gdbStartupCleanupCli.test.ts", + "vitest.reverse-tools.config.ts", + "vitest.reverse-tools-semantics.config.ts", + "stryker.config.mjs", + "stryker.semantics.config.mjs", + "tsconfig.reverse-tools-mutation.json", + "scripts/prepare-reverse-tools-mutation.mjs", + "scripts/lib/reverse-tools-mutation-targets.mjs", +])( + "selects the reverse-tool coverage and mutation gate for %s", + async (path) => { + const { directory, base } = await fixture(); + const head = await change(directory, [path]); + expect(await classify(directory, base, head)).toMatchObject({ + code: true, + reverse_tools: true, + }); + }, +); + +const reverseToolsVitestConfig = await readFile( + new URL("../../../vitest.reverse-tools.config.ts", import.meta.url), + "utf8", +); +const reverseToolsVitestInputs = [ + ...reverseToolsVitestConfig.matchAll(/"((?:\.\/)?(?:src|tests)\/[^"]+)"/gu), +].flatMap((match) => + match[1] === undefined ? [] : [match[1].replace(/^\.\//u, "")], +); +const reverseToolsMutationConfig = JSON.parse( + await readFile( + new URL("../../../tsconfig.reverse-tools-mutation.json", import.meta.url), + "utf8", + ), +) as { include: string[] }; +const reverseToolsInputs = [ + ...new Set([ + ...reverseToolsVitestInputs, + ...reverseToolsMutationConfig.include.map((path) => + path.replace(/^\.\//u, ""), + ), + ]), +]; + +it.each(reverseToolsInputs)( + "selects the reverse-tool quality gate for configured input %s", + async (path) => { + const { directory, base } = await fixture(); + const head = await change(directory, [path]); + expect(await classify(directory, base, head)).toMatchObject({ + code: true, + reverse_tools: true, + }); + }, +); + it.each([ "src/README.md", "README.md.ts", @@ -293,7 +416,10 @@ it.each(["", "not-a-commit"])( }, ); -it("selects only a trusted successful ancestor from recent CI push runs", async () => { +it("selects only a trusted successful ancestor from recent CI push runs", async ({ + skip, +}) => { + skip(process.platform === "win32", "the fake gh is a POSIX shell script"); const { directory, base } = await fixture(); const head = await change(directory, ["src/evm/client.ts"]); const bin = join(directory, "fake-bin"); @@ -326,7 +452,11 @@ it("selects only a trusted successful ancestor from recent CI push runs", async const output = join(directory, "baseline-output"); const result = await execFileAsync( process.execPath, - [new URL("../../../scripts/ci/baseline.mjs", import.meta.url).pathname], + [ + fileURLToPath( + new URL("../../../scripts/ci/baseline.mjs", import.meta.url), + ), + ], { cwd: directory, env: { @@ -344,7 +474,8 @@ it("selects only a trusted successful ancestor from recent CI push runs", async expect(await readFile(output, "utf8")).toBe(`base=${base}\nreason=\n`); }); -it("returns an empty baseline when GitHub lookup fails", async () => { +it("returns an empty baseline when GitHub lookup fails", async ({ skip }) => { + skip(process.platform === "win32", "the fake gh is a POSIX shell script"); const { directory, base } = await fixture(); const head = await change(directory, ["src/evm/client.ts"]); const bin = join(directory, "fake-bin"); @@ -354,7 +485,11 @@ it("returns an empty baseline when GitHub lookup fails", async () => { await chmod(gh, 0o755); const result = await execFileAsync( process.execPath, - [new URL("../../../scripts/ci/baseline.mjs", import.meta.url).pathname], + [ + fileURLToPath( + new URL("../../../scripts/ci/baseline.mjs", import.meta.url), + ), + ], { cwd: directory, env: { @@ -476,6 +611,10 @@ it.each([ { code: true, runtime: false, browser: false, package: false }, ], ["website/src/index.ts", { code: false, website: true, package: false }], + [ + "website/public/examples/notes-electron/preload.js", + { code: true, website: true, package: false }, + ], [ "website/locales/zh-CN/messages.po", { code: false, website: true, package: false }, @@ -769,6 +908,10 @@ it("rejects skipped selected work while allowing unselected lanes to skip", asyn }); it.each([ + [ + ".github/workflows/mutation.yml", + { workflows: true, code: true, reverse_tools: true, package: false }, + ], [ ".github/workflows/sqlite-runtime.yml", { workflows: true, code: true, package: false, website: false }, @@ -793,7 +936,16 @@ it.each([ "tsconfig.build.json", { code: true, package: true, browser: true, evm: true }, ], - ["docs/.vitepress/config.ts", { code: false, docs: true, package: false }], + ["docs/.vitepress/config.ts", { code: true, docs: true, package: false }], + [ + "docs/package.json", + { code: true, docs: true, runtime: false, package: false }, + ], + [ + "docs/.vitepress/package.json", + { code: true, docs: true, runtime: false, package: false }, + ], + ["docs/data.json", { code: false, docs: true, package: false }], ["CHANGELOG.md", { code: false, docs: true, package: false }], ["SECURITY.md", { code: false, docs: true, package: false }], [".github/pull_request_template.md", { code: false, docs: true }], @@ -811,6 +963,10 @@ it.each([ ".github/workflows/release.yml", { workflows: true, code: true, package: false, website: false }, ], + [ + ".github/workflows/pr-readiness.yml", + { workflows: true, code: true, package: false, website: false }, + ], [ ".github/workflows/real-ghidra-windows.yml", { workflows: true, code: true, package: false, website: false }, diff --git a/tests/boundary/filesystem/ciFailureReport.test.ts b/tests/boundary/filesystem/ciFailureReport.test.ts new file mode 100644 index 0000000000..c2a282906c --- /dev/null +++ b/tests/boundary/filesystem/ciFailureReport.test.ts @@ -0,0 +1,435 @@ +import { readFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { expect, it } from "vitest"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { normalizeWorkflowRun } from "../../../scripts/ci/workflow-run.mjs"; + +import { + readStaticCheckArtifact, + selectStaticCheckArtifact, +} from "../../../scripts/ci/artifacts.mjs"; +import { + classifyFailure, + createGitHubApi, + reportCiFailures, + type CiRun, + type CiJob, + type FailureReportApi, +} from "../../../scripts/ci/failure-report.mjs"; + +const sha = (digit: string) => digit.repeat(40); +const diagnostic = { + kind: "lint", + path: "src/example.ts", + code: "no-debugger", + message: "Unexpected debugger statement.", + severity: "error", +} as const; +const context = ( + sourceSha: string, + taskHash = "a".repeat(16), + prHeadSha?: string, +) => ({ + version: 1 as const, + sourceSha, + ...(prHeadSha ? { prHeadSha } : {}), + sourceClean: true, + nodeVersion: "v24.18.0", + npmVersion: "11.16.0", + turboVersion: "2.11.7", + platform: "linux", + architecture: "x64", + taskHashes: { "lint:built": taskHash }, +}); +const run = (overrides: Partial = {}) => + normalizeWorkflowRun({ + id: 10, + run_attempt: 1, + status: "completed", + conclusion: "failure", + event: "pull_request", + path: ".github/workflows/ci.yml@main", + head_sha: sha("a"), + head_branch: "feature", + repository: { full_name: "owner/rea" }, + pull_requests: [ + { base: { sha: sha("b"), repo: { full_name: "owner/rea" } } }, + ], + ...overrides, + }); +const job = (overrides: Partial = {}) => ({ + name: "Static checks", + status: "completed", + conclusion: "failure", + ...overrides, +}); + +const comparison = (): Parameters[0] => ({ + repository: "owner/rea", + defaultBranch: "main", + run: run(), + job: job(), + diagnostics: [diagnostic], + context: context(sha("a")), + mainRun: run({ + id: 20, + event: "push", + head_sha: sha("c"), + head_branch: "main", + pull_requests: [], + }), + mainJob: job(), + mainDiagnostics: [diagnostic], + mainContext: context(sha("c")), + executionSourcesVerified: true, + ancestry: { status: "ahead", mergeBaseSha: sha("c") }, + blobs: { "src/example.ts": true }, +}); + +it("attributes only an identical structured lint failure with matching task, context, ancestry, and source blob", () => { + const result = classifyFailure(comparison()); + expect(result).toEqual({ + classification: "matching-main-diagnostics", + reason: + "supported_diagnostics_match_failed_task_and_source_on_verified_main_ancestor", + }); +}); + +it.each([ + ["task hash", { mainContext: context(sha("c"), "b".repeat(16)) }], + ["dirty source", { context: { ...context(sha("a")), sourceClean: false } }], + [ + "toolchain", + { mainContext: { ...context(sha("c")), turboVersion: "other" } }, + ], + [ + "source sha", + { context: context(sha("d")), executionSourcesVerified: false }, + ], + [ + "diagnostic", + { mainDiagnostics: [{ ...diagnostic, message: "different message" }] }, + ], + ["content", { blobs: { "src/example.ts": false } }], + ["ancestry", { ancestry: { status: "diverged", mergeBaseSha: sha("e") } }], + ["canceled job", { job: job({ conclusion: "cancelled" }) }], + ["not a static job", { job: job({ name: "Test shard (1)" }) }], +] as const)("does not attribute on mismatched %s", (_label, overrides) => { + const result = classifyFailure({ ...comparison(), ...overrides }); + expect(result.classification).toBe("unknown"); +}); + +it.each([ + [{}, "diagnostic_source_blob_identity_unavailable"], + [ + { "src/example.ts": false }, + "diagnostic_source_content_not_identical_at_main_and_tested_base", + ], +] as const)( + "distinguishes unavailable blob identity from verified differences (%s)", + (blobs, reason) => { + expect(classifyFailure({ ...comparison(), blobs })).toEqual({ + classification: "unknown", + reason, + }); + }, +); + +it("writes informational JSON and Markdown from the supplied manifest using only read API calls", async () => { + const directory = await createTestTempDirectory("rea-ci-failure-test-"); + const calls: string[] = []; + const manifest = { + repository: "owner/rea", + truncation: [], + coverage: { exhaustive: false }, + runs: [ + { + id: 10, + runAttempt: 1, + workflow: "CI", + event: "pull_request", + conclusion: "failure", + status: "collected", + }, + { + id: 11, + runAttempt: 1, + workflow: "CI", + event: "pull_request", + conclusion: "success", + status: "collected", + }, + ], + }; + const failedRun = run(); + const api: FailureReportApi = { + async getRun(_repository, id) { + calls.push(`run:${id}`); + return failedRun; + }, + async getJobs(_repository, id) { + calls.push(`jobs:${id}`); + return [{ ...job(), run_id: Number(id), run_attempt: 1 }]; + }, + async getStaticEvidence() { + calls.push("artifact"); + return null; + }, + async getAncestry() { + throw new Error("must not compare without evidence"); + }, + async verifyExecutionSource() { + throw new Error("must not verify source without evidence"); + }, + async getBlobIdentity() { + throw new Error("must not inspect blobs without evidence"); + }, + }; + const report = await reportCiFailures({ + repository: "owner/rea", + defaultBranch: "main", + batchManifest: manifest, + api, + outputDirectory: directory, + }); + expect(calls).toEqual(["run:10", "jobs:10", "artifact"]); + expect(report).toMatchObject({ + informationalOnly: true, + gateEffect: "none; this report never changes CI conclusions", + counts: { unknown: 1 }, + }); + const json = JSON.parse( + await readFile(join(directory, "report.json"), "utf8"), + ); + const markdown = await readFile(join(directory, "report.md"), "utf8"); + expect(json.failures[0]).toMatchObject({ + classification: "unknown", + reason: "no_matching_verified_main_failure", + }); + expect(markdown).toContain("Informational only"); + expect(markdown).toContain("Unknown means evidence was incomplete"); +}); + +it.each([ + [22, false], + [6, true], +] as const)( + "finds verified main evidence after %s newer PR runs (extra incorrect main hint: %s)", + async (prCount, incorrectMainHints) => { + const directory = await createTestTempDirectory("rea-ci-main-proof-"); + const mainSha = sha("c"); + const prHeadSha = sha("a"); + const mergeSha = sha("d"); + const mainRun = run({ + id: 20, + event: "push", + head_sha: mainSha, + head_branch: "main", + pull_requests: [], + }); + const prRun = run({ + head_sha: prHeadSha, + pull_requests: [ + { base: { sha: mainSha, repo: { full_name: "owner/rea" } } }, + ], + }); + const evidence = (sourceSha: string, prHead?: string) => ({ + context: context(sourceSha, undefined, prHead), + diagnostics: [diagnostic], + }); + const lookups: number[] = []; + const api: FailureReportApi = { + async getRun(_repository, id) { + lookups.push(Number(id)); + return Number(id) === 20 ? mainRun : { ...prRun, id }; + }, + async getJobs(_repository, id) { + return [{ ...job(), run_id: Number(id), run_attempt: 1 }]; + }, + async getStaticEvidence(_repository, selectedRun) { + return Number(selectedRun.id) === 20 + ? evidence(mainSha) + : evidence(mergeSha, prHeadSha); + }, + async verifyExecutionSource(_repository, selectedRun, sourceContext) { + return selectedRun.event === "pull_request" + ? sourceContext.sourceSha === mergeSha && + sourceContext.prHeadSha === selectedRun.head_sha + : sourceContext.sourceSha === selectedRun.head_sha; + }, + async getAncestry() { + return { status: "ahead", mergeBaseSha: mainSha }; + }, + async getBlobIdentity() { + return { "src/example.ts": true }; + }, + }; + const result = await reportCiFailures({ + repository: "owner/rea", + defaultBranch: "main", + batchManifest: { + repository: "owner/rea", + runs: [ + ...(incorrectMainHints + ? [ + { + id: 30, + runAttempt: 1, + workflow: "CI", + event: "push", + headBranch: "main", + conclusion: "failure", + status: "collected", + }, + ] + : []), + ...Array.from({ length: prCount }, (_, index) => ({ + id: 100 + index, + runAttempt: 1, + workflow: "CI", + event: "pull_request", + headBranch: "feature", + conclusion: "failure", + status: "collected", + })), + { + id: 20, + runAttempt: 1, + workflow: "CI", + event: "push", + headBranch: "main", + conclusion: "failure", + status: "collected", + }, + ], + }, + api, + outputDirectory: directory, + }); + expect(result.failures[0]).toMatchObject({ + runId: 100, + classification: "matching-main-diagnostics", + reason: + "supported_diagnostics_match_failed_task_and_source_on_verified_main_ancestor", + }); + expect(lookups).toHaveLength( + Math.min(prCount, 20) + 1 + Number(incorrectMainHints), + ); + expect(lookups[0]).toBe(incorrectMainHints ? 30 : 20); + expect(new Set(lookups).size).toBe(lookups.length); + expect(result.boundedInput.mainCandidateRuns).toBe(1); + expect(result.failures).toHaveLength(Math.min(prCount, 20)); + }, +); + +it("proves checkout merge parents in base-then-PR-head order and rejects false identities", async () => { + const baseSha = sha("b"); + const headSha = sha("a"); + const mergeSha = sha("c"); + const prRun = run({ + head_sha: headSha, + pull_requests: [ + { base: { sha: baseSha, repo: { full_name: "owner/rea" } } }, + ], + }); + let parents = [baseSha, headSha]; + const calls: string[] = []; + const api = createGitHubApi({ + token: "fixture-token", + apiRequest: async (_method, endpoint) => { + calls.push(endpoint); + return { + sha: mergeSha, + parents: parents.map((parentSha) => ({ sha: parentSha })), + }; + }, + }); + const mergeContext = context(mergeSha, undefined, headSha); + expect( + await api.verifyExecutionSource("owner/rea", prRun, mergeContext), + ).toBe(true); + expect(calls).toEqual([`repos/owner/rea/commits/${mergeSha}`]); + + parents = [headSha, baseSha]; + expect( + await api.verifyExecutionSource("owner/rea", prRun, mergeContext), + ).toBe(false); + expect( + await api.verifyExecutionSource("owner/rea", prRun, { + ...mergeContext, + prHeadSha: sha("e"), + }), + ).toBe(false); +}); + +it("uses static artifacts only when listing is complete, unique, and tied to this run attempt", () => { + const artifact = { + id: 12, + name: "static-check-reports", + expired: false, + workflow_run: { id: 10, head_sha: sha("a") }, + }; + const runRecord = { id: 10, head_sha: sha("a") }; + expect( + selectStaticCheckArtifact( + { total_count: 1, artifacts: [artifact] }, + runRecord, + 1, + ), + ).toBe(artifact); + expect( + selectStaticCheckArtifact( + { total_count: 2, artifacts: [artifact] }, + runRecord, + 1, + ), + ).toBeNull(); + expect( + selectStaticCheckArtifact( + { total_count: 2, artifacts: [artifact, artifact] }, + runRecord, + 1, + ), + ).toBeNull(); + expect( + selectStaticCheckArtifact( + { + total_count: 1, + artifacts: [ + { ...artifact, workflow_run: { id: 11, head_sha: sha("a") } }, + ], + }, + runRecord, + 1, + ), + ).toBeNull(); + expect( + selectStaticCheckArtifact( + { total_count: 1, artifacts: [artifact] }, + runRecord, + 2, + ), + ).toBeNull(); +}); + +it("rejects artifact redirects that could send the bearer token to a non-HTTPS host", async () => { + const calls: string[] = []; + await expect( + readStaticCheckArtifact({ + repository: "owner/rea", + artifactId: 12, + token: "fixture-token", + fetchImpl: (async (url: string | URL | Request) => { + calls.push(String(url)); + return new Response(null, { + status: 302, + headers: { location: "http://untrusted.example/artifact.zip" }, + }); + }) as typeof fetch, + }), + ).rejects.toThrow("static_artifact_redirect_invalid"); + expect(calls).toEqual([ + "https://api.github.com/repos/owner/rea/actions/artifacts/12/zip", + ]); +}); diff --git a/tests/boundary/filesystem/ciFailureReportBudget.test.ts b/tests/boundary/filesystem/ciFailureReportBudget.test.ts new file mode 100644 index 0000000000..1caacde165 --- /dev/null +++ b/tests/boundary/filesystem/ciFailureReportBudget.test.ts @@ -0,0 +1,407 @@ +import { expect, it } from "vitest"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { + reportCiFailures, + createGitHubApi, + type CiRun, + type FailureReportApi, + type StaticDiagnostic, + type StaticEvidence, +} from "../../../scripts/ci/failure-report.mjs"; + +const repository = "owner/rea"; +const mainSha = "c".repeat(40); +const prSha = "a".repeat(40); +const diagnostic: StaticDiagnostic = { + kind: "lint", + path: "src/example.ts", + code: "no-debugger", + message: "Unexpected debugger statement.", + severity: "error", +}; +const evidence = ( + run: CiRun, + diagnostics: readonly StaticDiagnostic[], +): StaticEvidence => ({ + context: { + version: 1, + sourceSha: run.head_sha!, + sourceClean: true, + nodeVersion: "v24.18.0", + npmVersion: "11.16.0", + turboVersion: "2.11.7", + platform: "linux", + architecture: "x64", + taskHashes: { "lint:built": "a".repeat(16) }, + }, + diagnostics, +}); +const entry = (id: number, event = "pull_request", runAttempt = 1) => ({ + id, + runAttempt, + workflow: "CI", + event, + headBranch: event === "pull_request" ? "feature" : "main", + status: "collected", + conclusion: "failure", +}); + +function fixture(mainIds: readonly number[], prCount: number) { + const jobs: number[] = []; + const artifacts: number[] = []; + const paths: (readonly string[])[] = []; + const runs: Array< + Omit, "event"> & { event?: string } + > = [ + ...mainIds.map((id) => entry(id, "schedule")), + ...Array.from({ length: prCount }, (_, index) => entry(100 + index)), + ]; + const api: FailureReportApi = { + async getRun(_repository, id) { + const selected = runs.find((run) => run.id === Number(id))!; + return { + id, + run_attempt: selected.runAttempt, + status: "completed", + conclusion: "failure", + ...(selected.event ? { event: selected.event } : {}), + head_branch: selected.headBranch, + path: ".github/workflows/ci.yml@main", + head_sha: mainIds.includes(Number(id)) ? mainSha : prSha, + repository: { full_name: repository }, + pull_requests: + selected.event === "pull_request" + ? [{ base: { sha: mainSha, repo: { full_name: repository } } }] + : [], + }; + }, + async getJobs(_repository, id, attempt) { + jobs.push(Number(id)); + return [ + { + name: "Static checks", + status: "completed", + conclusion: "failure", + run_id: Number(id), + run_attempt: attempt, + }, + ]; + }, + async getStaticEvidence(_repository, run) { + artifacts.push(Number(run.id)); + return evidence(run, [diagnostic]); + }, + async getAncestry() { + return { status: "identical", mergeBaseSha: mainSha }; + }, + async getBlobIdentity(_repository, _ancestor, _descendant, requested) { + paths.push(requested); + return Object.fromEntries(requested.map((path) => [path, true])); + }, + async verifyExecutionSource() { + return true; + }, + }; + return { api, runs, jobs, artifacts, paths }; +} + +async function report(input: ReturnType) { + return await reportCiFailures({ + repository, + defaultBranch: "main", + batchManifest: { repository, runs: input.runs }, + api: input.api, + outputDirectory: await createTestTempDirectory("rea-ci-budget-"), + }); +} + +it("selects PR failures before the cap even after a day of scheduled main failures", async () => { + const input = fixture( + Array.from({ length: 24 }, (_, index) => 20 + index), + 1, + ); + input.runs.push({ + id: 200, + runAttempt: 1, + workflow: "CI", + headBranch: "feature", + status: "collected", + conclusion: "failure", + }); + const result = await report(input); + expect(result.failures).toMatchObject([ + { runId: 100, classification: "matching-main-diagnostics" }, + ]); + expect(result.boundedInput).toMatchObject({ + candidateRuns: 1, + mainCandidateRuns: 5, + manifestTruncated: false, + }); + expect(input.jobs).toEqual([100, 20]); +}); + +it("reuses five main job/artifact observations across twenty completed PR comparisons", async () => { + const input = fixture([20, 21, 22, 23, 24], 20); + const load = input.api.getStaticEvidence; + input.api.getStaticEvidence = async (...args) => { + const result = await load(...args); + return Number(args[1].id) < 24 + ? evidence(args[1], [{ ...diagnostic, message: "different" }]) + : result; + }; + const result = await report(input); + expect(result.counts).toEqual({ + "matching-main-diagnostics": 20, + unknown: 0, + }); + expect(input.paths).toHaveLength(100); + expect(input.jobs).toHaveLength(25); + expect(input.artifacts).toHaveLength(25); + for (const id of [20, 21, 22, 23, 24]) { + expect(input.jobs.filter((item) => item === id)).toHaveLength(1); + expect(input.artifacts.filter((item) => item === id)).toHaveLength(1); + } +}); + +it.each(["missing", "artifact error", "job error", "wrong job attempt"])( + "reuses unavailable main evidence without retries: %s", + async (failure) => { + const input = fixture([20], 3); + const load = input.api.getStaticEvidence; + const jobs = input.api.getJobs; + input.api.getJobs = async (...args) => { + const result = await jobs(...args); + if (Number(args[1]) !== 20) return result; + if (failure === "job error") throw new Error("job_snapshot_unavailable"); + return failure === "wrong job attempt" + ? result.map((job) => ({ ...job, run_attempt: 2 })) + : result; + }; + input.api.getStaticEvidence = async (...args) => { + const result = await load(...args); + if (Number(args[1].id) !== 20) return result; + if (failure === "artifact error") throw new Error("artifact_unavailable"); + return null; + }; + const result = await report(input); + expect(result.counts.unknown).toBe(3); + expect(input.jobs.filter((id) => id === 20)).toHaveLength(1); + expect(input.artifacts.filter((id) => id === 20)).toHaveLength( + failure === "job error" || failure === "wrong job attempt" ? 0 : 1, + ); + }, +); + +it.each([1, 2])( + "refreshes unavailable main evidence in the next report (attempt %s)", + async (attempt) => { + const input = fixture([20], 1); + expect((await report(input)).counts["matching-main-diagnostics"]).toBe(1); + input.runs[0]!.runAttempt = attempt; + const load = input.api.getStaticEvidence; + input.api.getStaticEvidence = async (...args) => { + const result = await load(...args); + return Number(args[1].id) === 20 ? null : result; + }; + expect((await report(input)).counts.unknown).toBe(1); + expect(input.jobs.filter((id) => id === 20)).toHaveLength(2); + expect(input.artifacts.filter((id) => id === 20)).toHaveLength(2); + }, +); + +it.each([1, 51])( + "fetches only PR paths with %s PR diagnostics and noisy main evidence", + async (count) => { + const input = fixture([20], 1); + const load = input.api.getStaticEvidence; + const diagnostics = Array.from({ length: count }, (_, index) => ({ + ...diagnostic, + path: `src/pr-${index}.ts`, + })); + input.api.getStaticEvidence = async (...args) => { + await load(...args); + const noise = + Number(args[1].id) === 20 + ? Array.from({ length: 60 }, (_, index) => ({ + ...diagnostic, + path: `src/main-${index}.ts`, + })) + : []; + return evidence(args[1], [...noise, ...diagnostics]); + }; + const result = await report(input); + expect(input.paths).toEqual([ + diagnostics.slice(0, 50).map((item) => item.path), + ]); + expect(result.failures[0]!.classification).toBe( + count === 1 ? "matching-main-diagnostics" : "unknown", + ); + if (count > 50) + expect(result.failures[0]!.reason).toBe( + "diagnostic_source_blob_identity_unavailable", + ); + }, +); + +it.each(["jobs", "artifact", "comparison"])( + "continues after an isolated main %s failure and preserves its provenance", + async (failure) => { + const input = fixture([20, 21], 3); + const jobs = input.api.getJobs; + const load = input.api.getStaticEvidence; + input.api.getJobs = async (...args) => { + const result = await jobs(...args); + if (Number(args[1]) === 20 && failure === "jobs") + throw new Error("main_jobs_unavailable"); + return result; + }; + input.api.getStaticEvidence = async (...args) => { + const result = await load(...args); + if (Number(args[1].id) === 20 && failure === "artifact") + throw new Error("main_artifact_unavailable"); + return Number(args[1].id) === 20 && failure === "comparison" + ? { + ...result!, + context: { ...result!.context, sourceSha: "d".repeat(40) }, + } + : result; + }; + input.api.verifyExecutionSource = async (_repository, run, context) => { + if (Number(run.id) === 20 && context.sourceSha !== run.head_sha) + throw new Error("main_comparison_unavailable"); + return true; + }; + const result = await report(input); + expect(result.counts).toEqual({ + "matching-main-diagnostics": 3, + unknown: 0, + }); + for (const item of result.failures) { + expect(item.candidateFailures).toEqual([ + { + runId: 20, + runAttempt: 1, + reason: `main_${failure}_unavailable`, + }, + ]); + } + expect(input.jobs.filter((id) => id === 20)).toHaveLength(1); + expect(input.jobs.filter((id) => id === 21)).toHaveLength(1); + expect(input.artifacts.filter((id) => id === 20)).toHaveLength( + failure === "jobs" ? 0 : 1, + ); + expect(input.artifacts.filter((id) => id === 21)).toHaveLength(1); + }, +); + +it.each([0, 1, 2])( + "preserves every failed job when artifact analysis fails at position %s", + async (position) => { + const input = fixture([20], 1); + const jobs = [ + { name: "Python producer", conclusion: "failure" }, + { name: "Test shard (1)", conclusion: "timed_out" }, + ]; + jobs.splice(position, 0, { name: "Static checks", conclusion: "failure" }); + input.api.getJobs = async (_repository, id, attempt) => + jobs.map((job) => ({ + ...job, + status: "completed", + run_id: Number(id), + run_attempt: attempt, + })); + input.api.getStaticEvidence = async (_repository, _run, job) => { + if (job.name === "Static checks") + throw new Error("static_artifact_unavailable"); + return null; + }; + const result = await report(input); + expect(result.counts.unknown).toBe(3); + expect(result.failures).toEqual( + jobs.map((job) => ({ + runId: 100, + runAttempt: 1, + runHead: prSha, + event: "pull_request", + job: job.name, + conclusion: job.conclusion, + classification: "unknown", + reason: + job.name === "Static checks" + ? "static_artifact_unavailable" + : "no_matching_verified_main_failure", + })), + ); + }, +); + +it("keeps unavailable job inventories at run scope", async () => { + const input = fixture([], 1); + input.api.getJobs = async () => { + throw new Error("job_inventory_unavailable"); + }; + expect((await report(input)).failures).toEqual([ + { + runId: 100, + runAttempt: 1, + classification: "unknown", + reason: "job_inventory_unavailable", + }, + ]); +}); + +it.each([ + ["request", "github_api_temporarily_unavailable"], + ["budget", "github_api_request_budget_exhausted"], + ["missing sha", "github_content_blob_identity_unavailable"], + ["invalid sha", "github_content_blob_identity_unavailable"], + ["directory", "github_content_blob_identity_unavailable"], +] as const)( + "retains unavailable blob evidence as a comparison error: %s", + async (failure, reason) => { + const input = fixture([20], 1); + const contentApi = createGitHubApi({ + token: "fixture-token", + apiRequest: async () => { + if (failure === "request") throw new Error(reason); + if (failure === "missing sha") return { type: "file" }; + if (failure === "invalid sha") + return { type: "file", sha: "not-a-sha" }; + if (failure === "directory") return []; + return { type: "file", sha: "e".repeat(40) }; + }, + }); + if (failure === "budget") { + for (let id = 1; id <= 250; id++) await contentApi.getRun(repository, id); + } + input.api.getBlobIdentity = contentApi.getBlobIdentity; + const result = await report(input); + expect(result.failures).toMatchObject([ + { + job: "Static checks", + classification: "unknown", + reason: "no_matching_verified_main_failure", + candidateFailures: [{ runId: 20, runAttempt: 1, reason }], + }, + ]); + }, +); + +it.each([true, false])( + "reports blob equality only from verified file SHAs (equal: %s)", + async (equal) => { + const api = createGitHubApi({ + token: "fixture-token", + apiRequest: async (_method, endpoint) => ({ + type: "file", + sha: + equal || endpoint.includes(`ref=${mainSha}`) + ? "e".repeat(40) + : "f".repeat(40), + }), + }); + expect( + await api.getBlobIdentity(repository, mainSha, prSha, ["src/example.ts"]), + ).toEqual({ "src/example.ts": equal }); + }, +); diff --git a/tests/boundary/filesystem/ciFeedback.test.ts b/tests/boundary/filesystem/ciFeedback.test.ts new file mode 100644 index 0000000000..e813b0d7c9 --- /dev/null +++ b/tests/boundary/filesystem/ciFeedback.test.ts @@ -0,0 +1,453 @@ +import { execFile } from "node:child_process"; +import { mkdir, readFile, readdir, rm, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { promisify } from "node:util"; +import { runInNewContext } from "node:vm"; +import { expect, it } from "vitest"; +import { parse } from "yaml"; +import { z } from "zod"; + +import { + parseStaticTaskContext, + recordStaticTaskContext, +} from "../../../scripts/ci/static-task-context.mjs"; +import { createGhApiRequest } from "../../../scripts/ci/github-api.mjs"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +const execute = promisify(execFile); +const scopeFlags = async () => + z + .object({ CI_FLAGS: z.array(z.string()) }) + .parse( + await import( + new URL("../../../scripts/ci/scopes.mjs", import.meta.url).href + ), + ).CI_FLAGS; +const workflow = async () => + z + .object({ + jobs: z.record( + z.string(), + z + .object({ + needs: z.union([z.string(), z.array(z.string())]).optional(), + if: z.string().optional(), + steps: z + .array( + z + .object({ + run: z.string().optional(), + if: z.string().optional(), + uses: z.string().optional(), + with: z.record(z.string(), z.unknown()).optional(), + env: z.record(z.string(), z.string()).optional(), + }) + .passthrough(), + ) + .optional(), + }) + .passthrough(), + ), + }) + .parse( + parse( + await readFile( + new URL("../../../.github/workflows/ci.yml", import.meta.url), + "utf8", + ), + ), + ).jobs; + +const evaluateCondition = ( + condition: string, + context: Record, +): boolean => + z + .boolean() + .parse( + runInNewContext( + condition.startsWith("${{") ? condition.slice(3, -2) : condition, + context, + { timeout: 1_000 }, + ), + ); + +it.each(["fast", "full"] as const)( + "requires coverage and both actual mutation preflights in %s qualification", + async (tier) => { + const jobs = await workflow(); + expect(jobs["reverse-tools-quality"]?.if).toBe( + "needs.changes.outputs.reverse_tools == 'true'", + ); + expect(jobs.required?.needs).toContain("reverse-tools-quality"); + const steps = jobs["reverse-tools-quality"]?.steps ?? []; + const commands = steps + .filter( + (step) => + step.if === undefined || + evaluateCondition(step.if, { + needs: { changes: { outputs: { tier } } }, + cancelled: () => false, + success: () => true, + failure: () => false, + always: () => true, + }), + ) + .flatMap((step) => step.run?.trim().split("\n") ?? []) + .map((command) => command.trim()); + const mutationCommands = commands.filter( + (command) => command.includes("mutation") || command.includes("stryker"), + ); + expect(mutationCommands).toEqual([ + "node scripts/prepare-reverse-tools-mutation.mjs", + "npx --no-install stryker run stryker.config.mjs --dryRunOnly", + "npx --no-install stryker run stryker.semantics.config.mjs --dryRunOnly", + ]); + const coverage = commands.indexOf("npm run test:coverage:reverse-tools"); + expect(coverage).toBeGreaterThanOrEqual(0); + expect(coverage).toBeLessThan(commands.indexOf(mutationCommands[0]!)); + + const flags = await scopeFlags(); + const context: Record< + string, + { result: string; outputs?: Record } + > = Object.fromEntries( + Object.keys(jobs).map((name) => [name, { result: "success" }]), + ); + context.changes = { + result: "success", + outputs: { + ...Object.fromEntries( + flags.map((flag) => [flag, tier === "full" ? "true" : "false"]), + ), + tier, + code: "true", + reverse_tools: "true", + }, + }; + for (const result of [ + "success", + "failure", + "cancelled", + "skipped", + "missing", + ]) { + const needs = { ...context }; + if (result === "missing") delete needs["reverse-tools-quality"]; + else needs["reverse-tools-quality"] = { result }; + const execution = execute( + process.execPath, + [ + fileURLToPath( + new URL("../../../scripts/ci/required.mjs", import.meta.url), + ), + ], + { + env: { + ...process.env, + CI_JOBS: JSON.stringify(needs), + CI_QUALIFICATION_DIRECTORY: "", + }, + }, + ); + if (result === "success") + await expect(execution).resolves.toMatchObject({ stderr: "" }); + else + await expect(execution).rejects.toMatchObject({ + code: 1, + stderr: expect.stringContaining("reverse-tools-quality"), + }); + } + }, +); + +it("retains available quality diagnostics after failure without keeping canceled uploads alive", async () => { + const steps = (await workflow())["reverse-tools-quality"]?.steps ?? []; + const upload = steps.find((step) => + step.uses?.startsWith("actions/upload-artifact@"), + ); + const condition = z.string().parse(upload?.if); + for (const cancelled of [false, true]) + expect( + evaluateCondition(condition, { + cancelled: () => cancelled, + success: () => false, + failure: () => true, + always: () => true, + }), + ).toBe(!cancelled); + const options = upload?.with ?? {}; + expect(z.string().parse(options.name)).toContain( + "${{ needs.changes.outputs.tier }}", + ); + expect(z.string().parse(options.name)).toContain("${{ github.run_attempt }}"); + expect( + z + .string() + .parse(options.path) + .trim() + .split("\n") + .map((path) => path.trim()), + ).toEqual( + expect.arrayContaining(["coverage/reverse-tools/", "reports/mutation/"]), + ); + expect(options["retention-days"]).toBe(7); + expect(options["if-no-files-found"]).toBe("warn"); +}); + +it("starts source feedback before build and requires both groups without waiving missing jobs", async () => { + const jobs = await workflow(); + expect( + jobs.static?.steps?.find( + (step) => step.run === "npm run ci:static-context", + )?.["continue-on-error"], + ).toBe(true); + expect(jobs["test-source"]?.needs).toBe("changes"); + expect(jobs["test-shard"]?.needs).toEqual(["changes", "build"]); + expect( + jobs["test-source"]?.steps?.some( + (step) => + step.run === "npm run test:fast -- --reporter=default --reporter=blob", + ), + ).toBe(true); + const required = fileURLToPath( + new URL("../../../scripts/ci/required.mjs", import.meta.url), + ); + const flags = await scopeFlags(); + for (const tier of ["fast", "full"]) { + for (const result of [ + "success", + "failure", + "cancelled", + "skipped", + "missing", + ]) { + const needs = Object.fromEntries( + Object.keys(jobs) + .filter((name) => name !== "required") + .map((name) => [name, { result: "success" }]), + ); + const outputs = { + ...Object.fromEntries( + flags.map((flag) => [flag, tier === "full" ? "true" : "false"]), + ), + tier, + code: "true", + }; + const context: Record< + string, + { result: string; outputs?: Record } + > = { ...needs, changes: { result: "success", outputs } }; + if (result === "missing") delete context["test-source"]; + else context["test-source"] = { result }; + const run = execute(process.execPath, [required], { + env: { + ...process.env, + CI_JOBS: JSON.stringify(context), + CI_QUALIFICATION_DIRECTORY: "", + }, + }); + if ( + result === "success" || + (tier === "full" && ["skipped", "missing"].includes(result)) + ) + await expect(run).resolves.toMatchObject({ stderr: "" }); + else + await expect(run).rejects.toMatchObject({ + code: 1, + stderr: expect.stringContaining("test-source"), + }); + } + } +}); + +it("records the actual static task graph with explicit source identity and no ambient environment", async () => { + const directory = await createTestTempDirectory("rea-ci-context-"); + const context = await recordStaticTaskContext({ outputDirectory: directory }); + const head = await execute("git", ["rev-parse", "HEAD"]); + expect(context.sourceSha).toBe(head.stdout.trim()); + expect(context.taskHashes["lint:built"]).toMatch(/^[a-f0-9]{16}$/u); + expect(context.taskHashes.jscpd).toMatch(/^[a-f0-9]{16}$/u); + expect(context).not.toHaveProperty("environmentVariables"); + expect(context).not.toHaveProperty("env"); + expect( + JSON.parse(await readFile(`${directory}/static-task-context.json`, "utf8")), + ).toEqual(context); + for (const invalid of [ + undefined, + {}, + { ...context, sourceSha: "bad" }, + { ...context, prHeadSha: "bad" }, + { ...context, taskHashes: { jscpd: "stale" } }, + ]) + expect(() => parseStaticTaskContext(invalid)).toThrow(); +}); + +it("rejects mutation methods at the shared GitHub transport before invoking the CLI", async () => { + const request = createGhApiRequest({ env: { PATH: "" } }); + await expect( + request("POST", "repos/owner/repo/actions/runs/1/rerun", { + requests: 0, + startedAt: Date.now(), + limitMs: 1000, + }), + ).rejects.toThrow("read-only GET"); +}); + +it("collects basic CI telemetry before optional dependency installation can fail", async () => { + const { jobs } = z + .object({ + jobs: z.object({ + report: z.object({ + steps: z.array( + z.object({ + id: z.string().optional(), + run: z.string().optional(), + if: z.string().optional(), + "continue-on-error": z.boolean().optional(), + }), + ), + }), + }), + }) + .parse( + parse( + await readFile( + new URL("../../../.github/workflows/ci-timings.yml", import.meta.url), + "utf8", + ), + ), + ); + const steps = jobs.report.steps; + const installation = steps.findIndex((step) => step.id === "failure-deps"); + for (const script of ["pr-ci-health.mjs", "job-timings-batch.mjs"]) { + const collection = steps.findIndex( + (step) => step.run === `node scripts/ci/${script}`, + ); + expect(collection).toBeGreaterThanOrEqual(0); + expect(installation).toBeGreaterThan(collection); + } + expect(steps[installation]).toMatchObject({ + run: "npm ci", + "continue-on-error": true, + }); + expect( + steps.find((step) => step.id === "npm-cache")?.["continue-on-error"], + ).toBe(true); + expect( + steps.find((step) => step.run === "node scripts/ci/failure-report.mjs")?.if, + ).toContain("steps.failure-deps.outcome == 'success'"); +}); + +it("merges source and built reports exactly once using the workflow's shard matrix", async () => { + const jobs = await workflow(); + const matrix = z + .object({ + matrix: z.object({ shard: z.string().regex(/^\$\{\{.*\}\}$/u) }), + }) + .parse(jobs["test-shard"]?.strategy).matrix.shard; + const shards = z + .array(z.string().regex(/^\d+\/\d+$/u)) + .min(1) + .parse( + runInNewContext( + matrix.slice(3, -2), + { + fromJSON: JSON.parse, + needs: { changes: { outputs: { tier: "fast" } } }, + }, + { timeout: 1_000 }, + ), + ); + const labels = Object.fromEntries( + ["source", "built"].map((group) => [ + group, + z + .string() + .min(1) + .parse( + jobs[group === "source" ? "test-source" : "test-shard"]?.steps?.find( + (step) => step.env?.VITEST_BLOB_LABEL, + )?.env?.VITEST_BLOB_LABEL, + ), + ]), + ); + const root = await createTestTempDirectory("rea-ci-group-merge-"); + const config = join(root, "vitest.config.mjs"); + await writeFile( + config, + `export default { test: { maxWorkers: 1, projects: [${["source", "built"].map((name) => `{ test: { name: "${name}", globals: true, include: ["${name}*.test.js"] } }`).join(",")}] } };`, + ); + const files = [ + "source.test.js", + ...Array.from( + { length: shards.length }, + (_, index) => `built-${index}.test.js`, + ), + ]; + for (const file of files) + await writeFile( + join(root, file), + 'it("runs its selected group", () => expect(true).toBe(true));', + ); + const vitest = fileURLToPath( + new URL("../../../node_modules/vitest/vitest.mjs", import.meta.url), + ); + const blobs = new Map(); + const runs = [ + { group: "source", shard: [] }, + ...shards.map((shard) => ({ group: "built", shard: [`--shard=${shard}`] })), + ]; + for (const { group, shard } of runs) { + await execute( + process.execPath, + [ + vitest, + "run", + "--config", + config, + "--project", + group, + "--reporter=blob", + ...shard, + ], + { + cwd: root, + env: { ...process.env, VITEST_BLOB_LABEL: labels[group]! }, + timeout: 20_000, + }, + ); + for (const filename of await readdir(join(root, ".vitest/blob"))) { + expect(blobs.has(filename)).toBe(false); + blobs.set(filename, await readFile(join(root, ".vitest/blob", filename))); + } + // Separate hosted jobs start with independent report directories. + await rm(join(root, ".vitest/blob"), { recursive: true }); + } + expect(blobs.size).toBe(runs.length); + await mkdir(join(root, ".vitest/blob"), { recursive: true }); + for (const [filename, bytes] of blobs) + await writeFile(join(root, ".vitest/blob", filename), bytes); + await execute( + process.execPath, + [ + vitest, + "--config", + config, + "--merge-reports", + "--reporter=json", + "--outputFile.json=merged.json", + ], + { cwd: root, timeout: 20_000 }, + ); + const merged = z + .object({ + success: z.literal(true), + testResults: z.array(z.object({ name: z.string() })), + }) + .parse(JSON.parse(await readFile(join(root, "merged.json"), "utf8"))); + expect( + merged.testResults.map(({ name }) => name.split("/").at(-1)).sort(), + ).toEqual(files.toSorted()); +}); diff --git a/tests/boundary/filesystem/ciJobTimingsBatch.test.ts b/tests/boundary/filesystem/ciJobTimingsBatch.test.ts index 6c2658c000..e0dcecc824 100644 --- a/tests/boundary/filesystem/ciJobTimingsBatch.test.ts +++ b/tests/boundary/filesystem/ciJobTimingsBatch.test.ts @@ -18,6 +18,8 @@ it("collects the latest attempt for recent completed CI and Release runs into a workflow_runs: [ { id: 42, + event: "push", + head_branch: "main", run_attempt: 3, status: "completed", conclusion: "failure", @@ -43,51 +45,9 @@ it("collects the latest attempt for recent completed CI and Release runs into a }, ], }, - [ - { - jobs: [ - { - name: "Release lane", - status: "completed", - conclusion: "success", - created_at: recent, - started_at: recent, - completed_at: new Date(Date.parse(recent) + 50_000).toISOString(), - steps: [], - }, - ], - }, - ], - [ - { - jobs: [ - { - name: "Failed lane", - status: "completed", - conclusion: "failure", - created_at: recent, - started_at: recent, - completed_at: new Date(Date.parse(recent) + 55_000).toISOString(), - steps: [], - }, - ], - }, - ], - [ - { - jobs: [ - { - name: "Cancelled lane", - status: "completed", - conclusion: "cancelled", - created_at: recent, - started_at: recent, - completed_at: new Date(Date.parse(recent) + 30_000).toISOString(), - steps: [], - }, - ], - }, - ], + completedJobs("Release lane", "success", recent, 50), + completedJobs("Failed lane", "failure", recent, 55), + completedJobs("Cancelled lane", "cancelled", recent, 30), ]); const result = await runBatch(fixture); const manifest = JSON.parse( @@ -116,8 +76,20 @@ it("collects the latest attempt for recent completed CI and Release runs into a expect(manifest.errors).toEqual([]); expect(manifest.runs).toMatchObject([ { id: 43, runAttempt: 2, conclusion: "success" }, - { id: 42, runAttempt: 3, conclusion: "failure" }, - { id: 41, runAttempt: 1, conclusion: "cancelled" }, + { + id: 42, + runAttempt: 3, + conclusion: "failure", + event: "push", + headBranch: "main", + }, + { + id: 41, + runAttempt: 1, + conclusion: "cancelled", + event: null, + headBranch: null, + }, ]); expect(manifest.runs).toEqual( expect.arrayContaining([ @@ -234,6 +206,31 @@ it("retains a partial manifest when its collection budget is exhausted", async ( expect(manifest.truncation.join(" ")).toContain("Time budget"); }); +function completedJobs( + name: string, + conclusion: string, + startedAt: string, + seconds: number, +) { + return [ + { + jobs: [ + { + name, + status: "completed", + conclusion, + created_at: startedAt, + started_at: startedAt, + completed_at: new Date( + Date.parse(startedAt) + seconds * 1000, + ).toISOString(), + steps: [], + }, + ], + }, + ]; +} + async function createFixture(responses: unknown[]) { const directory = await createTestTempDirectory("rea-ci-job-timings-batch-"); const fakeBin = join(directory, "bin"); diff --git a/tests/boundary/filesystem/ciPrHealth.test.ts b/tests/boundary/filesystem/ciPrHealth.test.ts new file mode 100644 index 0000000000..756fb47489 --- /dev/null +++ b/tests/boundary/filesystem/ciPrHealth.test.ts @@ -0,0 +1,617 @@ +import { chmod, mkdir, readFile, writeFile } from "node:fs/promises"; +import { delimiter, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +import { expect, it } from "vitest"; + +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { + classifyPrCi, + collectPrCiHealth, + matchRunToPullRequest, + writePrCiHealthReport, +} from "../../../scripts/ci/pr-ci-health.mjs"; +import { createGhApiRequest } from "../../../scripts/ci/github-api.mjs"; +import type { + PrCiPullRequest, + PrCiWorkflowRun, +} from "../../../scripts/ci/pr-ci-health.mjs"; +import { executeWorkflowFixture } from "../../support/workflowGit.js"; + +const reporter = fileURLToPath( + new URL("../../../scripts/ci/pr-ci-health.mjs", import.meta.url), +); +const observedAt = Date.parse("2026-10-14T00:00:00Z"); +const headSha = "a".repeat(40); +const mergeSha = "b".repeat(40); + +it("classifies workflow states without confusing absent, pending, and completed CI", () => { + const pull = validPull(); + const cases = [ + [ + "startup_failure", + run({ status: "completed", conclusion: "startup_failure" }), + ], + [ + "queued_pending_executing", + run({ status: "in_progress", conclusion: null }), + ], + ["approval_required", run({ status: "waiting", conclusion: null })], + ["completed_failure", run({ status: "completed", conclusion: "failure" })], + [ + "completed_canceled", + run({ status: "completed", conclusion: "cancelled" }), + ], + ["completed_success", run({ status: "completed", conclusion: "success" })], + ["completed_other", run({ status: "completed", conclusion: "neutral" })], + [ + "completed_other", + run({ status: "completed", conclusion: "constructor" }), + ], + ] as const; + for (const [category, evidence] of cases) { + for (const runsComplete of [true, false]) { + const result = classifyPrCi({ + pull, + runs: [ + { + run: { ...evidence, run_attempt: 1 }, + basis: "event_pr_number_and_head_sha", + }, + ], + runsComplete, + pullRequestInventoryComplete: true, + ambiguousRuns: [], + }); + expect(result).toMatchObject({ + category: runsComplete ? category : "unknown", + ...(runsComplete + ? {} + : { reason: "workflow_run_inventory_incomplete" }), + runs: [ + { + id: 77, + attempt: 1, + status: evidence.status, + conclusion: evidence.conclusion, + }, + ], + }); + } + } + for (const runsComplete of [true, false]) { + expect( + classifyPrCi({ + pull: { ...pull, mergeable: false }, + runs: [], + runsComplete, + pullRequestInventoryComplete: true, + ambiguousRuns: [], + }).category, + ).toBe("merge_conflict"); + } +}); + +it("requires complete inventories, known mergeability, and immutable head identity to report missing", () => { + const pull = validPull({ mergeable: true }); + const classify = (overrides = {}) => + classifyPrCi({ + pull, + runs: [], + runsComplete: true, + pullRequestInventoryComplete: true, + ambiguousRuns: [], + ...overrides, + }); + + expect(classify().category).toBe("missing"); + expect(classify({ runsComplete: false })).toMatchObject({ + category: "deferred", + reason: "lookup_inventory_incomplete", + }); + expect(classify({ pullRequestInventoryComplete: false }).category).toBe( + "deferred", + ); + expect( + classify({ + ambiguousRuns: [{ run: run(), basis: "ambiguous_head_identity" }], + }).category, + ).toBe("unknown"); + expect(classify({ pull: { ...pull, mergeable: undefined } }).category).toBe( + "unknown", + ); + expect(classify({ pull: { ...pull, identityValid: false } }).category).toBe( + "unknown", + ); +}); + +it("accepts event PR/head evidence for merge-SHA runs and refuses ambiguous branch identities", () => { + const pull = validPull(); + const associatedMergeRun = run({ + head_sha: mergeSha, + pull_requests: [ + { + number: pull.number!, + head: { sha: headSha, repo: { id: pull.head.repositoryId } }, + }, + ], + }); + expect( + matchRunToPullRequest({ + run: associatedMergeRun, + pull, + identityCounts: new Map([["73\u0000feature", 1]]), + }), + ).toEqual({ kind: "match", basis: "event_pr_number_and_head_sha" }); + expect( + matchRunToPullRequest({ + run: run({ + pull_requests: [ + { number: pull.number!, head: { sha: "c".repeat(40) } }, + ], + }), + pull, + identityCounts: new Map([["73\u0000feature", 1]]), + }).kind, + ).toBe("none"); + + const branchRun = run({ pull_requests: [] }); + expect( + matchRunToPullRequest({ + run: branchRun, + pull, + identityCounts: new Map([["73\u0000feature", 1]]), + inventoryComplete: false, + }).kind, + ).toBe("ambiguous"); + expect( + matchRunToPullRequest({ + run: branchRun, + pull, + identityCounts: new Map([["73\u0000feature", 1]]), + }), + ).toEqual({ + kind: "match", + basis: "event_unique_head_ref_and_repository_id", + }); + expect( + matchRunToPullRequest({ + run: run({ head_sha: mergeSha, pull_requests: [] }), + pull, + identityCounts: new Map([["73\u0000feature", 1]]), + }).kind, + ).toBe("none"); + expect( + matchRunToPullRequest({ + run: run({ + pull_requests: [{ number: pull.number!, head: { sha: headSha } }], + }), + pull, + identityCounts: new Map([["73\u0000feature", 1]]), + }).kind, + ).toBe("ambiguous"); + expect( + matchRunToPullRequest({ + run: branchRun, + pull, + identityCounts: new Map([["73\u0000feature", 2]]), + }).kind, + ).toBe("ambiguous"); + expect( + matchRunToPullRequest({ + run: run({ event: "workflow_dispatch" }), + pull, + identityCounts: new Map([["73\u0000feature", 1]]), + }).kind, + ).toBe("none"); +}); + +it("scans target-branch PRs through public gh GET requests and writes JSON and Markdown", async () => { + const fixture = await createFixture(); + const calls: string[] = []; + const apiRequest: NonNullable< + Parameters[0]["apiRequest"] + > = async (method, endpoint) => { + calls.push(`${method} ${endpoint}`); + if (method !== "GET") throw new Error("fixture rejected a non-GET request"); + if (endpoint.includes("/pulls?")) + return [ + rawPull(42, { mergeable: true }), + rawPull(43, { mergeable: false }), + rawPull(44, { + base: { ref: "release", repo: { id: 11 } }, + mergeable: true, + }), + rawPull(45, { draft: true, mergeable: true }), + ]; + if (endpoint.includes("/actions/workflows/ci.yml/runs?")) + return { + total_count: 1, + workflow_runs: [ + run({ + id: 9001, + event: "pull_request", + status: "completed", + conclusion: "failure", + head_sha: mergeSha, + pull_requests: [ + { + number: 42, + head: { sha: headSha, repo: { id: 73 } }, + }, + ], + }), + ], + }; + throw new Error(`unexpected GET endpoint: ${endpoint}`); + }; + const report = await collectPrCiHealth({ + repository: "owner/rea", + repositoryId: 11, + defaultBranch: "main", + now: observedAt, + apiRequest, + }); + expect(calls).toHaveLength(2); + expect(calls.every((call) => call.startsWith("GET "))).toBe(true); + expect(calls[0]).toContain("state=open"); + expect(calls[0]).not.toContain("base="); + expect(calls[1]).toContain("event=pull_request"); + expect(report.pullRequests).toHaveLength(2); + expect(report.pullRequests.map(({ number }) => number)).toEqual([42, 43]); + expect(report.pullRequests.map(({ ci }) => ci.category)).toEqual([ + "completed_failure", + "merge_conflict", + ]); + expect(report.scan).toMatchObject({ + status: "complete", + pullRequestInventoryComplete: true, + workflowRunInventoryComplete: true, + apiRequests: 2, + }); + + const output = join(fixture.directory, ".cache", "ci-pr-health"); + const paths = await writePrCiHealthReport(report, output); + const written = JSON.parse(await readFile(paths.jsonPath, "utf8")); + const markdown = await readFile(paths.markdownPath, "utf8"); + expect(written.counts.completed_failure).toBe(1); + expect(markdown).toContain("# Pull request CI health snapshot"); + expect(markdown).toContain("#42"); + expect(markdown).toContain("completed_failure"); +}); + +it("counts older open PRs when deciding whether a branch/ref fallback is unique", async () => { + const olderPull = rawPull(63, { + created_at: "2026-09-01T00:00:00Z", + head: { ...rawPull(63).head, ref: "shared-feature" }, + }); + const currentPull = rawPull(64, { + mergeable: true, + head: { ...olderPull.head }, + }); + const apiRequest: NonNullable< + Parameters[0]["apiRequest"] + > = async (_method, endpoint) => { + if (endpoint.includes("/pulls?")) return [currentPull, olderPull]; + if (endpoint.includes("/actions/workflows/ci.yml/runs?")) + return { + workflow_runs: [ + run({ + head_branch: "shared-feature", + pull_requests: [], + }), + ], + }; + throw new Error(`unexpected GET endpoint: ${endpoint}`); + }; + const report = await collectPrCiHealth({ + repository: "owner/rea", + repositoryId: 11, + defaultBranch: "main", + now: observedAt, + apiRequest, + }); + expect(report.scan.pullRequestInventoryComplete).toBe(true); + expect(report.pullRequests.map(({ number }) => number)).toEqual([64]); + expect(report.pullRequests[0]?.ci).toMatchObject({ + category: "unknown", + reason: "workflow_run_identity_ambiguous", + }); +}); + +it.each([false, true])( + "keeps page-capped scans uncertain (observed success: %s)", + async (observedSuccess) => { + const pull = rawPull(62, { mergeable: true }); + const apiRequest: NonNullable< + Parameters[0]["apiRequest"] + > = async (_method, endpoint) => { + if (endpoint.includes("/pulls?")) return [pull]; + const page = Number( + new URL(`https://fixture.invalid/${endpoint}`).searchParams.get("page"), + ); + return { + workflow_runs: Array.from({ length: 100 }, (_item, index) => + observedSuccess && page === 1 && index === 0 + ? run({ + run_attempt: 1, + pull_requests: [ + { number: 62, head: { sha: headSha, repo: { id: 73 } } }, + ], + }) + : run({ + id: page * 100 + index, + head_branch: "different-branch", + head_repository: { id: 999 }, + pull_requests: [], + }), + ), + }; + }; + const report = await collectPrCiHealth({ + repository: "owner/rea", + repositoryId: 11, + defaultBranch: "main", + now: observedAt, + apiRequest, + }); + expect(report.scan.workflowRunPages).toBe(5); + expect(report.scan.workflowRunInventoryComplete).toBe(false); + const firstPull = report.pullRequests[0]; + if (firstPull === undefined) throw new Error("fixture PR was not scanned"); + expect(firstPull.ci).toMatchObject({ + category: observedSuccess ? "unknown" : "deferred", + reason: observedSuccess + ? "workflow_run_inventory_incomplete" + : "lookup_inventory_incomplete", + }); + expect(firstPull.ci.runs).toHaveLength(observedSuccess ? 1 : 0); + }, +); + +it("keeps the result unknown when the head changes between API snapshots", async () => { + const apiRequest: NonNullable< + Parameters[0]["apiRequest"] + > = async (_method, endpoint) => { + if (endpoint.includes("/pulls?")) return [rawPull(63)]; + if (endpoint.includes("/actions/workflows/ci.yml/runs?")) + return { workflow_runs: [] }; + return { + ...rawPull(63, { mergeable: true }), + head: { + ...rawPull(63).head, + sha: mergeSha, + }, + }; + }; + const report = await collectPrCiHealth({ + repository: "owner/rea", + repositoryId: 11, + defaultBranch: "main", + now: observedAt, + apiRequest, + }); + expect(report.pullRequests[0]).toMatchObject({ + identityChangedDuringScan: true, + ci: { category: "unknown", reason: "pull_request_head_identity_invalid" }, + }); + expect(report.scan).toMatchObject({ + status: "partial", + errors: ["pull_request_identity_changed_during_scan"], + }); +}); + +it("public CLI uses only GET and produces a report from fixture GitHub responses", async () => { + const fixture = await createFixture(); + const ghCalls = join(fixture.directory, "gh-calls.jsonl"); + const ghResponse = join(fixture.directory, "gh-response.json"); + const fakeGh = join(fixture.fakeBin, "gh"); + await writeFile( + ghResponse, + JSON.stringify([ + [rawPull(51, { mergeable: true })], + { total_count: 0, workflow_runs: [] }, + ]), + ); + await writeFile( + fakeGh, + `#!/usr/bin/env node\nconst fs=require("node:fs");\nconst args=process.argv.slice(2);\nfs.appendFileSync(process.env.CI_GH_CALLS, JSON.stringify(args)+"\\n");\nif(args[0]!=="api"||args[1]!=="--method"||args[2]!=="GET")process.exit(41);\nconst page=Number(new URL("https://fixture.invalid/"+args.at(-1)).searchParams.get("page"));\nconst responses=JSON.parse(fs.readFileSync(process.env.CI_GH_RESPONSE,"utf8"));\nprocess.stdout.write(JSON.stringify(responses[page===1&&args.at(-1).includes("/pulls?")?0:1]));\n`, + ); + await chmod(fakeGh, 0o755); + const outputDirectory = join(fixture.directory, ".cache", "ci-pr-health"); + await executeWorkflowFixture(process.execPath, [reporter], { + cwd: fixture.directory, + env: { + ...process.env, + PATH: `${fixture.fakeBin}${delimiter}${process.env.PATH ?? ""}`, + CI_REPOSITORY: "owner/rea", + CI_REPOSITORY_ID: "11", + CI_DEFAULT_BRANCH: "main", + CI_PR_HEALTH_DIRECTORY: outputDirectory, + CI_GH_CALLS: ghCalls, + CI_GH_RESPONSE: ghResponse, + }, + }); + const calls = (await readFile(ghCalls, "utf8")) + .trim() + .split("\n") + .map((line) => JSON.parse(line)); + expect(calls).toHaveLength(2); + expect( + calls.every( + (args: string[]) => args[1] === "--method" && args[2] === "GET", + ), + ).toBe(true); + const report = JSON.parse( + await readFile(join(outputDirectory, "report.json"), "utf8"), + ); + expect(report.pullRequests[0].ci.category).toBe("missing"); + await expect( + readFile(join(outputDirectory, "summary.md"), "utf8"), + ).resolves.toContain("missing"); +}); + +it("rejects non-GET methods before launching the GitHub client", async () => { + const fixture = await createFixture(); + const request = createGhApiRequest({ + env: { + ...process.env, + PATH: `${fixture.fakeBin}${delimiter}${process.env.PATH ?? ""}`, + CI_GH_CALLS: join(fixture.directory, "should-not-exist"), + }, + }); + await expect( + request("POST", "repos/owner/rea/pulls", { + requests: 0, + startedAt: observedAt, + limitMs: 1_000, + }), + ).rejects.toThrow("read-only GET"); + await expect( + readFile(join(fixture.directory, "should-not-exist")), + ).rejects.toMatchObject({ code: "ENOENT" }); +}); + +function validPull(overrides: Partial = {}): PrCiPullRequest { + return { + number: 42, + identityValid: true, + mergeable: true, + head: { repositoryId: 73, ref: "feature", sha: headSha }, + ...overrides, + }; +} + +it.each([ + ["newer run", { id: 78, created_at: "2026-10-13T01:00:00Z" }, "unknown"], + [ + "older run", + { id: 78, created_at: "2026-10-12T23:00:00Z" }, + "completed_success", + ], + ["equal timestamps", { id: 78 }, "unknown"], + [ + "retry of an older run", + { id: 78, created_at: "2026-10-12T23:00:00Z", run_attempt: 2 }, + "unknown", + ], + ["unknown timestamp", { id: 78, created_at: "invalid" }, "unknown"], + ["newer attempt", { run_attempt: 3 }, "unknown"], + ["older attempt", { run_attempt: 1 }, "completed_success"], +] as const)( + "retains ambiguity evidence for a %s without reporting a stale completion", + (_label, overrides, category) => { + const currentAttempt = "id" in overrides ? 1 : 2; + const result = classifyPrCi({ + pull: validPull(), + runs: [ + { + run: run({ run_attempt: currentAttempt }), + basis: "event_pr_number_and_head_sha", + }, + ], + ambiguousRuns: [ + { + run: run({ run_attempt: currentAttempt, ...overrides }), + basis: "ambiguous_head_identity", + }, + ], + runsComplete: true, + pullRequestInventoryComplete: true, + }); + expect(result.category).toBe(category); + expect(result.runs).toHaveLength(2); + }, +); + +it("does not let an older proven run hide a newer run with an unverified repository ID", async () => { + const pull = rawPull(42); + const older = run({ + id: 70, + created_at: "2026-10-13T00:00:00Z", + pull_requests: [{ number: 42, head: { sha: headSha, repo: { id: 73 } } }], + }); + const newer = run({ + id: 71, + created_at: "2026-10-13T01:00:00Z", + status: "in_progress", + conclusion: null, + pull_requests: [{ number: 42, head: { sha: headSha } }], + }); + const report = await collectPrCiHealth({ + repository: "owner/rea", + repositoryId: 11, + defaultBranch: "main", + now: observedAt, + apiRequest: async (_method, endpoint) => { + if (endpoint.includes("/pulls?")) return [pull]; + if (endpoint.endsWith("/pulls/42")) return { ...pull, mergeable: true }; + return { workflow_runs: [newer, older] }; + }, + }); + expect(report.pullRequests[0]?.ci).toMatchObject({ + category: "unknown", + reason: "workflow_run_identity_ambiguous", + }); + expect(report.pullRequests[0]?.ci.runs).toHaveLength(2); +}); + +it("keeps a proven newest run even when two older matched runs have equal timestamps", () => { + const result = classifyPrCi({ + pull: validPull(), + runs: [ + run({ id: 70, run_attempt: 1 }), + run({ id: 71, run_attempt: 1, conclusion: "failure" }), + run({ + id: 72, + run_attempt: 1, + created_at: "2026-10-13T01:00:00Z", + status: "in_progress", + conclusion: null, + }), + ].map((run) => ({ run, basis: "event_pr_number_and_head_sha" })), + ambiguousRuns: [], + runsComplete: true, + pullRequestInventoryComplete: true, + }); + expect(result.category).toBe("queued_pending_executing"); + expect(result.runs).toHaveLength(3); +}); + +function run(overrides: Partial = {}): PrCiWorkflowRun { + return { + id: 77, + event: "pull_request", + head_branch: "feature", + head_repository: { id: 73 }, + head_sha: headSha, + status: "completed", + conclusion: "success", + created_at: "2026-10-13T00:00:00Z", + pull_requests: [], + ...overrides, + }; +} + +function rawPull(number: number, overrides: Record = {}) { + return { + number, + draft: false, + created_at: "2026-10-13T00:00:00Z", + html_url: `https://github.com/owner/rea/pull/${String(number)}`, + base: { ref: "main", repo: { id: 11 } }, + head: { + ref: `feature-${String(number)}`, + sha: headSha, + repo: { id: 73, full_name: "contributor/rea" }, + }, + ...overrides, + }; +} + +async function createFixture() { + const directory = await createTestTempDirectory("rea-ci-pr-health-"); + const fakeBin = join(directory, "bin"); + await mkdir(fakeBin); + return { directory, fakeBin }; +} diff --git a/tests/boundary/filesystem/ciQualification.test.ts b/tests/boundary/filesystem/ciQualification.test.ts index b6a5a1e1a4..673b09d34c 100644 --- a/tests/boundary/filesystem/ciQualification.test.ts +++ b/tests/boundary/filesystem/ciQualification.test.ts @@ -44,7 +44,7 @@ function run(overrides: Record = {}) { id: 7, status: "completed", conclusion: "success", - path: ".github/workflows/ci.yml", + path: ".github/workflows/ci.yml@main", event: "schedule", created_at: new Date(now - 120_000).toISOString(), head_sha: validSha, diff --git a/tests/boundary/filesystem/ciRequiredLanes.test.ts b/tests/boundary/filesystem/ciRequiredLanes.test.ts index 4c23844e76..d2191eb01c 100644 --- a/tests/boundary/filesystem/ciRequiredLanes.test.ts +++ b/tests/boundary/filesystem/ciRequiredLanes.test.ts @@ -121,7 +121,11 @@ it("exports every canonical scope with read authority for admission and keeps te ); expect(timings.on.workflow_run).toBeUndefined(); expect(timings.on.schedule).toHaveLength(1); - expect(timings.permissions).toEqual({ contents: "read", actions: "read" }); + expect(timings.permissions).toEqual({ + contents: "read", + actions: "read", + "pull-requests": "read", + }); const checkout = timings.jobs.report.steps.find((step) => step.uses?.startsWith("actions/checkout@"), ); @@ -154,6 +158,7 @@ it("holds native and provider fan-out behind static validation while portable fe "static", "build", "test-shard", + "test-source", "test", "docs", "required", @@ -369,7 +374,7 @@ it("keeps fast feedback separate from complete qualification and release proof", // release their concurrency slot without waiting for another runner. expect(ci.jobs.required?.if).toBe("${{ !cancelled() }}"); const shards = ci.jobs["test-shard"]?.steps ?? []; - const fast = shards.find((step) => step.run?.includes("npm run test:run")); + const fast = shards.find((step) => step.run?.includes("npm run test:built")); const full = shards.find((step) => step.run?.includes("test:ci:shard:run")); expect(fast?.if).toBe("needs.changes.outputs.tier == 'fast'"); expect(fast?.run).toContain("--reporter=blob"); @@ -506,3 +511,44 @@ it("rejects incomplete or malformed classification even when available jobs succ }); } }); + +it("fails fast only for ordinary PR matrices and rejects partial portable feedback", async () => { + const { jobs } = await workflow(); + expect(jobs["test-shard"]?.strategy).toMatchObject({ + "fail-fast": + "${{ github.event_name == 'pull_request' && needs.changes.outputs.tier == 'fast' }}", + }); + expect(jobs["package-e2e"]?.strategy).toMatchObject({ "fail-fast": false }); + const flags = await scopeFlags(); + for (const result of ["failure", "cancelled", "skipped", "missing"]) { + // Even an incorrectly successful report merge cannot hide matrix failure. + const needs: Record = { + changes: { + result: "success", + outputs: { + ...Object.fromEntries(flags.map((flag) => [flag, "false"])), + tier: "fast", + code: "true", + }, + }, + static: { result: "success" }, + build: { result: "success" }, + test: { result: "success" }, + ...(result === "missing" ? {} : { "test-shard": { result } }), + }; + await expect( + executeWorkflowFixture( + process.execPath, + [ + fileURLToPath( + new URL("../../../scripts/ci/required.mjs", import.meta.url), + ), + ], + { env: { ...process.env, CI_JOBS: JSON.stringify(needs) } }, + ), + ).rejects.toMatchObject({ + code: 1, + stderr: expect.stringContaining("test-shard"), + }); + } +}); diff --git a/tests/boundary/filesystem/ciStaticArtifacts.test.ts b/tests/boundary/filesystem/ciStaticArtifacts.test.ts new file mode 100644 index 0000000000..45915f4504 --- /dev/null +++ b/tests/boundary/filesystem/ciStaticArtifacts.test.ts @@ -0,0 +1,99 @@ +import { execFile } from "node:child_process"; +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { promisify } from "node:util"; +import { expect, it } from "vitest"; + +import { parseStaticArchive } from "../../../scripts/ci/artifacts.mjs"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +const execute = promisify(execFile); +const producer = fileURLToPath( + new URL("../../../scripts/ci/static-check-report.mjs", import.meta.url), +); + +it.each([3.5, 100, 0])( + "retains real lint failures when the duplication threshold is %s", + async (threshold) => { + const root = await createTestTempDirectory("rea-ci-static-archive-"); + await mkdir(join(root, "src")); + await mkdir(join(root, "tests")); + await writeFile( + join(root, ".oxlintrc.json"), + JSON.stringify({ rules: { "no-debugger": "error" } }), + ); + await writeFile( + join(root, ".jscpd.json"), + JSON.stringify({ threshold, minLines: 5, minTokens: 50 }), + ); + await writeFile( + join(root, "src/example.ts"), + "debugger;\nexport const example = true;\n", + ); + const duplicate = + "export function compute(value: number) {\n" + + Array.from({ length: 20 }, (_, index) => ` value += ${index};\n`).join( + "", + ) + + " return value;\n}\n"; + for (const name of threshold === 3.5 ? [] : ["first", "second"]) + await writeFile(join(root, `src/${name}.ts`), duplicate); + await expect( + execute(process.execPath, [producer, "lint", "src/example.ts"], { + cwd: root, + }), + ).rejects.toMatchObject({ code: 1 }); + const duplication = execute(process.execPath, [producer, "duplicates"], { + cwd: root, + }); + if (threshold === 0) + await expect(duplication).rejects.toMatchObject({ code: 1 }); + else await duplication; + const reports = join(root, ".cache/static-checks"); + expect( + JSON.parse( + await readFile(join(reports, "jscpd/jscpd-report.json"), "utf8"), + ), + ).toMatchObject({ reaCheck: { exitCode: threshold === 0 ? 1 : 0 } }); + await writeFile( + join(reports, "static-task-context.json"), + JSON.stringify({ + version: 1, + sourceSha: "a".repeat(40), + sourceClean: true, + nodeVersion: process.version, + npmVersion: "11.16.0", + turboVersion: "2.11.7", + platform: process.platform, + architecture: process.arch, + taskHashes: { "lint:built": "a".repeat(16), jscpd: "b".repeat(16) }, + }), + ); + const archive = join(root, "reports.zip"); + await execute( + "zip", + [ + "-q", + archive, + "oxlint.json", + "jscpd/jscpd-report.json", + "static-task-context.json", + ], + { cwd: reports }, + ); + const evidence = await parseStaticArchive(archive); + expect(evidence.diagnostics.filter(({ kind }) => kind === "lint")).toEqual([ + expect.objectContaining({ + kind: "lint", + path: "src/example.ts", + severity: "error", + }), + ]); + const duplicates = evidence.diagnostics.filter( + ({ kind }) => kind === "jscpd", + ); + if (threshold === 0) expect(duplicates.length).toBeGreaterThan(0); + else expect(duplicates).toEqual([]); + }, +); diff --git a/tests/boundary/filesystem/ciTestHistory.test.ts b/tests/boundary/filesystem/ciTestHistory.test.ts index 411e9ece9d..84950d49c5 100644 --- a/tests/boundary/filesystem/ciTestHistory.test.ts +++ b/tests/boundary/filesystem/ciTestHistory.test.ts @@ -35,7 +35,7 @@ function run(id: number, overrides: Record = {}) { id, status: "completed", conclusion: "success", - path: ".github/workflows/ci.yml", + path: ".github/workflows/ci.yml@main", event: "schedule", created_at: new Date(now - 120_000).toISOString(), head_branch: "main", diff --git a/tests/boundary/filesystem/ciTestProjects.test.ts b/tests/boundary/filesystem/ciTestProjects.test.ts new file mode 100644 index 0000000000..f0394c5bbf --- /dev/null +++ b/tests/boundary/filesystem/ciTestProjects.test.ts @@ -0,0 +1,75 @@ +import { execFileSync } from "node:child_process"; +import { readFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { describe, expect, it } from "vitest"; +import { + BUILT_TEST_PROJECTS, + SOURCE_TEST_PROJECTS, + projectForTest, +} from "../../../scripts/lib/test-projects.mjs"; +import { + developmentTestPlan, + parseDevelopmentTestRequest, +} from "../../../scripts/lib/development-tests.mjs"; + +const repository = process.cwd(); + +describe("canonical test-project ownership", () => { + it("assigns every tracked test file exactly once to source or built", () => { + const files = execFileSync("git", ["ls-files", "*.test.ts"], { + cwd: repository, + encoding: "utf8", + }) + .trim() + .split("\n") + .filter(Boolean); + const grouped = new Map(); + for (const file of files) { + const project = projectForTest(file); + grouped.set(project.group, [...(grouped.get(project.group) ?? []), file]); + } + + expect(grouped.get("source")).not.toBeUndefined(); + expect(grouped.get("built")).not.toBeUndefined(); + expect( + (grouped.get("source")?.length ?? 0) + + (grouped.get("built")?.length ?? 0), + ).toBe(files.length); + expect( + SOURCE_TEST_PROJECTS.reduce( + (count, project) => count + project.include.length, + 0, + ), + ).toBeGreaterThan(0); + expect( + BUILT_TEST_PROJECTS.every((project) => project.group === "built"), + ).toBe(true); + expect(() => projectForTest("tests/unowned/example.test.ts")).toThrow( + "No test project owns", + ); + }); + + it("owns the Hopper real-provider conformance test as built and requires preparation", () => { + const testPath = + "tests/conformance/providers/hopper/realHopperSearch.test.ts"; + expect(projectForTest(testPath)).toMatchObject({ + name: "boundary", + group: "built", + }); + expect(() => parseDevelopmentTestRequest("local", [testPath])).toThrow( + `Use test:focused for built test projects: ${testPath}`, + ); + const request = parseDevelopmentTestRequest("focused", [testPath]); + expect(developmentTestPlan(request)).toMatchObject({ needsBuild: true }); + }); + + it("uses canonical selectors in Vitest while retaining independent runner policy", async () => { + const config = await readFile(join(repository, "vitest.config.ts"), "utf8"); + expect(config).toContain( + "[...SOURCE_TEST_PROJECTS, ...BUILT_TEST_PROJECTS]", + ); + expect(config).toContain("PROJECT_POLICIES"); + expect(config).not.toContain('"src/{contracts,domain}/**/*.test.ts"'); + }); +}); diff --git a/tests/boundary/filesystem/ciTestReports.test.ts b/tests/boundary/filesystem/ciTestReports.test.ts index 3e2aadcd45..233d7c8eb6 100644 --- a/tests/boundary/filesystem/ciTestReports.test.ts +++ b/tests/boundary/filesystem/ciTestReports.test.ts @@ -3,6 +3,7 @@ import { mkdir, readFile, rm, writeFile } from "node:fs/promises"; import { join, resolve } from "node:path"; import { fileURLToPath } from "node:url"; import { promisify } from "node:util"; +import { runInNewContext } from "node:vm"; import { expect, it } from "vitest"; import { parse } from "yaml"; @@ -16,6 +17,32 @@ import { workflowGit } from "../../support/workflowGit.js"; const execute = promisify(execFile); const script = new URL("../../../scripts/ci/test-report.mjs", import.meta.url); + +// Each report invocation owns its CI state and summary destination. Inherited +// workflow flags must not change fixture outcomes or append to the real job. +const executeTimingReport = async ( + root: string, + environment: NodeJS.ProcessEnv = {}, +) => { + const summary = join(root, "summary.md"); + await writeFile(summary, ""); + const output = await execute(process.execPath, [fileURLToPath(script)], { + cwd: root, + env: { + ...process.env, + SHARD_RESULT: "success", + MERGE_RESULT: "success", + SOURCE_REQUIRED: "false", + SOURCE_RESULT: "", + CI_VERIFY_TEST_INVENTORY: "false", + CI_TEST_SELECTION: "", + ...environment, + GITHUB_STEP_SUMMARY: summary, + }, + }); + return { ...output, summary: await readFile(summary, "utf8") }; +}; + const stepsSchema = z.array( z.object({ name: z.string().optional(), @@ -48,9 +75,7 @@ const workflow = async () => it("merges available failed-shard diagnostics and skips unavailable shard reports", async () => { const steps = (await workflow()).test!.steps; const download = steps.find( - (step) => - step.uses?.includes("actions/download-artifact") && - step.with?.pattern === "vitest-report-*", + (step) => step.with?.pattern === "vitest-report-*", ); const merge = steps.findIndex((step) => step.run?.includes("test:ci:merge")); const upload = steps.findIndex((step) => step.with?.name === "test-results"); @@ -59,8 +84,7 @@ it("merges available failed-shard diagnostics and skips unavailable shard report ); expect(merge).toBeGreaterThan(0); expect(download?.if).toContain("!cancelled()"); - expect(download?.if).toContain("needs.test-shard.result == 'success'"); - expect(download?.if).toContain("needs.test-shard.result == 'failure'"); + expect(upload).toBeGreaterThan(merge); expect(gate).toBeGreaterThan(upload); const merges = steps.filter((step) => step.run?.includes("--reporter=json")); @@ -70,8 +94,6 @@ it("merges available failed-shard diagnostics and skips unavailable shard report "--reporter=./scripts/ci/test-timing-reporter.mjs", ); expect(step.if).toContain("!cancelled()"); - expect(step.if).toContain("needs.test-shard.result == 'success'"); - expect(step.if).toContain("needs.test-shard.result == 'failure'"); } expect(steps[upload]?.with?.path).toContain("test-timings.json"); expect(steps[upload]?.with).toMatchObject({ "include-hidden-files": true }); @@ -86,12 +108,23 @@ it("merges available failed-shard diagnostics and skips unavailable shard report ); const command = steps[gate]?.run; if (command === undefined) throw new Error("Missing shard result gate"); - for (const result of ["failure", "cancelled", "skipped", ""]) { - await expect( - execute("bash", ["-c", command], { - env: { ...process.env, CODE_REQUIRED: "true", SHARD_RESULT: result }, - }), - ).rejects.toMatchObject({ code: 1 }); + for (const tier of ["fast", "full"]) { + for (const shard of ["success", "failure", "cancelled", "skipped", ""]) { + for (const source of ["success", "failure", "cancelled", "skipped", ""]) { + const check = execute("bash", ["-c", command], { + env: { + ...process.env, + CODE_REQUIRED: "true", + SHARD_RESULT: shard, + SOURCE_REQUIRED: String(tier === "fast"), + SOURCE_RESULT: source, + }, + }); + if (shard === "success" && (tier !== "fast" || source === "success")) + await expect(check).resolves.toMatchObject({ stderr: "" }); + else await expect(check).rejects.toMatchObject({ code: 1 }); + } + } } await expect( execute("bash", ["-c", command], { @@ -100,6 +133,107 @@ it("merges available failed-shard diagnostics and skips unavailable shard report ).resolves.toMatchObject({ stderr: "" }); }); +// These workflow conditions use the shared boolean/string expression subset. +// Evaluate the checked-in expressions so failure-state coverage cannot drift +// into a separate test implementation of the collection policy. +function conditionValue( + condition: string | undefined, + needs: Record, + canceled: boolean, + timingReportExists = false, +) { + if (!condition) throw new Error("Missing report collection condition"); + return runInNewContext( + condition + .replaceAll("needs.test-shard", 'needs["test-shard"]') + .replaceAll("needs.test-source", 'needs["test-source"]'), + { + needs, + cancelled: () => canceled, + hashFiles: () => (timingReportExists ? "timing-report" : ""), + }, + { timeout: 1_000 }, + ); +} + +it("collects either fast test group while keeping full reports tied to full shards", async () => { + const steps = (await workflow()).test.steps; + const download = steps.find( + (step) => step.with?.pattern === "vitest-report-*", + ); + const fast = steps.find( + (step) => step.name === "Merge fast-suite JUnit and timing reports", + ); + const full = steps.find( + (step) => step.name === "Merge coverage, JUnit, and timing reports", + ); + for (const tier of ["fast", "full"]) { + for (const shard of ["success", "failure", "cancelled", "skipped", ""]) { + for (const source of ["success", "failure", "cancelled", "skipped", ""]) { + for (const [code, canceled] of [ + ["true", false], + ["true", true], + ["false", false], + ["false", true], + ] as const) { + const needs = { + changes: { outputs: { code, tier } }, + "test-shard": { result: shard }, + "test-source": { result: source }, + }; + const builtAvailable = ["success", "failure"].includes(shard); + const sourceAvailable = ["success", "failure"].includes(source); + const enabled = code === "true" && !canceled; + const available = + builtAvailable || (tier === "fast" && sourceAvailable); + const expected = enabled + ? { + download: available, + fast: tier === "fast" && available, + full: tier === "full" && builtAvailable, + } + : { download: false, fast: false, full: false }; + expect(conditionValue(download?.if, needs, canceled)).toBe( + expected.download, + ); + expect(conditionValue(fast?.if, needs, canceled)).toBe(expected.fast); + expect(conditionValue(full?.if, needs, canceled)).toBe(expected.full); + } + } + } + } + const summary = steps.find((step) => + step.run?.includes("scripts/ci/test-report.mjs"), + ); + expect(summary?.env?.SHARD_RESULT).toBe("${{ needs.test-shard.result }}"); + expect(summary?.env?.SOURCE_RESULT).toBe("${{ needs.test-source.result }}"); + expect(summary?.env?.SOURCE_REQUIRED).toBe( + "${{ needs.changes.outputs.tier == 'fast' }}", + ); +}); + +it("requires portable report validation even when the timing report is missing", async () => { + const summary = (await workflow()).test.steps.find((step) => + step.run?.includes("scripts/ci/test-report.mjs"), + ); + for (const tier of ["fast", "full"]) { + for (const timingReportExists of [false, true]) { + for (const code of ["true", "false"]) { + for (const canceled of [false, true]) { + expect( + conditionValue( + summary?.if, + { changes: { outputs: { code, tier } } }, + canceled, + timingReportExists, + ), + ).toBe(code === "true" && !canceled); + } + } + } + } +}); + it("keeps diagnosis available after merge failure and publishes only complete successful history", async () => { const steps = (await workflow()).test!.steps; const diagnosis = steps.find((step) => step.with?.name === "test-results"); @@ -144,14 +278,8 @@ it("rejects successful shard reports with missing or duplicate portable files", files: names.map((name) => ({ name, durationMs: 1 })), }), ); - return execute(process.execPath, [fileURLToPath(script)], { - cwd: root, - env: { - ...process.env, - SHARD_RESULT: "success", - MERGE_RESULT: "success", - CI_VERIFY_TEST_INVENTORY: "true", - }, + return executeTimingReport(root, { + CI_VERIFY_TEST_INVENTORY: "true", }); }; await expect( @@ -169,6 +297,16 @@ it("rejects successful shard reports with missing or duplicate portable files", readFile(join(root, ".cache/ci-test-history/test-results.json")), ).rejects.toMatchObject({ code: "ENOENT" }); } + await run(["tests/first.test.ts", "tests/second.test.ts"]); + await rm(join(root, "test-timings.json")); + await expect( + executeTimingReport(root, { CI_VERIFY_TEST_INVENTORY: "true" }), + ).rejects.toMatchObject({ + stderr: expect.stringContaining("ENOENT"), + }); + await expect( + readFile(join(root, ".cache/ci-test-history/test-results.json")), + ).rejects.toMatchObject({ code: "ENOENT" }); }); it("runs the existing onboarding and Python producer guards from their selected CI owners", async () => { @@ -220,17 +358,18 @@ it("exports portable timing history and removes it on a failed, partial, or inva { name: join(root, "tests/fast.test.ts"), durationMs: 10 }, ], }; - const run = async (value: unknown, shards = "success", merge = "success") => { + const run = async ( + value: unknown, + shards = "success", + merge = "success", + source?: string, + ) => { await writeFile(join(root, "test-timings.json"), JSON.stringify(value)); - await writeFile(summary, ""); - return execute(process.execPath, [fileURLToPath(script)], { - cwd: root, - env: { - ...process.env, - GITHUB_STEP_SUMMARY: summary, - SHARD_RESULT: shards, - MERGE_RESULT: merge, - }, + return executeTimingReport(root, { + SHARD_RESULT: shards, + SOURCE_REQUIRED: String(source !== undefined), + SOURCE_RESULT: source ?? "", + MERGE_RESULT: merge, }); }; await run(report); @@ -256,6 +395,24 @@ it("exports portable timing history and removes it on a failed, partial, or inva await expect(readFile(history)).rejects.toMatchObject({ code: "ENOENT" }); expect(await readFile(summary, "utf8")).toContain("diagnostic only"); } + await run(report, "success", "success", "success"); + expect(await readFile(history, "utf8")).toContain('"success":true'); + for (const [shards, source] of [ + ["skipped", "success"], + ["success", "skipped"], + ["success", "cancelled"], + ["success", "failure"], + ["success", ""], + ]) { + await run(report); + await run(report, shards, "success", source); + await expect(readFile(history)).rejects.toMatchObject({ code: "ENOENT" }); + const text = await readFile(summary, "utf8"); + expect(text).toContain(`Source result: ${source}.`); + expect(text).toContain("diagnostic only"); + if (shards === "skipped" || source !== "failure") + expect(text).toContain("Partial test diagnostics"); + } await run(report); await expect( run({ @@ -352,10 +509,17 @@ it("recovers complete module costs from real blob reports without executing test expect(file.durationMs).toBeCloseTo( Object.values(file.phases).reduce((sum, value) => sum + value, 0), ); - await execute(process.execPath, [fileURLToPath(script)], { - cwd: root, - env: { ...process.env, SHARD_RESULT: "success", MERGE_RESULT: "success" }, + const partial = await executeTimingReport(root, { + SHARD_RESULT: "skipped", + SOURCE_REQUIRED: "true", + SOURCE_RESULT: "success", }); + expect(partial.summary).toContain("Partial test diagnostics"); + expect(partial.summary).toContain("sample.test.js"); + await expect( + readFile(join(root, ".cache/ci-test-history/test-results.json")), + ).rejects.toMatchObject({ code: "ENOENT" }); + await executeTimingReport(root); expect( parseTestDurations( await readFile( @@ -387,10 +551,7 @@ it("recovers complete module costs from real blob reports without executing test files: [{ state: "failed" }], }); // Even a caller accidentally marking the merge successful cannot publish this report. - await execute(process.execPath, [fileURLToPath(script)], { - cwd: root, - env: { ...process.env, SHARD_RESULT: "success", MERGE_RESULT: "success" }, - }); + await executeTimingReport(root); await expect( readFile(join(root, ".cache/ci-test-history/test-results.json")), ).rejects.toMatchObject({ code: "ENOENT" }); @@ -420,21 +581,19 @@ it("records failed omissions from full reports without treating partial or unhan state: index === 1 ? "failed" : "passed", })), }; - const run = async (report: unknown) => { + const run = async (report: unknown, source = "failure") => { await writeFile(join(root, "test-timings.json"), JSON.stringify(report)); - await execute(process.execPath, [fileURLToPath(script)], { - cwd: root, - env: { - ...process.env, - SHARD_RESULT: "failure", - MERGE_RESULT: "failure", - CI_VERIFY_TEST_INVENTORY: "true", - CI_TEST_SELECTION: JSON.stringify( - planTestSelection(["src/sqlite/Provider.ts"], "pull_request", "fast"), - ), - GITHUB_SHA: "merge-sha", - CI_PR_HEAD_SHA: "pr-head-sha", - }, + await executeTimingReport(root, { + SHARD_RESULT: "failure", + SOURCE_REQUIRED: "true", + SOURCE_RESULT: source, + MERGE_RESULT: "failure", + CI_VERIFY_TEST_INVENTORY: "true", + CI_TEST_SELECTION: JSON.stringify( + planTestSelection(["src/sqlite/Provider.ts"], "pull_request", "fast"), + ), + GITHUB_SHA: "merge-sha", + CI_PR_HEAD_SHA: "pr-head-sha", }); return JSON.parse( await readFile( @@ -458,6 +617,8 @@ it("records failed omissions from full reports without treating partial or unhan { ...timing, reason: "interrupted" }, ]) expect(await run(report)).toMatchObject({ complete: false }); + for (const source of ["skipped", "cancelled", ""]) + expect(await run(timing, source)).toMatchObject({ complete: false }); await expect(run({ ...timing, files: "invalid" })).rejects.toMatchObject({ code: 1, }); diff --git a/tests/boundary/filesystem/ciVerifiedPackage.test.ts b/tests/boundary/filesystem/ciVerifiedPackage.test.ts index ff50450242..c2ee285249 100644 --- a/tests/boundary/filesystem/ciVerifiedPackage.test.ts +++ b/tests/boundary/filesystem/ciVerifiedPackage.test.ts @@ -29,7 +29,7 @@ it("validates a tarball against the exact checkout and current workflow attempt" join(fixture.artifactDirectory, "package.tgz"), ); expect(await readFile(fixture.output, "utf8")).toBe( - `tarball=${join(fixture.artifactDirectory, "package.tgz")}\n`, + `tarball=${join(fixture.artifactDirectory, "package.tgz")}\nversion=1.2.3\n`, ); }); @@ -46,6 +46,7 @@ it.each([ ["wrong verifier platform", { platform: "darwin" }], ["wrong verifier architecture", { architecture: "arm64" }], ["wrong package digest", { sha256: "c".repeat(64) }], + ["invalid package version", { packageVersion: "invalid\noutput=injection" }], ])("rejects %s", async (_label, override) => { const fixture = await createFixture({ override }); diff --git a/tests/boundary/filesystem/ciWorkflowRun.test.ts b/tests/boundary/filesystem/ciWorkflowRun.test.ts new file mode 100644 index 0000000000..cbebb5c4d1 --- /dev/null +++ b/tests/boundary/filesystem/ciWorkflowRun.test.ts @@ -0,0 +1,54 @@ +import { expect, it } from "vitest"; +import { normalizeWorkflowRun } from "../../../scripts/ci/workflow-run.mjs"; + +it.each([ + [".github/workflows/ci.yml", ".github/workflows/ci.yml", null], + [".github/workflows/ci.yml@main", ".github/workflows/ci.yml", "main"], + [ + ".github/workflows/ci.yml@refs/heads/release/v1", + ".github/workflows/ci.yml", + "refs/heads/release/v1", + ], + [ + ".github/workflows/ci.yml@feature@v1", + ".github/workflows/ci.yml", + "feature@v1", + ], + [ + ".github/workflows/release.yaml@main", + ".github/workflows/release.yaml", + "main", + ], +])( + "preserves API path %s while separating its file identity and ref", + (path, file, ref) => { + const raw = { id: 10, path, head_sha: "a".repeat(40) }; + const parsed = normalizeWorkflowRun(raw); + expect(parsed).toEqual({ ...raw, workflowPath: { file, ref } }); + expect(raw).not.toHaveProperty("workflowPath"); + }, +); + +it.each([ + undefined, + null, + 42, + "", + ".github/workflows/ci.yml@", + ".github/workflows/ci.yml@main\n", + ".github/workflows/ci.yml@main\0", + "../.github/workflows/ci.yml", + ".github/workflows/../ci.yml@main", + ".github/workflows/ci.yml-extra@main", + "https://example.test/ci.yml@main", +])( + "keeps malformed workflow path %s unknown instead of inventing a file identity", + (path) => { + const parsed = normalizeWorkflowRun({ + path, + workflowPath: { file: ".github/workflows/ci.yml", ref: "main" }, + }); + expect(parsed.path).toBe(path); + expect(parsed.workflowPath).toBeNull(); + }, +); diff --git a/tests/boundary/filesystem/clientConfigurationBom.test.ts b/tests/boundary/filesystem/clientConfigurationBom.test.ts index de64f58fcf..0fe946d526 100644 --- a/tests/boundary/filesystem/clientConfigurationBom.test.ts +++ b/tests/boundary/filesystem/clientConfigurationBom.test.ts @@ -16,6 +16,7 @@ import { } from "../../../src/application/SetupClientConfiguration.js"; import { systemUninstallHost } from "../../../src/application/Uninstall.js"; import { supportedClients } from "../../../src/application/SupportedClients.js"; +import { CURRENT_ENTRYPOINT_PATH } from "../../../src/process/CurrentEntrypointPath.js"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; describe.each([ @@ -137,8 +138,10 @@ it("restores original BOM-prefixed JSONC bytes after update and removal", async await mkdir(dirname(client.configPath), { recursive: true }); await writeFile(client.configPath, original); - const firstCommand = [join(home, "bin", "rea"), "mcp"]; - const updatedCommand = [join(home, "updated", "bin", "rea"), "mcp"]; + if (CURRENT_ENTRYPOINT_PATH === undefined) + throw new Error("test runner entrypoint is unavailable"); + const firstCommand = ["rea", "mcp"]; + const updatedCommand = [process.execPath, CURRENT_ENTRYPOINT_PATH, "mcp"]; expect( await configureClientConfiguration(client, {}, firstCommand), ).toMatchObject({ status: "configured" }); diff --git a/tests/boundary/filesystem/clientConfigurationDialects.test.ts b/tests/boundary/filesystem/clientConfigurationDialects.test.ts index ce867aab4d..e4955a6aca 100644 --- a/tests/boundary/filesystem/clientConfigurationDialects.test.ts +++ b/tests/boundary/filesystem/clientConfigurationDialects.test.ts @@ -1,5 +1,5 @@ import { mkdir, readFile, writeFile } from "node:fs/promises"; -import { join } from "node:path"; +import { join, relative, resolve } from "node:path"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { parse as parseJsonc, type ParseError } from "jsonc-parser"; @@ -9,7 +9,14 @@ import { configureClientConfiguration } from "../../../src/application/SetupClie import { systemUninstallHost } from "../../../src/application/Uninstall.js"; import { supportedClients } from "../../../src/application/SupportedClients.js"; import { PRODUCT_IDENTITY } from "../../../src/identity.js"; -import { isOwnedClientRegistrationCommand } from "../../../src/application/ClientRegistrationIdentity.js"; +import { + classifyClientRegistrationCommand, + isOwnedClientRegistrationCommand, +} from "../../../src/application/ClientRegistrationIdentity.js"; +import { + CURRENT_ENTRYPOINT_PATH, + normalizeEntrypointPath, +} from "../../../src/process/CurrentEntrypointPath.js"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; import { clearClientLocationEnvironment, @@ -37,6 +44,39 @@ const getClients = (home: string) => ); describe("additional client configuration dialects", () => { + it("does not uninstall a Node registration whose relative script only matches this process cwd", async () => { + if (CURRENT_ENTRYPOINT_PATH === undefined) + throw new Error("test runner entrypoint is unavailable"); + const home = await createTestTempDirectory("rea-relative-uninstall-"); + const cursor = supportedClients(home).find(({ name }) => name === "cursor"); + if (cursor === undefined) throw new Error("missing Cursor client"); + await mkdir(cursor.markerPath ?? home, { recursive: true }); + const relativeScript = relative(process.cwd(), CURRENT_ENTRYPOINT_PATH); + const registration = { + command: process.execPath, + args: [relativeScript, "mcp"], + }; + await writeFile( + cursor.configPath, + JSON.stringify({ mcpServers: { rea: registration } }), + ); + + expect( + await readClientRegistrationStatuses(home, CURRENT_ENTRYPOINT_PATH), + ).toContainEqual( + expect.objectContaining({ + client: "cursor", + command: [process.execPath, relativeScript, "mcp"], + state: "unknown", + }), + ); + await systemUninstallHost(home).removeClient(cursor); + const after = parseJsonc(await readFile(cursor.configPath, "utf8")) as { + readonly mcpServers: Readonly>; + }; + expect(after.mcpServers.rea).toEqual(registration); + }); + it("treats an empty XDG config home as unset for OpenCode setup", async () => { const home = await createTestTempDirectory("rea-opencode-empty-xdg-"); const client = supportedClients(home, "linux", { @@ -130,7 +170,9 @@ describe("additional client configuration dialects", () => { ], ).toBeUndefined(); }); +}); +describe("OpenCode JSONC registration edits", () => { it("preserves OpenCode JSONC comments and sibling registrations during update and removal", async () => { const home = await createTestTempDirectory("rea-opencode-jsonc-"); const client = getClients(home).find(({ name }) => name === "opencode"); @@ -489,28 +531,43 @@ describe("platform-aware client config paths", () => { ]), ); }); +}); +describe("client registration command ownership", () => { it("recognizes prior exact REA releases while rejecting foreign package commands", () => { + expect(isOwnedClientRegistrationCommand(["rea", "mcp"], undefined)).toBe( + true, + ); + const cwdMatchingEntry = resolve("./rea"); expect( - isOwnedClientRegistrationCommand([ - "npx", - "-y", - `${PRODUCT_IDENTITY.packageName}@3.2.0`, - "mcp", - ]), - ).toBe(true); + classifyClientRegistrationCommand(["./rea", "mcp"], cwdMatchingEntry), + ).toBe("unknown"); expect( - isOwnedClientRegistrationCommand(["npx", "-y", "another-package", "mcp"]), + isOwnedClientRegistrationCommand(["./rea", "mcp"], cwdMatchingEntry), ).toBe(false); expect( - isOwnedClientRegistrationCommand([ - "bunx", - `${PRODUCT_IDENTITY.packageName}@3.2.0`, - "mcp", - ]), + isOwnedClientRegistrationCommand( + ["npx", "-y", `${PRODUCT_IDENTITY.packageName}@3.2.0`, "mcp"], + undefined, + ), ).toBe(true); expect( - isOwnedClientRegistrationCommand(["bunx", "another-package", "mcp"]), + isOwnedClientRegistrationCommand( + ["npx", "-y", "another-package", "mcp"], + undefined, + ), + ).toBe(false); + expect( + isOwnedClientRegistrationCommand( + ["bunx", `${PRODUCT_IDENTITY.packageName}@3.2.0`, "mcp"], + undefined, + ), + ).toBe(true); + expect( + isOwnedClientRegistrationCommand( + ["bunx", "another-package", "mcp"], + undefined, + ), ).toBe(false); expect( isOwnedClientRegistrationCommand( @@ -518,6 +575,12 @@ describe("platform-aware client config paths", () => { "/packaged/rea.mjs", ), ).toBe(true); + expect( + isOwnedClientRegistrationCommand( + [process.execPath, resolve("unknown"), "mcp"], + normalizeEntrypointPath(""), + ), + ).toBe(false); expect( isOwnedClientRegistrationCommand( ["node", "/packaged/other-agent.mjs", "mcp"], diff --git a/tests/boundary/filesystem/clientConfigurationEnvironment.test.ts b/tests/boundary/filesystem/clientConfigurationEnvironment.test.ts index 9e0b340c5b..5c5340b02d 100644 --- a/tests/boundary/filesystem/clientConfigurationEnvironment.test.ts +++ b/tests/boundary/filesystem/clientConfigurationEnvironment.test.ts @@ -29,9 +29,15 @@ it.each([ const client = { name, format, configPath }; const custom = { REA_MCP_INPUT_SCHEMA_PROFILE: "compact", + REA_MCP_OUTPUT_SCHEMA_PROFILE: "off", REA_MCP_MAX_RESPONSE_BYTES: "33554432", REA_BROWSER_EXECUTABLE: join(root, "custom-browser"), GHIDRA_INSTALL_DIR: join(root, "old-ghidra"), + HOPPER_LOADER_ARGS_JSON: '["", "--loader", "Mach-O"]', + REA_REFERENCE_SECRET_PATTERNS_JSON: '["*.private", "local/**"]', + REA_GHIDRA_LANGUAGE_ID: "x86:LE:32:default", + REA_GHIDRA_COMPILER_SPEC_ID: "borlandcpp", + CALLER_SETTING: "preserve exactly", }; const original = serializeClientConfiguration( { @@ -53,9 +59,10 @@ it.each([ ).toMatchObject({ status: "configured" }); const configured = await readFile(configPath, "utf8"); const parsed = parseClientConfiguration(configured, format); - expect(effectiveClientServer(parsed, "rea")).toMatchObject({ - [environmentKey]: { ...custom, ...providers }, - }); + expect(effectiveClientServer(parsed, "rea")).toHaveProperty( + environmentKey, + { ...custom, ...providers }, + ); expect(effectiveClientServer(parsed, "other")).toEqual( effectiveClientServer( parseClientConfiguration(original, format), @@ -137,11 +144,51 @@ it.each([ { REA_MCP_INPUT_SCHEMA_PROFILE: "bogus" }, "REA_MCP_INPUT_SCHEMA_PROFILE", ], + [ + "empty output profile", + { REA_MCP_OUTPUT_SCHEMA_PROFILE: "" }, + "REA_MCP_OUTPUT_SCHEMA_PROFILE", + ], + [ + "invalid output profile", + { REA_MCP_OUTPUT_SCHEMA_PROFILE: "bogus" }, + "REA_MCP_OUTPUT_SCHEMA_PROFILE", + ], [ "invalid response budget", { REA_MCP_MAX_RESPONSE_BYTES: "1" }, "REA_MCP_MAX_RESPONSE_BYTES", ], + [ + "malformed loader JSON", + { HOPPER_LOADER_ARGS_JSON: "{" }, + "HOPPER_LOADER_ARGS_JSON", + ], + [ + "non-array loader JSON", + { HOPPER_LOADER_ARGS_JSON: "{}" }, + "HOPPER_LOADER_ARGS_JSON", + ], + [ + "NUL loader argument", + { HOPPER_LOADER_ARGS_JSON: JSON.stringify(["a\0b"]) }, + "HOPPER_LOADER_ARGS_JSON", + ], + [ + "malformed exclusion JSON", + { REA_REFERENCE_SECRET_PATTERNS_JSON: "{" }, + "REA_REFERENCE_SECRET_PATTERNS_JSON", + ], + [ + "empty exclusion pattern", + { REA_REFERENCE_SECRET_PATTERNS_JSON: '[""]' }, + "REA_REFERENCE_SECRET_PATTERNS_JSON", + ], + [ + "unpaired compiler specification", + { REA_GHIDRA_COMPILER_SPEC_ID: "borlandcpp" }, + "REA_GHIDRA_COMPILER_SPEC_ID", + ], ] as const)( "refuses %s before changing the configuration", async (_name, env, setting) => { diff --git a/tests/boundary/filesystem/clientConfigurationLayeredEnvironment.test.ts b/tests/boundary/filesystem/clientConfigurationLayeredEnvironment.test.ts new file mode 100644 index 0000000000..41f0b984c2 --- /dev/null +++ b/tests/boundary/filesystem/clientConfigurationLayeredEnvironment.test.ts @@ -0,0 +1,219 @@ +import { access, mkdir, readFile, rm, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { expect, it } from "vitest"; +import { + clientRegistrationEntry, + parseClientConfiguration, + effectiveClientServer, +} from "../../../src/application/ClientConfigurationDocument.js"; +import { + configureClientConfiguration, + inspectClientConfiguration, + clientConfigurationAligned, +} from "../../../src/application/SetupClientConfiguration.js"; +import { readClientRegistrationStatuses } from "../../../src/application/ClientRegistrationStatus.js"; +import { npxRegistrationCommand } from "../../../src/application/ClientRegistrationIdentity.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +const command = npxRegistrationCommand(process.platform); +const language = "x86:LE:32:default"; +const compiler = "borlandcpp"; + +const layered = async (lower: unknown, upper: unknown) => { + const home = await createTestTempDirectory("rea-layered-environment-"); + const directory = join(home, ".config/opencode"); + await mkdir(directory, { recursive: true }); + const paths = [ + join(directory, "opencode.json"), + join(directory, "opencode.jsonc"), + ]; + const sources = [lower, upper].map((environment) => + JSON.stringify({ + mcp: { + rea: { + ...clientRegistrationEntry("opencode", command, {}), + environment, + }, + }, + }), + ); + for (const [index, path] of paths.entries()) + await writeFile(path, sources[index]!); + const client = { + name: "opencode", + format: "opencode", + configPath: paths[1]!, + configPaths: paths, + } as const; + const status = async () => + ( + await readClientRegistrationStatuses(home, "rea", { + environment: { XDG_CONFIG_HOME: join(home, ".config") }, + }) + ).find(({ client }) => client === "opencode"); + const unchanged = async () => { + for (const [index, path] of paths.entries()) { + expect(await readFile(path, "utf8")).toBe(sources[index]); + await expect(access(`${path}.rea.backup`)).rejects.toMatchObject({ + code: "ENOENT", + }); + } + }; + return { client, paths, sources, status, unchanged }; +}; + +it("validates language/compiler settings in the effective OpenCode environment", async () => { + const fixture = await layered( + { REA_GHIDRA_LANGUAGE_ID: language }, + { REA_GHIDRA_COMPILER_SPEC_ID: compiler }, + ); + expect(await inspectClientConfiguration(fixture.client, {}, command)).toEqual( + { status: "already_current" }, + ); + expect(await clientConfigurationAligned(fixture.client, {}, command)).toBe( + true, + ); + expect(await fixture.status()).toMatchObject({ state: "aligned" }); + expect( + await configureClientConfiguration(fixture.client, {}, command), + ).toEqual({ status: "unchanged" }); + await fixture.unchanged(); +}); + +it("rejects a companion absent from the complete layered environment", async () => { + const fixture = await layered( + { CALLER_SETTING: "keep" }, + { REA_GHIDRA_COMPILER_SPEC_ID: compiler }, + ); + expect( + await inspectClientConfiguration(fixture.client, {}, command), + ).toMatchObject({ + status: "invalid", + remediation: expect.stringContaining("REA_GHIDRA_COMPILER_SPEC_ID"), + }); + expect(await clientConfigurationAligned(fixture.client, {}, command)).toBe( + false, + ); + expect(await fixture.status()).toMatchObject({ + state: "invalid", + remediation: expect.stringContaining("REA_GHIDRA_COMPILER_SPEC_ID"), + }); + expect( + await configureClientConfiguration(fixture.client, {}, command), + ).toMatchObject({ status: "failed" }); + await fixture.unchanged(); +}); + +it("keeps higher-priority values and does not copy inherited settings into the write layer", async () => { + const fixture = await layered( + { REA_GHIDRA_LANGUAGE_ID: language, HOPPER_LOADER_ARGS_JSON: '["lower"]' }, + { REA_GHIDRA_COMPILER_SPEC_ID: compiler, HOPPER_LOADER_ARGS_JSON: '[""]' }, + ); + expect( + await configureClientConfiguration( + fixture.client, + { REA_LOG_LEVEL: "debug" }, + command, + ), + ).toMatchObject({ status: "configured" }); + expect(await readFile(fixture.paths[0]!, "utf8")).toBe(fixture.sources[0]); + const configured = parseClientConfiguration( + await readFile(fixture.client.configPath, "utf8"), + "opencode", + ); + expect(effectiveClientServer(configured, "rea")).toHaveProperty( + "environment", + { + REA_GHIDRA_COMPILER_SPEC_ID: compiler, + HOPPER_LOADER_ARGS_JSON: '[""]', + REA_LOG_LEVEL: "debug", + }, + ); + expect( + await readFile(`${fixture.client.configPath}.rea.backup`, "utf8"), + ).toBe(fixture.sources[1]); + expect(await fixture.status()).toMatchObject({ state: "aligned" }); + expect(await clientConfigurationAligned(fixture.client, {}, command)).toBe( + true, + ); + expect( + await configureClientConfiguration(fixture.client, {}, command), + ).toEqual({ status: "unchanged" }); +}); + +it.each([ + ["invalid JSON", { HOPPER_LOADER_ARGS_JSON: "{" }], + ["invalid value type", { HOPPER_LOADER_ARGS_JSON: 42 }], + ["invalid environment map", []], +])( + "does not hide a lower-priority %s behind a valid override", + async (_name, lower) => { + const fixture = await layered(lower, { HOPPER_LOADER_ARGS_JSON: "[]" }); + expect( + await inspectClientConfiguration(fixture.client, {}, command), + ).toMatchObject({ + status: "invalid", + remediation: expect.stringContaining(fixture.paths[0]!), + }); + + expect(await fixture.status()).toMatchObject({ state: "invalid" }); + expect( + await configureClientConfiguration(fixture.client, {}, command), + ).toMatchObject({ status: "failed" }); + await fixture.unchanged(); + }, +); + +it("uses inherited settings when creating a missing write layer", async () => { + const fixture = await layered({ REA_GHIDRA_LANGUAGE_ID: language }, {}); + await rm(fixture.client.configPath); + const overrides = { REA_GHIDRA_COMPILER_SPEC_ID: compiler }; + expect( + await inspectClientConfiguration(fixture.client, overrides, command), + ).toEqual({ status: "create" }); + expect( + await configureClientConfiguration(fixture.client, overrides, command), + ).toMatchObject({ status: "configured" }); + const parsed = parseClientConfiguration( + await readFile(fixture.client.configPath, "utf8"), + "opencode", + ); + expect(effectiveClientServer(parsed, "rea")).toHaveProperty( + "environment", + overrides, + ); + expect(await readFile(fixture.paths[0]!, "utf8")).toBe(fixture.sources[0]); + expect(await fixture.status()).toMatchObject({ state: "aligned" }); + await expect( + access(`${fixture.client.configPath}.rea.backup`), + ).rejects.toMatchObject({ code: "ENOENT" }); +}); + +it("replaces the selected write layer rather than the last loaded layer", async () => { + const fixture = await layered( + { REA_GHIDRA_LANGUAGE_ID: language }, + { REA_GHIDRA_COMPILER_SPEC_ID: compiler }, + ); + const client = { ...fixture.client, configPath: fixture.paths[0]! }; + const overrides = { REA_LOG_LEVEL: "debug" }; + expect(await inspectClientConfiguration(client, {}, command)).toEqual({ + status: "already_current", + }); + expect( + await configureClientConfiguration(client, overrides, command), + ).toMatchObject({ status: "configured" }); + const parsed = parseClientConfiguration( + await readFile(client.configPath, "utf8"), + "opencode", + ); + expect(effectiveClientServer(parsed, "rea")).toHaveProperty("environment", { + REA_GHIDRA_LANGUAGE_ID: language, + ...overrides, + }); + expect(await readFile(fixture.paths[1]!, "utf8")).toBe(fixture.sources[1]); + expect(await readFile(`${client.configPath}.rea.backup`, "utf8")).toBe( + fixture.sources[0], + ); + expect(await clientConfigurationAligned(client, {}, command)).toBe(true); + expect(await fixture.status()).toMatchObject({ state: "aligned" }); +}); diff --git a/tests/boundary/filesystem/clientLifecycle.test.ts b/tests/boundary/filesystem/clientLifecycle.test.ts index 0e4054ae6b..b94bfa2b80 100644 --- a/tests/boundary/filesystem/clientLifecycle.test.ts +++ b/tests/boundary/filesystem/clientLifecycle.test.ts @@ -67,6 +67,7 @@ describe("client configuration filesystem lifecycle", () => { "kimi", "vscode", "trae", + "workbuddy", "grok_build", "omp", "pi", @@ -111,6 +112,7 @@ describe("client configuration filesystem lifecycle", () => { expect(configured).toContain("[mcp_servers.rea]"); expect(configured).toContain('command = "rea"'); expect(configured).toContain("startup_timeout_sec = 30"); + expect(configured).toContain("tool_timeout_sec = 450"); expect(configured).toContain('GHIDRA_INSTALL_DIR = "/opt/ghidra"'); expect(configured).toContain('JAVA_HOME = "/opt/jdk-21"'); expect( @@ -133,7 +135,7 @@ describe("client configuration filesystem lifecycle", () => { await mkdir(dirname(configPath), { recursive: true }); await writeFile( configPath, - `[mcp_servers.rea]\ncommand = "rea"\nargs = ["mcp"]\nstartup_timeout_sec = 30\n\n[mcp_servers.rea.env]\nJAVA_HOME = "/opt/jdk-21"\nHOPPER_LAUNCHER_PATH = "/Hopper Path"\nGHIDRA_INSTALL_DIR = "/opt/ghidra"\n`, + `[mcp_servers.rea]\ncommand = "rea"\nargs = ["mcp"]\nstartup_timeout_sec = 30\ntool_timeout_sec = 450\n\n[mcp_servers.rea.env]\nJAVA_HOME = "/opt/jdk-21"\nHOPPER_LAUNCHER_PATH = "/Hopper Path"\nGHIDRA_INSTALL_DIR = "/opt/ghidra"\n`, ); const original = await readFile(configPath, "utf8"); @@ -153,7 +155,52 @@ describe("client configuration filesystem lifecycle", () => { readFile(`${configPath}.rea.backup`, "utf8"), ).rejects.toThrow(); }); +}); +describe("Codex MCP tool deadlines", () => { + it("derives the Codex tool deadline from an explicit Ghidra startup allowance", async () => { + const home = await createTestTempDirectory("rea-codex-long-timeout-"); + roots.push(home); + const configPath = join(home, ".codex/config.toml"); + await mkdir(dirname(configPath), { recursive: true }); + expect( + await configureClientConfiguration( + { name: "codex", configPath, format: "toml" }, + { REA_GHIDRA_STARTUP_TIMEOUT_MS: "2700001" }, + ["rea", "mcp"], + ), + ).toMatchObject({ status: "configured" }); + expect(await readFile(configPath, "utf8")).toContain( + "tool_timeout_sec = 2821", + ); + expect(await readFile(configPath, "utf8")).toContain( + 'REA_GHIDRA_STARTUP_TIMEOUT_MS = "2700001"', + ); + }); + + it("preserves positive caller-selected Codex startup and tool deadlines", async () => { + const home = await createTestTempDirectory("rea-codex-custom-timeout-"); + roots.push(home); + const configPath = join(home, ".codex/config.toml"); + await mkdir(dirname(configPath), { recursive: true }); + await writeFile( + configPath, + '[mcp_servers.rea]\ncommand = "old-rea"\nargs = ["mcp"]\nstartup_timeout_sec = 5\ntool_timeout_sec = 10\n', + ); + expect( + await configureClientConfiguration( + { name: "codex", configPath, format: "toml" }, + {}, + ["rea", "mcp"], + ), + ).toMatchObject({ status: "configured" }); + const configured = await readFile(configPath, "utf8"); + expect(configured).toContain("startup_timeout_sec = 5"); + expect(configured).toContain("tool_timeout_sec = 10"); + }); +}); + +describe("client configuration filesystem lifecycle", () => { it("updates a symlink target without replacing the TOML config symlink", async () => { const home = await createTestTempDirectory("rea-toml-symlink-"); roots.push(home); @@ -1334,7 +1381,7 @@ describe("Codex configuration comments", () => { await configureClientConfiguration( { name: "codex", configPath, format: "toml" }, {}, - ["/isolated prefix/bin/rea", "mcp"], + [PRODUCT_IDENTITY.cliBinary, "mcp"], ), ).toMatchObject({ status: "configured" }); const configured = await readFile(configPath, "utf8"); @@ -1504,7 +1551,7 @@ describe("client configuration filesystem removal", () => { cursor, JSON.stringify({ mcpServers: { - rea: { command: "/isolated prefix/bin/rea", args: ["mcp"] }, + rea: { command: PRODUCT_IDENTITY.cliBinary, args: ["mcp"] }, other: { command: "other" }, }, }), diff --git a/tests/boundary/filesystem/clientRegistrationStatus.test.ts b/tests/boundary/filesystem/clientRegistrationStatus.test.ts index d3cb002ead..1612ac6aae 100644 --- a/tests/boundary/filesystem/clientRegistrationStatus.test.ts +++ b/tests/boundary/filesystem/clientRegistrationStatus.test.ts @@ -1,5 +1,5 @@ import { mkdir, writeFile } from "node:fs/promises"; -import { join, resolve } from "node:path"; +import { join, relative, resolve } from "node:path"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; @@ -51,7 +51,7 @@ describe("client registration status", () => { ]); await writeFile( join(home, ".codex/config.toml"), - `[mcp_servers.rea]\ncommand = "npx"\nargs = ["-y", "${PRODUCT_IDENTITY.registrationPackageSpecifier}", "mcp"]\nstartup_timeout_sec = 30\n`, + `[mcp_servers.rea]\ncommand = "npx"\nargs = ["-y", "${PRODUCT_IDENTITY.registrationPackageSpecifier}", "mcp"]\nstartup_timeout_sec = 30\ntool_timeout_sec = 450\n`, ); await writeFile( join(home, ".cursor/mcp.json"), @@ -112,7 +112,9 @@ describe("client registration status", () => { ]); }, ); +}); +describe("client registration launcher identity", () => { it("reports an unversioned npx registration as stale", async () => { const home = await createTestTempDirectory("rea-registrations-"); await mkdir(join(home, ".codex")); @@ -131,6 +133,122 @@ describe("client registration status", () => { }), ]); }); + + it("reports an unpinned bunx registration as stale rather than aligned", async () => { + const home = await createTestTempDirectory("rea-bunx-unpinned-"); + await writeFile( + join(home, ".claude.json"), + JSON.stringify({ + mcpServers: { + rea: { command: "bunx", args: ["rea-agents", "mcp"] }, + }, + }), + ); + + const [status] = await readClientRegistrationStatuses(home); + + expect(status).toMatchObject({ + client: "claude_code", + command: ["bunx", "rea-agents", "mcp"], + state: "stale", + }); + }); + + it("preserves direct command identity as unknown when the current entrypoint is absent", async () => { + const home = await createTestTempDirectory("rea-registration-unknown-"); + await mkdir(join(home, ".cursor")); + const command = ["/opt/rea/bin/rea", "mcp"]; + await writeFile( + join(home, ".cursor/mcp.json"), + JSON.stringify({ + mcpServers: { rea: { command: command[0], args: [command[1]] } }, + }), + ); + + const [status] = await readClientRegistrationStatuses(home, ""); + + expect(status).toMatchObject({ + client: "cursor", + command, + state: "unknown", + remediation: expect.stringContaining("registration was retained"), + }); + }); + + it("does not use the REA process cwd to resolve a client's relative launcher path", async () => { + const home = await createTestTempDirectory("rea-registration-relative-"); + await mkdir(join(home, ".cursor")); + const currentEntry = resolve("./rea.mjs"); + const configPath = join(home, ".cursor/mcp.json"); + const registration = ( + scriptPath: string, + executable = process.execPath, + ) => ({ + mcpServers: { + rea: { command: executable, args: [scriptPath, "mcp"] }, + }, + }); + await writeFile(configPath, JSON.stringify(registration("./rea.mjs"))); + + const [relativeStatus] = await readClientRegistrationStatuses( + home, + currentEntry, + ); + expect(relativeStatus).toMatchObject({ + command: [process.execPath, "./rea.mjs", "mcp"], + state: "unknown", + }); + + await writeFile(configPath, JSON.stringify(registration(currentEntry))); + const [absoluteStatus] = await readClientRegistrationStatuses( + home, + currentEntry, + ); + expect(absoluteStatus).toMatchObject({ + command: [process.execPath, currentEntry, "mcp"], + state: "aligned", + }); + + const relativeInterpreter = relative(process.cwd(), process.execPath); + await writeFile( + configPath, + JSON.stringify(registration(currentEntry, relativeInterpreter)), + ); + const [relativeInterpreterStatus] = await readClientRegistrationStatuses( + home, + currentEntry, + ); + expect(relativeInterpreterStatus).toMatchObject({ + command: [relativeInterpreter, currentEntry, "mcp"], + state: "unknown", + }); + }); + + it("reports the exact requested package version when it differs from this CLI", async () => { + const home = await createTestTempDirectory( + "rea-registration-version-drift-", + ); + await mkdir(join(home, ".codex")); + const command = ["npx", "-y", "rea-agents@4.1.0", "mcp"]; + await writeFile( + join(home, ".codex/config.toml"), + `[mcp_servers.rea]\ncommand = "${command[0]}"\nargs = ["${command[1]}", "${command[2]}", "${command[3]}"]\n`, + ); + + const [status] = await readClientRegistrationStatuses(home); + + expect(status).toMatchObject({ + client: "codex", + command, + package_reference: "rea-agents@4.1.0", + state: "stale", + remediation: expect.stringContaining( + `current REA CLI package is ${PRODUCT_IDENTITY.packageVersion}`, + ), + }); + expect(status?.remediation).toContain("does not resolve package-runner"); + expect(status?.remediation).toContain("rea setup --client codex"); + }); }); describe("commandcode registration status", () => { @@ -191,14 +309,14 @@ describe("commandcode registration status", () => { describe("Node-wrapped registration policy", () => { it.each([1, 30])( - "checks Codex startup timeout %d independently of launcher", + "accepts Codex caller-selected startup timeout %d independently of launcher", async (timeout) => { const home = await createTestTempDirectory("rea-node-registration-"); await mkdir(join(home, ".codex")); const entry = resolve("scripts/rea.mjs"); await writeFile( join(home, ".codex/config.toml"), - `[mcp_servers.rea]\ncommand = ${JSON.stringify(process.execPath)}\nargs = [${JSON.stringify(entry)}, "mcp"]\nstartup_timeout_sec = ${String(timeout)}\n`, + `[mcp_servers.rea]\ncommand = ${JSON.stringify(process.execPath)}\nargs = [${JSON.stringify(entry)}, "mcp"]\nstartup_timeout_sec = ${String(timeout)}\ntool_timeout_sec = 450\n`, ); const statuses = await readClientRegistrationStatuses(home, entry, { environment: {}, @@ -206,12 +324,30 @@ describe("Node-wrapped registration policy", () => { expect(statuses).toEqual([ expect.objectContaining({ client: "codex", - state: timeout === 30 ? "aligned" : "stale", + state: "aligned", }), ]); }, ); + it("keeps a short positive Codex tool timeout aligned and reports an advisory", async () => { + const home = await createTestTempDirectory("rea-codex-short-tool-timeout-"); + await mkdir(join(home, ".codex")); + const entry = resolve("scripts/rea.mjs"); + await writeFile( + join(home, ".codex/config.toml"), + `[mcp_servers.rea]\ncommand = ${JSON.stringify(process.execPath)}\nargs = [${JSON.stringify(entry)}, "mcp"]\nstartup_timeout_sec = 5\ntool_timeout_sec = 10\n`, + ); + const [status] = await readClientRegistrationStatuses(home, entry, { + environment: {}, + }); + expect(status).toMatchObject({ + client: "codex", + state: "aligned", + advisory: expect.stringContaining("tool_timeout_sec = 450 seconds"), + }); + }); + it.each([1, 30])( "checks Grok Build startup timeout %d independently of launcher", async (timeout) => { diff --git a/tests/boundary/filesystem/directAnalysisSnapshotReplay.test.ts b/tests/boundary/filesystem/directAnalysisSnapshotReplay.test.ts index 7adf193f4a..3782509aa2 100644 --- a/tests/boundary/filesystem/directAnalysisSnapshotReplay.test.ts +++ b/tests/boundary/filesystem/directAnalysisSnapshotReplay.test.ts @@ -136,6 +136,7 @@ interface SnapshotProviderOptions { readonly profile?: ReturnType; readonly identity?: ProviderIdentity; readonly supportedOperations?: readonly CapabilityDescriptor["operation"][]; + readonly limitations?: readonly string[]; } const makeProvider = ( @@ -209,6 +210,7 @@ const makeProvider = ( createAnalysisExecution(result, profile.provider, { analysisProfile: profile, rawResult: result, + limitations: options.limitations ?? [], }), ); }, @@ -243,6 +245,7 @@ const createSnapshotReplayFixture = async ( return { readConfiguration: () => parseConfig({}), createBinarySession: () => createTestBinarySession(provider), + createArtifactBinarySession: () => createTestBinarySession(provider), createManagedBinarySession: () => createTestBinarySession(provider), }; }, @@ -253,11 +256,11 @@ const withAlternateWorkflowProfile = ( snapshot: AnalysisSnapshot, path: string, current: Evidence, -): AnalysisSnapshot => { - const alternateProfile = createAnalysisProfile(REA_WORKFLOW_PROVIDER, { + alternateProfile = createAnalysisProfile(REA_WORKFLOW_PROVIDER, { workflow: "binary_overview", fixture: "alternate-profile", - }); + }), +): AnalysisSnapshot => { if ( typeof current.normalized_result !== "object" || current.normalized_result === null || @@ -466,6 +469,80 @@ const verifyWindowsSnapshotAdmission = async ( ).toHaveLength(1); }; +it.each([ + { tool: "binary_overview", arguments: {}, priorRevision: 4 }, + { + tool: "inspect_native_api", + arguments: { procedure: "0x1000" }, + priorRevision: 1, + }, +] as const)( + "refreshes unattributed $tool Evidence and replays the current revision", + async (scenario) => { + const fixture = await createSnapshotReplayFixture("rea-prior-coverage-"); + const limitation = "Original PDB names may be missing."; + const provider = fixture.createProvider({ + supportedOperations: workflowProviderOperations, + limitations: [limitation], + }); + const dependencies = fixture.dependencies(provider); + const options = { snapshotPath: fixture.snapshotPath }; + const first = parseEvidence( + await runDirectAnalysis( + dependencies, + fixture.path, + scenario.tool, + scenario.arguments, + options, + ), + ); + const saved = await readAnalysisSnapshot(fixture.snapshotPath); + if (!saved.ok) throw saved.error; + const priorProfile = createAnalysisProfile(REA_WORKFLOW_PROVIDER, { + upstream_analysis_profile: saved.value.binding.analysis_profile, + ...(scenario.priorRevision === undefined + ? {} + : { workflow_revision: scenario.priorRevision }), + }); + const prior = withAlternateWorkflowProfile( + saved.value, + fixture.path, + { ...first, limitations: ["Derived by an REA composed workflow."] }, + priorProfile, + ); + const written = await withArtifactResourceScope((resources) => + writeAnalysisSnapshot(prior, fixture.snapshotPath, true, resources), + ); + if (!written.ok) throw written.error; + const refreshed = parseEvidence( + await runDirectAnalysis( + dependencies, + fixture.path, + scenario.tool, + scenario.arguments, + options, + ), + ); + expect(refreshed.limitations).toEqual( + expect.arrayContaining(first.limitations), + ); + expect(refreshed.analysis_profile?.digest).not.toBe(priorProfile.digest); + expect(refreshed.normalized_result).toEqual(first.normalized_result); + const starts = [...fixture.starts]; + const calls = [...fixture.calls]; + const replayed = await runDirectAnalysis( + dependencies, + fixture.path, + scenario.tool, + scenario.arguments, + options, + ); + expect(replayed).toEqual(refreshed); + expect(fixture.starts).toEqual(starts); + expect(fixture.calls).toEqual(calls); + }, +); + describe("direct analysis composed snapshot replay", () => { it.skipIf(process.platform !== "win32")( "does not replay a CLI snapshot through a Windows junction", @@ -924,6 +1001,7 @@ describe("workflow snapshot profile and cancellation binding", () => { const dependencies: DirectAnalysisDependencies = { readConfiguration: () => parseConfig({}), createBinarySession: () => session, + createArtifactBinarySession: () => session, createManagedBinarySession: () => session, }; const result = await runDirectAnalysis( diff --git a/tests/boundary/filesystem/directoryArtifactReaderOwnership.test.ts b/tests/boundary/filesystem/directoryArtifactReaderOwnership.test.ts index 4145e7bca2..852275c2c0 100644 --- a/tests/boundary/filesystem/directoryArtifactReaderOwnership.test.ts +++ b/tests/boundary/filesystem/directoryArtifactReaderOwnership.test.ts @@ -1,4 +1,4 @@ -import { writeFile, type FileHandle } from "node:fs/promises"; +import { mkdir, rename, writeFile, type FileHandle } from "node:fs/promises"; import { join } from "node:path"; import { afterEach, describe, expect, it, vi } from "vitest"; @@ -47,6 +47,35 @@ afterEach(() => { }); describe("DirectoryArtifactReader resource ownership", () => { + it("rejects descendants after the selected root directory is replaced", async () => { + const { root, reader } = await makeFixture(); + const subdirectory = join(root, "sub"); + await mkdir(subdirectory); + await writeFile(join(subdirectory, "old.txt"), "original tree\n"); + const movedRoot = `${root}-original`; + try { + const entries = reader.entries()[Symbol.asyncIterator](); + // Root entries follow the filesystem's directory order; descendants are + // read only after the root is exhausted. + const rootEntries = [ + (await entries.next()).value?.path, + (await entries.next()).value?.path, + ]; + expect(rootEntries.sort()).toEqual(["sample.txt", "sub"]); + + await rename(root, movedRoot); + await mkdir(root); + await mkdir(subdirectory); + await writeFile(join(subdirectory, "new.txt"), "replacement tree\n"); + await expect(entries.next()).rejects.toMatchObject({ + reason: "path", + message: "Directory traversal escaped or changed its artifact root", + }); + } finally { + await reader.close(); + } + }); + it("closes the exact directory owner on early iterator return", async () => { const { reader } = await makeFixture(); let closeCalls = 0; diff --git a/tests/boundary/filesystem/doctorSystemHostFilesystem.test.ts b/tests/boundary/filesystem/doctorSystemHostFilesystem.test.ts index 22a2188731..0abf149071 100644 --- a/tests/boundary/filesystem/doctorSystemHostFilesystem.test.ts +++ b/tests/boundary/filesystem/doctorSystemHostFilesystem.test.ts @@ -23,10 +23,27 @@ it("threads the selected host and environment into system diagnostics", async () environment, execFileOutput: async (command, arguments_, options) => { calls.push({ command, arguments: arguments_, environment: options?.env }); - if (command === "sw_vers") return { stdout: "15.0\n", stderr: "" }; - if (command === "ldd") return { stdout: "linux-vdso.so.1", stderr: "" }; + if (command === "sw_vers") + return { + stdout: "15.0\n", + stderr: "", + stdoutBytes: 5, + stderrBytes: 0, + }; + if (command === "ldd") + return { + stdout: "linux-vdso.so.1", + stderr: "", + stdoutBytes: Buffer.byteLength("linux-vdso.so.1"), + stderrBytes: 0, + }; if (command === "which") - return { stdout: "/selected/bin/rea\n", stderr: "" }; + return { + stdout: "/selected/bin/rea\n", + stderr: "", + stdoutBytes: 18, + stderrBytes: 0, + }; throw new Error(`Unexpected diagnostic command: ${command}`); }, }); @@ -88,7 +105,12 @@ it.each([ environment: { HOME: join(root, "home") }, execFileOutput: async (_command, arguments_) => { calls.push(arguments_); - return { stdout: `${caskroom}\n`, stderr: "" }; + return { + stdout: `${caskroom}\n`, + stderr: "", + stdoutBytes: Buffer.byteLength(`${caskroom}\n`), + stderrBytes: 0, + }; }, }); @@ -111,7 +133,12 @@ it("reports no Homebrew Hopper when the cask has no receipt", async () => { environment: { HOME: caskroom }, execFileOutput: async (_command, arguments_) => { calls.push(arguments_); - return { stdout: `${caskroom}\n`, stderr: "" }; + return { + stdout: `${caskroom}\n`, + stderr: "", + stdoutBytes: Buffer.byteLength(`${caskroom}\n`), + stderrBytes: 0, + }; }, }); diff --git a/tests/boundary/filesystem/documentationFacts.test.ts b/tests/boundary/filesystem/documentationFacts.test.ts new file mode 100644 index 0000000000..64e19fe382 --- /dev/null +++ b/tests/boundary/filesystem/documentationFacts.test.ts @@ -0,0 +1,91 @@ +import { copyFile, mkdir, readFile, rm, writeFile } from "node:fs/promises"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +import { afterEach, describe, expect, it } from "vitest"; + +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +import { documentationFactIssues } from "../../../scripts/lib/docs-facts.mjs"; +import type { ProductCatalog } from "../../../scripts/lib/product-catalog.mjs"; + +const root = fileURLToPath(new URL("../../..", import.meta.url)); +const temporaryRoots: string[] = []; + +const README_FACT_FIXTURE_PATHS = [ + "README.md", + "docs/readme/README_zh.md", + "docs/readme/README_ja.md", + "docs/readme/README_ko.md", + "docs/readme/README_ar.md", + "website/locales/languages.json", + "docs/installation.md", + "AGENTS.md", + ".github/pull_request_template.md", +] as const; + +const README_LINK_DRIFT_MUTATIONS = [ + [ + "(../installation.md#supported-agents)", + "(../installation.md#missing-agents)", + ], + ["https://rea.tools/zh-CN/guides/", "https://rea.tools/zh-CN/missing/"], + ["https://rea.tools/zh-CN/guides/", "https://rea.tools/zh-TW/guides/"], + [ + "https://rea.tools/zh-CN/guides/", + "https://rea.tools/zh-CN/guides/?other=1", + ], + [ + 'src="../assets/rea-hopper-analysis.png"', + 'src="../assets/missing-hopper.png"', + ], + ['href="https://discord.gg/GkcryMnJDM"', 'href="https://discord.gg/missing"'], +] as const; + +const seedReadmeFactFixture = async (directory: string) => { + for (const path of README_FACT_FIXTURE_PATHS) { + const destination = join(directory, path); + await mkdir(dirname(destination), { recursive: true }); + await copyFile(join(root, path), destination); + } + await mkdir(join(directory, "skills/reverse-engineer-anything"), { + recursive: true, + }); +}; + +afterEach(async () => { + await Promise.all( + temporaryRoots + .splice(0) + .map((path) => rm(path, { recursive: true, force: true })), + ); +}); + +describe("translated README facts", () => { + it("keeps translated readmes linked to canonical setup and tool details", async () => { + const directory = await createTestTempDirectory("rea-readme-facts-"); + temporaryRoots.push(directory); + await seedReadmeFactFixture(directory); + const translatedPath = join(directory, "docs/readme/README_zh.md"); + const translated = await readFile(translatedPath, "utf8"); + const catalog: ProductCatalog = JSON.parse( + await readFile(join(root, "docs/public/product-catalog.json"), "utf8"), + ); + await expect(documentationFactIssues(directory, catalog)).resolves.toEqual( + [], + ); + for (const [original, changed] of README_LINK_DRIFT_MUTATIONS) { + expect(translated).toContain(original); + await writeFile( + translatedPath, + translated.replace(original, changed), + "utf8", + ); + await expect( + documentationFactIssues(directory, catalog), + ).resolves.toEqual([ + "docs/readme/README_zh.md: documentation links differ from README.md", + ]); + } + }); +}); diff --git a/tests/boundary/filesystem/dshClientConfiguration.test.ts b/tests/boundary/filesystem/dshClientConfiguration.test.ts index fd65e6b00c..0142716b64 100644 --- a/tests/boundary/filesystem/dshClientConfiguration.test.ts +++ b/tests/boundary/filesystem/dshClientConfiguration.test.ts @@ -1,4 +1,4 @@ -import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { access, mkdir, readFile, writeFile } from "node:fs/promises"; import { join, resolve } from "node:path"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; @@ -1704,6 +1704,119 @@ const dshStatus = async (home: string) => }) ).find(({ client }) => client === "dsh"); +describe("DeepSeek Harness runtime environment validation", () => { + it.each([ + ["HOPPER_LOADER_ARGS_JSON", "{"], + ["REA_REFERENCE_SECRET_PATTERNS_JSON", '[""]'], + ["REA_GHIDRA_COMPILER_SPEC_ID", "borlandcpp"], + ])("refuses invalid literal %s before writing", async (setting, value) => { + const { home, client } = await harnessClient(); + const entry = canonicalEntry(); + const original = stringifyYaml([ + { + insert: [ + { ...entry, config: { ...entry.config, env: { [setting]: value } } }, + ], + }, + ]); + await writeFile(client.configPath, original); + expect(await inspectClientConfiguration(client, {}, command)).toMatchObject( + { status: "invalid", remediation: expect.stringContaining(setting) }, + ); + expect(await dshStatus(home)).toMatchObject({ + state: "invalid", + remediation: expect.stringContaining(setting), + }); + expect( + await configureClientConfiguration(client, {}, command), + ).toMatchObject({ status: "failed" }); + expect(await readFile(client.configPath, "utf8")).toBe(original); + await expect( + access(`${client.configPath}.rea.backup`), + ).rejects.toMatchObject({ code: "ENOENT" }); + }); + + it("keeps an expression-backed companion unknown until a literal override replaces it", async () => { + const { home, client } = await harnessClient(); + const entry = canonicalEntry(); + const document = parseDocument( + stringifyYaml([ + { + insert: [ + { + ...entry, + config: { + ...entry.config, + command: "stale", + env: { + REA_GHIDRA_COMPILER_SPEC_ID: "borlandcpp", + CALLER_SETTING: "keep", + }, + }, + }, + ], + }, + ]), + ); + for (const setting of [ + "REA_GHIDRA_LANGUAGE_ID", + "HOPPER_LOADER_ARGS_JSON", + ]) { + const expression = document.createNode(`process.env.${setting}`); + expression.tag = "tag:yaml.org,2002:js"; + document.setIn([0, "insert", 0, "config", "env", setting], expression); + } + const original = document.toString(); + await writeFile(client.configPath, original); + expect(await inspectClientConfiguration(client, {}, command)).toMatchObject( + { status: "update" }, + ); + expect( + await configureClientConfiguration(client, {}, command), + ).toMatchObject({ status: "configured" }); + const configured = await readFile(client.configPath, "utf8"); + expect(configured).toContain("!!js process.env.REA_GHIDRA_LANGUAGE_ID"); + expect(configured).toContain("!!js process.env.HOPPER_LOADER_ARGS_JSON"); + expect(configured).toContain("CALLER_SETTING: keep"); + expect(await dshStatus(home)).toMatchObject({ state: "aligned" }); + expect( + await configureClientConfiguration(client, {}, command), + ).toMatchObject({ status: "unchanged" }); + await expect( + inspectClientConfiguration( + client, + { HOPPER_LOADER_ARGS_JSON: "{" }, + command, + ), + ).rejects.toMatchObject({ + settings: [{ setting: "HOPPER_LOADER_ARGS_JSON" }], + }); + await expect( + inspectClientConfiguration( + client, + { REA_GHIDRA_LANGUAGE_ID: "" }, + command, + ), + ).rejects.toMatchObject({ + settings: [{ setting: "REA_GHIDRA_LANGUAGE_ID" }], + }); + expect( + await inspectClientConfiguration( + client, + { + REA_GHIDRA_LANGUAGE_ID: "x86:LE:32:default", + HOPPER_LOADER_ARGS_JSON: '[""]', + }, + command, + ), + ).toMatchObject({ status: "update" }); + expect(await readFile(client.configPath, "utf8")).toBe(configured); + expect(await readFile(`${client.configPath}.rea.backup`, "utf8")).toBe( + original, + ); + }); +}); + describe("DeepSeek Harness source ownership", () => { it.each(["setup", "uninstall"])( "preserves unrelated rows in a later externally anchored operation during %s", diff --git a/tests/boundary/filesystem/ghidraJavaProbe.test.ts b/tests/boundary/filesystem/ghidraJavaProbe.test.ts index 415fbf3ddb..d48dc1ece9 100644 --- a/tests/boundary/filesystem/ghidraJavaProbe.test.ts +++ b/tests/boundary/filesystem/ghidraJavaProbe.test.ts @@ -88,6 +88,109 @@ it.each([ }, ); +it.each([ + { label: "omitted", property: "", shown: "bitness unknown" }, + { + label: "empty", + property: "sun.arch.data.model =\n", + shown: 'bitness unknown (sun.arch.data.model = "")', + }, + { + label: "malformed", + property: "sun.arch.data.model = 64junk\n", + shown: 'bitness unknown (sun.arch.data.model = "64junk")', + }, + { + label: "not exactly representable", + property: "sun.arch.data.model = 9007199254740993\n", + shown: 'bitness unknown (sun.arch.data.model = "9007199254740993")', + }, +])("keeps $label Java bitness unresolved", async ({ property, shown }) => { + const root = await createTestTempDirectory("rea-java-bitness-"); + const configuredHome = join(root, "selected-jdk"); + const reportedHome = join(root, "reported-jdk"); + const installDir = join(root, "ghidra"); + await mkdir(join(installDir, "Ghidra"), { recursive: true }); + await mkdir(join(reportedHome, "bin"), { recursive: true }); + await writeFile( + join( + reportedHome, + "bin", + process.platform === "win32" ? "javac.exe" : "javac", + ), + "fixture", + { + mode: 0o755, + }, + ); + await writeFile( + join(installDir, "Ghidra", "application.properties"), + `application.version=${SUPPORTED_GHIDRA_VERSION}\napplication.java.min=17\n`, + ); + mockSpawnSync.mockReturnValue({ + stdout: "", + stderr: ` java.version = 21.0.11\n java.home = ${reportedHome}\n${property}`, + status: 0, + }); + + const inspection = inspectGhidraInstallation({ + environment: {}, + installDir, + javaHome: configuredHome, + }); + + expect(inspection.status).toBe("unavailable"); + expect(inspection.checks.find(({ name }) => name === "java")).toMatchObject({ + status: "failed", + code: "version_unresolved", + detail: expect.stringContaining(shown), + }); + expect( + inspection.checks.find(({ name }) => name === "java")?.detail, + ).not.toMatch(/(?:0-bit|9007199254740992-bit)/); +}); + +it("keeps a known 32-bit Java report classified as unsupported", async () => { + const root = await createTestTempDirectory("rea-java-bitness-known-"); + const configuredHome = join(root, "selected-jdk"); + const reportedHome = join(root, "reported-jdk"); + const installDir = join(root, "ghidra"); + await mkdir(join(installDir, "Ghidra"), { recursive: true }); + await mkdir(join(reportedHome, "bin"), { recursive: true }); + await writeFile( + join( + reportedHome, + "bin", + process.platform === "win32" ? "javac.exe" : "javac", + ), + "fixture", + { + mode: 0o755, + }, + ); + await writeFile( + join(installDir, "Ghidra", "application.properties"), + `application.version=${SUPPORTED_GHIDRA_VERSION}\napplication.java.min=17\n`, + ); + mockSpawnSync.mockReturnValue({ + stdout: "", + stderr: ` java.version = 21.0.11\n java.home = ${reportedHome}\n sun.arch.data.model = 32\n`, + status: 0, + }); + + const inspection = inspectGhidraInstallation({ + environment: {}, + installDir, + javaHome: configuredHome, + }); + + expect(inspection.checks.find(({ name }) => name === "java")).toMatchObject({ + status: "failed", + code: "unsupported_version", + detail: expect.stringContaining("32-bit"), + }); +}); + it("bounds the Java version probe and reports an over-budget probe as a failed Java check", async () => { const root = await createTestTempDirectory("rea-java-probe-bound-"); const configuredHome = join(root, "selected-jdk"); diff --git a/tests/boundary/filesystem/ghidraLanguageCatalogBound.test.ts b/tests/boundary/filesystem/ghidraLanguageCatalogBound.test.ts new file mode 100644 index 0000000000..8555c1dbdc --- /dev/null +++ b/tests/boundary/filesystem/ghidraLanguageCatalogBound.test.ts @@ -0,0 +1,86 @@ +import { mkdir, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { afterEach, expect, it, vi } from "vitest"; + +import { readGhidraLanguageCatalog } from "../../../src/ghidra/GhidraLanguageCatalog.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +const LIMIT = 4 * 1024 * 1024; +const OVERSIZED = "oversized.ldefs"; + +// Count bytes delivered from the oversized definition, whichever read API is used. +const { readBytes } = vi.hoisted(() => ({ readBytes: { value: 0 } })); +vi.mock("node:fs/promises", async (importOriginal) => { + const actual = await importOriginal(); + const tracked = (path: unknown) => String(path).endsWith(OVERSIZED); + return { + ...actual, + readFile: async (...args: Parameters) => { + const bytes = await actual.readFile(...args); + if (tracked(args[0])) readBytes.value += bytes.length; + return bytes; + }, + open: async (...args: Parameters) => { + const handle = await actual.open(...args); + if (!tracked(args[0])) return handle; + const read = handle.read.bind(handle) as ( + ...readArgs: unknown[] + ) => Promise<{ bytesRead: number }>; + return Object.assign(handle, { + read: async (...readArgs: unknown[]) => { + const result = await read(...readArgs); + readBytes.value += result.bytesRead; + return result; + }, + }); + }, + }; +}); + +afterEach(() => { + readBytes.value = 0; +}); + +it("skips an oversized definition without buffering it beyond the catalog limit", async () => { + const root = await createTestTempDirectory("rea-ghidra-ldefs-bound-"); + const module = (name: string) => + join(root, "Ghidra/Processors", name, "data/languages"); + await mkdir(module("huge"), { recursive: true }); + await mkdir(module("x86"), { recursive: true }); + await writeFile( + join(module("huge"), OVERSIZED), + Buffer.concat([ + Buffer.from( + '', + ), + Buffer.alloc(2 * LIMIT, 0x20), + ]), + ); + await writeFile( + join(module("x86"), "x86.ldefs"), + '', + ); + + expect(await readGhidraLanguageCatalog(root)).toEqual([ + { id: "x86:LE:32:default", compilerSpecIds: [] }, + ]); + expect(readBytes.value).toBeGreaterThan(0); + expect(readBytes.value).toBeLessThanOrEqual(LIMIT + 1); +}); + +it("keeps a definition at exactly the catalog limit", async () => { + const root = await createTestTempDirectory("rea-ghidra-ldefs-limit-"); + const languages = join(root, "Ghidra/Processors/edge/data/languages"); + await mkdir(languages, { recursive: true }); + const document = Buffer.from( + '', + ); + await writeFile( + join(languages, "edge.ldefs"), + Buffer.concat([document, Buffer.alloc(LIMIT - document.length, 0x20)]), + ); + + expect(await readGhidraLanguageCatalog(root)).toEqual([ + { id: "edge:LE:32:default", compilerSpecIds: [] }, + ]); +}); diff --git a/tests/boundary/filesystem/ghidraLanguageCatalogCleanup.test.ts b/tests/boundary/filesystem/ghidraLanguageCatalogCleanup.test.ts new file mode 100644 index 0000000000..c009837658 --- /dev/null +++ b/tests/boundary/filesystem/ghidraLanguageCatalogCleanup.test.ts @@ -0,0 +1,203 @@ +import { mkdir, writeFile } from "node:fs/promises"; +import { basename, join } from "node:path"; +import { afterEach, expect, it, vi } from "vitest"; + +import { + GhidraLanguageCatalogCleanupError, + readGhidraLanguageCatalog, +} from "../../../src/ghidra/GhidraLanguageCatalog.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +interface FileControl { + closeFails: boolean; + /** Hold close() until every synchronized handle has reached it. */ + synchronizedClose: boolean; + invalidateOnFailure: boolean; + readFails: boolean; + invalidated: boolean; + closeCalls: number; +} + +// Per-file failure controls, keyed by definition file name, and a barrier +// that releases synchronized closes only after all of them are pending. +const { controls, barrier } = vi.hoisted(() => { + const state = { expected: 0, arrived: 0, release: () => {} }; + const gate = new Promise((resolve) => { + state.release = resolve; + }); + return { + controls: new Map(), + barrier: { + state, + arrive: async () => { + state.arrived += 1; + if (state.arrived >= state.expected) state.release(); + await gate; + }, + }, + }; +}); +vi.mock("node:fs/promises", async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + open: async (...args: Parameters) => { + const real = await actual.open(...args); + const control = controls.get(basename(String(args[0]))); + if (control === undefined) return real; + let released = false; + const release = async () => { + if (released) return; + released = true; + await real.close(); + }; + return { + get fd() { + return control.invalidated ? -1 : real.fd; + }, + read: (...readArgs: Parameters) => + control.readFails + ? Promise.reject(new Error("definition read failed")) + : real.read(...readArgs), + close: async () => { + control.closeCalls += 1; + if (control.synchronizedClose && control.closeCalls === 1) + await barrier.arrive(); + if (control.closeFails) { + if (control.invalidateOnFailure) { + await release(); + control.invalidated = true; + } + throw new Error(`definition close failed: ${String(args[0])}`); + } + await release(); + }, + }; + }, + }; +}); + +let preserveUnconfirmedOwner = false; + +const control = (name: string, overrides: Partial = {}) => { + const value: FileControl = { + closeFails: false, + synchronizedClose: false, + invalidateOnFailure: false, + readFails: false, + invalidated: false, + closeCalls: 0, + ...overrides, + }; + controls.set(name, value); + return value; +}; + +const installation = async (name: string): Promise => { + const root = await createTestTempDirectory("rea-ghidra-ldefs-cleanup-"); + const languages = join(root, "Ghidra/Processors", name, "data/languages"); + await mkdir(languages, { recursive: true }); + await writeFile( + join(languages, `${name}.ldefs`), + ``, + ); + return root; +}; + +// Let every retained owner close, then drain them through a healthy read. +afterEach(async () => { + if (preserveUnconfirmedOwner) { + preserveUnconfirmedOwner = false; + controls.clear(); + return; + } + for (const value of controls.values()) value.closeFails = false; + await readGhidraLanguageCatalog(await installation("drain")); + controls.clear(); +}); + +it("retries a retained close before the next read", async () => { + const retained = control("flaky.ldefs", { closeFails: true }); + await expect( + readGhidraLanguageCatalog(await installation("flaky")), + ).rejects.toMatchObject({ paths: [expect.stringContaining("flaky.ldefs")] }); + retained.closeFails = false; + expect(await readGhidraLanguageCatalog(await installation("x86"))).toEqual([ + { id: "x86:LE:32:default", compilerSpecIds: [] }, + ]); + expect(retained.closeCalls).toBe(2); +}); + +it("attempts every retained close and reports the remaining failures together", async () => { + const first = control("first.ldefs", { + closeFails: true, + synchronizedClose: true, + }); + const second = control("second.ldefs", { + closeFails: true, + synchronizedClose: true, + }); + barrier.state.expected = 2; + // Concurrent resolutions can each retain an owner. + const roots = [await installation("first"), await installation("second")]; + const results = await Promise.allSettled( + roots.map((root) => readGhidraLanguageCatalog(root)), + ); + expect(results.map(({ status }) => status)).toEqual(["rejected", "rejected"]); + second.closeFails = false; + const failure = await readGhidraLanguageCatalog( + await installation("x86"), + ).catch((cause: unknown) => cause); + expect(failure).toBeInstanceOf(GhidraLanguageCatalogCleanupError); + expect(failure).toMatchObject({ + paths: [expect.stringContaining("first.ldefs")], + }); + expect(second.closeCalls).toBe(2); + expect(first.closeCalls).toBe(2); +}); + +it("keeps the read failure when the close also fails", async () => { + control("damaged.ldefs", { readFails: true, closeFails: true }); + const failure = await readGhidraLanguageCatalog( + await installation("damaged"), + ).catch((cause: unknown) => cause); + expect(failure).toBeInstanceOf(GhidraLanguageCatalogCleanupError); + expect(failure).toMatchObject({ + readFailure: expect.objectContaining({ message: "definition read failed" }), + }); +}); + +it("reports retained cleanup for a catalog without definitions", async () => { + control("pending.ldefs", { closeFails: true }); + await expect( + readGhidraLanguageCatalog(await installation("pending")), + ).rejects.toBeInstanceOf(GhidraLanguageCatalogCleanupError); + const empty = await createTestTempDirectory("rea-ghidra-ldefs-empty-"); + await expect(readGhidraLanguageCatalog(empty)).rejects.toMatchObject({ + paths: [expect.stringContaining("pending.ldefs")], + }); +}); + +it("retains an invalidated descriptor owner and keeps later reads blocked", async () => { + const broken = control("broken.ldefs", { + closeFails: true, + invalidateOnFailure: true, + }); + const first = await readGhidraLanguageCatalog( + await installation("broken"), + ).catch((cause: unknown) => cause); + expect(first).toBeInstanceOf(GhidraLanguageCatalogCleanupError); + expect(first).toMatchObject({ + paths: [expect.stringContaining("broken.ldefs")], + }); + + const later = await readGhidraLanguageCatalog( + await installation("x86"), + ).catch((cause: unknown) => cause); + expect(later).toBeInstanceOf(GhidraLanguageCatalogCleanupError); + expect(later).toMatchObject({ + paths: [expect.stringContaining("broken.ldefs")], + }); + expect(broken.closeCalls).toBe(1); + preserveUnconfirmedOwner = true; +}); diff --git a/tests/boundary/filesystem/ghidraPspExtension.test.ts b/tests/boundary/filesystem/ghidraPspExtension.test.ts index 65a4d15124..33261d96e2 100644 --- a/tests/boundary/filesystem/ghidraPspExtension.test.ts +++ b/tests/boundary/filesystem/ghidraPspExtension.test.ts @@ -46,6 +46,7 @@ const target = { byteOrder: "little", type: 2, flags: 0x10a23001, + gnuAttributes: null, abiFlags: pspAbiFlags, }, } satisfies BinaryTarget; @@ -55,6 +56,8 @@ const host: GhidraInstallationHost = { readText: () => "application.version=12.1.3\napplication.java.min=21\n", executable: () => true, probeJava: () => ({ + osArch: "amd64", + osName: "Linux", version: "21.0.12", major: 21, home: "/jdk-21", @@ -310,34 +313,85 @@ describe.skipIf(process.platform === "win32")( expect(result.error._tag).toBe("AnalysisUnsupportedTargetError"); }, ); + + it("refuses a separately declared unsupported MSA argument ABI", async () => { + const { provider } = await installation(); + const altered = { + ...target, + mips: { ...target.mips, gnuAttributes: { fpAbi: 2, msa: 1 } }, + } as BinaryTarget; + expect(ghidraProcessorUnsupportedReason(altered)).toMatch(/MSA ABI/u); + expect(provider.inspectTargetSupport(altered).status).toBe("unsupported"); + }); }, ); +/** + * A PSPSDK-shaped ELF32 whose PT_MIPS_ABIFLAGS record matches the PSP profile. + * Optionally declare a section-header table beyond the captured file. + */ +const pspElfBytes = ({ sectionHeadersBeyondEof = false } = {}): Buffer => { + const abiOffset = 8192; + const bytes = Buffer.alloc(abiOffset + 24); + bytes.set([0x7f, 0x45, 0x4c, 0x46, 1, 1, 1]); + bytes.writeUInt16LE(2, 16); + bytes.writeUInt16LE(8, 18); + bytes.writeUInt32LE(1, 20); + bytes.writeUInt32LE(52, 28); + bytes.writeUInt32LE(0x10a23001, 36); + bytes.writeUInt16LE(52, 40); + bytes.writeUInt16LE(32, 42); + bytes.writeUInt16LE(1, 44); + if (sectionHeadersBeyondEof) { + bytes.writeUInt32LE(bytes.length + 64, 32); // e_shoff + bytes.writeUInt16LE(40, 46); // e_shentsize + bytes.writeUInt16LE(2, 48); // e_shnum + } + bytes.writeUInt32LE(0x70000003, 52); + bytes.writeUInt32LE(abiOffset, 56); + bytes.writeUInt32LE(24, 68); + bytes.writeUInt32LE(24, 72); + // GNU readelf -x .MIPS.abiflags, not bytes generated by REA. + bytes.set( + Buffer.from("000002000101000200000000000000000100000000000000", "hex"), + abiOffset, + ); + return bytes; +}; + describe.skipIf(process.platform === "win32")( "PSP inspected ABI declaration boundary", () => { + it("refuses a PSP image whose section headers lie beyond the captured file", async () => { + const { directory, provider } = await installation(); + const path = join(directory, "psp-truncated-sections.elf"); + await writeFile(path, pspElfBytes({ sectionHeadersBeyondEof: true })); + const parsed = await parseBinaryTarget(path); + if (!parsed.ok) throw parsed.error; + if ( + parsed.value.kind !== "executable" || + parsed.value.architecture !== "mips" + ) + throw new Error("PSP ELF was not recognized as MIPS"); + expect(parsed.value.mips.abiFlags).toEqual(pspAbiFlags); + expect(parsed.value.mips.limitations).toContainEqual( + expect.stringContaining("ABI declarations in unavailable headers"), + ); + expect(provider.inspectTargetSupport(parsed.value).status).toBe( + "unsupported", + ); + const result = await provider.resolveAnalysisProfile(parsed.value); + expect(result.ok).toBe(false); + if (result.ok) throw new Error("Incomplete ABI inspection was admitted"); + expect(result.error._tag).toBe("AnalysisUnsupportedTargetError"); + expect(result.error.message).toContain( + "MIPS ABI declaration inspection is incomplete", + ); + }); + it("retains independently inspected PSP ABI bytes through the real target reader", async () => { const { directory, provider } = await installation(); - const abiOffset = 8192; - const bytes = Buffer.alloc(abiOffset + 24); - bytes.set([0x7f, 0x45, 0x4c, 0x46, 1, 1, 1]); - bytes.writeUInt16LE(2, 16); - bytes.writeUInt16LE(8, 18); - bytes.writeUInt32LE(1, 20); - bytes.writeUInt32LE(52, 28); - bytes.writeUInt32LE(0x10a23001, 36); - bytes.writeUInt16LE(52, 40); - bytes.writeUInt16LE(32, 42); - bytes.writeUInt16LE(1, 44); - bytes.writeUInt32LE(0x70000003, 52); - bytes.writeUInt32LE(abiOffset, 56); - bytes.writeUInt32LE(24, 68); - bytes.writeUInt32LE(24, 72); - // GNU readelf -x .MIPS.abiflags, not bytes generated by REA. - bytes.set( - Buffer.from("000002000101000200000000000000000100000000000000", "hex"), - abiOffset, - ); + const bytes = pspElfBytes(); const path = join(directory, "psp-with-abi.elf"); await writeFile(path, bytes); const parsed = await parseBinaryTarget(path); @@ -386,6 +440,7 @@ describe.skipIf(process.platform === "win32")( byteOrder: "little", type: 2, flags: 0x10a23001, + gnuAttributes: null, abiFlags: { ...pspAbiFlags, [field]: value }, }, }; @@ -414,6 +469,7 @@ describe.skipIf(process.platform === "win32")( byteOrder: "little", type: 2, flags: 0x10a23001, + gnuAttributes: null, ...(abiFlags === undefined ? {} : { abiFlags }), }, }; diff --git a/tests/boundary/filesystem/harmony/application.test.ts b/tests/boundary/filesystem/harmony/application.test.ts index a50c440e9e..76fddc3740 100644 --- a/tests/boundary/filesystem/harmony/application.test.ts +++ b/tests/boundary/filesystem/harmony/application.test.ts @@ -20,7 +20,7 @@ import { artifactInventoryResultSchema } from "../../../../src/domain/artifactGr import { parseBinaryTarget } from "../../../../src/application/BinaryTargetResolver.js"; import { ZIP_PACKAGE_FORMATS, - ZIP_NON_ENTRY_TAIL_LIMITATION, + ZIP_NON_ENTRY_LIMITATION, } from "../../../../src/domain/zipPackageFormat.js"; const { runProviderAnalysis } = createDirectAnalysis({}); @@ -225,7 +225,7 @@ describe("HarmonyOS inventory coverage", () => { ); const inventory = await inventoryApplicationPackage(path); expect(inventory.normalized_result).toMatchObject({ - coverage_status: "complete", + coverage: { scope: "container-entries", status: "complete" }, }); const projection = requireDeterministicProjection( @@ -264,7 +264,7 @@ describe("HarmonyOS inventory coverage", () => { artifact_kind: "container", artifact_format: format, }); - expect(inventory.limitations).toContain(ZIP_NON_ENTRY_TAIL_LIMITATION); + expect(inventory.limitations).toContain(ZIP_NON_ENTRY_LIMITATION); }, ); diff --git a/tests/boundary/filesystem/hermesClientConfiguration.test.ts b/tests/boundary/filesystem/hermesClientConfiguration.test.ts index 218573d5c4..5923ebff68 100644 --- a/tests/boundary/filesystem/hermesClientConfiguration.test.ts +++ b/tests/boundary/filesystem/hermesClientConfiguration.test.ts @@ -114,18 +114,34 @@ describe("Hermes YAML shape diagnostics", () => { "!!omap [{rea: {command: rea, args: [mcp]}}]", "!!timestamp 2001-12-15", "!!binary cmVh", - ].flatMap((value) => [ - { - original: `${value}\n`, - problem: "The configuration root must be an object", - }, - { - original: `mcp_servers: ${value}\n`, - problem: '"mcp_servers" must be an object', - }, - ]), + ] + .flatMap((value) => [ + { + original: `${value}\n`, + problem: "The configuration root must be an object", + }, + { + original: `mcp_servers: ${value}\n`, + problem: '"mcp_servers" must be an object', + }, + ]) + .concat( + // An explicit null root is a value, not an empty file. + [ + "null\n", + "~\n", + "# Hermes configuration\nnull\n", + "--- ~\n", + "--- !!null\n", + '!!null ""\n', + "--- &empty\n", + ].map((original) => ({ + original, + problem: "The configuration root must be an object", + })), + ), )( - "blocks setup and uninstall before changes for tagged non-record YAML: $original", + "blocks setup and uninstall before changes for tagged or null non-record YAML: $original", async ({ original, problem }) => { const { home, client } = await hermesClient(); const cursor = join(home, ".cursor", "mcp.json"); @@ -271,17 +287,83 @@ mcp_servers: it("creates config.yaml when Hermes has no configuration yet", async () => { const { client } = await hermesClient(); + const environment = { REA_MCP_OUTPUT_SCHEMA_PROFILE: "off" }; expect( - (await configureClientConfiguration(client, {}, command)).status, + (await configureClientConfiguration(client, environment, command)).status, ).toBe("configured"); - expect(parseYaml(await readFile(client.configPath, "utf8"))).toEqual({ + expect( + parseYaml(await readFile(client.configPath, "utf8"), { version: "1.1" }), + ).toEqual({ mcp_servers: { - rea: { command: command[0], args: command.slice(1), enabled: true }, + rea: { + command: command[0], + args: command.slice(1), + env: environment, + enabled: true, + }, }, }); }); - it.each(["", "\n"])( + it.each(["off", "2026-10-14", "12:34:56"])( + "refuses an environment value %s that Hermes parses as a non-string", + async (value) => { + const { home, client } = await hermesClient(); + const original = `mcp_servers:\n rea:\n command: rea\n args: [mcp]\n env:\n CUSTOM_SETTING: ${value}\n`; + await writeFile(client.configPath, original); + expect( + await inspectClientConfiguration(client, {}, command), + ).toMatchObject({ + status: "invalid", + remediation: expect.stringContaining("CUSTOM_SETTING"), + }); + expect( + await configureClientConfiguration(client, {}, command), + ).toMatchObject({ status: "failed" }); + expect( + ( + await readClientRegistrationStatuses(home, "rea", { + environment: {}, + }) + ).find(({ client: name }) => name === "hermes"), + ).toMatchObject({ state: "invalid" }); + expect(await readFile(client.configPath, "utf8")).toBe(original); + await expect( + readFile(`${client.configPath}.rea.backup`, "utf8"), + ).rejects.toThrow(); + }, + ); + + // Comments and a bare document marker parse to a null document, like an + // empty file, and are kept ahead of the new registration. +}); + +describe("Hermes YAML empty configurations", () => { + it("rejects multiple YAML documents before treating an empty first root as absent", async () => { + const { client } = await hermesClient(); + const original = + "---\n---\nmcp_servers:\n rea:\n command: rea\n args: [mcp]\n"; + await writeFile(client.configPath, original); + expect(await inspectClientConfiguration(client, {}, command)).toMatchObject( + { status: "invalid" }, + ); + expect(await configureClientConfiguration(client, {}, command)).toEqual({ + status: "failed", + reason: "readback", + }); + expect(await readFile(client.configPath, "utf8")).toBe(original); + await expect( + readFile(`${client.configPath}.rea.backup`, "utf8"), + ).rejects.toThrow(); + }); + + it.each([ + "", + "\n", + "# Hermes configuration\n", + "---\n", + "---\n# Hermes configuration\n", + ])( "configures an empty config.yaml %j as a new document", async (original) => { const { home, client } = await hermesClient(); @@ -289,7 +371,10 @@ mcp_servers: expect( (await configureClientConfiguration(client, {}, command)).status, ).toBe("configured"); - expect(parseYaml(await readFile(client.configPath, "utf8"))).toEqual({ + const text = await readFile(client.configPath, "utf8"); + const kept = original.trimEnd(); + expect(text.slice(0, kept.length)).toBe(kept); + expect(parseYaml(text)).toEqual({ mcp_servers: { rea: { command: command[0], args: command.slice(1), enabled: true }, }, diff --git a/tests/boundary/filesystem/hopperLauncherSelection.test.ts b/tests/boundary/filesystem/hopperLauncherSelection.test.ts new file mode 100644 index 0000000000..5949ce23ea --- /dev/null +++ b/tests/boundary/filesystem/hopperLauncherSelection.test.ts @@ -0,0 +1,34 @@ +import { accessSync, constants, statSync } from "node:fs"; +import { mkdir, writeFile } from "node:fs/promises"; +import { dirname, join } from "node:path"; + +import { expect, it } from "vitest"; + +import { defaultHopperLauncherPath } from "../../../src/config/parseConfig.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +// A real system installation legitimately precedes this isolated user fixture. +const installedSystemHopper = (home: string): boolean => { + try { + const launcher = defaultHopperLauncherPath("darwin", home); + accessSync(launcher, constants.X_OK); + return statSync(launcher).isFile(); + } catch { + return false; + } +}; + +it.for(["Hopper Disassembler.app", "Hopper A.app"])( + "skips a directory launcher in %s and selects a real file", + async (name, context) => { + const home = await createTestTempDirectory("rea-hopper-selection-"); + if (installedSystemHopper(home)) context.skip(); + const invalid = join(home, "Applications", name, "Contents/MacOS/hopper"); + const valid = join(home, "Applications/Hopper Z.app/Contents/MacOS/hopper"); + await mkdir(invalid, { recursive: true }); + await mkdir(dirname(valid), { recursive: true }); + await writeFile(valid, "#!/bin/sh\nexit 0\n", { mode: 0o700 }); + + expect(defaultHopperLauncherPath("darwin", home)).toBe(valid); + }, +); diff --git a/tests/boundary/filesystem/installScript.test.ts b/tests/boundary/filesystem/installScript.test.ts index d7e7f78a30..861a9bdfce 100644 --- a/tests/boundary/filesystem/installScript.test.ts +++ b/tests/boundary/filesystem/installScript.test.ts @@ -186,6 +186,18 @@ describe("curl installer scenarios", { timeout: 20_000 }, () => { }); }); +describe("installer reported version", { timeout: 20_000 }, () => { + it("rejects an empty version from a successful installed command", async () => { + const fixture = await createFixture(); + await expect( + runInstaller(fixture, ["--version", "0.3.0"], { FAKE_REA_VERSION: "" }), + ).rejects.toMatchObject({ + stderr: + "REA installation failed: the installed REA version could not be read. Reinstall the requested version, then retry.\n", + }); + }); +}); + describe("installer TTY fallback", { timeout: 20_000 }, () => { it("skips automatic setup when stderr is redirected", async () => { const fixture = await createFixture(); @@ -392,6 +404,20 @@ describe("native macOS installer", { timeout: 20_000 }, () => { }); describe("installer semantic version parsing", { timeout: 20_000 }, () => { + it("rejects an explicitly empty version instead of selecting latest", async () => { + const fixture = await createFixture(); + await expect( + runInstaller(fixture, ["--version", "", "--dry-run"]), + ).rejects.toMatchObject({ + stdout: "", + stderr: + "REA installation failed: version must be an exact semantic version.\n", + }); + await expect(readFile(fixture.npmLog, "utf8")).rejects.toMatchObject({ + code: "ENOENT", + }); + }); + it.each([ "1.2.3-01", "1.2.3-$(printf unexpected)", @@ -559,7 +585,7 @@ if [ -n "\${FAKE_CURL_BODY:-}" ]; then printf '%s' "$FAKE_CURL_BODY"; else print `#!/bin/sh printf '%s\n' "$*" >> "$FAKE_REA_LOG" [ "\${FAKE_REA_VERSION_FAIL:-}" = "1" ] && exit 1 -[ "$1" = "--version" ] && printf '%s\n' "\${FAKE_REA_VERSION:-0.3.0}" +[ "$1" = "--version" ] && printf '%s\n' "\${FAKE_REA_VERSION-0.3.0}" exit 0 `, ), diff --git a/tests/boundary/filesystem/interfaceBuilderBundle.test.ts b/tests/boundary/filesystem/interfaceBuilderBundle.test.ts index 543248e635..1b2954b7b6 100644 --- a/tests/boundary/filesystem/interfaceBuilderBundle.test.ts +++ b/tests/boundary/filesystem/interfaceBuilderBundle.test.ts @@ -961,6 +961,33 @@ describe("bounded Interface Builder archive decoding", () => { ); }); + it("decodes an archive whose date the decoder cannot represent", async () => { + const root = await createTestTempDirectory("rea-ib-test-"); + const bundle = join(root, "Example.app"); + const resources = join(bundle, "Contents", "Resources"); + await mkdir(resources, { recursive: true }); + // plutil writes years past 9999 unsigned, which the decoder rejects. + await writeFile( + join(resources, "Far.nib"), + '$archiverNSKeyedArchiver$objects$null11507-08-17T05:20:00Z$top', + ); + + const result = await analyzeInterfaceBuilderBundle({ + bundlePath: bundle, + targetSha256: "f".repeat(64), + }); + + expect(result.documents).toHaveLength(1); + expect(result.limitations).toContainEqual( + expect.stringMatching( + /^Contents\/Resources\/Far\.nib: 1 date value\(s\)/u, + ), + ); + expect(result.limitations).not.toContainEqual( + expect.stringContaining("could not be decoded"), + ); + }); + it.skipIf(!xcodeIbtoolAvailable)( "decodes an Xcode-compiled storyboard NIB and recovers its UI routes", async () => { diff --git a/tests/boundary/filesystem/javascript-applications/electronApplicationAnalysis.test.ts b/tests/boundary/filesystem/javascript-applications/electronApplicationAnalysis.test.ts index 795075c99d..d04be62ee8 100644 --- a/tests/boundary/filesystem/javascript-applications/electronApplicationAnalysis.test.ts +++ b/tests/boundary/filesystem/javascript-applications/electronApplicationAnalysis.test.ts @@ -67,22 +67,18 @@ describe("static Electron application analysis", () => { }); }); + it( + "keeps Electron runtime entrypoints on their declared filenames", + exactRuntimeEntrypointFixture, + ); + it("keeps computed identifier member keys unknown", async () => { const root = await computedMemberFixtureDirectory(); const result = await reconstructJavaScriptArtifact({ input_path: root }); const graph = parseJavaScriptApplicationGraph(result.graph); - // Computed access keeps no known members; the unknown access is an - // explicit dynamic flag rather than a "*" sentinel inside members. - const accesses = graph.nodes.flatMap((node) => - node.kind === "native-export" - ? node.observations.map(({ properties }) => ({ - members: Array.isArray(properties.requested_members) - ? properties.requested_members - : [], - dynamicMemberAccess: properties.dynamic_member_access ?? false, - })) - : [], + const nativeBindings = graph.nodes.filter( + (node) => node.kind === "native-export", ); expect(result.electron_summary).toMatchObject({ @@ -90,7 +86,20 @@ describe("static Electron application analysis", () => { native_addon_bindings: 1, resolved_native_addon_bindings: 1, }); - expect(accesses).toEqual([{ members: [], dynamicMemberAccess: true }]); + expect(nativeBindings).toMatchObject([ + { + observations: [ + { + properties: { + requested_members: [], + namespace_access: false, + dynamic_member_access: true, + native_export_verification: "not-performed", + }, + }, + ], + }, + ]); }); it("maps Electron boundaries through bundler-renamed bindings", async () => { @@ -153,9 +162,210 @@ describe("static Electron application analysis", () => { }); }); +const exactRuntimeEntrypointFixture = async (): Promise => { + const root = await createTestTempDirectory("rea-electron-exact-entrypoints-"); + await mkdir(join(root, "renderer/page"), { recursive: true }); + await Promise.all([ + writeFile( + join(root, "main.cjs"), + String.raw` +const { BrowserWindow, utilityProcess } = require("electron"); +const path = require("node:path"); +new BrowserWindow({ webPreferences: { preload: "./entry" } }); +new BrowserWindow().loadURL("file:///renderer/index.html"); +new BrowserWindow().loadURL("renderer/bare.html"); +new BrowserWindow().loadURL("#home"); +new BrowserWindow().loadURL(""); +new BrowserWindow().loadURL("./renderer/other.html?theme=dark#home"); +new BrowserWindow().loadURL("./renderer/missing.html"); +new BrowserWindow().loadFile("./renderer/page"); +new BrowserWindow({ webPreferences: { preload: path.join(__dirname, "preload") } }); +new BrowserWindow({ webPreferences: { preload: "./explicit-preload.js" } }); +utilityProcess.fork("./worker"); +utilityProcess.fork("./worker#v1.js"); +utilityProcess.fork("./service.js"); +`, + ), + writeFile(join(root, "entry.html"), "sibling"), + writeFile(join(root, "preload.html"), "sibling"), + writeFile(join(root, "worker.js"), "module.exports = {};"), + writeFile(join(root, "renderer/index.html"), "renderer"), + writeFile( + join(root, "renderer/bare.html"), + "bare renderer URL", + ), + writeFile(join(root, "renderer/other.html"), "other"), + writeFile(join(root, "renderer/missing.html.js"), "module.exports = {};"), + writeFile( + join(root, "renderer/page/index.html"), + "directory", + ), + writeFile(join(root, "explicit-preload.js"), "module.exports = {};"), + writeFile(join(root, "service.js"), "module.exports = {};"), + ]); + + const result = await reconstructJavaScriptArtifact({ input_path: root }); + const graph = parseJavaScriptApplicationGraph(result.graph); + expectSuffixOnlyRenderersUnresolved(graph); + expectExactRendererPaths(graph); + const roles = graph.nodes.filter(({ kind }) => kind === "electron-preload"); + const roleByDeclaration = new Map( + roles.map((role) => [role.observations[0]?.properties.declared_path, role]), + ); + + expect(result.electron_summary.resolved_utility_entrypoints).toBe(2); + for (const [declaredPath, candidatePath] of [ + ["./entry", "entry"], + ["preload", "preload"], + ]) { + const role = roleByDeclaration.get(declaredPath); + expect(role?.observations[0]?.properties).toMatchObject({ + declared_path: declaredPath, + resolved_path: null, + resolution_status: "not-found", + candidate_path: candidatePath, + limitations: [ + expect.stringContaining( + "extension and directory fallback were not attempted", + ), + ], + }); + expect( + graph.edges.some( + (edge) => + edge.source_node_id === role?.node_id && edge.relation === "maps_to", + ), + ).toBe(false); + } + expect( + roleByDeclaration.get("./explicit-preload.js")?.observations[0]?.properties, + ).toMatchObject({ + resolved_path: "explicit-preload.js", + resolution_status: "resolved", + }); + + const utilityLaunches = graph.edges.filter( + (edge) => + edge.evidence.extractor.operation === "map-utility-process-launch", + ); + const resolvedWorker = utilityLaunches.find( + (edge) => edge.properties.declared_path === "./worker", + ); + expect(resolvedWorker?.evidence).toMatchObject({ + authority: "static-relationship-inference", + state: "inferred", + limitations: expect.arrayContaining([ + "Static syntax does not prove that this relationship executes at runtime.", + ]), + }); + expect(utilityLaunches).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + properties: expect.objectContaining({ + declared_path: "./worker", + resolved_path: "worker.js", + resolution_status: "resolved", + }), + }), + expect.objectContaining({ + properties: expect.objectContaining({ + declared_path: "./worker#v1.js", + resolved_path: null, + resolution_status: "not-found", + }), + }), + expect.objectContaining({ + properties: expect.objectContaining({ + declared_path: "./service.js", + resolved_path: "service.js", + resolution_status: "resolved", + }), + }), + ]), + ); + expect( + graph.edges.some( + (edge) => + edge.evidence.extractor.operation === + "resolve-utility-process-entrypoint" && + edge.properties.resolved_path === "worker.html", + ), + ).toBe(false); +}; + type Reconstruction = Awaited>; type ApplicationGraph = ReturnType; +const expectSuffixOnlyRenderersUnresolved = (graph: ApplicationGraph): void => { + const renderers = new Map( + graph.nodes + .filter(({ kind }) => kind === "electron-renderer") + .map((renderer) => [ + renderer.observations[0]?.properties.declared_path, + renderer, + ]), + ); + expect( + renderers.get("file:///renderer/index.html")?.observations[0]?.properties, + ).toMatchObject({ + resolved_path: "renderer/index.html", + resolution_status: "resolved", + }); + for (const declaredPath of [ + "./renderer/other.html?theme=dark#home", + "renderer/bare.html", + "./renderer/missing.html", + "#home", + "", + ]) { + const renderer = renderers.get(declaredPath); + expect(renderer?.observations[0]?.properties).toMatchObject({ + resolved_path: null, + resolution_status: "unavailable", + }); + expect(renderer?.observations[0]?.properties.limitations).toEqual( + expect.arrayContaining([ + expect.stringContaining("requires a URL protocol"), + ]), + ); + expect( + graph.edges.some( + (edge) => + edge.source_node_id === renderer?.node_id && + edge.relation === "maps_to", + ), + ).toBe(false); + } +}; + +const expectExactRendererPaths = (graph: ApplicationGraph): void => { + const renderers = new Map( + graph.nodes + .filter(({ kind }) => kind === "electron-renderer") + .map((renderer) => [ + renderer.observations[0]?.properties.declared_path, + renderer, + ]), + ); + for (const [declaredPath, candidatePath] of [ + ["./renderer/page", "renderer/page"], + ]) { + const renderer = renderers.get(declaredPath); + expect(renderer?.observations[0]?.properties).toMatchObject({ + resolved_path: null, + resolution_status: "not-found", + candidate_path: candidatePath, + }); + expect( + graph.edges.some( + (edge) => + edge.source_node_id === renderer?.node_id && + edge.relation === "maps_to", + ), + ).toBe(false); + } +}; + const expectElectronSummary = (result: Reconstruction): void => { expect(result.electron_summary).toMatchObject({ browser_windows: 3, diff --git a/tests/boundary/filesystem/javascript-applications/javascriptApplicationEmptyLiterals.test.ts b/tests/boundary/filesystem/javascript-applications/javascriptApplicationEmptyLiterals.test.ts index 0490ae2116..a575e41a8a 100644 --- a/tests/boundary/filesystem/javascript-applications/javascriptApplicationEmptyLiterals.test.ts +++ b/tests/boundary/filesystem/javascript-applications/javascriptApplicationEmptyLiterals.test.ts @@ -79,6 +79,10 @@ it("retains empty literal evidence across the application boundary surfaces", as ); }); +/** Endpoint identity keys tag literal values apart from candidates. */ +const literalEndpointKey = (value: string): string => + JSON.stringify(["literal", value]); + it("keeps bridge entry paths stable through worker validation and API seed matching", async () => { const output = await analyzeSource(` const { contextBridge, ipcRenderer } = require("electron"); @@ -144,7 +148,9 @@ it("analyzes a plain long-line loader without package metadata", async () => { for (const node of nodes) { expect(node.identity).toMatchObject({ strategy: "artifact-local-key", - key: "", + // A Worker string is relative to the unestablished document base, so + // its identity is the retained expression rather than a specifier. + key: node.kind === "endpoint" ? literalEndpointKey("") : 'new Worker("")', }); expect(node.observations.map(({ label }) => label)).toEqual([null]); } @@ -165,9 +171,9 @@ it("keeps exact values for empty literals without synthetic application labels", // a literal two-quote endpoint; both keep exact identities. expect(endpoints).toEqual( expect.arrayContaining([ - { key: "", labels: [null], values: [""] }, - { key: '""', labels: ['""'], values: ['""'] }, - { key: "/api", labels: ["/api"], values: ["/api"] }, + { key: literalEndpointKey(""), labels: [null], values: [""] }, + { key: literalEndpointKey('""'), labels: ['""'], values: ['""'] }, + { key: literalEndpointKey("/api"), labels: ["/api"], values: ["/api"] }, ]), ); expect(endpoints).toHaveLength(3); @@ -252,7 +258,7 @@ it("does not match an empty endpoint with a literal two-quote seed", async () => case_sensitive: true, }).filter(({ node_id: nodeId }) => endpoints.has(nodeId)); expect(matches.map(({ node_id: nodeId }) => endpoints.get(nodeId))).toEqual([ - '""', + literalEndpointKey('""'), ]); }); diff --git a/tests/boundary/filesystem/javascript-applications/javascriptApplicationTemplateLiterals.test.ts b/tests/boundary/filesystem/javascript-applications/javascriptApplicationTemplateLiterals.test.ts index 55807d6de7..d4774afa8c 100644 --- a/tests/boundary/filesystem/javascript-applications/javascriptApplicationTemplateLiterals.test.ts +++ b/tests/boundary/filesystem/javascript-applications/javascriptApplicationTemplateLiterals.test.ts @@ -54,8 +54,10 @@ const graphIdentities = async (source: string) => { // source locations and every derived identity stay comparable. it.each([ "fetch('https://api.example.com/v1/users');", + "client['get']('/api/items');", + "navigator['sendBeacon']('/api/events', payload);", "localStorage.setItem('token', 1);", - "new Worker('./worker.js');", + "new Worker(new URL('./worker.js', import.meta.url));", "const dep = require('./dep.js'); dep();", "import('./dep.js');", ])( diff --git a/tests/boundary/filesystem/javascript-applications/javascriptArtifactGraphEndpoints.test.ts b/tests/boundary/filesystem/javascript-applications/javascriptArtifactGraphEndpoints.test.ts new file mode 100644 index 0000000000..2976625e4e --- /dev/null +++ b/tests/boundary/filesystem/javascript-applications/javascriptArtifactGraphEndpoints.test.ts @@ -0,0 +1,76 @@ +import { rm, writeFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { expect, it } from "vitest"; + +import { createTestTempDirectory } from "../../../fixtures/temporaryDirectory.js"; +import { scanCanonicalArtifactInventory } from "../../../fixtures/artifactInventory.js"; +import { readJavaScriptArtifactFiles } from "../../../support/javascriptApplicationScope.js"; +import { createJavaScriptArtifactReader } from "../../../../src/artifacts/javascript/JavaScriptArtifactReader.js"; +import { analyzeJavaScriptArtifactFiles } from "../../../../src/application/javascript/JavaScriptArtifactAnalysis.js"; +import { buildJavaScriptArtifactGraph } from "../../../../src/application/javascript/JavaScriptArtifactGraphBuilder.js"; + +it("keeps ambiguous endpoint candidates inferred through artifact graph projection", async () => { + const root = await createTestTempDirectory("rea-js-endpoint-certainty-"); + await writeFile(join(root, "app.js"), 'form.get("shared"); fetch("shared");'); + const snapshot = await scanCanonicalArtifactInventory(root, {}); + const reader = createJavaScriptArtifactReader(root, "directory"); + try { + const files = await readJavaScriptArtifactFiles(reader, snapshot); + const analysis = analyzeJavaScriptArtifactFiles(files); + const graph = buildJavaScriptArtifactGraph(snapshot, files, analysis); + const endpoints = graph.nodes.filter(({ kind }) => kind === "endpoint"); + const candidate = endpoints.find(({ observations }) => + observations.some( + ({ properties }) => properties.interpretation === "candidate", + ), + ); + const observed = endpoints.find(({ observations }) => + observations.some( + ({ properties }) => + properties.value === "shared" && + properties.interpretation === undefined, + ), + ); + + expect(candidate).toBeDefined(); + expect(observed).toBeDefined(); + expect(candidate?.node_id).not.toBe(observed?.node_id); + const candidateObservation = candidate?.observations[0]; + expect(candidateObservation?.properties).toMatchObject({ + value: "shared", + interpretation: "candidate", + }); + expect(candidateObservation?.evidence).toMatchObject({ + authority: "static-relationship-inference", + state: "inferred", + confidence: "low", + location: { + available: true, + value: { kind: "source-range", source: "app.js" }, + }, + }); + expect(candidateObservation?.evidence.limitations).toEqual( + expect.arrayContaining([expect.stringContaining("only a candidate")]), + ); + expect(observed?.observations[0]?.evidence).toMatchObject({ + authority: "ast-static-analysis", + state: "observed", + confidence: "exact", + }); + const candidateEdge = graph.edges.find( + ({ target_node_id }) => target_node_id === candidate?.node_id, + ); + expect(candidateEdge?.properties.interpretation).toBe("candidate"); + expect(candidateEdge?.evidence).toMatchObject({ + state: "inferred", + confidence: "low", + }); + expect(candidateEdge?.evidence.limitations).toEqual( + expect.arrayContaining([expect.stringContaining("only a candidate")]), + ); + } finally { + await reader.close(); + await rm(root, { recursive: true, force: true }); + } +}); diff --git a/tests/boundary/filesystem/javascript-applications/javascriptArtifactReconstruction.test.ts b/tests/boundary/filesystem/javascript-applications/javascriptArtifactReconstruction.test.ts index 416337097f..47c6a93c6a 100644 --- a/tests/boundary/filesystem/javascript-applications/javascriptArtifactReconstruction.test.ts +++ b/tests/boundary/filesystem/javascript-applications/javascriptArtifactReconstruction.test.ts @@ -361,7 +361,7 @@ it("reconstructs package, Electron roles, Webpack/Rspack modules, and cross-laye expect.objectContaining({ properties: expect.objectContaining({ module_key: "1", - structural_fingerprint_algorithm: "babel-ast-v1", + structural_fingerprint_algorithm: "babel-ast-v2", }), }), ]), @@ -502,10 +502,72 @@ it("keeps ASAR analysis usable when unpacked native companion bytes are absent", "ASAR unpacked companion bytes were unavailable; no content hash or child artifact was produced.", ]), }); + if (missingNative === undefined) + throw new Error("Inventory omitted the unavailable native companion"); - await reconstructAsarArchive(archive, [ + const result = await reconstructAsarArchive(archive, [ expect.objectContaining({ kind: "javascript-asset" }), ]); + const missingEntry = result.graph.nodes.find( + ({ kind, observations }) => + kind === "asar-entry" && + observations.some(({ label }) => label === missingNative.logical_path), + ); + expect(missingEntry).toMatchObject({ + identity: { + strategy: "canonical-path", + stability: "artifact-version", + artifact_sha256: snapshot.manifest.root_sha256, + path: missingNative.logical_path, + }, + observations: [ + expect.objectContaining({ + label: missingNative.logical_path, + properties: expect.objectContaining({ + entry_sha256: null, + bytes: missingNative.declared_size, + inventory_artifact_id: null, + inventory_occurrence_id: missingNative.occurrence_id, + hash_status: "unavailable", + artifact_kind: "native-addon", + kind: "native-addon", + content_available: false, + }), + evidence: expect.objectContaining({ + limitations: expect.arrayContaining([ + "ASAR unpacked companion bytes were unavailable; no content hash or child artifact was produced.", + "The inventoried native-addon entry has unavailable bytes; no content artifact identity can be established.", + ]), + }), + }), + ], + }); + const unavailableAddon = result.graph.nodes.find( + ({ kind, observations }) => + kind === "native-addon" && + observations.some(({ label }) => label === missingNative.logical_path), + ); + expect(unavailableAddon).toMatchObject({ + identity: { + strategy: "canonical-path", + stability: "artifact-version", + artifact_sha256: snapshot.manifest.root_sha256, + path: missingNative.logical_path, + }, + observations: [ + expect.objectContaining({ + label: missingNative.logical_path, + properties: expect.objectContaining({ + availability: "unavailable", + hash_status: "unavailable", + artifact_kind: "native-addon", + bytes: missingNative.declared_size, + inventory_artifact_id: null, + inventory_occurrence_id: missingNative.occurrence_id, + }), + }), + ], + }); }); it("recurses into filesystem-backed ASAR containers without losing container-relative paths", async () => { diff --git a/tests/boundary/filesystem/javascript-applications/javascriptFileUrlPaths.test.ts b/tests/boundary/filesystem/javascript-applications/javascriptFileUrlPaths.test.ts index 3597c45033..89c582c508 100644 --- a/tests/boundary/filesystem/javascript-applications/javascriptFileUrlPaths.test.ts +++ b/tests/boundary/filesystem/javascript-applications/javascriptFileUrlPaths.test.ts @@ -21,6 +21,12 @@ const urlCases = [ ] as const; const unknownExpressions = [ + 'fileURLToPath("file:///app/%2Fpreload.cjs")', + 'fileURLToPath("file:///app/%5Cpreload.cjs")', + 'fileURLToPath("file:///app/%")', + 'fileURLToPath("file:///C:/app/preload.cjs")', + 'fileURLToPath("file://remote/share/preload.cjs")', + 'fileURLToPath("file:///app/preload.cjs", { windows: selectedPlatform })', 'fileURLToPath(new URL("./invalid%.cjs", import.meta.url))', 'fileURLToPath(new URL("./preload%2fname.cjs", import.meta.url))', 'fileURLToPath(new URL("./preload%5cname.cjs", import.meta.url))', @@ -34,7 +40,7 @@ const unknownExpressions = [ 'convertFileURLToPath(new URL("./preload.cjs", import.meta.url))', ]; -describe("source-relative file URL paths", () => { +describe("JavaScript file URL paths", () => { it.each(urlCases)( "resolves %s to the file read by Node", async (reference, target, sourcePath) => { @@ -79,6 +85,36 @@ describe("source-relative file URL paths", () => { }, ); + it("resolves a decoded literal file URL at the filesystem boundary", async () => { + const root = await createTestTempDirectory("rea-literal-file-url-"); + await mkdir(join(root, "app"), { recursive: true }); + await writeFile( + join(root, "app", "preload name.cjs"), + 'module.exports = "literal file URL preload";\n', + ); + await writeFile( + join(root, "main.mjs"), + `import { fileURLToPath } from "node:url";\nimport { BrowserWindow } from "electron";\nnew BrowserWindow({ webPreferences: { preload: fileURLToPath("file:///app/preload%20name.cjs") } });\n`, + ); + + const result = await analyzeJavaScriptApplication({ + input_path: root, + format: "directory", + }); + if (!result.ok) throw result.error; + const { graph } = javascriptApplicationAnalysisResultSchema.parse( + result.value.normalized_result, + ); + const preload = graph.nodes.find(({ kind }) => kind === "electron-preload"); + expect(preload?.observations).not.toHaveLength(0); + for (const observation of preload?.observations ?? []) + expect(observation.properties).toMatchObject({ + resolved_path: "app/preload name.cjs", + resolution_status: "resolved", + resolution_context: "filesystem-expression", + }); + }); + it.each(unknownExpressions)("leaves unsupported %s unknown", (expression) => { expect( staticPath(parseExpression(expression, { sourceType: "module" })), diff --git a/tests/boundary/filesystem/javascript-applications/javascriptModuleRelationships.test.ts b/tests/boundary/filesystem/javascript-applications/javascriptModuleRelationships.test.ts index cdd95bc979..860424c6df 100644 --- a/tests/boundary/filesystem/javascript-applications/javascriptModuleRelationships.test.ts +++ b/tests/boundary/filesystem/javascript-applications/javascriptModuleRelationships.test.ts @@ -1,3 +1,4 @@ +import { createHash } from "node:crypto"; import { writeFile } from "node:fs/promises"; import { join } from "node:path"; @@ -5,6 +6,7 @@ import { describe, expect, it } from "vitest"; import { createTestTempDirectory } from "../../../fixtures/temporaryDirectory.js"; import { parseJavaScriptApplicationGraph } from "../../../../src/domain/javascript/javascriptApplicationGraph.js"; +import { compareJavaScriptApplicationVersions } from "../../../../src/domain/javascript/javascriptApplicationVersionComparison.js"; import { findExportNode, @@ -14,7 +16,87 @@ import { writeFixtureFiles, } from "../../../support/javascriptApplicationFixture.js"; +describe("cross-version source module identity", () => { + it("keeps a source module unchanged when only a sibling file changes", async () => { + const root = await createTestTempDirectory("rea-source-module-sibling-"); + const versions = await Promise.all( + ["left", "right"].map(async (name) => { + const path = join(root, name); + await writeFixtureFiles(path, { + "main.js": 'require("./util.js");\n', + "util.js": `module.exports = ${JSON.stringify(name)};\n`, + }); + return reconstructJavaScriptFixture(path); + }), + ); + const [left, right] = versions; + if (left === undefined || right === undefined) + throw new Error("Missing JavaScript application version"); + const leftModule = findSourceModule(left.graph, "main.js"); + const rightModule = findSourceModule(right.graph, "main.js"); + if (leftModule === undefined || rightModule === undefined) + throw new Error("Missing main source module"); + const leftObservation = leftModule.observations.find( + ({ properties }) => properties.semantic_role === "source-module", + ); + const rightObservation = rightModule.observations.find( + ({ properties }) => properties.semantic_role === "source-module", + ); + expect(leftObservation?.properties).not.toHaveProperty("container_sha256"); + expect(rightObservation?.properties).not.toHaveProperty("container_sha256"); + expect(leftObservation?.properties).toEqual(rightObservation?.properties); + expect(leftObservation?.properties.source_sha256).toBe( + rightObservation?.properties.source_sha256, + ); + + const comparison = compareJavaScriptApplicationVersions({ + left: { + evidenceId: `ev_${"a".repeat(64)}`, + rootArtifactSha256: left.root_artifact_sha256, + graph: left.graph, + }, + right: { + evidenceId: `ev_${"b".repeat(64)}`, + rootArtifactSha256: right.root_artifact_sha256, + graph: right.graph, + }, + leftNativeEvidence: [], + rightNativeEvidence: [], + }); + + expect( + comparison.items.find( + ({ left_node_id }) => left_node_id === leftModule.node_id, + ), + ).toMatchObject({ + status: "unchanged", + right_node_id: rightModule.node_id, + match: { + status: "matched", + basis: "exact-module-source-digest", + confidence: "exact", + }, + }); + }); +}); + describe("TypeScript source module loader semantics", () => { + it("publishes each source module's own content digest", async () => { + const root = await createTestTempDirectory("rea-source-module-digest-"); + const source = "module.exports = 42;\n"; + await writeFixtureFiles(root, { "main.js": source }); + + const { graph } = await reconstructJavaScriptFixture(root); + const sourceModule = findSourceModule(graph, "main.js"); + const digest = createHash("sha256").update(source).digest("hex"); + + expect( + sourceModule?.observations.find( + ({ properties }) => properties.semantic_role === "source-module", + )?.properties, + ).toMatchObject({ source_sha256: digest }); + }); + it("keeps whole type-only declarations out of runtime module relationships", async () => { const root = await createTestTempDirectory("rea-type-module-links-"); await writeFixtureFiles(root, { diff --git a/tests/boundary/filesystem/javascript-applications/javascriptRuntimeReconciliation.test.ts b/tests/boundary/filesystem/javascript-applications/javascriptRuntimeReconciliation.test.ts index a3b2231a2b..ca747c90c0 100644 --- a/tests/boundary/filesystem/javascript-applications/javascriptRuntimeReconciliation.test.ts +++ b/tests/boundary/filesystem/javascript-applications/javascriptRuntimeReconciliation.test.ts @@ -8,6 +8,9 @@ import { createTestTempDirectory } from "../../../fixtures/temporaryDirectory.js import { createElectronActiveEvidence } from "../../../../src/application/javascript/ElectronActiveEvidence.js"; import { reconcileJavaScriptRuntimeEvidence } from "../../../../src/application/javascript/JavaScriptRuntimeReconciliationService.js"; import { createEvidence } from "../../../../src/domain/evidence.js"; +import { mappedRuntimePaths } from "../../../../src/domain/javascript/javascriptRuntimeStaticCandidates.js"; +import { projectRuntimeCaptures } from "../../../../src/domain/javascript/javascriptRuntimeReconciliationRuntime.js"; +import { electronPageInspectionSchema } from "../../../../src/domain/javascript/electronObservation.js"; import { parseRuntimeReconciliationInput, reconcileJavaScriptRuntime, @@ -419,3 +422,482 @@ it("imports an operator-provided cache layer through an explicit file mapping", sharedFile?.observations.map(({ properties }) => properties.path), ).toEqual(expect.arrayContaining(["app.js", "mapped/chunk.js"])); }); + +const sourceFreeRuntime = (runtimeRoot: string, scriptFile: string) => { + const runtime = electronRuntimeEvidence(runtimeRoot, "", { + scriptFile, + sourceIncluded: false, + includeWorker: false, + }); + const observation = electronPageInspectionSchema.parse( + runtime.normalized_result, + ); + for (const script of observation.scripts.items) + script.source = { included: false, reason: "not-requested" }; + return createEvidence(undefined, runtime.provider, { + operation: runtime.operation, + predicateType: runtime.predicate_type, + parameters: runtime.parameters, + result: observation, + confidence: runtime.confidence, + authority: runtime.authority, + }); +}; + +const reconcileScriptLocation = ( + application: Awaited>, + root: string, + scriptFile: string, + runtimeRoot = root, +) => { + return reconcileInput({ + static_layers: [ + { + role: "application", + analysis: application, + runtime_mappings: + runtimeRoot === root + ? [] + : [{ kind: "file-root", root: runtimeRoot, artifact_prefix: "" }], + }, + ], + runtime_observations: [sourceFreeRuntime(runtimeRoot, scriptFile)], + }).reconciliations.find(({ entity_kind }) => entity_kind === "script"); +}; + +it.skipIf(process.platform === "win32")( + "keeps distinct POSIX backslash and slash script locations separate", + async () => { + const root = await createTestTempDirectory("rea-backslash-runtime-"); + const literal = "literal\\name.js"; + await writeFile(join(root, literal), "export const marker = 'literal';\n"); + const application = await analyzeFixture(root); + expect( + reconcileScriptLocation(application, root, "literal/name.js"), + ).toMatchObject({ status: "unmatched" }); + expect(reconcileScriptLocation(application, root, literal)).toMatchObject({ + status: "matched", + basis: "artifact-path", + }); + }, +); + +it.skipIf(process.platform === "win32")( + "does not make POSIX literal and slash paths ambiguous, including Windows runtime mappings", + async () => { + const root = await createTestTempDirectory("rea-backslash-runtime-pair-"); + const literal = "literal\\name.js"; + await mkdir(join(root, "literal")); + await writeFile(join(root, literal), "export const marker = 'literal';\n"); + await writeFile( + join(root, "literal", "name.js"), + "export const marker = 'slash';\n", + ); + const application = await analyzeFixture(root); + const slash = reconcileScriptLocation(application, root, "literal/name.js"); + const backslash = reconcileScriptLocation(application, root, literal); + expect(slash).toMatchObject({ status: "matched", basis: "artifact-path" }); + expect(backslash).toMatchObject({ + status: "matched", + basis: "artifact-path", + }); + expect(slash?.static_node_id).not.toBe(backslash?.static_node_id); + // These recorded Windows paths exercise matching only, not native Windows IO. + expect( + reconcileScriptLocation( + application, + root, + "literal\\name.js", + "C:\\runtime", + ), + ).toMatchObject({ + status: "matched", + static_node_id: slash?.static_node_id, + }); + }, +); + +it.skipIf(process.platform === "win32")( + "preserves URL-prefix path semantics separately from logical artifact names", + async () => { + const root = await createTestTempDirectory("rea-backslash-runtime-url-"); + await writeFile( + join(root, "literal\\name.js"), + "export const marker = 42;\n", + ); + const analysis = await analyzeFixture(root); + const parsed = parseRuntimeReconciliationInput( + reconcileJavaScriptRuntimeInputSchema.parse({ + static_layers: [ + { + analysis, + role: "application", + runtime_mappings: [ + { + kind: "url-prefix", + prefix: "https://example.test/assets/", + artifact_prefix: "", + }, + ], + }, + ], + runtime_observations: [sourceFreeRuntime(root, "literal\\name.js")], + }), + ); + const layer = parsed.layers[0]; + const entity = projectRuntimeCaptures(parsed.captures).entities.find( + ({ kind }) => kind === "script", + ); + if (layer === undefined || entity === undefined) + throw new Error("Expected parsed static layer and runtime script"); + const mapped = (value: string) => + mappedRuntimePaths(layer, { + ...entity, + location: { kind: "url", value }, + }).map(({ path }) => path); + expect(mapped("https://example.test/assets/literal%5Cname.js")).toEqual([ + "literal\\name.js", + ]); + expect(mapped("https://example.test/assets/literal/name.js")).toEqual([ + "literal/name.js", + ]); + expect(mapped("https://example.test/assets/literal\\name.js")).toEqual([ + "literal/name.js", + ]); + expect(mapped("https://other.test/assets/literal/name.js")).toEqual([]); + expect(mapped("https://example.test/assets/%2e%2e/outside.js")).toEqual([]); + }, +); + +const rendererCapture = (runtimeRoot: string, filePath: string) => { + const runtime = sourceFreeRuntime(runtimeRoot, "main.cjs"); + const observation = electronPageInspectionSchema.parse( + runtime.normalized_result, + ); + observation.target.file_path = filePath; + for (const frame of observation.frames) frame.file_path = filePath; + return createEvidence(undefined, runtime.provider, { + operation: runtime.operation, + predicateType: runtime.predicate_type, + parameters: runtime.parameters, + result: observation, + confidence: runtime.confidence, + authority: runtime.authority, + }); +}; + +it.skipIf(process.platform === "win32")( + "interprets raw loadFile declarations using the captured native path format", + async () => { + const root = await createTestTempDirectory( + "rea-native-loadfile-declaration-", + ); + await mkdir(join(root, "dist")); + await writeFile(join(root, "dist", "index.html"), "slash file"); + await writeFile(join(root, "dist\\index.html"), "literal file"); + await writeFile( + join(root, "main.cjs"), + `const { BrowserWindow } = require('electron'); const win = new BrowserWindow(); win.loadFile(${JSON.stringify("dist\\index.html")});`, + ); + const analysis = await analyzeFixture(root); + const reconcileTarget = (runtimeRoot: string, path: string) => + reconcileInput({ + static_layers: [ + { + analysis, + role: "application", + runtime_mappings: [ + { kind: "file-root", root: runtimeRoot, artifact_prefix: "" }, + ], + }, + ], + runtime_observations: [rendererCapture(runtimeRoot, path)], + }).reconciliations.find(({ entity_kind }) => entity_kind === "target"); + // These Windows paths are recorded data, not native Windows execution. + expect( + reconcileTarget("C:\\runtime", "C:\\runtime\\dist\\index.html"), + ).toMatchObject({ status: "matched" }); + expect(reconcileTarget(root, join(root, "dist\\index.html"))).toMatchObject( + { status: "matched" }, + ); + expect( + reconcileTarget(root, join(root, "dist", "index.html")), + ).toMatchObject({ status: "unmatched" }); + }, +); + +it.skipIf(process.platform === "win32").each(["literal\\dir", "literal%5Cdir"])( + "maps a runtime root to the exact logical artifact prefix %j", + async (prefix) => { + const root = await createTestTempDirectory("rea-literal-artifact-prefix-"); + await mkdir(join(root, prefix)); + await writeFile( + join(root, prefix, "app.js"), + "export const marker = 42;\n", + ); + const analysis = await analyzeFixture(root); + const result = reconcileInput({ + static_layers: [ + { + analysis, + role: "application", + runtime_mappings: [ + { kind: "file-root", root: "/deployed", artifact_prefix: prefix }, + ], + }, + ], + runtime_observations: [sourceFreeRuntime("/deployed", "app.js")], + }); + expect( + result.reconciliations.find( + ({ entity_kind }) => entity_kind === "script", + ), + ).toMatchObject({ status: "matched", basis: "operator-file-mapping" }); + }, +); + +it.skipIf(process.platform === "win32")( + "uses resolved artifact paths instead of a raw declaration alias", + async () => { + const root = await createTestTempDirectory("rea-resolved-loadfile-alias-"); + const directory = "literal\\dir"; + await mkdir(join(root, directory)); + await writeFile(join(root, directory, "index.html"), "owned target"); + await writeFile(join(root, "index.html"), "different target"); + await writeFile( + join(root, directory, "package.json"), + JSON.stringify({ + name: "literal-renderer", + version: "1.0.0", + renderer: "./index.html", + }), + ); + const analysis = await analyzeFixture(root); + const reconcileTarget = (path: string) => + reconcileInput({ + static_layers: [{ analysis, role: "application" }], + runtime_observations: [rendererCapture(root, path)], + }).reconciliations.find(({ entity_kind }) => entity_kind === "target"); + expect(reconcileTarget(join(root, directory, "index.html"))).toMatchObject({ + status: "matched", + }); + expect(reconcileTarget(join(root, "index.html"))).toMatchObject({ + status: "unmatched", + }); + }, +); + +it("rejects unsafe logical artifact prefixes without decoding literal names", async () => { + const root = await createTestTempDirectory("rea-prefix-admission-"); + await writeFile(join(root, "app.js"), "export const marker = 42;\n"); + const analysis = await analyzeFixture(root); + const accepts = (artifact_prefix: string) => + reconcileJavaScriptRuntimeInputSchema.safeParse({ + static_layers: [ + { + analysis, + role: "application", + runtime_mappings: [ + { kind: "file-root", root: "/deployed", artifact_prefix }, + ], + }, + ], + runtime_observations: [sourceFreeRuntime("/deployed", "app.js")], + }).success; + for (const prefix of [ + "\0", + "/absolute", + ".", + "..", + "a/../b", + "a//b", + "a/", + "./a", + ]) + expect(accepts(prefix), prefix).toBe(false); + for (const prefix of ["", "literal\\dir", "literal%5Cdir", "literal%2Fdir"]) + expect(accepts(prefix), prefix).toBe(true); +}); + +it.skipIf(process.platform === "win32")( + "keeps a leading backslash declaration literal only for POSIX captures", + async () => { + const root = await createTestTempDirectory("rea-leading-loadfile-"); + await writeFile(join(root, "\\index.html"), "literal target"); + await writeFile( + join(root, "main.cjs"), + `const { BrowserWindow } = require('electron'); const win = new BrowserWindow(); win.loadFile(${JSON.stringify("\\index.html")});`, + ); + const analysis = await analyzeFixture(root); + const reconcileTarget = (runtimeRoot: string, path: string) => + reconcileInput({ + static_layers: [ + { + analysis, + role: "application", + runtime_mappings: [ + { kind: "file-root", root: runtimeRoot, artifact_prefix: "" }, + ], + }, + ], + runtime_observations: [rendererCapture(runtimeRoot, path)], + }).reconciliations.find(({ entity_kind }) => entity_kind === "target"); + expect(reconcileTarget(root, join(root, "\\index.html"))).toMatchObject({ + status: "matched", + }); + expect( + reconcileTarget("C:\\runtime", "C:\\runtime\\index.html"), + ).toMatchObject({ status: "unmatched" }); + }, +); + +it("keeps a raw nested loadFile target application-relative when the producer also resolves a nested decoy", async () => { + const root = await createTestTempDirectory("rea-raw-loadfile-ownership-"); + await mkdir(join(root, "dist")); + await mkdir(join(root, "sub", "dist"), { recursive: true }); + await writeFile(join(root, "dist", "index.html"), "application target"); + await writeFile(join(root, "sub", "dist", "index.html"), "nested decoy"); + await writeFile( + join(root, "sub", "main.cjs"), + "const { BrowserWindow } = require('electron'); const win = new BrowserWindow(); win.loadFile('dist/index.html');", + ); + const analysis = await analyzeFixture(root); + const reconcileTarget = (path: string) => + reconcileInput({ + static_layers: [{ analysis, role: "application" }], + runtime_observations: [rendererCapture(root, path)], + }).reconciliations.find(({ entity_kind }) => entity_kind === "target"); + expect(reconcileTarget(join(root, "dist", "index.html"))).toMatchObject({ + status: "matched", + }); + expect( + reconcileTarget(join(root, "sub", "dist", "index.html")), + ).toMatchObject({ status: "unmatched" }); +}); + +it + .skipIf(process.platform === "win32") + .each(["app", "literal\\app", "café", "cafe\u0301"])( + "maps raw renderer declarations through the logical application prefix %j", + async (prefix) => { + const root = await createTestTempDirectory("rea-prefixed-native-renderer-"); + await mkdir(join(root, prefix, "dist"), { recursive: true }); + await writeFile( + join(root, prefix, "dist", "index.html"), + "application target", + ); + await writeFile( + join(root, prefix, "main.cjs"), + "const { BrowserWindow } = require('electron'); const win = new BrowserWindow(); win.loadFile('dist/index.html');", + ); + const analysis = await analyzeFixture(root); + const reconcileTarget = (runtimeRoot: string, path: string) => + reconcileInput({ + static_layers: [ + { + analysis, + role: "application", + runtime_mappings: [ + { kind: "file-root", root: runtimeRoot, artifact_prefix: prefix }, + ], + }, + ], + runtime_observations: [rendererCapture(runtimeRoot, path)], + }).reconciliations.find(({ entity_kind }) => entity_kind === "target"); + expect( + reconcileTarget("/deployed", "/deployed/dist/index.html"), + ).toMatchObject({ status: "matched", basis: "operator-file-mapping" }); + expect( + reconcileTarget("C:\\deployed", "C:\\deployed\\dist\\index.html"), + ).toMatchObject({ status: "matched", basis: "operator-file-mapping" }); + expect(reconcileTarget("/deployed", "/deployed/index.html")).toMatchObject({ + status: "unmatched", + }); + }, +); + +it("confines raw renderer declarations to the operator-selected artifact prefix", async () => { + const root = await createTestTempDirectory("rea-native-prefix-isolation-"); + for (const app of ["appA", "appB"]) { + await mkdir(join(root, app, "dist"), { recursive: true }); + await writeFile(join(root, app, "dist", "index.html"), app); + await writeFile( + join(root, app, "main.cjs"), + "const { BrowserWindow } = require('electron'); const win = new BrowserWindow(); win.loadFile('dist/index.html');", + ); + } + const analysis = await analyzeFixture(root); + const result = reconcileInput({ + static_layers: [ + { + analysis, + role: "application", + runtime_mappings: [ + { kind: "file-root", root: "/deployed", artifact_prefix: "appA" }, + ], + }, + ], + runtime_observations: [ + rendererCapture("/deployed", "/deployed/dist/index.html"), + ], + }); + const target = result.reconciliations.find( + ({ entity_kind }) => entity_kind === "target", + ); + expect(target).toMatchObject({ + status: "matched", + candidate_static_count: 1, + }); + const node = result.graph.nodes.find( + ({ node_id }) => node_id === target?.static_node_id, + ); + expect( + node?.observations.flatMap(({ evidence }) => + evidence.location.available && + evidence.location.value.kind === "source-range" + ? [evidence.location.value.source] + : [], + ), + ).toEqual(["appA/main.cjs"]); +}); + +it("retains uncertainty for overlapping file mappings with distinct application interpretations", async () => { + const root = await createTestTempDirectory("rea-overlapping-native-prefix-"); + await mkdir(join(root, "app", "dist"), { recursive: true }); + await writeFile(join(root, "app", "dist", "index.html"), "shared target"); + for (const [path, declaration] of [ + ["app/main.cjs", "dist/index.html"], + ["app/dist/main.cjs", "index.html"], + ]) + await writeFile( + join(root, path!), + `const { BrowserWindow } = require('electron'); const win = new BrowserWindow(); win.loadFile(${JSON.stringify(declaration)});`, + ); + const analysis = await analyzeFixture(root); + const result = reconcileInput({ + static_layers: [ + { + analysis, + role: "application", + runtime_mappings: [ + { kind: "file-root", root: "/deployed", artifact_prefix: "app" }, + { + kind: "file-root", + root: "/deployed/dist", + artifact_prefix: "app/dist", + }, + ], + }, + ], + runtime_observations: [ + rendererCapture("/deployed", "/deployed/dist/index.html"), + ], + }); + expect( + result.reconciliations.find(({ entity_kind }) => entity_kind === "target"), + ).toMatchObject({ + status: "ambiguous", + confidence: "unknown", + candidate_static_count: 2, + }); +}); diff --git a/tests/boundary/filesystem/javascriptExportShapeComparison.test.ts b/tests/boundary/filesystem/javascriptExportShapeComparison.test.ts index 97090c029a..705fb7b187 100644 --- a/tests/boundary/filesystem/javascriptExportShapeComparison.test.ts +++ b/tests/boundary/filesystem/javascriptExportShapeComparison.test.ts @@ -306,7 +306,9 @@ describe("JavaScript export return-shape uncertain presence", () => { ).toBe("partial"); } }); +}); +describe("JavaScript export return-shape array presence", () => { it("excludes array holes while preserving observed unknown elements", async () => { const holes = 'export default () => ({ kind: "result", items: [,] });'; const empty = await analyzeSources({ @@ -331,6 +333,84 @@ describe("JavaScript export return-shape uncertain presence", () => { }), ); }); +}); + +describe("JavaScript export unpaired return locations", () => { + it.each([ + { + description: "two right return variants", + left: "export default function f() {}", + right: `export default function f(flag) { + if (flag) return { count: call() }; + return { total: call() }; + }`, + side: "right" as const, + lines: [2, 3], + }, + { + description: "one right return variant", + left: "export default function f() {}", + right: `export default function f() { + return { count: call() }; + }`, + side: "right" as const, + lines: [2], + }, + { + description: "two left return variants", + left: `export default function f(flag) { + if (flag) return { count: call() }; + return { total: call() }; + }`, + right: "export default function f() {}", + side: "left" as const, + lines: [2, 3], + }, + { + description: "one left return variant", + left: `export default function f() { + return { count: call() }; + }`, + right: "export default function f() {}", + side: "left" as const, + lines: [2], + }, + ])( + "retains each location for $description when the opposite export has no return", + async ({ left, right, side, lines }) => { + const graphs = await analyzeSources({ left, right }); + const result = compare(...graphs); + expect(result.coverage).toMatchObject({ + status: "partial", + unpaired_left_variants: side === "left" ? lines.length : 0, + unpaired_right_variants: side === "right" ? lines.length : 0, + }); + expect(result.changes).toHaveLength(lines.length); + expect( + result.changes.every( + ({ status, left_source_range, right_source_range }) => + status === "unknown" && + (side === "left" + ? left_source_range !== null && right_source_range === null + : left_source_range === null && right_source_range !== null), + ), + ).toBe(true); + const locations = result.changes + .map((change) => + side === "left" + ? change.left_source_range?.start.line + : change.right_source_range?.start.line, + ) + .sort((first, second) => (first ?? 0) - (second ?? 0)); + expect(locations).toEqual(lines); + expect(result.summary).toMatchObject({ + added: 0, + removed: 0, + changed: 0, + unknown: lines.length, + }); + }, + ); it("retains unpaired inventory locations when the opposite export has no return", async () => { const graphs = await analyzeSources({ diff --git a/tests/boundary/filesystem/javascriptHtmlEntrypoints.test.ts b/tests/boundary/filesystem/javascriptHtmlEntrypoints.test.ts index 29a9847df7..ee3f9ced0f 100644 --- a/tests/boundary/filesystem/javascriptHtmlEntrypoints.test.ts +++ b/tests/boundary/filesystem/javascriptHtmlEntrypoints.test.ts @@ -191,3 +191,185 @@ const htmlScriptRanges = (html: string) => { } return ranges; }; + +it("reports unanalyzed inline HTML scripts instead of claiming complete coverage", async () => { + const root = await createTestTempDirectory("rea-html-inline-scripts-"); + const html = [ + "", + '', + '', + "", + "", + '', + "", + ].join("\n"); + await mkdir(join(root, "www")); + await Promise.all([ + writeFile( + join(root, "package.json"), + JSON.stringify({ name: "game", version: "1.0.0" }), + ), + writeFile(join(root, "www", "index.html"), html), + writeFile(join(root, "www", "cordova.js"), "globalThis.cordova = {};"), + ]); + + const result = await reconstructJavaScriptArtifact({ input_path: root }); + const gaps = result.graph.nodes.filter( + ({ kind, observations }) => + kind === "unknown" && + observations.some( + ({ properties }) => + properties.operation === "parse-html-inline-scripts", + ), + ); + + expect(result.graph.coverage.status).toBe("partial"); + expect(gaps).toHaveLength(1); + expect(gaps[0]?.observations[0]).toMatchObject({ + properties: { + path: "www/index.html", + inline_script_count: 2, + inline_scripts: [ + { + script_type: "classic", + location: { + start: { line: 5, column: 8 }, + end: { line: 7, column: 0 }, + }, + }, + { + script_type: "module", + location: { + start: { line: 8, column: 22 }, + end: { line: 8, column: 67 }, + }, + }, + ], + }, + evidence: { + state: "unavailable", + artifact: { + sha256: createHash("sha256").update(html).digest("hex"), + }, + coverage: { status: "partial" }, + }, + }); + expect(result.graph.limitations).toContain( + "www/index.html has 2 inline JavaScript bodies starting at lines 5, 8 that were not analyzed; endpoints, storage, imports and other facts in them are absent from the evidence.", + ); + // The external script is still projected as before. + expect( + result.graph.edges.some( + ({ relation, properties }) => + relation === "loads" && properties.resolved_path === "www/cordova.js", + ), + ).toBe(true); +}); + +it("keeps complete coverage when HTML has only external scripts and data blocks", async () => { + const root = await createTestTempDirectory("rea-html-no-inline-scripts-"); + await Promise.all([ + writeFile( + join(root, "index.html"), + '', + ), + writeFile(join(root, "actual.js"), "globalThis.ownedActual = true;"), + ]); + + const result = await reconstructJavaScriptArtifact({ input_path: root }); + + expect(result.graph.coverage.status).toBe("complete"); + expect( + result.graph.limitations.some((limitation) => + limitation.includes("inline JavaScript"), + ), + ).toBe(false); +}); + +it.each(coordinateCases)( + "retains parse5 source coordinates for inline script bodies with $name", + async ({ prefix }) => { + const root = await createTestTempDirectory("rea-html-inline-coordinates-"); + const html = `${prefix}`; + await writeFile(join(root, "index.html"), html); + + const result = await reconstructJavaScriptArtifact({ input_path: root }); + const gap = result.graph.nodes + .flatMap(({ observations }) => observations) + .find( + ({ properties }) => + properties.operation === "parse-html-inline-scripts", + ); + + expect(gap?.properties.inline_scripts).toEqual([ + { script_type: "classic", location: inlineBodyRange(html) }, + ]); + }, +); + +const inlineBodyRange = (html: string) => { + const pending: DefaultTreeAdapterTypes.Node[] = [ + parse(html, { sourceCodeLocationInfo: true }), + ]; + while (pending.length > 0) { + const node = pending.pop(); + if (node === undefined) continue; + if (defaultTreeAdapter.isElementNode(node) && node.tagName === "script") { + const opening = node.sourceCodeLocation?.startTag; + const body = node.childNodes[0]?.sourceCodeLocation; + if (opening === undefined || body === undefined || body === null) + throw new Error("Missing parse5 inline script range"); + return { + start: { line: opening.endLine, column: opening.endCol - 1 }, + end: { line: body.endLine, column: body.endCol - 1 }, + }; + } + if ("childNodes" in node) { + for (const child of node.childNodes.toReversed()) pending.push(child); + } + } + throw new Error("Missing parse5 inline script"); +}; + +it("bounds retained inline script locations and names the omitted remainder", async () => { + const root = await createTestTempDirectory("rea-html-many-inline-scripts-"); + const count = 5_000; + await writeFile( + join(root, "index.html"), + "\n".repeat(count), + ); + + const result = await reconstructJavaScriptArtifact({ input_path: root }); + const gap = result.graph.nodes + .flatMap(({ observations }) => observations) + .find( + ({ properties }) => properties.operation === "parse-html-inline-scripts", + ); + const limit = { + name: "html_inline_script_locations_per_file", + value: 100, + unit: "items", + }; + + expect(gap?.properties.inline_script_count).toBe(count); + expect(gap?.properties.inline_scripts).toHaveLength(100); + // The gap record knows exactly which locations the bound dropped. + expect(gap?.evidence.coverage).toEqual({ + status: "partial", + truncated: true, + omitted_count: count - 100, + limits: [limit], + }); + // The graph is truncated too, but unanalyzed bodies hide an unknown total. + expect(result.graph.coverage).toMatchObject({ + status: "partial", + truncated: true, + omitted_count: null, + limits: expect.arrayContaining([limit]), + }); + expect(result.graph.limitations).toContain( + `index.html has ${String(count)} inline JavaScript bodies starting at lines 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 and ${String(count - 10)} more that were not analyzed; endpoints, storage, imports and other facts in them are absent from the evidence.`, + ); +}); diff --git a/tests/boundary/filesystem/javascriptRequireLiteralPaths.test.ts b/tests/boundary/filesystem/javascriptRequireLiteralPaths.test.ts index 65349b8e4f..ab47a41818 100644 --- a/tests/boundary/filesystem/javascriptRequireLiteralPaths.test.ts +++ b/tests/boundary/filesystem/javascriptRequireLiteralPaths.test.ts @@ -1,6 +1,7 @@ -import { writeFile } from "node:fs/promises"; +import { createHash } from "node:crypto"; +import { mkdir, writeFile } from "node:fs/promises"; import { createRequire } from "node:module"; -import { join, relative, sep } from "node:path"; +import { dirname, join, relative, sep } from "node:path"; import { describe, expect, it } from "vitest"; @@ -133,3 +134,101 @@ describe("literal CommonJS paths in application graphs", () => { }, ); }); + +describe("literal CommonJS percent bytes in application graphs", () => { + it("agrees with Node for requests whose literal names contain %2e or %2F", async () => { + const root = await createTestTempDirectory("rea-literal-percent-require-"); + const files = { + "app/lit%2eb.js": "module.exports = 'lit%2eb.js';", + "app/x%2Fy.js": "module.exports = 'x%2Fy.js';", + "app/sub/%2e%2e/x.js": "module.exports = 'sub/%2e%2e/x.js';", + "app/x.js": "module.exports = 'decoy for a decoded parent';", + "app/pkg/package.json": JSON.stringify({ main: "lit%2eb" }), + "app/pkg/lit%2eb.js": "module.exports = 'pkg/lit%2eb.js';", + "app/node_modules/dep/package.json": JSON.stringify({ name: "dep" }), + "app/node_modules/dep/lit%2eb.js": "module.exports = 'dep/lit%2eb.js';", + "escape.js": "module.exports = 'decoy outside the request directory';", + }; + const specifiers = [ + "./lit%2eb", + "./x%2Fy", + "./sub/%2e%2e/x", + "./pkg", + "dep/lit%2eb", + ]; + // CommonJS reads %2e%2e as a directory name, never as a parent segment. + const literalSegment = "./%2e%2e/escape"; + for (const [path, text] of Object.entries(files)) { + await mkdir(dirname(join(root, path)), { recursive: true }); + await writeFile(join(root, path), text); + } + const main = join(root, "app", "main.cjs"); + await writeFile( + main, + [...specifiers, literalSegment] + .map((specifier) => `require(${JSON.stringify(specifier)});`) + .join("\n"), + ); + const nodeRequire = createRequire(main); + const nodePath = (specifier: string) => + relative(root, nodeRequire.resolve(specifier)).split(sep).join("/"); + expect(() => nodeRequire.resolve(literalSegment)).toThrow( + /Cannot find module/u, + ); + + const { graph } = await analyze(root); + const requests = (specifier: string) => + graph.edges.filter( + (edge) => + edge.relation === "imports" && + edge.properties.specifier === specifier, + ); + for (const specifier of specifiers) { + const edges = requests(specifier); + expect(edges.length, specifier).toBeGreaterThan(0); + for (const edge of edges) + expect(edge.properties.resolved_path, specifier).toBe( + nodePath(specifier), + ); + } + const unresolved = requests(literalSegment); + expect(unresolved.length).toBeGreaterThan(0); + for (const edge of unresolved) + expect(edge.properties.resolved_path).toBeNull(); + }); +}); + +it("reports a present non-JavaScript require target as unavailable with inventory identity", async () => { + const present = await createTestTempDirectory("rea-require-opaque-present-"); + const absent = await createTestTempDirectory("rea-require-opaque-absent-"); + const bytes = Buffer.from([0xde, 0xc0, 0x17, 0x0b, 0x00, 0x01, 0x02]); + const main = 'require("./core.jsc");'; + await Promise.all([ + writeFile(join(present, "main.cjs"), main), + writeFile(join(present, "core.jsc"), bytes), + writeFile(join(absent, "main.cjs"), main), + ]); + const edge = async (root: string) => { + const { graph } = await analyze(root); + return graph.edges.find( + ({ properties }) => + properties.specifier === "./core.jsc" && + properties.module_link_kind === "require", + ); + }; + const presentEdge = await edge(present); + expect(presentEdge?.properties).toMatchObject({ + resolution_status: "unavailable", + candidate_path: "core.jsc", + resolved_path: null, + }); + expect(presentEdge?.evidence.limitations).toContainEqual( + expect.stringContaining( + `core.jsc exists in the inventoried artifact container (sha256 ${createHash("sha256").update(bytes).digest("hex")}, ${String(bytes.length)} bytes)`, + ), + ); + expect((await edge(absent))?.properties).toMatchObject({ + resolution_status: "not-found", + resolved_path: null, + }); +}); diff --git a/tests/boundary/filesystem/javascriptSourceMapLocations.test.ts b/tests/boundary/filesystem/javascriptSourceMapLocations.test.ts index 69a5af6621..21d5719e06 100644 --- a/tests/boundary/filesystem/javascriptSourceMapLocations.test.ts +++ b/tests/boundary/filesystem/javascriptSourceMapLocations.test.ts @@ -134,7 +134,9 @@ it("associates each source-map URL with its edge when aliases share one physical ); expect(mapEdges).toHaveLength(2); expect(mapEdges[0]?.target_node_id).toBe(mapEdges[1]?.target_node_id); - expect( + // Edge order follows content-derived edge IDs, so pair each edge with the + // script whose comment declared it. + const declaredBySource = new Map( mapEdges.map((edge) => { const target = nodes.get(edge.target_node_id); const observation = target?.observations.find( @@ -142,24 +144,37 @@ it("associates each source-map URL with its edge when aliases share one physical properties.declared_url_sha256 === edge.properties.declared_url_sha256, ); - return { - declared_url: observation?.properties.declared_url, - resolved_path: edge.properties.resolved_path, - }; + const location = edge.evidence.location; + return [ + location.available && location.value.kind === "source-range" + ? location.value.source + : null, + { + declared_url: observation?.properties.declared_url, + resolved_path: edge.properties.resolved_path, + }, + ]; }), - ).toEqual([ - { declared_url: "app.js.map", resolved_path: "dist/app.js.map" }, - { declared_url: "./app.js.map", resolved_path: "dist/app.js.map" }, - ]); + ); + expect(Object.fromEntries(declaredBySource)).toEqual({ + "dist/one.js": { + declared_url: "app.js.map", + resolved_path: "dist/app.js.map", + }, + "dist/two.js": { + declared_url: "./app.js.map", + resolved_path: "dist/app.js.map", + }, + }); }); -it("resolves a bare source-map file name and worker URL next to the script", async () => { +it("resolves a bare source-map file name and module worker URL next to the script", async () => { const root = await createTestTempDirectory("rea-source-map-relative-"); await mkdir(join(root, "dist")); await writeFile( join(root, "dist", "app.js"), [ - 'new Worker("worker.js");', + 'new Worker(new URL("worker.js", import.meta.url));', 'navigator.serviceWorker.register("sw.js");', "//# sourceMappingURL=app.js.map", ].join("\n"), @@ -191,6 +206,7 @@ it("resolves a bare source-map file name and worker URL next to the script", asy .update("app.js.map") .digest("hex"), resolved_path: "dist/app.js.map", + resolution_status: "resolved", }); expect( nodes @@ -212,12 +228,12 @@ it("resolves a bare source-map file name and worker URL next to the script", asy expect(workerEdge?.properties).toMatchObject({ resolved_path: "dist/worker.js", }); - const serviceWorkerEdge = result.graph.edges.find( - ({ relation, source_node_id }) => - relation === "maps_to" && - nodes.get(source_node_id)?.kind === "service-worker", - ); - expect(serviceWorkerEdge?.properties).toMatchObject({ - resolved_path: "dist/sw.js", - }); + // A service worker URL is relative to the unestablished document base. + expect( + result.graph.edges.some( + ({ relation, source_node_id }) => + relation === "maps_to" && + nodes.get(source_node_id)?.kind === "service-worker", + ), + ).toBe(false); }); diff --git a/tests/boundary/filesystem/keyedArchiveDateRange.test.ts b/tests/boundary/filesystem/keyedArchiveDateRange.test.ts new file mode 100644 index 0000000000..9a45bbaa61 --- /dev/null +++ b/tests/boundary/filesystem/keyedArchiveDateRange.test.ts @@ -0,0 +1,48 @@ +import { writeFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { build } from "plist"; +import { expect } from "vitest"; + +import { keyedArchiveResultSchema } from "../../../src/domain/apple/keyedArchive.js"; +import { parseEvidence } from "../../../src/domain/evidence.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { cliTest } from "../../support/cli/cliFixture.js"; + +cliTest( + "built CLI reports a date the decoder cannot represent as unknown", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-keyed-archive-date-"); + const path = join(root, "archive.plist"); + // plutil writes years past 9999 unsigned, which the decoder rejects. + const xml = build({ + $archiver: "NSKeyedArchiver", + $version: 100000, + $objects: ["$null", "__FAR__", new Date("2020-01-01T00:00:00Z")], + $top: { far: { CF$UID: 1 }, near: { CF$UID: 2 } }, + }).replace( + "__FAR__", + "11507-08-17T05:20:00Z", + ); + await writeFile(path, xml); + const output = await cli.run({ + arguments: ["inspect-keyed-archive", path, "--json"], + environment: { REA_LOG_LEVEL: "silent", REA_ANALYSIS_PROVIDER: "auto" }, + }); + expect(output.exitCode).toBe(0); + const result = keyedArchiveResultSchema.parse( + parseEvidence(output.json).normalized_result, + ); + expect(result.objects.map(({ id, value }) => ({ id, value }))).toEqual([ + { id: 0, value: "$null" }, + { id: 1, value: { $plist_type: "date", iso8601: null } }, + { + id: 2, + value: { $plist_type: "date", iso8601: "2020-01-01T00:00:00.000Z" }, + }, + ]); + expect(result.limitations).toContainEqual( + expect.stringContaining("1 date value(s)"), + ); + }, +); diff --git a/tests/boundary/filesystem/keyedArchiveXmlNesting.test.ts b/tests/boundary/filesystem/keyedArchiveXmlNesting.test.ts index e2dbe084dc..c1a2790e1d 100644 --- a/tests/boundary/filesystem/keyedArchiveXmlNesting.test.ts +++ b/tests/boundary/filesystem/keyedArchiveXmlNesting.test.ts @@ -2,6 +2,7 @@ import { writeFile } from "node:fs/promises"; import { join } from "node:path"; import { expect, it } from "vitest"; +import { parse } from "plist"; import { inspectBundleKeyedArchive } from "../../../src/artifacts/apple/KeyedArchiveReader.js"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; @@ -27,3 +28,76 @@ it.each([129, 3000])( }); }, ); + +it.each(["(}", "{)", "(@", "{key @", "(x @", "(x ", "{key ", "{key = x,"])( + "preserves the pinned OpenStep syntax diagnosis before a deep suffix after %s", + async (prefix) => { + const text = prefix + "(".repeat(130); + let pinned: unknown; + try { + parse(text); + } catch (cause) { + pinned = cause; + } + if (!(pinned instanceof Error) || pinned instanceof RangeError) + throw new Error( + "Malformed control must fail in the pinned syntax decoder", + ); + const root = await createTestTempDirectory("rea-keyed-openstep-malformed-"); + await writeFile(join(root, "Malformed.plist"), text); + const rejected = await inspectBundleKeyedArchive({ + bundlePath: root, + targetSha256: "a".repeat(64), + parameters: { path: "Malformed.plist" }, + platform: "linux", + }).catch((cause: unknown) => cause); + expect(rejected).toMatchObject({ + reason: "format", + message: `Cannot decode selected Foundation keyed archive: ${pinned.message}`, + }); + }, +); + +it("retains the limit classification for syntactically valid deep OpenStep values", async () => { + const root = await createTestTempDirectory("rea-keyed-openstep-limit-"); + await writeFile( + join(root, "Deep.plist"), + "(".repeat(129) + "x" + ")".repeat(129), + ); + const rejected = await inspectBundleKeyedArchive({ + bundlePath: root, + targetSha256: "a".repeat(64), + parameters: { path: "Deep.plist" }, + platform: "linux", + }).catch((cause: unknown) => cause); + expect(rejected).toMatchObject({ + reason: "limit", + message: + "Cannot decode selected Foundation keyed archive: Plist exceeds 128 nesting levels", + }); +}); + +it.each([ + ["array sibling", (deep: string) => `((x),${deep})`], + ["compound dictionary key", (deep: string) => `{(key)=${deep};}`], +] as const)( + "retains the named OpenStep depth limit after a completed %s", + async (_name, document) => { + const root = await createTestTempDirectory( + "rea-keyed-openstep-sibling-depth-", + ); + const deep = "(".repeat(30000) + "x" + ")".repeat(30000); + await writeFile(join(root, "Deep.plist"), document(deep)); + const rejected = await inspectBundleKeyedArchive({ + bundlePath: root, + targetSha256: "a".repeat(64), + parameters: { path: "Deep.plist" }, + platform: "linux", + }).catch((cause: unknown) => cause); + expect(rejected).toMatchObject({ + reason: "limit", + message: + "Cannot decode selected Foundation keyed archive: Plist exceeds 128 nesting levels", + }); + }, +); diff --git a/tests/boundary/filesystem/kimiClientConfiguration.test.ts b/tests/boundary/filesystem/kimiClientConfiguration.test.ts index e0580031aa..077c24d3ad 100644 --- a/tests/boundary/filesystem/kimiClientConfiguration.test.ts +++ b/tests/boundary/filesystem/kimiClientConfiguration.test.ts @@ -38,11 +38,16 @@ const exists = async (path: string): Promise => { } }; -const expectedRegistration = (launcher: readonly string[]) => ({ +const expectedRegistration = ( + launcher: readonly string[], + profile = "compact", + toolTimeoutMs = 450_000, +) => ({ command: launcher[0], args: launcher.slice(1), + env: { REA_MCP_INPUT_SCHEMA_PROFILE: profile }, startupTimeoutMs: MCP_STARTUP_POLICY.kimiStartupTimeoutMs, - toolTimeoutMs: MCP_STARTUP_POLICY.kimiToolTimeoutMs, + toolTimeoutMs, }); describe("Kimi Code configuration paths and skills", () => { @@ -107,6 +112,68 @@ describe("Kimi Code registration lifecycle", () => { ).toBe("unchanged"); }); + it("preserves selected timeout values through refresh and keeps backups idempotent", async () => { + const { home, client } = await kimiClient(); + const original = JSON.stringify({ + mcpServers: { + rea: { + ...expectedRegistration(["old-rea", "mcp"]), + startupTimeoutMs: 90_000, + toolTimeoutMs: 300_000, + }, + }, + }); + await writeFile(client.configPath, original); + + expect( + await configureClientConfiguration(client, {}, command), + ).toMatchObject({ + status: "configured", + backupPath: `${client.configPath}.rea.backup`, + }); + expect( + JSON.parse(await readFile(client.configPath, "utf8")).mcpServers.rea, + ).toEqual({ + ...expectedRegistration(command), + startupTimeoutMs: 90_000, + toolTimeoutMs: 300_000, + }); + expect(await readFile(`${client.configPath}.rea.backup`, "utf8")).toBe( + original, + ); + expect(await configureClientConfiguration(client, {}, command)).toEqual({ + status: "unchanged", + }); + expect( + ( + await readClientRegistrationStatuses(home, undefined, { + environment: {}, + }) + ).find(({ client: name }) => name === "kimi"), + ).toMatchObject({ + state: "aligned", + advisory: expect.stringContaining("toolTimeoutMs = 450000 ms"), + }); + }); + + it("derives a client deadline beyond Node's timer ceiling for an alternative provider selector", async () => { + const { client } = await kimiClient(); + expect( + await configureClientConfiguration( + client, + { + REA_ANALYSIS_PROVIDER: "ida", + REA_GHIDRA_STARTUP_TIMEOUT_MS: "2147483647", + }, + command, + ), + ).toMatchObject({ status: "configured" }); + expect( + JSON.parse(await readFile(client.configPath, "utf8")).mcpServers.rea + .toolTimeoutMs, + ).toBe(2_147_603_647); + }); + it("registers without type, preserves unrelated settings, backs up, and is idempotent", async () => { const { home, client } = await kimiClient(); const original = `{ @@ -159,6 +226,132 @@ describe("Kimi Code registration lifecycle", () => { }); }); +describe("Kimi Code input-schema profile selection", () => { + const kimiClient = async () => { + const home = await createTestTempDirectory("rea-kimi-schema-profile-"); + const client = kimiPath(home, process.platform, {}); + if (client?.markerPath === undefined) + throw new Error("missing Kimi Code client"); + await mkdir(client.markerPath, { recursive: true }); + return { home, client }; + }; + + it("preserves an explicitly stored full profile", async () => { + const { home, client } = await kimiClient(); + await writeFile( + client.configPath, + JSON.stringify({ + mcpServers: { + rea: { + command: "old-rea", + args: ["mcp"], + env: { REA_MCP_INPUT_SCHEMA_PROFILE: "full" }, + }, + }, + }), + ); + + expect( + (await configureClientConfiguration(client, {}, command)).status, + ).toBe("configured"); + expect( + JSON.parse(await readFile(client.configPath, "utf8")).mcpServers.rea, + ).toEqual(expectedRegistration(command, "full")); + expect( + ( + await readClientRegistrationStatuses(home, undefined, { + environment: {}, + }) + ).find(({ client: name }) => name === "kimi"), + ).toMatchObject({ state: "aligned", command }); + }); + + it.each(["full", "compact"] as const)( + "uses the caller's explicit %s profile", + async (profile) => { + const { client } = await kimiClient(); + expect( + ( + await configureClientConfiguration( + client, + { REA_MCP_INPUT_SCHEMA_PROFILE: profile }, + command, + ) + ).status, + ).toBe("configured"); + expect( + JSON.parse(await readFile(client.configPath, "utf8")).mcpServers.rea, + ).toEqual(expectedRegistration(command, profile)); + }, + ); + + it.each([ + ["full", "compact"], + ["compact", "full"], + ] as const)( + "lets explicit setup %s replace stored %s", + async (overrideProfile, storedProfile) => { + const { client } = await kimiClient(); + await writeFile( + client.configPath, + JSON.stringify({ + mcpServers: { + rea: { + command: "old-rea", + args: ["mcp"], + env: { REA_MCP_INPUT_SCHEMA_PROFILE: storedProfile }, + }, + }, + }), + ); + + expect( + ( + await configureClientConfiguration( + client, + { REA_MCP_INPUT_SCHEMA_PROFILE: overrideProfile }, + command, + ) + ).status, + ).toBe("configured"); + expect( + JSON.parse(await readFile(client.configPath, "utf8")).mcpServers.rea, + ).toEqual(expectedRegistration(command, overrideProfile)); + }, + ); + + it("reports an omitted-profile registration stale until setup adds compact", async () => { + const { home, client } = await kimiClient(); + await writeFile( + client.configPath, + JSON.stringify({ + mcpServers: { + rea: { + command: command[0], + args: command.slice(1), + startupTimeoutMs: MCP_STARTUP_POLICY.kimiStartupTimeoutMs, + toolTimeoutMs: 120_000, + }, + }, + }), + ); + + expect( + ( + await readClientRegistrationStatuses(home, undefined, { + environment: {}, + }) + ).find(({ client: name }) => name === "kimi"), + ).toMatchObject({ state: "stale", command }); + expect( + (await configureClientConfiguration(client, {}, command)).status, + ).toBe("configured"); + expect( + JSON.parse(await readFile(client.configPath, "utf8")).mcpServers.rea, + ).toEqual(expectedRegistration(command, "compact", 120_000)); + }); +}); + describe("Kimi Code package launchers", () => { const kimiClient = async () => { const home = await createTestTempDirectory("rea-kimi-launcher-"); @@ -178,7 +371,7 @@ describe("Kimi Code package launchers", () => { "keeps the %s launcher aligned on %s", async (runner, platform) => { const { home, client } = await kimiClient(); - const launcher = setupRegistrationCommand(platform, runner); + const launcher = setupRegistrationCommand(platform, runner, undefined); expect( (await configureClientConfiguration(client, {}, launcher)).status, ).toBe("configured"); @@ -309,7 +502,7 @@ describe("Kimi Code transport validation", () => { ); }); -describe("Kimi Code timeout validation", () => { +describe("Kimi Code timeout alignment", () => { const kimiClient = async () => { const home = await createTestTempDirectory("rea-kimi-timeout-"); const client = kimiPath(home, process.platform, {}); @@ -320,11 +513,11 @@ describe("Kimi Code timeout validation", () => { }; it.each([ - ["startupTimeoutMs", 1], - ["toolTimeoutMs", 1], - ["startupTimeoutMs", undefined], - ["toolTimeoutMs", undefined], - ] as const)("reports stale when %s is %j", async (field, value) => { + ["startupTimeoutMs", 1, "aligned"], + ["toolTimeoutMs", 1, "aligned"], + ["startupTimeoutMs", undefined, "stale"], + ["toolTimeoutMs", undefined, "stale"], + ] as const)("reports %s timeout %j as %s", async (field, value, state) => { const { home, client } = await kimiClient(); const registration = { ...expectedRegistration(command), @@ -340,7 +533,7 @@ describe("Kimi Code timeout validation", () => { environment: {}, }) ).find(({ client: name }) => name === "kimi"), - ).toMatchObject({ state: "stale" }); + ).toMatchObject({ state }); }); it("uses a direct installed REA command without requiring a package runner", async () => { diff --git a/tests/boundary/filesystem/macHopperInstallOwnership.test.ts b/tests/boundary/filesystem/macHopperInstallOwnership.test.ts new file mode 100644 index 0000000000..04b47c2fc9 --- /dev/null +++ b/tests/boundary/filesystem/macHopperInstallOwnership.test.ts @@ -0,0 +1,275 @@ +import { + access, + mkdir, + mkdtemp, + readFile, + rename, + rm, + stat, + writeFile, +} from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { afterEach, describe, expect, it } from "vitest"; + +import { + installMacHopperBundle, + type MacHopperBundleInstallOperations, +} from "../../../src/application/MacHopper.js"; + +const roots: string[] = []; + +afterEach(async () => { + await Promise.all( + roots.splice(0).map((root) => rm(root, { recursive: true, force: true })), + ); +}); + +const temporaryRoot = async (): Promise => { + const root = await mkdtemp(join(tmpdir(), "rea-hopper-owned-stage-test-")); + roots.push(root); + return root; +}; + +const bundleFiles = ( + overrides: Partial = {}, + captureRoot?: (root: string, mode: number) => void, +): MacHopperBundleInstallOperations => ({ + async createStagingRoot(parent) { + await mkdir(parent, { recursive: true }); + const root = await mkdtemp(join(parent, ".rea-hopper-stage-")); + captureRoot?.(root, (await stat(root)).mode & 0o777); + return root; + }, + async copyBundle(source, stage) { + const executable = join(stage, "Contents/MacOS/hopper"); + await mkdir(join(stage, "Contents/MacOS"), { recursive: true }); + await writeFile(executable, source); + }, + requireLauncher: async (path) => access(path).then(() => undefined), + async pathExists(path) { + try { + await access(path); + return true; + } catch (cause: unknown) { + if ( + typeof cause === "object" && + cause !== null && + "code" in cause && + cause.code === "ENOENT" + ) + return false; + throw cause; + } + }, + rename, + removeStagingRoot: (path) => rm(path, { recursive: true }), + ...overrides, +}); + +const writeBundle = async (path: string, content: string): Promise => { + await mkdir(join(path, "Contents/MacOS"), { recursive: true }); + await writeFile(join(path, "Contents/MacOS/hopper"), content); +}; + +describe("macOS Hopper staging isolation", () => { + it("rejects a staged bundle without its launcher before moving the original", async () => { + const root = await temporaryRoot(); + const destination = join(root, "Applications/Hopper Disassembler.app"); + await writeBundle(destination, "original installation"); + let stagingRoot = ""; + + const result = await installMacHopperBundle( + "replacement without launcher", + destination, + true, + bundleFiles( + { + async copyBundle(_source, stage) { + await mkdir(join(stage, "Contents/MacOS"), { recursive: true }); + }, + }, + (path) => (stagingRoot = path), + ), + () => undefined, + ); + + expect(result).toMatchObject({ + status: "failed", + reason: "launcher_missing", + diagnostic: expect.stringContaining("hopper"), + }); + expect( + await readFile(join(destination, "Contents/MacOS/hopper"), "utf8"), + ).toBe("original installation"); + await expect( + access(join(stagingRoot, "previous.app")), + ).rejects.toMatchObject({ + code: "ENOENT", + }); + await expect(access(stagingRoot)).rejects.toMatchObject({ code: "ENOENT" }); + }); + + it("leaves pre-existing staging-looking sibling directories untouched", async () => { + const root = await temporaryRoot(); + const applications = join(root, "Applications"); + const destination = join(applications, "Hopper Disassembler.app"); + const oldStage = `${destination}.rea-stage`; + const oldPrevious = `${oldStage}-previous`; + await Promise.all([ + writeBundle(oldStage, "user stage sentinel"), + writeBundle(oldPrevious, "user backup sentinel"), + ]); + let stagingRoot = ""; + let stagingMode = 0; + + await expect( + installMacHopperBundle( + "new bundle", + destination, + false, + bundleFiles({}, (path, mode) => { + stagingRoot = path; + stagingMode = mode; + }), + () => undefined, + ), + ).resolves.toEqual({ status: "installed" }); + + expect( + await readFile(join(oldStage, "Contents/MacOS/hopper"), "utf8"), + ).toBe("user stage sentinel"); + expect( + await readFile(join(oldPrevious, "Contents/MacOS/hopper"), "utf8"), + ).toBe("user backup sentinel"); + expect( + await readFile(join(destination, "Contents/MacOS/hopper"), "utf8"), + ).toBe("new bundle"); + expect(stagingMode).toBe(0o700); + await expect(access(stagingRoot)).rejects.toMatchObject({ code: "ENOENT" }); + }); +}); + +describe("macOS Hopper replacement ownership", () => { + it("keeps the committed replacement when owned-root cleanup fails", async () => { + const root = await temporaryRoot(); + const destination = join(root, "Applications/Hopper Disassembler.app"); + await writeBundle(destination, "previous installation"); + let stagingRoot = ""; + let failedCleanup = false; + const retained: (() => Promise)[] = []; + + const result = await installMacHopperBundle( + "new installation", + destination, + true, + bundleFiles( + { + removeStagingRoot: async (path) => { + if (!failedCleanup) { + failedCleanup = true; + throw new Error("simulated cleanup denial"); + } + await rm(path, { recursive: true }); + }, + }, + (path) => (stagingRoot = path), + ), + (cleanup) => retained.push(cleanup), + ); + + expect(result).toMatchObject({ + status: "installed", + cleanupFailure: expect.stringContaining("simulated cleanup denial"), + }); + expect( + await readFile(join(destination, "Contents/MacOS/hopper"), "utf8"), + ).toBe("new installation"); + expect( + await readFile( + join(stagingRoot, "previous.app/Contents/MacOS/hopper"), + "utf8", + ), + ).toBe("previous installation"); + expect(retained).toHaveLength(1); + + await retained[0]!(); + expect( + await readFile(join(destination, "Contents/MacOS/hopper"), "utf8"), + ).toBe("new installation"); + await expect(access(stagingRoot)).rejects.toMatchObject({ code: "ENOENT" }); + }); + + it("retains a moved backup and never overwrites a later destination on rollback", async () => { + const root = await temporaryRoot(); + const destination = join(root, "Applications/Hopper Disassembler.app"); + await writeBundle(destination, "original installation"); + let stagingRoot = ""; + let backupMoved = false; + let concurrentInstallCreated = false; + const rollbackRetained: (() => Promise)[] = []; + const realRename = rename; + + const result = await installMacHopperBundle( + "replacement installation", + destination, + true, + bundleFiles( + { + rename: async (source, target) => { + if (source === destination && target.endsWith("/previous.app")) { + await realRename(source, target); + backupMoved = true; + return; + } + if (source.endsWith("/new.app") && target === destination) { + await writeBundle(destination, "later installation"); + concurrentInstallCreated = true; + throw new Error("simulated competing install"); + } + await realRename(source, target); + }, + }, + (path) => (stagingRoot = path), + ), + (cleanup) => rollbackRetained.push(cleanup), + ); + + expect(result).toMatchObject({ + status: "failed", + diagnostic: expect.stringContaining("simulated competing install"), + cleanupFailure: expect.stringContaining( + "previous Hopper bundle remains owned", + ), + }); + expect(backupMoved).toBe(true); + expect(concurrentInstallCreated).toBe(true); + expect( + await readFile(join(destination, "Contents/MacOS/hopper"), "utf8"), + ).toBe("later installation"); + expect( + await readFile( + join(stagingRoot, "previous.app/Contents/MacOS/hopper"), + "utf8", + ), + ).toBe("original installation"); + expect(rollbackRetained).toHaveLength(1); + + await expect(rollbackRetained[0]!()).rejects.toThrow( + "destination is occupied", + ); + expect( + await readFile( + join(stagingRoot, "previous.app/Contents/MacOS/hopper"), + "utf8", + ), + ).toBe("original installation"); + await rm(destination, { recursive: true }); + await rollbackRetained[0]!(); + expect( + await readFile(join(destination, "Contents/MacOS/hopper"), "utf8"), + ).toBe("original installation"); + await expect(access(stagingRoot)).rejects.toMatchObject({ code: "ENOENT" }); + }); +}); diff --git a/tests/boundary/filesystem/mcpRegistryPublication.test.ts b/tests/boundary/filesystem/mcpRegistryPublication.test.ts new file mode 100644 index 0000000000..71d03f9cc1 --- /dev/null +++ b/tests/boundary/filesystem/mcpRegistryPublication.test.ts @@ -0,0 +1,76 @@ +import { expect, it } from "vitest"; +import { registryPublicationState } from "../../../.github/actions/publish-mcp/registry-state.mjs"; + +const server = { + name: "io.github.fixture/rea", + version: "6.5.0", + description: "Released metadata", +}; +const record = { + server, + _meta: { "io.modelcontextprotocol.registry/official": { status: "active" } }, +}; +const page = (servers: unknown[], metadata = {}) => + Response.json({ servers, metadata }); + +it("finds an identical existing version after paging past substring matches", async () => { + const urls: URL[] = []; + const result = await registryPublicationState(server, async (url: URL) => { + urls.push(url); + return urls.length === 1 + ? page([{ server: { ...server, name: `${server.name}-other` } }], { + nextCursor: "next", + }) + : page([record]); + }); + expect(result).toBe("present"); + expect(urls[0]?.searchParams.get("search")).toBe(server.name); + expect(urls[0]?.searchParams.get("version")).toBe(server.version); + expect(urls[0]?.searchParams.get("include_deleted")).toBe("true"); + expect(urls[1]?.searchParams.get("cursor")).toBe("next"); +}); + +it("permits publication only after a complete successful lookup finds no exact match", async () => { + await expect( + registryPublicationState(server, async () => page([])), + ).resolves.toBe("absent"); + await expect( + registryPublicationState( + server, + async () => new Response("unavailable", { status: 503 }), + ), + ).rejects.toThrow("HTTP 503"); + await expect( + registryPublicationState(server, async () => + Response.json({ metadata: {} }), + ), + ).rejects.toThrow("omitted servers"); + await expect( + registryPublicationState(server, async () => + page([], { nextCursor: "same" }), + ), + ).rejects.toThrow("non-advancing cursor"); +}); + +it("refuses changed, inactive or ambiguous published metadata", async () => { + await expect( + registryPublicationState(server, async () => + page([{ ...record, server: { ...server, description: "Different" } }]), + ), + ).rejects.toThrow("differs from released metadata"); + await expect( + registryPublicationState(server, async () => + page([ + { + ...record, + _meta: { + "io.modelcontextprotocol.registry/official": { status: "deleted" }, + }, + }, + ]), + ), + ).rejects.toThrow("not active"); + await expect( + registryPublicationState(server, async () => page([record, record])), + ).rejects.toThrow("ambiguous"); +}); diff --git a/tests/boundary/filesystem/mipsAbiFlags.test.ts b/tests/boundary/filesystem/mipsAbiFlags.test.ts index 1b724d31bd..6d26ac4ec6 100644 --- a/tests/boundary/filesystem/mipsAbiFlags.test.ts +++ b/tests/boundary/filesystem/mipsAbiFlags.test.ts @@ -2,9 +2,10 @@ import assert from "node:assert/strict"; import { createHash } from "node:crypto"; import { open, readFile, writeFile } from "node:fs/promises"; import { join } from "node:path"; -import { it } from "vitest"; +import { expect, it } from "vitest"; import { parseBinaryTarget } from "../../../src/application/BinaryTargetResolver.js"; import { AnalysisCancelledError } from "../../../src/domain/analysisErrorCore.js"; +import { jsonObjectSchema } from "../../../src/domain/jsonValue.js"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; import type { MipsElfMetadata } from "../../../src/domain/binaryTargetTypes.js"; import { readMipsElfMetadata } from "../../../src/application/MipsElfMetadata.js"; @@ -53,6 +54,62 @@ const fixture = (little = true): Buffer => { bytes.set([0, 0, 32, 2, 1, 1, 0, 5], abiOffset); return bytes; }; +const attributeSection = ( + attributes: readonly number[], + little = true, +): Buffer => { + const subsectionLength = 1 + 4 + attributes.length; + const vendor = Buffer.from("gnu\0", "latin1"); + const sectionLength = 4 + vendor.length + subsectionLength; + const bytes = Buffer.alloc(1 + sectionLength); + bytes[0] = 0x41; + if (little) { + bytes.writeUInt32LE(sectionLength, 1); + bytes.writeUInt32LE(subsectionLength, 1 + 4 + vendor.length + 1); + } else { + bytes.writeUInt32BE(sectionLength, 1); + bytes.writeUInt32BE(subsectionLength, 1 + 4 + vendor.length + 1); + } + vendor.copy(bytes, 5); + bytes[9] = 1; // Tag_File scope. + bytes.set(attributes, 14); + return bytes; +}; +const withGnuAttributes = ( + source: Buffer, + payloads: readonly Buffer[], + little = true, +): Buffer => { + const payloadStart = abiOffset + 24; + const payloadBytes = payloads.reduce( + (total, payload) => total + payload.length, + 0, + ); + const bytes = Buffer.alloc( + Math.max(source.length, payloadStart + payloadBytes), + ); + source.copy(bytes); + let payloadOffset = payloadStart; + const write16 = (value: number, offset: number) => + little + ? bytes.writeUInt16LE(value, offset) + : bytes.writeUInt16BE(value, offset); + const write32 = (value: number, offset: number) => + little + ? bytes.writeUInt32LE(value, offset) + : bytes.writeUInt32BE(value, offset); + write16(2 + payloads.length, 48); + write16(0, 50); // Section names are not needed for numeric-type detection. + payloads.forEach((payload, index) => { + const header = 128 + (2 + index) * 40; + write32(0x6ffffff5, header + 4); + write32(payloadOffset, header + 16); + write32(payload.length, header + 20); + payload.copy(bytes, payloadOffset); + payloadOffset += payload.length; + }); + return bytes; +}; const writeFixture = async (bytes: Buffer): Promise => { const directory = await createTestTempDirectory("rea-mips-abi-test-"); const path = join(directory, "fixture.elf"); @@ -77,7 +134,7 @@ const resolveFixture = async (bytes: Buffer) => { assert.deepEqual(await readFile(path), bytes); return result.value; }; -const inspect = async ( +const inspectDeclarations = async ( bytes: Buffer, little = true, checkCancelled: () => void = () => {}, @@ -90,11 +147,19 @@ const inspect = async ( metadata(little), checkCancelled, ); - return result.ok ? ok(result.value.abiFlags) : result; + return result; } finally { await handle.close(); } }; +const inspect = async ( + bytes: Buffer, + little = true, + checkCancelled: () => void = () => {}, +) => { + const result = await inspectDeclarations(bytes, little, checkCancelled); + return result.ok ? ok(result.value.abiFlags) : result; +}; const target = (mips: MipsElfMetadata) => ({ path: "/fixture.elf", sha256: "a".repeat(64), @@ -102,7 +167,7 @@ const target = (mips: MipsElfMetadata) => ({ format: "elf" as const, architecture: "mips" as const, availableArchitectures: ["mips"] as const, - mips, + mips: { gnuAttributes: null, ...mips }, }); for (const little of [true, false]) { @@ -156,7 +221,10 @@ for (const little of [true, false]) { else bytes.writeUInt32BE(sectionOffset, 32); const resolved = await resolveFixture(bytes); assert.equal(resolved.mips.abiFlags?.fpAbi, 5); - assert.equal(ghidraMipsUnsupportedReason(resolved), null); + assert.match( + ghidraMipsUnsupportedReason(resolved) ?? "", + /ABI declaration inspection is incomplete/u, + ); assert.match( resolved.mips.limitations?.join(" ") ?? "", /section header table/u, @@ -172,7 +240,7 @@ it("keeps declarations in uncaptured section headers unknown when program ABI da assert.equal(resolved.mips.abiFlags, null); assert.match( ghidraMipsUnsupportedReason(resolved) ?? "", - /inspected ABI.*unavailable headers remain unknown/u, + /ABI declaration inspection is incomplete.*unavailable headers remain unknown/u, ); }); for (const failure of ["conflicting", "outside", "duplicate"]) { @@ -370,6 +438,118 @@ it("projects a malformed out-of-probe ABI record as a target error", async () => assert.deepEqual(await readFile(path), bytes); }); +for (const little of [true, false]) { + it(`reads bounded GNU FP/MSA attributes through the full target resolver (${little ? "LE" : "BE"})`, async () => { + const bytes = withGnuAttributes( + fixture(little), + [attributeSection([4, 5, 8, 0], little)], + little, + ); + const result = await inspectDeclarations(bytes, little); + assert.ok(result.ok); + assert.deepEqual(result.value.gnuAttributes, { fpAbi: 5, msa: 0 }); + const resolved = await resolveFixture(bytes); + assert.deepEqual(resolved.mips.gnuAttributes, { fpAbi: 5, msa: 0 }); + assert.equal(ghidraMipsUnsupportedReason(resolved), null); + const profileParameters = jsonObjectSchema.parse( + ghidraMipsProfileParameters(resolved), + ); + expect(profileParameters.mips_elf).toMatchObject({ + gnu_attributes: { fp_abi: 5, msa: 0 }, + }); + }); +} + +it("reconciles FP tags with ABI flags while treating FP ANY as no requirement", async () => { + const any = withGnuAttributes(fixture(), [attributeSection([4, 0])]); + const accepted = await inspectDeclarations(any); + assert.ok(accepted.ok); + assert.equal(accepted.value.gnuAttributes?.fpAbi, 0); + + const conflict = withGnuAttributes(fixture(), [attributeSection([4, 1])]); + const rejected = await inspectDeclarations(conflict); + assert.ok(!rejected.ok); + assert.match( + rejected.error, + /conflicting MIPS ABI flags and GNU floating-point ABI/u, + ); +}); + +it("retains unsupported MSA declarations but refuses provider selection", async () => { + const bytes = withGnuAttributes(fixture(), [attributeSection([8, 1])]); + const resolved = await resolveFixture(bytes); + assert.deepEqual(resolved.mips.gnuAttributes, { fpAbi: null, msa: 1 }); + assert.match( + ghidraMipsUnsupportedReason(resolved) ?? "", + /MSA ABI attribute declares an unsupported/u, + ); +}); + +it.each([ + [0, null], + [1, null], + [5, null], + [2, /GNU floating-point ABI attribute/u], + [3, /GNU floating-point ABI attribute/u], +] as const)( + "applies recordless admission to GNU FP value %i", + async (fpAbi, reason) => { + const bytes = fixture(); + bytes.writeUInt32LE(0x50000001, 36); // Recordless MIPS32 legacy header. + bytes.writeUInt32LE(0, 52); // No PT_MIPS_ABIFLAGS. + bytes.writeUInt32LE(0, 172); // No SHT_MIPS_ABIFLAGS. + const withAttributes = withGnuAttributes(bytes, [ + attributeSection([4, fpAbi]), + ]); + const resolved = await resolveFixture(withAttributes); + const unsupported = ghidraMipsUnsupportedReason(resolved); + if (reason === null) assert.equal(unsupported, null); + else assert.match(unsupported ?? "", reason); + }, +); + +it.each([ + ["an unknown attribute tag", [6, 1], /tag 6/u], + [ + "a duplicate attribute section", + [4, 5], + /duplicate MIPS GNU attribute sections/u, + ], +] as const)( + "refuses %s while inspecting section contents", + async (label, attr, reason) => { + const payload = attributeSection(attr); + const bytes = withGnuAttributes( + fixture(), + label === "a duplicate attribute section" + ? [payload, payload] + : [payload], + ); + const result = await inspectDeclarations(bytes); + assert.ok(!result.ok); + assert.match(result.error, reason); + }, +); + +it("checks the GNU attribute size bound before attempting its file read", async () => { + const bytes = withGnuAttributes(fixture(), [attributeSection([4, 5])]); + bytes.writeUInt32LE(0xfffffff0, 224); + bytes.writeUInt32LE(4097, 228); + const result = await inspectDeclarations(bytes); + assert.ok(!result.ok); + assert.match(result.error, /outside REA's inspected bound of 4096 bytes/u); +}); + +it("uses section type and bounds the GNU attribute payload extent", async () => { + const bytes = withGnuAttributes(fixture(), [attributeSection([4, 5])]); + bytes.writeUInt32LE(0xffffffff, 208); // Unknown name; type remains authoritative. + bytes.writeUInt16LE(0, 50); // No section-name string table is required. + bytes.writeUInt32LE(bytes.length + 1, 224); + const result = await inspectDeclarations(bytes); + assert.ok(!result.ok); + assert.match(result.error, /GNU attribute section lies outside the file/u); +}); + it("preserves cancellation instead of turning it into malformed target data", async () => { const directory = await createTestTempDirectory("rea-mips-resolution-"); const path = join(directory, "fixture.elf"); diff --git a/tests/boundary/filesystem/mipsSharedLoad.test.ts b/tests/boundary/filesystem/mipsSharedLoad.test.ts index 6ad6a38c27..ba88cdb780 100644 --- a/tests/boundary/filesystem/mipsSharedLoad.test.ts +++ b/tests/boundary/filesystem/mipsSharedLoad.test.ts @@ -90,6 +90,7 @@ for (const little of [true, false]) { type: 3, flags: 0x74001005, abi_flags: null, + gnu_attributes: null, shared_load: { status: "complete", relocation_count: 2, diff --git a/tests/boundary/filesystem/mutationWorkflow.test.ts b/tests/boundary/filesystem/mutationWorkflow.test.ts new file mode 100644 index 0000000000..4369b67c65 --- /dev/null +++ b/tests/boundary/filesystem/mutationWorkflow.test.ts @@ -0,0 +1,145 @@ +import { execFile } from "node:child_process"; +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { promisify } from "node:util"; +import { expect, it } from "vitest"; +import { parse } from "yaml"; +import { z } from "zod"; + +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +const execute = promisify(execFile); +const stepSchema = z + .object({ + name: z.string().optional(), + run: z.string().optional(), + if: z.string().optional(), + uses: z.string().optional(), + env: z.record(z.string(), z.string()).optional(), + with: z.record(z.string(), z.unknown()).optional(), + }) + .passthrough(); +const readWorkflow = async () => + z + .object({ + on: z.record(z.string(), z.unknown()), + permissions: z.object({ contents: z.literal("read") }).strict(), + concurrency: z.object({ "cancel-in-progress": z.literal(false) }), + jobs: z.object({ + mutation: z + .object({ + strategy: z.object({ + "fail-fast": z.literal(false), + "max-parallel": z.literal(2), + matrix: z.object({ + include: z.array( + z.object({ suite: z.string(), config: z.string() }), + ), + }), + }), + steps: z.array(stepSchema), + }) + .passthrough(), + }), + }) + .parse( + parse( + await readFile( + new URL("../../../.github/workflows/mutation.yml", import.meta.url), + "utf8", + ), + ), + ); + +it("runs both complete canonical sweeps independently of PR and release gates", async () => { + const workflow = await readWorkflow(); + expect(Object.keys(workflow.on).sort()).toEqual([ + "schedule", + "workflow_dispatch", + ]); + const job = workflow.jobs.mutation; + expect(job["continue-on-error"]).toBeUndefined(); + expect(job.if).toBeUndefined(); + expect(job.strategy.matrix.include).toEqual([ + { suite: "reverse-tools", config: "stryker.config.mjs" }, + { + suite: "reverse-tools-semantics", + config: "stryker.semantics.config.mjs", + }, + ]); + const sweep = job.steps.find( + (step) => step.env?.MUTATION_CONFIG !== undefined, + ); + expect(sweep?.env?.MUTATION_CONFIG).toBe("${{ matrix.config }}"); + expect(sweep?.run?.trim().split("\n")).toEqual([ + "node scripts/prepare-reverse-tools-mutation.mjs", + 'npx --no-install stryker run "${MUTATION_CONFIG}"', + ]); + expect(job.steps.some((step) => step.run === "npm run test:prepare")).toBe( + true, + ); + const upload = job.steps.find((step) => + step.uses?.startsWith("actions/upload-artifact@"), + ); + expect(upload?.if).toBe("${{ !cancelled() }}"); + expect(upload?.with?.name).toBe( + "mutation-${{ matrix.suite }}-${{ github.run_attempt }}", + ); + expect(upload?.with?.["retention-days"]).toBe(7); + const paths = z.string().parse(upload?.with?.path).trim().split("\n"); + for (const { suite, config } of job.strategy.matrix.include) { + const canonical = z + .object({ + default: z.object({ + jsonReporter: z.object({ fileName: z.string() }), + htmlReporter: z.object({ fileName: z.string() }), + }), + }) + .parse( + await import(new URL(`../../../${config}`, import.meta.url).href), + ).default; + expect( + paths.map((path) => path.replace("${{ matrix.suite }}", suite)), + ).toEqual([ + canonical.jsonReporter.fileName, + canonical.htmlReporter.fileName, + ]); + } + for (const step of job.steps) + expect(step["continue-on-error"]).toBeUndefined(); +}); + +it("fails a successful sweep when either required report is absent or empty", async () => { + const steps = (await readWorkflow()).jobs.mutation.steps; + const guard = steps.find((step) => step.env?.REPORT_BASE !== undefined); + expect(guard?.if).toBeUndefined(); + const command = z.string().parse(guard?.run); + const guardIndex = steps.indexOf(guard!); + expect(guardIndex).toBeGreaterThan( + steps.findIndex((step) => step.env?.MUTATION_CONFIG), + ); + expect(guardIndex).toBeLessThan( + steps.findIndex((step) => + step.uses?.startsWith("actions/upload-artifact@"), + ), + ); + for (const state of ["missing", "empty-json", "empty-html", "complete"]) { + const directory = await createTestTempDirectory("rea-mutation-reports-"); + const base = join(directory, "reports", "reverse-tools"); + await mkdir(join(directory, "reports")); + if (state !== "missing") { + await writeFile(`${base}.json`, state === "empty-json" ? "" : "{}"); + await writeFile( + `${base}.html`, + state === "empty-html" ? "" : "", + ); + } + const result = execute("bash", ["-eu", "-c", command], { + cwd: directory, + env: { ...process.env, REPORT_BASE: base }, + }); + if (state === "complete") + await expect(result).resolves.toMatchObject({ stderr: "" }); + else await expect(result).rejects.toMatchObject({ code: 1 }); + } +}); diff --git a/tests/boundary/filesystem/nibNativeHierarchy.test.ts b/tests/boundary/filesystem/nibNativeHierarchy.test.ts index e2b948f571..5f7c07b3ed 100644 --- a/tests/boundary/filesystem/nibNativeHierarchy.test.ts +++ b/tests/boundary/filesystem/nibNativeHierarchy.test.ts @@ -220,7 +220,7 @@ describe.skipIf(process.platform !== "darwin")( REA_ANALYSIS_PROVIDER: "auto", }, }); - expect(output.exitCode).toBe(0); + expect(output.exitCode, JSON.stringify(output)).toBe(0); const result = interfaceBuilderAnalysisSchema.parse( parseEvidence(output.json).normalized_result, ); diff --git a/tests/boundary/filesystem/npmRelease.test.ts b/tests/boundary/filesystem/npmRelease.test.ts new file mode 100644 index 0000000000..c355e21b7d --- /dev/null +++ b/tests/boundary/filesystem/npmRelease.test.ts @@ -0,0 +1,132 @@ +import { createHash } from "node:crypto"; +import { writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { expect, it } from "vitest"; +import { + readNpmRelease, + requireMatchingNpmRelease, + stableUpdateVersion, + waitForNpmRelease, +} from "../../../scripts/ci/npm-release.mjs"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +const metadata = ( + version: string, + integrity: string, + tags: Record, +) => ({ + name: "rea-agents", + versions: { [version]: { name: "rea-agents", version, dist: { integrity } } }, + "dist-tags": tags, +}); + +it("accepts existing registry bytes only when content and selected channel match", async () => { + const root = await createTestTempDirectory("rea-npm-release-"); + const tarball = join(root, "package.tgz"); + const archive = "qualified archive"; + await writeFile(tarball, archive); + const integrity = `sha512-${createHash("sha512").update(archive).digest("base64")}`; + const published = await readNpmRelease("6.5.0", async () => + Response.json(metadata("6.5.0", integrity, { latest: "6.5.0" })), + ); + await expect( + requireMatchingNpmRelease(tarball, "6.5.0", published), + ).resolves.toBeUndefined(); + expect(stableUpdateVersion("6.5.0", published)).toBe("6.5.0"); + if (published.status !== "published") + throw new Error("Fixture release is absent"); + await expect( + requireMatchingNpmRelease(tarball, "6.5.0", { + ...published, + integrity: "sha512-wrong", + }), + ).rejects.toThrow("npm integrity differs"); + await expect( + requireMatchingNpmRelease(tarball, "6.5.0", { + ...published, + taggedVersion: "6.6.0", + }), + ).rejects.toThrow("refusing to change"); +}); + +it("keeps prerelease verification on next and the updater on stable latest", async () => { + const published = await readNpmRelease("6.6.0-rc.1", async () => + Response.json( + metadata("6.6.0-rc.1", "sha512-candidate", { + latest: "6.5.0", + next: "6.6.0-rc.1", + }), + ), + ); + expect(published).toMatchObject({ tag: "next", taggedVersion: "6.6.0-rc.1" }); + expect(stableUpdateVersion("6.6.0-rc.1", published)).toBe("6.5.0"); + if (published.status !== "published") + throw new Error("Fixture release is absent"); + expect(() => + stableUpdateVersion("6.6.0-rc.1", { + ...published, + stableVersion: "6.6.0-rc.1", + }), + ).toThrow("stable release"); +}); + +it("distinguishes absence from failed or malformed registry responses", async () => { + await expect( + readNpmRelease( + "6.5.0", + async () => new Response("missing", { status: 404 }), + ), + ).resolves.toEqual({ status: "absent" }); + await expect( + readNpmRelease("6.5.0", async () => + Response.json({ name: "rea-agents", versions: {}, "dist-tags": {} }), + ), + ).resolves.toEqual({ status: "absent" }); + await expect( + readNpmRelease( + "6.5.0", + async () => new Response("failed", { status: 503 }), + ), + ).rejects.toThrow("HTTP 503"); + await expect( + readNpmRelease("6.5.0", async () => Response.json({})), + ).rejects.toThrow(); + await expect( + readNpmRelease("6.5.0", async () => + Response.json(metadata("6.5.0", "", { latest: "6.5.0" })), + ), + ).rejects.toThrow(); +}); + +it("waits only for confirmed propagation absence and does not retry errors", async () => { + const published = await readNpmRelease("6.5.0", async () => + Response.json(metadata("6.5.0", "sha512-target", { latest: "6.5.0" })), + ); + let calls = 0; + const pauses: number[] = []; + await expect( + waitForNpmRelease( + "6.5.0", + async () => (++calls === 1 ? { status: "absent" } : published), + async (milliseconds) => { + pauses.push(milliseconds); + }, + ), + ).resolves.toEqual(published); + expect(calls).toBe(2); + expect(pauses).toEqual([15_000]); + let failures = 0; + await expect( + waitForNpmRelease( + "6.5.0", + async () => { + failures++; + throw new Error("unavailable"); + }, + async () => { + throw new Error("must not retry"); + }, + ), + ).rejects.toThrow("unavailable"); + expect(failures).toBe(1); +}); diff --git a/tests/boundary/filesystem/packageClientBackup.test.ts b/tests/boundary/filesystem/packageClientBackup.test.ts new file mode 100644 index 0000000000..8294c76c87 --- /dev/null +++ b/tests/boundary/filesystem/packageClientBackup.test.ts @@ -0,0 +1,65 @@ +import { readFile, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { expect, it } from "vitest"; + +import { + assertClientConfig, + terminalModesRestored, +} from "../../../scripts/verify-package-setup.mjs"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +it("requires positive canonical-input and echo terminal mode tokens", () => { + expect(terminalModesRestored("speed 38400 baud; icanon echo isig")).toBe( + true, + ); + expect(terminalModesRestored("speed 38400 baud; -icanon -echo isig")).toBe( + false, + ); + expect(terminalModesRestored("speed 38400 baud; -icanon echo isig")).toBe( + false, + ); + expect(terminalModesRestored("speed 38400 baud; icanon -echo isig")).toBe( + false, + ); +}); + +it("rejects a backup replaced with the post-setup client configuration", async () => { + const directory = await createTestTempDirectory("rea-client-backup-"); + const path = join(directory, "client.json"); + const original = Buffer.from('{"existing":true,"model":"caller"}\n'); + const configured = JSON.stringify({ + existing: true, + model: "caller", + mcpServers: { rea: { command: process.execPath, args: ["mcp"] } }, + }); + await writeFile(path, configured); + await writeFile(`${path}.rea.backup`, original); + await assertClientConfig(process.execPath, path, original); + + await writeFile(`${path}.rea.backup`, configured); + await expect( + assertClientConfig(process.execPath, path, original), + ).rejects.toThrow("packaged client backup failed"); + expect(await readFile(path, "utf8")).toBe(configured); +}); + +it("compares backup bytes even when invalid UTF-8 decodes to the same text", async () => { + const directory = await createTestTempDirectory("rea-client-backup-bytes-"); + const path = join(directory, "client.json"); + const original = Buffer.from('{"existing":true,"note":"\uFFFD"}\n'); + const configured = JSON.stringify({ + existing: true, + mcpServers: { rea: { command: process.execPath, args: ["mcp"] } }, + }); + const changed = Buffer.concat([ + Buffer.from('{"existing":true,"note":"'), + Buffer.from([0xff]), + Buffer.from('"}\n'), + ]); + expect(changed.toString("utf8")).toBe(original.toString("utf8")); + await writeFile(path, configured); + await writeFile(`${path}.rea.backup`, changed); + await expect( + assertClientConfig(process.execPath, path, original), + ).rejects.toThrow("packaged client backup failed"); +}); diff --git a/tests/boundary/filesystem/prReadiness.test.ts b/tests/boundary/filesystem/prReadiness.test.ts new file mode 100644 index 0000000000..f2b54353b0 --- /dev/null +++ b/tests/boundary/filesystem/prReadiness.test.ts @@ -0,0 +1,382 @@ +import { execFile } from "node:child_process"; +import { readFile } from "node:fs/promises"; +import { fileURLToPath } from "node:url"; +import { promisify } from "node:util"; +import { describe, expect, it, vi } from "vitest"; +import { parse } from "yaml"; +import { z } from "zod"; +import { + evaluateReadiness, + runReadinessCheck, + type ReadinessGitHub, + type ReadinessPullRequest, +} from "../../../scripts/ci/pr-readiness.mjs"; + +const template = await readFile( + new URL("../../../.github/pull_request_template.md", import.meta.url), + "utf8", +); +const readyBody = template + .slice( + template.indexOf("## Author readiness"), + template.indexOf("## Before merge"), + ) + .replace("Human owner: ", "Human owner: @contributor") + .replaceAll("- [ ]", "- [x]"); +const makePull = ( + changes: Partial = {}, +): ReadinessPullRequest => ({ + number: 42, + state: "open", + draft: false, + body: readyBody, + head: { sha: "a".repeat(40) }, + base: { ref: "main" }, + ...changes, +}); + +function fixture(changes: Partial = {}) { + const pull = makePull(changes); + const get = vi.fn(async () => ({ data: pull })); + const createCommitStatus = vi.fn( + async ( + _status: Parameters< + ReadinessGitHub["rest"]["repos"]["createCommitStatus"] + >[0], + ) => ({}), + ); + const paginate = vi.fn(async () => [pull]); + const github = { + rest: { + pulls: { get }, + repos: { + createCommitStatus, + listPullRequestsAssociatedWithCommit: vi.fn(), + }, + }, + paginate, + }; + const context = { + repo: { owner: "morluto", repo: "rea" }, + payload: { pull_request: { number: 42 } }, + serverUrl: "https://github.com", + runId: 123, + }; + const core = { info: vi.fn(), setFailed: vi.fn() }; + return { github, context, core, get, createCommitStatus, paginate, pull }; +} + +describe("initial author declaration", () => { + it("accepts the filled current template and uppercase checkmarks", () => { + expect(evaluateReadiness(makePull(), template).state).toBe("success"); + expect( + evaluateReadiness( + makePull({ body: readyBody.replaceAll("[x]", "[X]") }), + template, + ).state, + ).toBe("success"); + expect( + evaluateReadiness( + makePull({ body: readyBody.replaceAll("\n", "\r\n") }), + template, + ).state, + ).toBe("success"); + }); + + it("allows drafts without a body or declarations", () => { + expect( + evaluateReadiness(makePull({ draft: true, body: null }), template).state, + ).toBe("pending"); + }); + + it.each([0, 1, 2])("requires declaration %i to be checked", (index) => { + let position = 0; + const body = readyBody.replaceAll("- [x]", () => + position++ === index ? "- [ ]" : "- [x]", + ); + expect(evaluateReadiness(makePull({ body }), template).state).toBe( + "failure", + ); + }); + + it.each([ + null, + readyBody.replace("Human owner: @contributor", "Human owner:"), + readyBody.replace("Human owner: @contributor", "Human owner: @one @two"), + readyBody.replace( + "Human owner: @contributor", + "Human owner: @automation[bot]", + ), + readyBody + "\nHuman owner: @other\n", + readyBody.replace("## Author readiness", "## Other"), + readyBody + readyBody, + ``, + `~~~markdown\n${readyBody}\n~~~`, + `\`\`\`markdown\n${readyBody}\n\`\`\``, + readyBody + .split("\n") + .map((line) => `> ${line}`) + .join("\n"), + readyBody.replace( + "- [x] I declare this PR ready for initial maintainer review.", + "", + ), + readyBody + + "\n- [x] I declare this PR ready for initial maintainer review.\n", + ])( + "rejects missing, hidden, ambiguous, or incomplete declarations (%#)", + (body) => { + expect(evaluateReadiness(makePull({ body }), template).state).toBe( + "failure", + ); + }, + ); + + it("ignores checked examples in comments and fenced blocks before the actual declaration", () => { + for (const example of [ + `\n`, + `~~~markdown\n${readyBody}\n~~~\n`, + `\`\`\`markdown\n${readyBody}\n\`\`\`\n`, + ]) + expect( + evaluateReadiness(makePull({ body: example + readyBody }), template) + .state, + ).toBe("success"); + }); + + it("does not treat an HTML comment inside a code example as hiding the real declaration", () => { + const body = + "~~~html\n + + @@ -115,9 +117,10 @@ Blog Guides FAQ + Contributors

-
English
+
English
+ + + + + + + + + + +
+ + + +
+
+

Home / Contributors

+

Built together.

+

Thank you to everyone who helps make REA better.

+

+ Code, documentation, tests, translations and ideas — there are many + ways to contribute. +

+
+ How to contribute + Join the conversation → +
+
+
+
+

Contributors

+ +
+

+ +
+ + + +

+ From GitHub’s contributor list · Updated . +

+ +
+ +
+
+ + + +

+ + Top + + + + diff --git a/website/public/contributors/people.json b/website/public/contributors/people.json new file mode 100644 index 0000000000..001b81b5ad --- /dev/null +++ b/website/public/contributors/people.json @@ -0,0 +1,1518 @@ +{ + "repository": "morluto/rea", + "updated": "2026-10-15", + "contributors": [ + { + "avatar_url": "https://avatars.githubusercontent.com/u/76467478?v=4", + "html_url": "https://github.com/morluto", + "id": 76467478, + "login": "morluto", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/161784452?v=4", + "html_url": "https://github.com/N0zoM1z0", + "id": 161784452, + "login": "N0zoM1z0", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/208281322?v=4", + "html_url": "https://github.com/kaoru0822-kitauji", + "id": 208281322, + "login": "kaoru0822-kitauji", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/47457359?v=4", + "html_url": "https://github.com/rudycelekli", + "id": 47457359, + "login": "rudycelekli", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/in/15368?v=4", + "html_url": "https://github.com/apps/github-actions", + "id": 41898282, + "login": "github-actions[bot]", + "type": "Bot" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/114888155?v=4", + "html_url": "https://github.com/DeryFerd", + "id": 114888155, + "login": "DeryFerd", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/128062316?v=4", + "html_url": "https://github.com/sx4im", + "id": 128062316, + "login": "sx4im", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/29498139?v=4", + "html_url": "https://github.com/Y0UW1SH", + "id": 29498139, + "login": "Y0UW1SH", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/35291372?v=4", + "html_url": "https://github.com/XiaoBinGan", + "id": 35291372, + "login": "XiaoBinGan", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/65798732?v=4", + "html_url": "https://github.com/1052326311", + "id": 65798732, + "login": "1052326311", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/130491490?v=4", + "html_url": "https://github.com/BaaRai", + "id": 130491490, + "login": "BaaRai", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/164857476?v=4", + "html_url": "https://github.com/dakjdakd", + "id": 164857476, + "login": "dakjdakd", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/165009858?v=4", + "html_url": "https://github.com/shoyann", + "id": 165009858, + "login": "shoyann", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/191000457?v=4", + "html_url": "https://github.com/NIU-123370", + "id": 191000457, + "login": "NIU-123370", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/455140?v=4", + "html_url": "https://github.com/mvanhorn", + "id": 455140, + "login": "mvanhorn", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/3361008?v=4", + "html_url": "https://github.com/chqcjq", + "id": 3361008, + "login": "chqcjq", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/16836724?v=4", + "html_url": "https://github.com/nikolay-dementiev", + "id": 16836724, + "login": "nikolay-dementiev", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/85309073?v=4", + "html_url": "https://github.com/3QL", + "id": 85309073, + "login": "3QL", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/95101505?v=4", + "html_url": "https://github.com/viniciusdsandrade", + "id": 95101505, + "login": "viniciusdsandrade", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/169027046?v=4", + "html_url": "https://github.com/xu91102", + "id": 169027046, + "login": "xu91102", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/102741971?v=4", + "html_url": "https://github.com/pablodiazjorge", + "id": 102741971, + "login": "pablodiazjorge", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/140487034?v=4", + "html_url": "https://github.com/JayOfTheKeyboard", + "id": 140487034, + "login": "JayOfTheKeyboard", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/223882680?v=4", + "html_url": "https://github.com/yasshmittal", + "id": 223882680, + "login": "yasshmittal", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/11901124?v=4", + "html_url": "https://github.com/ac1982", + "id": 11901124, + "login": "ac1982", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/7439375?v=4", + "html_url": "https://github.com/keemsisi", + "id": 7439375, + "login": "keemsisi", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/76550342?v=4", + "html_url": "https://github.com/thangldw", + "id": 76550342, + "login": "thangldw", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/304086198?v=4", + "html_url": "https://github.com/peter-research", + "id": 304086198, + "login": "peter-research", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/35828647?v=4", + "html_url": "https://github.com/Totopo27", + "id": 35828647, + "login": "Totopo27", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/5654488?v=4", + "html_url": "https://github.com/larinyo", + "id": 5654488, + "login": "larinyo", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/207747646?v=4", + "html_url": "https://github.com/huiq777", + "id": 207747646, + "login": "huiq777", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/193022578?v=4", + "html_url": "https://github.com/jayhemnani9910", + "id": 193022578, + "login": "jayhemnani9910", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/73214171?v=4", + "html_url": "https://github.com/IvenKooLab", + "id": 73214171, + "login": "IvenKooLab", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/93173363?v=4", + "html_url": "https://github.com/HeadmasterEggy", + "id": 93173363, + "login": "HeadmasterEggy", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/153097222?v=4", + "html_url": "https://github.com/Yi-111-a", + "id": 153097222, + "login": "Yi-111-a", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/1432117?v=4", + "html_url": "https://github.com/Fazmin", + "id": 1432117, + "login": "Fazmin", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/275915537?v=4", + "html_url": "https://github.com/zhangguiping-xydt", + "id": 275915537, + "login": "zhangguiping-xydt", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/3128949?v=4", + "html_url": "https://github.com/xeropresence", + "id": 3128949, + "login": "xeropresence", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/74654766?v=4", + "html_url": "https://github.com/akushonkamen", + "id": 74654766, + "login": "akushonkamen", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/115929844?v=4", + "html_url": "https://github.com/1NormalGuy", + "id": 115929844, + "login": "1NormalGuy", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/131595336?v=4", + "html_url": "https://github.com/To3akaRin", + "id": 131595336, + "login": "To3akaRin", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/36695017?v=4", + "html_url": "https://github.com/flitzrrr", + "id": 36695017, + "login": "flitzrrr", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/86675342?v=4", + "html_url": "https://github.com/LuisTaffelli", + "id": 86675342, + "login": "LuisTaffelli", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/35878088?v=4", + "html_url": "https://github.com/fisforfaheem", + "id": 35878088, + "login": "fisforfaheem", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/633385?v=4", + "html_url": "https://github.com/AurelioB", + "id": 633385, + "login": "AurelioB", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/136579807?v=4", + "html_url": "https://github.com/Chaoqi31", + "id": 136579807, + "login": "Chaoqi31", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/52441697?v=4", + "html_url": "https://github.com/danbom", + "id": 52441697, + "login": "danbom", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/77732008?v=4", + "html_url": "https://github.com/jhuang-tw", + "id": 77732008, + "login": "jhuang-tw", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/102906380?v=4", + "html_url": "https://github.com/evenafterallisaid", + "id": 102906380, + "login": "evenafterallisaid", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/172472630?v=4", + "html_url": "https://github.com/chAng-L19", + "id": 172472630, + "login": "chAng-L19", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/75903587?v=4", + "html_url": "https://github.com/TonyWu2333", + "id": 75903587, + "login": "TonyWu2333", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/157806066?v=4", + "html_url": "https://github.com/shubhamahuja9999", + "id": 157806066, + "login": "shubhamahuja9999", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/48523873?v=4", + "html_url": "https://github.com/rohitg00", + "id": 48523873, + "login": "rohitg00", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/126580000?v=4", + "html_url": "https://github.com/sathariels", + "id": 126580000, + "login": "sathariels", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/405711?v=4", + "html_url": "https://github.com/maxazure", + "id": 405711, + "login": "maxazure", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/268763621?v=4", + "html_url": "https://github.com/LcokHcak", + "id": 268763621, + "login": "LcokHcak", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/3125615?v=4", + "html_url": "https://github.com/geekwolf", + "id": 3125615, + "login": "geekwolf", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/160818051?v=4", + "html_url": "https://github.com/Dragolone", + "id": 160818051, + "login": "Dragolone", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/155859586?v=4", + "html_url": "https://github.com/0xRafie", + "id": 155859586, + "login": "0xRafie", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/91750923?v=4", + "html_url": "https://github.com/BeqaKs", + "id": 91750923, + "login": "BeqaKs", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/280805072?v=4", + "html_url": "https://github.com/Aurantium-47", + "id": 280805072, + "login": "Aurantium-47", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/9948621?v=4", + "html_url": "https://github.com/Rinkia", + "id": 9948621, + "login": "Rinkia", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/80999801?v=4", + "html_url": "https://github.com/ShitK", + "id": 80999801, + "login": "ShitK", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/115921185?v=4", + "html_url": "https://github.com/Shenrui-Ma", + "id": 115921185, + "login": "Shenrui-Ma", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/38822679?v=4", + "html_url": "https://github.com/SII-YDD", + "id": 38822679, + "login": "SII-YDD", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/59321404?v=4", + "html_url": "https://github.com/Olorinm", + "id": 59321404, + "login": "Olorinm", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/28519211?v=4", + "html_url": "https://github.com/Nelglor", + "id": 28519211, + "login": "Nelglor", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/133689405?v=4", + "html_url": "https://github.com/cold-atom", + "id": 133689405, + "login": "cold-atom", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/68940212?v=4", + "html_url": "https://github.com/AS9929", + "id": 68940212, + "login": "AS9929", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/286664873?v=4", + "html_url": "https://github.com/marnixpluim", + "id": 286664873, + "login": "marnixpluim", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/72789060?v=4", + "html_url": "https://github.com/akram1089", + "id": 72789060, + "login": "akram1089", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/160517347?v=4", + "html_url": "https://github.com/Sriram-PR", + "id": 160517347, + "login": "Sriram-PR", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/289931646?v=4", + "html_url": "https://github.com/cyanseek", + "id": 289931646, + "login": "cyanseek", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/249198415?v=4", + "html_url": "https://github.com/felipeatp", + "id": 249198415, + "login": "felipeatp", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/194239734?v=4", + "html_url": "https://github.com/jimpablo", + "id": 194239734, + "login": "jimpablo", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/330186599?v=4", + "html_url": "https://github.com/qorvr", + "id": 330186599, + "login": "qorvr", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/101505019?v=4", + "html_url": "https://github.com/vilmerfrost", + "id": 101505019, + "login": "vilmerfrost", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/21329260?v=4", + "html_url": "https://github.com/zaydiscold", + "id": 21329260, + "login": "zaydiscold", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/205188959?v=4", + "html_url": "https://github.com/Zhong-fan", + "id": 205188959, + "login": "Zhong-fan", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/94890352?v=4", + "html_url": "https://github.com/Adolanium", + "id": 94890352, + "login": "Adolanium", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/166712943?v=4", + "html_url": "https://github.com/Aliciawque", + "id": 166712943, + "login": "Aliciawque", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/135997804?v=4", + "html_url": "https://github.com/rekcilyssup", + "id": 135997804, + "login": "rekcilyssup", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/29607505?v=4", + "html_url": "https://github.com/bugstan", + "id": 29607505, + "login": "bugstan", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/55243348?v=4", + "html_url": "https://github.com/bruno-candia", + "id": 55243348, + "login": "bruno-candia", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/249835913?v=4", + "html_url": "https://github.com/CassieFox23", + "id": 249835913, + "login": "CassieFox23", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/111882715?v=4", + "html_url": "https://github.com/txiki739", + "id": 111882715, + "login": "txiki739", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/81847?v=4", + "html_url": "https://github.com/claude", + "id": 81847, + "login": "claude", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/169631528?v=4", + "html_url": "https://github.com/Adkid-Zephyr", + "id": 169631528, + "login": "Adkid-Zephyr", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/8454518?v=4", + "html_url": "https://github.com/jcamargos", + "id": 8454518, + "login": "jcamargos", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/26080016?v=4", + "html_url": "https://github.com/jeonjw85", + "id": 26080016, + "login": "jeonjw85", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/10952625?v=4", + "html_url": "https://github.com/jaelliot", + "id": 10952625, + "login": "jaelliot", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/248806197?v=4", + "html_url": "https://github.com/JasonBuildAI", + "id": 248806197, + "login": "JasonBuildAI", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/190670431?v=4", + "html_url": "https://github.com/8rulerstar", + "id": 190670431, + "login": "8rulerstar", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/163161978?v=4", + "html_url": "https://github.com/dollarsmoney", + "id": 163161978, + "login": "dollarsmoney", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/122638311?v=4", + "html_url": "https://github.com/immanuwell", + "id": 122638311, + "login": "immanuwell", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/243105435?v=4", + "html_url": "https://github.com/Ha1baraA11", + "id": 243105435, + "login": "Ha1baraA11", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/76832285?v=4", + "html_url": "https://github.com/Melon-tgj", + "id": 76832285, + "login": "Melon-tgj", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/136733716?v=4", + "html_url": "https://github.com/ShreyanshJain105", + "id": 136733716, + "login": "ShreyanshJain105", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/81159574?v=4", + "html_url": "https://github.com/Sravan1011", + "id": 81159574, + "login": "Sravan1011", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/180283757?v=4", + "html_url": "https://github.com/Sunil56224972", + "id": 180283757, + "login": "Sunil56224972", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/112210438?v=4", + "html_url": "https://github.com/Swatantra-66", + "id": 112210438, + "login": "Swatantra-66", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/76486634?v=4", + "html_url": "https://github.com/RishTheFish21", + "id": 76486634, + "login": "RishTheFish21", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/146520753?v=4", + "html_url": "https://github.com/TahsinArafat", + "id": 146520753, + "login": "TahsinArafat", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/49832190?v=4", + "html_url": "https://github.com/ysntony", + "id": 49832190, + "login": "ysntony", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/100485512?v=4", + "html_url": "https://github.com/vinay121314", + "id": 100485512, + "login": "vinay121314", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/60581672?v=4", + "html_url": "https://github.com/williamblair333", + "id": 60581672, + "login": "williamblair333", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/18435004?v=4", + "html_url": "https://github.com/cainiao1992", + "id": 18435004, + "login": "cainiao1992", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/5994248?v=4", + "html_url": "https://github.com/yunaremaia", + "id": 5994248, + "login": "yunaremaia", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/154221037?v=4", + "html_url": "https://github.com/YunzhouLi-hub", + "id": 154221037, + "login": "YunzhouLi-hub", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/137368801?v=4", + "html_url": "https://github.com/zeynelacikgoez", + "id": 137368801, + "login": "zeynelacikgoez", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/208325033?v=4", + "html_url": "https://github.com/alfredo1491", + "id": 208325033, + "login": "alfredo1491", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/211421073?v=4", + "html_url": "https://github.com/all-black-493", + "id": 211421073, + "login": "all-black-493", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/152221558?v=4", + "html_url": "https://github.com/ShivekRanjan", + "id": 152221558, + "login": "ShivekRanjan", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/217162076?v=4", + "html_url": "https://github.com/shayanxcyber", + "id": 217162076, + "login": "shayanxcyber", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/132800182?v=4", + "html_url": "https://github.com/SeokMyungSeop", + "id": 132800182, + "login": "SeokMyungSeop", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/36766571?v=4", + "html_url": "https://github.com/LiquidSnake0", + "id": 36766571, + "login": "LiquidSnake0", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/92967809?v=4", + "html_url": "https://github.com/rollroyces", + "id": 92967809, + "login": "rollroyces", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/101842921?v=4", + "html_url": "https://github.com/rileygramlich", + "id": 101842921, + "login": "rileygramlich", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/38380971?v=4", + "html_url": "https://github.com/rasatpetabit", + "id": 38380971, + "login": "rasatpetabit", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/115410287?v=4", + "html_url": "https://github.com/JFSAS", + "id": 115410287, + "login": "JFSAS", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/153702595?v=4", + "html_url": "https://github.com/parbin-12", + "id": 153702595, + "login": "parbin-12", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/238757198?v=4", + "html_url": "https://github.com/pompiliodonatoapp", + "id": 238757198, + "login": "pompiliodonatoapp", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/131965864?v=4", + "html_url": "https://github.com/pirrozani", + "id": 131965864, + "login": "pirrozani", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/9910706?v=4", + "html_url": "https://github.com/pionxzh", + "id": 9910706, + "login": "pionxzh", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/43178362?v=4", + "html_url": "https://github.com/Petru101", + "id": 43178362, + "login": "Petru101", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/117924167?v=4", + "html_url": "https://github.com/thefolahan", + "id": 117924167, + "login": "thefolahan", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/144746926?v=4", + "html_url": "https://github.com/369Apps", + "id": 144746926, + "login": "369Apps", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/186841349?v=4", + "html_url": "https://github.com/JNTMTMTM", + "id": 186841349, + "login": "JNTMTMTM", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/161032298?v=4", + "html_url": "https://github.com/014-code", + "id": 161032298, + "login": "014-code", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/1079727?v=4", + "html_url": "https://github.com/suninrain086", + "id": 1079727, + "login": "suninrain086", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/11473889?v=4", + "html_url": "https://github.com/yiliang114", + "id": 11473889, + "login": "yiliang114", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/102887115?v=4", + "html_url": "https://github.com/wjl8636", + "id": 102887115, + "login": "wjl8636", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/185516466?v=4", + "html_url": "https://github.com/yashas1624", + "id": 185516466, + "login": "yashas1624", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/330743750?v=4", + "html_url": "https://github.com/xpeng5278-web", + "id": 330743750, + "login": "xpeng5278-web", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/93262000?v=4", + "html_url": "https://github.com/Seelly", + "id": 93262000, + "login": "Seelly", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/140315902?v=4", + "html_url": "https://github.com/vt0x78", + "id": 140315902, + "login": "vt0x78", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/38744951?v=4", + "html_url": "https://github.com/tyloryang", + "id": 38744951, + "login": "tyloryang", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/318237806?v=4", + "html_url": "https://github.com/taljeri", + "id": 318237806, + "login": "taljeri", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/116528231?v=4", + "html_url": "https://github.com/sqmagellan", + "id": 116528231, + "login": "sqmagellan", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/190618464?v=4", + "html_url": "https://github.com/skoodster", + "id": 190618464, + "login": "skoodster", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/146759278?v=4", + "html_url": "https://github.com/saurabh-sol", + "id": 146759278, + "login": "saurabh-sol", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/95270855?v=4", + "html_url": "https://github.com/Zburgers", + "id": 95270855, + "login": "Zburgers", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/98685501?v=4", + "html_url": "https://github.com/nKOxxx", + "id": 98685501, + "login": "nKOxxx", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/62732628?v=4", + "html_url": "https://github.com/muhammad-luay", + "id": 62732628, + "login": "muhammad-luay", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/274052178?v=4", + "html_url": "https://github.com/morconsultoriadigital-afk", + "id": 274052178, + "login": "morconsultoriadigital-afk", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/311719211?v=4", + "html_url": "https://github.com/maodns", + "id": 311719211, + "login": "maodns", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/243883506?v=4", + "html_url": "https://github.com/lyly-bonbon", + "id": 243883506, + "login": "lyly-bonbon", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/146933685?v=4", + "html_url": "https://github.com/lalomorales22", + "id": 146933685, + "login": "lalomorales22", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/95490470?v=4", + "html_url": "https://github.com/kvalls", + "id": 95490470, + "login": "kvalls", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/13846660?v=4", + "html_url": "https://github.com/kimtth", + "id": 13846660, + "login": "kimtth", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/237556902?v=4", + "html_url": "https://github.com/jiangjiang248", + "id": 237556902, + "login": "jiangjiang248", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/39996743?v=4", + "html_url": "https://github.com/jbernous", + "id": 39996743, + "login": "jbernous", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/22683417?v=4", + "html_url": "https://github.com/gongfpp", + "id": 22683417, + "login": "gongfpp", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/32026681?v=4", + "html_url": "https://github.com/fermanzolido", + "id": 32026681, + "login": "fermanzolido", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/6012706?v=4", + "html_url": "https://github.com/cevieiras", + "id": 6012706, + "login": "cevieiras", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/57933796?v=4", + "html_url": "https://github.com/balkan16", + "id": 57933796, + "login": "balkan16", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/274363472?v=4", + "html_url": "https://github.com/astralyx0129", + "id": 274363472, + "login": "astralyx0129", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/120994000?v=4", + "html_url": "https://github.com/FeeHaard", + "id": 120994000, + "login": "FeeHaard", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/10838655?v=4", + "html_url": "https://github.com/khanukov", + "id": 10838655, + "login": "khanukov", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/183067453?v=4", + "html_url": "https://github.com/Vaishnavi220506", + "id": 183067453, + "login": "Vaishnavi220506", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/8477196?v=4", + "html_url": "https://github.com/dhl84", + "id": 8477196, + "login": "dhl84", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/17119984?v=4", + "html_url": "https://github.com/fengdwx", + "id": 17119984, + "login": "fengdwx", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/47118053?v=4", + "html_url": "https://github.com/dandgabr", + "id": 47118053, + "login": "dandgabr", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/190232014?v=4", + "html_url": "https://github.com/DLME2024", + "id": 190232014, + "login": "DLME2024", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/199161495?v=4", + "html_url": "https://github.com/cursoragent", + "id": 199161495, + "login": "cursoragent", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/59381946?v=4", + "html_url": "https://github.com/CodeAlex52", + "id": 59381946, + "login": "CodeAlex52", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/210003561?v=4", + "html_url": "https://github.com/CmsChase", + "id": 210003561, + "login": "CmsChase", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/142112319?v=4", + "html_url": "https://github.com/MegaPhoenix92", + "id": 142112319, + "login": "MegaPhoenix92", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/180254320?v=4", + "html_url": "https://github.com/charan-rathore", + "id": 180254320, + "login": "charan-rathore", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/61664693?v=4", + "html_url": "https://github.com/kalayciburak", + "id": 61664693, + "login": "kalayciburak", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/71917936?v=4", + "html_url": "https://github.com/arielfikru", + "id": 71917936, + "login": "arielfikru", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/106865140?v=4", + "html_url": "https://github.com/ArianPashae", + "id": 106865140, + "login": "ArianPashae", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/122965317?v=4", + "html_url": "https://github.com/anujsingh-cse", + "id": 122965317, + "login": "anujsingh-cse", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/102934797?v=4", + "html_url": "https://github.com/Angelopgit", + "id": 102934797, + "login": "Angelopgit", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/169967362?v=4", + "html_url": "https://github.com/Bortlesboat", + "id": 169967362, + "login": "Bortlesboat", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/108879955?v=4", + "html_url": "https://github.com/sandr01", + "id": 108879955, + "login": "sandr01", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/101240385?v=4", + "html_url": "https://github.com/ALANDVO", + "id": 101240385, + "login": "ALANDVO", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/67451993?v=4", + "html_url": "https://github.com/akshitkhokhani", + "id": 67451993, + "login": "akshitkhokhani", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/180300697?v=4", + "html_url": "https://github.com/Akashmdshetty", + "id": 180300697, + "login": "Akashmdshetty", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/94559553?v=4", + "html_url": "https://github.com/aimalexe", + "id": 94559553, + "login": "aimalexe", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/192310219?v=4", + "html_url": "https://github.com/ahkurdev", + "id": 192310219, + "login": "ahkurdev", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/206692996?v=4", + "html_url": "https://github.com/Adarsh-mk7", + "id": 206692996, + "login": "Adarsh-mk7", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/95555196?v=4", + "html_url": "https://github.com/AbdulAzizWang", + "id": 95555196, + "login": "AbdulAzizWang", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/107977627?v=4", + "html_url": "https://github.com/555aaditya", + "id": 107977627, + "login": "555aaditya", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/257249127?v=4", + "html_url": "https://github.com/AWUQAQ", + "id": 257249127, + "login": "AWUQAQ", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/308706627?v=4", + "html_url": "https://github.com/ASVLCII", + "id": 308706627, + "login": "ASVLCII", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/190163814?v=4", + "html_url": "https://github.com/AICodeLion", + "id": 190163814, + "login": "AICodeLion", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/156576604?v=4", + "html_url": "https://github.com/Nik1313132", + "id": 156576604, + "login": "Nik1313132", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/226919216?v=4", + "html_url": "https://github.com/wunianze666-netizen", + "id": 226919216, + "login": "wunianze666-netizen", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/87424988?v=4", + "html_url": "https://github.com/Namnarak", + "id": 87424988, + "login": "Namnarak", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/95697060?v=4", + "html_url": "https://github.com/naimur-reza", + "id": 95697060, + "login": "naimur-reza", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/94893024?v=4", + "html_url": "https://github.com/ranmori", + "id": 94893024, + "login": "ranmori", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/27564626?v=4", + "html_url": "https://github.com/aznikline", + "id": 27564626, + "login": "aznikline", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/204237451?v=4", + "html_url": "https://github.com/lusknchars", + "id": 204237451, + "login": "lusknchars", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/92200217?v=4", + "html_url": "https://github.com/lg114", + "id": 92200217, + "login": "lg114", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/136307740?v=4", + "html_url": "https://github.com/Joshwong1908", + "id": 136307740, + "login": "Joshwong1908", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/79677753?v=4", + "html_url": "https://github.com/JosephDemarest", + "id": 79677753, + "login": "JosephDemarest", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/46970753?v=4", + "html_url": "https://github.com/JonahSimon", + "id": 46970753, + "login": "JonahSimon", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/102573344?v=4", + "html_url": "https://github.com/JingHao-Leon", + "id": 102573344, + "login": "JingHao-Leon", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/130693642?v=4", + "html_url": "https://github.com/Jackmeson1", + "id": 130693642, + "login": "Jackmeson1", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/1792039?v=4", + "html_url": "https://github.com/zahhak", + "id": 1792039, + "login": "zahhak", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/10250068?v=4", + "html_url": "https://github.com/troke12", + "id": 10250068, + "login": "troke12", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/69626391?v=4", + "html_url": "https://github.com/Ramcharan49", + "id": 69626391, + "login": "Ramcharan49", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/78690656?v=4", + "html_url": "https://github.com/HoneyTyagii", + "id": 78690656, + "login": "HoneyTyagii", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/157357792?v=4", + "html_url": "https://github.com/STiFLeR7", + "id": 157357792, + "login": "STiFLeR7", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/152292315?v=4", + "html_url": "https://github.com/hifzabuildsai", + "id": 152292315, + "login": "hifzabuildsai", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/107310514?v=4", + "html_url": "https://github.com/hemanthrayuduu", + "id": 107310514, + "login": "hemanthrayuduu", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/17862704?v=4", + "html_url": "https://github.com/haythemsellami", + "id": 17862704, + "login": "haythemsellami", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/52102987?v=4", + "html_url": "https://github.com/rizvihasan", + "id": 52102987, + "login": "rizvihasan", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/188205924?v=4", + "html_url": "https://github.com/fisHarly0", + "id": 188205924, + "login": "fisHarly0", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/225364157?v=4", + "html_url": "https://github.com/saya-ch", + "id": 225364157, + "login": "saya-ch", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/80974392?v=4", + "html_url": "https://github.com/giri256", + "id": 80974392, + "login": "giri256", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/148368732?v=4", + "html_url": "https://github.com/gefrezim", + "id": 148368732, + "login": "gefrezim", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/184471511?v=4", + "html_url": "https://github.com/Ganesh2325", + "id": 184471511, + "login": "Ganesh2325", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/281093441?v=4", + "html_url": "https://github.com/Franky100-pig", + "id": 281093441, + "login": "Franky100-pig", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/140405963?v=4", + "html_url": "https://github.com/imissapixel", + "id": 140405963, + "login": "imissapixel", + "type": "User" + }, + { + "avatar_url": "https://avatars.githubusercontent.com/u/119940611?v=4", + "html_url": "https://github.com/FengCi0", + "id": 119940611, + "login": "FengCi0", + "type": "User" + } + ] +} diff --git a/website/public/examples/dino-lab/index.html b/website/public/examples/dino-lab/index.html index 3f3272d4c0..6068c21470 100644 --- a/website/public/examples/dino-lab/index.html +++ b/website/public/examples/dino-lab/index.html @@ -51,6 +51,8 @@ + + @@ -116,9 +118,10 @@ Blog Guides FAQ + Contributors
-
English
+
English
-

- This command uses a shell with here-document support, such as - Bash or Zsh. Your agent can run the same JavaScript in your - platform’s shell. -

+
+
+ Windows · PowerShell + +
+ 
@'
+const { toCsv } = require('./csv.js');
+process.stdout.write(toCsv([{ id: 1, title: 'Hello, "REA"' }]));
+'@ | node
+
@@ -512,7 +526,8 @@ JS

REA · Reverse Engineer Anything

- SourceContributorsSourceMIT licenseJoin the communityFollow us on X diff --git a/website/public/get-started/index.html b/website/public/get-started/index.html index a81cdad8a0..2eaa17824c 100644 --- a/website/public/get-started/index.html +++ b/website/public/get-started/index.html @@ -51,6 +51,8 @@ + + @@ -115,9 +117,10 @@ Blog Guides FAQ + Contributors
-
English
+
English
-
- 
node -v
-npm -v
-
-

- If node -v prints v22.19+, v24.11+ or v26+, you're - ready. If it prints an older version or "not recognized" / "command - not found", install Node.js 24 LTS: -

-

Windows (PowerShell)

-
-
- Windows PowerShell -
- 
winget install OpenJS.NodeJS.LTS
-
-

- Type Y if asked to accept the terms. Close PowerShell, open it - again, then run node -v. -

-

- If PowerShell later says - "running scripts is disabled on this system" when you run - npx, run this once and try again: -

-
-
- Windows PowerShell -
- 
Set-ExecutionPolicy -Scope CurrentUser RemoteSigned
-
-

macOS

-

If you use Homebrew:

-
-
- macOS · Homebrew -
- 
brew install node
-
-

Otherwise:

-
-
- macOS -
- 
curl -fsSL https://fnm.vercel.app/install | bash
-
-

Close Terminal, open it again, then:

-
-
- macOS -
- 
fnm install 24
-fnm default 24
-
-

Linux

-
-
- Linux -
- 
curl -fsSL https://fnm.vercel.app/install | bash
-
-

- The installer needs curl and unzip (on - Debian/Ubuntu: sudo apt install curl unzip). Close the - terminal, open it again, then: -

-
-
- Linux -
- 
fnm install 24
-fnm default 24
-
-

- Your distribution's own nodejs package is often older - than 22.19, so check node -v if you use it instead. -

-

- Prefer a regular installer? Download the LTS version from - nodejs.org. -

-
-
-

Connect your coding agent

+
+

1. Prepare your computer

- Run setup to connect REA to your coding agent. Client choices - include Claude Code, Codex, Cursor, Gemini CLI, Windsurf, Grok - Build, and other supported MCP clients. Grok Bot's connector is - added from its chat. + Use the computer where your target files are stored. A local + coding agent can read those files and run tools for you.

-
- New to coding agents? -

- A coding agent is an AI assistant that works on your computer - instead of in a browser tab. It can read files and run commands, - and it asks before making changes. REA gives it - reverse-engineering tools. -

-

- If you don't have one yet, pick one, follow its install page and - sign in: -

+
+ How do I open a terminal?
  • - Claude Code - by Anthropic. Needs a paid Claude plan. + Windows: open the Start menu, type + PowerShell and press Enter.
  • - Codex - by OpenAI. Sign in with your ChatGPT account. + macOS: press ⌘ Space, + type Terminal and press Enter.
  • - OpenCode, open source. Connect the model provider of your choice. + Linux: press Ctrl Alt + T on most desktops.

- Then open it and copy the prompt below into it. The agent runs - REA's setup and shows you the plan before changing anything. + Paste a command with Ctrl V (macOS: + ⌘ V; Linux terminals: Ctrl + Shift V), then press Enter. +

+

+ Commands go in the terminal. Prompts labeled “Your coding agent” + go in the agent's chat.

+

+ You'll need Node.js 22 (22.19+), 24 + (24.11+) or 26+. npm comes with + Node.js. +

+
+ + Don't have Node.js, or not sure? + +

Open a terminal and check:

+
+
+ Terminal +
+ 
node -v
+npm -v
+
+

+ If node -v prints v22.19+, v24.11+ or v26+, you're + ready. If it prints an older version or "not recognized" / + "command not found", install + Node.js 24 LTS: +

+
+ Windows (PowerShell) +
+
+ Windows PowerShell +
+ 
winget install --exact --id OpenJS.NodeJS.LTS
+
+

+ Type Y if asked to accept the terms. Close PowerShell, open it + again, then run node -v. +

+

+ Download the Node.js installer + if winget is unavailable. +

+

+ If PowerShell later says + "running scripts is disabled on this system" when you + run npx, use npx.cmd instead. For + npm, use npm.cmd: +

+
+
+ Windows PowerShell +
+ 
npx.cmd rea-agents@latest setup
+
+
+
+ macOS +

+ For a first installation, download the + Node.js 24 LTS macOS installer (.pkg), open it and follow the installer. Then reopen Terminal. +

+
+ Already use Homebrew? +
+
+ macOS · Homebrew +
+ 
brew install node
+
+
+
+
+ Linux +

+ Install curl and unzip first if they are missing. On Debian or + Ubuntu, run sudo apt install curl unzip; on other + distributions, use your package manager. +

+
+
+ Linux +
+ 
curl -fsSL https://fnm.vercel.app/install | bash
+
+

Close the terminal, open it again, then:

+
+
+ Linux +
+ 
fnm install 24
+fnm use 24
+fnm default 24
+
+

+ Your distribution's own nodejs package is often + older than 22.19, so check node -v if you use it + instead. +

+
+

+ Prefer a regular installer? Download the LTS version from + nodejs.org. +

+
+ +
+ Make a working folder for your agent +

+ In your terminal, make a folder for the exercise, then move into + it: +

+
+
+ Terminal +
+ 
mkdir rea-work
+cd rea-work
+
+

+ Keep this terminal open for the next step. If you already have a + working folder, open it instead. +

+
+
+
+

2. Choose a coding agent

+

+ Your agent is the AI you talk to. REA supplies the inspection + tools it uses. Pick one client below; open its instructions if you + haven't used it before. +

+
+ Codex · use your ChatGPT account +

Install Codex using the command for your operating system:

+
+
+ Windows PowerShell + +
+ 
powershell -ExecutionPolicy ByPass -c "irm https://chatgpt.com/codex/install.ps1 | iex"
+
+
+
+ macOS / Linux +
+ 
curl -fsSL https://chatgpt.com/codex/install.sh | sh
+
+

+ After installation, reopen your terminal and return to the + working folder. Then start Codex: +

+
+
+ Terminal +
+ 
codex
+
+

+ Choose “Sign in with ChatGPT”, complete the browser sign-in, + then return to the terminal. When the message box appears, + continue to step 3. +

+

+ Codex installation and sign-in +

+
+
+ Cursor, Windsurf or VS Code · use an editor +
    +
  1. + Install the editor for your operating system and sign in. +
    2. +
  2. Open the rea-work folder you made in step 1.
    3. +
  3. + Open its AI chat and choose Agent mode. In Windsurf or Devin + Desktop, open the agent chat; in VS Code, set up GitHub + Copilot first. +
    4. +
+

+ Cursor setup · + Windsurf / Devin Desktop setup + · + VS Code and Copilot setup +

+
+
+ Claude Code · use your Claude account +

Install using the command for your operating system:

+
+
+ Windows PowerShell +
+ 
irm https://claude.ai/install.ps1 | iex
+
+
+
+ macOS / Linux +
+ 
curl -fsSL https://claude.ai/install.sh | bash
+
+

+ Reopen your terminal, return to your working folder and run + claude. Follow the sign-in prompts. Access uses an + eligible Claude subscription or a configured API account. +

+

+ Claude Code installation and account options +

+
+
+ Claude Desktop · use a desktop chat +

+ Install Claude Desktop on Windows or macOS and sign in. Quit it + completely, then run this command in a terminal you opened + outside Claude: +

+
+
+ Terminal + +
+ 
npx rea-agents@latest setup --client claude_desktop
+
+

+ Review and approve the plan, reopen Claude, then + check the connection in step 3. +

+

+ Download Claude Desktop + · + Claude Desktop connection details +

+
+
+ OpenCode · choose an AI model +
+
+ Terminal +
+ 
npm install --global opencode-ai
+
+

+ Run opencode in your working folder. Use + /connect in its chat to connect your model account, + then select a model. +

+

+ OpenCode installation and model setup +

+
+
+ All supported clients and setup IDs +

+ The npm 6.4.0 release supports the clients below. Use the ID + with --client when choosing a client explicitly. +

+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ClientSetup ID
Claude Codeclaude_code
Claude Desktopclaude_desktop
Codexcodex
Cursorcursor
Gemini CLIgemini_cli
Windsurfwindsurf
Devindevin
OpenCodeopencode
Antigravityantigravity
GitHub Copilot CLIcopilot_cli
Command Codecommandcode
Qwen Codeqwen_code
VS Codevscode
Grok Buildgrok_build
OMPomp
Pipi
Hermeshermes
+
+

+ Qoder (qoder), Factory Droid (droid), + Trae (trae), ZCode (zcode) and + DeepSeek Harness (dsh) are available on repository + main; use a source build or a release containing those + integrations. +

+

+ Client configuration and troubleshooting + · + Release availability +

+
+ Grok Bot · connect on its hosted computer +

+ In Grok Bot's chat, ask it to add a custom MCP server named + rea using this command on the Bot's computer: +

+
+
+ Grok Bot +
+ 
npx -y rea-agents@6.4.0 mcp
+
+

+ Grok Bot connector instructions +

+
+
+
+
+

3. Connect REA

+

Copy this into your coding agent:

>

+
Or run setup yourself -
- How do I open a terminal? -
    -
  • - Windows: open the Start menu, type - PowerShell and press Enter. -
    • -
  • - macOS: press ⌘ Space, - type Terminal and press Enter. -
    • -
  • - Linux: press Ctrl Alt - T on most desktops. -
    • -
-

- Paste a command with Ctrl V (macOS: - ⌘ V; Linux terminals: Ctrl - Shift V), then press Enter. -

-
- TerminalTerminal 
npx rea-agents@latest setup
+

For one specific client, add its setup ID:

+
+
+ Terminal +
+ 
npx rea-agents@latest setup --client codex
+
+

+ If npm asks to download the package, type y and + press Enter. In REA's client list, use the arrow keys to move, + Space to select your client and Enter to continue. +

Choose your client and review the setup plan. REA adds its MCP connection and the matching investigation skill, keeping a backup of existing configuration. It asks before applying the changes.

-

Restart your client when setup finishes.

+

+ MCP is the connection that lets your agent call REA. The skill + gives it a workflow for investigating software. +

+

+ For the first exercise, you can skip the optional Hopper + installation. Restart your client when setup finishes. +

How do I restart my client?

@@ -498,348 +853,931 @@ fnm default 24 CLI): exit the session and start it again.

-

- Give your agent a target path and a specific question. For a - JavaScript or Electron app, you might start with: -

-
-

- Look at /absolute/path/to/app and find how it - exports data. Show the entry points, calls and dependencies - involved, with the source locations for your findings. -

-
-

Replace the path with your local target.

-
- How do I get a folder's full path? -
    -
  • - Windows: in File Explorer, right-click the - folder and choose Copy as path. On Windows 10, hold - Shift while right-clicking. -
    • -
  • - macOS: in Finder, right-click the folder, - hold Option and choose Copy "…" as Pathname. -
    • -
  • - Any system: drag the folder into a terminal - window and its path appears. -
    • -
-

- In a terminal, put quotes around a path that contains spaces. -

-
-
-
-

Use the CLI

-

- Analyze a JavaScript or Electron application folder or ASAR - archive without running the app: -

-
+
- Static application analysis -
- 
npx -y rea-agents@latest analyze-javascript-application \
-  /absolute/path/to/app --json
-
-

- Replace the path with your extracted app directory or - .asar file. On Windows, a path such as - "D:/apps/example" works here. -

-
- Where do I find an app's .asar file? -

- Electron apps usually keep it in a - resources folder: -

-
    -
  • - Windows: - C:\Users\<you>\AppData\Local\Programs\<App>\resources\app.asar -
    • -
  • - macOS: - /Applications/<App>.app/Contents/Resources/app.asar - (right-click the app and choose - Show Package Contents) -
    • -
  • - Linux: often - /opt/<App>/resources/app.asar -
    • -
-

- Some apps install elsewhere. Search the app's folder for - app.asar. -

-
-

- The result identifies the app, shows connections between modules - and gives their source locations. Connections REA cannot resolve - are marked in the result. -

-

- If you'd like a permanent rea command in your shell: -

-
-
- Install the CLI -
- 
npm install --global rea-agents
-
-

- Continue with the JavaScript application guide -

-
- -
-

Set up native analysis

-

- Native binaries need a local analysis tool. Choose and configure - one below, then give REA your executable or library. -

-
- Choose a tool: Ghidra, Hopper or IDA -
-
-

Ghidra

-
-

- Use an existing Ghidra 12.1.x installation and a 64-bit - JDK 21 or newer on Linux or macOS, on x64 or arm64. Your - Ghidra installation needs a native decompiler for your - host architecture. Set the paths before running setup: -

-
- Connect your Ghidra installation -
-
- Ghidra connection · macOS / Linux -
- 
export GHIDRA_INSTALL_DIR=/absolute/path/to/ghidra_12.1.4_PUBLIC
-export JAVA_HOME=/absolute/path/to/jdk-21
-npx -y rea-agents@latest doctor --provider ghidra --json
-npx rea-agents@latest setup
-
-

- Replace both paths with your existing installations. - Setup checks them and records valid settings for your - selected client. On Linux arm64, you may need to build - Ghidra's native decompiler separately. -

-
-
- Connect Ghidra on Windows -

- Windows x64 has experimental, read-only support for - native x86 and x64 PE applications and DLLs on fixed - local NTFS drives. Use an existing Ghidra 12.1.x - installation and a 64-bit JDK 21 or newer: -

-
-
- Ghidra connection · Windows PowerShell - -
- 
$env:GHIDRA_INSTALL_DIR = "C:/tools/ghidra_12.1.4_PUBLIC"
-$env:JAVA_HOME = "C:/tools/jdk-21"
-npx -y rea-agents@latest doctor --provider ghidra --json
-npx rea-agents@latest setup
-
-

- Replace the two paths with your installations, then - review setup's plan. -

-
-
-
-
-

Hopper

-
-

- Connect Hopper on macOS or Linux. Run the setup command - above; it detects an existing installation or offers an - installation plan for your approval. -

-

- On macOS, complete Hopper's first-run choice of demo mode - or license activation before an unattended analysis. -

-
-
-
-

IDA

-
-

- The verified IDA setups use Windows, with an existing MCP - registration for an attached GUI or a headless database. - Linux and macOS headless analysis remains unverified. -

-
- - Connect an existing IDA MCP registration - -

- Keep the upstream server's working command, - args and optional env in a - JSON file. An existing mcpServers object - with an ida-pro-mcp entry is also accepted. - Point REA at that file before setup: -

-
-
- IDA connection · Windows PowerShell -
- 
$env:REA_IDA_MCP_CONFIG = "C:/analysis/ida-mcp.json"
-$env:REA_ANALYSIS_PROVIDER = "ida"
-npx -y rea-agents@latest doctor --provider ida --json
-npx rea-agents@latest setup
-
-

- The attached profile uses upstream legacy 1.4.0 tools - with the original binary already open in IDA. The - headless profile uses the upstream database-supervisor - API and a registration with - "mode": "headless"; real verification - covers an IDA 9.3 Windows x64 supervisor. Doctor checks - the registration without starting IDA. -

-

- Give REA the original executable path, with - --provider ida in a CLI query. For an - agent, include REA_IDA_MCP_CONFIG in the - REA registration's environment and select IDA when - opening the binary. -

-
-
-
-
-
-

- For an existing Ghidra setup, inspect a function with: -

-
-
- Function analysis -
- 
npx -y rea-agents@latest function /absolute/path/to/program main \
-  --provider ghidra --json
-
-

- Replace main with the function name or address you - want to inspect. -

-
- How native analysis sessions work -

- Each CLI command imports and analyzes the binary, then closes - its session. Your agent's MCP queries reuse the imported binary - while that session is open. -

-
-

- Follow a native analysis example - to see how the function result leads to a recovered calculation. -

-
- -
-

Getting help

-

If your agent doesn't see REA

-

- Restart or reconnect the client after setup. Check its REA - connection with doctor; for Codex: -

-
-
- Check one clientTerminal 
npx -y rea-agents@latest doctor --client codex --json
-

If a provider needs attention

-

Check the provider you're using. For Ghidra:

-
-
- Check one provider + Replace codex with your client's setup ID. A + successful check reports "healthy": true. Then ask + your agent: +

+
+

+ Check that REA is connected and show me which REA tools you can + use. +

+
+
+
+

4. Try your first investigation

+

+ The example is a six-file Electron app with an + Export notes button. Your agent can inspect its + code without installing or launching Electron. +

+
+
+ Your coding agent
- 
npx -y rea-agents@latest doctor --provider ghidra --json
+

+ Download + https://rea.tools/examples/notes-example.zip + into a new example folder and unpack it. Use REA to find what + happens when I click Export notes. Show where + notes.csv is written, which fields + it contains, and the relevant files and lines. +

-

The report gives the failed check and a suggested next step.

+

Look for a short explanation with file and line references:

+
+

+ The button sends a notes:export message. The main + process formats the notes as CSV and writes + notes.csv in the user’s Downloads folder. The + columns are id and title. +

+
+

+ Check a prediction in the guided exercise → +

+
+ + Download the example yourself, or use your own app + +

+ Download example ZIP +

+

+ If your agent can’t download files, unzip this archive yourself. + Give it the path to the notes-example folder and + the same export question. +

+
+

+ Use REA to inspect the app at + /absolute/path/to/app. Find how it exports data + and show the relevant files and lines. +

+
+
+ How do I get a folder's full path? +
    +
  • + Windows: in File Explorer, right-click the + folder and choose Copy as path. On Windows 10, hold + Shift while right-clicking. +
    • +
  • + macOS: in Finder, right-click the folder, + hold Option and choose Copy "…" as Pathname. +
    • +
  • + Any system: drag the folder into a terminal + window and its path appears. +
    • +
+

+ In a terminal, put quotes around a path that contains spaces. +

+
+
+
+
+

Analyze another kind of software

+

+ Choose the section for your target. A provider is the local + analysis tool REA uses for that kind of software. +

+
+ + Native executables and libraries · Hopper, Ghidra or IDA + +

+ Install one analysis tool on the computer running REA. The host + is your operating system; the target is the program you want to + inspect. +

+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ProviderWindowsmacOSLinux
HopperUnavailable12+Supported distributions
Ghidrax64 · experimentalx64 / arm64x64 / arm64
IDAVerified profilesVerified headless profileSee provider guide
+
+
+ Hopper · let setup install or connect it +

+ Connect Hopper on macOS or Linux. Run the setup command above; + it detects an existing installation or offers an installation + plan for your approval. +

+

+ Choose Hopper in setup and review its separate installation + plan. On macOS, choose demo mode or activate your license when + Hopper first opens. On Linux, setup prepares the supported + demo and its private display. +

+

+ Linux hosts: Ubuntu 24.04+, Fedora 41+, Nobara 44+, Arch + Linux, CachyOS or Omarchy. Hopper is commercial software; its + demo has vendor-defined limits. +

+
+
+ Terminal +
+ 
npx -y rea-agents@latest doctor --provider hopper --json
+
+

+ If macOS asks for Automation permission, allow the app that + starts REA in System Settings → Privacy & Security → + Automation, then retry. +

+

+ Hopper setup and troubleshooting +

+
+
+ + Ghidra · install the free analysis tool and Java + +
    +
  1. + Install a 64-bit JDK 21 or newer, including the Java + compiler. Choose the package matching your computer's + architecture. +
    2. +
  2. + Download Ghidra 12.1.4 from the release's Assets list and + extract the ghidra_…_PUBLIC_….zip archive. + Choose the release archive, not “Source code”. +
    3. +
  3. + Find the extracted folder containing + ghidraRun or ghidraRun.bat. Use + that folder for GHIDRA_INSTALL_DIR. +
    4. +
+

+ Download a JDK + · + Download Ghidra 12.1.4 +

+

+ Check Java. The first command prints java.home; + use that folder for JAVA_HOME. The second + confirms the compiler is installed: +

+
+
+ Terminal +
+ 
java -XshowSettings:properties -version
+javac -version
+
+
+ Connect Ghidra on Windows +

+ Windows x64 has experimental, read-only support for native + x86 and x64 PE applications and DLLs on fixed local NTFS + drives. Use an existing Ghidra 12.1.x installation and a + 64-bit JDK 21 or newer: +

+
+
+ Windows PowerShell +
+ 
$env:GHIDRA_INSTALL_DIR = "C:/tools/ghidra_12.1.4_PUBLIC"
+$env:JAVA_HOME = "C:/tools/jdk-21"
+npx -y rea-agents@latest doctor --provider ghidra --json
+npx rea-agents@latest setup
+
+

+ Replace the two paths with your installations, then review + setup's plan. +

+
+
+ Connect your Ghidra installation +
+
+ macOS / Linux +
+ 
export GHIDRA_INSTALL_DIR="/absolute/path/to/ghidra_12.1.4_PUBLIC"
+export JAVA_HOME="/absolute/path/to/jdk-21"
+
+

+ Replace both paths with your existing installations. For a + fresh macOS or Linux arm64 installation, build the native + decompiler for your computer before continuing. +

+
+ Build Ghidra's native decompiler +

+ On macOS, the build needs Xcode Command Line Tools, make + and bison. Use the guide for your computer: +

+ +
+

Then check Ghidra and save its paths for your client:

+
+
+ macOS / Linux + +
+ 
npx -y rea-agents@latest doctor --provider ghidra --json
+npx rea-agents@latest setup
+
+
+

+ Keep these commands in the same terminal session. Rerun setup + after setting the paths, approve the updated client + configuration, then restart your agent. +

+

+ Ghidra setup and architecture requirements + · + Windows Ghidra guide +

+
+
+ IDA · connect an existing installation +

+ Use the IDA provider guide to prepare an upstream MCP + registration. Choose an attached GUI with the original binary + open, or a headless supervisor that opens a private database. +

+

+ Verified profiles include the Windows legacy GUI and headless + supervisor, plus a macOS arm64 headless supervisor with IDA + Professional 9.1. The optional native GUI bridge has its own + version and platform requirements. +

+

+ The macOS headless profile and native GUI bridge require a + REA source build + or a release containing those changes. npm + 6.4.0 does not include them. +

+

+ From the built checkout, run + node scripts/rea.mjs doctor --provider ida --json + and node scripts/rea.mjs setup instead of the + npx commands below. +

+
+ Prepare the IDA registration file +

+ Install and configure the upstream server using the IDA + guide below. Then save its working registration as + ida-mcp.json in a text editor. On Windows, + choose “All files” when saving in Notepad so it doesn't add + .txt. +

+

+ For a headless supervisor providing + idb_open, idb_list and + idb_close, the file can look like this: +

+
+
+ ida-mcp.json + +
+ 
{
+  "command": "idalib-mcp",
+  "args": ["--stdio", "--max-workers", "1"],
+  "mode": "headless"
+}
+
+

+ For an attached legacy 1.4.0 GUI server, copy its working + command, args and optional + env into the file. Open the original binary in + IDA and start its upstream plugin before asking REA to + inspect it. +

+
+
+
+ Windows PowerShell +
+ 
$env:REA_IDA_MCP_CONFIG = "C:/analysis/ida-mcp.json"
+$env:REA_ANALYSIS_PROVIDER = "ida"
+npx -y rea-agents@latest doctor --provider ida --json
+npx rea-agents@latest setup
+
+
+
+ macOS / Linux +
+ 
export REA_IDA_MCP_CONFIG="/absolute/path/to/ida-mcp.json"
+export REA_ANALYSIS_PROVIDER="ida"
+npx -y rea-agents@latest doctor --provider ida --json
+npx rea-agents@latest setup
+
+

+ Use the JSON registration file you prepared with the provider + guide, then rerun setup and restart your agent. +

+

+ IDA profiles, registration examples and verified + platforms +

+
+

+ Once connected, give your agent the original executable's full + path and name your provider: +

+
+

+ Use REA with my configured provider to inspect + /absolute/path/to/program. List its functions and + strings, then explain the part that handles user input with + references to the code. +

+
+

+ Follow a native analysis example +

+
+
+ + Websites, Android, .NET, firmware and other targets + +

+ Open the guide for the target you have. Each guide gives the + additional tools, supported hosts and first command. +

+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TargetTools to prepareGuide
JavaScript / ElectronNode.js + npmOpen guide
WebsiteChrome / Chromium / EdgeOpen guide
.NET / Flutter / packagesNode.js + npm + Open guide +
Android APKJADX + JDK 17+ + Open guide +
Android deviceadb + Open guide +
Android resourcesApktool + Java + Open guide +
JEB projectJEB + MCP + Open guide +
FirmwareBinwalk / Unblob · Linux + Open guide +
WASMWABT + Open guide +
SQLite databaseNode.js 24.15+ / 26+ + Open guide +
EVM bytecodeNode.js · Linux x64 + Open guide +
ELF / Linux crashpwntools / GDB / pwndbg + Open guide +
HAR / SAZ / mitmproxyNode.js / mitmdump + Open guide +
PCAP / PCAPNGTShark / capinfos · Linux + Open guide +
Logic signalssigrok-cli · Linux + Open guide +
Process behaviorPTY · Linux / macOS + Open guide +
+
+

+ Flutter guide + · + Packages and resources + · + EVM bytecode + · All investigation guides +

+
+
+
+

Use REA from a terminal

+
+ CLI commands, global installation and curl +

+ Analyze a JavaScript or Electron application folder or ASAR + archive without running the app: +

+
+
+ Static application analysis +
+ 
npx -y rea-agents@latest analyze-javascript-application "/absolute/path/to/app" --json
+
+

+ Replace the path with your extracted app directory or + .asar file. On Windows, a path such as + "D:/apps/example" works here. +

+
+ Where do I find an app's .asar file? +

+ Electron apps usually keep it in a + resources folder: +

+
    +
  • + Windows: + C:\Users\<you>\AppData\Local\Programs\<App>\resources\app.asar +
    • +
  • + macOS: + /Applications/<App>.app/Contents/Resources/app.asar + (right-click the app and choose + Show Package Contents) +
    • +
  • + Linux: often + /opt/<App>/resources/app.asar +
    • +
+

+ Some apps install elsewhere. Search the app's folder for + app.asar. +

+
+

+ The result identifies the app, shows connections between modules + and gives their source locations. Connections REA cannot resolve + are marked in the result. +

+

+ If you'd like a permanent rea command in your + shell: +

+
+
+ Install the CLI +
+ 
npm install --global rea-agents
+
+

+ Continue with the JavaScript application guide +

+
+
+ Terminal +
+ 
npx -y rea-agents@latest function "/absolute/path/to/program" main --provider ghidra --json
+
+

+ Replace main with a function name or address from + the target's function list. Ghidra must be configured first. +

+

+ On macOS or Linux, you can also use the curl installer after + preparing Node.js: +

+
+
+ macOS / Linux +
+ 
curl -fsSL https://raw.githubusercontent.com/morluto/rea/main/install.sh | bash
+
+

+ Follow the setup prompts to connect your agent. If they don't + appear, run rea setup. +

+

+ If your shell can't find rea, use the executable + path printed by the installer. On Linux, the default setup + command is "$HOME/.local/bin/rea" setup. +

+
+
+ +
+

Getting help

+
+ If your agent doesn't see REA +

+ Restart or reconnect the client after setup. Check its REA + connection with doctor; for Codex: +

+

+ Use the client check in step 3 with your setup ID. In editor + clients, enable REA in MCP settings and allow its tools in the + agent chat. +

+
+
+ If a provider needs attention +

Check the provider you're using. For Ghidra:

+
+
+ Check one provider +
+ 
npx -y rea-agents@latest doctor --provider ghidra --json
+
+

+ The report gives the failed check and a suggested next step. +

+
+
+ Windows, WSL and paths with spaces +

+ Use Windows PowerShell for Windows clients and analysis tools. + If your agent runs inside WSL, install Node.js and REA inside + that Linux environment and use Linux paths there. Headless + providers run on the same host as REA. +

+

+ In a terminal, put quotes around a path that contains spaces. +

+

Windows setup command:

+
+
+ Windows PowerShell +
+ 
npx.cmd rea-agents@latest setup
+
+
+
+ Update or remove REA +

+ To refresh your agent connection, run the latest setup command + again, review its plan and restart your client. +

+
+
+ Terminal +
+ 
npx rea-agents@latest setup
+
+

+ If you installed the global CLI, update it with + rea update. Use + npx rea-agents@latest uninstall to remove REA's + client configuration and skill. +

+

+ Update and uninstall details +

+
-
-

What is reverse engineering?

-

- Finding out how software works by - examining the program itself. -

+
+

How it works

-

- The goal: understand a feature well enough to - explain it, - change it or - rebuild it. +

+ Reverse engineering means learning how software works by inspecting + the program itself. With REA, your agent can follow the code, + explain a feature and help you rebuild it.

-

- One question: - why does Calculator give 220 for - 200 + 10%? -

-
-
-

- Before REA · You investigate it - yourself +

    +
  1. +

    Ask about one feature

    +

    Give your agent a program or page and a question.

    +
    2. +
  2. +

    Inspect the program

    +

    REA returns code, references and runtime observations.

    +
    3. +
  3. +

    Explain it, then try a change

    +

    + Your agent uses the findings to explain or rebuild the feature.

    +
    4. +
+
+ See the full Calculator investigation +

+ Why does Calculator give 220 for + 200 + 10%? +

+
- - - -
-
-
- - Original x64 · selected instructions - - - 
0x180124945: MOV EAX, dword ptr [R13 + 0x18]
+                  
+                  
+                  
+
+
+
+ + Original x64 · selected instructions + + + 
0x180124945: MOV EAX, dword ptr [R13 + 0x18]
 0x180124949: CMP EAX, 0x5c
 0x18012494c: JZ 0x180124aba
 0x180124952: CMP EAX, 0x5b
@@ -395,272 +421,179 @@
 0x180124a29: MOV RDX, RAX
 0x180124a2c: LEA RCX, [RSP + 0xb8]
 0x180124a34: CALL 0x180123c10
-
- -
-

- You decode the instructions, follow the calls and reconstruct - the calculation. -

-
-
-

- With REA · You ask - your agent -

-

One prompt.

-

- Ask your agent in plain English. -

-
-
- Example prompt · Calculator - + +

Illustrative expert reasoning

+
-

- Use REA to inspect Windows Calculator. Why does - 200 + 10% give 220? +

+ You decode the instructions, follow the calls and + reconstruct the calculation.

-
-

- Example answer · based on the findings +

+
+

Ask your agent the same question

+
+
+ Example prompt · Calculator + +
+

+ Use REA to inspect Windows Calculator. Why does + 200 + 10% give 220?

-

- After +, the % button takes - a percentage of the first number. -

-
- 10% of 200 = 20200 + 20 = 220 +
+

+ Example answer · based on the findings +

+

+ After +, the % button takes + a percentage of the first number. +

+
+ 10% of 200 = 20200 + 20 = 220 +
+
+
+
REA supplies
+
+ The handler’s instructions, decompiled code and calls. +
+
+
+
Your agent explains
+
+ Which branch applies, what it calculates and how to + rebuild it. +
+
+
-
-
-
REA supplies
-
- The handler’s instructions, decompiled code and calls. -
-
-
-
Your agent explains
-
- Which branch applies, what it calculates and how to rebuild - it. -
-
-
-
-
- Selected assembly from the installed Calculator DLL. The thoughts - and reply illustrate the findings. - Continue with the Calculator example ↓ -
-
+
+ Selected assembly from the installed Calculator DLL. The + thoughts and reply illustrate the findings. + Continue with the Calculator example ↓ +
+ +

- Let's try two examples. - Change a game's speed, then return to Calculator and rebuild its % - button. + Let’s try two examples: change a game’s speed, then rebuild + Calculator’s % button.

Example 01 · Chrome’s dinosaur game

Why does the dinosaur get faster?

-
-
-
Goal
-
- Rebuild the game with - adjustable speed. -
-
-
-
Why reverse engineer it?
-
- To reproduce its acceleration, we need - the rule in the running game: how fast it - starts, how much it increases and when it stops. -
-
-
+

+ The game gets faster as you run. To rebuild it with a speed control, + we first need to find out how the speed changes. +

-
-

Play the reconstruction

-
- - - - 6 → 10 → 13 - - - -
- -

- Original rule: start at 6, then increase toward 13. -

-
- Move the slider to set your own speed. Space or Jump clears a - cactus. -
- -
-
-
-

REA returned

-

- The script actually running in the page. The - agent reads its update function and speed settings, then uses - that rule in a new game. -

-
+
+

1. Ask about the speed

Example prompt · Dinosaur gameExample question · Dinosaur game
-

- Inspected target: - wayou’s Chromium-derived browser edition. +

2. Find the speed rule

+

+ REA lets the agent read the game’s running code. The speed + starts at 6 and increases a little each update while it is below + 13.

-
-
- REA · loaded index.js · - selected lines +
+

3. Play the result

+

+ We rebuilt that rule in the mini-game below and added a speed + slider. Press Play, then choose your own speed. +

+
+
+ + + + 6 → 10 → 13 + + + +
+ - 
if (this.currentSpeed < this.config.MAX_SPEED) {
-  this.currentSpeed += this.config.ACCELERATION;
-}
-
-

- Start at 6. Add 0.001 each - update without a collision, while speed is below - 13. -

+

+ Original rule: start at 6, then increase toward 13. +

+
+ Move the slider to set your own speed. Space or Jump clears a + cactus. +
+ + Open the speed lab →
-
-

What each part does

-
    -
  1. - 01 · REARead the running game’s scriptReturn its actual code and settings to the agent. -
    2. -
  2. - 02 · Your agentRebuild the rule, add a controlUse the recovered acceleration in a small game with a speed - slider. -
    3. -
  3. - 03 · YouChange the speed and playTry the original acceleration, then choose your own - pace. -
    4. -
-
See the script, checks and how to try the analysis +

+ Inspected target: + wayou’s Chromium-derived browser edition. +

+
+
+ REA · loaded index.js · selected + lines +
+ 
if (this.currentSpeed < this.config.MAX_SPEED) {
+  this.currentSpeed += this.config.ACCELERATION;
+}
+
+

+ Start at 6. Add 0.001 each update without a collision, while speed + is below 13. +

+
+

What each part does

+
    +
  1. + 01 · REARead the running game’s scriptReturn its actual code and settings to the agent. +
    2. +
  2. + 02 · Your agentRebuild the rule, add a controlUse the recovered acceleration in a small game with a speed + slider. +
    3. +
  3. + 03 · YouChange the speed and playTry the original acceleration, then choose your own + pace. +
    4. +
+

REA inspected the HTTP browser edition through a local debugging connection and returned the loaded index.js, @@ -792,111 +804,20 @@ SPEED: 6

Example 02 · Windows Calculator

Rebuild Calculator’s % button.

-

- We saw why - 200 + 10% gives 220. - Now let’s recover the rules for both + and ×, and try them. -

-
-
-
Goal
-
- Build a small calculator that handles - both + and × correctly. -
-
-
-
Why reverse engineer it?
-
- The same button uses - different rules after + and ×. We inspect the - installed app to find which number the percentage is applied to. -
-
-
+

+ The % button does different things after + and ×. Let’s find both + rules and try them in a small calculator. +

-
-

Try the calculation

- -
- 200 + - 20 -
- -

- 10% of 200 = 20. Then 200 + 20 = 220. -

-
- Interactive illustration of the inspected percentage rule. -
-
-
-
-

REA returned

-

- One branch divides by 100. The other also multiplies by the - first number. - The agent uses both to recreate the % button. -

-
-

- For this example, previous = 200 and - current = 10. -

+
+

1. Ask about the % button

Example prompt · Rebuild the % buttonExample question · Calculator
-
-
- Readable summary of the % handler -
- 
if (operation == multiply || operation == divide) {
-  percent = current / 100;
-} else {
-  percent = current * previous / 100;
-}
-
+

2. Find both rules

+

+ REA helps the agent read the code behind the % button. It + applies two different rules: +

With +
- Take 10% of the first number.Take 10% of the first number.10% of 200 = 20 → 200 + 20 = 220
With ×
- Turn 10% into 0.1.200 × 0.1 = 20 + Turn 10% into 0.1.200 × 0.1 = 20
+
+

3. Try the % button

+

+ We recreated both rules below. Choose + or ×, press %, then + press =. +

+
+ +
+ 200 + + 20 +
+ +

+ 10% of 200 = 20. Then 200 + 20 = 220. +

+
+ Interactive illustration of the inspected percentage rule. +
+
+
-
-

- From the installed app to a working % button -

-
    -
  1. - 01 · REA reads calc.exeIt launches the Calculator appThe code opens ms-calculator:. The agent then - selects the installed app’s library. -
    2. -
  2. - 02 · REA reads the libraryReturn the % handler’s codeThe function checks the operator and uses the constant - 100. -
    3. -
  3. - 03 · Your agent interprets itExplain and reproduce the rulesCrosscheck the branch with Microsoft’s source, then try + and - × above. -
    4. -
-
See the real code and how the answer was checked +

+ For this example, previous = 200 and + current = 10. +

+
+
+ Readable summary of the % handler +
+ 
if (operation == multiply || operation == divide) {
+  percent = current / 100;
+} else {
+  percent = current * previous / 100;
+}
+
+
+

+ From the installed app to a working % button +

+
    +
  1. + 01 · REA reads calc.exeIt launches the Calculator appThe code opens ms-calculator:. The agent then + selects the installed app’s library. +
    2. +
  2. + 02 · REA reads the libraryReturn the % handler’s codeThe function checks the operator and uses the constant + 100. +
    3. +
  3. + 03 · Your agent interprets itExplain and reproduce the rulesCrosscheck the branch with Microsoft’s source, then try + + and × above. +
    4. +
+
@@ -1024,6 +1019,56 @@ SPEED: 6
+
+
+
+
+

REA case studies.

+

+ Explore a feature in a released app or game, with code and + checked results. +

+
+ Showcases → +
+
+
+

Android · APK analysis

+

How an authenticator generates a login code

+

+ Trace Aegis's six-digit login code through its released APK, + then move a demo clock to see why it changes every 30 seconds. +

+ Read the Android case study +
+
+

Electron · Feature tracing

+

How Notion copies text and HTML

+

+ Follow Notion's packaged JavaScript to the system clipboard and + see how a copy preserves rich block data. +

+ Read the Notion case study +
+
+

Native · Reconstruction

+

DX-Ball's sound-pan calculation

+

+ Recover how a ball's position becomes left/right sound in a 1996 + Windows game. +

+ Read the DX-Ball case study +
+
+
+
+
@@ -1043,77 +1088,12 @@ SPEED: 6
-

Want to go deeper?

+

Choose your target.

- Tell your agent what you want to understand or build. With REA, - you can work together on anything from - cloning this website to - reconstructing a game from its executable. + These guides cover setup and a first investigation for each kind + of software.

-
-
-
-
- - Example prompt · Website - - -
-

- Use REA to inspect - https://rea.tools/. Clone this - website for me. -

-
-

Yes, this one.

-
-
-
-
- - Example prompt · Game reconstruction - - -
-

- Use REA to reconstruct this game from its executable. - Recover the gameplay logic in C and test it against the - original. -

-
-

- See the DX-Ball case study → -

-
-
-

- For setup and analysis steps, choose your target: -

Native binaries

@@ -1140,36 +1120,90 @@ SPEED: 6 Browser observation guide
-
-
-
-
-

Any questions?

-

- Read the FAQ, - chat with us on Discord, - or - open a GitHub issue - for bugs and feature requests. -

+
+ Try a larger project +
+
+
+
+ + Example prompt · Website + + +
+

+ Use REA to inspect + https://rea.tools/. Clone this + website for me. +

+
+

Yes, this one.

+
+
+
+
+ + Example prompt · Game reconstruction + + +
+

+ Use REA to reconstruct this game from its executable. + Recover the gameplay logic in C and test it against the + original. +

+
+

+ See the DX-Ball case study → +

+
+
+
-

Join the community

-

- Share investigations and get help from other REA users on Discord. - Follow REA on X for project updates. -

-
- Join the community - Follow @reatools on X +
+

Any questions?

+
+

+ Read the FAQ — setup and common questions. +

+

+ Join Discord — get + help and discuss your investigations. +

+

+ GitHub issues + — report a bug or request a feature. +

+
@@ -1180,7 +1214,8 @@ SPEED: 6

REA · Reverse Engineer Anything

- SourceContributorsSourceMIT licenseJoin the communityFollow us on X diff --git a/website/public/showcase/aegis/index.html b/website/public/showcase/aegis/index.html index 7396c49b9c..0b06b93306 100644 --- a/website/public/showcase/aegis/index.html +++ b/website/public/showcase/aegis/index.html @@ -51,6 +51,8 @@ + + @@ -116,9 +118,10 @@ Blog Guides FAQ + Contributors
-
English
+
English