feat(native): match retained function inventories across builds (#5390)
Match retained native inventories through shared CLI/MCP workflows with conservative symbol/byte correspondence, unresolved structural candidates, complete Evidence provenance and bounded paged reads. Closes #5343.
This commit is contained in:
1 parent
e84468c0bb
commit
1417bf943b
38 files changed
+3422
-72
No files matched your search
@@ -1,5 +1,16 @@
|
||||
# Native, managed, and packaged artifacts
|
||||
|
||||
For release comparisons, export `list_procedures` Evidence from each target before
|
||||
closing it. Close clears session records: import both bundles after collecting the
|
||||
two targets, or supply portable inline Evidence. `match_native_functions` pairs unique exact names and, when supplied,
|
||||
complete observed body bytes. Reuse optional same-context `list_names`,
|
||||
`procedure_info`, `read_bytes` and existing `analyze_function` Evidence; avoid
|
||||
decompiling every function merely to match it. Its summary retains a parent for
|
||||
`inspect_analysis_view` `function-matches` collections `matches` and `functions`.
|
||||
Select columns, batch IDs and page individual facts; aliases remain unresolved.
|
||||
Unpaired observations do not prove additions/removals. Existing matched dossier
|
||||
pairs include inputs for `compare_functions`; otherwise analyze just chosen pairs.
|
||||
|
||||
## Desktop installation directories
|
||||
|
||||
When an installation directory is supplied and its runtime is unknown, use
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
# MCP runtime contracts
|
||||
|
||||
For pairing native functions before comparing selected dossiers, see
|
||||
[native function matching](native-function-matching.md). It operates on retained
|
||||
inventories, preserves aliases/unknowns and exposes selected fact pages.
|
||||
|
||||
Use this reference to call REA tools and interpret their replies on one MCP
|
||||
connection. Start with [tool results](#tool-results), [retained Evidence](#retained-application-evidence-inputs),
|
||||
and [session discovery](#identity-and-discovery). See [transports](#transports)
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
# Match native functions across builds
|
||||
|
||||
`match_native_functions` pairs two existing `list_procedures` Evidence inventories. Select `list_procedures` `detail: "summary"` to retain
|
||||
complete source rows without transferring the entire inventory. Its
|
||||
`parent_evidence_id` is the reference matching consumes.
|
||||
Targets may be opened sequentially; matching never opens a target, rereads its
|
||||
file, executes it or starts a provider. Preserve each inventory before closing its
|
||||
target, or import its Evidence bundle later. `close_binary` clears session
|
||||
records. Export each bundle before closing; after collecting both sides and
|
||||
closing the last target, import both bundles, then use their references. Keeping
|
||||
only an Evidence ID across close is insufficient. Complete inline Evidence and
|
||||
CLI `source_bundle` inputs are portable alternatives. Pass Evidence or exact
|
||||
session references as `left` and `right`:
|
||||
|
||||
```json
|
||||
{
|
||||
"left": { "kind": "retained-evidence", "evidence_id": "ev_<left digest>" },
|
||||
"right": { "kind": "retained-evidence", "evidence_id": "ev_<right digest>" }
|
||||
}
|
||||
```
|
||||
|
||||
Unique exact reported names are paired first. Optional `list_names` Evidence
|
||||
identifies dynamic/default names to exclude. If origin is unavailable, the
|
||||
existing automatic-name exclusion rule applies and each function retains that
|
||||
limitation. Exact names establish name correspondence, not unchanged code or
|
||||
original-source identity.
|
||||
|
||||
For unmatched candidates, optionally supply `left_observations` and
|
||||
`right_observations`. `procedure_info` or existing `analyze_function` Evidence
|
||||
gives exact inclusive body ranges; `read_bytes` supplies memory bytes. Observations
|
||||
must belong to the side's exact target/provider/profile and document context.
|
||||
Adjacent, consistent overlapping and partially returned reads compose by their
|
||||
observed extent; conflicting intervals remain gaps. A byte match requires every
|
||||
body byte, equal total size, equal ordered range lengths, a unique SHA-256
|
||||
fingerprint and the same known architecture. Extents are never guessed from the
|
||||
next entry, and relocations are not normalized. A larger mapped read can serve
|
||||
several observed bodies; only needed ranges need to be captured.
|
||||
|
||||
This first slice reuses existing range observations. Stripped builds can require
|
||||
many lightweight range queries when those facts are not retained yet. Matching
|
||||
does not hide that acquisition cost or silently start those queries. Full
|
||||
decompilation is unnecessary.
|
||||
|
||||
Repeated candidates stay `unresolved` groups without a Cartesian pair expansion.
|
||||
`only_left`/`only_right` mean no exact counterpart in the supplied observations;
|
||||
renaming, inlining, stripping, changed bytes or incomplete discovery can explain
|
||||
them. They do not prove source addition/removal. `source_coverage` explicitly keeps
|
||||
inventory completeness unknown and separately counts known byte fingerprints and
|
||||
resolved correspondence. Provider limitations remain accessible.
|
||||
|
||||
Default `detail: "summary"` retains complete matching Evidence in MCP and returns
|
||||
counts, both identities and coverage. Query its `parent_evidence_id`:
|
||||
|
||||
```json
|
||||
{
|
||||
"source": {
|
||||
"kind": "retained-evidence",
|
||||
"evidence_id": "ev_<matching digest>"
|
||||
},
|
||||
"view": {
|
||||
"kind": "function-matches",
|
||||
"collection": "functions",
|
||||
"states": ["matched"],
|
||||
"sides": ["left"],
|
||||
"fields": ["name", "body_sha256"],
|
||||
"offset": 0,
|
||||
"limit": 20
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`collection: "matches"` returns relation identities, states, basis and member
|
||||
counts. Large member lists are omitted by default; select `left`/`right` columns
|
||||
explicitly or page individual `functions`. Batch `match_ids`/`function_ids`, exact function `names`, state
|
||||
and side filters intersect. Mandatory identities and provenance survive column
|
||||
selection. Page coverage covers selected retained facts. Function identity binds
|
||||
inventory Evidence and the owning inventory’s `source_index`, preserving same-address alternatives.
|
||||
One validation, lazy member index and bounded recent filter are retained per
|
||||
immutable parent; no live target state is cached.
|
||||
|
||||
When both functions already have unambiguous `analyze_function` dossiers, a matched
|
||||
relation includes `compare_functions_input` with directly usable retained
|
||||
references. Otherwise analyze just the selected canonical addresses in their
|
||||
respective targets, then compare those dossiers. Matching never claims behavioral
|
||||
equivalence. Optional informative CFG/call/string fingerprints from supplied
|
||||
dossiers connect unresolved candidate groups only. They require observed CFG,
|
||||
callees and strings, complete observed call coverage, no unresolved calls, and
|
||||
the same analysis-profile identity. Empty leaf-function buckets are excluded.
|
||||
`structural_candidate_sha256` identifies that derived candidate signature; its
|
||||
dossier link preserves the full supporting facts. Group membership does not
|
||||
assert that every member pair is a candidate or equivalent.
|
||||
|
||||
CLI uses the same service and JSON input:
|
||||
|
||||
```bash
|
||||
rea match-native-functions request.json \
|
||||
--source_bundle left-bundle.json --source_bundle right-bundle.json \
|
||||
--evidence_output matching-bundle.json --json
|
||||
```
|
||||
|
||||
`source_bundle` resolves exact references from authenticated local bundles.
|
||||
Default summary requires `evidence_output`, retaining full matching Evidence and
|
||||
its selected source closure, including transitive Evidence links across supplied
|
||||
bundles. Missing dependencies fail with an integrity error; unrelated records
|
||||
are omitted. Publication never overwrites. Set `detail: "complete"`
|
||||
in the JSON for all groups inline. Import the bundle into MCP for views, or
|
||||
extract its matching Evidence record to a JSON file for the existing CLI
|
||||
`inspect-analysis-view` `evidence-file` source. Input bundles are unchanged.
|
||||
|
||||
Representation admission is bounded at 96 MiB before cloning selected observations
|
||||
and expanding matching indexes and repeated per-function source links. Work yields to cancellation. No worker/provider
|
||||
concurrency is introduced; this budget is not a bound on total process RSS.
|
||||
|
||||
The self-owned source fixture is [nativeFunctionMatching.c](https://github.com/morluto/rea/blob/main/tests/fixtures/nativeFunctionMatching.c).
|
||||
On Linux x86-64, compile with `cc -O2 -g0 -fno-pie -no-pie -DVERSION=1` and
|
||||
`-DVERSION=2`; copy the second binary and run `strip --strip-all` on that copy.
|
||||
It includes unchanged/changed code, a byte-identical rename, a removed/added
|
||||
observation, an inlined function, identical-code aliases and a renamed changed
|
||||
function with an observed call/string signature. Analyze without executing the
|
||||
fixture. Full-provider verification still needs caller-supplied Ghidra and Java.
|
||||
@@ -0,0 +1,707 @@
|
||||
import { expect, it } from "vitest";
|
||||
import { createHash } from "node:crypto";
|
||||
import { matchNativeFunctions } from "./NativeFunctionMatchingService.js";
|
||||
import { nativeFunctionMatchingSchema } from "../../domain/native/functionMatching.js";
|
||||
import { createEvidence } from "../../domain/evidence.js";
|
||||
import { createAnalysisProfile } from "../../domain/analysisProfile.js";
|
||||
import { functionDossierSchema } from "../../domain/hopperValues.js";
|
||||
import {
|
||||
nativeMatchingInventory,
|
||||
nativeMatchingDetails,
|
||||
nativeMatchingTarget,
|
||||
nativeMatchingProvider,
|
||||
} from "../../../tests/fixtures/nativeFunctionMatching.js";
|
||||
|
||||
const compare = async (input: unknown) => {
|
||||
const result = await matchNativeFunctions(input);
|
||||
if (!result.ok) throw result.error;
|
||||
return {
|
||||
...result.value,
|
||||
report: nativeFunctionMatchingSchema.parse(
|
||||
result.value.evidence.normalized_result,
|
||||
),
|
||||
};
|
||||
};
|
||||
|
||||
it("retains transitive provenance without treating dependencies as native observations and fails closed for a missing ancestor", async () => {
|
||||
const ancestor = createEvidence(undefined, nativeMatchingProvider, {
|
||||
operation: "fixture_provenance",
|
||||
parameters: {},
|
||||
result: { provenance: "Synthetic ancestor" },
|
||||
});
|
||||
const dependency = createEvidence(undefined, nativeMatchingProvider, {
|
||||
operation: "fixture_dependency",
|
||||
parameters: {},
|
||||
result: { provenance: "Synthetic dependency" },
|
||||
evidenceLinks: [ancestor.evidence_id],
|
||||
});
|
||||
const left = nativeMatchingInventory(
|
||||
"1",
|
||||
[{ address: "0x10", value: "main" }],
|
||||
[dependency.evidence_id],
|
||||
);
|
||||
const right = nativeMatchingInventory(
|
||||
"2",
|
||||
[{ address: "0x20", value: "main" }],
|
||||
[dependency.evidence_id],
|
||||
);
|
||||
const dependencies = new Map(
|
||||
[ancestor, dependency].map((record) => [record.evidence_id, record]),
|
||||
);
|
||||
const result = await matchNativeFunctions({ left, right }, undefined, (id) =>
|
||||
dependencies.get(id),
|
||||
);
|
||||
if (!result.ok) throw result.error;
|
||||
expect(
|
||||
new Set(result.value.sources.map((record) => record.evidence_id)),
|
||||
).toEqual(
|
||||
new Set(
|
||||
[left, right, dependency, ancestor].map((record) => record.evidence_id),
|
||||
),
|
||||
);
|
||||
expect(new Set(result.value.evidence.evidence_links)).toEqual(
|
||||
new Set([left.evidence_id, right.evidence_id]),
|
||||
);
|
||||
expect(
|
||||
nativeFunctionMatchingSchema.parse(result.value.evidence.normalized_result)
|
||||
.summary.matched,
|
||||
).toBe(1);
|
||||
const missing = await matchNativeFunctions(
|
||||
{ left, right },
|
||||
undefined,
|
||||
(id) => (id === dependency.evidence_id ? dependency : undefined),
|
||||
);
|
||||
expect(missing.ok).toBe(false);
|
||||
if (missing.ok) throw new Error("Missing ancestor accepted");
|
||||
expect(missing.error).toMatchObject({
|
||||
evidenceId: ancestor.evidence_id,
|
||||
reason: "missing",
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps symbol origin unknown when supplied metadata describes a different same-address alias", async () => {
|
||||
const left = nativeMatchingInventory("1", [
|
||||
{ address: "0x10", value: "reported_name" },
|
||||
]);
|
||||
const right = nativeMatchingInventory("2", [
|
||||
{ address: "0x20", value: "reported_name" },
|
||||
]);
|
||||
const alias = createEvidence(
|
||||
nativeMatchingTarget("1"),
|
||||
nativeMatchingProvider,
|
||||
{
|
||||
operation: "list_names",
|
||||
parameters: {},
|
||||
result: [
|
||||
{
|
||||
address: "0x10",
|
||||
value: "other_alias",
|
||||
symbol: {
|
||||
primary: false,
|
||||
dynamic: false,
|
||||
external: false,
|
||||
type: "function",
|
||||
source: "user_defined",
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
);
|
||||
const { report } = await compare({ left, right, left_observations: [alias] });
|
||||
expect(report.summary.matched).toBe(1);
|
||||
expect(report.matches[0]?.left[0]?.limitations).toContain(
|
||||
"Symbol origin unavailable; exact-name matching uses the reported name and the existing automatic-name exclusion rule.",
|
||||
);
|
||||
expect(report.matches[0]?.left[0]?.evidence_links).toContain(
|
||||
alias.evidence_id,
|
||||
);
|
||||
});
|
||||
|
||||
it("preserves contradictory symbol origins from inventory/name facts or multiple name observations, independently of byte correspondence", async () => {
|
||||
const observe = (operation: string, generated: boolean) =>
|
||||
createEvidence(nativeMatchingTarget("1"), nativeMatchingProvider, {
|
||||
operation,
|
||||
parameters: {},
|
||||
result: [
|
||||
{
|
||||
address: "0x10",
|
||||
value: "same_name",
|
||||
symbol: {
|
||||
primary: true,
|
||||
dynamic: generated,
|
||||
external: false,
|
||||
type: "function",
|
||||
source: generated ? "default" : "user_defined",
|
||||
},
|
||||
},
|
||||
],
|
||||
});
|
||||
const right = nativeMatchingInventory("2", [
|
||||
{ address: "0x20", value: "same_name" },
|
||||
]);
|
||||
for (const [left, names] of [
|
||||
[observe("list_procedures", false), [observe("list_names", true)]],
|
||||
[observe("list_procedures", true), [observe("list_names", false)]],
|
||||
[
|
||||
nativeMatchingInventory("1", [{ address: "0x10", value: "same_name" }]),
|
||||
[observe("list_names", false), observe("list_names", true)],
|
||||
],
|
||||
[
|
||||
nativeMatchingInventory("1", [{ address: "0x10", value: "same_name" }]),
|
||||
[observe("list_names", true), observe("list_names", false)],
|
||||
],
|
||||
] as const) {
|
||||
for (const withBytes of [false, true]) {
|
||||
const { report } = await compare({
|
||||
left,
|
||||
right,
|
||||
left_observations: [
|
||||
...names,
|
||||
...(withBytes ? nativeMatchingDetails("1", "0x10", "c3") : []),
|
||||
],
|
||||
right_observations: withBytes
|
||||
? nativeMatchingDetails("2", "0x20", "c3")
|
||||
: [],
|
||||
});
|
||||
expect(report.summary.matched).toBe(withBytes ? 1 : 0);
|
||||
expect(report.matches[0]!.basis).toBe(
|
||||
withBytes ? "identical-bytes" : null,
|
||||
);
|
||||
const member = report.matches[0]!.left[0]!;
|
||||
expect(member.limitations).toContain(
|
||||
"Conflicting generated and named symbol-origin observations; exact-name matching is unavailable. Independent complete body bytes remain eligible.",
|
||||
);
|
||||
expect(member.limitations.join(" ")).not.toContain(
|
||||
"Symbol origin unavailable",
|
||||
);
|
||||
for (const source of names)
|
||||
expect(member.evidence_links).toContain(source.evidence_id);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
it("keeps informative CFG/call/string fingerprints as unresolved candidates and refuses incomplete structural observations", async () => {
|
||||
const profile = createAnalysisProfile(nativeMatchingProvider, {
|
||||
fixture: "structural-candidates-v1",
|
||||
});
|
||||
const observe = (
|
||||
side: "1" | "2",
|
||||
operation: string,
|
||||
result: Parameters<typeof createEvidence>[2]["result"],
|
||||
complete = true,
|
||||
) =>
|
||||
createEvidence(nativeMatchingTarget(side), nativeMatchingProvider, {
|
||||
operation,
|
||||
parameters: {},
|
||||
result,
|
||||
analysisProfile: profile,
|
||||
relationshipCoverage: complete
|
||||
? { status: "complete" }
|
||||
: { status: "unknown", reason: "Missing direct call coverage" },
|
||||
});
|
||||
const left = observe("1", "list_procedures", [
|
||||
{ address: "0x10", value: "old_behavior" },
|
||||
]);
|
||||
const right = observe("2", "list_procedures", [
|
||||
{ address: "0x20", value: "new_behavior" },
|
||||
]);
|
||||
const dossier = (side: "1" | "2", address: string) => {
|
||||
const value = functionDossierSchema.parse(
|
||||
nativeMatchingDetails(side, address, "0102", true)[0]!.normalized_result,
|
||||
);
|
||||
value.basic_blocks = [
|
||||
{
|
||||
start: address,
|
||||
end: `0x${(BigInt(address) + 2n).toString(16)}`,
|
||||
successors: [],
|
||||
},
|
||||
];
|
||||
value.callees = [
|
||||
{
|
||||
address: "0x500",
|
||||
name: "puts",
|
||||
classification: null,
|
||||
body: { available: false, reason: "Synthetic external target" },
|
||||
},
|
||||
];
|
||||
value.unresolved_calls = [];
|
||||
value.referenced_strings = [
|
||||
{ address: "0x900", value: "marker", source_address: address },
|
||||
];
|
||||
return value;
|
||||
};
|
||||
const l = dossier("1", "0x10"),
|
||||
r = dossier("2", "0x20");
|
||||
const { report } = await compare({
|
||||
left,
|
||||
right,
|
||||
left_observations: [observe("1", "analyze_function", l)],
|
||||
right_observations: [observe("2", "analyze_function", r)],
|
||||
});
|
||||
expect(report.summary).toEqual({
|
||||
matched: 0,
|
||||
only_left: 0,
|
||||
only_right: 0,
|
||||
unresolved: 1,
|
||||
});
|
||||
expect(report.matches[0]?.left[0]?.structural_candidate_sha256).toMatch(
|
||||
/^[a-f0-9]{64}$/u,
|
||||
);
|
||||
expect(report.matches[0]?.left[0]?.structural_candidate_sha256).toBe(
|
||||
report.matches[0]?.right[0]?.structural_candidate_sha256,
|
||||
);
|
||||
expect(report.matches[0]?.compare_functions_input).toBeNull();
|
||||
const incomplete = await compare({
|
||||
left,
|
||||
right,
|
||||
left_observations: [observe("1", "analyze_function", l, false)],
|
||||
right_observations: [observe("2", "analyze_function", r)],
|
||||
});
|
||||
expect(incomplete.report.summary.unresolved).toBe(0);
|
||||
expect(incomplete.report.source_coverage.left_structural_candidates).toBe(0);
|
||||
const legacy = { ...l };
|
||||
delete legacy.facet_availability;
|
||||
const missingFacets = await compare({
|
||||
left,
|
||||
right,
|
||||
left_observations: [observe("1", "analyze_function", legacy)],
|
||||
right_observations: [observe("2", "analyze_function", r)],
|
||||
});
|
||||
expect(missingFacets.report.source_coverage.left_structural_candidates).toBe(
|
||||
0,
|
||||
);
|
||||
expect(missingFacets.report.summary.unresolved).toBe(0);
|
||||
l.callees = [];
|
||||
l.referenced_strings = [];
|
||||
r.callees = [];
|
||||
r.referenced_strings = [];
|
||||
const uninformative = await compare({
|
||||
left,
|
||||
right,
|
||||
left_observations: [observe("1", "analyze_function", l)],
|
||||
right_observations: [observe("2", "analyze_function", r)],
|
||||
});
|
||||
expect(uninformative.report.source_coverage.left_structural_candidates).toBe(
|
||||
0,
|
||||
);
|
||||
expect(uninformative.report.summary.unresolved).toBe(0);
|
||||
});
|
||||
|
||||
it("matches symbols before bytes, retains exact renamed bytes, changed pairs, unpaired observations and duplicate candidates", async () => {
|
||||
const left = nativeMatchingInventory("1", [
|
||||
{ address: "0x1000", value: "stable" },
|
||||
{ address: "0x1010", value: "changed" },
|
||||
{ address: "0x1020", value: "old_name" },
|
||||
{ address: "0x1030", value: "removed_observation" },
|
||||
{ address: "0x1040", value: "duplicate" },
|
||||
{ address: "0x1050", value: "duplicate" },
|
||||
]);
|
||||
const right = nativeMatchingInventory("2", [
|
||||
{ address: "0x2000", value: "stable" },
|
||||
{ address: "0x2010", value: "changed" },
|
||||
{ address: "0x2020", value: "renamed" },
|
||||
{ address: "0x2030", value: "added_observation" },
|
||||
{ address: "0x2040", value: "duplicate" },
|
||||
]);
|
||||
const result = await compare({
|
||||
left,
|
||||
right,
|
||||
left_observations: [
|
||||
...nativeMatchingDetails("1", "0x1010", "0102"),
|
||||
...nativeMatchingDetails("1", "0x1020", "030405", true),
|
||||
],
|
||||
right_observations: [
|
||||
...nativeMatchingDetails("2", "0x2010", "aabb"),
|
||||
...nativeMatchingDetails("2", "0x2020", "030405", true),
|
||||
],
|
||||
});
|
||||
expect(result.report.summary).toEqual({
|
||||
matched: 3,
|
||||
only_left: 1,
|
||||
only_right: 1,
|
||||
unresolved: 1,
|
||||
});
|
||||
const renamed = result.report.matches.find(
|
||||
(row) => row.basis === "identical-bytes",
|
||||
)!;
|
||||
expect(renamed.left[0]).toMatchObject({
|
||||
name: "old_name",
|
||||
body_bytes: 3,
|
||||
body_sha256: createHash("sha256")
|
||||
.update(Buffer.from("030405", "hex"))
|
||||
.digest("hex"),
|
||||
});
|
||||
expect(renamed.right[0]?.name).toBe("renamed");
|
||||
expect(renamed.compare_functions_input?.left.evidence_id).toBe(
|
||||
renamed.left[0]?.dossier_evidence_id,
|
||||
);
|
||||
expect(
|
||||
result.report.matches.find((row) => row.left[0]?.name === "changed"),
|
||||
).toMatchObject({ state: "matched", basis: "exact-symbol" });
|
||||
expect(
|
||||
result.report.matches.find((row) => row.state === "unresolved"),
|
||||
).toMatchObject({
|
||||
left: [expect.anything(), expect.anything()],
|
||||
right: [expect.anything()],
|
||||
});
|
||||
expect(new Set(result.evidence.evidence_links)).toEqual(
|
||||
new Set(result.sources.map((x) => x.evidence_id)),
|
||||
);
|
||||
expect(result.report.limitations.join(" ")).toContain("never prove absence");
|
||||
});
|
||||
|
||||
it("does not force repeated identical-byte candidates after an exact name consumes an alias", async () => {
|
||||
const left = nativeMatchingInventory("1", [
|
||||
{ address: "0x10", value: "known" },
|
||||
{ address: "0x20", value: "FUN_20" },
|
||||
]);
|
||||
const right = nativeMatchingInventory("2", [
|
||||
{ address: "0x30", value: "known" },
|
||||
{ address: "0x40", value: "FUN_40" },
|
||||
]);
|
||||
const { report } = await compare({
|
||||
left,
|
||||
right,
|
||||
left_observations: [
|
||||
...nativeMatchingDetails("1", "0x10", "c3"),
|
||||
...nativeMatchingDetails("1", "0x20", "c3"),
|
||||
],
|
||||
right_observations: [
|
||||
...nativeMatchingDetails("2", "0x30", "c3"),
|
||||
...nativeMatchingDetails("2", "0x40", "c3"),
|
||||
],
|
||||
});
|
||||
expect(report.summary).toEqual({
|
||||
matched: 1,
|
||||
unresolved: 1,
|
||||
only_left: 0,
|
||||
only_right: 0,
|
||||
});
|
||||
expect(
|
||||
report.matches.find((x) => x.state === "unresolved")?.left[0]?.name,
|
||||
).toBe("FUN_20");
|
||||
});
|
||||
|
||||
it("reconstructs discontiguous bodies from adjacent byte reads without next-procedure guesses and keeps address-space/64-bit identity", async () => {
|
||||
const address = "ram:0xfffffffffffffff0";
|
||||
const left = nativeMatchingInventory("1", [{ address, value: "FUN_f0" }]);
|
||||
const right = nativeMatchingInventory("2", [
|
||||
{ address: "0x20", value: "FUN_20" },
|
||||
]);
|
||||
const body = {
|
||||
available: true,
|
||||
provenance: "synthetic noncontiguous body",
|
||||
ranges: [
|
||||
{ start: address, end: "ram:0xfffffffffffffff1" },
|
||||
{ start: "ram:0xfffffffffffffff8", end: "ram:0xfffffffffffffff8" },
|
||||
],
|
||||
total_bytes: 3,
|
||||
span_bytes: 9,
|
||||
non_contiguous: true,
|
||||
contains_entry: true,
|
||||
};
|
||||
const info = createEvidence(
|
||||
nativeMatchingTarget("1"),
|
||||
nativeMatchingProvider,
|
||||
{
|
||||
operation: "procedure_info",
|
||||
parameters: {},
|
||||
result: {
|
||||
name: "a",
|
||||
entrypoint: address,
|
||||
basicblock_count: 1,
|
||||
length: 3,
|
||||
signature: null,
|
||||
locals: [],
|
||||
classification: null,
|
||||
body,
|
||||
},
|
||||
},
|
||||
);
|
||||
const read = (address: string, hex: string) =>
|
||||
createEvidence(nativeMatchingTarget("1"), nativeMatchingProvider, {
|
||||
operation: "read_bytes",
|
||||
parameters: {},
|
||||
result: {
|
||||
address,
|
||||
requested_bytes: hex.length / 2,
|
||||
returned_bytes: hex.length / 2,
|
||||
bytes_hex: hex,
|
||||
complete: true,
|
||||
},
|
||||
});
|
||||
const rightDetails = nativeMatchingDetails("2", "0x20", "010203");
|
||||
const { report } = await compare({
|
||||
left,
|
||||
right,
|
||||
left_observations: [
|
||||
info,
|
||||
read(address, "01"),
|
||||
read("ram:0xfffffffffffffff1", "02"),
|
||||
read("ram:0xfffffffffffffff8", "03"),
|
||||
],
|
||||
right_observations: rightDetails,
|
||||
});
|
||||
// Same concatenated bytes with a different body layout do not become a pair.
|
||||
expect(report.summary.matched).toBe(0);
|
||||
expect(report.matches[0]?.left[0]).toMatchObject({
|
||||
address,
|
||||
body_bytes: 3,
|
||||
body_sha256: createHash("sha256")
|
||||
.update(Buffer.from("010203", "hex"))
|
||||
.digest("hex"),
|
||||
});
|
||||
});
|
||||
|
||||
it("preserves byte correspondence when consistent dossiers differ only in request identity, without choosing an arbitrary dossier", async () => {
|
||||
const left = nativeMatchingInventory("1", [
|
||||
{ address: "0x10", value: "old_name" },
|
||||
]);
|
||||
const right = nativeMatchingInventory("2", [
|
||||
{ address: "0x20", value: "new_name" },
|
||||
]);
|
||||
const leftDetails = nativeMatchingDetails("1", "0x10", "0102", true);
|
||||
const another = createEvidence(
|
||||
nativeMatchingTarget("1"),
|
||||
nativeMatchingProvider,
|
||||
{
|
||||
operation: "analyze_function",
|
||||
parameters: { procedure: "old_name" },
|
||||
result: leftDetails[0]!.normalized_result,
|
||||
relationshipCoverage: { status: "complete" },
|
||||
},
|
||||
);
|
||||
for (const observations of [
|
||||
[...leftDetails, another],
|
||||
[another, ...leftDetails],
|
||||
]) {
|
||||
const { report } = await compare({
|
||||
left,
|
||||
right,
|
||||
left_observations: observations,
|
||||
right_observations: nativeMatchingDetails("2", "0x20", "0102", true),
|
||||
});
|
||||
expect(report.summary.matched).toBe(1);
|
||||
expect(report.matches[0]).toMatchObject({
|
||||
basis: "identical-bytes",
|
||||
compare_functions_input: null,
|
||||
left: [
|
||||
{
|
||||
dossier_evidence_id: null,
|
||||
body_bytes: 2,
|
||||
body_sha256: createHash("sha256")
|
||||
.update(Buffer.from("0102", "hex"))
|
||||
.digest("hex"),
|
||||
},
|
||||
],
|
||||
});
|
||||
expect(report.matches[0]!.left[0]!.evidence_links).toContain(
|
||||
another.evidence_id,
|
||||
);
|
||||
}
|
||||
const conflicting = nativeMatchingDetails("1", "0x10", "010203", true)[0]!;
|
||||
const { report } = await compare({
|
||||
left,
|
||||
right,
|
||||
left_observations: [...leftDetails, conflicting],
|
||||
right_observations: nativeMatchingDetails("2", "0x20", "0102", true),
|
||||
});
|
||||
expect(report.summary.matched).toBe(0);
|
||||
expect(report.matches[0]!.left[0]!.body_sha256).toBeNull();
|
||||
});
|
||||
|
||||
it("rejects repeated per-occurrence source-link expansion within the shared representation budget", async () => {
|
||||
const base = nativeMatchingDetails("1", "0x10", "c3")[0]!;
|
||||
const observations = Array.from({ length: 250 }, (_, observation) =>
|
||||
createEvidence(nativeMatchingTarget("1"), nativeMatchingProvider, {
|
||||
operation: "procedure_info",
|
||||
parameters: { procedure: "0x10", observation },
|
||||
result: base.normalized_result,
|
||||
}),
|
||||
);
|
||||
const result = await matchNativeFunctions({
|
||||
left: nativeMatchingInventory(
|
||||
"1",
|
||||
Array.from({ length: 1000 }, (_, i) => ({
|
||||
address: "0x10",
|
||||
value: `alias_${i}`,
|
||||
})),
|
||||
),
|
||||
right: nativeMatchingInventory("2", []),
|
||||
left_observations: observations,
|
||||
});
|
||||
expect(result.ok).toBe(false);
|
||||
if (!result.ok)
|
||||
expect(result.error._tag).toBe("AnalysisResourceConstraintError");
|
||||
});
|
||||
|
||||
it("retains missing bytes and conflicting overlapping observations as unknown instead of exact matches", async () => {
|
||||
const left = nativeMatchingInventory("1", [
|
||||
{ address: "0x10", value: "FUN_10" },
|
||||
]);
|
||||
const right = nativeMatchingInventory("2", [
|
||||
{ address: "0x20", value: "FUN_20" },
|
||||
]);
|
||||
const observations = [
|
||||
...nativeMatchingDetails("1", "0x10", "0102"),
|
||||
nativeMatchingDetails("1", "0x10", "ffff")[1]!,
|
||||
];
|
||||
const { report } = await compare({
|
||||
left,
|
||||
right,
|
||||
left_observations: observations,
|
||||
right_observations: nativeMatchingDetails("2", "0x20", "0102"),
|
||||
});
|
||||
expect(report.summary.matched).toBe(0);
|
||||
expect(report.matches[0]?.left[0]?.body_sha256).toBeNull();
|
||||
});
|
||||
|
||||
it("composes consistent overlapping and partial byte reads without losing complete body coverage", async () => {
|
||||
const left = nativeMatchingInventory("1", [
|
||||
{ address: "0x10", value: "FUN_10" },
|
||||
]);
|
||||
const right = nativeMatchingInventory("2", [
|
||||
{ address: "0x20", value: "FUN_20" },
|
||||
]);
|
||||
const info = nativeMatchingDetails("1", "0x10", "01020304")[0]!;
|
||||
const read = (address: string, hex: string, requested = hex.length / 2) =>
|
||||
createEvidence(nativeMatchingTarget("1"), nativeMatchingProvider, {
|
||||
operation: "read_bytes",
|
||||
parameters: { address, length: requested },
|
||||
result: {
|
||||
address,
|
||||
requested_bytes: requested,
|
||||
returned_bytes: hex.length / 2,
|
||||
bytes_hex: hex,
|
||||
complete: requested === hex.length / 2,
|
||||
},
|
||||
});
|
||||
const { report } = await compare({
|
||||
left,
|
||||
right,
|
||||
left_observations: [
|
||||
info,
|
||||
read("0x10", "010203", 6),
|
||||
read("0x11", "020304"),
|
||||
],
|
||||
right_observations: nativeMatchingDetails("2", "0x20", "01020304"),
|
||||
});
|
||||
expect(report.summary.matched).toBe(1);
|
||||
expect(report.source_coverage).toMatchObject({
|
||||
status: "complete-within-inputs",
|
||||
left_byte_fingerprints: 1,
|
||||
right_byte_fingerprints: 1,
|
||||
left_inventory_completeness: "unknown",
|
||||
});
|
||||
});
|
||||
|
||||
it("does not use generated symbols and preserves literal named FUN_ symbols when producer metadata proves their origin", async () => {
|
||||
const entries = [
|
||||
{ address: "0x10", value: "FUN_10" },
|
||||
{ address: "0x20", value: "apparently_named" },
|
||||
];
|
||||
const names = (side: "1" | "2") =>
|
||||
createEvidence(nativeMatchingTarget(side), nativeMatchingProvider, {
|
||||
operation: "list_names",
|
||||
parameters: {},
|
||||
result: entries.map((entry, i) => ({
|
||||
...entry,
|
||||
symbol: {
|
||||
primary: true,
|
||||
dynamic: i === 1,
|
||||
external: false,
|
||||
type: "function",
|
||||
source: i === 0 ? "user_defined" : "default",
|
||||
},
|
||||
})),
|
||||
});
|
||||
const { report } = await compare({
|
||||
left: nativeMatchingInventory("1", entries),
|
||||
right: nativeMatchingInventory("2", entries),
|
||||
left_observations: [names("1")],
|
||||
right_observations: [names("2")],
|
||||
});
|
||||
expect(report.summary.matched).toBe(1);
|
||||
expect(report.matches[0]?.left[0]?.name).toBe("FUN_10");
|
||||
expect(report.matches[0]?.left[0]?.evidence_links).toContain(
|
||||
names("1").evidence_id,
|
||||
);
|
||||
});
|
||||
|
||||
it("rejects tampered, non-inventory and mixed-target observations with typed failures", async () => {
|
||||
const left = nativeMatchingInventory("1", [
|
||||
{ address: "0x10", value: "main" },
|
||||
]);
|
||||
for (const input of [
|
||||
{ left: { ...left, normalized_result: [] }, right: left },
|
||||
{ left: nativeMatchingDetails("1", "0x10", "c3")[0], right: left },
|
||||
{
|
||||
left,
|
||||
right: left,
|
||||
left_observations: nativeMatchingDetails("2", "0x10", "c3"),
|
||||
},
|
||||
{
|
||||
left,
|
||||
right: left,
|
||||
left_observations: [
|
||||
{ kind: "retained-evidence", evidence_id: `ev_${"a".repeat(64)}` },
|
||||
],
|
||||
},
|
||||
]) {
|
||||
const result = await matchNativeFunctions(input);
|
||||
expect(result.ok).toBe(false);
|
||||
if (!result.ok)
|
||||
expect([
|
||||
"AnalysisInputError",
|
||||
"EvidenceIntegrityError",
|
||||
"EvidenceReferenceError",
|
||||
]).toContain(result.error._tag);
|
||||
}
|
||||
});
|
||||
|
||||
it("cooperatively preserves the exact cancellation reason and rejects expansion before cloning oversized input", async () => {
|
||||
const controller = new AbortController(),
|
||||
reason = new Error("stop matching");
|
||||
const left = nativeMatchingInventory(
|
||||
"1",
|
||||
Array.from({ length: 2000 }, (_, i) => ({
|
||||
address: `0x${i.toString(16)}`,
|
||||
value: `item${i}`,
|
||||
})),
|
||||
);
|
||||
const pending = matchNativeFunctions(
|
||||
{ left, right: left },
|
||||
{ signal: controller.signal },
|
||||
);
|
||||
setImmediate(() => controller.abort(reason));
|
||||
const cancelled = await pending;
|
||||
expect(cancelled.ok).toBe(false);
|
||||
if (!cancelled.ok) {
|
||||
expect(cancelled.error._tag).toBe("AnalysisCancelledError");
|
||||
expect(cancelled.error.cause).toBe(reason);
|
||||
}
|
||||
const large = await matchNativeFunctions({
|
||||
left: { padding: "x".repeat(26 * 1024 * 1024) },
|
||||
right: left,
|
||||
});
|
||||
expect(large.ok).toBe(false);
|
||||
if (!large.ok)
|
||||
expect(large.error._tag).toBe("AnalysisResourceConstraintError");
|
||||
});
|
||||
|
||||
it("keeps duplicate candidate storage linear and emits every input occurrence exactly once", async () => {
|
||||
const entries = Array.from({ length: 1000 }, (_, i) => ({
|
||||
address: `0x${i.toString(16)}`,
|
||||
value: "same",
|
||||
}));
|
||||
const { report } = await compare({
|
||||
left: nativeMatchingInventory("1", entries),
|
||||
right: nativeMatchingInventory("2", entries),
|
||||
});
|
||||
expect(report.matches).toHaveLength(1);
|
||||
expect(report.matches[0]?.left).toHaveLength(1000);
|
||||
expect(report.matches[0]?.right).toHaveLength(1000);
|
||||
expect(new Set(report.matches[0]?.left.map((x) => x.function_id)).size).toBe(
|
||||
1000,
|
||||
);
|
||||
});
|
||||
@@ -0,0 +1,250 @@
|
||||
import { NativeMatchingAdmission } from "./NativeMatchingAdmission.js";
|
||||
import { loadNativeMatchingBundleSources } from "./NativeMatchingBundleSources.js";
|
||||
import { RegularFileCleanupFailure } from "../RegularFileRead.js";
|
||||
import { matchFunctionInventories } from "../../domain/native/matchFunctionInventories.js";
|
||||
import type { ArtifactResourceScope } from "../../artifacts/ArtifactResourceScope.js";
|
||||
import { setImmediate } from "node:timers/promises";
|
||||
import { matchNativeFunctionsInputSchema } from "../../contracts/native/functionMatchingContracts.js";
|
||||
import { retainedEvidenceReferenceSchema } from "../../domain/evidenceReference.js";
|
||||
import {
|
||||
createEvidence,
|
||||
parseEvidence,
|
||||
type Evidence,
|
||||
} from "../../domain/evidence.js";
|
||||
import { readFunctionInventory } from "../../domain/native/functionInventory.js";
|
||||
import { nativeFunctionMatchingSchema } from "../../domain/native/functionMatching.js";
|
||||
import { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||
import {
|
||||
AnalysisCancelledError,
|
||||
AnalysisInputError,
|
||||
} from "../../domain/analysisErrorCore.js";
|
||||
import { analysisInputErrorFromIssues } from "../../domain/inputIssueProjection.js";
|
||||
import { err, ok, type Result } from "../../domain/result.js";
|
||||
import {
|
||||
resolveEvidenceInput,
|
||||
type EvidenceLookup,
|
||||
} from "../EvidenceInputResolver.js";
|
||||
import type { ExecutionOptions } from "../AnalysisProvider.js";
|
||||
import { FUNCTION_COMPARISON_PROVIDER } from "../InvestigationProviders.js";
|
||||
|
||||
const OPERATION = "match_native_functions";
|
||||
export interface NativeFunctionMatchingResult {
|
||||
readonly evidence: Evidence;
|
||||
readonly sources: readonly Evidence[];
|
||||
}
|
||||
|
||||
/** Pair existing observations through one source/analysis admission. */
|
||||
export const matchNativeFunctions = (
|
||||
rawInput: unknown,
|
||||
options?: ExecutionOptions,
|
||||
lookup?: EvidenceLookup,
|
||||
): Promise<Result<NativeFunctionMatchingResult, AnalysisError>> =>
|
||||
matchWithAdmission(
|
||||
rawInput,
|
||||
options,
|
||||
lookup,
|
||||
new NativeMatchingAdmission(OPERATION, options?.signal),
|
||||
);
|
||||
|
||||
const matchWithAdmission = async (
|
||||
rawInput: unknown,
|
||||
options: ExecutionOptions | undefined,
|
||||
lookup: EvidenceLookup | undefined,
|
||||
admission: NativeMatchingAdmission,
|
||||
): Promise<Result<NativeFunctionMatchingResult, AnalysisError>> => {
|
||||
try {
|
||||
const checkpoint = admission.checkpoint;
|
||||
await setImmediate();
|
||||
await checkpoint();
|
||||
const parsed = matchNativeFunctionsInputSchema.safeParse(rawInput);
|
||||
if (!parsed.success)
|
||||
return err(
|
||||
analysisInputErrorFromIssues(OPERATION, parsed.error.issues, rawInput),
|
||||
);
|
||||
const sources = new Map<string, Evidence>();
|
||||
const resolve = async (input: unknown): Promise<Evidence> => {
|
||||
const reference = retainedEvidenceReferenceSchema.safeParse(input);
|
||||
let raw = input;
|
||||
if (reference.success) {
|
||||
const found = resolveEvidenceInput(reference.data, lookup);
|
||||
if (!found.ok) throw found.error;
|
||||
raw = found.value;
|
||||
}
|
||||
await admission.admit(raw);
|
||||
const evidence = parseEvidence(raw);
|
||||
const cached = sources.get(evidence.evidence_id);
|
||||
if (cached !== undefined) return cached;
|
||||
sources.set(evidence.evidence_id, evidence);
|
||||
return evidence;
|
||||
};
|
||||
const left = await resolve(parsed.data.left),
|
||||
right = await resolve(parsed.data.right);
|
||||
const observations = async (inputs: readonly unknown[]) => {
|
||||
const result: Evidence[] = [];
|
||||
const seen = new Set<string>();
|
||||
for (const input of inputs) {
|
||||
const record = await resolve(input);
|
||||
if (!seen.has(record.evidence_id)) {
|
||||
seen.add(record.evidence_id);
|
||||
result.push(record);
|
||||
}
|
||||
}
|
||||
return result;
|
||||
};
|
||||
const leftObservations = await observations(parsed.data.left_observations);
|
||||
const rightObservations = await observations(
|
||||
parsed.data.right_observations,
|
||||
);
|
||||
const directLinks = [...sources.keys()].sort();
|
||||
// Expand provenance separately from role observations: dependencies may be
|
||||
// any Evidence operation and must not be interpreted as native body facts.
|
||||
for (const source of sources.values())
|
||||
for (const id of source.evidence_links)
|
||||
if (!sources.has(id))
|
||||
await resolve({ kind: "retained-evidence", evidence_id: id });
|
||||
const leftFunctions = await readFunctionInventory(
|
||||
left,
|
||||
leftObservations,
|
||||
checkpoint,
|
||||
);
|
||||
const rightFunctions = await readFunctionInventory(
|
||||
right,
|
||||
rightObservations,
|
||||
checkpoint,
|
||||
);
|
||||
const compatible =
|
||||
left.subject?.architecture != null &&
|
||||
left.subject.architecture === right.subject?.architecture;
|
||||
const { matches, summary } = await matchFunctionInventories(
|
||||
leftFunctions,
|
||||
rightFunctions,
|
||||
{ identicalBytesComparable: compatible, checkpoint },
|
||||
);
|
||||
const limitations = [
|
||||
...new Set([
|
||||
...left.limitations.map((x) => `Left: ${x}`),
|
||||
...right.limitations.map((x) => `Right: ${x}`),
|
||||
"Coverage is restricted to the supplied provider-observed inventories; missing/truncated inventory facts never prove absence from either binary.",
|
||||
"Only unique exact symbols and bytes are paired. Compatible observed CFG/call/string fingerprints propose unresolved candidates, never identity or equivalence. Missing structural coverage stays unknown.",
|
||||
"Name or byte correspondence is not behavioral equivalence; use analyze_function for selected pairs, then compare_functions. Existing dossier pairs include directly usable retained-reference inputs.",
|
||||
...(!compatible
|
||||
? [
|
||||
"Identical-byte matching unavailable: input architectures are missing or differ.",
|
||||
]
|
||||
: []),
|
||||
]),
|
||||
];
|
||||
const report = nativeFunctionMatchingSchema.parse({
|
||||
left: {
|
||||
evidence_id: left.evidence_id,
|
||||
sha256: left.subject!.digest.sha256,
|
||||
functions: leftFunctions.length,
|
||||
},
|
||||
right: {
|
||||
evidence_id: right.evidence_id,
|
||||
sha256: right.subject!.digest.sha256,
|
||||
functions: rightFunctions.length,
|
||||
},
|
||||
scope: "provider-observed-inputs",
|
||||
source_coverage: {
|
||||
status:
|
||||
summary.only_left + summary.only_right + summary.unresolved > 0
|
||||
? "partial"
|
||||
: "complete-within-inputs",
|
||||
left_inventory_completeness: "unknown",
|
||||
right_inventory_completeness: "unknown",
|
||||
left_byte_fingerprints: leftFunctions.filter(
|
||||
(x) => x.fingerprint !== null,
|
||||
).length,
|
||||
left_structural_candidates: leftFunctions.filter(
|
||||
(x) => x.member.structural_candidate_sha256 !== null,
|
||||
).length,
|
||||
right_structural_candidates: rightFunctions.filter(
|
||||
(x) => x.member.structural_candidate_sha256 !== null,
|
||||
).length,
|
||||
right_byte_fingerprints: rightFunctions.filter(
|
||||
(x) => x.fingerprint !== null,
|
||||
).length,
|
||||
},
|
||||
summary,
|
||||
matches,
|
||||
limitations,
|
||||
});
|
||||
await checkpoint();
|
||||
const evidence = createEvidence(undefined, FUNCTION_COMPARISON_PROVIDER, {
|
||||
operation: OPERATION,
|
||||
predicateType: "rea.native-function-matching",
|
||||
confidence: "derived",
|
||||
authority: "analyst-inference",
|
||||
parameters: {
|
||||
left_evidence_id: left.evidence_id,
|
||||
right_evidence_id: right.evidence_id,
|
||||
left_observation_ids: leftObservations.map((x) => x.evidence_id),
|
||||
right_observation_ids: rightObservations.map((x) => x.evidence_id),
|
||||
algorithm: "exact-symbol-body-bytes-and-structural-candidates-v1",
|
||||
},
|
||||
result: report,
|
||||
limitations,
|
||||
evidenceLinks: directLinks,
|
||||
});
|
||||
await checkpoint();
|
||||
return ok({ evidence, sources: [...sources.values()] });
|
||||
} catch (cause: unknown) {
|
||||
if (options?.signal?.aborted)
|
||||
return err(
|
||||
new AnalysisCancelledError(OPERATION, { cause: options.signal.reason }),
|
||||
);
|
||||
if (cause instanceof AnalysisError) return err(cause);
|
||||
return err(
|
||||
new AnalysisInputError(OPERATION, { cause }, [
|
||||
{
|
||||
path: [],
|
||||
reason: "invalid_value",
|
||||
message:
|
||||
cause instanceof Error
|
||||
? cause.message
|
||||
: "Invalid native matching observations",
|
||||
},
|
||||
]),
|
||||
);
|
||||
}
|
||||
};
|
||||
|
||||
/** Load selected portable records; do not retain unrelated records across bundles. */
|
||||
export const matchNativeFunctionsFromBundles = async (
|
||||
input: unknown,
|
||||
paths: readonly string[],
|
||||
resources: ArtifactResourceScope,
|
||||
options?: ExecutionOptions,
|
||||
): Promise<Result<NativeFunctionMatchingResult, AnalysisError>> => {
|
||||
try {
|
||||
const parsed = matchNativeFunctionsInputSchema.safeParse(input);
|
||||
if (!parsed.success)
|
||||
return err(
|
||||
analysisInputErrorFromIssues(OPERATION, parsed.error.issues, input),
|
||||
);
|
||||
const admission = new NativeMatchingAdmission(OPERATION, options?.signal);
|
||||
const loaded = await loadNativeMatchingBundleSources({
|
||||
input: parsed.data,
|
||||
paths,
|
||||
resources,
|
||||
admission,
|
||||
});
|
||||
return await matchWithAdmission(
|
||||
loaded.input,
|
||||
options,
|
||||
loaded.lookup,
|
||||
admission,
|
||||
);
|
||||
} catch (cause: unknown) {
|
||||
// This marker carries the exact descriptor owner; the caller must finish
|
||||
// that owner before classifying cancellation or unexpected I/O failures.
|
||||
if (cause instanceof RegularFileCleanupFailure) throw cause;
|
||||
if (options?.signal?.aborted)
|
||||
return err(
|
||||
new AnalysisCancelledError(OPERATION, { cause: options.signal.reason }),
|
||||
);
|
||||
if (cause instanceof AnalysisError) return err(cause);
|
||||
return err(new AnalysisInputError(OPERATION, { cause }));
|
||||
}
|
||||
};
|
||||
@@ -0,0 +1,101 @@
|
||||
import { expect, it } from "vitest";
|
||||
import { matchNativeFunctions } from "./NativeFunctionMatchingService.js";
|
||||
import { inspectAnalysisView } from "../analysisView/AnalysisViewService.js";
|
||||
import { nativeMatchingInventory } from "../../../tests/fixtures/nativeFunctionMatching.js";
|
||||
import { analysisViewResultSchema } from "../../domain/analysisView/analysisView.js";
|
||||
|
||||
it("pages individual function facts without transferring an entire ambiguous alias group, preserving mandatory identity and source coverage", async () => {
|
||||
const entries = Array.from({ length: 500 }, (_, i) => ({
|
||||
address: `0x${i.toString(16)}`,
|
||||
value: "same",
|
||||
}));
|
||||
const matched = await matchNativeFunctions({
|
||||
left: nativeMatchingInventory("1", entries),
|
||||
right: nativeMatchingInventory("2", entries),
|
||||
});
|
||||
if (!matched.ok) throw matched.error;
|
||||
const source = { kind: "inline", evidence: matched.value.evidence };
|
||||
const view = (query: unknown) => {
|
||||
const result = inspectAnalysisView({ source, view: query });
|
||||
if (!result.ok) throw result.error;
|
||||
return analysisViewResultSchema.parse(result.value.normalized_result);
|
||||
};
|
||||
const groups = view({
|
||||
kind: "function-matches",
|
||||
collection: "matches",
|
||||
fields: ["basis"],
|
||||
limit: 1,
|
||||
});
|
||||
expect(groups).toMatchObject({
|
||||
kind: "function-matches",
|
||||
coverage: { examined: 1, total: 1, exhausted: true },
|
||||
items: [
|
||||
{ state: "unresolved", basis: null, left_count: 500, right_count: 500 },
|
||||
],
|
||||
});
|
||||
if (groups.kind !== "function-matches") throw new Error("Wrong view");
|
||||
expect(groups.items[0]).not.toHaveProperty("left");
|
||||
const member = `${matched.value.sources[0]!.evidence_id}/123`;
|
||||
const functions = view({
|
||||
kind: "function-matches",
|
||||
collection: "functions",
|
||||
function_ids: [member],
|
||||
fields: ["name"],
|
||||
limit: 1,
|
||||
});
|
||||
expect(functions).toMatchObject({
|
||||
kind: "function-matches",
|
||||
coverage: { examined: 1, total: 1, exhausted: true },
|
||||
items: [
|
||||
{
|
||||
function_id: member,
|
||||
source_index: 123,
|
||||
address: "0x7b",
|
||||
name: "same",
|
||||
side: "left",
|
||||
state: "unresolved",
|
||||
evidence_links: [matched.value.sources[0]!.evidence_id],
|
||||
},
|
||||
],
|
||||
});
|
||||
expect(JSON.stringify(functions).length).toBeLessThan(6000);
|
||||
expect(
|
||||
view({
|
||||
kind: "function-matches",
|
||||
collection: "functions",
|
||||
names: [],
|
||||
limit: 1,
|
||||
}),
|
||||
).toMatchObject({ coverage: { total: 0 } });
|
||||
expect(
|
||||
view({
|
||||
kind: "function-matches",
|
||||
collection: "functions",
|
||||
names: ["same"],
|
||||
sides: ["left"],
|
||||
limit: 1,
|
||||
}),
|
||||
).toMatchObject({ coverage: { total: 500, next_offset: 1 } });
|
||||
expect(
|
||||
view({
|
||||
kind: "function-matches",
|
||||
collection: "functions",
|
||||
sides: ["right"],
|
||||
offset: 499,
|
||||
limit: 2,
|
||||
}),
|
||||
).toMatchObject({
|
||||
coverage: { examined: 1, total: 500, exhausted: true, next_offset: null },
|
||||
});
|
||||
expect(
|
||||
view({ kind: "function-matches", collection: "functions", states: [] }),
|
||||
).toMatchObject({ coverage: { total: 0, exhausted: true } });
|
||||
expect(view({ kind: "summary" })).toMatchObject({
|
||||
summary: {
|
||||
scope: "provider-observed-inputs",
|
||||
summary: { unresolved: 1 },
|
||||
left: { functions: 500 },
|
||||
right: { functions: 500 },
|
||||
},
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,60 @@
|
||||
import { setImmediate } from "node:timers/promises";
|
||||
import { AnalysisResourceConstraintError } from "../../domain/analysisErrorCore.js";
|
||||
|
||||
const NATIVE_MATCHING_REPRESENTATION_BYTES = 96 * 1024 * 1024;
|
||||
|
||||
/** Budget the actual selected observations, before cloning or index allocation. */
|
||||
export class NativeMatchingAdmission {
|
||||
#remaining = NATIVE_MATCHING_REPRESENTATION_BYTES;
|
||||
#work = 0;
|
||||
readonly #seen = new WeakSet<object>();
|
||||
constructor(
|
||||
readonly operation: string,
|
||||
readonly signal?: AbortSignal,
|
||||
) {}
|
||||
readonly checkpoint = async (bytes = 0): Promise<void> => {
|
||||
this.#remaining -= bytes;
|
||||
if (this.#remaining < 0)
|
||||
throw new AnalysisResourceConstraintError(
|
||||
this.operation,
|
||||
"capacity",
|
||||
"Native function matching exceeds representation capacity",
|
||||
{ maximum_bytes: NATIVE_MATCHING_REPRESENTATION_BYTES },
|
||||
);
|
||||
this.signal?.throwIfAborted();
|
||||
if (++this.#work % 256 === 0)
|
||||
await setImmediate(
|
||||
undefined,
|
||||
this.signal === undefined ? {} : { signal: this.signal },
|
||||
);
|
||||
};
|
||||
async admit(root: unknown): Promise<void> {
|
||||
const pending: Iterator<unknown>[] = [[root][Symbol.iterator]()];
|
||||
while (pending.length) {
|
||||
const next = pending[pending.length - 1]!.next();
|
||||
if (next.done) {
|
||||
pending.pop();
|
||||
continue;
|
||||
}
|
||||
const value = next.value;
|
||||
if (typeof value === "object" && value !== null) {
|
||||
if (this.#seen.has(value)) continue;
|
||||
this.#seen.add(value);
|
||||
await this.checkpoint(128);
|
||||
// Iterator traversal avoids a second width-sized array of child values.
|
||||
pending.push(
|
||||
(function* () {
|
||||
for (const key in value)
|
||||
if (Object.hasOwn(value, key)) {
|
||||
yield key;
|
||||
yield (value as Record<string, unknown>)[key];
|
||||
}
|
||||
})(),
|
||||
);
|
||||
} else
|
||||
await this.checkpoint(
|
||||
typeof value === "string" ? 64 + value.length * 4 : 128,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,165 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import type { z } from "zod";
|
||||
import type { ArtifactResourceScope } from "../../artifacts/ArtifactResourceScope.js";
|
||||
import type { EvidenceBundle } from "../../domain/evidenceBundle.js";
|
||||
import { immutableEvidence, type Evidence } from "../../domain/evidence.js";
|
||||
import {
|
||||
EvidenceIntegrityError,
|
||||
EvidenceReferenceError,
|
||||
} from "../../domain/evidenceErrors.js";
|
||||
import { digestCanonicalValue } from "../../domain/canonicalDigest.js";
|
||||
import { retainedEvidenceReferenceSchema } from "../../domain/evidenceReference.js";
|
||||
import type { matchNativeFunctionsInputSchema } from "../../contracts/native/functionMatchingContracts.js";
|
||||
import { readImmutableEvidenceBundle } from "../EvidenceBundleFiles.js";
|
||||
import type { EvidenceLookup } from "../EvidenceInputResolver.js";
|
||||
import type { NativeMatchingAdmission } from "./NativeMatchingAdmission.js";
|
||||
|
||||
type Request = z.output<typeof matchNativeFunctionsInputSchema>;
|
||||
interface SourceEntry {
|
||||
readonly file: number;
|
||||
readonly links: readonly string[];
|
||||
}
|
||||
interface Options {
|
||||
readonly input: Request;
|
||||
readonly paths: readonly string[];
|
||||
readonly resources: ArtifactResourceScope;
|
||||
readonly admission: NativeMatchingAdmission;
|
||||
}
|
||||
const reference = (evidence_id: string) => ({
|
||||
kind: "retained-evidence" as const,
|
||||
evidence_id,
|
||||
});
|
||||
// IDs and links have authenticated ASCII digest syntax. Own these short strings
|
||||
// independently of the discarded bundle's potentially large JSON backing store.
|
||||
const ownId = (value: string): string =>
|
||||
Buffer.from(value, "ascii").toString("ascii");
|
||||
|
||||
/** IDs bind semantic payloads; subjects add the display metadata IDs intentionally omit. */
|
||||
const recordSnapshot = async (
|
||||
bundle: EvidenceBundle,
|
||||
admission: NativeMatchingAdmission,
|
||||
): Promise<string> => {
|
||||
const hash = createHash("sha256");
|
||||
for (const record of bundle.records) {
|
||||
await admission.checkpoint();
|
||||
hash
|
||||
.update(record.evidence_id)
|
||||
.update(digestCanonicalValue(record.subject));
|
||||
}
|
||||
return hash.digest("hex");
|
||||
};
|
||||
|
||||
/** Index only graph metadata; keep at most the last whole bundle between passes. */
|
||||
export const loadNativeMatchingBundleSources = async ({
|
||||
input,
|
||||
paths,
|
||||
resources,
|
||||
admission,
|
||||
}: Options): Promise<{ input: Request; lookup: EvidenceLookup }> => {
|
||||
const records = new Map<string, Evidence>(),
|
||||
index = new Map<string, SourceEntry>();
|
||||
const rootIds: string[] = [];
|
||||
for (const value of [
|
||||
input.left,
|
||||
input.right,
|
||||
...input.left_observations,
|
||||
...input.right_observations,
|
||||
]) {
|
||||
const parsed = retainedEvidenceReferenceSchema.safeParse(value);
|
||||
if (parsed.success) rootIds.push(parsed.data.evidence_id);
|
||||
else {
|
||||
await admission.admit(value);
|
||||
const record = immutableEvidence(value);
|
||||
rootIds.push(record.evidence_id);
|
||||
records.set(record.evidence_id, record);
|
||||
index.set(record.evidence_id, { file: -1, links: record.evidence_links });
|
||||
}
|
||||
}
|
||||
const snapshots: string[] = [];
|
||||
let last: { file: number; bundle: EvidenceBundle } | undefined;
|
||||
for (let file = 0; file < paths.length; file++) {
|
||||
last = undefined;
|
||||
const loaded = await readImmutableEvidenceBundle(
|
||||
paths[file]!,
|
||||
admission.signal,
|
||||
resources,
|
||||
);
|
||||
if (!loaded.ok) throw loaded.error;
|
||||
snapshots.push(await recordSnapshot(loaded.value, admission));
|
||||
for (const record of loaded.value.records) {
|
||||
await admission.checkpoint(
|
||||
256 +
|
||||
4 *
|
||||
(record.evidence_id.length +
|
||||
record.evidence_links.reduce((sum, id) => sum + id.length, 0)),
|
||||
);
|
||||
if (!index.has(record.evidence_id))
|
||||
index.set(ownId(record.evidence_id), {
|
||||
file,
|
||||
links: record.evidence_links.map(ownId),
|
||||
});
|
||||
}
|
||||
last = { file, bundle: loaded.value };
|
||||
}
|
||||
const needed = new Set<string>(rootIds),
|
||||
files = new Set<number>();
|
||||
// Set iteration includes newly discovered nodes. Shared ancestors and cycles
|
||||
// are visited once, independent of source-file and record ordering.
|
||||
for (const id of needed) {
|
||||
await admission.checkpoint(128);
|
||||
const entry = index.get(id);
|
||||
if (entry === undefined)
|
||||
throw new EvidenceReferenceError(
|
||||
id,
|
||||
"missing",
|
||||
"Complete Evidence closure in the supplied bundles or inline inputs",
|
||||
null,
|
||||
);
|
||||
if (entry.file >= 0) files.add(entry.file);
|
||||
for (const linked of entry.links) needed.add(linked);
|
||||
}
|
||||
const retain = async (
|
||||
bundle: EvidenceBundle,
|
||||
file: number,
|
||||
): Promise<void> => {
|
||||
for (const record of bundle.records)
|
||||
if (
|
||||
needed.has(record.evidence_id) &&
|
||||
index.get(record.evidence_id)?.file === file
|
||||
) {
|
||||
await admission.admit(record);
|
||||
records.set(record.evidence_id, record);
|
||||
}
|
||||
};
|
||||
if (last !== undefined && files.has(last.file)) {
|
||||
await retain(last.bundle, last.file);
|
||||
files.delete(last.file);
|
||||
}
|
||||
last = undefined;
|
||||
for (const file of files) {
|
||||
const loaded = await readImmutableEvidenceBundle(
|
||||
paths[file]!,
|
||||
admission.signal,
|
||||
resources,
|
||||
);
|
||||
if (!loaded.ok) throw loaded.error;
|
||||
if ((await recordSnapshot(loaded.value, admission)) !== snapshots[file])
|
||||
throw new EvidenceIntegrityError(
|
||||
`Evidence bundle records changed during closure lookup: ${paths[file]}`,
|
||||
);
|
||||
await retain(loaded.value, file);
|
||||
}
|
||||
const left = rootIds[0]!,
|
||||
right = rootIds[1]!;
|
||||
const split = 2 + input.left_observations.length;
|
||||
return {
|
||||
input: {
|
||||
...input,
|
||||
left: reference(left),
|
||||
right: reference(right),
|
||||
left_observations: rootIds.slice(2, split).map(reference),
|
||||
right_observations: rootIds.slice(split).map(reference),
|
||||
},
|
||||
lookup: (id) => records.get(id),
|
||||
};
|
||||
};
|
||||
@@ -1,3 +1,4 @@
|
||||
import { registerNativeFunctionMatchingCommands } from "./cli/nativeFunctionMatchingCommands.js";
|
||||
import { registerCaptureCommands } from "./cli/captureCommands.js";
|
||||
import { registerSignalCommands } from "./cli/signalCommands.js";
|
||||
import { registerWasmCommands } from "./cli/wasmCommands.js";
|
||||
@@ -134,6 +135,7 @@ export const createCli = (
|
||||
registerFirmwareCommands(cli, logger, environment);
|
||||
registerBinaryDiagnosticsCommands(cli, logger, environment);
|
||||
registerAnalysisViewCommands(cli, logger);
|
||||
registerNativeFunctionMatchingCommands(cli, logger);
|
||||
registerFridaCommands(cli, logger, fridaInstrumentation);
|
||||
registerEvidenceCommands(cli, logger);
|
||||
registerEnvironmentCommands(cli, logger, environment);
|
||||
|
||||
@@ -39,6 +39,8 @@ interface CliEvidenceDelivery {
|
||||
readonly signal: AbortSignal;
|
||||
readonly detail: "complete" | "summary";
|
||||
readonly evidenceOutput: string | undefined;
|
||||
/** Derived workflows publish their selected source closure with the result. */
|
||||
readonly sourceEvidence?: readonly Evidence[];
|
||||
}
|
||||
|
||||
export const cliEvidenceDeliveryInputError = (
|
||||
@@ -115,7 +117,14 @@ export const publishCliAnalysisEvidence = async (
|
||||
const delivered = await finishAnalysisCommand(
|
||||
async () => {
|
||||
const written = await writeEvidenceBundle(
|
||||
createEvidenceBundle([complete]),
|
||||
createEvidenceBundle([
|
||||
...new Map(
|
||||
[...(input.sourceEvidence ?? []), complete].map((record) => [
|
||||
record.evidence_id,
|
||||
record,
|
||||
]),
|
||||
).values(),
|
||||
]),
|
||||
absoluteCallerPath(evidenceOutput),
|
||||
false,
|
||||
resources,
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
import { z } from "incur";
|
||||
import type { Logger } from "pino";
|
||||
import { matchNativeFunctionsFromBundles } from "../application/native/NativeFunctionMatchingService.js";
|
||||
import { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
|
||||
import { matchNativeFunctionsInputSchema } from "../contracts/native/functionMatchingContracts.js";
|
||||
import { CLI_COMMANDS } from "../cliCommandNames.js";
|
||||
import { parseCliJsonInput } from "../cliJsonInput.js";
|
||||
import { logCliCommand } from "../cliLogging.js";
|
||||
import { analysisInputErrorFromIssues } from "../domain/inputIssueProjection.js";
|
||||
import { projectAnalysisError } from "../domain/analysisErrorProjection.js";
|
||||
import type { Evidence } from "../domain/evidence.js";
|
||||
import { withCommandCancellation } from "./commandCancellation.js";
|
||||
import {
|
||||
cliEvidenceDeliveryInputError,
|
||||
publishCliAnalysisEvidence,
|
||||
} from "./analysisEvidenceDelivery.js";
|
||||
import { finishAnalysisCommand } from "./finishAnalysisCommand.js";
|
||||
import type { CliInstance } from "./types.js";
|
||||
|
||||
/** Resolve portable input bundles, then use the same read-only matching service as MCP. */
|
||||
export const registerNativeFunctionMatchingCommands = (
|
||||
cli: CliInstance,
|
||||
logger: Logger,
|
||||
): void => {
|
||||
cli.command(CLI_COMMANDS.matchNativeFunctions, {
|
||||
description:
|
||||
"Match two observed native function inventories without executing targets",
|
||||
args: z.object({
|
||||
input: z
|
||||
.string()
|
||||
.describe(
|
||||
"JSON or JSON file: left/right inventories, optional observations and detail",
|
||||
),
|
||||
}),
|
||||
options: z.object({
|
||||
source_bundle: z
|
||||
.array(z.string())
|
||||
.default([])
|
||||
.describe(
|
||||
"Canonical source Evidence bundle for retained references; repeat for each input",
|
||||
),
|
||||
evidence_output: z
|
||||
.string()
|
||||
.min(1)
|
||||
.optional()
|
||||
.describe(
|
||||
"Save full matching Evidence and selected sources; required for default summary; never overwrites",
|
||||
),
|
||||
}),
|
||||
run: ({ args, options }) =>
|
||||
withCommandCancellation((signal) =>
|
||||
logCliCommand(logger, CLI_COMMANDS.matchNativeFunctions, async () => {
|
||||
const input = await parseCliJsonInput(
|
||||
args.input,
|
||||
"match_native_functions",
|
||||
signal,
|
||||
);
|
||||
if (!input.ok) return input.error;
|
||||
const parsed = matchNativeFunctionsInputSchema.safeParse(input.value);
|
||||
if (!parsed.success)
|
||||
return projectAnalysisError(
|
||||
analysisInputErrorFromIssues(
|
||||
"match_native_functions",
|
||||
parsed.error.issues,
|
||||
input.value,
|
||||
),
|
||||
);
|
||||
const delivery = {
|
||||
providerId: "rea-function-comparison",
|
||||
operation: "match_native_functions",
|
||||
signal,
|
||||
detail: parsed.data.detail,
|
||||
evidenceOutput: options.evidence_output,
|
||||
};
|
||||
const invalid = cliEvidenceDeliveryInputError(delivery);
|
||||
if (invalid !== undefined) return projectAnalysisError(invalid);
|
||||
const resources = new ArtifactResourceScope();
|
||||
let sources: readonly Evidence[] = [];
|
||||
const complete = await finishAnalysisCommand(
|
||||
async () => {
|
||||
const result = await matchNativeFunctionsFromBundles(
|
||||
parsed.data,
|
||||
options.source_bundle,
|
||||
resources,
|
||||
{ signal },
|
||||
);
|
||||
if (!result.ok) return result;
|
||||
sources = result.value.sources;
|
||||
return { ok: true as const, value: result.value.evidence };
|
||||
},
|
||||
resources,
|
||||
delivery.providerId,
|
||||
delivery.operation,
|
||||
signal,
|
||||
);
|
||||
return publishCliAnalysisEvidence(complete, {
|
||||
...delivery,
|
||||
sourceEvidence: sources,
|
||||
});
|
||||
}),
|
||||
),
|
||||
});
|
||||
};
|
||||
@@ -4,6 +4,7 @@ export const CLI_COMMANDS = Object.freeze({
|
||||
inspectPacketCapture: "inspect-packet-capture",
|
||||
inspectSignalDecoders: "inspect-signal-decoders",
|
||||
decodeSignalRecording: "decode-signal-recording",
|
||||
matchNativeFunctions: "match-native-functions",
|
||||
compareWasmInterfaces: "compare-wasm-interfaces",
|
||||
inspectWasmArtifact: "inspect-wasm-artifact",
|
||||
inspectSqliteDatabase: "inspect-sqlite-database",
|
||||
|
||||
@@ -1,18 +1,5 @@
|
||||
import { z } from "zod";
|
||||
import { evidenceSchema } from "../domain/evidence.js";
|
||||
import { prefixedDigestSchema } from "../domain/digests.js";
|
||||
|
||||
/** Exact reference to Evidence already retained by the current session. */
|
||||
export const retainedEvidenceReferenceSchema = z.strictObject({
|
||||
kind: z.literal("retained-evidence"),
|
||||
evidence_id: prefixedDigestSchema("ev"),
|
||||
});
|
||||
|
||||
/** Portable inline Evidence or an exact, session-scoped reference. */
|
||||
export const evidenceInputSchema = z.union([
|
||||
evidenceSchema,
|
||||
retainedEvidenceReferenceSchema,
|
||||
]);
|
||||
|
||||
/** Evidence input after parsing the advertised boundary contract. */
|
||||
export type EvidenceInput = z.output<typeof evidenceInputSchema>;
|
||||
/** Canonical Evidence selection contracts are owned by the domain. */
|
||||
export {
|
||||
evidenceInputSchema,
|
||||
type EvidenceInput,
|
||||
} from "../domain/evidenceReference.js";
|
||||
@@ -0,0 +1,30 @@
|
||||
import { z } from "zod";
|
||||
import { evidenceInputSchema } from "../evidenceInputContracts.js";
|
||||
|
||||
// Borrow inline payloads until application admission; advertise the exact public
|
||||
// Evidence/reference union without a pre-admission recursive SDK clone.
|
||||
const input = z.unknown().nonoptional();
|
||||
input._zod.processJSONSchema = evidenceInputSchema._zod.processJSONSchema;
|
||||
|
||||
/** Inventories stay portable; optional observations avoid all-function decompilation. */
|
||||
export const matchNativeFunctionsInputSchema = z.strictObject({
|
||||
left: input.describe(
|
||||
"Left list_procedures Evidence or exact retained reference",
|
||||
),
|
||||
right: input.describe(
|
||||
"Right list_procedures Evidence or exact retained reference",
|
||||
),
|
||||
left_observations: z
|
||||
.array(input)
|
||||
.default([])
|
||||
.describe(
|
||||
"Optional same-context list_names, procedure_info, read_bytes or analyze_function Evidence",
|
||||
),
|
||||
right_observations: z
|
||||
.array(input)
|
||||
.default([])
|
||||
.describe(
|
||||
"Optional same-context list_names, procedure_info, read_bytes or analyze_function Evidence",
|
||||
),
|
||||
detail: z.enum(["summary", "complete"]).default("summary"),
|
||||
});
|
||||
@@ -1,3 +1,4 @@
|
||||
import { matchNativeFunctionsInputSchema } from "./native/functionMatchingContracts.js";
|
||||
import { z } from "zod";
|
||||
|
||||
import { isAbsolute } from "node:path";
|
||||
@@ -107,9 +108,14 @@ export const SESSION_TOOL_CONTRACTS = [
|
||||
"Compare complete artifact inventories by logical occurrence path, content identity, metadata, and graph relations. Pass the inventory Evidence nested in each inspect_artifact result as left and right, or pass its inspect_analysis_view summary while the complete parent Evidence remains retained in this session. Summary facts are not treated as a complete inventory; comparison resolves the exact retained parent. Every delta cites both complete inputs, and gaps yield truncated or unknown, never equivalence. Returns every change inline.",
|
||||
artifactComparisonInputSchema,
|
||||
),
|
||||
session(
|
||||
"match_native_functions",
|
||||
"Match two retained list_procedures inventories by unique exact reported symbols, then unique identical observed function bytes. Optional same-context list_names/procedure_info/read_bytes/analyze_function Evidence supplies symbol origin, exact bodies, bytes and dossier links as supplied static facts. Optional compatible CFG/call/string fingerprints from supplied dossiers propose unresolved groups only. Duplicate candidates remain unresolved groups; only_left/right mean unpaired within supplied observations, never proof of addition/removal. Default summary retains complete Evidence for filtered, selected-column, paged function-matches views. Complete detail returns every group. Existing dossier pairs provide direct retained-reference compare_functions inputs.",
|
||||
matchNativeFunctionsInputSchema,
|
||||
),
|
||||
session(
|
||||
"compare_functions",
|
||||
"Compare two explicit sets of analyze_function Evidence across identity, exact provider text, calls, references, strings, and address-normalized CFG topology. Missing or provider-incompatible facets remain truncated or unknown; every conclusion cites both Evidence sets.",
|
||||
"Compare two explicit analyze_function Evidence records or exact retained session references across identity, exact provider text, calls, references, strings, and address-normalized CFG topology. Missing or provider-incompatible facets remain truncated or unknown; every conclusion cites both Evidence sets.",
|
||||
functionComparisonInputSchema,
|
||||
),
|
||||
session(
|
||||
|
||||
@@ -74,6 +74,10 @@ export const TOOL_EXAMPLE_OVERRIDES: Readonly<
|
||||
left: ARTIFACT_COMPARISON_EXAMPLE.left,
|
||||
right: ARTIFACT_COMPARISON_EXAMPLE.right,
|
||||
},
|
||||
match_native_functions: {
|
||||
left: { kind: "retained-evidence", evidence_id: `ev_${"1".repeat(64)}` },
|
||||
right: { kind: "retained-evidence", evidence_id: `ev_${"2".repeat(64)}` },
|
||||
},
|
||||
compare_functions: {
|
||||
left: FUNCTION_COMPARISON_EXAMPLE.left,
|
||||
right: FUNCTION_COMPARISON_EXAMPLE.right,
|
||||
|
||||
@@ -705,6 +705,7 @@ export const TOOL_EFFECTS: Readonly<Record<string, ToolEffects>> = {
|
||||
compare_artifacts: sessionEvidence,
|
||||
compare_wasm_interfaces: sessionEvidence,
|
||||
compare_functions: sessionEvidence,
|
||||
match_native_functions: sessionEvidence,
|
||||
compare_bundles: effects({ mutatesSession: true }),
|
||||
find_changed_behavior: sessionEvidence,
|
||||
build_call_path: sessionEvidence,
|
||||
|
||||
@@ -1,3 +1,6 @@
|
||||
import { nativeFunctionMatchingSchema } from "../domain/native/functionMatching.js";
|
||||
import { nativeFunctionMatchingSummaryViewResultSchema } from "../domain/analysisView/analysisView.js";
|
||||
import { nativeByteReadSchema } from "../domain/native/nativeByteRead.js";
|
||||
import { analysisDetailOutputSchemaOf } from "./analysisDetail.js";
|
||||
import { nativeInventorySummaryViewResultSchema } from "../domain/analysisView/analysisView.js";
|
||||
import { nativeInventoryRowsSchemas } from "../domain/native/nativeInventory.js";
|
||||
@@ -158,15 +161,7 @@ export const officialOutputSchemas: Readonly<Record<string, z.ZodObject>> = {
|
||||
procedure_callers: evidenceResultOf(addressList),
|
||||
procedure_info: procedureInfoOutput,
|
||||
read_function_instructions: evidenceResultOf(functionInstructionWindowSchema),
|
||||
read_bytes: evidenceResultOf(
|
||||
z.object({
|
||||
address: z.string(),
|
||||
requested_bytes: z.number().int().min(1),
|
||||
returned_bytes: z.number().int().min(0),
|
||||
bytes_hex: z.string().regex(/^(?:[a-f0-9]{2})*$/u),
|
||||
complete: z.boolean(),
|
||||
}),
|
||||
),
|
||||
read_bytes: evidenceResultOf(nativeByteReadSchema),
|
||||
address_to_file_offset: evidenceResultOf(
|
||||
z.object({
|
||||
address: z.string(),
|
||||
@@ -595,6 +590,12 @@ export const sessionOutputSchemas = {
|
||||
compare_process_captures: evidenceResultOf(processCaptureComparisonSchema),
|
||||
compare_artifacts: evidenceResultOf(artifactComparisonResultSchema),
|
||||
compare_functions: evidenceResultOf(functionComparisonResultSchema),
|
||||
match_native_functions: evidenceResultOf(
|
||||
z.union([
|
||||
nativeFunctionMatchingSchema,
|
||||
nativeFunctionMatchingSummaryViewResultSchema,
|
||||
]),
|
||||
),
|
||||
compare_bundles: evidenceResultOf(bundleComparisonResultSchema),
|
||||
find_changed_behavior: evidenceResultOf(changedBehaviorResultSchema),
|
||||
build_call_path: evidenceResultOf(callPathResultSchema),
|
||||
|
||||
@@ -1,11 +1,9 @@
|
||||
import { procedureInfoSchema } from "../domain/native/procedureInfo.js";
|
||||
import { z } from "zod";
|
||||
|
||||
import {
|
||||
procedureClassificationSchema,
|
||||
procedureIdentitySchema,
|
||||
localVariableSchema,
|
||||
functionDossierSchema,
|
||||
functionBodySchema,
|
||||
} from "../domain/hopperValues.js";
|
||||
import { analysisProfileSchema } from "../domain/analysisProfile.js";
|
||||
import {
|
||||
@@ -598,18 +596,7 @@ export const segmentOutput = evidenceResultOf(
|
||||
z.array(memoryRegionOutput.extend({ sections: z.array(memoryRegionOutput) })),
|
||||
);
|
||||
|
||||
export const procedureInfoOutput = evidenceResultOf(
|
||||
z.object({
|
||||
name: z.string(),
|
||||
entrypoint: z.string(),
|
||||
basicblock_count: z.number().int().min(0),
|
||||
length: z.number().min(0),
|
||||
signature: nullableText,
|
||||
locals: z.array(localVariableSchema),
|
||||
classification: procedureClassificationSchema.nullable(),
|
||||
body: functionBodySchema,
|
||||
}),
|
||||
);
|
||||
export const procedureInfoOutput = evidenceResultOf(procedureInfoSchema);
|
||||
|
||||
export const symbolDiscoveryOutput = (property: "classes" | "protocols") =>
|
||||
evidenceResultOf(
|
||||
|
||||
@@ -1,3 +1,9 @@
|
||||
import {
|
||||
projectNativeFunctionMatchingView,
|
||||
nativeFunctionMatchingViewRequestSchema,
|
||||
nativeFunctionMatchingViewItemsSchema,
|
||||
nativeFunctionMatchingViewContextSchema,
|
||||
} from "./nativeFunctionMatchingView.js";
|
||||
import {
|
||||
protobufViewRequestSchema,
|
||||
protobufViewItemsSchema,
|
||||
@@ -243,6 +249,7 @@ const nativeViewSchema = z.strictObject({
|
||||
|
||||
/** Caller-selected projection of already completed analysis Evidence. */
|
||||
export const analysisViewRequestSchema = z.discriminatedUnion("kind", [
|
||||
nativeFunctionMatchingViewRequestSchema,
|
||||
summaryViewSchema,
|
||||
desktopViewRequestSchema,
|
||||
facetViewSchema,
|
||||
@@ -377,6 +384,7 @@ const viewResultBase = {
|
||||
"inspect_dart_aot",
|
||||
"inspect_il2cpp_metadata",
|
||||
...NATIVE_INVENTORY_OPERATIONS,
|
||||
"match_native_functions",
|
||||
]),
|
||||
parent_digest: digestSchema,
|
||||
view_digest: digestSchema,
|
||||
@@ -524,8 +532,25 @@ export const nativeInventorySummaryViewResultSchema = z.strictObject({
|
||||
summary: nativeInventorySummarySchema,
|
||||
});
|
||||
|
||||
/** Compact native matching discovery retains the complete matching report. */
|
||||
export const nativeFunctionMatchingSummaryViewResultSchema = z.strictObject({
|
||||
...summaryViewResultBase,
|
||||
artifact: z.null(),
|
||||
parent_operation: z.literal("match_native_functions"),
|
||||
summary: nativeFunctionMatchingViewContextSchema,
|
||||
});
|
||||
|
||||
/** Projected facts for one selected view of retained analysis Evidence. */
|
||||
export const analysisViewResultSchema = z.discriminatedUnion("kind", [
|
||||
z.strictObject({
|
||||
...viewResultBase,
|
||||
...nativeFunctionMatchingViewContextSchema.shape,
|
||||
kind: z.literal("function-matches"),
|
||||
artifact: z.null(),
|
||||
view: nativeFunctionMatchingViewRequestSchema,
|
||||
items: nativeFunctionMatchingViewItemsSchema,
|
||||
parent_operation: z.literal("match_native_functions"),
|
||||
}),
|
||||
z.strictObject({
|
||||
...viewResultBase,
|
||||
...apiSurfaceFields,
|
||||
@@ -574,6 +599,7 @@ export const analysisViewResultSchema = z.discriminatedUnion("kind", [
|
||||
dartAotSummaryViewResultSchema,
|
||||
il2cppMetadataSummaryViewResultSchema,
|
||||
nativeInventorySummaryViewResultSchema,
|
||||
nativeFunctionMatchingSummaryViewResultSchema,
|
||||
]),
|
||||
z.strictObject({
|
||||
...viewResultBase,
|
||||
@@ -844,6 +870,7 @@ const retainedViewProjectors = new Map<string, typeof projectWasmArtifactView>([
|
||||
["inspect_android_package", projectAndroidPackageView],
|
||||
["inspect_dart_aot", projectDartAotView],
|
||||
["inspect_il2cpp_metadata", projectIl2cppMetadataView],
|
||||
["match_native_functions", projectNativeFunctionMatchingView],
|
||||
["inspect_artifact", projectArtifactInventoryView],
|
||||
["inventory_artifact", projectArtifactInventoryView],
|
||||
]);
|
||||
|
||||
@@ -0,0 +1,311 @@
|
||||
import { z } from "zod";
|
||||
import {
|
||||
nativeFunctionMatchingSchema,
|
||||
nativeFunctionMatchingContextSchema,
|
||||
nativeFunctionMatchSchema,
|
||||
nativeFunctionMatchMemberSchema,
|
||||
nativeMatchStateSchema,
|
||||
type NativeFunctionMatching,
|
||||
} from "../native/functionMatching.js";
|
||||
import { isImmutableJsonSnapshot } from "../immutableJson.js";
|
||||
import { parseAnalysisViewResult } from "./analysisViewParent.js";
|
||||
import { analysisViewInputError } from "./analysisViewErrors.js";
|
||||
import {
|
||||
boundedViewMessages,
|
||||
completeWithinViewCoverage,
|
||||
pageViewCoverage,
|
||||
} from "./analysisViewCoverage.js";
|
||||
import { err, ok, type Result } from "../result.js";
|
||||
import type { AnalysisError } from "../analysisErrorBase.js";
|
||||
import type {
|
||||
AnalysisViewParent,
|
||||
AnalysisViewRequest,
|
||||
UnsignedAnalysisView,
|
||||
} from "./analysisView.js";
|
||||
import type { JsonValue } from "../jsonValue.js";
|
||||
|
||||
const matchRow = nativeFunctionMatchSchema.extend({
|
||||
left_count: z.number().int().nonnegative(),
|
||||
right_count: z.number().int().nonnegative(),
|
||||
});
|
||||
const functionRow = nativeFunctionMatchMemberSchema.extend({
|
||||
match_id: nativeFunctionMatchSchema.shape.match_id,
|
||||
state: nativeMatchStateSchema,
|
||||
side: z.enum(["left", "right"]),
|
||||
});
|
||||
const page = {
|
||||
kind: z.literal("function-matches"),
|
||||
states: z.array(nativeMatchStateSchema).optional(),
|
||||
match_ids: z.array(z.string()).optional(),
|
||||
function_ids: z.array(z.string()).optional(),
|
||||
offset: z.number().int().nonnegative().default(0),
|
||||
limit: z.number().int().positive().default(64),
|
||||
};
|
||||
export const nativeFunctionMatchingViewRequestSchema = z.discriminatedUnion(
|
||||
"collection",
|
||||
[
|
||||
z.strictObject({
|
||||
...page,
|
||||
collection: z.literal("matches"),
|
||||
fields: z.array(matchRow.keyof()).optional(),
|
||||
}),
|
||||
z.strictObject({
|
||||
...page,
|
||||
collection: z.literal("functions"),
|
||||
sides: z.array(z.enum(["left", "right"])).optional(),
|
||||
names: z.array(z.string()).optional(),
|
||||
fields: z.array(functionRow.keyof()).optional(),
|
||||
}),
|
||||
],
|
||||
);
|
||||
export const nativeFunctionMatchingViewItemsSchema = z.union([
|
||||
z.array(
|
||||
matchRow.partial().required({
|
||||
match_id: true,
|
||||
state: true,
|
||||
left_count: true,
|
||||
right_count: true,
|
||||
}),
|
||||
),
|
||||
z.array(
|
||||
functionRow.partial().required({
|
||||
function_id: true,
|
||||
match_id: true,
|
||||
state: true,
|
||||
side: true,
|
||||
inventory_evidence_id: true,
|
||||
source_index: true,
|
||||
address: true,
|
||||
evidence_links: true,
|
||||
}),
|
||||
),
|
||||
]);
|
||||
export const nativeFunctionMatchingViewContextSchema =
|
||||
nativeFunctionMatchingContextSchema;
|
||||
interface FunctionRow {
|
||||
readonly member: NativeFunctionMatching["matches"][number]["left"][number];
|
||||
readonly match_id: string;
|
||||
readonly state: NativeFunctionMatching["matches"][number]["state"];
|
||||
readonly side: "left" | "right";
|
||||
}
|
||||
interface Checked {
|
||||
readonly report: NativeFunctionMatching;
|
||||
functions?: readonly FunctionRow[];
|
||||
recent?: { readonly key: string; readonly rows: readonly number[] };
|
||||
}
|
||||
const checked = new WeakMap<object, Checked>();
|
||||
|
||||
type Request = z.output<typeof nativeFunctionMatchingViewRequestSchema>;
|
||||
const checkedReport = (
|
||||
parent: AnalysisViewParent,
|
||||
): Result<Checked, AnalysisError> => {
|
||||
const source = parent.normalizedResult;
|
||||
const reusable =
|
||||
typeof source === "object" &&
|
||||
source !== null &&
|
||||
parent.sealedResult === true &&
|
||||
isImmutableJsonSnapshot(source);
|
||||
const cached = reusable ? checked.get(source) : undefined;
|
||||
if (cached !== undefined) return ok(cached);
|
||||
const parsed = parseAnalysisViewResult(nativeFunctionMatchingSchema, parent);
|
||||
if (!parsed.ok) return parsed;
|
||||
const entry: Checked = {
|
||||
report: reusable ? (source as NativeFunctionMatching) : parsed.value,
|
||||
};
|
||||
if (reusable) checked.set(source, entry);
|
||||
return ok(entry);
|
||||
};
|
||||
const functionRows = (entry: Checked): readonly FunctionRow[] => {
|
||||
if (entry.functions !== undefined) return entry.functions;
|
||||
const functions: FunctionRow[] = [];
|
||||
for (const match of entry.report.matches)
|
||||
for (const side of ["left", "right"] as const)
|
||||
for (const member of match[side])
|
||||
functions.push({
|
||||
member,
|
||||
match_id: match.match_id,
|
||||
state: match.state,
|
||||
side,
|
||||
});
|
||||
entry.functions = functions;
|
||||
return functions;
|
||||
};
|
||||
const sharedContext = (
|
||||
parent: AnalysisViewParent,
|
||||
report: NativeFunctionMatching,
|
||||
) => ({
|
||||
parent_evidence_id: parent.evidenceId,
|
||||
parent_operation: "match_native_functions" as const,
|
||||
parent_digest: parent.evidenceId.slice(3),
|
||||
artifact: null,
|
||||
limitations: boundedViewMessages(
|
||||
[...parent.limitations, ...report.limitations],
|
||||
parent.evidenceId,
|
||||
"limitations",
|
||||
),
|
||||
unknowns: [
|
||||
"Unpaired observations and page exhaustion do not establish binary-wide function absence or behavioral equivalence.",
|
||||
],
|
||||
});
|
||||
const selected = (values: readonly string[] | undefined) =>
|
||||
values === undefined ? undefined : new Set(values);
|
||||
const filterRows = (entry: Checked, view: Request): readonly number[] => {
|
||||
const key = JSON.stringify([
|
||||
view.collection,
|
||||
view.states,
|
||||
view.match_ids,
|
||||
view.function_ids,
|
||||
view.collection === "functions" ? view.sides : null,
|
||||
view.collection === "functions" ? view.names : null,
|
||||
]);
|
||||
if (entry.recent?.key === key) return entry.recent.rows;
|
||||
const states = selected(view.states),
|
||||
ids = selected(view.match_ids),
|
||||
functions = selected(view.function_ids),
|
||||
sides = selected(view.collection === "functions" ? view.sides : undefined),
|
||||
names = selected(view.collection === "functions" ? view.names : undefined);
|
||||
const matchesIdentity = (row: { state: string; match_id: string }): boolean =>
|
||||
(states === undefined || states.has(row.state)) &&
|
||||
(ids === undefined || ids.has(row.match_id));
|
||||
const rows: number[] = [];
|
||||
if (view.collection === "matches")
|
||||
entry.report.matches.forEach((row, index) => {
|
||||
if (
|
||||
matchesIdentity(row) &&
|
||||
(functions === undefined ||
|
||||
[...row.left, ...row.right].some((member) =>
|
||||
functions.has(member.function_id),
|
||||
))
|
||||
)
|
||||
rows.push(index);
|
||||
});
|
||||
else
|
||||
functionRows(entry).forEach((row, index) => {
|
||||
if (
|
||||
matchesIdentity(row) &&
|
||||
(functions === undefined || functions.has(row.member.function_id)) &&
|
||||
(sides === undefined || sides.has(row.side)) &&
|
||||
(names === undefined || names.has(row.member.name))
|
||||
)
|
||||
rows.push(index);
|
||||
});
|
||||
// One recent selector and at most one row index per source occurrence.
|
||||
const count = entry.report.left.functions + entry.report.right.functions;
|
||||
if (key.length <= Math.max(256, count * 8)) entry.recent = { key, rows };
|
||||
return rows;
|
||||
};
|
||||
const selectedItems = (
|
||||
entry: Checked,
|
||||
view: Request,
|
||||
rows: readonly number[],
|
||||
) => {
|
||||
const indices = rows.slice(view.offset, view.offset + view.limit);
|
||||
const defaults =
|
||||
view.collection === "matches"
|
||||
? [
|
||||
"match_id",
|
||||
"state",
|
||||
"basis",
|
||||
"reason",
|
||||
"left_count",
|
||||
"right_count",
|
||||
"compare_functions_input",
|
||||
]
|
||||
: [
|
||||
"function_id",
|
||||
"match_id",
|
||||
"state",
|
||||
"side",
|
||||
"inventory_evidence_id",
|
||||
"source_index",
|
||||
"address",
|
||||
"name",
|
||||
"body_sha256",
|
||||
"structural_candidate_sha256",
|
||||
"body_bytes",
|
||||
"dossier_evidence_id",
|
||||
"evidence_links",
|
||||
];
|
||||
const fields = new Set<string>(view.fields ?? defaults);
|
||||
const columns = (row: Readonly<Record<string, JsonValue>>) =>
|
||||
Object.fromEntries([...fields].map((field) => [field, row[field]!]));
|
||||
if (view.collection === "matches")
|
||||
return indices.map((index) => {
|
||||
const row = entry.report.matches[index]!,
|
||||
left_count = row.left.length,
|
||||
right_count = row.right.length;
|
||||
return {
|
||||
...columns({ ...row, left_count, right_count }),
|
||||
match_id: row.match_id,
|
||||
state: row.state,
|
||||
left_count,
|
||||
right_count,
|
||||
};
|
||||
});
|
||||
const functions = functionRows(entry);
|
||||
return indices.map((index) => {
|
||||
const row = functions[index]!,
|
||||
member = row.member;
|
||||
return {
|
||||
...columns({
|
||||
...member,
|
||||
match_id: row.match_id,
|
||||
state: row.state,
|
||||
side: row.side,
|
||||
}),
|
||||
function_id: member.function_id,
|
||||
match_id: row.match_id,
|
||||
state: row.state,
|
||||
side: row.side,
|
||||
inventory_evidence_id: member.inventory_evidence_id,
|
||||
source_index: member.source_index,
|
||||
address: member.address,
|
||||
evidence_links: member.evidence_links,
|
||||
};
|
||||
});
|
||||
};
|
||||
|
||||
/** Query immutable matching facts without copying full alias groups into selected pages. */
|
||||
export const projectNativeFunctionMatchingView = (
|
||||
parent: AnalysisViewParent,
|
||||
view: AnalysisViewRequest,
|
||||
): Result<UnsignedAnalysisView, AnalysisError> => {
|
||||
if (view.kind !== "summary" && view.kind !== "function-matches")
|
||||
return err(
|
||||
analysisViewInputError([
|
||||
{
|
||||
path: ["view"],
|
||||
reason: "invalid_value",
|
||||
message:
|
||||
"Native matching supports summary and function-matches pages",
|
||||
},
|
||||
]),
|
||||
);
|
||||
const parsed = checkedReport(parent);
|
||||
if (!parsed.ok) return parsed;
|
||||
const entry = parsed.value;
|
||||
const {
|
||||
matches: _matches,
|
||||
limitations: _limitations,
|
||||
...context
|
||||
} = entry.report;
|
||||
const shared = sharedContext(parent, entry.report);
|
||||
if (view.kind === "summary")
|
||||
return ok({
|
||||
kind: "summary",
|
||||
view,
|
||||
...shared,
|
||||
summary: context,
|
||||
coverage: completeWithinViewCoverage(1, 1),
|
||||
});
|
||||
const rows = filterRows(entry, view),
|
||||
items = selectedItems(entry, view, rows);
|
||||
return ok({
|
||||
kind: "function-matches",
|
||||
view,
|
||||
...shared,
|
||||
...context,
|
||||
items,
|
||||
coverage: pageViewCoverage(view.offset, items.length, rows.length),
|
||||
});
|
||||
};
|
||||
@@ -0,0 +1,17 @@
|
||||
import { z } from "zod";
|
||||
import { prefixedDigestSchema } from "./digests.js";
|
||||
|
||||
/** Exact reference to Evidence already retained by the current session. */
|
||||
export const retainedEvidenceReferenceSchema = z.strictObject({
|
||||
kind: z.literal("retained-evidence"),
|
||||
evidence_id: prefixedDigestSchema("ev"),
|
||||
});
|
||||
|
||||
import { evidenceSchema } from "./evidence.js";
|
||||
|
||||
/** Portable observation or exact retained identity; semantic authentication follows admission. */
|
||||
export const evidenceInputSchema = z.union([
|
||||
evidenceSchema,
|
||||
retainedEvidenceReferenceSchema,
|
||||
]);
|
||||
export type EvidenceInput = z.output<typeof evidenceInputSchema>;
|
||||
@@ -1,6 +1,6 @@
|
||||
import { z } from "zod";
|
||||
import { overallStatus } from "./functionComparisonResults.js";
|
||||
import { evidenceSchema } from "./evidence.js";
|
||||
import { evidenceInputSchema } from "./evidenceReference.js";
|
||||
import { digestSchema } from "./../domain/digests.js";
|
||||
import { prefixedDigestSchema } from "./../domain/digests.js";
|
||||
|
||||
@@ -19,8 +19,8 @@ const evidenceIdSchema = prefixedDigestSchema("ev");
|
||||
|
||||
/** Inputs for explicit function-to-function comparison. */
|
||||
export const functionComparisonInputSchema = z.strictObject({
|
||||
left: evidenceSchema,
|
||||
right: evidenceSchema,
|
||||
left: evidenceInputSchema,
|
||||
right: evidenceInputSchema,
|
||||
});
|
||||
|
||||
const textDeltaSchema = z.object({
|
||||
|
||||
@@ -0,0 +1,193 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import type { z } from "zod";
|
||||
import type { functionBodySchema } from "../hopperValues.js";
|
||||
import type { nativeByteReadSchema } from "./nativeByteRead.js";
|
||||
import { addressCoordinate } from "./addressCoordinate.js";
|
||||
import { parseNativeAddress } from "./nativeAddress.js";
|
||||
|
||||
export type NativeMatchingCheckpoint = (bytes?: number) => Promise<void>;
|
||||
interface Interval {
|
||||
start: bigint;
|
||||
end: bigint;
|
||||
}
|
||||
interface Piece extends Interval {
|
||||
readonly hex: string;
|
||||
readonly links: readonly string[];
|
||||
}
|
||||
interface Fingerprint {
|
||||
readonly sha256: string;
|
||||
readonly key: string;
|
||||
readonly links: readonly string[];
|
||||
}
|
||||
const slice = (piece: Piece, start: bigint, end: bigint): Piece => ({
|
||||
...piece,
|
||||
start,
|
||||
end,
|
||||
hex:
|
||||
start === piece.start && end === piece.end
|
||||
? piece.hex
|
||||
: piece.hex.slice(
|
||||
Number(start - piece.start) * 2,
|
||||
Number(end - piece.start) * 2,
|
||||
),
|
||||
});
|
||||
const firstOverlapping = (pieces: readonly Piece[], start: bigint): number => {
|
||||
let lower = 0,
|
||||
upper = pieces.length;
|
||||
while (lower < upper) {
|
||||
const middle = (lower + upper) >>> 1;
|
||||
if (pieces[middle]!.end <= start) lower = middle + 1;
|
||||
else upper = middle;
|
||||
}
|
||||
return lower;
|
||||
};
|
||||
const mergeIntervals = (intervals: Interval[]): Interval[] => {
|
||||
intervals.sort((a, b) =>
|
||||
a.start < b.start ? -1 : a.start > b.start ? 1 : 0,
|
||||
);
|
||||
const merged: Interval[] = [];
|
||||
for (const interval of intervals) {
|
||||
const previous = merged.at(-1);
|
||||
if (previous !== undefined && interval.start <= previous.end) {
|
||||
if (interval.end > previous.end) previous.end = interval.end;
|
||||
} else merged.push({ ...interval });
|
||||
}
|
||||
return merged;
|
||||
};
|
||||
const excludeConflicts = (
|
||||
pieces: readonly Piece[],
|
||||
conflicts: readonly Interval[],
|
||||
): Piece[] => {
|
||||
const safe: Piece[] = [];
|
||||
let first = 0;
|
||||
for (const piece of pieces) {
|
||||
while (first < conflicts.length && conflicts[first]!.end <= piece.start)
|
||||
first++;
|
||||
let cursor = piece.start;
|
||||
for (
|
||||
let index = first;
|
||||
index < conflicts.length && conflicts[index]!.start < piece.end;
|
||||
index++
|
||||
) {
|
||||
const conflict = conflicts[index]!;
|
||||
if (conflict.start > cursor)
|
||||
safe.push(slice(piece, cursor, conflict.start));
|
||||
if (conflict.end > cursor) cursor = conflict.end;
|
||||
if (cursor >= piece.end) break;
|
||||
}
|
||||
if (cursor < piece.end) safe.push(slice(piece, cursor, piece.end));
|
||||
}
|
||||
return safe;
|
||||
};
|
||||
|
||||
/** Own exact observed byte intervals, their conflicts and whole-body hashing. */
|
||||
export class FunctionByteCoverage {
|
||||
readonly #pieces = new Map<string | null, Piece[]>();
|
||||
#prepared = false;
|
||||
constructor(readonly checkpoint: NativeMatchingCheckpoint) {}
|
||||
async add(
|
||||
read: z.output<typeof nativeByteReadSchema>,
|
||||
evidenceId: string,
|
||||
): Promise<void> {
|
||||
if (this.#prepared)
|
||||
throw new TypeError(
|
||||
"Cannot append byte observations after fingerprinting",
|
||||
);
|
||||
if (read.returned_bytes === 0) return;
|
||||
const coordinate = addressCoordinate(parseNativeAddress(read.address))!;
|
||||
let group = this.#pieces.get(coordinate.space);
|
||||
if (group === undefined) this.#pieces.set(coordinate.space, (group = []));
|
||||
await this.checkpoint(192 + read.bytes_hex.length * 2);
|
||||
group.push({
|
||||
start: coordinate.offset,
|
||||
end: coordinate.offset + BigInt(read.returned_bytes),
|
||||
hex: read.bytes_hex,
|
||||
links: [evidenceId],
|
||||
});
|
||||
}
|
||||
async #conflict(left: Piece, right: Piece): Promise<Interval | undefined> {
|
||||
const start = left.start > right.start ? left.start : right.start;
|
||||
const end = left.end < right.end ? left.end : right.end;
|
||||
for (let cursor = start; cursor < end;) {
|
||||
const next = cursor + 65536n < end ? cursor + 65536n : end;
|
||||
await this.checkpoint();
|
||||
if (slice(left, cursor, next).hex !== slice(right, cursor, next).hex)
|
||||
return { start, end };
|
||||
cursor = next;
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
async #prepareSpace(source: Piece[]): Promise<Piece[]> {
|
||||
source.sort((a, b) =>
|
||||
a.start < b.start
|
||||
? -1
|
||||
: a.start > b.start
|
||||
? 1
|
||||
: a.end > b.end
|
||||
? -1
|
||||
: a.end < b.end
|
||||
? 1
|
||||
: 0,
|
||||
);
|
||||
const admitted: Piece[] = [],
|
||||
conflicts: Interval[] = [];
|
||||
for (const incoming of source) {
|
||||
await this.checkpoint();
|
||||
for (
|
||||
let index = firstOverlapping(admitted, incoming.start);
|
||||
index < admitted.length && admitted[index]!.start < incoming.end;
|
||||
index++
|
||||
) {
|
||||
const conflict = await this.#conflict(admitted[index]!, incoming);
|
||||
if (conflict !== undefined) conflicts.push(conflict);
|
||||
}
|
||||
const end = admitted.at(-1)?.end;
|
||||
if (end === undefined || incoming.start >= end) admitted.push(incoming);
|
||||
else if (incoming.end > end)
|
||||
admitted.push(slice(incoming, end, incoming.end));
|
||||
}
|
||||
return excludeConflicts(admitted, mergeIntervals(conflicts));
|
||||
}
|
||||
async fingerprint(
|
||||
body: z.output<typeof functionBodySchema> | undefined,
|
||||
): Promise<Fingerprint | null> {
|
||||
if (!body?.available || !body.contains_entry || body.total_bytes === 0)
|
||||
return null;
|
||||
if (!this.#prepared) {
|
||||
for (const [space, pieces] of this.#pieces)
|
||||
this.#pieces.set(space, await this.#prepareSpace(pieces));
|
||||
this.#prepared = true;
|
||||
}
|
||||
const hash = createHash("sha256"),
|
||||
lengths: string[] = [],
|
||||
links = new Set<string>();
|
||||
for (const range of body.ranges) {
|
||||
const start = addressCoordinate(parseNativeAddress(range.start))!,
|
||||
end = addressCoordinate(parseNativeAddress(range.end))!;
|
||||
const pieces = this.#pieces.get(start.space) ?? [];
|
||||
lengths.push(String(end.offset - start.offset + 1n));
|
||||
let index = firstOverlapping(pieces, start.offset),
|
||||
cursor = start.offset;
|
||||
while (cursor <= end.offset) {
|
||||
const piece = pieces[index++];
|
||||
if (piece === undefined || piece.start > cursor || piece.end <= cursor)
|
||||
return null;
|
||||
const stop = piece.end < end.offset + 1n ? piece.end : end.offset + 1n;
|
||||
for (let chunk = cursor; chunk < stop;) {
|
||||
const next = chunk + 65536n < stop ? chunk + 65536n : stop;
|
||||
await this.checkpoint();
|
||||
hash.update(Buffer.from(slice(piece, chunk, next).hex, "hex"));
|
||||
chunk = next;
|
||||
}
|
||||
for (const link of piece.links) links.add(link);
|
||||
cursor = stop;
|
||||
}
|
||||
}
|
||||
const sha256 = hash.digest("hex");
|
||||
return {
|
||||
sha256,
|
||||
key: JSON.stringify([lengths, sha256]),
|
||||
links: [...links],
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,290 @@
|
||||
import { functionStructuralCandidate } from "./functionStructuralCandidate.js";
|
||||
import type { z } from "zod";
|
||||
import type { Evidence } from "../evidence.js";
|
||||
import { EvidenceIntegrityError } from "../evidenceErrors.js";
|
||||
import { digestCanonicalValue } from "../canonicalDigest.js";
|
||||
import {
|
||||
functionBodySchema,
|
||||
functionBodyEntryAgrees,
|
||||
functionDossierSchema,
|
||||
} from "../hopperValues.js";
|
||||
import { isAutoName } from "../functionComparisonNormalization.js";
|
||||
import {
|
||||
addressedValueSchema,
|
||||
nativeInventoryRowsSchemas,
|
||||
} from "./nativeInventory.js";
|
||||
import { parseNativeAddress } from "./nativeAddress.js";
|
||||
import { nativeByteReadSchema } from "./nativeByteRead.js";
|
||||
import { procedureInfoSchema } from "./procedureInfo.js";
|
||||
import type { NativeFunctionMatchMember } from "./functionMatching.js";
|
||||
import {
|
||||
FunctionByteCoverage,
|
||||
type NativeMatchingCheckpoint,
|
||||
} from "./functionByteCoverage.js";
|
||||
|
||||
type Body = z.infer<typeof functionBodySchema>;
|
||||
type AddressedValue = z.infer<typeof addressedValueSchema>;
|
||||
export interface MatchingFunction {
|
||||
readonly member: NativeFunctionMatchMember;
|
||||
readonly symbol: string | null;
|
||||
readonly fingerprint: string | null;
|
||||
}
|
||||
interface Detail {
|
||||
body?: Body;
|
||||
dossier?: string | null;
|
||||
candidate?: string | null;
|
||||
conflict: boolean;
|
||||
readonly links: string[];
|
||||
}
|
||||
interface Symbols {
|
||||
readonly values: AddressedValue[];
|
||||
readonly links: Set<string>;
|
||||
}
|
||||
interface Facts {
|
||||
readonly details: Map<string, Detail>;
|
||||
readonly symbols: Map<string, Symbols>;
|
||||
readonly bytes: FunctionByteCoverage;
|
||||
}
|
||||
|
||||
/** Supplementary observations must belong to this exact provider/target/profile/document. */
|
||||
const requireContext = (inventory: Evidence, observation: Evidence): void => {
|
||||
if (
|
||||
observation.predicate_type !== "rea.analysis" ||
|
||||
digestCanonicalValue(observation.subject) !==
|
||||
digestCanonicalValue(inventory.subject) ||
|
||||
digestCanonicalValue(observation.provider) !==
|
||||
digestCanonicalValue(inventory.provider) ||
|
||||
digestCanonicalValue(observation.analysis_profile) !==
|
||||
digestCanonicalValue(inventory.analysis_profile) ||
|
||||
(observation.parameters.document ?? null) !==
|
||||
(inventory.parameters.document ?? null)
|
||||
)
|
||||
throw new EvidenceIntegrityError(
|
||||
"Function observation does not belong to its selected inventory context",
|
||||
);
|
||||
};
|
||||
const addBody = (observation: Evidence, details: Facts["details"]): void => {
|
||||
const dossier =
|
||||
observation.operation === "analyze_function"
|
||||
? functionDossierSchema.parse(observation.normalized_result)
|
||||
: undefined;
|
||||
const procedure =
|
||||
dossier?.procedure ??
|
||||
(() => {
|
||||
const info = procedureInfoSchema.parse(observation.normalized_result);
|
||||
return { address: info.entrypoint, body: info.body };
|
||||
})();
|
||||
const address = parseNativeAddress(procedure.address);
|
||||
if (!functionBodyEntryAgrees(procedure.body, address))
|
||||
throw new EvidenceIntegrityError(
|
||||
"Procedure body entry membership contradicts its address",
|
||||
);
|
||||
let detail = details.get(address);
|
||||
if (detail === undefined)
|
||||
details.set(address, (detail = { conflict: false, links: [] }));
|
||||
if (
|
||||
detail.body !== undefined &&
|
||||
digestCanonicalValue(detail.body) !== digestCanonicalValue(procedure.body)
|
||||
)
|
||||
detail.conflict = true;
|
||||
detail.body = procedure.body;
|
||||
if (observation.operation === "analyze_function") {
|
||||
const candidate = functionStructuralCandidate(observation, dossier!);
|
||||
if (detail.dossier === undefined) {
|
||||
detail.dossier = observation.evidence_id;
|
||||
detail.candidate = candidate;
|
||||
} else {
|
||||
// Request identity and dossier selection are independent of body agreement.
|
||||
if (detail.dossier !== observation.evidence_id) detail.dossier = null;
|
||||
if (detail.candidate !== candidate) detail.candidate = null;
|
||||
}
|
||||
}
|
||||
detail.links.push(observation.evidence_id);
|
||||
};
|
||||
const addNames = async (
|
||||
observation: Evidence,
|
||||
symbols: Facts["symbols"],
|
||||
checkpoint: NativeMatchingCheckpoint,
|
||||
): Promise<void> => {
|
||||
for (const symbol of nativeInventoryRowsSchemas.names.parse(
|
||||
observation.normalized_result,
|
||||
)) {
|
||||
await checkpoint(128 + symbol.value.length * 2);
|
||||
const address = parseNativeAddress(symbol.address);
|
||||
let group = symbols.get(address);
|
||||
if (group === undefined)
|
||||
symbols.set(address, (group = { values: [], links: new Set() }));
|
||||
group.values.push(symbol);
|
||||
group.links.add(observation.evidence_id);
|
||||
}
|
||||
};
|
||||
const addBytes = async (
|
||||
observation: Evidence,
|
||||
bytes: FunctionByteCoverage,
|
||||
): Promise<void> => {
|
||||
const read = nativeByteReadSchema.parse(observation.normalized_result);
|
||||
if (
|
||||
(typeof observation.parameters.address === "string" &&
|
||||
parseNativeAddress(observation.parameters.address) !==
|
||||
parseNativeAddress(read.address)) ||
|
||||
(typeof observation.parameters.length === "number" &&
|
||||
observation.parameters.length !== read.requested_bytes)
|
||||
)
|
||||
throw new EvidenceIntegrityError(
|
||||
"Read-byte coordinates contradict the selected request",
|
||||
);
|
||||
await bytes.add(read, observation.evidence_id);
|
||||
};
|
||||
const reportedSymbol = (
|
||||
row: AddressedValue,
|
||||
known: Symbols | undefined,
|
||||
): {
|
||||
readonly value: string | null;
|
||||
readonly origin: "known" | "unknown" | "conflicting";
|
||||
} => {
|
||||
const names = [
|
||||
...(row.symbol === undefined ? [] : [row]),
|
||||
...(known?.values.filter((symbol) => symbol.value === row.value) ?? []),
|
||||
];
|
||||
const generated = names.some(
|
||||
(symbol) =>
|
||||
symbol.symbol?.dynamic === true || symbol.symbol?.source === "default",
|
||||
);
|
||||
const named = names.some(
|
||||
(symbol) =>
|
||||
symbol.symbol?.dynamic === false && symbol.symbol.source !== "default",
|
||||
);
|
||||
return {
|
||||
value:
|
||||
row.value.length === 0 || generated || (!named && isAutoName(row.value))
|
||||
? null
|
||||
: row.value,
|
||||
origin:
|
||||
generated && named
|
||||
? "conflicting"
|
||||
: names.some((symbol) => symbol.symbol !== undefined)
|
||||
? "known"
|
||||
: "unknown",
|
||||
};
|
||||
};
|
||||
const functionFact = async (
|
||||
{
|
||||
inventory,
|
||||
row,
|
||||
sourceIndex,
|
||||
}: { inventory: Evidence; row: AddressedValue; sourceIndex: number },
|
||||
facts: Facts,
|
||||
checkpoint: NativeMatchingCheckpoint,
|
||||
): Promise<MatchingFunction> => {
|
||||
const address = parseNativeAddress(row.address),
|
||||
detail = facts.details.get(address),
|
||||
symbols = facts.symbols.get(address);
|
||||
// Bound expansion before creating per-occurrence sets, arrays and subsequent
|
||||
// schema/Evidence serialization copies. Same-address aliases repeat these links.
|
||||
for (const link of detail?.links ?? [])
|
||||
await checkpoint(256 + 8 * link.length);
|
||||
for (const link of symbols?.links ?? [])
|
||||
await checkpoint(256 + 8 * link.length);
|
||||
await checkpoint(256 + 8 * inventory.evidence_id.length);
|
||||
const links = new Set([inventory.evidence_id]);
|
||||
for (const link of detail?.links ?? []) links.add(link);
|
||||
for (const link of symbols?.links ?? []) links.add(link);
|
||||
const body = detail?.conflict ? undefined : detail?.body;
|
||||
const fingerprint = await facts.bytes.fingerprint(body);
|
||||
for (const link of fingerprint?.links ?? []) {
|
||||
await checkpoint(256 + 8 * link.length);
|
||||
links.add(link);
|
||||
}
|
||||
const limitations: string[] = [];
|
||||
const symbol = reportedSymbol(row, symbols);
|
||||
if (fingerprint === null)
|
||||
limitations.push(
|
||||
detail?.conflict
|
||||
? "Conflicting procedure observations; body and dossier remain unknown."
|
||||
: "No complete unambiguous bytes for the observed function body.",
|
||||
);
|
||||
if (symbol.origin === "unknown")
|
||||
limitations.push(
|
||||
"Symbol origin unavailable; exact-name matching uses the reported name and the existing automatic-name exclusion rule.",
|
||||
);
|
||||
if (symbol.origin === "conflicting")
|
||||
limitations.push(
|
||||
"Conflicting generated and named symbol-origin observations; exact-name matching is unavailable. Independent complete body bytes remain eligible.",
|
||||
);
|
||||
if (detail?.dossier === null)
|
||||
limitations.push(
|
||||
"Multiple supplied dossiers describe this address; no dossier is selected automatically. Agreed body bytes and structural facts remain usable.",
|
||||
);
|
||||
return {
|
||||
member: {
|
||||
function_id: `${inventory.evidence_id}/${sourceIndex}`,
|
||||
inventory_evidence_id: inventory.evidence_id,
|
||||
source_index: sourceIndex,
|
||||
address,
|
||||
name: row.value,
|
||||
body_sha256: fingerprint?.sha256 ?? null,
|
||||
structural_candidate_sha256: detail?.conflict
|
||||
? null
|
||||
: (detail?.candidate ?? null),
|
||||
body_bytes: body?.available ? body.total_bytes : null,
|
||||
dossier_evidence_id: detail?.conflict ? null : (detail?.dossier ?? null),
|
||||
evidence_links: [...links].sort(),
|
||||
limitations,
|
||||
},
|
||||
symbol: symbol.value,
|
||||
fingerprint: fingerprint?.key ?? null,
|
||||
};
|
||||
};
|
||||
|
||||
/** Consume normalized producer facts once; never infer body extents from neighbor entries. */
|
||||
export const readFunctionInventory = async (
|
||||
inventory: Evidence,
|
||||
observations: readonly Evidence[],
|
||||
checkpoint: NativeMatchingCheckpoint,
|
||||
): Promise<MatchingFunction[]> => {
|
||||
if (
|
||||
inventory.operation !== "list_procedures" ||
|
||||
inventory.predicate_type !== "rea.analysis" ||
|
||||
inventory.subject === null
|
||||
)
|
||||
throw new EvidenceIntegrityError(
|
||||
"Expected artifact-bound list_procedures Evidence",
|
||||
);
|
||||
const rows = nativeInventoryRowsSchemas.names.parse(
|
||||
inventory.normalized_result,
|
||||
);
|
||||
const facts: Facts = {
|
||||
details: new Map(),
|
||||
symbols: new Map(),
|
||||
bytes: new FunctionByteCoverage(checkpoint),
|
||||
};
|
||||
for (const observation of observations) {
|
||||
await checkpoint();
|
||||
requireContext(inventory, observation);
|
||||
switch (observation.operation) {
|
||||
case "list_names":
|
||||
await addNames(observation, facts.symbols, checkpoint);
|
||||
break;
|
||||
case "read_bytes":
|
||||
await addBytes(observation, facts.bytes);
|
||||
break;
|
||||
case "procedure_info":
|
||||
case "analyze_function":
|
||||
addBody(observation, facts.details);
|
||||
break;
|
||||
default:
|
||||
throw new EvidenceIntegrityError(
|
||||
"Function matching observations must be list_names, procedure_info, read_bytes or analyze_function Evidence",
|
||||
);
|
||||
}
|
||||
}
|
||||
const functions: MatchingFunction[] = [];
|
||||
for (let sourceIndex = 0; sourceIndex < rows.length; sourceIndex++) {
|
||||
const row = rows[sourceIndex]!;
|
||||
await checkpoint(1024 + 4 * (row.address.length + row.value.length));
|
||||
functions.push(
|
||||
await functionFact({ inventory, row, sourceIndex }, facts, checkpoint),
|
||||
);
|
||||
}
|
||||
return functions;
|
||||
};
|
||||
@@ -0,0 +1,204 @@
|
||||
import { digestCanonicalValue } from "../canonicalDigest.js";
|
||||
import { z } from "zod";
|
||||
import { digestSchema, prefixedDigestSchema } from "../digests.js";
|
||||
import { retainedEvidenceReferenceSchema } from "../evidenceReference.js";
|
||||
|
||||
const evidenceId = prefixedDigestSchema("ev");
|
||||
export const nativeMatchStateSchema = z.enum([
|
||||
"matched",
|
||||
"only_left",
|
||||
"only_right",
|
||||
"unresolved",
|
||||
]);
|
||||
export const nativeFunctionMatchMemberSchema = z.strictObject({
|
||||
function_id: z.string().min(1),
|
||||
inventory_evidence_id: evidenceId,
|
||||
source_index: z.number().int().nonnegative(),
|
||||
address: z.string(),
|
||||
name: z.string(),
|
||||
body_sha256: digestSchema.nullable(),
|
||||
structural_candidate_sha256: digestSchema.nullable(),
|
||||
body_bytes: z.number().int().nonnegative().nullable(),
|
||||
dossier_evidence_id: evidenceId.nullable(),
|
||||
evidence_links: z.array(evidenceId).min(1),
|
||||
limitations: z.array(z.string()),
|
||||
});
|
||||
export const nativeFunctionMatchSchema = z.strictObject({
|
||||
match_id: prefixedDigestSchema("match"),
|
||||
state: nativeMatchStateSchema,
|
||||
basis: z.enum(["exact-symbol", "identical-bytes"]).nullable(),
|
||||
left: z.array(nativeFunctionMatchMemberSchema),
|
||||
right: z.array(nativeFunctionMatchMemberSchema),
|
||||
reason: z.string(),
|
||||
compare_functions_input: z
|
||||
.strictObject({
|
||||
left: retainedEvidenceReferenceSchema,
|
||||
right: retainedEvidenceReferenceSchema,
|
||||
})
|
||||
.nullable(),
|
||||
});
|
||||
export const nativeFunctionMatchingSummarySchema = z.strictObject({
|
||||
matched: z.number().int().nonnegative(),
|
||||
only_left: z.number().int().nonnegative(),
|
||||
only_right: z.number().int().nonnegative(),
|
||||
unresolved: z.number().int().nonnegative(),
|
||||
});
|
||||
export const nativeFunctionMatchingContextSchema = z.strictObject({
|
||||
left: z.strictObject({
|
||||
evidence_id: evidenceId,
|
||||
sha256: digestSchema,
|
||||
functions: z.number().int().nonnegative(),
|
||||
}),
|
||||
right: z.strictObject({
|
||||
evidence_id: evidenceId,
|
||||
sha256: digestSchema,
|
||||
functions: z.number().int().nonnegative(),
|
||||
}),
|
||||
scope: z.literal("provider-observed-inputs"),
|
||||
source_coverage: z.strictObject({
|
||||
status: z.enum(["complete-within-inputs", "partial"]),
|
||||
left_inventory_completeness: z.literal("unknown"),
|
||||
right_inventory_completeness: z.literal("unknown"),
|
||||
left_byte_fingerprints: z.number().int().nonnegative(),
|
||||
right_byte_fingerprints: z.number().int().nonnegative(),
|
||||
left_structural_candidates: z.number().int().nonnegative(),
|
||||
right_structural_candidates: z.number().int().nonnegative(),
|
||||
}),
|
||||
summary: nativeFunctionMatchingSummarySchema,
|
||||
});
|
||||
export const nativeFunctionMatchingSchema = nativeFunctionMatchingContextSchema
|
||||
.extend({
|
||||
matches: z.array(nativeFunctionMatchSchema),
|
||||
limitations: z.array(z.string()),
|
||||
})
|
||||
.superRefine((report, context) => {
|
||||
const seen = new Set<string>();
|
||||
const members = { left: new Set<string>(), right: new Set<string>() };
|
||||
const fingerprints = { left: 0, right: 0 };
|
||||
const candidates = { left: 0, right: 0 };
|
||||
const counts = { matched: 0, only_left: 0, only_right: 0, unresolved: 0 };
|
||||
let left = 0,
|
||||
right = 0;
|
||||
for (const match of report.matches) {
|
||||
counts[match.state]++;
|
||||
left += match.left.length;
|
||||
right += match.right.length;
|
||||
if (seen.has(match.match_id))
|
||||
context.addIssue({
|
||||
code: "custom",
|
||||
message: "Repeated match identity",
|
||||
});
|
||||
seen.add(match.match_id);
|
||||
if (
|
||||
match.match_id !==
|
||||
`match_${digestCanonicalValue({ state: match.state, basis: match.basis, left: match.left.map((x) => x.function_id), right: match.right.map((x) => x.function_id) })}`
|
||||
)
|
||||
context.addIssue({
|
||||
code: "custom",
|
||||
message:
|
||||
"Match identity does not bind its selected occurrences and basis",
|
||||
});
|
||||
for (const side of ["left", "right"] as const)
|
||||
for (const member of match[side]) {
|
||||
if (
|
||||
member.inventory_evidence_id !== report[side].evidence_id ||
|
||||
member.source_index >= report[side].functions ||
|
||||
member.function_id !==
|
||||
`${report[side].evidence_id}/${member.source_index}` ||
|
||||
members[side].has(member.function_id) ||
|
||||
!member.evidence_links.includes(member.inventory_evidence_id) ||
|
||||
(member.dossier_evidence_id !== null &&
|
||||
!member.evidence_links.includes(member.dossier_evidence_id))
|
||||
)
|
||||
context.addIssue({
|
||||
code: "custom",
|
||||
message:
|
||||
"Function occurrence identity, provenance or uniqueness disagrees with its selected inventory",
|
||||
});
|
||||
members[side].add(member.function_id);
|
||||
if (member.body_sha256 !== null) fingerprints[side]++;
|
||||
if (member.structural_candidate_sha256 !== null) candidates[side]++;
|
||||
}
|
||||
if (
|
||||
match.left.length + match.right.length === 0 ||
|
||||
(match.state !== "matched" &&
|
||||
(match.basis !== null || match.compare_functions_input !== null))
|
||||
)
|
||||
context.addIssue({
|
||||
code: "custom",
|
||||
message:
|
||||
"Unresolved/unpaired group cannot contain a definitive basis or comparison input",
|
||||
});
|
||||
if (
|
||||
match.basis === "identical-bytes" &&
|
||||
(match.left[0]?.body_sha256 == null ||
|
||||
match.left[0]?.body_sha256 !== match.right[0]?.body_sha256 ||
|
||||
match.left[0]?.body_bytes !== match.right[0]?.body_bytes)
|
||||
)
|
||||
context.addIssue({
|
||||
code: "custom",
|
||||
message:
|
||||
"Identical-byte match disagrees with its complete body digests and sizes",
|
||||
});
|
||||
if (
|
||||
match.compare_functions_input !== null &&
|
||||
(match.compare_functions_input.left.evidence_id !==
|
||||
match.left[0]?.dossier_evidence_id ||
|
||||
match.compare_functions_input.right.evidence_id !==
|
||||
match.right[0]?.dossier_evidence_id)
|
||||
)
|
||||
context.addIssue({
|
||||
code: "custom",
|
||||
message: "Comparison input does not identify this pair's dossiers",
|
||||
});
|
||||
|
||||
if (
|
||||
(match.state === "matched" &&
|
||||
(match.left.length !== 1 ||
|
||||
match.right.length !== 1 ||
|
||||
match.basis === null)) ||
|
||||
(match.state === "only_left" &&
|
||||
(match.left.length !== 1 || match.right.length !== 0)) ||
|
||||
(match.state === "only_right" &&
|
||||
(match.right.length !== 1 || match.left.length !== 0))
|
||||
)
|
||||
context.addIssue({
|
||||
code: "custom",
|
||||
message: "Match state contradicts its occurrences",
|
||||
});
|
||||
}
|
||||
if (
|
||||
report.source_coverage.status !==
|
||||
(counts.only_left + counts.only_right + counts.unresolved > 0
|
||||
? "partial"
|
||||
: "complete-within-inputs") ||
|
||||
report.source_coverage.left_byte_fingerprints !== fingerprints.left ||
|
||||
report.source_coverage.right_byte_fingerprints !== fingerprints.right ||
|
||||
report.source_coverage.left_structural_candidates !== candidates.left ||
|
||||
report.source_coverage.right_structural_candidates !== candidates.right
|
||||
)
|
||||
context.addIssue({
|
||||
code: "custom",
|
||||
message: "Source/matching coverage contradicts examined facts",
|
||||
});
|
||||
if (
|
||||
left !== report.left.functions ||
|
||||
right !== report.right.functions ||
|
||||
Object.keys(counts).some(
|
||||
(key) =>
|
||||
counts[key as keyof typeof counts] !==
|
||||
report.summary[key as keyof typeof counts],
|
||||
)
|
||||
)
|
||||
context.addIssue({
|
||||
code: "custom",
|
||||
message: "Matching summary or examined inventory coverage disagrees",
|
||||
});
|
||||
});
|
||||
export type NativeFunctionMatch = z.infer<typeof nativeFunctionMatchSchema>;
|
||||
export type NativeFunctionMatchMember = z.infer<
|
||||
typeof nativeFunctionMatchMemberSchema
|
||||
>;
|
||||
export type NativeFunctionMatching = z.infer<
|
||||
typeof nativeFunctionMatchingSchema
|
||||
>;
|
||||
@@ -0,0 +1,54 @@
|
||||
import type { Evidence } from "../evidence.js";
|
||||
import type { FunctionDossier } from "../hopperValues.js";
|
||||
import { digestCanonicalValue } from "../canonicalDigest.js";
|
||||
import {
|
||||
isAutoName,
|
||||
normalizeCfg,
|
||||
} from "../functionComparisonNormalization.js";
|
||||
import { assessFunctionCallCoverage } from "../functionCallCoverage.js";
|
||||
import { compareUnicodeCodePoints } from "../unicodeCodePointOrder.js";
|
||||
|
||||
/** An observed structural signature proposes candidates; it never proves identity. */
|
||||
export const functionStructuralCandidate = (
|
||||
evidence: Evidence,
|
||||
dossier: FunctionDossier,
|
||||
): string | null => {
|
||||
if (
|
||||
evidence.subject?.architecture == null ||
|
||||
dossier.facet_availability?.cfg?.available !== true ||
|
||||
dossier.facet_availability?.callees?.available !== true ||
|
||||
dossier.facet_availability?.strings_names?.available !== true ||
|
||||
dossier.unresolved_calls === undefined ||
|
||||
assessFunctionCallCoverage(evidence.relationship_coverage, dossier).blockers
|
||||
.length > 0 ||
|
||||
(dossier.swift_string_literal_omissions?.length ?? 0) > 0 ||
|
||||
evidence.analysis_profile === null
|
||||
)
|
||||
return null;
|
||||
const cfg = normalizeCfg(dossier.basic_blocks);
|
||||
if (
|
||||
cfg === null ||
|
||||
cfg.length === 0 ||
|
||||
dossier.callees.some(
|
||||
(callee) => isAutoName(callee.name) || callee.name.length === 0,
|
||||
)
|
||||
)
|
||||
return null;
|
||||
const calls = dossier.callees
|
||||
.map((callee) => callee.name)
|
||||
.sort(compareUnicodeCodePoints);
|
||||
const strings = [
|
||||
...dossier.referenced_strings.map((value) => value.value),
|
||||
...(dossier.swift_string_literals ?? []).map((value) => value.value),
|
||||
].sort(compareUnicodeCodePoints);
|
||||
// Empty leaf functions otherwise produce huge uninformative CFG-only buckets.
|
||||
if (calls.length === 0 && strings.length === 0) return null;
|
||||
return digestCanonicalValue({
|
||||
algorithm: "observed-cfg-calls-strings-v1",
|
||||
profile: evidence.analysis_profile.digest,
|
||||
architecture: evidence.subject.architecture,
|
||||
cfg,
|
||||
calls,
|
||||
strings,
|
||||
});
|
||||
};
|
||||
@@ -0,0 +1,197 @@
|
||||
import { digestCanonicalValue } from "../canonicalDigest.js";
|
||||
import type { MatchingFunction } from "./functionInventory.js";
|
||||
import type { NativeMatchingCheckpoint } from "./functionByteCoverage.js";
|
||||
import type {
|
||||
NativeFunctionMatch,
|
||||
NativeFunctionMatching,
|
||||
} from "./functionMatching.js";
|
||||
|
||||
type Groups = ReadonlyMap<string, MatchingFunction[]>;
|
||||
type PairingIndexes = readonly [Groups, Groups];
|
||||
interface MatchInput {
|
||||
readonly state: NativeFunctionMatch["state"];
|
||||
readonly basis: NativeFunctionMatch["basis"];
|
||||
readonly left: MatchingFunction[];
|
||||
readonly right: MatchingFunction[];
|
||||
readonly reason: string;
|
||||
}
|
||||
const index = (
|
||||
items: readonly MatchingFunction[],
|
||||
key: (item: MatchingFunction) => string | null,
|
||||
): Groups => {
|
||||
const groups = new Map<string, MatchingFunction[]>();
|
||||
for (const item of items) {
|
||||
const value = key(item);
|
||||
if (value === null) continue;
|
||||
let group = groups.get(value);
|
||||
if (group === undefined) groups.set(value, (group = []));
|
||||
group.push(item);
|
||||
}
|
||||
return groups;
|
||||
};
|
||||
|
||||
/** One request owns pairing state; ambiguous components never expand into pair products. */
|
||||
class InventoryMatcher {
|
||||
readonly consumed = new Set<MatchingFunction>();
|
||||
readonly matches: NativeFunctionMatch[] = [];
|
||||
readonly #parent = new Map<MatchingFunction, MatchingFunction>();
|
||||
readonly #ambiguous = new Set<MatchingFunction>();
|
||||
constructor(readonly checkpoint: NativeMatchingCheckpoint) {}
|
||||
add(input: MatchInput): void {
|
||||
const left = input.left.map((x) => x.member),
|
||||
right = input.right.map((x) => x.member);
|
||||
const leftDossier = left[0]?.dossier_evidence_id,
|
||||
rightDossier = right[0]?.dossier_evidence_id;
|
||||
this.matches.push({
|
||||
match_id: `match_${digestCanonicalValue({ state: input.state, basis: input.basis, left: left.map((x) => x.function_id), right: right.map((x) => x.function_id) })}`,
|
||||
state: input.state,
|
||||
basis: input.basis,
|
||||
left,
|
||||
right,
|
||||
reason: input.reason,
|
||||
compare_functions_input:
|
||||
input.state === "matched" && leftDossier != null && rightDossier != null
|
||||
? {
|
||||
left: { kind: "retained-evidence", evidence_id: leftDossier },
|
||||
right: { kind: "retained-evidence", evidence_id: rightDossier },
|
||||
}
|
||||
: null,
|
||||
});
|
||||
for (const item of [...input.left, ...input.right]) this.consumed.add(item);
|
||||
}
|
||||
async pairUnique(
|
||||
groups: PairingIndexes,
|
||||
basis: NonNullable<NativeFunctionMatch["basis"]>,
|
||||
): Promise<void> {
|
||||
for (const [key, left] of groups[0]) {
|
||||
await this.checkpoint();
|
||||
const right = groups[1].get(key);
|
||||
// Global multiplicity still matters after a different symbol consumes an alias.
|
||||
if (
|
||||
left.length !== 1 ||
|
||||
right?.length !== 1 ||
|
||||
this.consumed.has(left[0]!) ||
|
||||
this.consumed.has(right[0]!)
|
||||
)
|
||||
continue;
|
||||
this.add({
|
||||
state: "matched",
|
||||
basis,
|
||||
left,
|
||||
right,
|
||||
reason:
|
||||
basis === "exact-symbol"
|
||||
? "Unique exact reported symbol; this does not establish unchanged code or semantic equivalence."
|
||||
: "Unique identical observed function bytes, total size and ordered range lengths; relocations are not normalized.",
|
||||
});
|
||||
}
|
||||
}
|
||||
#find(item: MatchingFunction): MatchingFunction {
|
||||
let root = item;
|
||||
while (this.#parent.has(root) && this.#parent.get(root) !== root)
|
||||
root = this.#parent.get(root)!;
|
||||
let current = item;
|
||||
while (this.#parent.has(current) && this.#parent.get(current) !== root) {
|
||||
const next = this.#parent.get(current)!;
|
||||
this.#parent.set(current, root);
|
||||
current = next;
|
||||
}
|
||||
return root;
|
||||
}
|
||||
async connectCandidates(indexes: readonly PairingIndexes[]): Promise<void> {
|
||||
for (const groups of indexes)
|
||||
for (const [key, left] of groups[0]) {
|
||||
await this.checkpoint();
|
||||
const right = groups[1].get(key);
|
||||
if (right === undefined) continue;
|
||||
const pending = [...left, ...right].filter(
|
||||
(item) => !this.consumed.has(item),
|
||||
),
|
||||
first = pending[0];
|
||||
if (first === undefined) continue;
|
||||
for (const item of pending) {
|
||||
this.#parent.set(this.#find(item), this.#find(first));
|
||||
this.#ambiguous.add(item);
|
||||
}
|
||||
}
|
||||
}
|
||||
async finish(
|
||||
left: readonly MatchingFunction[],
|
||||
right: readonly MatchingFunction[],
|
||||
): Promise<void> {
|
||||
const groups = new Map<
|
||||
MatchingFunction,
|
||||
{ left: MatchingFunction[]; right: MatchingFunction[] }
|
||||
>();
|
||||
for (const [side, items] of [
|
||||
["left", left],
|
||||
["right", right],
|
||||
] as const)
|
||||
for (const item of items) {
|
||||
await this.checkpoint();
|
||||
if (this.consumed.has(item)) continue;
|
||||
if (!this.#ambiguous.has(item)) {
|
||||
this.add({
|
||||
state: side === "left" ? "only_left" : "only_right",
|
||||
basis: null,
|
||||
left: side === "left" ? [item] : [],
|
||||
right: side === "right" ? [item] : [],
|
||||
reason:
|
||||
"No exact correspondence in the supplied observed inventories. This is not proof of source addition/removal: renaming, inlining, stripping, byte changes or omitted functions can explain it.",
|
||||
});
|
||||
continue;
|
||||
}
|
||||
const root = this.#find(item);
|
||||
let group = groups.get(root);
|
||||
if (group === undefined)
|
||||
groups.set(root, (group = { left: [], right: [] }));
|
||||
group[side].push(item);
|
||||
}
|
||||
for (const group of groups.values())
|
||||
this.add({
|
||||
state: "unresolved",
|
||||
basis: null,
|
||||
...group,
|
||||
reason:
|
||||
"Repeated exact names/bytes, aliases or observed CFG/call/string fingerprints leave correspondence unresolved. Connected group membership does not imply pairwise equivalence. No candidate is forced into a pair.",
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
/** Match only exact unique facts, preserving every occurrence and all candidate ambiguity. */
|
||||
export const matchFunctionInventories = async (
|
||||
left: readonly MatchingFunction[],
|
||||
right: readonly MatchingFunction[],
|
||||
options: {
|
||||
readonly identicalBytesComparable: boolean;
|
||||
readonly checkpoint: NativeMatchingCheckpoint;
|
||||
},
|
||||
): Promise<{
|
||||
matches: NativeFunctionMatch[];
|
||||
summary: NativeFunctionMatching["summary"];
|
||||
}> => {
|
||||
const names: PairingIndexes = [
|
||||
index(left, (x) => x.symbol),
|
||||
index(right, (x) => x.symbol),
|
||||
];
|
||||
const bytes: PairingIndexes = [
|
||||
index(left, (x) =>
|
||||
options.identicalBytesComparable ? x.fingerprint : null,
|
||||
),
|
||||
index(right, (x) =>
|
||||
options.identicalBytesComparable ? x.fingerprint : null,
|
||||
),
|
||||
];
|
||||
const structural: PairingIndexes = [
|
||||
index(left, (x) => x.member.structural_candidate_sha256),
|
||||
index(right, (x) => x.member.structural_candidate_sha256),
|
||||
];
|
||||
const matcher = new InventoryMatcher(options.checkpoint);
|
||||
await matcher.pairUnique(names, "exact-symbol");
|
||||
await matcher.pairUnique(bytes, "identical-bytes");
|
||||
await matcher.connectCandidates([names, bytes, structural]);
|
||||
await matcher.finish(left, right);
|
||||
const summary = { matched: 0, only_left: 0, only_right: 0, unresolved: 0 };
|
||||
for (const match of matcher.matches) summary[match.state]++;
|
||||
return { matches: matcher.matches, summary };
|
||||
};
|
||||
@@ -0,0 +1,22 @@
|
||||
import { z } from "zod";
|
||||
|
||||
/** Provider-owned bytes and exact coverage of one requested memory interval. */
|
||||
export const nativeByteReadSchema = z
|
||||
.object({
|
||||
address: z.string(),
|
||||
requested_bytes: z.number().int().positive().max(Number.MAX_SAFE_INTEGER),
|
||||
returned_bytes: z.number().int().nonnegative().max(Number.MAX_SAFE_INTEGER),
|
||||
bytes_hex: z.string().regex(/^(?:[a-f0-9]{2})*$/u),
|
||||
complete: z.boolean(),
|
||||
})
|
||||
.superRefine((value, context) => {
|
||||
if (
|
||||
value.returned_bytes * 2 !== value.bytes_hex.length ||
|
||||
value.returned_bytes > value.requested_bytes ||
|
||||
value.complete !== (value.returned_bytes === value.requested_bytes)
|
||||
)
|
||||
context.addIssue({
|
||||
code: "custom",
|
||||
message: "Read-byte length and completeness disagree",
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,18 @@
|
||||
import { z } from "zod";
|
||||
import {
|
||||
functionBodySchema,
|
||||
localVariableSchema,
|
||||
procedureClassificationSchema,
|
||||
} from "../hopperValues.js";
|
||||
|
||||
/** Provider-neutral procedure metadata, including exact body ranges. */
|
||||
export const procedureInfoSchema = z.object({
|
||||
name: z.string(),
|
||||
entrypoint: z.string(),
|
||||
basicblock_count: z.number().int().min(0),
|
||||
length: z.number().min(0),
|
||||
signature: z.string().nullable(),
|
||||
locals: z.array(localVariableSchema),
|
||||
classification: procedureClassificationSchema.nullable(),
|
||||
body: functionBodySchema,
|
||||
});
|
||||
@@ -1,3 +1,4 @@
|
||||
import { procedureInfoSchema } from "../domain/native/procedureInfo.js";
|
||||
import { z } from "zod";
|
||||
import {
|
||||
nativeFunctionAnnotationsInputSchema,
|
||||
@@ -181,13 +182,11 @@ const procedureReferences = z
|
||||
),
|
||||
})
|
||||
.strict();
|
||||
const procedureInfo = z
|
||||
.object({
|
||||
const procedureInfo = procedureInfoSchema
|
||||
.safeExtend({
|
||||
name: z.string().min(1),
|
||||
entrypoint: ghidraCanonicalAddressSchema,
|
||||
basicblock_count: z.number().int().min(0),
|
||||
length: z.number().int().min(0),
|
||||
signature: z.string().nullable(),
|
||||
locals: z.array(localVariable),
|
||||
classification,
|
||||
body: ghidraFunctionBody,
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { nativeByteReadSchema } from "../domain/native/nativeByteRead.js";
|
||||
import { z } from "zod";
|
||||
import {
|
||||
bytePatternFields,
|
||||
@@ -320,25 +321,9 @@ const resultSchemas = {
|
||||
});
|
||||
}),
|
||||
inspect_native_load_image: nativeLoadImageObservationSchema,
|
||||
read_bytes: z
|
||||
.strictObject({
|
||||
address: canonicalAddress,
|
||||
requested_bytes: z.number().int().min(1),
|
||||
returned_bytes: z.number().int().min(0),
|
||||
bytes_hex: z.string().regex(/^(?:[a-f0-9]{2})*$/u),
|
||||
complete: z.boolean(),
|
||||
})
|
||||
.superRefine((value, context) => {
|
||||
if (
|
||||
value.returned_bytes * 2 !== value.bytes_hex.length ||
|
||||
value.returned_bytes > value.requested_bytes ||
|
||||
value.complete !== (value.returned_bytes === value.requested_bytes)
|
||||
)
|
||||
context.addIssue({
|
||||
code: "custom",
|
||||
message: "Read-byte length and completeness disagree",
|
||||
});
|
||||
}),
|
||||
read_bytes: nativeByteReadSchema
|
||||
.safeExtend({ address: canonicalAddress })
|
||||
.strict(),
|
||||
address_to_file_offset: z.strictObject({
|
||||
address: canonicalAddress,
|
||||
file_offset: z.number().int().min(0).max(Number.MAX_SAFE_INTEGER),
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { resolvePairedEvidenceRequest } from "../application/EvidenceInputResolver.js";
|
||||
import type { EvidenceMcpServer } from "./EvidenceMcpServer.js";
|
||||
|
||||
import type { BinarySessionPort } from "../application/binary/BinarySessionPort.js";
|
||||
@@ -38,11 +39,18 @@ export const registerFunctionComparisonTool = (
|
||||
contract.name,
|
||||
context.mcpReq.signal,
|
||||
async () => {
|
||||
const resolved = resolvePairedEvidenceRequest(
|
||||
input,
|
||||
(id) =>
|
||||
session.evidenceForAnalysis?.(id) ?? session.evidenceById(id),
|
||||
);
|
||||
if (!resolved.ok)
|
||||
return server.delivery.toCallToolResult(resolved, contract);
|
||||
let leftEvidence: Evidence;
|
||||
let rightEvidence: Evidence;
|
||||
try {
|
||||
leftEvidence = parseEvidence(input.left);
|
||||
rightEvidence = parseEvidence(input.right);
|
||||
leftEvidence = parseEvidence(resolved.value.left);
|
||||
rightEvidence = parseEvidence(resolved.value.right);
|
||||
} catch (cause: unknown) {
|
||||
return server.delivery.toCallToolResult(
|
||||
err(
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
import type { EvidenceMcpServer } from "./EvidenceMcpServer.js";
|
||||
import type { BinarySessionPort } from "../application/binary/BinarySessionPort.js";
|
||||
import { toolContract } from "../contracts/toolContracts.js";
|
||||
import { matchNativeFunctions } from "../application/native/NativeFunctionMatchingService.js";
|
||||
import { summarizeRetainedAnalysis } from "../application/analysisView/AnalysisViewService.js";
|
||||
import { withAdmittedAnalysis } from "./analysisAdmission.js";
|
||||
import { createEvidenceToolHandler } from "./contractToolHandler.js";
|
||||
import { toolRegistrationOptions } from "./toolRegistrationOptions.js";
|
||||
import { recordSessionEvidenceSources } from "./sessionEvidence.js";
|
||||
import { matchNativeFunctionsInputSchema } from "../contracts/native/functionMatchingContracts.js";
|
||||
import { analysisInputErrorFromIssues } from "../domain/inputIssueProjection.js";
|
||||
import { err, ok } from "../domain/result.js";
|
||||
import type { Logger } from "pino";
|
||||
|
||||
/** Bind one named contract to the same retained-Evidence workflow used by CLI. */
|
||||
export const registerNativeFunctionMatchingTool = (
|
||||
server: EvidenceMcpServer,
|
||||
session: BinarySessionPort,
|
||||
logger: Logger,
|
||||
): void => {
|
||||
const contract = toolContract("match_native_functions");
|
||||
const evidenceById = (id: string) =>
|
||||
session.evidenceForAnalysis?.(id) ?? session.evidenceById(id);
|
||||
const recordEvidence: BinarySessionPort["recordEvidence"] = (evidence) =>
|
||||
session.recordEvidence(evidence);
|
||||
const handler = createEvidenceToolHandler(
|
||||
server,
|
||||
async (input, options) => {
|
||||
const parsed = matchNativeFunctionsInputSchema.safeParse(input);
|
||||
if (!parsed.success)
|
||||
return err(
|
||||
analysisInputErrorFromIssues(
|
||||
contract.name,
|
||||
parsed.error.issues,
|
||||
input,
|
||||
),
|
||||
);
|
||||
const result = await matchNativeFunctions(
|
||||
parsed.data,
|
||||
options,
|
||||
evidenceById,
|
||||
);
|
||||
if (!result.ok) return result;
|
||||
const recorded = recordSessionEvidenceSources(
|
||||
recordEvidence,
|
||||
result.value.sources,
|
||||
);
|
||||
if (!recorded.ok) return recorded;
|
||||
return parsed.data.detail === "summary"
|
||||
? summarizeRetainedAnalysis(result.value.evidence, {
|
||||
evidenceById,
|
||||
recordEvidence,
|
||||
})
|
||||
: ok(result.value.evidence);
|
||||
},
|
||||
logger,
|
||||
recordEvidence,
|
||||
withAdmittedAnalysis({ kind: "session", session }),
|
||||
);
|
||||
server.registerTool(
|
||||
contract.name,
|
||||
toolRegistrationOptions(contract),
|
||||
handler(contract),
|
||||
);
|
||||
};
|
||||
@@ -1,3 +1,4 @@
|
||||
import { registerNativeFunctionMatchingTool } from "./registerNativeFunctionMatchingTool.js";
|
||||
import type { EvidenceMcpServer } from "./EvidenceMcpServer.js";
|
||||
import type { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
|
||||
import { workflowUnknownInput } from "../application/investigation/WorkflowUnknown.js";
|
||||
@@ -375,6 +376,7 @@ export const registerSessionTools = (
|
||||
registerProcessComparisonTool(server, session, compareContract);
|
||||
registerArtifactComparisonTool(server, session, compareArtifactsContract);
|
||||
registerFunctionComparisonTool(server, session, compareFunctionsContract);
|
||||
registerNativeFunctionMatchingTool(server, session, logger);
|
||||
registerBundleComparisonTool(
|
||||
server,
|
||||
session,
|
||||
|
||||
@@ -0,0 +1,276 @@
|
||||
import { readFile, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { expect } from "vitest";
|
||||
import { cliTest, type TestCli } from "../../support/cli/cliFixture.js";
|
||||
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
|
||||
import { connectLocalToolsMcp } from "../../fixtures/localToolsMcp.js";
|
||||
import {
|
||||
nativeMatchingInventory,
|
||||
nativeMatchingDetails,
|
||||
} from "../../fixtures/nativeFunctionMatching.js";
|
||||
import { createEvidence, parseEvidence } from "../../../src/domain/evidence.js";
|
||||
import { nativeMatchingProvider } from "../../fixtures/nativeFunctionMatching.js";
|
||||
import {
|
||||
createEvidenceBundle,
|
||||
parseEvidenceBundle,
|
||||
} from "../../../src/domain/evidenceBundle.js";
|
||||
import { nativeFunctionMatchingSchema } from "../../../src/domain/native/functionMatching.js";
|
||||
|
||||
const createMatchingFixture = async () => {
|
||||
const root = await createTestTempDirectory("rea-native-matching-");
|
||||
const ancestor = createEvidence(undefined, nativeMatchingProvider, {
|
||||
operation: "fixture_provenance",
|
||||
parameters: {},
|
||||
result: { source: "Synthetic ancestor" },
|
||||
});
|
||||
const unrelated = createEvidence(undefined, nativeMatchingProvider, {
|
||||
operation: "fixture_unrelated",
|
||||
parameters: {},
|
||||
result: { source: "Unselected synthetic record" },
|
||||
});
|
||||
const dependency = createEvidence(undefined, nativeMatchingProvider, {
|
||||
operation: "fixture_dependency",
|
||||
parameters: {},
|
||||
result: { source: "Synthetic dependency" },
|
||||
evidenceLinks: [ancestor.evidence_id],
|
||||
});
|
||||
const left = nativeMatchingInventory(
|
||||
"1",
|
||||
[
|
||||
{ address: "0x10", value: "main" },
|
||||
{ address: "0x20", value: "old_name" },
|
||||
],
|
||||
[dependency.evidence_id],
|
||||
);
|
||||
const right = nativeMatchingInventory("2", [
|
||||
{ address: "0x30", value: "main" },
|
||||
{ address: "0x40", value: "new_name" },
|
||||
]);
|
||||
const leftDetails = [
|
||||
...nativeMatchingDetails("1", "0x10", "9000", true),
|
||||
...nativeMatchingDetails("1", "0x20", "c3", true),
|
||||
];
|
||||
const rightDetails = [
|
||||
...nativeMatchingDetails("2", "0x30", "9001", true),
|
||||
...nativeMatchingDetails("2", "0x40", "c3", true),
|
||||
];
|
||||
const ancestorsPath = join(root, "ancestors.json");
|
||||
await writeFile(
|
||||
ancestorsPath,
|
||||
JSON.stringify(createEvidenceBundle([ancestor, unrelated])),
|
||||
);
|
||||
const importedPath = join(root, "dependencies.json");
|
||||
await writeFile(
|
||||
importedPath,
|
||||
JSON.stringify(createEvidenceBundle([ancestor, dependency])),
|
||||
);
|
||||
return {
|
||||
root,
|
||||
ancestor,
|
||||
dependency,
|
||||
unrelated,
|
||||
left,
|
||||
right,
|
||||
leftDetails,
|
||||
rightDetails,
|
||||
ancestorsPath,
|
||||
importedPath,
|
||||
};
|
||||
};
|
||||
const verifyPortableCli = async (
|
||||
cli: TestCli,
|
||||
fixture: Awaited<ReturnType<typeof createMatchingFixture>>,
|
||||
input: Record<string, unknown>,
|
||||
evidenceId: string,
|
||||
) => {
|
||||
const {
|
||||
root,
|
||||
ancestor,
|
||||
dependency,
|
||||
unrelated,
|
||||
left,
|
||||
right,
|
||||
leftDetails,
|
||||
rightDetails,
|
||||
ancestorsPath,
|
||||
} = fixture;
|
||||
const bundlePath = join(root, "sources.json"),
|
||||
inputPath = join(root, "request.json"),
|
||||
outputPath = join(root, "result.json");
|
||||
await writeFile(
|
||||
bundlePath,
|
||||
JSON.stringify(
|
||||
createEvidenceBundle([
|
||||
left,
|
||||
right,
|
||||
...leftDetails,
|
||||
...rightDetails,
|
||||
dependency,
|
||||
]),
|
||||
),
|
||||
);
|
||||
const original = await readFile(bundlePath);
|
||||
await writeFile(inputPath, JSON.stringify({ ...input, detail: "complete" }));
|
||||
const cliFull = await cli.run({
|
||||
arguments: [
|
||||
"match-native-functions",
|
||||
inputPath,
|
||||
"--source_bundle",
|
||||
ancestorsPath,
|
||||
"--source_bundle",
|
||||
bundlePath,
|
||||
"--json",
|
||||
],
|
||||
});
|
||||
expect(cliFull.exitCode, cliFull.stderr).toBe(0);
|
||||
expect(parseEvidence(cliFull.json).evidence_id).toBe(evidenceId);
|
||||
await writeFile(inputPath, JSON.stringify(input));
|
||||
const cliSummary = await cli.run({
|
||||
arguments: [
|
||||
"match-native-functions",
|
||||
inputPath,
|
||||
"--source_bundle",
|
||||
ancestorsPath,
|
||||
"--source_bundle",
|
||||
bundlePath,
|
||||
"--evidence_output",
|
||||
outputPath,
|
||||
"--json",
|
||||
],
|
||||
});
|
||||
expect(cliSummary.exitCode, cliSummary.stderr).toBe(0);
|
||||
expect(cliSummary.json).toMatchObject({
|
||||
summary: {
|
||||
normalized_result: { parent_evidence_id: evidenceId },
|
||||
},
|
||||
evidence_bundle: { path: outputPath },
|
||||
});
|
||||
const bundle = parseEvidenceBundle(
|
||||
JSON.parse(await readFile(outputPath, "utf8")),
|
||||
);
|
||||
const ids = new Set(bundle.records.map((x) => x.evidence_id));
|
||||
for (const record of bundle.records)
|
||||
for (const id of record.evidence_links)
|
||||
expect(
|
||||
ids.has(id),
|
||||
`Missing dependency ${id} of ${record.evidence_id}`,
|
||||
).toBe(true);
|
||||
expect(ids.has(ancestor.evidence_id)).toBe(true);
|
||||
expect(ids.has(dependency.evidence_id)).toBe(true);
|
||||
expect(ids.has(unrelated.evidence_id)).toBe(false);
|
||||
expect(ids.has(evidenceId)).toBe(true);
|
||||
expect(await readFile(bundlePath)).toEqual(original);
|
||||
const incomplete = await cli.run({
|
||||
arguments: [
|
||||
"match-native-functions",
|
||||
inputPath,
|
||||
"--source_bundle",
|
||||
bundlePath,
|
||||
"--evidence_output",
|
||||
join(root, "incomplete.json"),
|
||||
"--json",
|
||||
],
|
||||
});
|
||||
expect(incomplete.json).toMatchObject({
|
||||
code: "evidence_integrity_mismatch",
|
||||
});
|
||||
};
|
||||
|
||||
cliTest(
|
||||
"shared public CLI/MCP matching retains sources, pages exact facts, and feeds selected dossier pairs directly to compare_functions",
|
||||
async ({ cli }) => {
|
||||
const fixture = await createMatchingFixture();
|
||||
const { left, right, leftDetails, rightDetails, importedPath } = fixture;
|
||||
const { call } = await connectLocalToolsMcp();
|
||||
const imported = await call("import_evidence_bundle", {
|
||||
path: importedPath,
|
||||
});
|
||||
expect(imported.isError, JSON.stringify(imported)).not.toBe(true);
|
||||
const full = await call("match_native_functions", {
|
||||
left,
|
||||
right,
|
||||
left_observations: leftDetails,
|
||||
right_observations: rightDetails,
|
||||
detail: "complete",
|
||||
});
|
||||
expect(full.isError, JSON.stringify(full)).not.toBe(true);
|
||||
const evidence = parseEvidence(full.structuredContent),
|
||||
report = nativeFunctionMatchingSchema.parse(evidence.normalized_result);
|
||||
expect(report.summary).toEqual({
|
||||
matched: 2,
|
||||
only_left: 0,
|
||||
only_right: 0,
|
||||
unresolved: 0,
|
||||
});
|
||||
const reference = (evidence_id: string) => ({
|
||||
kind: "retained-evidence",
|
||||
evidence_id,
|
||||
});
|
||||
const input = {
|
||||
left: reference(left.evidence_id),
|
||||
right: reference(right.evidence_id),
|
||||
left_observations: leftDetails.map((x) => reference(x.evidence_id)),
|
||||
right_observations: rightDetails.map((x) => reference(x.evidence_id)),
|
||||
};
|
||||
const summary = await call("match_native_functions", input);
|
||||
expect(summary.isError, JSON.stringify(summary)).not.toBe(true);
|
||||
expect(
|
||||
parseEvidence(summary.structuredContent).normalized_result,
|
||||
).toMatchObject({
|
||||
kind: "summary",
|
||||
artifact: null,
|
||||
parent_evidence_id: evidence.evidence_id,
|
||||
summary: { summary: report.summary },
|
||||
});
|
||||
expect(parseEvidence(summary.structuredContent).locations).not.toEqual(
|
||||
expect.arrayContaining([
|
||||
expect.objectContaining({ kind: "artifact-path" }),
|
||||
]),
|
||||
);
|
||||
const page = await call("inspect_analysis_view", {
|
||||
source: reference(evidence.evidence_id),
|
||||
view: {
|
||||
kind: "function-matches",
|
||||
collection: "functions",
|
||||
sides: ["left"],
|
||||
fields: ["name", "body_sha256"],
|
||||
offset: 1,
|
||||
limit: 1,
|
||||
},
|
||||
});
|
||||
expect(page.isError, JSON.stringify(page)).not.toBe(true);
|
||||
expect(
|
||||
parseEvidence(page.structuredContent).normalized_result,
|
||||
).toMatchObject({
|
||||
artifact: null,
|
||||
coverage: { total: 2, next_offset: null },
|
||||
items: [
|
||||
{
|
||||
name: "old_name",
|
||||
function_id: `${left.evidence_id}/1`,
|
||||
address: "0x20",
|
||||
},
|
||||
],
|
||||
});
|
||||
expect(parseEvidence(page.structuredContent).locations).not.toEqual(
|
||||
expect.arrayContaining([
|
||||
expect.objectContaining({ kind: "artifact-path" }),
|
||||
]),
|
||||
);
|
||||
const compared = await call(
|
||||
"compare_functions",
|
||||
report.matches[0]!.compare_functions_input!,
|
||||
);
|
||||
expect(compared.isError, JSON.stringify(compared)).not.toBe(true);
|
||||
expect(parseEvidence(compared.structuredContent).operation).toBe(
|
||||
"compare_functions",
|
||||
);
|
||||
await verifyPortableCli(cli, fixture, input, evidence.evidence_id);
|
||||
const missing = await call("match_native_functions", {
|
||||
left: reference(`ev_${"a".repeat(64)}`),
|
||||
right: reference(right.evidence_id),
|
||||
});
|
||||
expect(missing.isError).toBe(true);
|
||||
},
|
||||
60000,
|
||||
);
|
||||
+32
@@ -0,0 +1,32 @@
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#ifndef VERSION
|
||||
#define VERSION 1
|
||||
#endif
|
||||
#define FN __attribute__((noinline, used))
|
||||
FN int stable(int x) { return x * 3 + 7; }
|
||||
FN int changed(int x) { return x + VERSION * 5; }
|
||||
#if VERSION == 1
|
||||
FN int old_behavior(int x) { puts("behavior marker"); return x + 7; }
|
||||
FN int old_name(int x) { return x ^ 0x13579; }
|
||||
FN int removed_observation(int x) { return x * 13 + 99; }
|
||||
FN int formerly_separate(int x) { return x + 12345; }
|
||||
#else
|
||||
FN int new_behavior(int x) { puts("behavior marker"); return x + 8; }
|
||||
FN int renamed(int x) { return x ^ 0x13579; }
|
||||
FN int added_observation(int x) { return x * 17 + 77; }
|
||||
static inline int formerly_separate(int x) { return x + 12345; }
|
||||
#endif
|
||||
FN int alias_one(int x) { return x + 42; }
|
||||
FN int alias_two(int x) { return x + 42; }
|
||||
int main(int argc, char **argv) {
|
||||
int x = argc > 1 ? atoi(argv[1]) : 2;
|
||||
int result = stable(x) + changed(x) + formerly_separate(x) + alias_one(x) + alias_two(x);
|
||||
#if VERSION == 1
|
||||
result += old_name(x) + removed_observation(x) + old_behavior(x);
|
||||
#else
|
||||
result += renamed(x) + added_observation(x) + new_behavior(x);
|
||||
#endif
|
||||
printf("%d\n", result);
|
||||
return 0;
|
||||
}
|
||||
+101
@@ -0,0 +1,101 @@
|
||||
import { createEvidence, type Evidence } from "../../src/domain/evidence.js";
|
||||
import { jsonValueSchema } from "../../src/domain/jsonValue.js";
|
||||
import { FUNCTION_COMPARISON_EXAMPLE } from "../../src/contracts/functionComparisonExample.js";
|
||||
import { functionDossierSchema } from "../../src/domain/hopperValues.js";
|
||||
|
||||
/** Explicit synthetic producer facts; they do not qualify any real provider. */
|
||||
export const nativeMatchingProvider = {
|
||||
id: "matching-fixture",
|
||||
name: "Synthetic native matching facts",
|
||||
version: "1",
|
||||
};
|
||||
export const nativeMatchingTarget = (side: "1" | "2") => ({
|
||||
path: `/tmp/matching-${side}.elf`,
|
||||
sha256: side.repeat(64),
|
||||
format: "elf" as const,
|
||||
architecture: "x86_64" as const,
|
||||
});
|
||||
export const nativeMatchingInventory = (
|
||||
side: "1" | "2",
|
||||
entries: readonly { address: string; value: string }[],
|
||||
evidenceLinks: readonly string[] = [],
|
||||
) =>
|
||||
createEvidence(nativeMatchingTarget(side), nativeMatchingProvider, {
|
||||
operation: "list_procedures",
|
||||
parameters: {},
|
||||
result: jsonValueSchema.parse(entries),
|
||||
evidenceLinks: [...evidenceLinks],
|
||||
});
|
||||
export const nativeMatchingBody = (address: string, bytes: number) => ({
|
||||
available: true as const,
|
||||
provenance: "explicit synthetic body",
|
||||
ranges: [
|
||||
{
|
||||
start: address,
|
||||
end: `0x${(BigInt(address) + BigInt(bytes) - 1n).toString(16)}`,
|
||||
},
|
||||
],
|
||||
total_bytes: bytes,
|
||||
span_bytes: bytes,
|
||||
non_contiguous: false,
|
||||
contains_entry: true,
|
||||
});
|
||||
export const nativeMatchingDetails = (
|
||||
side: "1" | "2",
|
||||
address: string,
|
||||
hex: string,
|
||||
dossier = false,
|
||||
): Evidence[] => {
|
||||
const target = nativeMatchingTarget(side),
|
||||
body = nativeMatchingBody(address, hex.length / 2);
|
||||
const procedure = {
|
||||
name: "fixture",
|
||||
entrypoint: address,
|
||||
basicblock_count: 1,
|
||||
length: hex.length / 2,
|
||||
signature: null,
|
||||
locals: [],
|
||||
classification: null,
|
||||
body,
|
||||
};
|
||||
let observation: Evidence;
|
||||
if (dossier) {
|
||||
const value = functionDossierSchema.parse(
|
||||
FUNCTION_COMPARISON_EXAMPLE.left.normalized_result,
|
||||
);
|
||||
value.procedure = {
|
||||
address,
|
||||
name: "fixture",
|
||||
classification: null,
|
||||
body,
|
||||
signature: null,
|
||||
locals: [],
|
||||
};
|
||||
value.basic_blocks = [];
|
||||
observation = createEvidence(target, nativeMatchingProvider, {
|
||||
operation: "analyze_function",
|
||||
parameters: { procedure: address },
|
||||
result: jsonValueSchema.parse(value),
|
||||
relationshipCoverage: { status: "complete" },
|
||||
});
|
||||
} else
|
||||
observation = createEvidence(target, nativeMatchingProvider, {
|
||||
operation: "procedure_info",
|
||||
parameters: { procedure: address },
|
||||
result: procedure,
|
||||
});
|
||||
return [
|
||||
observation,
|
||||
createEvidence(target, nativeMatchingProvider, {
|
||||
operation: "read_bytes",
|
||||
parameters: { address, length: hex.length / 2 },
|
||||
result: {
|
||||
address,
|
||||
requested_bytes: hex.length / 2,
|
||||
returned_bytes: hex.length / 2,
|
||||
bytes_hex: hex,
|
||||
complete: true,
|
||||
},
|
||||
}),
|
||||
];
|
||||
};
|
||||
Reference in new issue
Block a user