fix(session): clear provider-inferred unavailability after recovery (#8518)
* fix(session): clear provider-inferred unavailability after recovery A provider adapter or process failure marked every routed operation of that provider unavailable, but a later success cleared only the operation that succeeded. Sibling operations, and composed tools built on them, kept advertising the stale failure until each was retried individually. Move runtime availability into one owner that records whether each entry was observed for the operation or inferred from another operation's provider-wide failure. A success clears its own failure and that provider's inferred entries while keeping failures observed on other operations or providers and declared refusals. Outcomes are ordered by dispatch to the provider client, so an older in-flight success no longer erases a newer failure while a request queued behind a failure still clears it. Fixes #8404 Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(session): keep newer successes when older failures complete last Runtime availability kept one merged entry per operation and forgot successful dispatches. An older provider-wide failure that completed after a newer success re-inferred unavailability for siblings the newer success had already proved working, and a superseded refusal could survive its own later success. Record only the newest failure and newest success dispatch per operation and per provider, and project unavailability from them: an operation is unavailable while its own newest failure is newer than its own newest success, or, as inferred, while its provider's newest provider-wide failure is newer than any success from that provider. The projection no longer depends on reply completion order. The recovery tests now bound held-reply acknowledgements, release and join every held call in finally, and compare remediation with the owning formatters instead of copying their wording. Co-Authored-By: Claude Opus 5.5 <[email protected]> --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Kaoru0822andClaude Opus 5.5 authored and GitHub committed
1 parent
35ae7232bf
commit
3f61303939
4 files changed
+744
-58
No files matched your search
@@ -742,6 +742,19 @@ requirements. Compare the union of group names with `tools/list` when checking
|
||||
that a client received the full catalog. An operation absent from all groups is
|
||||
missing from the complete inventory; it is not an implicit unavailable entry.
|
||||
|
||||
Availability also reflects the active provider client's dispatched requests. A
|
||||
refusal or provider failure observed for an operation keeps that operation
|
||||
unavailable until a request for it dispatched later succeeds. A provider adapter
|
||||
or process failure also marks that provider's other declared-available
|
||||
operations unavailable until any request to that provider dispatched later
|
||||
succeeds; their remediation states that the status was inferred rather than
|
||||
observed for them. Recovery is not proof that each operation works, and failures
|
||||
observed on other operations or providers remain. Outcomes are ordered by
|
||||
dispatch to the provider client rather than reply completion, so a late reply
|
||||
from an older request does not override a newer outcome. Request deadlines and
|
||||
cancellations neither mark nor clear availability. A new provider client starts
|
||||
without these outcomes.
|
||||
|
||||
`detail` selects the projection and the result echoes it. The default
|
||||
`summary` keeps open state, target identity, the binding, current-route
|
||||
`capabilities`, run, activity and health status, and each provider candidate's
|
||||
|
||||
@@ -49,6 +49,7 @@ import { InvestigationRecords } from "../investigation/InvestigationRecords.js";
|
||||
import { recordEvidenceSources } from "../investigation/recordEvidenceSources.js";
|
||||
import { BinarySessionRecords } from "./BinarySessionRecords.js";
|
||||
import { binarySessionStatus } from "./BinarySessionStatus.js";
|
||||
import { SessionRuntimeAvailability } from "./SessionRuntimeAvailability.js";
|
||||
import { isSourceEvidenceRetainable } from "./AnalysisSnapshotCache.js";
|
||||
import type { BinarySessionDetail } from "../../contracts/sessionStatusContract.js";
|
||||
import {
|
||||
@@ -197,10 +198,7 @@ export class BinarySession
|
||||
readonly #bundleResourceFactory: BundleResourceClientFactory | undefined;
|
||||
#activeBundleResources: ActiveBundleResourceSession | undefined;
|
||||
#pendingBundleResourceCleanup: ActiveBundleResourceSession[] = [];
|
||||
readonly #runtimeUnavailability = new Map<
|
||||
string,
|
||||
{ readonly reason: string }
|
||||
>();
|
||||
readonly #runtimeAvailability = new SessionRuntimeAvailability();
|
||||
readonly #availabilityListeners = new Set<() => void | Promise<void>>();
|
||||
|
||||
constructor(
|
||||
@@ -1252,7 +1250,7 @@ export class BinarySession
|
||||
}),
|
||||
route: this.#currentRoute(),
|
||||
router: this.#providerRouter,
|
||||
runtimeUnavailability: this.#runtimeUnavailability,
|
||||
runtimeUnavailability: this.#runtimeAvailability.entries,
|
||||
runId: run?.runId,
|
||||
runtimeLineageSnapshots: client?.runtimeLineageSnapshots?.() ?? [],
|
||||
requestActivitySnapshots: client?.requestActivitySnapshots?.() ?? [],
|
||||
@@ -1414,6 +1412,7 @@ export class BinarySession
|
||||
// A failed reply cannot prove that a dispatched metadata write rolled back.
|
||||
// Stop replay before dispatch, including reads overlapping the write.
|
||||
if (capability.effects.mutatesArtifact) this.invalidateSnapshot();
|
||||
const dispatch = this.#runtimeAvailability.dispatch();
|
||||
const call = active.client.execute(name, arguments_, options);
|
||||
const result = unrecordedExecution(await call);
|
||||
const profiled = bindExecutionTarget(
|
||||
@@ -1442,7 +1441,15 @@ export class BinarySession
|
||||
this.#active === active
|
||||
)
|
||||
this.#active.nativeSourceAdmitted = true;
|
||||
this.#observeRuntimeAvailability(name, execution);
|
||||
if (
|
||||
this.#runtimeAvailability.observe(
|
||||
dispatch,
|
||||
capability,
|
||||
active.route.capabilities.values(),
|
||||
execution,
|
||||
)
|
||||
)
|
||||
this.#emitAvailabilityChanged();
|
||||
// Source facts belong to the observation, even when this query cannot be
|
||||
// replayed (for example a Ghidra procedure read without a document selector).
|
||||
if (execution.ok && execution.value.sourceEvidence !== undefined) {
|
||||
@@ -1486,56 +1493,8 @@ export class BinarySession
|
||||
return execution;
|
||||
}
|
||||
|
||||
#observeRuntimeAvailability(
|
||||
operation: AnalysisOperation,
|
||||
result: Result<AnalysisExecution, AnalysisError>,
|
||||
): void {
|
||||
if (result.ok) {
|
||||
if (this.#markRuntimeAvailable(operation))
|
||||
this.#emitAvailabilityChanged();
|
||||
return;
|
||||
}
|
||||
if (result.error._tag === "AnalysisCapabilityUnavailableError") {
|
||||
if (this.#markRuntimeUnavailable(operation, result.error.message))
|
||||
this.#emitAvailabilityChanged();
|
||||
return;
|
||||
}
|
||||
if (
|
||||
[
|
||||
"ProviderAdapterError",
|
||||
"HopperProcessError",
|
||||
"HopperStartError",
|
||||
].includes(result.error._tag)
|
||||
) {
|
||||
const capabilities = this.#active?.route.capabilities;
|
||||
const providerId = capabilities?.get(operation)?.provider.id;
|
||||
let changed = false;
|
||||
for (const descriptor of capabilities?.values() ?? [])
|
||||
if (providerId !== undefined && descriptor.provider.id === providerId)
|
||||
changed =
|
||||
this.#markRuntimeUnavailable(
|
||||
descriptor.operation,
|
||||
`${result.error._tag}: ${result.error.message}`,
|
||||
) || changed;
|
||||
if (changed) this.#emitAvailabilityChanged();
|
||||
}
|
||||
}
|
||||
|
||||
#markRuntimeAvailable(operation: string): boolean {
|
||||
return this.#runtimeUnavailability.delete(operation);
|
||||
}
|
||||
|
||||
#markRuntimeUnavailable(operation: string, reason: string): boolean {
|
||||
const current = this.#runtimeUnavailability.get(operation);
|
||||
if (current?.reason === reason) return false;
|
||||
this.#runtimeUnavailability.set(operation, { reason });
|
||||
return true;
|
||||
}
|
||||
|
||||
#clearRuntimeAvailability(): void {
|
||||
if (this.#runtimeUnavailability.size === 0) return;
|
||||
this.#runtimeUnavailability.clear();
|
||||
this.#emitAvailabilityChanged();
|
||||
if (this.#runtimeAvailability.clear()) this.#emitAvailabilityChanged();
|
||||
}
|
||||
|
||||
#emitAvailabilityChanged(): void {
|
||||
|
||||
@@ -0,0 +1,169 @@
|
||||
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||
import type { Result } from "../../domain/result.js";
|
||||
import type { CapabilityDescriptor } from "../AnalysisProvider.js";
|
||||
|
||||
/** Failures of the provider adapter or process rather than of one request. */
|
||||
const PROVIDER_WIDE_FAILURES: ReadonlySet<string> = new Set([
|
||||
"ProviderAdapterError",
|
||||
"HopperProcessError",
|
||||
"HopperStartError",
|
||||
]);
|
||||
|
||||
interface Failure {
|
||||
readonly dispatch: number;
|
||||
readonly reason: string;
|
||||
}
|
||||
|
||||
interface ProviderFailure extends Failure {
|
||||
readonly operation: string;
|
||||
}
|
||||
|
||||
/** Newest failure and newest success, both by dispatch, for one owner. */
|
||||
interface Outcomes<F extends Failure> {
|
||||
failure: F | undefined;
|
||||
succeeded: number;
|
||||
}
|
||||
|
||||
type RoutedOperation = Pick<
|
||||
CapabilityDescriptor,
|
||||
"operation" | "provider" | "available"
|
||||
>;
|
||||
|
||||
/** Reason recorded for an operation's own refusal or provider-wide failure. */
|
||||
export const runtimeFailureReason = (
|
||||
failure: AnalysisError,
|
||||
): string | undefined => {
|
||||
if (failure._tag === "AnalysisCapabilityUnavailableError")
|
||||
return failure.message;
|
||||
return PROVIDER_WIDE_FAILURES.has(failure._tag)
|
||||
? `${failure._tag}: ${failure.message}`
|
||||
: undefined;
|
||||
};
|
||||
|
||||
/** Explain unavailability inferred for an operation no request failed for. */
|
||||
export const inferredUnavailabilityReason = (
|
||||
providerId: string,
|
||||
failure: Pick<ProviderFailure, "operation" | "reason">,
|
||||
): string =>
|
||||
`Inferred from the ${failure.operation} failure (${failure.reason}). Retry after repairing the provider; a later successful ${providerId} request clears this.`;
|
||||
|
||||
/**
|
||||
* Own the runtime unavailability overlaid on the active route's declared
|
||||
* capabilities.
|
||||
*
|
||||
* An operation is unavailable while its newest refusal or provider-wide
|
||||
* failure was dispatched after its newest success. A declared-available
|
||||
* operation is also unavailable, as inferred, while its provider's newest
|
||||
* provider-wide failure was dispatched after any success from that provider.
|
||||
* One success therefore returns inferred siblings to their declared
|
||||
* availability without proving each is healthy, and keeps failures observed
|
||||
* for other operations.
|
||||
*
|
||||
* Only the newest dispatch of each outcome matters, so the projection does not
|
||||
* depend on the order in which overlapping replies complete. Dispatch is the
|
||||
* latest order this session owns; it does not establish how a provider
|
||||
* interleaves overlapping requests.
|
||||
*/
|
||||
export class SessionRuntimeAvailability {
|
||||
readonly #operations = new Map<string, Outcomes<Failure>>();
|
||||
readonly #providers = new Map<string, Outcomes<ProviderFailure>>();
|
||||
#projected: ReadonlyMap<string, { readonly reason: string }> = new Map();
|
||||
#dispatched = 0;
|
||||
|
||||
/** Current runtime unavailability keyed by operation. */
|
||||
get entries(): ReadonlyMap<string, { readonly reason: string }> {
|
||||
return this.#projected;
|
||||
}
|
||||
|
||||
/** Order one request immediately before it reaches the provider client. */
|
||||
dispatch(): number {
|
||||
this.#dispatched += 1;
|
||||
return this.#dispatched;
|
||||
}
|
||||
|
||||
/**
|
||||
* Record one dispatched request's outcome against the route it ran on and
|
||||
* report whether visible availability changed.
|
||||
*/
|
||||
observe(
|
||||
dispatch: number,
|
||||
capability: RoutedOperation,
|
||||
route: Iterable<RoutedOperation>,
|
||||
result: Result<unknown, AnalysisError>,
|
||||
): boolean {
|
||||
const operation = outcomes(this.#operations, capability.operation);
|
||||
const provider = outcomes(this.#providers, capability.provider.id);
|
||||
if (result.ok) {
|
||||
operation.succeeded = Math.max(operation.succeeded, dispatch);
|
||||
provider.succeeded = Math.max(provider.succeeded, dispatch);
|
||||
// A success can only remove unavailability.
|
||||
if (this.#projected.size === 0) return false;
|
||||
} else {
|
||||
const reason = runtimeFailureReason(result.error);
|
||||
if (reason === undefined) return false;
|
||||
operation.failure = newer(operation.failure, { dispatch, reason });
|
||||
if (PROVIDER_WIDE_FAILURES.has(result.error._tag))
|
||||
provider.failure = newer(provider.failure, {
|
||||
dispatch,
|
||||
reason,
|
||||
operation: capability.operation,
|
||||
});
|
||||
}
|
||||
return this.#project(route);
|
||||
}
|
||||
|
||||
/** Forget runtime outcomes when the active provider client is replaced. */
|
||||
clear(): boolean {
|
||||
this.#operations.clear();
|
||||
this.#providers.clear();
|
||||
if (this.#projected.size === 0) return false;
|
||||
this.#projected = new Map();
|
||||
return true;
|
||||
}
|
||||
|
||||
#project(route: Iterable<RoutedOperation>): boolean {
|
||||
const projected = new Map<string, { readonly reason: string }>();
|
||||
for (const descriptor of route) {
|
||||
const reason = this.#unavailableReason(descriptor);
|
||||
if (reason !== undefined) projected.set(descriptor.operation, { reason });
|
||||
}
|
||||
const changed =
|
||||
projected.size !== this.#projected.size ||
|
||||
[...projected].some(
|
||||
([operation, { reason }]) =>
|
||||
this.#projected.get(operation)?.reason !== reason,
|
||||
);
|
||||
this.#projected = projected;
|
||||
return changed;
|
||||
}
|
||||
|
||||
#unavailableReason(descriptor: RoutedOperation): string | undefined {
|
||||
const operation = this.#operations.get(descriptor.operation);
|
||||
if (
|
||||
operation?.failure !== undefined &&
|
||||
operation.failure.dispatch > operation.succeeded
|
||||
)
|
||||
return operation.failure.reason;
|
||||
// A declared refusal already explains its operation and is never dispatched.
|
||||
if (!descriptor.available) return undefined;
|
||||
const provider = this.#providers.get(descriptor.provider.id);
|
||||
return provider?.failure !== undefined &&
|
||||
provider.failure.dispatch > provider.succeeded
|
||||
? inferredUnavailabilityReason(descriptor.provider.id, provider.failure)
|
||||
: undefined;
|
||||
}
|
||||
}
|
||||
|
||||
const outcomes = <F extends Failure>(
|
||||
owners: Map<string, Outcomes<F>>,
|
||||
key: string,
|
||||
): Outcomes<F> => {
|
||||
const existing = owners.get(key);
|
||||
if (existing !== undefined) return existing;
|
||||
const created: Outcomes<F> = { failure: undefined, succeeded: 0 };
|
||||
owners.set(key, created);
|
||||
return created;
|
||||
};
|
||||
|
||||
const newer = <F extends Failure>(current: F | undefined, next: F): F =>
|
||||
current !== undefined && current.dispatch > next.dispatch ? current : next;
|
||||
@@ -1,14 +1,33 @@
|
||||
import { expect, it } from "vitest";
|
||||
import { expect, it, onTestFinished } from "vitest";
|
||||
|
||||
import {
|
||||
createAnalysisExecution,
|
||||
type AnalysisExecution,
|
||||
type AnalysisOperation,
|
||||
type AnalysisProvider,
|
||||
type CapabilityDescriptor,
|
||||
type ProviderIdentity,
|
||||
} from "../../../src/application/AnalysisProvider.js";
|
||||
import { toolContract } from "../../../src/contracts/toolContracts.js";
|
||||
import { expandToolAvailability } from "../../../src/contracts/toolOutputSchemaPrimitives.js";
|
||||
import { AnalysisTimeoutError } from "../../../src/domain/analysisErrorCore.js";
|
||||
import type { AnalysisError } from "../../../src/domain/analysisErrorBase.js";
|
||||
import {
|
||||
AnalysisCancelledError,
|
||||
AnalysisCapabilityUnavailableError,
|
||||
AnalysisTimeoutError,
|
||||
} from "../../../src/domain/analysisErrorCore.js";
|
||||
import { createAnalysisProfile } from "../../../src/domain/analysisProfile.js";
|
||||
import {
|
||||
inferredUnavailabilityReason,
|
||||
runtimeFailureReason,
|
||||
} from "../../../src/application/binary/SessionRuntimeAvailability.js";
|
||||
import { ProviderAdapterError } from "../../../src/domain/providerAdapterError.js";
|
||||
import { ProviderCleanupError } from "../../../src/domain/providerCleanupError.js";
|
||||
import { err, ok } from "../../../src/domain/result.js";
|
||||
import { err, ok, type Result } from "../../../src/domain/result.js";
|
||||
import { silentLogger } from "../../../src/logger.js";
|
||||
import { observed } from "../../fixtures/analysisExecution.js";
|
||||
import { createApplicationMcpHarness } from "../../fixtures/applicationMcpHarness.js";
|
||||
import { awaitAcknowledgement } from "../../fixtures/lifecycleAcknowledgement.js";
|
||||
import { parseMcpToolError } from "../../fixtures/mcpToolError.js";
|
||||
import {
|
||||
createBinarySessionTargets,
|
||||
@@ -309,3 +328,529 @@ it("keeps provider tools available after one request reaches its deadline", asyn
|
||||
await close();
|
||||
}
|
||||
});
|
||||
|
||||
type Reply = Result<AnalysisExecution, AnalysisError>;
|
||||
type Responder = (operation: string) => Promise<Reply>;
|
||||
|
||||
const FIXTURE = {
|
||||
id: "fixture",
|
||||
name: "Fixture analysis provider",
|
||||
version: "1",
|
||||
} as const satisfies ProviderIdentity;
|
||||
const AUXILIARY = {
|
||||
id: "fixture-auxiliary",
|
||||
name: "Auxiliary fixture provider",
|
||||
version: "1",
|
||||
} as const satisfies ProviderIdentity;
|
||||
|
||||
const liveCapability = (
|
||||
provider: ProviderIdentity,
|
||||
operation: Exclude<AnalysisOperation, "health">,
|
||||
declaredRefusal?: string,
|
||||
): CapabilityDescriptor => ({
|
||||
provider,
|
||||
operation,
|
||||
...(declaredRefusal === undefined
|
||||
? { available: true as const, reason: null }
|
||||
: { available: false as const, reason: declaredRefusal }),
|
||||
cachePolicy: "live",
|
||||
effects: {
|
||||
mutatesArtifact: false,
|
||||
launchesProcess: false,
|
||||
mayShowUi: false,
|
||||
mayAccessNetwork: false,
|
||||
mayWriteFilesystem: false,
|
||||
changesPermissions: false,
|
||||
requiresRoot: false,
|
||||
},
|
||||
limitations: [],
|
||||
});
|
||||
|
||||
const adapterFailure = (provider: ProviderIdentity, operation: string) =>
|
||||
Promise.resolve(err(new ProviderAdapterError(provider.id, operation)));
|
||||
const recordedReason = (failure: AnalysisError): string => {
|
||||
const reason = runtimeFailureReason(failure);
|
||||
if (reason === undefined) throw new Error(`${failure._tag} is not recorded`);
|
||||
return reason;
|
||||
};
|
||||
// Compare remediation with its owning formatters; the facts are their inputs.
|
||||
const observedFailure = (operation: string, provider: string = FIXTURE.id) =>
|
||||
recordedReason(new ProviderAdapterError(provider, operation));
|
||||
const inferredFailure = (operation: string, provider: string = FIXTURE.id) =>
|
||||
inferredUnavailabilityReason(provider, {
|
||||
operation,
|
||||
reason: observedFailure(operation, provider),
|
||||
});
|
||||
const refusal = (operation: string, reason: string) =>
|
||||
recordedReason(
|
||||
new AnalysisCapabilityUnavailableError(FIXTURE.id, operation, reason),
|
||||
);
|
||||
|
||||
/** Hold provider replies; release and join every launched call. */
|
||||
const heldCalls = (
|
||||
client: Awaited<ReturnType<typeof createApplicationMcpHarness>>["client"],
|
||||
) => {
|
||||
const held = new Map<
|
||||
string,
|
||||
{ readonly reply: Promise<Reply>; readonly received: () => void }
|
||||
>();
|
||||
const launched: {
|
||||
readonly release: (reply: Reply) => void;
|
||||
readonly response: Promise<unknown>;
|
||||
}[] = [];
|
||||
return {
|
||||
/** Serve a held reply when one is registered for this operation. */
|
||||
respond: (operation: string): Promise<Reply> | undefined => {
|
||||
const entry = held.get(operation);
|
||||
if (entry === undefined) return undefined;
|
||||
held.delete(operation);
|
||||
entry.received();
|
||||
return entry.reply;
|
||||
},
|
||||
dispatch: async (name: string, arguments_: Record<string, unknown>) => {
|
||||
const reply = createDeferred<Reply>();
|
||||
const received = createDeferred<void>();
|
||||
held.set(name, { reply: reply.promise, received: received.resolve });
|
||||
const response = client.callTool({ name, arguments: arguments_ });
|
||||
launched.push({ release: reply.resolve, response });
|
||||
await awaitAcknowledgement(received.promise, response, name);
|
||||
return {
|
||||
release: reply.resolve,
|
||||
isError: async () => (await response).isError === true,
|
||||
};
|
||||
},
|
||||
/** Release every held reply and wait for its MCP response. */
|
||||
join: async () => {
|
||||
for (const { release } of launched) release(observed("released"));
|
||||
await Promise.allSettled(launched.map(({ response }) => response));
|
||||
},
|
||||
};
|
||||
};
|
||||
|
||||
/** Connect a deep fixture and an auxiliary provider with scripted replies. */
|
||||
const connectScripted = async (options: {
|
||||
readonly operations: readonly Exclude<AnalysisOperation, "health">[];
|
||||
readonly auxiliaryOperations?: readonly Exclude<
|
||||
AnalysisOperation,
|
||||
"health"
|
||||
>[];
|
||||
readonly respond: { current: Responder };
|
||||
readonly respondAuxiliary?: { current: Responder };
|
||||
/** Operations the fixture declares unavailable, with their declared reason. */
|
||||
readonly declaredRefusals?: Readonly<Record<string, string>>;
|
||||
readonly families?: string;
|
||||
}) => {
|
||||
const [path] = await createBinarySessionTargets();
|
||||
const provider: AnalysisProvider = {
|
||||
identity: () => FIXTURE,
|
||||
resolveAnalysisProfile: () =>
|
||||
Promise.resolve(
|
||||
ok({ profile: createAnalysisProfile(FIXTURE, { fixture: true }) }),
|
||||
),
|
||||
capabilities: () =>
|
||||
options.operations.map((operation) =>
|
||||
liveCapability(
|
||||
FIXTURE,
|
||||
operation,
|
||||
options.declaredRefusals?.[operation],
|
||||
),
|
||||
),
|
||||
createClient: () => ({
|
||||
execute: (operation) =>
|
||||
operation === "health"
|
||||
? Promise.resolve(observed(null))
|
||||
: options.respond.current(operation),
|
||||
close: () => Promise.resolve(ok(null)),
|
||||
}),
|
||||
};
|
||||
const auxiliary: AnalysisProvider = {
|
||||
identity: () => AUXILIARY,
|
||||
capabilities: () =>
|
||||
(options.auxiliaryOperations ?? []).map((operation) =>
|
||||
liveCapability(AUXILIARY, operation),
|
||||
),
|
||||
createClient: () => ({
|
||||
execute: (operation) =>
|
||||
options.respondAuxiliary?.current(operation) ??
|
||||
Promise.resolve(ok(createAnalysisExecution(null, AUXILIARY))),
|
||||
close: () => Promise.resolve(ok(null)),
|
||||
}),
|
||||
};
|
||||
const session = createTestBinarySession(provider, {
|
||||
auxiliaryProviders: [auxiliary],
|
||||
});
|
||||
const { client, close } = await createApplicationMcpHarness({
|
||||
session,
|
||||
clientIdentity: { name: "provider-recovery", version: "1" },
|
||||
serverOptions: {
|
||||
environment:
|
||||
options.families === undefined
|
||||
? {}
|
||||
: { REA_MCP_TOOL_FAMILIES: options.families },
|
||||
logger: silentLogger,
|
||||
},
|
||||
});
|
||||
onTestFinished(close);
|
||||
expect(
|
||||
(await client.callTool({ name: "open_binary", arguments: { path } }))
|
||||
.isError,
|
||||
).not.toBe(true);
|
||||
const call = async (name: string, arguments_: Record<string, unknown>) =>
|
||||
(await client.callTool({ name, arguments: arguments_ })).isError === true
|
||||
? "error"
|
||||
: "ok";
|
||||
const tools = async () => {
|
||||
const reply = await client.callTool({
|
||||
name: "binary_session",
|
||||
arguments: {},
|
||||
});
|
||||
return expandToolAvailability(
|
||||
toolContract("binary_session").outputSchema.parse(reply.structuredContent)
|
||||
.result.tool_availability,
|
||||
);
|
||||
};
|
||||
const availability = async (names: readonly string[]) => {
|
||||
const advertised = await tools();
|
||||
return Object.fromEntries(
|
||||
names.map((name) => {
|
||||
const tool = advertised.find((entry) => entry.name === name);
|
||||
if (tool === undefined) throw new Error(`${name} is not advertised`);
|
||||
return [name, tool.available ? "available" : tool.remediation];
|
||||
}),
|
||||
);
|
||||
};
|
||||
return { client, call, availability, tools };
|
||||
};
|
||||
|
||||
const DIRECT_OPERATIONS = [
|
||||
"address_name",
|
||||
"current_address",
|
||||
"list_procedures",
|
||||
"procedure_assembly",
|
||||
"procedure_info",
|
||||
] as const;
|
||||
const ADDRESS = { address: "0x1000", document: "fixture" };
|
||||
const PROCEDURE = { procedure: "main" };
|
||||
|
||||
it("returns provider-inferred siblings to declared availability after one success", async () => {
|
||||
let broken = false;
|
||||
const respond = {
|
||||
current: (operation: string): Promise<Reply> => {
|
||||
if (operation === "procedure_info")
|
||||
return Promise.resolve(
|
||||
err(
|
||||
new AnalysisCapabilityUnavailableError(
|
||||
FIXTURE.id,
|
||||
operation,
|
||||
"fixture project is read-only",
|
||||
),
|
||||
),
|
||||
);
|
||||
if (broken) return adapterFailure(FIXTURE, operation);
|
||||
return Promise.resolve(observed("fixture-text"));
|
||||
},
|
||||
};
|
||||
const declared = "Fixture project was opened read-only.";
|
||||
const { call, availability } = await connectScripted({
|
||||
operations: [...DIRECT_OPERATIONS, "set_comment"],
|
||||
auxiliaryOperations: ["list_documents"],
|
||||
declaredRefusals: { set_comment: declared },
|
||||
respond,
|
||||
});
|
||||
const refused = refusal("procedure_info", "fixture project is read-only");
|
||||
expect(await call("procedure_info", PROCEDURE)).toBe("error");
|
||||
broken = true;
|
||||
expect(await call("address_name", ADDRESS)).toBe("error");
|
||||
expect(await call("procedure_assembly", PROCEDURE)).toBe("error");
|
||||
const names = [...DIRECT_OPERATIONS, "set_comment", "list_documents"];
|
||||
expect(await availability(names)).toEqual({
|
||||
address_name: observedFailure("address_name"),
|
||||
current_address: inferredFailure("procedure_assembly"),
|
||||
list_procedures: inferredFailure("procedure_assembly"),
|
||||
procedure_assembly: observedFailure("procedure_assembly"),
|
||||
// Separately observed and declared refusals keep their own reasons.
|
||||
procedure_info: refused,
|
||||
set_comment: declared,
|
||||
list_documents: "available",
|
||||
});
|
||||
|
||||
broken = false;
|
||||
expect(await call("address_name", ADDRESS)).toBe("ok");
|
||||
// One success is evidence the provider answers again, not that every
|
||||
// operation is healthy: failures observed on other operations remain.
|
||||
expect(await availability(names)).toEqual({
|
||||
address_name: "available",
|
||||
current_address: "available",
|
||||
list_procedures: "available",
|
||||
procedure_assembly: observedFailure("procedure_assembly"),
|
||||
procedure_info: refused,
|
||||
set_comment: declared,
|
||||
list_documents: "available",
|
||||
});
|
||||
expect(await call("procedure_assembly", PROCEDURE)).toBe("ok");
|
||||
expect(await availability(["procedure_assembly", "procedure_info"])).toEqual({
|
||||
procedure_assembly: "available",
|
||||
procedure_info: refused,
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps each provider's failures and ignores cancelled requests", async () => {
|
||||
let fixtureBroken = false;
|
||||
let auxiliaryBroken = true;
|
||||
const respond = {
|
||||
current: (operation: string): Promise<Reply> => {
|
||||
if (operation === "list_procedures")
|
||||
return Promise.resolve(err(new AnalysisCancelledError(operation)));
|
||||
return fixtureBroken
|
||||
? adapterFailure(FIXTURE, operation)
|
||||
: Promise.resolve(observed("fixture-text"));
|
||||
},
|
||||
};
|
||||
const respondAuxiliary = {
|
||||
current: (operation: string): Promise<Reply> =>
|
||||
auxiliaryBroken
|
||||
? adapterFailure(AUXILIARY, operation)
|
||||
: Promise.resolve(ok(createAnalysisExecution("doc", AUXILIARY))),
|
||||
};
|
||||
const { call, availability } = await connectScripted({
|
||||
operations: ["address_name", "current_address", "list_procedures"],
|
||||
auxiliaryOperations: ["current_document", "list_documents"],
|
||||
respond,
|
||||
respondAuxiliary,
|
||||
});
|
||||
const names = [
|
||||
"address_name",
|
||||
"current_address",
|
||||
"list_procedures",
|
||||
"current_document",
|
||||
"list_documents",
|
||||
];
|
||||
expect(await call("list_documents", {})).toBe("error");
|
||||
expect(await call("address_name", ADDRESS)).toBe("ok");
|
||||
const auxiliaryFailure = {
|
||||
current_document: inferredFailure("list_documents", AUXILIARY.id),
|
||||
list_documents: observedFailure("list_documents", AUXILIARY.id),
|
||||
};
|
||||
// Another provider's success does not clear the auxiliary provider's state.
|
||||
expect(await availability(names)).toEqual({
|
||||
address_name: "available",
|
||||
current_address: "available",
|
||||
list_procedures: "available",
|
||||
...auxiliaryFailure,
|
||||
});
|
||||
|
||||
fixtureBroken = true;
|
||||
expect(await call("address_name", ADDRESS)).toBe("error");
|
||||
// A cancelled request is not evidence that the provider recovered.
|
||||
expect(await call("list_procedures", {})).toBe("error");
|
||||
const fixtureFailure = {
|
||||
address_name: observedFailure("address_name"),
|
||||
current_address: inferredFailure("address_name"),
|
||||
list_procedures: inferredFailure("address_name"),
|
||||
};
|
||||
expect(await availability(names)).toEqual({
|
||||
...fixtureFailure,
|
||||
...auxiliaryFailure,
|
||||
});
|
||||
|
||||
auxiliaryBroken = false;
|
||||
expect(await call("current_document", {})).toBe("ok");
|
||||
expect(await availability(names)).toEqual({
|
||||
...fixtureFailure,
|
||||
current_document: "available",
|
||||
list_documents: auxiliaryFailure.list_documents,
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps newer failures when older successes complete after them", async () => {
|
||||
let reply: Responder = () => Promise.resolve(observed("text"));
|
||||
const respond = { current: (operation: string) => reply(operation) };
|
||||
const { client, call, availability } = await connectScripted({
|
||||
operations: [...DIRECT_OPERATIONS, "current_procedure"],
|
||||
respond,
|
||||
});
|
||||
const calls = heldCalls(client);
|
||||
respond.current = (operation) => calls.respond(operation) ?? reply(operation);
|
||||
const document = { document: "doc" };
|
||||
const refuse: Responder = (operation) =>
|
||||
Promise.resolve(
|
||||
err(
|
||||
new AnalysisCapabilityUnavailableError(
|
||||
FIXTURE.id,
|
||||
operation,
|
||||
"fixture refusal",
|
||||
),
|
||||
),
|
||||
);
|
||||
try {
|
||||
// Successes dispatched before an own refusal, before a provider failure,
|
||||
// or between the two complete after both.
|
||||
const olderAddress = await calls.dispatch("address_name", ADDRESS);
|
||||
const olderCurrent = await calls.dispatch("current_address", document);
|
||||
reply = refuse;
|
||||
expect(await call("address_name", ADDRESS)).toBe("error");
|
||||
expect(await call("current_procedure", document)).toBe("error");
|
||||
const between = await calls.dispatch("current_procedure", document);
|
||||
reply = (operation) => adapterFailure(FIXTURE, operation);
|
||||
expect(await call("procedure_assembly", PROCEDURE)).toBe("error");
|
||||
olderAddress.release(observed("old-name"));
|
||||
olderCurrent.release(observed("0x1000"));
|
||||
between.release(observed("main"));
|
||||
for (const settled of [olderAddress, olderCurrent, between])
|
||||
expect(await settled.isError()).toBe(false);
|
||||
expect(
|
||||
await availability([
|
||||
"address_name",
|
||||
"current_address",
|
||||
"current_procedure",
|
||||
"list_procedures",
|
||||
]),
|
||||
).toEqual({
|
||||
address_name: refusal("address_name", "fixture refusal"),
|
||||
current_address: inferredFailure("procedure_assembly"),
|
||||
// Its own later success superseded the refusal, not the provider failure.
|
||||
current_procedure: inferredFailure("procedure_assembly"),
|
||||
list_procedures: inferredFailure("procedure_assembly"),
|
||||
});
|
||||
|
||||
// A success dispatched behind a failure executed after it, even though
|
||||
// the failure was not yet observed when the success was dispatched.
|
||||
reply = () => Promise.resolve(observed("text"));
|
||||
const failure = await calls.dispatch("procedure_assembly", PROCEDURE);
|
||||
const queued = await calls.dispatch("current_address", document);
|
||||
failure.release(
|
||||
err(new ProviderAdapterError(FIXTURE.id, "procedure_assembly")),
|
||||
);
|
||||
expect(await failure.isError()).toBe(true);
|
||||
queued.release(observed("0x1000"));
|
||||
expect(await queued.isError()).toBe(false);
|
||||
expect(
|
||||
await availability([
|
||||
"address_name",
|
||||
"current_address",
|
||||
"current_procedure",
|
||||
"list_procedures",
|
||||
"procedure_assembly",
|
||||
]),
|
||||
).toEqual({
|
||||
address_name: refusal("address_name", "fixture refusal"),
|
||||
current_address: "available",
|
||||
current_procedure: "available",
|
||||
list_procedures: "available",
|
||||
procedure_assembly: observedFailure("procedure_assembly"),
|
||||
});
|
||||
} finally {
|
||||
await calls.join();
|
||||
}
|
||||
});
|
||||
|
||||
it("keeps newer successes when an older failure completes last", async () => {
|
||||
const respond = {
|
||||
current: (_operation: string): Promise<Reply> =>
|
||||
Promise.resolve(observed("text")),
|
||||
};
|
||||
const { client, call, availability } = await connectScripted({
|
||||
operations: DIRECT_OPERATIONS,
|
||||
respond,
|
||||
});
|
||||
const calls = heldCalls(client);
|
||||
respond.current = (operation) =>
|
||||
calls.respond(operation) ?? Promise.resolve(observed("text"));
|
||||
const names = ["address_name", "current_address", "list_procedures"];
|
||||
try {
|
||||
// A sibling dispatched later already proved the provider answers.
|
||||
const older = await calls.dispatch("address_name", ADDRESS);
|
||||
const newer = await calls.dispatch("current_address", { document: "doc" });
|
||||
newer.release(observed("0x1000"));
|
||||
expect(await newer.isError()).toBe(false);
|
||||
older.release(err(new ProviderAdapterError(FIXTURE.id, "address_name")));
|
||||
expect(await older.isError()).toBe(true);
|
||||
expect(await availability(names)).toEqual({
|
||||
address_name: observedFailure("address_name"),
|
||||
current_address: "available",
|
||||
list_procedures: "available",
|
||||
});
|
||||
|
||||
// A later success of the same operation supersedes its older failure.
|
||||
const stale = await calls.dispatch("procedure_assembly", PROCEDURE);
|
||||
expect(await call("procedure_assembly", PROCEDURE)).toBe("ok");
|
||||
stale.release(
|
||||
err(new ProviderAdapterError(FIXTURE.id, "procedure_assembly")),
|
||||
);
|
||||
expect(await stale.isError()).toBe(true);
|
||||
expect(await availability([...names, "procedure_assembly"])).toEqual({
|
||||
address_name: observedFailure("address_name"),
|
||||
current_address: "available",
|
||||
list_procedures: "available",
|
||||
procedure_assembly: "available",
|
||||
});
|
||||
} finally {
|
||||
await calls.join();
|
||||
}
|
||||
});
|
||||
|
||||
it("restores composed tools in a scoped catalog after one composed success", async () => {
|
||||
let broken = true;
|
||||
const respond = {
|
||||
current: (operation: string): Promise<Reply> => {
|
||||
if (broken) return adapterFailure(FIXTURE, operation);
|
||||
if (operation === "procedure_address")
|
||||
return Promise.resolve(observed("0x1"));
|
||||
if (operation === "address_name")
|
||||
return Promise.resolve(observed("fixture_name"));
|
||||
return Promise.resolve(observed("return 0;"));
|
||||
},
|
||||
};
|
||||
const { client, availability, tools } = await connectScripted({
|
||||
operations: [
|
||||
"address_name",
|
||||
"procedure_address",
|
||||
"procedure_callees",
|
||||
"procedure_callers",
|
||||
"procedure_pseudo_code",
|
||||
"search_procedures",
|
||||
"search_strings",
|
||||
"xrefs",
|
||||
],
|
||||
respond,
|
||||
families: "session,enhanced",
|
||||
});
|
||||
// Direct prerequisites are outside this scope; only composed tools can retry.
|
||||
const batchItemStatus = async () => {
|
||||
const reply = await client.callTool({
|
||||
name: "batch_decompile",
|
||||
arguments: { addresses: ["0x1"] },
|
||||
});
|
||||
expect(reply.isError).not.toBe(true);
|
||||
return reply.structuredContent;
|
||||
};
|
||||
expect(await batchItemStatus()).toMatchObject({
|
||||
normalized_result: { items: [{ status: "error" }] },
|
||||
});
|
||||
expect(await availability(["trace_feature", "get_call_graph"])).toEqual({
|
||||
// The batch still decompiles the unresolved selector; its failure is newest.
|
||||
trace_feature: inferredFailure("procedure_pseudo_code"),
|
||||
get_call_graph: observedFailure("procedure_address"),
|
||||
});
|
||||
|
||||
broken = false;
|
||||
expect(await batchItemStatus()).toMatchObject({
|
||||
normalized_result: { items: [{ status: "ok", pseudocode: "return 0;" }] },
|
||||
});
|
||||
expect(
|
||||
await availability(["batch_decompile", "trace_feature", "get_call_graph"]),
|
||||
).toEqual({
|
||||
batch_decompile: "available",
|
||||
trace_feature: "available",
|
||||
get_call_graph: "available",
|
||||
});
|
||||
expect(
|
||||
(await tools()).find(({ name }) => name === "get_call_graph"),
|
||||
).toMatchObject({
|
||||
default_mode_available: true,
|
||||
modes: [
|
||||
{ name: "forward", available: true, missing_operations: [] },
|
||||
{ name: "backward", available: true, missing_operations: [] },
|
||||
],
|
||||
});
|
||||
});
|
||||
Reference in new issue
Block a user