fix(environment): bound state at redacted writes; infer identity only from items

A scenario write whose URL was redacted has no request line, so it was
dropped and reads after it counted as initial state. Such a successful
write now joins the timeline with an empty path, whose root relates to
every type, so it ends every type's state; a limitation counts them.

A list whose recordings were all empty passed the "every listed item
carries it" check vacuously and published an identity field nothing
showed; it now needs at least one item, else the view stays recorded
as no-identity.

A value that differed between two request lines of one item tool, such
as a query variant, was called volatile, and the gate then substituted
each recording's own value; only repeats of one request line now show
volatility, so differing lines go through fields-differ.

List membership is compared only while a list can still be the whole
collection, and merged entities are indexed by type and collection once,
so paged captures no longer cost pages times entities (8,000 pages of
40,000 entities: 5.7 s to 0.23 s).

Docs and the tool contract say scenario state keeps the capture's
redaction markers and how redacted writes bound it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

docs/specs/agent-environments/state-view/spec.md

@@ -24,15 +24,16 @@ later increments.
 - Order primary-origin records by recorded start time, or by record order when
   any start time is missing, and say which. Bound each type's state by the first
   write under its root (API base, version segments, one segment); a query
-  `POST` is not a write, and a read overlapping the write is left out.
+  `POST` is not a write, a read overlapping the write is left out, and a write
+  whose URL was redacted bounds every type, with a limitation counting it.
 - Prove an item type's identity field from the requested path value its
   responses repeat; otherwise infer it from a fixed list of identity names that
-  every listed item carries distinctly. Never treat an envelope that is itself
-  an entity as a list.
+  every listed item carries distinctly, which needs at least one listed item.
+  Never treat an envelope that is itself an entity as a list.
 - Merge observations into one entity per collection and identity: item views'
   latest values win over list views', differing views keep their own values,
-  and a value that changed with no related write is volatile and keeps the
-  latest.
+  and a value that changed between repeats of one request line with no related
+  write is volatile and keeps the latest.
 - Keep an item or list plan only when it reproduces each request line's latest
   initial recording from the state, comparing JSON values without regard to
   member order; otherwise record the tool with a reason. Parse bodies only when

docs/web-network-captures.md

@@ -455,7 +455,11 @@ path has a segment such as `search`, `query`, `list`, `find`, `fuzzy`,
 successful reads that finished before the first such write under the same root:
 the path's API base (the literal segments every tool sharing its first segment
 has), any `v1`-style version segments, then one more segment. A read that
-overlaps that write in time is left out.
+overlaps that write in time is left out. A write whose URL a scenario redacted
+has an unknown target, so it ends the state of every type, and a limitation
+counts such writes. Values from a `capture_browser_scenario` parent keep the
+capture's redaction markers in place of declared secrets; the state is not
+refused for them, as the manifest is not.
 
 Each `GET` tool whose template ends in a parameter is an item tool, and its
 type's `collection_template` is the template without that parameter; any other

src/contracts/analysisViewToolContracts.ts

@@ -17,7 +17,7 @@ export const ANALYSIS_VIEW_TOOL_CONTRACTS = [
     ...toolContractMetadata("inspect_analysis_view"),
     kind: "application",
     description:
-      'Inspect selected views of completed binary-layout, JavaScript application, native analyze_function, inspect_artifact, inventory_artifact, inspect_wasm_artifact, inspect_macho, decode_interface_builder, inspect_evm_interface, inspect_managed_artifact, inspect_managed_members, inspect_android_package, inspect_dart_aot, inspect_il2cpp_metadata or inspect_web_network_capture Evidence without repeating analysis. Historical capture summaries return identity and counts without bodies or archive inventories. Capture views page retained records: view {kind: "capture", detail: "preview" | "complete", offset: 0, limit: 20}; preview selects recorded headline fields and binary descriptors, complete returns exact records with all sidecars. Offsets follow the selected parent record order, preserving original producer ordinals. Paging never reopens the source or reruns the decoder. For inspect_pcap_capture parents, kind pcap pages endpoints, dns-queries, tls-sni, protocols, http-uris or unattributed-addresses. Select an exact query for dns-addresses, or a tshark/capinfos stream for UTF-16 diagnostics text. Page totals count retained facts; source coverage remains explicit. Qlog summaries keep profile/scope and counts; kind qlog pages traces, streams, loss-declarations, incomplete-records, contributions, ranges or final-sizes. Except traces, select trace_index; stream-specific facets also require exact stream_id and direction. Facts retain all-recorded-events-before-selection scope, and interval overlap or endpoint loss declarations do not prove retransmission or network loss. The api-surface view groups retained HAR requests by method, origin and format-rule path template; templated segments are inferences. It reads no headers or bodies and reports no query values. The environment-manifest view projects the same HAR requests on the primary origin (most JSON-like records) into a replayable tool surface: view {kind: "environment-manifest", template_rules: ["after-uid-literal", "opaque-token", "sibling-variance"], include_query_parameter_names: true}. Each tool has a deterministic name, a path template and parameters from the four format rules plus the selected opt-in rules, observed query parameter names, a value-free body shape for POST/PUT/PATCH, status and content-type sets and record ordinals; records are pointers into the parent, excluded groups carry a reason and three fixed unknowns are always stated. Templates, parameters and names are inferences; the view copies no header, cookie or body values and refuses parents inspected with sensitive values. A capture_browser_scenario parent that selected response bodies or header values projects the same way, one record per network transaction in request order; it relies on capture-time redaction of declared secrets instead of that refusal, and a transaction whose URL was redacted is left unrepresented. The environment-state view extracts, from the same parents, the initial state the recorded reads observed: view {kind: "environment-state", template_rules: [...]} with the rules of the manifest it accompanies. It reads response bodies recorded before the first related write and reports entity types with an identity field (proved by a path parameter or inferred from a field name), entities in collection order with per-view values where views differ, documents for written non-entity paths, and a render plan per read tool: item and list views reproduce their latest recordings from the state, and every other read tool is a recorded view with a reason. Types and identity are inferences; values are recorded data, so it refuses parents inspected with sensitive values, and a scenario parent must have selected response bodies. For a retained record use source: {kind: "retained-evidence", evidence_id: "<parent_evidence_id>"}. Use normalized_result.parent_evidence_id from a summary receipt. Source kind is required. Portable source uses {kind: "inline", evidence: <complete Evidence>}. For layout, JavaScript, inventory, WASM, Mach-O, Interface Builder, EVM, managed artifact, Android package and Dart AOT sources, request a summary with view: {kind: "summary"}; summary is a view kind and has no facet. The WASM summary gives validation, counts, WAT digest/length, tool profile and execution/network states without reading WAT text. Select a layout mitigations/linkage facet, one section/symbol/module, or a stable page. Module pages include assets, bundled and source modules; use node_id for ambiguous or unavailable paths. Typed module references use view {kind: "javascript", collection: "modules", projection: "references", where: {path: "main.js", kind: "javascript-module"}, offset: 0, limit: 64}; where matches exact retained strings (null selects unavailable paths). Rows include canonical identity, retained source-range count and observation support states. Source coverage and selection remain explicit; exhausting matching rows does not prove runtime loading or complete analysis. For exaLine truncated
+      'Inspect selected views of completed binary-layout, JavaScript application, native analyze_function, inspect_artifact, inventory_artifact, inspect_wasm_artifact, inspect_macho, decode_interface_builder, inspect_evm_interface, inspect_managed_artifact, inspect_managed_members, inspect_android_package, inspect_dart_aot, inspect_il2cpp_metadata or inspect_web_network_capture Evidence without repeating analysis. Historical capture summaries return identity and counts without bodies or archive inventories. Capture views page retained records: view {kind: "capture", detail: "preview" | "complete", offset: 0, limit: 20}; preview selects recorded headline fields and binary descriptors, complete returns exact records with all sidecars. Offsets follow the selected parent record order, preserving original producer ordinals. Paging never reopens the source or reruns the decoder. For inspect_pcap_capture parents, kind pcap pages endpoints, dns-queries, tls-sni, protocols, http-uris or unattributed-addresses. Select an exact query for dns-addresses, or a tshark/capinfos stream for UTF-16 diagnostics text. Page totals count retained facts; source coverage remains explicit. Qlog summaries keep profile/scope and counts; kind qlog pages traces, streams, loss-declarations, incomplete-records, contributions, ranges or final-sizes. Except traces, select trace_index; stream-specific facets also require exact stream_id and direction. Facts retain all-recorded-events-before-selection scope, and interval overlap or endpoint loss declarations do not prove retransmission or network loss. The api-surface view groups retained HAR requests by method, origin and format-rule path template; templated segments are inferences. It reads no headers or bodies and reports no query values. The environment-manifest view projects the same HAR requests on the primary origin (most JSON-like records) into a replayable tool surface: view {kind: "environment-manifest", template_rules: ["after-uid-literal", "opaque-token", "sibling-variance"], include_query_parameter_names: true}. Each tool has a deterministic name, a path template and parameters from the four format rules plus the selected opt-in rules, observed query parameter names, a value-free body shape for POST/PUT/PATCH, status and content-type sets and record ordinals; records are pointers into the parent, excluded groups carry a reason and three fixed unknowns are always stated. Templates, parameters and names are inferences; the view copies no header, cookie or body values and refuses parents inspected with sensitive values. A capture_browser_scenario parent that selected response bodies or header values projects the same way, one record per network transaction in request order; it relies on capture-time redaction of declared secrets instead of that refusal, and a transaction whose URL was redacted is left unrepresented. The environment-state view extracts, from the same parents, the initial state the recorded reads observed: view {kind: "environment-state", template_rules: [...]} with the rules of the manifest it accompanies. It reads response bodies recorded before the first related write and reports entity types with an identity field (proved by a path parameter or inferred from a field name), entities in collection order with per-view values where views differ, documents for written non-entity paths, and a render plan per read tool: item and list views reproduce their latest recordings from the state, and every other read tool is a recorded view with a reason. Types and identity are inferences; values are recorded data, so it refuses parents inspected with sensitive values, and a scenario parent must have selected response bodies; its values keep the capture redaction markers, and a write whose URL it redacted ends the state of every type. For a retained record use source: {kind: "retained-evidence", evidence_id: "<parent_evidence_id>"}. Use normalized_result.parent_evidence_id from a summary receipt. Source kind is required. Portable source uses {kind: "inline", evidence: <complete Evidence>}. For layout, JavaScript, inventory, WASM, Mach-O, Interface Builder, EVM, managed artifact, Android package and Dart AOT sources, request a summary with view: {kind: "summary"}; summary is a view kind and has no facet. The WASM summary gives validation, counts, WAT digest/length, tool profile and execution/network states without reading WAT text. Select a layout mitigations/linkage facet, one section/symbol/module, or a stable page. Module pages include assets, bundled and source modules; use node_id for ambiguous or unavailable paths. Typed module references use view {kind: "javascript", collection: "modules", projection: "references", where: {path: "main.js", kind: "javascript-module"}, offset: 0, limit: 64}; where matches exact retained strings (null selects unavailable paths). Rows include canonical identity, retained source-range count and observation support states. Source coverage and sLine truncated
     inputSchema: inspectAnalysisViewInputSchema,
     outputSchema: evidenceResultOf(analysisViewResultSchema),
     examples: [

src/domain/analysisView/EnvironmentStateAccumulator.ts

@@ -13,10 +13,11 @@ import {
 } from "./EnvironmentManifestAccumulator.js";
 import { EnvironmentStateExtraction } from "./EnvironmentStateExtraction.js";
 import type { TemplateRule } from "./environmentManifestRules.js";
-import type {
-  ParsedBody,
-  StateRecordFacts,
-  StateTimeline,
+import {
+  isStateWrite,
+  type ParsedBody,
+  type StateRecordFacts,
+  type StateTimeline,
 } from "./environmentStateRules.js";
 import {
   base64PrefixChars,
@@ -52,13 +53,18 @@ const timingSchema = z.object({
   startedDateTime: z.string(),
   time: z.number().nonnegative(),
 });
+const redactedWriteSchema = z.object({
+  request: z.object({ method: z.string() }),
+  response: z.object({ status: z.number() }).optional(),
+});
 const contentSchema = z.object({
   response: z.object({ content: z.object({ text: z.string() }) }),
 });
 
 interface RecordEntry {
   readonly facts: StateRecordFacts;
-  readonly origin: string;
+  /** Null for a write whose URL was redacted: its target is unknown, so it bounds every type. */
+  readonly origin: string | null;
   readonly contentType: string;
   readonly timed: boolean;
   readonly record: WebNetworkCaptureRecord;
@@ -104,11 +110,20 @@ export class EnvironmentStateAccumulator extends EnvironmentManifestAccumulator
     try {
       const line = readRecordedRequestLine(record.reported, true, this.scratch);
       if (line === null) return this.overBudget();
-      if (line.kind === "excluded") return ok(undefined);
       const timing = timingSchema.safeParse(record.reported);
       const start = timing.success
         ? Date.parse(timing.data.startedDateTime)
         : Number.NaN;
+      const timed = timing.success && Number.isFinite(start);
+      if (line.kind === "excluded")
+        return line.reason === "url-redacted"
+          ? this.#redactedWrite(
+              record,
+              timing.success && timed
+                ? { start, end: start + timing.data.time }
+                : undefined,
+            )
+          : ok(undefined);
       // The entry keeps the line's strings and a reference to the already admitted record.
       if (
         !this.retain(
@@ -123,7 +138,6 @@ export class EnvironmentStateAccumulator extends EnvironmentManifestAccumulator
         )
       )
         return this.overBudget();
-      const timed = timing.success && Number.isFinite(start);
       this.#entries.set(record.ordinal, {
         facts: {
           ordinal: record.ordinal,
@@ -145,6 +159,36 @@ export class EnvironmentStateAccumulator extends EnvironmentManifestAccumulator
     }
   }
 
+  /** Keep a successful write whose URL was redacted, with an empty path whose root relates to every type. */
+  #redactedWrite(
+    record: WebNetworkCaptureRecord,
+    timing: { readonly start: number; readonly end: number } | undefined,
+  ): Result<void, AnalysisError> {
+    const parsed = redactedWriteSchema.safeParse(record.reported);
+    if (!parsed.success) return ok(undefined);
+    const status = parsed.data.response?.status ?? 0;
+    const facts: StateRecordFacts = {
+      ordinal: record.ordinal,
+      method: parsed.data.request.method,
+      path: "",
+      query: undefined,
+      status: status > 0 ? status : null,
+      start: timing?.start ?? record.ordinal,
+      end: timing?.end ?? record.ordinal,
+    };
+    if (!isStateWrite(facts)) return ok(undefined);
+    if (!this.retain(ENTRY_BYTES * 8 + textBytes(facts.method.length)))
+      return this.overBudget();
+    this.#entries.set(record.ordinal, {
+      facts,
+      origin: null,
+      contentType: "",
+      timed: timing !== undefined,
+      record,
+    });
+    return ok(undefined);
+  }
+
   /** Parse one retained response body as exact JSON, charging its decoding and tree as temporary work. */
   #body(ordinal: number): ParsedBody | null {
     const entry = this.#entries.get(ordinal);
@@ -175,8 +219,12 @@ export class EnvironmentStateAccumulator extends EnvironmentManifestAccumulator
     const manifest = this.finish();
     if (!manifest.ok) return manifest;
     const entries = [...this.#entries.values()].filter(
-      (entry) => entry.origin === manifest.value.origin,
+      (entry) =>
+        entry.origin === manifest.value.origin || entry.origin === null,
     );
+    const redactedWrites = entries.filter(
+      (entry) => entry.origin === null,
+    ).length;
     const timeline: StateTimeline =
       entries.length > 0 && entries.every((entry) => entry.timed)
         ? "recorded-start-time"
@@ -205,6 +253,11 @@ export class EnvironmentStateAccumulator extends EnvironmentManifestAccumulator
     }).run();
     if (state === null) return this.overBudget();
     const limitations = this.addLimitations([
+      ...(redactedWrites > 0
+        ? [
+            `${String(redactedWrites)} of the recorded writes had a redacted URL; each ends the initial state of every type, since its target is unknown.`,
+          ]
+        : []),
       ...(timeline === "record-order"
         ? [
             "Some records carry no start time, so the state's timeline is record order and overlapping requests are not detected.",

src/domain/analysisView/EnvironmentStateExtraction.ts

@@ -78,6 +78,8 @@ export class EnvironmentStateExtraction {
   readonly #entities = new Map<string, ObservedEntity>();
   readonly #volatile = new Map<string, Set<string>>();
   readonly #merged = new Map<string, StateEntity>();
+  /** Merged entities by type and collection, built once on first use after merging. */
+  #collections: Map<string, StateEntity[]> | undefined;
 
   constructor(input: {
     readonly tools: readonly StateToolFacts[];
@@ -280,6 +282,7 @@ export class EnvironmentStateExtraction {
     const proved = this.#identity.get(type)?.field;
     let pointer: string | undefined;
     let names: readonly string[] | undefined;
+    let sawItems = false;
     for (const record of recordings) {
       const found = this.#withBody(record.ordinal, (body) => {
         if (body === undefined) return null;
@@ -288,6 +291,7 @@ export class EnvironmentStateExtraction {
           located === undefined ? undefined : listItems(body, located);
         if (items === undefined) return undefined;
         pointer = located;
+        if (items.length > 0) sawItems = true;
         return items.length === 0
           ? (names ?? (proved === undefined ? IDENTITY_NAMES : [proved]))
           : (proved === undefined ? IDENTITY_NAMES : [proved]).filter((name) =>
@@ -301,7 +305,12 @@ export class EnvironmentStateExtraction {
       names = (names ?? found).filter((name) => found.includes(name));
     }
     const field = names?.[0];
-    if (field === undefined || pointer === undefined) {
+    // An identity name is only inferred from items that carry it; empty lists prove none.
+    if (
+      field === undefined ||
+      pointer === undefined ||
+      (proved === undefined && !sawItems)
+    ) {
       this.#recorded(tool, "no-identity");
       return;
     }
@@ -351,8 +360,12 @@ export class EnvironmentStateExtraction {
       };
       this.#entities.set(key, entity);
     }
+    // Only a repeated request line shows a value changing; other lines may differ by query.
+    const line = requestLine(record);
     const earlier = entity.observations.findLast(
-      (observation) => observation.tool === plan.tool.name,
+      (observation) =>
+        observation.tool === plan.tool.name &&
+        requestLine(this.#byOrdinal.get(observation.ordinal)!) === line,
     );
     for (const name of Object.keys(object))
       if (
@@ -400,9 +413,17 @@ export class EnvironmentStateExtraction {
   }
 
   #inCollection(type: string, collection: string): StateEntity[] {
-    return [...this.#merged.values()].filter(
-      (entity) => entity.type === type && entity.collection === collection,
-    );
+    if (this.#collections === undefined) {
+      this.#retain(ENTRY_BYTES * (this.#merged.size + 1));
+      this.#collections = new Map();
+      for (const entity of this.#merged.values()) {
+        const key = JSON.stringify([entity.type, entity.collection]);
+        const members = this.#collections.get(key);
+        if (members === undefined) this.#collections.set(key, [entity]);
+        else members.push(entity);
+      }
+    }
+    return this.#collections.get(JSON.stringify([type, collection])) ?? [];
   }
 
   /** A written path whose read is not an entity holds one document value: its latest init recording. */

src/domain/analysisView/environmentStateGate.ts

@@ -168,11 +168,14 @@ const listFacts = (
         body === undefined ? undefined : listItems(body, plan.pointer);
       if (body === undefined || items === undefined) return;
       items.forEach((item) => union.add(item));
-      const ids = items.map((item) => scalarText(item[field]) ?? "");
-      const expected = context
-        .inCollection(plan.type, collectionPathOf(record, false))
-        .map((entity) => entity.id);
-      if (record.query !== undefined || !same(ids, expected)) all = false;
+      // Once a list cannot be the whole collection, its membership is not compared again.
+      if (all && record.query === undefined) {
+        const ids = items.map((item) => scalarText(item[field]) ?? "");
+        const expected = context
+          .inCollection(plan.type, collectionPathOf(record, false))
+          .map((entity) => entity.id);
+        if (!same(ids, expected)) all = false;
+      } else all = false;
       const parent = plan.pointer === "" || !isObject(body) ? {} : body;
       const names = Object.keys(parent).filter(
         (name) => /total|count/iu.test(name) && parent[name] === items.length,

src/domain/analysisView/environmentStateView.test.ts

@@ -349,3 +349,47 @@ it("refuses when extraction would exceed the projection budget", () => {
     },
   });
 });
+
+it("ends every type's state at a write whose URL was redacted, and says so", () => {
+  const parent = captureOf([
+    ["GET", "/api/items/1", { id: 1, title: "Granite Ledger" }],
+    ["PUT", "/api/items/1", { id: 1 }],
+    ["GET", "/api/items/1", { id: 1, title: "Saltmarsh Depot" }],
+  ]);
+  const result = structuredClone(parent.normalizedResult) as {
+    records: { reported: { request: { url: string | null } } }[];
+  };
+  // A scenario keeps a redacted URL as null, so the write's target is unknown.
+  result.records[1]!.reported.request.url = null;
+  const state = stateOf({ ...parent, normalizedResult: result });
+  expect(state.types).toMatchObject([{ init_before_ordinal: 1 }]);
+  expect(state.entities).toMatchObject([
+    { fields: { title: "Granite Ledger" }, record_ordinals: [0] },
+  ]);
+  expect(state.limitations).toContain(
+    "1 of the recorded writes had a redacted URL; each ends the initial state of every type, since its target is unknown.",
+  );
+});
+
+it("infers no identity for a list whose recordings held no items", () => {
+  const state = stateOf(captureOf([["GET", "/api/widgets", []]]));
+  expect(state.types).toEqual([]);
+  expect(state.views).toEqual([
+    { tool: "get_widgets", shape: "recorded", reason: "no-identity" },
+  ]);
+});
+
+it("does not call a value volatile when it differs between request lines", () => {
+  const state = stateOf(
+    captureOf([
+      ["GET", "/api/items/1?lang=fr", { id: 1, name: "Bonjour" }],
+      ["GET", "/api/items/1", { id: 1, name: "Hello" }],
+    ]),
+  );
+  expect(state.types).toEqual([]);
+  expect(
+    state.views.map((view) =>
+      view.shape === "recorded" ? [view.tool, view.reason] : view.tool,
+    ),
+  ).toEqual([["get_items_by_id", "fields-differ"]]);
+});