Files
rea/src/domain/analysisView/EnvironmentStateWriteSimulation.ts
T
aa210b7ccc feat(environment): serve the writes that later reads explain (#7274)
* feat(environment): serve the writes callers make without recorded form fields

- A create member a recording sent empty ("", null, false, [] or {}) is no
  longer required; a call that leaves it out is applied as sending that value
  (body.blank).
- List views render envelope members that list the listed entities' ids
  (Paperless's `all`), so creates in such collections are confirmed.
- Updates move their entity in collections kept sorted by a field they write,
  and a failed whole-collection read after an update can teach that order.
- A later create that stores an unruled field otherwise is withheld at the
  create, and an item read may fill values a create's answer had not loaded.
- An unsent field respelled between null, [] and {} is unchanged.
- The skill asks for one create with every field and one with only what a
  person must give.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* docs(skill): show how to read signed-in pages before recording

Agents guessed capture keys (artifacts, dom, final) to see pages behind
sign-in, which inspect_web_page cannot reach.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(environment): accept every empty spelling recorded creates sent

A create that fills only what a person must give sends `null` where the
full create sent a string, and `null` or `false` for the same flag. The
merged plan now accepts each recorded empty value, and a call that leaves
the member out is applied as the earliest call sent it.

Also adds the spec folder for this change.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* docs(skill): keep the vary-fields step neutral on UI or API writes

Agents read "the form" as a cue to write through the UI, where some
applications' request bodies are not exposed; a body-unusable UI write
now re-records once with request steps.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* docs(environment): record verification for blank members and sorted writes

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(environment): address review of blank members and sorted writes

- Bulk updates move their entities in turn against the others' earlier
  values and install the final order once, as replay does.
- A member's recorded empty values stay accepted once a later call teaches
  its default; a left-out member with a default keeps the default.
- Write answers spell empty lists and maps as their recorded answers did.
- A rename can teach a sort when the creates chose one by a field the
  update did not write, such as ids that ascend with titles.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* feat(environment): serve the writes that later reads explain

- An item read whose object is marked deleted (deleted, isDeleted or
  is_deleted true) agrees with an absent entity, as EspoCRM answers a
  deleted record; such an object never seeds or adopts an entity.
- maxSize, max_results and maxResults are page sizes, for whole-collection
  list lines and the list query's limit role.
- An item view also shows the fields only reads after its type's first write
  show, when no initial entity holds them (Vikunja's task subscription),
  without requiring them; write answers leave such a field out for entities
  without it, and it no longer withholds the write as incomplete-entities.
- The skill asks updates to change the fields a person would change, and
  expects a deleted object to read back as 404 or marked deleted.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(environment): answer sent empty members as sent, and keep a placed create's order

- A write answer spells an empty list or map the call did not send as its
  recorded answer did; one the call sent is answered as sent, so a create
  sending `[]` is not answered `null` because the first recorded create was.
- A read listing the whole collection that placed a create leaves the create
  only the sorted inserts whose order that read follows, so a later rename
  cannot teach a sort an earlier read ruled out.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(environment): answer bulk updates without fields only later reads show

A bulk update's answer renders each entity with the same allowance a single
update's answer has: a field only reads after the type's first write showed,
which no initial entity holds, is left out for an entity without it, so a
bulk update of a seeded task is no longer refused for lacking it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* docs(environment): correct how the EspoCRM contact update task passes

The agent trajectories show gpt-oss passes it by creating a replacement contact
and deleting the original, since the served update refuses emailAddress; the
update itself is refused in both builds.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(environment): answer a read of a served-deleted entity marked deleted

An application that soft-deletes still answers a deleted record by id, marked
deleted. Serving now keeps an entity a served delete removed from a type whose
recorded item reads show that, until a reset or load and within the size
data/add loads, and answers a read of it with its last fields and the flag
true, instead of refusing the read.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(environment): keep unloaded create answers, check bulk answers with sent members, narrow inserts only on collection-order reads

- A create whose item read loaded values its answer had not records them
  (`observation.read.unloaded`), and a served create's answer keeps them as
  the recorded answer showed them, a zero-time owner included, while the state
  holds the read's.
- Replay checks a bulk update's recorded answers with each target's sent
  members, as serving answers them, so echoed `null` and `[]` no longer make
  the plan body-unusable.
- A whole-collection read whose query sorts it shows the query's order, so it
  no longer narrows a create's sorted insert candidates.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* docs(environment): record the census with main's unknown-clock rule merged

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(environment): learn a sort from a rename only when moving its entities reaches it

Serving moves only the entities a write changes in a collection kept sorted,
so replay now tries a sort after a rename only when the entities no write the
read exercised changed already follow it. Seeded folders listed out of title
order no longer teach a title sort that serving would answer with another
order.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(environment): keep unloaded create answers a sent value did not echo, and learn no sort from query-sorted reads

- A served create keeps a value its recorded answer had not loaded unless the
  call sent that member and the plan stores it as sent, so a create sending
  `active: false` is answered `false` as recorded while the state holds the
  read's `true`.
- A rename no longer teaches a sort from a whole-collection read its query
  sorts, whose ties need not follow the collection's own order.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(environment): learn the soft-delete flag in the state view and serve by it

Extraction now records the member later item reads set true on a record the
application soft-deleted, and the item view carries it as `deleted_flag`.
Serving reads that fact instead of scanning recorded bodies itself, so one
owner decides what a soft delete looks like.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(environment): learn the soft-delete flag from item reads with a query too

A client may read a deleted record back only with a query, such as a
cache-busting parameter. Every successful read of the item tool now shows the
flag; only plain later reads still add fields.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(environment): learn sorts only from orders a list read confirmed

- A sort retry after a failed whole-collection read requires the entities no
  write changed since a list last showed them to follow it already, rather than
  excusing every entity the update's tool ever wrote: an earlier rename a list
  confirmed in place stays where it was.
- An update takes its creates' sort only from a create a list read showed
  placed; replay still tries a sort a create guessed from the initial order.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(environment): drop entities read back as deleted and answer soft deletes as deleted

- A held entity a later read shows marked deleted is dropped, not adopted with
  the marked object's fields.
- Bulk-update replay renders answers with the optional and later-only fields
  serving uses, so a bulk update of created and seeded entities replays.
- A served delete of a soft-deleting type answers with the entity marked
  deleted, single and bulk.
- A deleted flag only query reads showed is a field of the item view, so a
  served tombstone keeps it where live records omit it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(environment): move updated entities only by placed sorts and keep query-sorted reads from confirming order

Replay moved an updated entity by a sort a create only guessed, while the
update plan exported only placed sorts, so a read could confirm an update
by a move serving would not make. Both now use the placed sort. A passing
list read clears its entities from the moved set only when the state
decides the order of every entity it lists, so a read whose query sorts
it no longer hides an earlier move from a later sort check.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(environment): try a learned sort by moving each pending write in turn, as serving does

#retrySort sorted the whole collection, which a list could match while
serving, moving each renamed entity in turn against the others' values
before their own writes, gave another order. The state now records the
values each write replaced, and the retry replays the writes since a
list in collection order last showed their entities.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(environment): answer a served delete with the deleted flag its recorded answer showed

An application may answer a delete with the record as it was and mark it
deleted only on later reads. Delete answers, single and bulk, now take
the flag's value from the recorded answer; reads of a served-deleted
entity still show it marked deleted.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* chore(environment): set aside two hunks next to lines main's saved-values change rewrites

Main's #7267 changes the lines beside the blank-member docs and the
empty-spelling check in explainUpdate. Setting them aside lets that merge
apply cleanly; the commit after it puts them back on top of main's text.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* chore(environment): restore the set-aside hunks on main and bump the skill to 51

Puts the blank-member docs and the empty-spelling check back beside main's
slug and composed text and heldSlugRule. Main is at skill 50 with
different text, so this branch's skill changes take 51.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* chore(environment): bump the skill to 52 above #7260's 51

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(environment): keep sorts a list showed in place ruled out, and created answers' nested members as recorded

A whole list in collection order that showed a renamed entity where a
sort by the renamed field would have moved it now rules that sort out, so
a later rename of the same entity cannot teach it. A create's answer
leaves out members its item read added inside an object the answer
showed, as the recorded answer did; the state still stores them.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(environment): compare soft-deleted read fields

* fix(environment): preserve initially deleted item observations

* fix(environment): index stored deleted entities separately

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
Co-authored-by: morluto <[email protected]>
2026-10-28 05:05:21 +08:00

1123 lines
42 KiB
TypeScript

import { CapturedWriteCompletion } from "./environmentStateWriteCompletion.js";
import type { JsonValue } from "../jsonValue.js";
import {
at,
changesState,
isObject,
isStateWrite,
listItems,
listsAll,
markedDeleted,
normalizedPath,
ownMember,
same,
scalarText,
type JsonObject,
type ParsedRequestBody,
type StateEntity,
type StateRecordFacts,
type StateToolFacts,
type StateType,
type StateView,
} from "./environmentStateRules.js";
import { UnseenEntities } from "./environmentStateWriteAdoption.js";
import { FailedReadBlame } from "./environmentStateWriteBlame.js";
import { addressedWrite } from "./environmentStateWriteAddress.js";
import { withoutMembers } from "./environmentStateResponseMembers.js";
import { linkedTarget } from "./environmentStateWriteLinks.js";
import {
laterItemFields,
updatedEntity,
} from "./environmentStateWriteApply.js";
import { projectedRead } from "./environmentStateWriteObserve.js";
import { OmittedFields } from "./environmentStateWriteOmitted.js";
import { SortedArrays } from "./environmentStateWriteSorted.js";
import { sameQuery } from "./environmentStateWriteStamps.js";
import {
checkDocumentRead,
checkItemRead,
checkListRead,
collectionTarget,
entityTarget,
isAbsence,
itemTarget,
succeeded,
withBodies,
showsPlacement,
SimulatedState,
type CheckOutcome,
type PendingWrite,
} from "./environmentStateWriteChecks.js";
import {
checkedLines,
servedLine,
type CheckedLines,
type ServedLine,
} from "./environmentStateCheckedLines.js";
import {
exercisedBy,
interference,
type WriteHistory,
forgetChecked,
runningAt,
shownFields,
shownWrites,
touchTarget,
WriteEffects,
type PendingTargets,
type TouchedTargets,
writtenFieldChecks,
} from "./environmentStateWriteAttribution.js";
import { applyBulk } from "./environmentStateWriteBulk.js";
import { applyCreate } from "./environmentStateWriteCreate.js";
import { applyDocument } from "./environmentStateWriteDocument.js";
import { applyDelete } from "./environmentStateWriteDelete.js";
import {
applyUpdate,
built,
type EffectContext,
type EffectResult,
type WriteDraft,
type WriteSimulationInput,
type WriteStep,
} from "./environmentStateWriteEffects.js";
import {
classifyWrite,
prefixRelated,
unservedPlan,
type InsertRule,
type StateWritePlan,
} from "./environmentStateWriteRules.js";
import { WriteTaint } from "./environmentStateWriteTaint.js";
import { writeVerdict } from "./environmentStateWriteVerdict.js";
import { compareValues, orderedBy } from "./environmentStateWriteExplain.js";
/** Whether members are in a sorted rule's order, ties kept in place as an insert keeps them. */
const keptInOrder = (
members: readonly JsonObject[],
rule: Extract<InsertRule, { rule: "sorted" }>,
): boolean =>
members.every((member, index) => {
if (index === 0) return true;
const order = compareValues(
ownMember(members[index - 1]!, rule.field),
ownMember(member, rule.field),
);
return (
order !== undefined &&
(rule.direction === "ascending" ? order <= 0 : order >= 0)
);
});
/** Whether a list read's query sets a parameter the view's query model reads as a sort or its direction. */
const sortedByQuery = (
view: Extract<StateView, { shape: "list" }>,
record: StateRecordFacts,
): boolean => {
const sent = new Set(
[...new URLSearchParams(record.query ?? "").keys()].map((key) =>
key.toLowerCase(),
),
);
return Object.entries(view.query?.parameters ?? {}).some(
([parameter, { role }]) =>
(role === "sort" || role === "direction") &&
sent.has(parameter.toLowerCase()),
);
};
const APPLY = {
create: applyCreate,
update: applyUpdate,
delete: applyDelete,
document: applyDocument,
} as const;
/**
* Replay the capture's timeline on a copy of the initial state: modelled writes apply their
* effects, other writes taint related types, and each read the served state would answer is
* compared with its recording and attributed to the writes it exercises.
*/
export class EnvironmentStateWriteSimulation {
readonly #input: WriteSimulationInput;
readonly #state: SimulatedState;
readonly #drafts = new Map<string, WriteDraft>();
readonly #toolOf = new Map<number, StateToolFacts>();
readonly #types = new Map<string, StateType>();
/** Each list or document view's checked request lines, with the recording a call of the line is answered on. */
readonly #checkedLines: ReadonlyMap<string, CheckedLines>;
/** Reads a write's effect was learned from, with the tools that borrowed each. */
readonly #observations = new Map<number, Set<string>>();
readonly #taint: WriteTaint;
readonly #touched: TouchedTargets = new Map();
readonly #effects = new WriteEffects();
/** Fields reads show the application leaves out, which serving then does not require. */
readonly omitted = new OmittedFields((bytes) => this.#input.io.retain(bytes));
/** Array fields reads showed a view lists sorted. */
readonly #sorted: SortedArrays;
readonly #lastCheck = new Map<string, number>();
/** When the latest modelled write to each target since its last check ended. */
readonly #touchedSince: PendingTargets = new Map();
/** Timeline indexes of writes the gate does not model. */
readonly #unmodelledAt: number[] = [];
readonly #writeCompletion: CapturedWriteCompletion;
readonly #history: WriteHistory = {
touched: this.#touched,
pending: this.#touchedSince,
checked: this.#lastCheck,
unmodelledAt: this.#unmodelledAt,
};
readonly #log: {
readonly index: number;
readonly tool: string;
readonly path: string;
}[] = [];
/** Entities a create inserted, by target, with the create's tool. */
readonly #inserted = new Map<string, string>();
/** Targets a write changed or placed since a passing list read last showed them, whose position no read has confirmed. */
readonly #moved = new Set<string>();
/** The writes to those targets' entities, in order. */
#pending: PendingWrite[] = [];
/** Sorts, by type, that a whole list in collection order ruled out: moving the writes it showed by them gives another order. */
readonly #refutedSorts = new Map<string, Set<string>>();
/** Creates whose insert rule a passing list check relied on, so it is no longer changed. */
readonly #placed = new Set<string>();
/** The order a read showed a type's collections sorted in after an update, when no create of it showed one. */
readonly #learnedSorts = new Map<string, InsertRule>();
/** Entities writes no plan models made, taken when an item read first shows them. */
readonly #unseen: UnseenEntities;
readonly #blame: FailedReadBlame;
constructor(input: WriteSimulationInput) {
this.#input = input;
this.#writeCompletion = new CapturedWriteCompletion(input.io.retain);
this.#state = new SimulatedState(input.entities, input.documents);
this.#unseen = new UnseenEntities(input.entities, {
state: this.#state,
io: input.io,
lastCheck: this.#lastCheck,
idField: (type) => this.#types.get(type)?.id_field,
});
this.#blame = new FailedReadBlame({
state: this.#state,
io: input.io,
records: input.records,
touched: this.#touched,
pending: this.#touchedSince,
effects: this.#effects,
inserted: this.#inserted,
idField: (type) => this.#types.get(type)?.id_field,
volatileOf: input.volatileOf,
});
for (const type of input.types)
this.#types.set(type.collection_template, type);
for (const tool of input.tools)
for (const ordinal of tool.record_ordinals)
this.#toolOf.set(ordinal, tool);
this.#checkedLines = checkedLines(input);
this.#sorted = new SortedArrays(input.views.values(), input.entities, {
state: this.#state,
io: input.io,
idField: (type) => this.#types.get(type)?.id_field,
});
this.#taint = new WriteTaint(
[...this.#types.keys()],
new Map(
input.documents.map((document) => [document.tool, document.path]),
),
this.#state,
);
this.#state.keepSorted((type) => this.#sortOf(type));
}
/**
* The sorted order a type's placed creates insert by, when they agree on one and nothing contradicted them: updates
* move their entity only by an order a list read showed, in replay as in serving.
*/
#sortOf(type: string): InsertRule | undefined {
const rules = new Map<string, InsertRule>();
for (const { plan, observed, contradictedBy } of this.#drafts.values())
if (
plan.effect === "create" &&
plan.target === type &&
observed !== undefined &&
plan.reason === null &&
contradictedBy === undefined &&
plan.insert?.rule === "sorted" &&
this.#placed.has(plan.tool)
)
rules.set(JSON.stringify(plan.insert), plan.insert);
if (rules.size === 0) return this.#learnedSorts.get(type);
return rules.size === 1 ? [...rules.values()][0] : undefined;
}
/** Every state-changing tool's plan, in manifest order, with its verdict. */
run(): StateWritePlan[] {
const documents = new Map<string, string>();
for (const tool of this.#input.tools)
if (this.#input.views.get(tool.name)?.shape === "document")
documents.set(normalizedPath(tool.path_template), tool.name);
const types = new Set(this.#types.keys());
for (const tool of this.#input.tools) {
if (
!changesState(tool.method, tool.path_template) ||
this.#input.readOnly.has(tool.name)
)
continue;
const link = linkedTarget(tool, classifyWrite(tool, types, documents), {
...this.#input,
isWrite: (record) => this.#isWrite(record),
viewOf: (ordinal) =>
this.#input.views.get(this.#toolOf.get(ordinal)?.name ?? ""),
});
const addressed =
link.effect === "none"
? addressedWrite(tool, {
...this.#input,
types,
idField: (type) => this.#types.get(type)?.id_field,
views: this.#input.views.values(),
})
: undefined;
const { effect, target } = addressed ?? link;
this.#input.io.retain(512);
this.#drafts.set(tool.name, {
plan: {
...unservedPlan(tool.name, effect, target, "no-state-target"),
reason: effect === "none" ? "no-state-target" : null,
...(addressed === undefined ? {} : { address: addressed.address }),
},
observed: undefined,
candidates: [],
confirmed: [],
contradictedBy: undefined,
sideEffect: false,
calls: 0,
updates: [],
sequences: new Map(),
steady: new Set(),
unstamped: new Set(),
kept: new Set(),
fixedWidth: new Map(),
ampersands: new Map(),
linked: link.collection,
});
}
// Reads may finish after writes with the same or a later start: index the
// complete capture before replay so ordinal tie-breaking cannot create proof.
for (const record of this.#input.records)
if (this.#isWrite(record))
this.#writeCompletion.write(
record.path,
this.#input.chronology.endMs(record),
this.#input.chronology.startMs(record),
);
this.#input.records.forEach((record, index) => this.#step(record, index));
const views = new Map(
this.views([...this.#input.views.values()]).map((view) => [
view.tool,
view,
]),
);
const sorts = new Map(
[...this.#types.keys()].map((type) => [type, this.#sortOf(type)]),
);
return [...this.#drafts.values()].map((draft) => {
// A view that became recorded cannot supply proof for a served write.
let kept = 0;
for (const ordinal of draft.confirmed) {
const tool = this.#toolOf.get(ordinal)?.name;
if (tool !== undefined && views.get(tool)?.shape !== "recorded")
draft.confirmed[kept++] = ordinal;
}
draft.confirmed.length = kept;
// An update moves its entity in a sorted collection when it writes the field sorted by.
const sort =
draft.plan.effect === "update" && draft.plan.target !== null
? sorts.get(draft.plan.target)
: undefined;
if (
sort?.rule === "sorted" &&
draft.plan.rules.some((rule) => rule.field === sort.field)
)
draft.plan.insert = sort;
return writeVerdict(draft, this.#input);
});
}
#step(record: StateRecordFacts, index: number): void {
const tool = this.#toolOf.get(record.ordinal);
// A redirected or unanswered write may have changed state, even made an entity, but only a 2xx one shows its effect.
if (this.#isWrite(record) && !succeeded(record)) {
this.#unmodelled(record.path, index);
this.#unseen.unplanned(index);
} else if (this.#isWrite(record)) this.#write(record, index, tool);
else if (tool?.method === "GET") this.#read(record, index, tool);
}
/** Whether a record may change state: a state write whose tool the capture does not show reading only. */
#isWrite(record: StateRecordFacts): boolean {
return (
isStateWrite(record) &&
!this.#input.readOnly.has(this.#toolOf.get(record.ordinal)?.name ?? "")
);
}
#context(
writer: string,
request?: { readonly ordinal: number; readonly body: ParsedRequestBody },
): EffectContext {
return {
isWrite: (record) => this.#isWrite(record),
state: this.#state,
io:
request === undefined
? this.#input.io
: {
...this.#input.io,
requestBody: (ordinal) =>
ordinal === request.ordinal
? request.body
: this.#input.io.requestBody(ordinal),
},
records: this.#input.records,
chronology: this.#input.chronology,
viewOf: (ordinal) => {
const view = this.#input.views.get(
this.#toolOf.get(ordinal)?.name ?? "",
);
return view === undefined ? undefined : this.#viewOf(view);
},
idField: (type) => this.#types.get(type)!.id_field,
volatileOf: this.#input.volatileOf,
observed: (ordinal) =>
this.#observations.set(
ordinal,
(this.#observations.get(ordinal) ?? new Set()).add(writer),
),
unknown: (type, collection, id) =>
this.#taint.unknown(type, collection, id),
initialDocument: (tool) =>
this.#input.documents.find((document) => document.tool === tool),
startedWith: (type) =>
this.#input.entities.filter((entity) => entity.type === type),
laterFields: (type) =>
laterItemFields(this.#input.views.values(), this.#input.entities, type),
pending: (target) =>
[...(this.#touched.get(target)?.keys() ?? [])].some(
(name) => name !== writer,
),
};
}
#write(
record: StateRecordFacts,
index: number,
tool: StateToolFacts | undefined,
): void {
const draft = tool === undefined ? undefined : this.#drafts.get(tool.name);
const effect = draft?.plan.effect;
if (
draft === undefined ||
tool === undefined ||
effect === undefined ||
effect === "none"
) {
this.#unmodelled(record.path, index);
this.#unseen.unplanned(index);
return;
}
const observeOnly = draft.plan.reason !== null;
if (!observeOnly) draft.calls += 1;
// Serving pins the exemplar's query, so a call with another may do something else and confirms nothing.
const pinned =
!built(draft) ||
sameQuery(
this.#input.io.queryOf(draft.plan.exemplar_ordinal ?? -1),
record.query,
);
const step: WriteStep = { record, index, tool, observeOnly, pinned };
const { address } = draft.plan;
const bulk = address !== undefined && address.by !== "action";
const apply = (body?: ParsedRequestBody): void => {
this.#state.track();
const context = this.#context(
tool.name,
body === undefined ? undefined : { ordinal: record.ordinal, body },
);
const results = bulk
? applyBulk(context, draft, step)
: [APPLY[effect](context, draft, step)];
// A call of a tool whose plan was never built, or one a linked plan does not place, is no more modelled than one of a tool without a plan.
if (
results.some((result) => result.kind === "unknown") &&
(!built(draft) || draft.linked !== undefined)
)
this.#unseen.unplanned(index);
for (const result of results)
this.#settle(result, {
index,
tool: tool.name,
path: record.path,
end: this.#input.chronology.endMs(record),
});
};
// Item deletes use their path; calls outside a built plan's query never interpret their body.
if (pinned && (effect !== "delete" || bulk))
this.#input.io.withRequestBody(record.ordinal, apply);
else apply();
}
/** Attribute an applied effect to its tool; a recorded outcome is the state's new baseline; an unknown one taints. */
#settle(
result: EffectResult,
write: {
readonly index: number;
readonly tool: string;
readonly path: string;
readonly end: number | undefined;
},
): void {
if (result.kind === "unknown") {
this.#unmodelled(write.path, write.index);
return;
}
if (result.entity !== undefined) {
this.#unseen.touched(result.entity);
const { type, collection, id } = result.entity;
this.#taint.known(type, collection, id);
}
if (result.kind === "observed") {
this.#effects.observed(this.#touched, result.targets);
this.#blame.rebased(result.targets[0]!, write.index);
// A recorded outcome replaces the entity's values, not its collection's membership.
this.#touched.delete(result.targets[0]!);
return;
}
this.#effects.supersede(this.#touched, write.tool, result);
if (result.entity !== undefined) {
const { type, collection, id } = result.entity;
const target = entityTarget(type, collection, id);
const before = this.#state.priorFields(target);
if (before !== undefined) this.#pending.push({ target, before });
}
for (const target of result.targets) {
this.#moved.add(target);
touchTarget(this.#touched, this.#touchedSince, target, {
...write,
fields: result.fields,
});
}
this.#log.push(write);
if (result.inserted === undefined) return;
this.#inserted.set(result.inserted, write.tool);
if (result.entity !== undefined)
this.#blame.inserted(result.entity, write.tool);
}
/**
* An item read of an entity an unmodelled write may have changed shows it again: the writes that
* changed it are checked on their own fields, and the state takes what the read recorded.
*/
#resync(
view: Extract<StateView, { shape: "item" }>,
record: StateRecordFacts,
entity: StateEntity,
): void {
const target = entityTarget(entity.type, entity.collection, entity.id);
const writes = this.#touched.get(target) ?? new Map();
this.#taint.known(entity.type, entity.collection, entity.id);
this.#touched.delete(target);
if (
isAbsence(record.status) ||
this.#input.io.withBody(record.ordinal, (body) =>
markedDeleted(
body === undefined || body === null
? undefined
: at(body, view.pointer),
),
)
) {
this.#removeEntity(entity);
return;
}
const object = this.#checkFields(view, record, entity, writes);
if (object !== undefined)
this.#state.put(updatedEntity(entity, { ...entity.fields, ...object }));
}
/** Whether an item read shows an entity an unmodelled write made, or one left unknown, again rather than checks it. */
#resynced(view: StateView, record: StateRecordFacts, index: number): boolean {
if (view.shape !== "item") return false;
if (this.#writeOverlapsRead(record)) return true;
const adopted = this.#unseen.adopt(view, { record, index });
if (adopted !== undefined) {
this.#taint.known(adopted.type, adopted.collection, adopted.id);
return true;
}
const { collection, id } = itemTarget(this.#state, view.type, record);
if (!this.#taint.unknown(view.type, collection, id)) return false;
// A read begun while a modelled write to the entity still ran may show either state, so it neither checks nor shows it.
const target = entityTarget(view.type, collection, id);
if (
runningAt(
this.#touchedSince,
target,
this.#input.chronology.startMs(record),
)
)
return true;
// A failed read that is no absence shows nothing of the entity, so it stays unknown for a usable read.
if (!succeeded(record) && !isAbsence(record.status)) return true;
const entity = this.#state.entity(view.type, collection, id);
// A recorded absence of an entity the state no longer holds is still a check of its delete.
if (entity === undefined) {
if (isAbsence(record.status)) return false;
this.#taint.known(view.type, collection, id);
return true;
}
this.#resync(view, record, entity);
return true;
}
#check(
view: StateView,
record: StateRecordFacts,
comparisonBody?: (body: JsonValue) => JsonValue,
): CheckOutcome | undefined {
if (view.shape === "recorded") return undefined;
const line = this.#line(view, record);
if (view.shape !== "item" && line === undefined) return undefined;
if (view.shape === "document")
return this.#input.io.withBody(record.ordinal, (body) =>
checkDocumentRead(this.#state, view, record, body),
);
const item = view.shape === "item";
const target = item
? itemTarget(this.#state, view.type, record)
: undefined;
const collection =
target === undefined ? line!.collection : target.collection;
if (this.#taint.under(collection)) return undefined;
if (!item) {
const ids = this.#state
.members(view.type, collection)
.map((entity) => entity.id);
// A recorded-membership list covers no collection target, so a write at the collection is checked here.
if (
this.#taint.tainted(collectionTarget(view.type, collection)) ||
this.#taint.blocksList(view.type, collection, ids)
)
return undefined;
}
// The served answer keeps the envelope of the item view's or the line's base recording.
const base = item ? view.envelope_ordinal : line!.ordinal;
const ordinals = { base, read: record.ordinal };
const minted = view.per_response_members;
return withBodies(this.#input.io.withBody, ordinals, (served, body) => {
const check = {
body:
body === undefined
? body
: withoutMembers(
comparisonBody === undefined ? body : comparisonBody(body),
minted,
),
base: served === undefined ? served : withoutMembers(served, minted),
idField: this.#types.get(view.type)?.id_field ?? "",
volatile: this.#input.volatileOf(view.type),
collection,
...(target === undefined ? {} : { itemId: target.id }),
};
return this.omitted.check(this.#state, view, record, check, (loosened) =>
loosened.shape === "item"
? checkItemRead(this.#state, loosened, record, check)
: checkListRead(this.#state, loosened, record, {
...check,
line: line!,
}),
);
});
}
/** The checked line serving answers a list or document read's call from, as replay chooses it. */
#line(view: StateView, record: StateRecordFacts): ServedLine | undefined {
return servedLine(this.#checkedLines.get(view.tool), record);
}
/** The view as the gate currently knows it: without the fields reads showed omitted, with the arrays they showed sorted. */
#viewOf(view: StateView): StateView {
return this.#sorted.view(this.omitted.view(view));
}
/** The views as reads showed them. */
views(views: readonly StateView[]): StateView[] {
return this.#sorted.views(this.omitted.views(views));
}
#read(record: StateRecordFacts, index: number, tool: StateToolFacts): void {
const original = this.#input.views.get(tool.name);
if (original === undefined) return;
let view = this.#sorted.comparisonView(this.omitted.view(original));
if (projectedRead(view, record, this.#input.io)) return;
if (this.#writeOverlapsRead(record)) return;
const borrowers = this.#observations.get(record.ordinal);
if (
this.#viewOf(original).shape !== "recorded" &&
!borrowers &&
this.#resynced(view, record, index)
)
return;
let outcome = this.#check(view, record);
if (outcome === undefined) return;
let keys = [...outcome.targets, ...outcome.unheld];
const blocked = (targets: readonly string[]) =>
targets.some(
(key) =>
this.#taint.tainted(key) ||
runningAt(
this.#touchedSince,
key,
this.#input.chronology.startMs(record),
),
);
if (blocked(keys)) return;
// A read the gate counts can show the view lists a written array sorted, or rule that out.
outcome = this.#sorted.retry(
view,
{ record, outcome },
(candidate, comparisonBody) =>
this.#check(candidate, record, comparisonBody),
blocked,
);
view = this.#viewOf(original);
keys = [...outcome.targets, ...outcome.unheld];
if (blocked(keys)) return;
let exercised = exercisedBy(this.#touched, keys, borrowers);
if (view.shape === "recorded") {
// Retiring a view withdraws ordering proof, not a remaining mismatch in values.
if (!outcome.pass)
this.#contradictOthers(
exercised,
record,
interference(this.#history, outcome),
);
return;
}
if (borrowers !== undefined) {
this.omitted.keep(view, outcome);
if (!outcome.pass)
this.#contradictOthers(
exercised,
record,
interference(this.#history, outcome),
);
return;
}
if (!outcome.pass && view.shape === "list") {
const retried =
this.#retryInsert(view, record, exercised) ??
this.#retrySort(view, record, exercised);
if (retried !== undefined) {
// Reordering can replace the entities on a query page. Confirmation
// and pending-write cleanup must use the successful check's coverage.
outcome = retried;
keys = [...outcome.targets, ...outcome.unheld];
if (blocked(keys)) return;
exercised = exercisedBy(this.#touched, keys, borrowers);
}
}
// Retain omissions only after the final check's coverage passes the ownership guards.
this.omitted.keep(view, outcome);
const interferes = interference(this.#history, outcome);
const unseen = outcome.pass
? this.#confirm(view, record, outcome, exercised)
: undefined;
// Only a read whose order the state decides for every entity it lists shows them in place.
if (
outcome.pass &&
view.shape === "list" &&
outcome.positional === undefined
) {
this.#refuteSorts(view, record);
const shown = new Set(outcome.targets);
for (const target of shown) this.#moved.delete(target);
this.#pending = this.#pending.filter((write) => !shown.has(write.target));
}
if (!outcome.pass) {
const interfered = new Set([...exercised].filter(interferes));
if (interfered.size === 0)
this.#mismatch({ record, index, keys, exercised, view });
else {
// An unmodelled write after some of these writes may explain the difference: those are checked on their own members only.
this.#contradictOthers(exercised, record, interferes);
if (view.shape === "item") this.#checkMembers(view, record, interfered);
else for (const key of keys) this.#taint.fail(key);
}
if (view.shape === "item") this.#adopt(view, record);
}
const pending = {
touched: this.#touched,
since: this.#touchedSince,
effects: this.#effects,
};
for (const key of keys) {
this.#lastCheck.set(key, index);
forgetChecked(pending, key, unseen?.get(key));
}
}
/** A failed read contradicts the writes it exercised that no unmodelled write came after, as a read a write's effect was learned from still does the others. */
#contradictOthers(
exercised: ReadonlySet<string>,
record: StateRecordFacts,
interferes: (name: string) => boolean,
): void {
for (const name of exercised)
if (!interferes(name))
this.#drafts.get(name)!.contradictedBy ??= record.ordinal;
}
/** A passing read confirms the writes it exercised that it shows something of, and names, per target, those it shows nothing of. */
#confirm(
view: StateView,
record: StateRecordFacts,
outcome: CheckOutcome,
exercised: ReadonlySet<string>,
): Map<string, Set<string>> {
const shown = shownWrites(this.#touched, this.#effects, {
keys: [...outcome.targets, ...outcome.unheld],
exercised,
fields: shownFields(view, this.#input.volatileOf),
});
for (const name of shown.confirmed) {
this.#drafts.get(name)!.confirmed.push(record.ordinal);
if (showsPlacement(view, outcome, this.#inserted, name)) {
this.#placed.add(name);
if (view.shape === "list") this.#narrowInserts(view, record, name);
}
}
return shown.unseen;
}
/** An overlapping captured write cannot supply a stable read baseline. */
#writeOverlapsRead(read: StateRecordFacts): boolean {
return this.#writeCompletion.runningAt(
read.path,
this.#input.chronology.startMs(read),
this.#input.chronology.endMs(read),
);
}
#unmodelled(path: string, index: number): void {
this.#taint.write(path);
this.#unmodelledAt.push(index);
}
/** Check the writes to an item read's entity on the members they sent, as a resynchronising read does. */
#checkMembers(
view: Extract<StateView, { shape: "item" }>,
record: StateRecordFacts,
only: ReadonlySet<string>,
): void {
const { collection, id } = itemTarget(this.#state, view.type, record);
const entity = this.#state.entity(view.type, collection, id);
if (entity === undefined) return;
const target = entityTarget(entity.type, entity.collection, entity.id);
const writes = [...(this.#touched.get(target) ?? [])];
const checked = new Map(writes.filter(([name]) => only.has(name)));
this.#checkFields(view, record, entity, checked);
}
/** Check the writes to an entity on the fields each wrote against an item read; the read's object, when it shows one. */
#checkFields(
view: Extract<StateView, { shape: "item" }>,
record: StateRecordFacts,
entity: StateEntity,
writes: ReadonlyMap<string, readonly string[] | null> = new Map(),
): JsonObject | undefined {
const object = this.#input.io.withBody(record.ordinal, (body) =>
body === undefined || body === null ? undefined : at(body, view.pointer),
);
if (!isObject(object)) return undefined;
for (const [name, pass] of writtenFieldChecks(entity, view, object, {
volatile: this.#input.volatileOf(view.type),
writes,
})) {
const draft = this.#drafts.get(name)!;
if (pass) draft.confirmed.push(record.ordinal);
else draft.contradictedBy ??= record.ordinal;
}
return object;
}
/** After a failed item check the state takes what the read recorded, so later checks start from it. */
#adopt(
view: Extract<StateView, { shape: "item" }>,
record: StateRecordFacts,
): void {
const { collection, id } = itemTarget(this.#state, view.type, record);
const entity = this.#state.entity(view.type, collection, id);
if (entity === undefined) {
this.#taint.fail(entityTarget(view.type, collection, id));
return;
}
const object = this.#input.io.withBody(record.ordinal, (body) =>
body === undefined || body === null ? undefined : at(body, view.pointer),
);
if (isObject(object) && !markedDeleted(object))
this.#state.put(updatedEntity(entity, { ...entity.fields, ...object }));
else this.#removeEntity(entity);
}
/** A read showed the entity gone: the state drops it, and no write keeps a claim on its collection through it. */
#removeEntity(entity: StateEntity): void {
this.#state.remove(entity);
this.#effects.removed(this.#touched, entity);
}
/** Contradict the writes a failed check exercised, or, when none, the related writes since its targets were last checked; a failed list stops being checked. */
#mismatch(failed: {
readonly record: StateRecordFacts;
readonly index: number;
readonly keys: readonly string[];
readonly exercised: ReadonlySet<string>;
readonly view: StateView;
}): void {
const ordinal = failed.record.ordinal;
const since = Math.min(
...failed.keys.map((key) => this.#lastCheck.get(key) ?? -1),
);
const blame = this.#blame.of({ ...failed, since });
for (const name of blame.confirmed)
this.#drafts.get(name)!.confirmed.push(ordinal);
for (const name of blame.contradicted)
this.#drafts.get(name)!.contradictedBy ??= ordinal;
if (failed.exercised.size === 0)
for (const write of this.#log)
if (
write.index > since &&
prefixRelated(write.path, failed.record.path)
)
this.#drafts.get(write.tool)!.sideEffect = true;
if (failed.view.shape !== "item")
for (const key of failed.keys) this.#taint.fail(key);
}
/** What `use` makes of a read's items when the read lists the whole collection the state holds, the only read that shows the order it is kept in. */
#wholeListing<T>(
view: Extract<StateView, { shape: "list" }>,
record: StateRecordFacts,
use: (items: readonly JsonObject[]) => T,
): T | undefined {
const line = this.#line(view, record);
const idField = this.#types.get(view.type)?.id_field;
if (line === undefined || idField === undefined) return undefined;
const held = new Set(
this.#state
.members(view.type, line.collection)
.map((entity) => entity.id),
);
return this.#input.io.withBody(record.ordinal, (body) => {
const items =
body === undefined ? undefined : listItems(body, view.pointer);
if (
items === undefined ||
items.length < 2 ||
items.length !== held.size ||
!items.every((item) =>
held.has(scalarText(ownMember(item, idField)) ?? ""),
)
)
return undefined;
return use(items);
});
}
/** A whole-collection read in collection order that placed a create leaves it only the sorted inserts whose order the read follows; one its query sorts shows the query's order instead. */
#narrowInserts(
view: Extract<StateView, { shape: "list" }>,
record: StateRecordFacts,
name: string,
): void {
if (sortedByQuery(view, record)) return;
const draft = this.#drafts.get(name)!;
const kept = this.#wholeListing(view, record, (items) =>
draft.candidates.filter(
(candidate) =>
candidate.rule !== "sorted" ||
same(candidate, draft.plan.insert) ||
orderedBy(items, candidate.field, candidate.direction),
),
);
if (kept !== undefined) draft.candidates = kept;
}
/**
* The successful check with the collection sorted by a field an update the read exercised writes, when the read
* lists the whole collection in that order and no create of the type was placed by a sort that excludes it; the
* type's creates then insert by it, and its updates move their entity by it. A sort the creates chose by a field the
* update did not write does not stop this when every whole-collection read that placed them followed this order too.
*/
#retrySort(
view: Extract<StateView, { shape: "list" }>,
record: StateRecordFacts,
exercised: ReadonlySet<string>,
): CheckOutcome | undefined {
const line = this.#line(view, record);
// A read its query sorts shows the query's order, which may break ties otherwise than the collection.
if (line === undefined || sortedByQuery(view, record)) return undefined;
const written = new Set(
[...exercised].flatMap((name) => {
const { plan } = this.#drafts.get(name)!;
return plan.effect === "update" && plan.target === view.type
? plan.rules.map((rule) => rule.field)
: [];
}),
);
// A sort by a field the update wrote has moved its entity already; one by another field may be a guess both fit.
const current = this.#sortOf(view.type);
if (current?.rule === "sorted" && written.has(current.field))
return undefined;
const creates = [...this.#drafts.values()].filter(
(draft) =>
draft.plan.effect === "create" &&
draft.plan.target === view.type &&
draft.observed !== undefined,
);
const rules =
this.#wholeListing(view, record, (items) =>
[...written].flatMap((field) =>
(["ascending", "descending"] as const).flatMap((direction) =>
orderedBy(items, field, direction)
? [{ rule: "sorted", field, direction } as const]
: [],
),
),
) ?? [];
// Serving moves only the entities writes changed since a list last showed them, so the others, including one an
// earlier write moved that a list then showed in place, must already be in the order tried.
const kept = this.#state
.members(view.type, line.collection)
.filter(
(entity) =>
!this.#moved.has(
entityTarget(entity.type, entity.collection, entity.id),
),
)
.map((entity) => entity.fields);
const refuted = this.#refutedSorts.get(view.type);
for (const rule of rules.filter(
(candidate) =>
(current === undefined || !same(candidate, current)) &&
refuted?.has(JSON.stringify(candidate)) !== true &&
keptInOrder(kept, candidate),
)) {
// A create placed by another sort shows the collection is not kept in this order.
if (
creates.some(
({ candidates }) =>
candidates.some((candidate) => candidate.rule === "sorted") &&
!candidates.some((candidate) => same(candidate, rule)),
)
)
continue;
// Serving moves each written entity in turn, against the others' values before their own writes.
const before = this.#state.replay(
view.type,
line.collection,
rule,
this.#pending,
);
const outcome = this.#check(view, record);
if (outcome?.pass === true) {
for (const draft of creates) draft.plan.insert = rule;
this.#learnedSorts.set(view.type, rule);
return outcome;
}
this.#state.restore(view.type, line.collection, before);
}
return undefined;
}
/** Note the sorts by a field a pending write changed that this passing whole list in collection order rules out. */
#refuteSorts(
view: Extract<StateView, { shape: "list" }>,
record: StateRecordFacts,
): void {
const line = this.#line(view, record);
if (
line === undefined ||
this.#wholeListing(view, record, () => true) === undefined
)
return;
const members = new Map(
this.#state
.members(view.type, line.collection)
.map((entity) => [
entityTarget(entity.type, entity.collection, entity.id),
entity.fields,
]),
);
const writes = this.#pending.filter((write) => members.has(write.target));
const fields = new Set<string>();
for (const [index, { target, before }] of writes.entries()) {
if (before === null) continue;
const after =
writes.slice(index + 1).find((later) => later.target === target)
?.before ?? members.get(target)!;
for (const field of Object.keys(after))
if (!same(ownMember(before, field), ownMember(after, field)))
fields.add(field);
}
const order = [...members.keys()];
const refuted = this.#refutedSorts.get(view.type) ?? new Set<string>();
for (const field of fields)
for (const direction of ["ascending", "descending"] as const) {
const rule = { rule: "sorted", field, direction } as const;
const replayed = this.#state.replayedOrder(
view.type,
line.collection,
rule,
writes,
);
if (replayed !== undefined && !same(replayed, order))
refuted.add(JSON.stringify(rule));
}
if (refuted.size > 0) this.#refutedSorts.set(view.type, refuted);
}
/** The successful check under another create insertion candidate, including its possibly different query-page coverage. */
#retryInsert(
view: Extract<StateView, { shape: "list" }>,
record: StateRecordFacts,
exercised: ReadonlySet<string>,
): CheckOutcome | undefined {
const line = this.#line(view, record);
if (line === undefined || !listsAll(view, line.ordinal)) return undefined;
const members = this.#state.members(view.type, line.collection);
const key = (entity: StateEntity) =>
entityTarget(entity.type, entity.collection, entity.id);
const inserted = members.filter((entity) =>
this.#inserted.has(key(entity)),
);
const creates = new Set(
inserted.map((entity) => this.#inserted.get(key(entity))!),
);
if (creates.size !== 1) return undefined;
const [name] = creates;
const draft = this.#drafts.get(name!)!;
// An item read confirms the entity, not where a list places it.
if (!exercised.has(name!) || this.#placed.has(name!)) return undefined;
for (const candidate of draft.candidates.slice(1)) {
this.#state.reinsert(view.type, line.collection, inserted, candidate);
const outcome = this.#check(view, record);
if (outcome?.pass === true) {
draft.plan.insert = candidate;
return outcome;
}
}
return undefined;
}
}