From 40b9c03f996270e944ff07cafb63be15ce903708 Mon Sep 17 00:00:00 2001 From: Edgar Arakelyan Date: Tue, 27 Oct 2026 16:51:03 -0700 Subject: [PATCH] feat(environment): serve query numbers, counted values and made objects (#7332) * feat(environment): serve the writes callers make without recorded form fields - A create member a recording sent empty ("", null, false, [] or {}) is no longer required; a call that leaves it out is applied as sending that value (body.blank). - List views render envelope members that list the listed entities' ids (Paperless's `all`), so creates in such collections are confirmed. - Updates move their entity in collections kept sorted by a field they write, and a failed whole-collection read after an update can teach that order. - A later create that stores an unruled field otherwise is withheld at the create, and an item read may fill values a create's answer had not loaded. - An unsent field respelled between null, [] and {} is unchanged. - The skill asks for one create with every field and one with only what a person must give. Co-Authored-By: Claude Opus 5.5 * docs(skill): show how to read signed-in pages before recording Agents guessed capture keys (artifacts, dom, final) to see pages behind sign-in, which inspect_web_page cannot reach. Co-Authored-By: Claude Opus 5.5 * fix(environment): accept every empty spelling recorded creates sent A create that fills only what a person must give sends `null` where the full create sent a string, and `null` or `false` for the same flag. The merged plan now accepts each recorded empty value, and a call that leaves the member out is applied as the earliest call sent it. Also adds the spec folder for this change. Co-Authored-By: Claude Opus 5.5 * docs(skill): keep the vary-fields step neutral on UI or API writes Agents read "the form" as a cue to write through the UI, where some applications' request bodies are not exposed; a body-unusable UI write now re-records once with request steps. Co-Authored-By: Claude Opus 5.5 * docs(environment): record verification for blank members and sorted writes Co-Authored-By: Claude Opus 5.5 * fix(environment): address review of blank members and sorted writes - Bulk updates move their entities in turn against the others' earlier values and install the final order once, as replay does. - A member's recorded empty values stay accepted once a later call teaches its default; a left-out member with a default keeps the default. - Write answers spell empty lists and maps as their recorded answers did. - A rename can teach a sort when the creates chose one by a field the update did not write, such as ids that ascend with titles. Co-Authored-By: Claude Opus 5.5 * feat(environment): serve the writes that later reads explain - An item read whose object is marked deleted (deleted, isDeleted or is_deleted true) agrees with an absent entity, as EspoCRM answers a deleted record; such an object never seeds or adopts an entity. - maxSize, max_results and maxResults are page sizes, for whole-collection list lines and the list query's limit role. - An item view also shows the fields only reads after its type's first write show, when no initial entity holds them (Vikunja's task subscription), without requiring them; write answers leave such a field out for entities without it, and it no longer withholds the write as incomplete-entities. - The skill asks updates to change the fields a person would change, and expects a deleted object to read back as 404 or marked deleted. Co-Authored-By: Claude Opus 5.5 * fix(environment): answer sent empty members as sent, and keep a placed create's order - A write answer spells an empty list or map the call did not send as its recorded answer did; one the call sent is answered as sent, so a create sending `[]` is not answered `null` because the first recorded create was. - A read listing the whole collection that placed a create leaves the create only the sorted inserts whose order that read follows, so a later rename cannot teach a sort an earlier read ruled out. Co-Authored-By: Claude Opus 5.5 * fix(environment): answer bulk updates without fields only later reads show A bulk update's answer renders each entity with the same allowance a single update's answer has: a field only reads after the type's first write showed, which no initial entity holds, is left out for an entity without it, so a bulk update of a seeded task is no longer refused for lacking it. Co-Authored-By: Claude Opus 5.5 * docs(environment): correct how the EspoCRM contact update task passes The agent trajectories show gpt-oss passes it by creating a replacement contact and deleting the original, since the served update refuses emailAddress; the update itself is refused in both builds. Co-Authored-By: Claude Opus 5.5 * fix(environment): answer a read of a served-deleted entity marked deleted An application that soft-deletes still answers a deleted record by id, marked deleted. Serving now keeps an entity a served delete removed from a type whose recorded item reads show that, until a reset or load and within the size data/add loads, and answers a read of it with its last fields and the flag true, instead of refusing the read. Co-Authored-By: Claude Opus 5.5 * fix(environment): keep unloaded create answers, check bulk answers with sent members, narrow inserts only on collection-order reads - A create whose item read loaded values its answer had not records them (`observation.read.unloaded`), and a served create's answer keeps them as the recorded answer showed them, a zero-time owner included, while the state holds the read's. - Replay checks a bulk update's recorded answers with each target's sent members, as serving answers them, so echoed `null` and `[]` no longer make the plan body-unusable. - A whole-collection read whose query sorts it shows the query's order, so it no longer narrows a create's sorted insert candidates. Co-Authored-By: Claude Opus 5.5 * docs(environment): record the census with main's unknown-clock rule merged Co-Authored-By: Claude Opus 5.5 * feat(environment): serve query numbers, counted values and made objects - A query parameter accepts a number or boolean, which serving sends as its JSON text, as agents send page sizes and flags. - A sequence may start at the value its first create took above every value its type held (Vikunja's per-project task index), and a later create's composed, template or started-sequence rule is adopted when it also spells the first create's value. - A made-object rule explains an object the application makes with each created entity: members the request stamped, members holding the entity's own id, and counts one past the held objects' values. - A create's answer names the id of the one object a one-element array member holds, so bulk creates link to their type. - The skill reads each listed object by id before writing. Co-Authored-By: Claude Opus 5.5 * feat(environment): let a linked create take path values it stores A create linked to its type's own collection, whose path has no parameters, accepts any path values when its plan stores each one in a field, as a Vikunja task stores its project; replay and serving apply it alike. Co-Authored-By: Claude Opus 5.5 * fix(environment): learn a sort from a rename only when moving its entities reaches it Serving moves only the entities a write changes in a collection kept sorted, so replay now tries a sort after a rename only when the entities no write the read exercised changed already follow it. Seeded folders listed out of title order no longer teach a title sort that serving would answer with another order. Co-Authored-By: Claude Opus 5.5 * fix(environment): keep unloaded create answers a sent value did not echo, and learn no sort from query-sorted reads - A served create keeps a value its recorded answer had not loaded unless the call sent that member and the plan stores it as sent, so a create sending `active: false` is answered `false` as recorded while the state holds the read's `true`. - A rename no longer teaches a sort from a whole-collection read its query sorts, whose ties need not follow the collection's own order. Co-Authored-By: Claude Opus 5.5 * docs(environment): verify query values, counted values and made objects Co-Authored-By: Claude Opus 5.5 * fix(environment): learn the soft-delete flag in the state view and serve by it Extraction now records the member later item reads set true on a record the application soft-deleted, and the item view carries it as `deleted_flag`. Serving reads that fact instead of scanning recorded bodies itself, so one owner decides what a soft delete looks like. Co-Authored-By: Claude Opus 5.5 * docs(environment): record the call-values checks at #7274 93821024 Co-Authored-By: Claude Opus 5.5 * fix(environment): learn the soft-delete flag from item reads with a query too A client may read a deleted record back only with a query, such as a cache-busting parameter. Every successful read of the item tool now shows the flag; only plain later reads still add fields. Co-Authored-By: Claude Opus 5.5 * docs(environment): record the call-values checks at #7274 45832406 Co-Authored-By: Claude Opus 5.5 * fix(environment): learn sorts only from orders a list read confirmed - A sort retry after a failed whole-collection read requires the entities no write changed since a list last showed them to follow it already, rather than excusing every entity the update's tool ever wrote: an earlier rename a list confirmed in place stays where it was. - An update takes its creates' sort only from a create a list read showed placed; replay still tries a sort a create guessed from the initial order. Co-Authored-By: Claude Opus 5.5 * fix(environment): drop entities read back as deleted and answer soft deletes as deleted - A held entity a later read shows marked deleted is dropped, not adopted with the marked object's fields. - Bulk-update replay renders answers with the optional and later-only fields serving uses, so a bulk update of created and seeded entities replays. - A served delete of a soft-deleting type answers with the entity marked deleted, single and bulk. - A deleted flag only query reads showed is a field of the item view, so a served tombstone keeps it where live records omit it. Co-Authored-By: Claude Opus 5.5 * fix(environment): check made objects as recorded and keep their ids and clocks honest - A made object's clock counts as a write clock, so a capture with no clock anchor withholds the create instead of stamping 1970. - Members holding the entity's identity follow the id written, generated or sent, through the type's id field. - A later create's made object is checked as recorded, before its own members are rewritten, so one naming another entity withholds the plan. - A later create's template is adopted only when it spells that create's value too, not just when the first create's text holds its source value. Co-Authored-By: Claude Opus 5.5 * fix(environment): move updated entities only by placed sorts and keep query-sorted reads from confirming order Replay moved an updated entity by a sort a create only guessed, while the update plan exported only placed sorts, so a read could confirm an update by a move serving would not make. Both now use the placed sort. A passing list read clears its entities from the moved set only when the state decides the order of every entity it lists, so a read whose query sorts it no longer hides an earlier move from a later sort check. Co-Authored-By: Claude Opus 5.5 * docs(environment): count six withheld plans that explain more, not seven The bulk task create counted among them is served on this branch. Co-Authored-By: Claude Opus 5.5 * fix(environment): try a learned sort by moving each pending write in turn, as serving does #retrySort sorted the whole collection, which a list could match while serving, moving each renamed entity in turn against the others' values before their own writes, gave another order. The state now records the values each write replaced, and the retry replays the writes since a list in collection order last showed their entities. Co-Authored-By: Claude Opus 5.5 * fix(environment): answer a served delete with the deleted flag its recorded answer showed An application may answer a delete with the record as it was and mark it deleted only on later reads. Delete answers, single and bulk, now take the flag's value from the recorded answer; reads of a served-deleted entity still show it marked deleted. Co-Authored-By: Claude Opus 5.5 * chore(environment): set aside two hunks next to lines main's saved-values change rewrites Main's #7267 changes the lines beside the blank-member docs and the empty-spelling check in explainUpdate. Setting them aside lets that merge apply cleanly; the commit after it puts them back on top of main's text. Co-Authored-By: Claude Opus 5.5 * chore(environment): restore the set-aside hunks on main and bump the skill to 51 Puts the blank-member docs and the empty-spelling check back beside main's slug and composed text and heldSlugRule. Main is at skill 50 with different text, so this branch's skill changes take 51. Co-Authored-By: Claude Opus 5.5 * chore(environment): bump the skill to 52 above #7260's 51 Co-Authored-By: Claude Opus 5.5 * chore(environment): put back the call-values hunks main's merge took its side for, and bump the skill to 53 The merge of main kept main's side where it overlapped this branch, so it could commit without a conflict. This restores this branch's side next to main's: made-object agreement beside the & slug spellings, sequence starts in the earlier evidence (now built by earlierOf), the combined value imports, and recording step 2 beside main's UI-first step 3. Main is at skill 50 and #7274 at 52, so this branch takes 53. Co-Authored-By: Claude Opus 5.5 * fix(environment): keep sorts a list showed in place ruled out, and created answers' nested members as recorded A whole list in collection order that showed a renamed entity where a sort by the renamed field would have moved it now rules that sort out, so a later rename of the same entity cannot teach it. A create's answer leaves out members its item read added inside an object the answer showed, as the recorded answer did; the state still stores them. Co-Authored-By: Claude Opus 5.5 * fix(environment): keep a later create's made object only with clocks its request stamped followsMadeObjects only checked that a made object's clocks parsed, so a later create whose object showed an older clock still passed, while serving stamps the write's clock. Its clocks must now be in the rule's format and within the stamp window of that create's request. Co-Authored-By: Claude Opus 5.5 * fix(environment): compare soft-deleted read fields * fix(environment): reject contradictory learned sequences * fix(environment): preserve initially deleted item observations * fix(environment): index stored deleted entities separately * fix(environment): replay lists that include deleted records * fix(environment): make no count of a made-object member earlier creates held alike A count only a later create shows starts at the exemplar's value, which is right only while the exemplar is the one create accepted: creates accepted after it showed the member unchanged, which a count would have moved. The merge now refuses such a count, as the top-level sequence check does. Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 Co-authored-by: morluto <76467478+morluto@users.noreply.github.com> --- .../skills/reverse-engineer-anything/SKILL.md | 2 +- .../references/environments.md | 6 +- docs/agent-environments.md | 8 +- .../write-call-values/plan.md | 13 ++ .../write-call-values/spec.md | 46 +++++ .../write-call-values/tasks.md | 13 ++ .../write-call-values/verification.md | 88 +++++++++ docs/web-network-captures.md | 24 ++- .../environment/environmentQueryText.test.ts | 40 +++++ .../environmentStateLaterReadsAnswer.test.ts | 7 + .../environmentStateMadeValuesAnswer.test.ts | 77 ++++++++ .../environmentStateWriteAnswer.ts | 61 +++++-- .../environment/environmentToolAnswer.ts | 11 +- .../environment/environmentTools.test.ts | 13 +- .../environment/environmentTools.ts | 21 ++- .../EnvironmentStateExtraction.ts | 3 + .../EnvironmentStateWriteSimulation.ts | 2 + .../environmentStateCreateCalls.test.ts | 49 +++++ .../analysisView/environmentStateGate.ts | 5 +- .../environmentStateLaterReads.test.ts | 8 + .../environmentStateMadeValues.test.ts | 143 +++++++++++++++ .../analysisView/environmentStateView.ts | 11 ++ .../environmentStateWriteApply.test.ts | 1 + .../environmentStateWriteApply.ts | 30 ++++ .../environmentStateWriteCreate.ts | 110 +++++++++++- .../environmentStateWriteCreateCalls.ts | 105 ++++++++++- .../environmentStateWriteEffects.ts | 4 + .../environmentStateWriteExplain.ts | 6 +- .../environmentStateWriteLinks.ts | 31 +++- .../environmentStateWriteRules.ts | 14 +- .../environmentStateWriteStamps.ts | 144 +++++++++++++++ .../environmentStateWriteValues.ts | 7 + src/generatedPackageMetadata.ts | 2 +- .../boundary/mcp/environmentServeMcp.test.ts | 74 +++++++- tests/fixtures/environmentStateLaterReads.ts | 12 ++ tests/fixtures/environmentStateMadeValues.ts | 167 ++++++++++++++++++ 36 files changed, 1307 insertions(+), 51 deletions(-) create mode 100644 docs/specs/agent-environments/write-call-values/plan.md create mode 100644 docs/specs/agent-environments/write-call-values/spec.md create mode 100644 docs/specs/agent-environments/write-call-values/tasks.md create mode 100644 docs/specs/agent-environments/write-call-values/verification.md create mode 100644 src/application/environment/environmentQueryText.test.ts create mode 100644 src/application/environment/environmentStateMadeValuesAnswer.test.ts create mode 100644 src/domain/analysisView/environmentStateMadeValues.test.ts create mode 100644 tests/fixtures/environmentStateMadeValues.ts diff --git a/.agents/skills/reverse-engineer-anything/SKILL.md b/.agents/skills/reverse-engineer-anything/SKILL.md index 8d437b23a8..9de416eb0a 100644 --- a/.agents/skills/reverse-engineer-anything/SKILL.md +++ b/.agents/skills/reverse-engineer-anything/SKILL.md @@ -2,7 +2,7 @@ name: reverse-engineer-anything description: Reverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. metadata: - version: "52" + version: "53" --- # REA diff --git a/.agents/skills/reverse-engineer-anything/references/environments.md b/.agents/skills/reverse-engineer-anything/references/environments.md index 930b88b80a..f8b3ca69be 100644 --- a/.agents/skills/reverse-engineer-anything/references/environments.md +++ b/.agents/skills/reverse-engineer-anything/references/environments.md @@ -132,8 +132,10 @@ client with it. Match `wait_for` locators to exactly one element. as a secret reference), submit, then `wait_for` an element only a signed-in user sees. 2. Read with `request` steps: each collection the environment needs, twice - before any write, so fields that change on their own show up; one item of - each. + before any write, so fields that change on their own show up; then each + object a list shows by id, up to ten per collection. An object known only + from a list that leaves out fields its own read shows cannot be read by id + or updated in the environment. 3. Write the way the application's UI does: fill its forms and click its buttons, so each request carries the fields the UI sends, including ones the server would otherwise derive. Give the click or key press that submits diff --git a/docs/agent-environments.md b/docs/agent-environments.md index ebf784cd7b..b5f24d175e 100644 --- a/docs/agent-environments.md +++ b/docs/agent-environments.md @@ -406,7 +406,8 @@ returns that record's recorded response. Selection compares the supplied path parameters against the record's path segments at the parameter indexes, then the supplied query values against the record's query string, and keeps the record with the most matches; ties keep manifest order. A query argument accepts -one string or an array of strings for repeated values; the array is matched +one string, a number or boolean sent as its JSON text, or an array of strings for +repeated values; the array is matched against every occurrence in captured order. The raw recorded query remains the replay evidence. The result reports where the answer came from: @@ -613,8 +614,9 @@ shape and carries no annotations. Path parameters are required strings whose description names the segment index, the templating rule and the observed values (sampled within the list budget above; the manifest keeps them all); -query parameter names accept an optional string or array of strings (for -repeated values in captured order); write methods accept an +query parameter names accept an optional string, number or boolean (a number or +boolean is sent as its JSON text) or array of strings (for repeated values in +captured order); write methods accept an optional `body` object described by the recorded top-level key types. Each input keeps its own argument: a query name equal to a path parameter's name is advertised as `query_`, a query name longer than 64 characters as diff --git a/docs/specs/agent-environments/write-call-values/plan.md b/docs/specs/agent-environments/write-call-values/plan.md new file mode 100644 index 0000000000..b0a4da6b4d --- /dev/null +++ b/docs/specs/agent-environments/write-call-values/plan.md @@ -0,0 +1,13 @@ +# Implementation plan + +1. **Query values.** Widen the query property schema and spell numbers and booleans as JSON text where a call enters + serving. +2. **Sequence start.** `start` on sequence rules and their schema; the exemplar records integer fields above every held + value; serving and replay floor the next value at `start`. +3. **Later rules.** Merging a later create adopts its `composed`, `template` or started `sequence` rule when it spells + the exemplar's value. +4. **Made objects.** The `made-object` rule and schema; its explanation, merge, later-call check, counts taken, serving + and own members. +5. **Bulk link.** `respondedIds` reads the object of a one-element array member; `anyLinkedSource` lets a linked + create take any path values its plan stores in fields, in replay and serving. +6. **Skill, tests and docs.** diff --git a/docs/specs/agent-environments/write-call-values/spec.md b/docs/specs/agent-environments/write-call-values/spec.md new file mode 100644 index 0000000000..7ff8f1b88d --- /dev/null +++ b/docs/specs/agent-environments/write-call-values/spec.md @@ -0,0 +1,46 @@ +# Serve the calls agents make and the values applications count + +Issue: https://github.com/morluto/rea/issues/4933 + +## Outcome + +Eval agents' trajectories on rebuilt environments, and the write plans of the recordings they ran on, showed three +gaps REA could close: + +- **Query values.** Served tools typed each query parameter as a string or an array of strings, while agents send page + sizes and flags as JSON numbers and booleans (`maxSize: 50`, `withItems: true`). The MCP server refused those calls + before REA saw them: 63 of 418 stored eval runs hit it, in EspoCRM, Homebox and PhotoPrism, and 18 never got the tool + to answer. +- **Counted and made values.** Vikunja numbers a project's tasks past those deleted earlier, so a created task's + `index` is not one past what the state holds; its `identifier` spells that index; and each created task gets a + `subscription` object the application makes with it. A second create disagreed with the first on all three, so + task creates were withheld. +- **Bulk creates.** Vikunja's UI creates a task through `POST /projects/{id}/tasks/bulk`, sending and answering the + task wrapped in `tasks: [ … ]`. The answer's id sat inside the array, so REA linked the call to no type. + +The diagnosis also found that a recorded update stores only what it sent (EspoCRM's updates never changed an email +address), and that an object known only from a list that leaves out fields its own read shows can be neither read by +id nor updated. Both are recording gaps the skill now addresses. + +## Behaviour + +- **Query values.** A query parameter accepts a number or boolean besides a string or an array of strings, and serving + sends it as its JSON text. +- **Sequences from a start.** A field the first create showed above every value its type held, which a later create + shows one past the first's, is a `sequence` with that `start`: serving never goes below it. +- **Rules that spell the first create.** A field the plan kept as shown takes a later create's rule when that rule also + spells the first create's value: a `composed` or `template` string, or the `sequence` above. +- **Made objects.** A created object no other entity of the type holds otherwise, with a member the request stamped, is + a `made-object`: stamped members take the write's clock, members holding the entity's identity take its id, counts + go one past the largest value held objects show from a `start`, and other members stay as recorded. +- **Linked bulk creates.** A create's answer names the id of the one object a one-element array member holds. A linked + create accepts any path values when it creates into its type's own collection, whose path has no parameters, and + stores each path value in a field, as a Vikunja task stores its project: no path value changes where it creates. +- **Skill.** Recordings read each listed object by id, up to ten per collection, before writing. + +Plans and views that need none of this are unchanged. + +## Acceptance + +- No write tool of the 130 recorded captures goes from served to withheld. +- Rebuilt from the same recordings, the branch passes at least the write tasks #7274 passes. diff --git a/docs/specs/agent-environments/write-call-values/tasks.md b/docs/specs/agent-environments/write-call-values/tasks.md new file mode 100644 index 0000000000..6ceed72189 --- /dev/null +++ b/docs/specs/agent-environments/write-call-values/tasks.md @@ -0,0 +1,13 @@ +# Tasks + +- [x] Accept numbers and booleans for query parameters. +- [x] Start a sequence from the value its first create took. +- [x] Adopt a later create's rule that spells the first create's value. +- [x] Learn, check and serve objects made with an entity. +- [x] Link creates whose answer wraps the entity in a one-element array, and let a linked create take path values + it stores. +- [x] Ask recordings to read each listed object by id. +- [x] Add view, serving, MCP boundary and unit tests. +- [x] Document the rules and the query argument types. +- [x] Measure the recorded captures and served write tasks. +- [x] Run static, formatting, documentation and test gates. diff --git a/docs/specs/agent-environments/write-call-values/verification.md b/docs/specs/agent-environments/write-call-values/verification.md new file mode 100644 index 0000000000..a2aad7b67f --- /dev/null +++ b/docs/specs/agent-environments/write-call-values/verification.md @@ -0,0 +1,88 @@ +# Verification + +Executed on macOS with Node v24.22.1 on this branch merged with #7274 at `45832406` (main `8706068b`), and on Ubuntu +24.04 VMs for the task runs. + +## Diagnosis + +The 418 eval runs stored on the comparison VMs, with their agent trajectories: + +- 63 runs had a call refused before REA saw it because a query value came as a JSON number or boolean (`maxSize`, + `pageSize`, `withItems`, `isLocation`, PhotoPrism's `count`, `offset`, `merged`, `public`), and in 18 the agent never + got that tool to answer. +- EspoCRM's contact update fails in every run: the served update accepts only the members recorded updates stored + (`description`, `name`, `lastName`), and a seeded contact known only from lists can be neither read by id nor + answered for. On #7274, gpt-oss passes the task by creating a replacement contact and deleting the original. +- Vikunja's task creates were withheld by an unexplained `index`, `identifier` and `subscription` (main's recording), + or had no state target, being bulk creates (four recordings). + +## Gates + +- `npm run -s check:fast`, `format:check`, `scan:todos`, `docs:check`, `verify:test-discovery`, `mcp-catalog:check` and + `jscpd`: passed. `npx knip` reports `staticFileUrlPath` and `ObjcClassImportBindingCoverage`, which main `8706068b` + added and this branch does not touch. +- `npx vitest run --project domain --project services`: 11,430 passed, 6 skipped. `--project adapters`: 7,437 passed, 19 + skipped, 3 failed: the Inspector cancellation test and two AppKit asset-catalog tests, which fail the same way on + #7274 at `e29e9c74`. +- `npm run -s test:focused --` the environment build, export and serve CLI boundaries, the analysis-view CLI and MCP + boundaries and the environment serve MCP boundary: passed. `outputSchemaProfile`, `toolSchemaValidity`, + `mcpSchemaValidation` and `outputSchemaSemanticConstraints`: passed; the compact catalog stays under its frame. + +## Tests + +- View: two task creates at a project path learn `index` as a `sequence` from 15, `identifier` as a `template` around + it and `subscription` as a `made-object` whose `id` counts from 21 and whose `entity_id` holds the task's own id; one + create alone keeps `index` as recorded; a bulk create whose body and answer wrap the task in `tasks` links to the + tasks type and is served; an object with no member the request stamped makes no rule. +- Serving: created tasks take index 15 then 16, `#15` then `#16`, and a subscription numbered 22 holding the second + task's id; a bulk create answers in its envelope, and one into another project stores that project. +- Query values: a number or boolean query value is spelled as its JSON text, and a list paged by a number answers as + with the same text; through the MCP server, `maxSize: 2` and `withItems: true` replay their recording exactly. The + served card of the `renamed` fixture grows 637 bytes, 91 for each of its seven query parameters. +- The seven behaviour tests fail on #7274 at `4ede1305`. + +## Write plans and views over recorded captures + +The 130 captures #7274 measured, built by #7274 at `45832406` and by this branch: 615 plans, 332 served and 333. + +- No write goes from served to withheld. Newly served: Vikunja's bulk task create in #7260's final recording. +- Vikunja's task create in main's recording is now explained, by its counter, template and made subscription, but + main's `unknown-clock` withholds it in both builds, as that capture has no clock anchor. Measured on #7274 at + `4ede1305`, before that rule, it was served too (339 to 341). +- Six plans that stay withheld explain more: Homebox entity creates learn `assetId` as a counter from 5 in four + recordings, and Vikunja task creates in two. +- In #7260's two Vikunja recordings whose UI writes lost their bodies, the bulk create now counts as a task write, so + the scratch task it made is no longer held as seeded (35 to 34 and 36 to 35 initial entities), a read tool that read + only that task replays its recordings, and later reads check fewer lines. Every other view is unchanged. + +## The same recordings, built by #7274 and by this branch + +Claude Code (`claude-sonnet-5-5`) recordings from earlier rounds, rebuilt by #7274 at `e839af3f` (its last two merges +change no census) and by this branch, their write tasks run with `fireworks_ai/gpt-oss-120b` then +`fireworks_ai/glm-5p3-flash`, each checked from the environment's state: + +| Recording | Write tasks | #7274 | This branch | +| ---------------------------------------------------------- | ----------- | ----------- | ----------- | +| Vikunja: #7260 final (bulk creates) | 3 + 2 | 1/5 1/5 | 2/5 2/5 | +| Vikunja: main (two), saved-values (two), #7260 (first two) | 6 × (3 + 2) | 6/30 6/30 | 6/30 6/30 | +| EspoCRM: main, #7260 (three), saved-values | 5 × 3 | 15/15 10/15 | 15/15 10/15 | +| Homebox: main, #7260 final, saved-values (two) | 4 × 2 | 6/8 6/8 | 6/8 6/8 | +| Total | 58 | 28/58 23/58 | 29/58 24/58 | + +No recording passes fewer tasks with this branch. + +- **Vikunja, bulk creates:** the create task passes with both models. The served bulk create numbers the task and makes + its subscription, and creates into Lighthouse Rota though the recording created into the inbox. +- **Refused calls:** agents' calls were refused at input validation 34 times with #7274 (30 in Vikunja) and once with + this branch, and agents made 286 tool calls against 317. +- **Main's `unknown-clock`** withholds Vikunja's task create and Homebox's tag create in main's recordings in both + builds, so those tasks fail in both. Rebuilt by #7274 at `4ede1305`, before that rule, this branch without the + linked-path change made main's Vikunja create task pass with both models (1/3 to 2/3) and changed no other result + among the recordings that finished. + +## Not verified + +- A fresh recording with skill 52, which reads each listed object by id. +- Path parameters given as numbers stay refused: they are required properties, which REA keeps free of combinators. +- A counter one create alone shows, and linked creates into a collection whose path has parameters, which still take + only their recorded path values. diff --git a/docs/web-network-captures.md b/docs/web-network-captures.md index 1eeb5a62c5..b0cf4c396b 100644 --- a/docs/web-network-captures.md +++ b/docs/web-network-captures.md @@ -733,8 +733,12 @@ order. Its `effect` is `create` for a `POST` to a type's plus one trailing parameter, or to that item path plus one action segment, `delete` for a `DELETE` to it, `document` for a `PUT`, `PATCH` or `POST` to a document's path, an `update` or `delete` of each entity a write at the -collection names by id in its body, and otherwise `none` with reason -`no-state-target`. A plan is built from the tool's first successful call: its +collection names by id in its body, a `create` for a `POST` whose answer names +an id (at its top level, in an object member or in the one object of a +one-element array member) that a later item read of a type first shows (served +only with the path values its recorded call sent, unless it creates into the +type's own collection, whose path has no parameters, and a `path` rule stores +each of its path values), and otherwise `none` with reason `no-state-target`. A plan is built from the tool's first successful call: its request body must be a usable JSON object (as response bodies are usable), and the written entity must show in that call's response or in a later read before the next write. Each field of a created entity gets the first rule that @@ -743,10 +747,14 @@ value's shape), `echo` of a body member (`sorted` when the application stored its strings in code-point order rather than as sent), `copy` of a body string of 8 or more characters, a `path` value, a `slug` of a body member, a `clock` value within 10 minutes of the request in its recorded format, a `sequence` one past the -largest value of its collection or type, a `unique` UUID, opaque token or +largest value of its collection or type, and never below its `start`, a `unique` UUID, opaque token or integer of 10 or more digits that no entity of the type holds (where every one holds a distinct integer of the same length, as time-based ids do), a -`nested-clock` object whose clocks the request stamped, a `template` around +`nested-clock` object whose clocks the request stamped, a `made-object` no +other entity of the type holds otherwise, whose `clocks` the request stamped and +whose other members hold the entity's own identity (`own`), one past the largest +value held objects show (`counts`, each from a `start`), or what every held +object shows, a `template` around a generated value, a `reference` to an entity, or a `composed` string; other fields keep the observed value. When the first read of a created entity before the next write shows fields the create's answer lacks, its clocks to another @@ -781,7 +789,13 @@ when, explained on its own, it agrees with it: both calls show the same fields but for a reference whose member a call without it did not send (one an earlier create added stays), every field both rule they rule alike (an echo one showed sorted stays sorted, unless a call stored its strings out of order as sent), -and a field neither rules holds one value. Members either call sent are accepted, only those every call sent are +and a field neither rules holds one value, unless the later call's rule for it +spells the earlier calls' value too: a `composed` or `template` string, or a +`sequence` started at that value (`start`), as Vikunja numbers a project's tasks +past those deleted before the capture. Two `made-object` rules agree in their +clocks and own members, and a count only the later call shows starts at the +first object's value; a later create's made object must hold clocks its own +request stamped, in the rule's format. Members either call sent are accepted, only those every call sent are required, and an echoed member a call left out takes the value that call stored (`body.defaults`). A member a create sent empty (`""`, `null`, `false`, `[]` or `{}`), as a form sends a field left as it was, is not required: a call that diff --git a/src/application/environment/environmentQueryText.test.ts b/src/application/environment/environmentQueryText.test.ts new file mode 100644 index 0000000000..1e52df3078 --- /dev/null +++ b/src/application/environment/environmentQueryText.test.ts @@ -0,0 +1,40 @@ +import { describe, expect, it } from "vitest"; + +import { MAX_SIZE_ROWS } from "../../../tests/fixtures/environmentStateLaterReads.js"; +import { servedFrom } from "../../../tests/fixtures/environmentStateServing.js"; +import { withQueryText } from "./environmentTools.js"; + +const contacts = () => servedFrom(MAX_SIZE_ROWS); + +const listTool = (served: ReturnType) => { + const tool = served.source.manifest.tools.find( + (candidate) => candidate.name === "get_contacts", + ); + if (tool === undefined) throw new Error("The manifest lacks get_contacts"); + return tool; +}; + +describe("query values given as numbers or booleans", () => { + it("spells a number or boolean query value as its JSON text and leaves others as given", () => { + const tool = listTool(contacts()); + const args = { + maxSize: 2, + offset: false, + body: { maxSize: 3 }, + }; + expect(withQueryText(tool, args)).toEqual({ + maxSize: "2", + offset: "false", + body: { maxSize: 3 }, + }); + const strings = { maxSize: "2", offset: ["0"] }; + expect(withQueryText(tool, strings)).toBe(strings); + }); + + it("answers a list paged by a number as it answers the same page size given as text", () => { + const served = contacts(); + expect(served.answer("get_contacts", { maxSize: 2, offset: 0 })).toEqual( + served.answer("get_contacts", { maxSize: "2", offset: "0" }), + ); + }); +}); diff --git a/src/application/environment/environmentStateLaterReadsAnswer.test.ts b/src/application/environment/environmentStateLaterReadsAnswer.test.ts index f1c12be2c3..78fa050912 100644 --- a/src/application/environment/environmentStateLaterReadsAnswer.test.ts +++ b/src/application/environment/environmentStateLaterReadsAnswer.test.ts @@ -1,6 +1,7 @@ import { describe, expect, it } from "vitest"; import { + INITIAL_TRASH_ROWS, LATER_BULK_ROWS, LATER_FIELD_ROWS, OMITTED_FLAG_ROWS, @@ -26,6 +27,12 @@ const names = (body: JsonValue | undefined): JsonValue[] => { }; describe("served writes modelled on later reads", () => { + it("replays the recorded list when it includes a deleted row", () => { + const contacts = servedFrom(INITIAL_TRASH_ROWS); + expect( + contacts.answer("get_contacts", { maxSize: "20", offset: "0" }).body, + ).toEqual(INITIAL_TRASH_ROWS[0]![2]); + }); it("keeps an initially deleted item readable without treating it as live", () => { const deleted = { id: 4, name: "Gone", deleted: true }; const rows = [ diff --git a/src/application/environment/environmentStateMadeValuesAnswer.test.ts b/src/application/environment/environmentStateMadeValuesAnswer.test.ts new file mode 100644 index 0000000000..f2dd88ce6b --- /dev/null +++ b/src/application/environment/environmentStateMadeValuesAnswer.test.ts @@ -0,0 +1,77 @@ +import { describe, expect, it } from "vitest"; + +import { + BULK_TASK_ROWS, + CHOSEN_ID_TASK_ROWS, + COUNTED_TASK_ROWS, +} from "../../../tests/fixtures/environmentStateMadeValues.js"; +import { servedFrom } from "../../../tests/fixtures/environmentStateServing.js"; +import { isObject } from "../../domain/analysisView/environmentStateRules.js"; +import type { JsonValue } from "../../domain/jsonValue.js"; + +const objectOf = (body: JsonValue | undefined) => { + if (!isObject(body)) throw new Error("Expected an object body"); + return body; +}; + +describe("served creates that count and make values", () => { + it("numbers created tasks on from the recorded counter and makes each one's subscription", () => { + const tasks = servedFrom(COUNTED_TASK_ROWS); + const first = objectOf( + tasks.answer("post_projects_tasks_by_id", { + id: "5", + body: { title: "Restock the fog-horn spares" }, + }).body, + ); + const second = objectOf( + tasks.answer("post_projects_tasks_by_id", { + id: "5", + body: { title: "Paint the gallery rail" }, + }).body, + ); + expect([first["index"], first["identifier"]]).toEqual([15, "#15"]); + expect([second["index"], second["identifier"]]).toEqual([16, "#16"]); + const read = objectOf( + tasks.answer("get_tasks_by_id", { id: String(second["id"]) }).body, + ); + expect(read["subscription"]).toMatchObject({ + id: 22, + entity: "task", + entity_id: second["id"], + }); + }); + + it("links a made object to the id the caller chose", () => { + const tasks = servedFrom(CHOSEN_ID_TASK_ROWS); + tasks.answer("post_projects_tasks_by_id", { + id: "5", + body: { id: 99, title: "Restock the fog-horn spares" }, + }); + const read = objectOf(tasks.answer("get_tasks_by_id", { id: "99" }).body); + expect(read["subscription"]).toMatchObject({ entity_id: 99 }); + }); + + it("answers a bulk create in the envelope it was sent in", () => { + const tasks = servedFrom(BULK_TASK_ROWS); + const body = objectOf( + tasks.answer("post_projects_tasks_bulk_by_id", { + id: "5", + body: { tasks: [{ title: "Restock the fog-horn spares" }] }, + }).body, + ); + expect(body["tasks"]).toMatchObject([ + { title: "Restock the fog-horn spares", index: 15, identifier: "#15" }, + ]); + }); + + it("creates into another project when the task stores its project and lands in one collection", () => { + const tasks = servedFrom(BULK_TASK_ROWS); + const body = objectOf( + tasks.answer("post_projects_tasks_bulk_by_id", { + id: "7", + body: { tasks: [{ title: "Paint the gallery rail" }] }, + }).body, + ); + expect(body["tasks"]).toMatchObject([{ project_id: 7 }]); + }); +}); diff --git a/src/application/environment/environmentStateWriteAnswer.ts b/src/application/environment/environmentStateWriteAnswer.ts index 0be2912e81..fb997d84f0 100644 --- a/src/application/environment/environmentStateWriteAnswer.ts +++ b/src/application/environment/environmentStateWriteAnswer.ts @@ -18,10 +18,14 @@ import { } from "../../domain/analysisView/environmentStateWriteExplain.js"; import { callRenewal } from "../../domain/analysisView/environmentStateWriteGenerated.js"; import { + heldObjects, + madeObjectValue, nestedClockValue, + nextMadeCount, unenveloped, } from "../../domain/analysisView/environmentStateWriteStamps.js"; import { + anyLinkedSource, linkedCollection, sameLinkedSource, } from "../../domain/analysisView/environmentStateWriteLinks.js"; @@ -38,6 +42,7 @@ import { seededInteger, seededLike, spelledLike, + startedSequence, } from "../../domain/analysisView/environmentStateWriteValues.js"; import { at, @@ -287,16 +292,41 @@ export const generator = ( exemplar, sequence( `${request.tool.name} ${rule.field} ${rule.scope === "collection" ? target.name : ""}`, - nextHeld( - rule.scope === "collection" - ? [...target.collection, ...startedIn(target.name)] - : [...ofType, ...started], - rule.field, - exemplar, + startedSequence( + nextHeld( + rule.scope === "collection" + ? [...target.collection, ...startedIn(target.name)] + : [...ofType, ...started], + rule.field, + exemplar, + ), + rule.start, ), ), ), ); + case "made-object": { + const clock = formatClock(write.clockMs, rule.format); + if (clock === undefined) + return err( + `The generated clock for ${rule.field} is outside its recorded four-digit year format.`, + ); + const base = ownMember(target.observed, rule.field); + const held = heldObjects( + [...ofType, ...started].map((entity) => entity.fields), + rule.field, + ); + return madeObjectValue(base, rule, clock, (count) => + sequence( + `${request.tool.name} ${rule.field}/${count.member}`, + nextMadeCount( + held, + count, + isObject(base) ? ownMember(base, count.member) : undefined, + ), + ), + ); + } case "unique": return unused(rule.field, (attempt) => generatedIntegerDigits(exemplar) === undefined @@ -438,12 +468,13 @@ const writtenCollection = ( return ok( createdCollection(serving, tool, request.args, plan.target ?? ""), ); - // No read showed where other path arguments create. - return sameLinkedSource( - tool.path_template, - exemplarPath(serving, plan), - pathArguments(tool, request.args), - ) + // No read showed where other path arguments create, unless they cannot change it. + return anyLinkedSource(plan, tool.path_template) || + sameLinkedSource( + tool.path_template, + exemplarPath(serving, plan), + pathArguments(tool, request.args), + ) ? ok(linked) : err( refused( @@ -455,7 +486,7 @@ const writtenCollection = ( /** * The path values a create's path fields take, decoded once as the gate learned them: a linked - * create accepts its exemplar's source only, so it takes that; another the collection this call + * create that accepts its exemplar's source only takes that; another the collection this call * selects, else its own arguments. */ const createdPathValues = ( @@ -466,7 +497,8 @@ const createdPathValues = ( ): ReadonlyMap => { const { tool, plan } = request; const exemplar = - linkedCollection(plan, tool.path_template) === undefined + linkedCollection(plan, tool.path_template) === undefined || + anyLinkedSource(plan, tool.path_template) ? undefined : exemplarPath(serving, plan); return ( @@ -521,6 +553,7 @@ const answerCreate = ( created, values.forTarget({ name: collection, collection: members, observed }), heldEntity(serving), + idField, ); if (!fields.ok) return err(refused(tool, fields.error)); const field = unloadableMember(fields.value); diff --git a/src/application/environment/environmentToolAnswer.ts b/src/application/environment/environmentToolAnswer.ts index 241f00f248..db1b35db3a 100644 --- a/src/application/environment/environmentToolAnswer.ts +++ b/src/application/environment/environmentToolAnswer.ts @@ -15,6 +15,7 @@ import { answerBulk } from "./environmentStateBulkAnswer.js"; import type { EnvironmentStateStore } from "./environmentStateStore.js"; import { answerWrite } from "./environmentStateWriteAnswer.js"; import { pinnedRequest } from "./environmentStateWriteQuery.js"; +import { withQueryText } from "./environmentTools.js"; /** How a served environment answers one tool call. */ export type EnvironmentToolAnswer = ( @@ -30,8 +31,14 @@ export const environmentToolAnswer = ( ): EnvironmentToolAnswer => store === undefined ? (tool, args, observedQuery) => - replayEnvironmentTool(source, tool, args, observedQuery) - : (tool, args, observedQuery) => { + replayEnvironmentTool( + source, + tool, + withQueryText(tool, args), + observedQuery, + ) + : (tool, given, observedQuery) => { + const args = withQueryText(tool, given); const plan = store.state.writes.get(tool.name); if (plan?.served !== true) return answerFromState({ source, store }, tool, args, observedQuery); diff --git a/src/application/environment/environmentTools.test.ts b/src/application/environment/environmentTools.test.ts index 85ae00b887..25dd8d4572 100644 --- a/src/application/environment/environmentTools.test.ts +++ b/src/application/environment/environmentTools.test.ts @@ -92,9 +92,14 @@ describe("environmentToolDefinitions", () => { expect(search.additionalProperties).toBe(false); expect(Object.keys(search.properties).sort()).toEqual(["query", "type"]); expect(search.properties.query).toEqual({ - oneOf: [{ type: "string" }, { type: "array", items: { type: "string" } }], + oneOf: [ + { type: "string" }, + { type: "number" }, + { type: "boolean" }, + { type: "array", items: { type: "string" } }, + ], description: - "Query parameter query; optional. Supply one string for one occurrence or an array to preserve repeated values in captured order; repeated values replay their recorded request.", + "Query parameter query; optional. Supply one value for one occurrence, a number or boolean sent as its JSON text, or an array of strings to preserve repeated values in captured order; repeated values replay their recorded request.", }); expect("body" in search.properties).toBe(false); }); @@ -522,10 +527,12 @@ describe("environmentToolInputSchema argument names", () => { ]); expect(Object.hasOwn(schema.properties, "constructor")).toBe(true); expect(schema.properties.query_id?.description).toBe( - "Query parameter id; optional. Supply one string for one occurrence or an array to preserve repeated values in captured order; repeated values replay their recorded request.", + "Query parameter id; optional. Supply one value for one occurrence, a number or boolean sent as its JSON text, or an array of strings to preserve repeated values in captured order; repeated values replay their recorded request.", ); expect(schema.properties.body?.oneOf).toEqual([ { type: "string" }, + { type: "number" }, + { type: "boolean" }, { type: "array", items: { type: "string" } }, ]); expect(schema.properties.request_body?.type).toBe("object"); diff --git a/src/application/environment/environmentTools.ts b/src/application/environment/environmentTools.ts index f334d9b503..62df5fda92 100644 --- a/src/application/environment/environmentTools.ts +++ b/src/application/environment/environmentTools.ts @@ -32,7 +32,7 @@ export const REPLAY_CATALOG_INSTRUCTIONS = /** JSON Schema advertised for one tool argument. */ export interface EnvironmentToolProperty extends EnvironmentToolSchema { readonly oneOf?: readonly ( - | { readonly type: "string" } + | { readonly type: "string" | "number" | "boolean" } | { readonly type: "array"; readonly items: { readonly type: "string" } } )[]; readonly description: string; @@ -325,6 +325,21 @@ export const hasRepeatedQueryArgument = ( return false; }; +/** The arguments with a number or boolean given for a query parameter spelled as its JSON text, as a URL carries it. */ +export const withQueryText = ( + tool: EnvironmentManifestTool, + args: Readonly>, +): Readonly> => { + let spelled: Record | undefined; + for (const argument of environmentToolArguments(tool).query.values()) { + const value = ownMember(args, argument); + if (typeof value !== "number" && typeof value !== "boolean") continue; + spelled ??= { ...args }; + spelled[argument] = JSON.stringify(value); + } + return spelled ?? args; +}; + /** Lazily render properties so admission precedes retaining a whole schema. */ function* environmentToolProperties( tool: EnvironmentManifestTool, @@ -351,9 +366,11 @@ function* environmentToolProperties( ? { oneOf: [ { type: "string" }, + { type: "number" }, + { type: "boolean" }, { type: "array", items: { type: "string" } }, ], - description: `Query parameter ${describeText(name, "query name")}; optional. Supply one string for one occurrence or an array to preserve repeated values in captured order; repeated values replay their recorded request.`, + description: `Query parameter ${describeText(name, "query name")}; optional. Supply one value for one occurrence, a number or boolean sent as its JSON text, or an array of strings to preserve repeated values in captured order; repeated values replay their recorded request.`, } : { type: "string", diff --git a/src/domain/analysisView/EnvironmentStateExtraction.ts b/src/domain/analysisView/EnvironmentStateExtraction.ts index 4a31b44ee7..6a108cd5c0 100644 --- a/src/domain/analysisView/EnvironmentStateExtraction.ts +++ b/src/domain/analysisView/EnvironmentStateExtraction.ts @@ -20,6 +20,7 @@ import { lastSegmentValue, STATE_BODY_LIMIT_BYTES, listItems, + markedDeleted, locateItem, locateList, projectsFields, @@ -862,7 +863,9 @@ export class EnvironmentStateExtraction { if (this.#collections === undefined) { this.#retain(ENTRY_BYTES * (this.#merged.size + 1)); this.#collections = new Map(); + // List serving selects live entities; item observations retain deleted entities separately. for (const entity of this.#merged.values()) { + if (markedDeleted(entity.fields)) continue; const key = JSON.stringify([entity.type, entity.collection]); const members = this.#collections.get(key); if (members === undefined) this.#collections.set(key, [entity]); diff --git a/src/domain/analysisView/EnvironmentStateWriteSimulation.ts b/src/domain/analysisView/EnvironmentStateWriteSimulation.ts index 2a16eb22b1..597e4d0ca3 100644 --- a/src/domain/analysisView/EnvironmentStateWriteSimulation.ts +++ b/src/domain/analysisView/EnvironmentStateWriteSimulation.ts @@ -295,6 +295,8 @@ export class EnvironmentStateWriteSimulation { calls: 0, updates: [], sequences: new Map(), + starts: new Map(), + createCollections: [], steady: new Set(), unstamped: new Set(), kept: new Set(), diff --git a/src/domain/analysisView/environmentStateCreateCalls.test.ts b/src/domain/analysisView/environmentStateCreateCalls.test.ts index 198d8bc78a..26aa4c3f3e 100644 --- a/src/domain/analysisView/environmentStateCreateCalls.test.ts +++ b/src/domain/analysisView/environmentStateCreateCalls.test.ts @@ -215,3 +215,52 @@ describe("merging a later create into a plan", () => { ).toBeUndefined(); }); }); + +describe("merging a later create's made object", () => { + const made = (counts: { member: string; start: number }[]): FieldRule => ({ + field: "subscription", + rule: "made-object", + clocks: ["created"], + format: { kind: "iso", separator: "T", fraction: 0, zone: "Z" }, + own: ["entity_id"], + counts, + }); + const body = { + accepted: ["title"], + fixed: [], + required: ["title"], + identity: null, + kinds: [], + }; + const title: FieldRule = { field: "title", rule: "echo" }; + const subscription = (id: number, task: number) => ({ + id, + entity: "task", + entity_id: task, + created: "2026-10-26T10:00:00Z", + }); + const plan = { rules: [title, made([])], body }; + const exemplar = { title: "First", subscription: subscription(31, 20) }; + const later = { + entity: { title: "Third", subscription: subscription(32, 22) }, + body: { title: "Third" }, + rules: [title, made([{ member: "id", start: 32 }])], + contract: body, + }; + const earlier = (accepted: number) => ({ + kept: new Set(), + clockOf: () => undefined, + createCollections: Array.from({ length: accepted }, () => "/api/tasks"), + }); + + it("counts a member only the later create shows while the exemplar is the one create accepted", () => { + expect( + mergedCreatePlan(plan, exemplar, later, earlier(1))?.rules, + ).toContainEqual(made([{ member: "id", start: 31 }])); + }); + + it("makes no count of a member creates past the exemplar held alike", () => { + // Creates of 31 and 31 then 32 are no count: serving would make 31, 32, 33. + expect(mergedCreatePlan(plan, exemplar, later, earlier(2))).toBeUndefined(); + }); +}); diff --git a/src/domain/analysisView/environmentStateGate.ts b/src/domain/analysisView/environmentStateGate.ts index fb059287d5..4e52a7c01a 100644 --- a/src/domain/analysisView/environmentStateGate.ts +++ b/src/domain/analysisView/environmentStateGate.ts @@ -8,6 +8,7 @@ import { isObject, lastSegmentValue, listItems, + markedDeleted, renderStateEntity, renderStateList, same, @@ -359,7 +360,9 @@ const listReproduces = ( collection, scalarText(ownMember(item, field)) ?? "", ); - return entity === undefined ? [] : [entity]; + return entity === undefined || markedDeleted(entity.fields) + ? [] + : [entity]; })); const rendered = members.map((entity) => { const item = observed.get(entity.id); diff --git a/src/domain/analysisView/environmentStateLaterReads.test.ts b/src/domain/analysisView/environmentStateLaterReads.test.ts index 1771e4abe8..c7a4ff3d74 100644 --- a/src/domain/analysisView/environmentStateLaterReads.test.ts +++ b/src/domain/analysisView/environmentStateLaterReads.test.ts @@ -6,6 +6,7 @@ import { } from "../../../tests/fixtures/analysisView.js"; import { GHOST_DELETED_ROWS, + INITIAL_TRASH_ROWS, LATER_FIELD_ROWS, LATER_MIXED_BULK_ROWS, MAX_SIZE_ROWS, @@ -25,6 +26,13 @@ const planOf = (rows: readonly TimedCaptureRow[], tool: string) => { }; describe("deletes the application answers for afterwards", () => { + it("replays an initial list that includes a deleted entity", () => { + expect( + viewOf(INITIAL_TRASH_ROWS).views.find( + (view) => view.tool === "get_contacts", + ), + ).toMatchObject({ shape: "recorded", reason: "list-differs" }); + }); it("confirms a delete by an item read that shows the entity marked deleted", () => { expect(planOf(SOFT_DELETED_ROWS, "delete_contacts_by_id")).toMatchObject({ served: true, diff --git a/src/domain/analysisView/environmentStateMadeValues.test.ts b/src/domain/analysisView/environmentStateMadeValues.test.ts new file mode 100644 index 0000000000..66375613c8 --- /dev/null +++ b/src/domain/analysisView/environmentStateMadeValues.test.ts @@ -0,0 +1,143 @@ +import { describe, expect, it } from "vitest"; + +import { + timedCaptureEvidence, + type TimedCaptureRow, +} from "../../../tests/fixtures/analysisView.js"; +import { + BULK_TASK_ROWS, + COUNTED_TASK_ROWS, + MISLINKED_TASK_ROWS, + ONE_TASK_ROWS, + RESPELLED_TASK_ROWS, + STALE_MADE_TASK_ROWS, + UNCLOCKED_TASK_ROWS, +} from "../../../tests/fixtures/environmentStateMadeValues.js"; +import { environmentStateViewOf } from "../../../tests/fixtures/environmentStateWrites.js"; +import { respondedIds } from "./environmentStateWriteLinks.js"; +import { createdMadeObject } from "./environmentStateWriteStamps.js"; + +const createOf = (rows: readonly TimedCaptureRow[]) => { + const plan = environmentStateViewOf(timedCaptureEvidence(rows)).writes.find( + (candidate) => candidate.effect === "create", + ); + if (plan === undefined) throw new Error("No create plan"); + return plan; +}; + +const ruleFor = (plan: ReturnType, field: string): unknown => + plan.rules.find((rule) => rule.field === field); + +describe("values an application counts and makes with each created entity", () => { + it("does not adopt a sequence that contradicts an earlier accepted create", () => { + const a = { id: 1, title: "A", index: 1 }; + const b = { id: 2, title: "B", index: 2 }; + const c = { id: 3, title: "C", index: 15 }; + const d = { id: 4, title: "D", index: 15 }; + const e = { id: 5, title: "E", index: 16 }; + const rows: readonly TimedCaptureRow[] = [ + ["GET", "/api/tasks", [a, b]], + ["GET", "/api/tasks/1", a], + ["POST", "/api/tasks", c, 201, { title: "C" }], + ["GET", "/api/tasks/3", c], + ["GET", "/api/tasks", [a, b, c]], + ["POST", "/api/tasks", d, 201, { title: "D" }], + ["GET", "/api/tasks/4", d], + ["GET", "/api/tasks", [a, b, c, d]], + ["POST", "/api/tasks", e, 201, { title: "E" }], + ["GET", "/api/tasks/5", e], + ["GET", "/api/tasks", [a, b, c, d, e]], + ]; + const plan = createOf(rows); + expect(plan).toMatchObject({ served: false }); + expect(ruleFor(plan, "index")).toBeUndefined(); + }); + it("learns a counter from where the first create stood, a template around it and the object made with each task", () => { + const plan = createOf(COUNTED_TASK_ROWS); + expect(plan).toMatchObject({ + served: true, + checked_record_ordinals: [4, 7], + }); + expect(ruleFor(plan, "index")).toEqual({ + field: "index", + rule: "sequence", + scope: "collection", + start: 15, + }); + expect(ruleFor(plan, "identifier")).toEqual({ + field: "identifier", + rule: "template", + from: "index", + }); + expect(ruleFor(plan, "subscription")).toMatchObject({ + rule: "made-object", + clocks: ["created"], + own: ["entity_id"], + counts: [{ member: "id", start: 21 }], + }); + }); + + it("keeps a value one create alone shows past what the type held", () => { + const plan = createOf(ONE_TASK_ROWS); + expect(ruleFor(plan, "index")).toBeUndefined(); + expect(ruleFor(plan, "subscription")).toMatchObject({ + rule: "made-object", + counts: [], + }); + }); + + it("links a create whose answer wraps the entity in a one-element array member", () => { + expect(createOf(BULK_TASK_ROWS)).toMatchObject({ + tool: "post_projects_tasks_bulk_by_id", + served: true, + body: { envelope: "tasks" }, + }); + expect(respondedIds({ tasks: [{ id: 14 }] })).toEqual(["14"]); + expect(respondedIds({ tasks: [{ id: 14 }, { id: 15 }] })).toEqual([]); + }); + + it("withholds a create whose only clock is its made object's when the capture has no clock anchor", () => { + expect(createOf(UNCLOCKED_TASK_ROWS)).toMatchObject({ served: true }); + expect( + createOf([ + ...UNCLOCKED_TASK_ROWS, + ["GET", "/api/health", { ready: true }, 200, undefined, -1], + ]), + ).toMatchObject({ served: false, reason: "unknown-clock" }); + }); + + it("withholds the creates when a later made object names another entity", () => { + expect(createOf(MISLINKED_TASK_ROWS)).toMatchObject({ + served: false, + reason: "unexplained-change", + }); + }); + + it("adopts no template that would not spell the later create's value", () => { + const plan = createOf(RESPELLED_TASK_ROWS); + expect(plan).toMatchObject({ served: false, reason: "unexplained-change" }); + expect(ruleFor(plan, "identifier")).toBeUndefined(); + }); + + it("withholds the creates when a later made object holds a clock its request did not stamp", () => { + expect(createOf(STALE_MADE_TASK_ROWS)).toMatchObject({ + served: false, + reason: "unexplained-change", + }); + }); + + it("makes no object rule without a clock the request stamped", () => { + expect( + createdMadeObject( + "subscription", + { id: 21, entity: "task", entity_id: 20 }, + { + startMs: Date.parse("2026-10-17T18:00:02Z"), + type: [], + entity: { id: 20 }, + idField: "id", + }, + ), + ).toBeUndefined(); + }); +}); diff --git a/src/domain/analysisView/environmentStateView.ts b/src/domain/analysisView/environmentStateView.ts index a1b7d232b2..aa98efe792 100644 --- a/src/domain/analysisView/environmentStateView.ts +++ b/src/domain/analysisView/environmentStateView.ts @@ -245,6 +245,7 @@ const fieldRuleSchema = z.discriminatedUnion("rule", [ ...ruled, rule: z.literal("sequence"), scope: z.enum(["collection", "type"]), + start: z.number().int().exactOptional(), }), z.strictObject({ ...ruled, rule: z.literal("unique") }), z.strictObject({ @@ -263,6 +264,16 @@ const fieldRuleSchema = z.discriminatedUnion("rule", [ members: z.array(z.string()).min(1), format: clockFormatSchema, }), + z.strictObject({ + ...ruled, + rule: z.literal("made-object"), + clocks: z.array(z.string()).min(1), + format: clockFormatSchema, + own: z.array(z.string()), + counts: z.array( + z.strictObject({ member: z.string(), start: z.number().int() }), + ), + }), z.strictObject({ ...ruled, rule: z.literal("held-clock"), diff --git a/src/domain/analysisView/environmentStateWriteApply.test.ts b/src/domain/analysisView/environmentStateWriteApply.test.ts index d8d65d7149..a4fc8133f1 100644 --- a/src/domain/analysisView/environmentStateWriteApply.test.ts +++ b/src/domain/analysisView/environmentStateWriteApply.test.ts @@ -51,6 +51,7 @@ const generator = () => { clock: NOW, "nested-clock": { at: NOW }, "held-clock": NOW, + "made-object": { at: NOW }, }; const generate = (rule: GeneratedRule) => { asked.push(rule); diff --git a/src/domain/analysisView/environmentStateWriteApply.ts b/src/domain/analysisView/environmentStateWriteApply.ts index 399d84f1b6..05a93adefe 100644 --- a/src/domain/analysisView/environmentStateWriteApply.ts +++ b/src/domain/analysisView/environmentStateWriteApply.ts @@ -236,6 +236,7 @@ const ruledValue = ( case "clock": case "nested-clock": case "held-clock": + case "made-object": return generate(rule); case "echo": return ok( @@ -356,6 +357,7 @@ export const createdFields = ( sentCall: WriteCall, generate: GenerateValue, referenced?: ReferencedEntity, + idField?: string, ): Applied => { if (plan.body === null) return err("The tool has no modelled body."); const call = withBlanks(plan.body, sentCall); @@ -377,10 +379,38 @@ export const createdFields = ( else if (rule.rule === "echo" || rule.rule === "reference") delete written[rule.field]; } + ownMadeMembers(plan.rules, written, idField); const unspelled = derive(plan.rules, observed, written, call.body ?? {}); return unspelled === undefined ? ok(written) : err(unspelled); }; +/** Set each made object's members that hold the entity's identity to the identity it was written with, generated or sent. */ +const ownMadeMembers = ( + rules: readonly FieldRule[], + written: JsonObject, + idField: string | undefined, +): void => { + const identity = + idField ?? rules.find((rule) => rule.rule === "identity")?.field; + const id = identity === undefined ? undefined : ownMember(written, identity); + if (id === undefined) return; + for (const rule of rules) { + if (rule.rule !== "made-object" || rule.own.length === 0) continue; + const made = ownMember(written, rule.field); + if (made === undefined || !isObject(made)) continue; + const owned: JsonObject = { ...made }; + for (const member of rule.own) + setOwnMember(owned, member, spelledIdentity(ownMember(made, member), id)); + setOwnMember(written, rule.field, owned); + } +}; + +/** An identity spelled as a recorded member spells it: a string, or a number when the identity is one. */ +const spelledIdentity = ( + exemplar: JsonValue | undefined, + id: JsonValue, +): JsonValue => (typeof exemplar === "string" ? (scalarText(id) ?? id) : id); + /** Rules an update gives a new value on every call. */ const REGENERATED: ReadonlySet = new Set([ "clock", diff --git a/src/domain/analysisView/environmentStateWriteCreate.ts b/src/domain/analysisView/environmentStateWriteCreate.ts index 616bc9b927..694c45d0c5 100644 --- a/src/domain/analysisView/environmentStateWriteCreate.ts +++ b/src/domain/analysisView/environmentStateWriteCreate.ts @@ -20,10 +20,14 @@ import { } from "./environmentStateWriteApply.js"; import { createdToken, + heldObjects, isOpaqueToken, + nextMadeCount, restampedFields, + stampedBy, } from "./environmentStateWriteStamps.js"; import { + anyLinkedSource, recordedPathValues, respondedIds, sameLinkedSource, @@ -91,6 +95,7 @@ import { isUuid, likeGeneratedInteger, nextSequence, + startedSequence, sequenceNumber, spelledLike, } from "./environmentStateWriteValues.js"; @@ -305,6 +310,8 @@ const refineCreate = ( const earlierOf = (draft: WriteDraft) => ({ kept: draft.kept, clockOf: (field: string) => clockEvidence(draft, field), + startOf: (field: string) => draft.starts.get(field), + createCollections: draft.createCollections, ampersands: draft.ampersands, }); @@ -466,6 +473,13 @@ const buildCreate = ( noteHeldAmpersands(draft.ampersands, explained.rules, held); noteAmpersands(draft.ampersands, explained.rules, evidence.observed.object); for (const entity of held) noteFixedWidths(draft.fixedWidth, entity); + const ruled = new Set(explained.rules.map((rule) => rule.field)); + for (const [field, value] of Object.entries(evidence.observed.object)) { + const integer = sequenceNumber(value); + if (ruled.has(field) || integer === undefined) continue; + const next = nextInteger(held, field, value); + if (next === undefined || integer >= next) draft.starts.set(field, integer); + } return true; }; @@ -505,10 +519,13 @@ const followsSequences = ( rule.scope === "type" || entity.collection === created.collection, ); const next = nextSequence( - nextInteger( - fieldsOf(scope), - rule.field, - ownMember(draft.observed, rule.field), + startedSequence( + nextInteger( + fieldsOf(scope), + rule.field, + ownMember(draft.observed, rule.field), + ), + rule.start, ), draft.sequences.get(key), ); @@ -523,9 +540,17 @@ const agreesLater = ( context: EffectContext, draft: WriteDraft, created: Pick, - call: { readonly body: JsonObject; readonly observed: JsonObject }, + call: { + readonly body: JsonObject; + readonly observed: JsonObject; + readonly startMs: number | undefined; + }, ): boolean => - followsCreateRules(context, draft, { ...created, observed: call.observed }) && + followsCreateRules(context, draft, { + ...created, + observed: call.observed, + startMs: call.startMs, + }) && !contradictsDefault(draft.plan.rules, call, created.fields) && !unruledDiffers(draft.plan.rules, call.observed, { written: created.fields, @@ -572,6 +597,7 @@ const followsCreateRules = ( draft: WriteDraft, created: Pick & { readonly observed: JsonObject; + readonly startMs: number | undefined; }, ): boolean => draft.plan.rules.every((rule) => { @@ -607,7 +633,56 @@ const followsCreateRules = ( return ( rule.rule !== "identity" || rule.generator !== "uuid" || isUuid(value) ); - }) && followsSequences(context, draft, created); + }) && + followsSequences(context, draft, created) && + followsMadeObjects(context, draft, created); + +/** The key a made object's count is taken under in a plan's sequences. */ +const madeKey = (field: string, member: string): string => `${field}/${member}`; + +/** Whether a later call's recorded made objects hold what serving makes: the exemplar's members, the entity's identity, clocks, and each count one past what held objects show or started with, past the plan's last. */ +const followsMadeObjects = ( + context: EffectContext, + draft: WriteDraft, + created: Pick & { + readonly observed: JsonObject; + readonly startMs: number | undefined; + }, +): boolean => + draft.plan.rules.every((rule) => { + if (rule.rule !== "made-object") return true; + const made = ownMember(created.observed, rule.field); + const exemplar = ownMember(draft.observed, rule.field); + if (!isObject(made) || !isObject(exemplar)) return false; + const id = scalarText( + ownMember(created.fields, context.idField(created.type)), + ); + const held = heldObjects( + [ + ...context.state.ofType(created.type), + ...context.startedWith(created.type), + ].map((entity) => entity.fields), + rule.field, + ); + return [...new Set([...Object.keys(exemplar), ...Object.keys(made)])].every( + (member) => { + const value = ownMember(made, member); + if (rule.clocks.includes(member)) + return stampedBy(value, rule.format, created.startMs); + if (rule.own.includes(member)) return scalarText(value) === id; + const count = rule.counts.find((entry) => entry.member === member); + if (count === undefined) + return same(value, ownMember(exemplar, member)); + return ( + sequenceNumber(value) === + nextSequence( + nextMadeCount(held, count, ownMember(exemplar, member)), + draft.sequences.get(madeKey(rule.field, member)), + ) + ); + }, + ); + }); /** Widen the plan's clock formats to the trimmed spelling a later call showed, and its other clocks with them unless a value the plan saw shows them at a fixed width. */ const fitClocks = (draft: WriteDraft, fields: JsonObject): void => { @@ -643,6 +718,17 @@ const takeSequences = ( ); if (value !== undefined) draft.sequences.set(key, value); } + for (const rule of draft.plan.rules) { + if (rule.rule !== "made-object") continue; + const made = ownMember(created.fields, rule.field); + for (const { member } of rule.counts) { + const value = isObject(made) + ? sequenceNumber(ownMember(made, member)) + : undefined; + if (value !== undefined) + draft.sequences.set(madeKey(rule.field, member), value); + } + } }; /** Segments of a recorded path as a create call supplies them: template literals, and each parameter's recorded spelling. */ @@ -709,6 +795,7 @@ const fromLinkedSource = ( draft: WriteDraft, step: WriteStep, ): boolean => + anyLinkedSource(draft.plan, step.tool.path_template) || sameLinkedSource( step.tool.path_template, context.records.find( @@ -838,6 +925,7 @@ export const applyCreate = ( ownMember(observed.object, rule.field) ?? null, ), referencedIn(context), + idField, ); // A later call that agrees with the plan widens it, as a later update does: to members it alone sent or left out, an // array it showed sorted, or a clock spelling. One that disagrees leaves the plan as it was. @@ -861,7 +949,11 @@ export const applyCreate = ( context, draft, { type, collection, fields: fields.value }, - { body: call.body, observed: observed.object }, + { + body: call.body, + observed: observed.object, + startMs: startOf(context, step), + }, )) ) { refuse(draft, step, "unexplained-change"); @@ -877,6 +969,8 @@ export const applyCreate = ( draft.plan.rules, ); const inserted = insertCreated(context, draft, entity(stored, id)); + context.io.retain(collection.length * 2 + 16); + draft.createCollections.push(collection); return { kind: "applied", targets: [inserted, collectionTarget(type, collection)], diff --git a/src/domain/analysisView/environmentStateWriteCreateCalls.ts b/src/domain/analysisView/environmentStateWriteCreateCalls.ts index 91e04bc91a..1383545146 100644 --- a/src/domain/analysisView/environmentStateWriteCreateCalls.ts +++ b/src/domain/analysisView/environmentStateWriteCreateCalls.ts @@ -1,5 +1,11 @@ import type { JsonValue } from "../jsonValue.js"; -import { ownMember, same, type JsonObject } from "./environmentStateRules.js"; +import { + isObject, + ownMember, + same, + scalarText, + type JsonObject, +} from "./environmentStateRules.js"; import { storedKinds } from "./environmentStateWriteApply.js"; import { fittedClockFormat } from "./environmentStateWriteClockFormats.js"; import { composedValue } from "./environmentStateWriteComposed.js"; @@ -9,7 +15,13 @@ import { type FieldRule, type WriteBody, } from "./environmentStateWriteRules.js"; -import { slugOf, type Ampersand } from "./environmentStateWriteValues.js"; +import { + containsDelimited, + replaceDelimited, + sequenceNumber, + slugOf, + type Ampersand, +} from "./environmentStateWriteValues.js"; type Default = { readonly member: string; readonly value: JsonValue }; type Blank = NonNullable[number]; @@ -18,6 +30,9 @@ type Blank = NonNullable[number]; export interface EarlierEvidence { readonly kept: ReadonlySet; readonly clockOf: (field: string) => JsonValue | undefined; + /** The integer the exemplar showed in an unruled field above every value its type held, where a sequence may start. */ + readonly startOf?: (field: string) => number | undefined; + readonly createCollections?: readonly string[]; /** How the plan's accepted calls and the type's held entities spelled `&` in each field's slug; absent where none showed one. */ readonly ampersands?: ReadonlyMap; } @@ -25,7 +40,7 @@ export interface EarlierEvidence { /** How a field's slug showed `&`: dropped, as the word, or in a way neither spelling gives. */ export type AmpersandSpelling = "dropped" | Ampersand | "unknown"; -/** The plan's rule for a field and a later call's agree: equal, or clocks one spelling covers given what earlier calls showed. */ +/** The plan's rule for a field and a later call's agree: equal, clocks one spelling covers given what earlier calls showed, slugs that differ only in how they spell `&`, or made objects alike but in the counts only the later call could show. */ const agreeing = ( held: FieldRule, later: FieldRule, @@ -39,6 +54,13 @@ const agreeing = ( ): FieldRule | undefined => { const alike = alikeRules(held, later, values.earlier.kept); if (alike !== undefined) return alike; + if (held.rule === "made-object" && later.rule === "made-object") + return madeAlike( + held, + later, + ownMember(values.exemplar, values.field), + values.earlier, + ); const spelled = ampersandSpelled( held, { rule: later, entity: values.laterEntity }, @@ -54,6 +76,41 @@ const agreeing = ( return format === undefined ? undefined : { ...held, format }; }; +type MadeObject = Extract; + +/** Two made-object rules as one: the same clocks, format and own members, and the earlier counts among the later's; a count only the later call shows starts at the exemplar's value, which no earlier object could show past, and only while the exemplar is the one create accepted, as earlier creates held it still. */ +const madeAlike = ( + held: MadeObject, + later: MadeObject, + exemplar: JsonValue | undefined, + earlier: EarlierEvidence, +): MadeObject | undefined => { + if ( + !same(held.clocks, later.clocks) || + !same(held.own, later.own) || + !same(held.format, later.format) || + held.counts.some( + ({ member }) => !later.counts.some((count) => count.member === member), + ) + ) + return undefined; + const counts: MadeObject["counts"] = []; + for (const { member } of later.counts) { + const kept = held.counts.find((count) => count.member === member); + // Accepted creates past the exemplar showed this member unchanged, which a count would have moved. + if (kept === undefined && (earlier.createCollections ?? []).length > 1) + return undefined; + const start = + kept?.start ?? + sequenceNumber( + isObject(exemplar) ? ownMember(exemplar, member) : undefined, + ); + if (start === undefined) return undefined; + counts.push({ member, start }); + } + return { ...held, counts }; +}; + /** A rule with its slugs spelled the default way. */ const unspelled = (rule: FieldRule): FieldRule => { if (rule.rule === "slug") { @@ -196,6 +253,44 @@ const joinedDefaults = ( return [...joined].map(([member, value]) => ({ member, value })); }; +/** A later call's rule for a field the plan kept as shown, when it spells the exemplar's value too: a string composed from the exemplar's fields or holding the value it templates, or a sequence that started at its value. */ +const spellsExemplar = ( + field: string, + rule: FieldRule, + calls: Pick, +): FieldRule | undefined => { + const value = ownMember(calls.exemplar, field); + if (rule.rule === "composed") + return value !== undefined && + composedValue(rule.parts, calls.exemplar) === value + ? rule + : undefined; + if (rule.rule === "template") { + const token = scalarText(ownMember(calls.exemplar, rule.from)); + const next = scalarText(ownMember(calls.later.entity, rule.from)); + return typeof value === "string" && + token !== undefined && + next !== undefined && + containsDelimited(value, token) && + replaceDelimited(value, token, next) === + ownMember(calls.later.entity, field) + ? rule + : undefined; + } + if (rule.rule !== "sequence") return undefined; + const start = calls.earlier.startOf?.(field); + const collections = calls.earlier.createCollections ?? []; + if ( + (rule.scope === "type" && collections.length > 1) || + (rule.scope === "collection" && + new Set(collections).size !== collections.length) + ) + return undefined; + return start !== undefined && sequenceNumber(value) === start + ? { ...rule, start } + : undefined; +}; + /** What two calls show of one field. */ interface FieldCalls { readonly before: FieldRule | undefined; @@ -273,6 +368,10 @@ const mergedField = ( default: { member: field, value: later.entity[field]! }, } : undefined; + if (before === undefined && after !== undefined) { + const rule = spellsExemplar(field, after, calls); + if (rule !== undefined) return { rule }; + } // This call echoes a member the plan's calls left out: what they stored there is its default. if (after !== undefined) return after.rule === "echo" && !calls.required.includes(field) diff --git a/src/domain/analysisView/environmentStateWriteEffects.ts b/src/domain/analysisView/environmentStateWriteEffects.ts index e0c74c51fe..4ace3c1121 100644 --- a/src/domain/analysisView/environmentStateWriteEffects.ts +++ b/src/domain/analysisView/environmentStateWriteEffects.ts @@ -94,6 +94,10 @@ export interface WriteDraft { readonly updates: UpdateCall[]; /** The last value each sequence took, by field and, for a collection's sequence, collection; the next call must follow it. */ readonly sequences: Map; + /** Integers the exemplar showed in fields no rule wrote, above every value its type held, where a later call's sequence may start. */ + readonly starts: Map; + /** Collections of accepted create calls, used to check late sequence adoption. */ + readonly createCollections: string[]; /** The collection a create writes into when its own path names no state type: the one its returned id was read under. */ readonly linked?: string | undefined; /** Fields an item action's call found already at the value it sets them to. */ diff --git a/src/domain/analysisView/environmentStateWriteExplain.ts b/src/domain/analysisView/environmentStateWriteExplain.ts index f40c00f6b2..6b2563724e 100644 --- a/src/domain/analysisView/environmentStateWriteExplain.ts +++ b/src/domain/analysisView/environmentStateWriteExplain.ts @@ -55,6 +55,7 @@ import { GENERATED, } from "./environmentStateWriteGenerated.js"; import { + createdMadeObject, createdNestedClock, createdToken, isClockNear, @@ -281,7 +282,10 @@ const generatedRule = ( return { field, rule: "unique" }; if (createdToken(field, evidence.entity, evidence.type) === "independent") return { field, rule: "unique" }; - return createdNestedClock(field, value, evidence); + return ( + createdNestedClock(field, value, evidence) ?? + createdMadeObject(field, value, evidence) + ); }; /** Pinned members less those a reference reads: it rebuilds the object they name for any held entity. */ diff --git a/src/domain/analysisView/environmentStateWriteLinks.ts b/src/domain/analysisView/environmentStateWriteLinks.ts index 3c428a20b3..9c751ccb94 100644 --- a/src/domain/analysisView/environmentStateWriteLinks.ts +++ b/src/domain/analysisView/environmentStateWriteLinks.ts @@ -23,6 +23,7 @@ import { import { ENTRY_BYTES, textBytes } from "./webNetworkRouteTemplates.js"; import { pathValuesOf, + type FieldRule, type WriteEffect, } from "./environmentStateWriteRules.js"; @@ -44,9 +45,14 @@ const identityValues = (object: JsonValue | undefined): string[] => }) : []; +/** The one object a one-element array holds, else the value itself. */ +const unwrapped = (value: JsonValue): JsonValue => + Array.isArray(value) && value.length === 1 ? value[0]! : value; + /** * The ids a write's response names for an entity it made: an identity-named member of the body or of - * an object member (an envelope) at any length, or a scalar body or sole scalar member long enough to name one. + * an object member (an envelope, or the one object of a one-element array member) at any length, or a + * scalar body or sole scalar member long enough to name one. */ export const respondedIds = (body: JsonValue | undefined): string[] => { if (body === undefined || body === null) return []; @@ -60,7 +66,7 @@ export const respondedIds = (body: JsonValue | undefined): string[] => { ...new Set([ ...identityValues(body), ...(members.length === 1 ? named(members[0]) : []), - ...members.flatMap(identityValues), + ...members.flatMap((member) => identityValues(unwrapped(member))), ]), ]; }; @@ -395,6 +401,27 @@ export const recordedPathValues = ( ); }; +/** Whether a linked create may take any path arguments: it creates into its type's own collection, whose path has no parameters, and stores each path value in a field, so none changes where it creates. */ +export const anyLinkedSource = ( + plan: { + readonly target: string | null; + readonly collection: string | null; + readonly rules: readonly FieldRule[]; + }, + template: string, +): boolean => + plan.target !== null && + plan.collection !== null && + normalizedPath(plan.collection) === plan.target && + !segmentsOf(plan.target).some(isParam) && + segmentsOf(template) + .filter(isParam) + .every((segment) => + plan.rules.some( + (rule) => rule.rule === "path" && rule.from === segment.slice(1, -1), + ), + ); + /** * Whether a linked create's call sends the path values its exemplar did, each as recorded or in the * other spelling once percent-decoded: the read that linked its target followed only that call. The diff --git a/src/domain/analysisView/environmentStateWriteRules.ts b/src/domain/analysisView/environmentStateWriteRules.ts index a91c60050f..3378f67be8 100644 --- a/src/domain/analysisView/environmentStateWriteRules.ts +++ b/src/domain/analysisView/environmentStateWriteRules.ts @@ -42,7 +42,8 @@ export const usesWriteClock = (plan: Pick): boolean => (rule) => rule.rule === "clock" || rule.rule === "held-clock" || - rule.rule === "nested-clock", + rule.rule === "nested-clock" || + rule.rule === "made-object", ); /** Rules that generate a value when served; the gate takes the recorded value instead. */ @@ -52,10 +53,12 @@ export type GeneratedRule = readonly rule: "identity"; readonly generator: "sequence" | "uuid" | "token"; } + /** One past the largest value its scope holds, and never below `start`, a value the first create took above them. */ | { readonly field: string; readonly rule: "sequence"; readonly scope: "collection" | "type"; + readonly start?: number; } | { readonly field: string; readonly rule: "unique" } /** The value kept with each UUID at the pointers (`*` for any array index) replaced by a fresh one, as rewritten child rows are. */ @@ -81,6 +84,15 @@ export type GeneratedRule = readonly field: string; readonly rule: "held-clock"; readonly format: ClockFormat; + } + /** An object made with the entity, which no initial entity holds: `clocks` take the write's clock, `own` the entity's identity, each of `counts` one past the largest value held objects show and never below its start; other members as the exemplar showed them. */ + | { + readonly field: string; + readonly rule: "made-object"; + readonly clocks: string[]; + readonly format: ClockFormat; + readonly own: string[]; + readonly counts: { readonly member: string; readonly start: number }[]; }; /** How one field of a written entity gets its value; fields no rule names keep the observed value. */ diff --git a/src/domain/analysisView/environmentStateWriteStamps.ts b/src/domain/analysisView/environmentStateWriteStamps.ts index 000b72583d..ae6f49bef9 100644 --- a/src/domain/analysisView/environmentStateWriteStamps.ts +++ b/src/domain/analysisView/environmentStateWriteStamps.ts @@ -20,6 +20,9 @@ import { containsDelimited, isUuid, readClock, + sequenceNumber, + spelledLike, + type ClockFormat, } from "./environmentStateWriteValues.js"; /** A clock value counts as written by the request when it lies this close to the request's start. */ @@ -243,6 +246,147 @@ export const createdNestedClock = ( return stamped ? nestedRule(field, value, members, evidence.type) : undefined; }; +/** Whether a clock is one a request starting at `startMs` stamped, in the given format. */ +export const stampedBy = ( + value: JsonValue | undefined, + format: ClockFormat, + startMs: number | undefined, +): boolean => { + const clock = readClock(value); + return ( + clock !== undefined && + startMs !== undefined && + same(clock.format, format) && + clock.ms >= startMs - TRUNCATION_MS && + clock.ms <= startMs + NESTED_WINDOW_MS + ); +}; + +/** The largest integer the objects hold at a member, spelled as the exemplar spells it. */ +const largestAt = ( + objects: readonly JsonObject[], + member: string, + exemplar: JsonValue | undefined, +): number | undefined => { + let largest: number | undefined; + for (const object of objects) { + const value = sequenceNumber(ownMember(object, member), exemplar); + if (value !== undefined && (largest === undefined || value > largest)) + largest = value; + } + return largest; +}; + +/** The objects held entities keep at a field. */ +export const heldObjects = ( + entities: readonly JsonObject[], + field: string, +): JsonObject[] => + entities.flatMap((entity) => { + const value = ownMember(entity, field); + return value !== undefined && isObject(value) ? [value] : []; + }); + +/** One past the largest integer held objects show at a member, never below a start. */ +export const nextMadeCount = ( + objects: readonly JsonObject[], + count: { readonly member: string; readonly start: number }, + exemplar: JsonValue | undefined, +): number => { + const largest = largestAt(objects, count.member, exemplar); + return Math.max( + largest === undefined ? count.start : largest + 1, + count.start, + ); +}; + +/** + * On a create, an object the application made with the entity: members holding a clock the request stamped, others the + * entity's own identity or one past the largest integer held objects show there, and the rest as every held object + * shows them. Undefined without a stamped clock, since only that shows the request made the object. + */ +export const createdMadeObject = ( + field: string, + value: JsonValue, + evidence: { + readonly startMs: number | undefined; + readonly type: readonly JsonObject[]; + readonly entity: JsonObject; + readonly idField: string; + }, +): FieldRule | undefined => { + const { startMs } = evidence; + if (!isObject(value) || startMs === undefined) return undefined; + const keys = Object.keys(value); + const clocks = keys.filter((key) => { + const ms = readClock(value[key])?.ms; + return ( + ms !== undefined && + ms >= startMs - TRUNCATION_MS && + ms <= startMs + NESTED_WINDOW_MS + ); + }); + const format = + clocks.length === 0 ? undefined : readClock(value[clocks[0]!])!.format; + if ( + format === undefined || + clocks.some((key) => !same(readClock(value[key])!.format, format)) + ) + return undefined; + const id = scalarText(ownMember(evidence.entity, evidence.idField)); + const own = keys.filter( + (key) => + !clocks.includes(key) && + id !== undefined && + scalarText(value[key]) === id, + ); + const held = heldObjects(evidence.type, field); + const counts = keys.flatMap((key) => { + if (clocks.includes(key) || own.includes(key)) return []; + const integer = sequenceNumber(value[key]); + const largest = largestAt(held, key, value[key]); + return integer !== undefined && + largest !== undefined && + integer === largest + 1 + ? [{ member: key, start: integer }] + : []; + }); + const varying = new Set([ + ...clocks, + ...own, + ...counts.map(({ member }) => member), + ]); + const alike = held.every((object) => + [...new Set([...Object.keys(object), ...keys])].every( + (key) => + varying.has(key) || same(ownMember(object, key), ownMember(value, key)), + ), + ); + return alike + ? { field, rule: "made-object", clocks, format, own, counts } + : undefined; +}; + +/** A served made object: the exemplar's, its clocks set to the write's clock and each count to the value `next` gives; its own members follow the entity's identity once written. */ +export const madeObjectValue = ( + base: JsonValue | undefined, + rule: Extract, + clock: JsonValue, + next: (count: { readonly member: string; readonly start: number }) => number, +): Result => { + if (base === undefined || !isObject(base)) + return err(`The plan holds no ${rule.field} object to make.`); + const made: JsonObject = { ...base }; + for (const member of rule.clocks) setOwnMember(made, member, clock); + for (const count of rule.counts) + setOwnMember( + made, + count.member, + spelledLike(ownMember(base, count.member) ?? "", next(count)), + ); + return ok(made); +}; + /** A served nested clock: the object the entity holds, or the observed one for a create, with its stamped members set to the clock. */ export const nestedClockValue = ( base: JsonValue | undefined, diff --git a/src/domain/analysisView/environmentStateWriteValues.ts b/src/domain/analysisView/environmentStateWriteValues.ts index aa62539e65..b2135f7ae0 100644 --- a/src/domain/analysisView/environmentStateWriteValues.ts +++ b/src/domain/analysisView/environmentStateWriteValues.ts @@ -249,6 +249,13 @@ export const nextSequence = ( ): number => taken === undefined ? (held ?? 1) : Math.max(held ?? taken + 1, taken + 1); +/** The next held integer of a sequence that never goes below its `start`. */ +export const startedSequence = ( + held: number | undefined, + start: number | undefined, +): number | undefined => + start === undefined ? held : Math.max(held ?? start, start); + /** An integer spelled as the recorded value spells integers: a number, or a string of digits. */ export const spelledLike = (exemplar: JsonValue, value: number): JsonValue => maskedCount(exemplar) !== undefined diff --git a/src/generatedPackageMetadata.ts b/src/generatedPackageMetadata.ts index 096f718c69..25bb35bb64 100644 --- a/src/generatedPackageMetadata.ts +++ b/src/generatedPackageMetadata.ts @@ -6,5 +6,5 @@ export const PACKAGE_METADATA = { serverSdkVersion: "2.3.1", clientSdkVersion: "2.3.1", coreSdkVersion: "2.3.1", - skillVersion: "52", + skillVersion: "53", } as const; diff --git a/tests/boundary/mcp/environmentServeMcp.test.ts b/tests/boundary/mcp/environmentServeMcp.test.ts index 83831a6496..ba29ccba49 100644 --- a/tests/boundary/mcp/environmentServeMcp.test.ts +++ b/tests/boundary/mcp/environmentServeMcp.test.ts @@ -1,5 +1,6 @@ import { createHash } from "node:crypto"; import { timedCaptureEvidence } from "../../fixtures/analysisView.js"; +import { INITIAL_TRASH_ROWS } from "../../fixtures/environmentStateLaterReads.js"; import { jsonObjectSchema } from "../../../src/domain/jsonValue.js"; import { @@ -23,9 +24,9 @@ import { const HTTP_BOUNDARY_TIMEOUT_MS = 90_000; const CARD_PATH = "/.well-known/agent-env.json"; -/** SHA-256 of the 13,882-byte card served for the fixture named "renamed", its descriptions naming each tool's recorded responses; pure replay must serve these bytes. */ +/** SHA-256 of the 14,519-byte card served for the fixture named "renamed", its descriptions naming each tool's recorded responses; pure replay must serve these bytes. */ const REPLAY_CARD_SHA256 = - "27370fe0a89556dcd0dd1bc6ddd0439d77b628349829afd56e4af4ee393d0a64"; + "995c280f0798f77e76f206c068eb6c2c1f8c4210a03ac1c01fba8cc07b5d7f09"; /** Serve the fixture pair as pure replay through the built CLI. */ const serveReplayOnly = async ( @@ -58,6 +59,73 @@ const connect = async (url: URL) => { return client; }; +processTest( + "replays a recorded list containing an initially deleted entity", + async ({ processes }) => { + const fixture = environmentStateServeFixture( + timedCaptureEvidence(INITIAL_TRASH_ROWS), + ); + const root = await createTestTempDirectory("rea-environment-trash-mcp-"); + const files = await writeEnvironmentServeFiles(root, fixture); + const started = await startListeningProcess(processes, [ + "environment-serve", + files.manifestPath, + "--evidence", + files.capturePath, + "--port", + "0", + ]); + const client = await connect(started.url); + try { + const response = await client.callTool({ + name: "get_contacts", + arguments: { maxSize: "20", offset: "0" }, + }); + expect(response.isError).toBeFalsy(); + expect(response.structuredContent).toMatchObject({ + body: INITIAL_TRASH_ROWS[0]![2], + }); + } finally { + await client.close(); + } + }, + HTTP_BOUNDARY_TIMEOUT_MS, +); + +processTest( + "accepts numbers and booleans for query parameters, sent as their JSON text", + async ({ processes }) => { + const fixture = environmentStateServeFixture( + timedCaptureEvidence([ + [ + "GET", + "/api/contacts?maxSize=2&withItems=true", + [{ id: 1, name: "Ada" }], + ], + ]), + ); + const started = await serveReplayOnly(processes, {}, fixture); + const client = await connect(started.url); + try { + const response = await client.callTool({ + name: "get_contacts", + arguments: { maxSize: 2, withItems: true }, + }); + expect(response.isError).toBeFalsy(); + expect(response.structuredContent).toMatchObject({ + body: [{ id: 1, name: "Ada" }], + replay: { + exact_match: true, + recorded_query: "?maxSize=2&withItems=true", + }, + }); + } finally { + await client.close(); + } + }, + HTTP_BOUNDARY_TIMEOUT_MS, +); + processTest( "replays recorded own JSON keys without interpreting inherited members as fields", async ({ processes }) => { @@ -113,6 +181,8 @@ processTest( ).toMatchObject({ oneOf: [ { type: "string" }, + { type: "number" }, + { type: "boolean" }, { type: "array", items: { type: "string" } }, ], }); diff --git a/tests/fixtures/environmentStateLaterReads.ts b/tests/fixtures/environmentStateLaterReads.ts index 18807c6486..bdac849871 100644 --- a/tests/fixtures/environmentStateLaterReads.ts +++ b/tests/fixtures/environmentStateLaterReads.ts @@ -18,6 +18,18 @@ const ADA = contact(1, "Ada Finch"); const MINA = contact(2, "Mina Okafor"); const RUI = contact(3, "Rui Castellanos"); +/** An initial trash-style list includes an entity already marked deleted. */ +export const INITIAL_TRASH_ROWS: readonly TimedCaptureRow[] = [ + [ + "GET", + "/api/contacts?maxSize=20&offset=0", + page([ADA, MINA, { ...RUI, deleted: true }]), + ], + ["GET", "/api/contacts/1", ADA], + ["GET", "/api/contacts/2", MINA], + ["GET", "/api/contacts/3", { ...RUI, deleted: true }], +]; + /** A contact deleted, then read by id as EspoCRM answers for soft-deleted records: still found, marked `deleted`. */ export const SOFT_DELETED_ROWS: readonly TimedCaptureRow[] = [ ["GET", "/api/contacts?maxSize=20&offset=0", page([ADA, MINA, RUI])], diff --git a/tests/fixtures/environmentStateMadeValues.ts b/tests/fixtures/environmentStateMadeValues.ts new file mode 100644 index 0000000000..2d9f01c31b --- /dev/null +++ b/tests/fixtures/environmentStateMadeValues.ts @@ -0,0 +1,167 @@ +import type { JsonObject } from "../../src/domain/analysisView/environmentStateRules.js"; +import type { TimedCaptureRow } from "./analysisView.js"; +import { bookClock } from "./environmentStateWrites.js"; + +/** How a capture of two task creates departs from Vikunja's. */ +interface TaskOptions { + /** Whether tasks carry a `created` clock of their own besides their subscription's. */ + readonly clocked?: boolean; + /** Whether each create sends the id the task then holds. */ + readonly sentId?: boolean; + /** A third create, whose subscription names this task rather than its own. */ + readonly thirdSubscribes?: number; + /** The first create's `identifier`, when not `#` and its index. */ + readonly firstIdentifier?: string; + /** Whether lists show each task's subscription, as item reads do. */ + readonly listsSubscriptions?: boolean; + /** A third create of its own task whose subscription shows a clock from before its request. */ + readonly staleThird?: boolean; +} + +const seed = (id: number, index: number, clocked: boolean): JsonObject => ({ + id, + title: `Seeded task ${String(id)}`, + project_id: 5, + index, + identifier: `#${String(index)}`, + subscription: null, + ...(clocked ? { created: "2026-10-01T09:00:00Z" } : {}), +}); + +/** Two task creates at a project path, numbered past tasks deleted before the capture and read back with the subscription the application made, then the list. */ +const taskRows = (options: TaskOptions = {}): TimedCaptureRow[] => { + const clocked = options.clocked ?? true; + const seeds = [seed(1, 1, clocked), seed(2, 2, clocked)]; + const made = ( + id: number, + index: number, + title: string, + row: number, + identifier = `#${String(index)}`, + ): JsonObject => ({ + id, + title, + project_id: 5, + index, + identifier, + subscription: null, + ...(clocked ? { created: bookClock(row) } : {}), + }); + const subscribed = ( + task: JsonObject, + subscription: number, + row: number, + entity = task["id"]!, + ): JsonObject => ({ + ...task, + subscription: { + id: subscription, + entity: "task", + entity_id: entity, + created: bookClock(row), + }, + }); + const first = made( + 20, + 15, + "Replace the lens gasket", + 2, + options.firstIdentifier, + ); + const second = made(21, 16, "Log the fog-signal tests", 5); + const sent = (task: JsonObject) => ({ + ...(options.sentId === true ? { id: task["id"]! } : {}), + title: task["title"]!, + }); + const firstRead = subscribed(first, 21, 2); + const secondRead = subscribed(second, 22, 5); + const listed = (task: JsonObject): JsonObject => { + if (options.listsSubscriptions !== false) return task; + const { subscription: _subscription, ...rest } = task; + return rest; + }; + return [ + ["GET", "/api/tasks", seeds.map(listed)], + ["GET", "/api/tasks/1", seeds[0]!], + ["POST", "/api/projects/5/tasks", first, 201, sent(first)], + ["GET", "/api/tasks/20", firstRead], + ["GET", "/api/tasks", [...seeds, firstRead].map(listed)], + ["POST", "/api/projects/5/tasks", second, 201, sent(second)], + ["GET", "/api/tasks/21", secondRead], + ["GET", "/api/tasks", [...seeds, firstRead, secondRead].map(listed)], + ...(options.thirdSubscribes === undefined && options.staleThird !== true + ? [] + : thirdRows( + made(22, 17, "Paint the gallery rail", 8), + subscribed( + made(22, 17, "Paint the gallery rail", 8), + 23, + options.staleThird === true ? 4 : 8, + options.thirdSubscribes, + ), + [...seeds, firstRead, secondRead].map(listed), + listed, + sent, + )), + ]; +}; + +/** A third create, its read-back and the list after it. */ +const thirdRows = ( + third: JsonObject, + read: JsonObject, + before: readonly JsonObject[], + listed: (task: JsonObject) => JsonObject, + sent: (task: JsonObject) => JsonObject, +): TimedCaptureRow[] => [ + ["POST", "/api/projects/5/tasks", third, 201, sent(third)], + ["GET", "/api/tasks/22", read], + ["GET", "/api/tasks", [...before, listed(read)]], +]; + +/** Vikunja's two task creates, each read back with a made subscription, then the list. */ +export const COUNTED_TASK_ROWS: readonly TimedCaptureRow[] = taskRows(); + +/** As COUNTED_TASK_ROWS, through a bulk path whose body and answer wrap the one task in `tasks`. */ +export const BULK_TASK_ROWS: readonly TimedCaptureRow[] = COUNTED_TASK_ROWS.map( + (row) => + row[0] === "POST" + ? [ + "POST", + `${row[1]}/bulk`, + { tasks: [row[2]!] }, + 201, + { tasks: [row[4]!] }, + ] + : row, +); + +/** COUNTED_TASK_ROWS up to the first create's reads: one create alone shows no counter. */ +export const ONE_TASK_ROWS: readonly TimedCaptureRow[] = + COUNTED_TASK_ROWS.slice(0, 5); + +/** Tasks with no clock of their own, so the subscription's is the only one the creates write. */ +export const UNCLOCKED_TASK_ROWS: readonly TimedCaptureRow[] = taskRows({ + clocked: false, +}); + +/** Creates that send the id the task takes. */ +export const CHOSEN_ID_TASK_ROWS: readonly TimedCaptureRow[] = taskRows({ + sentId: true, +}); + +/** A third create whose subscription names the second task, not its own, in a capture whose lists show no subscription. */ +export const MISLINKED_TASK_ROWS: readonly TimedCaptureRow[] = taskRows({ + thirdSubscribes: 21, + listsSubscriptions: false, +}); + +/** A first create spelling its identifier `TASK-15`, where the second spells `#16`. */ +export const RESPELLED_TASK_ROWS: readonly TimedCaptureRow[] = taskRows({ + firstIdentifier: "TASK-15", +}); + +/** A third create read back with a subscription whose clock is from before its request. */ +export const STALE_MADE_TASK_ROWS: readonly TimedCaptureRow[] = taskRows({ + staleThird: true, +});