mirror of
https://github.com/morluto/rea.git
synced 2026-11-02 22:10:55 +00:00
6e9ee97a69cb16c7150d8b550e705cba055ee4f0
291
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 gensokyo-dev
|
f72638f6b1 |
feat(investigation)!: retain records across engine cleanup (#8361)
Separate engine cleanup, investigation reset and terminal disposal in the shared application owner. Keep stable Evidence IDs and full Unknown revision history through engine close and target changes; serialize reset with all record writers and cleanup owners, optionally acknowledging a complete checkpoint before clearing. Bound retained representation without eviction. Validation: 43 focused files / 533 distinct tests, source/type/lint/schema/documentation checks, nine genuine source cases across eight families, real Ghidra session identity and cleanup, and nine actual local Codex CLI terra/high runs with independent full-result and checkpoint audits. See docs/verification/investigation-lifecycle.md for measurements, partial attempts and verification limits. Fixes #8347. BREAKING CHANGE: close_binary retains investigation records. Use reset_investigation to clear them. Cross-process recovery requires an explicitly acknowledged Evidence bundle checkpoint. Assisted-by: Suika (gensokyo-skills:suika-scatter-gather-planning) |
||
|
Kaoru0822andClaude Opus 5.5
|
c287a73276 |
fix(ghidra): report unbacked file-offset addresses without schema repair (#8308)
* fix(ghidra): report unbacked file-offset addresses without schema repair address_to_file_offset projected a well-formed address in uninitialized or unmapped Ghidra memory as generic invalid input: the headline said the arguments were invalid and recovery pointed at the input schema, while the real reason survived only in a nested issue. The bridge now gives each outcome its own code (address_unmapped, address_uninitialized, file_mapping_absent, file_mapping_ambiguous, file_offset_unrepresentable) instead of folding them into invalid_request. The Ghidra projection keeps the bridge reason as the headline, points to inspect_native_load_image, records the caller's exact address and the file_mapping outcome, and scopes diagnostics to the request. AnalysisInputError gains conformsToInputContract so a conforming request omits recovery.kind=inspect_input_contract. Malformed addresses and other bridge refusals keep their existing projection. Fixes #8244 Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(ghidra): match fixture block ends to bridge conventions The fake bridge treated .text's exclusive list_segments end (0x1100) as an inclusive bound, so address_to_file_offset mapped an address outside the block. Give the fixture's blocks one owner (start and size) and derive each reply from the production bridge convention: exclusive list_segments ends, inclusive byte-search coverage, half-open containment for file offsets. Add an MCP boundary test that reads the block edges from list_segments and search_bytes, maps the last file-backed byte and refuses the first address past each block with its outcome code. Co-Authored-By: Claude Opus 5.5 <[email protected]> --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
Entrpiandmorluto
|
b43c329df7 |
feat(artifacts): inventory and extract UF2 and DfuSe firmware containers (#8230)
* feat(artifacts): inventory and extract UF2 and DfuSe regions open_binary refused UF2 and DfuSe firmware containers as an unsupported format, so inspect_artifact and extract_artifact could not show which bytes a drag-and-drop or DFU update writes at which addresses. UF2 is recognized by a complete first block with all three magic numbers. Valid main-flash payloads are joined into address-contiguous regions per familyID, named family-0x<id>/0x<start>-0x<end>.bin, and known IDs are named from a small table copied from Microsoft's uf2families.json at a pinned commit. Numbering is counted per familyID and block count: absent, repeated, out-of-order and out-of-range block numbers are reported, and absent blocks leave coverage unknown. Not-main-flash blocks are excluded and reported; file-container blocks are reported and leave coverage unknown; invalid blocks and a partial final block are excluded with their position. A rewrite of one address with different bytes fails closed, naming both block spans. DfuSe is recognized only by its prefix together with a DFU 1.1a suffix. Each element becomes one region at its file range with its target's alternate setting and name. Structural faults fail closed with a byte offset; overlapping elements stay separate and are reported. A suffix CRC mismatch follows integrity_policy: fail refuses with a dfu-crc32 integrity error, and record-and-continue records a root contradiction that extraction also carries. HEX, UF2 and DfuSe share one retained-region reader and one run merger, so the HEX decoder and reader lose their private copies. * docs(artifacts): document UF2 and DfuSe images and their esptool lane Describe recognition, region names, metadata, numbering, failures, the DfuSe suffix CRC under integrity_policy, and limits, and link the page from the CLI and firmware guides and the agent skill reference. npm run verify:uf2 runs esptool merge-bin --format uf2 on generated inputs at known addresses, with and without MD5, with 256-byte payloads, and on a concatenation of two chips' outputs. Inspection and extraction through the built CLI must return every input at its address under the familyID esptool itself reports. * fix(artifacts): accept both DfuSe image size conventions UM0391 calls DFUImageSize the file length, and dfu-util's dfuse-pack.py writes that value, while ST's DfuFileMgr, MicroPython's dfu.py and pydfu leave the 16-byte suffix out. The decoder accepted only the second form, so it refused dfuse-pack.py output as malformed. Accept either value and keep the declared one in image_size; any other value still fails with the suffix offset and the file length. The tests pin both producers' output for one image, CRC included, from their build() functions transcribed and run under Python 3. * fix(artifacts): keep reused UF2 block numbers from hiding missing blocks esptool numbers every merged input from 0, so two inputs with the same block count share one numbering group. A block lost from one input left its number present in the other, and the file reported complete coverage. Each block number now records the target addresses it was seen at: equal address counts are complete copies, a resend to the same address is one block, and unequal counts make coverage unknown with a limitation and ambiguous_numbering_group_count. sequence_count becomes numbering_group_count, and region block positions become first_file_block and last_file_block. Review follow-ups in the same change: - compare_artifacts compares a DfuSe element's alternate setting and target name, so bytes moved to another device memory are changed; - the shared reader keeps no region bytes outside its map after close; - one domain type replaces four identical decoder failure types; - the UF2 decoder returns a typed limit instead of throwing on bytes beyond its admitted size, and caps retained regions, not runs; - DfuSe caps elements with bytes only, names empty elements and element-less targets, drops the constant bVersion and bcdDFU fields, and says whether the CRC matches when DFUImageSize fits neither convention; - classification evidence carries a suffix only when the probe saw the end, and root classification reads the tail only for a DfuSe prefix; - the UF2 domain region owns family_name, and the root records the family registry revision; the table's MIT notice is preserved; - docs pin the producer revisions (dfu-util ee6b393d and f9a537c9, MicroPython cb99ca98), give the MD5 structure's real meaning, and describe container contradictions in the MCP contract guide; - tests and the esptool lane add the equal-count inputs and a block lost from one of them, accepted boundaries beside each refusal, a DfuSe root larger than the classification prefix, and CLI refusals. * feat(artifacts): name the UF2 families that LibreTiny builds for LibreTiny writes UF2 for Realtek, Beken and Bouffalo Lab modules. The Microsoft registry revision REA already cites (cf3bd7f9) lists eight of those families: RTL8710A, RTL8710B, RTL8720C, RTL8720D, BK7231U, BK7231N, BK7251 and BL602. Their regions now carry those names. LN882H, which that revision omits, keeps its familyID with no name. * fix(artifacts): count DfuSe declarations without bytes instead of listing them Empty DfuSe elements have no entry, so DFUSE_MAX_ELEMENTS does not cap them, and the target walk kept a description string for each one. At 8 bytes per element, a file under the 128 MiB ceiling can declare about 16.7 million of them. The walk now counts every declaration without bytes and builds only the first NAMED_ELEMENTS descriptions, which is all that the limitation shows. The limitation text and empty_element_count do not change. The guide now says that both DfuSe limitations name the first four. On this host, decoding 4,000,000 empty elements (a 32 MB file) ran out of memory under a 128 MB V8 heap before this change, and it peaked at 1.59 GB RSS with the default heap. It now decodes in 114 ms with a 220 MB peak. --------- Co-authored-by: morluto <[email protected]> |
||
|
Entrpi
|
e669365c3b |
feat(ghidra): admit RISC-V ilp32 ELF with a derived calling convention (#8229)
* feat(ghidra): record the RISC-V float-ABI calling-convention decision Decision D3 (#7477), provisional: admit ELF32 soft-float (ilp32) RISC-V images through a lane that keeps the pinned RISCV:LE:32:default language and corrects the calling convention in the bridge. This commit is the contract only; admission is unchanged. - The domain owns the e_flags float-ABI vocabulary and parses .riscv.attributes and Tag_RISCV_arch into a declared, absent or unavailable ISA declaration. - The Ghidra profile owns the lane table (class and float ABI), lane ids built from fields, the ilp32-prototype-v1 policy and its __rea_ilp32 prototype, and the extensions Ghidra 12.1.4 decodes with p-code. - The session handshake accepts the bridge's calling-convention report. - docs/ghidra-riscv.md keeps the decision record and per-ABI policies for ilp32 and ilp32f; a test keeps it in step with the tables. - tests/conformance/c/riscv-float-abi.c is the source the policies were measured on. * feat(ghidra): admit RISC-V ilp32 ELF with a derived calling convention ESP32-C2, C3, C5, C6, C61, H2 and H21 builds are ELF32 soft-float (ilp32) images, which the RISC-V lane refused at its double-float check. Ghidra 12.1.4's generic RV32 compiler specification models ilp32d, so its stock prototype would read float and double arguments from fa0-fa7. - Target resolution reads the one SHT_RISCV_ATTRIBUTES section through the same bounded ELF declaration reader as range validation, and records Tag_RISCV_arch as declared, absent or unavailable. - Admission selects a lane from the class and float-ABI table. The soft-float lane also requires an RV32I Tag_RISCV_arch that declares only extensions Ghidra decodes with p-code; the refusal names any others (Zcb, Zcmp, Zcmt, Zb*, xesp*). - Soft-float profiles commit float_abi, arch, riscv_calling_convention ilp32-prototype-v1 and the lane elf32-exec-le-soft-float-generic-v1. Double-float profiles are unchanged. - ReaGhidraRiscvCallingConvention.java runs after analysis. It derives __rea_ilp32 from the default prototype without floating-point entries and rules, with fs0-fs11 killed by calls, self-checks its storage and assigns it to every non-thunk function, external function and function definition. Its report is written last. - The provider client requires that report exactly when the profile commits the policy, before any operation, and keeps it as session limitations and caller-project history. ELF32 single-float (ilp32f), quad-float and ELF64 soft- or single-float images stay refused, now with a reason built from the lane table. * test(ghidra): verify RISC-V ilp32 against real Ghidra verify:ghidra:riscv now also builds tests/conformance/c/riscv-float-abi.c with ESP32-C3's -march=rv32imc_zicsr_zifencei -mabi=ilp32, typed with DWARF and linked again with --strip-all. Every allocated section of the stripped image must keep its address and bytes. - riscv-checks.mjs holds the ilp32 storage of each fixture function together with the words the compiler's call site writes there. An LLVM objdump reading of the caller must find every word at its stated location: an unaligned named double, the a7/stack split, two floats in an integer pair, a large structure by reference, aligned variadic pairs and an indirect call. - Through CLI and stdio MCP, the soft-float builds must report the soft-float lane and ilp32-prototype-v1, __rea_ilp32 with exactly that storage (one integer word per location when stripped), caller pseudocode that passes the source constants and names no floating-point register, the prototype outcome in retained Evidence, and decoded fence.i and csrr. - The RV32 double-float image relabelled as soft-float, refused before, is admitted through the new lane. RV64 soft-float, single-float and unsupported bits are refused with their reasons. - riscv-checks.test.ts runs the checks offline against recorded LLVM and Ghidra output, including Ghidra's stock ilp32d prototypes as a negative control. * fix(ghidra): pass RISC-V ilp32 values over 8 bytes by reference The psABI passes any value wider than 2 x XLEN by reference, scalars included. __rea_ilp32 converted only structures and unions over 8 bytes to pointers, so a long double (16 bytes) was joined into a1-a4. Measured on Ghidra 12.1.4, rea_riscv_abi_quad(long double value, int sel) read value from a4:4,a3:4,a2:4,a1:4 and sel from a5, where clang passes a pointer in a1 and sel in a2. - ReaGhidraRiscvCallingConvention.java now has one derivation driven by a per-policy row (floating-point entry size, preserved floating-point registers). Every policy inserts a rule before the input list's final join that converts any value of 9 bytes or more to a pointer; a return that size already becomes a caller buffer. The self-check requires a 16-byte value to pass by reference. - The fixture gains rea_riscv_abi_quad. The call-site oracle now checks the words the by-reference pointers point at, for the large structure and the long double, and the recorded data comes from an LLVM 19 build. - docs/ghidra-riscv.md: the decision record and the ilp32 policy row. ilp32-prototype-v1 keeps its id: no release carries it yet. * test(ghidra): check RISC-V lane limitations by their facts The limitation tests froze a full explanatory sentence, the note count and array positions, so a harmless rewording failed them: the pattern Greptile flagged on #7486. Assert the facts the notes must carry instead: the pinned language, the static-linkage boundary, one note naming the declared ABI, the ABI the pinned language models and the prototype REA applies, and the declared ISA. * test(ghidra): check return storage in stripped RISC-V ilp32 builds The stripped branch of assertRiscvIlp32Storage compared parameter storage only, so an inferred return that lost the high word of a register pair, such as rea_riscv_abi_wide at a0:4 instead of a1:4,a0:4, still passed. Stripped returns must now use the typed build's storage, or none when the result goes through caller memory. Recovered type names are still not compared: a test accepts other type names and refuses the dropped high word. * docs(ghidra): record decision D3 as decided in #7477 |
||
|
Entrpi
|
e6d0618979 |
feat(artifacts): inventory and extract ESP flash layouts (#8228)
* feat(artifacts): inventory and extract ESP32 firmware images ESP32-family app and bootloader images (magic 0xE9) had no reader, so open_binary refused OTA downloads, app partitions and bootloaders. Add the esp-image container format behind inspect_artifact and extract_artifact. A pure decoder reads each byte once and keeps only fixed-size fields. Derived facts cover the chip, from a table cited to ESP-IDF v6.1; segment classification as ESP-IDF's should_map and should_load; checksum and appended SHA-256 results; app and bootloader descriptors; Secure Boot v2 signature blocks, observed and unverified; and undescribed trailing byte ranges. Truncation and undescribed bytes make coverage unknown. ESP8266 images are refused with a specific reason, and ciphertext stays unrecognized. Add the format to every target, Evidence, snapshot, live-summary and artifact vocabulary, document it, and add an optional verify:esp-images lane that compares esptool-built images with esptool image-info and espsecure signature-info-v2. Refs #3615 * test(artifacts): run the ESP image lane on esptool v5.6.0 The lane compared descriptor fields only when esptool image-info printed one, so a descriptor that REA reports and esptool does not print passed without comparison. Compare the descriptor kind in both directions. Record that the lane passed for all 13 images with esptool and espsecure v5.6.0, the version the layout facts cite, as well as with 4.12.0. * refactor(artifacts): define ESP image layout sizes once The decoder and the derived facts each defined the segment-header, appended-digest and flash-sector sizes, the erased-byte scan and the offset formatter. Define them once with the other fixed-layout structures and import them in both places. The signature slot count is the one constant that drives the slot scan. Correct the observedEnd comment: it is the end of the last chunk read. * fix(artifacts): report ESP-IDF's segment length limit by chip ESP-IDF's verify_segment_header rejects a segment of 16 MiB or more unless the chip maps flash through 32-bit cache addresses (SOC_SPI_MEM_SUPPORT_CACHE_32BIT_ADDR_MAP: ESP32-S3, P4, C5, C61, H4 and S31 in v6.1). Record that capability in the chip table and add a limitation for such segments; unknown chips get none. `loaded` repeats should_load. It does not mean the bootloader accepts the segment: verify_load_addresses still rejects one outside the chip's RAM, which REA does not model. Say so in the docs, the skill reference and the inventory limitation. * test(artifacts): check ESP lane tools first and verify extraction The lane started work before checking its tools, so a missing RISC-V target, ld.lld, esptool or espsecure surfaced as a raw spawn or command error. Check each one before building and name the variable to set. Warn when esptool is not v5.6.0, the release the layout facts cite. Assert each segment's class for the ESP32-C3 app, its variants and the bootloader, as the Xtensa image already did. Extract the app and the signed image through the CLI and compare every file with the bytes at esptool's segment offsets or in the appended signature sector; the source's SHA-256 must be unchanged and equal the reported root_sha256. * docs(artifacts): state what the ESP image lane does not establish The Verification section listed the lane's checks but not its gaps: only ESP32-C3 and ESP32 images come from a real producer, no real project build, OTA download or flash dump is inspected, no signature is verified, and the lane does not run in CI. Record the timing of the final lane run on esptool v5.6.0. * fix(artifacts): compare ESP segment declarations Segment entries hash only their data, so compare_artifacts reported a segment that moved to another load address with the same bytes as unchanged; only the image itself changed. Compare each segment's load_addr and data_len as its declaration, as Node SEA and PyInstaller members are compared, and count a changed declaration as a metadata change, also when content is unavailable. Header offsets are layout and the classification is derived, so neither is compared. * test(artifacts): run ESP lane commands as owned process trees The lane ran its tools through execFile. A timeout killed only the direct child, so a linker or assembler that Clang or GCC had started could keep running while the lane removed its workspace. Run every command through createVerifierCommands, check the tools inside withVerifierCleanup, and remove the workspace only after commands.close() has stopped every tree; a failed close keeps the workspace. Only a nonzero exit now counts as a refusal or an expected esptool failure; timeouts and spawn failures propagate. Supervising each command roughly doubles the lane's run time. * test(artifacts): cancel ESP decoding after its first decoded chunk The cancellation test requested cancel or close on setImmediate, which can land while the file is still being opened and checked, so it never showed that decoding had started. It also passed against a reader that ignored both requests while decoding. Acknowledge the decoder's first chunk, then cancel or close, join the operation, and assert that no later chunk reaches the decoder and that the reader ends closed. * docs(artifacts): record which ESP metadata comparison ignores Segments are compared by their declaration. The image's and the signature blocks' format_metadata are not: every field comes from their own bytes, which content comparison already covers, and a block's image_digest_matches follows the image's content. * feat(artifacts): inventory and extract ESP flash layouts A whole-flash dump of an ESP32-family chip was read as its bootloader image, with the partition table, apps and data left as undescribed bytes, or refused when it began with erased flash. Read it as an esp-flash-image container instead. The partition table is decoded as esp_partition_table_verify accepts it and is found at the caller's partition_table_offset, at the default 0x8000, or by scanning 0x1000-aligned sectors for a table whose MD5 verifies, which the inventory reports as derived. The bootloader is located at its chip's ROM bootloader offset. Each partition is an entry with its declaration in flat format_metadata; the bootloader and app partitions carry ESP image facts from the existing streaming decoder; the otadata records are reported with a selection derived without rollback or anti-rollback, and the running partition stays unknown. One unreadable partition does not hide the others, and extraction writes every partition, nvs_keys included, as ordinary data. Flash layouts are detected before single ESP images. The scan runs only when an ESP image sits at a ROM bootloader offset, and is refused when a bootloader at another chip's offset shows that the dump does not start at flash address 0. open_binary gains flash_offset and partition_table_offset, which require format esp-flash-image and are part of the snapshot target identity; the CLI takes them with --target-format esp-flash-image. Comparison treats a partition's label, type, subtype, range and flags as its declaration. * test(artifacts): verify ESP flash layouts with gen_esp32part, merge-bin and otatool Add the optional verify:esp-flash lane. It builds default and relocated partition tables with ESP-IDF's gen_esp32part.py, assembles dumps with esptool merge-bin, including one written from flash 0x1000, and writes OTA records with otatool through flash_file_tool.py, which replaces parttool's esptool calls with reads and writes of the dump file. It compares REA's bootloader and app facts with esptool image-info, every partition with gen_esp32part.py's decoding of the extracted table, and the OTA records and derived slot with otatool read_otadata after each switch. It also checks extracted bytes and source integrity, the scan and caller table offsets, flash_offset on the 0x1000 dump, a corrupted app beside valid partitions and a truncated dump. ESP32 linker scripts place the conformance bootloader and app in ESP32 address ranges. * fix(artifacts): read ESP flash layouts as the bootloader does Read the partition table as bootloader_utility_load_partition_table does: the last factory, otadata and same-subtype OTA declaration wins, and a limitation names repeats. Plan each otadata record within its partition and the dump. An otadata partition smaller than two sectors selects none, the bootloader's INVALID_INDEX; a record the dump lacks leaves the selection unknown while the other record stays reported. The fallbacks follow the bootloader's indexes: ota_0-fallback and factory-fallback name no partition when the table declares none, and the assumptions add GPIO holds and deep-sleep wake. Refuse a dump whose bootloader contradicts its origin before the default table offset is tried, whether flash_offset was omitted or given, and keep the ESP image reader's ESP8266 refusal. A table never lies inside the located bootloader image, so single app images are no longer scanned. An explicit esp-flash-image selection is decided before other container signatures, and the native commands' --target-format no longer takes it. Say when no bootloader confirms the default origin. Exempt the primary bootloader and partition_table entries of ESP-IDF 5.3 from placement limits, report every region inside a longer one, and name partitions by index when labels repeat ignoring case. Keep an undecodable label's bytes in label_hex and compare them. Word undescribed ranges as starting at a byte that is not erased, and treat a short erased region as erased. Add an MCP compare_artifacts test for partition declarations, reader identity and close tests, scan edge cases, the 0x2000 ROM offset and a literal otatool CRC. * test(artifacts): verify ESP primary layout entries and the otadata fallback Build a table with the primary bootloader and partition_table entries that gen_esp32part.py fills from --primary-bootloader-offset, and check that REA reads it with complete coverage, no overlap or misplacement limitation, and a table that agrees with gen_esp32part's decoding. Erase the otadata with otatool after the slot switches and check the records against read_otadata and the factory fallback. * fix(artifacts): keep ESP flash coverage complete for a dump cut short An esptool merge-bin dump ends after the last part it wrote, usually inside its last partition. Such a partition made the layout's coverage unknown, so artifact comparison of two merge-bin dumps reported every unchanged partition as unknown. The partition table names every partition whatever the file's length, so a partition cut short or left out by the end of the file hides no entry. As the ESP image reader does for a cut-short segment, its entry holds the bytes the file has and an entry limitation gives the declared size; a partition with no bytes stays in partitions_outside_dump. Coverage stays complete unless undescribed bytes remain or no bootloader is located. An MCP compare_artifacts test of two dumps that end inside ota_0 now classifies every partition, and fails on the old rule. * fix(artifacts): report written bytes in ESP partitions, erased slots included An app slot is erased for ESP-IDF's bootloader when its image header bytes are erased flash, and REA reported only that status. A slot can still hold data further in: a dump read from a device held a complete older app image 1 MiB into an OTA slot reported as erased, with no limitation. Each partition entry now records where the bytes the dump holds are written, relative to the partition: written_offset, the first byte that is not 0xFF, written_end, just past the last, and written_bytes, how many are not 0xFF. They are null for a partition outside the dump. The status stays erased, which is the bootloader's view, and a limitation names an erased app slot that holds written bytes and says they are not read as an image. The reader reads every partition byte once more to find them. * fix(artifacts): derive the ESP boot selection for every partition table The boot selection lived inside otadata, so a table without an otadata partition, the common factory-only layout, had none. ESP-IDF's bootloader_utility_get_selected_boot_partition returns the factory index when the table declares no otadata partition. Move it to a top-level boot_selection that every layout has. Its basis names the bootloader's branch: otadata, factory-fallback and ota_0-fallback from the records, no-otadata for a table without otadata, none for the bootloader's INVALID_INDEX, and unknown when the dump lacks a record the bootloader reads, which replaces the null selection. running_partition moves with it, and otadata keeps its partition index and records. The assumptions name what each branch leaves out: no records assumption without otadata, and none for none or unknown. The real-tool lane checks no-otadata on a table gen_esp32part built without otadata. * fix(artifacts): say when an ESP image is a flash dump read from 0x1000 An ESP32 dump read from flash 0x1000, as read-flash 0x1000 or merge-bin --target-offset 0x1000 writes it, opened automatically as its bootloader image with a megabyte of undescribed bytes and no next probe. The flash reader refuses that origin, but nothing told an agent which flash_offset to pass. espFlashDumpOrigin owns the inference: an image of a chip whose ROM loads the bootloader from flash R, at file offset F, is the bootloader of a dump read from R - F when a partition table sits where that origin puts 0x8000. The ESP image reader asks it for a non-app image at file offset 0, so the result does not depend on how the reader was created, and records flash_dump {flash_offset, table_file_offset}, null otherwise, with a limitation that says to reopen the file as esp-flash-image with that flash_offset. inspect_artifact proposes the same open_binary call as a next probe, and the flash reader's misplaced-origin refusal names the offset. * perf(artifacts): summarize nested ESP images in the flash layout summary view The summary view returned an ESP flash layout's root facts whole, so each app partition and the bootloader carried every segment and signature block, about half of a dump's summary. The summary view now gives each nested image a summary instead of its facts: status, chip, entry point, segment count, load ranges (each mapped, loaded or unclassified segment's address and declared length), the checksum and appended SHA-256 statuses, coverage and limitations. Complete detail keeps the full facts, every segment included, and the retained Evidence is unchanged. On a 16 MiB device dump the summary's root falls from 5,141 to 2,657 bytes, and the summary Evidence from 10,038 to 7,554. * test(artifacts): expect the misplaced image the ESP bootloader search reports |
||
|
morluto
|
245a581818 |
feat(kuna): verify release 1.986 and guide version selection (#8187)
* feat(kuna): verify release 1.986 and guide version selection * test(kuna): cover verified release doctor guidance |
||
|
morluto
|
ec107022fa | docs: guide audits of existing investigation tools (#8272) | ||
|
morluto
|
356cc50475 |
docs: guide design and verification of provider workflows (#8261)
* docs: guide provider workflow design and annotation comparisons * docs: connect provider tool design to workflow verification |
||
|
Entrpi
|
19ab239278 |
ci(ghidra): probe M-profile ARM and Xtensa decoding (#7482)
* ci(ghidra): probe M-profile ARM and Xtensa decoding The embedded plan relies on ARM:LE:32:Cortex, ARM:LE:32:v8-m and Xtensa:LE:32:default, but CI only exercised ARM:LE:32:v7. Import source-owned bytes for each language at 0x1000 with -noanalysis and check how they decode from the language's default context: Cortex and v8-m must start in Thumb state, v8-m must decode the ARMv8-M MSPLIM register, and Xtensa must decode entry and retw.n. Add the probe scripts the job runs, including the existing ARM context probe, to the ghidra_bridge change scope. * ci(ghidra): match decode-probe operands as complete operands The operand expectation searched the instruction text for a substring, so mrs/4/msp also accepted mrs r0,msplim. Compare it with each operand Ghidra reports for the instruction, so only a complete operand matches. The Cortex (msp, lr), v8-M (msplim, lr) and Xtensa (a1) expectations are unchanged. |
||
|
morluto
|
cd878ad57d |
refactor(ci): unify lane ownership and guard Python tests (#8150)
* refactor(ci): unify lane ownership and guard Python tests * fix(ci): reject entirely skipped bridge subtests |
||
|
morlutoandCursor Agent
|
77b09314ff |
test(ci): require provider checks and reduce test startup (#8145)
* test(ci): require provider coverage and reuse test sessions * fix(ci): prepare the runtime before Python producer tests The fast Python job ran the suite on a fresh checkout, so analysis-efficiency checks failed without compiled process modules. Install locked dependencies and prepare the runtime before the suite. * fix(ci): keep provider test reports inside the checkout The provider-report reader accepted an arbitrary path from argv. Resolve the path and reject anything outside the checkout before reading it. The packet and signal workflows write those reports under .cache. * fix(ci): require a test module in each named Python bridge family Discovery still runs every bridge test. The mitmproxy, native, and pwndbg families must each keep a test module, so deleting one cannot pass through the others. --------- Co-authored-by: Cursor Agent <[email protected]> |
||
|
|
860fdd8fbe |
feat(macho): report per-slice encryption status in inspect_macho (#5359)
* refactor(macho): extract non-throwing container and slice load-command readers Split readMachoImage into readMachoContainer, which classifies a file as thin, FAT or not Mach-O and keeps FAT range issues per slice, and readSliceLoadCommands, which reads one slice header and frames its load commands, returning every command framed before a failure. readMachoImage keeps its strict policy on top of them: commands framed before a framing defect still decode first, an unsupported FAT slice still yields to a later malformed one, and the first FAT range issue is still reported. A characterization test pins that precedence for inputs with several defects; it passes against both the previous and the new implementation. Part of #4306. * feat(macho): report per-slice encryption status in inspect_macho inspect_macho now reports the LC_ENCRYPTION_INFO(_64) state of every slice in a new `encryption` collection, read in process from load commands without reading code bytes. - Each slice is encrypted, not-encrypted, absent, malformed or unknown. Absent requires a completed load-command walk; malformed and unknown carry a reason and keep commands read before the failure. - The named range is end-exclusive in slice and input-file coordinates and maps to file-backed segment fragments, VM addresses and the overlapping nonempty sections. - A ciphertext range adds a limitation naming its input-file bytes. - The in-process read and the native tools are bracketed by one regular file descriptor and stat identity checks; a change reports artifact_changed. A missing or inaccessible target keeps the tool facts and reports encryption as non-exhaustive, never absent. - Retained Evidence recorded before this field is parsed by the owning schema and reports encryption as unknown in the macho view and in managed/native verification. Fixes #4306 * fix(macho): keep slice facts and failures under large tables and close errors - Frame load commands through a visitor instead of retaining every command, restoring the pre-refactor memory profile (2M eight-byte commands: peak heap 370 MiB -> 0.9 MiB). - Confirm a large FAT container from a valid slice's Mach-O header so a sibling's range fault stays per slice instead of hiding healthy slices. - Name only sections whose bytes the range and their segment both cover. - Throw inspection failures inside the owned descriptor callback so a close failure keeps them as its primary error. * fix(macho): map encrypted ranges only within segment VM extents mapRange gave every file-backed segment fragment VM addresses, even for a segment with no VM size, whose file bytes dyld never maps, and past vmsize when a segment declared more file bytes than memory. Read vmsize with the segment. A segment with no VM size keeps its fragment with null VM addresses. A file size beyond the VM size, or a VM range past the address space, makes the mapping unknown with a reason while the slice keeps its encryption status. The docs also claimed the encryption read never reads code bytes. Like every Mach-O container read, a 0xcafebabe input of at most 16 MiB is read whole to rule out a JVM class file; say so. * fix(macho): read encryption through the admitted descriptor facts Main's withArtifactRegularFile now hands callers the admitted handle and its initial stats. Use those facts instead of re-stating the handle, and pass the legacy Evidence payload straight to createEvidence. * fix(macho): retain a finished report when the target close fails The file owner keeps the completed callback value when the descriptor close fails or cancellation arrives during close. inspect_macho now validates that value and retains the report as the error's partial observation, as inspect_signature does, instead of returning only the cleanup or cancellation error. * test(native): supply readEncryption in merged dyld refusal cases main's new dyld refusal cases build NativeMachoInspectionContext without the encryption reader this branch requires. * test(views): bind the Mach-O encryption page to its facet main's facet-ownership case builds inspect_macho Evidence without the encryption coverage this branch requires; the fixture now carries one slice row and the loop checks the encryption page like the other facets. * fix(macho): bound encryption commands and range mapping across slices A slice retains at most two encryption load commands, since a second one already makes it malformed; the reason keeps the full count. Range mapping in every slice shares one budget of segment and section records, charged from each segment command's capacity before it is decoded. A slice past the budget keeps its encryption status and reports an unknown mapping. * style(macho): format encryption reader and binding test * fix(macho): stop retaining segments past the mapping budget Segment commands are collected during framing only while their mapping records still fit in the budget shared by all slices, and the budget is charged only when the slice maps its range. * docs(macho): clarify retained mapping allowance * fix(ci): own the Mach-O encryption test in Apple scope --------- Co-authored-by: t <t@t> Co-authored-by: gensokyo-dev <[email protected]> |
||
|
gensokyo-dev
|
f60e42a9ae |
fix(mcp)!: bound validation work before tool dispatch (#7905)
Reuse the pinned SDK's combined array-element and object-member admission before canonical validation, deferred runtime loading and tool execution. Share a configurable finite allowance across eager/deferred HTTP and stdio, and project refusals to bounded, actionable transport resource errors. Preserve canonical CLI/application contracts, caller-selected input values, ordinary validation semantics and complete Evidence access. Exercise actual transport boundaries, real artifact/TShark workflows, packaged installation, continued service and downstream identity/capacity regressions. Fixes #7882 BREAKING CHANGE: MCP tool arguments now default to 1,024 combined array elements and object members per call. Select a larger positive finite REA_MCP_MAX_INPUT_ELEMENTS allowance and restart for larger inline inputs, or use retained/file selectors where supported. Inputs are never truncated. Assisted-by: Reimu (gensokyo-skills:reimu-incident-triage) Assisted-by: Kogasa (gensokyo-skills:kogasa-surprise-testing) Assisted-by: Yukari (gensokyo-skills:yukari-boundary-analysis) |
||
|
Entrpiandgensokyo-dev
|
64196b08ca |
feat(artifacts): inventory and extract ESP32 firmware images (#7498)
* feat(artifacts): inventory and extract ESP32 firmware images ESP32-family app and bootloader images (magic 0xE9) had no reader, so open_binary refused OTA downloads, app partitions and bootloaders. Add the esp-image container format behind inspect_artifact and extract_artifact. A pure decoder reads each byte once and keeps only fixed-size fields. Derived facts cover the chip, from a table cited to ESP-IDF v6.1; segment classification as ESP-IDF's should_map and should_load; checksum and appended SHA-256 results; app and bootloader descriptors; Secure Boot v2 signature blocks, observed and unverified; and undescribed trailing byte ranges. Truncation and undescribed bytes make coverage unknown. ESP8266 images are refused with a specific reason, and ciphertext stays unrecognized. Add the format to every target, Evidence, snapshot, live-summary and artifact vocabulary, document it, and add an optional verify:esp-images lane that compares esptool-built images with esptool image-info and espsecure signature-info-v2. Refs #3615 * test(artifacts): run the ESP image lane on esptool v5.6.0 The lane compared descriptor fields only when esptool image-info printed one, so a descriptor that REA reports and esptool does not print passed without comparison. Compare the descriptor kind in both directions. Record that the lane passed for all 13 images with esptool and espsecure v5.6.0, the version the layout facts cite, as well as with 4.12.0. * refactor(artifacts): define ESP image layout sizes once The decoder and the derived facts each defined the segment-header, appended-digest and flash-sector sizes, the erased-byte scan and the offset formatter. Define them once with the other fixed-layout structures and import them in both places. The signature slot count is the one constant that drives the slot scan. Correct the observedEnd comment: it is the end of the last chunk read. * fix(artifacts): report ESP-IDF's segment length limit by chip ESP-IDF's verify_segment_header rejects a segment of 16 MiB or more unless the chip maps flash through 32-bit cache addresses (SOC_SPI_MEM_SUPPORT_CACHE_32BIT_ADDR_MAP: ESP32-S3, P4, C5, C61, H4 and S31 in v6.1). Record that capability in the chip table and add a limitation for such segments; unknown chips get none. `loaded` repeats should_load. It does not mean the bootloader accepts the segment: verify_load_addresses still rejects one outside the chip's RAM, which REA does not model. Say so in the docs, the skill reference and the inventory limitation. * test(artifacts): check ESP lane tools first and verify extraction The lane started work before checking its tools, so a missing RISC-V target, ld.lld, esptool or espsecure surfaced as a raw spawn or command error. Check each one before building and name the variable to set. Warn when esptool is not v5.6.0, the release the layout facts cite. Assert each segment's class for the ESP32-C3 app, its variants and the bootloader, as the Xtensa image already did. Extract the app and the signed image through the CLI and compare every file with the bytes at esptool's segment offsets or in the appended signature sector; the source's SHA-256 must be unchanged and equal the reported root_sha256. * docs(artifacts): state what the ESP image lane does not establish The Verification section listed the lane's checks but not its gaps: only ESP32-C3 and ESP32 images come from a real producer, no real project build, OTA download or flash dump is inspected, no signature is verified, and the lane does not run in CI. Record the timing of the final lane run on esptool v5.6.0. * fix(artifacts): compare ESP segment declarations Segment entries hash only their data, so compare_artifacts reported a segment that moved to another load address with the same bytes as unchanged; only the image itself changed. Compare each segment's load_addr and data_len as its declaration, as Node SEA and PyInstaller members are compared, and count a changed declaration as a metadata change, also when content is unavailable. Header offsets are layout and the classification is derived, so neither is compared. * test(artifacts): run ESP lane commands as owned process trees The lane ran its tools through execFile. A timeout killed only the direct child, so a linker or assembler that Clang or GCC had started could keep running while the lane removed its workspace. Run every command through createVerifierCommands, check the tools inside withVerifierCleanup, and remove the workspace only after commands.close() has stopped every tree; a failed close keeps the workspace. Only a nonzero exit now counts as a refusal or an expected esptool failure; timeouts and spawn failures propagate. Supervising each command roughly doubles the lane's run time. * test(artifacts): cancel ESP decoding after its first decoded chunk The cancellation test requested cancel or close on setImmediate, which can land while the file is still being opened and checked, so it never showed that decoding had started. It also passed against a reader that ignored both requests while decoding. Acknowledge the decoder's first chunk, then cancel or close, join the operation, and assert that no later chunk reaches the decoder and that the reader ends closed. * docs(artifacts): record which ESP metadata comparison ignores Segments are compared by their declaration. The image's and the signature blocks' format_metadata are not: every field comes from their own bytes, which content comparison already covers, and a block's image_digest_matches follows the image's content. * docs(artifacts): state that ESP mmu_page_size is a base-2 logarithm The app descriptor's mmu_page_size keeps ESP-IDF's name and value, the MMU page size in log base 2 as esp_app_desc_t declares it, so 16 means 64 KiB. Say so in docs/esp-image.md and the schema comment; a reader could otherwise take 16 as 16 KiB. * fix(esp-image): retain member consumption and share lifecycle regressions * test(artifacts): scope MCP catalogs and budget full-schema validation * test(artifacts): verify compact output profiles and document bounded catalogs * test(artifacts): close readers when cleanup-test admission fails * test(artifacts): compare ESP image limitations with their owners The decoder and reader tests froze complete limitation sentences, so a harmless rewording failed them. The wording now has two owners: ESP_IMAGE_LIMITATIONS in espImageFacts.ts, which the decoder, the inventory and the reader use, and ESP_DESCRIPTOR_LIMITATIONS beside the descriptor decoder. Tests compare delivered limitations with the owner, called with the facts each case establishes (segment index, declared and available bytes, offsets, chip, page range), next to the structured assertions they already make. Delivered text is unchanged. * test(esp-image): assert structured facts and owned guidance * fix(esp-image): keep spi_flash_mode and descriptor fields before a truncated tail ESP-IDF v6.1's esp_app_desc_t declares spi_flash_mode at byte 181, after mmu_page_size, as the committed reference layout shows; the descriptor now keeps it as stored. The decoder also required all 256 descriptor bytes, though it reads none past byte 181, so a file ending in the reserved tail lost the version, project, digest and revision facts it does hold. The app descriptor now needs only its fields; the segment's truncation is reported as before. * test(esp-image): pin the P-192 public key digest to espsecure's layout espsecure packs ECDSA signature blocks as "<BBBx32sB64s64s1031x" and zero-pads a 48-byte P-192 point to 64 bytes, so the signature starts at byte 101 for every SHA-256 curve. signature_info_v2 hashes sig_block[36:101] and _digest_sbv2_public_key the curve ID with the padded point. No test built a P-192 block; this one checks public_key_sha256 against that definition. * fix(esp-image): keep each app descriptor field the file holds in full A file can end anywhere inside the 256-byte esp_app_desc_t. The decoder needed every field byte, so a cut at byte 176 dropped the version, project and app_elf_sha256 it does hold. Once the 4-byte magic identifies an app descriptor, each field is now kept when the file holds all of its bytes and is null otherwise; observed_length records where the descriptor ends, so a cut field is told apart from text that is not UTF-8. The fields after the magic become nullable in the metadata schema. * fix(esp-image): report a signature block whose image digest does not match A CRC-valid Secure Boot v2 block can carry another image's digest. The block recorded image_digest_matches false, but no limitation said so, unlike a checksum or appended-SHA mismatch, so CLI and MCP summaries showed only the generic unverified note. The mismatch is now a limitation; coverage stays complete and signature_status stays unverified. The three other signature-sector limitations move into ESP_IMAGE_LIMITATIONS with their text unchanged, so all ESP image wording has one owner. * refactor(esp-image): remove redundant descriptor byte guards --------- Co-authored-by: gensokyo-dev <[email protected]> |
||
|
gensokyo-dev
|
6eb37a7749 |
fix(ci): restore qualification and reduce validation overhead
Use one typed-checker worker by default without changing semantic scope or caller overrides. Reuse the bounded CLI worker for pure argument validation while retaining fresh-process coverage and all upstream cases. Keep registration tests independent of real package-runner downloads while asserting the actual prewarm command and startup context. Pass the advertised binary_session definition by name to package verification, retaining default SDK validation and complete catalog/contract checks while avoiding compilation of unrelated tool outputs. Preserve all four installed-tarball scenarios and existing deadlines. Strengthen Hermes preview operation/backup assertions and clarify physical-path routing documentation. Assisted-by: Suika (gensokyo-skills:suika-scatter-gather-planning) |
||
|
|
1201c25542 |
fix(ci): align static checks and regression fixtures with current contracts (#7349)
* fix(ci): retire unused internal analysis declarations * test: use an admitted app bundle for provider routing (cherry picked from commit |
||
|
gensokyo-devand555aaditya
|
03f17cf41d |
feat(android): apply caller-selected R8 mapping evidence (#7456)
Apply caller-selected R8/ProGuard mappings to shared Android inspections, preserving raw identities, exact verification, ambiguity and provenance. Add retained summaries and bounded/cached parsing, with measured real R8/Conscrypt workflows and review fixes. Fixes #6329. Co-authored-by: 555aaditya <[email protected]> |
||
|
gensokyo-dev
|
2817d30ab2 |
fix(docs): preserve verification reports after experiment archival (#7462)
Move the retained allocation report and unchanged receipt JSON into verification documentation. Keep one report producer, repair current and archived links, document test-support ownership, and remove retired lint inputs. Fixes #7458. Validated focused/Python checks, report consistency, docs rendering, and real Bash CLI/MCP facts against an independent oracle. Greptile reviewed all seven files without comments. CI source/Windows/shard-8 failures match clean-main or existing-main controls; the requester authorized merging with those established failures. Codex review quota was unavailable. |
||
|
morluto
|
365fb03e73 | chore: archive analysis efficiency experiments | ||
|
Saim Shafiqueandgensokyo-dev
|
0d1a454e25 |
fix(ghidra): admit newer 12.x releases as compatible builds (#6924)
Admit same-major Ghidra releases at or above the verified minor through the existing compatible/unverified policy. Preserve the observed version, verified12.1.4 pin, installation-declared Java floor and separate NativeAOT verification gate. 367 targeted tests/static/docs, all10 real Java bridge-source compilation, and actual Linux12.2/JDK25 CLI/MCP queries and cleanup pass. JDK21 remains refused for12.2. Exact-main controls reproduce all3 unrelated CI assertions; canceled shards and an apt-blocked binary-layout lane remain unverified. Co-authored-by: gensokyo-dev <[email protected]> |
||
|
|
2c38becc4e |
feat(binary-layout): admit ELF32 i386, ARM and MIPS layout profiles (#7052)
* feat(binary-layout): admit ELF32 i386, ARM and MIPS layout profiles inspect_binary_layout refused every ELF32 object, so router and IoT firmware (ARM32, MIPS32) got no layout facts at all. The pwntools adapter now admits EXEC/DYN/REL objects in these profiles: ELF64 x86-64, ELF32 i386, ELF32 little-endian ARM and ELF32 little- or big-endian MIPS. Header fields are read in the declared byte order, the ELF32 header is bounds-checked at 52 bytes, and pwntools' own reported architecture, class and byte order must match the admitted profile. Other machines, classes and byte orders keep an unsupported_target refusal that names the admitted profiles. The output schema admits exactly those architecture combinations, and the Evidence subject now names the reported machine instead of always claiming x86_64. verify:binary:layout cross-compiles i386, ARM, MIPS and MIPSel executables and relocatable objects (never executed) and checks every section name, address, offset and size against GNU readelf through CLI and MCP. The real-binary-layout workflow installs those toolchains. Refs #6934 Co-Authored-By: Claude Opus 5.5 <[email protected]> * ci(binary-layout): install target libc headers for ELF32 fixtures --no-install-recommends skips the libc6-dev-*-cross packages the cross compilers only recommend, so the i686 fixture fell back to the host's /usr/include and failed to find bits/libc-header-start.h. Install the four target libc development packages explicitly. Refs #6934 Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(binary-layout): size the minimum symbol entry by ELF class The relocation reference check required a 24-byte Elf64_Sym entry, so every positive symbol reference in an ELF32 report was refused as unreadable output. Elf32_Sym is 16 bytes; take the minimum from the reported class. A 64-bit table with 16-byte entries is still refused. Refs #6934 Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(binary-layout): compare ELF32 symbols and relocations with readelf --------- Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: morluto <[email protected]> |
||
|
Hahaa909andmorluto
|
9750486e78 |
fix(mcp): normalize compact schemas for strict gateways (#3806)
* fix(mcp): make compact schemas gateway compatible * test(mcp): bind input precision assertions to the advertised profile * test(mcp): own schema assertions and validator lifetimes --------- Co-authored-by: morluto <[email protected]> |
||
|
morluto
|
4c680d099c | fix(lint): preserve portable advertised patterns through autofix | ||
|
morluto
|
c8269ca497 |
fix(ci): own mock isolation and normalize Git patch directories
Convert native directory separators only at Git patch argument production. Exercise fresh, applied and invalid dependency patches in private copies, preserving protectNTFS, and select the regression for Windows CI. Replace the spelling-based mock inventory with one binding-aware Oxlint rule. Canonical test projects own isolation for both Vitest and lint; external temporary CLI probes avoid concurrent test discovery. Validated actual local Git and Oxlint execution, resolved-policy parity, CI selection and report tests. Native Windows verification remains with CI. Refs: #7050 |
||
|
morluto
|
f3ed1657f8 |
feat(mcp): support runtime analysis limit recovery (#7049)
* feat(mcp): support runtime analysis limit recovery * refactor(config): remove unused limit re-exports * fix(android): keep fixture release metadata dependency-free * fix(settings): preserve provider budgets and recovery facts * fix(tests): resolve lint plugins from their repository owner * refactor(mcp): verify guidance delivery without frozen wording * fix(android): align availability with runtime timeout * test(recovery): assert limit facts instead of frozen wording * fix(mcp): share runtime settings with the default JADX provider * test(mcp): bound acknowledgements and release held work on failure * fix(catalog): declare runtime settings tools as MCP-only |
||
|
morluto
|
ab720b4216 | test(evaluation): preserve partial coverage in the Notes oracle | ||
|
morluto
|
627c085cee | chore(lint): enforce reviewed promise and fallback contracts | ||
|
morluto
|
76afdd35ae | chore(lint): guard validation ownership and asynchronous assertions | ||
|
morluto
|
475f09d5ca | chore(lint): enforce buffer and cancellation regression guards | ||
|
morluto
|
9140deae63 |
docs: clarify ownership and cleanup review guidance
Clarify structural parsing, admitted facts and freshness, useful facade and compatibility responsibilities, immutable JSON proof, and partial Evidence retention. Document fixture cleanup ownership, semantic fault injection and the limits of static checks; extend the existing PR owner hint. Correct the session-record architecture label. Validation: docs:check (including nine README consistency tests), rendered docs:build with link validation, touched Markdown formatting and git diff --check passed. Source and installed Vitest cleanup behavior were reviewed. No runtime behavior changed. |
||
|
morluto
|
efab65c2f9 |
docs: codify boundary regression checks
Turn recurring parser and projection failures into a selectable regression guide covering format assumptions, raw identity, missing values, partial work, independent limits, representation expansion, snapshots, and public delivery. Link it from the agent instructions and PR template without adding checkboxes. Clarify completed-observation ownership, digest representations, and exclusive success/failure budget accounting. Preserve capacity fixtures and use byte comparisons for large buffers to avoid diagnostic allocation blowups. |
||
|
morluto
|
0354759370 |
fix(artifacts): retain admitted ZIP ownership through extraction
Transfer the inventoried ZIP descriptor after inventory reader cleanup, then borrow it for materialization instead of reopening the pathname. Release the exact owner before publishing output; preserve failed owners for cleanup retry and completed inventory facts through cancellation or later tagged failures. This uses the existing admitted-descriptor reader API without a third carrier hash pass. Remove the superseded materialization wrapper. Cover identical-member carrier replacement, directory/missing replacements, late cancellation at both handoff and final close, mixed cleanup outcomes and retry. Root-owner test seams preserve real output-handle cleanup. Compiled CLI/MCP ZIP, archive and report regressions also run; no benchmark is claimed. |
||
|
gensokyo-dev
|
1184819cb9 | fix(verifiers): load shared source without compiler processes (#7012) | ||
|
|
02e049744f |
feat(environment): build agent environments from a prompt (#6946)
* feat(environment): report what an environment serves with an environment-coverage view
A new inspect_analysis_view kind over capture Evidence. It projects the
state an environment derives with the given template rules and reports
each tool as answered from state, replayed with its view reason, a served
write, or a withheld write with its reason, the records behind it and a
hint about what to record next. The hints cover every write reason, so a
new reason without one fails to compile.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* feat(environment): build an environment bundle in one command
rea environment-build <capture> [directory] inspects a HAR (or reads
capture Evidence), projects the environment manifest from the in-memory
Evidence, authenticates the pair, prepares the state and writes the bundle
environment-export writes. Every bundle file is byte-identical to the
three-step path for the same capture and options.
The report adds the capture and manifest identities, the coverage summary,
warnings (a loopback capture named after its file, credential-named body
members by pointer) and the next command. --dry-run writes nothing. The
default template rules are the four the published environments use.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* test(environment): cover environment-build and the coverage view
Domain tests check the hint table against the state view's reasons, the
summary, and the view against the environment-state view it summarizes.
Service tests build the fixture capture and compare every file with the
three-step path, a dry run, loopback naming and the refusals. A CLI test
compares a HAR built in one step with the three CLI steps byte for byte,
and an MCP test projects the view from every source kind.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* test(environment): build the real-agentenv lane's bundle with environment-build
The lane now runs the command users run: it builds the Grafana-shaped HAR
with environment-build and the packed checkout, checks the report, then
runs the bundle with agent-env as before.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* docs(environment): document environment-build and the coverage view
The agent environments guide opens with the one-step build: the report,
the hint for each withheld write reason, and what a bundle ships. The CLI
guide, the capture guide and the skill reference name the command and the
environment-coverage view.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(environment): harden environment-build after review
- The credential scan walks with its own stack, so a deeply nested body
cannot crash the build, and it also reports credential-named header
names, query names, form body members and string arrays.
- The build checks SIGINT between stages: cancelled before the bundle
stage it writes nothing, during the write it names the written bundle.
- A dry run opens the package tarball, so it refuses what a build refuses.
- Repeated template rules are refused, a HAR saved as .json gets a hint,
file-derived names follow the bundle name rules, `localhost.` counts as
a loopback origin, and the next command is shell-quoted.
- --capture-timeout-ms passes through to the HAR inspection, and a
missing directory is refused before inspection.
- The no-observation hint covers updates and deletes of entities the state
never held; the coverage view states that it does not apply the
serve-time admission checks; the inspect_analysis_view description
names the view.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* docs(environment): state the build's conditions and record its verification
The guides now state when the one-step bundle equals the three-step one,
which headers inspection excludes, what the credential warning scans, how
cancellation behaves, and that the coverage view does not apply the
serve-time admission checks. The spec folder records the census, the end
to end runs, the gates and the review.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(environment): dispatch a pending interrupt before each build stage check
The manifest, state and coverage stages run synchronously, so a SIGINT
that arrived during them was only dispatched once the bundle write had
begun. Each check now yields first, so an interrupt before the bundle
stage cancels the build with nothing written.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* feat(browser): send declared requests from a scenario page with a request step
A capture_browser_scenario step `request` sends one declared HTTP request
through the page's own fetch, with its cookies and session, the way the
application's frontend would, and reads the whole response so the capture
records the transaction like any other. The path must resolve to the start
URL's origin, and the page must be on that origin when the step runs.
Query values and header values may be literals or declared secrets, and a
header may read its value from a cookie or a local storage entry, which is
then redacted from later records like a declared secret. `expected_status`
fails the step on any other status. Scenarios without a request step are
unchanged.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* test(browser): verify request steps in a real browser through to a built environment
verify:browser:request records a write session against a fixture item API
that refuses writes without its CSRF cookie header and stored bearer
token. It checks that every request lands in its own step, that the page
values authenticated each write and appear in no record, header or body,
that an unexpected status fails the step and cancels the rest, and that
environment-build serves the recorded create, update and delete.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* feat(environment): export environment bundles over MCP
export_environment_bundle writes the bundle environment-build writes, from
capture Evidence retained in the session or one capture Evidence file,
into an absolute output directory that must not exist and whose parent
must exist, as extract_firmware and export_web_scripts require; it never
creates the parent or reuses a directory. It returns the same report,
whose schema now lives in the domain so the CLI and the tool share it, and
dry_run reports without writing. A loopback capture without a file needs a
name. The build's refusal messages no longer name CLI flags.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(environment): scan scenario credentials, own the dry-run tarball and serialize Evidence once
A browser scenario's retained header values and the query parameters of its recorded URLs are scanned for credential
names as a HAR's are, through a review projection of its records. A dry run admits and releases the package tarball
through the export's regular-file scope, so a close it cannot confirm keeps its owner, and checks for cancellation
after that last await. Evidence is written into one exactly sized buffer instead of chunks joined by a copy.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* docs(environment): record the review round's checks
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* docs(skill): build agent environments from a prompt with request steps and the bundle tool
Add the environments reference to the skill and install it with setup: record
against a disposable instance, sign in through the UI, write with request
steps and read each write back, check coverage, re-record by withheld reason
and export the bundle. Route it from SKILL.md, name agent environments in the
skill description, and document the request step, the export tool and
recording writes.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* refactor(browser): split the request lane and header resolution into smaller functions
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* style(environment): format the build spec
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(environment): say when an untimed capture learns no write clock
Coverage of a capture whose records are not all timed, which includes every
browser scenario capture, now states that the state learns no clock, so an
update stamped with a server-set modification time stays unexplained-change
however often it is recorded.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* docs(skill): give agents a whole scenario shape with step timeouts
Agents guessed the scenario schema field by field and two hung on steps
without timeout_ms. The reference now starts from a complete scenario with a
timeout on every step, and says a scenario capture cannot serve updates the
application stamps with a modification time.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(browser): keep request steps on their origin and their page values redacted
- Follow no redirects: a redirect fails the step and nothing reaches the
location it names (checked in real Chrome), and check the page's origin
again in the page as the request is sent.
- Send nothing once the step's time has run out or it was cancelled while
header values were read; the in-page deadline is what remains of the step's.
- Keep every page value redacted under a repeated or colliding label, keep a
value's first marker when it is read again, and leave values shorter than
8 characters unredacted rather than rewriting ids, ports and hosts.
- Refuse a header name given twice, User-Agent (Chrome drops it) and line
breaks in literal values and prefixes; fail before sending when a secret or
page value holds them.
- Append query entries after the path's literal query without re-encoding it.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* docs(environment): document request step limits and word bundle refusals for both surfaces
Document redirects, timeouts, page value redaction, HttpOnly cookie sources
and header refusals for request steps. Refusals shared by environment-build
and export_environment_bundle no longer name CLI flags, and the retained
loopback refusal no longer prints an empty file name. The skill reference
asks for read-backs after each write, shows the coverage view's source, names
package_tarball for unreleased builds and the browser path a scenario ships.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(environment): scan every form occurrence and MIME-less scenario bodies
The build's credential scan collapsed form bodies with Object.fromEntries, so a
repeated field kept only its last occurrence: `api_key=secret&api_key=` warned
about nothing although the Evidence ships the value, and repeated values were
undercounted. Form fields are now scanned as ordered name/value pairs, as
headers and query strings already are.
The review projection of a capture_browser_scenario dropped a request body
recorded without a media type, so the scan never saw it. Review keeps such a
body's text with an empty media type; the manifest projection still reads only
bodies it can classify.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(browser): close request step edge cases from the second review
- Append query entries correctly after a path ending in "?", "&" or "#".
- Refuse 3xx expected statuses, which an unfollowed redirect can never
report, and say a redirected write may already have been applied.
- Redact a page value also as fetch sends it, without surrounding spaces
and tabs.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* docs(environment): document page value body costs, redirects and MCP dry runs
Document that reading a page value shrinks the bodies a capture keeps under
the default allocation limit, that redirected writes may have landed, that
the MCP dry_run still checks its output directory and which refusals name
the tool, and keep the .har hint in the HAR-as-JSON refusal for both
surfaces.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* docs(environment): record the prompt-to-environment verification
Gates, the real-browser lanes, five environments built from prompts by an
agent on disposable VMs and evaluated with two models, and the two reviews.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* test(skill): count rollback failures from the managed skill files
The environments reference made the skill eight files, so the recovery tests now derive the counts they expect from
MANAGED_SKILL_FILES instead of fixing them at six and seven.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(adb): restore the portable Java float pattern a merge rewrote
A lint autofix during a merge unescaped the hyphens in javaFloatValuePattern, which is invalid in a v-mode character
class, so the advertised start_adb_activity schema failed to compile in every JS regex mode. The file now matches main.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* docs(browser): show the request step inside a whole scenario
Every JSON example in the browser scenario guide is a capture_browser_scenario input the catalog test parses, so the
request step example now sits in a launch scenario instead of standing alone.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(environment): scan structured HAR form params and the request side of unattributed scenario transactions
A HAR request recorded with postData.params and no text is scanned by those parameters, each occurrence counting, and
the review projection keeps the request side of a scenario transaction whose response cannot be attributed, so the
credential warning covers what evidence.json ships in both cases.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(environment): validate build paths and read dry-run tarballs
* fix(environment): report bodies too large for the credential scan
Text bodies over the scan's 4 MiB limit, including a response decoded only as a prefix, are reported as a
credential-scan-incomplete warning by record and side, never by content, so an absent credential-named-members
warning no longer reads as a complete check. The spec and agent-environments guide also state the path validation and
dry-run tarball read that
|
||
|
Ariel Fikru Avicennaandgensokyo-dev
|
cab78de506 |
feat(ghidra): analyze GameCube/Wii DOL executables through Ghidra (#6240)
* feat(domain): parse and validate GameCube/Wii DOL headers Add a pure parser for the 0x100-byte DOL header (7 text and 11 data slots, BSS, entry point). Used slots must lie after the header and inside the file and the 0x80000000..0x81800000 load window; text slots are 4-byte aligned; sections may not overlap in memory or in the file; the entry point must be inside a text section. BSS may overlap data sections, as .sdata/.sdata2 do in shipped DOLs. Refs #1741 * feat(binary): admit caller-selected GameCube/Wii DOL targets * feat(ghidra): import GameCube/Wii DOL targets through a section-map pre-script * test(ghidra): cover DOL launcher arguments, attestation and raw profile digests * test(ghidra): add the real verify:ghidra:dol lane and DOL docs * fix(ghidra): read only the DOL header and section ranges instead of the whole file * test(ghidra): derive DOL lane addresses from readelf and admit Gekko from the install catalog * fix(ghidra): attest DOL block permissions, space, overlay and load state * fix(ghidra): keep load-image permissions optional for older inspections * fix(contracts): accept DOL targets in binary_session output * fix(ghidra): adapt the DOL lane to the current upstream address schema Validate DOL load-image coordinates with the upstream observed-address schema, drop a duplicate import left by the rebase, and re-pin the raw profile digests to the unmodified upstream resolver. * fix(ghidra): preserve bridge inventory symbol provenance * fix(ghidra): align DOL verification with shared workflow contracts * feat(domain): parse and validate GameCube/Wii DOL headers Add a pure parser for the 0x100-byte DOL header (7 text and 11 data slots, BSS, entry point). Used slots must lie after the header and inside the file and the 0x80000000..0x81800000 load window; text slots are 4-byte aligned; sections may not overlap in memory or in the file; the entry point must be inside a text section. BSS may overlap data sections, as .sdata/.sdata2 do in shipped DOLs. Refs #1741 * feat(binary): admit caller-selected GameCube/Wii DOL targets * feat(ghidra): import GameCube/Wii DOL targets through a section-map pre-script * test(ghidra): cover DOL launcher arguments, attestation and raw profile digests * test(ghidra): add the real verify:ghidra:dol lane and DOL docs * fix(ghidra): read only the DOL header and section ranges instead of the whole file * test(ghidra): derive DOL lane addresses from readelf and admit Gekko from the install catalog * fix(ghidra): attest DOL block permissions, space, overlay and load state * fix(ghidra): keep load-image permissions optional for older inspections * fix(contracts): accept DOL targets in binary_session output * fix(ghidra): adapt the DOL lane to the current upstream address schema Validate DOL load-image coordinates with the upstream observed-address schema, drop a duplicate import left by the rebase, and re-pin the raw profile digests to the unmodified upstream resolver. * docs(ghidra): qualify the DOL memory window * refactor(ghidra): own DOL BSS ranges in the parsed header * docs(ghidra): record real DOL workflows and task costs * docs(ghidra): record final DOL integration verification * fix(mcp): share the native procedure search row contract * docs(ghidra): record public procedure search verification --------- Co-authored-by: gensokyo-dev <[email protected]> |
||
|
morluto
|
43c7255dbd | fix(android): preserve binary class selectors across metadata calls | ||
|
morluto
|
fbe77dc479 |
refactor(verification): separate readiness evidence scopes
Move the synthetic evaluator and its unchanged semantic tests out of the shipped runtime. Retain actual CLI and MCP Evidence separately from the synthetic nine-stage baseline in a versioned verifier artifact, execute its reported conformance checks, and assign the verifier to its CI lane. Validation: evaluator/state/classifier source suites, compiled CLI/MCP views, verify:readiness, static checks, build and docs passed. Readiness observed coverage is static analysis; fixture runtime behavior and reconstruction closure were not executed. |
||
|
morluto
|
41e10b240c | fix(filesystem): distinguish admission and bound-target failures | ||
|
gensokyo-dev
|
d662fa54a6 | fix(test): reserve heap headroom for shared CLI workers (#6941) | ||
|
morluto
|
2bd981b017 | refactor(tests): consolidate compiled CLI session ownership | ||
|
morluto
|
eea3271a33 | test(cli): load only Frida contracts for focused CLI checks | ||
|
morluto
|
eac392d085 |
revert(test): remove unproven SDK compiler sharing
This reverts commit 56eca648d. The equivalent completed two-client measurement increased elapsed and CPU time, so compiler sharing has not established a benefit. |
||
|
morluto
|
5909430771 | perf(test): share SDK catalog compilation across deferred clients | ||
|
morluto
|
cb689fae96 | test(package): join owned updater after registry fixture failure | ||
|
morluto
|
d0375a1ed0 | test(filesystem): probe actual permission denial before host assertions | ||
|
morluto
|
0c4bf68ce8 | merge: reconcile concurrent main fixes with issue remediation | ||
|
morluto
|
0d849b1ae8 | docs: separate retained notices from provider version support | ||
|
morluto
|
91f06f55a3 | ci(docs): render documentation in the selected docs lane | ||
|
kiranandgensokyo-dev
|
5d2c8c2b78 |
feat(cli)!: export row-oriented results as JSON Lines (#5629)
* feat(cli): add JSON Lines output for row-oriented results (#5487) - Support opt-in JSON Lines format (--format jsonl) for row-oriented results and collection views with clear record boundaries - Preserve stable source identities and stream records deterministically line-by-line - Emit final coverage and limitation metadata as a distinct line with https://rea.dev/schemas/jsonl-coverage-v1.json to prevent confusion with data rows - Distinguish error outputs with https://rea.dev/schemas/jsonl-error-v1.json - Reject non-row-oriented results that lack record boundaries with an unreadable_output error - Document JSON Lines streaming conventions in docs/cli.md - Add boundary tests covering empty results, multiline strings, partial coverage, large fields, and non-row rejection * fix(cli): preserve diagnostics, provenance, and nested collections in JSONL * fix(cli): frame classified failures and reject unrepresented JSONL facets --------- Co-authored-by: gensokyo-dev <[email protected]> |
||
|
morluto
|
1de720e02e | docs: stop and profile unexpectedly costly test runs |