Files
rea/docs/esp-image.md
Entrpiandgensokyo-dev 64196b08ca feat(artifacts): inventory and extract ESP32 firmware images (#7498)
* feat(artifacts): inventory and extract ESP32 firmware images

ESP32-family app and bootloader images (magic 0xE9) had no reader, so
open_binary refused OTA downloads, app partitions and bootloaders.

Add the esp-image container format behind inspect_artifact and
extract_artifact. A pure decoder reads each byte once and keeps only
fixed-size fields. Derived facts cover the chip, from a table cited to
ESP-IDF v6.1; segment classification as ESP-IDF's should_map and
should_load; checksum and appended SHA-256 results; app and bootloader
descriptors; Secure Boot v2 signature blocks, observed and unverified;
and undescribed trailing byte ranges. Truncation and undescribed bytes
make coverage unknown. ESP8266 images are refused with a specific
reason, and ciphertext stays unrecognized.

Add the format to every target, Evidence, snapshot, live-summary and
artifact vocabulary, document it, and add an optional verify:esp-images
lane that compares esptool-built images with esptool image-info and
espsecure signature-info-v2.

Refs #3615

* test(artifacts): run the ESP image lane on esptool v5.6.0

The lane compared descriptor fields only when esptool image-info printed
one, so a descriptor that REA reports and esptool does not print passed
without comparison. Compare the descriptor kind in both directions.

Record that the lane passed for all 13 images with esptool and espsecure
v5.6.0, the version the layout facts cite, as well as with 4.12.0.

* refactor(artifacts): define ESP image layout sizes once

The decoder and the derived facts each defined the segment-header,
appended-digest and flash-sector sizes, the erased-byte scan and the
offset formatter. Define them once with the other fixed-layout structures
and import them in both places. The signature slot count is the one
constant that drives the slot scan.

Correct the observedEnd comment: it is the end of the last chunk read.

* fix(artifacts): report ESP-IDF's segment length limit by chip

ESP-IDF's verify_segment_header rejects a segment of 16 MiB or more
unless the chip maps flash through 32-bit cache addresses
(SOC_SPI_MEM_SUPPORT_CACHE_32BIT_ADDR_MAP: ESP32-S3, P4, C5, C61, H4 and
S31 in v6.1). Record that capability in the chip table and add a
limitation for such segments; unknown chips get none.

`loaded` repeats should_load. It does not mean the bootloader accepts the
segment: verify_load_addresses still rejects one outside the chip's RAM,
which REA does not model. Say so in the docs, the skill reference and the
inventory limitation.

* test(artifacts): check ESP lane tools first and verify extraction

The lane started work before checking its tools, so a missing RISC-V
target, ld.lld, esptool or espsecure surfaced as a raw spawn or command
error. Check each one before building and name the variable to set. Warn
when esptool is not v5.6.0, the release the layout facts cite.

Assert each segment's class for the ESP32-C3 app, its variants and the
bootloader, as the Xtensa image already did. Extract the app and the
signed image through the CLI and compare every file with the bytes at
esptool's segment offsets or in the appended signature sector; the
source's SHA-256 must be unchanged and equal the reported root_sha256.

* docs(artifacts): state what the ESP image lane does not establish

The Verification section listed the lane's checks but not its gaps: only
ESP32-C3 and ESP32 images come from a real producer, no real project
build, OTA download or flash dump is inspected, no signature is verified,
and the lane does not run in CI.

Record the timing of the final lane run on esptool v5.6.0.

* fix(artifacts): compare ESP segment declarations

Segment entries hash only their data, so compare_artifacts reported a
segment that moved to another load address with the same bytes as
unchanged; only the image itself changed. Compare each segment's
load_addr and data_len as its declaration, as Node SEA and PyInstaller
members are compared, and count a changed declaration as a metadata
change, also when content is unavailable.

Header offsets are layout and the classification is derived, so neither
is compared.

* test(artifacts): run ESP lane commands as owned process trees

The lane ran its tools through execFile. A timeout killed only the direct
child, so a linker or assembler that Clang or GCC had started could keep
running while the lane removed its workspace. Run every command through
createVerifierCommands, check the tools inside withVerifierCleanup, and
remove the workspace only after commands.close() has stopped every tree;
a failed close keeps the workspace.

Only a nonzero exit now counts as a refusal or an expected esptool
failure; timeouts and spawn failures propagate. Supervising each command
roughly doubles the lane's run time.

* test(artifacts): cancel ESP decoding after its first decoded chunk

The cancellation test requested cancel or close on setImmediate, which
can land while the file is still being opened and checked, so it never
showed that decoding had started. It also passed against a reader that
ignored both requests while decoding. Acknowledge the decoder's first
chunk, then cancel or close, join the operation, and assert that no
later chunk reaches the decoder and that the reader ends closed.

* docs(artifacts): record which ESP metadata comparison ignores

Segments are compared by their declaration. The image's and the
signature blocks' format_metadata are not: every field comes from their
own bytes, which content comparison already covers, and a block's
image_digest_matches follows the image's content.

* docs(artifacts): state that ESP mmu_page_size is a base-2 logarithm

The app descriptor's mmu_page_size keeps ESP-IDF's name and value, the
MMU page size in log base 2 as esp_app_desc_t declares it, so 16 means
64 KiB. Say so in docs/esp-image.md and the schema comment; a reader
could otherwise take 16 as 16 KiB.

* fix(esp-image): retain member consumption and share lifecycle regressions

* test(artifacts): scope MCP catalogs and budget full-schema validation

* test(artifacts): verify compact output profiles and document bounded catalogs

* test(artifacts): close readers when cleanup-test admission fails

* test(artifacts): compare ESP image limitations with their owners

The decoder and reader tests froze complete limitation sentences, so a
harmless rewording failed them. The wording now has two owners:
ESP_IMAGE_LIMITATIONS in espImageFacts.ts, which the decoder, the inventory
and the reader use, and ESP_DESCRIPTOR_LIMITATIONS beside the descriptor
decoder. Tests compare delivered limitations with the owner, called with the
facts each case establishes (segment index, declared and available bytes,
offsets, chip, page range), next to the structured assertions they already
make. Delivered text is unchanged.

* test(esp-image): assert structured facts and owned guidance

* fix(esp-image): keep spi_flash_mode and descriptor fields before a truncated tail

ESP-IDF v6.1's esp_app_desc_t declares spi_flash_mode at byte 181, after
mmu_page_size, as the committed reference layout shows; the descriptor now
keeps it as stored. The decoder also required all 256 descriptor bytes,
though it reads none past byte 181, so a file ending in the reserved tail
lost the version, project, digest and revision facts it does hold. The app
descriptor now needs only its fields; the segment's truncation is reported
as before.

* test(esp-image): pin the P-192 public key digest to espsecure's layout

espsecure packs ECDSA signature blocks as "<BBBx32sB64s64s1031x" and
zero-pads a 48-byte P-192 point to 64 bytes, so the signature starts at
byte 101 for every SHA-256 curve. signature_info_v2 hashes sig_block[36:101]
and _digest_sbv2_public_key the curve ID with the padded point. No test built
a P-192 block; this one checks public_key_sha256 against that definition.

* fix(esp-image): keep each app descriptor field the file holds in full

A file can end anywhere inside the 256-byte esp_app_desc_t. The decoder
needed every field byte, so a cut at byte 176 dropped the version, project
and app_elf_sha256 it does hold. Once the 4-byte magic identifies an app
descriptor, each field is now kept when the file holds all of its bytes and
is null otherwise; observed_length records where the descriptor ends, so a
cut field is told apart from text that is not UTF-8. The fields after the
magic become nullable in the metadata schema.

* fix(esp-image): report a signature block whose image digest does not match

A CRC-valid Secure Boot v2 block can carry another image's digest. The
block recorded image_digest_matches false, but no limitation said so, unlike
a checksum or appended-SHA mismatch, so CLI and MCP summaries showed only
the generic unverified note. The mismatch is now a limitation; coverage
stays complete and signature_status stays unverified. The three other
signature-sector limitations move into ESP_IMAGE_LIMITATIONS with their
text unchanged, so all ESP image wording has one owner.

* refactor(esp-image): remove redundant descriptor byte guards

---------

Co-authored-by: gensokyo-dev <[email protected]>
2026-10-28 23:18:22 +08:00

19 KiB

ESP images

REA reads ESP32-family firmware images through inspect-artifact / inspect_artifact and extract-artifact / extract_artifact. OTA downloads, app partitions and second-stage bootloaders are all ESP images. Reading is static: no esptool, device or flash access is needed, and no code is run or disassembled.

rea inspect-artifact ./firmware.bin --json
rea inspect-artifact ./ota.bin --detail summary --evidence_output evidence.json --json
rea extract-artifact ./firmware.bin --json

For MCP, open the file with open_binary, then call inspect_artifact. The summary's root_format_metadata holds the image facts. Use its parent_evidence_id with inspect_analysis_view to select segment rows. extract_artifact writes one file per segment and per signature block.

For agent sessions, advertise the session,artifact,application tool families and consider setting both REA_MCP_INPUT_SCHEMA_PROFILE and REA_MCP_OUTPUT_SCHEMA_PROFILE to compact. The output profile bounds each advertised output schema to 4,096 bytes by relaxing nested client validation; REA still validates complete results against the canonical contracts. Pair this with retained summaries and selected inventory views to limit the context needed for one question. See MCP contracts for the profiles and their client compatibility tradeoffs.

The layout facts follow ESP-IDF v6.1 (esp_app_format.h, esp_app_desc.h, esp_bootloader_desc.h, esp_image_format.c, the ROM secure_boot.h headers and docs/en/security/secure-boot-v2.rst) and esptool v5.6.0 (bin_image.py, cmds.py, loader.py, espsecure).

Recognition

The container format is esp-image. A file is an ESP image when its bytes show all of these; the file name does not matter:

  • magic 0xE9 at offset 0 and 1 to 16 segments. A count of 0 is valid but loads nothing, and accepting it would admit x86 programs that begin E9 00;
  • a flash mode and flash speed code that ESP-IDF's enums define;
  • a hash_appended byte of 0 or 1, as esptool requires;
  • when the file holds the first segment header, a segment length that is a multiple of 4.

Later segments, the checksum and the digest are integrity facts. A truncated download is still an ESP image, with its integrity reported as unavailable.

Input Outcome
ESP8266 image (v1 or v3): same magic, 8-byte header Refused: "ESP8266 firmware images are not supported…"
Flash-encrypted image (ciphertext) Not recognized: the bytes carry no magic. Unknown, not malformed
ESP8266 v2 image (magic 0xEA) Not recognized

An ESP8266 image has its first segment header at offset 8, and that segment loads into ESP8266 DRAM, IRAM (Arduino's eboot links at 0x4010F000) or the IROM flash window, where v3 and Arduino app images begin. In an ESP32-family header the same bytes hold wp_pin and the drive settings, which esptool and ESP-IDF write as 0xEE and zeros. Inside another container, an ESP8266 image is classified as a plain file.

Recognition cannot exclude every file that starts with an x86 near jump (0xE9), such as a DOS .com program. An explicit --target-format dos-com or raw-binary selection skips ESP recognition.

Image facts

Root format_metadata has kind: "esp-image":

Field Label Meaning
header observed Every esp_image_header_t field, with ESP-IDF's names, for example entry_addr, chip_id, min_chip_rev_full
chip derived target, isa and the table's source (ESP-IDF v6.1) from chip_id; null for an ID the table lacks
checksum derived offset, stored, computed and status: valid, invalid or unavailable
appended_sha256 derived The digest hash_appended adds: offset, stored, computed and status, including not-appended
image_end derived End of the checksum block and any appended digest; null when the file ends before it
signature_sector_offset derived The Secure Boot v2 signature sector, when it holds at least one valid block
undescribed_trailing derived Byte ranges after the image that nothing describes; empty when every byte is described
descriptor observed esp_app_desc_t or esp_bootloader_desc_t fields, with ESP-IDF's names, and the descriptor's file offset

The descriptor's kind, derived from its magic, classifies the image as an app or a bootloader. Without a descriptor the role is unknown.

The ISA comes from chip_id, never from the series letter: ESP32-S31 is RISC-V. The chip table covers every esp_chip_id_t in ESP-IDF v6.1:

chip_id Target ISA chip_id Target ISA
0x0000 esp32 xtensa 0x0012 esp32p4 riscv
0x0002 esp32s2 xtensa 0x0014 esp32c61 riscv
0x0005 esp32c3 riscv 0x0017 esp32c5 riscv
0x0009 esp32s3 xtensa 0x0019 esp32h21 riscv
0x000C esp32c2 riscv 0x001C esp32h4 riscv
0x000D esp32c6 riscv 0x0020 esp32s31 riscv
0x0010 esp32h2 riscv

Other IDs, for example esptool v5.6.0's ESP32-E22 (0x1F), have chip: null and unclassified segments.

Integrity. The checksum is the XOR of every segment data byte, seeded with 0xEF, in the last byte of the next 16-byte block. The appended SHA-256 covers every byte through the checksum. Both are facts, not admission rules: a mismatch is reported with a limitation and the segments stay readable.

Descriptors. Both descriptors begin segment 0: ESP-IDF's bootloader reads the app descriptor there, and esptool places the bootloader descriptor there. Strings stop at the first NUL and are omitted, with a limitation, when they are not valid UTF-8. The bootloader descriptor's magic is a single byte (0x50), so its reserved bytes, which ESP-IDF zero-fills, must also be zero. A --ram-only-header image puts RAM segments first, so its app descriptor is not in segment 0 and its role is unknown. mmu_page_size keeps ESP-IDF's name and value: the MMU page size as a base-2 logarithm, as esp_app_desc_t declares it, so 16 means 64 KiB. It is 0 in images built before ESP-IDF v5.4. spi_flash_mode is the next byte, which ESP-IDF v6.1 names, kept as stored and not interpreted; releases that reserve it leave it 0. A file can end inside the app descriptor: every field it holds in full is kept, a field it cuts off is null, and observed_length gives the descriptor bytes the file holds. The segment's truncation is reported as usual.

Segments

Each header-declared segment is one entry named segment-<index>.bin, for example segment-00.bin, so two builds pair segments by position. Its source_location is the segment data in the image file. format_metadata (esp-image-segment) records index, load_addr, data_len, header_offset and classification.

classification follows ESP-IDF's bootloader (should_map and should_load in esp_image_format.c) on a cold boot:

Value Meaning
mapped Inside the chip's IROM, DROM or, where the MMU is per memory type, PSRAM range (soc.h)
loaded should_load selects it: not mapped, at or above 0x10000000. RAM ranges are not checked
not-loaded Below 0x10000000: the padding segments esptool writes at address 0, including --secure-pad-v2
unknown The chip is unknown, or the mapping depends on the bootloader's MMU page size

loaded does not mean the bootloader accepts the segment. Before loading, ESP-IDF's verify_load_addresses rejects the image when a loaded segment is not entirely in the chip's DRAM, IRAM, RTC memory or SPM, or overlaps the bootloader's own stack, data or loader code. For example, an ESP32-C6 segment at 0x43000000 is loaded here and rejected there. REA models no RAM ranges.

Padding segments are inventoried and extractable, but are never load regions. When waking from deep sleep, ESP-IDF also skips RTC-memory segments; that runtime case is not modelled. ESP32-C6 and ESP32-H2 map flash from 0x42000000 through 256 pages of the bootloader's configured MMU page size, which no image records. The app descriptor's page size describes the app's build, not the bootloader's. With page sizes from 8 KiB (which the hardware supports and esptool builds for) to 64 KiB, segments from 0x42200000 to 0x43000000 are unknown. Ranges follow ESP-IDF's should_map, not esptool's wider placement ranges.

A segment cut short by the end of the file is the bytes the file holds, with an entry limitation that gives the declared length.

Artifact comparison uses each segment's load_addr and data_len as its declaration, so a segment that moves with unchanged bytes is changed in metadata. Segments pair by index: inserting or removing one shifts the paths of the segments after it. Header offsets and classification are not compared; the image's own change covers the header and chip. The image's and the signature blocks' format_metadata carry no declaration: their fields come from their own bytes, which content comparison covers, and a block's image_digest_matches follows the image's content.

Secure Boot v2 signatures

A signature sector starts at the next 4 KiB boundary after the image and holds up to three 1216-byte slots. A slot holds a block when it has the 0xE7 magic and a matching CRC32 at offset 1196, the rule ESP-IDF and espsecure apply. Each slot is examined independently, so an erased or damaged slot does not hide a later block. Each block is an entry, signature-block-<slot>.bin, with esp-image-signature-block metadata:

Field Meaning
index, version, scheme Slot 0 to 2, version byte, and rsa (0x02) or ecdsa (0x03)
sha_version, curve_id For ECDSA, the third byte (0 is SHA-256, 1 SHA-384) and the curve ID byte
digest_algorithm, image_digest The digest the block records
image_digest_matches Whether it equals the digest of every byte before the sector
public_key_sha256 SHA-256 of the block's public key, the value eFuse key digests hold
signature_status Always unverified

Signatures are observed, not verified: REA checks no public key, signature or eFuse. public_key_sha256 is the digest espsecure signature-info-v2 prints, so it can be compared with espefuse summary. An unknown version or sha_version leaves the layout-dependent fields null. A slot with the magic but no matching CRC32 is not a block: it is reported in a limitation, whether or not another slot holds a block, and its bytes are undescribed. Secure Boot v1 signatures are not decoded.

Coverage and limitations

Bytes after the image are described when they are erased flash (0xFF) or a signature sector's blocks; erased slots and slack in that sector count as erased flash. Every other byte is in undescribed_trailing.

Situation Outcome
Segment data or a segment header beyond the end of the file Truncation limitation; checksum and SHA-256 unavailable; coverage unknown
File ends before the checksum or the appended digest That status is unavailable, with a limitation; coverage unknown, since a signature sector could follow
Checksum or SHA-256 mismatch invalid, with a limitation; coverage is unchanged
Erased flash after the image that runs past the three slots Described; no limitation
Erased flash that ends before the three slots are complete Limitation; coverage unknown, since a signed image cut there would have lost its sector
A signature sector the file cuts short Limitation; coverage unknown
Any other bytes after the image Ranges in undescribed_trailing, a limitation, and coverage unknown
Segment length that is not a multiple of 4 Limitation: ESP-IDF rejects such an image
Segment of 16 MiB or more, except on chips with 32-bit cache addresses Limitation: ESP-IDF rejects such an image

ESP32-S3, ESP32-P4, ESP32-C5, ESP32-C61, ESP32-H4 and ESP32-S31 map flash through 32-bit cache addresses (SOC_SPI_MEM_SUPPORT_CACHE_32BIT_ADDR_MAP), so ESP-IDF sets no 16 MiB segment limit for them. For an unknown chip the limit is not applied.

Undescribed bytes include other flash contents such as an app partition's stale data, a Secure Boot v1 signature, and the flash segments that esptool --ram-only-header writes after the checksum without counting them. REA reports their ranges and does not guess their structure. In the padding, in a sector without a valid block, in the sector's slack and after the sector, the first byte that is not erased starts a range to the end of that region. A slot that is neither a block nor erased is reported whole. Adjacent ranges are merged.

A whole-flash dump of a chip whose bootloader is at offset 0, such as ESP32-C3 or ESP32-S3, is recognized as that bootloader image; the partition table, apps and data follow as undescribed bytes.

Limits

An image has at most 16 segments and 3 signature blocks. Decoding reads each byte once and retains only fixed-size fields. It reads through erased flash to find the first byte that is not erased, and stops there. There is no file-size limit.

Analyzing a segment

Inventory does not disassemble code or build a memory map. open_binary admits neither RISC-V nor Xtensa as a raw-binary architecture, so a segment cannot yet be opened as a raw image. Each segment's source_location and load_addr give the file offset and address to map it at by hand, for example with the Rizin tool's om command.

Verification

Synthetic fixtures with computed checksums, digests and CRCs cover recognition, ESP8266 layouts, the ESP32, ESP32-C3, ESP32-C6 and ESP32-P4 memory maps, truncation at each structure, integrity mismatches, erased and undescribed trailing bytes, signature slots including damaged and unknown layouts, and the refusals. CLI and stdio MCP tests exercise the same artifact provider and advertised schemas.

The optional real lane builds ELF files from source, converts them with esptool and compares REA's facts with esptool image-info:

REA_RISCV_CLANG=/path/to/clang REA_RISCV_LLD=/path/to/ld.lld \
REA_ESPTOOL_PYTHON=/path/to/python-with-esptool \
  npm run verify:esp-images

It needs a Clang with the RISC-V target, ld.lld and a Python that can run esptool and espsecure. It checks each tool before building anything and names the variable to set when one is missing, and it warns when esptool is not v5.6.0, the version whose sources the layout facts cite. It builds an ESP32-C3 app, the same app with --dont-append-digest, --secure-pad-v2, --ram-only-header and --pad-to-size, a bootloader, and ESP8266 v1 and v3 images. It then checks:

  • entry point, chip ID, each segment's length, address and header offset, checksum, digest and descriptor fields against image-info;
  • each segment's classification in the app, its variants and the bootloader, whose linker scripts fix every section's region, including the not-loaded padding that --secure-pad-v2 adds;
  • after signing with RSA-3072 and appending ECDSA P-256 and P-384 blocks, each block's public_key_sha256 against espsecure signature-info-v2, and that its recorded digest matches;
  • extraction of the app and the signed image: each segment file equals the bytes after esptool's segment header offset, each signature block equals its slot in the sector espsecure appended, and the source's SHA-256 is unchanged and equals the reported root_sha256;
  • the --pad-to-size erased tail is described; the --ram-only-header flash segments are undescribed;
  • a flipped checksum, a flipped digest byte, a truncated segment (where image-info stops with "End of file reading segment"), and both ESP8266 refusals.

REA_XTENSA_ESP32_GCC adds an ESP32 (Xtensa) app built with Espressif's xtensa-esp32-elf-gcc. Without it the report marks that image as skipped.

Each command runs as a process tree owned by the shared verifier command owner, so a timeout also stops the linker or assembler a compiler driver starts. The lane removes its workspace only after every tree has stopped and keeps it when that cleanup cannot be confirmed.

The lane ran on macOS arm64 with Node 24.22.1, Homebrew Clang and LLD 21.1.8, and xtensa-esp32-elf-gcc 14.2.0 (crosstool-NG esp-14.2.0_20251107), once with esptool and espsecure v5.6.0 and once with 4.12.0. All checks passed for all 13 images in both runs. With v5.6.0 the complete run, including the builds and every esptool call, took 21.7 s elapsed, 16.9 s user CPU and 6.1 s system CPU under /usr/bin/time -l with warm caches. The largest single process peaked at a 253 MiB resident set; the combined peak of all processes was not measured.

The lane does not establish:

  • the other chips: only ESP32-C3 (RISC-V) and ESP32 (Xtensa) images come from a real producer; the other rows of the chip table are checked against the ESP-IDF v6.1 headers by reading them;
  • images from a real ESP-IDF project build, an OTA download or a device's flash, none of which it inspects;
  • that a signature is valid: it compares digests, and nothing verifies a signature.

It does not run in CI.