Repository navigation
Conversation
Int8Interpreter::invoke() read esp_timer_get_time() before and after every operator to fill a per-operator profile, whether or not anything printed it. On ESP32-S3 that is two function calls per operator; for a 32-operator streaming keyword-spotting model it was about 55 us of a 540 us call, more than many of the operators themselves. The interpreter now times operators only with Int8RuntimeOptions::profile_operations, which EspNNBackend sets from CONFIG_ESPNN_DELEGATE_PROFILE, and keeps the times in a table allocated only then. Otherwise it reads the timer at the start and end of each call and every 16 operators, for its cooperative yield. last_invoke_us() becomes the wall time of the call.
The ESP-NN delegate only built inside ESP-IDF, so checking a runtime change needed the ESP32-S3 emulator or a board. Its sources use few ESP-IDF APIs: the heap_caps allocator, esp_timer, a FreeRTOS mutex and a task yield. Outside ESP-IDF, espnn_runtime/CMakeLists.txt now builds the same sources as the espnn_backend library, with host/include standing in for those APIs and ESP-NN's portable C kernels fetched from GitHub at the v1.5.0 commit (or taken from ESPNN_LOCAL_PATH). backends/espressif/test/host builds it with the ExecuTorch runtime into espnn_host_runner, and test_host_runtime.py runs every fully delegated test model through it, over several steps and several loaded instances of a method, with planned and with caller-owned inputs and outputs, checking each output bit-exactly against ProgramEmulator. .ci/scripts/test_espressif_host.sh runs the test on a plain build and on an AddressSanitizer and UndefinedBehaviorSanitizer build, in a new job of the Espressif workflow that needs no ESP-IDF toolchain. The host build exercises the delegate's planner, interpreter and ExecuTorch glue, not ESP-NN's S3 assembly kernels, which the emulator jobs still cover.
On a 512 KB-SRAM part the delegate's own bookkeeping can cost more internal RAM than a small model's activations. For a 32-operator streaming keyword-spotting model on ESP32-S3 it kept 10.5 KB of per-operator state, 1.4 KB of operation records that are only needed while loading, and a separate 5.8 KB ESP-NN scratch buffer, and graph validation put four 1,024-byte bool arrays on the loading task's stack. Fields of the per-operator state that no operator uses together now share storage, operation records are decoded while loading instead of being kept, the planner frees its tables before allocating the arena, and graph validation uses bitsets. The invoke mutex is created by a static initializer instead of a function-local static, so invoke() no longer goes through a thread-safe static guard. ESPNN_DELEGATE_SCRATCH_IN_ARENA (off by default) plans each operator's ESP-NN scratch as an arena entry that lives only while the operator runs; for a model whose arena stays in internal RAM this replaces the separate buffer. DelegateStats now also reports the scratch, requantization and table bytes. The host test checks the delegate's arena against plan_activation_arena(), its Python mirror, and the sanitized host build runs with scratch in the arena and per-operator profiling on.
The delegate copied every tensor record and every per-channel requantization multiplier and shift out of its graph into RAM, although the graph already sits in memory-mapped flash or in PSRAM. For a 32-operator streaming keyword-spotting model on ESP32-S3 the two copies took 4.4 KB of internal RAM. They were needed because XTS3 v6 places those tables at offsets that are not word-aligned: the 22-byte header and the input/output table leave the tensor records, and so everything after them, 2 bytes off a word boundary. XTS3 v7 is v6 with the input/output table padded to a 4-byte boundary and each operation's parameter block starting on one. The serializer now writes v7. On a little-endian target the runtime uses a v7 graph's tensor records and requantization arrays (and softmax tables) where they are, and still copies them for v5 and v6 graphs, which it keeps accepting. When the delegate copies the graph's constants into PSRAM, the in-place pointers follow the copy. A program exported with this version needs a runtime from the same version or later; the runtime rejects unknown graph versions at init with DelegateInvalidCompatibility. espnn_host_interpreter runs a graph on the host build of the interpreter from addresses of every alignment, and test_host_interpreter.py checks that v6 and v7 graphs give bit-exact results from all of them and that v7 graphs keep no copy of their requantization parameters.
The delegate copied every input into its activation arena before running and every output out of it afterwards, transposing rank-4 tensors between NCHW and its NHWC device layout element by element. For a streaming keyword-spotting model on ESP32-S3 whose state tensors cross the delegate boundary on every step, that was both the largest share of the delegate's non-kernel time and the reason the arena held a copy of every input and output. With Int8RuntimeOptions::external_io, which EspNNBackend now sets, the planner leaves graph inputs and outputs out of the arena and the caller binds a buffer to each before invoke(). The backend binds the ExecuTorch tensor itself when its memory is 16-byte aligned, like the arena, and already in the device layout: an int8 tensor that is not rank 4, has one channel or one pixel, or has a channels-last dim order. Any other input or output (float, NCHW or misaligned) goes through a zero-initialized staging buffer that the first execute() needing one allocates, which keeps existing programs working unchanged. Because operators may read an input after writing an output, an output whose memory overlaps an input is always staged and copied out after the last operator; this makes binding one buffer as both an input and an output (an application updating a state tensor in place) safe instead of depending on the order of the delegate's copies. A graph tensor that the delegate returns as several outputs (when its value feeds several CPU operators) is written once and copied to the other outputs, and a graph input that is also returned as an output is staged and copied from there. The first COPY writing an output bound to caller memory, unless it fills it, clears it first, so the buffer's previous contents never show through; a long cat is written by several COPY operations. Pooling and convolution outputs wider or taller than 65535, which ESP-NN's kernels would truncate to uint16_t and so leave unwritten, are now rejected at load. A graph whose every activation is an input or output needs no arena at all. The interpreter keeps a pointer per tensor instead of an arena offset, so binding costs nothing per operator. plan_activation_arena() mirrors the external-I/O plan, the README describes how to export int8 channels-last I/O that is used in place, and the tests cover channels-last I/O without staging, misaligned caller buffers, an output bound to the memory of an input that operators read after the output is written, an input returned as an output (also with another output bound over it), a partial COPY into a bound output, a seven-way cat as an output, those limits, a graph with no arena, and, with the portable and quantized kernels now linked into the host runner, the programs that leave operators on the CPU between delegates.
A streaming model keeps a few frames of history in mutable buffers. Lowered to the ESP-NN delegate today, each buffer stays in the program: it is passed into the delegate and its new value out of it on every call, in float, or in int8 when the application exports it as a stateless input and output and carries the state itself. For a 32-filter streaming keyword-spotting model on ESP32-S3 with six state tensors, those copies, their layout conversions and the extra delegate arguments took about a fifth of each call. EspNNPartitioner(delegate_mutable_buffers=True), also exposed by to_espnn_edge(), gives a mutable buffer to the delegate that holds every access to it, through tag_mutated_buffer(), when the buffer is read and written with the same int8 quantization: a float32 buffer read through quantize ops (what PT2E produces for a cat and slice shift register) or an int8 buffer read through dequantize ops. Other buffers stay in the program, as they do by default. The partition's graph is then XTS3 version 8, which adds a table of (state, update) tensor pairs; each state's quantized initial value is stored with the constants. An update that the runtime cannot write into a state's slot (a graph input or output, another state, or an update shared by several states) gets a copy of its own, made after the last operator. The runtime keeps each state in a reserved arena slot and initializes it at load. The update is planned into the state's own slot when no operator reads the state at or after the update's first write, and copied over the state after the last operator otherwise. Every loaded method instance has its own state. reset_delegate_states(), or the reset_states backend option through executorch::runtime::set_option(), restores the initial state of every EspNNBackend delegate before its next execute(); reloading a method resets it alone. ProgramEmulator and the reference interpreter keep the same state across calls and can reset it, and plan_activation_arena() mirrors the in-place update. New tests cover which buffers the delegate takes, the reference against the quantized PyTorch model across steps and a reset, and, on the host build, states updated in place and by copy, a state updated from a graph input, a reset, two instances, and the parser's checks of the state table. The emulator suite gains a stateful program.
The pieces a streaming keyword-spotting model on ESP32-S3 needs (int8, channels-last or trivially laid out inputs and outputs left to the application's buffers, mutable buffers owned by the delegate, and a quantization that matches the model's reference) were each documented on their own. export_streaming_model.py puts them together on a small model whose program has one int8 input, one int8 output, no CPU operators and no planned memory, and the README adds the quantization choices that matter for such models and the device-side pattern: bind once, execute per frame, reset the delegate's state on a detection. The host test runs the example's program for several frames without any staging copy.
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25106
Note: Links to docs will display an error until the docs builds have been completed. ❌ 1 New FailureAs of commit a16cf92 with merge base 436abf7 ( NEW FAILURE - The following job has failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
This PR needs a
|
|
@claude review effort=max |
|
Claude finished @JakeStevens's task in 52m 19s —— View job PR Review: #25106 (effort=max)
Request Changes: external I/O is now always on, so every default float/NCHW program stages its I/O in a buffer allocated at the first Detailed reviewPaths are under SummaryThis PR adds a host build of the ESP-NN delegate with ASan/UBSan CI. It also shrinks the delegate's tables, scratch and load-time stack, uses XTS3 v7 tables in place, binds int8 graph I/O directly to ExecuTorch tensors, and lets the delegate own quantized mutable buffers with a process-wide reset. The blocking problems:
Two test gaps: the PSRAM constant-copy path never runs, and the default scratch configuration never runs under the sanitizers. Code Quality
Infrastructure
Testing
API Design
Thread Safety
Backward Compatibility
Performance
Specific Comments
RecommendationRequest Changes. Two problems block approval:
The missing PSRAM/rebase coverage and the unsanitized default configuration cover the riskiest new pointer handling, so add those tests in this PR too. The remaining items are minor. |
Record per-boundary conversion requirements at export so float and NCHW I/O can reuse the activation arena while compatible int8 I/O remains directly bound. Preserve independent ExecuTorch input/output planning and arena-backed I/O for older exports. Keep conversion metadata independent per partition. Host regression tests cover arena sizing, mixed boundaries, all four planning settings, and legacy exports. Validation: 268 host tests passed, 11 skipped. Independent review and six additional native host probes passed. Authored with Codex (OpenAI).
Remove the whole-method timing guarantee for concurrent resets. Each delegate independently applies a reset before its next execution. Authored with Codex (OpenAI).
The ESP32-S3 esp-nn kernels may load whole vectors past the end of a tensor they read; the general convolution reads up to 16 bytes past each row of its input. An input bound in place could be the caller's memory ending at the edge of mapped RAM. Graph inputs now always get slots in the delegate's activation arena and are copied into them, quantized or transposed if needed, and the arena gets 16 readable bytes after its last slot. The exporter records every input as a conversion slot, but the runtime plans inputs into the arena regardless of the recorded slots, so exports that left compatible inputs out of them still run. Int8Interpreter::bind_input() is removed so that nothing can bind one in place. Outputs keep their conversion slots and direct binding. Since inputs are copied before any operator runs, an output may be bound over an input's memory without staging, so the overlap check and the input staging slots go away.
Two esp-nn ESP32-S3 kernels return without writing all of their output, so the delegate produced whatever the output buffer held before. The max pool kernel handles only a multiple of four channels and writes nothing otherwise, and the pointwise convolution passes its number of output positions as a uint16_t width, leaving every position past 65535 unwritten. Both cases now use the ANSI kernels. invoke() also clears an output bound to memory outside the arena once it holds the invocation lock, so that a kernel leaving part of its output unwritten cannot let the caller's previous contents show through. The emulator and host suites gain a max pool over six channels.
An output that a later operator reads was still bound to the caller's buffer, so the ESP32-S3 kernels' vector loads past the end of a tensor could reach past that buffer, and an application that bound another output over it would change what the operator read. With external I/O, Int8Interpreter now gives every output that an operation reads an arena slot, whatever the export recorded, and reports those slots through conversion_slots(), so EspNNBackend copies such outputs out after the last operator like any other output in the arena. bind_output() refuses any output that has an arena slot. The host runner gains --share-output, and a host test binds two outputs to one buffer while an operator still reads the first.
Summary
This makes the ESP-NN delegate a good fit for small streaming models on ESP32-S3, like keyword spotting. For such a model, most of each step currently goes to the delegate's bookkeeping rather than the kernels: it times every operator, copies its tables into RAM, stages every input and output through its arena, and the streaming state goes in and out of the delegate on every call.
On a 32-operator keyword-spotting model, this brings a step from 594 µs to 444 µs and internal RAM from 48.6 KB to 32.2 KB, with bit-identical output. That puts ExecuTorch ahead of TFLite Micro running the same model on the same esp-nn kernels.
Models without streaming state don't gain much. On the backend's emulator test programs the runtime uses 0.2–4.7 KB less heap and runs 0.6–2% slower; the host build executes the same number of instructions, so it looks like code layout.
Review order
ProgramEmulatorhere.delegate_mutable_buffers=True: the delegate owns int8 mutable buffers and updates them in place. State is reset through thereset_statesbackend option (set_option) orreset_delegate_states().Compatibility
Anything that could change an existing export is opt-in, and the runtime still loads v5 and v6 programs. Programs exported after this PR use XTS3 v7 (v8 with delegate-owned state) and need the new runtime. The loader now rejects pooling and convolution outputs larger than 65535, which esp-nn truncates to
uint16_tand never handled correctly.Test plan
Every commit passes the Python suite and the host tests. The head also passes both emulator jobs and lintrunner. The device numbers are from an ESP32-S3 board over 330 test clips, measured before a few review fixes that aren't on this model's path.
I kept this as one PR, but I'm happy to split it into a stack if that's easier to review.
This PR was prepared with AI assistance (Claude).