Repository navigation
Conversation
Int8Interpreter::invoke() read esp_timer_get_time() before and after every operator to fill a per-operator profile, whether or not anything printed it. On ESP32-S3 that is two function calls per operator; for a 32-operator streaming keyword-spotting model it was about 55 us of a 540 us call, more than many of the operators themselves. The interpreter now times operators only with Int8RuntimeOptions::profile_operations, which EspNNBackend sets from CONFIG_ESPNN_DELEGATE_PROFILE, and keeps the times in a table allocated only then. Otherwise it reads the timer at the start and end of each call and every 16 operators, for its cooperative yield. last_invoke_us() becomes the wall time of the call.
The ESP-NN delegate only built inside ESP-IDF, so checking a runtime change needed the ESP32-S3 emulator or a board. Its sources use few ESP-IDF APIs: the heap_caps allocator, esp_timer, a FreeRTOS mutex and a task yield. Outside ESP-IDF, espnn_runtime/CMakeLists.txt now builds the same sources as the espnn_backend library, with host/include standing in for those APIs and ESP-NN's portable C kernels fetched from GitHub at the v1.5.0 commit (or taken from ESPNN_LOCAL_PATH). backends/espressif/test/host builds it with the ExecuTorch runtime into espnn_host_runner, and test_host_runtime.py runs every fully delegated test model through it, over several steps and several loaded instances of a method, with planned and with caller-owned inputs and outputs, checking each output bit-exactly against ProgramEmulator. .ci/scripts/test_espressif_host.sh runs the test on a plain build and on an AddressSanitizer and UndefinedBehaviorSanitizer build, in a new job of the Espressif workflow that needs no ESP-IDF toolchain. The host build exercises the delegate's planner, interpreter and ExecuTorch glue, not ESP-NN's S3 assembly kernels, which the emulator jobs still cover.
On a 512 KB-SRAM part the delegate's own bookkeeping can cost more internal RAM than a small model's activations. For a 32-operator streaming keyword-spotting model on ESP32-S3 it kept 10.5 KB of per-operator state, 1.4 KB of operation records that are only needed while loading, and a separate 5.8 KB ESP-NN scratch buffer, and graph validation put four 1,024-byte bool arrays on the loading task's stack. Fields of the per-operator state that no operator uses together now share storage, operation records are decoded while loading instead of being kept, the planner frees its tables before allocating the arena, and graph validation uses bitsets. The invoke mutex is created by a static initializer instead of a function-local static, so invoke() no longer goes through a thread-safe static guard. ESPNN_DELEGATE_SCRATCH_IN_ARENA (off by default) plans each operator's ESP-NN scratch as an arena entry that lives only while the operator runs; for a model whose arena stays in internal RAM this replaces the separate buffer. DelegateStats now also reports the scratch, requantization and table bytes. The host test checks the delegate's arena against plan_activation_arena(), its Python mirror, and the sanitized host build runs with scratch in the arena and per-operator profiling on.
The delegate copied every tensor record and every per-channel requantization multiplier and shift out of its graph into RAM, although the graph already sits in memory-mapped flash or in PSRAM. For a 32-operator streaming keyword-spotting model on ESP32-S3 the two copies took 4.4 KB of internal RAM. They were needed because XTS3 v6 places those tables at offsets that are not word-aligned: the 22-byte header and the input/output table leave the tensor records, and so everything after them, 2 bytes off a word boundary. XTS3 v7 is v6 with the input/output table padded to a 4-byte boundary and each operation's parameter block starting on one. The serializer now writes v7. On a little-endian target the runtime uses a v7 graph's tensor records and requantization arrays (and softmax tables) where they are, and still copies them for v5 and v6 graphs, which it keeps accepting. When the delegate copies the graph's constants into PSRAM, the in-place pointers follow the copy. A program exported with this version needs a runtime from the same version or later; the runtime rejects unknown graph versions at init with DelegateInvalidCompatibility. espnn_host_interpreter runs a graph on the host build of the interpreter from addresses of every alignment, and test_host_interpreter.py checks that v6 and v7 graphs give bit-exact results from all of them and that v7 graphs keep no copy of their requantization parameters.
The delegate copied every input into its activation arena before running and every output out of it afterwards, transposing rank-4 tensors between NCHW and its NHWC device layout element by element. For a streaming keyword-spotting model on ESP32-S3 whose state tensors cross the delegate boundary on every step, that was both the largest share of the delegate's non-kernel time and the reason the arena held a copy of every input and output. With Int8RuntimeOptions::external_io, which EspNNBackend now sets, the planner leaves graph inputs and outputs out of the arena and the caller binds a buffer to each before invoke(). The backend binds the ExecuTorch tensor itself when its memory is 16-byte aligned, like the arena, and already in the device layout: an int8 tensor that is not rank 4, has one channel or one pixel, or has a channels-last dim order. Any other input or output (float, NCHW or misaligned) goes through a zero-initialized staging buffer that the first execute() needing one allocates, which keeps existing programs working unchanged. Because operators may read an input after writing an output, an output whose memory overlaps an input is always staged and copied out after the last operator; this makes binding one buffer as both an input and an output (an application updating a state tensor in place) safe instead of depending on the order of the delegate's copies. A graph tensor that the delegate returns as several outputs (when its value feeds several CPU operators) is written once and copied to the other outputs, and a graph input that is also returned as an output is staged and copied from there. The first COPY writing an output bound to caller memory, unless it fills it, clears it first, so the buffer's previous contents never show through; a long cat is written by several COPY operations. Pooling and convolution outputs wider or taller than 65535, which ESP-NN's kernels would truncate to uint16_t and so leave unwritten, are now rejected at load. A graph whose every activation is an input or output needs no arena at all. The interpreter keeps a pointer per tensor instead of an arena offset, so binding costs nothing per operator. plan_activation_arena() mirrors the external-I/O plan, the README describes how to export int8 channels-last I/O that is used in place, and the tests cover channels-last I/O without staging, misaligned caller buffers, an output bound to the memory of an input that operators read after the output is written, an input returned as an output (also with another output bound over it), a partial COPY into a bound output, a seven-way cat as an output, those limits, a graph with no arena, and, with the portable and quantized kernels now linked into the host runner, the programs that leave operators on the CPU between delegates.
A streaming model keeps a few frames of history in mutable buffers. Lowered to the ESP-NN delegate today, each buffer stays in the program: it is passed into the delegate and its new value out of it on every call, in float, or in int8 when the application exports it as a stateless input and output and carries the state itself. For a 32-filter streaming keyword-spotting model on ESP32-S3 with six state tensors, those copies, their layout conversions and the extra delegate arguments took about a fifth of each call. EspNNPartitioner(delegate_mutable_buffers=True), also exposed by to_espnn_edge(), gives a mutable buffer to the delegate that holds every access to it, through tag_mutated_buffer(), when the buffer is read and written with the same int8 quantization: a float32 buffer read through quantize ops (what PT2E produces for a cat and slice shift register) or an int8 buffer read through dequantize ops. Other buffers stay in the program, as they do by default. The partition's graph is then XTS3 version 8, which adds a table of (state, update) tensor pairs; each state's quantized initial value is stored with the constants. An update that the runtime cannot write into a state's slot (a graph input or output, another state, or an update shared by several states) gets a copy of its own, made after the last operator. The runtime keeps each state in a reserved arena slot and initializes it at load. The update is planned into the state's own slot when no operator reads the state at or after the update's first write, and copied over the state after the last operator otherwise. Every loaded method instance has its own state. reset_delegate_states(), or the reset_states backend option through executorch::runtime::set_option(), restores the initial state of every EspNNBackend delegate before its next execute(); reloading a method resets it alone. ProgramEmulator and the reference interpreter keep the same state across calls and can reset it, and plan_activation_arena() mirrors the in-place update. New tests cover which buffers the delegate takes, the reference against the quantized PyTorch model across steps and a reset, and, on the host build, states updated in place and by copy, a state updated from a graph input, a reset, two instances, and the parser's checks of the state table. The emulator suite gains a stateful program.
The pieces a streaming keyword-spotting model on ESP32-S3 needs (int8, channels-last or trivially laid out inputs and outputs left to the application's buffers, mutable buffers owned by the delegate, and a quantization that matches the model's reference) were each documented on their own. export_streaming_model.py puts them together on a small model whose program has one int8 input, one int8 output, no CPU operators and no planned memory, and the README adds the quantization choices that matter for such models and the device-side pattern: bind once, execute per frame, reset the delegate's state on a detection. The host test runs the example's program for several frames without any staging copy.
Record per-boundary conversion requirements at export so float and NCHW I/O can reuse the activation arena while compatible int8 I/O remains directly bound. Preserve independent ExecuTorch input/output planning and arena-backed I/O for older exports. Keep conversion metadata independent per partition. Host regression tests cover arena sizing, mixed boundaries, all four planning settings, and legacy exports. Validation: 268 host tests passed, 11 skipped. Independent review and six additional native host probes passed. Authored with Codex (OpenAI).
Remove the whole-method timing guarantee for concurrent resets. Each delegate independently applies a reset before its next execution. Authored with Codex (OpenAI).
The ESP32-S3 esp-nn kernels may load whole vectors past the end of a tensor they read; the general convolution reads up to 16 bytes past each row of its input. An input bound in place could be the caller's memory ending at the edge of mapped RAM. Graph inputs now always get slots in the delegate's activation arena and are copied into them, quantized or transposed if needed, and the arena gets 16 readable bytes after its last slot. The exporter records every input as a conversion slot, but the runtime plans inputs into the arena regardless of the recorded slots, so exports that left compatible inputs out of them still run. Int8Interpreter::bind_input() is removed so that nothing can bind one in place. Outputs keep their conversion slots and direct binding. Since inputs are copied before any operator runs, an output may be bound over an input's memory without staging, so the overlap check and the input staging slots go away.
Two esp-nn ESP32-S3 kernels return without writing all of their output, so the delegate produced whatever the output buffer held before. The max pool kernel handles only a multiple of four channels and writes nothing otherwise, and the pointwise convolution passes its number of output positions as a uint16_t width, leaving every position past 65535 unwritten. Both cases now use the ANSI kernels. invoke() also clears an output bound to memory outside the arena once it holds the invocation lock, so that a kernel leaving part of its output unwritten cannot let the caller's previous contents show through. The emulator and host suites gain a max pool over six channels.
An output that a later operator reads was still bound to the caller's buffer, so the ESP32-S3 kernels' vector loads past the end of a tensor could reach past that buffer, and an application that bound another output over it would change what the operator read. With external I/O, Int8Interpreter now gives every output that an operation reads an arena slot, whatever the export recorded, and reports those slots through conversion_slots(), so EspNNBackend copies such outputs out after the last operator like any other output in the arena. bind_output() refuses any output that has an arena slot. The host runner gains --share-output, and a host test binds two outputs to one buffer while an operator still reads the first.
The host build fetched esp-nn v1.5.0 while idf_component.yml pins v1.4.1 for the device, so the host tests checked kernels that no device build runs. Fetch v1.4.1 by commit, matching the pin.
Without profiling, invoke() read the timer every sixteen operators to decide whether to yield, so the longest stretch without a yield depended on how slow those operators were. It now reads the cycle counter, a few cycles, after every operator, and the timer only once 2^20 cycles (4 ms at 240 MHz) have passed. A yield still comes within a second plus one operator. On an ESP32-S3 the streaming keyword model runs at 440 us per step this way, against 466 us with a timer read per operator. The host build gets an esp_cpu.h stand-in.
The host build had no PSRAM and esp-nn's portable kernels ask for no scratch, so the sanitized host runs never exercised the PSRAM constants copy or either scratch placement, which only the device used. With ESPNN_HOST_PSRAM the heap stand-in serves PSRAM allocations, and it now allocates exact sizes so AddressSanitizer sees any overrun. ESPNN_DELEGATE_CONSTANTS_TO_PSRAM becomes a host CMake option. With ESPNN_HOST_SCRATCH, a test-only esp_nn.h wrapper makes the convolution, depthwise and mean kernels request scratch and overwrite all of it before they run, so a scratch buffer that is too small fails under ASan and one placed over live activations fails the bit-exact checks. The interpreter runner frees its copy of the graph once the interpreter has copied it, as EspNNBackend does. The CI script runs a plain build and three sanitized ones: default options, optimized kernels, and every option on.
Int8Interpreter::load() allocated the requantization tables, the dilated convolution tile buffer and the separate ESP-NN scratch while it prepared the operations, before planning memory, so they took internal RAM first and could push the activation arena, which every operator reads and writes, out to PSRAM. prepare_operations() now only works out those sizes, plan_memory() allocates the arena, and the new allocate_operation_buffers() then allocates the tables and buffers and points the operations at them. The host heap stand-in can limit internal RAM through ESPNN_HOST_INTERNAL_BYTES, and a host test gives it room for the arena and the reserve only.
An output outside the arena whose ExecuTorch buffer is not 16-byte aligned is written to a staging buffer and copied out, and that buffer was allocated by the first execute() that needed one, in internal RAM regardless of the interpreter's reserve. A program could then fail, or starve the rest of the application, in the middle of running. EspNNBackend::init() now allocates the staging buffer, sized for the outputs outside the arena that could need it. Only misaligned outputs use it, so it goes to PSRAM when there is any and leaves internal RAM to the arena. Nothing is allocated during execute() again, as the README says. A new counter, DelegateStats::staged_outputs, counts the outputs written through it, and the host tests check that staging happens without growing the buffer.
An unplanned input or output that the application never gave memory reached EspNNBackend::execute() with a null data pointer. The delegate read or wrote through it, or for an output written in place, failed inside the interpreter with Error::Internal and no hint of the cause. execute() now rejects such a tensor before running, printing "input N has no data" or "output N has no data" and returning Error::InvalidArgument. The host runner can leave an unplanned output unbound to test this.
The ESP32-S3 kernels may rely on the arena's 16-byte alignment for their outputs, and EspNNBackend checked that before binding an ExecuTorch tensor, but Int8Interpreter::bind_output() itself took any pointer, so another caller of the interpreter could bind a buffer that a vector store would write around. bind_output() now refuses a null or misaligned buffer and documents the requirement, and EspNNBackend relies on it, staging any output it refuses. The interpreter runner checks that a misaligned buffer is refused before it binds the aligned one.
The device loader rejects a convolution, pooling or fully connected operation whose input, output or weight dimensions exceed 65535, because ESP-NN takes them as uint16_t, but check_runtime_limits() did not, so such a program exported and then failed at init() on the device. check_runtime_limits() now applies the same limits as the loader, with tests on both sides of 65535 for each operation.
With delegate_mutable_buffers, the partitioner gave each mutable buffer to whichever partition held its accesses. A method whose buffers fell in two partitions then had its state in two delegates, and since each delegate applies a reset at the start of its own execute(), a reset could reach one part of the state and not yet the other. A partition with more than 16 buffers failed to export, and owning a buffer could leave a partition with no runtime input or output, which an XTS3 graph needs. The partitioner now gives a method's buffers to the delegate only when one partition holds all of them and still has a runtime input and output once it owns them; constants do not count, since a shared one is duplicated into the partition. Otherwise the buffers stay in the program. A partition takes at most 16 buffers and leaves the rest in the program. Each case logs a warning. Python tests cover two partitions, 17 buffers, a state that is the only input or output, and a constant shared with the CPU, and a host test runs the two-partition program.
A delegate-owned state whose old value the partition also returned, directly or through an operation lowering treats as an alias such as clone, was serialized as both a state and a graph output. The runtime rejects that, so the program exported and then failed at init(). The serializer now returns such a state through a COPY after every other operation, the same way it separates a state's update from a graph output. A Python test checks the graph, and a host test runs both forms against the reference.
reset_delegate_states() and the reset_states backend option reset the state of every EspNNBackend delegate, so an application with several stateful methods could reset one of them only by loading it again. A method can now be loaded with a reset_group runtime spec from 0 to 7, passed through the existing LoadBackendOptionsMap, and reset_states accepts that group as an int, as does the new reset_delegate_states(int). Each group keeps its own generation counter, so a delegate restores its initial state at the start of its next execute() only when its own group was reset. Loading with an invalid group fails with Error::InvalidArgument. The host runner loads instance k in group k, and a host test resets one of two instances.
The README did not say which runtimes run a program, and its examples bound buffers without the 16-byte alignment that lets the delegate write outputs in place. It gains a "Program compatibility" section: programs need a runtime from the same ExecuTorch version or later, since older runtimes reject XTS3 version 7 and 8 graphs at init(), while older programs still run. The binding examples declare their buffers alignas(16). The DelegateStats and OpState comments now say what they count and share.
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25163
Note: Links to docs will display an error until the docs builds have been completed. ❌ 2 New FailuresAs of commit dc0504f with merge base 62e28de ( NEW FAILURES - The following jobs have failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
This PR needs a
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #25106. Until that merges, this diff also shows its commits; the twelve commits after 6a807c5 are new. I'll rebase onto main and mark this ready once #25106 lands.
Summary
These are the remaining fixes from the automated review of #25106, rebuilt on top of the I/O design that landed there (inputs copied into the arena, compatible int8 outputs written in place, export-time conversion slots). They make the delegate's memory use predictable: the arena gets internal RAM first, nothing is allocated during
execute(), and a method's streaming state is reset all at once or not at all. They also tighten a few contracts and extend the host tests to the configurations that previously only ran on the device.Review order
ESPNN_DELEGATE_CONSTANTS_TO_PSRAMon the host, and a test-only esp-nn wrapper that makes the kernels request and overwrite scratch, so both scratch placements run under ASan. The CI script runs a plain build and three sanitized ones.init()instead of on the firstexecute()that needs it. Only misaligned outputs use it, so it goes to PSRAM when there is any.Error::InvalidArgumentand a clear message.Int8Interpreter::bind_output()refuses null or misaligned buffers, and the backend relies on that check.check_runtime_limits()applies the loader's 65535 limits for convolution, pooling and fully connected tensors, so such programs fail at export instead of atinit().reset_groupload-time option (0 to 7, throughLoadBackendOptionsMap), andreset_statesaccepts that group as well astrue.alignas(16)in the binding examples, and small wording fixes.Results
On an ESP32-S3 (SenseCAP Watcher), an earlier revision of this work ran the streaming keyword model at 440 µs per step with the cycle-counter gate, against 466 µs with a timer read after every operator, and 32.2 KB of internal RAM. Everything else here is load-time or error-path behavior, checked on the host and in the emulator.
Compatibility
The XTS3 format is unchanged, so programs exported with #25106 load as before.
reset_statesstill takestrue, and the reset group defaults to 0. Three things can differ from #25106: a program with an output outside the arena now holds its staging buffer frominit()(in PSRAM when the board has it, internal RAM otherwise); stateful exports whose buffers end up in several partitions now keep them in the program instead of splitting them; and convolutions or pools wider than 65535 fail at export rather than on the device.Test plan
Every commit passes the Python suite, the sanitized host build with every option on, and lintrunner. The head also passes all four host builds and both emulator jobs.
This PR was prepared with AI assistance (Claude).