Skip to content

[ESP-NN] Follow-ups to #25106: load order, no allocation in execute(), per-method state reset - #25163

Draft
anantn wants to merge 25 commits into
pytorch:mainfrom
anantn:espnn-delegate-followups
Draft

anantn wants to merge 25 commits into
pytorch:mainfrom
anantn:espnn-delegate-followups

Conversation

@anantn

@anantn anantn commented Nov 17, 2026

Copy link
Copy Markdown

Depends on #25106. Until that merges, this diff also shows its commits; the twelve commits after 6a807c5 are new. I'll rebase onto main and mark this ready once #25106 lands.

Summary

These are the remaining fixes from the automated review of #25106, rebuilt on top of the I/O design that landed there (inputs copied into the arena, compatible int8 outputs written in place, export-time conversion slots). They make the delegate's memory use predictable: the arena gets internal RAM first, nothing is allocated during execute(), and a method's streaming state is reset all at once or not at all. They also tighten a few contracts and extend the host tests to the configurations that previously only ran on the device.

Review order

  1. Build the host tests against esp-nn 1.4.1, the version the device pins, instead of 1.5.0.
  2. Gate the cooperative yield on the cycle counter, so the timer is read only every ~4 ms instead of every 16 operators.
  3. Host coverage: a simulated PSRAM heap, ESPNN_DELEGATE_CONSTANTS_TO_PSRAM on the host, and a test-only esp-nn wrapper that makes the kernels request and overwrite scratch, so both scratch placements run under ASan. The CI script runs a plain build and three sanitized ones.
  4. Allocate the requantization tables, tile buffer and scratch after the arena, so the arena gets the first claim on internal RAM.
  5. Allocate the output staging buffer at init() instead of on the first execute() that needs it. Only misaligned outputs use it, so it goes to PSRAM when there is any.
  6. Reject inputs and outputs with no data pointer with Error::InvalidArgument and a clear message.
  7. Int8Interpreter::bind_output() refuses null or misaligned buffers, and the backend relies on that check.
  8. check_runtime_limits() applies the loader's 65535 limits for convolution, pooling and fully connected tensors, so such programs fail at export instead of at init().
  9. Give a method's mutable buffers to the delegate only when one partition holds all of them and still has a runtime input and output, and at most 16 of them. Anything else stays in the program, with a warning.
  10. Return a delegate-owned state that the partition also returns through a copy, since the runtime rejects a state that is also a graph output.
  11. Per-method reset: a reset_group load-time option (0 to 7, through LoadBackendOptionsMap), and reset_states accepts that group as well as true.
  12. README: a "Program compatibility" section, alignas(16) in the binding examples, and small wording fixes.

Results

On an ESP32-S3 (SenseCAP Watcher), an earlier revision of this work ran the streaming keyword model at 440 µs per step with the cycle-counter gate, against 466 µs with a timer read after every operator, and 32.2 KB of internal RAM. Everything else here is load-time or error-path behavior, checked on the host and in the emulator.

Compatibility

The XTS3 format is unchanged, so programs exported with #25106 load as before. reset_states still takes true, and the reset group defaults to 0. Three things can differ from #25106: a program with an output outside the arena now holds its staging buffer from init() (in PSRAM when the board has it, internal RAM otherwise); stateful exports whose buffers end up in several partitions now keep them in the program instead of splitting them; and convolutions or pools wider than 65535 fail at export rather than on the device.

Test plan

pytest -n 16 backends/espressif/test
.ci/scripts/test_espressif_host.sh
ESPRESSIF_NN_KERNELS=optimized bash .ci/scripts/test_espressif.sh
ESPRESSIF_NN_KERNELS=ansi bash .ci/scripts/test_espressif.sh
lintrunner

Every commit passes the Python suite, the sanitized host build with every option on, and lintrunner. The head also passes all four host builds and both emulator jobs.

This PR was prepared with AI assistance (Claude).

anantn and others added 25 commits November 15, 2026 18:28
Int8Interpreter::invoke() read esp_timer_get_time() before and after every
operator to fill a per-operator profile, whether or not anything printed it.
On ESP32-S3 that is two function calls per operator; for a 32-operator
streaming keyword-spotting model it was about 55 us of a 540 us call, more
than many of the operators themselves.

The interpreter now times operators only with
Int8RuntimeOptions::profile_operations, which EspNNBackend sets from
CONFIG_ESPNN_DELEGATE_PROFILE, and keeps the times in a table allocated only
then. Otherwise it reads the timer at the start and end of each call and
every 16 operators, for its cooperative yield. last_invoke_us() becomes the
wall time of the call.
The ESP-NN delegate only built inside ESP-IDF, so checking a runtime change
needed the ESP32-S3 emulator or a board. Its sources use few ESP-IDF APIs: the
heap_caps allocator, esp_timer, a FreeRTOS mutex and a task yield. Outside
ESP-IDF, espnn_runtime/CMakeLists.txt now builds the same sources as the
espnn_backend library, with host/include standing in for those APIs and
ESP-NN's portable C kernels fetched from GitHub at the v1.5.0 commit (or taken
from ESPNN_LOCAL_PATH).

backends/espressif/test/host builds it with the ExecuTorch runtime into
espnn_host_runner, and test_host_runtime.py runs every fully delegated test
model through it, over several steps and several loaded instances of a method,
with planned and with caller-owned inputs and outputs, checking each output
bit-exactly against ProgramEmulator. .ci/scripts/test_espressif_host.sh runs
the test on a plain build and on an AddressSanitizer and
UndefinedBehaviorSanitizer build, in a new job of the Espressif workflow that
needs no ESP-IDF toolchain.

The host build exercises the delegate's planner, interpreter and ExecuTorch
glue, not ESP-NN's S3 assembly kernels, which the emulator jobs still cover.
On a 512 KB-SRAM part the delegate's own bookkeeping can cost more internal
RAM than a small model's activations. For a 32-operator streaming
keyword-spotting model on ESP32-S3 it kept 10.5 KB of per-operator state,
1.4 KB of operation records that are only needed while loading, and a
separate 5.8 KB ESP-NN scratch buffer, and graph validation put four
1,024-byte bool arrays on the loading task's stack.

Fields of the per-operator state that no operator uses together now share
storage, operation records are decoded while loading instead of being kept,
the planner frees its tables before allocating the arena, and graph
validation uses bitsets. The invoke mutex is created by a static initializer
instead of a function-local static, so invoke() no longer goes through a
thread-safe static guard. ESPNN_DELEGATE_SCRATCH_IN_ARENA (off by default)
plans each operator's ESP-NN scratch as an arena entry that lives only while
the operator runs; for a model whose arena stays in internal RAM this
replaces the separate buffer.

DelegateStats now also reports the scratch, requantization and table bytes.
The host test checks the delegate's arena against plan_activation_arena(),
its Python mirror, and the sanitized host build runs with scratch in the
arena and per-operator profiling on.
The delegate copied every tensor record and every per-channel requantization
multiplier and shift out of its graph into RAM, although the graph already
sits in memory-mapped flash or in PSRAM. For a 32-operator streaming
keyword-spotting model on ESP32-S3 the two copies took 4.4 KB of internal RAM.
They were needed because XTS3 v6 places those tables at offsets that are not
word-aligned: the 22-byte header and the input/output table leave the
tensor records, and so everything after them, 2 bytes off a word boundary.

XTS3 v7 is v6 with the input/output table padded to a 4-byte boundary and each
operation's parameter block starting on one. The serializer now writes v7. On a
little-endian target the runtime uses a v7 graph's tensor records and
requantization arrays (and softmax tables) where they are, and still copies
them for v5 and v6 graphs, which it keeps accepting. When the delegate copies
the graph's constants into PSRAM, the in-place pointers follow the copy.

A program exported with this version needs a runtime from the same version or
later; the runtime rejects unknown graph versions at init with
DelegateInvalidCompatibility.

espnn_host_interpreter runs a graph on the host build of the interpreter from
addresses of every alignment, and test_host_interpreter.py checks that v6 and
v7 graphs give bit-exact results from all of them and that v7 graphs keep no
copy of their requantization parameters.
The delegate copied every input into its activation arena before running and
every output out of it afterwards, transposing rank-4 tensors between NCHW
and its NHWC device layout element by element. For a streaming
keyword-spotting model on ESP32-S3 whose state tensors cross the delegate
boundary on every step, that was both the largest share of the delegate's
non-kernel time and the reason the arena held a copy of every input and
output.

With Int8RuntimeOptions::external_io, which EspNNBackend now sets, the planner
leaves graph inputs and outputs out of the arena and the caller binds a buffer
to each before invoke(). The backend binds the ExecuTorch tensor itself when
its memory is 16-byte aligned, like the arena, and already in the device
layout: an int8 tensor that is not rank 4, has one channel or one pixel, or
has a channels-last dim order. Any other input or output (float, NCHW or
misaligned) goes through a zero-initialized staging buffer that the first
execute() needing one allocates, which keeps existing programs working
unchanged. Because operators may read an input after writing an output, an
output whose memory overlaps an input is always staged and copied out after
the last operator; this makes binding one buffer as both an input and an
output (an application updating a state tensor in place) safe instead of
depending on the order of the delegate's copies.

A graph tensor that the delegate returns as several outputs (when its value
feeds several CPU operators) is written once and copied to the other outputs,
and a graph input that is also returned as an output is staged and copied from
there. The first COPY writing an output bound to caller memory, unless it
fills it, clears it first, so the buffer's previous contents never show
through; a long cat is written by several COPY operations. Pooling and
convolution outputs wider or taller than 65535, which ESP-NN's kernels would
truncate to uint16_t and so leave unwritten, are now rejected at load. A graph
whose every activation is an input or output needs no arena at all. The
interpreter keeps a pointer per tensor instead of an arena offset, so binding
costs nothing per operator. plan_activation_arena() mirrors the external-I/O
plan, the README describes how to export int8 channels-last I/O that is used
in place, and the tests cover channels-last I/O without staging, misaligned
caller buffers, an output bound to the memory of an input that operators read
after the output is written, an input returned as an output (also with another
output bound over it), a partial COPY into a bound output, a seven-way cat as
an output, those limits, a graph with no arena, and, with the portable and
quantized kernels now linked into the host runner, the programs that leave
operators on the CPU between delegates.
A streaming model keeps a few frames of history in mutable buffers. Lowered
to the ESP-NN delegate today, each buffer stays in the program: it is passed
into the delegate and its new value out of it on every call, in float, or in
int8 when the application exports it as a stateless input and output and
carries the state itself. For a 32-filter streaming keyword-spotting model on
ESP32-S3 with six state tensors, those copies, their layout conversions and
the extra delegate arguments took about a fifth of each call.

EspNNPartitioner(delegate_mutable_buffers=True), also exposed by
to_espnn_edge(), gives a mutable buffer to the delegate that holds every
access to it, through tag_mutated_buffer(), when the buffer is read and
written with the same int8 quantization: a float32 buffer read through
quantize ops (what PT2E produces for a cat and slice shift register) or an
int8 buffer read through dequantize ops. Other buffers stay in the program, as
they do by default. The partition's graph is then XTS3 version 8, which adds a
table of (state, update) tensor pairs; each state's quantized initial value is
stored with the constants. An update that the runtime cannot write into a
state's slot (a graph input or output, another state, or an update shared by
several states) gets a copy of its own, made after the last operator.

The runtime keeps each state in a reserved arena slot and initializes it at
load. The update is planned into the state's own slot when no operator reads
the state at or after the update's first write, and copied over the state
after the last operator otherwise. Every loaded method instance has its own
state. reset_delegate_states(), or the reset_states backend option through
executorch::runtime::set_option(), restores the initial state of every
EspNNBackend delegate before its next execute(); reloading a method resets
it alone.

ProgramEmulator and the reference interpreter keep the same state across calls
and can reset it, and plan_activation_arena() mirrors the in-place update. New
tests cover which buffers the delegate takes, the reference against the
quantized PyTorch model across steps and a reset, and, on the host build,
states updated in place and by copy, a state updated from a graph input, a
reset, two instances, and the parser's checks of the state table. The emulator
suite gains a stateful program.
The pieces a streaming keyword-spotting model on ESP32-S3 needs (int8,
channels-last or trivially laid out inputs and outputs left to the
application's buffers, mutable buffers owned by the delegate, and a
quantization that matches the model's reference) were each documented on
their own. export_streaming_model.py puts them together on a small model whose
program has one int8 input, one int8 output, no CPU operators and no planned
memory, and the README adds the quantization choices that matter for such
models and the device-side pattern: bind once, execute per frame, reset the
delegate's state on a detection. The host test runs the example's program for
several frames without any staging copy.
Record per-boundary conversion requirements at export so float and NCHW I/O can reuse the activation arena while compatible int8 I/O remains directly bound. Preserve independent ExecuTorch input/output planning and arena-backed I/O for older exports.

Keep conversion metadata independent per partition. Host regression tests cover arena sizing, mixed boundaries, all four planning settings, and legacy exports.

Validation: 268 host tests passed, 11 skipped. Independent review and six additional native host probes passed.

Authored with Codex (OpenAI).
Remove the whole-method timing guarantee for concurrent resets. Each delegate independently applies a reset before its next execution.

Authored with Codex (OpenAI).
The ESP32-S3 esp-nn kernels may load whole vectors past the end of a tensor
they read; the general convolution reads up to 16 bytes past each row of its
input. An input bound in place could be the caller's memory ending at the edge
of mapped RAM.

Graph inputs now always get slots in the delegate's activation arena and are
copied into them, quantized or transposed if needed, and the arena gets 16
readable bytes after its last slot. The exporter records every input as a
conversion slot, but the runtime plans inputs into the arena regardless of the
recorded slots, so exports that left compatible inputs out of them still run.
Int8Interpreter::bind_input() is removed so that nothing can bind one in
place. Outputs keep their conversion slots and direct binding. Since inputs
are copied before any operator runs, an output may be bound over an input's
memory without staging, so the overlap check and the input staging slots go
away.
Two esp-nn ESP32-S3 kernels return without writing all of their output, so
the delegate produced whatever the output buffer held before. The max pool
kernel handles only a multiple of four channels and writes nothing otherwise,
and the pointwise convolution passes its number of output positions as a
uint16_t width, leaving every position past 65535 unwritten. Both cases now
use the ANSI kernels.

invoke() also clears an output bound to memory outside the arena once it holds
the invocation lock, so that a kernel leaving part of its output unwritten
cannot let the caller's previous contents show through. The emulator and host
suites gain a max pool over six channels.
An output that a later operator reads was still bound to the caller's buffer,
so the ESP32-S3 kernels' vector loads past the end of a tensor could reach
past that buffer, and an application that bound another output over it would
change what the operator read.

With external I/O, Int8Interpreter now gives every output that an operation
reads an arena slot, whatever the export recorded, and reports those slots
through conversion_slots(), so EspNNBackend copies such outputs out after the
last operator like any other output in the arena. bind_output() refuses any
output that has an arena slot. The host runner gains --share-output, and a
host test binds two outputs to one buffer while an operator still reads the
first.
The host build fetched esp-nn v1.5.0 while idf_component.yml pins v1.4.1 for
the device, so the host tests checked kernels that no device build runs. Fetch
v1.4.1 by commit, matching the pin.
Without profiling, invoke() read the timer every sixteen operators to decide
whether to yield, so the longest stretch without a yield depended on how slow
those operators were. It now reads the cycle counter, a few cycles, after
every operator, and the timer only once 2^20 cycles (4 ms at 240 MHz) have
passed. A yield still comes within a second plus one operator. On an ESP32-S3
the streaming keyword model runs at 440 us per step this way, against 466 us
with a timer read per operator. The host build gets an esp_cpu.h stand-in.
The host build had no PSRAM and esp-nn's portable kernels ask for no scratch,
so the sanitized host runs never exercised the PSRAM constants copy or either
scratch placement, which only the device used.

With ESPNN_HOST_PSRAM the heap stand-in serves PSRAM allocations, and it now
allocates exact sizes so AddressSanitizer sees any overrun.
ESPNN_DELEGATE_CONSTANTS_TO_PSRAM becomes a host CMake option. With
ESPNN_HOST_SCRATCH, a test-only esp_nn.h wrapper makes the convolution,
depthwise and mean kernels request scratch and overwrite all of it before they
run, so a scratch buffer that is too small fails under ASan and one placed over
live activations fails the bit-exact checks. The interpreter runner frees its
copy of the graph once the interpreter has copied it, as EspNNBackend does.
The CI script runs a plain build and three sanitized ones: default options,
optimized kernels, and every option on.
Int8Interpreter::load() allocated the requantization tables, the dilated
convolution tile buffer and the separate ESP-NN scratch while it prepared the
operations, before planning memory, so they took internal RAM first and could
push the activation arena, which every operator reads and writes, out to
PSRAM.

prepare_operations() now only works out those sizes, plan_memory() allocates
the arena, and the new allocate_operation_buffers() then allocates the tables
and buffers and points the operations at them. The host heap stand-in can
limit internal RAM through ESPNN_HOST_INTERNAL_BYTES, and a host test gives
it room for the arena and the reserve only.
An output outside the arena whose ExecuTorch buffer is not 16-byte aligned is
written to a staging buffer and copied out, and that buffer was allocated by
the first execute() that needed one, in internal RAM regardless of the
interpreter's reserve. A program could then fail, or starve the rest of the
application, in the middle of running.

EspNNBackend::init() now allocates the staging buffer, sized for the outputs
outside the arena that could need it. Only misaligned outputs use it, so it
goes to PSRAM when there is any and leaves internal RAM to the arena. Nothing
is allocated during execute() again, as the README says. A new counter,
DelegateStats::staged_outputs, counts the outputs written through it, and the
host tests check that staging happens without growing the buffer.
An unplanned input or output that the application never gave memory reached
EspNNBackend::execute() with a null data pointer. The delegate read or wrote
through it, or for an output written in place, failed inside the interpreter
with Error::Internal and no hint of the cause.

execute() now rejects such a tensor before running, printing "input N has
no data" or "output N has no data" and returning Error::InvalidArgument.
The host runner can leave an unplanned output unbound to test this.
The ESP32-S3 kernels may rely on the arena's 16-byte alignment for their
outputs, and EspNNBackend checked that before binding an ExecuTorch tensor,
but Int8Interpreter::bind_output() itself took any pointer, so another caller
of the interpreter could bind a buffer that a vector store would write around.

bind_output() now refuses a null or misaligned buffer and documents the
requirement, and EspNNBackend relies on it, staging any output it refuses.
The interpreter runner checks that a misaligned buffer is refused before it
binds the aligned one.
The device loader rejects a convolution, pooling or fully connected operation
whose input, output or weight dimensions exceed 65535, because ESP-NN takes
them as uint16_t, but check_runtime_limits() did not, so such a program
exported and then failed at init() on the device.

check_runtime_limits() now applies the same limits as the loader, with tests
on both sides of 65535 for each operation.
With delegate_mutable_buffers, the partitioner gave each mutable buffer to
whichever partition held its accesses. A method whose buffers fell in two
partitions then had its state in two delegates, and since each delegate
applies a reset at the start of its own execute(), a reset could reach one
part of the state and not yet the other. A partition with more than 16
buffers failed to export, and owning a buffer could leave a partition with no
runtime input or output, which an XTS3 graph needs.

The partitioner now gives a method's buffers to the delegate only when one
partition holds all of them and still has a runtime input and output once it
owns them; constants do not count, since a shared one is duplicated into the
partition. Otherwise the buffers stay in the program. A partition takes at
most 16 buffers and leaves the rest in the program. Each case logs a warning.
Python tests cover two partitions, 17 buffers, a state that is the only
input or output, and a constant shared with the CPU, and a host test runs the
two-partition program.
A delegate-owned state whose old value the partition also returned, directly
or through an operation lowering treats as an alias such as clone, was
serialized as both a state and a graph output. The runtime rejects that, so
the program exported and then failed at init().

The serializer now returns such a state through a COPY after every other
operation, the same way it separates a state's update from a graph output.
A Python test checks the graph, and a host test runs both forms against the
reference.
reset_delegate_states() and the reset_states backend option reset the state
of every EspNNBackend delegate, so an application with several stateful
methods could reset one of them only by loading it again.

A method can now be loaded with a reset_group runtime spec from 0 to 7, passed
through the existing LoadBackendOptionsMap, and reset_states accepts that
group as an int, as does the new reset_delegate_states(int). Each group keeps
its own generation counter, so a delegate restores its initial state at the
start of its next execute() only when its own group was reset. Loading with
an invalid group fails with Error::InvalidArgument. The host runner loads
instance k in group k, and a host test resets one of two instances.
The README did not say which runtimes run a program, and its examples bound
buffers without the 16-byte alignment that lets the delegate write outputs in
place. It gains a "Program compatibility" section: programs need a runtime
from the same ExecuTorch version or later, since older runtimes reject XTS3
version 7 and 8 graphs at init(), while older programs still run. The
binding examples declare their buffers alignas(16). The DelegateStats and
OpState comments now say what they count and share.
@pytorch-bot

pytorch-bot Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25163

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures

As of commit dc0504f with merge base 62e28de (image):

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 17, 2026
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants