Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
a18ed53
Read the ESP-NN delegate's per-operator timers only when profiling
anantn Nov 15, 2026
90d07e2
Build and test the ESP-NN delegate on the host
anantn Nov 15, 2026
3a99bfa
Shrink the ESP-NN delegate's tables, load-time stack and scratch
anantn Nov 15, 2026
0177939
Use XTS3 tables and requantization parameters in place
anantn Nov 15, 2026
f7e0a3d
Use ExecuTorch tensors in place for ESP-NN delegate inputs and outputs
anantn Nov 15, 2026
a279668
Let the ESP-NN delegate own quantized mutable buffers
anantn Nov 16, 2026
dfbccc9
Document and show the streaming export recipe for the ESP-NN delegate
anantn Nov 15, 2026
b14df9a
Fix ESP-NN arena reuse for boundary conversions
rascani Nov 17, 2026
7ddab8c
Clarify ESP-NN state reset documentation
rascani Nov 17, 2026
f13e923
Apply clang-format to the ESP-NN host runner
anantn Nov 17, 2026
a3e9931
Always copy ESP-NN delegate inputs
anantn Nov 17, 2026
7e0e414
Fall back to portable kernels where esp-nn S3 leaves outputs unwritten
anantn Nov 17, 2026
6a807c5
Keep ESP-NN delegate outputs that operators read in the arena
anantn Nov 17, 2026
2776db9
Build the ESP-NN host tests against the device's esp-nn
anantn Nov 17, 2026
807df86
Read the cycle counter for the ESP-NN delegate's cooperative yield
anantn Nov 17, 2026
ccd20e6
Cover PSRAM and ESP-NN scratch in the ESP-NN host tests
anantn Nov 17, 2026
c4006aa
Give the ESP-NN delegate's arena the first claim on internal RAM
anantn Nov 17, 2026
d6b2edc
Allocate the ESP-NN delegate's staging buffer at init
anantn Nov 17, 2026
2439c2c
Reject ESP-NN delegate inputs and outputs without data
anantn Nov 17, 2026
7d44dd0
Require 16-byte aligned buffers in Int8Interpreter::bind_output
anantn Nov 17, 2026
a337dfd
Check ESP-NN kernel extents when exporting
anantn Nov 17, 2026
3eb5858
Give a method's ESP-NN state to one delegate or none
anantn Nov 17, 2026
7b12323
Return a copy of an ESP-NN state that the partition also returns
anantn Nov 17, 2026
f20454a
Reset ESP-NN delegate state by group
anantn Nov 17, 2026
dc0504f
Document ESP-NN program compatibility and buffer alignment
anantn Nov 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions .ci/scripts/test_espressif_host.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
#!/usr/bin/env bash
# Copyright (c) Meta Platforms, Inc. and affiliates.
# All rights reserved.
#
# This source code is licensed under the BSD-style license found in the
# LICENSE file in the root directory of this source tree.

# Builds the ESP-NN delegate for the host and checks fully delegated programs
# against the bit-exact Python reference: plain, and with ASan/UBSan over
# esp-nn's portable kernels, over its optimized C kernels, and with the
# delegate's non-default options. The sanitized builds make the kernels use
# scratch, in a separate buffer or in the arena. Needs no ESP-IDF toolchain or
# emulator.

set -euo pipefail

et_root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
build_dir="${ESPRESSIF_HOST_BUILD_DIR:-${et_root}/cmake-out-espnn-host}"
cd "${et_root}"

run_variant() {
local variant=$1
shift
cmake -S backends/espressif/test/host -B "${build_dir}/${variant}" -G Ninja \
-DCMAKE_BUILD_TYPE=RelWithDebInfo \
-DPYTHON_EXECUTABLE="$(command -v python)" "$@"
cmake --build "${build_dir}/${variant}"
ESPNN_HOST_RUNNER="${build_dir}/${variant}/espnn_host_runner" \
ESPNN_HOST_INTERPRETER="${build_dir}/${variant}/espnn_host_interpreter" \
python -m pytest -v --tb=short backends/espressif/test/test_host_runtime.py \
backends/espressif/test/test_host_interpreter.py
}

run_variant plain
scratch=(-DESPNN_HOST_SANITIZE=ON -DESPNN_HOST_SCRATCH=ON)
run_variant sanitized "${scratch[@]}"
run_variant sanitized-optimized "${scratch[@]}" -DESPNN_HOST_OPTIMIZED=ON
run_variant sanitized-options "${scratch[@]}" -DESPNN_HOST_OPTIMIZED=ON \
-DESPNN_HOST_PSRAM=ON -DESPNN_DELEGATE_CONSTANTS_TO_PSRAM=ON \
-DESPNN_DELEGATE_PROFILE=ON -DESPNN_DELEGATE_SCRATCH_IN_ARENA=ON
22 changes: 22 additions & 0 deletions .github/workflows/test-espressif.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ on:
- tools/cmake/preset/esp_baremetal.cmake
- CMakePresets.json
- .ci/scripts/test_espressif.sh
- .ci/scripts/test_espressif_host.sh
- .ci/scripts/test_espressif_idf_component.sh
- .ci/scripts/test_espressif_idf_consumer.sh
- .ci/scripts/esp_idf_mirror.py
Expand Down Expand Up @@ -75,6 +76,27 @@ jobs:
install_executorch --use-pt-pinned-commit --minimal
ESPRESSIF_NN_KERNELS="${{ matrix.kernels }}" bash .ci/scripts/test_espressif.sh

test-host:
name: ESP-NN delegate on the host
needs: docker-image
uses: pytorch/test-infra/.github/workflows/linux_job_v3.yml@main
permissions:
id-token: write
contents: read
with:
runner: mt-l-x86iavx512-8-64
docker-image: ${{ needs.docker-image.outputs.docker-registry }}/ci-image:executorch-ubuntu-22.04-clang12-${{ needs.docker-image.outputs.ci-docker-hash }}
submodules: recursive
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: 45
script: |
set -euo pipefail
CONDA_ENV=$(conda env list --json | jq -r '.envs | .[-1]')
conda activate "${CONDA_ENV}"
source .ci/scripts/utils.sh
install_executorch --use-pt-pinned-commit --minimal
bash .ci/scripts/test_espressif_host.sh

test-idf-component:
name: ESP-IDF component
needs: docker-image
Expand Down
219 changes: 212 additions & 7 deletions backends/espressif/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ backends/espressif/
test/ pytest suite (lowering, operators, CPU fallback, parser)
examples/espressif/
quantize_and_lower.py minimal quantize -> lower -> .pte example
export_streaming_model.py streaming model with delegate-owned state
export_espnn_test_programs.py the test models as BundleIO programs for the runner
camera_stream/ live camera classification on the ExecuTorch runtime
```
Expand Down Expand Up @@ -100,6 +101,170 @@ wrote mobilenet_v2_espnn.pte

Calibrate on a few hundred real samples; random data gives useless scales.

### Inputs and outputs

The delegate keeps rank-4 tensors in NHWC order. Inputs are always copied
into slots in the delegate's activation arena, quantized or transposed if
needed, because ESP-NN kernels may load whole vectors past the end of a
tensor they read. For the same reason, an output that an operator reads keeps
its arena slot and is copied out at the end. The delegate writes any other
int8 output directly when its buffer is 16-byte aligned and already in this
layout: the tensor is not rank 4, has one channel or one pixel, or has a
channels-last dim order. Each output is handled independently, whether its
buffer is memory-planned by ExecuTorch or supplied by the caller.

Float outputs and NCHW outputs with several channels and pixels need dtype
or layout conversion. The exporter records which outputs need conversion,
so their int8 buffers can share the arena with intermediate tensors. Older
exports without this metadata use the arena for all outputs. An output
outside the arena whose buffer turns out misaligned is written to a staging
buffer and copied out. The delegate allocates that buffer at `init()`, sized
for every such output, in PSRAM when there is any.

For the smallest and fastest program, give the delegate int8, channels-last
inputs and outputs and leave them out of ExecuTorch's memory plan, so that
the application's own buffers are used without conversion:

```python
from executorch.exir import ExecutorchBackendConfig
from executorch.exir.passes import MemoryPlanningPass
from executorch.exir.passes.quantize_io_pass import quantize_input, quantize_output

# Example inputs (and calibration data) in channels-last memory format. An
# output that PyTorch would make contiguous, such as a slice of a cat, needs
# .clone(memory_format=torch.channels_last) at the end of forward().
example_inputs = (torch.randn(1, 3, 96, 96).to(memory_format=torch.channels_last),)
# ... quantize and calibrate as above ...
program = torch.export.export(quantized, example_inputs)
quantize_input(program, 0)
quantize_output(program, 0)
edge, report = to_espnn_edge(program)
config = ExecutorchBackendConfig(
memory_planning_pass=MemoryPlanningPass(
alloc_graph_input=False, alloc_graph_output=False
)
)
pte = edge.to_executorch(config).buffer
```

The application then binds its buffers once, with `Method::set_input()` and
`Method::set_output_data_ptr()`, and each step is a call to
`Method::execute()`. Declare them `alignas(16)`. An output may share memory
with an input, for example to update a state tensor in place, since the
inputs are copied in before any operator runs.

### Streaming state

A streaming model keeps a few frames of history between calls in a mutable
buffer. By default the buffer stays in the program, which passes it to the
delegate and back on every call. With `delegate_mutable_buffers=True`, the
delegate that reads and updates a buffer owns it instead: the program loses
that input and output, and the delegate keeps the buffer as int8 in its arena,
written in place when no operator reads the old value after the new one is
produced.

```python
class Streaming(torch.nn.Module):
def __init__(self):
super().__init__()
self.register_buffer("frames", torch.zeros(1, 40, 2, 1))
self.conv = torch.nn.Conv2d(40, 32, (3, 1))

def forward(self, x): # x: (1, 40, 1, 1), one new frame
window = torch.cat([self.frames, x], dim=2)
self.frames.copy_(window[:, :, 1:])
return self.conv(window)

# ... prepare_pt2e, calibrate, convert_pt2e as above, then reset the buffer
# that calibration updated before exporting.
edge, report = to_espnn_edge(
torch.export.export(quantized, (torch.zeros(1, 40, 1, 1),)),
delegate_mutable_buffers=True,
)
print(report["delegated_buffers"]) # ['frames']
```

The delegate takes a buffer only when one partition holds every access to it
and the buffer is read and written with the same int8 quantization, as a
`cat` and slice shift register is with `EspNNQuantizer`. It takes a method's
buffers only when they all end up in one partition that keeps another input
and output, and at most 16 of them. Other buffers stay in the program, with a
warning.

On the device, each loaded method has its own state. It starts from the
buffer's initial value when the method is loaded and persists across
`execute()` calls. `reset_delegate_states()` (from `espnn/espnn_backend.h`),
or the `reset_states` backend option set to `true`, restores the initial
state of every EspNNBackend delegate before its next `execute()`. To reset
some methods alone, load them with a `reset_group` from 0 to 7 and pass that
group to `reset_states`, or load the method again. Buffers that stay in the
program are not reset this way. After an `execute()` that fails, the state is
undefined until it is reset. The reset may be requested from any task.

```cpp
executorch::backends::espressif::reset_delegate_states();
// or, through the generic backend-option API:
executorch::runtime::BackendOptions<1> options;
options.set_option("reset_states", true);
executorch::runtime::set_option("EspNNBackend", options.view());

// A method in reset group 1, reset alone:
executorch::runtime::BackendOptions<1> group;
group.set_option("reset_group", 1);
executorch::runtime::LoadBackendOptionsMap load_options;
load_options.set_options("EspNNBackend", group.view());
auto method = program->load_method(
"forward", &memory_manager, nullptr, nullptr, &load_options);
options.set_option("reset_states", 1);
executorch::runtime::set_option("EspNNBackend", options.view());
```

### Streaming keyword spotting

[`examples/espressif/export_streaming_model.py`](../../examples/espressif/export_streaming_model.py)
puts the pieces above together for a streaming keyword-spotting model: one
frame of features in, one score out, the frame history in delegate-owned
buffers, int8 input and output in the application's memory. The resulting
program has one input and one output, no CPU operators, and needs no
planned memory.

Quantization choices that matter for such models:

- `EspNNQuantizer` observes activations with histograms. For small streaming
models, min/max ranges (`minmax_quantizer()` in the example) often track a
float model, or a TensorFlow Lite reference, more closely.
- To reproduce an existing int8 model exactly, give each activation the
reference model's scale and zero point: after `prepare_pt2e`, replace its
observers with `FixedQParamsObserver(scale, zero_point, ...)`.
- Weight scales are `max|w| / 127` per output channel; check that a layer with
very small weights is not clamped by the weight observer's `eps`.
- Calibrate on consecutive frames, so the histories see realistic values, and
zero the buffers before exporting, since calibration updated them.

On the device, the application binds its buffers once and calls `execute()`
for each frame:

```cpp
#include <espnn/espnn_backend.h>
#include <executorch/runtime/executor/method.h>

// After Program::load() and load_method() with a method allocator sized by
// method_meta(), and no planned buffers (the program has none). The buffers
// are 16-byte aligned so that the delegate writes the score in place.
alignas(16) int8_t features[40];
alignas(16) int8_t score;
method.set_input(EValue(Tensor(&features_impl)), 0); // int8[1, 40, 1, 1]
method.set_output_data_ptr(&score, sizeof(score), 0); // int8[1, 1]
for (;;) {
compute_features(features); // writes the bound input buffer
method.execute();
if (score > threshold) {
on_detection();
executorch::backends::espressif::reset_delegate_states();
}
}
```

## Running on the ESP32-S3

You need ESP-IDF and an activated ESP-IDF shell.
Expand Down Expand Up @@ -174,7 +339,8 @@ Other options, in menuconfig under *ExecuTorch ESP-NN backend*:
|---|---|---|
| `ESPNN_DELEGATE_CONSTANTS_TO_PSRAM` | y | Copies weights from memory-mapped flash into PSRAM when they fit (faster than flash cache misses on large models) |
| `ESPNN_DELEGATE_DUAL_CORE` | n | Splits large convolutions across both cores |
| `ESPNN_DELEGATE_PROFILE` | n | Prints a per-operator time table after a delegate's first run |
| `ESPNN_DELEGATE_SCRATCH_IN_ARENA` | n | Plans each operator's ESP-NN scratch in the activation arena instead of a separate buffer sized for the largest operator. Saves internal RAM when the arena fits there; for a large model it moves the scratch to PSRAM with the arena |
| `ESPNN_DELEGATE_PROFILE` | n | Times every operator and prints a per-operator time table after a delegate's first run. The timer reads cost about 1 µs per operator on the ESP32-S3, so leave it off in production |

### The original ESP32

Expand Down Expand Up @@ -206,13 +372,50 @@ ANSI C reference kernels against the same bundled outputs. Both jobs target
ESP32-S3. Logs include the kernel configuration in their filenames; artifacts
retain the bundled fixtures, SDK configuration, and resolved component versions.

### On the host

The delegate also builds outside ESP-IDF, for Linux and macOS, over ESP-NN's
portable C kernels. `host/include` stands in for the few ESP-IDF APIs it uses,
and ESP-NN is fetched from GitHub (or taken from `-DESPNN_LOCAL_PATH=...`).
`test_host_runtime.py` runs programs through the ExecuTorch runtime, this
build and the portable kernels, and checks every output against the host
reference (bit-exactly for fully delegated programs), with no board or
emulator:

```bash
cmake -S backends/espressif/test/host -B cmake-out-espnn-host -G Ninja
cmake --build cmake-out-espnn-host
ESPNN_HOST_RUNNER=cmake-out-espnn-host/espnn_host_runner \
pytest backends/espressif/test/test_host_runtime.py
```

`-DESPNN_HOST_SANITIZE=ON` builds with AddressSanitizer and
UndefinedBehaviorSanitizer, `-DESPNN_HOST_OPTIMIZED=ON` uses ESP-NN's optimized
C kernels instead of the reference ones, `-DESPNN_HOST_PSRAM=ON` makes the
heap stand-in serve PSRAM allocations, and `-DESPNN_HOST_SCRATCH=ON` makes
the convolution and mean kernels request scratch and overwrite all of it, as
a check on where the delegate puts it. `ESPNN_DELEGATE_CONSTANTS_TO_PSRAM`,
`ESPNN_DELEGATE_PROFILE` and `ESPNN_DELEGATE_SCRATCH_IN_ARENA` are CMake
options here, off by default. `.ci/scripts/test_espressif_host.sh`
runs a plain build and three sanitized ones that cover these options. The host
build checks the delegate's logic, not ESP-NN's S3 assembly kernels, which only
the emulator and the board run.

The code follows the ExecuTorch coding style and passes the repository's
`lintrunner` checks:

```bash
lintrunner --paths-cmd "git ls-files backends/espressif examples/espressif"
```

## Program compatibility

A program runs on a delegate runtime from the same ExecuTorch version as the
exporter or a later one. Programs exported now carry XTS3 version 7 graphs,
or version 8 for delegates that own state, which older runtimes reject at
`init()` with `DelegateInvalidCompatibility`. The runtime still runs
programs from older exporters (XTS3 versions 5 and 6).

## Limitations

- Only int8 is delegated. Float32 operators run on ExecuTorch's portable
Expand All @@ -223,15 +426,17 @@ lintrunner --paths-cmd "git ls-files backends/espressif examples/espressif"
norm fall back to the CPU.
- Dual-core execution only helps large convolutions, and it is off by default.
- The delegate handle comes from ExecuTorch's runtime allocator, but each
delegate allocates its activation arena, scratch buffers and optional PSRAM
weight copy with ESP-IDF's `heap_caps` allocator at `init()`, so it can
choose between internal RAM and PSRAM. Nothing is allocated during
`execute()`.
delegate allocates its activation arena, scratch and staging buffers and
optional PSRAM weight copy with ESP-IDF's `heap_caps` allocator at
`init()`, so it can choose between internal RAM and PSRAM. Nothing is
allocated during `execute()`, except that with `ESPNN_DELEGATE_DUAL_CORE`
ESP-NN's worker task grows its own scratch buffer the first time a large
convolution needs it.

## License

Every source file in this backend and its examples is released under the
BSD 3-Clause license (`SPDX-License-Identifier: BSD-3-Clause`), the same
license as ExecuTorch. See the `LICENSE` file at the repository root. ESP-NN
is Apache-2.0 and is fetched from the Espressif component registry, not
vendored.
is Apache-2.0 and is fetched from the Espressif component registry (or, for
the host build, from GitHub), not vendored.
Loading
Loading