Skip to content

[ESP-NN] Streaming int8 models: host tests, in-place tables and I/O, delegate-owned state - #25106

Open
anantn wants to merge 14 commits into
pytorch:mainfrom
anantn:espnn-delegate-int8-state
Open

anantn wants to merge 14 commits into
pytorch:mainfrom
anantn:espnn-delegate-int8-state

Conversation

@anantn

@anantn anantn commented Nov 16, 2026 •

Copy link
Copy Markdown

Summary

This makes the ESP-NN delegate a good fit for small streaming models on ESP32-S3, like keyword spotting. For such a model, most of each step currently goes to the delegate's bookkeeping rather than the kernels: it times every operator, copies its tables into RAM, stages every input and output through its arena, and the streaming state goes in and out of the delegate on every call.

On a 32-operator keyword-spotting model, this brings a step from 594 µs to 444 µs and internal RAM from 48.6 KB to 32.2 KB, with bit-identical output. That puts ExecuTorch ahead of TFLite Micro running the same model on the same esp-nn kernels.

ESP32-S3, 240 MHz µs per step internal RAM
TFLite Micro + esp-nn 613 32.5 KB
ExecuTorch main 594 48.6 KB
This PR 444 32.2 KB

Models without streaming state don't gain much. On the backend's emulator test programs the runtime uses 0.2–4.7 KB less heap and runs 0.6–2% slower; the host build executes the same number of instructions, so it looks like code layout.

Review order

  1. Read per-operator timers only when profiling is enabled (about 55 µs per step).
  2. Build and test the delegate on the host with esp-nn's portable kernels, with a CI job running plain and ASan/UBSan builds. The later commits are checked bit-exactly against ProgramEmulator here.
  3. Shrink the delegate's per-operator state and load-time stack, with an option to plan scratch in the arena.
  4. Read tensor tables and requantization parameters in place from the program (XTS3 v7).
  5. Use ExecuTorch tensors directly for int8 channels-last inputs and outputs. Everything else is staged as before; the aliasing rules are in the README.
  6. Opt-in delegate_mutable_buffers=True: the delegate owns int8 mutable buffers and updates them in place. State is reset through the reset_states backend option (set_option) or reset_delegate_states().
  7. A streaming export example and README guidance.

Compatibility

Anything that could change an existing export is opt-in, and the runtime still loads v5 and v6 programs. Programs exported after this PR use XTS3 v7 (v8 with delegate-owned state) and need the new runtime. The loader now rejects pooling and convolution outputs larger than 65535, which esp-nn truncates to uint16_t and never handled correctly.

Test plan

pytest -n 16 backends/espressif/test
.ci/scripts/test_espressif_host.sh
ESPRESSIF_NN_KERNELS=optimized bash .ci/scripts/test_espressif.sh
ESPRESSIF_NN_KERNELS=ansi bash .ci/scripts/test_espressif.sh

Every commit passes the Python suite and the host tests. The head also passes both emulator jobs and lintrunner. The device numbers are from an ESP32-S3 board over 330 test clips, measured before a few review fixes that aren't on this model's path.

I kept this as one PR, but I'm happy to split it into a stack if that's easier to review.

This PR was prepared with AI assistance (Claude).

Int8Interpreter::invoke() read esp_timer_get_time() before and after every
operator to fill a per-operator profile, whether or not anything printed it.
On ESP32-S3 that is two function calls per operator; for a 32-operator
streaming keyword-spotting model it was about 55 us of a 540 us call, more
than many of the operators themselves.

The interpreter now times operators only with
Int8RuntimeOptions::profile_operations, which EspNNBackend sets from
CONFIG_ESPNN_DELEGATE_PROFILE, and keeps the times in a table allocated only
then. Otherwise it reads the timer at the start and end of each call and
every 16 operators, for its cooperative yield. last_invoke_us() becomes the
wall time of the call.
The ESP-NN delegate only built inside ESP-IDF, so checking a runtime change
needed the ESP32-S3 emulator or a board. Its sources use few ESP-IDF APIs: the
heap_caps allocator, esp_timer, a FreeRTOS mutex and a task yield. Outside
ESP-IDF, espnn_runtime/CMakeLists.txt now builds the same sources as the
espnn_backend library, with host/include standing in for those APIs and
ESP-NN's portable C kernels fetched from GitHub at the v1.5.0 commit (or taken
from ESPNN_LOCAL_PATH).

backends/espressif/test/host builds it with the ExecuTorch runtime into
espnn_host_runner, and test_host_runtime.py runs every fully delegated test
model through it, over several steps and several loaded instances of a method,
with planned and with caller-owned inputs and outputs, checking each output
bit-exactly against ProgramEmulator. .ci/scripts/test_espressif_host.sh runs
the test on a plain build and on an AddressSanitizer and
UndefinedBehaviorSanitizer build, in a new job of the Espressif workflow that
needs no ESP-IDF toolchain.

The host build exercises the delegate's planner, interpreter and ExecuTorch
glue, not ESP-NN's S3 assembly kernels, which the emulator jobs still cover.
On a 512 KB-SRAM part the delegate's own bookkeeping can cost more internal
RAM than a small model's activations. For a 32-operator streaming
keyword-spotting model on ESP32-S3 it kept 10.5 KB of per-operator state,
1.4 KB of operation records that are only needed while loading, and a
separate 5.8 KB ESP-NN scratch buffer, and graph validation put four
1,024-byte bool arrays on the loading task's stack.

Fields of the per-operator state that no operator uses together now share
storage, operation records are decoded while loading instead of being kept,
the planner frees its tables before allocating the arena, and graph
validation uses bitsets. The invoke mutex is created by a static initializer
instead of a function-local static, so invoke() no longer goes through a
thread-safe static guard. ESPNN_DELEGATE_SCRATCH_IN_ARENA (off by default)
plans each operator's ESP-NN scratch as an arena entry that lives only while
the operator runs; for a model whose arena stays in internal RAM this
replaces the separate buffer.

DelegateStats now also reports the scratch, requantization and table bytes.
The host test checks the delegate's arena against plan_activation_arena(),
its Python mirror, and the sanitized host build runs with scratch in the
arena and per-operator profiling on.
The delegate copied every tensor record and every per-channel requantization
multiplier and shift out of its graph into RAM, although the graph already
sits in memory-mapped flash or in PSRAM. For a 32-operator streaming
keyword-spotting model on ESP32-S3 the two copies took 4.4 KB of internal RAM.
They were needed because XTS3 v6 places those tables at offsets that are not
word-aligned: the 22-byte header and the input/output table leave the
tensor records, and so everything after them, 2 bytes off a word boundary.

XTS3 v7 is v6 with the input/output table padded to a 4-byte boundary and each
operation's parameter block starting on one. The serializer now writes v7. On a
little-endian target the runtime uses a v7 graph's tensor records and
requantization arrays (and softmax tables) where they are, and still copies
them for v5 and v6 graphs, which it keeps accepting. When the delegate copies
the graph's constants into PSRAM, the in-place pointers follow the copy.

A program exported with this version needs a runtime from the same version or
later; the runtime rejects unknown graph versions at init with
DelegateInvalidCompatibility.

espnn_host_interpreter runs a graph on the host build of the interpreter from
addresses of every alignment, and test_host_interpreter.py checks that v6 and
v7 graphs give bit-exact results from all of them and that v7 graphs keep no
copy of their requantization parameters.
The delegate copied every input into its activation arena before running and
every output out of it afterwards, transposing rank-4 tensors between NCHW
and its NHWC device layout element by element. For a streaming
keyword-spotting model on ESP32-S3 whose state tensors cross the delegate
boundary on every step, that was both the largest share of the delegate's
non-kernel time and the reason the arena held a copy of every input and
output.

With Int8RuntimeOptions::external_io, which EspNNBackend now sets, the planner
leaves graph inputs and outputs out of the arena and the caller binds a buffer
to each before invoke(). The backend binds the ExecuTorch tensor itself when
its memory is 16-byte aligned, like the arena, and already in the device
layout: an int8 tensor that is not rank 4, has one channel or one pixel, or
has a channels-last dim order. Any other input or output (float, NCHW or
misaligned) goes through a zero-initialized staging buffer that the first
execute() needing one allocates, which keeps existing programs working
unchanged. Because operators may read an input after writing an output, an
output whose memory overlaps an input is always staged and copied out after
the last operator; this makes binding one buffer as both an input and an
output (an application updating a state tensor in place) safe instead of
depending on the order of the delegate's copies.

A graph tensor that the delegate returns as several outputs (when its value
feeds several CPU operators) is written once and copied to the other outputs,
and a graph input that is also returned as an output is staged and copied from
there. The first COPY writing an output bound to caller memory, unless it
fills it, clears it first, so the buffer's previous contents never show
through; a long cat is written by several COPY operations. Pooling and
convolution outputs wider or taller than 65535, which ESP-NN's kernels would
truncate to uint16_t and so leave unwritten, are now rejected at load. A graph
whose every activation is an input or output needs no arena at all. The
interpreter keeps a pointer per tensor instead of an arena offset, so binding
costs nothing per operator. plan_activation_arena() mirrors the external-I/O
plan, the README describes how to export int8 channels-last I/O that is used
in place, and the tests cover channels-last I/O without staging, misaligned
caller buffers, an output bound to the memory of an input that operators read
after the output is written, an input returned as an output (also with another
output bound over it), a partial COPY into a bound output, a seven-way cat as
an output, those limits, a graph with no arena, and, with the portable and
quantized kernels now linked into the host runner, the programs that leave
operators on the CPU between delegates.
A streaming model keeps a few frames of history in mutable buffers. Lowered
to the ESP-NN delegate today, each buffer stays in the program: it is passed
into the delegate and its new value out of it on every call, in float, or in
int8 when the application exports it as a stateless input and output and
carries the state itself. For a 32-filter streaming keyword-spotting model on
ESP32-S3 with six state tensors, those copies, their layout conversions and
the extra delegate arguments took about a fifth of each call.

EspNNPartitioner(delegate_mutable_buffers=True), also exposed by
to_espnn_edge(), gives a mutable buffer to the delegate that holds every
access to it, through tag_mutated_buffer(), when the buffer is read and
written with the same int8 quantization: a float32 buffer read through
quantize ops (what PT2E produces for a cat and slice shift register) or an
int8 buffer read through dequantize ops. Other buffers stay in the program, as
they do by default. The partition's graph is then XTS3 version 8, which adds a
table of (state, update) tensor pairs; each state's quantized initial value is
stored with the constants. An update that the runtime cannot write into a
state's slot (a graph input or output, another state, or an update shared by
several states) gets a copy of its own, made after the last operator.

The runtime keeps each state in a reserved arena slot and initializes it at
load. The update is planned into the state's own slot when no operator reads
the state at or after the update's first write, and copied over the state
after the last operator otherwise. Every loaded method instance has its own
state. reset_delegate_states(), or the reset_states backend option through
executorch::runtime::set_option(), restores the initial state of every
EspNNBackend delegate before its next execute(); reloading a method resets
it alone.

ProgramEmulator and the reference interpreter keep the same state across calls
and can reset it, and plan_activation_arena() mirrors the in-place update. New
tests cover which buffers the delegate takes, the reference against the
quantized PyTorch model across steps and a reset, and, on the host build,
states updated in place and by copy, a state updated from a graph input, a
reset, two instances, and the parser's checks of the state table. The emulator
suite gains a stateful program.
The pieces a streaming keyword-spotting model on ESP32-S3 needs (int8,
channels-last or trivially laid out inputs and outputs left to the
application's buffers, mutable buffers owned by the delegate, and a
quantization that matches the model's reference) were each documented on
their own. export_streaming_model.py puts them together on a small model whose
program has one int8 input, one int8 output, no CPU operators and no planned
memory, and the README adds the quantization choices that matter for such
models and the device-side pattern: bind once, execute per frame, reset the
delegate's state on a detection. The host test runs the example's program for
several frames without any staging copy.
@anantn
anantn requested a review from larryliu0820 as a code owner November 16, 2026 02:56
@pytorch-bot

pytorch-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25106

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit a16cf92 with merge base 436abf7 (image):

NEW FAILURE - The following job has failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 16, 2026
@linux-foundation-easycla

linux-foundation-easycla Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.

@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@JakeStevens

Copy link
Copy Markdown
Contributor

@claude review effort=max

@claude

claude Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Claude finished @JakeStevens's task in 52m 19s —— View job


PR Review: #25106 (effort=max)

  • Gather context: diff against merge base 62e28de, 7 commits, 32 files
  • Independent review panel (3 reviewers)
  • Consolidate findings
  • Fact-check every finding against the code
  • Post review

Request Changes: external I/O is now always on, so every default float/NCHW program stages its I/O in a buffer allocated at the first execute() without honoring internal_reserve_bytes; the new load order can push the arena into PSRAM; and a reset from another task can leave a multi-partition method's states one execution apart.

Detailed review

Paths are under backends/espressif/, except those starting with .ci/ or runtime/.

Summary

This PR adds a host build of the ESP-NN delegate with ASan/UBSan CI. It also shrinks the delegate's tables, scratch and load-time stack, uses XTS3 v7 tables in place, binds int8 graph I/O directly to ExecuTorch tensors, and lets the delegate own quantized mutable buffers with a process-wide reset.

The blocking problems:

  • The memory behavior of every default (float/NCHW I/O) program changes.
  • The new load order can demote the arena to PSRAM.
  • A reset can race between state-owning partitions.

Two test gaps: the PSRAM constant-copy path never runs, and the default scratch configuration never runs under the sanitizers.

Code Quality

  • More than 16 eligible buffers in one partition fails the export (partition/espnn_partitioner.py:306-332).
    • _tag_quantized_mutable_buffers() delegates every compatible buffer, but check_runtime_limits() rejects more than MAX_GRAPH_STATES (16) per partition (serialization/xts3.py:2028-2032).
    • A streaming model with a history buffer in each of 17 or more layers therefore raises ValueError in to_backend.
    • This contradicts the class docstring, which says "Any other buffer stays in the program" (:351). Fix this →
  • check_runtime_limits() misses the new extent limits (serialization/xts3.py:2014-2017).
    • Its docstring promises "the same limits espnn_runtime/src/xts3_graph.cpp enforces".
    • This PR adds output H/W > 65535 rejections for convolution and pooling (espnn_runtime/src/xts3_graph.cpp:158-160, :412-413).
    • Only the upsample extent is mirrored (xts3.py:2050-2058). The input and fully-connected extent limits (:500-501) are missing as well.
    • Such graphs export cleanly and then fail device init() with DelegateInvalidCompatibility. An example is a padded window that widens 65535 to 65536, which test/test_xts3_parser_host.py:193-203 builds by hand. Fix this →
  • A missing output buffer becomes a bare Error::Internal (espnn_runtime/src/espnn_backend.cpp:353, :398).
    • arena_aligned(nullptr) is true, so a tensor without data is bound in place.
    • One way to get there: an output left out of the memory plan that never got set_output_data_ptr(). Method::execute() checks only inputs (runtime/executor/method.cpp:1963-1969).
    • invoke() then returns false without a message (espnn_runtime/src/int8_interpreter.cpp:1921-1930), and execute() returns Error::Internal (espnn_backend.cpp:425-426). Operator failures, by contrast, print the operator. Fix this →
  • The cooperative-yield bound now depends on 16 operator durations (espnn_runtime/src/int8_interpreter.cpp:1955-1964; low confidence on real-world impact).
    • Without profiling, the clock is read only after every 16th operator. The longest run without vTaskDelay() grows from 1 s plus one operator to 1 s plus 16.
    • A graph whose convolutions read weights from flash or PSRAM moves closer to the 5 s task-watchdog default that the yield exists to avoid.
    • The saving is one esp_timer_get_time() per operator. Fix this →

Infrastructure

  • Host CI tests a different esp-nn version than devices build with (espnn_runtime/host/espnn_host.cmake:15-18).
    • The host build fetches esp-nn v1.5.0.
    • Device builds resolve espressif/esp-nn: "1.4.1" (idf_component.yml:11); espnn_runtime/idf_component.yml:13 allows ^1.4.0.
    • So the bit-exact host tests validate kernels, and possibly APIs, that firmware built from this tree does not use. Fix this →

Testing

  • The PSRAM constant-copy path never runs on the host (espnn_runtime/src/int8_interpreter.cpp:385-409).
    • The host heap stub has no SPIRAM (espnn_runtime/host/include/esp_heap_caps.h), and espnn_host.cmake:78-82 does not expose ESPNN_DELEGATE_CONSTANTS_TO_PSRAM.
    • So copy_constants_to_psram, rebase_constants() and the new tensors_ re-point for in-place v7 tables never run in CI.
    • In these paths a missed pointer becomes a use-after-free once the backend frees the processed buffer (espnn_runtime/src/espnn_backend.cpp:293-294). That is exactly what the sanitized build is for. Fix this →
  • The default scratch configuration never runs under ASan/UBSan (.ci/scripts/test_espressif_host.sh:24-26). The sanitized variant also turns on SCRATCH_IN_ARENA and PROFILE.
    • The separately allocated, newly shrunk scratch buffer and the every-16-operators path are tested only without sanitizers.
    • In the in-arena variant, a scratch overrun stays inside the arena allocation, so ASan cannot see it.
    • Only the *_ansi.c kernels are built (espnn_host.cmake:44). I believe esp-nn's ANSI getters report zero conv/depthwise scratch; if so, neither placement is ever exercised with a non-zero size, and ESPNN_HOST_OPTIMIZED=ON may be needed. Low confidence: esp-nn's sources are not in this checkout. Fix this →

API Design

  • The in-place I/O alignment contract is neither documented nor checked (espnn_runtime/include/espnn/int8_interpreter.h:131-145, espnn_runtime/src/int8_interpreter.cpp:988-1001).
    • bind_input() and bind_output() are documented as taking any device-layout buffer of bytes, and they accept any pointer.
    • The backend, however, binds only 16-byte-aligned buffers, because "the kernels may rely on the activation arena's 16-byte alignment" (espnn_runtime/src/espnn_backend.cpp:111-115).
    • If the requirement is real, a direct Int8Interpreter user with external_io can get wrong results or a fault from a misaligned buffer. If it is not, the backend stages misaligned int8 tensors for nothing.
    • Settle the requirement and enforce it in one place. Fix this →
  • State reset is process-wide only (espnn_runtime/include/espnn/espnn_backend.h:75-86, README.md:183-189). Needs discussion.
    • reset_delegate_states() and the reset_states option reset every EspNNBackend delegate in the process.
    • The only per-method reset is reloading the method, which reallocates its arena, tables and staging buffer.
    • So two independent streams cannot be reset separately, for example two keyword spotters, or a stateful model running next to another.
    • Program::load_method() already accepts per-method LoadBackendOptionsMap runtime specs. A reset group assigned at load time and taken by reset_states would fit the existing API.

Thread Safety

  • A reset from another task can leave a method's partitions one execution apart (espnn_runtime/src/espnn_backend.cpp:324-331).
    • Each delegate compares the global generation at the start of its own execute().
    • A method can have two state-owning partitions under the default allow_cpu_fallback=True, because README.md:178 only requires that each buffer's accesses sit in one partition.
    • If reset_delegate_states() runs on another task between the two delegates' execute() calls, the first partition finishes on its old state and the second restarts.
    • On the next execution the first partition resets and the second does not. They stay one execution apart until a reset lands between executions.
    • This contradicts espnn_backend.h:82-84 ("that execution finishes with the old state and the next one starts from the initial state") and README.md:189 ("The reset may be requested from any task").
    • It is also untested: test/test_mutable_buffers.py always uses allow_cpu_fallback=False. Fix this →

Backward Compatibility

  • Every default float/NCHW program now stages its I/O (espnn_runtime/src/espnn_backend.cpp:273, :220-253). external_io is now on unconditionally, so graph inputs and outputs leave the lifetime-planned arena. Float or NCHW tensors, which the default export produces, go through the staging buffer instead.
    • staging_slot() sizes one buffer for every input and output slot, including slots bound in place.
    • Before this PR these tensors shared arena space with intermediates by lifetime, so total RAM can grow by up to the full I/O size. For example, an image classifier's input region used to be reused by later activations.
    • The buffer is allocated by the first execute(), internal RAM first, with no internal_reserve_bytes check (:238-243). The interpreter's own buffers do honor the reserve.
    • So the buffer can take the internal RAM the reserve keeps for Wi-Fi and task stacks, or fail at run time with MemoryAllocationFailed. The previous README promised "Nothing is allocated during execute()". Fix this →
  • New programs need new firmware, and that is not documented (serialization/xts3.py:26; minor).
    • Every new program is XTS3 v7, or v8 with delegate-owned state. Runtimes from before this PR reject it at init() with DelegateInvalidCompatibility.
    • Only a commit message says so.
    • There is no supported way to emit v6: only the test helper patches graph.version (test/test_host_runtime.py:113-117). Fix this →

Performance

  • The arena no longer gets first claim on internal RAM (espnn_runtime/src/int8_interpreter.cpp:379-380).
    • load() now runs prepare_operations() before plan_memory().
    • So these buffers take internal RAM through allocate_preferring_internal(…, true, internal_reserve_bytes) before the arena does (:584):
      • the separate scratch (:947, the default configuration);
      • the phase buffer (:936);
      • the v5/v6 requantization tables (:629).
    • The arena stays internal only if the largest internal block still exceeds its size plus the reserve. An arena that fit internally before can now land in PSRAM and slow every operator.
    • Only scratch_in_arena needs the scratch sizes before planning. Compute the sizes first, and allocate the separate buffers after the arena. Fix this →

Specific Comments

  • espnn_runtime/src/int8_interpreter.cpp:214-216: the union comment says pool[] shares storage only with the CONCAT/COPY input list. It also holds the convolution and transpose-convolution windows (:712-717, :840-841).
  • espnn_runtime/src/espnn_backend.cpp:459-461: a reset_states option with a non-bool value is reported as "unknown option reset_states". Say that the value must be a bool.
  • espnn_runtime/include/espnn/espnn_backend.h:38-47: "held by the initialized delegates" reads as current memory, but destroy() never subtracts. After the README's reload-to-reset, the sums still count destroyed delegates. Say "allocated by the delegates initialized since…".
  • README.md:230-231: the snippet needs alignas(16) on features and score, because a plain int8_t score is 1-byte aligned. Otherwise the delegate stages both, contrary to "int8 input and output in the application's memory" (:204).

Recommendation

Request Changes. Two problems block approval:

  • The staging and allocation-order changes alter memory behavior for programs that work today.
  • The cross-partition reset race contradicts the documented reset contract.

The missing PSRAM/rebase coverage and the unsanitized default configuration cover the riskiest new pointer handling, so add those tests in this PR too. The remaining items are minor.

@mergennachin
mergennachin requested a review from rascani November 16, 2026 16:35
Record per-boundary conversion requirements at export so float and NCHW I/O can reuse the activation arena while compatible int8 I/O remains directly bound. Preserve independent ExecuTorch input/output planning and arena-backed I/O for older exports.

Keep conversion metadata independent per partition. Host regression tests cover arena sizing, mixed boundaries, all four planning settings, and legacy exports.

Validation: 268 host tests passed, 11 skipped. Independent review and six additional native host probes passed.

Authored with Codex (OpenAI).
Remove the whole-method timing guarantee for concurrent resets. Each delegate independently applies a reset before its next execution.

Authored with Codex (OpenAI).
The ESP32-S3 esp-nn kernels may load whole vectors past the end of a tensor
they read; the general convolution reads up to 16 bytes past each row of its
input. An input bound in place could be the caller's memory ending at the edge
of mapped RAM.

Graph inputs now always get slots in the delegate's activation arena and are
copied into them, quantized or transposed if needed, and the arena gets 16
readable bytes after its last slot. The exporter records every input as a
conversion slot, but the runtime plans inputs into the arena regardless of the
recorded slots, so exports that left compatible inputs out of them still run.
Int8Interpreter::bind_input() is removed so that nothing can bind one in
place. Outputs keep their conversion slots and direct binding. Since inputs
are copied before any operator runs, an output may be bound over an input's
memory without staging, so the overlap check and the input staging slots go
away.
Two esp-nn ESP32-S3 kernels return without writing all of their output, so
the delegate produced whatever the output buffer held before. The max pool
kernel handles only a multiple of four channels and writes nothing otherwise,
and the pointwise convolution passes its number of output positions as a
uint16_t width, leaving every position past 65535 unwritten. Both cases now
use the ANSI kernels.

invoke() also clears an output bound to memory outside the arena once it holds
the invocation lock, so that a kernel leaving part of its output unwritten
cannot let the caller's previous contents show through. The emulator and host
suites gain a max pool over six channels.
An output that a later operator reads was still bound to the caller's buffer,
so the ESP32-S3 kernels' vector loads past the end of a tensor could reach
past that buffer, and an application that bound another output over it would
change what the operator read.

With external I/O, Int8Interpreter now gives every output that an operation
reads an arena slot, whatever the export recorded, and reports those slots
through conversion_slots(), so EspNNBackend copies such outputs out after the
last operator like any other output in the arena. bind_output() refuses any
output that has an arena slot. The host runner gains --share-output, and a
host test binds two outputs to one buffer while an operator still reads the
first.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants