Skip to content

Add Cadence w8a32 split GRU operator (#25195) - #25195

Open
mgiordy wants to merge 5 commits into
pytorch:mainfrom
mgiordy:export-D99592763
Open

mgiordy wants to merge 5 commits into
pytorch:mainfrom
mgiordy:export-D99592763

Conversation

@mgiordy

@mgiordy mgiordy commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary:

Adds cadence::quantized_w8a32_split_gru, one step of a split GRU with int8 weights and biases and fp32 activations. A split GRU runs two independent GRUs: split i reads every other input feature starting at feature i, and the i-th half of the hidden state.

cadence::split_gru is a float op for the whole split GRU, so the quantizer sees it as a single node. MixedW8A32SplitGruPattern in CadenceW8A32MixedQuantizer fuses it for a single frame (batch 1, seq 1) whose per-split input and hidden sizes are multiples of 4. Other shapes stay in fp32. The op schemas, fakes and reference implementations come with it; the quantized reference runs the float one on the dequantized weights.

Differential Revision: D99592763

Copilot AI balanced review requested due to automatic review settings November 18, 2026 00:32
@pytorch-bot

pytorch-bot Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25195

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 6e46298 with merge base ec5e3cf (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 18, 2026
@meta-codesync

meta-codesync Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

@mgiordy has exported this pull request. If you are a Meta employee, you can view the originating Diff in D99592763.

@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unsupported split-GRU shapes fail export instead of falling back to fp32, and normalization fusion can silently replace nondefault epsilon with 1e-5, changing results.

2 open issues
What changed in this PR

Adds a two-way split GRU to Cadence’s mixed-precision quantization pipeline, using int8 weights and biases with fp32 activations.

Changes:

  • Adds split-GRU schemas, fake implementations, references, and shape-gated quantization.
  • Extends W8A32 patterns to convolution, transposed convolution, and layer normalization, using the symmetric [-127, 127] quantization range.
  • Adds numerical checks, quantizer annotation tests, and unsupported-shape coverage.
File Description
backends/​cadence/​aot/​tests/​test_ref_implementations.py Checks W8A32 operator numerics and input validation.
backends/​cadence/​aot/​tests/​test_quantizer_ops.py Tests annotations, shape restrictions, quantization range, and operator preservation.
backends/​cadence/​aot/​ref_implementations.py Implements float split GRU and new W8A32 references.
backends/​cadence/​aot/​quantizer/​quantizer.py Registers new patterns and symmetric quantization settings.
backends/​cadence/​aot/​quantizer/​patterns.py Adds eligibility checks and fusion for the new operators.
backends/​cadence/​aot/​quantizer/​BUCK Declares the operator-registration dependency.
backends/​cadence/​aot/​ops_registrations.py Defines operator schemas and fake implementations for shape inference.

Review effort: Balanced

Comment on lines +765 to +767
"split_gru(Tensor inputs, Tensor hidden, "
"Tensor w_i_0, Tensor w_h_0, Tensor b_i_0, Tensor b_h_0, "
"Tensor w_i_1, Tensor w_h_1, Tensor b_i_1, Tensor b_h_1) -> Tensor"
Comment on lines +1805 to +1824
if len(layer_norm_node.args) > 4 and layer_norm_node.args[4] != 1e-5:
return (PartitionAnchors(empty=True), layer_norm_node)

return (
PartitionAnchors(
inputs=[],
weights=[(layer_norm_node, 2)],
biases=[(layer_norm_node, 3)],
output=[],
others=[(layer_norm_node, 0)],
),
layer_norm_node,
)

def replacement_op(self) -> OpOverload:
return torch.ops.cadence.quantized_w8a32_layernorm.default

def fuse(self, gm: fx.GraphModule, anchor_node: fx.Node) -> fx.Node | None:
if len(anchor_node.args) < 4:
return None
Summary:

# Summary
This diff includes a symmetric 8 bit quantizer in ET.

Differential Revision: D91777784
Summary:

Adds `cadence::quantized_w8a32_transpose_conv2d`, a transpose convolution with int8 weights and bias and fp32 activations, for a (1, kW) kernel over a single [1, C, 1, W] frame.

`MixedW8A32TransposeConv2dPattern` in `CadenceW8A32MixedQuantizer` matches `conv_transpose2d` with H stride 1, no padding, output padding or dilation, groups 1 and input channels that are a multiple of 4. Other transpose convolutions stay in fp32. The fused op runs in NHWC, with permutes around it. The op schema, fake and reference implementation come with it.

Differential Revision: D91787300
Summary:

Adds `cadence::quantized_w8a32_conv2d`, a convolution with int8 weights and bias and fp32 activations, for a (1, kW) kernel with any W stride over a single [1, C, 1, W] frame.

`MixedW8A32Conv2dPattern` in `CadenceW8A32MixedQuantizer` matches `conv2d` with H stride 1, no padding or dilation, groups 1 and channels that are multiples of 4. Other convolutions stay in fp32. The fused op runs in NHWC, with permutes around it. The op schema, fake and reference implementation come with it.

Differential Revision: D93113537
Summary:

Adds `cadence::quantized_w8a32_layernorm`, a layer norm over the last dim with int8 weight and bias and fp32 activations.

`MixedW8A32LayerNormPattern` in `CadenceW8A32MixedQuantizer` matches `layer_norm` over one dim whose size is a multiple of 4, with weight, bias and the default epsilon. Other layer norms stay in fp32. The op schema, fake and reference implementation come with it.

Differential Revision: D93112974
Summary:

Adds `cadence::quantized_w8a32_split_gru`, one step of a split GRU with int8 weights and biases and fp32 activations. A split GRU runs two independent GRUs: split i reads every other input feature starting at feature i, and the i-th half of the hidden state.

`cadence::split_gru` is a float op for the whole split GRU, so the quantizer sees it as a single node. `MixedW8A32SplitGruPattern` in `CadenceW8A32MixedQuantizer` fuses it for a single frame (batch 1, seq 1) whose per-split input and hidden sizes are multiples of 4. Other shapes stay in fp32. The op schemas, fakes and reference implementations come with it; the quantized reference runs the float one on the dequantized weights.

Differential Revision: D99592763
@meta-codesync meta-codesync Bot changed the title Add Cadence w8a32 split GRU operator Add Cadence w8a32 split GRU operator (#25195) Nov 18, 2026

This branch was previously deployed

1 inactive deployment
cadence — 6e462989 Deployed Nov 18, 2026 by mgiordy via hifi5s-op-test / hifi5s #39063
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. meta-exported

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants