Repository navigation
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25195
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 6e46298 with merge base ec5e3cf ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
This PR needs a
|
There was a problem hiding this comment.
🟡 Changes recommended
Unsupported split-GRU shapes fail export instead of falling back to fp32, and normalization fusion can silently replace nondefault epsilon with 1e-5, changing results.
2 open issues
What changed in this PR
Adds a two-way split GRU to Cadence’s mixed-precision quantization pipeline, using int8 weights and biases with fp32 activations.
Changes:
- Adds split-GRU schemas, fake implementations, references, and shape-gated quantization.
- Extends W8A32 patterns to convolution, transposed convolution, and layer normalization, using the symmetric
[-127, 127]quantization range. - Adds numerical checks, quantizer annotation tests, and unsupported-shape coverage.
| File | Description |
|---|---|
backends/cadence/aot/tests/test_ref_implementations.py |
Checks W8A32 operator numerics and input validation. |
backends/cadence/aot/tests/test_quantizer_ops.py |
Tests annotations, shape restrictions, quantization range, and operator preservation. |
backends/cadence/aot/ref_implementations.py |
Implements float split GRU and new W8A32 references. |
backends/cadence/aot/quantizer/quantizer.py |
Registers new patterns and symmetric quantization settings. |
backends/cadence/aot/quantizer/patterns.py |
Adds eligibility checks and fusion for the new operators. |
backends/cadence/aot/quantizer/BUCK |
Declares the operator-registration dependency. |
backends/cadence/aot/ops_registrations.py |
Defines operator schemas and fake implementations for shape inference. |
Review effort: Balanced
| "split_gru(Tensor inputs, Tensor hidden, " | ||
| "Tensor w_i_0, Tensor w_h_0, Tensor b_i_0, Tensor b_h_0, " | ||
| "Tensor w_i_1, Tensor w_h_1, Tensor b_i_1, Tensor b_h_1) -> Tensor" |
| if len(layer_norm_node.args) > 4 and layer_norm_node.args[4] != 1e-5: | ||
| return (PartitionAnchors(empty=True), layer_norm_node) | ||
|
|
||
| return ( | ||
| PartitionAnchors( | ||
| inputs=[], | ||
| weights=[(layer_norm_node, 2)], | ||
| biases=[(layer_norm_node, 3)], | ||
| output=[], | ||
| others=[(layer_norm_node, 0)], | ||
| ), | ||
| layer_norm_node, | ||
| ) | ||
|
|
||
| def replacement_op(self) -> OpOverload: | ||
| return torch.ops.cadence.quantized_w8a32_layernorm.default | ||
|
|
||
| def fuse(self, gm: fx.GraphModule, anchor_node: fx.Node) -> fx.Node | None: | ||
| if len(anchor_node.args) < 4: | ||
| return None |
Summary: # Summary This diff includes a symmetric 8 bit quantizer in ET. Differential Revision: D91777784
Summary: Adds `cadence::quantized_w8a32_transpose_conv2d`, a transpose convolution with int8 weights and bias and fp32 activations, for a (1, kW) kernel over a single [1, C, 1, W] frame. `MixedW8A32TransposeConv2dPattern` in `CadenceW8A32MixedQuantizer` matches `conv_transpose2d` with H stride 1, no padding, output padding or dilation, groups 1 and input channels that are a multiple of 4. Other transpose convolutions stay in fp32. The fused op runs in NHWC, with permutes around it. The op schema, fake and reference implementation come with it. Differential Revision: D91787300
Summary: Adds `cadence::quantized_w8a32_conv2d`, a convolution with int8 weights and bias and fp32 activations, for a (1, kW) kernel with any W stride over a single [1, C, 1, W] frame. `MixedW8A32Conv2dPattern` in `CadenceW8A32MixedQuantizer` matches `conv2d` with H stride 1, no padding or dilation, groups 1 and channels that are multiples of 4. Other convolutions stay in fp32. The fused op runs in NHWC, with permutes around it. The op schema, fake and reference implementation come with it. Differential Revision: D93113537
Summary: Adds `cadence::quantized_w8a32_layernorm`, a layer norm over the last dim with int8 weight and bias and fp32 activations. `MixedW8A32LayerNormPattern` in `CadenceW8A32MixedQuantizer` matches `layer_norm` over one dim whose size is a multiple of 4, with weight, bias and the default epsilon. Other layer norms stay in fp32. The op schema, fake and reference implementation come with it. Differential Revision: D93112974
Summary: Adds `cadence::quantized_w8a32_split_gru`, one step of a split GRU with int8 weights and biases and fp32 activations. A split GRU runs two independent GRUs: split i reads every other input feature starting at feature i, and the i-th half of the hidden state. `cadence::split_gru` is a float op for the whole split GRU, so the quantizer sees it as a single node. `MixedW8A32SplitGruPattern` in `CadenceW8A32MixedQuantizer` fuses it for a single frame (batch 1, seq 1) whose per-split input and hidden sizes are multiples of 4. Other shapes stay in fp32. The op schemas, fakes and reference implementations come with it; the quantized reference runs the float one on the dequantized weights. Differential Revision: D99592763
b7a6938 to
6e46298
Compare


Summary:
Adds
cadence::quantized_w8a32_split_gru, one step of a split GRU with int8 weights and biases and fp32 activations. A split GRU runs two independent GRUs: split i reads every other input feature starting at feature i, and the i-th half of the hidden state.cadence::split_gruis a float op for the whole split GRU, so the quantizer sees it as a single node.MixedW8A32SplitGruPatterninCadenceW8A32MixedQuantizerfuses it for a single frame (batch 1, seq 1) whose per-split input and hidden sizes are multiples of 4. Other shapes stay in fp32. The op schemas, fakes and reference implementations come with it; the quantized reference runs the float one on the dequantized weights.Differential Revision: D99592763