Skip to content

Apply 1D weights along the collapsed dimension in SUM - #7343

Open
bhaskargurram-ai wants to merge 4 commits into
SciTools:mainfrom
bhaskargurram-ai:fix/collapsed-1d-weights
Open

bhaskargurram-ai wants to merge 4 commits into
SciTools:mainfrom
bhaskargurram-ai:fix/collapsed-1d-weights

Conversation

@bhaskargurram-ai

@bhaskargurram-ai bhaskargurram-ai commented Oct 18, 2026 •

Copy link
Copy Markdown

Description

Closes #5083.

Cube.collapsed accepts 1D weights for a single collapsed dimension (the docstring says weights "should be 1d (for collapsing over a 1d coordinate) or match the shape of the cube"). MEAN and RMS handle these through np.average/da.ma.average, and on the real-data path the collapsed dimension is moved to the end first. _sum, however, multiplies the weights with the data directly. With lazy data the collapsed dimension stays in place, so the 1D weights are broadcast against the last dimension:

  • if the lengths differ, you get the error from Lazy SUM with 1D weights sometimes fails #5083 (operands could not be broadcast together with shapes (2,) (2, 3))
  • if they are equal, the result is silently wrong. A lazy (3, 3) cube collapsed over the first dimension with weights=[1, 0, 0] gives [9, 0, 0] instead of [0, 1, 2] (the real-data result).

Along the way I found a few related failures with 1D weights:

  • SUM with returned=True failed for real and lazy data (MaskError: Mask and data not compatible)
  • weights given as a list failed on the real-data path (AttributeError: 'list' object has no attribute 'ndim')
  • WPERCENTILE rejected 1D weights through Cube.collapsed. The shape check data.shape[axis : axis + 1] is empty for the axis=-1 that collapsed passes, so the error said weights should be ().

Changes:

  • _sum aligns 1D weights with the given axis, as np.average does, for both the sum and the returned weights total.
  • _Weights converts array-likes that aren't arrays (e.g. lists) with np.asanyarray. Lazy arrays are left as they are.
  • _weighted_percentile checks (data.shape[axis],), which also works for a negative axis.

Tests:

  • test_Cube.py: lazy SUM with 1D weights along y and x (also inherited by the cube/str/dim-metadata weight variants), the unit adaptation test for the lazy case that the existing comment said fails with Lazy SUM with 1D weights sometimes fails #5083 (I removed those comments), and a new Test_collapsed__1d_weights_array class covering the square-cube case, returned=True, list weights, lazy weights and WPERCENTILE
  • test_WPERCENTILE.py: 1D weights with axis=-1

The 12 new tests fail on main and pass with this change. I also ran lib/iris/tests/unit and lib/iris/tests/integration with and without the change: the only differences are the new tests. The other failures there come from missing test data, ncgen and Natural Earth downloads in my environment, and happen on main too.

I haven't added the What's New fragment yet because it needs the PR number. I'll add changelog/<PR>.bugfix.rst once the PR is open.

Checklist

  • Included a What's New entry (to add once the PR number is known)
  • Incorporated type hints in any changed code (no signatures changed)
  • Checked if tests need updating
  • Checked if benchmarks need updating
  • Checked if documentation needs updating
  • Checked if dependencies need updating
  • Confirmed that the GitHub 'checks' on this PR are passing ✅

Like np.average, weighted aggregations accept 1D weights for a single
collapsed dimension. MEAN and RMS (via np.average) and the real-data
path of Cube.collapsed (which moves the collapsed dimension to the end)
handle these, but SUM multiplied the weights with the data directly. For
lazy data, where the collapsed dimension stays in place, the weights
were broadcast against the last dimension: this raised a broadcasting
error, or silently weighted the wrong dimension when both had the same
length. SUM with returned=True also failed for 1D weights, with real
and lazy data.

Align 1D weights with the collapsed axis in _sum. Also accept weights
given as a list in Cube.collapsed, and fix the shape check of 1D weights
in WPERCENTILE for the negative axis that Cube.collapsed passes.

Fixes SciTools#5083
@CLAassistant

CLAassistant commented Oct 18, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

trexfeathers and others added 2 commits November 19, 2026 16:46
_Weights now converts array-likes such as lists to arrays, so the test checks the values instead of the identity of the list.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

Lazy SUM with 1D weights sometimes fails

3 participants