Repository navigation
Apply 1D weights along the collapsed dimension in SUM - #7343
Open
bhaskargurram-ai wants to merge 4 commits into
Open
bhaskargurram-ai wants to merge 4 commits into
bhaskargurram-ai wants to merge 4 commits into
Conversation
Like np.average, weighted aggregations accept 1D weights for a single collapsed dimension. MEAN and RMS (via np.average) and the real-data path of Cube.collapsed (which moves the collapsed dimension to the end) handle these, but SUM multiplied the weights with the data directly. For lazy data, where the collapsed dimension stays in place, the weights were broadcast against the last dimension: this raised a broadcasting error, or silently weighted the wrong dimension when both had the same length. SUM with returned=True also failed for 1D weights, with real and lazy data. Align 1D weights with the collapsed axis in _sum. Also accept weights given as a list in Cube.collapsed, and fix the shape check of 1D weights in WPERCENTILE for the negative axis that Cube.collapsed passes. Fixes SciTools#5083
_Weights now converts array-likes such as lists to arrays, so the test checks the values instead of the identity of the list.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Closes #5083.
Cube.collapsedaccepts 1D weights for a single collapsed dimension (the docstring says weights "should be 1d (for collapsing over a 1d coordinate) or match the shape of the cube"). MEAN and RMS handle these throughnp.average/da.ma.average, and on the real-data path the collapsed dimension is moved to the end first._sum, however, multiplies the weights with the data directly. With lazy data the collapsed dimension stays in place, so the 1D weights are broadcast against the last dimension:operands could not be broadcast together with shapes (2,) (2, 3))(3, 3)cube collapsed over the first dimension withweights=[1, 0, 0]gives[9, 0, 0]instead of[0, 1, 2](the real-data result).Along the way I found a few related failures with 1D weights:
SUMwithreturned=Truefailed for real and lazy data (MaskError: Mask and data not compatible)AttributeError: 'list' object has no attribute 'ndim')WPERCENTILErejected 1D weights throughCube.collapsed. The shape checkdata.shape[axis : axis + 1]is empty for theaxis=-1thatcollapsedpasses, so the error said weights should be().Changes:
_sumaligns 1D weights with the given axis, asnp.averagedoes, for both the sum and thereturnedweights total._Weightsconverts array-likes that aren't arrays (e.g. lists) withnp.asanyarray. Lazy arrays are left as they are._weighted_percentilechecks(data.shape[axis],), which also works for a negative axis.Tests:
test_Cube.py: lazy SUM with 1D weights along y and x (also inherited by the cube/str/dim-metadata weight variants), the unit adaptation test for the lazy case that the existing comment said fails with Lazy SUM with 1D weights sometimes fails #5083 (I removed those comments), and a newTest_collapsed__1d_weights_arrayclass covering the square-cube case,returned=True, list weights, lazy weights and WPERCENTILEtest_WPERCENTILE.py: 1D weights withaxis=-1The 12 new tests fail on
mainand pass with this change. I also ranlib/iris/tests/unitandlib/iris/tests/integrationwith and without the change: the only differences are the new tests. The other failures there come from missing test data,ncgenand Natural Earth downloads in my environment, and happen onmaintoo.I haven't added the What's New fragment yet because it needs the PR number. I'll add
changelog/<PR>.bugfix.rstonce the PR is open.Checklist