Skip to content

Conversation

Copilot
Copy link
Contributor

@Copilot Copilot AI commented Aug 31, 2025

Fixes #116133

Core Fix

  • Introduced SumOfSquaredAbsoluteDifferences<T> operation that computes T.Abs(x - mean) before squaring
  • Updated Tensor.StdDev to use this new operation, ensuring mathematically correct results for complex numbers

Test Improvements

  • Converted TensorStdDevTests to TensorStdDevFloatTests Theory test with multiple test cases including sequential numbers, same values, and mixed positive/negative values
  • Converted TensorStdDevComplexTests to Theory test with test cases for complex numbers from the original issue, purely imaginary numbers, and real complex numbers
  • Added separate TensorStdDevNonContiguousTests to ensure non-contiguous tensor calculations work correctly
  • Added a TestComplex struct that throws NotImplementedException for interface methods not essential for StdDev calculations

Test Coverage

The new Theory tests cover:

  • Cases where Abs doesn't matter: Real numbers (float values) where |x|² = x² for the standard deviation calculation
  • Cases where Abs is critical: Complex numbers where |z|² ≠ z², ensuring the fix works correctly
  • Edge cases: All same values (should return 0), mixed positive/negative values, purely imaginary numbers, and real complex numbers

Both TensorPrimitives.StdDev and Tensor.StdDev now return identical mathematically correct results for all test cases.

@Copilot Copilot AI changed the title [WIP] Tensor.StdDev vs TensorPrimitivers.StdDev differences for Complex input Fix Tensor.StdDev vs TensorPrimitives.StdDev differences for Complex input Aug 31, 2025
@Copilot Copilot AI requested a review from jeffhandley August 31, 2025 07:16
Copilot finished work on behalf of jeffhandley August 31, 2025 07:16
…tion

- Fix typo: "TensorPrimitivers" -> "TensorPrimitives" in test comment
- Replace custom SumOfSquaredMagnitudeDifferences operation with direct use of TensorPrimitives.StdDev
- Leverage TensorPrimitives vectorization optimizations where T.Abs is only called when necessary
- Remove unnecessary custom operation that always called T.Abs for all types
- Use TryGetSpan for contiguous tensors and fallback to FlattenTo for non-contiguous tensors

Co-authored-by: jeffhandley <[email protected]>
[Fact]
public static void TensorStdDevComplexTests()
{
// Test case from issue: Tensor.StdDev vs TensorPrimitivers.StdDev differences for Complex input
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed the typo in commit 4585313.

@Copilot Copilot AI requested a review from jeffhandley August 31, 2025 10:22
Copilot finished work on behalf of jeffhandley August 31, 2025 10:22
@Copilot Copilot AI requested a review from jeffhandley August 31, 2025 11:18
Copilot finished work on behalf of jeffhandley August 31, 2025 11:18
…or non-essential methods and convert StdDev tests to Theory tests

Co-authored-by: jeffhandley <[email protected]>
@Copilot Copilot AI requested a review from jeffhandley August 31, 2025 13:02
Copilot finished work on behalf of jeffhandley August 31, 2025 13:02
@jeffhandley
Copy link
Member

@tannergooding I explored a couple options on this and this seems best to me. For my own reference/learning, I looked into how NumPy handles complex numbers for Euclidean Distance and Standard Deviation of complex numbers and found it also resorts to an absolute value of the diff for StdDev but not Distance, increasing my confidence this was the correct approach.

I do worry slightly about introducing the Abs call purely to support complex numbers, but I couldn't find a way to distinguish between real and (custom) complex numbers reliably from the interfaces. If you have a trick up your sleeve here, that could change the implementation.

@jeffhandley jeffhandley marked this pull request as ready for review August 31, 2025 14:01
@Copilot Copilot AI review requested due to automatic review settings August 31, 2025 14:01
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR fixes mathematical correctness issues between Tensor.StdDev and TensorPrimitives.StdDev when calculating standard deviation for complex numbers. The core problem was that the tensor implementation was squaring complex differences directly instead of taking the absolute value before squaring, which is required for mathematically correct standard deviation calculation.

Key changes:

  • Introduced SumOfSquaredMagnitudeDifferences<T> operation that properly computes T.Abs(x - mean) before squaring
  • Updated Tensor.StdDev to use the new operation for consistent results with TensorPrimitives.StdDev
  • Enhanced test coverage with Theory tests covering both real and complex number scenarios

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
src/libraries/System.Numerics.Tensors/tests/TensorTests.cs Converts existing tests to Theory tests with comprehensive test cases for both float and complex numbers, adds TestComplex struct for testing
src/libraries/System.Numerics.Tensors/src/System/Numerics/Tensors/netcore/TensorOperation.cs Adds new SumOfSquaredMagnitudeDifferences operation that correctly applies absolute value before squaring
src/libraries/System.Numerics.Tensors/src/System/Numerics/Tensors/netcore/Tensor.cs Updates StdDev implementation to use the new mathematically correct operation

@jeffhandley jeffhandley enabled auto-merge (squash) September 16, 2025 19:20
@jeffhandley jeffhandley merged commit 0c5f13b into main Sep 17, 2025
88 checks passed
@jeffhandley
Copy link
Member

/backport to release/10.0

Copy link
Contributor

Started backporting to release/10.0: https://github.com/dotnet/runtime/actions/runs/17786818289

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Tensor.StdDev vs TensorPrimitives.StdDev differences for Complex input
4 participants