SNOW-1952124 decimal coercion #3159

sfc-gh-mkeller · 2025-03-14T20:42:23Z

Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

Fixes SNOW-1952124
Fill out the following pre-review checklist:
- I am adding a new automated test(s) to verify correctness of my new code
  - If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing
- I am adding new logging messages
- I am adding a new telemetry message
- I am adding new credentials
- I am adding a new dependency
- If this is a new feature/behavior, I'm adding the Local Testing parity changes.
- I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: Thread-safe Developer Guidelines
Please describe how your code solves the related issue.

We found a problem where ints and floats should be able to fit into Decimal columns without problems. We have seen this discrepancy in fillna and replace functionalities.
In this PR I fix this behavior, but place it behind a flag argument and test these new/old behaviours.

sfc-gh-snowflakedb-snyk-sa · 2025-03-14T20:42:35Z

🎉 Snyk checks have passed. No issues have been found so far.

✅ security/snyk check is complete. No issues have been found. (View Details)

✅ license/snyk check is complete. No issues have been found. (View Details)

Copilot

Pull Request Overview

This PR addresses SNOW-1952124 by enhancing the coercion of int and float values into Decimal columns for the fillna and replace DataFrame methods. Key changes include:

Adding new tests in tests/integ/test_dataframe.py to verify decimal coercion behavior when using the include_decimal flag.
Updating the implementation in src/snowflake/snowpark/dataframe_na_functions.py to support the include_decimal flag in functions fill and replace.
Enhancing type utility functions and adding a helper function (null_data4) in tests/utils.py to support the new decimal coercion behavior.

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/integ/test_dataframe.py	Added tests verifying decimal coercion with include_decimal flag.
src/snowflake/snowpark/dataframe_na_functions.py	Updated fill and replace functions, including _is_value_type_matching_for_na_function with a new include_decimal parameter.
tests/utils.py	Introduced null_data4 to support testing of Decimal columns.
src/snowflake/snowpark/_internal/type_utils.py	Minor code style and import updates related to type handling.
CHANGELOG.md	Documented the new behavior in fillna and replace functions.

Comments suppressed due to low confidence (1)

src/snowflake/snowpark/dataframe_na_functions.py:60

Consider flattening the tuple for int_types by concatenating DecimalType instead of nesting tuples (e.g., int_types = int_types + (DecimalType,)). This will ensure the isinstance check behaves as expected when include_decimal is True.

int_types = (int_types, DecimalType)

Copilot · 2025-03-14T20:50:07Z

src/snowflake/snowpark/dataframe_na_functions.py

+        int_types = (int_types, DecimalType)
+        float_types = (float_types, DecimalType)


Consider flattening the tuple for float_types by concatenating DecimalType instead of nesting tuples (e.g., float_types = float_types + (DecimalType,)). This adjustment will allow isinstance checks to correctly include DecimalType when include_decimal is True.

Suggested change

int_types = (int_types, DecimalType)

float_types = (float_types, DecimalType)

int_types = int_types + (DecimalType,)

float_types = float_types + (DecimalType,)

From https://docs.python.org/3/library/functions.html#isinstance

If classinfo is a tuple of type objects (or recursively, other such tuples)

So the code should work as expected. Unless a maintainer would like me to change this I won't

sfc-gh-dryazanov · 2025-03-14T21:00:54Z

tests/integ/test_dataframe.py

+    # Fill Decimal columns with int
+    Utils.check_answer(
+        TestData.null_data4(session).fillna(123, include_decimal=True),
+        [
+            Row(decimal.Decimal(1), decimal.Decimal(123)),
+            Row(decimal.Decimal(123), 2),
+        ],
+        sort=False,
+    )
+    # Fill Decimal columns with float
+    Utils.check_answer(
+        TestData.null_data4(session).fillna(123.0, include_decimal=True),
+        [
+            Row(decimal.Decimal(1), decimal.Decimal(123)),
+            Row(decimal.Decimal(123), 2),
+        ],
+        sort=False,
+    )


It just occurred to me that, for symmetry, we should support filling int/long/float/double columns with decimals... But for some reason Spark does not allow that, so I don't know what the right answer is.

I think that's because int/float can be safely fit into decimal, but not the other way around

Agree with the type conversion. And spark explicitly mentioned that decimal can't be the input for replace api.

sfc-gh-wshangguan · 2025-03-14T21:57:40Z

src/snowflake/snowpark/_internal/type_utils.py

@@ -94,6 +94,8 @@
    )

 if TYPE_CHECKING:
+    import snowflake.snowpark.column


Why do we have this import?

Type checking. My editor was telling me that the type-hint at some part of this file was broken

sfc-gh-wshangguan · 2025-03-14T21:57:55Z

src/snowflake/snowpark/_internal/type_utils.py

-                        field.name
-                        if context._should_use_structured_type_semantics()
-                        else quote_name(field.name, keep_case=True),
+                        (


Adding for better readability?

My editor has black formatting built-in on save, it was done by it automatically. I can revert this if people would like me to!

sfc-gh-aling · 2025-03-17T18:22:28Z

tests/integ/test_dataframe.py

+    # Replace Decimal value with int
+    Utils.check_answer(


it seems related to coercion in localtesting, you can temporarily skip the local testing by:

def test_replace_with_coercion(session, local_testing_mode): # existing code # ... if local_testing_mode: # SNOW-xxxx local test gap return # your new test code

sfc-gh-mkeller requested review from a team as code owners March 14, 2025 20:42

sfc-gh-mkeller requested review from sfc-gh-jdu, sfc-gh-aling and sfc-gh-yuwang March 14, 2025 20:42

sfc-gh-mkeller requested review from Copilot and removed request for sfc-gh-jdu, sfc-gh-aling and sfc-gh-yuwang March 14, 2025 20:49

Copilot AI reviewed Mar 14, 2025

View reviewed changes

sfc-gh-dryazanov reviewed Mar 14, 2025

View reviewed changes

sfc-gh-wshangguan reviewed Mar 14, 2025

View reviewed changes

sfc-gh-yixie requested review from sfc-gh-jdu, sfc-gh-aling, sfc-gh-aalam and sfc-gh-jrose March 14, 2025 22:21

sfc-gh-aalam approved these changes Mar 14, 2025

View reviewed changes

sfc-gh-aling reviewed Mar 17, 2025

View reviewed changes

sfc-gh-mkeller force-pushed the mkeller/SNOW-1952124/decimal-coercion branch from c6e04b4 to a6e8153 Compare March 17, 2025 18:46

sfc-gh-aling approved these changes Mar 17, 2025

View reviewed changes

sfc-gh-mkeller requested a review from a team March 17, 2025 18:55

sfc-gh-jdu approved these changes Mar 17, 2025

View reviewed changes

sfc-gh-heshah self-requested a review March 17, 2025 23:12

sfc-gh-wshangguan and others added 6 commits March 18, 2025 20:58

add Decimaltype

8ff535d

formatting and type-hint fixes

29ca3cd

adding arguments to protect against beraking change for Decimal types

02f5ad3

adding test for new behaviour

99c9e3b

adding changelog

cf255d9

adding doc-strings

2cb0443

fix local testing tests

862c431

sfc-gh-mkeller force-pushed the mkeller/SNOW-1952124/decimal-coercion branch from a6e8153 to 862c431 Compare March 19, 2025 03:58

sfc-gh-mkeller merged commit 0a02767 into main Mar 19, 2025
37 of 38 checks passed

sfc-gh-mkeller deleted the mkeller/SNOW-1952124/decimal-coercion branch March 19, 2025 04:44

github-actions bot locked and limited conversation to collaborators Mar 19, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SNOW-1952124 decimal coercion #3159

SNOW-1952124 decimal coercion #3159

sfc-gh-mkeller commented Mar 14, 2025

sfc-gh-snowflakedb-snyk-sa commented Mar 14, 2025 •

edited

Loading

Copilot AI left a comment

Copilot AI Mar 14, 2025

Provide additional feedback

Please help us improve GitHub Copilot by sharing more details about this comment.

sfc-gh-mkeller Mar 14, 2025

sfc-gh-dryazanov Mar 14, 2025

sfc-gh-mkeller Mar 14, 2025

sfc-gh-wshangguan Mar 14, 2025

sfc-gh-wshangguan Mar 14, 2025

sfc-gh-mkeller Mar 14, 2025

sfc-gh-wshangguan Mar 14, 2025

sfc-gh-mkeller Mar 14, 2025

sfc-gh-aling Mar 17, 2025

		int_types = (int_types, DecimalType)
		float_types = (float_types, DecimalType)

SNOW-1952124 decimal coercion #3159

SNOW-1952124 decimal coercion #3159

Conversation

sfc-gh-mkeller commented Mar 14, 2025

sfc-gh-snowflakedb-snyk-sa commented Mar 14, 2025 • edited Loading

🎉 Snyk checks have passed. No issues have been found so far.

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Copilot AI Mar 14, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfc-gh-snowflakedb-snyk-sa commented Mar 14, 2025 •

edited

Loading