SNOW-1952124 decimal coercion

[sfc-gh-mkeller]

1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

   Fixes SNOW-1952124

2. Fill out the following pre-review checklist:

   • I am adding a new automated test(s) to verify correctness of my new code
     • If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing
   • I am adding new logging messages
   • I am adding a new telemetry message
   • I am adding new credentials
   • I am adding a new dependency
   • If this is a new feature/behavior, I'm adding the Local Testing parity changes.
   • I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: Thread-safe Developer Guidelines

3. Please describe how your code solves the related issue.

   We found a problem where ints and floats should be able to fit into Decimal columns without problems. We have seen this discrepancy in fillna and replace functionalities.
   In this PR I fix this behavior, but place it behind a flag argument and test these new/old behaviours.

[sfc-gh-snowflakedb-snyk-sa]

🎉 Snyk checks have passed. No issues have been found so far.

✅ security/snyk check is complete. No issues have been found. (View Details)

✅ license/snyk check is complete. No issues have been found. (View Details)

[Copilot]

Pull Request Overview

This PR addresses SNOW-1952124 by enhancing the coercion of int and float values into Decimal columns for the fillna and replace DataFrame methods. Key changes include:

• Adding new tests in tests/integ/test_dataframe.py to verify decimal coercion behavior when using the include_decimal flag.
• Updating the implementation in src/snowflake/snowpark/dataframe_na_functions.py to support the include_decimal flag in functions fill and replace.
• Enhancing type utility functions and adding a helper function (null_data4) in tests/utils.py to support the new decimal coercion behavior.

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/integ/test_dataframe.py	Added tests verifying decimal coercion with include_decimal flag.
src/snowflake/snowpark/dataframe_na_functions.py	Updated fill and replace functions, including _is_value_type_matching_for_na_function with a new include_decimal parameter.
tests/utils.py	Introduced null_data4 to support testing of Decimal columns.
src/snowflake/snowpark/_internal/type_utils.py	Minor code style and import updates related to type handling.
CHANGELOG.md	Documented the new behavior in fillna and replace functions.

Comments suppressed due to low confidence (1)

src/snowflake/snowpark/dataframe_na_functions.py:60

• Consider flattening the tuple for int_types by concatenating DecimalType instead of nesting tuples (e.g., int_types = int_types + (DecimalType,)). This will ensure the isinstance check behaves as expected when include_decimal is True.

int_types = (int_types, DecimalType)

[Copilot]

Consider flattening the tuple for float_types by concatenating DecimalType instead of nesting tuples (e.g., float_types = float_types + (DecimalType,)). This adjustment will allow isinstance checks to correctly include DecimalType when include_decimal is True.

Suggested change

	int_types = (int_types, DecimalType)
	float_types = (float_types, DecimalType)
	int_types = int_types + (DecimalType,)
	float_types = float_types + (DecimalType,)

[sfc-gh-mkeller]

From https://docs.python.org/3/library/functions.html#isinstance

If classinfo is a tuple of type objects (or recursively, other such tuples)

So the code should work as expected. Unless a maintainer would like me to change this I won't

[sfc-gh-dryazanov]

It just occurred to me that, for symmetry, we should support filling int/long/float/double columns with decimals... But for some reason Spark does not allow that, so I don't know what the right answer is.

[sfc-gh-mkeller]

I think that's because int/float can be safely fit into decimal, but not the other way around

[sfc-gh-wshangguan]

Agree with the type conversion. And spark explicitly mentioned that decimal can't be the input for replace api.

[sfc-gh-wshangguan]

Why do we have this import?

[sfc-gh-mkeller]

Type checking. My editor was telling me that the type-hint at some part of this file was broken

[sfc-gh-wshangguan]

Adding for better readability?

[sfc-gh-mkeller]

My editor has black formatting built-in on save, it was done by it automatically. I can revert this if people would like me to!

[sfc-gh-aling]

it seems related to coercion in localtesting, you can temporarily skip the local testing by:

def test_replace_with_coercion(session, local_testing_mode):
    # existing code
    # ...
    if local_testing_mode:
        # SNOW-xxxx local test gap
        return
    # your new test code