SNOW-1918055: Update agg error for unsupported aggregation functions #3133

sfc-gh-lmukhopadhyay · 2025-03-07T21:43:30Z

Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

Fixes SNOW-1918055
Fill out the following pre-review checklist:
- I am adding a new automated test(s) to verify correctness of my new code
  - If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing
- I am adding new logging messages
- I am adding a new telemetry message
- I am adding new credentials
- I am adding a new dependency
- If this is a new feature/behavior, I'm adding the Local Testing parity changes.
- I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: Thread-safe Developer Guidelines
Please describe how your code solves the related issue.

Updating groupby.agg and agg error for unsupported aggregation functions to match pandas. It will now return
'SeriesGroupBy' object has no attribute 'COUNT'

Signed-off-by: Labanya Mukhopadhyay <[email protected]>

…rror

Signed-off-by: Labanya Mukhopadhyay <[email protected]>

…rror

Signed-off-by: Labanya Mukhopadhyay <[email protected]>

sfc-gh-joshi · 2025-03-08T00:12:01Z

src/snowflake/snowpark/modin/plugin/_internal/aggregation_utils.py

+        bool
+            True if all functions in the list are snowflake supported aggregation functions, otherwise,
+        return False
+        list
+            The list of unsupported functions used for aggregation.


nit: usually I see docstrings for functions like this explicitly mention tuple[bool, list] as the return type, and describe what each member of the tuple means rather than separating out the values.

sfc-gh-joshi · 2025-03-08T00:13:31Z

src/snowflake/snowpark/modin/plugin/_internal/aggregation_utils.py

-            for value in agg_func.values()
-        )
+            if not is_supported_func:
+                supported_flag = False


Why not just return here? Is your intent to combine the unsupported_arguments lists?

The first function that's unsupported will be returned. The case with multiple unsupported functions needs to be handled which will require returning in this loop, so I'll make those changes as well as using repr_aggregate_function(agg_func, agg_kwargs)!

sfc-gh-joshi · 2025-03-08T00:14:31Z

src/snowflake/snowpark/modin/plugin/_internal/aggregation_utils.py

    """
    # validate agg_func, only snowflake builtin agg function or dict of snowflake builtin agg
    # function can be implemented in distributed way.
+    unsupported_arguments: list[str] = []
+    supported_flag = True


I prefer the name "is_supported" so there's 0 ambiguity about the meaning of the flag's T/F value.

sfc-gh-joshi · 2025-03-08T00:15:05Z

src/snowflake/snowpark/modin/plugin/compiler/snowflake_query_compiler.py

+        ) = check_is_aggregation_supported_in_snowflake(agg_func, agg_kwargs, axis)
+        if not is_supported:
+            raise AttributeError(
+                f"'SeriesGroupBy' object has no attribute '{unsupported_arguments}'"


what happens if this is an aggregation that native pandas supports but we do not?

This should be returning False since we check if there is a corresponding Snowflake aggregation function via get_snowflake_agg_func(). The overall checking logic for if a function is supported should not be changing here

tests/integ/modin/groupby/test_groupby_basic_agg.py

sfc-gh-jjiao · 2025-03-08T00:45:11Z

CHANGELOG.md

@@ -54,12 +54,13 @@
 - Fixed a bug where creating a Dataframe with large number of values raised `Unsupported feature 'SCOPED_TEMPORARY'.` error if thread-safe session was disabled.
 - Fixed a bug where `df.describe` raised internal SQL execution error when the dataframe is created from reading a stage file and CTE optimization is enabled.
 - Fixed a bug where `df.order_by(A).select(B).distinct()` would generate invalid SQL when simplified query generation was enabled using `session.conf.set("use_simplified_query_generation", True)`.
-  - Disabled simplified query generation by default.


Good catch!

sfc-gh-jjiao · 2025-03-08T00:48:37Z

src/snowflake/snowpark/modin/plugin/_internal/aggregation_utils.py

@@ -851,40 +851,50 @@ def _is_supported_snowflake_agg_func(
                    The value can be different for different aggregation functions.
    Returns:
        is_valid: bool. Whether it is valid to implement with snowflake or not.


I suspected this comment is the result of copy-pasting , Whether it is valid to implement with snowflake or not. is not consistent of the semantic of this function (to check "check if the aggregation function is supported with snowflake")

sfc-gh-jjiao · 2025-03-08T00:57:08Z

src/snowflake/snowpark/modin/plugin/_internal/aggregation_utils.py

    """
    if isinstance(agg_func, tuple) and len(agg_func) == 2:
        # For named aggregations, like `df.agg(new_col=("old_col", "sum"))`,
        # take the second part of the named aggregation.
        agg_func = agg_func[0]
-    return get_snowflake_agg_func(agg_func, agg_kwargs, axis, _is_df_agg) is not None
+    if get_snowflake_agg_func(agg_func, agg_kwargs, axis, _is_df_agg) is None:
+        return False, agg_func


is agg_func guaranteed to be a list?

sfc-gh-jjiao · 2025-03-08T01:05:44Z

src/snowflake/snowpark/modin/plugin/_internal/aggregation_utils.py

+    unsupported_list: list[str] = []
+    for func in agg_funcs:
+        is_supported, unsupported_list = _is_supported_snowflake_agg_func(
+            func, agg_kwargs, axis, _is_df_agg
+        )


does the unsupported_list need to be appended to in the for loop ? it seems like it has been replaced/overwritten every time here

sfc-gh-jjiao

Could we please add a unit test for this function _are_all_agg_funcs_supported_by_snowflake ? we could use some functions that for sure is not going to be supported. I suspect the current code change has a bug. Thanks

Signed-off-by: Labanya Mukhopadhyay <[email protected]>

…rror

Signed-off-by: Labanya Mukhopadhyay <[email protected]>

sfc-gh-jjiao · 2025-03-19T20:22:17Z

tests/unit/modin/test_aggregation_utils.py

+        # ("max", {}, True),  # snowflake supported aggregation function str
+        # (np.sum, {}, True),  # snowflake supported aggregation function numpy function
+        # (np.quantile, {}, False),  # snowflake unsupported aggregation function
+        # (
+        #     {"col1": "max", "col2": np.sum},
+        #     {},
+        #     True,
+        # ),  # dictionary fo aggregation functions


why are we commenting these lines out ?

Signed-off-by: Labanya Mukhopadhyay <[email protected]>

sfc-gh-snowflakedb-snyk-sa · 2025-03-20T08:25:22Z

🎉 Snyk checks have passed. No issues have been found so far.

✅ security/snyk check is complete. No issues have been found. (View Details)

✅ license/snyk check is complete. No issues have been found. (View Details)

Signed-off-by: Labanya Mukhopadhyay <[email protected]>

…rror

sfc-gh-lmukhopadhyay added 4 commits March 7, 2025 13:34

SNOW-1918055: Add groupby agg error details

ef9abd2

Signed-off-by: Labanya Mukhopadhyay <[email protected]>

Merge branch 'main' into lmukhopadhyay-SNOW-1918055-groupby-agg-fix-e…

1855cd6

…rror

update changelog

7af1aa0

Signed-off-by: Labanya Mukhopadhyay <[email protected]>

update comments

31b3fb3

Signed-off-by: Labanya Mukhopadhyay <[email protected]>

github-actions bot added the snowpark-pandas label Mar 7, 2025

sfc-gh-lmukhopadhyay added NO-PANDAS-CHANGEDOC-UPDATES This PR does not update Snowpark pandas docs and removed snowpark-pandas labels Mar 7, 2025

Merge branch 'main' into lmukhopadhyay-SNOW-1918055-groupby-agg-fix-e…

97692e9

…rror

github-actions bot added the snowpark-pandas label Mar 7, 2025

fix agg error check

36ccf27

Signed-off-by: Labanya Mukhopadhyay <[email protected]>

sfc-gh-lmukhopadhyay marked this pull request as ready for review March 7, 2025 23:55

sfc-gh-lmukhopadhyay requested a review from a team as a code owner March 7, 2025 23:56

sfc-gh-lmukhopadhyay requested review from sfc-gh-mvashishtha and sfc-gh-joshi March 7, 2025 23:56

sfc-gh-lmukhopadhyay changed the title ~~SNOW-1918055: Update groupby.agg error for unsupported aggregation functions~~ SNOW-1918055: Update agg error for unsupported aggregation functions Mar 7, 2025

sfc-gh-joshi requested changes Mar 8, 2025

View reviewed changes

sfc-gh-jjiao reviewed Mar 8, 2025

View reviewed changes

sfc-gh-jjiao requested changes Mar 8, 2025

View reviewed changes

sfc-gh-lmukhopadhyay added 3 commits March 20, 2025 00:40

update agg to support multiple incorrect funcs and tests

ca8a20b

Signed-off-by: Labanya Mukhopadhyay <[email protected]>

Merge branch 'main' into lmukhopadhyay-SNOW-1918055-groupby-agg-fix-e…

5ebd77c

…rror

update quantile xfail

ba43142

Signed-off-by: Labanya Mukhopadhyay <[email protected]>

sfc-gh-jjiao reviewed Mar 19, 2025

View reviewed changes

update unit tests

d97987a

Signed-off-by: Labanya Mukhopadhyay <[email protected]>

sfc-gh-lmukhopadhyay added 3 commits March 20, 2025 16:53

cleanup

9955b7d

Signed-off-by: Labanya Mukhopadhyay <[email protected]>

update docstring

b124a2b

Signed-off-by: Labanya Mukhopadhyay <[email protected]>

Merge branch 'main' into lmukhopadhyay-SNOW-1918055-groupby-agg-fix-e…

f20fb7d

…rror

sfc-gh-lmukhopadhyay requested a review from sfc-gh-joshi March 21, 2025 19:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SNOW-1918055: Update agg error for unsupported aggregation functions #3133

SNOW-1918055: Update agg error for unsupported aggregation functions #3133

sfc-gh-lmukhopadhyay commented Mar 7, 2025 •

edited

Loading

sfc-gh-joshi Mar 8, 2025

sfc-gh-joshi Mar 8, 2025

sfc-gh-lmukhopadhyay Mar 8, 2025

sfc-gh-joshi Mar 8, 2025

sfc-gh-joshi Mar 8, 2025

sfc-gh-lmukhopadhyay Mar 8, 2025

sfc-gh-jjiao Mar 8, 2025

sfc-gh-jjiao Mar 8, 2025

sfc-gh-jjiao Mar 8, 2025

sfc-gh-jjiao Mar 8, 2025 •

edited

Loading

sfc-gh-jjiao left a comment •

edited

Loading

sfc-gh-jjiao Mar 19, 2025

sfc-gh-snowflakedb-snyk-sa commented Mar 20, 2025 •

edited

Loading

SNOW-1918055: Update agg error for unsupported aggregation functions #3133

Are you sure you want to change the base?

SNOW-1918055: Update agg error for unsupported aggregation functions #3133

Conversation

sfc-gh-lmukhopadhyay commented Mar 7, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfc-gh-jjiao Mar 8, 2025 • edited Loading

Choose a reason for hiding this comment

sfc-gh-jjiao left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfc-gh-snowflakedb-snyk-sa commented Mar 20, 2025 • edited Loading

🎉 Snyk checks have passed. No issues have been found so far.

sfc-gh-lmukhopadhyay commented Mar 7, 2025 •

edited

Loading

sfc-gh-jjiao Mar 8, 2025 •

edited

Loading

sfc-gh-jjiao left a comment •

edited

Loading

sfc-gh-snowflakedb-snyk-sa commented Mar 20, 2025 •

edited

Loading