[SPARK-51087][PYTHON][CONNECT] Raise a warning when memory-profiler is not installed for memory profiling

[xinrong-meng]

What changes were proposed in this pull request?

Raise a warning when memory-profiler is not installed for memory profiling.

Why are the changes needed?

Better usability of PySpark UDF memory profiling.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing and manual tests as shown below:

>>> import memory_profiler  # when memory-profiler is not installed
Traceback (most recent call last):
...
ModuleNotFoundError: No module named 'memory_profiler'

>>> from pyspark.sql.functions import pandas_udf
>>> 
>>> df = spark.range(10)
>>> @pandas_udf("long")
... def add1(x):
...   return x + 1
... 
>>> added = df.select(add1("id"))
>>> 
>>> spark.conf.set("spark.sql.pyspark.udf.profiler", "memory")
>>> added.show()
+--------+                                                                      
|add1(id)|
+--------+
|       1|
|       2|
|       3|
|       4|
|       5|
|       6|
|       7|
|       8|
|       9|
|      10|
+--------+
>>> spark.profile.show(type="memory")
/Users/xinrong.meng/spark/python/pyspark/sql/profiler.py:141: UserWarning: Install the 'memory_profiler' library in the cluster to enable memory profiling
...
>>> spark.profile.dump(path='...', type="memory")
/Users/xinrong.meng/spark/python/pyspark/sql/profiler.py:225: UserWarning: Install the 'memory_profiler' library in the cluster to enable memory profiling
...

Was this patch authored or co-authored using generative AI tooling?

No.

[ueshin]

What if the server has memory-profiler, but the client doesn't, or vice versa?
This seems to check the client side only, but actually I don't think we need the library in the client side to show the result?

[xinrong-meng]

Good point, how do you like the current approach? @ueshin

[ueshin]

The worker side warning message won't be seen from the client.
How about showing warning message in the client side when:

• The client doesn't have the library installed
• There are no data to show when calling spark.profile.show or ..dump
  • because the server side may have the library and already measured.

to make sure the library is installed?

[xinrong-meng]

Thank you @ueshin ! Now it raises a warning when the client/driver does not have memory-profiler and there is no result profile

[xinrong-meng]

Merged to master and branch-4.0 thanks!