[#1750] feat(remote merge): Support Spark.

[zhengchenyu]

What changes were proposed in this pull request?

Support spark Framework

Why are the changes needed?

#1750

Does this PR introduce any user-facing change?

No.

How was this patch tested?

unit test, integration test, real job in cluster.

[github-actions]

Test Results

 3 029 files  +18   3 029 suites  +18   6h 41m 32s ⏱️ + 1m 40s
 1 182 tests + 7   1 180 ✅ + 7   2 💤 ±0  0 ❌ ±0 
14 961 runs  +62  14 931 ✅ +62  30 💤 ±0  0 ❌ ±0 

Results for commit 0c89203. ± Comparison against base commit 4300f93.

♻️ This comment has been updated with latest results.

[zuston]

From my sight, this feature now can't be used in Spark SQL. Maybe RDD could use this.

[zuston]

Aha! . How about renaming the RemoteMergeSparkSQLTest . RM looks Yarn ResourceManager

[zhengchenyu]

For me, when I mention RM, the first thing that comes to my mind is Yarn ResourceManager. But I don't have any other good names.

[zhengchenyu]

Or I will rename RM to RMS or RS. In fact, merge sort is implemented on the server side, but there is no combine, and remote sort is fine.
If so, the previous code and documentation need to be modified. Maybe a new PR is needed.

[zhengchenyu]

From my sight, this feature now can't be used in Spark SQL. Maybe RDD could use this.

This test is based on draft pr apache/spark#50248.

[jerqi]

From my sight, this feature now can't be used in Spark SQL. Maybe RDD could use this.

This test is based on draft pr apache/spark#50248.

cc @LuciferYang

[jerqi]

From my sight, this feature now can't be used in Spark SQL. Maybe RDD could use this.

This test is based on draft pr apache/spark#50248.

This will break the code implement of Spark. You would better to insert a new logic plan represents the distribution and partitioning after shuffling. You only need to implement some optimization rules.

[zhengchenyu]

From my sight, this feature now can't be used in Spark SQL. Maybe RDD could use this.

This test is based on draft pr apache/spark#50248.

This will break the code implement of Spark. You would better to insert a new logic plan represents the distribution and partitioning after shuffling. You only need to implement some optimization rules.

Are you talking about changes to Spark? My initial idea was also to see if I could add a new rule. Maybe for map side, I could add new rules. But for reduce, adding a new SortExec is determined by determining whether distribution and partitioning match, which is not easy to do by adding a new Rule.
For the draft pr about changes to spark. It is only a draft to verify the feasibility of this proposal. There are still some code architectures that need to be refactored. For example, some partial aggregation in memory logic, add some logic to the rule.

[LuciferYang]

also cc @summaryzb

[LuciferYang]

From my sight, this feature now can't be used in Spark SQL. Maybe RDD could use this.

This test is based on draft pr apache/spark#50248.

Does this one depend on SPARK-51398 being merged first?

[jerqi]

From my sight, this feature now can't be used in Spark SQL. Maybe RDD could use this.

This test is based on draft pr apache/spark#50248.

This will break the code implement of Spark. You would better to insert a new logic plan represents the distribution and partitioning after shuffling. You only need to implement some optimization rules.

Are you talking about changes to Spark? My initial idea was also to see if I could add a new rule. Maybe for map side, I could add new rules. But for reduce, adding a new SortExec is determined by determining whether distribution and partitioning match, which is not easy to do by adding a new Rule. For the draft pr about changes to spark. It is only a draft to verify the feasibility of this proposal. There are still some code architectures that need to be refactored. For example, some partial aggregation in memory logic, add some logic to the rule.

Yes, Meta Cosco ever did similar things. You can see
apache/spark#32944
apache/spark#34702

cc @c21 Excuse me, sorry to bother you. Is it possible that we don't change the code of Spark and only add some rules to implement this feature? Could you give us some suggestion?