[WIP][SPARK-50892][SQL]Add UnionLoopExec, physical operator for recursion, to perform execution of recursive queries #49955
Conversation
Pajaraja
changed the title
…mizer/InlineCTE.scala.rej
| @@ -4421,6 +4421,12 @@ | |||
| ], | |||
| "sqlState" : "38000" | |||
| }, | |||
| "RECURSION_LEVEL_LIMIT_EXCEEDED" : { | |||
| "message" : [ | |||
| "Recursion level limit <levelLimit> reached but query has not exhausted, try increasing CTE_RECURSION_LEVEL_LIMIT" | |||
There was a problem hiding this comment.
shall we mention the config name here? Otherwise this error message is not actionable.
| @@ -1031,6 +1040,9 @@ object ColumnPruning extends Rule[LogicalPlan] { | |||
| } else { | |||
| p | |||
| } | |||
| // TODO: Pruning `UnionLoop`s needs to take into account both the outer `Project` and the inner | |||
| // `UnionLoopRef` nodes. | |||
There was a problem hiding this comment.
This might hurt performance a lot. Let's figure it out
| // propagated to UnionLoopExec. | ||
| // Limit node is constructed by placing GlobalLimit over LocalLimit (look at Limit apply method) | ||
| // that is the reason why we match it this way. | ||
| case g @ GlobalLimit(IntegerLiteral(limit), l @ LocalLimit(_, p @ Project(_, ul: UnionLoop))) => |
There was a problem hiding this comment.
nit: case Limit(...) matches a GlobalLimit wrapping a LocalLimit with the same value, we should use it instead.
| @@ -34,7 +34,9 @@ import org.apache.spark.sql.internal.SQLConf | |||
| * @param id The id of the loop, inherited from [[CTERelationDef]] within which the Union lived. | |||
| * @param anchor The plan of the initial element of the loop. | |||
| * @param recursion The plan that describes the recursion with an [[UnionLoopRef]] node. | |||
| * @param limit An optional limit that can be pushed down to the node to stop the loop earlier. | |||
There was a problem hiding this comment.
I think the previous comment is good enough to describe this parameter.
| * Note here: limit can be applied in the main query calling the recursive CTE, and not | ||
| * inside the recursive term of recursive CTE. | ||
| */ | ||
| case class UnionLoopExec( |
There was a problem hiding this comment.
nit: let's move it to a new file
| override val output: Seq[Attribute], | ||
| limit: Option[Int] = None) extends LeafExecNode { | ||
|
|
||
| override def innerChildren: Seq[QueryPlan[_]] = Seq(anchor, recursion) |
There was a problem hiding this comment.
why do they have to be inner children?
| plan: LogicalPlan, currentLimit: Int) = { | ||
| // In case limit is defined, we create a (global) limit node above the plan and execute | ||
| // the newly created plan. | ||
| // Note here: global limit requires coordination (shuffle) between partitions. |
There was a problem hiding this comment.
Then it's better to use local limit? It's just a best effort to reduce the generated records.
| var currentLevel = 1 | ||
|
|
||
| // Main loop for obtaining the result of the recursive query. | ||
| while (prevCount > 0 && (limit.isEmpty || currentLimit > 0)) { |
There was a problem hiding this comment.
one idea: the key here is to get the row count of the current iteration, so that we can decide if we should keep iterating or not. The shuffle is only to save recomputing of the query. But for very simple queries (e.g. local scan with simple filter/project), shuffle is probably more expensive than recomputing. We should detect such case and avoid shuffle.
What changes were proposed in this pull request?
This PR introduces UnionLoopExec, physical operator for recursion: UnionLoop is converted to UnionLoopExec during execution.
For now only UNION ALL case is supported.
The execution is performed by iteratively substituting UnionLoopRef with the plan obtained in previous step, as long as we are still generating new elements
In addition, small changes to Optimizer.scala are added to push down the Limit to UnionLoopExec in case it is present in the query.
Why are the changes needed?
Support for recursive CTE.
Does this PR introduce any user-facing change?
No.
#How was this patch tested?
Added golden files tests for various use cases of recursive CTEs: cte-recursion.sql and with.sql (tests are run with SQLQueryTestSuite). The outputs of the tests are checked with the outputs of the same (or syntactically slightly adapted) queries in Snowflake and PostgreSQL engines.
Added two tests with parameterized identifier with recursive CTEs to ParametersSuite.
Was this patch authored or co-authored using generative AI tooling?
No.