[SPARK-51016][SQL] Fix for incorrect results on retry for Left Outer Join with indeterministic join keys #49708
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Added a new lazy boolean exprValHasIndeterministicCharacter in the Expression class, to indicate whether the expression's evaluation result was obtained using some indeterministic calculation.
The main difference between existing deterministic boolean and this is , that the former determines if same expression is evaluated more than once in a given iteration would its value change or not. Also on that basis, any leaf expression would always be deterministic.
But the latter indicates if an expression's evaluation used/uses any non deterministic component. So here even the leaf expression ( like Attribute) can have non deterministic character flag as true, if its pointing to some result obtained via indeterministic calculation.
Apart from that code of ShuffleDependency is augmented to indicate if the hashing expression is using any non deterministic component.
Why are the changes needed?
The changes are needed to ensure that in case of non deterministic join keys, the whole Shuffle stage is re-executed instead of just the failing task of the stage. This is achieved by fixing the bug in the Rdd.outputDeterministicLevel which now takes into account the indeterministic nature of ShuffleDependency
Does this PR introduce any user-facing change?
No
How was this patch tested?
Added some basic tests. Will be adding more tests to validate the functional bug of incorrect results.
Was this patch authored or co-authored using generative AI tooling?
No