Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-51016][SQL] Fix for incorrect results on retry for Left Outer Join with indeterministic join keys #49708

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

ahshahid
Copy link

What changes were proposed in this pull request?

Added a new lazy boolean exprValHasIndeterministicCharacter in the Expression class, to indicate whether the expression's evaluation result was obtained using some indeterministic calculation.
The main difference between existing deterministic boolean and this is , that the former determines if same expression is evaluated more than once in a given iteration would its value change or not. Also on that basis, any leaf expression would always be deterministic.
But the latter indicates if an expression's evaluation used/uses any non deterministic component. So here even the leaf expression ( like Attribute) can have non deterministic character flag as true, if its pointing to some result obtained via indeterministic calculation.

Apart from that code of ShuffleDependency is augmented to indicate if the hashing expression is using any non deterministic component.

Why are the changes needed?

The changes are needed to ensure that in case of non deterministic join keys, the whole Shuffle stage is re-executed instead of just the failing task of the stage. This is achieved by fixing the bug in the Rdd.outputDeterministicLevel which now takes into account the indeterministic nature of ShuffleDependency

Does this PR introduce any user-facing change?

No

How was this patch tested?

Added some basic tests. Will be adding more tests to validate the functional bug of incorrect results.

Was this patch authored or co-authored using generative AI tooling?

No

@ahshahid ahshahid changed the title [WIP][SPARK-51016][SQL] Fix for incorrect results on retry for Left Outer Join with indeterministic join keys [SPARK-51016][SQL] Fix for incorrect results on retry for Left Outer Join with indeterministic join keys Jan 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant