Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Refactor QueryPlanSerde to allow logic to be moved to individual classes per expression #1331

Merged
merged 8 commits into from
Jan 24, 2025

Conversation

andygrove
Copy link
Member

Which issue does this PR close?

Part of #1330

Rationale for this change

See #1330 for rationale.

What changes are included in this PR?

  • Refactor QueryPlanSerde to move inner methods of exprToProto to top-level methods to make them accessible from other places
  • Refactor ArrayRemove serde to demonstrate the benefit of this refactor

The ArrayRemove code in QueryPlanSerde now becomes:

case expr: ArrayRemove => CometArrayRemove.convert(expr, input, binding)

Later, we can move to a map of registered converters instead of a match statement for better extensibility. For example:

converters.get(expression.getClass).convert(expr, input, binding)

How are these changes tested?

There are no functional changes, so rely on existing tests.

@andygrove
Copy link
Member Author

@erenavsarogullari @NoeB @dharanad @jatin510 please let me know if you have any feedback on this approach

@andygrove
Copy link
Member Author

@parthchandra @mbutrovich fyi

Copy link
Contributor

@comphead comphead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @andygrove. it definitely improves the access and testability for methods which have been inner

import org.apache.comet.shims.CometExprShim

trait CometExpression {
def checkSupport(expr: Expression): Boolean
trait CometExpressionSerde {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is presumably not just for arrays, so perhaps should be in a common file?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, currently, there is only one implementation, which is why it is here at the moment, but this can move to a common class. Perhaps I should just move this into QueryPlanSerde source file

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I moved the trait and added some scaladocs.

withInfo(expr, "unsupported arguments for GetArrayStructFields", child)
None
}
case expr: ArrayRemove => CometArrayRemove.convert(expr, input, binding)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although there is a big diff, I suppose they are just code moving and this is the only change?

Copy link
Member Author

@andygrove andygrove Jan 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, there are no functional changes. I just moved the inner methods to the top-level and that required adding extra parameters in some cases (such as input and binding).

*/
def exprToProto(
expr: Expression,
input: Seq[Attribute],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason to switching to singular input from inputs?
The doc above still says inputs. Should we switch to plural inputs since it is Seq?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In main, we currently have an inconsistency where exprToProto uses input and exprToProtoInternal uses inputs.

  def exprToProto(
      expr: Expression,
      input: Seq[Attribute],
      binding: Boolean = true): Option[Expr] = {
def exprToProtoInternal(expr: Expression, inputs: Seq[Attribute]): Option[Expr] = {

I changed both to input for consistency, but I agree that inputs is better. I have changed it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kazuyukitanimura fyi, I fixed the above

Copy link
Contributor

@kazuyukitanimura kazuyukitanimura left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove

@andygrove andygrove merged commit a5b36b4 into apache:main Jan 24, 2025
74 checks passed
@andygrove andygrove deleted the qps-refactor branch January 24, 2025 21:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants