Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spark support only i32 indexed arrays while comet is trying to support both i32 and i64 #1114

Closed
SemyonSinchenko opened this issue Nov 23, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@SemyonSinchenko
Copy link
Member

What is the problem the feature request solves?

At the moment Comet is trying to support both List and LargeList for handling array functions:

But in the spark the maximal size of any array is equal to java.lang.Integer.MAX_VALUES - 15 and any attempt to create an array of the bigger size will throw an exception:

  def cannotCreateArrayWithElementsExceedLimitError(
      numElements: Long, additionalErrorMessage: String): SparkRuntimeException = {
    new SparkRuntimeException(
      errorClass = "_LEGACY_ERROR_TEMP_2176",
      messageParameters = Map(
        "numElements" -> numElements.toString(),
        "maxRoundedArrayLength"-> ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH.toString(),
        "additionalErrorMessage" -> additionalErrorMessage))
  }

Describe the potential solution

Refactoring of all the code in list.rs and drop the support of arrow LargeList. It will simplify an existing code and also simplify resolving of the #1042

Additional context

I can try to make a refactoring and check if tests are passed.

@SemyonSinchenko SemyonSinchenko added the enhancement New feature or request label Nov 23, 2024
@SemyonSinchenko SemyonSinchenko closed this as not planned Won't fix, can't repro, duplicate, stale Nov 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant