Different semantics of casting from int64 to timestamp between Comet and Spark

[viirya]

What is the problem the feature request solves?

In Spark, casting from LongType to Timestamp is to casting seconds to microseconds. arrow-rs's cast kernel basically simply reinterprets Int64Array as TimestampMicrosecondArray. So the scale is different, i.e., seconds v.s. microseconds.

We need to add this special case into Comet's cast kernel.

Describe the potential solution

No response

Additional context

No response

[andygrove]

This cast is currently disabled by default, but it would be good to implement this. I have added this to the 0.2.0 milestone.

[justahuman1]

I can take this

[andygrove]

Thanks @justahuman1!

A good place to start is org.apache.comet.expressions.CometCast#canCastFromLong. This currently returns false for casting to timestamp, so enabling it here is the first step.

Next, you can enable the following test in CometCastSuite:

  ignore("cast LongType to TimestampType") {
    // java.lang.ArithmeticException: long overflow
    castTest(generateLongs(), DataTypes.TimestampType)
  }

[parthchandra]

Also check CometExpressionSuite.test("cast timestamp and timestamp_ntz"). This reads timestamps as longs from a parquet file (which may store the values as either millis or micros).