Support casting date literal to timestamp #3831

yuancu · 2025-06-30T11:19:55Z

Description

Before this PR, only the following casts are supported

	date	time	timestamp
date str	✅	❌	❌
time str	❌	✅	❌
timestamp str	✅	✅	✅

With this PR:

	date	time	timestamp
date str	✅	❌	✅
time str	❌	✅	❌
timestamp str	✅	✅	✅

The castings that remain impossible are in place to avoid swallowing errors silently.

An example case it solves
... | where date_time > '1950-10-11' | fields date_time

Before this PR

calcite

EnumerableCalc(expr#0=[{inputs}], expr#1=['1950-10-11':VARCHAR], expr#2=[TIMESTAMP($t1)], expr#3=[>($t0, $t2)], date_time=[$t0], $condition=[$t3])
  CalciteEnumerableIndexScan(table=[[OpenSearch, dates]], PushDownContext=[[PROJECT->[date_time]], OpenSearchRequestBuilder(sourceBuilder={"from":0,"timeout":"1m","_source":{"includes":["date_time"],"excludes":[]}}, requestedTotalSize=2147483647, pageSize=null, startFrom=0)])

Run into follow exception when executed

{
  "error": {
    "reason": "There was internal problem at backend",
    "details": "java.sql.SQLException: exception while executing query: timestamp:1950-10-11 in unsupported format, please use 'yyyy-MM-dd HH:mm:ss[.SSSSSSSSS]'",
    "type": "RuntimeException"
  },
  "status": 500
}

v2

{
 "error": {
   "reason": "Invalid Query",
   "details": "timestamp:1950-10-11 in unsupported format, please use 'yyyy-MM-dd HH:mm:ss[.SSSSSSSSS]'",
   "type": "SemanticCheckException"
 },
 "status": 400
}

After this PR

calcite

CalciteEnumerableIndexScan(table=[[OpenSearch, dates]], PushDownContext=[[PROJECT->[date_time], FILTER->>($0, '1950-10-11 00:00:00')], OpenSearchRequestBuilder(sourceBuilder={"from":0,"timeout":"1m","query":{"range":{"date_time":{"from":"1950-10-11T00:00:00.000Z","to":null,"include_lower":false,"include_upper":true,"boost":1.0}}},"_source":{"includes":["date_time"],"excludes":[]},"sort":[{"_doc":{"order":"asc"}}]}, requestedTotalSize=2147483647, pageSize=null, startFrom=0)])

v2

{
    "root": {
        "name": "ProjectOperator",
        "description": {
            "fields": "[date_time]"
        },
        "children": [
            {
                "name": "OpenSearchIndexScan",
                "description": {
                    "request": "OpenSearchQueryRequest(indexName=dates, sourceBuilder={\"from\":0,\"size\":10000,\"timeout\":\"1m\",\"query\":{\"range\":{\"date_time\":{\"from\":\"1950-10-11T00:00:00.000Z\",\"to\":null,\"include_lower\":false,\"include_upper\":true,\"boost\":1.0}}},\"_source\":{\"includes\":[\"date_time\"],\"excludes\":[]},\"sort\":[{\"_doc\":{\"order\":\"asc\"}}]}, needClean=true, searchDone=false, pitId=*, cursorKeepAlive=1m, searchAfter=null, searchResponse=null)"
                },
                "children": []
            }
        ]
    }
}

Callout: The exception thrown when failed to parse a date/time string is changed to ExpressionEvaluationException (from SemanticCheckException). Users relying on the exception type will be affected.

Implementation Notes

With Calcite, the original way was just to make literals with types like EXPR_TIMESTAMP. The values inside are not validated: https://github.com/apache/calcite/blob/a6973bde8107a7c2a605a82c779a64fe28bdb5d1/core/src/main/java/org/apache/calcite/rex/RexBuilder.java#L835
Now we explicitly call TIMESTAMP, TIME, or DATE functions to cast string values.

Concerns

Calling UDFs for casting may render it impossible to e.g. pushdown
Calling UDFs for casting may make the execution slower

Related Issues

Resolves #3728

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

date str -> timestamp timestamp str -> date timestamp str -> time Signed-off-by: Yuanchun Shen <[email protected]>

Signed-off-by: Yuanchun Shen <[email protected]>

LantaoJin · 2025-07-01T04:53:48Z

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalcitePPLCastFunctionIT.java

+        rows("New York"),
+        rows("Ontario"),
+        rows("Quebec"),
+        rows((Object) null),


Good job to support passing null in verifyDataRows

LantaoJin · 2025-07-01T04:55:42Z

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalcitePPLCastFunctionIT.java

+                        "source=%s | eval a = cast('09:07:42' as DATE) | fields a",
+                        TEST_INDEX_DATE_FORMATS)));
+
+    verifyErrorMessageContains(t, "date:09:07:42 in unsupported format, please use 'yyyy-MM-dd'");


seems we are not aligning with Spark, so this is the behviour of v2 or mysql?

This was the behavior of v2.

In MySQL, cast a time string to date will return NULL; in some other databases like Postgres, it raises an error.

Signed-off-by: Yuanchun Shen <[email protected]>

dai-chen

Calling UDFs for casting may render it impossible to e.g. pushdown

Is this still a concern if expression pushdown is supported?

Meanwhile in V2, even though expression pushdown is available, I recall there is performance degradation when CAST is involved—queries may fall back from DSL search to expression script evaluation. Probably we can double check and see if there’s a possible workaround in V3.

dai-chen · 2025-07-03T18:53:58Z

There happens to be a new issue #3842 related.

Signed-off-by: Yuanchun Shen <[email protected]>

yuancu · 2025-07-04T07:13:33Z

Is this still a concern if expression pushdown is supported?

It should not be a issue when script pushdown is supported. Besides, if it is a literal that has to be cast, it can be pushed-down as a range query with #3798. I added an example with physical plan in the PR description to demonstrate this.

Probably we can double check and see if there’s a possible workaround in V3.

#3798 circumvents script push-down by converting the string literal to another DSL-recognizable literal then pushing down.

qianheng-aws · 2025-07-09T06:54:18Z

core/src/main/java/org/opensearch/sql/data/model/ExprTimestampValue.java

-              .toInstant();
-    } catch (DateTimeParseException e) {
+          DateTimeParser.parseDateOrTimestamp(timestamp).atZone(ZoneOffset.UTC).toInstant();
+    } catch (SemanticCheckException e) {


I think SemanticCheckException is inappropriate here. This issue isn't related to semantic at all. Should be something like IllegalArgumentException.

Changed to ExpressionEvaluationException when failed parsing datetime literals/strings.

qianheng-aws · 2025-07-09T07:24:20Z

core/src/main/java/org/opensearch/sql/data/model/ExprTimestampValue.java

-              .atZone(ZoneOffset.UTC)
-              .toInstant();
-    } catch (DateTimeParseException e) {
+          DateTimeParser.parseDateOrTimestamp(timestamp).atZone(ZoneOffset.UTC).toInstant();


Shall we support cast time str -> timestamp in the future? Maybe replace DATE_TIME_FORMATTER_VARIABLE_NANOS with DATE_TIME_FORMATTER_VARIABLE_NANOS_OPTIONAL directly if that's the case.

And I wonder if using DateTimeFormatter with multiple patterns should be more efficient than DateTimeParser.parseDateOrTimestamp which try to parseDate and fallback to parseTimeStamp when throwing exception.

Fixed by replacing parse*Or* calls with new DateTimeFormatters

qianheng-aws · 2025-07-09T07:41:08Z

Not related to this issue, I found there are some inefficient implementation for the date related functions, like branch selection at runtime for TIMESTAMP which should be determined in compiling phase based on the type of args, and redundant transformation between String and ExprStringValue for DATE. I don't check all of them.

Signed-off-by: Yuanchun Shen <[email protected]>

…ate/time strings Signed-off-by: Yuanchun Shen <[email protected]>

yuancu · 2025-07-10T07:35:31Z

One problem I discovered during testing:

Errors thrown during executing PPL with Calcite will often result in 500 error with Calcite.

For example query source=t0001 | where @timestamp > '12:00' has the following response with Calcite:

{
  "error": {
    "reason": "There was internal problem at backend",
    "details": "java.sql.SQLException: exception while executing query: timestamp:12:00 in unsupported format, please use 'yyyy-MM-dd HH:mm:ss[.SSSSSSSSS]'",
    "type": "RuntimeException"
  },
  "status": 500
}

But in v2, it's:

{
  "error": {
    "reason": "Invalid Query",
    "details": "timestamp:12:00 in unsupported format, please use 'yyyy-MM-dd HH:mm:ss[.SSSSSSSSS]'",
    "type": "ExpressionEvaluationException"
  },
  "status": 400
}

This is because exceptions thrown during statement.executeQuery() will be wrapped with SQLException (see AvaticaConnection.java#L579), which is then captured by us and wrapped with a RuntimeException:

  try (PreparedStatement statement = OpenSearchRelRunners.run(context, rel)) {
    ResultSet result = statement.executeQuery();
    buildResultSet(result, rel.getRowType(), context.querySizeLimit, listener);
  } catch (SQLException e) {
    throw new RuntimeException(e);
  }

A simple fix: retrieving the wrapped exception from the SQLException and re-throw it. Should I implement this?

example stacktrace

[2025-07-10T14:28:48,782][ERROR][o.o.s.p.r.RestPPLQueryAction] [7cf34de73b85] Error happened during query handling
java.lang.RuntimeException: java.sql.SQLException: exception while executing query: timestamp:12:00 in unsupported format, please use 'yyyy-MM-dd HH:mm:ss[.SSSSSSSSS]'
	at org.opensearch.sql.opensearch.executor.OpenSearchExecutionEngine.lambda$execute$6(OpenSearchExecutionEngine.java:203) ~[?:?]
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
	at org.opensearch.sql.opensearch.executor.OpenSearchExecutionEngine.lambda$execute$7(OpenSearchExecutionEngine.java:196) ~[?:?]
	at org.opensearch.sql.opensearch.client.OpenSearchNodeClient.schedule(OpenSearchNodeClient.java:193) ~[?:?]
	at org.opensearch.sql.opensearch.executor.OpenSearchExecutionEngine.execute(OpenSearchExecutionEngine.java:194) ~[?:?]
	at org.opensearch.sql.executor.QueryService.lambda$executeWithCalcite$0(QueryService.java:107) ~[?:?]
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
	at org.opensearch.sql.executor.QueryService.executeWithCalcite(QueryService.java:96) ~[?:?]
	at org.opensearch.sql.executor.QueryService.execute(QueryService.java:72) ~[?:?]
	at org.opensearch.sql.executor.execution.QueryPlan.execute(QueryPlan.java:69) ~[?:?]
	at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$submit$0(OpenSearchQueryManager.java:31) ~[?:?]
	at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$withCurrentContext$1(OpenSearchQueryManager.java:45) ~[?:?]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:916) ~[opensearch-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.base/java.lang.Thread.run(Thread.java:1575) [?:?]
Caused by: java.sql.SQLException: exception while executing query: timestamp:12:00 in unsupported format, please use 'yyyy-MM-dd HH:mm:ss[.SSSSSSSSS]'
	at org.apache.calcite.avatica.Helper.createException(Helper.java:56) ~[?:?]
	at org.apache.calcite.avatica.Helper.createException(Helper.java:41) ~[?:?]
	at org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:579) ~[?:?]
	at org.apache.calcite.avatica.AvaticaPreparedStatement.executeQuery(AvaticaPreparedStatement.java:137) ~[?:?]
	at org.opensearch.sql.opensearch.executor.OpenSearchExecutionEngine.lambda$execute$6(OpenSearchExecutionEngine.java:200) ~[?:?]
	... 15 more
Caused by: org.opensearch.sql.exception.ExpressionEvaluationException: timestamp:12:00 in unsupported format, please use 'yyyy-MM-dd HH:mm:ss[.SSSSSSSSS]'
	at org.opensearch.sql.data.model.ExprTimestampValue.<init>(ExprTimestampValue.java:43) ~[?:?]
	at org.opensearch.sql.calcite.utils.datetime.DateTimeConversionUtils.convertToTimestampValue(DateTimeConversionUtils.java:62) ~[?:?]
	at org.opensearch.sql.expression.function.udf.datetime.TimestampFunction.timestamp(TimestampFunction.java:75) ~[?:?]
	at Baz$1$1.moveNext(Unknown Source) ~[?:?]
	at org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.<init>(Linq4j.java:666) ~[?:?]
	at org.apache.calcite.linq4j.Linq4j.enumeratorIterator(Linq4j.java:99) ~[?:?]
	at org.apache.calcite.linq4j.AbstractEnumerable.iterator(AbstractEnumerable.java:33) ~[?:?]
	at org.apache.calcite.avatica.MetaImpl.createCursor(MetaImpl.java:83) ~[?:?]
	at org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:186) ~[?:?]
	at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:64) ~[?:?]
	at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:43) ~[?:?]
	at org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:575) ~[?:?]
	at org.apache.calcite.avatica.AvaticaPreparedStatement.executeQuery(AvaticaPreparedStatement.java:137) ~[?:?]
	at org.opensearch.sql.opensearch.executor.OpenSearchExecutionEngine.lambda$execute$6(OpenSearchExecutionEngine.java:200) ~[?:?]
	... 15 more

yuancu · 2025-07-10T07:47:59Z

Not related to this issue, I found there are some inefficient implementation for the date related functions, like branch selection at runtime for TIMESTAMP which should be determined in compiling phase based on the type of args, and redundant transformation between String and ExprStringValue for DATE. I don't check all of them.

I spotted a few for functions under java/org/opensearch/sql/expression/function/udf/datetime. I can fix them in another new PR.

Besides, almost all datetime functions convert inputs to ExprValues to reuse v2's implementation. I can eliminate them to enhance efficiency if the anticipated improvement is significant.

yuancu added 3 commits June 30, 2025 17:59

Support following datetime casts in v2:

7a120db

date str -> timestamp timestamp str -> date timestamp str -> time Signed-off-by: Yuanchun Shen <[email protected]>

Invoke datetime UDF to cast date/time/timestamp with Calcite

0e98650

Signed-off-by: Yuanchun Shen <[email protected]>

Test cast to date/time/timestamp

bc2c9e2

Signed-off-by: Yuanchun Shen <[email protected]>

yuancu requested review from ps48, kavithacm, derek-ho, joshuali925, dai-chen, YANG-DB, mengweieric, Swiddis, penghuo, seankao-az, MaxKsyunz, Yury-Fridlyand, anirudha, forestmvey, acarbonetto, GumpacG, ykmr1224, LantaoJin, noCharger and qianheng-aws as code owners June 30, 2025 11:19

yuancu marked this pull request as draft June 30, 2025 11:27

Use stricter date and time formatters

3c42fe1

Signed-off-by: Yuanchun Shen <[email protected]>

yuancu changed the title ~~Support casting date string to timestamp~~ [BUG] Support casting date string to timestamp Jul 1, 2025

yuancu changed the title ~~[BUG] Support casting date string to timestamp~~ Support casting date string to timestamp Jul 1, 2025

LantaoJin added bug Something isn't working calcite calcite migration releated labels Jul 1, 2025

LantaoJin reviewed Jul 1, 2025

View reviewed changes

yuancu added 2 commits July 1, 2025 16:18

Add a timestamp formatter for instantiating ExprTimestampValue

8d1d530

Signed-off-by: Yuanchun Shen <[email protected]>

Merge remote-tracking branch 'origin/main' into cast-datetime

669b461

Signed-off-by: Yuanchun Shen <[email protected]>

yuancu marked this pull request as ready for review July 1, 2025 09:34

dai-chen reviewed Jul 1, 2025

View reviewed changes

yuancu changed the title ~~Support casting date string to timestamp~~ Support casting date literal to timestamp Jul 2, 2025

Merge remote-tracking branch 'origin/main' into cast-datetime

28d2461

Signed-off-by: Yuanchun Shen <[email protected]>

qianheng-aws reviewed Jul 9, 2025

View reviewed changes

LantaoJin mentioned this pull request Jul 10, 2025

Add ClickBench IT Suite #3860

Merged

7 tasks

yuancu added 2 commits July 10, 2025 14:02

Simplify date/time expr constructors & parsers

f8f01f4

Signed-off-by: Yuanchun Shen <[email protected]>

Change to ExpressionEvaluationException when fail parsing malformat d…

bd9cea2

…ate/time strings Signed-off-by: Yuanchun Shen <[email protected]>

yuancu requested review from LantaoJin and qianheng-aws July 10, 2025 08:36

qianheng-aws approved these changes Jul 15, 2025

View reviewed changes

LantaoJin approved these changes Jul 22, 2025

View reviewed changes

LantaoJin merged commit 30aba65 into opensearch-project:main Jul 22, 2025
24 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Support casting date literal to timestamp #3831

Support casting date literal to timestamp #3831

yuancu commented Jun 30, 2025 •

edited

Loading

Uh oh!

LantaoJin Jul 1, 2025

Uh oh!

LantaoJin Jul 1, 2025

Uh oh!

yuancu Jul 1, 2025

Uh oh!

dai-chen left a comment

Uh oh!

dai-chen commented Jul 3, 2025

Uh oh!

yuancu commented Jul 4, 2025 •

edited

Loading

Uh oh!

qianheng-aws Jul 9, 2025

Uh oh!

yuancu Jul 10, 2025

Uh oh!

qianheng-aws Jul 9, 2025

Uh oh!

yuancu Jul 10, 2025

Uh oh!

qianheng-aws commented Jul 9, 2025

Uh oh!

yuancu commented Jul 10, 2025 •

edited

Loading

Uh oh!

yuancu commented Jul 10, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Support casting date literal to timestamp #3831

Support casting date literal to timestamp #3831

Conversation

yuancu commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

LantaoJin Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

LantaoJin Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

yuancu Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

dai-chen left a comment

Choose a reason for hiding this comment

Uh oh!

dai-chen commented Jul 3, 2025

Uh oh!

yuancu commented Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qianheng-aws Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

yuancu Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

qianheng-aws Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

yuancu Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

qianheng-aws commented Jul 9, 2025

Uh oh!

yuancu commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yuancu commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yuancu commented Jun 30, 2025 •

edited

Loading

yuancu commented Jul 4, 2025 •

edited

Loading

yuancu commented Jul 10, 2025 •

edited

Loading

yuancu commented Jul 10, 2025 •

edited

Loading