Support filter script push down for calcite #3812

qianheng-aws · 2025-06-23T10:30:15Z

Description

Implement filter script push down for calcite by registering a new script language for java code string. This new script language can only be invoked by Node client since it requires a option of engine_type which is not allowed in Rest client.

Related Issues

Resolves #3379

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Heng Qian <[email protected]>

...ch/src/main/java/org/opensearch/sql/opensearch/storage/script/PPLCompoundedScriptEngine.java

opensearch/src/main/java/org/opensearch/sql/opensearch/request/PredicateAnalyzer.java

opensearch/src/main/java/org/opensearch/sql/opensearch/storage/script/CalciteScriptEngine.java

penghuo · 2025-06-23T21:52:48Z

This new script language can only be invoked by Node client since it requires a option of engine_type which is not allowed in Rest client.

This is a side project that runs the plugin as a standalone CLI and leverages DSL queries to interact with the OpenSearch cluster.
Is there any workaround, can we encode the engine_type with script_code?

LantaoJin · 2025-06-24T00:27:31Z

This new script language can only be invoked by Node client since it requires a option of engine_type which is not allowed in Rest client.

This is a side project that runs the plugin as a standalone CLI and leverages DSL queries to interact with the OpenSearch cluster.

Is there any workaround, can we encode the engine_type with script_code?

Yes, we can check the "calcite" related package information in script code to determine the engine type, rather than encode additional information.

qianheng-aws · 2025-06-24T02:40:15Z

This new script language can only be invoked by Node client since it requires a option of engine_type which is not allowed in Rest client.

This is a side project that runs the plugin as a standalone CLI and leverages DSL queries to interact with the OpenSearch cluster. Is there any workaround, can we encode the engine_type with script_code?

Add option incompatible with rest client is also for security consideration. Otherwise, users can use Rest client to submit java code script without limitation.

Signed-off-by: Heng Qian <[email protected]>

…shDown

songkant-aws · 2025-06-24T03:34:04Z

opensearch/src/main/java/org/opensearch/sql/opensearch/storage/script/CalciteScriptEngine.java

+  public static Function1<DataContext, Object[]> compile(String code, Object reason) {
+    try {
+      ClassBodyEvaluator cbe = new ClassBodyEvaluator();
+      cbe.setClassName("Reducer");
+      cbe.setExtendedClass(Utilities.class);
+      cbe.setImplementedInterfaces(new Class[] {Function1.class, Serializable.class});
+      cbe.setParentClassLoader(RexExecutable.class.getClassLoader());
+      cbe.cook(new Scanner((String) null, new StringReader(code)));
+      Class c = cbe.getClazz();
+      Constructor<Function1<DataContext, Object[]>> constructor = c.getConstructor();
+      return (Function1) constructor.newInstance();
+    } catch (IOException
+        | InstantiationException
+        | IllegalAccessException
+        | InvocationTargetException
+        | NoSuchMethodException
+        | CompileException var5) {
+      Exception e = var5;
+      throw new RuntimeException("While compiling " + reason, e);
+    }
+  }


Minor: Since we already get the code string, we can new RexExecutable and call getFunction() to avoid this copied method?

Signed-off-by: Heng Qian <[email protected]>

penghuo · 2025-06-24T22:57:49Z

Add option incompatible with rest client is also for security consideration. Otherwise, users can use Rest client to submit java code script without limitation.

Current implementation depend on Script.java in Core deny any options, if it is not true in future, user can run ANY java code.

if (options.size() > 1 || options.size() == 1 && options.get(CONTENT_TYPE_OPTION) == null) {
    options.remove(CONTENT_TYPE_OPTION);

    throw new IllegalArgumentException("illegal compiler options [" + options + "] specified");
}

Are there any workarounds we can consider? Instead of using raw Java code, can we leverage LINQ4j Enumerable or V2 Expression to limit user input to only allowlisted expressions and reduce the risk?

dai-chen · 2025-06-26T04:29:32Z

opensearch/src/main/java/org/opensearch/sql/opensearch/request/PredicateAnalyzer.java

+      // Compile code when creating to detect exception as early as possible
+      JavaTypeFactoryImpl typeFactory =
+          new JavaTypeFactoryImpl(rexBuilder.getTypeFactory().getTypeSystem());
+      RexToLixTranslator.InputGetter getter =
+          new ScriptInputGetter(typeFactory, rowType, fieldTypes);
+      this.code = CalciteScriptEngine.translate(rexBuilder, List.of(rexNode), getter, rowType);


I'm thinking is it possible we delay this compilation to Java until compile() in ScriptEngine? In this way, we serialize and deserialize RexCall instead, similar as V2 expression pushdown? Just some thoughts.

RexNode is unserializable. We've though of similar method as mentioned in the option2 here: #3379 (comment).

@songkant-aws is working on option2 but seems it's not feasible.

Leveraging Kryo serialization for classes without zero-argument constructor requires explicitly registering specific class serializer. That means we may need to spend more efforts on creating separate customized Kryo serializers for Calcite Expression classes/Enumerable classes.

If we go this way, I'm not confident yet to ensure all kinds of expressions can be correctly serialized.

RexNode is unserializable.

You mean it's not by JDK or Kryo right? Does other serializer work, or even Json's?

Do we know how many classes related to RexCall we need to serialize? Just thinking if that's not many and if we can ignore fields not needed by code-gen, can we have customize serializer?

@qianheng-aws I checked the option 2 you posted. I feel it's different?

What I was thinking is:

Coordinator: RexNode -> serialized code

Worker: Code -> RexNode -> Linq4j expression -> Interpret or compile to Java

In this way, the script engine can be bounded to allowlisted RexNode.

@songkant-aws Not sure if you've explored the idea above. If so, could you share your PoC branch if any? I can do more test from my side. Thanks!

@dai-chen For the sake of convenience, I pulled Heng's code in my local and add my Kryo serialization prototype code on top of it. And then pushed the code to my personal branch: https://github.com/songkant-aws/sql/tree/kryo-serialization

Basically, the logic is similar. I just feel adding additional serialization logic for third party classes requires too much effort.

Do we know how many classes related to RexCall we need to serialize?

The common RexNode we need may not be too many. Like RexCall, RexLiteral, RexInputRef, RexLambda, RexLambdaRef... I think the biggest blocker is the inner fields of them. Like RexCall, it has plenty of SqlOperators implemented by Calcite or us.

@songkant-aws @qianheng-aws Got it. Will have a look and also see if other options on my side. Thanks!

LantaoJin · 2025-07-01T02:02:32Z

opensearch/src/main/java/org/opensearch/sql/opensearch/request/PredicateAnalyzer.java

+        throw new UnsupportedScriptException(
+            "DSL will evaluate both branches of OR with isNUll, prevent push-down to avoid NPE");


is it an existed issue in v2?

I think V2 won't have this issue. Firstly, it seems doesn't support function isEmpty and isBlank. And secondly, it won't have such issue if it includes the null handling logic in one expression.

We have such issue because we break isEmpty into 2 expressions(isNull or isEmpty, all from Calcite) and Calcite's isEmpty doesn't have null handling in its implementation.

LantaoJin · 2025-07-01T02:07:07Z

opensearch/src/main/java/org/opensearch/sql/opensearch/request/PredicateAnalyzer.java

+        throw new UnsupportedScriptException(
+            "ScriptQueryExpression requires a valid current time from hook, but it is not set");


non-blocker: can we add a UT for this branch?

LantaoJin · 2025-07-01T02:14:09Z

opensearch/src/main/java/org/opensearch/sql/opensearch/storage/script/CalciteScriptEngine.java

+      return switch (exprType) {
+        case INTEGER -> EnumUtils.convert(docValueExpr, Long.class);
+        case FLOAT -> EnumUtils.convert(docValueExpr, Double.class);
+        default -> docValueExpr;


can the exprType be a SHORT or BINARY? for doc value. Seems all numeric types will impact.

It imitates the same logic as v2. As said in the description, it only needs specific handling for Integer and Long.

sql/opensearch/src/main/java/org/opensearch/sql/opensearch/storage/script/core/ExpressionScript.java

Lines 139 to 143 in 870b82c

/**

* DocValue only support long and double so cast to integer and float if needed. The doc value

* must be Long and Double for expr type Long/Integer and Double/Float respectively. Otherwise

* there must be bugs in our engine that causes the mismatch.

*/

LantaoJin · 2025-07-01T02:18:59Z

opensearch/src/main/java/org/opensearch/sql/opensearch/storage/script/CalciteScriptEngine.java

+  }
+
+  /**
+   * This function is copied from Calcite RexExecutorImpl It's used to compile RexNode expression to


Seems you copied the decompiled code instead source code. Can you check it?:
java.util.Iterator var5, Expressions.declare(16, Expressions.methodDecl(1, etc.

LantaoJin · 2025-07-01T02:55:30Z

...h/src/main/java/org/opensearch/sql/opensearch/storage/script/filter/CalciteFilterScript.java

+ * Calcite script executor that executes the generated code on each document and determine if the
+ * document is supposed to be filtered out or not.
+ */
+@EqualsAndHashCode(callSuper = false)


should we use true here?

qianheng-aws · 2025-07-09T06:18:22Z

Mark draft currently for security consideration.

Support filter script push down for calcite

1894875

Signed-off-by: Heng Qian <[email protected]>

qianheng-aws requested review from ps48, kavithacm, derek-ho, joshuali925, dai-chen, YANG-DB, mengweieric, Swiddis, penghuo, seankao-az, MaxKsyunz, Yury-Fridlyand, anirudha, forestmvey, acarbonetto, GumpacG, ykmr1224, LantaoJin and noCharger as code owners June 23, 2025 10:30

Fix UT

13acaf5

Signed-off-by: Heng Qian <[email protected]>

penghuo reviewed Jun 23, 2025

View reviewed changes

qianheng-aws added 2 commits June 24, 2025 11:15

Change CompoundedScriptEngine name

b37dc92

Signed-off-by: Heng Qian <[email protected]>

Merge remote-tracking branch 'refs/remotes/origin/main' into ScriptPu…

1abf8aa

…shDown

songkant-aws reviewed Jun 24, 2025

View reviewed changes

Remove duplicated code after merging main

bc6ecac

Signed-off-by: Heng Qian <[email protected]>

dai-chen reviewed Jun 26, 2025

View reviewed changes

Swiddis added the enhancement New feature or request label Jun 26, 2025

dai-chen mentioned this pull request Jun 26, 2025

[FEATURE] Calcite Engine Framework: Pushdown scripts #3379

Open

LantaoJin reviewed Jul 1, 2025

View reviewed changes

qianheng-aws marked this pull request as draft July 9, 2025 06:17

songkant-aws mentioned this pull request Jul 10, 2025

Filter script pushdown with RelJson serialization in Calcite #3859

Open

7 tasks

		throw new UnsupportedScriptException(
		"DSL will evaluate both branches of OR with isNUll, prevent push-down to avoid NPE");

		throw new UnsupportedScriptException(
		"ScriptQueryExpression requires a valid current time from hook, but it is not set");

	/**
	* DocValue only support long and double so cast to integer and float if needed. The doc value
	* must be Long and Double for expr type Long/Integer and Double/Float respectively. Otherwise
	* there must be bugs in our engine that causes the mismatch.
	*/

Support filter script push down for calcite #3812

Are you sure you want to change the base?

Support filter script push down for calcite #3812

Uh oh!

Conversation

qianheng-aws commented Jun 23, 2025

Description

Related Issues

Check List

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

penghuo commented Jun 23, 2025

Uh oh!

LantaoJin commented Jun 24, 2025

Uh oh!

qianheng-aws commented Jun 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

penghuo commented Jun 24, 2025

Uh oh!

dai-chen Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

songkant-aws Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dai-chen Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dai-chen Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qianheng-aws commented Jul 9, 2025

Uh oh!

Uh oh!

dai-chen Jun 26, 2025 •

edited

Loading

songkant-aws Jun 26, 2025 •

edited

Loading

dai-chen Jun 26, 2025 •

edited

Loading

dai-chen Jun 26, 2025 •

edited

Loading