ESQL - Add K mandatory param for KNN function #129763

carlosdelest · 2025-06-20T11:28:25Z

Right now KNN function sets k as an option:

| WHERE KNN(vector, [0, 1, 2], {"k": 10})

This is a problem as it makes k optional. Setting a default value makes no sense until we use LIMIT for setting k (#129353).

Until we use LIMIT, this PR makes k a mandatory parameter for KNN function:

Users must explicitly set the value of K they use
K is a first class citizen in KNN. This makes sense as it will be the most tweaked option by users.

The new function format will be:

| WHERE KNN(vector, [0, 1, 2], 10)

In case users want to set num_candidates, they can do so via option:

| WHERE KNN(vector, [0, 1, 2], 10, {"num_candidates": 50})

After #129353 is done, we can make this parameter optional and check that a LIMIT can be used to set K on behalf of the user. Users will be able to override the default K by setting it explicitly.

This PR removes a test for the k option, which makes no sense now, and closes the following related issues for this test:

Closes #129447
Closes #129512

github-actions · 2025-06-20T11:28:34Z

🔍 Preview links for changed docs:

docs/reference/query-languages/esql/kibana/docs/functions/knn.md

🔔 The preview site may take up to 3 minutes to finish building. These links will become live once it completes.

carlosdelest · 2025-06-20T11:32:58Z

muted-tests.yml

@@ -511,9 +511,6 @@ tests:
 - class: org.elasticsearch.entitlement.runtime.policy.FileAccessTreeTests
  method: testWindowsAbsolutPathAccess
  issue: https://github.com/elastic/elasticsearch/issues/129168
- class: org.elasticsearch.xpack.esql.qa.multi_node.EsqlSpecIT


This test is removed in this PR as it's already being tested in all other tests

carlosdelest · 2025-06-20T11:34:50Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/knn-function.csv-spec

@@ -29,31 +29,12 @@ chartreuse | [127.0, 255.0, 0.0]
 // end::knn-function-result[]
 ;

-knnSearchWithKOption


Removed test - k is already added to all other tests

carlosdelest · 2025-06-20T11:35:11Z

...k/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/vector/Knn.java

@@ -100,14 +109,6 @@ public Knn(
                    description = "Floating point number used to decrease or increase the relevance scores of the query."
                        + "Defaults to 1.0."
                ),
-                @MapParam.MapParamEntry(


k is removed as an option

carlosdelest · 2025-06-20T11:36:07Z

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/VerifierTests.java

@@ -2172,24 +2173,25 @@ private void checkFullTextFunctionNullArgs(String functionInvocation, String arg
        );
    }

-    public void testFullTextFunctionsConstantQuery() throws Exception {


Renamed as we're checking other params not being null now

elasticsearchmachine · 2025-06-20T13:43:43Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

ioanatia

does what it says 👍

kderusso

LGTM, one question

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

…k-param

# Conflicts: # docs/reference/query-languages/esql/kibana/definition/functions/knn.json # docs/reference/query-languages/esql/kibana/docs/functions/knn.md # muted-tests.yml # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/EsqlFunctionRegistry.java

…o non-issue/knn-k-param # Conflicts: # docs/reference/query-languages/esql/kibana/definition/functions/knn.json # muted-tests.yml # server/src/main/java/org/elasticsearch/TransportVersions.java

…itten

carlosdelest · 2025-07-01T17:11:10Z

Hey @ioanatia @tteofili @kderusso I've removed the usage of TransportVersions by not serializing k parameter in a6faf49.

Once the query builder is created as part of the query rewriting, there is no need for storing k as it will be included in the query builder options, so we don't really need to serialize it to data nodes.

I think this is better for not dealing with transport versions, but LMK otherwise 👍

github-actions · 2025-07-02T07:06:05Z

🔍 Preview links for changed docs

docs/reference/query-languages/esql/kibana/docs/functions/knn.md

ioanatia · 2025-07-02T07:51:35Z

muted-tests.yml

@@ -236,6 +236,9 @@ tests:
 - class: org.elasticsearch.packaging.test.DockerTests
  method: test012SecurityCanBeDisabled
  issue: https://github.com/elastic/elasticsearch/issues/116636
+- class: org.elasticsearch.index.shard.StoreRecoveryTests


bad merge? same as the other ones that are added?

🤦 ouch. Sorry about that.

Opened #130523 to fix

ioanatia · 2025-07-02T07:59:15Z

...k/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/vector/Knn.java

+            return TypeResolution.TYPE_RESOLVED;
+        }
+
+        return isType(k(), dt -> dt == INTEGER, sourceText(), THIRD, "integer").and(isFoldable(k(), sourceText(), THIRD))


are we missing a test here on what would happen if k is not a constant (or not foldable)?

I believe that is covered by this test

carlosdelest · 2025-07-02T09:48:02Z

bwc tests fail - opened #130441 to fix

carlosdelest added 7 commits June 20, 2025 12:48

Add k as a param

d25776a

Add testing

bdfc1c2

Add CSV tests

ab1943e

Remove knnSearchWithKOption as it makes no sense now

5d401fc

Check k is a constant

3522afb

Fix ITs

3dac389

Spotless

0e9f2e6

elasticsearchmachine added the v9.1.0 label Jun 20, 2025

carlosdelest commented Jun 20, 2025

View reviewed changes

carlosdelest mentioned this pull request Jun 20, 2025

ES|QL - dense_vector approximate nearest neighbour search support #126710

Open

6 tasks

carlosdelest added 2 commits June 20, 2025 13:47

Fix tests

6a3d6f7

Fix capability for multi cluster tests

2d454ca

carlosdelest marked this pull request as ready for review June 20, 2025 13:43

elasticsearchmachine removed Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch Team:Search - Relevance The Search organization Search Relevance team labels Jun 20, 2025

carlosdelest requested review from tteofili, afoucret, ioanatia, ChrisHegarty, kderusso and svilen-mihaylov-elastic June 20, 2025 13:44

carlosdelest added the >non-issue label Jun 20, 2025

elasticsearchmachine added the serverless-linked Added by automation, don't add manually label Jun 20, 2025

ioanatia approved these changes Jun 20, 2025

View reviewed changes

tteofili approved these changes Jun 23, 2025

View reviewed changes

kderusso approved these changes Jun 23, 2025

View reviewed changes

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java Show resolved Hide resolved

ioanatia removed auto-backport Automatically create backport pull requests when merged v8.19.0 labels Jun 23, 2025

ioanatia added 2 commits June 26, 2025 12:32

Merge remote-tracking branch 'elasticsearch/main' into non-issue/knn-…

de8c564

…k-param

muted-tests merge conflicts

d54ce01

elasticsearchmachine added v9.2.0 and removed v9.1.0 labels Jun 26, 2025

ioanatia and others added 7 commits June 26, 2025 20:26

Merge branch 'main' into non-issue/knn-k-param

c659629

Merge branch 'main' into non-issue/knn-k-param

123f2e4

Removed k param from serialization to avoid TransportVersions change

a6faf49

Merge remote-tracking branch 'carlosdelest/non-issue/knn-k-param' int…

2c864e9

…o non-issue/knn-k-param # Conflicts: # docs/reference/query-languages/esql/kibana/definition/functions/knn.json # muted-tests.yml # server/src/main/java/org/elasticsearch/TransportVersions.java

Fix merge

f4356a2

Fix error for multi cluster tests, where k can already have been rewr…

4c94b4a

…itten

Merge branch 'main' into non-issue/knn-k-param

28ef9c3

carlosdelest enabled auto-merge (squash) July 2, 2025 07:05

ioanatia reviewed Jul 2, 2025

View reviewed changes

Merge remote-tracking branch 'origin/main' into non-issue/knn-k-param

9f3ff2f

Merge branch 'main' into non-issue/knn-k-param

dfa0c3c

carlosdelest merged commit 315aba6 into elastic:main Jul 2, 2025
32 checks passed

carlosdelest mentioned this pull request Jul 3, 2025

Revert bad merge in commit 315aba69 for muted-tests.yml #130523

Merged

mridula-s109 pushed a commit to mridula-s109/elasticsearch that referenced this pull request Jul 3, 2025

ESQL - Add K mandatory param for KNN function (elastic#129763)

b68de5b

carlosdelest mentioned this pull request Jul 7, 2025

Fix bad merge done in commit 315aba6 #130735

Merged

ESQL - Add K mandatory param for KNN function #129763

ESQL - Add K mandatory param for KNN function #129763

Uh oh!

Conversation

carlosdelest commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Jun 20, 2025

Uh oh!

ioanatia left a comment

Choose a reason for hiding this comment

Uh oh!

kderusso left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

carlosdelest commented Jul 1, 2025

Uh oh!

github-actions bot commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlosdelest commented Jul 2, 2025

Uh oh!

Uh oh!

Uh oh!

carlosdelest commented Jun 20, 2025 •

edited

Loading

github-actions bot commented Jun 20, 2025 •

edited

Loading

github-actions bot commented Jul 2, 2025 •

edited

Loading