Upgrade to Lucene 10.1.0

[ChrisHegarty]

This commit upgrades to Lucene 10.1.0.

[javanna]

LGTM thanks (only one small thing in the buildkite pipelines that I think should get reverted, although it should not take effect)

[chatman]

@ChrisHegarty I'm working on apache/lucene#14131, but in order to test it with Elasticsearch, I needed to test with Elasticsearch that uses Lucene 10. Hence, I'm looking at this branch. However, the work in apache/lucene#14131 integrates cuvs-java that uses Project Panama's features in JDK 22. However, Elasticsearch (as of this branch) still is on JDK21. Hence, trying to "gradlew compileJava" results in the following:

=======================================
Elasticsearch Build Hamster says Hello!
  Gradle Version        : 8.12
  OS Info               : Linux 6.8.0-49-generic (amd64)
  JDK Version           : 22.0.2+9-70 (Oracle)
  JAVA_HOME             : /opt/jdk-22.0.2
  Random Testing Seed   : 5173214438CDD58D
  In FIPS 140 mode      : false
=======================================

> Task :server:compileJava FAILED
warning: /home/ishan/.m2/repository/com/nvidia/cuvs/cuvs-java/25.02/cuvs-java-25.02.jar(/module-info.class): major version 66 is newer than 65, the highest major version supported by this compiler.
  It is recommended that the compiler be upgraded.
error: warnings found and -Werror specified
1 error
1 warning

[Incubating] Problems report is available at: file:///home/ishan/code/elasticsearch/build/reports/problems/problems-report.html

FAILURE: Build failed with an exception.

(the patch is https://termbin.com/v0co on top of this branch, along with the Lucene PR for cuVS being available from a local maven snapshot based on 10.2).

Any ideas or thoughts, please? Is there a plan (or will there be) to raise minimum JDK level to 22?

[ChrisHegarty]

Any ideas or thoughts, please? Is there a plan (or will there be) to raise minimum JDK level to 22?

There is certainly some complications here, but we can resolve them.

Elasticsearch compiles with a minimum of JDK 21, but runs with the latest JDK, currently 23. We mostly do something similar with Lucene. I need to look a bit more at how this is all supposed to work in Lucene first, then I'll see how we can get this working with ES.

[chatman]

Elasticsearch compiles with a minimum of JDK 21, but runs with the latest JDK, currently 23. We mostly do something similar with Lucene. I need to look a bit more at how this is all supposed to work in Lucene first, then I'll see how we can get this working with ES.

I'm trying to get a branch working with it: https://github.com/SearchScale/elasticsearch/tree/cuvs-integration
Mainly, upgraded things to minimum JDK22 (so that it doesn't complain that CuVS is compiled with JDK22).
I'll send you some ideas on a dev setup on that CuVS-Java PR so that you can kindly test it out.

[ChrisHegarty]

Linking this test PR that we need before merging the Lucene 10.1.0 upgrade - #121104

[tlrx]

Nice work @ChrisHegarty 👏

[mark-vieira]

Any ideas or thoughts, please? Is there a plan (or will there be) to raise minimum JDK level to 22?

There are no plans to bump the minimum Java version at this time. The soonest we would do this would be with the release of Java 25, as our general policy is for minimum runtime version to be LTS versions.