build: Add Spark 4.0 to release build script

[parthchandra]

Which issue does this PR close?

Closes #1989

Updates the build script to build spark 4.0 jars.
Build now requires JDK 17 for all versions.

[andygrove]

@parthchandra will the jars for Spark 3.x still work with JDK 11, or do we also need to update the target java version in the Maven build?

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 58.49%. Comparing base (f09f8af) to head (f347a53).
⚠️ Report is 568 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2514      +/-   ##
============================================
+ Coverage     56.12%   58.49%   +2.36%     
- Complexity      976     1447     +471     
============================================
  Files           119      146      +27     
  Lines         11743    13550    +1807     
  Branches       2251     2356     +105     
============================================
+ Hits           6591     7926    +1335     
- Misses         4012     4390     +378     
- Partials       1140     1234      +94     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[parthchandra]

@parthchandra will the jars for Spark 3.x still work with JDK 11, or do we also need to update the target java version in the Maven build?

Let me verify that.

[parthchandra]

@parthchandra will the jars for Spark 3.x still work with JDK 11, or do we also need to update the target java version in the Maven build?

Let me verify that.

The jars for Spark 3.x will work with JDK 11.

The pom file already specifies

    <java.version>11</java.version>
    <maven.compiler.source>${java.version}</maven.compiler.source>
    <maven.compiler.target>${java.version}</maven.compiler.target>

For the spark 4.0 profile this is overridden with

        <java.version>17</java.version>
        <maven.compiler.source>${java.version}</maven.compiler.source>
        <maven.compiler.target>${java.version}</maven.compiler.target>

Also the compiler plugin specifies the configured version so it is all consistent.

        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-compiler-plugin</artifactId>
          <version>${maven-compiler-plugin.version}</version>
          <configuration>
            <source>${java.version}</source>
            <target>${java.version}</target>
            <skipMain>true</skipMain>
            <skip>true</skip>
          </configuration>
        </plugin>

I also built with JDK 17 and the byte code versions for the default profile (Spark 3.5) were compatible with JDK 8 (major version 52) for the classes I checked. With the spark-4.0 profile the bytecode versions corresponded to JDK 17 (major version 61)

[andygrove]

The jars for Spark 3.x will work with JDK 11.

The pom file already specifies

    <java.version>11</java.version>
    <maven.compiler.source>${java.version}</maven.compiler.source>
    <maven.compiler.target>${java.version}</maven.compiler.target>

The java.version gets overridden by the JDK profiles though:

    <profile>
      <id>jdk17</id>
      <activation>
        <jdk>17</jdk>
      </activation>
      <properties>
        <java.version>17</java.version>
        <maven.compiler.source>${java.version}</maven.compiler.source>
        <maven.compiler.target>${java.version}</maven.compiler.target>
      </properties>
    </profile>

So if I build with JDK 17, then I get JDK 17 classes. If I try and run with JDK 11, I get:

java.lang.UnsupportedClassVersionError: org/apache/spark/sql/comet/execution/shuffle/CometBypassMergeSortShuffleWriter has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0

I think we need to override the maven.compiler.target (or java.version) for each Spark version to set it to the minimum supported JDK version for that Spark version.

[parthchandra]

Added jdk 11 explicitly for spark 3.4, 3.5 profiles

[andygrove]

Thanks @parthchandra. I tested this locally, and LGTM