Skip to content

Commit 101244f

Browse files
jiyong-lee-devjiyong-lee-dev
committed
scala 2.13 dependency
set scala dependencies for 2.13 fix delta-core to delta-spark fix for scala 2.13 [SPARK-45629][CORE][SQL][CONNECT][ML][STREAMING][BUILD][EXAMPLES] Fix `Implicit definition should have explicit type` ### What changes were proposed in this pull request? This PR aims to fix `Implicit definition should have explicit type` in Scala 2.13. This pr includes: 1. Declaration types for global variables of implicit 2. Add scala.annotation.warn ### Why are the changes needed? - For implicit global variables without explicit type declaration, will get warnning : warning: Implicit definition should have explicit type (inferred String) [quickfixable] - No modifications are required for local variables. Additionally, to handle cases involving reflection-related types like ClassTag in implicit variables, the [scala.annotation.warn](https://github.com/scala.annotation.warn) annotation is used to suppress the warning. Furthermore, warnings generated in Spark will be treated as errors: [error] ... Implicit definition should have explicit type (inferred org.json4s.DefaultFormats.type) [quickfixable] ... [error] implicit val formats = org.json4s.DefaultFormats Jira link: SPARK-45314: https://issues.apache.org/jira/browse/SPARK-45629 Related issue link about `implicit` : scala/bug#5265 ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? Most of the testing is completed through CI, and the example module is locally compiled and tested in IDEA Additionally, there are some writing changes that are verified through demo code ### Was this patch authored or co-authored using generative AI tooling? no Closes apache#43526 from laglangyue/SPARK-45629. Lead-authored-by: tangjiafu <[email protected]> Co-authored-by: tangjiafu <[email protected]> Signed-off-by: yangjie01 <[email protected]> # Conflicts: # connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala # core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala # pom.xml # project/SparkBuild.scala # sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala # sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/OperatorStateMetadata.scala Revert "[SPARK-45629][CORE][SQL][CONNECT][ML][STREAMING][BUILD][EXAMPLES] Fix `Implicit definition should have explicit type`" This reverts commit bc67612. Fix Implicit definition should have explicit type fix install error add args for scala 2.13 fix args update pom java version update pom jvm target version fix implicit types rollback scala 2.13.8 rollback fmt blank space fix
1 parent c0f0b5e commit 101244f

File tree

117 files changed

+249
-247
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

117 files changed

+249
-247
lines changed

‎R/pkg/tests/fulltests/test_client.R

+2-2
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ test_that("multiple packages don't produce a warning", {
3737

3838
test_that("sparkJars sparkPackages as character vectors", {
3939
args <- generateSparkSubmitArgs("", "", c("one.jar", "two.jar", "three.jar"), "",
40-
c("com.databricks:spark-avro_2.12:2.0.1"))
40+
c("com.databricks:spark-avro_2.13:2.0.1"))
4141
expect_match(args, "--jars one.jar,two.jar,three.jar")
42-
expect_match(args, "--packages com.databricks:spark-avro_2.12:2.0.1")
42+
expect_match(args, "--packages com.databricks:spark-avro_2.13:2.0.1")
4343
})

‎assembly/pom.xml

+2-2
Original file line numberDiff line numberDiff line change
@@ -20,12 +20,12 @@
2020
<modelVersion>4.0.0</modelVersion>
2121
<parent>
2222
<groupId>org.apache.spark</groupId>
23-
<artifactId>spark-parent_2.12</artifactId>
23+
<artifactId>spark-parent_2.13</artifactId>
2424
<version>3.5.1</version>
2525
<relativePath>../pom.xml</relativePath>
2626
</parent>
2727

28-
<artifactId>spark-assembly_2.12</artifactId>
28+
<artifactId>spark-assembly_2.13</artifactId>
2929
<name>Spark Project Assembly</name>
3030
<url>https://spark.apache.org/</url>
3131
<packaging>pom</packaging>

‎common/kvstore/pom.xml

+2-2
Original file line numberDiff line numberDiff line change
@@ -21,12 +21,12 @@
2121
<modelVersion>4.0.0</modelVersion>
2222
<parent>
2323
<groupId>org.apache.spark</groupId>
24-
<artifactId>spark-parent_2.12</artifactId>
24+
<artifactId>spark-parent_2.13</artifactId>
2525
<version>3.5.1</version>
2626
<relativePath>../../pom.xml</relativePath>
2727
</parent>
2828

29-
<artifactId>spark-kvstore_2.12</artifactId>
29+
<artifactId>spark-kvstore_2.13</artifactId>
3030
<packaging>jar</packaging>
3131
<name>Spark Project Local DB</name>
3232
<url>https://spark.apache.org/</url>

‎common/network-common/pom.xml

+2-2
Original file line numberDiff line numberDiff line change
@@ -21,12 +21,12 @@
2121
<modelVersion>4.0.0</modelVersion>
2222
<parent>
2323
<groupId>org.apache.spark</groupId>
24-
<artifactId>spark-parent_2.12</artifactId>
24+
<artifactId>spark-parent_2.13</artifactId>
2525
<version>3.5.1</version>
2626
<relativePath>../../pom.xml</relativePath>
2727
</parent>
2828

29-
<artifactId>spark-network-common_2.12</artifactId>
29+
<artifactId>spark-network-common_2.13</artifactId>
3030
<packaging>jar</packaging>
3131
<name>Spark Project Networking</name>
3232
<url>https://spark.apache.org/</url>

‎common/network-shuffle/pom.xml

+2-2
Original file line numberDiff line numberDiff line change
@@ -21,12 +21,12 @@
2121
<modelVersion>4.0.0</modelVersion>
2222
<parent>
2323
<groupId>org.apache.spark</groupId>
24-
<artifactId>spark-parent_2.12</artifactId>
24+
<artifactId>spark-parent_2.13</artifactId>
2525
<version>3.5.1</version>
2626
<relativePath>../../pom.xml</relativePath>
2727
</parent>
2828

29-
<artifactId>spark-network-shuffle_2.12</artifactId>
29+
<artifactId>spark-network-shuffle_2.13</artifactId>
3030
<packaging>jar</packaging>
3131
<name>Spark Project Shuffle Streaming Service</name>
3232
<url>https://spark.apache.org/</url>

‎common/network-yarn/pom.xml

+2-2
Original file line numberDiff line numberDiff line change
@@ -21,12 +21,12 @@
2121
<modelVersion>4.0.0</modelVersion>
2222
<parent>
2323
<groupId>org.apache.spark</groupId>
24-
<artifactId>spark-parent_2.12</artifactId>
24+
<artifactId>spark-parent_2.13</artifactId>
2525
<version>3.5.1</version>
2626
<relativePath>../../pom.xml</relativePath>
2727
</parent>
2828

29-
<artifactId>spark-network-yarn_2.12</artifactId>
29+
<artifactId>spark-network-yarn_2.13</artifactId>
3030
<packaging>jar</packaging>
3131
<name>Spark Project YARN Shuffle Service</name>
3232
<url>https://spark.apache.org/</url>

‎common/sketch/pom.xml

+2-2
Original file line numberDiff line numberDiff line change
@@ -21,12 +21,12 @@
2121
<modelVersion>4.0.0</modelVersion>
2222
<parent>
2323
<groupId>org.apache.spark</groupId>
24-
<artifactId>spark-parent_2.12</artifactId>
24+
<artifactId>spark-parent_2.13</artifactId>
2525
<version>3.5.1</version>
2626
<relativePath>../../pom.xml</relativePath>
2727
</parent>
2828

29-
<artifactId>spark-sketch_2.12</artifactId>
29+
<artifactId>spark-sketch_2.13</artifactId>
3030
<packaging>jar</packaging>
3131
<name>Spark Project Sketch</name>
3232
<url>https://spark.apache.org/</url>

‎common/tags/pom.xml

+2-2
Original file line numberDiff line numberDiff line change
@@ -21,12 +21,12 @@
2121
<modelVersion>4.0.0</modelVersion>
2222
<parent>
2323
<groupId>org.apache.spark</groupId>
24-
<artifactId>spark-parent_2.12</artifactId>
24+
<artifactId>spark-parent_2.13</artifactId>
2525
<version>3.5.1</version>
2626
<relativePath>../../pom.xml</relativePath>
2727
</parent>
2828

29-
<artifactId>spark-tags_2.12</artifactId>
29+
<artifactId>spark-tags_2.13</artifactId>
3030
<packaging>jar</packaging>
3131
<name>Spark Project Tags</name>
3232
<url>https://spark.apache.org/</url>

‎common/unsafe/pom.xml

+2-2
Original file line numberDiff line numberDiff line change
@@ -21,12 +21,12 @@
2121
<modelVersion>4.0.0</modelVersion>
2222
<parent>
2323
<groupId>org.apache.spark</groupId>
24-
<artifactId>spark-parent_2.12</artifactId>
24+
<artifactId>spark-parent_2.13</artifactId>
2525
<version>3.5.1</version>
2626
<relativePath>../../pom.xml</relativePath>
2727
</parent>
2828

29-
<artifactId>spark-unsafe_2.12</artifactId>
29+
<artifactId>spark-unsafe_2.13</artifactId>
3030
<packaging>jar</packaging>
3131
<name>Spark Project Unsafe</name>
3232
<url>https://spark.apache.org/</url>

‎common/utils/pom.xml

+2-2
Original file line numberDiff line numberDiff line change
@@ -21,12 +21,12 @@
2121
<modelVersion>4.0.0</modelVersion>
2222
<parent>
2323
<groupId>org.apache.spark</groupId>
24-
<artifactId>spark-parent_2.12</artifactId>
24+
<artifactId>spark-parent_2.13</artifactId>
2525
<version>3.5.1</version>
2626
<relativePath>../../pom.xml</relativePath>
2727
</parent>
2828

29-
<artifactId>spark-common-utils_2.12</artifactId>
29+
<artifactId>spark-common-utils_2.13</artifactId>
3030
<packaging>jar</packaging>
3131
<name>Spark Project Common Utils</name>
3232
<url>https://spark.apache.org/</url>

‎connector/avro/pom.xml

+2-2
Original file line numberDiff line numberDiff line change
@@ -20,12 +20,12 @@
2020
<modelVersion>4.0.0</modelVersion>
2121
<parent>
2222
<groupId>org.apache.spark</groupId>
23-
<artifactId>spark-parent_2.12</artifactId>
23+
<artifactId>spark-parent_2.13</artifactId>
2424
<version>3.5.1</version>
2525
<relativePath>../../pom.xml</relativePath>
2626
</parent>
2727

28-
<artifactId>spark-avro_2.12</artifactId>
28+
<artifactId>spark-avro_2.13</artifactId>
2929
<properties>
3030
<sbt.project.name>avro</sbt.project.name>
3131
</properties>

‎connector/connect/client/jvm/pom.xml

+2-2
Original file line numberDiff line numberDiff line change
@@ -21,12 +21,12 @@
2121
<modelVersion>4.0.0</modelVersion>
2222
<parent>
2323
<groupId>org.apache.spark</groupId>
24-
<artifactId>spark-parent_2.12</artifactId>
24+
<artifactId>spark-parent_2.13</artifactId>
2525
<version>3.5.1</version>
2626
<relativePath>../../../../pom.xml</relativePath>
2727
</parent>
2828

29-
<artifactId>spark-connect-client-jvm_2.12</artifactId>
29+
<artifactId>spark-connect-client-jvm_2.13</artifactId>
3030
<packaging>jar</packaging>
3131
<name>Spark Project Connect Client</name>
3232
<url>https://spark.apache.org/</url>

‎connector/connect/common/pom.xml

+2-2
Original file line numberDiff line numberDiff line change
@@ -21,12 +21,12 @@
2121
<modelVersion>4.0.0</modelVersion>
2222
<parent>
2323
<groupId>org.apache.spark</groupId>
24-
<artifactId>spark-parent_2.12</artifactId>
24+
<artifactId>spark-parent_2.13</artifactId>
2525
<version>3.5.1</version>
2626
<relativePath>../../../pom.xml</relativePath>
2727
</parent>
2828

29-
<artifactId>spark-connect-common_2.12</artifactId>
29+
<artifactId>spark-connect-common_2.13</artifactId>
3030
<packaging>jar</packaging>
3131
<name>Spark Project Connect Common</name>
3232
<url>https://spark.apache.org/</url>

‎connector/connect/server/pom.xml

+2-2
Original file line numberDiff line numberDiff line change
@@ -21,12 +21,12 @@
2121
<modelVersion>4.0.0</modelVersion>
2222
<parent>
2323
<groupId>org.apache.spark</groupId>
24-
<artifactId>spark-parent_2.12</artifactId>
24+
<artifactId>spark-parent_2.13</artifactId>
2525
<version>3.5.1</version>
2626
<relativePath>../../../pom.xml</relativePath>
2727
</parent>
2828

29-
<artifactId>spark-connect_2.12</artifactId>
29+
<artifactId>spark-connect_2.13</artifactId>
3030
<packaging>jar</packaging>
3131
<name>Spark Project Connect Server</name>
3232
<url>https://spark.apache.org/</url>

‎connector/docker-integration-tests/pom.xml

+2-2
Original file line numberDiff line numberDiff line change
@@ -21,12 +21,12 @@
2121
<modelVersion>4.0.0</modelVersion>
2222
<parent>
2323
<groupId>org.apache.spark</groupId>
24-
<artifactId>spark-parent_2.12</artifactId>
24+
<artifactId>spark-parent_2.13</artifactId>
2525
<version>3.5.1</version>
2626
<relativePath>../../pom.xml</relativePath>
2727
</parent>
2828

29-
<artifactId>spark-docker-integration-tests_2.12</artifactId>
29+
<artifactId>spark-docker-integration-tests_2.13</artifactId>
3030
<packaging>jar</packaging>
3131
<name>Spark Project Docker Integration Tests</name>
3232
<url>https://spark.apache.org/</url>

‎connector/kafka-0-10-assembly/pom.xml

+2-2
Original file line numberDiff line numberDiff line change
@@ -20,12 +20,12 @@
2020
<modelVersion>4.0.0</modelVersion>
2121
<parent>
2222
<groupId>org.apache.spark</groupId>
23-
<artifactId>spark-parent_2.12</artifactId>
23+
<artifactId>spark-parent_2.13</artifactId>
2424
<version>3.5.1</version>
2525
<relativePath>../../pom.xml</relativePath>
2626
</parent>
2727

28-
<artifactId>spark-streaming-kafka-0-10-assembly_2.12</artifactId>
28+
<artifactId>spark-streaming-kafka-0-10-assembly_2.13</artifactId>
2929
<packaging>jar</packaging>
3030
<name>Spark Integration for Kafka 0.10 Assembly</name>
3131
<url>https://spark.apache.org/</url>

‎connector/kafka-0-10-sql/pom.xml

+2-2
Original file line numberDiff line numberDiff line change
@@ -20,13 +20,13 @@
2020
<modelVersion>4.0.0</modelVersion>
2121
<parent>
2222
<groupId>org.apache.spark</groupId>
23-
<artifactId>spark-parent_2.12</artifactId>
23+
<artifactId>spark-parent_2.13</artifactId>
2424
<version>3.5.1</version>
2525
<relativePath>../../pom.xml</relativePath>
2626
</parent>
2727

2828
<groupId>org.apache.spark</groupId>
29-
<artifactId>spark-sql-kafka-0-10_2.12</artifactId>
29+
<artifactId>spark-sql-kafka-0-10_2.13</artifactId>
3030
<properties>
3131
<sbt.project.name>sql-kafka-0-10</sbt.project.name>
3232
</properties>

‎connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/JsonUtils.scala

+3-5
Original file line numberDiff line numberDiff line change
@@ -21,14 +21,14 @@ import scala.collection.mutable.HashMap
2121
import scala.util.control.NonFatal
2222

2323
import org.apache.kafka.common.TopicPartition
24-
import org.json4s.NoTypeHints
24+
import org.json4s.{Formats, NoTypeHints}
2525
import org.json4s.jackson.Serialization
2626

2727
/**
2828
* Utilities for converting Kafka related objects to and from json.
2929
*/
3030
private object JsonUtils {
31-
private implicit val formats = Serialization.formats(NoTypeHints)
31+
private implicit val formats: Formats = Serialization.formats(NoTypeHints)
3232

3333
/**
3434
* Read TopicPartitions from json string
@@ -96,10 +96,8 @@ private object JsonUtils {
9696
*/
9797
def partitionOffsets(partitionOffsets: Map[TopicPartition, Long]): String = {
9898
val result = new HashMap[String, HashMap[Int, Long]]()
99-
implicit val order = new Ordering[TopicPartition] {
100-
override def compare(x: TopicPartition, y: TopicPartition): Int = {
99+
implicit val order: Ordering[TopicPartition] = (x: TopicPartition, y: TopicPartition) => {
101100
Ordering.Tuple2[String, Int].compare((x.topic, x.partition), (y.topic, y.partition))
102-
}
103101
}
104102
val partitions = partitionOffsets.keySet.toSeq.sorted // sort for more determinism
105103
partitions.foreach { tp =>

‎connector/kafka-0-10-token-provider/pom.xml

+2-2
Original file line numberDiff line numberDiff line change
@@ -20,13 +20,13 @@
2020
<modelVersion>4.0.0</modelVersion>
2121
<parent>
2222
<groupId>org.apache.spark</groupId>
23-
<artifactId>spark-parent_2.12</artifactId>
23+
<artifactId>spark-parent_2.13</artifactId>
2424
<version>3.5.1</version>
2525
<relativePath>../../pom.xml</relativePath>
2626
</parent>
2727

2828
<groupId>org.apache.spark</groupId>
29-
<artifactId>spark-token-provider-kafka-0-10_2.12</artifactId>
29+
<artifactId>spark-token-provider-kafka-0-10_2.13</artifactId>
3030
<properties>
3131
<sbt.project.name>token-provider-kafka-0-10</sbt.project.name>
3232
</properties>

‎connector/kafka-0-10/pom.xml

+2-2
Original file line numberDiff line numberDiff line change
@@ -20,12 +20,12 @@
2020
<modelVersion>4.0.0</modelVersion>
2121
<parent>
2222
<groupId>org.apache.spark</groupId>
23-
<artifactId>spark-parent_2.12</artifactId>
23+
<artifactId>spark-parent_2.13</artifactId>
2424
<version>3.5.1</version>
2525
<relativePath>../../pom.xml</relativePath>
2626
</parent>
2727

28-
<artifactId>spark-streaming-kafka-0-10_2.12</artifactId>
28+
<artifactId>spark-streaming-kafka-0-10_2.13</artifactId>
2929
<properties>
3030
<sbt.project.name>streaming-kafka-0-10</sbt.project.name>
3131
</properties>

‎connector/kinesis-asl-assembly/pom.xml

+2-2
Original file line numberDiff line numberDiff line change
@@ -20,12 +20,12 @@
2020
<modelVersion>4.0.0</modelVersion>
2121
<parent>
2222
<groupId>org.apache.spark</groupId>
23-
<artifactId>spark-parent_2.12</artifactId>
23+
<artifactId>spark-parent_2.13</artifactId>
2424
<version>3.5.1</version>
2525
<relativePath>../../pom.xml</relativePath>
2626
</parent>
2727

28-
<artifactId>spark-streaming-kinesis-asl-assembly_2.12</artifactId>
28+
<artifactId>spark-streaming-kinesis-asl-assembly_2.13</artifactId>
2929
<packaging>jar</packaging>
3030
<name>Spark Project Kinesis Assembly</name>
3131
<url>https://spark.apache.org/</url>

‎connector/kinesis-asl/pom.xml

+2-2
Original file line numberDiff line numberDiff line change
@@ -19,13 +19,13 @@
1919
<modelVersion>4.0.0</modelVersion>
2020
<parent>
2121
<groupId>org.apache.spark</groupId>
22-
<artifactId>spark-parent_2.12</artifactId>
22+
<artifactId>spark-parent_2.13</artifactId>
2323
<version>3.5.1</version>
2424
<relativePath>../../pom.xml</relativePath>
2525
</parent>
2626

2727
<!-- Kinesis integration is not included by default due to ASL-licensed code. -->
28-
<artifactId>spark-streaming-kinesis-asl_2.12</artifactId>
28+
<artifactId>spark-streaming-kinesis-asl_2.13</artifactId>
2929
<packaging>jar</packaging>
3030
<name>Spark Kinesis Integration</name>
3131

‎connector/protobuf/pom.xml

+2-2
Original file line numberDiff line numberDiff line change
@@ -20,12 +20,12 @@
2020
<modelVersion>4.0.0</modelVersion>
2121
<parent>
2222
<groupId>org.apache.spark</groupId>
23-
<artifactId>spark-parent_2.12</artifactId>
23+
<artifactId>spark-parent_2.13</artifactId>
2424
<version>3.5.1</version>
2525
<relativePath>../../pom.xml</relativePath>
2626
</parent>
2727

28-
<artifactId>spark-protobuf_2.12</artifactId>
28+
<artifactId>spark-protobuf_2.13</artifactId>
2929
<properties>
3030
<sbt.project.name>protobuf</sbt.project.name>
3131
</properties>

‎connector/spark-ganglia-lgpl/pom.xml

+2-2
Original file line numberDiff line numberDiff line change
@@ -19,13 +19,13 @@
1919
<modelVersion>4.0.0</modelVersion>
2020
<parent>
2121
<groupId>org.apache.spark</groupId>
22-
<artifactId>spark-parent_2.12</artifactId>
22+
<artifactId>spark-parent_2.13</artifactId>
2323
<version>3.5.1</version>
2424
<relativePath>../../pom.xml</relativePath>
2525
</parent>
2626

2727
<!-- Ganglia integration is not included by default due to LGPL-licensed code -->
28-
<artifactId>spark-ganglia-lgpl_2.12</artifactId>
28+
<artifactId>spark-ganglia-lgpl_2.13</artifactId>
2929
<packaging>jar</packaging>
3030
<name>Spark Ganglia Integration</name>
3131

‎core/pom.xml

+2-2
Original file line numberDiff line numberDiff line change
@@ -20,12 +20,12 @@
2020
<modelVersion>4.0.0</modelVersion>
2121
<parent>
2222
<groupId>org.apache.spark</groupId>
23-
<artifactId>spark-parent_2.12</artifactId>
23+
<artifactId>spark-parent_2.13</artifactId>
2424
<version>3.5.1</version>
2525
<relativePath>../pom.xml</relativePath>
2626
</parent>
2727

28-
<artifactId>spark-core_2.12</artifactId>
28+
<artifactId>spark-core_2.13</artifactId>
2929
<packaging>jar</packaging>
3030
<name>Spark Project Core</name>
3131
<url>https://spark.apache.org/</url>

0 commit comments

Comments
 (0)