Skip to content

[SPARK-52612][INFRA] Add an env NO_PROVIDED_SPARK_JARS to control collection behavior of sbt/package for spark-avro.jar and spark-protobuf.jar #51321

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from
Closed

[SPARK-52612][INFRA] Add an env NO_PROVIDED_SPARK_JARS to control collection behavior of sbt/package for spark-avro.jar and spark-protobuf.jar #51321

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
de37ad5
init
LuciferYang Jun 30, 2025
003f330
test
LuciferYang Jun 30, 2025
5956b91
test
LuciferYang Jul 1, 2025
0d249f0
more test
LuciferYang Jul 1, 2025
afec878
revert
LuciferYang Jul 1, 2025
7fdb7ae
rename
LuciferYang Jul 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/_plugins/build_api_docs.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ def build_spark_if_necessary

print_header "Building Spark."
cd(SPARK_PROJECT_ROOT)
command = "build/sbt -Phive -Pkinesis-asl clean package"
command = "NO_PROVIDED_SPARK_JARS=0 build/sbt -Phive -Pkinesis-asl clean package"
puts "Running '#{command}'; this may take a few minutes..."
system(command) || raise("Failed to build Spark")
$spark_package_is_built = true
Expand Down
9 changes: 8 additions & 1 deletion project/SparkBuild.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1517,6 +1517,9 @@ object CopyDependencies {
val fid = (LocalProject("connect") / assembly).value
val fidClient = (LocalProject("connect-client-jvm") / assembly).value
val fidProtobuf = (LocalProject("protobuf") / assembly).value
val noProvidedSparkJars: Boolean = sys.env.getOrElse("NO_PROVIDED_SPARK_JARS", "1") == "1" ||
sys.env.getOrElse("NO_PROVIDED_SPARK_JARS", "true")
.toLowerCase(Locale.getDefault()) == "true"

(Compile / dependencyClasspath).value.map(_.data)
.filter { jar => jar.isFile() }
Expand All @@ -1532,12 +1535,16 @@ object CopyDependencies {
// Don't copy the spark connect common JAR as it is shaded in the spark connect.
} else if (jar.getName.contains("connect-client-jvm")) {
// Do not place Spark Connect client jars as it is not built-in.
} else if (noProvidedSparkJars && jar.getName.contains("spark-avro")) {
// Do not place Spark Avro jars as it is not built-in.
} else if (jar.getName.contains("spark-connect") &&
!SbtPomKeys.profiles.value.contains("noshade-connect")) {
Files.copy(fid.toPath, destJar.toPath)
} else if (jar.getName.contains("spark-protobuf") &&
!SbtPomKeys.profiles.value.contains("noshade-protobuf")) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've checked, and there is no profile named noshade-protobuf in the project, so there's no need to make a judgment about it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the current pr, I've chosen not to touch those profiles that are suspected to be non-existent. If it's possible to clean them up, I'll handle that in a separate PR.

Files.copy(fidProtobuf.toPath, destJar.toPath)
if (!noProvidedSparkJars) {
Files.copy(fidProtobuf.toPath, destJar.toPath)
}
} else {
Files.copy(jar.toPath(), destJar.toPath())
}
Expand Down