[SPARK-52612][INFRA] Add an env NO_PROVIDED_SPARK_JARS to control collection behavior of sbt/package for spark-avro.jar and spark-protobuf.jar
#51321
+9
−2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
It seems that failing to gather the |
LuciferYang
changed the title
spark-avro-*.jar to the assembly/jars/scala-2.13 directory during sbt/packagespark-avro-*.jar to the classpath of the benchmark.
LuciferYang
changed the title
spark-avro-*.jar to the classpath of the benchmark.spark-avro-*.jar to the classpath of the benchmark.
LuciferYang
commented
Jun 30, 2025
|
After running The aforementioned phenomenon may lead to several issues:
On the other hand, if Therefore, it appears that there are negative consequences regardless of whether the packaging results include or exclude Currently, I can think of two solutions:
|
LuciferYang
changed the title
spark-avro-*.jar to the classpath of the benchmark.
|
For me, I prefer (1) because Maven has been always our official build system for Spark distribution. BTW, do you know when this situation starts to happen? |
The |
LuciferYang
commented
Jul 1, 2025
| } else if (jar.getName.contains("spark-connect") && | ||
| !SbtPomKeys.profiles.value.contains("noshade-connect")) { | ||
| Files.copy(fid.toPath, destJar.toPath) | ||
| } else if (jar.getName.contains("spark-protobuf") && | ||
| !SbtPomKeys.profiles.value.contains("noshade-protobuf")) { |
There was a problem hiding this comment.
I've checked, and there is no profile named noshade-protobuf in the project, so there's no need to make a judgment about it.
There was a problem hiding this comment.
In the current pr, I've chosen not to touch those profiles that are suspected to be non-existent. If it's possible to clean them up, I'll handle that in a separate PR.
|
The tests should have passed. It seems that the current solution involves relatively minor modifications. Let's run a full benchmarks to verify the effects of the changes: If everything goes well, I'll update the pr title and description tomorrow. |
|
Thank you so much, @LuciferYang . Looking forward to seeing a good result. |
All micro benchmarks can pass successfully, and there will no longer be logs similar to "
|
LuciferYang
changed the title
NO_PROVIDED_SPARK_JARS to control collection behavior of sbt/package for spark-avro.jar and spark-protobuf.jar
|
Could you help review this pull request If you have time? @dongjoon-hyun Thanks ~ |
dongjoon-hyun
approved these changes
Jul 2, 2025
dongjoon-hyun
pushed a commit
that referenced
this pull request
Jul 2, 2025
…ollection behavior of `sbt/package` for `spark-avro.jar` and `spark-protobuf.jar` ### What changes were proposed in this pull request? This pr introduces an environment variable named `NO_PROVIDED_SPARK_JARS`, which controls the behavior of the `sbt/package` command so that it only collects `spark-avro.jar` and `spark-protobuf.jar` into the `assembly/target/scala-2.13/jars` directory during documentation generation. ### Why are the changes needed? 1. To ensure that, by default, the `sbt/package` command does not collect jars with a `provided` scope, such as `spark-avro.jar` and `spark-protobuf.jar`, into the `assembly/target/scala-2.13/jars` directory, maintaining consistency with Maven's behavior. 2. To ensure that, during documentation generation, the `sbt/package` command collects the necessary jars into the `assembly/target/scala-2.13/jars` directory to ensure that no dependencies are missing for the documentation generation task. 3. To avoid the following error when executing benchmark tasks using GitHub Actions: ``` 25/06/28 07:03:45 ERROR SparkContext: Failed to add file:///home/runner/work/spark/spark/assembly/target/scala-2.13/jars/spark-avro_2.13-4.1.0-SNAPSHOT.jar to Spark environment java.lang.IllegalArgumentException: requirement failed: File spark-avro_2.13-4.1.0-SNAPSHOT.jar was already registered with a different path (old path = /home/runner/work/spark/spark/connector/avro/target/scala-2.13/spark-avro_2.13-4.1.0-SNAPSHOT.jar, new path = /home/runner/work/spark/spark/assembly/target/scala-2.13/jars/spark-avro_2.13-4.1.0-SNAPSHOT.jar ... ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Passed GitHub Actions. - Manually confirmed that benchmark tasks are not affected and that the ERROR log described above no longer appears during benchmark task execution. ### Was this patch authored or co-authored using generative AI tooling? No Closes #51321 from LuciferYang/SPARK-52612. Authored-by: yangjie01 <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 591e1c3) Signed-off-by: Dongjoon Hyun <[email protected]>
|
Merged to master/4.0. |
|
Thank you @dongjoon-hyun ~ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This pr introduces an environment variable named
NO_PROVIDED_SPARK_JARS, which controls the behavior of thesbt/packagecommand so that it only collectsspark-avro.jarandspark-protobuf.jarinto theassembly/target/scala-2.13/jarsdirectory during documentation generation.Why are the changes needed?
To ensure that, by default, the
sbt/packagecommand does not collect jars with aprovidedscope, such asspark-avro.jarandspark-protobuf.jar, into theassembly/target/scala-2.13/jarsdirectory, maintaining consistency with Maven's behavior.To ensure that, during documentation generation, the
sbt/packagecommand collects the necessary jars into theassembly/target/scala-2.13/jarsdirectory to ensure that no dependencies are missing for the documentation generation task.To avoid the following error when executing benchmark tasks using GitHub Actions:
Does this PR introduce any user-facing change?
No
How was this patch tested?
Was this patch authored or co-authored using generative AI tooling?
No