Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUILD] Upgrade RoaringBitmap to 1.3.19 #49710

Draft
wants to merge 4 commits into
base: master
Choose a base branch
from
+32 −20
Draft

[BUILD] Upgrade RoaringBitmap to 1.3.19 #49710

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d4b1054
init
LuciferYang Jan 28, 2025
cfa126b
add bench result
LuciferYang Jan 28, 2025
34d60a9
init
LuciferYang Jan 28, 2025
71b7d39
test java 21
LuciferYang Jan 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/build_and_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ on:
java:
required: false
type: string
default: 17
default: 21
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will revert after test

branch:
description: Branch to run the build against
required: false
Expand Down
2 changes: 1 addition & 1 deletion LICENSE-binary
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -392,7 +392,7 @@ org.json4s:json4s-jackson_2.13
org.json4s:json4s-scalap_2.13
org.lz4:lz4-java
org.objenesis:objenesis
org.roaringbitmap:RoaringBitmap
com.github.RoaringBitmap.RoaringBitmap:roaringbitmap
org.rocksdb:rocksdbjni
org.scala-lang:scala-compiler
org.scala-lang:scala-library
Expand Down
4 changes: 2 additions & 2 deletions common/network-common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,8 +136,8 @@
<artifactId>tink</artifactId>
</dependency>
<dependency>
<groupId>org.roaringbitmap</groupId>
<artifactId>RoaringBitmap</artifactId>
<groupId>com.github.RoaringBitmap.RoaringBitmap</groupId>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, interesting.

<artifactId>roaringbitmap</artifactId>
</dependency>

<!-- Test dependencies -->
Expand Down
4 changes: 2 additions & 2 deletions common/network-shuffle/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,8 +63,8 @@
<artifactId>guava</artifactId>
</dependency>
<dependency>
<groupId>org.roaringbitmap</groupId>
<artifactId>RoaringBitmap</artifactId>
<groupId>com.github.RoaringBitmap.RoaringBitmap</groupId>
<artifactId>roaringbitmap</artifactId>
</dependency>

<!-- Test dependencies -->
Expand Down
8 changes: 4 additions & 4 deletions core/benchmarks/MapStatusesConvertBenchmark-jdk21-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@
MapStatuses Convert Benchmark
================================================================================================

OpenJDK 64-Bit Server VM 21.0.5+11-LTS on Linux 6.8.0-1017-azure
OpenJDK 64-Bit Server VM 21.0.6+7-LTS on Linux 6.8.0-1020-azure
AMD EPYC 7763 64-Core Processor
MapStatuses Convert: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Num Maps: 50000 Fetch partitions:500 716 730 14 0.0 715747604.0 1.0X
Num Maps: 50000 Fetch partitions:1000 1592 1619 29 0.0 1591519021.0 0.4X
Num Maps: 50000 Fetch partitions:1500 2500 2507 7 0.0 2499934291.0 0.3X
Num Maps: 50000 Fetch partitions:500 726 733 6 0.0 726407701.0 1.0X
Num Maps: 50000 Fetch partitions:1000 1644 1659 14 0.0 1644292738.0 0.4X
Num Maps: 50000 Fetch partitions:1500 2482 2514 31 0.0 2482389083.0 0.3X


8 changes: 4 additions & 4 deletions core/benchmarks/MapStatusesConvertBenchmark-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@
MapStatuses Convert Benchmark
================================================================================================

OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.8.0-1017-azure
OpenJDK 64-Bit Server VM 17.0.14+7-LTS on Linux 6.8.0-1020-azure
AMD EPYC 7763 64-Core Processor
MapStatuses Convert: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Num Maps: 50000 Fetch partitions:500 612 614 3 0.0 611543498.0 1.0X
Num Maps: 50000 Fetch partitions:1000 1389 1398 15 0.0 1388971632.0 0.4X
Num Maps: 50000 Fetch partitions:1500 2178 2222 39 0.0 2177711722.0 0.3X
Num Maps: 50000 Fetch partitions:500 718 727 13 0.0 717856848.0 1.0X
Num Maps: 50000 Fetch partitions:1000 1730 1751 25 0.0 1729819265.0 0.4X
Num Maps: 50000 Fetch partitions:1500 2662 2692 27 0.0 2662049238.0 0.3X


4 changes: 2 additions & 2 deletions core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -240,8 +240,8 @@
<artifactId>zstd-jni</artifactId>
</dependency>
<dependency>
<groupId>org.roaringbitmap</groupId>
<artifactId>RoaringBitmap</artifactId>
<groupId>com.github.RoaringBitmap.RoaringBitmap</groupId>
<artifactId>roaringbitmap</artifactId>
</dependency>
<dependency>
<groupId>org.scala-lang.modules</groupId>
Expand Down
2 changes: 1 addition & 1 deletion dev/deps/spark-deps-hadoop-3-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
HikariCP/2.5.1//HikariCP-2.5.1.jar
JLargeArrays/1.5//JLargeArrays-1.5.jar
JTransforms/3.1//JTransforms-3.1.jar
RoaringBitmap/1.3.0//RoaringBitmap-1.3.0.jar
ST4/4.0.4//ST4-4.0.4.jar
activation/1.1.1//activation-1.1.1.jar
aircompressor/2.0.2//aircompressor-2.0.2.jar
Expand Down Expand Up @@ -252,6 +251,7 @@ parquet-jackson/1.15.0//parquet-jackson-1.15.0.jar
pickle/1.5//pickle-1.5.jar
py4j/0.10.9.9//py4j-0.10.9.9.jar
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
roaringbitmap/1.3.19//roaringbitmap-1.3.19.jar
rocksdbjni/9.8.4//rocksdbjni-9.8.4.jar
scala-collection-compat_2.13/2.7.0//scala-collection-compat_2.13-2.7.0.jar
scala-compiler/2.13.15//scala-compiler-2.13.15.jar
Expand Down
17 changes: 14 additions & 3 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -356,6 +356,17 @@
<enabled>false</enabled>
</snapshots>
</repository>
<repository>
Copy link
Contributor Author

@LuciferYang LuciferYang Jan 28, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new version of RoaringBitmap requires configuring additional repositories:

image

The second method requires additional configuration of a GitHub token, so Method One was chosen here.

<id>jitpack.io</id>
<name>JitPack</name>
<url>https://jitpack.io</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
<repository>
<!--
This is used as a fallback when the first try fails.
Expand Down Expand Up @@ -868,9 +879,9 @@
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.roaringbitmap</groupId>
<artifactId>RoaringBitmap</artifactId>
<version>1.3.0</version>
<groupId>com.github.RoaringBitmap.RoaringBitmap</groupId>
<artifactId>roaringbitmap</artifactId>
<version>1.3.19</version>
</dependency>

<!-- Netty Begin -->
Expand Down
1 change: 1 addition & 0 deletions project/SparkBuild.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -277,6 +277,7 @@ object SparkBuild extends PomBuild {
// Google Mirror of Maven Central, placed first so that it's used instead of flaky Maven Central.
// See https://storage-download.googleapis.com/maven-central/index.html for more info.
"gcs-maven-central-mirror" at "https://maven-central.storage-download.googleapis.com/maven2/",
"jitpack" at "https://jitpack.io",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wondering if this can cause a flakiness like the default Maven.

Due to the Apache Spark's high traffic, the Maven was flaky. So, we are using GCS Maven as the first cache here inevitably.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Understand your concerns. We can wait for a while longer to see if RoaringBitmap can resume its publish on Maven Central

DefaultMavenRepository,
Resolver.mavenLocal,
Resolver.file("ivyLocal", file(Path.userHome.absolutePath + "/.ivy2/local"))(Resolver.ivyStylePatterns)
Expand Down
Loading