update rapids version for 24.10 release #1248

nvliyuan · 2024-10-23T09:35:24Z

This pr is to update the spark-rapids script version to 24.10.0 and update the readme doc

nvliyuan · 2024-10-24T02:11:50Z

@viadea please help review. CC @jayadeep-jayaraman @cjac

cjac · 2024-10-24T03:02:10Z

Oh hey, thanks for the ping. I'll check it out.

cjac · 2024-10-24T03:06:23Z

/gcbrun

cjac

Let's get the docs updated to reflect the versions of Dataproc supported in Q4 2024

Our current supported versions follow:
2.2-debian12
2.1-debian11
2.0-debian10
2.2-ubuntu22
2.2-ubuntu20
2.0-ubuntu18
2.2-rocky9
2.1-rocky8
2.0-rocky8

And if all of these platforms are not supported by spark-rapids, then we'll need to merge in my changes to dask-rapids.

cjac · 2024-10-24T03:03:50Z

spark-rapids/README.md

    *   NCCL 2.11.4+
-    *   Ubuntu 18.04, Ubuntu 20.04 or Rocky Linux 7, Rocky Linux8, Debian 10, Debian 11
+    *   Ubuntu 20.04, Ubuntu 22.04, CentOS 7, or Rocky Linux 8, Debian 10, Debian 11


Do you have support for CentOS 7, stil, really? I commend you. Do the tests exercise that platform?

I would also add Rocky Linux 9.

If you do not have support for rocky linux 9 yet, we can merge my work from spark-dask. I'll continue my review.

Thanks for pointing it out, we don't support centos7 now, update a link in the doc for the soft/hardware requirements.

File related pr for the doc issue.

cjac · 2024-10-24T03:05:34Z

spark-rapids/README.md

@@ -63,7 +63,7 @@ export CUDA_VER=11.5

 gcloud dataproc clusters create $CLUSTER_NAME  \
    --region $REGION \
-    --image-version=2.0-ubuntu18 \
+    --image-version=2.1-ubuntu20 \


Please recommend 2.2 where possible. If 2.1 doesn't have other representation, then this is fine, but we should be emphasizing that 2.2 is the better choice.

update to 2.2 image

nvliyuan · 2024-11-05T07:30:33Z

Hi @cjac , can we merge this pr?

nvliyuan · 2024-11-11T01:48:56Z

Hi @cjac , any update?

cjac · 2024-11-11T06:44:27Z

I apologize for the delay here.

I'm caught up behind adding installation from local disk as an option to rapids/rapids.sh ; I had begun seeing weekly cdn related build failures, so I'm bringing the packages closer to the cluster to improve ci/cd test performance.

Unfortunately, conda does not presently install directly from direct attached media, opting instead to copy the packages to an intermediate temp directory before unpacking.

conda/conda#14377

If movement on spark-rapids is urgent and merits putting down dask rapids instead of finishing it, putting it down, and moving on to spark-rapids work, I may be able to switch context. I prefer to finish the other first, but if nv wants to see a new version of spark-rapids before the middle of December, then let me know and I'll switch tracks for a bit.

My current estimate for completion of dask-rapids work is later this week. Then I will take a look at spark-rapids/ for the first time since it got its own directory.

C.J.

nvliyuan · 2024-12-24T06:39:56Z

Hi @cjac , not sure that can we merge this pr now?

cjac · 2024-12-24T23:42:21Z

I haven't tested it thoroughly yet. I've been caught up in refactoring shared code into templates. I would like to generate this file from components rather than copy/pasting between scripts.

Can you let me know what you think of #1282 please?

nvliyuan · 2024-12-25T01:40:35Z

Let's merge the pr for now, it is just a version update, thx

cjac · 2024-12-25T02:12:10Z

let me try it in my environment...

cjac · 2024-12-25T02:28:24Z

running without cuda-version specified produces a request for:

cjac · 2024-12-25T02:31:45Z

incorrectly specifying rapids-runtime as --metadata rapids-runtime="RAPIDS" produces usable error message.

cjac · 2024-12-25T02:55:37Z

@nvliyuan - do you mind if I commit to your branch?

nvliyuan · 2024-12-25T02:57:33Z

please feel free to commit, thanks

cjac · 2024-12-25T03:05:21Z

a re-re-re-run took 4m35.536s to complete ; This looks good to me. Let me run it through the automated tests.

It looks like it built the kernel more than once.

NG code in templates/spark-rapids/ caches builds to GCS after the first run completes so subsequent similar runs will have less work to do.

[1] https://github.com/LLC-Technologies-Collier/initialization-actions/tree/template-gpu-20241219/templates/spark-rapids

cjac · 2024-12-25T03:05:27Z

/gcbrun

cjac · 2024-12-25T03:08:42Z

there's a known problem with our build system. Un momento por favor.

cjac · 2024-12-25T03:14:03Z

/gcbrun

cjac · 2024-12-25T03:15:54Z

/gcbrun

cjac · 2024-12-25T03:20:36Z

failure on 2.2-rocky9 ; I'll spin that up in my env.

cjac · 2024-12-25T03:38:43Z

oof. I forgot DKMS took so long to run. I think the test is timing out for waiting on dnf -y -q module install nvidia-driver:latest-dkms maybe

cjac · 2024-12-25T03:40:52Z

dnf -y -q install cuda-toolkit is taking a long time, too

cjac · 2024-12-25T03:43:47Z

the run takes 14m9.444s on rocky9

# echo $?
0

cjac · 2024-12-25T03:51:41Z

/gcbrun

cjac · 2024-12-25T04:08:50Z

[edited to add: I was incorrect to assume that nvliyuan/initialization-actions' master tracks GoogleCloudDataproc/initialization-actions' master]

I'm sorry, I seem to have done something to the commit history here. The diffstat looks very wrong at this point.

$ git diff master | diffstat
 CONTRIBUTING.md                         |   20 ++---
 cloudbuild/Dockerfile                   |   22 +++++-
 cloudbuild/presubmit.sh                 |    1 
 cloudbuild/run-presubmit-on-k8s.sh      |   34 +++++++--
 dask/dask.sh                            |  213 ++++++++++++++++++++++++++++++++++++++++++++--------------
 dask/test_dask.py                       |   14 +++
 gpu/Dockerfile                          |   40 +++++++++++
 gpu/README.md                           |   28 +++----
 gpu/bazel.screenrc                      |   11 +++
 gpu/env.json.sample                     |    7 +
 gpu/install_gpu_driver.sh               |  652 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---------------------------------------------------------
 gpu/manual-test-runner.sh               |   77 +++++++++++++++++++++
 gpu/run-bazel-tests.sh                  |   24 ++++++
 gpu/test_gpu.py                         |  167 ++++++++++++++++++++++++++++++++-------------
 h2o/sample-script.py                    |   11 ---
 horovod/horovod.sh                      |    4 -
 horovod/test_horovod.py                 |   13 ++-
 hue/README.md                           |  118 ++++++++++++++++++++++++++++++++
 hue/another-query.png                   |binary
 hue/create-hive-table.png               |binary
 hue/hue-ui.png                          |binary
 hue/simple-hiveql.png                   |binary
 integration_tests/dataproc_test_case.py |   18 +++-
 rapids/BUILD                            |    2 
 rapids/Dockerfile                       |   40 +++++++++++
 rapids/bazel.screenrc                   |   17 ++++
 rapids/env.json.sample                  |    7 +
 rapids/manual-test-runner.sh            |   77 +++++++++++++++++++++
 rapids/rapids.sh                        |  814 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-----------------------------------------------------------
 rapids/run-bazel-tests.sh               |   23 ++++++
 rapids/test_rapids.py                   |  137 +++++++++++--------------------------
 rapids/verify_rapids_dask.py            |   19 -----
 rapids/verify_rapids_dask_yarn.py       |   19 +++++
 spark-rapids/README.md                  |   19 -----
 spark-rapids/spark-rapids.sh            |   14 ++-
 35 files changed, 1943 insertions(+), 719 deletions(-)

cjac · 2024-12-25T04:10:12Z

my apologies. ambiguous use of 'master' here. When diffed against origin master's commit, 169e98e I see what I expect.

Signed-off-by: liyuan <[email protected]>

…ew Dockerfile

cjac · 2024-12-25T04:22:28Z

/gcbrun

cjac · 2024-12-25T04:31:56Z

many tests have passed. still standing by for full green run.

cjac · 2024-12-25T04:53:37Z

okay, that looks good.

cjac

LGTM.

cjac marked this pull request as draft October 24, 2024 04:26

cjac requested changes Oct 24, 2024

View reviewed changes

nvliyuan mentioned this pull request Nov 27, 2024

Release Checklist v24.10 NVIDIA/spark-rapids#11249

Closed

update v2410 rapids release

b1b84ae

Signed-off-by: liyuan <[email protected]>

nvliyuan and others added 7 commits December 24, 2024 20:10

update the readme doc

762c72a

Signed-off-by: liyuan <[email protected]>

update the readme doc

5bbac9a

Signed-off-by: liyuan <[email protected]>

update v2412 version

5e73027

Signed-off-by: liyuan <[email protected]>

do not recreate git clone on second pass

a12b4fe

do not clone compute-gpu-monitoring if it is already extant

8a2a968

tricking the test framework into only running our tests while using n…

57d95f2

…ew Dockerfile

gathering timing data for some long-running sections of the installer

ab83665

cjac force-pushed the rapids-v2410 branch 2 times, most recently from a3e5e99 to ab83665 Compare December 25, 2024 04:16

over-commitment on the disk space cleaned up

00caa8b

cjac self-requested a review December 25, 2024 04:53

revert to master for final squash+merge

f5acb7e

cjac approved these changes Dec 25, 2024

View reviewed changes

cjac marked this pull request as ready for review December 25, 2024 04:56

cjac merged commit 8089389 into GoogleCloudDataproc:master Dec 25, 2024
1 of 2 checks passed

update rapids version for 24.10 release #1248

update rapids version for 24.10 release #1248

Conversation

nvliyuan commented Oct 23, 2024

nvliyuan commented Oct 24, 2024

cjac commented Oct 24, 2024

cjac commented Oct 24, 2024

cjac left a comment

Choose a reason for hiding this comment

cjac Oct 24, 2024

Choose a reason for hiding this comment

cjac Oct 24, 2024

Choose a reason for hiding this comment

nvliyuan Oct 25, 2024

Choose a reason for hiding this comment

nvliyuan Oct 28, 2024

Choose a reason for hiding this comment

cjac Oct 24, 2024

Choose a reason for hiding this comment

nvliyuan Oct 25, 2024

Choose a reason for hiding this comment

nvliyuan commented Nov 5, 2024

nvliyuan commented Nov 11, 2024

cjac commented Nov 11, 2024

nvliyuan commented Dec 24, 2024

cjac commented Dec 24, 2024

nvliyuan commented Dec 25, 2024

cjac commented Dec 25, 2024

cjac commented Dec 25, 2024 • edited Loading

cjac commented Dec 25, 2024

cjac commented Dec 25, 2024

nvliyuan commented Dec 25, 2024

cjac commented Dec 25, 2024

cjac commented Dec 25, 2024

cjac commented Dec 25, 2024

cjac commented Dec 25, 2024

cjac commented Dec 25, 2024

cjac commented Dec 25, 2024

cjac commented Dec 25, 2024

cjac commented Dec 25, 2024

cjac commented Dec 25, 2024

cjac commented Dec 25, 2024

cjac commented Dec 25, 2024 • edited Loading

cjac commented Dec 25, 2024 • edited Loading

cjac commented Dec 25, 2024

cjac commented Dec 25, 2024

cjac commented Dec 25, 2024

cjac left a comment

Choose a reason for hiding this comment

cjac commented Dec 25, 2024 •

edited

Loading

cjac commented Dec 25, 2024 •

edited

Loading

cjac commented Dec 25, 2024 •

edited

Loading