Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

eckit::linalg::sparse::LinearAlgebraTorch backend #165

Open
wants to merge 1 commit into
base: develop
Choose a base branch
from

Conversation

pmaciel
Copy link
Member

@pmaciel pmaciel commented Feb 18, 2025

This PR adds a sparse lienar algebra backend to allow for GPU-based matrix multiplications (and other operations), which translates into a significant performance increase for interpolations, in the right conditions (right environment, advanced use of mir.)

It makes use of a deployed version of PyTorch (findable by CMake), specifically its lower level component "Torch", which is part of the same package (this is how it is released to the public.) I've exposed all possible hardware configuration options, contemporary. But obviously, the better development is to improve the whole workflow to avoid copies to/from the CPU/GPU, so this develolpment is purelly a stepping stone -- it has already allowed me to run both mars-client (C) and pgen on GPUs, and of course mir. It would be great to follow this up with a publication. Possibly, this could be configurable in earthkit-regrid for maximum marketing :-)

I've held back this development for several months, but I couldn't find a definite response on when to post it -- here it is.

@codecov-commenter
Copy link

codecov-commenter commented Feb 18, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 64.09%. Comparing base (f1591f4) to head (b548c96).

Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #165      +/-   ##
===========================================
- Coverage    64.09%   64.09%   -0.01%     
===========================================
  Files         1083     1083              
  Lines        55759    55759              
  Branches      4120     4120              
===========================================
- Hits         35738    35736       -2     
- Misses       20021    20023       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@wdeconinck
Copy link
Member

This looks neat!
I'd like to test this before merging, preferably on our HPC.
Could you get a working Torch that I could link into e.g. with a setup using the nvidia compiler?

As a side, it would be nice to clean up and remove unused backends (armadillo, viennacl, ... )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants