[DOC]: Improve documentation for cmake usage

[mwiesenberger]

Is this a duplicate?

• I confirmed there appear to be no duplicate issues for this bug and that I agree to the Code of Conduct

Is this for new documentation, or an update to existing docs?

Update

Describe the incorrect/future/missing documentation

I am trying to write a CMakeLists.txt file for my project that depends on thrust. For the sake of argument here is a minimal main

// main.cpp
#include <iostream>
#include <thrust/device_vector.h>

int main()
{
   thrust::device_vector<double> v(100, 7);
   v[3] = 8;
    std::cout << "Element is "<<v[3]<<"\n";
    return 0;
}

I want to compile this program for various values of the the THRUST_DEVICE_SYSTEM i.e.

• for CPP: g++ -O2 -Wall -std=c++17 -DTHRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_CPP main.cpp -o main -I<path/to/thrust/thrust>
• for OMP: g++ -fopenmp -O2 -Wall -std=c++17 -DTHRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_OMP main.cpp -o main -I<path/to/thrust/thrust>
• for CUDA: nvcc --compiler-bindir g++ -O2 -x cu -std=c++17 main.cpp -o main -I<path/to/thrust/thrust>

My problem is that I do not find instructions of how to achieve this with the cmake configuration provided in cccl. In the cccl README https://github.com/NVIDIA/cccl there is a (short) paragraph telling me that cccl should be configured with cmake and I can install it using

git clone https://github.com/NVIDIA/cccl.git
cd cccl
cmake --preset install -DCMAKE_INSTALL_PREFIX=/usr/local/
cd build/install
ninja install

(which I did) and to follow the example https://github.com/NVIDIA/cccl/tree/main/examples/basic for the rest. That example essentially tells me to use the CPM package manager to find cccl and that I can compile for a GPU, however the other two possibilities (CPP and OMP) are not discussed.
This link https://nvidia.github.io/cccl/thrust/cmake_options.html essentially lists a couple of options that I should see using cmake-gui but it does not tell me anything about what cmake targets are defined that have those options.
Finally, I found this hidden https://github.com/NVIDIA/cccl/blob/main/lib/cmake/thrust/README.md which tells me that there is a thrust_create_target function but since the file is so tugged away in a sub-folder I am unsure if this is still a maintained feature or just a remainder of the thrust integration into cccl.

My first issue is that I now have four different places that describe cmake behaviour in cccl/thrust but none describe how I achieve my above goal. Could you update your documentation in this regard?

Here is what I tried so far

# Attempt 1
cmake_minimum_required( VERSION 3.26)
project( thrust-test VERSION 0.1 LANGUAGES CXX CUDA)
find_package( cccl 2.7 REQUIRED)
add_executable( main main.cu ) # Rename main.cpp to main.cu else build error
target_link_libraries( main PRIVATE CCCL::CCCL)

# cmake -B build .
# cd build
# cmake --build .

Configuring and building leads to a valid GPU build with THRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_CUDA. But how do I compile for CPP and OMP?

# Attempt 2
# Same CMakeLists.txt as above but
cmake -B build . -DTHRUST_DEVICE_SYSTEM=CPP
cd build
cmake --build . --verbose

Here, I get a warning from cmake that THRUST_DEVICE_SYSTEM was not used by the project and the build still compiles for a GPU. This leaves me a bit at a loss as to how the options from https://nvidia.github.io/cccl/thrust/cmake_options.html are supposed to appear?

# Attempt 3
cmake_minimum_required( VERSION 3.26)
project( thrust-test VERSION 0.1 LANGUAGES CXX CUDA)
find_package( Thrust 2.7 REQUIRED)
add_executable( main main.cu ) # Rename main.cpp to main.cu else build error
target_link_libraries( main PRIVATE Thrust::Thrust)

# cmake -B build .
# cd build
# cmake --build .

Works the same as Attempt 1? So which should I use? Is there a difference?

# Attempt 4
cmake_minimum_required( VERSION 3.26)
project( thrust-test VERSION 0.1 LANGUAGES CXX CUDA)
find_package( Thrust 2.7 REQUIRED)
thrust_create_target( ThrustTarget FROM_OPTIONS DEVICE CPP)
add_executable( main main.cpp ) # Use cpp file ending
target_link_libraries( main PRIVATE ThrustTarget)

# cmake -B build . -DTHRUST_DEVICE_SYSTEM=OMP
# cd build
# cmake --build .

Now this actually compiles my target for CPP and OMP, but unfortunately it fails for CUDA because the file ending is wrong and the c++ compiler is invoked instead of nvcc. Also, contrary to the documentation https://github.com/NVIDIA/cccl/blob/main/lib/cmake/thrust/README.md the THRUST_DEVICE_SYSTEM option does not appear in my cmake-gui only THRUST_DISPATCH_TYPE. I would also like to selectively load the CUDA language only if the user wants to compile for cuda. How do I do this?

At this point I don't know how to proceed, I am still learning CMake and I would greatly appreciate a working example that achieves what I want in cmake.

If this is a correction, please provide a link to the incorrect documentation. If this is a new documentation request, please link to where you have looked.

Places I have looked:
https://github.com/NVIDIA/cccl
https://github.com/NVIDIA/cccl/tree/main/examples/basic
https://nvidia.github.io/cccl/thrust/cmake_options.html
https://github.com/NVIDIA/cccl/blob/main/lib/cmake/thrust/README.md

[alliepiper]

It is a bit of a mess -- things have changed over the years and need to be cleaned up. I'll look into updating the docs to make this more cohesive. To answer your question:

https://nvidia.github.io/cccl/thrust/cmake_options.html is for configuring and building the thrust tests/examples as part of a developer build. The options listed there are not available in the installed CMake packages.

The CCCL package provides Thrust and is the preferred way to include Thrust. By default, CCCL defines a CCCL::Thrust target which is configured from CMake options that can be discovered by using the ccmake/cmake-gui utilities or inspecting the CMakeCache.txt file in the build directory. This CCCL::Thrust target is also available through the CCCL::CCCL target transitively. The option to change the device system is CCCL_THRUST_DEVICE_SYSTEM.

Your first example was most of the way there, it just needed to define that variable:

cmake_minimum_required( VERSION 3.26)
project( thrust-test VERSION 0.1 LANGUAGES CXX CUDA)
find_package( CCCL 2.7 REQUIRED)
add_executable( main main.cu )
target_link_libraries( main PRIVATE CCCL::CCCL)

# cmake -B build . -DCCCL_THRUST_DEVICE_SYSTEM=TBB # (or OMP)
# cd build
# cmake --build .

https://github.com/NVIDIA/cccl/blob/main/lib/cmake/thrust/README.md describes in more detail how to configure thrust as a user. The thrust_create_target(...) CMake function is used to generate interface targets for a specific host/device configuration of Thrust. Creating a thrust target that uses OpenMP/TBB is detailed here.

If you want to use both TBB and OMP from the same build you can use thrust_create_target directly:

cmake_minimum_required( VERSION 3.26)
project( thrust-test VERSION 0.1 LANGUAGES CXX CUDA)
find_package(CCCL 2.7 REQUIRED)

# Create two thrust targets, one that uses TBB, another that uses OpenMP:
thrust_create_target(ThrustTBB DEVICE TBB)
thrust_create_target(ThrustOMP DEVICE OMP)

add_executable( main_omp main.cu )
target_link_libraries( main_omp PRIVATE ThrustOMP)
add_executable( main_tbb main.cu )
target_link_libraries( main_tbb PRIVATE ThrustTBB)

[mwiesenberger]

Thank you for the fast reply I appreciate the help.

I have tried your solutions and I have a couple of follow-up questions

• In neither of the examples I can see options appearing in the cmake-gui. I can confirm that -DCCCL_THRUST_DEVICE_SYSTEM=OMP works as advertised, it is just that CCCL_THRUST_DEVICE_SYSTEM does not appear in cmake-gui nor in ccmake. Is there something special I need to activate in my CMakeLists.txt for this to happen?
• Is there a prefered way to check if the CCCL_THRUST_DEVICE_SYSTEM is set to CUDA by the user and only then load the CUDA language in cmake? On some systems I do not have CUDA installed nor have a GPU and thus, if I set it to CPP I would not want to have to depend on CUDA
• On the issue of the file ending: whenever the DEVICE is set to CUDA I am forced to name my file main.cu instead of main.cpp. This is not the case for CPP OMP or TBB. I do not want to name my file main.cu because I write no cuda kernels in there and it is a pure C++ file. How do I set the -x cu compiler option if the user sets DEVICE to CUDA?
• Is the THRUST_DEVICE_SYSTEM macro still the preferred way to check in code (in the cpp file not in cmake) what the device backend is? Or does this now also change to CCCL_THRUST_DEVICE_SYSTEM?

// main.cpp
#include <iostream>
#include <thrust/device_vector.h>

int main()
{
#if THRUST_DEVICE_SYSTEM==THRUST_DEVICE_SYSTEM_CPP
   std::cout  << "Thrust device system is CPP\n";
#elif THRUST_DEVICE_SYSTEM==THRUST_DEVICE_SYSTEM_TBB
   std::cout  << "Thrust device system is TBB\n";
#elif THRUST_DEVICE_SYSTEM==THRUST_DEVICE_SYSTEM_OMP
   std::cout  << "Thrust device system is OMP\n";
#elif THRUST_DEVICE_SYSTEM==THRUST_DEVICE_SYSTEM_CUDA
   std::cout  << "Thrust device system is CUDA\n";
#endif
   return 0;
}

[alliepiper]

CCCL_THRUST_DEVICE_SYSTEM does not appear in cmake-gui nor in ccmake

These variables are marked as advanced and hidden by default, since most users of CCCL just want the CUDA backend. You'll need to toggle advanced mode (press t in ccmake, check the Advanced box in cmake-gui) to make them appear.

Is there a prefered way to check if the CCCL_THRUST_DEVICE_SYSTEM is set to CUDA by the user and only then load the CUDA language in cmake?

Something like this should work:

project(Foo CXX) # CXX always enabled

if (CCCL_THRUST_DEVICE_SYSTEM STREQUAL CUDA)
  enable_language(CUDA) # Enable CUDA only when needed
endif()

find_package(CCCL)
...

How do I set the -x cu compiler option if the user sets DEVICE to CUDA?

This can be overridden by setting the LANGUAGE source file property:

set_source_files_properties(main.cpp PROPERTIES LANGUAGE CUDA)

Is the THRUST_DEVICE_SYSTEM macro still the preferred way to check in code (in the cpp file not in cmake) what the device backend is?

Yes, this is still the preferred way to check in-source which system is used.