Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I wish I could use cuDF to do [...]
We have been experimenting with JIT-compilation in CUDF and gotten impressive advantages, like:
- Lower register pressure
- Higher throughput due to operator fusion and better codegen
- Lower memory usage due to the compound operations not requiring intermediate memory allocations
- Support more complex operators and types
The pressing challenge remains with the JIT kernel compilation time.
This is primarily expensive because we use C++/CUDA for JIT-time compilation.
An NVRTC time trace shows that the majority of the uncached compilation time is spent in the CUDA C++ Frontend, accounting for ~90% of the offline compilation time:
profile.json (view with chrome://tracing / perfetto)
For latency-sensitive workloads with dynamic expressions, this compilation time is prohibitively expensive.
Describe the solution you'd like
A clear and concise description of what you want to happen.
LTO-IR is an intermediate program representation similar to LLVM IR. It is intended to be very close to the final assembly/SASS output.
With LTO-IR, we'd only pay for the JIT-time cost of linking the compile-time LTO-IR kernel with the runtime-selected operators and a final pass to generate SASS from it.
The cost of LTO-IR-linking should be significantly lower (yet to be measured) than going from CUDA C++ to SASS, as we'll only be processing the relevant partially-optimized code at runtime.
While the operators can be CUDA code, we still risk paying the cost of compiling CUDA/C++ code at runtime.
The way to solve this would be to have precompiled LTO-IR operators in a single LTO-IR program compiled from our existing CUDA operators; we would at runtime select and plumb them together using LTO-IR.
This will be trivial to do with the in-progress Row-IR (#19467) as ASTs only have a number of operators.
We can always fall back to CUDA/C++ if requested by the user or for more complex operators.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
- Pre-compiled headers (still in evaluation)
- User-controlled caching of programs
Additional context
Add any other context, code examples, or references to existing implementations about the feature request here.
Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I wish I could use cuDF to do [...]
We have been experimenting with JIT-compilation in CUDF and gotten impressive advantages, like:
The pressing challenge remains with the JIT kernel compilation time.
This is primarily expensive because we use C++/CUDA for JIT-time compilation.
An NVRTC time trace shows that the majority of the uncached compilation time is spent in the CUDA C++ Frontend, accounting for ~90% of the offline compilation time:
profile.json (view with
chrome://tracing/ perfetto)For latency-sensitive workloads with dynamic expressions, this compilation time is prohibitively expensive.
Describe the solution you'd like
A clear and concise description of what you want to happen.
LTO-IR is an intermediate program representation similar to LLVM IR. It is intended to be very close to the final assembly/SASS output.
With LTO-IR, we'd only pay for the JIT-time cost of linking the compile-time LTO-IR kernel with the runtime-selected operators and a final pass to generate SASS from it.
The cost of LTO-IR-linking should be significantly lower (yet to be measured) than going from CUDA C++ to SASS, as we'll only be processing the relevant partially-optimized code at runtime.
While the operators can be CUDA code, we still risk paying the cost of compiling CUDA/C++ code at runtime.
The way to solve this would be to have precompiled LTO-IR operators in a single LTO-IR program compiled from our existing CUDA operators; we would at runtime select and plumb them together using LTO-IR.
This will be trivial to do with the in-progress
Row-IR(#19467) as ASTs only have a number of operators.We can always fall back to CUDA/C++ if requested by the user or for more complex operators.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context, code examples, or references to existing implementations about the feature request here.