In Docker Python doesn't see quant_cuda library.

[TomekGniazdowski]

To make it work "site-packages/quant_cuda-0.0.0-py3.9-linux-x86_64.egg/llmtune/engine/quant" should be added to the PATH.

[imsrgadich]

Do this to get it to work.

import sys
sys.path.extend(['/home/<user_env_path>/llm-tools/lib/python3.11/site-packages/quant_cuda-0.0.0-py3.11-linux-x86_64.egg/llmtune/engine/quant'])

[osbm]

I have made a packaged version of that cuda module

pip install git+https://github.com/osbm/quant_cuda

But your machine does need to be able to compile cuda (just like this repo).

[osbm]

And as of now i could not find a way to activate gpus during build time of a docker image. And it is troublesome to install cuda compiled python packages in docker. I am currently doing it by specifying the architecture.

Dockerfile:

FROM nvidia/cuda:12.1.1-cudnn8-devel-ubuntu20.04
ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y git build-essential python3 python3-pip
# there is no gpu access during docker build time :(
ENV TORCH_CUDA_ARCH_LIST Turing
RUN pip install git+https://github.com/osbm/quant_cuda

And run:

docker build -t quant_cuda .
docker run quant_cuda python3 -c "import torch; import quant_cuda; print(dir(quant_cuda))"

Prints out:

==========
== CUDA ==
==========

CUDA Version 12.1.1

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

WARNING: The NVIDIA Driver was not detected.  GPU functionality will not be available.
   Use the NVIDIA Container Toolkit to start this container with GPU support; see
   https://docs.nvidia.com/datacenter/cloud-native/ .

['__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'vecquant2matmul', 'vecquant2recons_v2', 'vecquant3matmul', 'vecquant4matmul', 'vecquant4recons_v1', 'vecquant4recons_v2', 'vecquant8matmul']

But sadly this image is 17.6GB. I need to find a way to lower this, but here you go