Cannot get GPU to work #1944

ape-nq · 2024-04-02T10:05:11Z

ape-nq
Apr 2, 2024

I've tried v2.11.0-cublas-cuda12 and compiling current main branch with Mistral.

When doing chat completion log says that layers are offloaded to GPU:

 GRPC(mistral-7b-openorca.Q6_K.gguf-127.0.0.1:32909): stderr ggml_cuda_init: found 1 CUDA devices:
 GRPC(mistral-7b-openorca.Q6_K.gguf-127.0.0.1:32909): stderr   Device 0: NVIDIA GeForce RTX 4080, compute capability 8.9, VMM: yes
 GRPC(mistral-7b-openorca.Q6_K.gguf-127.0.0.1:32909): stderr llm_load_tensors: ggml ctx size =    0.22 MiB
 GRPC(mistral-7b-openorca.Q6_K.gguf-127.0.0.1:32909): stderr llm_load_tensors: offloading 5 repeating layers to GPU
 GRPC(mistral-7b-openorca.Q6_K.gguf-127.0.0.1:32909): stderr llm_load_tensors: offloaded 5/33 layers to GPU
 GRPC(mistral-7b-openorca.Q6_K.gguf-127.0.0.1:32909): stderr llm_load_tensors:        CPU buffer size =  5666.11 MiB
 GRPC(mistral-7b-openorca.Q6_K.gguf-127.0.0.1:32909): stderr llm_load_tensors:      CUDA0 buffer size =   853.28 MiB

But I get error messages from llama.cpp:

 GRPC(mistral-7b-openorca.Q6_K.gguf-127.0.0.1:32909): stderr ggml_cuda_compute_forward: ADD failed
 GRPC(mistral-7b-openorca.Q6_K.gguf-127.0.0.1:32909): stderr GGML_ASSERT: /root/LocalAI/backend/cpp/llama/llama.cpp/ggml-cuda.cu:2392: false

Also I'm seeing weird values that look like pointers (e.g. TopP, TopK, Temperature) in the config in debug log:

Configuration read: &{PredictionOptions:{Model:mistral-7b-openorca.Q6_K.gguf Language: N:0 TopP:0xc0006f7d38 TopK:0xc0006f7d30 Temperature:0xc0006
f7d10 Maxtokens:0xc0006f7dc0 Echo:false Batch:0 IgnoreEOS:false RepeatPenalty:0 Keep:0 FrequencyPenalty:0 PresencePenalty:0 TFZ:0 TypicalP:0 Seed:0xc0006f7df0 NegativePrompt: RopeFreq
Base:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Name:mistral F16:0xc0006f7cf8 Threads:0xc0006f7d88 Debug:0xc00047d838 Roles:map[] Embeddings
:false Backend:llama-cpp

My mistral.yaml:

backend: llama-cpp
context_size: 4096
f16: true
gpu_layers: 5
mmap: true
name: mistral
parameters:
  model: mistral-7b-openorca.Q6_K.gguf
  temperature: 0.2
  top_k: 40
  top_p: 0.95
stopwords:
- <|im_end|>
template:
  chat: chatml-block
  chat_message: chatml
  completion: completion
threads: 4

Any ideas, what I'm doing wrong?

ape-nq · 2024-04-02T14:10:21Z

ape-nq
Apr 2, 2024
Author

Also getting errors when running make BUILD_TYPE=cublas GRPC_BACKENDS=backend-assets/grpc/llama-cpp test-llama-gguf:

{"level":"debug","time":"2024-04-02T14:07:36Z","message":"GRPC(codellama-7b-instruct.Q4_K_M.gguf-127.0.0.1:41969): stderr ggml_cuda_compute_forward: RMS_NORM failed"}
{"level":"debug","time":"2024-04-02T14:07:36Z","message":"GRPC(codellama-7b-instruct.Q4_K_M.gguf-127.0.0.1:41969): stderr GGML_ASSERT: /root/LocalAI/backend/cpp/llama/llama.cpp/ggml-cuda.cu:2392: false"}
[127.0.0.1]:47426 500 - POST /v1/chat/completions

API test API with ephemeral models Applying models [It] runs openllama gguf(llama-cpp) [llama-gguf]
/root/LocalAI/core/http/api_test.go:465

  [FAILED] Unexpected error:
      <*openai.APIError | 0xc00053ae60>: 
      error, status code: 500, message: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF
      {
          Code: <int>500,
          Message: "could not load model: rpc error: code = Unavailable desc = error reading from server: EOF",
          Param: nil,
          Type: "",
          HTTPStatusCode: 500,
          InnerError: nil,
      }
  occurred
  In [It] at: /root/LocalAI/core/http/api_test.go:492 @ 04/02/24 14:08:22.4

1 reply

ape-nq Apr 2, 2024
Author

I tried the cuda12 release builds:

https://github.com/mudler/LocalAI/releases

Start with debug:

$ ./local-ai-cuda12-Linux-x86_64 --debug

curl from Quickstart guide:

curl http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{ "model": "mistral", "messages": [{"role": "user", "content": "How are you doing?", "temperature": 0.1}] }'

I'm getting two types of error:

CUDA error: the provided PTX was compiled with an unsupported toolchain.
ggml_cuda_compute_forward: ADD failed
- https://github.com/mudler/LocalAI/releases/tag/v2.11.0

Am I missing some flag or config somewhere?

ape-nq · 2024-04-03T09:32:35Z

ape-nq
Apr 3, 2024
Author

Is my driver simply too old?

$ nvidia-smi
Wed Apr  3 09:31:16 2024       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.147.05   Driver Version: 525.147.05   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:A1:00.0 Off |                  N/A |
| 50%   37C    P8     4W / 320W |      6MiB / 16376MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

0 replies

ape-nq · 2024-04-03T09:34:02Z

ape-nq
Apr 3, 2024
Author

nvcc compiler version:

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Mon_Apr__3_17:16:06_PDT_2023
Cuda compilation tools, release 12.1, V12.1.105
Build cuda_12.1.r12.1/compiler.32688072_0

0 replies

ape-nq · 2024-04-03T12:08:45Z

ape-nq
Apr 3, 2024
Author

I switched machines and believe I have compatible CUDA driver and toolkit now:

[email protected]:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Mon_Apr__3_17:16:06_PDT_2023
Cuda compilation tools, release 12.1, V12.1.105
Build cuda_12.1.r12.1/compiler.32688072_0
[email protected]:~$ nvidia-smi 
Wed Apr  3 12:07:25 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3090        On  | 00000000:01:00.0 Off |                  N/A |
|  0%   37C    P8              24W / 420W |      3MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

But I'm still getting the same errors.

0 replies

ape-nq · 2024-04-05T12:59:53Z

ape-nq
Apr 5, 2024
Author

Got the v2.11.0 release binary local-ai-cuda12-Linux-x86_64 it working with these Versions:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Wed_Nov_22_10:17:15_PST_2023
Cuda compilation tools, release 12.3, V12.3.107
Build cuda_12.3.r12.3/compiler.33567101_0
Fri Apr  5 12:48:02 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3080        On  |   00000000:05:00.0 Off |                  N/A |
|  0%   25C    P8             26W /  320W |    6811MiB /  10240MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
+-----------------------------------------------------------------------------------------+

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Cannot get GPU to work #1944

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 5 comments 1 reply

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

Cannot get GPU to work #1944

Uh oh!

Uh oh!

ape-nq Apr 2, 2024

Replies: 5 comments · 1 reply

Uh oh!

Uh oh!

ape-nq Apr 2, 2024 Author

Uh oh!

ape-nq Apr 2, 2024 Author

Uh oh!

ape-nq Apr 3, 2024 Author

Uh oh!

ape-nq Apr 3, 2024 Author

Uh oh!

ape-nq Apr 3, 2024 Author

Uh oh!

ape-nq Apr 5, 2024 Author

ape-nq
Apr 2, 2024

Replies: 5 comments 1 reply

ape-nq
Apr 2, 2024
Author

ape-nq Apr 2, 2024
Author

ape-nq
Apr 3, 2024
Author

ape-nq
Apr 3, 2024
Author

ape-nq
Apr 3, 2024
Author

ape-nq
Apr 5, 2024
Author