Unable to specify dtype

[drewwells]

Bug description

Warning in the logs about specifying dtype. It's infering the support of the CPU rather than the metal GPU I am using (apple m4).

Steps to reproduce

Start chat-ui with a model I'm using Deepseek-R1 gguf
Start the chat-ui, view the logs

Screenshots

Context

Logs

docker-compose-chat-ui[3275734]: chat-ui  | dtype not specified for "model". Using the default dtype (fp32) for this device (cpu).

Specs

• OS: OS X
• Browser: Orion
• chat-ui commit: chat-ui-db Docker Image: f82eabd33648

Config

MODELS=`[
    {
      "name": "unsloth/DeepSeek-R1-Distill-Llama-70B-GGUF",
      "id": "unsloth/DeepSeek-R1-Distill-Llama-70B-GGUF",
      "displayName": "unsloth/deepseek-r1-distill-llama-70b",
      "description": "Size: 74.98gb",
      "modelUrl": "https://huggingface.co/unsloth/DeepSeek-R1-Distill-Llama-70B-GGUF",
      "websiteUrl": "https://unsloth.ai/blog/deepseek-r1",
      "logoUrl": "https://cdn-avatars.huggingface.co/v1/production/uploads/62ecdc18b72a69615d6bd857/E4lkPz1TZNLzIFr_dR273.png",
      "chunkCharLength": 768,
      "preQuery": "query: ",
      "prePassage": "passage: ",
      "parameters": {
        "temperature": 0.6,
      },
...

Notes

I can't find anything about dtype in this repo, so should be helpful just to help others find some background information here. I believe for my m4 gpu, fp32 that it keeps choosing is not ideal.

[nsarrazin]

Hi, thanks for opening the issue, I think (though I could be wrong) that GPU support is not available with transformers.js using node.

However it would be nice to have an option to control the dtype for sure.