4 bit and 8 bit bnb quants only generate empty strings or one token repeated endlessly

#32

by nicorinn-google - opened Jul 24, 2024

Jul 24, 2024

I'm using the same exact notebook for the 27b-it and 9b-it versions, so the issue is definitely related to this model. Any ideas of what the cause may be?

mdouglas

Aug 1, 2024

Hi @nicorinn-google , please use torch_dtype=torch.bfloat16 when loading with from_pretrained(). There's a PR to update the model card examples here: #33.

lkv

Google org 4 days ago

Hi @mdouglas , Kindly update the bitsandbytes examples to load the model using torch_dtype=torch.bfloat16. I have tested and reproduced. Please refer this gist file for reference. If you have any concerns let me know will assist you.

Thank you.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment