IQ3m
Gives out gibberish. Tried to redownload to ensure it wasn't just a fluke and still, it gives incoherent responses even on neutralized text gen settings. Maybe gonna try different quants.
Yeah, it just iq3m. IQ3ks works fine.
static IQ3 quants are generally of very low quality - the Q2_K is usually far better. If you want, I can make imatrix quants for this model, where IQ3 should behave way better.
Oh, sorry for my ignorance. Now I know a little more, thanks.
And no, I'll probably stick to mistral based models for the time :)
Oh, sorry for my ignorance. Now I know a little more, thanks.
We had no idea about this either until recently. It was a total surprise that the non-imatrix IQ3 quants are so bad (and that the imatrix IQ3 quants behave normally). We decided to stop generating them and at some point, we will stop displaying them for old repos.