Discussion: your claims

[electroglyph]

"Beats Q8_0 perplexity at half the size -- and even beats F16"

I'm sure you know that perplexity pales against KLD for this kind of comparison, so claiming better than Q8 seems pretty disingenuous.

I don't think I need to explain how claiming the quantization is better than the full precision model is kind of a mathematically insane claim.

If you quantize a model, and then the quantized model is better at one benchmark than the base model, that's still quantization error and it has most likely regressed somewhere else.

Apex Quality has ~2.5x worse KLD than Q8_0, you should compare it to other bit sizes for a better size/accuracy comparison.

Apex Quality has ~4.5x worse KLD than Unsloth UD-Q8_K_XL

[mudler]

Apex Quality has ~2.5x worse KLD than Q8_0, you should compare it to other bit sizes for a better size/accuracy comparison.

Apex Quality has ~4.5x worse KLD than Unsloth UD-Q8_K_XL

You are not comparing sizes, and you are even just selecting Quality, which by per-se is like selecting what you prefer best to steer to your view, I-Matrix versions are the ones that have better results.

Benchmarks are in the repo, and everyone can draw conclusions, there is literally nothing to hide, and I welcome criticism to refine this methodology even better to provide better quants, but I dislike random comments which are cheap to write without even taking the time to review and test the models.

Look, to me it's not about marketing bullshit, but it's to really have better models to run on my restricted HW because I've found other quants really bad when it comes at coding. I've could have kept this for me, but I work in the open so here it is why it's in the wild.

After the release, independent benchmarks confirmed that apex-quants does indeed have very good results, especially at long context sizes compared to other quants. For what it stands it provides very strong capabilities which I've typically find quite short with other quants.

The claims are relative to the sizes, this is my focus here. The fact is that punch benchmarks above if you compare to other quants relative to the other size, and that's why you can see in the table Q5 from Unsloth which you should compare it against. Benchmarks shows that numbers on Q8 are comparable and in the same range with the quants, at half of the size. There is a loss which is neglectable, but it still saves much more space for a small quality degradation. Now you can read the data from the angle you prefer, but the benchmarks.. are there.

That being said, if you don't like the quants, just don't use it, I don't literally care and do not like attacks that are cheap to write without taking even the efforts of reading the data, because this work takes my free time away, and people like you really push it below the line.

[lee-b]

I do agree that there are some (not insurmountable) issues with the claims, and the presentation, and just that some clarifications that would be helpful, too.

Beats Q8_0 perplexity at half the size -- and even beats F16.

As @electroglyph suggests, I think this is a questionable claim and would need much stronger evidence (so should likely just be dropped). It doesn't seem to make sense that you can train a model on trillions of parameters with massive compute, get certain results at F16, then apply a few post-training tweaks on the weights, and actually improve the accuracy, across the full domain of tasks. I think it's probably overfitting, if it does achieve better results on benchmarks? Point is: don't claim it just because benchmarks SEEM to show it.

exploiting the structural sparsity of Mixture-of-Experts models to achieve lossless compression that uniform quantization cannot.

This "lossless compression" claim seems too strong, or at least too wide when it's obviously not lossless at all quants. The next sentence already mitigates this claim, but at that point, the claim has been made, unqualified. I don't think this is an attempt at deception: tighter wording would help, is all.

APEX assigns quantization precision per tensor type and per layer

I thought UD 2.0 quants already did this kind of thing? Perhaps not with MoE awareness in WHAT they quantize, but I thought they did the same thing -- quantization precision per tensor type and per layer? I could use clarification on why APEX is better than UD 2.0, which seems to be essentially state of the art in the GGUF world... at least according to unsloth and their own charts ;)

As a helpful clarification of that, the charts are lacking some data that would be helpful:

Mini is the lowest quant that's claimed "safe" for MiniMax M2.7, and it's a compelling size between various UD quants that may JUST squeeze 128K context at usable quality into 4x24GB rigs for that model in particular. BUT, in the "Perplexity vs. Size" chart in this repo, Mini is way out there on its own, with bartowski shown as way below it:

The chart is literally presenting sparse data around Mini here, just as it looks visually (of course). I'm not sure that's a great comparison as bartowski is already known to be worse than UD 2.0 and probably AesSedai. Additional samples, from similarly sized UD/bartowski quants etc. should be included to convince that APEX is superior, or be clear about any similarities, imho.

I feel this is particularly important here, the lower quants have the bigger losses, and a few hundred MB can be more important at this scale (both in maintaining usability, and in squeezing into devices), and so this is where one can really show off how APEX performs (or show where improvement potential remains). I would really like to see i1-UD-IQ2* and UD-i1-Q2* quants in comparison. Also, my own personal experience is that it generally matters little at this quant level, what the actual quant is called; file size becomes the dominant factor in accuracy loss. An ~12GB model file is ~9% bigger than 11GB, so comparisons for closer file sizes between quant methods / types would be welcome.

This size-similar comparison point is important, especially because one of the repo's own introductory claims is about size benefits:

Beats Q8_0 perplexity at half the size -- and even beats F16. APEX outperforms Unsloth Dynamic 2.0 (UD) quantizations on perplexity, HellaSwag, and inference speed while being 2x smaller: APEX I-Quality (21.3 GB) achieves PPL 6.552 and 83.5% HellaSwag vs Unsloth UD-Q8_K_XL (45.3 GB) at PPL 6.536 and 82.5% HellaSwag. At the consumer tier, APEX Mini (12.2 GB) beats bartowski IQ2_M on every metric.

But note the (unintended, I'm sure) switcheroo here: this paragraph starts off saying that it beats other models at half the size. But the final "consumer tier" factoid is actually the inverse: the APEX model is beating bartowski (again, not the best to compare with) but it's bigger, so that would be expected in my book, and doesn't support the paragraph's wider claim.

I don't mean to be all negative in this comment about accuracy/presentation, though. On a more positive note, on the Accuracy vs. Benchmark Comparison chart, the legend is covering what look like perhaps the most convincing results :D Moving that legend to the top right would help a lot.