Eval bug: same K and V cache type enforced for models with no V cache

[wolfpld]

Name and Version

version: 10216 (876a432)
built with Clang 22.1.8 for Linux x86_64

Operating systems

Linux

GGML backends

CUDA

Hardware

Models

GLM-5.2

Problem description & steps to reproduce

When loading GLM-5.2 with -ctk q5_1 and no -ctv:

llama_init_from_model: model does not support different K (q5_1) and V (f16) cache types

First Bad Commit

No response

Relevant log output

[fairydreaming]

This is intentional - it's to make sure that any existing logic in the code that depends on V cache type still works correctly (so we don't have to introduce ...and if it's a model without separate V cache then check K cache type instead... conditions). Simply set V cache type to the same value as K cache type and you are good to go.

[JomboKabary]

This is intentional - it's to make sure that any existing logic in the code that depends on V cache type still works correctly (so we don't have to introduce ...and if it's a model without separate V cache then check K cache type instead... conditions). Simply set V cache type to the same value as K cache type and you are good to go.

@fairydreaming I understand the desire to simplify the logic for models without a separate V-cache (like MLA), but this enforcement is a significant regression for users targeting high context windows on limited VRAM.

Previously I could run GLM-5.2 with -ctk q8_0 -ctv q5_1 fine. Keeping K at q8_0 preserved the model's attention accuracy while reducing q5_1 for V allowed me to fit more context without any significant degradation unlike enfocring both to q5_1 the degradation is severe.

By enforcing params.type_k == params.type_v in llama-context.cpp, I am now forced into two sub-optimal choices, either I set both to q8_0 I lose around 32K of context to stay within VRAM limits. Or I set both to q5_1, which as I said, the model's output quality degrades noticeably compared to the hybrid setup.

The specific check causing this is at src/llama-context.cpp:3561:

if ((model->hparams.is_mla() || model->arch == LLM_ARCH_DEEPSEEK4) && params.type_k != params.type_v) {
LLAMA_LOG_ERROR("%s: model does not support different K (%s) and V (%s) cache types\n", func, ggml_type_name(params.type_k), ggml_type_name(params.type_v));
return nullptr;
}

Even if the architecture uses a unified latent for KV, the implementation previously allowed these flags to independently affect the quantization of the cache segments, which provided a very balanced middle ground for performance.

Could we consider removing this strict enforcement, or at least allowing it as an "at your own risk" override with a special command line flag? It would restore the ability to run long-context models that were working fine just a few versions ago.

Thank you.

[fairydreaming]

This is intentional - it's to make sure that any existing logic in the code that depends on V cache type still works correctly (so we don't have to introduce ...and if it's a model without separate V cache then check K cache type instead... conditions). Simply set V cache type to the same value as K cache type and you are good to go.

@fairydreaming I understand the desire to simplify the logic for models without a separate V-cache (like MLA), but this enforcement is a significant regression for users targeting high context windows on limited VRAM.

Previously I could run GLM-5.2 with -ctk q8_0 -ctv q5_1 fine. Keeping K at q8_0 preserved the model's attention accuracy while reducing q5_1 for V allowed me to fit more context without any significant degradation unlike enfocring both to q5_1 the degradation is severe.
By enforcing params.type_k == params.type_v in llama-context.cpp, I am now forced into two sub-optimal choices, either I set both to q8_0 I lose around 32K of context to stay within VRAM limits. Or I set both to q5_1, which as I said, the model's output quality degrades noticeably compared to the hybrid setup.

Well, you only had an illusion of control there as only -ctk value was taken into account. If you run the server with --verbose you will see for example:

0.46.797.395 I llama_kv_cache: size =  671.50 MiB ( 65536 cells,  79 layers,  1/1 seqs), K (q8_0):  671.50 MiB, V (q5_1):    0.00 MiB

V cache buffer was zero-length for GLM-5.2 no matter what you set in -ctv. The reason is that there's a single common latent representation used as both K and V. It's kept in llama.cpp K buffer, but internally it's used as both K and V. That's why it's not possible to set different K and V quantizations as in DSA they are the same latent vectors kept in a single buffer. Even for MLA models since @ggerganov implemented #19067 there was no separate V cache.

Even if the architecture uses a unified latent for KV, the implementation previously allowed these flags to independently affect the quantization of the cache segments, which provided a very balanced middle ground for performance.

Yes, implementation allowed to set different quantization values, but used only K cache type and ignored the second value. Well, not really ignored as it's used for some other checks, but no cache buffer was created based on this type.

Could we consider removing this strict enforcement, or at least allowing it as an "at your own risk" override with a special command line flag? It would restore the ability to run long-context models that were working fine just a few versions ago.

Maybe the changes that you see in memory usage are the result of #25407 that finally implemented full GLM-5.2 DSA architecture? It added lightning indexer KV cache which increased overall memory usage.

[wolfpld]

Maybe the changes that you see in memory usage are the result of #25407 that finally implemented full GLM-5.2 DSA architecture? It added lightning indexer KV cache which increased overall memory usage.

There is also the MTP layer, which for some time was loaded unconditionally. All in all, I have also observed increased memory pressure with GLM 5.2 after these changes.

This is intentional - it's to make sure that any existing logic in the code that depends on V cache type still works correctly (so we don't have to introduce ...and if it's a model without separate V cache then check K cache type instead... conditions).

Seems easy to fix? Just set the V cache type to K, if model only has K, and V was not set, at ~parameter parsing stage. This gives the single code path you describe.

[fairydreaming]

Maybe the changes that you see in memory usage are the result of #25407 that finally implemented full GLM-5.2 DSA architecture? It added lightning indexer KV cache which increased overall memory usage.

There is also the MTP layer, which for some time was loaded unconditionally. All in all, I have also observed increased memory pressure with GLM 5.2 after these changes.

This is intentional - it's to make sure that any existing logic in the code that depends on V cache type still works correctly (so we don't have to introduce ...and if it's a model without separate V cache then check K cache type instead... conditions).

Seems easy to fix? Just set the V cache type to K, if model only has K, and V was not set, at ~parameter parsing stage. This gives the single code path you describe.

@wolfpld Considered this too, but the "V was not set at parameter parsing stage" doesn't seem simple to check, as V cache type is set to GGML_TYPE_F16 by default, so it's not known inside the code if someone explicitly set it to f16 or if it was simply not set by the user at all. Therefore probably another internal boolean flag like v_type_set would be needed (likely in several places) and passed around, which sounded like PITA.

[fairydreaming]

@JomboKabary If you are low on VRAM try this: #26474
It shall considerably reduce indexer cache size. In my test for q8_0 K type and 65536 cells indexer cache went from 663.00 MiB to 178.50 MiB, so ~0.5GB saved.

[JomboKabary]

Well, you only had an illusion of control there as only -ctk value was taken into account.

I think that explains it then. I wasn't aware that -ctv was a placebo for MLA models in previous versions. Seeing the VRAM usage spike on my RTX 3090 (24GB) made me think the enforcement was the culprit, but it makes total sense that it was actually the DSA indexer cache from #25407 pushing the memory footprint over the edge and causing the spill into System RAM and drop my t/s generation speed.

@fairydreaming Thanks for the technical explanation and for opening #26474 to optimize the indexer cache. I guess that ~0.5GB saving is probably what I need to keep the 200k context fully on the GPU. I'll try to test the PR branch. Thank you.

[fairydreaming]

@JomboKabary b10243 and later already contain this fix, let me know how it works for you.