support by channel group on k of pa second token - #20
Open
tiger100256-hu wants to merge 4 commits into
Open
Conversation
Signed-off-by: HU Yuan2 <yuan2.hu@intel.com>
Comment on lines
+968
to
+969
| -DKV_CACHE_COMPRESSION_BY_CHANNEL_SUBGROUP={kvcache_quantization_by_channel_group} | ||
| -DKV_CACHE_COMPRESSION_BY_CHANNEL={kvcache_quantization_by_channel} |
Owner
There was a problem hiding this comment.
Why no merge the two cases?
Comment on lines
+49
to
+50
| #define K_SCALE_ZP_SIZE 0 // no scale/zp | ||
| #define V_SCALE_ZP_SIZE 0 // no scale/zp |
Owner
There was a problem hiding this comment.
Sounds no need to split KV_SCALE_ZP_SIZE
Comment on lines
+35
to
+42
| #if KV_CACHE_COMPRESSION_BY_CHANNEL_SUBGROUP | ||
| #define SUB_GROUP_SIZE 16 | ||
| #define GROUPS_PER_BLOCK (KV_BLOCK_SIZE / SUB_GROUP_SIZE) // 256/16 = 16 | ||
| #define K_SCALE_ZP_SIZE (4 * GROUPS_PER_BLOCK) // scale/zp bytes | ||
| #define V_SCALE_ZP_SIZE 4 // scale/zp bytes | ||
| #elif KV_CACHE_COMPRESSION_BY_CHANNEL | ||
| #define K_SCALE_ZP_SIZE 4 // scale/zp bytes | ||
| #define V_SCALE_ZP_SIZE 4 // scale/zp bytes |
Owner
There was a problem hiding this comment.
Please merge the two cases... we only need to extend KV_CACHE_COMPRESSION_BY_CHANNEL with a SUB_GROUP_SIZE.. if it equals to KV_BLOCK_SIZE, then it is only a group.
Comment on lines
58
to
62
| kvcache_quantization_by_token = int(kv_cache_quantization_mode == "by_token") | ||
| print(f"{kv_cache_quantization_mode=}, {kvcache_quantization_by_token=}") | ||
| kvcache_quantization_by_channel = int(kv_cache_quantization_mode == "by_channel") | ||
| kvcache_quantization_by_channel_group = int(kv_cache_quantization_mode == "by_channel_group") | ||
| print(f"{kv_cache_quantization_mode=}, {kvcache_quantization_by_token=}, {kvcache_quantization_by_channel_group=}, {kvcache_quantization_by_channel=}") | ||
|
|
Owner
There was a problem hiding this comment.
Probably by refer to XuChen's PR to use an enum type is better: compressed_kvcache = 0, 1, 2 for fp16, by_token, by_channel respectively.
…nnnel" mode use by channel and sub_group_size to replace the "by channel_group" mode add enum for different mode print gpu compile kernel params Signed-off-by: HU Yuan2 <yuan2.hu@intel.com>
Signed-off-by: HU Yuan2 <yuan2.hu@intel.com>
ceciliapeng2011
force-pushed
the
dev/quant_by_channel
branch
from
March 30, 2026 08:18
e08682a to
226cfdb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.