feat: MoE expert cache — keep hot routed experts resident in VRAM - #5

Merged

thecodacus merged 23 commits into

perffrom

fable5/moe-expert-cache

Jul 24, 2026

Commits on Jul 24, 2026

feat: llama-moe-trace tool + expert-cache policy simulator
thecodacus
committed
3d3efff
fix: moe-trace — common_init_from_params returns ptr with accessors
thecodacus
committed
d7506be
fix: moe-trace — include arg.h for common_params_parse
thecodacus
committed
380d831
fix: moe-trace — take output path from MOE_TRACE_OUT env, -o not in this arg set
thecodacus
committed
0bc799a
fix: moe-trace — topk tensor is a strided view; copy full byte range and index by nb
thecodacus
committed
5c7f475
feat: moe-trace simulator v2 — model upload bytes/token and miss/upload cost per policy
thecodacus
committed
a9de136
feat: mul_mat_id tolerates id=-1 (skip) on CPU and CUDA fallback path
thecodacus
committed
f009677
feat: id=-1 skip support in CUDA mul_mat_id fast paths + tests
thecodacus
committed
d5b1a81
fix: skip_ids MUL_MAT_ID test cases were inside an #if 0 block
thecodacus
committed
119ead3
fix: mm_ids_helper compact map corruption when ids contain -1
thecodacus
committed
e4ed3cd
fix: mmvf id=-1 skip guard (float mul_mat_id vec path)
thecodacus
committed
a3f26d8
fix: zero-init ids_dst before mm_ids_helper — garbage tail entries in the
thecodacus
committed
2fc7267
feat: route MUL_MAT_ID with skip-capable ids away from mmq/mmf
thecodacus
committed
b69dc94
fix: op_params write in test — ggml_set_op_params_i32 is internal-only
thecodacus
committed
15e71ee
feat: MoE expert cache loader — hot-pack build from routing profile
thecodacus
committed
477ca49
feat: MoE expert cache graph wiring — dual hot/cold mul_mat_id in build_moe_ffn
thecodacus
committed
458ebe5
fix: flatten routed ids before map get_rows (batched-gather shape rule)
thecodacus
committed
2449739
fix: topk ids are a strided view - cont before flatten for map lookup
thecodacus
committed
5b96fcc
fix: sched expert-copy path skips id=-1 (hot/cold expert split)
thecodacus
committed
8a14a58
fix: restructure MoE pack dual path into full FFN chains per residency side
thecodacus
committed
aaa885c
feat: wire MoE expert cache into deepseek2 graph (GLM-4.7-Flash)
thecodacus
committed
f780f03
feat: add --moe-cache-profile and --moe-cache-slots flags
thecodacus
committed
f39fc1a
docs: fix moe-trace usage comment (output path is MOE_TRACE_OUT env)
thecodacus
committed
cca05a3

Navigation Menu

feat: MoE expert cache — keep hot routed experts resident in VRAM - #5

feat: MoE expert cache — keep hot routed experts resident in VRAM#5
thecodacus merged 23 commits into
perffrom
fable5/moe-expert-cache

Commits on Jul 24, 2026

feat: llama-moe-trace tool + expert-cache policy simulator

fix: moe-trace — common_init_from_params returns ptr with accessors

fix: moe-trace — include arg.h for common_params_parse

fix: moe-trace — take output path from MOE_TRACE_OUT env, -o not in this arg set

fix: moe-trace — topk tensor is a strided view; copy full byte range and index by nb

feat: moe-trace simulator v2 — model upload bytes/token and miss/upload cost per policy

feat: mul_mat_id tolerates id=-1 (skip) on CPU and CUDA fallback path

feat: id=-1 skip support in CUDA mul_mat_id fast paths + tests

fix: skip_ids MUL_MAT_ID test cases were inside an #if 0 block

fix: mm_ids_helper compact map corruption when ids contain -1

fix: mmvf id=-1 skip guard (float mul_mat_id vec path)

fix: zero-init ids_dst before mm_ids_helper — garbage tail entries in the

feat: route MUL_MAT_ID with skip-capable ids away from mmq/mmf

fix: op_params write in test — ggml_set_op_params_i32 is internal-only

feat: MoE expert cache loader — hot-pack build from routing profile

feat: MoE expert cache graph wiring — dual hot/cold mul_mat_id in build_moe_ffn

fix: flatten routed ids before map get_rows (batched-gather shape rule)

fix: topk ids are a strided view - cont before flatten for map lookup

fix: sched expert-copy path skips id=-1 (hot/cold expert split)

fix: restructure MoE pack dual path into full FFN chains per residency side

feat: wire MoE expert cache into deepseek2 graph (GLM-4.7-Flash)

feat: add --moe-cache-profile and --moe-cache-slots flags

docs: fix moe-trace usage comment (output path is MOE_TRACE_OUT env)