Skip to content

roofline: emit per-expert routing histogram (tokens_per_expert) for MoE launches - #53

Merged
mgehre-amd merged 1 commit into
gfx11from
rogarcia.roofline-moe-tokens-per-expert
Jul 17, 2026
Merged

roofline: emit per-expert routing histogram (tokens_per_expert) for MoE launches#53
mgehre-amd merged 1 commit into
gfx11from
rogarcia.roofline-moe-tokens-per-expert

Conversation

@roberteg16

@roberteg16 roberteg16 commented Jul 16, 2026

Copy link
Copy Markdown

Summary

count_active_experts copies the MUL_MAT_ID ids tensor to host once per invocation to count the distinct routed experts. The same pass now also fills a per-expert token-count array. begin_op stores it per invocation and write_report emits it as tokens_per_expert (length n_experts, index e = routings to expert e, Σ = n_tokens*top_k) next to experts_used.

  • No new device work or synchronization — the ids read-back already happens for experts_used; this replaces the seen[] bitmap with a counts[] array and adds one JSON field.
  • Present only for MoE launches whose ids were read back; non-MoE rows are unchanged, and consumers that don't know the field ignore it.

Test plan

  • ggml-cuda-roofline.cpp compiles clean with the project's exact build flags (host-only TU).
  • End-to-end on gfx1151: GGML_ROOFLINE_OUT=... llama-bench -m granite-3.0-1b-a400m-Q4_K_M.gguf -p 64 -ngl 99 → all 71 MUL_MAT_ID rows carry tokens_per_expert, with len == n_experts, Σ == M*top_k, and nonzero == experts_used on every row.

…oE launches

count_active_experts already reads the MUL_MAT_ID ids tensor back to host once
per invocation to count distinct routed experts; the same pass now also fills a
per-expert token-count array. begin_op stores it per invocation and write_report
emits it as "tokens_per_expert" (length n_experts, index e = routings to expert e,
Σ = n_tokens*top_k) alongside experts_used.

No new device work or sync — the ids read-back already happens for experts_used.
The consumer renders it as the MoE routing-load distribution under --show-histograms.
@roberteg16
roberteg16 marked this pull request as ready for review July 16, 2026 10:45
@mgehre-amd
mgehre-amd merged commit 0941c4c into gfx11 Jul 17, 2026
5 checks passed
@mgehre-amd
mgehre-amd deleted the rogarcia.roofline-moe-tokens-per-expert branch July 17, 2026 06:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants