Skip to content

opencl: make the MoE expert scatter deterministic - #26464

Draft
wanghqc wants to merge 1 commit into
ggml-org:masterfrom
qualcomm:hq/opencl-moe-deterministic-scatter
Draft

opencl: make the MoE expert scatter deterministic#26464
wanghqc wants to merge 1 commit into
ggml-org:masterfrom
qualcomm:hq/opencl-moe-deterministic-scatter

Conversation

@wanghqc

@wanghqc wanghqc commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Overview

This PR is to fix a bug in MoE scatter.

  • MoE prompt processing on the OpenCL backend is not reproducible run-to-run: the same binary on the same prompt returns one of several outputs.
  • Measured on granite-3.0-3b-a800m-Q4_K_M (Adreno X2-90, greedy, llama-simple -n 64, 3 prompts × 5 runs each), every prompt produced 2–3 distinct outputs across 5 runs.

Root cause

  • kernel_moe_scatter assigns each token its slot within its expert via atomic_inc(&slot_counter[expert]), so the token→slot packing depends on which work-item wins the atomic and varies every run.
  • The ragged MoE prefill GEMM is sensitive to that packing order; the non-ragged path happens to be immune.

Fix

  • This PR replaces the atomic slot grab with a deterministic rank: a token's slot is its position in flat (token, k) order among the tokens routed to the same expert — a fixed function of the routing input.
  • One workgroup per expert walks the routing list in blocks of 64 with a workgroup scan and a running carry.
  • Default on; GGML_OPENCL_MOE_STABLE_SCATTER=0 restores the previous behavior.
  • 2 files, backend-only (ggml-opencl.cpp + kernels/moe_sort_by_expert.cl).

Additional information

Determinism verification

  • With the fix: 5/5 identical runs on every prompt, and the output matches an O(N·topK) reference implementation exactly.
  • Six MoE models on the X2-90, 3 runs each, all single-output and coherent: gpt-oss-20b-MXFP4, Qwen3-30B-A3B-Q4_K_M, Qwen3-unsloth-30B-A3B-Q4_0, Qwen3.6-35B-A3B-MXFP4_MOE, and granite-3.0-3b in Q4_K_M and MXFP4_MOE.
  • gpt-oss also 3/3 on the Adreno 840.

Cost

Prefill only — the scatter runs on the GEMM path, decode never calls it: pp512 −0.63% on granite-3.0-3b (40 experts), −0.15% on Qwen3-30B-A3B (128 experts).

Scope

  • This fixes run-to-run reproducibility.
  • The ragged and non-ragged paths still produce different roundings of equal quality (MUL_MAT_ID max err 2.68e-05 vs 2.79e-05 against the CPU reference);
  • A host comment claiming the ragged path is byte-identical is corrected as part of this change. That difference is a rounding choice, not a wrong answer.

Testing

test-backend-ops -o MUL_MAT_ID: Adreno X2-90, 383 OK / 0 FAIL.
The Adreno 840 declines all MUL_MAT_ID cases at these shapes in test-backend-ops; its evidence is the real-model determinism runs above.

Requirements

kernel_moe_scatter assigns each token's slot within an expert via
atomic_inc, so the packing order depends on which work-item wins the
atomic. The ragged MoE prefill GEMM is sensitive to that order, so
prompt processing is not reproducible: the same binary on the same
prompt returns one of several outputs (Adreno X2-90,
granite-3.0-3b-a800m: 2-3 distinct outputs over five greedy runs,
diverging ~30 tokens in). Dense models and token generation are
unaffected.

Assign the slot instead as the token's rank in flat (n, k) order among
the tokens routed to the same expert - a fixed function of the routing
input. One workgroup per expert ranks its tokens with a workgroup scan.

The scatter runs only on the prefill GEMM path. Cost on the X2-90:
pp512 -0.6% (granite-3.0-3b-a800m), -0.2% (Qwen3-30B-A3B); token
generation never calls it.

test-backend-ops MUL_MAT_ID 383/383. GGML_OPENCL_MOE_STABLE_SCATTER=0
restores the atomic version.
@github-actions github-actions Bot added ggml changes relating to the ggml tensor library for machine learning OpenCL Issues specific to the OpenCL backend labels Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ggml changes relating to the ggml tensor library for machine learning OpenCL Issues specific to the OpenCL backend

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant