Skip to content

Fuse rmsnorm and quant#294

Draft
apinge wants to merge 1 commit into
zejunchen-zejun:Qwen3.5_v0.5.9from
apinge:qwen3_5_update_rmsnorm_quant
Draft

Fuse rmsnorm and quant#294
apinge wants to merge 1 commit into
zejunchen-zejun:Qwen3.5_v0.5.9from
apinge:qwen3_5_update_rmsnorm_quant

Conversation

@apinge

@apinge apinge commented Jul 10, 2026

Copy link
Copy Markdown

Motivation

This PR fuses dynamic quant kernel into rmsnorm.

Modifications

Accuracy Tests

python3 /opt/sglang/benchmark/gsm8k/bench_sglang.py    --port 9080       --tokenizer-path /model/offical_qwen3p5_397B_ptpc/    --max-new-tokens 4096 
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 200/200 [01:18<00:00,  2.55it/s]
Accuracy: 0.970
Invalid: 0.000
Latency: 78.430 s
Output throughput: 465.588 token/s

Benchmarking and Profiling

Test with concurrency=10 397B

Before

image

After

image

Checklist

Review Process

  1. Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • /tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci
  4. After green CI and required approvals, ask Merge Oncalls to merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant