Skip to content

fast::rms_norm: accept Option<&Array> for weight - #347

Open
Ogilthorp3 wants to merge 1 commit into
oxiglade:mainfrom
Ogilthorp3:fast-rms-norm-optional-weight
Open

fast::rms_norm: accept Option<&Array> for weight#347
Ogilthorp3 wants to merge 1 commit into
oxiglade:mainfrom
Ogilthorp3:fast-rms-norm-optional-weight

Conversation

@Ogilthorp3

Copy link
Copy Markdown

Summary

fast::rms_norm currently requires a non-optional weight: impl AsRef<Array>. This PR changes it to weight: impl Into<Option<&'a Array>>, matching:

  • The existing fast::layer_norm pattern in the same file (weight: impl Into<Option<&'a Array>>).
  • Python's mx.fast.rms_norm(x, None, eps) semantics — when weight is None, the kernel skips the per-element multiply.

Why

The underlying C function mlx_fast_rms_norm already supports a null weight (it checks weight.ctx and routes to the no-affine path internally, skipping the multiply kernel pass). Only the Rust wrapper needed updating to expose this.

When callers want plain RMS normalization without an affine weight — common in attention's q/k norm step where the weight would otherwise be all-ones — passing None saves one Metal kernel dispatch per call. Concrete impact on a 40-layer LLM at long-context prefill: ~80 saved kernel passes per forward (two q/k norms × 40 attention layers).

Backward compatibility

Existing callers continue to compile via the auto-derived From<&'a Array> for Option<&'a Array> blanket impl. The single owned-Array test caller in test_rms_norm was updated to pass &weight.

Validation

$ cargo test --release -p mlx-rs --lib fast::tests -- --test-threads=1
  test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 574 filtered out

$ cargo test --release -p mlx-rs --lib nn::normalization -- --test-threads=1
  test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 575 filtered out

The serial flag is needed to avoid the known Metal command-buffer assertion when running tests in parallel on M-series; that's pre-existing behavior, not introduced by this PR.

Discovery context

Found while investigating a 2× LM-prefill performance gap between mlx-rs and Python's mlx_lm bindings on Apple M4 Max. Per-op microbenches showed exact parity at the FFI boundary, but the qwen3_5 LinearAttention's q/k norm path was constructing an ones array and calling rms_norm with it, then multiplying by a scale array afterward — where Python's qwen3_5 implementation passes None and applies a scalar multiply inline. This signature gap was one of the structural divergences from the Python API.

Companion application-side fix in our downstream consumer:

```rust
// before
let qk_ones = self.qk_ones_weight.as_dtype(qkv_dtype)?;
let q_norm = mlx_rs::fast::rms_norm(&q, &qk_ones, 1e-6)?;
let q = q_norm.multiply(&q_scale)?;

// after (matches Python: q = inv_scale**2 * fast.rms_norm(q, None, 1e-6))
let q_norm = mlx_rs::fast::rms_norm(&q, None, 1e-6)?;
let q = q_norm.multiply(&q_scale)?;
```

Aligns with the existing layer_norm_device pattern in the same file
and matches Python's mx.fast.rms_norm(x, None, eps) semantics. The
underlying C function (mlx_fast_rms_norm) already supports a null
weight pointer — it checks weight.ctx and routes to the no-affine
path internally, skipping the per-element multiply kernel pass.
Only the Rust wrapper needed updating.

Why it matters: when callers want plain RMS normalization without
an affine weight (common in attention's q/k norm step where the
weight would be all-ones), passing None saves one Metal kernel
dispatch per call. For a 40-layer LLM at 1045-token prefill with
two q/k norms per attention layer, that's ~80 saved kernel passes
per forward.

Existing call sites continue to compile via auto-derived
From<&'a Array> for Option<&'a Array>. The single owned-Array test
caller (test_rms_norm) was changed to pass &weight.

Validated:
  cargo test --release -p mlx-rs --lib fast::tests -- --test-threads=1
    test result: ok. 6 passed; 0 failed
  cargo test --release -p mlx-rs --lib nn::normalization -- --test-threads=1
    test result: ok. 5 passed; 0 failed

Discovered while investigating a 2x LM-prefill perf gap between
mlx-rs and Python's mlx_lm bindings on Apple M4 Max — the qwen3_5
LinearAttention's q/k norm path was forced to construct an ones
array and call rms_norm with it, then multiply by a scale array
afterward, where Python passes None and applies a scalar multiply
inline. Microbenches and per-layer profiling confirmed parity at
the per-op level; this signature gap was one of the structural
divergences from Python's API.
@Ogilthorp3

Copy link
Copy Markdown
Author

Hi — friendly check-in on this one (open ~7 days, MERGEABLE, no CI runs yet). Happy to address review feedback or rebase if needed. The downstream context: removes a redundant Metal kernel pass per call, aligns the Rust signature with the existing mx.fast.rms_norm(x, None, eps) Python semantics + the C function (which already supports null weight). Anything I can do to help move it forward?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant