Skip to content

Track expert usage stats for MoE models across the mesh #1185

Description

@michaelneale

What

For MoE models (e.g. Mixtral/Qwen-MoE/gpt-oss class), track which experts get activated during inference and surface that as a mesh-wide statistic — purely as a curiosity/observability feature.

Why

  • MoE routing is invisible today. We split MoE models across nodes but have no idea how balanced the expert activation actually is.
  • Interesting signal for split planning: if a handful of experts dominate, expert-aware placement could beat naive layer splits.
  • Fun/insightful thing to show in the console — a live histogram of expert activation across the mesh.

Sketch

  • Expose per-token expert selection counts from the embedded staged runtime (skippy) as a lightweight counter, off by default or cheap enough to always run.
  • Aggregate locally per model, then gossip a compact summary (additive field, ignorable by older nodes) so any node can render mesh-wide totals.
  • Surface via /api/status (or a dedicated stats route) and a small console visualization; optionally emit as OTLP metrics through skippy-metrics.

Notes

  • Must be additive on the wire — older peers should ignore the new gossip field.
  • Keep the hot path cost near zero; prefer counters incremented in the runtime over per-token event streams.
  • Privacy: counts only, no prompt or token content.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions