Instella moe

[csabakecskemeti]

Overview

This MR adds support for the Instella-MoE architecture.

I would like to state upfront that this implementation was heavily assisted by Claude Opus 5 to help navigate the ggml graph logic and the conversion script plumbing. I have done my best to thoroughly study and understand the architectural changes to ensure they are implemented correctly as much as I could.

Architecture Overview
Instella-MoE is a derivative of DeepSeek-V3. It inherits several advanced features, including:
MLA (Multi-head Latent Attention) using the absorption optimization to keep the KV cache small.
Sigmoid-based MoE routing with shared experts.
YaRN RoPE scaling and the GGML_ROPE_TYPE_NORM (adjacent-pair) rotation convention.
However, it introduces two critical architectural "deltas" that required a new architecture definition (LLM_ARCH_INSTELLA_MOE) to avoid cluttering the existing deepseek2.cpp logic.

The Two Deltas

1. Gated Attention The model introduces a learned gate_proj matrix (mapped to blk.N.attn_gate)
   . This acts as a learned volume knob for attention: it produces a per-dimension value, squashes it with a sigmoid, and multiplies it by the attention output before the final o_proj
   . This allows the model to dynamically suppress attention for tokens where it adds little value.

2. FarSkip-Collective This is a structural change to the residual stream designed by AMD to overlap network communication in multi-GPU setups.
   Dual Streams: Each layer maintains two streams: res_full (contains everything) and res_nort (excludes routed experts).
   Parallel Execution: The next layer's Attention reads from the "no-routed" stream, while the FFN reads from the "full" stream.

These are sounds very neat optimizations. I have to admit I haven't tested performance so far.

Because of how these streams are bound, the Attention and FFN/MoE blocks effectively run in parallel over the same input rather than in the standard sequence, as I've understood. This necessitated a custom graph loop in src/models/instella-moe.cpp to ensure mathematical parity with the reference implementation. And this is completely done by Opus!
Implementation
The strategy it has chosen to implement this as a separate architecture rather than adding flags to deepseek2.cpp. So no risk to existing Deepseek models. It also copied the common code with all comments, and some of those may not hold anymore.

Please take this as an initial state for support the InstellaMoEForCausalLM architecture. I hope some with deeper knowledge of all crux of the numerical magic will chime in!

Additional information

Verification
I have verified the implementation by converting the model and testing text generation. It generates coherent, on-topic reasoning output with performance consistent with other MoE models of this size.

HF GGUF Repository: DevQuasar/amd.Instella-MoE-16B-A3B-Think-GGUF
I am happy to provide further details or adjust the implementation based on reviewer feedback.

Requirements

• I have read and agree with the contributing guidelines
  YES
• AI usage disclosure:
  YES
  Opus 5 has been used in a highly autonomous way. After the architecture adoption I've tested the code, and manually reviewed by myself as deep as I've could. Added extra comments and requested extra comments for the AI itself.

[ggml-gh-bot]

Hi @csabakecskemeti, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.