Is your feature request related to a problem? Please describe.
We adapted Qwen3-0.6B to run end-to-end on NPU2 (Strix Point) with IRON. Two questions before we contribute:
- Does upstream accept model applications beyond Llama (e.g. Qwen3)? What are the acceptance criteria (app structure, test/benchmark requirements, weight handling)?
- Are there plans for first-class decode-oriented attention (fused GQA with runtime seq_pos, KV cache streamed from DRAM)? Today a model port must assemble decode attention from GEMV + Softmax + Transpose per layer.
Describe the solution you'd like
- A place (or documented criteria) for model applications beyond Llama
- A decode-attention operator (fused scores-GEMV + online-softmax + context-GEMV) as a standard operator
For reference, we have a working pure-IRON Qwen3-0.6B decode (28 layers in a single OperatorSequence, ~65 ms/token, logits cosine 0.9985 vs HuggingFace):
https://github.com/ZhongYic00/IRON/tree/feat/qwen3-decode
- app:
iron/applications/qwen3_0.6b/ (dialogue runner, HF cosine/e2e checks, TPOT bench, README)
- new operators:
decode_attn (fused decode attention, runtime seq_pos, S_KV up to 4096), qk_norm, gemv_argmax(_bf16), etc.
Describe alternatives you've considered
Hand-rolling per-model decode attention from GEMV + Softmax + Transpose (what the Llama app does). Works, but it is ~10 ops per layer with per-op array reconfiguration; a fused decode_attn operator collapses that to one.
Is your feature request related to a problem? Please describe.
We adapted Qwen3-0.6B to run end-to-end on NPU2 (Strix Point) with IRON. Two questions before we contribute:
Describe the solution you'd like
For reference, we have a working pure-IRON Qwen3-0.6B decode (28 layers in a single OperatorSequence, ~65 ms/token, logits cosine 0.9985 vs HuggingFace):
https://github.com/ZhongYic00/IRON/tree/feat/qwen3-decode
iron/applications/qwen3_0.6b/(dialogue runner, HF cosine/e2e checks, TPOT bench, README)decode_attn(fused decode attention, runtime seq_pos, S_KV up to 4096),qk_norm,gemv_argmax(_bf16), etc.Describe alternatives you've considered
Hand-rolling per-model decode attention from GEMV + Softmax + Transpose (what the Llama app does). Works, but it is ~10 ops per layer with per-op array reconfiguration; a fused decode_attn operator collapses that to one.