Non-record: Initial small random mini-MoE#1692

Open

joshEng1 wants to merge 5 commits into

openai:mainfrom

Commits on Apr 4, 2026

Add non-record random MLP up adapter ablation experiment
joshEng1
committed

Commits on Apr 17, 2026

cheap MoE style random MLP variant
joshEng1
committed

Commits on Apr 18, 2026

Add support for GQA in scaled dot-product attention with fallback mechanism
joshEng1
committed
Updated with early smoke results
joshEng1
committed
Update README.md with additional insights on model efficiency and performance trade-offs
joshEng1
committed