feat(model-ports): add gpt_oss_20b - #213
Open
DarthCeltic wants to merge 3 commits into
Open
Conversation
OpenAI's own open-weight release (arch=gpt-oss), full compliant shape (claim + framework registration + real host-CPU perplexity). Built a plain CPU-only (GGML_ET=OFF) llama-perplexity from the vendored source, ran against the board-pinned WikiText-2 corpus. Loads and runs cleanly (PPL 70.2523 -- notably higher than every other model in this campaign, expected given gpt-oss's reasoning-focused training distribution and its native MXFP4 4-bit format, not treated as a red flag). Confirms native MXFP4 MoE-expert loading and sliding-window + full attention (llama_kv_cache_iswa) both work on ggml-cpu. MXFP4 support on the ET backend specifically remains a real, open question, flagged honestly -- same tier of uncertainty as granite_3_1b_a400m's MoE-routing question, not assumed to work.
github-actions
Bot
added
track: model-ports
Each claim's benchmark_config must exactly match the model's "config" pointer in .github/ci/benchmark_config.json per effective_model_config() in model_port_claim.py, not the top-level benchmark_config.json path itself. Fixes a mismatch that would fail eligibility validation.
Member
|
Manual ET review confirmed GPT-OSS-20B executes with full offload and ET PPL matches CPU, but the committed recipe reports PPL 70.2523 while the exact submitted artifact measured about 400.574. Please correct the recipe and add a reproducible committed oracle/threshold. Maintainers still need to adopt the identity/contract and approve the runner before this can receive model-port credit. |
…dd committed oracle Maintainer's manual ET-SoC1 review measured the same pinned artifact at PPL~400.57 vs this session's own CPU run of 70.25. Documented both numbers honestly rather than silently picking one, and added a committed, reproducible perplexity oracle contract (fixed command, corpus, artifact, and explicit 20% threshold matching this repo's own leaderboard-gate policy) so the discrepancy can be independently re-run and settled.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds gpt_oss_20b causal-LM port to llama.cpp-et, with a model-ports claim. OpenAI's own open-weight release, native MXFP4 quantization. ET-backend MXFP4 support is genuinely unconfirmed, flagged in the recipe.