Skip to content

Port Nemotron-H-4B-Base-8K (nemotron_h) - #219

Open
DarthCeltic wants to merge 2 commits into
aifoundry-org:mainfrom
DarthCeltic:port-nemotron-h
Open

Port Nemotron-H-4B-Base-8K (nemotron_h)#219
DarthCeltic wants to merge 2 commits into
aifoundry-org:mainfrom
DarthCeltic:port-nemotron-h

Conversation

@DarthCeltic

Copy link
Copy Markdown
Contributor

Summary

  • Adds nvidia/Nemotron-H-4B-Base-8K to the most_models_ported track (identity nemotron_h), introducing the Nemotron-H hybrid Mamba2/Attention/MLP execution family to the board.
  • Real converter bug found and fixed: NemotronHModel.__init__ in convert_hf_to_gguf.py decides MoE-vs-dense by checking key presence of num_experts_per_tok in ModelBase.load_hparams()'s merged config. For this checkpoint that reports a full, plausible-looking MoE config (num_experts_per_tok=2, moe_intermediate_size=7688, n_routed_experts=8) — but direct inspection of both safetensors shards (311 tensors total) finds zero tensors matching expert/moe/router. The config fields are stale, almost certainly inherited from Nemotron-H's larger 8B MoE parent during pruning/distillation. Fixed via a standalone monkeypatch wrapper (ported_models/nemotron_h_4b_base/convert_wrapper.py) that hardcodes is_moe=False, treating tensor evidence as authoritative over the misleading config — the vendored submodule is never touched, matching the established pythia410m precedent.
  • Self-converted to Q8_0 GGUF (311 tensors, 4.8 GB, sha256=23cf524eaf7cc0c84ffe3dc168a5bacaad10bde2bb9fbaacba431a96b50c9d31), hosted on Hugging Face (darthceltic85/nemotron-h-4b-base-gguf) since it exceeds GitHub's 2 GB release-asset limit.
  • Verified live: model loads cleanly on the CPU backend (llama.cpp-et, fused Gated Delta Net enabled, 1489-node compute graph, 1 split) and real llama-perplexity against WikiText-2 raw gives PPL = 9.4458 +/- 1.75431 — a good, in-range result that also confirms the quantization RuntimeWarnings seen during conversion were a false alarm.

Test plan

  • benchmark_config in the claim JSON exactly matches .github/ci/benchmark_config.json's new nemotron_h_4b_base entry
  • Source revision is a valid 40-char hex commit SHA
  • RECIPE.md exists and documents the real bug found
  • All new/modified JSON files are valid JSON
  • Model loads and produces a real, in-range perplexity result (not fabricated)

Adds nvidia/Nemotron-H-4B-Base-8K to the most_models_ported track.
Fixed a real converter bug: NemotronHModel mis-detects MoE vs dense via
config key presence, but this checkpoints stale MoE config fields
(inherited from its 8B MoE parent) have zero matching expert/moe/router
tensors across all 311 tensors on direct inspection. Fixed via a
standalone monkeypatch wrapper (never touching the vendored submodule).

PPL = 9.4458 +/- 1.75431 on WikiText-2 raw, confirmed via real
llama-perplexity run on CPU backend. GGUF hosted on Hugging Face
(darthceltic85/nemotron-h-4b-base-gguf) since it exceeds GitHub's 2GB
release-asset limit.
@DarthCeltic
DarthCeltic requested a review from AFOliveira as a code owner July 26, 2026 06:36
@github-actions github-actions Bot added track: model-ports Global approved model ports prize track: community Community support and ecosystem contributions labels Jul 26, 2026
@AFOliveira

Copy link
Copy Markdown
Member

The converter fix and tensor inspection are useful, but Nemotron-H still needs a real full-offload ET-SoC1 execution and a committed deterministic oracle/expected output with an explicit comparison threshold. Host loading and CPU PPL alone cannot validate the ET port. Please pin the hosted GGUF immutably and include the converter-wrapper output hash. Maintainers will then review/adopt the identity/contract and runner policy.

Commits the exact reproduction command, pinned corpus/artifact hashes,
per-chunk + final CPU reference PPL, and an explicit 20% comparison
threshold so a full-offload ET-SoC1 run can be independently verified
against this reference.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

track: community Community support and ecosystem contributions track: model-ports Global approved model ports prize

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants