Gemma 4 E2B support (umbrella / tracking)

[ai-hpc]

Goal

Run google/gemma-4-E2B-it (GGUF) correctly on the Orin Nano — including
Per-Layer Embeddings (PLE) — so genie-claw can use Gemma 4 E2B (better
quality at the 2B scale) in place of the current model. Then port the validated
math to TensorRT-Edge-LLM (NVIDIA/TensorRT-Edge-LLM#72).

Why this is a big change

The engine today is hardcoded to the Qwen3 / Llama-3 pre-norm + SwiGLU path:
one head_dim, one rope_theta, SiLU baked into the FFN, 2 norms/layer, no
sliding-window mask, no logit softcap, and a BPE-only tokenizer. Gemma 4 diverges
on nearly every axis — dual head dim (256 local / 512 global), dual RoPE, GeGLU,
4 norms + a PLE pathway, KV-sharing across the last 20 layers, final-logit
softcap, and a SentencePiece tokenizer.

So step 1 is an architecture abstraction so model-family behavior is
data-driven (an arch "recipe"), not hardcoded — then Gemma 4 is one recipe among
others and future archs are cheap. The existing Qwen3/Llama path must route
through the new abstraction with byte-identical output (pure refactor) before
any Gemma code lands.

Note: even llama.cpp does not implement Gemma 4 E2B's PLE in its forward graph
(ggml-org/llama.cpp#22243),
so the PLE work here is genuinely new.

Full design: docs/gemma4-plan.md (on feat/gemma4-support).

Open question to resolve first

llama.cpp registers altup_* / laurel_* tensors under arch gemma4 (Gemma-3n
lineage), but HF's Gemma4TextDecoderLayer uses only PLE. Confirm from the
real GGUF tensor dump whether E2B carries AltUp/LAuReL before #93 — it
materially changes the forward pass.

Tasks

1. Gemma 4 [1/6]: architecture abstraction — make the forward path arch-driven #89 — architecture abstraction (do first; pure refactor, byte-identical Qwen3 output)
2. Gemma 4 [2/6]: recognize gemma4 GGUF + parse hparams + fail loud #90 — recognize gemma4 GGUF + parse hparams + fail loud
3. Gemma 4 [3/6]: SentencePiece/unigram tokenizer (262k vocab) #91 — SentencePiece/unigram tokenizer (parallelizable; host-verifiable on x86)
4. Gemma 4 [4/6]: dense forward — GeGLU, embed scale, 4-norm, dual head dim, dual RoPE, sliding window, KV-sharing, softcap #92 — dense forward (GeGLU, embed scale, 4-norm, dual head dim, dual RoPE, sliding window, KV-sharing, softcap)
5. Gemma 4 [5/6]: Per-Layer Embeddings (PLE) forward pipeline (+ AltUp/LAuReL if present) #93 — Per-Layer Embeddings (PLE) (+ AltUp/LAuReL if present)
6. Gemma 4 [6/6]: validation — HF logit/argmax parity + perf + memory on Orin #94 — validation (HF parity, perf, memory on Orin)

[ai-hpc]

Resolved the open architecture questions by dumping the real gemma-4-E2B-it-Q4_K_M.gguf on the Orin:

AltUp/LAuReL: NONE — Gemma 4 E2B is PLE-only (HAS altup: False, laurel: False, per_layer: True). Matches HF; no Gemma-3n complexity needed. The forward = standard Gemma decoder + Per-Layer Embeddings.

Schema (authoritative, from GGUF):

• 35 layers, embd 1536, head_count 8, head_count_kv 1 (MQA)
• head dim: key_length=512 (full/global), key_length_swa=256 (sliding/local)
• sliding_window_pattern: 4 sliding : 1 full (full layers at idx 4,9,…,34)
• feed_forward_length: 6144 (layers 0–14), 12288 (layers 15–34, double-wide)
• shared_kv_layers=20, sliding_window=512, final_logit_softcapping=30
• dual RoPE: freq_base=1e6/dim 512 (full), freq_base_swa=1e4/dim 256 (sliding)
• blk.0: attn_{q,k,v,output} + attn_{q,k}_norm + 4 norms (attn_norm, post_attention_norm, ffn_norm, post_ffw_norm) + PLE (inp_gate, proj, post_norm) + layer_output_scale

#91 tokenizer DONE + verified: tokenizer.ggml.model=gemma4 is unigram SentencePiece (carries scores; ignore the extra merge table). genie's encode_spm now matches llama.cpp byte-for-byte on the real model (Hello world -> 2 9259 1902, etc.).

Next: #92 dense forward (now fully specified), then #93 PLE.