Gemma 31B & 26B model support

[callmephil]

Is it possible to add support for the bigger models, such as Gemma 4 31B & 26B?

[DenisovAV]

The engine runs any .litertlm checkpoint you install — it routes by file type, there is no per-model allow-list in the plugin. So supporting a given model comes down to two things: does a LiteRT-LM (.litertlm) build exist, and does it fit device memory.

For 26B and 31B, no .litertlm build is published — I checked, and litert-community only ships E2B (~2.6 GB), E4B (~3.7 GB) and 12B (~6.5 GB). google/gemma-4-26B-A4B and google/gemma-4-31B are the base safetensors weights, which Google positions for consumer GPUs and workstations, not edge.

You can convert them to .litertlm yourself with the LiteRT Torch Generative API, e.g.:

litert-torch export-hf --model=google/gemma-4-31B --output_dir=./gemma4-31b

(see https://developers.google.com/edge/litert-lm/tutorials/convert-and-run). The resulting file loads in flutter_gemma as a ModelType.gemma4 .litertlm model — nothing in the engine gates on model size.

But they will not fit on a phone. 31B is fully dense (~30.7B params active on every forward pass) — roughly 15-16 GB of weights even at int4. 26B A4B is Mixture-of-Experts, but all 26B parameters stay resident in memory (only ~4B are active per token); MoE saves compute, not memory — so ~13 GB at int4. That is desktop-only territory, and even there it needs a lot of RAM. On mobile it is not feasible regardless of format.

The largest Gemma 4 that already exists as .litertlm is 12B (litert-community/gemma-4-12B-it-litert-lm, ~6.5 GB), runnable on desktop with enough RAM. On mobile, E2B/E4B remain the practical ceiling.

[callmephil]

Thanks for the tips. I'll do the conversion with Torch then; I'm running the app on MacOS with 128GB ram, so hardware shouldn't be a problem at this point.

I was able to download the 12B model, but it looks like the model isn't initializing, or at least does not generate any responses, unlike 2B/4B. It does not produce an error or crash either.

[DenisovAV]

Figured out the cause: the 12B build is GPU-only. Its .litertlm pins the decoder section to GPU (section_backend_constraint: gpu in the loader metadata), so it can't create an engine on the CPU backend — that's the "loads but no output" you hit. On GPU it loads and generates fine (I pulled the full 6.5 GB file and it returned a normal response).

Quick unblock: make sure the model runs on PreferredBackend.gpu (or leave preferredBackend unset — the engine tries GPU first by default). Heads-up: a few of the example app's ModelConfigs default preferredBackend to PreferredBackend.cpu, so if you based yours on those, that's likely it.

To confirm that's the whole story, could you share:

• the preferredBackend you pass (or your ModelConfig.preferredBackend),
• which file — gemma-4-12B-it.litertlm or -web,
• your maxTokens.

If you were on CPU, switching to GPU should fix it. (I'll also fix the error message — a GPU-only model on CPU currently says "model may be invalid", which is what sent you the wrong way.)