Support for Gemma 4 E2B QAT mobile checkpoint (<1 GB)

[TobiasJu]

Hi there, Google's Gemma 4 documentation lists a dedicated mobile QAT format for Gemma 4 E2B, alongside the existing BF16/Q4_0 checkpoints:

Format	Memory
BF16	11.4 GB
Q4_0	2.9 GB
Mobile	1.1 GB
Mobile (text-only)	0.84 GB

Source: https://ai.google.dev/gemma/docs/core?hl=de#gemma-4-inference-memory-requirements

We're currently shipping the standard gemma-4-E2B-it.litertlm (~2.6 GB) via flutter_gemma, since the mobile QAT checkpoint doesn't seem to be available as a pre-converted .litertlm bundle yet (only as -mobile-ct/-mobile-transformers weights on Hugging Face, with no documented conversion path to LiteRT-LM).

Is there a plan/timeline to add the QAT mobile checkpoint (ideally the text-only variant at 0.84 GB) to flutter_gemma's supported model list? This would be a significant win for RAM-constrained Android devices. Happy to test a preview build if one exists.

[remingtonc]

Isn't the Gemma 4 E2B LiteRT-LM version already QAT?

https://huggingface.co/litert-community/gemma-4-E2B-it-litert-lm

LiteRT-LM uses a state of the art Gemma-4 mobile quantization scheme that uses a mixture of 2bit, 4bit and 8 bit weights. This means that for text only use cases the weight footprint in memory can be as low as 0.8 GB while the runtime uses memory mapping to support the 1.12GB of embedding parameters. This approach gives significant working memory savings on some platforms as seen in the more detailed data below. Additionally the Vision and Audio models are loaded on demand to further reduce memory consumption.

[TobiasJu]

Thanks for the hint! But I tested huggingface.co/litert-community/gemma-4-E2B-it-litert-lm and it uses ~2,6GB of RAM running on an S24:

Applications Memory Usage (in Kilobytes):
Uptime: 112515457 Realtime: 165238961

** MEMINFO in pid 26945 [com.app.plate] **
                   Pss  Private  Private  SwapPss      Rss     Heap     Heap     Heap
                 Total    Dirty    Clean    Dirty    Total     Size    Alloc     Free
                ------   ------   ------   ------   ------   ------   ------   ------
  Native Heap   199616   181120    18496   285539   199968   780920   700693    74408
  Dalvik Heap     1401     1144      236     4330     1904    27646     3070    24576
 Dalvik Other      320      144      144     1518      416                           
        Stack      492      464       28     1640      500                           
       Ashmem        3        0        0        0      140                           
    Other dev       17        8        8        0      396                           
     .so mmap    30026      112    28708     3720    46856                           
    .jar mmap      323        0       60        0     9252                           
    .apk mmap    26050      188    25860     1420    26148                           
    .ttf mmap       22        0       20        0       24                           
    .dex mmap        3        0        0        0      240                           
    .oat mmap      248        0       44        0     6176                           
    .art mmap      366       28      108      655     9892                           
   Other mmap   293192        8   293084     1292   293624                           
   EGL mtrack    35485    35485        0        0    35485                           
    GL mtrack  2587252  2587252        0        0  2587252                           
      Unknown    11771     8224     3544    10607    11988                           
        TOTAL  3497308  2814177   370340   310721  3230261   808566   703763    98984
 
 App Summary
                       Pss(KB)                        Rss(KB)
                        ------                         ------
           Java Heap:     1280                          11796
         Native Heap:   181120                         199968
                Code:    55032                          88800
               Stack:      464                            500
            Graphics:  2622737                        2622737
       Private Other:   323884
              System:   312791
             Unknown:                                  306460
 
           TOTAL PSS:  3497308            TOTAL RSS:  3230261       TOTAL SWAP PSS:   310721
 
 Objects
               Views:        6         ViewRootImpl:        1
         AppContexts:        5           Activities:        1
              Assets:       73        AssetManagers:        0
       Local Binders:       40        Proxy Binders:       81
       Parcel memory:       41         Parcel count:       73
    Death Recipients:        1             WebViews:        0
 
 Native Allocations
                         Count                       Total(kB)
                        ------                         ------
    Other (malloced):      353                             32
 Other (nonmalloced):      138                             94
 
 SQL
         MEMORY_USED:      288
  PAGECACHE_OVERFLOW:      147          MALLOC_SIZE:       46
 
 DATABASES
      pgsz     dbsz   Lookaside(b) cache hits cache misses cache size  Dbname
PER CONNECTION STATS
         4      104             20    62   184     1  /data/user/0/com.app.plate/no_backup/androidx.work.workdb
         4        8                    0     0     0    (attached) temp
         4      104             21   285    28     1  /data/user/0/com.app.plate/no_backup/androidx.work.workdb (1)
POOL STATS
     cache hits  cache misses    cache size  Dbname
            347           216             2  /data/user/0/com.app.plate/no_backup/androidx.work.workdb
 
 Asset Allocations
    : 17K
    : 9K
    : 21K

Update it's nearly the same RAM usage for PreferredBackend.npu or PreferredBackend.gpu.

[DenisovAV]

QAT doesn't reduce memory the way it's being assumed here. Model size is set by the quantization bit-width (this build uses a 2/4/8-bit mixture) and the parameter count — not by whether it was QAT or PTQ. Post-training quantization takes the bf16 model and quantizes to int4: small file, but accuracy drops at low bit-widths. Quantization-aware training bakes the quantization into fine-tuning so the model learns to live at 2/4-bit: same small size, accuracy close to bf16. A non-QAT model quantized to the same 2/4/8-bit mix would be the same size. QAT buys quality at that size, not a smaller size — and this build already ships that quantization, so there is nothing lower to switch to.

The "reduced to 1 GB" figure in Google's QAT blog (https://blog.google/innovation-and-ai/technology/developers-tools/quantization-aware-training-gemma-4/) is quantization vs full-precision bf16 — a reduction any int4 quantization achieves; QAT is just how they kept int4 from losing quality. It is not a QAT-vs-PTQ size difference, and it is not a file size. Both the blog's "under 1 GB" and the card's "0.8 GB" are the same narrow number: text transformer weights only, with the 1.12 GB of Per-Layer Embeddings excluded and the vision/audio weights excluded (loaded on demand). It's an accounting boundary for the text weights alone, not the footprint of the running model.

The quantized weights plus embeddings have to be resident in accelerator memory to run on the GPU, so ~2.6 GB for this model is inherent, not a leak. What you can actually trim without changing the model: load text-only with getActiveModel(supportImage: false, supportAudio: false) — the vision/audio executors are loaded on demand, so not enabling them avoids their ~0.66 GB; and keep maxTokens near the 1024 floor, which keeps the KV cache small.

Bottom line: QAT won't lower your memory — it's a quality technique and it's already applied. For this model the real levers are loading text-only and keeping the context window small; the model weights themselves stay resident on the accelerator.