moondream/moondream3 vram usage

[piotrkochan]

While moondream2 is super small and consume ~3-5GB of VRAM , moondream3 model eats over 16GB. Why? Is there any way to make it consume less? Model is advertised to be "tiny".

[Theheavenlyprogrammer333]

1. It’s dramatically larger internally

Moondream 3 is a ~9 billion-parameter mixture-of-experts (MoE) vision-language model.

Only about 2 B parameters are active per inference, but the full network still has to be stored in memory.

Result: memory footprint rises even if runtime compute is optimized.

2. Longer context = more memory

Context window jumped from ~2k to 32k tokens.

Large context windows increase KV-cache and attention memory usage — often a hidden VRAM cost.

How to reduce VRAM usage
Use a quantized build

There is a 4-bit INT4 version:

Size drops from ~18 GB → ~6 GB (~66% reduction).

Runs in roughly 7 GB VRAM instead of ~20 GB.

Accuracy loss is small (~2–5%).

This is the biggest practical lever you have.
source : https://huggingface.co/alecccdd/moondream3-preview-4bit?