The gap
APEX's whole value is quantizing large MoE models well — the benchmark suite
already includes a 122B model. But the pipeline's imatrix step (apex_pipeline.sh
Phase 6) shells out to stock llama-imatrix, which needs the entire F16 model
resident (RAM+VRAM, via -ngl). That's the one step that still forces users onto
a big-memory box, even though every other phase scales down gracefully.
Concretely: the APEX quantize step is already memory-light (it streams tensors).
I recently quantized Laguna-S-2.1 (118B MoE, 235 GB BF16 → three APEX tiers) on a
laptop with no GPU and 54 GB RAM — llama-quantize handled it fine. Published:
https://huggingface.co/Myric/Laguna-S-2.1-APEX-GGUF (i-quality/i-compact/i-mini,
built with your generate_config.sh). The only reason I couldn't do the whole APEX
I-pipeline on that box is Phase 6: I had to reuse an existing imatrix because stock
llama-imatrix can't build one for a 118B model on that hardware. That last wall is
what this proposal removes.
The idea
A band-serialized imatrix generator: an independent reimplementation of
llama.cpp's importance-matrix computation (per-input-channel Σx²) in PyTorch via
forward hooks, that processes the model in bands of N layers — load a band's
weights from the shards once, push all calibration chunks through it, cache
inter-band activations, free the band. Peak resident weights = band layers,
independent of total model size. Each weight is read from disk once.
This decouples imatrix generation from llama.cpp's memory model, so a 100B–1T model
can be imatrix'd on a modest box (or a small GPU) and quantized anywhere.
Validation (public, independently checkable)
Reproduced end-to-end on granite-4.0-h-tiny and published with both imatrices +
the quants so anyone can verify — https://huggingface.co/Myric/granite-4.0-h-tiny-APEX-GGUF
(see the "The torch-imatrix variant" section; both granite-4.0-h-tiny.imatrix and
granite-4.0-h-tiny-torch.imatrix are in the repo for direct diffing). Same APEX
i-quality recipe + Q6_K base + eval, swapping only the imatrix:
| imatrix source |
i-quality PPL (wikitext-2, 200×512) |
Δ vs bf16 |
| band-serialized PyTorch (this tool) |
8.864 |
−0.04% |
stock llama-imatrix (ground truth) |
8.901 |
+0.38% |
At parity — the PyTorch imatrix is marginally better here, well inside the error
bar. It matches llama-imatrix tensor-for-tensor (368/368 names), median per-tensor
correlation 0.995, GGUF format bit-exact. Memory: band=4 peaked ~6.4 GB GPU
vs ~13 GB for the full model.
(Known, harmless wrinkle: down_*/ssm_out correlate less per-tensor because their
inputs are post-nonlinearity — SiLU-gated / SSM-scan outputs where bf16 + kernel
differences amplify. It doesn't move PPL, since the quantizer only needs relative
per-channel importance. On pure-transformer MoE the wrinkle largely vanishes.)
How it would slot in
An opt-in backend for Phase 6, default unchanged:
IMATRIX_BACKEND=llama (default) → today's llama-imatrixIMATRIX_BACKEND=serialized → the band generator over the HF safetensors
(already downloaded in Phase 2), writing the same imatrix.dat Phase 7 consumes.
The PyTorch/transformers/accelerate dependency would be gated behind the opt-in
backend (a separate requirements-serialized.txt), never required for the default
path.
Questions for maintainers
- Interest / placement — would you want this in-repo as an optional Phase 6
backend, or as a linked companion tool? I have a working port of the Phase 6
wiring ready either way.
- Arch coverage — it's validated end-to-end on Granite (hybrid Mamba-2/MoE);
the name-mapping is per-arch. I'd add + validate a mapping for a headline APEX
target (Qwen3.5 MoE) as the gating step before a PR. Any preference on which
model to validate against?
- Scope — the prototype favors correctness (eager attention, per-band CPU↔GPU
activation transfers). Happy to note the production TODOs (sdpa, batched chunks,
on-GPU activations) or land them first — your call on the bar for merge.
Happy to open a PR once you signal interest and the Qwen validation lands. Code +
validation harness already exist; I can share the branch.
The gap
APEX's whole value is quantizing large MoE models well — the benchmark suite
already includes a 122B model. But the pipeline's imatrix step (
apex_pipeline.shPhase 6) shells out to stock
llama-imatrix, which needs the entire F16 modelresident (RAM+VRAM, via
-ngl). That's the one step that still forces users ontoa big-memory box, even though every other phase scales down gracefully.
Concretely: the APEX quantize step is already memory-light (it streams tensors).
I recently quantized Laguna-S-2.1 (118B MoE, 235 GB BF16 → three APEX tiers) on a
laptop with no GPU and 54 GB RAM —
llama-quantizehandled it fine. Published:https://huggingface.co/Myric/Laguna-S-2.1-APEX-GGUF (i-quality/i-compact/i-mini,
built with your
generate_config.sh). The only reason I couldn't do the whole APEXI-pipeline on that box is Phase 6: I had to reuse an existing imatrix because stock
llama-imatrixcan't build one for a 118B model on that hardware. That last wall iswhat this proposal removes.
The idea
A band-serialized imatrix generator: an independent reimplementation of
llama.cpp's importance-matrix computation (per-input-channel Σx²) in PyTorch via
forward hooks, that processes the model in bands of N layers — load a band's
weights from the shards once, push all calibration chunks through it, cache
inter-band activations, free the band. Peak resident weights =
bandlayers,independent of total model size. Each weight is read from disk once.
This decouples imatrix generation from llama.cpp's memory model, so a 100B–1T model
can be imatrix'd on a modest box (or a small GPU) and quantized anywhere.
Validation (public, independently checkable)
Reproduced end-to-end on
granite-4.0-h-tinyand published with both imatrices +the quants so anyone can verify — https://huggingface.co/Myric/granite-4.0-h-tiny-APEX-GGUF
(see the "The torch-imatrix variant" section; both
granite-4.0-h-tiny.imatrixandgranite-4.0-h-tiny-torch.imatrixare in the repo for direct diffing). Same APEXi-quality recipe + Q6_K base + eval, swapping only the imatrix:
llama-imatrix(ground truth)At parity — the PyTorch imatrix is marginally better here, well inside the error
bar. It matches
llama-imatrixtensor-for-tensor (368/368 names), median per-tensorcorrelation 0.995, GGUF format bit-exact. Memory: band=4 peaked ~6.4 GB GPU
vs ~13 GB for the full model.
(Known, harmless wrinkle:
down_*/ssm_outcorrelate less per-tensor because theirinputs are post-nonlinearity — SiLU-gated / SSM-scan outputs where bf16 + kernel
differences amplify. It doesn't move PPL, since the quantizer only needs relative
per-channel importance. On pure-transformer MoE the wrinkle largely vanishes.)
How it would slot in
An opt-in backend for Phase 6, default unchanged:
IMATRIX_BACKEND=llama(default) → today'sllama-imatrixIMATRIX_BACKEND=serialized→ the band generator over the HF safetensors(already downloaded in Phase 2), writing the same
imatrix.datPhase 7 consumes.The PyTorch/transformers/accelerate dependency would be gated behind the opt-in
backend (a separate
requirements-serialized.txt), never required for the defaultpath.
Questions for maintainers
backend, or as a linked companion tool? I have a working port of the Phase 6
wiring ready either way.
the name-mapping is per-arch. I'd add + validate a mapping for a headline APEX
target (Qwen3.5 MoE) as the gating step before a PR. Any preference on which
model to validate against?
activation transfers). Happy to note the production TODOs (sdpa, batched chunks,
on-GPU activations) or land them first — your call on the bar for merge.
Happy to open a PR once you signal interest and the Qwen validation lands. Code +
validation harness already exist; I can share the branch.