feat: granite 4.2 - #700
Open
vegah wants to merge 3 commits into
Open
Conversation
Adds the granite family so granite-4.2-3B runs on FastFlowLM. The engine is plain C++ in src/common/models/, so it is picked up by the existing source glob and needs no CMake change; no binary, xclbin or closed component is touched. src/include/models/granite/q4nx_host.hpp q4nx q4 -> bf16 on the host src/include/models/granite/granite_npu.hpp the causal_lm implementation src/common/models/granite_npu.cpp forward/prefill, KV cache, sampler src/common/AutoModel/modeling_granite.cpp load_model, chat template, generate + registration in all_models.hpp / automodel.hpp Granite needs head_dim 64 at hidden 2560, and every shipped design at hidden >= 2560 is head_dim 128, so no existing engine could be reused. The q4nx layout is derived from the published model file and the MIT headers in this repository; nothing was disassembled or reverse-engineered. GRANITE_DEBUG_DUMP=<path> writes (start_pos, ids[], logits[]) at prefill or a chosen decode step, so a forward pass can be diffed against a reference rather than argued about from output text. It found a real bug during this port and is left in, off unless the variable is set.
Weights: https://huggingface.co/vegahyo/Granite-4.2-3B-NPU2 -- a q4nx conversion of IBM's Apache-2.0 granite-4.2-3b, redistributed under the same licence with attribution. Happy to move it under the FastFlowLM org if you would rather host it. model_info.json is the hub's own tree listing for that repo, fetched from the file_url the entry records, so the two cannot disagree. Two things worth a look: * `chat_template.jinja` is in the `files` list. This model has no chat_template in its tokenizer_config, so without the file automodel.cpp reports "No template file found and no chat_template in tokenizer_config.json" and the model will not start. Five shipped entries already list it for the same reason. * `flm_min_version` is 1.0.4 on the assumption that this lands in the next release; adjust if it lands elsewhere. On an older binary the entry is correctly flagged rather than silently unusable. Verified against a local build: `flm list` shows the entry, every file passes the hash check against the published repo, and `flm serve granite:3b` answers correctly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
granite-4.2-3bdoes not run on FastFlowLM today, and no shipped engine can bereused for it: granite is
head_dim 64athidden 2560, every design shippedat
hidden >= 2560ishead_dim 128, and head_dim cannot be padded. This addsthe family so
flm run granite:3bworks.This is the first of two PRs. This one is the model and an open host engine
— self-contained, no binary, no build change, useful on its own. A second PR
will propose that AIE kernel source can live in the repository, with the
granite NPU kernels as the worked example; it is independent of this one and
neither requires the other. Flagging it here so the second is not a surprise.
Technical Details
Nine new files and two existing ones touched (one
#include; one enum entry, onemap entry and one
case). The engine is plain C++ undersrc/common/models/,so the existing source glob at
src/CMakeLists.txtpicks it up and no CMakechange is needed. No binary,
.xclbin, DLL or closed component is added,modified or referenced.
The engine runs on the CPU. It implements the eleven
causal_lmvirtualsdirectly rather than through a closed engine library, because there is no
closed engine for this geometry.
Weights: https://huggingface.co/vegahyo/Granite-4.2-3B-NPU2 — a q4nx
conversion of IBM's Apache-2.0
granite-4.2-3b, redistributed under the samelicence with attribution. Happy to move it under the FastFlowLM org if you
would rather host it; that is a one-line change to the entry.
model_info.jsonis the hub's own tree listing for that repo, fetched from thefile_urlthe entry records, so the two cannot disagree.Two details worth a look:
chat_template.jinjais in thefileslist. This model has nochat_templatein itstokenizer_config.json, so without the fileautomodel.cppreports "No template file found and no chat_template intokenizer_config.json" and it will not start. Five shipped entries already
list it for the same reason.
flm_min_versionis1.0.5, assuming this lands after 1.0.4 — adjust if itlands elsewhere. On an older binary the entry is then correctly flagged
rather than silently unusable.
GRANITE_DEBUG_DUMP=<path>writes(start_pos, ids[], logits[])at prefill ora chosen decode step so a forward pass can be diffed against a reference rather
than argued about from output text. It is inert unless the variable is set, and
it found a real bug during this port. Happy to drop it if you would rather not
carry it.
Test Plan
Built from this branch on top of
main(6002e0f), MSVC Release, Windows,Ryzen AI 9 HX 370, then:
flm list— the entry appears and is version-gated correctlymodel_list.jsonis checked against the publishedrepo's hashes by the existing downloader path
flm serve granite:3b+ an OpenAI-format chat completion — checks that theengine produces a coherent answer, closes its
<think>block and stops onEOS rather than on the token cap
written from
config.json, at prefill and at decode steps 1 and 50src/test/holds per-engine developer harnesses that need model weights and anNPU and are not run by any workflow. I did not add a
src/test/granite_npu/tomatch, to keep the diff to the feature — glad to add one in that pattern if
you want it, it is mechanical.
Test Result
Build: succeeds, no new warnings attributable to these files.
All five declared files pass the hash check against the published repository,
so the artefact users will download is byte-identical to the one these results
were produced on.
Numerical agreement against the independent numpy forward pass: cosine