Skip to content

feat: granite 4.2 - #700

Open
vegah wants to merge 3 commits into
ROCm:mainfrom
vegah:feat/granite-4.2
Open

feat: granite 4.2#700
vegah wants to merge 3 commits into
ROCm:mainfrom
vegah:feat/granite-4.2

Conversation

@vegah

@vegah vegah commented Sep 2, 2026

Copy link
Copy Markdown

Motivation

granite-4.2-3b does not run on FastFlowLM today, and no shipped engine can be
reused for it: granite is head_dim 64 at hidden 2560, every design shipped
at hidden >= 2560 is head_dim 128, and head_dim cannot be padded. This adds
the family so flm run granite:3b works.

This is the first of two PRs. This one is the model and an open host engine
— self-contained, no binary, no build change, useful on its own. A second PR
will propose that AIE kernel source can live in the repository, with the
granite NPU kernels as the worked example; it is independent of this one and
neither requires the other. Flagging it here so the second is not a surprise.

Technical Details

Nine new files and two existing ones touched (one #include; one enum entry, one
map entry and one case). The engine is plain C++ under src/common/models/,
so the existing source glob at src/CMakeLists.txt picks it up and no CMake
change is needed
. No binary, .xclbin, DLL or closed component is added,
modified or referenced.

src/include/models/granite/q4nx_host.hpp    q4nx q4 -> bf16 on the host
src/include/models/granite/granite_npu.hpp  the causal_lm implementation
src/common/models/granite_npu.cpp           forward/prefill, KV cache, sampler
src/common/AutoModel/modeling_granite.cpp   load_model, chat template, generate
src/include/AutoModel/{all_models,automodel,modeling_granite}.hpp  registration
src/{model_list,model_info}.json            the granite:3b entry
docs/docs/models/granite.md                 model card + index link

The engine runs on the CPU. It implements the eleven causal_lm virtuals
directly rather than through a closed engine library, because there is no
closed engine for this geometry.

Weights: https://huggingface.co/vegahyo/Granite-4.2-3B-NPU2 — a q4nx
conversion of IBM's Apache-2.0 granite-4.2-3b, redistributed under the same
licence with attribution. Happy to move it under the FastFlowLM org if you
would rather host it; that is a one-line change to the entry.
model_info.json is the hub's own tree listing for that repo, fetched from the
file_url the entry records, so the two cannot disagree.

Two details worth a look:

  • chat_template.jinja is in the files list. This model has no
    chat_template in its tokenizer_config.json, so without the file
    automodel.cpp reports "No template file found and no chat_template in
    tokenizer_config.json"
    and it will not start. Five shipped entries already
    list it for the same reason.
  • flm_min_version is 1.0.5, assuming this lands after 1.0.4 — adjust if it
    lands elsewhere. On an older binary the entry is then correctly flagged
    rather than silently unusable.

GRANITE_DEBUG_DUMP=<path> writes (start_pos, ids[], logits[]) at prefill or
a chosen decode step so a forward pass can be diffed against a reference rather
than argued about from output text. It is inert unless the variable is set, and
it found a real bug during this port. Happy to drop it if you would rather not
carry it.

Test Plan

Built from this branch on top of main (6002e0f), MSVC Release, Windows,
Ryzen AI 9 HX 370, then:

  1. flm list — the entry appears and is version-gated correctly
  2. every file declared in model_list.json is checked against the published
    repo's hashes by the existing downloader path
  3. flm serve granite:3b + an OpenAI-format chat completion — checks that the
    engine produces a coherent answer, closes its <think> block and stops on
    EOS rather than on the token cap
  4. the forward pass was diffed against an independent numpy implementation
    written from config.json, at prefill and at decode steps 1 and 50

src/test/ holds per-engine developer harnesses that need model weights and an
NPU and are not run by any workflow. I did not add a src/test/granite_npu/ to
match, to keep the diff to the feature — glad to add one in that pattern if
you want it
, it is mechanical.

Test Result

Build: succeeds, no new warnings attributable to these files.

$ flm list
  - granite:3b ✅

$ flm serve granite:3b     # then POST /v1/chat/completions "What is 2+2?"
[FLM]  Checking file: config.json...            Success!
[FLM]  Checking file: model.q4nx...             Success!
[FLM]  Checking file: tokenizer.json...         Success!
[FLM]  Checking file: tokenizer_config.json...  Success!
[FLM]  Checking file: chat_template.jinja...    Success!
[FLM]  granite (host engine): hidden 2560, layers 40, heads 40/8, head_dim 64,
       attn_scale 0.125
[FLM]  Prefill chunk 1/1 with 44 tokens

finish_reason: stop, 24 tokens, 8.5 tok/s decode
  The user asked "What is 2+2?" I should answer simply: 4.
  </think>
  4

All five declared files pass the hash check against the published repository,
so the artefact users will download is byte-identical to the one these results
were produced on.

Numerical agreement against the independent numpy forward pass: cosine

0.9999 at prefill and at decode steps 1 and 50, same argmax at all three.

Adds the granite family so granite-4.2-3B runs on FastFlowLM. The engine is
plain C++ in src/common/models/, so it is picked up by the existing source glob
and needs no CMake change; no binary, xclbin or closed component is touched.

  src/include/models/granite/q4nx_host.hpp   q4nx q4 -> bf16 on the host
  src/include/models/granite/granite_npu.hpp the causal_lm implementation
  src/common/models/granite_npu.cpp          forward/prefill, KV cache, sampler
  src/common/AutoModel/modeling_granite.cpp  load_model, chat template, generate
  + registration in all_models.hpp / automodel.hpp

Granite needs head_dim 64 at hidden 2560, and every shipped design at hidden
>= 2560 is head_dim 128, so no existing engine could be reused.

The q4nx layout is derived from the published model file and the MIT headers in
this repository; nothing was disassembled or reverse-engineered.

GRANITE_DEBUG_DUMP=<path> writes (start_pos, ids[], logits[]) at prefill or a
chosen decode step, so a forward pass can be diffed against a reference rather
than argued about from output text. It found a real bug during this port and is
left in, off unless the variable is set.
Weights: https://huggingface.co/vegahyo/Granite-4.2-3B-NPU2 -- a q4nx
conversion of IBM's Apache-2.0 granite-4.2-3b, redistributed under the same
licence with attribution. Happy to move it under the FastFlowLM org if you
would rather host it.

model_info.json is the hub's own tree listing for that repo, fetched from the
file_url the entry records, so the two cannot disagree.

Two things worth a look:

  * `chat_template.jinja` is in the `files` list. This model has no
    chat_template in its tokenizer_config, so without the file automodel.cpp
    reports "No template file found and no chat_template in
    tokenizer_config.json" and the model will not start. Five shipped entries
    already list it for the same reason.

  * `flm_min_version` is 1.0.4 on the assumption that this lands in the next
    release; adjust if it lands elsewhere. On an older binary the entry is
    correctly flagged rather than silently unusable.

Verified against a local build: `flm list` shows the entry, every file passes
the hash check against the published repo, and `flm serve granite:3b` answers
correctly.
@vegah vegah changed the title Feat/granite 4.2 feat: granite 4.2 Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant