Expected single token encoding for <|image_start|> for LFM2-VL-450M

[YingjieHe]

When running FLM2-VL-450M models on ios with flutter, it will throw the error Expected single token encoding for <|image_start|> at

cactus/cactus/models/model_lfm2vl.cpp

Line 226 in 10d04d7

throw std::runtime_error("Expected single token encoding for " + token_text);

.

It looks like the tokenizer cannot recognize the image start token

verified the downloaded model, we do have <|image_start|> in tokenizer.json with special: true

    {
      "id": 499,
      "content": "<|image_end|>",
      "single_word": false,
      "lstrip": false,
      "rstrip": false,
      "normalized": false,
      "special": true
    },

Model version: https://huggingface.co/Cactus-Compute/LFM2-VL-450M/commit/35f3e4525551681aa9c044cf653587337151b792
Cactus version: synced to head(10d04d7)

[obielin]

Hi Henry, I'd like to wrap my hands on this one if it's still open.

I faced a similar issue while working with special tokens in custom tokenizers — the token was correctly marked as special: true in tokenizer.json, but the tokenizer at runtime wasn't registering the added_tokens map, so multi-character tokens like <|image_start|> got split into multiple BPE pieces instead of encoding to a single ID.

Here's how I'd like to approach it:

• Reproduce locally with LFM2-VL-450M at head (10d04d7) and confirm <|image_start|> encodes into more than one token.
• Trace how model_lfm2vl.cpp (around line 226) loads the tokenizer — specifically whether the special / added tokens from tokenizer.json are being registered before encoding happens.
• If they're being skipped, wire up the special tokens map at tokenizer init so they encode to their assigned single IDs (e.g. <|image_end|> → 499, and the corresponding ID for <|image_start|>).
• Add a small round-trip unit test for the image special tokens to prevent regressions.

Does this direction sound good to you? I'm Happy to adjust if you'd prefer a different angle. If you can assign the issue to me, I'll get started.