Non-degenerate BPE vocabularies have the property: all tokens with byte-length greater than 1 have at least two lower ranked parents in the merge map which can be merged to produce that token.
So, for any token "ABCD" there is at least one pair of smaller token spans which can be merged to produce that sequence.
Some newer token vocabularies break this rule, (looking at you, Qwen3.5), and have tokens in the vocabulary which cannot be produced by a merge of 2 smaller parents.
The current vocab impls prohibit this; as do the encoder algorithms.
Non-degenerate BPE vocabularies have the property: all tokens with byte-length greater than 1 have at least two lower ranked parents in the merge map which can be merged to produce that token.
So, for any token "ABCD" there is at least one pair of smaller token spans which can be merged to produce that sequence.
Some newer token vocabularies break this rule, (looking at you, Qwen3.5), and have tokens in the vocabulary which cannot be produced by a merge of 2 smaller parents.
The current vocab impls prohibit this; as do the encoder algorithms.