Suffix decode

[kashif]

Overview

Initial implementation of https://suffix-decoding.github.io/, which is a model-free spec. decoding method. We build the tree online over the current request and tokens generated so far, and it's best when the matched suffix is longer.

Additional information

Currently, we build only the online tree only and no global corpus tree. vLLM optionally keeps the global cross-request cache (up to 10k past requests). We implement the per-request/prompt tree only.

Requirements

• I have read and agree with the contributing guidelines YES
• AI usage disclosure: YES for tests and docs and fixing code vs. reference code

[ServeurpersoCom]

The concept is very elegant! It treats the context and previous outputs as a memory of sequences, proposes the continuation of a previously seen pattern in one block, and then has the LLM verify it, so it mimics the effect of MTP without requiring MTP heads. It becomes extremely effective whenever patterns repeat, especially in code, structured documents, JSON, and agentic loops.

[am17an]

Can you add some unit-tests to showcase the functionality?

[Green-Sky]

without looking too deep, sounds just like ngram with tree, like #8648 (tho that pr is probably dead)

[ServeurpersoCom]

Launching CI :p

[ServeurpersoCom]

Tested on my prod (Qwen 3.6 dense 27B Q8, single slot): default config drops decode from 42 to 31 t/s on normal prompts. The per-request tree has count=1 almost everywhere so min_prob filters nothing, and short junk drafts eat the perf in verification.

Attached commit adds a n_min gate (default 3) like the other draftless impls -> back to 41 t/s worst case, long matches untouched. On a MoE the gap is even bigger: 110 t/s without the gate vs 185 t/s with it. Default can go back down once a global cross-request cache gives real frequency stats.

I'd suggest adding this patch and default values until the global cross-request cache from the paper is implemented:
with real frequency stats accumulated across requests, min_prob becomes meaningful again as the adaptive gate and n_min can go back down to 1 :

ServeurpersoCom@2cfce6c

diff --git a/common/arg.cpp b/common/arg.cpp
index 8f77830..7d6f2d3 100644
--- a/common/arg.cpp
+++ b/common/arg.cpp
@@ -4198,6 +4198,16 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
             params.speculative.ngram_suffix.n_max = value;
         }
     ).set_spec().set_examples({LLAMA_EXAMPLE_SPECULATIVE, LLAMA_EXAMPLE_SERVER, LLAMA_EXAMPLE_CLI}));
+    add_opt(common_arg(
+        {"--spec-ngram-suffix-n-min"}, "N",
+        string_format("discard drafts shorter than this for ngram-suffix speculative decoding (default: %d)", params.speculative.ngram_suffix.n_min),
+        [](common_params & params, int value) {
+            if (value < 1) {
+                throw std::invalid_argument("ngram-suffix n-min must be at least 1");
+            }
+            params.speculative.ngram_suffix.n_min = value;
+        }
+    ).set_spec().set_examples({LLAMA_EXAMPLE_SPECULATIVE, LLAMA_EXAMPLE_SERVER, LLAMA_EXAMPLE_CLI}));
     add_opt(common_arg(
         {"--spec-ngram-suffix-max-factor"}, "F",
         string_format("draft up to match_len * F tokens for ngram-suffix speculative decoding (default: %.1f)", params.speculative.ngram_suffix.max_factor),
diff --git a/common/common.h b/common/common.h
index 7065636..3871ce1 100644
--- a/common/common.h
+++ b/common/common.h
@@ -370,6 +370,7 @@ struct common_params_speculative_ngram_cache {
 struct common_params_speculative_ngram_suffix {
     int32_t max_depth = 24;    // suffix-tree depth = max context-match length
     int32_t n_max     = 24;    // maximum number of drafted tokens
+    int32_t n_min     = 3;     // discard drafts shorter than this
     float   max_factor = 1.0f; // draft up to match_len * max_factor tokens
     float   min_prob   = 0.1f; // stop drafting below this frequency probability
 };
diff --git a/common/speculative.cpp b/common/speculative.cpp
index ee98261..85b0a78 100644
--- a/common/speculative.cpp
+++ b/common/speculative.cpp
@@ -1822,6 +1822,11 @@ struct common_speculative_impl_ngram_suffix : public common_speculative_impl {
             common_suffix_draft draft = st.tree.speculate(
                     context, params.n_max, params.max_factor, params.min_prob);

+            // a short draft is unlikely to pay for its verification batch.
+            if ((int32_t) draft.tokens.size() < params.n_min) {
+                continue;
+            }
+
             *dp.result = std::move(draft.tokens);
         }
     }

[ServeurpersoCom]

Quick demo on a dense model: chat baseline runs at 41 t/s, then repetitive XML generation ramps up to 122 t/s as the suffix tree warms up :

suffix-decode.mp4

[Green-Sky]

@ServeurpersoCom opening a minimal pr in a sec that can improve perf further for all non draft-model speculators for models with recurrent state.

edit: here #26499

[kashif]

thanks! cherry picking the commit thanks @ServeurpersoCom