common : add env to override n_rs_seq

[Green-Sky]

Overview

This allows us to manually specify how many recurrent state rollback states we use for speculation. Currently it is set to 0 for all non-draft-model speculators, which can hurt performance a lot. But on the other side you have to be careful to be conservative with the rollback states, as they balloon quick as they are unquantized f32.

Additional information

Without rollback, partial accepted sequences need to perform a full checkpoint rollback, and on my setup that is prohibitively expensive.

Exampel I use:

$ LLAMA_N_RS_SEQ=7 GGML_OP_OFFLOAD_MIN_BATCH=8 llama-server ... --spec-type ngram-mod --spec-ngram-mod-n-min 7 --spec-ngram-mod-n-max 7 --spec-ngram-mod-n-match = 24

which for qwen_3_5_moe with 168192ctx gives:

llama_context: n_rs_seq      = 7
llama_context: n_outputs_max = 8
llama_context: n_ctx_seq (168192) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
llama_context:  CUDA_Host  output buffer size =     0.95 MiB
llama_kv_cache:      CUDA0 KV buffer size =  1745.16 MiB
llama_kv_cache: size = 1745.16 MiB (168192 cells,  10 layers,  1/1 seqs), K (q8_0):  872.58 MiB, V (q8_0):  872.58 MiB
llama_kv_cache: attn_rot_k = 1, n_embd_head_k_all = 256
llama_kv_cache: attn_rot_v = 1, n_embd_head_k_all = 256
llama_memory_recurrent:      CUDA0 RS buffer size =   502.50 MiB
llama_memory_recurrent: size =  502.50 MiB (     1 cells,  40 layers,  1 seqs  7 rs_seq), R (f32):   22.50 MiB, S (f32):  480.00 MiB

which is 500MiB of recurrent memory, which is a lot on a 8gig card.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yea I had ornith 35B on a leash for >100k tokens before it sniffed out whats wrong.

[Green-Sky]

ngram speculation is currently hard to benchmark. I ran the speed bench, but that rarely triggers ngram spec. In practice, agents with edit toolcalls make use of ngram spec a lot. Like a hole bunch.
Agentic harnesses use edit toolcalls that need to provide diff like context around the change, to uniquely identify the place. This means reciting from context verbatim -> ngram time.

speed bench with the example settings from above:

Comparison: baseline=speed_bench_ornith_35b_coding_baseline.json speculative=speed_bench_ornith_35b_coding_ngram-mod-7-7-24.json

category	base_avg_pred_t/s	spec_avg_pred_t/s	decode_speedup	base_avg_latency	spec_avg_latency	latency_speedup	accept_rate
coding	10.75	10.37	0.97x	107.939s	112.918s	0.96x	0.6668

^the ~400MiB extra memory are mostly at fault fault here I think. Also no tool calls.