fix(skippy): preserve token ordering during sampled verification

[i386]

Why this matters

While checking several predicted tokens, Mesh could let an earlier decision see tokens from later in the batch or count tokens twice. That can make accelerated generation choose different tokens from normal generation.

This change makes each prediction see only the token sequence that should exist at that exact point.

Technical details

• remove the complete eagerly recorded verification batch before sampled row evaluation
• re-admit each input token exactly once and in position order
• fail closed on token-history underflow or sampler/history disagreement
• retain token history for non-output verification stages

This keeps repetition penalties, grammar state, and other history-sensitive sampling behavior aligned with serial decoding.

Validation

• bash scripts/prepare-llama.sh — complete 65-patch queue applied successfully
• cargo test -p skippy-runtime --quiet — 68 passed

This PR is based directly on main and has no dependency on the companion fixes.

Summary by CodeRabbit

• Bug Fixes
  • Improved sampled verification so token history remains consistent during validation.
  • Added checks to prevent invalid or prematurely appended tokens from affecting verification.
  • Ensured non-output verification correctly tracks complete token batches.
  • Preserved sampler state consistency when processing verified tokens.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 97a5a2b6-e446-4839-979e-172889fa8aa5

📥 Commits

Reviewing files that changed from the base of the PR and between 73e29c7 and 13bc833.

📒 Files selected for processing (1)

• third_party/llama.cpp/patches/0048-Fix-sampled-verification-token-history.patch

🚧 Files skipped from review as they are similar to previous changes (1)

• third_party/llama.cpp/patches/0048-Fix-sampled-verification-token-history.patch

---

📝 Walkthrough

Walkthrough

The PR replaces the previous sampled-verification patch with token-history validation, sampler-state alignment checks, causal token recording, and full-batch recording when output sampling is disabled.

Changes

Sampled verification

Layer / File(s)	Summary
Token-history validation and recording third_party/llama.cpp/patches/0048-Fix-sampled-verification-token-history.patch	Sampled verification removes eagerly appended token history, rejects history underflow and sampler-state mismatches, records input tokens before sampling their rows, and records the full batch when output sampling is disabled. The previous grammar-state patch is deleted.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant SampledVerification
  participant TokenHistory
  participant SamplerState
  SampledVerification->>TokenHistory: Remove eagerly appended tokens
  SampledVerification->>SamplerState: Validate history alignment
  SampledVerification->>TokenHistory: Record tokens before sampling

Loading

Possibly related PRs

• Mesh-LLM/mesh-llm#1172: Both PRs modify sampled speculative verification state handling in skippy.cpp.

Suggested reviewers: michaelneale

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: preserving token ordering during sampled verification.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch agent/fix-sampled-verification-token-history

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

This pull request is currently a draft. Reviews will not take place until the PR is marked as ready for review.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@third_party/llama.cpp/patches/0048-Fix-sampled-verification-token-history.patch`:
- Around line 45-46: Update the non-sampled branch around skippy_record_tokens
to reject sampler/history disagreement before recording tokens: if
sampling_accepted_token_count exceeds session->token_history.size(), return
SKIPPY_STATUS_RUNTIME_ERROR; otherwise preserve the existing recording behavior.
Add a regression test covering accepted sampler tokens exceeding the existing
history, including non-output recording, and verify the runtime error is
returned.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: e03bffd0-6518-432f-90f8-2910cac2017d

📥 Commits

Reviewing files that changed from the base of the PR and between 4bd1453 and 73e29c7.

📒 Files selected for processing (2)

• third_party/llama.cpp/patches/0048-Fix-sampled-verification-token-history.patch
• third_party/llama.cpp/patches/0048-Preserve-grammar-state-during-sampled-verification.patch

💤 Files with no reviewable changes (1)

• third_party/llama.cpp/patches/0048-Preserve-grammar-state-during-sampled-verification.patch