Skip to content

verify: the verdict reasoning is bounded twice, in different units — a multi-byte reply is truncated to a third of the cap and carries two markers #1948

Description

@macanderson

Problem

Two caps now bound Verdict::reasoning, they disagree about units, and both fire on the same value.

For an ASCII reply the second bound is now dead code: 4,000 chars is 4,000 bytes, so text.len() <= 4_000 always holds and it returns unchanged.

For a non-ASCII reply the two compose badly. A verifier answering in Japanese produces ~3 bytes per character, so a reasoning already clipped to 4,000 chars is ~12,000 bytes. bound_forwarded_reasoning then truncates it again at byte 4,000 — roughly 1,333 characters, a third of the intended budget — and appends its own marker. The worker's revision prompt receives:

<~1333 chars of reasoning>
[verifier reasoning truncated]

...where the first marker ([… verifier reasoning clipped at 4000 characters]) was itself cut off mid-way or lost entirely, depending on where byte 4,000 lands. So a verifier that answers in a non-Latin script gets materially less of its reasoning forwarded than one that answers in English, for no stated reason.

This is the same class of bug clipping_a_multibyte_reply_does_not_panic (crates/stella-pipeline/src/verify/tests/parse.rs) was written to prevent — that test pins the construction bound against a byte-slice panic; nothing covers the interaction with the forwarding bound.

Fix direction

Decide which bound is load-bearing and delete or subordinate the other. Most likely shape:

  1. bounded_reasoning at construction is the structural bound (that is what fix(stella-pipeline): bound the verdict reasoning at the point it is constructed (#1787) #1932 argued for, and it is the one no future consumer can forget). Keep it.
  2. bound_forwarded_reasoning then has nothing left to do for Verdict::reasoning. Check its other callers (rg -n 'bound_forwarded_reasoning' crates/) — if the verdict path is the only one, remove it and FORWARDED_REASONING_MAX_CHARS with it. If other, unbounded text is forwarded through it, keep it but fix the unit to characters so it agrees with its own name and with the sibling cap.

Either way the two constants should not both exist at 4,000 with different meanings.

Verify

Witness: a verifier reply of ~4,000 multi-byte characters (e.g. "これはテストです。".repeat(..) prefixed with a FAIL token, the fixture shape already used in clipping_a_multibyte_reply_does_not_panic) is forwarded to the worker at the same character length as an equivalent ASCII reply, and carries exactly one truncation marker rather than two.

Constraints

  • crates/stella-pipeline/src/pipeline.rs is a grandfathered god file closed to growth — the forwarding call site is there, so a fix must be net-zero or land in a sibling.
  • Model output is runtime data (invariant 5): any slicing stays on character boundaries.

Related

#1787 (the parent issue — item 2 shipped as #1932 and introduced the second cap), #1932.

Found while widening the verdict parse protocol for #1787 item 1; out of scope for that change, which did not touch either bound.

Metadata

Metadata

Assignees

Labels

P2Polish — worth doing, not urgentarea:pipelinestella-pipeline — triage→…→judge orchestrationgoal:enginePillar 4 — engine tuned and optimized: budget, wall-clock, resolve rate

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions