Skip to content

Fit OpenCode generation budget to context - #30

Merged
bingran-you merged 1 commit into
mainfrom
bry/fit-dynamic-generation-budget
Jul 14, 2026
Merged

Fit OpenCode generation budget to context#30
bingran-you merged 1 commit into
mainfrom
bry/fit-dynamic-generation-budget

Conversation

@bingran-you

Copy link
Copy Markdown
Contributor

Summary

  • measure the irreducible OpenCode prompt before reserving completion tokens
  • reduce only the per-turn generation allowance when the prompt needs part of that reserve
  • reject prompts only when no generation token can fit
  • prevent extra_body from overriding enforced max_tokens, n, or logprobs
  • cover the real 12,306-token GRPO regression plus exact boundary and full-evaluation-context cases

Verification

  • pytest pipelines/benchflow-task-posttrain/tests -q (228 passed)
  • ruff check pipelines/benchflow-task-posttrain/src pipelines/benchflow-task-posttrain/tests
  • python -m compileall -q pipelines/benchflow-task-posttrain/src
  • git diff --check

@bingran-you
bingran-you merged commit fd84939 into main Jul 14, 2026
1 check passed
@bingran-you
bingran-you deleted the bry/fit-dynamic-generation-budget branch July 14, 2026 22:21

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3dbf7d80e2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +425 to +426
for key, value in extra_body.items()
if key not in RESERVED_GENERATION_KWARGS

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Honor max_tokens carried in extra_body

When a client sends the completion cap only in extra_body (for example the request shape added in the new tests, but with a smaller value), this filter drops max_tokens before the cap is read from only top-level max_completion_tokens/max_tokens. Those requests now fall back to config.max_tokens_per_call (4096 by default), so a caller asking for a short generation can be forwarded to TRL with thousands of tokens and exceed the expected per-call or idle budget; the reserved value should still be consumed as the requested cap before it is omitted from generation_kwargs.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant