Fit OpenCode generation budget to context - #30
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3dbf7d80e2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| for key, value in extra_body.items() | ||
| if key not in RESERVED_GENERATION_KWARGS |
There was a problem hiding this comment.
Honor max_tokens carried in extra_body
When a client sends the completion cap only in extra_body (for example the request shape added in the new tests, but with a smaller value), this filter drops max_tokens before the cap is read from only top-level max_completion_tokens/max_tokens. Those requests now fall back to config.max_tokens_per_call (4096 by default), so a caller asking for a short generation can be forwarded to TRL with thousands of tokens and exceed the expected per-call or idle budget; the reserved value should still be consumed as the requested cap before it is omitted from generation_kwargs.
Useful? React with 👍 / 👎.
Summary
extra_bodyfrom overriding enforcedmax_tokens,n, orlogprobsVerification
pytest pipelines/benchflow-task-posttrain/tests -q(228 passed)ruff check pipelines/benchflow-task-posttrain/src pipelines/benchflow-task-posttrain/testspython -m compileall -q pipelines/benchflow-task-posttrain/srcgit diff --check