Fix worker status for unusable terminal backend responses by ushaket · Pull Request #615 · vllm-project/guidellm

ushaket · 2026-02-26T09:37:58Z

Summary

This PR fixes a scheduler correctness bug where requests could be marked as completed even when the backend resolved without a usable terminal response. It adds explicit terminal-response validation in the worker so malformed/empty terminal results are surfaced as errored with a clear diagnostic instead of being counted as successful requests.

Details

Added terminal response validation in WorkerProcess to guard final status transitions.
Updated request finalization logic so unusable terminal responses set:
- status: errored
- error message: [UNUSABLE_BACKEND_RESPONSE] backend resolved without a usable terminal response payload
Implemented GenerationResponse-aware usability criteria:
- usable if non-empty text or output_metrics.total_tokens > 0
- None terminal response is always unusable
- non-GenerationResponse fallback remains bool(response) for generic/test compatibility
Added regression tests in tests/unit/scheduler/test_worker.py for:
- no terminal response object -> errored
- empty GenerationResponse -> errored
- token-bearing GenerationResponse with empty text -> completed
Preserved existing cancellation/error flow behavior outside terminal-response validation.

Test Plan

Run targeted worker regression tests:
- uv run pytest -q tests/unit/scheduler/test_worker.py -k "terminal_response or empty_generation_response or generation_response_with_tokens or invalid_initialization"
Verify expected outcomes:
- missing terminal response is not marked completed
- empty GenerationResponse is not marked completed
- non-empty signal (output_tokens > 0) is accepted as completed
Confirm no lint issues on touched files.

Related Issues

Resolves Worker marks completed/successful when final response has no usable data #613

"I certify that all code in this PR is my own, except as noted below."

Use of AI

Includes AI-assisted code completion
Includes code generated by an AI application
Includes AI-generated tests (NOTE: AI written tests should have a docstring that includes ## WRITTEN BY AI ##)

git log

commit ffedece
Author: Uri Shaket ushaket@redhat.com
Date: Mon Mar 2 23:55:49 2026 +0200

mvoe validation logic to RequestHandlers

Signed-off-by: Uri Shaket <ushaket@redhat.com>

commit 03cdec7
Author: Uri Shaket ushaket@redhat.com
Date: Mon Mar 2 23:57:34 2026 +0200

revert worker changes

Signed-off-by: Uri Shaket <ushaket@redhat.com>

commit e5fda32
Author: Uri Shaket ushaket@redhat.com
Date: Wed Jul 15 11:25:12 2026 +0300

fix tests

Signed-off-by: Uri Shaket <ushaket@redhat.com>

commit d459b2e
Author: Uri Shaket ushaket@redhat.com
Date: Wed Jul 15 13:23:05 2026 +0300

self CR

Signed-off-by: Uri Shaket <ushaket@redhat.com>

commit 36036d0
Author: Uri Shaket ushaket@redhat.com
Date: Thu Jul 16 17:28:58 2026 +0300

Clarify unusable backend response error message

Signed-off-by: Uri Shaket <ushaket@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

commit 1789158
Author: Uri Shaket ushaket@redhat.com
Date: Tue Jul 21 20:39:29 2026 +0300

refactor: centralize post_validation hook for response validation

Add post_validation to OpenAIRequestHandler and OpenAIWSRequestHandler
protocols with a permissive no-op default. TextCompletionsRequestHandler
and ResponsesRequestHandler override with a check for text, tool_calls,
or output_tokens. PoolingRequestHandler and EmbeddingsRequestHandler
keep the default no-op since they produce non-text output.

Validation is called from the backend resolve methods (http.py,
websocket.py) after compile, not inside each handler's compile methods.
Cancellation paths skip validation for partial responses.

Fixes tool-call-only false negative: responses with tool_calls but no
text are now correctly treated as valid output.

Assisted-by: Claude
Signed-off-by: Uri Shaket <ushaket@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

commit 6de940a
Author: Uri Shaket ushaket@redhat.com
Date: Thu Jul 23 09:44:58 2026 +0300

refactor: deduplicate post_validation and add docstrings

Extract _validate_text_response helper shared by
TextCompletionsRequestHandler and ResponsesRequestHandler.
Add docstrings to all post_validation overrides.

Assisted-by: Claude
Signed-off-by: Uri Shaket <ushaket@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

Assisted-by: Claude
Co-authored-by: Cursor cursoragent@cursor.com
Signed-off-by: Uri Shaket ushaket@redhat.com

ushaket · 2026-02-26T09:43:33Z

This introduce worker dependency on GenerationResponse, not sure if that's the right way to go,
The reason I added it is that the we need to somehow know what we're looking for, in the future we might have ResponseT that doesn't return text and total_tokens, so if we want to keep this check in the worker, we'll need to be aware of the different ResponseT possible

sjmonson · 2026-03-02T20:45:42Z

Hmm, yeah I am not a fan of depending on GenerationResponse in the worker. Can you maybe just check inside the OpenAIRequestHandler.compile_streaming and OpenAIRequestHandler.compile_non_streaming methods? It does not generalize as nicely but what qualifies as an unusable response may differ between backends/endpoints. If you throw an exception in those methods it should propagate up to the worker.

ushaket · 2026-03-02T22:01:04Z

Done

sjmonson

Can you rebase and get rid of the merge commits?

mergify · 2026-03-30T13:06:09Z

@ushaket, this project requires a linear history on feature branches.
Your PR contains merge commits. Please rebase your branch against main
and remove them.

You can do this by running:
git pull --rebase upstream main

dbutenhof

I have really mixed feelings about this. I don't like the idea of hacking this into each request handler separately -- and you've only handled a subset of the http request handlers in this PR. Maybe that's OK, but it feels weird.

I like your original idea of triggering this through the worker, perhaps by delegating to a backend validate_response method (which could, if necessary, delegate to individual request handlers where more specialized knowledge is required). Your check, for example, is pretty much generic and might even be a default implementation on the base Backend class that everyone would get by default.

(The risk, of course, is that such a default might break some weird special handlers that don't have "normal" text/token outputs; which means a generalization would require extensive testing through all the validations.)

SkiHatDuckie · 2026-07-16T12:36:00Z

+            {"choices": [], "usage": {}},
+        ],
+    )
+    def test_non_streaming_raises_for_unusable_terminal_payload(


These tests (and the ones starting at 671) feel like duplicates of each other. But that's an issue for a different PR.

Tests are working for me, and I don't really have any additional complaints to add onto what the others have said.

ushaket · 2026-07-16T14:52:28Z

@dbutenhof Thanks for the feedback, I agree the per-handler approach is a bit awkward, and it currently only covers text/chat.

I prototyped extending this to Responses (same text / tool_calls / tokens check) and Embeddings (raw data[].embedding check, since those responses never have text). Happy to include either or both in this PR if that’s useful.

How would you like to proceed?

Keep this PR scoped to text/chat and take broader coverage in a follow-up
Add Responses here
Add Responses + Embeddings here
Pull @sjmonson in and decide together (especially vs. a Backend.validate_response hook)

I’m fine with any of these, just want to align before I push more surface area into this PR.

dbutenhof · 2026-07-20T18:54:48Z

Keep this PR scoped to text/chat and take broader coverage in a follow-up Add Responses here Add Responses + Embeddings here Pull @sjmonson in and decide together (especially vs. a Backend.validate_response hook)

I’m fine with any of these, just want to align before I push more surface area into this PR.

Yeah, the risk is the "special cases" that don't have text, like embeddings, Geospatial pooling requests (I think), and maybe others.

While I like the idea of a meaningful Backend implementation that "works for most", the downside is a lot of awkward testing to be sure it doesn't break anything.

A safer alternative is probably a base implementation that does nothing, overridden by the various backends & request format handlers as necessary... I do prefer centralizing it from the worker loop, delegating through the backend to the request format... but as most of the implementation probably falls to the request formatter, maybe it doesn't matter much.

jaredoconnell · 2026-07-20T22:02:36Z

Keep this PR scoped to text/chat and take broader coverage in a follow-up Add Responses here Add Responses + Embeddings here Pull @sjmonson in and decide together (especially vs. a Backend.validate_response hook)
I’m fine with any of these, just want to align before I push more surface area into this PR.

Yeah, the risk is the "special cases" that don't have text, like embeddings, Geospatial pooling requests (I think), and maybe others.

While I like the idea of a meaningful Backend implementation that "works for most", the downside is a lot of awkward testing to be sure it doesn't break anything.

A safer alternative is probably a base implementation that does nothing, overridden by the various backends & request format handlers as necessary... I do prefer centralizing it from the worker loop, delegating through the backend to the request format... but as most of the implementation probably falls to the request formatter, maybe it doesn't matter much.

Dave's proposal sounds good. It can have a default implementation that is permissive, and can be overridden whenever you need to validate this. The function could maybe be called "post_validation".

One thing that needs to be kept in mind is that for the chat completions endpoints and the responses API, no text is valid due to tool calls, and for other endpoints it's valid due to non-token content.

Signed-off-by: Uri Shaket <ushaket@redhat.com>

Signed-off-by: Uri Shaket <ushaket@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Add post_validation to OpenAIRequestHandler and OpenAIWSRequestHandler protocols with a permissive no-op default. TextCompletionsRequestHandler and ResponsesRequestHandler override with a check for text, tool_calls, or output_tokens. PoolingRequestHandler and EmbeddingsRequestHandler keep the default no-op since they produce non-text output. Validation is called from the backend resolve methods (http.py, websocket.py) after compile, not inside each handler's compile methods. Cancellation paths skip validation for partial responses. Fixes tool-call-only false negative: responses with tool_calls but no text are now correctly treated as valid output. Assisted-by: Claude Signed-off-by: Uri Shaket <ushaket@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>

ushaket · 2026-07-21T18:31:36Z

Thanks @dbutenhof, @jaredoconnell , pushed the refactor:

post_validation added to the handler protocol with a no-op default, overridden in TextCompletionsRequestHandler and ResponsesRequestHandler
Call site is in the backend resolve methods after compile, so new handlers get the no-op automatically
Fixed the tool-call blind spot @jaredoconnell flagged: check now passes when any of text, tool_calls, or output_tokens is present
Pooling and embeddings keep the no-op
Cancellation paths skip validation

Kept it at the backend resolve level rather than the worker loop since the implementation lives on the request handlers anyway as @dbutenhof pointed out

jaredoconnell

This seems like an appropriate design. One thing is that OpenAIRequestHandler is a protocol, so there is no default implementation. So no-op calls need to be added to RealtimeTranscriptionWSRequestHandler and EmbeddingsRequestHandler.

And I added a comment regarding de-duplicating the function you added.

jaredoconnell · 2026-07-21T19:16:51Z

+    def post_validation(self, response: GenerationResponse) -> None:
+        has_text = bool(response.text and response.text.strip())
+        has_tool_calls = bool(response.tool_calls)
+        output_tokens = response.output_metrics.total_tokens or 0
+        if not has_text and not has_tool_calls and output_tokens <= 0:
+            raise ValueError(
+                "[UNUSABLE_BACKEND_RESPONSE] backend resolved with empty "
+                "response payload"
+            )
+


It might be worth de-duplicating this and the other identical post_validation methods by making them both call the same helper function.

Done, extracted _validate_text_response helper, both handlers call it now.

dbutenhof

I think there are a lot of compromises in this approach, but it's probably the lowest touch and safest. I'd like to see more consistent style in your validation methods, including docstrings; but aside from that, it's probably as good as it gets.

dbutenhof · 2026-07-21T19:35:03Z

        """
        ...

+    def post_validation(self, response: GenerationResponse) -> None:


Duplicating this is ugly -- but I guess I also have to agree with not constructing a proper class hierarchy in this PR that would allow sharing it. That was why I was hoping to put it on Backend, because the backend classes do have a working hierarchy while the request handlers are a lot more chaotic. But this is probably safer since it minimizes the blast radius.

Agreed, extracted a shared _validate_text_response helper to avoid the copy-paste.

dbutenhof · 2026-07-21T19:36:44Z

        )

+    def post_validation(self, response: GenerationResponse) -> None:
+        has_text = bool(response.text and response.text.strip())


You wrote a big docstring for the null implementations, but nothing for the ones that do something? 😁

added docstrings to all three overrides (Text, Responses, Pooling).

dbutenhof · 2026-07-21T19:37:41Z

    pooling-specific request structure with nested data fields.
    """

+    def post_validation(self, response: GenerationResponse) -> None:  # noqa: ARG002


And ... completely different style here, without a docstring?

added a docstring explaining why pooling skips validation.

Extract _validate_text_response helper shared by TextCompletionsRequestHandler and ResponsesRequestHandler. Add docstrings to all post_validation overrides. Assisted-by: Claude Signed-off-by: Uri Shaket <ushaket@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>

ushaket marked this pull request as draft February 26, 2026 09:38

ushaket marked this pull request as ready for review March 2, 2026 21:59

sjmonson requested changes Mar 3, 2026

View reviewed changes

dbutenhof added this to the v0.7.0 milestone Mar 30, 2026

mergify Bot added the needs-rebase label Mar 30, 2026

ushaket force-pushed the fix/worker-unusable-terminal-response branch from 60e1abc to d35b823 Compare March 30, 2026 13:12

mergify Bot removed the needs-rebase label Mar 30, 2026

ushaket force-pushed the fix/worker-unusable-terminal-response branch from d35b823 to d82eb51 Compare March 30, 2026 13:18

sjmonson modified the milestones: v0.7.0, v0.7.2 Jul 1, 2026

ushaket force-pushed the fix/worker-unusable-terminal-response branch from d82eb51 to 4388149 Compare July 15, 2026 07:54

dbutenhof requested review from SkiHatDuckie, dbutenhof and jaredoconnell July 15, 2026 11:31

dbutenhof added community contribution An opportunity for contribution from the GuideLLM community already invested in this area. bug Represents a user-visible defect labels Jul 15, 2026

dbutenhof reviewed Jul 15, 2026

View reviewed changes

jaredoconnell reviewed Jul 15, 2026

View reviewed changes

Comment thread src/guidellm/backends/openai/request_handlers.py Outdated

SkiHatDuckie approved these changes Jul 16, 2026

View reviewed changes

ushaket added 4 commits July 21, 2026 17:45

mvoe validation logic to RequestHandlers

ffedece

Signed-off-by: Uri Shaket <ushaket@redhat.com>

revert worker changes

03cdec7

Signed-off-by: Uri Shaket <ushaket@redhat.com>

fix tests

e5fda32

Signed-off-by: Uri Shaket <ushaket@redhat.com>

self CR

d459b2e

Signed-off-by: Uri Shaket <ushaket@redhat.com>

ushaket and others added 2 commits July 21, 2026 17:45

Clarify unusable backend response error message

36036d0

Signed-off-by: Uri Shaket <ushaket@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>

ushaket force-pushed the fix/worker-unusable-terminal-response branch from 509ce95 to 1789158 Compare July 21, 2026 18:14

jaredoconnell reviewed Jul 21, 2026

View reviewed changes

dbutenhof requested changes Jul 21, 2026

View reviewed changes

dbutenhof modified the milestones: v0.7.2, v0.8.0 Jul 24, 2026

Uh oh!

Conversation

ushaket commented Feb 26, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Details

Test Plan

Related Issues

Use of AI

git log

Uh oh!

ushaket commented Feb 26, 2026

Uh oh!

sjmonson commented Mar 2, 2026

Uh oh!

ushaket commented Mar 2, 2026

Uh oh!

sjmonson left a comment

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Mar 30, 2026

Uh oh!

dbutenhof left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ushaket commented Jul 16, 2026

Uh oh!

dbutenhof commented Jul 20, 2026

Uh oh!

jaredoconnell commented Jul 20, 2026

Uh oh!

ushaket commented Jul 21, 2026

Uh oh!

jaredoconnell left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dbutenhof left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ushaket commented Feb 26, 2026 •

edited by github-actions Bot

Loading