Problem Statement
When using the websocket backend for a task which has some sort of correspondence between input packets and output tokens (e.g realtime transcription), we might want more metrics than just the classical LLM metrics such as TTFT, TPOT, E2E etc, but might want to know things like the Round-Trip Time (RTT), which is essentially the user experienced system lag.
Without exact alignment between input packets and output tokens and their timings - we cannot know the exact RTT, but we can estimate it in 3 different ways:
- TTFT - Time between the first sent packet time and the first received token time (
T_recv[0] - T_sent[0]). Basically the RTT of the first token, which is already an OK proxy
- TTLT (Time To Last Token) - Time between the last sent packet time and the last received token time (
T_recv[-1] - T_sent[-1]). The RTT of the last token, also an OK proxy
- Avg RTT (approximate) - The time between the average sent packet time and the average received packet time (
T_recv.mean() - T_sent.mean()). This is a much more stable metric, because like TPOT it averages over all tokens instead of measuring just once per request. However, this is an approximate estimation of the true RTT, because it assumes that the sent packets and received tokens match each other uniformly in time, which is never exactly true.
Proposed Solution
When the websocket backend is used, report TTLT and Avg RTT in addition to the rest of the metrics. They are always simple to calculate with the websocket backend regardless of modality and endpoint, they provide useful information for all websocket use cases, and they will provide a unique advantage for realtime transcription benchmarking by approximating the experienced system lag.
Alternatives Considered
We could just accept TTFT as the only proxy, but it's very different from actual system lag in actual streaming applications.
We could also try to measure the true RTT, but that requires having a dataset with accurate timestamp labels, and to use some forced alignment to align predicted tokens with input audio packets, and accommodate for prediction errors, and it requires a different solution for each future websocket modality. It's an overly complicated solution that will create a fragile benchmark that can't generalize to other datasets.
Usage Examples
python3 -m vllm.entrypoints.openai.api_server \
--model mistralai/Voxtral-Mini-4B-Realtime-2602 \
--tokenizer-mode mistral \
--config-format mistral \
--load-format mistral \
--trust-remote-code \
--compilation-config '{"cudagraph_mode":"PIECEWISE"}' \
--tensor-parallel-size 1 \
--max-model-len 45000 \
--max-num-batched-tokens 8192 \
--max-num-seqs 16 \
--gpu-memory-utilization 0.90 \
--host 0.0.0.0 --port 8000
guidellm benchmark \
--target http://localhost:8000/v1 \
--request-type realtime_transcription \
--backend openai_realtime_ws \
--data /workspace/custom-audio-dataset/hf_dataset \
--profile synchronous \
--max-requests 10 \
--output-dir /workspace/repo/runs/2026-04-23T13-41-41 \
--outputs json,html,csv
# TTLT & Avg RTT always reported when `backend=openai_realtime_ws`Additional Context
#713
https://en.wikipedia.org/wiki/Round-trip_delay
I am interested in creating a PR for this 😄
Problem Statement
When using the websocket backend for a task which has some sort of correspondence between input packets and output tokens (e.g realtime transcription), we might want more metrics than just the classical LLM metrics such as TTFT, TPOT, E2E etc, but might want to know things like the Round-Trip Time (RTT), which is essentially the user experienced system lag.
Without exact alignment between input packets and output tokens and their timings - we cannot know the exact RTT, but we can estimate it in 3 different ways:
T_recv[0] - T_sent[0]). Basically the RTT of the first token, which is already an OK proxyT_recv[-1] - T_sent[-1]). The RTT of the last token, also an OK proxyT_recv.mean() - T_sent.mean()). This is a much more stable metric, because like TPOT it averages over all tokens instead of measuring just once per request. However, this is an approximate estimation of the true RTT, because it assumes that the sent packets and received tokens match each other uniformly in time, which is never exactly true.Proposed Solution
When the websocket backend is used, report TTLT and Avg RTT in addition to the rest of the metrics. They are always simple to calculate with the websocket backend regardless of modality and endpoint, they provide useful information for all websocket use cases, and they will provide a unique advantage for realtime transcription benchmarking by approximating the experienced system lag.
Alternatives Considered
We could just accept TTFT as the only proxy, but it's very different from actual system lag in actual streaming applications.
We could also try to measure the true RTT, but that requires having a dataset with accurate timestamp labels, and to use some forced alignment to align predicted tokens with input audio packets, and accommodate for prediction errors, and it requires a different solution for each future websocket modality. It's an overly complicated solution that will create a fragile benchmark that can't generalize to other datasets.
Usage Examples
Additional Context
#713
https://en.wikipedia.org/wiki/Round-trip_delay
I am interested in creating a PR for this 😄