Nemotron ASR Server

An OpenAI-compatible speech-to-text server for NVIDIA's nvidia/nemotron-3.5-asr-streaming-0.6b model, served with FastAPI on top of NVIDIA NeMo.

It implements OpenAI's /v1/audio/transcriptions endpoint — so existing OpenAI clients/SDKs work by just changing the base URL — plus a WebSocket endpoint for real-time streaming.

Transcription only. This project intentionally implements just audio transcription. There is no text generation / chat.

• ✅ POST /v1/audio/transcriptions — json, text, verbose_json, srt, vtt
• ✅ Word-level and segment-level timestamps (OpenAI timestamp_granularities[])
• ✅ Automatic language identification (32+ locales) or forced language
• ✅ WS /v1/audio/transcriptions/stream — real-time streaming transcription
• ✅ Interactive OpenAPI docs at /docs
• ✅ Docker-first: docker compose up
• ✅ Runs on the NVIDIA DGX Spark (GB10) — see below

Does it run on the DGX Spark (GB10)?

Yes. This server was built and verified end-to-end on a DGX Spark (GB10 Grace-Blackwell, aarch64, CUDA 13). It uses native NeMo PyTorch inference, which is what makes this work:

• The base image is NVIDIA's nvcr.io/nvidia/pytorch:25.09-py3, whose CUDA build targets Blackwell sm_121 on aarch64.
• The model (EncDecRNNTBPEModelWithPrompt) loads and transcribes on the GB10 with no source changes.

The one thing that does not work on the Spark is the NVIDIA Riva deployment path (nemo2riva), which has unresolved aarch64 dependency conflicts. We sidestep Riva entirely and run the model directly in NeMo.

Quick start

git clone https://github.com/briancaffey/nemotron-asr-server.git
cd nemotron-asr-server
docker compose up --build

The server listens on http://0.0.0.0:8105 on the host (mapped to 8000 in the container). On first start it downloads the model into your host ~/.cache/huggingface cache (reused on subsequent runs). Wait for Model loaded on cuda in the logs, then open the docs:

http://localhost:8105/docs

Requirements

• NVIDIA GPU + recent driver, and the NVIDIA Container Toolkit
• Docker with Compose v2

Usage

cURL

# Plain text
curl http://localhost:8105/v1/audio/transcriptions \
  -F file=@audio.wav -F response_format=text

# JSON (default): {"text": "..."}
curl http://localhost:8105/v1/audio/transcriptions \
  -F file=@audio.wav

# verbose_json with word + segment timestamps
curl http://localhost:8105/v1/audio/transcriptions \
  -F file=@audio.wav \
  -F response_format=verbose_json \
  -F 'timestamp_granularities[]=word' \
  -F 'timestamp_granularities[]=segment'

# SRT / VTT subtitles
curl http://localhost:8105/v1/audio/transcriptions \
  -F file=@audio.wav -F response_format=srt

See examples/transcribe.sh for more.

OpenAI Python SDK

Point the SDK at this server — no other changes:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8105/v1", api_key="not-needed")

with open("audio.wav", "rb") as f:
    result = client.audio.transcriptions.create(
        model="nvidia/nemotron-3.5-asr-streaming-0.6b",
        file=f,
        response_format="verbose_json",
        timestamp_granularities=["word", "segment"],
    )
print(result.text)
for w in result.words:
    print(w.start, w.end, w.word)

Real-time streaming (WebSocket)

pip install websockets soundfile numpy
python examples/stream_client.py audio.wav

The client streams 16 kHz mono PCM frames and prints partial transcripts as audio arrives, then a final result with timestamps. Protocol details are documented at the top of app/streaming.py.

API

POST /v1/audio/transcriptions

multipart/form-data fields (OpenAI-compatible):

Field	Type	Default	Notes
file	file	—	Required. Any ffmpeg-decodable audio (wav, mp3, m4a, flac, ogg, webm, …).
model	string	server model	Informational; this server serves one model.
language	string	(auto)	Locale like en-US, es-ES, fr-FR. Omit for auto-detection.
response_format	string	json	json | text | verbose_json | srt | vtt.
temperature	float	0.0	Accepted for compatibility; RNNT greedy decoding is deterministic.
timestamp_granularities[]	string[]	[segment]	Any of word, segment (used with verbose_json).
prompt	string	—	Accepted for compatibility; not used by this model.

verbose_json mirrors OpenAI's shape:

{
  "task": "transcribe",
  "language": "en-US",
  "duration": 5.46,
  "text": "Well, I don't wish to see it any more ...",
  "words": [{"word": "Well,", "start": 0.72, "end": 1.04}, ...],
  "segments": [{"id": 0, "start": 0.72, "end": 5.2, "text": "Well, ..."}, ...]
}

Segment objects include OpenAI's extra fields (seek, tokens, avg_logprob, compression_ratio, no_speech_prob, temperature) with neutral defaults, since NeMo's RNNT decoder doesn't expose them.

WS /v1/audio/transcriptions/stream

Send an optional JSON config frame, then binary audio frames, then {"type": "end"}. Receive {"type": "partial", ...} messages and a final {"type": "final", "text", "words", "segments"}. See app/streaming.py.

Other endpoints

• GET /v1/models — OpenAI-style model listing
• GET /healthz — readiness/liveness ({"status","ready","device",...})
• GET /docs — interactive Swagger UI

Configuration

Environment variables (prefix NEMOTRON_ASR_):

Variable	Default	Description
NEMOTRON_ASR_MODEL	nvidia/nemotron-3.5-asr-streaming-0.6b	HF model id or local .nemo path.
NEMOTRON_ASR_DEFAULT_LANGUAGE	auto	Language used when a request omits one.
NEMOTRON_ASR_EAGER_LOAD	true	Load the model at startup vs. on first request.
NEMOTRON_ASR_SAMPLE_RATE	16000	Model input sample rate (audio is resampled to this).
PORT / HOST	8000 / 0.0.0.0	In-container bind address. Host port is set by Compose (8105).

Port mapping is host 8105 → container 8000 (see docker-compose.yml).

How it works

• Audio: uploads are decoded to 16 kHz mono via ffmpeg, so any common format is accepted.
• Language: the model is conditioned on a language-ID prompt. auto runs language identification; an explicit language forces a locale. The model emits inline <xx-XX> tags that the server strips from text and uses to report the detected language.
• Streaming: the WebSocket endpoint accumulates audio and re-decodes incrementally to emit partial hypotheses, with a final pass that adds word/segment timestamps.
• Compatibility shim: on current NeMo main, transcribing bare audio files trips a prompt-index dataset that expects a per-cut language. The server installs a small shim so the request's language (or auto) drives the prompt index directly. See app/model.py.

Running without Docker

# Inside an NVIDIA PyTorch container (or an env with a Blackwell-capable torch):
apt-get update && apt-get install -y libsndfile1 ffmpeg
pip install "nemo_toolkit[asr] @ git+https://github.com/NVIDIA/NeMo.git@main"
pip install -r requirements.txt
uvicorn app.main:app --host 0.0.0.0 --port 8000

License & attribution

• This repository is licensed under the Apache License 2.0 — see LICENSE and NOTICE.
• The model, nvidia/nemotron-3.5-asr-streaming-0.6b, is the property of NVIDIA and distributed under its own license (OpenMDW-1.1), separate from this repo. It is downloaded at runtime and is not redistributed here. Review the model card and comply with its license before use.
• Built on NVIDIA NeMo (Apache-2.0).