Skip to content

TTS provider availability check produces false positives, breaking multi-agent TTS when local servers aren't running #574

@cosarah

Description

@cosarah

Summary

getAvailableProvidersWithVoices in lib/audio/voice-resolver.ts marks keyless local TTS providers (e.g. VoxCPM, Lemonade) as available whenever their defaultBaseUrl is set in the static registry — even when the user has neither configured nor deployed the server. The inflated voice list is then handed to the agent-profile generator, which can assign an unreachable provider to an agent, causing TTS failures at runtime.

Reproduction

  1. Locally deploy only the Lemonade TTS server (e.g. http://localhost:13305/v1). Do not deploy VoxCPM.
  2. Do not configure VoxCPM in settings (no baseUrl/serverBaseUrl).
  3. Generate a classroom via /generation-preview.
  4. Observe agent profiles: some agents get voiceConfig.providerId = 'voxcpm-tts', others 'lemonade-tts'.
  5. During TTS generation, VoxCPM-assigned agents error out (connection refused on 127.0.0.1:8000); Lemonade-assigned agents succeed.

Root cause

lib/audio/voice-resolver.ts:133-142:

```ts
const isKeylessLocalProvider =
!config.requiresApiKey &&
!!(
providerConfig?.serverBaseUrl?.trim() ||
providerConfig?.baseUrl?.trim() ||
config.defaultBaseUrl // ← always truthy for built-in keyless local providers
);
```

VoxCPM (lib/audio/constants.ts:746-761) declares requiresApiKey: false and defaultBaseUrl: 'http://127.0.0.1:8000'. The third disjunct hits unconditionally, so VoxCPM is reported as available regardless of user config or server liveness. Lemonade hits the same disjunct (defaultBaseUrl: 'http://localhost:13305/v1'), but in the repro it happens to be running.

The inflated list is consumed by app/generation-preview/page.tsx:676-704 (getAvailableVoicesForGeneration) and fed to /api/generate/agent-profiles as availableVoices. The LLM is explicitly prompted to "try to use different voices for each agent" (app/api/generate/agent-profiles/route.ts:102-107), so it distributes voices across providers — happily handing out VoxCPM voices that cannot be served.

At TTS time the failure is silent per agent: app/generation-preview/page.tsx:885-933 just increments ttsFailCount. The course is generated with a partially broken agent voice assignment.

Impact

  • Imported and freshly generated classrooms can ship with agents bound to TTS providers that error at playback.
  • Effect is non-deterministic from the user's perspective: "some agents work, some don't".
  • Affects any keyless local provider with a defaultBaseUrl in the registry. Today: VoxCPM, Lemonade. Future keyless local providers will inherit the same bug.

Proposed fix

Drop the config.defaultBaseUrl fallback from the availability check so that a keyless local provider only counts as available when the user has explicitly set serverBaseUrl or baseUrl:

```ts
const isKeylessLocalProvider =
!config.requiresApiKey &&
!!(providerConfig?.serverBaseUrl?.trim() || providerConfig?.baseUrl?.trim());
```

UX follow-up: in TTS settings, offer a one-click "Use default URL" action that writes defaultBaseUrl into the user config, so default-port deployments stay one click away.

Optional hardening (separate change):

  • Health-probe keyless local providers (with short cache) before listing them.
  • Surface per-agent TTS failures during generation instead of silently counting them.

Affected files

  • `lib/audio/voice-resolver.ts` — availability logic
  • `app/generation-preview/page.tsx` — consumer; silent failure counter
  • `app/api/generate/agent-profiles/route.ts` — voice distribution prompt

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions