Live API v2

[dlarocque]

Adds startAudioConversation, a new helper function for creating real-time voice conversations with the model.

The existing LiveSession API is low-level and requires a lot of boilerplate to manage microphone audio and playback. This helper handles all that complexity, making it much easier for developers to add voice features to their apps.

• Handles microphone permissions and captures audio.
• Processes raw audio into the format required by the server.
• Plays back the model's audio response smoothly, without gaps or clicks.
• Supports user interruptions (talking over the model).
• Integrates function calling via a simple handler.
• Includes unit tests with mocks for the Web Audio API.

Sample

const liveSession = await model.connect();
let conversationController;

// This function must be called from within a click handler.
async function startConversation() {
  try {
    conversationController = await startAudioConversation(liveSession);
  } catch (e) {
    // Handle AI-specific errors
    if (e instanceof AIError) {
      console.error("AI Error:", e.message);
    }
    // Handle microphone permission and hardware errors
    else if (e instanceof DOMException) {
      console.error("Microphone Error:", e.message);
    }
    // Handle other unexpected errors
    else {
      console.error("An unexpected error occurred:", e);
    }
  }
}

// Later, to stop the conversation:
// if (conversationController) {
//   await conversationController.stop();
// }

[changeset-bot]

⚠️ No Changeset found

Latest commit: 774fecc

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[google-oss-bot]

Size Report ¹

Affected Products

• @firebase/ai

  Type	Base (3bb3f36)	Merge (a95ef8c)	Diff
  browser	47.2 kB	54.5 kB	+7.26 kB (+15.4%)
  main	50.6 kB	58.0 kB	+7.37 kB (+14.6%)
  module	47.2 kB	54.5 kB	+7.26 kB (+15.4%)

• firebase

  Type	Base (3bb3f36)	Merge (a95ef8c)	Diff
  firebase-ai.js	37.3 kB	43.3 kB	+6.06 kB (+16.3%)

Test Logs

• Base (3bb3f36): https://github.com/firebase/firebase-js-sdk/actions/runs/17079519885
• Merge (a95ef8c): https://github.com/firebase/firebase-js-sdk/actions/runs/17136697879

1. https://storage.googleapis.com/firebase-sdk-metric-reports/imVimFeD9F.html

[google-oss-bot]

Size Analysis Report ¹

Affected Products

• @firebase/ai

  • LiveGenerativeModel

    Size

    Type	Base (3bb3f36)	Merge (a95ef8c)	Diff
    size	8.39 kB	8.41 kB	+26 B (+0.3%)
    size-with-ext-deps	26.0 kB	26.0 kB	+26 B (+0.1%)

  • LiveSession

    Size

    Type	Base (3bb3f36)	Merge (a95ef8c)	Diff
    size	4.92 kB	4.95 kB	+26 B (+0.5%)
    size-with-ext-deps	22.5 kB	22.5 kB	+26 B (+0.1%)

  • ResponseModality

    Size

    Type	Base (3bb3f36)	Merge (a95ef8c)	Diff
    size	2.47 kB	2.49 kB	+14 B (+0.6%)
    size-with-ext-deps	20.0 kB	20.0 kB	+14 B (+0.1%)

  • getLiveGenerativeModel

    Size

    Type	Base (3bb3f36)	Merge (a95ef8c)	Diff
    size	10.9 kB	10.9 kB	+26 B (+0.2%)
    size-with-ext-deps	28.5 kB	28.5 kB	+26 B (+0.1%)

  • startAudioConversation

    Size

    Type	Base (3bb3f36)	Merge (a95ef8c)	Diff
    size	?	8.52 kB	? (?)
    size-with-ext-deps	?	26.1 kB	? (?)

    Dependency

    Type	Base (3bb3f36)	Merge (a95ef8c)	Diff
    functions	?	decodeInstanceIdentifier registerAI startAudioConversation	?
    classes	?	AIError AIService AudioConversationRunner Backend GoogleAIBackend VertexAIBackend	?
    variables	?	12 dependencies AIErrorCode AI_TYPE AUDIO_PROCESSOR_NAME BackendType DEFAULT_LOCATION SERVER_INPUT_SAMPLE_RATE SERVER_OUTPUT_SAMPLE_RATE Task audioProcessorWorkletString logger name version	?
    enums	?		?

    External Dependency

    Module	Base (3bb3f36)	Merge (a95ef8c)	Diff
    @firebase/app	?	_registerComponent registerVersion	?
    @firebase/component	?	Component	?
    @firebase/logger	?	Logger	?
    @firebase/util	?	FirebaseError	?

Test Logs

• Base (3bb3f36): https://github.com/firebase/firebase-js-sdk/actions/runs/17079519885
• Merge (a95ef8c): https://github.com/firebase/firebase-js-sdk/actions/runs/17136697879

1. https://storage.googleapis.com/firebase-sdk-metric-reports/WSfEIGEANz.html

[hsubox76]

Is there supposed to be something here?

[hsubox76]

Do you want to use a Deferred? I think this is the deferred pattern. Here's a usage: https://github.com/firebase/firebase-js-sdk/blob/realtime-for-web/packages/analytics/src/index.test.ts#L132 I don't know that you gain a lot except that it reads clearly that you're doing the deferred pattern, and it is a bit bulkier than you need (you don't need wrapCallback or reject). Maybe you can just refactor this so you put both stopPromise and stopResolver into a deferred object like:

  private stopDeferred: { promise: Promise<void>; resolve: () => void } = {
    resolve: () => {},
    promise: new Promise(resolve => {
      this.stopDeferred.resolve = resolve;
    })
  };

And then you just refer to this.stopDeferred.resolve() and this.stopDeferred.promise everywhere you reference them. That way it's clear this is a Deferred and they belong to each other.