Voice pipeline bring-up tested on ESP32-LyraT v4.3 (I2S2 @ 24 kHz → Jetson Orin Nano APE)

[ai-hpc]

Recording how the voice pipeline was brought up and tested on an ESP32-LyraT v4.3 audio dev board as the front-end for the Jetson Orin Nano Super 8 GB, so it's captured as genie-voice-runtime takes over the audio/capture/playback slice from genie-claw.

Hardware setup

• Board: ESP32-LyraT v4.3 (ES8388 codec, onboard mics + speaker) used as the mic/speaker front-end.
• Wiring: LyraT codec I2S → Jetson I2S2, captured and played through the Jetson APE (Audio Processing Engine / AHUB).
• ALSA view on the Jetson (verified): capture is card 1: APE [NVIDIA Jetson Orin Nano APE] via the XBAR → ADMAIF path (tegra-dlink-*); card 0: HDA is HDMI only.
• Sample rate: 24 kHz, chosen to match the AHUB I2S2 rate — audio_sample_rate = 24000 ("Match the AHUB's I2S2 Sample Rate (24 kHz on LyraT)").

Pipeline path (as deployed today in genie-claw)

LyraT mic → I2S2 → APE/ADMAIF → ALSA capture (24 kHz)
  → deepfilternet (capture-side denoise)
  → whisper.cpp STT (ggml-small, CUDA, whisper-server :8178; tiny fallback under memory pressure)
  → agent (genie-claw)
  → piper TTS (en_US-amy-medium)
  → ALSA playback → I2S2 → LyraT speaker

Mode / settings used

• Push-to-talk (wakeword_script = ""); voice_continuous = true for follow-ups without re-wake.
• Half-duplex gate: post_tts_silence_ms = 1500 — the LyraT mic and speaker share the board, so we gate the mic after aplay to avoid capturing TTS bleed.
• audio_device = "auto" / audio_output_device = "auto"; audio_denoiser = "deepfilternet"; STT swap to whisper-tiny below swap_stt_mb = 200.

Evidence

• A 5 s capture from the board is retained (lyrat.wav: 48 kHz / 2-ch / 16-bit PCM raw, downmixed + resampled to 24 kHz mono for the pipeline).

Open items to validate / carry into genie-voice-runtime

• Robust capture-device selection for the APE/ADMAIF card (vs auto/hardcoded plughw).
• Capture-rate handling: board/I2S2 at 24 kHz, whisper wants 16 kHz, lyrat.wav is 48 kHz/2-ch — pin the resample/downmix path and confirm no quality loss.
• End-to-end latency budget on the LyraT path (capture → STT → agent → TTS → playback).
• STT accuracy (WER) on real LyraT captures vs clean reference.
• Wake-word path (currently push-to-talk only).
• AEC / barge-in to replace the half-duplex silence gate.

Some details above are derived from the deployed geniepod.toml + the Jetson's ALSA/cards rather than a written test log — please correct any board-wiring specifics.