Seekable playback: convert committed AAC take to M4A for Review scrubbing

[fel-cesar]

Summary

The Record Tab Review state (#49) requires scrubbing the draft waveform to any position. Recording commits takes as ADTS AAC (.aac), which is not reliably seekable, so #49 waives the scrubbing acceptance criterion and defers it here to be solved and reviewed separately.

This issue covers both halves of the problem: the commit format and the scrubbing it needs to enable.

Background / why the format is the way it is

Recording captures into ADTS AAC (.aac) rather than the preset .m4a on purpose (see src/hooks/useRecorder.ts):

• An MP4/M4A file is only valid once its moov atom is written on a clean stop(). A process kill mid-recording leaves it unplayable and un-appendable.
• ADTS is a self-framing stream: a killed segment stays playable, and multiple segments concatenate by byte-append. This is what makes pause / resume-after-kill possible.

The trade-off: ADTS AAC has no container index / moov atom, so accurate absolute seeking (scrubbing) is unreliable. We do not want to give up kill-safe resumable capture to get seeking, so the two formats must coexist.

Current vs expected

• Current: committed takes are stored as ADTS .aac. Playback works, but scrubbing to an arbitrary position is not reliably accurate.
• Expected: Review can scrub to any position accurately, reading from a seekable file, without regressing resumable capture.

Proposed approach

Build a local (native) module — an Expo local/native module or config-plugin-backed module, since this is CNG / Android-only — that, after a take is committed, converts/remuxes the committed ADTS .aac into an M4A (MP4/AAC) file with a proper moov atom.

• Keep ADTS AAC as the capture/resume format (do not regress kill-safe resumable takes).
• Run the AAC → M4A conversion as part of / immediately after commit (commitRecording in src/hooks/useRecorder.ts), and resolve the committed take's playable URI to the M4A output.
• Feed the seekable M4A into the existing playback/seek path (src/hooks/useAudioPlayback.ts) that backs the Review scrub UI (src/app/tabs/drafting/record/components/RecordingWaveform.tsx).

Scope

In scope

• Local native module to convert/remux a committed AAC take to seekable M4A.
• Wiring the converted M4A into the commit flow and playback URI resolution.
• Accurate scrubbing in the Review state on the converted file.

Out of scope

• Changing the capture/resume format (stays ADTS AAC).
• Waveform amplitude generation / visual design beyond enabling accurate seek.

Acceptance criteria

• A committed take is converted to a seekable M4A (MP4/AAC) file after commit.
• Review-state scrubbing seeks accurately to arbitrary positions on the converted file (satisfies the Build Record Tab with Draft Recording and Playback #49-waived scrubbing AC).
• Capture, pause, and resume-after-kill continue to work unchanged (still ADTS AAC).
• Conversion failure is handled gracefully (logged; take remains playable even if non-seekable) — no data loss.
• Android device QA recorded on the PR (native module + audio path).

Verification / test plan

• Unit-test the commit → convert → resolve-URI path (mock the native module).
• Android device QA: record a take, commit, then scrub the Review waveform to multiple positions and confirm playback resumes accurately from each.

File pointers

• src/hooks/useRecorder.ts — AAC/ADTS capture decision (comment near the useAudioRecorder config) and commitRecording.
• src/hooks/useAudioPlayback.ts — seek() / playback used by Review.
• src/app/tabs/drafting/record/components/RecordingWaveform.tsx — Review scrub UI (onSeek, durationMs).
• app.config.ts / plugins/ — where a local native module / config plugin would be wired (CNG, Android-only).

Links

Refs #49
Related: #96 (playback engine + waveform decision)

[fel-cesar]

Spike #176 — findings: m4a vs ADTS capture (bounded, ~half day)

Ran the bounded kill-test spike on a physical Android device (dev client). TL;DR: there is no single crash-safe and natively-seekable capture format available through an expo-audio preset, which confirms ADTS capture is the right call. Two real options remain for producing a seekable take (below) — flagging before any more native code per the ask.

1. Default HIGH_QUALITY .m4a (mpeg4) preset — kill mid-take

Result: lossy on crash. After a mid-take force-kill, only a prefix of the take is playable; audio recorded near the moment of the kill is lost. MediaRecorder streams samples into mdat while recording but only finalizes the moov (sample index) on a clean stop(), so a killed take has a missing/stale index and can't recover the tail. (Exact symptom can vary by device/player — open-with-prefix vs fail-to-open — but it's lossy either way.)

Video:

m4a-recording-edited.mp4

2. Our ADTS config (aac_adts / .aac) — kill mid-take

Result: fully playable + resumable. The partial take plays end-to-end after a kill, because every AAC frame is self-framing (no central index to go stale). This is what makes byte-append concatenation and resume-after-kill work.

Video:

good-record-edited.mp4

3. Output formats exposed by expo-audio

AndroidOutputFormat is fixed to: default | 3gp | mpeg4 | amrnb | amrwb | aac_adts | mpeg2ts | webm. No fragmented-MP4 (fMP4) option. Of these, only aac_adts and mpeg2ts are self-framing / crash-tolerant; none are both crash-safe and natively seekable via a preset. (Confirmed by inspecting the shipped types — no device needed.)

4. Fragmented MP4

As predicted, fMP4 (moof/mdat fragments) is not reachable through a preset. Getting it would require a custom AudioRecord → MediaCodec → fragmented muxer pipeline — a much larger native project than the remux module. Ruled out for this feature.

Conclusion

No single-format path through expo-audio gives us crash-safety + native seek. ADTS capture stands. That leaves the two valid options called out in the ticket for turning captured ADTS into a seekable take:

• B — ADTS capture + remux on commit: losslessly repackage the merged ADTS into a seekable .m4a via Android MediaMuxer (AacRemux). Not brute force — it's the platform API JS can't reach.
• A — segment manifest + global seek in playback (JS only): keep the ordered ADTS segments, play them as one logical stream, map global position → (segment, offset). No Kotlin.

The A-vs-B decision hinges on intra-ADTS seek accuracy and cross-segment boundary smoothness, which I have not measured yet.

Scope flag before more native code

Holding here per "post findings before building any more native code" — how do you want to proceed?

• (b) go straight to Option B (keep the remux) and delete the Option A prototype, or
• (c) merge-at-sync/server and play a single file.

[fel-cesar]

Append: fMP4 findings — why we're not taking that path

fMP4 (fragmented MP4) is the one container that is both crash-safe and seekable:
an init segment (ftyp + skeletal moov) is written up front, then each fragment
(moof index + mdat audio) is self-describing — so flushed fragments stay playable
after a kill, and each carries a sample table for precise seek. It would collapse both
concerns the ADTS+remux path splits (crash-safe capture vs. seekable playback) into a
single file, retiring AacRemux.

It is reachable natively: androidx.media3.muxer.FragmentedMp4Muxer
(setFragmentDurationMs, default 2000ms) produces fMP4. But not through expo-audio
— AndroidOutputFormat exposes no fMP4, so this means replacing the recorder with a
custom AudioRecord → MediaCodec → FragmentedMp4Muxer pipeline.

Why we're not doing it now:

• Much larger than the remux module. We'd own raw mic capture, AAC encode, the
  muxer, a foreground-service (microphone FGS type) lifecycle, and metering —
  everything expo-audio currently handles for us.
• Crash-recovery is not free. A kill can leave a torn trailing fragment, so we'd
  need a relaunch repair/truncate step. More importantly, media3 writes samples "in
  batches," so the real loss window depends on disk-flush cadence — unverified, and
  the single biggest risk to prove before trusting the crash-safety claim.
• Doesn't remove multi-segment. FragmentedMp4Muxer can't be reopened to append, so
  resume-after-kill still yields N files — same as ADTS today — and fMP4 files don't
  byte-concatenate (each has its own init segment), so merging needs a real remux pass.

Decision: stay on crash-safe ADTS capture; the A-vs-B choice (JS manifest vs.
ADTS→M4A remux) is unchanged. fMP4 is the theoretically ideal format but a disproportionate
native subsystem for this feature. Revisit only if a future ticket needs a single
crash-safe + seekable artifact and is scoped for a dedicated native recorder — first step
would be a throwaway spike that records fragments, hard-kills the process, and measures
the actual tail loss to validate the flush window.

[fel-cesar]

fluent-mobile-176-spike.zip

Device test update — Option C (m4a segments + manifest + global seek)

Reran the final device test with Record .m4a + Keep manifest + Segment playback forced on, killing the app two ways: Recents swipe and adb shell am force-stop com.eten.fluent (system-style kill).

📹 Screen recording attached below.

Screen_Recording_20260713_183434_Fluent.mp4

What worked

• Gap-free sequential playback across multiple independently-finalized .m4a segments — little to no perceptible gap at boundaries.
• Accurate global seek across the whole take: maps to the right

segment and the correct offset within it (relative to total duration), auto-switching .m4a files while scrubbing.

• No native remux needed — each .m4a is self-seekable (own moov/sample table), so manifest + useSegmentedAudioPlayback plays N files as one logical, seekable stream.
• Contrary to the step-1 kill test and the documented "no moov on abrupt kill," most hard kills (swipe and am force-stop) left the in-flight segment playable — likely because expo-audio releases/finalizes the recorder on teardown and the muxer runs in a separate media-server process.

Downsides

• .m4a segments are not reliably mergeable into one file in JS (MP4 is a box structure; would need ffmpeg/native).
• In-flight crash-safety is best-effort, not guaranteed — some runs still produced a corrupt (moov-less) segment. It's timing/OEM-dependent, so it can't be leaned on for data safety. The manifest writer drops any moov-less segment (probe → 0ms), so the failure mode degrades to "lose the last chunk," never "whole take unplayable." ADTS remains the only guaranteed crash-safe capture.

I also tried to reproduce the "corrupt state" locally more aggressively (using the CLI am force-stop to match how Android actually kills the app) and it happens less often than the docs/research imply — though frequency is still unknown.

Recommendation — depends on how reliable we need crash recovery to be

• If occasional unrecoverable in-flight chunks are acceptable → Option C, with a recovery prompt. When the in-flight segment can't be recovered, tell the user that chunk couldn't be saved and prompt them to re-record it. Given how hard the corruption has been to reproduce in practice, this is my lean — but it's a product/reliability call, not purely technical.
• If we need near-guaranteed recovery → don't ship C as-is. Either stay on Option B (ADTS + MediaMuxer remux, or merge at sync/server) for guaranteed crash-safe capture + accurate seek, or invest in a merge/repair step before relying on m4a segments.

A/B/C stays deferred pending the tech-lead call per this issue — no native code deleted or graduated. The Option A/C prototype (useSegmentedAudioPlayback, manifest writer/resolver, runtime debug flags, MP4/ADTS duration probes) is all on fel-cesar/spike/explore-m4a for side-by-side comparison.