[AGENT] Design long-running recording architecture

[BASIC-BIT]

[AGENT]

Summary

Design and implement an architecture for effectively long-running Chronote recordings across desktop and Discord without unbounded memory, disk, network, or AI-call backlog.

Product direction

The desired path is staged:

1. Near term: compressed upload after stop for desktop recordings.
2. Next: rolling upload and rolling transcription for desktop recordings.
3. Long term: Discord meetings and desktop recordings should support effectively indefinite sessions by rolling artifacts, transcripts, summaries, and processing windows.

Current constraints

• Desktop recorder stores WAV sources locally until Stop.
• Desktop upload submits completed source files after capture stops.
• Desktop upload currently shares the 500 MB total upload cap.
• Desktop upload reads completed files into memory before POSTing.
• Discord meeting transcription can backlog behind global transcription concurrency and rate limits.
• Meeting end waits on pending transcription before notes generation.

Proposed scope

• Define bounded local storage rules for active recordings.
• Compress desktop audio during or immediately after capture.
• Add resumable/chunked upload semantics for desktop sources.
• Add rolling server-side processing units with durable status.
• Track queue depth, age, and per-meeting backlog.
• Decide when to generate rolling summaries versus final summaries.
• Define recovery behavior after desktop crash, network loss, backend restart, or worker restart.
• Revisit transcription concurrency/rate limits with production-safe configuration rather than hardcoded low limits.

Acceptance criteria

1. A design exists for bounded long-running capture across desktop and Discord.
2. Desktop no longer requires a full WAV file to remain below a single upload cap before any upload can start.
3. Uploads can resume or recover without duplicating completed meetings.
4. Transcription work can be processed incrementally without unbounded backlog.
5. Meeting end latency is bounded or at least observable and recoverable.
6. Metrics expose queue depth, queue age, and per-meeting processing lag.
7. Tests cover chunk state transitions, retry/recovery, and final meeting assembly.

[BASIC-BIT]

[AGENT]

Additional findings from local desktop meeting failure

A real local desktop recording hit Media file is too large. before the upload intent was created. The failed path exposed several concrete loss and long-meeting gaps that should be folded into this parent architecture issue.

Current implementation details

• Desktop records local mic and system output as separate WAV files under the Tauri app data recording directory.
• Desktop requests /api/desktop/recordings/intent only after Stop, using final source fileSize values.
• The API rejects a desktop recording when any source or the combined source total exceeds PERSONAL_MEDIA_UPLOAD_MAX_BYTES.
• The desktop app removes the local recording directory after the upload attempt, even if upload intent or upload fails.
• The desktop upload path reads each completed source file fully into memory before submitting the signed POST.
• Personal media processing is batch-oriented: download full source, normalize, split for transcription if needed, then generate final notes.
• Discord meetings already stream audio to a local MP3 during capture, but artifact upload errors can be swallowed and local files are still cleaned up after end processing.
• Discord meetings still have a hard 2-hour timeout and no durable rolling meeting state.

Architecture direction

Use a layered approach instead of one large rewrite:

1. Local loss prevention: never delete the only local artifact until durable storage or an explicit user delete succeeds.
2. Recoverable local desktop store: write self-contained sealed segments plus a manifest so a killed app can recover everything except possibly the active segment.
3. Rolling cloud upload: upload sealed segments to S3 as they close, instead of waiting for one full file under a single size cap.
4. Rolling processing: represent segments as durable processing units and transcribe/process them incrementally.
5. Live local meeting processing: expose in-progress transcript/status once segment processing exists.
6. Discord infinite meetings: reuse the segment/manifest model for bot meetings, with durable active meeting state and bounded finalization.

Design note

S3 is not an appendable filesystem. A “stream to S3 file” stopgap should probably be modeled as either multipart upload with careful recovery semantics or, more simply, a sequence of sealed segment objects plus a manifest. Segment objects fit better with crash recovery, retry, partial processing, and future live transcript/status.

Linked issues

• [AGENT] Stop deleting local recording artifacts before durable persistence #256
• [AGENT] Add recoverable desktop recording store with segment manifests #260
• [AGENT] Add rolling S3 segment upload for desktop recordings #257
• [AGENT] Process personal recording segments incrementally #259
• [AGENT] Make Discord meetings rolling-artifact and restart-recoverable #258