Deep-dive: binding the socket in ~1 second — and the shutdown race hiding inside deferred startup

[debpalash]

Part of a series of engineering deep-dives from VoiceStudio's reliability work. This one covers the startup restructure that binds the socket in ~1 second — and the shutdown race hiding inside it.

Why restructure startup at all

VoiceStudio's backend drags in torch, torchaudio, and ~30 API routers at import time. On a cold disk that meant the socket didn't even bind for tens of seconds — and to the desktop shell (and the user), "socket not bound" is indistinguishable from "backend dead". Roughly 1 in 5 of every issue ever filed against the app was some flavor of can't reach the backend / backend died.

The fix (#1550) splits startup:

• Module scope stays fast — FastAPI app object, health endpoint, a startup-progress ledger. Socket binds in ~1s.
• Phase A — everything heavy (env/prefs migration, native preloads, ML imports, router fan-out) runs on an executor thread, reporting each step to the ledger.
• Phase A finalize — route registration happens back on the event loop with no awaits inside, so it's atomic with respect to in-flight requests.
• Phase B — the old lifespan body (DB, background services).

While that runs, a StartupGateMiddleware 503s everything except /health and /startup/progress, and /health answers 503 with the current step name — so the shell can narrate "Loading ML runtime…" instead of showing a dead spinner, and a startup that dies tells you which step killed it.

The race: cancelling a thread that hasn't started

Here's the subtle part. Shutdown must not let interpreter teardown race the import work still running on the executor thread (that's our #1000 bug class — torch imports dying mid-teardown with bizarre errors). So shutdown joins the thread, bounded:

if _phase_a_started.is_set() and not _phase_a_finished.is_set():
    await asyncio.to_thread(_phase_a_finished.wait, 20.0)

Three ways this innocent-looking join can deadlock or get skipped, and what closes each:

1. The queued-but-not-running callable. If shutdown samples _phase_a_started while the executor callable is still queued, the flag is unset, the join is skipped, and teardown races the imports anyway. Fix: set the flag before submission, not inside the callable:

_phase_a_started.set()
await asyncio.shield(loop.run_in_executor(None, _phase_a_build))

2. The cancelled queued callable. Now the opposite hazard: with the flag set before submission, a task cancellation landing while the callable is still queued would (unshielded) cancel the inner future — the callable never runs, _phase_a_finished is never set, and shutdown stalls on the full 20s timeout every time. asyncio.shield means the build always runs to completion and always sets the event; the cancellation still propagates to the startup task itself.

3. The build that exits without reporting. Every exit path of the build — including the idempotent already-built early return — must set the event, or a shutdown that observed started can wait on an event nothing will set:

def _phase_a_build():
    _phase_a_started.set()
    try:
        if not _phase_a_built:
            _phase_a_build_inner()
    finally:
        _phase_a_finished.set()

An AST-level regression test pins all three properties (flag-before-submit, shield present, finally-sets-finished) so a refactor can't quietly reintroduce any of them.

One more detail: dying loudly, but not instantly

If Phase A fails, the backend should die with import-crash semantics (traceback to stderr, hard exit, crash marker attribution). But the whole point of the progress ledger is that the shell's last poll should capture which step failed — so the failure path marks the step failed, prints the traceback, then waits one beat asynchronously before os._exit(1):

await asyncio.sleep(1.0)   # NOT time.sleep — a blocking sleep would freeze
os._exit(1)                # the loop and the ledger poll it exists to serve

(An earlier revision used time.sleep there. A reviewer caught it: a blocking sleep freezes the event loop, making the "one last poll" beat serve nothing.)

Results

• Socket bound in ~1s instead of tens of seconds; integration test budget is 8s for the full bind-and-answer cycle.
• Startup failures name their step ("FATAL: backend startup failed during 'ml_imports'") in the crash marker the shell reads.
• The shell narrates startup step-by-step from /startup/progress.

PR: #1550. The fault-injection harness that exercises the death paths end-to-end: #1551.