Describe the bug
On current main (verified on both ca7fb9c and latest main including #1577), every synthesis on my macOS deploy logs:
WARNING [omnivoice.watermark] Watermark embedding failed (passing through original): CppCompileError: C++ compile error
The failure is fail-open, so audio ships unmarked — and each attempt burns a failed inductor C++ compile first: ~30-40s on the first take after startup, ~5-8s on later takes (10 consecutive failures observed). So the watermark neither works nor is cheap.
Notable: with #1577's startup prefetch, the AudioSeal generator loads fine at startup (AudioSeal generator loaded (16-bit message mode)); it's the first actual embed that hits the C++ compile error inside the GPU-pool worker.
Environment
Repro
- Start the backend on macOS arm64 with the watermark pref enabled (default).
- Any synthesis (
/v1/audio/speech, /generate, ...).
- Log shows the CppCompileError above; response takes ~30-40s longer than the same synth with the pref off; audio is unmarked.
What I checked
- Standalone works: running
mark_synthetic(waveform, sr, context=...) directly in the same .venv succeeds (5-35s: the inductor compile itself is fine on this machine), both with my normal shell env and with a launchd-like scrubbed env (env -i PATH=/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin HOME=...).
- So the failure is specific to the in-process context: the embed path running inside the GPU-pool worker thread of the live backend.
mark_synthetic catches and logs only the exception class, so the underlying inductor error text is not visible in the log — worth capturing exc_info or the chained traceback to pin it.
- Persisted across restarts, all 10/10 attempts, two different main cuts.
Impact
- EU AI Act Art. 50(2) marking silently not applied (fail-open by design, but 100% failure on this platform means no marking at all).
- 30-40s latency penalty per first take, 5-8s per later take, for nothing.
Suggested angles
- Log the full traceback at least once per process for this failure class (the generic
CppCompileError message hides the real compiler error).
- Check whether the pool worker's thread context breaks torch inductor's C++ compile (e.g. compile happening concurrently with the startup preload holding the pool, or a forked-thread cwd/TMPDIR difference).
Related: #1577 (watermark lifecycle/prefetch — generator load now works; embed does not), #1586-class crash reports may share the latency symptom.
Describe the bug
On current
main(verified on both ca7fb9c and latest main including #1577), every synthesis on my macOS deploy logs:The failure is fail-open, so audio ships unmarked — and each attempt burns a failed inductor C++ compile first: ~30-40s on the first take after startup, ~5-8s on later takes (10 consecutive failures observed). So the watermark neither works nor is cheap.
Notable: with #1577's startup prefetch, the AudioSeal generator loads fine at startup (
AudioSeal generator loaded (16-bit message mode)); it's the first actual embed that hits the C++ compile error inside the GPU-pool worker.Environment
watermark.invisiblepref at default (on)Repro
/v1/audio/speech,/generate, ...).What I checked
mark_synthetic(waveform, sr, context=...)directly in the same.venvsucceeds (5-35s: the inductor compile itself is fine on this machine), both with my normal shell env and with a launchd-like scrubbed env (env -i PATH=/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin HOME=...).mark_syntheticcatches and logs only the exception class, so the underlying inductor error text is not visible in the log — worth capturingexc_infoor the chained traceback to pin it.Impact
Suggested angles
CppCompileErrormessage hides the real compiler error).Related: #1577 (watermark lifecycle/prefetch — generator load now works; embed does not), #1586-class crash reports may share the latency symptom.