output.pipe shuts Liquidsoap down if the device runs out of free space

[MikaSappi]

Description

Liquidsoap restarts itself (or shuts down and is restarted by systemd) when output.pipe runs out of storage. First I thought the issue wasn't because of no available space on the storage media, because the archive disk still had good headroom. After thorough investigation, I see that the archive cleaner hasn't ran as it's supposed to (this is on me 😁, it should've been a recurrent run). Now it seems that it has ran the archive cleaning function first when the script's initialized and a second time an hour after the start:

grep -iE '2025/03/20.*archive_cl'
2025/03/20 06:58:20 [archive_cleaner:3] 
2025/03/20 06:58:20 [archive_cleaner:3] 
2025/03/20 07:58:20 [archive_cleaner:3] 
2025/03/20 07:58:20 [archive_cleaner:3] 

I'm thinking I might move the archive cleaner outside of Liquidsoap because it really serves no purpose running inside Liquidsoap.

Still, the issue persists: if the storage's full, Liquidsoap stops.

Steps to reproduce

...

# Archiving function, local, /radio/archive/YYYY-MM-DD-UUID
output.file(
shine,
  {time.string("/radio/archive/%Y-%m-%d/%Y-%m-%d-%H_%M_%S.mp3")},
  radio,
  reopen_when={0m}
)

# Function to save the output file every hour and remove files older than 35d
thread.run(
  every=3600.,
  {
    list.iter(
      fun (msg) -> log(msg, label="archive_cleaner"),
      list.append(
        process.read.lines(
        log_errors=false,
          "find /radio/archive/* -type f -mtime +35 -delete"
        ),
        process.read.lines(
        log_errors=false,
          "find /radio/archive/* -type d -empty -delete"
        )
      )
    )
  }
)

Expected behavior

It shouldn't stop the service. I think the best end result would just be either outputting the output.pipe to output.dummy if storage is full or something like that.

Log extracts

The script wasn't in a debugging phase, so I didn't have the log level at 4. This is what I've got:

LIQ log:

2025/03/20 06:56:47 [clock:2] Source output_file failed while streaming: Lang.Runtime_error { kind: "system", msg: "Sys_error(\"No space left on device\")", pos: [at stdlib.ml, line 379, char 33-33, at src/core/outputs/pipe_output.ml, line 415, char 36-36] }!
2025/03/20 06:56:47 [clock:2] Raised by primitive operation at Stdlib.output_substring in file "stdlib.ml", line 379, characters 7-40
2025/03/20 06:56:47 [clock:2] Called from 2025/03/20 06:58:20 >>> LOG START

journalctl:

Mar 20 06:56:49 AP-ubuntu-4gb-hel1-2 liquidsoap[138748]: Fatal error: exception Sys_error("No space left on device")
Mar 20 06:56:49 AP-ubuntu-4gb-hel1-2 liquidsoap[138748]: Raised by primitive operation at Stdlib.close_out in file "stdlib.ml", line 392, characters 19-27
Mar 20 06:56:49 AP-ubuntu-4gb-hel1-2 liquidsoap[138748]: Called from Liquidsoap_runtime__Main.daemonize in file "src/runtime/main.ml", line 643, characters 4-15
Mar 20 06:56:49 AP-ubuntu-4gb-hel1-2 liquidsoap[138748]: Called from Liquidsoap_runtime__Main.(fun) in file "src/runtime/main.ml", line 667, characters 42-54
Mar 20 06:56:49 AP-ubuntu-4gb-hel1-2 liquidsoap[138748]: Called from Stdlib__List.iter in file "list.ml", line 110, characters 12-15
Mar 20 06:56:49 AP-ubuntu-4gb-hel1-2 liquidsoap[138748]: Called from Lifecycle.make_action.action in file "src/core/tools/lifecycle.ml", line 63, characters 4-20
Mar 20 06:56:49 AP-ubuntu-4gb-hel1-2 liquidsoap[138748]: Called from Stdlib__List.iter in file "list.ml", line 110, characters 12-15
Mar 20 06:56:49 AP-ubuntu-4gb-hel1-2 liquidsoap[138748]: Called from Stdlib__List.iter in file "list.ml", line 110, characters 12-15
Mar 20 06:56:49 AP-ubuntu-4gb-hel1-2 liquidsoap[138748]: Called from Liquidsoap_runtime__Runner.run in file "src/runtime/runner.default.ml" (inlined), line 25, characters 13-26
Mar 20 06:56:49 AP-ubuntu-4gb-hel1-2 liquidsoap[138748]: Called from Dune__exe__Liquidsoap in file "src/bin/liquidsoap.ml", line 1, characters 9-41

Script extracts

No response

Liquidsoap version

Liquidsoap 2.3.1+dev
Copyright (c) 2003-2024 Savonet team
Liquidsoap is open-source software, released under GNU General Public License.
See <http://liquidsoap.info> for more information.

Liquidsoap build config

* Liquidsoap version  : 2.3.1+dev

 * Compilation options
   - Release build       : false
   - Git SHA             : (none)
   - OCaml version       : 4.14.0
   - OS type             : Unix
   - Libs versions       : angstrom=0.16.1 asetmap=0.8.1 astring=0.8.5 backoff=0.1.1 base=v0.16.3 base.base_internalhash_types=v0.16.3 base.caml=v0.16.3 base.shadow_stdlib=v0.16.3 base64=3.5.1 bigarray=[distributed with Ocaml] bigarray-compat=1.1.0 bigstringaf=0.10.0 bytes=[distributed with OCaml 4.02 or above] camlp-streams camomile.lib=2.0 cohttp=v6.0.0 cohttp-lwt=v6.0.0 cohttp-lwt-unix=v6.0.0 conduit=7.1.0 conduit-lwt=7.1.0 conduit-lwt-unix=7.1.0 cry=1.0.3 ctypes=0.23.0 ctypes-foreign=0.23.0 ctypes.stubs=0.23.0 curl=0.9.2 domain-name=0.4.1 dtools=0.4.5 dune-build-info=3.17.2 dune-private-libs.dune-section=3.17.2 dune-site=3.17.2 dune-site.private=3.17.2 duppy=0.9.4 fdkaac=0.3.3 ffmpeg-av=1.2.2 ffmpeg-avcodec=1.2.2 ffmpeg-avdevice=1.2.2 ffmpeg-avfilter=1.2.2 ffmpeg-avutil=1.2.2 ffmpeg-swresample=1.2.2 ffmpeg-swscale=1.2.2 fileutils=v0.6.6 flac=1.0.0 flac.decoder=1.0.0 flac.ogg=1.0.0 fmt=0.9.0 gen=1.1 http=v6.0.0 http.__private__.http_bytebuffer=v6.0.0 integers ipaddr=5.6.0 ipaddr-sexp=5.6.0 ipaddr.unix=5.6.0 lame=0.3.7 liquidsoap-lang=2.3.1 liquidsoap-lang.console=2.3.1 liquidsoap_builtins liquidsoap_core liquidsoap_fdkaac liquidsoap_ffmpeg liquidsoap_flac liquidsoap_lame liquidsoap_mad liquidsoap_ndi liquidsoap_ogg liquidsoap_ogg_flac liquidsoap_optionals liquidsoap_opus liquidsoap_oss liquidsoap_prometheus liquidsoap_runtime liquidsoap_shine liquidsoap_sqlite liquidsoap_srt liquidsoap_stereotool liquidsoap_vorbis logs=0.7.0 logs.fmt=0.7.0 logs.lwt=0.7.0 lwt lwt.unix macaddr=5.6.0 mad=0.5.3 magic-mime=1.3.1 mem_usage=0.1.2 menhirLib=20240715 metadata=0.3.0 mm=0.8.6 mm.audio=0.8.6 mm.base=0.8.6 mm.image=0.8.6 mm.midi=0.8.6 mm.video=0.8.6 multicore-magic=2.3.1 multicore-magic.__private__.multicore_magic_atomic_array_ocaml4=2.3.1 ndi ocplib-endian ocplib-endian.bigstring ogg=1.0.0 ogg.decoder=1.0.0 opus=1.0.0 opus.decoder=1.0.0 posix-base=2.2.0 posix-socket=2.2.0 posix-socket.constants=2.2.0 posix-socket.stubs=2.2.0 posix-socket.types=2.2.0 ppx_compare.runtime-lib=v0.16.0 ppx_hash.runtime-lib=v0.16.0 ppx_sexp_conv.runtime-lib=v0.16.0 prometheus=1.2 prometheus-app=1.2 re=1.12.0 saturn_lockfree=0.5.0 sedlex=3.3 seq=[distributed with OCaml 4.07 or above] sexplib0=v0.16.0 shine=0.2.3 sqlite3=5.3.0 srt=0.3.3 srt.constants=0.3.3 srt.stubs=0.3.3 srt.stubs.locked=0.3.3 srt.types=0.3.3 stdlib-shims=0.3.0 stereotool str=[distributed with Ocaml] stringext=1.6.0 threads=[distributed with Ocaml] threads.posix=[internal] unix=[distributed with Ocaml] uri=4.4.0 uri-sexp=4.4.0 uri.services=4.4.0 vorbis=1.0.0 vorbis.decoder=1.0.0 xml-light=2.5
   - architecture        : arm64
   - host                : aarch64-unknown-linux-gnu
   - target              : aarch64-unknown-linux-gnu
   - system              : linux
   - ocamlopt_cflags     : -O2 -fno-strict-aliasing -fwrapv -pthread -fPIC
   - native_c_compiler   : gcc -O2 -fno-strict-aliasing -fwrapv -pthread -fPIC -D_FILE_OFFSET_BITS=64
   - native_c_libraries  : -lm

 * Configured paths
   - mode              : default
   - standard library  : (set by dune-site)
   - scripted binaries : (set by dune-site)
   - rundir            : (set by dune-site)
   - logdir            : (set by dune-site)
   - user cache        : $HOME/.cache/liquidsoap (override with $LIQ_CACHE_USER_DIR)
   - system cache      : (set by dune-site) (override with $LIQ_CACHE_SYSTEM_DIR)
   - camomile files    : (set by dune-site)

 * Supported input formats
   - MP3               : yes
   - AAC               : no (requires faad)
   - Ffmpeg            : yes
   - Flac (native)     : yes
   - Flac (ogg)        : yes
   - Opus              : yes
   - Speex             : no (requires speex)
   - Theora            : no (requires theora)
   - Vorbis            : yes
   - WAV/AIFF          : yes (native)

 * Supported output formats
   - FDK-AAC           : yes
   - FFmpeg            : yes
   - MP3               : yes
   - MP3 (fixed-point) : yes
   - Flac (native)     : yes
   - Flac (ogg)        : yes
   - Opus              : yes
   - Speex             : no (requires speex)
   - Theora            : no (requires theora)
   - Vorbis            : yes
   - WAV/AIFF          : yes (native)

 * Tags
   - AAC               : no (requires faad)
   - FFmpeg            : yes
   - FLAC (native)     : yes
   - Flac (ogg)        : yes
   - Native decoder    : yes
   - Vorbis            : yes

 * Input / output
   - ALSA              : no (requires alsa)
   - AO                : no (requires ao)
   - FFmpeg            : yes
   - JACK              : no (requires bjack)
   - NDI               : yes
   - OSS               : yes
   - Portaudio         : no (requires portaudio)
   - Pulseaudio        : no (requires pulseaudio)
   - SRT               : yes

 * Audio manipulation
   - FFmpeg            : yes
   - LADSPA            : no (requires ladspa)
   - Lilv              : no (requires lilv)
   - Samplerate        : no (requires samplerate)
   - SoundTouch        : no (requires soundtouch)
   - StereoTool        : yes

 * Video manipulation
   - camlimages        : no (requires camlimages)
   - FFmpeg            : yes
   - frei0r            : no (requires frei0r)
   - ImageLib          : no (requires imagelib)
   - SDL               : no (requires tsdl-image & tsdl-ttf)

 * MIDI manipulation
   - DSSI              : no (requires dssi)

 * Visualization
   - GD                : no (requires gd)
   - Graphics          : no (requires graphics)
   - SDL               : no (requires tsdl-image & tsdl-ttf)

 * Additional libraries
   - FFmpeg filters    : yes
   - FFmpeg devices    : yes
   - inotify           : no (requires inotify)
   - irc               : no (requires irc-client-unix)
   - jemalloc          : no (requires jemalloc)
   - lo                : no (requires lo)
   - memtrace          : no (requires memtrace)
   - osc               : no (requires osc-unix)
   - ssl               : no (requires ssl)
   - sqlite3           : yes
   - tls               : no (requires tls-liquidsoap)
   - posix-time2       : no (requires posix)
   - windows service   : no (requires winsvc)
   - YAML support      : no (requires yaml)
   - XML playlists     : no (requires xmlplaylist)

 * Monitoring
   - Prometheus        : yes

Installation method

From OPAM

Additional Info

No response

[toots]

Thanks for the report. I believe that the underlying issue is that at the moment the whole process will shut down if you have an exception in a thread.

In general we want to aim toward making the stream run as much as possible. However, I am not sure how to handle and properly surface errors thrown in asynchronous tasks. If we start just logging them, the use might also miss very important things that they need to act upon.

I'm leaving this open to this about it.

[toots]

I'm gonna move this to a discussion. Re-reading it, I'm not sure what exactly we could do. A full storage is a major application error, I believe most application would shutdown under these circumstances for a multitude of reasons like not being able to write a log, etc.