[bug]: Backup to CIFS/Samba share stalls due to fsync serialization in backup-fs FUSE daemon

[7thgenerationdesign]

Prerequisites

• I have searched for existing issues that already report this problem.

Server Hardware

FriendlyElec CM3588 (RK3588, ARM64 - 4× Cortex-A55 + 4× Cortex-A76, 32 GB RAM), PXVIRT (ARM64 Proxmox fork) hypervisor, StartOS 0.4.0 VM: 16 GB RAM, 4 vCPUs (A76 cores), VirtIO SCSI disk, kernel 6.1.141 aarch64

StartOS Version

0.4.0-beta.9

Client OS

Linux

Client OS Version

Pop!_OS 24.04

Browser

Firefox

Browser Version

150.0.2

Current Behavior

Backups to a CIFS/Samba network share stall and eventually time out. After an initial burst of ~33 MB/s throughput (matching local disk performance), the backup-fs FUSE daemon becomes blocked on backend flush operations and throughput collapses to near zero. The rsync process hits its hardcoded 300-second I/O timeout and exits with code 30.

Believed root cause: backup-fs calls fsync on every write to the backup filesystem. Because the FUSE daemon is single-threaded, each fsync blocks all concurrent FUSE requests until the backend flush completes. On CIFS, SMB flush round-trip latency (~1–5 ms even to a local NVMe-backed Samba server) is high enough to serialize the entire I/O pipeline. During stalls, backup-fs spends ~90% of its time in uninterruptible sleep (D state) waiting on CIFS flush completions. I/O pressure (/proc/pressure/io) climbs from ~11% to 43–62%.

The same fsync bottleneck also manifests on local ext4-formatted spinning HDDs (blocked on jbd2_log_wait_commit during journal commits), confirming this is not CIFS-specific but rather a function of backend flush latency.

The issue does not manifest when backing up to an NVMe SSD passed through as a VirtIO SCSI block device (direct ext4, sub-millisecond fsync latency). Backups to this target complete fully with stable throughput and zero stalls.

Expected Behavior

Backups to a CIFS/Samba network share should complete successfully. Backend flush latency should not block the FUSE request pipeline.

Steps to Reproduce

1. Set up a Samba share on the LAN (tested with both NVMe SSD and USB HDD backends on the Samba server)
2. In StartOS UI: System → Create Backup → Network Folder, configure the CIFS share
3. Start a backup including at least one large service (e.g., Jellyfin ~99 GB, Nextcloud ~68 GB)
4. Observe initial throughput of ~33 MB/s, followed by throughput collapse within minutes
5. Backup eventually times out (rsync exit code 30) or hangs indefinitely

Anything else?

Diagnostic Evidence:

• dd sequential writes through backup-fs FUSE to the same CIFS mount achieve 35–40 MB/s sustained — raw throughput is not the limitation
• During stalls, backup-fs is blocked in CIFS flush waits (visible via /proc/<pid>/stack and wchan)
• Identical stall pattern on local ext4 HDD (blocked on jbd2_log_wait_commit instead of CIFS flush) confirms the issue is fsync frequency, not the network transport
• NVMe via VirtIO SCSI (sub-ms fsync) completes the same backup with zero stalls, confirming the threshold is backend flush latency

Potential fix:
Reduce fsync frequency in backup-fs. Options include:

• Batch writes and sync at configurable intervals rather than per-write
• Use fdatasync instead of fsync (avoids metadata journal flushes)
• Enable FUSE multithreading or async request handling so metadata/read operations aren't blocked behind data syncs
• Allow O_DSYNC or periodic sync as an alternative to per-write fsync

Workaround:

Use an NVMe SSD passed through as a VirtIO SCSI block device (direct ext4) as the backup target. Fsync latency on NVMe is low enough that the serialization bottleneck does not manifest.

Additional Context:

This may be ARM64/RK3588-specific in practice — it appears that no other users have reported CIFS backup stalls, possibly because x86_64 systems with faster I/O subsystems keep fsync latency below the threshold. However, the underlying per-write fsync design would theoretically affect any backend with non-trivial flush latency.

[gStart9]

Start a backup including at least one large service (e.g., Jellyfin ~99 GB, Nextcloud ~68 GB)

Jellyfin pulls media data form /mnt/filebrowser and/or /mnt/nextcloud and so it shouldn't be backed up twice. I should note, another user complained about this as well (as you saw yesterday, @dr-bonez in #Community - Jellyfin (using cifs share on x86_64 linux using a USB-attached HDD):

Failed: Unknown Error: Error: rsync exited with code 30 [sender] io timeout after 300 seconds -- exiting rsync error: timeout in data send/receive (code 30) at io.c(201) [sender=3.4.1]

And on another try, a different error (SIGUSR1):

Failed: Unknown Error: Error: rsync exited with code 30 [sender] io timeout after 301 seconds -- exiting rsync error: timeout in data send/receive (code 30) at io.c(201) [sender=3.4.1] rsync error: received SIGUSR1 (code 19) at main.c(1600) [generator=3.4.1]

[kwsantiago]

Seeing very slow backups on 0.4.0-beta.9, but with a different signature that may be a separate mode from the CIFS/fsync path here.

Target is a local USB drive (/dev/sda1, ext-formatted), not CIFS/Samba. During the slowdown, mount.backup-fs ran for ~3.5h in R (running) state at ~98.8% CPU with I/O-wait ~0.5% — i.e. CPU-bound, not blocked on fsync (no D-state, no I/O pressure). The three rsync processes sat essentially idle (CPU times 0:06 / 0:00 / 0:11), and the encrypted target grew at only ~70 KB/s (df delta +4.1 MB over 60s).

Key difference from this report: mine did not time out / fail — it eventually completed after ~3.5 hours (rsync never hit the 300s timeout; no exit-30). So this looks less like the CIFS flush-stall and more like a CPU-bound path in backup-fs that just runs extremely slowly to completion. Same practical pain (multi-hour backup, service offline throughout), different mechanism.

Is the CPU-bound case the same root cause, or worth tracking separately? Happy to grab more diagnostics next time.

Env: StartOS 0.4.0-beta.9, x86_64, 16 GB RAM, ~13 packages, backup to local USB drive.