[bug]: pre-update volume snapshot (cp --reflink) hangs for minutes on heavily-fragmented volumes (e.g. Monero LMDB)

[MattDHill]

Summary

The pre-install/update volume snapshot added in 900d86ab8 ("feat: preserve volumes on failed install + migrate ext4 to btrfs") uses cp -a --reflink=always and assumes the reflink is "instant on btrfs" (per the comment in core/src/volume.rs). That assumption holds for low-fragmentation data but breaks badly for heavily-fragmented volumes — a btrfs reflink is O(number of extents), and a random-write mmap'd database fragments into millions of extents. For a Monero full node (one ~259 GiB LMDB data.mdb), the snapshot cp runs for ~10–12 minutes, during which the install/update appears hung, its completion notification is withheld, and the package cannot be re-installed.

This is not a data duplication / disk-overrun problem — it is genuinely a reflink (CoW), confirmed below (real free space does not drop). The problem is purely the time the reflink takes on a fragmented extent tree.

Affected code

• core/src/volume.rs → snapshot_volumes_for_install() runs cp -a --reflink=always <pkgVolumeDir> <pkgVolumeDir>.install-backup.
• Called from core/src/service/service_map.rs:311, in the install/update finalization, after the s9pk is downloaded/verified/renamed (:298) and before the previous service is stopped (service.uninstall(...) at :351).
• Skipped when no volume dir exists (fresh installs are unaffected) → only updates/reinstalls over an existing volume hit it.
• Introduced in 900d86ab8 (~0.4.0-beta.4). Reproduced on 0.4.0-beta.9.

Symptom

Updating monerod 0.18.4.6:4 → :6 on a synced full node:

• UI/CLI install sits at the final step for ~10–12 minutes.
• The "install failed" notification (after a cancel) only arrived ~10 minutes later — notification issuance and any re-attempt are blocked behind the synchronous cp.
• ps: cp -a --reflink=always …/volumes/monerod …/volumes/monerod.install-backup in D state for the whole window; disk saturated (~100% util, multi-second write latency) doing metadata/extent-tree writes — not file data (free space stayed flat throughout).

Root cause

A btrfs reflink (FICLONERANGE) inserts a reference for every source extent and bumps each extent's backref — cost scales with extent count, not file size. Monero's blockchain is a single ~259 GiB LMDB, a memory-mapped random-write DB, which fragments pathologically on btrfs. filefrag (which only reads the extent map) could not finish enumerating data.mdb's extents in 19+ minutes — the extent count is effectively uncountable in reasonable time, and cp --reflink has to process that same list.

Bitcoin Core does not hit this: its block files are append-only and low-fragmentation (a 128 MiB blk*.dat here had 30 extents).

Evidence (measured on the affected box)

Reflink IS normally instant (control, real low-frag file):

bitcoind blk*.dat: 128 MiB, 30 extents
cp -a --reflink=always: real 0m0.001s
btrfs free-space delta: -4096 bytes (one metadata block ⇒ true reflink, no data copy)

Reflink cost scales with fragmentation, not size (controlled microbenchmark, identical 3 GiB files on the same btrfs, monerod quiesced, bare cp with no trailing sync):

file               size     extents     cp -a --reflink=always
seq.bin            3 GiB         22      0.001 s
fragmented file    3 GiB    419,047      2.18 / 2.27 / 2.26 s   (~2.24 s)

Identical size; the 419k-extent file reflinks ~2,240× slower than the 22-extent file ⇒ cost is O(extent count), ≈5.3 µs/extent. (An earlier run showed seq≈frag≈13 s — that was an artifact of a trailing global sync flushing the live monerod's writes; removing it exposes the real per-operation cost.)

Real-world anchor: the production cp of monerod's 259 GiB data.mdb ran ~10–12 min (≈660 s); at ≈5.3 µs/extent that implies ~10⁸ extents, consistent with filefrag being unable to enumerate it in 19+ minutes.

btrfs subvolume snapshot barely moves with fragmentation (proposed fix):

btrfs subvolume snapshot   empty subvol             0.082 s
btrfs subvolume snapshot   subvol w/ 419,047-extent file   0.489 s
cp -a --reflink=always     same 419,047-extent file        2.24 s

A subvolume snapshot shares the subvolume's b-tree root instead of cloning each extent, so it does not traverse the extent tree (the 0.082→0.489 s difference is the commit of freshly-written metadata, not extent traversal). It's already 4.6× faster at 419k extents and the gap widens with fragmentation — at monerod's ~10⁸-extent scale: snapshot ≈ sub-second vs cp ≈ minutes.

No data duplication (CoW confirmed): btrfs filesystem usage "Free (estimated)" stayed flat (~198 GiB) across the entire multi-minute snapshot; the du of the backup "grows" only because reflinked extents are counted under both paths.

Reproduction conditions

1. btrfs package-data (the new default after the ext4→btrfs migration).
2. Update or reinstall over an existing, large, heavily-fragmented volume (random-write DB: LMDB / SQLite-WAL / etc.). Fresh installs are unaffected.
3. Worsened by a near-full filesystem — this box was at 96.6% data usage (btrfs metadata allocation slows as it fills).

Monero is the clearest trigger; any random-write-DB-heavy package is susceptible. (Variance between nodes is expected: the cost is a function of this volume's extent count, which depends on sync/prune history and fs fullness — so "it worked elsewhere" doesn't contradict this.)

Secondary issues observed

• Snapshot runs while the old service is still running (:311 before the stop at :351). So cp clones a live, actively-written LMDB → the snapshot may be inconsistent, it contends with the running service's I/O, and the file keeps fragmenting during the clone.
• It blocks the install future — the failure/success notification and any re-install are gated behind the synchronous cp. The box looks wedged ("request never leaves the server") for the duration.
• Retained backup of a live volume slowly leaks real space: as the live copy diverges post-snapshot (CoW un-shares), space is consumed. Observed free space drift 198 → 190 GiB while a stalled backup sat retained and monerod kept writing.
• Orphaned .install-backup on external cancel/kill: the Rust cleanup only runs on a normal error/success return, so a SIGKILL'd cp (e.g. user cancel) leaves a 200+ GiB stale backup dir. It's reclaimed on the next attempt (volume.rs:63 deletes a stale backup first), but lingers meanwhile.

Recommended fixes (in priority order)

1. Use btrfs subvolume snapshot instead of cp --reflink. It shares the subvolume's b-tree root rather than cloning per-extent, so it doesn't traverse the extent tree (measured 4.6× faster at 419k extents, gap widening with fragmentation). Requires package volume dirs to be btrfs subvolumes (created as subvolumes at volume-creation; existing dirs need a one-time conversion). This is the proper fix and matches the btrfs-migration direction.
2. If volumes can't be subvolumes near-term: bound the reflink — skip it above a size/extent threshold (proceed without a rollback point, exactly as the ext4 path already does), and/or run it asynchronously with cancellation + progress so it never blocks the notification or re-attempt.
3. Stop the service before snapshotting (move the snapshot after the existing-service stop) for a consistent, uncontended snapshot.
4. Clean up the orphaned .install-backup on cancellation, and account for the diverging-space cost of a retained backup over a live volume.

Workaround (for users hitting this now)

The snapshot is slow but finite — let the update run without cancelling and it completes in ~10–12 min (confirmed: monerod reached 0.18.4.6:6 once allowed to finish). Cancelling mid-cp is what strands the package in updating and leaves an orphaned .install-backup.

Environment

• StartOS / start-cli 0.4.0-beta.9
• package-data: btrfs on LVM/LUKS (/dev/mapper/STARTOS_…_package-data), ssd,discard=async,space_cache=v2, 1.8 TiB, 96.6% used, plain dirs on subvolid 5 (volumes are not subvolumes)
• Package: monerod (community registry), 259 GiB monerod volume (single LMDB data.mdb)

cc @dr-bonez — this is your 900d86ab8 feature; happy to test a patch on this box (it reproduces every time).