[bug]: Read-only file system (os error 30) when syncing Bitcoin Core

[otech47]

Server Hardware

Start9 Server One 16 GB RAM, 2 TB NVMe (Samsung SSD 990 EVO Plus 2TB, firmware 2B2QKXG7). AMI BIOS 5.19. Boot and data both on same disk. It shipped with StartOS 0.3.5, then I flashed it to 0.4 with a full clean disk overwrite

StartOS Version

0.4.0-beta.9

Client OS

n/a for this report (server-side bug, independent of the client device)

Client OS Version

n/a

Browser

n/a (server-side, browser-independent)

Browser Version

n/a

Current Behavior

• Brand-new Server One, freshly flashed to StartOS 0.4, with a clean install of Bitcoin Core. No customizations.
• A few percent into the Bitcoin sync, the data volume goes read-only and login to the web GUI breaks.
• The GUI page itself loads fine. The problem is signing in: entering the correct password returns an error instead of logging in (exact text below).
• Only a hard power cycle recovers it. After that, login works again, but resuming the sync brings the failure back.
• Reproducible every time once the sync has run a while.

Expected Behavior

• A service doing sustained disk writes (Bitcoin initial sync) should be able to run to completion without the server going read-only or needing a hard reset.

Steps to Reproduce

1. Brand-new Start9 Server One (recommended packaged hardware, 16 GB RAM, 2 TB NVMe, no customizations). It shipped with StartOS 0.3.5.
2. On 0.3.5: booted it, briefly started and stopped a Bitcoin Core sync. Did not run it long, so I cannot say for sure 0.3.5 is unaffected.
3. Flashed StartOS 0.4 (0.4.0-beta.9), which fully overwrote the disk. The flash went smoothly.
4. Installed Bitcoin Core fresh on 0.4 (Bitcoin Core 31.0, StartOS package 31.0:14) and started it (initial block download).
5. A few percent into the sync the failure starts: the data volume goes read-only, GUI login then fails with the error, and only a hard power cycle recovers it. Recurs on resuming the sync.

• The unit arrived a little banged up from shipping (via BitSaga), so my first thought was a hardware fault. Diagnosis ruled that out. The drive itself seems healthy

Anything else?

SSHed in to debug and under sustained write load the kernel log repeats this:

nvme nvme0: controller is down; will reset: CSTS=0x3, PCI_STATUS=0x11

Around the same time btrfs logs checksum errors on the data volume:

BTRFS error (device dm-3): ... errs: wr 0, rd 0, flush 0, corrupt 31
BTRFS warning (device dm-3): csum failed root 5 ino 9997 ... csum 0x.. expected 0x..

btrfs then remounts read-only to protect itself. That is what breaks login. The GUI page still loads, but signing in with the correct password returns:

RPC ERROR: Database Error
IO Error: Read-only file system (os error 30): Read-only file system (os error 30)

The drive does not look like it is failing: SMART PASSED, 0 media errors, 0% wear, 100% spare, ~56C, already on the latest Samsung firmware (2B2QKXG7). The resets also seem to happen at light load (very roughly one every 20 to 40 min idle), and heavy writes (Bitcoin IBD) seem to make them frequent enough to trip read-only quickly.

Theory: this looks like a power-management interaction on this DRAM-less drive, which leans on Host Memory Buffer (HMB). HMB and NVMe power-state handling are coupled in the kernel, so a drive that mishandles a power-state transition while also using HMB is a plausible fit for a controller that wedges under load. One reference on that HMB / power-control coupling: https://bugs.launchpad.net/bugs/1950042 ("Let NVMe with HMB use native power control again"). Not 100% sure but it fits what I see, and disabling both APST and HMB is what stabilized it here.

Workaround I am running (seems to hold so far): two kernel params on the x86 cmdline:

• nvme_core.default_ps_max_latency_us=0 (disables APST)
• nvme.max_host_mem_size_mb=0 (disables HMB)

With both, Bitcoin Core ran ~27 min of continuous block validation (81 GB written, 0% to ~10%) with no controller resets and no new corruption, and that held across a reboot. APST-off alone did not seem to be enough in my testing. HMB-off seemed to be the one that settled it. Small sample, so suggestive rather than proven.

If it is worth pursuing at the startos level, the x86 cmdline appears to be set in debian/startos/postinst:

GRUB_CMDLINE_LINUX="boot=startos console=ttyS0,115200n8 console=tty0"

Maybe those two params (or a proper per-drive quirk) belong there. It looks similar to the Pi USB quirks already set in build/image-recipe/raspberrypi/squashfs/etc/default/grub.d/raspberrypi.cfg (usb-storage.quirks=...)

It's also possible I just need more amps/watts from the circuit I have the box plugged into. I didn't look too hard into this theory.

This might also be the same underlying thing as the closed #2140 (Read-only file system / Database Error) and possibly #2336 (very slow IBD, which could be the controller resetting and retrying). Not sure.

[helix-nine]

Thanks for the unusually thorough report — the kernel logs and your workaround testing are genuinely helpful. Before we ship a system-wide change, we want to confirm the actual mechanism rather than lock in the APST/HMB theory, because (a) it's a small sample so far (you noted ~27 min, "suggestive rather than proven"), and (b) there's upstream evidence on this same drive (Samsung 990 EVO Plus) where disabling APST alone did not stop the controller resets — so we'd be guessing. A closed PR (#3338) tried to encode the workaround but didn't match the cause, and its HMB half wouldn't have taken effect anyway.

A few things would help us pin it down. No rush, and incremental is totally fine.

Drive + controller state (run right after a clean boot, while healthy):

• sudo nvme id-ctrl /dev/nvme0 | grep -iE 'hmb|hmpre|hmmin' — does the drive even request HMB, and how much?
• sudo nvme get-feature /dev/nvme0 -f 0x0c -H — current APST table/config
• sudo nvme get-feature /dev/nvme0 -f 0x02 -H — current power-management state
• sudo lspci -vvv -s <nvme-bus-address> (the LnkCtl / ASPM lines; get the address from lspci | grep -i nvme) — PCIe link power state
• uname -a and cat /sys/module/nvme_core/parameters/default_ps_max_latency_us

Isolate the variable — this is the key ask. Instead of testing both knobs together, run the Bitcoin IBD long enough to normally trigger a reset under each config separately, and report which one actually holds:

1. Baseline (no extra params) — confirm it still resets
2. nvme_core.default_ps_max_latency_us=0 only
3. nvme.max_host_mem_size_mb=0 only
4. pcie_aspm=off only
5. combinations only if no single one is enough

That tells us whether this is APST, HMB, ASPM, or something else — which determines the right fix (and whether a targeted PCI-ID kernel quirk is better than a global param).

During/after a failure:

• Full dmesg from boot through the first reset (not just the two lines) — we want what the controller was doing immediately before CSTS=0x3
• sudo nvme smart-log /dev/nvme0 captured under sustained load (temperature/throttling/critical warnings) — to rule out thermal
• If feasible: does it reproduce on a different power supply/circuit, or with the drive in a different machine? That helps rule out the power-delivery angle you raised.

The isolation runs (which single param, if any, prevents the reset over a long sync) are by far the most valuable — everything else is supporting context.