io_uring engine busy-spins at 100% CPU when idle (no backoff in `UringIoEngineShard::run`)

[vishwasgarg18]

Summary

The io_uring I/O engine runs a dedicated per-shard thread (foyer-uring-{i}) whose loop never blocks. When there is no work, no submissions queued and no completions outstanding the loop still runs continuously, pinning a full CPU core per engine instance even though the cache is completely idle (zero get/put traffic).

Because the loop uses only non-blocking checks (a try_recv() on the submission channels plus a non-blocking drain of the io_uring completion queue), it never parks. Every HybridCache that selects the io_uring engine therefore keeps one core at ~100% for the lifetime of the process, regardless of load.

• Affected: foyer-storage (io_uring engine). Reproduced on 0.22.3; the loop is unchanged on main at the time of writing.
• Not affected: the psync engine (it parks when idle).
• Type: performance / resource-usage bug (wasted CPU), not a correctness bug. (There is a separate correctness issue in the same engine: Use-after-free in io_uring #1286.)

Root cause

UringIoEngineShard::run (foyer-storage/src/io/engine/uring.rs) is structured as:

fn run(mut self) {
    loop {
        // 1) SUBMISSION side — non-blocking:
        'prepare: loop {
            if self.read_inflight + self.write_inflight >= self.io_depth { break 'prepare; }
            let ctx = /* self.read_rx.try_recv() / self.write_rx.try_recv() */;
            let ctx = match ctx { Some(ctx) => ctx, None => break 'prepare }; // both channels empty -> leave
            // build SQE, push to submission queue
        }

        // 2) submit only if something is in flight
        if self.read_inflight + self.write_inflight > 0 {
            self.uring.submit().unwrap();
        }

        // 3) COMPLETION side — non-blocking drain:
        for cqe in self.uring.completion() {
            // reap, send result back
        }
    }
}

When the cache is idle:

1. Both submission channels are empty → try_recv() returns Empty → break 'prepare.
2. read_inflight + write_inflight == 0 → submit() is skipped.
3. self.uring.completion() yields nothing → the for body never runs.
4. The outer loop immediately restarts and repeats forever.

Nothing in the loop ever blocks: try_recv() is the non-blocking receive, and completion() is a non-blocking iterator over whatever CQEs currently exist. So the thread stays runnable with no work to do and consumes a full core. (In a thread dump the hot frame is Receiver::try_recv, which is the top of the loop, but the completion-side drain is equally non-blocking.)

Because UringIoEngineConfig defaults to threads: 1, this is one pinned core per engine; an application that builds N HybridCache instances backed by io_uring pins N cores at idle.

Reproduction

Minimal: build the io_uring engine, submit no I/O, and observe a foyer-uring-* thread pinning a core.

#[tokio::test(flavor = "multi_thread")]
async fn uring_idle_busy_spin() {
    let dir = tempfile::tempdir().unwrap();
    let device = FsDeviceBuilder::new(dir).build().unwrap();

    let _engine = UringIoEngineConfig::new()
        .boxed()
        .build(IoEngineBuildContext { spawner: Spawner::current() })
        .await
        .unwrap();

    // No reads or writes are ever issued.
    // Observe: a `foyer-uring-0` thread sits at ~100% CPU.
    std::thread::sleep(std::time::Duration::from_secs(10));
}

Observe with top -H -p <pid> (or htop, press H): a foyer-uring-0 thread reports ~100% CPU for the full 10 s despite zero I/O.

To make it deterministic, sample the engine thread's CPU time from /proc/self/task/<tid>/stat (fields utime+stime) before and after the sleep and assert it grew by roughly the wall-clock interval. On the current code the accumulated CPU time ≈ elapsed wall time (spinning); with a fix it should be ≈ 0.

The equivalent happens through the public API: a HybridCacheBuilder configured .with_io_engine_config(UringIoEngineConfig::new()) and then left idle pins a core with no cache operations.

Impact

• One core pinned at ~100% per io_uring engine instance, continuously, independent of load. An idle process is not idle.
• Scales linearly with the number of HybridCache instances (e.g. an app with a data cache and a metadata cache pins two cores).
• On small hosts this is a large fraction of total CPU; it also steals CPU from co-located work and shows up as constant baseline utilization, wasted power, and (for CPU-bound co-tenants) contention.
• Only manifests when the io_uring engine is selected (including via any auto/default path that chooses io_uring on capable kernels).

Proposed fix: adaptive spin-then-block

The loop already tracks read_inflight / write_inflight, so the idle condition ("submission channels empty and in-flight count == 0") is directly detectable at the existing break 'prepare. When idle, the thread should park instead of re-looping. To preserve latency under load, wrap it as spin-then-block:

1. Fast path (work present): current behavior — reap completions and pick up submissions without blocking.
2. Bounded adaptive spin: on finding nothing, spin for a short, bounded window in case work is imminent (avoids a park/wake cycle under sustained throughput).
3. Park (idle): if still empty, block until either a submission or a completion arrives.

The thread must wake for either source (a new submission on the channel, or a completion on the CQ). Two candidate implementations:

• recv_timeout() on the submission channel — smallest change: block on the channel with a short timeout; the timeout bounds how often the CQ is re-polled. No new fd, no unsafe; small latency floor when idle.
• [Recommended] Registered eventfd + submit_and_wait(1) — block in io_uring_enter waiting on completions, and arm/register an eventfd so a submission also produces a completion and wakes the same wait. One blocking call covers both sources; zero periodic wakeups. This matches how tokio-uring / monoio integrate an io_uring reactor.

Either is contained to the engine loop.

Correctness properties a fix should uphold

• No lost wakeups: a submission or a completion arriving while the thread is parked always wakes it; work is never left unreaped (no deadlock/stall).
• Idle ⇒ ~0% CPU: with no in-flight ops and no submissions for a bounded interval, the engine thread consumes ~0% CPU.
• No latency regression under load: completion-to-delivery latency under sustained load stays within an acceptable delta of the current pure-spin behavior.
• Prompt shutdown: a parked thread still exits promptly when the engine/shard is dropped (today shutdown is delivered via the channel Disconnected; the fix must preserve that wakeup).

Environment

• foyer-storage 0.22.3 (also present on main).
• Linux with io_uring available; io_uring engine selected. Default engine config (threads: 1, sqpoll: false, iopoll: false) — so this is a userspace busy-poll, not kernel SQPOLL.

Related

• The io_uring engine was made opt-in (default psync) in a downstream project specifically because of this idle-CPU cost: make foyer io_uring engine opt-in (default psync) beyondoss/glidefs#64
• Separate correctness issue in the same engine (use-after-free): Use-after-free in io_uring #1286

I'm happy to open a PR implementing the spin-then-block fix (starting with the recv_timeout variant) if that's welcome.