LRU eviction panics when Cache::clear() races a concurrent insert (weight/indexer accounting)

[jbrown9513]

Summary

clear() on an in-memory LRU cache is not safe against a concurrent insert(): when a fill inserts while another thread is clearing the same cache, foyer's LRU accounting goes inconsistent and a strict_assert!/assert! fires, panicking the worker (or the clearing task). It reproduces under moderate concurrency; the window is small, so it is timing-dependent and shows up mainly on slower/loaded machines.

Version

• foyer 0.22.3 (foyer-memory 0.22.3)
• Reproduced on aarch64-apple-darwin (Rust 1.96.1); originally observed on an x86_64 Linux CI runner.

Assertion traces

Two distinct assertions fire depending on which side wins the race — both are the same underlying clear vs insert unsoundness:

1. LRU weight accounting, on the clearing side:

foyer-memory-0.22.3/src/eviction/lru.rs:238: assertion `left == right` failed
  left: 6291844
  right: 0

This is the tail of Lru::clear:

assert!(self.list.is_empty());
assert!(self.high_priority_list.is_empty());
assert!(self.pin_list.is_empty());
assert_eq!(self.high_priority_weight, 0);   // <-- left: 6291844, right: 0

high_priority_weight is still charged for an entry whose list membership was concurrently mutated by an in-flight insert, so the weight the clear expects to have drained back to 0 is non-zero.

2. Indexer sentry, on the insert side (same run, different iteration):

thread 'tokio-rt-worker' panicked at foyer-memory-0.22.3/src/indexer/sentry.rs:47:9:
assertion failed: !record.is_in_indexer()

Scenario

A single Cache/HybridCache memory tier with:

• background/concurrent insert() calls (cold fills), and
• a clear() called on the same cache from another task.

clear() takes &mut self on the eviction structure, but foyer applies eviction ops (acquire/Op::mutable) lazily under separate lock acquisitions, so an insert's weight bump and its list/indexer linkage are not atomic with respect to a concurrent clear(). clear() can observe a half-applied insert (weight charged but not linked, or linked but not yet drained), tripping the end-of-clear invariants.

Minimal repro sketch

// One in-memory cache; hammer inserts of distinct keys from N tasks while
// a second loop calls clear() repeatedly. Under thread pressure this panics
// inside foyer within seconds to a couple of minutes.
let cache = /* CacheBuilder::new(cap).with_eviction_config(LruConfig::default()).build() */;
for t in 0..24 {
    let cache = cache.clone();
    tokio::spawn(async move {
        let mut i = t;
        loop { cache.insert(i, vec![0u8; 64 * 1024]); i += 7; }
    });
}
for _ in 0..4000 { cache.clear(); tokio::task::yield_now().await; }

Impact / workaround

We hit this in a cache-purge path (POST /cache/purge) that calls clear() on live stores while reads (and therefore fills/inserts) are in flight — a production crash risk under traffic. Our application-side workaround is a serialize-admissions-against-clear flag: we stop admitting new entries for the duration of the clear window, so no insert() overlaps clear(). That avoids the panic but is a band-aid; clear() racing insert() on the same cache should be memory-safe (or at least not panic) inside foyer.

Happy to provide a self-contained reproduction crate if useful.