ObjectAllocatorImpl.Dispose() nulls objectPages before the LogSizeTracker resizer is stopped, FailFasting the process

[hexonal]

Summary

ObjectAllocatorImpl.Dispose() nulls objectPages before the LogSizeTracker resizer is stopped, so a resizer already inside ShiftAddresses can reach EvictRecordsInRange and dereference the null array. Since #2056 that NullReferenceException reaches Environment.FailFast, which kills the process.

This is a store-shutdown race, not a data-path bug. It surfaces in CI as a whole test run aborting mid-suite with no failing assertion.

Observed

Garnet Standalone (ubuntu-latest, net8.0, Release, Garnet.test.complexstring), during Garnet.test.HyperLogLogTests:

Process terminated. OnPagesClosedWorker failed; allocator state is unrecoverable.
ClosedUntilAddress=114752, OngoingCloseUntilAddress=131072
   at System.Environment.FailFast(System.String, System.Exception)
   at Tsavorite.core.AllocatorBase`2.OnPagesClosedWorker()
   at Tsavorite.core.AllocatorBase`2.OnPagesClosed(Int64)
   at Tsavorite.core.LightEpoch.Drain(Int64)
   at Tsavorite.core.LightEpoch.BumpCurrentEpoch(System.Action)
   at Tsavorite.core.AllocatorBase`2.ShiftHeadAddress(Int64)
   at Tsavorite.core.AllocatorBase`2.ShiftAddressesWithWait(Int64, Int64, Boolean)
   at Tsavorite.core.LogAccessor`2.ShiftAddresses(Int64, Int64, Boolean)
   at Tsavorite.core.LogSizeTracker`2.ResizeIfNeeded(System.Threading.CancellationToken)
   at Tsavorite.core.LogSizeTracker`2+<ResizerTask>d__41.MoveNext()

System.NullReferenceException: Object reference not set to an instance of an object.
   at Tsavorite.core.ObjectAllocatorImpl`1.EvictRecordsInRange(Int64 startAddress, Int64 endAddress, EvictionSource source, Boolean isRecovery)
      in libs/storage/Tsavorite/cs/src/core/Allocator/ObjectAllocatorImpl.cs:line 399
   at Tsavorite.core.AllocatorBase`2.OnPagesClosedWorkerCore()
   at Tsavorite.core.AllocatorBase`2.OnPagesClosedWorker()

The whole run aborted after 383 passing tests; the remainder never executed.

Mechanism

Line numbers below are against 8c54cc278 (current main).

AllocatorBase.Dispose() states the required ordering explicitly (AllocatorBase.cs:501):

public virtual void Dispose()
{
    disposed = true;

    // Stop the size-tracker resizer and wait for it to exit BEFORE tearing down the epoch, buffer pool, flush event
    // (and, by the owner, the log device) that it uses; otherwise a still-running resizer can spin on or dereference
    // these cleared resources. ...
    logSizeTracker?.Stop(wait: true);
    ...

ObjectAllocatorImpl.Dispose() tears down its own resources first and only then calls that (ObjectAllocatorImpl.cs:492):

public override void Dispose()
{
    var localValues = Interlocked.Exchange(ref objectPages, null);   // (1) objectPages is gone
    if (localValues != null)
    {
        freePagePool.Dispose();
        foreach (var value in localValues)
            value.Clear();                                            // (2) every ObjectIdMap cleared
        base.Dispose();                                               // (3) only now: disposed = true; Stop(wait: true)
    }
}

Between (1) and (3) the allocator is in a state the resizer is not prepared for: objectPages == null and disposed == false.

LogSizeTracker checks runState at LogSizeTracker.cs:275 and then calls ShiftAddresses; a resizer already past that check runs to completion. It reaches ObjectAllocatorImpl.cs:411:

var objectIdMap = objectPages[GetPageIndexForAddress(address)].objectIdMap;

which is the null dereference in the trace.

Two consequences worth noting:

• if (disposed) return; in OnPagesClosedWorkerCore would not close this window. disposed is set inside base.Dispose(), i.e. after the null, so the hazardous interval is exactly the one where disposed is still false. This is the most likely way a well-meaning fix lands and the flake survives.
• Step (2) gives a second, quieter signature of the same window: a resizer that already loaded its objectIdMap local walks a cleared map rather than crashing.

Why it started showing up on 2026-08-12

The race is older than the crash. #2056 (eeaa6ffc4) changed the handler in AllocatorBase.OnPagesClosedWorker (AllocatorBase.cs:1707):

-                logger?.LogCritical(ex, "OnPagesClosedWorker failed, page closing will not resume. ...");
-                throw;
+                Environment.FailFast($"OnPagesClosedWorker failed; allocator state is unrecoverable. ...", ex);

Before that, the identical NRE unwound into ResizerTask's catch-all and was invisible — the store was being torn down anyway, so the wedged OngoingCloseUntilAddress was harmless. From 2026-08-12 the same interleaving kills the host. The FailFast is not the bug; it made an existing one visible.

Suggested direction

Hoist the background-work shutdown ahead of the derived teardown, e.g. a protected void StopBackgroundWork() on AllocatorBase (setting disposed = true and logSizeTracker?.Stop(wait: true)) that each derived Dispose() calls as its first statement. Stop CASes on runState (LogSizeTracker.cs:173), so the later call from base.Dispose() is a no-op.

I have not opened a PR: ObjectAllocatorImpl.cs has active owners and the epoch/resizer invariants here are yours, not mine. Happy to put one up if that is useful.

Two related observations, offered without a claim that they are the same defect:

• SpanByteAllocatorImpl.Dispose() (SpanByteAllocatorImpl.cs:224) has the identical shape — Interlocked.Exchange(ref freePagePool, null) before base.Dispose() — and freePagePool is reached from the same worker via FreePage → ReturnPage.
• GarnetDatabase.Dispose calls Store?.Dispose() before SizeTracker?.Stop(), and that Stop is wait: false, which widens the window from the Garnet side.

One thing I could not establish: the reported ClosedUntilAddress=114752 / OngoingCloseUntilAddress=131072 do not by themselves pin line 399 — ClosedUntilAddress is only advanced after EvictRecordsInRange returns for the whole page, so the arithmetic is equally consistent with an NRE in the record walk further down. The stack frame is what identifies the line.