[Tsavorite] Handle ObjectAllocator device read failures - #2075
Merged
Conversation
Propagate device errors and short reads instead of treating callback completion as success. Abort unsafe flush and truncation work, preserve fresh snapshot behavior, and add deterministic regression coverage. Fixes #2073 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8ade2dd0-3bdc-4523-9ca9-5b5dcddec9d2
Ted Hart (TedHartMS)
requested review from
Badrish Chandramouli (badrishc)
and
a balanced review from Copilot
August 18, 2026 21:14
Contributor
There was a problem hiding this comment.
Pull request overview
Adds reliable ObjectAllocator device-read failure propagation to prevent truncation and flush corruption.
Changes:
- Propagates device errors, short reads, and exceptions.
- Stops dependent flush/truncation operations after failed reads.
- Adds regression tests for failure, retry, and snapshot paths.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
ObjectAllocatorDeviceFailureTests.cs |
Adds deterministic device-failure regression tests. |
PageAsyncResultTypes.cs |
Carries complete read completion details. |
ObjectAllocatorImpl.cs |
Validates reads and safely routes failures. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Badrish Chandramouli (badrishc)
approved these changes
Aug 19, 2026
Embed the caught exception's full ToString output in the OnPagesClosedWorker FailFast message so CI logs include its type, message, inner exceptions, and stack trace. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8ade2dd0-3bdc-4523-9ca9-5b5dcddec9d2
Format the caught page-close exception inside a try/finally so Environment.FailFast always executes. Fall back to the exception type if a custom Exception.ToString implementation throws. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8ade2dd0-3bdc-4523-9ca9-5b5dcddec9d2
Badrish Chandramouli (badrishc)
approved these changes
Aug 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root Cause
ObjectAllocatorImpl.AsyncReadPageCallbacklogged device read failures but signaled completion without preserving the error. Its header-read and partial-sector read-back callers therefore continued after failed reads. Zero-error short reads were also treated as successful completions.Description of Change
Device read completion now carries the error code, transferred byte count, and exception, and existing-data reads succeed only when the requested length is transferred.
Key changes:
EndOfStreamExceptionfailuresOnTruncatecallbacks and allowing a later retryFlushedUntilAddressToString()output in theOnPagesClosedWorkerfatal message so CI logs contain its type, message, inner exceptions, and stack trace even when the runtime omits theFailFastexception argument; formatting runs insidetry/finallyso a broken customToString()cannot bypass fail-fast terminationKey Technical Details
Affected types:
PageAsyncReadResult<TContext>- carries the full device completion resultObjectAllocatorImpl<TStoreFunctions>- validates exact reads and prevents unsafe follow-on workAllocatorBase<TStoreFunctions, TAllocator>- preserves complete page-close exception diagnostics in fail-fast outputObjectAllocatorDeviceFailureTests- injects device errors and EOF-style short readsTesting
Tsavorite.corebuilds successfully for .NET 8 and .NET 10Issues Fixed
Fixes #2073