test(directory): assert rolling upgrade invariants by phase - #10705
Merged
ReubenBond merged 5 commits intoAug 20, 2026
Merged
Conversation
Deactivate registrations which lose distributed directory recovery after the recovered range unlocks, and serialize conditional cleanup with activation registration. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR addresses a distributed grain directory recovery edge case during rolling upgrades where multiple valid activations for a single grain can be recovered, ensuring that only the deterministically selected “winner” remains active and that integrity validation does not fail due to stale-but-valid losing activations.
Changes:
- Track all recovered activation candidates during range recovery, compute which recovered activations are duplicates (non-winners), and retire those duplicates.
- Gate integrity checking on completion of any in-flight recovery cleanup so validation doesn’t race deactivation.
- Add conditional deactivation logic in the catalog/activation lifecycle so deletes only deactivate an activation when its address (and membership version) matches, including during Creating/Activating states.
Show a summary per file
| File | Description |
|---|---|
test/Orleans.GrainDirectory.Tests/GrainDirectory/GrainDirectoryPartitionTests.cs |
Extends recovery-ordering test to assert duplicate detection behavior. |
src/Orleans.Runtime/GrainDirectory/GrainDirectoryPartition.cs |
Returns duplicates from recovery, kicks off post-recovery cleanup, and makes integrity checks await cleanup completion. |
src/Orleans.Runtime/Catalog/Catalog.cs |
Makes DeleteActivations conditional on the target activation address (and membership version) matching the current context. |
src/Orleans.Runtime/Catalog/ActivationData.cs |
Adds an address-matching deactivation operation that runs through the activation’s serialized operation queue when needed. |
Review details
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 4/4 changed files
- Comments generated: 1
- Review effort level: Lite
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Treat cross-version Local/Distributed activation conflicts as a bounded mixed-mode state while retaining strict registration and ownership checks after convergence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep the bounded-grain checkpoint message separate from phase-specific activation and partition validation output. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
During a mixed LocalGrainDirectory/DistributedGrainDirectory rolling upgrade, compatibility recovery can expose one older local activation alongside one newer distributed activation. This cross-membership transition state converges when the legacy host exits; exact registration uniqueness and partition integrity are steady-state guarantees after every host uses DistributedGrainDirectory and directory views converge.
Make the rolling-upgrade validation phase-aware. Mixed checkpoints require conflicts to contain exactly one older LocalGrainDirectory activation and one newer DistributedGrainDirectory activation, while bounded tracked grains continue to prove fresh authoritative lookup and traffic progress. Homogeneous checkpoints require unique activations and fresh matching registrations, and the fully distributed phase also validates partition ownership and complete directory integrity.
Fixes #10519
Microsoft Reviewers: Open in CodeFlow