-
Notifications
You must be signed in to change notification settings - Fork 869
feat(shard distributor): use smoothed load for initial ephemeral shard placement #7457
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
AndreasHolt
wants to merge
67
commits into
cadence-workflow:master
Choose a base branch
from
AndreasHolt:initial-placement-with-statistics
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
feat(shard distributor): use smoothed load for initial ephemeral shard placement #7457
AndreasHolt
wants to merge
67
commits into
cadence-workflow:master
from
AndreasHolt:initial-placement-with-statistics
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Andreas Holt <[email protected]>
Signed-off-by: Andreas Holt <[email protected]>
… is being reassigned in AssignShard Signed-off-by: Andreas Holt <[email protected]>
Signed-off-by: Andreas Holt <[email protected]>
Signed-off-by: Andreas Holt <[email protected]>
Signed-off-by: Andreas Holt <[email protected]>
…to not overload etcd's 128 max ops per txn Signed-off-by: Andreas Holt <[email protected]>
…s txn and retry monotonically Signed-off-by: Andreas Holt <[email protected]>
…ents Signed-off-by: Andreas Holt <[email protected]>
…shard metrics, move out to staging to separate function Signed-off-by: Andreas Holt <[email protected]>
Signed-off-by: Andreas Holt <[email protected]>
Signed-off-by: Andreas Holt <[email protected]>
… And more idiomatic naming of collection vs singular type Signed-off-by: Andreas Holt <[email protected]>
…ook more like executor key tests Signed-off-by: Andreas Holt <[email protected]>
…ey in BuildShardKey, as we don't use it Signed-off-by: Andreas Holt <[email protected]>
…o "statistics" Signed-off-by: Andreas Holt <[email protected]>
…ollow conventions Signed-off-by: Andreas Holt <[email protected]>
…o ewma) Signed-off-by: Andreas Holt <[email protected]>
…t heartbeat Signed-off-by: Andreas Holt <[email protected]>
…rdStatistics Signed-off-by: Andreas Holt <[email protected]>
Signed-off-by: Andreas Holt <[email protected]>
…unt, for logging Signed-off-by: Andreas Holt <[email protected]>
… for ephemeral shards Signed-off-by: Andreas Holt <[email protected]>
…xecutor properly Signed-off-by: Andreas Holt <[email protected]>
…e with new load based selection Signed-off-by: Andreas Holt <[email protected]>
Signed-off-by: Andreas Holt <[email protected]>
Signed-off-by: Andreas Holt <[email protected]>
Signed-off-by: Andreas Holt <[email protected]>
…eartbeat TTL Signed-off-by: Andreas Holt <[email protected]>
…o ewma) Signed-off-by: Andreas Holt <[email protected]>
…eartbeat TTL Signed-off-by: Andreas Holt <[email protected]> Signed-off-by: Theis Randeris Mathiassen <[email protected]>
…o ewma) Signed-off-by: Andreas Holt <[email protected]> Signed-off-by: Theis Randeris Mathiassen <[email protected]>
…t heartbeat Signed-off-by: Andreas Holt <[email protected]> Signed-off-by: Theis Randeris Mathiassen <[email protected]>
…rdStatistics Signed-off-by: Andreas Holt <[email protected]> Signed-off-by: Theis Randeris Mathiassen <[email protected]>
Signed-off-by: Andreas Holt <[email protected]> Signed-off-by: Theis Randeris Mathiassen <[email protected]>
Signed-off-by: Theis Randeris Mathiassen <[email protected]>
Signed-off-by: Andreas Holt <[email protected]> Signed-off-by: Theis Randeris Mathiassen <[email protected]>
Signed-off-by: Theis Randeris Mathiassen <[email protected]>
Signed-off-by: Theis Randeris Mathiassen <[email protected]>
Signed-off-by: Theis Randeris Mathiassen <[email protected]>
Signed-off-by: Andreas Holt <[email protected]>
…adencefork into heartbeat-shard-statistics Signed-off-by: Andreas Holt <[email protected]>
Signed-off-by: Andreas Holt <[email protected]>
…on from heartbeat TTL Signed-off-by: Andreas Holt <[email protected]>
…-statistics Signed-off-by: Andreas Holt <[email protected]>
Signed-off-by: Andreas Holt <[email protected]>
…dd+modify tests Modified pickLeastLoadedExecutor to skip executrs with non ACTIVE status when selecting an executor for shard assignment. Updated error message to reflect when no active executors are available. Add tests to cover this behavior. Adjust previous tests to also mock executors s.t. we get executors status Signed-off-by: Andreas Holt <[email protected]>
Signed-off-by: Theis Mathiassen <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changed?
assignEphemeralShardto call a newpickLeastLoadedExecutorhelper that sums smoothed shard load per executor (falling back to shard count) and log the selected targetAggregateLoadandAssignedCounttags so handler logs can show load totals when placing ephemeral shardsWhy?
How did you test it?
TestPickLeastLoadedExecutorandShardNotFound_Ephemeral_LoadBasedin handler_test.go to verify logic.Potential risks
If shard stats are missing or stale, the aggregated load will calculate as zero. In this case, the logic degrades to the previous behavior (selecting based on shard count), minimizing the risk of bad placement.
Release notes
Ephemeral shard placement now favors the executor with the lowest smoothed load (with shard-count tie breaker) and logs the inputs for each decision.
Documentation Changes