Add WeightedRandomPicker #1412

Jooho · 2025-08-19T23:20:20Z

What type of PR is this?
/kind feature

What this PR does / why we need it:
Implements weighted random sampling picker with 4 normalization options:

None (default): Pure weighted random sampling
Square Root: Moderate score difference reduction
Capping: Limits max score ratio to prevent hot-spotting
Logarithmic: Maximum load distribution for extreme score variations

Addresses hot-spotting issues in max-score-picker while maintaining score-based preferences through configurable normalization strategies.

Which issue(s) this PR fixes:

Fixes ##1411

Does this PR introduce a user-facing change?:

Implements weighted random sampling picker with 4 normalization options: - None (default): Pure weighted random sampling - Square Root: Moderate score difference reduction - Capping: Limits max score ratio to prevent hot-spotting - Logarithmic: Maximum load distribution for extreme score variations Addresses hot-spotting issues in max-score-picker while maintaining score-based preferences through configurable normalization strategies. Signed-off-by: Jooho Lee <[email protected]>

netlify · 2025-08-19T23:20:25Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`257fdb0`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/68af359fe90b0e0008a43ee9
😎 Deploy Preview	https://deploy-preview-1412--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

linux-foundation-easycla · 2025-08-19T23:20:26Z

The committers listed above are authorized under a signed CLA.

✅ login: Jooho / name: Jooho Lee (257fdb0, 799543d, 41b64c5, be7ff49, f72bd77)

k8s-ci-robot · 2025-08-19T23:20:28Z

Welcome @Jooho!

It looks like this is your first PR to kubernetes-sigs/gateway-api-inference-extension 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/gateway-api-inference-extension has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2025-08-19T23:20:29Z

Hi @Jooho. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

pkg/epp/scheduling/framework/plugins/picker/weighted_random_picker.go

pkg/epp/scheduling/framework/plugins/picker/picker_test.go

pkg/epp/scheduling/framework/plugins/picker/weighted_random_picker.go

Signed-off-by: Jooho Lee <[email protected]>

pkg/epp/scheduling/framework/plugins/picker/weighted_random_picker.go

nirrozenbaum · 2025-08-21T10:39:48Z

pkg/epp/scheduling/framework/plugins/picker/weighted_random_picker.go

+		randomGenerator.Shuffle(len(scoredPods), func(i, j int) {
+			scoredPods[i], scoredPods[j] = scoredPods[j], scoredPods[i]
+		})
+		if p.maxNumOfEndpoints < len(scoredPods) {
+			scoredPods = scoredPods[:p.maxNumOfEndpoints]
+		}


this part should probably leverage and reuse the existing random picker. there is no point in rewriting the logic.
I think it would be good to have random picker as a private field of the weightedRandomPicker.

nirrozenbaum · 2025-08-21T10:40:26Z

pkg/epp/scheduling/framework/plugins/picker/weighted_random_picker.go

+type weightedRandomPickerParameters struct {
+	MaxNumOfEndpoints int `json:"maxNumOfEndpoints"`
+}


IMO we should remove this and reuse the common picker paramters struct that is used by the other pickers

pkg/epp/scheduling/framework/plugins/picker/weighted_random_picker.go

…logic Signed-off-by: Jooho Lee <[email protected]>

… all weights are zero Signed-off-by: Jooho Lee <[email protected]>

nirrozenbaum · 2025-08-24T06:15:15Z

pkg/epp/scheduling/framework/plugins/picker/picker_test.go

+	tests := []struct {
+		name                  string
+		input                 []*types.ScoredPod
+		maxPods               int // maxNumOfEndpoints for this test
+		iterations            int
+		expectedProbabilities map[string]float64 // pod name -> expected probability
+		tolerancePercent      float64            // acceptable deviation percentage


I like the way you implemented a probability based test. I think we should give it a try and hopefully we don't see flaky tests in CI.
I have several minor comments:

can we set iterations as a const or a local variable and just use it across all the tests? I didn't understand why different tests have different number of iterations. also I assume 500 iterations should be enough to have representative results.

can we move expectedProb to be calculated in the code and not set as input? it looks like magic numbers and this is an easy calculation (summing the scores, dividing score / sum).

tolerance can also be a local variable or a const, isn't it? why should the tolerance be different between tests?

nit: I think it's easier to understand if we use tolerance value in the range of [0,1] (instead of dividing by 100).

I used 1000 iterations because 500 iterations sometimes make tests fail. The others followed your suggestions
Thx!

nirrozenbaum · 2025-08-24T15:30:42Z

/ok-to-test

yafengio · 2025-08-25T09:15:52Z

pkg/epp/scheduling/framework/plugins/picker/weighted_random_picker.go

+			continue
+		}
+
+		// Generate random number U in (0,1)


Can it be updated to// Generate random number U in [0.0, 1.0) here?

It will be clearer.

No 0 is always excluded so the existing comment is correct.

u := randomGenerator.Float64() if u == 0 { u = 1e-10 // Avoid log(0) #<-- this will be like 0.0000000001 }

Signed-off-by: Jooho Lee <[email protected]>

k8s-ci-robot · 2025-08-27T16:43:17Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Jooho
Once this PR has been reviewed and has the lgtm label, please ask for approval from nirrozenbaum. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added the kind/feature Categorizes issue or PR as related to a new feature. label Aug 19, 2025

k8s-ci-robot requested review from danehans and liu-cong August 19, 2025 23:20

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Aug 19, 2025

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Aug 19, 2025

Jooho mentioned this pull request Aug 19, 2025

Poor performance with bursty workloads llm-d/llm-d-inference-scheduler#298

Open

nirrozenbaum reviewed Aug 20, 2025

View reviewed changes

pkg/epp/scheduling/framework/plugins/picker/weighted_random_picker.go Outdated Show resolved Hide resolved

nirrozenbaum reviewed Aug 20, 2025

View reviewed changes

pkg/epp/scheduling/framework/plugins/picker/weighted_random_picker.go Outdated Show resolved Hide resolved

nirrozenbaum reviewed Aug 20, 2025

View reviewed changes

pkg/epp/scheduling/framework/plugins/picker/picker_test.go Outdated Show resolved Hide resolved

kfswain reviewed Aug 20, 2025

View reviewed changes

remove normalization and add topN logic

be7ff49

Signed-off-by: Jooho Lee <[email protected]>

nirrozenbaum reviewed Aug 21, 2025

View reviewed changes

pkg/epp/scheduling/framework/plugins/picker/weighted_random_picker.go Outdated Show resolved Hide resolved

nirrozenbaum reviewed Aug 21, 2025

View reviewed changes

pkg/epp/scheduling/framework/plugins/picker/weighted_random_picker.go Outdated Show resolved Hide resolved

Jooho added 2 commits August 21, 2025 10:24

removed all the optimization parts and implemented only the core WRS …

f72bd77

…logic Signed-off-by: Jooho Lee <[email protected]>

Change to delegate selection to RandomPicker for uniform picking when…

41b64c5

… all weights are zero Signed-off-by: Jooho Lee <[email protected]>

nirrozenbaum reviewed Aug 24, 2025

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 24, 2025

yafengio reviewed Aug 25, 2025

View reviewed changes

nirrozenbaum self-assigned this Aug 25, 2025

Jooho changed the title ~~Add WeightedRandomPicker with score normalization~~ Add WeightedRandomPicker Aug 26, 2025

follw up comments

257fdb0

Signed-off-by: Jooho Lee <[email protected]>

Add WeightedRandomPicker #1412

Are you sure you want to change the base?

Add WeightedRandomPicker #1412

Conversation

Jooho commented Aug 19, 2025

Uh oh!

netlify bot commented Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

linux-foundation-easycla bot commented Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Aug 19, 2025

Uh oh!

k8s-ci-robot commented Aug 19, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nirrozenbaum commented Aug 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Aug 27, 2025

Uh oh!

Uh oh!

netlify bot commented Aug 19, 2025 •

edited

Loading

linux-foundation-easycla bot commented Aug 19, 2025 •

edited

Loading