handle picking multiple destinations in scheduling layer #1059

nirrozenbaum · 2025-06-24T19:59:43Z

This PR implements the multiple destination handling in the scheduling layer, as defined in the EPP protocol:
https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/main/docs/proposals/004-endpoint-picker-protocol#destination-endpoint

please pay attention that this PR updates only the scheduling layer.
as a follow up, we should update the request-control layer and the handlers to use the multiple returned endpoints according to the protocol.
In order to keep this PR tightly scoped, the director keeps using a single targetPod to keep the current functionality.

k8s-ci-robot · 2025-06-24T19:59:50Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nirrozenbaum

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [nirrozenbaum]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

netlify · 2025-06-24T20:00:32Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`951b5af`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/686a16a377bb4800087e7b43
😎 Deploy Preview	https://deploy-preview-1059--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

nirrozenbaum · 2025-06-30T12:57:06Z

cc @kfswain @ahg-g

Signed-off-by: Nir Rozenbaum <[email protected]>

kfswain · 2025-07-08T20:28:07Z

pkg/epp/requestcontrol/director.go

@@ -238,7 +238,8 @@ func (d *Director) prepareRequest(ctx context.Context, reqCtx *handlers.RequestC
 		return reqCtx, errutil.Error{Code: errutil.Internal, Msg: "results must be greater than zero"}
 	}
 	// primary profile is used to set destination
-	targetPod := result.ProfileResults[result.PrimaryProfileName].TargetPod.GetPod()
+	// TODO should use multiple destinations according to epp protocol. current code assumes a single target


Can we bind an issue to this TODO?

we do have #414 which covers this point current PR implements first half (scheduling part) and next PR should mark the issue as completed (handle request control + hndlers).
a new issue will be a duplicate

kfswain · 2025-07-08T20:45:48Z

pkg/epp/scheduling/framework/plugins/picker/random_picker.go

-// RandomPickerFactory defines the factory function for RandomPicker.
-func RandomPickerFactory(name string, _ json.RawMessage, _ plugins.Handle) (plugins.Plugin, error) {
-	return NewRandomPicker().WithName(name), nil
+func RandomPickerFactory(name string, rawParameters json.RawMessage, _ plugins.Handle) (plugins.Plugin, error) {


I'm wondering if we want to parse the json before here and just push everything into an unstructured map.

It's out of scope of this PR, definitely, and I do understand the issue that a complex object would still need its type inferred... But it feels strange to do json parsing in every factory. Is there a reason that I missed that it was done this way?

same answer as in the other thread.. it was initially implemented this way and this is yet another point that potentially can be improved in config api.

kfswain · 2025-07-08T20:47:16Z

/lgtm
/hold

Holding if we want to spin up an issue to track the TODO work, thanks Nir!

nirrozenbaum · 2025-07-08T21:17:13Z

/unhold

will take care of next part using issue #414 (current PR was focused on scheduling and left the behavior of other layers as is)

kfswain · 2025-07-08T22:06:26Z

SGTM! Thanks Nir!

…sigs#1059) * implement multiple destination as the output of the scheduler Signed-off-by: Nir Rozenbaum <[email protected]> * updated max score picker unit tests to cover multiple pods Signed-off-by: Nir Rozenbaum <[email protected]> * imports Signed-off-by: Nir Rozenbaum <[email protected]> * unit-test fix Signed-off-by: Nir Rozenbaum <[email protected]> --------- Signed-off-by: Nir Rozenbaum <[email protected]>

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jun 24, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 24, 2025

k8s-ci-robot requested review from danehans and liu-cong June 24, 2025 19:59

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jun 24, 2025

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 25, 2025

nirrozenbaum force-pushed the multiple-dest branch from 0668a03 to bd6c559 Compare June 25, 2025 09:26

k8s-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Jun 25, 2025

nirrozenbaum force-pushed the multiple-dest branch from bd6c559 to 58cd3c4 Compare June 30, 2025 09:34

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 30, 2025

nirrozenbaum changed the title ~~[WIP] handle picking multiple destinations~~ handle picking multiple destinations Jun 30, 2025

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 30, 2025

nirrozenbaum changed the title ~~handle picking multiple destinations~~ handle picking multiple destinations in scheduling layer Jun 30, 2025

nirrozenbaum force-pushed the multiple-dest branch from 293bbec to 61c36ab Compare June 30, 2025 13:20

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 3, 2025

nirrozenbaum force-pushed the multiple-dest branch from 61c36ab to 73c9eaf Compare July 3, 2025 09:20

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 3, 2025

nirrozenbaum mentioned this pull request Jul 3, 2025

verify fallback to default values if parameters are not defined in config file #1111

Merged

nirrozenbaum force-pushed the multiple-dest branch from 8762ad7 to 44a5172 Compare July 3, 2025 19:06

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 5, 2025

nirrozenbaum added 4 commits July 6, 2025 08:51

implement multiple destination as the output of the scheduler

60c9d9d

Signed-off-by: Nir Rozenbaum <[email protected]>

updated max score picker unit tests to cover multiple pods

610d779

Signed-off-by: Nir Rozenbaum <[email protected]>

imports

d8e1d79

Signed-off-by: Nir Rozenbaum <[email protected]>

unit-test fix

951b5af

Signed-off-by: Nir Rozenbaum <[email protected]>

nirrozenbaum force-pushed the multiple-dest branch from 44a5172 to 951b5af Compare July 6, 2025 06:24

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 6, 2025

nirrozenbaum mentioned this pull request Jul 8, 2025

Add fallback logic to support multiple endpoints #1122

Open

kfswain reviewed Jul 8, 2025

View reviewed changes

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 8, 2025

k8s-ci-robot assigned kfswain Jul 8, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 8, 2025

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 8, 2025

k8s-ci-robot merged commit 91e3047 into kubernetes-sigs:main Jul 8, 2025
9 checks passed

nirrozenbaum deleted the multiple-dest branch July 9, 2025 05:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

handle picking multiple destinations in scheduling layer #1059

handle picking multiple destinations in scheduling layer #1059

Uh oh!

nirrozenbaum commented Jun 24, 2025 •

edited

Loading

Uh oh!

k8s-ci-robot commented Jun 24, 2025

Uh oh!

netlify bot commented Jun 24, 2025 •

edited

Loading

Uh oh!

nirrozenbaum commented Jun 30, 2025

Uh oh!

kfswain Jul 8, 2025

Uh oh!

nirrozenbaum Jul 8, 2025

Uh oh!

kfswain Jul 8, 2025

Uh oh!

nirrozenbaum Jul 8, 2025 •

edited

Loading

Uh oh!

kfswain commented Jul 8, 2025

Uh oh!

nirrozenbaum commented Jul 8, 2025

Uh oh!

Uh oh!

kfswain commented Jul 8, 2025

Uh oh!

Uh oh!

handle picking multiple destinations in scheduling layer #1059

handle picking multiple destinations in scheduling layer #1059

Uh oh!

Conversation

nirrozenbaum commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Jun 24, 2025

Uh oh!

netlify bot commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

nirrozenbaum commented Jun 30, 2025

Uh oh!

kfswain Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

nirrozenbaum Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

kfswain Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

nirrozenbaum Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kfswain commented Jul 8, 2025

Uh oh!

nirrozenbaum commented Jul 8, 2025

Uh oh!

Uh oh!

kfswain commented Jul 8, 2025

Uh oh!

Uh oh!

nirrozenbaum commented Jun 24, 2025 •

edited

Loading

netlify bot commented Jun 24, 2025 •

edited

Loading

nirrozenbaum Jul 8, 2025 •

edited

Loading