AEP-8818: InPlace Update Mode #8818

omerap12 · 2025-11-15T13:55:39Z

What type of PR is this?

/kind documentation

What this PR does / why we need it:

AEP for #8720

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

NONE

Signed-off-by: Omer Aplatony <[email protected]>

k8s-ci-robot · 2025-11-15T13:55:42Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

k8s-ci-robot · 2025-11-15T13:55:47Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: omerap12
Once this PR has been reviewed and has the lgtm label, please assign gjtempleton for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

vertical-pod-autoscaler/enhancements/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: Omer Aplatony <[email protected]>

omerap12 · 2025-11-16T11:55:37Z

/cc @adrianmoisey @maxcao13

Signed-off-by: Omer Aplatony <[email protected]>

omerap12 · 2025-11-20T09:51:12Z

/kind api-review

k8s-ci-robot · 2025-11-20T09:51:15Z

@omerap12: The label(s) kind/api-review cannot be applied, because the repository doesn't have them.

In response to this:

/kind api-review

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

omerap12 · 2025-11-20T09:51:38Z

/label kind/api-review

k8s-ci-robot · 2025-11-20T09:51:41Z

@omerap12: The label(s) /label kind/api-review cannot be applied. These labels are supported: api-review, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, team/katacoda, refactor, ci-short, ci-extended, ci-full. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

In response to this:

/label kind/api-review

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

omerap12 · 2025-11-20T21:12:23Z

/label api-review

iamzili · 2025-12-05T12:30:42Z

/lgtm

@omerap12 If you have a draft PR that makes sense to review, then let me know Omer.

k8s-ci-robot · 2025-12-05T12:30:47Z

@iamzili: changing LGTM is restricted to collaborators

In response to this:

/lgtm

@omerap12 If you have a draft PR that makes sense to review, then let me know Omer.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

maxcao13 · 2025-12-05T20:20:02Z

I'll take a look at this next week @omerap12 sorry for the delay 🥲

Signed-off-by: Omer Aplatony <[email protected]>

adrianmoisey · 2025-12-07T14:39:02Z

vertical-pod-autoscaler/enhancements/8818-in-place-only/README.md

+	    klog.V(4).InfoS("Can't in-place update pod, waiting for next loop", "pod", klog.KObj(pod))
+	return utils.InPlaceDeferred


Minor nit, are these supposed to be indented the same level?

No sure, I'll try to fmt soon

adrianmoisey · 2025-12-07T14:39:32Z

vertical-pod-autoscaler/enhancements/8818-in-place-only/README.md

+
+
+Retry is handled entirely by the Kubelet based on pod conditions:
+- `PodResizePending` (reason: `Deferred`) - Kubelet will retry automatically, , VPA continues to defer.


Suggested change

- `PodResizePending` (reason: `Deferred`) - Kubelet will retry automatically, , VPA continues to defer.

- `PodResizePending` (reason: `Deferred`) - Kubelet will retry automatically, VPA continues to defer.

adrianmoisey · 2025-12-07T14:40:16Z

vertical-pod-autoscaler/enhancements/8818-in-place-only/README.md

+
+### Behavior when Feature Gate is Disabled
+
+- When `InPlace` feature gate is disabled and a VPA is configured with `UpdateMode: InPlace`, the updater will skip processing that VPA entirely (not fall back to eviction).


Just want to check: it won't evict and it won't in-place update?

Also, what does the admission-controller do when the feature gate is disabled but a pod is set to InPlace?

The admission controller will deny the the request ref

Just want to check: it won't evict and it won't in-place update?

That’s what I assumed, because if someone wants to use in-place mode only, it likely means the workload can’t be evicted. In that case, I think the correct action is to do nothing.

Well, what if someone does this:

Upgrades to this version of VPA and enables the feature gate

Uses the InPlace mode on a VPA

Disables the feature gate

Deletes a Pod from the VPA pointing at InPlace

Does the admission-controller:

Set the resources as per the recommendation (as if the VPA was in "Initial" mode)

Ignore the pod (as if the VPA was in "Off" mode)

Something else..

TBH I didn't test it. but it should be 1

Just checked, we set the resources as per the recommendation.

Cool, that's worth clarifying here

adrianmoisey · 2025-12-07T18:20:30Z

vertical-pod-autoscaler/enhancements/8818-in-place-only/README.md

+
+- Guarantee that all updates will eventually succeed (node capacity constraints may prevent this)
+- Provide mechanisms to automatically increase node capacity to accommodate updates
+- Change the behavior of existing update modes (Auto, Recreate, InPlaceOrRecreate)


Suggested change

- Change the behavior of existing update modes (Auto, Recreate, InPlaceOrRecreate)

- Change the behavior of existing update modes (Off, Recreate, InPlaceOrRecreate)

Not sure if adding Auto makes sense, since it's deprecated? I'll let you decide.

maxcao13 · 2025-12-08T18:10:04Z

vertical-pod-autoscaler/enhancements/8818-in-place-only/README.md

+- Apply recommendations during pod admission (like all other modes)
+- Attempt in-place updates for running pods under the same conditions as `InPlaceOrRecreate`
+- Never add pods to `podsForEviction` if in-place updates fail
+- Continuously retry failed in-place update


Should we have a backoff policy for retrying, or do we think linear retry is sufficient if we keep failing?

maxcao13 · 2025-12-08T18:14:07Z

vertical-pod-autoscaler/enhancements/8818-in-place-only/README.md

+- Allow VPA to eventually apply updates when cluster conditions improve
+- Respect the existing in-place update infrastructure from AEP-4016
+
+## Non-Goals


Do we think we should have some small note that this update mode is subject to the behavior of the inplacepodverticalscaling gate, such that it's possible (but improbable) that a resize can cause an OOMkill during a memory limit downsize?

Though I don't actually know the probability of this happening if a limit gets resized close to the usage, I think it may be useful to callout since we emphasize that brief disruptions are unnacceptable.

I think to mitigate risk here we may want to recommend that if you absolutely cannot tolerate disruption (i.e. unintended OOMkill), then you can either:

disallow memory limits for your no disruption container

if you must allow VPA to set memory limits, then you should configure the VPA to generate more generous/conservative memory limit recommendations as a safety buffer.

^ Though this may or may not be better for our docs, instead of getting into it in the AEP here.

Thoughts? cc @adrianmoisey

I think you're right.
I was thinking a similar though on the "Provide a truly non-disruptive VPA update mode that never evicts pods" goal.

I think it may be worth softening the language in the AEP (since we can't make guarantees that resizes are non-disruptive)

I also agree that most of what you suggested may be good for the docs

Related: #8805

[WIP] In Place Only VPA

10a2270

Signed-off-by: Omer Aplatony <[email protected]>

k8s-ci-robot added the do-not-merge/needs-area label Nov 15, 2025

k8s-ci-robot requested review from adrianmoisey and kwiesmueller November 15, 2025 13:55

k8s-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. area/vertical-pod-autoscaler and removed do-not-merge/needs-area labels Nov 15, 2025

removed spaces

29e7843

Signed-off-by: Omer Aplatony <[email protected]>

omerap12 changed the title ~~[WIP] In Place Only VPA~~ AEP-8720: InPlace Update Mode Nov 15, 2025

omerap12 changed the title ~~AEP-8720: InPlace Update Mode~~ AEP-8818: InPlace Update Mode Nov 15, 2025

omerap12 added 3 commits November 15, 2025 15:57

Fixed AEP number

c6d27c0

Signed-off-by: Omer Aplatony <[email protected]>

Fixed function

26456cb

Signed-off-by: Omer Aplatony <[email protected]>

fmt

e76adef

Signed-off-by: Omer Aplatony <[email protected]>

omerap12 marked this pull request as ready for review November 16, 2025 11:55

k8s-ci-robot requested a review from maxcao13 November 16, 2025 11:55

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 16, 2025

Use kubelet as the for retry

19e34c0

Signed-off-by: Omer Aplatony <[email protected]>

k8s-ci-robot added the api-review Categorizes an issue or PR as actively needing an API review. label Nov 20, 2025

github-project-automation bot added this to API Reviews Nov 20, 2025

Update AEP

f46365c

Signed-off-by: Omer Aplatony <[email protected]>

adrianmoisey reviewed Dec 7, 2025

View reviewed changes

maxcao13 reviewed Dec 8, 2025

View reviewed changes

		klog.V(4).InfoS("Can't in-place update pod, waiting for next loop", "pod", klog.KObj(pod))
		return utils.InPlaceDeferred



		Retry is handled entirely by the Kubelet based on pod conditions:
		- `PodResizePending` (reason: `Deferred`) - Kubelet will retry automatically, , VPA continues to defer.


		### Behavior when Feature Gate is Disabled

		- When `InPlace` feature gate is disabled and a VPA is configured with `UpdateMode: InPlace`, the updater will skip processing that VPA entirely (not fall back to eviction).

	- Change the behavior of existing update modes (Auto, Recreate, InPlaceOrRecreate)
	- Change the behavior of existing update modes (Off, Recreate, InPlaceOrRecreate)

AEP-8818: InPlace Update Mode #8818

Are you sure you want to change the base?

AEP-8818: InPlace Update Mode #8818

Conversation

omerap12 commented Nov 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Uh oh!

k8s-ci-robot commented Nov 15, 2025

Uh oh!

k8s-ci-robot commented Nov 15, 2025

Uh oh!

omerap12 commented Nov 16, 2025

Uh oh!

omerap12 commented Nov 20, 2025

Uh oh!

k8s-ci-robot commented Nov 20, 2025

Uh oh!

omerap12 commented Nov 20, 2025

Uh oh!

k8s-ci-robot commented Nov 20, 2025

Uh oh!

omerap12 commented Nov 20, 2025

Uh oh!

iamzili commented Dec 5, 2025

Uh oh!

k8s-ci-robot commented Dec 5, 2025

Uh oh!

maxcao13 commented Dec 5, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

omerap12 commented Nov 15, 2025 •

edited

Loading