kserve: move platform resources to kserve namespace by danish9039 · Pull Request #3515 · kubeflow/community-distribution

danish9039 · 2026-06-29T05:12:57Z

Summary of Changes

This follow-up moves KServe-owned platform resources from kubeflow to the
upstream kserve namespace.

Install the standard upstream KServe bundle in kserve, including
controllers, webhooks, certificates, namespaced RBAC, and
inferenceservice-config.
Move the KServe Models Web Application and its NetworkPolicy to kserve
while preserving routing through kubeflow/kubeflow-gateway.
Keep user InferenceService resources and serving workloads in Profile
namespaces.
Keep direct Kustomize installation self-contained with Namespace/kserve;
add equivalent namespace ownership and restricted Pod Security labels to the
kubeflow-namespaces foundation chart.
Disable Istio sidecar injection explicitly on KServe controller pod templates
while retaining explicit injection for the Models Web Application.
Preserve Kubeflow aggregate roles and disable the restricted-Pod-Security
incompatible local model node agent.
Update installation tests, workflow diagnostics, chart parity, synchronized
version handling, and upgrade cleanup guidance.

Ownership Boundaries

The component Kustomization owns Namespace/kserve for direct Kustomize
installation. Helm component charts do not create namespaces; the
kubeflow-namespaces foundation chart owns Namespace/kserve for Helm
installation.

This pull request only aligns the existing Models Web Application chart with
the relocation. Generated KServe control-plane charts and their synchronization
pipeline remain separate future Helm work.

Upgrade Notes

Controllers in the old and new namespaces use different leader-election
Leases, so the old kubeflow controllers must be scaled to zero before the new
controllers start. The old Models Web Application route and namespaced
resources must also be removed to prevent duplicate routing and stale
resources.

The documented cleanup does not delete KServe CRDs, user InferenceService
objects, Profile namespaces, or model-serving workloads. Existing Helm
installations must uninstall the Models Web Application release from
kubeflow and install a new release in kserve, because Helm cannot move a
release between namespaces.

Related Issues

Follow-up to #3505 and the namespace recommendation in the pull request
discussion.

Contributor Checklist

I have tested these changes with Kustomize. See Installation Prerequisites.
All commits are signed off to satisfy the DCO check.
I have considered adding my company to the adopters page to support Kubeflow and help the community, since I expect help from the community for my issue (see 1 and 2).

Verification

python3 tests/kserve_manifest_test.py
pre-commit hooks for every changed or added file
git diff --check upstream/master...HEAD
shell syntax validation for the affected test and synchronization scripts
Kustomize rendering for the KServe control plane, Models Web Application,
shared namespaces, and complete example
strict kubeconform validation of both KServe renderings
Helm linting for the namespace foundation and Models Web Application charts
complete tests/helm_kustomize_compare_all.sh all parity matrix
independent adversarial review of RBAC, webhooks, certificates, routing,
NetworkPolicies, Pod Security, Istio injection, upgrade cleanup, and Helm
ownership

Not run locally: live-cluster admission, runtime, and in-place migration tests
because no active Kubernetes context was available.

Related Issues

Tracking issue:
#3561

github-actions · 2026-06-29T05:13:06Z

Welcome to the Kubeflow Community Distribution Repository

Thanks for opening your first PR. Your contribution means a lot to the Kubeflow community.

Before making more PRs:
Please ensure your PR follows our Contributing Guide.
Please also be aware that many components are synchronized from upstream via the scripts in /scripts.
So in some cases you have to fix the problem in the upstream repositories first, but you can use a PR against kubeflow/community-distribution to test the platform integration.

Community Resources:

CNCF Slack channels: https://www.kubeflow.org/docs/about/community/#kubeflow-slack-channels
Community meetings: https://www.kubeflow.org/docs/about/community/

Thanks again for helping to improve Kubeflow.

google-oss-prow · 2026-06-29T05:13:13Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kimwnasptd for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copilot

Pull request overview

This pull request moves KServe control-plane resources from the kubeflow namespace to the upstream kserve namespace while preserving Kubeflow-facing behavior (notably keeping the KServe Models Web Application in kubeflow).

Changes:

Add and manage Namespace/kserve, and update the KServe installation kustomization to install control-plane resources there (including webhook NetworkPolicy coverage).
Update installation and continuous integration scripts to wait for and collect diagnostics from the kserve namespace.
Document upgrade cleanup steps to remove legacy kubeflow-scoped KServe control-plane resources.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
`tests/PSS_enable.sh`	Includes `kserve` in Pod Security Standards enforcement labeling.
`tests/kserve_install.sh`	Updates waits and diagnostics to reflect KServe control-plane deployment in `kserve`.
`README.md`	Documents the new namespace split and adds upgrade cleanup instructions.
`common/kubeflow-namespace/base/kubeflow/kustomization.yaml`	Stops applying the old `kubeflow`-scoped KServe NetworkPolicy.
`common/kubeflow-namespace/base/kubeflow/kserve.yaml`	Removes the old `kubeflow`-scoped KServe webhook NetworkPolicy manifest.
`applications/kserve/UPGRADE.md`	Adds upgrade guidance for cleaning up legacy `kubeflow`-scoped KServe control-plane resources.
`applications/kserve/kserve/namespace.yaml`	Introduces managed `Namespace/kserve` with labels for control-plane and Pod Security Standards.
`applications/kserve/kserve/kustomization.yaml`	Installs upstream KServe resources into `kserve`, adds aggregated roles and `kserve`-scoped webhook NetworkPolicy, and updates local patches accordingly.
`applications/kserve/kserve/kserve-networkpolicy.yaml`	Adds `kserve`-scoped NetworkPolicy allowing admission webhook ingress on port 9443.
`applications/kserve/kserve/aggregated-roles.yaml`	Expands aggregated Kubeflow KServe roles to include additional KServe resources.
`.github/workflows/kserve_test.yaml`	Adds `common/kubeflow-namespace/**` trigger and includes `kserve` namespace in diagnostics capture.
`.github/workflows/kserve_models_web_application_test.yaml`	Adds `common/kubeflow-namespace/**` trigger.
`.github/workflows/full_kubeflow_integration_test.yaml`	Includes `kserve` namespace in diagnostics capture.

danish9039 · 2026-07-26T10:53:09Z

/retest

danish9039 · 2026-07-26T11:47:45Z

/retest

danish9039 · 2026-07-27T07:24:16Z

@juliusvonkohout for review

Copilot

Pull request overview

Copilot reviewed 30 out of 31 changed files in this pull request and generated 3 comments.

danish9039 · 2026-07-30T07:01:05Z

/retest

Move the KServe control plane and Models Web Application into the kserve namespace while preserving user InferenceService resources in Profile namespaces and routing through the Kubeflow gateway. Align namespace foundation ownership, workload-specific Istio injection, NetworkPolicies, the existing Models Web Application chart, synchronization behavior, installation tests, and upgrade cleanup guidance. Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

The manifest test asserted properties that the live KinD run in tests/kserve_test.sh already proves, and asserted documentation prose and shell source text that no rendered resource depends on. Remove the assertions covered by the cluster run: the Models Web Application route, which Test 3 proves by requesting /kserve-endpoints, and the inferenceservice-config gateway and path template, which Test 2a proves by path-based routing. Remove the assertions on prose and source text. The root README upgrade steps are version-specific and are rewritten each release, so pinning their wording fails an unrelated release-notes edit. The kustomization comment assertions only prevent removed comments from being retyped. Keep the checks a green cluster run cannot make: resources left behind in kubeflow, sidecar injection selecting the Models Web Application through a pod label rather than an annotation, the storage initializer security context that no restricted namespace exercises in CI, namespace fields on cluster-scoped resources that the API server silently drops, the disabled local model node agent whose rejected pods create no pods to wait for, and the webhook certificate and authority references for the llmisvc and localmodel chains that no test calls. Document on each remaining test why the cluster run cannot catch it. Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

…n test The installation section still told operators the Models Web Application remains in the kubeflow namespace, which contradicts both the rendering and the upgrade guidance in the same document. State that the control plane and the Models Web Application are both in kserve, that user InferenceService resources stay in the profile namespaces, and that the Models Web Application is still reached through the Kubeflow gateway. Assert the exact namespace of every namespaced resource rather than the absence of the old one. A resource that declares no namespace is applied into whichever namespace the caller is using, which is default for the installation scripts, so a check for "not kubeflow" stayed green for a resource that had silently lost its namespace. The check now covers the control plane and the Models Web Application through one shared rule for cluster scope, which the cluster scope test reuses. Watch applications/kserve/kserve in the comparison workflow. That rendering became an input to the namespace parity check, but the workflow did not watch it, so a change to it alone would skip the comparison. Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

danish9039 · 2026-08-05T20:03:07Z

/retest

danish9039 · 2026-08-05T20:56:03Z

@juliusvonkohout Addressed the Copilot review comments

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

danish9039 · 2026-08-10T05:11:35Z

@juliusvonkohout ready to merge

Copilot

Pull request overview

Copilot reviewed 31 out of 32 changed files in this pull request and generated no new comments.

Suppressed comments (3)

tests/kserve_manifest_test.py:229

This filtering only validates resources that already have cert-manager.io/inject-ca-from; self.assertTrue(annotated_resources) proves only that one annotation remains. Removing the annotation from any individual conversion CRD or webhook configuration therefore leaves this test green, although the test documentation states that it guards every certificate chain. Compare the annotated resource identities with the expected set, or assert the annotation while traversing every webhook-backed resource.
tests/kserve_manifest_test.py:223
all(...) is true for an empty dnsNames list. If a future synchronization removes the DNS names from the local-model or LLM certificate, this regression test remains green even though the certificate cannot match its webhook service. Assert that the list is nonempty before checking every suffix.

This issue also appears on line 225 of the same file.
tests/kserve_manifest_test.py:151

The Profile controller Pod Security overlay currently enforces baseline, not restricted (applications/dashboard/upstream/profile-controller/overlays/kubeflow-pss/namespace-labels-kubeflow-pss.yaml:19-20). This security-policy statement is therefore inaccurate; describe restricted enforcement as an installation choice rather than the repository default.

juliusvonkohout · 2026-08-10T14:41:21Z

+          drop:
+          - ALL
+        runAsNonRoot: true
+        runAsUser: 1000


Suggested change

runAsUser: 1000

christian-heusel · 2026-08-10T15:52:27Z

+## Upgrade Cleanup
+
+When upgrading from a version that installed KServe control-plane resources in
+the `kubeflow` namespace, plan for a short control-plane interruption. Do not
+delete KServe CRDs, user `InferenceService` objects, or profile namespaces.
+
+The standard upstream bundle also uses the current upstream workload labels.
+Update custom monitoring and policies that select the legacy `app: kserve` pod
+label to select the `control-plane` label instead.
+
+First remove the unused cluster-scoped authorization for the disabled local
+model node agent:
+
+```sh
+kubectl delete --ignore-not-found \
+  clusterrolebinding/kserve-localmodelnode-agent-rolebinding \
+  clusterrole/kserve-localmodelnode-agent-role
+```
+
+Before applying the new manifests, scale the old controllers to zero.
+Controllers in different namespaces use different leader-election Leases and
+must not run concurrently:
+
+```sh
+kubectl scale -n kubeflow --replicas=0 \
+  deployment/kserve-controller-manager \
+  deployment/kserve-localmodel-controller-manager \
+  deployment/llmisvc-controller-manager
+```
+
+Remove the old Models Web Application route before applying the new manifests
+so the old and new VirtualServices do not claim `/kserve-endpoints/`
+concurrently. Then remove the remaining old namespaced resources:
+
+```sh
+kubectl delete -n kubeflow --ignore-not-found \
+  virtualservice.networking.istio.io/kserve-models-web-application
+kubectl delete -n kubeflow --ignore-not-found \
+  serviceaccount/kserve-models-web-application \
+  configmap/kserve-models-web-application-config \
+  service/kserve-models-web-application \
+  deployment/kserve-models-web-application \
+  authorizationpolicy.security.istio.io/kserve-models-web-application \
+  networkpolicy.networking.k8s.io/kserve-models-web-application
+```
+
+Apply the new release manifests using the normal upgrade procedure. Then verify
+the new KServe controllers, Models Web Application, certificates, and webhook
+endpoints in the `kserve` namespace:
+
+```sh
+kubectl rollout status -n kserve --timeout=120s \
+  deployment/kserve-controller-manager \
+  deployment/kserve-localmodel-controller-manager \
+  deployment/llmisvc-controller-manager \
+  deployment/kserve-models-web-application
+kubectl wait -n kserve --for=condition=Ready --timeout=120s \
+  certificate.cert-manager.io/serving-cert \
+  certificate.cert-manager.io/llmisvc-serving-cert \
+  certificate.cert-manager.io/localmodel-serving-cert
+kubectl get endpoints -n kserve \
+  kserve-webhook-server-service \
+  llmisvc-webhook-server-service \
+  localmodel-webhook-server-service
+```
+
+After the new control plane is ready, delete the old `kubeflow`-scoped
+resources:
+
+```sh
+kubectl delete -n kubeflow --ignore-not-found \
+  serviceaccount/kserve-controller-manager \
+  serviceaccount/kserve-localmodel-controller-manager \
+  serviceaccount/kserve-localmodelnode-agent \
+  serviceaccount/llmisvc-controller-manager \
+  networkpolicy.networking.k8s.io/kserve \
+  role.rbac.authorization.k8s.io/kserve-leader-election-role \
+  role.rbac.authorization.k8s.io/llmisvc-leader-election-role \
+  rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding \
+  rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding \
+  configmap/inferenceservice-config \
+  secret/kserve-webhook-server-secret \
+  secret/kserve-webhook-server-cert \
+  secret/llmisvc-webhook-server-cert \
+  secret/localmodel-webhook-server-cert \
+  service/kserve-controller-manager-metrics-service \
+  service/kserve-controller-manager-service \
+  service/kserve-webhook-server-service \
+  service/llmisvc-controller-manager-service \
+  service/llmisvc-webhook-server-service \
+  service/localmodel-webhook-server-service \
+  deployment.apps/kserve-controller-manager \
+  deployment.apps/kserve-localmodel-controller-manager \
+  deployment.apps/llmisvc-controller-manager \
+  daemonset.apps/kserve-localmodelnode-agent \
+  lease.coordination.k8s.io/kserve-controller-manager-leader-lock \
+  lease.coordination.k8s.io/llminferenceservice-kserve-controller-manager \
+  certificate.cert-manager.io/serving-cert \
+  certificate.cert-manager.io/llmisvc-serving-cert \
+  certificate.cert-manager.io/localmodel-serving-cert \
+  issuer.cert-manager.io/selfsigned-issuer
+```


Why are these notes duplicated here from the main README.md? 🤔 I think they can be dropped here.

Also I think such notes should always be versioned, otherwise it's unsure on which upgrades they apply 😅

Yes please remove duplicates

christian-heusel · 2026-08-10T15:57:01Z

Where are those roles coming from and why do we need to introduce them? 🤔 AFAIU the upstream overlay already creates rules like this in the kubeflow overlay 😅

The kubeflow overlay is not that well maintained. This is why we wanted to switch to the regular one, also in light of the helm efforts.

juliusvonkohout · 2026-08-10T16:40:27Z

Please check #3515 (review)

juliusvonkohout · 2026-08-10T16:44:25Z

Please rebase after #3576

Copilot AI review requested due to automatic review settings June 29, 2026 05:12

google-oss-prow Bot added the do-not-merge/work-in-progress label Jun 29, 2026

google-oss-prow Bot added the size/L label Jun 29, 2026

google-oss-prow Bot requested review from TobiasGoerke and tarekabouzeid June 29, 2026 05:13

Copilot started reviewing on behalf of danish9039 June 29, 2026 05:13 View session

Copilot AI reviewed Jun 29, 2026

View reviewed changes

Comment thread applications/kserve/kserve/kustomization.yaml

danish9039 force-pushed the pr-3505-kserve-namespace-followup branch 2 times, most recently from a41eace to f635136 Compare July 20, 2026 13:36

google-oss-prow Bot added size/XL and removed size/L labels Jul 20, 2026

danish9039 force-pushed the pr-3505-kserve-namespace-followup branch 2 times, most recently from 6961ac0 to 48761fc Compare July 25, 2026 13:24

google-oss-prow Bot added size/XXL and removed size/XL labels Jul 25, 2026

danish9039 changed the title ~~kserve: move control plane to kserve namespace~~ kserve: move platform resources to kserve namespace Jul 25, 2026

danish9039 force-pushed the pr-3505-kserve-namespace-followup branch from 48761fc to 6b61122 Compare July 25, 2026 13:27

danish9039 marked this pull request as ready for review July 25, 2026 14:04

google-oss-prow Bot removed the do-not-merge/work-in-progress label Jul 25, 2026

google-oss-prow Bot requested review from juliusvonkohout and kimwnasptd July 25, 2026 14:04

danish9039 marked this pull request as draft July 25, 2026 14:04

google-oss-prow Bot added the do-not-merge/work-in-progress label Jul 25, 2026

danish9039 force-pushed the pr-3505-kserve-namespace-followup branch from 6b61122 to fb1b194 Compare July 25, 2026 15:36

danish9039 marked this pull request as ready for review July 27, 2026 07:22

google-oss-prow Bot removed the do-not-merge/work-in-progress label Jul 27, 2026

danish9039 mentioned this pull request Jul 27, 2026

KServe platform resources should be installed in the kserve namespace #3561

Open

9 tasks

juliusvonkohout requested a review from Copilot July 27, 2026 14:35

Copilot started reviewing on behalf of juliusvonkohout July 27, 2026 14:35 View session

Copilot AI reviewed Jul 27, 2026

View reviewed changes

Comment thread README.md Outdated

Comment thread tests/helm_kustomize_compare.sh

Comment thread tests/kserve_manifest_test.py Outdated

danish9039 added 3 commits August 6, 2026 00:44

danish9039 force-pushed the pr-3505-kserve-namespace-followup branch from d43a56d to 7e95d7d Compare August 5, 2026 19:14

juliusvonkohout requested a balanced review from Copilot August 6, 2026 17:59

Copilot AI reviewed Aug 6, 2026

Copilot started reviewing on behalf of juliusvonkohout August 6, 2026 18:37 View session

juliusvonkohout requested a balanced review from Copilot August 10, 2026 14:09

Copilot started reviewing on behalf of juliusvonkohout August 10, 2026 14:10 View session

Copilot AI reviewed Aug 10, 2026

View reviewed changes

juliusvonkohout reviewed Aug 10, 2026

View reviewed changes

juliusvonkohout mentioned this pull request Aug 10, 2026

Update kserve/kserve manifests to version v0.20.0 #3576

Merged

3 tasks

christian-heusel reviewed Aug 10, 2026

View reviewed changes

Uh oh!

Conversation

danish9039 commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of Changes

Ownership Boundaries

Upgrade Notes

Related Issues

Contributor Checklist

Verification

Related Issues

Uh oh!

github-actions Bot commented Jun 29, 2026

Uh oh!

google-oss-prow Bot commented Jun 29, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

danish9039 commented Jul 26, 2026

Uh oh!

danish9039 commented Jul 26, 2026

Uh oh!

danish9039 commented Jul 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danish9039 commented Jul 30, 2026

Uh oh!

danish9039 commented Aug 5, 2026

Uh oh!

danish9039 commented Aug 5, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

danish9039 commented Aug 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

juliusvonkohout Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

christian-heusel Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

christian-heusel Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout commented Aug 10, 2026

Uh oh!

juliusvonkohout commented Aug 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

danish9039 commented Jun 29, 2026 •

edited

Loading