KEP 831-Kubeflow-Helm-Support: Support Helm as an Alternative for Kustomize

[juliusvonkohout]

Helm KEP from @varodrig @chasecadet @juliusvonkohout
Fixed branch from #830

Placeholder: #831
Implementation: kubeflow/manifests#2730

[varodrig]

/lgtm

[andreyvelich]

I can see that you re-created the KEP.

/hold for community to review.
cc @kubeflow/wg-training-leads @kubeflow/wg-automl-leads @kubeflow/wg-manifests-leads @kubeflow/wg-data-leads @kubeflow/release-team @kubeflow/wg-pipeline-leads @kubeflow/wg-notebooks-leads @kubeflow/red-hat @kubeflow/wg-deployment-leads @kubeflow/kubeflow-steering-committee

[thesuperzapper]

Foundationally, this is still a discussion around if we have an "official" distribution or not.

This KEP proposes a single "mega" helm chart with all components inside, this is by definition, an opinionated "distribution" of Kubeflow Platform.

The community has discussed and rejected having official distributions in the past, for reasons that are still applicable:

1. The fundamental goal of the project is to make AI/ML tools for Kubernetes (Pipelines, Notebooks, Trainer, Feature Store, etc.)
2. Our strength is that our tools can run on any Kubernetes cluster, with no preference for any deployment method, cloud vendor, or anything else.
3. If we make opinionated deployment decisions, less people will be able to adopt our tools, or include them in downstream platforms.
4. There is strong evidence that it's not possible to create a successful "generic" distribution. See the fact that multiple successful distributions exist today, each with different opinionated approaches.
5. Those in the OWNERS file of an official distribution would have the ability to make decisions that affect the whole community. This likely means that 1-3 consulting/cloud/platform companies make decisions that benefit them, rather than the goal of making our tools the standard.

PS: I want to stress that your motivations about making Kubeflow easier to use are great, and I am sure some users would love a Kubeflow Distribution that looks like this. (In fact, there are at least 2 that I am aware of which are similar to your proposal already, so perhaps you can collaborate with them). However, it's critical to keep the project neutral and focused on the tools themselves.

---

Also, while it's clearly not the intention of this KEP, there is a separate discussion around if automatically-generated helm charts based on the existing component kustomize manifests would be useful for downstream distributions. But that would be a completely separate proposal.

[lburgazzoli]

there is a separate discussion around if automatically-generated helm charts based on the existing component kustomize manifests would be useful for downstream distributions.

@thesuperzapper is this discussion already happening somewhere ?

[shaikmoeed]

Thanks for this great feature—it's really going useful for us! As a suggestion, it would be awesome to include some migration documentation on transitioning from Kustomize to Helm in Goals. Also, any guidance on achieving a zero-downtime migration would be much appreciated. It will help users who are willing to utilise this.

Thanks again for all the great work!

[chasecadet]

Thanks for this great feature—it's really going useful for us! As a suggestion, it would be awesome to include some migration documentation on transitioning from Kustomize to Helm in Goals. Also, any guidance on achieving a zero-downtime migration would be much appreciated. It will help users who are willing to utilise this.

Thanks again for all the great work!

Hey @shaikmoeed thanks for the feedback! One thing that would really help with this initiative is if you would be willing to tell us how Helm would help you and what you are looking for from these Helm charts. Your insights would be super awesome!

[shaikmoeed]

Hey @shaikmoeed thanks for the feedback! One thing that would really help with this initiative is if you would be willing to tell us how Helm would help you and what you are looking for from these Helm charts. Your insights would be super awesome!

@chasecadet Thanks for asking! We maintain a customized local version of kubeflow/manifests (with patches like Istio, OAuth2 and other fixes), which makes our upgrades challenging. As we manage most of our k8s service using Helm with ArgoCD, this initiative would simplify our process by letting us enable only the needed components and manage our patches more cleanly—eliminating the need to maintain local copies and keeping our upgrade PRs much smaller and easier to review.

[chasecadet]

Hey @shaikmoeed thanks for the feedback! One thing that would really help with this initiative is if you would be willing to tell us how Helm would help you and what you are looking for from these Helm charts. Your insights would be super awesome!

@chasecadet Thanks for asking! We maintain a customized local version of kubeflow/manifests (with patches like Istio, OAuth2 and other fixes), which makes our upgrades challenging. As we manage most of our k8s service using Helm with ArgoCD, this initiative would simplify our process by letting us enable only the needed components and manage our patches more cleanly—eliminating the need to maintain local copies and keeping our upgrade PRs much smaller and easier to review.

Good to know! Also, we'd love to know about how you use Kubeflow and your use cases, but probably for a different medium. Feel free to hit me up on the CNCF slack (chasecadet) if you'd like to share. Also let's ensure your org is on the adopters list so you get credit for being bleeding edge!

[juliusvonkohout]

Foundationally, this is still a discussion around if we have an "official" distribution or not.

This KEP proposes a single "mega" helm chart with all components inside, this is by definition, an opinionated "distribution" of Kubeflow Platform.

Actually that is not the case. It is not a distribution, since they would still be the community manifests, not derived/deviated from it.

But you are right in the sense that we should have helm charts for the individual components and combine them as we combine the kustomize manifests of the individual components. In the end we have a wonderful heterogeneous userbase and they want both options . If possible we should just combine the smaller ones in a meta helm chart or so similar to the kustomize overlay https://github.com/kubeflow/manifests/blob/master/example/kustomization.yaml . In the end the goal is to have something helm based with a similar structure / goal as the current kustomize manifests. Please make constructive suggestions in the KEP how we can emphasize this more.

"Also, while it's clearly not the intention of this KEP, there is a separate discussion around if automatically-generated helm charts based on the existing component kustomize manifests would be useful for downstream distributions. But that would be a completely separate proposal.

Also here I have to object, this is not a separate discussion. Automatically-generated helm charts are within this KEP. But that is an implementation detail of the single source of truth requirement. CC @lburgazzoli

[juliusvonkohout]

Hey @shaikmoeed thanks for the feedback! One thing that would really help with this initiative is if you would be willing to tell us how Helm would help you and what you are looking for from these Helm charts. Your insights would be super awesome!

@chasecadet Thanks for asking! We maintain a customized local version of kubeflow/manifests (with patches like Istio, OAuth2 and other fixes), which makes our upgrades challenging. As we manage most of our k8s service using Helm with ArgoCD, this initiative would simplify our process by letting us enable only the needed components and manage our patches more cleanly—eliminating the need to maintain local copies and keeping our upgrade PRs much smaller and easier to review.

Hello, are you familiar with https://github.com/kubeflow/manifests?tab=readme-ov-file#upgrading-and-extending ? Maybe this can help you until the helm manifests are ready.

[juliusvonkohout]

@shaikmoeed https://github.com/kubeflow/community/blob/master/ADOPTERS.md is the file where you can add your company.

[andreyvelich]

Thank you for this effort team!
I left my initial comments.
I will review it again later this week.

[google-oss-prow]

New changes are detected. LGTM label has been removed.

[TheCodingSheikh]

Hey, i made a helm chart to install kubeflow.
Doesnt require modification, helm install will work out of the box, it is based on the manifets repo and argo.
Highly customizable, there is an example to expose with ingress and integrate keycloak.

Check it out and open to feedback

https://github.com/TheCodingSheikh/helm-charts/tree/main/charts/kubeflow

[chasecadet]

Hey, i made a helm chart to install kubeflow. Doesnt require modification, helm install will work out of the box, it is based on the manifets repo and argo. Highly customizable, there is an example to expose with ingress and integrate keycloak.

Check it out and open to feedback

https://github.com/TheCodingSheikh/helm-charts/tree/main/charts/kubeflow

This is awesome! I think what the KSC is working on determining is whether this is a "distribution" or something we move within the main Kubeflow repos. The boundaries of what is core Kubeflow vs. component manifests we support etc.. Right now you can in theory install everything with a single kustomize command, but using best in breed cloud services or other configurations requires effort. Then do you template the patches and how do you potentially tie that into a CI/CD system nicely or just use ArgoCD. This is a great contribution and we eagerly await the KSCs decision so we can figure out the best path forward. Please continue to give feedback here if you have it!

[juliusvonkohout]

From the @kubeflow/kubeflow-steering-committee meeting today:

Voted for by all meeting attending KSC members (4).
"Kubeflow Working Groups (WGs) are allowed, but not required, to provide standalone helm charts for their projects."

Not more not less, so do not interpret too much. This is not supposed to answer all questions and possibilities, but to allow us to move forward with the GSOC project and see how far we get.

@chasecadet

[rareddy]

Good news! Thank you for sharing the information.

[kromanow94]

Hi all,

Thanks for the detailed proposal and all the great work in pushing forward official Helm support for Kubeflow.

I'd like to propose and outline an alternative chart structure that addresses some of the limitations of Helm's dependency system while supporting modularity, reusability, and vendor extensibility.

---

🔍 Motivation

The current proposal describes a structure where each Kubeflow component maintains its own Helm Chart, and a top-level AIO (all-in-one) chart acts as an umbrella by declaring those components as dependencies. While this works well, it introduces a known limitation:

Helm only supports one level of dependencies.
This means that the AIO Helm Chart cannot itself be used as a dependency of another Helm Chart (e.g., by a vendor or platform team), because Helm does not allow recursive umbrella charts.
Reference: https://github.com/helm/helm/issues/2247

To support advanced use cases such as vendor charts or GitOps-first workflows, I propose an alternative that avoids Helm dependencies altogether. This structure introduces a clear separation between rendered manifests and reusable logic. It uses Git submodules for composition rather than relying on Helm’s dependency resolution, making the architecture more extensible and suitable for complex multi-layered setups.

---

🧩 Proposed Structure

Each major component (e.g., Pipelines, Central Dashboard, Notebooks) is maintained under their respective repository. In this structure, the templates/ directory is consistently split into two subdirectories:

• helpers/: contains shared template functions (e.g., _*.tpl), brought in via git submodules pointing to helm-toolkit

• manifests/: contains component-specific Kubernetes YAML templates rendered by Helm The file structure separates templated manifests from helper functions, and uses Git submodules to compose the AIO chart. Here's how it looks in practice:

Shared Templates Repository (helm-toolkit)

helm-toolkit/
└── templates/
    ├── _common.labels.tpl
    ├── _kubeflow.centraldashboard.tpl                 <- Helpers to be reused in controller, jupyter-web-app
    ├── _kubeflow.centraldashboard.controller.tpl
    ├── _kubeflow.centraldashboard.jupyter-web-app.tpl
    ├── _kubeflow.pipelines.tpl                        <- Helpers to be reused in mlpipeline, mlpipeline-cache, and so on
    ├── _kubeflow.pipelines.mlpipeline.tpl
    ├── _kubeflow.pipelines.mlpipeline-cache.tpl
    └── ...

🔁 This repo is included as a submodule in all components under templates/helpers/.

Component Chart Example (kubeflow/experimental/helm/{centraldashboard,notebooks})

kubeflow/
└── experimental/
    └── helm/
        ├── centraldashboard/
        │   ├── Chart.yaml
        │   └── templates/
        │       ├── helpers/             ← Git submodule → helm-toolkit/templates
        │       └── manifests/
        │           ├── deployment.yaml
        │           └── configmap.yaml
        └── notebooks/
            ├── Chart.yaml
            └── templates/
                ├── helpers/             ← Git submodule → helm-toolkit/templates
                └── manifests/
                    ├── controller
                    │   ├── deployment.yaml
                    │   └── configmap.yaml
                    └── jupyter-web-app
                        ├── deployment.yaml
                        └── configmap.yaml

# Alternative setup with one Helm Chart for all components under this repo

kubeflow/
└── experimental/
    └── helm/
        ├── Chart.yaml
        └── templates/
            ├── helpers/                  <- Git submodule → helm-toolkit/templates
            └── manifests/
                ├── centraldashboard/
                │    ├── deployment.yaml
                │    └── configmap.yaml
                └── notebooks
                     ├── controller
                     │   ├── deployment.yaml
                     │   └── configmap.yaml
                     └── jupyter-web-app
                         ├── deployment.yaml
                         └── configmap.yaml

Component Chart Example (pipelines/experimental/helm)

pipelines/
└── experimental/
    └── helm/
        ├── Chart.yaml
        └── templates/
            ├── helpers/                  <- Git submodule → helm-toolkit/templates
            └── manifests/
                ├── mlpipeline-api
                │   ├── deployment.yaml
                │   └── configmap.yaml
                ├── mlpipeline-cache
                │   ├── deployment.yaml
                │   └── configmap.yaml
                ├── metadata-writer
                │   ├── deployment.yaml
                │   └── configmap.yaml
                └── scheduledworkflow
                    ├── deployment.yaml
                    └── configmap.yaml

AIO Helm Chart (manifests/experimental/helm)

manifests/
└── experimental/
    └── helm/
        ├── Chart.yaml                  ← AIO Chart - does NOT use dependencies in Chart.yaml
        └── templates/
            ├── helpers/                ← Git submodule → helm-toolkit/templates
            ├── centraldashboard/       ← Git submodule → kubeflow/.../centraldashboard/templates/manifests
            ├── pipelines/              ← Git submodule → pipelines/.../templates/manifests
            ├── istio-integration/      ← Templated manifests like Gateway, cluster-jwks-proxy, RequestAuthentication. This could also be a part of the helm-toolkit.
            │   ├── cluster-jwks-proxy/
            │   ├── external-auth/
            │   ├── istio-m2m/
            │   └── gateway.yaml
            └── dex-integration/        ← Templated manifests like VirtualService. This could also be a part of the helm-toolkit.
                └── virtualservice.yaml

---

✅ Benefits

Structural Scalability

• Avoids Helm's recursive dependency limitation — the AIO chart does not define dependencies, allowing it to be included in vendor charts.

• Modular and composable architecture — components are deployable standalone, and can be composed without Helm’s dependency system.

• Vendor flexibility — enables easy extension or composition without hitting Helm’s single-level dependency limit.

Maintainability & Reusability

• Clear separation of concerns — reusable template logic (helpers/) is decoupled from component manifests.

• Centralized helper management — shared _*.tpl functions are maintained in helm-toolkit, reducing duplication.

• DRY by design — all charts pull from the same set of helpers via submodules, ensuring consistency.

Compatibility & Workflow

• GitOps-friendly — works seamlessly with tools like ArgoCD and Flux due to the flat structure and explicit submodules.

---

🧠 Origin and Reference

This structure comes from my experience developing and maintaining a working "Mega Kubeflow Helm Chart", available here:
➡️ https://github.com/kromanow94/kubeflow-manifests/releases/tag/kubeflow-0.5.0

It also draws inspiration from other large-scale Helm projects such as openstack-helm, which has a similar modular layout. In openstack-helm:

• Common logic is extracted into a dedicated chart called helm-toolkit,

• Each service (e.g., Neutron) is defined as a separate Helm chart,

• AIO-style orchestration is done in the openstack chart.

You can explore more here:

• openstack-helm (all-in-one chart)

• helm-toolkit (shared helpers)

• neutron (single component example)

🚀 Final Thoughts

If one-level nesting is acceptable, the current direction is sound. But if we want to support higher-level reuse and allow vendors/platforms to compose Kubeflow flexibly, this alternative provides a scalable, modular, and Helm-compliant solution.

Would the community be open to exploring this model as a prototype or RFC-style implementation? I'm happy to contribute a working example or engage in further discussion!

[juliusvonkohout]

@kromanow94 do you want to guide @kunal-511 regarding "AIO Helm Chart (manifests/experimental/helm)"

he started here https://github.com/kubeflow/manifests/pull/3154/files

[juliusvonkohout]

The GSOC project with Helm is ongoing, so i plan to merge this proposal soon.
/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: juliusvonkohout

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [juliusvonkohout]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[andreyvelich]

@juliusvonkohout Before merging this KEP, we need to ensure that we update the proposal.
For example, we will only maintain Helm Charts for individual projects.
/hold

[juliusvonkohout]

@juliusvonkohout Before merging this KEP, we need to ensure that we update the proposal. For example, we will only maintain Helm Charts for individual projects. /hold

can you make direct code suggestions then ?

[andreyvelich]

@juliusvonkohout I left my suggestions.
@thesuperzapper @kimwnasptd Do you want to leave your comments as well ?

[andreyvelich]

This KEP should be renamed:

Suggested change

	# 831-Helm-Manifests: Installing Kubeflow with Helm
	# 831-Helm-Manifests: Support Helm for Individual Kubeflow Tools

[juliusvonkohout]

But is should also allow users to combine the Helm charts on their own as needed downstream. Therefore at least the title i would leave as it is

[andreyvelich]

What does install Kubeflow mean ?
As we discussed here, we have two concepts: kubeflow/kubeflow#7734:

• Kubeflow projects
• Kubeflow AI reference platform

If we say that this KEP is to implements Helm Charts for Kubeflow Projects we should say:

Implement Helm Charts for Kubeflow Projects.

[chasecadet]

What does install Kubeflow mean ? As we discussed here, we have two concepts: kubeflow/kubeflow#7734:

* Kubeflow projects

* Kubeflow AI reference platform

If we say that this KEP is to implements Helm Charts for Kubeflow Projects we should say:

Implement Helm Charts for Kubeflow Projects.

A leader's greatest defense against upheaval isn’t control or secrecy—it’s the trust and goodwill of the people. When leaders become resented, or when the community grows tired and disengaged, they’re not overthrown by force—they’re abandoned.

When people no longer feel heard or aligned with leadership, they stop defending it. They become indifferent, or worse, open to influence from alternatives that promise clarity, participation, and direction. That’s the real risk—not rebellion, but quiet attrition. And once that trust erodes, it’s incredibly hard to get back.

We are making the same mistake. By resisting a clear, consistent definition of Kubeflow as a platform—and instead pushing a vague, ever-shifting ecosystem narrative—we alienate the very community we rely on for innovation and resilience. This approach hasn't been meaningfully opened to the community, yet "Kubeflow platform" continues to surface organically across talks, Slack threads, and real user deployments. We're not leading; we're obstructing.

If we want to survive and thrive, we must choose clarity, openness, and alignment with the community. Not doing so is a slow self-destruction. Our sin of certainty is leading us astray and we aren't fixing the real problem, which is community apathy.

In this PR I an even see @juliusvonkohout's usual stoic self show frustration. This is a thread being pulled from a sweater that will soon unravel. @akgraner might have something to say as well here.

[andreyvelich]

This needs to be updated.

[juliusvonkohout]

maybe @kunal-511 can do so.

[andreyvelich]

This needs to be updated.

[juliusvonkohout]

@kunal-511

[andreyvelich]

Why do we need to put Helm Charts in the experimental folder if we maintain them in the individual Kubeflow Project repos ?

[varodrig]

agree with this comment

[juliusvonkohout]

Because you want someone to mentor this. I do not have the time to deal with each maintainer and fix their tests, i need the extended platform tests and make sure that the rendered output is identical with kustomize. All of this must be reusable and scalable without code duplication. So i can incubate in /experimental with full proper tests support and then upstream. Otherwise i would rather cancel the whole helm effort and move to something time efficient, because i value my time and that is not worth it.

[andreyvelich]

In that case, we should have plan how we can keep those Helm Charts synchronized between kubeflow/manifests and other repositories.

Since this KEP is affecting every Kubeflow project it requires collaboration across working groups and project maintainers.

[juliusvonkohout]

The same way as we synchronize kustomize. @kunal-511 will include them in our synchronization scripts. So upstream should become source after incubation as it is with kustomize.

[andreyvelich]

Suggested change

	* Develop subcharts for each major Kubeflow component (Pipelines, Platform, Notebooks, Dashboard, Katib, Training Operator, Istio, etc.) and make them deployable simultaneously to replicate the kustomize manifests.
	* Develop subcharts for each major Kubeflow tool (Pipelines, Trainer, Katib, Model Registry, Spark Operator, Notebooks).

[juliusvonkohout]

without istio etc helm charts it becomes useless for me and violates the original GSOC Idea. This change would make it useless for me to mentor this. I would waste 80 % of my and the students time. Then i rather close the PR and focus on something useful for the community. I need to be at least able to install all components individually even without a combined chart as we do already for kustomize.

[andreyvelich]

Same comment as here: #832 (comment)

[chasecadet]

How much time have we spent on this? To block a community-supported Helm chart? Fascinating.

[franciscojavierarceo]

I 100% agree with @chasecadet here.

[andreyvelich]

Working Group, KSC, or active maintainers of Kubeflow projects have not agreed on the Helm Charts that is proposed in this KEP. If @kubeflow/wg-manifests-leads is sponsoring this proposal, we should get at least +1 from @kimwnasptd.

As KSC voted previously, we agreed to have Helm Charts only for standalone Kubeflow project: #832 (comment)

[chasecadet]

Let them eat cake.

[juliusvonkohout]

Yes, but we explicitly did not vote to block something. We voted to encourage something. This Helm becomes useful for large installations, where you need templating for multiple charts. E.g you install KFP, model registy and Notebooks. Then helm finding what is needed as dependencies (istio, knative etc.) and setting common thing is the big benefit. Otherwise i can just wrap kustomize as it is in helm and be done in a few minutes with GSOC. In the end the goal is to have helm charts where i can say i want single or multi-tenant mode and based on that it install dependencies and sets up resources differently. Otherwise the effort is more ore less the same for the installation, kustomization and maintenance as with kustomize.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[juliusvonkohout]

The implementation is progressing as experimental POC.