Which Kubernetes Features Would Help Kubearchive, and Vice Versa?

[adambkaplan]

Today I had an excellent discussion with @dims at KubeCon about the difference between submitting projects as a CNCF Sandbox vs. Kubernetes SIG-sponsored project. To paraphrase the conversation, it is not sufficient for a Kubernetes SIG-sponsored project to be beneficial to the Kubernetes end user community. Great candidates for SIG sponsorship drive improvements to the Kubernetes project itself through KEPs. Otherwise, CNCF Sandbox (or a component within an existing CNCF project) is the better long term home.

Taking Kueue as an example, it does not merely provide a tool for those that operate batch workloads on Kubernetes. It exposes gaps in how the Kubernetes scheduler works today and provides a space for improvements to be validated. Kueue's measures of success include "this project is obsolete, use XXX Kubernetes features instead."

I could think of 3-4 KEP-candidate ideas off the top of my head driven by my experience with Kubearchive. However, I would like to hear from the folks in the community who are doing the actual work first:

• Which features in Kubearchive would you like to see be included in Kubernetes itelf?
• Which features in Kubernetes would make Kubearchive easier to implement?
• Which features in Kubearchive are impossible to implement because Kubernetes doesn't provide what we need today?

To help organize this discussion, I suggest that folks create separate replies per KEP-level idea. This will keep the individual reply threads focused and on-topic.

[adambkaplan]

For additional context- I think KubeArchive belongs in the Kubernetes project itself as a SIG-sponsored project. SIG-Apps is probably the best sponsor, as they are the primary sponsor of kueue.

Donation to Kubernetes is not risk-free. Some of our practices, code structure, CRD names/api groups, and even release cadence may need to adjust to k8s standards. The potential benefits outweigh IMO outweigh the costs:

• Broader community adoption
• Larger base of contributors
• Improved project practices
• Access to CNCF and k8s resources
• Future improvements to Kubernetes itself.

[skoved]

Assuming that SIG-Apps (or whatever other SIG might be will to adopt KubeArchive) is broadly aligned with our vision for the project I personally feel like it's not too much to work to get ourselves aligned with the other k8s projects. Do you have insights into how SIGs generally go about working with projects that they sponsor? Would we be attending SIG meetings to discuss key work being done/project trajectory?

[skoved]

I can see KubeArchive being sponsored by SIG-Apps (or one of the other Kubernetes SIGs) would be a benefit for us. I'm not super familiar with the KEPs so I was going through the list. I saw this one about time to live for Jobs/Pods which was designed to make it easier to clean up resources. I feel like KubeArchive could provide an enhancement in here so that the data the pods/jobs provide is still accessible but don't create clutter on the cluster itself and don't take up space in ETCD

That KEP is "stable" so not sure if this is the type of thing you're looking for though. I always get a bit lost when going through stuff in kubernetes because of just how large the project is.

[skoved]

I also think it would be really cool if resources with a finite amount of run time (batch/Jobs, Tekton/PipelineRuns, etc.) had some specific key, which would be the same across them all) that you could query to determine if it was running or completed AND if it has completed, whether it was successful or failed. This would make configuration of KubeArchive for resources like this much simpler for users because KubeArchive wouldn't need to be told where to look for this information or require custom logic for each individual resource

[adambkaplan]

I feel like KubeArchive could provide an enhancement in here so that the data the pods/jobs provide is still accessible but don't create clutter on the cluster itself and don't take up space in ETCD

This is the sort of long term vision that I believe the Kubernetes community is looking for 😄. Kubearchive at present solves this problem. Long term - valid question if this should be considered part of a "conformant" Kubernetes distribution?

[maruiz93]

Something that would also help KubeArchive would be if Kubernetes always filled out the deletionTimestmp when deleting a resource. As far as I know, this is only done when there are finalizers and no --force is applied on the deletion.

[adambkaplan]

Logs features "inspired by" Kubearchive:

• /logs endpoint for core Kubernetes workload types (Job, Deployment, etc.). This is currently done client-side in kubectl.
• Extend current pod /logs to optionally stream all container logs, with filter options (ex: include/exclude init containers, exclude container by name, etc.)
• logs subresource for CRDs

[maruiz93]

Extend current pod /logs to optionally stream all container logs, with filter options (ex: include/exclude init containers, exclude container by name, etc.)

Regarding this, when kubectl logs several containers (e.g. kubectl logs -l <my-label-selector> --all-containers) they are shown all mixed, no established format for a consumable way of knowing when one container logs end and when the next one starts and whose pod/container the logs belong too.
Ideally we should have some changes in this output and a established standard of how to show them.

[adambkaplan]

CloudEvent "feeds" for Kubernetes object events

In some ways this gets back to old architecture. However, instead of us configuring the event queue and broker (Kafka), Kubernetes handles that detail. Projects like Kubearchive and Velero can then subscribe to these feeds and perform "edge-driven" actions.

Does our current architecture allow Kubearchive to create an unbound number of watches on the Kubernetes apiserver? Is there a risk to cluster availability if too many namespace-scoped KubearchiveConfig objects are created?

[adambkaplan]

Publish Log Forwarding Destinations

Currently cluster admins must tell Kubearchive where pod + container logs are forwarded in its deployment, which generally requires each deployment to have separate configuration for the log forwarding destination. This is challenging to scale in an enterprise, which may have hundreds of clusters to manage.

Tekton Results avoids this problem by relying on log forwarders to be deployed via operators, and publishing their log forward destinations in a respective custom resource definition. This presents its own ecosystem challenge - many projects do not want to maintain operators or custom resource definitions. A KEP along the lines of the KEP-1755 might help - publish the log forward destination(s) through a ConfigMap in the kube-public namespace.