Add kube_pod_status_ready_time

[gnuletik]

Hi!

Since v2.8.0, kube state metrics supports the following metrics:

• kube_pod_status_ready_time
• kube_pod_status_containers_ready_time

See kubernetes/kube-state-metrics#1938

However, I don't see these listed in the supported metrics list.

Is it possible to add support for these?

Thanks!

[tibernardinelli]

I also recommend to stop ignoring the metric kube_pod_start_time kubernetes_state.py#L327

With those metrics we can understand how long pod's take to start. This is particularly important for those who uses event-driven auto-scale in Kubernetes like Keda.

kube_pod_status_ready_time - kube_pod_start_time

[mertcankloian]

Absolutely right! These metrics and additional, receive_request_ms or equal one in datadog side are red dots for auto scaling rules and observability actually. They should be in usage as an active.

kube_pod_status_ready_time - kube_pod_start_time - receive_request_ms

[JQLSpec]

This would be very nice to have in Kube State Core. I can easily derive pod start times in my internal Grafana/Prometheus instances, being able to do this in DataDog would be amazing.

[bgaillard]

Hi, we also need the kube_pod_status_ready_time, in our case the use case is simply to identity problematic pods which are running in our clusters.

@tibernardinelli I think Datadog is not really "ignoring" the kube_pod_start_time, it appears the source code is doing a conversion of this metric into the kubernetes_state.pod.uptime.

We can find the name conversion which is done in this file kubernetes_state_transformers.go.

There is a similar renaming in the source code for the kube_pod_created metric which is renamed into kubernetes_state.pod.age.

It allows to do stuffs like that to get the time "from pod scheduling until pod creation".

I'm not sure to understand why the metrics have been renamed, for me it adds lot of confusion because the "golden source" should be the Kube State Metrics reference documentation IMO. Perhaps it's to respect other Datadog metrics conventions but this appears to not being homogeneous because for example the Datadog kubernetes_state.endpoint.created metric exists (so creating a kubernetes_state.pod.created metric would be possible 🤔?).

The renaming is probably associated to what's described here https://github.com/prometheus/OpenMetrics/blob/main/specification/OpenMetrics.md#suffixes 🤔 (at least for the _created) ?.

Anyway the PR which added those 2 (badly named?) metrics is #7764, I suppose we could take it as an example to add the missing ready time metrics.

[daniel-laszlo]

We would like to measure performance of our Kubernetes clusters and create SLOs by subtracting two metrics from each other

kube_pod_status_initialized_time - kube_pod_created.

Would it be possible to add and expose these metrics? Thank you.

[maanti]

I had to use the following workaround to collect some additional metrics. It's possible to define several integrations and collect missing metrics using prometheus integration:

      annotations:
        ad.datadoghq.com/kube-state-metrics.check_names: '["kubernetes_state", "prometheus"]'
        ad.datadoghq.com/kube-state-metrics.init_configs: '[{}, {}]'
        ad.datadoghq.com/kube-state-metrics.instances: |
          [
            {
              "kube_state_url": "http://%%host%%:8080/metrics"
            },
            {
              "prometheus_url": "http://%%host%%:8080/metrics",
              "namespace": "kubernetes_state",
              "metrics": [
                "kube_node_deletion_timestamp",
                "kube_pod_created",
                "kube_pod_start_time"
              ]
            }
          ]

Note, it's v1 annotations syntax. Check https://docs.datadoghq.com/getting_started/containers/autodiscovery if you are using v2