CP-35087: Upgrade Prometheus to 3.x with backward compatibility

Prometheus 3.0 was released and includes performance improvements and new
features. We want to upgrade to the latest stable version (3.7.3) while
maintaining backward compatibility for customers who may be using custom
Prometheus 2.x images.

Impact:

Before: Chart used Prometheus 2.55.1 by default. Prometheus 3.x images would
produce warnings about unknown flag "--enable-feature=agent" since the agent
mode flag changed between versions.

After: Chart uses Prometheus 3.7.3 by default. Both 2.x and 3.x versions work
correctly with automatic flag detection based on image version.

Scope: All customers, but particularly those using custom Prometheus images
via server.image.tag or components.prometheus.image.tag overrides.

Implementation Approach:

Prometheus 3.x changed the agent mode flag from `--enable-feature=agent` to
just `--agent`. To support both versions seamlessly, we needed version-aware
flag selection that respects the same image tag fallback chain used for
actual image generation.

Solution:

1. Updated Chart.AppVersion from v2.55.1 to v3.7.3 in helm/Chart.yaml

2. Created cloudzero-agent.prometheusAgentFlag helper in helm/templates/_helpers.tpl
   that:
   - Checks if mode is "agent" or "federated" (using existing mode derivation logic)
   - Uses same tag fallback as image generation: server.image.tag →
     components.prometheus.image.tag → Chart.AppVersion
   - Returns "--enable-feature=agent" for v2.x tags
   - Returns "--agent" for v3.x and newer tags
   - Returns empty string for server/clustered modes

3. Updated helm/templates/agent-deploy.yaml and helm/templates/agent-daemonset.yaml
   to use the new helper instead of hardcoded flags

4. Updated helm/tests/agent_mode_derivation_test.yaml to expect "--agent" flag
   for default Chart.AppVersion (now 3.7.3)

5. Added comprehensive test suite in helm/tests/prometheus_version_flag_test.yaml
   covering:
   - Prometheus 2.x uses --enable-feature=agent
   - Prometheus 3.x uses --agent
   - server.image.tag precedence over components.prometheus.image.tag
   - Both agent and federated modes
   - Server mode (no agent flag)
   - Fallback to Chart.AppVersion

Validation:

- All 255 Helm unit tests pass (added 8 new tests)
- Schema validation tests pass
- Go unit tests pass
- Deployed to brahms cluster successfully
- Verified Prometheus 3.7.3 starts correctly in agent mode with no warnings
- Confirmed all expected metrics (container, node, pod, GPU) flowing correctly
- Zero dropped metrics in production deployment
- Manually tested both v2.x and v3.x tags produce correct flags
- Verified server.image.tag override precedence works correctly

Author: evan-cz

helm/Chart.yaml

@@ -8,7 +8,7 @@ kubeVersion: ">= 1.21.0-0"
 maintainers:
   - name: CloudZero
     email: [email protected]
-appVersion: "v2.55.1"
+appVersion: "v3.7.3"
 dependencies:
   - name: kube-state-metrics
     version: "5.36.*"

helm/templates/_helpers.tpl

@@ -1294,3 +1294,34 @@ alloy-config.river
 prometheus.yml
 {{- end -}}
 {{- end -}}
+
+{{/*
+Get the appropriate Prometheus agent mode flag based on version and mode
+
+Determines whether Prometheus should run in agent mode and which flag to use:
+- Prometheus 2.x uses --enable-feature=agent
+- Prometheus 3.x uses --agent
+- Returns empty string if not in agent/federated mode
+
+The cloudzero-agent.Values.components.agent.mode helper already handles all the
+complex mode derivation logic, so we just check if it returns "agent" or "federated"
+and then determine the appropriate version-specific flag.
+
+Uses the same tag fallback chain as image generation:
+server.image.tag -> components.prometheus.image.tag -> Chart.AppVersion
+
+Usage: {{ include "cloudzero-agent.prometheusAgentFlag" . }}
+Returns: string (either "--agent", "--enable-feature=agent", or empty string)
+*/}}
+{{- define "cloudzero-agent.prometheusAgentFlag" -}}
+  {{- $mode := include "cloudzero-agent.Values.components.agent.mode" . -}}
+  {{- if or (eq $mode "agent") (eq $mode "federated") -}}
+    {{- /* Use same fallback chain as image generation: server.image.tag -> components.prometheus.image.tag -> Chart.AppVersion */ -}}
+    {{- $tag := .Values.server.image.tag | default .Values.components.prometheus.image.tag | default .Chart.AppVersion -}}
+    {{- if hasPrefix "v2." $tag -}}
+      --enable-feature=agent
+    {{- else -}}
+      --agent
+    {{- end -}}
+  {{- end -}}
+{{- end -}}

helm/templates/agent-daemonset.yaml

@@ -111,9 +111,9 @@ spec:
             - --web.enable-lifecycle
             - --web.console.libraries=/etc/prometheus/console_libraries
             - --web.console.templates=/etc/prometheus/consoles
-            {{- /* In federated mode, default to Prometheus agent unless server.agentMode is explicitly false */ -}}
-            {{- if or (not .Values.server) (eq .Values.server.agentMode nil) (eq .Values.server.agentMode true) }}
-            - --enable-feature=agent
+            {{- $agentFlag := include "cloudzero-agent.prometheusAgentFlag" . }}
+            {{- if $agentFlag }}
+            - {{ $agentFlag }}
             {{- end }}
           ports:
             - containerPort: 9090

helm/templates/agent-deploy.yaml

@@ -148,8 +148,9 @@ spec:
                   - /checks/app/config/validator.yml
           args:
             {{ toYaml .Values.server.args | nindent 12}}
-            {{- if or (eq (include "cloudzero-agent.Values.components.agent.mode" .) "agent") (eq (include "cloudzero-agent.Values.components.agent.mode" .) "federated") }}
-            - --enable-feature=agent
+            {{- $agentFlag := include "cloudzero-agent.prometheusAgentFlag" . }}
+            {{- if $agentFlag }}
+            - {{ $agentFlag }}
             {{- end }}
             - --log.level={{ .Values.server.logging.level | default "info" }}
           ports:

helm/tests/agent_mode_derivation_test.yaml

@@ -68,7 +68,7 @@ tests:
       # Should have agent mode flag (default)
       - contains:
           path: spec.template.spec.containers[1].args
-          content: --enable-feature=agent
+          content: --agent
         template: templates/agent-deploy.yaml
 
   # Test automatic mode: server.agentMode=false derives mode as "server"
@@ -85,7 +85,7 @@ tests:
       # Should NOT have agent mode flag (server mode)
       - notContains:
           path: spec.template.spec.containers[1].args
-          content: --enable-feature=agent
+          content: --agent
         template: templates/agent-deploy.yaml
       # Should have Prometheus config (not Alloy)
       - isNotEmpty:
@@ -106,7 +106,7 @@ tests:
       # Should have agent mode flag (default)
       - contains:
           path: spec.template.spec.containers[1].args
-          content: --enable-feature=agent
+          content: --agent
         template: templates/agent-deploy.yaml
       # Should have Prometheus config (not Alloy)
       - isNotEmpty:
@@ -155,7 +155,7 @@ tests:
       # Should have agent mode flag
       - contains:
           path: spec.template.spec.containers[1].args
-          content: --enable-feature=agent
+          content: --agent
         template: templates/agent-deploy.yaml
 
   # Test that components.agent.mode="server" works
@@ -170,7 +170,7 @@ tests:
       # Should NOT have agent mode flag
       - notContains:
           path: spec.template.spec.containers[1].args
-          content: --enable-feature=agent
+          content: --agent
         template: templates/agent-deploy.yaml
 
   # Test that components.agent.mode="clustered" works