CP-35087: Upgrade Prometheus to 3.x with backward compatibility

[evan-cz]

Prometheus 3.0 was released and includes performance improvements and new features. We want to upgrade to the latest stable version (3.7.3) while maintaining backward compatibility for customers who may be using custom Prometheus 2.x images.

Impact:

Before: Chart used Prometheus 2.55.1 by default. Prometheus 3.x images would produce warnings about unknown flag "--enable-feature=agent" since the agent mode flag changed between versions.

After: Chart uses Prometheus 3.7.3 by default. Both 2.x and 3.x versions work correctly with automatic flag detection based on image version.

Scope: All customers, but particularly those using custom Prometheus images via server.image.tag or components.prometheus.image.tag overrides.

Implementation Approach:

Prometheus 3.x changed the agent mode flag from --enable-feature=agent to just --agent. To support both versions seamlessly, we needed version-aware flag selection that respects the same image tag fallback chain used for actual image generation.

Solution:

1. Updated Chart.AppVersion from v2.55.1 to v3.7.3 in helm/Chart.yaml

2. Created cloudzero-agent.prometheusAgentFlag helper in helm/templates/_helpers.tpl that:

   • Checks if mode is "agent" or "federated" (using existing mode derivation logic)
   • Uses same tag fallback as image generation: server.image.tag → components.prometheus.image.tag → Chart.AppVersion
   • Returns "--enable-feature=agent" for v2.x tags
   • Returns "--agent" for v3.x and newer tags
   • Returns empty string for server/clustered modes

3. Updated helm/templates/agent-deploy.yaml and helm/templates/agent-daemonset.yaml to use the new helper instead of hardcoded flags

4. Updated helm/tests/agent_mode_derivation_test.yaml to expect "--agent" flag for default Chart.AppVersion (now 3.7.3)

5. Added comprehensive test suite in helm/tests/prometheus_version_flag_test.yaml covering:

   • Prometheus 2.x uses --enable-feature=agent
   • Prometheus 3.x uses --agent
   • server.image.tag precedence over components.prometheus.image.tag
   • Both agent and federated modes
   • Server mode (no agent flag)
   • Fallback to Chart.AppVersion

Validation:

• All 255 Helm unit tests pass (added 8 new tests)
• Schema validation tests pass
• Go unit tests pass
• Deployed to brahms cluster successfully
• Verified Prometheus 3.7.3 starts correctly in agent mode with no warnings
• Confirmed all expected metrics (container, node, pod, GPU) flowing correctly
• Zero dropped metrics in production deployment
• Manually tested both v2.x and v3.x tags produce correct flags
• Verified server.image.tag override precedence works correctly