manifests: centralize the OTLP endpoint in an ate-otel-config ConfigMap

[Da Huang (git286)]

OTEL_EXPORTER_OTLP_ENDPOINT is written in 9 places: hardcoded inline in 4 base workload manifests, then re-patched in 5 spots across the kind overlays. Changing the collector address means editing all 9 and knowing which install path renders which.

Replace them with one checked-in ConfigMap per environment, consumed by every component via envFrom:

manifests/ate-install/ate-otel-config.yaml (GKE)
manifests/ate-install/kind/ate-otel-config.yaml (kind, same name)

The GKE copy is listed in manifests/ate-install/base, so token-client, agentgateway and agentgateway-token-client all inherit it; the kind overlay lists its own copy of the same ConfigMap name. No overlay builds on both, so the two never collide.

A kustomize-only fix does not work here. The GKE path applies the base directory raw and hack/install-ate.sh's targeted redeploys apply single files with no Kustomize, so the mechanism has to survive kubectl apply -f <one-file> -- which rules out configMapGenerator (hash-suffixed names) and replacements. envFrom on a stable name reaches every path.

deploy_ate_system applies the ConfigMap before the rendered bundle, the same way it already applies the namespace. A container whose envFrom target is missing does not start, and a raw directory apply is ordered by filename, so ate-api-server.yaml and ate-controller.yaml would otherwise be created first and sit in CreateContainerConfigError until the ConfigMap caught up.

This also fixes a latent bug in the targeted redeploys. deploy_ate_apiserver, deploy_atelet and deploy_atenet apply raw files even in kind mode, silently reverting the endpoint to the GKE value. They now apply the environment's ConfigMap via a new apply_otel_config helper, which picks the file by ATE_INSTALL_KIND rather than applying the base copy unconditionally -- applying the base one on kind would break telemetry for every component at once.

atenet-router needs no special handling: since a98f85b its --otlp-collector-address defaults to $OTEL_EXPORTER_OTLP_ENDPOINT, so Envoy's own spans follow the ConfigMap along with everything else.

OTEL_TRACES_SAMPLER deliberately stays an inline kind patch on ate-api-server rather than moving into the ConfigMap. The ConfigMap is shared by every component via envFrom, so putting parentbased_always_on there would pin the router and the rest of the control plane to 100% and undo the per-component ratios from 15eecd0.

Note that a ConfigMap edit does not roll the consuming pods the way an inline env change did, since the pod template is unchanged. Callers must follow a change with kubectl rollout restart. This is documented in both ConfigMaps, in docs/observability.md, and in the tracing best practices, which previously told authors to hardcode the variable.

Fixes #745

It's a good idea to open an issue first for discussion.

• Tests pass
• Appropriate changes to documentation are included in the PR

[Krisztian F (krisztianfekete)]

This LGTM, added some non-blocking comments!

[Krisztian F (krisztianfekete)]

Maybe not for this PR, but AGENTGATEWAY_OTLP_ADDRESS is also hardcoded in multiple places.

[Da Huang (git286)]

Thanks! Will leave it as a follow-up.

[Krisztian F (krisztianfekete)]

Maybe we can add that ateom workers get these values at pod creation, so a rollout-restart of the controller doesn't change existing workers (until they are restarted).

[Da Huang (git286)]

Added. One correction while verifying it: the workers don't need a separate restart. The controller SSA-applies the WorkerPool Deployment on every reconcile with the endpoint as a literal, so a controller restart changes the pod template and rolls the workers automatically. Confirmed on kind.