feat: run deploy and evaluate scripts inside the Kagenti cluster

[yoavkatz]

Context

Currently deploy-benchmark.sh, deploy-agent.sh, and evaluate-benchmark.sh (orchestrated by deploy-and-evaluate.sh) run on a local developer machine. They assume a local shell environment with access to kubectl connected to a KinD cluster named kind-kagenti, a local container runtime (podman/docker), and kind for image syncing.

This issue tracks the work to run these scripts inside a container in the Kagenti cluster itself — as a Kubernetes Job or Pod, rather than from a developer laptop.

---

What works already (no changes needed)

All HTTP-based steps already work from inside the cluster:

• Keycloak auth (http://keycloak.localtest.me:8080) — reachable via Istio ingress
• Kagenti API calls (http://kagenti-api.localtest.me:8080) — same
• MCP health checks — same
• GitHub raw content fetches — standard internet egress

---

What needs to change

1. Image pull + KinD sync (biggest structural change)

deploy-benchmark.sh and deploy-agent.sh both pull images locally, check whether the KinD node already has them, and sync via kind load image-archive. This entire flow is local-machine-centric:

• No podman/docker socket is available inside a cluster container
• kind load image-archive requires exec access to the KinD node container — impossible from inside the cluster
• Image-ID comparison via podman/docker exec kagenti-control-plane crictl inspecti ... won't work

Resolution: Gate the image sync block on [ -z "$KUBERNETES_SERVICE_HOST" ]. When in-cluster, skip straight to using $REMOTE_IMAGE_NAME directly. The --local-image flag remains functional for local use.

Affected: deploy-benchmark.sh:104–202, deploy-agent.sh:205–303.

2. kubectl — removed from in-cluster path

All cluster interaction goes through the Kagenti API and Keycloak over HTTP. No kubectl is needed in-cluster, so no ServiceAccount or RBAC is required. The Job uses the default ServiceAccount.

kubectl wait readiness checks are replaced with curl retry loops against the service endpoints directly (see item 3).

3. Port-forwards are meaningless from inside the cluster

evaluate-benchmark.sh sets up:

• OTEL collector: localhost:4327 → otel-collector.kagenti-system:4317
• Prometheus: localhost:9191 → prometheus.istio-system:9090

Resolution: Gate port-forward setup on [ -z "$KUBERNETES_SERVICE_HOST" ]. When in-cluster, use cluster DNS directly via dedicated URL helper functions.

Affected: evaluate-benchmark.sh:218–262.

4. *.localtest.me URLs don't resolve from inside the cluster

localtest.me is a DNS wildcard that resolves to 127.0.0.1. From inside a pod, this points to the container's own loopback — not the ingress.

Resolution: A shared lib/urls.sh with named functions for each service. Each function checks KUBERNETES_SERVICE_HOST and returns the appropriate URL.

Current (localtest.me)	In-cluster replacement
http://kagenti-api.localtest.me:8080	http://kagenti-api.kagenti-system.svc.cluster.local:8080
http://keycloak.localtest.me:8080	http://keycloak.keycloak.svc.cluster.local:8080
http://{tool}.team1.localtest.me:8080	http://{tool}-mcp.team1.svc.cluster.local:8000
http://{agent}.team1.localtest.me:8080	http://{agent}.team1.svc.cluster.local:8080
http://mcp-gateway-istio.gateway-system.localtest.me:8080	http://mcp-gateway-istio.gateway-system.svc.cluster.local:8080

5. Cluster context check — remove when in-cluster

evaluate-benchmark.sh:126–145 checks kubectl config current-context and prompts interactively if not kind-kagenti. Inside a pod there is no named context, and interactive prompts hang forever.

Resolution: Gate on [ -z "$KUBERNETES_SERVICE_HOST" ]. Skip entirely when in-cluster.

6. Python/uv runner containerization

evaluate-benchmark.sh:314–373 runs the Python harness via uv sync + uv run. The container image bakes in the source and pre-installs dependencies at build time — no uv sync at runtime.

---

Deliverables

1. exgentic_a2a_runner/lib/urls.sh — shared URL helper functions sourced by all scripts
2. exgentic_a2a_runner/Dockerfile — runner image with curl, jq, bash, python3.12, uv, pre-built venv; no kubectl
3. Gate image sync on [ -z "$KUBERNETES_SERVICE_HOST" ] in deploy-benchmark.sh and deploy-agent.sh
4. Replace kubectl wait with curl retry loops in evaluate-benchmark.sh
5. Gate port-forward setup on [ -z "$KUBERNETES_SERVICE_HOST" ] in evaluate-benchmark.sh
6. Gate context check on [ -z "$KUBERNETES_SERVICE_HOST" ] in evaluate-benchmark.sh
7. exgentic_a2a_runner/k8s/job.yaml — Kubernetes Job with runner image and secret injection (OPENAI_API_KEY, KEYCLOAK_PASSWORD, etc.), default ServiceAccount

---

Verification

1. Local path unchanged: run deploy-and-evaluate.sh on a laptop with KinD — behavior identical to before
2. Build image: docker build -t ghcr.io/exgentic/runner:test exgentic_a2a_runner/
3. Launch Job: kubectl apply -f exgentic_a2a_runner/k8s/job.yaml
4. kubectl logs -f job/<name> — should see all 3 steps execute
5. Verify benchmark pod and agent pod become Ready in team1
6. Verify evaluation produces results

[yoavkatz]

Plan: Run deploy and evaluate scripts inside the Kagenti cluster

Context

deploy-benchmark.sh, deploy-agent.sh, evaluate-benchmark.sh, and deploy-and-evaluate.sh currently run on a developer laptop. They depend on a local container runtime (podman/docker), kind, and a kubeconfig pointing at a KinD cluster. This plan makes them runnable as a Kubernetes Job inside the cluster itself, while keeping the local dev workflow unchanged.

The trigger for each change is KUBERNETES_SERVICE_HOST — a standard env var injected into all pods by Kubernetes. When set, use in-cluster paths; otherwise fall back to existing local behavior.

---

Changes

1. Shared URL helper library — exgentic_a2a_runner/lib/urls.sh (new file)

All three scripts source this file. It defines named functions for each service:

kagenti_api_url()   # kagenti-api.kagenti-system / kagenti-api.localtest.me
keycloak_url()      # keycloak.keycloak / keycloak.localtest.me
mcp_tool_url()      # {name}-mcp.{ns}:8000 (in-cluster) / {name}.{ns}.localtest.me:8080
agent_url()         # {name}.{ns}:8080 (in-cluster) / {name}.{ns}.localtest.me:8080
mcp_gateway_url()   # {name}.{ns}:8080 (in-cluster) / {name}.{ns}.localtest.me:8080
otel_endpoint()     # otel-collector.kagenti-system:4317 / localhost:${OTEL_COLLECTOR_LOCAL_PORT}
prometheus_url()    # prometheus.istio-system:9090 / localhost:${PROMETHEUS_LOCAL_PORT}

Each function checks [ -n "$KUBERNETES_SERVICE_HOST" ] and returns the appropriate URL.

Each script adds source "$(dirname "$0")/lib/urls.sh" near the top, then replaces every inline URL construction with the corresponding function call.

2. Remove image pull + KinD sync — deploy-benchmark.sh, deploy-agent.sh

Gate the entire image-sync block on [ -z "$KUBERNETES_SERVICE_HOST" ]:

if [ -z "$KUBERNETES_SERVICE_HOST" ]; then
  # existing: container runtime detection, pull, tag, crictl compare, kind load
fi

When in-cluster, skip straight to using $REMOTE_IMAGE_NAME directly. The --local-image flag remains functional for local use.

Affected blocks: deploy-benchmark.sh:104–202, deploy-agent.sh:205–303.

3. Replace kubectl wait with endpoint polling — evaluate-benchmark.sh

Remove the kubectl wait --for=condition=ready calls (lines 162–207). Replace with curl retry loops against the same endpoints already tested a few lines later:

• MCP server readiness: POST initialize to $(mcp_tool_url ...)/mcp (reuse existing health-check pattern from deploy-benchmark.sh)
• Agent readiness: GET $(agent_url ...)/.well-known/agent-card.json

Use a loop with a timeout (e.g. 60 attempts × 1s sleep) consistent with the existing kubectl wait --timeout=60s.

4. Remove port-forwards — evaluate-benchmark.sh

Gate the kubectl port-forward subprocess setup on [ -z "$KUBERNETES_SERVICE_HOST" ] (lines 218–262):

if [ -z "$KUBERNETES_SERVICE_HOST" ]; then
  # existing: port-forward setup, PID tracking, cleanup trap
fi

When in-cluster, otel_endpoint() and prometheus_url() return cluster DNS addresses directly. The OTEL_EXPORTER_OTLP_ENDPOINT and PROMETHEUS_URL env vars are set from these functions regardless of path.

5. Remove interactive context check — evaluate-benchmark.sh

Gate the kubectl config current-context block (lines 126–145) on [ -z "$KUBERNETES_SERVICE_HOST" ]. When in-cluster, skip entirely.

6. Runner container image — exgentic_a2a_runner/Dockerfile (new file)

Build an image that bakes in the source and pre-installs dependencies at build time (no uv sync at runtime).

FROM python:3.12-slim
RUN pip install uv
WORKDIR /app
COPY . .
RUN uv sync --frozen
ENTRYPOINT ["uv", "run", "exgentic-a2a-runner"]

Also install curl, jq, bash for the shell scripts. No kubectl needed.

The evaluate script, when in-cluster, skips uv sync and .venv activation — the entrypoint handles that. The scripts invoke uv run exgentic-a2a-runner directly.

7. Kubernetes Job manifest — exgentic_a2a_runner/k8s/job.yaml (new file)

A Job manifest with:

• The runner image
• All required secrets injected as env vars (OPENAI_API_KEY, KEYCLOAK_PASSWORD, etc.)
• Default ServiceAccount — no RBAC needed since all cluster interaction goes through the Kagenti API and Keycloak over HTTP, not the Kubernetes API

---

Files to modify

File	Change
deploy-benchmark.sh	Gate image-sync block; replace URL literals with lib/urls.sh functions
deploy-agent.sh	Gate image-sync block; replace URL literals with lib/urls.sh functions
evaluate-benchmark.sh	Gate port-forward + context-check blocks; replace kubectl wait with curl loops; replace URL literals
deploy-and-evaluate.sh	Replace URL literals with lib/urls.sh functions
exgentic_a2a_runner/lib/urls.sh	New — shared URL helper functions
exgentic_a2a_runner/Dockerfile	New — runner container image
exgentic_a2a_runner/k8s/job.yaml	New — Kubernetes Job with secret injection, default ServiceAccount

---

Verification

1. Local path unchanged: run deploy-and-evaluate.sh on a laptop with KinD — behavior identical to before
2. Build image: docker build -t ghcr.io/exgentic/runner:test exgentic_a2a_runner/
3. Launch Job: kubectl apply -f exgentic_a2a_runner/k8s/job.yaml
4. Follow logs: kubectl logs -f job/<name> — should see all 3 steps execute
5. Verify benchmark pod becomes Ready in team1
6. Verify agent pod becomes Ready in team1
7. Verify evaluation produces results (check job exit code 0)