diff --git a/docs/integrator/eks-dynamo-networking.md b/docs/integrator/eks-dynamo-networking.md index 6c8fb3531..d10fdca7e 100644 --- a/docs/integrator/eks-dynamo-networking.md +++ b/docs/integrator/eks-dynamo-networking.md @@ -1,30 +1,44 @@ # EKS Dynamo Networking Prerequisites For `*-eks-ubuntu-inference-dynamo` recipes, AICR configures -`dynamo-platform` with Kubernetes-native discovery and the standard NATS -event plane for KV-cache and runtime events: -- `nats` on TCP `4222` - -This NATS dependency is new as of the Dynamo 1.2 bump, which switched discovery -to the NATS event plane. A cluster whose system-node security group only -allowlisted the pre-1.2 control-plane ports will not have `4222` open, so a -bundle that worked on Dynamo 1.0.x can start failing purely from the version -bump — add the `4222` rule below. - -Frontend-to-worker inference request/response traffic is separate: Dynamo 1.2 -defaults `DYN_REQUEST_PLANE` to TCP, and AICR does not override it to NATS. The -worker runtime relays local vLLM ZMQ KV-cache events onto the NATS-backed event -plane so the KV router or EPP can consume live cache state. - -If system components and GPU workloads are on different node groups/security groups, these ports may be blocked from GPU nodes to system nodes. Typical symptoms: -- `JetStream not available` (NATS unreachable) -- Dynamo frontend and vLLM worker pods stuck in `CrashLoopBackOff`, with - `Exception: Failed to connect to NATS: timed out` in the frontend log -- Worker startup probes failing with `connection refused` because the process - exits before serving -- The `inference-perf` performance validator failing after its workload-readiness - (10 min) and health (5 min) gates lapse — roughly 15 min — while `deployment` - and `conformance` pass; the workload never reaches a ready state +`dynamo-platform` with Kubernetes-native discovery. As of the Dynamo 1.4+ +bump, AICR no longer installs bundled NATS by default: the request plane +defaults to TCP and the KV event plane defaults to ZMQ +(`ai-dynamo/dynamo#11951`). This removes the old `4222` NATS requirement, +but it does **not** remove the underlying cross-nodegroup networking +requirement — the request plane and KV events are now **direct +frontend↔worker pod-to-pod connections** instead of both sides talking to a +`dynamo-platform-nats` StatefulSet on the system nodegroup, and Frontend +pods still run on the system nodegroup while workers run on the GPU +nodegroup, so traffic still crosses the same GPU↔system nodegroup SG +boundary as before. + +> **TODO before merging (tracked in NVIDIA/aicr#1836):** the port(s) below +> are not yet confirmed against a real Dynamo 1.4+ EKS deployment. What's +> known from the AICR recipes: the ZMQ KV-event endpoint is set explicitly +> per worker via `--kv-events-config`, e.g. +> `{"enable_kv_cache_events":true,"publisher":"zmq","endpoint":"tcp://*:5557"}` +> (see `tests/manifests/dynamo-vllm-smoke-test.yaml`), offset by `+dp_rank` +> for dp_rank > 0. The TCP request plane does not have one fixed, +> documented port the way NATS had `4222` — confirm the actual listening +> port(s) on a live cluster before finalizing the SG rule below: +> ```shell +> kubectl exec -n dynamo-system -- ss -tlnp +> kubectl exec -n dynamo-system -- ss -tlnp +> ``` + +If the GPU and system node groups sit in different security groups, these +ports may be blocked from GPU nodes to the frontend's node (and vice versa). +Typical symptoms: +- Dynamo frontend and vLLM worker pods stuck in `CrashLoopBackOff`, or a + frontend that starts cleanly but never successfully routes a request + through to a worker +- Worker startup probes failing with `connection refused` because the + process exits before serving +- The `inference-perf` performance validator failing after its + workload-readiness (10 min) and health (5 min) gates lapse — roughly + 15 min — while `deployment` and `conformance` pass; the workload never + reaches a ready state You can confirm reachability directly from a GPU node before re-running. The toleration is required because the GPU node groups on these clusters are @@ -32,9 +46,9 @@ tainted (`NoSchedule`/`NoExecute`); without it the probe pod stays `Pending` and never runs: ```shell -kubectl run nats-probe --rm -i --restart=Never --image=busybox:1.36 \ +kubectl run tcp-probe --rm -i --restart=Never --image=busybox:1.36 \ --overrides='{"spec":{"nodeSelector":{"":""},"tolerations":[{"operator":"Exists"}]}}' \ - -- sh -c 'nc -zv -w 5 dynamo-platform-nats.dynamo-system.svc.cluster.local 4222' + -- sh -c 'nc -zv -w 5 ' ``` The conformance validator's `ai-service-metrics` check adds a third requirement: @@ -64,8 +78,9 @@ SG rule below remains the reliable cluster-side guarantee. ## Required Security Group Rules -Allow ingress from the GPU node security group to the system node security group on: -- TCP `4222` - NATS event plane (dynamo-platform) +Allow ingress from the GPU node security group to the system node security +group on: +- TCP `` - Dynamo request plane + KV events (dynamo-platform) — confirm exact port(s) on-cluster, see TODO above - TCP `9090` - Prometheus (required for the `ai-service-metrics` conformance check) The `9090` rule is required as a fallback guarantee: the orchestrator *prefers* @@ -94,9 +109,9 @@ aws ec2 describe-instances \ --query "Reservations[0].Instances[0].SecurityGroups[*].GroupId" \ --output text -# 2) Allow NATS + Prometheus from GPU SG -> system SG +# 2) Allow Dynamo request/event-plane + Prometheus from GPU SG -> system SG aws ec2 authorize-security-group-ingress --group-id \ - --protocol tcp --port 4222 --source-group + --protocol tcp --port --source-group aws ec2 authorize-security-group-ingress --group-id \ --protocol tcp --port 9090 --source-group diff --git a/recipes/components/dynamo-platform/values.yaml b/recipes/components/dynamo-platform/values.yaml index e6fbeece9..4dc20d44e 100644 --- a/recipes/components/dynamo-platform/values.yaml +++ b/recipes/components/dynamo-platform/values.yaml @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -# Dynamo Platform Helm values (v1.2.1) +# Dynamo Platform Helm values (v) # NVIDIA Dynamo inference serving platform operator. # Provides OpenAI-compatible endpoints, KV-cache-aware routing, # disaggregated prefill/decode, and SLA-driven autoscaling. @@ -23,13 +23,18 @@ global: # Kubernetes-native discovery replaces etcd for AICR Dynamo workloads. etcd: install: false - # Dynamo 1.2 defaults the runtime event plane to NATS for distributed - # Kubernetes backends. Install the bundled NATS chart so workloads can use - # the runtime default instead of per-workload DYN_EVENT_PLANE overrides. - # Request/response traffic stays on Dynamo's request plane default (TCP) - # unless a workload explicitly sets DYN_REQUEST_PLANE. - nats: - install: true + # Dynamo 1.4+ defaults the request plane to TCP and the KV event plane to + # ZMQ, so the bundled NATS chart is no longer needed for AICR's defaults. + # global.nats.install is intentionally left unset (chart default: false). + # Re-enable only if a workload explicitly opts into a NATS-based transport + # (legacy DYN_REQUEST_PLANE=nats, DYN_EVENT_PLANE=nats, or a NATS + # request-trace sink), via `nats: install: true`. + # + # Upgrade note: on an in-place `helm upgrade` of a standing cluster, + # accepting this default removes bundled NATS and drops NATS_SERVER from + # operator-managed workload specs, which triggers rolling restarts. Fresh + # bundle installs are unaffected. Pass --set global.nats.install=true + # during the upgrade to opt out and avoid the restart. # Disable kai-scheduler sub-chart install — managed as a separate AICR component. # Keep enabled: true so the Dynamo operator detects and uses the external scheduler. kai-scheduler: @@ -41,17 +46,6 @@ global: install: false enabled: true -nats: - config: - jetstream: - fileStore: - pvc: - # Mirror the chart default so AICR can warn when no StorageClass is - # selected and inject --storage-class into the bundled NATS PVC. - enabled: true - size: 10Gi - storageClassName: "" - dynamo-operator: # Upgrade CRDs via the platform chart (no separate dynamo-crds 1.0 chart) upgradeCRD: true diff --git a/recipes/registry.yaml b/recipes/registry.yaml index e8671d265..dee33b573 100644 --- a/recipes/registry.yaml +++ b/recipes/registry.yaml @@ -524,18 +524,14 @@ components: helm: defaultRepository: https://helm.ngc.nvidia.com/nvidia/ai-dynamo defaultChart: dynamo-platform - defaultVersion: "1.2.1" + defaultVersion: "" defaultNamespace: dynamo-system - storageClassPaths: - - nats.config.jetstream.fileStore.pvc.storageClassName nodeScheduling: system: nodeSelectorPaths: - dynamo-operator.controllerManager.nodeSelector - - nats.podTemplate.merge.spec.nodeSelector tolerationPaths: - dynamo-operator.controllerManager.tolerations - - nats.podTemplate.merge.spec.tolerations - name: agentgateway-crds displayName: agentgateway-crds diff --git a/tests/chainsaw/ai-conformance/README.md b/tests/chainsaw/ai-conformance/README.md index be9e76982..8d57c3ef0 100644 --- a/tests/chainsaw/ai-conformance/README.md +++ b/tests/chainsaw/ai-conformance/README.md @@ -162,7 +162,7 @@ chainsaw test \ | Namespaces, CRDs | 2m | Should exist immediately after deployment | | cert-manager, agentgateway, skyhook, monitoring, kai-scheduler | 5m | Standard Deployment rollout | | gpu-operator, nvidia-dra-driver-gpu | 10m | GPU driver compilation on nodes is slow | -| dynamo-platform | 5m | Operator + Grove + NATS event-plane startup | +| dynamo-platform | 5m | Operator + Grove startup | ## Assertion Patterns diff --git a/tests/chainsaw/ai-conformance/cluster/assert-dynamo.yaml b/tests/chainsaw/ai-conformance/cluster/assert-dynamo.yaml index a5037510c..4a75d4117 100644 --- a/tests/chainsaw/ai-conformance/cluster/assert-dynamo.yaml +++ b/tests/chainsaw/ai-conformance/cluster/assert-dynamo.yaml @@ -13,12 +13,13 @@ # limitations under the License. # Assert Dynamo platform components are healthy. -# Chart: dynamo-platform 1.2.1 +# Chart: dynamo-platform # Provides NVIDIA Dynamo inference serving: OpenAI-compatible endpoints, # KV-cache-aware routing, disaggregated prefill/decode, SLA-driven autoscaling. # -# AICR uses Kubernetes-native discovery, so etcd remains disabled. NATS is -# enabled for Dynamo's Kubernetes event plane and worker KV-cache events. +# AICR uses Kubernetes-native discovery, so etcd remains disabled. As of +# Dynamo 1.4+, the request plane defaults to TCP and the KV event plane to +# ZMQ, so bundled NATS is no longer installed by default (NVIDIA/aicr#1836). # Dynamo Operator — manages DynamoComponent and DynamoGraphDeployment CRs apiVersion: apps/v1 @@ -30,15 +31,6 @@ status: (conditions[?type == 'Available']): - status: "True" --- -# NATS — event plane used by Dynamo KV-cache-aware routing on Kubernetes -apiVersion: apps/v1 -kind: StatefulSet -metadata: - name: dynamo-platform-nats - namespace: dynamo-system -status: - (readyReplicas > `0`): true ---- # Grove Operator — pod lifecycle management for DynamoGraphDeployments apiVersion: apps/v1 kind: Deployment diff --git a/tests/chainsaw/ai-conformance/cluster/chainsaw-test.yaml b/tests/chainsaw/ai-conformance/cluster/chainsaw-test.yaml index d858f14b1..c19893381 100644 --- a/tests/chainsaw/ai-conformance/cluster/chainsaw-test.yaml +++ b/tests/chainsaw/ai-conformance/cluster/chainsaw-test.yaml @@ -116,7 +116,7 @@ spec: # ── Dynamo Platform ──────────────────────────────────────────────── - name: assert-dynamo - description: Verify Dynamo operator, Grove, and NATS event plane are healthy. + description: Verify Dynamo operator and Grove are healthy. try: - assert: file: assert-dynamo.yaml diff --git a/tests/chainsaw/ai-conformance/kind-inference-dynamo/assert-dynamo.yaml b/tests/chainsaw/ai-conformance/kind-inference-dynamo/assert-dynamo.yaml index 9401afa0a..04c9153a4 100644 --- a/tests/chainsaw/ai-conformance/kind-inference-dynamo/assert-dynamo.yaml +++ b/tests/chainsaw/ai-conformance/kind-inference-dynamo/assert-dynamo.yaml @@ -13,8 +13,9 @@ # limitations under the License. # Assert Dynamo platform components are healthy on the kind inference stack. -# AICR uses Kubernetes-native discovery plus NATS for Dynamo's Kubernetes -# event plane and worker KV-cache events. +# AICR uses Kubernetes-native discovery. As of Dynamo 1.4+, the request +# plane defaults to TCP and the KV event plane to ZMQ, so bundled NATS is +# no longer installed by default (NVIDIA/aicr#1836). apiVersion: apps/v1 kind: Deployment metadata: @@ -25,14 +26,6 @@ status: - status: "True" --- apiVersion: apps/v1 -kind: StatefulSet -metadata: - name: dynamo-platform-nats - namespace: dynamo-system -status: - (readyReplicas > `0`): true ---- -apiVersion: apps/v1 kind: Deployment metadata: name: grove-operator diff --git a/tests/chainsaw/ai-conformance/kind-inference-dynamo/chainsaw-test.yaml b/tests/chainsaw/ai-conformance/kind-inference-dynamo/chainsaw-test.yaml index eefb665ad..d8e0b87b0 100644 --- a/tests/chainsaw/ai-conformance/kind-inference-dynamo/chainsaw-test.yaml +++ b/tests/chainsaw/ai-conformance/kind-inference-dynamo/chainsaw-test.yaml @@ -118,7 +118,7 @@ spec: # ── Dynamo Platform ──────────────────────────────────────────────── - name: assert-dynamo - description: Verify Dynamo operator, Grove, and NATS event plane are healthy. + description: Verify Dynamo operator and Grove are healthy. try: - assert: file: assert-dynamo.yaml diff --git a/tests/manifests/dynamo-vllm-smoke-test.yaml b/tests/manifests/dynamo-vllm-smoke-test.yaml index 6497a8bcb..f07e9aa52 100644 --- a/tests/manifests/dynamo-vllm-smoke-test.yaml +++ b/tests/manifests/dynamo-vllm-smoke-test.yaml @@ -20,11 +20,11 @@ # serving Qwen/Qwen3-0.6B (public, ungated, ~1.2GB). # No HuggingFace token required. # -# Uses the Dynamo 1.2 Kubernetes path: Kubernetes-native discovery plus the -# NATS-backed event plane from the dynamo-platform chart. Request/response -# traffic uses Dynamo's request plane default (TCP). The frontend runs -# KV-cache-aware routing; the local vLLM engine publishes KV events through -# ZMQ and the Dynamo worker runtime relays them onto the NATS event plane. +# Uses the Dynamo 1.4+ Kubernetes path: Kubernetes-native discovery, with +# the request plane on TCP (default) and the KV event plane on ZMQ +# (default) — no bundled NATS relay in between. The frontend runs +# KV-cache-aware routing; the local vLLM engine publishes KV events over +# ZMQ and the KV router consumes them directly, end-to-end. # # Usage: # kubectl apply -f tests/manifests/dynamo-vllm-smoke-test.yaml -n dynamo-system