diff --git a/MTLS.md b/MTLS.md deleted file mode 100644 index e9e6bed7f..000000000 --- a/MTLS.md +++ /dev/null @@ -1,264 +0,0 @@ -# OpenShell Gateway on ROSA: mTLS, OIDC, and External Connectivity - -> **Working memory.** Canonical specifications live in `specs/platform/openshell-gateway*.spec.md`. - -## Result - -**Local openshell CLI successfully connected to the hosted OpenShell gateway on ROSA** via OIDC authentication, created a sandbox, and received a running pod. Full e2e demo: 11/11 pass. - -``` -openshell CLI (local) → NLB (L4) → HAProxy passthrough → gateway pod → sandbox pod -``` - -Demo script: `components/pr-test/e2e-openshell.sh` - -## Architecture - -### Cluster State (namespace: `acp-api-01` + `tenant-a`) - -| Component | Image | Status | -|---|---|---| -| `ambient-control-plane` | `quay.io/ambient_code/acp_control_plane:gateway-oidc-mtls` | Running, reconciling | -| `ambient-api-server` | `quay.io/ambient_code/acp_api_server:gateway-postgres-fix` | Running | -| `openshell-gateway` | `ghcr.io/nvidia/openshell/gateway:0.0.88` | Running (Deployment) | -| `openshell-gateway-db` | `registry.redhat.io/rhel9/postgresql-16:latest` | Running (Deployment) | -| `demo-sandbox` | `ghcr.io/nvidia/openshell-community/sandboxes/base:latest` | Running | - -### Gateway Configuration - -```toml -[openshell.gateway.tls] -cert_path = "/etc/openshell-tls/server/tls.crt" -key_path = "/etc/openshell-tls/server/tls.key" -client_ca_path = "/etc/openshell-tls/client-ca/ca.crt" # RETAINED with OIDC - -[openshell.gateway.auth] -allow_unauthenticated_users = false - -[openshell.gateway.oidc] -issuer = "https://keycloak-acp-api-01.apps.rosa.vteam-stage.7fpc.p3.openshiftapps.com/realms/ambient-code" -audience = "ambient-frontend" -roles_claim = "groups" -admin_role = "ambient-admins" -user_role = "ambient-users" -``` - -### TLS Mode: Optional mTLS (has_ca && has_oidc) - -OpenShell's TLS modes are determined by `require_client_auth = has_client_ca && !has_oidc`: - -| `client_ca_path` | OIDC configured | `require_client_auth` | Mode | -|---|---|---|---| -| set | no | **true** | Full mTLS — all clients must present certs | -| set | yes | **false** | Optional mTLS — certs validated when present, OIDC for others | -| unset | yes | false | HTTPS + OIDC only | -| unset | no | false | HTTPS only (allow_unauthenticated) | - -We use **optional mTLS**: sandbox supervisors still authenticate via client certificates, while CLI users authenticate via OIDC Bearer tokens. HAProxy re-encrypt/passthrough works because client cert is not required. - -## Networking - -### The CloudFront Problem - -ROSA's default ingress path: `Client → CloudFront (L7) → ALB → HAProxy → backend` - -CloudFront breaks gRPC: -- Buffers HTTP/2 streams -- Enforces 30s idle timeout killing streaming connections -- Strips `te: trailers` headers -- Does not support bidirectional streaming - -Both passthrough and re-encrypt Routes through the default ingress controller time out. - -### The Solution: NLB-Backed IngressController - -Created a secondary IngressController with an NLB (L4 TCP): - -```yaml -apiVersion: operator.openshift.io/v1 -kind: IngressController -metadata: - name: grpc - namespace: openshift-ingress-operator -spec: - domain: grpc.vteam-stage.7fpc.p3.openshiftapps.com - endpointPublishingStrategy: - type: LoadBalancerService - loadBalancer: - providerParameters: - type: AWS - aws: - type: NLB - scope: External - routeSelector: - matchLabels: - router: grpc - replicas: 1 -``` - -Passthrough Route labeled for the NLB router: - -```yaml -apiVersion: route.openshift.io/v1 -kind: Route -metadata: - name: openshell-gateway-grpc - namespace: tenant-a - labels: - router: grpc -spec: - host: openshell-gateway-tenant-a.grpc.vteam-stage.7fpc.p3.openshiftapps.com - port: - targetPort: grpc - tls: - termination: passthrough - to: - kind: Service - name: openshell-gateway -``` - -NLB hostname: `af7661bc2f5404ef3b9307fb3ea670bf-86d9f83cd3c5ca72.elb.us-east-1.amazonaws.com` - -### NetworkPolicy (Critical) - -The reconciler creates `openshell-gateway-allow-sandbox` which only allows ingress from pods in the same namespace. Router pods in `openshift-ingress` are blocked. A separate NetworkPolicy is required: - -```yaml -apiVersion: networking.k8s.io/v1 -kind: NetworkPolicy -metadata: - name: openshell-gateway-allow-router - namespace: tenant-a -spec: - podSelector: - matchLabels: - app.kubernetes.io/instance: openshell-gateway - app.kubernetes.io/name: openshell - ingress: - - from: - - namespaceSelector: - matchLabels: - kubernetes.io/metadata.name: openshift-ingress - ports: - - port: 8080 - protocol: TCP - - port: 8081 - protocol: TCP -``` - -Without this, the NLB passthrough silently fails — TLS handshake hangs with zero bytes read. - -## Code Changes - -### 1. `manifests.go` — Keep `client_ca_path` with OIDC (critical fix) - -**File:** `components/ambient-control-plane/internal/gateway/manifests.go` - -Removed the block at lines 154-163 that stripped `client_ca_path` from gateway.toml when OIDC was configured. OpenShell handles this correctly via `require_client_auth = has_ca && !has_oidc` — OIDC presence automatically switches from required to optional mTLS. - -**Before:** OIDC + mTLS were mutually exclusive (code removed `client_ca_path`). -**After:** OIDC + mTLS are complementary (optional mTLS, OIDC for CLI users, mTLS for sandboxes). - -### 2. `reconciler.go` — RHEL postgres image default - -**File:** `components/ambient-control-plane/internal/gateway/reconciler.go` - -Changed default postgres image from `postgres:16` (Docker Hub, rate-limited) to `registry.redhat.io/rhel9/postgresql-16:latest` (matches API server DB pattern). The RHEL image detection (`strings.Contains(pgImage, "rhel")`) correctly switches env vars to `POSTGRESQL_*` format. - -### 3. `kustomize.go` — Database field in Resource struct - -**File:** `components/ambient-sdk/go-sdk/kustomize/kustomize.go` - -Added `Database map[string]any` field to the `Resource` struct. Without this, `acpctl apply` silently dropped the `database:` key from YAML manifests, causing gateways to revert to sqlite. - -### 4. `apply/cmd.go` — Database wiring in acpctl apply - -**File:** `components/ambient-cli/cmd/acpctl/apply/cmd.go` - -Added `databaseFromResource()` helper and wired the `Database` field through both create (`applyGateway`) and patch (`buildGatewayPatch`) paths. - -### 5. `manifests_test.go` — Updated test - -**File:** `components/ambient-control-plane/internal/gateway/manifests_test.go` - -Updated `TestApplyConfigOverrides_OIDCDisablesMTLS` to expect `client_ca_path` retained (not removed) when OIDC is enabled. - -## Images - -| Image | Tag | Registry | -|---|---|---| -| `acp_control_plane` | `gateway-oidc-mtls` | `quay.io/ambient_code/` | -| `acp_api_server` | `gateway-postgres-fix` | `quay.io/ambient_code/` | - -## CLI Connection Flow - -```bash -# 1. Login to acpctl (get OIDC token) -acpctl login --password-grant --username admin --password admin \ - --issuer-url https://keycloak-acp-api-01.apps.rosa.vteam-stage.7fpc.p3.openshiftapps.com/realms/ambient-code \ - --client-id ambient-frontend --client-secret "acp-ZaA3W6_uxDSwZDisz0Yz6A" \ - --url https://ambient-api-acp-api-01.apps.rosa.vteam-stage.7fpc.p3.openshiftapps.com - -# 2. Port-forward to gateway (until DNS is configured for the NLB route) -oc port-forward -n tenant-a svc/openshell-gateway 7443:8080 & - -# 3. Register gateway and inject OIDC token -acpctl gateway setup-cli --project tenant-a --gateway-url "https://localhost:7443" - -# 4. Use openshell CLI -OPENSHELL_GATEWAY_INSECURE=true openshell -g tenant-a-openshell-gateway status -OPENSHELL_GATEWAY_INSECURE=true openshell -g tenant-a-openshell-gateway sandbox create --name demo -OPENSHELL_GATEWAY_INSECURE=true openshell -g tenant-a-openshell-gateway provider list -``` - -## Remaining Work - -### DNS for NLB Route — RESOLVED - -Fixed by using `dnsManagementPolicy: Managed` on the IngressController. OpenShift automatically creates Route53 records for the NLB. The managed hostname `openshell-gateway-tenant-a.grpc.apps.rosa.vteam-stage.7fpc.p3.openshiftapps.com` resolves to the NLB. - -### Reconciler Improvements (Open) - -1. **NetworkPolicy for router ingress**: The reconciler should create `openshell-gateway-allow-router` automatically on OpenShift. See `specs/platform/openshell-gateway-routing.spec.md`. - -2. **Route management**: The reconciler should create/update passthrough Routes with `router: grpc` label. See `specs/platform/openshell-gateway-routing.spec.md`. - -3. **Gateway restart on ConfigMap change**: Needs hash annotation on ConfigMap content. - -4. **Keycloak token TTL**: 5-minute TTL causes frequent expiry. openshell CLI should auto-renew via refresh token. - -5. **Control plane RBAC**: ClusterRole `ambient-control-plane` needs `configmaps`, `persistentvolumeclaims`, `deployments`, `networkpolicies` permissions. Currently patched manually; needs proper fix in `components/manifests/base/rbac/control-plane-clusterrole.yaml`. - -### Commit Changes - -Code changes committed in PR #415. Spec updates in `spec/openshell-gateway-subspecs` branch. - -## Spec References - -| Spec | Scope | -|---|---| -| `specs/platform/openshell-gateway.spec.md` | Core provisioning, reconciler, deployment resources | -| `specs/platform/openshell-gateway-tls.spec.md` | TLS, optional mTLS, cert-manager, SAN management | -| `specs/platform/openshell-gateway-oidc.spec.md` | OIDC authentication, role validation, Keycloak | -| `specs/platform/openshell-gateway-routing.spec.md` | Gateway API, NLB passthrough, NetworkPolicy | -| `specs/platform/openshell-gateway-database.spec.md` | PostgreSQL provisioning, workload switching | - -## Debugging Reference - -### Symptom → Root Cause Map - -| Symptom | Root Cause | -|---|---| -| Route times out (all types) on ROSA | CloudFront L7 CDN in front of default ingress — use NLB | -| TLS handshake: 0 bytes read, immediate EOF | NetworkPolicy blocking router → gateway ingress | -| `DecryptError` in gateway logs | Stale client cert from sandbox after cert rotation | -| grpcurl hangs but openssl s_client works | grpcurl blocked by NetworkPolicy (different source namespace) | -| 503 Service Unavailable from route | SNI mismatch — HAProxy can't match passthrough route hostname | -| `role 'openshell-user' required` | OIDC `roles_claim` not configured — JWT has `groups` not `roles` | -| `invalid peer certificate: UnknownIssuer` | Self-signed CA — use `OPENSHELL_GATEWAY_INSECURE=true` or trust CA | -| `acpctl apply` silently reverts to sqlite | `kustomize.Resource` missing `Database` field | -| Docker Hub `toomanyrequests` for postgres | Changed default to `registry.redhat.io/rhel9/postgresql-16:latest` | -| Cert deletion loop (every 30s) | ConfigMap SANs don't match API SANs exactly | -| `GROUPS` variable returns `1000` | Bash builtin collision — use `USER_GROUPS` instead | -| `openshell gateway add` opens browser | No `--no-browser` flag — write metadata.json directly | -| `openshell sandbox create` hangs | Blocking interactive command — background and poll for pod status | diff --git a/Makefile b/Makefile index 25ffe26f4..0fdc18ade 100755 --- a/Makefile +++ b/Makefile @@ -106,18 +106,16 @@ KIND_HOST ?= LOCAL_IMAGES ?= false LOCAL_RUNNER ?= false LOCAL_VERTEX ?= false -OPENSHELL_USE_GATEWAY ?= true ANTHROPIC_VERTEX_PROJECT_ID ?= $(shell echo $$ANTHROPIC_VERTEX_PROJECT_ID) CLOUD_ML_REGION ?= $(shell echo $$CLOUD_ML_REGION) # Default to ADC location if not set (created by: gcloud auth application-default login) GOOGLE_APPLICATION_CREDENTIALS ?= $(or $(shell echo $$GOOGLE_APPLICATION_CREDENTIALS),$(HOME)/.config/gcloud/application_default_credentials.json) VERTEX_CRED ?= $(GOOGLE_APPLICATION_CREDENTIALS) -# OpenShell Gateway Configuration (OPENSHELL_USE_GATEWAY=true by default) +# OpenShell Gateway Configuration # Provisions tenant namespaces with an OpenShell gateway each. # Override with OPENSHELL_TENANTS="ns1 ns2" to change the set of tenant namespaces. # Skip acpctl apply for specific tenants: SKIP_TENANT_SETUP="tenant-c" -OPENSHELL_USE_GATEWAY ?= true OPENSHELL_TENANTS ?= tenant-a tenant-b tenant-c vteam-product-swarm codebase-maintainers SKIP_TENANT_SETUP ?= AGENT_SANDBOX_VERSION ?= v0.5.1 @@ -193,10 +191,7 @@ update-agent-sandbox: ## Update agent-sandbox CRD version. Usage: make update-ag ##@ Building -MCP_BUILD_TARGETS := build-mcp build-credential-sidecars -ifeq ($(OPENSHELL_USE_GATEWAY),true) MCP_BUILD_TARGETS := -endif build-all: build-runner-openshell build-api-server build-control-plane build-ambient-ui $(MCP_BUILD_TARGETS) ## Build all container images @@ -205,7 +200,6 @@ build-ambient-ui: ## Build ambient-ui image @cd components && $(CONTAINER_ENGINE) build $(PLATFORM_FLAG) $(BUILD_FLAGS) \ -f ambient-ui/Dockerfile \ --build-arg GIT_COMMIT=$(shell git rev-parse HEAD) \ - --build-arg OPENSHELL_USE_GATEWAY=$(OPENSHELL_USE_GATEWAY) \ -t $(AMBIENT_UI_IMAGE) . @echo "$(COLOR_GREEN)✓$(COLOR_RESET) Ambient UI built: $(AMBIENT_UI_IMAGE)" @@ -490,7 +484,7 @@ local-test-dev: ## Run local developer experience tests @echo "$(COLOR_BLUE)▶$(COLOR_RESET) Running local developer experience tests..." @./tests/e2e/local-dev-test.sh $(if $(filter true,$(CI_MODE)),--ci,) -test-openshell-dual-tenant: ## Test dual-tenant OpenShell gateway provisioning (requires kind-up OPENSHELL_USE_GATEWAY=true) +test-openshell-dual-tenant: ## Test dual-tenant OpenShell gateway provisioning (requires kind-up) @echo "$(COLOR_BLUE)▶$(COLOR_RESET) Running dual-tenant OpenShell sandbox provisioning test..." @API_URL="http://localhost:$(KIND_FWD_API_SERVER_PORT)" ./tests/e2e/openshell-dual-tenant.sh @@ -936,16 +930,7 @@ kind-up: preflight-cluster build-cli ## Start kind cluster and deploy the platfo fi @echo "$(COLOR_BLUE)▶$(COLOR_RESET) Waiting for pods..." @./tests/infra/wait-for-ready.sh - @echo "$(COLOR_BLUE)▶$(COLOR_RESET) Configuring OpenShell mode (gateway=$(OPENSHELL_USE_GATEWAY))..." - @kubectl set env deployment/ambient-api-server -n $(NAMESPACE) \ - OPENSHELL_USE_GATEWAY=$(OPENSHELL_USE_GATEWAY) $(QUIET_REDIRECT) - @kubectl set env deployment/ambient-control-plane -n $(NAMESPACE) \ - OPENSHELL_USE_GATEWAY=$(OPENSHELL_USE_GATEWAY) $(QUIET_REDIRECT) - @if [ "$(OPENSHELL_USE_GATEWAY)" = "true" ]; then \ - echo "$(COLOR_GREEN)✓$(COLOR_RESET) OpenShell: gateway mode (default)"; \ - else \ - echo "$(COLOR_GREEN)✓$(COLOR_RESET) OpenShell: pod mode"; \ - fi + @echo "$(COLOR_GREEN)✓$(COLOR_RESET) OpenShell: gateway mode" @echo "$(COLOR_BLUE)▶$(COLOR_RESET) Configuring SSO..." @NAMESPACE=$(NAMESPACE) KIND_FWD_AMBIENT_UI_PORT=$(KIND_FWD_AMBIENT_UI_PORT) KIND_FWD_KEYCLOAK_PORT=$(KIND_FWD_KEYCLOAK_PORT) \ ./scripts/setup-kind-sso.sh @@ -979,7 +964,7 @@ kind-up: preflight-cluster build-cli ## Start kind cluster and deploy the platfo @KIND_CLUSTER_NAME=$(KIND_CLUSTER_NAME) KIND_HTTP_PORT=$(KIND_HTTP_PORT) CONTAINER_ENGINE=$(CONTAINER_ENGINE) ./tests/infra/extract-token.sh @echo "$(COLOR_GREEN)✓$(COLOR_RESET) Kind cluster '$(KIND_CLUSTER_NAME)' ready!" @# OpenShell gateway setup if requested - @if [ "$(OPENSHELL_USE_GATEWAY)" = "true" ] && [ "$(NO_SETUP)" != "true" ]; then \ + @if [ "$(NO_SETUP)" != "true" ]; then \ echo "$(COLOR_BLUE)▶$(COLOR_RESET) Installing OpenShell gateway prerequisites ($(OPENSHELL_TENANTS))..."; \ NAMESPACE=$(NAMESPACE) \ OPENSHELL_TENANTS="$(OPENSHELL_TENANTS)" \ @@ -1023,11 +1008,6 @@ kind-up: preflight-cluster build-cli ## Start kind cluster and deploy the platfo done; \ kill $$PF_PID 2>/dev/null || true; \ fi - @# Vertex AI setup if requested (non-gateway) - @if [ "$(OPENSHELL_USE_GATEWAY)" != "true" ]; then \ - echo "$(COLOR_BLUE)▶$(COLOR_RESET) Configuring Vertex AI..."; \ - $(MAKE) --no-print-directory kind-setup-vertex VERTEX_CRED="$(VERTEX_CRED)"; \ - fi @if [ -f .dev-bootstrap.env ] && [ -f ./scripts/bootstrap-workspace.sh ]; then \ echo "$(COLOR_BLUE)▶$(COLOR_RESET) Bootstrapping developer workspace..."; \ ./scripts/bootstrap-workspace.sh || \ @@ -1367,7 +1347,6 @@ kind-reload-ambient-ui: check-kind check-kubectl check-local-context ## Rebuild @cd components && $(CONTAINER_ENGINE) build $(PLATFORM_FLAG) \ -f ambient-ui/Dockerfile \ --build-arg GIT_COMMIT=$(shell git rev-parse HEAD) \ - --build-arg OPENSHELL_USE_GATEWAY=$(OPENSHELL_USE_GATEWAY) \ -t $(AMBIENT_UI_IMAGE) . $(QUIET_REDIRECT) $(call kind-reload-component,$(AMBIENT_UI_IMAGE),ambient-ui,Ambient UI,ambient-ui) @@ -1514,20 +1493,10 @@ kind-status: check-kind ## Show all kind clusters and their port assignments fi kind-setup-vertex: check-kubectl _kind-require-cluster ## Configure Vertex AI for the kind cluster (VERTEX_CRED=~/.config/gcloud/application_default_credentials.json) - @if [ "$(OPENSHELL_USE_GATEWAY)" = "true" ]; then \ - echo "$(COLOR_BLUE)▶$(COLOR_RESET) Gateway mode — setting up vertex provider declarations"; \ - for ns in $(OPENSHELL_TENANTS); do \ - echo "$(COLOR_BLUE)▶$(COLOR_RESET) Setting up vertex provider in $$ns..."; \ - ./scripts/setup-vertex-provider.sh "$$ns" "$${VERTEX_CRED:-$$HOME/.config/gcloud/application_default_credentials.json}"; \ - done; \ - else \ - NAMESPACE=$(NAMESPACE) \ - GOOGLE_APPLICATION_CREDENTIALS="$(GOOGLE_APPLICATION_CREDENTIALS)" \ - ANTHROPIC_VERTEX_PROJECT_ID="$(ANTHROPIC_VERTEX_PROJECT_ID)" \ - CLOUD_ML_REGION="$(CLOUD_ML_REGION)" \ - AMBIENT_UI_URL="http://$$(if [ -n "$(KIND_HOST)" ]; then echo "$(KIND_HOST)"; else echo "localhost"; fi):$(KIND_FWD_AMBIENT_UI_PORT)" \ - ./scripts/setup-vertex-kind.sh; \ - fi + @for ns in $(OPENSHELL_TENANTS); do \ + echo "$(COLOR_BLUE)▶$(COLOR_RESET) Setting up vertex provider in $$ns..."; \ + ./scripts/setup-vertex-provider.sh "$$ns" "$${VERTEX_CRED:-$$HOME/.config/gcloud/application_default_credentials.json}"; \ + done kind-clean: kind-down ## Alias for kind-down @@ -1660,10 +1629,7 @@ _kind-require-cluster: ## Internal: Fail fast if kind cluster is not running (echo "$(COLOR_RED)✗$(COLOR_RESET) No ambient Kind cluster found. Run 'make kind-up' first, or set KIND_CLUSTER_NAME to an existing cluster." && exit 1) KIND_CORE_IMAGES := $(RUNNER_OPENSHELL_IMAGE) $(API_SERVER_IMAGE) $(CONTROL_PLANE_IMAGE) $(AMBIENT_UI_IMAGE) -KIND_MCP_IMAGES := $(MCP_IMAGE) $(GITHUB_MCP_IMAGE) $(JIRA_MCP_IMAGE) $(K8S_MCP_IMAGE) $(GOOGLE_MCP_IMAGE) -ifeq ($(OPENSHELL_USE_GATEWAY),true) KIND_MCP_IMAGES := -endif _kind-preload-runner: ## Internal: Pull runner image from Quay, retag, and load into kind cluster @echo "$(COLOR_BLUE)▶$(COLOR_RESET) Pre-loading runner image into kind ($(KIND_CLUSTER_NAME))..." diff --git a/components/ambient-cli/demo-openshell.sh b/components/ambient-cli/demo-openshell.sh index 1e0c3e42e..d22f550d8 100644 --- a/components/ambient-cli/demo-openshell.sh +++ b/components/ambient-cli/demo-openshell.sh @@ -12,7 +12,7 @@ # Every command is printed with its output so the workflow is easy to follow. # # Prerequisites: -# - kind-up with OPENSHELL_USE_GATEWAY=true (default) +# - kind-up with OpenShell gateway prerequisites # - acpctl built (make build-cli) # - openshell CLI installed and available in $PATH # - Keycloak reachable (KEYCLOAK_URL, default http://localhost:11880) diff --git a/components/ambient-control-plane/cmd/ambient-control-plane/main.go b/components/ambient-control-plane/cmd/ambient-control-plane/main.go index 629352e97..27a4a308e 100644 --- a/components/ambient-control-plane/cmd/ambient-control-plane/main.go +++ b/components/ambient-control-plane/cmd/ambient-control-plane/main.go @@ -164,7 +164,6 @@ func runKubeMode(ctx context.Context, cfg *config.ControlPlaneConfig) error { PlatformMode: cfg.PlatformMode, MPPConfigNamespace: cfg.MPPConfigNamespace, OpenShellEnabled: cfg.OpenShellEnabled, - OpenShellUseGateway: cfg.OpenShellUseGateway, OpenShellRunnerImage: cfg.OpenShellRunnerImage, OpenShellPolicyName: cfg.OpenShellPolicyName, ServiceIdentity: cfg.ServiceIdentity, @@ -218,56 +217,47 @@ func runKubeMode(ctx context.Context, cfg *config.ControlPlaneConfig) error { inf.RegisterHandler("projects", projectReconciler.Reconcile) inf.RegisterHandler("project_settings", projectSettingsReconciler.Reconcile) - // Initialize API-driven gateway reconciler (if enabled) gatewayErrCh := make(chan error, 1) - if cfg.OpenShellUseGateway { - kubeCfg, kubeErr := buildKubeConfig(cfg.Kubeconfig) - if kubeErr != nil { - return fmt.Errorf("build kubeconfig for gateway reconciler: %w", kubeErr) - } - gwClientset, csErr := kubernetes.NewForConfig(kubeCfg) - if csErr != nil { - return fmt.Errorf("create kubernetes clientset for gateway reconciler: %w", csErr) - } - gwDynamic, dynErr := dynamic.NewForConfig(kubeCfg) - if dynErr != nil { - return fmt.Errorf("create dynamic client for gateway reconciler: %w", dynErr) - } - gwReconciler := reconciler.NewGatewayReconciler(factory, gwDynamic, gwClientset, provisioner, log.Logger) - go func() { - gatewayErrCh <- gwReconciler.Run(ctx) - }() - log.Info().Msg("gateway reconciler enabled") + kubeCfg, kubeErr := buildKubeConfig(cfg.Kubeconfig) + if kubeErr != nil { + return fmt.Errorf("build kubeconfig for gateway reconciler: %w", kubeErr) + } + gwClientset, csErr := kubernetes.NewForConfig(kubeCfg) + if csErr != nil { + return fmt.Errorf("create kubernetes clientset for gateway reconciler: %w", csErr) + } + gwDynamic, dynErr := dynamic.NewForConfig(kubeCfg) + if dynErr != nil { + return fmt.Errorf("create dynamic client for gateway reconciler: %w", dynErr) + } + gwReconciler := reconciler.NewGatewayReconciler(factory, gwDynamic, gwClientset, provisioner, log.Logger) + go func() { + gatewayErrCh <- gwReconciler.Run(ctx) + }() + log.Info().Msg("gateway reconciler enabled") + + var resolveCred openshell.CredentialResolver + if cfg.OpenShellGatewayTLSEnabled { + tlsResolver := openshell.NewTLSResolver(provisionerKube, cfg.OpenShellGatewayClientTLSSecret, cfg.OpenShellGatewayTLSServerName, log.Logger) + resolveCred = tlsResolver.CredentialsForNamespace + log.Info().Str("secret", cfg.OpenShellGatewayClientTLSSecret).Str("server_name", cfg.OpenShellGatewayTLSServerName).Msg("OpenShell gateway TLS enabled") } else { - close(gatewayErrCh) - } - - var gateway *openshell.GatewayClient - if cfg.OpenShellUseGateway { - var resolveCred openshell.CredentialResolver - if cfg.OpenShellGatewayTLSEnabled { - tlsResolver := openshell.NewTLSResolver(provisionerKube, cfg.OpenShellGatewayClientTLSSecret, cfg.OpenShellGatewayTLSServerName, log.Logger) - resolveCred = tlsResolver.CredentialsForNamespace - log.Info().Str("secret", cfg.OpenShellGatewayClientTLSSecret).Str("server_name", cfg.OpenShellGatewayTLSServerName).Msg("OpenShell gateway TLS enabled") - } else { - resolveCred = openshell.InsecureResolver() - log.Info().Msg("OpenShell gateway TLS disabled (plaintext)") - } - gateway = openshell.NewGatewayClient(cfg.OpenShellGatewayServiceName, cfg.OpenShellGatewayGRPCPort, resolveCred, cfg.OpenShellGatewaySATokenPath, log.Logger, openshell.WithTokenProvider(tokenProvider)) - defer func() { - if err := gateway.Close(); err != nil { - log.Warn().Err(err).Msg("failed to close gateway client") - } - }() - log.Info().Msg("OpenShell gateway mode enabled") + resolveCred = openshell.InsecureResolver() + log.Info().Msg("OpenShell gateway TLS disabled (plaintext)") } + gateway := openshell.NewGatewayClient(cfg.OpenShellGatewayServiceName, cfg.OpenShellGatewayGRPCPort, resolveCred, cfg.OpenShellGatewaySATokenPath, log.Logger, openshell.WithTokenProvider(tokenProvider)) + defer func() { + if err := gateway.Close(); err != nil { + log.Warn().Err(err).Msg("failed to close gateway client") + } + }() sessionReconcilers := createSessionReconcilers(cfg.Reconcilers, factory, kube, projectKube, provisioner, gateway, kubeReconcilerCfg, log.Logger, inf) for _, sessionRec := range sessionReconcilers { inf.RegisterHandler("sessions", sessionRec.Reconcile) } - podSyncer := reconciler.NewPodStatusSyncer(factory, provisionerKube, gateway, cfg.OpenShellUseGateway, cfg.PlatformMode, cfg.MPPConfigNamespace, log.Logger) + podSyncer := reconciler.NewPodStatusSyncer(factory, provisionerKube, gateway, cfg.PlatformMode, cfg.MPPConfigNamespace, log.Logger) tsErrCh := make(chan error, 1) go func() { diff --git a/components/ambient-control-plane/internal/config/config.go b/components/ambient-control-plane/internal/config/config.go index 5d5820753..34fa30d7e 100755 --- a/components/ambient-control-plane/internal/config/config.go +++ b/components/ambient-control-plane/internal/config/config.go @@ -50,7 +50,6 @@ type ControlPlaneConfig struct { NoProxy string ImagePullSecret string OpenShellEnabled bool - OpenShellUseGateway bool OpenShellRunnerImage string OpenShellPolicyName string OpenShellGatewayServiceName string @@ -121,7 +120,6 @@ func Load() (*ControlPlaneConfig, error) { NoProxy: os.Getenv("NO_PROXY"), ImagePullSecret: os.Getenv("IMAGE_PULL_SECRET"), OpenShellEnabled: os.Getenv("OPENSHELL_ENABLED") == "true", - OpenShellUseGateway: os.Getenv("OPENSHELL_USE_GATEWAY") == "true", OpenShellRunnerImage: envOrDefault("OPENSHELL_RUNNER_IMAGE", "quay.io/ambient_code/acp_runner_openshell:latest"), OpenShellPolicyName: envOrDefault("OPENSHELL_POLICY_CONFIGMAP", "openshell-policy"), OpenShellGatewayServiceName: envOrDefault("OPENSHELL_GATEWAY_SERVICE_NAME", "openshell-gateway"), diff --git a/components/ambient-control-plane/internal/reconciler/gateway_reconciler.go b/components/ambient-control-plane/internal/reconciler/gateway_reconciler.go index f6cc74c32..49d1a8fc8 100644 --- a/components/ambient-control-plane/internal/reconciler/gateway_reconciler.go +++ b/components/ambient-control-plane/internal/reconciler/gateway_reconciler.go @@ -51,6 +51,24 @@ var gatewayGVR = schema.GroupVersionResource{ Resource: "gateways", } +var osRouteGVR = schema.GroupVersionResource{ + Group: "route.openshift.io", + Version: "v1", + Resource: "routes", +} + +var ingressControllerGVR = schema.GroupVersionResource{ + Group: "operator.openshift.io", + Version: "v1", + Resource: "ingresscontrollers", +} + +var networkPolicyGVR = schema.GroupVersionResource{ + Group: "networking.k8s.io", + Version: "v1", + Resource: "networkpolicies", +} + const ( trustedCAConfigMapName = "gateway-trusted-ca" trustedCAKey = "ca-bundle.crt" @@ -71,6 +89,7 @@ type GatewayReconciler struct { gatewayAPIGatewayName string gatewayAPIGatewayNamespace string baseDomain string + nlbRouteDomain string } func NewGatewayReconciler( @@ -119,12 +138,16 @@ func (r *GatewayReconciler) Run(ctx context.Context) error { r.hasCertManager = r.detectCertManager() r.baseDomain = r.detectBaseDomain(ctx) r.hasGatewayAPI = r.detectGatewayAPI(ctx) + if r.isOpenShift && !r.hasGatewayAPI { + r.nlbRouteDomain = r.detectNLBRouteDomain(ctx) + } r.logger.Info(). Int("manifest_files", len(manifests)). Bool("openshift", r.isOpenShift). Bool("cert_manager", r.hasCertManager). Bool("gateway_api", r.hasGatewayAPI). Str("base_domain", r.baseDomain). + Str("nlb_route_domain", r.nlbRouteDomain). Dur("interval", gatewaySyncInterval). Msg("gateway reconciler started") @@ -407,6 +430,9 @@ func (r *GatewayReconciler) detectOpenShift() bool { func (r *GatewayReconciler) reconcileGRPCRoute(ctx context.Context, projectClient *sdkclient.Client, gw *types.Gateway, namespace string) error { if !r.hasGatewayAPI { + if r.isOpenShift { + return r.reconcilePassthroughRoute(ctx, projectClient, gw, namespace) + } return nil } @@ -425,7 +451,7 @@ func (r *GatewayReconciler) reconcileGRPCRoute(ctx context.Context, projectClien return nil } - stsUID, err := r.getWorkloadUID(ctx, namespace, routeName) + stsUID, _, err := r.getWorkloadUID(ctx, namespace, routeName) if err != nil { r.logger.Info().Err(err).Str("namespace", namespace).Msg("workload not yet available, creating GRPCRoute without OwnerReference (will be set on next reconcile)") stsUID = "" @@ -640,15 +666,16 @@ func (r *GatewayReconciler) readCACert(ctx context.Context, namespace string) (s return string(decoded), nil } -func (r *GatewayReconciler) getWorkloadUID(ctx context.Context, namespace, name string) (string, error) { - for _, res := range []string{"deployments", "statefulsets"} { +func (r *GatewayReconciler) getWorkloadUID(ctx context.Context, namespace, name string) (string, string, error) { + kinds := map[string]string{"deployments": "Deployment", "statefulsets": "StatefulSet"} + for res, kind := range kinds { gvr := schema.GroupVersionResource{Group: "apps", Version: "v1", Resource: res} obj, err := r.dynamicClient.Resource(gvr).Namespace(namespace).Get(ctx, name, metav1.GetOptions{}) if err == nil { - return string(obj.GetUID()), nil + return string(obj.GetUID()), kind, nil } } - return "", fmt.Errorf("get workload %s: not found as deployment or statefulset", name) + return "", "", fmt.Errorf("get workload %s: not found as deployment or statefulset", name) } func (r *GatewayReconciler) reconcileGRPCRouteAddress(ctx context.Context, projectClient *sdkclient.Client, gw *types.Gateway, hostname string) error { @@ -1095,3 +1122,189 @@ func (r *GatewayReconciler) deleteIfExists(ctx context.Context, gvr schema.Group } return nil } + +func (r *GatewayReconciler) reconcilePassthroughRoute(ctx context.Context, projectClient *sdkclient.Client, gw *types.Gateway, namespace string) error { + routeName := "openshell-gateway-grpc" + + routeDomain := r.nlbRouteDomain + if routeDomain == "" { + routeDomain = r.baseDomain + } + if routeDomain == "" { + r.logger.Debug().Str("gateway_name", gw.Name).Msg("no route domain available, skipping passthrough Route") + return nil + } + + var hostname string + if gw.Route != nil && gw.Route.Host != "" { + hostname = gw.Route.Host + } else { + hostname = fmt.Sprintf("openshell-gateway-%s.%s", namespace, routeDomain) + } + + stsUID, workloadKind, err := r.getWorkloadUID(ctx, namespace, "openshell-gateway") + if err != nil { + r.logger.Info().Err(err).Str("namespace", namespace).Msg("workload not yet available, creating passthrough Route without OwnerReference") + stsUID = "" + workloadKind = "" + } + + route := r.buildPassthroughRoute(namespace, routeName, hostname, stsUID, workloadKind, r.nlbRouteDomain != "") + if err := r.applyUnstructured(ctx, osRouteGVR, namespace, routeName, route); err != nil { + return fmt.Errorf("apply passthrough Route: %w", err) + } + + if err := r.reconcileRouterNetworkPolicy(ctx, namespace, stsUID, workloadKind); err != nil { + r.logger.Warn().Err(err).Str("namespace", namespace).Msg("failed to reconcile router NetworkPolicy") + } + + routeAddress := "grpcs://" + hostname + if gw.RouteAddress == routeAddress { + return nil + } + + patch := types.NewGatewayPatchBuilder().RouteAddress(routeAddress).Build() + if _, err := projectClient.Gateways().Update(ctx, gw.ID, patch); err != nil { + return fmt.Errorf("update routeAddress for gateway %s: %w", gw.ID, err) + } + + r.logger.Info(). + Str("gateway_name", gw.Name). + Str("route_address", routeAddress). + Msg("updated gateway routeAddress (passthrough Route)") + return nil +} + +func (r *GatewayReconciler) buildPassthroughRoute(namespace, routeName, hostname, stsUID, workloadKind string, useNLBRouter bool) *unstructured.Unstructured { + labels := map[string]interface{}{ + "app.kubernetes.io/name": "openshell", + "app.kubernetes.io/component": "gateway", + "app.kubernetes.io/managed-by": "agent-control-plane", + } + if useNLBRouter { + labels["router"] = "grpc" + } + metadata := map[string]interface{}{ + "name": routeName, + "namespace": namespace, + "labels": labels, + "annotations": map[string]interface{}{ + "haproxy.router.openshift.io/timeout": "3600s", + }, + } + if stsUID != "" && workloadKind != "" { + metadata["ownerReferences"] = []interface{}{ + map[string]interface{}{ + "apiVersion": "apps/v1", + "kind": workloadKind, + "name": "openshell-gateway", + "uid": stsUID, + "controller": true, + }, + } + } + return &unstructured.Unstructured{ + Object: map[string]interface{}{ + "apiVersion": "route.openshift.io/v1", + "kind": "Route", + "metadata": metadata, + "spec": map[string]interface{}{ + "host": hostname, + "port": map[string]interface{}{ + "targetPort": "grpc", + }, + "tls": map[string]interface{}{ + "termination": "passthrough", + }, + "to": map[string]interface{}{ + "kind": "Service", + "name": "openshell-gateway", + "weight": int64(100), + }, + }, + }, + } +} + +func (r *GatewayReconciler) reconcileRouterNetworkPolicy(ctx context.Context, namespace, stsUID, workloadKind string) error { + policyName := "openshell-gateway-allow-router" + metadata := map[string]interface{}{ + "name": policyName, + "namespace": namespace, + "labels": map[string]interface{}{ + "app.kubernetes.io/name": "openshell", + "app.kubernetes.io/component": "gateway", + "app.kubernetes.io/managed-by": "agent-control-plane", + }, + } + if stsUID != "" && workloadKind != "" { + metadata["ownerReferences"] = []interface{}{ + map[string]interface{}{ + "apiVersion": "apps/v1", + "kind": workloadKind, + "name": "openshell-gateway", + "uid": stsUID, + "controller": true, + }, + } + } + policy := &unstructured.Unstructured{ + Object: map[string]interface{}{ + "apiVersion": "networking.k8s.io/v1", + "kind": "NetworkPolicy", + "metadata": metadata, + "spec": map[string]interface{}{ + "podSelector": map[string]interface{}{ + "matchLabels": map[string]interface{}{ + "app.kubernetes.io/instance": "openshell-gateway", + "app.kubernetes.io/name": "openshell", + }, + }, + "ingress": []interface{}{ + map[string]interface{}{ + "from": []interface{}{ + map[string]interface{}{ + "namespaceSelector": map[string]interface{}{ + "matchLabels": map[string]interface{}{ + "kubernetes.io/metadata.name": "openshift-ingress", + }, + }, + }, + }, + "ports": []interface{}{ + map[string]interface{}{ + "port": int64(8080), + "protocol": "TCP", + }, + map[string]interface{}{ + "port": int64(8081), + "protocol": "TCP", + }, + }, + }, + }, + }, + }, + } + return r.applyUnstructured(ctx, networkPolicyGVR, namespace, policyName, policy) +} + +func (r *GatewayReconciler) detectNLBRouteDomain(ctx context.Context) string { + if domain := os.Getenv("NLB_ROUTE_DOMAIN"); domain != "" { + return domain + } + + ic, err := r.dynamicClient.Resource(ingressControllerGVR).Namespace("openshift-ingress-operator").Get(ctx, "grpc", metav1.GetOptions{}) + if err != nil { + if !k8serrors.IsNotFound(err) { + r.logger.Warn().Err(err).Msg("failed to read IngressController/grpc for NLB domain detection") + } + return "" + } + + domain, _, _ := unstructured.NestedString(ic.Object, "spec", "domain") + if domain != "" { + r.logger.Info().Str("nlb_route_domain", domain).Msg("detected NLB route domain from IngressController/grpc") + } + return domain +} diff --git a/components/ambient-control-plane/internal/reconciler/kube_reconciler.go b/components/ambient-control-plane/internal/reconciler/kube_reconciler.go index f33f4fc43..4d5019cfa 100644 --- a/components/ambient-control-plane/internal/reconciler/kube_reconciler.go +++ b/components/ambient-control-plane/internal/reconciler/kube_reconciler.go @@ -105,7 +105,6 @@ type KubeReconcilerConfig struct { PlatformMode string MPPConfigNamespace string OpenShellEnabled bool - OpenShellUseGateway bool OpenShellRunnerImage string OpenShellPolicyName string ServiceIdentity string @@ -318,10 +317,7 @@ func (r *SimpleKubeReconciler) deprovisionAsync(session types.Session, nextPhase } func (r *SimpleKubeReconciler) provisionSession(ctx context.Context, session types.Session) error { - if r.cfg.OpenShellUseGateway { - return r.provisionSessionSandbox(ctx, session) - } - return r.provisionSessionPod(ctx, session) + return r.provisionSessionSandbox(ctx, session) } func (r *SimpleKubeReconciler) provisionSessionPod(ctx context.Context, session types.Session) error { @@ -673,9 +669,6 @@ func (r *SimpleKubeReconciler) appendPromptToEntrypoint(ctx context.Context, ent } func (r *SimpleKubeReconciler) inferenceExecEnv(agent *types.Agent) map[string]string { - if !r.cfg.OpenShellUseGateway { - return nil - } baseURL := "https://inference.local" if agent != nil { if v, ok := agent.Environment["ANTHROPIC_BASE_URL"]; ok && v != "" { @@ -1683,10 +1676,7 @@ func (r *SimpleKubeReconciler) resolveMaxSeq(ctx context.Context, sdk *sdkclient } func (r *SimpleKubeReconciler) buildSandboxEnv(ctx context.Context, session types.Session, projectName string, sdk *sdkclient.Client, providerNames []string, hasMLflowProvider bool) map[string]string { - workspacePath := "/workspace" - if r.cfg.OpenShellUseGateway { - workspacePath = "/sandbox/workspace" - } + workspacePath := "/sandbox/workspace" env := map[string]string{ "SESSION_ID": session.ID, @@ -1718,18 +1708,11 @@ func (r *SimpleKubeReconciler) buildSandboxEnv(ctx context.Context, session type } } - if r.cfg.OpenShellUseGateway { - // In gateway mode the supervisor proxy handles all inference via - // inference.local — regardless of provider (Vertex, Anthropic, etc.). - // The runner must activate inference routing mode so requests go - // through the proxy instead of directly to the provider API. - env["ACP_OPENSHELL_INFERENCE"] = "true" - // Set at sandbox level so every tool (claude, opencode, etc.) gets - // them — not just processes launched through a specific wrapper. - env["ANTHROPIC_BASE_URL"] = "https://inference.local" - env["ANTHROPIC_API_KEY"] = "notused" - env["CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS"] = "1" - } else if r.cfg.VertexEnabled { + env["ACP_OPENSHELL_INFERENCE"] = "true" + env["ANTHROPIC_BASE_URL"] = "https://inference.local" + env["ANTHROPIC_API_KEY"] = "notused" + env["CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS"] = "1" + if false && r.cfg.VertexEnabled { env["USE_VERTEX"] = "1" env["CLAUDE_CODE_USE_VERTEX"] = "1" env["ANTHROPIC_VERTEX_PROJECT_ID"] = r.cfg.VertexProjectID @@ -1761,10 +1744,7 @@ func (r *SimpleKubeReconciler) buildSandboxEnv(ctx context.Context, session type env["HTTPS_PROXY"] = r.cfg.HTTPSProxy } noProxy := r.cfg.NoProxy - if r.cfg.OpenShellUseGateway && noProxy == "" { - // In gateway mode the sandbox network namespace has no direct route - // to cluster IPs or DNS — all traffic must traverse the supervisor - // proxy. Only loopback is excluded. + if noProxy == "" { noProxy = "127.0.0.1,localhost" } if noProxy != "" { @@ -1799,10 +1779,7 @@ func (r *SimpleKubeReconciler) buildSandboxEnv(ctx context.Context, session type } func (r *SimpleKubeReconciler) deprovisionSession(ctx context.Context, session types.Session, nextPhase string) error { - if r.cfg.OpenShellUseGateway { - return r.deprovisionSessionSandbox(ctx, session, nextPhase) - } - return r.deprovisionSessionPod(ctx, session, nextPhase) + return r.deprovisionSessionSandbox(ctx, session, nextPhase) } func (r *SimpleKubeReconciler) deprovisionSessionPod(ctx context.Context, session types.Session, nextPhase string) error { @@ -1913,10 +1890,7 @@ func (r *SimpleKubeReconciler) finalSandboxSnapshot(ctx context.Context, session } func (r *SimpleKubeReconciler) cleanupSession(ctx context.Context, session types.Session) error { - if r.cfg.OpenShellUseGateway { - return r.cleanupSessionSandbox(ctx, session) - } - return r.cleanupSessionPod(ctx, session) + return r.cleanupSessionSandbox(ctx, session) } func (r *SimpleKubeReconciler) cleanupSessionPod(ctx context.Context, session types.Session) error { @@ -2496,7 +2470,7 @@ func (r *SimpleKubeReconciler) ensurePod(ctx context.Context, namespace string, saName := serviceAccountName(session.ID) runnerImage := r.cfg.RunnerImage - if r.cfg.OpenShellUseGateway && r.cfg.OpenShellRunnerImage != "" { + if r.cfg.OpenShellRunnerImage != "" { runnerImage = r.cfg.OpenShellRunnerImage } imagePullPolicy := "Always" @@ -2505,11 +2479,9 @@ func (r *SimpleKubeReconciler) ensurePod(ctx context.Context, namespace string, } labels := sessionLabels(session.ID, session.ProjectID) - useMCPSidecar := r.cfg.MCPImage != "" && r.cfg.CPTokenURL != "" && r.cfg.CPTokenPublicKey != "" && !r.cfg.OpenShellUseGateway - if r.cfg.OpenShellUseGateway && r.cfg.MCPImage != "" { - r.logger.Debug().Str("session_id", session.ID).Msg("MCP sidecar disabled: OPENSHELL_USE_GATEWAY is enabled") - } else if r.cfg.MCPImage != "" && !useMCPSidecar { - r.logger.Warn().Str("session_id", session.ID).Msg("MCP sidecar disabled: CP_TOKEN_URL or CPTokenPublicKey not configured") + useMCPSidecar := false + if r.cfg.MCPImage != "" { + r.logger.Debug().Str("session_id", session.ID).Msg("MCP sidecar disabled: gateway handles MCP") } containers := []interface{}{ @@ -2547,7 +2519,7 @@ func (r *SimpleKubeReconciler) ensurePod(ctx context.Context, namespace string, credentialSidecarMode := false var credTmpVolumes []interface{} - if r.cfg.CPTokenURL != "" && r.cfg.CPTokenPublicKey != "" && !r.cfg.OpenShellUseGateway { + if false { credSidecars, credMCPURLs, credTmpVols := r.buildCredentialSidecars(session.ID, namespace, credentialIDs, r.cfg.OpenShellEnabled) credTmpVolumes = credTmpVols containers = append(containers, credSidecars...) @@ -2963,9 +2935,6 @@ func (r *SimpleKubeReconciler) resolveCredentialIDs(ctx context.Context, sdk *sd totalBindings := len(agentBindings) + len(projectBindings) + len(globalBindings) if totalBindings == 0 { // Only log when NOT using gateway - gateway resolves credentials from agent provider declarations - if !r.cfg.OpenShellUseGateway { - r.logger.Info().Str("project_id", projectID).Msg("no credential bindings found for project; no credentials will be injected") - } return map[string]string{}, nil } diff --git a/components/ambient-control-plane/internal/reconciler/kube_reconciler_test.go b/components/ambient-control-plane/internal/reconciler/kube_reconciler_test.go index d3e8fe25a..9f0ac4b3a 100644 --- a/components/ambient-control-plane/internal/reconciler/kube_reconciler_test.go +++ b/components/ambient-control-plane/internal/reconciler/kube_reconciler_test.go @@ -260,43 +260,29 @@ func TestCredentialMCPURLsJSON(t *testing.T) { } func TestCredentialSidecarsGating_GatewayMode(t *testing.T) { - // This test verifies the gating logic that happens in ensurePod before buildCredentialSidecars is called. - // When OpenShellUseGateway=true, buildCredentialSidecars should NOT be called even if credentials exist. - tests := []struct { name string cpTokenURL string cpTokenPublicKey string - openShellUseGateway bool shouldBuildSidecars bool }{ { - name: "gateway disabled, tokens configured", - cpTokenURL: "http://cp:8080", - cpTokenPublicKey: "test-key", - openShellUseGateway: false, - shouldBuildSidecars: true, - }, - { - name: "gateway enabled, tokens configured", + name: "tokens configured — gateway handles credentials", cpTokenURL: "http://cp:8080", cpTokenPublicKey: "test-key", - openShellUseGateway: true, shouldBuildSidecars: false, }, { - name: "gateway disabled, missing token URL", + name: "missing token URL — gateway handles credentials", cpTokenURL: "", cpTokenPublicKey: "test-key", - openShellUseGateway: false, shouldBuildSidecars: false, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - // This mirrors the gating logic from ensurePod line 631 - shouldCall := tt.cpTokenURL != "" && tt.cpTokenPublicKey != "" && !tt.openShellUseGateway + shouldCall := false if shouldCall != tt.shouldBuildSidecars { t.Errorf("shouldCallBuildCredentialSidecars = %v, want %v", shouldCall, tt.shouldBuildSidecars) @@ -381,51 +367,25 @@ func TestResolveSandboxImage_EmptyAllowlist(t *testing.T) { func TestUseMCPSidecar_GatewayModeDisablesMCP(t *testing.T) { tests := []struct { - name string - mcpImage string - cpTokenURL string - cpTokenPublicKey string - openShellUseGateway bool - expectedUseMCP bool + name string + mcpImage string + expectedUseMCP bool }{ { - name: "all configured, gateway disabled", - mcpImage: "mcp:latest", - cpTokenURL: "http://cp:8080", - cpTokenPublicKey: "test-key", - openShellUseGateway: false, - expectedUseMCP: true, - }, - { - name: "all configured, gateway enabled", - mcpImage: "mcp:latest", - cpTokenURL: "http://cp:8080", - cpTokenPublicKey: "test-key", - openShellUseGateway: true, - expectedUseMCP: false, - }, - { - name: "missing token URL, gateway disabled", - mcpImage: "mcp:latest", - cpTokenURL: "", - cpTokenPublicKey: "test-key", - openShellUseGateway: false, - expectedUseMCP: false, + name: "MCP image configured — gateway handles MCP", + mcpImage: "mcp:latest", + expectedUseMCP: false, }, { - name: "missing image, gateway enabled", - mcpImage: "", - cpTokenURL: "http://cp:8080", - cpTokenPublicKey: "test-key", - openShellUseGateway: true, - expectedUseMCP: false, + name: "no MCP image", + mcpImage: "", + expectedUseMCP: false, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - // This mirrors the logic from ensurePod - useMCPSidecar := tt.mcpImage != "" && tt.cpTokenURL != "" && tt.cpTokenPublicKey != "" && !tt.openShellUseGateway + useMCPSidecar := false if useMCPSidecar != tt.expectedUseMCP { t.Errorf("useMCPSidecar = %v, want %v", useMCPSidecar, tt.expectedUseMCP) @@ -665,7 +625,6 @@ func TestBuildSandboxEnv_MLflowInjection(t *testing.T) { MLflowWorkspace: tt.workspace, MLflowEnableAsyncTraceLogging: "true", MLflowAutologExcludeFlavors: tt.excludeFlavors, - OpenShellUseGateway: true, }, provisioner: &mockProvisioner{}, } diff --git a/components/ambient-control-plane/internal/reconciler/pod_sync.go b/components/ambient-control-plane/internal/reconciler/pod_sync.go index 11c835389..cf88e4c3e 100644 --- a/components/ambient-control-plane/internal/reconciler/pod_sync.go +++ b/components/ambient-control-plane/internal/reconciler/pod_sync.go @@ -26,19 +26,17 @@ type PodStatusSyncer struct { factory *SDKClientFactory kube *kubeclient.KubeClient gateway *openshell.GatewayClient - useGateway bool platformMode string mppConfigNamespace string logger zerolog.Logger errorFirstSeen map[string]time.Time } -func NewPodStatusSyncer(factory *SDKClientFactory, kube *kubeclient.KubeClient, gateway *openshell.GatewayClient, useGateway bool, platformMode, mppConfigNamespace string, logger zerolog.Logger) *PodStatusSyncer { +func NewPodStatusSyncer(factory *SDKClientFactory, kube *kubeclient.KubeClient, gateway *openshell.GatewayClient, platformMode, mppConfigNamespace string, logger zerolog.Logger) *PodStatusSyncer { return &PodStatusSyncer{ factory: factory, kube: kube, gateway: gateway, - useGateway: useGateway, platformMode: platformMode, mppConfigNamespace: mppConfigNamespace, logger: logger.With().Str("component", "pod-status-syncer").Logger(), @@ -63,20 +61,7 @@ func (s *PodStatusSyncer) Run(ctx context.Context) error { } func (s *PodStatusSyncer) syncOnce(ctx context.Context) { - if s.useGateway { - s.syncGatewaySandboxes(ctx) - return - } - - namespaces, err := s.listManagedNamespaces(ctx) - if err != nil { - s.logger.Warn().Err(err).Msg("failed to list managed namespaces") - return - } - - for _, ns := range namespaces { - s.syncNamespace(ctx, ns) - } + s.syncGatewaySandboxes(ctx) } func (s *PodStatusSyncer) listManagedNamespaces(ctx context.Context) ([]string, error) { diff --git a/components/ambient-control-plane/internal/reconciler/project_reconciler.go b/components/ambient-control-plane/internal/reconciler/project_reconciler.go index 5bf530710..6912ae20b 100644 --- a/components/ambient-control-plane/internal/reconciler/project_reconciler.go +++ b/components/ambient-control-plane/internal/reconciler/project_reconciler.go @@ -193,8 +193,8 @@ func (r *ProjectReconciler) controlPlaneRBACRules() []interface{} { rules := []interface{}{ map[string]interface{}{ "apiGroups": []interface{}{""}, - "resources": []interface{}{"secrets", "serviceaccounts", "services"}, - "verbs": []interface{}{"get", "list", "watch", "create", "delete", "deletecollection"}, + "resources": []interface{}{"secrets", "serviceaccounts", "services", "persistentvolumeclaims", "configmaps"}, + "verbs": []interface{}{"get", "list", "watch", "create", "update", "delete", "deletecollection"}, }, map[string]interface{}{ "apiGroups": []interface{}{""}, @@ -203,8 +203,28 @@ func (r *ProjectReconciler) controlPlaneRBACRules() []interface{} { }, map[string]interface{}{ "apiGroups": []interface{}{"rbac.authorization.k8s.io"}, - "resources": []interface{}{"rolebindings"}, - "verbs": []interface{}{"get", "list", "watch", "create", "delete"}, + "resources": []interface{}{"roles", "rolebindings"}, + "verbs": []interface{}{"get", "list", "watch", "create", "update", "delete"}, + }, + map[string]interface{}{ + "apiGroups": []interface{}{"apps"}, + "resources": []interface{}{"statefulsets", "deployments"}, + "verbs": []interface{}{"get", "list", "watch", "create", "update", "delete"}, + }, + map[string]interface{}{ + "apiGroups": []interface{}{"batch"}, + "resources": []interface{}{"jobs"}, + "verbs": []interface{}{"get", "list", "watch", "create", "update", "delete"}, + }, + map[string]interface{}{ + "apiGroups": []interface{}{"networking.k8s.io"}, + "resources": []interface{}{"networkpolicies"}, + "verbs": []interface{}{"get", "list", "watch", "create", "update", "delete"}, + }, + map[string]interface{}{ + "apiGroups": []interface{}{"route.openshift.io"}, + "resources": []interface{}{"routes", "routes/custom-host"}, + "verbs": []interface{}{"get", "list", "watch", "create", "update", "delete"}, }, } @@ -220,11 +240,6 @@ func (r *ProjectReconciler) controlPlaneRBACRules() []interface{} { "resources": []interface{}{"imagestreams", "imagestreamtags", "imagestreamimages"}, "verbs": []interface{}{"get", "list", "create", "update", "delete"}, }, - map[string]interface{}{ - "apiGroups": []interface{}{"route.openshift.io"}, - "resources": []interface{}{"routes"}, - "verbs": []interface{}{"get", "list", "create", "update", "delete"}, - }, ) } diff --git a/components/ambient-ui/Dockerfile b/components/ambient-ui/Dockerfile index 0d943f8fe..f27d1e7a4 100644 --- a/components/ambient-ui/Dockerfile +++ b/components/ambient-ui/Dockerfile @@ -35,9 +35,6 @@ ENV NEXT_TELEMETRY_DISABLED=1 ARG GIT_COMMIT=unknown ENV NEXT_PUBLIC_GIT_COMMIT=$GIT_COMMIT -ARG OPENSHELL_USE_GATEWAY=false -ENV NEXT_PUBLIC_OPENSHELL_USE_GATEWAY=$OPENSHELL_USE_GATEWAY - RUN npm run build # Prepare standalone output with OpenShift-compatible permissions in the builder diff --git a/components/manifests/base/rbac/control-plane-clusterrole.yaml b/components/manifests/base/rbac/control-plane-clusterrole.yaml index f8a8cd4a2..b63bc5538 100644 --- a/components/manifests/base/rbac/control-plane-clusterrole.yaml +++ b/components/manifests/base/rbac/control-plane-clusterrole.yaml @@ -28,15 +28,19 @@ rules: # ConfigMaps (watch platform-config, create/delete gateway configs including CA bundles) - apiGroups: [""] resources: ["configmaps"] - verbs: ["get", "watch", "list", "create", "update", "patch", "delete"] + verbs: ["get", "list", "watch", "create", "update", "patch", "delete", "deletecollection"] # Deployments, StatefulSets (gateway provisioning + cleanup) - apiGroups: ["apps"] resources: ["deployments", "statefulsets", "statefulsets/finalizers"] - verbs: ["create", "get", "list", "update", "patch", "delete"] + verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] +# PersistentVolumeClaims (gateway database storage) +- apiGroups: [""] + resources: ["persistentvolumeclaims"] + verbs: ["get", "list", "watch", "create", "update", "delete", "deletecollection"] # NetworkPolicies (gateway isolation + session cleanup) - apiGroups: ["networking.k8s.io"] resources: ["networkpolicies"] - verbs: ["create", "get", "list", "update", "patch", "delete", "deletecollection"] + verbs: ["get", "list", "watch", "create", "update", "patch", "delete", "deletecollection"] # ClusterRoles and ClusterRoleBindings (gateway RBAC) - apiGroups: ["rbac.authorization.k8s.io"] resources: ["clusterroles", "clusterrolebindings"] @@ -60,6 +64,14 @@ rules: - apiGroups: ["gateway.networking.k8s.io"] resources: ["gateways"] verbs: ["get", "list"] +# OpenShift Routes (NLB passthrough Route for gateway exposure on ROSA/AWS) +- apiGroups: ["route.openshift.io"] + resources: ["routes", "routes/custom-host"] + verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] +# IngressControllers (auto-detect NLB domain for passthrough Route hostnames) +- apiGroups: ["operator.openshift.io"] + resources: ["ingresscontrollers"] + verbs: ["get"] # Cluster ingress config (auto-detect base domain for Gateway API hostnames) - apiGroups: ["config.openshift.io"] resources: ["ingresses"] diff --git a/components/manifests/overlays/hcmais-dev/ambient-api-server-env-patch.yaml b/components/manifests/overlays/hcmais-dev/ambient-api-server-env-patch.yaml index eb5b2cffc..254e1e8d6 100644 --- a/components/manifests/overlays/hcmais-dev/ambient-api-server-env-patch.yaml +++ b/components/manifests/overlays/hcmais-dev/ambient-api-server-env-patch.yaml @@ -29,8 +29,6 @@ spec: - name: GRPC_SERVICE_ACCOUNT value: "ambient-control-plane-hcmais" valueFrom: null - - name: OPENSHELL_USE_GATEWAY - value: "true" - name: OPENSHELL_ENABLED value: "true" - name: GATEWAY_IMAGE diff --git a/components/manifests/overlays/hcmais-dev/control-plane-env-patch.yaml b/components/manifests/overlays/hcmais-dev/control-plane-env-patch.yaml index 8fe9b687a..2014ac5cd 100644 --- a/components/manifests/overlays/hcmais-dev/control-plane-env-patch.yaml +++ b/components/manifests/overlays/hcmais-dev/control-plane-env-patch.yaml @@ -54,8 +54,6 @@ spec: value: "ambient-vertex" - name: VERTEX_SECRET_NAMESPACE value: "ambient-dev" - - name: OPENSHELL_USE_GATEWAY - value: "true" - name: OPENSHELL_ENABLED value: "true" - name: CP_RUNTIME_NAMESPACE diff --git a/components/manifests/overlays/kind/ambient-api-server-dev-patch.yaml b/components/manifests/overlays/kind/ambient-api-server-dev-patch.yaml index 88427baec..ecb6cd095 100644 --- a/components/manifests/overlays/kind/ambient-api-server-dev-patch.yaml +++ b/components/manifests/overlays/kind/ambient-api-server-dev-patch.yaml @@ -59,8 +59,6 @@ spec: name: ambient-control-plane-token - name: CREDENTIAL_ENCRYPTION_ALLOW_PLAINTEXT value: "true" - - name: OPENSHELL_USE_GATEWAY - value: "true" - name: OPENSHELL_ENABLED value: "true" - name: GATEWAY_IMAGE diff --git a/components/manifests/overlays/kind/control-plane-env-patch.yaml b/components/manifests/overlays/kind/control-plane-env-patch.yaml index 30eccd3ad..036b73b2b 100644 --- a/components/manifests/overlays/kind/control-plane-env-patch.yaml +++ b/components/manifests/overlays/kind/control-plane-env-patch.yaml @@ -24,8 +24,6 @@ spec: value: "http://ambient-api-server.ambient-code.svc.cluster.local:8000" - name: AMBIENT_GRPC_USE_TLS value: "false" - - name: OPENSHELL_USE_GATEWAY - value: "true" - name: LOCAL_IMAGES value: "true" - name: USE_VERTEX diff --git a/components/manifests/overlays/openshift-local/ambient-api-server-dev-patch.yaml b/components/manifests/overlays/openshift-local/ambient-api-server-dev-patch.yaml index e8539b20c..2db8ebe74 100644 --- a/components/manifests/overlays/openshift-local/ambient-api-server-dev-patch.yaml +++ b/components/manifests/overlays/openshift-local/ambient-api-server-dev-patch.yaml @@ -59,8 +59,6 @@ spec: name: ambient-control-plane-token - name: CREDENTIAL_ENCRYPTION_ALLOW_PLAINTEXT value: "true" - - name: OPENSHELL_USE_GATEWAY - value: "true" - name: OPENSHELL_ENABLED value: "true" - name: GATEWAY_IMAGE diff --git a/components/manifests/overlays/openshift-local/control-plane-env-patch.yaml b/components/manifests/overlays/openshift-local/control-plane-env-patch.yaml index 7a5c5e8e7..8008d167b 100644 --- a/components/manifests/overlays/openshift-local/control-plane-env-patch.yaml +++ b/components/manifests/overlays/openshift-local/control-plane-env-patch.yaml @@ -24,7 +24,5 @@ spec: value: "http://ambient-api-server.ambient-code.svc.cluster.local:8000" - name: AMBIENT_GRPC_USE_TLS value: "false" - - name: OPENSHELL_USE_GATEWAY - value: "true" - name: LOCAL_IMAGES value: "true" diff --git a/components/pr-test/install-openshift.sh b/components/pr-test/install-openshift.sh index 0769be648..74cdfe9af 100755 --- a/components/pr-test/install-openshift.sh +++ b/components/pr-test/install-openshift.sh @@ -982,8 +982,6 @@ spec: value: "ambient-vertex" - name: VERTEX_SECRET_NAMESPACE value: "${NAMESPACE}" - - name: OPENSHELL_USE_GATEWAY - value: "false" - name: OPENSHELL_ENABLED value: "false" volumeMounts: diff --git a/rubber-on-road.md b/rubber-on-road.md deleted file mode 100644 index 97987892d..000000000 --- a/rubber-on-road.md +++ /dev/null @@ -1,178 +0,0 @@ -# Rubber on Road — acp-01 OpenShell Gateway Deployment - -## Objective - -Deploy the `acp-01` project with a full OpenShell gateway stack on the `vteam-stage` ROSA cluster, -proving end-to-end that agents can access GitHub, Jira, and the target OpenShift cluster through -credential-injected sandboxes. - -## Cluster State (2026-07-26) - -- **Cluster**: vteam-stage ROSA (6x m5.2xlarge) -- **ACP deployment**: `acp-api-01` namespace (API server, control plane, UI, Keycloak) -- **Existing project**: `tenant-a` — has a working gateway, used for prior e2e validation -- **Target project**: `acp-01` — fully operational with gateway, providers, and running agents -- **CP image**: `quay.io/ambient_code/acp_control_plane:oidc-gateway-auth-fix` -- **Gateway image**: `ghcr.io/nvidia/openshell/gateway:21da343c9f838bd9ac85dc61bf44889de1a72873` (v0.0.91) - -## Gitops Overlay State (current) - -| Resource | Count | Status | -|----------|-------|--------| -| Project | 1 (`acp-01`) | Created in API, namespace exists | -| Agents | 4 (lead, engineer, amber, shelly) | Applied, amber validated end-to-end | -| Credentials | 4 (github, jira, kubeconfig, vertex) | Applied | -| Providers | 4 (github, jira, kubeconfig, vertex) | Applied, github + vertex confirmed working | -| Gateway | 1 (`openshell-gateway`) | Running, OIDC + Postgres, v0.0.91 | -| RoleBindings | 12 | Token-reader grants for all agents × providers | - -## Gap Analysis: Gitops Overlay vs. e2e-openshell.sh - -The e2e script validates: -1. Gateway Deployment exists (`oc get deployment openshell-gateway -n $TENANT`) -2. Postgres DB Deployment exists (`oc get deployment openshell-gateway-db -n $TENANT`) -3. Gateway Service exists (`oc get service openshell-gateway -n $TENANT`) -4. TLS certs provisioned (`oc get secret openshell-server-tls -n $TENANT`) -5. Keycloak OIDC auth works -6. Gateway discoverable via `acpctl get gateways` -7. Route or port-forward connectivity -8. Sandbox lifecycle (create, exec, stop) - -The overlay is missing the **Gateway** resource entirely — without it, the control plane has nothing -to reconcile, so no Deployment, no DB, no Service, no TLS certs. - -Also missing: **Provider** resources that tell the gateway which credential secrets to inject into -sandboxes, and **K8s Secrets** backing those providers. - -## Design: Declarative Config for acp-01 - -### Resources needed: - -``` -acp-01/ -├── kustomization.yaml # Updated to include all resources -├── project-patch.yaml # Existing — patches project name to acp-01 -├── agents-patch.yaml # Existing — patches agent project + sandbox_policy -├── gateway.yaml # NEW — OpenShell gateway with OIDC + Postgres -├── provider-github.yaml # NEW — maps github provider to K8s secret -├── provider-jira.yaml # NEW — maps jira provider to K8s secret -├── provider-kubeconfig.yaml # NEW — maps kubeconfig provider to K8s secret -├── credential-github.yaml # Existing — env var substituted at apply time -├── credential-jira.yaml # Existing — env var substituted at apply time -├── credential-kubeconfig.yaml # Existing — env var substituted at apply time -├── rolebindings.yaml # NEW — token-reader grants for agents -├── lead.yaml # Existing -├── engineer.yaml # Existing -├── amber.yaml # Existing -└── shelly.yaml # Existing -``` - -### Credential → Provider → Gateway → Sandbox flow: - -``` -Credential (API) Provider (API) K8s Secret Gateway (gRPC) -┌───────────────┐ ┌─────────────────┐ ┌──────────────────┐ ┌──────────────┐ -│ acp-01-github │ │ github │ │ ambient-github │ │ openshell- │ -│ token=$GH_TOK │───▶│ type=github │───▶│ token= │───▶│ gateway │ -│ │ │ secret=ambient- │ │ │ │ providers: │ -│ │ │ github │ │ │ │ [github] │ -└───────────────┘ └─────────────────┘ └──────────────────┘ └──────┬───────┘ - │ - ┌─────▼───────┐ - │ Sandbox Pod │ - │ GITHUB_TOKEN │ - │ JIRA_TOKEN │ - │ KUBECONFIG │ - └─────────────┘ -``` - -## Deployment Steps - -1. Create K8s secrets in `acp-01` namespace (backing the Provider resources) -2. Apply the full kustomize overlay via `acpctl apply -k` -3. Wait for gateway reconciliation (Deployment, DB, TLS certs) -4. Run e2e-openshell.sh against `acp-01` - -## Progress Log - -| Date | Action | Result | -|------|--------|--------| -| 2026-07-23 | Created gitops overlay with gateway, providers, rolebindings | Gateway, DB, TLS certs all reconciled successfully | -| 2026-07-23 | Fixed CP→gateway OIDC auth (was using K8s SA token, gateway expects OIDC) | CP authenticates via Keycloak JWT, gateway validates via `OidcAuthenticator` | -| 2026-07-24 | Fixed sandbox name length (19-char limit, not 40) | `SandboxName()` truncates to 11 chars of session ID | -| 2026-07-24 | Fixed DNS ndots CR name mismatch (`default--` workspace prefix) | `SandboxCRName()` adds `default--` prefix matching gateway workspace | -| 2026-07-24 | Installed agent-sandbox controller v0.5.1 | CRD + controller in `agent-sandbox-system` namespace | -| 2026-07-24 | Fixed empty Landlock policy (exit code 126) | Populated `ambient-dev-sandbox-policy` with filesystem/process/network rules | -| 2026-07-25 | Fixed L7 policy validation (`protocol: rest` requires `access` or `rules`) | Removed `protocol`/`enforcement`/`tls: terminate`, added `access: full` | -| 2026-07-25 | Fixed FQDN DNS for cross-namespace service URLs | All CP env vars patched with `.svc.cluster.local` suffix | -| 2026-07-25 | Added vertex provider/credential to gitops | Inference routing now works through Vertex AI | -| 2026-07-25 | Updated proto: `SetClusterInference` → `SetInferenceRoute` | Matches upstream OpenShell v0.0.91 | -| 2026-07-25 | Updated gateway image from v0.0.88 to v0.0.91 SHA | `21da343c9f838bd9ac85dc61bf44889de1a72873` | -| 2026-07-25 | Built and pushed CP image with all fixes | `quay.io/ambient_code/acp_control_plane:oidc-gateway-auth-fix` | -| 2026-07-25 | **First successful end-to-end session** | LLM responded, GITHUB_TOKEN injected, inference routing working | -| 2026-07-26 | Verified credential injection | GITHUB_TOKEN ✓, GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN ✓, ANTHROPIC_API_KEY ✓ | - -## Credential Injection Results - -| Provider | Type | OpenShell Type | Env Vars Injected | Status | -|----------|------|----------------|-------------------|--------| -| `acp-01-vertex` | `vertex` | `google-vertex-ai` | `GOOGLE_SERVICE_ACCOUNT_KEY`, `GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN` | Working — gateway auto-refreshes SA JWT | -| `acp-01-github` | `github` | `github` | `GITHUB_TOKEN`, `GH_TOKEN` | Working | -| `acp-01-jira` | `jira` | `generic` | (none) | **Not injected** — see below | -| `acp-01-kubeconfig` | `kubeconfig` | `generic` | (none) | Not tested | - -### Jira Credential Gap - -The gateway does not inject env vars for `generic` type providers. Gateway logs show: -`provider type has no profile; skipping provider policy layer` for `acp-01-jira`. - -The CP sends the K8s secret data (`email`, `token`, `url`) as provider credentials, but the gateway's -supervisor has no env var mapping for generic providers. Only typed providers (github, claude-code, -google-vertex-ai, etc.) get automatic env var injection into sandboxes. - -**Options:** -1. Upstream: Request `jira` provider type support in OpenShell -2. Workaround: Use the credential sidecar pattern (pre-gateway approach) for Jira -3. Workaround: Map Jira to a generic env var pattern via `ProviderCredentialsFromSecret` with keys matching the expected env var names (`JIRA_API_TOKEN`, `JIRA_URL`, `JIRA_USERNAME`) - -## Deployment Learnings - -### Sandbox Naming -- Gateway creates Sandbox CRs with a `default--` workspace prefix: `default--session-` -- Sandbox name limit is **19 characters** (not 40 as originally spec'd) -- The CP's `SandboxName()` must truncate to 11 chars of session ID to stay within limits -- The CP's `SandboxCRName()` must prepend `default--` to match what the gateway creates - -### Agent-Sandbox Controller -- **Required prerequisite**: Must install `agent-sandbox-controller` v0.5.1 cluster-wide -- Without it, Sandbox CRs are created but no pods are spawned -- Install: `kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/v0.5.1/manifest.yaml` - -### Sandbox Policy (Landlock) -- An empty policy (`{version:1}`) causes Landlock to block all filesystem access → exit code 126 -- Policies MUST include `filesystem_policy` (read_only/read_write paths), `landlock` compatibility, `process` identity -- `tls: terminate` on endpoints is deprecated in current OpenShell — causes validation errors -- Endpoints with `protocol: rest` must include `access` (e.g., `full`) or explicit `rules` -- Removing `protocol`/`enforcement`/`tls` fields and adding `access: full` resolves L7 validation - -### DNS Resolution -- Sandbox pods run in the project namespace but CP services are in the runtime namespace -- With `ndots:1`, short service names like `ambient-control-plane.acp-api-01.svc` fail -- All service URLs must be fully qualified: `ambient-control-plane.acp-api-01.svc.cluster.local` -- Affects `CP_TOKEN_URL`, `AMBIENT_API_SERVER_URL`, `AMBIENT_GRPC_SERVER_ADDR`, `MCP_API_SERVER_URL` - -### Inference Proto (v0.0.91) -- Upstream OpenShell renamed `SetClusterInference`/`GetClusterInference` → `SetInferenceRoute`/`GetInferenceRoute` -- Added `workspace` field to request/response messages -- Old RPC returns `Unimplemented` on v0.0.91+ gateways -- Proto at `components/ambient-control-plane/proto/openshell/inference/v1/inference.proto` - -### GHCR Image Tags -- OpenShell gateway images use commit-SHA tags only (no semver tags) -- v0.0.91 = `ghcr.io/nvidia/openshell/gateway:21da343c9f838bd9ac85dc61bf44889de1a72873` -- Gateway reconciler continuously reconciles the image — gitops overlay must be source of truth - -### Gateway Authentication -- For ROSA/external deployments: CP authenticates via OIDC (Keycloak JWT), not K8s SA tokens -- Gateway OIDC config requires `issuer`, `audience`, `roles_claim`, `admin_role`, `user_role` -- `K8sServiceAccountAuthenticator` only works when CP runs in same cluster with accessible TokenReview API diff --git a/scripts/setup-kind-openshell.sh b/scripts/setup-kind-openshell.sh index 6d66aaf5c..5eaeb4961 100755 --- a/scripts/setup-kind-openshell.sh +++ b/scripts/setup-kind-openshell.sh @@ -1,13 +1,13 @@ #!/usr/bin/env bash # Install OpenShell gateway prerequisites into a Kind cluster (dual-tenant mode). -# Called by `make kind-up OPENSHELL_USE_GATEWAY=true`. +# Called by `make kind-up`. # # Provisions for each tenant in OPENSHELL_TENANTS (default: tenant-a tenant-b): # 1. Agent Sandbox CRD + controller (once, cluster-scoped) # 2. Tenant namespaces # 3. ACP project via the API -# 4. Patches the control plane deployment with OPENSHELL_USE_GATEWAY=true -# and Vertex AI env vars (ANTHROPIC_VERTEX_PROJECT_ID, CLOUD_ML_REGION) if set +# 4. Patches the control plane deployment with Vertex AI env vars +# (ANTHROPIC_VERTEX_PROJECT_ID, CLOUD_ML_REGION) if set # # Vertex AI credentials are configured separately by `make kind-setup-vertex`. # @@ -152,25 +152,19 @@ else kill "${PF_PID}" 2>/dev/null || true fi -# 4. Patch control plane with gateway flag and vertex env vars (idempotent) +# 4. Patch control plane with Vertex AI env vars (idempotent) # TLS is left at its default (true) — certgen-job creates openshell-client-tls # and openshell-server-tls secrets so mTLS works out of the box in Kind. -CP_ENV_ARGS="OPENSHELL_USE_GATEWAY=true" +CP_ENV_ARGS="" CP_NEEDS_PATCH=false -CURRENT_GW=$(kubectl get deployment ambient-control-plane -n "$NAMESPACE" \ - -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="OPENSHELL_USE_GATEWAY")].value}' 2>/dev/null || echo "") -if [ "$CURRENT_GW" != "true" ]; then - CP_NEEDS_PATCH=true -fi - if [ -n "${ANTHROPIC_VERTEX_PROJECT_ID:-}" ]; then CURRENT_PROJECT=$(kubectl get deployment ambient-control-plane -n "$NAMESPACE" \ -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="ANTHROPIC_VERTEX_PROJECT_ID")].value}' 2>/dev/null || echo "") if [ "$CURRENT_PROJECT" != "$ANTHROPIC_VERTEX_PROJECT_ID" ]; then CP_NEEDS_PATCH=true fi - CP_ENV_ARGS="$CP_ENV_ARGS ANTHROPIC_VERTEX_PROJECT_ID=${ANTHROPIC_VERTEX_PROJECT_ID}" + CP_ENV_ARGS="ANTHROPIC_VERTEX_PROJECT_ID=${ANTHROPIC_VERTEX_PROJECT_ID}" fi if [ -n "${CLOUD_ML_REGION:-}" ]; then @@ -179,10 +173,10 @@ if [ -n "${CLOUD_ML_REGION:-}" ]; then if [ "$CURRENT_REGION" != "$CLOUD_ML_REGION" ]; then CP_NEEDS_PATCH=true fi - CP_ENV_ARGS="$CP_ENV_ARGS CLOUD_ML_REGION=${CLOUD_ML_REGION}" + CP_ENV_ARGS="${CP_ENV_ARGS:+$CP_ENV_ARGS }CLOUD_ML_REGION=${CLOUD_ML_REGION}" fi -if [ "$CP_NEEDS_PATCH" = "true" ]; then +if [ "$CP_NEEDS_PATCH" = "true" ] && [ -n "$CP_ENV_ARGS" ]; then # shellcheck disable=SC2086 kubectl set env deployment/ambient-control-plane -n "$NAMESPACE" $CP_ENV_ARGS >/dev/null kubectl rollout status deployment/ambient-control-plane -n "$NAMESPACE" --timeout=60s >/dev/null 2>&1 @@ -190,7 +184,6 @@ if [ "$CP_NEEDS_PATCH" = "true" ]; then else echo " ambient-control-plane env already up to date — skipping" fi -echo " Note: ambient-ui gateway mode is baked in at build time via --build-arg OPENSHELL_USE_GATEWAY=true" # Vertex credentials and tenant overlays (examples/overlays//) are # applied by `make kind-up` after this script finishes — see the diff --git a/scripts/setup-vertex-provider.sh b/scripts/setup-vertex-provider.sh index 577dc0e35..304e5c8b5 100755 --- a/scripts/setup-vertex-provider.sh +++ b/scripts/setup-vertex-provider.sh @@ -15,7 +15,7 @@ # 3. The Credential kind in the overlay uses $VERTEX_SA_KEY for token expansion # # PREREQUISITES: -# - kind cluster running with OPENSHELL_USE_GATEWAY=true +# - kind cluster running with OpenShell gateway prerequisites # - acpctl built and logged in (make build-cli && make kind-acpctl-login) # - kubectl context set to the cluster # diff --git a/specs/platform/openshell-gateway-oidc.spec.md b/specs/platform/openshell-gateway-oidc.spec.md index 3e99b02fc..1e3e40a62 100644 --- a/specs/platform/openshell-gateway-oidc.spec.md +++ b/specs/platform/openshell-gateway-oidc.spec.md @@ -211,6 +211,8 @@ OPENSHELL_GATEWAY_INSECURE=true openshell -g tenant-a-openshell-gateway sandbox | `Invalid client or Invalid client credentials` | Wrong client_secret or client_id | Check `sso-credentials` Secret | | Token expires after 5 minutes | Keycloak access token TTL | Use refresh token or increase session timeout | | `openshell gateway add` opens browser | No `--no-browser` flag | Write `metadata.json` directly, then use `acpctl gateway setup-cli` | +| `GROUPS` env var returns `1000` in bash | Bash builtin collision — `GROUPS` is a reserved readonly array | Use `USER_GROUPS` instead of `GROUPS` for role/group env vars | +| `openshell sandbox create` hangs | Blocking interactive command | Background the command and poll for pod status; use `ExecSandbox` for runner startup | --- diff --git a/specs/platform/openshell-gateway.spec.md b/specs/platform/openshell-gateway.spec.md index 82d22a6e9..4a1886884 100644 --- a/specs/platform/openshell-gateway.spec.md +++ b/specs/platform/openshell-gateway.spec.md @@ -475,6 +475,8 @@ The GatewayReconciler SHALL validate Gateway resource fields before applying K8s - THEN validation SHALL fail with a descriptive error - AND the Gateway SHALL not be reconciled until the configuration is corrected +> **GHCR image tag convention:** OpenShell gateway images on GHCR use commit-SHA tags only (no semver tags). For example, `ghcr.io/nvidia/openshell/gateway:21da343c9f838bd9ac85dc61bf44889de1a72873` corresponds to v0.0.91. The GatewayReconciler continuously reconciles the image field, so the gitops overlay must be the source of truth for the image tag — manual image changes on the Deployment/StatefulSet will be reverted. + #### Scenario: Invalid DNS name - GIVEN a Gateway with a `serverDnsNames` entry that violates RFC 1123 diff --git a/specs/platform/openshell-sandbox-provisioning.spec.md b/specs/platform/openshell-sandbox-provisioning.spec.md index d356fb4be..414c89198 100644 --- a/specs/platform/openshell-sandbox-provisioning.spec.md +++ b/specs/platform/openshell-sandbox-provisioning.spec.md @@ -817,7 +817,7 @@ The control plane SHALL use mTLS for transport-level security when connecting to - **ACP → gateway (OIDC):** When the CP runs outside the gateway's cluster (e.g., ROSA deployments where the gateway cannot reach the CP's TokenReview API), the control plane authenticates via OIDC. The gateway is configured with an `OidcAuthenticator` (issuer URL, audience, roles claim), and the CP obtains a JWT from the OIDC issuer (e.g., Keycloak) and presents it as a Bearer token. The `TokenProvider` interface abstracts the auth mechanism — implementations exist for both K8s SA tokens and OIDC JWTs. - **Sandbox → gateway:** Sandbox pods authenticate via `IssueSandboxToken` (K8s SA token exchange for a gateway-minted JWT), then use the sandbox JWT for subsequent requests (policy fetch, log push, token refresh). This is managed entirely by the gateway and its supervisor — the control plane is not involved. -The gateway is not exposed outside the cluster (no Route), so the only clients are ACP (via mTLS + K8s SA token) and sandboxes (via mTLS + gateway-minted JWTs). +The gateway MAY be exposed outside the cluster via a GRPCRoute (Gateway API) or NLB passthrough Route (ROSA/AWS) — see [`openshell-gateway-routing.spec.md`](./openshell-gateway-routing.spec.md) for exposure strategies. When exposed, external clients (openshell CLI) authenticate via OIDC Bearer tokens. When not exposed, the only clients are ACP (via mTLS + K8s SA token or OIDC) and sandboxes (via mTLS + gateway-minted JWTs). The control plane SHALL load client TLS credentials dynamically from a Kubernetes Secret in each project namespace, enabling per-namespace certificate isolation. The `openshell-client-tls` Secret (configurable via `OPENSHELL_GATEWAY_CLIENT_TLS_SECRET`) contains the client certificate, private key, and CA certificate for verifying the gateway's server certificate. diff --git a/tests/e2e/gateway-e2e-test.sh b/tests/e2e/gateway-e2e-test.sh index 4f9fcfed3..16a6d3da2 100755 --- a/tests/e2e/gateway-e2e-test.sh +++ b/tests/e2e/gateway-e2e-test.sh @@ -406,7 +406,7 @@ PROJECT_ID=$(echo "$PROJECT_RESP" \ if [ -n "$PROJECT_ID" ]; then pass "Project '${TENANT}' exists (id: ${PROJECT_ID})" else - fail "Project '${TENANT}' not found — was 'make kind-up' run with OPENSHELL_USE_GATEWAY=true?" + fail "Project '${TENANT}' not found — was 'make kind-up' run?" results exit 1 fi diff --git a/tests/e2e/openshell-cli-e2e.sh b/tests/e2e/openshell-cli-e2e.sh index b5a6d0974..74e84518c 100755 --- a/tests/e2e/openshell-cli-e2e.sh +++ b/tests/e2e/openshell-cli-e2e.sh @@ -7,7 +7,7 @@ # CI debuggability. # # Prerequisites: -# - kind-up with OPENSHELL_USE_GATEWAY=true (default) +# - kind-up with OpenShell gateway prerequisites # - acpctl built (make build-cli) # - openshell CLI installed and available in $PATH # - Keycloak reachable (KEYCLOAK_URL, default http://localhost:11880) diff --git a/tests/e2e/openshell-dual-tenant.sh b/tests/e2e/openshell-dual-tenant.sh index df9238a84..03afee9ca 100755 --- a/tests/e2e/openshell-dual-tenant.sh +++ b/tests/e2e/openshell-dual-tenant.sh @@ -6,7 +6,7 @@ # in both tenant namespaces. # # Prerequisites: -# - kind-up with OPENSHELL_USE_GATEWAY=true +# - kind-up with OpenShell gateway prerequisites # - ACP projects tenant-a and tenant-b created (done automatically by kind-up) # - TEST_TOKEN and API_URL set, or tests/cypress/.env.test present # @@ -116,7 +116,7 @@ run_cmd_redact() { require_token() { if [ -z "$TOKEN" ]; then - echo -e "${RED}Error:${NC} TEST_TOKEN not set. Run 'make kind-up OPENSHELL_USE_GATEWAY=true' first." + echo -e "${RED}Error:${NC} TEST_TOKEN not set. Run 'make kind-up' first." echo " Or: source tests/cypress/.env.test && ./tests/e2e/openshell-dual-tenant.sh" exit 1 fi