fix(api): restore image resolution so e2e-cloud-test can boot boxes

.github/workflows/README.md

@@ -195,7 +195,7 @@ Runs VM-based E2E integration tests on an ephemeral AWS EC2 self-hosted runner.
 - `AWS_SUBNET_ID` - Subnet with auto-assign public IP
 - `AWS_SECURITY_GROUP_ID` - Security group allowing outbound HTTPS
 
-**Required AWS resources** (provisioned by `scripts/ci/setup-aws-oidc.sh`):
+**Required AWS resources** (provisioned by `scripts/ci/setup-ci-runner.sh`):
 - OIDC identity provider (`token.actions.githubusercontent.com`)
 - IAM role `boxlite-e2e-github-actions` with trust policy for this repo
 - IAM instance profile `boxlite-e2e-runner` with `ec2:TerminateInstances` on self
@@ -207,6 +207,75 @@ Runs VM-based E2E integration tests on an ephemeral AWS EC2 self-hosted runner.
 3. `e2e-tests` - Build runtime, run Rust/CLI/Python/Node/C integration tests
 4. `stop-runner` - Terminate instance, deregister runner
 
+### `e2e-cloud.yml`
+
+Runs the SDK → API → Runner → libkrun VM regression suite against the
+**always-on Tokyo stack** (`boxlite-e2e-ci-*`) instead of a per-run KVM
+runner. Built for fast iteration on REST-path bugs (`e2e-test.yml` is
+slower because it spins up a fresh EC2 every run).
+
+**Why:** The same regression goal as `e2e-test.yml`, but the cost
+profile is different — `e2e-cloud.yml` deploys-and-tests against an
+already-deployed cloud stack, so a typical run is 8-15 min and adds
+no per-run EC2 cost. `e2e-test.yml` and this workflow are
+complementary, not redundant — they exercise the same test code but
+the deployed-stack path also catches infra-only regressions (LB
+config, RDS schema, ECS task def drift) that the self-bootstrap
+path doesn't.
+
+**Architecture:** Three-job required-gate pattern:
+1. `changes` (ubuntu-latest) — paths-filter cheap detector
+2. `e2e` (ubuntu-latest, only if `changes` says relevant or `workflow_dispatch`) —
+   builds Api image + runner binary from this checkout, deploys to Tokyo,
+   builds the Python SDK from source, runs `pytest scripts/test/e2e/cases/`
+3. `e2e-gate` (always runs) — collapses outcome into one required check
+
+**Triggers:**
+- Push to `main`
+- Pull request to `main`
+- Manual dispatch (`workflow_dispatch`)
+
+  Path matching happens inside the `changes` job (so branch
+  protection can require the gate's status check on every PR).
+
+**Cost:** Build + deploy + test on GitHub-hosted ubuntu-latest (free for
+public repos / billable minutes for private). The Tokyo stack itself
+runs 24×7 — its baseline is the cost driver, not this workflow.
+
+**Authentication:** GitHub OIDC → AWS STS (no stored AWS credentials),
+**separate** IAM role from `e2e-test.yml`:
+
+| Workflow | Role | Region | Scope |
+|----------|------|--------|-------|
+| `e2e-test.yml` | `boxlite-e2e-github-actions` | us-east-1 | self-hosted runner provisioning (ec2:RunInstances, terminate, register/deregister GH runner) |
+| `e2e-cloud.yml` | `boxlite-e2e-cloud-github-actions` | ap-northeast-1 | Tokyo stack deploy + exec (ecr push, ecs update-service / execute-command, ssm send-command to runner, s3 builds/, ssm parameter read) |
+
+**Required variables** (Settings → Variables → Actions):
+- `AWS_ACCOUNT_ID` (shared)
+- `AWS_E2E_CLOUD_REGION` = `ap-northeast-1`
+- `AWS_E2E_CLOUD_ROLE_ARN` = `arn:aws:iam::<acct>:role/boxlite-e2e-cloud-github-actions`
+
+**Required AWS resources** (provisioned by `scripts/ci/setup-e2e-cloud-oidc.sh`):
+- OIDC identity provider (`token.actions.githubusercontent.com`) — shared
+  with `e2e-test.yml`
+- IAM role `boxlite-e2e-cloud-github-actions` with trust policy limited
+  to `repo:boxlite-ai/boxlite:{ref:refs/heads/main, pull_request, environment:e2e-cloud}`
+- SSM SecureString parameter `/boxlite/e2e-ci/admin-api-key` (sourced
+  by the workflow at runtime; never committed)
+- Tokyo `boxlite-e2e-ci-*` stack already deployed (SST `e2e-ci` stage)
+
+**Concurrency:** `e2e-cloud-shared` — singleton lock across every PR
+and every push. The Tokyo stack is a shared singleton, so per-ref
+grouping would let PRs race each other's ECS rolling updates.
+
+**Stack state after a run:** The Tokyo stack is left running THIS
+workflow run's Api image + runner binary. There's no auto-restore to
+`main` HEAD between runs (cost trade-off). Console / direct stack
+inspection therefore reflects "whatever the last e2e-cloud run
+deployed" — a `git log --grep e2e-cloud` on `main` and the ECS task
+definition's image tag (`api-<sha>`) together identify the running
+revision.
+
 ## Trigger Behavior
 
 | Change | warm-caches | build-runtime | build-wheels | build-node |

.github/workflows/_deploy-single-service.yml

@@ -0,0 +1,183 @@
+# Reusable inner workflow: build + deploy ONE ECS service.
+# Called from deploy-app-services.yml's matrix-equivalent (3 jobs, one
+# per service). Parameterized by service_name + dockerfile so the same
+# logic services Proxy / OtelCollector / SshGateway.
+#
+# Mirrors deploy-api.yml's deploy job (minus the S3-related env patches
+# which only apply to Api):
+#   1. Resolve cluster
+#   2. ECR login + buildx build of the parameterized Dockerfile
+#   3. Clone live TD, swap image, strip readonly fields, register
+#   4. UpdateService --force-new-deployment + wait services-stable
+#   5. Assert PRIMARY taskDefinition == NEW_TD_ARN
+#
+# Unlike Api, these services don't sit behind ApiLoadBalancer (only Api
+# has an ALB target group; Proxy/SshGateway have their own LBs;
+# OtelCollector is internal). So the ALB-healthy poll is omitted here —
+# `wait services-stable` + PRIMARY assertion are sufficient.
+name: _deploy-single-service
+
+on:
+  workflow_call:
+    inputs:
+      service_name:
+        description: 'Exact ECS service name (e.g. Proxy, OtelCollector, SshGateway)'
+        type: string
+        required: true
+      dockerfile:
+        description: 'Repo-relative Dockerfile path (build context = repo root)'
+        type: string
+        required: true
+
+# Belt-and-suspenders against the outer-workflow concurrency: also
+# serialize per-service here, so any caller (deploy-app-services or
+# a future direct dispatch) racing on the same ECS service queues
+# instead of fighting over TD revisions. Different services parallelize.
+concurrency:
+  group: deploy-single-service-${{ inputs.service_name }}
+  cancel-in-progress: false
+
+permissions:
+  contents: read
+  id-token: write
+
+env:
+  AWS_REGION: ${{ vars.AWS_E2E_CLOUD_REGION }}
+  AWS_ROLE_ARN: ${{ vars.AWS_E2E_CLOUD_ROLE_ARN }}
+  STACK_PREFIX: boxlite-e2e-ci
+  ECS_CLUSTER_PATTERN: boxlite-e2e-ci-ClusterCluster-
+  ECR_REPO: sst-asset
+
+jobs:
+  deploy:
+    name: Deploy ${{ inputs.service_name }}
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    steps:
+      - uses: actions/checkout@v5
+
+      - name: Configure AWS credentials (OIDC)
+        uses: aws-actions/configure-aws-credentials@v4
+        with:
+          role-to-assume: ${{ env.AWS_ROLE_ARN }}
+          aws-region: ${{ env.AWS_REGION }}
+          role-session-name: deploy-${{ inputs.service_name }}-${{ github.run_id }}
+
+      - name: Resolve cluster
+        id: resources
+        run: |
+          set -euo pipefail
+          CLUSTER_COUNT=$(aws ecs list-clusters \
+            --query "length(clusterArns[?contains(@, '${ECS_CLUSTER_PATTERN}')])" --output text)
+          [ "$CLUSTER_COUNT" -eq 1 ] || { echo "::error::Expected 1 cluster, found $CLUSTER_COUNT"; exit 1; }
+          CLUSTER=$(aws ecs list-clusters \
+            --query "clusterArns[?contains(@, '${ECS_CLUSTER_PATTERN}')]|[0]" \
+            --output text | awk -F/ '{print $NF}')
+          echo "cluster=$CLUSTER" >> "$GITHUB_OUTPUT"
+          echo "::notice::cluster=$CLUSTER service=${{ inputs.service_name }}"
+
+      - name: ECR login
+        id: ecr_login
+        uses: aws-actions/amazon-ecr-login@v2
+
+      - name: Build & push image (${{ inputs.dockerfile }})
+        id: build
+        run: |
+          set -euo pipefail
+          REGISTRY="${{ steps.ecr_login.outputs.registry }}"
+          # Lowercase service for the ECR tag (deterministic + matches
+          # the convention "${service-lowercase}-${sha}").
+          TAG_NAME=$(echo "${{ inputs.service_name }}" | tr '[:upper:]' '[:lower:]')
+          IMAGE="${REGISTRY}/${ECR_REPO}:${TAG_NAME}-${{ github.sha }}"
+          docker buildx build --platform linux/amd64 --load \
+            -f "${{ inputs.dockerfile }}" -t "$IMAGE" .
+          docker push "$IMAGE"
+          echo "image=$IMAGE" >> "$GITHUB_OUTPUT"
+          echo "::notice::Built and pushed $IMAGE"
+
+      - name: Resolve Api role ARNs (borrow source for missing per-service roles)
+        id: api_roles
+        run: |
+          set -euo pipefail
+          # The e2e-ci stack's per-service Task/Execution roles
+          # (Proxy/Otel/SshGw{Task,Execution}Role-*) were deleted from
+          # IAM out-of-band, but the live ECS TDs still reference the
+          # dead ARNs → AssumeRole NoSuchEntity → new tasks can't start.
+          # As a temporary unblock (until admin runs `sst deploy
+          # --stage e2e-ci` to recreate properly), borrow the Api
+          # service's roles which were recreated earlier in this PR's
+          # session. Both Api roles trust ecs-tasks.amazonaws.com (no
+          # principal restriction), so any service can assume them. The
+          # cost is least-privilege: borrowed task role grants S3 +
+          # CloudWatch perms the borrower may not need. Acceptable for
+          # e2e-ci which has no production traffic.
+          API_TD=$(aws ecs describe-services --cluster "${{ steps.resources.outputs.cluster }}" \
+            --services Api --query 'services[0].taskDefinition' --output text)
+          read -r TASK_ROLE EXEC_ROLE <<<"$(aws ecs describe-task-definition \
+            --task-definition "$API_TD" \
+            --query 'taskDefinition.[taskRoleArn,executionRoleArn]' --output text)"
+          [ -n "$TASK_ROLE" ] && [ "$TASK_ROLE" != "None" ] \
+            || { echo "::error::Could not resolve Api taskRoleArn"; exit 1; }
+          [ -n "$EXEC_ROLE" ] && [ "$EXEC_ROLE" != "None" ] \
+            || { echo "::error::Could not resolve Api executionRoleArn"; exit 1; }
+          echo "task_role_arn=$TASK_ROLE" >> "$GITHUB_OUTPUT"
+          echo "exec_role_arn=$EXEC_ROLE" >> "$GITHUB_OUTPUT"
+          echo "::notice::Will borrow Api roles for ${{ inputs.service_name }}: $(basename "$TASK_ROLE"), $(basename "$EXEC_ROLE")"
+
+      - name: Register new TD + UpdateService + wait stable
+        env:
+          CLUSTER:   ${{ steps.resources.outputs.cluster }}
+          IMAGE:     ${{ steps.build.outputs.image }}
+          SERVICE:   ${{ inputs.service_name }}
+          TASK_ROLE: ${{ steps.api_roles.outputs.task_role_arn }}
+          EXEC_ROLE: ${{ steps.api_roles.outputs.exec_role_arn }}
+        run: |
+          set -euo pipefail
+
+          OLD_TD_ARN=$(aws ecs describe-services --cluster "$CLUSTER" --services "$SERVICE" \
+            --query 'services[0].taskDefinition' --output text)
+          echo "Old TD: $OLD_TD_ARN"
+
+          # Clone old TD, swap container image, OVERRIDE task & execution
+          # roles to point at Api's (the per-service roles are gone from
+          # IAM — see api_roles step above), strip readonly fields.
+          # Plaintext env (e.g. credentials) lives in TD — do NOT cat the file.
+          aws ecs describe-task-definition --task-definition "$OLD_TD_ARN" \
+            --query 'taskDefinition' --output json \
+            | jq --arg img "$IMAGE" --arg trole "$TASK_ROLE" --arg erole "$EXEC_ROLE" '
+                .containerDefinitions[0].image = $img
+                | .taskRoleArn = $trole
+                | .executionRoleArn = $erole
+                | del(.taskDefinitionArn, .revision, .status, .requiresAttributes,
+                      .compatibilities, .registeredAt, .registeredBy)' \
+            > /tmp/new-td.json
+
+          NEW_TD_ARN=$(aws ecs register-task-definition \
+            --cli-input-json file:///tmp/new-td.json \
+            --query 'taskDefinition.taskDefinitionArn' --output text)
+          echo "::notice::New TD: $NEW_TD_ARN"
+
+          aws ecs update-service --cluster "$CLUSTER" --service "$SERVICE" \
+            --task-definition "$NEW_TD_ARN" --force-new-deployment >/dev/null
+          aws ecs wait services-stable --cluster "$CLUSTER" --services "$SERVICE"
+          echo "Service stable — verifying PRIMARY..."
+
+          PRIMARY_TD=$(aws ecs describe-services --cluster "$CLUSTER" --services "$SERVICE" \
+            --query 'services[0].deployments[?status==`PRIMARY`]|[0].taskDefinition' \
+            --output text)
+          if [ "$PRIMARY_TD" != "$NEW_TD_ARN" ]; then
+              echo "::error::ECS rolled back. PRIMARY=$PRIMARY_TD expected $NEW_TD_ARN"
+              aws ecs list-tasks --cluster "$CLUSTER" --service-name "$SERVICE" \
+                  --desired-status STOPPED --max-results 5 \
+                  --query 'taskArns' --output json \
+                | python3 -c "import json,sys; [print(t) for t in json.load(sys.stdin)]" \
+                | while read TASK; do
+                    [ -n "$TASK" ] || continue
+                    echo "--- stopped task $(basename "$TASK") ---"
+                    aws ecs describe-tasks --cluster "$CLUSTER" --tasks "$TASK" \
+                        --query 'tasks[0].[stoppedReason,containers[0].exitCode,containers[0].reason]' \
+                        --output text || true
+                  done
+              exit 1
+          fi
+          echo "::notice::PRIMARY deployment confirmed for ${SERVICE} on $NEW_TD_ARN"

.github/workflows/build-c.yml

@@ -15,12 +15,28 @@
 # Triggers:
 # - release: When a GitHub release is published
 # - workflow_dispatch: Manual trigger for testing
+# - workflow_call:     Reusable from .github/workflows/e2e-cloud.yml.
+#                      Pass target_filter='linux-x64-gnu' to skip the
+#                      macOS/arm64 matrix entries (only the Tokyo
+#                      runner target is needed).
 name: Build C SDK
 
+# Default GITHUB_TOKEN to read-only. The upload_release job opts into
+# `contents: write` explicitly when it needs to publish release assets.
+permissions:
+  contents: read
+
 on:
   release:
     types: [ published ]
   workflow_dispatch:
+  workflow_call:
+    inputs:
+      target_filter:
+        description: 'Optional. Build only the given target (e.g. "linux-x64-gnu"). Empty = build all matrix platforms.'
+        type: string
+        default: ''
+        required: false
 
 env:
   CARGO_TERM_COLOR: always
@@ -30,6 +46,33 @@ jobs:
   config:
     uses: ./.github/workflows/config.yml
 
+  # Compute the matrix the `build` job iterates over. When called via
+  # workflow_call with target_filter set, narrow to that single entry.
+  # `matrix` context is not available in job-level if:, so we filter
+  # the JSON here and have `build` consume `setup_matrix.outputs.platforms`.
+  setup_matrix:
+    name: Setup matrix
+    needs: config
+    runs-on: ubuntu-latest
+    outputs:
+      platforms: ${{ steps.filter.outputs.platforms }}
+    steps:
+      - id: filter
+        env:
+          ALL_PLATFORMS: ${{ needs.config.outputs.platforms }}
+          TARGET_FILTER: ${{ inputs.target_filter }}
+        run: |
+          if [ -n "${TARGET_FILTER:-}" ]; then
+            FILTERED=$(printf '%s' "$ALL_PLATFORMS" | jq -c --arg t "$TARGET_FILTER" '[.[] | select(.target == $t)]')
+            COUNT=$(printf '%s' "$FILTERED" | jq 'length')
+            [ "$COUNT" -ge 1 ] || { echo "::error::No matrix entry matches target_filter=$TARGET_FILTER"; exit 1; }
+            echo "Filtered to $COUNT entry(ies): $FILTERED"
+          else
+            FILTERED="$ALL_PLATFORMS"
+            echo "No filter — full matrix"
+          fi
+          echo "platforms=$FILTERED" >> "$GITHUB_OUTPUT"
+
   # ============================================================================
   # BUILD
   # ============================================================================
@@ -48,12 +91,12 @@ jobs:
   # ============================================================================
   build:
     name: Build (${{ matrix.target }})
-    needs: config
+    needs: [config, setup_matrix]
     runs-on: ${{ matrix.os }}
     strategy:
       fail-fast: false
       matrix:
-        include: ${{ fromJson(needs.config.outputs.platforms) }}
+        include: ${{ fromJson(needs.setup_matrix.outputs.platforms) }}
 
     steps:
       - uses: actions/checkout@v4