fix(e2e): drop xfail strict=True on tests the cloud stack now passes #752
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,183 @@ | ||
| # Reusable inner workflow: build + deploy ONE ECS service. | ||
| # Called from deploy-app-services.yml's matrix-equivalent (3 jobs, one | ||
| # per service). Parameterized by service_name + dockerfile so the same | ||
| # logic services Proxy / OtelCollector / SshGateway. | ||
| # | ||
| # Mirrors deploy-api.yml's deploy job (minus the S3-related env patches | ||
| # which only apply to Api): | ||
| # 1. Resolve cluster | ||
| # 2. ECR login + buildx build of the parameterized Dockerfile | ||
| # 3. Clone live TD, swap image, strip readonly fields, register | ||
| # 4. UpdateService --force-new-deployment + wait services-stable | ||
| # 5. Assert PRIMARY taskDefinition == NEW_TD_ARN | ||
| # | ||
| # Unlike Api, these services don't sit behind ApiLoadBalancer (only Api | ||
| # has an ALB target group; Proxy/SshGateway have their own LBs; | ||
| # OtelCollector is internal). So the ALB-healthy poll is omitted here — | ||
| # `wait services-stable` + PRIMARY assertion are sufficient. | ||
| name: _deploy-single-service | ||
|
|
||
| on: | ||
| workflow_call: | ||
| inputs: | ||
| service_name: | ||
| description: 'Exact ECS service name (e.g. Proxy, OtelCollector, SshGateway)' | ||
| type: string | ||
| required: true | ||
| dockerfile: | ||
| description: 'Repo-relative Dockerfile path (build context = repo root)' | ||
| type: string | ||
| required: true | ||
|
|
||
| # Belt-and-suspenders against the outer-workflow concurrency: also | ||
| # serialize per-service here, so any caller (deploy-app-services or | ||
| # a future direct dispatch) racing on the same ECS service queues | ||
| # instead of fighting over TD revisions. Different services parallelize. | ||
| concurrency: | ||
| group: deploy-single-service-${{ inputs.service_name }} | ||
| cancel-in-progress: false | ||
|
|
||
| permissions: | ||
| contents: read | ||
| id-token: write | ||
|
|
||
| env: | ||
| AWS_REGION: ${{ vars.AWS_E2E_CLOUD_REGION }} | ||
| AWS_ROLE_ARN: ${{ vars.AWS_E2E_CLOUD_ROLE_ARN }} | ||
| STACK_PREFIX: boxlite-e2e-ci | ||
| ECS_CLUSTER_PATTERN: boxlite-e2e-ci-ClusterCluster- | ||
| ECR_REPO: sst-asset | ||
|
|
||
| jobs: | ||
| deploy: | ||
| name: Deploy ${{ inputs.service_name }} | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 30 | ||
| steps: | ||
| - uses: actions/checkout@v5 | ||
|
|
||
| - name: Configure AWS credentials (OIDC) | ||
| uses: aws-actions/configure-aws-credentials@v4 | ||
|
Comment on lines
+57
to
+60
There was a problem hiding this comment. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
rg -nP --glob '.github/workflows/*.yml' '^\s*-\s*uses:\s*(?!\./)[^@]+@(?![0-9a-fA-F]{40}\b)[^\s]+'Repository: boxlite-ai/boxlite Length of output: 211 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== Workflow files found =="
fd -e yml -e yaml .github/workflows || true
echo
echo "== Count =="
echo "yml:"; find .github/workflows -type f -name '*.yml' 2>/dev/null | wc -l || true
echo "yaml:"; find .github/workflows -type f -name '*.yaml' 2>/dev/null | wc -l || true
echo
echo "== Show top-level filenames =="
find .github/workflows -maxdepth 1 -type f -print 2>/dev/null || trueRepository: boxlite-ai/boxlite Length of output: 1218 🏁 Script executed: #!/bin/bash
set -euo pipefail
# Re-run detection over both yml and yaml, without relying on --glob filter.
WORKDIR=".github/workflows"
echo "== Non-SHA-pinned third-party uses: entries (yml+yaml) =="
if [ -d "$WORKDIR" ]; then
rg -nP --hidden --glob "${WORKDIR}/**/*.yml" --glob "${WORKDIR}/**/*.yaml" \
'^\s*-\s*uses:\s*(?!\./)(?![^@]+@)[^@]+@(?![0-9a-fA-F]{40}\b)[^\s]+' \
"$WORKDIR" || true
else
echo "No $WORKDIR directory found"
fiRepository: boxlite-ai/boxlite Length of output: 120 🏁 Script executed: #!/bin/bash
set -euo pipefail
# More reliable: explicitly match common checkout/configure uses lines and filter out local/sha-pinned.
WORKDIR=".github/workflows"
echo "== Candidate uses lines =="
if [ -d "$WORKDIR" ]; then
rg -n --hidden --glob "${WORKDIR}/**/*.yml" --glob "${WORKDIR}/**/*.yaml" \
'^\s*-\s*uses:\s*(?!\./)' \
"$WORKDIR" | head -n 200
fiRepository: boxlite-ai/boxlite Length of output: 350 🏁 Script executed: #!/bin/bash
set -euo pipefail
# Print the workflow files mentioned in the original comment (if they exist) so we can update the claim.
for f in \
.github/workflows/_deploy-single-service.yml \
.github/workflows/build-runner-binary.yml \
.github/workflows/deploy-api.yml \
.github/workflows/deploy-runner.yml \
.github/workflows/deploy-app-services.yml
do
if [ -f "$f" ]; then
echo "== $f =="
sed -n '1,120p' "$f"
echo
else
echo "== Missing: $f =="
fi
doneRepository: boxlite-ai/boxlite Length of output: 26031 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== Workflow files found =="
fd -e yml -e yaml .github/workflows || true
echo
echo "== Count =="
echo "yml:"; find .github/workflows -type f -name '*.yml' 2>/dev/null | wc -l || true
echo "yaml:"; find .github/workflows -type f -name '*.yaml' 2>/dev/null | wc -l || true
echo
echo "== Show top-level filenames =="
find .github/workflows -maxdepth 1 -type f -print 2>/dev/null || trueRepository: boxlite-ai/boxlite Length of output: 1218 🏁 Script executed: #!/bin/bash
set -euo pipefail
# Re-run detection over both yml and yaml, without relying on --glob filter.
WORKDIR=".github/workflows"
echo "== Non-SHA-pinned third-party uses: entries (yml+yaml) =="
if [ -d "$WORKDIR" ]; then
rg -nP --hidden --glob "${WORKDIR}/**/*.yml" --glob "${WORKDIR}/**/*.yaml" \
'^\s*-\s*uses:\s*(?!\./)(?![^@]+@)[^@]+@(?![0-9a-fA-F]{40}\b)[^\s]+' \
"$WORKDIR" || true
else
echo "No $WORKDIR directory found"
fiRepository: boxlite-ai/boxlite Length of output: 120 🏁 Script executed: #!/bin/bash
set -euo pipefail
# More reliable: explicitly match common checkout/configure uses lines and filter out local/sha-pinned.
WORKDIR=".github/workflows"
echo "== Candidate uses lines (first 200) =="
if [ -d "$WORKDIR" ]; then
rg -n --hidden --glob "${WORKDIR}/**/*.yml" --glob "${WORKDIR}/**/*.yaml" \
'^\s*-\s*uses:\s*(?!\./)' \
"$WORKDIR" | head -n 200
fiRepository: boxlite-ai/boxlite Length of output: 362 🏁 Script executed: #!/bin/bash
set -euo pipefail
# Print the workflow files mentioned in the original comment (if they exist) so we can update the claim.
for f in \
.github/workflows/_deploy-single-service.yml \
.github/workflows/build-runner-binary.yml \
.github/workflows/deploy-api.yml \
.github/workflows/deploy-runner.yml \
.github/workflows/deploy-app-services.yml
do
if [ -f "$f" ]; then
echo "== $f =="
sed -n '1,140p' "$f"
echo
else
echo "== Missing: $f =="
fi
doneRepository: boxlite-ai/boxlite Length of output: 30525 🏁 Script executed: #!/bin/bash
set -euo pipefail
python3 - <<'PY'
import os, re, sys, pathlib
root = pathlib.Path(".github/workflows")
files = sorted([p for p in root.rglob("*.yml") if p.is_file()])
uses_re = re.compile(r'^\s*-\s*uses:\s*([^\s#]+)|^\s*uses:\s*([^\s#]+)\s*(?:#.*)?$')
local_re = re.compile(r'^\./|^\.\./|^\.github/')
bad = []
for f in files:
txt = f.read_text(encoding="utf-8", errors="replace").splitlines()
for i,line in enumerate(txt, 1):
# capture either group 1 (dash uses) or group 2 (job uses:)
m = re.match(r'^\s*-\s*uses:\s*([^\s#]+)', line)
val = None
if m:
val = m.group(1)
else:
m2 = re.match(r'^\s*uses:\s*([^\s#]+)\s*(?:#.*)?$', line)
if m2:
val = m2.group(1)
if not val:
continue
if '@' not in val:
continue
if local_re.match(val):
continue
# anything like owner/repo/path@ref
before, ref = val.rsplit('@',1)
# treat refs that are exactly 40-hex as pinned
if re.fullmatch(r'[0-9a-fA-F]{40}', ref):
continue
bad.append((str(f), i, val))
print(f"Total workflow files scanned: {len(files)}")
print(f"Non-SHA-pinned uses found: {len(bad)}")
for f,i,val in bad:
print(f"{f}:{i}: {val}")
PYRepository: boxlite-ai/boxlite Length of output: 8741 🏁 Script executed: #!/bin/bash
set -euo pipefail
python3 - <<'PY'
import pathlib, re
root = pathlib.Path(".github/workflows")
files = sorted([p for p in root.rglob("*.yml") if p.is_file()])
pattern = re.compile(r'`@v`\d+|`@master`|`@main`|`@release`|`@stable`|`@latest`|`@canary`')
hits = []
for f in files:
for i,line in enumerate(f.read_text(encoding="utf-8", errors="replace").splitlines(),1):
m = re.match(r'^\s*-\s*uses:\s*([^\s#]+)', line) or re.match(r'^\s*uses:\s*([^\s#]+)\s*(?:#.*)?$', line)
if not m:
continue
val = m.group(1)
if '@' not in val:
continue
if val.startswith('./') or val.startswith('../') or val.startswith('.github/'):
continue
if pattern.search(val):
hits.append((str(f), i, val))
print("Examples of common floating tags:")
for f,i,val in hits[:200]:
print(f"{f}:{i}: {val}")
print(f"(showing {min(len(hits),200)} of {len(hits)})")
PYRepository: boxlite-ai/boxlite Length of output: 8536 🏁 Script executed: #!/bin/bash
set -euo pipefail
python3 - <<'PY'
import pathlib, re
root = pathlib.Path(".github/workflows")
files = sorted([p for p in root.rglob("*.yml") if p.is_file()])
def extract_uses_line(line):
m = re.match(r'^\s*-\s*uses:\s*([^\s#]+)', line)
if m:
return m.group(1)
m = re.match(r'^\s*uses:\s*([^\s#]+)\s*(?:#.*)?$', line)
return m.group(1) if m else None
bad = []
local_re = re.compile(r'^(?:\./|\.\./|\.github/)')
sha40 = re.compile(r'^[0-9a-fA-F]{40}$')
for f in files:
for idx, line in enumerate(f.read_text(encoding="utf-8", errors="replace").splitlines(), 1):
val = extract_uses_line(line)
if not val or '@' not in val:
continue
if local_re.match(val):
continue
_, ref = val.rsplit('@', 1)
if not sha40.match(ref):
bad.append((str(f), idx, val))
print(f"Total workflow files scanned: {len(files)}")
print(f"Non-SHA-pinned uses found: {len(bad)}")
for f, idx, val in bad:
print(f"{f}:{idx}: {val}")
PYRepository: boxlite-ai/boxlite Length of output: 8741 Pin third-party GitHub Actions to commit SHAs (floating Scanning
Replace each 🧰 Tools🪛 zizmor (1.25.2)[warning] 57-57: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false (artipacked) [error] 57-57: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy) (unpinned-uses) [error] 60-60: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy) (unpinned-uses) 🤖 Prompt for AI AgentsSource: Linters/SAST tools |
||
| with: | ||
| role-to-assume: ${{ env.AWS_ROLE_ARN }} | ||
| aws-region: ${{ env.AWS_REGION }} | ||
| role-session-name: deploy-${{ inputs.service_name }}-${{ github.run_id }} | ||
|
|
||
| - name: Resolve cluster | ||
| id: resources | ||
| run: | | ||
| set -euo pipefail | ||
| CLUSTER_COUNT=$(aws ecs list-clusters \ | ||
| --query "length(clusterArns[?contains(@, '${ECS_CLUSTER_PATTERN}')])" --output text) | ||
| [ "$CLUSTER_COUNT" -eq 1 ] || { echo "::error::Expected 1 cluster, found $CLUSTER_COUNT"; exit 1; } | ||
| CLUSTER=$(aws ecs list-clusters \ | ||
| --query "clusterArns[?contains(@, '${ECS_CLUSTER_PATTERN}')]|[0]" \ | ||
| --output text | awk -F/ '{print $NF}') | ||
| echo "cluster=$CLUSTER" >> "$GITHUB_OUTPUT" | ||
| echo "::notice::cluster=$CLUSTER service=${{ inputs.service_name }}" | ||
|
|
||
| - name: ECR login | ||
| id: ecr_login | ||
| uses: aws-actions/amazon-ecr-login@v2 | ||
|
|
||
| - name: Build & push image (${{ inputs.dockerfile }}) | ||
| id: build | ||
| run: | | ||
| set -euo pipefail | ||
| REGISTRY="${{ steps.ecr_login.outputs.registry }}" | ||
| # Lowercase service for the ECR tag (deterministic + matches | ||
| # the convention "${service-lowercase}-${sha}"). | ||
| TAG_NAME=$(echo "${{ inputs.service_name }}" | tr '[:upper:]' '[:lower:]') | ||
| IMAGE="${REGISTRY}/${ECR_REPO}:${TAG_NAME}-${{ github.sha }}" | ||
| docker buildx build --platform linux/amd64 --load \ | ||
| -f "${{ inputs.dockerfile }}" -t "$IMAGE" . | ||
| docker push "$IMAGE" | ||
| echo "image=$IMAGE" >> "$GITHUB_OUTPUT" | ||
| echo "::notice::Built and pushed $IMAGE" | ||
|
|
||
| - name: Resolve Api role ARNs (borrow source for missing per-service roles) | ||
| id: api_roles | ||
| run: | | ||
| set -euo pipefail | ||
| # The e2e-ci stack's per-service Task/Execution roles | ||
| # (Proxy/Otel/SshGw{Task,Execution}Role-*) were deleted from | ||
| # IAM out-of-band, but the live ECS TDs still reference the | ||
| # dead ARNs → AssumeRole NoSuchEntity → new tasks can't start. | ||
| # As a temporary unblock (until admin runs `sst deploy | ||
| # --stage e2e-ci` to recreate properly), borrow the Api | ||
| # service's roles which were recreated earlier in this PR's | ||
| # session. Both Api roles trust ecs-tasks.amazonaws.com (no | ||
| # principal restriction), so any service can assume them. The | ||
| # cost is least-privilege: borrowed task role grants S3 + | ||
| # CloudWatch perms the borrower may not need. Acceptable for | ||
| # e2e-ci which has no production traffic. | ||
| API_TD=$(aws ecs describe-services --cluster "${{ steps.resources.outputs.cluster }}" \ | ||
| --services Api --query 'services[0].taskDefinition' --output text) | ||
| read -r TASK_ROLE EXEC_ROLE <<<"$(aws ecs describe-task-definition \ | ||
| --task-definition "$API_TD" \ | ||
| --query 'taskDefinition.[taskRoleArn,executionRoleArn]' --output text)" | ||
| [ -n "$TASK_ROLE" ] && [ "$TASK_ROLE" != "None" ] \ | ||
| || { echo "::error::Could not resolve Api taskRoleArn"; exit 1; } | ||
| [ -n "$EXEC_ROLE" ] && [ "$EXEC_ROLE" != "None" ] \ | ||
| || { echo "::error::Could not resolve Api executionRoleArn"; exit 1; } | ||
| echo "task_role_arn=$TASK_ROLE" >> "$GITHUB_OUTPUT" | ||
| echo "exec_role_arn=$EXEC_ROLE" >> "$GITHUB_OUTPUT" | ||
| echo "::notice::Will borrow Api roles for ${{ inputs.service_name }}: $(basename "$TASK_ROLE"), $(basename "$EXEC_ROLE")" | ||
|
|
||
| - name: Register new TD + UpdateService + wait stable | ||
| env: | ||
| CLUSTER: ${{ steps.resources.outputs.cluster }} | ||
| IMAGE: ${{ steps.build.outputs.image }} | ||
| SERVICE: ${{ inputs.service_name }} | ||
| TASK_ROLE: ${{ steps.api_roles.outputs.task_role_arn }} | ||
| EXEC_ROLE: ${{ steps.api_roles.outputs.exec_role_arn }} | ||
|
Comment on lines
+98
to
+133
There was a problem hiding this comment. Avoid unconditional role replacement with Api roles. Line 149 and Line 150 force every service to use Api task/execution roles on each deploy. That turns a temporary break-glass workaround into a persistent least-privilege regression and can hide role drift after IAM is repaired. Keep existing service roles by default, and only fallback to Api roles when the current role ARNs are actually invalid/unresolvable. Also applies to: 141-151 🧰 Tools🪛 zizmor (1.25.2)[info] 114-114: code injection via template expansion (template-injection): may expand into attacker-controllable code (template-injection) [error] 125-125: code injection via template expansion (template-injection): may expand into attacker-controllable code (template-injection) 🤖 Prompt for AI Agents |
||
| run: | | ||
| set -euo pipefail | ||
|
|
||
| OLD_TD_ARN=$(aws ecs describe-services --cluster "$CLUSTER" --services "$SERVICE" \ | ||
| --query 'services[0].taskDefinition' --output text) | ||
| echo "Old TD: $OLD_TD_ARN" | ||
|
|
||
| # Clone old TD, swap container image, OVERRIDE task & execution | ||
| # roles to point at Api's (the per-service roles are gone from | ||
| # IAM — see api_roles step above), strip readonly fields. | ||
| # Plaintext env (e.g. credentials) lives in TD — do NOT cat the file. | ||
| aws ecs describe-task-definition --task-definition "$OLD_TD_ARN" \ | ||
| --query 'taskDefinition' --output json \ | ||
| | jq --arg img "$IMAGE" --arg trole "$TASK_ROLE" --arg erole "$EXEC_ROLE" ' | ||
| .containerDefinitions[0].image = $img | ||
| | .taskRoleArn = $trole | ||
| | .executionRoleArn = $erole | ||
| | del(.taskDefinitionArn, .revision, .status, .requiresAttributes, | ||
| .compatibilities, .registeredAt, .registeredBy)' \ | ||
| > /tmp/new-td.json | ||
|
|
||
| NEW_TD_ARN=$(aws ecs register-task-definition \ | ||
| --cli-input-json file:///tmp/new-td.json \ | ||
| --query 'taskDefinition.taskDefinitionArn' --output text) | ||
| echo "::notice::New TD: $NEW_TD_ARN" | ||
|
|
||
| aws ecs update-service --cluster "$CLUSTER" --service "$SERVICE" \ | ||
| --task-definition "$NEW_TD_ARN" --force-new-deployment >/dev/null | ||
| aws ecs wait services-stable --cluster "$CLUSTER" --services "$SERVICE" | ||
| echo "Service stable — verifying PRIMARY..." | ||
|
|
||
| PRIMARY_TD=$(aws ecs describe-services --cluster "$CLUSTER" --services "$SERVICE" \ | ||
| --query 'services[0].deployments[?status==`PRIMARY`]|[0].taskDefinition' \ | ||
| --output text) | ||
| if [ "$PRIMARY_TD" != "$NEW_TD_ARN" ]; then | ||
| echo "::error::ECS rolled back. PRIMARY=$PRIMARY_TD expected $NEW_TD_ARN" | ||
| aws ecs list-tasks --cluster "$CLUSTER" --service-name "$SERVICE" \ | ||
| --desired-status STOPPED --max-results 5 \ | ||
| --query 'taskArns' --output json \ | ||
| | python3 -c "import json,sys; [print(t) for t in json.load(sys.stdin)]" \ | ||
| | while read TASK; do | ||
| [ -n "$TASK" ] || continue | ||
| echo "--- stopped task $(basename "$TASK") ---" | ||
| aws ecs describe-tasks --cluster "$CLUSTER" --tasks "$TASK" \ | ||
| --query 'tasks[0].[stoppedReason,containers[0].exitCode,containers[0].reason]' \ | ||
| --output text || true | ||
| done | ||
| exit 1 | ||
| fi | ||
| echo "::notice::PRIMARY deployment confirmed for ${SERVICE} on $NEW_TD_ARN" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
e2e-cloud.ymldocumentation is out of sync with the actual workflow.This section describes a 3-job
changes/e2e/e2e-gatepattern and PR triggers, but the workflow currently has deploy sub-jobs (deploy_runner,deploy_api,deploy_app_services), no gate job, and nopull_requesttrigger. Please update this section to match the file behavior so branch-protection and operator expectations stay accurate.🤖 Prompt for AI Agents