Skip to content

Commit 2e81a54

Browse files
TaylorMutchTaylorMutch
committed
ci(canary): add kind-based helm chart smoke test to Release Canary
Add a kubernetes job to release-canary.yml that creates a kind cluster, installs the published 0.0.0-dev helm chart, port-forwards the gateway, registers it via openshell gateway add, and runs openshell status against it. Pair the workflow with a test-release-canary skill that documents manual dispatch, local reproduction, and failure-mode triage; cross-reference it from CI.md and the CONTRIBUTING.md skills table. Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
1 parent 702bb56 commit 2e81a54

4 files changed

Lines changed: 222 additions & 0 deletions

File tree

‎.agents/skills/test-release-canary/SKILL.md‎

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
---
2+
name: test-release-canary
3+
description: Manually dispatch and iterate on the Release Canary workflow that smoke-tests published OpenShell artifacts (install.sh on macOS/Ubuntu/Fedora, Helm chart on kind) after each Release Dev publish. Use when changing `.github/workflows/release-canary.yml`, validating a release before tagging, debugging a canary failure, or reproducing a canary job locally. Trigger keywords - release canary, release-canary, canary failed, canary dispatch, test release canary, post-release smoke, install.sh canary, helm chart canary, kind canary, dispatch canary.
4+
---
5+
6+
# Test Release Canary
7+
8+
The Release Canary (`.github/workflows/release-canary.yml`) smoke-tests the artifacts a `Release Dev` run just published. It is the last automated checkpoint before tagging a public release: if the canary is red, the published `dev` artifacts do not install on a stock environment.
9+
10+
## What the canary verifies
11+
12+
| Job | Runner | Verifies |
13+
|---|---|---|
14+
| `macos` | `macos-latest-xlarge` | `install.sh` resolves the Homebrew formula, brew installs the cask, and `openshell status` reaches the brew-services–backed local gateway with the VM driver. |
15+
| `ubuntu` | `ubuntu-latest` | `install.sh` installs the Debian package, the post-install systemd user service starts, and `openshell status` reaches the local gateway with the Docker driver. |
16+
| `fedora` | `fedora:latest` container | `install.sh` installs the RPM packages, the local gateway starts under Podman, and `openshell status` succeeds. |
17+
| `kubernetes` | `ubuntu-latest` + kind | `helm install oci://ghcr.io/nvidia/openshell/helm-chart --version 0.0.0-dev` succeeds in a kind cluster, the gateway pod becomes Ready, port-forward exposes 8080, and the released CLI registers the in-cluster gateway and runs `openshell status` against it. |
18+
19+
`install.sh` defaults to the *latest tagged* release — the canary is therefore checking that the most recent public release still installs, not the just-published `dev` build. The `kubernetes` job is the exception: it pins to `0.0.0-dev` chart + `:dev` images.
20+
21+
## Trigger paths
22+
23+
The workflow has two triggers:
24+
25+
```yaml
26+
on:
27+
workflow_dispatch:
28+
workflow_run:
29+
workflows: ["Release Dev"]
30+
types: [completed]
31+
```
32+
33+
- **Automatic.** Every successful `Release Dev` run (on `main` or a manual dispatch of Release Dev) fires the canary. Each job gates on `github.event.workflow_run.conclusion == 'success'` so a failed Release Dev does not run the canary.
34+
- **Manual.** `workflow_dispatch` lets you run the canary on demand against any branch's workflow definition.
35+
36+
When dispatched manually, `github.event.workflow_run.head_sha` is empty and the workflow falls back to `github.sha` (the branch tip) for the `install.sh` URL.
37+
38+
## Manual dispatch
39+
40+
Run the canary as-is on the current branch:
41+
42+
```shell
43+
gh workflow run release-canary.yml --ref "$(git branch --show-current)"
44+
```
45+
46+
Watch the run that starts:
47+
48+
```shell
49+
sleep 5 # let GitHub register the dispatch
50+
gh run list --workflow release-canary.yml --limit 1
51+
gh run watch "$(gh run list --workflow release-canary.yml --limit 1 --json databaseId --jq '.[0].databaseId')"
52+
```
53+
54+
View only failed jobs after completion:
55+
56+
```shell
57+
gh run view <run-id> --log-failed
58+
```
59+
60+
## Iterating on the canary itself
61+
62+
When you change `release-canary.yml` on a branch, a manual dispatch on that branch tests *your branch's workflow logic* against *main's published artifacts* (`0.0.0-dev` chart, `:dev` images, latest tagged install.sh assets). This is what you want for iterating on the canary — you're validating that the canary still works against known-good artifacts.
63+
64+
Note `install.sh` is pulled from `raw.githubusercontent.com/NVIDIA/OpenShell/${head_sha}/install.sh`, so changes to `install.sh` on your branch *are* exercised even though the binaries it downloads are from the latest public tag.
65+
66+
## Testing artifacts from a specific SHA
67+
68+
`Release Dev` publishes two chart versions for every dev build (see `.github/actions/release-helm-oci/action.yml:89-102`):
69+
70+
- `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev` — floating, overwritten on every main push.
71+
- `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev.<sha>` — immutable, `appVersion` set to the same SHA so it pulls `ghcr.io/nvidia/openshell/gateway:<sha>` and `:supervisor:<sha>`.
72+
73+
To smoke-test the chart for a specific dev build, dispatch `Release Dev` on the branch first, then run the kind canary steps locally pointed at the SHA-pinned chart (see "Local kind reproduction" below). The release-canary workflow itself does not currently expose `chart_version` / `image_tag` inputs.
74+
75+
## Local kind reproduction
76+
77+
The `kubernetes` job can be reproduced on any machine with Docker and `mise install`-provided `kubectl` + `helm`:
78+
79+
```shell
80+
kind create cluster --name release-canary-local
81+
82+
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \
83+
--version 0.0.0-dev \
84+
--namespace openshell --create-namespace \
85+
--set server.disableTls=true \
86+
--set pkiInitJob.enabled=false \
87+
--wait --timeout 5m
88+
89+
kubectl wait --namespace openshell \
90+
--for=condition=Ready pod \
91+
--selector="app.kubernetes.io/name=openshell,app.kubernetes.io/instance=openshell" \
92+
--timeout=300s
93+
94+
kubectl port-forward --namespace openshell svc/openshell 8080:8080 &
95+
openshell gateway add http://127.0.0.1:8080 --local --name kind
96+
openshell status
97+
```
98+
99+
Swap `0.0.0-dev` for `0.0.0-dev.<sha>` to pin to a specific dev build. Tear down with `kind delete cluster --name release-canary-local`.
100+
101+
Loopback registration auto-derives the gateway name to `openshell` if `--name` is omitted, which collides with the `install.sh`-installed local gateway — always pass `--name kind` (or another distinct name) when registering in addition to a local install.
102+
103+
## Diagnosing failures
104+
105+
| Symptom | Likely cause | Where to look |
106+
|---|---|---|
107+
| `macos`/`ubuntu`/`fedora` job fails on `install.sh` | Latest tagged release missing an asset, checksum mismatch, or `install.sh` regression on this branch. | Job log around the `curl … install.sh \| sh` step. |
108+
| `macos`/`ubuntu`/`fedora` job fails on `openshell status` | Local gateway service did not start (systemd/brew/podman). Often a driver issue. | Service logs in the job log; `OPENSHELL_DRIVERS` env in the "Ensure …" step. |
109+
| `kubernetes` job fails on `helm install --wait` | Chart did not deploy in 5 min — usually image pull failure or readiness probe failing. | "Diagnostics on failure" step dumps `helm status`, manifest, pod describe, pod logs. |
110+
| `kubernetes` job fails on `kubectl wait` | Gateway pod stuck `CrashLoopBackOff` or `ImagePullBackOff`. | Diagnostics dump; check `:dev` image existence at `ghcr.io/nvidia/openshell/gateway`. |
111+
| `kubernetes` job fails on `openshell gateway add` or `status` | Port-forward not reachable, or CLI/gateway proto mismatch. | `port-forward.log` and `openshell gateway list` in the diagnostics dump. |
112+
113+
The `kubernetes` job's diagnostics step (only runs `if: failure()`) emits, in order: helm status, rendered manifest, `kubectl get all`, pod descriptions, pod logs (200 lines per container), port-forward log, gateway list, CLI version. Read it top-to-bottom — most failures fall out by the manifest or pod logs.
114+
115+
## Related
116+
117+
- `helm-dev-environment` skill — local k3d-based dev environment (more featureful than the canary's kind cluster, but uses Skaffold-built local images, not published artifacts).
118+
- `watch-github-actions` skill — generic `gh run` workflow monitoring.
119+
- `debug-openshell-cluster` skill — runtime gateway/sandbox diagnostics that pair with the kind job's diagnostics dump.

‎.github/workflows/release-canary.yml‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,3 +71,95 @@ jobs:
7171
run: |
7272
curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh | sh
7373
openshell status
74+
75+
kubernetes:
76+
name: Kubernetes Helm (kind)
77+
if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
78+
runs-on: ubuntu-latest
79+
timeout-minutes: 20
80+
env:
81+
KIND_CLUSTER_NAME: release-canary-${{ github.run_id }}
82+
RELEASE_NAME: openshell
83+
RELEASE_NAMESPACE: openshell
84+
KIND_GATEWAY_NAME: kind
85+
steps:
86+
- name: Install Helm
87+
uses: azure/setup-helm@v4
88+
89+
- name: Create kind cluster
90+
uses: helm/kind-action@v1
91+
with:
92+
cluster_name: ${{ env.KIND_CLUSTER_NAME }}
93+
wait: 120s
94+
95+
- name: Install OpenShell Helm chart from GHCR OCI
96+
run: |
97+
set -euo pipefail
98+
helm install "$RELEASE_NAME" oci://ghcr.io/nvidia/openshell/helm-chart \
99+
--version 0.0.0-dev \
100+
--namespace "$RELEASE_NAMESPACE" --create-namespace \
101+
--set server.disableTls=true \
102+
--set pkiInitJob.enabled=false \
103+
--wait --timeout 5m
104+
105+
- name: Verify gateway pod is Ready
106+
run: |
107+
set -euo pipefail
108+
kubectl wait --namespace "$RELEASE_NAMESPACE" \
109+
--for=condition=Ready pod \
110+
--selector="app.kubernetes.io/name=openshell,app.kubernetes.io/instance=${RELEASE_NAME}" \
111+
--timeout=300s
112+
113+
- name: Port-forward gateway service
114+
run: |
115+
set -euo pipefail
116+
nohup kubectl port-forward --namespace "$RELEASE_NAMESPACE" \
117+
"svc/${RELEASE_NAME}" 8080:8080 \
118+
> port-forward.log 2>&1 &
119+
echo $! > port-forward.pid
120+
for _ in $(seq 1 30); do
121+
if (echo > /dev/tcp/127.0.0.1/8080) >/dev/null 2>&1; then
122+
echo "port-forward is reachable"
123+
exit 0
124+
fi
125+
sleep 1
126+
done
127+
echo "port-forward did not become reachable" >&2
128+
cat port-forward.log >&2
129+
exit 1
130+
131+
- name: Install OpenShell CLI
132+
run: |
133+
set -euo pipefail
134+
mkdir -p "${HOME}/.config/openshell"
135+
printf 'OPENSHELL_DRIVERS=docker\n' > "${HOME}/.config/openshell/gateway.env"
136+
curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh | sh
137+
138+
- name: Register kind gateway and check status
139+
run: |
140+
set -euo pipefail
141+
openshell gateway add http://127.0.0.1:8080 --local --name "$KIND_GATEWAY_NAME"
142+
openshell status
143+
144+
- name: Diagnostics on failure
145+
if: failure()
146+
run: |
147+
set +e
148+
echo "--- helm status ---"
149+
helm status "$RELEASE_NAME" --namespace "$RELEASE_NAMESPACE"
150+
echo "--- helm get manifest ---"
151+
helm get manifest "$RELEASE_NAME" --namespace "$RELEASE_NAMESPACE"
152+
echo "--- get all ---"
153+
kubectl get all --namespace "$RELEASE_NAMESPACE"
154+
echo "--- describe pods ---"
155+
kubectl describe pods --namespace "$RELEASE_NAMESPACE"
156+
echo "--- pod logs ---"
157+
kubectl logs --namespace "$RELEASE_NAMESPACE" \
158+
--selector="app.kubernetes.io/name=openshell,app.kubernetes.io/instance=${RELEASE_NAME}" \
159+
--tail=200 --all-containers --prefix
160+
echo "--- port-forward log ---"
161+
cat port-forward.log 2>/dev/null
162+
echo "--- openshell gateway list ---"
163+
openshell gateway list 2>/dev/null
164+
echo "--- openshell version ---"
165+
openshell --version 2>/dev/null

‎CI.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,3 +111,13 @@ The bot's full administrator documentation is internal to NVIDIA. The only comma
111111
| `.github/workflows/e2e-gate.yml` | Posts the required `E2E Gate` check on the PR. Re-evaluates after the gated workflow completes. |
112112
| `.github/workflows/e2e-gate-check.yml` | Reusable gate logic shared by E2E and GPU E2E. |
113113
| `.github/workflows/e2e-label-help.yml` | When a `test:e2e*` label is applied, posts a PR comment telling the maintainer the next manual step (re-run an existing workflow run, or `/ok to test <SHA>` to refresh the mirror). |
114+
115+
## Release workflows
116+
117+
These workflows run after merge to publish dev/tagged artifacts and verify them. They are not PR-gated.
118+
119+
| File | Role |
120+
|---|---|
121+
| `.github/workflows/release-dev.yml` | Publishes the rolling `dev` build on every push to `main`. Builds gateway/supervisor images and binaries, packages, wheels, and pushes the Helm chart as `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev` (plus an immutable `0.0.0-dev.<sha>` pin). Also dispatchable manually. |
122+
| `.github/workflows/release-tag.yml` | Publishes a tagged public release. |
123+
| `.github/workflows/release-canary.yml` | Smoke-tests published artifacts on `macos`, `ubuntu`, `fedora`, and `kubernetes` (kind + Helm) runners. Triggers automatically when `Release Dev` succeeds, and via `workflow_dispatch` on any branch (`gh workflow run release-canary.yml --ref <branch>`). The `kubernetes` job pins to `0.0.0-dev` artifacts; the other jobs install the latest tagged release via `install.sh`. See the `test-release-canary` skill for the manual-dispatch playbook and local kind reproduction. |

‎CONTRIBUTING.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,7 @@ Skills live in `.agents/skills/`. Your agent's harness can discover and load the
7575
| Reviewing | `review-github-pr` | Summarize PR diffs and key design decisions |
7676
| Reviewing | `review-security-issue` | Assess security issues for severity and remediation |
7777
| Reviewing | `watch-github-actions` | Monitor CI pipeline status and logs |
78+
| Reviewing | `test-release-canary` | Dispatch and iterate on the Release Canary workflow that smoke-tests published artifacts |
7879
| Triage | `triage-issue` | Assess, classify, and route community-filed issues |
7980
| Platform | `generate-sandbox-policy` | Generate YAML sandbox policies from requirements or API docs |
8081
| Platform | `tui-development` | Development guide for the ratatui-based terminal UI |

0 commit comments

Comments
 (0)