Skip to content

Commit 0a5b5ed

Browse files
committed
ci(e2e): add RHEL 10 Docker/Podman/SELinux compatibility suite
PR #2092 adds SELinux relabeling (:z/:Z) support to the Docker and Podman driver bind mounts, but our existing E2E lanes run inside the Ubuntu-based ghcr.io/nvidia/openshell/ci container image, which does not enforce SELinux. That leaves the new relabeling behavior, and any future SELinux-sensitive change, without a host that can actually deny a mislabeled mount. Add an opt-in `test:e2e-rhel` label that runs a new reusable workflow, e2e-rhel-selinux.yml, on a bare (non-containerized) RHEL 10 runner. The job installs Docker Engine alongside RHEL's preinstalled Podman using the commands from the PR discussion (get.docker.com plus the --ip-forward-no-drop dockerd override needed for the two engines to coexist), verifies SELinux is enforcing, runs the standard e2e:docker and e2e:podman suites against both engines on the same host, and fails the job if the audit log shows AVC denials during the run. This requires an org-provisioned GitHub-hosted "larger runner" using the RHEL 10 partner image (public preview per the June 2026 GitHub changelog) under the linux-amd64-rhel10 label; until that runner exists, jobs using the default label will queue. The suite is wired in as optional/non-blocking, matching the existing test:e2e-kubernetes pattern, since it exercises unproven infrastructure. Documentation in CI.md and CONTRIBUTING.md is updated to list the new label alongside the existing E2E labels. Signed-off-by: Eric Curtin <eric.curtin@docker.com>
1 parent 45614a3 commit 0a5b5ed

5 files changed

Lines changed: 237 additions & 7 deletions

File tree

.github/workflows/branch-e2e.yml

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ jobs:
2424
run_core_e2e: ${{ steps.labels.outputs.run_core_e2e }}
2525
run_gpu_e2e: ${{ steps.labels.outputs.run_gpu_e2e }}
2626
run_kubernetes_ha_e2e: ${{ steps.labels.outputs.run_kubernetes_ha_e2e }}
27+
run_rhel_e2e: ${{ steps.labels.outputs.run_rhel_e2e }}
2728
run_any_e2e: ${{ steps.labels.outputs.run_any_e2e }}
2829
steps:
2930
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
@@ -41,12 +42,14 @@ jobs:
4142
run_core_e2e=true
4243
run_gpu_e2e=true
4344
run_kubernetes_ha_e2e=true
45+
run_rhel_e2e=true
4446
else
4547
run_core_e2e="$(jq -r 'index("test:e2e") != null' <<< "$LABELS_JSON")"
4648
run_gpu_e2e="$(jq -r 'index("test:e2e-gpu") != null' <<< "$LABELS_JSON")"
4749
run_kubernetes_ha_e2e="$(jq -r 'index("test:e2e-kubernetes") != null' <<< "$LABELS_JSON")"
50+
run_rhel_e2e="$(jq -r 'index("test:e2e-rhel") != null' <<< "$LABELS_JSON")"
4851
fi
49-
if [ "$run_core_e2e" = "true" ] || [ "$run_gpu_e2e" = "true" ] || [ "$run_kubernetes_ha_e2e" = "true" ]; then
52+
if [ "$run_core_e2e" = "true" ] || [ "$run_gpu_e2e" = "true" ] || [ "$run_kubernetes_ha_e2e" = "true" ] || [ "$run_rhel_e2e" = "true" ]; then
5053
run_any_e2e=true
5154
else
5255
run_any_e2e=false
@@ -55,6 +58,7 @@ jobs:
5558
echo "run_core_e2e=$run_core_e2e"
5659
echo "run_gpu_e2e=$run_gpu_e2e"
5760
echo "run_kubernetes_ha_e2e=$run_kubernetes_ha_e2e"
61+
echo "run_rhel_e2e=$run_rhel_e2e"
5862
echo "run_any_e2e=$run_any_e2e"
5963
} >> "$GITHUB_OUTPUT"
6064
@@ -145,6 +149,19 @@ jobs:
145149
extra-helm-values: deploy/helm/openshell/ci/values-high-availability.yaml
146150
external-postgres-secret: openshell-ha-pg
147151

152+
rhel-selinux-e2e:
153+
needs: [pr_metadata, build-supervisor]
154+
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_rhel_e2e == 'true'
155+
# Optional proof-of-life suite on a new runner class (requires an
156+
# org-provisioned RHEL 10 larger runner). Not part of the required E2E
157+
# gate; see kubernetes-ha-e2e above for the same pattern.
158+
permissions:
159+
contents: read
160+
packages: read
161+
uses: ./.github/workflows/e2e-rhel-selinux.yml
162+
with:
163+
image-tag: ${{ github.sha }}
164+
148165
core-e2e-result:
149166
name: Core E2E result
150167
needs: [pr_metadata, build-gateway, build-supervisor, e2e, kubernetes-e2e]

.github/workflows/e2e-label-help.yml

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ permissions: {}
1919
jobs:
2020
hint:
2121
name: Post next-step hint for E2E label
22-
if: github.event.label.name == 'test:e2e' || github.event.label.name == 'test:e2e-gpu' || github.event.label.name == 'test:e2e-kubernetes'
22+
if: github.event.label.name == 'test:e2e' || github.event.label.name == 'test:e2e-gpu' || github.event.label.name == 'test:e2e-kubernetes' || github.event.label.name == 'test:e2e-rhel'
2323
runs-on: ubuntu-latest
2424
permissions:
2525
pull-requests: write
@@ -55,6 +55,11 @@ jobs:
5555
build_summary="gateway and supervisor images"
5656
status_summary="This is an optional proof-of-life suite; failures are visible in the workflow run but do not publish a required CI gate status."
5757
;;
58+
test:e2e-rhel)
59+
suite_summary="the RHEL 10 Docker/Podman/SELinux compatibility suite"
60+
build_summary="supervisor image"
61+
status_summary="This is an optional proof-of-life suite on a new runner class; failures are visible in the workflow run but do not publish a required CI gate status."
62+
;;
5863
*) echo "Unrecognized label $LABEL_NAME"; exit 1 ;;
5964
esac
6065
Lines changed: 205 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,205 @@
1+
name: RHEL 10 Docker/Podman/SELinux Compatibility
2+
3+
# Exercises the Docker and Podman compute drivers on a bare RHEL 10 runner
4+
# with SELinux enforcing. This is the cross-compatibility check requested in
5+
# https://github.com/NVIDIA/OpenShell/pull/2092: RHEL ships Podman but not
6+
# Docker, so this job installs Docker Engine alongside the preinstalled
7+
# Podman, confirms both engines coexist on one host, and runs the standard
8+
# Docker/Podman e2e suites against a real SELinux-enforcing kernel (unlike
9+
# the Ubuntu-based `ghcr.io/nvidia/openshell/ci` image used by e2e-test.yml,
10+
# which does not enforce SELinux). This gives bind-mount relabeling changes
11+
# (e.g. the `selinux_label`/`:z`/`:Z` support added in #2092) a host that can
12+
# actually deny mislabeled mounts.
13+
#
14+
# Runs on a bare runner (no `container:` image) because installing/configuring
15+
# dockerd and reading SELinux/audit state requires direct access to the host,
16+
# which a containerized step cannot provide.
17+
#
18+
# Requires a GitHub-hosted "larger runner" provisioned with the RHEL 10
19+
# partner image under this repo's runner label (see
20+
# https://github.blog/changelog/2026-06-25-red-hat-enterprise-linux-runner-images-are-now-in-public-preview/).
21+
# That provisioning happens in GitHub org settings, not in this repo; until an
22+
# org admin creates a runner with the `linux-amd64-rhel10` label, jobs using
23+
# the default `runner` input will queue indefinitely.
24+
25+
on:
26+
workflow_call:
27+
inputs:
28+
image-tag:
29+
description: "Image tag to test (typically the commit SHA)"
30+
required: true
31+
type: string
32+
runner:
33+
description: "GitHub Actions runner label (must be a RHEL 10 image)"
34+
required: false
35+
type: string
36+
default: "linux-amd64-rhel10"
37+
checkout-ref:
38+
description: "Git ref to check out for test inputs (defaults to the workflow SHA)"
39+
required: false
40+
type: string
41+
default: ""
42+
mise-version:
43+
description: "mise version to install on the bare RHEL runner"
44+
required: false
45+
type: string
46+
default: "v2026.4.25"
47+
48+
permissions:
49+
contents: read
50+
packages: read
51+
52+
jobs:
53+
rhel-selinux-compat:
54+
name: RHEL 10 Docker + Podman + SELinux
55+
runs-on: ${{ inputs.runner }}
56+
timeout-minutes: 45
57+
permissions:
58+
contents: read
59+
packages: read
60+
env:
61+
MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
62+
MISE_VERSION: ${{ inputs.mise-version }}
63+
IMAGE_TAG: ${{ inputs.image-tag }}
64+
OPENSHELL_SUPERVISOR_IMAGE: ${{ format('ghcr.io/nvidia/openshell/supervisor:{0}', inputs.image-tag) }}
65+
OPENSHELL_REGISTRY: ghcr.io/nvidia/openshell
66+
OPENSHELL_REGISTRY_HOST: ghcr.io
67+
OPENSHELL_REGISTRY_NAMESPACE: nvidia/openshell
68+
OPENSHELL_REGISTRY_USERNAME: ${{ github.actor }}
69+
OPENSHELL_REGISTRY_PASSWORD: ${{ secrets.GITHUB_TOKEN }}
70+
steps:
71+
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
72+
with:
73+
ref: ${{ inputs['checkout-ref'] || github.sha }}
74+
75+
- name: Record host facts
76+
run: |
77+
set -euo pipefail
78+
cat /etc/os-release
79+
uname -a
80+
getenforce || echo "getenforce not available"
81+
82+
- name: Require SELinux enforcing
83+
run: |
84+
set -euo pipefail
85+
mode="$(getenforce)"
86+
echo "SELinux mode: ${mode}"
87+
if [ "${mode}" != "Enforcing" ]; then
88+
echo "::error::SELinux must be Enforcing on this runner for the compatibility check to be meaningful (got: ${mode})"
89+
exit 1
90+
fi
91+
92+
- name: Record job start time for audit search
93+
id: audit_window
94+
run: echo "start=$(date -u '+%m/%d/%Y %H:%M:%S')" >> "$GITHUB_OUTPUT"
95+
96+
- name: Install build and test dependencies
97+
run: |
98+
set -euo pipefail
99+
sudo dnf install -y \
100+
ca-certificates \
101+
curl \
102+
git \
103+
gcc \
104+
gcc-c++ \
105+
make \
106+
clang \
107+
clang-devel \
108+
z3-devel \
109+
pkgconf-pkg-config \
110+
openssl-devel \
111+
openssh-clients \
112+
python3 \
113+
cmake \
114+
socat \
115+
unzip \
116+
xz \
117+
jq \
118+
rsync \
119+
zstd \
120+
audit \
121+
policycoreutils
122+
123+
- name: Install Docker Engine
124+
# RHEL doesn't ship Docker; install it alongside the preinstalled
125+
# Podman, per https://github.com/NVIDIA/OpenShell/pull/2092#issuecomment-4856873804.
126+
run: |
127+
set -euo pipefail
128+
curl -fsSL https://get.docker.com | sudo bash
129+
sudo sed -i "s#/usr/bin/dockerd#/usr/bin/dockerd --ip-forward-no-drop#g" /usr/lib/systemd/system/docker.service
130+
sudo systemctl daemon-reload
131+
sudo systemctl enable --now docker
132+
133+
- name: Grant runner user access to the Docker socket
134+
run: |
135+
set -euo pipefail
136+
sudo groupadd -f docker
137+
sudo usermod -aG docker "$(id -un)"
138+
sudo chmod 666 /var/run/docker.sock
139+
140+
- name: Verify Podman is available
141+
run: |
142+
set -euo pipefail
143+
if ! command -v podman >/dev/null 2>&1; then
144+
echo "podman not preinstalled; installing via dnf"
145+
sudo dnf install -y podman
146+
fi
147+
podman --version
148+
149+
- name: Verify Docker and Podman coexist
150+
run: |
151+
set -euo pipefail
152+
docker version
153+
docker info
154+
podman version
155+
podman info
156+
157+
- name: Install mise
158+
run: |
159+
curl https://mise.run | MISE_VERSION="${MISE_VERSION}" sh
160+
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
161+
echo "$HOME/.local/share/mise/shims" >> "$GITHUB_PATH"
162+
163+
- name: Install tools
164+
run: mise install --locked
165+
166+
- name: Log in to GHCR with Docker
167+
run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${{ github.actor }}" --password-stdin
168+
169+
- name: Log in to GHCR with Podman
170+
run: echo "${{ secrets.GITHUB_TOKEN }}" | podman login ghcr.io -u "${{ github.actor }}" --password-stdin
171+
172+
- name: Run Docker-backed e2e suite
173+
run: mise run --no-deps --skip-deps e2e:docker
174+
175+
- name: Run Podman-backed e2e suite
176+
run: mise run --no-deps --skip-deps e2e:podman
177+
178+
- name: Check for unexpected SELinux denials
179+
run: |
180+
set -euo pipefail
181+
echo "Searching audit log for AVC denials since ${AUDIT_WINDOW_START}"
182+
if sudo ausearch -m avc,user_avc -ts "${AUDIT_WINDOW_START}" 2>/dev/null | tee /tmp/avc-denials.log | grep -q .; then
183+
echo "::error::SELinux denied one or more operations during the e2e suites; see the log below"
184+
cat /tmp/avc-denials.log
185+
exit 1
186+
fi
187+
echo "No SELinux denials recorded."
188+
env:
189+
AUDIT_WINDOW_START: ${{ steps.audit_window.outputs.start }}
190+
191+
- name: Collect SELinux/audit context on failure
192+
if: failure()
193+
run: |
194+
set +e
195+
echo "=== getenforce ==="
196+
getenforce
197+
echo "=== recent AVC denials ==="
198+
sudo ausearch -m avc,user_avc -ts "${AUDIT_WINDOW_START}" 2>/dev/null
199+
echo "=== dockerd status ==="
200+
sudo systemctl status docker --no-pager
201+
echo "=== recent kernel log ==="
202+
sudo journalctl -k --no-pager | tail -n 200
203+
exit 0
204+
env:
205+
AUDIT_WINDOW_START: ${{ steps.audit_window.outputs.start }}

CI.md

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,15 +10,17 @@ PR CI that runs on NVIDIA self-hosted runners uses NVIDIA's copy-pr-bot. The bot
1010

1111
`Branch Checks` run automatically after copy-pr-bot mirrors the PR. `Required CI Gates` posts PR-head statuses that verify the mirror exists, is current, and ran the expected push-based workflows. E2E suites are opt-in because they are more expensive and publish temporary images.
1212

13-
Three opt-in labels enable the long-running E2E suites:
13+
Four opt-in labels enable the long-running E2E suites:
1414

1515
- `test:e2e` runs the standard E2E suite in `Branch E2E Checks`
1616
- `test:e2e-gpu` runs GPU E2E in `Branch E2E Checks`
1717
- `test:e2e-kubernetes` runs Kubernetes E2E with the HA Helm overlay
1818
(`replicaCount: 2` and bundled PostgreSQL) in `Branch E2E Checks`
19+
- `test:e2e-rhel` runs the Docker/Podman/SELinux compatibility suite on a bare
20+
RHEL 10 runner in `Branch E2E Checks`
1921

2022
When multiple labels are present, `Branch E2E Checks` builds the shared gateway and supervisor images once and fans out all enabled suites in parallel.
21-
The `OpenShell / E2E` and `OpenShell / GPU E2E` required statuses are evaluated from separate suite result jobs inside that workflow. `test:e2e-kubernetes` is optional while HA behavior is under active iteration: failures are visible in the workflow run but do not publish a required CI gate status.
23+
The `OpenShell / E2E` and `OpenShell / GPU E2E` required statuses are evaluated from separate suite result jobs inside that workflow. `test:e2e-kubernetes` and `test:e2e-rhel` are optional: failures are visible in the workflow run but do not publish a required CI gate status. `test:e2e-rhel` additionally requires an org-provisioned RHEL 10 larger runner (see `.github/workflows/e2e-rhel-selinux.yml`).
2224

2325
The GitHub ruleset should require the `OpenShell / ...` statuses published by `Required CI Gates`, not the push-triggered workflow jobs directly.
2426

@@ -71,7 +73,7 @@ Flow:
7173

7274
1. Open the PR. copy-pr-bot mirrors it to `pull-request/<N>` automatically.
7375
2. The mirror push runs `Branch Checks` automatically. `Required CI Gates` keeps the PR blocked until the mirror exists, matches the PR head SHA, and the required push-based workflow succeeds. The first `Branch E2E Checks` run only resolves metadata and skips expensive jobs unless an E2E label is already set.
74-
3. A maintainer applies `test:e2e`, `test:e2e-gpu`, and/or `test:e2e-kubernetes`. `E2E Label Help` posts a comment with a link to the existing gated workflow run.
76+
3. A maintainer applies `test:e2e`, `test:e2e-gpu`, `test:e2e-kubernetes`, and/or `test:e2e-rhel`. `E2E Label Help` posts a comment with a link to the existing gated workflow run.
7577
4. The maintainer opens that link and clicks **Re-run all jobs**. This time `pr_metadata` sees the label and the build/E2E jobs run.
7678
5. When the run finishes, the matching `OpenShell / ...` gate status flips to green automatically.
7779
6. New commits push to the mirror automatically and re-trigger `Branch Checks` plus any labeled E2E jobs in `Branch E2E Checks`.
@@ -110,7 +112,8 @@ The bot's full administrator documentation is internal to NVIDIA. The only comma
110112
| File | Role |
111113
|---|---|
112114
| `.github/workflows/branch-checks.yml` | Required non-E2E PR checks. Triggers on `push: pull-request/[0-9]+`. |
113-
| `.github/workflows/branch-e2e.yml` | Opt-in standard, GPU, and Kubernetes HA E2E. Triggers on `push: pull-request/[0-9]+` and runs jobs selected by `test:e2e`, `test:e2e-gpu`, or `test:e2e-kubernetes`. |
115+
| `.github/workflows/branch-e2e.yml` | Opt-in standard, GPU, Kubernetes HA, and RHEL 10 compatibility E2E. Triggers on `push: pull-request/[0-9]+` and runs jobs selected by `test:e2e`, `test:e2e-gpu`, `test:e2e-kubernetes`, or `test:e2e-rhel`. |
116+
| `.github/workflows/e2e-rhel-selinux.yml` | Reusable workflow that installs Docker alongside Podman on a bare RHEL 10 runner, verifies SELinux is enforcing, and runs the Docker/Podman e2e suites. Requires an org-provisioned RHEL 10 larger runner. |
114117
| `.github/workflows/helm-lint.yml` | Helm chart validation. Triggers on `push: pull-request/[0-9]+` and skips lint jobs unless Helm inputs changed. |
115118
| `.github/actions/pr-gate/action.yml` | Composite action that resolves PR metadata and verifies the required label is set. |
116119
| `.github/actions/pr-merge-base/action.yml` | Composite action that resolves and fetches the merge-base commit for `pull-request/<N>` push workflows. |

CONTRIBUTING.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -305,4 +305,4 @@ DCO sign-off is separate from cryptographic commit signing. CI requires signing
305305

306306
## CI
307307

308-
How PR CI runs, the `test:e2e`, `test:e2e-gpu`, and `test:e2e-kubernetes` labels, copy-pr-bot, and commit-signing setup are documented in [CI.md](CI.md).
308+
How PR CI runs, the `test:e2e`, `test:e2e-gpu`, `test:e2e-kubernetes`, and `test:e2e-rhel` labels, copy-pr-bot, and commit-signing setup are documented in [CI.md](CI.md).

0 commit comments

Comments
 (0)