build(python): publish manylinux_2_28 wheels for broader glibc compatibility

Build released Linux wheels inside a PyPA manylinux_2_28 container so the
resulting binary runs on any Linux distribution with glibc >= 2.28
(RHEL 8, Debian 10+, Ubuntu 18.04+). The previous native build on the
noble (glibc 2.39) CI image produced manylinux_2_39 wheels that uv
refused to install on common LTS hosts such as Ubuntu 22.04.

- Add deploy/docker/Dockerfile.python-wheels-linux based on
  quay.io/pypa/manylinux_2_28_{x86_64,aarch64}.
- Add build:python:wheel:linux:docker mise task and per-arch aliases.
  Re-point python:build:linux:{amd64,arm64} (used by release workflows)
  to the Docker-based path. The legacy native build:python:wheel:linux
  task remains for fast local iteration.
- Mount the host docker socket and set up buildx in the
  build-python-wheels-linux job in release-tag.yml and release-dev.yml,
  mirroring the existing macOS wheel job.
- Update architecture/build-containers.md to describe the new flow.

Signed-off-by: phamgialinhlx <phamgialinhlx2@gmail.com>

Author: phamgialinhlx

.github/workflows/release-dev.yml

@@ -120,6 +120,8 @@ jobs:
         username: ${{ github.actor }}
         password: ${{ secrets.GITHUB_TOKEN }}
       options: --privileged
+      volumes:
+        - /var/run/docker.sock:/var/run/docker.sock
     env:
       MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
       SCCACHE_MEMCACHED_ENDPOINT: ${{ vars.SCCACHE_MEMCACHED_ENDPOINT }}
@@ -129,6 +131,9 @@ jobs:
         with:
           fetch-depth: 0
 
+      - name: Set up Docker Buildx
+        uses: ./.github/actions/setup-buildx
+
       - name: Mark workspace safe for git
         run: git config --global --add safe.directory "$GITHUB_WORKSPACE"
 

.github/workflows/release-tag.yml

@@ -140,6 +140,8 @@ jobs:
         username: ${{ github.actor }}
         password: ${{ secrets.GITHUB_TOKEN }}
       options: --privileged
+      volumes:
+        - /var/run/docker.sock:/var/run/docker.sock
     env:
       MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
       SCCACHE_MEMCACHED_ENDPOINT: ${{ vars.SCCACHE_MEMCACHED_ENDPOINT }}
@@ -150,6 +152,9 @@ jobs:
           ref: ${{ inputs.tag || github.ref }}
           fetch-depth: 0
 
+      - name: Set up Docker Buildx
+        uses: ./.github/actions/setup-buildx
+
       - name: Mark workspace safe for git
         run: git config --global --add safe.directory "$GITHUB_WORKSPACE"
 

architecture/build-containers.md

@@ -43,8 +43,8 @@ Both the standalone artifact and the deployed container image use the `openshell
 
 OpenShell also publishes Python wheels for `linux/amd64`, `linux/arm64`, and macOS ARM64.
 
-- Linux wheels are built natively on matching Linux runners via `build:python:wheel:linux:amd64` and `build:python:wheel:linux:arm64` in `tasks/python.toml`.
-- There is no local Linux multiarch wheel build task. Release workflows own the per-arch Linux wheel production.
+- Released Linux wheels are built per-arch inside `deploy/docker/Dockerfile.python-wheels-linux`, which uses the PyPA `manylinux_2_28_{x86_64,aarch64}` images as a base. The resulting wheels are tagged `manylinux_2_28` and install on any Linux distribution shipping glibc >= 2.28 (RHEL 8, Debian 10+, Ubuntu 18.04+). The matrix job invokes `mise run python:build:linux:amd64` / `python:build:linux:arm64`, which fan out to `build:python:wheel:linux:{amd64,arm64}:docker`.
+- For fast local iteration, `build:python:wheel:linux:amd64` / `build:python:wheel:linux:arm64` still produce a non-portable wheel tagged for the host's glibc. These tasks are no longer used by release workflows.
 - The macOS ARM64 wheel is cross-compiled with `deploy/docker/Dockerfile.python-wheels-macos` via `build:python:wheel:macos`.
 - Release workflows mirror the CLI layout: a Linux matrix job for amd64/arm64, a separate macOS job, and release jobs that download the per-platform wheel artifacts directly before publishing.
 

deploy/docker/Dockerfile.python-wheels-linux

@@ -0,0 +1,119 @@
+# syntax=docker/dockerfile:1.6
+
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+# Build the Linux Python wheel inside a PyPA manylinux_2_28 container so the
+# resulting binary is compatible with any Linux distribution shipping glibc
+# >= 2.28 (RHEL 8, Ubuntu 18.04+, Debian 10+). The host CI runner is on noble
+# (glibc 2.39), which previously produced manylinux_2_39 wheels that uv refuses
+# to install on Ubuntu 22.04 / Debian 11 (glibc 2.31 / 2.35).
+
+ARG TARGETARCH
+ARG MANYLINUX_AMD64_IMAGE=quay.io/pypa/manylinux_2_28_x86_64:latest
+ARG MANYLINUX_ARM64_IMAGE=quay.io/pypa/manylinux_2_28_aarch64:latest
+ARG PYTHON_VERSION=cp312-cp312
+ARG RUST_VERSION=1.95.0
+
+# Selector stages — Docker resolves only the matching one based on TARGETARCH.
+FROM ${MANYLINUX_AMD64_IMAGE} AS base-amd64
+FROM ${MANYLINUX_ARM64_IMAGE} AS base-arm64
+FROM base-${TARGETARCH} AS builder
+
+ARG TARGETARCH
+ARG PYTHON_VERSION
+ARG RUST_VERSION
+ARG CARGO_TARGET_CACHE_SCOPE=default
+
+ENV PATH="/opt/python/${PYTHON_VERSION}/bin:/root/.cargo/bin:${PATH}"
+
+# manylinux_2_28 ships gcc-toolset-14, cmake, and patchelf. We add clang for
+# bindgen-driven crates (libclang-dev equivalent) and openssl-devel for any
+# Rust crates that link against the system libssl during dependency resolution.
+RUN dnf install -y --setopt=install_weak_deps=False \
+        clang \
+        llvm-devel \
+        openssl-devel \
+        perl-core \
+        perl-IPC-Cmd \
+    && dnf clean all
+
+RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs \
+      | sh -s -- -y --default-toolchain ${RUST_VERSION} --profile minimal
+RUN python -m pip install --no-cache-dir maturin
+
+WORKDIR /build
+
+# Copy dependency manifests first for better layer caching.
+COPY Cargo.toml Cargo.lock ./
+COPY crates/openshell-cli/Cargo.toml crates/openshell-cli/Cargo.toml
+COPY crates/openshell-core/Cargo.toml crates/openshell-core/Cargo.toml
+COPY crates/openshell-ocsf/Cargo.toml crates/openshell-ocsf/Cargo.toml
+COPY crates/openshell-providers/Cargo.toml crates/openshell-providers/Cargo.toml
+COPY crates/openshell-router/Cargo.toml crates/openshell-router/Cargo.toml
+COPY crates/openshell-sandbox/Cargo.toml crates/openshell-sandbox/Cargo.toml
+COPY crates/openshell-server/Cargo.toml crates/openshell-server/Cargo.toml
+COPY crates/openshell-bootstrap/Cargo.toml crates/openshell-bootstrap/Cargo.toml
+COPY crates/openshell-policy/Cargo.toml crates/openshell-policy/Cargo.toml
+COPY crates/openshell-prover/Cargo.toml crates/openshell-prover/Cargo.toml
+COPY crates/openshell-tui/Cargo.toml crates/openshell-tui/Cargo.toml
+COPY crates/openshell-core/build.rs crates/openshell-core/build.rs
+COPY proto/ proto/
+
+# Create dummy source files to build dependencies.
+RUN mkdir -p crates/openshell-cli/src crates/openshell-core/src crates/openshell-ocsf/src \
+        crates/openshell-policy/src crates/openshell-providers/src crates/openshell-prover/src \
+        crates/openshell-router/src crates/openshell-sandbox/src crates/openshell-server/src \
+        crates/openshell-bootstrap/src crates/openshell-tui/src && \
+    echo "fn main() {}" > crates/openshell-cli/src/main.rs && \
+    echo "fn main() {}" > crates/openshell-sandbox/src/main.rs && \
+    echo "fn main() {}" > crates/openshell-server/src/main.rs && \
+    touch crates/openshell-core/src/lib.rs && \
+    touch crates/openshell-ocsf/src/lib.rs && \
+    touch crates/openshell-providers/src/lib.rs && \
+    touch crates/openshell-router/src/lib.rs && \
+    touch crates/openshell-bootstrap/src/lib.rs && \
+    touch crates/openshell-policy/src/lib.rs && \
+    touch crates/openshell-prover/src/lib.rs && \
+    touch crates/openshell-tui/src/lib.rs
+
+# Warm the dependency build (cached unless Cargo.toml/lock changes).
+RUN --mount=type=cache,id=cargo-registry-python-wheels-linux-${TARGETARCH},sharing=locked,target=/root/.cargo/registry \
+    --mount=type=cache,id=cargo-git-python-wheels-linux-${TARGETARCH},sharing=locked,target=/root/.cargo/git \
+    --mount=type=cache,id=cargo-target-python-wheels-linux-${TARGETARCH}-${CARGO_TARGET_CACHE_SCOPE},sharing=locked,target=/build/target \
+    cargo build --release -p openshell-cli --features bundled-z3 2>/dev/null || true
+
+# Copy actual source code and Python packaging files.
+COPY crates/ crates/
+COPY pyproject.toml README.md ./
+COPY python/ python/
+
+# Touch source files so cargo rebuilds them (not the cached dummy).
+RUN touch crates/openshell-cli/src/main.rs \
+    crates/openshell-cli/src/lib.rs \
+    crates/openshell-bootstrap/src/lib.rs \
+    crates/openshell-core/src/lib.rs \
+    crates/openshell-providers/src/lib.rs \
+    crates/openshell-router/src/lib.rs \
+    crates/openshell-sandbox/src/main.rs \
+    crates/openshell-server/src/main.rs \
+    crates/openshell-core/build.rs \
+    proto/*.proto
+
+# Declare version ARGs here (not earlier) so the git-hash-bearing values do not
+# invalidate the expensive dependency-build layers above on every commit.
+ARG OPENSHELL_CARGO_VERSION
+ARG OPENSHELL_IMAGE_TAG
+RUN --mount=type=cache,id=cargo-registry-python-wheels-linux-${TARGETARCH},sharing=locked,target=/root/.cargo/registry \
+    --mount=type=cache,id=cargo-git-python-wheels-linux-${TARGETARCH},sharing=locked,target=/root/.cargo/git \
+    --mount=type=cache,id=cargo-target-python-wheels-linux-${TARGETARCH}-${CARGO_TARGET_CACHE_SCOPE},sharing=locked,target=/build/target \
+    if [ -n "${OPENSHELL_CARGO_VERSION:-}" ]; then \
+      sed -i -E '/^\[workspace\.package\]/,/^\[/{s/^version[[:space:]]*=[[:space:]]*".*"/version = "'"${OPENSHELL_CARGO_VERSION}"'"/}' Cargo.toml; \
+    fi && \
+    maturin build --release --features bundled-z3 \
+      --compatibility manylinux_2_28 \
+      --out /wheels && \
+    ls -la /wheels/*.whl
+
+FROM scratch AS wheels
+COPY --from=builder /wheels/*.whl /

mise.lock

@@ -83,23 +83,93 @@ ls -la "$WHEEL_OUTPUT_DIR"/*.whl
 hide = true
 
 ["build:python:wheel:linux:amd64"]
-description = "Build Python wheel for Linux amd64 natively"
+description = "Build Python wheel for Linux amd64 natively (host glibc; non-portable)"
 depends = ["EXPECTED_HOST_ARCH=amd64 WHEEL_OUTPUT_DIR=target/wheels/linux-amd64 build:python:wheel:linux"]
 hide = true
 
-["python:build:linux:amd64"]
-description = "Alias for build:python:wheel:linux:amd64"
-depends = ["build:python:wheel:linux:amd64"]
-hide = true
-
 ["build:python:wheel:linux:arm64"]
-description = "Build Python wheel for Linux arm64 natively"
+description = "Build Python wheel for Linux arm64 natively (host glibc; non-portable)"
 depends = ["EXPECTED_HOST_ARCH=arm64 WHEEL_OUTPUT_DIR=target/wheels/linux-arm64 build:python:wheel:linux"]
 hide = true
 
+["build:python:wheel:linux:docker"]
+description = "Build a portable manylinux_2_28 Python wheel via Docker (glibc >= 2.28)"
+depends = ["python:proto"]
+run = """
+#!/usr/bin/env bash
+set -euo pipefail
+
+source tasks/scripts/container-engine.sh
+
+WHEEL_OUTPUT_DIR=${WHEEL_OUTPUT_DIR:?Set WHEEL_OUTPUT_DIR to a per-platform wheel output directory}
+TARGETARCH=${TARGETARCH:?Set TARGETARCH to amd64 or arm64}
+
+sha256_16() {
+  if command -v sha256sum >/dev/null 2>&1; then
+    sha256sum "$1" | awk '{print substr($1, 1, 16)}'
+  else
+    shasum -a 256 "$1" | awk '{print substr($1, 1, 16)}'
+  fi
+}
+
+sha256_16_stdin() {
+  if command -v sha256sum >/dev/null 2>&1; then
+    sha256sum | awk '{print substr($1, 1, 16)}'
+  else
+    shasum -a 256 | awk '{print substr($1, 1, 16)}'
+  fi
+}
+
+CARGO_VERSION=${OPENSHELL_CARGO_VERSION:-}
+if [ -z "$CARGO_VERSION" ] && [ -n "${CI:-}" ]; then
+  CARGO_VERSION=$(uv run python tasks/scripts/release.py get-version --cargo)
+fi
+
+LOCK_HASH=$(sha256_16 Cargo.lock)
+RUST_SCOPE=${RUST_TOOLCHAIN_SCOPE:-rustup-1.95.0}
+CACHE_SCOPE_INPUT="v1|python-wheels-linux-${TARGETARCH}|manylinux_2_28|${LOCK_HASH}|${RUST_SCOPE}"
+CARGO_TARGET_CACHE_SCOPE=$(printf '%s' "$CACHE_SCOPE_INPUT" | sha256_16_stdin)
+
+rm -rf "$WHEEL_OUTPUT_DIR"
+mkdir -p "$WHEEL_OUTPUT_DIR"
+
+ce build \
+  -f deploy/docker/Dockerfile.python-wheels-linux \
+  --target wheels \
+  --build-arg "TARGETARCH=${TARGETARCH}" \
+  --build-arg "CARGO_TARGET_CACHE_SCOPE=${CARGO_TARGET_CACHE_SCOPE}" \
+  ${CARGO_VERSION:+--build-arg "OPENSHELL_CARGO_VERSION=${CARGO_VERSION}"} \
+  ${OPENSHELL_IMAGE_TAG:+--build-arg "OPENSHELL_IMAGE_TAG=${OPENSHELL_IMAGE_TAG}"} \
+  --output "type=local,dest=${WHEEL_OUTPUT_DIR}" \
+  .
+
+ls -la "$WHEEL_OUTPUT_DIR"/*.whl
+"""
+hide = true
+
+["build:python:wheel:linux:amd64:docker"]
+description = "Build portable manylinux_2_28 wheel for Linux amd64"
+depends = ["TARGETARCH=amd64 WHEEL_OUTPUT_DIR=target/wheels/linux-amd64 build:python:wheel:linux:docker"]
+hide = true
+
+["build:python:wheel:linux:arm64:docker"]
+description = "Build portable manylinux_2_28 wheel for Linux arm64"
+depends = ["TARGETARCH=arm64 WHEEL_OUTPUT_DIR=target/wheels/linux-arm64 build:python:wheel:linux:docker"]
+hide = true
+
+# Release-pipeline aliases. These produce manylinux_2_28-tagged wheels via
+# Docker so the published artifacts install on any glibc >= 2.28 host (RHEL 8,
+# Ubuntu 18.04+, Debian 10+). The native `build:python:wheel:linux:amd64` /
+# `build:python:wheel:linux:arm64` tasks remain available for fast local
+# iteration and produce wheels tagged for the host glibc only.
+["python:build:linux:amd64"]
+description = "Build portable manylinux_2_28 wheel for Linux amd64 (release path)"
+depends = ["build:python:wheel:linux:amd64:docker"]
+hide = true
+
 ["python:build:linux:arm64"]
-description = "Alias for build:python:wheel:linux:arm64"
-depends = ["build:python:wheel:linux:arm64"]
+description = "Build portable manylinux_2_28 wheel for Linux arm64 (release path)"
+depends = ["build:python:wheel:linux:arm64:docker"]
 hide = true
 
 ["build:python:wheel:macos"]