Skip to content
ci

fix(server): fail-fast unschedulable KV waits #2147

Sign in to view logs

fix(server): fail-fast unschedulable KV waits

fix(server): fail-fast unschedulable KV waits #2147

Workflow file for this run

.github/workflows/ci.yml at 0c43f73
name: ci
on:
push:
branches: [main]
pull_request:
# Workflow-level: dedupe PR pushes only. The group is keyed on the SHA for a
# push, so two pushes to main never share a group and never cancel each other.
# That is deliberate: `documentation-checkpoint` and `commit-protocol-tag` below
# are DIFF-scoped over `github.event.before..github.sha`, so a cancelled main run
# leaves its own commit range permanently unvalidated (no later run re-covers it,
# because the next run's `before` is this run's `sha`).
#
# The expensive TREE-scoped jobs carry their own job-level group instead, which
# does collapse superseded main pushes. Net effect: a superseded push to main
# keeps only its 2 per-commit gates and drops the other 6 jobs.
#
# 2026-08-04: those two jobs DID carry `ci-doc-`/`ci-commit-` groups keyed on
# `github.ref` with `cancel-in-progress: true`, directly contradicting their own
# comments and this one. It was not theoretical: two consecutive pushes to main
# had their documentation-checkpoint and commit-protocol-tag cancelled by the
# next push, so commits cdec2d10 and 0435746d's ranges went unvalidated by the
# gates written to cover them. The groups are removed; these two jobs must never
# regain one. They are a checkout plus a Python script, so the cost of always
# running them is negligible against a silently skipped gate.
concurrency:
group: ci-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
agent-record:
# Tree-scoped: validates the record as it stands at HEAD, so only the newest
# push to a ref is meaningful. Cancellable.
concurrency:
group: ci-agent-record-${{ github.ref }}-${{ github.repository }}
cancel-in-progress: true
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Canonical roadmap tables and links are consistent
run: |
python3 scripts/check-agent-record.py
python3 tests/scripts/test_agent_record.py
python3 tests/scripts/test_doc_checkpoint.py
- name: Accepted binary-release design and record anchors stay in sync
run: |
python3 scripts/check-release-binary-contract.py
python3 tests/scripts/test_check_release_binary_contract.py
- name: Binary release manifests are deterministic and fail closed
run: |
python3 tests/scripts/test_release_manifest.py
- name: Extracted release archives fail closed on content and metadata drift
run: |
python3 tests/scripts/test_release_archive.py
- name: Release workflow is dry-run safe and least privilege
run: |
python3 scripts/check-release-workflow.py
python3 tests/scripts/test_release_pipeline.py
python3 tests/scripts/test_release_index.py
- name: CPU release metadata and tier execution gates fail closed
run: |
python3 tests/scripts/test_release_metadata.py
python3 tests/scripts/test_release_accelerator_metadata.py
python3 tests/scripts/test_release_macos_metadata.py
python3 tests/scripts/test_cpu_release_gates.py
- name: README stays a human-readable user-facing document
run: |
python3 scripts/check-readme-structure.py
python3 tests/scripts/test_check_readme_structure.py
- name: Benchmarks and features stay human-readable keyed tables
run: |
python3 scripts/check-public-doc-tables.py
python3 tests/scripts/test_check_public_doc_tables.py
- name: Architecture-support checklist matches the row states
run: |
python3 scripts/check-model-checklist.py
python3 tests/scripts/test_check_model_checklist.py
- name: Public supported-model list matches the C++ registry
run: |
python3 scripts/check-supported-models.py
python3 tests/scripts/test_check_supported_models.py
- name: Every production env var is documented or classified
run: |
python3 scripts/check-env-doc.py
python3 tests/scripts/test_check_env_doc.py
- name: Model add+RMSNorm glue routes through the fusion catalog
run: |
python3 scripts/check-fusion-consistency.py
python3 tests/scripts/test_check_fusion_consistency.py
- name: Model decode is born on the runner (device-resident logits)
run: |
python3 scripts/check-runner-routing-consistency.py
python3 tests/scripts/test_check_runner_routing_consistency.py
- name: Examples are thin clients of the public ABI, capabilities reach it
run: |
python3 scripts/check-surface-coverage.py
python3 tests/scripts/test_check_surface_coverage.py
- name: Critical regression tests remain executable and CTest-registered
run: |
python3 scripts/check-test-registration.py
python3 tests/scripts/test_check_test_registration.py
- name: cuBLASLt GEMM invocation stays dtype-faithful (template parity)
run: |
python3 scripts/check-gemv-invocation-consistency.py
python3 tests/scripts/test_check_gemv_invocation_consistency.py
- name: Protocol prose matches the checkers that enforce it
env:
EVENT_NAME: ${{ github.event_name }}
PR_BASE: ${{ github.event.pull_request.base.sha }}
PR_HEAD: ${{ github.event.pull_request.head.sha }}
PUSH_BASE: ${{ github.event.before }}
PUSH_HEAD: ${{ github.sha }}
run: |
set -eu
python3 scripts/check-prompt-contract.py
python3 -m unittest \
tests.scripts.test_waivers \
tests.scripts.test_check_commit_trailers
python3 tests/scripts/test_check_prompt_contract.py
python3 tests/scripts/test_agent_gates.py
if [ "$EVENT_NAME" = "pull_request" ]; then
base="$PR_BASE"
head="$PR_HEAD"
else
base="$PUSH_BASE"
head="$PUSH_HEAD"
fi
python3 scripts/check-commit-trailers.py --range "$base..$head"
- name: NOW.md stays a short one-Read resume surface
run: |
python3 scripts/check-now-current.py
python3 tests/scripts/test_check_now_current.py
- name: A gated row may never lose a gate command that can FAIL
# An EXACT PIN over the SET of rows whose spec names a runnable command
# -- not a count, and not shrink-only. Not a count, because a count
# cannot tell a row that LOST its command from one that legitimately
# left the gated population, and the fix for the second reads as
# lowering the number for the first. Not shrink-only, because the unit
# test asserts the set EQUALS RUNNABLE_BASELINE, so GROWTH reds this job
# too: any movement, up or down, re-pins the baseline in the same
# change, naming the rows and the reason.
run: |
python3 scripts/check-gate-commands.py --check
python3 tests/scripts/test_check_gate_commands.py
- name: Agent role machinery and role discipline
env:
EVENT_NAME: ${{ github.event_name }}
PR_BASE: ${{ github.event.pull_request.base.sha }}
PR_HEAD: ${{ github.event.pull_request.head.sha }}
PUSH_BASE: ${{ github.event.before }}
PUSH_HEAD: ${{ github.sha }}
run: |
set -eu
pending_args=()
if [ "$EVENT_NAME" = "pull_request" ]; then
base="$PR_BASE"
head="$PR_HEAD"
pending_args=(--pending-pr-head "$PR_HEAD")
else
base="$PUSH_BASE"
head="$PUSH_HEAD"
fi
python3 scripts/check-role-discipline.py \
--base "$base" --head "$head" "${pending_args[@]}"
python3 tests/scripts/test_agent_role.py
python3 tests/scripts/test_agent_onboard.py
- name: Claim view, helper queue and PR reviewability
run: |
python3 scripts/claim-view.py --check
python3 scripts/ready-for-helper.py --check
python3 scripts/upstream-inventory.py --check
python3 tests/scripts/test_claim_view.py
python3 tests/scripts/test_upstream_inventory.py
- name: Live-state rows are reconciled against Git reality
# The classifier reads its evidence from `origin/main`. This job's
# checkout already uses `fetch-depth: 0`, which DOES leave
# `refs/remotes/origin/*` behind, so the ref is present today -- the
# fetch makes that independent of the checkout block above rather than
# a standing assumption about it, and refreshes the ref besides.
# Without a resolvable `origin/main` the audit aborts by design
# (require_origin_main): "no origin/main" would otherwise read as
# "every ACTIVE row is abandoned", absence of information wearing
# absence of work's face.
#
# The refspec is EXPLICIT and forced, not a bare `git fetch origin
# main`. A bare fetch only updates a remote-tracking ref when a
# CONFIGURED `remote.origin.fetch` refspec matches, and checkout sets
# that config from its own inputs (at shallower settings it narrows to
# the triggering ref, on a PR `refs/pull/N/merge`); with no match it
# writes FETCH_HEAD only. Naming the refspec creates the ref whatever
# the checkout is configured to do.
run: |
git fetch -q origin +refs/heads/main:refs/remotes/origin/main
python3 scripts/audit-live-rows.py --check
python3 tests/scripts/test_audit_live_rows.py
documentation-checkpoint:
# Gate: every code/test/benchmark/spike/lifecycle iteration refreshes both
# user-facing status surfaces in that same commit, including void attempts.
#
# DIFF-scoped: deliberately carries NO concurrency group. Its verdict covers
# this push's own `before..sha` range and nothing re-covers that range later,
# so cancelling it would silently exempt those commits from the gate.
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Every feature checkpoint updates STATUS, BENCHMARKS and NOW
env:
EVENT_NAME: ${{ github.event_name }}
PR_BASE: ${{ github.event.pull_request.base.sha }}
PR_HEAD: ${{ github.event.pull_request.head.sha }}
PUSH_BASE: ${{ github.event.before }}
PUSH_HEAD: ${{ github.sha }}
run: |
set -eu
if [ "$EVENT_NAME" = "pull_request" ]; then
base="$PR_BASE"
head="$PR_HEAD"
else
base="$PUSH_BASE"
head="$PUSH_HEAD"
fi
python3 scripts/check-doc-checkpoint.py --base "$base" --head "$head"
python3 scripts/check-now-current.py --base "$base" --head "$head"
pending_args=()
if [ "$EVENT_NAME" = "pull_request" ]; then
pending_args=(--pending-pr-head "$PR_HEAD")
fi
python3 scripts/check-role-discipline.py \
--base "$base" --head "$head" "${pending_args[@]}"
commit-protocol-tag:
# Gate: every NEW commit must carry the FOLLOWING_AGENTS_PROTOCOL trailer,
# asserting the contributor read AGENTS.md. See .agents/ai-coding-assistants.md.
#
# DIFF-scoped: deliberately carries NO concurrency group, same reasoning as
# documentation-checkpoint above. It walks `before..sha` per push, so a
# cancelled run means those commits are never checked for the trailer.
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Every new commit carries FOLLOWING_AGENTS_PROTOCOL
run: |
set -eu
if [ "${{ github.event_name }}" = "pull_request" ]; then
base="${{ github.event.pull_request.base.sha }}"
head="${{ github.event.pull_request.head.sha }}"
else
base="${{ github.event.before }}"
head="${{ github.sha }}"
fi
# Enforce on the FIRST-PARENT mainline (the commits the submitter
# directly lands on the target branch, incl. merge commits which are
# skipped below). Merged-in feature-branch commits are owned by the
# submitter via the merge and are not re-checked here.
# New branch / unreachable base (all-zero SHA): only check the tip commit.
if ! git cat-file -e "${base}^{commit}" 2>/dev/null; then
commits="$head"
else
commits="$(git rev-list --first-parent "${base}..${head}")"
fi
fail=0
for c in $commits; do
# Skip merge commits (>1 parent) — they are not authored content.
if [ "$(git rev-list --parents -n1 "$c" | wc -w)" -gt 2 ]; then continue; fi
if ! git log -1 --format=%B "$c" | grep -q 'FOLLOWING_AGENTS_PROTOCOL'; then
echo "::error::commit $c is missing the FOLLOWING_AGENTS_PROTOCOL trailer — read AGENTS.md"
git log -1 --oneline "$c"
fail=1
fi
done
if [ "$fail" -ne 0 ]; then
echo "One or more commits lack FOLLOWING_AGENTS_PROTOCOL. See AGENTS.md / .agents/ai-coding-assistants.md."
exit 1
fi
echo "OK: all new commits carry FOLLOWING_AGENTS_PROTOCOL."
pr-size:
# PR-only: helper PRs stay cheap to review, which is what stops the serial
# operator from becoming the bottleneck. Non-row branches are reported only.
if: github.event_name == 'pull_request'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Helper PRs stay reviewable
run: |
python3 scripts/check-pr-size.py \
--base "${{ github.event.pull_request.base.sha }}" \
--head "${{ github.event.pull_request.head.sha }}" \
--branch "${{ github.event.pull_request.head.ref }}" \
--pr-number "${{ github.event.pull_request.number }}"
cuda-arch-features:
# Configure-tier assertions on the CUDA per-arch FEATURE TABLE. Needs neither
# a CUDA toolkit nor a GPU: it drives cmake/CudaArchFeatures.cmake directly,
# so the resolution that decides which architectures get fp4-mma /
# cutlass-nvfp4 / cutlass-fp8 / marlin-nvfp4 / fa2 (whose historical failure
# mode was a SILENT capability drop) is checked on every push.
#
# Tree-scoped (drives cmake at HEAD): cancellable.
concurrency:
group: ci-cuda-arch-features-${{ github.ref }}-${{ github.repository }}
cancel-in-progress: true
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: CUDA feature table and per-source gencode resolve exactly
run: |
cmake -P cmake/CudaArchFeaturesTest.cmake
cmake -P cmake/CudaSourceGencodeTest.cmake
cmake -P cmake/TritonAOTMultiArchTest.cmake
cmake -P cmake/TritonAOTDefaultTest.cmake
# Same tier, no CUDA involved: the in-source build guard (issue #85) is a
# configure-time predicate, so it is asserted the same way — by driving the
# module directly, with no compiler and no writes to the tree.
- name: In-source build guard fires on `cmake .` and only on `cmake .`
run: |
cmake -P cmake/InSourceGuardTest.cmake
python3 -m unittest tests.scripts.test_check_cuda_fat_gencode
python3 -m unittest tests.scripts.test_check_triton_aot_multiarch
cuda-fat-build:
# W1 release prerequisite: compile the primary heterogeneous CUDA archive,
# then inspect both compile_commands.json and the linked archive. No GPU is
# used; runtime/correctness evidence remains independent.
permissions:
contents: read
concurrency:
group: ci-cuda-fat-build-${{ github.ref }}-${{ github.repository }}
cancel-in-progress: true
runs-on: ubuntu-latest
timeout-minutes: 180
container: nvidia/cuda:13.3.0-devel-ubuntu24.04
steps:
- name: Install build tools
run: |
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
binutils ca-certificates cmake g++ git ninja-build python3
rm -rf /var/lib/apt/lists/*
- uses: actions/checkout@v4
- name: Configure ten-SM static library
run: |
cmake -S . -B build-cuda-fat -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
-DVLLM_CPP_CUDA=ON \
-DVLLM_CPP_CUDA_ARCHITECTURES='80;86;87;89;90a;100a;103a;110;120a;121a' \
-DVLLM_CPP_CUTLASS_FETCH=ON \
-DVLLM_CPP_TRITON=ON \
-DVLLM_CPP_BUILD_TESTS=OFF \
-DVLLM_CPP_BUILD_EXAMPLES=OFF \
-DVLLM_CPP_SERVER=OFF
- name: Build and audit exact per-source gencode
run: |
cmake --build build-cuda-fat --target vllm --parallel 2
python3 scripts/check-cuda-fat-gencode.py \
--compile-commands build-cuda-fat/compile_commands.json \
--library build-cuda-fat/libvllm.a
python3 scripts/check-triton-aot-multiarch.py \
--vendored-root src/vt/cuda/triton_aot_vendored \
--library build-cuda-fat/libvllm.a
# BUILD-TRITON-DEFAULT-ON (#219), spec Tests item 3. The configure above
# passes -DVLLM_CPP_TRITON=ON explicitly, so it cannot witness the DEFAULT.
# This one passes no such flag: the option must resolve ON by itself AND
# reach every CUDA translation unit as VLLM_CPP_TRITON=1 /
# VLLM_CPP_TRITON_CHUNKO_BF16=1, which is the row's actual claim. Configure
# only, and it reuses the CUTLASS the job already fetched (FETCH stays ON
# as the fallback if that path ever moves).
- name: The computed Triton default reaches the CUDA translation units
run: |
cmake -S . -B build-cuda-default -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
-DVLLM_CPP_CUDA=ON \
-DVLLM_CPP_CUDA_ARCHITECTURES='80;86;87;89;90a;100a;103a;110;120a;121a' \
-DVLLM_CPP_CUTLASS_DIR="$PWD/build-cuda-fat/_deps/cutlass-src" \
-DVLLM_CPP_CUTLASS_FETCH=ON \
-DVLLM_CPP_BUILD_TESTS=OFF \
-DVLLM_CPP_BUILD_EXAMPLES=OFF \
-DVLLM_CPP_SERVER=OFF
VLLM_CPP_DEFAULT_BUILD_DIR=build-cuda-default \
python3 -m unittest tests.scripts.test_triton_default_definitions
vulkan-spirv-freshness:
# The Vulkan backend commits its SPIR-V (src/vt/vulkan/vulkan_spirv.h) instead
# of compiling shaders at build time, so the build needs no shader toolchain on
# any machine -- strictly better than llama.cpp, which requires glslc on every
# build box. The trade is an obligation to regenerate by hand, and NOTHING
# ENFORCED IT: `gen-vulkan-spirv.py --check` existed but no job ran it, so a
# .comp edit without a regenerate shipped silently. That is the failure mode
# this job closes, before the shader surface grows (BACKEND-VULKAN, VK-A1).
#
# The glslang DOWNLOAD URL is the pin. An exact version-string assertion was
# deliberately NOT used: the committed SPIR-V was measured byte-identical under
# both the 16.4.0 that produced it and the 16.5.0 pinned here, so gating on the
# string would be brittle without buying anything. The URL being fixed is what
# makes the run deterministic.
#
# Tree-scoped (checks HEAD's artifact): cancellable.
concurrency:
group: ci-vulkan-spirv-freshness-${{ github.ref }}-${{ github.repository }}
cancel-in-progress: true
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install pinned glslang
run: |
set -euo pipefail
curl -fsSL -o /tmp/glslang.tar.gz \
https://github.com/KhronosGroup/glslang/releases/download/16.5.0/glslang-16.5.0-linux-x86_64-release.tar.gz
mkdir -p /tmp/glslang
tar xzf /tmp/glslang.tar.gz -C /tmp/glslang
/tmp/glslang/bin/glslang --version | head -1
echo "/tmp/glslang/bin" >> "$GITHUB_PATH"
- name: Committed SPIR-V is not stale
run: python3 scripts/gen-vulkan-spirv.py --check
build-test-vulkan:
# The Vulkan backend had NO CI leg at all: `VLLM_CPP_VULKAN=ON` appeared
# nowhere in this workflow, so tests/vt/test_vulkan_backend.cpp ran on no
# machine. It rotted exactly as you would expect. Accelerator-seam row S5
# (af0b21ba) gave unified-memory devices a portable CPU reference tier, which
# means a missed GetOp no longer throws; the Metal sibling test was updated for
# that as Metal work continued, and the Vulkan one was not, so its "the
# unimplemented ops throw" assertion sat RED and INVISIBLE from S5 until
# VK-A1 built the backend by hand (2026-08-06). That is the gap this closes.
#
# Runs GPU-FREE on `mesa-vulkan-drivers`' llvmpipe software ICD, which is the
# arrangement the fan-out spike already recorded as working. The suite is
# scoped to the Vulkan-specific targets rather than all of ctest: the full
# suite is already covered by build-test-cpu, and this lane exists to keep the
# Vulkan-only TUs compiled and executed, not to duplicate it.
#
# Tree-scoped: cancellable.
concurrency:
group: ci-build-test-vulkan-${{ github.ref }}-${{ github.repository }}
cancel-in-progress: true
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install the software Vulkan ICD
# Loader + llvmpipe only. No shader toolchain: the backend consumes the
# COMMITTED SPIR-V, which is the whole point of the committed-artifact
# route, and the vulkan-spirv-freshness job above is what keeps it honest.
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get install -y --no-install-recommends libvulkan1 mesa-vulkan-drivers
- name: Configure
run: cmake -S . -B build-vulkan -DVLLM_CPP_BUILD_TESTS=ON -DVLLM_CPP_VULKAN=ON
- name: Build
# Bounded parallelism for the same reason as build-test-cpu: an unbounded
# parallel link OOM-kills the runner.
run: cmake --build build-vulkan -j 2 --target test_vulkan_backend test_backend_cross_device
- name: Vulkan backend gate
run: ./build-vulkan/tests/test_vulkan_backend
- name: Cross-device numerics vs the CPU oracle
run: ./build-vulkan/tests/test_backend_cross_device
device-leakage:
# The DSR RATCHET (work row `S1` of .agents/specs/accelerator-seam-audit.md).
# Counts device-specific references in `src/vllm/` + `include/vllm/` — the
# layer that is supposed to be device-agnostic — and fails on ANY increase
# over scripts/device-leakage-baseline.json. A reduction must lower the
# baseline in the SAME commit, so the number can only ever move down.
#
# It exists because the audit re-measured the leakage and found it had DRIFTED
# UPWARD with no bad commit: DeepSeek-V2, Qwen3-Coder and the attention-registry
# work each added a device test in passing. Leakage grows silently under
# well-executed work, which is a job for a ratchet, not a cleanup.
#
# Needs neither a CUDA toolkit nor a GPU — pure static analysis, like the
# cuda-arch-features job above. The mutation suite proves the checker actually
# catches a planted leak; an unpoliced checker is worse than none.
#
# Tree-scoped: the ratchet compares HEAD against the committed baseline, so
# only the newest push matters. Cancellable.
concurrency:
group: ci-device-leakage-${{ github.ref }}-${{ github.repository }}
cancel-in-progress: true
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Shared-layer device leakage does not grow (DSR ratchet)
run: |
python3 scripts/check-device-leakage.py --report
python3 tests/scripts/test_device_leakage.py
build-test-cpu:
# Tree-scoped and the most expensive lane in the workflow. Cancellable.
concurrency:
group: ci-build-test-cpu-${{ github.ref }}-${{ github.repository }}
cancel-in-progress: true
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Configure
run: |
cmake -S . -B build \
-DVLLM_CPP_BUILD_TESTS=ON \
-DCMAKE_EXPORT_COMPILE_COMMANDS=ON
- name: Build
# Bounded parallelism: a bare `-j` lets Make link ALL test executables at
# once, which OOM-kills the runner (ld signal 9) during the parallel link.
run: cmake --build build -j 2
- name: Installed server archive is reproducible and runnable
env:
VLLM_CPP_PACKAGE_TEST_BUILD_DIR: ${{ github.workspace }}/build
run: python3 tests/scripts/test_server_package.py
- name: CPU ISA objects use only their declared feature flags
run: |
python3 scripts/check-cpu-isa-build.py \
--compile-commands build/compile_commands.json
- name: Test
run: ctest --test-dir build --output-on-failure
- name: Force every x86 ISA tier available on this runner
run: |
tiers=(portable)
if [[ "$(uname -m)" == "x86_64" ]]; then
tiers+=(sse2)
if grep -qiw f16c /proc/cpuinfo && grep -qiw avx /proc/cpuinfo; then
tiers+=(sse2+f16c)
fi
if grep -qiw avx2 /proc/cpuinfo; then
tiers+=(avx2)
fi
if grep -qiw avx512f /proc/cpuinfo && \
grep -qiw avx512bw /proc/cpuinfo && \
grep -qiw avx512vl /proc/cpuinfo; then
tiers+=(avx512)
fi
fi
for tier in "${tiers[@]}"; do
echo "Running test_ops_matmul_elem with VT_CPU_MATMUL_TIER=${tier}"
VT_CPU_MATMUL_TIER="${tier}" build/tests/test_ops_matmul_elem
done
build-test-cpu-arm64:
# Independent Arm execution evidence: the x86 lane cannot prove HWCAP
# dispatch, Arm instructions, or the host ABI. The native runner exercises
# every available rich tier; qemu's Cortex-A53 model supplies the required
# feature-poor execution and illegal-tier refusal gate.
concurrency:
group: ci-build-test-cpu-arm64-${{ github.ref }}-${{ github.repository }}
cancel-in-progress: true
runs-on: ubuntu-24.04-arm
steps:
- uses: actions/checkout@v4
- name: Configure focused Arm CPU gate
run: |
cmake -S . -B build-arm \
-DVLLM_CPP_BUILD_TESTS=ON \
-DVLLM_CPP_BUILD_EXAMPLES=OFF \
-DVLLM_CPP_SERVER=OFF \
-DVLLM_CPP_CUDA=OFF \
-DCMAKE_EXPORT_COMPILE_COMMANDS=ON
- name: Build Arm ISA and kernel gates
run: |
cmake --build build-arm \
--target test_cpu_isa_arm test_ops_matmul_elem \
test_ops_quant_dot test_ops_quant_repack \
-j 2
- name: Arm ISA objects use only their declared feature flags
run: |
python3 scripts/check-arm-isa-build.py \
--compile-commands build-arm/compile_commands.json
- name: Force baseline and every rich tier available on the native runner
run: |
build-arm/tests/test_cpu_isa_arm
VT_CPU_MATMUL_TIER=portable build-arm/tests/test_ops_matmul_elem
VT_CPU_MATMUL_TIER=neon build-arm/tests/test_ops_matmul_elem
VT_CPU_Q8_DOT=portable \
VT_CPU_QUANT_MMLA=portable \
build-arm/tests/test_ops_quant_dot
VT_CPU_QUANT_REPACK=portable build-arm/tests/test_ops_quant_repack
if grep -qiw asimddp /proc/cpuinfo; then
VT_CPU_Q8_DOT=sdot build-arm/tests/test_ops_quant_dot
fi
if grep -qiw asimddp /proc/cpuinfo && grep -qiw i8mm /proc/cpuinfo; then
VT_CPU_QUANT_MMLA=i8mm build-arm/tests/test_ops_quant_dot
VT_CPU_QUANT_REPACK=i8mm build-arm/tests/test_ops_quant_repack
fi
- name: Feature-poor Cortex-A53 executes baseline and refuses rich tiers
run: |
sudo apt-get update
sudo apt-get install --yes qemu-user
qemu-aarch64 -cpu cortex-a53 build-arm/tests/test_cpu_isa_arm
VT_CPU_MATMUL_TIER=portable \
qemu-aarch64 -cpu cortex-a53 build-arm/tests/test_ops_matmul_elem
VT_CPU_MATMUL_TIER=neon \
qemu-aarch64 -cpu cortex-a53 build-arm/tests/test_ops_matmul_elem
VT_CPU_Q8_DOT=portable VT_CPU_QUANT_MMLA=portable \
qemu-aarch64 -cpu cortex-a53 build-arm/tests/test_ops_quant_dot
if VT_CPU_Q8_DOT=sdot \
qemu-aarch64 -cpu cortex-a53 build-arm/tests/test_ops_quant_dot; then
echo "Cortex-A53 unexpectedly accepted the DotProd tier" >&2
exit 1
fi
if VT_CPU_QUANT_MMLA=i8mm \
qemu-aarch64 -cpu cortex-a53 build-arm/tests/test_ops_quant_dot; then
echo "Cortex-A53 unexpectedly accepted the i8mm tier" >&2
exit 1
fi
sanitize-cpu:
# The DYNAMIC detector lanes (VLLM_CPP_SANITIZE, see CMakeLists.txt). The
# plain build-test-cpu job above proves the suite PASSES; it cannot see a
# read one past a tensor row, a use-after-free of an engine-owned container,
# a signed-overflow index computation, or an unsynchronized access between
# the serving threads — every one of which stays green under -O2 until it
# silently corrupts a token stream. These jobs run the SAME suite with
# ASan+UBSan and with TSan.
#
# Two jobs, not one: the runtimes are mutually exclusive. Both are CPU-tier
# (the lane refuses to configure with the CUDA backend on) and both are
# `continue-on-error` for their FIRST landing so a pre-existing finding
# cannot block unrelated work — the finding is triaged, then this flag is
# removed and the lane becomes binding. Removing it is tracked as the
# closing step of the hardening-adoption row.
strategy:
fail-fast: false
matrix:
lane: ["address,undefined", "thread"]
continue-on-error: true
# Tree-scoped: cancellable. `matrix.lane` MUST be part of the group key, or
# the two mutually-exclusive sanitizer legs of the SAME run would land in one
# group and cancel each other, leaving only whichever started second.
concurrency:
group: ci-sanitize-cpu-${{ matrix.lane }}-${{ github.ref }}-${{ github.repository }}
cancel-in-progress: true
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Configure
run: |
cmake -S . -B build-sanitize \
-DVLLM_CPP_BUILD_TESTS=ON \
-DVLLM_CPP_CUDA=OFF \
-DVLLM_CPP_SANITIZE='${{ matrix.lane }}'
- name: Build
run: cmake --build build-sanitize -j 2
- name: Test
# ctest runs the suite serially: the sanitizer runtimes multiply peak RSS,
# and a parallel run OOM-kills the runner before it reports a finding.
env:
UBSAN_OPTIONS: print_stacktrace=1
ASAN_OPTIONS: detect_leaks=1:strict_string_checks=1
# The production DevicePool deliberately retains scratch blocks. Its
# detector lane uses exact allocations and real frees so ASan can
# distinguish that cache from a leak and can see use-after-free.
VT_POOL_BYPASS: "1"
run: ctest --test-dir build-sanitize --output-on-failure