gate(ROCM): M4 -- pinned vLLM-ROCm oracle on gfx1100 + ROCm device-golden lane in the Qwen3-0.6B near-tie gate #3631
Workflow file for this run
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: ci | |
| on: | |
| push: | |
| branches: [main] | |
| # `closed` so a merged or abandoned pull request ENTERS the concurrency | |
| # group and supersedes its own in-flight run (#822). Every job below is | |
| # skipped on it -- directly through its own `if:`, or transitively by | |
| # `needs:`-ing a job that is -- except the two `windows-msvc-*` PR proofs, | |
| # whose whole job mapping is pinned byte-for-byte by | |
| # `scripts/check-release-workflow.py` and can carry neither key. Those two | |
| # therefore still start on a close; recorded as owed in | |
| # .agents/specs/ci-concurrency.md and tracked by #874. | |
| pull_request: | |
| types: [opened, synchronize, reopened, closed] | |
| # THE BASELINE LANE (issue #274, spec .agents/specs/main-verifiability.md). | |
| # | |
| # The push lane below cannot answer "is main green?", by construction. Every | |
| # expensive job carries a job-level group keyed on `github.ref`, which is the | |
| # constant `refs/heads/main` for every push, so consecutive pushes cancel each | |
| # other's long jobs. Measured over 40 consecutive main runs at 0eb049f7: 26 | |
| # cancelled, 12 failure, 1 success. The cancel instant equals the next push's | |
| # start instant (run 31485402200 died 11:46:33, run 31488132224 started | |
| # 11:46:32; 31482845117 died 11:05:12, 31485054749 started 11:05:11). At 55 | |
| # pushes/day and a 99-minute suite, that is structural, not bad luck. | |
| # | |
| # This lane runs the SAME jobs on a cadence, in its own concurrency groups so | |
| # a push cannot cancel it, and publishes a verdict via `baseline-summary`. | |
| # Every 4h: at 55 pushes/day a red baseline then names ~9 commits (three | |
| # bisect steps), and successive runs still cannot overlap a 99-minute suite. | |
| # Making every MERGE non-cancellable was rejected: ~55 overlapping runs/day, | |
| # ~190 h/day of job time on a pool where run 31485402200 already sat queued 37 | |
| # minutes without starting a single job -- and at that rate the newest | |
| # complete baseline would be several commits stale anyway. | |
| schedule: | |
| - cron: '17 */4 * * *' | |
| # The hybrid half: pin a baseline on a SHA you care about right after merging | |
| # it, without paying for a full run on all 55 of the day's pushes. | |
| # gh workflow run ci.yml --ref main | |
| workflow_dispatch: | |
| # Workflow-level: dedupe PR pushes only. The group is keyed on the SHA for a | |
| # push, so two pushes to main never share a group and never cancel each other. | |
| # That is deliberate: `documentation-checkpoint` and `commit-protocol-tag` below | |
| # are DIFF-scoped over `github.event.before..github.sha`, so a cancelled main run | |
| # leaves its own commit range permanently unvalidated (no later run re-covers it, | |
| # because the next run's `before` is this run's `sha`). | |
| # | |
| # The expensive TREE-scoped jobs carry their own job-level group instead, which | |
| # does collapse superseded main pushes. Net effect: a superseded push to main | |
| # keeps only its 2 per-commit gates and drops the other 6 jobs. | |
| # | |
| # 2026-08-04: those two jobs DID carry `ci-doc-`/`ci-commit-` groups keyed on | |
| # `github.ref` with `cancel-in-progress: true`, directly contradicting their own | |
| # comments and this one. It was not theoretical: two consecutive pushes to main | |
| # had their documentation-checkpoint and commit-protocol-tag cancelled by the | |
| # next push, so commits cdec2d10 and 0435746d's ranges went unvalidated by the | |
| # gates written to cover them. The groups are removed; these two jobs must never | |
| # regain one. They are a checkout plus a Python script, so the cost of always | |
| # running them is negligible against a silently skipped gate. | |
| # | |
| # 2026-08-11 (#274): `github.event_name` joins the key. `cancel-in-progress` is | |
| # already false for every non-PR event, so this cancels nothing new -- it stops | |
| # a `workflow_dispatch` baseline fired on a SHA that was just pushed from | |
| # sharing group `ci-<sha>-<repo>` with that push's run and QUEUEING behind it. | |
| # For `push` and `pull_request` the added token is a constant, so their | |
| # partition into groups is exactly what it was. | |
| # 2026-08-15 (#822): the push lane becomes LATEST-ONLY. The group keys on | |
| # `github.ref` for a push rather than `github.sha`, so consecutive pushes to main | |
| # share a group and the older one is cancelled. | |
| # | |
| # This was previously unsafe and is now safe. It was unsafe because a cancelled | |
| # run's diff-scoped range was never re-covered: the next run's `before` is this | |
| # run's `sha`. The gates now walk from the last SUCCESSFULLY gated commit | |
| # (`last-gated-commit`), so a cancelled run is lossless -- the next run walks a | |
| # wider range and reports the same red. Reverting that base to `before` without | |
| # reverting this reintroduces #863. | |
| # | |
| # `schedule` and `workflow_dispatch` keep their own partition through the | |
| # event_name token and stay non-cancellable, so the baseline lane (#274) is | |
| # untouched. | |
| concurrency: | |
| group: ci-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref }}-${{ github.repository }} | |
| cancel-in-progress: ${{ github.event_name == 'pull_request' || github.event_name == 'push' }} | |
| jobs: | |
| agent-record: | |
| # Tree-scoped: validates the record as it stands at HEAD, so only the newest | |
| # push to a ref is meaningful. Cancellable. | |
| concurrency: | |
| group: ci-agent-record-${{ github.event_name }}-${{ github.ref }}-${{ github.repository }} | |
| cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }} | |
| # NO job-level `if:`, deliberately, and no `always()`. | |
| # | |
| # `check-release-binary-contract.py` and `check-test-registration.py` count | |
| # a checker as wired into CI only when the job owning its step carries no | |
| # `if:` at all: a registration behind a condition is not a registration. | |
| # #865 gave this job one, and both checkers plus three of their suites went | |
| # RED on `main` (#873). | |
| # | |
| # The closed-pull-request skip #822 needs is carried by `needs:` instead. | |
| # `last-gated-commit` excludes the closed action and runs on every other | |
| # lane, so a closed pull request skips it and skips this job with it, while | |
| # push, pull_request and the baseline lane all still run it. That is also | |
| # why `always()` must NOT come back here: it would defeat the skip. | |
| # | |
| # The asymmetry that buys is deliberate but real: without `always()` the | |
| # implicit `success()` couples this job to `last-gated-commit` SUCCEEDING, | |
| # not merely running, while `documentation-checkpoint` and | |
| # `commit-protocol-tag` on the same `needs:` keep `always()` and run | |
| # regardless. A failure in the sha resolver therefore silences this job's | |
| # checkers rather than reding them -- which is why that job does nothing but | |
| # resolve a string and falls back instead of failing. | |
| needs: [last-gated-commit] | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v4 | |
| with: | |
| fetch-depth: 0 | |
| - name: Canonical roadmap tables and links are consistent | |
| run: | | |
| python3 scripts/check-agent-record.py | |
| python3 tests/scripts/test_agent_record.py | |
| python3 tests/scripts/test_check_issue_index_append_only.py | |
| python3 tests/scripts/test_doc_checkpoint.py | |
| - name: Accepted binary-release design and record anchors stay in sync | |
| run: | | |
| python3 scripts/check-release-binary-contract.py | |
| python3 tests/scripts/test_check_release_binary_contract.py | |
| - name: Binary release manifests are deterministic and fail closed | |
| run: | | |
| python3 tests/scripts/test_release_manifest.py | |
| - name: Extracted release archives fail closed on content and metadata drift | |
| run: | | |
| python3 tests/scripts/test_release_archive.py | |
| - name: Release workflow is dry-run safe and least privilege | |
| run: | | |
| python3 scripts/check-release-workflow.py | |
| python3 scripts/check-windows-release-state.py | |
| python3 scripts/check-container-matrix.py | |
| python3 scripts/check-container-workflow.py | |
| python3 tests/scripts/test_release_pipeline.py | |
| python3 tests/scripts/test_check_windows_release_state.py | |
| python3 tests/scripts/test_release_postpublish_audit.py | |
| python3 tests/scripts/test_check_container_matrix.py | |
| python3 tests/scripts/test_check_container_workflow.py | |
| python3 tests/scripts/test_release_index.py | |
| - name: CPU release metadata and tier execution gates fail closed | |
| run: | | |
| python3 tests/scripts/test_release_metadata.py | |
| python3 tests/scripts/test_release_accelerator_metadata.py | |
| python3 tests/scripts/test_release_macos_metadata.py | |
| python3 tests/scripts/test_release_windows_metadata.py | |
| python3 tests/scripts/test_cpu_release_gates.py | |
| - name: README stays a human-readable user-facing document | |
| run: | | |
| python3 scripts/check-readme-structure.py | |
| python3 tests/scripts/test_check_readme_structure.py | |
| - name: Benchmarks and features stay human-readable keyed tables | |
| run: | | |
| python3 scripts/check-public-doc-tables.py | |
| python3 tests/scripts/test_check_public_doc_tables.py | |
| - name: Architecture-support checklist matches the row states | |
| run: | | |
| python3 scripts/check-model-checklist.py | |
| python3 tests/scripts/test_check_model_checklist.py | |
| - name: Public supported-model list matches the C++ registry | |
| run: | | |
| python3 scripts/check-supported-models.py | |
| python3 tests/scripts/test_check_supported_models.py | |
| - name: Every production env var is documented or classified | |
| run: | | |
| python3 scripts/check-env-doc.py | |
| python3 tests/scripts/test_check_env_doc.py | |
| - name: The docs site's content invariants hold | |
| run: | | |
| python3 scripts/check-site.py | |
| python3 tests/scripts/test_check_site.py | |
| - name: Model add+RMSNorm glue routes through the fusion catalog | |
| run: | | |
| python3 scripts/check-fusion-consistency.py | |
| python3 tests/scripts/test_check_fusion_consistency.py | |
| - name: Structural checkers ignore text the compiler never sees | |
| run: | | |
| python3 tests/scripts/test_checker_text.py | |
| - name: Every fp4 resident upload keeps its post-upload residency step | |
| run: | | |
| python3 scripts/check-fp4-resident-consistency.py | |
| python3 tests/scripts/test_check_fp4_resident_consistency.py | |
| - name: An arch-independent CUDA op is registered from an unconditional TU | |
| run: | | |
| python3 scripts/check-cuda-op-arch-gate.py --report | |
| python3 tests/scripts/test_check_cuda_op_arch_gate.py | |
| - name: Model decode is born on the runner (device-resident logits) | |
| run: | | |
| python3 scripts/check-runner-routing-consistency.py | |
| python3 tests/scripts/test_check_runner_routing_consistency.py | |
| - name: Examples are thin clients of the public ABI, capabilities reach it | |
| run: | | |
| python3 scripts/check-surface-coverage.py | |
| python3 tests/scripts/test_check_surface_coverage.py | |
| - name: The x86_64 CPU floor harness runs from a clean checkout | |
| # It shipped unable to run at all: `OUT=evi` with no `mkdir -p`, so every | |
| # redirection failed, every leg was discarded for a non-zero exit, and | |
| # the series died blaming contention it had never reached. Stub engines, | |
| # so this needs no model, no GPU and no quiet box. | |
| run: | | |
| python3 tests/scripts/test_cpu_x86_llamacpp_floor.py | |
| - name: The IndexTTS-2.5 port still matches the shipped checkpoint (#634) | |
| # These three read committed copies of the checkpoint's own config and | |
| # safetensors header, so they need no network, no weights and no GPU. | |
| # They landed in #727 and earlier WITHOUT a CI line, which made them | |
| # run-if-you-remember rather than gates; wiring them is the point. | |
| run: | | |
| python3 tests/scripts/test_indextts2_config_contract.py | |
| python3 tests/scripts/test_indextts2_emotion_arch_covered.py | |
| python3 tests/scripts/test_indextts2_pth_manifest.py | |
| python3 tests/scripts/test_indextts2_convert.py | |
| python3 tests/scripts/test_indextts2_reference_path.py | |
| python3 tests/scripts/test_vocoder1d_single_home.py | |
| - name: Critical regression tests remain executable and CTest-registered | |
| run: | | |
| python3 scripts/check-test-registration.py | |
| python3 tests/scripts/test_check_test_registration.py | |
| - name: No gate resolves its checkpoint by readdir order (#471) | |
| # Registered HERE and not only in `scripts/agent-preflight.sh`: a checker | |
| # reachable through a hook gates whoever chooses to run the hook, and | |
| # AGENTS.md is explicit that hooks are bypassable convenience, never | |
| # proof. The `--self-test` leg sweeps the checker's own fixture corpus in | |
| # both directions, so a narrowed pattern fails here rather than passing | |
| # quietly on a tree it no longer inspects. | |
| run: | | |
| python3 scripts/check-snapshot-pins.py | |
| python3 scripts/check-snapshot-pins.py --self-test | |
| python3 tests/scripts/test_check_snapshot_pins.py | |
| - name: Every oracle is named, pinned, and honest about gateability (#647) | |
| # The fallback rule (AGENTS.md "When vLLM has no implementation") admits | |
| # a secondary oracle only at a recorded pin. Ungated, the registry decays | |
| # the way the scattered pins it replaced already had: an upstream gets | |
| # compared against at whatever revision happened to be checked out, and | |
| # the pin that does exist lands in one spec where no other spec can see | |
| # it. `--self-test` sweeps the fixture corpus in both directions. | |
| run: | | |
| python3 scripts/check-oracle-pins.py | |
| python3 scripts/check-oracle-pins.py --self-test | |
| python3 tests/scripts/test_check_oracle_pins.py | |
| - name: cuBLASLt GEMM invocation stays dtype-faithful (template parity) | |
| run: | | |
| python3 scripts/check-gemv-invocation-consistency.py | |
| python3 tests/scripts/test_check_gemv_invocation_consistency.py | |
| - name: The main baseline lane and its reader stay honest | |
| # Registered in TWO places, deliberately: here and in the `SUITES` array | |
| # of `scripts/agent-preflight.sh`. It shipped in neither, so the 24 tests | |
| # that guard the baseline lane ran on no machine -- the same class of | |
| # defect as an unregistered CTest target, one layer up. #408 tracks | |
| # whether a checker should catch the class; this closes the instance. | |
| run: | | |
| python3 tests/scripts/test_main_baseline.py | |
| - name: Preflight never reports green over a block that did not run | |
| # Registered in TWO places, like the baseline suite above: here and in | |
| # the `SUITES` array of `scripts/agent-preflight.sh`. This lane is the | |
| # one that matters more of the two, because the defect it pins (#998) | |
| # is preflight lying about ITSELF, and a suite that only ever runs from | |
| # the script it audits is checked by exactly the run it cannot trust. | |
| run: | | |
| python3 tests/scripts/test_agent_preflight_skip_report.py | |
| - name: Protocol prose matches the checkers that enforce it | |
| # PART tree-scoped, PART diff-scoped, and the two must be told apart on | |
| # the baseline lane (#274). `documentation-checkpoint` and | |
| # `commit-protocol-tag` opt OUT of `schedule`/`workflow_dispatch` | |
| # wholesale because they are diff-scoped end to end; this job cannot, | |
| # because most of it is tree-scoped and the baseline needs it. So the | |
| # RANGE-SCOPED call is guarded here instead of the whole job. | |
| # | |
| # Without that guard this step DID kill the lane: a `schedule` payload | |
| # has no `github.event.before`, `PUSH_BASE` renders empty, and under | |
| # `set -eu` the walk aborts with | |
| # `--range: range must be exactly BASE..HEAD` (exit 2) | |
| # so `agent-record` could never be green and `baseline-summary`, which | |
| # `needs:` it, could never publish GREEN. Reproduced by replaying this | |
| # body with EVENT_NAME=schedule, PUSH_BASE="". | |
| env: | |
| EVENT_NAME: ${{ github.event_name }} | |
| run: | | |
| set -eu | |
| python3 scripts/check-prompt-contract.py | |
| python3 -m unittest \ | |
| tests.scripts.test_check_commit_trailers | |
| python3 tests/scripts/test_check_prompt_contract.py | |
| python3 tests/scripts/test_agent_gates.py | |
| - name: NOW.md stays a short one-Read resume surface | |
| run: | | |
| python3 scripts/check-now-current.py | |
| python3 tests/scripts/test_check_now_current.py | |
| - name: A gated row may never lose a gate command that can FAIL | |
| # An EXACT PIN over the SET of rows whose spec names a runnable command | |
| # -- not a count, and not shrink-only. Not a count, because a count | |
| # cannot tell a row that LOST its command from one that legitimately | |
| # left the gated population, and the fix for the second reads as | |
| # lowering the number for the first. Not shrink-only, because the unit | |
| # test asserts the set EQUALS RUNNABLE_BASELINE, so GROWTH reds this job | |
| # too: any movement, up or down, re-pins the baseline in the same | |
| # change, naming the rows and the reason. | |
| run: | | |
| python3 scripts/check-gate-commands.py --check | |
| python3 tests/scripts/test_check_gate_commands.py | |
| - name: The GPU mutex has exactly one truth (#777) | |
| # Registered HERE and not only in preflight: a `flock` on the wrong file | |
| # SUCCEEDS, so a second mutex is silent at runtime and surfaces only as | |
| # timing noise attributed to whoever else was on the box. Nothing at run | |
| # time can catch it; this is the only place it can be caught. | |
| run: python3 tests/scripts/test_gpu_lock_one_truth.py | |
| - name: Agent role machinery and role discipline | |
| # Same split as the step above: the two suites are tree-scoped and run | |
| # on every lane; `check-role-discipline.py --base/--head` is DIFF-scoped | |
| # and is skipped where there is no range. | |
| # | |
| # It would not have aborted on its own -- passing `--base ""` makes | |
| # `commits_in_range` fall back to the head commit alone | |
| # (scripts/check-role-discipline.py:328-333), so the baseline lane would | |
| # have reported a PASS covering one commit while looking like it covered | |
| # a range. Skipping loudly beats passing vacuously. Recorded as risk 10 | |
| # in .agents/specs/main-verifiability.md. | |
| env: | |
| EVENT_NAME: ${{ github.event_name }} | |
| PR_BASE: ${{ github.event.pull_request.base.sha }} | |
| PR_HEAD: ${{ github.event.pull_request.head.sha }} | |
| PUSH_BASE: ${{ github.event.before }} | |
| LAST_GREEN: ${{ needs.last-gated-commit.outputs.sha }} | |
| PUSH_HEAD: ${{ github.sha }} | |
| run: | | |
| set -eu | |
| pending_args=() | |
| if [ "$EVENT_NAME" = "pull_request" ]; then | |
| base="$PR_BASE" | |
| head="$PR_HEAD" | |
| pending_args=(--pending-pr-head "$PR_HEAD") | |
| else | |
| # The last SUCCESSFULLY gated commit, not the previous push: a | |
| # cancelled run must not advance the base, or its commits are | |
| # skipped forever. That is what lets the push lane be latest-only | |
| # (#822, #863). Falls back to `before`, today's behaviour. | |
| base="${LAST_GREEN:-}" | |
| [ -n "$base" ] || base="$PUSH_BASE" | |
| head="$PUSH_HEAD" | |
| fi | |
| if [ -z "$base" ]; then | |
| echo "no diff range on the $EVENT_NAME lane: role discipline is" | |
| echo "diff-scoped and every commit's own push already ran it." | |
| else | |
| python3 scripts/check-role-discipline.py \ | |
| --base "$base" --head "$head" "${pending_args[@]}" | |
| fi | |
| python3 tests/scripts/test_agent_role.py | |
| python3 tests/scripts/test_agent_onboard.py | |
| - name: Claim view, helper queue and PR reviewability | |
| run: | | |
| python3 scripts/claim-view.py --check | |
| python3 scripts/ready-for-helper.py --check | |
| python3 scripts/upstream-inventory.py --check | |
| python3 tests/scripts/test_claim_view.py | |
| python3 tests/scripts/test_upstream_inventory.py | |
| - name: Live-state rows are reconciled against Git reality | |
| # The classifier reads its evidence from `origin/main`. This job's | |
| # checkout already uses `fetch-depth: 0`, which DOES leave | |
| # `refs/remotes/origin/*` behind, so the ref is present today -- the | |
| # fetch makes that independent of the checkout block above rather than | |
| # a standing assumption about it, and refreshes the ref besides. | |
| # Without a resolvable `origin/main` the audit aborts by design | |
| # (require_origin_main): "no origin/main" would otherwise read as | |
| # "every ACTIVE row is abandoned", absence of information wearing | |
| # absence of work's face. | |
| # | |
| # The refspec is EXPLICIT and forced, not a bare `git fetch origin | |
| # main`. A bare fetch only updates a remote-tracking ref when a | |
| # CONFIGURED `remote.origin.fetch` refspec matches, and checkout sets | |
| # that config from its own inputs (at shallower settings it narrows to | |
| # the triggering ref, on a PR `refs/pull/N/merge`); with no match it | |
| # writes FETCH_HEAD only. Naming the refspec creates the ref whatever | |
| # the checkout is configured to do. | |
| # | |
| # The `row/*` refspec is the OTHER half of the same argument (#726). | |
| # The classifier's IN-FLIGHT verdict is reached only through a ref named | |
| # `row/<ID>`, and fetching `main` alone left none in the checkout -- so | |
| # IN-FLIGHT was unreachable and a row whose work is genuinely in flight | |
| # produced the same verdict as one nobody is working on. Every PR that | |
| # moves a row to ACTIVE before its code lands failed here, which is the | |
| # normal shape of the work. `require_branch_information` now aborts | |
| # rather than misreport if this refspec is ever dropped again. | |
| # | |
| # It does not cover a FORK, whose branch is a ref `origin` cannot hold; | |
| # that case is covered inside the audit, by reading `origin/main..HEAD`. | |
| # | |
| # Cost measured at `af026e524`: 267 `row/*` heads carrying 648 commits | |
| # not on `origin/main`, over a `fetch-depth: 0` checkout that already | |
| # holds main's history. | |
| run: | | |
| git fetch -q origin +refs/heads/main:refs/remotes/origin/main \ | |
| '+refs/heads/row/*:refs/remotes/origin/row/*' | |
| python3 scripts/audit-live-rows.py --check | |
| python3 tests/scripts/test_audit_live_rows.py | |
| documentation-checkpoint: | |
| # Gate: every code/test/benchmark/spike/lifecycle iteration refreshes both | |
| # user-facing status surfaces in that same commit, including void attempts. | |
| # | |
| # DIFF-scoped: deliberately carries NO concurrency group. Its verdict covers | |
| # this push's own `before..sha` range and nothing re-covers that range later, | |
| # so cancelling it would silently exempt those commits from the gate. | |
| # | |
| # Not on the baseline lane (#274): `schedule` and `workflow_dispatch` | |
| # payloads have no `github.event.before`, so the range below would be empty | |
| # and the gate vacuous. The push that landed each commit already ran this, | |
| # which is exactly why it carries no group. The baseline's subject is the | |
| # TREE at a SHA, and `scripts/main-baseline.py` names the jobs it covered. | |
| # always(): `last-gated-commit` is skipped on the PR lane, and a | |
| # skipped dependency would otherwise skip this gate too. | |
| if: always() && github.event.action != 'closed' && (github.event_name != 'schedule' && github.event_name != 'workflow_dispatch') | |
| needs: [last-gated-commit] | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v4 | |
| with: | |
| fetch-depth: 0 | |
| - name: Every feature checkpoint updates STATUS, BENCHMARKS and NOW | |
| env: | |
| EVENT_NAME: ${{ github.event_name }} | |
| PR_BASE: ${{ github.event.pull_request.base.sha }} | |
| PR_HEAD: ${{ github.event.pull_request.head.sha }} | |
| PUSH_BASE: ${{ github.event.before }} | |
| LAST_GREEN: ${{ needs.last-gated-commit.outputs.sha }} | |
| PUSH_HEAD: ${{ github.sha }} | |
| run: | | |
| set -eu | |
| if [ "$EVENT_NAME" = "pull_request" ]; then | |
| base="$PR_BASE" | |
| head="$PR_HEAD" | |
| else | |
| # The last SUCCESSFULLY gated commit, not the previous push: a | |
| # cancelled run must not advance the base, or its commits are | |
| # skipped forever. That is what lets the push lane be latest-only | |
| # (#822, #863). Falls back to `before`, today's behaviour. | |
| base="${LAST_GREEN:-}" | |
| [ -n "$base" ] || base="$PUSH_BASE" | |
| head="$PUSH_HEAD" | |
| fi | |
| python3 scripts/check-doc-checkpoint.py --base "$base" --head "$head" | |
| python3 scripts/check-now-current.py --base "$base" --head "$head" | |
| pending_args=() | |
| if [ "$EVENT_NAME" = "pull_request" ]; then | |
| pending_args=(--pending-pr-head "$PR_HEAD") | |
| fi | |
| python3 scripts/check-role-discipline.py \ | |
| --base "$base" --head "$head" "${pending_args[@]}" | |
| last-gated-commit: | |
| # The head of the most recent SUCCESSFUL push run of this workflow on this | |
| # branch. It is the base the diff-scoped gates walk from, instead of | |
| # `github.event.before`. | |
| # | |
| # `before` is the previous push's sha whether or not that push was gated, so | |
| # a cancelled run's commits are skipped and NOTHING re-covers them -- the | |
| # reason a superseded main run could not be cancelled at all (#822, #863). | |
| # Basing on the last GREEN commit makes a cancelled run lossless: the next | |
| # run simply walks a wider range. | |
| # | |
| # Carries NO concurrency group, like the gates that consume it. | |
| # | |
| # This is workflow plumbing resolving an input, not a checker. The | |
| # protocol's ban on network calls is about a CHECKER failing on | |
| # connectivity; this falls back to `before` when the query returns nothing, | |
| # so a failed query is the status quo rather than a skipped gate. | |
| # | |
| # It runs on EVERY lane except a closed pull request, and resolves a sha | |
| # only on the push lane -- there is no gated `main` run to find for a pull | |
| # request or a schedule. The wider condition is deliberate: `agent-record` | |
| # may not carry an `if:` of its own without un-registering three checkers | |
| # (#873), so this job's condition is the closed-PR guard it inherits through | |
| # `needs:`. A skipped dependency skips its dependents, which is the same | |
| # skip #822 asked for, expressed where a checker can still see the gate. | |
| if: github.event.action != 'closed' | |
| runs-on: ubuntu-latest | |
| permissions: | |
| actions: read | |
| contents: read | |
| outputs: | |
| sha: ${{ steps.resolve.outputs.sha }} | |
| steps: | |
| - id: resolve | |
| env: | |
| GH_TOKEN: ${{ github.token }} | |
| REPO: ${{ github.repository }} | |
| BRANCH: ${{ github.ref_name }} | |
| EVENT_NAME: ${{ github.event_name }} | |
| run: | | |
| set -eu | |
| # Only the push lane has a gated `main` history to resolve against. | |
| # Every other lane leaves this EMPTY and its consumers fall back, | |
| # which is what they already do when the query finds nothing. | |
| sha="" | |
| if [ "$EVENT_NAME" = "push" ]; then | |
| sha="$(gh api \ | |
| "repos/$REPO/actions/workflows/ci.yml/runs?branch=$BRANCH&event=push&status=success&per_page=1" \ | |
| --jq '.workflow_runs[0].head_sha' 2>/dev/null || true)" | |
| fi | |
| # NOT `[ ... ] && sha=""`: that returns non-zero when the test is | |
| # false, which aborts the step under `set -e`. | |
| if [ "$sha" = "null" ]; then sha=""; fi | |
| echo "last successfully gated commit: ${sha:-<none, falling back to before>}" | |
| echo "sha=$sha" >> "$GITHUB_OUTPUT" | |
| commit-protocol-tag: | |
| # Gate: every NEW commit must carry the FOLLOWING_AGENTS_PROTOCOL trailer, | |
| # asserting the contributor read AGENTS.md. See .agents/ai-coding-assistants.md. | |
| # | |
| # DIFF-scoped: deliberately carries NO concurrency group, same reasoning as | |
| # documentation-checkpoint above. It walks `before..sha` per push, so a | |
| # cancelled run means those commits are never checked for the trailer. | |
| # | |
| # Not on the baseline lane (#274), same reasoning: no `before` on a | |
| # `schedule` or `workflow_dispatch` payload, so the walk has no range. | |
| needs: [last-gated-commit] | |
| if: always() && github.event.action != 'closed' && github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v4 | |
| with: | |
| fetch-depth: 0 | |
| - name: Every new commit carries FOLLOWING_AGENTS_PROTOCOL | |
| env: | |
| LAST_GREEN: ${{ needs.last-gated-commit.outputs.sha }} | |
| run: | | |
| set -eu | |
| if [ "${{ github.event_name }}" = "pull_request" ]; then | |
| base="${{ github.event.pull_request.base.sha }}" | |
| head="${{ github.event.pull_request.head.sha }}" | |
| else | |
| # Same self-healing base as the strict walk below (#822, #863). | |
| base="${LAST_GREEN:-}" | |
| [ -n "$base" ] || base="${{ github.event.before }}" | |
| head="${{ github.sha }}" | |
| fi | |
| # Enforce on the FIRST-PARENT mainline (the commits the submitter | |
| # directly lands on the target branch, incl. merge commits which are | |
| # skipped below). Merged-in feature-branch commits are owned by the | |
| # submitter via the merge and are not re-checked here. | |
| # New branch / unreachable base (all-zero SHA): only check the tip commit. | |
| if ! git cat-file -e "${base}^{commit}" 2>/dev/null; then | |
| commits="$head" | |
| else | |
| commits="$(git rev-list --first-parent "${base}..${head}")" | |
| fi | |
| fail=0 | |
| for c in $commits; do | |
| # Skip merge commits (>1 parent) — they are not authored content. | |
| if [ "$(git rev-list --parents -n1 "$c" | wc -w)" -gt 2 ]; then continue; fi | |
| if ! git log -1 --format=%B "$c" | grep -q 'FOLLOWING_AGENTS_PROTOCOL'; then | |
| echo "::error::commit $c is missing the FOLLOWING_AGENTS_PROTOCOL trailer — read AGENTS.md" | |
| git log -1 --oneline "$c" | |
| fail=1 | |
| fi | |
| done | |
| if [ "$fail" -ne 0 ]; then | |
| echo "One or more commits lack FOLLOWING_AGENTS_PROTOCOL. See AGENTS.md / .agents/ai-coding-assistants.md." | |
| exit 1 | |
| fi | |
| echo "OK: all new commits carry FOLLOWING_AGENTS_PROTOCOL." | |
| - name: Every new commit satisfies the strict trailer contract | |
| # DIFF-scoped, and therefore HERE rather than in agent-record. It lived | |
| # in that job until #863: agent-record carries a cancellable group keyed | |
| # on `github.ref`, which for a push is the constant `refs/heads/main`, so | |
| # consecutive pushes cancelled it. Measured on run 31851003245, the push | |
| # of 51e0cb5b1: agent-record cancelled, this job succeeded, and the | |
| # commit fails the strict walk. The grep above passed it because the | |
| # marker was PRESENT three times -- presence is not parseability, and | |
| # only the strict walk knows the difference. | |
| env: | |
| EVENT_NAME: ${{ github.event_name }} | |
| PR_BASE: ${{ github.event.pull_request.base.sha }} | |
| PR_HEAD: ${{ github.event.pull_request.head.sha }} | |
| PUSH_BASE: ${{ github.event.before }} | |
| LAST_GREEN: ${{ needs.last-gated-commit.outputs.sha }} | |
| PUSH_HEAD: ${{ github.sha }} | |
| # Passed as an ENV VAR, never interpolated into the script: a body is | |
| # attacker-controlled text on a fork pull request. | |
| PR_BODY: ${{ github.event.pull_request.body }} | |
| run: | | |
| set -eu | |
| if [ "$EVENT_NAME" = "pull_request" ]; then | |
| base="$PR_BASE" | |
| head="$PR_HEAD" | |
| else | |
| # The last SUCCESSFULLY gated commit, not the previous push. A | |
| # cancelled or failed run must not advance the base, or its commits | |
| # are skipped forever -- that is the whole reason a superseded main | |
| # run could not be cancelled before (#822, #863). Falls back to | |
| # `before`, which is the previous behaviour, when nothing is found. | |
| base="${LAST_GREEN:-}" | |
| [ -n "$base" ] || base="$PUSH_BASE" | |
| head="$PUSH_HEAD" | |
| fi | |
| if [ -z "$base" ] || ! git cat-file -e "${base}^{commit}" 2>/dev/null; then | |
| echo "no usable diff range on the $EVENT_NAME lane: base=${base:-<empty>}" | |
| echo "checking the tip commit alone rather than passing vacuously." | |
| python3 scripts/check-commit-trailers.py --range "$head~1..$head" | |
| else | |
| echo "strict trailer walk over ${base}..${head}" | |
| python3 scripts/check-commit-trailers.py --range "$base..$head" | |
| fi | |
| # `squash_merge_commit_message = PR_BODY` makes the pull request body | |
| # the landed commit message, so it is held to the contract its commit | |
| # will be held to, by the same checker (#848). `${PR_BODY:-}` because | |
| # this runs under `set -eu` and the variable is UNSET, not empty, on a | |
| # lane with no pull request. An empty body is not skipped: it lands a | |
| # commit carrying no trailers at all. | |
| if [ "$EVENT_NAME" = "pull_request" ]; then | |
| body_file="$(mktemp)" | |
| printf '%s' "${PR_BODY:-}" > "$body_file" | |
| python3 scripts/check-commit-trailers.py \ | |
| --message-file "$body_file" --filled | |
| fi | |
| pr-size: | |
| # PR-only. The per-class LINE BUDGETS this job used to enforce were retired | |
| # 2026-08-10, and the fail-closed BINARY GUARD was retired 2026-08-13 (see | |
| # scripts/check-pr-size.py for both); the job name is kept because it is a | |
| # required check. What it enforces now: explicit path classification, the | |
| # checker-change mutation-evidence contract, and the role check that keeps | |
| # product paths on a PR. Nothing here measures the size of a diff. | |
| if: github.event.action != 'closed' && (github.event_name == 'pull_request') | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v4 | |
| with: | |
| fetch-depth: 0 | |
| - name: PR path classification and checker evidence | |
| run: | | |
| python3 scripts/check-pr-size.py \ | |
| --base "${{ github.event.pull_request.base.sha }}" \ | |
| --head "${{ github.event.pull_request.head.sha }}" \ | |
| --branch "${{ github.event.pull_request.head.ref }}" \ | |
| --pr-number "${{ github.event.pull_request.number }}" | |
| cuda-arch-features: | |
| # Configure-tier assertions on the CUDA per-arch FEATURE TABLE. Needs neither | |
| # a CUDA toolkit nor a GPU: it drives cmake/CudaArchFeatures.cmake directly, | |
| # so the resolution that decides which architectures get fp4-mma / | |
| # cutlass-nvfp4 / cutlass-fp8 / marlin-nvfp4 / fa2 (whose historical failure | |
| # mode was a SILENT capability drop) is checked on every push. | |
| # | |
| # Tree-scoped (drives cmake at HEAD): cancellable. | |
| concurrency: | |
| group: ci-cuda-arch-features-${{ github.event_name }}-${{ github.ref }}-${{ github.repository }} | |
| cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }} | |
| # Skip when the pull request was CLOSED: the run exists only to enter | |
| # the concurrency group and supersede the in-flight one (#822). | |
| if: github.event.action != 'closed' | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - name: CUDA feature table and per-source gencode resolve exactly | |
| run: | | |
| cmake -P cmake/CudaArchFeaturesTest.cmake | |
| cmake -P cmake/CudaSourceGencodeTest.cmake | |
| cmake -P cmake/TritonAOTMultiArchTest.cmake | |
| cmake -P cmake/TritonAOTDefaultTest.cmake | |
| # Same tier, no CUDA involved: the in-source build guard (issue #85) is a | |
| # configure-time predicate, so it is asserted the same way — by driving the | |
| # module directly, with no compiler and no writes to the tree. | |
| - name: In-source build guard fires on `cmake .` and only on `cmake .` | |
| run: | | |
| cmake -P cmake/InSourceGuardTest.cmake | |
| python3 -m unittest tests.scripts.test_check_cuda_fat_gencode | |
| python3 -m unittest tests.scripts.test_check_triton_aot_multiarch | |
| cuda-fat-build: | |
| # W1 release prerequisite: compile the primary heterogeneous CUDA archive, | |
| # then inspect both compile_commands.json and the linked archive. No GPU is | |
| # used; runtime/correctness evidence remains independent. | |
| permissions: | |
| contents: read | |
| concurrency: | |
| group: ci-cuda-fat-build-${{ github.event_name }}-${{ github.ref }}-${{ github.repository }} | |
| cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }} | |
| # Skip when the pull request was CLOSED: the run exists only to enter | |
| # the concurrency group and supersede the in-flight one (#822). | |
| if: github.event.action != 'closed' | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 180 | |
| container: nvidia/cuda:13.3.0-devel-ubuntu24.04 | |
| steps: | |
| - name: Install build tools | |
| run: | | |
| apt-get update | |
| DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ | |
| binutils ca-certificates cmake g++ git ninja-build python3 | |
| rm -rf /var/lib/apt/lists/* | |
| - uses: actions/checkout@v4 | |
| - name: Configure ten-SM static library | |
| run: | | |
| cmake -S . -B build-cuda-fat -G Ninja \ | |
| -DCMAKE_BUILD_TYPE=Release \ | |
| -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \ | |
| -DVLLM_CPP_CUDA=ON \ | |
| -DVLLM_CPP_CUDA_ARCHITECTURES='80;86;87;89;90a;100a;103a;110;120a;121a' \ | |
| -DVLLM_CPP_CUTLASS_FETCH=ON \ | |
| -DVLLM_CPP_TRITON=ON \ | |
| -DVLLM_CPP_BUILD_TESTS=OFF \ | |
| -DVLLM_CPP_BUILD_EXAMPLES=OFF \ | |
| -DVLLM_CPP_SERVER=OFF | |
| - name: Build and audit exact per-source gencode | |
| run: | | |
| cmake --build build-cuda-fat --target vllm --parallel 2 | |
| python3 scripts/check-cuda-fat-gencode.py \ | |
| --compile-commands build-cuda-fat/compile_commands.json \ | |
| --library build-cuda-fat/libvllm.a | |
| python3 scripts/check-triton-aot-multiarch.py \ | |
| --vendored-root src/vt/cuda/triton_aot_vendored \ | |
| --library build-cuda-fat/libvllm.a | |
| # BUILD-TRITON-DEFAULT-ON (#219), spec Tests item 3. The configure above | |
| # passes -DVLLM_CPP_TRITON=ON explicitly, so it cannot witness the DEFAULT. | |
| # This one passes no such flag: the option must resolve ON by itself AND | |
| # reach every CUDA translation unit as VLLM_CPP_TRITON=1 / | |
| # VLLM_CPP_TRITON_CHUNKO_BF16=1, which is the row's actual claim. Configure | |
| # only, and it reuses the CUTLASS the job already fetched (FETCH stays ON | |
| # as the fallback if that path ever moves). | |
| - name: The computed Triton default reaches the CUDA translation units | |
| run: | | |
| cmake -S . -B build-cuda-default -G Ninja \ | |
| -DCMAKE_BUILD_TYPE=Release \ | |
| -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \ | |
| -DVLLM_CPP_CUDA=ON \ | |
| -DVLLM_CPP_CUDA_ARCHITECTURES='80;86;87;89;90a;100a;103a;110;120a;121a' \ | |
| -DVLLM_CPP_CUTLASS_DIR="$PWD/build-cuda-fat/_deps/cutlass-src" \ | |
| -DVLLM_CPP_CUTLASS_FETCH=ON \ | |
| -DVLLM_CPP_BUILD_TESTS=OFF \ | |
| -DVLLM_CPP_BUILD_EXAMPLES=OFF \ | |
| -DVLLM_CPP_SERVER=OFF | |
| VLLM_CPP_DEFAULT_BUILD_DIR=build-cuda-default \ | |
| python3 -m unittest tests.scripts.test_triton_default_definitions | |
| vulkan-spirv-freshness: | |
| # The Vulkan backend commits its SPIR-V (src/vt/vulkan/vulkan_spirv.h) instead | |
| # of compiling shaders at build time, so the build needs no shader toolchain on | |
| # any machine -- strictly better than llama.cpp, which requires glslc on every | |
| # build box. The trade is an obligation to regenerate by hand, and NOTHING | |
| # ENFORCED IT: `gen-vulkan-spirv.py --check` existed but no job ran it, so a | |
| # .comp edit without a regenerate shipped silently. That is the failure mode | |
| # this job closes, before the shader surface grows (BACKEND-VULKAN, VK-A1). | |
| # | |
| # The glslang DOWNLOAD URL is the pin. An exact version-string assertion was | |
| # deliberately NOT used: the committed SPIR-V was measured byte-identical under | |
| # both the 16.4.0 that produced it and the 16.5.0 pinned here, so gating on the | |
| # string would be brittle without buying anything. The URL being fixed is what | |
| # makes the run deterministic. | |
| # | |
| # Tree-scoped (checks HEAD's artifact): cancellable. | |
| concurrency: | |
| group: ci-vulkan-spirv-freshness-${{ github.event_name }}-${{ github.ref }}-${{ github.repository }} | |
| cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }} | |
| # Skip when the pull request was CLOSED: the run exists only to enter | |
| # the concurrency group and supersede the in-flight one (#822). | |
| if: github.event.action != 'closed' | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - name: Install pinned glslang | |
| run: | | |
| set -euo pipefail | |
| curl -fsSL -o /tmp/glslang.tar.gz \ | |
| https://github.com/KhronosGroup/glslang/releases/download/16.5.0/glslang-16.5.0-linux-x86_64-release.tar.gz | |
| mkdir -p /tmp/glslang | |
| tar xzf /tmp/glslang.tar.gz -C /tmp/glslang | |
| /tmp/glslang/bin/glslang --version | head -1 | |
| echo "/tmp/glslang/bin" >> "$GITHUB_PATH" | |
| - name: Committed SPIR-V is not stale | |
| run: python3 scripts/gen-vulkan-spirv.py --check | |
| build-test-vulkan: | |
| # The Vulkan backend had NO CI leg at all: `VLLM_CPP_VULKAN=ON` appeared | |
| # nowhere in this workflow, so tests/vt/test_vulkan_backend.cpp ran on no | |
| # machine. It rotted exactly as you would expect. Accelerator-seam row S5 | |
| # (af0b21ba) gave unified-memory devices a portable CPU reference tier, which | |
| # means a missed GetOp no longer throws; the Metal sibling test was updated for | |
| # that as Metal work continued, and the Vulkan one was not, so its "the | |
| # unimplemented ops throw" assertion sat RED and INVISIBLE from S5 until | |
| # VK-A1 built the backend by hand (2026-08-06). That is the gap this closes. | |
| # | |
| # Runs GPU-FREE on `mesa-vulkan-drivers`' llvmpipe software ICD, which is the | |
| # arrangement the fan-out spike already recorded as working. The suite is | |
| # scoped to the Vulkan-specific targets rather than all of ctest: the full | |
| # suite is already covered by build-test-cpu, and this lane exists to keep the | |
| # Vulkan-only TUs compiled and executed, not to duplicate it. | |
| # | |
| # Tree-scoped: cancellable. | |
| concurrency: | |
| group: ci-build-test-vulkan-${{ github.event_name }}-${{ github.ref }}-${{ github.repository }} | |
| cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }} | |
| # Skip when the pull request was CLOSED: the run exists only to enter | |
| # the concurrency group and supersede the in-flight one (#822). | |
| if: github.event.action != 'closed' | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - name: Install the software Vulkan ICD | |
| # Loader + llvmpipe only. No shader toolchain: the backend consumes the | |
| # COMMITTED SPIR-V, which is the whole point of the committed-artifact | |
| # route, and the vulkan-spirv-freshness job above is what keeps it honest. | |
| run: | | |
| set -euo pipefail | |
| sudo apt-get update | |
| sudo apt-get install -y --no-install-recommends libvulkan1 mesa-vulkan-drivers | |
| - name: Configure | |
| run: cmake -S . -B build-vulkan -DVLLM_CPP_BUILD_TESTS=ON -DVLLM_CPP_VULKAN=ON | |
| - name: Build | |
| # Bounded parallelism for the same reason as build-test-cpu: an unbounded | |
| # parallel link OOM-kills the runner. | |
| run: cmake --build build-vulkan -j 2 --target test_vulkan_backend test_backend_cross_device | |
| - name: Vulkan backend gate | |
| run: ./build-vulkan/tests/test_vulkan_backend | |
| - name: Cross-device numerics vs the CPU oracle | |
| run: ./build-vulkan/tests/test_backend_cross_device | |
| device-leakage: | |
| # The DSR RATCHET (work row `S1` of .agents/specs/accelerator-seam-audit.md). | |
| # Counts device-specific references in `src/vllm/` + `include/vllm/` — the | |
| # layer that is supposed to be device-agnostic — and fails on ANY increase | |
| # over scripts/device-leakage-baseline.json. A reduction must lower the | |
| # baseline in the SAME commit, so the number can only ever move down. | |
| # | |
| # It exists because the audit re-measured the leakage and found it had DRIFTED | |
| # UPWARD with no bad commit: DeepSeek-V2, Qwen3-Coder and the attention-registry | |
| # work each added a device test in passing. Leakage grows silently under | |
| # well-executed work, which is a job for a ratchet, not a cleanup. | |
| # | |
| # Needs neither a CUDA toolkit nor a GPU — pure static analysis, like the | |
| # cuda-arch-features job above. The mutation suite proves the checker actually | |
| # catches a planted leak; an unpoliced checker is worse than none. | |
| # | |
| # Tree-scoped: the ratchet compares HEAD against the committed baseline, so | |
| # only the newest push matters. Cancellable. | |
| concurrency: | |
| group: ci-device-leakage-${{ github.event_name }}-${{ github.ref }}-${{ github.repository }} | |
| cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }} | |
| # Skip when the pull request was CLOSED: the run exists only to enter | |
| # the concurrency group and supersede the in-flight one (#822). | |
| if: github.event.action != 'closed' | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - name: Shared-layer device leakage does not grow (DSR ratchet) | |
| run: | | |
| python3 scripts/check-device-leakage.py --report | |
| python3 tests/scripts/test_device_leakage.py | |
| windows-msvc-cpu: | |
| # Native Windows release portability must be proven before merge. This PR | |
| # lane calls the authoritative driver but retains no artifact and has no | |
| # release, upload, write-token, or OIDC authority (#117). | |
| if: github.event_name == 'pull_request' | |
| permissions: | |
| contents: read | |
| runs-on: windows-2022 | |
| timeout-minutes: 180 | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - name: Prove PowerShell, static CRT, and unsupported-tier contracts | |
| run: ./scripts/build-windows-release.ps1 -ContractTest | |
| - name: Build and execute the native Windows CPU focused gate | |
| env: | |
| EVIDENCE_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} | |
| SOURCE_SHA: ${{ github.sha }} | |
| run: | | |
| $env:VERSION = (Get-Content release/release-version.json -Raw | ConvertFrom-Json).version | |
| $env:SOURCE_DATE_EPOCH = (git show -s --format=%ct HEAD).Trim() | |
| ./scripts/build-windows-release.ps1 ` | |
| -Backend cpu ` | |
| -ArtifactId windows-x86_64-msvc-cpu ` | |
| -BuildDir $env:GITHUB_WORKSPACE/build-pr-windows-cpu | |
| windows-msvc-vulkan: | |
| # Independent Vulkan build/loader/backend proof on the native MSVC ABI. | |
| # Like the CPU lane it is PR-only, read-only, and retains no release asset. | |
| if: github.event_name == 'pull_request' | |
| permissions: | |
| contents: read | |
| runs-on: windows-2022 | |
| timeout-minutes: 180 | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - name: Prove PowerShell, static CRT, and unsupported-tier contracts | |
| run: ./scripts/build-windows-release.ps1 -ContractTest | |
| - name: Build and execute the native Windows Vulkan focused gate | |
| env: | |
| EVIDENCE_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} | |
| SOURCE_SHA: ${{ github.sha }} | |
| run: | | |
| $env:VERSION = (Get-Content release/release-version.json -Raw | ConvertFrom-Json).version | |
| $env:SOURCE_DATE_EPOCH = (git show -s --format=%ct HEAD).Trim() | |
| ./scripts/build-windows-release.ps1 ` | |
| -Backend vulkan ` | |
| -ArtifactId windows-x86_64-msvc-vulkan ` | |
| -BuildDir $env:GITHUB_WORKSPACE/build-pr-windows-vulkan | |
| build-test-cpu: | |
| # Tree-scoped and the most expensive lane in the workflow. Cancellable. | |
| concurrency: | |
| group: ci-build-test-cpu-${{ github.event_name }}-${{ github.ref }}-${{ github.repository }} | |
| cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }} | |
| # Skip when the pull request was CLOSED: the run exists only to enter | |
| # the concurrency group and supersede the in-flight one (#822). | |
| if: github.event.action != 'closed' | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - name: Configure | |
| run: | | |
| cmake -S . -B build \ | |
| -DVLLM_CPP_BUILD_TESTS=ON \ | |
| -DCMAKE_EXPORT_COMPILE_COMMANDS=ON | |
| - name: Build | |
| # Bounded parallelism: a bare `-j` lets Make link ALL test executables at | |
| # once, which OOM-kills the runner (ld signal 9) during the parallel link. | |
| run: cmake --build build -j 2 | |
| - name: Installed server archive is reproducible and runnable | |
| env: | |
| VLLM_CPP_PACKAGE_TEST_BUILD_DIR: ${{ github.workspace }}/build | |
| run: python3 tests/scripts/test_server_package.py | |
| - name: CPU ISA objects use only their declared feature flags | |
| run: | | |
| python3 scripts/check-cpu-isa-build.py \ | |
| --compile-commands build/compile_commands.json | |
| - name: Test | |
| run: ctest --test-dir build --output-on-failure | |
| - name: Force every x86 ISA tier available on this runner | |
| run: | | |
| tiers=(portable) | |
| if [[ "$(uname -m)" == "x86_64" ]]; then | |
| tiers+=(sse2) | |
| if grep -qiw f16c /proc/cpuinfo && grep -qiw avx /proc/cpuinfo; then | |
| tiers+=(sse2+f16c) | |
| fi | |
| if grep -qiw avx2 /proc/cpuinfo; then | |
| tiers+=(avx2) | |
| fi | |
| if grep -qiw avx512f /proc/cpuinfo && \ | |
| grep -qiw avx512bw /proc/cpuinfo && \ | |
| grep -qiw avx512vl /proc/cpuinfo; then | |
| tiers+=(avx512) | |
| fi | |
| fi | |
| for tier in "${tiers[@]}"; do | |
| echo "Running test_ops_matmul_elem with VT_CPU_MATMUL_TIER=${tier}" | |
| VT_CPU_MATMUL_TIER="${tier}" build/tests/test_ops_matmul_elem | |
| done | |
| build-test-cpu-arm64: | |
| # Independent Arm execution evidence: the x86 lane cannot prove HWCAP | |
| # dispatch, Arm instructions, or the host ABI. The native runner exercises | |
| # every available rich tier; qemu's Cortex-A53 model supplies the required | |
| # feature-poor execution and illegal-tier refusal gate. | |
| concurrency: | |
| group: ci-build-test-cpu-arm64-${{ github.event_name }}-${{ github.ref }}-${{ github.repository }} | |
| cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }} | |
| # Skip when the pull request was CLOSED: the run exists only to enter | |
| # the concurrency group and supersede the in-flight one (#822). | |
| if: github.event.action != 'closed' | |
| runs-on: ubuntu-24.04-arm | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - name: Configure focused Arm CPU gate | |
| run: | | |
| cmake -S . -B build-arm \ | |
| -DVLLM_CPP_BUILD_TESTS=ON \ | |
| -DVLLM_CPP_BUILD_EXAMPLES=OFF \ | |
| -DVLLM_CPP_SERVER=OFF \ | |
| -DVLLM_CPP_CUDA=OFF \ | |
| -DCMAKE_EXPORT_COMPILE_COMMANDS=ON | |
| - name: Build Arm ISA and kernel gates | |
| run: | | |
| cmake --build build-arm \ | |
| --target test_cpu_isa_arm test_ops_matmul_elem \ | |
| test_ops_quant_dot test_ops_quant_repack \ | |
| -j 2 | |
| - name: Arm ISA objects use only their declared feature flags | |
| run: | | |
| python3 scripts/check-arm-isa-build.py \ | |
| --compile-commands build-arm/compile_commands.json | |
| - name: Force baseline and every rich tier available on the native runner | |
| run: | | |
| build-arm/tests/test_cpu_isa_arm | |
| VT_CPU_MATMUL_TIER=portable build-arm/tests/test_ops_matmul_elem | |
| VT_CPU_MATMUL_TIER=neon build-arm/tests/test_ops_matmul_elem | |
| VT_CPU_Q8_DOT=portable \ | |
| VT_CPU_QUANT_MMLA=portable \ | |
| build-arm/tests/test_ops_quant_dot | |
| VT_CPU_QUANT_REPACK=portable build-arm/tests/test_ops_quant_repack | |
| if grep -qiw asimddp /proc/cpuinfo; then | |
| VT_CPU_Q8_DOT=sdot build-arm/tests/test_ops_quant_dot | |
| fi | |
| if grep -qiw asimddp /proc/cpuinfo && grep -qiw i8mm /proc/cpuinfo; then | |
| VT_CPU_QUANT_MMLA=i8mm build-arm/tests/test_ops_quant_dot | |
| VT_CPU_QUANT_REPACK=i8mm build-arm/tests/test_ops_quant_repack | |
| fi | |
| - name: Feature-poor Cortex-A53 executes baseline and refuses rich tiers | |
| run: | | |
| sudo apt-get update | |
| sudo apt-get install --yes qemu-user | |
| qemu-aarch64 -cpu cortex-a53 build-arm/tests/test_cpu_isa_arm | |
| VT_CPU_MATMUL_TIER=portable \ | |
| qemu-aarch64 -cpu cortex-a53 build-arm/tests/test_ops_matmul_elem | |
| VT_CPU_MATMUL_TIER=neon \ | |
| qemu-aarch64 -cpu cortex-a53 build-arm/tests/test_ops_matmul_elem | |
| VT_CPU_Q8_DOT=portable VT_CPU_QUANT_MMLA=portable \ | |
| qemu-aarch64 -cpu cortex-a53 build-arm/tests/test_ops_quant_dot | |
| if VT_CPU_Q8_DOT=sdot \ | |
| qemu-aarch64 -cpu cortex-a53 build-arm/tests/test_ops_quant_dot; then | |
| echo "Cortex-A53 unexpectedly accepted the DotProd tier" >&2 | |
| exit 1 | |
| fi | |
| if VT_CPU_QUANT_MMLA=i8mm \ | |
| qemu-aarch64 -cpu cortex-a53 build-arm/tests/test_ops_quant_dot; then | |
| echo "Cortex-A53 unexpectedly accepted the i8mm tier" >&2 | |
| exit 1 | |
| fi | |
| sanitize-cpu: | |
| # The DYNAMIC detector lanes (VLLM_CPP_SANITIZE, see CMakeLists.txt). The | |
| # plain build-test-cpu job above proves the suite PASSES; it cannot see a | |
| # read one past a tensor row, a use-after-free of an engine-owned container, | |
| # a signed-overflow index computation, or an unsynchronized access between | |
| # the serving threads — every one of which stays green under -O2 until it | |
| # silently corrupts a token stream. These jobs run the SAME suite with | |
| # ASan+UBSan and with TSan. | |
| # | |
| # Two jobs, not one: the runtimes are mutually exclusive. Both are CPU-tier | |
| # (the lane refuses to configure with the CUDA backend on) and both are | |
| # `continue-on-error` for their FIRST landing so a pre-existing finding | |
| # cannot block unrelated work — the finding is triaged, then this flag is | |
| # removed and the lane becomes binding. Removing it is tracked as the | |
| # closing step of the hardening-adoption row. | |
| strategy: | |
| fail-fast: false | |
| matrix: | |
| lane: ["address,undefined", "thread"] | |
| continue-on-error: true | |
| # Tree-scoped: cancellable. `matrix.lane` MUST be part of the group key, or | |
| # the two mutually-exclusive sanitizer legs of the SAME run would land in one | |
| # group and cancel each other, leaving only whichever started second. | |
| concurrency: | |
| group: ci-sanitize-cpu-${{ matrix.lane }}-${{ github.event_name }}-${{ github.ref }}-${{ github.repository }} | |
| cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }} | |
| # Skip when the pull request was CLOSED: the run exists only to enter | |
| # the concurrency group and supersede the in-flight one (#822). | |
| if: github.event.action != 'closed' | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - name: Configure | |
| run: | | |
| cmake -S . -B build-sanitize \ | |
| -DVLLM_CPP_BUILD_TESTS=ON \ | |
| -DVLLM_CPP_CUDA=OFF \ | |
| -DVLLM_CPP_SANITIZE='${{ matrix.lane }}' | |
| - name: Build | |
| run: cmake --build build-sanitize -j 2 | |
| - name: Test | |
| # ctest runs the suite serially: the sanitizer runtimes multiply peak RSS, | |
| # and a parallel run OOM-kills the runner before it reports a finding. | |
| env: | |
| UBSAN_OPTIONS: print_stacktrace=1 | |
| ASAN_OPTIONS: detect_leaks=1:strict_string_checks=1 | |
| # The production DevicePool deliberately retains scratch blocks. Its | |
| # detector lane uses exact allocations and real frees so ASan can | |
| # distinguish that cache from a leak and can see use-after-free. | |
| VT_POOL_BYPASS: "1" | |
| run: ctest --test-dir build-sanitize --output-on-failure | |
| baseline-summary: | |
| # THE PUBLISHED VERDICT (issue #274, spec .agents/specs/main-verifiability.md). | |
| # | |
| # Baseline lane ONLY. It can never run on a `pull_request` or a `push`, so it | |
| # blocks no contributor and changes nothing about what a PR must pass. | |
| # | |
| # It fails when any covered job is red, `sanitize-cpu` INCLUDED. That job is | |
| # `continue-on-error: true` for the push/PR lanes and stays that way -- its | |
| # removal is the closing step of the hardening row, not this one. What this | |
| # job refuses is a BASELINE that reports green while a job it covered is red, | |
| # which would rebuild the very defect #274 was filed about. | |
| # | |
| # It reads per-job conclusions from the Actions API for its own run, NOT from | |
| # `needs.<job>.result`: `continue-on-error` makes that context report | |
| # `success` for a failed job, which is the trap the whole row is about. Run | |
| # 31448896841 is the live proof -- run conclusion `success`, both sanitizer | |
| # lanes `failure`. | |
| # | |
| # On the day this lands the first run is EXPECTED RED, on the six sanitizer | |
| # failures of #274 finding #1 / #301. That is the correct first verdict. A | |
| # baseline that hid them would certify nothing. | |
| if: always() && github.event.action != 'closed' && (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') | |
| needs: | |
| - agent-record | |
| - cuda-arch-features | |
| - cuda-fat-build | |
| - vulkan-spirv-freshness | |
| - build-test-vulkan | |
| - device-leakage | |
| - build-test-cpu | |
| - build-test-cpu-arm64 | |
| - sanitize-cpu | |
| permissions: | |
| contents: read | |
| actions: read | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - name: Publish the per-job baseline verdict for this SHA | |
| env: | |
| GH_TOKEN: ${{ github.token }} | |
| run: | | |
| python3 scripts/main-baseline.py \ | |
| --run-id "${{ github.run_id }}" --emit-summary |