Skip to content

chore/add npu ci image build step #361

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Meihan-chen merged 2 commits into from
Jul 23, 2026
Merged

chore/add npu ci image build step #361

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
35d436e
chore/add npu ci image build step
yzeyu71 Jul 20, 2026
ab17afb
add ci image build test
yzeyu71 Jul 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 26 additions & 11 deletions .buildkite/README-NPU-CI.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,31 +12,46 @@ Three steps run in order:

**`pre-commit-npu`** runs the pre-commit gate on all files, always.

**`image-build-npu`** builds and pushes the NPU test image via buildctl/buildkit.
It only runs when `Dockerfile.npu` has changed (PR trigger) or on a scheduled
build — otherwise the step is skipped and the pre-built default image is used.
**`npu-gate`** is a block step that pauses the pipeline for manual interaction.
It appears only on PR triggers (`build.source != "schedule"`). You select which
NPU suites to run via a multi-select field:

- **`image-build`** — triggers a fresh image build in the `image-build-npu` step;
when selected, `smk` is automatically included and all test steps use the newly
built image instead of the pre-built default.
- **`smk`** — runs the smoke test suite.
- **`nightly`** — runs the nightly test suite.

**`image-build-npu`** the `image-build-npu` step will build and
push a fresh NPU test image tagged with the current commit. When `image-build`
is selected, all test steps generated by `upload-npu-suites` use the newly built
image instead of the pre-built default image.

This allows testing code changes that require an updated NPU environment
(e.g., modifications to `docker/Dockerfile.npu`) before they are merged.

**`upload-npu-suites`** reads the `NPU_SUITES` environment variable (or
`buildkite-agent meta-data get npu-suites` for PR triggers) and generates
individual test jobs via [`npu_suites.py`](./npu_suites.py).

## Triggers

The pipeline supports three trigger modes:
The pipeline supports two trigger modes:

**PR trigger.** On a PR trigger, the pre-built default image is used and
test suites are selected manually via the block step.
**PR trigger.** The `npu-gate` block step appears for manual suite selection.
Suites not selected in the block step are skipped. The `image-build-npu` step
builds a new image only when `image-build` is chosen in the block.

**Schedule trigger.** On a scheduled build, a new image is always built
regardless of file changes. The suites to run are determined by the
**Schedule trigger.** There is no block step — the `npu-gate` step is omitted
entirely. A new image is **always** built, and the suites are determined by the
`NPU_SUITES` environment variable (set in the scheduled build's pipeline
configuration), which selects the corresponding entries from the `SUITES` dict.
configuration), which lists suite names from the `SUITES` dict.

## Adding a test

Suites and test mappings are defined in [`npu_suites.py`](./npu_suites.py). Two
suites are predefined — `smk` (always runs) and `nightly` (runs on schedule or
with the `run-ci-npu-nightly` label).
suites are predefined — `smk` (runs with the `run-ci-npu-smk` label) and `nightly`
(runs on schedule or with the `run-ci-npu-nightly` label).

Each entry is a 4-tuple:

Expand Down
34 changes: 28 additions & 6 deletions .buildkite/npu_suites.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,14 @@
import json
import os
import subprocess
import sys

NPU_QUEUE = "ascend-a3"
CI_IMAGE = os.environ.get("IMAGE_BUILD", "quay.io/ascend/vime:0.3.0-a3-vllm0.22.1rc1")
DEFAULT_CI_IMAGE = "quay.io/ascend/vime:0.3.0-a3-vllm0.22.1rc1"
IMAGE_REGISTRY = "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry"
IMAGE_NAME = "vime-ci-npu"
VIME_IMAGE_TAG = os.environ.get("BUILDKITE_COMMIT", "latest")
BUILDKITE_SOURCE = os.environ.get("BUILDKITE_SOURCE", "")

# (test_name, resource_class, extra_args, env_overrides)
SUITES = {
Expand All @@ -33,7 +35,7 @@
}


def selected_suites() -> list:
def _read_suite_values() -> list[str]:
raw = os.environ.get("NPU_SUITES")
if raw is None:
try:
Expand All @@ -45,10 +47,24 @@ def selected_suites() -> list:
).stdout
except subprocess.CalledProcessError:
raw = ""
values = [v.strip() for v in raw.replace(",", "\n").splitlines()]
unknown = [v for v in values if v and v not in SUITES]
return [v.strip() for v in raw.replace(",", "\n").splitlines()]


def _ci_image() -> str:
values = _read_suite_values()
if ("image-build" in values) or (BUILDKITE_SOURCE == "schedule"):
return f"{IMAGE_REGISTRY}/{IMAGE_NAME}:{VIME_IMAGE_TAG}"
return DEFAULT_CI_IMAGE

@Meihan-chen Meihan-chen Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Selecting only image-build uploads zero test steps, so the build can pass without testing the image. Could we require include smk by default?

@yzeyu71 yzeyu71 Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Modified npu-suites.py will determine whether image-build is selected to add SMK tests by default.


def selected_suites() -> list:
values = _read_suite_values()
unknown = [v for v in values if v and v not in SUITES and v != "image-build"]
if unknown:
raise SystemExit(f"unknown suite(s) {unknown}; expected {sorted(SUITES)}")
if "image-build" in values:
# image-build auto-includes smk tests
values.append("smk")
return [s for s in SUITES if s in values]


Expand Down Expand Up @@ -81,13 +97,14 @@ def npu_step(suite: str, test_name: str, resource_class: str, extra_args: str, e
label = f":fire: {suite}: {test_name}{' ' + extra_args if extra_args else ''}"
step = {
"label": label,
"depends_on": "image-build-npu",
"command": command,
"agents": {
"queue": NPU_QUEUE,
"resource_class": resource_class,
},
"timeout_in_minutes": 180,
"image": CI_IMAGE,
"image": _ci_image(),
"plugins": [
{
"kubernetes": {
Expand All @@ -104,7 +121,12 @@ def npu_step(suite: str, test_name: str, resource_class: str, extra_args: str, e

def main() -> None:
steps = [npu_step(suite, *entry) for suite in selected_suites() for entry in SUITES[suite]]
print(json.dumps({"steps": steps}, indent=2))
json_str = json.dumps({"steps": steps}, indent=2)

print("--- Generated Pipeline JSON:", file=sys.stderr)
print(json_str, file=sys.stderr)

print(json_str)


if __name__ == "__main__":
Expand Down
89 changes: 89 additions & 0 deletions .buildkite/pipeline-npu-image.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
steps:
- label: ":docker: Build and Push NPU Test Image - A3"
key: image-build-npu
depends_on:
- pre-commit-npu
- npu-gate
timeout_in_minutes: 240
skip: __SKIP_IMAGE_BUILD__
agents:
queue: "ascend-a3"
resource_class: "npu-2"
plugins:
- kubernetes:
metadata:
annotations:
vault.hashicorp.com/agent-init-first: "true"
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-inject-perms-ca.pem: "0400"
vault.hashicorp.com/agent-inject-perms-cert.pem: "0400"
vault.hashicorp.com/agent-inject-perms-config.json: "0400"
vault.hashicorp.com/agent-inject-perms-key.pem: "0400"
vault.hashicorp.com/agent-inject-secret-ca.pem: internal/data/ascend/buildkitd
vault.hashicorp.com/agent-inject-secret-cert.pem: internal/data/ascend/buildkitd
vault.hashicorp.com/agent-inject-secret-config.json: internal/data/ascend/buildkitd
vault.hashicorp.com/agent-inject-secret-key.pem: internal/data/ascend/buildkitd
vault.hashicorp.com/agent-inject-template-ca.pem: "{{- with secret \"internal/data/ascend/buildkitd\" -}}\n{{ .Data.data.RootCA }}\n{{- end }}"
vault.hashicorp.com/agent-inject-template-cert.pem: "{{- with secret \"internal/data/ascend/buildkitd\" -}}\n{{ .Data.data.ClientCaCert }}\n{{- end }}"
vault.hashicorp.com/agent-inject-template-config.json: "{{- with secret \"internal/data/ascend/buildkitd\" -}}\n{{ .Data.data.dockerConfig }}\n{{- end }}"
vault.hashicorp.com/agent-inject-template-key.pem: "{{- with secret \"internal/data/ascend/buildkitd\" -}}\n{{ .Data.data.ClientCaKey }}\n{{- end }}"
vault.hashicorp.com/agent-pre-populate-only: "true"
vault.hashicorp.com/agent-run-as-group: "1000"
vault.hashicorp.com/agent-run-as-user: "1000"
vault.hashicorp.com/agent-service-account-token-volume-name: token-vol
vault.hashicorp.com/role: ascend-gha-runners
vault.hashicorp.com/secret-volume-path: /home/user/.docker/
vault.hashicorp.com/tls-skip-verify: "true"
podSpecPatch:
volumes:
- name: token-vol
projected:
defaultMode: 420
sources:
- serviceAccountToken:
audience: api
expirationSeconds: 600
path: token
env:
VIME_IMAGE_TAG: "${BUILDKITE_COMMIT}"
IMAGE_NAME: "vime-ci-npu"
IMAGE_REGISTRY: "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry"
BUILDKITD_ADDR: "tcp://buildkitd-service.buildkitd:1234"
command: |
set -ex
echo "--- Building and pushing NPU Test Image"
echo "Image: $${IMAGE_REGISTRY}/$${IMAGE_NAME}:$${VIME_IMAGE_TAG}"
echo "buildkitd address: $${BUILDKITD_ADDR}"

if ! command -v buildctl &> /dev/null; then
echo "Installing buildctl..."
mkdir -p /tmp/buildkit
BUILDKIT_VERSION="v0.29.0"
wget -q "https://gh-proxy.test.osinfra.cn/https://github.com/moby/buildkit/releases/download/$${BUILDKIT_VERSION}/buildkit-$${BUILDKIT_VERSION}.linux-arm64.tar.gz" -O /tmp/buildkit.tar.gz
tar -xzf /tmp/buildkit.tar.gz -C /tmp/buildkit
cp /tmp/buildkit/bin/buildctl /usr/local/bin/
fi

sed -i '/^RUN git config --global http.sslVerify false/i RUN git config --global url."https://gh-proxy.test.osinfra.cn/https://github.com/".insteadOf "https://github.com/"' docker/Dockerfile.npu
sed -i '/^# syntax=docker\/dockerfile:1\.7$$/d' docker/Dockerfile.npu

export DOCKER_CONFIG=/home/user/.docker
buildctl \
--addr="$${BUILDKITD_ADDR}" \
--tlscacert=/home/user/.docker/ca.pem \
--tlscert=/home/user/.docker/cert.pem \
--tlskey=/home/user/.docker/key.pem \
build \
--frontend dockerfile.v0 \
--local context=. \
--local dockerfile=./docker \
--opt filename=Dockerfile.npu \
--opt build-arg:BASE_IMAGE=swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/vllm-ascend/vllm-ascend \
--opt build-arg:APTMIRROR=http://cache-service.nginx-pypi-cache.svc.cluster.local:8081 \
--opt build-arg:PIP_INDEX_URL=http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple \
--secret id=dockerconfig,src=/home/user/.docker/config.json \
--output type=image,name=$${IMAGE_REGISTRY}/$${IMAGE_NAME}:$${VIME_IMAGE_TAG},push=true \
--progress=plain

echo "--- Image pushed successfully"
echo "$${IMAGE_REGISTRY}/$${IMAGE_NAME}:$${VIME_IMAGE_TAG}"
109 changes: 17 additions & 92 deletions .buildkite/pipeline-npu.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,109 +45,34 @@ steps:
multiple: true
required: true
options:
- label: "run-npu-ci-image-build (test new image, auto include smk)"
value: image-build
- label: "run-npu-ci-smk"
value: smk
- label: "run-npu-ci-nightly"
value: nightly

# - label: ":buildkit: Build and Push NPU Test Image - A3"
# key: image-build-npu
# depends_on: pre-commit-npu
# if: __IMAGE_BUILD_IF__ == true
# timeout_in_minutes: 240
# agents:
# queue: "ascend-a3"
# resource_class: "npu-2"
# plugins:
# - kubernetes:
# metadata:
# annotations:
# vault.hashicorp.com/agent-init-first: "true"
# vault.hashicorp.com/agent-inject: "true"
# vault.hashicorp.com/agent-inject-perms-ca.pem: "0400"
# vault.hashicorp.com/agent-inject-perms-cert.pem: "0400"
# vault.hashicorp.com/agent-inject-perms-config.json: "0400"
# vault.hashicorp.com/agent-inject-perms-key.pem: "0400"
# vault.hashicorp.com/agent-inject-secret-ca.pem: internal/data/ascend/buildkitd
# vault.hashicorp.com/agent-inject-secret-cert.pem: internal/data/ascend/buildkitd
# vault.hashicorp.com/agent-inject-secret-config.json: internal/data/ascend/buildkitd
# vault.hashicorp.com/agent-inject-secret-key.pem: internal/data/ascend/buildkitd
# vault.hashicorp.com/agent-inject-template-ca.pem: "{{- with secret \"internal/data/ascend/buildkitd\" -}}\n{{ .Data.data.RootCA }}\n{{- end }}"
# vault.hashicorp.com/agent-inject-template-cert.pem: "{{- with secret \"internal/data/ascend/buildkitd\" -}}\n{{ .Data.data.ClientCaCert }}\n{{- end }}"
# vault.hashicorp.com/agent-inject-template-config.json: "{{- with secret \"internal/data/ascend/buildkitd\" -}}\n{{ .Data.data.dockerConfig }}\n{{- end }}"
# vault.hashicorp.com/agent-inject-template-key.pem: "{{- with secret \"internal/data/ascend/buildkitd\" -}}\n{{ .Data.data.ClientCaKey }}\n{{- end }}"
# vault.hashicorp.com/agent-pre-populate-only: "true"
# vault.hashicorp.com/agent-run-as-group: "1000"
# vault.hashicorp.com/agent-run-as-user: "1000"
# vault.hashicorp.com/agent-service-account-token-volume-name: token-vol
# vault.hashicorp.com/role: ascend-gha-runners
# vault.hashicorp.com/secret-volume-path: /home/user/.docker/
# vault.hashicorp.com/tls-skip-verify: "true"
# podSpecPatch:
# volumes:
# - name: token-vol
# projected:
# defaultMode: 420
# sources:
# - serviceAccountToken:
# audience: api
# expirationSeconds: 600
# path: token
# env:
# VIME_IMAGE_TAG: "${BUILDKITE_COMMIT}"
# IMAGE_NAME: "vime-ci-npu"
# IMAGE_REGISTRY: "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry"
# BUILDKITD_ADDR: "tcp://buildkitd-service.buildkitd:1234"
# command: |
# set -ex
#
# echo "--- Building and pushing NPU Test Image"
# echo "Image: $${IMAGE_REGISTRY}/$${IMAGE_NAME}:$${VIME_IMAGE_TAG}"
# echo "buildkitd address: $${BUILDKITD_ADDR}"
#
# if ! command -v buildctl &> /dev/null; then
# echo "Installing buildctl..."
# mkdir -p /tmp/buildkit
# BUILDKIT_VERSION="v0.29.0"
# wget -q "https://gh-proxy.test.osinfra.cn/https://github.com/moby/buildkit/releases/download/$${BUILDKIT_VERSION}/buildkit-$${BUILDKIT_VERSION}.linux-arm64.tar.gz" -O /tmp/buildkit.tar.gz
# tar -xzf /tmp/buildkit.tar.gz -C /tmp/buildkit
# cp /tmp/buildkit/bin/buildctl /usr/local/bin/
# fi
#
# sed -i '/^RUN git config --global http.sslVerify false/i RUN git config --global url."https://gh-proxy.test.osinfra.cn/https://github.com/".insteadOf "https://github.com/"' docker/Dockerfile.npu
# sed -i '/^# syntax=docker\/dockerfile:1\.7$$/d' docker/Dockerfile.npu
#
# export DOCKER_CONFIG=/home/user/.docker
# buildctl \
# --addr="$${BUILDKITD_ADDR}" \
# --tlscacert=/home/user/.docker/ca.pem \
# --tlscert=/home/user/.docker/cert.pem \
# --tlskey=/home/user/.docker/key.pem \
# build \
# --frontend dockerfile.v0 \
# --local context=. \
# --local dockerfile=./docker \
# --opt filename=Dockerfile.npu \
# --opt build-arg:APTMIRROR=http://cache-service.nginx-pypi-cache.svc.cluster.local:8081 \
# --opt build-arg:PIP_INDEX_URL=http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple \
# --secret id=dockerconfig,src=/home/user/.docker/config.json \
# --output type=image,name=$${IMAGE_REGISTRY}/$${IMAGE_NAME}:$${VIME_IMAGE_TAG},push=true \
# --progress=plain
#
# echo "--- Image pushed successfully"
# echo "$${IMAGE_REGISTRY}/$${IMAGE_NAME}:$${VIME_IMAGE_TAG}"

- label: ":pipeline: upload NPU suites"
key: upload-npu-suites
depends_on:
- npu-gate
Comment on lines +57 to +58

@gemini-code-assist gemini-code-assist Bot Jul 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

On scheduled builds (nightly runs), the npu-gate block step is skipped because of its if: build.source != "schedule" condition. Since image-build-npu depends on npu-gate directly, Buildkite will automatically skip image-build-npu whenever npu-gate is skipped. This breaks the scheduled nightly builds which are intended to always build the image.

To fix this, define the dependency on npu-gate with allow_failure: true using Buildkite's object-based dependency syntax. This allows the step to run when npu-gate is skipped.

        depends_on:
          - step: pre-commit-npu
          - step: npu-gate
            allow_failure: true

- pre-commit-npu
agents:
queue: "ascend-a3"
resource_class: "npu-2"
timeout_in_minutes: 180
command: |
export IMAGE_BUILD="quay.io/ascend/vime:0.3.0-a3-vllm0.22.1rc1"
echo "IMAGE_BUILD: $${IMAGE_BUILD}"
python .buildkite/npu_suites.py | buildkite-agent pipeline upload

NPU_SUITES=$$(buildkite-agent meta-data get "npu-suites" --default "")

@CalvinXKY CalvinXKY Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. image-build-npu only checks block meta-data npu-suites, while _ci_image() prefers env NPU_SUITES. On schedule (env-only, no block), NPU_SUITES=image-build,... will point tests at $COMMIT without building that tag.
    use one shared condition for build + image selection (env and/or meta-data, and ideally build.source == "schedule") ?

  2. Update the README/header which still describe the old auto-build behavior.

@yzeyu71 yzeyu71 Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Modified according to the recommendation. Check build.source == "schedule" to determine if triggered on a scheduled time. If yes, execute the build image. Readme has been updated.

if [[ "$$BUILDKITE_SOURCE" == "schedule" ]]; then
echo "Scheduled build — building the image."
echo "BUILDKITE_SOURCE: $$BUILDKITE_SOURCE"
SKIP_IMAGE_BUILD=false
elif [[ "$$NPU_SUITES" == *"image-build"* ]]; then
echo "image-build selected — building the image."
echo "NPU_SUITES: $$NPU_SUITES"
SKIP_IMAGE_BUILD=false
else
echo "Skipping image build because image-build is not present in the npu-suites."
SKIP_IMAGE_BUILD=true
fi
sed -e "s/__SKIP_IMAGE_BUILD__/$${SKIP_IMAGE_BUILD}/g" .buildkite/pipeline-npu-image.yaml | buildkite-agent pipeline upload
python .buildkite/npu_suites.py | buildkite-agent pipeline upload
Loading