Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions .clang-format
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Copyright 2026 Google LLC.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# This file configures the clang-format tool for C++ code. It enforces the
# Google C++ Style Guide rules across all modifications:
# https://google.github.io/styleguide/cppguide.html.
#
# Details on configuration formatting and style options can be found at:
# https://clang.llvm.org/docs/ClangFormatStyleOptions.html

---
Language: Cpp
BasedOnStyle: Google
...
64 changes: 64 additions & 0 deletions .github/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# tpu-raiden CI/CD

GitHub Actions CI for tpu-raiden, modeled on
[`torch_tpu`](https://github.com/google-pytorch/torch_tpu)'s setup. The existing
Kokoro jobs under [`kokoro/`](../kokoro) are kept and run alongside these.

## Runners

- **CPU** — `linux-x86-n2-32` runs the presubmit + nightly build/unit-test jobs.
- **TPU v5e** — `linux-x86-ct5lp-224-8tpu` runs the device tests (`device_test.yml`).
v5e is used because the v7 (`linux-x86-tpu7x-224-4tpu`) scale-set is not
available to this repo.

Both are self-hosted scale-sets and can take several minutes to cold-start a
runner, so a freshly triggered job may sit `queued` for a while before a runner
picks it up — that is normal, not a failure. CPU and TPU jobs are split across
separate workflows so a slow TPU runner never blocks the CPU gate.

## Workflows

| Workflow | Trigger | What it does |
| --- | --- | --- |
| [`lint.yml`](workflows/lint.yml) | PR | Apache license headers (`addlicense`) + commit-message check. GitHub-hosted. |
| [`clang_format.yml`](workflows/clang_format.yml) | PR | Google C++ style on changed `.cc/.h` via pinned `clang-format==18`. GitHub-hosted. |
| [`presubmit.yml`](workflows/presubmit.yml) | PR | Build JAX extension + OSS-safe CPU unit tests + import smoke, on `n2-32`. |
| [`nightly.yml`](workflows/nightly.yml) | push to `main` / daily cron 09:00 UTC / dispatch | CPU build + unit tests + `tpu_raiden_jax` wheel (twine-checked, uploaded as an artifact), file a tracking issue on failure, on `n2-32`. |
| [`device_test.yml`](workflows/device_test.yml) | PR / daily cron / dispatch | Build JAX extension, verify TPU visible, run device tests on v5e (`ct5lp`). |

Shared logic lives in [`ci/tools/`](../ci/tools): `install_clang18.sh` (the
clang-18 the build needs) and `bazel_test.sh` (bazel binary + dummy torch_tpu
override + flags, matching `build.sh`).

## Scope notes

- **JAX-only build.** The JAX path generates a dummy `torch_tpu` Bazel module
(same as `build.sh` / `kokoro/.../presubmit.sh`), so no secrets are needed. The
**Torch** extension/wheel additionally needs a `torch_tpu` checkout + a local
`torch` (deploy key) — out of scope here for now.
- **Test set.** `CPU_TEST_TARGETS` are device-free; `DEVICE_TEST_TARGETS` exercise
the v5e. Both are a curated OSS-loadable set. `//rpc/...` and
`//kv_cache/global_registry/...` (and the `kv_cache_store` tests that depend on
them) load Google-internal gRPC rules and are excluded by design — see
[`ci/wheel/BUILD.bazel`](../ci/wheel/BUILD.bazel). Expand the lists as coverage
grows; for multiple parallel device tests, add a `--run_under` accelerator-lock
helper.

## One-time setup to turn this on

1. **Runner** — `google/tpu-raiden` must be granted the
`linux-x86-ct5lp-224-8tpu` scale-set (confirmed picking up jobs).
2. **Bazel remote cache** — bucket `gs://tpu-raiden-bazel-cache` (already used by
Kokoro); the runner service account needs object read (and write for the
nightly's read-write cache).
3. **Labels** — create `ci:nightly-failed` for the failure-issue action.
`GITHUB_TOKEN` is provided automatically.

## Landing & running

`google/tpu-raiden` is a one-way Copybara mirror of an internal google3 repo
(commits carry `PiperOrigin-RevId:`), so these files land **internally** and sync
out — there is no external push/PR path for merges. Opening a PR still triggers
the workflows for validation. Once on `main`, presubmit fires on PRs and
`nightly.yml` can be kicked with `gh workflow run "CI - Nightly"` by anyone with
write access.
21 changes: 21 additions & 0 deletions .github/actionlint.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Copyright 2026 Google LLC.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Declares the self-hosted runner labels used by the workflows so actionlint
# does not flag them as unknown. CPU jobs run on n2-32; TPU device tests on the
# v5e scale-set (the v7/tpu7x scale-set is not available to this repo).
self-hosted-runner:
labels:
- linux-x86-n2-32 # CPU
- linux-x86-ct5lp-224-8tpu # TPU v5e, 8 chips
72 changes: 72 additions & 0 deletions .github/workflows/clang_format.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# Copyright 2026 Google LLC.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Enforces the Google C++ Style Guide on Pull Requests via clang-format, so
# unformatted C++ cannot be merged into main. Only files changed in the PR are
# checked, and the clang-format version is pinned to 18 to match the clang-18
# toolchain raiden builds with (see ci/build_wheel.sh).

name: Clang Format

on:
workflow_dispatch:
pull_request:
branches:
- main

permissions:
contents: read

# Cancel old jobs when a contributor pushes new commits to their PR.
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
cancel-in-progress: ${{ github.ref != 'main' }}

jobs:
run_clang_format:
name: Clang format check
runs-on: ubuntu-latest
steps:
- name: Check out PR HEAD
uses: actions/checkout@v5

- name: Fetch PR base commit
run: git fetch --no-tags --depth=1 origin ${{ github.event.pull_request.base.sha }}

# uv is a fast, single-binary replacement for pip/pipx/virtualenv. Modern
# runners enforce PEP 668 ("Externally Managed Environments"), which blocks
# global `pip install`; uvx runs the pinned clang-format wheel in an
# isolated sandbox without touching the host interpreter.
- name: Setup uv
uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
with:
enable-cache: true

- name: Run clang-format on changed C/C++ files
env:
BASE_SHA: ${{ github.event.pull_request.base.sha }}
run: |
# Collect C/C++ sources changed in this PR (added/copied/modified/renamed).
mapfile -t FILES < <(
git diff --name-only --diff-filter=ACMR "${BASE_SHA}" HEAD -- \
'*.c' '*.cc' '*.cpp' '*.cxx' '*.h' '*.hpp' '*.hh'
)
if [[ ${#FILES[@]} -eq 0 ]]; then
echo "No C/C++ files changed; nothing to check."
exit 0
fi
printf 'Checking:\n'; printf ' %s\n' "${FILES[@]}"
# clang-format 18.x matches the clang-18 build toolchain. --dry-run
# --Werror makes any required reformatting a hard failure.
uvx --from 'clang-format==18.1.8' clang-format --dry-run --Werror "${FILES[@]}"
77 changes: 77 additions & 0 deletions .github/workflows/device_test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# Copyright 2026 Google LLC.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# TPU device tests on the v5e runner (linux-x86-ct5lp-224-8tpu). Kept separate
# from the CPU presubmit/nightly so a slow-to-provision TPU runner never blocks
# the CPU gate. v5e is used because the v7 (tpu7x) scale-set is not available to
# this repo. Builds the JAX extension, confirms the TPU is visible, and runs the
# device tests.

name: CI - Device tests (v5e)

on:
workflow_dispatch:
pull_request:
branches:
- main
schedule:
- cron: "0 9 * * *" # 1 AM PST daily, alongside the nightly CPU build.

permissions:
contents: read
issues: write

concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
cancel-in-progress: ${{ github.ref != 'main' }}

env:
DEVICE_TEST_TARGETS: >-
//tpu_raiden/frameworks/jax:raw_transfer_test_gl

jobs:
device_tests:
name: "Device tests on v5e"
runs-on: linux-x86-ct5lp-224-8tpu
container:
image: "us-docker.pkg.dev/ml-oss-artifacts-published/ml-public-container/ml-build:latest"
options: --privileged
steps:
- uses: actions/checkout@v5
- name: Fix Git workspace ownership
run: git config --global --add safe.directory "$GITHUB_WORKSPACE"

- name: Install clang-18
run: ./ci/tools/install_clang18.sh

- name: Build JAX extension
run: ./build.sh jax

- name: Verify TPU is visible
run: |
python3 -c "import jax; d=jax.devices(); print('JAX devices:', d); assert any('TPU' in str(x).upper() for x in d), 'no TPU visible'; print('TPU visible OK')"

- name: Run device tests
env:
RAIDEN_REMOTE_CACHE: "https://storage.googleapis.com/tpu-raiden-bazel-cache"
RAIDEN_REMOTE_UPLOAD: "false"
run: ./ci/tools/bazel_test.sh jax -- ${DEVICE_TEST_TARGETS}

- name: Upload test logs on failure
uses: actions/upload-artifact@v4
if: failure()
with:
name: bazel-testlogs-device-v5e
path: bazel-testlogs/
retention-days: 7
72 changes: 72 additions & 0 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# Copyright 2026 Google LLC.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

name: "Lint checks"

on:
workflow_dispatch:
pull_request:
branches:
- main

permissions:
contents: read
pull-requests: read
issues: read

# Cancel old jobs when they are replaced by new runs.
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
# Don't cancel in-progress jobs for the main branch.
cancel-in-progress: ${{ github.ref != 'main' }}

jobs:
run_lint:
name: "Lint check"
runs-on: ubuntu-latest
steps:
- name: "Check out code"
uses: "actions/checkout@v5"
- name: "Set up Go"
uses: "actions/setup-go@v6"
with:
go-version: "1.24"
cache: false
- name: "Validate License Headers"
shell: "bash"
run: |
go run github.com/google/addlicense@v1.1.1 -check -s=only .
- name: "Validate Commit Messages"
if: (success() || failure()) && github.event_name == 'pull_request'
shell: "bash"
run: |
# Fix dubious ownership issue in Actions.
git config --global --add safe.directory '*'

# Get the exact number of commits in the PR to minimize git fetch depth.
PR_COMMITS="${{ github.event.pull_request.commits }}"
PR_COMMITS="${PR_COMMITS:-0}"

# Fetch the PR branch head and the base commit to establish the range.
git fetch --no-tags --depth=$((PR_COMMITS + 1)) origin "${{ github.event.pull_request.head.sha }}"
git fetch --no-tags --depth=1 origin "${{ github.event.pull_request.base.sha }}"

# Get commit messages between base and head of the pull request.
commit_msgs=$(git log "${{ github.event.pull_request.base.sha }}..${{ github.event.pull_request.head.sha }}" --format=%B)

if echo "$commit_msgs" | grep -qF "[NO DESCRIPTION SET]"; then
echo "::error::Found '[NO DESCRIPTION SET]' in one or more commit messages."
exit 1
fi
echo "Commit messages look good."
Loading
Loading