Skip to content

Capture cpu profile for keynote bench in nightly job #5163

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
joshua-spacetime wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Capture cpu profile for keynote bench in nightly job #5163

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
be64920
Capture profile for keynote bench in nightly job
joshua-spacetime Jun 1, 2026
e6e1c18
testing
joshua-spacetime Jun 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 52 additions & 3 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -273,8 +273,11 @@ jobs:
runs-on: spacetimedb-benchmark-runner
timeout-minutes: 60
env:
CARGO_TARGET_DIR: ${{ github.workspace }}/target
RUST_BACKTRACE: full
CARGO_TARGET_DIR: ${{ github.workspace }}/target
CARGO_PROFILE_RELEASE_DEBUG: line-tables-only
Copy link
Copy Markdown
Contributor

@jdetter jdetter Jun 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may cause cargo cache misses with the new vm caching change but that's more of a me problem for later

KEYNOTE_PROFILE_DIR: ${{ github.workspace }}/keynote-profile
STDB_V8_FLAGS: "--perf-basic-prof --perf-basic-prof-only-functions --interpreted-frames-native-stack"
steps:
- name: Find Git ref
env:
Expand Down Expand Up @@ -306,7 +309,14 @@ jobs:
prefix-key: v1

- name: Build keynote benchmark binaries
run: cargo build --release -p spacetimedb-cli -p spacetimedb-standalone
run: >
cargo
--config 'build.rustflags = ["-C", "force-frame-pointers=yes"]'
build
--release
-p spacetimedb-cli
-p spacetimedb-standalone
--features spacetimedb-standalone/perfmap
Copy link
Copy Markdown
Contributor

@jdetter jdetter Jun 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will also probably cause a cache miss - if this workflow ends up being too slow I can bake a new cache with all of these options enabled so we're more likely to get a compilation cache hit.

Copy link
Copy Markdown
Contributor Author

@joshua-spacetime joshua-spacetime Jun 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, I think --features spacetimedb-standalone/perfmap should be the default for all of our release builds.


# Node 24 is the current Active LTS line.
- name: Set up Node.js
Expand All @@ -323,7 +333,46 @@ jobs:
working-directory: crates/bindings-typescript

- name: Run keynote-2 benchmark regression check
run: cargo ci keynote-bench
id: keynote_bench
run: |
mkdir -p "$KEYNOTE_PROFILE_DIR"
set +e
samply record \
Comment thread
joshua-spacetime marked this conversation as resolved.
--save-only \
--unstable-presymbolicate \
--profile-name keynote-bench \
-o "$KEYNOTE_PROFILE_DIR/profile.json.gz" \
-- cargo ci keynote-bench
status=$?
ls -lh "$KEYNOTE_PROFILE_DIR" || true
exit "$status"

- name: Upload keynote benchmark profile
if: always()
# if: failure() && steps.keynote_bench.outcome == 'failure'
uses: actions/upload-artifact@v4
with:
name: keynote-bench-profile
path: ${{ env.KEYNOTE_PROFILE_DIR }}
if-no-files-found: warn
retention-days: 14

- name: Report keynote benchmark profile instructions
if: always()
# if: failure() && steps.keynote_bench.outcome == 'failure'
run: |
{
echo "## Keynote benchmark profile"
echo
echo "The keynote benchmark failed after recording a Samply/Firefox profile."
echo
echo "1. Open this run's Summary page: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
echo "2. Download the artifact named \`keynote-bench-profile\`."
echo "3. Open <https://profiler.firefox.com/> and load \`profile.json.gz\`."
echo "4. Compare hot frames against a recent nightly \`keynote-bench-nightly-profile\` artifact."
} >> "$GITHUB_STEP_SUMMARY"

echo "::error title=Keynote benchmark profile uploaded::Download the keynote-bench-profile artifact from this run and open profile.json.gz in https://profiler.firefox.com/."

lints:
name: Lints
Expand Down
108 changes: 108 additions & 0 deletions .github/workflows/keynote-bench-nightly.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
name: Keynote Bench Nightly

on:
schedule:
# Nightly at 3 AM UTC.
- cron: '0 3 * * *'
Copy link
Copy Markdown
Contributor

@jdetter jdetter Jun 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer this to run more around 8am UTC - when we do maintenance on the runners we typically like to do it in the middle of the night when nobody is working to reduce the impact on productivity. If you keep this as 3AM UTC you will just run the risk of it getting cancelled sometimes.

workflow_dispatch:
inputs:
ref:
description: "Git ref to benchmark"
required: false
default: master

permissions:
contents: read

concurrency:
group: keynote-bench-nightly
cancel-in-progress: true

jobs:
keynote_bench:
name: Keynote Bench
runs-on: spacetimedb-benchmark-runner
timeout-minutes: 60
env:
CARGO_TARGET_DIR: ${{ github.workspace }}/target
CARGO_PROFILE_RELEASE_DEBUG: line-tables-only
KEYNOTE_PROFILE_DIR: ${{ github.workspace }}/keynote-profile
STDB_V8_FLAGS: "--perf-basic-prof --perf-basic-prof-only-functions --interpreted-frames-native-stack"
steps:
- name: Checkout sources
uses: actions/checkout@v4
with:
ref: ${{ github.event.inputs.ref || 'master' }}

- uses: dsherret/rust-toolchain-file@v1
- name: Set default rust toolchain
run: rustup default $(rustup show active-toolchain | cut -d' ' -f1)

- name: Cache Rust dependencies
uses: Swatinem/rust-cache@v2
with:
workspaces: ${{ github.workspace }}
shared-key: spacetimedb
save-if: false
prefix-key: v1

- name: Build keynote benchmark binaries
run: >
cargo
--config 'build.rustflags = ["-C", "force-frame-pointers=yes"]'
build
--release
-p spacetimedb-cli
-p spacetimedb-standalone
--features spacetimedb-standalone/perfmap

# Node 24 is the current Active LTS line.
- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: 24

- uses: ./.github/actions/setup-pnpm
with:
run_install: true

- name: Build TypeScript SDK
run: pnpm build
working-directory: crates/bindings-typescript

- name: Run keynote-2 benchmark regression check
id: keynote_bench
run: |
mkdir -p "$KEYNOTE_PROFILE_DIR"
samply record \
--save-only \
--unstable-presymbolicate \
--profile-name keynote-bench-nightly \
-o "$KEYNOTE_PROFILE_DIR/profile.json.gz" \
-- cargo ci keynote-bench
ls -lh "$KEYNOTE_PROFILE_DIR" || true

- name: Upload keynote benchmark profile
if: always()
uses: actions/upload-artifact@v4
with:
name: keynote-bench-nightly-profile
path: ${{ env.KEYNOTE_PROFILE_DIR }}
if-no-files-found: warn
retention-days: 90
Copy link
Copy Markdown
Contributor Author

@joshua-spacetime joshua-spacetime Jun 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note for @jdetter: I'm not sure if or how much of an impact this will have on our github actions spend. Happy to reconfigure this. I would just like some level of cpu profile history for diagnostic purposes.

Copy link
Copy Markdown
Contributor

@jdetter jdetter Jun 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be fine for now, if the spend here becomes too large then we can always move these to an S3 bucket or something

Copy link
Copy Markdown
Contributor

@jdetter jdetter Jun 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How large are these artifacts?

Copy link
Copy Markdown
Contributor Author

@joshua-spacetime joshua-spacetime Jun 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

~4MB each.


- name: Report keynote benchmark profile instructions
if: failure() && steps.keynote_bench.outcome == 'failure'
run: |
{
echo "## Keynote benchmark profile"
echo
echo "The nightly keynote benchmark failed after recording a Samply/Firefox profile."
echo
echo "1. Open this run's Summary page: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
echo "2. Download the artifact named \`keynote-bench-nightly-profile\`."
echo "3. Open <https://profiler.firefox.com/> and load \`profile.json.gz\`."
echo "4. Compare hot frames against earlier nightly \`keynote-bench-nightly-profile\` artifacts."
} >> "$GITHUB_STEP_SUMMARY"

echo "::error title=Keynote benchmark profile uploaded::Download the keynote-bench-nightly-profile artifact from this run and open profile.json.gz in https://profiler.firefox.com/."
Loading