Skip to content

Catch SIGINT/SIGTERM/SIGHUP in start-runner.sh so the VM always powers off#133

Open
ophirt wants to merge 1 commit into
framna-dk:mainfrom
ophirt:fix-start-runner-trap-signals
Open

Catch SIGINT/SIGTERM/SIGHUP in start-runner.sh so the VM always powers off#133
ophirt wants to merge 1 commit into
framna-dk:mainfrom
ophirt:fix-start-runner-trap-signals

Conversation

@ophirt

@ophirt ophirt commented Jun 2, 2026

Copy link
Copy Markdown

Problem

start-runner.sh powers the VM off when the runner exits, so Tartelet can recreate a fresh ephemeral VM:

function onexit {
  sudo shutdown -h now
}
trap onexit EXIT

The script is launched inside a GUI Terminal session (open -a Terminal ~/start-runner.sh). If that session is torn down — or the runner process is otherwise terminated by a signal — the shell receives SIGHUP/SIGTERM, and a zsh EXIT trap does not run when the shell is killed by a signal. So onexit never fires, shutdown -h now never runs, and the VM stays powered on with no runner.

Because Tartelet only provisions a replacement VM after it observes the current one stop, such a VM becomes a permanent "offline zombie": it shows offline in GitHub's runners page indefinitely and is never recycled, silently shrinking the pool while healthy VMs absorb all jobs.

We hit this in production: a runner died ~3s into boot with an I/O error on its console (the Terminal session went away before it reached "Listening for Jobs"), start-runner.sh got SIGTERM, the EXIT trap was skipped, and the VM never powered off.

Fix

Trap the termination signals too, so onexit (and therefore shutdown -h now) runs whether the shell exits normally or is killed by a signal:

-trap onexit EXIT
+trap onexit EXIT INT TERM HUP

onexit only runs sudo shutdown -h now, so invoking it on a signal (and again on the subsequent EXIT) is idempotent — the machine is halting either way.

Notes

  • One-line change inside the generated start-runner.sh heredoc; no Swift API or type changes.

🤖 Generated with Claude Code

@ophirt
ophirt requested a review from simonbs as a code owner June 2, 2026 14:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant