Skip to content

feat(deploy): runner artifact build + SSM rollout (dual-track sh/mjs, supersedes #901)#911

Draft
law-chain-hot wants to merge 10 commits into
mainfrom
feat/ops-runner-deploy-scripts
Draft

feat(deploy): runner artifact build + SSM rollout (dual-track sh/mjs, supersedes #901)#911
law-chain-hot wants to merge 10 commits into
mainfrom
feat/ops-runner-deploy-scripts

Conversation

@law-chain-hot

Copy link
Copy Markdown
Contributor

Summary

Product-side (boxlite) half of the Ops Console runner build + rollout capability. Supersedes #901 (older draft on the smoke branch, which self-flagged a stale base and a START_BOX guest-hash bug).

Contents:

  • Runner deploy scripts, dual-track: scripts/deploy/{build-runner-binary,runner-update-binary}.{sh,mjs} — bash is the authority (adopted from ci(deploy): isolate non-release runner runtime caches #724's runtime-payload rewrite), .mjs twins added for the single-language orchestration direction. SSM remote payload is parsed from the .sh at runtime (single payload authority, cannot drift).
  • Parity harness: scripts/deploy/tests/parity/* — PATH-shim fixture that byte-compares the two implementations' command sequence, artifacts, and SSM payload.
  • SST/env plumbing: apps/infra/scripts/sst-with-cloudflare.mjs (+ self-host env sync from SSM), apps/infra/sst.config.ts, and the four service Dockerfile bumps that the deploy path needs.

Coordinated release

This PR pairs with the Ops Console repo's feat/ops-hardening branch (orchestration .mjs, per-job worktree isolation, evidence-to-S3, post-deploy smoke, promote-by-digest). Merge order: this PR to main first, then repoint Ops Console OPS_PRODUCT_REF/OPS_RUNNER_TOOL_REF from codex/ops-service-deploy-integration to main, then the Ops Console change.

Verification

  • parity harness PASS (SSM payload byte-identical between .sh/.mjs)
  • dev runner i-0ee6bc569d5a1e9ef: build → artifact → S3 → SSM rollout → hot-adopt → health verified, both bash and js implementations, multiple runs
  • cleanly mergeable to main (no conflicts)

Draft because

  • pending final rebase confirmation + coordinated merge with the Ops Console PR
  • prod promote / cross-region ECR replication are separate, operator-in-person steps

Fleet re-runs and canary-then-fleet overlaps hit runners that already run the
exact incoming bits. Compare sha256 of the incoming binary against the current
symlink target before creating a release dir: identical -> log + exit 0.
Hash comparison (not version label) keeps the no-op safe when a dev artifact
reuses a version name.
The 'deferred' branch of verify_embedded_runtime_hash passed silently when no
runtime cache dir existed yet — a runner whose embedded guest key could never
materialize installed cleanly and then failed every box start with a hash
mismatch. Now: wait up to 20s for the runner's self-extraction, verify the
guest hash, and return FATAL on a miss so restart_with_target routes to the
existing auto-rollback branch. Better no install than a runner that cannot
start boxes.
…oad)

Root fix for 'Guest binary hash mismatch' on dev artifacts: the build script
now locates the embedded runtime dir BY the fresh guest's sha256 and packs it
into the tarball as boxlite-runtime.tar.gz, so the shipped runtime always
matches the compile-time key. Supersedes the interim same-binary skip and
guest-extraction fuse on these files (to be re-evaluated on top of the new
install flow if still needed).
PATH-shim fixture records the semantic command sequence (cargo/go/make/git/
tar/aws/curl) each implementation issues and byte-compares artifacts,
sidecars, and the base64 SSM payload. Red until the .mjs twins land.
…rity)

Line-for-line port of build-runner-binary.sh: same CLI, same artifact names,
same sidecars, same command sequence (parity-tested). Selected at the
orchestration layer via OPS_RUNNER_IMPL; bash stays the default.
…ority)

Ports the local orchestrator half (region/instance resolution, artifact
staging, S3 presign, SSM dispatch + poll). The remote upgrade payload is not
duplicated: the .mjs parses the heredoc out of the sibling .sh at runtime and
expands it with bash here-document semantics, so the two halves cannot drift.
Parity test proves the SSM payload byte-identical.
@coderabbitai

coderabbitai Bot commented Jul 3, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: e4c94d81-f6be-4402-9315-2f0a3b2512e1

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/ops-runner-deploy-scripts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Add BuildKit cache mounts (yarn Berry global cache, Go build + module
caches) to the api/otel-collector/proxy/ssh-gateway/runner image builds so
deps and compiled objects persist across builds instead of recompiling from
cold every time. Also add the dockerfile:1 syntax header to the Go images.
Layer ordering, build commands, and outputs are unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant