chore(ci): retire the Ada Cloud Run deploy workflow — 6/6 runs failed, and nothing calls a deployed Ada - #411
Merged
Merged
Conversation
…, and nothing calls a deployed Ada `.github/workflows/deploy-ada.yml` has failed every run since it was added in #236 — six runs between 2026-05-04 and 2026-07-27, all dying on the same line: gcloud builds submit ... ERROR: (gcloud.builds.submit) PERMISSION_DENIED: The caller does not have permission. The cause is narrow. The WIF service account was never granted a Cloud Build role. `deploy-gke.yml` uses the same `secrets.WIF_SA` and works because it never calls Cloud Build at all — it runs docker build/push on the runner, then kubectl. Auth was fine the whole time; the source tarball uploads successfully on every run and only `builds.create` is denied. The IAM was fixable. It wasn't worth fixing, because a green build would have deployed a service nothing can reach: - ingress=internal with no LB (agent/cloudbuild.yaml says as much), and no in-VPC caller exists - no Ada manifests in k8s/, no run.app or ada.ai-identity.co reference anywhere in the tree - day-to-day Ada runs locally via agent/launcher So the workflow fired on every push touching agent/**, went red, and nothing downstream noticed for three months — which is exactly how a real failure hides. Removing it is the honest state: Ada's deploy is manual. The deploy artifacts stay. agent/Dockerfile and agent/cloudbuild.yaml are the manual path documented in docs/ada-deployment.md, and they remain the on-ramp if Ada ever needs hosting. The runbook gains a status section recording why CI was retired and the two things still unverified before any first real deploy: the Cloud Build IAM grants, and a secret-name mismatch (the runbook creates ada-admin-key / ada-runtime-key; cloudbuild.yaml defaults to ai-identity-ADA_ADMIN_KEY / ai-identity-ADA_RUNTIME_KEY — one is wrong, and the build never got far enough to find out). Unrelated to the 2026-07-27 history rewrite. That force-push merely triggered another run: 413ea00 touches no files under agent/, but a rewrite makes GitHub treat the whole tree as changed. The path filter itself was sound — it correctly excluded the launcher-only change in #408. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Companion to the deletion in this PR. Filed as a `_CI:_` entry under Changed, matching how the other CI/deploy changes in [Unreleased] are recorded — the bold multi-sentence format is used for product and security changes, not internal tooling. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
.github/workflows/deploy-ada.ymlhas never succeeded. Six runs since it was added in #236 (2026-05-04 → 2026-07-27), every one dying on the same line:Why it failed
Narrow cause: the WIF service account was never granted a Cloud Build role.
deploy-gke.ymluses the samesecrets.WIF_SAand works — because it never calls Cloud Build at all. It runsdocker build/docker pushon the runner, then kubectl. Ada's workflow is the only thing in the repo that callsgcloud builds submit, so it's the only thing that needed IAM nobody ever requested.Auth itself was never the problem: the source tarball uploads successfully on every run, so the SA has GCS write. Only
cloudbuild.builds.createis denied.Why retire instead of granting the IAM
The IAM was fixable. It wasn't worth fixing, because a green build would deploy a service nothing can reach:
--ingress=internalwith no load balancer —agent/cloudbuild.yamlsays so itself ("no external HTTPS LB / serverless NEG fronting Ada (none was ever built)"), and no in-VPC caller exists.k8s/; norun.apporada.ai-identity.coreference anywhere in the tree. The only Ada reference outsideagent/ismarketing/sales/decks/refresh_ada_evidence.py, which queries the AI Identity API for audit records — it never calls the Ada service.agent/launcher, which execsserve.pyon:8000from a hardcoded local path.So the workflow fired on every push touching
agent/**, went red, and nothing downstream noticed for three months. That's the real cost: a permanently red workflow trains everyone to ignore failures on this repo, which is how a genuine failure hides.What this changes
.github/workflows/deploy-ada.yml.agent/Dockerfileandagent/cloudbuild.yaml— these are not dead. They're the manual deploy path documented in the runbook, and the on-ramp if Ada ever needs hosting.cloudbuild.yaml's header now says the deploy is manual and points at the runbook.docs/ada-deployment.mdwith a status section, so nobody re-adds the workflow without knowing what happened.Two things still unverified before any first real deploy
Recorded in the runbook rather than silently left as landmines. Neither was ever exercised end-to-end:
roles/cloudbuild.builds.editor, plus the Cloud Build SA grants already in Prerequisites §5.ada-admin-key/ada-runtime-key, butagent/cloudbuild.yamldefaults toai-identity-ADA_ADMIN_KEY/ai-identity-ADA_RUNTIME_KEY. One is wrong. The build never got far enough to find out — fixing the IAM alone would just have surfaced this next. This needs agcloud secrets listto settle; I couldn't check (local gcloud creds need reauth).Not related to the history rewrite
The 2026-07-27 force-push only triggered another run.
413ea008touches no files underagent/, but a history rewrite makes GitHub treat the whole tree as changed. The path filter itself was sound — it correctly excluded the launcher-only change in #408.🤖 Generated with Claude Code