Skip to content

chore(ci): retire the Ada Cloud Run deploy workflow — 6/6 runs failed, and nothing calls a deployed Ada - #411

Merged
Levaj2000 merged 2 commits into
mainfrom
chore/retire-ada-cloudrun-deploy
Jul 27, 2026
Merged

chore(ci): retire the Ada Cloud Run deploy workflow — 6/6 runs failed, and nothing calls a deployed Ada#411
Levaj2000 merged 2 commits into
mainfrom
chore/retire-ada-cloudrun-deploy

Conversation

@Levaj2000

Copy link
Copy Markdown
Owner

.github/workflows/deploy-ada.yml has never succeeded. Six runs since it was added in #236 (2026-05-04 → 2026-07-27), every one dying on the same line:

gcloud builds submit --project=... --region=us-east1 --config=agent/cloudbuild.yaml
ERROR: (gcloud.builds.submit) PERMISSION_DENIED: The caller does not have permission.

Why it failed

Narrow cause: the WIF service account was never granted a Cloud Build role.

deploy-gke.yml uses the same secrets.WIF_SA and works — because it never calls Cloud Build at all. It runs docker build/docker push on the runner, then kubectl. Ada's workflow is the only thing in the repo that calls gcloud builds submit, so it's the only thing that needed IAM nobody ever requested.

Auth itself was never the problem: the source tarball uploads successfully on every run, so the SA has GCS write. Only cloudbuild.builds.create is denied.

Why retire instead of granting the IAM

The IAM was fixable. It wasn't worth fixing, because a green build would deploy a service nothing can reach:

  • --ingress=internal with no load balancer — agent/cloudbuild.yaml says so itself ("no external HTTPS LB / serverless NEG fronting Ada (none was ever built)"), and no in-VPC caller exists.
  • No Ada manifests in k8s/; no run.app or ada.ai-identity.co reference anywhere in the tree. The only Ada reference outside agent/ is marketing/sales/decks/refresh_ada_evidence.py, which queries the AI Identity API for audit records — it never calls the Ada service.
  • Day-to-day Ada runs locally via agent/launcher, which execs serve.py on :8000 from a hardcoded local path.

So the workflow fired on every push touching agent/**, went red, and nothing downstream noticed for three months. That's the real cost: a permanently red workflow trains everyone to ignore failures on this repo, which is how a genuine failure hides.

What this changes

  • Deletes .github/workflows/deploy-ada.yml.
  • Keeps agent/Dockerfile and agent/cloudbuild.yaml — these are not dead. They're the manual deploy path documented in the runbook, and the on-ramp if Ada ever needs hosting. cloudbuild.yaml's header now says the deploy is manual and points at the runbook.
  • Documents the retirement in docs/ada-deployment.md with a status section, so nobody re-adds the workflow without knowing what happened.

Two things still unverified before any first real deploy

Recorded in the runbook rather than silently left as landmines. Neither was ever exercised end-to-end:

  1. IAM — a deploying principal needs roles/cloudbuild.builds.editor, plus the Cloud Build SA grants already in Prerequisites §5.
  2. The secret names disagree. The runbook §4 creates ada-admin-key / ada-runtime-key, but agent/cloudbuild.yaml defaults to ai-identity-ADA_ADMIN_KEY / ai-identity-ADA_RUNTIME_KEY. One is wrong. The build never got far enough to find out — fixing the IAM alone would just have surfaced this next. This needs a gcloud secrets list to settle; I couldn't check (local gcloud creds need reauth).

Not related to the history rewrite

The 2026-07-27 force-push only triggered another run. 413ea008 touches no files under agent/, but a history rewrite makes GitHub treat the whole tree as changed. The path filter itself was sound — it correctly excluded the launcher-only change in #408.

🤖 Generated with Claude Code

…, and nothing calls a deployed Ada

`.github/workflows/deploy-ada.yml` has failed every run since it was added in
#236 — six runs between 2026-05-04 and 2026-07-27, all dying on the same line:

    gcloud builds submit ...
    ERROR: (gcloud.builds.submit) PERMISSION_DENIED: The caller does not have permission.

The cause is narrow. The WIF service account was never granted a Cloud Build
role. `deploy-gke.yml` uses the same `secrets.WIF_SA` and works because it
never calls Cloud Build at all — it runs docker build/push on the runner, then
kubectl. Auth was fine the whole time; the source tarball uploads successfully
on every run and only `builds.create` is denied.

The IAM was fixable. It wasn't worth fixing, because a green build would have
deployed a service nothing can reach:

  - ingress=internal with no LB (agent/cloudbuild.yaml says as much), and no
    in-VPC caller exists
  - no Ada manifests in k8s/, no run.app or ada.ai-identity.co reference
    anywhere in the tree
  - day-to-day Ada runs locally via agent/launcher

So the workflow fired on every push touching agent/**, went red, and nothing
downstream noticed for three months — which is exactly how a real failure
hides. Removing it is the honest state: Ada's deploy is manual.

The deploy artifacts stay. agent/Dockerfile and agent/cloudbuild.yaml are the
manual path documented in docs/ada-deployment.md, and they remain the on-ramp
if Ada ever needs hosting. The runbook gains a status section recording why CI
was retired and the two things still unverified before any first real deploy:
the Cloud Build IAM grants, and a secret-name mismatch (the runbook creates
ada-admin-key / ada-runtime-key; cloudbuild.yaml defaults to
ai-identity-ADA_ADMIN_KEY / ai-identity-ADA_RUNTIME_KEY — one is wrong, and
the build never got far enough to find out).

Unrelated to the 2026-07-27 history rewrite. That force-push merely triggered
another run: 413ea00 touches no files under agent/, but a rewrite makes
GitHub treat the whole tree as changed. The path filter itself was sound —
it correctly excluded the launcher-only change in #408.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Jul 27, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
ai-identity-landing Ready Ready Preview, Comment Jul 27, 2026 4:46pm
dashboard Ready Ready Preview, Comment Jul 27, 2026 4:46pm

Companion to the deletion in this PR. Filed as a `_CI:_` entry under
Changed, matching how the other CI/deploy changes in [Unreleased] are
recorded — the bold multi-sentence format is used for product and
security changes, not internal tooling.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Levaj2000
Levaj2000 merged commit 64a8537 into main Jul 27, 2026
6 checks passed
@Levaj2000
Levaj2000 deleted the chore/retire-ada-cloudrun-deploy branch July 27, 2026 16:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant