Skip to content

fix(deploy-prod): push to ghcr.io — the fleet left ECR a while ago#1003

Merged
jee7s merged 2 commits into
mainfrom
fix/deploy-prod-ghcr
Jul 15, 2026
Merged

fix(deploy-prod): push to ghcr.io — the fleet left ECR a while ago#1003
jee7s merged 2 commits into
mainfrom
fix/deploy-prod-ghcr

Conversation

@jee7s

@jee7s jee7s commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

The failure

v1.0.0's approved deploy failed at Log in to Amazon ECR: checkin-deploy-prod is not authorized for ecr:GetAuthorizationToken — and that's correct: the role's own policy comment says "Images are built/pushed to ghcr.io by the checkin CI… the deploy role no longer needs ecr:*". The prod task definition already pulls ghcr.io/innovationtreehouse/checkin-prod (verified live: image: ghcr.io/innovationtreehouse/checkin-prod:bootstrap, repositoryCredentials: ghcr-pull-credentials). The workflow was the last piece of the fleet still pointed at the vestigial ECR registry — latent because this was the first prod deploy ever.

Fix (mirrors deploy-dev.yml)

  • IMAGE_REPO: ghcr.io/innovationtreehouse/checkin-prod replaces the ECR registry/repo envs
  • docker/login-action with GITHUB_TOKEN replaces amazon-ecr-login; packages: write added to permissions
  • Both :sha and :<release-tag> tags still pushed; report step updated

Trade-off note (security review L3): ECR's IMMUTABLE tag setting doesn't exist on GHCR — unique :sha tags remain the integrity practice, consistent with the entire fleet (dev, arbor, s-read, bootstrap). Infra follow-up available: delete the now-unused aws_ecr_repository.checkin resources.

After merge

Third cut of v1.0.0 (the release workflow runs from the tag's commit). Validate + gate will re-run; reviewer approves once more; deploy proceeds on GHCR.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Hq9naXJVyJnLYpFZapZW24

v1.0.0's approved deploy failed at 'Log in to Amazon ECR':
checkin-deploy-prod has no ecr:* permissions — CORRECTLY, per its own
policy comment ('images are built/pushed to ghcr.io... the deploy role no
longer needs ecr:*'). The prod task definition already pulls
ghcr.io/innovationtreehouse/checkin-prod via the shared
ghcr-pull-credentials secret; this workflow was the only piece still
targeting the vestigial ECR registry (latent: first prod deploy ever).

Mirrors deploy-dev.yml: docker/login-action with GITHUB_TOKEN,
packages:write permission, IMAGE_REPO env, both :sha and :release tags
pushed. Note the accepted trade-off: GHCR has no ECR-style immutable
tags — unique :sha tags remain the practice, matching the whole fleet.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hq9naXJVyJnLYpFZapZW24
@jee7s jee7s enabled auto-merge July 15, 2026 02:58
@jee7s jee7s requested review from dkaygithub and thpr July 15, 2026 02:58
Full audit of the never-exercised deploy job against live AWS + the proven
deploy-dev.yml found three more guaranteed first-run failures:

1. docker build had no -f: there is NO root Dockerfile — the image builds
   from checkin-app/Dockerfile with repo-root context (deploy-dev's exact
   invocation, incl. the NEXT_PUBLIC_GIT_SHA build-arg prod also lacked).
2. update-service never set --desired-count: the service is STAGED at 0
   by infra (ignore_changes; 'the deploy pipeline scales this up') — a 0/0
   service is instantly 'stable', so the run would go green serving nothing.
3. No post-rollout smoke existed (the header's 'smoke' is CI's image-boot
   check): added a real probe of APP_URL with retries, failing the run
   unless production actually answers.

Verified clean in the same audit: capacity-provider name + strategy match
the live cluster; migrate run-task pattern (host/EC2, no network config);
log-dump group/stream names; host port 4000 clear of wiki; DDL secret
wiring; report step.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hq9naXJVyJnLYpFZapZW24
@jee7s

jee7s commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

Scrubbed the whole deploy job per review — three more guaranteed first-run failures found and fixed on this branch: (1) docker build lacked -f checkin-app/Dockerfile (no root Dockerfile exists — the build dies preparing context; also added the NEXT_PUBLIC_GIT_SHA build-arg for parity with dev); (2) update-service never set --desired-count 1 — the service is staged at 0 by infra design, and a 0/0 service is instantly 'stable', so the run would have gone green while serving nothing; (3) no post-rollout smoke existed at all — added a real retried probe of the deployed APP_URL that fails the run unless production answers. Audited clean: capacity provider name/strategy vs the live cluster, the host/EC2 migrate run-task pattern, log group/stream names, host port 4000 vs wiki, DDL secret wiring. One capacity note for later: with the app (512 MiB) running, the instance has ~12 MiB free — the NEXT release's migrate task (512 MiB) won't place without a second instance or a smaller migrate reservation; flagging as an infra follow-up before v1.0.1.

@jee7s jee7s disabled auto-merge July 15, 2026 03:04
@jee7s jee7s enabled auto-merge July 15, 2026 03:04
@jee7s jee7s added this pull request to the merge queue Jul 15, 2026
Merged via the queue into main with commit 8d68cf2 Jul 15, 2026
6 checks passed
@github-actions

Copy link
Copy Markdown

✅ Deployed to dev

Commit 8d68cf2 is live on https://ops-dev.innovationtreehouse.org.

  • Image: ghcr.io/innovationtreehouse/checkin-dev:8d68cf2
  • Migrations: applied · Service: stable
  • Deploy run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants