Skip to content

Add production stage to CI pipeline #2483

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

nhatnghiho
Copy link
Contributor

Issues:

Resolves #CryptoAlg-2985

Description of changes:

Enable production stage for the CI pipeline with a few upgrades:

  • The transition from pre-prod to prod requires manual approval
  • Push Docker images built in pre-production to our production ECR directly rather than rebuilding. Right now only available for Linux Docker images (only takes 10 minutes for Linux Docker images vs 1.5hr to rebuild from scratch)
  • The pipeline's execution mode is QUEUED, which means executions will get stuck and piled on at the manual approval step, regardless of what type of changes. I added a skip condition to skip production stage (including the manual approval step) to avoid unnecessary executions

Testing:

Test deploy the pipeline using instructions in README

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and the ISC license.

@codecov-commenter
Copy link

codecov-commenter commented Jun 11, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.85%. Comparing base (39071f3) to head (6521351).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2483      +/-   ##
==========================================
- Coverage   78.85%   78.85%   -0.01%     
==========================================
  Files         640      640              
  Lines      109604   109604              
  Branches    15525    15512      -13     
==========================================
- Hits        86432    86424       -8     
- Misses      22474    22483       +9     
+ Partials      698      697       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@nhatnghiho nhatnghiho marked this pull request as ready for review June 16, 2025 14:38
@nhatnghiho nhatnghiho requested a review from a team as a code owner June 16, 2025 14:38
skmcgrail
skmcgrail previously approved these changes Jun 17, 2025
Comment on lines 196 to 201
# Pipeline is run everytime we push to main branch. Avoid unnecessary hold up if these updates are non-CI related
if not IS_DEV:
start_index = next(
(i for i, stage in enumerate(base_pipeline.stages) if stage.stage_name == "PromoteToProduction"), #TODO: Revert this
None
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't follow what this is doing or what the impact is, also do you have an issue for this TODO so we don't forget?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The TODO is redundant, I'll remove that.

But the idea is -- the pipeline is run everytime we push to main branch. With every stage, we would check to see if we need to redeploy. However, the manual approval step doesn't have such check. The pipeline will just sit and wait at the manual approval step until we approve it. This will lead to a pile of irrelevant pipeline executions stuck waiting at this step. That's why I put in the skip condition for this manual approval step so we can silently skip it (and since I'm at it already, when the manual approval step is skipped, I also configure the pipeline to skip all the subsequent production stages)

Comment on lines 61 to 66
# Create date-based tag
img_push_date=$(date +%Y-%m-%d)
if [[ -n "${PLATFORM:-}" && ${PLATFORM} == "windows" ]]; then
img_push_date=$(date +%Y-%m-%d-%H)
fi
date_tag="${tag%_latest}_${img_push_date}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This creates today's tag which will be different than the pre-prod image, is there anyway to connect a prod image to the pre-prod image it came from?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated code to reflect this. Since our images have a lifecycle of 30 days though, which depends on the time the images are published to ECR, the date tag and the expiration time will not correspond to each other. But preserving the date tag would make it easier for debugging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants