Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: a new prep-deps workflow with caching #29979

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

HowardBraham
Copy link
Contributor

@HowardBraham HowardBraham commented Jan 29, 2025

Description

Major changes

  • prep-deps
    • There's a new workflow that parallels the CircleCI workflow. prep-deps runs first and caches the result, then prep-build-test-webpack makes an artifact, then the benchmarks run.
    • Several of the existing workflows were changed to now depend on prep-deps
    • Many of the file changes are just using the new setup-environment@caching and deleting the independent "Checkout repository" step
  • vars.USE_CACHING
    • This allows us to toggle the caching feature on and off in a centralized way. I left instructions as a comment in main.yml for how to toggle it. We can discuss implementing this in a different way, but passing this variable through different workflows is actually kind of a pain (that's the first way I implemented it). Several of the jobs are 30-60 seconds faster when this is turned on.
  • test-short-suite
    • I think this could be slightly controversial, but I think it's for the better.
    • I noticed that there were 7 individual really short tests, that each ran on their own VM. Each VM instance took about 1m30s to execute. I combined these together into a single VM that runs all the short tests in sequence, and takes about 1m20s combined. The if: always() statements make all independent tests run, even if one fails.
    • The 7 jobs were: test-lint-shellcheck, test-lint-changelog, test-lint-lockfile, test-deps-audit, test-yarn-dedupe, test-deps-depcheck, validate-lavamoat-allow-scripts
    • If this were running on VMs that cost money, this would be a no-brainer. As is on GitHub runners, it doesn't really cost us any time or money. All we can really say is 7 separate jobs burn fossil fuels for a slight bit of convenience in terms of seeing individual test failures in the GitHub PR page.

Prerequisites to merging this PR

Open in GitHub Codespaces

Related issues

Split off from: #29955

@HowardBraham HowardBraham added the team-extension-platform Extension Platform team label Jan 29, 2025
@HowardBraham HowardBraham self-assigned this Jan 29, 2025
Copy link
Contributor

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

@metamaskbot metamaskbot added the team-tiger Tiger team (for tech debt reduction + performance improvements) label Jan 29, 2025
@HowardBraham HowardBraham removed the team-tiger Tiger team (for tech debt reduction + performance improvements) label Jan 29, 2025
@HowardBraham HowardBraham marked this pull request as ready for review January 29, 2025 18:41
@HowardBraham HowardBraham added the DO-NOT-MERGE Pull requests that should not be merged label Jan 29, 2025
@HowardBraham HowardBraham force-pushed the self-hosted-benchmarks branch from c9f1e1e to 6266945 Compare January 29, 2025 21:50
@metamaskbot
Copy link
Collaborator

Builds ready [1283bfe]
Page Load Metrics (1641 ± 69 ms)
PlatformPageMetricMin (ms)Max (ms)Average (ms)StandardDeviation (ms)MarginOfError (ms)
ChromeHomefirstPaint14061928164214569
domContentLoaded13981902162013967
load14071926164114469
domInteractive21118382411
backgroundConnect106324157
firstReactRender15101362411
getState479192110
initialActions01000
loadScripts9981388117711555
setupStore66316168
uiStartup15862435188520096

Copy link
Contributor

@NicholasEllul NicholasEllul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left one question & reviewed usage of caching from a security POV. No concerns from our end. I'll let the others chime in with regards to reviewing the functionality

uses: metamask/github-tools/.github/actions/setup-environment@main
uses: metamask/github-tools/.github/actions/setup-environment@caching
with:
should-cache-restore: ${{ vars.USE_CACHING }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What was the background behind choosing an env variable over harcoding?

Copy link
Contributor Author

@HowardBraham HowardBraham Jan 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pro: It's way fewer lines of code to do it this way. Setting a variable in main.yml and passing it in to a bunch of workflows is harder, much less clean, and easier to accidentally mess up.

Con: You lose some control and repeatability of workflows.

I'm kind of in a divided mind myself.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see what you mean. Other than slower performance due to the extra yarn install, are there any issues if the settings were accidentally set out out of sync?

I suspect most of the time we will want caching for these, and only in the odd case we may set a couple workflows to be false. In which case ${{ vars.USE_CACHING }} could be seen as being synonymous to true (but with a var that can be flipped without going through the peer review flow).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If prep-deps had USE_CACHING set to false, and later steps had USE_CACHING set to true, it would be a problem.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose another option is to just set all of these to true, and then turning it off has to happen in a PR with about 10 files, but maybe that's okay?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would probably also hard-code true. This also allows us to 100% trust that it will be cached (or not), as someone cannot silently change it in the background without anyone knowing - it will be shown in the commit history.

@NicholasEllul
Copy link
Contributor

Change setup-environment@caching back to setup-environment@main

(Not blocking) Are there plans to version changes? While for now pointing directly to the main branch allows for immediate updates, it can result in challenges down the road if a breaking change is introduced. Could be handy to have things set up now to save time in the long run.

@@ -12,14 +12,11 @@ jobs:
runs-on: ubuntu-latest
if: github.event.pull_request.merged == true
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Setup environment
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would not merge the Checkout repository and Setup environment steps together. What if there is some occasion where you want only one? Feels like this step is doing too much at once

@@ -42,7 +42,9 @@ jobs:
- name: Setup environment
if: ${{ steps.needs-beta-build.outputs.NEEDS_BETA_BUILD == 'true' }}
uses: metamask/github-tools/.github/actions/setup-environment@main
uses: metamask/github-tools/.github/actions/setup-environment@caching
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like setup environment has too many arguments and it's doing too much. I wonder if this could be separated to multiple smaller workflows?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I see what you mean. I split this into two actions: setup-environment and setup-environment-secure.

- name: Setup environment
uses: metamask/github-tools/.github/actions/setup-environment@main
uses: metamask/github-tools/.github/actions/setup-environment@caching
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should change from @main (or @caching) to @commithash-on-main

@@ -15,62 +15,52 @@ on:
merge_group:

jobs:
prep-deps:
Copy link
Contributor

@itsyoboieltr itsyoboieltr Feb 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Running prep-deps and making all other jobs dependent on it is a bit dangerous, especially given the limitations of gh actions we recently encountered (max 20 workflows in one file). It is very likely that in the near future (which probably means, when I get off release rotation in March) we will need to split main.yml into multiple smaller workflows (probably each workflow will go in its own file, or we need to batch them somehow, like the short suite), which means that they will not be able to depend on prep-deps.

Also keep in mind that we need to migrate 40 more workflows.

More context here: https://consensys.slack.com/archives/C087GPTR5HQ/p1738658608653149

workflow_call:

jobs:
test-short-suite:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the performance gains, but it does not feel "clean" to do so many things in one workflow

@HowardBraham HowardBraham requested review from a team as code owners February 11, 2025 07:20
@HowardBraham HowardBraham force-pushed the self-hosted-benchmarks branch from 6266945 to abcd5a1 Compare February 11, 2025 07:24
@HowardBraham HowardBraham removed the request for review from a team February 11, 2025 07:25
@metamaskbot
Copy link
Collaborator

Builds ready [b6e58b6]
Page Load Metrics (1643 ± 73 ms)
PlatformPageMetricMin (ms)Max (ms)Average (ms)StandardDeviation (ms)MarginOfError (ms)
ChromeHomefirstPaint24419601586341164
domContentLoaded14021937162415675
load14401962164315373
domInteractive18100392210
backgroundConnect76622178
firstReactRender1592372512
getState35111136
initialActions00000
loadScripts9741426116711756
setupStore65514147
uiStartup164025511890209100

@HowardBraham HowardBraham force-pushed the self-hosted-benchmarks branch 3 times, most recently from e1404b6 to 61028ec Compare February 18, 2025 18:03
Base automatically changed from self-hosted-benchmarks to main February 18, 2025 19:51
@metamaskbot
Copy link
Collaborator

Builds ready [1fa54b9]
Page Load Metrics (1729 ± 53 ms)
PlatformPageMetricMin (ms)Max (ms)Average (ms)StandardDeviation (ms)MarginOfError (ms)
ChromeHomefirstPaint15661957174011455
domContentLoaded15521898170310349
load15711965172911153
domInteractive25118422411
backgroundConnect1081282010
firstReactRender1481302311
getState55412136
initialActions01000
loadScripts1095139812348943
setupStore767192010
uiStartup18032704201919091
Bundle size diffs
  • background: 0 Bytes (0.00%)
  • ui: 0 Bytes (0.00%)
  • common: 0 Bytes (0.00%)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
DO-NOT-MERGE Pull requests that should not be merged team-extension-platform Extension Platform team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants