Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: a new prep-deps workflow with caching #29979

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 2 additions & 5 deletions .github/workflows/add-release-label.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,14 +12,11 @@ jobs:
runs-on: ubuntu-latest
if: github.event.pull_request.merged == true
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Setup environment
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would not merge the Checkout repository and Setup environment steps together. What if there is some occasion where you want only one? Feels like this step is doing too much at once

uses: metamask/github-tools/.github/actions/setup-environment@caching
with:
fetch-depth: 0 # This is needed to checkout all branches

- name: Setup environment
uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565

- name: Get the next semver version
id: get-next-semver-version
env:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/build-beta.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ jobs:

- name: Setup environment
if: ${{ steps.needs-beta-build.outputs.NEEDS_BETA_BUILD == 'true' }}
uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
uses: metamask/github-tools/.github/actions/setup-environment-secure@caching

- name: Run beta build
if: ${{ steps.needs-beta-build.outputs.NEEDS_BETA_BUILD == 'true' }}
Expand Down
5 changes: 1 addition & 4 deletions .github/workflows/build-storybook.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,8 @@ jobs:
name: Build storybook
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Setup environment
uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
uses: metamask/github-tools/.github/actions/setup-environment-secure@caching

- name: Build storybook
run: yarn storybook:build
Expand Down
5 changes: 1 addition & 4 deletions .github/workflows/check-attributions.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,8 @@ jobs:
check-attributions:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Setup environment
uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
uses: metamask/github-tools/.github/actions/setup-environment@caching
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should change from @main (or @caching) to @commithash-on-main


- name: Check attributions changes
run: yarn attributions:check
5 changes: 1 addition & 4 deletions .github/workflows/check-pr-labels.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,8 @@ jobs:
pull-requests: read

steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Setup environment
uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
uses: metamask/github-tools/.github/actions/setup-environment@caching

- name: Check PR has required labels
env:
Expand Down
5 changes: 1 addition & 4 deletions .github/workflows/check-template-and-add-labels.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,8 @@ jobs:
check-template-and-add-labels:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Setup environment
uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
uses: metamask/github-tools/.github/actions/setup-environment@caching

- name: Check template and add labels
id: check-template-and-add-labels
Expand Down
5 changes: 1 addition & 4 deletions .github/workflows/close-bug-report.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,8 @@ jobs:
runs-on: ubuntu-latest
if: github.event.pull_request.merged == true && startsWith(github.event.pull_request.head.ref, 'Version-v')
steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Setup environment
uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
uses: metamask/github-tools/.github/actions/setup-environment@caching

- name: Close release bug report issue
env:
Expand Down
58 changes: 29 additions & 29 deletions .github/workflows/codeql-analysis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,14 @@
# the `language` matrix defined below to confirm you have the correct set of
# supported CodeQL languages.
#
name: "CodeQL"
name: 'CodeQL'

on:
push:
branches: [ main, Version-v*, cla-signatures, master, snaps ]
branches: [main, Version-v*, cla-signatures, master, snaps]
pull_request:
# The branches below must be a subset of the branches above
branches: [ main ]
branches: [main]
schedule:
- cron: '28 12 * * 0'

Expand All @@ -32,39 +32,39 @@ jobs:
strategy:
fail-fast: false
matrix:
language: [ 'javascript' ]
language: ['javascript']
# CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python', 'ruby' ]
# Learn more about CodeQL language support at https://git.io/codeql-language-support

steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Checkout repository
uses: actions/checkout@v4

# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v3
with:
languages: ${{ matrix.language }}
# If you wish to specify custom queries, you can do so here or in a config file.
# By default, queries listed here will override any specified in a config file.
# Prefix the list here with "+" to use these queries and those in the config file.
# queries: ./path/to/local/query, your-org/your-repo/queries@main
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v3
with:
languages: ${{ matrix.language }}
# If you wish to specify custom queries, you can do so here or in a config file.
# By default, queries listed here will override any specified in a config file.
# Prefix the list here with "+" to use these queries and those in the config file.
# queries: ./path/to/local/query, your-org/your-repo/queries@main

# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below)
- name: Autobuild
uses: github/codeql-action/autobuild@v3
# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below)
- name: Autobuild
uses: github/codeql-action/autobuild@v3

# ℹ️ Command-line programs to run using the OS shell.
# 📚 https://git.io/JvXDl
# ℹ️ Command-line programs to run using the OS shell.
# 📚 https://git.io/JvXDl

# ✏️ If the Autobuild fails above, remove it and uncomment the following three lines
# and modify them (or add more) to build your code if your project
# uses a compiled language
# ✏️ If the Autobuild fails above, remove it and uncomment the following three lines
# and modify them (or add more) to build your code if your project
# uses a compiled language

#- run: |
# make bootstrap
# make release
#- run: |
# make bootstrap
# make release

- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v3
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v3
2 changes: 1 addition & 1 deletion .github/workflows/create-bug-report.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ jobs:

- name: Setup environment
if: steps.extract_version.outputs.version
uses: metamask/github-tools/.github/actions/setup-environment@main
uses: metamask/github-tools/.github/actions/setup-environment@caching

- name: Create bug report issue on planning repo
if: steps.extract_version.outputs.version
Expand Down
32 changes: 15 additions & 17 deletions .github/workflows/crowdin-action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,26 +9,24 @@ on:
branches:
- main
schedule:
- cron: "0 */12 * * *"
- cron: '0 */12 * * *'

jobs:
synchronize-with-crowdin:
runs-on: ubuntu-latest

steps:
- name: Checkout
uses: actions/checkout@v4

- name: Checkout
uses: actions/checkout@v4

- name: crowdin action
uses: crowdin/github-action@a3160b9e5a9e00739392c23da5e580c6cabe526d
with:
upload_translations: true
download_translations: true
github_user_name: metamaskbot
github_user_email: [email protected]
env:
GITHUB_ACTOR: metamaskbot
GITHUB_TOKEN: ${{ secrets.METAMASKBOT_CROWDIN_TOKEN }}
CROWDIN_PROJECT_ID: ${{ secrets.CROWDIN_PROJECT_ID }}
CROWDIN_PERSONAL_TOKEN: ${{ secrets.CROWDIN_PERSONAL_TOKEN }}
- name: crowdin action
uses: crowdin/github-action@a3160b9e5a9e00739392c23da5e580c6cabe526d
with:
upload_translations: true
download_translations: true
github_user_name: metamaskbot
github_user_email: [email protected]
env:
GITHUB_ACTOR: metamaskbot
GITHUB_TOKEN: ${{ secrets.METAMASKBOT_CROWDIN_TOKEN }}
CROWDIN_PROJECT_ID: ${{ secrets.CROWDIN_PROJECT_ID }}
CROWDIN_PERSONAL_TOKEN: ${{ secrets.CROWDIN_PERSONAL_TOKEN }}
7 changes: 2 additions & 5 deletions .github/workflows/fitness-functions.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,11 @@ jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup environment
uses: metamask/github-tools/.github/actions/setup-environment@caching
with:
fetch-depth: 0 # This is needed to checkout all branches

- name: Setup environment
uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565

- name: Run fitness functions
env:
BASE_REF: ${{ github.event.pull_request.base.ref }}
Expand Down
84 changes: 35 additions & 49 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,75 +15,65 @@ on:
merge_group:

jobs:
prep-deps:
Copy link
Contributor

@itsyoboieltr itsyoboieltr Feb 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Running prep-deps and making all other jobs dependent on it is a bit dangerous, especially given the limitations of gh actions we recently encountered (max 20 workflows in one file). It is very likely that in the near future (which probably means, when I get off release rotation in March) we will need to split main.yml into multiple smaller workflows (probably each workflow will go in its own file, or we need to batch them somehow, like the short suite), which means that they will not be able to depend on prep-deps.

Also keep in mind that we need to migrate 40 more workflows.

More context here: https://consensys.slack.com/archives/C087GPTR5HQ/p1738658608653149

runs-on: ubuntu-latest
steps:
- name: Setup environment
uses: metamask/github-tools/.github/actions/setup-environment@caching
with:
should-cache-restore: 'false'
should-cache-save: ${{ vars.USE_CACHING }}
# -- If you're a repo admin, you can change vars.USE_CACHING at:
# https://github.com/MetaMask/metamask-extension/settings/variables/actions
# -- If you're just a contributor, you can change vars.USE_CACHING by following this documentation:
# https://docs.github.com/en/rest/actions/variables?apiVersion=2022-11-28#update-a-repository-variable

lint-workflows:
name: Lint workflows
uses: metamask/github-tools/.github/workflows/lint-workflows.yml@1d657e262aea7e3f216754febb624831527d2565

test-lint-shellcheck:
name: Test lint shellcheck
uses: ./.github/workflows/test-lint-shellcheck.yml
uses: metamask/github-tools/.github/workflows/lint-workflows.yml@caching

test-lint:
name: Test lint
needs: prep-deps
uses: ./.github/workflows/test-lint.yml

test-circular-deps:
name: Test circular deps
needs: prep-deps
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Setup environment
uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
uses: metamask/github-tools/.github/actions/setup-environment@caching
with:
should-cache-restore: ${{ vars.USE_CACHING }}

- name: Check circular dependencies
run: yarn circular-deps:check

test-lint-changelog:
name: Test lint changelog
uses: ./.github/workflows/test-lint-changelog.yml

test-lint-lockfile:
name: Test lint lockfile
uses: ./.github/workflows/test-lint-lockfile.yml

test-deps-audit:
name: Test deps audit
uses: ./.github/workflows/test-deps-audit.yml

test-yarn-dedupe:
name: Test yarn dedupe
uses: ./.github/workflows/test-yarn-dedupe.yml

test-deps-depcheck:
name: Test deps depcheck
uses: ./.github/workflows/test-deps-depcheck.yml
test-short-suite:
needs: prep-deps
uses: ./.github/workflows/test-short-suite.yml

test-storybook:
name: Test storybook
needs: prep-deps
uses: ./.github/workflows/test-storybook.yml

validate-lavamoat-allow-scripts:
name: Validate lavamoat allow scripts
uses: ./.github/workflows/validate-lavamoat-allow-scripts.yml

validate-lavamoat-policy-build:
name: Validate lavamoat policy build
needs: prep-deps
uses: ./.github/workflows/validate-lavamoat-policy-build.yml

validate-lavamoat-policy-webapp:
name: Validate lavamoat policy webapp
needs: prep-deps
uses: ./.github/workflows/validate-lavamoat-policy-webapp.yml

prep-build-test-browserify:
needs: prep-deps
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Setup environment
uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
uses: metamask/github-tools/.github/actions/setup-environment@caching
with:
should-cache-restore: ${{ vars.USE_CACHING }}

- run: yarn build:test

Expand All @@ -94,13 +84,13 @@ jobs:
path: ./dist/

prep-build-test-webpack:
needs: prep-deps
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Setup environment
uses: metamask/github-tools/.github/actions/setup-environment@1d657e262aea7e3f216754febb624831527d2565
uses: metamask/github-tools/.github/actions/setup-environment@caching
with:
should-cache-restore: ${{ vars.USE_CACHING }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What was the background behind choosing an env variable over harcoding?

Copy link
Contributor Author

@HowardBraham HowardBraham Jan 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pro: It's way fewer lines of code to do it this way. Setting a variable in main.yml and passing it in to a bunch of workflows is harder, much less clean, and easier to accidentally mess up.

Con: You lose some control and repeatability of workflows.

I'm kind of in a divided mind myself.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see what you mean. Other than slower performance due to the extra yarn install, are there any issues if the settings were accidentally set out out of sync?

I suspect most of the time we will want caching for these, and only in the odd case we may set a couple workflows to be false. In which case ${{ vars.USE_CACHING }} could be seen as being synonymous to true (but with a var that can be flipped without going through the peer review flow).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If prep-deps had USE_CACHING set to false, and later steps had USE_CACHING set to true, it would be a problem.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose another option is to just set all of these to true, and then turning it off has to happen in a PR with about 10 files, but maybe that's okay?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would probably also hard-code true. This also allows us to 100% trust that it will be cached (or not), as someone cannot silently change it in the background without anyone knowing - it will be shown in the commit history.


- run: yarn webpack --test --no-lavamoat --no-cache --browser=chrome --lockdown --sentry --snow --env production
env:
Expand All @@ -114,6 +104,7 @@ jobs:

run-tests:
name: Run tests
needs: prep-deps
uses: ./.github/workflows/run-tests.yml

run-benchmarks:
Expand Down Expand Up @@ -171,14 +162,9 @@ jobs:
runs-on: ubuntu-latest
needs:
- lint-workflows
- test-lint-shellcheck
- test-lint
- test-lint-changelog
- test-lint-lockfile
- test-yarn-dedupe
- test-deps-depcheck
- test-short-suite
- test-storybook
- validate-lavamoat-allow-scripts
- validate-lavamoat-policy-build
- validate-lavamoat-policy-webapp
- run-tests
Expand Down
Loading
Loading