Let's include E2E tests for the website

[UlisesGascon]

Motivation

We currently don’t have a reliable way to test UI changes on the website beyond building and manually checking it. Manual testing is error-prone. This PR introduces new tests using E2E testing with headless browsers, which run the web app, query the DOM, click elements, and so on.

It’s also difficult to cover all pieces with traditional tests. After discussing with @KoolTheba, we decided to include visual regression tests (snapshot vs snapshot). This lets the testing framework detect UI changes with adjustable sensitivity.

Snapshot testing can vary significantly between machines, so we introduced Chromatic to run these tests consistently and provide a dashboard where we can review and approve or reject changes.

Main Changes

• Added a pipeline that runs E2E tests using headless browsers and pushes UI differences to Chromatic.
• Added simple tests supporting Playwright using different actions and resolutions.
• Added network request mocks, useful for scenarios like viewer: show checks missing from API with a ? #799.

Assumptions

• We are okay using a new third-party service (free tier) for visual regression tests in CI. Tests can also run locally without any third-party service by setting the ENABLE_SNAPSHOTS environment variable.
• This is likely a first iteration of the approach; we can refine or simplify it later.
• The dependencies are installed directly on the CI when running this pipeline or manually when you want to run this tests locally. So no changes where made on the yarn.lock or package.json files.

⚠️ Blockers

• enable secrets: GitHub Actions seems unable to read secrets. The same settings work in CI but not in PR runs. We need to fix this before merging (cc: @spencerschrock).

Notes

• Thanks to @KoolTheba for suggesting this E2E approach and for the feedback on browser rendering differences between CI and local environments. I still couldn’t get it working in Docker, so I opted for Chromatic.
• The Chromatic project dashboard (login required) is available, and I can add you to the team so you can review visual changes: https://www.chromatic.com/build?appId=691883dc4fa0c14fef366886. If you’re new to regression testing or Chromatic, this video might help: https://www.youtube.com/watch?v=Y1eTK-j66PU.

Important

I made many changes while writing this. It would be very helpful if you run everything locally following the instructions to confirm the docs make sense and are easy to follow.

Screenshots

Local UI

All tests:

when running tests locally

One test suite in detail:

If you run snapshots:

When discrepancies appear:

Chromatic UI

Our dashboard:

[netlify]

✅ Deploy Preview for ossf-scorecard ready!

Name	Link
🔨 Latest commit	f5736ce
🔍 Latest deploy log	https://app.netlify.com/projects/ossf-scorecard/deploys/6918a116f4acdc0008c4fdde
😎 Deploy Preview	https://deploy-preview-900--ossf-scorecard.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[spencerschrock]

• enable secrets: GitHub Actions seems unable to read secrets

That is a limitation of GitHub Actions (link):

With the exception of GITHUB_TOKEN, secrets are not passed to the runner when a workflow is triggered from a forked repository.