Skip to content

Add pre-deploy test gate and fix Pages two-publisher race - #887

Merged
Marina He (hemarina) merged 5 commits into
mainfrom
feat/release-test-gate
May 20, 2026
Merged

Add pre-deploy test gate and fix Pages two-publisher race#887
Marina He (hemarina) merged 5 commits into
mainfrom
feat/release-test-gate

Conversation

@hemarina

@hemarina Marina He (hemarina) commented May 20, 2026

Copy link
Copy Markdown
Contributor

Fix #884

Summary

Hardens the release pipeline by:

  1. Gating deploys behind a pre-deploy website test job (build + unit tests + Playwright e2e), mirroring Azure/ai-app-templates.
  2. Eliminating the two-publisher race between legacy Jekyll and actions/deploy-pages that produced the recent outage where the live site rendered the raw README.md instead of the gallery.

Why

Releases were dispatched manually with no guarantee the site built, passed tests, or rendered correctly. The recent outage exposed a second problem: Pages was configured with build_type: legacy (source = gh-pages branch), so every push to that branch auto-triggered GitHub's built-in pages-build-deployment (Jekyll) in parallel with actions/deploy-pages. Jekyll rendered the root README.md (no _config.yml/index.html), finished ~2 minutes after deploy-pages, and clobbered the gallery output. The workflow happily reported success either way.

This PR closes both gaps in one shot: a test gate catches broken builds before they reach production, and switching Pages to "GitHub Actions" as the source removes the auto-trigger entirely so actions/deploy-pages becomes the only publisher.

What changes

New reusable workflow .github/workflows/website-test.yml

  • Single job. Always runs npm ci + npm test + npm run build (the lightweight gate for PRs and pushes).
  • Playwright steps (npx playwright install --with-deps chromium + npx playwright test) are gated by if: github.event_name == 'workflow_call' || github.event_name == 'workflow_dispatch', so the heavy browser suite only runs as part of a release (called from release.yml) or when explicitly dispatched — not on every PR.
  • Triggers: pull_request/push on main (paths-filtered to website/** and the workflow itself), plus workflow_call/workflow_dispatch.
  • Uploads the Playwright report as a 7-day artifact on failure.
  • Checkout pins ref: ${{ github.sha }} for defense-in-depth (immutable input to the test job).

.github/workflows/release.yml

  • Adds a guard job with permissions: {} that fails fast if github.ref != refs/heads/main, preventing accidental release runs from feature branches.
  • Adds a test-website job (calls website-test.yml) gated by guard.
  • deploy now needs: [guard, test-website], so the deploy chain only proceeds after both pass.
  • Removes the now-redundant npm test step from deploy (covered by the test gate).
  • Removes the sync job (see below).
  • Checkout pins ref: ${{ github.sha }}.

.github/workflows/sync-gh-pages.yml (removed)

Previously force-pushed the website build to the gh-pages branch. That push is what triggered Jekyll. With Pages source switched to "GitHub Actions", there is no gh-pages branch to keep in sync — actions/deploy-pages publishes the artifact directly. Deleting the workflow removes the auto-trigger.

.github/workflows/test-deploy.yml (removed)

Superseded by website-test.yml, which fires on the same triggers and runs a superset of its checks.

Action SHA refresh

  • actions/upload-pages-artifact v4 → v5 (fc324d3547104276b827a68afc52ff2a11cc49c9)
  • actions/deploy-pages v4 → v5 (cd2ce8fcbc39b97be8ca5fce6e763baed58fa128)
  • actions/upload-artifact v7.0.0 → v7.0.1 (043fb46d1a93c77aae656e7c1c64a875d1fc6a0a)
  • actions/github-script v9.0.0 SHA refresh (d746ffe35508b1917358783b479e04febd2b8f71) in discover-templates-extensions.yml and extension-submission.yml

The v4→v5 upgrade was blocked by an artifact format mismatch (upload v5 uses immutable artifacts under upload-artifact@v7, which deploy-pages@v4 couldn't consume). Moving both to v5 in lockstep resolves it and unblocks the revert from #883.

Playwright spec changes

  • New website/e2e/gallery-functionality.spec.ts — 7 deploy-gate tests covering: homepage hero, a healthy template count (≥50), card rendering, narrowing search ("azure"), empty-state messaging, Language-section tag filter (Python), and clear-search restoration. Reads the total from the [role="status"] "Viewing N templates" live region rather than counting .fui-Card (which is page-bound to 20 by pagination).
  • Fixed existing specs to match current UI behavior:
    • Fluent UI SearchBox only updates the URL on onSearch (Enter) — added .press("Enter") after .fill(...) in gallery-filters.spec.ts and homepage.spec.ts.
    • .clear() doesn't fire Fluent UI's clear-X handler — replaced with fill("") + press("Enter") in gallery-filters.spec.ts.
    • getting-started.spec.ts: corrected hero text ("Ship in minutes" — was "Ship to Azure"); added exact: true on the "Pick a template" / "Deploy to Azure" headings to avoid strict-mode collision with the hero h1.

website/.gitignore

  • Adds /playwright-report, /test-results, /playwright/.cache to keep local Playwright artifacts out of the repo.
  • Adds !*.png to override the root-level *.png rule, so website/static/**/*.png and any future visual-regression baselines are tracked.

Removed an accidentally-committed file

  • website/playwright-report/index.html (526KB report bundle) was committed in an earlier revision of this branch. Deleted.

Operational steps still required (cannot be done from PR)

  • Repo Settings → Pages → set Source to "GitHub Actions" (currently "Deploy from a branch / gh-pages").
  • Repo Settings → Environments → github-pages → update Deployment branches from gh-pages to main.

Without these two clicks, the workflow change alone won't take effect.

Validation

  • All 26 Playwright specs pass locally against npm run serve (Docusaurus production build).
  • PR-mode CI (pull_request) runs only the lightweight gate (npm ci + npm test + npm run build).
  • Release-mode (workflow_call from release.yml) runs the full Playwright suite as the deploy gate.

Notes for reviewers

  • Branch protection check Test deployment will need to be renamed to whatever job name website-test.yml emits once this lands — call out in a follow-up.
  • All actions are SHA-pinned.

Mirrors the pattern used by Azure/ai-app-templates:
- New reusable workflow .github/workflows/website-test.yml runs unit tests
  (jest), builds the Docusaurus site, and runs Playwright e2e tests against
  the built artifact.
- release.yml gains a test-website job that runs before sync, so a failing
  test blocks the deploy.
- Removes .github/workflows/test-deploy.yml (now redundant: website-test.yml
  triggers on pull_request/push to main with the same paths filter and runs a
  superset of its checks).
- Drops the now-duplicate `npm test` step from the deploy job (already
  covered by the gate).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a stronger pre-release validation gate for the Docusaurus website by adding a reusable website test workflow (unit + build + Playwright e2e) and wiring it into the manual release pipeline, replacing the previous PR-only build/test workflow.

Changes:

  • Added a new reusable workflow (website-test.yml) that runs npm ci, npm test, npm run build, and Playwright e2e tests, uploading the Playwright report on failure.
  • Updated the release workflow to run the website test workflow as a prerequisite before proceeding with sync/deploy, and removed the redundant npm test from deploy.
  • Removed the old test-deploy.yml workflow.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
.github/workflows/website-test.yml New reusable CI workflow that runs unit tests, build, and Playwright e2e tests for the website.
.github/workflows/release.yml Adds a website test gate job and makes the sync job depend on it; removes redundant deploy-time unit test step.
.github/workflows/test-deploy.yml Deletes the prior PR-only workflow in favor of the new consolidated website test workflow.

Comment thread .github/workflows/release.yml Outdated
Comment thread .github/workflows/website-test.yml
@hemarina Marina He (hemarina) changed the title Add website test gate to release workflow Add pre-deploy test gate and fix Pages two-publisher race May 20, 2026
@hemarina
Marina He (hemarina) merged commit c1268ef into main May 20, 2026
2 checks passed
@hemarina
Marina He (hemarina) deleted the feat/release-test-gate branch May 20, 2026 23:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

add testing gate before release pipeline

3 participants