Skip to content

Add automated aicollection tag validation to npm test #885

Description

Problem

The aicollection tag in website/static/templates.json is reserved for templates listed in the Microsoft AI collection at https://azure.github.io/ai-app-templates/. Today we enforce that rule only through manual PR review.

This came up in #780, which proposes adding 36 templates all carrying the aicollection tag. Checking each entry by hand against the AI collection site is slow and error-prone, and at scale it blocks otherwise-clean PRs from getting merged.

Proposed approach

Add a check to the existing npm test suite in website/ that:

  1. Fetches the canonical list of templates from https://azure.github.io/ai-app-templates/ (ideally from a JSON manifest, not by scraping HTML)
  2. For each entry in templates.json carrying the aicollection tag, asserts the source repo URL (or another stable identifier) appears in the canonical list
  3. Fails with a clear message that names the offending templates if any aicollection-tagged entry is missing from the canonical list

Open questions

  • Does the AI collection site expose a machine-readable manifest, or should we check in a local allow-list and refresh it on a cadence?
  • How should we handle network failures in CI, soft-fail with a warning or skip the check entirely?
  • Should this also cover other reserved tags like msft, popular, and featured?

Acceptance criteria

  • npm test fails when a template carries aicollection without a match in the canonical source
  • The failure message names the mistagged templates
  • CI behavior is documented in the contributor guide
  • Existing templates.json passes the new check, or any known violations are tracked in a follow-up

Context

Raised in the review of #780.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions