test: introduce api conformance test

this test runs on each PR and uses a new conformance workflow to compare the base (main) branch openapi spec to the one on this PR
if one of our "stable" APIs change, the test will fail.

this workflow needs a different version of `yq` than the one that comes with `ubuntu`. Additonally yq has issues writing files to disk in the workflows environment, so the yq itself needs to happen in the same container as the openapi-diff.

This means I needed to write a new Contaninerfile and build a new Container which has two stages: first install yq (since the openapi diff image is distroless) and second use yq to parse the two openapi specs and THEN use openapi-diff to actually diff the two specs.

As a follow up to this, we can add an optional way to make it so that it is OK to make these change if properly documented or in a changelog or something.

related to #3237

Signed-off-by: Charlie Doern <[email protected]>

Author: cdoern

.github/images/Containerfile

@@ -0,0 +1,50 @@
+# Multi-stage Docker container for OpenAPI schema comparison
+# This container combines openapi-diff tool with yq for advanced YAML filtering
+
+# Stage 1: Download and prepare yq binary
+# We need a custom yq binary because the base openapi-diff container doesn't include it
+FROM debian:bullseye-slim AS downloader
+
+WORKDIR /tmp
+
+RUN apt-get update && apt-get install -y curl && \
+    apt-get clean && rm -rf /var/lib/apt/lists/*
+
+RUN curl -sSL https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -o /yq && \
+    chmod +x /yq
+
+# Stage 2: Final container with both openapi-diff and yq tools
+# Start from the official openapi-diff container which provides the core comparison functionality
+FROM openapitools/openapi-diff:2.1.2
+
+WORKDIR /workspace
+
+# Copy the yq binary from the downloader stage to the final container
+# This gives us both tools in one container: openapi-diff for comparison and yq for filtering
+COPY --from=downloader /yq /usr/bin/yq
+
+WORKDIR /workspace
+
+# Environment variable containing the YAML filter condition
+# This filter focuses the comparison on only the most critical API endpoints:
+# - paths: Only specific API paths that are most critical for compatibility:
+#   - /v1/inference: Core inference endpoints
+#   - /v1/vector-io: Vector input/output operations
+#   - /v1/vector-dbs: Vector database operations
+#   - /v1/openai/v1/responses: OpenAI-compatible response formats
+#   - /v1/openai/v1/chat: OpenAI-compatible chat endpoints
+# anything else (openapi, info, servers, is just there to make openapi-diff happy)
+ENV FILTER_CONDITION='{ "openapi": .openapi, "info": .info, "servers": .servers, "tags": .tags, "x-tagGroups": .["x-tagGroups"], "components": .components, "paths": (.paths | with_entries(select(.key | test("^/(v1/inference|v1/vector-io|v1/vector-dbs|v1/openai/v1/responses|v1/openai/v1/chat)")))) }'
+
+# Custom entrypoint that combines yq filtering with openapi-diff comparison
+# This allows us to compare only the filtered, relevant parts of the API spec
+ENTRYPOINT ["/bin/sh", "-c", "\
+  echo '=== Running yq filter ===' && \
+  # Filter the base branch OpenAPI spec to only relevant endpoints and save to base.yaml
+  yq eval \"$FILTER_CONDITION\" /workspace/base/docs/_static/llama-stack-spec.yaml > /workspace/base.yaml && \
+  # Filter the current revision OpenAPI spec to only relevant endpoints and save to revision.yaml
+  yq eval \"$FILTER_CONDITION\" /workspace/docs/_static/llama-stack-spec.yaml > /workspace/revision.yaml && \
+  echo '=== Running original openapi-diff entrypoint ===' && \
+  # Run the OpenAPI diff tool on the filtered specs
+  exec java -jar /app/openapi-diff.jar /workspace/base.yaml /workspace/revision.yaml --fail-on-incompatible \
+"]

.github/workflows/README.md

@@ -5,6 +5,7 @@ Llama Stack uses GitHub Actions for Continuous Integration (CI). Below is a tabl
 | Name | File | Purpose |
 | ---- | ---- | ------- |
 | Update Changelog | [changelog.yml](changelog.yml) | Creates PR for updating the CHANGELOG.md |
+| API Conformance Tests | [conformance.yml](conformance.yml) | Run the API Conformance test suite on the changes. |
 | Installer CI | [install-script-ci.yml](install-script-ci.yml) | Test the installation script |
 | Integration Auth Tests | [integration-auth-tests.yml](integration-auth-tests.yml) | Run the integration test suite with Kubernetes authentication |
 | SqlStore Integration Tests | [integration-sql-store-tests.yml](integration-sql-store-tests.yml) | Run the integration test suite with SqlStore |

.github/workflows/conformance.yml

@@ -0,0 +1,66 @@
+# API Conformance Tests
+# This workflow ensures that API changes maintain backward compatibility and don't break existing integrations
+# It runs schema validation and OpenAPI diff checks to catch breaking changes early
+
+name: API Conformance Tests
+
+# Descriptive name that appears in GitHub Actions UI
+run-name: Run the API Conformance test suite on the changes.
+
+on:
+  # Trigger on pushes to main branch to ensure all production code is tested
+  push:
+    branches: [ main ]
+  # Trigger on PR events to catch issues before they reach main
+  pull_request:
+    branches: [ main ]
+    types: [opened, synchronize, reopened] # opened: new PR, synchronize: new commits, reopened: PR reopened after being closed
+    paths:
+      # Only run when relevant files change to avoid unnecessary CI runs
+      - 'llama_stack/**'           # Core library code changes
+      - '!llama_stack/ui/**'       # Exclude UI changes (handled by separate workflow)
+      - 'tests/**'                 # Test changes that might affect API behavior
+      - 'uv.lock'                  # Dependency changes
+      - 'pyproject.toml'           # Project configuration changes
+      - '.github/workflows/conformance.yml' # This workflow itself
+  schedule:
+    # If changing the cron schedule, update the provider in the test-matrix job
+    - cron: '0 0 * * *'  # Daily at 12 AM UTC - test latest client to catch breaking changes in dependencies
+    - cron: '1 0 * * 0'  # Weekly on Sunday at 1 AM UTC - test vllm provider specifically
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/main' && github.run_id || github.ref }}
+  # Cancel in-progress runs when new commits are pushed to avoid wasting CI resources
+  cancel-in-progress: true
+
+jobs:
+  # Job to check if API schema changes maintain backward compatibility
+  check-schema-compatibility:
+    runs-on: ubuntu-latest
+    steps:
+      # Using specific version 4.1.7 because 5.0.0 fails when trying to run this locally using `act`
+      # This ensures consistent behavior between local testing and CI
+      - name: Checkout PR Code
+        uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
+
+      # Checkout the base branch to compare against (usually main)
+      # This allows us to diff the current changes against the previous state
+      - name: Checkout Base Branch
+        uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
+        with:
+          ref: ${{ github.event.pull_request.base.ref }}
+          path: 'base'
+
+      # Build a custom container that includes both openapi-diff and yq tools
+      # The default openapi-diff container doesn't have yq, which we need for advanced YAML filtering
+      - name: Build OpenAPI Diff container
+        run: |
+          docker build -t openapi-diff-yq -f ./.github/images/Containerfile .
+
+      # Run the OpenAPI diff tool to detect breaking changes in the API specification
+      # This step will fail if incompatible changes are detected, preventing breaking changes from being merged
+      - name: Run OpenAPI Diff
+        run: |
+          docker run --rm \
+            -v "$(pwd):/workspace" \
+            openapi-diff-yq