test: introduce api conformance test

this test runs on each PR and uses the openapi-diff-action to compare the base (main) branch openapi spec to the one on this PR
if one of our "stable" APIs change, the test will fail.

As a follow up to this, we can add an optional way to make it so that it is OK to make these change if properly documented or in a changelog or something.

related to #3237

Signed-off-by: Charlie Doern <[email protected]>

Author: cdoern

.github/workflows/README.md

@@ -5,6 +5,7 @@ Llama Stack uses GitHub Actions for Continuous Integration (CI). Below is a tabl
 | Name | File | Purpose |
 | ---- | ---- | ------- |
 | Update Changelog | [changelog.yml](changelog.yml) | Creates PR for updating the CHANGELOG.md |
+| API Conformance Tests | [conformance.yml](conformance.yml) | Run the API Conformance test suite on the changes. |
 | Installer CI | [install-script-ci.yml](install-script-ci.yml) | Test the installation script |
 | Integration Auth Tests | [integration-auth-tests.yml](integration-auth-tests.yml) | Run the integration test suite with Kubernetes authentication |
 | SqlStore Integration Tests | [integration-sql-store-tests.yml](integration-sql-store-tests.yml) | Run the integration test suite with SqlStore |

.github/workflows/conformance.yml

@@ -0,0 +1,70 @@
+name: API Conformance Tests
+
+run-name: Run the API Conformance test suite on the changes.
+
+on:
+  push:
+    branches: [ main ]
+  pull_request:
+    branches: [ main ]
+    types: [opened, synchronize, reopened]
+    paths:
+      - 'llama_stack/**'
+      - '!llama_stack/ui/**'
+      - 'tests/**'
+      - 'uv.lock'
+      - 'pyproject.toml'
+      - '.github/workflows/conformance.yml' # This workflow
+      - '.github/actions/setup-ollama/action.yml'
+      - '.github/actions/setup-test-environment/action.yml'
+      - '.github/actions/run-and-record-tests/action.yml'
+  schedule:
+    # If changing the cron schedule, update the provider in the test-matrix job
+    - cron: '0 0 * * *'  # (test latest client) Daily at 12 AM UTC
+    - cron: '1 0 * * 0'  # (test vllm) Weekly on Sunday at 1 AM UTC
+
+concurrency:
+  # Skip concurrency for pushes to main - each commit should be tested independently
+  group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/main' && github.run_id || github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  check-schema-compatibility:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout PR Code
+        uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # 5.0.0
+
+      - name: Checkout Base Branch
+        uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # 5.0.0
+        with:
+          ref: ${{ github.event.pull_request.base.ref }}
+          path: 'base'
+
+      # does this include yq?
+      - name: Install dependencies
+        uses: ./.github/actions/setup-runner
+
+      - name: Filter Schemas
+        run: |
+          # Define the filter to match all paths that start with the specified prefixes.
+          FILTER_CONDITION='
+          .paths |= with_entries(select(
+              (.key | startswith("/v1/inference")) or
+              (.key | startswith("/v1/vector-io")) or
+              (.key | startswith("/v1/vector-dbs")) or
+              (.key | startswith("/v1/openai/v1/responses")) or
+              (.key | startswith("/v1/openai/chat"))
+          ))
+          '
+
+          # Filter the base and revision schema files
+          yq eval "$FILTER_CONDITION" base/docs/_static/llama-stack-spec.yaml > base-filtered.yaml
+          yq eval "$FILTER_CONDITION" docs/_static/llama-stack-spec.yaml > revision-filtered.yaml
+
+      - name: Run OpenAPI Diff on Filtered Schemas
+        uses: OpenAPITools/openapi-diff-action@c71c94cb937760c067818cdc5e14b3a1115fb840 # v2.1.2
+        with:
+          base: 'base-filtered.yaml'
+          revision: 'revision-filtered.yaml'
+          fail-on-incompatible: true