Skip to content

Commit 216b33c

Browse files
cdoerncdoern
committed
test: introduce api conformance test
this test runs on each PR and uses a new conformance workflow to compare the base (main) branch openapi spec to the one on this PR if one of our "stable" APIs change, the test will fail. this workflow needs a different version of `yq` than the one that comes with `ubuntu`. Additonally yq has issues writing files to disk in the workflows environment, so the yq itself needs to happen in the same container as the openapi-diff. This means I needed to write a new Contaninerfile and build a new Container which has two stages: first install yq (since the openapi diff image is distroless) and second use yq to parse the two openapi specs and THEN use openapi-diff to actually diff the two specs. As a follow up to this, we can add an optional way to make it so that it is OK to make these change if properly documented or in a changelog or something. related to #3237 Signed-off-by: Charlie Doern <[email protected]>
1 parent d73955a commit 216b33c

File tree

3 files changed

+113
-0
lines changed

3 files changed

+113
-0
lines changed

‎.github/images/Containerfile‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# Multi-stage container for OpenAPI schema comparison
2+
# This container combines openapi-diff tool with yq for advanced YAML filtering
3+
4+
# Stage 1: Download and prepare yq binary
5+
# We need a custom yq binary because the base openapi-diff container doesn't include it
6+
FROM debian:bullseye-slim AS downloader
7+
8+
WORKDIR /tmp
9+
10+
RUN apt-get update && apt-get install -y curl && \
11+
apt-get clean && rm -rf /var/lib/apt/lists/*
12+
13+
RUN curl -sSL https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -o /yq && \
14+
chmod +x /yq
15+
16+
# Stage 2: Final container with both openapi-diff and yq tools
17+
# Start from the official openapi-diff container which provides the core comparison functionality
18+
FROM openapitools/openapi-diff:2.1.2
19+
20+
WORKDIR /workspace
21+
22+
# Copy the yq binary from the downloader stage to the final container
23+
# This gives us both tools in one container: openapi-diff for comparison and yq for filtering
24+
COPY --from=downloader /yq /usr/bin/yq
25+
26+
WORKDIR /workspace
27+
28+
# Environment variable containing the YAML filter condition
29+
# This filter focuses the comparison on only the most critical API endpoints:
30+
# - paths: Only specific API paths that are most critical for compatibility:
31+
# - /v1/inference: Core inference endpoints
32+
# - /v1/vector-io: Vector input/output operations
33+
# - /v1/vector-dbs: Vector database operations
34+
# - /v1/openai/v1/responses: OpenAI-compatible response formats
35+
# - /v1/openai/v1/chat: OpenAI-compatible chat endpoints
36+
# anything else (openapi, info, servers, is just there to make openapi-diff happy)
37+
ENV FILTER_CONDITION='{ "openapi": .openapi, "info": .info, "servers": .servers, "tags": .tags, "x-tagGroups": .["x-tagGroups"], "components": .components, "paths": (.paths | with_entries(select(.key | test("^/(v1/inference|v1/vector-io|v1/vector-dbs|v1/openai/v1/responses|v1/openai/v1/chat)")))) }'
38+
39+
# Custom entrypoint that combines yq filtering with openapi-diff comparison
40+
# This allows us to compare only the filtered, relevant parts of the API spec
41+
ENTRYPOINT ["/bin/sh", "-c", "\
42+
echo '=== Running yq filter ===' && \
43+
# Filter the base branch OpenAPI spec to only relevant endpoints and save to base.yaml
44+
yq eval \"$FILTER_CONDITION\" /workspace/base/docs/_static/llama-stack-spec.yaml > /workspace/base.yaml && \
45+
# Filter the current revision OpenAPI spec to only relevant endpoints and save to revision.yaml
46+
yq eval \"$FILTER_CONDITION\" /workspace/docs/_static/llama-stack-spec.yaml > /workspace/revision.yaml && \
47+
echo '=== Running original openapi-diff entrypoint ===' && \
48+
# Run the OpenAPI diff tool on the filtered specs
49+
exec java -jar /app/openapi-diff.jar /workspace/base.yaml /workspace/revision.yaml --fail-on-incompatible \
50+
"]

‎.github/workflows/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ Llama Stack uses GitHub Actions for Continuous Integration (CI). Below is a tabl
55
| Name | File | Purpose |
66
| ---- | ---- | ------- |
77
| Update Changelog | [changelog.yml](changelog.yml) | Creates PR for updating the CHANGELOG.md |
8+
| API Conformance Tests | [conformance.yml](conformance.yml) | Run the API Conformance test suite on the changes. |
89
| Installer CI | [install-script-ci.yml](install-script-ci.yml) | Test the installation script |
910
| Integration Auth Tests | [integration-auth-tests.yml](integration-auth-tests.yml) | Run the integration test suite with Kubernetes authentication |
1011
| SqlStore Integration Tests | [integration-sql-store-tests.yml](integration-sql-store-tests.yml) | Run the integration test suite with SqlStore |

‎.github/workflows/conformance.yml‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# API Conformance Tests
2+
# This workflow ensures that API changes maintain backward compatibility and don't break existing integrations
3+
# It runs schema validation and OpenAPI diff checks to catch breaking changes early
4+
5+
name: API Conformance Tests
6+
7+
run-name: Run the API Conformance test suite on the changes.
8+
9+
on:
10+
push:
11+
branches: [ main ]
12+
pull_request:
13+
branches: [ main ]
14+
types: [opened, synchronize, reopened]
15+
paths:
16+
- 'llama_stack/**'
17+
- '!llama_stack/ui/**'
18+
- 'tests/**'
19+
- 'uv.lock'
20+
- 'pyproject.toml'
21+
- '.github/workflows/conformance.yml' # This workflow itself
22+
schedule:
23+
# If changing the cron schedule, update the provider in the test-matrix job
24+
- cron: '0 0 * * *' # Daily at 12 AM UTC - test latest client to catch breaking changes in dependencies
25+
- cron: '1 0 * * 0' # Weekly on Sunday at 1 AM UTC - test vllm provider specifically
26+
27+
concurrency:
28+
group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/main' && github.run_id || github.ref }}
29+
# Cancel in-progress runs when new commits are pushed to avoid wasting CI resources
30+
cancel-in-progress: true
31+
32+
jobs:
33+
# Job to check if API schema changes maintain backward compatibility
34+
check-schema-compatibility:
35+
runs-on: ubuntu-latest
36+
steps:
37+
# Using specific version 4.1.7 because 5.0.0 fails when trying to run this locally using `act`
38+
# This ensures consistent behavior between local testing and CI
39+
- name: Checkout PR Code
40+
uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
41+
42+
# Checkout the base branch to compare against (usually main)
43+
# This allows us to diff the current changes against the previous state
44+
- name: Checkout Base Branch
45+
uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
46+
with:
47+
ref: ${{ github.event.pull_request.base.ref }}
48+
path: 'base'
49+
50+
# Build a custom container that includes both openapi-diff and yq tools
51+
# The default openapi-diff container doesn't have yq, which we need for advanced YAML filtering
52+
- name: Build OpenAPI Diff container
53+
run: |
54+
docker build -t openapi-diff-yq -f ./.github/images/Containerfile .
55+
56+
# Run the OpenAPI diff tool to detect breaking changes in the API specification
57+
# This step will fail if incompatible changes are detected, preventing breaking changes from being merged
58+
- name: Run OpenAPI Diff
59+
run: |
60+
docker run --rm \
61+
-v "$(pwd):/workspace" \
62+
openapi-diff-yq

0 commit comments

Comments
 (0)