eval: migrate azure-deploy integration tests to vally eval suites#2833
Draft
JasonYeMSFT wants to merge 12 commits into
Draft
eval: migrate azure-deploy integration tests to vally eval suites#2833JasonYeMSFT wants to merge 12 commits into
JasonYeMSFT wants to merge 12 commits into
Conversation
Identified 20 integration tests in tests/azure-deploy/integration.test.ts
that started from empty workspaces and asked the agent to both create and
deploy applications. This conflates azure-prepare (scaffolding) and
azure-deploy (running azd up) responsibilities.
Add per-scenario fixtures under evals/azure-deploy/fixture/ so each vally
stimulus provides a pre-built azure.yaml + infra/ and the agent is only
evaluated on deployment execution:
Fixtures added:
- swa-bicep/ Static Web App with Bicep infra
- app-service-bicep/ App Service with Bicep infra
- functions-bicep/ Azure Functions with Bicep infra
- dts-bicep/ Durable Task Scheduler with Bicep infra (DTS-specific
Microsoft.DurableTask/* resources + role 0ad04412)
- container-apps-bicep/ Container Apps with Bicep infra (two-phase pattern:
SystemAssigned identity, placeholder image, AcrPull 7f951dda)
- swa-terraform/ Static Web App with Terraform infra
- app-service-terraform/ App Service with Terraform infra
- functions-terraform/ Azure Functions with Terraform infra
- container-apps-terraform/ Container Apps with Terraform infra
(SystemAssigned identity, AcrPull, lifecycle ignore_changes)
New eval suite: evals/azure-deploy/e2e-eval.yaml
Migrates all 20 vanilla/terraform integration tests. Each stimulus:
- Loads the appropriate fixture into the agent environment
- Prompts "deploy this prepared app" (not "create and deploy")
- Uses earlyTerminate on azd up/provision
- Graders check azure-deploy skill invocation + azd command execution
- Container Apps stimuli also verify correct IaC patterns in fixture files
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 01827688-80a4-46ef-90c2-15e7de402bc4
Migrate describe('skill-invocation') tests from
tests/azure-deploy/integration.test.ts to routing-eval.yaml.
Four routing stimuli, each with a matching pre-built fixture so the
agent has concrete project context:
- 'Deployment Prompt' (swa-bicep fixture, tier: smoke)
- 'Publish to Azure Prompt' (swa-bicep fixture, tier: full)
- 'Azure Functions Deployment Prompt' (functions-bicep fixture, tier: full)
- 'Post-Deployment Role Assignment Prompt' (container-apps-bicep, tier: full)
All use earlyTerminate on skill-call or tool-call-count:10, and
skill-invocation grader requiring azure-deploy.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 01827688-80a4-46ef-90c2-15e7de402bc4
output-eval.yaml:
- Add 'AZD Pattern Module Priority - Container App' stimulus from
tests/azure-deploy/avm/integration.test.ts describe('avm-azd-pattern-preference').
Graders check AZD pattern keyword presence and pattern-before-resource ordering.
deploy-eval.yaml:
- Bump default timeout 45m → 55m to cover brownfield scenarios.
- Add 13 brownfield stimuli across describe('brownfield-dotnet'),
describe('brownfield-javascript'), describe('brownfield-python'):
brownfield-dotnet (8):
- Brownfield .NET - eShop (git clone dotnet/eShop)
- Brownfield .NET - MvcMovie 10 (sparse checkout AspNetCore.Docs)
- Brownfield .NET Aspire - Azure Functions
- Brownfield .NET Aspire - Client Apps Integration
- Brownfield .NET Aspire - Container Build (+file-matches on two-phase Bicep pattern)
- Brownfield .NET Aspire - Custom Resources (No Deploy) (disallows azd up)
- Brownfield .NET Aspire - Database Containers
- Brownfield .NET Aspire - Orleans Voting (+output-matches for container env vars)
brownfield-javascript (3):
- Brownfield JavaScript - nodejs-demoapp (git clone benc-uk/nodejs-demoapp)
- Brownfield .NET Aspire - JavaScript
- Brownfield .NET Aspire - Node
brownfield-python (2):
- Brownfield Python - Flask Calculator (git clone UltiRequiem/flask-calculator)
- Brownfield .NET Aspire - Python
All brownfield stimuli use environment.commands (git clone / sparse-checkout),
earlyTerminate on azd up/provision, and require azure-prepare + azure-deploy
skill invocations — matching the full pipeline tested by the original Jest tests.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 01827688-80a4-46ef-90c2-15e7de402bc4
Contributor
There was a problem hiding this comment.
Pull request overview
This PR migrates azure-deploy integration coverage from Jest-based integration tests to Vally eval suites under evals/azure-deploy/, introducing prepared fixture projects (azure.yaml + infra/) so deployment tests focus on azure-deploy rather than conflating azure-prepare responsibilities.
Changes:
- Added a routing eval suite (
routing-eval.yaml) to measureazure-deployskill invocation rate for deployment-style prompts. - Added/expanded E2E deployment eval coverage (
e2e-eval.yaml,deploy-eval.yaml) including vanilla (fixture-based) and brownfield (repo-clone) scenarios. - Added multiple Bicep/Terraform fixture projects (Static Web Apps, App Service, Functions, Durable Task Scheduler, Container Apps) used by the eval stimuli.
Show a summary per file
| File | Description |
|---|---|
| evals/azure-deploy/routing-eval.yaml | Vally routing/skill-invocation stimuli for azure-deploy. |
| evals/azure-deploy/output-eval.yaml | Output-quality eval suite for azure-deploy (AVM/AZD guidance). |
| evals/azure-deploy/e2e-eval.yaml | Fixture-based E2E deploy eval stimuli (vanilla + terraform scenarios). |
| evals/azure-deploy/deploy-eval.yaml | Full deploy eval suite (fixture-based + brownfield repo-clone scenarios). |
| evals/azure-deploy/fixture/swa-terraform/infra/main.tf | Terraform SWA infra fixture used by deploy evals. |
| evals/azure-deploy/fixture/swa-terraform/index.html | SWA sample static content fixture (Terraform scenario). |
| evals/azure-deploy/fixture/swa-terraform/azure.yaml | azd app definition for SWA Terraform fixture. |
| evals/azure-deploy/fixture/swa-bicep/infra/resources.bicep | Bicep SWA resource module fixture. |
| evals/azure-deploy/fixture/swa-bicep/infra/main.parameters.json | Parameters template for SWA Bicep fixture. |
| evals/azure-deploy/fixture/swa-bicep/infra/main.bicep | Bicep SWA subscription-scope entrypoint fixture. |
| evals/azure-deploy/fixture/swa-bicep/index.html | SWA sample static content fixture (Bicep scenario). |
| evals/azure-deploy/fixture/swa-bicep/azure.yaml | azd app definition for SWA Bicep fixture. |
| evals/azure-deploy/fixture/functions-terraform/infra/main.tf | Terraform Functions infra fixture used by deploy evals. |
| evals/azure-deploy/fixture/functions-terraform/host.json | Azure Functions host metadata fixture (Terraform scenario). |
| evals/azure-deploy/fixture/functions-terraform/azure.yaml | azd app definition for Functions Terraform fixture. |
| evals/azure-deploy/fixture/functions-bicep/package.json | Functions app package metadata fixture (Bicep scenario). |
| evals/azure-deploy/fixture/functions-bicep/infra/resources.bicep | Bicep Functions resource module fixture. |
| evals/azure-deploy/fixture/functions-bicep/infra/main.parameters.json | Parameters template for Functions Bicep fixture. |
| evals/azure-deploy/fixture/functions-bicep/infra/main.bicep | Bicep Functions subscription-scope entrypoint fixture. |
| evals/azure-deploy/fixture/functions-bicep/host.json | Azure Functions host metadata fixture (Bicep scenario). |
| evals/azure-deploy/fixture/functions-bicep/azure.yaml | azd app definition for Functions Bicep fixture. |
| evals/azure-deploy/fixture/dts-bicep/infra/resources.bicep | Bicep Durable Task Scheduler infra fixture. |
| evals/azure-deploy/fixture/dts-bicep/infra/main.parameters.json | Parameters template for DTS Bicep fixture. |
| evals/azure-deploy/fixture/dts-bicep/infra/main.bicep | Bicep DTS subscription-scope entrypoint fixture. |
| evals/azure-deploy/fixture/dts-bicep/host.json | Azure Functions host metadata fixture for DTS scenario. |
| evals/azure-deploy/fixture/dts-bicep/azure.yaml | azd app definition for DTS Bicep fixture. |
| evals/azure-deploy/fixture/container-apps-terraform/package.json | Container Apps sample app metadata fixture (Terraform scenario). |
| evals/azure-deploy/fixture/container-apps-terraform/infra/main.tf | Terraform Container Apps infra fixture. |
| evals/azure-deploy/fixture/container-apps-terraform/Dockerfile | Container image build fixture for Terraform CA scenario. |
| evals/azure-deploy/fixture/container-apps-terraform/azure.yaml | azd app definition for Container Apps Terraform fixture. |
| evals/azure-deploy/fixture/container-apps-terraform/app.js | Container Apps sample Node app fixture (Terraform scenario). |
| evals/azure-deploy/fixture/container-apps-bicep/package.json | Container Apps sample app metadata fixture (Bicep scenario). |
| evals/azure-deploy/fixture/container-apps-bicep/infra/resources.bicep | Bicep Container Apps resource module fixture. |
| evals/azure-deploy/fixture/container-apps-bicep/infra/main.parameters.json | Parameters template for Container Apps Bicep fixture. |
| evals/azure-deploy/fixture/container-apps-bicep/infra/main.bicep | Bicep Container Apps subscription-scope entrypoint fixture. |
| evals/azure-deploy/fixture/container-apps-bicep/infra/abbreviations.json | Naming abbreviation inputs for Bicep CA fixture. |
| evals/azure-deploy/fixture/container-apps-bicep/Dockerfile | Container image build fixture for Bicep CA scenario. |
| evals/azure-deploy/fixture/container-apps-bicep/azure.yaml | azd app definition for Container Apps Bicep fixture. |
| evals/azure-deploy/fixture/container-apps-bicep/app.js | Container Apps sample Node app fixture (Bicep scenario). |
| evals/azure-deploy/fixture/app-service-terraform/package.json | App Service sample app metadata fixture (Terraform scenario). |
| evals/azure-deploy/fixture/app-service-terraform/infra/main.tf | Terraform App Service infra fixture. |
| evals/azure-deploy/fixture/app-service-terraform/azure.yaml | azd app definition for App Service Terraform fixture. |
| evals/azure-deploy/fixture/app-service-terraform/app.js | App Service sample Node app fixture (Terraform scenario). |
| evals/azure-deploy/fixture/app-service-bicep/package.json | App Service sample app metadata fixture (Bicep scenario). |
| evals/azure-deploy/fixture/app-service-bicep/infra/resources.bicep | Bicep App Service resource module fixture. |
| evals/azure-deploy/fixture/app-service-bicep/infra/main.parameters.json | Parameters template for App Service Bicep fixture. |
| evals/azure-deploy/fixture/app-service-bicep/infra/main.bicep | Bicep App Service subscription-scope entrypoint fixture. |
| evals/azure-deploy/fixture/app-service-bicep/azure.yaml | azd app definition for App Service Bicep fixture. |
| evals/azure-deploy/fixture/app-service-bicep/app.js | App Service sample Node app fixture (Bicep scenario). |
Review details
Comments suppressed due to low confidence (1)
evals/azure-deploy/output-eval.yaml:12
- This eval file defines
name: azure-deploy-eval, which duplicates the existingevals/azure-deploy/eval.yamlsuite name. The Vally runner (tests/run-vally-test.ts) loads all.yamlfiles underevals/<skill>/, so this will run the same suite twice and merge results under the same normalized test names, skewing pass rates and increasing runtime/cost.
- Files reviewed: 48/48 changed files
- Comments generated: 2
- Review effort level: Low
Comment on lines
+17
to
+20
| name: azure-deploy-e2e-eval | ||
| description: | | ||
| End-to-end evaluation suite for the azure-deploy skill. | ||
| Tests deployment execution quality across: |
Comment on lines
+1210
to
+1214
| - type: tool-calls | ||
| config: | ||
| disallowed: | ||
| - name: "(?i)^(bash|powershell|pwsh)$" | ||
| command: "(?i)azd\\s+(up|deploy)\\b" |
Add 'Use azd as the deployment tool.' to all 40 stimuli across: - deploy-eval.yaml (33 prompts) - routing-eval.yaml (4 prompts) - output-eval.yaml (3 prompts) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 01827688-80a4-46ef-90c2-15e7de402bc4
Map each stimulus back to its original Jest describe group: - vanilla-static-web-apps-deploy (2) - vanilla-app-service-deploy (2) - vanilla-azure-functions-deploy (3) - durable-task-scheduler-deploy (1) - vanilla-azure-container-apps-deploy (2) - terraform-static-web-apps-deploy (2) - terraform-app-service-deploy (2) - terraform-azure-functions-deploy (3) - terraform-azure-container-apps-deploy (3) - brownfield-dotnet (8) - brownfield-javascript (3) - brownfield-python (2) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 01827688-80a4-46ef-90c2-15e7de402bc4
…y-eval 32 stimuli had both a tool-calls grader checking for azd up/provision and an earlyTerminate using tool-call-match on the same command. tool-call-match fires when the call starts (before execution), so the grader may not see a completed call in the trajectory. Switching to tool-call-result ensures early termination happens after the tool call completes, allowing the tool-calls grader to evaluate the result. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 01827688-80a4-46ef-90c2-15e7de402bc4
Replace azd up/provision tool-call-result earlyTerminate with assistant-message-match on Azure deployment link patterns: - *.azurewebsites.net (App Service) - *.azurestaticapps.net (Static Web Apps) - *.azurecontainerapps.io (Container Apps) - *.web.core.windows.net (Storage static website) This matches the original shouldEarlyTerminateForCompletedDeployment logic in tests/azure-deploy/utils.ts. Tests now run through actual deployment and terminate when the agent surfaces a live app URL. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 01827688-80a4-46ef-90c2-15e7de402bc4
Add has-deployment-url takeScreenshot condition to the 17 stimuli that had takeScreenshot configured in the original Jest tests. Screenshot triggers when the agent outputs any Azure deployment link: - *.azurewebsites.net (App Service) - *.azurestaticapps.net (Static Web Apps) - *.azurecontainerapps.io (Container Apps) - *.web.core.windows.net (Storage static website) Stimuli with takeScreenshot (17): - SWA Bicep (2), App Service Bicep (2) - SWA Terraform (2), App Service Terraform (2) - Brownfield .NET: eShop, MvcMovie 10, Aspire Azure Functions, Aspire Client Apps Integration - Brownfield JavaScript: nodejs-demoapp, Aspire JS, Aspire Node - Brownfield Python: Flask Calculator, Aspire Python Stimuli without takeScreenshot (16): - Azure Functions (Bicep/Terraform), DTS, Container Apps (Bicep/Terraform), Brownfield .NET: Container Build, Custom Resources, DB Containers, Orleans Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 01827688-80a4-46ef-90c2-15e7de402bc4
The dts-bicep fixture was missing Node.js app code. The azure.yaml declares language:js host:function, so azd expects deployable source. Added (Azure Functions v4 model): - package.json — @azure/functions v4, @azure/durabletask deps - src/functions/orderOrchestrator.js — orchestrator coordinating 3 sequential steps: validateOrder → processPayment → sendConfirmation - src/functions/orderActivities.js — the three activity functions - src/functions/startOrder.js — HTTP POST trigger to start the pipeline Updated deploy-eval.yaml to include all new files in the stimulus environment.files list. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 01827688-80a4-46ef-90c2-15e7de402bc4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Migrates all 40 integration tests from
tests/azure-deploy/**/integration.test.tsto vally eval suites underevals/azure-deploy/.Key changes:
evals/azure-deploy/routing-eval.yaml— 4 routing (skill-invocation) stimuli,runs: 5, threshold 0.8evals/azure-deploy/output-eval.yaml— 3 AVM output-quality stimuli (2 existing + 1 new AZD pattern preference fromavm/integration.test.ts)evals/azure-deploy/deploy-eval.yaml— 33 E2E stimuli:environment.commands(git clone/sparse-checkout of real repos)Fixture directories created for vanilla tests:
swa-bicep,app-service-bicep,functions-bicep,dts-bicep,container-apps-bicep,swa-terraform,app-service-terraform,functions-terraform,container-apps-terraformThe core problem addressed: vanilla tests previously conflated
azure-prepareandazure-deployresponsibilities by starting from empty workspaces. Fixtures now provide pre-builtazure.yaml+ infra so tests focus on the deploy skill.Checklist
cd tests && npm test)fix:,feat:,feature:,chore:,misc:,test:,eval:Related Issues