Skip to content

eval: migrate azure-deploy integration tests to vally eval suites #2833

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JasonYeMSFT (JasonYeMSFT) merged 22 commits into from
Aug 5, 2026
Merged

eval: migrate azure-deploy integration tests to vally eval suites #2833

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
6d4db53
feat: add azure-deploy e2e eval fixtures and suite
JasonYeMSFT Jul 13, 2026
98f32a2
feat: add azure-deploy routing eval suite
JasonYeMSFT Jul 13, 2026
3a84b69
feat: migrate remaining azure-deploy tests to vally evals
JasonYeMSFT Jul 13, 2026
de63121
Reorganize eval suites
JasonYeMSFT Jul 13, 2026
efa6446
eval: add azd deployment tool hint to all azure-deploy eval prompts
JasonYeMSFT Jul 14, 2026
c9f2da8
limit to 1 output test per run
JasonYeMSFT Jul 14, 2026
d97fc6b
eval: add category tags to azure-deploy deploy-eval stimuli
JasonYeMSFT Jul 15, 2026
46d7028
eval: switch earlyTerminate to tool-call-result in azure-deploy deplo…
JasonYeMSFT Jul 15, 2026
7d14b8d
eval: change earlyTerminate to detect deploy URL in assistant output
JasonYeMSFT Jul 15, 2026
9518bb7
eval: add takeScreenshot tag to azure-deploy deploy-eval stimuli
JasonYeMSFT Jul 15, 2026
a6d0dac
update CI test command
JasonYeMSFT Jul 15, 2026
ba6d31b
eval: add application code to DTS fixture for order processing workflow
JasonYeMSFT Jul 15, 2026
8bcc661
eval: rename whiteboard to test-sample-app in SWA fixture HTML
JasonYeMSFT Jul 15, 2026
1dcf20f
restore test prompts
JasonYeMSFT Jul 21, 2026
c071eb3
increase all max_turns to 80
JasonYeMSFT Jul 21, 2026
d441c96
relax IaC check path
JasonYeMSFT Jul 22, 2026
217052d
remove incorrect bicep check
JasonYeMSFT Jul 28, 2026
bef3c39
add deployed link grader
JasonYeMSFT Jul 28, 2026
2ea69d7
Merge branch 'main' into jasonyemsft-azure-deploy-eval-fixture-migration
JasonYeMSFT Jul 30, 2026
0601b37
clear eval suites
JasonYeMSFT Jul 30, 2026
11e04d4
disallow provision command for no deploy test
JasonYeMSFT Jul 30, 2026
88b0fc7
properly dispatch azure-deploy test runs
JasonYeMSFT Jul 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/test-all-integration.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ jobs:
core.setOutput("has_azure_deploy", hasAzureDeploy.toString());

// Build matrix excluding azure-deploy
const matrix = skills.filter(s => s !== "azure-deploy");
const matrix = skills.filter(s => s !== "azure-skills/azure-deploy");
core.setOutput("matrix", JSON.stringify(matrix));

# Call the azure-deploy workflow when azure-deploy is in the skills list
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/test-azure-deploy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,7 @@ jobs:
MODEL_OVERRIDE: ${{ inputs.model-override }}
TEST_GROUP: ${{ matrix.test-group }}
run: |
npm run test:integration -- azure-deploy "$TEST_GROUP"
npm run test:vally -- --plugin azure-skills --skill azure-deploy --tag "category=$TEST_GROUP"
continue-on-error: true

- name: Generate report
Expand Down
Loading
Loading