eval: use vally turns for multi-turn tests#2806
Open
JasonYeMSFT wants to merge 3 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Migrates Vally integration eval suites from the legacy tags.followUp multi-turn mechanism to Vally’s native turns: format, and updates the custom executor + validators/types accordingly.
Changes:
- Update the Vally executor to run multi-turn prompts using
stimulus.turns(and mark executor as multi-turn / prepared-workspace capable). - Remove
followUptag plumbing (tag helper, stimulus validator logic, and result typing). - Convert multi-turn stimuli in affected eval suites to
turns:arrays.
Show a summary per file
| File | Description |
|---|---|
| tests/vally/vally-executor.ts | Switches multi-turn handling to stimulus.turns and passes follow-up prompts into the agent runner. |
| tests/vally/tag-helpers.ts | Removes getFollowUp tag helper (no longer needed with turns:). |
| tests/run-vally-test.ts | Removes followUp from the typed tag shape for Vally output processing. |
| scripts/src/vally/validate-stimulus.ts | Drops validation logic for tags.followUp. |
| evals/azure-upgrade/eval.yaml | Converts relevant stimuli from prompt + followUp to turns:. |
| evals/azure-resource-visualizer/eval.yaml | Converts stimuli to turns: (comments still need to be updated to stop referencing followUp). |
| evals/azure-reliability/eval.yaml | Converts multi-turn flows to turns: (file header docs still reference followUp). |
| evals/azure-prepare/e2e-eval.yaml | Converts multi-turn tests to turns: and removes followUp usage (one whitespace-only line remains). |
| evals/azure-enterprise-infra-planner/eval.yaml | Converts multi-turn flows to turns: (some comments/formatting still reference followUp / have trailing whitespace). |
Review details
- Files reviewed: 9/9 changed files
- Comments generated: 12
- Review effort level: Low
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Migrates multi-turn eval tests across multiple skills to use vally turns format. Removes the single-turn stimulus validation logic from scripts and tag helpers, and updates the vally executor to handle the new format. The change only affects how the user prompt input gets passed to the executor. Test results won't be affected.
Affected evals: azure-enterprise-infra-planner, azure-prepare, azure-reliability, azure-resource-visualizer, and azure-upgrade.
Checklist
cd tests && npm test)fix:,feat:,feature:,chore:,misc:,test:,eval:tests/,npm run test:integration -- <skill>ornpm run test:vally -- --skill <skill>)Related Issues