Skip to content

eval: add max turn limit for foundry skill e2e #3060

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
anchenyi merged 2 commits into from
Aug 12, 2026
Merged

eval: add max turn limit for foundry skill e2e #3060

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c21c404
chore: add max turn limitation and change region
anchenyi Aug 11, 2026
40e9338
chore: update max turn limit
anchenyi Aug 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions evals/azure-skills/microsoft-foundry/eval.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -486,6 +486,8 @@ stimuli:

# ── Foundry E2E checks ──
- name: "Golden Path - Create and deploy hosted agent"
constraints:
max_turns: 50
tags:
id: golden-path-create-and-deploy-hosted-agent
type: foundry-e2e
Expand All @@ -496,7 +498,7 @@ stimuli:
Create a Python hosted agent for B2B customer onboarding and deploy it to a new Foundry project. Use the Responses protocol. After it is done, run in locally to make sure it can run successfully; then deploy it to foundry and ensure it can respond to users correctly.

Foundry model: gpt-5.4-nano
Region: North Central US
Region: eastus2
Comment thread
anchenyi marked this conversation as resolved.
graders:
- type: skill-invocation
config:
Expand All @@ -520,6 +522,8 @@ stimuli:
files:
- src: fixture/openai-agents-sdk
dest: .
constraints:
max_turns: 70
tags:
id: migration-openai-agents-sdk-to-foundry
type: foundry-e2e
Expand All @@ -530,7 +534,7 @@ stimuli:
This project is our existing Python customer-support agent built using OpenAI Agents SDK and self-hosted as a container on our internal platform. Re-host it on Microsoft Foundry with the minimum code changes necessary, preserving its existing architecture and behavior. Run it locally to make sure it works, create a new Foundry project with Foundry models and deploy the agent there, then invoke the deployed agent to make sure it works after deployment.

Foundry model: gpt-5.4-nano
Region: North Central US
Region: eastus2
graders:
- type: skill-invocation
config:
Expand Down
Loading