Sync .github/skills directory with azure-sdk-tools for PR 16380#44771
Open
azure-sdk-automation[bot] wants to merge 5 commits into
Open
Sync .github/skills directory with azure-sdk-tools for PR 16380#44771azure-sdk-automation[bot] wants to merge 5 commits into
azure-sdk-automation[bot] wants to merge 5 commits into
Conversation
Wires environment.skills: [<skill dirs>] into the mock workflow eval YAML specs so Vally loads the relevant skill per-eval instead of relying on --skill-dir at the pipeline level.
- Add missing environment.skills block to sdk-local-update-metadata-001 (all other positive stimuli in this file already had it) - Replace ambiguous/fictional 'contoso' package paths with the real fixture package (sdk/compute/azure-mgmt-compute, azure-resourcemanager-compute) so the model doesn't need to ask for clarification on a made-up path - Add explicit version/release-type/date to the version-update stimuli since the tool genuinely requires that info Verified locally: suite score went from ~42% to consistently 83-92% across repeated runs (threshold 80%).
The trigger.eval.yaml suite reproducibly failed at ~70%% because vague/ informational prompts (SDK release checklist, walk me through the release process, how to publish a new SDK version, release a new version of azure-sdk) weren't covered by the skill's narrow trigger phrase list, so the model didn't reliably invoke the skill. Added those phrasings to the description, Triggers section, and Examples, and clarified that the skill should also handle how-to/ checklist style questions, not just direct action requests. Verified locally: trigger.eval.yaml went from 70%% to 100%% across two repeated runs (threshold 80%%).
Per Copilot PR review feedback: anti-trigger and negative stimuli in these eval specs didn't set environment.skills, so the skill was never loaded/available for those prompts. That made the disallowed skill-invocation / tool-calls grader checks trivially true (the skill literally couldn't be invoked), providing no real signal. Added environment.skills: [..] to all anti-trigger/negative stimuli in: - azsdk-common-apiview-feedback-resolution/evals/trigger.eval.yaml - azsdk-common-pipeline-fixer/evals/trigger.eval.yaml - azsdk-common-pipeline-analysis/evals/trigger.eval.yaml - azsdk-common-prepare-release-plan/evals/trigger.eval.yaml - azsdk-common-sdk-release/evals/trigger.eval.yaml - azsdk-common-generate-sdk-locally/evals/eval.yaml (sdk-local-negative-001) Verified locally: all 6 suites still pass with the skill loaded and available - the model correctly declines to invoke it on out-of-scope prompts. No new failures introduced (5 trigger suites at 100%, generate-sdk-locally-eval at 91.7%, same pre-existing unrelated failure as before on sdk-local-analyzer-errors-001).
azure-sdk-automation
Bot
requested review from
a team,
ArthurMa1978,
chunyu3,
haolingdong-msft and
lirenhe
as code owners
July 17, 2026 19:16
azure-sdk-automation
Bot
added
EngSys
microsoft-github-policy-service
Bot
added
the
customer-reported
Contributor
|
Thank you for your contribution @azure-sdk-automation[bot]! We will review the pull request and get back to you soon. |
Next Steps to Merge✅ All automated merging requirements have been met! To get your PR merged, see aka.ms/azsdk/specreview/merge.Comment generated by summarize-checks workflow run. |
|
Azure Pipelines: Successfully started running 10 pipeline(s). 6 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
helen229
approved these changes
Jul 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sync .github/skills directory with azure-sdk-tools for PR Azure/azure-sdk-tools#16380 See eng/common workflow