-
Notifications
You must be signed in to change notification settings - Fork 323
Add Vally skill-effectiveness experiments and Python test scenarios #383
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
58 commits
Select commit
Hold shift + click to select a range
ffce92d
Add Vally skill-effectiveness experiments and Python test scenarios
LarryOsterman 3f410d7
Update vally version
LarryOsterman e573b16
Update package info
LarryOsterman 6e8b2e8
Fix review comments: delete artifact, fix exception handling, add exc…
Copilot f7b82c6
Fix review 4708288806 comments: exceptions, duplicate files, tab chec…
Copilot 3efe77a
Removed experiment specific evaluations; cleaned up existing eval.yam…
LarryOsterman 3a38b55
Removed bogus files
LarryOsterman 9304c2f
Merge remote-tracking branch 'origin/main' into larryo/split/vally-py…
LarryOsterman 093b72c
test: cover python static graders
Copilot 75af211
fix: address all 10 comments from review 4739490318
Copilot 999107e
fix: rename grader_model to judge_model in all Python eval specs (Val…
Copilot 15cfed6
Improved reliability of prompts
LarryOsterman f01affb
Fixed test harness for azure-appconfiguration-py skill
LarryOsterman 02e6068
Fix --junit output path and update Node.js version prerequisite
Copilot 485360c
Fix fastapi-router-py template.py to use {{}} placeholder syntax
Copilot 23f7c34
fix: update Node.js version to 22 in test-harness workflow
Copilot 130fadf
fix: skip template Python files containing {{...}} placeholders in sy…
Copilot 508f4a4
fix: remove invalid 'execution' and 'portability' fields from GraderM…
Copilot 06f85f5
fix: allow esbuild builds in tests workspace
Copilot 4cf28ea
fix: restore required vally grader metadata
Copilot 8a31474
fix: remove obsolete execution and portability fields from rust-cargo…
Copilot 25d5932
Fix remaining review thread feedback
Copilot a7abe62
Fixes from rust branch
LarryOsterman 21c1fd4
Apply remaining changes
Copilot 749ea30
Fix grader metadata typing
Copilot 14e6067
Remove test-harness.yml workflow (belongs to PR #380)
Copilot b164e4b
Merge branch 'main' into larryo/split/vally-python-scenarios
LarryOsterman 7f2ac5e
Fix remaining review thread feedback
Copilot 71b68fe
Restore test-harness.yml deleted in error
LarryOsterman 218e5cf
Merge branch 'larryo/split/vally-python-scenarios' of https://github.…
LarryOsterman 90cfcef
Potential fix for pull request finding
LarryOsterman cd6e12c
Add @microsoft/vally-cli to test devDependencies and update lockfile
Copilot ab5e378
Fix Windows-only backslashes and README pnpm --dir path
Copilot 257fddd
Move Vally expect_skills constraints into each stimulus
Copilot c4527a9
Fix azure-eventhub-rust review feedback
Copilot 8bd6e47
Fix remaining fastapi and eval review issues
Copilot 46eef54
Updated timeouts on skills
LarryOsterman d630874
Merge branch 'larryo/split/vally-python-scenarios' of https://github.…
LarryOsterman a056f57
Fix review 4766529446: experiment evals, vally probe, timestamps, dea…
Copilot 27ae1ab
Removed skill_effectiveness_eval.yaml files for python
LarryOsterman 2ab1939
Repoint python skill experiments to eval.yaml and remove generator sc…
LarryOsterman 8dd88a4
Regenerate rust skills and add guidance on structures.
LarryOsterman 4df2ef1
Fix 5 issues from review 4768872057: pnpm-workspace bool, cargo clipp…
Copilot 2bfc649
Cleaned up eval job; simplified storage skills somewhat
LarryOsterman 753cb01
Fix azure-mgmt-apimanagement-py: remove invalid require_envvar from D…
Copilot 3db2b05
Fix cargo clippy --D warnings in Rust eval files and fix -rs typo in …
Copilot a46e299
Fix pydantic-models-py: simplify template to prevent file truncation …
Copilot 6477298
Fix pydantic vally camel_case_aliases workspace output path
Copilot 41f59a4
Fix vally CI: add file save instructions and improve error handling e…
Copilot 53b86dd
Fix review 4790830267: add file save instructions and fix eval output…
Copilot e610fee
Removed experimental files that are no longer needed
LarryOsterman 00f93d5
fix: add #Requires -Version 7.0 to run-all-evals.ps1 (review 4739841645)
Copilot 5a119cd
Fix 6 issues from review 4791599641: workspace file requirements and …
Copilot 6d9859e
Fix all review #4791806947 comments: syntax checks, compare-experimen…
Copilot 6902bbf
Fix azure-ai-projects-py: wrong import paths and invalid credential p…
Copilot d746d89
Fix review 4791806947: add save instructions to eval.yaml stimuli, fi…
Copilot 1f5882f
Fix azure-monitor-opentelemetry-py: add explicit connection_string ex…
Copilot e355a66
Fix --junit flag, add datetime import, add rust grader plugin support…
Copilot File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The PR description has been updated to reflect the actual scope: 145 changed files spanning
tests/scenarios, plugin skill content under.github/plugins/, repository validators, and a CI workflow update under.github/. The original "178 files undertests/" count was from an earlier draft before files were reorganized.