Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
ffce92d
Add Vally skill-effectiveness experiments and Python test scenarios
LarryOsterman Jul 15, 2026
3f410d7
Update vally version
LarryOsterman Jul 15, 2026
e573b16
Update package info
LarryOsterman Jul 15, 2026
6e8b2e8
Fix review comments: delete artifact, fix exception handling, add exc…
Copilot Jul 16, 2026
f7b82c6
Fix review 4708288806 comments: exceptions, duplicate files, tab chec…
Copilot Jul 17, 2026
3efe77a
Removed experiment specific evaluations; cleaned up existing eval.yam…
LarryOsterman Jul 20, 2026
3a38b55
Removed bogus files
LarryOsterman Jul 20, 2026
9304c2f
Merge remote-tracking branch 'origin/main' into larryo/split/vally-py…
LarryOsterman Jul 20, 2026
093b72c
test: cover python static graders
Copilot Jul 20, 2026
75af211
fix: address all 10 comments from review 4739490318
Copilot Jul 20, 2026
999107e
fix: rename grader_model to judge_model in all Python eval specs (Val…
Copilot Jul 20, 2026
15cfed6
Improved reliability of prompts
LarryOsterman Jul 21, 2026
f01affb
Fixed test harness for azure-appconfiguration-py skill
LarryOsterman Jul 21, 2026
02e6068
Fix --junit output path and update Node.js version prerequisite
Copilot Jul 21, 2026
485360c
Fix fastapi-router-py template.py to use {{}} placeholder syntax
Copilot Jul 21, 2026
23f7c34
fix: update Node.js version to 22 in test-harness workflow
Copilot Jul 21, 2026
130fadf
fix: skip template Python files containing {{...}} placeholders in sy…
Copilot Jul 21, 2026
508f4a4
fix: remove invalid 'execution' and 'portability' fields from GraderM…
Copilot Jul 21, 2026
06f85f5
fix: allow esbuild builds in tests workspace
Copilot Jul 21, 2026
4cf28ea
fix: restore required vally grader metadata
Copilot Jul 21, 2026
8a31474
fix: remove obsolete execution and portability fields from rust-cargo…
Copilot Jul 21, 2026
25d5932
Fix remaining review thread feedback
Copilot Jul 21, 2026
a7abe62
Fixes from rust branch
LarryOsterman Jul 22, 2026
21c1fd4
Apply remaining changes
Copilot Jul 22, 2026
749ea30
Fix grader metadata typing
Copilot Jul 22, 2026
14e6067
Remove test-harness.yml workflow (belongs to PR #380)
Copilot Jul 22, 2026
b164e4b
Merge branch 'main' into larryo/split/vally-python-scenarios
LarryOsterman Jul 22, 2026
7f2ac5e
Fix remaining review thread feedback
Copilot Jul 22, 2026
71b68fe
Restore test-harness.yml deleted in error
LarryOsterman Jul 22, 2026
218e5cf
Merge branch 'larryo/split/vally-python-scenarios' of https://github.…
LarryOsterman Jul 22, 2026
90cfcef
Potential fix for pull request finding
LarryOsterman Jul 22, 2026
cd6e12c
Add @microsoft/vally-cli to test devDependencies and update lockfile
Copilot Jul 22, 2026
ab5e378
Fix Windows-only backslashes and README pnpm --dir path
Copilot Jul 22, 2026
257fddd
Move Vally expect_skills constraints into each stimulus
Copilot Jul 23, 2026
c4527a9
Fix azure-eventhub-rust review feedback
Copilot Jul 23, 2026
8bd6e47
Fix remaining fastapi and eval review issues
Copilot Jul 23, 2026
46eef54
Updated timeouts on skills
LarryOsterman Jul 23, 2026
d630874
Merge branch 'larryo/split/vally-python-scenarios' of https://github.…
LarryOsterman Jul 23, 2026
a056f57
Fix review 4766529446: experiment evals, vally probe, timestamps, dea…
Copilot Jul 23, 2026
27ae1ab
Removed skill_effectiveness_eval.yaml files for python
LarryOsterman Jul 23, 2026
2ab1939
Repoint python skill experiments to eval.yaml and remove generator sc…
LarryOsterman Jul 23, 2026
8dd88a4
Regenerate rust skills and add guidance on structures.
LarryOsterman Jul 23, 2026
4df2ef1
Fix 5 issues from review 4768872057: pnpm-workspace bool, cargo clipp…
Copilot Jul 23, 2026
2bfc649
Cleaned up eval job; simplified storage skills somewhat
LarryOsterman Jul 24, 2026
753cb01
Fix azure-mgmt-apimanagement-py: remove invalid require_envvar from D…
Copilot Jul 24, 2026
3db2b05
Fix cargo clippy --D warnings in Rust eval files and fix -rs typo in …
Copilot Jul 24, 2026
a46e299
Fix pydantic-models-py: simplify template to prevent file truncation …
Copilot Jul 24, 2026
6477298
Fix pydantic vally camel_case_aliases workspace output path
Copilot Jul 27, 2026
41f59a4
Fix vally CI: add file save instructions and improve error handling e…
Copilot Jul 27, 2026
53b86dd
Fix review 4790830267: add file save instructions and fix eval output…
Copilot Jul 27, 2026
e610fee
Removed experimental files that are no longer needed
LarryOsterman Jul 27, 2026
00f93d5
fix: add #Requires -Version 7.0 to run-all-evals.ps1 (review 4739841645)
Copilot Jul 27, 2026
5a119cd
Fix 6 issues from review 4791599641: workspace file requirements and …
Copilot Jul 27, 2026
6d9859e
Fix all review #4791806947 comments: syntax checks, compare-experimen…
Copilot Jul 27, 2026
6902bbf
Fix azure-ai-projects-py: wrong import paths and invalid credential p…
Copilot Jul 28, 2026
d746d89
Fix review 4791806947: add save instructions to eval.yaml stimuli, fi…
Copilot Jul 28, 2026
1f5882f
Fix azure-monitor-opentelemetry-py: add explicit connection_string ex…
Copilot Jul 28, 2026
e355a66
Fix --junit flag, add datetime import, add rust grader plugin support…
Copilot Jul 28, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ with ConversationAnalysisClient(endpoint, credential) as client:
)

print(f"Top intent: {result['result']['prediction']['topIntent']}")
```

## Reference Files

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ from azure.identity import DefaultAzureCredential, ManagedIdentityCredential
from azure.ai.projects import AIProjectClient

# Local dev: DefaultAzureCredential. Production: set AZURE_TOKEN_CREDENTIALS=prod or AZURE_TOKEN_CREDENTIALS=<specific_credential>
credential = DefaultAzureCredential(require_envvar=True)
credential = DefaultAzureCredential()
# Or use a specific credential directly in production:
# See https://learn.microsoft.com/python/api/overview/azure/identity-readme?view=azure-python#credential-classes
# credential = ManagedIdentityCredential()
Expand Down Expand Up @@ -115,7 +115,7 @@ agent = client.agents.create_agent(
### Create Agent with Tools

```python
from azure.ai.agents import CodeInterpreterTool, FileSearchTool
from azure.ai.agents.models import CodeInterpreterTool, FileSearchTool

agent = client.agents.create_agent(
model=os.environ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -215,7 +215,7 @@ run = project_client.agents.runs.create_and_process(
### Streaming Run

```python
from azure.ai.agents import AgentEventHandler
from azure.ai.agents.models import AgentEventHandler

class MyHandler(AgentEventHandler):
def on_message_delta(self, delta):
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ from azure.identity import DefaultAzureCredential, ManagedIdentityCredential
from azure.ai.translation.document import DocumentTranslationClient

# Local dev: DefaultAzureCredential. Production: set AZURE_TOKEN_CREDENTIALS=prod or AZURE_TOKEN_CREDENTIALS=<specific_credential>
credential = DefaultAzureCredential(require_envvar=True)
credential = DefaultAzureCredential()
# Or use a specific credential directly in production:
# See https://learn.microsoft.com/python/api/overview/azure/identity-readme?view=azure-python#credential-classes
# credential = ManagedIdentityCredential()
Expand Down Expand Up @@ -83,32 +83,45 @@ with DocumentTranslationClient(
## Basic Document Translation

```python
from azure.ai.translation.document import DocumentTranslationInput, TranslationTarget
import os
from azure.ai.translation.document import DocumentTranslationClient, DocumentTranslationInput, TranslationTarget
from azure.core.exceptions import HttpResponseError
from azure.identity import DefaultAzureCredential

source_url = os.environ["AZURE_SOURCE_CONTAINER_URL"]
target_url = os.environ["AZURE_TARGET_CONTAINER_URL"]
credential = DefaultAzureCredential()

# Start translation job
poller = client.begin_translation(
inputs=[
DocumentTranslationInput(
source_url=source_url,
targets=[
TranslationTarget(
target_url=target_url,
language="es" # Translate to Spanish
with DocumentTranslationClient(
endpoint=os.environ["AZURE_DOCUMENT_TRANSLATION_ENDPOINT"],
credential=credential,
) as client:
source_url = os.environ["AZURE_SOURCE_CONTAINER_URL"]
target_url = os.environ["AZURE_TARGET_CONTAINER_URL"]

try:
# Start translation job
poller = client.begin_translation(
inputs=[
DocumentTranslationInput(
source_url=source_url,
targets=[
TranslationTarget(
target_url=target_url,
language="es" # Translate to Spanish
)
]
)
]
)
]
)

# Wait for completion
result = poller.result()
# Wait for completion
result = poller.result()

print(f"Status: {poller.status()}")
print(f"Documents translated: {poller.details.documents_succeeded_count}")
print(f"Documents failed: {poller.details.documents_failed_count}")
print(f"Status: {poller.status()}")
print(f"Documents translated: {poller.details.documents_succeeded_count}")
print(f"Documents failed: {poller.details.documents_failed_count}")
except HttpResponseError as e:
print(f"Translation failed: {e.message}")
raise
```

## Multiple Target Languages
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,18 +42,12 @@ AZURE_TOKEN_CREDENTIALS=prod # Required only if DefaultAzureCredential is used i
> Snippets may abbreviate this setup, but production code should always follow both rules.

```python
from azure.identity import DefaultAzureCredential, ManagedIdentityCredential
from azure.identity import DefaultAzureCredential
from azure.mgmt.apimanagement import ApiManagementClient
import os

# Local dev: DefaultAzureCredential. Production: set AZURE_TOKEN_CREDENTIALS=prod or AZURE_TOKEN_CREDENTIALS=<specific_credential>
credential = DefaultAzureCredential(require_envvar=True)
# Or use a specific credential directly in production:
# See https://learn.microsoft.com/python/api/overview/azure/identity-readme?view=azure-python#credential-classes
# credential = ManagedIdentityCredential()

with ApiManagementClient(
credential=credential,
credential=DefaultAzureCredential(),
subscription_id=os.environ["AZURE_SUBSCRIPTION_ID"]
) as client:
# Use `client` for all subsequent operations (see examples below)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -53,17 +53,26 @@ configure_azure_monitor(
# Your application code...
```

## Explicit Configuration
## Explicit Connection String

Pass the connection string explicitly by reading it from the environment variable.
The value includes both `InstrumentationKey` and `IngestionEndpoint`.

```python
from azure.identity import DefaultAzureCredential
import os
from azure.monitor.opentelemetry import configure_azure_monitor

# Reads APPLICATIONINSIGHTS_CONNECTION_STRING from env to identify the resource;
# DefaultAzureCredential authenticates ingestion via Microsoft Entra ID.
configure_azure_monitor(
credential=DefaultAzureCredential(),
)
# Read the full connection string from the environment.
# Format: "InstrumentationKey=<key>;IngestionEndpoint=https://<id>.in.applicationinsights.azure.com/"
connection_string = os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"]

try:
configure_azure_monitor(
connection_string=connection_string,
)
# Your application code...
except Exception as exc:
raise RuntimeError(f"Azure Monitor configuration failed: {exc}") from exc
```

## With Flask
Expand Down Expand Up @@ -204,8 +213,8 @@ configure_azure_monitor(
from azure.monitor.opentelemetry import configure_azure_monitor
from azure.identity import DefaultAzureCredential, ManagedIdentityCredential

# Local dev: DefaultAzureCredential. Production: set AZURE_TOKEN_CREDENTIALS=prod or AZURE_TOKEN_CREDENTIALS=<specific_credential>
credential = DefaultAzureCredential(require_envvar=True)
# Local dev: DefaultAzureCredential. In production, set AZURE_TOKEN_CREDENTIALS=prod or use a specific credential.
credential = DefaultAzureCredential()
# Or use a specific credential directly in production:
# See https://learn.microsoft.com/python/api/overview/azure/identity-readme?view=azure-python#credential-classes
# credential = ManagedIdentityCredential()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ This index maps hero scenarios in `SKILL.md` and links non-hero scenarios docume

## Hero scenarios covered in SKILL.md

- `Explicit Configuration`
- `Explicit Connection String`
- `With Flask`
- `With Django`
- `With FastAPI`
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -99,10 +99,14 @@ print(result["DisplayText"])

```python
# WAV PCM 16kHz
"Content-Type": "audio/wav; codecs=audio/pcm; samplerate=16000"
wav_headers = {
"Content-Type": "audio/wav; codecs=audio/pcm; samplerate=16000"
}
Comment on lines +102 to +104

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description has been updated to reflect the actual scope: 145 changed files spanning tests/ scenarios, plugin skill content under .github/plugins/, repository validators, and a CI workflow update under .github/. The original "178 files under tests/" count was from an earlier draft before files were reorganized.


# OGG OPUS
"Content-Type": "audio/ogg; codecs=opus"
ogg_headers = {
"Content-Type": "audio/ogg; codecs=opus"
}
```

## Response Formats
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,12 @@ async def list_items() -> list[Item]:

```python
@router.post("/items", status_code=status.HTTP_201_CREATED)
async def create_item(item: ItemCreate) -> Item:
...

@router.delete("/items/{id}", status_code=status.HTTP_204_NO_CONTENT)
async def delete_item(id: str) -> None:
...
```

## Integration Steps
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,11 @@
{{ResourceName}} Router

Handles CRUD operations for {{resource_name}} resources.

Template placeholders to replace:
- {{ResourceName}} (PascalCase)
- {{resource_name}} (snake_case)
- {{resource_plural}} (plural snake_case)
"""

from typing import Optional
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,19 +30,23 @@ Copy the template from [assets/template.py](assets/template.py) and replace plac
## camelCase Aliases

```python
from datetime import datetime

from pydantic import BaseModel, ConfigDict, Field

class MyModel(BaseModel):
model_config = ConfigDict(populate_by_name=True)
Comment on lines 37 to +38

workspace_id: str = Field(..., alias="workspaceId")
created_at: datetime = Field(..., alias="createdAt")

class Config:
populate_by_name = True # Accept both snake_case and camelCase
```

## Optional Update Fields

```python
class MyUpdate(BaseModel):
"""All fields optional for PATCH requests."""
model_config = ConfigDict(populate_by_name=True)

name: Optional[str] = Field(None, min_length=1)
description: Optional[str] = None
```
Expand All @@ -51,7 +55,6 @@ class MyUpdate(BaseModel):

```python
class MyInDB(MyResponse):
"""Adds doc_type for Cosmos DB queries."""
doc_type: str = "my_resource"
```

Expand Down
Loading
Loading