Copilot Studio Agent Health Check - #283
Conversation
Initial addition of the Agent Architecture Advisor skill for Copilot Studio and Azure AI Foundry. Includes agent reasoning instructions (SKILL.md), human-facing overview (README.md), metadata, Python analysis scripts (ingest_agent.py, analyze_topics.py, cost_model.py), requirements.txt, and comprehensive reference documentation. Enables agent architecture review, design, migration, and cost modeling with evidence-based recommendations.
…and detail on functionality
There was a problem hiding this comment.
Pull request overview
Adds a new submission under submissions/agent-architecture-advisor/ providing an “Agent Architecture Advisor” skill: a structured, rules-driven architecture review/design workflow (with references) plus deterministic Python helpers for ingestion, topic analysis, and cost/break-even modeling.
Changes:
- Introduces the
agent-architecture-advisorskill (SKILL.md) with multi-mode routing (Design / Review / Modernize) and an opinionated report contract. - Adds deterministic analysis scripts (
ingest_agent.py,analyze_topics.py,cost_model.py) plus an optionalrequirements.txt. - Adds a substantial set of reference materials (
references/*) and a human-facingREADME.md, plusmetadata.jsonfor gallery listing.
Reviewed changes
Copilot reviewed 20 out of 20 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| submissions/agent-architecture-advisor/SKILL.md | Skill instructions: mode routing + shared reasoning core + report/output contract. |
| submissions/agent-architecture-advisor/scripts/requirements.txt | Optional dependency list (PyYAML) for higher-fidelity YAML parsing. |
| submissions/agent-architecture-advisor/scripts/ingest_agent.py | Ingests Copilot Studio/Foundry exports into a normalized JSON model for downstream analysis. |
| submissions/agent-architecture-advisor/scripts/analyze_topics.py | Deterministic topic-level measurements (collisions, orchestration graph, config checks, complexity, variables, grounding). |
| submissions/agent-architecture-advisor/scripts/cost_model.py | Deterministic cost model + break-even calculation with rate-card overrides. |
| submissions/agent-architecture-advisor/references/token-optimization.md | Advisory guidance for token/weight cost levers and how to present them. |
| submissions/agent-architecture-advisor/references/requirement-model.md | Requirement schema + elicitation/inference guidance shared across modes. |
| submissions/agent-architecture-advisor/references/reference-links.json | Allowlisted “source of truth” external links for guidance/pricing fetch protocol. |
| submissions/agent-architecture-advisor/references/rate-card.md | Human-readable explanation of the cost model’s scaffold rates/assumptions. |
| submissions/agent-architecture-advisor/references/rate-card.json | Machine-readable fallback rates/assumptions consumed by cost_model.py. |
| submissions/agent-architecture-advisor/references/platform-capabilities.md | Platform capability envelope reference used for ceiling classification. |
| submissions/agent-architecture-advisor/references/output-templates.md | Canonical report template and mode-specific adjustments. |
| submissions/agent-architecture-advisor/references/migration-classic-to-agentic.md | Reference for classic→agentic modernization (Mode C) mapping and outputs. |
| submissions/agent-architecture-advisor/references/hybrid-patterns.md | EXTEND/hybrid reference architectures and boundary contract requirements. |
| submissions/agent-architecture-advisor/references/harness-rules.md | Skill scope harness + agentic harness capability model + operating loop rules. |
| submissions/agent-architecture-advisor/references/foundry-design-templates.md | Foundry target-state spec templates for EXTEND/MIGRATE outputs. |
| submissions/agent-architecture-advisor/references/ceiling-rules.md | Ruleset for classifying findings (GOVERNANCE/CEILING/COST/DESIGN/CONFIG) and deriving verdicts. |
| submissions/agent-architecture-advisor/references/agent-quality-dimensions.md | Cross-cutting operational dimensions and their mapping to rules. |
| submissions/agent-architecture-advisor/README.md | Human-facing overview, usage positioning, prompts, and installation notes. |
| submissions/agent-architecture-advisor/metadata.json | Gallery metadata (name/description/platforms/tags/authors/version/dates). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
@microsoft-github-policy-service agree |
Explicit addition of foundry exported agent file Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.
Suppressed comments (3)
submissions/agent-architecture-advisor/SKILL.md:83
- This section says “five batched rounds” but then lists 6 rounds (1–6). The mismatch can cause the skill to skip the operational-readiness elicitation step.
Elicit the requirement model through five batched rounds. Ask 3–5 questions per round
submissions/agent-architecture-advisor/README.md:11
- The README claims the skill can be used in “GitHub Copilot”, but this repo’s submission metadata platform list appears limited to values like “Copilot Studio”, “Cowork”, and “Scout” (and the PR metadata lists only those). This creates a support/expectations mismatch for gallery consumers.
This skill can be used across Copilot Studio, Copilot Cowork, Scout, and GitHub
Copilot. The same architecture-review reasoning applies regardless of which
host invokes it.
submissions/agent-architecture-advisor/scripts/ingest_agent.py:656
- In the .yaml/.yml ingestion path, the file is re-read and
ingest_foundry(...)is called unconditionally. This overwrites the earlier Copilot Studio topic parse (format/platform) and can also add spurious parse warnings/unreadable entries when PyYAML is absent.
text = path.read_text(encoding="utf-8", errors="replace")
ingest_foundry(text, model, path.name)
|
Thanks for putting this together, Anamika Kumari (@Annamika) ! There's real depth here and a thoughtful mapping of the process from ideation to production-ready agent building, and the questions one should ask themselves in between. My main thought here is on scope. The way we tend to frame it: a skill is a reusable procedure that earns its place on one specific job. Reading this end to end, one trigger opens onto a whole practice, review, design, diagnosis, platform and design choice, harness upgrade considerations for existing agents (in the case of MCS), and cost at scale, each with its own machinery. That is closer to an army of skills around one subject than a single procedure, as you can see with some of those specific jobs living as a standalone skill elsewhere in the gallery. I believe holding that much in one place is also what makes the parts start to disagree: harness-rules.md talk about extended agentic loop and orchestration capabilities in Copilot Studio's GHCP harness, while hybrid-patterns.md still treats some of these as a ceiling for Copilot Studio and reframes its scope differently. As much as decision trees are very helpful for users, from an architectural advisory standpoint we try to be careful with scenario>design strict patterns. Similar concern applies to the harness migration/upgrade: a fixed 1:1 mapping in a space that is moving fast, and tools like https://github.com/[microsoft/copilot-studio-plugin](https://github.com/microsoft/copilot-studio-plugin) are built to carry exactly that mapping forward (even for terminology matters, as we have moved from "migrating" from "classic" to "upgrading" from "standard harness". So rather than a list of edits, my suggestion is to let each of these procedures be the size it actually is, a focused review, a modernizer that defers to the plugin, a cost model that stands alone. Scoped that way the contradictions mostly dissolve on their own, and the strong ideas at the center get enough room. Happy to talk it through. Thank you! |
|
Hi Adriana Trujillo (@adrianatruji) Thanks for the review the scope point is fair, and it holds up against the gallery. Most of what I bundled already exists as focused skills (AI Platform Advisor, EADA, Harness Picker, Topic Blueprint, the test planners, Knowledge Readiness). One trigger over all of that is the problem you flagged. So rather than patch, I'd extract one focused skill: Copilot Studio Agent Linter - reads an exported agent and finds structural defects: trigger-phrase collisions, unreachable topics, missing fallback/escalation, variable-lifecycle issues. Deterministic, one job. I'll push the update to the same PR for review. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 20 out of 20 changed files in this pull request and generated 1 comment.
Suppressed comments (1)
submissions/agent-architecture-advisor/scripts/cost_model.py:204
foundry_cost()currently adds the “supporting uplift” ontofixed_monthly(fixed + supporting) even thoughsupportingis computed from(variable + fixed). That makes the reported fixed cost volume-dependent and undercounts Foundry per-conversation cost (sincecost_per_conversationexcludes the variable portion of the uplift). This distorts the break-even calculation and the cost comparison logic.
fixed = SEARCH_FIXED_MONTHLY + HOSTING_FIXED_MONTHLY
supporting = (variable + fixed) * SUPPORTING_UPLIFT
total = variable + fixed + supporting
return {
| if fmt == "solution_zip": | ||
| ingest_solution_zip(path, model, verbose) | ||
| elif fmt == "agent_yaml": | ||
| text = path.read_text(encoding="utf-8", errors="replace") | ||
|
|
||
| # Heuristic: Foundry agent definitions often have top-level 'model'/'model_deployment' and 'tools'. | ||
| # Prefer failing with an explicit warning over mis-parsing as a Copilot Studio topic. | ||
| looks_like_foundry = ( | ||
| re.search(r"^\s*(model|model_deployment)\s*:", text, re.M) | ||
| and re.search(r"^\s*tools\s*:", text, re.M) | ||
| ) | ||
|
|
||
| if looks_like_foundry: | ||
| ingest_foundry(text, model, path.name) | ||
| else: | ||
| model["source"]["format"] = "copilot_studio_agent_yaml" | ||
| model["agent"]["platform"] = "copilot_studio" | ||
| model["source"]["parsed_files"].append(path.name) | ||
| parse_topic_yaml(text, path.name, model, verbose) | ||
| text = path.read_text(encoding="utf-8", errors="replace") | ||
| ingest_foundry(text, model, path.name) | ||
| elif fmt == "customizations_xml": | ||
| text = path.read_text(encoding="utf-8", errors="replace") |
- Introduced a new requirements.txt file to specify PyYAML as an optional dependency. - This improves the reliability of parsing agent YAML exports, with a fallback to regex parsing if PyYAML is not available.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.
Suppressed comments (3)
submissions/copilot-studio-agent-health-check/SKILL.md:4
- The PR title/description says this change adds an “Agent Architecture Advisor” skill (and mentions Azure AI Foundry + GitHub Copilot support), but the submitted skill is “Copilot Studio Agent Health Check”, explicitly scoped as a deterministic linter that does not do platform advisory or modernization strategy. Please update the PR title/description (or rename/swap the submission) so the PR metadata matches what’s actually being added.
---
name: copilot-studio-agent-health-check
description: "Statically lints an exported Copilot Studio agent for structural defects, across both the classic topic-based experience and the new agentic (skill-based) experience. For topic agents it finds trigger-phrase collisions that cause misrouting, unreachable topics, missing fallback and escalation paths, slot-filling with no correction path, and variable-lifecycle bugs. For agentic agents it finds skill-description collisions that confuse routing, missing or thin skill descriptions, empty skill instructions, and references to unwired tools. It also runs cross-cutting checks on both experiences: unsafe or unrecognised Power Fx expressions, dead-end dialog branches, orchestration-mode-aware trigger/description quality, inconsistent OData parameter quoting, variable names that look like leaked secrets, and connected-agent handoff integrity. Use this whenever someone uploads a Copilot Studio agent export (solution .zip or the botcomponents folder) and wants it checked, or asks why their agent misroutes, gives empty answers, gets stuck, or might be leaking a secret — even when they only describe the symptom. Deterministic and evidence-backed: it reports defects in the artifact, it does not test security, model cost, or recommend a platform."
---
submissions/copilot-studio-agent-health-check/scripts/ingest_agent.py:328
parse_modernization_layer()documentsparent-agent/instructions.mdas the modernization-layer instructions source, but the current search order iterates over*instructions.mdanywhere in the export beforeparent-agent/*.md. That can pick an unrelated instructions file first and misattribute the agent’s main instructions.
# Markdown instructions (parent-agent/instructions.md, */instructions.md).
# Only adopt these as the agent's main instructions if config gave us none.
if not model["instructions"].get("present"):
for md in sorted(root.rglob("*instructions.md")) + sorted(root.rglob("parent-agent/*.md")):
try:
submissions/copilot-studio-agent-health-check/scripts/lint_agent.py:1420
lint()reads the inputschema_versionfrom the normalized model (and uses it to decide whether to emitschema_warning), but the returned result hard-codes"schema_version": "1.0". That makes the output inconsistent and hides the schema version that triggered the warning.
return {
"schema_version": "1.0",
"schema_warning": schema_warning,
"agent_name": model.get("agent", {}).get("name"),
Anamika Kumari (Annamika)
changed the title
Adds the Agent Architecture Advisor skill for reviewing, designing, and modernizing Copilot Studio and Azure AI Foundry agents.
The skill: