From 403d758e2df481fb34f86b4d0b6ae6d29363cdb0 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 7 Jul 2026 02:30:15 +0000 Subject: [PATCH 1/3] Initial plan From 183d44b095fbad41272661125c8c246f0d793c21 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 7 Jul 2026 02:53:19 +0000 Subject: [PATCH 2/3] docs: add custom agent prerequisites, preflight pattern, and tool-unavailable fallback - github-agentic-workflows.md: add Custom Agent Tool Prerequisites section documenting prerequisites (Copilot CLI), failure modes table, and the tool-unavailable fallback pattern with explicit outcome recording - create-agentic-workflow.md: add Preflight Check for Custom Agent Runtimes subsection inside Ad Hoc Evaluation Mode with probe/success/failure steps and a cross-reference to the prerequisites doc - agent-persona-explorer.md: add preflight probe step before Phase 3 scenarios and a Tooling Availability field to the Phase 5 report template; recompile lock file Closes #43828 Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --- .github/aw/create-agentic-workflow.md | 11 ++++++++ .github/aw/github-agentic-workflows.md | 26 +++++++++++++++++++ .../workflows/agent-persona-explorer.lock.yml | 2 +- .github/workflows/agent-persona-explorer.md | 18 ++++++++++++- 4 files changed, 55 insertions(+), 2 deletions(-) diff --git a/.github/aw/create-agentic-workflow.md b/.github/aw/create-agentic-workflow.md index 15a72102978..9cc9089a26e 100644 --- a/.github/aw/create-agentic-workflow.md +++ b/.github/aw/create-agentic-workflow.md @@ -60,6 +60,17 @@ Use this mode for exploratory testing, persona walkthroughs, and "what workflow - Exit ad hoc evaluation mode only when the user explicitly asks to create, implement, or write the workflow file. - End by offering to turn the recommendation into `.github/workflows/.md` if the user wants to proceed. +### Preflight Check for Custom Agent Runtimes + +When the evaluation workflow depends on a custom agent tool (for example the `agentic-workflows` tool), run a preflight check before executing scenarios: + +1. **Probe the tool** — make a single minimal call (for example a one-sentence prompt) and confirm the response is non-empty and error-free. +2. **On success** — proceed with evaluation scenarios normally. +3. **On failure** — record `{ "status": "tool-unavailable", "reason": "" }`, switch to direct design reasoning using `.github/aw/*.md` reference files, and label all derived recommendations as **inferred**. +4. **Always publish a result** — emit the report regardless of tool availability; include a `Tooling Availability` note that lists which tools succeeded, which failed, and the exact errors observed. + +See [github-agentic-workflows.md](github-agentic-workflows.md#custom-agent-tool-prerequisites) for the full failure-mode table and fallback pattern. + ## Design Checklist ### 1. Pick the workflow ID diff --git a/.github/aw/github-agentic-workflows.md b/.github/aw/github-agentic-workflows.md index 11b22d257dd..4ff5dd503e5 100644 --- a/.github/aw/github-agentic-workflows.md +++ b/.github/aw/github-agentic-workflows.md @@ -83,6 +83,32 @@ Use the smallest trigger that matches the requested automation. See also: [workflow-constraints.md](workflow-constraints.md) +## Custom Agent Tool Prerequisites + +Workflows that invoke a custom agent tool (for example `tools.agentic-workflows:`) depend on the corresponding CLI runtime being present in the workflow sandbox. + +### Prerequisites + +- **Copilot CLI**: Required for the default `copilot` engine. If not installed, custom agent tool calls return `Copilot CLI not installed` and the tool is unavailable. +- Verify availability at runtime by attempting a lightweight probe call before committing to a full evaluation run. + +### Common Failure Modes + +| Failure | Cause | Recovery | +|---|---|---| +| `Copilot CLI not installed` | The Copilot CLI binary is absent from the sandbox | Record explicitly; fall back to direct design reasoning | +| Tool call timeout | Agent invocation exceeded the allowed time window | Record as `tool-unavailable`; continue with remaining scenarios | +| Empty or malformed response | CLI is present but the agent returned no usable content | Record the raw error; mark quality score as `N/A` | + +### Tool-Unavailable Fallback Pattern + +When a custom agent tool call fails or is unavailable: + +1. Record the outcome as `{ "status": "tool-unavailable", "reason": "" }` alongside the other scenario results. +2. Derive the expected design recommendation using direct reasoning from the `.github/aw/*.md` reference files. +3. Label the derived recommendation as **inferred** (not agent-produced) so aggregate quality scores remain comparable across runs. +4. Include a `## Tooling Availability` section in the published report summarising which tools succeeded, which failed, and the exact error messages received. + ## Ad Hoc Scenario Evaluation Installed gh-aw agents should support scenario evaluation requests that do not create workflow files. diff --git a/.github/workflows/agent-persona-explorer.lock.yml b/.github/workflows/agent-persona-explorer.lock.yml index 084f2a5d23a..26eeca814f0 100644 --- a/.github/workflows/agent-persona-explorer.lock.yml +++ b/.github/workflows/agent-persona-explorer.lock.yml @@ -1,4 +1,4 @@ -# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"a4d920f0c4fd428423b016bde55627cdf09b0569f166c7bdeb1c28796f1ff790","body_hash":"ba8f28145047b919e751f35cecb011044729c0b40a96145b782f6ab777995888","strict":true,"agent_id":"pi","agent_model":"copilot/gpt-5.4","detection_agent_id":"copilot","engine_versions":{"copilot":"1.0.68","pi":"0.80.3"}} +# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"a4d920f0c4fd428423b016bde55627cdf09b0569f166c7bdeb1c28796f1ff790","body_hash":"7c0c4c34c015bda8ec81c07b5a3168b18e57df73ab52fcbfcede7a80e2a647a9","strict":true,"agent_id":"pi","agent_model":"copilot/gpt-5.4","detection_agent_id":"copilot","engine_versions":{"copilot":"1.0.68","pi":"0.80.3"}} # gh-aw-manifest: {"version":1,"secrets":["COPILOT_GITHUB_TOKEN","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GH_AW_OTEL_GRAFANA_AUTHORIZATION","GH_AW_OTEL_GRAFANA_ENDPOINT","GH_AW_OTEL_SENTRY_AUTHORIZATION","GH_AW_OTEL_SENTRY_ENDPOINT","GITHUB_TOKEN"],"actions":[{"repo":"actions/cache/restore","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/cache/save","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/checkout","sha":"9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0","version":"v7.0.0"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/setup-go","sha":"924ae3a1cded613372ab5595356fb5720e22ba16","version":"v6.5.0"},{"repo":"actions/setup-node","sha":"48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e","version":"v6.4.0"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"docker/build-push-action","sha":"f9f3042f7e2789586610d6e8b85c8f03e5195baf","version":"v7.2.0"},{"repo":"docker/setup-buildx-action","sha":"d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5","version":"v4.1.0"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.27.24","digest":"sha256:b283b1f037d2e068532fe178a06f2944696c3933ba11604979134e7896ac6f8c","pinned_image":"ghcr.io/github/gh-aw-firewall/agent:0.27.24@sha256:b283b1f037d2e068532fe178a06f2944696c3933ba11604979134e7896ac6f8c"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.24","digest":"sha256:1743dbac9bf4225f3acfdcbce4f77f5a3e61e22b2e929305525f00196693c015","pinned_image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.24@sha256:1743dbac9bf4225f3acfdcbce4f77f5a3e61e22b2e929305525f00196693c015"},{"image":"ghcr.io/github/gh-aw-firewall/cli-proxy:0.27.24","digest":"sha256:6006b1c579ca550e023697b5dfd832ae03361328a4c0f2eb49fb181a6b8d7a4b","pinned_image":"ghcr.io/github/gh-aw-firewall/cli-proxy:0.27.24@sha256:6006b1c579ca550e023697b5dfd832ae03361328a4c0f2eb49fb181a6b8d7a4b"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.27.24","digest":"sha256:d4647c3bfaf80889eb1dbd3d4e2063340cbf94c0ca6c5747bbfc8507b12f3485","pinned_image":"ghcr.io/github/gh-aw-firewall/squid:0.27.24@sha256:d4647c3bfaf80889eb1dbd3d4e2063340cbf94c0ca6c5747bbfc8507b12f3485"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.3.33","digest":"sha256:f0e1f1d562f01e737f3ef125cb4d7057ff87a68a97927f5ea9fd6d3f5091da06","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.3.33@sha256:f0e1f1d562f01e737f3ef125cb4d7057ff87a68a97927f5ea9fd6d3f5091da06"},{"image":"ghcr.io/github/gh-aw-node","digest":"sha256:529d02eb970b1161aa25c593a9c3df57fdfad5a8add328cb3b6eccef66f3183b","pinned_image":"ghcr.io/github/gh-aw-node@sha256:529d02eb970b1161aa25c593a9c3df57fdfad5a8add328cb3b6eccef66f3183b"},{"image":"ghcr.io/github/github-mcp-server:v1.5.0","digest":"sha256:e25564dccc9110a70a77b9df560cbde11aa392fcb5f08b9abe5c4ebc6d146ea4","pinned_image":"ghcr.io/github/github-mcp-server:v1.5.0@sha256:e25564dccc9110a70a77b9df560cbde11aa392fcb5f08b9abe5c4ebc6d146ea4"}]} # This file was automatically generated by gh-aw. DO NOT EDIT. To debug this workflow, load the skill at https://github.com/github/gh-aw/blob/main/debug.md # diff --git a/.github/workflows/agent-persona-explorer.md b/.github/workflows/agent-persona-explorer.md index 51fe35afb5f..180e76e6cd1 100644 --- a/.github/workflows/agent-persona-explorer.md +++ b/.github/workflows/agent-persona-explorer.md @@ -133,6 +133,18 @@ Store all scenarios in cache memory. **Token Budget Optimization**: Test a **representative subset of 3-4 scenarios** from the 6 generated above (not all) to reduce token consumption and ensure budget remains for Phase 5 publishing. +### Preflight: Verify Agent Tool Availability + +Before running any scenario, probe the "agentic-workflows" custom agent with a single minimal prompt (for example: "What trigger should I use for a weekly digest?"). + +- **If the probe succeeds** (non-empty, error-free response): proceed with scenario testing below. +- **If the probe fails** (for example `Copilot CLI not installed`, timeout, or empty response): + 1. Record `{ "status": "tool-unavailable", "reason": "" }` in cache memory. + 2. For each selected scenario, derive the expected design recommendation using direct reasoning from `.github/aw/*.md` reference files and label it **inferred**. + 3. Skip the scenario-by-scenario agent invocations below and proceed directly to Phase 4 and Phase 5. + 4. Set the average quality score to `N/A` (tool-unavailable) rather than a numeric rating. + 5. Include a `### Tooling Availability` subsection in the published report that states which tools were probed, the outcome, and the exact error message. + {{#if experiments.sub_agent_strategy == 'batch' }} Invoke the "agentic-workflows" custom agent **once** with all 3-4 selected scenarios presented together in a structured list: @@ -238,7 +250,11 @@ Example: - **Agent**: [name] - **Personas This Run**: [3 persona names] - **Scenarios Tested**: [count - should be 3-4, selected from the 6 generated in Phase 2 (2 per persona × 3 personas)] -- **Average Quality Score**: [X.X/5.0] +- **Average Quality Score**: [X.X/5.0 or N/A (tool-unavailable)] + +### Tooling Availability +- **agentic-workflows tool**: [available | unavailable — reason: ""] +- *(list any other tools probed)* ### Key Findings (3-5 bullet points max) [High-level insights - keep concise] From ca4d30bbafb586997495926d3bb16e18e3a42991 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 7 Jul 2026 02:54:06 +0000 Subject: [PATCH 3/3] docs: address review feedback on preflight probe and fallback wording Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --- .github/aw/create-agentic-workflow.md | 2 +- .github/aw/github-agentic-workflows.md | 2 +- .github/workflows/agent-persona-explorer.lock.yml | 2 +- .github/workflows/agent-persona-explorer.md | 4 ++-- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/.github/aw/create-agentic-workflow.md b/.github/aw/create-agentic-workflow.md index 9cc9089a26e..956934ac5dc 100644 --- a/.github/aw/create-agentic-workflow.md +++ b/.github/aw/create-agentic-workflow.md @@ -64,7 +64,7 @@ Use this mode for exploratory testing, persona walkthroughs, and "what workflow When the evaluation workflow depends on a custom agent tool (for example the `agentic-workflows` tool), run a preflight check before executing scenarios: -1. **Probe the tool** — make a single minimal call (for example a one-sentence prompt) and confirm the response is non-empty and error-free. +1. **Probe the tool** — make a single minimal call (for example `"Are you available?"`) and confirm the response is non-empty, error-free, and contains a recognisable affirmative or workflow guidance signal. 2. **On success** — proceed with evaluation scenarios normally. 3. **On failure** — record `{ "status": "tool-unavailable", "reason": "" }`, switch to direct design reasoning using `.github/aw/*.md` reference files, and label all derived recommendations as **inferred**. 4. **Always publish a result** — emit the report regardless of tool availability; include a `Tooling Availability` note that lists which tools succeeded, which failed, and the exact errors observed. diff --git a/.github/aw/github-agentic-workflows.md b/.github/aw/github-agentic-workflows.md index 4ff5dd503e5..dad69f874af 100644 --- a/.github/aw/github-agentic-workflows.md +++ b/.github/aw/github-agentic-workflows.md @@ -105,7 +105,7 @@ Workflows that invoke a custom agent tool (for example `tools.agentic-workflows: When a custom agent tool call fails or is unavailable: 1. Record the outcome as `{ "status": "tool-unavailable", "reason": "" }` alongside the other scenario results. -2. Derive the expected design recommendation using direct reasoning from the `.github/aw/*.md` reference files. +2. Derive the design recommendation using direct reasoning from the `.github/aw/*.md` reference files. 3. Label the derived recommendation as **inferred** (not agent-produced) so aggregate quality scores remain comparable across runs. 4. Include a `## Tooling Availability` section in the published report summarising which tools succeeded, which failed, and the exact error messages received. diff --git a/.github/workflows/agent-persona-explorer.lock.yml b/.github/workflows/agent-persona-explorer.lock.yml index 26eeca814f0..653cbe18315 100644 --- a/.github/workflows/agent-persona-explorer.lock.yml +++ b/.github/workflows/agent-persona-explorer.lock.yml @@ -1,4 +1,4 @@ -# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"a4d920f0c4fd428423b016bde55627cdf09b0569f166c7bdeb1c28796f1ff790","body_hash":"7c0c4c34c015bda8ec81c07b5a3168b18e57df73ab52fcbfcede7a80e2a647a9","strict":true,"agent_id":"pi","agent_model":"copilot/gpt-5.4","detection_agent_id":"copilot","engine_versions":{"copilot":"1.0.68","pi":"0.80.3"}} +# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"a4d920f0c4fd428423b016bde55627cdf09b0569f166c7bdeb1c28796f1ff790","body_hash":"ad03242b179857adcb3e0aeb23ff396d00d1c56ac316d5f0e11a19f897ba7ad5","strict":true,"agent_id":"pi","agent_model":"copilot/gpt-5.4","detection_agent_id":"copilot","engine_versions":{"copilot":"1.0.68","pi":"0.80.3"}} # gh-aw-manifest: {"version":1,"secrets":["COPILOT_GITHUB_TOKEN","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GH_AW_OTEL_GRAFANA_AUTHORIZATION","GH_AW_OTEL_GRAFANA_ENDPOINT","GH_AW_OTEL_SENTRY_AUTHORIZATION","GH_AW_OTEL_SENTRY_ENDPOINT","GITHUB_TOKEN"],"actions":[{"repo":"actions/cache/restore","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/cache/save","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/checkout","sha":"9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0","version":"v7.0.0"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/setup-go","sha":"924ae3a1cded613372ab5595356fb5720e22ba16","version":"v6.5.0"},{"repo":"actions/setup-node","sha":"48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e","version":"v6.4.0"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"docker/build-push-action","sha":"f9f3042f7e2789586610d6e8b85c8f03e5195baf","version":"v7.2.0"},{"repo":"docker/setup-buildx-action","sha":"d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5","version":"v4.1.0"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.27.24","digest":"sha256:b283b1f037d2e068532fe178a06f2944696c3933ba11604979134e7896ac6f8c","pinned_image":"ghcr.io/github/gh-aw-firewall/agent:0.27.24@sha256:b283b1f037d2e068532fe178a06f2944696c3933ba11604979134e7896ac6f8c"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.24","digest":"sha256:1743dbac9bf4225f3acfdcbce4f77f5a3e61e22b2e929305525f00196693c015","pinned_image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.24@sha256:1743dbac9bf4225f3acfdcbce4f77f5a3e61e22b2e929305525f00196693c015"},{"image":"ghcr.io/github/gh-aw-firewall/cli-proxy:0.27.24","digest":"sha256:6006b1c579ca550e023697b5dfd832ae03361328a4c0f2eb49fb181a6b8d7a4b","pinned_image":"ghcr.io/github/gh-aw-firewall/cli-proxy:0.27.24@sha256:6006b1c579ca550e023697b5dfd832ae03361328a4c0f2eb49fb181a6b8d7a4b"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.27.24","digest":"sha256:d4647c3bfaf80889eb1dbd3d4e2063340cbf94c0ca6c5747bbfc8507b12f3485","pinned_image":"ghcr.io/github/gh-aw-firewall/squid:0.27.24@sha256:d4647c3bfaf80889eb1dbd3d4e2063340cbf94c0ca6c5747bbfc8507b12f3485"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.3.33","digest":"sha256:f0e1f1d562f01e737f3ef125cb4d7057ff87a68a97927f5ea9fd6d3f5091da06","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.3.33@sha256:f0e1f1d562f01e737f3ef125cb4d7057ff87a68a97927f5ea9fd6d3f5091da06"},{"image":"ghcr.io/github/gh-aw-node","digest":"sha256:529d02eb970b1161aa25c593a9c3df57fdfad5a8add328cb3b6eccef66f3183b","pinned_image":"ghcr.io/github/gh-aw-node@sha256:529d02eb970b1161aa25c593a9c3df57fdfad5a8add328cb3b6eccef66f3183b"},{"image":"ghcr.io/github/github-mcp-server:v1.5.0","digest":"sha256:e25564dccc9110a70a77b9df560cbde11aa392fcb5f08b9abe5c4ebc6d146ea4","pinned_image":"ghcr.io/github/github-mcp-server:v1.5.0@sha256:e25564dccc9110a70a77b9df560cbde11aa392fcb5f08b9abe5c4ebc6d146ea4"}]} # This file was automatically generated by gh-aw. DO NOT EDIT. To debug this workflow, load the skill at https://github.com/github/gh-aw/blob/main/debug.md # diff --git a/.github/workflows/agent-persona-explorer.md b/.github/workflows/agent-persona-explorer.md index 180e76e6cd1..26c872e9b7d 100644 --- a/.github/workflows/agent-persona-explorer.md +++ b/.github/workflows/agent-persona-explorer.md @@ -135,12 +135,12 @@ Store all scenarios in cache memory. ### Preflight: Verify Agent Tool Availability -Before running any scenario, probe the "agentic-workflows" custom agent with a single minimal prompt (for example: "What trigger should I use for a weekly digest?"). +Before running any scenario, probe the "agentic-workflows" custom agent with a minimal availability prompt such as "Are you available? Reply with yes or no.". - **If the probe succeeds** (non-empty, error-free response): proceed with scenario testing below. - **If the probe fails** (for example `Copilot CLI not installed`, timeout, or empty response): 1. Record `{ "status": "tool-unavailable", "reason": "" }` in cache memory. - 2. For each selected scenario, derive the expected design recommendation using direct reasoning from `.github/aw/*.md` reference files and label it **inferred**. + 2. Derive the design recommendation using direct reasoning from the `.github/aw/*.md` reference files and label it **inferred**. 3. Skip the scenario-by-scenario agent invocations below and proceed directly to Phase 4 and Phase 5. 4. Set the average quality score to `N/A` (tool-unavailable) rather than a numeric rating. 5. Include a `### Tooling Availability` subsection in the published report that states which tools were probed, the outcome, and the exact error message.