Skip to content

Commit ca4d30b

Browse files
Copilotpelikhan
andauthored
docs: address review feedback on preflight probe and fallback wording
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
1 parent 183d44b commit ca4d30b

4 files changed

Lines changed: 5 additions & 5 deletions

File tree

.github/aw/create-agentic-workflow.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ Use this mode for exploratory testing, persona walkthroughs, and "what workflow
6464

6565
When the evaluation workflow depends on a custom agent tool (for example the `agentic-workflows` tool), run a preflight check before executing scenarios:
6666

67-
1. **Probe the tool** — make a single minimal call (for example a one-sentence prompt) and confirm the response is non-empty and error-free.
67+
1. **Probe the tool** — make a single minimal call (for example `"Are you available?"`) and confirm the response is non-empty, error-free, and contains a recognisable affirmative or workflow guidance signal.
6868
2. **On success** — proceed with evaluation scenarios normally.
6969
3. **On failure** — record `{ "status": "tool-unavailable", "reason": "<error>" }`, switch to direct design reasoning using `.github/aw/*.md` reference files, and label all derived recommendations as **inferred**.
7070
4. **Always publish a result** — emit the report regardless of tool availability; include a `Tooling Availability` note that lists which tools succeeded, which failed, and the exact errors observed.

.github/aw/github-agentic-workflows.md

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

.github/workflows/agent-persona-explorer.lock.yml

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

.github/workflows/agent-persona-explorer.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -135,12 +135,12 @@ Store all scenarios in cache memory.
135135

136136
### Preflight: Verify Agent Tool Availability
137137

138-
Before running any scenario, probe the "agentic-workflows" custom agent with a single minimal prompt (for example: "What trigger should I use for a weekly digest?").
138+
Before running any scenario, probe the "agentic-workflows" custom agent with a minimal availability prompt such as "Are you available? Reply with yes or no.".
139139

140140
- **If the probe succeeds** (non-empty, error-free response): proceed with scenario testing below.
141141
- **If the probe fails** (for example `Copilot CLI not installed`, timeout, or empty response):
142142
1. Record `{ "status": "tool-unavailable", "reason": "<exact error>" }` in cache memory.
143-
2. For each selected scenario, derive the expected design recommendation using direct reasoning from `.github/aw/*.md` reference files and label it **inferred**.
143+
2. Derive the design recommendation using direct reasoning from the `.github/aw/*.md` reference files and label it **inferred**.
144144
3. Skip the scenario-by-scenario agent invocations below and proceed directly to Phase 4 and Phase 5.
145145
4. Set the average quality score to `N/A` (tool-unavailable) rather than a numeric rating.
146146
5. Include a `### Tooling Availability` subsection in the published report that states which tools were probed, the outcome, and the exact error message.

0 commit comments

Comments
 (0)