Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .github/aw/create-agentic-workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,17 @@ Use this mode for exploratory testing, persona walkthroughs, and "what workflow
- Exit ad hoc evaluation mode only when the user explicitly asks to create, implement, or write the workflow file.
- End by offering to turn the recommendation into `.github/workflows/<workflow-id>.md` if the user wants to proceed.

### Preflight Check for Custom Agent Runtimes

When the evaluation workflow depends on a custom agent tool (for example the `agentic-workflows` tool), run a preflight check before executing scenarios:

1. **Probe the tool** — make a single minimal call (for example `"Are you available?"`) and confirm the response is non-empty, error-free, and contains a recognisable affirmative or workflow guidance signal.
2. **On success** — proceed with evaluation scenarios normally.
3. **On failure** — record `{ "status": "tool-unavailable", "reason": "<error>" }`, switch to direct design reasoning using `.github/aw/*.md` reference files, and label all derived recommendations as **inferred**.
4. **Always publish a result** — emit the report regardless of tool availability; include a `Tooling Availability` note that lists which tools succeeded, which failed, and the exact errors observed.

See [github-agentic-workflows.md](github-agentic-workflows.md#custom-agent-tool-prerequisites) for the full failure-mode table and fallback pattern.

## Design Checklist

### 1. Pick the workflow ID
Expand Down
26 changes: 26 additions & 0 deletions .github/aw/github-agentic-workflows.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion .github/workflows/agent-persona-explorer.lock.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

18 changes: 17 additions & 1 deletion .github/workflows/agent-persona-explorer.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,18 @@ Store all scenarios in cache memory.

**Token Budget Optimization**: Test a **representative subset of 3-4 scenarios** from the 6 generated above (not all) to reduce token consumption and ensure budget remains for Phase 5 publishing.

### Preflight: Verify Agent Tool Availability

Before running any scenario, probe the "agentic-workflows" custom agent with a minimal availability prompt such as "Are you available? Reply with yes or no.".

- **If the probe succeeds** (non-empty, error-free response): proceed with scenario testing below.
- **If the probe fails** (for example `Copilot CLI not installed`, timeout, or empty response):
1. Record `{ "status": "tool-unavailable", "reason": "<exact error>" }` in cache memory.
2. Derive the design recommendation using direct reasoning from the `.github/aw/*.md` reference files and label it **inferred**.
3. Skip the scenario-by-scenario agent invocations below and proceed directly to Phase 4 and Phase 5.
4. Set the average quality score to `N/A` (tool-unavailable) rather than a numeric rating.
5. Include a `### Tooling Availability` subsection in the published report that states which tools were probed, the outcome, and the exact error message.

{{#if experiments.sub_agent_strategy == 'batch' }}
Invoke the "agentic-workflows" custom agent **once** with all 3-4 selected scenarios presented together in a structured list:

Expand Down Expand Up @@ -238,7 +250,11 @@ Example:
- **Agent**: [name]
- **Personas This Run**: [3 persona names]
- **Scenarios Tested**: [count - should be 3-4, selected from the 6 generated in Phase 2 (2 per persona × 3 personas)]
- **Average Quality Score**: [X.X/5.0]
- **Average Quality Score**: [X.X/5.0 or N/A (tool-unavailable)]

### Tooling Availability
- **agentic-workflows tool**: [available | unavailable — reason: "<exact error>"]
- *(list any other tools probed)*

### Key Findings (3-5 bullet points max)
[High-level insights - keep concise]
Expand Down