Skip to content

refactor(tests): distill safe-output smoke suite from 27 to 5 pipelines#1498

Merged
jamesadevine merged 4 commits into
mainfrom
copilot/distill-safe-output-pipelines
Jul 13, 2026
Merged

refactor(tests): distill safe-output smoke suite from 27 to 5 pipelines#1498
jamesadevine merged 4 commits into
mainfrom
copilot/distill-safe-output-pipelines

Conversation

Copilot AI commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Summary

27 per-tool daily agentic smoke pipelines provided redundant Stage 1→2→3 coverage: the deterministic executor-e2e suite already validates each tool's ADO REST logic independently, so 23 individual agentic loops added no incremental signal. This PR collapses them to 5 pipelines while fully compensating lost Stage 3 coverage in the deterministic suite.

Step 1 — Add 4 missing executor-e2e signal scenarios

  • New scripts/ado-script/src/executor-e2e/scenarios/signals.ts covers noop, missing-tool, missing-data, report-incomplete — the four NDJSON-only tools that had no Stage 3 deterministic coverage. Assert verifies status: "ok" (or expectedFailure for report-incomplete). Wired into index.ts as signalScenarios. Executor-e2e now covers 23/23 tools.

Step 2 — One omnibus canary pipeline replaces 23 per-tool smokes

  • tests/safe-outputs/canary.md asks the agent to emit three diverse tools in a single run: noop (proof-of-life), create-work-item (ADO write path), add-build-tag (build API path). One green run simultaneously validates Stage 1 (agent emits multiple correct calls), Stage 2 (detection clears heterogeneous NDJSON), and Stage 3 (two distinct ADO REST paths). Scheduled daily around 03:00.

Step 3 — Keep 4 infrastructure pipelines unchanged
azure-cli, noop-target, janitor, smoke-failure-reporter retained; smoke-failure-reporter prompt updated to query only canary + azure-cli instead of the old Daily safe-output smoke: * glob.

Step 4 — Delete 23 per-tool smoke pairs
All pipeline IDs were TBD (never registered in ADO), so no operational de-registration needed.

Step 5 — Docs and test fixture updates

  • tests/safe-outputs/README.md and REGISTERED.md rewritten for 5-pipeline layout with updated "Adding a new safe output" convention.
  • tests/executor-e2e/README.md coverage table updated.
  • Test fixtures in src/mcp_author/tests.rs, src/audit/pipeline_graph.rs, src/inspect/lint.rs, and tests/inspect_integration.rs migrated from the deleted create-pull-request.md to canary.md / tests/fixtures/complete-agent.md.
Dimension Before After
Agentic smoke pipelines 27 5
Stage 3 executor coverage 19/23 23/23
Daily end-to-end agentic runs 23 2 (canary + azure-cli)

Test plan

cargo test — all tests pass (0 failures). npm tests for ado-script pass (510/510) including the executor-e2e index test that validates scenario wiring.

Copilot AI and others added 4 commits July 13, 2026 11:35
…g-data/report-incomplete

Co-authored-by: jamesadevine <4742697+jamesadevine@users.noreply.github.com>
Co-authored-by: jamesadevine <4742697+jamesadevine@users.noreply.github.com>
…failure reporter

Co-authored-by: jamesadevine <4742697+jamesadevine@users.noreply.github.com>
…ry.md/complete-agent.md

Co-authored-by: jamesadevine <4742697+jamesadevine@users.noreply.github.com>
@jamesadevine
jamesadevine marked this pull request as ready for review July 13, 2026 20:10
@jamesadevine
jamesadevine merged commit ad65fcc into main Jul 13, 2026
22 checks passed
@jamesadevine
jamesadevine deleted the copilot/distill-safe-output-pipelines branch July 13, 2026 20:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants