refactor(tests): distill safe-output smoke suite from 27 to 5 pipelines#1498
Merged
Conversation
…g-data/report-incomplete Co-authored-by: jamesadevine <4742697+jamesadevine@users.noreply.github.com>
Co-authored-by: jamesadevine <4742697+jamesadevine@users.noreply.github.com>
…failure reporter Co-authored-by: jamesadevine <4742697+jamesadevine@users.noreply.github.com>
…ry.md/complete-agent.md Co-authored-by: jamesadevine <4742697+jamesadevine@users.noreply.github.com>
Copilot created this pull request from a session on behalf of
jamesadevine
July 13, 2026 20:08
View session
jamesadevine
marked this pull request as ready for review
July 13, 2026 20:10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
27 per-tool daily agentic smoke pipelines provided redundant Stage 1→2→3 coverage: the deterministic
executor-e2esuite already validates each tool's ADO REST logic independently, so 23 individual agentic loops added no incremental signal. This PR collapses them to 5 pipelines while fully compensating lost Stage 3 coverage in the deterministic suite.Step 1 — Add 4 missing executor-e2e signal scenarios
scripts/ado-script/src/executor-e2e/scenarios/signals.tscoversnoop,missing-tool,missing-data,report-incomplete— the four NDJSON-only tools that had no Stage 3 deterministic coverage. Assert verifiesstatus: "ok"(orexpectedFailureforreport-incomplete). Wired intoindex.tsassignalScenarios. Executor-e2e now covers 23/23 tools.Step 2 — One omnibus canary pipeline replaces 23 per-tool smokes
tests/safe-outputs/canary.mdasks the agent to emit three diverse tools in a single run:noop(proof-of-life),create-work-item(ADO write path),add-build-tag(build API path). One green run simultaneously validates Stage 1 (agent emits multiple correct calls), Stage 2 (detection clears heterogeneous NDJSON), and Stage 3 (two distinct ADO REST paths). Scheduleddaily around 03:00.Step 3 — Keep 4 infrastructure pipelines unchanged
azure-cli,noop-target,janitor,smoke-failure-reporterretained;smoke-failure-reporterprompt updated to query onlycanary+azure-cliinstead of the oldDaily safe-output smoke: *glob.Step 4 — Delete 23 per-tool smoke pairs
All pipeline IDs were TBD (never registered in ADO), so no operational de-registration needed.
Step 5 — Docs and test fixture updates
tests/safe-outputs/README.mdandREGISTERED.mdrewritten for 5-pipeline layout with updated "Adding a new safe output" convention.tests/executor-e2e/README.mdcoverage table updated.src/mcp_author/tests.rs,src/audit/pipeline_graph.rs,src/inspect/lint.rs, andtests/inspect_integration.rsmigrated from the deletedcreate-pull-request.mdtocanary.md/tests/fixtures/complete-agent.md.Test plan
cargo test— all tests pass (0 failures). npm tests for ado-script pass (510/510) including the executor-e2e index test that validates scenario wiring.