Fix flaky-test detector PR-evidence loss and raise scan limits - #13919
Merged
Conversation
The daily flaky-test detector was producing no-ops: it found 0 PR sources
and frequently reported scanComplete:false.
Two root causes:
1. PR approval filter saw no GitHub token. The detector's gh pr view
approval/draft/base filter ran unauthenticated because the gh-aw agent
entrypoint strips GH_TOKEN/COPILOT_GITHUB_TOKEN from the agent process
("no secrets in agent runtime"), so the injected GH_TOKEN: github.token
never reached gh. Every PR was dropped as metadata-unavailable
(prSources=0), leaving only rolling-main evidence (~1/3 of the signal).
Fix: switch GitHub tools to mode: gh-proxy, which mounts a
pre-authenticated gh CLI inside the agent container, and remove the
ineffective GH_TOKEN env injection.
2. MaxBuilds was at/below real build volume. Definition 75 produces ~60
failed builds per 14 days; with -MaxBuilds 60 the build-list query came
back as a full page, which the detector treats as possibly-truncated
(scanComplete:false), blocking all action. Raise the PR-pipeline scan to
-MaxBuilds 200 and the quarantine scan to -MaxBuilds 150, with
-MaxArtifactDownloads 400 so the larger set doesn't re-trip the artifact
cap. Documented the heuristic inline.
Also move the repro workspace under /tmp/gh-aw/agent/ so its contents are
captured as a run artifact, and recompile the lock with the upgraded gh-aw
compiler.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes the flaky-test detector workflow producing no-op runs by restoring approved-PR evidence (via gh authentication inside the agent container) and by increasing scan limits so the AzDO build-list query is no longer consistently treated as truncated. It also updates the generated workflow lockfile produced by a newer gh-aw compiler.
Changes:
- Switch the workflow to
tools.github.mode: gh-proxyand remove the ineffectiveGH_TOKENinjection that never reachedghinside the agent runtime. - Raise
Get-FlakyTests.ps1scan caps (-MaxBuilds/-MaxArtifactDownloads) for both PR pipeline (def 75) and quarantine pipeline (def 344) scans, and document the truncation heuristic inline. - Move the parallel repro workspace under
/tmp/gh-aw/agent/so it’s captured in run artifacts; recompile the.agent.lock.ymlwith the upgraded compiler.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/flaky-test-detector.agent.md |
Enables gh-proxy mode, raises scan limits, and relocates the repro workspace to an artifact-captured directory. |
.github/workflows/flaky-test-detector.agent.lock.yml |
Regenerated lock with new gh-aw tooling and adds gh-proxy plumbing (needs follow-up fixes noted in PR comments). |
.github/aw/actions-lock.json |
Updates the pinned action entries to match the newer gh-aw compilation output. |
Copilot's findings
- Files reviewed: 3/3 changed files
- Comments generated: 3
The select-copilot-pat action and its frontmatter wiring (pre-activation job, PAT-pool checkout/select step, and engine.env COPILOT_GITHUB_TOKEN case expression) are removed from all four agentic workflows. They now use the default COPILOT_GITHUB_TOKEN secret directly. This also resolves the compiler warning about engine.env referencing the built-in pre_activation job in a needs expression. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AlesProkop
approved these changes
Jun 2, 2026
ViktorHofer
enabled auto-merge (squash)
June 2, 2026 13:03
This was referenced Jun 3, 2026
This was referenced Aug 11, 2026
Bump Microsoft.Build from 18.4.0 to 18.9.6
SkylineCommunications/Skyline.DataMiner.CICD.Packages#172
Open
This was referenced Aug 18, 2026
Closed
Open
This was referenced Aug 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The daily flaky-test detector workflow (
flaky-test-detector.agent.md) has been producing no-op runs since it merged. Investigating the first real runs surfaced two independent root causes.1. The PR approval filter never had a GitHub token (lost ~2/3 of evidence)
The detector flags a test as flaky only when it fails across multiple independent sources — approved PRs and rolling
mainbuilds. The approved-PR half was always empty (prSources: 0).Cause: the detector's
gh pr viewapproval/draft/base filter runs as a bash subprocess inside the gh-aw agent container. The gh-aw agent entrypoint deliberately stripsGH_TOKEN/COPILOT_GITHUB_TOKENfrom the agent process (the documented "no secrets in agent runtime" guarantee), so theGH_TOKENwe injected viaengine.envnever reachedgh. Every PR was dropped as "metadata-unavailable", leaving only rolling-mainevidence.Fix: set
tools.github.mode: gh-proxy, the documented mechanism that mounts a pre-authenticatedghCLI inside the agent container (token provided via the trusted proxy, not as a readable env var). The detector'sghcalls now authenticate with the workflow's existing read-only token. The ineffectiveGH_TOKENinjection is removed.2.
-MaxBuildssat at/below real build volume -> permanentscanComplete: falseDefinition 75 produces ~60 failed builds per 14-day window. With
-MaxBuilds 60, the build-list query returned a full page, which the detector treats as possibly-truncated (scanComplete: false) because the AzDO API exposes no reliable total count — and a truncated scan is a hard no-op by design.Fix: raise the PR-pipeline scan to
-MaxBuilds 200and the quarantine scan (def 344) to-MaxBuilds 150, both with-MaxArtifactDownloads 400so the larger build set does not re-trip the artifact-download cap. The heuristic is documented inline so the values stay above real volume.3. Remove the Copilot PAT-rotation stop-gap from all agentic workflows
The
select-copilot-patshared action and its frontmatter wiring (apre-activationjob, a PAT-pool checkout/select step, and anengine.envCOPILOT_GITHUB_TOKENcase(...)expression) are removed from all four agentic workflows (flaky-test-detector,close-stale-prs,review,review-on-open). They now use the defaultCOPILOT_GITHUB_TOKENsecret directly, and the.github/actions/select-copilot-pat/folder is deleted.This also resolves the gh-aw compile warning about
engine.envreferencing the built-inpre_activationjob in aneedsexpression.Other changes
/tmp/reproto/tmp/gh-aw/agent/reproso its contents are captured as a run artifact (clears a gh-aw compile lint)..lock.ymlfiles with the upgraded gh-aw compiler (this accounts for the large lock diff: new firewall/proxy container versions + cli-proxy plumbing for gh-proxy mode).Validation
gh aw compilesucceeds with no errors or warnings.cli-proxycontainer is pulled andstart_cli_proxy.shruns, so a pre-authenticatedghis available in the agent container.select-copilot-pat,copilot_pat_number, orCOPILOT_GITHUB_TOKEN_<n>in.github/.