Skip to content

Fix flaky-test detector PR-evidence loss and raise scan limits - #13919

Merged
ViktorHofer merged 2 commits into
mainfrom
flaky-detector-tuning
Jun 2, 2026
Merged

Fix flaky-test detector PR-evidence loss and raise scan limits#13919
ViktorHofer merged 2 commits into
mainfrom
flaky-detector-tuning

Conversation

@ViktorHofer

@ViktorHofer ViktorHofer commented Jun 2, 2026

Copy link
Copy Markdown
Member

Problem

The daily flaky-test detector workflow (flaky-test-detector.agent.md) has been producing no-op runs since it merged. Investigating the first real runs surfaced two independent root causes.

1. The PR approval filter never had a GitHub token (lost ~2/3 of evidence)

The detector flags a test as flaky only when it fails across multiple independent sources — approved PRs and rolling main builds. The approved-PR half was always empty (prSources: 0).

Cause: the detector's gh pr view approval/draft/base filter runs as a bash subprocess inside the gh-aw agent container. The gh-aw agent entrypoint deliberately strips GH_TOKEN/COPILOT_GITHUB_TOKEN from the agent process (the documented "no secrets in agent runtime" guarantee), so the GH_TOKEN we injected via engine.env never reached gh. Every PR was dropped as "metadata-unavailable", leaving only rolling-main evidence.

Fix: set tools.github.mode: gh-proxy, the documented mechanism that mounts a pre-authenticated gh CLI inside the agent container (token provided via the trusted proxy, not as a readable env var). The detector's gh calls now authenticate with the workflow's existing read-only token. The ineffective GH_TOKEN injection is removed.

2. -MaxBuilds sat at/below real build volume -> permanent scanComplete: false

Definition 75 produces ~60 failed builds per 14-day window. With -MaxBuilds 60, the build-list query returned a full page, which the detector treats as possibly-truncated (scanComplete: false) because the AzDO API exposes no reliable total count — and a truncated scan is a hard no-op by design.

Fix: raise the PR-pipeline scan to -MaxBuilds 200 and the quarantine scan (def 344) to -MaxBuilds 150, both with -MaxArtifactDownloads 400 so the larger build set does not re-trip the artifact-download cap. The heuristic is documented inline so the values stay above real volume.

3. Remove the Copilot PAT-rotation stop-gap from all agentic workflows

The select-copilot-pat shared action and its frontmatter wiring (a pre-activation job, a PAT-pool checkout/select step, and an engine.env COPILOT_GITHUB_TOKEN case(...) expression) are removed from all four agentic workflows (flaky-test-detector, close-stale-prs, review, review-on-open). They now use the default COPILOT_GITHUB_TOKEN secret directly, and the .github/actions/select-copilot-pat/ folder is deleted.

This also resolves the gh-aw compile warning about engine.env referencing the built-in pre_activation job in a needs expression.

Other changes

  • Move the parallel-repro workspace from /tmp/repro to /tmp/gh-aw/agent/repro so its contents are captured as a run artifact (clears a gh-aw compile lint).
  • Recompile the .lock.yml files with the upgraded gh-aw compiler (this accounts for the large lock diff: new firewall/proxy container versions + cli-proxy plumbing for gh-proxy mode).

Validation

  • gh aw compile succeeds with no errors or warnings.
  • Lock confirms gh-proxy wiring: the cli-proxy container is pulled and start_cli_proxy.sh runs, so a pre-authenticated gh is available in the agent container.
  • No remaining references to select-copilot-pat, copilot_pat_number, or COPILOT_GITHUB_TOKEN_<n> in .github/.

The daily flaky-test detector was producing no-ops: it found 0 PR sources
and frequently reported scanComplete:false.

Two root causes:

1. PR approval filter saw no GitHub token. The detector's gh pr view
   approval/draft/base filter ran unauthenticated because the gh-aw agent
   entrypoint strips GH_TOKEN/COPILOT_GITHUB_TOKEN from the agent process
   ("no secrets in agent runtime"), so the injected GH_TOKEN: github.token
   never reached gh. Every PR was dropped as metadata-unavailable
   (prSources=0), leaving only rolling-main evidence (~1/3 of the signal).
   Fix: switch GitHub tools to mode: gh-proxy, which mounts a
   pre-authenticated gh CLI inside the agent container, and remove the
   ineffective GH_TOKEN env injection.

2. MaxBuilds was at/below real build volume. Definition 75 produces ~60
   failed builds per 14 days; with -MaxBuilds 60 the build-list query came
   back as a full page, which the detector treats as possibly-truncated
   (scanComplete:false), blocking all action. Raise the PR-pipeline scan to
   -MaxBuilds 200 and the quarantine scan to -MaxBuilds 150, with
   -MaxArtifactDownloads 400 so the larger set doesn't re-trip the artifact
   cap. Documented the heuristic inline.

Also move the repro workspace under /tmp/gh-aw/agent/ so its contents are
captured as a run artifact, and recompile the lock with the upgraded gh-aw
compiler.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings June 2, 2026 12:26
@ViktorHofer
ViktorHofer requested a review from a team as a code owner June 2, 2026 12:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes the flaky-test detector workflow producing no-op runs by restoring approved-PR evidence (via gh authentication inside the agent container) and by increasing scan limits so the AzDO build-list query is no longer consistently treated as truncated. It also updates the generated workflow lockfile produced by a newer gh-aw compiler.

Changes:

  • Switch the workflow to tools.github.mode: gh-proxy and remove the ineffective GH_TOKEN injection that never reached gh inside the agent runtime.
  • Raise Get-FlakyTests.ps1 scan caps (-MaxBuilds / -MaxArtifactDownloads) for both PR pipeline (def 75) and quarantine pipeline (def 344) scans, and document the truncation heuristic inline.
  • Move the parallel repro workspace under /tmp/gh-aw/agent/ so it’s captured in run artifacts; recompile the .agent.lock.yml with the upgraded compiler.
Show a summary per file
File Description
.github/workflows/flaky-test-detector.agent.md Enables gh-proxy mode, raises scan limits, and relocates the repro workspace to an artifact-captured directory.
.github/workflows/flaky-test-detector.agent.lock.yml Regenerated lock with new gh-aw tooling and adds gh-proxy plumbing (needs follow-up fixes noted in PR comments).
.github/aw/actions-lock.json Updates the pinned action entries to match the newer gh-aw compilation output.

Copilot's findings

  • Files reviewed: 3/3 changed files
  • Comments generated: 3

Comment thread .github/workflows/flaky-test-detector.agent.lock.yml
Comment thread .github/workflows/flaky-test-detector.agent.lock.yml
Comment thread .github/workflows/flaky-test-detector.agent.lock.yml
The select-copilot-pat action and its frontmatter wiring (pre-activation
job, PAT-pool checkout/select step, and engine.env COPILOT_GITHUB_TOKEN
case expression) are removed from all four agentic workflows. They now use
the default COPILOT_GITHUB_TOKEN secret directly.

This also resolves the compiler warning about engine.env referencing the
built-in pre_activation job in a needs expression.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ViktorHofer
ViktorHofer enabled auto-merge (squash) June 2, 2026 13:03
@ViktorHofer
ViktorHofer merged commit 8d2c7a6 into main Jun 2, 2026
10 checks passed
@ViktorHofer
ViktorHofer deleted the flaky-detector-tuning branch June 2, 2026 13:27
This was referenced Aug 11, 2026
This was referenced Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants