Skip to content

[Flaky Test] Quarantine 4 flaky tests - #13937

Merged
ViktorHofer merged 1 commit into
mainfrom
flaky-test/quarantine-4-tests-2026-06-03-d24a0b17a84046ab
Jun 5, 2026
Merged

[Flaky Test] Quarantine 4 flaky tests#13937
ViktorHofer merged 1 commit into
mainfrom
flaky-test/quarantine-4-tests-2026-06-03-d24a0b17a84046ab

Conversation

@github-actions

@github-actions github-actions Bot commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

Quarantines 4 tests that are failing intermittently in CI. None could be reproduced on this Linux/.NET runner, so all are quarantined with [ActiveIssue] pending a determinism fix.

Microsoft.Build.UnitTests.ToolTask_Tests.ToolTaskCanChangeCanonicalErrorFormat

Action: Quarantine (7d) — [ActiveIssue("https://github.com/dotnet/msbuild/issues/13931", TestPlatforms.Windows)]

Tracked by #13931

  • 6 distinct sources (1 approved PR + 5 rolling main builds)
  • Legs: CoreOnWindows, FullOnWindows
  • First seen: 2026-05-22 / Last seen: 2026-05-26
  • Sample error: Assert.Contains() Failure — findstr.exe stdout mismatch
  • Could not reproduce on Linux runner; Windows-scoped quarantine applied.

Microsoft.Build.UnitTests.ToolTask_Tests.OverrideStdOutImportanceToHigh

Action: Quarantine (7d) — [ActiveIssue("https://github.com/dotnet/msbuild/issues/13934", TestPlatforms.Windows)]

Tracked by #13934

  • 3 distinct sources (0 PRs + 3 rolling main builds: 1430301, 1431561, 1440318)
  • Legs: CoreOnWindows, FullOnWindows
  • First seen: 2026-05-21 / Last seen: 2026-05-29
  • Sample error: Assert.Contains() Failure — "hello world" not found in tool stdout
  • Could not reproduce on Linux runner; Windows-scoped quarantine applied.

Microsoft.Build.Engine.UnitTests.MSBuildTaskHostTests.CompileNet35WinFormsApp

Action: Quarantine (7d) — [ActiveIssue("https://github.com/dotnet/msbuild/issues/13933", TestPlatforms.Windows)]

Tracked by #13933

  • 3 distinct sources (1 approved PR + 2 rolling main builds: 1437213, 1445319)
  • 3 distinct error hashes — classic flake signature
  • Legs: FullOnWindows Release
  • First seen: 2026-05-25 / Last seen: 2026-06-02
  • Windows/.NET 3.5 only test ([WindowsNet35OnlyFact]); cannot reproduce on Linux.

Microsoft.Build.UnitTests.BinaryLoggerTests.BinaryLoggerShouldEmbedFilesViaTaskOutput

Action: Quarantine (7d) — [ActiveIssue("https://github.com/dotnet/msbuild/issues/13932", TestPlatforms.Linux)]

Tracked by #13932

  • 3 distinct sources (0 PRs + 3 rolling main builds: 1430551, 1431677, 1445471)
  • Legs: CoreOnLinux
  • First seen: 2026-05-21 / Last seen: 2026-06-02 (12-day spread)
  • Sample error: zipArchive.Entries does not contain testtaskoutputfile.txt — likely a timing/ordering race in task-output file embedding
  • Ran 25 sequential iterations locally — all passed (could not reproduce). Linux-scoped quarantine applied.

Generated by Flaky Test Triage and Fix · sonnet46 2.8M · ◷

@github-actions
Quarantine 4 flaky tests
8546057 
- ToolTask_Tests.OverrideStdOutImportanceToHigh (Windows, tracked by #13934)
- ToolTask_Tests.ToolTaskCanChangeCanonicalErrorFormat (Windows, tracked by #13931)
- MSBuildTaskHostTests.CompileNet35WinFormsApp (Windows, tracked by #13933)
- BinaryLoggerTests.BinaryLoggerShouldEmbedFilesViaTaskOutput (Linux, tracked by #13932)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot added the flaky-test Tests that have failed but passed on rerun. label Jun 3, 2026
@ViktorHofer ViktorHofer mentioned this pull request Jun 3, 2026
Merged
This was referenced Jun 3, 2026
Closed
Closed
Closed
Open
@ViktorHofer
ViktorHofer marked this pull request as ready for review June 5, 2026 12:11
Copilot AI review requested due to automatic review settings June 5, 2026 12:11
@ViktorHofer
ViktorHofer enabled auto-merge (squash) June 5, 2026 12:12
Copilot AI reviewed Jun 5, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Quarantines four intermittently failing unit tests by adding [ActiveIssue] attributes scoped to the affected OS, keeping the rest of the test suite running while the linked determinism issues are investigated.

Changes:

  • Quarantined two ToolTask_Tests failures on Windows via [ActiveIssue(..., TestPlatforms.Windows)].
  • Quarantined a .NET 3.5 WinForms task-host test on Windows via [ActiveIssue(..., TestPlatforms.Windows)].
  • Quarantined a BinaryLogger embedding test on Linux via [ActiveIssue(..., TestPlatforms.Linux)].
Show a summary per file
File Description
src/Utilities.UnitTests/ToolTask_Tests.cs Adds Windows-scoped [ActiveIssue] quarantines for two flaky ToolTask-related tests.
src/Build.UnitTests/MSBuildTaskHostTests.cs Adds a Windows-scoped [ActiveIssue] quarantine for the flaky .NET 3.5 WinForms compilation test.
src/Build.UnitTests/BinaryLogger_Tests.cs Adds a Linux-scoped [ActiveIssue] quarantine for intermittent file-embedding verification in binlogs.

Copilot's findings

  • Files reviewed: 3/3 changed files
  • Comments generated: 0

This was referenced Jun 5, 2026
Closed
Closed
Closed
Closed
Closed
@ViktorHofer
ViktorHofer merged commit a21bdfd into main Jun 5, 2026
11 checks passed
@ViktorHofer
ViktorHofer deleted the flaky-test/quarantine-4-tests-2026-06-03-d24a0b17a84046ab branch June 5, 2026 13:48
@ly6093909-ops

ly6093909-ops commented Jun 5, 2026

Copy link
Copy Markdown

Quarantines 4 tests that are failing intermittently in CI. None could be reproduced on this Linux/.NET runner, so all are quarantined with [ActiveIssue] pending a determinism fix.

Microsoft.Build.UnitTests.ToolTask_Tests.ToolTaskCanChangeCanonicalErrorFormat

Action: Quarantine (7d) — [ActiveIssue("https://github.com/dotnet/msbuild/issues/13931", TestPlatforms.Windows)]

Tracked by #13931

  • 6 distinct sources (1 approved PR + 5 rolling main builds)
  • Legs: CoreOnWindows, FullOnWindows
  • First seen: 2026-05-22 / Last seen: 2026-05-26
  • Sample error: Assert.Contains() Failure — findstr.exe stdout mismatch
  • Could not reproduce on Linux runner; Windows-scoped quarantine applied.

Microsoft.Build.UnitTests.ToolTask_Tests.OverrideStdOutImportanceToHigh

Action: Quarantine (7d) — [ActiveIssue("https://github.com/dotnet/msbuild/issues/13934", TestPlatforms.Windows)]

Tracked by #13934

  • 3 distinct sources (0 PRs + 3 rolling main builds: 1430301, 1431561, 1440318)
  • Legs: CoreOnWindows, FullOnWindows
  • First seen: 2026-05-21 / Last seen: 2026-05-29
  • Sample error: Assert.Contains() Failure — "hello world" not found in tool stdout
  • Could not reproduce on Linux runner; Windows-scoped quarantine applied.

Microsoft.Build.Engine.UnitTests.MSBuildTaskHostTests.CompileNet35WinFormsApp

Action: Quarantine (7d) — [ActiveIssue("https://github.com/dotnet/msbuild/issues/13933", TestPlatforms.Windows)]

Tracked by #13933

  • 3 distinct sources (1 approved PR + 2 rolling main builds: 1437213, 1445319)
  • 3 distinct error hashes — classic flake signature
  • Legs: FullOnWindows Release
  • First seen: 2026-05-25 / Last seen: 2026-06-02
  • Windows/.NET 3.5 only test ([WindowsNet35OnlyFact]); cannot reproduce on Linux.

Microsoft.Build.UnitTests.BinaryLoggerTests.BinaryLoggerShouldEmbedFilesViaTaskOutput

Action: Quarantine (7d) — [ActiveIssue("https://github.com/dotnet/msbuild/issues/13932", TestPlatforms.Linux)]

Tracked by #13932

  • 3 distinct sources (0 PRs + 3 rolling main builds: 1430551, 1431677, 1445471)
  • Legs: CoreOnLinux
  • First seen: 2026-05-21 / Last seen: 2026-06-02 (12-day spread)
  • Sample error: zipArchive.Entries does not contain testtaskoutputfile.txt — likely a timing/ordering race in task-output file embedding
  • Ran 25 sequential iterations locally — all passed (could not reproduce). Linux-scoped quarantine applied.

Generated by Flaky Test Triage and Fix · sonnet46 2.8M · ◷

@ly6093909-ops

ly6093909-ops commented Jun 5, 2026

Copy link
Copy Markdown

Quarantines 4 tests that are failing intermittently in CI. None could be reproduced on this Linux/.NET runner, so all are quarantined with [ActiveIssue] pending a determinism fix.

Microsoft.Build.UnitTests.ToolTask_Tests.ToolTaskCanChangeCanonicalErrorFormat

Action: Quarantine (7d) — [ActiveIssue("https://github.com/dotnet/msbuild/issues/13931", TestPlatforms.Windows)]

Tracked by #13931

  • 6 distinct sources (1 approved PR + 5 rolling main builds)
  • Legs: CoreOnWindows, FullOnWindows
  • First seen: 2026-05-22 / Last seen: 2026-05-26
  • Sample error: Assert.Contains() Failure — findstr.exe stdout mismatch
  • Could not reproduce on Linux runner; Windows-scoped quarantine applied.

Microsoft.Build.UnitTests.ToolTask_Tests.OverrideStdOutImportanceToHigh

Action: Quarantine (7d) — [ActiveIssue("https://github.com/dotnet/msbuild/issues/13934", TestPlatforms.Windows)]

Tracked by #13934

  • 3 distinct sources (0 PRs + 3 rolling main builds: 1430301, 1431561, 1440318)
  • Legs: CoreOnWindows, FullOnWindows
  • First seen: 2026-05-21 / Last seen: 2026-05-29
  • Sample error: Assert.Contains() Failure — "hello world" not found in tool stdout
  • Could not reproduce on Linux runner; Windows-scoped quarantine applied.

Microsoft.Build.Engine.UnitTests.MSBuildTaskHostTests.CompileNet35WinFormsApp

Action: Quarantine (7d) — [ActiveIssue("https://github.com/dotnet/msbuild/issues/13933", TestPlatforms.Windows)]

Tracked by #13933

  • 3 distinct sources (1 approved PR + 2 rolling main builds: 1437213, 1445319)
  • 3 distinct error hashes — classic flake signature
  • Legs: FullOnWindows Release
  • First seen: 2026-05-25 / Last seen: 2026-06-02
  • Windows/.NET 3.5 only test ([WindowsNet35OnlyFact]); cannot reproduce on Linux.

Microsoft.Build.UnitTests.BinaryLoggerTests.BinaryLoggerShouldEmbedFilesViaTaskOutput

Action: Quarantine (7d) — [ActiveIssue("https://github.com/dotnet/msbuild/issues/13932", TestPlatforms.Linux)]

Tracked by #13932

  • 3 distinct sources (0 PRs + 3 rolling main builds: 1430551, 1431677, 1445471)
  • Legs: CoreOnLinux
  • First seen: 2026-05-21 / Last seen: 2026-06-02 (12-day spread)
  • Sample error: zipArchive.Entries does not contain testtaskoutputfile.txt — likely a timing/ordering race in task-output file embedding
  • Ran 25 sequential iterations locally — all passed (could not reproduce). Linux-scoped quarantine applied.

Generated by Flaky Test Triage and Fix · sonnet46 2.8M · ◷

@ly6093909-ops

ly6093909-ops commented Jun 5, 2026

Copy link
Copy Markdown

Quarantines 4 tests that are failing intermittently in CI. None could be reproduced on this Linux/.NET runner, so all are quarantined with [ActiveIssue] pending a determinism fix.

Microsoft.Build.UnitTests.ToolTask_Tests.ToolTaskCanChangeCanonicalErrorFormat

Action: Quarantine (7d) — [ActiveIssue("https://github.com/dotnet/msbuild/issues/13931", TestPlatforms.Windows)]

Tracked by #13931

  • 6 distinct sources (1 approved PR + 5 rolling main builds)
  • Legs: CoreOnWindows, FullOnWindows
  • First seen: 2026-05-22 / Last seen: 2026-05-26
  • Sample error: Assert.Contains() Failure — findstr.exe stdout mismatch
  • Could not reproduce on Linux runner; Windows-scoped quarantine applied.

Microsoft.Build.UnitTests.ToolTask_Tests.OverrideStdOutImportanceToHigh

Action: Quarantine (7d) — [ActiveIssue("https://github.com/dotnet/msbuild/issues/13934", TestPlatforms.Windows)]

Tracked by #13934

  • 3 distinct sources (0 PRs + 3 rolling main builds: 1430301, 1431561, 1440318)
  • Legs: CoreOnWindows, FullOnWindows
  • First seen: 2026-05-21 / Last seen: 2026-05-29
  • Sample error: Assert.Contains() Failure — "hello world" not found in tool stdout
  • Could not reproduce on Linux runner; Windows-scoped quarantine applied.

Microsoft.Build.Engine.UnitTests.MSBuildTaskHostTests.CompileNet35WinFormsApp

Action: Quarantine (7d) — [ActiveIssue("https://github.com/dotnet/msbuild/issues/13933", TestPlatforms.Windows)]

Tracked by #13933

  • 3 distinct sources (1 approved PR + 2 rolling main builds: 1437213, 1445319)
  • 3 distinct error hashes — classic flake signature
  • Legs: FullOnWindows Release
  • First seen: 2026-05-25 / Last seen: 2026-06-02
  • Windows/.NET 3.5 only test ([WindowsNet35OnlyFact]); cannot reproduce on Linux.

Microsoft.Build.UnitTests.BinaryLoggerTests.BinaryLoggerShouldEmbedFilesViaTaskOutput

Action: Quarantine (7d) — [ActiveIssue("https://github.com/dotnet/msbuild/issues/13932", TestPlatforms.Linux)]

Tracked by #13932

  • 3 distinct sources (0 PRs + 3 rolling main builds: 1430551, 1431677, 1445471)
  • Legs: CoreOnLinux
  • First seen: 2026-05-21 / Last seen: 2026-06-02 (12-day spread)
  • Sample error: zipArchive.Entries does not contain testtaskoutputfile.txt — likely a timing/ordering race in task-output file embedding
  • Ran 25 sequential iterations locally — all passed (could not reproduce). Linux-scoped quarantine applied.

Generated by Flaky Test Triage and Fix · sonnet46 2.8M · ◷

-1739730.pdf

This was referenced Jun 6, 2026
Merged
Merged
This was referenced Aug 11, 2026
Merged
Closed
Merged
Open
Closed
This was referenced Aug 19, 2026
Merged
Merged
Open
Merged
Closed
Merged
Closed
Closed
Closed
Closed
Closed
Closed
Open
Open
Open
Open
Open
Open
Open
Closed
Closed
Closed
Open
Open
Closed
This was referenced Aug 26, 2026
Merged
Open
Merged
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@ViktorHofer ViktorHofer ViktorHofer approved these changes

Assignees

No one assigned

Labels

flaky-test Tests that have failed but passed on rerun.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ly6093909-ops @ViktorHofer