[Apple mobile] Stage the Helix R2R app payload outside the crossgen2 input directory by davidnguyen-tech · Pull Request #132489 · dotnet/runtime

davidnguyen-tech · 2026-08-18T20:38:36Z

Fixes #131922

Problem

The build agent sends Apple mobile test files to Helix in publish/. On the Helix Mac, ProxyProjectForAOTOnHelix.proj acts as an MSBuild stand-in for the original test project: it ReadyToRun-compiles the shipped assemblies and builds the Apple app. It then copied Crossgen2 output back into publish/, which is also the compiler input directory.

Helix retries reuse the work-item directory. A retry therefore fed attempt 1's already-compiled, IL-stripped assemblies back into Crossgen2. The resulting app crashed before test discovery:

Fatal error.
A method body required at runtime was stripped from the ReadyToRun image.

This is retry input corruption in the build orchestration, not a Crossgen2 compiler bug.

Fix

Keep publish/ as the stable Crossgen2 input for every attempt.

For CoreCLR Mach-O R2R, the proxy now:

recreates obj/R2R/ before compiling;
creates obj/r2r-app-bundle-source/ from publish/ plus the current R2R output;
points AppleBuildDir and the bundle item lists at that directory.

Crossgen2 and AppleAppBuilder output can no longer become input to a later retry, and partial output from a killed attempt is removed.

Regression coverage

The new Apple.Build.Tests host test runs the real proxy targets twice in the same temporary work-item directory. It verifies that:

publish/ keeps the same files and contents;
final bundle paths are declared before the bundle source is created;
the bundle source contains exactly the publish inputs plus the current attempt's R2R output;
files left by a killed or previous attempt are removed.

Apple MSBuild changes set the apple_msbuild path-filter variable. A dedicated macOS x64 job then runs the test project directly through the normal runtime build entry point; non-PR rolling builds run it as well. The test does not require an Apple SDK or device.

Validation

./build.sh --projects "$(pwd)/src/mono/msbuild/apple/tests/Apple.Build.Tests/Apple.Build.Tests.csproj" -c Release -test: 1 passed, 0 failed.
An earlier form of the same staging fix ran System.Net.Security.Unit.Tests three times in one tvOS Helix work-item directory in build 1558829. All three attempts rebuilt R2R, launched the app, reached 128 tests, and had no stripped-IL or APP_CRASH failure. The work item remained red because of a separate certificate-context test failure.

Note

This description and change were prepared with GitHub Copilot assistance and reviewed by the submitting developer.

The Helix ReadyToRun step copied the crossgen2 output over the publish directory, but that directory is also the crossgen2 input. When the AOT build runs twice in the same work item - a Helix work item retry re-runs the whole command in the same directory - the second run compiles the already compiled assemblies again, and --strip-il-bodies drops the IL of an image that no longer has the original method bodies. The app then dies at startup with "A method body required at runtime was stripped from the ReadyToRun image". Stage the app payload in obj/r2r-publish instead: copy the pristine publish tree there, overlay the crossgen2 output on top of it, and point AppleBuildDir and the Apple*ToBundle items at the staging directory. AppleAppBuilder bundles every top level entry of its AppDir, so redirecting the item lists alone is not enough, the AppDir itself has to move out of the pristine input directory. Also fail with a clear error if the publish directory already contains R2R output, and remove obj/R2R before compiling so that a partially written attempt cannot look up to date to the next one. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

azure-pipelines · 2026-08-18T20:39:18Z

Azure Pipelines: Successfully started running 3 pipeline(s). 13 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run.

Copilot

Pull request overview

This PR addresses a Helix retry hazard in Apple mobile CoreCLR test app ReadyToRun (R2R) compilation by preventing crossgen2 outputs from being written back into the publish/ directory (which is also the crossgen2 input). Instead, it stages a pristine app payload into an intermediate directory and overlays R2R outputs there, ensuring retries recompile from original IL inputs.

Changes:

Introduces an obj/r2r-publish/ staging directory: copies the original publish/ payload into it, then overlays crossgen2 outputs, and builds the app bundle from the staged payload.
Ensures retry safety by recreating the staging directory on every run and deleting obj/R2R before recompilation.
Adds guardrails: warnings when R2R outputs are unexpectedly empty and a fail-fast error if stale *.r2r.dylib outputs are detected in publish/.

A payload can legitimately arrive with ReadyToRun output in it: a test project that sets PublishReadyToRun itself keeps that setting on the build machine, where AppleBuild.props only suppresses the default, so the composite image is published and then shipped inside the work item. Failing the build on the mere presence of a *.r2r.dylib would break the first attempt of such a work item, and the guard is not what keeps the crossgen2 inputs pristine, the staging directory is. Keep the diagnostic, drop the hard failure, and say what the consequence is. Also make the comment on the obj/R2R cleanup match what the SDK actually tracks: the composite image is an output of _CreateR2RImages, the per assembly files written next to it are not. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Measuring the actual payload showed the check has no signal. The build machine does not strip IL when it publishes ReadyToRun for these projects, because the in tree crossgen2 targets never pass the composite strip argument: an agent published payload has 145956 method bodies and zero stripped ones, and only the Helix compilation produces stripped bodies. So the only thing that can still put a .r2r.dylib in the publish directory is a build machine that pre compiled the app, where compiling again is fine, and the message claimed the opposite. The staging directory is what keeps the crossgen2 inputs pristine. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

The Apple Helix ReadyToRun proxy build had no test surface, so the retry bug it just got fixed for could come back unnoticed. Cover it from Wasm.Build.Tests, the only in-tree suite that already runs MSBuild driven build tests on a CI build machine. The test ships the real ProxyProjectForAOTOnHelix.proj into the test payload and drives it with `dotnet msbuild`, so the production target logic is what is under test rather than a copy of it. It lays out a Helix shaped work item, stands in for crossgen2 with a stub props file, and runs the R2R targets twice in the same directory with leftovers from a killed attempt planted in between. Exact manifests assert that the publish directory stays byte for byte identical across both runs, that the staging directory is rebuilt from scratch, and that AppleBuildDir and the Apple*ToBundle item lists resolve inside the staging directory. Wire src/mono/msbuild/apple/* into the wasmbuildtests path set so a change to the proxy project actually runs the test on PR CI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/mono/wasm/Wasm.Build.Tests/AppleHelixR2RTests.cs:356

ReplaceDirectoryContent uses TryDeleteDirectory, which swallows IOException/UnauthorizedAccessException. Since this method is intended to ensure the crossgen output directory is clean between attempts, suppressing delete failures can leave stale files behind and make the test nondeterministic (or validate the wrong behavior). It’s better to delete the directory deterministically (or fail) and then recreate/populate it.

    private static void ReplaceDirectoryContent(string directory, (string RelativePath, string Content)[] files)
    {
        TryDeleteDirectory(directory);
        foreach ((string relativePath, string content) in files)
            WriteFile(Path.Combine(directory, ToNativePath(relativePath)), content);

The run script builds SDK_FOR_WORKLOAD_TESTING_PATH from $(dirname $0), so it can be a relative path, and the child msbuild runs with a different working directory. Resolve the host to a full path the way BuildEnvironment already does, and point DOTNET_ROOT, DOTNET_INSTALL_DIR and PATH at it for the same reason that file gives: the repo build environment sets them to its own dotnet. Narrow the path trigger to src/mono/msbuild/apple/data/*. The test only ever loads the proxy project from data/, and the AppleBuild props and targets next to it are shielded by the stub Directory.Build files, so triggering the whole browser wasm test matrix on a change to those would buy no coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

The previous commit prepended the resolved host directory to PATH by interpolating the parent value, which yields a trailing separator when the parent has no PATH at all. An empty PATH entry means the working directory on unix, and the child runs with the work item publish directory as its working directory, so keep the separator out in that case. Also say in the path trigger comment which files under apple/data the test actually covers: it replaces the Directory.Build files with stubs, so a change to those runs the test without exercising them. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/mono/wasm/Wasm.Build.Tests/AppleHelixR2RTests.cs:378

ReplaceDirectoryContent deletes the directory via TryDeleteDirectory, which silently swallows IOException/UnauthorizedAccessException. In this method that behavior can hide real test setup failures and leave stale files behind (e.g., if deletion fails on Windows due to file locks), potentially making the test flaky or misleading. Prefer a deterministic delete that fails the test when the directory cannot be cleared; keep TryDeleteDirectory only for best-effort cleanup in the finally path.

    private static void ReplaceDirectoryContent(string directory, (string RelativePath, string Content)[] files)
    {
        TryDeleteDirectory(directory);
        foreach ((string relativePath, string content) in files)
            WriteFile(Path.Combine(directory, ToNativePath(relativePath)), content);

davidnguyen-tech · 2026-08-19T08:43:47Z

/azp run runtime-extra-platforms

azure-pipelines · 2026-08-19T08:44:07Z

Azure Pipelines: Successfully started running 1 pipeline(s).

dotnet-policy-service · 2026-08-21T10:39:54Z

Tagging subscribers to this area: @akoeplinger, @matouskozak, @simonrozsival
See info in area-owners.md if you want to be subscribed.

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 780282fa-426e-4ebe-96ab-14cd314cce54

davidnguyen-tech · 2026-08-21T14:13:40Z

Right now, the PR introduces a new subset - will change it not to.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 780282fa-426e-4ebe-96ab-14cd314cce54

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Suppressed comments (1)

src/mono/msbuild/apple/tests/Directory.Build.props:14

The root props identifies this as a test project because it is under tests and ends in .Tests, but importing only the repository root bypasses src/libraries/Directory.Build.props. That is the import that enables eng/testing/tests.props and its implicit xUnit references, so the new Xunit usages cannot compile in this project. Import the libraries props here (it imports the root props itself).

  <Import Project="..\..\..\..\..\Directory.Build.props" />

azure-pipelines · 2026-08-21T14:38:29Z

Azure Pipelines: Successfully started running 5 pipeline(s). 11 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run.

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/mono/msbuild/apple/tests/Directory.Build.props:14

This import bypasses src/libraries/Directory.Build.props, which is the layer that sets EnableTestSupport and imports eng/testing/tests.props (including the xUnit package setup). With only the root props imported, this project uses Fact/ITestOutputHelper without the implicit xUnit references and does not get the test-support properties used by the -test invocation. Import the libraries props (as Wasm.Build.Tests does) and configure this project for default source items/test support before relying on it as regression coverage.

  <Import Project="..\..\..\..\..\Directory.Build.props" />

src/mono/msbuild/apple/tests/Directory.Build.targets:4

The matching targets import also bypasses src/libraries/Directory.Build.targets, so eng/testing/tests.targets is never loaded. That file defines the Test target (and GenerateRunScript/RunTests) that the new pipeline invokes with -test; importing only the repository-root targets leaves that command unavailable. Route this import through the libraries targets after wiring the test project into the standard test-support chain.

  <Import Project="..\..\..\..\..\Directory.Build.targets" />

davidnguyen-tech · 2026-08-21T14:51:48Z

/azp run runtime-extra-platforms

azure-pipelines · 2026-08-21T14:52:01Z

Azure Pipelines: Successfully started running 1 pipeline(s).

BrzVlad · 2026-08-24T18:26:06Z

Might fix #126458 as well.

BrzVlad · 2026-08-25T05:57:38Z

+      #
+      # Apple MSBuild changes trigger this host test project directly on an ordinary macOS build host.
+      # It only drives MSBuild and file orchestration, so it needs no Apple mobile workload or device.
+      - template: /eng/pipelines/common/platform-matrix.yml


I'm not that familiar with the infrastructure bits or the logic of this new test but I'm wondering whether this is overkill. Do we really need this new job that will only ever have a single test which seems quite complex to me, trying to mimic helix packaging behavior. I'm wondering if it would be simpler and also provide better coverage to have a simple runtime test, that we make sure is being sent to helix as an independent work item. Maybe as part of a helix retry, the work item would receive some additional env var that the test could end up checking. If this env var is not set, the test would fail, making sure we hit the retry path. Then, on second run, the test would pass, effectively testing this logic regardless of configuration, r2r or not r2r, mobile or desktop, also testing exactly what happens as part of normal test runs.

@akoeplinger for second opinion in case I'm speaking nonsense

I agree it looks like an overkill at the moment.

I was thinking more in long-term. This could be the foundation for more tests - which might be easier to add than this initial one.

I'm not sure how much we want to test infra though.

I would not add a dedicated job as that requires provisioning azure runner and other stuff. We could add this to the tvOS CoreCLR job (either as another workitem) or a build stage.

I don't have a strong opinion on whether to test this or not :)

matouskozak · 2026-08-26T17:34:19Z

Might fix #126458 as well.

If I'm reading this PR right, I think this is scoped purely for CoreCLR R2R and shouldn't affect Mono.

matouskozak · 2026-08-26T18:14:21Z

Overall looking good, except for the test inclusion/location.

Before merging this, can we test if this fixes also the other tests where we disabled the IL stripping (e.g. #130468 and others)

Copilot AI lite review requested due to automatic review settings August 18, 2026 20:38

dotnet-policy-service Bot assigned davidnguyen-tech Aug 18, 2026

github-actions Bot added the area-ReadyToRun label Aug 18, 2026

Copilot started reviewing on behalf of davidnguyen-tech August 18, 2026 20:39 View session

Copilot AI reviewed Aug 18, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings August 18, 2026 21:57

Copilot started reviewing on behalf of davidnguyen-tech August 18, 2026 21:57 View session

Copilot AI reviewed Aug 18, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings August 18, 2026 23:28

Copilot started reviewing on behalf of davidnguyen-tech August 18, 2026 23:29 View session

Copilot AI reviewed Aug 18, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings August 19, 2026 00:39

Copilot started reviewing on behalf of davidnguyen-tech August 19, 2026 00:39 View session

Copilot AI reviewed Aug 19, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings August 19, 2026 01:54

Copilot started reviewing on behalf of davidnguyen-tech August 19, 2026 01:55 View session

Copilot AI reviewed Aug 19, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings August 19, 2026 02:53

Copilot started reviewing on behalf of davidnguyen-tech August 19, 2026 02:54 View session

Copilot AI reviewed Aug 19, 2026

View reviewed changes

build-analysis Bot mentioned this pull request Aug 19, 2026

slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes." dotnet/dnceng#1883

Open

3 tasks

davidnguyen-tech temporarily deployed to copilot-pat-pool August 21, 2026 10:39 — with GitHub Actions Inactive

davidnguyen-tech removed the area-ReadyToRun label Aug 21, 2026

davidnguyen-tech temporarily deployed to copilot-pat-pool August 21, 2026 10:39 — with GitHub Actions Inactive

davidnguyen-tech requested a review from matouskozak August 21, 2026 10:43

davidnguyen-tech mentioned this pull request Aug 21, 2026

Exclude desktop-only Security test from mobile builds #132467

Closed

Copilot AI reviewed Aug 21, 2026

View reviewed changes

Comment thread src/mono/msbuild/apple/tests/Directory.Build.props

davidnguyen-tech added 2 commits August 21, 2026 15:55

Apply unstaged changes for CI filtering

8d5d92c

Clarify direct Apple build test wiring

528e6c9

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 780282fa-426e-4ebe-96ab-14cd314cce54

Copilot AI review requested due to automatic review settings August 21, 2026 14:13

Copilot started reviewing on behalf of davidnguyen-tech August 21, 2026 14:14 View session

Record required Copilot attribution

c71a7f8

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 780282fa-426e-4ebe-96ab-14cd314cce54

davidnguyen-tech marked this pull request as draft August 21, 2026 14:17

Copilot AI reviewed Aug 21, 2026

View reviewed changes

Comment thread src/mono/msbuild/apple/tests/Apple.Build.Tests/Apple.Build.Tests.csproj

Comment thread src/mono/msbuild/apple/tests/Directory.Build.targets

davidnguyen-tech marked this pull request as ready for review August 21, 2026 14:38

Copilot AI review requested due to automatic review settings August 21, 2026 14:38

Copilot started reviewing on behalf of davidnguyen-tech August 21, 2026 14:38 View session

Copilot AI reviewed Aug 21, 2026

View reviewed changes

This was referenced Aug 21, 2026

Android WebSocket failure #121518

Open

[ci-scan] Build break: Mono AOT compile of JIT.Generics_ro fails with recursive/invalid generic instantiation on linux-x64 MiniF [Content truncated due to length] #132559

Open

davidnguyen-tech requested review from BrzVlad and akoeplinger August 24, 2026 18:13

BrzVlad reviewed Aug 25, 2026

View reviewed changes

Uh oh!

Conversation

davidnguyen-tech commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Regression coverage

Validation

Uh oh!

azure-pipelines Bot commented Aug 18, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

davidnguyen-tech commented Aug 19, 2026

Uh oh!

azure-pipelines Bot commented Aug 19, 2026

Uh oh!

dotnet-policy-service Bot commented Aug 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

davidnguyen-tech commented Aug 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

azure-pipelines Bot commented Aug 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

davidnguyen-tech commented Aug 21, 2026

Uh oh!

azure-pipelines Bot commented Aug 21, 2026

Uh oh!

BrzVlad commented Aug 24, 2026

Uh oh!

BrzVlad Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

davidnguyen-tech Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

matouskozak Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

matouskozak commented Aug 26, 2026

Uh oh!

matouskozak commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

davidnguyen-tech commented Aug 18, 2026 •

edited

Loading

matouskozak Aug 26, 2026 •

edited

Loading

matouskozak commented Aug 26, 2026 •

edited

Loading