Official build failure trying to sign Microsoft.NET.Sdk.WebAssembly.Pack

[sbomer]

Official builds started failing after #84082. Link to build: https://dev.azure.com/dnceng/internal/_build/results?buildId=2156574&view=logs&j=4d50a8bf-a143-51c7-5cc8-defff437e23b&t=24058bbf-eb7e-5a58-7620-a06c45604c2a

The publish stage step Download Intermediate Artifacts says it downloaded the package:

Downloaded IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NET.Sdk.WebAssembly.Pack.8.0.0-preview.4.23212.5.nupkg to D:\a\_work\1\s\artifacts\PackageDownload\IntermediateArtifacts\MonoRuntimePacks\Shipping\Microsoft.NET.Sdk.WebAssembly.Pack.8.0.0-preview.4.23212.5.nupkg

But the next step Prepare artifacts and upload to build fails:

D:\a\_work\1\s\src\installer\prepare-artifacts.proj(146,5): error : Item to sign 'Microsoft.NET.Sdk.WebAssembly.Pack.8.0.0-preview.4.23212.5.nupkg' was not found in the artifacts
##[error]src\installer\prepare-artifacts.proj(146,5): error : (NETCORE_ENGINEERING_TELEMETRY=Build) Item to sign 'Microsoft.NET.Sdk.WebAssembly.Pack.8.0.0-preview.4.23212.5.nupkg' was not found in the artifacts
D:\a\_work\1\s\src\installer\prepare-artifacts.proj(146,5): error : Item to sign 'Microsoft.NET.Sdk.WebAssembly.Pack.8.0.0-preview.4.23212.5.symbols.nupkg' was not found in the artifacts
##[error]src\installer\prepare-artifacts.proj(146,5): error : (NETCORE_ENGINEERING_TELEMETRY=Build) Item to sign 'Microsoft.NET.Sdk.WebAssembly.Pack.8.0.0-preview.4.23212.5.symbols.nupkg' was not found in the artifacts
  Writing build manifest file 'D:\a\_work\1\s\artifacts\log\Release\AssetManifest/Manifest.xml'...

The failure comes from arcade here:
https://github.com/dotnet/arcade/blob/5ec6af6931967583ebedeeccc85c25769de902bc/src/Microsoft.DotNet.Build.Tasks.Feed/src/SigningInformationModelFactory.cs#L54-L60

I believe the PushToAzureDevOpsArtifacts invocation here:

runtime/src/installer/prepare-artifacts.proj

Lines 146 to 163 in 887c043

	<PushToAzureDevOpsArtifacts
	AzureDevOpsCollectionUri="$(SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)"
	AzureDevOpsProject="$(SYSTEM_TEAMPROJECT)"
	AzureDevOpsBuildId="$(BUILD_BUILDID)"
	ItemsToSign="@(ItemsToSignPostBuild)"
	StrongNameSignInfo="@(StrongNameSignInfo)"
	FileSignInfo="@(FileSignInfo)"
	FileExtensionSignInfo="@(FileExtensionSignInfo)"
	ItemsToPush="@(ItemsToPush)"
	ManifestBuildData="@(ManifestBuildData)"
	ManifestRepoUri="$(BUILD_REPOSITORY_NAME)"
	ManifestBranch="$(BUILD_SOURCEBRANCH)"
	ManifestBuildId="$(BUILD_BUILDNUMBER)"
	ManifestCommit="$(BUILD_SOURCEVERSION)"
	IsStableBuild="$(IsStableBuild)"
	AssetManifestPath="$(AssetManifestFile)"
	AssetsTemporaryDirectory="$(TempWorkingDir)"
	PublishingVersion ="3" />

includes the new package in ItemsToSign (which gets populated from the PackageDownload directory mentioned earlier), but not in ItemsToPush, and this is what causes the failure. I think the fix would be to add the new package to NupkgToPublishFile somewhere in this logic (I'm assuming the package should get published):

runtime/src/installer/prepare-artifacts.proj

Lines 188 to 259 in 887c043

	<DownloadedArtifactFile Include="$(DownloadDirectory)**" />
	<Error
	Condition="'@(DownloadedArtifactFile)' == ''"
	Text="No downloaded artifacts found." />
	<DownloadedSymbolNupkgFile Include="$(DownloadDirectory)**\*.symbols.nupkg" />
	<DownloadedWixPdbFile Include="$(DownloadDirectory)**\*.wixpdb" />
	<DownloadedWorkloadsVSInsertionFile Include="$(DownloadDirectory)*\workloads-vs\**\*" />
	<DownloadedNupkgFile
	Include="$(DownloadDirectory)**\*.nupkg"
	Exclude="@(DownloadedSymbolNupkgFile)" />
	<!-- Add files that are not affected by filtering. -->
	<UploadToBlobStorageFile
	Include="@(DownloadedArtifactFile)"
	Exclude="@(DownloadedSymbolNupkgFile);@(DownloadedNupkgFile);@(DownloadedWixPdbFile);@(DownloadedWorkloadsVSInsertionFile)" />
	<!--
	Filter out the RID-specific (Runtime) nupkgs and RID-agnostic nupkgs. RID-specific packages
	are published from every job. RID-agnostic nupkgs are built with the same ID/version by
	every job, so one specific job's outputs must be picked to sign and publish.
	-->
	<!-- RID-specific framework packs. -->
	<RuntimeNupkgFile
	Include="
	$(DownloadDirectory)**\Microsoft.*.Runtime.*.nupkg;
	$(DownloadDirectory)**\Microsoft.*.App.Host.*.nupkg;
	$(DownloadDirectory)**\Microsoft.*.App.Crossgen2.*.nupkg"
	Exclude="@(DownloadedSymbolNupkgFile)" />
	<!-- VS insertion packages, carrying RID-specific installers. -->
	<RuntimeNupkgFile
	Include="$(DownloadDirectory)**\VS.Redist.Common.*.nupkg"
	Exclude="@(DownloadedSymbolNupkgFile)" />
	<!--
	Workloads VS insertion artifacts produced by src/workloads/workloads.csproj. Only grab
	the zip artifacts as they're grouped by SDK feature band which correlates with specific VS versions.
	-->
	<WorkloadsVSInsertionFile
	Include="
	$(DownloadDirectory)*\workloads-vs\**\*.zip"/>
	<!--
	Runtime packages associated with some identity packages. Need to exclude "runtime.native.*"
	because Libraries produces some "runtime.native.Foo" packages with
	"runtime.<rid>.runtime.native.Foo" identity packages.
	-->
	<RuntimeNupkgFile
	Include="$(DownloadDirectory)**\runtime.*.nupkg"
	Exclude="
	$(DownloadDirectory)**\runtime.native.*.nupkg;
	@(DownloadedSymbolNupkgFile)" />
	<!--
	Packages that aren't matched above as RID-specific are considered RID-agnostic. Also include
	the AllConfigurations packages from the Libraries build.
	-->
	<RidAgnosticNupkgToPublishFile
	Include="
	$(DownloadDirectory)**\Microsoft.NET.Workload.Mono.Toolchain.*Manifest-*.nupkg;
	$(DownloadDirectory)*\$(PublishRidAgnosticPackagesFromPlatform)\**\*.nupkg;
	$(DownloadDirectory)*\*AllConfigurations\**\*.nupkg"
	Exclude="@(RuntimeNupkgFile);@(DownloadedSymbolNupkgFile)" />
	<TransportPackagesToPublishFile
	Include="$(DownloadDirectory)**\*Transport*.nupkg"
	Exclude="@(RuntimeNupkgFile);@(RidAgnosticNupkgToPublishFile);@(DownloadedSymbolNupkgFile)" />
	<NupkgToPublishFile Include="@(RuntimeNupkgFile);@(RidAgnosticNupkgToPublishFile);@(TransportPackagesToPublishFile)" />

@maraf @ViktorHofer

[lewing]

We can revert this to unblock if needed

[lewing]

@joeloff any thoughts on this one?

[lewing]

opened #84743 while we look into this

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Official builds started failing after #84082. Link to build: https://dev.azure.com/dnceng/internal/_build/results?buildId=2156574&view=logs&j=4d50a8bf-a143-51c7-5cc8-defff437e23b&t=24058bbf-eb7e-5a58-7620-a06c45604c2a

The publish stage step Download Intermediate Artifacts says it downloaded the package:

Downloaded IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NET.Sdk.WebAssembly.Pack.8.0.0-preview.4.23212.5.nupkg to D:\a\_work\1\s\artifacts\PackageDownload\IntermediateArtifacts\MonoRuntimePacks\Shipping\Microsoft.NET.Sdk.WebAssembly.Pack.8.0.0-preview.4.23212.5.nupkg

But the next step Prepare artifacts and upload to build fails:

D:\a\_work\1\s\src\installer\prepare-artifacts.proj(146,5): error : Item to sign 'Microsoft.NET.Sdk.WebAssembly.Pack.8.0.0-preview.4.23212.5.nupkg' was not found in the artifacts
##[error]src\installer\prepare-artifacts.proj(146,5): error : (NETCORE_ENGINEERING_TELEMETRY=Build) Item to sign 'Microsoft.NET.Sdk.WebAssembly.Pack.8.0.0-preview.4.23212.5.nupkg' was not found in the artifacts
D:\a\_work\1\s\src\installer\prepare-artifacts.proj(146,5): error : Item to sign 'Microsoft.NET.Sdk.WebAssembly.Pack.8.0.0-preview.4.23212.5.symbols.nupkg' was not found in the artifacts
##[error]src\installer\prepare-artifacts.proj(146,5): error : (NETCORE_ENGINEERING_TELEMETRY=Build) Item to sign 'Microsoft.NET.Sdk.WebAssembly.Pack.8.0.0-preview.4.23212.5.symbols.nupkg' was not found in the artifacts
  Writing build manifest file 'D:\a\_work\1\s\artifacts\log\Release\AssetManifest/Manifest.xml'...

The failure comes from arcade here:
https://github.com/dotnet/arcade/blob/5ec6af6931967583ebedeeccc85c25769de902bc/src/Microsoft.DotNet.Build.Tasks.Feed/src/SigningInformationModelFactory.cs#L54-L60

I believe the PushToAzureDevOpsArtifacts invocation here:

runtime/src/installer/prepare-artifacts.proj

Lines 146 to 163 in 887c043

	<PushToAzureDevOpsArtifacts
	AzureDevOpsCollectionUri="$(SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)"
	AzureDevOpsProject="$(SYSTEM_TEAMPROJECT)"
	AzureDevOpsBuildId="$(BUILD_BUILDID)"
	ItemsToSign="@(ItemsToSignPostBuild)"
	StrongNameSignInfo="@(StrongNameSignInfo)"
	FileSignInfo="@(FileSignInfo)"
	FileExtensionSignInfo="@(FileExtensionSignInfo)"
	ItemsToPush="@(ItemsToPush)"
	ManifestBuildData="@(ManifestBuildData)"
	ManifestRepoUri="$(BUILD_REPOSITORY_NAME)"
	ManifestBranch="$(BUILD_SOURCEBRANCH)"
	ManifestBuildId="$(BUILD_BUILDNUMBER)"
	ManifestCommit="$(BUILD_SOURCEVERSION)"
	IsStableBuild="$(IsStableBuild)"
	AssetManifestPath="$(AssetManifestFile)"
	AssetsTemporaryDirectory="$(TempWorkingDir)"
	PublishingVersion ="3" />

includes the new package in ItemsToSign (which gets populated from the PackageDownload directory mentioned earlier), but not in ItemsToPush, and this is what causes the failure. I think the fix would be to add the new package to NupkgToPublishFile somewhere in this logic (I'm assuming the package should get published):

runtime/src/installer/prepare-artifacts.proj

Lines 188 to 259 in 887c043

	<DownloadedArtifactFile Include="$(DownloadDirectory)**" />
	<Error
	Condition="'@(DownloadedArtifactFile)' == ''"
	Text="No downloaded artifacts found." />
	<DownloadedSymbolNupkgFile Include="$(DownloadDirectory)**\*.symbols.nupkg" />
	<DownloadedWixPdbFile Include="$(DownloadDirectory)**\*.wixpdb" />
	<DownloadedWorkloadsVSInsertionFile Include="$(DownloadDirectory)*\workloads-vs\**\*" />
	<DownloadedNupkgFile
	Include="$(DownloadDirectory)**\*.nupkg"
	Exclude="@(DownloadedSymbolNupkgFile)" />
	<!-- Add files that are not affected by filtering. -->
	<UploadToBlobStorageFile
	Include="@(DownloadedArtifactFile)"
	Exclude="@(DownloadedSymbolNupkgFile);@(DownloadedNupkgFile);@(DownloadedWixPdbFile);@(DownloadedWorkloadsVSInsertionFile)" />
	<!--
	Filter out the RID-specific (Runtime) nupkgs and RID-agnostic nupkgs. RID-specific packages
	are published from every job. RID-agnostic nupkgs are built with the same ID/version by
	every job, so one specific job's outputs must be picked to sign and publish.
	-->
	<!-- RID-specific framework packs. -->
	<RuntimeNupkgFile
	Include="
	$(DownloadDirectory)**\Microsoft.*.Runtime.*.nupkg;
	$(DownloadDirectory)**\Microsoft.*.App.Host.*.nupkg;
	$(DownloadDirectory)**\Microsoft.*.App.Crossgen2.*.nupkg"
	Exclude="@(DownloadedSymbolNupkgFile)" />
	<!-- VS insertion packages, carrying RID-specific installers. -->
	<RuntimeNupkgFile
	Include="$(DownloadDirectory)**\VS.Redist.Common.*.nupkg"
	Exclude="@(DownloadedSymbolNupkgFile)" />
	<!--
	Workloads VS insertion artifacts produced by src/workloads/workloads.csproj. Only grab
	the zip artifacts as they're grouped by SDK feature band which correlates with specific VS versions.
	-->
	<WorkloadsVSInsertionFile
	Include="
	$(DownloadDirectory)*\workloads-vs\**\*.zip"/>
	<!--
	Runtime packages associated with some identity packages. Need to exclude "runtime.native.*"
	because Libraries produces some "runtime.native.Foo" packages with
	"runtime.<rid>.runtime.native.Foo" identity packages.
	-->
	<RuntimeNupkgFile
	Include="$(DownloadDirectory)**\runtime.*.nupkg"
	Exclude="
	$(DownloadDirectory)**\runtime.native.*.nupkg;
	@(DownloadedSymbolNupkgFile)" />
	<!--
	Packages that aren't matched above as RID-specific are considered RID-agnostic. Also include
	the AllConfigurations packages from the Libraries build.
	-->
	<RidAgnosticNupkgToPublishFile
	Include="
	$(DownloadDirectory)**\Microsoft.NET.Workload.Mono.Toolchain.*Manifest-*.nupkg;
	$(DownloadDirectory)*\$(PublishRidAgnosticPackagesFromPlatform)\**\*.nupkg;
	$(DownloadDirectory)*\*AllConfigurations\**\*.nupkg"
	Exclude="@(RuntimeNupkgFile);@(DownloadedSymbolNupkgFile)" />
	<TransportPackagesToPublishFile
	Include="$(DownloadDirectory)**\*Transport*.nupkg"
	Exclude="@(RuntimeNupkgFile);@(RidAgnosticNupkgToPublishFile);@(DownloadedSymbolNupkgFile)" />
	<NupkgToPublishFile Include="@(RuntimeNupkgFile);@(RidAgnosticNupkgToPublishFile);@(TransportPackagesToPublishFile)" />

@maraf @ViktorHofer

Author:	sbomer
Assignees:	maraf
Labels:	arch-wasm, packaging, needs-area-label
Milestone:	8.0.0

[maraf]

This seems like the right pattern. I'll update it

runtime/src/installer/prepare-artifacts.proj

Lines 248 to 253 in 0ac097f

	<RidAgnosticNupkgToPublishFile
	Include="
	$(DownloadDirectory)**\Microsoft.NET.Workload.Mono.Toolchain.*Manifest-*.nupkg;
	$(DownloadDirectory)*\$(PublishRidAgnosticPackagesFromPlatform)\**\*.nupkg;
	$(DownloadDirectory)*\*AllConfigurations\**\*.nupkg"
	Exclude="@(RuntimeNupkgFile);@(DownloadedSymbolNupkgFile)" />

I'm curious, I ran the official build for the PR branch yesterday, and it passed. Does this build run something more?