Publish artifacts and symbols one file at a time by epananth · Pull Request #7086 · dotnet/arcade

epananth · 2021-03-10T18:10:37Z

To double check:

The right tests are in and and the right validation has happened. Guidance: https://github.com/dotnet/core-eng/tree/master/Documentation/Validation

Issue -> #6987

Both the old functionality to download the entire artifact folder is still present and can be turned on by enabling the artifact download task and switching UseStreamingPublishing= false in the publish-assets.yml

There is definitely some redundant code, I'm hoping code review will help me make it better.

NOTE : PdbArtifacts are still downloaded like we did earlier, cos there is not way to get the file names before hand to download them from PdbArtifacts directory.

…ng-pr

epananth · 2021-03-16T07:46:20Z

Ran some tests

Runtime Latest build -> https://dnceng.visualstudio.com/internal/_build/results?buildId=1040969&view=results
vs Runtime build with new hosted pool -> https://dnceng.visualstudio.com/internal/_build/results?buildId=1040959&view=results

Other promotion builds
dotnet-sdk -
https://dnceng.visualstudio.com/internal/_build/results?buildId=1040973&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=6e277ba4-1c1e-552d-b96f-db0aeb4be20a

dotnet-winforms -
https://dnceng.visualstudio.com/internal/_build/results?buildId=1040984&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=6e277ba4-1c1e-552d-b96f-db0aeb4be20a

…ng-pr

epananth · 2021-03-16T08:07:06Z

More test results from day before

https://dnceng.visualstudio.com/internal/_build/results?buildId=1039004&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=6e277ba4-1c1e-552d-b96f-db0aeb4be20a - 20 m 48 seconds
https://dnceng.visualstudio.com/internal/_build/results?buildId=1039011&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=6e277ba4-1c1e-552d-b96f-db0aeb4be20a - 20 m 46 seconds
https://dnceng.visualstudio.com/internal/_build/results?buildId=1039034&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=6e277ba4-1c1e-552d-b96f-db0aeb4be20a - 21 m 42 seconds

this download with the new hosted pool took 31 minutes https://dnceng.visualstudio.com/internal/_build/results?buildId=1038957&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=cecdd0ae-1666-570d-4521-d44005725bca, compared to the whole thing taking https://dev.azure.com/dnceng/internal/_build/results?buildId=1039004&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=6e277ba4-1c1e-552d-b96f-db0aeb4be20a 20 m 48 seconds..

epananth · 2021-03-16T08:10:39Z

Odd one

Runtime took 31 mins
https://dnceng.visualstudio.com/internal/_build/results?buildId=1040978&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=6e277ba4-1c1e-552d-b96f-db0aeb4be20a

epananth · 2021-03-16T08:13:41Z

I ran tests for different promotions parallelly, and they all succeeded. I have also added retry logic for downloading files. So I think I covered most of the tests, we had discussed.

ghost

First round of feedback. Still need to finish looking at all the files.

ghost

Second pass. Two files to go

…ng-pr

ghost

I only have a couple more nits and comments but provided you have some test builds with these latest changes that we can take a look at, this LGTM,

mmitche · 2021-04-06T16:00:36Z

        /// </summary>
        /// <param name="client">Azdo client</param>
-        /// <param name="artifact">If it is PackageArtifacts or BlobArtifacts</param>
+        /// <param name="ArtifactName">If it is PackageArtifacts or BlobArtifacts</param>


Nit: the parameter name is artifactName (casing)

Updated this

mmitche · 2021-04-06T16:01:11Z

-
-        }
-
-        private async Task PublishPackagesToAzureStorageNugetByDownloadingEntireFolderAsync(HashSet<PackageArtifactModel> packagesToPublish,


mmitche

LGTM. Thanks for iterating on this. Complicated stuff.

michellemcdaniel

I'm happy with how it looks in the staging pipeline (the only failures make sense/are because the files had already been uploaded since we tried on a bar id that already has been published). LGTM

epananth · 2021-04-06T17:32:47Z

Latest test against runtime

Publishing	Link	Time taken to publish
New infra Bar Id - 85015	https://dev.azure.com/dnceng/internal/_build/results?buildId=1073502&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=6e277ba4-1c1e-552d-b96f-db0aeb4be20a	24mins 56s
old infra Bar Id - 85015	https://dev.azure.com/dnceng/internal/_build/results?buildId=1072291&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=6e277ba4-1c1e-552d-b96f-db0aeb4be20a	23mins 44secs + 3 mins 48 secs
New infra (bar id - 84977)	https://dev.azure.com/dnceng/internal/_build/results?buildId=1073701&view=logs&j=33961d73-1e6b-5306-1d3b-c6c083bff7de&t=577fb5b9-bfb7-5430-eb7a-5c5120b89b4d	25 mins 51 sec
old infra (bar id - 84977)	https://dev.azure.com/dnceng/internal/_build/results?buildId=1071605&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=6e277ba4-1c1e-552d-b96f-db0aeb4be20a	31 mins 32 sec + 3 min 8 sec

epananth · 2021-04-07T17:09:16Z

I did some test against installer and I am currently facing some IO exception, used by another process for blob publishing. I am investigating this issue, so I am not going to merge this PR till that issue is fixed.

epananth · 2021-04-08T22:25:26Z

Aspnet core -> https://dev.azure.com/dnceng/internal/_build/results?buildId=1079097&view=results
Installer -> https://dev.azure.com/dnceng/internal/_build/results?buildId=1079093&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=6e277ba4-1c1e-552d-b96f-db0aeb4be20a
Runtime -> https://dev.azure.com/dnceng/internal/_build/results?buildId=1079083&view=logs&s=380ab8c6-4139-55ff-15c4-fb48260f4dca&j=ba23343f-f710-5af9-782d-5bd26b102304

epananth · 2021-04-08T22:30:58Z

In the last change, I added the following

Added await to CompareLocalPackageToFeedPackage
Making sure PackageArchiveReader was using the correct scope
added retry to ContainerIdAsync
More logs for downloading file
Added retry to directory delete
Updated the test

epananth · 2021-04-09T00:06:05Z

@mmitche mentioned my change is good to merge. :)

epananth · 2021-04-09T00:07:42Z

Installer build first time - https://dev.azure.com/dnceng/internal/_build/results?buildId=1079093&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=6e277ba4-1c1e-552d-b96f-db0aeb4be20a

did delete-build-from-channel and retried build - https://dev.azure.com/dnceng/internal/_build/results?buildId=1078127&view=results

* Improve publishing performance

epananth added 2 commits March 10, 2021 10:05

Improve publishing performance

2ca72c8

remove unwanted param

8197834

epananth changed the title ~~Publishing pr~~ Publish artifacts and symbols one file at a time Mar 10, 2021

epananth added 9 commits March 10, 2021 14:17

some fixes

9d3a66e

more update

271e5c2

minor fixes

23fd68d

Merge branch 'main' of https://github.com/dotnet/arcade into publishi…

086bc12

…ng-pr

Merge branch 'main' of https://github.com/dotnet/arcade into publishi…

583c310

…ng-pr

Disabled package download

b82c1c0

fix

ff093e1

Remove some logging

91f474a

More fix

68e8ea8

epananth marked this pull request as ready for review March 16, 2021 07:51

Merge branch 'main' of https://github.com/dotnet/arcade into publishi…

7c1325a

…ng-pr

ghost reviewed Mar 16, 2021

View reviewed changes

Comment thread eng/publishing/v3/publish-assets.yml Outdated

ghost reviewed Mar 16, 2021

View reviewed changes

Comment thread src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs Outdated

ghost reviewed Mar 16, 2021

View reviewed changes

Comment thread src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs Outdated

ghost reviewed Mar 16, 2021

View reviewed changes

epananth added 7 commits March 17, 2021 18:48

Merge branch 'main' of https://github.com/dotnet/arcade into publishi…

30e72f6

…ng-pr

test sdk

037401b

test

6d73f0b

review comments

ff239e8

merge conflict

42edf24

test

696ae33

review comments:

3cc26df

epananth requested review from a user and michellemcdaniel and removed request for a user April 5, 2021 18:23

ghost reviewed Apr 5, 2021

View reviewed changes

Comment thread src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs Outdated

ghost reviewed Apr 5, 2021

View reviewed changes

Comment thread src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs

ghost approved these changes Apr 5, 2021

View reviewed changes

review comment

86215a9

mmitche reviewed Apr 6, 2021

View reviewed changes

mmitche approved these changes Apr 6, 2021

View reviewed changes

michellemcdaniel approved these changes Apr 6, 2021

View reviewed changes

last change

8de357d

This was referenced Apr 6, 2021

Downloading artifacts before publishing is exceeding 10gb, in runtime #6987

Closed

Unblock runtime publishing #6988

Closed

epananth added 2 commits April 8, 2021 14:13

IO exception fix

8a5d431

test fix

07f63c4

epananth requested review from a user and michellemcdaniel April 8, 2021 23:39

ghost approved these changes Apr 9, 2021

View reviewed changes

epananth merged commit 23b4c45 into dotnet:main Apr 12, 2021

akoeplinger pushed a commit to akoeplinger/arcade that referenced this pull request Apr 12, 2021

Publish artifacts and symbols one file at a time (dotnet#7086)

0a92032

* Improve publishing performance

epananth deleted the publishing-pr branch April 15, 2021 05:30

danmoseley mentioned this pull request Feb 7, 2022

Add RuntimeConfiguration.NotRelease danmoseley/arcade#1

Closed

1 task


		}

		private async Task PublishPackagesToAzureStorageNugetByDownloadingEntireFolderAsync(HashSet<PackageArtifactModel> packagesToPublish,

Uh oh!

Conversation

epananth commented Mar 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

To double check:

Uh oh!

epananth commented Mar 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

epananth commented Mar 16, 2021

Uh oh!

epananth commented Mar 16, 2021

Uh oh!

epananth commented Mar 16, 2021

Uh oh!

Uh oh!

ghost left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ghost left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ghost left a comment

Choose a reason for hiding this comment

Uh oh!

mmitche Apr 6, 2021

Choose a reason for hiding this comment

Uh oh!

epananth Apr 6, 2021

Choose a reason for hiding this comment

Uh oh!

mmitche Apr 6, 2021

Choose a reason for hiding this comment

Uh oh!

mmitche left a comment

Choose a reason for hiding this comment

Uh oh!

michellemcdaniel left a comment

Choose a reason for hiding this comment

Uh oh!

epananth commented Apr 6, 2021

Uh oh!

epananth commented Apr 7, 2021

Uh oh!

epananth commented Apr 8, 2021

Uh oh!

epananth commented Apr 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

epananth commented Apr 9, 2021

Uh oh!

epananth commented Apr 9, 2021

Uh oh!

Reviewers

Assignees

Labels

epananth commented Mar 10, 2021 •

edited

Loading

epananth commented Mar 16, 2021 •

edited

Loading

epananth commented Apr 8, 2021 •

edited

Loading