How to modify the container registry?

[Varorbc]

I noticed that by default, the image is pulled from docker.io. Due to network issues, I need to modify the registry for pulling the image. What should I do?

[davidfowl]

https://learn.microsoft.com/en-us/dotnet/api/aspire.hosting.containerresourcebuilderextensions.withimageregistry?view=dotnet-aspire-8.0

You can see how the integrations are implemented:

aspire/src/Aspire.Hosting.PostgreSQL/PostgresBuilderExtensions.cs

Line 107 in 5fd2b10

.WithImageRegistry(PostgresContainerImageTags.Registry)

[avendel]

We are trying to use WithImageRegistry to pull images from a central Azure Container Registry (ACR) to which we push our released images. That is, another registry than the one created by azd provision or azd up, which we don't really need.

The scenario we are looking for is this, which I think will be quite common for companies wanting to have more control over the exact binaries/containers deployed and make sure the same image is used in all environments.

1. Use Aspire together with azd.
2. Build docker images in a prior build step in order to push them to a central ACR.
3. Use azd deploy to deploy the application to multiple environments, pulling from the ACR from step 2.

Here is an attempt at visualizing this:

graph TB
    subgraph Azure Environments
        Test
        Staging
        Production
    end
    
    subgraph "Pipeline runners"
        BuildStep["Build Step"]
        DeploySteps["Deploy Steps"]
    end
    CentralACR["Central ACR"]

    BuildStep-->|docker push| CentralACR
    DeploySteps-->|azd deploy -e test --no-prompt| Test
    DeploySteps-->|azd deploy -e staging --no-prompt| Staging
    DeploySteps-->|azd deploy -e prod --no-prompt| Production

    Test -->|docker pull| CentralACR
    Staging -->|docker pull| CentralACR
    Production -->|docker pull| CentralACR

Loading

We have encountered three problems with this, and are looking for advice, especially for the last one related to authentication.

1. It is not possible to use AddProject<T>, since this will create and push new images on azd deploy. We had to get around that by using AddContainer if builder.ExecutionContext.IsPublishMode == true. This is a bit awkward since it means we have to duplicate the application setup code for local development and Azure deployment.

2. When we use azd deploy in a separate build step from azd provision, the process fails with message

   failed deploying service '': logging in to registry: could not determine container registry endpoint, ensure 'registry' has been set in the docker options or 'AZURE_CONTAINER_REGISTRY_ENDPOINT' environment variable has been set

   The AZURE_CONTAINER_REGISTRY_ENDPOINT is not available unless you do azd up or azd provision, but we don't have any use for it since we have specified another registry for our images.

3. If we get around problem 2 by using azd up instead, how is authentication with the central ACR supposed to be handled?
   Azure Developer Cli handles authentication with the ACR it created, but how does it deal with another registry specified with WithImageRegistry? The recommendations from How to authenticate for AddContainer #3440 will not work, since the commands are run from where azd is executed, but I assume ACA pulls images in the Azure environment created by azd.

   How does authentication with a registry specified with WithImageRegistry work?

[davidfowl]

For what you want to accomplish, the azd + aspire combination isn’t flexible enough. With the changes in 9.2 it might be possible to tell azd what your container registry is in order to trick it to build and push containers there (by using the new AddAzureContainerAppEnviroment resource).

1. Yea, it’s weird. You can do trickery with annotations or by looking at how PublishAsDockerFile works, but I don’t recommend that.
2. Do you do it in a separate step or job? Azd deploy depends on state from azd provision, if it can’t find that state then those errors occur.
3. I don’t think you want to use WithImageRegistry. That’s for static values and the auth story is non existent for local dev (people are using az acr login as a pre step in the app host).

If you are willing to continue to use azd for deploying then I think the simplest change you could make (which is not really simple) is updating to 9.2 (if you haven’t already) and using AddAzureContainerAppEnviroment and ConfigureInfrastructure to manipulate the bicep in a way that points azd to your ACR instead of the ones that bicep tries to create.

See @FullStackChef have the same problem #8923 (comment)

[FullStackChef]

So the good news is I had a very similar scenario to this working (more or less - there are a couple of manual tweaks I was still having to make via infra synth)
The bad news is this was work done as part of my role in an organisation I finished up at yesterday -meaning I no longer have access to the code. I will try to write up my process in the next couple of days while it's still semi fresh in my head.

[avendel]

Thanks for your replies. Will have a look at the code @FullStackChef has posted. For now we have chosen to go with az rather than azd and use Aspire for local development.

But in summary, what I'm aiming for is to make sure a container image we have built and tested is also the one deployed to different environments, pulled from a central registry containing tested and released images. I feel like this is one of the appeals of containerized deployment: The software binaries running in production - with its dependencies - are the same - with the same dependencies - as what we have tested.

[FullStackChef]

@avendel
Here is part of the solution (https://github.com/Fieldstate-Forge/Aspire.Hosting.Extensions/tree/main/Fieldstate.Aspire.Hosting.Azure.AppContainers)

I'm almost finished writing up what I've done -but in the meantime the concept here is

• Remove the container registry from the CAE and re-create it in it's own module
• Accept the CAE managed identity as a parameter - we need to grant this role access to the registry
• Reference the registry in the CAE
• Remove the outputs from the CAE
• We set the ContainerImage Name in the project to an empty string - it's a bit hacky but it prevents the standard CAE context from trying to add reference expressions that no longer exist - we rebuild the image manually
• Add a configuration task that sets the registry information

Notes: I've not written any unit tests yet, I have run some manual tests

• Manually validated generated bicep using azd infra synth (using new registry and the aspire PublishAsExisiting() method)
• Deploys targeting a new registry (no changes to bicep output required -e.g no azd infra synth)
• Deploy using existing - fails based on the fact that I was too lazy to set up a registry before hand (fails where expected)

With this the process flow should be possible but sing out if you get stuck.

@davidfowl is this too hacky to contribute to the aspire repo?

[FullStackChef]

Note - I've commited the infra synth output for reference sake as well

[davidfowl]

@davidfowl is this too hacky to contribute to the aspire repo?

Absolutely too hacky 😄 as is but with small tweaks to azd and aspire it could be made not hacky.

1. An AzureContainerRegistryResource type (we have this new abstraction https://github.com/dotnet/aspire/blob/main/src/Aspire.Hosting.Azure/IAzureContainerRegistry.cs)
2. First class support in the AzureContainerAppEnvironmentResource to support a referencing an existing IAzureContainerRegistry

• In this mode we wouldn't create the container registry but we would still create the managed identity to setup role assignments for pull access.

4. Potential azd changes? Unsure if anything needs to change here if the acr registry endpoint gets output like normal.