Keycloak Postgres Aspire networking issues

[sdukehart-omnesoft]

Utilizing Aspire 9.1.0, I've been working on an extension to try to extend IResourceBuilder<KeycloakResource> to be able to utilize the Keycloak image's ability to integrate with a PostgreSQL database.

/// <summary>
///     Aspire Keycloak integration extensions for permitting working with PostgreSQL.
/// </summary>
/// <seealso href="https://www.keycloak.org/server/all-config#category-database" />
public static class KeycloakPostgresExtension
{
    private const string PostgresDbType = "postgres";
    private const string LogLevel = "info";
    private const string Localhost = "localhost";
    private const string LocalhostIp = "127.0.0.1";
    private const string PostgresDefaultPort = "5432";
    private const string Host = "Host";
    private const string Port = "Port";
    private const string Database = "Database";
    private const string Username = "Username";
    private const string Password = "Password";
    private const string KeycloakDbType = "KC_DB";
    private const string KeycloakLogLevel = "KC_LOG_LEVEL";
    private const string KeycloakDbUrlHost = "KC_DB_URL_HOST";
    private const string KeycloakDbUrlPort = "KC_DB_URL_PORT";
    private const string KeycloakDbUrlDatabase = "KC_DB_URL_DATABASE";
    private const string KeycloakDbSchema = "KC_DB_SCHEMA";
    private const string KeycloakDbUsername = "KC_DB_USERNAME";
    private const string KeycloakDbPassword = "KC_DB_PASSWORD";
    private const string KeycloakHttpEnabled = "KC_HTTP_ENABLED";
    private const string KeycloakHostname = "KC_HOSTNAME";

    public static IResourceBuilder<KeycloakResource> WithPostgres(
        this IResourceBuilder<KeycloakResource> keycloak,
        IResourceBuilder<AzurePostgresFlexibleServerDatabaseResource> pgDatabase,
        IResourceBuilder<ParameterResource>? databaseName = null,
        IResourceBuilder<ParameterResource>? schema = null)
    {
        ArgumentNullException.ThrowIfNull(pgDatabase);

        keycloak.WithEnvironment(async context =>
        {
            context.EnvironmentVariables[KeycloakDbType] = PostgresDbType;
            context.EnvironmentVariables[KeycloakLogLevel] = LogLevel;

            var connectionString =
                await pgDatabase.Resource.ConnectionStringExpression.GetValueAsync(CancellationToken.None);
            if (string.IsNullOrWhiteSpace(connectionString))
                throw new InvalidOperationException(
                    $"Invalid state; no connection string found for {pgDatabase.Resource.Name}");

            var connectionComponents = ConnectionStringToDictionary(connectionString);

            var hostname = connectionComponents[Host].Trim();
            var isLocalhost = hostname.ToLower() == Localhost || hostname == LocalhostIp;
            if (isLocalhost)
                // Aspire networking requires the name of the postgres container rather than localhost
                hostname = pgDatabase.Resource.Parent.Name;
            context.EnvironmentVariables[KeycloakDbUrlHost] = hostname;

            var port = connectionComponents[Port];
            if (isLocalhost)
                // Aspire networking requires the internal port of the postgres container rather than proxied port
                port = PostgresDefaultPort;
            context.EnvironmentVariables[KeycloakDbUrlPort] = port;

            if (databaseName is null && !connectionComponents.ContainsKey(Database))
                throw new InvalidOperationException(
                    $"Invalid state; no database name was provided or could be found in the connection string for {pgDatabase.Resource.Name}");

            context.EnvironmentVariables[KeycloakDbUrlDatabase] =
                databaseName is not null ? databaseName.Resource : connectionComponents[Database];

            if (schema is not null) context.EnvironmentVariables[KeycloakDbSchema] = schema.Resource;

            context.EnvironmentVariables[KeycloakDbUsername] = connectionComponents[Username];
            context.EnvironmentVariables[KeycloakDbPassword] = connectionComponents[Password];
        });
        return keycloak;
    }

    private static Dictionary<string, string> ConnectionStringToDictionary(string connectionString)
    {
        return connectionString.Split(';').Select(x => x.Split('=')).ToDictionary(x => x[0], x => x[1]);
    }
}

Unfortunately, there seems to be a bit of weirdness that I've had to include in it related to the Aspire-provided connection string and Aspire's networking.

The Aspire-provided connection string here looks something like Host=localhost;Port=63030;Username=<USERNAME_VALUE>;Password=<PASSWORD_VALUE>;Database=<DATABASE_NAME>, where the <...> placeholders have accurate information. Additionally, the Host and Port match with what the Aspire dashboard shows for the given database. However, using these values that correspond to the Aspire proxy in front of the Postgres database fail to permit traffic through.

If I go and look at the Aspire dashboard's environment variables for the Keycloak instance, I see a ConnectionStrings__<DATABASE_NAME> entry. However, it's value here is Host=postgres;Port=5432;Username=<USERNAME_VALUE>;Password=<PASSWORD_VALUE>;Database=<DATABASE_NAME>.

Using those Host and Port values allows my Keycloak instance to successfully communicate with my Postgres instance.

So ultimately my questions are:

1. Why am I getting a different connection string in the attached code than what is finally provided to the Aspire dashboard?
2. Why would the working solution require Host=postgres;Port=5432 instead of the proxied endpoint?

Thanks for any insights! And hopefully the Postgres extension will be useful to others!

[sdukehart-omnesoft]

This seems to be somewhat related to #2111 & #844.

[danegsta]

They port reported by the dashboard is the port the container is listening to requests from the host network; it’s effectively a port for the user to access the service (it would also be used by any services running on the host to talk to the container).

The host and port that you’re seeing in the connection string environment variable are for the internal container network for container to container communication. Service discovery automatically uses container networking for communication between container services.

[sdukehart-omnesoft]

I get that there's a distinction between the two. However, there seems to be inconsistency between what is provided via .WithEnvironment(async context => and what is available post-deployment. The expectations would be:

• That Aspire's dashboard would show the proxied endpoint in this case - it's not, and is showing the internal port
• That the proxied endpoint would be accessible by the Keycloak container app - it's not; connections cannot go from this container app to the Postgres instance via the proxied endpoint

[danegsta]

Container to container service discovery explicitly uses the internal container network because container to host networking for Aspire services is unreliable. It can work in some environments, but in many setups the host services are unreachable due to how Docker (and Podman) implement their network gateway and that the host services are listening only on localhost. The service discovery logic takes into account both the target and requesting service when generating connection info (so the host and port info for connecting to a given service could vary depending on where the request is coming from).

The host and port information reported on the dashboard should be for the local proxy backing all services on the host (unless proxying was disabled for a given endpoint). If you’re running Docker Desktop, you can likely reach that endpoint from a container, but you’ll need to use host.docker.internal:<service port> as localhost in a container isn’t able to access host ports (different networks). If you aren’t using Docker Desktop, you won’t be able to reach Aspire service host ports (proxy or otherwise) from your container.

[danegsta]

See: #6846 (comment) for an explanation on why container to host service communication won’t work in some circumstances.

#5628 is the PR where we enabled the container to container service discovery behavior.