Reuse default AWSOptions in client-specific config

[chase-miller]

Describe the feature

Expose the ability to register a service-client with service-specific config that reuses the default AWSOptions.

Use Case

I'd like to register an aws service with client-specific config overrides that uses the default AWSOptions. Without the ability to do this, I would have to duplicate option declarations (e.g. Credentials or ServiceURL) for each service-specific config I need to setup.

This is particularly important in scenarios where registrations can be modified outside of startup (e.g. WebApplicationFactory).

Ultimately I want my DI registrations to be the source of truth for how the AWS SDK behaves, and I want these registrations to be clean and DRY.

Here's a contrived example. An explicit AWSOptions instantiation is used for emphasis.

using Amazon.Extensions.NETCore.Setup;
using Amazon.Runtime;
using Amazon.S3;
using Amazon.SQS;

var builder = WebApplication.CreateBuilder(args);

builder.Services
    .AddDefaultAWSOptions(new AWSOptions
    {
        Credentials = new BasicAWSCredentials("accessKey", "accessSecret"),
        DefaultClientConfig =
        {
            ServiceURL = "localhost:6379",
        },
    })
    .AddAWSService<IAmazonSQS>() // sqs will use the default "global" options added above
    .AddSingleton<IAmazonS3>(sp =>
    {
        // The only way to create an AWSOptions object with an AmazonS3Config DefaultClientConfig prop is via this extension method.
        var awsOptions = sp.GetRequiredService<IConfiguration>().GetAWSOptions<AmazonSQSConfig>();

        // We have to declare these options again.
        awsOptions.Credentials = new BasicAWSCredentials("accessKey", "accessSecret");
        awsOptions.DefaultClientConfig.ServiceURL = "localhost:6379";

        var s3Config = (AmazonS3Config)awsOptions.DefaultClientConfig;
        s3Config.ForcePathStyle = true;
        s3Config.MaxErrorRetry = 10;
        // Set any other s3-specific config here

        return awsOptions.CreateServiceClient<IAmazonS3>();
    });

var app = builder.Build();

app.Run();

Proposed Solution

There are a number of approaches that could be taken. Here are some ideas in order of my personal preference (although I'd do 1 & 2 together):

1.

Expose an extension method that allows for service-specific config customization.

builder.Services
    .AddDefaultAWSOptions(sp => sp.GetRequiredService<IConfiguration>().GetAWSOptions())
    .AddAWSService<IAmazonSQS>()
    .AddAWSService<IAmazonS3, AmazonSQSConfig>((IServiceProvider sp, AmazonS3Config s3Config) =>
    {
        s3Config.ForcePathStyle = true;
        s3Config.MaxErrorRetry = 10;

        return s3Config;
    });

2.

Expose an extension method that provides a (new) AWSOptions object created using the default AWSOptions if one is registered.

builder.Services
    .AddDefaultAWSOptions(sp => sp.GetRequiredService<IConfiguration>().GetAWSOptions())
    .AddAWSService<IAmazonSQS>()
    .AddAWSService<IAmazonS3, AmazonS3Config>((IServiceProvider sp, AWSOptions awsOptions) =>
    {
        var sqsOptions = (AmazonS3Config)awsOptions.DefaultClientConfig;

        sqsOptions.ForcePathStyle = true;
        sqsOptions.MaxErrorRetry = 10;

        return awsOptions;
    })

3.

Expose a method/constructor that creates an AWSOptions object from another. Note that this example also updates AWSOptions to take a generic for ease of use, but that's not strictly necessary as long as there's some means of creating one outside of the IConfiguration extension method. Note that this also exposes a new AddAWSService extension method that takes a Func<IServiceProvider, AWSOptions> arg.

builder.Services
    .AddDefaultAWSOptions(sp => sp.GetRequiredService<IConfiguration>().GetAWSOptions())
    .AddAWSService<IAmazonSQS>()
    .AddAWSService<IAmazonS3>(sp =>
    {
        var awsOptions = sp.GetRequiredService<AWSOptions>();
        var sqsOptions = new AWSOptions<AmazonS3Config>(awsOptions);

        sqsOptions.DefaultClientConfig.ForcePathStyle = true;
        sqsOptions.DefaultClientConfig.MaxErrorRetry = 10;

        return sqsOptions;
    })

Other Information

The most reasonable approach I could muster is to create an ugly ApplyPropsFrom extension method.

using Amazon.Extensions.NETCore.Setup;
using Amazon.S3;
using Amazon.SQS;

var builder = WebApplication.CreateBuilder(args);

builder.Services
    .AddDefaultAWSOptions(sp => sp.GetRequiredService<IConfiguration>().GetAWSOptions())
    .AddAWSService<IAmazonSQS>()
    .AddSingleton<IAmazonS3>(sp =>
    {
        var globalAwsOptions = sp.GetRequiredService<AWSOptions>();
        var config = sp.GetRequiredService<IConfiguration>();

        var s3Options = config.GetAWSOptions<AmazonS3Config>();

        s3Options.ApplyPropsFrom(globalAwsOptions);

        var s3Config = (AmazonS3Config)s3Options.DefaultClientConfig;
        s3Config.ForcePathStyle = true;
        s3Config.MaxErrorRetry = 10;
        // Set any other s3-specific config here

        return s3Options.CreateServiceClient<IAmazonS3>();
    });

var app = builder.Build();

app.Run();

using Amazon.Extensions.NETCore.Setup;
using Amazon.Runtime;

namespace MyApp;

public static class AWSOptionsExtensions
{
    public static void ApplyPropsFrom(this AWSOptions options, AWSOptions? other)
    {
        if (other == null)
        {
            return;
        }

        // Hope and pray that AWSOptions props never change...
        options.Credentials = other.Credentials;
        options.Region = other.Region;
        options.Logging = other.Logging;
        options.Profile = other.Profile;
        options.ExternalId = other.ExternalId;
        options.ProfilesLocation = other.ProfilesLocation;
        options.SessionName = other.SessionName;
        options.DefaultConfigurationMode = other.DefaultConfigurationMode;
        options.SessionRoleArn = other.SessionRoleArn;

        options.DefaultClientConfig?.ApplyPropsFrom(other.DefaultClientConfig);
    }

    public static void ApplyPropsFrom(this ClientConfig config, ClientConfig? other)
    {
        if (other == null)
        {
            return;
        }

        // Hope and pray that ClientConfig props never change...
        config.ServiceId = other.ServiceId;
        config.DefaultConfigurationMode = other.DefaultConfigurationMode;
        config.RegionEndpoint = other.RegionEndpoint;
        config.ThrottleRetries = other.ThrottleRetries;
        config.UseHttp = other.UseHttp;
        config.UseAlternateUserAgentHeader = other.UseAlternateUserAgentHeader;
        config.ServiceURL = other.ServiceURL;
        config.SignatureVersion = other.SignatureVersion;
        config.ClientAppId = other.ClientAppId;
        config.SignatureMethod = other.SignatureMethod;
        config.LogResponse = other.LogResponse;
        config.BufferSize = other.BufferSize;
        config.ProgressUpdateInterval = other.ProgressUpdateInterval;
        config.ResignRetries = other.ResignRetries;
        config.ProxyCredentials = other.ProxyCredentials;
        config.LogMetrics = other.LogMetrics;
        config.DisableLogging = other.DisableLogging;
        config.AllowAutoRedirect = other.AllowAutoRedirect;
        config.UseDualstackEndpoint = other.UseDualstackEndpoint;
        config.UseFIPSEndpoint = other.UseFIPSEndpoint;
        config.DisableRequestCompression = other.DisableRequestCompression;
        config.RequestMinCompressionSizeBytes = other.RequestMinCompressionSizeBytes;
        config.DisableHostPrefixInjection = other.DisableHostPrefixInjection;
        config.EndpointDiscoveryEnabled = other.EndpointDiscoveryEnabled;
        config.IgnoreConfiguredEndpointUrls = other.IgnoreConfiguredEndpointUrls;
        config.EndpointDiscoveryCacheLimit = other.EndpointDiscoveryCacheLimit;
        config.RetryMode = other.RetryMode;
        config.TelemetryProvider = other.TelemetryProvider;
        config.AccountIdEndpointMode = other.AccountIdEndpointMode;
        config.RequestChecksumCalculation = other.RequestChecksumCalculation;
        config.ResponseChecksumValidation = other.ResponseChecksumValidation;
        config.ProxyHost = other.ProxyHost;
        config.ProxyPort = other.ProxyPort;
        config.Profile = other.Profile;
        config.AWSTokenProvider = other.AWSTokenProvider;
        config.AuthenticationRegion = other.AuthenticationRegion;
        config.AuthenticationServiceName = other.AuthenticationServiceName;
        config.MaxErrorRetry = other.MaxErrorRetry;
        config.FastFailRequests = other.FastFailRequests;
        config.CacheHttpClient = other.CacheHttpClient;
        config.HttpClientCacheSize = other.HttpClientCacheSize;
        config.EndpointProvider = other.EndpointProvider;
        config.MaxConnectionsPerServer = other.MaxConnectionsPerServer;
        config.HttpClientFactory = other.HttpClientFactory;

        if (other.Timeout.HasValue)
        {
            config.Timeout = other.Timeout.Value;
        }
    }
}

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

AWS .NET SDK and/or Package version used

<PackageVersion Include="AWSSDK.Core" Version="3.7.402.19" />
<PackageVersion Include="AWSSDK.S3" Version="3.7.201.1" />
<PackageVersion Include="AWSSDK.SQS" Version="3.7.400.102" />
<PackageVersion Include="AWSSDK.SSO" Version="3.7.400.113" />
<PackageVersion Include="AWSSDK.SSOOIDC" Version="3.7.400.114" />
<PackageVersion Include="AWSSDK.SecurityToken" Version="3.7.202.11" />
<PackageVersion Include="AWSSDK.SimpleEmail" Version="3.7.200.29" />

Targeted .NET Platform

.NET 8

Operating System and version

macOS

[chase-miller]

Here's a possible implementation of part of what I propose. There are two big issues with this as-is:

1. AWSOptionsExtensions.ApplyPropsFrom() is hacky and brittle, and it doesn't have knowledge of what was set on the service-specific config AWSOptions vs the default AWSOptions. It would probably be best to update the core methods of the lib to facilitate the notion of creating AWSOptions (optionally from IConfiguration) using an existing AWSOptions object for unset properties.
2. It would be better imo to use the internal ClientFactory.CreateServiceClient() over AWSOptions.CreateServiceClient<TService>() for the first extension method below.

AdditionalServiceCollectionExtensions.cs:

using Amazon.Extensions.NETCore.Setup;
using Amazon.Runtime;
using Microsoft.Extensions.DependencyInjection;

namespace AWSSDK.Extensions.NETCore.Setup.ServiceConfig;

public static class AdditionalServiceCollectionExtensions
{
    // this should really be added to the lib's existing `ServiceCollectionExtensions.cs` class and reworked to use the internal `ClientFactory`
    public static IServiceCollection AddAWSService<TService>(
        this IServiceCollection services,
        Func<IServiceProvider, AWSOptions> optionsFunc,
        ServiceLifetime lifetime = ServiceLifetime.Singleton)
        where TService : IAmazonService
    {
        var descriptor = new ServiceDescriptor(
            typeof(TService),
            sp =>
            {
                var userOptions = optionsFunc(sp);
                return userOptions.CreateServiceClient<TService>();
            },
            lifetime);

        services.Add(descriptor);

        return services;
    }
}

ClientConfigServiceProviderExtensions.cs:

using Amazon.Extensions.NETCore.Setup;
using Amazon.Runtime;
using Microsoft.Extensions.DependencyInjection;

namespace AWSSDK.Extensions.NETCore.Setup.ServiceConfig;

public static class ClientConfigServiceProviderExtensions
{
    public static AWSOptions GetAWSOptions<TConfig>(this IServiceProvider sp)
        where TConfig : ClientConfig, new()
    {
        return sp.GetRequiredKeyedService<AWSOptions>(typeof(TConfig));
    }
}

ClientConfigServiceCollectionExtensions.cs:

using Amazon.Extensions.NETCore.Setup;
using Amazon.Runtime;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;

namespace AWSSDK.Extensions.NETCore.Setup.ServiceConfig;

public static class ClientConfigServiceCollectionExtensions
{
    public static IServiceCollection AddAWSOptions<TConfig>(
        this IServiceCollection services,
        Func<IServiceProvider, AWSOptions, TConfig, AWSOptions> optionsFunc = null,
        ServiceLifetime lifetime = ServiceLifetime.Singleton)
        where TConfig : ClientConfig, new()
    {
        var serviceDescriptor = new ServiceDescriptor(
            serviceType: typeof(AWSOptions),
            serviceKey: typeof(TConfig),
            (sp, _) => sp.GetOptionsViaFunc(optionsFunc),
            lifetime);

        services.Add(serviceDescriptor);

        return services;
    }

    public static IServiceCollection AddAWSService<TService, TConfig>(
        this IServiceCollection services,
        ServiceLifetime lifetime = ServiceLifetime.Singleton)
        where TService : IAmazonService
        where TConfig : ClientConfig, new()
    {
        return services.AddAWSService<TService>(
            sp =>
            {
                var options = sp.GetAWSOptions<TConfig>();
                return options;
            },
            lifetime);
    }

    public static IServiceCollection AddAWSService<TService, TConfig>(
        this IServiceCollection services,
        Func<IServiceProvider, AWSOptions, TConfig, AWSOptions> optionsFunc,
        ServiceLifetime lifetime = ServiceLifetime.Singleton)
        where TService : IAmazonService
        where TConfig : ClientConfig, new()
    {
        return services.AddAWSService<TService>(
            sp => sp.GetOptionsViaFunc(optionsFunc),
            lifetime);
    }

    public static IServiceCollection AddAWSService<TService, TConfig>(
        this IServiceCollection services,
        Action<IServiceProvider, TConfig> configModifier,
        ServiceLifetime lifetime = ServiceLifetime.Singleton)
        where TService : IAmazonService
        where TConfig : ClientConfig, new()
    {
        return services.AddAWSService<TService, TConfig>(
            (sp, options, config) =>
            {
                configModifier(sp, config);
                return options;
            },
            lifetime);
    }

    private static AWSOptions GetOptionsViaFunc<TConfig>(this IServiceProvider sp, Func<IServiceProvider, AWSOptions, TConfig, AWSOptions> optionsFunc)
        where TConfig : ClientConfig, new()
    {
        var configuration = sp.GetRequiredService<IConfiguration>();
        var defaultOptions = sp.GetService<AWSOptions>() ?? configuration.GetAWSOptions();

        var newOptions = configuration.GetAWSOptions<TConfig>();

        // this is really unfortunate, but I don't know of another way to recognize the globally-registered AWSOptions while creating service-specific config.
        // See https://github.com/aws/aws-sdk-net/issues/3754
        newOptions.ApplyPropsFrom(defaultOptions);

        var clientConfig = (TConfig)newOptions.DefaultClientConfig;

        return optionsFunc(sp, newOptions, clientConfig);
    }
}

AWSOptionsExtensions.cs:

using Amazon.Extensions.NETCore.Setup;
using Amazon.Runtime;

namespace AWSSDK.Extensions.NETCore.Setup.ServiceConfig;

public static class AWSOptionsExtensions
{
    /// <remarks>
    ///  https://github.com/aws/aws-sdk-net/issues/3754
    /// </remarks>
    public static void ApplyPropsFrom(this AWSOptions options, AWSOptions? other)
    {
        if (other == null)
        {
            return;
        }

        // Hope and pray that AWSOptions props never change...
        options.Credentials ??= other.Credentials;
        options.Region ??= other.Region;
        options.Logging ??= other.Logging;
        options.Profile ??= other.Profile;
        options.ExternalId ??= other.ExternalId;
        options.ProfilesLocation ??= other.ProfilesLocation;
        options.SessionName ??= other.SessionName;
        options.DefaultConfigurationMode ??= other.DefaultConfigurationMode;
        options.SessionRoleArn ??= other.SessionRoleArn;

        options.DefaultClientConfig?.ApplyPropsFrom(other.DefaultClientConfig);
    }

    /// <remarks>
    ///  https://github.com/aws/aws-sdk-net/issues/3754
    /// </remarks>
    public static void ApplyPropsFrom(this ClientConfig config, ClientConfig? other)
    {
        if (other == null)
        {
            return;
        }

        // Hope and pray that ClientConfig props never change...
        config.ServiceId ??= other.ServiceId;
        config.DefaultConfigurationMode = other.DefaultConfigurationMode;
        config.RegionEndpoint ??= other.RegionEndpoint;
        config.ThrottleRetries = other.ThrottleRetries;
        config.UseHttp = other.UseHttp;
        config.UseAlternateUserAgentHeader = other.UseAlternateUserAgentHeader;
        config.ServiceURL ??= other.ServiceURL;
        config.SignatureVersion ??= other.SignatureVersion;
        config.ClientAppId ??= other.ClientAppId;
        config.SignatureMethod = other.SignatureMethod;
        config.LogResponse = other.LogResponse;
        config.BufferSize = other.BufferSize;
        config.ProgressUpdateInterval = other.ProgressUpdateInterval;
        config.ResignRetries = other.ResignRetries;
        config.ProxyCredentials ??= other.ProxyCredentials;
        config.LogMetrics = other.LogMetrics;
        config.DisableLogging = other.DisableLogging;
        config.AllowAutoRedirect = other.AllowAutoRedirect;
        config.UseDualstackEndpoint = other.UseDualstackEndpoint;
        config.UseFIPSEndpoint = other.UseFIPSEndpoint;
        config.DisableRequestCompression = other.DisableRequestCompression;
        config.RequestMinCompressionSizeBytes = other.RequestMinCompressionSizeBytes;
        config.DisableHostPrefixInjection = other.DisableHostPrefixInjection;
        config.EndpointDiscoveryEnabled = other.EndpointDiscoveryEnabled;
        config.IgnoreConfiguredEndpointUrls = other.IgnoreConfiguredEndpointUrls;
        config.EndpointDiscoveryCacheLimit = other.EndpointDiscoveryCacheLimit;
        config.RetryMode = other.RetryMode;
        config.TelemetryProvider ??= other.TelemetryProvider;
        config.AccountIdEndpointMode = other.AccountIdEndpointMode;
        config.RequestChecksumCalculation = other.RequestChecksumCalculation;
        config.ResponseChecksumValidation = other.ResponseChecksumValidation;
        config.ProxyHost ??= other.ProxyHost;
        config.ProxyPort = other.ProxyPort;
        config.Profile ??= other.Profile;
        config.AWSTokenProvider ??= other.AWSTokenProvider;
        config.AuthenticationRegion ??= other.AuthenticationRegion;
        config.AuthenticationServiceName ??= other.AuthenticationServiceName;
        config.MaxErrorRetry = other.MaxErrorRetry;
        config.FastFailRequests = other.FastFailRequests;
        config.CacheHttpClient = other.CacheHttpClient;
        config.HttpClientCacheSize = other.HttpClientCacheSize;
        config.EndpointProvider ??= other.EndpointProvider;
        config.MaxConnectionsPerServer ??= other.MaxConnectionsPerServer;
        config.HttpClientFactory ??= other.HttpClientFactory;

        if (other.Timeout.HasValue && !config.Timeout.HasValue)
        {
            config.Timeout = other.Timeout.Value;
        }
    }
}

[normj]

Hi @chase-miller. Just giving you quick feedback before I go on vacation for a week, possibly somebody else on the team will chime in while I'm gone.

I haven't thought through the implementation but a version of your option 1 is my preferred user experience where a user can provide an Action<TServiceConfig> and we will call that action with the config we have constructed based on the values from default config. The user can do any last minute modifications/overrides to the service config before we create the service client with it.

[chase-miller]

@normj thanks for the feedback and have a great vacation!