Open
Description
Describe the bug
We were on version 3.7.400 of AWSSDK.* packages (and later, for those packages that had further version bumps). When moving to 4.0.1, we started seeing sporadic error logs with Npgsql.PostgresException (0x80004005): 28000: PAM authentication failed for user. These errors would last somewhere between 1-30 minutes, 2-5 times/day. Reverting the SDK update solves the issue.
I've been through the migration doc but don't see what could cause this. We're looking for some insight. Before tweaking various values or making code changes trying to get the error to go away, we'd like to understand what changed.
Regression Issue
- Select this option if this issue appears to be a regression.
Expected Behavior
No PAM authentication failed errors
OR
A document explaining what changed and what specific values/settings are needed
Current Behavior
Npgsql.PostgresException (0x80004005): 28000: PAM authentication failed for user "app_user"
at Npgsql.Internal.NpgsqlConnector.ReadMessageLong(Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
at Npgsql.Internal.NpgsqlConnector.Authenticate(String username, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|214_1(NpgsqlConnector conn, SslMode sslMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
at Npgsql.NpgsqlDataSource.OpenConnectionAsync(CancellationToken cancellationToken)
at Npgsql.NpgsqlDataSource.OpenConnectionAsync(CancellationToken cancellationToken)
at Core.Persistence.Postgres.ConnectionFactory.GetConnection() in /app/Core/Persistence/Postgres/ConnectionFactory.cs:line 57
at Core.Catalog.ProductsRepository.Get(List`1 productIds) in /app/Core/Catalog/ProductsRepository.cs:line 125
at Domain.Products.GetProductsWorkflow.Execute(RequestWithCompanyId`1 request) in /app/Domain/Products/GetProductsWorkflow.cs:line 29
at Api.Controllers.SupplierController.GetProducts(GetProductsRequest request) in /app/Api/Controllers/SupplierController.cs:line 51
at lambda_method31(Closure, Object)
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
Exception data:
Severity: FATAL
SqlState: 28000
MessageText: PAM authentication failed for user "app_user"
File: auth.c
Line: 321
Routine: auth_failed
Reproduction Steps
Upgrade
<PackageReference Include="AWSSDK.Extensions.NETCore.Setup" Version="3.7.400" />
<PackageReference Include="AWSSDK.KeyManagementService" Version="3.7.400.137" />
<PackageReference Include="AWSSDK.RDS" Version="3.7.410.75" />
<PackageReference Include="AWSSDK.SecretsManager" Version="3.7.400.137" />
<PackageReference Include="AWSSDK.SQS" Version="3.7.400.137" />
to
<PackageReference Include="AWSSDK.Extensions.NETCore.Setup" Version="4.0.1" />
<PackageReference Include="AWSSDK.KeyManagementService" Version="4.0.0.4" />
<PackageReference Include="AWSSDK.RDS" Version="4.0.0.4" />
<PackageReference Include="AWSSDK.SecretsManager" Version="4.0.0.4" />
<PackageReference Include="AWSSDK.SQS" Version="4.0.0.4" />
Possible Solution
No response
Additional Information/Context
Configuration of NpgsqlDataSource:
public ConnectionFactory(IConfiguration configuration)
{
var dataSourceBuilder = new NpgsqlDataSourceBuilder(GetConnectionString(configuration));
dataSourceBuilder.UsePeriodicPasswordProvider(
(builder, cancellationToken) =>
new ValueTask<string>(
RDSAuthTokenGenerator.GenerateAuthTokenAsync(
FallbackCredentialsFactory.GetCredentials(),
FallbackRegionFactory.GetRegionEndpoint(),
builder.Host,
builder.Port,
builder.Username)),
TimeSpan.FromMinutes(13.5), // https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.Connecting.html
TimeSpan.FromSeconds(3));
DataSource = dataSourceBuilder.Build();
}Usage of NpgsqlDataSource:
public async Task<NpgsqlConnection> GetConnection()
{
return await DataSource.OpenConnectionAsync();
}After upgrading to v4, we got a warning that FallbackCredentialsFactory is obsolete. We tried both of the following for that line but the error was still present.
DefaultAWSCredentialsIdentityResolver.GetCredentials(),and
#pragma warning disable CS0618 // Type or member is obsolete
FallbackCredentialsFactory.GetCredentials(),
#pragma warning restore CS0618 // Type or member is obsoleteOtherwise, we had no Postgres or RDS changes with this version bump.
AWS .NET SDK and/or Package version used
<PackageReference Include="AWSSDK.Extensions.NETCore.Setup" Version="4.0.1" />
<PackageReference Include="AWSSDK.KeyManagementService" Version="4.0.0.4" />
<PackageReference Include="AWSSDK.RDS" Version="4.0.0.4" />
<PackageReference Include="AWSSDK.SecretsManager" Version="4.0.0.4" />
<PackageReference Include="AWSSDK.SQS" Version="4.0.0.4" />
Targeted .NET Platform
net8.0
Operating System and version
mcr.microsoft.com/dotnet/sdk:8.0-alpine