[BUG] Missing secretsmanager:GetSecretValue permission unclear when using DD_API_KEY_SECRET_ARN with the AWS Lambda Extension

[blimmer]

Agent Environment

Latest extension layer version 42 (arn:aws:lambda:us-west-2:464622532012:layer:Datadog-Extension:42): https://github.com/DataDog/datadog-lambda-extension/releases/tag/v42

Describe what happened:

I am using the DD_API_KEY_SECRET_ARN environment variable for my lambda function to secure our Datadog API key. However, I forgot to grant the lambda secretsmanager:GetSecretValue on that secret (see also DataDog/datadog-cdk-constructs#174).

However, the logs from the extension make it challenging to figure out what's wrong:

2023-05-03 19:24:03 UTC | DD_EXTENSION | ERROR | No API key configured
2023-05-03 19:24:03 UTC | DD_EXTENSION | ERROR | Unable to load trace agent config: you must specify an API Key, either via a configuration file or the DD_API_KEY env var
TELEMETRY	Name: datadog-agent	State: Subscribed	Types: [Platform, Function, Extension]
2023-05-03 19:24:03 UTC | DD_EXTENSION | ERROR | Unexpected nil instance of the trace-agent
[ERROR] [1683141852646] LAMBDA_RUNTIME Failed to get next invocation. No Response from endpoint
2023-05-03 19:24:17 UTC | DD_EXTENSION | ERROR | No API key configured
2023-05-03 19:24:18 UTC | DD_EXTENSION | ERROR | Unable to load trace agent config: you must specify an API Key, either via a configuration file or the DD_API_KEY env var
TELEMETRY	Name: datadog-agent	State: Already subscribed	Types: [Platform, Function, Extension]
2023-05-03 19:24:18 UTC | DD_EXTENSION | ERROR | Unexpected nil instance of the trace-agent
EXTENSION	Name: datadog-agent	State: Registered	Events: [SHUTDOWN, INVOKE]
START RequestId: a5a8fd29-7f6a-4203-b4eb-8ee72bb79afe Version: $LATEST
2023-05-03T19:24:24.799Z a5a8fd29-7f6a-4203-b4eb-8ee72bb79afe Task timed out after 12.01 seconds

END RequestId: a5a8fd29-7f6a-4203-b4eb-8ee72bb79afe
REPORT RequestId: a5a8fd29-7f6a-4203-b4eb-8ee72bb79afe	Duration: 12006.28 ms	Billed Duration: 10000 ms	Memory Size: 128 MB	Max Memory Used: 99 MB	
XRAY TraceId: 1-6452b4d2-4147d071097687c7737c48e3	SegmentId: 15906fc3250ce1cf	Sampled: true	

Previous versions (I tried layer version 30) expose the actual error. This is how I figured out what the problem was.

2023-05-03 19:31:00 UTC | DD_EXTENSION | ERROR | Error while trying to read an API Key from Secrets Manager: Secrets Manager read error: AccessDeniedException: User: arn:aws:sts::<account>:assumed-role/ExampleHandler-ExampleHandlerFunctionServiceRole01-12PEF15E2FPCU/ExampleHandler-dev is not authorized to perform: secretsmanager:GetSecretValue on resource: arn:aws:secretsmanager:us-west-2:<account>:secret:DATADOG_API_KEY-ABC123 because no identity-based policy allows the secretsmanager:GetSecretValue action
status code: 400, request id: 51476c7a-afa6-4346-9896-5ae6de8ebc36
2023-05-03 19:31:00 UTC | DD_EXTENSION | ERROR | No API key configured, exiting
2023-05-03 19:31:00 UTC | DD_EXTENSION | ERROR | Unable to load trace agent config: you must specify an API Key, either via a configuration file or the DD_API_KEY env var
LOGS	Name: datadog-agent	State: Subscribed	Types: [Platform, Function, Extension]
EXTENSION	Name: datadog-agent	State: Ready	Events: [INVOKE, SHUTDOWN]
2023-05-03 19:31:04 UTC | DD_EXTENSION | ERROR | Error while trying to read an API Key from Secrets Manager: Secrets Manager read error: AccessDeniedException: User: arn:aws:sts::<account>:assumed-role/ExampleHandler-ExampleHandlerFunctionServiceRole01-12PEF15E2FPCU/ExampleHandler-dev is not authorized to perform: secretsmanager:GetSecretValue on resource: arn:aws:secretsmanager:us-west-2:<account>:secret:DATADOG_API_KEY-ABC123 because no identity-based policy allows the secretsmanager:GetSecretValue action
status code: 400, request id: 73da0eca-dcb4-415b-997f-da744bc9fd09
2023-05-03 19:31:04 UTC | DD_EXTENSION | ERROR | No API key configured, exiting
LOGS	Name: datadog-agent	State: Already subscribed	Types: [Platform, Function, Extension]
2023-05-03 19:31:04 UTC | DD_EXTENSION | ERROR | Unable to load trace agent config: you must specify an API Key, either via a configuration file or the DD_API_KEY env var
EXTENSION	Name: datadog-agent	State: Registered	Events: [INVOKE, SHUTDOWN]
START RequestId: df0615c8-e2d8-47e6-ac45-dcf0b03fe944 Version: $LATEST
RequestId: df0615c8-e2d8-47e6-ac45-dcf0b03fe944 Error: exit status 255
Extension.Crash
END RequestId: df0615c8-e2d8-47e6-ac45-dcf0b03fe944
REPORT RequestId: df0615c8-e2d8-47e6-ac45-dcf0b03fe944	Duration: 4656.33 ms	Billed Duration: 4657 ms	Memory Size: 128 MB	Max Memory Used: 58 MB	
XRAY TraceId: 1-6452b673-3de0018c3f97312a2557c9ef	SegmentId: 5d66a08e391f75eb	Sampled: true	

Describe what you expected:

I expected the SecretsManager failure to be exposed in the logs so I could more easily recover from the issue.

Steps to reproduce the issue:

1. Store your Datadog API key in secrets manager. Copy the ARN of the secret.
2. Deploy an AWS Lambda function that's instrumented using the Lambda Extension. I'm using AWS CDK, but that's not important. You can set up the Lambda function + extension using any of the methods described here: https://docs.datadoghq.com/serverless/installation/nodejs/?tab=awscdk
3. Set the DD_API_KEY_SECRET_ARN to the ARN from step 1.
4. DO NOT grant secretsmanager:GetSecretValue to the Lambda function.
5. Trigger the Lambda function from the AWS console.

The Lambda function will time out. In the logs (see example above), you'll see no indication of the missing GetSecretValue permission.

Additional environment details (Operating System, Cloud provider, etc):

Lambda, Node 18

[lonedone]

The same issue is still here on the 49 version layer

[MohamedAmineB]

still here in v63 :(