Request has expired errors when EC2 credential uses IAM role - StaticCredentialsProvider freezes STS session tokens

[elkh510]

Problem

When the EC2 cloud is configured with a credentialsId pointing to a Jenkins AWS credential that internally uses iamRoleArn (role-based credential), provisioning agents periodically fails with Request has expired every ~1 hour:

Exception during provisioning
  SdkClientException: Request attempt 1 failure: Request has expired.
  ...
  Ec2Exception: Request has expired. (Service: Ec2, Status Code: 400) (SDK Attempt Count: 17)

Reconnecting to EC2 due to RequestExpired or ExpiredToken error

After reconnectToEc2() the very next call to describeImages succeeds immediately.

Root Cause

When credentialsId is set but roleArn is not set at the EC2 cloud level, createCredentialsProvider takes the 2-parameter path:

ec2-plugin/src/main/java/hudson/plugins/ec2/EC2Cloud.java

Line 1084 in 127f0f1

} finally {

AmazonWebServicesCredentials credentials = getCredentials(credentialsId);
if (credentials != null) {
    return StaticCredentialsProvider.create(credentials.resolveCredentials());
}

credentials.resolveCredentials() on a role-based Jenkins credential performs STS AssumeRole once and returns temporary AwsSessionCredentials. These are then frozen inside StaticCredentialsProvider - no auto-refresh.

After the STS session expires (default 1 hour), all subsequent describeImages calls fail. The SDK retries 17 times (due to hardcoded numRetries(16)) before throwing, then reconnectToEc2() is called which creates a fresh Ec2Client with new credentials - hence working immediately after reconnect.

The 5-parameter path with roleArn correctly uses StsAssumeRoleCredentialsProvider which auto-refreshes:

ec2-plugin/src/main/java/hudson/plugins/ec2/EC2Cloud.java

Line 1095 in 127f0f1

if (jenkinsInstance.isQuietingDown()) {

Fix / Workaround

Configure roleArn at the EC2 cloud level (not inside the Jenkins credential) and use instance profile or static keys as credentialsId. This routes through StsAssumeRoleCredentialsProvider with automatic credential refresh:

- amazonEC2:
    useInstanceProfileForCredentials: true
    roleArn: "arn:aws:iam::ACCOUNT:role/ROLE_NAME"
    roleSessionName: "Jenkins"

Expected Behavior

If credentialsId resolves to a credential containing an IAM role ARN, the resulting STS session credentials should be wrapped in an auto-refreshing provider rather than StaticCredentialsProvider.

Related Issues

• [JENKINS-71554] EC2ConnectionUpdater is not resestablishing connection #1901 (JENKINS-71554) - describes similar RequestExpired symptoms but focuses on the keepalive path; the root cause reported there overlaps with this issue
• PR JENKINS-71554: Reconnect on "RequestExpired" exception #886 - Added reconnectToEc2() on RequestExpired in EC2ConnectionUpdater (partial fix, doesn't address StaticCredentialsProvider expiry)
• PR Reconnect to EC2 also when error code is ExpiredToken #1008 - Added ExpiredToken handling alongside RequestExpired in provision catch block
• PR Pick fresh profile credentials #1120 - Fixed DefaultCredentialsProvider being cached as static instance; similar spirit to this issue but different code path