Skip to content

feat(client): retry PermissionDenied on established connections (opt-in) - #1466

Open
sachinsharma3191 wants to merge 1 commit into
temporalio:mainfrom
sachinsharma3191:feat/1448-retry-permission-denied-established
Open

feat(client): retry PermissionDenied on established connections (opt-in)#1466
sachinsharma3191 wants to merge 1 commit into
temporalio:mainfrom
sachinsharma3191:feat/1448-retry-permission-denied-established

Conversation

@sachinsharma3191

Copy link
Copy Markdown

Summary

  • Adds retry_permission_denied_when_established flag to RetryOptions (default: false), so existing behavior is unchanged.
  • Tracks "established" state per-connection via an Arc<AtomicBool> on ConnectionInner, set to true after the first successful RPC response.
  • In TonicErrorHandler::handle(), PermissionDenied is retried only when both the opt-in flag is enabled and the connection is established — this handles transient auth failures (token rotation races, IAM propagation delays) without retrying genuinely unauthorized initial connections.
  • Scoped to PermissionDenied only; Unauthenticated is left to credential refresh flows.

Open design questions (for reviewer)

  1. Config location: Flag is on RetryOptions (flows through ConnectionOptions.retry_options). Alternative: a dedicated field on ConnectionOptions. Current placement keeps retry concerns together.
  2. "Established" lifetime: The flag is sticky per ConnectionInner (never resets on reconnect). This is intentional — if creds were valid once, a PermissionDenied on a reconnected transport is likely transient. If reset-on-reconnect is desired, we'd need a callback from the transport layer.

Addresses #1448

Test plan

  • permission_denied_not_retried_by_default — default behavior unchanged
  • permission_denied_not_retried_when_not_established — opt-in enabled but no successful RPC yet → not retried
  • permission_denied_retried_when_established_and_opted_in — opt-in + established → retried
  • All 158 temporalio-client lib tests pass
  • All 389 temporalio-sdk-core lib tests pass
  • Full workspace cargo build succeeds

Add `retry_permission_denied_when_established` flag to `RetryOptions`
(default: false). When enabled, `PermissionDenied` errors are retried
only after at least one successful RPC has completed on the connection,
proving that credentials were valid at some point.

This handles transient auth failures on established connections (e.g.
token rotation races, brief IAM propagation delays) without retrying
genuinely unauthorized initial connections.

The "established" state is tracked per-connection via an
`Arc<AtomicBool>` set on first successful RPC response.

Addresses temporalio#1448
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant