fix: add a retry policy and async fetch for Connect's remote JWKS

[markgoddard]

We sometimes see API requests to the TLS API of Connect fail with:

Jwks remote fetch is failed

This has been traced to Envoy's JWT authenticator. Previously we
configured it without retry policy, leaving it vulnerable to
intermittent failures.

We also used on-demand JWKS fetch, rather than a background thread
(async fetch). This required each thread to individually cache the JWKS,
increasing the likeliness of being affected by IdP flakiness.

This change addresses both of these issues.

Fixes: cofide/cofide-connect#223

Docs: https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/http/jwt_authn/v3/config.proto#extensions-filters-http-jwt-authn-v3-remotejwks

[gemini-code-assist]

Summary of Changes

Hello @markgoddard, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the reliability and robustness of the JWT authentication process within Envoy. By introducing a retry mechanism and enabling asynchronous fetching of JSON Web Key Sets (JWKS), the changes aim to prevent authentication failures caused by intermittent network issues or Identity Provider flakiness, ensuring a more stable service operation.

Highlights

• Retry Policy for JWKS Fetch: Configured Envoy's JWT authenticator with a retry policy, including 5 retries on 5xx errors and resets, to mitigate intermittent failures during remote JWKS fetching.
• Asynchronous JWKS Fetch: Enabled asynchronous fetching of JWKS in a background thread, ensuring the listener waits for the fetch to complete, which improves resilience against IdP flakiness and reduces individual thread caching.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request aims to improve the reliability of fetching remote JWKS in Envoy by introducing a retry policy and enabling asynchronous fetching. These are excellent changes for enhancing resilience against intermittent failures from the identity provider. However, there's a critical configuration error in the YAML structure where retry_policy is misplaced. I've provided a specific comment with a code suggestion to correct the indentation. Once this is fixed, the change will be ready.

[alec-w]

Good find 👍