Replace SQS singleton with per-pipeline client cache with TTL eviction - #291
Replace SQS singleton with per-pipeline client cache with TTL eviction#291gshikhar2021 wants to merge 2 commits into
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request refactors the SQS client management strategy to improve scalability and resource management. By moving from a single global client to a per-pipeline cache, the system can now effectively handle multiple concurrent pipelines. The addition of TTL-based eviction ensures that inactive clients are cleaned up, preventing memory leaks and stale connections. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request replaces the single global SQS client with a multi-tenant, TTL-evicted client cache (SQSClientCache) to support multiple pipelines. The review feedback highlights critical issues that need to be addressed: using only the resource name as a cache key can lead to cross-namespace cache collisions, so namespaced names should be used instead. Additionally, the cache currently lacks support for credential rotation because it does not compare incoming AWS configurations with cached ones, and the cleanup routine is susceptible to a panic if initialized with an invalid TTL.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| if apierrors.IsNotFound(err) { | ||
| awsclienthandler.DeleteSQSClient(req.Name) | ||
| return ctrl.Result{}, nil | ||
| } |
There was a problem hiding this comment.
Using only req.Name as the cache key can lead to cache collisions if multiple SourceCrawler resources with the same name exist in different namespaces. Since this operator can watch multiple namespaces, we should use the namespaced name (e.g., req.NamespacedName.String()) as the cache key to prevent cross-namespace client leakage.
| if apierrors.IsNotFound(err) { | |
| awsclienthandler.DeleteSQSClient(req.Name) | |
| return ctrl.Result{}, nil | |
| } | |
| if apierrors.IsNotFound(err) { | |
| awsclienthandler.DeleteSQSClient(req.NamespacedName.String()) | |
| return ctrl.Result{}, nil | |
| } |
| if _, err := awsclienthandler.NewSQSClientFromConfig(ctx, sourceAWSConfig, sourceCrawlerCR.Name); err != nil { | ||
| return ctrl.Result{}, r.handleError(ctx, sourceCrawlerCR, fmt.Errorf("failed to create SQS client: %w", err)) | ||
| } | ||
| return handleSQSWakeUp(ctx, sqsQueueURL, sourceCrawlerConfig.S3Config.Bucket, sourceCrawlerConfig.S3Config.Prefix), nil | ||
| return handleSQSWakeUp(ctx, sqsQueueURL, sourceCrawlerConfig.S3Config.Bucket, sourceCrawlerConfig.S3Config.Prefix, sourceCrawlerCR.Name), nil |
There was a problem hiding this comment.
Using sourceCrawlerCR.Name as the cache key can cause cache collisions across different namespaces. We should use the namespaced name string of the resource to ensure uniqueness.
namespacedName := types.NamespacedName{Namespace: sourceCrawlerCR.Namespace, Name: sourceCrawlerCR.Name}.String()
if _, err := awsclienthandler.NewSQSClientFromConfig(ctx, sourceAWSConfig, namespacedName); err != nil {
return ctrl.Result{}, r.handleError(ctx, sourceCrawlerCR, fmt.Errorf("failed to create SQS client: %w", err))
}
return handleSQSWakeUp(ctx, sqsQueueURL, sourceCrawlerConfig.S3Config.Bucket, sourceCrawlerConfig.S3Config.Prefix, namespacedName), nil| type sqsEntry struct { | ||
| client *sqs.Client | ||
| lastUsed time.Time | ||
| } |
There was a problem hiding this comment.
To support credential rotation and updates, we should store the AWSConfig in the sqsEntry so that we can detect when the configuration has changed and recreate the client accordingly.
| type sqsEntry struct { | |
| client *sqs.Client | |
| lastUsed time.Time | |
| } | |
| type sqsEntry struct { | |
| client *sqs.Client | |
| awsConfig AWSConfig | |
| lastUsed time.Time | |
| } |
| entry, ok := sqsClientCache.GetClient(pipelineName) | ||
| if ok { | ||
| logger.Info("Using existing SQS client for the pipeline", "pipelineName", pipelineName) | ||
| return entry.client, nil | ||
| } |
There was a problem hiding this comment.
If the AWS credentials are rotated or updated in the Secret, the controller will continue using the cached client with the old/stale credentials because it ignores the incoming awsConfig when a cached client exists. We should compare the incoming configuration with the cached one and recreate the client if they differ.
| entry, ok := sqsClientCache.GetClient(pipelineName) | |
| if ok { | |
| logger.Info("Using existing SQS client for the pipeline", "pipelineName", pipelineName) | |
| return entry.client, nil | |
| } | |
| entry, ok := sqsClientCache.GetClient(pipelineName) | |
| if ok && awsConfig != nil && entry.awsConfig == *awsConfig { | |
| logger.Info("Using existing SQS client for the pipeline", "pipelineName", pipelineName) | |
| return entry.client, nil | |
| } |
| newEntry := &sqsEntry{client: sqsClient, lastUsed: time.Now()} | ||
| sqsClientCache.SetClient(pipelineName, newEntry) |
There was a problem hiding this comment.
Store the AWSConfig in the new cache entry to enable configuration comparison on subsequent calls.
| newEntry := &sqsEntry{client: sqsClient, lastUsed: time.Now()} | |
| sqsClientCache.SetClient(pipelineName, newEntry) | |
| var cfgVal AWSConfig | |
| if awsConfig != nil { | |
| cfgVal = *awsConfig | |
| } | |
| newEntry := &sqsEntry{ | |
| client: sqsClient, | |
| awsConfig: cfgVal, | |
| lastUsed: time.Now(), | |
| } | |
| sqsClientCache.SetClient(pipelineName, newEntry) |
| func (c *SQSClientCache) startCleanup(ctx context.Context, ttl time.Duration) { | ||
| ticker := time.NewTicker(ttl / 2) |
There was a problem hiding this comment.
If ttl is less than or equal to 0, time.NewTicker(ttl / 2) will panic. We should add a defensive check to prevent panics if an invalid TTL is passed.
func (c *SQSClientCache) startCleanup(ctx context.Context, ttl time.Duration) {
if ttl <= 0 {
return
}
ticker := time.NewTicker(ttl / 2)
db88168 to
05bc52d
Compare
05bc52d to
fd51748
Compare
1 participant
No description provided.