[draft] background goroutine get job info #4160
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: fscnick <[email protected]>
Signed-off-by: fscnick <[email protected]>
fscnick
commented
Oct 30, 2025
Comment on lines
+63
to
+65
| time.AfterFunc(queryInterval, func() { | ||
| w.taskQueue <- task | ||
| }) |
There was a problem hiding this comment.
This wouldn't block the current goroutine.
Comment on lines
+768
to
+770
| if err := rayDashboardClient.StopJob(ctx, rayJobInstance.Status.JobId); err != nil { | ||
| logger.Error(err, "Failed to stop job for RayJob") | ||
| } |
There was a problem hiding this comment.
Is it okay to call StopJob to remove the cache placeholder before deleting the RayCluster because the status of retry calls deleteClusterResources?
fscnick
commented
Oct 30, 2025
|
|
||
| for i := 0; i < workerSize; i++ { | ||
| // TODO: observability for these goroutine | ||
| // TODO: should we consider the stop ? |
There was a problem hiding this comment.
should we consider the stop? The goroutine fetches the JobInfo and keeps the copy of JobInfo in the memory not updating something. Once the program exits, all the copy are gone.
fscnick
commented
Oct 30, 2025
| ) | ||
|
|
||
| type RayDashboardClientInterface interface { | ||
| InitClient(client *http.Client, dashboardURL string) |
There was a problem hiding this comment.
Remove this method from the interface because the different implementation might have different input arguments.
| keys := cacheStorage.Keys() | ||
| expiredThreshold := time.Now().Add(-cacheExpiry) | ||
| for _, key := range keys { | ||
| if cached, ok := cacheStorage.Peek(key); ok { |
There was a problem hiding this comment.
Peek wouldn't update the recent-ness of cache.
Signed-off-by: fscnick <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
Currently this pr is a draft for demonstrating the design. If it is okay, we could continue to polish it.
The operator query the job info with block operation. It might impact the efficiency of the reconciliation.
In this PR, it introduces the background goroutine to fetch the
JobInfoand cache it.implementation design:
When the dashboard client is initializing, it will also initialize the singleton of worker pool and cache storage along with a cache cleanup goroutine.
When
GetJobInfois called, it returns the cache if hit. Or, it put a placeholder and add a task to the background goroutine to update the JobInfo periodically.The placeholder will be remove via calling
StopJob. it might happen on retrying or deleting the RayJob.Additionally, this pr takes the feedback in #4043 into account.
Related issue number
Closes #4087
Checks