[Feature] add `ttlSeconds` field to `RayClusterSpec`

[danielgafni]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

It would be nice it there was a ttlSeconds field in RayClusterSpec that would allow the KubeRay controller to automatically cleanup old RayCluster instances.

Use case

I would like to have a staging-like environment where a RayCluster can be shared across multiple data processing steps - for faster startup time.

When a new step runs, it would check whether there is already a RayCluster available to run on. If there isn't, it would create one. However, it's channeling to determine whether a step should delete a RayCluster, because it's unclear whether there are going to be other steps targeting the same cluster launched soon.

Therefore, the only reliable cleanup mechanism can be implemented on the controller side (I mean, I could make a custom external schedule, but that's inferior UX), similarly how RayJobSpec has ttlSecondsAfterFinished.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[andrewsykim]

We've discussed this feature in the past and I think it makes sense, but can you share why you couldn't use RayJob? RayJob would be more efficiency since the cluster would be deleted immediately.

[danielgafni]

I don't want to delete the RayCluster, I want to reuse it across multiple steps to avoid unnecessary (in a dev/staging environment) waiting. I don't know when exactly would the next step be launched, so I just want to have some buffer time when the RayCluster is still available.

I would use a RayJob in a production environment where I want the steps to be isolated and reliable.

[andrewsykim]

This makes sense, are you open to adding a PR for this feature? I'm happy to review it.

[danielgafni]

I don't really know any Go but can try doing it with the help of my friend Claude

[kevin85421]

The behavior seems strange to me. In #4036, the RayCluster is deleted when the time reaches CreationTimestamp + Spec.TTLSeconds no matter how many jobs are running in the RayCluster. What happens if a Ray job is submitted to a RayCluster, and the cluster is deleted before the job finishes because it has expired?

Is terminating the RayCluster after it has been idle (with no running Ray jobs) for a period of time the ideal behavior for your use case? @owenowenisme is working on moving the queries of Ray dashboards into a background goroutine pool and then implementing idle termination: #2999 (comment).

[kevin85421]

cc @rueian @Future-Outlier PTAL and share your thoughts if you have any.

[Future-Outlier]

I agree that #2998 this is make more sense, since it will complex the logic in RayJob when a Ray job is submitted to a RayCluster, and the RayCluster is going to be deleted, but we also want to provide an option for users to configure that this RayCluster will wait for RayJob to finish.

I also super agree that "the RayCluster is deleted when the time reaches CreationTimestamp + Spec.TTLSeconds no matter how many jobs are running in the RayCluster"
This is super dangerous for most of users.

We should keep the controller's logic simple in summary.

[danielgafni]

For my use case it would be sufficient, since I'd also bump ttlSeconds with each job submission, ensuring there is enough cluster lifetime left to complete the job.

However, I agree that for most users this might not be desired.

The main goal I'm pursuing here is the reusal of (ephemeral) RayCluster across multiple jobs. So it doesn't make sense to wait for just one job to finish.

Could we consider tracking multiple jobs instead? Not sure how feasible that would be.

An alternative approach would be to allow RayJob to cleanup the cluster it targets with clusterSelector. Then we could create multiple RayJob instances running on the same cluster, and the controller could delete it once the last ttlSecondsAfteeFinished expires.

[kevin85421]

Thank you for the reply!

since I'd also bump ttlSeconds with each job submission, ensuring there is enough cluster lifetime left to complete the job.

This approach seems hacky to me.

The main goal I'm pursuing here is the reusal of (ephemeral) RayCluster across multiple jobs. So it doesn't make sense to wait for just one job to finish.
Could we consider tracking multiple jobs instead? Not sure how feasible that would be.

“Idle” means that there are no running Ray jobs in the cluster. It does not refer to just a single Ray job.

An alternative approach would be to allow RayJob to cleanup the cluster it targets with clusterSelector.

This is also not good behavior for me. In HTTPMode, the underlying assumption is that the cluster isn't owned by the CR, so we should not use the RayJob CR to manage the RayCluster. If we do, it feels like an abstraction leak to me.

[danielgafni]

“Idle” means that there are no running Ray jobs in the cluster. It does not refer to just a single Ray job.

Oh great, that's even better then! Sure, let's track the idle status rather than the cluster creation timestamp.

[andrewsykim]

Tracking idle status would require us to connect to the Ray cluster and perdiocally list jobs, which I thought we were trying to avoid? Not just jobs but also client sessions that may have created tasks / actors. Seems like a lot of complexity we would have to add in RayCluster controller.

I think allowing users to set their own TTL is a reasonable use-case separate from terminating idle clusters after a timeout. It's up to the user to ensure their jobs completed within the TTL and I think that's ok. I think this is also generally useful for platform teams that may want to timebox the life of Ray clusters without necessarily caring about what job runs there. If you care about job completion you should use RayJob anyways.

[andrewsykim]

I also super agree that "the RayCluster is deleted when the time reaches CreationTimestamp + Spec.TTLSeconds no matter >how many jobs are running in the RayCluster"
This is super dangerous for most of users.

We should keep the controller's logic simple in summary.

@Future-Outlier I think these two statements are countering each other. Deleting a cluster after a TTL is actually very simple, compared to trying to track idle clusters.

Can you also expand on why it's dangerous? The feature is not on by default, a user is opting into the TTL so they understand a cluster is deleted regardless of the timeout. If a users finds it "dangerous" they do not need to enable the feature.