Inconsistent cleanup of client.shutdown

[Adam-D-Lewis]

Bug

I'm not sure if this is an issue with dask-kubernetes only or the distributed library, but I have a KubeCluster which I've connected to with a client. When I run client.shutdown(), the dask-scheduler pod shuts down, but the workers shut down inconsistently, at times all of them remain up.

Workarounds

I'd be interested in a work around if possible to ensure the workers shut down. I don't have direct access to the cluster object (unless it can somehow be recovered from the client object) or I'd run cluster.scale(0) first. Perhaps something using client.run? Or perhaps I could recreate the cluster object with KubeCluster.from_name(cluster_name)? It's not clear how I would find the cluster-name, though. Is that the name of the dask-scheduler pod?

[jacobtomlinson]

Thanks for raising this @Adam-D-Lewis.

Sadly client.shutdown() doesn't work with may cluster managers, including KubeCluster. This is because the scheduler itself is not in control of the cluster resources, only the KubeCluster object is.

Also I'm afraid KubeCluster doesn't support from_name right now. But this is the exact use case that dask-ctl is designed to handle, but from_name is part of the dask-ctl spec.

Once the scheduler closes the worker tasks should exit after a timeout.

But the main way you can control this today is to call cluster.close() on the KubeCluster object.

[Adam-D-Lewis]

Thanks @jacobtomlinson, that's super helpeful. Could you clarify a bit on the from_name method? As I understand it, from_name is listed in the KubeCluster API documentation, but it's not currently working, and there is a desire to have it work eventually through integration with dask-ctl, but it's not been implemented yet, is that right?

[jacobtomlinson]

The from_name method has not yet been implemented on KubeCluster, therefore it is not possible to use it with dask-ctl. There is a WIP PR #318 which attempts to add this, but we are currently evaluating a more substantial rewrite instead as KubeCluster has some design flaws, such as those covered in #197.

If you've found from_name in the docs you may have seen it on the HelmCluster reference which does indeed support dask-ctl.

[Adam-D-Lewis]

Ah, perhaps you're right. Thanks again for your helpful response. I'll close this issue since it's been answered. 👍