Ability to re-connect KubeCluster to a running dask-kubernetes cluster

[hhuuggoo]

So, at Saturn Cloud, we actually deploy KubeCluster in a k8s deployment (and we wrap it with an http api). I don't think our HTTP api would be useful for anyone else, but since we run it as a deployment, we've had to build in logic to be able to reconnect KubeCluster with an existing cluster.

This code will check for the existence of the scheduler pod, and worker pods, and then talk to the scheduler to match up worker pods with worker names, and re-create the in-memory state of the KubeCluster

Would this be useful for anyone if we upstream it?

[jacobtomlinson]

Thanks for raising this @hhuuggoo. It would be great to see this upstreamed.

I've been working on #318 which does exactly this, but I was intending on starting again once dask/distributed#5033 as I think that will be cleaner.

One of the challenges of doing this is managing the state held by the KubeCluster object, things like the worker_options which are needed to create new workers. How have you solved this at Saturn?

[hhuuggoo]

oh ic - we don't solve that problem. We're doing this.

                pod = worker_pods.pop(host, None)
                if pod:
                    worker = Worker(
                        scheduler_address,
                        name=name,
                        pod_template=self.pod_template,
                        **common_options,
                    )
                    worker._pod = pod
                    worker.status = Status.running
                    self.workers[name] = worker

So since we control it - we know that worker_options are not being passed in.

[hhuuggoo]

One of the challenges of doing this is managing the state held by the KubeCluster object, things like the worker_options which are needed to create new workers. How have you solved this at Saturn?

I suppose we could store these parameters in k8s directly as a configmap or a secret? I'm not sure how flexible worker_options are but in the worst case we could pickle and store it as a secret. I know pickles would not be ideal but ....

[jacobtomlinson]

Ah gotcha! I didn't know if you were using a ConfigMap or Secret or something.

Once dask/distributed#5033 is merged that state will be able to be handed off to the scheduler, then retrieved again when a new Cluster object connects. This should solve the problem, it just needs finishing off.

[jacobtomlinson]

I'm going to close this out as a duplicate of #185 so we can track things in one place.