Thanks, @in-han , I think the coordinator running too long does not affect the follower who became a leader, the follower should watch the leader key is expired. could you show more details logs?

Thanks, @in-han , I think the coordinator running too long does not affect the follower who became a leader, the follower should watch the leader key is expired. could you show more details logs?

Yes, follower can watch key expiration. But follower can campaign leader with tow conditions: a) pd leader key is expired; b) follower is etcd leader, in this case, the old pd leader was still the etcd leader.

This is the logs:

Why lease expired but the old PD still is the leader? does it re-election and the old pd became leader again?

BTW, 2000 TiKVs is the largest cluster size I have seen. amazing! here is an issue trace to improve the performance of the balance region #3744. If you have an interest, may you can help us.

Pd leader lease timeout because it encountered timeout when write key to embed ETCD. You are right, in this case the old pd became leader again.

Ah, after deployment, we also feel that this cluster is too large!

I think there maybe two ways to resolve this problem :
a) speed up coordinator stop. There is already a PR had optimized this problem ,but it wasn't enough. I have some ideas and will try to make a PR.
b) as you mentioned, optimize scheduler performance can also resolve this problem, I will have a look at the PR.

@nolouch Thanks for your reply

/remove-type bug
/type enhancement