Expose RPC to promote a replica as primary #508
Description
If a primary dies, let's make it possible for an external supervisor to promote a replica as a primary. To avoid the problem of multiple primaries, let's make the primary ask for a lease from an external supervisor (that can, for example, use Consul).
When a replica is promoted as a primary, it needs to be up-to-date; otherwise some writes to the previous primary may be lost. So when we promote a replica, the first thing it should do is sync up from the durable store. The recovery procedure should be the same as if a primary lost its volume and had to restore from backup.
Please note that the RPC to promote replicas should be used as a last resort option when it's impossible to recover a primary with regular means because there's is inherently the possiblity of data loss.