[Design] Valkey Pod naming strategy

[jdheyburn]

Summary

Outlining a strategy for how Valkey Pods should be named.

Motivation

In the pre-0.1.0 release, Valkey pods are created with this naming (for 3 shards, 2 replicas):

valkeycluster-sample-blc5d-7c9d77c4cf-nxkfm
valkeycluster-sample-cgh4v-7c9d77c4cf-jkkpn
valkeycluster-sample-jpqb8-7c9d77c4cf-98chz
valkeycluster-sample-tlhgj-7c9d77c4cf-5pfsp
valkeycluster-sample-twstb-7c9d77c4cf-hvgjj
valkeycluster-sample-zjzqb-7c9d77c4cf-sx6dm

This doesn't provide much information on what shards the pods are part of, and what their desired replication role is (master / replica).

While the proposed ValkeyNode CRD abstracts this away, we would still like the pod name to be recognisable in its primitive form.

Detailed Design

We can include varying attributes to the pod names:

Desired role in name

# Format:
valkey-<NAME>-<SHARD#>-<DESIRED_ROLE>

# Examples
valkeyNode: valkey-cache-0-master
- pod_statefulset: valkey-cache-0-master-0
- pod_deployment: valkey-cache-0-master-REPLICA_SET_HASH-POD_HASH

valkeyNode: valkey-cache-0-replica-0
- pod_statefulset: valkey-cache-0-replica-0-0
- pod_deployment: valkey-cache-0-replica-0-REPLICA_SET_HASH-POD_HASH

StatefulSet's will always append on the pod ordinal index -0 to pods. Given ValkeyNode would abstract the pod name away, does it really matter?

Pros:

• We know which pod is desired to be the master
  • Applicable to zone pinning requirements
  • Easier glance at kubectl get pods

Cons:

• During failover, the actual role will differ from the role of the pod
  • Can be mitigated by making ValkeyNode the entry point: kubectl get valkeynode -l shard=SHARD_NAME

## Others

Please include other suggestions in the comments below

Open Questions

• Do we want to include the type of Valkey (Cluster / Standalone / Sentinel)? Maybe just for sentinel?

References

No response

Implementation

• I'm willing to implement this design and submit a PR
• This design has been discussed with maintainers

[judgeaxl]

Are replicas supposed to fall back to the original master node when it comes back up? If not, then a replica which has taken over a master will simply be named incorrectly until it fails again. Also, if it's not reverting back, then what would you name the new node taking over the failed master?

I've just spent some time understanding the CloudNative-PG operator for PostgreSQL and they put the current role of each pod in a label, which is easy to both query and view.

[jdheyburn]

This PR discussed this point a bit. Currently the route we're going down is:

<name>-<shard_index>-<pod_index>

Where the 0 pod_index always starts off as the primary. Labels will be used to dictate the pods current replication role. We may have in the future fixed labelling such that pod index 0 is labelled "expected/desired-roe=primary" and the others are labelled "replica".

e.g.

# expected primary for shard 0
mycluster-0-0
# replica for shard 0, but it can be failed over to
mycluster-0-1

# expected primary for shard 1
mycluster-1-0
# replicas for shard 1
mycluster-1-1
mycluster-1-2

How does that align with how the PG operator works?

[judgeaxl]

Cool, I think that makes a lot of sense, except I would not put too much weight on pod 0 being the primary beyond first startup other than just being the natural order of things.

CNPG uses its own pod controller, no deployments or stateful sets, so pods are just db_identifier-0...N and whichever is the primary at any given time is just found via labels, and probably some internal tracking.

PG is slightly differently than Valkey I think, since mostly PG deployments only have one primary and the rest are replicas, either stand-by or read-only, so it might not be a completely fair apples to apples comparison. I get the impression that sharded Valkey clusters are more common than sharded PG clusters, but I have no actual data to back that up.

One reason they don't use stateful sets is they don't want to have the last pod in the set end up being the master and then being the one going away when you scale down, since stateful sets only grow/shrink at the highest numbered pod. I actually think the design of stateful sets is a bit odd in that regard, unless you have a workload where all pods are completely equal or at the very least maintain their role throughout the lifetime of the app.

[judgeaxl]

And I see you do cover the movement of the primaries in the PR, so that's great! I'll read those to get the full overview of the discussions in the future!

[jdheyburn]

CNPG uses its own pod controller, no deployments or stateful sets, so pods are just db_identifier-0...N and whichever is the primary at any given time is just found via labels, and probably some internal tracking.

We considered having direct pod management (bypassing StatefulSet/Deployment). In the end we want to adopt the Unix philosophy and delegate control of pod management to existing controllers, then there isn't a dependency on the Valkey Operator to recreate pods - that's managed by the built-in controllers.

• [Design] Pod management design #14

The ValkeyNode CRD proposal abstracts away the concrete implementation of the pod manager (Deployment, StatefulSet). Later we could even provide a Direct option to have the ValkeyNode controller directly manage pods.

One reason they don't use stateful sets is they don't want to have the last pod in the set end up being the master and then being the one going away when you scale down, since stateful sets only grow/shrink at the highest numbered pod. I actually think the design of stateful sets is a bit odd in that regard, unless you have a workload where all pods are completely equal or at the very least maintain their role throughout the lifetime of the app.

Another design decision is to utilise Singleton Deployments/StatefulSets to allow us to get the most control of pod placement, then we don't run into that problem.