You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: 083-stretch-cluster.md
+12-10Lines changed: 12 additions & 10 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,21 +1,21 @@
1
1
# Stretch Kafka cluster
2
2
3
3
The Strimzi Kafka operator currently manages Kafka clusters within a single Kubernetes cluster.
4
-
This proposal aims to extend support to stretch Kafka clusters, where brokers and controllers of a single Kafka cluster are distributed across multiple Kubernetes clusters.
4
+
This proposal aims to extend this support to stretch Kafka clusters, where brokers and controllers of a single Kafka cluster are distributed across multiple Kubernetes clusters.
5
5
6
6
## Current situation
7
7
8
-
At present, the availability of Strimzi-managed Kafka clusters is directly tied to the availability of the underlying Kubernetes cluster.
8
+
At present, the availability of Strimzi-managed Kafka clusters is limited by the availability of the underlying Kubernetes cluster.
9
9
If a Kubernetes cluster experiences an outage, the entire Kafka cluster becomes unavailable, disrupting all connected Kafka clients.
10
10
11
11
## Motivation
12
12
13
-
A stretch Kafka cluster allows Kafka nodes to be distributed across multiple Kubernetes clusters, significantly enhancing resilience by enabling the system to tolerate the outage of an entire Kubernetes cluster without disrupting service to clients.
14
-
This configuration ensures high availability and seamless client operations, even in the event of cluster-specific failures.
13
+
A stretch Kafka cluster allows Kafka nodes to be distributed across multiple Kubernetes clusters. This approach also facilitates maby valuable use cases, such as:
15
14
16
-
In addition to improving fault tolerance, this approach also facilitates other valuable use cases, such as:
15
+
-**High Availability**: Distributing Kafka brokers across multiple Kubernetes cluster significantly enhances resilience by enabling the system to tolerate the outage of a Kubernetes cluster without disrupting service to clients.
16
+
17
+
-**Migration Flexibility**: The ability to move Kafka nodes between Kubernetes environments without downtime, supporting maintenance or migrations.
17
18
18
-
-**Migration Flexibility**: The ability to move Kafka clusters between Kubernetes environments without downtime, supporting maintenance or migrations.
19
19
-**Resource Optimization**: Efficiently utilizing resources across multiple clusters, which can be advantageous in environments with varying cluster capacities or during scaling operations.
20
20
21
21
### Limitations and Considerations
@@ -24,18 +24,20 @@ While a stretch Kafka cluster offers several advantages, it also introduces some
24
24
-**Increased Network Complexity and Costs**: The communication between brokers and controllers across clusters relies on network connectivity, which can be less reliable and more costly than intra-cluster communication.
25
25
This necessitates careful consideration of network architecture and associated costs.
26
26
27
-
-**Latency Requirements**: The stretch Kafka cluster is best suited for environments with low-latency network connections between the Kubernetes clusters.High latency can adversely affect the performance and synchronization of Kafka nodes, potentially leading to delays or errors in replication and client communication.
27
+
-**Latency Requirements**: Stretch Kafka clusters are best suited for environments with low-latency network connections between the Kubernetes clusters.
28
+
High latency can adversely affect the performance and synchronization of Kafka nodes, potentially leading to delays or errors in replication and client communication.
28
29
Defining the minimal acceptable latency between clusters is crucial to ensure optimal performance.
29
30
30
31
## Proposal
31
32
32
33
This proposal seeks to enhance the Strimzi Kafka operator to support stretch Kafka clusters, distributing brokers and controllers across multiple Kubernetes clusters.
33
-
The intent is to focus on high-availability of the data plane.The proposal outlines high-level topology and design concepts for such deployments, with a plan to incrementally include finer design and implementation details for various aspects.
34
+
The intent is to focus on high-availability of the data plane.
35
+
The proposal outlines high-level topology and design concepts for such deployments, with a plan to incrementally include finer design and implementation details for various aspects.
Ideally, an odd number of clusters (at least three) is needed to maintain quorum in the event of a cluster outage.
40
+
Ideally, an odd number of clusters (at least three) will be needed to maintain Kafka controller quorum and tolerate a Kubernetes cluster outage.
39
41
40
42
-**Low Latency**: Kafka clusters should be deployed in environments that allow low-latency communication between Kafka brokers and controllers.
41
43
Stretch Kafka clusters should be deployed in environments such as data centers or availability zones within a single region, and not across distant regions where high latency could impair performance.
@@ -45,7 +47,7 @@ While Zookeeper-based deployments are still supported, they are outside the scop
45
47
46
48
### Design
47
49
48
-
The cluster operator will be deployed in all Kubernetes clusters and will manage Kafka brokers/controllers running on that cluster.
50
+
The cluster operator will be deployed in all Kubernetes clusters and will manage Kafka brokers and controllers running on that cluster.
49
51
One Kubernetes cluster will act as the control point for defining custom resources (Kafka, KafkaNodePool) required for stretch Kafka cluster.
50
52
The KafkaNodePool custom resource will be extended to include information about a Kubernetes cluster where the pool should be deployed.
51
53
The cluster operator will create necessary resources (StrimziPodSets, services etc.) on the target clusters specified within the KafkaNodePool resource.
0 commit comments