-
Notifications
You must be signed in to change notification settings - Fork 8
Use_case_Topic_API
Providing a Kubernetes API for Kafka topics is an attractive means of enabling gitops/infra-as-code for Kafka.
But it's difficult because the Kafka Admin
client provides a way for Kafka applications to create topics behind the back of the infra layer.
This can result in different metadata state within the Kafka cluster than the Kubernetes resource state would imply.
For example, a topic created via the Admin wouldn't have an associated topic resource in the kube control plane.
This Admin
API is used by applications in the Kafka ecosystem (e.g. Kafka Connect, Kafka Streams, arbitrary user applications).
A proxy could provide an alternative to mechanisms like the bi-directional Strimzi Topic Operator.
All client interaction happens through the Kafka proxy which interacts with an API for resources that will serve as a single source of truth (Kube api server with topics CRs in the Kubernetes case). A privileged mechanism (e.g. kube operator) unidirectionally syncs the source of truth obtained from this API with the Kafka cluster itself.
In diagram form
Kafka Client ───────────────────→ Proxy
KRPC (CreateTopics) │
│ HTTP PUT
↓
Kube Client ───────────────────→ Kube apiserver <----> etcd
|
| WATCH Create
↓
Operator ────────────────→ Kafka Broker
KRPC (CreateTopics)
- Kube and Kafka still have different authz/permissioning models
- Kube and Kafka have different rules about what constitutes legal names for their resources. This requires some policy for dealing with names that are legal in one but not the other.
- Topics created internally within the Kafka cluster a problematic:
- True internal topics like
__consumer_offsets
and__transaction_state
- Topics created via side-effect of receiving
Metadata
could also be problematic, i.e. Although proxy can intercept theMetadata
request, it would also need to know whether the topic existed within the Kafka cluster already. Or just make aCreateTopics
request for everyMetadata
that permitted topic creation.
- True internal topics like