Use_case_Topic_API

What?

Providing a Kubernetes API for Kafka topics is an attractive means of enabling gitops/infra-as-code for Kafka. But it's difficult because the Kafka Admin client provides a way for Kafka applications to create topics behind the back of the infra layer. This can result in different metadata state within the Kafka cluster than the Kubernetes resource state would imply. For example, a topic created via the Admin wouldn't have an associated topic resource in the kube control plane. This Admin API is used by applications in the Kafka ecosystem (e.g. Kafka Connect, Kafka Streams, arbitrary user applications).

A proxy could provide an alternative to mechanisms like the bi-directional Strimzi Topic Operator.

How?

All client interaction happens through the Kafka proxy which interacts with an API for resources that will serve as a single source of truth (Kube api server with topics CRs in the Kubernetes case). A privileged mechanism (e.g. kube operator) unidirectionally syncs the source of truth obtained from this API with the Kafka cluster itself.

In diagram form

Kafka Client ───────────────────→ Proxy
             KRPC (CreateTopics)    │
                                    │ HTTP PUT
                                    ↓
Kube Client ───────────────────→ Kube apiserver  <----> etcd
                                    |
                                    | WATCH Create
                                    ↓
                                Operator ────────────────→ Kafka Broker
                                        KRPC (CreateTopics)

Challenges

• Kube and Kafka still have different authz/permissioning models
• Kube and Kafka have different rules about what constitutes legal names for their resources. This requires some policy for dealing with names that are legal in one but not the other.
• Topics created internally within the Kafka cluster a problematic:
  • True internal topics like __consumer_offsets and __transaction_state
  • Topics created via side-effect of receiving Metadata could also be problematic, i.e. Although proxy can intercept the Metadata request, it would also need to know whether the topic existed within the Kafka cluster already. Or just make a CreateTopics request for every Metadata that permitted topic creation.