Add Kafka to TDP

[Nuttymoon]

Apache Kafka

Presentation of the project

Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

Kafka is the de facto standard for building scalable fault-tolerant messaging systems.

Why should it be included in TDP?

• As a big data Apache project, Kafka is very well integrated with many Hadoop projects that are part of TDP (Hive, Spark, Ranger, etc.)
• Kafka brings complementary functionalities that are needed to build full-fledged data lakes

Targeted version of the project

Kafka 3.2.x (latest version)

Note: Unfortunately Kafka Raft metadata mode (that allows running Kafka without ZooKeeper) is still not production-ready in this release. See KRaft (aka KIP-500) mode Preview Release.

Potential impacts on TDP

ZooKeeper

Kafka 3.x uses ZooKeeper version 3.6.3 (see dependencies.gradle) whereas the version packaged in TDP is 3.4.6.

Here are a few things that I have noted:

• Building Kafka 3.2 using ZK 3.4.6 as a dependency fails. Kafka 3.x uses new client APIs introduced in ZK 3.5.2 (e.g. ZKClientConfig)
• If we look at Confluent interoperability documentation we can see that ZK 3.4.6 can only be used with Kafka < 0.11, which is clearly not a viable option for TDP

This brings me to the following questions: Can we integrate another version of ZooKeeper into TDP? If yes, how will we modify tdp-collection's architecture?

[leopaul36]

I think we will all agree that Kafka fits well into TDP due to its integration with the other components.

As for the ZooKeeper dependency, we can try and see if our current K̶a̶f̶k̶a̶ ZooKeeper role is able to deploy a ZooKeeper in version 3.6.x. If not maybe we can do a ZooKeeper subtask inside the Kafka role like we did for Solr in the Ranger role ?

What do you think @gboutry @rpignolet @nschung ?

[Nuttymoon]

@leopaul36 did you mean

see if our current Kafka ZooKeeper role is able to deploy a ZooKeeper in version 3.6.x

[leopaul36]

Yes, my bad.

[rpignolet]

Doing ZooKeeper subtask inside Kafka involves having 2 ZooKeeper clusters running at the same time ?

[leopaul36]

Doing ZooKeeper subtask inside Kafka involves having 2 ZooKeeper clusters running at the same time ?

Yes, it is always a good practice to have a dedicated ZooKeeper cluster for Kafka anyway.

[Nuttymoon]

Ok I will try reusing the current ZooKeeper role and create a new independent service zookeeperkafka. It should work fine.

[nschung]

Kafka should be integrated as an option. It can be deployed in dedicated architecture. Therefore, Kafka can use its compatible ZK version IMO.
However, it's better to have the same ZK version of TDP and Kafka.

[rpignolet]

When reviewing PR related to Kafka, I think we didn't discuss enough about adding Kafka to TDP.

For me the goal, when adding a service/component to TDP, is to have a consistent ecosystem i.e. the new service/component should interact with other service/component inside TDP.

Another point is about the dedicated ZooKeeper, we should talk about it because, just to add Kafka we deploy another ZooKeeper cluster and this is really the good solution ?

I think that it should, at least, be moved to tdp-collection-extras and not in tdp-collection core if we absolutely want to have it.

[Edouard-R]

As suggested in Issue #340 of tdp-collection, starting with a Kafka 2.8.1 may be a good idea.
If we can't find a recent version of Kafka (2.X) compatible with ZK 3.4.6, maybe we should simply wait for the moment we'll fix an entire new set of versions for all our component.

[Nuttymoon]

As suggested by @rpignolet and @Edouard-R I am going to move Kafka and it's dedicated ZooKeeper to tdp-collection-extras for now. We can reconsider a graduation to tdp-collection later on.