Python implementation of Kafka Streams?

[dalejin2014]

We are interested in using kafka streaming.
Is it on the road map for confluent kafka python library?

[ewencp]

@dalejin2014 We'd love to have native stream processing libraries in different languages and having really good Kafka clients is the basis for that. That said, we don't have a timeline for adding this yet.

[miguno]

@dalejin2014: As @ewencp mentioned we don't have a timeline yet. The reason for this is that we first want to ensure we have a strong foundation in the form of the Java implementation of Kafka Streams before venturing into non-JVM languages.

That said, of course I took a note of your request. :-)

Do you mind sharing some information about your use case where you'd use Kafka Streams from Python?

[dalejin2014]

We are interested in developing a commenting feature kind of like google doc.
The use case is as follows:

• users in a thread should be notified on the events at a frequency of their choosing (realtime, hourly, daily, etc)

So we are thinking about using Kafka Streaming since it provides us:

• windowing
• group-by
• accumulation
• etc

Is there an easy way to port the features from Java client?

[miguno]

Thanks for sharing the background info @dalejin2014.

Is there an easy way to port the features from Java client?

It's not super-hard but also not trivial. Also, one would need to continuously maintain any such Kafka Streams libraries for other languages with the same commitment and high quality as the current Kafka Streams library for Java, so "porting" is not a one-off effort but an ongoing time investment. Hence our current decision to focus our efforts first on the Java implementation of Kafka Streams.

[jacqvdm]

+100 :)

[zzbennett]

Kafka Streams for Python would be so amazing. I'm currently evaluating stream processing frameworks and I like what I've been reading about Kafka Streams. My use case is essentially this: I'm laying down the infrastructure to enable realtime analytics and processing of log/event data. The primary users of this data are data scientists who would be standing up their own Kafka streams apps mostly for doing transformations, joins, partitioning and windowed analytics. I think Kafka streams fits this use case nicely since the streams library eliminates a lot of the boiler plate code involved in configuring Kafka consumers and producers but leaves developers the freedom and flexibility to do lots of cool stuff with the data in each Kafka topic. The only catch is that not many of the data scientists are well versed in Java--our language of choice is Python for almost everything. As much as I like Kafka and as excited as I am about Kafka Streams, getting the data scientists on board with writing Java will be an uphill battle.

With that said, have there been any developments with regards to supporting a Python based Kafka Streams library?

[miguno]

@zzbennett I hear you, Elizabeth. :-)

Unfortunately our short-term roadmap does not include work on a Python library of Kafka Streams. (We'd definitely welcome contributors though!) Same situation for e.g. kafka-python, a community project.

I'm kinda hesitant to suggest this, but perhaps it would be worth a try to experiment with Jython? IIRC some Ruby users have been experimenting with Kafka Streams' Java library via JRuby. FWIW, there are a few community/external projects already working on various "wrappers" (in a broad sense) for Kafka's Streams and Connect APIs, but they haven't been released yet; I don't remember off the top of my hat whether a Python-based one was amongst that.

[zzbennett]

Thanks for your reply @miguno and thanks for the suggestions. Jython might be a good option for prototyping. I may actually be able to drum up support for Scala based Streams apps, which would work a bit better with the Java libraries.

As far as contributing, I may even end up putting together a Python port of Kafka Streams for our uses cases. Eventually with the help of some collaborators in the kafka python community we'd hopefully be able to contribute something upstream. But I suppose we can cross that bridge when we get there. At any rate, thanks again for the help!

[murphyke]

@zzbennett Somebody in my group was talking about working on this also. If you create a repo with issues laying out the work and then solicit help, you may find yourself with some contributors reasonably soon.

[zzbennett]

@murphyke that would be super. I actually just created a repo last weekend to start working on it (https://github.com/python-kafka-streams/python-kafka-streams). I haven't committed any work or created any tickets yet, but hopefully I'll get a chance to do that in the next couple of days. Feel free to send people over there if they are itching to work on it. Once a little momentum gets built up I'll post to some user groups to solicit help.

[supertramp01]

@zzbennett I'd love to contribute to the python-kafka-streams repo.

[ayanguha]

I would love to work on this, as well as love the idea itself :)

Wondering if someone has some initial design which I can start working with?

so... what's best practice? use Jython?