confluent_kafka/cimpl.cpython-36m-x86_64-linux-gnu.so: undefined symbol: rd_kafka_event_AlterConsumerGroupOffsets_result

[shaileshjagnani]

Description

On centos 7, getting error when importing confluent kafka 2.2.0 in python script.
The confluent-kafka module was built using the source along with librdkafka.

Getting below error when importing the module
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib64/python3.6/site-packages/confluent_kafka/init.py", line 19, in
from .deserializing_consumer import DeserializingConsumer
File "/usr/lib64/python3.6/site-packages/confluent_kafka/deserializing_consumer.py", line 19, in
from confluent_kafka.cimpl import Consumer as _ConsumerImpl
ImportError: /usr/lib64/python3.6/site-packages/confluent_kafka/cimpl.cpython-36m-x86_64-linux-gnu.so: undefined symbol: rd_kafka_event_AlterConsumerGroupOffsets_result

How to reproduce

1. Build the confluent-kafka rpm using the source with below steps

2. dnf install -y python3-devel

3. sed -i -E 's/(name = ")/\1python3-/g' setup.py

4. python3 setup.py sdist bdist_rpm --release="%{timestamp}%{?dist}" --spec-only --build-requires=python3-devel

5. SPEC=$(basename $(find -name *.spec) )

6. rpmbuild -bb dist/"${SPEC}" --define "_sourcedir pwd/dist" --define "_specdir pwd/dist" --define "_rpmdir pwd/dist" --define 'timestamp %(date +"%Y%m%d%H%M")' --define "debug_package %{nil}"

7. Prerequisite to create the confluent-kafka rpm, create the librdkafka as below

8. curl -L https://github.com/confluentinc/librdkafka/archive/refs/tags/v2.2.0.tar.gz | tar xzf -

9. cd librdkafka-2.2.0/

10. ./configure --prefix=/usr

11. make -j

12. make install

Install the rpm created on the centos box and then try to import the module.

Checklist

Please provide the following information:

• confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()):
• Apache Kafka broker version:
• Client configuration: {...}
• Operating system:
• Provide client logs (with 'debug': '..' as necessary)
• Provide broker log excerpts
• Critical issue

[pranavrth]

Is installation from the wheel not working using python3 -m pip install confluent_kafka?

[sbivol]

I got the same error on Ubuntu 22.04 after following a similar procedure.
The solution was to remove the OS-provided librdkafka:

sudo apt purge librdkafka1 librdkafka-dev "librdkafka++1"

Both confluent-kafka=1.9 (from PyPI) and confluent-kafka=2.2 (built locally) seem to be working fine after this.

Is installation from the wheel not working using python3 -m pip install confluent_kafka?

Binary wheels are not published for all compatible Python versions, and wheels built from source don't include librdkafka.so for some reason.

[sandeepkumar1-fareye]

Is there any workaround available? This issue is open for a long time.

[MSeal]

Yes we're planning on revamping and upgrading the release process to include all supported versions of python across common os's as it's been left without upgrade for a while now.