Openelemetry plugin for kafka connect

[radzikp]

It would be nice to have kafka connect plugin for opentelemetry where messages are produced ansynchronously and kafka client is seperated from the 'service'.

An example: The outbox pattern.

The workaround for this:
Create a custom transformation on kafka connect as 3rd party custom library to glue spans in one trace and propagate the context to the Kafka.

[dharmaplay]

any tentative date when can we get the kafka connector for opentelemetry?

[mateuszrzeszutek]

Hey @dharmaplay ,
Unfortunately we don't have enough time and resources to implement it on our own. We would greatly appreciate an external contribution for this.

[deepeshhitachi]

We are blocked as we are using opentelemetry and want to put opentelemetry data into timescale DB using Kafka connect but it is extremely difficult to parse - is kafka connect not support OpenTelemetry at all? We wanted to push from Kafka to Promscale using kafka connect instead of OTEL connector.

[vidourem]

I am trying to implement end-to-end traceability of the transactional outbox pattern with Debezium. There is a SMT for Kafka Connect that extracts the context previously recorded in a database table and propagates it in a ConnectRecord. I just made an evolution of this SMT to work with OpenTelemetry. It works perfectly. However, the Kafka ProducerRecord instrumentation with the interceptor always starts a new trace while the context is present in the record. Is this normal behavior? Could you evolve the instrumentation to extract the context from the record and use it if it exists?

[vidourem]

The issue I encounter between the ConnectRecord and the ProducerRecord is similar to #6297 and #2801 . I don't see any other solutions at the moment except to extend the ProducerRecord's interceptor, have I missed anything?

[Cirilla-zmh]

Hi @vidourem , the reason for the new trace in the Kafka producer might be due to an incorrect context propagation. I don’t believe any improvements are necessary for the kafka-clients instrumentations. Instead, you may need to retrieve the context created by the Kafka producer instrumentation and propagate it in your connector.