Kafka Producer Setup (Core)

[JayeshWani7]

Title

Setup Kafka Producer for Event Streaming

Description

We need a reliable Kafka producer to send ingestion events into the pipeline.

Goal

Enable scalable event streaming.

Tasks

• Setup Kafka connection
• Define event schema
• Handle retries

🛠️ Tech Context

• Go
• Kafka

Acceptance Criteria

• Events reliably delivered
• Retry mechanism works

[Saranyadharani]

@JayeshWani7

I'd like to work on Issue #3 (Kafka Producer Setup).

A few questions before I start:

1. Is there a preferred Kafka library for this project (Sarama, Confluent)?
2. Should I follow the same pattern as the existing Redis queue setup?
3. Any specific event schema format to follow?

Thank you!

[JayeshWani7]

Hi @Saranyadharani ,

1 -> Use segmentio/kafka-go
2 -> Yes follow the same pattern as the existing one
3 ->

type QueueMessage struct {
    EventID   string    `json:"event_id"`
    Source    string    `json:"source"`
    EventType string    `json:"event_type"`
    Timestamp time.Time `json:"timestamp"`
    Data      string    `json:"data"`  // raw JSON
}

For Kafka, optionally add:

CorrelationID - for tracing across services
Version - for schema evolution
But start with the existing structure to maintain compatibility.

[JayeshWani7]

Will you be able to implement it? @Saranyadharani

[Saranyadharani]

Hi @JayeshWani7,

Yes, I can implement this. Thank you for the clarification!

My approach:

• Use segmentio/kafka-go library as suggested
• Follow the existing Redis queue pattern
• Add retry mechanism with exponential backoff
• Include CorrelationID and Version fields for tracing

will finsh this within 1-2 days

[JayeshWani7]

@Saranyadharani Are you still working on this issue ? let me know if you are stuck somewhere

[Saranyadharani]

@JayeshWani7 PR #8 is ready for review! All Copilot comments have been addressed and all checks are passing.