Skip to content

Replace bitnami Kafka Docker images with confluent ones #21206

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 17 commits into
base: master
Choose a base branch
from
Open

Replace bitnami Kafka Docker images with confluent ones #21206

wants to merge 17 commits into from

Conversation

AAraKKe
Copy link
Contributor

@AAraKKe AAraKKe commented Aug 29, 2025 •
edited
Loading

What does this PR do?

Replace bitnami docker images for Kafka and Kafka Consumer integrations with confluent images instead.

Motivation

Bitnami has removed their open images from the public regsitry. They provided a legacy registry where they are keeping them but if we need to move we better move to a separate source. The legacy ones will be frozen and won't be updated anymore.

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • Add the qa/skip-qa label if the PR doesn't need to be tested during QA.
  • If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

AAraKKe added 2 commits August 29, 2025 16:02
@AAraKKe
Replace bitnami Kafka Docker image with confluent one
00289a9 
This change is applied to both the kafka and kafka_consumer application
since the test setup is similar

Issue AI-6103
@AAraKKe
Remove duplicate depends_on key from docker-compose
fb3f56f
@AAraKKe AAraKKe added the qa/skip-qa Automatically skip this PR for the next QA label Aug 29, 2025
@AAraKKe AAraKKe changed the title Remove duplicate depends_on key from docker-compose Replace bitnami Kafka Docker images with confluent ones Aug 29, 2025
@github-actions GitHub Actions
Copy link

⚠️ The qa/skip-qa label has been added with shippable changes

The following files, which will be shipped with the agent, were modified in this PR and
the qa/skip-qa label has been added.

You can ignore this if you are sure the changes in this PR do not require QA. Otherwise, consider removing the label.

List of modified files that will be shipped with the agent 
kafka/hatch.toml
kafka_consumer/hatch.toml

@ofek
Copy link
Contributor

ofek commented Aug 29, 2025

Nice 👍

@codecov Codecov
Copy link

codecov bot commented Aug 29, 2025 •
edited
Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.01%. Comparing base (a5c52c1) to head (34abe16).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

AAraKKe added 12 commits August 29, 2025 18:15
@AAraKKe
Add wait_for_health when calling docker_run
3179d9e
@AAraKKe
Merge remote-tracking branch 'origin/master' into aarakke/AI-6103/rem…
965d908 
…ove-kafka-bitnami
@AAraKKe
Modify certificates to point to the right host (try fixing the ssl tr…
de62ad5 
…ansport issue)
@AAraKKe
Remove platform from compose file. Left by mistake
94d2d4d
@AAraKKe
Remove compose platform for Kafka integration as well
56838e0
@AAraKKe
Cleanup docker compose and hatch file for kafka consumer
bcb4d57
@AAraKKe
Fix transport issue with kafka consumer
5aa0324
@AAraKKe
Remove hatch environment added by mistake
5737bb2
@AAraKKe
Remove PLAINTEXT listener. Not needed since we do not run tests from …
8fb2ff1 
…within the docker network
@AAraKKe
Use the proper SSL configuration for kafka consumer
b3e9e20
@AAraKKe
Add SSL client properties and improve health checks in Docker setup
8e30d6c
@AAraKKe
Clean up conftests and setup ruff config
855231c
@AAraKKe AAraKKe marked this pull request as ready for review September 5, 2025 09:04
@AAraKKe AAraKKe requested a review from a team as a code owner September 5, 2025 09:04
@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 5, 2025

⚠️ The qa/skip-qa label has been added with shippable changes

The following files, which will be shipped with the agent, were modified in this PR and
the qa/skip-qa label has been added.

You can ignore this if you are sure the changes in this PR do not require QA. Otherwise, consider removing the label.

List of modified files that will be shipped with the agent 
kafka_consumer/pyproject.toml
kafka/hatch.toml
kafka_consumer/hatch.toml

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 5, 2025

⚠️ The qa/skip-qa label has been added with shippable changes

The following files, which will be shipped with the agent, were modified in this PR and
the qa/skip-qa label has been added.

You can ignore this if you are sure the changes in this PR do not require QA. Otherwise, consider removing the label.

List of modified files that will be shipped with the agent 
kafka/changelog.d/21206.fixed
kafka_consumer/changelog.d/21206.fixed
kafka/pyproject.toml
kafka_consumer/pyproject.toml
kafka/hatch.toml
kafka_consumer/hatch.toml

sarah-witt
sarah-witt approved these changes Sep 8, 2025
View reviewed changes
Copy link
Contributor

@sarah-witt sarah-witt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thanks so much! Just left one non-blocking comment

kafka/tests/compose/docker-compose.yml
@@ -1,23 +1,32 @@
services:
zookeeper:
image: bitnami/zookeeper:3.8.0
image: confluentinc/cp-zookeeper:7.2.0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit- should we use CONFLUENT_VERSION here? Or create ZK_VERSION again? I had the same comment for the other integrations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sarah-witt sarah-witt sarah-witt approved these changes

Assignees
No one assigned
Labels
integration/kafka_consumer integration/kafka qa/skip-qa Automatically skip this PR for the next QA team/agent-integrations
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@AAraKKe @ofek @sarah-witt