Skip to content

Conversation

@piochelepiotr
Copy link
Contributor

What does this PR do?

This PR adds the kafka_cluster_id dynamic tag to all Kafka broker metrics collected by the Kafka integration. The tag is dynamically populated from the JMX bean kafka.server:type=KafkaServer,name=ClusterId and is added to metrics from the Kafka brokers.
Consumer and producer metrics are intentionally excluded from this change as they don't represent broker-level metrics.

Motivation

The Kafka cluster ID is a unique identifier for each Kafka cluster that is essential for building reliable observability features. By tagging all broker metrics with the cluster ID, we enable:
Cluster discovery: Automatically getting a comprehensive list of all monitored Kafka clusters
Data correlation: Linking Kafka broker metrics with Data Streams metrics for end-to-end observability
Multi-cluster environments: Better differentiation and filtering of metrics in environments with multiple Kafka clusters
This change is made possible by the dynamic tags functionality added in DataDog/jmxfetch#581, which allows us to dynamically fetch tag values from JMX beans and apply them to metrics at collection time.

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • Add the qa/skip-qa label if the PR doesn't need to be tested during QA.
  • If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

@piochelepiotr piochelepiotr requested a review from a team as a code owner October 16, 2025 21:17
@piochelepiotr piochelepiotr changed the title Add kafka_cluster_id tag to Kafka cluster metrics DSMON-1102: Add kafka_cluster_id tag to Kafka cluster metrics Oct 16, 2025
@codecov
Copy link

codecov bot commented Oct 16, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.99%. Comparing base (d74af14) to head (c1c533a).
⚠️ Report is 10 commits behind head on master.

Additional details and impacted files
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Kyle-Neale Kyle-Neale added this pull request to the merge queue Oct 27, 2025
Merged via the queue into master with commit 95c8bcf Oct 27, 2025
26 checks passed
@Kyle-Neale Kyle-Neale deleted the piotr-wolski/add-kafka-cluster-id-tag branch October 27, 2025 15:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants