KAFKA-16524: Metrics for KIP-853

[kevin-wu24]

This PR implements the following gauges in KafkaRaftMetrics, BrokerServerMetrics, and ControllerServerMetrics:

KafkaRaftMetrics:

• number-of-voters: number of voters for a KRaft topic partition. All of the replicas will report this metric.
• number-of-observers: number of observers being tracked by the KRaft topic partition leader.
• uncommitted-voter-change: 1 if there is a voter change that has not been committed, 0 otherwise.

number-of-observers and uncommitted-voter-change will only be present on the active controller, since it does not make sense for other replicas to report these metrics. If number-of-offline-voters is implemented, it will also only be reported by the leader.

In order to make these two metrics thread-safe, KafkaRaftMetrics needs to be passed into LeaderState, and therefore QuorumState. This introduces a circularity since the KafkaRaftMetrics constructor takes in QuorumState. To break the circularity for now, the logic using QuorumState will be moved to a KafkaRaftMetrics#initialize method.

BrokerServerMetrics:

• ignored-static-voters: 1 if reading the voter set from the log, 0 if the voter set comes from static configuration.

ControllerServerMetrics:

• IgnoredStaticVoters: 1 if reading the voter set from the log, 0 if the voter set comes from static configuration. (same as the metric above but for controllers)
• IsObserver: 1 if the controller node is not a voter for the KRaft cluster metadata partition, 0 otherwise. This metric will not be implemented in this PR. Instead, the current-state metric in conjunction with the presence of kafka.server.controller metrics can be used to provide the same information.

To implement the ignored-static-voters metrics, this PR introduces the ExternalKRaftMetrics class, which allows for higher layer metrics objects to be accessible within the raft module. number-of-voters and ignored-static-voters will be updated in KRaftControlRecordStateMachine#handleBatch when a new VotersRecord is read from log.

Testing:

• Unit tests in KRaftControlRecordStateMachineTest and KafkaRaftMetricsTest.
• Added/modified tests in KafkaRaftClientReconfigTest to verify new metrics behavior.

Committer Checklist (excluded from commit message)

• Verify design and implementation
• Verify test coverage and CI build status
• Verify documentation (including upgrade notes)

[github-actions]

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

[jsancio]

Thanks for the changes @kevin-wu24 . Quick review of src/main.

[jsancio]

Thanks for your improvements @kevin-wu24

[kevin-wu24]

Pushed some commits. Changes of note:

• Moved DefaultExternalKRaftMetrics to /core/src/main/scala/kafka/raft to avoid circular dependency and avoid BrokerServerMetrics and ControllerMetadataMetrics needing to implement another interface.
• I'm not sure whether or not it makes sense to provide a mock implementation of ExternalKRaftMetrics for the purpose of testing. The most recent changes just use the DefaultExternalKRaftMetrics implementation directly, since we want to validate that the metrics values in the brokerServerMetrics and controllerMetadataMetrics objects contained in the implementation are actually being updated. This has the side effect of requiring dependencies being added in the build file for raft tests and allowing the relevant imports.
• Added tests in KafkaMetricsTest for the newly-added number-of-voters, number-of-observers, and uncommitted-voter-change.
• Modified tests in KRaftControlRecordStateMachineTest to test the two ignored-static-voters metrics.
• Added testLeaderMetricsAreReset in KafkaRaftClientReconfigTest to cover the case where a leader loses leadership and regains it, checking that number-of-observers and uncommitted-voter-change are not using the values from previous leadership.

[jsancio]

Thanks for the changes. I reviewed src/main.

[kevin-wu24]

Thanks for the review @jsancio. Pushed some commits:

• Added MockExternalKRaftMetrics to prevent additional raft module dependencies. Added DefaultExternalKRaftMetricsTest since raft layer tests use the mock instead.
• The staticVoterSet passed to KRaftControlRecordStateMachine can never be null. Instead the default value for controller.quorum.voters is the empty list in both implementation and testing. Fixed updating logic for the ignored-static-voters metric so that it always stays false when the mentioned config is not set. Only when that config is actually set can the metric's value ever go from false to true.
• In the case of truncateNewEntries truncating the entire log, if controller.quorum.voters is set, ignored-static-voters metric can go from true back to false. Implementation now covers for this case.
• KRaftControlRecordStateMachineTest checks both ignored-static-voters and number-of-voters metric values.

[jsancio]

Thanks. Submitted a quick review.

[jsancio]

Thanks for the update @kevin-wu24 . Let's make sure that the metrics that represent a boolean expose the value as 1 or 0 and not as true or false.

[jsancio]

LGTM