KAFKA-18659: librdkafka compressed produce fails unless api versions returns produce v0 #18727

ijuma · 2025-01-28T08:14:27Z

Advertise support for produce v0-v2, but return an UnsupportedVersion error if a produce request with these versions is attempted.

Since clients pick the highest supported version by both client and broker during version negotiation, this solves the problem with minimal tech debt (even though it's not ideal that the protocol definition becomes inconsistent with the actual protocol support). Adjust the protocol html generation to exclude produce v0-v2 and fix an existing bug in the response output where removed versions were not excluded.

Add an integration test to verify the behavior for produce v0-v2.

Reference to related librdkafka issue: confluentinc/librdkafka#4956

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

core/src/main/scala/kafka/server/KafkaApis.scala

ijuma · 2025-01-29T06:05:15Z

clients/src/main/java/org/apache/kafka/common/protocol/Protocol.java

@@ -208,7 +211,7 @@ public static String toHtml() {
            // Responses
            b.append("<b>Responses:</b><br>\n");
            Schema[] responses = key.messageType.responseSchemas();
-            for (int i = 0; i < responses.length; i++) {
+            for (int i = oldestVersion; i < responses.length; i++) {


This bug fix is unrelated to the main change in this PR.

ijuma · 2025-01-29T06:06:10Z

clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java

@@ -69,11 +74,6 @@ public ProduceRequest build(short version) {
            return build(version, true);
        }

-        // Visible for testing only
-        public ProduceRequest buildUnsafe(short version) {


This is unused.

It seems we can merge private ProduceRequest build(short version, boolean validate) into public ProduceRequest build(short version). Otherwise, IDE complains the method has a "always true" input variable validate.

…returns produce v0

ijuma · 2025-01-29T14:57:41Z

The test failures look related:

FAILED ❌ KafkaAdminClientTest > testAdminClientApisAuthenticationFailure()
FAILED ❌ ProcessorTest > testParseRequestHeaderWithUnsupportedApiVersion()
FAILED ❌ KafkaApisTest > shouldReplaceProducerFencedWithInvalidProducerEpochInProduceResponse()
FAILED ❌ KafkaApisTest > testTransactionalParametersSetCorrectly()

Investigating.

chia7712

@ijuma thanks for this fix. Does this patch enable Kafka clients to communicate with older servers that support Produce v0-v2? If so, should we disable that request on client-side?

chia7712 · 2025-01-29T15:19:27Z

clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java

@@ -69,11 +74,6 @@ public ProduceRequest build(short version) {
            return build(version, true);
        }

-        // Visible for testing only
-        public ProduceRequest buildUnsafe(short version) {


It seems we can merge private ProduceRequest build(short version, boolean validate) into public ProduceRequest build(short version). Otherwise, IDE complains the method has a "always true" input variable validate.

chia7712 · 2025-01-29T15:22:32Z

core/src/main/scala/kafka/server/KafkaApis.scala

+    // See `ProduceRequest.MIN_VERSION` for details on why we need to do this
+    if (produceRequest.version < ProduceRequest.MIN_VERSION) {
+      requestHelper.sendErrorResponseMaybeThrottle(request, Errors.UNSUPPORTED_VERSION.exception())
+      return;


; is unnecessary

ijuma · 2025-01-29T16:24:21Z

@chia7712 Good point regarding the client behavior, I'll look into adjusting that too.

ijuma · 2025-01-29T16:26:04Z

Thinking about it, there is probably a cleaner way to make this change. Let me try that.

junrao · 2025-01-29T17:53:01Z

clients/src/main/resources/common/message/ProduceRequest.json

@@ -44,7 +47,7 @@
  // transaction V2 (KIP_890 part 2) is enabled, the produce request will also include the function for a
  // AddPartitionsToTxn call. If V2 is disabled, the client can't use produce request version higher than 11 within
  // a transaction.
-  "validVersions": "3-12",
+  "validVersions": "0-12",


Instead of bringing back v0 and v1, could we just customize the minVersion of the produce ApiKey in ApiResponse?

I am indeed exploring something involving customizing ApiKey/ApiResponse (that's what I meant when I said that there may be a cleaner way to make the change). But I was planning to leave 0-12 here and reduce the range within the code.

The main advantage is that you get appropriate errors if you try to produce with v0-2. The option you suggest would result in a disconnection. I think a disconnection is too confusing if the given version is included as part of the api versions response.

Thoughts?

It seems that UNSUPPORTED_VERSION was added in 0.10.0 https://github.com/apache/kafka/pull/986/files#diff-0ae188e241ee35148145e7e04ed2acb3c20356650194f3603811c51b886ebe20. Produce v0/v1 already existed in 0.9. So, even if we send UNSUPPORTED_VERSION in the error, the 0.9 client won't be able to understand it. So, it really only helps with Produce v2 request.

I agree that it's nearly impossible to make the experience for these super ancient clients good. I was more thinking about client developers implementing the kafka protocol.

Yes, we could make an effort to send a more meaningful error code for produce v0-v2. But since they are not really supported on the broker, just doing a disconnect like any other unsupported version also seems reasonable.

ijuma changed the title ~~(WIP) KAFKA-18659: librdkafka compressed produce fails unless api versions …~~ (WIP) KAFKA-18659: librdkafka compressed produce fails unless api versions returns produce v0 Jan 28, 2025

github-actions bot added core Kafka Broker clients small Small PRs labels Jan 28, 2025

ijuma changed the title ~~(WIP) KAFKA-18659: librdkafka compressed produce fails unless api versions returns produce v0~~ KAFKA-18659: librdkafka compressed produce fails unless api versions returns produce v0 Jan 28, 2025

stanislavkozlovski reviewed Jan 28, 2025

View reviewed changes

core/src/main/scala/kafka/server/KafkaApis.scala Outdated Show resolved Hide resolved

ijuma force-pushed the kafka-18659-librdkafka-compressed-produce-fails branch from 30ea3d7 to 6b145a6 Compare January 29, 2025 06:03

ijuma commented Jan 29, 2025

View reviewed changes

ijuma force-pushed the kafka-18659-librdkafka-compressed-produce-fails branch from 6b145a6 to 6bbf021 Compare January 29, 2025 06:07

KAFKA-18659: librdkafka compressed produce fails unless api versions …

a252374

…returns produce v0

ijuma force-pushed the kafka-18659-librdkafka-compressed-produce-fails branch from 6bbf021 to a252374 Compare January 29, 2025 06:10

chia7712 reviewed Jan 29, 2025

View reviewed changes

junrao reviewed Jan 29, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KAFKA-18659: librdkafka compressed produce fails unless api versions returns produce v0 #18727

KAFKA-18659: librdkafka compressed produce fails unless api versions returns produce v0 #18727

ijuma commented Jan 28, 2025 •

edited

Loading

ijuma Jan 29, 2025

ijuma Jan 29, 2025

chia7712 Jan 29, 2025

ijuma commented Jan 29, 2025

chia7712 left a comment

chia7712 Jan 29, 2025

chia7712 Jan 29, 2025

ijuma commented Jan 29, 2025

ijuma commented Jan 29, 2025

junrao Jan 29, 2025

ijuma Jan 29, 2025

junrao Jan 29, 2025

ijuma Jan 29, 2025

junrao Jan 29, 2025

KAFKA-18659: librdkafka compressed produce fails unless api versions returns produce v0 #18727

Are you sure you want to change the base?

KAFKA-18659: librdkafka compressed produce fails unless api versions returns produce v0 #18727

Conversation

ijuma commented Jan 28, 2025 • edited Loading

Committer Checklist (excluded from commit message)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ijuma commented Jan 29, 2025

chia7712 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ijuma commented Jan 29, 2025

ijuma commented Jan 29, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ijuma commented Jan 28, 2025 •

edited

Loading