Skip to content

Conversation

@velo
Copy link
Collaborator

@velo velo commented Dec 23, 2025

Summary

Adds kafka-clients JAR to Flink's lib directory to fix NoClassDefFoundError: org/apache/kafka/common/security/auth/AuthenticateCallbackHandler when using MSK IAM authentication.

Problem

When connecting to MSK with IAM authentication, the aws-msk-iam-auth library requires Kafka's authentication interfaces which are defined in kafka-clients. While kafka-clients is included as a transitive dependency in the shaded uber JAR, it's not accessible from Flink's lib classloader where aws-msk-iam-auth is loaded.

Solution

  • Added kafka-clients (3.9.1) as a provided dependency
  • Added maven-dependency-plugin execution to copy the JAR to target
  • Updated Dockerfile to include kafka-clients-*.jar in /opt/flink/lib

Test Plan

  • Built project successfully with mvn clean install -Pfast
  • Verified kafka-clients-3.9.1.jar is copied to target directory

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants