Connecting SASL_SSL Kafka Cluster #583

tommyJimmy87 · 2019-03-06T17:33:05Z

I'm using Security Protocol set to "SASL_SSL" which means I need to specify all the JKS and JAAS in order to connect to the Cluster. I'm providing those information through the property file, but I found out that when the KafkaAdminClient runs it gets wrong SSL configuration because of this :

// Configure SSL configs (if security protocol is SSL)
      if (securityProtocol.equals(SecurityProtocol.SSL.name)) {
        setStringConfigIfExists(configs, adminClientConfigs, SslConfigs.SSL_TRUSTMANAGER_ALGORITHM_CONFIG);
        setStringConfigIfExists(configs, adminClientConfigs, SslConfigs.SSL_KEYMANAGER_ALGORITHM_CONFIG);
        setStringConfigIfExists(configs, adminClientConfigs, SslConfigs.SSL_KEYSTORE_TYPE_CONFIG);
        setStringConfigIfExists(configs, adminClientConfigs, SslConfigs.SSL_KEYSTORE_LOCATION_CONFIG);
        setStringConfigIfExists(configs, adminClientConfigs, SslConfigs.SSL_TRUSTSTORE_TYPE_CONFIG);
        setStringConfigIfExists(configs, adminClientConfigs, SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG);
        setStringConfigIfExists(configs, adminClientConfigs, SslConfigs.SSL_SECURE_RANDOM_IMPLEMENTATION_CONFIG);
        setPasswordConfigIfExists(configs, adminClientConfigs, SslConfigs.SSL_KEYSTORE_PASSWORD_CONFIG);
        setPasswordConfigIfExists(configs, adminClientConfigs, SslConfigs.SSL_KEY_PASSWORD_CONFIG);
        setPasswordConfigIfExists(configs, adminClientConfigs, SslConfigs.SSL_TRUSTSTORE_PASSWORD_CONFIG);
      }

I'm not using SSL but SASL_SSL and I still need those properties to be set in the KafkaClientAdmin configurations.

The class where i found that code portion is KafkaCruiseControlUtils.java.

This is the error I'm facing :

org.apache.kafka.common.KafkaException: Failed create new KafkaAdminClient
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:370)
	at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:62)
	at com.linkedin.kafka.cruisecontrol.KafkaCruiseControlUtils.createAdminClient(KafkaCruiseControlUtils.java:176)
	at com.linkedin.kafka.cruisecontrol.KafkaCruiseControlUtils.describeLogDirs(KafkaCruiseControlUtils.java:161)
	at com.linkedin.kafka.cruisecontrol.servlet.response.KafkaClusterState.populateKafkaBrokerLogDirState(KafkaClusterState.java:323)
	at com.linkedin.kafka.cruisecontrol.servlet.response.KafkaClusterState.getJsonStructure(KafkaClusterState.java:252)
	at com.linkedin.kafka.cruisecontrol.servlet.response.KafkaClusterState.getJSONString(KafkaClusterState.java:85)
	at com.linkedin.kafka.cruisecontrol.servlet.response.KafkaClusterState.discardIrrelevantAndCacheRelevant(KafkaClusterState.java:454)
	at com.linkedin.kafka.cruisecontrol.servlet.response.AbstractCruiseControlResponse.discardIrrelevantResponse(AbstractCruiseControlResponse.java:43)
	at com.linkedin.kafka.cruisecontrol.servlet.response.AbstractCruiseControlResponse.writeSuccessResponse(AbstractCruiseControlResponse.java:34)
	at com.linkedin.kafka.cruisecontrol.servlet.KafkaCruiseControlServletUtils.syncRequest(KafkaCruiseControlServletUtils.java:118)
	at com.linkedin.kafka.cruisecontrol.servlet.KafkaCruiseControlServlet.doGet(KafkaCruiseControlServlet.java:159)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:535)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:564)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
	at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
	at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:673)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:591)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.common.KafkaException: java.lang.IllegalArgumentException: No serviceName defined in either JAAS or Kafka config
	at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:153)
	at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:140)
	at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:65)
	at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:88)
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:346)
	... 39 more
Caused by: java.lang.IllegalArgumentException: No serviceName defined in either JAAS or Kafka config
	at org.apache.kafka.common.security.kerberos.KerberosLogin.getServiceName(KerberosLogin.java:303)
	at org.apache.kafka.common.security.kerberos.KerberosLogin.configure(KerberosLogin.java:92)
	at org.apache.kafka.common.security.authenticator.LoginManager.<init>(LoginManager.java:63)
	at org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:114)
	at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:142)
	... 43 more

Tag Cruise Control version is 2.0.36. Kafka Version is 1.1.x.

The text was updated successfully, but these errors were encountered:

efeg · 2019-03-07T02:35:48Z

@tommyJimmy87 Thanks for reporting the issue!
Cruise Control does not support SASL_SSL, yet -- i.e. it currently supports SSL and Plaintext.
Would you be interested in submitting a patch to add SASL_SSL support?

becketqin · 2019-03-07T07:25:42Z

@tommyJimmy87 Are you trying to use Kerberos for authentication? If so, one of the following configs is needed:

sasl.kerberos.service.name
sasl.jaas.config

It would be more helpful if you can share the configurations you are using.

tommyJimmy87 · 2019-03-07T09:06:28Z

@becketqin We are not using kerberos... we are just using SASL_SSL with the username and password in the JAAS.. but apparently as @efeg said is not supported.

tommyJimmy87 · 2019-03-07T09:07:27Z

@efeg I can take a look how complicate is to patch it.

becketqin · 2019-03-07T09:28:36Z

@tommyJimmy87 Hmm, CC should support all the mechanisms support by Kafka itself. Have you set sasl.mechanism in your config? By default it is going to be GSSAPI and kerberos will be used. With SASL_SSL, change the that config to PLAIN and have a try.

becketqin · 2019-03-07T09:30:57Z

Also, please make sure you are using the org.apache.kafka.common.security.plain.PlainLoginModule with your username and password in the jaas config file.

tommyJimmy87 · 2019-03-07T09:49:05Z

@becketqin Thx, here my config file :

broker.metric.sample.store.topic=__KafkaCruiseControlModelTrainingSamples
sample.store.class=com.linkedin.kafka.cruisecontrol.monitor.sampling.KafkaSampleStore
ssl.key.password=xxxxxxxxx (password is present)
min.samples.per.broker.metrics.window=1
num.partition.metrics.windows=1
ssl.keystore.password=xxxxxxxxx (password is present)
sample.store.topic.replication.factor=2
ssl.keystore.location=/etc/kafka/secrets/jks/kafka.keystore.jks
capacity.config.file=/etc/kafka-cruise-control/config/capacity.json
webserver.ui.diskpath=/app/cruise-control-ui/dist/
zookeeper.connect=ZKIP:2181/devkafka
partition.metrics.window.ms=300000
metric.sampler.partition.assignor.class=com.linkedin.kafka.cruisecontrol.monitor.sampling.DefaultMetricSamplerPartitionAssignor
ssl.truststore.location=/etc/kafka/secrets/jks/kafka.truststore.jks
ssl.client.auth=true
partition.metric.sample.store.topic=__KafkaCruiseControlPartitionMetricSamples
sasl.mechanism=PLAIN
security.protocol=SASL_SSL
producer.security.protocol=SASL_SSL
client.id=kafka-cruise-control
broker.metrics.window.ms=300000
default.goals=com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.PotentialNwOutGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.TopicReplicaDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderBytesInDistributionGoal
producer.sasl.mechanism=PLAIN
bootstrap.servers=SASL_SSL://myIP:9094
cluster.configs.file=/etc/kafka-cruise-control/config/clusterConfigs.json
num.sample.loading.threads=8
num.broker.metrics.windows=20
ssl.truststore.password=xxxxxxxxx (password is present)
ssl.endpoint.identification.algorithm=
metric.sampler.class=com.linkedin.kafka.cruisecontrol.monitor.sampling.CruiseControlMetricsReporterSampler
consumer.sasl.mechanism=PLAIN
consumer.security.protocol=SASL_SSL
num.metric.fetcher=1
auto.create.topics.enable=true
metric.sampling.interval.ms=120000
metric.reporter.topic.pattern=__CruiseControlMetrics

And also I'm providing my JAAS file path in the KAFKA_OPTS env variable that should be added when CC runs.
We are using the same JAAS file for several application.

tommyJimmy87 · 2019-03-07T09:51:59Z

@becketqin The problem is just the KafkaAdminClient Configs, the ConsumerConf for example are created with the right values (jks.. password and jaas are present in the ConsumerConf but not in the KafkaAdminClient).

tommyJimmy87 · 2019-03-07T11:17:27Z

@becketqin Uhm.. with the tag 0.1.39 is working actually.. earlier I was using 2.0.36 Tag... what's the different ? Master branch should be at the 0.1.39 right ? I guess it was a little bit confusing.

becketqin · 2019-03-07T11:32:44Z

@tommyJimmy87 Thanks for the update. This sounds like a bug in 2.0.36. But I could be wrong. @efeg would be more familiar with the diff between those two versions.

efeg · 2019-03-07T17:43:32Z

@tommyJimmy87 The reason why the tag 0.1.39 is working, but 2.0.36 is not is because former versions (i.e. 0.1.*) do not call describeLogDirs to show logDirs in the response of kafka_cluster_state (i.e. the endpoint that your request is sent to). The describeLogDirs creates and uses a KafkaAdminClient to return the corresponding response (supported for Kafka: 1.1.0+ with Cruise Control: 2.*).

To support SASL_SSL in 2.*, I believe there are (at least) two places to update (and test) -- i.e. you may search for the string if (securityProtocol.equals(SecurityProtocol.SSL.name)) to find the exact locations:

^ So I suspect that all we need would be:

else if (securityProtocol.equals(SecurityProtocol.SASL_SSL.name)) {
  setStringConfigIfExists(configs, adminClientConfigs, SaslConfigs.SASL_JAAS_CONFIG);
  setPasswordConfigIfExists(configs, adminClientConfigs, SaslConfigs.*); // Needs update for any other relevant password configs
  setStringConfigIfExists(configs, adminClientConfigs, SaslConfigs.*); // Needs update for any other relevant string configs.
}

tommyJimmy87 · 2019-03-08T13:06:59Z

@efeg Ok as I thought. If you want I can contribute on this.

efeg · 2019-03-08T16:00:30Z

@tommyJimmy87 Sure, we appreciate contributions!

Sugaroverdose · 2019-04-05T13:04:39Z

I've encountered the same issue with SASL_PLAINTEXT and sasl.mechanism=SCRAM-SHA-256:
Cruise Control does connects to the cluster but right after that it tries to start AdminClient which does not inherit client.id, jaas.config and sasl.mechanism

UPD: managed to fix this by adding
import org.apache.kafka.common.config.SaslConfigs;
and
setStringConfigIfExists(configs, adminClientConfigs, SaslConfigs.SASL_MECHANISM);
into KafkaCruiseControlUtils.java
i guess the same should be added into CruiseControlMetricsUtils.java

efeg · 2019-04-05T17:50:35Z

@Sugaroverdose Sounds great! Would you consider creating a PR with the patch?

Sugaroverdose · 2019-04-05T20:41:20Z

@efeg i'll try to
it seems like that CruiseControlMetricsUtils.java does not inherit securityProtocol, so idk how it works with something different than plaintext

Sugaroverdose · 2019-04-09T10:31:52Z

@tommyJimmy87 you may try latest migrate_to_kafka_2_0 branch, it should work in your case now

poyyi · 2019-05-02T14:17:44Z

I'm novice in adopting to cruise-contol using 2.029,trying with Sasl_ssl going by above discussion it looks I'm may not be able use it sasl_ssl
could you suggest which release shld I pick ..
Thank u

tommyJimmy87 · 2019-05-03T07:55:18Z

@poyyi What is your Cruise Control Metric Jar configuration?

poyyi · 2019-05-04T23:15:45Z

@tommy
it is cruise-control-metrics-reporter-0.1.0-SNAPSHOT.jar
META-INF/
META-INF/MANIFEST.MF
com/
com/linkedin/
com/linkedin/kafka/
com/linkedin/kafka/cruisecontrol/
com/linkedin/kafka/cruisecontrol/metricsreporter/
com/linkedin/kafka/cruisecontrol/metricsreporter/CruiseControlMetricsReporter$1.class
com/linkedin/kafka/cruisecontrol/metricsreporter/CruiseControlMetricsReporter.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/YammerMetricProcessor$Context.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/YammerMetricProcessor.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/CruiseControlMetric$MetricClassId.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/CruiseControlMetric.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/RawMetricType$MetricScope.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/RawMetricType.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/BrokerMetric.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/MetricSerde$1.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/MetricSerde.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/MetricsUtils.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/PartitionMetric.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/TopicMetric.class
com/linkedin/kafka/cruisecontrol/metricsreporter/CruiseControlMetricsReporterConfig.class
com/linkedin/kafka/cruisecontrol/metricsreporter/exception/

on the kafka servers.. "2024" is sasl_ssl port
metric.reporters=com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter
cruise.control.metrics.reporter.bootstrap.servers=0.0.0.0:2024
auto.create.topics.enable=true

CC property file
bootstrap.servers=BK1-IP:2024,BK2-IP:2024,BK3-IP:2024,BK4-IP:2024,BK5-IP:2024
zookeeper.connect=ZK1-IP:2181,ZK2-IP:2181,ZK3-IP:2181
sample.store.class=com.linkedin.kafka.cruisecontrol.monitor.sampling.KafkaSampleStore
ssl.key.password=XXXXX@123
ssl.keystore.password=XXXXX@123
ssl.keystore.location=/opt/app/work/kafka-0.10.01/certs/kafka.server.truststore.jks
ssl.truststore.location=/opt/app/work/kafka-0.10.01/certs/kafka.server.truststore.jks
ssl.client.auth=true
sasl.mechanism=PLAIN
security.protocol=SASL_SSL
producer.security.protocol=SASL_SSL
producer.sasl.mechanism=PLAIN
ssl.truststore.password=XXXXX@123
ssl.endpoint.identification.algorithm=
consumer.sasl.mechanism=PLAIN
consumer.security.protocol=SASL_SSL

tommyJimmy87 · 2019-05-05T08:44:41Z

I guess you are missing all the Cruise Control configs for Sasl SSL.

Try with these also :


KAFKA_METRIC_REPORTERS: "com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter"
  KAFKA_CRUISE_CONTROL_METRICS_REPORTER_SECURITY_PROTOCOL: "SASL_SSL"
  KAFKA_CRUISE_CONTROL_METRICS_REPORTER_SASL_MECHANISM: "PLAIN"

  KAFKA_CRUISE_CONTROL_METRICS_REPORTER_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: ""
  KAFKA_CRUISE_CONTROL_METRICS_REPORTER_SSL_CLIENT_AUTH: "true"
  KAFKA_CRUISE_CONTROL_METRICS_REPORTER_SSL_KEYSTORE_LOCATION: '/path/to/kafka.server.keystore.jks'
  KAFKA_CRUISE_CONTROL_METRICS_REPORTER_SSL_TRUSTSTORE_LOCATION: '/path/to/kafka.server.truststore.jks'

  KAFKA_CRUISE_CONTROL_METRICS_REPORTER_SASL_MECHANISM_INTER_BROKER_PROTOCOL: "{{.kafkaSaslMechanismInterBrokerProtocol}}"
  KAFKA_CRUISE_CONTROL_METRICS_REPORTER_SASL_ENABLED_MECHANISMS: "{{.kafkaSaslEnabledMechanisms}}"

  KAFKA_CRUISE_CONTROL_METRICS_REPORTER_LISTENER_SECURITY_PROTOCOL_MAP: "INTERNAL:SASL_SSL,CLIENT:SASL_SSL,BROKER:SASL_SSL,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_SSL:SASL_SSL"

  KAFKA_CRUISE_CONTROL_METRICS_REPORTER_LISTENER_SECURITY_PROTOCOL_MAP: "INTERNAL:SSL,CLIENT:SSL,BROKER:SSL,PLAINTEXT:PLAINTEXT,SSL:SSL"

  KAFKA_CRUISE_CONTROL_METRICS_REPORTER_LISTENER_SECURITY_PROTOCOL_MAP: "INTERNAL:PLAINTEXT,CLIENT:PLAINTEXT,BROKER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL"
 KAFKA_CRUISE_CONTROL_METRICS_REPORTER_ADVERTISED_LISTENERS={advertised_listeners_list}

KAFKA_CRUISE_CONTROL_METRICS_REPORTER_SSL_TRUSTSTORE_PASSWORD={trustStorePass}
KAFKA_CRUISE_CONTROL_METRICS_REPORTER_SSL_KEYSTORE_PASSWORD={keyStorePass}
KAFKA_CRUISE_CONTROL_METRICS_REPORTER_SSL_KEY_PASSWORD={keyPass}
KAFKA_CRUISE_CONTROL_METRICS_REPORTER_BOOTSTRAP_SERVERS={yoursBootstrapServer}

poyyi · 2019-05-17T23:03:10Z

@jimmy sorry could not try immly on SASL_SSL (but tired with SSL which i run into some other issue described below ).. i have few queries on your suggestions for SASL_SSL

1)Above mentioned properties shld go into server.properties on brokers?
2) shld they be given exactly like how you gave above
KAFKA_METRIC_REPORTERS: ?, currently i have it as below in my server.properties
metric.reporters=com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter
cruise.control.metrics.reporter.security.protocol=SSL

PL note version I'm using:
Cruise control version: Cruise-control-2.0.42
Kafka :kafka_2.12-0.11.0.0 (Kafka version : 0.11.0.1)

I ran with SSL pl find my setting and Exception I'm getting for

Failures
"GET /kafkacruisecontrol/kafka_cluster_state?json=true HTTP/1.1" 500 4462 79

Success
"GET /kafkacruisecontrol/user_tasks?json=true HTTP/1.1" 200 2637 8
"GET /kafkacruisecontrol/state?substates=EXECUTOR&verbose=true&json=true HTTP/1.1" 200 61 2
"GET /kafkacruisecontrol/state?substates=MONITOR&verbose=true&json=true HTTP/1.1" 200 288 1
"GET /kafkacruisecontrol/state?substates=ANALYZER&verbose=true&json=true HTTP/1.1" 200 2866 2
"GET /kafkacruisecontrol/load?allow_capacity_estimation=true&json=true HTTP/1.1" 200 3074 2
"GET /kafkacruisecontrol/user_tasks?json=true HTTP/1.1" 200 3075 2
"GET /kafkacruisecontrol/state?substates=EXECUTOR&verbose=true&json=true HTTP/1.1" 200 61 1
"GET /kafkacruisecontrol/user_tasks?json=true HTTP/1.1" 200 3275 2

Exception:
GET request '/kafka_cluster_state' due to: 'Failed to populate broker logDir state.'. (com.linkedin.kafka.cruisecontrol.servlet.KafkaCruiseControlServlet)
java.lang.RuntimeException: Failed to populate broker logDir state.
at com.linkedin.kafka.cruisecontrol.servlet.response.KafkaClusterState.getJSONString(KafkaClusterState.java:90)
at com.linkedin.kafka.cruisecontrol.servlet.response.KafkaClusterState.discardIrrelevantAndCacheRelevant(KafkaClusterState.java:456)
at com.linkedin.kafka.cruisecontrol.servlet.response.AbstractCruiseControlResponse.discardIrrelevantResponse(AbstractCruiseControlResponse.java:49)
Caused by: org.apache.kafka.common.errors.UnsupportedVersionException: The broker does not support DESCRIBE_LOG_DIRS
---------------Cruisecontrol.properties Setting----
Cruise control.properties
security.protocol=SSL
ssl.truststore.type=JKS
ssl.truststore.location=/opt/app/workload/cruise-control-2.0.42/config/certs/kafka.server.truststore.jks
ssl.truststore.password=pxxxxxxxx@123
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
#sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="uxxxxxx" password="pxxxxxxxxx";

---------Broker server.properties----------------
listeners=PLAINTEXT://bk1:2021,SASL_PLAINTEXT://bk1:2022,SSL://bk1:2023,SASL_SSL://bk1:2024
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
security.inter.broker.protocol=SASL_SSL
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
super.users=User:mxxxx
allow.everyone.if.no.acl.found=true
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer

advertised.listeners=PLAINTEXT://bk1:2021,SASL_PLAINTEXT://bk1:2022,SSL://bk1:2023,SASL_SSL://bk1:2024
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1

############################# Log Basics #############################

log.dirs=/opt/app/workload/enabler/kafka_2.12-0.11.0.0/kafka-logs

#####Cruise control setting##############
group.initial.rebalance.delay.ms=0
metric.reporters=com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter
cruise.control.metrics.reporter.security.protocol=SSL
cruise.control.metrics.reporter.bootstrap.servers=0.0.0.0:2023
auto.create.topics.enable=true
cruise.control.metrics.reporter.ssl.truststore.location=/opt/app/workload/kafka_2.12-0.11.0.0/config/certs/kafka.server.truststore.jks
cruise.control.metrics.reporter.ssl.truststore.password=pxxxxxxxx@123
cruise.control.metrics.reporter.ssl.keystore.location=/opt/app/workload/kafka_2.12-0.11.0.0/config/certs/kafka.server.keystore.jks
cruise.control.metrics.reporter.ssl.keystore.password=pxxxxxxxx@123
cruise.control.metrics.reporter.ssl.key.password=pxxxxxxxx@123
broker.rack=Rack1

poyyi · 2019-05-24T19:53:41Z

@tommyJimmy87 ...did you get chance to look at my query

ishita07-danaher · 2019-12-04T08:22:17Z

I am facing the same issue. Can you please update on this ?

Arullaldivakar · 2021-03-30T14:40:55Z

Does cruise control support TLS SASL/Scram enabled Kafka secure cluster?

HirossxD · 2024-07-23T09:26:53Z

I have the same issue using SASL_PLAINTEXT, CC not working on kerberized kafka cluster,

i am using older commit of this repo: "repo_commit": "6e8f2ef0209903dc1036857edff1e1020058f02e"

seems like AdminClientConfig does not fetch/load sasl.kereberos.service.name , it is null even if specified in cruise_control_jaas.conf and cruisecontrol.properties files (or kafka_opts)

[2024-07-23 04:29:52,734] INFO COMMIT INFO: 2.5.126---6e8f2ef0209903dc1036857edff1e1020058f02e (com.linkedin.kafka.cruisecontrol.KafkaCruiseControl)
[2024-07-23 04:29:52,744] INFO AdminClientConfig values:
        bootstrap.servers = [poc-brkfk-1:9092, poc-brkfk-2:9092, poc-brkfk-3:9092]
        client.dns.lookup = use_all_dns_ips
        client.id =
        connections.max.idle.ms = 300000
        default.api.timeout.ms = 60000
        metadata.max.age.ms = 300000
        metric.reporters = []
        metrics.num.samples = 2
        metrics.recording.level = INFO
        metrics.sample.window.ms = 30000
        receive.buffer.bytes = 65536
        reconnect.backoff.max.ms = 50
        reconnect.backoff.ms = 50
        request.timeout.ms = 180000
        retries = 2147483647
        retry.backoff.ms = 100
        sasl.client.callback.handler.class = null
        sasl.jaas.config = [hidden]
        sasl.kerberos.kinit.cmd = /usr/bin/kinit
        sasl.kerberos.min.time.before.relogin = 60000
        sasl.kerberos.service.name = null
        sasl.kerberos.ticket.renew.jitter = 0.05
        sasl.kerberos.ticket.renew.window.factor = 0.8
        sasl.login.callback.handler.class = null
        sasl.login.class = null
        sasl.login.connect.timeout.ms = null
        sasl.login.read.timeout.ms = null
        sasl.login.refresh.buffer.seconds = 300
        sasl.login.refresh.min.period.seconds = 60
        sasl.login.refresh.window.factor = 0.8
        sasl.login.refresh.window.jitter = 0.05
        sasl.login.retry.backoff.max.ms = 10000
        sasl.login.retry.backoff.ms = 100
        sasl.mechanism = GSSAPI
        sasl.oauthbearer.clock.skew.seconds = 30
        sasl.oauthbearer.expected.audience = null
        sasl.oauthbearer.expected.issuer = null
        sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
        sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
        sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
        sasl.oauthbearer.jwks.endpoint.url = null
        sasl.oauthbearer.scope.claim.name = scope
        sasl.oauthbearer.sub.claim.name = sub
        sasl.oauthbearer.token.endpoint.url = null
        security.protocol = SASL_PLAINTEXT
        security.providers = null
        send.buffer.bytes = 131072
        socket.connection.setup.timeout.max.ms = 30000
        socket.connection.setup.timeout.ms = 10000
        ssl.cipher.suites = null
        ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
        ssl.endpoint.identification.algorithm = https
        ssl.engine.factory.class = null
        ssl.key.password = null
        ssl.keymanager.algorithm = SunX509
        ssl.keystore.certificate.chain = null
        ssl.keystore.key = null
        ssl.keystore.location = null
        ssl.keystore.password = null
        ssl.keystore.type = JKS
        ssl.protocol = TLSv1.3
        ssl.provider = null
        ssl.secure.random.implementation = null
        ssl.trustmanager.algorithm = PKIX
        ssl.truststore.certificates = null
        ssl.truststore.location = null
        ssl.truststore.password = null
        ssl.truststore.type = JKS
 (org.apache.kafka.clients.admin.AdminClientConfig)
[2024-07-23 04:29:52,781] INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics)
[2024-07-23 04:29:52,783] INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics)
[2024-07-23 04:29:52,783] INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics)
[2024-07-23 04:29:52,783] ERROR Uncaught exception on thread Thread[main,5,main] (com.linkedin.kafka.cruisecontrol.KafkaCruiseControlMain)
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
        at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:538) ~[kafka-clients-3.1.0.jar:?]
        at org.apache.kafka.clients.admin.Admin.create(Admin.java:143) ~[kafka-clients-3.1.0.jar:?]
        at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:49) ~[kafka-clients-3.1.0.jar:?]
        at com.linkedin.kafka.cruisecontrol.KafkaCruiseControlUtils.createAdminClient(KafkaCruiseControlUtils.java:582) ~[cruise-control-2.5.126.jar:?]
        at com.linkedin.kafka.cruisecontrol.KafkaCruiseControl.<init>(KafkaCruiseControl.java:120) ~[cruise-control-2.5.126.jar:?]
        at com.linkedin.kafka.cruisecontrol.async.AsyncKafkaCruiseControl.<init>(AsyncKafkaCruiseControl.java:34) ~[cruise-control-2.5.126.jar:?]
        at com.linkedin.kafka.cruisecontrol.KafkaCruiseControlApp.<init>(KafkaCruiseControlApp.java:36) ~[cruise-control-2.5.126.jar:?]
        at com.linkedin.kafka.cruisecontrol.KafkaCruiseControlServletApp.<init>(KafkaCruiseControlServletApp.java:32) ~[cruise-control-2.5.126.jar:?]
        at com.linkedin.kafka.cruisecontrol.KafkaCruiseControlUtils.getCruiseControlApp(KafkaCruiseControlUtils.java:920) ~[cruise-control-2.5.126.jar:?]
        at com.linkedin.kafka.cruisecontrol.KafkaCruiseControlMain.main(KafkaCruiseControlMain.java:37) ~[cruise-control-2.5.126.jar:?]
Caused by: org.apache.kafka.common.KafkaException: java.lang.IllegalArgumentException: No serviceName defined in either JAAS or Kafka config
        at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:184) ~[kafka-clients-3.1.0.jar:?]
        at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192) ~[kafka-clients-3.1.0.jar:?]
        at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81) ~[kafka-clients-3.1.0.jar:?]
        at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105) ~[kafka-clients-3.1.0.jar:?]
        at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:509) ~[kafka-clients-3.1.0.jar:?]
        ... 9 more
Caused by: java.lang.IllegalArgumentException: No serviceName defined in either JAAS or Kafka config
        at org.apache.kafka.common.security.kerberos.KerberosLogin.getServiceName(KerberosLogin.java:303) ~[kafka-clients-3.1.0.jar:?]
        at org.apache.kafka.common.security.kerberos.KerberosLogin.configure(KerberosLogin.java:92) ~[kafka-clients-3.1.0.jar:?]
        at org.apache.kafka.common.security.authenticator.LoginManager.<init>(LoginManager.java:61) ~[kafka-clients-3.1.0.jar:?]
        at org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:105) ~[kafka-clients-3.1.0.jar:?]
        at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:170) ~[kafka-clients-3.1.0.jar:?]
        at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192) ~[kafka-clients-3.1.0.jar:?]
        at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81) ~[kafka-clients-3.1.0.jar:?]
        at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105) ~[kafka-clients-3.1.0.jar:?]
        at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:509) ~[kafka-clients-3.1.0.jar:?]
        ... 9 more

i will try same procedure on the latest commit, will see if issue persist.

HirossxD · 2024-07-23T11:21:42Z

switched to commit f23332a and cruisecontrol started without an issue..

here are some necessary configuration of cruisecontrol.properties

security.protocol=SASL_PLAINTEXT
sasl.mechanism=GSSAPI
sasl.kerberos.service.name=kafka

cruise.control.metrics.reporter.security.protocol=SASL_PLAINTEXT
cruise.control.metrics.reporter.sasl.mechanism=GSSAPI
cruise.control.metrics.reporter.sasl.kerberos.service.name=kafka

# Kerberos principal and keytab for Cruise Control
cruise.control.metrics.reporter.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \
    useKeyTab=true \
    storeKey=true \
    keyTab="/etc/security/keytabs/kafka.service.keytab" \
    principal="kafka/[email protected]";



# Kerberos configuration file location
java.security.auth.login.config=/opt/cruise-control/config/cruise_control_jaas.conf
java.security.krb5.conf=/etc/krb5.conf

but cruisecontrol logs says

 INFO Collected 1 broker metric samples for 1 brokers. (com.linkedin.kafka.cruisecontrol.monitor.sampling.SamplingFetcher)

i have just setup cc parameters only on one broker yet.

current broker configuration lines related to CC and kerberos are

metric.reporters=com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter
cruise.control.metrics.topic.auto.create=true
cruise.control.metrics.topic.num.partitions=1
cruise.control.metrics.topic.replication.factor=1



cruise.control.metrics.reporter.bootstrap.servers=poc-brkfk-1:9092


cruise.control.metrics.reporter.security.protocol=SASL_PLAINTEXT
cruise.control.metrics.reporter.sasl.mechanism=GSSAPI

cruise.control.metrics.reporter.sasl.kerberos.service.name=kafka

# Kerberos principal and keytab for Cruise Control
cruise.control.metrics.reporter.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \
useKeyTab=true \
storeKey=true \
keyTab="/etc/security/keytabs/kafka.service.keytab" \
principal="kafka/[email protected]";

so it is functional i guess.

Sugaroverdose mentioned this issue Apr 5, 2019

Inherit sasl.mechanism and sasl.jaas.config for AdminClient #643

Merged

efeg mentioned this issue May 21, 2019

Kerberos Support #326

Closed

efeg added the usability Improves the ease of use or learnability of the system. label May 21, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Connecting SASL_SSL Kafka Cluster #583

Connecting SASL_SSL Kafka Cluster #583

tommyJimmy87 commented Mar 6, 2019 •

edited

Loading

efeg commented Mar 7, 2019

becketqin commented Mar 7, 2019

tommyJimmy87 commented Mar 7, 2019

tommyJimmy87 commented Mar 7, 2019

becketqin commented Mar 7, 2019

becketqin commented Mar 7, 2019

tommyJimmy87 commented Mar 7, 2019 •

edited

Loading

tommyJimmy87 commented Mar 7, 2019 •

edited

Loading

tommyJimmy87 commented Mar 7, 2019

becketqin commented Mar 7, 2019

efeg commented Mar 7, 2019

tommyJimmy87 commented Mar 8, 2019

efeg commented Mar 8, 2019

Sugaroverdose commented Apr 5, 2019 •

edited

Loading

efeg commented Apr 5, 2019

Sugaroverdose commented Apr 5, 2019

Sugaroverdose commented Apr 9, 2019

poyyi commented May 2, 2019

tommyJimmy87 commented May 3, 2019

poyyi commented May 4, 2019

tommyJimmy87 commented May 5, 2019

poyyi commented May 17, 2019 •

edited

Loading

poyyi commented May 24, 2019

ishita07-danaher commented Dec 4, 2019

Arullaldivakar commented Mar 30, 2021

HirossxD commented Jul 23, 2024

HirossxD commented Jul 23, 2024 •

edited

Loading

Connecting SASL_SSL Kafka Cluster #583

Connecting SASL_SSL Kafka Cluster #583

Comments

tommyJimmy87 commented Mar 6, 2019 • edited Loading

efeg commented Mar 7, 2019

becketqin commented Mar 7, 2019

tommyJimmy87 commented Mar 7, 2019

tommyJimmy87 commented Mar 7, 2019

becketqin commented Mar 7, 2019

becketqin commented Mar 7, 2019

tommyJimmy87 commented Mar 7, 2019 • edited Loading

tommyJimmy87 commented Mar 7, 2019 • edited Loading

tommyJimmy87 commented Mar 7, 2019

becketqin commented Mar 7, 2019

efeg commented Mar 7, 2019

tommyJimmy87 commented Mar 8, 2019

efeg commented Mar 8, 2019

Sugaroverdose commented Apr 5, 2019 • edited Loading

efeg commented Apr 5, 2019

Sugaroverdose commented Apr 5, 2019

Sugaroverdose commented Apr 9, 2019

poyyi commented May 2, 2019

tommyJimmy87 commented May 3, 2019

poyyi commented May 4, 2019

tommyJimmy87 commented May 5, 2019

poyyi commented May 17, 2019 • edited Loading

poyyi commented May 24, 2019

ishita07-danaher commented Dec 4, 2019

Arullaldivakar commented Mar 30, 2021

HirossxD commented Jul 23, 2024

HirossxD commented Jul 23, 2024 • edited Loading

tommyJimmy87 commented Mar 6, 2019 •

edited

Loading

tommyJimmy87 commented Mar 7, 2019 •

edited

Loading

tommyJimmy87 commented Mar 7, 2019 •

edited

Loading

Sugaroverdose commented Apr 5, 2019 •

edited

Loading

poyyi commented May 17, 2019 •

edited

Loading

HirossxD commented Jul 23, 2024 •

edited

Loading