Kafka node is crashing because of log file access issue

[ashishpatil09]

Hi Team

I need your help with one issue.
I have one tb-kafka and zookeeper running in k8s and 3 replicas of tb-node. but kafka node is keep crashing and restarting after some time with following error
ERROR Failed to clean up log for __consumer_offsets-35 in dir /kafka-logs/kafka due to IOException (kafka.server.LogDirFailureChannel)
java.io.IOException: Permission denied
at java.io.RandomAccessFile.setLength(Native Method)
at kafka.log.AbstractIndex.$anonfun$resize$1(AbstractIndex.scala:189)
at kafka.log.AbstractIndex.resize(AbstractIndex.scala:175)
at kafka.log.AbstractIndex.$anonfun$trimToValidSize$1(AbstractIndex.scala:241)
at kafka.log.AbstractIndex.trimToValidSize(AbstractIndex.scala:241)
at kafka.log.LogSegment.onBecomeInactiveSegment(LogSegment.scala:507)

Then I tried to reduce the replica of tb-node to 2 and I have now tb-node-0 and tb-node-1 pods are running but I could see js_eval.responses.tb-node-2. folders inside kafka-logs/kafka
Does this cause the Kafka crash or anything i am missing in my config?