Post Deploy/UCI/Orchestrator pod going into crashloopbackoff

[chitranshu-keshav]

Post Build and Deploy job execution, the UCI orchestrator pod is entering the CrashLoopBackoff state as it's startup is failing. The logs and the details analysis of the same can be referenced in the Sunbird ED discussion thread 421 (https://github.com/orgs/Sunbird-Ed/discussions/421)

@ChakshuGautam @abhay-samagra can you please get the required updates done in UCI code to resolve this issue. @subhashchandrab would be able to provide you any additional details that you may need

cc: @subhashchandrab @vinukumar-vs @Krishnaj20 @anandvarada

[vinukumar-vs]

@ChakshuGautam @abhay-samagra Any suggestion on this?

[vinukumar-vs]

@abhay-samagra You can reach-out to @gandham-santhosh for NOC team help.

@subhashchandrab @anandvarada

[gandham-santhosh]

@subhashchandrab can you please setup a meeting with @abhay-samagra to discuss this?

[Krishnaj20]

@subhashchandrab - as you can see from the discussions in this thread, @abhay-samagra 's team is not able to replicate the issue on their side. So, they have asked for your help in debugging this issue. Appreciate if you can co-ordinate with @abhay-samagra and close this thread.

@anandvarada

[anandvarada-diksha]

@Krishnaj20 i have added @chitranshu-keshav @abhay-samagra to meeting invite with OCI team on this today at 12 pm.

[abhay-samagra]

@suresh12, @karntrehan, @surabhi-mahawar - For your reference.

[ddevadat]

For the db issue the resolution given was to set the below value in the orchestrator-config configmap

CASSANDRA_MIGRATION_COUNT=6

After this orchestrator pod comes up, but now its throwing some other error like below

e=Doubt%20Solving%20Bot&match=true
		at org.springframework.web.reactive.function.client.WebClientResponseException.create(WebClientResponseException.java:196) ~[spring-webflux-5.3.13.jar!/:5.3.13]
		Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
	*__checkpoint ⇢ 400 from GET http://uci-service.production.svc.cluster.local:9999/admin/v1/bot/search/?name=Doubt%20Solving%20Bot&match=true [DefaultWebClient]
Original Stack Trace:
			at org.springframework.web.reactive.function.client.WebClientResponseException.create(WebClientResponseException.java:196)
			at org.springframework.web.reactive.function.client.DefaultClientResponse.lambda$createException$1(DefaultClientResponse.java:207)
			at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:106)
			at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
			at reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:101)
			at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:127)
			at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
			at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:295)
			at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337)
			at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
			at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:159)
			at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:142)
			at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)
			at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:142)
			at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:400)
			at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:419)
			at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:473)
			at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:702)
			at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
			at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
			at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
			at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
			at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
			at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
			at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
			at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
			at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
			at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
			at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
			at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
			at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
			at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
			at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
			at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
			at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
			at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
			at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
			at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
			at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
			at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
			at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
			at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
			at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
			at java.base/java.lang.Thread.run(Thread.java:835)

[ddevadat]

after rebuilding uci from
https://github.com/ocisunbird/uci-apis.git
branch: release-5.0.0-postgres-ssl-latest

redeployed uci
able to do the search command

[Krishnaj20]

@ddevadat @chitranshu-keshav - is this issue resolved? Pls confirm.

[ddevadat]

this issue is fixed