AMQP Integration | Messages Not Getting Published : 4.20.0.0

[midhunpjos]

ISSUE TYPE

• Bug Report

COMPONENT NAME

AMQP

CLOUDSTACK VERSION

4.20.0.0

CONFIGURATION

Advanced Zone -

OS / ENVIRONMENT

Cloudstack Over Ubuntu 22.04

SUMMARY

Post Upgrade to 4.20.0.0 Messages are not getting Published to AMQP server. The Configuration was working fine for 4.19.1.3

STEPS TO REPRODUCE

Rabbit MQ server is configured as per documentation
https://docs.cloudstack.apache.org/en/latest/adminguide/events.html#amqp-configuration
I can see successful connections to RabbitMQ server, However Event Messages are not getting delivered to RabbitMQ server 
Logs only below warning is printed

2024-12-17 17:11:29,345 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (Work-Job-Executor-3:[ctx-c572a287, job-5493/job-5494]) (logid:85a9626c) Done executing com.cloud.vm.VmWorkStart for job-5494
2024-12-17 17:11:29,346 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-3:[ctx-c572a287, job-5493/job-5494]) (logid:85a9626c) Remove job-5494 from job monitoring
2024-12-17 17:11:29,357 WARN  [o.a.c.m.w.WebhookServiceImpl] (API-Job-Executor-4:[ctx-8ec9aabd, job-5493, ctx-0af47dea]) (logid:85a9626c) Skipping delivering event [ID: null, description: {"details":"Vm Id: 2fba756b-5c49-4f83-8a8a-df58c100356e","event":"VM.START","status":"Completed"}] to any webhook as account ID is missing
2024-12-17 17:11:29,357 WARN  [o.a.c.f.e.EventDistributorImpl] (API-Job-Executor-4:[ctx-8ec9aabd, job-5493, ctx-0af47dea]) (logid:85a9626c) Failed to publish event [category: ActionEvent, type: VM.START] on bus webhookEventBus

EXPECTED RESULTS

ACTUAL RESULTS

 Event Messages are not getting delivered to RabbitMQ server 

[shwstppr]

@midhunpjos Based on the code the above logs warnings shouldn't affect publishing on AMQP as they are just for the internal webhook bus.
Do you also get a warning like Failed to publish event ... on bus RabbitMQ or similar?

[midhunpjos]

@shwstppr No Such errors are noted. Only below is observed to "webhookEventBus"

2024-12-17 13:38:20,257 WARN [o.a.c.f.e.EventDistributorImpl] (API-Job-Executor-1:[ctx-783f33de, job-5294, ctx-20e8c097]) (logid:a9c95eb8) Failed to publish event [category: ActionEvent, type: DOMAIN.DELETE] on bus webhookEventBus
2024-12-17 13:39:39,380 WARN [o.a.c.f.e.EventDistributorImpl] (qtp1845623216-28:[ctx-dfd62cbd, ctx-6665cd71]) (logid:e406f3c2) Failed to publish event [category: ActionEvent, type: ACCOUNT.UPDATE] on bus webhookEventBus
2024-12-17 13:40:26,818 WARN [o.a.c.f.e.EventDistributorImpl] (qtp1845623216-28:[ctx-f0540b0b, ctx-bfbbd97c]) (logid:7cac820e) Failed to publish event [category: ActionEvent, type: DOMAIN.UPDATE] on bus webhookEventBus
2024-12-17 14:45:24,075 WARN [o.a.c.f.e.EventDistributorImpl] (API-Job-Executor-2:[ctx-389322c3, job-5305, ctx-6bc27d39]) (logid:08254309) Failed to publish event [category: ActionEvent, type: VM.STOP] on bus webhookEventBus
2024-12-17 14:45:56,455 WARN [o.a.c.f.e.EventDistributorImpl] (API-Job-Executor-3:[ctx-ca8648fb, job-5307, ctx-e34a8c0c, ctx-cca6e98a]) (logid:9329b5aa) Failed to publish event [category: ActionEvent, type: VOLUME.DESTROY] on bus webhookEventBus
2024-12-17 14:45:56,880 WARN [o.a.c.f.e.EventDistributorImpl] (API-Job-Executor-3:[ctx-ca8648fb, job-5307, ctx-e34a8c0c]) (logid:9329b5aa) Failed to publish event [category: ActionEvent, type: FIREWALL.CLOSE] on bus webhookEventBus
2024-12-17 14:45:56,925 WARN [o.a.c.f.e.EventDistributorImpl] (API-Job-Executor-3:[ctx-ca8648fb, job-5307, ctx-e34a8c0c]) (logid:9329b5aa) Failed to publish event [category: ActionEvent, type: VM.DESTROY] on bus webhookEventBus
2024-12-17 17:11:29,357 WARN [o.a.c.f.e.EventDistributorImpl] (API-Job-Executor-4:[ctx-8ec9aabd, job-5493, ctx-0af47dea]) (logid:85a9626c) Failed to publish event [category: ActionEvent, type: VM.START] on bus webhookEventBus

[shwstppr]

Can you please check logs when you started MS to find a long message like,
Found x event buses ...
This should help identify what all event buses were discovered by MS. Either it is not correctly discovering the Rabbit MQ bus or some issue with the RabbitMQ bus code. Though there should be some log in the latter case.

[midhunpjos]

@shwstppr I am unable to find any logs related to "event buses". searching RabbitMQ Found below:

2024-12-17 17:37:26,941 INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean RabbitMQEventBus.
2024-12-17 17:37:31,975 INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean RabbitMQEventBus.

I can see successful logins to RabbitMQ Node, Further The is same is tested in Multiple environments and similar issue is noted there.

[shwstppr]

@midhunpjos checked later, that Found x event buses... is a TRACE level log.
Will try to reproduce this at my end

[midhunpjos]

@shwstppr Noted. Trace Mode resulted in below log
root@acs01-gul:/etc/cloudstack/management# less /var/log/cloudstack/management/management-server.log |grep -i 'event buses'
2024-12-18 11:34:51,466 TRACE [o.a.c.f.e.EventDistributorImpl] (main:[]) (logid:) Found 0 event buses :

[shwstppr]

@midhunpjos can you please try moving your RbbitMQ spring-event-bus-context.xml file from /etc/cloudstack/management/META-INF/cloudstack/core/ to /etc/cloudstack/management/META-INF/cloudstack/event/?

If using encrypted credentials for the connection then cred.properties file as well (if present)

[midhunpjos]

@midhunpjos can you please try moving your RbbitMQ spring-event-bus-context.xml file from /etc/cloudstack/management/META-INF/cloudstack/core/ to /etc/cloudstack/management/META-INF/cloudstack/event/?

If using encrypted credentials for the connection then cred.properties file as well (if present)

@shwstppr Thanks for the update. Everything is back to normal after making the changes.