L2 Isolation with VXLAN - Virtual Router fails to start

[thanasiskalos]

problem

Hello

On a cloudstack environment I have 2 hosts KVM and one CLoudstack Management on Cisco UCS. Each KVM has 5 Physical Interfaces with following assignments :

cloudbr0 Management enp8s0
cloudbr1 storage enp10s0
cloudbr2 GUEST / VXLAN enp12s0
cloudbr3 GUEST / VLAN enp11s0
cloudbr4 Public enp13s0

The Guest L2 isolation with VLAN works fine. The issue is for L2 Isolation with VxLAN. From my findings the issue lies on the boot / starting of the Virtual router assigned for the L2 Isolated traffic with Source NAT.

On cloudstack logs I can see following error :

2025-01-28 10:30:48,556 WARN [o.a.c.m.w.WebhookServiceImpl] (API-Job-Executor-62:[ctx-999fb26a, job-731, ctx-0a0f21b0]) (logid:24c44654) Skipping delivering event [ID: null, description: {"details":"Router Id: aae0418b-27f6-4b3b-a443-435373c9798c","event":"ROUTER.START","status":"Completed"}] to any webhook as account ID is missing
2025-01-28 10:30:48,561 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-62:[ctx-999fb26a, job-731]) (logid:24c44654) Unexpected exception while executing org.apache.cloudstack.api.command.admin.router.StartRouterCmd com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Unable to start instance due to Unable to start VM:aae0418b-27f6-4b3b-a443-435373c9798c due to error in finalizeStart, not retrying
Caused by: com.cloud.utils.exception.ExecutionException: Unable to start VM:aae0418b-27f6-4b3b-a443-435373c9798c due to error in finalizeStart, not retrying
2025-01-28 10:30:48,561 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-62:[ctx-999fb26a, job-731]) (logid:24c44654) Complete async job-731, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Resource [Host:1] is unreachable: Host 1: Unable to start instance due to Unable to start VM:aae0418b-27f6-4b3b-a443-435373c9798c due to error in finalizeStart, not retrying"}

2025-01-28 10:30:49,622 INFO [c.c.a.ApiServlet] (qtp1513608173-3590:[ctx-c69f5e49, ctx-0f588a43]) (logid:7b92e8a2) (userId=2 accountId=2 sessionId=node014urk8gp4cwaq77ikzdwi15hj4) 10.200.54.123 -- GET jobId=24c44654-1cd3-4214-a847-364d04ecf450&command=queryAsyncJobResult&response=json&sessionkey=_9Ti7ZhqYfOlALzSBX9PBY0BbnI 200 {"queryasyncjobresultresponse":{"accountid":"2e416228-d996-11ef-8240-e0071bd1d7d0","account":"admin","domainid":"dd511c8f-d995-11ef-8240-e0071bd1d7d0","domainpath":"ROOT","userid":"2e42450d-d996-11ef-8240-e0071bd1d7d0","cmd":"org.apache.cloudstack.api.command.admin.router.StartRouterCmd","jobstatus":2,"jobprocstatus":0,"jobresultcode":530,"jobresulttype":"object","jobresult":{"errorcode":530,"errortext":"Resource [Host:1] is unreachable: Host 1: Unable to start instance due to Unable to start VM:aae0418b-27f6-4b3b-a443-435373c9798c due to error in finalizeStart, not retrying"},"jobinstancetype":"DomainRouter","jobinstanceid":"aae0418b-27f6-4b3b-a443-435373c9798c","created":"2025-01-28T10:30:13+0200","completed":"2025-01-28T10:30:48+0200","jobid":"24c44654-1cd3-4214-a847-364d04ecf450"}}
2025-01-28 10:30:49,798 DEBUG [c.c.a.ApiServlet] (qtp1513608173-3755:[ctx-76482e37]) (logid:4add14b0) ===START=== 10.200.54.123 -- GET listall=true&projectid=-1&id=aae0418b-27f6-4b3b-a443-435373c9798c&page=1&pagesize=20&command=listRouters&response=json&sessionkey=_9Ti7ZhqYfOlALzSBX9PBY0BbnI

Any advice appreciated

versions

cloudstack : 4.20.0

KVM Hosts : Rocky Linux 8.10

Linux Bridge

The steps to reproduce the bug

Cloustack with 5 Physical Network Interfaces.
2 x Guest Interfaces
1 x VLAN Isolation on a Physical Interface
1 x VxLAN Isolation on a Physical Interface

What to do about it?

No response

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[thanasiskalos]

Further Logs from the KVM host

/var/log/cloudstack/agent/agent.log

2025-01-28 12:13:26,985 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:[]) (logid:4c5ab8a5) Trying to fetch storage pool 0c3cae2d-85a4-3870-b64d-2d2b7061f9c8 from libvirt
2025-01-28 12:13:27,063 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:[]) (logid:4c5ab8a5) Trying to fetch storage pool 0c3cae2d-85a4-3870-b64d-2d2b7061f9c8 from libvirt
2025-01-28 12:13:27,313 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-2:[]) (logid:4c5ab8a5) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy' is not available. Transformation
s will not be applied.
2025-01-28 12:13:27,313 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-2:[]) (logid:4c5ab8a5) Groovy scripting engine is not initialized. Data transformation skipped.
2025-01-28 12:13:28,266 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-2:[]) (logid:4c5ab8a5) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not available. Transformations w
ill not be applied.
2025-01-28 12:13:28,267 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-2:[]) (logid:4c5ab8a5) Groovy scripting engine is not initialized. Data transformation skipped.
2025-01-28 12:13:54,437 WARN [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-2:[]) (logid:4c5ab8a5) Execution of process [64156] for command [/usr/share/cloudstack-common/scripts/network/domr/router_proxy.
sh get_template_version.sh 169.254.78.45 ] failed.
2025-01-28 12:13:54,438 WARN [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-2:[]) (logid:4c5ab8a5) Exception [Stream closed] occurred when attempting to run command [/usr/share/cloudstack-common/scripts/n
etwork/domr/router_proxy.sh get_template_version.sh 169.254.78.45 ]. java.io.IOException: Stream closed
at java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:168)
at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:287)
at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:330)
at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:190)
at java.base/java.io.InputStreamReader.read(InputStreamReader.java:177)
at java.base/java.io.BufferedReader.fill(BufferedReader.java:162)
at java.base/java.io.BufferedReader.readLine(BufferedReader.java:329)
at java.base/java.io.BufferedReader.readLine(BufferedReader.java:396)
at com.cloud.utils.script.OutputInterpreter.processError(OutputInterpreter.java:41)
at com.cloud.utils.script.Script.execute(Script.java:314)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:553)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:541)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.execute(VirtualRoutingResource.java:496)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeQueryCommand(VirtualRoutingResource.java:215)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:129)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:35)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:29)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1945)
at com.cloud.agent.Agent.processRequest(Agent.java:686)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1109)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)