SBC cluster nodes don't bootstrap

[fsgh42]

continued from https://taloscommunity.slack.com/archives/C04D4PDAJT0/p1738879422196599

recap:

• I selfhost both omni and imagefactory
• I want to deploy an omni backed talos cluster across multiple providers: hetzner, a local VM on libvirt/qemu and a radxa SBC (rock4se)
• cluster is created via clustertemplate and shows up in omni, machines are added to it:
  • the hetzner VM is the controlplane and boots fine, joins cluster, healthy status
  • the libvirt VM is a worker and boots fine, joins cluster, healthy status
  • the radxa SBC gets stuck during boot

I can use kubectl to access the kubernetes API. All Pods are up and running, but the third node never shows up.

I noticed that once a machine joins the cluster, that (on all three machines) as part of an upgrade sequence talos for some reason attempts to pull installer images that I did not explicitly set (they have image IDs unknown to me).
As discussed in slack, I think those unknown image IDs are the customization.systemExtensions and overlay sections of the original schematic.
Initially, I saw log lines from talos indicating that those images don't exist - which makes sense - I never registered those schemas. So with the help or Artem I was able to brute force their data origins and registered them with my (selfhosted) registry, which made that error message go away.

I am now at the point, where the rock4se SBC does not report any errors, but is stuck somehow. It checks in with omni and gets added to the desired cluster, though. It just never reaches ready state like the other nodes. These are the last logs I see before it reports a timeout and reboots:

18/01/2013 09:50:41
[talos] phase config (9/10): failed
18/01/2013 09:50:41
[talos] initialize sequence: failed
18/01/2013 09:50:41
[talos] maintenanceUpgrade sequence: 5 phase(s)
18/01/2013 09:50:41
[talos] phase upgrade (1/5): 1 tasks(s)
18/01/2013 09:50:41
[talos] task upgrade (1/1): starting
18/01/2013 09:50:41
[talos] task upgrade (1/1): performing upgrade via "if.xfx1.de/installer/5bea1ff0359b26d60605d1bfa3f826d531f0d8c65ca0d489c3c72f2163f8ab91:v1.9.3"
18/01/2013 09:50:41
[talos] pulling "if.xfx1.de/installer/5bea1ff0359b26d60605d1bfa3f826d531f0d8c65ca0d489c3c72f2163f8ab91:v1.9.3" 
...
18/01/2013 10:10:41
[talos] task upgrade (1/1): failed: 1 error(s) occurred:
18/01/2013 10:10:41
timeout
18/01/2013 10:10:42
[talos] phase upgrade (1/5): failed
18/01/2013 10:10:42
[talos] maintenanceUpgrade sequence: failed
18/01/2013 10:10:42
[talos] upgrade failed: error running phase 1 in maintenanceUpgrade sequence: task 1/1: failed, 1 error(s) occurred:
18/01/2013 10:10:42
kauditd_printk_skb: 29 callbacks suppressed
18/01/2013 10:10:42
audit: type=1305 audit(1739014808.056:42): op=set audit_enabled=0 old=1 auid=4294967295 ses=4294967295 subj=kernel res=1
18/01/2013 10:10:42
timeout 

I am not sure how to further debug this.
Looking at my instance of imagefactory I can see that there is no connection attempt. I earlier argued that this is a network problem, but both the SBC and the libvirt VM are exposed to the same network (the latter by macvtap) and get IPs from the same pools.

Any pointers are appreciated.

[Unix4ever]

It looks like network connectivity problem.
As another option you can try using the standard image factory and see if the issue persists.

[fsgh42]

I tried that and it still gets stuck at that log message every time.
I don't think it is something with the imagefactory, that is just the last message that sticks out.
All other hosts I tried on the same physical network, VM or metal, boot just fine, using the same imagefactory, so this problem is specific to this SBC.

Omni logs show this:

omni-1  | {"level":"error","ts":1739210784.357962,"caller":"task/task.go:82","msg":"restarting task","component":"omni_runtime","controller":"MachineStatusController","namespace":"default","type":"Machines.omni.sidero.dev","id":"32373632-3965-3337-3035-313238666639","job":"reconcile","task":"32373632-3965-3337-3035-313238666639","interval":5.486052491,"error":"failed to list resource definitions: rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority\""}

If I understand correctly, this means the node's talos API certificates are not set up correctly (yet). The machine is visible in omni though. Could it be that at the time they are created, the SBCs RTC is not synced yet and the "not valid after" field of the certificate is set to something that would, for a machine with a working RTC, still be in the past?
The SBC issuing its certificate and RTC sync are just one second apart from one another, I can see that in the logs:

18/01/2013 09:50:37 [talos] adjusting time (jump) by 105752h53m10.545319508s via 162.159.200.123, state TIME_OK, status STA_NANO {"component": "controller-runtime", "controller": "time.SyncController"} 
18/01/2013 09:50:37 [talos] synchronized RTC with system clock {"component": "controller-runtime", "controller": "time.SyncController"}
18/01/2013 09:50:38 [talos] configuring siderolink connection {"component": "controller-runtime", "controller": "siderolink.ManagerController", "peer_endpoint": "[fdae:41e4:649b:9304:f357:9044:b75b:b11]:50889", "next_peer_endpoint": ""} 
18/01/2013 09:50:38 [talos] siderolink connection configured ...
...
18/01/2013 09:50:38 [talos] server certificate issued {"component": "controller-runtime", "controller": "runtime.MaintenanceServiceController", "fingerprint": "txdePGiNuVHi+bUuNpueYMPNUUQY9NGJaaDbWJ54oCQ="} 

[Unix4ever]

I have an update for you: we actually found the bug in Talos 1.9.
Version 1.9.0-1.9.3 are affected. The fix will land in 1.9.4.

[fsgh42]

Thank you! I was about to go crazy here.
I was able to narrow this down to arm64 arch: when creating a VM based on qemu/aarch64, it suffers the same problems. qemu/amd64 works fine, though. Can you provide a reference? I'd be interested in what triggered that.

[Unix4ever]

That's the issue #919

The bug was first introduced in 1.9. The containerd is misconfigured in the maintenance mode so it fails to pull the images.
We will try to fix the issue from both Omni side and Talos side.
As Talos fix will land only in the next backport release.

[fsgh42]

Thank you! I'll wait for talos v1.9.4 and report if that fixes my issue.
ll also check if 1.8.x works in the meantime.

[utkuozdemir]

1.8.x is not affected by the issue, it should work fine :)

[fsgh42]

indeed, it does 👍