expected number of pods for kube-apiserver to be 3, got 0, but everything is running?

[runningman84]

My test cluster is a it strange I can use kubectl to access the api, the staticpods are ready but talosctl health report that the api server is not ready. talosctl upgrade-k8s is also stuck because the kube api server is not ready....

➜  talos talosctl -n 10.0.30.81  get staticpods        
NODE         NAMESPACE   TYPE        ID                        VERSION
10.0.30.81   k8s         StaticPod   kube-apiserver            1
10.0.30.81   k8s         StaticPod   kube-controller-manager   1
10.0.30.81   k8s         StaticPod   kube-scheduler            1
➜  talos talosctl -n 10.0.30.81  get staticpodstatus   
NODE         NAMESPACE   TYPE              ID                                            VERSION   READY
10.0.30.81   k8s         StaticPodStatus   kube-system/kube-apiserver-master1            1         True
10.0.30.81   k8s         StaticPodStatus   kube-system/kube-controller-manager-master1   1         True
10.0.30.81   k8s         StaticPodStatus   kube-system/kube-scheduler-master1            1         True
➜  talos talosctl -n 10.0.30.81  get manifests         
NODE         NAMESPACE      TYPE       ID                                                                                                                  VERSION
10.0.30.81   controlplane   Manifest   00-kubelet-bootstrapping-token                                                                                      1
10.0.30.81   controlplane   Manifest   01-csr-approver-role-binding                                                                                        1
10.0.30.81   controlplane   Manifest   01-csr-node-bootstrap                                                                                               1
10.0.30.81   controlplane   Manifest   01-csr-renewal-role-binding                                                                                         1
10.0.30.81   controlplane   Manifest   05-flannel                                                                                                          1
10.0.30.81   controlplane   Manifest   10-kube-proxy                                                                                                       1
10.0.30.81   controlplane   Manifest   11-core-dns                                                                                                         1
10.0.30.81   controlplane   Manifest   11-core-dns-svc                                                                                                     1
10.0.30.81   controlplane   Manifest   11-kube-config-in-cluster                                                                                           1
10.0.30.81   controlplane   Manifest   99-https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml                       1
10.0.30.81   controlplane   Manifest   99-https://raw.githubusercontent.com/alex1989hu/kubelet-serving-cert-approver/main/deploy/standalone-install.yaml   1
➜  talos 
➜  talos talosctl -n 10.0.30.81 health                 
discovered nodes: ["10.0.30.80" "10.0.30.82" "10.0.30.83"]
waiting for etcd to be healthy: ...
waiting for etcd to be healthy: OK
waiting for etcd members to be consistent across nodes: ...
waiting for etcd members to be consistent across nodes: OK
waiting for etcd members to be control plane nodes: ...
waiting for etcd members to be control plane nodes: OK
waiting for apid to be ready: ...
waiting for apid to be ready: OK
waiting for all nodes memory sizes: ...
waiting for all nodes memory sizes: OK
waiting for all nodes disk sizes: ...
waiting for all nodes disk sizes: OK
waiting for kubelet to be healthy: ...
waiting for kubelet to be healthy: OK
waiting for all nodes to finish boot sequence: ...
waiting for all nodes to finish boot sequence: OK
waiting for all k8s nodes to report: ...
waiting for all k8s nodes to report: OK
waiting for all k8s nodes to report ready: ...
waiting for all k8s nodes to report ready: OK
waiting for all control plane static pods to be running: ...
waiting for all control plane static pods to be running: OK
waiting for all control plane components to be ready: ...
waiting for all control plane components to be ready: expected number of pods for kube-apiserver to be 3, got 0

[smira]

The checks are done via kubectl get pods -n kube-system, so you need to start looking there.

[runningman84]

the strange thing is that there are no kube-apiserver pods in the kube-system ns:

➜  talos kubectl get pods -n kube-system
NAME                                     READY   STATUS                   RESTARTS       AGE
coredns-85b955d87b-464rw                 1/1     Running                  0              11h
coredns-85b955d87b-88x5w                 1/1     Running                  0              11h
coredns-85b955d87b-fbdvt                 0/1     Completed                2              2d14h
coredns-85b955d87b-h6j9n                 0/1     Completed                0              21h
coredns-85b955d87b-k9q9g                 0/1     Completed                0              21h
coredns-85b955d87b-plnc6                 1/1     Running                  1 (12h ago)    12h
coredns-85b955d87b-rlhv9                 0/1     Completed                0              2d17h
coredns-85b955d87b-vh5m7                 0/1     Completed                0              12h
coredns-85b955d87b-wf5mm                 0/1     Completed                0              12h
descheduler-28519800-4hszx               0/1     Completed                0              2m48s
external-dns-89b8dcd4b-dwjb8             1/1     Running                  2 (56m ago)    78m
kube-controller-manager-master1          1/1     Running                  1 (79m ago)    79m
kube-controller-manager-master2          1/1     Running                  2 (79m ago)    78m
kube-controller-manager-master3          1/1     Running                  0              74m
kube-flannel-27psk                       1/1     Running                  8 (63m ago)    78m
kube-flannel-r8wnp                       1/1     Running                  8 (63m ago)    78m
kube-flannel-txkd5                       1/1     Running                  20 (60m ago)   11h
kube-proxy-j56dq                         1/1     Running                  4 (11h ago)    12h
kube-proxy-k4gql                         1/1     Running                  0              78m
kube-proxy-mq5w6                         1/1     Running                  0              78m
kube-scheduler-master1                   1/1     Running                  1 (79m ago)    79m
kube-scheduler-master2                   1/1     Running                  2 (79m ago)    78m
kube-scheduler-master3                   1/1     Running                  0              74m
local-path-provisioner-8648b54f9-lqlwg   1/1     Running                  1 (12h ago)    12h
metrics-server-6db4d75b97-4hdn8          1/1     Running                  0              11h
metrics-server-6db4d75b97-4pbhq          0/1     Completed                1              2d14h
metrics-server-6db4d75b97-cnbn6          0/1     Completed                0              22h
metrics-server-6db4d75b97-h4jvc          0/1     Completed                0              17h
metrics-server-6db4d75b97-m6fnt          0/1     ContainerStatusUnknown   0              12h
➜  talos kubectl get events -n kube-system
LAST SEEN   TYPE      REASON                    OBJECT                                       MESSAGE
56m         Normal    NoSourceArtifact          helmchart/authelia-authelia                  no artifact available for HelmRepository source 'authelia'
55m         Normal    ChartPullSucceeded        helmchart/authelia-authelia                  pulled 'authelia' chart with version '0.8.58'
57m         Warning   Failed                    helmrepository/authelia                      failed to fetch Helm repository index: failed to cache index to temporary file: Get "https://charts.authelia.com/index.yaml": dial tcp: lookup charts.authelia.com on 10.96.0.10:53: read udp 10.244.0.71:48409->10.96.0.10:53: read: connection refused
55m         Normal    NewArtifact               helmrepository/authelia                      stored fetched index of size 208kB from 'https://charts.authelia.com'
58m         Normal    LeaderElection            lease/cert-manager-controller                cert-manager-96c6895cd-tsw9l-external-cert-manager-controller became leader
60m         Warning   FailedCreatePodSandBox    pod/coredns-85b955d87b-464rw                 (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "cda5b4f8726cee7a2dc0cc74ed3a30e0e8ab80b55ce86f7c27539cdea7dc3518": plugin type="flannel" failed (add): loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no such file or directory
60m         Warning   FailedCreatePodSandBox    pod/coredns-85b955d87b-88x5w                 (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "495afe21aa2e0b05f74bf014219be9b529a6c80c830c8511a4857530e9772f34": plugin type="flannel" failed (add): loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no such file or directory
60m         Normal    SandboxChanged            pod/coredns-85b955d87b-plnc6                 Pod sandbox changed, it will be killed and re-created.
58m         Warning   FailedGetResourceMetric   horizontalpodautoscaler/coredns              failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)
58m         Normal    Pulled                    pod/descheduler-28519080-6xdgs               Container image "registry.k8s.io/descheduler/descheduler:v0.27.1" already present on machine
58m         Normal    Completed                 job/descheduler-28519080                     Job completed
58m         Normal    Scheduled                 pod/descheduler-28519740-n6rfn               Successfully assigned kube-system/descheduler-28519740-n6rfn to master2
58m         Normal    Pulled                    pod/descheduler-28519740-n6rfn               Container image "registry.k8s.io/descheduler/descheduler:v0.27.1" already present on machine
58m         Normal    Created                   pod/descheduler-28519740-n6rfn               Created container descheduler
58m         Normal    Started                   pod/descheduler-28519740-n6rfn               Started container descheduler
58m         Normal    SuccessfulCreate          job/descheduler-28519740                     Created pod: descheduler-28519740-n6rfn
58m         Normal    Completed                 job/descheduler-28519740                     Job completed
58m         Normal    Scheduled                 pod/descheduler-28519745-nhs8m               Successfully assigned kube-system/descheduler-28519745-nhs8m to master2
58m         Normal    Pulled                    pod/descheduler-28519745-nhs8m               Container image "registry.k8s.io/descheduler/descheduler:v0.27.1" already present on machine
58m         Normal    Created                   pod/descheduler-28519745-nhs8m               Created container descheduler
58m         Normal    Started                   pod/descheduler-28519745-nhs8m               Started container descheduler
58m         Normal    SuccessfulCreate          job/descheduler-28519745                     Created pod: descheduler-28519745-nhs8m
57m         Normal    Completed                 job/descheduler-28519745                     Job completed
53m         Normal    Scheduled                 pod/descheduler-28519750-rt2zj               Successfully assigned kube-system/descheduler-28519750-rt2zj to master2
53m         Normal    Pulled                    pod/descheduler-28519750-rt2zj               Container image "registry.k8s.io/descheduler/descheduler:v0.27.1" already present on machine
53m         Normal    Created                   pod/descheduler-28519750-rt2zj               Created container descheduler
53m         Normal    Started                   pod/descheduler-28519750-rt2zj               Started container descheduler
53m         Normal    SuccessfulCreate          job/descheduler-28519750                     Created pod: descheduler-28519750-rt2zj
52m         Normal    Completed                 job/descheduler-28519750                     Job completed
48m         Normal    Scheduled                 pod/descheduler-28519755-hqld5               Successfully assigned kube-system/descheduler-28519755-hqld5 to master2
48m         Normal    Pulled                    pod/descheduler-28519755-hqld5               Container image "registry.k8s.io/descheduler/descheduler:v0.27.1" already present on machine
48m         Normal    Created                   pod/descheduler-28519755-hqld5               Created container descheduler
48m         Normal    Started                   pod/descheduler-28519755-hqld5               Started container descheduler
48m         Normal    SuccessfulCreate          job/descheduler-28519755                     Created pod: descheduler-28519755-hqld5
48m         Normal    Completed                 job/descheduler-28519755                     Job completed
43m         Normal    Scheduled                 pod/descheduler-28519760-q4m7f               Successfully assigned kube-system/descheduler-28519760-q4m7f to master2
43m         Normal    Pulled                    pod/descheduler-28519760-q4m7f               Container image "registry.k8s.io/descheduler/descheduler:v0.27.1" already present on machine
43m         Normal    Created                   pod/descheduler-28519760-q4m7f               Created container descheduler
43m         Normal    Started                   pod/descheduler-28519760-q4m7f               Started container descheduler
43m         Normal    SuccessfulCreate          job/descheduler-28519760                     Created pod: descheduler-28519760-q4m7f
42m         Normal    Completed                 job/descheduler-28519760                     Job completed
38m         Normal    Scheduled                 pod/descheduler-28519765-b77g8               Successfully assigned kube-system/descheduler-28519765-b77g8 to master2
38m         Normal    Pulled                    pod/descheduler-28519765-b77g8               Container image "registry.k8s.io/descheduler/descheduler:v0.27.1" already present on machine
38m         Normal    Created                   pod/descheduler-28519765-b77g8               Created container descheduler
38m         Normal    Started                   pod/descheduler-28519765-b77g8               Started container descheduler
38m         Normal    SuccessfulCreate          job/descheduler-28519765                     Created pod: descheduler-28519765-b77g8
37m         Normal    Completed                 job/descheduler-28519765                     Job completed
33m         Normal    Scheduled                 pod/descheduler-28519770-z999z               Successfully assigned kube-system/descheduler-28519770-z999z to master2
33m         Normal    Pulled                    pod/descheduler-28519770-z999z               Container image "registry.k8s.io/descheduler/descheduler:v0.27.1" already present on machine
33m         Normal    Created                   pod/descheduler-28519770-z999z               Created container descheduler
33m         Normal    Started                   pod/descheduler-28519770-z999z               Started container descheduler
33m         Normal    SuccessfulCreate          job/descheduler-28519770                     Created pod: descheduler-28519770-z999z
32m         Normal    Completed                 job/descheduler-28519770                     Job completed
28m         Normal    Scheduled                 pod/descheduler-28519775-whdpp               Successfully assigned kube-system/descheduler-28519775-whdpp to master2
28m         Normal    Pulled                    pod/descheduler-28519775-whdpp               Container image "registry.k8s.io/descheduler/descheduler:v0.27.1" already present on machine
28m         Normal    Created                   pod/descheduler-28519775-whdpp               Created container descheduler
28m         Normal    Started                   pod/descheduler-28519775-whdpp               Started container descheduler
28m         Normal    SuccessfulCreate          job/descheduler-28519775                     Created pod: descheduler-28519775-whdpp
27m         Normal    Completed                 job/descheduler-28519775                     Job completed
23m         Normal    Scheduled                 pod/descheduler-28519780-22bsd               Successfully assigned kube-system/descheduler-28519780-22bsd to master2
23m         Normal    Pulled                    pod/descheduler-28519780-22bsd               Container image "registry.k8s.io/descheduler/descheduler:v0.27.1" already present on machine
23m         Normal    Created                   pod/descheduler-28519780-22bsd               Created container descheduler
23m         Normal    Started                   pod/descheduler-28519780-22bsd               Started container descheduler
23m         Normal    SuccessfulCreate          job/descheduler-28519780                     Created pod: descheduler-28519780-22bsd
22m         Normal    Completed                 job/descheduler-28519780                     Job completed
18m         Normal    Scheduled                 pod/descheduler-28519785-jtj2x               Successfully assigned kube-system/descheduler-28519785-jtj2x to master2
18m         Normal    Pulled                    pod/descheduler-28519785-jtj2x               Container image "registry.k8s.io/descheduler/descheduler:v0.27.1" already present on machine
18m         Normal    Created                   pod/descheduler-28519785-jtj2x               Created container descheduler
18m         Normal    Started                   pod/descheduler-28519785-jtj2x               Started container descheduler
18m         Normal    SuccessfulCreate          job/descheduler-28519785                     Created pod: descheduler-28519785-jtj2x
17m         Normal    Completed                 job/descheduler-28519785                     Job completed
13m         Normal    Scheduled                 pod/descheduler-28519790-9r9th               Successfully assigned kube-system/descheduler-28519790-9r9th to master2
13m         Normal    Pulled                    pod/descheduler-28519790-9r9th               Container image "registry.k8s.io/descheduler/descheduler:v0.27.1" already present on machine
13m         Normal    Created                   pod/descheduler-28519790-9r9th               Created container descheduler
13m         Normal    Started                   pod/descheduler-28519790-9r9th               Started container descheduler
13m         Normal    SuccessfulCreate          job/descheduler-28519790                     Created pod: descheduler-28519790-9r9th
12m         Normal    Completed                 job/descheduler-28519790                     Job completed
8m3s        Normal    Scheduled                 pod/descheduler-28519795-nfnlq               Successfully assigned kube-system/descheduler-28519795-nfnlq to master2
8m3s        Normal    Pulled                    pod/descheduler-28519795-nfnlq               Container image "registry.k8s.io/descheduler/descheduler:v0.27.1" already present on machine
8m3s        Normal    Created                   pod/descheduler-28519795-nfnlq               Created container descheduler
8m3s        Normal    Started                   pod/descheduler-28519795-nfnlq               Started container descheduler
8m4s        Normal    SuccessfulCreate          job/descheduler-28519795                     Created pod: descheduler-28519795-nfnlq
7m59s       Normal    Completed                 job/descheduler-28519795                     Job completed
3m3s        Normal    Scheduled                 pod/descheduler-28519800-4hszx               Successfully assigned kube-system/descheduler-28519800-4hszx to master2
3m3s        Normal    Pulled                    pod/descheduler-28519800-4hszx               Container image "registry.k8s.io/descheduler/descheduler:v0.27.1" already present on machine
3m3s        Normal    Created                   pod/descheduler-28519800-4hszx               Created container descheduler
3m3s        Normal    Started                   pod/descheduler-28519800-4hszx               Started container descheduler
3m4s        Normal    SuccessfulCreate          job/descheduler-28519800                     Created pod: descheduler-28519800-4hszx
2m59s       Normal    Completed                 job/descheduler-28519800                     Job completed
58m         Warning   TooManyMissedTimes        cronjob/descheduler                          too many missed start times. Set or decrease .spec.startingDeadlineSeconds or check clock skew
58m         Normal    JobAlreadyActive          cronjob/descheduler                          Not starting job because prior execution is running and concurrency policy is Forbid
58m         Normal    SuccessfulDelete          cronjob/descheduler                          Deleted job descheduler-28519075
58m         Normal    SawCompletedJob           cronjob/descheduler                          Saw completed job: descheduler-28519080, status: Complete
58m         Normal    SuccessfulCreate          cronjob/descheduler                          Created job descheduler-28519740
58m         Normal    SuccessfulDelete          cronjob/descheduler                          Deleted job descheduler-28519080
58m         Normal    SawCompletedJob           cronjob/descheduler                          Saw completed job: descheduler-28519740, status: Complete
58m         Normal    SuccessfulCreate          cronjob/descheduler                          Created job descheduler-28519745
57m         Normal    SuccessfulDelete          cronjob/descheduler                          Deleted job descheduler-28519740
57m         Normal    SawCompletedJob           cronjob/descheduler                          Saw completed job: descheduler-28519745, status: Complete
53m         Normal    SuccessfulCreate          cronjob/descheduler                          Created job descheduler-28519750
48m         Normal    SuccessfulCreate          cronjob/descheduler                          Created job descheduler-28519755
43m         Normal    SuccessfulCreate          cronjob/descheduler                          Created job descheduler-28519760
38m         Normal    SuccessfulCreate          cronjob/descheduler                          Created job descheduler-28519765
33m         Normal    SuccessfulCreate          cronjob/descheduler                          Created job descheduler-28519770
28m         Normal    SuccessfulCreate          cronjob/descheduler                          Created job descheduler-28519775
23m         Normal    SuccessfulCreate          cronjob/descheduler                          Created job descheduler-28519780
3m4s        Normal    SuccessfulCreate          cronjob/descheduler                          (combined from similar events): Created job descheduler-28519800
58m         Normal    RemoveFailedPods          pod/external-dns-89b8dcd4b-6668b             pod evicted from master3 node by sigs.k8s.io/descheduler
58m         Normal    RemoveFailedPods          pod/external-dns-89b8dcd4b-75xxg             pod evicted from master1 node by sigs.k8s.io/descheduler
58m         Normal    RemoveFailedPods          pod/external-dns-89b8dcd4b-dbz72             pod evicted from master3 node by sigs.k8s.io/descheduler
58m         Normal    Pulled                    pod/external-dns-89b8dcd4b-dwjb8             Container image "docker.io/bitnami/external-dns:0.13.6-debian-11-r30" already present on machine
58m         Normal    RemoveFailedPods          pod/external-dns-89b8dcd4b-px4kp             pod evicted from master2 node by sigs.k8s.io/descheduler
58m         Normal    Pulled                    pod/kube-flannel-r8wnp                       Container image "ghcr.io/siderolabs/flannel:v0.23.0" already present on machine
60m         Warning   BackOff                   pod/kube-flannel-txkd5                       Back-off restarting failed container kube-flannel in pod kube-flannel-txkd5_kube-system(82c08c0e-3bc8-49f0-b458-1de2c691c26c)
56m         Normal    NoSourceArtifact          helmchart/kube-system-descheduler            no artifact available for HelmRepository source 'descheduler'
54m         Normal    ChartPullSucceeded        helmchart/kube-system-descheduler            pulled 'descheduler' chart with version '0.27.1'
60m         Normal    SandboxChanged            pod/local-path-provisioner-8648b54f9-lqlwg   Pod sandbox changed, it will be killed and re-created.
58m         Normal    RemoveFailedPods          pod/local-path-provisioner-8648b54f9-rk9d5   pod evicted from master1 node by sigs.k8s.io/descheduler
60m         Warning   FailedCreatePodSandBox    pod/metrics-server-6db4d75b97-4hdn8          (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8de848ec5440448dc42f0e3e9656b444686c81984873cfb027f248f082184da2": plugin type="flannel" failed (add): loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no such file or directory

[smira]

Then you'd have to figure out why kubelet doesn't register them. It's not Talos at this point, but plain Kubernetes, or the way you configured things.

[runningman84]

ok I have found some issue in the kubelet log:

10.0.30.81: {"ts":1711188676157.6511,"caller":"kubelet/kubelet.go:1921","msg":"Failed creating a mirror pod for","pod":{"name":"kube-apiserver-master1","namespace":"kube-system"},"err":"Pod \"kube-apiserver-master1\" is invalid: spec.containers[0].resources.requests: Invalid value: \"512Mi\": must be less than or equal to memory limit of 256Mi"}

I do not think that I have changed anything in the requests or limits but I can take a look into that...

[runningman84]

the pod definition does not contain any limit:

  talos talosctl get staticpods -o yaml -n 10.0.30.81
node: 10.0.30.81
metadata:
    namespace: k8s
    type: StaticPods.kubernetes.talos.dev
    id: kube-apiserver
    version: 1
    owner: k8s.ControlPlaneStaticPodController
    phase: running
    created: 2024-03-23T08:43:10Z
    updated: 2024-03-23T08:43:10Z
spec:
    apiVersion: v1
    kind: Pod
    metadata:
        annotations:
            talos.dev/config-file-version: "111"
            talos.dev/config-version: "1"
            talos.dev/secrets-version: "1"
        creationTimestamp: null
        labels:
            k8s-app: kube-apiserver
            tier: control-plane
        name: kube-apiserver
        namespace: kube-system
    spec:
        containers:
            - command:
                - /usr/local/bin/kube-apiserver
                - --admission-control-config-file=/system/config/kubernetes/kube-apiserver/admission-control-config.yaml
                - --advertise-address=$(POD_IP)
                - --allow-privileged=true
                - --anonymous-auth=false
                - --api-audiences=https://10.0.30.80:6443
                - --audit-log-maxage=30
                - --audit-log-maxbackup=10
                - --audit-log-maxsize=100
                - --audit-log-path=/var/log/audit/kube/kube-apiserver.log
                - --audit-policy-file=/system/config/kubernetes/kube-apiserver/auditpolicy.yaml
                - --authorization-mode=Node,RBAC
                - --bind-address=0.0.0.0
                - --client-ca-file=/system/secrets/kubernetes/kube-apiserver/ca.crt
                - --enable-admission-plugins=NodeRestriction
                - --enable-bootstrap-token-auth=true
                - --encryption-provider-config=/system/secrets/kubernetes/kube-apiserver/encryptionconfig.yaml
                - --etcd-cafile=/system/secrets/kubernetes/kube-apiserver/etcd-client-ca.crt
                - --etcd-certfile=/system/secrets/kubernetes/kube-apiserver/etcd-client.crt
                - --etcd-keyfile=/system/secrets/kubernetes/kube-apiserver/etcd-client.key
                - --etcd-servers=https://localhost:2379
                - --kubelet-client-certificate=/system/secrets/kubernetes/kube-apiserver/apiserver-kubelet-client.crt
                - --kubelet-client-key=/system/secrets/kubernetes/kube-apiserver/apiserver-kubelet-client.key
                - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
                - --profiling=false
                - --proxy-client-cert-file=/system/secrets/kubernetes/kube-apiserver/front-proxy-client.crt
                - --proxy-client-key-file=/system/secrets/kubernetes/kube-apiserver/front-proxy-client.key
                - --requestheader-allowed-names=front-proxy-client
                - --requestheader-client-ca-file=/system/secrets/kubernetes/kube-apiserver/aggregator-ca.crt
                - --requestheader-extra-headers-prefix=X-Remote-Extra-
                - --requestheader-group-headers=X-Remote-Group
                - --requestheader-username-headers=X-Remote-User
                - --secure-port=6443
                - --service-account-issuer=https://10.0.30.80:6443
                - --service-account-key-file=/system/secrets/kubernetes/kube-apiserver/service-account.pub
                - --service-account-signing-key-file=/system/secrets/kubernetes/kube-apiserver/service-account.key
                - --service-cluster-ip-range=10.96.0.0/12
                - --tls-cert-file=/system/secrets/kubernetes/kube-apiserver/apiserver.crt
                - --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
                - --tls-min-version=VersionTLS12
                - --tls-private-key-file=/system/secrets/kubernetes/kube-apiserver/apiserver.key
              env:
                - name: POD_IP
                  valueFrom:
                    fieldRef:
                        fieldPath: status.podIP
              image: registry.k8s.io/kube-apiserver:v1.29.3
              name: kube-apiserver
              resources:
                requests:
                    cpu: 200m
                    memory: 512Mi
              securityContext:
                allowPrivilegeEscalation: false
                capabilities:
                    add:
                        - NET_BIND_SERVICE
                    drop:
                        - ALL
                seccompProfile:
                    type: RuntimeDefault
              volumeMounts:
                - mountPath: /system/secrets/kubernetes/kube-apiserver
                  name: secrets
                  readOnly: true
                - mountPath: /system/config/kubernetes/kube-apiserver
                  name: config
                  readOnly: true
                - mountPath: /var/log/audit/kube
                  name: audit
        hostNetwork: true
        priority: 2e+09
        priorityClassName: system-cluster-critical
        securityContext:
            runAsGroup: 65534
            runAsNonRoot: true
            runAsUser: 65534
        volumes:
            - hostPath:
                path: /system/secrets/kubernetes/kube-apiserver
              name: secrets
            - hostPath:
                path: /system/config/kubernetes/kube-apiserver
              name: config
            - hostPath:
                path: /var/log/audit/kube
              name: audit
    status: {}

But there is some limitrange

➜  talos kubectl get limitrange -n kube-system -o yaml
apiVersion: v1
items:
- apiVersion: v1
  kind: LimitRange
  metadata:
    creationTimestamp: "2024-03-17T12:04:15Z"
    labels:
      kustomize.toolkit.fluxcd.io/name: basics
      kustomize.toolkit.fluxcd.io/namespace: flux-system
    name: mem-limit-range
    namespace: kube-system
    resourceVersion: "35800"
    uid: f7c72632-b4ac-417d-9852-97120ee78a13
  spec:
    limits:
    - default:
        memory: 256Mi
      defaultRequest:
        memory: 64Mi
      type: Container
kind: List
metadata:
  resourceVersion: ""

Remove that fixed it...

➜  talos talosctl -n 10.0.30.81 health
discovered nodes: ["10.0.30.80" "10.0.30.82" "10.0.30.83"]
waiting for etcd to be healthy: ...
waiting for etcd to be healthy: OK
waiting for etcd members to be consistent across nodes: ...
waiting for etcd members to be consistent across nodes: OK
waiting for etcd members to be control plane nodes: ...
waiting for etcd members to be control plane nodes: OK
waiting for apid to be ready: ...
waiting for apid to be ready: OK
waiting for all nodes memory sizes: ...
waiting for all nodes memory sizes: OK
waiting for all nodes disk sizes: ...
waiting for all nodes disk sizes: OK
waiting for kubelet to be healthy: ...
waiting for kubelet to be healthy: OK
waiting for all nodes to finish boot sequence: ...
waiting for all nodes to finish boot sequence: OK
waiting for all k8s nodes to report: ...
waiting for all k8s nodes to report: OK
waiting for all k8s nodes to report ready: ...
waiting for all k8s nodes to report ready: OK
waiting for all control plane static pods to be running: ...
waiting for all control plane static pods to be running: OK
waiting for all control plane components to be ready: ...
waiting for all control plane components to be ready: OK
waiting for kube-proxy to report ready: ...
waiting for kube-proxy to report ready: OK
waiting for coredns to report ready: ...
waiting for coredns to report ready: OK
waiting for all k8s nodes to report schedulable: ...
waiting for all k8s nodes to report schedulable: OK

Thans for you guidance!