Need help with Kepler on Talos (no metrics)

[kaznak]

Hi all,

I'm new to Talos Linux and I'm really liking it so far! I'm running a Talos cluster at home and I'm trying to keep an eye on its power consumption.

I found Kepler( https://sustainable-computing.io/ ) , which is supposed to measure power consumption, but I'm having trouble getting it to work. I had to set pod-security.kubernetes.io/enforce: privileged, but that didn't fix the problem.

The logs show that it can't access these files:

1. /sys/devices/system/cpu/cpu0/online
2. /dev/cpu/0/msr
3. /sys/kernel/tracing/events/writeback/writeback_dirty_page/id

I'm not a Linux kernel expert, but it seems like these files should be accessible. Is there any way to get Kepler working on Talos?

Here is some informations about this problem.

➜ talosctl get cpu 
NODE            NAMESPACE   TYPE        ID        VERSION   MANUFACTURER           MODEL                                     CORES   THREADS
192.168.11.32   hardware    Processor   LGA1151   1         Intel(R) Corporation   Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz   6       12

➜ talosctl get ram                                                             
NODE            NAMESPACE   TYPE           ID               VERSION   MANUFACTURER   MODEL              SIZEMIB
192.168.11.32   hardware    MemoryModule   ChannelA-DIMM1   1         Samsung        M378A5244CB0-CTD   4096
192.168.11.32   hardware    MemoryModule   ChannelB-DIMM1   1         Samsung        M378A5244CB0-CTD   4096

➜ kubectl -n obs-prometheus logs kepler-pn57x   
WARNING: failed to read int from file: open /sys/devices/system/cpu/cpu0/online: no such file or directory
I0129 15:18:37.168036       1 exporter.go:103] Kepler running on version: v0.7.12-dirty
I0129 15:18:37.168126       1 config.go:293] using gCgroup ID in the BPF program: true
I0129 15:18:37.168159       1 config.go:295] kernel version: 6.12
I0129 15:18:37.168202       1 rapl_msr_util.go:129] failed to open path /dev/cpu/0/msr: no such file or directory
I0129 15:18:37.168537       1 power.go:78] Unable to obtain power, use estimate method
I0129 15:18:37.168551       1 redfish.go:169] failed to get redfish credential file path
I0129 15:18:37.169474       1 acpi.go:71] Could not find any ACPI power meter path. Is it a VM?
I0129 15:18:37.169480       1 power.go:79] using none to obtain power
E0129 15:18:37.169488       1 accelerator.go:154] [DUMMY] doesn't contain GPU
E0129 15:18:37.169506       1 exporter.go:154] failed to init GPU accelerators: no devices found
WARNING: failed to read int from file: open /sys/devices/system/cpu/cpu0/online: no such file or directory
I0129 15:18:37.171435       1 exporter.go:84] Number of CPUs: 12
W0129 15:18:37.390337       1 exporter.go:135] failed to attach tp/writeback/writeback_dirty_page: reading file "/sys/kernel/tracing/events/writeback/writeback_dirty_page/id": open /sys/kernel/tracing/events/writeback/writeback_dirty_page/id: no such file or directory. Kepler will not collect page cache write events. This will affect the DRAM power model estimation on VMs.
I0129 15:18:37.395382       1 watcher.go:83] Using in cluster k8s config
I0129 15:18:37.495932       1 watcher.go:229] k8s APIserver watcher was started
I0129 15:18:37.517744       1 process_energy.go:129] Using the Ratio Power Model to estimate PROCESS_TOTAL Power
I0129 15:18:37.517789       1 process_energy.go:130] Feature names: [bpf_cpu_time_ms]
I0129 15:18:37.522379       1 process_energy.go:129] Using the Ratio Power Model to estimate PROCESS_COMPONENTS Power
I0129 15:18:37.522419       1 process_energy.go:130] Feature names: [bpf_cpu_time_ms bpf_cpu_time_ms bpf_cpu_time_ms   gpu_compute_util]
I0129 15:18:37.541930       1 regressor.go:276] Created predictor linear for trainer: "SGDRegressorTrainer"
I0129 15:18:37.541968       1 model.go:125] Requesting for Machine Spec: &{genuineintel intel_core_i7_8700 12 1 7 4600 2}
I0129 15:18:37.542008       1 node_platform_energy.go:53] Using the Regressor/AbsPower Power Model to estimate Node Platform Power
I0129 15:18:37.560806       1 regressor.go:276] Created predictor linear for trainer: "SGDRegressorTrainer"
I0129 15:18:37.560839       1 regressor.go:276] Created predictor linear for trainer: "SGDRegressorTrainer"
I0129 15:18:37.560856       1 regressor.go:276] Created predictor linear for trainer: "SGDRegressorTrainer"
I0129 15:18:37.560881       1 regressor.go:276] Created predictor linear for trainer: "SGDRegressorTrainer"
I0129 15:18:37.560900       1 model.go:125] Requesting for Machine Spec: &{genuineintel intel_core_i7_8700 12 1 7 4600 2}
I0129 15:18:37.560952       1 node_component_energy.go:57] Using the Regressor/AbsPower Power Model to estimate Node Component Power
I0129 15:18:37.561225       1 prometheus_collector.go:95] Registered Container Prometheus metrics
I0129 15:18:37.561356       1 prometheus_collector.go:100] Registered VM Prometheus metrics
I0129 15:18:37.561403       1 prometheus_collector.go:104] Registered Node Prometheus metrics
I0129 15:18:37.563529       1 exporter.go:194] starting to listen on 0.0.0.0:9102
I0129 15:18:37.563745       1 exporter.go:208] Started Kepler in 395.842121ms
E0129 15:18:37.650029       1 watcher.go:189] parsing pod kepler-pn57x obs-prometheus ContainerStatuses issue : container kepler-exporter did not start yet status, InitContainerStatuses issue :<nil>, EphemeralContainerStatuses issue :<nil>

Any help would be appreciated!

Thanks,

[smira]

It's hard to guess, but it shouldn't be Talos-specific.

You can check presence of the files on the host with talosctl ls <path> or even talosctl read <path> to see the value.

If the container doesn't see these files, it's simply misconfigured, e.g. it might need to have privileged enabled, but that is better asked on Kepler side.

[kaznak]

Thank you for your answer.

I confirmed with talosctl ls that the above files are not present, so if it is not a Talos-specific issue, does that mean that these files should not be present originally?

I then booted the same machine from a ubuntu USB image and /dev/cpu/0/msr was present. This leads me to believe that this file may not exist due to differences in kernel configuration.

As for privileges, I assigned them to the namespace running pod-security.kubernetes.io/enforce: privileged. Is there anything else I should do?

In any case, Kepler seems to be the cause of this problem. I will contact them as well.

Thank you very much.

[smira]

I don't see anything immediately missing in Talos kernel config, for specifically cpu/online, but CONFIG_X86_MSR is disabled, as by default Talos Linux hardens the kernel configuration, and exposing this is considered insecure.

[kaznak]

The Kepler team has not responded yet. However, this problem might be caused by the lack of capability to get metrics from the hardware or Kepler. Your information about disabling CONFIG_X86_MSR was helpful. Thank you for your answer.