Skip to content

Conversation

@dylandreimerink
Copy link
Member

@dylandreimerink dylandreimerink commented Aug 27, 2025

info: return ErrRestrictedKernel when program info is restricted

On systems where the `kernel.kptr_restrict` and
`net.core.bpf_jit_harden` sysctls are enabled, certain fields in the
program info may be restricted. When this is the case xlated and jitted
instruction, line info, and function info are unavailable.

When such fields are unavailable, we do get *Len fields set to non-zero
values, but the kernel will have not written any data to the pointers
provided.

At present when this happens we don't recognize this and attempt to
parse the empty buffers resulting in a `parse func info: offset 0:
type ID 0 is a *btf.Void, but expected a Func` error.

The only clue we have that this is happening is that the kernel will
zero out the insns pointer. This commit adds logic to detect when
this happens and then returns a dedicated `ErrRestrictedKernel` error
so this case can be handled gracefully.

This is similar to / based on the approach bpftool has taken to handle this.

kallsyms: return ErrRestrictedKernel when reading zero address

Reading a zero address for an existing kallsyms address is never valid,
so make internal.AssignAddresses return ErrRestrictedKernel if it encounters
a zero address due to kernel.kptr_restrict being set on the host.

To enable users to gracefully fall back to a non-ksym implementation, tolerate
ErrRestrictedKernel during ksym resolution unless there are required ksyms in
the CollectionSpec.

@dylandreimerink dylandreimerink force-pushed the feature/handle-restricted-prog-info branch from 27470e0 to 83c838b Compare August 27, 2025 16:36
@dylandreimerink dylandreimerink marked this pull request as ready for review August 28, 2025 09:19
@dylandreimerink dylandreimerink requested a review from a team as a code owner August 28, 2025 09:19
florianl
florianl previously approved these changes Aug 28, 2025
@dylandreimerink dylandreimerink force-pushed the feature/handle-restricted-prog-info branch from 83c838b to 33b393e Compare September 15, 2025 11:41
@ti-mo ti-mo force-pushed the feature/handle-restricted-prog-info branch from 33b393e to 1286cdb Compare October 14, 2025 15:47
@ti-mo ti-mo changed the title info: Return dedicated error when program info is restricted Return ErrRestrictedKernel when kernel pointers are restricted using sysctls Oct 14, 2025
@ti-mo ti-mo force-pushed the feature/handle-restricted-prog-info branch from 1286cdb to bc642ec Compare October 14, 2025 15:54
ti-mo
ti-mo previously approved these changes Oct 14, 2025
florianl
florianl previously approved these changes Oct 14, 2025
dylandreimerink and others added 2 commits October 15, 2025 09:41
On systems where the `kernel.kptr_restrict` and
`net.core.bpf_jit_harden` sysctls are enabled, certain fields in the
program info may be restricted. When this is the case xlated and jitted
instruction, line info, and function info are unavailable.

When such fields are unavailable, we do get *Len fields set to non-zero
values, but the kernel will have not written any data to the pointers
provided.

At present when this happens we don't recognize this and attempt to
parse the empty buffers resulting in a `parse func info: offset 0:
type ID 0 is a *btf.Void, but expected a Func` error.

The only clue we have that this is happening is that the kernel will
zero out the insns pointer. This commit adds logic to detect when
this happens and then returns a dedicated `ErrRestrictedKernel` error
so this case can be handled gracefully.

Signed-off-by: Dylan Reimerink <[email protected]>
Co-authored-by: Timo Beckers <[email protected]>
Reading a zero address for an existing kallsyms address is never valid,
so make internal.AssignAddresses return ErrRestrictedKernel if it encounters
a zero address due to kernel.kptr_restrict being set on the host.

To enable users to gracefully fall back to a non-ksym implementation, tolerate
ErrRestrictedKernel during ksym resolution unless there are required ksyms in
the CollectionSpec.

Signed-off-by: Timo Beckers <[email protected]>
@ti-mo ti-mo dismissed stale reviews from florianl and themself via 30ee5fc October 15, 2025 07:45
@ti-mo ti-mo force-pushed the feature/handle-restricted-prog-info branch from bc642ec to 30ee5fc Compare October 15, 2025 07:45
@ti-mo ti-mo merged commit 9a014ef into main Oct 15, 2025
16 of 18 checks passed
@ti-mo ti-mo deleted the feature/handle-restricted-prog-info branch October 15, 2025 10:20
jrife added a commit to jrife/cilium that referenced this pull request Oct 23, 2025
Environments like GKE COS enable BPF JIT hardening (
net.core.bpf_jit_harden=2). With this enabled, BPF_OBJ_GET_INFO_BY_FD
does not provide xlated instructions, JITed instructions, or other
related info in its response when querying program info. So, any Cilium
feature that relies on these fields won't work. Currently, there are
three such places:

1. The `HaveBPFJIT` probe calls info.JitedSize() to determine if the
   program was JITed.
2. The `HaveDeadCodeElim` probe calls info.Instructions() to inspect the
   final BPF instructions to see if dead code elimination was applied.
3. `verifyUnusedMaps` calls info.Instructions() in an attempt to walk
   the final program instructions and see which maps are referenced
   after dead code elimination.

Recently, cilium/ebpf#1858 started returning ErrRestrictedKernel when
querying these fields in such environments [1]. Before, these probes
would silently fail, but now they fail loudly blocking Cilium startup.

`verifyUnusedMaps` hits ErrRestrictedKernel when querying program
instructions. AFAICT this does not impact functionality, but may result
in unused maps being loaded.

```
msg="verifying unused maps: getting instructions for program tail_icmp6_send_time_exceeded: instructions: restricted by kernel.kptr_restrict and/or net.core.bpf_jit_harden sysctls" module=agent.datapath.loader
```

The version bump to cilium/ebpf in cilium/cilium was reverted in
cilium#42327 [2] to fix these probe failures, so undo this and
forward fix Cilium by tolerating ErrRestrictedKernel in the probes.
This reverts Cilium back to its previous behavior, silently ignoring
these probe failures in environments where BPF JIT hardening is enabled,
while allowing cilium/ebpf to be upgraded.

[1]: cilium/ebpf#1858
[2]: cilium#42327

Signed-off-by: Jordan Rife <[email protected]>
github-merge-queue bot pushed a commit to cilium/cilium that referenced this pull request Oct 28, 2025
Environments like GKE COS enable BPF JIT hardening (
net.core.bpf_jit_harden=2). With this enabled, BPF_OBJ_GET_INFO_BY_FD
does not provide xlated instructions, JITed instructions, or other
related info in its response when querying program info. So, any Cilium
feature that relies on these fields won't work. Currently, there are
three such places:

1. The `HaveBPFJIT` probe calls info.JitedSize() to determine if the
   program was JITed.
2. The `HaveDeadCodeElim` probe calls info.Instructions() to inspect the
   final BPF instructions to see if dead code elimination was applied.
3. `verifyUnusedMaps` calls info.Instructions() in an attempt to walk
   the final program instructions and see which maps are referenced
   after dead code elimination.

Recently, cilium/ebpf#1858 started returning ErrRestrictedKernel when
querying these fields in such environments [1]. Before, these probes
would silently fail, but now they fail loudly blocking Cilium startup.

`verifyUnusedMaps` hits ErrRestrictedKernel when querying program
instructions. AFAICT this does not impact functionality, but may result
in unused maps being loaded.

```
msg="verifying unused maps: getting instructions for program tail_icmp6_send_time_exceeded: instructions: restricted by kernel.kptr_restrict and/or net.core.bpf_jit_harden sysctls" module=agent.datapath.loader
```

The version bump to cilium/ebpf in cilium/cilium was reverted in
#42327 [2] to fix these probe failures, so undo this and
forward fix Cilium by tolerating ErrRestrictedKernel in the probes.
This reverts Cilium back to its previous behavior, silently ignoring
these probe failures in environments where BPF JIT hardening is enabled,
while allowing cilium/ebpf to be upgraded.

[1]: cilium/ebpf#1858
[2]: #42327

Signed-off-by: Jordan Rife <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants