Oh where oh where can my arg5 be?

[rzezeski]

As I mentioned in #61, I'm seeing some issues around usdt while working on oxidecomputer/opte#145. I'm going to eschew the background and get straight to it.

Issue

EDIT: This is an M1 (ARM) mac.

On macOS I cannot seem to pass more than five arguments to a probe. As part of my change to OPTE (in the above link) I added an epoch argument to the port__process__return probe, now giving it a total of six arguments, pushing the res argument to arg5 position. After doing this the probe always fires with an arg5 of NULL.

$ sudo dtrace -Zn 'port-process-return { trace(arg5); }'
Password:
dtrace: description 'port-process-return ' matched 0 probes
CPU     ID                    FUNCTION:NAME
  5  14747 _ZN4opte6engine4port4Port25port_process_return_probe17hec39bcf71fd676a8E:port-process-return                 0
  2  14747 _ZN4opte6engine4port4Port25port_process_return_probe17hec39bcf71fd676a8E:port-process-return                 0
  4  14747 _ZN4opte6engine4port4Port25port_process_return_probe17hec39bcf71fd676a8E:port-process-return                 0
  9  14747 _ZN4opte6engine4port4Port25port_process_return_probe17hec39bcf71fd676a8E:port-process-return                 0
  6  14747 _ZN4opte6engine4port4Port25port_process_return_probe17hec39bcf71fd676a8E:port-process-return                 0
  7  14747 _ZN4opte6engine4port4Port25port_process_return_probe17hec39bcf71fd676a8E:port-process-return                 0
  8  14747 _ZN4opte6engine4port4Port25port_process_return_probe17hec39bcf71fd676a8E:port-process-return                 0
  8  14747 _ZN4opte6engine4port4Port25port_process_return_probe17hec39bcf71fd676a8E:port-process-return                 0
  7  14747 _ZN4opte6engine4port4Port25port_process_return_probe17hec39bcf71fd676a8E:port-process-return                 0
  8  14747 _ZN4opte6engine4port4Port25port_process_return_probe17hec39bcf71fd676a8E:port-process-return                 0
  8  14747 _ZN4opte6engine4port4Port25port_process_return_probe17hec39bcf71fd676a8E:port-process-return                 0
  6  14747 _ZN4opte6engine4port4Port25port_process_return_probe17hec39bcf71fd676a8E:port-process-return                 0
  6  14747 _ZN4opte6engine4port4Port25port_process_return_probe17hec39bcf71fd676a8E:port-process-return                 0

Reproduce

1. Clone the opte repo onto a macOS box and checkout the opte-143-set-fw-rules branch.
2. In one window trace arg5 of the port-process-return probe.

$ sudo dtrace -Zn 'port-process-return { trace(arg5); }'

3. In another window run the OPTE unit tests with the usdt feature.

$ cd opte
$ cargo test --features usdt

[bnaecker]

Thanks for filing this, and the other related issue! To check, are you running on an x86 or ARM-based platform?

[bnaecker]

So I can't repro this, but I'm on an x86 system, and I recall your mac is pretty new. If it's ARM, that definitely points to us screwing up the registers on ARM that should be used for storing the probe arguments.

Here's my output from your steps:

bnaecker@sandstone : ~/file-cabinet/oxide/opte $ sudo dtrace -Zn 'port-process-return { trace(arg5); }'
Password:
dtrace: system integrity protection is on, some features will not be available

dtrace: description 'port-process-return ' matched 0 probes
CPU     ID                    FUNCTION:NAME
  7  23079 _ZN4opte6engine4port4Port25port_process_return_probe17h0e527bc92f9d898bE:port-process-return   140585278138192
  0  23079 _ZN4opte6engine4port4Port25port_process_return_probe17h0e527bc92f9d898bE:port-process-return   140585279186208
  1  23079 _ZN4opte6engine4port4Port25port_process_return_probe17h0e527bc92f9d898bE:port-process-return   140585267651728
  4  23079 _ZN4opte6engine4port4Port25port_process_return_probe17h0e527bc92f9d898bE:port-process-return   140585266595072
  0  23079 _ZN4opte6engine4port4Port25port_process_return_probe17h0e527bc92f9d898bE:port-process-return   140585279178816
  3  23079 _ZN4opte6engine4port4Port25port_process_return_probe17h0e527bc92f9d898bE:port-process-return   140585277080448
  5  23079 _ZN4opte6engine4port4Port25port_process_return_probe17h0e527bc92f9d898bE:port-process-return   140585265571728
  0  23079 _ZN4opte6engine4port4Port25port_process_return_probe17h0e527bc92f9d898bE:port-process-return   140585279178672
  7  23079 _ZN4opte6engine4port4Port25port_process_return_probe17h0e527bc92f9d898bE:port-process-return   140585278151248
  4  23079 _ZN4opte6engine4port4Port25port_process_return_probe17h0e527bc92f9d898bE:port-process-return   140585266595408
  6  23079 _ZN4opte6engine4port4Port25port_process_return_probe17h0e527bc92f9d898bE:port-process-return   140585264505968
  7  23079 _ZN4opte6engine4port4Port25port_process_return_probe17h0e527bc92f9d898bE:port-process-return   140585278135424
  6  23079 _ZN4opte6engine4port4Port25port_process_return_probe17h0e527bc92f9d898bE:port-process-return   140585264516624

[rzezeski]

Yes this is an M1 mac.

[bnaecker]

This is probably going to be difficult to track down, since I don't have an ARM mac to test with.

You should be able to use up to 6 arguments without issue. More than that will generate a compiler error. So we should still be good here. It looks like we're using the correct registers for general-purpose parameter transfer here, at least, if I'm understanding the inline assembly documentation on AArch64 register operands and the ARM ABI correctly.

I'll install a toolchain to build this, and try to find a way to test it. Thanks for filing, hopefully we can figure it out soon.

[rzezeski]

@bnaecker Thanks for looking into this. This is very low on my priority list so I wouldn't stress over it too much. The probes work on illumos fine and I can just run there.

[ratmice]

Seems today github added m1 runners to their actions, so wonder if this issue can be perturbed by CI now.