stapsdt, monormoprhisation, and linker --gc-sections.

[conradludgate]

I've spent a few days trying to debug why I'm getting weird behaviour with usdt on linux.

What I'm observing is many stapsdt notes that end up linked into my ELF file have the probe address unresolved. The conditions for this bug to appear are quite hard to minimise for:

1. It requires a probe to be part of a monomorphised function
2. The monomorphised function needs to be instantiated across multiple codegen units
3. Using the lld or mold linker

By eliminating any of these the behaviour would not manifest. After some time chatting with peers. I was able to find that the bug disappears with RUSTFLAGS=-Clink-args=-Wl,--no-gc-sections. This leads me to conclude the bug is the following:

1. A function is monomorphised across several CGU.
2. This monomorphised function contains the stapsdt note.
3. The linker deduplicates these functions when applying GC
4. The linker does not remove any dangling notes.

I've found 2 ways to resolve the issue in my testing. But I don't understand whether there's any other consequences of the code they produce. They both relate to the following line

usdt/usdt-impl/src/stapsdt.rs

Line 156 in 4329591

.pushsection .note.stapsdt, "", "note"

Retained

The first fix is to mark the note as "retained". This ensures that the functions don't get de-duplicated and thus the notes do not dangle. This is not ideal as this causes binary bloat.

        .pushsection .note.stapsdt, "R", "note"

Comdat

The second fix I don't fully understand but it works in all my testing and I can't find any problems with the resulting ELF. It uses the following change:

        .pushsection .note.stapsdt, "G", "note", ".note.stapsdt.{prov}.{probe}", comdat

cc @aapoalas

[conradludgate]

For the full context, here's the full assembly I'm now using (not using the usdt crate, but it's heavily inspired for now https://github.com/tokio-rs/tokio/blob/ed3c773d39f227f6863591a76b82523117e402de/tokio/src/util/usdt/stapsdt.rs#L28-L64)

[aapoalas]

Usdt should already be producing notes with "aGR", ie. Including retain

[aapoalas]

#450

[conradludgate]

That applies to the base, not the note.

[aapoalas]

That applies to the base, not the note.

Oop, my bad; I was too hasty in my reading. That's an interesting failure mode... I wonder if the ? option might have any play here; I don't quite understand it but it has something to do with grouping of notes which does relate to garbage collection in some ways as well. Maybe.

[conradludgate]

I did see that in the systemtap code, but it didn't work for me. I did have high hopes thinking it would help since they mentioned clang/C++ which would indeed observe the same behaviour

[conradludgate]

?
section is a member of the previously-current section’s group, if any

This does look promising, but I fear the "if any" might be a problem.

[aapoalas]

Not sure if this exact blog post will enlighten us, but the website in general is a pretty definite source of deep information on ELF sections: https://maskray.me/blog/2021-07-25-comdat-and-section-group

[aapoalas]

comdat[1]

indicates that only one copy of this section should be retained

I wonder if this can break using the same probe in multiple places? That is officially a supported thing in STAPSDT, and at least sometimes used.

[conradludgate]

I did have that concern too. But it seems to work regardless - I'm using the same probes in multiple places and I see those notes in the elf. Maybe I'm just getting lucky.

However I've found it's not sufficient anyway. I'm still seeing the original issue when I use my latest tokio patch as part of a much bigger project.

[conradludgate]

Thanks to #t-compiler > ✔ inline assembly group inheritance @ 💬, it seems we have found a workable solution. This produces much better results in my testing.