Flag `-Zindirect-branch-cs-prefix` (retpoline mitigation)

[ojeda]

Proposal

Add a new flag to the compiler, -Zindirect-branch-cs-prefix, for x86, equivalent to Clang's and GCC's -mindirect-branch-cs-prefix:

• From https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html#index-mindirect-branch-cs-prefix:

  Add CS prefix to call and jmp to indirect thunk with branch target in r8-r15 registers so that the call and jmp instruction length is 6 bytes to allow them to be replaced with lfence; call *%r8-r15 or lfence; jmp *%r8-r15 at run-time.

• From https://clang.llvm.org/docs/ClangCommandLineReference.html#cmdoption-clang-mindirect-branch-cs-prefix:

  Add cs prefix to call and jmp to indirect thunk

This flag is used in the Linux kernel when the retpoline mitigation is enabled in order to inline more cases (i.e. to replace more retpoline thunk calls (__x86_indirect_thunk*) with lfence; call *%reg). In the Linux kernel, it gets used together with -mindirect-branch=thunk-extern -mindirect-branch-register (GCC) or -mretpoline-external-thunk (Clang) when supported by the C compiler. The rustc equivalent of the main retpoline flags (-Zretpoline, -Zretpoline-external-thunk) have already landed in rustc (rust-lang/rust#116852, rust-lang/rust#135927).

The flag, at least in the Linux kernel, is used only for retpoline builds, and thus one could consider just doing it unconditionally when other retpoline flags are enabled. However, that would mean less flexibility compared to C compilers and other projects (e.g. other kernels) may not support the inlining and thus may not want the prefix to be added to begin with.

Another aspect is whether to have this as a modifier of one of the existing flags or as an independent flag. Matching the naming of existing flags in C compilers (especially when both GCC and Clang already agree) is clearer, especially for niche flags. For instance, in the Linux kernel Makefiles, the C and Rust flags would be assigned in consecutive lines, which makes it clear it is intended to have the same purpose if their name match. On the other hand, it would be less consistent with other rustc flags.

The implementation, including tests, using the -Zindirect-branch-cs-prefix name (i.e. matching GCC and Clang), is already available at rust-lang/rust#140740. The flag is added with the intent of later stabilizing it.

More context:

• https://git.kernel.org/linus/68cf4f2a72ef8786e6b7af6fd9a89f27ac0f520d.
• https://git.kernel.org/linus/bbe2df3f6b6da7848398d55b1311d58a16ec21e4.
• https://gcc.gnu.org/g:2196a681d7810ad8b227bf983f38ba716620545e.
• https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102952.
• https://bugs.llvm.org/show_bug.cgi?id=52323.

Cc: @davidtwco

Process

The main points of the Major Change Process are as follows:

• File an issue describing the proposal.
• A compiler team member or contributor who is knowledgeable in the area can second by writing @rustbot second.
  • Finding a "second" suffices for internal changes. If however, you are proposing a new public-facing feature, such as a -C flag, then full team check-off is required.
  • Compiler team members can initiate a check-off via @rfcbot fcp merge on either the MCP or the PR.
• Once an MCP is seconded, the Final Comment Period begins. If no objections are raised after 10 days, the MCP is considered approved.

You can read more about Major Change Proposals on forge.

[rustbot]

Important

This issue is not meant to be used for technical discussion. There is a Zulip stream for that.
Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

Concerns or objections can formally be registered here by adding a comment.

@rustbot concern reason-for-concern
<description of the concern>

Concerns can be lifted with:

@rustbot resolve reason-for-concern

See documentation at https://forge.rust-lang.org

cc @rust-lang/compiler