How I solved the problem of cfg/cfg_attr expansion in proc macro attributes

[Veetaha]

This is in response to the reddit comment by @jplatte.

When developing a hack for cfg/cfg_attr handing in proc macro attributes, I tried almost everything possible. I even tried generating a dummy struct with a derive() (as suggested in @jplatte's Reddit comment). It looked smth like this:

#[derive(bon::__ExpandCfgs)]
#[expand_cfgs(
    // Outputs a call to the proc macro attribute specified in this parameter
    callback = bon::builder,

    // Passes through these params at the end of `callback(...)`. Here
    // should be the parameters passed to the `#[builder(...)]` by the user
    // that are required for its business logic
    params(/* */),

    // The item the proc-macro-attribute was placed on top of
    item(fn greet(/* */) {/* */} )
)]
// Add a `cfg_attr` for every cfg expression. They will be evaluated
// by the compiler automatically before the derive(bon::__ExpandCfgs) runs.
// This way __ExpandCfgs can see which cfgs evaluated to `true`, and consider
// all other cfgs to be `false`.
#[cfg_attr(feature = "foo", expand_cfgs(cfg(feature = "foo")))]
#[cfg_attr(feature = "bar", expand_cfgs(cfg(feature = "bar")))]
struct DummyStruct;

Now let's suppose that only feature = "bar" cfg evaluates to true.

In this case, the logic of derive(bon::__ExpandCfgs) would expand to the following code:

                   // v- here goes the comma-separated list of all cfg predicates that evaluate to `true`
#[bon::builder(__cfgs(feature = "foo") /* params */)]
/* item */

Then bon::builder would parse the special __cfgs(...) syntax in front of its parameters to collect the cfgs expressions that evaluate to true. Then there is a bit of code that traverses the function or impl block that bon::builder was placed on and evaluates the cfg/cfg_attr manually (it basically reimplements these attrs inside of bon::builder). Yeah... that's unfortunate:

bon/bon-macros/src/normalization/cfg/mod.rs

Lines 191 to 243 in 3217b4b

	fn eval_cfgs(true_predicates: &BTreeSet<String>, attrs: &mut Vec<syn::Attribute>) -> Result<bool> {
	let mut cfg_attr_expansions = vec![];
	for (i, attr) in attrs.iter().enumerate() {
	let syntax = match CfgSyntax::from_meta(&attr.meta)? {
	Some(syntax) => syntax,
	_ => continue,
	};
	let expansion = match syntax {
	CfgSyntax::Cfg(predicate) => {
	if !true_predicates.contains(&predicate.to_string()) {
	// The cfg predicate is false. No need to keep iterating
	// because the entire syntax this attribute is attached to
	// should be removed. Signal the caller to remove it via `false`.
	return Ok(false);
	}
	// Just remove the attribute. It evaluated to `true`
	None
	}
	CfgSyntax::CfgAttr(cfg_attr) => {
	let predicate = cfg_attr.predicate.to_token_stream().to_string();
	// We can't both iterate over the attributes and mutate them,
	// so collect the planned actions in a separate vector, and
	// do the mutations after the iteration.
	true_predicates
	.contains(&predicate)
	.then(|| cfg_attr.then_branch)
	}
	};
	cfg_attr_expansions.push((i, expansion));
	}
	// It's important to iterate in reverse order to avoid index invalidation
	for (i, metas) in cfg_attr_expansions.iter().rev() {
	let metas = if let Some(metas) = metas {
	metas
	} else {
	attrs.remove(*i);
	continue;
	};
	let replacement = metas.iter().map(|meta| syn::parse_quote!(#[#meta]));
	attrs.splice(i..=i, replacement);
	}
	Ok(true)
	}

I implemented this logic, tested it and it indeed worked! Yes, but... it didn't work well with Rust Analyzer. The entire function body with the #[bon::builder] annotation was completely excluded from RA's analysis. No code completions and on-hover hints worked in it. Why? I'm not entirely sure, but I think what causes this is that function item is passed as an argument to the derive macro like #[expand_cfgs(item(fn greet(/**/) {/* */}))]. This somehow makes RA lose the span information, I guess.

I even tried a crazy hack of passing the function item in a nested expression inside of the enum's discriminant value expression. However, I stumbled with a bug in rustc, in that it invalidly expands such code (see rust-lang/rust#130004). I suppose no one ever tried doing such a crazy hack so no one ever caught such a bug.

So, in the end, I didn't use the derive syntax approach to evaluate the cfgs, instead I used the macro_rules! that can be observed here:

bon/bon/src/private/mod.rs

Lines 85 to 188 in 3217b4b

	/// This is all a big embarrassing workaround, please don't oversee 😳😳😳.
	///
	/// Anyway, if you are curious what the hell is going on here, then here is
	/// an explanation 😸. So... where to start 🤔. Ah! The problem!
	///
	/// ## The problem
	///
	/// Proc macro attributes (like `#[builder]`) see all the `#[cfg(...)]` and `#[cfg_attr(...)]`
	/// attributes unexpanded. For example, if you write smth like this:
	///
	/// ```
	/// #[bon::builder]
	/// fn func(
	/// #[cfg(windows)]
	/// windows_only_param: u32,
	/// ) {}
	///
	/// ```
	///
	/// then the `#[builder]` macro will see the full `#[cfg(...)]` attribute with
	/// the `windows_only_param` it is attached to verbatim. The `#[cfg(...)]` isn't
	/// removed by the time the `#[builder]`'s macro expansion is invoked.
	///
	/// It is a problem because the `#[builder]` macro needs to know the exact list
	/// of members it has to generate setters for. It doesn't know whether the
	/// the `windows` predicate evaluates to `true` or `false`, especially if this was
	/// a more complex predicate. So it can't decide whether to generate a setter for
	/// the `windows_only_param` or not.
	///
	/// ## The solution
	///
	/// This macro allows us to evaluate the `cfg` predicates by using a variation of
	/// [the trick] shared by @recatek.
	///
	/// When the `#[builder]` macro finds any usage of `#[cfg(...)]` or `#[cfg_attr(...)]`
	/// it generates a call to this macro with all `cfg` predicates collected from the
	/// item it was placed on. The `#[builder]` macro deduplicates and sorts the `cfg`
	/// predicates and passes them as `$pred` to this macro.
	///
	/// This macro then dispatches to `__eval_cfg_callback_true` or `__eval_cfg_callback_false`
	/// by defining a conditional `use ...` statement for each predicate and collects the
	/// results of the evaluation in the `$results` list.
	///
	/// For the last call to this macro (when no more `$pred` are left) the macro calls back
	/// to the proc macro attribute that called it with the results of the evaluation and
	/// the original parameters and the item which are passed through via the `$rest` macro variable.
	///
	/// [the trick]: https://users.rust-lang.org/t/supporting-or-evaluating-cfg-in-proc-macro-parameters/93240/2
	#[macro_export]
	#[doc(hidden)]
	macro_rules! __eval_cfg_callback {
	(
	{ $($results:tt)* }
	( $pred_id:ident: $($pred:tt)* )
	$($rest:tt)*
	) => {
	// The `pred_id` is required to be a unique identifier for the current
	// predicate evaluation so that we can use in a `use` statement to define
	// a new unique name for the macro to call.
	#[cfg($($pred)*)]
	#[doc(hidden)]
	use $crate::__eval_cfg_callback_true as $pred_id;
	#[cfg(not($($pred)*))]
	#[doc(hidden)]
	use $crate::__eval_cfg_callback_false as $pred_id;
	// The trick here is that `$pred_id` now resolves either to
	// `__eval_cfg_callback_true` or `__eval_cfg_callback_false`
	// depending on the evaluation of the cfg predicate, so by
	// invoking it as a macro, that macro internally pushes either
	// `true` or `false` to the `$results` list.
	$pred_id! {
	{ $($results)* }
	$($rest)*
	}
	};
	// The terminal case for the recursion when there are no more predicates left.
	// We have collected all the results of the cfg evaluations and now we can
	// delegate them to the proc macro attribute that called this macro.
	(
	// The results of the cfg evaluation
	{ $($results:tt)* }
	// The proc macro attribute to invoke with the results
	$final_macro:path,
	// The number of times this macro was called recursively from the proc macro
	$recursion_counter:literal,
	// Parameters to pass to the proc macro attribute after the cfg results
	( $($macro_params:tt)* )
	// The item to attach the proc macro attribute to
	$($item:tt)*
	) => {
	// The special `__cfgs(...)` prefix is parsed by the proc macro attribute
	// to get the results of the cfg evaluations.
	#[$final_macro(__cfgs($recursion_counter, $($results)*) $($macro_params)*)]
	$($item)*
	};
	}

)

So the #[bon::builder] macro generates a call to this macro_rules which basically does the similar job of expanding the cfgs, collecting the results and passing them as __cfg(...) in front of the callback proc macro attribute parameters. This, surprisingly works well with RA. RA can even recursively expand the macro with its "Expand macro recursively" feature.

Note that both of these solutions suffer from a very silly problem that you need to generate unique names for the struct under the derive (if you use the derive approach) or the symbols created by use statements in the macro_rules! approach current used in bon. You probably don't want macros to generate non-deterministic random identifiers on each rerun to avoid any unknown headaches that this may bring, so there is a bit of heuristic solution to this problem implemented here:

bon/bon-macros/src/normalization/cfg/mod.rs

Lines 129 to 189 in 3217b4b

	/// The macro `__eval_cfg_callback` needs to generate a use statement for
	/// every `cfg` predicate. To do that it needs to assign a unique name for
	/// every `use` statement so they doesn't collide with other items in
	/// the same scope and with each other.
	///
	/// But.. How in the world can we generate a unique name for every `use`
	/// if proc macros are supposed to be stateless and deterministic? 😳
	///
	/// We could use a random number here, but that would make the output
	/// non-deterministic, which is not good for reproducible builds and
	/// generally may lead to some unexpected headaches 🤧.
	///
	/// That's a silly problem, and here is a silly solution that doesn't
	/// work in 100% of the cases but it's probably good enough 😸.
	///
	/// We just need to use some existing name as a source of uniqueness.
	/// The name of the item under the macro is a good candidate for that.
	/// If the item is a function, then we can use the function name as that
	/// reliable source of uniqueness.
	///
	/// If the item is an `impl` block, then we have a bit of a problem because
	/// the `impl` block doesn't have a unique identifier attached to it, especially
	/// if the `self_ty` of the `impl` block isn't some simple syntax like a path.
	///
	/// However, in most of the cases it will be a simple path, so its combination
	/// with the name of the first function in the `impl` block should be unique enough.
	fn unique_invocation_name(&self) -> Result<String> {
	let path_to_ident =
	|path: &syn::Path| path.segments.iter().map(|segment| &segment.ident).join("_");
	// Include the name of the proc macro in the unique name to avoid
	// collisions when different proc macros are placed on the same item
	// and they use this code to generate unique names.
	let macro_path_str = path_to_ident(&self.macro_path);
	let item_name = match &self.item {
	syn::Item::Fn(item) => item.sig.ident.to_string(),
	syn::Item::Impl(item) => {
	let self_ty = item
	.self_ty
	.as_path()
	.map(|path| path_to_ident(&path.path))
	.unwrap_or_default();
	let first_fn = item
	.items
	.iter()
	.find_map(|item| match item {
	syn::ImplItem::Fn(method) => Some(method.sig.ident.to_string()),
	_ => None,
	})
	.unwrap_or_default();
	format!("impl_{self_ty}_fn_{first_fn}")
	}
	_ => bail!(&Span::call_site(), "Unsupported item type"),
	};
	Ok(format!("__eval_cfg_{macro_path_str}_{item_name}"))
	}
	}

[jplatte]

I think the __ExpandCfgs approach is very different from what I suggested on reddit, I don't fully understand it but it seems to be trying to get the info of which cfgs evaluate to true / false into the attribute parameters, such that the attribute can then evaluate cfgs it sees.

That is not what I was talking about: Rather, the idea would be that #[bon::builder] could delegate most of its work (of generating impl blocks) to a derive macro¹, which would make rustc remove all cfg(<evals to false>) arguments for you. Assuming bon were to generate untyped builders with everything just being Option... I know it's not nearly this simple, and maybe that actually means this idea doesn't work, but maybe it does:

// user written code:
#[bon::builder]
fn greet(
    #[cfg(all())] name: &str,
    #[cfg(any())] level: Option<u32>,
) -> String {
    format!("Hello {name}!")
}

// expands to
fn greet() -> GreetBuilder {
    GreetBuilder {
        #[cfg(all())]
        name: None,
        #[cfg(any())]
        level: None,
    }
}

#[derive(bon::Builder)]
#[bon(__private_fn_builder)]
struct GreetBuilder<'a> {
    #[cfg(all())]
    name: Option<&'a str>,
    #[cfg(any(())]
    level: Option<Option<u32>>,
}

// what the Builder derive sees
#[bon(__private_fn_builder)]
struct GreetBuilder<'a> {
    name: Option<&'a str>,
}

// so it can generate an `impl<'a> GreetBuilder<'a>`
// knowing that there is (only) a name field

Am I talking nonsense? I applied this pattern successfully before so I know the general idea works, but maybe it's not applicable here.

¹ either always, or only if there are any cfgs / cfg_attrs in the input as an optimization

[Veetaha]

Aha, I see what's your suggestion now. It makes sense indeed, although it would be rather complicated to apply to bon because the current generated builder looks like this:

type GreetBuilderInitialState = (
    Unset<Required>,
    Unset<Optional>,
);

struct GreetBuilder<State = GreetBuilderInitialState> {
    __private_members: State,
    /* other unimportant stuff */
}

Where the State is a type state expressed as a tuple with one element per each function argument (and attributes aren't supported inside of tuple types). It's theoretically possible to update the codegen to have a field per member in the builder, but that leads to other complications (and bon already was there).

However, the idea looks nice, I'll keep it in mind! If I find other limitations with the current approach I'll consider it, but yeah, it's not easily applicable in the current design of the builder struct. Anyway thank you, for the suggestion🐱

[jplatte]

Huh, I didn't know that attributes are forbidden on tuple fields!
I wonder why that is, considering that it works with tuple structs.

[Veetaha]

Yeah, there are a lot of places in Rust where attributes aren't supported (yet?). It's really easy to check which context supports attributes by just reviewing the syn's AST models. For example the TypeTuple in question here doesn't have them, and I think attributes are not allowed inside of basically any type.

The Field of the tuple struct does have the attributes.