Temporary lifetime extension for blocks

[dianne]

This implements a revised version of the temporary lifetime extension semantics I suggested in #145838 (comment), with the goal of making temporary lifetimes and drop order more consistent between extending and non-extending blocks. As a consequence, this undoes the breaking change introduced by #145838 (but in exchange has a much larger surface area).

The change this PR hopes to enforce is a general rule: any expression's temporaries should have the same relative drop order regardless of whether the expression is in an extending context or not: let _ = $expr; and drop($expr); should have the same drop order. To achieve that, this PR applies lifetime extension rules bottom-up, starting from borrow expressions and super let, rather than top-down from let statements and consts/statics. For example:

// This `temp()` is now extended past the block tail in all contexts.
{ &temp() }

now extends the lifetime of temp() to outlive the block tail in Rust 2024 regardless of whether the block is an extending expression in a let statement initializer (in which context it was already extended to outlive the block before this PR). The scoping rules for tails of extending blocks remain the same: extending subexpressions' temporary scopes are extended based on the source of the lifetime extension (e.g. to match the scope of a parent let statement's bindings). For blocks not extended by any other source, extending borrows in the tail expression now share a temporary scope with the result of the block. This can in turn extend nested blocks within blocks' tail expressions:

// This `temp()` is extended past the outer block tail.
// It is now dropped after the reference to it at the `;`.
f({{ &temp() }});

// This context-sensitivity is consistent with `let`:
// This `temp()` was already extended.
// It is still dropped after `x` at the end of its scope.
let x = {{ &temp() }};

Since this uses the same rules as let, it only applies to extending sub-expressions.

// This `temp()` is still never extended in any context.
// In Rust 2024, it is dropped at the end of the block tail.
{ identity(&temp()) }

This also applies to if expressions' blocks and to match arms in all editions, since lifetime extension applies to both of them as well and they both drop their non-extended temporaries. This is where breakage from #145838 was observed:

if cond { &temp() } else { &temp() }

now extends temp() to have the same temporary scope as the result of the if expression.

As a further consequence, this makes super let in if expressions' blocks more consistent with block expressions:

if cond() {
    super let x = temp();
    &temp
} else {
    super let x = temp();
    &temp
}

previously only worked in extending contexts (since the super lets would be extended), and now it works everywhere.

Reference PR: rust-lang/reference#2051
Edition Guide PR: rust-lang/edition-guide#379

@rustbot label +T-lang

[dianne]

@rustbot label -stable-nominated

I'm not intending to stable-nominate this, at least. Someone else can, but I don't expect it's needed or that it would be accepted.

[traviscross]

Does this only affect code in Rust 2024, or would you expect any visible difference in earlier editions?

[dianne]

It should only be visible in Rust 2024. The only extending expressions that introduce temporary drop scopes are Rust 2024 block tail expressions. Edit: this is also visible on earlier editions through if expressions' blocks.

Suppose we have a macro extending!, for which $expr is extending if extending!($expr) is extending. Under this PR, in a non-extending context, { extending!(&temp()) } would give temp() the same temporary scope as the result of the block. Prior to Rust 2021, they're already in the same scope, due to extending! being unable to introduce temporary scopes.

Or to generalize this, the aim of this PR is that in a non-extending context, extending!(&temp()) should give temp() the same temporary scope as the expansion, similar to how let x = extending!(&temp()); gives temp() the same scope as x. This already holds in Rust 2021 and prior.

If new expressions are added to Rust that are both extending and temporary scopes, I'd want this behavior to apply to them as well.

[traviscross]

Since this would effectively reduce the scope of the Rust 2024 tail expression temporary scope change, we'd also want to be sure to reflect that in the behavior of the tail-expr-drop-order lint.

[dianne]

I haven't done extensive testing, but see this test diff for that lint: lint-tail-expr-drop-order-borrowck.rs. I'm applying the lifetime extension rules on all editions, and lifetime extension prevents the temporary scope from being registered as potentially forwards-incompatible (even though the extended scopes are technically the same as the old scopes in old editions). Though I think I've convinced myself at this point that lifetime extension doesn't need to be applied to block tails of non-extending old-edition blocks¹, so potentially the lint change could be implemented in some other way instead.

Footnotes

1. I was worried about mixed-edition code, but I don't think it's an issue anymore. ↩

[bors]

☔ The latest upstream changes (presumably #146666) made this pull request unmergeable. Please resolve the merge conflicts.

[dianne]

I've made some revisions. This should now properly handle if expressions' blocks, meaning it affects all editions (since if blocks are both terminating in all editions and extending when the if expression is extending). Of note, I didn't notice at the time, but I think #145838 affected all editions as well (including the real-world breakage), due to if blocks working like that.

I think the implementation will likely need optimization and cleanup, but it might take a bit of refactoring to get it to a good place, so I'd like to get a vibe check on the design first, if there's room for it in a lang team meeting.

@rustbot label +I-lang-nominated

[rustbot]

r? @nnethercote

rustbot has assigned @nnethercote.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[craterbot]

🚧 Experiment pr-146098-10 is now running

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[craterbot]

🎉 Experiment pr-146098-10 is completed!
📊 4 regressed and 5 fixed (53 total)
📊 20 spurious results on the retry-regessed-list.txt, consider a retry¹ if this is a significant amount.
📰 Open the summary report.

⚠️ If you notice any spurious failure please add them to the denylist!
ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

Footnotes

1. re-run the experiment with crates=https://crater-reports.s3.amazonaws.com/pr-146098-10/retry-regressed-list.txt ↩

[dianne]

The ryanpbrewster.actix-interrupt and ncomm-executors test failures both look nondeterministic. The failure in ryanpbrewster.actix-interrupt was a test that sleeps for a long time, which made it prone to timing out, and the failure in ncomm-executors was a test that measures timing.

[BoxyUwU]

I would have expected that whenever we enter an extending node we update the extended_parent. Why do we only set an extended parent if the scope drops temps?

[dianne]

It's to avoid HashMap insertions by only setting the extended_parent when it will give extendible temporaries non-default temporary scopes. The result should be the same in either case, but when the extended_parent is set we record borrow expressions' operands' temporary scopes in a table, which is probably best to avoid. I'll add a comment with a better explanation.

As for why it's the same, consider the context of a borrow expression in the HIR; we need to check the extended_parent to determine its operand's temporary scope. It'll have some (possibly empty) chain of extending: true nodes directly above it.

• If the chain leads all the way to a let statement, const, or static, the extended_parent will have been set already by resolve_local; the borrow operand will have an extended temporary scope.
• Otherwise, if there's no drop_temps: true node in the extending: true chain, the borrow operand's temporary scope will be its default temporary scope. Setting the extended_parent to the node above the chain won't change this, since there's no drop_temps: true nodes in it that would affect the default temporary scope. As such, we can leave it unset.
• If there is a drop_temps: true node in the extending: true chain, we should make sure the borrow operand lives past it; that's the change this PR makes. To do that, we use the default temporary scope at whatever expression started the extending: true chain. e.g. in f({ &temp() }), the outermost node in the extending chain is the block tail subexpression, so we use the default temporary scope at the block expression for the borrow operand. The trick here is the same as in the previous case: the only thing that will change the default temporary scope is drop_temps: true nodes, so we can put off actually setting the extended_parent until we hit one.

[dianne]

Another option here would be to get rid of the ScopeTree altogether (#97426). It's effectively a second copy of each HIR body but stored in an IndexMap. We could probably save a lot of work (and reallocations) by not building it. That'd be a bigger refactor, but it's one I've been thinking about anyway. The way I imagine it, without the ScopeTree, we also wouldn't have a special table for lifetime-extended temporaries, so we wouldn't need to worry about minimizing insertions into it.

[dianne]

Updated the comment to add some more detail: (diff)

[BoxyUwU]

I struggled a bit to understand this function due to it using a loop instead of being a recursive fn. Do you know if there's a reason why it's not like that? If it can be a recursive fn want to open a separate PR changing that?

[dianne]

I'm not actually sure why it's a loop. I'll look into it. I have a suspicion it used to be recursive; it was previously named record_rvalue_scope_rec and it was called by a function named record_rvalue_scope, so there's probably some justification in a commit message or PR description somewhere. I'd also find recursion more readable, so I'll see if that justification still applies.

[dianne]

Looks like it was always a loop. The split was introduced in #95563, and it was a loop both before and after that. As far as I'm aware, it's just a stylistic choice (assuming tail recursion optimizes properly). I've opened #150548 to make it recursive.

[BoxyUwU]

How much of this PR is actually user-facing. Is it theoretically possible to split out like 99% of this PR but with blocks as having extending: false and then this PR is just flipping a bool?

[dianne]

The actual change this PR makes is the addition of this conditional assignment. e037e93 is a self-contained refactor that helps make it expressible, and 8c84070 is the less self-contained stuff, including the actual change. I could split things apart further if that would help.

As for blocks, they should already be extending: true in stable. The change this PR makes is in what extending: true means:

• Currently (and in that first commit), if you have an unbroken chain of extending: true subexpressions starting at the initializer of a let statement¹ and going to an expression with a lifetime-extendible temporary², that expression's lifetime-extendible temporaries are lifetime-extended to match scope of the let statement's bindings. This means the temp() in { &temp() } only outlives the block tail if the block appears in a lifetime-extending context.
• After this PR's change, I have a different mental model: the lifetime-extendible temporaries of an extending: true subexpression are treated as lifetime-extendible temporaries of the parent expression (and if they become lifetime-extendible temporaries of a let statement initializer, they're lifetime-extended to match the scope of its bindings). This means the temp() in { &temp() } always outlives the block tail, but how far it lives past that depends on the context the block is in.

Footnotes

1. The body expressions of static and constant items and the tail expressions of const blocks are treated like let statement initializers for this purpose. ↩

2. Lifetime-extendible temporaries are produced by borrow operators (which lifetime-extend their operands if they become temporaries) and blocks with super let statements (which lifetime-extend their super lets). ↩

[bors]

☔ The latest upstream changes (presumably #150559) made this pull request unmergeable. Please resolve the merge conflicts.

[rustbot]

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.