Skip to content

Conversation

ggevay
Copy link
Contributor

@ggevay ggevay commented Sep 3, 2025

Fixes https://github.com/MaterializeInc/database-issues/issues/9643 by making memoize_expr (which is called from extract_common) not go into temporal filters.

Note that inline_expressions solves this problem when full MFP optimization is run, but extract_common only runs memoize_expressions.

The PR adds a regression test in temporal.slt. (temporal.td also tests various temporal filters.) Additionally, the requirement that the non-mz_now side of a temporal filter should still be processed by memoize_expressions is tested by the existing EXPLAINs in filter-pushdown.slt.

The PR also makes the existing tests slightly stronger by putting actual data into one of the existing tables, and actually querying the existing materialized views to check for evaluation errors, in addition to planning errors.

Nightly (subset): https://buildkite.com/materialize/nightly/builds/13060

Edit: I've thought about feature flagging this change, but unfortunately the optimizer flags are not wired to this part of the code. But I think this is a low-risk change.

New Nightly run: https://buildkite.com/materialize/nightly/builds/13227

Motivation

Tips for reviewer

Checklist

  • This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
  • This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
  • If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
  • If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
  • If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

@ggevay ggevay added the A-optimization Area: query optimization and transformation label Sep 3, 2025
@ggevay ggevay force-pushed the extract_common-vs-mznow branch from 030fda2 to 254bc48 Compare September 3, 2025 12:56
soft_assert_or_log!(
!is_temporal_filter,
"MfpPlan::create_from and is_temporal_filter should agree"
);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm also open to refactoring these into one method if deemed worthwhile.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eh, seems fine as is. Maybe worth adding a comment that temporal filters are necessarily binary operations?

Copy link
Contributor Author

@ggevay ggevay Sep 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eh, seems fine as is.

Ok, I'm relieved :)

Maybe worth adding a comment that temporal filters are necessarily binary operations?

Below the let is_temporal_filter = line, there is the following old comment:

// Supported temporal predicates are exclusively binary operators.

Or did you mean in the doc comment of the fn is_temporal_filter? That kinda has it already, albeit somewhat implicitly, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(I ended up factoring this out, due to Frank's comment.)

@ggevay ggevay marked this pull request as ready for review September 3, 2025 13:27
@ggevay ggevay requested a review from a team as a code owner September 3, 2025 13:27
Copy link
Contributor

@mgree mgree left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor nits---looks good to me!

soft_assert_or_log!(
!is_temporal_filter,
"MfpPlan::create_from and is_temporal_filter should agree"
);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eh, seems fine as is. Maybe worth adding a comment that temporal filters are necessarily binary operations?

@frankmcsherry
Copy link
Contributor

To discuss: why not use MapFilterProject::extract_temporal, and ultimately SafeMfpPlan to go from MFPs containing temporal operators to those with and without. (( i.e. I can't tell why the approach here is the right solution, vs using existing methods ))

Copy link
Contributor

@frankmcsherry frankmcsherry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unless there is a rush, I think there's potentially a clearer way to do this, where we centralize the "am I a temporal filter" logic, deduplicating that and removing the need for tests that the two bits of logic agree.

/// - no mz_now() anywhere inside the other side of the comparison.
///
/// If yes, then it returns the other side of the comparison.
pub fn is_temporal_filter(&mut self) -> Option<&mut MirScalarExpr> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit, but naming-wise I would expect is_foo(&mut self) -> ... to return a bool and not be &mut. Perhaps instead as_mut_temporal_filter_expr? That's a bit horrid, but probably clearer when used and less surprising.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reading the other changes, I think what we actually want here is

/// If `self` expresses a temporal filter, normalize it to start with `mz_now()` and return references.
/// 
/// A temporal filter is an expression of the form `mz_now() <BINOP> <EXPR>`,
/// for a restricted set of `BINOP` and `EXPR` that do not themselves contain `mz_now()`.
/// Expressions may conform to this once their expressions are swapped.
///
/// If the expression is not a temporal filter it will be unchanged.
pub fn as_mut_temporal_filter(&mut self) -> Option<(&BinaryFunc, &mut MirScalarExpr)>

which I think centralizes the logic in one place, rather than have two flavors and tests to make sure they stay the same.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, ok, this is what I meant in https://github.com/MaterializeInc/materialize/pull/33500/files#r2318981605
If you deem this refactoring worthwhile, then I'll do it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, see the latest commit.

@ggevay ggevay changed the title Avoid deconstructing temporal filters in ANF Avoid deconstructing temporal filters in memoize_expr Sep 8, 2025
@ggevay ggevay force-pushed the extract_common-vs-mznow branch from 06ee435 to 25e178c Compare September 8, 2025 14:44
@ggevay ggevay force-pushed the extract_common-vs-mznow branch from 25e178c to a7545e2 Compare September 8, 2025 14:59
Copy link
Contributor

@frankmcsherry frankmcsherry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code changes look great, thank you!

@ggevay
Copy link
Contributor Author

ggevay commented Sep 8, 2025

Thanks for the review! (The Nightly fail looks unrelated.)

@ggevay ggevay merged commit dc9ce2d into MaterializeInc:main Sep 8, 2025
300 of 316 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-optimization Area: query optimization and transformation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants