Skip to content

[FLINK-36962] Avoid pushing down non-deterministic filter in FILTER_RULES #26747

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

Au-Miner
Copy link
Contributor

@Au-Miner Au-Miner commented Jul 3, 2025

What is the purpose of the change

Currently, filters with non deterministic features will be pushed down to join/agg/union, which can result in errors.

Brief change log

  • Added non deterministic check in rules in FILTER_RULE.
  • Add Test and ITCase in related operator.

Verifying this change

Existent tests and new added tests can verify this change.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@flinkbot
Copy link
Collaborator

flinkbot commented Jul 3, 2025

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@Au-Miner Au-Miner force-pushed the FLINK-36962 branch 2 times, most recently from af7a18d to 36a8b17 Compare July 9, 2025 09:57
@Au-Miner Au-Miner marked this pull request as ready for review July 9, 2025 09:57
@Au-Miner Au-Miner changed the title [FLINK-36962] Avoid pushing down non-deterministic filter after stream join to source [FLINK-36962] Avoid pushing down non-deterministic filter in FILTER_RULES Jul 9, 2025
@Au-Miner Au-Miner changed the title [FLINK-36962] Avoid pushing down non-deterministic filter in FILTER_RULES [FLINK-38089] Avoid pushing down non-deterministic filter in FILTER_RULES Jul 11, 2025
@Au-Miner Au-Miner changed the title [FLINK-38089] Avoid pushing down non-deterministic filter in FILTER_RULES [FLINK-36962] Avoid pushing down non-deterministic filter in FILTER_RULES Jul 11, 2025
@Au-Miner Au-Miner closed this Jul 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants