GH-46777: [C++] Use SimplifyIsIn only when the value_set of the expression is lower than a threshold

[raulcd]

Rationale for this change

Using SimplifyIsIn when the value set is large has a substantial performance penalty.

What changes are included in this PR?

Ensure we do not use the simplification when the value_set on the expression is higher than a threshold (50).

Are these changes tested?

I've tested locally that the reproducer goes back to pre change levels.

$ python read.py 
=== PYARROW VERSION 20 ===
Retrieved 10,000,000 rows in 3.08 seconds.

I have added a test for large sets and validate the expression is not being modified.

Are there any user-facing changes?

No

• GitHub Issue: [Python] Severe performance regression in isin() filter After pyarrow v18 #46777

[github-actions]

⚠️ GitHub issue #46777 has been automatically assigned in GitHub to PR creator.

[raulcd]

@zanmato1984 @pitrou how would you proceed about testing this? Should I just try to add a new benchmark that exercises this scenario? The current Python reproducer has validated the solution fixes the original reported problem and there are currently tests with less values but I haven't found tests with a high value_set.

[pitrou]

Do we have any benchmarks for expression simplification already? Otherwise, we shouldn't bother adding any.

[zanmato1984]

how would you proceed about testing this? Should I just try to add a new benchmark that exercises this scenario?

I think it's good to have both tests and benchmarks. I assume the test shouldn't be very heavy (50 elements value set), is it?

[pitrou]

It would be nice to have this in 21.0. Do you want to update this PR @raulcd ?

[raulcd]

Sure, I am working on it at the moment, will try to push soon

[raulcd]

I don't know our test utilities but couldn't find something like this, do we have any utility to do this?

[pitrou]

We don't.

[pitrou]

+1 except for two nits

[raulcd]

👍

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 0b34e6b.

There were 9 benchmark results indicating a performance regression:

• Commit Run on test-mac-arm at 2025-07-07 11:41:06Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-10, scale_factor=1
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-11, scale_factor=1
• and 7 more (see the report linked below)

The full Conbench report has more details. It also includes information about 53 possible false positives for unstable benchmarks that are known to sometimes produce them.