Skip to content

Use maybe constant in duckdb convert #3258

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

Use maybe constant in duckdb convert #3258

wants to merge 2 commits into from

Conversation

joseph-isaacs
Copy link
Contributor

No description provided.

@joseph-isaacs
fix
fd48df5 
Signed-off-by: Joe Isaacs <[email protected]>
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented May 9, 2025

Benchmarks: TPC-H on NVME

Table of Results
name PR fd48df5 base 4ba0763 ratio (PR/base) unit
tpch_q01/DataFusion:arrow 66353621 6.30896e+07 1.05174 ns
tpch_q02/DataFusion:arrow 16628414 1.60858e+07 1.03373 ns
tpch_q03/DataFusion:arrow 24235450 2.40555e+07 1.00748 ns
tpch_q04/DataFusion:arrow 19682385 1.94899e+07 1.00988 ns
tpch_q05/DataFusion:arrow 32880483 3.26827e+07 1.00605 ns
tpch_q06/DataFusion:arrow 6497996 6.46192e+06 1.00558 ns
tpch_q07/DataFusion:arrow 53621257 5.41268e+07 0.990661 ns
tpch_q08/DataFusion:arrow 18984507 1.95661e+07 0.970277 ns
tpch_q09/DataFusion:arrow 34479307 3.50258e+07 0.984398 ns
tpch_q10/DataFusion:arrow 38373917 3.94352e+07 0.973087 ns
tpch_q11/DataFusion:arrow 7932266 7.85888e+06 1.00934 ns
tpch_q12/DataFusion:arrow 17494867 1.83858e+07 0.951542 ns
tpch_q13/DataFusion:arrow 15174619 1.49341e+07 1.01611 ns
tpch_q14/DataFusion:arrow 8135218 8.22721e+06 0.988818 ns
tpch_q15/DataFusion:arrow 18496868 1.81441e+07 1.01945 ns
tpch_q16/DataFusion:arrow 14796501 1.50676e+07 0.982006 ns
tpch_q17/DataFusion:arrow 47864781 5.01335e+07 0.954747 ns
tpch_q18/DataFusion:arrow 105958698 1.06813e+08 0.992 ns
tpch_q19/DataFusion:arrow 17762332 1.73871e+07 1.02158 ns
tpch_q20/DataFusion:arrow 23729005 2.50055e+07 0.948953 ns
tpch_q21/DataFusion:arrow 83868271 8.23785e+07 1.01808 ns
tpch_q22/DataFusion:arrow 10553749 1.08832e+07 0.969725 ns
tpch_q01/DataFusion:parquet 103544481 1.03985e+08 0.99576 ns
tpch_q02/DataFusion:parquet 51041406 5.17403e+07 0.986492 ns
tpch_q03/DataFusion:parquet 47725424 4.85838e+07 0.982333 ns
tpch_q04/DataFusion:parquet 31659068 3.23548e+07 0.978496 ns
tpch_q05/DataFusion:parquet 64496114 6.57862e+07 0.98039 ns
tpch_q06/DataFusion:parquet 28741050 2.89758e+07 0.991898 ns
tpch_q07/DataFusion:parquet 81969169 8.40089e+07 0.97572 ns
tpch_q08/DataFusion:parquet 59973605 6.0937e+07 0.984191 ns
tpch_q09/DataFusion:parquet 85870395 8.55858e+07 1.00332 ns
tpch_q10/DataFusion:parquet 95272341 9.64794e+07 0.987489 ns
tpch_q11/DataFusion:parquet 29356088 2.90249e+07 1.01141 ns
tpch_q12/DataFusion:parquet 41306507 4.01834e+07 1.02795 ns
tpch_q13/DataFusion:parquet 45006700 4.39738e+07 1.02349 ns
tpch_q14/DataFusion:parquet 33211072 3.39992e+07 0.976819 ns
tpch_q15/DataFusion:parquet 54257208 5.45014e+07 0.995519 ns
tpch_q16/DataFusion:parquet 35215289 3.54012e+07 0.994748 ns
tpch_q17/DataFusion:parquet 97282969 9.49789e+07 1.02426 ns
tpch_q18/DataFusion:parquet 117206524 1.17654e+08 0.996201 ns
tpch_q19/DataFusion:parquet 71500997 7.40524e+07 0.965546 ns
tpch_q20/DataFusion:parquet 56723985 5.87408e+07 0.965666 ns
tpch_q21/DataFusion:parquet 102206964 1.03131e+08 0.991041 ns
tpch_q22/DataFusion:parquet 51288119 5.21795e+07 0.982916 ns
tpch_q01/DataFusion:vortex-file-compressed 66553752 6.45365e+07 1.03126 ns
tpch_q02/DataFusion:vortex-file-compressed 39358056 4.02487e+07 0.977871 ns
tpch_q03/DataFusion:vortex-file-compressed 32929055 3.31058e+07 0.994662 ns
tpch_q04/DataFusion:vortex-file-compressed 16009746 1.66131e+07 0.963681 ns
tpch_q05/DataFusion:vortex-file-compressed 49691058 5.00318e+07 0.99319 ns
tpch_q06/DataFusion:vortex-file-compressed 4693757 5.07536e+06 0.924812 ns
tpch_q07/DataFusion:vortex-file-compressed 65847091 6.83338e+07 0.963609 ns
tpch_q08/DataFusion:vortex-file-compressed 54960046 5.70951e+07 0.962606 ns
tpch_q09/DataFusion:vortex-file-compressed 77490947 7.72096e+07 1.00364 ns
tpch_q10/DataFusion:vortex-file-compressed 46485406 4.65384e+07 0.998861 ns
tpch_q11/DataFusion:vortex-file-compressed 26347298 2.68468e+07 0.981394 ns
tpch_q12/DataFusion:vortex-file-compressed 18438511 1.82861e+07 1.00833 ns
tpch_q13/DataFusion:vortex-file-compressed 19498725 1.89195e+07 1.03062 ns
tpch_q14/DataFusion:vortex-file-compressed 12552869 1.28791e+07 0.974668 ns
tpch_q15/DataFusion:vortex-file-compressed 23213355 2.31348e+07 1.0034 ns
tpch_q16/DataFusion:vortex-file-compressed 23337527 2.26932e+07 1.02839 ns
tpch_q17/DataFusion:vortex-file-compressed 77951286 7.89707e+07 0.987091 ns
tpch_q18/DataFusion:vortex-file-compressed 95652841 9.59999e+07 0.996385 ns
tpch_q19/DataFusion:vortex-file-compressed 18169718 1.8525e+07 0.980823 ns
tpch_q20/DataFusion:vortex-file-compressed 27629025 2.76055e+07 1.00085 ns
tpch_q21/DataFusion:vortex-file-compressed 88240110 8.88753e+07 0.992853 ns
tpch_q22/DataFusion:vortex-file-compressed 12914262 1.43129e+07 0.902284 ns
tpch_q01/DuckDB:parquet 64000000 6.5e+07 0.984615 ns
tpch_q02/DuckDB:parquet 36000000 3.7e+07 0.972973 ns
tpch_q03/DuckDB:parquet 80000000 7.9e+07 1.01266 ns
tpch_q04/DuckDB:parquet 38000000 3.9e+07 0.974359 ns
tpch_q05/DuckDB:parquet 58000000 5.8e+07 1 ns
tpch_q06/DuckDB:parquet 31000000 3e+07 1.03333 ns
tpch_q07/DuckDB:parquet 62000000 6.3e+07 0.984127 ns
tpch_q08/DuckDB:parquet 70000000 6.9e+07 1.01449 ns
tpch_q09/DuckDB:parquet 100000000 1e+08 1 ns
tpch_q10/DuckDB:parquet 327000000 3.28e+08 0.996951 ns
tpch_q11/DuckDB:parquet 18000000 1.9e+07 0.947368 ns
tpch_q12/DuckDB:parquet 46000000 4.5e+07 1.02222 ns
tpch_q13/DuckDB:parquet 55000000 5.7e+07 0.964912 ns
tpch_q14/DuckDB:parquet 49000000 5e+07 0.98 ns
tpch_q15/DuckDB:parquet 62000000 6.5e+07 0.953846 ns
tpch_q16/DuckDB:parquet 69000000 7e+07 0.985714 ns
tpch_q17/DuckDB:parquet 62000000 6.1e+07 1.01639 ns
tpch_q18/DuckDB:parquet 76000000 7.5e+07 1.01333 ns
tpch_q19/DuckDB:parquet 71000000 7e+07 1.01429 ns
tpch_q20/DuckDB:parquet 60000000 6e+07 1 ns
tpch_q21/DuckDB:parquet 117000000 1.18e+08 0.991525 ns
tpch_q22/DuckDB:parquet 46000000 4.5e+07 1.02222 ns
tpch_q01/DuckDB:vortex-file-compressed 53000000 5.4e+07 0.981481 ns
tpch_q02/DuckDB:vortex-file-compressed 40000000 4.3e+07 0.930233 ns
tpch_q03/DuckDB:vortex-file-compressed 78000000 7.9e+07 0.987342 ns
tpch_q04/DuckDB:vortex-file-compressed 40000000 4.1e+07 0.97561 ns
tpch_q05/DuckDB:vortex-file-compressed 1136000000 1.138e+09 0.998243 ns
tpch_q06/DuckDB:vortex-file-compressed 15000000 1.6e+07 0.9375 ns
tpch_q07/DuckDB:vortex-file-compressed 70000000 7.3e+07 0.958904 ns
tpch_q08/DuckDB:vortex-file-compressed 76000000 7.7e+07 0.987013 ns
tpch_q09/DuckDB:vortex-file-compressed 99000000 1.01e+08 0.980198 ns
tpch_q10/DuckDB:vortex-file-compressed 317000000 3.21e+08 0.987539 ns
tpch_q11/DuckDB:vortex-file-compressed 26000000 2.7e+07 0.962963 ns
tpch_q12/DuckDB:vortex-file-compressed 39000000 4.2e+07 0.928571 ns
tpch_q13/DuckDB:vortex-file-compressed 82000000 8.9e+07 0.921348 ns
tpch_q14/DuckDB:vortex-file-compressed 34000000 3.7e+07 0.918919 ns
tpch_q15/DuckDB:vortex-file-compressed 60000000 6.1e+07 0.983607 ns
tpch_q16/DuckDB:vortex-file-compressed 70000000 7.1e+07 0.985915 ns
tpch_q17/DuckDB:vortex-file-compressed 53000000 5.5e+07 0.963636 ns
tpch_q18/DuckDB:vortex-file-compressed 72000000 7.4e+07 0.972973 ns
tpch_q19/DuckDB:vortex-file-compressed 48000000 5.1e+07 0.941176 ns
tpch_q20/DuckDB:vortex-file-compressed 47000000 5e+07 0.94 ns
tpch_q21/DuckDB:vortex-file-compressed 285000000 2.96e+08 0.962838 ns
tpch_q22/DuckDB:vortex-file-compressed 38000000 4e+07 0.95 ns
tpch_q01/DuckDB:duckdb 27000000 2.7e+07 1 ns
tpch_q02/DuckDB:duckdb 21000000 2.1e+07 1 ns
tpch_q03/DuckDB:duckdb 56000000 5.5e+07 1.01818 ns
tpch_q04/DuckDB:duckdb 23000000 2.4e+07 0.958333 ns
tpch_q05/DuckDB:duckdb 33000000 3.3e+07 1 ns
tpch_q06/DuckDB:duckdb 10000000 1e+07 1 ns
tpch_q07/DuckDB:duckdb 31000000 3.2e+07 0.96875 ns
tpch_q08/DuckDB:duckdb 31000000 3.3e+07 0.939394 ns
tpch_q09/DuckDB:duckdb 69000000 6.9e+07 1 ns
tpch_q10/DuckDB:duckdb 284000000 2.84e+08 1 ns
tpch_q11/DuckDB:duckdb 11000000 1.2e+07 0.916667 ns
tpch_q12/DuckDB:duckdb 30000000 2.9e+07 1.03448 ns
tpch_q13/DuckDB:duckdb 42000000 4.2e+07 1 ns
tpch_q14/DuckDB:duckdb 30000000 3e+07 1 ns
tpch_q15/DuckDB:duckdb 38000000 4.1e+07 0.926829 ns
tpch_q16/DuckDB:duckdb 60000000 6e+07 1 ns
tpch_q17/DuckDB:duckdb 30000000 2.9e+07 1.03448 ns
tpch_q18/DuckDB:duckdb 46000000 4.6e+07 1 ns
tpch_q19/DuckDB:duckdb 43000000 4.3e+07 1 ns
tpch_q20/DuckDB:duckdb 34000000 3.5e+07 0.971429 ns
tpch_q21/DuckDB:duckdb 65000000 6.5e+07 1 ns
tpch_q22/DuckDB:duckdb 32000000 3.3e+07 0.969697 ns

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented May 9, 2025

Benchmarks: TPC-H on S3

Table of Results
name PR fd48df5 base 4ba0763 ratio (PR/base) unit
tpch_q01/DataFusion:parquet 221055413 2.20501e+08 1.00251 ns
tpch_q02/DataFusion:parquet 759309834 7.70195e+08 0.985867 ns
tpch_q03/DataFusion:parquet 359473347 3.50557e+08 1.02543 ns
tpch_q04/DataFusion:parquet 248785974 2.35529e+08 1.05629 ns
tpch_q05/DataFusion:parquet 605373644 5.59647e+08 1.08171 ns
tpch_q06/DataFusion:parquet 146669929 1.51101e+08 0.970676 ns
tpch_q07/DataFusion:parquet 552963753 5.10152e+08 1.08392 ns
tpch_q08/DataFusion:parquet 780459752 7.75248e+08 1.00672 ns
tpch_q09/DataFusion:parquet 632630206 5.99383e+08 1.05547 ns
tpch_q10/DataFusion:parquet 535392302 5.46267e+08 0.980092 ns
tpch_q11/DataFusion:parquet 486717885 4.72106e+08 1.03095 ns
tpch_q12/DataFusion:parquet 295192353 3.10627e+08 0.950312 ns
tpch_q13/DataFusion:parquet 196200237 2.27351e+08 0.862983 ns
tpch_q14/DataFusion:parquet 285547214 3.01072e+08 0.948436 ns
tpch_q15/DataFusion:parquet 397510062 3.97375e+08 1.00034 ns
tpch_q16/DataFusion:parquet 375884644 3.65088e+08 1.02957 ns
tpch_q17/DataFusion:parquet 547127541 5.24247e+08 1.04364 ns
tpch_q18/DataFusion:parquet 485651784 4.85544e+08 1.00022 ns
tpch_q19/DataFusion:parquet 397870370 3.84672e+08 1.03431 ns
tpch_q20/DataFusion:parquet 517008059 5.36924e+08 0.962908 ns
tpch_q21/DataFusion:parquet 708526232 7.2543e+08 0.976699 ns
tpch_q22/DataFusion:parquet 289244068 2.69097e+08 1.07487 ns
tpch_q01/DataFusion:vortex-file-compressed 368435686 3.32716e+08 1.10736 ns
tpch_q02/DataFusion:vortex-file-compressed 156715831 1.7604e+08 0.890228 ns
tpch_q03/DataFusion:vortex-file-compressed 352695884 3.57581e+08 0.986338 ns
tpch_q04/DataFusion:vortex-file-compressed 167021415 1.60354e+08 1.04158 ns
tpch_q05/DataFusion:vortex-file-compressed 361128490 3.70288e+08 0.975263 ns
tpch_q06/DataFusion:vortex-file-compressed 11882345 1.22061e+07 0.973473 ns
tpch_q07/DataFusion:vortex-file-compressed 334386498 3.21159e+08 1.04119 ns
tpch_q08/DataFusion:vortex-file-compressed 391756200 3.91937e+08 0.999538 ns
tpch_q09/DataFusion:vortex-file-compressed 469408651 4.55536e+08 1.03045 ns
tpch_q10/DataFusion:vortex-file-compressed 385299582 3.90548e+08 0.986562 ns
tpch_q11/DataFusion:vortex-file-compressed 157660878 1.43359e+08 1.09976 ns
tpch_q12/DataFusion:vortex-file-compressed 194380896 2.02628e+08 0.959298 ns
tpch_q13/DataFusion:vortex-file-compressed 152525894 1.55926e+08 0.978193 ns
tpch_q14/DataFusion:vortex-file-compressed 243904026 2.37677e+08 1.0262 ns
tpch_q15/DataFusion:vortex-file-compressed 411333861 4.17247e+08 0.985829 ns
tpch_q16/DataFusion:vortex-file-compressed 87634362 9.45373e+07 0.926982 ns
tpch_q17/DataFusion:vortex-file-compressed 370254854 4.02663e+08 0.919516 ns
tpch_q18/DataFusion:vortex-file-compressed 410962900 4.21237e+08 0.97561 ns
tpch_q19/DataFusion:vortex-file-compressed 56232047 5.13221e+07 1.09567 ns
tpch_q20/DataFusion:vortex-file-compressed 279745788 2.71406e+08 1.03073 ns
tpch_q21/DataFusion:vortex-file-compressed 391712102 3.99498e+08 0.98051 ns
tpch_q22/DataFusion:vortex-file-compressed 112585180 1.06615e+08 1.056 ns

gatesn
gatesn requested changes May 12, 2025
View reviewed changes
Copy link
Contributor

@gatesn gatesn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Going to request changes just to block this as an experiment-only. I think the fix looks more like: #3274

@robert3005
Copy link
Contributor

wait, this is on of the options mentioned in the issue. Why is asking for an existing stat an antipattern now? If it is we should remove this api?

@gatesn
Copy link
Contributor

gatesn commented May 13, 2025

It's an anti-pattern because this call site wants an estimate of "is_constant" with some predicatable performance guarantees. Query the stats fits the performance requirement, but there are other sources that also may fit the requirement. e.g. ConstantArray that doesn't pre-populate the stats. Or dictionary array with constant codes.

So we should add an API that lets this and other call sites express "what" they want, not "how" they want it

@gatesn
Copy link
Contributor

gatesn commented May 14, 2025

Fixed by #3283

@gatesn gatesn closed this May 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gatesn gatesn gatesn requested changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@joseph-isaacs @robert3005 @gatesn