Skip to content

feat[duckdb]: add s3 duckdb benchmarks #3286

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 71 commits into
base: develop
Choose a base branch
from

Conversation

joseph-isaacs
Copy link
Contributor

@joseph-isaacs joseph-isaacs commented May 13, 2025

No description provided.

Signed-off-by: Joe Isaacs <[email protected]>
Signed-off-by: Joe Isaacs <[email protected]>
Signed-off-by: Joe Isaacs <[email protected]>
Signed-off-by: Joe Isaacs <[email protected]>
@joseph-isaacs joseph-isaacs marked this pull request as draft May 13, 2025 15:14
@joseph-isaacs joseph-isaacs changed the title wip feat[duckdb]: add s3 duckdb benchmarks May 13, 2025
Signed-off-by: Joe Isaacs <[email protected]>
Signed-off-by: Joe Isaacs <[email protected]>
@joseph-isaacs joseph-isaacs added the benchmark Run benchmarks on this branch label May 13, 2025
@github-actions github-actions bot removed the benchmark Run benchmarks on this branch label May 13, 2025
@joseph-isaacs joseph-isaacs added the benchmark Run benchmarks on this branch label May 13, 2025
@github-actions github-actions bot removed the benchmark Run benchmarks on this branch label May 13, 2025
Copy link
Contributor

github-actions bot commented May 13, 2025

Benchmarks: random_access

Table of Results
name PR 3da5be0 base 9e35e49 ratio (PR/base) unit
random-access/vortex-tokio-local-disk 1616039 1.62036e+06 0.99733 ns
random-access/parquet-tokio-local-disk 195027997 1.87744e+08 1.0388 ns

Signed-off-by: Joe Isaacs <[email protected]>
Signed-off-by: Joe Isaacs <[email protected]>
Signed-off-by: Joe Isaacs <[email protected]>
@joseph-isaacs joseph-isaacs requested a review from 0ax1 May 13, 2025 15:29
Signed-off-by: Joe Isaacs <[email protected]>
Signed-off-by: Joe Isaacs <[email protected]>
@joseph-isaacs joseph-isaacs added the benchmark Run benchmarks on this branch label May 13, 2025
@github-actions github-actions bot removed the benchmark Run benchmarks on this branch label May 13, 2025
@joseph-isaacs joseph-isaacs added the benchmark Run benchmarks on this branch label May 13, 2025
@github-actions github-actions bot removed the benchmark Run benchmarks on this branch label May 13, 2025
Signed-off-by: Joe Isaacs <[email protected]>
@joseph-isaacs joseph-isaacs added the benchmark Run benchmarks on this branch label May 13, 2025
@github-actions github-actions bot removed the benchmark Run benchmarks on this branch label May 13, 2025
Signed-off-by: Joe Isaacs <[email protected]>
Signed-off-by: Joe Isaacs <[email protected]>
Signed-off-by: Joe Isaacs <[email protected]>
Signed-off-by: Joe Isaacs <[email protected]>
Signed-off-by: Joe Isaacs <[email protected]>
Signed-off-by: Joe Isaacs <[email protected]>
Signed-off-by: Joe Isaacs <[email protected]>
Signed-off-by: Joe Isaacs <[email protected]>
Signed-off-by: Joe Isaacs <[email protected]>
Signed-off-by: Joe Isaacs <[email protected]>
Copy link
Contributor

Benchmarks: TPC-H on NVME

Table of Results
name PR 2f69f0d base 9e35e49 ratio (PR/base) unit
tpch_q01/DataFusion:arrow 65402430 6.36832e+07 1.027 ns
tpch_q02/DataFusion:arrow 15837177 1.60915e+07 0.984192 ns
tpch_q03/DataFusion:arrow 23768865 2.42024e+07 0.982085 ns
tpch_q04/DataFusion:arrow 19376282 1.92987e+07 1.00402 ns
tpch_q05/DataFusion:arrow 32561074 3.24644e+07 1.00298 ns
tpch_q06/DataFusion:arrow 6480182 6.2792e+06 1.03201 ns
tpch_q07/DataFusion:arrow 54445360 5.39084e+07 1.00996 ns
tpch_q08/DataFusion:arrow 18716029 1.96532e+07 0.952316 ns
tpch_q09/DataFusion:arrow 35012498 3.3336e+07 1.05029 ns
tpch_q10/DataFusion:arrow 39517532 3.86627e+07 1.02211 ns
tpch_q11/DataFusion:arrow 8015598 8.0015e+06 1.00176 ns
tpch_q12/DataFusion:arrow 17669954 1.81483e+07 0.973643 ns
tpch_q13/DataFusion:arrow 14434959 1.4995e+07 0.962654 ns
tpch_q14/DataFusion:arrow 8283142 8.63954e+06 0.958748 ns
tpch_q15/DataFusion:arrow 18200351 1.81477e+07 1.0029 ns
tpch_q16/DataFusion:arrow 14981013 1.48538e+07 1.00856 ns
tpch_q17/DataFusion:arrow 50411171 5.04596e+07 0.999039 ns
tpch_q18/DataFusion:arrow 106095028 1.09107e+08 0.972392 ns
tpch_q19/DataFusion:arrow 18075793 1.8232e+07 0.991433 ns
tpch_q20/DataFusion:arrow 23213291 2.39814e+07 0.96797 ns
tpch_q21/DataFusion:arrow 82655480 8.51307e+07 0.970924 ns
tpch_q22/DataFusion:arrow 10461366 1.10417e+07 0.947444 ns
tpch_q01/DataFusion:parquet 102876789 1.03483e+08 0.994145 ns
tpch_q02/DataFusion:parquet 52548242 5.06915e+07 1.03663 ns
tpch_q03/DataFusion:parquet 46866163 4.96556e+07 0.943824 ns
tpch_q04/DataFusion:parquet 31416965 3.19996e+07 0.981792 ns
tpch_q05/DataFusion:parquet 65142597 6.47552e+07 1.00598 ns
tpch_q06/DataFusion:parquet 28110947 2.90508e+07 0.967648 ns
tpch_q07/DataFusion:parquet 80400747 8.28903e+07 0.969966 ns
tpch_q08/DataFusion:parquet 59765153 6.31168e+07 0.946898 ns
tpch_q09/DataFusion:parquet 86685119 8.4986e+07 1.01999 ns
tpch_q10/DataFusion:parquet 96326465 9.72906e+07 0.990091 ns
tpch_q11/DataFusion:parquet 28992228 3.028e+07 0.957471 ns
tpch_q12/DataFusion:parquet 39408532 4.05198e+07 0.972574 ns
tpch_q13/DataFusion:parquet 41353080 4.19738e+07 0.985212 ns
tpch_q14/DataFusion:parquet 32998918 3.37054e+07 0.97904 ns
tpch_q15/DataFusion:parquet 54151393 5.78603e+07 0.935899 ns
tpch_q16/DataFusion:parquet 34922176 3.52204e+07 0.991533 ns
tpch_q17/DataFusion:parquet 93969768 9.91757e+07 0.947508 ns
tpch_q18/DataFusion:parquet 116460269 1.18735e+08 0.980845 ns
tpch_q19/DataFusion:parquet 72822834 7.39573e+07 0.984661 ns
tpch_q20/DataFusion:parquet 56452371 5.83423e+07 0.967606 ns
tpch_q21/DataFusion:parquet 100932206 1.04868e+08 0.962468 ns
tpch_q22/DataFusion:parquet 49984584 5.20434e+07 0.96044 ns
tpch_q01/DataFusion:vortex-file-compressed 55993109 6.00395e+07 0.932604 ns
tpch_q02/DataFusion:vortex-file-compressed 37620300 3.84867e+07 0.977488 ns
tpch_q03/DataFusion:vortex-file-compressed 29413749 2.97581e+07 0.988429 ns
tpch_q04/DataFusion:vortex-file-compressed 15684938 1.59813e+07 0.981457 ns
tpch_q05/DataFusion:vortex-file-compressed 48074354 4.80013e+07 1.00152 ns
tpch_q06/DataFusion:vortex-file-compressed 4099193 4.30203e+06 0.95285 ns
tpch_q07/DataFusion:vortex-file-compressed 63541905 6.50409e+07 0.976953 ns
tpch_q08/DataFusion:vortex-file-compressed 54327611 5.36938e+07 1.0118 ns
tpch_q09/DataFusion:vortex-file-compressed 74714888 7.55756e+07 0.988611 ns
tpch_q10/DataFusion:vortex-file-compressed 44512417 4.63155e+07 0.961069 ns
tpch_q11/DataFusion:vortex-file-compressed 26647562 2.66373e+07 1.00039 ns
tpch_q12/DataFusion:vortex-file-compressed 17428117 1.75363e+07 0.993828 ns
tpch_q13/DataFusion:vortex-file-compressed 17918097 1.84109e+07 0.973235 ns
tpch_q14/DataFusion:vortex-file-compressed 9861172 9.90231e+06 0.995846 ns
tpch_q15/DataFusion:vortex-file-compressed 17545871 1.76572e+07 0.993693 ns
tpch_q16/DataFusion:vortex-file-compressed 22955842 2.39856e+07 0.957069 ns
tpch_q17/DataFusion:vortex-file-compressed 76947420 7.84679e+07 0.980622 ns
tpch_q18/DataFusion:vortex-file-compressed 94881965 9.53244e+07 0.995359 ns
tpch_q19/DataFusion:vortex-file-compressed 17504322 1.91811e+07 0.912583 ns
tpch_q20/DataFusion:vortex-file-compressed 26915310 2.71027e+07 0.993085 ns
tpch_q21/DataFusion:vortex-file-compressed 86637509 8.88686e+07 0.974894 ns
tpch_q22/DataFusion:vortex-file-compressed 13173820 1.31344e+07 1.003 ns
tpch_q01/DuckDB:parquet 56000000 6.5e+07 0.861538 ns
tpch_q02/DuckDB:parquet 34000000 3.6e+07 0.944444 ns
tpch_q03/DuckDB:parquet 63000000 8e+07 0.7875 ns
tpch_q04/DuckDB:parquet 34000000 3.9e+07 0.871795 ns
tpch_q05/DuckDB:parquet 53000000 6e+07 0.883333 ns
tpch_q06/DuckDB:parquet 27000000 3e+07 0.9 ns
tpch_q07/DuckDB:parquet 55000000 6.1e+07 0.901639 ns
tpch_q08/DuckDB:parquet 60000000 6.8e+07 0.882353 ns
tpch_q09/DuckDB:parquet 92000000 9.6e+07 0.958333 ns
tpch_q10/DuckDB:parquet 237000000 3.27e+08 0.724771 ns
tpch_q11/DuckDB:parquet 17000000 1.9e+07 0.894737 ns
tpch_q12/DuckDB:parquet 41000000 4.6e+07 0.891304 ns
tpch_q13/DuckDB:parquet 49000000 5.4e+07 0.907407 ns
tpch_q14/DuckDB:parquet 45000000 5e+07 0.9 ns
tpch_q15/DuckDB:parquet 55000000 6.3e+07 0.873016 ns
tpch_q16/DuckDB:parquet 54000000 6.9e+07 0.782609 ns
tpch_q17/DuckDB:parquet 56000000 6.3e+07 0.888889 ns
tpch_q18/DuckDB:parquet 67000000 7.4e+07 0.905405 ns
tpch_q19/DuckDB:parquet 66000000 7e+07 0.942857 ns
tpch_q20/DuckDB:parquet 52000000 5.9e+07 0.881356 ns
tpch_q21/DuckDB:parquet 103000000 1.19e+08 0.865546 ns
tpch_q22/DuckDB:parquet 38000000 4.5e+07 0.844444 ns
tpch_q01/DuckDB:vortex-file-compressed 41000000 4.8e+07 0.854167 ns
tpch_q02/DuckDB:vortex-file-compressed 60000000 6e+07 1 ns
tpch_q03/DuckDB:vortex-file-compressed 69000000 7.8e+07 0.884615 ns
tpch_q04/DuckDB:vortex-file-compressed 46000000 4.4e+07 1.04545 ns
tpch_q05/DuckDB:vortex-file-compressed 61000000 6.3e+07 0.968254 ns
tpch_q06/DuckDB:vortex-file-compressed 15000000 1.6e+07 0.9375 ns
tpch_q07/DuckDB:vortex-file-compressed 62000000 6.3e+07 0.984127 ns
tpch_q08/DuckDB:vortex-file-compressed 79000000 8.4e+07 0.940476 ns
tpch_q09/DuckDB:vortex-file-compressed 101000000 9.9e+07 1.0202 ns
tpch_q10/DuckDB:vortex-file-compressed 206000000 2.91e+08 0.707904 ns
tpch_q11/DuckDB:vortex-file-compressed 30000000 3.6e+07 0.833333 ns
tpch_q12/DuckDB:vortex-file-compressed 44000000 4.4e+07 1 ns
tpch_q13/DuckDB:vortex-file-compressed 62000000 6e+07 1.03333 ns
tpch_q14/DuckDB:vortex-file-compressed 32000000 3.2e+07 1 ns
tpch_q15/DuckDB:vortex-file-compressed 49000000 5.1e+07 0.960784 ns
tpch_q16/DuckDB:vortex-file-compressed 65000000 7.8e+07 0.833333 ns
tpch_q17/DuckDB:vortex-file-compressed 56000000 5e+07 1.12 ns
tpch_q18/DuckDB:vortex-file-compressed 72000000 8.1e+07 0.888889 ns
tpch_q19/DuckDB:vortex-file-compressed 45000000 4.3e+07 1.04651 ns
tpch_q20/DuckDB:vortex-file-compressed 52000000 5.1e+07 1.01961 ns
tpch_q21/DuckDB:vortex-file-compressed 135000000 1.34e+08 1.00746 ns
tpch_q22/DuckDB:vortex-file-compressed 43000000 5e+07 0.86 ns
tpch_q01/DuckDB:duckdb 25000000 2.6e+07 0.961538 ns
tpch_q02/DuckDB:duckdb 19000000 2.1e+07 0.904762 ns
tpch_q03/DuckDB:duckdb 44000000 5.4e+07 0.814815 ns
tpch_q04/DuckDB:duckdb 22000000 2.3e+07 0.956522 ns
tpch_q05/DuckDB:duckdb 31000000 3.3e+07 0.939394 ns
tpch_q06/DuckDB:duckdb 10000000 9e+06 1.11111 ns
tpch_q07/DuckDB:duckdb 32000000 3.2e+07 1 ns
tpch_q08/DuckDB:duckdb 30000000 3.2e+07 0.9375 ns
tpch_q09/DuckDB:duckdb 65000000 7e+07 0.928571 ns
tpch_q10/DuckDB:duckdb 204000000 2.87e+08 0.710801 ns
tpch_q11/DuckDB:duckdb 10000000 1.2e+07 0.833333 ns
tpch_q12/DuckDB:duckdb 28000000 3e+07 0.933333 ns
tpch_q13/DuckDB:duckdb 41000000 4.1e+07 1 ns
tpch_q14/DuckDB:duckdb 29000000 2.9e+07 1 ns
tpch_q15/DuckDB:duckdb 40000000 3.9e+07 1.02564 ns
tpch_q16/DuckDB:duckdb 45000000 5.9e+07 0.762712 ns
tpch_q17/DuckDB:duckdb 30000000 3e+07 1 ns
tpch_q18/DuckDB:duckdb 44000000 4.6e+07 0.956522 ns
tpch_q19/DuckDB:duckdb 42000000 4.3e+07 0.976744 ns
tpch_q20/DuckDB:duckdb 33000000 3.5e+07 0.942857 ns
tpch_q21/DuckDB:duckdb 61000000 6.5e+07 0.938462 ns
tpch_q22/DuckDB:duckdb 29000000 3.3e+07 0.878788 ns

Copy link
Contributor

Benchmarks: TPC-H on S3

Table of Results
name PR 2f69f0d base 9e35e49 ratio (PR/base) unit
tpch_q01/DataFusion:parquet 225638500 2.23278e+08 1.01057 ns
tpch_q02/DataFusion:parquet 755874239 7.42937e+08 1.01741 ns
tpch_q03/DataFusion:parquet 370227617 3.61696e+08 1.02359 ns
tpch_q04/DataFusion:parquet 225949597 2.20978e+08 1.0225 ns
tpch_q05/DataFusion:parquet 565738167 5.40244e+08 1.04719 ns
tpch_q06/DataFusion:parquet 145098123 1.4822e+08 0.978935 ns
tpch_q07/DataFusion:parquet 520013197 5.17807e+08 1.00426 ns
tpch_q08/DataFusion:parquet 787673582 7.64368e+08 1.03049 ns
tpch_q09/DataFusion:parquet 616412020 6.24478e+08 0.987084 ns
tpch_q10/DataFusion:parquet 536368109 5.42375e+08 0.988925 ns
tpch_q11/DataFusion:parquet 465001946 4.79299e+08 0.970171 ns
tpch_q12/DataFusion:parquet 297779532 2.85609e+08 1.04261 ns
tpch_q13/DataFusion:parquet 209730982 2.04739e+08 1.02438 ns
tpch_q14/DataFusion:parquet 283116694 2.78967e+08 1.01488 ns
tpch_q15/DataFusion:parquet 392933113 3.88214e+08 1.01216 ns
tpch_q16/DataFusion:parquet 380381994 3.80911e+08 0.99861 ns
tpch_q17/DataFusion:parquet 537323569 5.3103e+08 1.01185 ns
tpch_q18/DataFusion:parquet 467935670 4.5768e+08 1.02241 ns
tpch_q19/DataFusion:parquet 377174808 3.7988e+08 0.992879 ns
tpch_q20/DataFusion:parquet 523912712 5.34086e+08 0.980952 ns
tpch_q21/DataFusion:parquet 753846189 7.36134e+08 1.02406 ns
tpch_q22/DataFusion:parquet 276174609 2.94713e+08 0.937097 ns
tpch_q01/DataFusion:vortex-file-compressed 161890903 1.6387e+08 0.987924 ns
tpch_q02/DataFusion:vortex-file-compressed 156021088 1.55103e+08 1.00592 ns
tpch_q03/DataFusion:vortex-file-compressed 253614279 2.52822e+08 1.00314 ns
tpch_q04/DataFusion:vortex-file-compressed 157984419 1.54881e+08 1.02004 ns
tpch_q05/DataFusion:vortex-file-compressed 279881685 2.3632e+08 1.18434 ns
tpch_q06/DataFusion:vortex-file-compressed 11040375 9.91384e+06 1.11363 ns
tpch_q07/DataFusion:vortex-file-compressed 251771624 2.51384e+08 1.00154 ns
tpch_q08/DataFusion:vortex-file-compressed 310387383 3.32188e+08 0.934373 ns
tpch_q09/DataFusion:vortex-file-compressed 319387888 3.39751e+08 0.940066 ns
tpch_q10/DataFusion:vortex-file-compressed 301198501 2.87032e+08 1.04936 ns
tpch_q11/DataFusion:vortex-file-compressed 129357829 1.36377e+08 0.948533 ns
tpch_q12/DataFusion:vortex-file-compressed 185631733 1.926e+08 0.963821 ns
tpch_q13/DataFusion:vortex-file-compressed 141723855 1.38601e+08 1.02253 ns
tpch_q14/DataFusion:vortex-file-compressed 135967473 1.41498e+08 0.960915 ns
tpch_q15/DataFusion:vortex-file-compressed 215652572 2.08269e+08 1.03545 ns
tpch_q16/DataFusion:vortex-file-compressed 102069094 9.57879e+07 1.06557 ns
tpch_q17/DataFusion:vortex-file-compressed 270468045 2.45489e+08 1.10175 ns
tpch_q18/DataFusion:vortex-file-compressed 322380758 3.0944e+08 1.04182 ns
tpch_q19/DataFusion:vortex-file-compressed 48282483 5.54774e+07 0.870308 ns
tpch_q20/DataFusion:vortex-file-compressed 222984300 2.18183e+08 1.02201 ns
tpch_q21/DataFusion:vortex-file-compressed 381166400 3.91913e+08 0.972579 ns
tpch_q22/DataFusion:vortex-file-compressed 108254145 1.02801e+08 1.05305 ns
tpch_q01/DuckDB:parquet 460000000 nan nan nan
tpch_q02/DuckDB:parquet 1102000000 nan nan nan
tpch_q03/DuckDB:parquet 719000000 nan nan nan
tpch_q04/DuckDB:parquet 445000000 nan nan nan
tpch_q05/DuckDB:parquet 895000000 nan nan nan
tpch_q06/DuckDB:parquet 300000000 nan nan nan
tpch_q07/DuckDB:parquet 821000000 nan nan nan
tpch_q08/DuckDB:parquet 1025000000 nan nan nan
tpch_q09/DuckDB:parquet 960000000 nan nan nan
tpch_q10/DuckDB:parquet 1050000000 nan nan nan
tpch_q11/DuckDB:parquet 644000000 nan nan nan
tpch_q12/DuckDB:parquet 400000000 nan nan nan
tpch_q13/DuckDB:parquet 345000000 nan nan nan
tpch_q14/DuckDB:parquet 449000000 nan nan nan
tpch_q15/DuckDB:parquet 785000000 nan nan nan
tpch_q16/DuckDB:parquet 442000000 nan nan nan
tpch_q17/DuckDB:parquet 669000000 nan nan nan
tpch_q18/DuckDB:parquet 666000000 nan nan nan
tpch_q19/DuckDB:parquet 620000000 nan nan nan
tpch_q20/DuckDB:parquet 682000000 nan nan nan
tpch_q21/DuckDB:parquet 1012000000 nan nan nan
tpch_q22/DuckDB:parquet 396000000 nan nan nan
tpch_q01/DuckDB:vortex-file-compressed 349000000 nan nan nan
tpch_q02/DuckDB:vortex-file-compressed 1078000000 nan nan nan
tpch_q03/DuckDB:vortex-file-compressed 745000000 nan nan nan
tpch_q04/DuckDB:vortex-file-compressed 500000000 nan nan nan
tpch_q05/DuckDB:vortex-file-compressed 972000000 nan nan nan
tpch_q06/DuckDB:vortex-file-compressed 98000000 nan nan nan
tpch_q07/DuckDB:vortex-file-compressed 830000000 nan nan nan
tpch_q08/DuckDB:vortex-file-compressed 1076000000 nan nan nan
tpch_q09/DuckDB:vortex-file-compressed 940000000 nan nan nan
tpch_q10/DuckDB:vortex-file-compressed 985000000 nan nan nan
tpch_q11/DuckDB:vortex-file-compressed 663000000 nan nan nan
tpch_q12/DuckDB:vortex-file-compressed 566000000 nan nan nan
tpch_q13/DuckDB:vortex-file-compressed 403000000 nan nan nan
tpch_q14/DuckDB:vortex-file-compressed 496000000 nan nan nan
tpch_q15/DuckDB:vortex-file-compressed 642000000 nan nan nan
tpch_q16/DuckDB:vortex-file-compressed 439000000 nan nan nan
tpch_q17/DuckDB:vortex-file-compressed 738000000 nan nan nan
tpch_q18/DuckDB:vortex-file-compressed 835000000 nan nan nan
tpch_q19/DuckDB:vortex-file-compressed 503000000 nan nan nan
tpch_q20/DuckDB:vortex-file-compressed 820000000 nan nan nan
tpch_q21/DuckDB:vortex-file-compressed 1177000000 nan nan nan
tpch_q22/DuckDB:vortex-file-compressed 505000000 nan nan nan

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
do not merge Pull requests that are not intended to merge feature New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant