Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](parquet)Fix data column and null map column not equal when reading Parquet complex type cross-page data #47734

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

[fix](parquet)Fix data column and null map column not equal when reading Parquet complex type cross-page data #47734

wants to merge 5 commits into from
+122 −10

Conversation

hubgeter
Copy link
Contributor

What problem does this PR solve?

Related PR: #23277

Problem Summary:
Previously, you may encounter this error when reading parquet complex types. This PR is mainly to fix this problem.

[fragment_mgr.cpp:549] report error status: cur path:  xxx. parquet. Read parquet file   xxxx.parquet failed, reason = [E-1721][E-1721] Size of filter doesn't match size of column: size=3156, filter.size=12544

0#  doris::Exception::Exception(int, std::basic_string_view<char, std::char_traits<char> > const&) at //ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:173
1#  doris::Exception::Exception<unsigned long&, unsigned long&>(int, std::basic_string_view<char, std::char_traits<char> > const&, unsigned long&, unsigned long&) at //ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187
2#  doris::vectorized::ColumnVector<unsigned char>::filter(doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 16ul> const&) at /be/src/vec/columns/columns_common.h:86
3#  doris::vectorized::ColumnNullable::filter(doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 16ul> const&) at /be/src/vec/columns/column_nullable.cpp:373
4#  doris::vectorized::ColumnStruct::filter(doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 16ul> const&) at /be/src/vec/columns/column_struct.cpp:289
5#  doris::vectorized::ColumnArray::filter_generic(doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 16ul> const&) at /be/src/vec/common/cow.h:402
6#  doris::vectorized::ColumnArray::filter_nullable(doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 16ul> const&) at /be/src/vec/columns/column_array.cpp:877
7#  doris::vectorized::ColumnNullable::filter(doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 16ul> const&) at /be/src/vec/columns/column_nullable.cpp:371
8#  doris::vectorized::Block::filter_block_internal(doris::vectorized::Block*, std::vector<unsigned int, std::allocator<unsigned int> > const&, doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 16ul> const&) at /be/src/vec/core/block.cpp:790
9#  doris::vectorized::RowGroupReader::next_batch(doris::vectorized::Block*, unsigned long, unsigned long*, bool*) at /be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:0
10# doris::vectorized::ParquetReader::get_next_block(doris::vectorized::Block*, unsigned long*, bool*) at /be/src/common/status.h:486
11# doris::vectorized::IcebergTableReader::get_next_block(doris::vectorized::Block*, unsigned long*, bool*) at /be/src/common/status.h:491
12# doris::vectorized::VFileScanner::_get_block_wrapped(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /be/src/common/status.h:491
13# doris::vectorized::VFileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /be/src/common/status.h:491
14# doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /be/src/vec/exec/scan/vscanner.cpp:0
15# doris::vectorized::VScanner::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /be/src/vec/exec/scan/vscanner.cpp:101
16# doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) at /be/src/common/status.h:378

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Feb 11, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter
Copy link
Contributor Author

run buildall

@hubgeter hubgeter force-pushed the fix_parquet_complex_null_map branch from d2820a8 to be21e62 Compare February 11, 2025 06:18
@hubgeter
Copy link
Contributor Author

run buildall

@hubgeter hubgeter force-pushed the fix_parquet_complex_null_map branch from be21e62 to 080a260 Compare February 11, 2025 07:52
@hubgeter
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31270 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 080a260e8bf4921c2f7e51f46ad162bc2393dcfb, data reload: false

------ Round 1 ----------------------------------
q1	17558	5256	5099	5099
q2	2057	299	169	169
q3	10433	1226	742	742
q4	10230	1001	533	533
q5	7787	2384	2293	2293
q6	186	164	132	132
q7	874	751	590	590
q8	9283	1225	1018	1018
q9	4851	4756	4661	4661
q10	6858	2292	1896	1896
q11	488	269	256	256
q12	346	353	219	219
q13	17784	3627	3070	3070
q14	241	243	200	200
q15	514	457	448	448
q16	614	617	570	570
q17	562	860	319	319
q18	6730	6222	6074	6074
q19	1618	942	536	536
q20	308	318	181	181
q21	2692	2154	1972	1972
q22	364	320	292	292
Total cold run time: 102378 ms
Total hot run time: 31270 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5260	5184	5188	5184
q2	240	326	228	228
q3	2131	2682	2272	2272
q4	1422	1858	1341	1341
q5	4209	4085	4147	4085
q6	200	164	132	132
q7	1838	1796	1673	1673
q8	2584	2669	2587	2587
q9	7125	7089	7130	7089
q10	2982	3215	2759	2759
q11	578	544	501	501
q12	697	791	622	622
q13	3554	3808	3309	3309
q14	293	309	267	267
q15	498	449	474	449
q16	635	660	629	629
q17	1128	1613	1334	1334
q18	7523	7337	7276	7276
q19	846	858	993	858
q20	1928	2025	1829	1829
q21	5493	5004	4722	4722
q22	643	603	533	533
Total cold run time: 51807 ms
Total hot run time: 49679 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 188662 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 080a260e8bf4921c2f7e51f46ad162bc2393dcfb, data reload: false

query1	1312	944	936	936
query2	6240	1867	1814	1814
query3	10911	4418	4477	4418
query4	56841	24686	22945	22945
query5	5185	506	474	474
query6	335	186	194	186
query7	4886	520	293	293
query8	307	244	246	244
query9	5638	2505	2513	2505
query10	429	301	258	258
query11	14990	14998	14822	14822
query12	161	107	104	104
query13	1048	525	392	392
query14	10620	6354	6748	6354
query15	215	193	173	173
query16	7152	628	487	487
query17	1063	750	560	560
query18	1653	420	324	324
query19	189	195	168	168
query20	130	124	155	124
query21	209	123	103	103
query22	4327	4558	4236	4236
query23	33708	33508	33240	33240
query24	5638	2440	2417	2417
query25	460	470	411	411
query26	695	272	153	153
query27	1761	515	340	340
query28	2785	2459	2450	2450
query29	560	588	430	430
query30	214	186	153	153
query31	881	866	795	795
query32	74	61	66	61
query33	460	360	326	326
query34	746	880	518	518
query35	803	832	755	755
query36	937	1001	923	923
query37	136	101	78	78
query38	4238	4294	4176	4176
query39	1467	1456	1412	1412
query40	209	118	102	102
query41	52	50	49	49
query42	125	109	109	109
query43	499	504	483	483
query44	1315	819	822	819
query45	185	177	166	166
query46	880	1061	670	670
query47	1859	1866	1787	1787
query48	379	414	302	302
query49	685	528	415	415
query50	749	753	422	422
query51	4270	4283	4205	4205
query52	108	109	96	96
query53	249	270	194	194
query54	470	496	411	411
query55	81	85	82	82
query56	271	282	276	276
query57	1172	1172	1141	1141
query58	246	242	247	242
query59	2673	2918	2771	2771
query60	303	271	260	260
query61	126	124	121	121
query62	751	795	673	673
query63	229	190	190	190
query64	1871	1033	705	705
query65	3295	3296	3142	3142
query66	715	403	306	306
query67	16014	15332	15409	15332
query68	5164	845	518	518
query69	516	290	271	271
query70	1221	1022	1093	1022
query71	437	292	259	259
query72	5912	3605	3781	3605
query73	1362	748	354	354
query74	8951	9131	8756	8756
query75	3404	3138	2657	2657
query76	3719	1181	737	737
query77	537	391	275	275
query78	9916	10087	9305	9305
query79	2748	804	598	598
query80	705	523	433	433
query81	489	275	232	232
query82	684	150	118	118
query83	189	171	149	149
query84	296	98	72	72
query85	782	353	300	300
query86	411	292	294	292
query87	4459	4565	4284	4284
query88	3611	2219	2208	2208
query89	426	319	282	282
query90	1900	190	193	190
query91	140	137	107	107
query92	70	57	57	57
query93	2265	998	577	577
query94	656	405	292	292
query95	349	264	255	255
query96	490	568	266	266
query97	2795	2866	2735	2735
query98	232	210	194	194
query99	1322	1402	1233	1233
Total cold run time: 295343 ms
Total hot run time: 188662 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.63 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 080a260e8bf4921c2f7e51f46ad162bc2393dcfb, data reload: false

query1	0.03	0.02	0.03
query2	0.09	0.04	0.03
query3	0.24	0.06	0.06
query4	1.62	0.10	0.10
query5	0.42	0.41	0.38
query6	1.14	0.66	0.65
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.59	0.52	0.52
query10	0.57	0.58	0.57
query11	0.16	0.10	0.10
query12	0.14	0.11	0.10
query13	0.61	0.60	0.61
query14	2.67	2.83	2.85
query15	0.91	0.85	0.85
query16	0.35	0.37	0.36
query17	0.98	1.02	1.01
query18	0.21	0.19	0.19
query19	1.92	1.85	1.95
query20	0.01	0.02	0.01
query21	15.37	0.89	0.54
query22	0.75	1.23	0.61
query23	15.00	1.39	0.61
query24	6.58	1.52	0.78
query25	0.54	0.25	0.07
query26	0.61	0.17	0.15
query27	0.06	0.05	0.05
query28	9.87	0.90	0.44
query29	12.53	4.16	3.42
query30	0.24	0.09	0.06
query31	2.82	0.60	0.38
query32	3.23	0.54	0.46
query33	3.00	3.03	3.03
query34	15.61	5.06	4.46
query35	4.52	4.52	4.50
query36	0.65	0.49	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 104.58 s
Total hot run time: 30.63 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.81% (11263/26308)
Line Coverage: 32.79% (94618/288569)
Region Coverage: 31.94% (48499/151825)
Branch Coverage: 27.81% (24459/87940)
Coverage Report: http://coverage.selectdb-in.cc/coverage/080a260e8bf4921c2f7e51f46ad162bc2393dcfb_080a260e8bf4921c2f7e51f46ad162bc2393dcfb/report/index.html

@hubgeter hubgeter force-pushed the fix_parquet_complex_null_map branch from 080a260 to bdda3ee Compare February 12, 2025 06:19
@github-actions github-actions bot added the lfs-detected! Warning Label for use when LFS is detected in the commits of a Pull Request label Feb 12, 2025
@github-actions GitHub Actions
Copy link
Contributor

Possible file(s) that should be tracked in LFS detected: 🚨

The following file(s) exceeds the file size limit: 1048576 bytes, as set in the .yml configuration files:

  • regression-test/data/external_table_p2/hive/test_parquet_complex_cross_page.out

Consider using git-lfs to manage large files.

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

Possible file(s) that should be tracked in LFS detected: 🚨

The following file(s) exceeds the file size limit: 1048576 bytes, as set in the .yml configuration files:

  • regression-test/data/external_table_p2/hive/test_parquet_complex_cross_page.out

Consider using git-lfs to manage large files.

@github-actions github-actions bot removed the lfs-detected! Warning Label for use when LFS is detected in the commits of a Pull Request label Feb 12, 2025
@hubgeter hubgeter marked this pull request as ready for review February 12, 2025 06:43
@hubgeter hubgeter force-pushed the fix_parquet_complex_null_map branch from 357f163 to 90d6a7f Compare February 12, 2025 06:43
@hubgeter
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31402 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 90d6a7fe20152d379007b0365442ab276024df78, data reload: false

------ Round 1 ----------------------------------
q1	17599	5280	5080	5080
q2	2056	310	167	167
q3	10408	1355	722	722
q4	10217	1000	528	528
q5	7580	2395	2345	2345
q6	191	167	142	142
q7	926	746	605	605
q8	9323	1312	1104	1104
q9	4926	4797	4565	4565
q10	6835	2309	1912	1912
q11	473	271	267	267
q12	350	365	218	218
q13	17755	3656	3089	3089
q14	235	227	214	214
q15	502	465	448	448
q16	636	609	574	574
q17	587	858	325	325
q18	6655	6154	6247	6154
q19	1213	958	542	542
q20	317	318	194	194
q21	2808	2282	1900	1900
q22	368	342	307	307
Total cold run time: 101960 ms
Total hot run time: 31402 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5100	5196	5126	5126
q2	228	327	230	230
q3	2195	2618	2273	2273
q4	1441	1813	1395	1395
q5	4217	4127	4134	4127
q6	219	160	125	125
q7	1854	1819	1661	1661
q8	2601	2687	2613	2613
q9	7281	7087	7087	7087
q10	2972	3181	2760	2760
q11	578	505	481	481
q12	666	791	643	643
q13	3422	3914	3202	3202
q14	274	302	270	270
q15	512	461	462	461
q16	684	678	632	632
q17	1152	1535	1370	1370
q18	7628	7291	7332	7291
q19	808	811	889	811
q20	1989	2013	1858	1858
q21	5414	4882	4774	4774
q22	660	585	539	539
Total cold run time: 51895 ms
Total hot run time: 49729 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183034 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 90d6a7fe20152d379007b0365442ab276024df78, data reload: false

query1	969	378	377	377
query2	6556	1841	1853	1841
query3	6786	208	206	206
query4	26325	23794	23497	23497
query5	4801	680	470	470
query6	293	200	201	200
query7	4610	499	296	296
query8	288	242	219	219
query9	8635	2499	2502	2499
query10	505	305	253	253
query11	15370	15078	14786	14786
query12	153	107	105	105
query13	1646	516	380	380
query14	9191	6100	6641	6100
query15	209	202	178	178
query16	7284	620	480	480
query17	1128	688	535	535
query18	1944	397	293	293
query19	195	189	163	163
query20	122	115	119	115
query21	212	120	104	104
query22	4197	4393	4572	4393
query23	34687	33451	32991	32991
query24	7653	2403	2371	2371
query25	541	438	382	382
query26	1232	262	150	150
query27	2533	471	329	329
query28	4289	2416	2404	2404
query29	748	531	412	412
query30	235	185	154	154
query31	936	837	797	797
query32	71	70	60	60
query33	557	405	299	299
query34	796	832	507	507
query35	801	815	722	722
query36	969	967	865	865
query37	118	117	74	74
query38	4182	4371	4077	4077
query39	1462	1379	1378	1378
query40	205	118	108	108
query41	58	51	54	51
query42	119	106	105	105
query43	499	495	472	472
query44	1276	802	796	796
query45	175	169	164	164
query46	856	1034	635	635
query47	1747	1792	1711	1711
query48	372	411	304	304
query49	770	496	418	418
query50	673	715	428	428
query51	4153	4100	4054	4054
query52	106	103	91	91
query53	228	248	192	192
query54	478	480	405	405
query55	80	76	82	76
query56	289	281	267	267
query57	1134	1149	1066	1066
query58	268	248	261	248
query59	2689	2693	2638	2638
query60	295	284	262	262
query61	149	148	140	140
query62	831	748	644	644
query63	223	188	190	188
query64	4429	1081	770	770
query65	3202	3151	3112	3112
query66	1128	400	299	299
query67	15800	15533	15221	15221
query68	7898	761	505	505
query69	480	300	269	269
query70	1197	1164	1107	1107
query71	453	308	272	272
query72	5865	3515	3727	3515
query73	746	735	353	353
query74	8975	9178	8784	8784
query75	3184	3198	2670	2670
query76	3275	1160	737	737
query77	480	387	344	344
query78	9972	10238	9290	9290
query79	2195	811	596	596
query80	633	534	451	451
query81	499	280	242	242
query82	198	127	99	99
query83	177	171	148	148
query84	245	90	74	74
query85	811	337	302	302
query86	374	298	254	254
query87	4444	4504	4325	4325
query88	3237	2207	2193	2193
query89	382	324	281	281
query90	1972	196	187	187
query91	138	140	108	108
query92	73	60	59	59
query93	1931	1010	573	573
query94	668	409	307	307
query95	356	270	254	254
query96	484	544	264	264
query97	2750	2873	2699	2699
query98	233	202	201	201
query99	1317	1433	1244	1244
Total cold run time: 271103 ms
Total hot run time: 183034 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.64 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 90d6a7fe20152d379007b0365442ab276024df78, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.03
query3	0.24	0.06	0.06
query4	1.61	0.10	0.11
query5	0.41	0.41	0.40
query6	1.18	0.67	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.53	0.53
query10	0.59	0.58	0.57
query11	0.15	0.10	0.10
query12	0.15	0.11	0.11
query13	0.62	0.59	0.61
query14	2.69	2.72	2.71
query15	0.92	0.86	0.84
query16	0.37	0.38	0.37
query17	1.07	1.09	1.06
query18	0.22	0.20	0.20
query19	1.91	1.76	2.04
query20	0.01	0.01	0.01
query21	15.35	0.87	0.53
query22	0.76	1.17	0.63
query23	15.00	1.39	0.63
query24	6.97	1.34	0.99
query25	0.50	0.16	0.13
query26	0.64	0.16	0.14
query27	0.05	0.05	0.05
query28	9.50	0.88	0.43
query29	12.54	3.94	3.24
query30	0.25	0.09	0.06
query31	2.82	0.57	0.38
query32	3.22	0.54	0.46
query33	2.97	2.99	3.02
query34	15.90	5.11	4.51
query35	4.52	4.50	4.59
query36	0.65	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.16	0.13	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.03 s
Total hot run time: 30.64 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.94% (11299/26312)
Line Coverage: 32.87% (94869/288588)
Region Coverage: 32.05% (48656/151829)
Branch Coverage: 27.90% (24535/87938)
Coverage Report: http://coverage.selectdb-in.cc/coverage/90d6a7fe20152d379007b0365442ab276024df78_90d6a7fe20152d379007b0365442ab276024df78/report/index.html

@morningman morningman force-pushed the fix_parquet_complex_null_map branch from 90d6a7f to 30a118a Compare February 14, 2025 12:36
@morningman
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31556 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 30a118a67444fe18691a957f2b6ea5501bdb95b6, data reload: false

------ Round 1 ----------------------------------
q1	17570	5182	5107	5107
q2	2047	306	170	170
q3	10401	1282	703	703
q4	10218	1008	549	549
q5	7521	2452	2326	2326
q6	189	173	135	135
q7	904	744	630	630
q8	9311	1333	1035	1035
q9	5104	4602	4848	4602
q10	6840	2317	1892	1892
q11	479	275	257	257
q12	350	340	215	215
q13	17765	3678	3058	3058
q14	234	237	205	205
q15	506	463	450	450
q16	623	593	603	593
q17	574	930	342	342
q18	6821	6318	6422	6318
q19	1211	945	561	561
q20	310	323	190	190
q21	2783	2131	1921	1921
q22	360	331	297	297
Total cold run time: 102121 ms
Total hot run time: 31556 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5147	5111	5145	5111
q2	229	328	240	240
q3	2154	2681	2303	2303
q4	1423	1856	1356	1356
q5	4250	4149	4145	4145
q6	209	164	122	122
q7	1877	1823	1653	1653
q8	2649	2579	2534	2534
q9	7361	7142	7115	7115
q10	2993	3220	2719	2719
q11	571	515	479	479
q12	665	766	626	626
q13	3536	4050	3292	3292
q14	289	307	270	270
q15	510	462	448	448
q16	621	671	630	630
q17	1140	1629	1308	1308
q18	7595	7310	7273	7273
q19	796	824	889	824
q20	1973	2046	1904	1904
q21	5291	4890	4878	4878
q22	659	570	532	532
Total cold run time: 51938 ms
Total hot run time: 49762 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183248 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 30a118a67444fe18691a957f2b6ea5501bdb95b6, data reload: false

query1	960	373	373	373
query2	6532	1847	1828	1828
query3	6811	215	207	207
query4	26109	23714	25564	23714
query5	4372	657	508	508
query6	281	185	171	171
query7	4599	495	293	293
query8	285	245	220	220
query9	8618	2505	2509	2505
query10	468	300	273	273
query11	15719	15333	14878	14878
query12	160	108	110	108
query13	1665	510	401	401
query14	9311	6619	6202	6202
query15	207	198	179	179
query16	7133	660	482	482
query17	933	712	565	565
query18	1950	403	315	315
query19	203	185	161	161
query20	126	133	121	121
query21	209	125	108	108
query22	4153	4378	4337	4337
query23	34406	33244	32901	32901
query24	7707	2392	2377	2377
query25	548	488	410	410
query26	1236	268	153	153
query27	2118	504	325	325
query28	3958	2386	2373	2373
query29	776	536	406	406
query30	223	178	146	146
query31	934	863	824	824
query32	80	64	62	62
query33	586	346	294	294
query34	790	848	489	489
query35	796	826	734	734
query36	982	985	912	912
query37	121	92	76	76
query38	4094	4106	4092	4092
query39	1491	1413	1399	1399
query40	205	112	103	103
query41	54	52	50	50
query42	122	104	99	99
query43	479	495	483	483
query44	1253	770	775	770
query45	177	175	163	163
query46	857	1029	631	631
query47	1733	1782	1701	1701
query48	396	405	311	311
query49	772	481	408	408
query50	681	739	418	418
query51	4173	4341	4105	4105
query52	109	100	92	92
query53	221	252	191	191
query54	477	491	391	391
query55	84	79	79	79
query56	293	262	244	244
query57	1108	1144	1086	1086
query58	253	238	238	238
query59	2646	2724	2471	2471
query60	282	269	255	255
query61	116	120	123	120
query62	797	720	668	668
query63	231	182	194	182
query64	4372	1003	666	666
query65	3266	3171	3148	3148
query66	1152	403	285	285
query67	15746	15502	15342	15342
query68	8137	756	490	490
query69	469	287	256	256
query70	1139	1192	1105	1105
query71	451	288	263	263
query72	5202	3466	3697	3466
query73	727	742	338	338
query74	9201	9007	8794	8794
query75	3803	3170	2702	2702
query76	3690	1167	755	755
query77	832	372	278	278
query78	9951	10107	9289	9289
query79	2477	793	588	588
query80	631	536	445	445
query81	490	279	241	241
query82	701	125	95	95
query83	164	170	152	152
query84	283	89	76	76
query85	770	344	303	303
query86	383	308	313	308
query87	4360	4537	4311	4311
query88	3701	2156	2147	2147
query89	386	311	289	289
query90	1897	195	187	187
query91	163	135	104	104
query92	66	61	54	54
query93	1865	1016	568	568
query94	659	412	292	292
query95	347	267	255	255
query96	480	565	273	273
query97	2792	2872	2754	2754
query98	234	209	202	202
query99	1647	1417	1246	1246
Total cold run time: 271803 ms
Total hot run time: 183248 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.88 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 30a118a67444fe18691a957f2b6ea5501bdb95b6, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.02	0.04
query3	0.24	0.07	0.07
query4	1.61	0.10	0.11
query5	0.40	0.41	0.39
query6	1.18	0.67	0.65
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.60	0.52	0.53
query10	0.59	0.58	0.58
query11	0.16	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.71	2.74	2.72
query15	0.92	0.85	0.86
query16	0.40	0.37	0.38
query17	0.98	1.02	1.01
query18	0.21	0.19	0.19
query19	1.87	1.79	1.95
query20	0.02	0.01	0.01
query21	15.37	0.96	0.54
query22	0.76	1.13	0.67
query23	15.02	1.43	0.64
query24	10.05	1.13	1.09
query25	0.49	0.14	0.12
query26	0.79	0.18	0.13
query27	0.05	0.05	0.05
query28	8.99	0.86	0.43
query29	12.53	3.96	3.26
query30	0.25	0.09	0.06
query31	2.84	0.57	0.39
query32	3.22	0.55	0.46
query33	3.03	3.02	3.05
query34	15.86	5.17	4.50
query35	4.58	4.56	4.59
query36	0.66	0.50	0.48
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.8 s
Total hot run time: 30.88 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 43.34% (11317/26112)
Line Coverage: 33.34% (95018/284986)
Region Coverage: 32.15% (48731/151565)
Branch Coverage: 27.96% (24568/87854)
Coverage Report: http://coverage.selectdb-in.cc/coverage/30a118a67444fe18691a957f2b6ea5501bdb95b6_30a118a67444fe18691a957f2b6ea5501bdb95b6/report/index.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
dev/2.1.x dev/3.0.x
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@hubgeter @Thearas @doris-robot @morningman