Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat](scan) Adaptive scan concurrency #44690

Merged
merged 14 commits into from
Feb 6, 2025

Conversation

zhiqiang-hhhh
Copy link
Contributor

@zhiqiang-hhhh zhiqiang-hhhh commented Nov 27, 2024

What problem does this PR solve?

Implementation of adaptive scan concurrency. We submit all scanner to scheduler at once in the past. It will introduce much problems:

  1. The execution of scan task is not even between different query
  2. Memory peak of scan task
  3. When running scan task consumes all free block, other remaining scanner will be bubbles in scan scheduler.

So that we want to adjust concurrency of scanner of each scan operator by monitoring system scan pressure.

  1. Make full utilization of resource even if there is just one query.
  2. Distribute resource between all scanners.

Two new session variables are introduced:

  1. min_scanner_concurrnency: Each scan operator will have at least min_scanner_concurrnency scanner is running;
  2. min_scan_scheduler_concurrency: Minimum active threads of scan scheduler.

The original num_scanner_threads is used as max_scanner_concurrency now.

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

Sorry, something went wrong.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zhiqiang-hhhh zhiqiang-hhhh force-pushed the feat-concurrcy-scan branch 3 times, most recently from 0bf7245 to 990b804 Compare December 2, 2024 11:13
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

1 similar comment
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

2 similar comments
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh zhiqiang-hhhh force-pushed the feat-concurrcy-scan branch 2 times, most recently from 5b91373 to 5bc8071 Compare December 6, 2024 07:54
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

MetricUnit::NANOSECONDS);

SimplifiedScanScheduler::SimplifiedScanScheduler(std::string sched_name,
std::shared_ptr<CgroupCpuCtl> cg_cpu_ctl)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: pass by value and use std::move [modernize-pass-by-value]

be/src/vec/exec/scan/scanner_scheduler.cpp:399:

-         : _is_stop(false), _cgroup_cpu_ctl(cg_cpu_ctl), _sched_name(sched_name) {
+         : _is_stop(false), _cgroup_cpu_ctl(cg_cpu_ctl), _sched_name(std::move(sched_name)) {

@zhiqiang-hhhh zhiqiang-hhhh force-pushed the feat-concurrcy-scan branch 2 times, most recently from 9a06917 to 041a17b Compare December 16, 2024 07:36
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

MetricUnit::NANOSECONDS);

SimplifiedScanScheduler::SimplifiedScanScheduler(std::string sched_name,
std::shared_ptr<CgroupCpuCtl> cg_cpu_ctl)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: pass by value and use std::move [modernize-pass-by-value]

be/src/vec/exec/scan/scanner_scheduler.cpp:397:

- _sched_name(sched_name) {
+ _sched_name(std::move(sched_name)) {

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40800 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 504645872e4216e94f69fc2d4dc418efefbf2a89, data reload: false

------ Round 1 ----------------------------------
q1	17870	8070	7841	7841
q2	2055	185	175	175
q3	10698	1031	1180	1031
q4	10418	781	751	751
q5	7601	2839	2868	2839
q6	241	150	149	149
q7	1028	629	612	612
q8	9355	2015	2010	2010
q9	6518	6147	6228	6147
q10	6961	2329	2350	2329
q11	469	280	256	256
q12	434	239	224	224
q13	17774	2959	2980	2959
q14	242	216	212	212
q15	548	504	491	491
q16	677	592	601	592
q17	1021	651	616	616
q18	7429	6849	6837	6837
q19	1342	1180	1153	1153
q20	480	181	189	181
q21	3684	3088	3106	3088
q22	372	319	307	307
Total cold run time: 107217 ms
Total hot run time: 40800 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7748	8469	7749	7749
q2	326	235	233	233
q3	3072	3100	3030	3030
q4	2163	2039	1959	1959
q5	5912	5927	5902	5902
q6	233	150	152	150
q7	2306	1905	1892	1892
q8	3561	3663	3611	3611
q9	9014	9166	9180	9166
q10	3573	3619	3596	3596
q11	603	505	528	505
q12	815	657	660	657
q13	15392	3098	3160	3098
q14	314	301	304	301
q15	568	505	510	505
q16	721	676	636	636
q17	1921	1746	1698	1698
q18	8782	8123	7731	7731
q19	1843	1681	1697	1681
q20	2128	1926	1973	1926
q21	6035	5842	5828	5828
q22	653	607	586	586
Total cold run time: 77683 ms
Total hot run time: 62440 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 199031 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 504645872e4216e94f69fc2d4dc418efefbf2a89, data reload: false

query1	1316	952	944	944
query2	6241	2495	2567	2495
query3	10974	4728	4774	4728
query4	26483	23958	23885	23885
query5	4369	525	507	507
query6	301	197	197	197
query7	3995	300	307	300
query8	317	247	223	223
query9	8433	2667	2663	2663
query10	475	266	261	261
query11	16024	15419	15255	15255
query12	165	107	121	107
query13	1564	418	396	396
query14	9833	7483	7337	7337
query15	240	181	186	181
query16	7723	473	485	473
query17	1525	656	625	625
query18	2026	345	374	345
query19	208	173	177	173
query20	127	119	124	119
query21	206	107	107	107
query22	4881	4719	4592	4592
query23	34735	33775	33728	33728
query24	9674	2797	2871	2797
query25	619	400	418	400
query26	1331	162	160	160
query27	2457	347	357	347
query28	6292	2448	2461	2448
query29	922	426	436	426
query30	242	157	157	157
query31	1024	871	855	855
query32	97	58	64	58
query33	736	311	309	309
query34	1199	540	538	538
query35	907	783	765	765
query36	1128	963	982	963
query37	148	84	81	81
query38	4344	4232	4377	4232
query39	1546	1456	1460	1456
query40	205	107	104	104
query41	53	51	54	51
query42	123	107	112	107
query43	566	534	531	531
query44	1294	823	824	823
query45	196	177	178	177
query46	1227	780	807	780
query47	2043	1986	2002	1986
query48	486	322	305	305
query49	1024	409	411	409
query50	933	435	434	434
query51	7247	7141	7210	7141
query52	111	94	95	94
query53	275	201	211	201
query54	772	426	439	426
query55	88	82	86	82
query56	272	260	254	254
query57	1340	1202	1223	1202
query58	253	239	235	235
query59	3359	3151	2920	2920
query60	270	254	247	247
query61	115	110	120	110
query62	887	711	698	698
query63	227	234	193	193
query64	4053	685	664	664
query65	3341	3213	3224	3213
query66	806	319	313	313
query67	16326	15618	15573	15573
query68	5740	598	580	580
query69	514	264	270	264
query70	1158	1157	1080	1080
query71	523	249	249	249
query72	6823	3805	4077	3805
query73	907	353	358	353
query74	9729	8973	8857	8857
query75	3776	2684	2704	2684
query76	3801	1339	1454	1339
query77	578	286	274	274
query78	10424	9577	9553	9553
query79	2005	617	621	617
query80	720	445	457	445
query81	517	233	237	233
query82	679	124	121	121
query83	186	156	146	146
query84	236	69	67	67
query85	1354	416	301	301
query86	389	299	290	290
query87	4574	4415	4506	4415
query88	3444	2207	2197	2197
query89	462	308	291	291
query90	2081	194	192	192
query91	147	104	107	104
query92	67	55	61	55
query93	1528	627	612	612
query94	850	301	299	299
query95	371	246	255	246
query96	648	271	263	263
query97	2889	2668	2692	2668
query98	224	198	201	198
query99	1609	1327	1342	1327
Total cold run time: 291672 ms
Total hot run time: 199031 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.68 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 504645872e4216e94f69fc2d4dc418efefbf2a89, data reload: false

query1	0.03	0.03	0.03
query2	0.11	0.04	0.04
query3	0.28	0.06	0.06
query4	1.59	0.08	0.08
query5	0.42	0.41	0.40
query6	1.18	0.66	0.65
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.63	0.55	0.55
query10	0.61	0.63	0.60
query11	0.25	0.13	0.13
query12	0.25	0.13	0.13
query13	0.62	0.61	0.60
query14	2.68	2.71	2.80
query15	1.01	0.88	0.88
query16	0.38	0.38	0.39
query17	1.05	1.04	1.05
query18	0.16	0.16	0.16
query19	2.01	1.84	2.04
query20	0.02	0.04	0.03
query21	15.35	0.68	0.67
query22	3.88	6.90	2.22
query23	16.58	1.96	1.31
query24	5.47	0.31	0.28
query25	0.18	0.09	0.09
query26	0.57	0.18	0.18
query27	0.09	0.09	0.09
query28	11.12	1.09	1.06
query29	12.58	3.46	3.45
query30	0.27	0.05	0.05
query31	2.84	0.43	0.42
query32	3.24	0.51	0.50
query33	3.12	3.06	3.09
query34	16.21	4.38	4.38
query35	4.53	4.41	4.45
query36	0.68	0.50	0.50
query37	0.20	0.17	0.17
query38	0.16	0.15	0.16
query39	0.05	0.04	0.04
query40	0.20	0.14	0.15
query41	0.10	0.06	0.05
query42	0.06	0.05	0.05
query43	0.06	0.05	0.05
Total cold run time: 110.89 s
Total hot run time: 33.68 s

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh zhiqiang-hhhh force-pushed the feat-concurrcy-scan branch 2 times, most recently from e6b5b6c to 01a41c7 Compare December 23, 2024 13:18
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40342 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 01a41c777acef54100899009ae1a290bfced0459, data reload: false

------ Round 1 ----------------------------------
q1	17650	8069	7700	7700
q2	2073	178	187	178
q3	10518	1136	1087	1087
q4	10217	804	769	769
q5	7584	2867	2805	2805
q6	238	149	143	143
q7	1039	636	620	620
q8	9428	1980	2009	1980
q9	6538	6017	6054	6017
q10	6965	2321	2306	2306
q11	453	261	257	257
q12	432	225	231	225
q13	17781	3000	2948	2948
q14	254	218	220	218
q15	552	513	501	501
q16	678	606	587	587
q17	989	581	575	575
q18	7541	6760	6714	6714
q19	1347	1165	1158	1158
q20	470	188	191	188
q21	3555	3099	3057	3057
q22	361	328	309	309
Total cold run time: 106663 ms
Total hot run time: 40342 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7709	7672	7634	7634
q2	332	226	232	226
q3	2924	3046	3018	3018
q4	2098	1989	1975	1975
q5	5842	5863	5867	5863
q6	234	147	146	146
q7	2273	1815	1836	1815
q8	3460	3569	3569	3569
q9	8989	9089	9061	9061
q10	3546	3558	3554	3554
q11	606	527	500	500
q12	833	614	643	614
q13	12238	3060	3098	3060
q14	310	278	301	278
q15	541	509	504	504
q16	657	660	652	652
q17	1903	1721	1682	1682
q18	8683	8043	7898	7898
q19	1704	1612	1616	1612
q20	2111	1892	1894	1892
q21	6203	5961	5920	5920
q22	649	616	572	572
Total cold run time: 73845 ms
Total hot run time: 62045 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.78% (10092/26025)
Line Coverage: 29.77% (85126/285920)
Region Coverage: 28.88% (43451/150431)
Branch Coverage: 25.42% (22148/87118)
Coverage Report: http://coverage.selectdb-in.cc/coverage/01a41c777acef54100899009ae1a290bfced0459_01a41c777acef54100899009ae1a290bfced0459/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 197062 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 01a41c777acef54100899009ae1a290bfced0459, data reload: false

query1	1327	1010	978	978
query2	6291	2421	2300	2300
query3	10959	4797	4787	4787
query4	26772	23323	23310	23310
query5	5289	511	485	485
query6	309	194	200	194
query7	4020	300	308	300
query8	303	240	238	238
query9	8499	2738	2724	2724
query10	521	251	261	251
query11	15686	15247	15177	15177
query12	167	101	101	101
query13	1562	423	394	394
query14	10191	6623	6971	6623
query15	239	177	183	177
query16	7867	458	507	458
query17	1526	624	596	596
query18	2043	345	339	339
query19	325	178	197	178
query20	131	125	125	125
query21	211	109	108	108
query22	4894	4821	4744	4744
query23	34810	33663	33631	33631
query24	8652	2720	2658	2658
query25	597	413	399	399
query26	722	155	159	155
query27	2618	347	355	347
query28	5252	2506	2505	2505
query29	832	459	447	447
query30	232	153	154	153
query31	1039	838	828	828
query32	93	61	57	57
query33	704	295	317	295
query34	951	565	543	543
query35	916	750	746	746
query36	1111	943	937	937
query37	134	76	73	73
query38	4288	4180	4206	4180
query39	1512	1473	1483	1473
query40	209	102	100	100
query41	54	45	46	45
query42	112	106	105	105
query43	505	514	483	483
query44	1200	806	820	806
query45	186	168	171	168
query46	1163	755	769	755
query47	2059	1993	1980	1980
query48	472	316	327	316
query49	981	430	423	423
query50	914	419	429	419
query51	7433	7198	7226	7198
query52	112	95	102	95
query53	261	190	198	190
query54	650	419	450	419
query55	83	83	86	83
query56	294	265	241	241
query57	1313	1189	1192	1189
query58	254	231	221	221
query59	3266	3077	3009	3009
query60	297	257	261	257
query61	107	106	104	104
query62	870	685	726	685
query63	227	193	194	193
query64	3763	682	663	663
query65	3234	3197	3205	3197
query66	780	293	297	293
query67	15972	15536	15473	15473
query68	5067	561	622	561
query69	484	259	250	250
query70	1126	1136	1077	1077
query71	491	272	269	269
query72	7071	3959	3903	3903
query73	811	356	364	356
query74	9145	8805	8776	8776
query75	3749	2658	2692	2658
query76	3680	1156	1179	1156
query77	565	301	295	295
query78	10438	9506	9499	9499
query79	1147	626	634	626
query80	756	443	446	443
query81	515	233	221	221
query82	430	122	119	119
query83	201	150	150	150
query84	242	77	69	69
query85	1055	310	306	306
query86	334	293	294	293
query87	4595	4418	4469	4418
query88	2770	2279	2256	2256
query89	421	289	287	287
query90	2020	192	193	192
query91	146	107	107	107
query92	63	51	55	51
query93	1474	594	588	588
query94	828	283	268	268
query95	366	332	255	255
query96	598	278	277	277
query97	2869	2747	2698	2698
query98	218	194	201	194
query99	1536	1320	1326	1320
Total cold run time: 285577 ms
Total hot run time: 197062 ms

@zhiqiang-hhhh
Copy link
Contributor Author

run build all

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31671 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 22aca5d2db5309651f5ff0c588e0ecdd04b2f485, data reload: false

------ Round 1 ----------------------------------
q1	17579	5263	5117	5117
q2	2050	313	182	182
q3	10452	1289	743	743
q4	10213	1016	542	542
q5	7545	2339	2445	2339
q6	189	165	134	134
q7	908	753	593	593
q8	9292	1282	1074	1074
q9	4906	4652	4848	4652
q10	6831	2311	1875	1875
q11	462	272	262	262
q12	347	361	227	227
q13	17774	3691	3160	3160
q14	239	235	208	208
q15	532	462	453	453
q16	656	615	589	589
q17	560	873	342	342
q18	6699	6263	6185	6185
q19	1451	953	553	553
q20	318	330	203	203
q21	2798	2252	1939	1939
q22	367	324	299	299
Total cold run time: 102168 ms
Total hot run time: 31671 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5200	5197	5189	5189
q2	240	328	233	233
q3	2152	2696	2289	2289
q4	1426	1881	1409	1409
q5	4263	4129	4133	4129
q6	204	163	124	124
q7	1914	1871	1749	1749
q8	2629	2598	2560	2560
q9	7291	7266	7148	7148
q10	3059	3247	2795	2795
q11	580	519	492	492
q12	706	825	650	650
q13	3441	4032	3281	3281
q14	294	286	273	273
q15	527	486	471	471
q16	675	685	627	627
q17	1170	1631	1334	1334
q18	7618	7420	7237	7237
q19	776	770	838	770
q20	1984	2048	1886	1886
q21	5479	5205	4752	4752
q22	633	567	577	567
Total cold run time: 52261 ms
Total hot run time: 49965 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191141 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 22aca5d2db5309651f5ff0c588e0ecdd04b2f485, data reload: false

query1	1303	969	939	939
query2	6318	1812	1836	1812
query3	11121	4705	4873	4705
query4	26098	24144	23116	23116
query5	3940	699	493	493
query6	296	211	198	198
query7	3996	527	309	309
query8	305	254	233	233
query9	8479	2611	2615	2611
query10	472	331	262	262
query11	15764	15728	15071	15071
query12	189	113	113	113
query13	1577	532	404	404
query14	10108	6547	6870	6547
query15	210	203	187	187
query16	7442	743	506	506
query17	1114	731	572	572
query18	1835	423	323	323
query19	211	201	184	184
query20	139	124	119	119
query21	202	156	111	111
query22	4504	4425	4349	4349
query23	34689	33438	33166	33166
query24	7552	2428	2479	2428
query25	514	466	394	394
query26	1174	280	154	154
query27	2093	522	348	348
query28	4130	2466	2446	2446
query29	661	587	465	465
query30	224	198	164	164
query31	924	898	829	829
query32	82	71	63	63
query33	571	389	347	347
query34	828	937	542	542
query35	858	891	811	811
query36	997	1030	914	914
query37	127	107	81	81
query38	4262	4272	4349	4272
query39	1556	1476	1429	1429
query40	220	122	111	111
query41	52	49	51	49
query42	127	108	112	108
query43	490	524	486	486
query44	1313	814	828	814
query45	189	177	168	168
query46	885	1078	661	661
query47	1871	1867	1780	1780
query48	405	434	315	315
query49	787	510	461	461
query50	749	781	447	447
query51	4348	4330	4349	4330
query52	114	115	107	107
query53	243	269	220	220
query54	506	533	422	422
query55	88	83	86	83
query56	274	289	260	260
query57	1195	1205	1094	1094
query58	260	264	248	248
query59	2821	2901	2585	2585
query60	284	283	262	262
query61	143	111	111	111
query62	794	745	720	720
query63	236	196	188	188
query64	4026	1035	676	676
query65	3404	3278	3266	3266
query66	1050	407	308	308
query67	16249	15622	15262	15262
query68	8231	791	502	502
query69	483	307	282	282
query70	1174	1112	1101	1101
query71	406	306	286	286
query72	5100	3590	3827	3590
query73	721	707	358	358
query74	8894	9089	8750	8750
query75	3226	3212	2735	2735
query76	3234	1195	747	747
query77	525	414	289	289
query78	10522	10473	9735	9735
query79	1963	834	595	595
query80	627	534	441	441
query81	508	283	253	253
query82	373	171	121	121
query83	163	178	164	164
query84	241	99	77	77
query85	741	344	309	309
query86	366	305	274	274
query87	4463	4651	4415	4415
query88	2895	2253	2239	2239
query89	396	322	288	288
query90	1872	195	196	195
query91	135	140	110	110
query92	82	64	59	59
query93	2265	1010	575	575
query94	663	415	304	304
query95	356	272	257	257
query96	488	569	273	273
query97	2849	2909	2776	2776
query98	252	200	197	197
query99	1286	1427	1285	1285
Total cold run time: 275488 ms
Total hot run time: 191141 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.06% (10971/26087)
Line Coverage: 32.33% (92745/286839)
Region Coverage: 31.48% (47550/151045)
Branch Coverage: 27.53% (24087/87488)
Coverage Report: http://coverage.selectdb-in.cc/coverage/22aca5d2db5309651f5ff0c588e0ecdd04b2f485_22aca5d2db5309651f5ff0c588e0ecdd04b2f485/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 31795 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 22aca5d2db5309651f5ff0c588e0ecdd04b2f485, data reload: false

------ Round 1 ----------------------------------
q1	17732	5184	5178	5178
q2	2056	299	177	177
q3	10505	1304	709	709
q4	10255	1009	538	538
q5	8592	2390	2448	2390
q6	190	164	141	141
q7	926	778	630	630
q8	9312	1355	1121	1121
q9	5001	4597	4654	4597
q10	6818	2336	1903	1903
q11	476	275	262	262
q12	352	360	220	220
q13	17769	3725	3092	3092
q14	241	248	207	207
q15	514	466	461	461
q16	625	632	606	606
q17	597	883	344	344
q18	6650	6338	6167	6167
q19	1294	972	571	571
q20	313	339	196	196
q21	2889	2362	1968	1968
q22	377	339	317	317
Total cold run time: 103484 ms
Total hot run time: 31795 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5361	5214	5128	5128
q2	235	328	230	230
q3	2169	2689	2298	2298
q4	1405	1891	1342	1342
q5	4276	4220	4158	4158
q6	209	169	124	124
q7	1893	1917	1826	1826
q8	2599	2633	2675	2633
q9	7181	7065	7219	7065
q10	3056	3202	2787	2787
q11	583	500	507	500
q12	706	788	645	645
q13	3590	3856	3245	3245
q14	268	286	291	286
q15	518	458	457	457
q16	648	677	639	639
q17	1138	1606	1358	1358
q18	7630	7369	7134	7134
q19	863	826	950	826
q20	1981	1986	1862	1862
q21	5478	5040	4900	4900
q22	672	642	574	574
Total cold run time: 52459 ms
Total hot run time: 50017 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183480 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 22aca5d2db5309651f5ff0c588e0ecdd04b2f485, data reload: false

query1	988	387	382	382
query2	6555	1818	1766	1766
query3	6810	223	210	210
query4	25942	23925	23257	23257
query5	4380	677	485	485
query6	307	207	193	193
query7	4610	519	308	308
query8	300	236	222	222
query9	8616	2631	2631	2631
query10	461	302	269	269
query11	15205	15247	15002	15002
query12	161	114	107	107
query13	1651	518	398	398
query14	9083	6143	6236	6143
query15	226	206	177	177
query16	7328	630	448	448
query17	1210	718	570	570
query18	1966	406	307	307
query19	205	192	157	157
query20	123	114	120	114
query21	209	124	102	102
query22	4225	4234	3960	3960
query23	33976	33112	32995	32995
query24	7669	2403	2387	2387
query25	549	450	392	392
query26	1222	276	150	150
query27	2535	467	333	333
query28	4342	2447	2389	2389
query29	754	555	416	416
query30	236	186	159	159
query31	959	883	762	762
query32	79	64	70	64
query33	554	349	294	294
query34	781	843	506	506
query35	818	793	737	737
query36	972	959	880	880
query37	127	102	78	78
query38	4190	4125	4033	4033
query39	1452	1390	1386	1386
query40	214	120	108	108
query41	60	50	50	50
query42	122	114	114	114
query43	488	490	472	472
query44	1296	792	798	792
query45	176	170	162	162
query46	852	1041	645	645
query47	1749	1753	1721	1721
query48	381	413	299	299
query49	797	515	408	408
query50	686	747	414	414
query51	4210	4197	4089	4089
query52	106	109	95	95
query53	231	254	178	178
query54	484	478	402	402
query55	83	86	81	81
query56	258	260	261	260
query57	1121	1139	1086	1086
query58	259	265	246	246
query59	2497	2803	2666	2666
query60	286	267	266	266
query61	121	145	120	120
query62	804	727	679	679
query63	230	186	189	186
query64	4355	1000	700	700
query65	3217	3128	3127	3127
query66	1134	405	306	306
query67	15687	15553	15475	15475
query68	7495	753	499	499
query69	475	300	258	258
query70	1186	1139	1085	1085
query71	392	323	268	268
query72	6086	3552	3669	3552
query73	740	729	362	362
query74	9002	9110	8927	8927
query75	3137	3147	2709	2709
query76	3257	1153	752	752
query77	475	375	279	279
query78	9892	10112	9294	9294
query79	1864	819	591	591
query80	621	522	448	448
query81	524	277	240	240
query82	226	154	123	123
query83	177	172	157	157
query84	243	94	75	75
query85	733	339	299	299
query86	368	290	286	286
query87	4384	4562	4476	4476
query88	3156	2305	2242	2242
query89	384	303	289	289
query90	1982	200	194	194
query91	132	138	110	110
query92	82	58	59	58
query93	1863	1017	569	569
query94	676	389	301	301
query95	356	267	264	264
query96	497	560	268	268
query97	2733	2842	2709	2709
query98	241	204	201	201
query99	1345	1414	1279	1279
Total cold run time: 268404 ms
Total hot run time: 183480 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.81 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 22aca5d2db5309651f5ff0c588e0ecdd04b2f485, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.63	0.10	0.10
query5	0.40	0.41	0.39
query6	1.20	0.66	0.66
query7	0.02	0.01	0.02
query8	0.04	0.04	0.03
query9	0.58	0.52	0.51
query10	0.57	0.55	0.55
query11	0.15	0.10	0.10
query12	0.15	0.11	0.12
query13	0.63	0.60	0.59
query14	2.78	2.82	2.71
query15	0.91	0.85	0.87
query16	0.38	0.38	0.37
query17	1.02	1.06	1.05
query18	0.21	0.20	0.19
query19	1.87	1.79	2.00
query20	0.02	0.01	0.01
query21	15.37	0.94	0.53
query22	0.76	1.17	0.73
query23	14.86	1.36	0.61
query24	6.87	1.00	1.70
query25	0.52	0.22	0.09
query26	0.52	0.16	0.13
query27	0.06	0.05	0.05
query28	10.31	0.89	0.45
query29	12.62	3.95	3.28
query30	0.25	0.09	0.06
query31	2.85	0.58	0.38
query32	3.23	0.54	0.47
query33	2.96	3.00	3.02
query34	15.76	5.19	4.51
query35	4.56	4.58	4.58
query36	0.66	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.03
query43	0.04	0.04	0.03
Total cold run time: 105.54 s
Total hot run time: 30.81 s

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31516 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a4b6fc36e77f4297742da2626d036f3b6152a8e5, data reload: false

------ Round 1 ----------------------------------
q1	17583	5177	5074	5074
q2	2065	328	169	169
q3	10376	1248	761	761
q4	10222	1024	545	545
q5	7548	2448	2325	2325
q6	197	167	134	134
q7	922	775	602	602
q8	9315	1322	1045	1045
q9	4901	4559	4774	4559
q10	6809	2285	1923	1923
q11	503	272	254	254
q12	346	360	220	220
q13	17764	3714	3137	3137
q14	226	228	205	205
q15	519	469	469	469
q16	624	611	596	596
q17	571	857	361	361
q18	6608	6201	6173	6173
q19	1766	963	559	559
q20	318	323	195	195
q21	2830	2160	1911	1911
q22	373	343	299	299
Total cold run time: 102386 ms
Total hot run time: 31516 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5166	5097	5093	5093
q2	247	325	231	231
q3	2306	2702	2328	2328
q4	1428	1840	1355	1355
q5	4251	4126	4152	4126
q6	206	160	124	124
q7	1884	1800	1693	1693
q8	2636	2689	2575	2575
q9	7304	7150	7140	7140
q10	3067	3232	2808	2808
q11	585	525	486	486
q12	692	779	637	637
q13	3499	3915	3313	3313
q14	282	295	273	273
q15	496	464	466	464
q16	634	697	631	631
q17	1126	1550	1380	1380
q18	7555	7418	7214	7214
q19	789	808	905	808
q20	1984	2035	1914	1914
q21	5533	5044	4834	4834
q22	610	582	528	528
Total cold run time: 52280 ms
Total hot run time: 49955 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191029 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a4b6fc36e77f4297742da2626d036f3b6152a8e5, data reload: false

query1	1326	988	906	906
query2	6226	1842	1846	1842
query3	11095	4628	4682	4628
query4	54587	25714	23487	23487
query5	5341	517	514	514
query6	367	196	187	187
query7	5139	501	293	293
query8	318	249	248	248
query9	6469	2642	2643	2642
query10	418	317	269	269
query11	15394	15026	15260	15026
query12	163	107	106	106
query13	1194	524	391	391
query14	10630	6473	6814	6473
query15	210	195	189	189
query16	7037	673	483	483
query17	1106	732	598	598
query18	1454	403	315	315
query19	215	200	165	165
query20	126	128	125	125
query21	210	130	104	104
query22	4521	4396	4433	4396
query23	34042	33445	33482	33445
query24	5849	2496	2521	2496
query25	460	451	392	392
query26	723	281	155	155
query27	1862	481	341	341
query28	2818	2489	2453	2453
query29	589	553	426	426
query30	220	185	156	156
query31	878	879	782	782
query32	70	63	62	62
query33	437	350	301	301
query34	768	860	518	518
query35	813	840	751	751
query36	961	1000	939	939
query37	129	101	81	81
query38	4233	4330	4413	4330
query39	1472	1452	1409	1409
query40	217	122	101	101
query41	53	51	54	51
query42	123	108	107	107
query43	505	539	499	499
query44	1342	805	824	805
query45	175	172	170	170
query46	909	1063	674	674
query47	1806	1918	1782	1782
query48	397	420	309	309
query49	702	529	430	430
query50	690	756	423	423
query51	4295	4253	4267	4253
query52	115	109	99	99
query53	251	271	201	201
query54	524	529	431	431
query55	84	77	80	77
query56	294	270	246	246
query57	1176	1184	1120	1120
query58	246	249	241	241
query59	2799	2899	2749	2749
query60	285	283	263	263
query61	125	117	119	117
query62	784	751	695	695
query63	228	200	197	197
query64	1738	1054	666	666
query65	3257	3219	3161	3161
query66	756	409	301	301
query67	15983	15682	15335	15335
query68	5416	785	507	507
query69	530	301	268	268
query70	1254	1137	1120	1120
query71	451	300	289	289
query72	5914	3506	3836	3506
query73	1128	735	351	351
query74	9195	9002	8947	8947
query75	3268	3141	2701	2701
query76	3992	1170	745	745
query77	545	394	285	285
query78	10128	10059	9314	9314
query79	2023	839	585	585
query80	650	534	443	443
query81	517	280	238	238
query82	412	156	118	118
query83	173	167	159	159
query84	290	90	81	81
query85	727	352	300	300
query86	345	317	281	281
query87	4330	4500	4336	4336
query88	3255	2243	2277	2243
query89	390	342	307	307
query90	1863	198	190	190
query91	127	137	107	107
query92	82	60	56	56
query93	2728	999	580	580
query94	659	381	297	297
query95	354	270	265	265
query96	484	567	272	272
query97	2758	2865	2744	2744
query98	232	227	199	199
query99	1288	1424	1268	1268
Total cold run time: 295237 ms
Total hot run time: 191029 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.12 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a4b6fc36e77f4297742da2626d036f3b6152a8e5, data reload: false

query1	0.03	0.03	0.03
query2	0.10	0.04	0.05
query3	0.29	0.06	0.06
query4	1.60	0.07	0.08
query5	0.42	0.39	0.39
query6	1.16	0.64	0.66
query7	0.02	0.02	0.02
query8	0.06	0.05	0.05
query9	0.62	0.53	0.52
query10	0.58	0.58	0.57
query11	0.26	0.13	0.12
query12	0.25	0.12	0.13
query13	0.63	0.61	0.61
query14	2.81	2.74	2.80
query15	0.98	0.89	0.86
query16	0.36	0.38	0.38
query17	1.02	1.03	1.06
query18	0.19	0.19	0.17
query19	1.93	1.83	2.00
query20	0.02	0.02	0.01
query21	15.35	0.96	0.65
query22	0.93	0.99	0.80
query23	14.72	1.54	0.74
query24	5.54	0.56	0.28
query25	0.17	0.09	0.08
query26	0.55	0.22	0.18
query27	0.08	0.08	0.09
query28	11.01	1.15	0.53
query29	12.54	4.11	3.37
query30	0.28	0.07	0.06
query31	2.81	0.60	0.42
query32	3.23	0.59	0.50
query33	3.00	3.05	3.04
query34	16.41	5.19	4.45
query35	4.51	4.53	4.57
query36	0.63	0.52	0.51
query37	0.21	0.16	0.17
query38	0.16	0.16	0.15
query39	0.05	0.04	0.04
query40	0.19	0.15	0.15
query41	0.10	0.05	0.05
query42	0.06	0.06	0.05
query43	0.05	0.04	0.04
Total cold run time: 105.91 s
Total hot run time: 31.12 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.05% (10989/26132)
Line Coverage: 32.33% (92780/286973)
Region Coverage: 31.48% (47562/151110)
Branch Coverage: 27.52% (24080/87506)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a4b6fc36e77f4297742da2626d036f3b6152a8e5_a4b6fc36e77f4297742da2626d036f3b6152a8e5/report/index.html

@zhiqiang-hhhh
Copy link
Contributor Author

run cloud_p0

Copy link
Contributor

@kaka11chen kaka11chen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

github-actions bot commented Feb 6, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 6, 2025
Copy link
Contributor

github-actions bot commented Feb 6, 2025

PR approved by anyone and no changes requested.

@yiguolei yiguolei merged commit 017f85f into apache:master Feb 6, 2025
25 of 26 checks passed
@zhiqiang-hhhh zhiqiang-hhhh deleted the feat-concurrcy-scan branch February 6, 2025 06:58
yiguolei pushed a commit that referenced this pull request Feb 18, 2025

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
### What problem does this PR solve?

Unit test for scanner schedule. Adaptive scan schedule is introduced by
#44690

* ScannerContext::init
* ScannerContext::_push_back_scan_task
* ScannerContext::_get_margin
* ScannerContext::_pull_next_scan_task
* ScannerContext::_schedule_scan_task 
* Additional test for scan operator, make sure
`adaptive_pipeline_task_serial_read_on_limit` is working correctlly.
* ScannerContext::get_free_block
* ScannerContext::return_free_block
* ScannerContext::get_block_from_queue
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Implementation of adaptive scan concurrency. We submit all scanner to
scheduler at once in the past. It will introduce much problems:

1. The execution of scan task is not even between different query
2. Memory peak of scan task
3. When running scan task consumes all free block, other remaining
scanner will be bubbles in scan scheduler.

So that we want to adjust concurrency of scanner of each scan operator
by monitoring system scan pressure.

1. Make full utilization of resource even if there is just one query.
2. Distribute resource between all scanners.

Two new session variables are introduced:
1. min_scanner_concurrnency: Each scan operator will have at least
min_scanner_concurrnency scanner is running;
2. min_scan_scheduler_concurrency: Minimum active threads of scan
scheduler.

The original num_scanner_threads is used as max_scanner_concurrency now.

None
、
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
### What problem does this PR solve?

Unit test for scanner schedule. Adaptive scan schedule is introduced by
apache#44690

* ScannerContext::init
* ScannerContext::_push_back_scan_task
* ScannerContext::_get_margin
* ScannerContext::_pull_next_scan_task
* ScannerContext::_schedule_scan_task 
* Additional test for scan operator, make sure
`adaptive_pipeline_task_serial_read_on_limit` is working correctlly.
* ScannerContext::get_free_block
* ScannerContext::return_free_block
* ScannerContext::get_block_from_queue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. not-merge/2.1 not-merge/3.0 reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants