Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Enhancement] config rf wait time in cloud mode #47579

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

[Enhancement] config rf wait time in cloud mode #47579

wants to merge 2 commits into from
+39 −30

Conversation

xzj7019
Copy link
Contributor

@xzj7019 xzj7019 commented Feb 7, 2025 •
edited
Loading

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: config rf wait time in cloud mode

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@xzj7019 xzj7019 marked this pull request as draft February 7, 2025 04:37
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@xzj7019
Copy link
Contributor Author

xzj7019 commented Feb 7, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31349 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b9dcb01c2627017719e4533095e6904546d478b3, data reload: false

------ Round 1 ----------------------------------
q1	17585	5233	5019	5019
q2	2044	300	175	175
q3	10483	1245	773	773
q4	10202	998	546	546
q5	7765	2302	2426	2302
q6	195	171	134	134
q7	912	760	590	590
q8	9303	1287	1098	1098
q9	4782	4628	4785	4628
q10	6813	2313	1875	1875
q11	476	272	247	247
q12	348	354	211	211
q13	17754	3664	3080	3080
q14	223	237	214	214
q15	516	468	461	461
q16	616	611	571	571
q17	578	840	359	359
q18	6875	6140	6116	6116
q19	1198	938	539	539
q20	306	324	179	179
q21	2815	2103	1936	1936
q22	356	330	296	296
Total cold run time: 102145 ms
Total hot run time: 31349 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5070	5074	5083	5074
q2	234	323	228	228
q3	2179	2678	2335	2335
q4	1482	1844	1383	1383
q5	4221	4120	4121	4120
q6	209	163	126	126
q7	1859	1802	1632	1632
q8	2553	2664	2532	2532
q9	7211	7218	7186	7186
q10	3026	3202	2763	2763
q11	566	500	501	500
q12	681	784	604	604
q13	3524	3879	3264	3264
q14	294	303	257	257
q15	522	455	477	455
q16	650	705	638	638
q17	1138	1560	1338	1338
q18	7713	7297	7335	7297
q19	775	775	843	775
q20	1990	1997	1839	1839
q21	5438	5003	4693	4693
q22	646	582	553	553
Total cold run time: 51981 ms
Total hot run time: 49592 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190159 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b9dcb01c2627017719e4533095e6904546d478b3, data reload: false

query1	1308	956	928	928
query2	6200	1815	1843	1815
query3	10993	4547	4568	4547
query4	56870	26030	23718	23718
query5	5225	494	472	472
query6	401	180	191	180
query7	5311	508	286	286
query8	324	250	233	233
query9	7351	2494	2523	2494
query10	414	312	254	254
query11	16060	15230	14871	14871
query12	166	111	110	110
query13	1300	568	415	415
query14	10253	6420	7201	6420
query15	213	206	196	196
query16	7079	668	499	499
query17	1046	702	576	576
query18	1542	403	320	320
query19	204	189	157	157
query20	138	119	122	119
query21	207	120	104	104
query22	4258	4335	4467	4335
query23	33914	33505	33158	33158
query24	5707	2446	2418	2418
query25	467	452	397	397
query26	658	269	158	158
query27	1774	500	339	339
query28	3178	2434	2423	2423
query29	581	575	427	427
query30	212	201	157	157
query31	906	905	802	802
query32	75	65	70	65
query33	448	367	308	308
query34	785	846	522	522
query35	813	829	741	741
query36	927	1005	922	922
query37	142	104	78	78
query38	4281	4303	4317	4303
query39	1482	1424	1461	1424
query40	218	120	104	104
query41	49	56	50	50
query42	128	104	104	104
query43	513	494	479	479
query44	1344	808	798	798
query45	178	175	163	163
query46	907	1073	652	652
query47	1869	1846	1780	1780
query48	389	427	314	314
query49	707	531	425	425
query50	726	749	438	438
query51	4191	4287	4379	4287
query52	109	108	92	92
query53	242	271	185	185
query54	493	479	414	414
query55	88	77	78	77
query56	262	288	256	256
query57	1175	1167	1122	1122
query58	261	241	260	241
query59	2828	2800	2632	2632
query60	302	284	280	280
query61	123	118	115	115
query62	750	752	672	672
query63	230	189	189	189
query64	1476	1040	684	684
query65	3311	3146	3118	3118
query66	670	398	298	298
query67	15768	15471	15347	15347
query68	4577	784	522	522
query69	494	296	257	257
query70	1182	1108	1110	1108
query71	414	305	277	277
query72	5234	3696	3800	3696
query73	800	747	341	341
query74	8987	9136	8918	8918
query75	3243	3127	2658	2658
query76	2980	1164	736	736
query77	578	380	289	289
query78	10107	10385	9327	9327
query79	983	897	582	582
query80	785	542	445	445
query81	523	272	236	236
query82	263	153	128	128
query83	188	164	148	148
query84	279	97	75	75
query85	880	352	297	297
query86	320	314	272	272
query87	4521	4550	4430	4430
query88	2761	2192	2149	2149
query89	430	305	287	287
query90	1659	194	187	187
query91	134	146	106	106
query92	67	60	54	54
query93	1545	1005	575	575
query94	601	394	299	299
query95	342	303	251	251
query96	473	577	262	262
query97	2750	2857	2753	2753
query98	227	195	201	195
query99	1311	1399	1292	1292
Total cold run time: 292158 ms
Total hot run time: 190159 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.51 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b9dcb01c2627017719e4533095e6904546d478b3, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.04	0.03
query3	0.24	0.07	0.06
query4	1.61	0.10	0.11
query5	0.41	0.42	0.40
query6	1.15	0.67	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.58	0.55	0.52
query10	0.57	0.59	0.59
query11	0.15	0.11	0.11
query12	0.15	0.10	0.11
query13	0.62	0.60	0.60
query14	2.72	2.85	2.81
query15	0.92	0.85	0.83
query16	0.37	0.38	0.39
query17	1.06	1.03	1.02
query18	0.21	0.19	0.20
query19	1.91	1.84	2.03
query20	0.01	0.01	0.01
query21	15.37	0.89	0.55
query22	0.75	1.07	0.65
query23	15.06	1.39	0.62
query24	7.21	2.02	0.55
query25	0.51	0.25	0.18
query26	0.55	0.17	0.13
query27	0.05	0.05	0.05
query28	9.21	0.84	0.41
query29	12.58	4.04	3.29
query30	0.25	0.08	0.06
query31	2.83	0.59	0.38
query32	3.23	0.54	0.45
query33	2.98	3.04	3.05
query34	15.62	5.10	4.50
query35	4.55	4.52	4.53
query36	0.69	0.50	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.17	0.14	0.13
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 104.75 s
Total hot run time: 30.51 s

@xzj7019 xzj7019 marked this pull request as ready for review February 7, 2025 07:52
englefly
englefly previously approved these changes Feb 8, 2025
View reviewed changes
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 8, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 8, 2025
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 8, 2025

PR approved by anyone and no changes requested.

@xzj7019 xzj7019 force-pushed the config_rftime_cloudmode branch from b9dcb01 to bb683f8 Compare February 10, 2025 06:16
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Feb 10, 2025
englefly
englefly reviewed Feb 10, 2025
View reviewed changes
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java
sessionVariable.setVarOnce(SessionVariable.RUNTIME_FILTER_WAIT_TIME_MS, "300000");
sessionVariable.setVarOnce(SessionVariable.RUNTIME_FILTER_WAIT_TIME_MS,
String.valueOf(Math.max(VariableMgr.getDefaultSessionVariable().getRuntimeFilterWaitTimeMs(),
1000 * sessionVariable.getQueryTimeoutS())));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if wait time is query_timeout, it is equivalents to wait infinitely.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is expected in cloud mode currently..

@xzj7019
Copy link
Contributor Author

xzj7019 commented Feb 10, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31314 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bb683f8f5233fcdccb49e7ebc36c4e10b76f5d83, data reload: false

------ Round 1 ----------------------------------
q1	17616	5338	5076	5076
q2	2050	297	176	176
q3	10403	1240	747	747
q4	10206	992	526	526
q5	7546	2393	2270	2270
q6	187	167	134	134
q7	905	737	591	591
q8	9289	1292	1086	1086
q9	4772	4794	4699	4699
q10	6830	2306	1866	1866
q11	483	279	252	252
q12	340	348	219	219
q13	17769	3693	3064	3064
q14	224	224	218	218
q15	511	468	458	458
q16	627	621	585	585
q17	565	855	328	328
q18	6543	6245	6077	6077
q19	1212	941	535	535
q20	310	322	197	197
q21	2948	2177	1902	1902
q22	374	331	308	308
Total cold run time: 101710 ms
Total hot run time: 31314 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5539	5123	5152	5123
q2	239	330	229	229
q3	2242	2810	2434	2434
q4	1481	1877	1403	1403
q5	4347	4141	4138	4138
q6	205	165	126	126
q7	1857	1815	1672	1672
q8	2586	2709	2662	2662
q9	7140	7044	7166	7044
q10	3002	3181	2793	2793
q11	576	519	502	502
q12	716	768	626	626
q13	3424	3984	3325	3325
q14	303	298	257	257
q15	504	473	464	464
q16	656	670	662	662
q17	1103	1582	1347	1347
q18	7664	7384	7295	7295
q19	801	783	850	783
q20	1971	2041	1868	1868
q21	5397	4905	4709	4709
q22	613	575	548	548
Total cold run time: 52366 ms
Total hot run time: 50010 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191195 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bb683f8f5233fcdccb49e7ebc36c4e10b76f5d83, data reload: false

query1	1304	956	925	925
query2	6220	1956	1895	1895
query3	10947	4356	4345	4345
query4	53667	25610	23625	23625
query5	5081	551	510	510
query6	360	194	187	187
query7	4988	514	295	295
query8	316	246	237	237
query9	5979	2529	2503	2503
query10	418	311	259	259
query11	15230	15413	14970	14970
query12	162	111	118	111
query13	1086	524	383	383
query14	10677	6899	6511	6511
query15	205	193	200	193
query16	7047	647	492	492
query17	1074	737	598	598
query18	1511	436	324	324
query19	208	211	179	179
query20	133	133	128	128
query21	215	126	107	107
query22	4435	4493	4443	4443
query23	33910	33327	33593	33327
query24	5982	2406	2446	2406
query25	453	480	395	395
query26	737	283	154	154
query27	1983	487	338	338
query28	2972	2439	2410	2410
query29	573	559	454	454
query30	214	197	179	179
query31	905	874	808	808
query32	81	66	67	66
query33	439	423	315	315
query34	775	861	511	511
query35	796	823	757	757
query36	943	1005	908	908
query37	140	108	80	80
query38	4226	4294	4303	4294
query39	1501	1487	1633	1487
query40	215	120	103	103
query41	50	52	48	48
query42	125	105	112	105
query43	499	531	514	514
query44	1323	805	796	796
query45	176	174	170	170
query46	889	1086	671	671
query47	1829	1857	1772	1772
query48	400	407	314	314
query49	671	527	421	421
query50	709	746	437	437
query51	4308	4309	4339	4309
query52	105	106	92	92
query53	234	257	193	193
query54	488	500	412	412
query55	82	94	85	85
query56	280	283	265	265
query57	1170	1172	1134	1134
query58	250	238	252	238
query59	2846	2903	2711	2711
query60	286	274	286	274
query61	124	118	119	118
query62	737	754	667	667
query63	233	194	196	194
query64	1923	1032	742	742
query65	3298	3237	3278	3237
query66	869	400	304	304
query67	15809	15626	15569	15569
query68	5507	769	501	501
query69	517	296	259	259
query70	1165	1140	1140	1140
query71	458	344	270	270
query72	5932	3682	3833	3682
query73	1252	741	340	340
query74	9037	9119	8956	8956
query75	3199	3140	2708	2708
query76	3924	1183	744	744
query77	535	367	276	276
query78	10039	10124	9263	9263
query79	2003	787	588	588
query80	667	545	455	455
query81	505	329	246	246
query82	411	145	117	117
query83	181	176	154	154
query84	297	95	75	75
query85	741	355	305	305
query86	335	296	302	296
query87	4497	4497	4440	4440
query88	3191	2192	2182	2182
query89	443	314	277	277
query90	1814	189	191	189
query91	130	141	109	109
query92	76	56	58	56
query93	2627	1001	571	571
query94	675	416	303	303
query95	353	269	262	262
query96	490	555	268	268
query97	2774	2850	2704	2704
query98	230	209	203	203
query99	1324	1410	1252	1252
Total cold run time: 293150 ms
Total hot run time: 191195 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.5 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bb683f8f5233fcdccb49e7ebc36c4e10b76f5d83, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.10	0.10
query5	0.42	0.42	0.39
query6	1.18	0.66	0.65
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.60	0.51	0.52
query10	0.58	0.59	0.58
query11	0.16	0.10	0.10
query12	0.15	0.11	0.11
query13	0.63	0.60	0.60
query14	2.69	2.71	2.73
query15	0.91	0.86	0.83
query16	0.37	0.38	0.37
query17	1.01	1.02	1.02
query18	0.20	0.20	0.19
query19	1.88	1.80	1.96
query20	0.02	0.01	0.01
query21	15.35	0.94	0.56
query22	0.76	1.12	0.76
query23	14.93	1.34	0.58
query24	7.38	1.43	0.78
query25	0.50	0.18	0.07
query26	0.57	0.18	0.13
query27	0.06	0.05	0.04
query28	9.40	0.83	0.44
query29	12.60	3.92	3.29
query30	0.24	0.08	0.06
query31	2.83	0.58	0.39
query32	3.22	0.55	0.46
query33	2.96	3.03	3.01
query34	15.75	5.11	4.48
query35	4.53	4.50	4.54
query36	0.64	0.51	0.49
query37	0.10	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.17	0.12	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 105.03 s
Total hot run time: 30.5 s

@xzj7019
Copy link
Contributor Author

xzj7019 commented Feb 10, 2025

run p0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@englefly englefly englefly left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@xzj7019 @hello-stephen @doris-robot @englefly