Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[enhance](mtmv)mv_infos not hold read lock of tables #47415

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

zddr
Copy link
Contributor

@zddr zddr commented Jan 24, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@zddr
Copy link
Contributor Author

zddr commented Jan 24, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32251 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b6aac2442572ac756fbb2862bf526eb1a9007db9, data reload: false

------ Round 1 ----------------------------------
q1	17587	5467	5359	5359
q2	2049	303	171	171
q3	10418	1311	751	751
q4	10223	961	530	530
q5	7536	2448	2181	2181
q6	195	171	133	133
q7	945	783	625	625
q8	9235	1345	1151	1151
q9	5154	4893	4838	4838
q10	6819	2349	1897	1897
q11	460	280	257	257
q12	360	354	224	224
q13	17779	3761	3137	3137
q14	230	238	215	215
q15	532	479	487	479
q16	638	614	589	589
q17	573	857	337	337
q18	6992	6327	6518	6327
q19	1207	958	540	540
q20	327	318	191	191
q21	2896	2267	2003	2003
q22	368	332	316	316
Total cold run time: 102523 ms
Total hot run time: 32251 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5610	5529	5540	5529
q2	245	337	231	231
q3	2228	2670	2327	2327
q4	1448	1861	1398	1398
q5	4354	4736	4655	4655
q6	163	159	124	124
q7	2054	1996	1831	1831
q8	2674	2838	2705	2705
q9	7208	7148	7078	7078
q10	3054	3290	2764	2764
q11	601	518	485	485
q12	686	778	636	636
q13	3546	3938	3288	3288
q14	306	292	279	279
q15	520	473	484	473
q16	640	681	638	638
q17	1215	1729	1263	1263
q18	7742	7535	7172	7172
q19	828	1194	1090	1090
q20	2000	2066	1882	1882
q21	5705	5276	4894	4894
q22	588	608	559	559
Total cold run time: 53415 ms
Total hot run time: 51301 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184630 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b6aac2442572ac756fbb2862bf526eb1a9007db9, data reload: false

query1	974	383	371	371
query2	6524	2049	1994	1994
query3	6797	216	209	209
query4	33284	23283	23008	23008
query5	4357	618	455	455
query6	291	207	186	186
query7	4610	499	305	305
query8	291	235	216	216
query9	9425	2584	2572	2572
query10	469	305	242	242
query11	17704	15286	15006	15006
query12	145	107	105	105
query13	1663	511	398	398
query14	10891	6923	6716	6716
query15	234	188	172	172
query16	8092	631	483	483
query17	1607	739	572	572
query18	2136	403	299	299
query19	225	186	157	157
query20	128	115	110	110
query21	208	128	106	106
query22	4194	4305	4224	4224
query23	33942	33171	33577	33171
query24	7132	2279	2349	2279
query25	473	477	408	408
query26	1058	248	212	212
query27	1980	473	337	337
query28	5424	2429	2392	2392
query29	535	524	423	423
query30	233	189	158	158
query31	954	864	749	749
query32	93	58	62	58
query33	522	353	332	332
query34	753	854	489	489
query35	797	808	726	726
query36	989	1036	961	961
query37	122	102	79	79
query38	4104	4146	4128	4128
query39	1448	1415	1381	1381
query40	201	120	99	99
query41	51	53	51	51
query42	122	104	105	104
query43	500	516	493	493
query44	1303	797	812	797
query45	175	168	170	168
query46	845	1040	645	645
query47	1825	1820	1765	1765
query48	387	398	313	313
query49	794	487	400	400
query50	621	665	403	403
query51	4196	4139	4170	4139
query52	113	111	93	93
query53	235	271	194	194
query54	485	495	423	423
query55	79	76	80	76
query56	276	257	253	253
query57	1113	1166	1085	1085
query58	250	258	240	240
query59	3003	3109	2846	2846
query60	276	278	264	264
query61	121	118	119	118
query62	807	730	652	652
query63	224	217	202	202
query64	4596	1006	651	651
query65	3228	3133	3173	3133
query66	1170	416	373	373
query67	15943	15420	15519	15420
query68	2101	831	542	542
query69	455	307	275	275
query70	1195	1117	1139	1117
query71	341	290	268	268
query72	5903	3828	3810	3810
query73	650	742	374	374
query74	10150	8819	8613	8613
query75	3123	3181	2673	2673
query76	2737	1150	790	790
query77	463	357	275	275
query78	10082	10108	9291	9291
query79	1948	828	600	600
query80	1598	537	445	445
query81	532	279	238	238
query82	355	160	125	125
query83	269	176	152	152
query84	237	103	82	82
query85	767	354	310	310
query86	387	325	300	300
query87	4462	4461	4366	4366
query88	3428	2201	2161	2161
query89	396	324	295	295
query90	1677	192	192	192
query91	131	139	117	117
query92	59	60	54	54
query93	999	881	537	537
query94	670	414	312	312
query95	334	265	259	259
query96	500	598	284	284
query97	2813	2885	2723	2723
query98	223	210	205	205
query99	1273	1370	1256	1256
Total cold run time: 278005 ms
Total hot run time: 184630 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.56 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b6aac2442572ac756fbb2862bf526eb1a9007db9, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.03
query3	0.25	0.07	0.06
query4	1.62	0.10	0.10
query5	0.40	0.41	0.39
query6	1.16	0.67	0.65
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.58	0.52	0.51
query10	0.55	0.56	0.55
query11	0.15	0.10	0.11
query12	0.14	0.10	0.11
query13	0.62	0.60	0.59
query14	2.70	2.76	2.80
query15	0.90	0.82	0.83
query16	0.39	0.39	0.37
query17	1.08	1.08	1.07
query18	0.22	0.22	0.20
query19	1.96	1.86	2.02
query20	0.01	0.01	0.01
query21	15.35	0.92	0.59
query22	0.76	0.71	0.68
query23	15.33	1.49	0.55
query24	3.49	1.61	2.63
query25	0.16	0.22	0.12
query26	0.18	0.14	0.14
query27	0.06	0.05	0.05
query28	14.73	0.94	0.43
query29	12.61	3.99	3.31
query30	0.25	0.10	0.07
query31	2.81	0.61	0.38
query32	3.22	0.54	0.46
query33	3.04	3.01	3.04
query34	16.60	5.20	4.54
query35	4.65	4.59	4.60
query36	0.64	0.52	0.50
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 107.29 s
Total hot run time: 31.56 s

@zddr
Copy link
Contributor Author

zddr commented Feb 7, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31196 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1f41a2500686344cbc60a79f16ab957172c0c0c7, data reload: false

------ Round 1 ----------------------------------
q1	17580	5343	5090	5090
q2	2041	311	165	165
q3	10408	1249	755	755
q4	10197	1034	537	537
q5	7537	2387	2335	2335
q6	191	167	130	130
q7	903	725	588	588
q8	9278	1285	1006	1006
q9	4850	4574	4519	4519
q10	6825	2325	1877	1877
q11	486	269	254	254
q12	348	364	215	215
q13	17775	3700	3065	3065
q14	221	232	208	208
q15	497	479	453	453
q16	630	634	572	572
q17	586	877	336	336
q18	6647	6161	6119	6119
q19	1204	955	558	558
q20	311	323	189	189
q21	2928	2126	1920	1920
q22	367	334	305	305
Total cold run time: 101810 ms
Total hot run time: 31196 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5161	5151	5083	5083
q2	234	323	243	243
q3	2124	2655	2263	2263
q4	1434	1829	1375	1375
q5	4265	4135	4159	4135
q6	205	169	125	125
q7	1887	1843	1669	1669
q8	2606	2687	2588	2588
q9	7383	7283	7275	7275
q10	3069	3266	2824	2824
q11	594	537	503	503
q12	757	792	616	616
q13	3582	4032	3320	3320
q14	276	287	276	276
q15	525	466	470	466
q16	658	659	649	649
q17	1149	1601	1354	1354
q18	7640	7552	7445	7445
q19	805	875	823	823
q20	1967	2028	1865	1865
q21	5402	4984	4722	4722
q22	582	568	526	526
Total cold run time: 52305 ms
Total hot run time: 50145 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189907 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1f41a2500686344cbc60a79f16ab957172c0c0c7, data reload: false

query1	1800	999	924	924
query2	6311	1880	1871	1871
query3	11166	4698	4579	4579
query4	27467	24060	23452	23452
query5	5768	662	494	494
query6	302	198	185	185
query7	3993	498	304	304
query8	305	253	226	226
query9	8505	2499	2486	2486
query10	529	326	272	272
query11	15251	15087	14875	14875
query12	163	109	107	107
query13	1705	530	379	379
query14	10621	6508	6741	6508
query15	227	201	173	173
query16	7783	697	486	486
query17	1558	779	599	599
query18	2084	435	336	336
query19	246	196	182	182
query20	135	122	117	117
query21	210	138	114	114
query22	4676	4394	4504	4394
query23	34189	33652	33200	33200
query24	7172	2408	2428	2408
query25	488	458	399	399
query26	803	275	151	151
query27	2007	507	327	327
query28	4274	2449	2412	2412
query29	635	553	432	432
query30	218	197	153	153
query31	910	858	816	816
query32	72	64	67	64
query33	552	355	296	296
query34	817	892	515	515
query35	827	832	742	742
query36	983	994	907	907
query37	127	105	84	84
query38	4316	4407	4345	4345
query39	1522	1475	1429	1429
query40	204	120	103	103
query41	57	54	50	50
query42	117	110	107	107
query43	506	516	491	491
query44	1301	796	776	776
query45	183	168	163	163
query46	901	1078	645	645
query47	1906	1935	1798	1798
query48	383	414	317	317
query49	715	527	413	413
query50	732	777	426	426
query51	4342	4285	4214	4214
query52	107	104	95	95
query53	227	257	191	191
query54	506	507	411	411
query55	82	80	82	80
query56	285	273	276	273
query57	1174	1181	1103	1103
query58	257	250	240	240
query59	2712	2844	2726	2726
query60	303	274	265	265
query61	117	118	120	118
query62	828	761	650	650
query63	230	189	197	189
query64	3716	1034	688	688
query65	3295	3222	3262	3222
query66	800	393	296	296
query67	15968	15534	15219	15219
query68	7030	784	509	509
query69	482	298	262	262
query70	1164	1101	1118	1101
query71	411	336	267	267
query72	5814	3499	3817	3499
query73	743	741	355	355
query74	8923	9256	8901	8901
query75	3186	3142	2666	2666
query76	3049	1169	758	758
query77	477	374	280	280
query78	9981	9947	9295	9295
query79	1998	828	592	592
query80	701	532	444	444
query81	527	273	284	273
query82	474	156	122	122
query83	173	166	143	143
query84	244	92	73	73
query85	733	362	294	294
query86	332	321	294	294
query87	4481	4504	4392	4392
query88	2840	2187	2165	2165
query89	388	312	283	283
query90	2082	192	191	191
query91	131	133	108	108
query92	80	59	54	54
query93	2032	1042	571	571
query94	669	384	301	301
query95	344	267	253	253
query96	487	565	268	268
query97	2777	2834	2744	2744
query98	219	220	191	191
query99	1306	1428	1246	1246
Total cold run time: 276949 ms
Total hot run time: 189907 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.93 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1f41a2500686344cbc60a79f16ab957172c0c0c7, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.62	0.11	0.10
query5	0.40	0.42	0.40
query6	1.16	0.64	0.65
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.61	0.52	0.51
query10	0.59	0.57	0.59
query11	0.16	0.11	0.11
query12	0.14	0.12	0.11
query13	0.60	0.60	0.61
query14	2.73	2.72	2.83
query15	0.93	0.85	0.86
query16	0.39	0.37	0.37
query17	1.01	1.05	1.02
query18	0.21	0.20	0.19
query19	1.89	1.81	1.92
query20	0.02	0.01	0.01
query21	15.46	0.90	0.55
query22	0.77	1.22	0.77
query23	14.76	1.37	0.64
query24	7.17	1.85	1.08
query25	0.51	0.17	0.08
query26	0.68	0.16	0.14
query27	0.05	0.05	0.05
query28	9.59	0.88	0.44
query29	12.54	3.95	3.32
query30	0.25	0.08	0.06
query31	2.84	0.61	0.38
query32	3.23	0.54	0.47
query33	2.99	2.99	3.05
query34	15.74	5.14	4.51
query35	4.50	4.47	4.49
query36	0.67	0.49	0.51
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.18	0.13	0.14
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.12 s
Total hot run time: 30.93 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants