Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

branch-3.0: [fix](restore) Add a local snapshot lock to protect snapshot dir #47279 #47292

Open
wants to merge 1 commit into
base: branch-3.0
Choose a base branch
from

Conversation

github-actions[bot]
Copy link
Contributor

Cherry-picked from #47279

@github-actions github-actions bot requested a review from dataroaring as a code owner January 21, 2025 13:19
@Thearas
Copy link
Contributor

Thearas commented Jan 21, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Jan 21, 2025
@Thearas
Copy link
Contributor

Thearas commented Jan 21, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40943 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 62ce0fe663087a73a5e8e171c2a3a3391d65c824, data reload: false

------ Round 1 ----------------------------------
q1	17572	7436	7321	7321
q2	2075	172	161	161
q3	10753	1041	1164	1041
q4	10560	747	748	747
q5	7733	2851	2819	2819
q6	236	146	142	142
q7	977	615	608	608
q8	9588	1998	2059	1998
q9	6589	6391	6397	6391
q10	7010	2290	2338	2290
q11	469	255	255	255
q12	403	218	213	213
q13	17790	2990	3014	2990
q14	243	213	212	212
q15	559	526	531	526
q16	703	606	625	606
q17	971	576	600	576
q18	7284	6607	6715	6607
q19	1386	1052	1097	1052
q20	485	204	204	204
q21	3971	3228	3196	3196
q22	1088	1007	988	988
Total cold run time: 108445 ms
Total hot run time: 40943 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7248	7239	7236	7236
q2	327	247	237	237
q3	2872	2995	2922	2922
q4	2041	1854	1747	1747
q5	5737	5731	5741	5731
q6	230	139	146	139
q7	2273	1800	1826	1800
q8	3331	3581	3540	3540
q9	8820	8887	8860	8860
q10	3582	3534	3562	3534
q11	603	518	502	502
q12	760	570	605	570
q13	8853	3198	3211	3198
q14	304	268	273	268
q15	573	533	533	533
q16	709	678	649	649
q17	1861	1616	1559	1559
q18	8256	7784	7632	7632
q19	1645	1408	1643	1408
q20	2127	1894	1881	1881
q21	5534	5420	5256	5256
q22	1169	1076	1027	1027
Total cold run time: 68855 ms
Total hot run time: 60229 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197546 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 62ce0fe663087a73a5e8e171c2a3a3391d65c824, data reload: false

query1	1341	921	925	921
query2	6213	2103	2121	2103
query3	10822	4381	4212	4212
query4	66325	28849	23430	23430
query5	4899	456	463	456
query6	432	183	175	175
query7	5655	318	310	310
query8	314	229	225	225
query9	9340	2689	2681	2681
query10	458	270	253	253
query11	17471	15158	15664	15158
query12	166	102	102	102
query13	1531	436	425	425
query14	10642	7687	7511	7511
query15	207	183	181	181
query16	7114	504	522	504
query17	1064	604	596	596
query18	1934	316	331	316
query19	221	156	161	156
query20	118	110	113	110
query21	217	107	108	107
query22	4900	4525	4403	4403
query23	34865	34139	34244	34139
query24	6168	2964	2904	2904
query25	523	420	440	420
query26	658	178	173	173
query27	1859	364	374	364
query28	4365	2487	2447	2447
query29	684	450	424	424
query30	242	166	167	166
query31	995	828	849	828
query32	61	55	62	55
query33	441	290	273	273
query34	918	519	523	519
query35	835	750	731	731
query36	1087	975	978	975
query37	136	76	74	74
query38	4215	4033	4066	4033
query39	1541	1457	1469	1457
query40	206	97	100	97
query41	47	47	51	47
query42	112	102	100	100
query43	558	502	514	502
query44	1190	814	838	814
query45	187	172	174	172
query46	1161	710	740	710
query47	1987	1897	1896	1896
query48	490	383	388	383
query49	730	393	417	393
query50	857	432	427	427
query51	7400	7272	7165	7165
query52	98	86	85	85
query53	266	179	176	176
query54	554	444	447	444
query55	78	78	77	77
query56	258	246	244	244
query57	1204	1088	1088	1088
query58	209	218	215	215
query59	3140	3043	2964	2964
query60	277	252	249	249
query61	134	103	103	103
query62	793	657	666	657
query63	211	194	189	189
query64	1386	650	643	643
query65	3252	3174	3180	3174
query66	716	310	306	306
query67	15846	15510	15632	15510
query68	3953	599	578	578
query69	441	266	263	263
query70	1198	1134	1149	1134
query71	363	255	254	254
query72	6363	4006	4053	4006
query73	756	344	356	344
query74	10131	9265	9024	9024
query75	3367	2597	2669	2597
query76	1972	977	1140	977
query77	499	277	266	266
query78	10578	9566	9587	9566
query79	1658	572	600	572
query80	913	433	414	414
query81	513	253	235	235
query82	1259	132	113	113
query83	248	142	147	142
query84	286	89	77	77
query85	908	300	292	292
query86	342	303	254	254
query87	4535	4268	4358	4268
query88	3744	2414	2352	2352
query89	412	289	285	285
query90	2041	192	188	188
query91	200	146	146	146
query92	70	48	48	48
query93	1906	533	541	533
query94	881	307	293	293
query95	376	262	253	253
query96	620	277	288	277
query97	3335	3162	3181	3162
query98	214	203	198	198
query99	1590	1304	1288	1288
Total cold run time: 319996 ms
Total hot run time: 197546 ms

)

To avoid concurrent modification of a snapshot dir.
@w41ter w41ter force-pushed the auto-pick-47279-branch-3.0 branch from 62ce0fe to 2168a44 Compare February 7, 2025 03:52
@w41ter
Copy link
Contributor

w41ter commented Feb 7, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41034 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2168a446d316a41cb3a3fa8c948b77093f4d760c, data reload: false

------ Round 1 ----------------------------------
q1	17556	7403	7302	7302
q2	2049	169	167	167
q3	10597	1074	1176	1074
q4	10591	743	733	733
q5	7751	2981	2895	2895
q6	246	149	147	147
q7	965	615	610	610
q8	9362	2018	2081	2018
q9	6690	6478	6445	6445
q10	7011	2302	2323	2302
q11	464	277	270	270
q12	399	212	213	212
q13	17789	2978	3003	2978
q14	252	215	207	207
q15	560	543	524	524
q16	655	581	597	581
q17	991	585	534	534
q18	7406	6619	6620	6619
q19	1386	1119	1025	1025
q20	494	203	199	199
q21	4259	3202	3284	3202
q22	1114	1037	990	990
Total cold run time: 108587 ms
Total hot run time: 41034 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7290	7274	7288	7274
q2	339	230	235	230
q3	2952	2941	2979	2941
q4	2098	1879	1825	1825
q5	5735	5766	5801	5766
q6	234	147	153	147
q7	2207	1807	1820	1807
q8	3395	3607	3581	3581
q9	8997	8950	8870	8870
q10	3650	3615	3618	3615
q11	626	494	494	494
q12	812	603	625	603
q13	10047	3193	3221	3193
q14	335	273	276	273
q15	575	525	539	525
q16	698	649	638	638
q17	1898	1611	1620	1611
q18	8325	7802	7512	7512
q19	1673	1547	1629	1547
q20	2111	1847	1906	1847
q21	5659	5468	5320	5320
q22	1139	1089	1046	1046
Total cold run time: 70795 ms
Total hot run time: 60665 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 198585 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2168a446d316a41cb3a3fa8c948b77093f4d760c, data reload: false

query1	1293	958	938	938
query2	6247	2055	2005	2005
query3	10909	4532	4341	4341
query4	66821	29350	23965	23965
query5	5037	446	440	440
query6	416	180	177	177
query7	5722	304	313	304
query8	316	226	221	221
query9	9469	2722	2724	2722
query10	471	268	266	266
query11	17622	15412	16013	15412
query12	162	103	104	103
query13	1565	451	439	439
query14	9704	7821	7148	7148
query15	210	185	175	175
query16	7093	513	491	491
query17	1082	571	566	566
query18	1821	310	315	310
query19	232	165	154	154
query20	117	109	116	109
query21	211	111	108	108
query22	4971	4418	4468	4418
query23	34505	34050	34522	34050
query24	6209	2926	2851	2851
query25	539	450	430	430
query26	669	170	177	170
query27	1960	355	352	352
query28	4633	2528	2498	2498
query29	723	493	462	462
query30	251	169	166	166
query31	988	824	850	824
query32	69	58	56	56
query33	477	310	295	295
query34	931	510	524	510
query35	913	740	711	711
query36	1081	967	977	967
query37	126	77	79	77
query38	4127	3993	3920	3920
query39	1697	1483	1474	1474
query40	214	107	106	106
query41	50	52	52	52
query42	118	103	112	103
query43	533	493	508	493
query44	1225	849	839	839
query45	189	179	174	174
query46	1151	774	718	718
query47	2075	1944	1955	1944
query48	477	376	386	376
query49	771	377	396	377
query50	862	427	442	427
query51	7414	7107	7087	7087
query52	108	90	87	87
query53	259	182	178	178
query54	556	464	438	438
query55	78	79	78	78
query56	268	241	235	235
query57	1252	1124	1092	1092
query58	222	211	209	209
query59	3119	3102	2959	2959
query60	279	254	256	254
query61	111	111	151	111
query62	853	740	725	725
query63	220	193	190	190
query64	1354	707	642	642
query65	3286	3228	3190	3190
query66	719	337	303	303
query67	16075	15672	15810	15672
query68	4092	583	555	555
query69	421	267	265	265
query70	1166	1120	1127	1120
query71	356	260	253	253
query72	6455	4030	4035	4030
query73	765	347	351	347
query74	10465	9042	9391	9042
query75	3391	2652	2697	2652
query76	1909	1090	1135	1090
query77	504	279	281	279
query78	10611	9678	9617	9617
query79	2005	600	610	600
query80	1377	421	426	421
query81	541	241	236	236
query82	1182	121	118	118
query83	266	142	156	142
query84	287	77	82	77
query85	1040	314	293	293
query86	414	271	294	271
query87	4565	4450	4313	4313
query88	3820	2367	2348	2348
query89	424	294	293	293
query90	1961	185	188	185
query91	187	151	149	149
query92	67	50	50	50
query93	2797	557	549	549
query94	884	304	304	304
query95	356	258	256	256
query96	623	278	287	278
query97	3321	3158	3175	3158
query98	215	199	198	198
query99	1696	1436	1412	1412
Total cold run time: 323390 ms
Total hot run time: 198585 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.1 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2168a446d316a41cb3a3fa8c948b77093f4d760c, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.06
query4	1.62	0.11	0.10
query5	0.52	0.52	0.51
query6	1.13	0.73	0.73
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.56	0.52	0.49
query10	0.56	0.56	0.55
query11	0.15	0.11	0.10
query12	0.17	0.12	0.11
query13	0.61	0.60	0.60
query14	2.73	2.75	2.76
query15	0.90	0.83	0.82
query16	0.40	0.38	0.38
query17	1.05	0.96	1.02
query18	0.23	0.22	0.22
query19	1.87	1.80	1.91
query20	0.02	0.01	0.01
query21	15.36	0.59	0.57
query22	2.81	2.15	1.74
query23	16.77	1.07	0.82
query24	3.10	1.28	2.21
query25	0.27	0.15	0.22
query26	0.41	0.14	0.14
query27	0.04	0.04	0.04
query28	9.67	1.10	1.07
query29	12.55	3.22	3.19
query30	0.26	0.06	0.06
query31	2.86	0.39	0.39
query32	3.23	0.47	0.46
query33	3.02	3.01	3.01
query34	16.96	4.56	4.55
query35	4.65	4.56	4.58
query36	0.65	0.47	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.03 s
Total hot run time: 33.1 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants