Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](cloud-mow)Fix not release delete bitmap lock when checking txn state is visible #47580

Merged
merged 2 commits into from
Feb 8, 2025

Conversation

hust-hhb
Copy link
Contributor

@hust-hhb hust-hhb commented Feb 7, 2025

What problem does this PR solve?

Fe should check txn state before calculate delete bitmap, if txn state is visible, just retrurn ok to be, no need to recalculate delete bitmap again.This check operation should do before get delete bitmap lock, otherwise delete bitmap lock will not relase until time out.
Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Feb 7, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hust-hhb hust-hhb changed the title [fix](cloud-mow)Fix no release delete bitmap lock when checking txn state is visible [fix](cloud-mow)Fix not release delete bitmap lock when checking txn state is visible Feb 7, 2025
@hust-hhb
Copy link
Contributor Author

hust-hhb commented Feb 7, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31570 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 27c138975d1012f8bb5e9c0e3216e2727a33db47, data reload: false

------ Round 1 ----------------------------------
q1	17613	5306	5194	5194
q2	2061	308	181	181
q3	10385	1311	714	714
q4	10219	1038	551	551
q5	7515	2347	2408	2347
q6	201	169	136	136
q7	910	745	605	605
q8	9306	1288	1088	1088
q9	5086	4607	4762	4607
q10	6814	2316	1882	1882
q11	493	275	252	252
q12	347	363	226	226
q13	17775	3671	3067	3067
q14	250	229	217	217
q15	521	483	478	478
q16	646	601	576	576
q17	569	880	345	345
q18	6696	6365	6103	6103
q19	1215	953	561	561
q20	317	337	198	198
q21	2997	2185	1937	1937
q22	363	332	305	305
Total cold run time: 102299 ms
Total hot run time: 31570 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5148	5126	5188	5126
q2	230	329	235	235
q3	2144	2693	2302	2302
q4	1471	1881	1450	1450
q5	4238	4208	4153	4153
q6	205	164	124	124
q7	1874	1849	1719	1719
q8	2605	2621	2639	2621
q9	7173	7227	7271	7227
q10	2908	3209	2779	2779
q11	556	514	510	510
q12	681	797	648	648
q13	3436	3917	3251	3251
q14	282	307	282	282
q15	511	467	452	452
q16	657	704	645	645
q17	1150	1646	1357	1357
q18	7571	7369	7239	7239
q19	821	874	1036	874
q20	1953	2014	1886	1886
q21	5352	4922	4947	4922
q22	627	599	548	548
Total cold run time: 51593 ms
Total hot run time: 50350 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189462 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 27c138975d1012f8bb5e9c0e3216e2727a33db47, data reload: false

query1	1317	950	920	920
query2	6058	1820	1768	1768
query3	10960	4485	4432	4432
query4	53653	26324	23125	23125
query5	4879	611	481	481
query6	315	197	196	196
query7	4861	533	295	295
query8	297	257	238	238
query9	5445	2509	2522	2509
query10	431	345	275	275
query11	15090	15106	14905	14905
query12	164	115	109	109
query13	1065	540	374	374
query14	10131	6267	6156	6156
query15	214	205	185	185
query16	7153	665	465	465
query17	1101	708	554	554
query18	1745	418	318	318
query19	206	190	163	163
query20	128	120	123	120
query21	214	128	106	106
query22	4355	4443	4366	4366
query23	34097	33278	33438	33278
query24	6540	2451	2420	2420
query25	477	446	413	413
query26	704	272	155	155
query27	2704	508	342	342
query28	3343	2474	2428	2428
query29	583	571	434	434
query30	211	187	151	151
query31	892	870	823	823
query32	72	69	63	63
query33	431	364	319	319
query34	796	881	517	517
query35	780	828	743	743
query36	959	1036	895	895
query37	129	105	80	80
query38	4371	4196	4359	4196
query39	1481	1418	1464	1418
query40	224	119	108	108
query41	51	50	51	50
query42	129	105	112	105
query43	509	514	495	495
query44	1305	808	808	808
query45	179	172	166	166
query46	882	1098	664	664
query47	1858	1906	1851	1851
query48	392	421	309	309
query49	692	526	416	416
query50	745	756	431	431
query51	4273	4323	4306	4306
query52	105	110	97	97
query53	246	277	202	202
query54	502	527	403	403
query55	86	84	87	84
query56	279	279	264	264
query57	1177	1194	1073	1073
query58	240	226	233	226
query59	2603	2672	2600	2600
query60	281	271	271	271
query61	122	122	113	113
query62	685	708	650	650
query63	228	196	189	189
query64	1929	1123	763	763
query65	3183	3153	3137	3137
query66	781	392	316	316
query67	15893	15487	15442	15442
query68	4441	764	519	519
query69	515	309	275	275
query70	1192	1112	1119	1112
query71	441	291	266	266
query72	5836	3681	3785	3681
query73	822	742	348	348
query74	9279	9154	8867	8867
query75	3185	3171	2669	2669
query76	3470	1169	754	754
query77	524	374	280	280
query78	10154	10131	9267	9267
query79	2099	830	592	592
query80	1053	523	456	456
query81	535	269	238	238
query82	446	149	122	122
query83	215	176	158	158
query84	289	94	77	77
query85	768	369	310	310
query86	380	292	285	285
query87	4473	4582	4481	4481
query88	2851	2243	2170	2170
query89	387	318	290	290
query90	1561	197	193	193
query91	138	142	110	110
query92	64	60	55	55
query93	1325	1066	582	582
query94	682	411	310	310
query95	357	282	272	272
query96	486	560	266	266
query97	2817	2874	2726	2726
query98	225	211	205	205
query99	1349	1468	1252	1252
Total cold run time: 290290 ms
Total hot run time: 189462 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.54 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 27c138975d1012f8bb5e9c0e3216e2727a33db47, data reload: false

query1	0.03	0.02	0.03
query2	0.07	0.04	0.03
query3	0.23	0.06	0.07
query4	1.63	0.10	0.11
query5	0.40	0.41	0.41
query6	1.16	0.66	0.65
query7	0.02	0.01	0.02
query8	0.04	0.03	0.02
query9	0.60	0.53	0.52
query10	0.59	0.59	0.57
query11	0.16	0.10	0.10
query12	0.14	0.11	0.10
query13	0.62	0.59	0.60
query14	2.71	2.84	2.73
query15	0.92	0.86	0.86
query16	0.38	0.39	0.39
query17	1.00	1.02	1.02
query18	0.21	0.20	0.19
query19	1.91	1.76	1.98
query20	0.01	0.02	0.01
query21	15.36	0.93	0.56
query22	0.76	1.26	0.73
query23	14.80	1.38	0.64
query24	6.90	1.36	0.69
query25	0.48	0.09	0.22
query26	0.62	0.16	0.13
query27	0.06	0.05	0.04
query28	8.85	0.92	0.43
query29	12.60	3.92	3.30
query30	0.24	0.09	0.07
query31	2.83	0.59	0.40
query32	3.23	0.55	0.46
query33	3.08	3.07	3.08
query34	15.78	5.12	4.49
query35	4.54	4.50	4.56
query36	0.65	0.49	0.49
query37	0.09	0.06	0.06
query38	0.05	0.05	0.03
query39	0.03	0.03	0.02
query40	0.16	0.14	0.13
query41	0.08	0.04	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.09 s
Total hot run time: 30.54 s

@hust-hhb
Copy link
Contributor Author

hust-hhb commented Feb 7, 2025

run buildall

Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 7, 2025
Copy link
Contributor

github-actions bot commented Feb 7, 2025

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented Feb 7, 2025

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 31696 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 37cffc517635987077a650c58d2425a259be672c, data reload: false

------ Round 1 ----------------------------------
q1	17594	5181	5014	5014
q2	2056	306	165	165
q3	10405	1243	734	734
q4	10218	1011	541	541
q5	7520	2298	2328	2298
q6	186	168	136	136
q7	896	760	595	595
q8	9309	1292	1090	1090
q9	4948	4538	4707	4538
q10	6811	2300	1887	1887
q11	505	269	253	253
q12	345	356	223	223
q13	17791	3705	3343	3343
q14	239	227	215	215
q15	510	488	474	474
q16	621	624	593	593
q17	586	864	361	361
q18	7122	6606	6279	6279
q19	1210	933	553	553
q20	297	323	193	193
q21	2742	2270	1905	1905
q22	353	321	306	306
Total cold run time: 102264 ms
Total hot run time: 31696 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5139	5130	5105	5105
q2	242	322	229	229
q3	2149	2672	2248	2248
q4	1501	1820	1414	1414
q5	4226	4092	4144	4092
q6	203	166	124	124
q7	1888	1792	1676	1676
q8	2599	2509	2485	2485
q9	7195	7148	7199	7148
q10	3023	3205	2783	2783
q11	582	505	492	492
q12	695	770	632	632
q13	3429	4010	3291	3291
q14	298	300	258	258
q15	486	470	451	451
q16	675	692	648	648
q17	1108	1607	1303	1303
q18	7608	7448	7407	7407
q19	793	789	910	789
q20	1997	2007	1909	1909
q21	5327	5006	4763	4763
q22	606	562	547	547
Total cold run time: 51769 ms
Total hot run time: 49794 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190416 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 37cffc517635987077a650c58d2425a259be672c, data reload: false

query1	1288	954	944	944
query2	6229	1820	1775	1775
query3	10983	4472	4431	4431
query4	54176	24292	23489	23489
query5	5178	586	451	451
query6	329	183	179	179
query7	4890	510	292	292
query8	306	242	245	242
query9	5582	2492	2493	2492
query10	441	315	268	268
query11	15260	15008	14945	14945
query12	150	110	108	108
query13	1062	517	388	388
query14	10160	6331	6218	6218
query15	218	212	184	184
query16	7099	664	468	468
query17	1098	724	593	593
query18	1574	425	321	321
query19	201	199	177	177
query20	133	127	134	127
query21	212	132	106	106
query22	4382	4460	4576	4460
query23	33924	33611	33452	33452
query24	6894	2426	2459	2426
query25	481	468	396	396
query26	699	280	181	181
query27	2223	507	333	333
query28	3285	2479	2401	2401
query29	591	575	427	427
query30	216	190	159	159
query31	926	838	848	838
query32	75	66	69	66
query33	470	347	297	297
query34	750	866	482	482
query35	799	830	748	748
query36	962	986	898	898
query37	131	102	77	77
query38	4382	4376	4333	4333
query39	1507	1422	1449	1422
query40	217	121	100	100
query41	52	49	50	49
query42	116	103	110	103
query43	504	521	475	475
query44	1294	801	797	797
query45	177	174	170	170
query46	891	1067	659	659
query47	1853	1900	1826	1826
query48	398	426	303	303
query49	707	542	433	433
query50	710	753	424	424
query51	4273	4324	4237	4237
query52	113	111	100	100
query53	239	266	183	183
query54	471	508	401	401
query55	81	81	81	81
query56	252	277	254	254
query57	1150	1203	1145	1145
query58	248	245	243	243
query59	2744	2856	2920	2856
query60	292	286	266	266
query61	128	177	123	123
query62	756	791	682	682
query63	225	188	184	184
query64	1519	1023	752	752
query65	3386	3175	3126	3126
query66	780	388	298	298
query67	15931	15716	15372	15372
query68	5562	780	509	509
query69	545	295	256	256
query70	1181	1125	1109	1109
query71	437	298	277	277
query72	6034	3693	3807	3693
query73	1374	790	349	349
query74	9230	9167	8777	8777
query75	3179	3179	2687	2687
query76	3920	1172	743	743
query77	527	356	287	287
query78	10180	10241	9329	9329
query79	1593	829	581	581
query80	1104	540	451	451
query81	537	271	245	245
query82	459	152	120	120
query83	278	170	150	150
query84	284	96	79	79
query85	767	352	298	298
query86	395	290	282	282
query87	4440	4658	4480	4480
query88	2784	2195	2148	2148
query89	402	313	281	281
query90	1508	194	186	186
query91	134	138	107	107
query92	60	63	62	62
query93	1339	1029	576	576
query94	651	398	299	299
query95	345	266	257	257
query96	484	544	266	266
query97	2788	2865	2825	2825
query98	222	205	196	196
query99	1321	1419	1275	1275
Total cold run time: 292837 ms
Total hot run time: 190416 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.49 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 37cffc517635987077a650c58d2425a259be672c, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.03	0.03
query3	0.24	0.07	0.06
query4	1.62	0.10	0.10
query5	0.44	0.41	0.41
query6	1.14	0.67	0.66
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.53	0.52
query10	0.59	0.59	0.57
query11	0.15	0.10	0.11
query12	0.14	0.11	0.12
query13	0.60	0.60	0.59
query14	2.69	2.81	2.82
query15	0.91	0.85	0.86
query16	0.37	0.38	0.38
query17	1.02	1.04	1.02
query18	0.22	0.20	0.20
query19	2.00	1.75	2.04
query20	0.02	0.01	0.01
query21	15.36	0.94	0.55
query22	0.76	1.22	0.74
query23	14.79	1.38	0.65
query24	6.91	2.06	0.51
query25	0.52	0.21	0.15
query26	0.63	0.16	0.15
query27	0.05	0.05	0.04
query28	9.31	0.81	0.42
query29	12.58	3.99	3.29
query30	0.25	0.08	0.06
query31	2.83	0.59	0.38
query32	3.23	0.55	0.47
query33	3.00	3.00	3.06
query34	15.89	5.14	4.51
query35	4.52	4.51	4.55
query36	0.67	0.49	0.48
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.03
query40	0.17	0.14	0.13
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 104.7 s
Total hot run time: 30.49 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit acb6bd2 into apache:master Feb 8, 2025
26 of 28 checks passed
dataroaring pushed a commit that referenced this pull request Feb 8, 2025
…state is visible #47580 (#47652)

### What problem does this PR solve?
pick #47580
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.4-merged p0_b reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants