Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

branch-3.0: [fix](group commit) fix wal reader handle empty block #48290 #48333

Merged
merged 1 commit into from
Feb 26, 2025

Conversation

github-actions[bot]
Copy link
Contributor

Cherry-picked from #48290

### What problem does this PR solve?

Problem Summary:

If kill BE, wal writer may write block length and does not write block
data. Then when replay wal, will get:
```
failed to replay wal...st=[INTERNAL_ERROR]cur path: . failed to deserialize row
```
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Feb 25, 2025
@hello-stephen
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39917 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d6cab4529ad100a43b2e5c3264fb6d2f252995df, data reload: false

------ Round 1 ----------------------------------
q1	17592	6778	6585	6585
q2	2071	162	170	162
q3	10627	1093	1159	1093
q4	10564	755	780	755
q5	7741	2795	2840	2795
q6	221	134	132	132
q7	982	606	605	605
q8	9344	1925	2000	1925
q9	6578	6390	6357	6357
q10	7021	2260	2297	2260
q11	462	265	251	251
q12	394	211	210	210
q13	17794	2952	3012	2952
q14	242	216	200	200
q15	542	466	465	465
q16	663	581	581	581
q17	970	584	485	485
q18	7159	6764	6682	6682
q19	1391	1011	1123	1011
q20	490	212	207	207
q21	4034	3231	3339	3231
q22	1065	973	979	973
Total cold run time: 107947 ms
Total hot run time: 39917 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6592	6603	6644	6603
q2	332	233	229	229
q3	2889	2771	2895	2771
q4	2049	1846	1795	1795
q5	5821	5749	5766	5749
q6	209	129	127	127
q7	2244	1800	1855	1800
q8	3385	3546	3497	3497
q9	8815	8857	8892	8857
q10	3572	3524	3470	3470
q11	588	491	503	491
q12	820	602	611	602
q13	9636	3197	3173	3173
q14	304	266	281	266
q15	494	474	470	470
q16	712	661	647	647
q17	1844	1613	1618	1613
q18	8182	7822	7877	7822
q19	1662	1552	1464	1464
q20	2023	1861	1852	1852
q21	5423	5417	5416	5416
q22	1101	1047	1057	1047
Total cold run time: 68697 ms
Total hot run time: 59761 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197083 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d6cab4529ad100a43b2e5c3264fb6d2f252995df, data reload: false

query1	1308	904	892	892
query2	6305	1974	2012	1974
query3	10821	4265	4152	4152
query4	61357	28615	23358	23358
query5	5214	452	443	443
query6	399	176	169	169
query7	5476	305	301	301
query8	297	221	224	221
query9	8594	2589	2546	2546
query10	456	271	261	261
query11	17676	15484	15714	15484
query12	168	104	106	104
query13	1459	436	438	436
query14	10059	7415	6950	6950
query15	199	179	177	177
query16	7099	503	515	503
query17	1136	582	585	582
query18	1888	334	332	332
query19	220	166	158	158
query20	116	109	110	109
query21	208	103	108	103
query22	4680	4312	4745	4312
query23	34496	34209	34206	34206
query24	6302	2978	2937	2937
query25	551	415	446	415
query26	672	175	171	171
query27	1901	346	354	346
query28	4183	2435	2442	2435
query29	716	484	420	420
query30	244	154	160	154
query31	974	815	855	815
query32	66	55	56	55
query33	425	287	271	271
query34	944	502	517	502
query35	844	744	784	744
query36	1090	978	963	963
query37	121	70	64	64
query38	4104	4023	4067	4023
query39	1529	1491	1485	1485
query40	204	98	98	98
query41	50	55	52	52
query42	111	100	101	100
query43	527	482	485	482
query44	1204	822	820	820
query45	188	168	168	168
query46	1145	729	746	729
query47	2033	1952	1937	1937
query48	481	380	394	380
query49	734	392	391	391
query50	841	448	420	420
query51	7319	7375	7190	7190
query52	107	89	93	89
query53	265	182	184	182
query54	590	484	476	476
query55	80	80	78	78
query56	274	263	236	236
query57	1312	1215	1157	1157
query58	223	212	224	212
query59	3049	2970	2883	2883
query60	279	246	247	246
query61	107	109	105	105
query62	762	678	671	671
query63	215	187	181	181
query64	1435	665	638	638
query65	3278	3219	3199	3199
query66	699	303	311	303
query67	15916	15900	15753	15753
query68	4222	583	558	558
query69	424	263	261	261
query70	1189	1134	1085	1085
query71	348	259	252	252
query72	6400	4144	4080	4080
query73	806	353	346	346
query74	10381	8877	9179	8877
query75	3382	2648	2650	2648
query76	1922	948	1058	948
query77	482	277	264	264
query78	10696	9622	9483	9483
query79	2102	599	590	590
query80	1351	431	426	426
query81	526	237	237	237
query82	1264	92	85	85
query83	164	150	147	147
query84	279	82	75	75
query85	982	306	293	293
query86	363	306	281	281
query87	4473	4413	4343	4343
query88	4105	2383	2367	2367
query89	418	293	287	287
query90	1970	186	190	186
query91	180	148	148	148
query92	60	47	52	47
query93	2990	545	550	545
query94	774	298	282	282
query95	359	259	254	254
query96	635	284	279	279
query97	3342	3165	3208	3165
query98	212	201	201	201
query99	1602	1294	1327	1294
Total cold run time: 316128 ms
Total hot run time: 197083 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.13 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d6cab4529ad100a43b2e5c3264fb6d2f252995df, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.62	0.10	0.10
query5	0.53	0.53	0.51
query6	1.14	0.73	0.74
query7	0.02	0.02	0.02
query8	0.04	0.05	0.03
query9	0.56	0.50	0.50
query10	0.54	0.56	0.56
query11	0.15	0.10	0.10
query12	0.14	0.12	0.11
query13	0.61	0.60	0.60
query14	2.73	2.75	2.85
query15	0.91	0.82	0.82
query16	0.39	0.38	0.36
query17	1.07	1.06	1.05
query18	0.23	0.22	0.22
query19	1.83	1.76	2.01
query20	0.02	0.01	0.01
query21	15.35	0.59	0.57
query22	2.51	2.78	2.05
query23	17.04	1.11	0.81
query24	3.56	1.91	0.69
query25	0.17	0.07	0.10
query26	0.56	0.14	0.14
query27	0.04	0.05	0.05
query28	9.47	0.53	0.55
query29	12.56	3.25	3.22
query30	0.25	0.06	0.06
query31	2.86	0.38	0.38
query32	3.28	0.47	0.46
query33	2.98	3.03	3.00
query34	16.88	4.50	4.45
query35	4.54	4.53	4.52
query36	0.70	0.48	0.48
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 106.09 s
Total hot run time: 32.13 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit f00c103 into branch-3.0 Feb 26, 2025
22 of 24 checks passed
@github-actions github-actions bot deleted the auto-pick-48290-branch-3.0 branch February 26, 2025 01:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants