Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feat](binlog) Support getting binlogs in batch #47557

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[feat](binlog) Support getting binlogs in batch #47557

wants to merge 1 commit into from
+35 −14

Conversation

w41ter
Copy link
Contributor

@w41ter w41ter commented Feb 6, 2025 •
edited
Loading

What problem does this PR solve?

Issue Number: close #xxx

Related PR: selectdb/ccr-syncer#400

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@w41ter
Copy link
Contributor Author

w41ter commented Feb 6, 2025

run buildall

1 similar comment
@w41ter
Copy link
Contributor Author

w41ter commented Feb 6, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31788 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2240de6f6cdc097da7df6dd1bb6f4c4e3a85476b, data reload: false

------ Round 1 ----------------------------------
q1	17594	5297	5065	5065
q2	2059	313	175	175
q3	10385	1264	774	774
q4	10210	1040	528	528
q5	7525	2386	2412	2386
q6	190	169	138	138
q7	902	763	620	620
q8	9337	1301	1166	1166
q9	4924	4627	4702	4627
q10	6836	2344	1933	1933
q11	514	287	261	261
q12	352	371	216	216
q13	17770	3652	3073	3073
q14	233	228	210	210
q15	508	471	469	469
q16	636	624	570	570
q17	583	842	390	390
q18	6601	6272	6151	6151
q19	1902	945	564	564
q20	319	323	188	188
q21	2955	2270	1958	1958
q22	367	337	326	326
Total cold run time: 102702 ms
Total hot run time: 31788 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5192	5122	5162	5122
q2	236	326	231	231
q3	2198	2716	2334	2334
q4	1485	1886	1424	1424
q5	4278	4175	4195	4175
q6	210	164	130	130
q7	1865	1840	1718	1718
q8	2630	2615	2608	2608
q9	7241	7219	7146	7146
q10	3079	3229	2803	2803
q11	569	503	488	488
q12	676	795	640	640
q13	3418	3959	3284	3284
q14	276	293	280	280
q15	510	462	469	462
q16	673	691	644	644
q17	1180	1645	1320	1320
q18	7663	7397	7323	7323
q19	798	869	964	869
q20	1965	2043	1887	1887
q21	5473	4997	4756	4756
q22	655	583	552	552
Total cold run time: 52270 ms
Total hot run time: 50196 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190362 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2240de6f6cdc097da7df6dd1bb6f4c4e3a85476b, data reload: false

query1	1314	966	933	933
query2	6201	1868	1808	1808
query3	11005	4456	4492	4456
query4	53380	24966	23386	23386
query5	5141	576	480	480
query6	355	211	196	196
query7	4978	516	298	298
query8	320	251	240	240
query9	6188	2553	2560	2553
query10	428	321	282	282
query11	15296	15021	15017	15017
query12	159	110	107	107
query13	1090	542	374	374
query14	10346	6555	6859	6555
query15	197	202	190	190
query16	7013	670	495	495
query17	1055	725	573	573
query18	1529	417	332	332
query19	209	200	163	163
query20	121	125	136	125
query21	210	120	107	107
query22	4282	4399	4370	4370
query23	34053	33416	33260	33260
query24	5681	2427	2504	2427
query25	484	460	404	404
query26	677	275	157	157
query27	1766	487	333	333
query28	3067	2441	2402	2402
query29	590	575	440	440
query30	217	188	154	154
query31	906	892	792	792
query32	82	65	63	63
query33	428	350	308	308
query34	785	867	523	523
query35	807	850	769	769
query36	957	972	937	937
query37	130	100	76	76
query38	4392	4270	4219	4219
query39	1537	1443	1491	1443
query40	212	121	105	105
query41	68	51	48	48
query42	122	106	100	100
query43	484	521	478	478
query44	1341	827	809	809
query45	186	182	165	165
query46	903	1062	662	662
query47	1842	1832	1782	1782
query48	394	435	367	367
query49	697	523	447	447
query50	765	770	430	430
query51	4301	4292	4239	4239
query52	107	108	96	96
query53	239	280	213	213
query54	493	502	414	414
query55	84	80	82	80
query56	280	279	257	257
query57	1156	1172	1120	1120
query58	245	241	247	241
query59	2694	2897	2707	2707
query60	287	287	267	267
query61	121	120	118	118
query62	739	771	679	679
query63	237	205	199	199
query64	1501	1139	698	698
query65	3279	3245	3326	3245
query66	748	397	301	301
query67	15876	15655	15195	15195
query68	4431	819	515	515
query69	492	310	291	291
query70	1192	1149	1106	1106
query71	416	291	263	263
query72	6316	3772	3754	3754
query73	810	740	351	351
query74	9052	9014	8768	8768
query75	3179	3170	2715	2715
query76	3141	1189	756	756
query77	571	376	274	274
query78	10135	10290	9311	9311
query79	1676	815	584	584
query80	723	535	447	447
query81	495	291	239	239
query82	415	155	126	126
query83	177	172	153	153
query84	285	100	72	72
query85	742	346	307	307
query86	339	296	298	296
query87	4415	4645	4366	4366
query88	2895	2235	2193	2193
query89	423	318	280	280
query90	1641	196	193	193
query91	132	135	107	107
query92	70	62	110	62
query93	1352	1005	578	578
query94	632	425	293	293
query95	344	267	257	257
query96	476	568	264	264
query97	2764	2860	2753	2753
query98	239	220	203	203
query99	1323	1423	1276	1276
Total cold run time: 288070 ms
Total hot run time: 190362 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.2 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2240de6f6cdc097da7df6dd1bb6f4c4e3a85476b, data reload: false

query1	0.04	0.03	0.03
query2	0.09	0.05	0.04
query3	0.28	0.04	0.05
query4	1.61	0.07	0.08
query5	0.42	0.42	0.40
query6	1.17	0.65	0.67
query7	0.02	0.02	0.02
query8	0.05	0.05	0.04
query9	0.62	0.52	0.52
query10	0.58	0.59	0.58
query11	0.24	0.12	0.12
query12	0.24	0.12	0.12
query13	0.62	0.62	0.60
query14	2.82	2.72	2.81
query15	1.02	0.88	0.87
query16	0.36	0.37	0.39
query17	1.02	1.05	1.04
query18	0.18	0.18	0.18
query19	1.98	1.91	2.00
query20	0.02	0.01	0.01
query21	15.35	0.98	0.65
query22	0.92	1.08	0.79
query23	14.71	1.56	0.73
query24	5.41	0.57	0.28
query25	0.17	0.09	0.09
query26	0.55	0.21	0.17
query27	0.08	0.08	0.08
query28	10.99	1.15	0.55
query29	12.60	4.05	3.38
query30	0.28	0.07	0.06
query31	2.83	0.62	0.43
query32	3.23	0.58	0.49
query33	3.04	3.11	3.09
query34	16.62	5.10	4.48
query35	4.47	4.53	4.47
query36	0.63	0.51	0.49
query37	0.21	0.18	0.16
query38	0.17	0.16	0.15
query39	0.05	0.04	0.04
query40	0.20	0.16	0.15
query41	0.10	0.05	0.05
query42	0.07	0.05	0.05
query43	0.05	0.05	0.04
Total cold run time: 106.11 s
Total hot run time: 31.2 s

@w41ter w41ter force-pushed the batch_get_binlogs branch from 2240de6 to adf4fa4 Compare February 7, 2025 02:13
@w41ter
Copy link
Contributor Author

w41ter commented Feb 7, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31763 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit adf4fa41389ec3da8ea3d261d5388457016d14b2, data reload: false

------ Round 1 ----------------------------------
q1	17595	5258	5109	5109
q2	2052	306	172	172
q3	10424	1269	751	751
q4	10208	1002	561	561
q5	7543	2425	2388	2388
q6	194	166	133	133
q7	902	761	603	603
q8	9324	1301	1130	1130
q9	4961	4739	4647	4647
q10	6820	2332	1894	1894
q11	469	274	246	246
q12	356	356	222	222
q13	17772	3727	3086	3086
q14	239	227	211	211
q15	529	468	454	454
q16	621	628	577	577
q17	578	884	346	346
q18	6596	6291	6311	6291
q19	1593	964	554	554
q20	304	317	185	185
q21	2817	2160	1907	1907
q22	360	326	296	296
Total cold run time: 102257 ms
Total hot run time: 31763 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5185	5126	5178	5126
q2	245	325	229	229
q3	2236	2690	2362	2362
q4	1433	1856	1388	1388
q5	4269	4117	4110	4110
q6	210	169	122	122
q7	1848	1814	1716	1716
q8	2643	2609	2495	2495
q9	7259	7279	7126	7126
q10	3024	3254	2846	2846
q11	587	523	496	496
q12	687	779	630	630
q13	3483	3977	3263	3263
q14	274	303	274	274
q15	503	463	474	463
q16	629	659	631	631
q17	1172	1661	1317	1317
q18	7633	7502	7410	7410
q19	780	791	1036	791
q20	1961	2033	1871	1871
q21	5557	5045	4720	4720
q22	637	593	536	536
Total cold run time: 52255 ms
Total hot run time: 49922 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190131 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit adf4fa41389ec3da8ea3d261d5388457016d14b2, data reload: false

query1	1316	969	954	954
query2	6218	1883	1847	1847
query3	11069	4681	4525	4525
query4	54540	25352	23239	23239
query5	5286	557	525	525
query6	376	208	187	187
query7	5106	510	299	299
query8	324	255	240	240
query9	6676	2511	2530	2511
query10	432	309	294	294
query11	15577	14988	14884	14884
query12	164	110	109	109
query13	1175	528	384	384
query14	10702	6419	6332	6332
query15	221	209	186	186
query16	7068	672	460	460
query17	1085	732	588	588
query18	1523	429	324	324
query19	224	195	172	172
query20	126	123	127	123
query21	257	123	106	106
query22	4277	4539	4399	4399
query23	33883	33448	33369	33369
query24	5644	2444	2453	2444
query25	501	456	402	402
query26	720	275	157	157
query27	1753	467	346	346
query28	2899	2468	2408	2408
query29	596	553	434	434
query30	222	186	162	162
query31	895	853	807	807
query32	75	63	61	61
query33	445	366	299	299
query34	764	845	504	504
query35	817	850	728	728
query36	964	1011	927	927
query37	125	111	84	84
query38	4306	4307	4221	4221
query39	1471	1441	1433	1433
query40	207	124	105	105
query41	50	57	49	49
query42	124	109	117	109
query43	505	533	466	466
query44	1333	804	804	804
query45	176	174	163	163
query46	880	1076	659	659
query47	1855	1896	1814	1814
query48	387	423	312	312
query49	694	492	424	424
query50	731	757	419	419
query51	4289	4296	4240	4240
query52	110	106	95	95
query53	255	260	189	189
query54	486	486	423	423
query55	86	80	83	80
query56	258	295	275	275
query57	1163	1198	1119	1119
query58	251	238	259	238
query59	2750	2903	2733	2733
query60	288	291	263	263
query61	123	121	155	121
query62	767	730	676	676
query63	232	197	190	190
query64	1857	1055	692	692
query65	3282	3216	3148	3148
query66	779	394	294	294
query67	16160	15545	15221	15221
query68	5354	781	525	525
query69	526	354	266	266
query70	1211	1142	1102	1102
query71	434	302	281	281
query72	5978	3720	3788	3720
query73	1162	749	342	342
query74	8904	9159	9035	9035
query75	3210	3186	2752	2752
query76	3904	1171	750	750
query77	546	381	276	276
query78	10108	10059	9219	9219
query79	2581	802	603	603
query80	666	532	447	447
query81	499	272	238	238
query82	367	153	132	132
query83	170	177	155	155
query84	294	92	83	83
query85	758	411	309	309
query86	382	303	316	303
query87	4488	4599	4391	4391
query88	3402	2171	2166	2166
query89	406	317	278	278
query90	1763	190	186	186
query91	135	138	109	109
query92	81	62	56	56
query93	2449	1013	600	600
query94	666	396	307	307
query95	347	266	258	258
query96	473	559	266	266
query97	2806	2866	2735	2735
query98	237	221	200	200
query99	1357	1395	1246	1246
Total cold run time: 295564 ms
Total hot run time: 190131 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.18 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit adf4fa41389ec3da8ea3d261d5388457016d14b2, data reload: false

query1	0.03	0.03	0.03
query2	0.10	0.05	0.05
query3	0.27	0.05	0.05
query4	1.61	0.07	0.07
query5	0.41	0.40	0.40
query6	1.17	0.66	0.64
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.61	0.52	0.52
query10	0.57	0.59	0.58
query11	0.25	0.12	0.12
query12	0.25	0.13	0.13
query13	0.63	0.62	0.61
query14	2.68	2.83	2.79
query15	1.00	0.87	0.86
query16	0.36	0.38	0.37
query17	1.06	1.06	1.07
query18	0.18	0.17	0.20
query19	1.95	1.86	2.01
query20	0.02	0.01	0.01
query21	15.48	0.97	0.66
query22	0.92	1.07	0.80
query23	14.70	1.54	0.75
query24	5.38	0.60	0.29
query25	0.16	0.09	0.08
query26	0.57	0.23	0.18
query27	0.09	0.08	0.08
query28	11.01	1.14	0.54
query29	12.53	4.00	3.36
query30	0.27	0.08	0.05
query31	2.82	0.62	0.43
query32	3.23	0.58	0.49
query33	3.12	3.08	3.01
query34	16.52	5.20	4.47
query35	4.53	4.48	4.47
query36	0.63	0.50	0.50
query37	0.20	0.16	0.16
query38	0.17	0.16	0.14
query39	0.04	0.04	0.04
query40	0.19	0.16	0.16
query41	0.11	0.06	0.05
query42	0.07	0.05	0.05
query43	0.06	0.04	0.05
Total cold run time: 106.02 s
Total hot run time: 31.18 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@w41ter @hello-stephen @doris-robot