Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[chore](binlog) Log download binlog cost #47360

Merged
merged 1 commit into from
Jan 24, 2025

Conversation

w41ter
Copy link
Contributor

@w41ter w41ter commented Jan 23, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@w41ter
Copy link
Contributor Author

w41ter commented Jan 23, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32049 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d9cb6b9db652f7f7c9255e2ba1151f325bec97b3, data reload: false

------ Round 1 ----------------------------------
q1	17571	5483	5376	5376
q2	2053	307	187	187
q3	10522	1214	715	715
q4	10227	976	534	534
q5	7650	2375	2126	2126
q6	203	168	135	135
q7	886	753	610	610
q8	9233	1356	1150	1150
q9	5192	4828	4852	4828
q10	6875	2349	1886	1886
q11	493	279	253	253
q12	344	361	221	221
q13	17774	3754	3065	3065
q14	230	221	206	206
q15	509	459	485	459
q16	629	606	599	599
q17	560	855	325	325
q18	6881	6531	6299	6299
q19	1727	953	554	554
q20	317	335	189	189
q21	2769	2230	2026	2026
q22	370	333	306	306
Total cold run time: 103015 ms
Total hot run time: 32049 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5556	5525	5827	5525
q2	241	333	235	235
q3	2218	2645	2433	2433
q4	1539	1903	1416	1416
q5	4427	4823	4732	4732
q6	170	159	125	125
q7	2128	1990	1873	1873
q8	2677	2865	2751	2751
q9	7475	7307	7353	7307
q10	3091	3327	2823	2823
q11	568	497	485	485
q12	633	738	579	579
q13	3535	3972	3348	3348
q14	299	321	275	275
q15	529	471	498	471
q16	660	690	654	654
q17	1257	1733	1271	1271
q18	7685	7599	7396	7396
q19	821	1154	1109	1109
q20	1999	2047	1910	1910
q21	5866	5193	5144	5144
q22	619	595	583	583
Total cold run time: 53993 ms
Total hot run time: 52445 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192864 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d9cb6b9db652f7f7c9255e2ba1151f325bec97b3, data reload: false

query1	1339	961	946	946
query2	6197	2040	1991	1991
query3	11070	4635	4491	4491
query4	60942	28461	23839	23839
query5	5482	598	467	467
query6	416	190	198	190
query7	5472	525	300	300
query8	333	254	237	237
query9	8053	2685	2654	2654
query10	450	302	259	259
query11	17383	15203	15691	15203
query12	166	110	110	110
query13	1349	561	436	436
query14	12400	7494	7057	7057
query15	213	195	187	187
query16	7277	701	479	479
query17	1123	755	602	602
query18	1958	414	332	332
query19	210	186	168	168
query20	133	114	113	113
query21	213	124	107	107
query22	4393	4771	4374	4374
query23	34353	33719	33474	33474
query24	5715	2302	2408	2302
query25	473	463	420	420
query26	644	276	156	156
query27	1708	496	343	343
query28	4124	2487	2450	2450
query29	551	584	416	416
query30	226	191	159	159
query31	917	908	807	807
query32	66	58	55	55
query33	426	355	313	313
query34	779	878	515	515
query35	809	870	769	769
query36	1005	1099	970	970
query37	127	104	77	77
query38	4355	4326	4243	4243
query39	1494	1425	1494	1425
query40	211	118	104	104
query41	50	49	50	49
query42	125	103	108	103
query43	532	542	504	504
query44	1350	820	848	820
query45	185	178	168	168
query46	925	1082	691	691
query47	1854	1875	1832	1832
query48	407	425	332	332
query49	717	500	406	406
query50	657	661	404	404
query51	4314	4305	4236	4236
query52	104	104	99	99
query53	240	287	197	197
query54	495	535	421	421
query55	91	80	102	80
query56	266	263	267	263
query57	1198	1228	1123	1123
query58	245	229	243	229
query59	3134	3286	3018	3018
query60	292	293	273	273
query61	118	113	110	110
query62	736	703	642	642
query63	234	200	202	200
query64	1299	1089	658	658
query65	3293	3136	3183	3136
query66	667	383	301	301
query67	16014	15565	15701	15565
query68	5358	855	544	544
query69	490	294	272	272
query70	1232	1170	1126	1126
query71	452	313	265	265
query72	6025	3868	3873	3868
query73	783	767	364	364
query74	9961	8995	9107	8995
query75	3243	3214	2666	2666
query76	3793	1180	795	795
query77	469	393	276	276
query78	10022	10127	9379	9379
query79	2382	846	610	610
query80	624	530	449	449
query81	484	354	237	237
query82	228	147	119	119
query83	177	185	149	149
query84	292	91	71	71
query85	759	344	296	296
query86	369	315	296	296
query87	4344	4514	4325	4325
query88	4243	2193	2145	2145
query89	463	334	291	291
query90	1686	191	192	191
query91	131	132	106	106
query92	60	58	55	55
query93	2206	887	543	543
query94	709	414	295	295
query95	333	265	260	260
query96	490	629	275	275
query97	2807	2846	2746	2746
query98	226	195	197	195
query99	1281	1396	1260	1260
Total cold run time: 310219 ms
Total hot run time: 192864 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.86 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d9cb6b9db652f7f7c9255e2ba1151f325bec97b3, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.61	0.11	0.10
query5	0.43	0.42	0.40
query6	1.16	0.65	0.66
query7	0.03	0.02	0.02
query8	0.04	0.03	0.04
query9	0.58	0.50	0.49
query10	0.56	0.57	0.55
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.62	0.60	0.60
query14	2.82	2.84	2.74
query15	0.90	0.83	0.83
query16	0.38	0.39	0.37
query17	1.04	1.06	1.07
query18	0.23	0.21	0.21
query19	1.91	1.88	2.00
query20	0.01	0.02	0.01
query21	15.36	0.96	0.58
query22	0.76	0.84	0.73
query23	15.21	1.43	0.62
query24	2.89	0.69	1.70
query25	0.14	0.22	0.16
query26	0.22	0.15	0.15
query27	0.06	0.05	0.05
query28	14.08	1.07	0.45
query29	12.58	3.99	3.29
query30	0.24	0.09	0.06
query31	2.85	0.62	0.41
query32	3.23	0.55	0.47
query33	3.08	3.03	3.12
query34	16.38	5.28	4.58
query35	4.61	4.60	4.51
query36	0.64	0.50	0.50
query37	0.10	0.07	0.05
query38	0.05	0.04	0.03
query39	0.04	0.02	0.03
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.77 s
Total hot run time: 30.86 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 41.95% (10938/26074)
Line Coverage: 32.21% (92327/286600)
Region Coverage: 31.33% (47280/150894)
Branch Coverage: 27.39% (23934/87394)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d9cb6b9db652f7f7c9255e2ba1151f325bec97b3_d9cb6b9db652f7f7c9255e2ba1151f325bec97b3/report/index.html

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 24, 2025
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@w41ter w41ter merged commit 88ceb24 into apache:master Jan 24, 2025
27 of 29 checks passed
@w41ter w41ter deleted the stat_ingest_binlog_cost branch January 24, 2025 06:13
dataroaring pushed a commit that referenced this pull request Jan 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.x dev/2.1.x-conflict dev/3.0.4-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants