Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[opt](kerberos) use ticket cache instead of principal+keytab on BE side #47299

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

[opt](kerberos) use ticket cache instead of principal+keytab on BE side #47299

wants to merge 6 commits into from

Conversation

morningman
Copy link
Contributor

@morningman morningman commented Jan 21, 2025 •
edited
Loading

What problem does this PR solve?

Overview

Previously, BE node use principal and keytab to do the kerberos authentication.
But only the modified hadoop libhdfs support authenticating in this way, the origin libhdfs
only support setting kerberos ticket cache path, or use system level kerberos authentication context.

This pull request introduces a comprehensive Kerberos authentication module for the BE.
The module is designed to handle Kerberos ticket management, including initialization, authentication, and periodic ticket refresh.
It provides a robust interface for integrating Kerberos authentication, ensuring secure and efficient credential management.

Key Components

  1. KerberosConfig (kerberos_config.h and kerberos_config.cpp):

    • This class encapsulates the configuration settings required for Kerberos authentication, such as principal, keytab path, and refresh intervals.
    • Provides methods to set and retrieve configuration parameters.

  2. KerberosTicketCache (kerberos_ticket_cache.h and kerberos_ticket_cache.cpp):

    • Manages the Kerberos ticket cache, including initialization, login, and periodic refresh of tickets.
    • Supports operations like writing to the ticket cache and checking if a refresh is needed.
    • Utilizes a background thread to periodically refresh tickets based on configured intervals.
    • The default cache file will be written in /tmp dir, but can be modified using kerberos_ccache_path in be.conf

  3. KerberosTicketMgr (kerberos_ticket_mgr.h and kerberos_ticket_mgr.cpp):

    • Acts as a manager for multiple Kerberos ticket caches, handling their lifecycle, including creation, access, and cleanup.
    • Provides methods to get or set ticket caches and retrieve cache file paths.
    • Includes a background thread for cleaning up expired ticket caches every 1 hour. If a cache is longer being referenced, it will be removed.

  4. HdfsMgr

    • A simple and new class to manager the hdfs fs handler.
    • It replace the old HdfsHandlerCache
    • It will check HdfsHandler every 1 hour, and remove unused HdfsHandler after 24 hours.

Mainly Changes

  1. Introduce a comprehensive kerberos ticket cache management on BE side
  2. Use ticket cache path instead of principal and keytab to do the kerberos authentication of libhdfs.
  3. Fix the issue that kerberos_krb5_conf_path in be.conf does not take effect.
  4. Add a new system table backend_kerberos_ticket_cache, to view the krb ticket cache of each backend:
Doris > select * from information_schema.backend_kerberos_ticket_cache\G
*************************** 1. row ***************************
                  BE_ID: 1738304534666
                  BE_IP: 172.20.32.136
              PRINCIPAL: hdfs/[email protected]
                 KEYTAB: /path/to/hdfs.keytab
      SERVICE_PRINCIPAL: krbtgt/[email protected]
      TICKET_CACHE_PATH: /tmp/doris_krb_ce93d5ebb2a6554c7ba9f43aee3a9e6c
              HASH_CODE: ce93d5ebb2a6554c7ba9f43aee3a9e6c
             START_TIME: 2025-02-01 00:08:26
            EXPIRE_TIME: 2025-02-01 00:09:26
              AUTH_TIME: 2025-02-01 00:08:26
              REF_COUNT: 1
REFRESH_INTERVAL_SECOND: 3600

Usage

The user interface remains unchanged.

  1. set krb5.conf path in be.conf kerberos_krb5_conf_path, default is /etc/krb5.conf
  2. provide kerberos principal the keytab path as usual.

Configurations

be.conf

  1. kerberos_ccache_path
    The dir where kerber ticket cache file saved. the file name as format doris_krb_xxxx

  2. kerberos_krb5_conf_path
    The path of krb5.conf file

  3. kerberos_refresh_interval_second
    The min interval to refresh a kerberos ticket cache file. default is 1h.

  4. cleanup logic

    If the ticket cache is not used for 1 day, it will be deleted.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman morningman force-pushed the be_ticket_path branch 4 times, most recently from 972fb7e to 8c80cf5 Compare January 27, 2025 17:37
@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32324 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8c80cf56ce658e77d06f21449423b647d0bbc2c5, data reload: false

------ Round 1 ----------------------------------
q1	17590	5654	5360	5360
q2	2046	325	177	177
q3	10464	1221	742	742
q4	10215	966	545	545
q5	7552	2371	2171	2171
q6	191	175	130	130
q7	882	759	622	622
q8	9221	1373	1236	1236
q9	5166	4981	4898	4898
q10	6833	2346	1863	1863
q11	482	276	266	266
q12	353	356	228	228
q13	17770	3699	3069	3069
q14	232	235	216	216
q15	534	476	476	476
q16	633	639	593	593
q17	589	874	338	338
q18	7066	6359	6493	6359
q19	1921	959	535	535
q20	321	324	186	186
q21	2840	2173	1990	1990
q22	367	352	324	324
Total cold run time: 103268 ms
Total hot run time: 32324 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5570	5508	5452	5452
q2	237	328	230	230
q3	2254	2651	2307	2307
q4	1394	1838	1338	1338
q5	4296	4738	4673	4673
q6	169	159	127	127
q7	2049	1951	1811	1811
q8	2617	2860	2722	2722
q9	7315	7293	7256	7256
q10	3020	3265	2780	2780
q11	591	514	498	498
q12	653	723	607	607
q13	3541	3958	3288	3288
q14	289	300	277	277
q15	507	472	459	459
q16	659	694	648	648
q17	1264	1750	1283	1283
q18	7747	7515	7279	7279
q19	826	952	1168	952
q20	2045	2051	1887	1887
q21	5832	5185	4909	4909
q22	601	595	554	554
Total cold run time: 53476 ms
Total hot run time: 51337 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192037 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8c80cf56ce658e77d06f21449423b647d0bbc2c5, data reload: false

query1	1310	949	951	949
query2	6316	2177	2093	2093
query3	10971	4488	4514	4488
query4	60430	29239	23098	23098
query5	5362	610	452	452
query6	440	193	179	179
query7	5514	503	300	300
query8	329	236	225	225
query9	8065	2733	2690	2690
query10	448	314	263	263
query11	17660	15096	15515	15096
query12	164	121	107	107
query13	1432	552	418	418
query14	11449	7119	7006	7006
query15	214	206	194	194
query16	6589	633	493	493
query17	1107	733	581	581
query18	1713	391	330	330
query19	237	192	168	168
query20	127	117	112	112
query21	211	132	105	105
query22	4673	4730	4517	4517
query23	34069	33565	33348	33348
query24	5572	2274	2337	2274
query25	463	462	411	411
query26	658	273	158	158
query27	1713	501	335	335
query28	4001	2543	2496	2496
query29	546	551	414	414
query30	212	189	157	157
query31	925	895	809	809
query32	65	64	62	62
query33	428	354	289	289
query34	756	867	526	526
query35	808	843	761	761
query36	1015	1026	980	980
query37	120	98	76	76
query38	4348	4337	4337	4337
query39	1502	1473	1431	1431
query40	205	112	97	97
query41	52	48	49	48
query42	135	104	106	104
query43	526	530	509	509
query44	1334	850	852	850
query45	184	184	161	161
query46	877	1054	658	658
query47	1964	1945	1878	1878
query48	391	429	326	326
query49	723	492	416	416
query50	676	676	392	392
query51	4310	4302	4229	4229
query52	115	106	93	93
query53	239	263	191	191
query54	489	532	429	429
query55	81	76	79	76
query56	260	268	268	268
query57	1176	1210	1132	1132
query58	245	237	235	235
query59	3341	3449	3185	3185
query60	284	273	251	251
query61	118	124	120	120
query62	731	713	640	640
query63	227	187	193	187
query64	1241	1008	682	682
query65	3243	3192	3202	3192
query66	735	394	300	300
query67	15984	15749	15347	15347
query68	5038	838	592	592
query69	486	302	263	263
query70	1214	1142	1076	1076
query71	401	283	258	258
query72	6051	3933	3825	3825
query73	798	744	359	359
query74	10071	8954	8827	8827
query75	3200	3131	2688	2688
query76	3796	1156	767	767
query77	490	347	269	269
query78	10138	9913	9343	9343
query79	2812	797	615	615
query80	1583	593	435	435
query81	554	274	231	231
query82	348	156	114	114
query83	270	173	148	148
query84	299	102	85	85
query85	772	363	344	344
query86	453	326	310	310
query87	4570	4505	4432	4432
query88	3977	2193	2121	2121
query89	387	314	293	293
query90	1583	193	192	192
query91	136	144	108	108
query92	66	57	57	57
query93	2849	855	536	536
query94	759	400	298	298
query95	338	274	260	260
query96	500	630	286	286
query97	2843	2902	2731	2731
query98	226	199	199	199
query99	1284	1338	1251	1251
Total cold run time: 310140 ms
Total hot run time: 192037 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.23 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8c80cf56ce658e77d06f21449423b647d0bbc2c5, data reload: false

query1	0.03	0.03	0.05
query2	0.07	0.04	0.03
query3	0.24	0.06	0.07
query4	1.64	0.11	0.11
query5	0.44	0.42	0.39
query6	1.14	0.67	0.64
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.50	0.52
query10	0.58	0.57	0.55
query11	0.15	0.10	0.10
query12	0.14	0.11	0.10
query13	0.61	0.60	0.60
query14	2.83	2.84	2.82
query15	0.90	0.83	0.82
query16	0.38	0.39	0.38
query17	1.04	1.04	1.07
query18	0.22	0.21	0.21
query19	1.88	2.03	1.83
query20	0.02	0.01	0.01
query21	15.37	0.95	0.58
query22	0.76	0.74	0.66
query23	15.32	1.44	0.63
query24	3.49	1.44	1.65
query25	0.12	0.08	0.18
query26	0.36	0.15	0.14
query27	0.06	0.05	0.07
query28	14.51	0.90	0.42
query29	12.59	3.91	3.26
query30	0.26	0.09	0.07
query31	2.83	0.61	0.40
query32	3.22	0.55	0.46
query33	2.98	3.04	3.02
query34	16.50	5.15	4.46
query35	4.53	4.48	4.59
query36	0.65	0.52	0.49
query37	0.09	0.07	0.06
query38	0.04	0.03	0.03
query39	0.04	0.03	0.03
query40	0.16	0.14	0.13
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.96 s
Total hot run time: 31.23 s

@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31973 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f946e312640070cd561cb465feee2d6c4839e80b, data reload: false

------ Round 1 ----------------------------------
q1	17574	5401	5346	5346
q2	2041	307	169	169
q3	10448	1220	719	719
q4	10292	970	513	513
q5	8951	2349	2100	2100
q6	197	161	130	130
q7	888	743	611	611
q8	9236	1351	1172	1172
q9	5212	4889	4918	4889
q10	6816	2326	1886	1886
q11	478	277	257	257
q12	335	351	216	216
q13	18189	3755	3088	3088
q14	247	233	212	212
q15	508	481	463	463
q16	605	611	572	572
q17	567	862	328	328
q18	6872	6422	6319	6319
q19	2739	962	521	521
q20	300	317	185	185
q21	2760	2139	1965	1965
q22	361	335	312	312
Total cold run time: 105616 ms
Total hot run time: 31973 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5580	5456	5519	5456
q2	233	330	230	230
q3	2262	2628	2327	2327
q4	1390	1777	1366	1366
q5	4305	4690	4690	4690
q6	167	167	125	125
q7	2049	1969	1819	1819
q8	2649	2839	2674	2674
q9	7275	7255	7319	7255
q10	3023	3280	2806	2806
q11	568	513	482	482
q12	635	725	624	624
q13	3506	3855	3300	3300
q14	285	315	285	285
q15	525	497	480	480
q16	663	668	635	635
q17	1214	1757	1263	1263
q18	7546	7550	7252	7252
q19	814	1094	1083	1083
q20	1996	2037	1927	1927
q21	5740	5125	4858	4858
q22	600	581	568	568
Total cold run time: 53025 ms
Total hot run time: 51505 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191165 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f946e312640070cd561cb465feee2d6c4839e80b, data reload: false

query1	1333	941	939	939
query2	6293	2099	2101	2099
query3	11061	4738	4456	4456
query4	61279	28722	22990	22990
query5	5645	569	424	424
query6	417	192	176	176
query7	5518	510	287	287
query8	319	218	232	218
query9	8373	2679	2661	2661
query10	467	308	248	248
query11	17300	15068	15603	15068
query12	158	109	104	104
query13	1436	546	439	439
query14	11602	6758	6599	6599
query15	201	211	193	193
query16	7334	672	519	519
query17	1177	745	588	588
query18	1768	408	307	307
query19	217	185	164	164
query20	118	116	115	115
query21	211	124	104	104
query22	4571	4789	4358	4358
query23	33988	33993	33420	33420
query24	5629	2342	2271	2271
query25	456	455	409	409
query26	663	254	155	155
query27	1764	472	320	320
query28	4388	2506	2486	2486
query29	557	530	427	427
query30	209	188	168	168
query31	932	893	815	815
query32	65	60	56	56
query33	433	358	297	297
query34	742	835	513	513
query35	791	838	799	799
query36	987	1009	954	954
query37	130	99	78	78
query38	4318	4307	4220	4220
query39	1543	1456	1446	1446
query40	196	115	106	106
query41	50	49	47	47
query42	126	106	103	103
query43	539	536	532	532
query44	1319	834	815	815
query45	186	174	174	174
query46	889	1056	649	649
query47	1938	1939	1866	1866
query48	386	415	321	321
query49	759	480	391	391
query50	639	694	400	400
query51	4297	4278	4293	4278
query52	113	105	97	97
query53	227	255	190	190
query54	486	514	423	423
query55	89	82	83	82
query56	268	269	245	245
query57	1195	1205	1152	1152
query58	245	244	244	244
query59	3239	3457	3316	3316
query60	273	261	241	241
query61	120	117	118	117
query62	721	773	644	644
query63	219	188	185	185
query64	1384	1095	642	642
query65	3235	3177	3131	3131
query66	765	400	287	287
query67	15917	15610	15505	15505
query68	5043	753	529	529
query69	478	288	252	252
query70	1169	1117	1134	1117
query71	407	293	266	266
query72	6078	3836	3737	3737
query73	769	756	358	358
query74	10283	8926	8850	8850
query75	3204	3127	2636	2636
query76	3798	1164	747	747
query77	482	389	285	285
query78	9935	9836	9227	9227
query79	3016	784	584	584
query80	674	523	458	458
query81	492	269	239	239
query82	355	150	114	114
query83	174	169	155	155
query84	289	91	73	73
query85	788	367	302	302
query86	376	316	295	295
query87	4422	4545	4521	4521
query88	4545	2166	2136	2136
query89	474	322	290	290
query90	1651	189	187	187
query91	136	134	106	106
query92	63	56	52	52
query93	2506	854	529	529
query94	730	376	281	281
query95	333	255	249	249
query96	500	658	286	286
query97	2784	2849	2786	2786
query98	225	201	195	195
query99	1313	1345	1250	1250
Total cold run time: 311705 ms
Total hot run time: 191165 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.13 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f946e312640070cd561cb465feee2d6c4839e80b, data reload: false

query1	0.04	0.03	0.03
query2	0.10	0.03	0.03
query3	0.24	0.07	0.07
query4	1.62	0.10	0.10
query5	0.41	0.43	0.41
query6	1.15	0.65	0.67
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.58	0.50	0.49
query10	0.56	0.55	0.56
query11	0.15	0.11	0.10
query12	0.14	0.11	0.12
query13	0.61	0.60	0.59
query14	2.81	2.87	2.75
query15	0.92	0.82	0.83
query16	0.40	0.40	0.37
query17	1.04	1.07	1.08
query18	0.23	0.21	0.20
query19	1.91	1.79	1.95
query20	0.01	0.01	0.01
query21	15.35	1.00	0.56
query22	0.76	0.88	0.65
query23	15.22	1.50	0.57
query24	3.60	1.44	1.52
query25	0.28	0.22	0.06
query26	0.20	0.14	0.14
query27	0.05	0.07	0.05
query28	14.16	1.00	0.43
query29	12.58	3.93	3.25
query30	0.25	0.09	0.07
query31	2.84	0.60	0.38
query32	3.22	0.56	0.46
query33	2.98	3.06	3.12
query34	16.45	5.32	4.50
query35	4.55	4.54	4.57
query36	0.66	0.48	0.47
query37	0.10	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.14	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.63 s
Total hot run time: 31.13 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.09% (11005/26147)
Line Coverage: 32.37% (92981/287241)
Region Coverage: 31.51% (47651/151208)
Branch Coverage: 27.54% (24119/87572)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f946e312640070cd561cb465feee2d6c4839e80b_f946e312640070cd561cb465feee2d6c4839e80b/report/index.html

@morningman
Copy link
Contributor Author

run buildall

@morningman morningman changed the title [opt](kerberos) use ticket patch instead of keytab [opt](kerberos) use ticket cache instead of keytab on BE side Jan 28, 2025
@morningman morningman closed this Jan 28, 2025
@morningman morningman reopened this Jan 28, 2025
@morningman morningman marked this pull request as ready for review January 28, 2025 09:31
@doris-robot
Copy link

TPC-H: Total hot run time: 32501 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7124069f5634503d3e44868c616e30f30661e1f7, data reload: false

------ Round 1 ----------------------------------
q1	17568	5519	5398	5398
q2	2053	324	193	193
q3	10373	1244	743	743
q4	10214	972	531	531
q5	7523	2457	2163	2163
q6	194	171	130	130
q7	901	771	603	603
q8	9216	1359	1182	1182
q9	5345	4930	4865	4865
q10	6856	2334	1919	1919
q11	477	275	256	256
q12	345	361	221	221
q13	17780	3815	3150	3150
q14	227	240	206	206
q15	518	488	447	447
q16	624	628	586	586
q17	576	901	335	335
q18	7110	6499	6493	6493
q19	1845	945	554	554
q20	314	345	195	195
q21	2908	2202	2019	2019
q22	361	337	312	312
Total cold run time: 103328 ms
Total hot run time: 32501 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5598	5482	5490	5482
q2	239	329	232	232
q3	2305	2679	2300	2300
q4	1374	1801	1351	1351
q5	4385	4786	4702	4702
q6	173	161	125	125
q7	2095	1993	1835	1835
q8	2668	2824	2760	2760
q9	7390	7240	7254	7240
q10	3051	3313	2782	2782
q11	574	507	478	478
q12	631	746	563	563
q13	3439	4013	3343	3343
q14	305	313	295	295
q15	512	481	461	461
q16	658	682	644	644
q17	1257	1748	1266	1266
q18	7874	7638	7367	7367
q19	826	1181	1094	1094
q20	2021	2081	1935	1935
q21	5882	5453	5060	5060
q22	605	595	578	578
Total cold run time: 53862 ms
Total hot run time: 51893 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191616 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7124069f5634503d3e44868c616e30f30661e1f7, data reload: false

query1	1316	957	926	926
query2	6184	2142	2121	2121
query3	10967	4487	4586	4487
query4	32483	23890	23138	23138
query5	4351	613	457	457
query6	292	201	202	201
query7	3988	502	319	319
query8	295	270	231	231
query9	9621	2704	2710	2704
query10	473	323	242	242
query11	17779	15381	14862	14862
query12	156	110	102	102
query13	1563	538	401	401
query14	8716	7528	6395	6395
query15	272	213	192	192
query16	7994	670	521	521
query17	1610	740	579	579
query18	2115	412	308	308
query19	204	181	154	154
query20	124	116	130	116
query21	207	127	105	105
query22	4490	4462	4450	4450
query23	34487	33993	33321	33321
query24	6642	2378	2254	2254
query25	485	475	406	406
query26	820	276	158	158
query27	2344	474	334	334
query28	5192	2524	2515	2515
query29	670	572	431	431
query30	214	187	152	152
query31	966	880	833	833
query32	87	64	58	58
query33	479	350	329	329
query34	754	877	530	530
query35	821	871	747	747
query36	1006	1035	960	960
query37	123	101	88	88
query38	4452	4461	4294	4294
query39	1499	1409	1428	1409
query40	201	113	105	105
query41	49	50	47	47
query42	125	99	100	99
query43	530	543	507	507
query44	1327	835	844	835
query45	191	172	172	172
query46	881	1058	648	648
query47	1882	1888	1829	1829
query48	408	398	335	335
query49	745	484	401	401
query50	642	683	398	398
query51	4285	4295	4288	4288
query52	109	104	100	100
query53	241	282	200	200
query54	505	513	433	433
query55	91	81	80	80
query56	277	289	260	260
query57	1234	1207	1136	1136
query58	266	236	243	236
query59	3292	3306	3321	3306
query60	300	266	262	262
query61	123	120	120	120
query62	775	751	715	715
query63	228	194	198	194
query64	3624	1049	652	652
query65	3312	3235	3145	3145
query66	793	391	310	310
query67	15968	15561	15349	15349
query68	2075	820	558	558
query69	443	295	265	265
query70	1208	1135	1135	1135
query71	340	288	258	258
query72	6121	4037	3959	3959
query73	652	758	359	359
query74	10165	8971	8903	8903
query75	3182	3138	2658	2658
query76	2796	1169	745	745
query77	460	374	363	363
query78	10006	10014	9371	9371
query79	3187	809	584	584
query80	802	516	438	438
query81	515	278	240	240
query82	710	152	124	124
query83	174	174	155	155
query84	232	94	74	74
query85	770	350	301	301
query86	418	321	296	296
query87	4458	4540	4476	4476
query88	5103	2183	2171	2171
query89	390	332	285	285
query90	1805	187	184	184
query91	137	137	107	107
query92	64	60	52	52
query93	2714	880	540	540
query94	769	415	276	276
query95	339	261	252	252
query96	489	608	281	281
query97	2806	2849	2771	2771
query98	263	191	189	189
query99	1278	1368	1254	1254
Total cold run time: 283516 ms
Total hot run time: 191616 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.51 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7124069f5634503d3e44868c616e30f30661e1f7, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.03
query3	0.24	0.07	0.07
query4	1.60	0.10	0.10
query5	0.43	0.42	0.40
query6	1.17	0.64	0.65
query7	0.02	0.01	0.02
query8	0.05	0.03	0.03
query9	0.61	0.49	0.51
query10	0.56	0.57	0.54
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.62	0.60	0.59
query14	2.72	2.72	2.83
query15	0.89	0.83	0.85
query16	0.39	0.38	0.36
query17	1.09	1.07	1.05
query18	0.23	0.21	0.20
query19	1.98	1.84	2.00
query20	0.01	0.01	0.02
query21	15.37	0.96	0.55
query22	0.74	0.99	0.56
query23	15.26	1.38	0.60
query24	2.84	1.72	1.17
query25	0.23	0.16	0.07
query26	0.25	0.14	0.14
query27	0.05	0.05	0.05
query28	13.98	0.94	0.43
query29	12.59	3.88	3.20
query30	0.24	0.09	0.06
query31	2.83	0.60	0.39
query32	3.23	0.55	0.45
query33	2.99	3.11	3.01
query34	16.47	5.15	4.43
query35	4.52	4.50	4.47
query36	0.65	0.51	0.49
query37	0.10	0.06	0.05
query38	0.05	0.04	0.03
query39	0.04	0.03	0.02
query40	0.16	0.14	0.12
query41	0.09	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.75 s
Total hot run time: 30.51 s

@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32325 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b566aa4b3d3ef0ec297769ed599fa7f74e8f9164, data reload: false

------ Round 1 ----------------------------------
q1	17581	5431	5351	5351
q2	2045	301	167	167
q3	10439	1252	754	754
q4	10223	966	553	553
q5	8122	2447	2199	2199
q6	191	181	135	135
q7	905	769	610	610
q8	9223	1325	1120	1120
q9	5125	4920	4881	4881
q10	6942	2331	1872	1872
q11	470	277	255	255
q12	348	352	221	221
q13	17761	3712	3189	3189
q14	234	227	209	209
q15	505	467	477	467
q16	623	606	583	583
q17	577	869	326	326
q18	6852	6394	6503	6394
q19	1247	965	557	557
q20	318	324	192	192
q21	2793	2174	1987	1987
q22	370	333	303	303
Total cold run time: 102894 ms
Total hot run time: 32325 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5490	5534	5451	5451
q2	239	332	226	226
q3	2256	2797	2317	2317
q4	1392	1794	1374	1374
q5	4329	4709	4616	4616
q6	174	160	126	126
q7	2051	1931	1840	1840
q8	2629	2850	2704	2704
q9	7279	7182	7198	7182
q10	3037	3281	2809	2809
q11	592	535	499	499
q12	636	728	606	606
q13	3425	3854	3083	3083
q14	268	287	261	261
q15	506	460	479	460
q16	632	669	633	633
q17	1201	1739	1232	1232
q18	7167	7198	7100	7100
q19	778	1117	1003	1003
q20	1991	1942	1830	1830
q21	5480	5145	4963	4963
q22	591	610	560	560
Total cold run time: 52143 ms
Total hot run time: 50875 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.09% (11005/26148)
Line Coverage: 32.36% (92944/287246)
Region Coverage: 31.51% (47652/151209)
Branch Coverage: 27.54% (24121/87572)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b566aa4b3d3ef0ec297769ed599fa7f74e8f9164_b566aa4b3d3ef0ec297769ed599fa7f74e8f9164/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 196460 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b566aa4b3d3ef0ec297769ed599fa7f74e8f9164, data reload: false

query1	1337	957	926	926
query2	13109	9139	9298	9139
query3	6792	216	219	216
query4	33275	23216	22945	22945
query5	4344	655	460	460
query6	294	203	201	201
query7	4625	506	316	316
query8	303	252	235	235
query9	9509	2728	2683	2683
query10	441	339	265	265
query11	18325	15064	14953	14953
query12	150	110	101	101
query13	1635	534	412	412
query14	9685	7610	7662	7610
query15	222	197	201	197
query16	7205	672	469	469
query17	1541	739	567	567
query18	1968	385	312	312
query19	241	183	170	170
query20	141	117	115	115
query21	217	124	108	108
query22	4434	4629	4390	4390
query23	34057	33309	33509	33309
query24	6500	2398	2416	2398
query25	527	483	426	426
query26	748	288	162	162
query27	1888	483	337	337
query28	5173	2532	2523	2523
query29	622	615	460	460
query30	185	190	163	163
query31	1003	891	847	847
query32	88	64	64	64
query33	501	393	324	324
query34	778	907	513	513
query35	840	829	782	782
query36	1050	1037	973	973
query37	125	116	81	81
query38	4387	4421	4261	4261
query39	1493	1413	1457	1413
query40	169	118	104	104
query41	51	50	49	49
query42	120	101	108	101
query43	530	552	501	501
query44	1409	828	851	828
query45	182	174	175	174
query46	896	1066	672	672
query47	1899	1940	1817	1817
query48	394	419	329	329
query49	783	522	418	418
query50	678	700	418	418
query51	4287	4374	4261	4261
query52	106	103	102	102
query53	239	285	192	192
query54	495	526	410	410
query55	80	86	90	86
query56	278	275	275	275
query57	1179	1177	1156	1156
query58	247	256	243	243
query59	3448	3332	3328	3328
query60	306	281	280	280
query61	145	121	144	121
query62	790	715	669	669
query63	236	199	199	199
query64	3242	1041	679	679
query65	3315	3249	3257	3249
query66	900	396	321	321
query67	16698	16451	16109	16109
query68	6808	837	537	537
query69	477	303	262	262
query70	1200	1173	1147	1147
query71	434	288	257	257
query72	5155	3888	3845	3845
query73	668	760	366	366
query74	10267	9045	8841	8841
query75	4034	3147	2660	2660
query76	3539	1176	791	791
query77	760	409	276	276
query78	9967	9959	9318	9318
query79	3112	819	607	607
query80	739	545	439	439
query81	451	279	236	236
query82	686	155	124	124
query83	202	172	159	159
query84	278	90	74	74
query85	740	349	379	349
query86	366	315	311	311
query87	4568	4557	4464	4464
query88	3304	2222	2170	2170
query89	391	341	305	305
query90	1973	193	195	193
query91	136	143	112	112
query92	67	58	55	55
query93	1062	891	549	549
query94	665	410	305	305
query95	345	272	261	261
query96	491	616	292	292
query97	2786	2845	2758	2758
query98	227	195	205	195
query99	1492	1362	1285	1285
Total cold run time: 290250 ms
Total hot run time: 196460 ms

@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32387 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5d6f86865b24de4f0636e38dc96ebffe54f2f24b, data reload: false

------ Round 1 ----------------------------------
q1	17736	5520	5455	5455
q2	2065	304	169	169
q3	10521	1223	735	735
q4	10217	982	531	531
q5	7603	2411	2176	2176
q6	192	166	139	139
q7	906	748	608	608
q8	9231	1384	1192	1192
q9	5134	4922	4902	4902
q10	6855	2354	1883	1883
q11	474	275	256	256
q12	354	363	219	219
q13	17791	3727	3083	3083
q14	237	230	210	210
q15	527	481	468	468
q16	641	618	588	588
q17	583	880	328	328
q18	7387	6476	6384	6384
q19	1415	962	560	560
q20	361	329	188	188
q21	2851	2247	2001	2001
q22	367	338	312	312
Total cold run time: 103448 ms
Total hot run time: 32387 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5526	5503	5542	5503
q2	237	336	241	241
q3	2274	2672	2329	2329
q4	1397	1859	1405	1405
q5	4322	4728	4698	4698
q6	170	160	127	127
q7	2082	1983	1918	1918
q8	2662	2897	2705	2705
q9	7315	7274	7250	7250
q10	3035	3291	2827	2827
q11	583	529	489	489
q12	665	746	592	592
q13	3639	3972	3374	3374
q14	288	293	271	271
q15	540	469	470	469
q16	666	692	649	649
q17	1262	1730	1274	1274
q18	7824	7420	7441	7420
q19	836	1209	1089	1089
q20	2029	2068	1918	1918
q21	5861	5294	5088	5088
q22	645	611	554	554
Total cold run time: 53858 ms
Total hot run time: 52190 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.13% (11043/26209)
Line Coverage: 32.41% (93176/287519)
Region Coverage: 31.55% (47731/151296)
Branch Coverage: 27.57% (24152/87600)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5d6f86865b24de4f0636e38dc96ebffe54f2f24b_5d6f86865b24de4f0636e38dc96ebffe54f2f24b/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 192557 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5d6f86865b24de4f0636e38dc96ebffe54f2f24b, data reload: false

query1	1329	956	921	921
query2	6115	2059	1997	1997
query3	10967	4575	4382	4382
query4	61366	32948	23275	23275
query5	5548	625	470	470
query6	355	204	204	204
query7	5030	503	297	297
query8	314	249	244	244
query9	5834	2567	2559	2559
query10	430	321	262	262
query11	15586	15296	15479	15296
query12	165	108	105	105
query13	1030	548	404	404
query14	10405	7925	7377	7377
query15	226	220	187	187
query16	7272	621	464	464
query17	1112	745	586	586
query18	1905	408	309	309
query19	193	185	160	160
query20	117	119	116	116
query21	216	128	100	100
query22	4397	4389	4591	4389
query23	34075	33296	33474	33296
query24	5634	2446	2355	2355
query25	505	496	429	429
query26	659	282	183	183
query27	1686	510	341	341
query28	4160	2506	2444	2444
query29	577	608	455	455
query30	226	200	165	165
query31	940	880	829	829
query32	70	62	62	62
query33	441	397	319	319
query34	742	875	559	559
query35	828	855	758	758
query36	1028	1045	976	976
query37	125	108	82	82
query38	4370	4273	4193	4193
query39	1501	1434	1455	1434
query40	210	114	102	102
query41	51	53	55	53
query42	128	102	106	102
query43	536	530	516	516
query44	1349	834	833	833
query45	188	180	167	167
query46	910	1072	663	663
query47	1889	1889	1821	1821
query48	386	406	366	366
query49	730	504	404	404
query50	668	681	398	398
query51	4302	4363	4283	4283
query52	111	110	100	100
query53	231	275	195	195
query54	501	516	428	428
query55	82	81	82	81
query56	269	266	263	263
query57	1190	1221	1148	1148
query58	244	252	237	237
query59	3155	3244	3026	3026
query60	290	280	264	264
query61	121	115	123	115
query62	752	746	676	676
query63	242	193	197	193
query64	1341	1048	683	683
query65	3275	3183	3171	3171
query66	805	391	292	292
query67	16003	15609	15535	15535
query68	5036	795	529	529
query69	495	305	265	265
query70	1218	1123	1147	1123
query71	402	300	265	265
query72	6203	3851	3959	3851
query73	820	755	360	360
query74	10279	8994	9124	8994
query75	3256	3157	2683	2683
query76	3773	1181	786	786
query77	566	370	292	292
query78	10182	10005	9295	9295
query79	2479	843	613	613
query80	1133	581	445	445
query81	548	282	249	249
query82	347	154	123	123
query83	282	174	160	160
query84	291	92	77	77
query85	751	368	303	303
query86	424	320	317	317
query87	4473	4566	4324	4324
query88	3510	2159	2138	2138
query89	405	330	296	296
query90	1624	198	193	193
query91	139	140	109	109
query92	70	57	55	55
query93	2097	857	535	535
query94	762	405	284	284
query95	328	283	268	268
query96	494	632	284	284
query97	2760	2919	2757	2757
query98	227	216	191	191
query99	1276	1379	1294	1294
Total cold run time: 304088 ms
Total hot run time: 192557 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.87 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5d6f86865b24de4f0636e38dc96ebffe54f2f24b, data reload: false

query1	0.04	0.04	0.03
query2	0.09	0.04	0.05
query3	0.23	0.06	0.07
query4	1.64	0.08	0.10
query5	0.42	0.39	0.39
query6	1.17	0.68	0.67
query7	0.02	0.01	0.01
query8	0.06	0.05	0.04
query9	0.56	0.49	0.51
query10	0.56	0.54	0.55
query11	0.17	0.13	0.11
query12	0.15	0.12	0.13
query13	0.61	0.60	0.61
query14	2.74	2.88	2.78
query15	0.92	0.84	0.83
query16	0.39	0.38	0.38
query17	1.01	1.01	1.04
query18	0.21	0.20	0.19
query19	1.88	1.79	2.08
query20	0.01	0.02	0.01
query21	15.37	0.98	0.67
query22	0.76	0.80	0.71
query23	14.95	1.50	0.68
query24	2.19	0.33	0.22
query25	0.14	0.09	0.08
query26	0.30	0.19	0.18
query27	0.08	0.07	0.08
query28	13.45	1.23	0.55
query29	12.66	4.11	3.39
query30	0.25	0.10	0.06
query31	2.82	0.62	0.40
query32	3.22	0.58	0.48
query33	2.99	3.12	3.04
query34	16.52	5.17	4.55
query35	4.55	4.51	4.64
query36	0.61	0.48	0.48
query37	0.19	0.15	0.16
query38	0.17	0.15	0.15
query39	0.05	0.04	0.05
query40	0.17	0.14	0.13
query41	0.11	0.06	0.06
query42	0.06	0.05	0.06
query43	0.05	0.05	0.05
Total cold run time: 104.54 s
Total hot run time: 30.87 s

@morningman
Copy link
Contributor Author

run buildall

1 similar comment
@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32510 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 212bd323f67a09356324fc898cc8d4a1e413c190, data reload: false

------ Round 1 ----------------------------------
q1	17776	5517	5354	5354
q2	2060	314	171	171
q3	10532	1262	761	761
q4	10222	988	544	544
q5	7775	2423	2168	2168
q6	194	167	132	132
q7	914	774	609	609
q8	9249	1398	1225	1225
q9	5316	4908	4976	4908
q10	6817	2347	1885	1885
q11	463	294	254	254
q12	346	373	223	223
q13	17774	3750	3128	3128
q14	244	235	217	217
q15	521	469	460	460
q16	634	627	593	593
q17	607	863	336	336
q18	7279	6454	6635	6454
q19	1758	970	558	558
q20	311	315	186	186
q21	3084	2241	2037	2037
q22	379	346	307	307
Total cold run time: 104255 ms
Total hot run time: 32510 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5529	5520	5557	5520
q2	237	330	235	235
q3	2263	2672	2358	2358
q4	1538	1847	1418	1418
q5	4336	4749	4794	4749
q6	172	159	127	127
q7	2094	2029	1789	1789
q8	2668	2835	2747	2747
q9	7369	7282	7426	7282
q10	3063	3336	2767	2767
q11	576	522	482	482
q12	649	749	610	610
q13	3574	3998	3284	3284
q14	283	292	260	260
q15	514	475	453	453
q16	653	685	658	658
q17	1263	1753	1292	1292
q18	7867	7591	7488	7488
q19	833	1144	1112	1112
q20	2013	2030	1920	1920
q21	5976	5186	5306	5186
q22	602	643	569	569
Total cold run time: 54072 ms
Total hot run time: 52306 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191703 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 212bd323f67a09356324fc898cc8d4a1e413c190, data reload: false

query1	1305	960	936	936
query2	6184	2106	2058	2058
query3	10916	4514	4244	4244
query4	60814	28685	23411	23411
query5	5412	600	441	441
query6	420	185	191	185
query7	5480	523	298	298
query8	335	256	230	230
query9	7956	2718	2701	2701
query10	435	307	264	264
query11	17036	15248	15484	15248
query12	170	116	109	109
query13	1370	544	444	444
query14	11177	7513	6441	6441
query15	218	219	201	201
query16	7300	681	498	498
query17	1107	748	603	603
query18	1955	420	334	334
query19	217	185	165	165
query20	118	114	115	114
query21	219	134	105	105
query22	4360	4683	4352	4352
query23	34010	33465	33539	33465
query24	5462	2377	2304	2304
query25	482	467	393	393
query26	642	278	150	150
query27	1667	487	323	323
query28	3945	2521	2492	2492
query29	536	558	442	442
query30	208	191	157	157
query31	962	875	838	838
query32	67	58	57	57
query33	498	347	299	299
query34	734	894	502	502
query35	831	849	789	789
query36	1003	1023	968	968
query37	123	103	79	79
query38	4275	4397	4415	4397
query39	1505	1443	1434	1434
query40	212	113	100	100
query41	56	49	49	49
query42	119	104	104	104
query43	523	523	510	510
query44	1344	816	806	806
query45	179	176	174	174
query46	893	1066	664	664
query47	1899	1958	1860	1860
query48	393	420	313	313
query49	711	482	415	415
query50	649	700	407	407
query51	4303	4371	4214	4214
query52	101	101	93	93
query53	237	256	195	195
query54	488	500	419	419
query55	85	80	80	80
query56	255	262	251	251
query57	1184	1206	1116	1116
query58	241	233	239	233
query59	3145	3409	3114	3114
query60	287	266	265	265
query61	115	117	119	117
query62	744	758	669	669
query63	221	185	194	185
query64	1300	988	701	701
query65	3241	3171	3131	3131
query66	727	391	303	303
query67	16016	15743	15470	15470
query68	5047	828	526	526
query69	497	296	262	262
query70	1199	1136	1129	1129
query71	419	281	254	254
query72	6129	3890	3920	3890
query73	811	775	360	360
query74	10143	8910	8831	8831
query75	3220	3252	2687	2687
query76	3835	1189	766	766
query77	486	372	276	276
query78	10109	10105	9362	9362
query79	2522	801	600	600
query80	691	526	443	443
query81	502	281	239	239
query82	305	150	119	119
query83	179	168	155	155
query84	284	88	74	74
query85	741	358	299	299
query86	400	325	295	295
query87	4605	4487	4333	4333
query88	3666	2210	2200	2200
query89	386	329	291	291
query90	1676	188	189	188
query91	136	139	110	110
query92	66	56	53	53
query93	2912	876	528	528
query94	689	420	298	298
query95	328	263	258	258
query96	506	616	284	284
query97	2831	2850	2731	2731
query98	225	198	197	197
query99	1301	1384	1236	1236
Total cold run time: 307968 ms
Total hot run time: 191703 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.17 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 212bd323f67a09356324fc898cc8d4a1e413c190, data reload: false

query1	0.03	0.04	0.03
query2	0.09	0.05	0.04
query3	0.23	0.05	0.05
query4	1.66	0.09	0.09
query5	0.41	0.40	0.40
query6	1.16	0.66	0.66
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.54	0.50	0.50
query10	0.55	0.57	0.57
query11	0.16	0.12	0.12
query12	0.16	0.13	0.12
query13	0.61	0.60	0.60
query14	2.74	2.87	2.77
query15	0.90	0.85	0.84
query16	0.37	0.37	0.38
query17	1.07	1.07	1.07
query18	0.19	0.19	0.18
query19	1.84	1.90	2.02
query20	0.02	0.01	0.02
query21	15.35	0.96	0.66
query22	0.77	0.80	0.76
query23	14.93	1.50	0.68
query24	2.24	0.35	0.21
query25	0.14	0.09	0.09
query26	0.28	0.19	0.18
query27	0.09	0.08	0.08
query28	13.44	1.23	0.55
query29	12.67	4.19	3.46
query30	0.25	0.08	0.06
query31	2.83	0.60	0.40
query32	3.23	0.57	0.48
query33	2.94	3.05	3.03
query34	16.56	5.14	4.50
query35	4.59	4.55	4.55
query36	0.61	0.50	0.51
query37	0.20	0.16	0.17
query38	0.16	0.14	0.15
query39	0.05	0.04	0.04
query40	0.16	0.15	0.12
query41	0.11	0.05	0.04
query42	0.06	0.06	0.05
query43	0.06	0.05	0.05
Total cold run time: 104.52 s
Total hot run time: 31.17 s

@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32514 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2acb8f61c0d213a6d9e268d70af623736ce04db7, data reload: false

------ Round 1 ----------------------------------
q1	17783	5571	5463	5463
q2	2060	320	175	175
q3	10559	1215	755	755
q4	10294	977	532	532
q5	8704	2352	2179	2179
q6	194	168	135	135
q7	907	778	621	621
q8	9253	1345	1147	1147
q9	5272	4906	4925	4906
q10	6867	2357	1888	1888
q11	486	290	260	260
q12	346	363	224	224
q13	17786	3735	3111	3111
q14	222	234	214	214
q15	531	487	455	455
q16	615	639	586	586
q17	585	882	336	336
q18	6905	6510	6490	6490
q19	1998	950	534	534
q20	327	311	187	187
q21	2812	2193	2009	2009
q22	380	351	307	307
Total cold run time: 104886 ms
Total hot run time: 32514 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5550	5494	5511	5494
q2	236	324	232	232
q3	2196	2668	2317	2317
q4	1360	1824	1355	1355
q5	4331	4735	4735	4735
q6	175	163	125	125
q7	2080	1930	1822	1822
q8	2613	2856	2728	2728
q9	7265	7170	7268	7170
q10	3017	3298	2791	2791
q11	557	509	492	492
q12	628	724	589	589
q13	3592	3925	3332	3332
q14	283	313	272	272
q15	522	473	471	471
q16	637	699	674	674
q17	1242	1744	1275	1275
q18	7577	7559	7185	7185
q19	828	1125	1086	1086
q20	2027	2020	1888	1888
q21	5767	5280	4906	4906
q22	607	606	554	554
Total cold run time: 53090 ms
Total hot run time: 51493 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191232 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2acb8f61c0d213a6d9e268d70af623736ce04db7, data reload: false

query1	1313	929	969	929
query2	6236	2117	2020	2020
query3	10915	4329	4407	4329
query4	61142	29012	23021	23021
query5	5487	603	436	436
query6	427	205	200	200
query7	5477	506	296	296
query8	333	255	233	233
query9	7796	2701	2657	2657
query10	446	307	253	253
query11	17321	14957	15511	14957
query12	172	111	108	108
query13	1394	536	434	434
query14	10956	7432	6845	6845
query15	216	212	207	207
query16	7203	659	477	477
query17	1105	753	578	578
query18	1920	427	331	331
query19	207	192	173	173
query20	123	118	112	112
query21	209	122	108	108
query22	4655	4628	4559	4559
query23	34107	33351	33578	33351
query24	5579	2260	2336	2260
query25	468	471	414	414
query26	642	286	157	157
query27	1743	509	347	347
query28	3994	2554	2488	2488
query29	546	569	446	446
query30	218	188	155	155
query31	927	899	826	826
query32	73	60	58	58
query33	431	382	293	293
query34	761	869	519	519
query35	829	829	764	764
query36	1002	1058	936	936
query37	118	96	77	77
query38	4326	4402	4316	4316
query39	1499	1459	1423	1423
query40	204	113	102	102
query41	58	56	62	56
query42	129	104	109	104
query43	534	523	500	500
query44	1302	824	830	824
query45	189	174	170	170
query46	875	1075	655	655
query47	1897	1914	1827	1827
query48	385	410	314	314
query49	739	488	409	409
query50	672	687	417	417
query51	4351	4346	4227	4227
query52	105	104	96	96
query53	231	265	197	197
query54	492	498	415	415
query55	87	85	87	85
query56	259	256	244	244
query57	1185	1228	1136	1136
query58	268	232	237	232
query59	3290	3242	3145	3145
query60	281	273	262	262
query61	131	120	130	120
query62	719	758	701	701
query63	219	187	191	187
query64	1285	1020	667	667
query65	3240	3166	3138	3138
query66	653	395	305	305
query67	15996	15668	15273	15273
query68	6097	821	531	531
query69	488	307	261	261
query70	1187	1136	1046	1046
query71	435	280	260	260
query72	5350	3870	3880	3870
query73	919	765	368	368
query74	10352	9073	8742	8742
query75	3249	3152	2727	2727
query76	3765	1170	765	765
query77	491	357	308	308
query78	10060	10063	9280	9280
query79	3069	800	603	603
query80	673	560	462	462
query81	495	272	245	245
query82	227	153	130	130
query83	185	185	149	149
query84	295	94	75	75
query85	757	346	296	296
query86	384	298	304	298
query87	4446	4446	4536	4446
query88	4476	2198	2201	2198
query89	390	322	289	289
query90	1644	190	192	190
query91	136	134	109	109
query92	66	55	53	53
query93	2522	859	542	542
query94	699	405	302	302
query95	340	271	256	256
query96	487	619	291	291
query97	2863	2862	2793	2793
query98	222	189	195	189
query99	1302	1359	1257	1257
Total cold run time: 310163 ms
Total hot run time: 191232 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.8 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2acb8f61c0d213a6d9e268d70af623736ce04db7, data reload: false

query1	0.03	0.03	0.04
query2	0.08	0.05	0.04
query3	0.23	0.05	0.05
query4	1.65	0.08	0.10
query5	0.41	0.40	0.40
query6	1.16	0.65	0.67
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.55	0.50	0.51
query10	0.56	0.56	0.56
query11	0.18	0.12	0.12
query12	0.16	0.12	0.12
query13	0.62	0.60	0.59
query14	2.74	2.75	2.77
query15	0.92	0.82	0.82
query16	0.38	0.39	0.38
query17	1.02	1.08	1.01
query18	0.18	0.17	0.19
query19	1.98	1.87	1.97
query20	0.02	0.02	0.02
query21	15.37	1.04	0.71
query22	0.78	0.77	0.69
query23	14.98	1.49	0.67
query24	2.18	0.38	0.22
query25	0.14	0.08	0.09
query26	0.28	0.18	0.18
query27	0.08	0.08	0.08
query28	13.38	1.27	0.53
query29	12.66	4.17	3.38
query30	0.25	0.08	0.06
query31	2.89	0.60	0.39
query32	3.23	0.57	0.48
query33	3.13	2.96	3.11
query34	16.19	5.12	4.53
query35	4.53	4.55	4.56
query36	0.61	0.48	0.46
query37	0.20	0.16	0.15
query38	0.15	0.14	0.15
query39	0.05	0.04	0.04
query40	0.17	0.13	0.13
query41	0.10	0.06	0.05
query42	0.06	0.05	0.06
query43	0.05	0.05	0.05
Total cold run time: 104.4 s
Total hot run time: 30.8 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.14% (11044/26210)
Line Coverage: 32.41% (93201/287543)
Region Coverage: 31.55% (47739/151300)
Branch Coverage: 27.56% (24146/87600)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2acb8f61c0d213a6d9e268d70af623736ce04db7_2acb8f61c0d213a6d9e268d70af623736ce04db7/report/index.html

@morningman
Copy link
Contributor Author

run buildall

@morningman morningman changed the title [opt](kerberos) use ticket cache instead of keytab on BE side [opt](kerberos) use ticket cache instead of principal+keytab on BE side Feb 5, 2025
@doris-robot
Copy link

TPC-H: Total hot run time: 32396 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8a653b95bf23fa397cb2a3fa4456c8db74fbfebe, data reload: false

------ Round 1 ----------------------------------
q1	17746	5474	5365	5365
q2	2051	303	168	168
q3	10543	1235	740	740
q4	10227	969	535	535
q5	7743	2401	2176	2176
q6	194	176	136	136
q7	1014	760	604	604
q8	9246	1414	1199	1199
q9	5235	4985	4927	4927
q10	6875	2328	1907	1907
q11	472	272	265	265
q12	343	382	224	224
q13	17780	3650	3078	3078
q14	226	225	213	213
q15	516	467	472	467
q16	639	625	589	589
q17	576	888	330	330
q18	7172	6475	6544	6475
q19	2069	963	553	553
q20	308	315	186	186
q21	2798	2280	1956	1956
q22	352	338	303	303
Total cold run time: 104125 ms
Total hot run time: 32396 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5596	5617	5526	5526
q2	243	323	232	232
q3	2292	2671	2316	2316
q4	1383	1809	1338	1338
q5	4358	4746	4722	4722
q6	176	163	130	130
q7	2068	2007	1831	1831
q8	2591	2832	2749	2749
q9	7351	7333	7350	7333
q10	3038	3298	2737	2737
q11	568	509	485	485
q12	634	743	608	608
q13	3664	3883	3246	3246
q14	275	285	291	285
q15	518	475	477	475
q16	636	707	635	635
q17	1234	1757	1266	1266
q18	7676	7420	7317	7317
q19	840	1201	1074	1074
q20	1990	2035	1891	1891
q21	5758	5286	5042	5042
q22	643	630	572	572
Total cold run time: 53532 ms
Total hot run time: 51810 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192003 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8a653b95bf23fa397cb2a3fa4456c8db74fbfebe, data reload: false

query1	1329	968	949	949
query2	6250	2036	1979	1979
query3	10937	4518	4372	4372
query4	61485	29332	23064	23064
query5	5526	595	454	454
query6	441	206	177	177
query7	5568	511	293	293
query8	336	245	232	232
query9	8353	2680	2685	2680
query10	468	315	248	248
query11	17453	15050	15593	15050
query12	160	104	107	104
query13	1461	543	452	452
query14	11620	7234	7334	7234
query15	215	192	205	192
query16	7294	644	458	458
query17	1077	743	570	570
query18	1952	404	296	296
query19	198	177	151	151
query20	114	114	110	110
query21	213	123	106	106
query22	4358	4771	4435	4435
query23	33812	33367	33509	33367
query24	5539	2316	2298	2298
query25	523	456	388	388
query26	645	284	162	162
query27	1729	514	330	330
query28	4228	2523	2481	2481
query29	572	557	445	445
query30	216	195	157	157
query31	948	880	820	820
query32	68	58	56	56
query33	437	348	307	307
query34	762	869	518	518
query35	811	888	774	774
query36	976	1027	1002	1002
query37	125	102	77	77
query38	4328	4415	4173	4173
query39	1518	1599	1417	1417
query40	214	113	100	100
query41	51	53	48	48
query42	119	98	102	98
query43	518	534	480	480
query44	1336	845	848	845
query45	205	188	168	168
query46	913	1076	692	692
query47	1906	1895	1864	1864
query48	395	416	340	340
query49	705	491	410	410
query50	643	680	397	397
query51	4329	4308	4308	4308
query52	114	109	96	96
query53	235	269	189	189
query54	501	523	426	426
query55	87	81	85	81
query56	258	275	254	254
query57	1190	1228	1113	1113
query58	240	229	254	229
query59	3027	3161	3091	3091
query60	281	276	254	254
query61	115	113	114	113
query62	772	729	668	668
query63	220	185	186	185
query64	1253	1008	653	653
query65	3230	3145	3159	3145
query66	715	396	300	300
query67	15608	15553	15635	15553
query68	2873	780	572	572
query69	431	304	268	268
query70	1221	1092	1113	1092
query71	351	289	276	276
query72	5139	3775	3866	3775
query73	641	751	365	365
query74	9723	9214	8927	8927
query75	3124	3158	2681	2681
query76	1897	1179	791	791
query77	379	359	270	270
query78	10061	10168	9375	9375
query79	1544	831	597	597
query80	1513	513	436	436
query81	556	274	244	244
query82	878	154	124	124
query83	239	174	156	156
query84	286	101	72	72
query85	789	362	294	294
query86	406	316	307	307
query87	4414	4481	4503	4481
query88	3991	2189	2226	2189
query89	393	319	288	288
query90	1747	192	192	192
query91	135	148	108	108
query92	59	56	52	52
query93	1392	863	543	543
query94	670	413	291	291
query95	342	276	254	254
query96	499	611	286	286
query97	2793	2909	2729	2729
query98	224	203	194	194
query99	1281	1362	1296	1296
Total cold run time: 303216 ms
Total hot run time: 192003 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.17% (11066/26243)
Line Coverage: 32.43% (93315/287746)
Region Coverage: 31.57% (47805/151419)
Branch Coverage: 27.58% (24175/87666)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8a653b95bf23fa397cb2a3fa4456c8db74fbfebe_8a653b95bf23fa397cb2a3fa4456c8db74fbfebe/report/index.html

@doris-robot
Copy link

ClickBench: Total hot run time: 30.86 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8a653b95bf23fa397cb2a3fa4456c8db74fbfebe, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.05	0.05
query3	0.23	0.05	0.06
query4	1.64	0.08	0.08
query5	0.42	0.40	0.41
query6	1.18	0.66	0.66
query7	0.03	0.02	0.01
query8	0.06	0.05	0.04
query9	0.56	0.49	0.51
query10	0.56	0.56	0.55
query11	0.16	0.12	0.12
query12	0.16	0.13	0.14
query13	0.62	0.59	0.61
query14	2.84	2.89	2.75
query15	0.91	0.84	0.83
query16	0.36	0.38	0.38
query17	1.06	1.05	1.06
query18	0.19	0.20	0.19
query19	1.94	1.85	2.03
query20	0.02	0.02	0.01
query21	15.36	0.95	0.65
query22	0.78	0.78	0.72
query23	14.98	1.46	0.67
query24	2.19	0.34	0.22
query25	0.15	0.09	0.09
query26	0.28	0.19	0.18
query27	0.08	0.08	0.08
query28	13.40	1.34	0.55
query29	12.65	4.15	3.41
query30	0.25	0.08	0.05
query31	2.84	0.61	0.38
query32	3.23	0.58	0.48
query33	3.04	2.96	3.08
query34	16.40	5.11	4.56
query35	4.52	4.52	4.55
query36	0.62	0.48	0.49
query37	0.19	0.15	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.17	0.13	0.14
query41	0.10	0.06	0.05
query42	0.06	0.05	0.04
query43	0.06	0.05	0.04
Total cold run time: 104.61 s
Total hot run time: 30.86 s

@CalvinKirs
Copy link
Member

run external

@CalvinKirs
Copy link
Member

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31640 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit aec6645ece06680213062717ba89e00f08fcd80d, data reload: false

------ Round 1 ----------------------------------
q1	17642	5301	5078	5078
q2	2070	299	174	174
q3	10530	1262	759	759
q4	10235	1006	526	526
q5	7642	2407	2377	2377
q6	192	169	132	132
q7	904	776	600	600
q8	9306	1298	1131	1131
q9	4908	4659	4672	4659
q10	6829	2332	1875	1875
q11	464	270	254	254
q12	345	348	221	221
q13	17759	3705	3099	3099
q14	229	230	210	210
q15	527	479	470	470
q16	617	614	595	595
q17	575	880	349	349
q18	7107	6304	6188	6188
q19	2049	960	562	562
q20	311	321	190	190
q21	2786	2130	1888	1888
q22	374	323	303	303
Total cold run time: 103401 ms
Total hot run time: 31640 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5214	5093	5127	5093
q2	243	329	231	231
q3	2159	2693	2414	2414
q4	1426	1805	1355	1355
q5	4260	4201	4177	4177
q6	209	166	127	127
q7	1893	1808	1791	1791
q8	2633	2723	2551	2551
q9	7300	7156	7271	7156
q10	3023	3243	2805	2805
q11	568	515	488	488
q12	712	795	636	636
q13	3312	3977	3267	3267
q14	267	282	274	274
q15	530	466	464	464
q16	660	698	621	621
q17	1160	1600	1379	1379
q18	7623	7512	7331	7331
q19	826	839	870	839
q20	1979	2040	1873	1873
q21	5465	5116	4683	4683
q22	588	601	545	545
Total cold run time: 52050 ms
Total hot run time: 50100 ms

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gavinchou gavinchou Awaiting requested review from gavinchou gavinchou is a code owner

@dataroaring dataroaring Awaiting requested review from dataroaring dataroaring is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
dev/2.1.x-experimental dev/3.0.x-experimental
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@morningman @hello-stephen @doris-robot @CalvinKirs