Skip to content

[draft][feature](filecache)normal queue cold hot seperation #57302

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 17 commits into
base: master
Choose a base branch
from
Open

[draft][feature](filecache)normal queue cold hot seperation #57302

wants to merge 17 commits into from
+479 −35

Conversation

@xuchenhao
Copy link
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Oct 24, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@xuchenhao
Copy link
Contributor Author

run buildall

@xuchenhao
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-DS: Total hot run time: 190046 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3581305573a6c06f92af9e5e3c91534e9910d0bf, data reload: false

query1	1090	422	398	398
query2	6567	1731	1707	1707
query3	6755	224	228	224
query4	26434	23843	23183	23183
query5	4912	646	482	482
query6	357	254	235	235
query7	4667	510	302	302
query8	323	294	264	264
query9	8737	2608	2620	2608
query10	523	349	299	299
query11	15912	15075	15018	15018
query12	200	125	118	118
query13	1723	629	446	446
query14	12808	9394	9336	9336
query15	276	193	186	186
query16	7901	689	560	560
query17	1663	857	738	738
query18	2155	443	346	346
query19	318	223	186	186
query20	144	134	150	134
query21	229	152	126	126
query22	4760	4796	4424	4424
query23	34913	33734	33646	33646
query24	8277	2543	2535	2535
query25	625	517	458	458
query26	1391	281	166	166
query27	2857	533	364	364
query28	4459	2214	2212	2212
query29	795	671	509	509
query30	318	239	201	201
query31	943	871	782	782
query32	86	73	71	71
query33	610	409	349	349
query34	817	887	530	530
query35	851	839	781	781
query36	949	1018	928	928
query37	141	136	103	103
query38	3759	3665	3557	3557
query39	1464	1414	1417	1414
query40	221	129	115	115
query41	64	59	58	58
query42	128	110	122	110
query43	514	516	473	473
query44	1212	747	744	744
query45	189	190	173	173
query46	894	986	636	636
query47	1797	1831	1710	1710
query48	436	420	330	330
query49	777	508	418	418
query50	646	677	406	406
query51	3959	3926	3911	3911
query52	110	107	97	97
query53	240	277	196	196
query54	600	610	524	524
query55	91	100	91	91
query56	344	344	353	344
query57	1185	1193	1123	1123
query58	303	286	272	272
query59	2549	2634	2464	2464
query60	362	334	338	334
query61	163	160	157	157
query62	798	744	661	661
query63	233	197	189	189
query64	4421	1196	857	857
query65	4035	3979	3961	3961
query66	1078	423	332	332
query67	15535	15276	15034	15034
query68	9226	872	595	595
query69	464	387	293	293
query70	1347	1274	1236	1236
query71	494	345	315	315
query72	5743	4963	5020	4963
query73	684	590	359	359
query74	9083	8851	8631	8631
query75	4261	3521	2800	2800
query76	4002	1165	753	753
query77	968	400	332	332
query78	9668	9841	8956	8956
query79	1959	863	588	588
query80	677	587	508	508
query81	498	268	236	236
query82	459	169	133	133
query83	299	277	269	269
query84	310	115	92	92
query85	964	472	426	426
query86	333	309	293	293
query87	3791	3756	3630	3630
query88	3046	2279	2266	2266
query89	387	331	295	295
query90	1951	215	226	215
query91	165	167	141	141
query92	84	74	68	68
query93	1146	1016	644	644
query94	694	439	338	338
query95	413	325	336	325
query96	508	571	287	287
query97	2989	2971	2874	2874
query98	235	230	210	210
query99	1474	1431	1329	1329
Total cold run time: 282915 ms
Total hot run time: 190046 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 27.44 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3581305573a6c06f92af9e5e3c91534e9910d0bf, data reload: false

query1	0.06	0.04	0.05
query2	0.09	0.06	0.05
query3	0.27	0.09	0.08
query4	1.62	0.12	0.12
query5	0.30	0.27	0.25
query6	1.18	0.64	0.64
query7	0.03	0.03	0.03
query8	0.05	0.04	0.04
query9	0.64	0.53	0.52
query10	0.59	0.58	0.57
query11	0.16	0.11	0.11
query12	0.16	0.15	0.12
query13	0.62	0.60	0.60
query14	1.01	1.02	1.00
query15	0.85	0.85	0.84
query16	0.40	0.38	0.39
query17	1.02	1.03	1.05
query18	0.21	0.20	0.19
query19	1.87	1.80	1.78
query20	0.02	0.01	0.01
query21	15.48	0.19	0.13
query22	5.09	0.08	0.06
query23	15.65	0.28	0.10
query24	2.90	0.85	0.32
query25	0.07	0.07	0.06
query26	0.14	0.14	0.15
query27	0.07	0.06	0.05
query28	4.34	1.16	0.93
query29	12.59	3.98	3.24
query30	0.28	0.15	0.12
query31	2.82	0.58	0.38
query32	3.23	0.55	0.48
query33	3.09	3.09	3.02
query34	15.76	5.16	4.55
query35	4.59	4.58	4.61
query36	0.69	0.51	0.51
query37	0.13	0.07	0.07
query38	0.06	0.04	0.03
query39	0.04	0.03	0.03
query40	0.17	0.14	0.14
query41	0.09	0.04	0.03
query42	0.04	0.04	0.03
query43	0.05	0.04	0.04
Total cold run time: 98.52 s
Total hot run time: 27.44 s

@xuchenhao
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

ClickBench: Total hot run time: 28.01 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4105506cc7752c376acdbb17e2dc62c23e9f6be5, data reload: false

query1	0.06	0.05	0.05
query2	0.09	0.06	0.05
query3	0.25	0.09	0.09
query4	1.61	0.12	0.12
query5	0.28	0.27	0.25
query6	1.19	0.65	0.67
query7	0.03	0.02	0.02
query8	0.06	0.05	0.04
query9	0.63	0.54	0.52
query10	0.59	0.60	0.58
query11	0.18	0.12	0.12
query12	0.16	0.12	0.13
query13	0.63	0.62	0.60
query14	1.01	1.02	1.00
query15	0.88	0.85	0.87
query16	0.41	0.39	0.41
query17	1.07	1.07	1.01
query18	0.22	0.21	0.22
query19	1.90	1.87	1.81
query20	0.02	0.01	0.02
query21	15.42	0.21	0.14
query22	5.03	0.08	0.05
query23	15.68	0.27	0.11
query24	2.98	0.55	0.53
query25	0.08	0.09	0.07
query26	0.15	0.13	0.13
query27	0.07	0.05	0.05
query28	4.89	1.18	0.94
query29	12.59	4.18	3.35
query30	0.31	0.16	0.11
query31	2.82	0.60	0.38
query32	3.25	0.56	0.49
query33	3.04	3.11	3.14
query34	15.96	5.16	4.57
query35	4.63	4.61	4.63
query36	0.69	0.51	0.50
query37	0.10	0.07	0.07
query38	0.08	0.05	0.04
query39	0.04	0.03	0.03
query40	0.17	0.14	0.14
query41	0.09	0.03	0.03
query42	0.05	0.03	0.03
query43	0.06	0.04	0.04
Total cold run time: 99.45 s
Total hot run time: 28.01 s

@xuchenhao
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

ClickBench: Total hot run time: 27.76 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a313c66861569f763c13407451807fcd7601eb5b, data reload: false

query1	0.06	0.06	0.05
query2	0.09	0.06	0.05
query3	0.26	0.08	0.09
query4	1.60	0.12	0.11
query5	0.28	0.26	0.25
query6	1.18	0.67	0.64
query7	0.03	0.05	0.03
query8	0.05	0.04	0.04
query9	0.63	0.52	0.51
query10	0.58	0.58	0.57
query11	0.16	0.12	0.12
query12	0.16	0.14	0.13
query13	0.63	0.60	0.61
query14	1.01	1.03	1.02
query15	0.86	0.86	0.87
query16	0.39	0.39	0.41
query17	1.03	1.06	1.04
query18	0.21	0.20	0.20
query19	1.95	1.89	1.82
query20	0.02	0.01	0.01
query21	15.43	0.18	0.13
query22	5.19	0.08	0.05
query23	15.65	0.27	0.11
query24	2.66	1.16	0.45
query25	0.07	0.06	0.06
query26	0.14	0.12	0.14
query27	0.07	0.05	0.06
query28	4.53	1.15	0.96
query29	12.55	3.97	3.29
query30	0.29	0.15	0.12
query31	2.82	0.61	0.39
query32	3.24	0.56	0.48
query33	3.11	3.04	3.08
query34	15.95	5.19	4.56
query35	4.60	4.57	4.62
query36	0.67	0.52	0.49
query37	0.10	0.07	0.07
query38	0.06	0.05	0.04
query39	0.04	0.03	0.02
query40	0.17	0.14	0.15
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 98.7 s
Total hot run time: 27.76 s

@xuchenhao
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-DS: Total hot run time: 190770 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 736a4751f1cb8f4d9b1462bedc9f415e4bff5764, data reload: false

query1	1087	428	406	406
query2	6557	1728	1695	1695
query3	6757	231	220	220
query4	26704	23944	23019	23019
query5	4973	679	464	464
query6	355	265	239	239
query7	4658	513	307	307
query8	332	273	276	273
query9	8761	2617	2628	2617
query10	551	349	291	291
query11	16138	15047	14889	14889
query12	210	126	119	119
query13	1668	566	437	437
query14	12760	9336	9400	9336
query15	270	198	176	176
query16	7742	669	511	511
query17	1582	879	761	761
query18	2119	497	352	352
query19	293	233	198	198
query20	135	148	152	148
query21	238	141	125	125
query22	4738	4650	4564	4564
query23	34708	34067	33908	33908
query24	8324	2499	2508	2499
query25	625	574	493	493
query26	1278	297	172	172
query27	2835	527	401	401
query28	4583	2271	2258	2258
query29	873	692	575	575
query30	321	250	215	215
query31	976	872	855	855
query32	95	82	82	82
query33	617	413	371	371
query34	838	943	547	547
query35	865	901	831	831
query36	1006	1068	971	971
query37	140	116	99	99
query38	3632	3626	3458	3458
query39	1544	1402	1410	1402
query40	221	130	117	117
query41	60	59	58	58
query42	122	120	116	116
query43	487	509	453	453
query44	1256	744	754	744
query45	199	184	180	180
query46	895	987	650	650
query47	1736	1806	1743	1743
query48	402	447	314	314
query49	750	492	419	419
query50	685	702	420	420
query51	3872	3916	3853	3853
query52	110	110	109	109
query53	239	272	193	193
query54	614	586	538	538
query55	94	81	88	81
query56	335	318	302	302
query57	1178	1213	1113	1113
query58	302	280	273	273
query59	2690	2670	2595	2595
query60	366	354	333	333
query61	171	157	161	157
query62	817	732	672	672
query63	235	196	199	196
query64	4463	1194	903	903
query65	4052	3940	3992	3940
query66	1094	441	330	330
query67	15631	15346	15050	15050
query68	8474	957	596	596
query69	493	334	287	287
query70	1354	1245	1332	1245
query71	500	339	312	312
query72	6072	4959	4919	4919
query73	709	579	367	367
query74	9369	8841	8968	8841
query75	4033	3337	2810	2810
query76	3745	1161	731	731
query77	813	415	319	319
query78	9551	9734	9055	9055
query79	1990	836	593	593
query80	637	563	491	491
query81	494	261	231	231
query82	446	167	133	133
query83	267	270	254	254
query84	255	115	98	98
query85	910	497	441	441
query86	348	310	319	310
query87	3712	3767	3658	3658
query88	3562	2249	2219	2219
query89	404	327	295	295
query90	1995	218	219	218
query91	172	173	140	140
query92	84	68	71	68
query93	1561	964	646	646
query94	711	463	362	362
query95	403	336	314	314
query96	489	581	283	283
query97	2927	2977	2893	2893
query98	243	218	209	209
query99	1498	1418	1296	1296
Total cold run time: 283185 ms
Total hot run time: 190770 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.11 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 736a4751f1cb8f4d9b1462bedc9f415e4bff5764, data reload: false

query1	0.06	0.04	0.05
query2	0.08	0.06	0.05
query3	0.26	0.08	0.08
query4	1.62	0.11	0.12
query5	0.29	0.27	0.25
query6	1.19	0.66	0.63
query7	0.03	0.03	0.03
query8	0.06	0.05	0.04
query9	0.64	0.52	0.51
query10	0.58	0.63	0.60
query11	0.16	0.11	0.11
query12	0.16	0.12	0.12
query13	0.61	0.60	0.60
query14	1.00	1.02	1.00
query15	0.86	0.83	0.87
query16	0.40	0.39	0.40
query17	1.05	1.07	1.01
query18	0.22	0.20	0.20
query19	1.96	1.84	1.82
query20	0.02	0.01	0.02
query21	15.45	0.18	0.13
query22	5.18	0.06	0.05
query23	15.66	0.26	0.10
query24	2.59	1.35	0.90
query25	0.07	0.06	0.06
query26	0.14	0.14	0.14
query27	0.06	0.05	0.05
query28	4.40	1.13	0.95
query29	12.65	3.88	3.28
query30	0.30	0.15	0.10
query31	2.82	0.59	0.38
query32	3.24	0.55	0.48
query33	3.00	3.11	3.08
query34	15.83	5.13	4.50
query35	4.58	4.63	4.61
query36	0.70	0.52	0.49
query37	0.11	0.07	0.07
query38	0.06	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.15	0.15
query41	0.08	0.03	0.03
query42	0.04	0.04	0.03
query43	0.04	0.05	0.03
Total cold run time: 98.47 s
Total hot run time: 28.11 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou Awaiting requested review from gavinchou gavinchou is a code owner

@dataroaring dataroaring Awaiting requested review from dataroaring dataroaring is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@xuchenhao @Thearas @doris-robot