Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[enhance](mtmv)MTMV support history partition #46569

Open
wants to merge 18 commits into
base: master
Choose a base branch
from
Open

[enhance](mtmv)MTMV support history partition #46569

wants to merge 18 commits into from

Conversation

zddr
Copy link
Contributor

@zddr zddr commented Jan 7, 2025 •
edited
Loading

What problem does this PR solve?

wneh base table drop some partition, MTMV can not drop these partition by set partition_sync_limit
Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
MTMV support history partition

Release note

MTMV support history partition
None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jan 7, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zddr
Copy link
Contributor Author

zddr commented Jan 7, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33297 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1403833296612bd0b78066210e4af1d4cd225403, data reload: false

------ Round 1 ----------------------------------
q1	17603	6323	6331	6323
q2	2047	326	176	176
q3	10405	1299	780	780
q4	10224	893	445	445
q5	7877	2214	2025	2025
q6	209	176	146	146
q7	886	773	601	601
q8	9240	1407	1186	1186
q9	5351	4982	5040	4982
q10	6765	2331	1856	1856
q11	473	283	275	275
q12	356	360	225	225
q13	17761	3673	3131	3131
q14	229	250	215	215
q15	566	519	496	496
q16	654	636	579	579
q17	558	864	327	327
q18	6985	6524	6434	6434
q19	2012	972	572	572
q20	305	317	189	189
q21	2923	2203	2023	2023
q22	366	343	311	311
Total cold run time: 103795 ms
Total hot run time: 33297 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6317	6270	6267	6267
q2	236	325	233	233
q3	2264	2636	2308	2308
q4	1433	1849	1392	1392
q5	4379	4869	4838	4838
q6	190	182	143	143
q7	2087	1940	1816	1816
q8	2665	2846	2710	2710
q9	7410	7331	7346	7331
q10	3052	3358	2875	2875
q11	578	522	507	507
q12	647	789	627	627
q13	3526	3922	3175	3175
q14	310	309	288	288
q15	570	520	507	507
q16	647	698	669	669
q17	1229	1751	1260	1260
q18	7787	7576	7411	7411
q19	852	1198	1084	1084
q20	2018	2037	1936	1936
q21	5634	5133	4915	4915
q22	641	609	581	581
Total cold run time: 54472 ms
Total hot run time: 52873 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 198726 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1403833296612bd0b78066210e4af1d4cd225403, data reload: false

query1	1298	967	932	932
query2	6403	2452	2244	2244
query3	11089	4704	4997	4704
query4	33223	23794	23516	23516
query5	4389	643	472	472
query6	291	215	184	184
query7	3976	517	322	322
query8	303	250	242	242
query9	9655	2835	2801	2801
query10	477	311	259	259
query11	18011	15407	15011	15011
query12	153	112	103	103
query13	1562	553	410	410
query14	10111	7804	7707	7707
query15	247	211	196	196
query16	7792	655	514	514
query17	1573	806	612	612
query18	1784	427	315	315
query19	234	191	183	183
query20	134	118	111	111
query21	216	134	118	118
query22	4479	4597	4470	4470
query23	35679	34395	33847	33847
query24	6372	2324	2374	2324
query25	461	475	403	403
query26	1139	292	162	162
query27	2419	481	358	358
query28	5700	2566	2563	2563
query29	664	581	430	430
query30	173	191	155	155
query31	1023	969	849	849
query32	68	62	58	58
query33	499	357	307	307
query34	814	856	526	526
query35	849	856	775	775
query36	1062	1045	987	987
query37	116	107	81	81
query38	4408	4405	4444	4405
query39	1505	1469	1490	1469
query40	209	123	109	109
query41	47	45	45	45
query42	122	109	106	106
query43	528	544	497	497
query44	1385	844	850	844
query45	193	177	176	176
query46	926	1109	695	695
query47	1987	2020	1934	1934
query48	410	439	333	333
query49	740	511	437	437
query50	675	687	413	413
query51	7198	7140	6981	6981
query52	108	104	97	97
query53	233	279	191	191
query54	494	518	425	425
query55	82	87	86	86
query56	281	270	252	252
query57	1256	1244	1204	1204
query58	252	254	247	247
query59	3327	3426	3134	3134
query60	277	269	271	269
query61	147	107	112	107
query62	868	819	783	783
query63	257	202	209	202
query64	3916	1059	667	667
query65	3413	3264	3289	3264
query66	787	419	315	315
query67	16771	15938	15601	15601
query68	8314	727	538	538
query69	470	294	262	262
query70	1256	1129	1141	1129
query71	445	284	268	268
query72	6503	3934	3922	3922
query73	670	800	369	369
query74	10096	9365	8852	8852
query75	3949	3176	2724	2724
query76	3775	1205	815	815
query77	765	367	281	281
query78	10211	10337	9565	9565
query79	5080	824	600	600
query80	741	567	444	444
query81	474	277	235	235
query82	663	159	118	118
query83	192	163	147	147
query84	274	94	76	76
query85	751	366	313	313
query86	355	322	342	322
query87	4763	4453	4510	4453
query88	3695	2279	2245	2245
query89	479	337	297	297
query90	1876	191	193	191
query91	134	133	106	106
query92	62	57	55	55
query93	2782	871	561	561
query94	706	404	291	291
query95	339	278	257	257
query96	499	624	284	284
query97	2888	2945	2839	2839
query98	239	214	191	191
query99	1722	1554	1437	1437
Total cold run time: 302408 ms
Total hot run time: 198726 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.79 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1403833296612bd0b78066210e4af1d4cd225403, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.04
query3	0.23	0.06	0.06
query4	1.62	0.10	0.10
query5	0.43	0.42	0.42
query6	1.15	0.66	0.67
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.57	0.51	0.51
query10	0.56	0.56	0.54
query11	0.15	0.10	0.11
query12	0.14	0.12	0.11
query13	0.60	0.59	0.59
query14	2.74	2.74	2.80
query15	0.91	0.83	0.83
query16	0.38	0.39	0.38
query17	1.06	1.06	1.05
query18	0.23	0.21	0.21
query19	1.96	1.86	2.02
query20	0.01	0.02	0.01
query21	15.37	0.91	0.57
query22	0.76	0.78	0.63
query23	15.35	1.39	0.54
query24	2.85	1.30	1.69
query25	0.14	0.18	0.14
query26	0.27	0.15	0.14
query27	0.04	0.07	0.05
query28	14.43	1.50	1.05
query29	12.55	3.92	3.28
query30	0.25	0.09	0.06
query31	2.81	0.61	0.37
query32	3.22	0.54	0.46
query33	3.08	3.14	3.09
query34	16.62	5.09	4.58
query35	4.54	4.49	4.56
query36	0.66	0.48	0.47
query37	0.10	0.06	0.06
query38	0.04	0.03	0.04
query39	0.04	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 106.32 s
Total hot run time: 31.79 s

@zddr
Copy link
Contributor Author

zddr commented Jan 8, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32729 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4d25b767808c7c32b18e58da301efbd3f90c8599, data reload: false

------ Round 1 ----------------------------------
q1	17806	6165	6086	6086
q2	2047	287	163	163
q3	10542	1287	748	748
q4	10229	860	430	430
q5	7734	2179	1990	1990
q6	210	182	152	152
q7	909	745	623	623
q8	9222	1364	1154	1154
q9	5278	4916	4897	4897
q10	6770	2300	1881	1881
q11	474	275	269	269
q12	341	364	225	225
q13	17798	3669	3104	3104
q14	234	234	209	209
q15	562	521	502	502
q16	625	622	583	583
q17	557	868	321	321
q18	7114	6419	6364	6364
q19	1798	964	564	564
q20	309	320	188	188
q21	2795	2161	1966	1966
q22	357	331	310	310
Total cold run time: 103711 ms
Total hot run time: 32729 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6321	6248	6263	6248
q2	243	326	234	234
q3	2277	2688	2295	2295
q4	1441	1858	1421	1421
q5	4350	4780	4900	4780
q6	191	180	143	143
q7	2134	2000	1787	1787
q8	2628	2802	2697	2697
q9	7367	7181	7267	7181
q10	3044	3311	2814	2814
q11	587	537	501	501
q12	671	741	602	602
q13	3502	3898	3132	3132
q14	282	300	284	284
q15	571	511	511	511
q16	659	693	648	648
q17	1210	1748	1252	1252
q18	7649	7590	7263	7263
q19	837	1207	1082	1082
q20	2032	1997	1899	1899
q21	5744	5238	5066	5066
q22	612	629	544	544
Total cold run time: 54352 ms
Total hot run time: 52384 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187524 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4d25b767808c7c32b18e58da301efbd3f90c8599, data reload: false

query1	975	371	378	371
query2	6506	2375	2366	2366
query3	6712	215	217	215
query4	33898	23524	23354	23354
query5	4379	629	465	465
query6	271	193	173	173
query7	4638	485	305	305
query8	293	244	232	232
query9	9556	2767	2740	2740
query10	456	308	266	266
query11	18145	15309	14871	14871
query12	162	108	104	104
query13	1670	533	403	403
query14	9731	6886	7143	6886
query15	223	192	186	186
query16	8104	609	428	428
query17	1572	729	558	558
query18	2044	386	289	289
query19	207	165	152	152
query20	141	108	108	108
query21	208	118	105	105
query22	4232	4336	4082	4082
query23	33823	32815	33002	32815
query24	6398	2215	2286	2215
query25	478	449	382	382
query26	1195	268	150	150
query27	1994	461	322	322
query28	5422	2455	2450	2450
query29	721	543	418	418
query30	231	184	154	154
query31	960	882	816	816
query32	86	65	60	60
query33	522	358	290	290
query34	767	816	519	519
query35	792	814	759	759
query36	970	1048	937	937
query37	116	98	81	81
query38	3985	3995	3933	3933
query39	1497	1420	1392	1392
query40	203	108	101	101
query41	45	43	43	43
query42	118	102	97	97
query43	508	520	496	496
query44	1311	833	809	809
query45	183	171	167	167
query46	871	1022	646	646
query47	1892	1818	1741	1741
query48	385	403	327	327
query49	776	464	389	389
query50	620	631	382	382
query51	6986	6959	6838	6838
query52	98	96	91	91
query53	225	252	184	184
query54	481	521	404	404
query55	81	80	78	78
query56	258	271	245	245
query57	1157	1162	1107	1107
query58	248	222	234	222
query59	3098	3227	2974	2974
query60	270	255	253	253
query61	113	113	126	113
query62	878	751	731	731
query63	223	195	187	187
query64	4637	988	642	642
query65	3262	3200	3154	3154
query66	1069	408	300	300
query67	15816	15636	15422	15422
query68	7449	703	512	512
query69	458	304	270	270
query70	1201	1170	1054	1054
query71	428	281	257	257
query72	6166	3828	3920	3828
query73	635	767	360	360
query74	10118	9088	8746	8746
query75	3944	3143	2629	2629
query76	3558	1155	768	768
query77	767	359	262	262
query78	9779	9879	9289	9289
query79	2741	786	597	597
query80	666	521	430	430
query81	493	278	227	227
query82	504	154	131	131
query83	165	162	146	146
query84	241	87	70	70
query85	788	350	322	322
query86	398	327	310	310
query87	4477	4504	4259	4259
query88	5069	2205	2178	2178
query89	418	329	301	301
query90	1825	187	191	187
query91	129	138	104	104
query92	68	55	50	50
query93	1910	892	532	532
query94	663	386	294	294
query95	343	260	251	251
query96	483	616	287	287
query97	2864	2946	2794	2794
query98	213	203	204	203
query99	1411	1442	1381	1381
Total cold run time: 290264 ms
Total hot run time: 187524 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.37 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4d25b767808c7c32b18e58da301efbd3f90c8599, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.04	0.03
query3	0.24	0.07	0.07
query4	1.60	0.11	0.10
query5	0.43	0.43	0.42
query6	1.20	0.65	0.65
query7	0.03	0.01	0.02
query8	0.04	0.03	0.03
query9	0.60	0.51	0.50
query10	0.56	0.56	0.56
query11	0.15	0.10	0.10
query12	0.14	0.11	0.10
query13	0.61	0.60	0.60
query14	2.84	2.84	2.71
query15	0.90	0.85	0.84
query16	0.37	0.38	0.39
query17	1.00	1.03	1.07
query18	0.22	0.21	0.21
query19	1.93	1.85	2.02
query20	0.01	0.01	0.01
query21	15.37	0.91	0.56
query22	0.74	0.80	0.68
query23	15.23	1.46	0.50
query24	3.27	0.98	1.99
query25	0.22	0.20	0.13
query26	0.18	0.14	0.14
query27	0.07	0.06	0.05
query28	14.52	1.49	1.05
query29	12.55	3.82	3.20
query30	0.25	0.09	0.06
query31	2.84	0.59	0.38
query32	3.23	0.55	0.46
query33	3.12	3.05	3.18
query34	16.85	5.12	4.55
query35	4.54	4.53	4.54
query36	0.64	0.51	0.50
query37	0.09	0.06	0.06
query38	0.04	0.05	0.03
query39	0.03	0.03	0.02
query40	0.16	0.14	0.13
query41	0.09	0.03	0.03
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 107.07 s
Total hot run time: 31.37 s

@zddr
Copy link
Contributor Author

zddr commented Jan 8, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32468 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1141188642e03d805b4de91dcd8b7149727e4f5e, data reload: false

------ Round 1 ----------------------------------
q1	17582	6146	6015	6015
q2	2053	300	167	167
q3	10422	1276	700	700
q4	10211	850	438	438
q5	7512	2139	1906	1906
q6	209	176	143	143
q7	882	738	628	628
q8	9236	1335	1139	1139
q9	5216	4828	4863	4828
q10	6741	2280	1832	1832
q11	474	270	266	266
q12	345	360	215	215
q13	17746	3658	3084	3084
q14	225	240	223	223
q15	565	501	497	497
q16	622	616	602	602
q17	553	836	315	315
q18	7110	6464	6496	6464
q19	1242	968	541	541
q20	304	323	188	188
q21	2973	2162	1965	1965
q22	370	325	312	312
Total cold run time: 102593 ms
Total hot run time: 32468 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6239	6232	6205	6205
q2	241	325	230	230
q3	2235	2635	2266	2266
q4	1407	1812	1377	1377
q5	4271	4720	4631	4631
q6	187	181	141	141
q7	2138	1961	1860	1860
q8	2556	2766	2645	2645
q9	7223	7216	7134	7134
q10	3012	3345	2784	2784
q11	601	518	501	501
q12	630	767	571	571
q13	3468	3834	3211	3211
q14	289	305	289	289
q15	567	533	505	505
q16	641	686	636	636
q17	1183	1705	1251	1251
q18	7741	7475	7111	7111
q19	783	1112	1036	1036
q20	1876	1964	1818	1818
q21	5341	5090	4777	4777
q22	580	613	588	588
Total cold run time: 53209 ms
Total hot run time: 51567 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189034 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1141188642e03d805b4de91dcd8b7149727e4f5e, data reload: false

query1	969	378	393	378
query2	6523	2384	2452	2384
query3	6719	219	214	214
query4	33600	23345	23535	23345
query5	4345	641	466	466
query6	285	205	181	181
query7	4629	495	294	294
query8	300	246	231	231
query9	9475	2725	2731	2725
query10	474	312	259	259
query11	17790	15251	15136	15136
query12	159	110	108	108
query13	1698	537	408	408
query14	10642	7470	7062	7062
query15	254	196	188	188
query16	8594	618	468	468
query17	1605	707	540	540
query18	2070	391	289	289
query19	206	176	152	152
query20	118	111	110	110
query21	204	119	103	103
query22	4342	4504	4285	4285
query23	34093	32984	33101	32984
query24	6345	2266	2233	2233
query25	498	439	377	377
query26	1181	277	158	158
query27	2019	459	329	329
query28	5319	2487	2426	2426
query29	708	528	408	408
query30	227	185	151	151
query31	941	872	801	801
query32	82	61	57	57
query33	533	348	280	280
query34	759	829	505	505
query35	814	791	713	713
query36	1013	1052	964	964
query37	122	99	80	80
query38	4167	4159	4079	4079
query39	1440	1421	1419	1419
query40	210	112	100	100
query41	46	45	47	45
query42	119	101	103	101
query43	513	523	493	493
query44	1285	826	811	811
query45	171	165	166	165
query46	853	1035	640	640
query47	1816	1853	1795	1795
query48	379	407	319	319
query49	779	497	410	410
query50	619	638	382	382
query51	7056	6905	6843	6843
query52	104	103	97	97
query53	220	245	181	181
query54	473	485	388	388
query55	80	83	80	80
query56	265	262	246	246
query57	1218	1143	1093	1093
query58	239	224	226	224
query59	3241	3249	2972	2972
query60	265	269	264	264
query61	111	108	110	108
query62	825	757	672	672
query63	224	194	193	193
query64	4094	1080	720	720
query65	3241	3167	3172	3167
query66	1075	404	309	309
query67	15771	15688	15552	15552
query68	8685	685	510	510
query69	458	282	246	246
query70	1217	1121	1120	1120
query71	455	279	261	261
query72	6220	3782	3774	3774
query73	646	741	441	441
query74	9457	9091	9009	9009
query75	4384	3145	2639	2639
query76	3942	1180	754	754
query77	774	361	316	316
query78	9971	10037	9272	9272
query79	3191	777	592	592
query80	623	503	432	432
query81	461	263	225	225
query82	629	151	118	118
query83	159	164	147	147
query84	232	84	69	69
query85	778	363	302	302
query86	366	314	292	292
query87	4299	4285	4428	4285
query88	4584	2185	2170	2170
query89	403	332	295	295
query90	1936	183	183	183
query91	130	131	108	108
query92	66	56	56	56
query93	1560	859	523	523
query94	659	373	286	286
query95	331	256	251	251
query96	489	613	279	279
query97	2830	2907	2822	2822
query98	224	200	193	193
query99	1691	1505	1404	1404
Total cold run time: 292404 ms
Total hot run time: 189034 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.04 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1141188642e03d805b4de91dcd8b7149727e4f5e, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.03
query3	0.24	0.07	0.07
query4	1.62	0.10	0.10
query5	0.42	0.40	0.43
query6	1.15	0.64	0.65
query7	0.02	0.02	0.01
query8	0.04	0.02	0.03
query9	0.58	0.50	0.50
query10	0.55	0.57	0.54
query11	0.15	0.10	0.10
query12	0.13	0.10	0.11
query13	0.60	0.61	0.60
query14	2.81	2.71	2.73
query15	0.90	0.82	0.80
query16	0.37	0.38	0.38
query17	1.02	1.09	1.05
query18	0.22	0.21	0.21
query19	1.98	1.89	1.99
query20	0.02	0.02	0.01
query21	15.36	0.87	0.58
query22	0.78	0.72	0.61
query23	15.37	1.42	0.61
query24	2.95	0.79	0.92
query25	0.23	0.14	0.13
query26	0.32	0.16	0.12
query27	0.06	0.05	0.04
query28	13.59	1.51	1.05
query29	12.54	3.96	3.31
query30	0.25	0.09	0.07
query31	2.81	0.60	0.38
query32	3.22	0.54	0.47
query33	3.11	3.09	3.08
query34	16.62	5.06	4.42
query35	4.47	4.41	4.51
query36	0.66	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.12
query41	0.09	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 105.78 s
Total hot run time: 31.04 s

@zddr
Copy link
Contributor Author

zddr commented Jan 9, 2025

run external

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@zddr @Thearas @doris-robot