Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature](sql-dialect)support convert hive view and presto view use sql convertor service #46308

Merged
merged 11 commits into from
Jan 24, 2025

Conversation

liujiwen-up
Copy link
Contributor

@liujiwen-up liujiwen-up commented Jan 2, 2025

What problem does this PR solve?

Problem Summary:

  1. Supports presto view query
  2. Supports hive view and presto view rewriting based on dialect service
  3. Migrated dialect_compatible case to external_table_p0

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jan 2, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@liujiwen-up
Copy link
Contributor Author

run buildall

@liujiwen-up
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32503 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1b4559222ececa5d12fabc53aa7171b6ddedadbc, data reload: false

------ Round 1 ----------------------------------
q1	17580	6052	6005	6005
q2	2043	308	168	168
q3	10420	1312	760	760
q4	10270	869	454	454
q5	8293	2242	2059	2059
q6	207	182	147	147
q7	894	774	595	595
q8	9226	1334	1130	1130
q9	5306	4949	4895	4895
q10	6786	2330	1856	1856
q11	487	280	253	253
q12	345	353	219	219
q13	17756	3708	3031	3031
q14	246	228	216	216
q15	585	514	483	483
q16	622	635	588	588
q17	558	843	309	309
q18	6907	6472	6307	6307
q19	1951	968	558	558
q20	303	312	192	192
q21	3081	2201	1970	1970
q22	355	338	308	308
Total cold run time: 104221 ms
Total hot run time: 32503 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6263	6170	6225	6170
q2	232	322	235	235
q3	2228	2660	2315	2315
q4	1397	1847	1339	1339
q5	4359	4756	4808	4756
q6	185	178	144	144
q7	2044	2018	1816	1816
q8	2632	2822	2722	2722
q9	7380	7292	7304	7292
q10	3072	3328	2829	2829
q11	572	505	503	503
q12	614	736	566	566
q13	3538	3918	3253	3253
q14	294	301	285	285
q15	565	511	524	511
q16	646	678	657	657
q17	1223	1757	1269	1269
q18	7710	7615	7400	7400
q19	831	1121	1091	1091
q20	2017	2084	1942	1942
q21	5656	5252	4870	4870
q22	589	642	589	589
Total cold run time: 54047 ms
Total hot run time: 52554 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197395 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1b4559222ececa5d12fabc53aa7171b6ddedadbc, data reload: false

query1	1292	962	912	912
query2	6370	2338	2413	2338
query3	11131	4796	4845	4796
query4	33153	23700	23364	23364
query5	3864	631	457	457
query6	273	196	203	196
query7	3988	488	306	306
query8	315	257	245	245
query9	9513	2661	2667	2661
query10	452	306	265	265
query11	17843	15358	15225	15225
query12	154	107	104	104
query13	1590	529	398	398
query14	10131	6773	7381	6773
query15	339	219	198	198
query16	8580	639	455	455
query17	1543	778	575	575
query18	2178	408	310	310
query19	208	190	179	179
query20	118	117	114	114
query21	200	140	110	110
query22	4584	4571	4428	4428
query23	34392	33957	33864	33864
query24	6756	2368	2297	2297
query25	454	493	408	408
query26	714	271	153	153
query27	2144	439	352	352
query28	5295	2464	2480	2464
query29	518	550	431	431
query30	211	184	153	153
query31	1023	930	843	843
query32	97	57	58	57
query33	480	376	304	304
query34	766	861	502	502
query35	828	836	744	744
query36	1053	1051	978	978
query37	116	96	82	82
query38	4211	4245	4150	4150
query39	1563	1476	1455	1455
query40	226	122	103	103
query41	46	44	42	42
query42	121	104	104	104
query43	523	546	526	526
query44	1346	828	843	828
query45	197	174	170	170
query46	881	1071	666	666
query47	2035	2014	1936	1936
query48	376	426	335	335
query49	719	486	397	397
query50	645	691	391	391
query51	7344	7391	7159	7159
query52	102	105	91	91
query53	228	265	194	194
query54	481	488	421	421
query55	88	79	81	79
query56	255	271	245	245
query57	1270	1273	1170	1170
query58	248	235	230	230
query59	3198	3394	3214	3214
query60	277	269	251	251
query61	110	107	129	107
query62	860	838	768	768
query63	233	193	201	193
query64	3417	1043	644	644
query65	3317	3285	3293	3285
query66	785	420	311	311
query67	16509	15798	15627	15627
query68	8594	694	519	519
query69	480	291	248	248
query70	1243	1179	1164	1164
query71	425	295	264	264
query72	6265	4028	4110	4028
query73	733	754	349	349
query74	10349	9250	9076	9076
query75	4045	3186	2670	2670
query76	3578	1194	770	770
query77	743	374	357	357
query78	10527	10323	9511	9511
query79	3707	822	577	577
query80	652	508	438	438
query81	497	281	237	237
query82	638	151	123	123
query83	172	164	144	144
query84	245	83	80	80
query85	797	352	308	308
query86	394	310	303	303
query87	4406	4439	4426	4426
query88	4831	2169	2161	2161
query89	419	335	291	291
query90	1798	187	184	184
query91	132	128	106	106
query92	69	53	52	52
query93	1961	829	529	529
query94	634	398	298	298
query95	335	301	256	256
query96	477	617	281	281
query97	2945	3032	2848	2848
query98	226	203	192	192
query99	1727	1591	1472	1472
Total cold run time: 298321 ms
Total hot run time: 197395 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.37 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1b4559222ececa5d12fabc53aa7171b6ddedadbc, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.03
query3	0.24	0.06	0.07
query4	1.62	0.10	0.10
query5	0.42	0.42	0.39
query6	1.15	0.65	0.65
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.58	0.51	0.50
query10	0.56	0.56	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.62	0.60
query14	2.83	2.86	2.92
query15	0.90	0.82	0.84
query16	0.39	0.40	0.37
query17	1.00	1.05	0.97
query18	0.22	0.20	0.22
query19	1.87	1.82	1.99
query20	0.01	0.02	0.01
query21	15.37	0.99	0.60
query22	0.75	0.75	0.63
query23	15.32	1.43	0.58
query24	3.14	1.60	0.87
query25	0.20	0.21	0.11
query26	0.28	0.14	0.13
query27	0.06	0.06	0.05
query28	13.94	1.52	1.06
query29	12.58	3.93	3.31
query30	0.25	0.10	0.07
query31	2.82	0.59	0.37
query32	3.23	0.53	0.46
query33	3.10	3.10	3.20
query34	16.81	5.09	4.50
query35	4.56	4.52	4.53
query36	0.64	0.48	0.47
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.02
query40	0.18	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.39 s
Total hot run time: 31.37 s

@liujiwen-up
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32621 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1b4559222ececa5d12fabc53aa7171b6ddedadbc, data reload: false

------ Round 1 ----------------------------------
q1	17573	6123	6402	6123
q2	2055	295	164	164
q3	10561	1273	721	721
q4	10194	866	429	429
q5	7517	2185	1948	1948
q6	209	182	148	148
q7	902	768	599	599
q8	9237	1335	1132	1132
q9	5228	4875	4824	4824
q10	6736	2312	1857	1857
q11	478	292	255	255
q12	337	350	224	224
q13	17820	3816	3304	3304
q14	231	234	223	223
q15	577	511	489	489
q16	646	619	602	602
q17	571	854	328	328
q18	6808	6385	6287	6287
q19	3015	966	543	543
q20	300	306	181	181
q21	2724	2117	1937	1937
q22	360	331	303	303
Total cold run time: 104079 ms
Total hot run time: 32621 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6395	6202	6253	6202
q2	235	321	241	241
q3	2266	2633	2368	2368
q4	1392	1835	1327	1327
q5	4322	4704	4866	4704
q6	197	187	149	149
q7	2093	1982	1812	1812
q8	2628	2820	2702	2702
q9	7322	7341	7311	7311
q10	3030	3363	2838	2838
q11	562	508	478	478
q12	705	798	625	625
q13	3455	3895	3278	3278
q14	294	334	302	302
q15	581	537	490	490
q16	640	684	652	652
q17	1214	1753	1257	1257
q18	7742	7395	7422	7395
q19	812	1096	1125	1096
q20	1981	2070	1982	1982
q21	5739	5249	4973	4973
q22	637	635	618	618
Total cold run time: 54242 ms
Total hot run time: 52800 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197038 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1b4559222ececa5d12fabc53aa7171b6ddedadbc, data reload: false

query1	1285	959	930	930
query2	6283	2390	2332	2332
query3	11063	4888	4886	4886
query4	33144	23931	23400	23400
query5	4766	603	475	475
query6	291	210	196	196
query7	3999	488	308	308
query8	305	243	262	243
query9	9354	2643	2663	2643
query10	447	304	256	256
query11	17907	15638	15134	15134
query12	173	112	113	112
query13	1740	547	414	414
query14	12420	7706	7453	7453
query15	243	207	196	196
query16	7838	643	492	492
query17	1514	767	597	597
query18	2059	419	324	324
query19	241	185	157	157
query20	132	113	119	113
query21	203	136	106	106
query22	4549	4733	4398	4398
query23	34393	34380	33593	33593
query24	6054	2281	2318	2281
query25	460	499	402	402
query26	839	284	155	155
query27	2020	459	328	328
query28	5373	2496	2454	2454
query29	552	552	418	418
query30	204	190	158	158
query31	1001	949	842	842
query32	91	57	61	57
query33	483	354	302	302
query34	774	830	514	514
query35	789	843	758	758
query36	1029	1068	969	969
query37	118	95	77	77
query38	4133	4301	4428	4301
query39	1535	1490	1480	1480
query40	215	126	113	113
query41	44	49	45	45
query42	123	106	114	106
query43	527	542	497	497
query44	1336	841	817	817
query45	178	180	177	177
query46	871	1055	660	660
query47	1991	2039	1969	1969
query48	383	413	321	321
query49	720	499	428	428
query50	648	667	378	378
query51	7076	7278	7059	7059
query52	99	96	91	91
query53	231	264	191	191
query54	472	488	406	406
query55	83	81	81	81
query56	256	285	244	244
query57	1272	1249	1188	1188
query58	237	225	235	225
query59	3262	3258	3101	3101
query60	273	267	255	255
query61	117	108	111	108
query62	888	804	761	761
query63	226	195	185	185
query64	3137	1029	650	650
query65	3324	3208	3282	3208
query66	919	406	301	301
query67	16252	15917	15596	15596
query68	7515	686	514	514
query69	486	285	253	253
query70	1225	1145	1109	1109
query71	448	280	252	252
query72	6424	3798	3921	3798
query73	647	742	356	356
query74	10218	8955	8958	8955
query75	4195	3147	2664	2664
query76	3736	1169	774	774
query77	861	353	266	266
query78	10174	10084	9475	9475
query79	3637	821	585	585
query80	720	504	416	416
query81	497	271	226	226
query82	629	149	122	122
query83	185	166	148	148
query84	284	86	71	71
query85	770	356	300	300
query86	335	311	303	303
query87	4481	4476	4394	4394
query88	4046	2185	2181	2181
query89	424	327	303	303
query90	1867	220	183	183
query91	134	138	104	104
query92	66	55	51	51
query93	1443	848	525	525
query94	659	398	277	277
query95	341	264	242	242
query96	476	610	276	276
query97	2889	2924	2843	2843
query98	213	201	198	198
query99	1719	1577	1459	1459
Total cold run time: 296708 ms
Total hot run time: 197038 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.36 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1b4559222ececa5d12fabc53aa7171b6ddedadbc, data reload: false

query1	0.03	0.02	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.06
query4	1.62	0.10	0.10
query5	0.42	0.39	0.41
query6	1.16	0.66	0.64
query7	0.03	0.02	0.01
query8	0.04	0.03	0.03
query9	0.58	0.52	0.50
query10	0.55	0.56	0.54
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.60	0.59	0.60
query14	2.72	2.79	2.75
query15	0.89	0.84	0.82
query16	0.39	0.38	0.39
query17	1.07	1.01	1.01
query18	0.23	0.21	0.21
query19	1.94	1.87	2.00
query20	0.02	0.01	0.01
query21	15.38	0.96	0.58
query22	0.74	0.76	0.64
query23	15.31	1.51	0.63
query24	3.32	1.53	0.96
query25	0.12	0.19	0.12
query26	0.34	0.17	0.13
query27	0.08	0.05	0.06
query28	13.87	1.48	1.04
query29	12.58	3.96	3.28
query30	0.25	0.09	0.06
query31	2.83	0.60	0.38
query32	3.22	0.54	0.45
query33	3.09	3.08	3.19
query34	16.79	5.13	4.48
query35	4.58	4.54	4.52
query36	0.63	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.04
Total cold run time: 106.52 s
Total hot run time: 31.36 s

@liujiwen-up
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33157 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6c8a8f97fd50d5ddfda4ef8ef7ae8b61f0cdab06, data reload: false

------ Round 1 ----------------------------------
q1	17576	6192	6153	6153
q2	2059	308	170	170
q3	10575	1274	749	749
q4	10212	888	440	440
q5	7567	2261	2034	2034
q6	207	185	148	148
q7	899	771	604	604
q8	9246	1504	1292	1292
q9	5433	5063	5041	5041
q10	6770	2302	1876	1876
q11	496	291	256	256
q12	337	363	223	223
q13	17763	3661	3101	3101
q14	228	238	208	208
q15	554	511	494	494
q16	631	626	604	604
q17	561	897	339	339
q18	6852	6425	6320	6320
q19	1843	992	587	587
q20	317	315	183	183
q21	2932	2244	2028	2028
q22	365	334	307	307
Total cold run time: 103423 ms
Total hot run time: 33157 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6477	6321	6315	6315
q2	247	337	243	243
q3	2282	2654	2357	2357
q4	1439	1839	1374	1374
q5	4367	4784	4904	4784
q6	200	184	141	141
q7	2081	1957	1787	1787
q8	2675	2875	2765	2765
q9	7300	7246	7299	7246
q10	3061	3351	2880	2880
q11	579	509	480	480
q12	649	763	621	621
q13	3482	3920	3253	3253
q14	289	332	281	281
q15	560	521	504	504
q16	660	683	670	670
q17	1240	1742	1267	1267
q18	7721	7506	7508	7506
q19	906	1334	1200	1200
q20	2039	2019	1849	1849
q21	5655	5271	4918	4918
q22	656	605	593	593
Total cold run time: 54565 ms
Total hot run time: 53034 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 198129 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6c8a8f97fd50d5ddfda4ef8ef7ae8b61f0cdab06, data reload: false

query1	1306	939	912	912
query2	6313	2267	2262	2262
query3	11094	4803	4927	4803
query4	33052	23785	23447	23447
query5	3975	622	458	458
query6	288	207	187	187
query7	3983	499	307	307
query8	303	247	244	244
query9	9410	2676	2660	2660
query10	477	317	262	262
query11	18171	15246	15269	15246
query12	163	110	102	102
query13	1780	542	408	408
query14	12601	8037	8120	8037
query15	258	204	188	188
query16	8486	608	475	475
query17	1617	764	569	569
query18	2185	411	302	302
query19	210	178	157	157
query20	120	131	113	113
query21	231	121	104	104
query22	4583	4521	4365	4365
query23	34248	33645	33740	33645
query24	5957	2316	2390	2316
query25	475	464	390	390
query26	756	270	159	159
query27	2049	467	352	352
query28	5131	2474	2459	2459
query29	643	562	451	451
query30	211	182	150	150
query31	995	936	848	848
query32	74	62	59	59
query33	521	359	292	292
query34	756	842	534	534
query35	858	835	749	749
query36	1043	1068	961	961
query37	117	102	78	78
query38	4182	4334	4167	4167
query39	1523	1475	1453	1453
query40	205	115	98	98
query41	45	43	43	43
query42	123	109	102	102
query43	532	522	500	500
query44	1373	822	841	822
query45	190	178	164	164
query46	904	1059	674	674
query47	1957	2012	1941	1941
query48	384	416	330	330
query49	737	494	403	403
query50	678	680	415	415
query51	7277	7328	7158	7158
query52	109	104	95	95
query53	228	255	185	185
query54	497	521	423	423
query55	83	81	78	78
query56	270	274	241	241
query57	1262	1252	1177	1177
query58	244	222	234	222
query59	3199	3180	3102	3102
query60	277	260	259	259
query61	106	111	106	106
query62	896	814	771	771
query63	235	198	195	195
query64	3139	1027	669	669
query65	3345	3291	3295	3291
query66	841	417	305	305
query67	16457	15807	15591	15591
query68	8362	682	503	503
query69	489	293	250	250
query70	1285	1159	1114	1114
query71	447	283	262	262
query72	6319	3884	3733	3733
query73	657	753	436	436
query74	10592	9315	9176	9176
query75	3995	3150	2672	2672
query76	3631	1183	793	793
query77	757	379	315	315
query78	10182	10103	9538	9538
query79	3868	798	579	579
query80	725	528	434	434
query81	476	265	235	235
query82	575	144	125	125
query83	186	160	147	147
query84	279	85	72	72
query85	782	354	307	307
query86	352	311	312	311
query87	4628	4447	4525	4447
query88	3838	2166	2147	2147
query89	429	331	301	301
query90	1893	187	188	187
query91	135	151	103	103
query92	65	55	53	53
query93	1907	860	522	522
query94	661	408	291	291
query95	338	267	249	249
query96	477	614	276	276
query97	2951	3015	2830	2830
query98	212	197	197	197
query99	1691	1548	1493	1493
Total cold run time: 298922 ms
Total hot run time: 198129 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.35 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6c8a8f97fd50d5ddfda4ef8ef7ae8b61f0cdab06, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.04
query3	0.23	0.07	0.07
query4	1.61	0.10	0.11
query5	0.44	0.40	0.40
query6	1.15	0.66	0.64
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.55	0.53	0.51
query10	0.56	0.56	0.56
query11	0.15	0.11	0.11
query12	0.15	0.11	0.11
query13	0.62	0.60	0.61
query14	2.87	2.84	2.78
query15	0.91	0.83	0.82
query16	0.38	0.39	0.38
query17	1.00	1.06	1.05
query18	0.23	0.21	0.21
query19	1.92	1.88	1.97
query20	0.02	0.01	0.01
query21	15.37	0.95	0.56
query22	0.77	0.88	0.90
query23	14.97	1.48	0.61
query24	2.93	1.89	1.54
query25	0.22	0.18	0.12
query26	0.32	0.14	0.13
query27	0.05	0.05	0.05
query28	14.44	1.47	1.05
query29	12.56	3.95	3.27
query30	0.25	0.09	0.06
query31	2.84	0.60	0.38
query32	3.23	0.54	0.48
query33	3.08	3.19	3.09
query34	16.76	5.08	4.53
query35	4.58	4.44	4.51
query36	0.64	0.52	0.51
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.02
query40	0.17	0.13	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.45 s
Total hot run time: 32.35 s

@morningman morningman self-assigned this Jan 7, 2025
/**
* Convert view definition SQL based on current SQL dialect
*/
private String convertViewDefinition(String originSql, ConnectContext ctx) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest the extract a common method to do the conversion.
This code is just same as in convertOriginStmt() in ConnectProcessor.java

* .....
* }
*/
if (originalText != null && originalText.contains("/* Presto View: ")) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extract this logic to a separate method, like parseTrinoViewXXX(), to make the code more clear

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@liujiwen-up
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32480 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6f00487c79adfc77762dc44327d48985b3abf20d, data reload: false

------ Round 1 ----------------------------------
q1	17580	6315	6070	6070
q2	2038	311	166	166
q3	10416	1256	707	707
q4	10219	847	430	430
q5	7528	2174	1963	1963
q6	207	182	147	147
q7	907	764	620	620
q8	9228	1342	1177	1177
q9	5272	4857	4821	4821
q10	6735	2322	1852	1852
q11	490	279	261	261
q12	340	356	220	220
q13	17776	3674	3096	3096
q14	235	225	209	209
q15	558	491	505	491
q16	650	638	589	589
q17	553	843	333	333
q18	6974	6482	6262	6262
q19	1975	984	575	575
q20	313	312	192	192
q21	2823	2137	1998	1998
q22	360	330	301	301
Total cold run time: 103177 ms
Total hot run time: 32480 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6320	6249	6197	6197
q2	238	329	232	232
q3	2216	2645	2330	2330
q4	1432	1849	1425	1425
q5	4352	4780	4767	4767
q6	181	175	142	142
q7	2098	2084	1800	1800
q8	2614	2759	2701	2701
q9	7275	7140	7198	7140
q10	3074	3323	2765	2765
q11	603	516	505	505
q12	691	760	605	605
q13	3399	3793	3141	3141
q14	273	301	293	293
q15	566	541	504	504
q16	649	687	644	644
q17	1198	1717	1236	1236
q18	7621	7332	7232	7232
q19	795	1114	1002	1002
q20	1880	1947	1807	1807
q21	5378	5103	4871	4871
q22	587	604	568	568
Total cold run time: 53440 ms
Total hot run time: 51907 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189782 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6f00487c79adfc77762dc44327d48985b3abf20d, data reload: false

query1	965	372	365	365
query2	6520	2324	2430	2324
query3	6711	216	211	211
query4	33963	23507	23872	23507
query5	4309	586	447	447
query6	291	203	185	185
query7	4640	487	300	300
query8	319	262	241	241
query9	9612	2764	2770	2764
query10	468	311	265	265
query11	18025	15349	15099	15099
query12	153	118	110	110
query13	1669	549	421	421
query14	10206	6745	7703	6745
query15	251	200	198	198
query16	8854	566	467	467
query17	1637	748	562	562
query18	2106	406	298	298
query19	228	188	165	165
query20	125	113	112	112
query21	204	165	106	106
query22	4443	4471	4360	4360
query23	34445	33564	33500	33500
query24	6411	2320	2299	2299
query25	478	449	380	380
query26	1188	273	148	148
query27	2017	454	344	344
query28	5190	2451	2442	2442
query29	729	539	407	407
query30	223	188	147	147
query31	989	908	806	806
query32	82	61	56	56
query33	513	350	298	298
query34	762	835	505	505
query35	818	833	745	745
query36	1044	1042	1000	1000
query37	117	97	80	80
query38	4228	4080	4041	4041
query39	1516	1434	1437	1434
query40	200	112	98	98
query41	46	45	71	45
query42	118	103	99	99
query43	510	526	481	481
query44	1301	804	807	804
query45	183	169	172	169
query46	861	1032	637	637
query47	1897	1903	1861	1861
query48	391	399	333	333
query49	767	472	406	406
query50	612	646	378	378
query51	6900	7023	6965	6965
query52	104	104	88	88
query53	215	251	187	187
query54	470	482	388	388
query55	85	82	79	79
query56	252	264	234	234
query57	1188	1184	1112	1112
query58	240	239	231	231
query59	3018	3184	2793	2793
query60	276	253	256	253
query61	124	119	138	119
query62	886	804	773	773
query63	234	190	193	190
query64	4039	977	654	654
query65	3268	3180	3231	3180
query66	1078	418	323	323
query67	16040	15755	15375	15375
query68	8057	707	513	513
query69	467	295	254	254
query70	1182	1159	1112	1112
query71	433	288	264	264
query72	6203	3899	3829	3829
query73	653	736	353	353
query74	9510	9033	8944	8944
query75	3857	3154	2636	2636
query76	3590	1170	745	745
query77	770	377	277	277
query78	10282	10029	9345	9345
query79	3586	805	591	591
query80	734	508	439	439
query81	497	273	228	228
query82	635	152	122	122
query83	158	170	155	155
query84	245	90	72	72
query85	805	337	316	316
query86	405	323	287	287
query87	4405	4616	4493	4493
query88	4916	2231	2228	2228
query89	411	335	310	310
query90	1799	187	191	187
query91	135	137	109	109
query92	73	57	56	56
query93	1521	795	533	533
query94	662	385	281	281
query95	338	265	254	254
query96	483	617	276	276
query97	2879	3022	2817	2817
query98	229	206	201	201
query99	1794	1576	1416	1416
Total cold run time: 293466 ms
Total hot run time: 189782 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.4 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6f00487c79adfc77762dc44327d48985b3abf20d, data reload: false

query1	0.03	0.03	0.04
query2	0.06	0.04	0.03
query3	0.23	0.07	0.07
query4	1.61	0.11	0.11
query5	0.42	0.43	0.41
query6	1.15	0.65	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.58	0.52	0.51
query10	0.56	0.56	0.56
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.60	0.62	0.59
query14	2.74	2.75	2.74
query15	0.90	0.83	0.84
query16	0.36	0.39	0.37
query17	1.05	1.06	1.03
query18	0.23	0.22	0.20
query19	1.98	1.86	1.99
query20	0.02	0.01	0.01
query21	15.36	0.88	0.59
query22	0.75	1.03	0.62
query23	15.10	1.43	0.58
query24	3.12	1.93	1.97
query25	0.17	0.16	0.08
query26	0.24	0.14	0.14
query27	0.06	0.06	0.04
query28	14.67	1.44	1.05
query29	12.57	4.04	3.39
query30	0.25	0.08	0.06
query31	2.83	0.60	0.37
query32	3.23	0.54	0.46
query33	3.12	3.12	3.15
query34	16.85	5.09	4.49
query35	4.48	4.43	4.50
query36	0.78	0.49	0.48
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.16	0.14	0.13
query41	0.07	0.02	0.02
query42	0.04	0.03	0.02
query43	0.03	0.03	0.04
Total cold run time: 106.93 s
Total hot run time: 32.4 s

@liujiwen-up
Copy link
Contributor Author

run buildall

morrySnow
morrySnow previously approved these changes Jan 9, 2025
Comment on lines 459 to 460
String trinoViewSql = parseTrinoViewDefinition(originalText);
return trinoViewSql != null ? trinoViewSql : originalText;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not return originalText instead of return null when parse failed?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. you can return originalText directly in parseTrinoViewDefinition's return value

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and better add a UT for this method

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Contributor

github-actions bot commented Jan 9, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 9, 2025
Copy link
Contributor

github-actions bot commented Jan 9, 2025

PR approved by anyone and no changes requested.

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jan 14, 2025
@liujiwen-up
Copy link
Contributor Author

run buildall

@liujiwen-up
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32525 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit de5650e48e4bc448519cd66aaa0578cb3848ef7b, data reload: false

------ Round 1 ----------------------------------
q1	17580	6157	6037	6037
q2	2049	303	163	163
q3	10424	1199	741	741
q4	10204	860	426	426
q5	7488	2185	1921	1921
q6	210	183	152	152
q7	896	764	591	591
q8	9228	1321	1114	1114
q9	5157	4840	4853	4840
q10	6770	2322	1866	1866
q11	474	267	255	255
q12	343	356	227	227
q13	18040	3725	3132	3132
q14	234	235	203	203
q15	566	530	519	519
q16	649	620	574	574
q17	573	847	335	335
q18	6932	6389	6424	6389
q19	3778	941	559	559
q20	302	317	193	193
q21	2771	2187	1977	1977
q22	364	341	311	311
Total cold run time: 105032 ms
Total hot run time: 32525 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6362	6206	6214	6206
q2	242	334	237	237
q3	2290	2648	2364	2364
q4	1429	1825	1363	1363
q5	4316	4756	4841	4756
q6	180	182	148	148
q7	2062	1956	1808	1808
q8	2618	2772	2674	2674
q9	7269	7232	7287	7232
q10	3046	3291	2752	2752
q11	607	541	507	507
q12	707	753	647	647
q13	3493	3946	3361	3361
q14	281	292	279	279
q15	589	538	532	532
q16	669	692	654	654
q17	1209	1746	1294	1294
q18	7711	7576	7322	7322
q19	830	1054	1130	1054
q20	2013	2079	1917	1917
q21	5488	5250	4936	4936
q22	590	619	537	537
Total cold run time: 54001 ms
Total hot run time: 52580 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195173 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit de5650e48e4bc448519cd66aaa0578cb3848ef7b, data reload: false

query1	1320	945	929	929
query2	6288	2114	2054	2054
query3	10983	4501	4438	4438
query4	60946	29725	23165	23165
query5	5486	624	452	452
query6	429	202	194	194
query7	5583	529	311	311
query8	333	247	228	228
query9	8311	2775	2751	2751
query10	446	321	261	261
query11	17610	15119	15371	15119
query12	165	116	115	115
query13	1452	546	447	447
query14	10971	7176	7564	7176
query15	211	222	193	193
query16	7261	618	476	476
query17	1118	716	572	572
query18	1890	417	308	308
query19	203	182	171	171
query20	125	111	114	111
query21	212	125	106	106
query22	4595	4651	4235	4235
query23	33967	33782	33656	33656
query24	5429	2340	2357	2340
query25	461	451	400	400
query26	676	293	161	161
query27	1789	471	341	341
query28	4188	2513	2475	2475
query29	542	553	431	431
query30	218	191	154	154
query31	962	886	825	825
query32	71	58	63	58
query33	427	366	309	309
query34	810	867	531	531
query35	834	832	781	781
query36	977	1052	949	949
query37	120	100	76	76
query38	4339	4430	4286	4286
query39	1504	1457	1432	1432
query40	193	121	108	108
query41	58	52	52	52
query42	125	105	104	104
query43	538	541	493	493
query44	1429	846	839	839
query45	183	178	168	168
query46	875	1085	660	660
query47	1891	1917	1829	1829
query48	399	415	337	337
query49	709	502	429	429
query50	703	679	401	401
query51	7034	6961	7119	6961
query52	101	105	93	93
query53	236	257	187	187
query54	481	510	452	452
query55	85	79	89	79
query56	264	269	250	250
query57	1239	1200	1157	1157
query58	228	238	248	238
query59	3050	3052	2803	2803
query60	275	290	256	256
query61	115	113	114	113
query62	781	766	684	684
query63	225	183	186	183
query64	1305	1013	651	651
query65	3269	3187	3150	3150
query66	680	405	300	300
query67	16034	15606	15512	15512
query68	5066	853	546	546
query69	483	302	254	254
query70	1183	1141	1112	1112
query71	409	281	264	264
query72	6380	3961	3858	3858
query73	805	763	365	365
query74	9951	9023	8920	8920
query75	3215	3109	2652	2652
query76	3825	1159	751	751
query77	470	371	280	280
query78	10377	9949	9237	9237
query79	3189	806	601	601
query80	1588	541	449	449
query81	546	277	240	240
query82	357	151	114	114
query83	274	176	159	159
query84	293	100	71	71
query85	776	373	360	360
query86	445	306	294	294
query87	4453	4608	4641	4608
query88	4539	2213	2178	2178
query89	411	329	284	284
query90	1607	193	190	190
query91	134	139	106	106
query92	68	58	57	57
query93	2924	885	549	549
query94	771	402	305	305
query95	329	273	254	254
query96	494	616	275	275
query97	2811	2858	2781	2781
query98	217	201	198	198
query99	1396	1480	1355	1355
Total cold run time: 315543 ms
Total hot run time: 195173 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.92 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit de5650e48e4bc448519cd66aaa0578cb3848ef7b, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.04	0.03
query3	0.24	0.08	0.06
query4	1.62	0.11	0.10
query5	0.43	0.43	0.41
query6	1.17	0.65	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.51	0.52
query10	0.56	0.57	0.55
query11	0.15	0.10	0.11
query12	0.14	0.11	0.11
query13	0.62	0.59	0.60
query14	2.73	2.82	2.75
query15	0.90	0.83	0.83
query16	0.40	0.38	0.40
query17	1.01	1.02	1.01
query18	0.24	0.22	0.21
query19	1.90	1.82	2.03
query20	0.01	0.01	0.01
query21	15.36	0.99	0.57
query22	0.76	0.82	0.81
query23	15.12	1.48	0.54
query24	3.14	1.57	1.45
query25	0.19	0.11	0.15
query26	0.33	0.15	0.14
query27	0.07	0.03	0.05
query28	14.13	1.48	1.05
query29	12.63	3.91	3.27
query30	0.25	0.09	0.07
query31	2.84	0.59	0.38
query32	3.23	0.56	0.47
query33	3.00	3.05	3.05
query34	16.74	5.14	4.51
query35	4.53	4.54	4.49
query36	0.66	0.49	0.48
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.02
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.36 s
Total hot run time: 31.92 s

@liujiwen-up
Copy link
Contributor Author

External Regression

@liujiwen-up
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32568 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit de5650e48e4bc448519cd66aaa0578cb3848ef7b, data reload: false

------ Round 1 ----------------------------------
q1	17577	6146	6064	6064
q2	2042	304	169	169
q3	10428	1230	744	744
q4	10202	856	424	424
q5	7574	2162	1939	1939
q6	204	179	146	146
q7	890	756	594	594
q8	9260	1358	1095	1095
q9	5187	4858	4863	4858
q10	6755	2305	1869	1869
q11	485	277	253	253
q12	337	360	213	213
q13	18046	3683	3102	3102
q14	224	223	210	210
q15	589	515	525	515
q16	641	614	581	581
q17	572	837	320	320
q18	6885	6469	6452	6452
q19	1866	963	530	530
q20	305	326	191	191
q21	3026	2180	1994	1994
q22	363	326	305	305
Total cold run time: 103458 ms
Total hot run time: 32568 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6369	6462	6300	6300
q2	247	322	237	237
q3	2250	2707	2470	2470
q4	1458	1860	1423	1423
q5	4444	4855	4914	4855
q6	190	180	141	141
q7	2110	1995	1835	1835
q8	2630	2807	2711	2711
q9	7346	7373	7568	7373
q10	3145	3319	2835	2835
q11	597	536	516	516
q12	740	781	660	660
q13	3761	4174	3618	3618
q14	306	299	295	295
q15	584	535	520	520
q16	682	731	662	662
q17	1258	1750	1282	1282
q18	7590	7537	7237	7237
q19	839	1061	1140	1061
q20	1996	2040	1899	1899
q21	5692	5234	4929	4929
q22	600	597	558	558
Total cold run time: 54834 ms
Total hot run time: 53417 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 194760 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit de5650e48e4bc448519cd66aaa0578cb3848ef7b, data reload: false

query1	1302	984	915	915
query2	6325	2066	2008	2008
query3	11001	4496	4452	4452
query4	61840	31138	23312	23312
query5	5296	594	442	442
query6	419	203	192	192
query7	5543	516	321	321
query8	333	239	236	236
query9	8454	2763	2730	2730
query10	435	300	241	241
query11	17541	15331	15708	15331
query12	157	108	113	108
query13	1426	547	433	433
query14	11352	7750	6579	6579
query15	210	212	192	192
query16	6532	608	464	464
query17	1111	722	552	552
query18	848	379	344	344
query19	207	201	179	179
query20	130	117	118	117
query21	215	124	107	107
query22	4484	4407	4547	4407
query23	34241	33708	33735	33708
query24	5502	2295	2246	2246
query25	447	462	395	395
query26	634	274	156	156
query27	1804	486	337	337
query28	4095	2539	2479	2479
query29	536	542	415	415
query30	207	192	162	162
query31	963	883	828	828
query32	69	59	56	56
query33	450	374	328	328
query34	723	891	535	535
query35	799	843	760	760
query36	990	1072	965	965
query37	122	98	81	81
query38	4290	4243	4215	4215
query39	1657	1450	1486	1450
query40	207	114	102	102
query41	54	51	52	51
query42	126	107	105	105
query43	521	520	503	503
query44	1306	849	872	849
query45	190	187	170	170
query46	893	1076	648	648
query47	1948	1948	1856	1856
query48	401	414	325	325
query49	722	497	394	394
query50	663	658	390	390
query51	6979	6989	6915	6915
query52	105	98	92	92
query53	226	250	184	184
query54	487	517	450	450
query55	85	103	77	77
query56	273	272	259	259
query57	1244	1218	1138	1138
query58	227	230	223	223
query59	2996	3044	2878	2878
query60	278	266	256	256
query61	125	117	115	115
query62	755	781	719	719
query63	226	185	182	182
query64	1275	1000	702	702
query65	3247	3121	3183	3121
query66	721	403	307	307
query67	16015	15593	15415	15415
query68	5025	830	537	537
query69	490	317	259	259
query70	1227	1158	1105	1105
query71	420	287	261	261
query72	6481	3863	3785	3785
query73	806	757	357	357
query74	10039	8853	8851	8851
query75	3218	3254	2680	2680
query76	3803	1149	780	780
query77	477	373	274	274
query78	10200	10087	9327	9327
query79	2728	811	619	619
query80	1693	533	429	429
query81	570	317	231	231
query82	355	148	124	124
query83	267	170	161	161
query84	288	99	68	68
query85	771	339	304	304
query86	450	307	296	296
query87	4520	4629	4544	4544
query88	3713	2212	2204	2204
query89	396	336	289	289
query90	1609	189	190	189
query91	132	137	106	106
query92	70	56	54	54
query93	2608	864	540	540
query94	755	402	296	296
query95	327	254	252	252
query96	500	606	292	292
query97	2774	2871	2766	2766
query98	225	201	196	196
query99	1415	1494	1380	1380
Total cold run time: 313337 ms
Total hot run time: 194760 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.39 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit de5650e48e4bc448519cd66aaa0578cb3848ef7b, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.03	0.04
query3	0.24	0.07	0.07
query4	1.62	0.10	0.11
query5	0.41	0.42	0.40
query6	1.16	0.66	0.64
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.59	0.51	0.50
query10	0.55	0.56	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.60	0.60	0.61
query14	2.71	2.72	2.84
query15	0.91	0.84	0.83
query16	0.38	0.39	0.37
query17	1.05	1.09	1.08
query18	0.22	0.21	0.20
query19	1.84	1.84	1.98
query20	0.01	0.01	0.01
query21	15.36	0.89	0.59
query22	0.76	0.75	0.63
query23	15.35	1.41	0.56
query24	2.66	1.39	0.25
query25	0.16	0.12	0.14
query26	0.35	0.14	0.14
query27	0.06	0.04	0.04
query28	13.02	1.55	1.05
query29	12.56	3.85	3.21
query30	0.26	0.09	0.06
query31	2.85	0.62	0.38
query32	3.23	0.55	0.46
query33	3.01	3.00	2.98
query34	16.88	5.14	4.43
query35	4.54	4.49	4.55
query36	0.65	0.52	0.48
query37	0.09	0.05	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.07	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 104.91 s
Total hot run time: 30.39 s

Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 17, 2025
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

@suxiaogang223 suxiaogang223 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@morningman morningman merged commit 12d84c1 into apache:master Jan 24, 2025
31 of 34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.x-experimental dev/3.0.x-experimental reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants