Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fix](Variant) variant should not implicit be short key column when create mv #46444

Merged
merged 2 commits into from
Jan 7, 2025
Merged

[Fix](Variant) variant should not implicit be short key column when create mv #46444

merged 2 commits into from
Jan 7, 2025

Conversation

eldenmoon
Copy link
Member

@eldenmoon eldenmoon commented Jan 6, 2025 •
edited
Loading

What problem does this PR solve?

When creating mv, the variant column should not be part of short key since it's not supported sorting
Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jan 6, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@eldenmoon
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32887 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d0da5f4d4a942778eab3a01901a323d4c0cdd709, data reload: false

------ Round 1 ----------------------------------
q1	17583	6154	6013	6013
q2	2046	311	187	187
q3	10396	1227	776	776
q4	10264	874	427	427
q5	8624	2229	1990	1990
q6	216	185	150	150
q7	889	759	593	593
q8	9226	1369	1181	1181
q9	5256	4969	5009	4969
q10	6795	2301	1856	1856
q11	479	280	250	250
q12	343	379	219	219
q13	17812	3801	3162	3162
q14	243	228	206	206
q15	565	510	510	510
q16	624	618	605	605
q17	575	868	330	330
q18	7117	6524	6429	6429
q19	2930	999	551	551
q20	298	306	179	179
q21	2727	2233	1995	1995
q22	358	338	309	309
Total cold run time: 105366 ms
Total hot run time: 32887 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6426	6256	6243	6243
q2	231	329	235	235
q3	2183	2644	2314	2314
q4	1395	1800	1360	1360
q5	4338	4760	4875	4760
q6	189	179	151	151
q7	2097	1972	1814	1814
q8	2649	2838	2673	2673
q9	7370	7317	7318	7317
q10	3046	3350	2773	2773
q11	570	495	493	493
q12	630	764	574	574
q13	3502	3839	3275	3275
q14	301	318	267	267
q15	564	528	509	509
q16	651	687	653	653
q17	1232	1743	1242	1242
q18	7642	7537	7483	7483
q19	846	1175	1112	1112
q20	1963	2074	1894	1894
q21	5745	5169	4955	4955
q22	628	605	589	589
Total cold run time: 54198 ms
Total hot run time: 52686 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196652 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d0da5f4d4a942778eab3a01901a323d4c0cdd709, data reload: false

query1	1297	928	918	918
query2	6385	2320	2282	2282
query3	11130	4791	4824	4791
query4	32933	23785	23516	23516
query5	4263	620	448	448
query6	273	211	200	200
query7	3979	489	303	303
query8	311	273	230	230
query9	9520	2638	2630	2630
query10	471	318	260	260
query11	17991	15368	15265	15265
query12	164	111	102	102
query13	1612	540	428	428
query14	10434	6797	6735	6735
query15	268	212	190	190
query16	8031	646	474	474
query17	1516	757	595	595
query18	2155	404	309	309
query19	189	196	162	162
query20	125	118	115	115
query21	200	123	114	114
query22	4915	4590	4501	4501
query23	34608	34380	33723	33723
query24	7394	2348	2310	2310
query25	467	450	392	392
query26	705	279	156	156
query27	2772	461	341	341
query28	5610	2462	2419	2419
query29	527	559	427	427
query30	209	184	161	161
query31	979	926	840	840
query32	86	60	54	54
query33	477	352	293	293
query34	783	853	546	546
query35	820	849	760	760
query36	1038	1048	954	954
query37	114	107	74	74
query38	4163	4233	4368	4233
query39	1517	1488	1493	1488
query40	211	123	114	114
query41	51	50	46	46
query42	121	113	106	106
query43	522	538	492	492
query44	1382	835	841	835
query45	188	174	170	170
query46	890	1088	674	674
query47	1992	1967	1979	1967
query48	392	421	319	319
query49	718	491	387	387
query50	643	660	405	405
query51	7240	7249	7173	7173
query52	107	100	127	100
query53	229	260	189	189
query54	468	500	417	417
query55	88	83	81	81
query56	256	255	243	243
query57	1228	1235	1145	1145
query58	236	236	228	228
query59	3358	3191	3207	3191
query60	279	261	243	243
query61	106	105	103	103
query62	875	819	732	732
query63	230	198	205	198
query64	3209	1013	655	655
query65	3345	3294	3305	3294
query66	779	406	329	329
query67	16380	15768	15422	15422
query68	9791	711	532	532
query69	469	292	252	252
query70	1237	1153	1127	1127
query71	419	299	248	248
query72	6427	3904	3834	3834
query73	647	752	373	373
query74	10345	9068	8875	8875
query75	4538	3180	2646	2646
query76	4777	1214	794	794
query77	773	372	281	281
query78	10186	10154	9503	9503
query79	5488	797	581	581
query80	712	547	475	475
query81	494	264	229	229
query82	615	156	119	119
query83	197	170	145	145
query84	284	93	66	66
query85	783	364	295	295
query86	356	308	309	308
query87	4626	4376	4487	4376
query88	4464	2221	2166	2166
query89	462	331	290	290
query90	1972	191	192	191
query91	131	137	101	101
query92	67	63	51	51
query93	3299	851	530	530
query94	673	381	272	272
query95	340	264	261	261
query96	490	607	293	293
query97	2915	2935	2810	2810
query98	235	205	199	199
query99	1706	1521	1432	1432
Total cold run time: 305872 ms
Total hot run time: 196652 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.14 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d0da5f4d4a942778eab3a01901a323d4c0cdd709, data reload: false

query1	0.06	0.03	0.03
query2	0.08	0.04	0.04
query3	0.24	0.08	0.07
query4	1.61	0.10	0.10
query5	0.41	0.38	0.40
query6	1.16	0.66	0.67
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.52	0.50
query10	0.55	0.56	0.56
query11	0.14	0.10	0.10
query12	0.15	0.11	0.11
query13	0.62	0.61	0.60
query14	2.73	2.78	2.76
query15	0.90	0.82	0.81
query16	0.39	0.37	0.38
query17	1.05	1.04	1.04
query18	0.23	0.20	0.21
query19	1.85	1.84	1.95
query20	0.01	0.02	0.01
query21	15.36	0.96	0.58
query22	0.74	0.82	0.72
query23	15.23	1.46	0.56
query24	3.15	1.30	0.84
query25	0.24	0.14	0.05
query26	0.27	0.15	0.13
query27	0.06	0.04	0.04
query28	13.72	1.56	1.03
query29	12.57	3.91	3.22
query30	0.25	0.10	0.06
query31	2.82	0.58	0.38
query32	3.22	0.54	0.47
query33	3.12	3.05	3.03
query34	16.95	5.14	4.52
query35	4.54	4.50	4.52
query36	0.65	0.49	0.49
query37	0.09	0.07	0.06
query38	0.04	0.04	0.03
query39	0.04	0.03	0.02
query40	0.16	0.13	0.12
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.18 s
Total hot run time: 31.14 s

@eldenmoon
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32718 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a2a5b652166e8c7d4a36d467de347769dee31378, data reload: false

------ Round 1 ----------------------------------
q1	17586	6125	6056	6056
q2	2047	308	174	174
q3	10405	1310	739	739
q4	10215	861	437	437
q5	7528	2200	2008	2008
q6	205	176	150	150
q7	904	744	621	621
q8	9240	1369	1181	1181
q9	5252	4876	4829	4829
q10	6763	2332	1861	1861
q11	457	286	256	256
q12	347	361	225	225
q13	17784	3599	3055	3055
q14	235	227	219	219
q15	573	502	489	489
q16	614	609	604	604
q17	559	842	321	321
q18	7093	6460	6389	6389
q19	1260	980	578	578
q20	316	333	199	199
q21	2836	2226	2027	2027
q22	360	330	300	300
Total cold run time: 102579 ms
Total hot run time: 32718 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6184	6233	6202	6202
q2	229	319	229	229
q3	2267	2672	2303	2303
q4	1397	1837	1372	1372
q5	4340	4727	4755	4727
q6	183	177	145	145
q7	2048	2025	1843	1843
q8	2583	2760	2693	2693
q9	7324	7203	7150	7150
q10	3075	3332	2810	2810
q11	589	515	511	511
q12	654	777	641	641
q13	3606	3888	3190	3190
q14	285	307	285	285
q15	571	519	498	498
q16	660	700	630	630
q17	1208	1729	1261	1261
q18	7674	7329	7070	7070
q19	854	880	1134	880
q20	1920	1991	1891	1891
q21	5537	5120	4816	4816
q22	620	624	550	550
Total cold run time: 53808 ms
Total hot run time: 51697 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189278 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a2a5b652166e8c7d4a36d467de347769dee31378, data reload: false

query1	956	386	363	363
query2	6531	2292	2335	2292
query3	6704	227	213	213
query4	33586	23686	23275	23275
query5	4361	614	452	452
query6	278	214	187	187
query7	4629	487	302	302
query8	298	236	231	231
query9	9613	2643	2644	2643
query10	463	325	267	267
query11	18015	15504	15036	15036
query12	151	106	103	103
query13	1651	518	392	392
query14	9200	6601	7221	6601
query15	239	205	199	199
query16	8149	622	463	463
query17	1574	729	550	550
query18	2098	413	298	298
query19	217	179	161	161
query20	124	114	114	114
query21	215	127	103	103
query22	4180	4524	4155	4155
query23	34564	33797	33311	33311
query24	6695	2361	2315	2315
query25	519	511	381	381
query26	1229	272	149	149
query27	2396	450	328	328
query28	5458	2429	2438	2429
query29	695	524	413	413
query30	227	182	148	148
query31	973	886	833	833
query32	73	60	59	59
query33	507	364	295	295
query34	754	828	505	505
query35	799	803	734	734
query36	1022	1010	969	969
query37	113	98	71	71
query38	4276	4304	4085	4085
query39	1459	1417	1467	1417
query40	200	116	105	105
query41	46	46	44	44
query42	119	105	103	103
query43	527	530	478	478
query44	1270	795	807	795
query45	177	173	170	170
query46	855	1035	642	642
query47	1898	1978	1900	1900
query48	380	402	319	319
query49	785	470	374	374
query50	614	646	377	377
query51	7084	7103	7015	7015
query52	102	97	91	91
query53	223	266	184	184
query54	469	508	409	409
query55	79	80	78	78
query56	252	264	241	241
query57	1220	1180	1139	1139
query58	235	226	238	226
query59	3051	3239	3009	3009
query60	277	271	267	267
query61	113	104	111	104
query62	895	778	737	737
query63	223	195	199	195
query64	4363	989	653	653
query65	3260	3220	3186	3186
query66	1080	426	310	310
query67	15999	15692	15521	15521
query68	8705	699	501	501
query69	444	295	263	263
query70	1231	1174	1131	1131
query71	421	276	255	255
query72	6177	3879	3844	3844
query73	649	739	360	360
query74	10486	9103	9233	9103
query75	4359	3135	2648	2648
query76	3839	1181	765	765
query77	792	367	265	265
query78	10183	10158	9409	9409
query79	3399	796	578	578
query80	699	529	432	432
query81	471	276	284	276
query82	549	156	126	126
query83	186	160	145	145
query84	275	86	79	79
query85	765	382	305	305
query86	354	319	292	292
query87	4446	4531	4326	4326
query88	4236	2182	2159	2159
query89	420	329	359	329
query90	1873	187	190	187
query91	137	136	104	104
query92	70	57	54	54
query93	1614	839	527	527
query94	674	391	288	288
query95	333	263	251	251
query96	487	612	275	275
query97	2886	2980	2845	2845
query98	227	204	198	198
query99	1669	1569	1444	1444
Total cold run time: 293843 ms
Total hot run time: 189278 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.04 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a2a5b652166e8c7d4a36d467de347769dee31378, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.03
query3	0.23	0.08	0.06
query4	1.61	0.10	0.11
query5	0.41	0.42	0.42
query6	1.15	0.64	0.65
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.58	0.50	0.48
query10	0.56	0.58	0.56
query11	0.14	0.11	0.11
query12	0.15	0.11	0.11
query13	0.60	0.60	0.59
query14	2.73	2.72	2.72
query15	0.90	0.83	0.82
query16	0.38	0.38	0.38
query17	1.07	1.06	1.05
query18	0.22	0.21	0.21
query19	1.88	1.84	1.86
query20	0.01	0.02	0.02
query21	15.36	0.90	0.57
query22	0.74	0.73	0.68
query23	15.32	1.43	0.49
query24	3.13	1.83	0.60
query25	0.18	0.11	0.17
query26	0.36	0.16	0.14
query27	0.04	0.08	0.05
query28	13.09	1.52	1.05
query29	12.59	3.92	3.26
query30	0.26	0.10	0.07
query31	2.85	0.56	0.38
query32	3.22	0.54	0.46
query33	3.07	3.14	3.18
query34	16.62	5.13	4.51
query35	4.51	4.53	4.52
query36	0.65	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.26 s
Total hot run time: 31.04 s

@wm1581066 wm1581066 added the usercase Important user case type label label Jan 7, 2025
@eldenmoon eldenmoon requested a review from starocean999 January 7, 2025 03:59
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 7, 2025
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 7, 2025

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 7, 2025

PR approved by anyone and no changes requested.

@eldenmoon eldenmoon merged commit bdab3bd into apache:master Jan 7, 2025
33 of 35 checks passed
@eldenmoon eldenmoon deleted the var-key branch January 7, 2025 08:10
github-actions bot pushed a commit that referenced this pull request Jan 7, 2025
@eldenmoon
[Fix](Variant) variant should not implicit be short key column when c…
3c9824b 
…reate mv (#46444)

When creating mv, the variant column should not be part of short key
since it's not supported sorting
@github-actions github-actions bot mentioned this pull request Jan 7, 2025
Merged
eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Jan 7, 2025
@eldenmoon
[Fix](Variant) variant should not implicit be short key column when c…
3e9a01d 
…reate mv (apache#46444)

When creating mv, the variant column should not be part of short key
since it's not supported sorting
@eldenmoon eldenmoon mentioned this pull request Jan 7, 2025
Merged
yiguolei pushed a commit to eldenmoon/incubator-doris that referenced this pull request Jan 8, 2025
@eldenmoon @yiguolei
[Fix](Variant) variant should not implicit be short key column when c…
38ad966 
…reate mv (apache#46444)

When creating mv, the variant column should not be part of short key
since it's not supported sorting
eldenmoon added a commit that referenced this pull request Jan 8, 2025
@github-actions @eldenmoon
branch-3.0: [Fix](Variant) variant should not implicit be short key c…
b4f1d7d 
…olumn when create mv #46444 (#46538)

Cherry-picked from #46444

Co-authored-by: lihangyu <lihangyu@selectdb.com>
yiguolei pushed a commit to eldenmoon/incubator-doris that referenced this pull request Jan 8, 2025
@eldenmoon @yiguolei
[Fix](Variant) variant should not implicit be short key column when c…
e787ca7 
…reate mv (apache#46444)

When creating mv, the variant column should not be part of short key
since it's not supported sorting
yiguolei pushed a commit that referenced this pull request Jan 9, 2025
@eldenmoon
[Fix](Variant) variant should not implicit be short key column when c…
ca5a4f6 
…reate mv (#46539)

cherry-pick from #46444
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@starocean999 starocean999 starocean999 approved these changes

@morningman morningman morningman approved these changes

@dataroaring dataroaring Awaiting requested review from dataroaring dataroaring is a code owner

@CalvinKirs CalvinKirs Awaiting requested review from CalvinKirs CalvinKirs is a code owner

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.4-merged reviewed usercase Important user case type label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@eldenmoon @Thearas @doris-robot @morningman @starocean999 @yiguolei @wm1581066