[opt](coordinator) optimize parallel degree of shuffle when use nereids #44754

924060929 · 2024-11-28T12:53:39Z

What problem does this PR solve?

optimize parallel degree of shuffle when use nereids , this pr can fix some performance rollback when upgrade doris from 1.2 to 2.x/3.x

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

924060929 · 2024-11-28T12:53:49Z

run buildall

doris-robot · 2024-11-28T12:53:54Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

github-actions · 2024-11-28T14:10:29Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-11-28T14:10:31Z

PR approved by anyone and no changes requested.

924060929 · 2024-12-17T08:12:19Z

run buildall

924060929 · 2024-12-17T08:27:49Z

run buildall

924060929 · 2024-12-17T10:18:36Z

run buildall

924060929 · 2024-12-17T11:37:22Z

run buildall

924060929 · 2024-12-18T04:21:20Z

run buildall

github-actions · 2024-12-18T04:21:48Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

924060929 · 2024-12-18T08:29:34Z

run buildall

github-actions · 2024-12-18T08:30:11Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

924060929 · 2024-12-18T08:36:12Z

run buildall

github-actions · 2024-12-18T08:36:58Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

924060929 · 2024-12-18T16:55:35Z

run buildall

924060929 · 2024-12-19T11:02:21Z

run buildall

924060929 · 2024-12-20T07:04:39Z

run buildall

doris-robot · 2024-12-20T08:21:16Z

TPC-H: Total hot run time: 37787 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 97e762ce46f8ee066cbdb971f8a38724ee6e957c, data reload: false

------ Round 1 ----------------------------------
q1	17603	8298	7262	7262
q2	1382	166	152	152
q3	10672	1064	1160	1064
q4	10574	768	679	679
q5	7625	2764	2748	2748
q6	246	147	147	147
q7	996	625	606	606
q8	9262	1858	1926	1858
q9	5420	4458	4480	4458
q10	7003	2284	2329	2284
q11	462	260	275	260
q12	404	218	227	218
q13	17799	2963	2987	2963
q14	251	215	209	209
q15	542	500	496	496
q16	684	608	590	590
q17	994	564	522	522
q18	7310	6741	6603	6603
q19	1354	1041	947	947
q20	362	158	153	153
q21	3998	3252	3253	3252
q22	383	316	320	316
Total cold run time: 105326 ms
Total hot run time: 37787 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7241	7220	7206	7206
q2	282	200	195	195
q3	2902	2855	2804	2804
q4	1929	1772	1806	1772
q5	5609	5756	5632	5632
q6	230	142	147	142
q7	2249	1798	1794	1794
q8	3394	3525	3528	3525
q9	6783	6761	6749	6749
q10	3588	3548	3542	3542
q11	607	523	505	505
q12	794	611	583	583
q13	13741	3136	3064	3064
q14	299	276	285	276
q15	540	501	492	492
q16	697	652	629	629
q17	1836	1597	1691	1597
q18	8423	7918	7649	7649
q19	1755	1625	1566	1566
q20	2122	1876	1843	1843
q21	5448	5470	5402	5402
q22	648	615	556	556
Total cold run time: 71117 ms
Total hot run time: 57523 ms

doris-robot · 2025-01-08T12:15:31Z

TPC-H: Total hot run time: 30584 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d4bb8fd067cf63d43a8a9389b954327b6a006fb3, data reload: false

------ Round 1 ----------------------------------
q1	17581	6150	6113	6113
q2	2045	308	162	162
q3	10429	1218	731	731
q4	10213	850	418	418
q5	7520	2180	1919	1919
q6	207	183	148	148
q7	881	748	618	618
q8	9221	1291	1133	1133
q9	3856	2865	2970	2865
q10	6744	2295	1855	1855
q11	492	280	272	272
q12	355	366	227	227
q13	17754	3638	3071	3071
q14	237	229	207	207
q15	561	516	491	491
q16	635	605	604	604
q17	567	847	330	330
q18	7208	6371	6486	6371
q19	1246	973	551	551
q20	293	316	178	178
q21	3153	2249	2009	2009
q22	375	332	311	311
Total cold run time: 101573 ms
Total hot run time: 30584 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6239	6269	6292	6269
q2	225	318	223	223
q3	2252	2660	2322	2322
q4	1420	1820	1368	1368
q5	4291	4774	4638	4638
q6	183	170	145	145
q7	1948	1897	1846	1846
q8	2643	2862	2720	2720
q9	5143	5060	4918	4918
q10	3001	3312	2830	2830
q11	600	550	500	500
q12	713	733	626	626
q13	3447	3872	3219	3219
q14	273	288	275	275
q15	562	508	505	505
q16	690	681	653	653
q17	1202	1753	1236	1236
q18	7838	7435	7196	7196
q19	832	1192	1025	1025
q20	1900	1958	1812	1812
q21	5466	5117	4883	4883
q22	630	613	597	597
Total cold run time: 51498 ms
Total hot run time: 49806 ms

doris-robot · 2025-01-08T12:27:13Z

TPC-DS: Total hot run time: 183628 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d4bb8fd067cf63d43a8a9389b954327b6a006fb3, data reload: false

query1	970	387	381	381
query2	6532	2404	2322	2322
query3	6704	221	216	216
query4	33470	23422	23166	23166
query5	4320	635	474	474
query6	273	206	202	202
query7	4623	522	310	310
query8	327	257	255	255
query9	9659	2750	2751	2750
query10	468	312	269	269
query11	18069	15682	14998	14998
query12	164	108	102	102
query13	1687	553	397	397
query14	10208	6345	7129	6345
query15	224	134	123	123
query16	8079	608	465	465
query17	1565	764	584	584
query18	1971	296	208	208
query19	195	216	145	145
query20	121	110	113	110
query21	209	122	104	104
query22	4107	4236	4179	4179
query23	33949	32817	32748	32748
query24	6279	1941	2011	1941
query25	488	429	361	361
query26	1227	279	153	153
query27	2082	429	285	285
query28	5713	2486	2446	2446
query29	723	512	386	386
query30	198	121	94	94
query31	961	879	776	776
query32	95	63	64	63
query33	506	336	284	284
query34	800	868	521	521
query35	746	753	686	686
query36	1023	1039	953	953
query37	119	100	78	78
query38	4004	4268	3925	3925
query39	1474	1404	1578	1404
query40	211	118	104	104
query41	50	46	45	45
query42	118	101	108	101
query43	516	534	498	498
query44	1339	816	818	816
query45	125	117	109	109
query46	753	915	542	542
query47	1837	1852	1746	1746
query48	387	419	323	323
query49	792	493	409	409
query50	653	683	405	405
query51	7065	6994	6874	6874
query52	102	108	98	98
query53	236	265	186	186
query54	414	374	340	340
query55	87	91	82	82
query56	280	278	243	243
query57	1175	1167	1067	1067
query58	242	228	230	228
query59	3059	3236	2878	2878
query60	292	267	260	260
query61	115	114	114	114
query62	859	780	700	700
query63	236	202	199	199
query64	4500	961	604	604
query65	3276	3182	3200	3182
query66	1071	433	325	325
query67	15892	15672	15286	15286
query68	7616	712	540	540
query69	486	303	283	283
query70	1219	1164	1152	1152
query71	391	296	268	268
query72	2237	1258	935	935
query73	589	640	357	357
query74	10164	8895	9065	8895
query75	3252	3164	2669	2669
query76	3337	1179	787	787
query77	454	377	297	297
query78	10002	10078	9478	9478
query79	3239	837	605	605
query80	653	540	452	452
query81	432	170	115	115
query82	620	163	129	129
query83	161	169	187	169
query84	234	97	73	73
query85	741	371	314	314
query86	397	301	298	298
query87	4380	4260	4301	4260
query88	4651	2230	2202	2202
query89	419	336	299	299
query90	1875	195	197	195
query91	134	129	108	108
query92	65	58	54	54
query93	1745	878	526	526
query94	707	415	298	298
query95	341	263	262	262
query96	497	664	294	294
query97	2843	2945	2795	2795
query98	228	206	198	198
query99	1639	1514	1348	1348
Total cold run time: 285762 ms
Total hot run time: 183628 ms

doris-robot · 2025-01-08T12:32:33Z

ClickBench: Total hot run time: 31.51 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d4bb8fd067cf63d43a8a9389b954327b6a006fb3, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.04
query3	0.25	0.07	0.07
query4	1.60	0.10	0.10
query5	0.43	0.41	0.40
query6	1.14	0.64	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.57	0.51	0.50
query10	0.54	0.56	0.54
query11	0.15	0.11	0.10
query12	0.14	0.11	0.11
query13	0.60	0.59	0.60
query14	2.81	2.88	2.75
query15	0.89	0.83	0.82
query16	0.37	0.36	0.36
query17	1.05	1.05	1.02
query18	0.23	0.22	0.21
query19	1.92	1.83	1.99
query20	0.02	0.02	0.01
query21	15.38	0.98	0.59
query22	0.77	0.83	0.73
query23	15.17	1.42	0.55
query24	3.42	1.19	1.17
query25	0.35	0.12	0.12
query26	0.26	0.13	0.13
query27	0.07	0.06	0.06
query28	13.43	1.50	1.05
query29	12.88	3.95	3.30
query30	0.25	0.09	0.06
query31	2.82	0.61	0.37
query32	3.23	0.56	0.45
query33	3.07	3.03	3.05
query34	16.82	5.11	4.49
query35	4.55	4.50	4.52
query36	0.65	0.50	0.47
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.5 s
Total hot run time: 31.51 s

doris-robot · 2025-01-08T12:37:04Z

TeamCity be ut coverage result:
Function Coverage: 38.87% (10130/26061)
Line Coverage: 29.91% (85693/286475)
Region Coverage: 29.02% (43732/150704)
Branch Coverage: 25.56% (22320/87340)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d4bb8fd067cf63d43a8a9389b954327b6a006fb3_d4bb8fd067cf63d43a8a9389b954327b6a006fb3/report/index.html

924060929 · 2025-01-08T13:17:35Z

run buildall

doris-robot · 2025-01-08T13:41:13Z

TPC-H: Total hot run time: 32749 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4770b30d683d74ec2436dfdf18c0ffadc0a5fead, data reload: false

------ Round 1 ----------------------------------
q1	17581	6131	6091	6091
q2	2050	308	169	169
q3	10419	1274	712	712
q4	10215	897	453	453
q5	7551	2172	1962	1962
q6	204	187	154	154
q7	908	748	599	599
q8	9247	1403	1209	1209
q9	5381	4977	4983	4977
q10	6758	2335	1856	1856
q11	479	288	267	267
q12	343	356	219	219
q13	17758	3701	3069	3069
q14	230	233	218	218
q15	574	511	519	511
q16	638	631	578	578
q17	559	852	322	322
q18	7131	6448	6350	6350
q19	1241	986	562	562
q20	307	324	185	185
q21	2863	2171	1978	1978
q22	359	334	308	308
Total cold run time: 102796 ms
Total hot run time: 32749 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6214	6213	6220	6213
q2	245	333	233	233
q3	2291	2674	2299	2299
q4	1427	1806	1366	1366
q5	4328	4762	4756	4756
q6	188	180	142	142
q7	2106	2004	1831	1831
q8	2598	2840	2686	2686
q9	7337	7184	7228	7184
q10	3051	3313	2821	2821
q11	584	518	507	507
q12	648	766	591	591
q13	3543	3927	3178	3178
q14	316	318	279	279
q15	569	523	509	509
q16	650	686	628	628
q17	1225	1756	1244	1244
q18	7657	7347	7333	7333
q19	844	1071	1121	1071
q20	2009	2020	1891	1891
q21	5664	5330	4958	4958
q22	649	621	591	591
Total cold run time: 54143 ms
Total hot run time: 52311 ms

doris-robot · 2025-01-08T13:53:09Z

TPC-DS: Total hot run time: 187933 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4770b30d683d74ec2436dfdf18c0ffadc0a5fead, data reload: false

query1	957	375	383	375
query2	6519	2490	2264	2264
query3	6705	218	219	218
query4	33914	23386	23434	23386
query5	4316	601	453	453
query6	291	234	190	190
query7	4640	474	292	292
query8	291	228	231	228
query9	9356	2742	2743	2742
query10	458	299	257	257
query11	18515	15326	15062	15062
query12	158	112	105	105
query13	1674	541	423	423
query14	9694	6406	7259	6406
query15	245	195	193	193
query16	8214	611	482	482
query17	1611	754	567	567
query18	2127	406	289	289
query19	222	191	167	167
query20	119	113	112	112
query21	215	123	106	106
query22	4360	4263	3922	3922
query23	33901	33120	33369	33120
query24	6388	2249	2270	2249
query25	494	448	375	375
query26	897	274	157	157
query27	2010	455	317	317
query28	5408	2466	2456	2456
query29	687	544	414	414
query30	231	185	152	152
query31	951	863	802	802
query32	93	63	59	59
query33	514	353	292	292
query34	748	830	512	512
query35	792	807	734	734
query36	1022	1036	920	920
query37	117	108	78	78
query38	4068	4012	4012	4012
query39	1462	1452	1416	1416
query40	203	116	106	106
query41	48	49	46	46
query42	123	104	102	102
query43	538	518	497	497
query44	1301	849	803	803
query45	177	173	164	164
query46	856	1021	634	634
query47	1820	1859	1769	1769
query48	401	410	319	319
query49	730	483	403	403
query50	619	643	390	390
query51	6848	6883	6828	6828
query52	102	100	90	90
query53	220	248	189	189
query54	479	483	416	416
query55	85	79	81	79
query56	271	288	247	247
query57	1168	1176	1104	1104
query58	245	234	240	234
query59	3026	3073	3136	3073
query60	284	262	257	257
query61	121	117	113	113
query62	821	776	724	724
query63	239	186	188	186
query64	3772	1033	638	638
query65	3249	3132	3148	3132
query66	945	416	309	309
query67	15870	15738	15491	15491
query68	8250	725	517	517
query69	467	295	268	268
query70	1209	1148	1149	1148
query71	434	285	263	263
query72	6251	3795	3846	3795
query73	662	758	364	364
query74	10221	9211	8795	8795
query75	4048	3137	2640	2640
query76	3553	1189	774	774
query77	784	375	276	276
query78	10024	9930	9454	9454
query79	3799	798	582	582
query80	734	573	427	427
query81	497	270	240	240
query82	628	150	125	125
query83	200	176	151	151
query84	291	88	76	76
query85	803	387	299	299
query86	409	309	298	298
query87	4275	4550	4228	4228
query88	4749	2226	2198	2198
query89	405	334	291	291
query90	1799	188	186	186
query91	138	136	106	106
query92	68	59	54	54
query93	1968	859	521	521
query94	652	387	281	281
query95	339	262	252	252
query96	495	607	289	289
query97	2926	2938	2806	2806
query98	237	205	204	204
query99	1670	1480	1378	1378
Total cold run time: 291930 ms
Total hot run time: 187933 ms

doris-robot · 2025-01-08T13:57:13Z

TeamCity be ut coverage result:
Function Coverage: 38.86% (10128/26061)
Line Coverage: 29.92% (85722/286475)
Region Coverage: 29.02% (43736/150704)
Branch Coverage: 25.56% (22324/87340)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4770b30d683d74ec2436dfdf18c0ffadc0a5fead_4770b30d683d74ec2436dfdf18c0ffadc0a5fead/report/index.html

doris-robot · 2025-01-08T13:58:29Z

ClickBench: Total hot run time: 31.77 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4770b30d683d74ec2436dfdf18c0ffadc0a5fead, data reload: false

query1	0.04	0.03	0.05
query2	0.07	0.04	0.04
query3	0.24	0.07	0.07
query4	1.60	0.10	0.10
query5	0.42	0.41	0.41
query6	1.15	0.65	0.64
query7	0.03	0.02	0.01
query8	0.04	0.03	0.03
query9	0.58	0.50	0.49
query10	0.54	0.56	0.56
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.73	2.88	2.73
query15	0.90	0.83	0.83
query16	0.38	0.38	0.37
query17	0.96	1.05	1.01
query18	0.23	0.22	0.21
query19	1.88	1.89	1.99
query20	0.01	0.01	0.01
query21	15.37	0.94	0.56
query22	0.77	0.88	0.72
query23	15.16	1.43	0.61
query24	3.02	1.17	1.26
query25	0.14	0.09	0.12
query26	0.32	0.15	0.14
query27	0.05	0.07	0.05
query28	14.09	1.54	1.05
query29	12.56	3.99	3.30
query30	0.25	0.09	0.06
query31	2.82	0.61	0.38
query32	3.23	0.54	0.46
query33	3.08	3.18	3.11
query34	16.83	5.16	4.56
query35	4.53	4.50	4.49
query36	0.83	0.49	0.48
query37	0.10	0.05	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.03
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.24 s
Total hot run time: 31.77 s

924060929 · 2025-01-08T14:16:16Z

run buildall

doris-robot · 2025-01-08T14:39:52Z

TPC-H: Total hot run time: 32939 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 06227897b1ac8eea9c4be3607787f23d62da03d1, data reload: false

------ Round 1 ----------------------------------
q1	17601	6173	6111	6111
q2	2049	311	186	186
q3	10397	1271	707	707
q4	10232	858	438	438
q5	7488	2225	1996	1996
q6	208	186	151	151
q7	911	764	632	632
q8	9230	1369	1174	1174
q9	5430	4994	4979	4979
q10	6845	2296	1853	1853
q11	510	297	254	254
q12	360	352	221	221
q13	17771	3701	3061	3061
q14	252	233	214	214
q15	590	524	519	519
q16	647	617	579	579
q17	591	854	322	322
q18	6911	6553	6444	6444
q19	1224	967	561	561
q20	320	343	195	195
q21	2871	2248	2032	2032
q22	371	332	310	310
Total cold run time: 102809 ms
Total hot run time: 32939 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6233	6291	6299	6291
q2	236	331	251	251
q3	2313	2650	2325	2325
q4	1414	1768	1360	1360
q5	4377	4833	4633	4633
q6	183	172	137	137
q7	1958	1885	1712	1712
q8	2520	2670	2617	2617
q9	6980	6917	6837	6837
q10	2940	3186	2678	2678
q11	570	527	496	496
q12	649	714	569	569
q13	3356	3763	3172	3172
q14	269	309	272	272
q15	564	507	510	507
q16	647	687	643	643
q17	1187	1706	1231	1231
q18	7326	7116	7071	7071
q19	845	1168	1066	1066
q20	1940	2005	1833	1833
q21	5505	5051	4760	4760
q22	615	608	594	594
Total cold run time: 52627 ms
Total hot run time: 51055 ms

doris-robot · 2025-01-08T14:51:37Z

TPC-DS: Total hot run time: 189893 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 06227897b1ac8eea9c4be3607787f23d62da03d1, data reload: false

query1	872	393	400	393
query2	4630	2441	2427	2427
query3	5161	219	218	218
query4	31999	23665	23411	23411
query5	2803	635	489	489
query6	270	200	178	178
query7	4255	477	310	310
query8	301	242	239	239
query9	7944	2761	2756	2756
query10	455	308	251	251
query11	18156	15317	15162	15162
query12	144	110	107	107
query13	1545	554	395	395
query14	10047	7238	7171	7171
query15	254	208	190	190
query16	8128	624	430	430
query17	1623	762	555	555
query18	2091	389	301	301
query19	196	187	158	158
query20	119	108	109	108
query21	209	122	102	102
query22	4369	4373	4136	4136
query23	34985	33290	33958	33290
query24	6742	2320	2292	2292
query25	453	443	387	387
query26	1054	257	167	167
query27	2026	461	324	324
query28	5127	2473	2476	2473
query29	530	511	431	431
query30	274	188	158	158
query31	929	857	787	787
query32	91	66	61	61
query33	524	354	331	331
query34	753	856	506	506
query35	831	825	729	729
query36	1035	1057	978	978
query37	130	102	81	81
query38	4238	4183	4035	4035
query39	1448	1426	1409	1409
query40	199	115	102	102
query41	49	47	46	46
query42	122	111	103	103
query43	529	528	492	492
query44	1347	823	824	823
query45	189	173	173	173
query46	862	1035	654	654
query47	1858	1822	1738	1738
query48	400	412	342	342
query49	857	487	416	416
query50	635	652	400	400
query51	7106	7015	6855	6855
query52	107	103	92	92
query53	226	260	188	188
query54	499	496	408	408
query55	80	85	79	79
query56	263	266	253	253
query57	1150	1147	1086	1086
query58	243	244	236	236
query59	3087	3221	2978	2978
query60	277	270	258	258
query61	145	114	116	114
query62	838	772	702	702
query63	236	211	201	201
query64	4244	1037	670	670
query65	3324	3161	3236	3161
query66	926	457	341	341
query67	15692	15602	15354	15354
query68	5963	735	536	536
query69	439	306	267	267
query70	1219	1164	1147	1147
query71	376	293	270	270
query72	6127	3905	3910	3905
query73	648	745	364	364
query74	9645	8949	8880	8880
query75	3155	3191	2675	2675
query76	3083	1173	863	863
query77	390	361	301	301
query78	9919	9903	9334	9334
query79	2902	793	625	625
query80	683	512	466	466
query81	504	274	235	235
query82	451	154	123	123
query83	180	174	162	162
query84	290	93	75	75
query85	843	356	314	314
query86	439	322	271	271
query87	4417	4271	4249	4249
query88	5083	2216	2199	2199
query89	398	328	296	296
query90	1858	197	198	197
query91	138	141	114	114
query92	66	58	54	54
query93	2243	853	517	517
query94	701	391	295	295
query95	338	274	266	266
query96	497	628	280	280
query97	2785	2973	2826	2826
query98	244	211	198	198
query99	1443	1498	1398	1398
Total cold run time: 279698 ms
Total hot run time: 189893 ms

doris-robot · 2025-01-08T14:56:57Z

ClickBench: Total hot run time: 31.05 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 06227897b1ac8eea9c4be3607787f23d62da03d1, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.04
query3	0.24	0.07	0.07
query4	1.62	0.11	0.10
query5	0.43	0.43	0.40
query6	1.17	0.65	0.66
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.60	0.50	0.50
query10	0.54	0.56	0.55
query11	0.14	0.11	0.10
query12	0.14	0.12	0.11
query13	0.60	0.60	0.60
query14	2.86	2.82	2.82
query15	0.89	0.82	0.83
query16	0.39	0.37	0.38
query17	1.04	1.08	1.07
query18	0.23	0.21	0.21
query19	1.90	1.83	2.04
query20	0.01	0.01	0.01
query21	15.36	0.92	0.59
query22	0.75	0.86	0.72
query23	15.14	1.48	0.63
query24	3.28	1.35	0.44
query25	0.26	0.11	0.11
query26	0.26	0.16	0.13
query27	0.06	0.04	0.06
query28	13.61	1.53	1.05
query29	12.58	3.98	3.30
query30	0.25	0.09	0.06
query31	2.82	0.60	0.38
query32	3.23	0.56	0.47
query33	3.01	3.11	3.13
query34	16.73	5.11	4.46
query35	4.54	4.46	4.47
query36	0.66	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.18	0.14	0.13
query41	0.07	0.02	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 106.01 s
Total hot run time: 31.05 s

doris-robot · 2025-01-08T15:00:26Z

TeamCity be ut coverage result:
Function Coverage: 38.87% (10130/26061)
Line Coverage: 29.93% (85737/286475)
Region Coverage: 29.02% (43735/150704)
Branch Coverage: 25.56% (22323/87340)
Coverage Report: http://coverage.selectdb-in.cc/coverage/06227897b1ac8eea9c4be3607787f23d62da03d1_06227897b1ac8eea9c4be3607787f23d62da03d1/report/index.html

924060929 · 2025-01-08T17:55:52Z

run external

924060929 · 2025-01-08T17:56:39Z

run external_p0

924060929 · 2025-01-09T14:00:55Z

run buildall

doris-robot · 2025-01-09T14:44:14Z

TeamCity be ut coverage result:
Function Coverage: 39.34% (10252/26058)
Line Coverage: 30.52% (87405/286395)
Region Coverage: 29.56% (44542/150678)
Branch Coverage: 26.12% (22810/87322)
Coverage Report: http://coverage.selectdb-in.cc/coverage/cf12af01ad1d6be56654958953edc8fbad48fe68_cf12af01ad1d6be56654958953edc8fbad48fe68/report/index.html

doris-robot · 2025-01-09T14:48:45Z

TPC-H: Total hot run time: 32393 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cf12af01ad1d6be56654958953edc8fbad48fe68, data reload: false

------ Round 1 ----------------------------------
q1	17581	6167	6080	6080
q2	2048	314	170	170
q3	10404	1237	721	721
q4	10188	863	430	430
q5	7525	2157	1973	1973
q6	203	178	148	148
q7	895	755	598	598
q8	9254	1349	1086	1086
q9	5067	4865	4818	4818
q10	6761	2309	1849	1849
q11	506	281	258	258
q12	341	371	224	224
q13	17740	3691	3081	3081
q14	229	233	214	214
q15	559	503	490	490
q16	631	606	587	587
q17	557	848	330	330
q18	6854	6537	6291	6291
q19	1537	938	546	546
q20	316	319	191	191
q21	2896	2181	1982	1982
q22	362	346	326	326
Total cold run time: 102454 ms
Total hot run time: 32393 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6272	6207	6226	6207
q2	239	327	237	237
q3	2206	2682	2300	2300
q4	1394	1809	1366	1366
q5	4317	4752	4675	4675
q6	183	176	144	144
q7	2080	1934	1765	1765
q8	2572	2819	2654	2654
q9	7372	7170	7173	7170
q10	3062	3336	2774	2774
q11	570	511	499	499
q12	654	755	593	593
q13	3521	3877	3280	3280
q14	281	294	275	275
q15	570	523	504	504
q16	669	696	645	645
q17	1237	1720	1250	1250
q18	7622	7389	7419	7389
q19	814	1125	1072	1072
q20	1984	2061	1843	1843
q21	5745	5387	4935	4935
q22	607	625	619	619
Total cold run time: 53971 ms
Total hot run time: 52196 ms

doris-robot · 2025-01-09T15:01:01Z

TPC-DS: Total hot run time: 195394 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cf12af01ad1d6be56654958953edc8fbad48fe68, data reload: false

query1	1317	972	953	953
query2	6239	2381	2313	2313
query3	10973	4481	4619	4481
query4	36159	23893	23468	23468
query5	4990	602	474	474
query6	305	207	196	196
query7	4000	493	314	314
query8	338	258	242	242
query9	9457	2736	2703	2703
query10	519	323	257	257
query11	18120	15217	15010	15010
query12	156	106	101	101
query13	1704	520	384	384
query14	11792	7154	6765	6765
query15	258	206	185	185
query16	7300	584	446	446
query17	1385	733	578	578
query18	1833	395	313	313
query19	206	194	171	171
query20	124	113	113	113
query21	203	135	106	106
query22	4520	4486	4464	4464
query23	34072	34626	34073	34073
query24	6342	2420	2277	2277
query25	464	462	404	404
query26	759	269	162	162
query27	2097	461	336	336
query28	5180	2508	2448	2448
query29	516	542	428	428
query30	227	191	153	153
query31	973	906	840	840
query32	68	59	58	58
query33	478	361	309	309
query34	749	859	528	528
query35	787	820	771	771
query36	1003	1062	956	956
query37	130	107	82	82
query38	4217	4169	4077	4077
query39	1542	1439	1427	1427
query40	214	126	114	114
query41	64	62	59	59
query42	131	106	106	106
query43	530	527	504	504
query44	1339	852	855	852
query45	191	171	219	171
query46	882	1053	691	691
query47	1875	1879	1856	1856
query48	397	421	317	317
query49	703	483	415	415
query50	637	666	421	421
query51	7089	6970	7058	6970
query52	103	101	99	99
query53	222	246	186	186
query54	478	522	423	423
query55	90	82	77	77
query56	258	256	248	248
query57	1225	1192	1099	1099
query58	259	233	247	233
query59	3120	3245	3154	3154
query60	281	259	249	249
query61	121	117	120	117
query62	856	782	734	734
query63	234	190	202	190
query64	2108	1027	678	678
query65	3384	3257	3240	3240
query66	699	439	308	308
query67	16010	15819	15479	15479
query68	8485	749	523	523
query69	471	304	258	258
query70	1221	1126	1121	1121
query71	445	289	257	257
query72	6296	3841	3757	3757
query73	671	736	363	363
query74	10063	8943	8751	8751
query75	4644	3121	2643	2643
query76	4190	1177	765	765
query77	818	356	269	269
query78	10209	10078	9316	9316
query79	2414	808	606	606
query80	583	528	438	438
query81	488	292	238	238
query82	443	144	117	117
query83	220	175	147	147
query84	244	88	71	71
query85	810	356	294	294
query86	381	313	295	295
query87	4338	4409	4350	4350
query88	4478	2179	2266	2179
query89	400	329	280	280
query90	1760	188	185	185
query91	132	136	108	108
query92	65	55	49	49
query93	2092	865	537	537
query94	650	397	288	288
query95	328	277	244	244
query96	486	604	279	279
query97	2847	2872	2752	2752
query98	221	200	196	196
query99	1384	1489	1362	1362
Total cold run time: 297279 ms
Total hot run time: 195394 ms

doris-robot · 2025-01-09T15:06:21Z

ClickBench: Total hot run time: 31.41 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cf12af01ad1d6be56654958953edc8fbad48fe68, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.04
query3	0.24	0.06	0.07
query4	1.62	0.10	0.10
query5	0.41	0.42	0.41
query6	1.15	0.65	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.59	0.51	0.50
query10	0.56	0.57	0.56
query11	0.15	0.10	0.10
query12	0.15	0.12	0.11
query13	0.61	0.60	0.60
query14	2.71	2.76	2.86
query15	0.90	0.84	0.81
query16	0.38	0.38	0.37
query17	1.03	1.05	1.04
query18	0.24	0.21	0.21
query19	1.97	1.87	2.00
query20	0.02	0.01	0.01
query21	15.37	0.95	0.58
query22	0.76	0.80	0.69
query23	15.26	1.50	0.55
query24	3.03	1.32	1.05
query25	0.19	0.24	0.09
query26	0.17	0.14	0.13
query27	0.05	0.08	0.05
query28	13.87	1.52	1.04
query29	12.55	3.91	3.27
query30	0.25	0.09	0.08
query31	2.82	0.60	0.38
query32	3.22	0.53	0.46
query33	3.19	3.06	3.20
query34	16.84	5.07	4.49
query35	4.46	4.47	4.45
query36	0.64	0.50	0.48
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.16	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.02	0.02
Total cold run time: 106.06 s
Total hot run time: 31.41 s

github-actions · 2025-01-10T02:07:57Z

PR approved by at least one committer and no changes requested.

924060929 changed the title ~~[opt](coordinator) optimize shuffle parallel when use nereids~~ [opt](coordinator) optimize parallel degree of shuffle when use nereids Nov 28, 2024

morrySnow previously approved these changes Nov 28, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 28, 2024

github-actions bot added the reviewed label Nov 28, 2024

924060929 force-pushed the opt_parallel branch from 7e13881 to f16364c Compare December 17, 2024 08:12

924060929 dismissed morrySnow’s stale review via 0b5a164 December 17, 2024 08:27

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Dec 17, 2024

924060929 force-pushed the opt_parallel branch from c48cdb7 to b5f0211 Compare December 17, 2024 11:37

924060929 force-pushed the opt_parallel branch from b5f0211 to d461498 Compare December 18, 2024 04:21

924060929 force-pushed the opt_parallel branch from aa4e689 to d78387b Compare December 18, 2024 13:31

924060929 force-pushed the opt_parallel branch from d78387b to 3a44f7d Compare December 19, 2024 11:02

924060929 force-pushed the opt_parallel branch from 406f6c5 to 97e762c Compare December 20, 2024 07:04

test

4770b30

test

0622789

fix

cf12af0

starocean999 approved these changes Jan 10, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 10, 2025

englefly approved these changes Jan 10, 2025

View reviewed changes

924060929 merged commit 584a256 into apache:master Jan 10, 2025
24 of 26 checks passed

924060929 deleted the opt_parallel branch January 10, 2025 03:58

[opt](coordinator) optimize parallel degree of shuffle when use nereids #44754

[opt](coordinator) optimize parallel degree of shuffle when use nereids #44754

Conversation

924060929 commented Nov 28, 2024 • edited Loading

What problem does this PR solve?

Check List (For Author)

Check List (For Reviewer who merge this PR)

924060929 commented Nov 28, 2024

doris-robot commented Nov 28, 2024

github-actions bot commented Nov 28, 2024

github-actions bot commented Nov 28, 2024

924060929 commented Dec 17, 2024

924060929 commented Dec 17, 2024

924060929 commented Dec 17, 2024

924060929 commented Dec 17, 2024

924060929 commented Dec 18, 2024

github-actions bot commented Dec 18, 2024

sh-checker report

924060929 commented Dec 18, 2024

github-actions bot commented Dec 18, 2024

sh-checker report

924060929 commented Dec 18, 2024

github-actions bot commented Dec 18, 2024

sh-checker report

924060929 commented Dec 18, 2024

924060929 commented Dec 19, 2024

924060929 commented Dec 20, 2024

doris-robot commented Dec 20, 2024

doris-robot commented Jan 8, 2025

doris-robot commented Jan 8, 2025

doris-robot commented Jan 8, 2025

doris-robot commented Jan 8, 2025

924060929 commented Jan 8, 2025

doris-robot commented Jan 8, 2025

doris-robot commented Jan 8, 2025

doris-robot commented Jan 8, 2025

doris-robot commented Jan 8, 2025

924060929 commented Jan 8, 2025

doris-robot commented Jan 8, 2025

doris-robot commented Jan 8, 2025

doris-robot commented Jan 8, 2025

doris-robot commented Jan 8, 2025

924060929 commented Jan 8, 2025

924060929 commented Jan 8, 2025

924060929 commented Jan 9, 2025

doris-robot commented Jan 9, 2025

doris-robot commented Jan 9, 2025

doris-robot commented Jan 9, 2025

doris-robot commented Jan 9, 2025

github-actions bot commented Jan 10, 2025

924060929 commented Nov 28, 2024 •

edited

Loading

`sh-checker report`

`sh-checker report`

`sh-checker report`