Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](macOS) Fix build scripts for macOS #45751

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[fix](macOS) Fix build scripts for macOS #45751

wants to merge 1 commit into from

Conversation

0xderek
Copy link

@0xderek 0xderek commented Dec 20, 2024 •
edited
Loading

What problem does this PR solve?

Issue Number: close #45750

Related PR: None

Problem Summary: Build scripts do not work on macOS

Release note

  • Replaced nproc with sysctl -n hw.logicalcpu to get CPU count on macOS.
  • Avoided lowercase parameter expansion due to macOS default Bash version (3.2).

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Dec 20, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@0xderek
Copy link
Author

0xderek commented Dec 23, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39569 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c557afd6a4f286d500a182e89c2e7bedf5ac707d, data reload: false

------ Round 1 ----------------------------------
q1	17586	7458	7260	7260
q2	2049	183	180	180
q3	10630	1089	1134	1089
q4	10578	746	770	746
q5	7603	2675	2638	2638
q6	238	154	147	147
q7	980	634	585	585
q8	9466	1852	1913	1852
q9	6712	6390	6319	6319
q10	6950	2277	2308	2277
q11	462	261	248	248
q12	424	218	225	218
q13	18030	3002	3132	3002
q14	240	216	230	216
q15	580	533	495	495
q16	675	594	578	578
q17	980	581	580	580
q18	7305	6559	6518	6518
q19	1346	1074	1121	1074
q20	480	192	190	190
q21	3984	3418	3048	3048
q22	383	324	309	309
Total cold run time: 107681 ms
Total hot run time: 39569 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7216	7246	7251	7246
q2	330	230	235	230
q3	2913	2780	2896	2780
q4	2087	1791	1833	1791
q5	5642	5634	5623	5623
q6	227	141	145	141
q7	2213	1796	1822	1796
q8	3393	3519	3452	3452
q9	8767	8900	8894	8894
q10	3584	3506	3549	3506
q11	609	516	531	516
q12	814	630	594	594
q13	12373	3144	3094	3094
q14	307	285	290	285
q15	570	497	519	497
q16	676	651	644	644
q17	1851	1609	1606	1606
q18	8315	7856	7753	7753
q19	1729	1544	1515	1515
q20	2079	1850	1891	1850
q21	5658	5407	5489	5407
q22	660	577	626	577
Total cold run time: 72013 ms
Total hot run time: 59797 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.80% (10094/26015)
Line Coverage: 29.78% (85117/285862)
Region Coverage: 28.90% (43464/150403)
Branch Coverage: 25.43% (22153/87116)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c557afd6a4f286d500a182e89c2e7bedf5ac707d_c557afd6a4f286d500a182e89c2e7bedf5ac707d/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 196581 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c557afd6a4f286d500a182e89c2e7bedf5ac707d, data reload: false

query1	1291	929	945	929
query2	6258	2297	2360	2297
query3	10996	4555	4500	4500
query4	33159	23313	23360	23313
query5	4120	459	493	459
query6	272	179	178	178
query7	3995	306	312	306
query8	290	225	253	225
query9	9496	2779	2730	2730
query10	454	242	240	240
query11	17760	15319	15040	15040
query12	155	107	102	102
query13	1563	436	425	425
query14	8602	7448	7641	7448
query15	272	190	198	190
query16	8221	478	437	437
query17	1626	613	608	608
query18	2246	323	334	323
query19	381	177	178	177
query20	126	119	120	119
query21	207	110	113	110
query22	4792	4740	4429	4429
query23	34861	33597	33887	33597
query24	10532	2500	2554	2500
query25	661	459	401	401
query26	1530	151	151	151
query27	2890	329	334	329
query28	7865	2492	2507	2492
query29	937	433	425	425
query30	222	151	151	151
query31	1028	840	906	840
query32	104	58	55	55
query33	768	301	287	287
query34	1146	518	531	518
query35	884	760	752	752
query36	1120	947	984	947
query37	146	73	75	73
query38	4485	4132	4253	4132
query39	1550	1463	1454	1454
query40	266	110	102	102
query41	45	42	46	42
query42	119	102	100	100
query43	540	497	500	497
query44	1354	831	844	831
query45	196	172	175	172
query46	1181	743	734	734
query47	2066	1946	1994	1946
query48	438	332	334	332
query49	950	391	390	390
query50	852	414	399	399
query51	7220	7230	7257	7230
query52	101	102	92	92
query53	264	185	185	185
query54	987	420	405	405
query55	90	83	76	76
query56	260	244	244	244
query57	1301	1173	1173	1173
query58	238	223	215	215
query59	3360	3146	3227	3146
query60	282	264	246	246
query61	113	105	106	105
query62	926	699	677	677
query63	219	190	186	186
query64	4806	708	669	669
query65	3331	3299	3264	3264
query66	971	322	305	305
query67	16647	15574	15446	15446
query68	5949	553	562	553
query69	511	264	249	249
query70	1206	1162	1129	1129
query71	490	262	240	240
query72	6502	4143	3971	3971
query73	799	370	378	370
query74	10159	8754	8835	8754
query75	3858	2667	2669	2667
query76	3898	1057	1158	1057
query77	653	292	346	292
query78	10164	9564	9404	9404
query79	1880	613	616	613
query80	1202	421	422	421
query81	504	244	238	238
query82	459	128	117	117
query83	192	142	145	142
query84	285	75	73	73
query85	1112	309	319	309
query86	350	261	307	261
query87	4827	4386	4416	4386
query88	3756	2240	2237	2237
query89	424	291	288	288
query90	2128	190	188	188
query91	173	107	105	105
query92	65	52	51	51
query93	2728	577	577	577
query94	969	284	282	282
query95	356	257	255	255
query96	632	277	275	275
query97	2776	2678	2719	2678
query98	215	195	192	192
query99	1623	1340	1309	1309
Total cold run time: 306979 ms
Total hot run time: 196581 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.57 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c557afd6a4f286d500a182e89c2e7bedf5ac707d, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.08	0.07
query4	1.59	0.10	0.11
query5	0.42	0.41	0.41
query6	1.14	0.64	0.64
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.56	0.52	0.50
query10	0.56	0.59	0.57
query11	0.14	0.10	0.10
query12	0.13	0.12	0.11
query13	0.60	0.61	0.59
query14	2.82	2.71	2.86
query15	0.90	0.81	0.82
query16	0.38	0.38	0.39
query17	1.06	1.06	1.02
query18	0.22	0.21	0.21
query19	1.99	1.89	1.98
query20	0.02	0.01	0.01
query21	15.36	0.58	0.58
query22	2.52	2.25	1.25
query23	17.02	0.97	0.88
query24	2.75	1.38	1.53
query25	0.11	0.15	0.10
query26	0.60	0.14	0.14
query27	0.04	0.04	0.05
query28	10.15	1.10	1.08
query29	12.56	3.25	3.24
query30	0.25	0.07	0.06
query31	2.85	0.39	0.37
query32	3.24	0.47	0.45
query33	3.10	3.06	3.06
query34	16.92	4.47	4.46
query35	4.50	4.46	4.49
query36	0.66	0.49	0.49
query37	0.10	0.06	0.06
query38	0.04	0.03	0.03
query39	0.04	0.03	0.02
query40	0.17	0.12	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 106.06 s
Total hot run time: 32.57 s

@0xderek
Copy link
Author

0xderek commented Jan 2, 2025

It has been some days. Can someone take a look? Thanks.

@0xderek
[fix](macOS) Fix build scripts for macOS
738b6aa 
- Replaced `nproc` with `sysctl -n hw.logicalcpu` to get CPU count on macOS.
- Avoided lowercase parameter expansion due to macOS default Bash version (3.2).
@0xderek 0xderek force-pushed the fix-script-on-mac branch from c557afd to 738b6aa Compare January 3, 2025 13:10
@0xderek
Copy link
Author

0xderek commented Jan 7, 2025

@Thearas
Hello, can you take a look? Thanks!

@Thearas
Copy link
Contributor

Thearas commented Jan 8, 2025

@Thearas Hello, can you take a look? Thanks!

Hi! Thanks for your contribution.
I'll ask the committer in charge of compilation to review it ASAP.

@hello-stephen
Copy link
Contributor

run buildall

hello-stephen
hello-stephen approved these changes Jan 8, 2025
View reviewed changes
Copy link
Contributor

@hello-stephen hello-stephen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 8, 2025
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 8, 2025

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 8, 2025

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 32634 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 738b6aa5a3611fabee70d459bd07d6311c7d91f9, data reload: false

------ Round 1 ----------------------------------
q1	17623	6103	6070	6070
q2	2048	304	167	167
q3	10420	1258	725	725
q4	10218	852	432	432
q5	7539	2260	1953	1953
q6	203	176	144	144
q7	881	742	612	612
q8	9244	1347	1149	1149
q9	5150	4923	4890	4890
q10	6757	2321	1838	1838
q11	466	280	265	265
q12	344	363	221	221
q13	17769	3645	3043	3043
q14	227	228	217	217
q15	555	511	490	490
q16	638	617	582	582
q17	563	840	323	323
q18	7052	6554	6472	6472
q19	1241	1000	566	566
q20	308	317	199	199
q21	2995	2160	1968	1968
q22	363	333	308	308
Total cold run time: 102604 ms
Total hot run time: 32634 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6259	6241	6372	6241
q2	233	323	227	227
q3	2226	2600	2365	2365
q4	1378	1835	1354	1354
q5	4340	4702	4737	4702
q6	184	178	139	139
q7	2107	1968	1822	1822
q8	2581	2766	2671	2671
q9	7321	7110	7215	7110
q10	2988	3323	2827	2827
q11	684	512	506	506
q12	659	754	586	586
q13	3491	3874	3234	3234
q14	303	311	265	265
q15	553	510	494	494
q16	638	692	657	657
q17	1219	1714	1256	1256
q18	7780	7461	7358	7358
q19	816	1058	1056	1056
q20	1908	1961	1872	1872
q21	5478	5157	4783	4783
q22	595	609	582	582
Total cold run time: 53741 ms
Total hot run time: 52107 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187599 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 738b6aa5a3611fabee70d459bd07d6311c7d91f9, data reload: false

query1	973	377	373	373
query2	6516	2490	2257	2257
query3	6708	214	215	214
query4	33498	23510	23186	23186
query5	4326	605	453	453
query6	287	199	188	188
query7	4639	505	311	311
query8	305	243	243	243
query9	9502	2734	2725	2725
query10	477	317	239	239
query11	17931	15189	14797	14797
query12	163	111	108	108
query13	1673	543	423	423
query14	10747	6889	6511	6511
query15	233	198	187	187
query16	8067	576	479	479
query17	1576	742	561	561
query18	2079	420	284	284
query19	213	175	153	153
query20	122	112	111	111
query21	212	123	99	99
query22	4335	4214	4345	4214
query23	34199	33168	32839	32839
query24	6320	2250	2249	2249
query25	488	461	381	381
query26	1206	269	153	153
query27	1999	448	324	324
query28	5168	2475	2432	2432
query29	717	525	428	428
query30	223	181	148	148
query31	919	870	775	775
query32	72	57	60	57
query33	504	346	282	282
query34	761	829	492	492
query35	811	820	718	718
query36	1006	1016	979	979
query37	116	104	78	78
query38	3992	4134	4049	4049
query39	1449	1438	1579	1438
query40	204	112	98	98
query41	48	43	49	43
query42	121	101	99	99
query43	518	539	488	488
query44	1292	818	818	818
query45	175	167	168	167
query46	839	1021	632	632
query47	1817	1886	1773	1773
query48	368	399	319	319
query49	770	469	375	375
query50	622	645	396	396
query51	6964	6965	6970	6965
query52	104	101	88	88
query53	226	247	183	183
query54	472	474	408	408
query55	80	76	78	76
query56	251	259	241	241
query57	1212	1120	1074	1074
query58	240	235	227	227
query59	3080	3179	3178	3178
query60	256	257	245	245
query61	109	104	109	104
query62	832	779	697	697
query63	225	192	187	187
query64	4269	1037	634	634
query65	3202	3151	3157	3151
query66	1042	421	301	301
query67	15846	15678	15378	15378
query68	7236	695	523	523
query69	467	291	245	245
query70	1219	1165	1154	1154
query71	443	279	255	255
query72	6223	3852	3816	3816
query73	662	741	357	357
query74	9348	8829	8585	8585
query75	4084	3141	2644	2644
query76	3719	1178	776	776
query77	787	373	279	279
query78	10097	10006	9476	9476
query79	3782	801	582	582
query80	647	510	437	437
query81	499	277	218	218
query82	667	150	119	119
query83	175	247	147	147
query84	245	89	73	73
query85	797	348	317	317
query86	409	318	306	306
query87	4501	4557	4469	4469
query88	4839	2231	2194	2194
query89	407	317	294	294
query90	1843	190	193	190
query91	148	132	106	106
query92	66	58	61	58
query93	1977	818	529	529
query94	663	398	277	277
query95	339	263	252	252
query96	490	608	288	288
query97	2874	2968	2744	2744
query98	235	206	205	205
query99	1428	1489	1401	1401
Total cold run time: 291032 ms
Total hot run time: 187599 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.68 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 738b6aa5a3611fabee70d459bd07d6311c7d91f9, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.03	0.04
query3	0.24	0.07	0.07
query4	1.61	0.10	0.10
query5	0.43	0.41	0.40
query6	1.16	0.65	0.64
query7	0.03	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.51	0.49
query10	0.56	0.56	0.56
query11	0.16	0.10	0.10
query12	0.13	0.10	0.11
query13	0.62	0.60	0.60
query14	2.75	2.87	2.75
query15	0.88	0.82	0.82
query16	0.38	0.37	0.39
query17	1.00	1.08	0.97
query18	0.24	0.21	0.21
query19	1.89	1.78	1.96
query20	0.01	0.01	0.01
query21	15.36	0.95	0.56
query22	0.75	0.78	0.61
query23	15.36	1.39	0.53
query24	3.58	0.51	1.71
query25	0.30	0.12	0.09
query26	0.26	0.16	0.14
query27	0.07	0.09	0.05
query28	13.86	1.52	1.05
query29	12.58	3.95	3.26
query30	0.26	0.09	0.07
query31	2.81	0.59	0.38
query32	3.25	0.55	0.46
query33	3.07	3.15	3.08
query34	16.78	5.15	4.54
query35	4.56	4.52	4.49
query36	0.82	0.50	0.48
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 106.97 s
Total hot run time: 30.68 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.86% (10128/26061)
Line Coverage: 29.92% (85706/286475)
Region Coverage: 29.02% (43729/150704)
Branch Coverage: 25.55% (22319/87340)
Coverage Report: http://coverage.selectdb-in.cc/coverage/738b6aa5a3611fabee70d459bd07d6311c7d91f9_738b6aa5a3611fabee70d459bd07d6311c7d91f9/report/index.html

@gavinchou
Copy link
Contributor

run p0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hello-stephen hello-stephen hello-stephen approved these changes

@gavinchou gavinchou gavinchou approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Bug] Build scripts do not work on macOS
5 participants
@0xderek @Thearas @doris-robot @hello-stephen @gavinchou