Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[question] MsBuild tools.microsoft.msbuild:max_cpu_count #17290

Closed
1 task
fellinga opened this issue Nov 7, 2024 · 6 comments · Fixed by #17292 or #17301
Closed
1 task

[question] MsBuild tools.microsoft.msbuild:max_cpu_count #17290

fellinga opened this issue Nov 7, 2024 · 6 comments · Fixed by #17292 or #17301
Assignees

Comments

@fellinga
Copy link

fellinga commented Nov 7, 2024

Hello,

we are using conan in CI/CD and we were wondering if we can utilize all of the CPUs in a MsBuild build when calling "conan build .".
We already know that we can mimic /maxcpucount:2 with tools.microsoft.msbuild:max_cpu_count=2 in our default.conf file, but how can we set /maxcpucount without any arguments? Because setting /maxcpucount would use all available CPUs by default... "If you include the -maxcpucount switch without specifying a value, MSBuild will use up to the number of processors on the computer." (source).

Another question: is there a better way to set tools.microsoft.msbuild:max_cpu_count=2 in CI/CD besides manually appending it to the default.conf?

Thanks in advance!

Have you read the CONTRIBUTING guide?

  • I've read the CONTRIBUTING guide
@memsharded memsharded self-assigned this Nov 7, 2024
@memsharded
Copy link
Member

Hi @fellinga

Thanks for your question.

We already know that we can mimic /maxcpucount:2 with tools.microsoft.msbuild:max_cpu_count=2 in our default.conf file, but how can we set /maxcpucount without any arguments?

It is not clear if you are using MSBuild (VS native project) or via CMake. In any case, note that you can also use the tools.build:jobs conf, and it will define the <ProcessorNumber>{}</ProcessorNumber> in MSBuildToolchain or to /MP flags in CMakeToolchain. This is equally or even more important for multi-processor compilation parallelism than max_cpu_count.

Because setting /maxcpucount would use all available CPUs by default.

Not really, if the solution doesn't have multiple projects to build in parallel. For intra-project compilation parallelism, you need the above conf.

 njobs = conanfile.conf.get("tools.build:jobs",
                               default=_cpu_count(),
                               check_type=int)

This is how the default number of jobs is computed, by default it is already the cpu_count, and that is used for the compilation parallelism

I see it seems there is a small bug in the max_cpu_count management for CMake, but need clarification if you are using it or not.

@memsharded
Copy link
Member

One thing more: the max_cpu_count is not set by default to cpu_count, because our experience is that having both /MP and /m:x flags at the same time to the number of cpus, is counter productive, it can easily become slower because having too many concurrent process. I wouldn't recommend setting max_cpu_count to the number of cpu-counts, at least without deactivating or reducing first the tools.build:jobs

@fellinga
Copy link
Author

fellinga commented Nov 8, 2024

Hi @memsharded,

thanks for answering so quickly! We use MSBuild with VisualStudio and do not use CMake at all.

I'll try to explain my issue a little better, sorry if it was not clear enough:
Currently we run "MSBuild.exe mySolution.sln /maxcpucount /p:Configuration=Release in gitlab-ci and are quite happy with it but want to change this command to "conan install . " so that we can use a common pipeline for all projects. Therefore we added:

def build(self): msbuild = MSBuild(self) msbuild.build("mySolution.sln")

to our conanfiles. Now when i call "conan install . " it generates "MSBuild.exe mySolution.sln /p:Configuration=Release" which is almost exactly what we want besides the missing maxcpucount argument. I do not want to specify a hardcoded number of cores since my CI runners do not all have the same amount of CPUs. How can we solve this issue?

Thanks!

@memsharded
Copy link
Member

Understood. Yes, I think this specific use case is not covered by the current conf value, but I think it might be doable to take it into account, i will reopen (it was closed by by #17292 and will give it a try.

Thanks for the feedback!

@memsharded
Copy link
Member

Proposing #17301, and the maxcpucount=0 value to mean "all cores", I think it is better than inventing some other arbitrary string.

@memsharded
Copy link
Member

#17301 was merged, so using maxcpucount=0 Conan conf will mean => /m which means all cores. It will be in next Conan 2.10 release. Thanks for the feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants