Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Speed up the build and make it use less memory #744

Open
kordejong opened this issue Nov 30, 2024 · 4 comments
Open

Speed up the build and make it use less memory #744

kordejong opened this issue Nov 30, 2024 · 4 comments
Assignees
@kordejong
Labels
05: target: all All targets or not related to a specific target 15: feature: configuration Configuration stuff: cmake, ... 25: kind: enhancement 35: medium priority I need this soon
Milestone
0.3.10

Comments

@kordejong
Copy link
Member

kordejong commented Nov 30, 2024 •
edited
Loading

Building Conda packages on Conda-Forge resources will likely timeout soon. The Windows builds take sometimes almost six hours and the Windows runners complain that they are running out of memory.

See if we can profile the build:

  • Time used to build each target, including individual object files
  • Memory used to build each target

Based on profile info, figure out how to speed up the build and make it use less memory.

See if using precompiled headers speed up the compilation significantly:

  • They are now easy to use from CMake. Source files don't have to be changed.
  • At first, make this an optional thing, default off
  • Get some experience with it. Time it w/o using ccache and with. Compare influence on size of build artifacts.
  • Start with partitioned array headers
@kordejong kordejong added 25: kind: enhancement 05: target: all All targets or not related to a specific target 15: feature: configuration Configuration stuff: cmake, ... 35: medium priority I need this soon labels Nov 30, 2024
@kordejong kordejong added this to the 0.3.10 milestone Nov 30, 2024
@kordejong
Copy link
Member Author

Example from QGIS: qgis/QGIS#59667

@kordejong kordejong changed the title See if using precompiled headers speed up the compilation significantly Speed up the build and make it use less memory Dec 4, 2024
@kordejong kordejong self-assigned this Dec 4, 2024
@kordejong
Copy link
Member Author

Ninja dumps a file called .ninja.log which contains information about how long it took to build each translation unit. A tool called ninjatracing can then be used to convert that information into a trace file which can be visualized.

See also: https://crascit.com/2022/06/24/build-performance-insights/

@kordejong
Copy link
Member Author

It is possible to query the amount of memory and limit the number of parallel jobs used during the build:

set_property(GLOBAL PROPERTY JOB_POOLS one_jobs=1 two_jobs=2)
cmake_host_system_information(RESULT _memfree QUERY AVAILABLE_PHYSICAL_MEMORY)
add_library(big big1.c big2.f90)
if(_memfree LESS 1000)
  set_property(TARGET big PROPERTY JOB_POOL_COMPILE one_jobs)
endif()

@kordejong
Copy link
Member Author

https://devblogs.microsoft.com/cppblog/finding-build-bottlenecks-with-cpp-build-insights/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kordejong kordejong

Labels
05: target: all All targets or not related to a specific target 15: feature: configuration Configuration stuff: cmake, ... 25: kind: enhancement 35: medium priority I need this soon
Projects
None yet
Milestone
0.3.10
Development

No branches or pull requests
1 participant
@kordejong