Add --compiled-modules=existing command line option
#50586
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Occasionally when developing new compiler features, it is not possible for me to precompile the package I'm developing (because it relies on the new compiler features, which are only available if Revise'd in). `--compiled-modules=no` is an option, but for packages with deep dependency stacks, this is not practical (in addition to slowing down use of development utilities like Revise, Cthulhu or Plots). Originally I tried to add a mode to `--compiled-modules` that would avoid using compiled modules for anything that's dev'ed, but that's a pretty complicated thing to figure out in the loading code, because you could have a non-`dev`'ed package that depends on a `dev`'ed package and you'd want to avoid loading that as well, but we don't technically have explicit dependency links at the loading level. This sidesteps all that and just adds a simpler option: `nonew`. This option simply uses any pre-existing cache files if they exist, but refuses to create new ones. This does effectively the same thing, because the only packages with stale cache files are usually the ones that I've edited. However, the semantics are much simpler for loading to implement.
| @@ -101,7 +101,7 @@ The following is a complete list of command-line switches available when launchi | |||
| |`--startup-file={yes*\|no}` |Load `JULIA_DEPOT_PATH/config/startup.jl`; if `JULIA_DEPOT_PATH` environment variable is unset, load `~/.julia/config/startup.jl`| | |||
| |`--handle-signals={yes*\|no}` |Enable or disable Julia's default signal handlers| | |||
| |`--sysimage-native-code={yes*\|no}` |Use native code from system image if available| | |||
| |`--compiled-modules={yes*\|no}` |Enable or disable incremental precompilation of modules| | |||
| |`--compiled-modules={yes*\|no|nonew}` |Enable or disable incremental precompilation of modules| | |||
There was a problem hiding this comment.
Needs some extra docs. How about
nonew uses any precompiled modules that already exist, but will not compile new modules.
There was a problem hiding this comment.
Reasonable to put in the docs, but I'd keep it out of the cli usage note, since it's a bit of a niche flag.
|
This will also be useful for CReduce+Julia!
Maybe |
Sure |
oscardssmith
changed the title
--compiled-modules=nonew command line option--compiled-modules=existing command line option
KristofferC
pushed a commit
that referenced
this pull request
Dec 19, 2023
Equivalent of #50586; implements #51474. With `--pkgimages=existing`, it's possible to disable the (often slow) generation of package images, without losing the ability to use existing ones. That's important now that we're moving more and more packages outside of the system image, e.g., running with `--pkgimages=no` otherwise takes close to 30s here before the Pkg REPL is usable. The main motivation for this is PkgEval, where generating package images is not very useful, yet disabling generation of them makes each job (which requires Pkg to drive the test process) take a significantly longer time. For example, `--pkgimages=yes` vs `no`: ```julia ❯ JULIA_DEBUG=loading ./julia --project=Example.jl --pkgimages=yes # no precompilation of REPL.jl ┌ Debug: Loading object cache file /Users/tim/Julia/src/julia/build/dev/usr/share/julia/compiled/v1.11/REPL/u0gqU_XmENM.dylib for REPL [3fa0cd96-eef1-5676-8a61-b3b8758bbffb] └ @ Base loading.jl:1116 julia> using Example # short time precompiling + pkgimg generation for Example.jl ┌ Debug: Loading object cache file /Users/tim/.julia/compiled/v1.11/Example/lLvWP_tJaso.dylib for Example [7876af07-990d-54b4-ab0e-23690620f79a] └ @ Base loading.jl:1116 ``` ```julia ❯ JULIA_DEBUG=loading ./julia --project=Example.jl --pkgimages=no ┌ Debug: Rejecting cache file /Users/tim/Julia/src/julia/build/dev/usr/share/julia/compiled/v1.11/REPL/u0gqU_XmENM.ji for REPL [3fa0cd96-eef1-5676-8a61-b3b8758bbffb] since the flags are mismatched │ current session: use_pkgimages = false, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2 │ cache file: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2 └ @ Base loading.jl:3289 # long time precompiling REPL.jl ┌ Debug: Loading cache file /Users/tim/.julia/compiled/v1.11/REPL/u0gqU_CWvWI.ji for REPL [3fa0cd96-eef1-5676-8a61-b3b8758bbffb] └ @ Base loading.jl:1119 julia> using Example # short time precompiling Example ┌ Debug: Loading cache file /Users/tim/.julia/compiled/v1.11/Example/lLvWP_CWvWI.ji for Example [7876af07-990d-54b4-ab0e-23690620f79a] └ @ Base loading.jl:1119 ``` With the new `--pkgimages=existing`: ```julia ❯ JULIA_DEBUG=loading ./julia --project=Example.jl --pkgimages=existing # no precompilation of REPL.jl ┌ Debug: Loading object cache file /Users/tim/Julia/src/julia/build/dev/usr/share/julia/compiled/v1.11/REPL/u0gqU_XmENM.dylib for REPL [3fa0cd96-eef1-5676-8a61-b3b8758bbffb] └ @ Base loading.jl:1116 julia> using Example # short time precompiling Example ┌ Debug: Loading cache file /Users/tim/.julia/compiled/v1.11/Example/lLvWP_CWvWI.ji for Example [7876af07-990d-54b4-ab0e-23690620f79a] └ @ Base loading.jl:1119 ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Occasionally when developing new compiler features, it is not possible for me to precompile the package I'm developing (because it relies on the new compiler features, which are only available if Revise'd in).
--compiled-modules=nois an option, but for packages with deep dependency stacks, this is not practical (in addition to slowing down use of development utilities like Revise, Cthulhu or Plots). Originally I tried to add a mode to--compiled-modulesthat would avoid using compiled modules for anything that's dev'ed, but that's a pretty complicated thing to figure out in the loading code, because you could have a non-dev'ed package that depends on adev'ed package and you'd want to avoid loading that as well, but we don't technically have explicit dependency links at the loading level. This sidesteps all that and just adds a simpler option:nonewexisting. This option simply uses any pre-existing cache files if they exist, but refuses to create new ones. This does effectively the same thing, because the only packages with stale cache files are usually the ones that I've edited. However, the semantics are much simpler for loading to implement.