Repeated calls to isfile to look for identical ji files in precompile causes high disk pressure #56366
Closed
Description
Running something like:
@eval Base begin
const isfile_cnt = Dict{String, Int64}()
function Base.Filesystem.isfile(st::StatStruct)
if haskey(isfile_cnt, st.desc)
isfile_cnt[st.desc] += 1
else
isfile_cnt[st.desc] = 1
end
filemode(st) & 0xf000 == 0x8000
end
end
empty!(Base.isfile_cnt)
using Pkg
Pkg.precompile()
@show sort(collect(Base.isfile_cnt); by=x->x[2], rev=true)gives for a big environment (e.g. DifferentialEquations.jl):
"~/.julia/compiled/v1.10/Preferences/pWSk8_J52EP.ji" => 152
"~/.julia/compiled/v1.10/Preferences/pWSk8_bmGsQ.ji" => 152
"~/julia/compiled/v1.10/SuiteSparse_jll/ME9At_BncKj.ji" => 124
"~/julia/compiled/v1.10/SuiteSparse_jll/ME9At_rJPx9.ji" => 124
"~/julia/compiled/v1.10/SuiteSparse_jll/ME9At_9T2BZ.ji" => 124
"~/.julia/compiled/v1.10/PrecompileTools/AQ9Mk_bmGsQ.ji" => 123
"~/.julia/compiled/v1.10/PrecompileTools/AQ9Mk_J52EP.ji" => 123
"~/julia/compiled/v1.10/SparseArrays/P9ieR_eNkqo.ji" => 122
"~/julia/compiled/v1.10/SparseArrays/P9ieR_BncKj.ji" => 122
"~/julia/compiled/v1.10/SparseArrays/P9ieR_rJPx9.ji" => 122
"~/.julia/compiled/v1.10/ArrayInterface/7bROb_EHPmX.ji" => 112
"~/.julia/compiled/v1.10/DocStringExtensions/KRdZs_bmGsQ.ji" => 111
where we are checking the same file over and over.
This seems to come from
Line 1834 in 2cdfe06
Noticed by @topolarity when a no-op instantiate was slow on Windows
Activity